Robots.txt is a file that's only purpose is to communicate with web crawlers and robots. It tells these bots and crawlers which areas on your site to scan, and which to ignore. 

If your file isn't setup correctly, you may be blocking necessary components of the ad tech from properly crawling your site, and this can severely impact performance. 

We are checking to see 3 main things

1. Is there a robots.txt? Can we get to it? Are they blocking "the world" (example below)?

404 status code = GOOD (means your site is up, just doesn't have this file and doesn't explicitly tell anyone not to crawl it)

500 status code = BAD (your site could be down so skip it and don't crawl anything)

2. and 3. Are they blocking Grapeshot or Google Adsense ?

https://www.grapeshot.com/crawler/

"Grapeshot assists advertisers to contextually place adverts on pages, to do this it is necessary to examine, or crawl, the page to determine which category, or categories, it best matches."

**BAD (They are blocking google grapeshot)**

User-agent: grapeshot
Disallow: /

**BAD (they are blocking the world)**

User-agent: *
Disallow: /


**BAD (They are blocking google adsense)**

User-agent: Mediapartners-Google
Disallow: /

Please update the file via FTP, or contact your managed host or technical support for assistance. 

Yay! You are all set!

Did this answer your question?