Thursday, September 24, 2009

Block all except adsense bot in robots.txt

I ran into a problem today when i was adding adsense to one of my
phpBB forums ( Short Jokes )

I had implemented a url rewrite mod in this forum, so in order to avoid duplicate content I had to block my old urls being indexed (such as viewtopic.php?*** ). I wrote a robots.txt to block all bots from accessing such pages.

All went fine until it was time to implement adsense. I was not getting relevant ads on the old style urls since its getting blocked.

After a bit of googling I got this piece of code which solved the problem.
Put this on top of all in my robots.txt file.

User-agent: Mediapartners-Google
Disallow:
Allow: /*


Now let me watch it for sometime and see if the quality of ads is improving.

No comments: