SEO Class in Chicago, IL
Are you a Google Analytics enthusiast?
More SEO Content
Disallow Some Bots
Posted 31 January 2008 - 05:18 AM
this will allow all the robots
now if i add this:
will that prevent the Zao bot to crawl my website or when it sees the first lines will start to crawl ignoring the lines below?
Posted 31 January 2008 - 06:56 AM
The above will disallow bot Zao from spidering anything on your site, assuming the bot in question actually follows robots.txt standards. Some bad bots (read: Scraper Bots) don't follow robots.txt instructions.
Posted 31 January 2008 - 04:26 PM
You are right about that but, please explain to me what would happen if i have allowed all bots in the same robots.txt and then disallowed some bots particulary.
I was wondering how they consider this situation. maybe they come to the first block where i am allowing all of them user-agent: * and when they see Disallow: nothing they start crawling ???
Thanks ones again,
Posted 01 February 2008 - 01:45 AM
Per the specs the more specific stuff should come first if memory serves, though in reality I don't think it really matters since even good bots extend well beyond the original specs these days and attempt to account for common errors. They do read the file from the top down though, so it's always good practice follow the specs as closely as possible.
So to do your example per the specs would be:
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users