3 replies to this topic

[dimmond]

I need a help about the robots.txt .. namely i want to say allow all bots except these three bots but i have no idea how. i.e.
User-Agent: *
Disallow:
this will allow all the robots
now if i add this:
User-agent: Zao
Disallow: /
will that prevent the Zao bot to crawl my website or when it sees the first lines will start to crawl ignoring the lines below?
Thanks

[Randy]

Welcome dimmond !

CODE

User-agent: Zao
Disallow: /

The above will disallow bot Zao from spidering anything on your site, assuming the bot in question actually follows robots.txt standards. Some bad bots (read: Scraper Bots) don't follow robots.txt instructions.

[dimmond]

Thanks Randy,

You are right about that but, please explain to me what would happen if i have allowed all bots in the same robots.txt and then disallowed some bots particulary.
I was wondering how they consider this situation. maybe they come to the first block where i am allowing all of them user-agent: * and when they see Disallow: nothing they start crawling ???

Thanks ones again,
Elizabeth Dimmond

[Randy]

ahhh, I get what you're asking now Elizabeth.

Per the specs the more specific stuff should come first if memory serves, though in reality I don't think it really matters since even good bots extend well beyond the original specs these days and attempt to account for common errors. They do read the file from the top down though, so it's always good practice follow the specs as closely as possible.

So to do your example per the specs would be:

CODE

User-agent: Zao
Disallow: /
User-agent: *
Disallow: