28 replies to this topic

[Scottie]

Welcome to the forum, Guidaro!

Corey, I think I've welcomed you before, but just in case I haven't- to you too.

[Corey Bryant]

Thanks scottiecl - but please don't make me remember But I think you have. LOL

[don1]

I am having a different situation with the robots.txt

In October I had 394 entries via the robots.txt and 212 exited immediately (without looking at/crawling other pages). That's only 48% that entered the site. I have been having a difficult time getting these buggers to crawl new pages. Googlebot shows up every few days but does not go anywhere. Why? Are they looking for changes in the file before crawling anywhere else? Have I told them to stop? I have tested the site with our search spider and they all seem to work as planned. I'm stumped.

[Think Web]

Sometimes web crawlers will look for the "Last-Modified" HTTP response header. If there hasn't been a change to the page since the last spider session, it may move on to another site.

[don1]

That makes sense if they were entering the index page or other html page but these critters are hitting the robots.txt first and turning tail. I'm going to take a shot at removing the file and see if they drill any further. I know this will create an error but I'm desparate.

[dragonlady7]

Getting back to the robots.txt and why to bother question-- Robots.txt is a good starting place. It makes sure you're square with everyone who's legit. And it also means you can complain if someone displays pages you didn't want spidered-- it means you're aware of the issue, at least, and it means you've got a leg to stand on if you need to complain about a noncompliant bot.

It's also very handy if you want to get further into things-- start watching your logs for bots who never request it, and there you have your rogues that you should think about banning. You'll have to use more advanced techniques to keep the others off your site, but at least you've got a starting place.

[bwelford]

... but these critters are hitting the robots.txt first and turning tail.

There's nothing to worry about here. It seems to be the normal behaviour. I less often see the opposite behaviour where a robot keeps going after getting the robots.txt file. Usually another robot from the same family will come and look for other files. That's much more typical in my log files.

[Corey Bryant]

Sorry to dredge this up, but I have a question re: the robots.txt.

If the first line is:
User-agent: *
Disallow:

But then goes on to say:
User-agent: SpankBot
Disallow: /

Would SpankBot be banned or would it review the first one?

Thanks!!!

[dragonlady7]

I think that you'd simply omit the first one.
Just put in the
"User-agent: SpankBot
Disallow: /"
line, and don't bother with the other.

Other bots will read it, say "I am not SpankBot", and go on their merry way spidering your site.

That's how it's supposed to work, anyway. (Yes, I looked it up!!!)

[Alan Perkins]

Would SpankBot be banned or would it review the first one?

Definitive Answer: It would be banned. "*" means "any robot not mentioned elsewhere".

In October I had 394 entries via the robots.txt and 212 exited immediately (without looking at/crawling other pages).

Hits on robots.txt are usually from robots. Web stats for "entry", "exit" and "path" don't work too well for robots, since robots can change IP address between requests and they don't accept cookies. Just ignore it.

[meta]

Many threads have included mentions of "bad" bots that nose through things that the webmaster does not want indexed. What exactly are they doing with the stuff? Why do they want it?

[dragonlady7]

Often they're spammers, simply looking for more email addresses to harvest. Sometimes they're just the kind of bad bots that the exclusion protocol was written for-- not malicious, but overeager. Sometimes people hide good things in those directories, as well, like cgi-bins and the like. Sometimes a bad bot will go straight for the robots.txt to find out what it's not supposed to take, so it can take it!
So, there are a number of reasons.
Most people need not worry about them, but advanced users have plenty to keep up on their toes about!

[Corey Bryant]

Great - thanks for the information!! Sorry I did not get back to you sooner. I just was not e-mailed anything about all these wonderful responses.

[dzinerbear]

Hi all,

I have a question about excluding certain spiders from my site. I stumbled across a site that will generate a robots.txt file to include what it feels are a whole bunch of "nasty" bots.

The URL is here for anyone who wants to take a look at it:
http://www.1-hit.com...t-generator.htm

As I scan the list I recognize many of the names, so I'm pretty confident that they are bots I'm not interested in indexing my site. But in reading this thread, it seems that the spiders may or may not abide by my robots.txt file, so is there any point in excluding these 20 or so spiders.

Thanks
Dzinerbear