3 replies to this topic

[falcon]

Hi.

I am still learning about SEO/SEM and have a question. Until recently, I was naive and did not know that there were such things as bad spiders that ignored robots.txt.

When I learned of these little beasties, I alerted my company's Networking department. They plan to put NT security or logon pages in place. Will this prevent the spiders from accessing the directories/pages? Or, should any information we don't want spidered be moved off a machine that's accessible to the outside world?

Thanks.

Falcon

PS I did search for information on this topic, but didn't find anything. If this has been covered elsewhere, please just point me in the right direction.

[bobsledbob]

This is probably more a "technobabble" kind of discussion, rather than search engine friendly design.

However, yes adding a login page or "NT security" (basic authentication) to your web pages will prevent a bad robot from gaining access.

It would also obviously prevent anyone else from getting there so long as they don't have an appropriate username and password.

If you have super sensitive information, get it off the webserver. An IIS webserver is not necessarily the safest thing to have exposed to the internet. But really, it doesn't matter which kind of server, if it's sensitive, get it off the network period.

For ways to deter bad robots, you might want to read this topic which somewhat addresses it: http://www.highranki...?showtopic=4254

[Scottie]

Is there a particular kind of beastie you are trying to block? The more you know about it, the more effective you can be at blocking it.

There are many techniques to try and foil e-mail harvesters, ranging from obscuring the e-mail addresses to feeding the bots a page full of spam-reporting e-mails.

For the most part, it's not worth wasting your time trying to thwart them as a group. If you see in your logs that some are eating up your bandwidth on a regular basis or doing other misbehaving, you can block them at the IP level. If they are really bad beasties, they won't pay attention to your robots.txt anyway.

[falcon]

Thanks very much for your replies.

No, there's not any one in particular that I've seen, but I will look into that.

I will speak to our Networking folks about your feedback.

Thanks again.

Falcon