SEO Class in Chicago, IL
Are you a Google Analytics enthusiast?
More SEO Content
Posted 17 March 2004 - 01:56 PM
I am still learning about SEO/SEM and have a question. Until recently, I was naive and did not know that there were such things as bad spiders that ignored robots.txt.
When I learned of these little beasties, I alerted my company's Networking department. They plan to put NT security or logon pages in place. Will this prevent the spiders from accessing the directories/pages? Or, should any information we don't want spidered be moved off a machine that's accessible to the outside world?
PS I did search for information on this topic, but didn't find anything. If this has been covered elsewhere, please just point me in the right direction.
Posted 17 March 2004 - 06:27 PM
However, yes adding a login page or "NT security" (basic authentication) to your web pages will prevent a bad robot from gaining access.
It would also obviously prevent anyone else from getting there so long as they don't have an appropriate username and password.
If you have super sensitive information, get it off the webserver. An IIS webserver is not necessarily the safest thing to have exposed to the internet. But really, it doesn't matter which kind of server, if it's sensitive, get it off the network period.
For ways to deter bad robots, you might want to read this topic which somewhat addresses it: http://www.highranki...?showtopic=4254
Posted 17 March 2004 - 11:53 PM
There are many techniques to try and foil e-mail harvesters, ranging from obscuring the e-mail addresses to feeding the bots a page full of spam-reporting e-mails.
For the most part, it's not worth wasting your time trying to thwart them as a group. If you see in your logs that some are eating up your bandwidth on a regular basis or doing other misbehaving, you can block them at the IP level. If they are really bad beasties, they won't pay attention to your robots.txt anyway.
Posted 18 March 2004 - 09:45 AM
No, there's not any one in particular that I've seen, but I will look into that.
I will speak to our Networking folks about your feedback.
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users