Are you a Google Analytics enthusiast?
More SEO Content
Does Anybody Block Automated Link Checkers?
Posted 12 September 2004 - 05:44 AM
Maybe it is just me, but one particular link checker regularly keeps crawling through every page of my site, and probably will become even more often the more backlinks I get.
I presume that the only disadvantage of blocking any Link Checker is that the linked site may eventually delete my link on their site?
Posted 12 September 2004 - 06:38 AM
You can do some user-agent detection at the server level and redirect those that don't obey robots.txt, but that's usually a bit more than most want to get into doing.
Posted 12 September 2004 - 11:08 AM
Guess I will just have to put up with it crawling all over me....or not link to sites that don't trust me enough to keep thjeir site on mine - shame that they cannot instruct it to just crawl the one page where I told them their site could be found?
Posted 12 September 2004 - 11:21 AM
Just don't be terribly surprised if your friendly neighborhood spider keeps right on crawling.
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users