Are you a Google Analytics enthusiast?
More SEO Content
Does Robots.txt Affect Seo
Posted 26 December 2004 - 07:05 AM
To stop people finding these books online withoutg having paid for them, I've created robots.txt files to stop SEs from spidering the pages where they can be found.
However, I'm concerned that this might interfere with my SEO efforts.
Does anyone know anything about this???
Posted 26 December 2004 - 07:28 AM
Keep the following in mind, though -- robots.txt won't stop compliant spiders from being aware of those pages. It'll just keep them from being indexed, so it's possible that on a site: search those pages will come up. There just won't be any snippets or cached copies of them.
Also, not all spiders are robots.txt-compliant. The big ones (googlebot, slurp and msnbot) are, but there are lots of others out there. You may want to consider password-protecting those pages instead of just using the robots exclusion protocol if it's really important to you that only those who you want to authorize to view those pages are able to.
Posted 26 December 2004 - 10:54 AM
Posted 26 December 2004 - 12:01 PM
Think of it as being useful for controlling the bandwidth usage a spider uses and for avoiding dupe content issues. It's pretty useless for almost everything else.
Even if you have a robots.txt that says that the whole site is spiderable, if you password protect the pages in question the SE's will be blocked just like people.
I recommend using the robots.txt to block off duplicate content and folders with no useful content (images, cgi-bin, etc) and then use real security for everything else.
Posted 26 December 2004 - 12:47 PM
robots.txt offers zero protection on the security side of things. It simply wasn't designed to be a security measure.
If you don't need or want to have ironclad protection it would be better to set up a blanket username/password system and send that to everyone who purchases something. So every purchaser would get the same username and password to download your file(s).
That would work better than nothing at all. Then you would only have to worry about real people sending the information to all of their friends. And at least you could change the password from time to time.
Posted 19 January 2005 - 10:20 AM
Thanks for your help and advice!
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users