I'm trying to get a better understanding of the robots.txt file and how it helps (preventing the spiders from accessing certain pages) and hinders (malicious blocking of the spiders on links pages).
I know that I have read about this somewhere on the forum but cannot find anything in the search.
1/ Where is a robots.txt file located?
2/ How does a browser to a page view it?
3/ what code am I looking for it is being used maliciously on links pages?
4/ Anything else of relevance?
Thanks in anticipation
Edited by e-breaks, 08 January 2005 - 05:30 PM.