Are you a Google Analytics enthusiast?
More SEO Content
Robots.txt & Sitemap.xml Redundant?
Posted 15 August 2007 - 03:09 PM
Posted 15 August 2007 - 05:22 PM
Posted 15 August 2007 - 06:17 PM
Oh, OK, if both have different purposes which would you say is more crucial to have? And don't they both function as ways for the robots to see all of the files available for spidering? If so, then what exactly is the difference? Thanks for your help on this.
Posted 15 August 2007 - 06:21 PM
robots.txt is for EXCLUDING urls from compliant crawlers.
and neither are crucial or even necessary.
Posted 15 August 2007 - 06:28 PM
Posted 16 August 2007 - 03:26 AM
robots.txt tells a bot what NOT to do with files or folders
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users