Are you a Google Analytics enthusiast?
More SEO Content
Posted 08 January 2004 - 09:11 AM
but in my robots.txt file i have disallowed the 'nothinghere' folder
would a spider follow the link and happily crawl the page, or would it request the robots.txt file first and say "no chance, son; not allowed in there!"
Posted 08 January 2004 - 09:20 AM
Posted 08 January 2004 - 02:51 PM
Posted 08 January 2004 - 06:05 PM
The robots.txt file is requested by both Google and Inktomi at the beginning of every crawl session, even if they are only minutes apart.
Posted 08 January 2004 - 07:16 PM
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users