Are you a Google Analytics enthusiast?
More SEO Content
Robots.txt - Googlebot
Posted 12 November 2004 - 01:03 PM
Is this because of a possible update of some kind going on this week, or does getting rid of the robots.txt file (that disallowed nothing anyway) entice googlebot for some reason?
My old robots.txt code:
Posted 12 November 2004 - 01:53 PM
Posted 12 November 2004 - 03:10 PM
Posted 12 November 2004 - 11:02 PM
Just curious. What would you include in a robots.txt file that would make Google spider your site or how many times Google would spider?
Posted 13 November 2004 - 01:16 AM
All the robots.txt file can do is tell google to go away or to not visit certain areas. It can't ask it to come more often or to visit certain areas.
Basically, it EXcludes, but doesn't INclude.
Posted 13 November 2004 - 07:46 AM
Posted 13 November 2004 - 08:26 AM
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users