Are you a Google Analytics enthusiast?
More SEO Content
Filetype Exclusion For Googlebot
Posted 15 March 2004 - 06:05 AM
says that googlebot supports wildcard file extensions in robots.txt files
But I can't find anything on Google's own pages to verify this claim.
Does anyone know about this?
Posted 15 March 2004 - 08:46 AM
Yes I think I've found it. I wasn't looking for such a complicated exclusion as yours. I simply wanted to exclude files of a particular type in the same directory.
My thinking on this is a little naive. So it is probably not even worth doing.
It occured to me that if I were to tell googlebot not to bother with my .aspx pages since they are all restricted to logged-in users, it might prompt the bot to use its resources more productively by trawling deeper on/for pages that are available to it...
.... OK, you can start laughing at me now
Posted 15 March 2004 - 08:52 AM
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users