Are you a Google Analytics enthusiast?
More SEO Content
Robots.txt Syntax For Wildcards?
Posted 12 May 2008 - 06:08 AM
Posted 12 May 2008 - 07:23 AM
Posted 13 May 2008 - 08:29 AM
Do I need to have two seperate sets of directions in my robots.txt, one for google, yahoo and msn and a second for all others?
Posted 13 May 2008 - 09:06 AM
If you want those instructions to apply to only certain spiders, then you need to set that in the User-agent field. If not, you can use a User-agent: * to cover them all.
Those spiders who don't understand/use wildcards will simply ignore those lines.
Posted 14 May 2008 - 02:45 AM
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users