Are you a Google Analytics enthusiast?
More SEO Content
Quick Robots.txt Question
Posted 14 November 2008 - 02:09 AM
Mean none of my site can be spidered?
Posted 14 November 2008 - 07:36 AM
The / means everything. So everything will be disallowed.
Posted 14 November 2008 - 10:21 AM
The "/" does not literally mean everything. It means "everything beginning with a /", just as "/secret/" means "everything beginning with /secret/". Since URLs are specified as relative to the root directory of the site, everything DOES begin with a "/". This is why, for practical purpses, "/ means everything".
Posted 14 November 2008 - 11:40 AM
Use robots.txt to disallow directories (including root, which means everything)
Use the robots metatag to disallow specific individual pages
Use the nofollow attribute to disallow links on pages
Use system level permissions for actual security and the ultimate disallow (overkill for preventing spidering of your images, required for preventing access to your customer database).
Posted 14 November 2008 - 12:44 PM
I just realized for the last year and a half that was my robots.txt file!
Though there were a few pages that were also under the 'allow' part.
Posted 14 November 2008 - 12:54 PM
Glad we could help - I suspect you SEO efforts for this site will be more fruitful from now on.
If you need a more complicated robots.txt than a simple "allow all", the you can do a search for "robots.txt generator" in your favorite search engine and use a tool I designed for the job. It's usually the first result.
Posted 14 November 2008 - 03:33 PM
I haven't done much with the site lately, but traffic has been down lately, and I noted on google webmaster tools there was a problem with my sitemap. So I re-upped it,, and still a problem, so I checked my robots file and seen the problem. I was banned from google for a while, and had someoen fix the site, so I guess they didn't want the spiders to go to any bad directories, so blocked them off. Just forgot to fix it after google put us back in!
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users