Are you a Google Analytics enthusiast?
More SEO Content
Posted 23 September 2004 - 04:52 AM
Is there anyway to see what is in the robots.txt file for a given site?
Posted 23 September 2004 - 05:06 AM
but can be dangerous to the inexperienced ....
Posted 23 September 2004 - 05:08 AM
Posted 23 September 2004 - 03:49 PM
If you don't need to do any of the above, then the spiders will usually find all of your site, or you can include the meta tag "Meta Name="Robots" in your meta tags at the top of your HTML to make sure.
I do know that spiders do like robots.txt files and generate a hit when they find it. Not sure if it makes any difference in spidering a site or not though.....
Posted 23 September 2004 - 04:01 PM
I think it would be a bit much to say that spiders 'like' robots.txt; an ethical search bot will have its spiders check the robots.txt file to see where you don't want them to go.
Its not that they particularly want to look at it, its that they feel an obligation to look at it (with the micro benefit of not having to deal with the support calls to remove content from their index)
The robots.txt file is just another file on your site - of course a hit on it will show in your logs, the same as any other hit!
Posted 27 September 2004 - 09:41 AM
I felt a bit like homer simpson asking about the file without puttin git in the browser first....doh!!!
But as OWG says, now I can hack the world..muh..hah.haha.ahaha.ahahaa
Just joking, just wanted to see an advanced version from a big site operator to compare to mine.
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users