Are you a Google Analytics enthusiast?
More SEO Content
Posted 16 April 2004 - 01:06 PM
Posted 16 April 2004 - 01:10 PM
Posted 16 April 2004 - 01:13 PM
Posted 16 April 2004 - 01:17 PM
Posted 16 April 2004 - 01:18 PM
Posted 16 April 2004 - 01:46 PM
Posted 16 April 2004 - 02:08 PM
Robots.txt is generally used to prevetn certain pages to get crawled n indexed.
Posted 17 April 2004 - 12:26 AM
However, it is standard practice to exclude your Picture file. This is because, the robots surfing your pictures may consume a great deal of bandwidth at your expense. And all you get out of it is the possibility that these robot reports are used by some "people" to make huge databases of images to steal at will.
Or at least, that was what I was taught at the vBulletin forum. Anyway, I exclude my picture files and that is all I ever can see any need to exclude.
Oh....another reason though, is if you have PDF files or such, intended only for download, and that duplicate the standard HTML content of your site. Definitely, you want to exclude those, so they are not mistaken for "mirror pages."
(This of course does not apply to PDF files that do not have duplicate content and that you WANT to be spidered.)
Edited by kristof, 17 April 2004 - 12:38 AM.
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users