5 replies to this topic

[lister]

Just to confirm - can someone please check that my robots.txt file is correct:

User-agent: *
Disallow: /images/
Disallow: /img/

> what I am trying to do is allow all bots all access but I dont want the images being archived and indexed.

Thanks - appreciate it! (I guess it is vital to make sure the robots.txt is correctly written otherwise no bot visits!!)

[1dmf]

it all depends if the image files are in those directories.

You are blocking a directory with that command, and *IF* the spider honours the robots.txt protocol then it won't index anything in those folders, it has nothing to do with file type.

you can check out the syntax here http://www.robotstxt.org/ , G! also have a checker if you have a GWMT account.

[lister]

it all depends if the image files are in those directories.

You are blocking a directory with that command, and *IF* the spider honours the robots.txt protocol then it won't index anything in those folders, it has nothing to do with file type.

you can check out the syntax here http://www.robotstxt.org/ , G! also have a checker if you have a GWMT account.

perhaps it is better to allow the bots to do exactly what they want - indeed, is there any need for the robots.txt at all?

[NASA]

well if you don't want your website images showing up on google image search, then it's a good idea to block the images folder.

there is only a need IMO for a robots.txt file *IF* you are trying to block specific content/folders from being indexed, otherwise there is no need for one.

[adibranch]

useful for wildcard blocks though, with all those pesky duplicated sortby? pages..

[Michael Martinez]

there is only a need IMO for a robots.txt file *IF* you are trying to block specific content/folders from being indexed, otherwise there is no need for one.

They are also useful for showing robots where your sitemap files are.