Jump to content

  • Log in with Facebook Log in with Twitter Log In with Google      Sign In   
  • Create Account

Subscribe to HRA Now!

 



Are you a Google Analytics enthusiast?

Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE! 

 



 

 www.CustomReportSharing.com 

From the folks who brought you High Rankings!



Photo

Quick Question On Robots.txt


  • Please log in to reply
5 replies to this topic

#1 lister

lister

    HR 5

  • Active Members
  • PipPipPipPipPip
  • 408 posts

Posted 14 May 2009 - 02:06 AM

Just to confirm - can someone please check that my robots.txt file is correct:

User-agent: *
Disallow: /images/
Disallow: /img/

> what I am trying to do is allow all bots all access but I dont want the images being archived and indexed.

Thanks - appreciate it! (I guess it is vital to make sure the robots.txt is correctly written otherwise no bot visits!!)

#2 1dmf

1dmf

    Keep Asking, Keep Questioning, Keep Learning

  • Active Members
  • PipPipPipPipPipPipPip
  • 2,160 posts
  • Location:Worthing - England

Posted 14 May 2009 - 04:03 AM

it all depends if the image files are in those directories.

You are blocking a directory with that command, and *IF* the spider honours the robots.txt protocol then it won't index anything in those folders, it has nothing to do with file type.

you can check out the syntax here http://www.robotstxt.org/ , G! also have a checker if you have a GWMT account.



#3 lister

lister

    HR 5

  • Active Members
  • PipPipPipPipPip
  • 408 posts

Posted 14 May 2009 - 07:28 AM

QUOTE(1dmf @ May 14 2009, 10:03 AM) View Post
it all depends if the image files are in those directories.

You are blocking a directory with that command, and *IF* the spider honours the robots.txt protocol then it won't index anything in those folders, it has nothing to do with file type.

you can check out the syntax here http://www.robotstxt.org/ , G! also have a checker if you have a GWMT account.


perhaps it is better to allow the bots to do exactly what they want - indeed, is there any need for the robots.txt at all?

#4 NASA

NASA

    HR 4

  • Active Members
  • PipPipPipPip
  • 183 posts

Posted 14 May 2009 - 08:57 AM

well if you don't want your website images showing up on google image search, then it's a good idea to block the images folder.

there is only a need IMO for a robots.txt file *IF* you are trying to block specific content/folders from being indexed, otherwise there is no need for one.



#5 adibranch

adibranch

    HR 5

  • Active Members
  • PipPipPipPipPip
  • 332 posts

Posted 20 May 2009 - 10:08 AM

useful for wildcard blocks though, with all those pesky duplicated sortby? pages..

#6 Michael Martinez

Michael Martinez

    HR 10

  • Active Members
  • PipPipPipPipPipPipPipPipPipPip
  • 5,013 posts
  • Location:Georgia

Posted 20 May 2009 - 11:51 PM

QUOTE(NASA @ May 14 2009, 06:57 AM) View Post
there is only a need IMO for a robots.txt file *IF* you are trying to block specific content/folders from being indexed, otherwise there is no need for one.


They are also useful for showing robots where your sitemap files are.






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

SPAM FREE FORUM!
 
If you are just registering to spam,
don't bother. You will be wasting your
time as your spam will never see the
light of day!