Jump to content

  • Log in with Facebook Log in with Twitter Log In with Google      Sign In   
  • Create Account

Subscribe to HRA Now!

 



Are you a Google Analytics enthusiast?

Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE! 

 



 

 www.CustomReportSharing.com 

From the folks who brought you High Rankings!



Photo

Robots.txt Syntax


  • Please log in to reply
4 replies to this topic

#1 clandestino

clandestino

    HR 3

  • Members
  • PipPipPip
  • 84 posts

Posted 09 December 2010 - 03:50 AM

Hi All,

I think I may have stopped the Search Engines from crawling my whole site. I'm hoping someone can help me out. I would really appreciate it.

If structured my Robots.txt file as follows --

CODE
User-agent: *
Disallow: /

http://www.CompanyDomain.com/index.php?p=date&CatalogSetView=Text
http://www.CompanyDomain.com/index.php?p=name&CatalogSetView=Text
http://www.CompanyDomain.com/index.php?p=price_desc&CatalogSetView=Text

Which of the following will it do? --

1) Disallow all three URL's or

2) Disallow the whole website.

If I want to disallow all three URL's, can I use the above shortcut or do I have to include the Disallow: field on every line like this --

CODE
User-agent: *
Disallow: /http://www.CompanyDomain.com/index.php?p=date&CatalogSetView=Text
Disallow: /http://www.CompanyDomain.com/index.php?p=name&CatalogSetView=Text
Disallow: /http://www.CompanyDomain.com/index.php?p=price_desc&CatalogSetView=Text


Thanks for the help!

#2 Jill

Jill

    Recovering SEO

  • Admin
  • 32,864 posts

Posted 09 December 2010 - 10:12 AM

You would use this:

CODE
User-agent: *

Disallow: /index.php?p=date&CatalogSetView=Text
Disallow: /index.php?p=name&CatalogSetView=Text
Disallow: /index.php?p=price_desc&CatalogSetView=Text




#3 clandestino

clandestino

    HR 3

  • Members
  • PipPipPip
  • 84 posts

Posted 09 December 2010 - 04:02 PM

QUOTE(Jill @ Dec 9 2010, 07:12 AM) View Post
You would use this:

CODE
User-agent: *

Disallow: /index.php?p=date&CatalogSetView=Text
Disallow: /index.php?p=name&CatalogSetView=Text
Disallow: /index.php?p=price_desc&CatalogSetView=Text


Thanks for the quick answer. I will implement ASAP.

Just for my education, do you think the syntax that was used --

CODE
User-agent: *
Disallow: /

http://www.CompanyDomain.com/index.php?p=date&CatalogSetView=Text
http://www.CompanyDomain.com/index.php?p=name&CatalogSetView=Text
http://www.CompanyDomain.com/index.php?p=price_desc&CatalogSetView=Text


would prevent the whole site from being crawled? I don't think the Robot will get past the first Disallow: field --

CODE
User-agent: *
Disallow: /


I suspect that's the case but I've been wrong before. I don't want to wait to find out, though!

Thanks for all your help.



#4 Jill

Jill

    Recovering SEO

  • Admin
  • 32,864 posts

Posted 09 December 2010 - 04:48 PM

Come to think of it, yes it would prevent the whole site from being crawled!

Because you're saying at the first line to disallow everything with just the /

#5 clandestino

clandestino

    HR 3

  • Members
  • PipPipPip
  • 84 posts

Posted 09 December 2010 - 04:55 PM

QUOTE(Jill @ Dec 9 2010, 01:48 PM) View Post
Come to think of it, yes it would prevent the whole site from being crawled!

Because you're saying at the first line to disallow everything with just the /


Grazie!




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

SPAM FREE FORUM!
 
If you are just registering to spam,
don't bother. You will be wasting your
time as your spam will never see the
light of day!