4 replies to this topic

[ChuckFinley]

Hi All,

I think I may have stopped the Search Engines from crawling my whole site. I'm hoping someone can help me out. I would really appreciate it.

If structured my Robots.txt file as follows --

CODE

User-agent: *
Disallow: /

http://www.CompanyDomain.com/index.php?p=date&CatalogSetView=Text
http://www.CompanyDomain.com/index.php?p=name&CatalogSetView=Text
http://www.CompanyDomain.com/index.php?p=price_desc&CatalogSetView=Text

Which of the following will it do? --

1) Disallow all three URL's or

2) Disallow the whole website.

If I want to disallow all three URL's, can I use the above shortcut or do I have to include the Disallow: field on every line like this --

CODE

User-agent: *
Disallow: /http://www.CompanyDomain.com/index.php?p=date&CatalogSetView=Text
Disallow: /http://www.CompanyDomain.com/index.php?p=name&CatalogSetView=Text
Disallow: /http://www.CompanyDomain.com/index.php?p=price_desc&CatalogSetView=Text

Thanks for the help!

[Jill]

You would use this:

CODE

User-agent: *

Disallow: /index.php?p=date&CatalogSetView=Text
Disallow: /index.php?p=name&CatalogSetView=Text
Disallow: /index.php?p=price_desc&CatalogSetView=Text

[ChuckFinley]

You would use this:

CODE

User-agent: *

Disallow: /index.php?p=date&CatalogSetView=Text
Disallow: /index.php?p=name&CatalogSetView=Text
Disallow: /index.php?p=price_desc&CatalogSetView=Text

Thanks for the quick answer. I will implement ASAP.

Just for my education, do you think the syntax that was used --

CODE

User-agent: *
Disallow: /

http://www.CompanyDomain.com/index.php?p=date&CatalogSetView=Text
http://www.CompanyDomain.com/index.php?p=name&CatalogSetView=Text
http://www.CompanyDomain.com/index.php?p=price_desc&CatalogSetView=Text

would prevent the whole site from being crawled? I don't think the Robot will get past the first Disallow: field --

CODE

User-agent: *
Disallow: /

I suspect that's the case but I've been wrong before. I don't want to wait to find out, though!

Thanks for all your help.

[Jill]

Come to think of it, yes it would prevent the whole site from being crawled!

Because you're saying at the first line to disallow everything with just the /

[ChuckFinley]

Come to think of it, yes it would prevent the whole site from being crawled!

Because you're saying at the first line to disallow everything with just the /

Grazie!