11 replies to this topic

[skr325]

Working on a robots.txt file and have noticed that many of the entries are listed multiple times in various cases as in the following example:

Disallow: /exampleone/
Disallow: /ExampleOne/
Disallow: /exampleOne/

It looks like Google is indexing these pages (with content not just URLs) regardless of these exclusions. So my own "duh-rometer" tells me something is amiss.

1. Can someone please explain what impact these 3 identical listings would have on a the bot?
2. Should I have only one entry, and if so, which one?
3. Are files and directories in the robots.txt case sensitive?

Thanks,
Steve

[Jill]

Can you post the entire robots.txt here? How long has it been up? Something is definitely amiss if Google is indexing that directory.

[skr325]

Thanks Jill-

here is the full text. It's been up at least 6 months.

User-agent: *
Disallow: /AccessAccount/
Disallow: /accessaccount/
Disallow: /accessaccountsearch/
Disallow: /Accessaccountsearch/
Disallow: /AccessAccountsearch/
Disallow: /AccessAccountSearch/
Disallow: /AccessaccountSearch/
Disallow: /accessAccountSearch/
Disallow: /accessAccountSearch/
Disallow: /accessaccountSearch/
Disallow: /Advantage/
Disallow: /advantage/
Disallow: /bin/
Disallow: /classes/
Disallow: /Demo/
Disallow: /demo/
Disallow: /Downloads/
Disallow: /downloads/
Disallow: /eBinder/
Disallow: /ebinder/
Disallow: /Binder/
Disallow: /binder/
Disallow: /BinderView/
Disallow: /binderview/
Disallow: /binderView/
Disallow: /eBV/
Disallow: /ebv/
Disallow: /FaxBack/
Disallow: /faxBack/
Disallow: /faxback/
Disallow: /Faxback/
Disallow: /Home/
Disallow: /home/
Disallow: /Sign-In/
Disallow: /Sign-in/
Disallow: /sign-In/
Disallow: /sign-in/
Disallow: /SignIn/
Disallow: /Signin/
Disallow: /signIn/
Disallow: /signin/
Disallow: /UnauthorizedAccess/
Disallow: /unauthorizedAccess/
Disallow: /Unauthorizedaccess/
Disallow: /unauthorizedaccess/
Disallow: /Utilities/
Disallow: /utilities/
Disallow: /Xsl/
Disallow: /XSL/
Disallow: /xsl/
Disallow: /SignUp/
Disallow: /signUp/
Disallow: /Signup/
Disallow: /signup/
Disallow: /CustomerSupport/YourAccount.aspx
Disallow: /customersupport/YourAccount.aspx
Disallow: /customersupport/YourAccount.aspx
Sitemap: http://www.example.com/sitemap.xml.gz

[TriExpert]

It's important to have a blank line between the Disallow directives and the Sitemap directives. But that's advice out of a book; I've never tested to see exactly what FUBARs if you fail to do so.

Edited by TriExpert, 24 April 2008 - 02:13 PM.

[skr325]

Still looking for some love here. Any thoughts from Jill or others?

-Steve

[projectphp]

1. Can someone please explain what impact these 3 identical listings would have on a the bot?

They'l block different things.

2. Should I have only one entry, and if so, which one?

No, all of them as appropriate, i.e. if you link to various CaSings, you need all of 'em.

3. Are files and directories in the robots.txt case sensitive?

Yes, because the WEB IS.

You might wanna use a robots noindex metatag, JIC, for all these pages.

[skr325]

I've started back through the pages to block at that level, but my original question still stands - why are these pages still in the index if the robots.txt is correct?

-steve

[Randy]

Because the search engines apparently already knew those pages existed for one.

robots.txt doesn't say to the search engines Never include these pages in your index. Instead it says Do not spider/crawl these pages, even if you've spidered them before, don't spider them anymore.

This points to one of those really horrid things about using an IIS server and not being careful in how you use Case in your URLs. On a Unix/Linux box each of those addresses would have been seen as being unique, so if the path was really all lowercase on the server every other attempted case would be returned as a 404 Not Found.

When you're on an IIS server you need to be extremely careful with how you link to your files. Choose one way and do it that way every time.

[Jill]

Are they still indexed with content and not just the URLs?

[incrediblehelp]

Are you internally linking to different version on your own website or other websites you control?

[skr325]

Are they still indexed with content and not just the URLs?

Yup!

Are you internally linking to different version on your own website or other websites you control?

I am not sure what you mean? Am I linking to these pages?

[Randy]

I am not sure what you mean? Am I linking to these pages?

I would imagine Jaan is asking if you've checked all of your own links to make sure you're being 100% consistent with the Case being used. Under they fact that the search engine spiders had to find links that contained the different case in the first place to even visit the different variations of the page urls.