9 replies to this topic

[Say Yebo]

I'm assessing a site for a potential new client and when I checked www.client.com/robots.txt to see their robots file, I got their home page.

I thought they somehow had their robots.txt file redirecting to their home page, but a quick header/redirect check shows it's a direct link. I get a 200 OK result.

I've never seen this before. What's it all about?

[Ron Carnell]

Try to look at www.client.com/nopage.here and I suspect you'll see exactly the same thing. If so, it's just a very poorly executed custom 404 "plan." (sic)

[Jill]

Yep, and that's very bad news as it means they could have tons of non existent URLs all being indexed with the home page content!

[Say Yebo]

Urgh. OK, I'll have to look into that then. Thanks guys!

Try to look at www.client.com/nopage.here and I suspect you'll see exactly the same thing. If so, it's just a very poorly executed custom 404 "plan." (sic)

Looks like you're spot on Ron! That IS the case.

Yep, and that's very bad news as it means they could have tons of non existent URLs all being indexed with the home page content!

And indeed they do. I just looked and found over 60 versions :-(

Many of the versions look like this...with the word token in it: client.org/index.cfm?CFID=31111&CFTOKEN=37878467

They have a CMS - might they be creating duplicates of their home page every time they edit it?

Is this even fixable?

Edited by Jill, 03 August 2010 - 08:45 PM.

[qwerty]

Sure. Set up the server to respond properly to requests for nonexistent URLs. The next time the crawler requests those URLs it will get a 404 instead of a 200.

[Say Yebo]

Sure. Set up the server to respond properly to requests for nonexistent URLs. The next time the crawler requests those URLs it will get a 404 instead of a 200.

Thanks Qwerty....sounds simple enough.

I've just found another issue: Some of those duplicated page have a different domain name featuring the design company's name. My client is a .org, but about 20% of the duplicated home pages are on: client.designco.com/index.cfm....

Will the 404 take care of that too?

[Jill]

Those should be excluded via a robots.txt file if possible.

[Say Yebo]

Those should be excluded via a robots.txt file if possible.

Thank you Jill. Living and learning!

[qwerty]

I think I disagree, but it may just be that I'm not clear on the situation. If you've got URLs indexed and you don't want them indexed, then blocking them with the robots exclusion protocol won't help: if search engines are told they can't request pages that they already know exist (or believe exist), then disallowing those pages will keep search engines from requesting them again. If they don't request them, they're never going to find out that the pages don't exist.

I think you'd be better off either setting up 301s from those URLs to others, or just letting the search engines receive 404s when they request them. Those methods will eventually drop those URLs from the index.

The part I clearly don't get is that these are apparently pages from a different domain: designco.org vs. designco.com and subdomains thereof. Is G reporting that there are .com pages on the .org domain??

[Say Yebo]

The part I clearly don't get is that these are apparently pages from a different domain: designco.org vs. designco.com and subdomains thereof. Is G reporting that there are .com pages on the .org domain??

Well, Google is finding their home page content on client.org and client.designco.com. When I pointed this out, their design company 301'd the client.designco.com to client.org. But I'm not sure why that situation existed in the first place.

As I am still waiting to see which SEO company gets awarded the job, I have not spent any more time on the subject, but I felt inclined to point out the problem as it may help us get the job. (Or maybe not!)