6 replies to this topic

[Jack]

On my sites server, I have sub-directories setup for testing new changes to the site prior to applying them to the live shop. It turns out that google is indexing all of the pages in those sub-directories. They are not mentioned anywhere (robots file, sitemap, etc.) but they are setup as sub-domanis, like test.mydomain.com. All I can think is that google is using the IP, which they share, to locate them. Regardless of how they are finding them, can some please explain a way to stop it, short of adding no archive tags to all of the files or making them password protected? I thought of adding another IP and setting the sub-domains up on it but without knowing just how it is google is doing it, that may be a wasted effort. I would appreciate any thoughts on how to set this up properly.

[chrishirst]

For Google (or any other search tool) to find the URL there must be a link to it somewhere, search engines cannot see how the domains and subdomains your server are structured.

[Jill]

I believe there's a way to block the subdomains, perhaps through robots.txt?

[qwerty]

Yes, each subdomain can have its own robots.txt file.

[Jack]

For Google (or any other search tool) to find the URL there must be a link to it somewhere, search engines cannot see how the domains and subdomains your server are structured.

No, there isn't such a link. No mention of it anywhere on the server other than in its own files. That's why I think it is being reached via the IP.

[Jack]

I believe there's a way to block the subdomains, perhaps through robots.txt?

Thanks, but google doesn't always pay attention to the robot file. They won't rank the pages if told not to search them but they may search them and list any they think should be listed. But, now that you joggled my brain, I'll try adding a 301 to the sub-domains .htaccess file.

[qwerty]

If the purpose of a given subdomain is internal testing, I don't think a 301 is what you want. If you redirect requests for pages within the subdomain, you're not going to be able to access those pages for testing.

Robots.txt will keep Google out of the pages in the subdomain, but it's true that that doesn't stop them from being aware of the pages there and including them in the index (even if they don't include the content on them). You can use Webmaster Tools to have Google remove URLs, however.