3 replies to this topic

[UweKeim]

Recently I discovered that pages of a competitor of us are in the Google index, although the URLs of the pages are not having any links pointing to them.

(I downloaded the complete site with WinHTTrack and really found no internal links)

The URLs that Google found and indexed are e.g.:

http://www.my-cool-website.com/somefolder/

On such URLs, no internal and no extern links are pointing to.

Instead, I found links to documents like:

http://www.my-cool-website.com/somefolder/some-document.pdf

I do not understand why non-refered documents are still being crawled by Google.

So my question is: Could anyone kindly explain me why non-refered URLs are found by Google?

What I do assume is that for each found URL, Google also moves up the folder hierarchy and crawls each found document, even if it is not refered directly.

Thanks
Uwe

[Jill]

Google finds urls in all kinds of crazy ways these days even if they're not linked to.

[UweKeim]

Thanks a lot, Jill

[josephfranklynmcelroy]

Does the site have an xml sitemap? Pages to be indexed can be placed there... look for sitemap: tag in robots.txt or see if www.domain.com/sitemap.xml (or urllist.txt) exists

Recently I discovered that pages of a competitor of us are in the Google index, although the URLs of the pages are not having any links pointing to them.