Can Google Index Unlinked Pages?, Can Google index unlinked pages? Options

[klewando]

Hi. There is a discussion on my Web design mailing list about whether Google can index pages to which there are no links, such as those under development which are not intended to be public yet. Is this possible?

Thanks,
Kim

[torka]

It is possible for them to spider those pages, assuming there's no password to prevent it or robots.txt to disallow the spider. It apparently has happened in the past; there are a number of theories around about means they might use to find the page in the first place, although I don't know if anyone's ever determined for sure how the spider finds these orphan pages.

It's also possible the page might show up briefly in the index, but unless the page gets links pointing to it, the page won't stay in the index, and even during the potential time it might be indexed it's doubtful that it would rank very well for any even mildly competitive term with no inbound links.

That said, that's no guarantee that no one will ever find the page -- people enter obscure/oddball search terms all the time, and you can't always predict what phrases a page might show up for. We all have the experience of finding referers in our logs from totally off-the-wall searches. So "security through obscurity" is unreliable at best -- all it takes is one freaky search during that time when the page is in the index and the proverbial cat could be out of the proverbial bag.

If the concern is keeping the SEs away from a page or preventing the general public from stumbling across it, IMHO the only way to insure that is to password-protect it. There are enough bots out there that don't obey robots.txt that you can't rely on robots.txt to "protect" a page from indexing.

If the idea is to create an orphan page and somehow get it to rank for something, that won't happen unless the page gets inbound links from somewhere. Without links, it won't "stick" in Google's index even if it does briefly show up initially.

I think we've got a thread or two around here where this was tested...

--Torka (IMG:http://www.highrankings.com/forum/style_emoticons/default/mf_prop.gif)

[Alan Perkins]

Welcome to the forums klewando (IMG:http://www.highrankings.com/forum/style_emoticons/default/bye1.gif)

There is a discussion on my Web design mailing list about whether Google can index pages to which there are no links, such as those under development which are not intended to be public yet. Is this possible?

Yes.

Here are some possible ways (not saying that any of these are probable):

1) The URL appears up as a referrer in a pubic referrer log, or in trackbacks, etc.
2) Google crawls ISP logs
3) The Google Toolbar (with Pagerank display enabled) reports the existence of the URL to Google
4) Somebody submits the URL to Google using their Add URL form

[davidbrett]

Hi Kim,

This has happened to me once. We were developing a site last year and we had nearly finished the job. However we weren't happy with it and in the end we didn't go live with the site.

A couple of months later I saw that all of the search engines had picked up the site - even though no links were pointing at it. It even gained good rankings in MSN and Yahoo.

This post has been edited by davidbrett: Nov 22 2005, 11:01 AM

[sonnyyu]

I would like add few more;-

1. Google Web Accelerator
2. Google Sitemap Xml Submite
3. Froogle Submite



my $0.0199

[Michael Martinez]

Matt Cutts has reportedly stated that they don't get URLs from the Toolbar.

A few months ago, several of us here submitted orphaned pages to Google through their ADD URL feature and none of the pages were spidered.

The other suggestions seem plausible. You can, on rare occasions, find some referer data in open logs, for example.

I'm not sure Web accelerator feeds the index. I haven't seen any discussions of this possibility. A number of people have speculated that AdSense might feed the index, but Google engineers have denied that being so.

[chrishirst]

Submitting doesn't get you in Google, eveyone from here who tried it came up with the same result, only the Yahoo crawler turned up.

The Google, MSN and Yahoo toolbars. I've used all three and checked various unlinked pages on a couple of crawled sites. Not a crawler in sight.

Adsense, Had it on several unlinked pages only the Media-Partners bot visited. Same with using Opera with the ads running.

[qwerty]

Ditto that (except I didn't try the Yahoo or MSN toolbars).

And my submitted page, after about half a year, is still in the Yahoo index, so I guess one could say that's a legitimate way to get into Yahoo. It was dropped for a little while, but it came back without any action on my part.

[michaelhood]

Matt Cutts has reportedly stated that they don't get URLs from the Toolbar.

I've had pages with:

- no inbound links
- no web-facing server logs
- etc etc

3 of us were working on a beta of a site, similar to the story above, and this site was indxed in G, and remained in the index. This happened in Oct 04.

[Jill]

3 of us were working on a beta of a site, similar to the story above, and this site was indxed in G, and remained in the index. This happened in Oct 04.

I'd have to see it with my own eyes to believe it.

[sonnyyu]

I would like add few more;-

1. Google VPN
2. Port 443 (SSL Certificate Scan)
3. Register .gov, .mil, .edu domain.



my $0.01999

[making-it-big]

To comment on the discussion above I have seen similar results from google on new sites. No link and just starting it gets me to wonder if they are looking at domain registration or something. Google was the first to pick up the site and this even BEFORE we started to design the site. All we did was buy the domain name and the hosting package. No I do not know for sure - even though it didn't any backlinks - but our server COULD have a list accessable to bots of new sites on their server. To this day I do not even see anything from our server as backlinks (google or yahoo). It is strange indeed.

I also see that Google is picking up MSN results and have seen Google trying to crawl funky broken up URL's like (/directory/file...) with the 3 dots. The only sites that has URLs like that (when doing a search in Google and Yahoo) is msn results and scrapers of those results. Thes links are cut off and are at the end of search results but are not linked. But who is to say there isn't something somewhere that is actually linked like that but is not showing up in searches - I highly doubt it but it could be possible.

Just my few cents worth.

This post has been edited by Jill: Nov 29 2005, 06:32 PM

[Randy]

No link and just starting it gets me to wonder if they are looking at domain registration or something.

Well, there's probably no way to prove it one way or another. But since Google became an official ICANN Accredited Registrar earlier this year...

Let's just say that it's a pretty safe bet that they have a lot more information regarding domains and their ownership now than they had in the past. (IMG:http://www.highrankings.com/forum/style_emoticons/default/wink.gif)

[making-it-big]

I wouldn't doubt that they would use that information. It was rather quick that they picked up the site that is for sure. Yahoo was quick to do it also. MSN didn't until we gathered a few links.

[dmcconkey]

Let's just say that it's a pretty safe bet that they have a lot more information regarding domains and their ownership now than they had in the past.

Yeah, big brother is starting to scare me. Next thing we know, our tax filings will go through them.

But I have a hard time believing in G crawling new domain registrations. Sure they could take a guess at the index address and crawl from there. But what about some of the claims in this thread of G crawling non linked folders in established sites?

Best idea I've heard is that G gets these pages from our PR toolbar queries.

-Dan