6 replies to this topic

[Mel]

Just noticed this while looking for something else:

Search appliance can crawl password protected pages
which says

"The Google Search Appliance now has the capability to crawl password-protected areas and HTTPs content."

Hmmm... if the search appliance can do it I wonder about Googlebot?

[dimok]

Mel, I think Googlebot will not have permissions on oritected pages (i.e. login and password)
So it wouldn't crawl anything secure.

[Ron Carnell]

Hmmm... if the search appliance can do it I wonder about Googlebot?

Sure. All you have to do is install and configure Googlebot on your Enterprise server.

[dragonlady7]

Yeah, the Search Appliance is for Intranets, and generally the appliance has been given the proper permissions. Googlebot has nothing to do with it...

[Alan Perkins]

Crawling password-protected content is not a big deal. It's one extra line in the HTTP request header - a line that specifies the user name and password to access the page.

There is no standard way of communicating this information to a crawler, though. It needs outside influence - such as an Intranet control panel, or a trusted feed arrangement.

IMO, if Internet scale search engines really wanted to freely index password-protected content, they would have defined and implemented a standard for it long ago. It could even have been incorporated into robots.txt.

[Jill]

Hmmm... if the search appliance can do it I wonder about Googlebot?

Don't think it's a question of if they can. It's a question of do they want to?

Jill

[dimok]

Don't think it's a question of if they can.  It's a question of do they want to?

Why do you think so?

To visit a protected page bot (ha, it would not) have to contact with site administrator in order to get pass to the page. Impossible, I suppose