Jump to content

  • Log in with Facebook Log in with Twitter Log In with Google      Sign In   
  • Create Account

Subscribe to HRA Now!

 



Are you a Google Analytics enthusiast?

Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE! 

 



 

 www.CustomReportSharing.com 

From the folks who brought you High Rankings!



Photo
- - - - -

Search Appliance Can Crawl Pw Protected Pages


This topic has been archived. This means that you cannot reply to this topic.
6 replies to this topic

#1 Mel

Mel

    HR 5

  • Active Members
  • PipPipPipPipPip
  • 353 posts

Posted 05 August 2003 - 11:24 PM

Just noticed this while looking for something else:

Search appliance can crawl password protected pages
which says

"The Google Search Appliance now has the capability to crawl password-protected areas and HTTPs content."

Hmmm... if the search appliance can do it I wonder about Googlebot? :P

#2 dimok

dimok

    HR 4

  • Active Members
  • PipPipPipPip
  • 110 posts

Posted 06 August 2003 - 04:47 AM

Mel, I think Googlebot will not have permissions on oritected pages (i.e. login and password) :chef:
So it wouldn't crawl anything secure.

#3 Ron Carnell

Ron Carnell

    HR 6

  • Moderator
  • 966 posts

Posted 06 August 2003 - 05:32 AM

Hmmm... if the search appliance can do it I wonder about Googlebot?


Sure. All you have to do is install and configure Googlebot on your Enterprise server. :chef:

#4 dragonlady7

dragonlady7

    HR 6

  • Active Members
  • PipPipPipPipPipPip
  • 618 posts

Posted 06 August 2003 - 08:02 AM

Yeah, the Search Appliance is for Intranets, and generally the appliance has been given the proper permissions. Googlebot has nothing to do with it...

#5 Alan Perkins

Alan Perkins

    Token male admin

  • Admin
  • 1,642 posts

Posted 06 August 2003 - 08:21 AM

Crawling password-protected content is not a big deal. It's one extra line in the HTTP request header - a line that specifies the user name and password to access the page.

There is no standard way of communicating this information to a crawler, though. It needs outside influence - such as an Intranet control panel, or a trusted feed arrangement.

IMO, if Internet scale search engines really wanted to freely index password-protected content, they would have defined and implemented a standard for it long ago. It could even have been incorporated into robots.txt.

#6 Jill

Jill

    Recovering SEO

  • Admin
  • 33,004 posts

Posted 06 August 2003 - 10:02 AM

Hmmm... if the search appliance can do it I wonder about Googlebot?


Don't think it's a question of if they can. It's a question of do they want to?

Jill

#7 dimok

dimok

    HR 4

  • Active Members
  • PipPipPipPip
  • 110 posts

Posted 06 August 2003 - 10:10 AM

Don't think it's a question of if they can.  It's a question of do they want to?

Why do you think so?

To visit a protected page bot (ha, it would not) have to contact with site administrator in order to get pass to the page. Impossible, I suppose :applause:




SPAM FREE FORUM!
 
If you are just registering to spam,
don't bother. You will be wasting your
time as your spam will never see the
light of day!