to figure out whether Google had spidered my (just-)updated index page. Googlebot has been visiting my site quite a lot lately and I'm ecstatic over the results, but I was just curious. Anyhow, while in Notepad doing a "find next" to locate google searches and bots, I noticed that several times, the bot appeared within fractions of a minute to spider a page a searcher had just clicked through to from the SERPs.
It could've been coincidence-- it was a high-traffic time, and Google was randomly (to me, it seemed) spidering great swathes of my site.
Details:
Google searcher came to /foo/bar.html and loaded the entire page, images, .CSS, .JS, and all. (08 Sept 16:06:25)
16:07:13, 64.68.82.38 Googlebot/2.1 shows up to get foo/bar.html, then at
16:11:03 bot 64.68.82.18 got robots.txt, then at
16:11:05 64.68.82.18 got fred/barney.html, then at
16:14:12 bot #64.68.82.18 got bar/foo/ HTTP/1.0.
So, it got the specific page the searcher had clicked on, then robots.txt, an unrelated page, and then the first page's entire directory.
I don't know what that means, but I thought I'd point it out. Isn't that interesting? I didn't know Google did that, and still don't have proof.
Googlebot came back repeatedly over the next ten minutes and spidered the first directory, the second directory, and a number of unrelated pages.
It also repeatedly got robots.txt, my .css file, and two different .js files once each.
:learn:
But, again, this is just one inexperienced webmaster-newbie with Notepad and the Find function and a PR 5 site, so...
BTW, I'm not entirely sure what HTTP/1.0 and HTTP/1.1 actually mean, but it seems that it indicates that the visitor is examining the contents of a directory to see what to get? I'm not sure, so any enlightenment is welcome.
<edit-- spelling. I used the spellchecker and it didn't know Googlebot, spidered, HTTP, html, webmaster, I'm, or newbie. Hmm...>
[Moved to Statistics and Tracking ROI forum. - Jill]
Edited by Jill, 09 September 2003 - 06:17 PM.
