Are you a Google Analytics enthusiast?
More SEO Content
Deep Crawl By Google
Posted 30 March 2004 - 08:40 AM
I've looked through some archives and can't find much info about this... I've got some sites that have been up for 6 months to a year and are very well indexed, ranked, optimized, running well and all the rest. Virtually every page is indexed by Google.
HOWEVER, I have a couple of sites that I've had up for three or four weeks or so and they've not yet been indexed beyond the first page. In all cases, there are a few incoming links from good sites, there is a very clear text-link navigation on each page, and there is a site map (called sitemap.asp and sitemap.html respectively) but still no deep crawl...
Can some of you kind souls educated me about deep crawls - i.e. what can i do to encourage or speed this along (i'm assuming either wait or keep plugging away for inbound links which is advisable anyways) ???
Thanks ahead of time!
Posted 30 March 2004 - 08:51 AM
I have had a site that was up for just under 2 months, Gbot visited 20 odd times taking only the first page, then bingo, it took the lot. At first it freaked me out, I was looking for a reason as to why it was happening, what was different about these pages, counted words, checked code etc, then just though OK as you wish.
I have just looked at a forum (not this one) and there were 4 googlebots all there at the same time (or 4 pages being spidered I should say) as they wish so shall they do
Posted 30 March 2004 - 08:52 AM
And... hey presto within a few days (about a month after Google's first visit) the full crawl happened.
Provided you have a good internal text linking, I'm sure it's just a matter of time.
Posted 30 March 2004 - 08:54 AM
Posted 30 March 2004 - 09:38 AM
The halfway indexed site, has been regularly refreshed in Google, but the other one is still not showing any dates in the results, and the new pages are still not listed.
Posted 30 March 2004 - 09:47 AM
On my business website I typically link to sites I've launched recently since its not bad to have a link to those sites plus it helps educate the visitors about our work and recent news...
I'm considering changing this link and perhaps targetting a few new inbound links going to an interior page rather than the homepage. Any thoughts here?
Posted 30 March 2004 - 10:03 AM
Links going straight to the correct page should help visitors and your rankings.
I'm considering changing this link and perhaps targetting a few new inbound links going to an interior page rather than the homepage.
I never get questions back, after sending the proper link in replies to enquiries.
You might get the case that Google is only going to bother indexing that one page and still leave out the rest. If your home page links to a deeper page, then you would expect the spider to follow it. but you would not always expect it when an external site links to one of your pages.
If think that the main Google servers are lagging behind with processing all the new data, especially ever since all those other document types were added since the data is growing exponentially.
Posted 30 March 2004 - 02:40 PM
This means that if your site is slow (as in slow hosting or heavy traffic) Googlebot will slow down too - in order to not do bad things to your server.
I believe the same thinking also applies to dynamic sites - it slows down in order to not crash the site - especially on deep crawls.
Is this the only factor? No. I doubt it's even the major factor (that's probably links and useability), but it IS a factor. Slow doesn't mean stopped - once again, as long as you have checked to make sure there are no spider traps or other issues, then - be patient.
Posted 31 March 2004 - 10:03 AM
When I originally posted this I had two specific websites in mind. One is a client's site which is a brand new domain and site and the other is one of my 2 websites for my business (I have two business websites which are branded differently and appeal to different clientelle. One also involves a partner of mine in business while the other is just my business.)
Anyhu, I put a link on the homepage of business site A (which gets spidered daily for the most part) going to an interior page on business site B yesterday or perhaps the day before and low and behold today ALL pages in business site B have been indexed. It has been about 2 to 3 weeks since the new version of the site for business site B was uploaded and launched. Prior to that the homepage of business site B had been spidered for years but nothing further as it was mostly a flash site (before I got a hold of things!!!).
I'm not sure if its b/c of the homepage link or just a matter of time, but the deep crawl has come! However, the other website I was originally referring to still has not been deep crawled.
Posted 31 March 2004 - 10:04 AM
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users