10 replies to this topic

[Katok]

Hi everybody.

I got a fresh site, which is not indexed yet, and I use the robots.txt file to exclude the directory with cgi-bin scripts from being indexed at all. Yesterday I looked thru the server logs with awstats and lo and behold - Googlebot came along.

Only what troubles me is that the bot only asked for the home page and the robots.txt - no other page got crawled so far, although I have a link to the site map on every page.

Is that normal for the G-bot? Or am I doing smth terribly wrong? I just need an early warning. I'm starting to think that maybe I shouldn't have used robots.txt at all... Is that theoretically possible that because cgi-bin is not accessible, G-bot cannot follow the links from the home page? I previously used robots.txt in a similar situation successfully.

My robots.txt goes as following:

User-agent: *
Disallow: /cgi-bin/

Thanks,
Katok

[Connie]

your robots.txt file is not hurting you. I think what you described is fairly normal for a new site. My understanding is that Google uses one bot to find sites and others to actually crawl them.

[Renagade Master]

The way I understand it, is that a bot finds a sites and reports it back to the mothership(!) for parsing later. So maybe you will see more activity soon.

[Katok]

Thanks guys

So I'll just sit there and wait for Google to come index it all.

[Randy]

What you're seeing is completely normal.

Keep buiding new content, or if that's already in good shape keep getting more links pointing to your site. Now is the time to do either/both of those as it will pay big dividends in a month or two.

[Scottie]

Once it is indexed, expect a 4-6 month delay before it ranks well for anything. There seems to be a time delay now built in to Google's also- I'd say it is to combat the mini-nets and other link pop scams.

[lorax]

>> There seems to be a time delay now built in to Google's algo

aka The Sandbox

I've seen googlebot query a site's robot.txt file and/or index file a dozen times or more before it finally crawled something else.

[Scottie]

Not the same thing as the sandbox, IMO.

The sandbox is where the hundreds of links you've acquired for a new site are sent to for a period of time, not adding anything to your PR.

The time delay filter affects all new sites as far as I can tell- no matter how many or how few links there are.

[lorax]

I believe you are right in saying that the value of links 'placed on probation' is part of the Sandbox. But so too is the delay or dissapearance of a site from the SERPs.

Some programmers have viewed Google as uncomfortable to rank newer websites until they have proven their viability to exist for more than a period of "x" months. Thus the term "Sandbox Effect" applies to the idea that all new websites have their ratings placed in a holding tank until such time is deemed appropriate before a ranking can commence.

http://www.webpronew...dboxEffect.html

The fact that the site is new is a key ingredient as is the number of backlinks acquired and the time over which they were acquired. In rereading the original post I don't think Katok's site is in the Sandbox. Sounds more like it just hasn't been indexed yet.

FWIW. I've listened to Matt Cutts (engineer) of Google on several occassions and the underlying message I hear is that if looks like spam, smells like spam, and acts like spam, Google will treat it as spam until they're sure it isn't. So if your website has meteoric success with links, SERP position, and is relatively new - you could very well be in for a time-out.

[Scottie]

I don't like lumping them together, regardless of what the venerable WPN has to say on the matter.

The sandbox is discussed as a penalty sort of situation; the time delay for new sites is unavoidable no matter how you promote it from what I've seen.

I think there are ways to avoid having your links sandboxed, but as far as I know there is no magic bullet to avoid the rankings delay for a new site.

[Alan Perkins]

Yesterday I looked thru the server logs with awstats and lo and behold - Googlebot came along.

Only what troubles me is that the bot only asked for the home page and the robots.txt - no other page got crawled so far, although I have a link to the site map on every page.

Is that normal for the G-bot?

As others have said, it's normal. To be extra safe, though, make sure that the HTTP response to Googlebot is 200 (OK), rather than something nasty like a 3xx, 4xx or 5xx series response.