A Google Sandbox Test - March 1, 2006, Tracking a newly active domain Options

[Michael Martinez]

I've been promising for weeks to release a new domain and see what happens to it. I decided to let it go live today even though it still lacks one feature I've been meaning to add. I'm just too sick to make a fuss over it. But the timing is too good to pass up the opportunity. I stopped blocking robots this afternoon and put a link to the domain on Xenite this evening. I also added a link to one forum at SF-FANDOM.

So, I'll use this thread to officially report how the site does over the next few months. If I get into really technical details, I may post them at Spider-Food. But I don't expect this to get too technical.

The site launch is much lower key than I intended. This may be a better live test than it could have been, since I could have used my connections to get some immediate visibility. Until I add the blog, though, I don't want to make a big fuss. The site has other resources, but the blog was going to provide the flagship content.

[incrediblehelp]

I remember you saying something about this in other threads.

So you are positive that this website has no other IBL's other than the one you just added? Also are you going to hide the identity of the URL from everyone to ensure your testing is accurate?

Also will you be willing to let everyone know how long this website has been online with robots.txt blocking spiders? As we have learned in many other threads and forums it seems spiders are getting worse and worse at adhering to robots.txt.

I remember my initial concern with this test was that if the website was registered and live even for a small duration that it could skew your testing. Like I said before I wish you just bought a domain and put it online the same day the test starts to make sure no other factors could have been involved.

Good luck. I will be following.

[Michael Martinez]

I remember you saying something about this in other threads.

So you are positive that this website has no other IBL's other than the one you just added?  Also are you going to hide the identity of the URL from everyone to ensure your testing is accurate?

I am not going to mention the site by name or URL in this or other discussions, but the link I placed on Xenite is prominent. It's on the first page. This is a natural way of announcing a site, and the goal is indeed to garner what I would describe as links from genuinely interested parties. My original plan was to put out a press release and get a few very visible sites to announce it as well. I may still do that in a week or two.

At the time of this posting, I know of no other backlinks. I can and will generate more of my own later on.

Also will you be willing to let everyone know how long this website has been online with robots.txt blocking spiders?  As we have learned in many other threads and forums it seems spiders are getting worse and worse at adhering to robots.txt.

The site has been online since September 2005. Robots.txt was in place since that time as well. According to my server logs, several robots have fetched the robots.txt and main index page but they haven't gone any further. I cannot find the URL in either Google or MSN. Yahoo! reports the URL but shows no information about it.

I remember my initial concern with this test was that if the website was registered and live even for a small duration that it could skew your testing.  Like I said before I wish you just bought a domain and put it online the same day the test starts to make sure no other factors could have been involved.

The purpose of the test is only to see how long it will take a previously unindexed site to get past the initial period of little to no trust.

Getting a proper domain name on a last-minute notice is not easy. This is a very specific research-oriented domain and the name was important (for branding, not for SEO).

There is another domain of a similar name which has been online but inactive for 3 years. That domain name already possesses some visibility. So I am also going for a legal and moral test with this domain name.

The other domain name has very few links, but there are some very important links pointing to it.

[Deeves]

Just wondering, but most people say it takes about 6 months for new sites/domains to get out of the google sandbox. Can someone clarify if the time starts from when the domain has been registered (since some people say doing a 5 yr registration can help you get out faster) or is it from the time you launch your site and google first spiders it.

If it is from the time of domain registration.. then this experiment seems quite flawed.

[incrediblehelp]

I doubt many consider the start time of the "sandbox" to be from the time of registration. Many buy domains, but may not do much with them for years or months.

My only point was that if was a prominent domain it could get exposure and links even with the robots.txt blocking spiders which could skew the test.

[seadog]

Just wondering, but most people say it takes about 6 months for new sites/domains to get out of the google sandbox. Can someone clarify if the time starts from when the domain has been registered (since some people say doing a 5 yr registration can help you get out faster) or is it from the time you launch your site and google first spiders it.

If it is from the time of domain registration.. then this experiment seems quite flawed.

From my experience its from the date the site is crawled. If the site remains consistent and grows(content and links) but still remains relevant it can come out of the box

[Michael Martinez]

There are many possibilities and hypotheses. One domain can really only effectively test one or a very small number of them. Whatever the outcome of this exercise in observation of the progress of a natural domain's path to visibility, it won't be anything definitive. I will not characterize it as such at any point.

[ACpeter]

I find all of this interesting because we launched another (new) site just about two week prior to this test starting.

Our site was a newly registered name. On that site we're using Article Manager to create an entire series of short articles addressing specific industry or job related topics.

Part of the reason for using Article Manager was that it includes an easy to use WebFeed. That let's us put relevant 'headlines' from articles on other sites, adding to the visitors experience. However, we did a couple of things to help the new site get crawled.

-From the home page of the new site, we created a link to a site map page that provides static links to each article.

-On more established sites where we have added the syndicated content, we have also added static links direct to each article (Article Manager utilizes a javascript code in the syndication. The syndicated headlines are not included in the chached text of the pages where the syndicated content resides)

Right now I am giggling because the new site is appearing in every search engine. Now, a few days ago G showed about 75 indexed pages (which there were actually about 80 articles or pages in total) Today that's changed and most of the pages are not showing up in G. I am assuming that's part of the sand box issue.

Overall, we're getting great listings (sometimes 2 in top 10, no kidding) in MSN. Yahoo is doing ok too and G is even including the site, although not as regularly.

I would think most of the action is because of links from some of our more established sites, but like I said, in some cases the new site is already getting 2 pages listed in the top 10.

I guess this forum does help if you read it (IMG:http://www.highrankings.com/forum/style_emoticons/default/thumbup1.gif)

[Jill]

Right now I am giggling because the new site is appearing in every search engine. Now, a few days ago G showed about 75 indexed pages (which there were actually about 80 articles or pages in total) Today that's changed and most of the pages are not showing up in G. I am assuming that's part of the sand box issue.

No, actually the sandbox/aging delay has nothing to do with indexing, it's all about rankings for keyword phrases that have some slight competition.

[ACpeter]

Jill,
Thank you. I understand what you're saying, but I guess I wasn't clear. The new site is indexed AND already coming up in organic searches. I understand (from reading here) that this may not be long lived for a new site, but at this time we have seen some pages from the new site already showing up in searches for good KW's. Not sure if that's because it's linked to from the established sites or just luck.

Sorry if I wasn't clear. We'll keep you informed how this plays out.

[Michael Martinez]

Right now I am giggling because the new site is appearing in every search engine. Now, a few days ago G showed about 75 indexed pages (which there were actually about 80 articles or pages in total) Today that's changed and most of the pages are not showing up in G. I am assuming that's part of the sand box issue.

Google is going through an update process that is not expected to end before April. You have to be patient with whatever you see in Google and understand that other people are probably seeing other results.

[Michael Martinez]

An update:

Google
The site was crawled on March 1.

I used the page design from another site for a template and did not realize I had overlooked a header from the old site. So Google crawled several pages with the wrong header before I was able to fix the problem. I noticed the problem right away, so Google fetched those pages while I was inserting the correct headline.

The cache dates all read March 1.

There are two prominent links to the domain on Xenite and SF-FANDOM. Actually, the SF-FANDOM link is embedded in a forum message, but it shows up as a URL reference in Google, so they crawled the forum and indexed the message announcing the site.

The site comes up for its primary title (techically, a secondary page comes up first -- and it's one with an abbreviated HTML formatting that I whipped together quickly without following the standard from the other pages).

The site's title is not competitive, but it does contain a two-word expression which is the title for another site that is well-linked. There will be some head-to-head competition between these two sites in the long run, as far as that two-word expression goes.

There are only about 10 pages on the site right now, but it will grow. I intend to set up a blog, although I've asked my partner to do that and she has been very busy. That may be a "do it yourself, Michael" kind of busy or it may be a "I'm really busy and will get to it ... eventually" kind of busy.

So, the site will grow and should build up some internal linkage. I have not yet sent out any announcements to the community that would be most interested in the site, except for some links I posted to a mailing list that is not archived by the search engines.

Two other sites that I don't control are linking to the new domain. I did include it in an RSS feed that I generate, and one of those sites is carrying the RSS feed. The other site appears to have provided a natural link.

The site carrying my RSS feed is not a spam site. It carries news and comments in another language but also picks up some RSS feeds for margin link content. I am reluctant to classify RSS feeds as "natural linkage" but this appears to me to be a natural, non-spam site making good use of available RSS content.


Yahoo!
Yahoo! has also indexed the site, as well as one link or reference from my blog (Google apparently has not transferred that link into its main index, but I know they indexed the blog article).

Yahoo! has picked up two other pages that carry the RSS feed in addition to the one that Google picked up. One of the pages is the "home" page for the RSS feed on Xenite, and the other appears to be a dynamic content page that carries no ads. It appears to be an old user start page service (RSS was originally used for start pages, before personal aggregator software was developed). I didn't know any of these services were still around.

The domain comes up second for an EXACT FIND (quotes around the expression) search on the site name. It does not show up in the top ten for the two-word combo I mentioned above.


MSN
MSN has indexed two pages from the site and it got there from the SF-FANDOM forum message. I can find no other references to the domain in MSN.

The domain comes up for its name on MSN.

It also comes up first for an EXACT FIND search on the two-word combination I described above. And it comes up first for a FIND ALL search on the combo.


Ask
Ask has not yet indexed the domain or picked up any references to it, except for caching Xenite's main index on March 6.


AOL
AOL uses Google's database and it's showing pretty much the same information as Google.

[Michael Martinez]

Directory Submission

I have just now decided to submit the site to Yahoo!'s directory. There is an appropriate category (sort of) that I feel the site would fit into. Technically, I think it would better fit into an educational category, but I doubt Yahoo!'s editors would agree.

It has been several years since I submitted a domain to Yahoo!'s directory. I am well aware of the frustrations of many Webmasters who have submitted to Yahoo!, even through paid inclusion, without getting any results. This particular site is unique and should prove to be a useful resource for a long-established and growing online community.

Of course, since it's a non-commercial site, I used the free submission tool. I can only assume it's now queued up behind a huge backlog of spam sites, mis-catalogued sites, commercial sites hoping to slip in under the radar, honest hobbyist sites, productive NGO sites, etc.

Maybe in a year someone at Yahoo! will look at the site and say, "Hm. Maybe ...."

I'm not going to hold my breath.

So, now I've done that, I might as well submit to other directories. My criteria for directories are:

1) Must be free
2) Must have related listings already
3) Must be indexed by Google
4) Cannot shove GoogleAds or Yahoo! Ads in my face
5) Must have a reasonable submission process
6) Prefer not to be a DMOZ clone

I don't care about whether they use a URL redirection script. I'm not looking for link popularity or PageRank.


DMOZ Technically, I haven't decided on whether to submit the site to DMOZ. Frankly, I can do without the headaches of dealing with DMOZ's loony edits showing up in Google's search results.

JoeAnt I don't have time to become an editor at JoeAnt and I don't care to pay them $40.00 for a link. So, I'll pass on JoeAnt at this time. But if they run another discount special on submissions, I'll probably submit the site at that time.

GoGuides is unbelievably sparse in content compared to what they used to have. So there seems to be no point in submitting to them.

Gimpsy is just too weird. I pass.

Xoron has been shut down.

Zeal I agonized through Zeal's Java-based submission process. Fortunately, I signed up for the service 6 years ago and passed their exam, although I've long since lost my "Zealot" status due to inactivity or something, so the submission will be reviewed.

Lycos has disabled their free InSite submisson service. I think I used it last year, but now you can only do paid submissions. I did add a link to the site in the My Favorite Links section on MyLycos, but I seriously doubt that will influence anything.

WebWorldIndex Never heard of them, but they are listed in Brian Turner's directory of free Web sites and Google has indexed a lot of their pages, so they seem to be legitimate.

TurnPike Emporium Directory Another one from Brian's list of free submission directories (which, btw, needs to be updated as a couple of them now require paid submission). Looks sort of like a DMOZ clone? Oh well. The deed is done.

Amray looked interesting, but their requirement that I use an email address from the domain being submitted killed the deal. Too bad. I liked their other stringent requirements (such as if you host with a specific company, they say Go Away (IMG:http://www.highrankings.com/forum/style_emoticons/default/lol.gif) ).

Brad Enslen's SciFiMatter.com. SciFiMatter is a specialty directory. Brad is a neat guy, too.

SciFiSource.com. Another specialty directory.


And that is it for my directory submission. At least for now. I'll leave DMOZ as an option, especially if Turnpike is not associated with them. But I really think DMOZ could do us all a big favor by going quietly into the night and not coming back.

One of my old stand-bys, Solar Flare, has apparently discontinued its directory service. They used to send me traffic, and I'm sorry to see that Eoghann had to give up on what I felt was a truly useful resource for the science fiction and fantasy community.

[Michael Martinez]

Ack! I just remembered another specialty directory devoted explicitly to the topic of the new domain. I have submitted the site to OneRing.Virbius.org. They redirect, but they are an important resource for these these types of sites.

NOTE: I've received some private messages about this domain. I don't mind discussing it in PMs, but everyone please keep in mind that if you post the URL in this discussion thread, that could spoil the experiment. I only get one shot with this domain name. I don't have any others waiting in the wings at this time.

Thanks.

[Michael Martinez]

I need to get to sleep, but I wanted to post a quick update. In looking at my referer stats, I see that the new site has actually gotten 2 hits from a natural search query (that is, a query I have not used myself to look for the site in the search engines).

People/robots from more than 20 countries have visited the site. The traffic is still relatively small, but as I have an international audience and have announced the site to a small group of people who subscribe to a mailing list I maintain, the number of countries is reasonable.

Referrals come from MSN, Google, Netcraft, Whois.sc, Yahoo mail (probably one of my subscribers clicking on the link in the email I sent out), a few sites in my network, and of course by direct request.

There is also a Tagworld reference I'm not familiar with. I'll have to check it out some other time.

The natural query has seven terms in it. The actual name of the site (not used in the query) only has five terms in it. Right now, the site does come up in Google for that five-word name.

Traffic has been fairly steady throughout the month. A couple of days have produced fewer than 10 visits but most days there are between 10 and 30 visits.

MSNBot has shown up 34 times. Googlebot has made 42+ fetches.

Inktomi (Slurp/Yahoo!'s robot) has made 25+ fetches.

There are some hits from AOL proxy servers. So far as I know, it's impossible to know (without setting cookies) how many people actually look at a site through a proxy server. At least, the raw server logs should not be indicating accurate viewer data for proxy servers unless you are setting cookies. I don't set cookies on this domain.

Over 200 sites have fetched content from the domain so far. But quite a few of them appear to be different instances of search engine and proxy server robots. Maybe 30-40 sites are robots that I recognize.