17 replies to this topic

[Mooro]

Hi there,

I've got a site with about 5,000,000 products.

The site has both a logical navigational structure and a series of sitemaps and has been online just a few months.

When I Google site:myurl.com it returns around 500,000 pages.

I've increased the crawl rate in Google webmaster tools to try to get them reading more of my pages but it hasn't had an effect.

Other than five million off site keyword targetted links, which by any stretch of the imagination is a stupendously huge and impossible task, what can I do to get Google to index more of my pages?

Or..... Am I being impatient? I understand it's a lot of pages to read and digest, should I just keep on doing what I'm doing and wait some more?

Any input would be appreciated, thank you.

[Randy]

Other than five million off site keyword targetted links, which by any stretch of the imagination is a stupendously huge and impossible task, what can I do to get Google to index more of my pages?

It's not necessarily a one-to-one ratio that you need. All of those pages will need links that can be found and followed by the spiders in order to get in the index, but those don't necessarily have to be links from external sites to each of your product pages. If you get links from other sites that point to pages above the product pages in your site structure --say to category pages that then link to product pages or that then link to subcat pages that then link to product pages-- that's usually good enough. Assuming of course the product pages are fairly unique.

Or..... Am I being impatient?

I'd have to say the answer to this question is a Yes.

Sounds like you've gotten started off on the right foot since you have 500K pages indexed.

It takes time and effort to build up the kind of trust and authority you're shooting for. It just doesn't happen overnight. Or in just a few months.

Keep creating solid content. Keep attracting solid links by getting the word out. Keep doing things the right way so that users are the top priority. These things will encourage the search engines to sit up and take notice, eventually building up decent trust for and authority in your site.

It's a never ending process, but for the scale you're looking at you're probably looking at 18-24 months for this to come to full bloom.

[Mooro]

Thanks Randy!

The marketplace I'm in is huge, 965,000,000 results for my core term, all my competitors are multi million pound turnover companies, this is just me with a budget that's nothing to the other players, as you suggest I'll keep on doing the right thing for my visitors, it is after all those who count the most!

I'm impatient, I know it.... I'm sure deep down I'm doing all the right things but I want results now!

That said, I'll not moan, as you say we're off on the right foot and time plus effort will hopefully get me the listings I'm so keen to see.

I'm certain the formula for SEO is something like this:

((time + effort) * relevancy)

Thanks again, appreciated!

[Michael Martinez]

I would not advise you to try to obtain 5,000,000 inbound links, but you should certainly obtain as many links as you can without violating people's rights and search engine guidelines, at least until your site has enough internal PageRank to sustain its crawling needs.

[Mooro]

I'm thinking I've got my next six months of SEO strategy worked out, or much closer to having a clear vision of what I need to do.

• More 100% unique high quality content on site = Finish building in house CMS
• Press releases writing and distribution at every relevant opportunity = Build on site news publishing system.
• Articles focusing on a target of deep linked keywords, distro to relevant sites = Lesser quality copy for off site based around our own unique on site copy.
• Affiliate marketing as a merchant = Finish off back end systems; sign up, log in, view sales, invoices and payments.
• Strategic partnerships = Find, contact and convert possible link partners into Affiliates
• Social networking / FaceBook = Devise and write FaceBook application

Pretty much in that order too.

Not forgeting constant monitoring, evaluation and on site evolution.......

Enough to keep me busy!

Thanks all.

[Michael Martinez]

And get links. Without knowing more about your site, that SEO strategy would not be adequate for such a large site of generic or mediocre quality.

It's great that you're planning to add useful, valuable content but waiting for people to find it will be slow going. The problem is that asking a search engine to value 5,000,000 pages of content with minimal PageRank is taking a very long approach.

You'll want establish visibility in other ways by asking people to write about your site, or at least mention your site. You might want to work with a publicist.

And maybe PPC advertising would help you, too.

Think in terms of how to build a brand, not just how to add content to the Web.

[Mooro]

In terms of content I'm biased of course but it's far from mediocre, it's high quality (as in 320 kbps / drm free / legal) em-pee-three downloads. Very, very, competitive marketplace, the players and biggest competitors have budgets of millions, I've not, I love a challenge tho!



Are not the bulk of the points I raised all going in the direction of link development?

Yes, I need someone spinning a web of good propaganda about us!

I've found that whilst there are terms we're not ranking organically for we are still getting visitors from them thanks to coming up in image searches and product search too. The depth of the indexing is getting deeping and the uniques, search terms and customers are growing.

It's a journey with at least a year before I start getting anywhere near the destination.

Of course the ultimate aim is to have this thing suck traffic in like it were matter into a black hole!



I'm only concerned about organic here, PPC doesn't turn me on, nor is it realistic as an answer to the indexing issue.

I get traffic I know some folks would adore but it's not enough for me, Randy was right when he agreed I was impatient!

[Michael-F]

I'm in a similar situation. I've got about 500,000 different pages on my site, and about 75,000 of those show up in a site:url.com search.

However, even pages that don't show up in that search still rank for the terms they're targeting. They still show up in Google Shopping queries, and they're generally rank 1 for their terms.

Basically, the site: searches don't tell the whole story, and you don't need it to show your entire site for every page to be able to rank.

[Jill]

Basically, the site: searches don't tell the whole story, and you don't need it to show your entire site for every page to be able to rank.

BINGO!

[Michael Martinez]

In terms of content I'm biased of course but it's far from mediocre, it's high quality (as in 320 kbps / drm free / legal) em-pee-three downloads. Very, very, competitive marketplace, the players and biggest competitors have budgets of millions, I've not, I love a challenge tho!



Are not the bulk of the points I raised all going in the direction of link development?

Not in a highly competitive industry, no. Your competitors will be obtaining links. Therefore, much though I hate to say it, you probably need to obtain links, too.

I'm only concerned about organic here, PPC doesn't turn me on, nor is it realistic as an answer to the indexing issue.

I get traffic I know some folks would adore but it's not enough for me, Randy was right when he agreed I was impatient!

Although this discussion is about indexing, your ultimate goal should be increasing converting traffic. A robust link profile will help you there in non-search related ways.

I will say no more on the subject. I just got a sense that you don't place much priority on link building. It's a time-consuming task but you need to include it in your SEO strategy as more than a "build it and they will link" philosophy.

[Mooro]

However, even pages that don't show up in that search still rank for the terms they're targeting. They still show up in Google Shopping queries, and they're generally rank 1 for their terms.

Basically, the site: searches don't tell the whole story, and you don't need it to show your entire site for every page to be able to rank.

Yes! Same here. There are worm holes into Google, my product feeds bring users, image search brings users, our API drives partner sites; that too brings in users, I've seen results where we're not listed organically but there are a number of ways of getting to our site through those results.

Not in a highly competitive industry, no. Your competitors will be obtaining links. Therefore, much though I hate to say it, you probably need to obtain links, too.
Although this discussion is about indexing, your ultimate goal should be increasing converting traffic. A robust link profile will help you there in non-search related ways.

I will say no more on the subject. I just got a sense that you don't place much priority on link building. It's a time-consuming task but you need to include it in your SEO strategy as more than a "build it and they will link" philosophy.

I agree a build it and they will link pholosophy isn't enough, however everything I'm doing is targetted and strategic so to tell me I need links and that my goal is converting traffic is something I take umbrage towards. Don't get me wrong I appreciate the train of thought expressed as keystrokes I just feel it's egregiously belittling and brings nothing of value to the table.

I am employing and will continue to employ a strategic, tightly targetted, geographic, contexually relevant and robust link strategy built with conversion in mind, I will not however be employing a strategist or egg-suck tutor.

[Mooro]

On a positive note Google is crawling me more now, on a really positive note traffic has doubled in the last week.

103,745 pages crawled yesterday, that being the most ever so far, the kilobytes downloaded is up but not by a huge factor as I've become obsessed with optimisation and have everything set to max compression, I have however seen a sligh increase in load times but only a couple of hundred milliseconds, I can play with cache settings at my side to compensate that.

Randy hit the nail on the head, I'm off on a good footing but I'm impatient! I'll keep doing what I'm doing as I'm seeing results!

[BBCoach]

I have to ask an obvious question Mooro. Are you wanting to make money or are you wanting guests to your website?

If it's money that you're after, then read this thread for money making ideas. 8 Ways I Promote My Website

If you've done the SEO part, then you are on the right track for the long term; however, there's a lot you can do to make money now.

Just a thought.

[Mooro]

@BBCoach - thanks for the link - and to answer your question, YES, I want to make money!

As for your list:

1) SEO = investment of time
2) PPC = not of interest to me despite Google sending me free voucher after free voucher with increases in value each time
3) Bona fide email campaigns = marketing to customer yes but buying lists to mail cold = no - I'll also SMS our customers who ordered by mobile
4) Featured items on eBay = Wouldn't work for me
5) Local search = Not viable, party as nobody will search for my product and then a geo term, second I want my home address off Google's first page
6) Posting in industry specific forums and online communities - YES
7) Posting in directories - YES

The latter two being governed by relevancy of course.

I see more ways of spending money there than I do making it.

This link was retweeted today: www.seomoz.org/blog/testing-how-crawl-priority-works which I found interesting, I went to bed last think after reading this thinking how I can alter my navigation for optimal crawlage which gives me more of an insight into my original question.

[chrishirst]

Well after reading the usual pseudo scientific BS from seomoz I came to the conclusion they were meaning something that the rest of us a very, very aware of!

Build links to ALL your pages not just the home page.

Not sure how (or why) they manage to get so much verbiage and so many useless "diagrams" into stating something so bloody obvious!!!!



(the answer probably is) Because they have their collective heads, firmly planted up their collective rectal orifices!