Skip navigation
newsletter signup

Determining Keyword Competition

August 5, 2009
[The following is an email conversation I had with a fellow HRA subscriber. – Jill]

Hi Jill,

I'm a Keyword Discovery subscriber, and I use the tool for organic SEO purposes. I've recently discovered that the keyword analysis information (competition and thus KEI info) is based on exact match search instead of partial match. I asked KD why they do this, and they said it's because it gives you a more "realistic picture" of the true competition on any given keyword.

I'm struggling to understand this. The vast majority of people don't put quotes around their search phrases. I choose the keywords and phrases I want to target based on the KEI info returned by Keyword Discovery. But if this info is based on exact match instead of partial match, then aren't we barking up the wrong tree?

Help! I feel like I must be really dense here, because I just can't understand... Your thoughts and input are greatly appreciated.

Ellen


++Jill's Response++

Hi Ellen,

It's because you're not looking at how many people are searching for the words. You're trying to learn *how many other websites might be optimizing* for those words.

That said, putting it in quotes still isn't a very good measure for how many other sites are possibly optimized for the phrase. It's only telling you how many pages have that exact phrase somewhere on them. A much better way to check the competition is to do it manually at Google via the "Allintitle:" command.

For more info, please read this article:

Avoiding SEO Brain Freeze Part One – Hunting for Keyword Phrases

Specifically Step 6 (but you should find the whole thing helpful).

For what it's worth, we use Google's Keyword Research tool these days because it's much more accurate than the others, is free, and you get info straight from the horse's mouth.

Hope this helps!

Jill

++Ellen's Follow-up Questions++

Jill,

Thanks so much for taking the time to reply.

So you're actually collecting your keyword data from two different sources: the number of searches from Google's keyword research tool and the amount of competition using an Allintitle search? Then do you just compare the info from both sources to decide which keywords you're going to target? Do you not use KEI info for that purpose?

So one last question about this whole exact-match vs. partial-match thing for determining competition... Isn't your competition in any given search based on the way the user conducts that search? For example, if the user has done a search with quotes, obviously you'll have less competition in that given search than you would if they searched without quotes?

I may be missing something here, but it seems that if you want to target the majority of search engine users, then shouldn't you gauge your competition based on the number of sites returned using the most common search method (partial/broad search)?

Thanks,

Ellen


++Jill's Response++


Hi Ellen,

You are correct, I'm getting my data from two sources: Google's keyword research tool, and Google's Allintitle data. I create a spreadsheet with the number of searches and the Allintitle number and then it becomes clear which are the "keyword gems."

I don't use KEI at all because it is a useless measurement – all it tells you is which other pages have those words on their page somewhere. That doesn't necessarily mean they're your competition.

You said:
Isn't your competition in any given search based on the
way the user conducts that search? So for example, if the
user has done a search with quotes, obviously you'll have
less competition in that given search than you would if they
searched without quotes?

We assume that the users don't use quotes, since most don't.

Your competition is the number of pages that are OPTIMIZED for the phrase in question. Because it's hard to know for sure who actually optimized and who didn't, the Allintitle command is a quick and dirty way to get an idea. If they've got the phrase in their title tag, then they're at least minimally optimized for it. You then can dig further at the sites that show up for the phrase in question and eyeball them to determine if they really are optimized for the phrase, and if you think your site can beat them.

You also said:
I may be missing something here, but it seems that if you
want to target the majority of search engine users, then
shouldn't you gauge your competition based on the number
of sites returned using the most common search method
(partial/broad search)?

No, because those pages may or may not be optimized. In all likelihood, they may just have one of the words somewhere on the page. So they are not really your competition. You're not alone in being confused, however. It's a common mistake for people to think that somehow the number of searches for a particular keyword phrase tells them how competitive the phrase is.

Here's an article that goes into more detail on that:

Why Use Keyword Research Tools

Here is where you can see all the Keyword Research articles from the past newsletters.

That should help reinforce what I'm saying. You may also want to check out our Keyword Research Forum.

Good luck!

Jill
---
Jill Whalen is the CEO of High Rankings, a Boston SEO Consulting Agency.

If you learned from this article, be sure to sign up for the High Rankings Advisor SEO Newsletter so you can be the first to receive similar articles in the future!
 
 
Post Comment

 Jayne Reddyhoff said:
Jill

This was a very timely newsletter as today a colleague and I have been discussing how to determine keyword competitiveness and whether to maintain our subscriptions to either Wordtracker or Keyword Discovery now that Google’s research tool is so much better than it used to be.

We use Google’s tool extensively for exactly the reasons you gave. But, when we put a list of keywords from Google into Wordtracker’s competitiveness check, it throws out any that have not appeared in its sample database.

If we are researching a relatively low-traffic-volume business area for a client, it is very common for search terms which Google tells us are popular enough to be interesting, not to feature in Wordtracker’s list! I believe this is even more of a problem in the UK (where we are based) than it is in the US.

The limitation of Google’s tool is the absence of a quick way to find the number of competing sites for a list of keywords. We have used allintitle and a straightforward phrase-match search but is there any way to carry out a competitiveness check on a list of candidate keywords all at once rather than doing them one at a time?

By the way, when I tried a search on the phrase ‘google analytics tutorial’, allintitle came back with 10 results, the phrase match search came back with 145. Which one should I believe?

Thanks again for an interesting article

Regards
Jayne
 Ryan said:
Just curious Jill. What ratio/formula do you use to compare competition to monthly searches to determine "high reward" keywords that are really worth going for.

Ideally - finding ones with high searches and low competition.

Do you just divide, or do you use some weighting factors?
 Jill - Deciding How Competitive You Can Go said:
Ryan, I don't have a specific ratio, it's more of a gut check. I actually started a forum thread on that topic a few weeks ago, however, which you might find helpful.

Click the link in my name to find it.

Jane, Google does show a bar graph with what they judge to be the competitiveness of each keyword. It's not very accurate though.

And yes, an exact phrase match will almost always come back with more results than an allintitle search. Just because a phrase is on the page, doesn't mean it's also in the Title tag.

It's not a choice of believing one or another. It's using all the information at your disposal and then making an informed decision as to which keyword phrases you're comfortable optimizing.
 Ryan said:
If I recall correctly, the google competition bar is only advertiser competition - which is not the same as sites trying to rank. Advertiser competition will vary along with cost per keyword, profitability of keyword, industry, niche, etc.
 Jill said:
@Ryan, correct. One can assume if it's got heavy advertiser competition it probably has high organic as well. But the reverse may not be true.
 Alana Nelson said:
Hi Jill,

This article is very timely for me as well, since you're doing a Recession Buster review for our website redesign.

As a small business owner, about four years ago I took on the task of improving our website and learning about SEO (although at the time I had never heard the term "search engine optimization). One of the first resources I found was your newsletter, and after verifying your suggestions a few times using other SEO sources, I decided to pretty much use you as my "guru". And I have not regretted that decision once.

So thank you for the info about the Google tools. I had used the Trellian research tool which you'd talked about a couple of years ago, and I know it helped. Now I look forward to taking my site to the next level of optimization using the Google tools.

And of course, I'm excited to hear what suggestions you have after your quick review of my site. Thanks again for sharing your expertise!
 Jill said:
@Alana...oh crap...that was going to be my big revelation to you in the website review. Will start digging for some more nuggets!
 Ellen Bell said:
Jill,

Thanks for featuring my question in your newsletter. Good to know I'm not the only person who might be pondering this issue!

I've been scouring the related articles you mentioned, as well as the forum posts on keyword research, and I hope you'll indulge me by answering a couple more follow-up questions.

First, I have a website I've optimized for a pretty competitive keyphrase. According to what I've read in the articles and forum threads, my understanding is that I should examine the top sites showing up in an allintitle search, because these guys will be my real competition. I should evaluate things like their PR, # and quality of inbound links, etc. to determine if I stand a chance of beating them in the SERPs.

So I did an allintitle search for my target keyphrase, intending to scrutinize the top 10 or 20 results, and much to my surprise, my website is #13. But if I just do a broad/partial match search on my target keyphrase, my site is languishing somewhere around #550. What does it mean??

And second question: In one of the forum threads, I saw a forum member reference doing an intitle search on Google. How does an intitle search differ from allintitle? I can see some slight variances in results for each, so I assume there must be some difference?

Thanks!
Ellen
 Jill said:
More good questions, Ellen!

Actually, it's not the allintitle ones you need to scrutinize, although that can help. But the ones that show up in the top 20 results for your phrase are probably the ones you want to see if you can beat.

I know that seems confusing given what I've previously said about Allintitle. But remember, that's just really a quickie way to get some idea of the competition. Doesn't mean they are your exact competition.

As to "intitle" vs. "allintitle" I don't actually know the difference. I think Google may have some documentation on them in their Advanced Search area. Neither of them work exactly as they should, but again, just provide some frame of reference.
 April Masitwe said:
Hi I am a new visitor to your site but I found the information quite insightful. I ended up here through searching in google - wanting to see if there is no way one could enter a list of keyword phrases and simultaneously establish the number of competing sites for each keyword phrase. It is quite tedious working on each keyword phrase at a time to determine the number of competing sites.

I am new to developing own mini sites, and have no operating site at the moment. Will join and update my details as soon as my first site is up and running.
 Michael McDonald said:
I have a couple of answers and a problem.

First the good news. To answer the question about the "intitle" vs. "allintitle" terms that Ellen presents, the "intitle" term searches for page terms that include a "title word." The Google expanation is below.

intitle:
The query intitle:term restricts results to documents containing term in the title. For instance, [ flu shot intitle:help ] will return documents that mention the word “help” in their titles, and mention the words “flu” and “shot” anywhere in the document (title or not).

Google's page below describes these search terms and others:
http://www.googleguide.com/advanced_operators.html#intitle

Now the bad news. I tried to use the "allintitle" search method on Google's home page. If I try to search on phrases that are similar, such as "CAD Bay Area", "computer aided design bay area" and "Design Bay Area" I will get locked out of Google search and the results page comes up as:

We're sorry...
... but it appears your computer is sending automated requests.

with a further link explaining "•The 'We're Sorry' message appears when Google detects that a computer on your network is sending automated traffic to Google. Automated queries are against our Terms of Service."

Is there some way that you know of to avoid this problem yet still search for the best terms to use in my keywords?

Thank you,
Michael
 Jill Whalen said:
Hi Michael,

This has been a problem for the past few months. I did find a workaround, which I hope still works. That is, you go to Google's Advanced search page and do the title search there. They have a field for "words that appear in the title." It's slow going, but it's better than nothing.

Basically, they don't like SEOs who are the only ones doing allintitle searches so they're blocking us out.
 Laurie said:
Hi Jill,

I have taken your SEO tutorials on Lynda.com and have found them to be extremely helpful and more useful information provided by you than any other resource I've found. So, thank you for all your great help!

I would like to comment on the "all intitle" problem discussed previously...I am doing a google advanced search and it does not have a field entitled "words that appear in the title". The only options are "all these words" "this exact wording or phrase" and "one of more of these words".

I tried "this exact wording or phrase" and it gave me a result of 1,380,000 vs. a result of 9 results by using the allintitle:" " search. These results are obviously inaccurate. Please help!
 Jill Whalen said:
Hi Laurie,

Glad you liked the Lynda course!

What you need to do in Google's Advanced Search page is click the plus box that says "Date, usage rights, numeric range, and more." That's where they're hiding the title thing these days. Keep it quiet though, or they might take it away all together! :)
 Caitlin Grussing said:
Hello! It seems that the above info was shared some time ago, and I'm wondering how much of the information is still relevant at the end of 2013? Do you have any new information that would help us determine keyword competitiveness? Thanks!