17 replies to this topic

[Hippo]

Is there any information out there on how to "programatically" determine your ranking on search engines? Several softwares do it, such as WebPosition Gold, Search Engine Commando, IBP from Axandra, or other web-based software.

What is the programming involved in doing this? Is it XML? And how difficult is it? Are there any resources out there that explain it?

By the way, if this is some sort of trade secret that should not be discussed, please feel free to let me know that as well. (Or email me)

Thanks!

[OldWelshGuy]

Hi Cory, IMO the only programme out there that can determine your position in the rankings 100% accurately is the very same Algo that the SE uses, anything else is different.

Some of these tools are good for making your life easier with regard examining certain areas of SEO, but unless you can get hold of the Algo from Google then its all assumptions based on search results, ie they build their prog around their own prog, Kind of like a self fulfilling prophecy.

If you want to see why sites rank above you then you have to sit down and look at WHY, you need to look at every aspect of the pages above you to find out why they appear above, not below you.

I have found that if I was close to the top of a large pool, then the sites above me might be the tiniest of scores above, so much so that it was a nigh on impossible task for me to decipher as there are many elements and combinations of elements in the 'if, then' department.

And as for 'insider trade secrets' if anyone on this forumhad the google algo, they would not be on this forum, they would be sat on a beach, counting the millions they made from selling that algo to the spammers

[Jill]

OWG, I think Corey just wants to know how to create a program that automatically checks rankings.

Corey, I can't help you there, but why reinvent the wheel when there are numerous programs already out there that do the same thing? You'd have to talk to a programmer to learn how to program your own software.

Good luck!

Jill

[Alan Perkins]

Is there any information out there on how to "programatically" determine your ranking on search engines? Several softwares do it, such as WebPosition Gold, Search Engine Commando, IBP from Axandra, or other web-based software.

Most rank checkers just submit the query direct to the search engine and parse the HTML search results.

In many circumstances this contravenes the engine's TOS and/or robots.txt file.

[Ron Carnell]

The methodology used is called "scraping," so if you do a Google search on screen scraper you'll likely get a whole more information than you really wanted.

First, remember that Internet Explorer, or any other browser, is just a software application that knows how to (1) talk to servers on the Web, (2) request specific HTML pages from those servers, and (3) display those pages on your computer. A screen scraper does (1) and (2), as well, but instead of displaying the page, it analyzes it. Since the search engines results pages follow a distinct pattern, it's not too difficult for a scraper to read through the page(s) and find where you rank for any given keyword. It's doing exactly what you and IE do; it's just doing it a whole lot faster.

[Ron Carnell]

LOL. Yea, what Alan said ... much more succinctly.

[powerofeyes]

Hello Coree,
You can write rank checker program in any programming language or application development program provided you know the exact syntax to query the google database. When I was lurking in WMW threads I came across brett's posting on a rank check program for google written in Perl script. Just check if it helps,
http://www.webmaster...orum13/1156.htm
I am always overwhelmed by brett's coding skills. And this code seems to take only 30 mins for him.
Also check this thread ASP designer has talked about writing a rank checker in VB. http://www.webmaster.../forum47/97.htm
Hope that helps, infact I am writing one in VB.NET with ASP designer's suggestion the commands used are excellent check it out,

VIJAY.

[bumblingnewbee]

hi, remember, its against googles terms to use automated searchs for web position information.

while it may not get your site banned, it may get your IP banned.

With that in mind, here is the link to a program already written. It is fast..ok very fast, effective and accurate and you can do very, very deep scans. Best of all, it's free.

[http://www.cleversta...le-monitor.htm]

Cheers.

if you have any questions, you can shoot me an e-mail
Warmest Regards

Edited by Jill, 01 December 2003 - 06:13 PM.

[Jill]

Bumblingbewbee, not sure if I welcomed you already, so if not...welcome!

Isn't that software querying Google automatically, against Google's TOS, just like WPG and TopDog?

Jill

[bumblingnewbee]

Hi Jill,

Yep Yep...is a very big no, no by most search engines. The safety is, they can only identify you IP address as the source of the query, but, they do not know what it is your looking for with regards to the web site.

Hense my warning...

It can get your IP banned from using the service (nothing worse than changing IP address...wink wink...lol)

But the chances of getting your site banned are slim and none. The program looks for your site address ie..http://yourdomainname.com.... when it finds it for that keyword or phrase it's on to the next one. if your not listed, depending upon how deep you go, it returns an N/A. , but it does not submit your site address as a part of the query. Hense the safety.

The best part about this one is, it's free (my favorite part) and it only queries your search terms, not your site.


But...my disclaimer is, use at your own risk....

[GeordieSEO]

Web position simply runs lots of queries to the search engines, true this gives the IP address of your machine or proxy but it doesn't necessarily give the domain name, depending on how it has been setup. e.g. I am behind a proxy server and appear to be a query from a NTL fat pipe connection and our web servers are hosted at a completely different location and IP. Web position saves the web pages and then searches the results for your domain name.

I believe Google will penalise a site if you run too many queries a site in a short period of time.

The main reason why they don't like auto search programs is to prevent server overloads and denial of service attacks.

The simplest way is to run the autosearches when the search engines are quiet and to increase the gap between submission of the auto queries.

There is a tool, which I can't remember if it is free or not, which uses Google's API to generate position results. Under the terms and conditions of Google's API you can safely perform up to 1000 searches per day.

Does anybody know of somewhere that has been banned by Google for oversearching? (Rather than a deliberate denial of service attack). Given the number of ISP which use proxy server there must be millions of searches appearing to coming from a proxy at any given time.

[Alan Perkins]

There is a tool, which I can't remember if it is free or not, which uses Google's API to generate position results. Under the terms and conditions of Google's API you can safely perform up to 1000 searches per day.

You need your own API license to run any tool based on the API. The terms and conditions of the API don't allow commercial use.

There is no way, within Google's various terms and conditions, to run automated queries for commercial purposes.

[Jill]

Yep Yep...is a very big no, no by most search engines. The safety is, they can only identify you IP address as the source of the query, but, they do not know what it is your looking for with regards to the web site.

Just so long as you and those who use it understand that they are violating Google's terms of service every time they use it (whether or not they get caught).

IMO, some of these software programmers seem to think that if they make their program so you can't get caught, then it's okay. I guess that's something that everyone has to decide for themselves. I liken it to speeding...

Jill

[alltuckerdout]

Does that mean that I shouldn't be using Digital Point either to run search queries?

Can I get in trouble for this?

Thanks,

Janet

[Jill]

If it uses the API, it's doubtful you'd get in trouble.

If you're using it for commercial purposes, then use your own judgement with what you're comfortable with.

I do any rankings checks at Yahoo which uses Google's database.

Jill