Are you a Google Analytics enthusiast?
More SEO Content
Posted 27 September 2005 - 03:38 PM
1)query a specific SE with a designated search phrase
2)grab some fields from the WHOis data for sites ranking in specific spots for that phrase
3)export the data as CSV
Now, I am something of a programming nitwit inadvertently turned aspiring geek. This is to be my spare time activity. I assume I need to get into the whole API business - that's all currently a mystery to me. But I am willing to put my nose to the proverbial grindstone. Am I on the right track; can API's be modified to do this: am I dreaming; is there a good place to start (I've looked through a number of downloadable scripts and not yet come close to relevant); should I go back to my few real world skills? My hobbies used to include harmless pursuits like fishing - I am at the top of a slippery slope...
Posted 27 September 2005 - 03:45 PM
The API provides you with the specifications for querying a remote database to retrieve specific data sets. Think of it as a highly specialized programming language intended to be used by any application capable of sending and receiving data in a dynamic format.
You would have to have access to one or more registrars' WHOIS databases. I don't know if any of them make that data readily available. I do know that many of them block other registrars from querying their databases, so you'll never be able to get all the domain information out there through any APIs.
And you cannot scrape because they are using Altavista's authentication system, where you are presented with a graphic and have to type in the letters and/or numbers displayed in the graphic.
Posted 27 September 2005 - 04:00 PM
The only problem I see is going to be pulling the WhoIs data into the CSV. I have a WhoIs thingee I built long ago and far away that tries to determine the WhoIs registry that you would need to conenct with for any IP/Domain Name, but I don't recall seeing any of those that offer API connections exactly.
I suppose you could always link out to it to fire off a browser for the WhoIs portion, though you may be able to pull the info into a data file too. I've never tried it honestly, but I can almost formulate how it could be done from a general query from a web application.
Does whois.sc have an API by chance? That would be the first place I would look.
If you want to grab the stuff I've made it's over on my personal site. The basic API's (save one I haven't found the time to clean up/finish up yet) are GPL'd and available for download. The WhoIs is there too, but I've not made the code available for that one. I'll put it on my To Do list though.
Posted 27 September 2005 - 04:14 PM
Thanks for the input Micheal and Randy. Though Randy, I take your "I can almost formulate" as somewhat daunting after looking at what you have formulated. I will take at look at your site. Everyone needs a hobby.
Posted 28 September 2005 - 07:10 AM
There are SDK's out there --and plenty of example API scripts-- for both Google and Yahoo. MSN is pretty new so there's not a lot of documentation yet. They only released it like a week and a half ago. But in my little testing I did one weekend, it's pretty similar to Google's API since it uses SOAP. The variable calls are different, but it's close enough to deal with, even without good docs.
The way I'm going to proceed from here on out is pretty simple really. I'm planning to compartmentalize things a bit. Have a file that has the Yahoo way of doing thing, a file that has the Google way of doing things and a file that has the MSN way of doing things. Then in my main file I'll standardize what I'm going to call each variable so that it will work for each of the three seamlessly. Then simply include whichever file needs to be included, based upon the user's choice of engine to query.
Once this foundation is set, adding in capabilities of other API's should be a pretty simple matter. Much better than re-writing everything from scratch each time!
Of course I need to find a day or two I can dedicate to the project. But once it's done I'll most certainly release the code under GPL as normal. I find the API stuff, specifically the Search API stuff, fascinating. I personally think it lends itself to all sorts of neat applications that are as yet untapped.
If you have info on the WhoIs API I'd certainly be interested! I've never looked into it honestly. Does it go through Whois.sc by chance? They seem to be the most active in this sort of data sharing.
Posted 28 September 2005 - 09:26 AM
i agree.. i brought this topic up recently with some designers and programmers here (where i work), and they all kind of just stared at me with a funny face. they had the usual response like, "it won't directly benefit our clients.. they have no use for this type of thing, etc.."
to which i replied, "isn't it our job to find a use?" regardless, i am amazed at the degree to which 'professionals' (especially the programmers) are still in the dark about API's and their potential. i told them the sky was the limit concerning how we might use them to our benefit as well as the clients. point being, i was trying to spark a discussion on possible useful implementations, to no avail.
i'd love to go over potential uses or brainstorm with someone.. anyone?
Posted 28 September 2005 - 01:26 PM
I'm sure there are lots of others out there that simply haven't captured my attention yet.
Yes, I'd love to brainstorm concepts. That's always where the really intriguing ideas come from.
FWIW, I've already purchased a domain and my intent is to someday make many of these installable apps people can download to their own computer. That doesn't help out with Google's API query limit, but it certainly will with Yahoo and MSN.
Posted 28 September 2005 - 01:55 PM
i purchased a domain recently as well.. i plan to focus it on this very subject (more the brainstorming and various uses of API's than my own application development). i'm not much of a programmer to begin with. but i agree the uses are unlimited and potentially ground breaking.
maybe we could start by listing some of the current uses and work from there.. but before we do, should this be moved to its own thread? i don't want to dilute the original question posed by clueless.
Posted 28 September 2005 - 03:04 PM
But there are also the more niche developments for information junkies like me. This kind of think opens up all sorts of industry specific information retrieval/crunching tools. I, for instance, would like to keep an eye on .travel domains as they proliferate. Who's using them, how many, where are they being registered, does their eventual SERP position suggest that the domain functions like an edu/gov? As more major players offer the API option, the directions multiply. The possibilities are endless and a little dizzying and I don't know why everyone isn't more enthusiastic.
I would love to hear everyone else's thoughts/schemes. In the meantime, I am embarking on a new learning curve - though today it looked more like a learning line. . .
Posted 28 September 2005 - 03:56 PM
There was another story from earlier this year (I can't remember where I read it and would have to fnid it) that focused on how the big data repository folks folks were just coming to the realization that opening up their data to other developers basically got them a Free workforce. The idea being that it does no good to have all of the data if it can't be accessed to produce the information that Real People need.
That's the bandwagon all of the data providers are jumping on. And IMO their right in doing so.
The question is, which developers --who are either in the trenches themselves or can get feedback from those in the trenches-- will make best use of the various data streams.
As a developer, that's what excites me. If you have an idea because you're in the trenches or have a good ear and listen to people who are in the trenches you can usually build an application that will accomplish what needs to get done.
Posted 28 September 2005 - 06:26 PM
Posted 29 September 2005 - 10:04 AM
and you know that if microsoft is following suite there must be some potential in this, as they are not known for being the most 'open' company when it comes to their own data.
Posted 29 September 2005 - 11:27 AM
Their initial API SDK release, and the way it stood up until yesterday when I last checked, required one to download, pay for and install Visual Studio. Depending upon the version you're talking up to $1,500 or so per person, and it's still only going to give you .NET compliant code. Not PHP. Not Java. Not anything else.
I personally don't think MS has a grasp on the benefits of the whole API movement yet. Of if they do have a clue it's not filtered down to the folks who control those developer areas yet. It would have been childs play for them to release multiple SDKs the same way everybody else has.
That's okay though. Some of us in the Open Source community are pretty resourceful when push comes to shove. We'll just create our own documentaiton.
Posted 29 September 2005 - 12:15 PM
I figure it's much like riding a bike, in that it'll all come back quickly. But I would need at least a couple of weeks to familiarize myself with software that I haven't used in a loooooooong time.
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users