Are you a Google Analytics enthusiast?
More SEO Content
Is Frequent Spidering Important?
Posted 15 December 2004 - 05:02 PM
In advanced SEO, this info is indispensable, how would one do IP cloak if you didn't know ALL the robot IPs, the big engines don't use just one crawler - and Google and MSN are insane these days with the shear amount of bots and time they spend crawling around. Black hat maybe, but still - impossible without this info. Not to mention how does one construct a robots.txt file without this info - rely on others??...please, engines play with IP and Agent settings all the time to detect spam/cloak/whatever...I like to get this info first hand.
As a bonus, I have almost mapped complete segments of bots, their agent names - for Yahoo, MSN, and Google and what they tend to do (text crawls, image crawls, status 200 check - usually the big engines have purpose for their bots too).
Anyway, just wanted to share that I'm kinda astonished that many of you don't even care...wow! For me, it's easy - all I do is SEO, that is my job title so I don't need to worry about selling users, etc - just getting placement ...so I've got the time to look into data like this...
Posted 15 December 2004 - 05:23 PM
Black hat maybe? Black hat definitely! To be clear, we do not recommend cloaking under any circumstances, and therefore it is unnecessary for us to have to care about the spiders user agents, etc. for that purpose.
Cloaking is always a bad idea.
(If anyone wants to discuss cloaking further, please start a new thread as that's not the topic of this one.)
Posted 16 December 2004 - 09:31 AM
And I guess that's where the misunderstanding falls.
See, most of the people here seem to fall into one of two camps.
Either they're business people trying to build their businesses and learn enough about SEO to optimize their own business's site(s), or at least to know enough to avoid being scammed by a hired SEO...
Or they're people who provide services to these businesses.
Business people -- at least the smart ones -- recognize that positioning and the frequency of spider visits are nice bragging rights, but they don't do diddly to put food on the table or pay the rent. What counts in business, no matter whether in it for the long haul or you're just trying to make a quick buck, is sales.
For us, "advanced SEO" is SEO that actually benefits the bottom line. And there's a lot more to it than just getting top rankings "by any means necessary."
Of course, we care whether the spiders visit and what position our sites get! We just don't care to the degree that you seem to.
We recognize that things that prevent spiders from visiting may also prove to be barriers to humans trying to use the site. So, we try to make our site navigation to be as easy, intuitive and barrier-free as we possibly can. And we try to make the site code as clean as possible, and the page downloads as quick as possible, and the pages as usable and accessible as possible. These things are good for humans and SE spiders, even if they don't fall under the category of "positioning."
And, naturally, we want to get as much traffic as we can, and we know that higher ranking sites tend to get more traffic, so we want to rank as well as we can. But we also know that all the traffic in the world won't do any good if none of it converts.
What doth it profit a man to have top ten rankings and hourly spider crawls if every human who visiteth his site sayeth, yea verily , and hightaileth it for the competition within five seconds of arrival?
So we check for spider visits, sure. But more importantly, we're looking at the paths that human visitors take through our sites. We're checking at what points they enter the site, and at what points they leave. We're tracking conversion ratios and click through rates. Instead of updating our robots.txt and IP-address cloaking scripts, we're tweaking site architecture, page layouts and content to see if we can get a higher conversion ratio.
The name of the game isn't traffic or positioning, it's conversions.
Which is, I think, why the question got raised in the first place. Why do people care so much about how often the spiders visit? Frequency of spider visits has nothing to do with how highly a page will rank, and nothing at all to do with how well a page will convert.
We may call ourselves SEOs or SEMs or Internet Marketers or whatever. But in the final analysis, very few of us here see our business as simply "positioning" and we apparently take a broader definition of "SEO" than what you do. We're here to help ourselves or our clients make better websites (which is not the same thing as "websites that are better at tricking the SEs") that lead to higher sales and more profits.
I think more business people are discovering that while having a website is cool, having a website that actually makes them money is even cooler. And they're becoming more sophisticated about what it takes to get there. It's going to take time, but IMHO the days are numbered when SEOs could make a killing with simply positioning and take no responsibility for whether the site converts any of its traffic to sales. At least with the smart business people...
My YMMV, of course.
Posted 16 December 2004 - 10:34 AM
I always disagree with statements about SEOs not being involved in marketing, and their only responsibility being rankings, but that leads to discussions of what we ought to call ourselves, and that is
Posted 16 December 2004 - 11:53 AM
Just becaue you don't agree with me, I guess - doesn't mean my definition of SEO is any less valid than yours, maybe not even different - I might even go as far to bring more flames to my post by saying my definition is more complete...
Of course I wouldn't be ignoring conversion, or sales, accurately I don't ignore ANYTHING....damn - that's lame that you would say that. In my job though, I have a marketing department to feed me that info so I can focus more on getting the placement the marketing deparment wants. Hey, it's not the same out there for everyone else I know - and I thought I was sharing some decent info about how you could use spider info, that abandoning it is IN MY OPINION - scary and well, I don't believe you should...but hey I don't care what other people do in SEO, just what I do and whether someone wants to take what I've learned to their project is up to them, and I find it disturbing that just because I mention IP cloaking I'm suddenly in "the other camp"...ya, the one you're NOT in.
Sorry, I was very disturbed by the comments made - the thread had asked the question I was answering the only way I can - with MY ANSWER TO THE QUESTION...lame.
Let the lightning strike...hope someone besides site moderators/admins get to read this before it's nuked and I have to leave the site permanently - it's not worth getting a new account if you nuke me
Posted 16 December 2004 - 12:18 PM
A flame, as I understand it, is a personal attack on someone, and I'm just not seeing that in regards to anything anyone posted that disagreed with your post.
Please point them out to me and I will delete asap.
Posted 16 December 2004 - 12:34 PM
Well Jill, the whole "we" angle pretty much sets an us vs you attitude - that the poster was a moderator, so now I feel like it's "High Rankings" vs "Me"...like this...
The thread asked me a question that I answered. "Why is frequent spidering important?". In fact - the replies to my post are OFF TOPIC stating reasons you wouldn't want to know about spiders, fair enough - but off topic. I stated reason you might want to know, and there are more reasons I have that I didn't state, that I guess will be my own since I don't feel like sharing on here anymore espeically if a moderator can come in, flame a user post with off topic information...if you don't agree, that's fair too - I already feel like I'm on an island here, or at least "the other camp" that none of you are in...oh well, it sure is nice and peacful at my camp
Thanks for at least looking into Jill, appreciated. OWG, you still ROCK!
Posted 16 December 2004 - 02:39 PM
However, I don't feel that it is in any way flaming you personally. Post about cloaking here, and you will definitely have the moderators jumping all over you to make sure that the people reading here know "our" (the forum's) official stance on it.
Again, it's nothing personal though. Cloak, don't cloak, whatever. Just know that this forum is against it, and why we are.
(Heck I just did a whole session on this very topic here at SES, and the other panel members do understand that my stance is nothing personal.)
Posted 16 December 2004 - 05:19 PM
Also, great post, Torka!
All good reasons IMO...consistent with helping interested visitors quickly find the site and easily get around.
To expand on those thoughts a bit...
From an overall business standpoint, there is a lot you can do with that information. For example, you can identify opportunities to improve navigation - find pages that are difficult to find and change the navigational structure to make it easier for spiders and visitors to find those pages.
Knowing when the spider comes around can also make it faster/easier to test different description tags (which can make your Google snippet more "user friendly") quickly see the effects of different wording, etc.
(Oh yeah, OWG I think you rock, too. )
Posted 16 December 2004 - 07:49 PM
SEO is search engine optimization. Methods and rules in the creation of online content that suits itself to ranking within search engines, within a set of phrases. Writing things that make people feel like they need something, selling stuff to people that's marketing. They are two wildly different things no matter how and where they intersect. I don't know how to sell people stuff, what to tell them to click on things to buy more stuff, but, I do know how to get you page one on Google, Yahoo, and MSN (et al). That's why I make the distinction between them that way. Fair enough? And you have your reasons for making your distinctions too, fair enough. I'm sorry that I made the mistake of saying that a very risky method of manipulating search rankings could be described as "Advanced SEO", my mistake - it is not. Obtaining something honestly with good content that is accesbile and dishonest manipulation of organic results are two different things. Right on!
I'd like you all to know that you've singled someone out in a way that's completely self elevating - it's terrible and simply how I feel. Like your opinion on this whole thing, so is my feeling - unchanged. And, really, as much as I understand why and that you had to defend your opinion, knowing it's your resume....oh just wait...imagine if it were mine, and maybe you'd see why I didn't like what Torka said, and feel as though it's a flame because it attacks me professionally. Come on dude, you knew what you were saying and how you were saying it when you wrote all that - and all I get is dogpiled by site admins and staff...a hand clap and bunch of "I don't see what you're talking about" stuff...damn it, I didn't have the time to go on and on and on, which SEOs tends to have to do to explain even a fraction of the whole picture. You could have given me a break and just let it go...but oh no...
Thank you though to Bobette for at least for trying to salvage something from my post...even if you don't see the flames guys - I really do feel 'em. But, who am I to argue with the almighty Torka....*kneels before his master*....
...throw me that bottle there OWG, I need a bite....and hey, thanks for stopping by dude, you rock!
Posted 17 December 2004 - 12:07 AM
You were not flamed, however. Nobody called you stupid. Nobody called you a dumbass. You read that into it as is your perogative.
Bottom line as far as I'm concerned in regards to this thread...there are numerous reasons to look at when and why spiders come to your site.
The original question was asking why do people care if/when they get FREQUENT spidering, not what spiders are coming to their site.
Frequent spidering does not equal high rankings.
And if you're not deceiving search engines you don't need to care what the latest spider/user agents or whatever, are doing at your site. You may certainly want to watch your logs to make sure your site is indeed getting spidered, and you may even want to see how frequent this is. It won't help your rankings, however, if they come more often.
Oh, and cloaking is bad.
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users