Jump to content

  • Log in with Facebook Log in with Twitter Log In with Google      Sign In   
  • Create Account

Subscribe to HRA Now!

 



Are you a Google Analytics enthusiast?

Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE! 

 



 

 www.CustomReportSharing.com 

From the folks who brought you High Rankings!


Sponsored Content

 

 
 

Photo
- - - - -

Is Frequent Spidering Important?


  • This topic is locked This topic is locked
42 replies to this topic

#31 TechnoCrow

TechnoCrow

    HR 2

  • Active Members
  • PipPip
  • 26 posts
  • Location:CANADA

Posted 15 December 2004 - 05:02 PM

The reason I watch *very* carefully what the spiders are doing on my site is to ensure there are no navigation problems, frequency of visits to determine best times to change content for quick crawls, knowing what site to put a link on if I want a crawler to follow it...man, there's a ton of reasons that I have - I even parse log files for complete lists of what/who crawled and when. I find it surprising that a lot of people don't care...fair enough, I like to know as much as I can about my opponent (the SE).

In advanced SEO, this info is indispensable, how would one do IP cloak if you didn't know ALL the robot IPs, the big engines don't use just one crawler - and Google and MSN are insane these days with the shear amount of bots and time they spend crawling around. Black hat maybe, but still - impossible without this info. Not to mention how does one construct a robots.txt file without this info - rely on others??...please, engines play with IP and Agent settings all the time to detect spam/cloak/whatever...I like to get this info first hand.

As a bonus, I have almost mapped complete segments of bots, their agent names - for Yahoo, MSN, and Google and what they tend to do (text crawls, image crawls, status 200 check - usually the big engines have purpose for their bots too).

Anyway, just wanted to share that I'm kinda astonished that many of you don't even care...wow! For me, it's easy - all I do is SEO, that is my job title so I don't need to worry about selling users, etc - just getting placement smile.gif...so I've got the time to look into data like this...

#32 Jill

Jill

    High Rankings Advisor

  • Admin
  • 32,312 posts

Posted 15 December 2004 - 05:23 PM

QUOTE
In advanced SEO, this info is indispensable, how would one do IP cloak if you didn't know ALL the robot IPs, the big engines don't use just one crawler - and Google and MSN are insane these days with the shear amount of bots and time they spend crawling around. Black hat maybe, but still - impossible without this info. Not to mention how does one construct a robots.txt file without this info - rely on others??...please, engines play with IP and Agent settings all the time to detect spam/cloak/whatever...I like to get this info first hand.


Black hat maybe? Black hat definitely! To be clear, we do not recommend cloaking under any circumstances, and therefore it is unnecessary for us to have to care about the spiders user agents, etc. for that purpose.

Cloaking is always a bad idea.

(If anyone wants to discuss cloaking further, please start a new thread as that's not the topic of this one.)

#33 torka

torka

    Vintage Babe

  • Moderator
  • 4,392 posts
  • Location:Triangle area, NC, USA, Earth (usually)

Posted 16 December 2004 - 09:31 AM

QUOTE(TechnoCrow @ Dec 15 2004, 05:02 PM)
Anyway, just wanted to share that I'm kinda astonished that many of you don't even care...wow! For me, it's easy - all I do is SEO, that is my job title so I don't need to worry about selling users, etc - just getting placement smile.gif...so I've got the time to look into data like this...

And I guess that's where the misunderstanding falls. smile.gif

See, most of the people here seem to fall into one of two camps.

Either they're business people trying to build their businesses and learn enough about SEO to optimize their own business's site(s), or at least to know enough to avoid being scammed by a hired SEO...

Or they're people who provide services to these businesses.

Business people -- at least the smart ones wink.gif -- recognize that positioning and the frequency of spider visits are nice bragging rights, but they don't do diddly to put food on the table or pay the rent. What counts in business, no matter whether in it for the long haul or you're just trying to make a quick buck, is sales.

For us, "advanced SEO" is SEO that actually benefits the bottom line. smile.gif And there's a lot more to it than just getting top rankings "by any means necessary."

Of course, we care whether the spiders visit and what position our sites get! We just don't care to the degree that you seem to.

We recognize that things that prevent spiders from visiting may also prove to be barriers to humans trying to use the site. So, we try to make our site navigation to be as easy, intuitive and barrier-free as we possibly can. And we try to make the site code as clean as possible, and the page downloads as quick as possible, and the pages as usable and accessible as possible. These things are good for humans and SE spiders, even if they don't fall under the category of "positioning."

And, naturally, we want to get as much traffic as we can, and we know that higher ranking sites tend to get more traffic, so we want to rank as well as we can. But we also know that all the traffic in the world won't do any good if none of it converts.

What doth it profit a man to have top ten rankings and hourly spider crawls if every human who visiteth his site sayeth, yea verily yuk.gif, and hightaileth it for the competition within five seconds of arrival? smile.gif

So we check for spider visits, sure. But more importantly, we're looking at the paths that human visitors take through our sites. We're checking at what points they enter the site, and at what points they leave. We're tracking conversion ratios and click through rates. Instead of updating our robots.txt and IP-address cloaking scripts, we're tweaking site architecture, page layouts and content to see if we can get a higher conversion ratio.

The name of the game isn't traffic or positioning, it's conversions.

Which is, I think, why the question got raised in the first place. Why do people care so much about how often the spiders visit? Frequency of spider visits has nothing to do with how highly a page will rank, and nothing at all to do with how well a page will convert.

We may call ourselves SEOs or SEMs or Internet Marketers or whatever. But in the final analysis, very few of us here see our business as simply "positioning" and we apparently take a broader definition of "SEO" than what you do. We're here to help ourselves or our clients make better websites (which is not the same thing as "websites that are better at tricking the SEs") that lead to higher sales and more profits.

I think more business people are discovering that while having a website is cool, having a website that actually makes them money is even cooler. And they're becoming more sophisticated about what it takes to get there. It's going to take time, but IMHO the days are numbered when SEOs could make a killing with simply positioning and take no responsibility for whether the site converts any of its traffic to sales. At least with the smart business people... wink.gif

My penny.gif YMMV, of course. thumbup1.gif

--Torka mf_prop.gif

#34 Randy

Randy

    Convert Me!

  • Moderator
  • 17,540 posts

Posted 16 December 2004 - 10:02 AM

appl.gif

#35 qwerty

qwerty

    HR 10

  • Moderator
  • 8,287 posts
  • Location:Somerville, MA

Posted 16 December 2004 - 10:34 AM

Well stated, Torka.

I always disagree with statements about SEOs not being involved in marketing, and their only responsibility being rankings, but that leads to discussions of what we ought to call ourselves, and that is offtopic.gif

#36 TechnoCrow

TechnoCrow

    HR 2

  • Active Members
  • PipPip
  • 26 posts
  • Location:CANADA

Posted 16 December 2004 - 11:53 AM

I guess I'll be a little more careful sharing my OPINION, my gawd man, you flamed me for sharing?!?...

Just becaue you don't agree with me, I guess - doesn't mean my definition of SEO is any less valid than yours, maybe not even different - I might even go as far to bring more flames to my post by saying my definition is more complete...

Of course I wouldn't be ignoring conversion, or sales, accurately I don't ignore ANYTHING....damn - that's lame that you would say that. In my job though, I have a marketing department to feed me that info so I can focus more on getting the placement the marketing deparment wants. Hey, it's not the same out there for everyone else I know - and I thought I was sharing some decent info about how you could use spider info, that abandoning it is IN MY OPINION - scary and well, I don't believe you should...but hey I don't care what other people do in SEO, just what I do and whether someone wants to take what I've learned to their project is up to them, and I find it disturbing that just because I mention IP cloaking I'm suddenly in "the other camp"...ya, the one you're NOT in.

Sorry, I was very disturbed by the comments made - the thread had asked the question I was answering the only way I can - with MY ANSWER TO THE QUESTION...lame.

Let the lightning strike...hope someone besides site moderators/admins get to read this before it's nuked and I have to leave the site permanently - it's not worth getting a new account if you nuke me smile.gif

#37 Jill

Jill

    High Rankings Advisor

  • Admin
  • 32,312 posts

Posted 16 December 2004 - 12:18 PM

Technocrow, where do you see a flame, and why would we nuke your post for your opinion?

A flame, as I understand it, is a personal attack on someone, and I'm just not seeing that in regards to anything anyone posted that disagreed with your post.

Please point them out to me and I will delete asap.

#38 TechnoCrow

TechnoCrow

    HR 2

  • Active Members
  • PipPip
  • 26 posts
  • Location:CANADA

Posted 16 December 2004 - 12:34 PM

QUOTE(Jill @ Dec 16 2004, 01:18 PM)
Technocrow, where do you see a flame?

Well Jill, the whole "we" angle pretty much sets an us vs you attitude - that the poster was a moderator, so now I feel like it's "High Rankings" vs "Me"...like this...

QUOTE
For us, "advanced SEO" is SEO that actually benefits the bottom line.  And there's a lot more to it than just getting top rankings "by any means necessary."


QUOTE
Of course, we care whether the spiders visit and what position our sites get! We just don't care to the degree that you seem to.


QUOTE
We may call ourselves SEOs or SEMs or Internet Marketers or whatever. But in the final analysis, very few of us here see our business as simply "positioning" and we apparently take a broader definition of "SEO" than what you do. We're here to help ourselves or our clients make better websites (which is not the same thing as "websites that are better at tricking the SEs") that lead to higher sales and more profits.


The thread asked me a question that I answered. "Why is frequent spidering important?". In fact - the replies to my post are OFF TOPIC stating reasons you wouldn't want to know about spiders, fair enough - but off topic. I stated reason you might want to know, and there are more reasons I have that I didn't state, that I guess will be my own since I don't feel like sharing on here anymore espeically if a moderator can come in, flame a user post with off topic information...if you don't agree, that's fair too - I already feel like I'm on an island here, or at least "the other camp" that none of you are in...oh well, it sure is nice and peacful at my camp smile.gif

Thanks for at least looking into Jill, appreciated. OWG, you still ROCK!

#39 Jill

Jill

    High Rankings Advisor

  • Admin
  • 32,312 posts

Posted 16 December 2004 - 02:39 PM

I understand you feeling the us vs. them thing, and that's probably correct. We (all the moderators here) have the same feeling on the subject of cloaking and spam, which is why they are moderators to begin with.

However, I don't feel that it is in any way flaming you personally. Post about cloaking here, and you will definitely have the moderators jumping all over you to make sure that the people reading here know "our" (the forum's) official stance on it.

Again, it's nothing personal though. Cloak, don't cloak, whatever. Just know that this forum is against it, and why we are.

(Heck I just did a whole session on this very topic here at SES, and the other panel members do understand that my stance is nothing personal.)

Jill

#40 BobetteKyle

BobetteKyle

    HR 6

  • Active Members
  • PipPipPipPipPipPip
  • 889 posts
  • Location:Near St. Louis, Missouri

Posted 16 December 2004 - 05:19 PM

What Jill said. I do not see any flames here either ... politely disagreeing and/or "taking sides" isn't the same as flaming.

Also, great post, Torka!

QUOTE(TechnoCrow)
The reason I watch *very* carefully what the spiders are doing on my site is to ensure there are no navigation problems, frequency of visits to determine best times to change content for quick crawls, knowing what site to put a link on if I want a crawler to follow it.

All good reasons IMO...consistent with helping interested visitors quickly find the site and easily get around. thumbup1.gif

To expand on those thoughts a bit...

From an overall business standpoint, there is a lot you can do with that information. For example, you can identify opportunities to improve navigation - find pages that are difficult to find and change the navigational structure to make it easier for spiders and visitors to find those pages.

Knowing when the spider comes around can also make it faster/easier to test different description tags (which can make your Google snippet more "user friendly") quickly see the effects of different wording, etc.

(Oh yeah, OWG I think you rock, too. thumbup1.gif)

#41 OldWelshGuy

OldWelshGuy

    Work is Fun

  • Moderator
  • 4,713 posts
  • Location:Neath, South Wales, UK

Posted 16 December 2004 - 05:26 PM

I will be rocking this Christmas with any luck

#42 TechnoCrow

TechnoCrow

    HR 2

  • Active Members
  • PipPip
  • 26 posts
  • Location:CANADA

Posted 16 December 2004 - 07:49 PM

This is so ridiculous. When you say "we" and "you" (and all gang up together) that means High Rankings is "we" and "you" is me...the opinion you expressed Torka, as well put, correct, and eloquent as it maybe - still describes me as someone who is so far from "your" (High Rankings) definition of SEO that well, maybe I'm just a dumbass to people who don't know any better. Man you elevated yourself off of my post so nicely I can hardly believe it. Also, what's with the crack that I seem to work for stupid people since my only focus is placement and only stupid people care about that. I know you're mixing your definition of marketing and optimization - and oh yeah, why don't we make that clear too - as clear as mud.

SEO is search engine optimization. Methods and rules in the creation of online content that suits itself to ranking within search engines, within a set of phrases. Writing things that make people feel like they need something, selling stuff to people that's marketing. They are two wildly different things no matter how and where they intersect. I don't know how to sell people stuff, what to tell them to click on things to buy more stuff, but, I do know how to get you page one on Google, Yahoo, and MSN (et al). That's why I make the distinction between them that way. Fair enough? And you have your reasons for making your distinctions too, fair enough. I'm sorry that I made the mistake of saying that a very risky method of manipulating search rankings could be described as "Advanced SEO", my mistake - it is not. Obtaining something honestly with good content that is accesbile and dishonest manipulation of organic results are two different things. Right on!

I'd like you all to know that you've singled someone out in a way that's completely self elevating - it's terrible and simply how I feel. Like your opinion on this whole thing, so is my feeling - unchanged. And, really, as much as I understand why and that you had to defend your opinion, knowing it's your resume....oh just wait...imagine if it were mine, and maybe you'd see why I didn't like what Torka said, and feel as though it's a flame because it attacks me professionally. Come on dude, you knew what you were saying and how you were saying it when you wrote all that - and all I get is dogpiled by site admins and staff...a hand clap and bunch of "I don't see what you're talking about" stuff...damn it, I didn't have the time to go on and on and on, which SEOs tends to have to do to explain even a fraction of the whole picture. You could have given me a break and just let it go...but oh no...

Thank you though to Bobette for at least for trying to salvage something from my post...even if you don't see the flames guys - I really do feel 'em. But, who am I to argue with the almighty Torka....*kneels before his master*.... notworthy.gif

...throw me that bottle there OWG, I need a bite....and hey, thanks for stopping by dude, you rock!

#43 Jill

Jill

    High Rankings Advisor

  • Admin
  • 32,312 posts

Posted 17 December 2004 - 12:07 AM

Feeling like you're being flamed and being flamed are two different things.

You were not flamed, however. Nobody called you stupid. Nobody called you a dumbass. You read that into it as is your perogative.

Bottom line as far as I'm concerned in regards to this thread...there are numerous reasons to look at when and why spiders come to your site.

The original question was asking why do people care if/when they get FREQUENT spidering, not what spiders are coming to their site.

Frequent spidering does not equal high rankings.

And if you're not deceiving search engines you don't need to care what the latest spider/user agents or whatever, are doing at your site. You may certainly want to watch your logs to make sure your site is indeed getting spidered, and you may even want to see how frequent this is. It won't help your rankings, however, if they come more often.

Oh, and cloaking is bad.

Thread closed.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users