Is Frequent Spidering Important? Options

[Scottie]

I'm always interested in what people think more frequent spider visits actually do to help you.

The spider visits, sucks up some bandwidth, finds all the pages the same as they were the last time she visited, and goes on to the next site. How do you think this affects your website?

If you are constantly changing your pages to add fresh content, the spiders will naturally visit more often because they don't want outdated pages in their index. This happens as well if you are constantly tweaking keyword density or changing the page structure or other things that people think they can do to affect their rankings. But all that "tweaking" can lead to incorrect cause-and-effect conclusions.

In the meantime, who's monitoring the sales? Who is tweaking the copy to sell more if all you are ever doing is worrying about rankings? Does anyone know what happens when a searcher actually reaches their site?

I suspect most people don't know why they want spiders visiting more often; they just think it's a good sign. Or possibly they think the spiders give them a rankings boost every time they visit... I don't know. Why else are people so concerned about spider visits?

If your page is in the index already and you haven't changed it, it doesn't matter when then next spider visit is! Or am I missing something? Is there some other value to getting your site crawled that has nothing to do with new content?

[qwerty]

I think the only sensible reason to hope for frequent spidering of a site that isn't updated often is the desire for the spiders to catch new content as soon as possible after it's created.

I'm currently running some tests on my site, and because the site isn't spidered often (because I don't update it on a regular basis), I have to wait for the search engines to get around to finding my new content in order to get any results for my tests. So I have to wait a couple of weeks. Big deal.

I have read a number of articles that announce that search engines "like" fresh content, but I don't think any of them have been clear about how search engines treat this content that they like so much. Yes, they spider it more often, but I don't think any of those articles come out and claim that this means better rankings.

[Jill]

Can't say that I've ever checked a spider log in my life.

If something is out of my control such as that, then why would I waste my time checking it and worrying about it?

[Scottie]

We get so many posts where people seem to think that getting spidered frequently is a big advantage... it just isn't unless you are changing your site regularly.

It's certainly not a goal in and of itself! It's just a means to get changes and new content into the indexes.

[Jill]

The problem, as I see it, is that people read info from other sites and then get it mixed up in their heads.

They hear that the spiders will visit more popular sites, more often, and think that somehow the spidering MAKES their site more popular.

The most common thing we deal with is a mix up of cause and effect, in this biz. Just too many people who only half understand stuff posting all over the net their misunderstood info.

Anyone who wants to do SEO needs to figure stuff out for themselves, do their own tests, be careful of mixing up cause and effect, and use common sense to decide what the results they see mean.

[qwerty]

Anyone who wants to do SEO needs to figure stuff out for themselves, do their own tests...

...or just read here (IMG:http://www.highrankings.com/forum/style_emoticons/default/smile.gif)

[Jill]

Well, yeah, read here...but I would recommend that nobody believe anything (even from me) without testing it for themselves.

[maleman]

Well, yeah, read here...but I would recommend that nobody believe anything (even from me) without testing it for themselves.

The proof's in the pudding!

[leadegroot]

One thing I have noted in checking the search engines for completeness-of-listing is that the pages which change most often (specifically blog pages), and the most recently created stuff generally, tends to appear at the top of the listing for a site:mydomain.com check.
Now, that may not be indicative for a real SERP query, but it is one thing that will lead people to believe that more recent content ranks better, and to see more recent content, the spider has to drop in regularly.
Putting that together, people may erroneously conclude that they need the spider to visit more often.
Which doesn't work logically at all if pages aren't actually changing.
But who ever said folks were logical?
(IMG:http://www.highrankings.com/forum/style_emoticons/default/wink.gif)

Lea
<edit>because my fingers cannot spell today (IMG:http://www.highrankings.com/forum/style_emoticons/default/sad.gif) </edit>

This post has been edited by leadegroot: Dec 11 2004, 09:18 PM

[Scottie]

Exactly! Fresh content is great. Spiders will visit more often if you have it, it may or may not rank better than old content, but spiders visiting the same exact pages several times a week doesn't do a thing.

[Jill]

Now, I don't want to read the next post saying, "Okay, then how much do I have to change my content to make the spider think it's fresh?!!!"

It's not about tricking the engines into thinking your content is fresh. If you really want the spiders to visit often, for whatever reason, then you actually need REAL fresh content.

But again, you don't need fresh content to rank well, and you don't need the spiders to visit often to rank well. I have some very competitive keyword phrases that have ranked well for years on some client's sites that have never been updated since I did them like 5 or 6 years ago.

Fresh doesn't always equal better, and the engines know this.

[Connie]

Scottie,

It boost your ego when the spiders are crawling. You live in the hope that in this crawl you will get better rankings (IMG:http://www.highrankings.com/forum/style_emoticons/default/lol.gif) . People tend to forget about cached pages (or data). You are right it's no big deal as long as the spiders hit your site regularly. Have to admit I would really be worried if the spiders completely ignored me for a month even though my site is pretty static.

[Marctwo]

I get googlebot through my site every day without fail. This means that I can make subtle changes to the SEO and see results in google within a couple of days.

This is obviously great for someone like me who has a lot to learn. (IMG:http://www.highrankings.com/forum/style_emoticons/default/smile.gif)

[anthonyparsons.c...]

Yep....just means nothing at all to me.

[leadegroot]

Gerry McGovern has published an apropos article this week - You'd almost think he was subscribed here (IMG:http://www.highrankings.com/forum/style_emoticons/default/wink.gif)
http://www.gerrymcgovern.com/nt/2004/nt_20..._publishing.htm