42 replies to this topic

[lyn]

The only reason I can think of to get excited about frequent spider visits is... if you are already planning to try some changes and you want to see their effect on rankings , it would be helpful to know that the spiders will be coming by on a regular basis. Just so you don't have to wait too long to see the difference.

Still kind of chicken & egg, though...

I've never actually paid any attention to spider visits. Does a visit from a spider always result in an update to the SERPs?

L.

[labatsman]

Have to admit I would really be worried if the spiders completely ignored me for a month

I would also be worried if the spiders didn't visit.

[OldWelshGuy]

Have to admit I would really be worried if the spiders completely ignored me for a month

You wouldn't be saying that if you had a forum. the little buggers eat datatransfer like ther is no tomorrow, running around eating everything in sight

[qwerty]

True. When I started my tiny little forum, my bandwidth more than quintupled. Hardly anyone posts in there, and a decent number of people read, but it's mostly spiders crawling around non-stop.

[jerry]

The new Beta MSN search engine has a sliding control that lets you select how important freshness is to your search.

Freshness might be important to me if I were to search for a baseball player's batting average. I wouldn't care about a site that had not been updated in over six months.

Whether this "freshness" feature will catch on, we'll have to wait and see, but it looks like the new MSN search engine thinks maybe so.

Jerry Minchey

[Connie]

You wouldn't be saying that if you had a forum. the little buggers eat datatransfer like ther is no tomorrow, running around eating everything in sight 

Your probably right. I'm not going to start one to talk to myself. Tried that years ago in another area and it was a flop. Guess I will have to be satisfied as long as the little spiders are interested at all.

[rwt001]

Now, that may not be indicative for a real SERP query, but it is one thing that will lead people to believe that more recent content ranks better, and to see more recent content, the spider has to drop in regularly.
Putting that together, people may erroneously conclude that they need the spider to visit more often.
Which doesn't work logically at all if pages aren't actually changing.
But who ever said folks were logical?

Sure folks are not logical, but wouldn't we expect spiders or the SE' that send them out to be logical?

There are lots of stale pages on the web. Lots of dynamic content too.

Surely the spiders know the last date any given web page was revised. If that is the case, if a page remained static (no changes) after a few spider visits wouldn't it make sense that the spider would schedule itself to visit less frequently?

I would think that scaling back the frequency of spidering stale pages would make sense to conserve "spider resources".

If this is true, a page that got updated say, less than once a year, could expect to wait quite a while for the next spider visit. Does this ring true for any of you SEM webmaster techies?

[Haystack]

I don't have a techie answer to this, but I figure search engines are going to want to revisit stale pages on a regular basis to make sure they still exist. It certainly seems like this is done pretty regularly these days because I rarely run across dead pages from SERPs.

[Connie]

I think Ed may have hit the nail on the head. If you have a fairly static site like mine spiders still need to crawl to make sure pages still exist. I see some activity every day from spiders.

When I update a few pages or add a new page that usually causes an increase in spider activity for a few days.

The adding of new content is where a forum can become a valuable asset to a site. Especially if the forum is active.

Bob would you give the key words again for finding your forum or PM me the url. I didn't look at it when you mentioned it previously. I would like to see your forum.

[sonnyyu]

Now, I don't want to read the next post saying, "Okay, then how much do I have to change my content to make the spider think it's fresh?!!!" -Jill

I happen to have answer if 61.8% or more content changed spider will think it's fresh.


Search 0.618 at google you will find out why.

[Marctwo]

What if you change 31% one day and 31% the next.

[qwerty]

Bob would you give the key words again for finding your forum or PM me the url. I didn't look at it when you mentioned it previously. I would like to see your forum.

It's not an SEO forum, so I doubt many people here would find it very interesting, but it's at www.annasinkovska.com/fashion-forum/

[torka]

Search 0.618 at google you will find out why.

Hmmm. I'd say, though, it's equally likely the answer is 42...

--Torka

[lyn]

Search 0.618 at google you will find out why.

Phi on your cryptic algorithms!!
Phi, I say!!!

[flyer]

> You wouldn't be saying that if you had a forum. the little buggers eat datatransfer like ther is no tomorrow, running around eating everything in sight

Well, it's easy enough to keep them out of the forum pages if you want to.


> One thing I have noted in checking the search engines for completeness-of-listing is that the pages which change most often (specifically blog pages), and the most recently created stuff generally, tends to appear at the top of the listing for a site:mydomain.com check.

Does the order the pages display in a site: listing mean anything? Is it officially supposed to be the most recent first, or is that just something you're guessing? That doesn't seem to be the case with my sites.