14 replies to this topic

[ajitky]

Hello All,

I have foll. 2 Qs.

SE (esp. Google & msn-inktomi) reindexing approach for a site & all its pages already indexed and present in its database...so when SE reindex this site again, does it....

1. Directly refers the individual page urls it already has in its index and checks for server response (to find if page still exists) or new changes on particular pg.? or

2. Use the normal approach i:e visits INDEX / HOME pg. first and starts crawling the pgs. again? so as to remove pgs. not found and add any new pgs. it finds to its index.


My second Qs. is related to first one...
I have a site indexed with major SE, now I want to RENAME many of the pages on my site which were named quite generally and are not pulling enough traffic. I want to name them after the most important K/W phrase used on page, just to get that extra little points in ranking. SO HOW DO I DO IT WITHOUT LOOSING ANY/MUCH TRAFFIC?

Thanks in advance.

Ajit

[OldWelshGuy]

Hi ajitky,
When re naming pages you have to think about more than just the Se's, you have to think that maybe people have visited you site, and added pages to their favourites/bookmarks, or even linked to them from their own pages.

With this in mind I always leave the original page name up, put a re-direct with a delay on it, and a message for your visitor telling them your page has moved, also put a clickable link on there to the new page as well for the bots to follow.

You can then alter your navigation structure within your site to the new pages, leaving the 'old' pages orphaned and acting as a safeguard against links bookmarks etc, and preventing the dreaded 404 error page not found page of death. If anyone was to get this, they will assume your company no longer trades and will remove you from their favs/bookmarks/link page.

By doing this I have found that there is no need to worry about 'how' your site is indexed, as no matter how it is, it will still be indexable in both the old, and new page setup.

[ajitky]

Hello OldWelshGuy,

Thanks for ur reply...when u say "leave the original page...." does it mean I should remove the content on that original page and move it to new page, so that SE won't find it as a duplicate page thereby thinking I am spaming.

[Peter]

Ajitky,

Perhaps there is some confusion on your part between indexing and ranking. These are 2 different processes.

Ranking happens the moment a person does a search.

Indexing is done once after a page has been visited by a spider.

The indexing process just creates a list of factors that are important to the search engine. This list of factors is then used for the ranking process.

A search engine does not check if a site is still online every time it presents its results. That is impossible. It just recrawls a site every now and then to check for changes and if pages are still there.


The second question:

You are talking about renaming pages. I assume you mean changing the file name. The thing that is important here is linking. You have to change all the links that point at those pages. For links in your own web site that is no problem,.. but for links that come from other web sites it can be a problem. But If I understand you correctly, those pages don't bring in much traffic so they probably have few links pointing at them from other sites.

To prevent loosing traffic from back links, you can do the following.

Rename all pages, but put them in a new directory (with the keyword in the name).
Make all links inside your web site point at the new pages. (make sure these pages are no longer reachable from within your own pages.)
Add the following meta tag to the old pages:
<meta name="robots" content="noindex, nofollow">
Email all web masters that link to the old files and ask them to change the links to link to the new file.
Once all new pages have been indexed, and all backlinks are pointing at your new pages, you can delete the old pages.

Regards,

Peter

[ajitky]

Thanks for that advice.

[isoseo]

Hello,

This question may not be related to previous discussion, but it is about re-indexing, so it might fit in here.

We had a major rehaul of our existing (commercial) site: same content with new look and new page/directory names. (I was not prudent enough to do a re-direct on the old pages or in .htaccess, but this I think is not really important to the problem below.)

Google got the overwhelming majority of new pages indexed in 1-20 days. Altavista indexed most of them some 2 months after uploading. All the other "major" SE-s, however, do not seem to care much about the change, even though the new pages are 4-months old now!

This I find funny because of the following:

There is nothing that would make SE-s frown upon or ignore our site. No dubious optimization techniques applied, not on the pages, not in the linking strategy, nowhere. The site has tons of unique content (e.g detailed manuals and instructions for our products). The rehaul was - I am sure - for the better, making the site cleaner, more user-friendly, and more SE-friendly.

All this is reflected in Google, where the site has vastly improved rankings for a lot of keywords and phrases; it has a PR 6; it is re-crawled almost on a weekly basis; and the traffic referred is 3-4 times higher than before.

All other well known SE-s (or database providers) have already crawled the new site. We have about 350 indexable content files (htm, pdf, ppt). Wisenutbot/Looksmart, Slurp and Teoma has requested such pages 900+ times, FAST-WebCrawler 300 times.

Now, Teoma, Alltheweb, Lycos, HotBot and MSN contain 1, 2, 4, 25 and 27 of our pages, respectively. Actually, Teoma (the next Google, haha) still lists 40 of the old pages!

So, why do they index pages if they do not include them in their database? Why do their bots keep requesting pages deleted 4 months ago (because they do, esp. Slurp)? Why does Teoma list old pages after getting more than 300 Error404-s for them in this 4 months?

And finally, how can I trust the results of these SE-s for ANY search if new pages will not make it into their databases for such a long time and old ones sometimes just do not seem to drop out?

Thanks for any helpful thoughts.

[isoseo]

So, why do they index pages

Sorry, I meant "crawl".

[isoseo]

Trying to make it shorter:

What is the reason for bots to constantly crawl our site, SEs still not updating our data in their indices for months?

[Peter]

What exactly do you mean with

not updating our data in their indices for months

They do update the index, but if nothing changed,.. your position won't change.
Google is by far the fastest. The others can be faster too but that would hurt their own business because they have a service called: "paid inclusion". If they would crawl every site every month or more often, then their paid inclusion service wouldn't make that much money anymore.

Updating the index does not automatically mean an increase in rank. Some sites are visited more often because of reasons only search engines know exactly. I think it mostly depends on how often something on your web site changes.

Regards,

Peter

[isoseo]

if nothing changed,.. your position won't change

Actually, our complete site had been re-designed in July. Since then the new pages have been extensively crawled by all major SE-bots, having downloaded them by the hundred! However, many SEs still contain mainly our old deleted pages or nothing (Google and AV are the exceptions).

I also suspect that there is a conflict of interest between paid-inclusion schemes and keeping the index up-to-date, but should this really mean that they crawl our site in August, and do not include those pages in the index even in November?

Why do they bother to crawl then at all? Why do some of them keep our non-existing pages? It just makes no sense.

[Peter]

I can think of only one reason,.. paid inclusion generates money. Perhaps they do crawl sites more often because they enter through back links, but appearently that doesn't mean they index the pages.

If anybody knows of other reasons,.. please post them.

By the way,. I mentioned that when your site doesn't change, your position won't change,.. that is not 100% correct. You can still gain or loose positions due to shifts in positions of other pages,.. but that generally doesn't make enormous differences.

Peter

[Jill]

I also suspect that there is a conflict of interest between paid-inclusion schemes and keeping the index up-to-date, but should this really mean that they crawl our site in August, and do not include those pages in the index even in November?

Your suspicions are correct. They hope you will pay them to get indexed if you're in a hurry and they drag their feet for awhile.

Jokes on them, however. Now they have databases that are way out of date. All in the hopes that they'll make more money. 'Cept who cares about being included in an out of date search engine?

It's a vicious circle that they've created out of greed.

Jill

[SearchRank]

What is the reason for bots to constantly crawl our site, SEs still not updating our data in their indices for months?

Google and All The Web (FAST) are extremely quick to index pages and update indices. Inktomi does a pretty good job but is slower unless you are using PFI.

Alta Vista is slow and sometimes, submitting either the home page or the pages you want crawled has prompted them to update information in their index.

Teoma is usually pretty slow to update their index with changes in sites already listed there. Its hit and miss. Sometimes they pick up changes in 2-3 months; other times they don't. In that case, you may want to submit the URLs that need to be updated through their PFI program.

WiseNut's index is so out of date it is not even funny. To make a long story short here: Too bad Looksmart ever bought this engine and screwed it up! There's no way to submit to them either for free or PFI so your stuck waiting on them. Because they only feed Looksmart supplemental results as well as being a stand alone site, they are not all that important at this point.

Bottom line is I believe there is the process of continually crawling the Internet but then there is a separate process of the SE actually updating their indices.

[isoseo]

All The Web (FAST) are extremely quick to index [...] Alta Vista is slow

I'd defend AltaVista to some degree.

Well, it was not extremely quick with our site, either, but in the end it has found and indexed most of it - and it keeps coming back. (And sends some traffic, if not that much.). Their site looks good now, and they seem to be very much alive. Too bad they blew it back then.

On the other hand, FAST was fast only in purging our old pages, so we are now non-existent in the their world (as opposed to 635 pages in Google...).

As to Jill's note: it really is sad to see such short-sightedness (now regardless of their handling our site). This whole thing would be more interesting with more than one serious player.

[domokun]

You can then alter your navigation structure within your site to the new pages, leaving the 'old' pages orphaned and acting as a safeguard against links bookmarks etc, and preventing the dreaded 404 error page not found page of death.

wow. good advice!
i never even thought of doing that before - its so simple!!
i used to set up directories and stick to them

mydomain/myfolder/mypost.html

but frequently (especially when i was in a bad mood!) id start from scratch meaning that anyone who had linked to me would have seen a 404



ive been so stoopid! thanks owg