60 replies to this topic

[SearchRank]

I have seen that Inktomi WILL crawl links from a site map and include the pages it finds.

[BrianR]

Thanks, David - that's a relief!

BrianR

[MakeMeTop]

>If I use PFI to get a sitemap page listed in Inktomi, are you saying that it will not crawl the links on that page ..

Hate to rain on anyone's parade here - but to get this absolutely clear:

There are two classes of INK spider - the natural class and the PFI class.

The natural class will crawl from links it has found naturally on the web. If it finds a sitemap through it being linked to by other pages it has in its' crawl database - it will crawl the links from the sitemap.

The PFI class will crawl the pages it has been paid to crawl. It crawls no other pages - ever! It will not follow any links - ever!

Paying for a sitemap to be in PFI with the vain hope that it is a cheap way of getting all your pages indexed is a waste of both time and money (IMHO)

[Jill]

Thanks for that, Barry! I wanted to post that myself, but I wasn't 100% sure. I was hoping you'd chime in.



Jill

[SearchRank]

As I also stated in this post...

"We submitted a site map thru PFI from a site that was brand new, had no incoming links and no pages listed in the Inktomi index and you know what - it found and included all the pages listed on that site map. "

Maybe what your saying (MakeMeTop) about the PFI spider not crawling additional links is true but in this case it did.

Bottom line for one is to check!

If a client wants all their pages included via PFI and they are not already in there, then pay for them to be included. If they are, save them their money.

If they do not have the $$ to shell out for PFI and no pages are in there, then try submitting the home page or a site map and even after this, check to see if other pages were included.

It's easy to say that a PFI spider will not crawl additional links which may be true, but some spider crawled those links from the site map we submitted - whether the PFI or INK, I do not know but it occurred.

BTW I have seen this happen other times as well.

[harpsound]

I had help on another thread on this topic where my homepage link is dormant on AlltheWeb (June2002) and appears to be so on MSN with almost no other satellite pages in either index. I took advice and have paid for PFI of my homepage to both Inktomi and Fast to kick start things so I am following this thread with interest.

How long will it take before I see some evidence that there has been uptake on the PFI ala a current homepage cached in allthe web and a few more linked pages in MSN? It has been a week since submission. I have a PR4 or better in Google and rank highly for almost all my key phrases. The site has no subterfuge and is updated weekly.

Thanks for this great resource Jill and to all the contributors who are so free with their sharing.

Stephen V

[SearchRank]

How long will it take before I see some evidence that there has been uptake on the PFI ala a current homepage cached in allthe web and a few more linked pages in MSN?

Fast and Inktomi indexes PFI pages in 24-72 hours and then it depends on how frequently All The Web updates the public index which they are pretty fast at doing.

I would say anywhere from 3 days to no more than a month for both ATW and MSN.

[BrianR]

So, in sum, Barry says that the Site map PFI strategy doesn't work and David says that it most certainly does.

I guess I'll just have to try it for myself and report back...

BrianR

[SearchRank]

So, in sum, Barry says that the Site map PFI strategy doesn't work and David says that it most certainly does.

Just to clarify, I said it "can" work, not that it always does. It has worked for me a few times.

[projectphp]

Just to clarify, I said it "can" work, not that it always does. It has worked for me a few times

Either it does or not. I can't see this as being variable. IMHO, it is more likely a side effect, e.g. many high profile SE don't havew a robots.txt file (e.g. http://www.ninemsn.com.au/robots.txt is non existent). Therefore, every single SERP is allowed to be indexed. There are many other SERPs in various engines that are crawlable. So, if one of your pages were thus indexed, this may work, but surely there are better ways to achive the same result.

I am with Barry. There is no way this will work. It may by-proxy, but not directly.

Edited by projectphp, 23 September 2003 - 01:08 AM.

[compar]

Hopefully Barry -- MakeMeTop -- can answer this. Suppose I have a site which is already been spidered and index by Inktomi's free spider. But I would still like to do some fine tuning and hence desire the frequent crawling from a PFI. Now if I PFI the homepage, which version of the home page will show up in Inktomi's index. The one from the free spidering or the one from the PFI?

[MakeMeTop]

Now if I PFI the homepage, which version of the home page will show up in Inktomi's index.

The PFI version. However, I would be a little cautious about putting in your homepage if it is already in the index. It can have somewhat bizarre effects sometimes.

That is not to say I have never done this - and it can work well - but think about it before hand - and if your site homepage is in L$ US - forget it. The L$ listing will override your PFI entry.

[harpsound]

An unofficial experiment

It has been a week did my PFI for Inktomi and Fast. Both AlltheWeb and MSN have updated with all the old out of date (2002) pages removed.

Each SE has my current homepage and the site map only. Is that as good as it is going to get? No answer required as I will update any changes.

Regards
Stephen V

[harpsound]

An update

PFI Homepage only to fast and Inktomi 3 weeks ago.
My Homepage and map updated a week ago on both MSN and AlltheWeb.
No other pages crawled and MSN retains a few old links

Reason for doing this as as both spiders stopped updating circa June 2002.

I have been getting a little paranoid this last week as I realized that I put a free Google search box on my site about June 2002. Are these competing spiders/SEs penalizing you for loyalties elsewhere? Seems that the ethics of some SEs are less than exemplary.

Stephen

[Jill]

Are these competing spiders/SEs penalizing you for loyalties elsewhere?

No way. Gotta be just a coincidence. The other engines haven't sunk that low...yet.

Jill