19 replies to this topic

[BrianR]

David & Bob

Agreed. Except...

Every time I've checked site listings, Inktomi lists considerably less pages than both Google and Alltheweb - especially for dynamic pages.

If I've got to pay $39 to get a site map listed (and hence get all other dynamic pages listed) and then another $25 (or whatever it is) to get a second page listed for keyphrase testing, then it's still a bargain.

Sure, no-one wants to spend bucks they don't have to, but the very quick returns you get should far outweigh a $39 investment to get a site map listed. I'll let you know when I've tried it out on a small dynamic site sometime next month...

BrianR

[MakeMeTop]

There appear to be a few urban myths starting to make the rounds here!

i) submitting a site map through PFI to get all your pages crawled for $39.00 is wrong, wrong, wrong (and I'll say it again) - wrong! PFI spiders do not crawl from site maps! If you want to be naturally spidered - just get inward links from pages already crawled by INKs natural spider. PFI is for rapid inclusion of the page submitted - nothing else.

ii) you may change a URL at anytime during the currency of your PFI agreement provided the URL is from the same domain. If you change the URL, the previous URL will drop out of the index until picked up by natural spidering and will re-appear after the next natural index update. If you are happy to wait the couple of months this could take - then fine, you can replace a well-optimised page by another URL.

[SearchRank]

PFI spiders do not crawl from site maps! - MakeMeTop

We submitted a site map thru PFI from a site that was brand new, had no incoming links and no pages listed in the Inktomi index and you know what - it found and included all the pages listed on that site map.

True if a site is already mature and has some incoming links that Inktomi will pick it up on its own and true if you want rapid inclusion from a page that is not already in the index, then submit through PFI.

As for switching URLs within PFI and whether the one switched out drops from the index - my question is why would one want to submit a URL thru PFI and then switch with another anyway? If one optimizes a page and wants it crawled quickly and frequently, then leave it be. There's no need to switch things around, possibly to try to cheat the system. The only cases where we have done this is if a client does a site redesign and changes their file structure or they drop one keyword phrase for another which requires optimization of another page on the site.

[MakeMeTop]

>you know what - it found and included all the pages listed on that site map.

I know it didn't, I'm afraid. And I have been watching this over thousands of URLs and hundreds of domains for years. Your pages being picked up by INK are nothing to do with your PFI submission, but a lot to do with INK spidering new sites quicker and quicker. The record for me is 36 hours for an entire site after a Yahoo submission.

I also put a link from a site already in INK to another site's home page and discovered the whole site crawled and indexed within 72 hours - before I even had a chance to put it into PFI. So, I didn't bother!

PFI spiders come from specific IPs - but are in the same C block as standard Slurp spiders. It is easy to get the two confused.

If you want me to get someone from INK to confirm what I have said - I'll attempt to do so. PFI spiders crawling links is economic suicide for Inktomi. It doesn't happen, now - or ever in the past.

But INK is spidering new content faster than ever before.

Edited by MakeMeTop, 22 September 2003 - 08:57 AM.

[SearchRank]

Could it be then that when one submits a page or pages to Inktomi thru PFI and the site has not already been indexed by the regular spider, that it would prompt it to visit and crawl the site? Maybe this is what happened in the example I have been using?!?!