19 replies to this topic

[alnany]

I've been basically trying whatever I read and made sense. I've heard that building a sitemap helps the spider crawl the site better? How about a text version of the website? Thanks.

[Mel]

Hi Al

A sitemap is IMO almost a necessity on any website, though strictly it is not necessary if you have good links to all the pages in your site in a place the the spiders can find easily.

I see no reason to create a text version of any website.

[alnany]

Aw man... We've spent quite sometime on the text version... But thanks, at least now we know :T

[Jill]

Welcome, alnany!

It depends.

You might need a text version if your site is in all Flash. Is it? Or all graphics?

As to a site map, it's important if you feel your users need one to better navigate your site, and/or if your current navigation is not spider-friendly.

Jill

[qwerty]

I'm not sure what you plan to use the text version for. Is this for making it easier to print pages? If so, that's fine, but it can also be accomplished by calling a different style sheet.

Also, if you're going to have extra versions of every page, I'd recommend using your robots.txt file to disallow spiders from accessing the text version. You don't want the search engines finding duplicate content.

[Kenny]

Hello, can you please further explain how to use a "spider.txt"? ie. where do I place it, what the contents should be, how it works, etc. Thanks in advance!

[qwerty]

You must mean the robots.txt.

It's a plain text file that gets uploaded to your site's root directory, e.g., www.mydomain.com/robots.txt

What it does is specify which robots are to be refused access to which directories. Have a look at this page and let me know if you have any other questions.

[Mel]

If you want a good list of "bad" spiders to exclude in your robots.txt file Brette Tabke allows free use of his searchengineworld robots file, which has a pretty exhaustive list, but remember that you will have to customize the file to suit your site architecture.

[balz]

Just out of curiosity, what makes a spider "bad"?

b.

[qwerty]

I can think of two kinds of spiders that could be called bad: sometimes legitimate spiders from legitimate search engines will crawl too much. They'll request a ton of pages and burn up your bandwidth. I recently read about either Scooter (AltaVista) or Slurp (Inktomi) doing that to some directories (it's never happened to me, though). Under such circumstances, one might decide to disallow the spider, even though it would mean that you could get dropped from that SE's index.

The other kind would be something like an email harvester. Of course, it's not likely such a spider would obey the robots protocol, so disallowing it wouldn't accomplish much.

[Marc]

Site Maps are a must have....

here's my take on that subject, why you should have one!

http://www.highranki.....ite map&st=15

Take Care!

[Cruise Planner]

A related question about sitemaps...

When submitting to search engines, is it better to submit the url for the sitemap or the url for the homepage, or even both?

[qwerty]

Here's what it says on Google's Add URL page:

Only the top-level page from a host is necessary; you do not need to submit each individual page. Our crawler, Googlebot, will be able to find the rest. Google updates its index on a regular basis, so updated or outdated link submissions are not necessary. Dead links will 'fade out' of our index on our next crawl when we update our entire index

So submit the home page if you're going to submit anything, and make sure the home page has a text link to the site map on it.

[Scottie]

You will find most of us don't submit to search engines at all... there is no need.

Spiders will find your site by crawling links to it. If you submit and there are no other links to it, it won't stay in the index for long.

If you do submit (it doesn't hurt) you can submit both your homepage and the sitemap. If I were only going to submit one, I'd submit the home page.

[garyhall]

You will find most of us don't submit to search engines at all... there is no need.

You are correct and not a doubt about it. BUT try to explain that to
a customer who is paying you big bucks. You got to show him
something. Sitting on your hands is not an option when starting out
with a new customer. Basically I do the submitting for free after I
have been paid of course. It is just good PR to show the customer
you are doing something.

Years ago while walking through our large corporation goofing off,
I discovered if I carried some papers with me I wouldn't be bothered
by anyone. Appearances sometimes is most important, if you get the
drift.

Yahoo really irks me paying them the $299.00 bucks, but they are
sorta like the big gorilla, if you know what I mean.

Regards,

Gary