7 replies to this topic

[blue]

Hello,
I am looking to prevent SE spiders from crawling and indexing one of my pages on my web site. Does anyone know of anything i can put in my HTML to do this? I ran across this, <meta name="MSSmartTagsPreventParsing" content="TRUE">
Will this prevent an SE spider from crawling/indexing a page? Any input is appreciated.
-Blue

[Mel]

Hi Blue:

You can put a robots tag in your head section of the page you do not want to be spidered like this:
<META NAME="ROBOTS"' CONTENT="NOINDEX">

or there is a special tag that only works for Googlebot:
<META NAME="GOOGLEBOT" CONTENT="NOARCHIVE">.

[Jill]

Your best bet is to use the robots exclusion in your robots.txt file. Check out RobotsTxt.org for more details.

The Meta robots tag has been known to be not always obeyed by the spiders.

Jill

[Alan Perkins]

I ran across this, <meta name="MSSmartTagsPreventParsing" content="TRUE">
Will this prevent an SE spider from crawling/indexing a page?

No.

You should either use the robots meta tag (as described by Mel above) or the robots.txt protocol. Details of both can be found at http://www.robotstxt.org/

[blue]

robotstxt.org was very helpful. This is what i am going to place in my header tags,

<META NAME="ROBOTS" CONTENT="NOINDEX">

<META NAME="ROBOTS" CONTENT="NOFOLLOW">

<META NAME="GOOGLEBOT" CONTENT="NOARCHIVE">

Is it ok to use all three, or will it cause a problem?

[qwerty]

If you're going to use a robots meta tag, I believe the correct version it to combine your first two examples:

<meta name="robots" content="noindex, nofollow">

[Jill]

Be sure to use the robots.txt also.

Jill

[Alan Perkins]

robotstxt.org was very helpful. This is what i am going to place in my header tags,

<META NAME="ROBOTS" CONTENT="NOINDEX">

<META NAME="ROBOTS" CONTENT="NOFOLLOW">

<META NAME="GOOGLEBOT" CONTENT="NOARCHIVE">

Is it ok to use all three, or will it cause a problem?

You just need this:

<META NAME="ROBOTS" CONTENT="NOINDEX,NOFOLLOW">

Forget the Googlebot tag. That does something slightly different (it allows indexing but disallows the cached page from being seen. You want to disallow indexing.)

The Meta robots tag does not prevent robots accessing your page, it prevents compliant search engines from indexing your pages - a subtle but important difference. Use the robots.txt file to prevent compliant robots accessing your pages.