5 replies to this topic

[adi_azar]

Hello guys,
I want to prevent the search engine crawler from following a URL in the result page. This URL points to a page called addData.jsp in my site. I am planning to add a robots tag to the page addData.jsp with the following content:
<meta name="robots" content="noindex,nofollow">

Does that solve the problem?

Adi

[Tom Philo]

yes, for the bots that honor the robot protocol that works.

To keep others that do not honor it I would also restrict the page via security so that would stop the non-honoring bots from going there and they would get an accessed denied (if that does not break your app.)

[Shane]

If you're only wanting to prevent the following of one URL, that code won't work. It prevents the following of all URLs on that page. You can either use a nofollow attribute on the link or put that file in your robots.txt.

[torka]

Nofollow on the link won't do anything to stop them from crawling that URL. We've had other threads discussing that; I believe some folks have tested it and found, for instance, that Yahoo doesn't appear to be paying any attention to nofollow. AFAIK, Google has never said it would prevent them from following the link, it just indicates you don't trust the page you've linked to.

--Torka

[Alan Perkins]

I want to prevent the search engine crawler from following a URL in the result page. This URL points to a page called addData.jsp in my site. I am planning to add a robots tag to the page  addData.jsp with the following content:
<meta name="robots" content="noindex,nofollow">

Does that solve the problem?

No. The only part of the robots tag that is reliable is "noindex". Support for "nofollow" is very spotty. Sometimes it is obeyed. Sometimes it is crawled but not indexed. Sometimes it is crawled and indexed, i.e. ignored completely.

The best solution is to use a robots.txt file. That prevents the URL being crawled and the content being indexed.

[Shane]

Nofollow on the link won't do anything to stop them from crawling that URL.

Man, I couldn't have been more wrong on that. Thanks for correcting it!