6 replies to this topic

[tellico4x4]

I know that it is possible to enter a meta tag to keep a robot from crawling a page (<meta name="robots" content="noindex,nofollow" />) and I know that it is possible to create your robots.txt file so that it won't crawl your site or certain directories and/or files.

My question is, does anyone know if there is a way to keep robots from crawling certain parts of a page? For example, if I wanted the robot to crawl an entire page expect for one paragraph on that page...

We have some drop-down lists on our site that we'd rather the robots skip over in order to get to the meaningful content.

Thanks in advance for any help that you could offer.

[chrishirst]

You can't block sections of pages (well you could with a bit of javascript [not recommended] trickery) but as crawlers don't read drop-downs (<select>) anyway it makes no difference.

But you can block pages and folders with  Robots.txt[/hr]

[tellico4x4]

Perhaps it is the way our site is built, but the choices in our drop-down lists show up in our search results quite often.

The first drop-down list that we have shows several different vehicle manufacturers so that our customer can just see parts for their vehicle. On many of our searches results, it lists all of those vehicles and that uses up all of our words for the description underneath our link...

We're trying to get the bots to skip over all those vehicles so that it can show more meaningful information in the search results.

[WebSlave]

Our site does that sometimes, too... we use php... does that make a difference?

[lyn]

from the WAG department...

...would it help if the dropdowns were written in at the end of the page and positioned with CSS? At least the crawlers would find them last and, possibly, be less inclined to use them in the description snippet.

???

[Tom Philo]

Would make no difference where on the page words are positioned. If a SE thinks that the matching text for the terms serached for is the best - it will still display it as the snippet.

There are some corporate SE products that allow you to block portions of a page from being searched by embedding special HTML tags that they reconize, but on the Internet there are no code standards to do so.

If the SE see the JS dropdown selections then the JS is only used to execute the selection chosen, and the HTML code behind the selection is in normal HTML being indexed by the SE and not being skipped by the SE. Maybe ensure that every item being used the menu is set inside <--! <script></script> --> tags

[Jill]

Remeber, the descriptions (or snippets) that the engines use is going to be different every time, depending on the search query. So you may solve the "problem" for one phrase, but you may get some other description that is just as bad, or worse.

The best way to control the descriptions is to make sure your Meta keyword tag uses the search query phrase that you're talking about, then you'll have a good chance of that showing instead.