3 replies to this topic

[EddyGonzalez]

I have a set of hundreds of subfolders in the following format

/laptops/acer/netbook/red/
/laptops/acer/netbook/blue/
...

In Robots.txt I want to disallow all these subfolders but I want the pages i.e.

/laptops/acer/netbook/page1.html to be crawled

How can I do this without having to do:
User-agent: *
Disallow: /laptops/acer/netbook/black
Disallow: /laptops/acer/netbook/white
Disallow: /laptops/acer/netbook/fast
Disallow: /laptops/acer/netbook/slow
etc... hundreds of time?

thanks
Eddy

[Alan Perkins]

The best way to achieve what you want is to structure the URLs a little differently in order to make disallowing easier. Examples:

• /ni/laptops/acer/netbook/red/ => Disallow: /ni/
• /laptops/acer/netbook/red/?noindex => Disallow: /*noindex*

Failing that, I would do the hundreds of disallows or use something else like a combination of nofollow on links to the pages, and noindex on the pages themselves.

There is a risky way to do what you want, which is:

CODE

User-agent: Googlebot
Disallow: /laptops/acer/netbook/
Allow: /laptops/acer/netbook/*.html$

However, I can't vouch for this ... you'd have to try it and see.

[Michael Martinez]

I have a set of hundreds of subfolders in the following format

/laptops/acer/netbook/red/
/laptops/acer/netbook/blue/
...

In Robots.txt I want to disallow all these subfolders but I want the pages i.e.

/laptops/acer/netbook/page1.html to be crawled

If you don't have index pages in those folders, you may be leaving your server to exposure at some level. At the very least, the user experience won't be very productive. I typically handle this by using HTTP meta refresh tags in blank index pages for the folders, rather than fiddle with .htaccess or robots.txt. That gives me the flexibility to add a full content index page later on without having to risk fudging up the vital crawl/access management files for the whole Website.

It's purely a matter of personal site management style. In my opinion, there is no one right way to handle the situation. But you definitely want to remember what incidental visitors may see if they hit those URLs.

[EddyGonzalez]

Thanks for the tips guys.

tried:
User-agent: Googlebot
Disallow: /laptops/acer/netbook/
Allow: /laptops/acer/netbook/*.html$

and it seems to work well but I appreciate your caution Michael.

eddy