3 replies to this topic

[Marchy]

I have a Joomla website which I have built a directory in it with SOBI2. There are 2 categories in it that does not have good information and I would like the robots to not visit them. I tried a few things with the robot text but it does not work. The problem I have is that the url does not show a specific category like if you have www.example.com/fruit/apples.html than you know that it is the fruit category that you will disallow. But I just have a long url with not really anything I can see as a categpory. I am not too familiar with this.

[Randy]

Are the the URL addresses dynamic and have variables and values? (eg something like www.domain.com/page.php?variable=value)

If so, look for something in the variable/value pairs that denote the category information. The variable name is often going to be something like catID or cat_id and the value may be a number or text string.

Once you identify the category id variable and the values you need to exclude the rest is pretty easy to do with robots.txt since Google allows you to exclude specific dynamic url variable/value pairs. And you can even use wildcard characters to make sure you get them all.

As an example, if your subdirectory was "sobi2" and you found a category id variable named of "cat_id" and you wanted to exclude a category that had a value of 1254 you could exclude this in robots.txt like

CODE

Disallow: /sobi2/*cat_id=1254

[Marchy]

Thanks Randy, I am pretty sure. I got it right. I have this question. I can you make sure that it will be noindex because when I look at the the source page you do not see the meta noindex? I just wanted to have a way to make sure it was disallowed. I took one of my entry that had never been crawled and i will check it from time to time if it get crawled.

[Randy]

You lost me Marchy.

If you're making changes to your robots.txt file you won't see anything in the source code of the pages. The same is also true going the opposite direction. They're two ways to reach the same goal.