Jump to content

  • Log in with Facebook Log in with Twitter Log In with Google      Sign In   
  • Create Account

Subscribe to HRA Now!


Are you a Google Analytics enthusiast?

Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE! 




From the folks who brought you High Rankings!

- - - - -

Dynamic Robots.txt

  • Please log in to reply
3 replies to this topic

#1 Shammi


    HR 1

  • Members
  • Pip
  • 2 posts
  • Location:Dhaka

Posted 02 October 2014 - 08:13 AM

Let's say I have a web site for hosting community generated content that targets a very specific set of users. Now, let's say in the interest of fostering a better community I have an off-topic area where community members can post or talk about anything they want, regardless of the site's main theme.

Now, I want most of the content to get indexed by Google. The notable exception is the off-topic content. Each thread has it's own page, but all the threads are listed in the same folder so I can't just exclude search engines from a folder somewhere. It has to be per-page. A traditional robots.txt file would get huge, so how else could I accomplish this?

#2 chrishirst


    A not so moderate moderator.

  • Moderator
  • 7,718 posts
  • Location:Blackpool UK

Posted 02 October 2014 - 08:49 AM

Robots.txt is for per site  exclusions.


The robots meta directive is for "per page" granularity of exclusions.

#3 Jill


    Recovering SEO

  • Admin
  • 33,244 posts

Posted 02 October 2014 - 08:58 AM

Curious as to why you don't want to content indexed? So what if it's off topic? Others still may be interested in finding it.

#4 Michael Martinez

Michael Martinez

    HR 10

  • Active Members
  • PipPipPipPipPipPipPipPipPipPip
  • 5,325 posts
  • Location:Georgia

Posted 02 October 2014 - 05:11 PM

If you don't have the resources to implement the "robots" meta directives on the pages, you can try using wildcards in "robots.txt" but that can get messy if you don't have unique identifiers in the page URLs of each section of your forum.

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

We are now a read-only forum.
No new posts or registrations allowed.