Jump to content

  • Log in with Facebook Log in with Twitter Log In with Google      Sign In   
  • Create Account

Subscribe to HRA Now!

 



Are you a Google Analytics enthusiast?

Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE! 

 



 

 www.CustomReportSharing.com 

From the folks who brought you High Rankings!



Photo

I Need Help On Preventing Google To Crawl A Specific Or Group Urls


  • This topic is locked This topic is locked
No replies to this topic

#1 Mir

Mir

    HR 1

  • Members
  • Pip
  • 1 posts

Posted 06 December 2010 - 07:03 AM

Hi guys,

NOTE: I'm not good in english smile.gif

First of all, I already searched here about my problem but I didn't find a solution for my problem

I have a problem about the site I handled, it uses Drupal 6. Now my problem is I already have a "robots.txt", if I am not wrong, this file is used to prevent google bots or other bots to crawl the page. I added some links I don't want to be crawled, after few days I checked the site on google using this, site:www.yourdomain.com. The pages I added on the robots.txt is still crawled by google.

Here's my robots.txt

CODE
User-agent: *
# Directories
Disallow: /includes/
Disallow: /misc/
Disallow: /modules/
Disallow: /profiles/
Disallow: /scripts/
Disallow: /sites/
Disallow: /themes/
Disallow: /blog/
# Files
Disallow: /CHANGELOG.txt
Disallow: /cron.php
Disallow: /INSTALL.mysql.txt
Disallow: /INSTALL.pgsql.txt
Disallow: /install.php
Disallow: /INSTALL.txt
Disallow: /LICENSE.txt
Disallow: /MAINTAINERS.txt
Disallow: /update.php
Disallow: /UPGRADE.txt
Disallow: /xmlrpc.php
# global path
Disallow: /admin/*
Disallow: /comment/*
Disallow: /contact/*
Disallow: /logout/*
Disallow: /search/*
Disallow: /search/node/*
Disallow: /node/*
Disallow: /node/add/*
Disallow: /visit/*
Disallow: /taxonomy/*
Disallow: /taxonomy/term/*
Disallow: /user/*
Disallow: /user/login/*
# direct path
Disallow: /taxonomy/term/12
Disallow: /taxonomy/term/6
Disallow: /taxonomy/term/21
Disallow: /taxonomy/term/36
Disallow: /taxonomy/term/5
Disallow: /taxonomy/term/14
Disallow: /taxonomy/term/42
Disallow: /taxonomy/term/40
Disallow: /taxonomy/term/13
Disallow: /taxonomy/term/9?page
Disallow: /taxonomy/term/36?page=2
Disallow: /taxonomy/term/14?page=1
Disallow: /taxonomy/term/14?page=2
Disallow: /taxonomy/term/9?page=5
Disallow: /taxonomy/term/36?page=1
Disallow: /taxonomy/term/42?page=1
Disallow: /taxonomy/term/46?page=1
Disallow: /taxonomy/term/12?page=1
Disallow: /taxonomy/term/13?page=1
Disallow: /taxonomy/term/48?page=1
Disallow: /taxonomy/term/46?page=2
Disallow: /taxonomy/term/9?page=2
Disallow: /?q=taxonomy/term/12
Disallow: /?q=taxonomy/term/6
Disallow: /?q=taxonomy/term/21
Disallow: /?q=taxonomy/term/36
Disallow: /?q=taxonomy/term/5
Disallow: /?q=taxonomy/term/14
Disallow: /?q=taxonomy/term/42
Disallow: /?q=taxonomy/term/40
Disallow: /?q=taxonomy/term/13
Disallow: /?q=taxonomy/term/9?page
Disallow: /?q=taxonomy/term/36?page=2
Disallow: /?q=taxonomy/term/14?page=1
Disallow: /?q=taxonomy/term/14?page=2
Disallow: /?q=taxonomy/term/9?page=5
Disallow: /?q=taxonomy/term/36?page=1
Disallow: /?q=taxonomy/term/42?page=1
Disallow: /?q=taxonomy/term/46?page=1
Disallow: /?q=taxonomy/term/12?page=1
Disallow: /?q=taxonomy/term/13?page=1
Disallow: /?q=taxonomy/term/48?page=1
Disallow: /?q=taxonomy/term/46?page=2
Disallow: /?q=taxonomy/term/9?page=2
# Paths (clean URLs)
Disallow: /admin/*
Disallow: /comment/reply/
Disallow: /contact/*
Disallow: /logout/*
Disallow: /node/add/*
Disallow: /search/*
Disallow: /search/node/*
Disallow: /node/*
Disallow: /node/305
Disallow: /node/224
Disallow: /node/282
Disallow: /node/273
Disallow: /visit/*
Disallow: /visit/titancasino/*
Disallow: /user/*
Disallow: /user/register/
Disallow: /user/register/
Disallow: /user/password/
Disallow: /user/password/
Disallow: /user/login/*
Disallow: /user/login?destination
Disallow: /user/login?destination=comment%2Freply%2F213%23comment-form
Disallow: /articoli/test2.htm
Disallow: /taxonomy/*
Disallow: /taxonomy/term/*
# Paths (no clean URLs)
Disallow: /?q=admin/*
Disallow: /?q=comment/reply/
Disallow: /?q=contact/*
Disallow: /?q=logout/*
Disallow: /?q=node/add/*
Disallow: /?q=search/*
Disallow: /?q=search/node/*
Disallow: /?q=node/*
Disallow: /?q=node/305
Disallow: /?q=node/224
Disallow: /?q=node/282
Disallow: /?q=node/273
Disallow: /?q=visit/*
Disallow: /?q=visit/titancasino
Disallow: /?q=taxonomy/*
Disallow: /?q=taxonomy/term/*
Disallow: /?q=user/*
Disallow: /?q=user/register/
Disallow: /?q=user/register/
Disallow: /?q=user/password/
Disallow: /?q=user/password/
Disallow: /?q=user/login/*
Disallow: /?q=user/login?destination
Disallow: /?q=user/login?destination=comment%2Freply%2F213%23comment-form
Disallow: /?q=articoli/test2.htm
# Paths (clean URLs) - fixed!
Disallow: /admin
Disallow: /comment/reply
Disallow: /contact
Disallow: /logout
Disallow: /node/add
Disallow: /search*
Disallow: /search/node
Disallow: /node*
Disallow: /node/305
Disallow: /node/224
Disallow: /node/282
Disallow: /node/273
Disallow: /visit*
Disallow: /visit/titancasino
Disallow: /taxonomy*
Disallow: /taxonomy/term*
Disallow: /user*
Disallow: /user/register
Disallow: /user/register
Disallow: /user/password
Disallow: /user/password
Disallow: /user/login*
Disallow: /user/login?destination
Disallow: /user/login?destination=comment%2Freply%2F213%23comment-form
Disallow: /articoli/test2.htm
# Paths (no clean URLs) - fixed!
Disallow: /?q=admin
Disallow: /?q=comment/reply
Disallow: /?q=contact
Disallow: /?q=logout
Disallow: /?q=node/add*
Disallow: /?q=search*
Disallow: /?q=search/node
Disallow: /?q=node*
Disallow: /?q=node/305
Disallow: /?q=node/224
Disallow: /?q=node/282
Disallow: /?q=node/273
Disallow: /?q=visit*
Disallow: /?q=visit/titancasino
Disallow: /?q=taxonomy*
Disallow: /?q=taxonomy/term*
Disallow: /?q=user*
Disallow: /?q=user/register
Disallow: /?q=user/register
Disallow: /?q=user/password
Disallow: /?q=user/password
Disallow: /?q=user/login*
Disallow: /?q=user/login?destination
Disallow: /?q=user/login?destination=comment%2Freply%2F213%23comment-form
Disallow: /?q=articoli/test2.htm
# Sitemap
Sitemap:  http://www.yoursite.com/sitemap.xml





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

SPAM FREE FORUM!
 
If you are just registering to spam,
don't bother. You will be wasting your
time as your spam will never see the
light of day!