Best Answer Alan Perkins , 09 October 2013 - 09:29 AM
How can I add that to the robots.txt file without stopping google from crawing the whole site.
You can't do it with robots.txt.
There are ways to attempt to NOINDEX /, even when / is a HTTP redirect, but I'm not going to tell you what they are because it's a really bad idea! Basically, you need to accept that / is going to be indexed. It's no big deal.Go to the full post