3 replies to this topic

[Comtrad]

I am looking in google webmasters and there are over 1k links restricted by the robots.txt. I understand why they would be restricted, which is because we have it so it can't index User ID sessions. However every link that is restricted has a user ID that we do not have on our site.

http://example.com/M...?UserID=2667723 URL restricted by robots.txt [?] Sep 10, 2007
http://example.com/M...?UserID=2667724 URL restricted by robots.txt [?] Sep 10, 2007
http://example.com/M...?UserID=2667725 URL restricted by robots.txt [?] Sep 10, 2007
http://example.com/M...?UserID=2667726 URL restricted by robots.txt [?] Sep 10, 2007
http://example.com/M...?UserID=2667727 URL restricted by robots.txt [?] Sep 10, 2007
http://example.com/M...?UserID=2667728 URL restricted by robots.txt [?] Sep 7, 2007

Those 6 User ID's are not registered on our site. However it shows up usually in that order with each link for over 1k links being restricted. Can google make their own ID's?

Anyone seen this before?

This is our robots.txt

CODE

User-agent:*
Disallow: /*?
Disallow: /sendlink.aspx

[Randy]

Can google make their own ID's?

Can they? Sure, they could. But they don't as far as I know.

Anyone seen this before?

Yup, sure have. In the vast majority of cases it was just referrer spam, where someone adds something to your url to get it to show up in your stats on the hope that

1) You'll click through to their site to see what they're doing, bringing them (very unqualified) traffic;

2) Your stats are not protected and get spidered by the search engines, which can end up giving them a link that doesn't really appear on your site at all.

[Comtrad]

Can they? Sure, they could. But they don't as far as I know.
Yup, sure have. In the vast majority of cases it was just referrer spam, where someone adds something to your url to get it to show up in your stats on the hope that

1) You'll click through to their site to see what they're doing, bringing them (very unqualified) traffic;

2) Your stats are not protected and get spidered by the search engines, which can end up giving them a link that doesn't really appear on your site at all.

How would I check if my stats are protected or not, and go about fixing it?

[Randy]

Easy to check. When you go to your Stats page does it ask you for a username and password?

If the server requires a User/Password they're protected. If not, they're not.

How to fix it depends upon what type of server you're on, what control panel is installed, your server permissions, etc. Many control panels these days give you an easy way to password protect directories, so that would be the first place to start. Or if you're on a *nix system you could always do it manually by uploading .htaccess and .htpasswd files.