4 replies to this topic

[SERPico]

Today when i did a site:www.mydomain.com search i noticed that one page that i excluded from indexing through Robots.txt was still indexed by Google.

CODE

User-agent: *
Disallow: /page-name.html

Have a i made a mistake in the above robots.txt?

The page in question also had this:

CODE

<meta name="robots" content="noindex,nofollow">

How is this possible?

Or does Google completely ignore robots.txt nowadays?

[chrishirst]

And was it displayed as a URL only?

disallow: only tells bots NOT to index the page contents, it doesn't tell it NOT to list the URL should it find links to it.

same with the meta robots, the page has to be indexed before the bots know not to index it.

[SERPico]

Yep, as a URL only - it didn't had a cache link like other pages though.

I just thought it was odd the link was being displayed, it was also displayed under omitted results as well.

I guess it's normal to see this then

[Randy]

Yup, completely normal.

After all, Google does know the url address exists...

[SERPico]

Thanks guys!