8 replies to this topic

[aquatix]

If you go to google or yahoo and search for terms like <removed keyword phrase> you will see my site at or near the top.

For some reason, no description shows! I checked my meta tags and they appear fine... at least like my other sites who have no such problem. This site has existed for 9 months or so and no description is showing in the SERPS. I know I'm losing visitors because of this.

I'm going crazy here... any ideas?

Edited by Randy, 02 January 2008 - 07:37 AM.
Removed keyword and domain info so this thread doesn't start to rank for your phrase.

[rush4rk]

It takes a while for discription to change.
had the same problem.

Edited by Randy, 02 January 2008 - 07:39 AM.
Removed quote with keyword phrase references.

[Randy]

It's your robots.txt file Aquatix.

Right now your robots.txt file is set up to disallow all spiders from all pages. You'll want to correct that ASAP.

[aquatix]

It's your robots.txt file Aquatix.

Right now your robots.txt file is set up to disallow all spiders from all pages. You'll want to correct that ASAP.

Thanks Randy! You are right, it was different than the robots.txt on other my other sites so I just copied from those. Now it reads:

User-agent: *
Disallow: /*.htm
Disallow: /backup/
Disallow: /css/
Disallow: /images/
Disallow: /js/
Disallow: /lists/
Disallow: /old/
Disallow: /php/
Disallow: /sitemap/

Is that right?

[qwerty]

Assuming I'm looking at the right site (the first in your signature), it looks like all your pages are .html files, so blocking .htm pages shouldn't be a problem. Of course, if there are no .htm pages on the server, you don't need to tell robots to stay away from them.

[aquatix]

Assuming I'm looking at the right site (the first in your signature), it looks like all your pages are .html files, so blocking .htm pages shouldn't be a problem. Of course, if there are no .htm pages on the server, you don't need to tell robots to stay away from them.

Actually, that site is fine. It's the second site in my sig (saltwaterfish) that is the problem.

[Randy]

I'm not sure why you're using

Disallow: /*.htm

Personally I wouldn't, even if I didn't currently have any .htm pages on the site. The reason being partly that I wouldn't want to take the chance of there never being a .htm page that I wanted to be spidered and partly because robots.txt can get really screwy sometimes since every spider out there could be using their own non-standard implementation of the standards.

For instance, in the robots.txt standard I believe their is supposed to be an implied wildcard at the end of each line. (Alan correct me if I'm wrong, you're the expert on the robots exclusion standards!) With this wildcard implementation if you had a line that said:

[/b]Disallow: /lederhosen[/b]

the spiders should technically exclude anything in your /lederhosen/ subdirectory and any file at the root level that started with the text string lederhosen. For example, files named lederhosen.htm, lederhosen.html, lederhosen.c lederhosen_red, etc

Given this, I would be worried about some engine actually sticking to the wildcard standard and making
Disallow: /*.htm
also disallow .html files.

Gonna move this to the Robots section of the forum, where those more knowledgeable about the standards will see it.

[aquatix]

Ok, so now that I've fixed it, how long would you say until the SERPS show my description?

[Randy]

It depends upon how often the spiders visit your site.

You should start seeing the datacenters getting updated within a few days of the spider visiting your site to see the change in your robots.txt, then send out a spider to fetch your pages. So anywhere from a week to several weeks depending upon what sort of schedule your site is on.