Is The Robots.txt Header Imp?, It is passing html header to plain text Options

[Aji]

The pages under domain X is not getting cached and the header for its robots.txt says,

CODE

HTTP/1.x 200 OK
Date: Tue, 13 Jun 2006 07:06:47 GMT
Server: Apache/1.3.36 (Unix) mod_fastcgi/2.4.2 mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 PHP/4.4.2 FrontPage/5.0.2.2635.SR1.2 mod_ssl/2.8.27 OpenSSL/0.9.7a
X-Powered-By: PHP/4.4.2
Content-Type: text/html
Connection: close

The pages under domain Y is getting properly cached and the header for its robots.txt says,

CODE

HTTP/1.x 200 OK
Date: Tue, 13 Jun 2006 06:28:33 GMT
Server: Apache/1.3.36 (Unix) mod_fastcgi/2.4.2 mod_gzip/1.3.26.1a mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 PHP/4.4.2 FrontPage/5.0.2.2635.SR1.2 mod_ssl/2.8.27 OpenSSL/0.9.7a
Last-Modified: Tue, 02 May 2006 08:44:40 GMT
Etag: "3801cb-132-44571bf8"
Accept-Ranges: bytes
Content-Length: 306
Content-Type: text/plain
Age: 11710
Connection: keep-alive

The difference that I can see if
Content-Type: text/html and Content-Type: text/plain

Can this be a possible reason for no cache with Google? What are the other possibilities for the pages not getting cached (rather loosing the caches from Google)? I am assuming no penalty and no filters from Google as these sites were doing good and are very helpful ones with no SEO done.

Thanks,
Aji

This post has been edited by Aji: Jun 13 2006, 04:54 AM

[Randy]

The two should make no difference Aji.

You say it's not getting indexed. Is it getting spidered? Have you checked the log files to see if the spiders are visiting various pages?

There are a thousand reasons why a site may not be getting indexed. With as flakey as Google has been since Big Daddy began, I would bet it has much more to do with that than anything else, assuming you don't have the entire domain blocked in the robots.txt file.