I am helping out a client whose site is hosted elsewhere. His complaint is that although he is indexed in the major search engines, he does not come up well for his important keyword phrases. I reviewed his site, and for the most part it's sound. Good tags, content, text links, etc. So I scratched my head and kept looking. I finally got FTP access from his webmaster and see that there is no robots.txt file. He also does not have anything regarding "follow,index" in his Meta tags. Could this be a reason he's not performing well? Will spiders crawl a site that has no robots.txt? Any help you could provide would be much appreciated!
Subscribe to HRA Now!
Are you a Google Analytics enthusiast?
Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE!
www.CustomReportSharing.com
From the folks who brought you High Rankings!
More SEO Content
International SEM | Social Media | Search Friendly Design | SEO | Paid Search / PPC | Seminars | Forum Threads | Q&A | Copywriting | Keyword Research | Web Analytics / Conversions | Blogging | Dynamic Sites | Linking | SEO Services | Site Architecture | Search Engine Spam | Wrap-ups | Business Issues | HRA Questions | Online Courses
Sponsored Content
Missing Robots.txt File
Started by
outdoormind
, Feb 17 2004 03:05 PM
4 replies to this topic
#2
Alan Perkins
-
- Admin
-
- 1,559 posts
Token male admin
- Location:UK
Posted 17 February 2004 - 03:40 PM
Welcome to the forums outdoormind. 
No. (Highly unlikely, anyway).Could this be a reason he's not performing well?
Yes.Will spiders crawl a site that has no robots.txt?
#3
OldWelshGuy
-
- Moderator
-
- 4,713 posts
Work is Fun
- Location:Neath, South Wales, UK
Posted 17 February 2004 - 03:45 PM
Welcome outdoormind
Yep they will spider, they will also appear as a failed filed request in your stats, I am forever having to put robots.txt in after getting a gentle reminder from the failed request stats
Yep they will spider, they will also appear as a failed filed request in your stats, I am forever having to put robots.txt in after getting a gentle reminder from the failed request stats
#4
SearchRank
-
- Active Members
-
- 2,333 posts
HR 7
- Location:Phoenix, AZ
Posted 17 February 2004 - 03:48 PM
Welcome, outdoormind!
As Alan said, the robots.txt file or the absence of it is not going to negatively affect positioning efforts. There are many factors that could cause one to not have good positioning. Maybe they are going after key phrases that are too competitive? Maybe they do not have good overall search engine saturation? Maybe their content is not properly optimized to reflect the keywords they do want to target? I'd be looking at these and other factors.
Robots.txt is really about "what not to allow" rather than what to allow.
As Alan said, the robots.txt file or the absence of it is not going to negatively affect positioning efforts. There are many factors that could cause one to not have good positioning. Maybe they are going after key phrases that are too competitive? Maybe they do not have good overall search engine saturation? Maybe their content is not properly optimized to reflect the keywords they do want to target? I'd be looking at these and other factors.
Robots.txt is really about "what not to allow" rather than what to allow.
#5
outdoormind
-
- Members
-
- 2 posts
HR 1
Posted 17 February 2004 - 04:05 PM
Thanks for your help everyone. It just worried me because I made the unfortunate mistake one time of having the wrong robots.txt file on a client's site, (the disallow / --ouch!) so I wanted to make sure and never make a robots.txt mistake again!
His targeted keywords are extremely competitive, so I will focus on placing keywords throughout his text.
And thanks for the warm welcome!
His targeted keywords are extremely competitive, so I will focus on placing keywords throughout his text.
And thanks for the warm welcome!
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users
