Can you help better explain the robots.txt file ?
Is it needed, not needed, etc... And if it's needed, what should it look like?
Thanks!
-- Josh
Subscribe to HRA Now!
Are you a Google Analytics enthusiast?
Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE!
www.CustomReportSharing.com
From the folks who brought you High Rankings!
More SEO Content
International SEM | Social Media | Search Friendly Design | SEO | Paid Search / PPC | Seminars | Forum Threads | Q&A | Copywriting | Keyword Research | Web Analytics / Conversions | Blogging | Dynamic Sites | Linking | SEO Services | Site Architecture | Search Engine Spam | Wrap-ups | Business Issues | HRA Questions | Online Courses
Sponsored Content
Robots.txt
Started by
josh1r
, Mar 11 2004 04:21 PM
2 replies to this topic
#2
Jill
-
- Admin
-
- 32,316 posts
High Rankings Advisor
Posted 11 March 2004 - 06:40 PM
Josh I split this post off from the other topic that wasn't relevant.
You can learn about robots.txt at http://www.robotstxt.org and also do a search here for robots.txt and you will find our numerous other threads on it.
Jill
You can learn about robots.txt at http://www.robotstxt.org and also do a search here for robots.txt and you will find our numerous other threads on it.
Jill
#3
Randy
-
- Moderator
-
- 17,540 posts
Convert Me!
Posted 12 March 2004 - 07:30 AM
Quickie explanation since I'm here. 
robots.txt is a file you put at the root level of your domain that tells good spiders what not to do. If you don't have any areas of your site that you want to be Off Limits, you don't need one technically.
On the other hand, every search engine spider is going to look for a robots.txt file when they visit your site for a crawl. So rather than them getting a 404 Not Found error, I think it's always better to upload a blank robots.txt file. Since it's used only to tell spiders what you don't want them to crawl, if the file is blank you're telling the spiders that you want them to spider and index everything.
robots.txt is a file you put at the root level of your domain that tells good spiders what not to do. If you don't have any areas of your site that you want to be Off Limits, you don't need one technically.
On the other hand, every search engine spider is going to look for a robots.txt file when they visit your site for a crawl. So rather than them getting a 404 Not Found error, I think it's always better to upload a blank robots.txt file. Since it's used only to tell spiders what you don't want them to crawl, if the file is blank you're telling the spiders that you want them to spider and index everything.
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users
