If you want all pages to get indexed and all pages are linked from home page that has plenty of incoming links, is a robots.txt file still needed? For what?
Subscribe to HRA Now!
Are you a Google Analytics enthusiast?
Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE!
www.CustomReportSharing.com
From the folks who brought you High Rankings!
More SEO Content
International SEM | Social Media | Search Friendly Design | SEO | Paid Search / PPC | Seminars | Forum Threads | Q&A | Copywriting | Keyword Research | Web Analytics / Conversions | Blogging | Dynamic Sites | Linking | SEO Services | Site Architecture | Search Engine Spam | Wrap-ups | Business Issues | HRA Questions | Online Courses
Sponsored Content
Robots.txt
Started by
rohgan03
, Mar 05 2004 02:53 PM
3 replies to this topic
#2
SearchRank
-
- Active Members
-
- 2,333 posts
HR 7
- Location:Phoenix, AZ
Posted 05 March 2004 - 02:59 PM
The robots.txt is more for excluding spiders from indexing rather than trying to get them to index.
#3
Alan Perkins
-
- Admin
-
- 1,559 posts
Token male admin
- Location:UK
Posted 05 March 2004 - 03:00 PM
No, it's not needed.
It's good form to have an empty one to show you explictly permit robots. Also if you implement a 404 handler, an empty robots.txt file will prevent your 404 handler having to handle requests for robots.txt.
It's good form to have an empty one to show you explictly permit robots. Also if you implement a 404 handler, an empty robots.txt file will prevent your 404 handler having to handle requests for robots.txt.
#4
arman
-
- Active Members
-
- 87 posts
HR 3
Posted 05 March 2004 - 10:53 PM
Robots.text file is helpful if you want to exclude certain area of your site from the bot or spider.
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users
