Hello all and thanks in advance. In your experience is having Robots.txt & Sitemap.xml for the same website redundant? Thanks!
Subscribe to HRA Now!
Are you a Google Analytics enthusiast?
Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE!
www.CustomReportSharing.com
From the folks who brought you High Rankings!
More SEO Content
International SEM | Social Media | Search Friendly Design | SEO | Paid Search / PPC | Seminars | Forum Threads | Q&A | Copywriting | Keyword Research | Web Analytics / Conversions | Blogging | Dynamic Sites | Linking | SEO Services | Site Architecture | Search Engine Spam | Wrap-ups | Business Issues | HRA Questions | Online Courses
Sponsored Content
Robots.txt & Sitemap.xml Redundant?
Started by
Tyl3r
, Aug 15 2007 03:09 PM
7 replies to this topic
#2
chrishirst
-
- Moderator
-
- 5,881 posts
A not so moderate moderator.
- Location:Blackpool UK
Posted 15 August 2007 - 05:22 PM
well, as they serve two totally different purposes, why would you think that??
#3
qwerty
-
- Moderator
-
- 8,289 posts
HR 10
- Location:Somerville, MA
Posted 15 August 2007 - 05:50 PM
You can use robots.txt to point to the sitemap, so they can work together, but of course you need both to do that.
#4
Tyl3r
-
- Members
-
- 5 posts
HR 1
Posted 15 August 2007 - 06:17 PM
QUOTE(chrishirst @ Aug 15 2007, 03:22 PM) 
well, as they serve two totally different purposes, why would you think that??
Oh, OK, if both have different purposes which would you say is more crucial to have? And don't they both function as ways for the robots to see all of the files available for spidering? If so, then what exactly is the difference? Thanks for your help on this.
#5
chrishirst
-
- Moderator
-
- 5,881 posts
A not so moderate moderator.
- Location:Blackpool UK
Posted 15 August 2007 - 06:21 PM
Nope,
robots.txt is for EXCLUDING urls from compliant crawlers.
and neither are crucial or even necessary.
robots.txt is for EXCLUDING urls from compliant crawlers.
and neither are crucial or even necessary.
#6
Tyl3r
-
- Members
-
- 5 posts
HR 1
Posted 15 August 2007 - 06:28 PM
So a robots.txt basically tells the robots what to do with a series of files....OK, that makes sense. Thanks again, you rule Comic Book Store Owner dude.
#7
chrishirst
-
- Moderator
-
- 5,881 posts
A not so moderate moderator.
- Location:Blackpool UK
Posted 16 August 2007 - 03:26 AM
Not quite.
robots.txt tells a bot what NOT to do with files or folders
robots.txt tells a bot what NOT to do with files or folders
#8
Jill
-
- Admin
-
- 32,317 posts
High Rankings Advisor
Posted 17 August 2007 - 02:15 AM
Why is this an either/or situation?
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users
