I have a general understanding of this files purpose, but is there any reason that a non-search engine user would be requesting it?
I had an IP today request this file 15 times over 4 hours and I don't know why.
Subscribe to HRA Now!
SEO Class in Chicago, IL
Learn How To Optimize Your Website on July 26, 2013
Looking for personalized in-depth SEO training among your peers?
High Rankings is offering a 1-day customized SEO training class in Chicago. Class size is limited so please sign-up now if you want in!
Are you a Google Analytics enthusiast?
Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE!
www.CustomReportSharing.com
From the folks who brought you High Rankings!
More SEO Content
International SEM | Social Media | Search Friendly Design | SEO | Paid Search / PPC | Seminars | Forum Threads | Q&A | Copywriting | Keyword Research | Web Analytics / Conversions | Blogging | Dynamic Sites | Linking | SEO Services | Site Architecture | Search Engine Spam | Wrap-ups | Business Issues | HRA Questions | Online Courses
What Uses Robots.txt?
Started by
clubmilwaukee
, Apr 01 2004 08:03 PM
2 replies to this topic
#2
Jill
-
- Admin
-
- 32,375 posts
High Rankings Advisor
Posted 01 April 2004 - 08:07 PM
Hackers sometimes like to look at them to see if you have any stuff you don't want indexed which might be of interest to them.
But it's probably just some rogue spider or something, I would think.
Jill
But it's probably just some rogue spider or something, I would think.
Jill
#3
clubmilwaukee
-
- Active Members
-
- 83 posts
HR 3
Posted 01 April 2004 - 08:34 PM
Actually, turns out it was looksmart!
Problem is, that's the only thing they ask for, then the crawler leaves
Problem is, that's the only thing they ask for, then the crawler leaves
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users
