High Rankings Search Engine Optimization ForumHigh Rankings Advisor Search Marketing Newsletter

Welcome Guest ( Log In | Register )

Important Announcement: ***Need an Affordable SEO Website Review?***
 
 Reply to this topicStart new topic
> Robots.txt, when does a spider request it?
domokun
post Jan 8 2004, 09:11 AM
Post #1


Web jockey
****

Group: Active Members
Posts: 249
Joined: 17-October 03
User's local time:
Feb 9 2010, 08:48 PM
Member No.: 1,108
if an external site links to my site, like so

www.wishihadakeyboard.com/nothinghere/index.html

but in my robots.txt file i have disallowed the 'nothinghere' folder

would a spider follow the link and happily crawl the page, or would it request the robots.txt file first and say "no chance, son; not allowed in there!"

?
Go to the top of the page
 
+Quote Post
Jill
post Jan 8 2004, 09:20 AM
Post #2


High Rankings Advisor
Group Icon

Group: Admin
Posts: 29,201
Joined: 21-July 03
User's local time:
Feb 9 2010, 03:48 PM
From: Ashland, MA
Member No.: 2
It might index the URL, but it won't follow the file and index the information or add it to it's database.

Jill
Go to the top of the page
 
+Quote Post
meta
post Jan 8 2004, 02:51 PM
Post #3


HR 5
*****

Group: Active Members
Posts: 301
Joined: 31-July 03
User's local time:
Feb 9 2010, 05:48 PM
From: Chicago
Member No.: 165
Jill, what does is mean to index the url if it is not added to the database?
Go to the top of the page
 
+Quote Post
Matt B
post Jan 8 2004, 06:05 PM
Post #4


The modem is the message.
******

Group: Active Members
Posts: 558
Joined: 21-July 03
User's local time:
Feb 9 2010, 05:48 PM
From: Canton, OH
Member No.: 4
It means that the spider may still follow the link, but not add the page to the database of results.

The robots.txt file is requested by both Google and Inktomi at the beginning of every crawl session, even if they are only minutes apart.
Go to the top of the page
 
+Quote Post
Jill
post Jan 8 2004, 07:16 PM
Post #5


High Rankings Advisor
Group Icon

Group: Admin
Posts: 29,201
Joined: 21-July 03
User's local time:
Feb 9 2010, 03:48 PM
From: Ashland, MA
Member No.: 2
It also means that the link may show up in a search result, but there will be no other information about it, no title, nothing in the cache, no description.

Jill
Go to the top of the page
 
+Quote Post
« Next Oldest · Robots.txt · Next Newest »
 

  
 Fast ReplyReply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 

Display Mode: Standard · Switch to: Linear+ · Switch to: Outline

Track this topic · Email this topic · Print this topic · Subscribe to this forum

This forum is sponsored by High Rankings, a Boston SEO Agency
- Lo-Fi Version Time is now: 9th February 2010 - 03:48 PM