2 replies to this topic

[sinbad]

I currently don't use a robots.txt file on a site I maintain because, if the file isn't done correctly it could do the more harm than good for what I understand it's purpose to be. That being to keep robots out of areas you don't want spidered. Now heres my problem. The site uses a access database to generate the content (item name,descrip,price etc) for the product pages the database is in a folder of its own. It has been suggested that I use disallow/database folder. Now if I disallow database folder how would the spider be able to pick up the content for these pages? Wouldn't I be blocking it?

One last thing I do use the robot meta tag on the .asp pages

Thank for your help

[SearchRank]

If there is no actual links to your database for a spider to follow (I doubt there is), there is no need to worry about a spider getting into there. Spiders only follow links. The only reason then to block a spider from a specific area of your site would be if there was a link to something that you didn't want it following.

Also, if you do block a spider from accessing your dbase folder, it is not going to hinder then from indexing product pages. The dbase merely generates info and then produces a page. A spider will see the page in its finished state, not the raw state such as a template before data is input into it.

[sinbad]

Thanks searchrank thats what I need to know.