8 replies to this topic

[iadvancemedia]

Dear Members

I am newbie in SEO. While uploading my website on IIS server, I was suggested to delete Robot.txt file. As a beginner I did the same as I was told.

I want to know, how important is the existence of Robot.txt file on the server. What should be its correct location? What it should contain and what shouldn't?

Also Note that I submitted my website to Google on 4 december 2003 (approx.) and still I did not received a single hit from googlebot. What could be the cause of delay?

Do I need to upload robot.txt file and resubmit?

Regards
iAdvanceMedia

[SearchRank]

Google would rather find your site by indexing one of the top directories or finding a link to your site on another site or page it already has in its index. With that said, are you trying to get your site listed in top directories or on other sites that you know are already listed in Google?

[Jill]

It's not that important if you don't have any sections of your site that you want blocked from spidering.

However, it's good form to have one. You should upload one that allows all robots to all sections of your site, if that's what you want to have happen.

Learn more at RobotsTxt.org.

(I moved your post to the technology and coding section of the forum.)

Jill

[qwerty]

I'm curious... who or what suggested that you delete your robots.txt?

[Grumpus]

Yes. There's no real need for a robots.txt file. All it does is tell a spider what not to do. As Jill says, it's a good idea to have one so that there is no doubt, but it is far better NOT to have one than to have one with an error in it. If you add one - make sure it's correct.

Submitting your site to Google really (as far as I can tell) does nothing more than have the power of a single low ranked inbound link for your site. Google will want to find several links to your site before it'll crawl it - and it'll need several more to go very deeply and frequently. So, you'll need 3-4 links to your site before Google pays any attention to it (or maybe one link from a PR4 or better page) - and you'll need even more if you want that attention to be enthusiastic.

G.

[iadvancemedia]

Thanks for the valuable information.

If I decide to have one. What should be the location?

Note that I have one password protected region for my clients.

Regards

[qwerty]

If you decide you want one, it has to go in the root directory: www.domain.com/robots.txt. That's the only place a spider will look for it, so it's useless anywhere else.

[powerofeyes]

Also Note that I submitted my website to Google on 4 december 2003 (approx.) and still I did not received a single hit from googlebot. What could be the cause of delay?

There is never a need to submit site to google, Infact what ever Brand new site we do we place the link of that site in couple of sites which googlebot visits twice or thrice a week, Bang we see that new site in the google Index within 2 or 3 days most of them fully indexed if it is a small site,
So links from quality sites is all that matters, We have been following this trend for a very long time and all the time it hardly takes 2 or 3 days to get a new site into the index,
Faster than Inktomi PFI crawl and everybody say Inktomi PFI helps to tweak pages, for us Googlebot helps to tweak pages, we tweak pages and wait for the results to get updated in a week and we see the ranking changes,

So build quality links,

VIJAY,

[iadvancemedia]

Hello all.

Vijay, is that the reason you have website 'Lambostuff' in your sign off even when it do not have any concern with SEO or Web Design/ Markeing?

Regards