61 replies to this topic

[redbull1977]

Hello there! I am a web developer and recently my boss asked me the following:

Since we submit an optimized site what is going on with spiders, and that is how often do they spiders pass from my site, do we have to resubmit the site to engines and directories, will the spiders pass forever and if that happens should i refresh the site's content?

Could anyone help me?

[chrishirst]

welcome to HR redbull1977

no need to resubmit ever! to anywhere. If you have incoming links the crawlers will keep finding your site and if you have changed the content the results will reflect that after two or three days.

[Randy]

Welcome Redbull !

[Michael Martinez]

Resubmission is only useful if your site isn't getting crawled frequently. Generally speaking, the crawlers tend to come back more often as you add more content to your site.

Many people coax Google and Yahoo! into regularly crawling their sites by submitting 1-2 pages per week for a couple of months. After that, they just add new content to their site maps and are happy with their crawling experience.

Your mileage may vary.

[Haystack]

Many people coax Google and Yahoo! into regularly crawling their sites by submitting 1-2 pages per week for a couple of months.

That time would likely be better spent finding additional links to the site, because the site won't rank competitively without inbound links, and more important sites (based on link popularity) tend to get crawled more often.

[Jill]

To repeat what Chrishirst said, submitting your site to the spidering search engines is never necessary, or something that is worth doing.

[Michael Martinez]

That time would likely be better spent finding additional links to the site, because the site won't rank competitively without inbound links

Neither point is true. It only takes a minute to submit a site for crawling to Google. And you can rank just fine in most searches without backlinks at Google. Many sites do so, including a large number of my own.

Now, your other point, about more important sites getting crawled more often is true, but any site can get crawled regularly and relatively frequently if it demonstrates to Google that it is updating regularly with new content.

And let me reiterate that I am not saying submission is NECESSARY, I am saying that it gets new content crawled MUCH faster than simply waiting for Google to find you through links.

It takes considerably less time to submit a new page to Google than to get someone to link to you. Google also frequently crawls new content within a week, but not always. They state explicitly that they do NOT guarantee a submitted site will be crawled or that a crawled site (regardless of how they found it) will be indexed.

The general reluctance in the SEO industry to submitting content to Google, so far as I know, was established a few years ago when many of us determined that Google seemed to penalize submitted sites. That is, until Google found a site through its own crawling, sites did not perform well in its index. People saw no advantage to being listed quickly if they achieved no significant visibility.

But that hasn't been a problem for at least six months, maybe longer than that. I was on a long hiatus last year and during that time Google implemented a fair number of changes. They have implemented many more changes since December.

Right now, there is no apparent disadvantage to submitting a site directly to Google, and there IS an advantage of getting a submitted site into the index (with visibility) sooner than waiting for a spider to crawl a link.

I have been submitting new content on a weekly basis since mid-December, and it has consistently shown up in Google's results within 7-10 days (except for one page where I made a technical error, and it appeared in the index after I fixed the error).

Edited by Michael Martinez, 21 April 2005 - 11:26 PM.

[Haystack]

Neither point is true. It only takes a minute to submit a site for crawling to Google. And you can rank just fine in most searches without backlinks at Google. Many sites do so, including a large number of my own.

If it takes a minute to submit a site for crawling, why not spend two minutes to do something that will have both short and long term benefits? I've never seen a site with no backlinks rank just fine.

The general reluctance in the SEO industry to submitting content to Google, so far as I know, was established a few years ago when many of us determined that Google seemed to penalize submitted sites.

I believe that's a misconception from people who didn't understand the importance of link popularity. If Google is aware of a site, yet determines that not a single web page on the whole web links to it, changes are pretty good that it won't rank competitively. This isn't a penalty.

I have been submitting new content on a weekly basis since mid-December, and it has consistently shown up in Google's results within 7-10 days (except for one page where I made a technical error, and it appeared in the index after I fixed the error).

That doesn't sound much different from what one would expect from content linked to from an indexed site.

[Jill]

I have been submitting new content on a weekly basis since mid-December, and it has consistently shown up in Google's results within 7-10 days (except for one page where I made a technical error, and it appeared in the index after I fixed the error).

I have been not submitting new content to Google for 5 years, and all my new content gets added in that timeframe as well.

Submitting pages to the add url form doesn't seem to have any benefit as far as I can tell. Page gets added either way.

[Michael Martinez]

If it takes a minute to submit a site for crawling, why not spend two minutes to do something that will have both short and long term benefits?

If we could really get an indexed backlink in two minutes, rather than two weeks (or longer), then I would say you have a point.

I've never seen a site with no backlinks rank just fine.

But I have, and so have other people. And, of course, it's hard to come up with specific examples on the fly because sites don't remain unlinked to for long if they place highly in search results.

For example, of my own recent new sites, I cannot think of anything that now don't have backlinks, but they all ranked pretty well (top ten) when starting out.

You're not going to see that happen for a site that focuses on a hyperoptimized expression like "real estate" or "search engine optimization", but most people don't search for hyperoptimized expressions.

The general reluctance in the SEO industry to submitting content to Google, so far as I know, was established a few years ago when many of us determined that Google seemed to penalize submitted sites.

I believe that's a misconception from people who didn't understand the importance of link popularity. If Google is aware of a site, yet determines that not a single web page on the whole web links to it, changes are pretty good that it won't rank competitively. This isn't a penalty.

It has been a very long time since Google determined rankings on the basis of link popularity. That "seeming penalty" was actually considered a reward for quality design and content, in that it got listed in the search results at all without backlinks. It was also proof that Google would indeed index and list pages without backlinks.

It's just that people wanted to get their sites into Google with top rankings, so the strategy shifted to getting another site (preferably a directory) to link to your site, and then to submit that OTHER page for crawling. Quite a few people created hallway pages on throwaway domains just to get their new sites crawled. They reported great success for a few months. I haven't seen anyone mention using this technique in at least a couple of years. I suppose, like so many other techniques, it no longer works, or no longer works as well as other techniques.

That doesn't sound much different from what one would expect from content linked to from an indexed site.

Show me a site that went up on April 1 and got into Google by April 7 through being crawled by a reciprocal link.

It takes longer than that for Google to update the cache on most sites. I get my new content into the index faster than Google can re-index my site map and crawl the new pages from there.

I spend a good deal of time checking my logs. I see when Google fetches a page. If submitting to Google were a waste of time, I wouldn't continue doing it.

I like getting my content into the index faster than other people. But I don't mind explaining how I do it.

[Michael Martinez]

I have been not submitting new content to Google for 5 years, and all my new content gets added in that timeframe as well.

Submitting pages to the add url form doesn't seem to have any benefit as far as I can tell.  Page gets added either way.

Technically, my new content gets indexed and listed within 2-3 days if I create/submit it on Thursday or Friday. If I create the content on Monday and add it to the site map, there is a good chance it will be added by Google within the next week.

However, Jill, you and I have large content sites and we can easily cross-link to new content from pages that we know will be crawled.

I have been submitting new content directly to test the submission tool. People who have been waiting several weeks to get crawled lose nothing by submitting their pages to Google.

A 1-minute process that works (and it DOES work) is not a waste of time by any measure with which I am familiar.

Right now, a lot of people are complaining about not getting listed with Google after months of waiting for a crawl. So why shouldn't they create a new page of content and submit to Google? They have nothing to lose and everything to gain.



NOTE ON EDIT: For what it's worth, I created only one cross-promotional link to my newest Google paper on Xenite (it's on the main index). A couple of people have already linked to it because I mentioned it on the Spider-Food forum. This is as pure a test as I have available right now for seeing how long it takes GoogleBot to find the new page. The page is not yet listed in my site map, which Google tends to fetch frequently (but the index page is also frequently fetched).

The page went live on April 19 just before midnight Central Time. So far, only MSNBot and AIPBot have fetched the actual page.

GoogleBot fetched my main index page about 13-1/2 hours after the new link to the paper was added. They show the cache for the fetched index page with the link to the paper already. So, they have the link.

MSN also only shows the main index with the link. But they are still a step ahead of Google.

Edited by Michael Martinez, 22 April 2005 - 10:25 AM.

[qwerty]

If you want to submit, submit. I'm sure it does no harm.

One question, Michael: are these new pages which you're submitting orphaned, or are they linked to from pages that are getting crawled regularly? A farily simple experiment would be to submit a page that has no links pointing to it and see how long it takes to get it into the index.

[chrishirst]

Funny how I should be reading this when a Google alert for a page I created and uploaded Tuesday 19th at around 8PM was crawled by googlebot 5pm next day and added to the index today (Friday 22nd 3pm)
googlebot was on the site from 7AM through to 11pm (as most days)

Not a submission form in sight.

[Michael Martinez]

One question, Michael: are these new pages which you're submitting orphaned, or are they linked to from pages that are getting crawled regularly? A farily simple experiment would be to submit a page that has no links pointing to it and see how long it takes to get it into the index.

I debated whether to post my example as an EDIT to my previous message or followup. I'm sorry I didn't just followup.

All of the pages I submit are orphaned at the time of submission, as far as the Google index is concerned. Generally, when I create a new site, I write a news feature article about it on Xenite, add it to the appropriate site map, and if there is related content I add it to the cross-promotional link boxes on those pages. I also usually mention it on the main index.

So, in practice, almost any new content I create gets anywhere from 3 to 10 inbound links from my own network on the first day. How quickly Google finds those links is a matter of timing.

Google is currently fetching 50-100 pages per day from Xenite. For a while, it was fetching about 10 pages (that lasted only a couple of weeks) in early to mid-March. During the slow-fetch period, new content still appeared as expected, so Google was not following the links on pages.

Right now, I don't have any other projects on the boards. But I'll try to remember to do the orphaned submit test with the next new site.

[qwerty]

I've just created an orphaned page (which will remain orphaned) and submitted it to Google, Yahoo, and MSN. I'll check every day to see if any of them have indexed it, assuming I remember to do so