Are you a Google Analytics enthusiast?
More SEO Content
Re-spidering An Existing Page
Posted 04 September 2003 - 11:16 AM
I have a question regarding page spidering. Once a page has been spidered will google continue to re-spider the page, or will the page be re-spidered only if the content (page size) has changed ? If its based on page size changes, what percentage of change is necessary. If re-spidering is based on other factors please explain.
Posted 04 September 2003 - 11:23 AM
This can depend on your server and how its set up, and how big and how popular your site is.
The first problem is to get the spider to visit your page, and he will tend to visit higher PR sites more often, likes to spider html in preference to dynamic pages, and will usually only take so many pages on each visit.
But once he does come calling Googlebot uses the if-modifies-since query to ask the server if the page has been modified since it last visted, and if the server is set up right it will wither return a 304 not modified response if the page has not changed or if it has will return the page.
Posted 04 September 2003 - 11:40 AM
Our site has a page rank value of 5 and 6. Google does come on a regular basis. What I have found is that there are certain pages that are re-indexed on every visit. What I'm trying for - is to have other pages in that re-occurrirng loop. The additional pages that aren't in that loop have the same page rank as the re-occuring pages. This is why I was wondering if page modification was a determining factor, and if so what percent of page change triggered the re-indexing flag.
Posted 04 September 2003 - 03:29 PM
The server is IIS/5.0. I went to the link you gave, and I didn't see anything refferencing change date or a modification flag / date field.
Posted 05 September 2003 - 12:03 AM
The site is a dynamic .com site. I'll read the article and see what's in store for me.
Thanks for the help...
Posted 05 September 2003 - 02:46 AM
Another thing to check for is to see if your site is returning lots of 304 responses, since some dynamic pages will serve up a 304 every time the page is called.
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users