How Does A Se Access Dynamic Content?, what exactly doe the SE read on a dynamically genated page? Options

[stepheneighmey]

i've heard that search engines don't actually go into the database to read the information generated by a dynamically generated link but rather they read the contents of the html page that is generated by this URL. How do they do this exactly, do they "see" the html page when it is quickly brought up in a browser by a user? any insight would be appreciated, thank you.

[projectphp]

Man, you got a lot of confused ideas going on. You need to start again completely with how you think about this.

SEs have robots that crawl the web viewing pages. It has nothing to do with users, browsers or databases. http://en.wikipedia.org/wiki/Web_crawler is a good start. Have a read and come back and ask questions.

This post has been edited by projectphp: Mar 28 2007, 11:00 AM

[stepheneighmey]

Perhaps my question was unfocused, I do realize that search bots have nothing to do with users, browsers or databases. I was wondering how content pulled from a search query on a website were indexed by a search engine. I read the Wikipedia article and it helped a bit and perhaps even answered my question. This forum (and Jill) has been a great resource in the past so I thought I'd try here first. Thanks for pointing me to the article.

[projectphp]

Huh? Still not being clear (IMG:style_emoticons/default/smile.gif)

SEs don't search, or fill in forms, or parse javascript. Does that help?

[madams]

I think stefh means...

A spider sees a htm page in a cirtain way. On page text and off page code and judges the page (including many other factors) accordinly.

As I see it, the question is...
What do the spiders see, if anything, for a dynamically generated page. Do they go rummiging around the database putting together the page parts that "people" see.

(Not highjacking, just my take on the question. I am interested in the answer)

[smc_online]

Try this tool: http://www.seo-browser.com/

I find it helpful in looking at a page the way a SE would. Enjoy!

Sorry! I forgot to point out that a SE does not see it the way a person sees it but at the same time it does. Meaning, it doesn't matter where the data on the page is coming from (DB or whatever). It sees the on page content but without all the pretty pictures and stuff.

Does this help?

[chrishirst]

There is absolutely nothing different about a page that is "static HTML" delivered from a server hard drive or a page that is generated "dynamically" by of collection of scripts.

[Martin C]

they read the contents of the html page that is generated by this URL.

They do it in the same way as a human visiting the website does.

They click on an item, the data is retrieve from the database and displayed to the screen using HTML code. It is the HTML code that displays the data to the screen that is used by the search engine.

Each dynamic display will generate a URL

[Jill]

I was wondering how content pulled from a search query on a website were indexed by a search engine.

They can't and they don't.

That content will not be seen by the search engine if it can only be found through a search query on a website.

The way to make that content accessible, is to create hard-coded links to that content. So, for instance, you could do the query on the website, and then add the resulting URL to your main navigation as a link.

For instance, say you did a search at your site for red lederhosen. The resulting page would be a nice one (presumably) with many types of red lederhosen and would make a great red lederhosen page for other people who may not have thought to search for it. So you could add that page (the URL) to your navigation, and now you basically have a static red lederhosen page on your dynamic site!

Does that help clarify things for you a bit?

It is a very hard topic to wrap your head around at first. And it appears that ProjectPHP (being a PHP specialist and all) gets it so well that he forgets that we mere mortals don't always understand how dynamically generated sites work!

[stepheneighmey]

okay. well here is an example of my question... i have a page of dynamically generated content that is pulled up on my site through a search function. the product number is SAF-2000523. the only way this product can be brought up on the site is by either putting this exact term in the site's search function, or clicking on this item as it comes up in a page of results (that is from a hard coded link) for the term "safety". there is no hard coded link on my site to this specific product, yet, if i type SAF-2000523 into google:

http://www.google.com/search?hl=en&q=S...G=Google+Search

i get exactly one result returned and it is my exact product. how does google "see" this product since there is no hard coded link to this product on my site? though, as i mentioned, there is a hard coded link to this particular section of the site, just not this specific product. i hope this makes a bit more sense.

[Randy]

Check the Google cache of the page. You'll see this text:

These terms only appear in links pointing to this page: saf 2000523 saf2000523

Which probably means there's a link out there somewhere pointing to the page. It doesn't have to be a link on your own site. I could be a forum post or a link on any other site pointing to your search results page.

[stepheneighmey]

does that mean there has to be a link somewhere on the web pointing to this url? i would actually be surprised if there was but i imagine it's possible. there is definitely not a link directly to this item on my site.

[lyn]

All it would really take is for someone on a forum or in a blog to mention your carabiner and link to the web page.
Not sure (like you (IMG:style_emoticons/default/searchme.gif) ) why someone would attach that kind of importance to a 'biner but it must have happened or Google couldn't have found the link to it!
Do you have a lot of similar hardware items appearing in Google results?

L.

[Randy]

Given a couple of things we know about how spiders operate and what you've said it would almost have to be a link on some other site linking to your internal search results page Stephen.

1. We know that spiders don't submit forms. If they did everybody would see all sorts of incomplete orders in their database that were begin started by the spiders.
2. We know that spiders find pages by looking for links in the code of pages they grab from the Web. Technically they don't click and crawl the way we people do. They go to a single page, grab the html code, parse through it looking for links, then go back out to the new pages they found.

Since they can only find pages by finding links; and since you've confirmed that you don't link to the dynamic search page at all; the only thing left to consider is that someone else is in fact linking to that page from somewhere.

[zammy]

does that mean there has to be a link somewhere on the web pointing to this url? i would actually be surprised if there was but i imagine it's possible. there is definitely not a link directly to this item on my site.

Stepheneighmey, There are many sources from which Google can find your pages. Most obvious ones are like those already been previously suggested, but one that has not been suggested is through the Google Toolbar. If you or your users have it installed on their browsers, chances are, Google found those "hidden" pages through that data goldmine.