7 replies to this topic

[donaldcroswell]

Hello all,

I have been reading about how to make my ASP dynamic pages spidered by the Googlebot.
It seams I have to get the client to request a page that doesn't exist, then when the server goes to the 404 error message, which is now a script to write the page requested, the server will generate a page based on the URL requested.

I am wondering if there are any flaws in this technique or even if the Googlebot will follow all of the links on the page. I can see how the setup works but it seems that somehow the folks at google would only want to spider pages that exist, not pages generated by a 404 request.

Also I have this idea of a tiny spider crawling the web and finding my pages one at a time. When the spider crawls the web and finds a page with 20 links, I am assuming that each pages is requested from my server one at a time and that my server won't have an overload from 20 page requests which all have 20 links on each page, and so on.....till the bot is requesting thousands of pages at once.

Excuse my ignorance here but I know nothing about stress testing a server and what ASP and Access can handle.

Thanks for any input
Don

[Jill]

Moved your post to the technobabble category. More likely to get some responses here...

Jill

[Think Web]

404 pages can and do get spidered.

http://www.google.co...filetype:htm...

ERROR 404: Page Not Found
Error 404: Page not found !!!!! ...
www.thomas.maxwell.net/404.htm- 21k - Cached - Similar pages

[SEOCub]

Your logs will be ful of 404s. You will need to write your own logs if you want to analyze traffic in the same way as you do now.
See this post from another forum: http://www.ihelpyous...6733#post106733
That thread has an indepth disucssion of how to implement URL rewrite with ASP.

Downside of writing your own logs with Access is that the database will fill up very quickly ifyou have a reasonably trafficked web site. Think of upgrading to MySQL if you want to write your own logs and flush out the database out every other day or so.

[SEOCub]

I also thought I woudl mention that ASP pages doe get spidered by Google.
http://www.google.co...c...l:index.asp
The trick is to keep the arguments down to 2. If you need to use more than 2 arguments then append the value to another viz:
something.asp?Var=123_MyVal_XYZ
where the underscore is the separator and you really have three args with values:
123, MyVal and XYZ

[donaldcroswell]

Jill-Thanks for putting me in the right place.

Everyone-Thanks for your helpful information.

Forums like this help me realise that this planet is full of thousands of helpful and friendly people.

I'm feeling all warm and fuzzy.

Thanks again.

[Scottie]

I'm feeling all warm and fuzzy.

You're sitting too close to SEOCub...

[SEOCub]

nothing wrong with general human warmth, eh? 'tis the season for giving thanks and much cheer...just doing my bit in the larger interest of humanity.