I posted a bit about this earlier. I am working on a site that is being indexed by google no problem- but yahoo, msn, etc just the main page is being indexed- which leads me to believe it is a technical problem. Any thoughts??
thanks!
hobby
Subscribe to HRA Now!
SEO Class in Chicago, IL
Learn How To Optimize Your Website on July 26, 2013
Looking for personalized in-depth SEO training among your peers?
High Rankings is offering a 1-day customized SEO training class in Chicago. Class size is limited so please sign-up now if you want in!
Are you a Google Analytics enthusiast?
Share and download Custom Google Analytics Reports, dashboards and advanced segments--for FREE!
www.CustomReportSharing.com
From the folks who brought you High Rankings!
More SEO Content
International SEM | Social Media | Search Friendly Design | SEO | Paid Search / PPC | Seminars | Forum Threads | Q&A | Copywriting | Keyword Research | Web Analytics / Conversions | Blogging | Dynamic Sites | Linking | SEO Services | Site Architecture | Search Engine Spam | Wrap-ups | Business Issues | HRA Questions | Online Courses
Site Not Being Indexed By Most Search Engines?
Started by
hobby
, Nov 21 2004 09:27 AM
5 replies to this topic
#2
qwerty
-
- Moderator
-
- 8,323 posts
HR 10
- Location:Somerville, MA
Posted 21 November 2004 - 10:06 AM
What are the log files telling you? Are Slurp and MSNBot requesting pages other than the home page? If so, what response are they getting?
#3
hobby
-
- Active Members
-
- 27 posts
HR 2
Posted 21 November 2004 - 11:44 AM
Bob- I am new at this and just getting into analysing logs. I am embarrassed to say I don;t know how to look for this in logs. I do have all the raw logs for this site.
If there is somewhere on board that breaks down this process I can learn how to do.
Thanks!
hobby
If there is somewhere on board that breaks down this process I can learn how to do.
Thanks!
hobby
#4
qwerty
-
- Moderator
-
- 8,323 posts
HR 10
- Location:Somerville, MA
Posted 21 November 2004 - 02:02 PM
It depends on what you're using to analyze the files. Most software that's designed for this is going to give you a report on which spiders came in, and many will get into more specifics, telling you what they requested, when, and whether it was successful.
But if you're just reading the files in a text editor, just do a search on the name of the spider, like Slurp.
For example, here are a couple of lines from my logs:
In the first line, Slurp is requesting /about/ and it gets a 302 (temporarily moved) response, redirecting it to the default file for that directory, /about/index.htm, for which it gets a 200 (OK) response.
But if you're just reading the files in a text editor, just do a search on the name of the spider, like Slurp.
For example, here are a couple of lines from my logs:
CODE
2004-03-09 10:32:22 66.196.72.64 - 66.155.40.24 80 GET /about/ - 302 303 185 Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysearch/slurp)
2004-03-09 10:33:51 66.196.72.64 - 66.155.40.24 80 GET /about/index.htm - 200 6648 186 Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysearch/slurp)
2004-03-09 10:33:51 66.196.72.64 - 66.155.40.24 80 GET /about/index.htm - 200 6648 186 Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysearch/slurp)
In the first line, Slurp is requesting /about/ and it gets a 302 (temporarily moved) response, redirecting it to the default file for that directory, /about/index.htm, for which it gets a 200 (OK) response.
#5
hobby
-
- Active Members
-
- 27 posts
HR 2
Posted 29 November 2004 - 10:46 AM
Just an update- I think I have found problem. You can only browse the site with cookies enabled and must have session ID. So.... though google can read - none of the other major search engines bots can get through.
Is there way to create an exception for bots? Spot if the user agent is one of a common bot, stop the deman for a cookie?
Note: not sure if I should re-ask in technology/coding section..
Thanks.
hobby
Is there way to create an exception for bots? Spot if the user agent is one of a common bot, stop the deman for a cookie?
Note: not sure if I should re-ask in technology/coding section..
Thanks.
hobby
#6
Randy
-
- Moderator
-
- 17,540 posts
Convert Me!
Posted 29 November 2004 - 11:33 AM
That'll do it every time hobby. Good find!
If it's not absolutely necessary to have the Cookie/SID set to track everything, you can simply Request rather than Require both. How you do that is going to depend a bit upon which scripting language is being used. Usually that's going to be a very small change in the coding of your site.
FWIW, for most e-commerce sites there is no reason to Require a cookie or session id until they add something to their shopping cart. That's when you need to start tracking.
If it's not absolutely necessary to have the Cookie/SID set to track everything, you can simply Request rather than Require both. How you do that is going to depend a bit upon which scripting language is being used. Usually that's going to be a very small change in the coding of your site.
FWIW, for most e-commerce sites there is no reason to Require a cookie or session id until they add something to their shopping cart. That's when you need to start tracking.
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users
