Are you a Google Analytics enthusiast?
More SEO Content
My Site Is Based On X-cart!
Posted 05 April 2008 - 04:33 AM
I prepared very first time a robot.txt file today, I am a little bit afraid to upload it,
My site is based on X-cart software, we generate html page by catalog option.
1-I disallowed all php Pages for duplicating purpose,
2-i disallowed all ftp folders but images folder.
Is there anything important to disallow or I disallowed any important thing that I shouldn’t?
Posted 05 April 2008 - 08:55 AM
For instance, if your dynamic cart is installed in a subdirectory one level down from root named /cart/ and your html version is in a different subdirecotry you could simply put the following in your robots.txt file
If the dynamic cart is installed at the root level of your domain with the html version in a subdirectory you can simply exclude the cart.php page since all other x-cart pages feed off of it. That would look something like
Posted 05 April 2008 - 09:07 AM
You may want to paste in here how you're disallowing those, just in case.
Posted 07 April 2008 - 04:27 AM
I disallowed all php pages one by one,
If I just disallow: cart.php
Is it enough for all php pages?
Thanks once again.
Posted 07 April 2008 - 08:06 AM
For instance, with one x-cart site I watch over the dynamic store is located at www.domain.com/store/ with the html catalog located at www.domain.com/store/catalog. The dynamic side of things uses a file called home.php in the /store/ subdirectory for many pages. So to exclude those via robots.txt my instruction would look like:
That alone would get all of the base pages and category pages since x-cart shows cat pages like www.domain.com/store/home.php?cat=124
Additionally, when I drill down through the dynamic side of things to the individual product pages I see that x-cart uses a different php file named product.php followed by some variables and values. So to block all of the pages with a single robots.txt exclusion I would change my robots.txt to read:
To take it a bit further, I have several extra pages set up for things like Ordering Information, Return Policy, Shipping Info and so on. The html converter makes these pages too, so I'll probably want to exclude the dynamic version. In my x-cart installation these dynamic pages are called via www.domain.com/store/pages.php?pageid=# So to exclude these pages also I'd change my robots.txt to read:
So with just three lines I've manage to exclude the entire dynamic side of my store, forcing everything to go through my html catalog version first. If I wanted to I could also exclude the Search page, which is located at /store/search.php in my case, or since that's the page my html catalog points to also I can leave it be. The search engne spiders aren't going to go past it anyway since it uses an html form to perform a site search. For these reasons I've chosen not to exclude the search page, though I could.
Basically have a look around the dynamic side of your site and make note of the path (eg /store/ in my case) and filenames being used. Jot them down as you're surfing around. Once you have the exact files and locations being used to produce the dynamic pages you can easily restrict the spiders from seeing or using them.
Posted 07 April 2008 - 10:44 AM
but yet I am thinking about other folders…..
There are many folders in my sub directory and as my little knowledge I disallow almost all folders excluding images folder,
Please say something about it Sir,
Posted 07 April 2008 - 05:08 PM
x-cart and many shopping carts automatically set up a bunch of subdirectories because that's how they keep things at least a little bit neat and tidy. However there is no need to exclude any of these because the search engines will never even see that they exist. The files in those folders are dynamically included in other files, but there is no direct link to any of them in the resulting html code.
Anything that stands a chance of getting linked to that you don't want the engines to index, feel free to exclude. Honestly, that usually won't be many folders for the average site.
However don't think excluding a subdirectory via robots.txt is anything close to a security measure, because it's not. Quite the opposite in fact, since there are bad bots out there that specifically look for Disallowed folders and files to see if they can use them as a way to break into a site. Robots.txt is not a form of site security.
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users