Are you a Google Analytics enthusiast?
More SEO Content
Is Robots.txt Necessary For Unlinked File
Posted 08 July 2008 - 05:06 PM
# robots.txt for www.mysite.com
Posted 08 July 2008 - 07:13 PM
Just remember that any person can look at your robots.txt file and then go view the page in question, however. So be sure there's nothing there you don't actually want people to see.
Posted 08 July 2008 - 07:52 PM
robots.txt would keep it out of all of the nice bots. But having the file referenced in your robots.txt is going to ensure it gets picked up by the bad bots or any other scurrilous characters out there who mine robots.txt to find stuff people don't want others to see.
Posted 09 July 2008 - 08:23 AM
So let me rephrase this (you know both good and bad attorneys rephrase questions to make the answer fit...) as:
In root of the website is a standard index.php with a navigation system which does not include either my_private_file.html or perhaps some_private_folder or perhaps even .invisible.html.
Either with or without robots.txt do bad bots and/or unscrupulous programs have the ability to find these private-ish files or folders even though they are not identified anywhere in navigation and there is an index file in place?
Posted 09 July 2008 - 09:23 AM
I wouldn't name it anything obvious. eg hidden.html, invisible.html, setup.php, admin.php.
How about another obfuscation idea. It's not perfect and not secure but you might like it if security isn't a huge issue.
Set up your file and name it something a little wonky. Maybe even put a number in it so the rogue bots wouldn't stumble across it easily.
Then in the HTML code itself include a meta robots tag with a value of noindex, nofollow, noarchive. In theory if the bad bots don't find your page because of the odd filename that's good. And if a good bot does happen to stumble across it the meta robots tag will tell them to stay away. Just like robots.txt would, cept you're not putting it out there for the bad bots to find because there is no robots.txt entry.
Posted 09 July 2008 - 04:43 PM
People can't browse the directory so they can't find the file, and the directory and everything in it gets excluded via robots.txt.
Posted 09 July 2008 - 07:29 PM
Me too. 5 months ago, I put 4 PDF files in a directory and excluded this dir via robots.txt. A couple weeks ago I did a "site" search for .pdf files and the 4 PDF files came up in the return.
So, is this a hint that excluding a dir with robots.txt doesn't mean much nowadays?
Posted 09 July 2008 - 08:14 PM
Big G does honor this tag, I believe.
Posted 09 July 2008 - 10:57 PM
Really? I'd have to see that with my own eyes to believe it. If you're talking about Google, they're generally good at obeying robots.txt but I haven't checked all that closely lately.
Posted 09 July 2008 - 11:07 PM
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users