Robots.txt Update
I turfed my robots.txt file, however thanks to the Belchfire.net member who emailed me and pointed out a formatting error. Why did I delete it? Because I was unable to confirm the precise formatting of the file. If it was written in stone, and I could be assured of 100% compliance by all search engines, I would have just made the necessary changes and left it in place. But since I couldn’t, I figured, ‘What’s the point?’. In addition, a robots.txt file can be just an ad for what you don’t what indexed. So now you’ve just told everyone where the stuff is you want kept secret.
A better method is to not publish what you don’t want known and then use other methods to restrict/block access to those files and folders such as .htaccess files or code within your pages to restrict access by IP or user-agent. Since I’ve always done this anyway the robots.txt file was redundance for me in this regard, so I saw no reason to maintain it. Lack of a robots.txt file will not effect your web site’s ability to be indexed by search engines.