More Robots.txt Trouble
Huh. Well I donno what’s going on but no matter what I do with my robots.txt file, even after validating it (automatically and manually), and editing it to correct some formatting errors, Yahoo still reads it but won’t index any pages. Strange thing is I haven’t touched my robots.txt file in years. So until I can figure this out, I’ve renamed my robots.txt file to _robots.txt so search spiders don’t find it. As of right now after renaming it, Yahoo is now indexing my sites.
I can tell by looking at my web logs. My web logs indicated that the only request Yahoo (Slurp) made was for the robots.txt file (about three times an hour). The moment I renamed it, Yahoo immediately started indexing files. So weird…