[ask] why there are people visit my wp site robot.txt?
- Thread starter rockoki
- Start date
Some people will put sensitive directories in there that they don't want crawled. By visiting robots.txt files you can find some interesting stuff sometimes.
yeah its ironic since those are the stuff you want to hide from the public search, and yet its so easily accessed through a plain .txt file
errrr.. would have been funny if you said
"sure which visa card are you talking about? whats the card number?"
back on topic, robots.txt files are alright, i doubt anyone would actually put any info on there that could harm them. then again there are stupid peoples out there.
User Agent: *
Disallow: /members/photos
Disallow: /members/passwords/
You're probs never gonna see something like that. More like boring shit. like
/photos/, cos you dont want them appear on google image results and other fuckers nicking your pictures and putting them on their blogs or something
and it's not like this info is any good anyways, just disallow directory listing, or put an empty index.html file in there.. voila/
emmm, are you sire it's people and not bots. there's the obvious ones like googe, bing, yahoo. but there are also hundreds of crappy SE out there, that even I [with 4 years experience in seo] cannot name,