robots.txt file permissions

[mitchb]

From ​RT ticket 883503:

I was scrolling through error_log... We don't consider .txt a "safe" extension. I can't decide whether or not I agree, but it's certainly wrong in the case of robots.txt. Can we make "robots.txt" a safe filename?

This is probably not entirely trivial on static-cat's side, since IIRC it extracts the extension, but also not more involved than a special case for this filename.

-- Geoffrey Thomas geofft@…

[ezyang]

The fact that .txt is not a "safe" extension is absolutely bogus: there is absolutely no expectation on the user end that you can write to a .txt file and then expect it to NOT be shown by Apache without twiddling permissions via .htaccess. We should fix this by making .txt a safe extension.

Geofft has some concerns about changing security properties this late in the game, but I think this is a clear cut case of something we should display by default.

[mitchb]

I think the concern here was that some programs might use something like config.txt as a settings file, and spitting that out on the web would be a compromise of that program. Whether that was a valid concern or not, the ship has sailed. We cannot add txt to the list of safe extensions now; we have 2600 users, any number of which may have verified that their .txt files are safe from viewing on the web. We can't yank the rug out from under them; we don't even know who's done this.

Regardless of what you'd expect with a normal web server, remember that many of our users have no clue how a web server works. And if they checked to make sure that their file wasn't visible on the web, that's a completely reasonable thing to expect not to change.

If Geoff wants to special-case robots.txt, I think that's fine. And it's what this ticket was about. If you want to discuss .txt files in general, let's do it in another ticket or on scripts-team, but I don't think it'll fly.

[adehnert]

Anybody who is (mysteriously) motivated to see rehashings of blacklist vs. whitelist and "Can we allow .txt" should go look at:

• -c scripts -i sipb, on 2011-03-21
• -c scripts -i 1685762, on 2011-08-03

There are likely others, but I'm not finding them quickly. (This comment largely added for convenience next time we decide to wank about this, so we can conveniently reference our arguments and counterarguments by copying and pasting instead of re-typing them each time.)

[ezyang]

Bumping priority, we should announce it, do it, and close this sucker.