classification
Title: Brotli encoding is not recognized by mimetypes
Type: enhancement Stage:
Components: Library (Lib) Versions: Python 3.7, Python 3.6, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Andrey, christian.heimes, r.david.murray
Priority: normal Keywords:

Created on 2017-11-14 07:43 by Andrey, last changed 2017-11-15 18:19 by r.david.murray.

Messages (11)
msg306188 - (view) Author: Andrey (Andrey) Date: 2017-11-14 07:43
Brotli (.br) encoding is not recognized by mimetypes module.
mimetypes doesn't have API for adding encodings.
The encoding is supported by most browsers: https://caniuse.com/#feat=brotli
msg306189 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2017-11-14 08:43
You can just add the encoding to mimetypes.encodings_map:

mimetypes.encodings_map['.br'] = 'brotli'
msg306195 - (view) Author: Andrey (Andrey) Date: 2017-11-14 09:49
Yes, however would it make sense to add it, so mimetypes recognizes it by default? I will make a PR if it makes sense.
msg306203 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-11-14 12:59
It looks like Brotli is a de-facto standard (RFC 7932 is an informational RFC), and it says the IANA registry has been updated.  However, this appears to be http-only, with no corresponding program used for compression of files, and thus no "file extension" to be registered, or program name to put in the encodings_map table.  We don't have most of the types listed in the http content encodings registry in mimetypes, so I don't at this time see any reason to add br.

mimetypes is aimed at files, not http.  In what context do you want to use mimetimes to look up br?
msg306215 - (view) Author: Andrey (Andrey) Date: 2017-11-14 14:34
When "type, encoding = mimetypes.guess_type('file.js.br')" is used not only the "encoding" is not set to "brotli", but the type is also not recognized.
Such code is used in aiohttp for example.

My proposal is to add 'br' to mimetypes.encodings_map, not to mimetypes.types_map
msg306221 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-11-14 16:29
OK, so there are "real" filenames that have .br on the end?  That wasn't clear from the RFC and discussions I found.  Based on the documentation, br is for use in the Content-Encoding header.

What does aiohttp do when it gets a brotli encoded file?  What does it do with the other types in the http content-encoding table? (https://www.iana.org/assignments/http-parameters/http-parameters.xhtml#content-coding)

Where is the standard that talks about the '.br' file extension?
msg306222 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-11-14 16:31
Note, I'm not saying we shouldn't do this, I'd just like to get good documentation of why linked to this issue.
msg306226 - (view) Author: Andrey (Andrey) Date: 2017-11-14 17:20
Currently aiohttp doesn't support Brotli. (I'm here because I would like to get that support)

When it gets (as a server) a request for file.html it checks if file.html.gz exists (and accept-encoding contains 'gzip'). It then send file.html.gz to the client and sets Content-Encoding to gzip.

It uses mimetypes.guess_type to set Content-Type and Content-Encoding.
msg306239 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-11-14 21:58
Hmm. OK, the docs do say that value should be usable as a content-encoding header.

However, absent any sort of program that actually does brotli encoding/decoding to and from files, this would be essentially useless based on the aiohttp's algorithm (or any other use I can think of for the mimetimes module in this context).  Does such a program exist?  I didn't find one searching the web.
msg306251 - (view) Author: Andrey (Andrey) Date: 2017-11-15 05:42
NodeJS: https://www.npmjs.com/package/brotli
Python:
https://pypi.python.org/pypi/brotlipy
https://pypi.python.org/pypi/Brotli
Ubuntu: https://launchpad.net/ubuntu/+source/brotli

That said I don't expect an average user to compress their files as .br, but it is useful for site/webapp developers to have static resources as .br
msg306293 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-11-15 18:19
OK, bouncing around from those URL I finally found this:

https://github.com/google/ngx_brotli/issues/28

That indicates unambiguously that someone does expect files to be pre-compressed on the server and that the main brotli package provides a cli tool (I could not find any reference to that in the brotli docs themselves, such as they are).

So, I think it is appropriate to add this, and one can hope that the program name is indeed 'brotli' so that it matches up with the existing mimetypes docs :)
History
Date User Action Args
2017-11-15 18:19:31r.david.murraysetmessages: + msg306293
2017-11-15 05:42:46Andreysetmessages: + msg306251
2017-11-14 21:58:00r.david.murraysetmessages: + msg306239
2017-11-14 17:20:39Andreysetmessages: + msg306226
2017-11-14 16:31:15r.david.murraysetmessages: + msg306222
2017-11-14 16:29:26r.david.murraysetmessages: + msg306221
2017-11-14 14:34:03Andreysetmessages: + msg306215
2017-11-14 12:59:05r.david.murraysetnosy: + r.david.murray
messages: + msg306203
2017-11-14 09:49:30Andreysetmessages: + msg306195
2017-11-14 08:43:21christian.heimessetnosy: + christian.heimes

messages: + msg306189
versions: - Python 3.4, Python 3.5, Python 3.8
2017-11-14 07:43:50Andreycreate