Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Brotli encoding is not recognized by mimetypes #76202

Closed
Andrey mannequin opened this issue Nov 14, 2017 · 13 comments
Closed

Brotli encoding is not recognized by mimetypes #76202

Andrey mannequin opened this issue Nov 14, 2017 · 13 comments
Labels
3.9 only security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@Andrey
Copy link
Mannequin

Andrey mannequin commented Nov 14, 2017

BPO 32021
Nosy @tiran, @bitdancer, @maxking, @csabella
PRs
  • bpo-32021: Support brotli .br encoding in mimetypes #12200
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2020-01-12.22:33:29.295>
    created_at = <Date 2017-11-14.07:43:50.631>
    labels = ['type-feature', 'library', '3.9']
    title = 'Brotli encoding is not recognized by mimetypes'
    updated_at = <Date 2020-01-12.22:33:29.293>
    user = 'https://bugs.python.org/Andrey'

    bugs.python.org fields:

    activity = <Date 2020-01-12.22:33:29.293>
    actor = 'maxking'
    assignee = 'none'
    closed = True
    closed_date = <Date 2020-01-12.22:33:29.295>
    closer = 'maxking'
    components = ['Library (Lib)']
    creation = <Date 2017-11-14.07:43:50.631>
    creator = 'Andrey'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 32021
    keywords = ['patch']
    message_count = 13.0
    messages = ['306188', '306189', '306195', '306203', '306215', '306221', '306222', '306226', '306239', '306251', '306293', '359766', '359880']
    nosy_count = 5.0
    nosy_names = ['christian.heimes', 'r.david.murray', 'maxking', 'cheryl.sabella', 'Andrey']
    pr_nums = ['12200']
    priority = 'normal'
    resolution = None
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue32021'
    versions = ['Python 3.9']

    @Andrey
    Copy link
    Mannequin Author

    Andrey mannequin commented Nov 14, 2017

    Brotli (.br) encoding is not recognized by mimetypes module.
    mimetypes doesn't have API for adding encodings.
    The encoding is supported by most browsers: https://caniuse.com/#feat=brotli

    @Andrey Andrey mannequin added 3.8 only security fixes 3.7 (EOL) end of life stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels Nov 14, 2017
    @tiran
    Copy link
    Member

    tiran commented Nov 14, 2017

    You can just add the encoding to mimetypes.encodings_map:

    mimetypes.encodings_map['.br'] = 'brotli'

    @tiran tiran removed the 3.8 only security fixes label Nov 14, 2017
    @Andrey
    Copy link
    Mannequin Author

    Andrey mannequin commented Nov 14, 2017

    Yes, however would it make sense to add it, so mimetypes recognizes it by default? I will make a PR if it makes sense.

    @bitdancer
    Copy link
    Member

    It looks like Brotli is a de-facto standard (RFC 7932 is an informational RFC), and it says the IANA registry has been updated. However, this appears to be http-only, with no corresponding program used for compression of files, and thus no "file extension" to be registered, or program name to put in the encodings_map table. We don't have most of the types listed in the http content encodings registry in mimetypes, so I don't at this time see any reason to add br.

    mimetypes is aimed at files, not http. In what context do you want to use mimetimes to look up br?

    @Andrey
    Copy link
    Mannequin Author

    Andrey mannequin commented Nov 14, 2017

    When "type, encoding = mimetypes.guess_type('file.js.br')" is used not only the "encoding" is not set to "brotli", but the type is also not recognized.
    Such code is used in aiohttp for example.

    My proposal is to add 'br' to mimetypes.encodings_map, not to mimetypes.types_map

    @bitdancer
    Copy link
    Member

    OK, so there are "real" filenames that have .br on the end? That wasn't clear from the RFC and discussions I found. Based on the documentation, br is for use in the Content-Encoding header.

    What does aiohttp do when it gets a brotli encoded file? What does it do with the other types in the http content-encoding table? (https://www.iana.org/assignments/http-parameters/http-parameters.xhtml#content-coding)

    Where is the standard that talks about the '.br' file extension?

    @bitdancer
    Copy link
    Member

    Note, I'm not saying we shouldn't do this, I'd just like to get good documentation of why linked to this issue.

    @Andrey
    Copy link
    Mannequin Author

    Andrey mannequin commented Nov 14, 2017

    Currently aiohttp doesn't support Brotli. (I'm here because I would like to get that support)

    When it gets (as a server) a request for file.html it checks if file.html.gz exists (and accept-encoding contains 'gzip'). It then send file.html.gz to the client and sets Content-Encoding to gzip.

    It uses mimetypes.guess_type to set Content-Type and Content-Encoding.

    @bitdancer
    Copy link
    Member

    Hmm. OK, the docs do say that value should be usable as a content-encoding header.

    However, absent any sort of program that actually does brotli encoding/decoding to and from files, this would be essentially useless based on the aiohttp's algorithm (or any other use I can think of for the mimetimes module in this context). Does such a program exist? I didn't find one searching the web.

    @Andrey
    Copy link
    Mannequin Author

    Andrey mannequin commented Nov 15, 2017

    NodeJS: https://www.npmjs.com/package/brotli
    Python:
    https://pypi.python.org/pypi/brotlipy
    https://pypi.python.org/pypi/Brotli
    Ubuntu: https://launchpad.net/ubuntu/+source/brotli

    That said I don't expect an average user to compress their files as .br, but it is useful for site/webapp developers to have static resources as .br

    @bitdancer
    Copy link
    Member

    OK, bouncing around from those URL I finally found this:

    google/ngx_brotli#28

    That indicates unambiguously that someone does expect files to be pre-compressed on the server and that the main brotli package provides a cli tool (I could not find any reference to that in the brotli docs themselves, such as they are).

    So, I think it is appropriate to add this, and one can hope that the program name is indeed 'brotli' so that it matches up with the existing mimetypes docs :)

    @csabella
    Copy link
    Contributor

    Now that two years have passed, is there any additional info as to whether this should be added or not? It does seem that the format is active (https://github.com/google/brotli).

    @csabella csabella added 3.8 only security fixes 3.9 only security fixes and removed 3.7 (EOL) end of life labels Jan 10, 2020
    @maxking
    Copy link
    Contributor

    maxking commented Jan 12, 2020

    New changeset b2b4a51 by Abhilash Raj (Philip McMahon) in branch 'master':
    bpo-32021: Support brotli .br encoding in mimetypes (bpo-12200)
    b2b4a51

    @maxking maxking removed the 3.8 only security fixes label Jan 12, 2020
    @maxking maxking closed this as completed Jan 12, 2020
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.9 only security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants