Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Accelerate mimetypes.init on Windows #88748

Closed
zooba opened this issue Jul 7, 2021 · 6 comments
Closed

Accelerate mimetypes.init on Windows #88748

zooba opened this issue Jul 7, 2021 · 6 comments
Assignees
@zooba
Labels
3.10 only security fixes 3.11 only security fixes OS-windows performance Performance or resource usage

Comments

@zooba
Copy link
Member

zooba commented Jul 7, 2021

BPO 44582
Nosy @pfmoore, @tjguk, @zware, @zooba, @miss-islington
PRs
  • bpo-44582: Accelerate mimetypes.init on Windows with a native accelerator #27059
  • [3.10] bpo-44582: Accelerate mimetypes.init on Windows with a native accelerator (GH-27059) #27071
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/zooba'
closed_at = <Date 2021-07-08.17:03:31.718>
created_at = <Date 2021-07-07.20:06:39.588>
labels = ['3.11', '3.10', 'OS-windows', 'performance']
title = 'Accelerate mimetypes.init on Windows'
updated_at = <Date 2021-07-08.17:03:31.718>
user = 'https://github.com/zooba'

    bugs.python.org fields:

    activity = <Date 2021-07-08.17:03:31.718>
actor = 'steve.dower'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2021-07-08.17:03:31.718>
closer = 'steve.dower'
components = ['Windows']
creation = <Date 2021-07-07.20:06:39.588>
creator = 'steve.dower'
dependencies = []
files = []
hgrepos = []
issue_num = 44582
keywords = ['patch']
message_count = 6.0
messages = ['397110', '397113', '397114', '397146', '397147', '397157']
nosy_count = 5.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'steve.dower', 'miss-islington']
pr_nums = ['27059', '27071']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue44582'
versions = ['Python 3.10', 'Python 3.11']
    @zooba
    Copy link
    Member Author

    zooba commented Jul 7, 2021

    mimetypes.init is slow on Windows because of the big registry search, which involves touching thousands of entries in order to add a few hundred into the initial table.

    Things get even worse when audit hooks are enabled, because the winreg methods need to be hooked. However, we know that this particular operation is coming from core, and so we could skip these hooks.

    Both issues can be trivially solved (helped) with a native accelerator function.

    @zooba zooba added 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes labels Jul 7, 2021
    @zooba zooba self-assigned this Jul 7, 2021
    @zooba zooba added performance Performance or resource usage 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes labels Jul 7, 2021
    @zooba zooba self-assigned this Jul 7, 2021
    @zooba zooba added performance Performance or resource usage OS-windows labels Jul 7, 2021
    @zooba
    Copy link
    Member Author

    zooba commented Jul 7, 2021

    It's only a rough benchmark, but my timings for python -c "import mimetypes; mimetypes.init() were ~200ms without the accelerator and ~100ms with the accelerator.

    Since that includes all of startup, it's actually a better than 50% reduction for mimetypes.init(). I'm happy to take that without questioning precisely how much better it is ;)

    @zooba
    Copy link
    Member Author

    zooba commented Jul 7, 2021

    mimetypes doesn't really support clearing, but python.exe -m timeit "import mimetypes as M; M.types_map.clear(); M.init()" is pretty close.

    Without the accelerator: 133ms
    With the accelerator: 27.8ms

    @zooba
    Copy link
    Member Author

    zooba commented Jul 8, 2021

    New changeset bbf2fb6 by Steve Dower in branch 'main':
    bpo-44582: Accelerate mimetypes.init on Windows with a native accelerator (GH-27059)
    bbf2fb6

    @miss-islington
    Copy link
    Contributor

    New changeset 08697ac by Miss Islington (bot) in branch '3.10':
    bpo-44582: Accelerate mimetypes.init on Windows with a native accelerator (GH-27059)
    08697ac

    @zooba
    Copy link
    Member Author

    zooba commented Jul 8, 2021

    Dropping the 3.9 backport. Pretty sure it only needs a clinic regen, but it's also not worth the risk of changed behaviour.

    We could consider it again after 3.10 has shipped if people really want it (and can help validate it).

    @zooba zooba removed the 3.9 only security fixes label Jul 8, 2021
    @zooba zooba closed this as completed Jul 8, 2021
    @zooba zooba removed the 3.9 only security fixes label Jul 8, 2021
    @zooba zooba closed this as completed Jul 8, 2021
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @zooba zooba

    Labels
    3.10 only security fixes 3.11 only security fixes OS-windows performance Performance or resource usage
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @zooba @miss-islington