Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Thread locks in zlib module may go wrong in rare case #85901

Closed
animalize mannequin opened this issue Sep 7, 2020 · 9 comments
Closed

Thread locks in zlib module may go wrong in rare case #85901

animalize mannequin opened this issue Sep 7, 2020 · 9 comments
Assignees
@gpshead
Labels
3.8 only security fixes 3.9 only security fixes 3.10 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@animalize
Copy link
Mannequin

animalize mannequin commented Sep 7, 2020

BPO 41735
Nosy @gpshead, @ambv, @animalize, @miss-islington
PRs
  • bpo-41735: Fix thread locks in zlib module may go wrong in rare case. #22126
  • [3.9] bpo-41735: Fix thread locks in zlib module may go wrong in rare case #22130
  • [3.8] bpo-41735: Fix thread locks in zlib module may go wrong in rare case #22132
  • bpo-41735: Fix thread lock in zlib.Decompress.flush() may go wrong #29587
  • [3.9] bpo-41735: Fix thread lock in zlib.Decompress.flush() may go wrong #29588
  • [3.10] bpo-41735: Fix thread lock in zlib.Decompress.flush() may go wrong (GH-29587) #29811
  • [3.8] [3.9] bpo-41735: Fix thread lock in zlib.Decompress.flush() may go wrong (GH-29588) #29812
    		•
    Files
  • test_flush.py
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/gpshead'
closed_at = <Date 2021-11-27.00:24:26.190>
created_at = <Date 2020-09-07.03:14:46.922>
labels = ['3.8', 'type-bug', 'library', '3.9', '3.10']
title = 'Thread locks in zlib module may go wrong in rare case'
updated_at = <Date 2021-11-27.04:14:33.454>
user = 'https://github.com/animalize'

    bugs.python.org fields:

    activity = <Date 2021-11-27.04:14:33.454>
actor = 'malin'
assignee = 'gregory.p.smith'
closed = True
closed_date = <Date 2021-11-27.00:24:26.190>
closer = 'gregory.p.smith'
components = ['Library (Lib)']
creation = <Date 2020-09-07.03:14:46.922>
creator = 'malin'
dependencies = []
files = ['50445']
hgrepos = []
issue_num = 41735
keywords = ['patch']
message_count = 9.0
messages = ['376473', '392045', '392049', '406447', '407115', '407116', '407119', '407122', '407132']
nosy_count = 4.0
nosy_names = ['gregory.p.smith', 'lukasz.langa', 'malin', 'miss-islington']
pr_nums = ['22126', '22130', '22132', '29587', '29588', '29811', '29812']
priority = 'normal'
resolution = 'fixed'
stage = 'commit review'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue41735'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']
    @animalize
    Copy link
    Mannequin Author

    animalize mannequin commented Sep 7, 2020

    The code in zlib module:

    self->zst.next_in = data->buf;  // set next_in
...
ENTER_ZLIB(self);   // acquire thread lock

    self->zst is a z_stream struct defined in zlib, used to record states of a compress/decompress stream:

    typedef struct z_stream_s {
    Bytef    *next_in;  /* next input byte */
    uInt     avail_in;  /* number of bytes available at next_in */
    uLong    total_in;  /* total number of input bytes read so far */

    Bytef    *next_out; /* next output byte will go here */
    uInt     avail_out; /* remaining free space at next_out */
    uLong    total_out; /* total number of bytes output so far */
    
    ... // Other states
} z_stream;

    Setting next_in before acquiring the thread lock may mix up compress/decompress state in other threads.

    Moreover, modify ENTER_ZLIB macro, don't release the GIL when the thread lock can be acquired immediately. This behavior is the same as the bz2/lzma modules.

    @animalize animalize mannequin added 3.8 only security fixes 3.9 only security fixes 3.10 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Sep 7, 2020
    @ambv
    Copy link
    Contributor

    ambv commented Apr 27, 2021

    Thanks! ✨ 🍰 ✨

    @ambv ambv closed this as completed Apr 27, 2021
    @ambv ambv closed this as completed Apr 27, 2021
    @animalize
    Copy link
    Mannequin Author

    animalize mannequin commented Apr 27, 2021

    Thanks for review.

    @animalize
    Copy link
    Mannequin Author

    animalize mannequin commented Nov 17, 2021

    Sorry, I found an omission.

    The previous PRs fixed the bug in these methods:

        zlib.Compress.compress()
    zlib.Decompress.decompress()

    This method also has this bug, fix in PR29587 (main/3.10) and PR29588 (3.9-):

        zlib.Decompress.flush()
    
Attached file `test_flush.py` can reliably reproduce the bug.

    This time I carefully checked bz2/lzma/zlib modules, it should be no problem anymore.

    Gregory P. Smith should understand these codes, add him to nosy list.

    @animalize animalize mannequin reopened this Nov 17, 2021
    @animalize animalize mannequin reopened this Nov 17, 2021
    @gpshead
    Copy link
    Member

    gpshead commented Nov 27, 2021

    New changeset 7edb627 by Ma Lin in branch 'main':
    bpo-41735: Fix thread lock in zlib.Decompress.flush() may go wrong (GH-29587)
    7edb627

    @gpshead
    Copy link
    Member

    gpshead commented Nov 27, 2021

    New changeset 86c1265 by Ma Lin in branch '3.9':
    [3.9] bpo-41735: Fix thread lock in zlib.Decompress.flush() may go wrong (GH-29588)
    86c1265

    @gpshead
    Copy link
    Member

    gpshead commented Nov 27, 2021

    Thanks malin!

    @gpshead gpshead closed this as completed Nov 27, 2021
    @gpshead gpshead closed this as completed Nov 27, 2021
    @miss-islington
    Copy link
    Contributor

    New changeset 57100c8 by Miss Islington (bot) in branch '3.10':
    [3.10] bpo-41735: Fix thread lock in zlib.Decompress.flush() may go wrong (GH-29587) (GH-29811)
    57100c8

    @animalize
    Copy link
    Mannequin Author

    animalize mannequin commented Nov 27, 2021

    Thanks for review!

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @gpshead gpshead

    Labels
    3.8 only security fixes 3.9 only security fixes 3.10 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    3 participants
    @ambv @gpshead @miss-islington