Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

race condition in threading._newname() #56075

Closed
PeterSaveliev mannequin opened this issue Apr 18, 2011 · 10 comments
Closed

race condition in threading._newname() #56075

PeterSaveliev mannequin opened this issue Apr 18, 2011 · 10 comments
Assignees
Labels
stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@PeterSaveliev
Copy link
Mannequin

PeterSaveliev mannequin commented Apr 18, 2011

BPO 11866
Nosy @rhettinger, @amauryfa, @pitrou, @bitdancer
Files
  • thread_test.py: simple test python script that illustrates _newname() race
  • newname_race_fix.patch: race condition fix (itertools, w/o locking)
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/rhettinger'
    closed_at = <Date 2014-10-04.21:50:27.904>
    created_at = <Date 2011-04-18.11:29:02.372>
    labels = ['type-bug', 'library']
    title = 'race condition in threading._newname()'
    updated_at = <Date 2014-10-04.21:50:27.900>
    user = 'https://bugs.python.org/PeterSaveliev'

    bugs.python.org fields:

    activity = <Date 2014-10-04.21:50:27.900>
    actor = 'r.david.murray'
    assignee = 'rhettinger'
    closed = True
    closed_date = <Date 2014-10-04.21:50:27.904>
    closer = 'r.david.murray'
    components = ['Library (Lib)']
    creation = <Date 2011-04-18.11:29:02.372>
    creator = 'Peter.Saveliev'
    dependencies = []
    files = ['21704', '21866']
    hgrepos = []
    issue_num = 11866
    keywords = ['patch']
    message_count = 10.0
    messages = ['133963', '135004', '135019', '141990', '141994', '142015', '142016', '169656', '228498', '228499']
    nosy_count = 6.0
    nosy_names = ['rhettinger', 'amaury.forgeotdarc', 'pitrou', 'r.david.murray', 'python-dev', 'Peter.Saveliev']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue11866'
    versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

    @PeterSaveliev
    Copy link
    Mannequin Author

    PeterSaveliev mannequin commented Apr 18, 2011

    The _newname() function has no locking.

    It is called from the new thread constructor. Such constructor is executed within parent thread. So, when several threads create new threads simultaneously, there can be race condition, leading to the same name for two (or even more) threads.

    8<--------------------------------

    >>> import threading
    >>> import dis
    >>> dis.dis(threading._newname)
    403           0 LOAD_GLOBAL              0 (_counter)
                  3 LOAD_CONST               1 (1)
                  6 BINARY_ADD         
                  7 STORE_GLOBAL             0 (_counter)
    
    404          10 LOAD_FAST                0 (template)
                 13 LOAD_GLOBAL              0 (_counter)
                 16 BINARY_MODULO      
                 17 RETURN_VALUE       
    >>> 
    8<

    The race condition can be achieved between BINARY_ADD and STORE_GLOBAL. Several threads can do BINARY_ADD before the first one will do STORE_GLOBAL. All racing threads will get the same name.

    8<--------------------------------

    $ for i in `seq 0 100`; do python thread_test.py |\
     awk -F Thread- '{print $2}' |\
     grep -vE '^$' |\
     sort |\
     uniq -c -d; done
          2 35
          2 12
          ...
    8<

    As you see, there are cases when several threads can get same name.

    Proposals: use thread-safe increment counter (with atomic get-increment) like itertools.counter() (that does not release GIL)

    @PeterSaveliev PeterSaveliev mannequin added extension-modules C modules in the Modules dir type-bug An unexpected behavior, bug, or error labels Apr 18, 2011
    @pitrou
    Copy link
    Member

    pitrou commented May 2, 2011

    Do you want to provide a patch?

    @pitrou pitrou added stdlib Python modules in the Lib dir and removed extension-modules C modules in the Modules dir labels May 2, 2011
    @PeterSaveliev
    Copy link
    Mannequin Author

    PeterSaveliev mannequin commented May 3, 2011

    Ok, patch attached.

    Patch made for Python: 2.6
    Tested Python versions: 2.6, 2.7

    @PeterSaveliev
    Copy link
    Mannequin Author

    PeterSaveliev mannequin commented Aug 12, 2011

    Any news? I hope, the change is trivial enough…

    @amauryfa
    Copy link
    Member

    Are you sure that counter.next() cannot release the GIL? Remember that any DECREF can trigger the garbage collector and execute arbitrary code...

    @PeterSaveliev
    Copy link
    Mannequin Author

    PeterSaveliev mannequin commented Aug 13, 2011

    counter.next() is a C routine and it is atomic from Python's point of view — if I understand right.

    The test shows that original threading.py leads to a (rare) race here, while with counter object there is no race condition.

    @rhettinger
    Copy link
    Contributor

    I think the patch is correct.

    FWIW, my style is to prebind the next method, making the counter completely self-contained (like a closure):

    +_counter = itertools.count().next
     def _newname(template="Thread-%d"):
         global _counter
    -    _counter = _counter + 1
    -    return template % _counter
    +    return template % _counter()

    @rhettinger rhettinger self-assigned this Aug 13, 2011
    @pitrou
    Copy link
    Member

    pitrou commented Sep 1, 2012

    With or without Raymond's suggested change, "global _counter" is not necessary anymore.
    Note that this fix only works on implementations where itertools.count().next is atomic.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Oct 4, 2014

    New changeset a6b4add67168 by R David Murray in branch '2.7':
    bpo-11866: Eliminate race condition in the computation of names for new threads.
    https://hg.python.org/cpython/rev/a6b4add67168

    New changeset a6906b9e21d5 by R David Murray in branch '3.4':
    bpo-11866: Eliminate race condition in the computation of names for new threads.
    https://hg.python.org/cpython/rev/a6906b9e21d5

    New changeset e9afcce9a154 by R David Murray in branch 'default':
    Merge: bpo-11866: Eliminate race condition in the computation of names for new threads.
    https://hg.python.org/cpython/rev/e9afcce9a154

    @bitdancer
    Copy link
    Member

    I committed a version of this that uses Raymond's suggestion and also makes sure that the first thread is named Thread-1 as it has been in the past (there was a test failure without that fix, but I also think it makes sense for it to be Thread-1 since one would normally think of the main thread as thread 0).

    I did not try to turn the test script into a test case because it would take a long time to run if it is to have a reasonable chance of displaying the bug, and it would never be a consistent failure.

    Thanks, Peter; sorry it took so long to get this committed.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants