Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test_io leaks references on AMD64 Fedora Rawhide Refleaks 3.8 #81442

Closed
vstinner opened this issue Jun 13, 2019 · 7 comments
Closed

test_io leaks references on AMD64 Fedora Rawhide Refleaks 3.8 #81442

vstinner opened this issue Jun 13, 2019 · 7 comments
Labels
3.8 only security fixes

Comments

@vstinner
Copy link
Member

BPO 37261
Nosy @vstinner, @miss-islington
PRs
  • bpo-37261: Fix support.catch_unraisable_exception() #14052
  • [3.8] bpo-37261: Fix support.catch_unraisable_exception() (GH-14052) #14056
  • bpo-37261: Document sys.unraisablehook corner cases #14059
  • [3.8] bpo-37261: Document sys.unraisablehook corner cases (GH-14059) #14089
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2019-06-13.13:45:26.159>
    created_at = <Date 2019-06-13.01:43:10.730>
    labels = ['3.8']
    title = 'test_io leaks references on AMD64 Fedora Rawhide Refleaks 3.8'
    updated_at = <Date 2019-06-14.16:59:57.981>
    user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2019-06-14.16:59:57.981>
    actor = 'miss-islington'
    assignee = 'none'
    closed = True
    closed_date = <Date 2019-06-13.13:45:26.159>
    closer = 'vstinner'
    components = []
    creation = <Date 2019-06-13.01:43:10.730>
    creator = 'vstinner'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 37261
    keywords = ['patch']
    message_count = 7.0
    messages = ['345449', '345450', '345514', '345516', '345518', '345604', '345614']
    nosy_count = 2.0
    nosy_names = ['vstinner', 'miss-islington']
    pr_nums = ['14052', '14056', '14059', '14089']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue37261'
    versions = ['Python 3.8']

    @vstinner
    Copy link
    Member Author

    test_io leaks references on AMD64 Fedora Rawhide Refleaks 3.8:
    https://buildbot.python.org/all/#/builders/229/builds/10

    test_io leaked [23208, 23204, 23208] references, sum=69620
    test_io leaked [7657, 7655, 7657] memory blocks, sum=22969

    The issue has been introduced by my change:

    commit c15a682
    Author: Victor Stinner <vstinner@redhat.com>
    Date: Thu Jun 13 00:23:49 2019 +0200

    bpo-37223: test_io: silence destructor errors (GH-14031)
    
    * bpo-18748: Fix _pyio.IOBase destructor (closed case) (GH-13952)
    
    _pyio.IOBase destructor now does nothing if getting the closed
    attribute fails to better mimick _io.IOBase finalizer.
    
    (cherry picked from commit 4f6f7c5a611905fb6b81671547f268c226bc646a)
    
    * bpo-37223: test_io: silence destructor errors (GH-13954)
    
    Implement also MockNonBlockWriterIO.seek() method.
    
    (cherry picked from commit b589cef9c4dada2fb84ce0fae5040ecf16d9d5ef)
    
    * bpo-37223, test_io: silence last 'Exception ignored in:' (GH-14029)
    
    Use catch_unraisable_exception() to ignore 'Exception ignored in:'
    error when the internal BufferedWriter of the BufferedRWPair is
    destroyed. The C implementation doesn't give access to the
    internal BufferedWriter, so just ignore the warning instead.
    
    (cherry picked from commit 913fa1c8245d1cde6edb4254f4fb965cc91786ef)
    

    It seems like the root issue is the usage of catch_unraisable_exception() in test_io.

    class catch_unraisable_exception:
        def __init__(self):
            self.unraisable = None
            self._old_hook = None
    
        def _hook(self, unraisable):
            self.unraisable = unraisable
    
        def __enter__(self):
            self._old_hook = sys.unraisablehook
            sys.unraisablehook = self._hook
            return self
    
        def __exit__(self, *exc_info):
            # Clear the unraisable exception to explicitly break a reference cycle
            del self.unraisable
            sys.unraisablehook = self._old_hook

    *Sometimes* "del self.unraisable" of __exit__() triggers a new unraisable exception which calls again the _hook() method which sets again the unraisable attribute.

    catch_unraisable_exception resurects _io.BufferedWriter objects, and then "del self.unraisable" calls again their finalizer: iobase_finalize() is called again, and iobase_finalize() calls PyErr_WriteUnraisable() on close() failure.

    @vstinner vstinner added the 3.8 only security fixes label Jun 13, 2019
    @vstinner
    Copy link
    Member Author

    gdb traceback on "del self.unraisable":

    (gdb) where
    #0 _PyErr_WriteUnraisableDefaultHook (
    args=(<type at remote 0x7a9fa0>, ValueError('flush of closed file',), <traceback at remote 0x7fffe9cfb5f0>, None, <_io.BufferedWriter at remote 0x7fffe9bcb890>)) at Python/errors.c:1294
    #1 0x00000000005726ec in sys_unraisablehook (module=<module at remote 0x7fffeac10710>,
    unraisable=(<type at remote 0x7a9fa0>, ValueError('flush of closed file',), <traceback at remote 0x7fffe9cfb5f0>, None, <_io.BufferedWriter at remote 0x7fffe9bcb890>)) at ./Python/sysmodule.c:702
    #2 0x0000000000435741 in _PyMethodDef_RawFastCallKeywords (method=0x7bb2e0 <sys_methods+1216>, self=<module at remote 0x7fffeac10710>, args=0x7ffffffd1d00,
    nargs=1, kwnames=0x0) at Objects/call.c:650
    #3 0x00000000004359f4 in _PyCFunction_Vectorcall (func=<built-in method unraisablehook of module object at remote 0x7fffeac10710>, args=0x7ffffffd1d00,
    nargsf=1, kwnames=0x0) at Objects/call.c:737
    #4 0x0000000000534666 in _PyObject_Vectorcall (callable=<built-in method unraisablehook of module object at remote 0x7fffeac10710>, args=0x7ffffffd1d00,
    nargsf=1, kwnames=0x0) at ./Include/cpython/abstract.h:127
    #5 0x00000000005346b1 in _PyObject_FastCall (func=<built-in method unraisablehook of module object at remote 0x7fffeac10710>, args=0x7ffffffd1d00, nargs=1)
    at ./Include/cpython/abstract.h:147
    #6 0x0000000000536da4 in _PyErr_WriteUnraisableMsg (err_msg_str=0x0, obj=<_io.BufferedWriter at remote 0x7fffe9bcb890>) at Python/errors.c:1379
    #7 0x0000000000536ee6 in PyErr_WriteUnraisable (obj=<_io.BufferedWriter at remote 0x7fffe9bcb890>) at Python/errors.c:1425

    #8 0x00000000005eb97d in iobase_finalize (self=<_io.BufferedWriter at remote 0x7fffe9bcb890>) at ./Modules/_io/iobase.c:299
    #9 0x0000000000479fd2 in PyObject_CallFinalizer (self=<_io.BufferedWriter at remote 0x7fffe9bcb890>) at Objects/object.c:307
    #10 0x000000000047a044 in PyObject_CallFinalizerFromDealloc (self=<_io.BufferedWriter at remote 0x7fffe9bcb890>) at Objects/object.c:325
    #11 0x00000000005eb9e0 in _PyIOBase_finalize (self=<_io.BufferedWriter at remote 0x7fffe9bcb890>) at ./Modules/_io/iobase.c:320

    #12 0x00000000005f27d3 in buffered_dealloc (self=0x7fffe9bcb890) at ./Modules/_io/bufferedio.c:387
    #13 0x000000000047ebff in _Py_Dealloc (op=<_io.BufferedWriter at remote 0x7fffe9bcb890>) at Objects/object.c:2213
    #14 0x00000000005f210a in _Py_DECREF (filename=0x71eef3 "./Modules/_io/bufferedio.c", lineno=2138, op=<_io.BufferedWriter at remote 0x7fffe9bcb890>)
    at ./Include/object.h:478
    #15 0x00000000005f716a in bufferedrwpair_dealloc (self=0x7fffe9a90d70) at ./Modules/_io/bufferedio.c:2138

    #16 0x000000000047ebff in _Py_Dealloc (op=<_io.BufferedRWPair at remote 0x7fffe9a90d70>) at Objects/object.c:2213
    #17 0x000000000048d1b6 in _Py_DECREF (filename=0x69fe40 "./Include/object.h", lineno=541, op=<_io.BufferedRWPair at remote 0x7fffe9a90d70>)
    at ./Include/object.h:478
    #18 0x000000000048d1e2 in _Py_XDECREF (op=<_io.BufferedRWPair at remote 0x7fffe9a90d70>) at ./Include/object.h:541

    # del UnraisableHookArgs

    #19 0x000000000048d569 in structseq_dealloc (obj=0x7fffe9ad1360) at Objects/structseq.c:85
    #20 0x000000000047ebff in _Py_Dealloc (
        op=(<type at remote 0x7a9fa0>, ValueError('flush of closed file',), <unknown at remote 0x7fffe9a8ad70>, None, <_io.BufferedRWPair at remote 0x7fffe9a90d70>))
        at Objects/object.c:2213
    #21 0x0000000000463be6 in _Py_DECREF (filename=0x696dc9 "Objects/dictobject.c", lineno=1585, 
        op=(<type at remote 0x7a9fa0>, ValueError('flush of closed file',), <unknown at remote 0x7fffe9a8ad70>, None, <_io.BufferedRWPair at remote 0x7fffe9a90d70>))
        at ./Include/object.h:478
    #22 0x0000000000468428 in delitem_common (mp=0x7fffe9b17950, hash=-822296730167156087, ix=0, 
        old_value=(<type at remote 0x7a9fa0>, ValueError('flush of closed file',), <unknown at remote 0x7fffe9a8ad70>, None, <_io.BufferedRWPair at remote 0x7fffe9a90d70>)) at Objects/dictobject.c:1585
    #23 0x0000000000468663 in _PyDict_DelItem_KnownHash (op={'_old_hook': <built-in method unraisablehook of module object at remote 0x7fffeac10710>}, 
        key='unraisable', hash=-822296730167156087) at Objects/dictobject.c:1637
    #24 0x00000000004684e5 in PyDict_DelItem (op={'_old_hook': <built-in method unraisablehook of module object at remote 0x7fffeac10710>}, key='unraisable')
        at Objects/dictobject.c:1603

    # del self.unraisable

    #25 0x000000000046ef21 in _PyObjectDict_SetItem (tp=0xa58830, dictptr=0x7fffe9a8ad30, key='unraisable', value=0x0) at Objects/dictobject.c:4614

    @vstinner
    Copy link
    Member Author

    New changeset 6d22cc8 by Victor Stinner in branch 'master':
    bpo-37261: Fix support.catch_unraisable_exception() (GH-14052)
    6d22cc8

    @miss-islington
    Copy link
    Contributor

    New changeset b4f5b21 by Miss Islington (bot) in branch '3.8':
    bpo-37261: Fix support.catch_unraisable_exception() (GH-14052)
    b4f5b21

    @vstinner
    Copy link
    Member Author

    I tested manually "./python -m test test_io -m CBufferedRWPairTest -R 3:3": it failed before the fix, it now pass on 3.8 and master with the fix. I close the issue.

    Enjoy resurrected objects...

    @vstinner
    Copy link
    Member Author

    New changeset 212646c by Victor Stinner in branch 'master':
    bpo-37261: Document sys.unraisablehook corner cases (GH-14059)
    212646c

    @miss-islington
    Copy link
    Contributor

    New changeset 3b976d1 by Miss Islington (bot) in branch '3.8':
    bpo-37261: Document sys.unraisablehook corner cases (GH-14059)
    3b976d1

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 only security fixes
    Projects
    None yet
    Development

    No branches or pull requests

    2 participants