classification
Title: Multiple tests are leaking references in AMD64 Windows8.1 Refleaks 3.x and x86 Gentoo Refleaks 3.x buildbots
Type: Stage: resolved
Components: Versions: Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: lukasz.langa, miss-islington, ncoghlan, pablogsal, steve.dower
Priority: release blocker Keywords: patch

Created on 2019-08-26 14:30 by pablogsal, last changed 2019-08-29 11:57 by ncoghlan. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 15514 merged pablogsal, 2019-08-26 14:31
PR 15518 merged miss-islington, 2019-08-26 15:14
Messages (10)
msg350533 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-08-26 14:30
BUILDBOT FAILURE REPORT
=======================

Builder name: AMD64 Windows8.1 Refleaks 3.x
Builder url: https://buildbot.python.org/all/#/builders/80/
Build url: https://buildbot.python.org/all/#/builders/80/builds/683

Failed tests
------------



Test leaking resources
----------------------

- test_tools is leaking references

- test_exceptions is leaking references

- test_compile is leaking references

- test_trace is leaking references

- test_dis is leaking references

- test_builtin is leaking references

- test_syntax is leaking references

- test_named_expressions is leaking memory blocks

- test_syntax is leaking memory blocks

- test_symtable is leaking references

- test_typing is leaking references

- test_future is leaking references

- test_global is leaking references

- test_opcodes is leaking references

- test_grammar is leaking references

- test_scope is leaking references

- test_named_expressions is leaking references

- test_ast is leaking references

- test_ast is leaking memory blocks


Build summary
-------------

== Tests result: FAILURE then FAILURE ==

373 tests OK.

10 slowest tests:
- test_multiprocessing_spawn: 25 min 17 sec
- test_asyncio: 19 min 45 sec
- test_mailbox: 15 min 38 sec
- test_distutils: 14 min 8 sec
- test_concurrent_futures: 13 min 58 sec
- test_zipfile: 9 min 36 sec
- test_venv: 8 min 8 sec
- test_compileall: 6 min 37 sec
- test_lib2to3: 6 min 11 sec
- test_regrtest: 5 min 43 sec

16 tests failed:
    test_ast test_builtin test_compile test_dis test_exceptions
    test_future test_global test_grammar test_named_expressions
    test_opcodes test_scope test_symtable test_syntax test_tools
    test_trace test_typing

30 tests skipped:
    test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll
    test_fcntl test_fork1 test_gdb test_grp test_ioctl test_kqueue
    test_multiprocessing_fork test_multiprocessing_forkserver test_nis
    test_openpty test_ossaudiodev test_pipes test_poll test_posix
    test_pty test_pwd test_readline test_resource test_spwd
    test_syslog test_threadsignals test_wait3 test_wait4
    test_xxtestfuzz test_zipfile64

20 re-run tests:
    test_ast test_asyncgen test_builtin test_compile test_compileall
    test_dis test_exceptions test_future test_global test_grammar
    test_named_expressions test_opcodes test_scope test_symtable
    test_syntax test_tarfile test_threading test_tools test_trace
    test_typing

Total duration: 1 hour 21 min


Tracebacks
----------

```traceback
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\proactor_events.py", line 765, in _loop_self_reading
    f.result()  # may raise
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 808, in _poll
    value = callback(transferred, key, ov)
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 457, in finish_recv
    raise ConnectionResetError(*exc.args)
ConnectionResetError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request
.Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652af9f0> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652af3c0> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Error on reading from the event loop self pipe
loop: <ProactorEventLoop running=True closed=False debug=False>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 453, in finish_recv
    return ov.getresult()
OSError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request


Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\proactor_events.py", line 765, in _loop_self_reading
    f.result()  # may raise
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 808, in _poll
    value = callback(transferred, key, ov)
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 457, in finish_recv
    raise ConnectionResetError(*exc.args)
ConnectionResetError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request
.Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652afe10> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652afb50> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Error on reading from the event loop self pipe
loop: <ProactorEventLoop running=True closed=False debug=False>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 453, in finish_recv
    return ov.getresult()
OSError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request


Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\proactor_events.py", line 765, in _loop_self_reading
    f.result()  # may raise
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 808, in _poll
    value = callback(transferred, key, ov)
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 457, in finish_recv
    raise ConnectionResetError(*exc.args)
ConnectionResetError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request
.


Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\proactor_events.py", line 765, in _loop_self_reading
    f.result()  # may raise
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 808, in _poll
    value = callback(transferred, key, ov)
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 457, in finish_recv
    raise ConnectionResetError(*exc.args)
ConnectionResetError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request
.Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652af730> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652af890> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Error on reading from the event loop self pipe
loop: <ProactorEventLoop running=True closed=False debug=False>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 453, in finish_recv
    return ov.getresult()
OSError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request


Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\proactor_events.py", line 765, in _loop_self_reading
    f.result()  # may raise
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 808, in _poll
    value = callback(transferred, key, ov)
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 457, in finish_recv
    raise ConnectionResetError(*exc.args)
ConnectionResetError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request
.Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652af7e0> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652afec0> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Error on reading from the event loop self pipe
loop: <ProactorEventLoop running=True closed=False debug=False>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 453, in finish_recv
    return ov.getresult()
OSError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request


Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb61c51f70> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Error on reading from the event loop self pipe
loop: <ProactorEventLoop running=True closed=False debug=False>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 453, in finish_recv
    return ov.getresult()
OSError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request


Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\proactor_events.py", line 765, in _loop_self_reading
    f.result()  # may raise
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 808, in _poll
    value = callback(transferred, key, ov)
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 457, in finish_recv
    raise ConnectionResetError(*exc.args)
ConnectionResetError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request
.Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652af260> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652af310> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Error on reading from the event loop self pipe
loop: <ProactorEventLoop running=True closed=False debug=False>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 453, in finish_recv
    return ov.getresult()
OSError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request

```

Current builder status
----------------------

The builder is failing currently

Commits
-------



Other builds with similar failures
----------------------------------

-  https://buildbot.python.org/all/#/builders/1/builds/694
-  https://buildbot.python.org/all/#/builders/224/builds/70
-  https://buildbot.python.org/all/#/builders/223/builds/84
msg350534 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-08-26 14:31
Bisecting points at commit 5dbe0f59b7a4f39c7c606b48056bc29e406ebf78 as the commit that introduced the reference leaks
msg350536 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-08-26 14:57
See also https://bugs.python.org/issue37947
msg350538 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-08-26 15:14
New changeset 4901dc46da5ecb131f8d902a0fbd704934f209e1 by Pablo Galindo in branch 'master':
bpo-37954: Fix reference leak in the symtable (GH-15514)
https://github.com/python/cpython/commit/4901dc46da5ecb131f8d902a0fbd704934f209e1
msg350539 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-08-26 15:17
Scheduled two custom builds of the 3.x refleak buildbots:

https://buildbot.python.org/all/#/builders/80/builds/684
https://buildbot.python.org/all/#/builders/1/builds/695
msg350542 - (view) Author: miss-islington (miss-islington) Date: 2019-08-26 15:49
New changeset ed8af33cce5554545d2bd079b23fe551d26fb4bd by Miss Islington (bot) in branch '3.8':
bpo-37954: Fix reference leak in the symtable (GH-15514)
https://github.com/python/cpython/commit/ed8af33cce5554545d2bd079b23fe551d26fb4bd
msg350543 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-08-26 15:53
Scheduled another two custom builds on the 3.8 branch:

https://buildbot.python.org/all/#/builders/223/builds/85
https://buildbot.python.org/all/#/builders/224/builds/71
msg350547 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-08-26 16:30
Great work, Pablo! This does *not* look like it was an easy one to track down.
msg350644 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-08-27 19:35
Thank you very much, Steve! This was indeed a bit tricky :)

The last refleak buildbot is green again, so closing this.
msg350765 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2019-08-29 11:57
Thank you Pablo!

Even having seen your fix, it still took me a couple of rescans of the original PR to figure out exactly how I'd broken it in the first place (refactoring to call an existing function without noticing that the replaced code included an extra DECREF).
History
Date User Action Args
2019-08-29 11:57:37ncoghlansetmessages: + msg350765
2019-08-27 19:35:24pablogsalsetstatus: open -> closed
resolution: fixed
messages: + msg350644

stage: patch review -> resolved
2019-08-26 16:30:09steve.dowersetnosy: + steve.dower
messages: + msg350547
2019-08-26 15:53:15pablogsalsetmessages: + msg350543
2019-08-26 15:49:48miss-islingtonsetnosy: + miss-islington
messages: + msg350542
2019-08-26 15:17:16pablogsalsetmessages: + msg350539
2019-08-26 15:14:18miss-islingtonsetpull_requests: + pull_request15203
2019-08-26 15:14:10pablogsalsetmessages: + msg350538
2019-08-26 14:57:08pablogsalsetmessages: + msg350536
2019-08-26 14:31:59pablogsalsetkeywords: + patch
stage: patch review
pull_requests: + pull_request15200
2019-08-26 14:31:40pablogsalsetnosy: + ncoghlan
messages: + msg350534
2019-08-26 14:30:13pablogsalcreate