BUILDBOT FAILURE REPORT
=======================

Builder name: AMD64 Windows8.1 Refleaks 3.x
Builder url: https://buildbot.python.org/all/#/builders/80/
Build url: https://buildbot.python.org/all/#/builders/80/builds/683

Failed tests
------------



Test leaking resources
----------------------

- test_tools is leaking references

- test_exceptions is leaking references

- test_compile is leaking references

- test_trace is leaking references

- test_dis is leaking references

- test_builtin is leaking references

- test_syntax is leaking references

- test_named_expressions is leaking memory blocks

- test_syntax is leaking memory blocks

- test_symtable is leaking references

- test_typing is leaking references

- test_future is leaking references

- test_global is leaking references

- test_opcodes is leaking references

- test_grammar is leaking references

- test_scope is leaking references

- test_named_expressions is leaking references

- test_ast is leaking references

- test_ast is leaking memory blocks


Build summary
-------------

== Tests result: FAILURE then FAILURE ==

373 tests OK.

10 slowest tests:
- test_multiprocessing_spawn: 25 min 17 sec
- test_asyncio: 19 min 45 sec
- test_mailbox: 15 min 38 sec
- test_distutils: 14 min 8 sec
- test_concurrent_futures: 13 min 58 sec
- test_zipfile: 9 min 36 sec
- test_venv: 8 min 8 sec
- test_compileall: 6 min 37 sec
- test_lib2to3: 6 min 11 sec
- test_regrtest: 5 min 43 sec

16 tests failed:
    test_ast test_builtin test_compile test_dis test_exceptions
    test_future test_global test_grammar test_named_expressions
    test_opcodes test_scope test_symtable test_syntax test_tools
    test_trace test_typing

30 tests skipped:
    test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll
    test_fcntl test_fork1 test_gdb test_grp test_ioctl test_kqueue
    test_multiprocessing_fork test_multiprocessing_forkserver test_nis
    test_openpty test_ossaudiodev test_pipes test_poll test_posix
    test_pty test_pwd test_readline test_resource test_spwd
    test_syslog test_threadsignals test_wait3 test_wait4
    test_xxtestfuzz test_zipfile64

20 re-run tests:
    test_ast test_asyncgen test_builtin test_compile test_compileall
    test_dis test_exceptions test_future test_global test_grammar
    test_named_expressions test_opcodes test_scope test_symtable
    test_syntax test_tarfile test_threading test_tools test_trace
    test_typing

Total duration: 1 hour 21 min


Tracebacks
----------

```traceback
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\proactor_events.py", line 765, in _loop_self_reading
    f.result()  # may raise
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 808, in _poll
    value = callback(transferred, key, ov)
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 457, in finish_recv
    raise ConnectionResetError(*exc.args)
ConnectionResetError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request
.Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652af9f0> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652af3c0> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Error on reading from the event loop self pipe
loop: <ProactorEventLoop running=True closed=False debug=False>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 453, in finish_recv
    return ov.getresult()
OSError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request


Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\proactor_events.py", line 765, in _loop_self_reading
    f.result()  # may raise
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 808, in _poll
    value = callback(transferred, key, ov)
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 457, in finish_recv
    raise ConnectionResetError(*exc.args)
ConnectionResetError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request
.Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652afe10> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652afb50> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Error on reading from the event loop self pipe
loop: <ProactorEventLoop running=True closed=False debug=False>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 453, in finish_recv
    return ov.getresult()
OSError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request


Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\proactor_events.py", line 765, in _loop_self_reading
    f.result()  # may raise
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 808, in _poll
    value = callback(transferred, key, ov)
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 457, in finish_recv
    raise ConnectionResetError(*exc.args)
ConnectionResetError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request
.


Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\proactor_events.py", line 765, in _loop_self_reading
    f.result()  # may raise
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 808, in _poll
    value = callback(transferred, key, ov)
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 457, in finish_recv
    raise ConnectionResetError(*exc.args)
ConnectionResetError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request
.Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652af730> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652af890> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Error on reading from the event loop self pipe
loop: <ProactorEventLoop running=True closed=False debug=False>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 453, in finish_recv
    return ov.getresult()
OSError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request


Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\proactor_events.py", line 765, in _loop_self_reading
    f.result()  # may raise
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 808, in _poll
    value = callback(transferred, key, ov)
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 457, in finish_recv
    raise ConnectionResetError(*exc.args)
ConnectionResetError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request
.Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652af7e0> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652afec0> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Error on reading from the event loop self pipe
loop: <ProactorEventLoop running=True closed=False debug=False>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 453, in finish_recv
    return ov.getresult()
OSError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request


Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb61c51f70> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Error on reading from the event loop self pipe
loop: <ProactorEventLoop running=True closed=False debug=False>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 453, in finish_recv
    return ov.getresult()
OSError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request


Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\proactor_events.py", line 765, in _loop_self_reading
    f.result()  # may raise
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 808, in _poll
    value = callback(transferred, key, ov)
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 457, in finish_recv
    raise ConnectionResetError(*exc.args)
ConnectionResetError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request
.Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652af260> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652af310> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Error on reading from the event loop self pipe
loop: <ProactorEventLoop running=True closed=False debug=False>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 453, in finish_recv
    return ov.getresult()
OSError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request

```

Current builder status
----------------------

The builder is failing currently

Commits
-------



Other builds with similar failures
----------------------------------

-  https://buildbot.python.org/all/#/builders/1/builds/694
-  https://buildbot.python.org/all/#/builders/224/builds/70
-  https://buildbot.python.org/all/#/builders/223/builds/84

Bisecting points at commit 5dbe0f59b7a4f39c7c606b48056bc29e406ebf78 as the commit that introduced the reference leaks

See also https://bugs.python.org/issue37947

New changeset 4901dc46da5ecb131f8d902a0fbd704934f209e1 by Pablo Galindo in branch 'master':
bpo-37954: Fix reference leak in the symtable (GH-15514)
https://github.com/python/cpython/commit/4901dc46da5ecb131f8d902a0fbd704934f209e1

Scheduled two custom builds of the 3.x refleak buildbots:

https://buildbot.python.org/all/#/builders/80/builds/684
https://buildbot.python.org/all/#/builders/1/builds/695

New changeset ed8af33cce5554545d2bd079b23fe551d26fb4bd by Miss Islington (bot) in branch '3.8':
bpo-37954: Fix reference leak in the symtable (GH-15514)
https://github.com/python/cpython/commit/ed8af33cce5554545d2bd079b23fe551d26fb4bd

Scheduled another two custom builds on the 3.8 branch:

https://buildbot.python.org/all/#/builders/223/builds/85
https://buildbot.python.org/all/#/builders/224/builds/71

Great work, Pablo! This does *not* look like it was an easy one to track down.

Thank you very much, Steve! This was indeed a bit tricky :)

The last refleak buildbot is green again, so closing this.

Thank you Pablo!

Even having seen your fix, it still took me a couple of rescans of the original PR to figure out exactly how I'd broken it in the first place (refactoring to call an existing function without noticing that the replaced code included an extra DECREF).