AMD64 Windows8.1 Refleaks 3.x:
http://buildbot.python.org/all/#/builders/80/builds/118

test_asyncio leaked [4, 4, 3] memory blocks, sum=11

I reproduced the issue. I'm running test.bisect to try to isolate this bug.

It seems to be related to sendfile():

C:\vstinner\python\master>python -m test -R 3:3 test_asyncio -m test.test_asyncio.test_events.ProactorEventLoopTests.test_sendfile_close_peer_in_middle_of_receiving
Running Debug|x64 interpreter...
Run tests sequentially
0:00:00 [1/1] test_asyncio
beginning 6 repetitions
123456
......
test_asyncio leaked [1, 2, 1] memory blocks, sum=4
test_asyncio failed

1 test failed:
test_asyncio

Total duration: 1 sec
Tests result: FAILURE

Andrew, please take a look. I'll have very limited (to no at all) time to work on Python in the next 2 weeks.

The test is still leaking memory blocks. Any progress on investigating the issue?

Oh, I found again this bug while working on my PR 7827 (detect handle leaks on Windows in regrtest).

See also bpo-33735: my commit 23401fb fixes a false alarm in regrtest when hunting leaks in test_multiprocessing_spawn or test_multiprocessing_forkserver.

New changeset df8e1fb by Victor Stinner in branch 'master':
bpo-32710: test_asyncio: test_sendfile reset policy (GH-11461)
df8e1fb

Update:

• test.test_asyncio.test_sendfile.ProactorEventLoopTests.test_sendfile_close_peer_in_the_middle_of_receiving leaks 1 reference per run: this bug is caused by bpo-35682 and fixed by PR 11462
• test.test_asyncio.test_sendfile.ProactorEventLoopTests.test_sendfile_fallback_close_peer_in_the_middle_of_receiving leaks 1 reference per run: I don't understand why.

I spent a lot of time to investigate test_sendfile_fallback_close_peer_in_the_middle_of_receiving() leak and I don't understand the issue. The main loop is BaseEventLoop._sendfile_fallback(). For the specific case of this test, the loop can be simplified to:

        proto = _SendfileFallbackProtocol(transp)
        try:
            while True:
                data = b'x' * (1024 * 64)
                await proto.drain()
                transp.write(data)
        finally:
            await proto.restore()

The server closes the connection after it gets 1024 bytes. The client socket gets a ConnectionAbortedError exception in _ProactorBaseWritePipeTransport._loop_writing() which calls _fatal_error():

        except OSError as exc:
            self._fatal_error(exc, 'Fatal write error on pipe transport')

_fatal_error() calls _force_close() which sets _closing to True and calls protocol.connection_lost(). In the meanwhile, drain() raises ConnectionError because is_closing() is true:

    async def drain(self):
        if self._transport.is_closing():
            raise ConnectionError("Connection closed by peer")
        ...

Interesting commit. No idea if it's related.

commit 79790bc
Author: Victor Stinner <vstinner@redhat.com>
Date: Fri Jun 8 00:25:52 2018 +0200

bpo-33694: Fix race condition in asyncio proactor (GH-7498)

The cancellation of an overlapped WSARecv() has a race condition
which causes data loss because of the current implementation of
proactor in asyncio.

No longer cancel overlapped WSARecv() in _ProactorReadPipeTransport
to work around the race condition.

Remove the optimized recv_into() implementation to get simple
implementation of pause_reading() using the single _pending_data
attribute.

Move _feed_data_to_bufferred_proto() to protocols.py.

Remove set_protocol() method which became useless.

Attached test_aiosend.py is a simplified version of test to trigger the reference leak. Copy it to Lib/test/ and run:

vstinner@WIN C:\vstinner\python\master>python -m test test_aiosend -R 3:3
Running Debug|x64 interpreter...
Run tests sequentially
0:00:00 [1/1] test_aiosend
beginning 6 repetitions
123456
......
test_aiosend leaked [1, 1, 1] references, sum=3
test_aiosend leaked [1, 2, 1] memory blocks, sum=4
test_aiosend failed

== Tests result: FAILURE ==

1 test failed:
test_aiosend

Total duration: 548 ms
Tests result: FAILURE

It took me 1 year, a few sleepless nights, multiple attempts to understand the leak, but I eventually found it! WSASend() doesn't release the memory if it fails immediately. I wrote PR 11469 to fix the memory leak. ReadFile() has the same bug, I also fixed it.

By the way, the _overlapped.Overlapped type has no traverse function: it may help the garbage collector to add once, since asyncio is famous for building reference cycles by design (Future.set_exception()).

New changeset a234e14 by Victor Stinner in branch 'master':
bpo-32710: Fix leak in Overlapped_WSASend() (GH-11469)
a234e14

I ran test_asyncio refleak hunting on Windows, and there is no more leak!

vstinner@WIN C:\vstinner\python\master>python -m test test_asyncio -R 3:3
(...)
Total duration: 13 min 24 sec
Tests result: SUCCESS

I will close this PR once the 3.7 backport is merged.

New changeset 88ad48b by Miss Islington (bot) in branch '3.7':
bpo-32710: Fix leak in Overlapped_WSASend() (GH-11469)
88ad48b

New changeset 5485085 by Victor Stinner in branch 'master':
bpo-32710: Fix _overlapped.Overlapped memory leaks (GH-11489)
5485085

New changeset 059997d by Miss Islington (bot) in branch '3.7':
bpo-32710: Fix _overlapped.Overlapped memory leaks (GH-11489)
059997d

Ok, _overlapped.Overlapped should now have a few less memory leaks :-)