with the attached code note how HttpClient.connection_lost callback is never called for the madis server. The madis server is an apache server, I tried with the OSX apache server and could not reproduce the issue so it seems something particular about their apache version or configuration.  This is a pretty critical issue as close() does not release the socket.

updating to make default the error case (madis)

madis-data.ncep.noaa.gov side does not complete ssl shutdown process.

Thanks so much for the patch!

may want to change spelling of what was supposed to be "shutdown" =)  Also think it's worth a comment stating why it's needed?  Like certain Apache servers were noticed to not complete the SSL shutdown process.

Can you guys create a PR on github.com/python/cpython?

There are two PRs for this issue. Is one depending on the other?

PR 409 has merge conflict that needs to be resolved.

> There are two PRs for this issue. Is one depending on the other?

No, they all address slightly different bugs. All need to be merged though.

> PR 409 has merge conflict that needs to be resolved.

Yes, it seems that it's based on 3.6 branch. I asked the author to rebase it.

New changeset a608d2d5a7f1aabe9bcbfc220135c5e126189390 by Yury Selivanov (Nikolay Kim) in branch 'master':
bpo-29406: asyncio SSL contexts leak sockets after calling close with certain servers (#409)
https://github.com/python/cpython/commit/a608d2d5a7f1aabe9bcbfc220135c5e126189390

New changeset 6e14fd2a14cef6ea0709ad234ab41198c2195591 by Yury Selivanov in branch '3.6':
[3.6] bpo-29406: asyncio SSL contexts leak sockets after calling close with certain servers (GH-409) (#2062)
https://github.com/python/cpython/commit/6e14fd2a14cef6ea0709ad234ab41198c2195591

New changeset 1395c58ef7b98f087d1d5d50962fe7a8c032f34d by Yury Selivanov in branch '3.5':
[3.5] bpo-29406: asyncio SSL contexts leak sockets after calling close with certain servers (GH-409) (#2063)
https://github.com/python/cpython/commit/1395c58ef7b98f087d1d5d50962fe7a8c032f34d

While testing current 3.6 top of trunk, I noticed spurious error messages being reported that were not causing the tests to fail.  Investigating further, I found the culprit to be this issue's 6e14fd2a14cef6ea0709ad234ab41198c2195591.  Curiously, the errors aren't showing up on buildbots although I can reproduce 100% on both platforms I've tried: Debian Linux and macOS 10.12.  For some reason that needs to be explored, using -W on regrtest causes the messages to be suppressed and the buildbots use -w by default.

$ ./python
Python 3.6.1+ (remotes/upstream/3.6:76eabd3a21, Jun 10 2017, 15:20:44)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

# without -w or -W
$ ./python -m test test_asyncio
Run tests sequentially
0:00:00 [1/1] test_asyncio
Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0xb55423b4>
transport: None
Traceback (most recent call last):
  File "/py/3x/unix/source/Lib/asyncio/sslproto.py", line 660, in _process_write_backlog
    self._transport.write(chunk)
AttributeError: 'NoneType' object has no attribute 'write'
Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0xb5542504>
transport: None
Traceback (most recent call last):
  File "/py/3x/unix/source/Lib/asyncio/sslproto.py", line 660, in _process_write_backlog
    self._transport.write(chunk)
AttributeError: 'NoneType' object has no attribute 'write'
Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0xb639f8bc>
transport: None
Traceback (most recent call last):
  File "/py/3x/unix/source/Lib/asyncio/sslproto.py", line 660, in _process_write_backlog
    self._transport.write(chunk)
AttributeError: 'NoneType' object has no attribute 'write'
Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0xb6126f84>
transport: None
Traceback (most recent call last):
  File "/py/3x/unix/source/Lib/asyncio/sslproto.py", line 660, in _process_write_backlog
    self._transport.write(chunk)
AttributeError: 'NoneType' object has no attribute 'write'
Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0xb6392fbc>
transport: None
Traceback (most recent call last):
  File "/py/3x/unix/source/Lib/asyncio/sslproto.py", line 660, in _process_write_backlog
    self._transport.write(chunk)
AttributeError: 'NoneType' object has no attribute 'write'
Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0xb671dedc>
transport: None
Traceback (most recent call last):
  File "/py/3x/unix/source/Lib/asyncio/sslproto.py", line 660, in _process_write_backlog
    self._transport.write(chunk)
AttributeError: 'NoneType' object has no attribute 'write'
1 test OK.

Total duration: 13 sec
Tests result: SUCCESS

# with -w
$ ./python -m test -w test_asyncio
Run tests sequentially
0:00:00 [1/1] test_asyncio
Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0xb4b15b5c>
transport: None
Traceback (most recent call last):
  File "/py/3x/unix/source/Lib/asyncio/sslproto.py", line 660, in _process_write_backlog
    self._transport.write(chunk)
AttributeError: 'NoneType' object has no attribute 'write'
Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0xb581f264>
transport: None
Traceback (most recent call last):
  File "/py/3x/unix/source/Lib/asyncio/sslproto.py", line 660, in _process_write_backlog
    self._transport.write(chunk)
AttributeError: 'NoneType' object has no attribute 'write'
Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0xb53f068c>
transport: None
Traceback (most recent call last):
  File "/py/3x/unix/source/Lib/asyncio/sslproto.py", line 660, in _process_write_backlog
    self._transport.write(chunk)
AttributeError: 'NoneType' object has no attribute 'write'
Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0xb65faf14>
transport: None
Traceback (most recent call last):
  File "/py/3x/unix/source/Lib/asyncio/sslproto.py", line 660, in _process_write_backlog
    self._transport.write(chunk)
AttributeError: 'NoneType' object has no attribute 'write'
Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0xb5816c04>
transport: None
Traceback (most recent call last):
  File "/py/3x/unix/source/Lib/asyncio/sslproto.py", line 660, in _process_write_backlog
    self._transport.write(chunk)
AttributeError: 'NoneType' object has no attribute 'write'
Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0xb582dea4>
transport: None
Traceback (most recent call last):
  File "/py/3x/unix/source/Lib/asyncio/sslproto.py", line 660, in _process_write_backlog
    self._transport.write(chunk)
AttributeError: 'NoneType' object has no attribute 'write'
1 test OK.

Total duration: 13 sec
Tests result: SUCCESS

# with -W -> no messages
$ ./python -m test -W test_asyncio
Run tests sequentially
0:00:00 [1/1] test_asyncio
1 test OK.

Total duration: 13 sec
Tests result: SUCCESS

Er, "For some reason that needs to be explored, using -W on regrtest causes the messages to be suppressed and the buildbots use -w by default."

That should be "use -W by default".

FTR, same behavior with master (3.7) top of trunk.

Nikolay, can you please take a look?

yes, i am on it

> yes, i am on it

Nick, have you been able to find what the problem is? If not, we'll need to pull this change out of 3.6.2.

> > yes, i am on it

> Nick, have you been able to find what the problem is? If not, we'll need to pull this change out of 3.6.2.

It's interesting to see how `_process_write_backlog` gets called after `_finalize`. One option would be to simply guard `transport.write` with `if transport is not None`, but I'm afraid that we'll only mask the bug then.

> It's interesting to see how `_process_write_backlog` gets called after `_finalize`. One option would be to simply guard `transport.write` with `if transport is not None`, but I'm afraid that we'll only mask the bug then.

OTOH we weren't setting the transport to None before, we were just closing it. Is setting it to None really necessary?

Alright, let's play it safe. I'm going to be reverting the change from all branches. Nick, if you are able to figure this out please create a new PR.

New changeset 09663de203396b28483a570bb268c130b59639cf by Yury Selivanov in branch 'master':
Revert "bpo-29406: asyncio SSL contexts leak sockets after calling close with certain servers (#409)" (#2111)
https://github.com/python/cpython/commit/09663de203396b28483a570bb268c130b59639cf

New changeset 83d30bd667924549bacf1428ac3e475cdf9792ae by Yury Selivanov in branch '3.6':
Revert "[3.6] bpo-29406: asyncio SSL contexts leak sockets after calling close with certain servers (GH-409) (#2062)" (#2112)
https://github.com/python/cpython/commit/83d30bd667924549bacf1428ac3e475cdf9792ae

New changeset 4e9dfe214dff000c5b5ad0d8bd8a393feed1d3cf by Yury Selivanov in branch '3.5':
Revert "[3.5] bpo-29406: asyncio SSL contexts leak sockets after calling close with certain servers (GH-409) (#2063)" (#2113)
https://github.com/python/cpython/commit/4e9dfe214dff000c5b5ad0d8bd8a393feed1d3cf

See also issue 29970.

This is not problem with madis-data.ncep.noaa.gov not doing ssl shutdown, this is problem with asyncio not doing it.

Patch from this #30698 issue fix this too.

Let’s close this issue then. I don’t like it anyway.

> On Jun 19, 2017, at 10:21 AM, Grzegorz Grzywacz <report@bugs.python.org> wrote:
> 
> 
> Grzegorz Grzywacz added the comment:
> 
> This is not problem with madis-data.ncep.noaa.gov not doing ssl shutdown, this is problem with asyncio not doing it.
> 
> Patch from this #30698 issue fix this too.
> 
> ----------
> nosy: +grzgrzgrz3
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue29406>
> _______________________________________

Yury, based on the most recent comments, can this issue be closed now?

> Yury, based on the most recent comments, can this issue be closed now?

Not sure, I'm reviewing the patch for #30698, let's see if it fixes the problem.

This is marked as Critical so either we should change the priority or fix it soon :)

I'm picking up the issue.
Will provide an updated PR soon.

I'm skeptical about `critical` priority.

The bugfix could be backported to Python 3.6 but I very not sure about the need for 3.5.
It is a desirable fix but not secure.

Yury Selivanov please confirm.

> The bugfix could be backported to Python 3.6 but I very not sure about the need for 3.5.
> It is a desirable fix but not secure.

The attached script looks very innocent, and it's worrisome that it's that easy to make an asyncio SSL server to leak sockets in some contexts.  I think it's OK to fix 3.5, but let's make the final decision when we have an approved PR.

I merged Nick's PR once, but then it introduced some problems on buildbots (https://bugs.python.org/msg295659), so I reverted the commit.

I'd suggest to look at the attached script again (scratch_1.py) and try to understand what exactly causes the problem with that specific Apache server version.  I'd also suggest to come up with a *functional* unittest that reproduces it (you can grab some testing utilities from the uvloop project).

my understanding is that the PR in https://bugs.python.org/issue30698 fixes this issue no?  If so can we get it merged?

Andrew, Yury, PR 4402 is still open. Should the fix land in 3.7?

I have removed the SSL component because it's not a bug in the ssl module.

I'll take a look on the evening

ping...

Dima, we'll likely address this in 3.8, when we land new SSL implementation.  Meanwhile you can use uvloop where all these bugs should be fixed.

Since bpo-issue44011 is fixed, this can be closed now @asvetlov.