classification
Title: ResourceWarning: unclosed warning
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: martin.panter, vstinner
Priority: normal Keywords:

Created on 2017-09-19 16:30 by vstinner, last changed 2017-11-24 01:28 by vstinner. This issue is now closed.

Messages (5)
msg302541 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-19 16:30
_socket.socket object destructor emits a ResourceWarning if the socket is not closed. 

The problem is this warning:

build/Lib/contextlib.py:60: ResourceWarning: unclosed <socket.socket [closed] fd=3, family=AddressFamily.AF_INET, type=2049, proto=6>
  self.gen = func(*args, **kwds)

The message says "unclosed" and "closed" in the same sentence. It's confusing.

In fact, the Python module "socket" has a socket.socket class based on the _socket.socket of the C module "_socket".

The Python module has a private _closed attribute set to True as soon as close() was closed. *But* the underlying _socket.socket.close() is only called once the "io refs" counter reachs zero.

The Python module allows to open to "fork" the socket using the makefile() method: the "io refs" counter is increased in that case. makefile() creates a raw socket.SocketIO() object which will call the close() method of the original socket.

Ok, let's come back at the C level. The _socket.socket destructor checks if _socket.socket.close() was called to decide if the ResourceWarning should be emitted or not.

Maybe SocketIO should raise the ResourceWarning?

IMHO the minimum patch is to modify socket.socket.__repr__() to include the "io refs" counter. So a developer knowing the implementation can understand the surprising warning.
msg302934 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-25 09:29
socket.SocketIO inherits from io.RawIOBase which inherits from io.IOBase.

io.IOBase has a finalizer which calls the close() method. I tried to add a __del__ method to socket.SocketIO, but the object was already closed by the finalizer.

io.FileIO uses a trick at the C level: it sets an internal "finalizing" attribute to check if close() was called by IOBase finalizer. If it's the case, a ResourceWarning is emitted.

I don't see a simpler way to emit a ResourceWarning in SocketIO. Maybe it's not the right approach.
msg302949 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2017-09-25 12:07
I’m curious how you manage to trigger the warning in the “closed” state. The Python I have handy is half a year out of date, but all my attempts to trigger the warning either produce the less confusing version,

ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('0.0.0.0', 0)>

or there is no warning at all due IOBase.__del__ (see Issue 19829).

If your SocketIO was wrapped in a BufferedReader/Writer/RWPair, then that could easily close the SocketIO object before SocketIO.__del__ is called. You would also have to override the wrapper’s __del__ method, rather than (or as well as) SocketIO.__del__.
msg302952 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-25 12:33
> I’m curious how you manage to trigger the warning in the “closed” state.

It comes from Refleak buildbots. Example:

0:13:47 load avg: 3.24 [ 52/407] test_urllib2net passed (239 sec)
beginning 6 repetitions
123456
Resource 'http://www.imdb.com' is not available
/buildbot/buildarea/3.x.ware-gentoo-x86.refleak/build/Lib/test/libregrtest/refleak.py:253: ResourceWarning: unclosed <socket.socket [closed] fd=3, family=AddressFamily.AF_INET, type=2049, proto=6>
  gc.collect()
......

I failed to reproduce this exact warning. I don't know if it's related to the "Resource 'http://www.imdb.com' is not available" message.
msg306871 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-24 01:28
I'm not confortable to expose the value of the private _io_refs attribute in repr(socket.socket). I'm not sure that anything should really be done here. I close the issue.
History
Date User Action Args
2017-11-24 01:28:20vstinnersetstatus: open -> closed
resolution: out of date
messages: + msg306871

stage: resolved
2017-09-25 12:33:40vstinnersetmessages: + msg302952
2017-09-25 12:07:00martin.pantersetmessages: + msg302949
2017-09-25 09:29:01vstinnersetmessages: + msg302934
2017-09-19 16:30:16vstinnersetnosy: + martin.panter
2017-09-19 16:30:09vstinnercreate