Author haypo
Recipients haypo, serhiy.storchaka, yselivanov
Date 2016-03-15.13:15:15
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1458047716.42.0.114286591066.issue26567@psf.upfronthosting.co.za>
In-reply-to
Content
Python emits ResourceWarning when an object using limited resource is destroyed without being explicitly closed: files, sockets, etc. The problem is that it's hard to find where the object comes from, since the warning can occur very late, in the garbage collector, etc.

I propose to reuse tracemalloc.get_object_traceback() to show were the object was allocated, when tracemalloc traces memory allocations. In practice, I propose to add a new "source" parameter to warnings.showwarning().

Attached patch:

* Add a new keyword-only source parameter to warnings.showwarning()
* Add C function PyErr_ResourceWarning() to pass source
* showwarning() uses tracemalloc.get_object_traceback() to get the traceback were the object was allocated
* Modify socket.socket, io.FileIO and os.scandir destructor to use PyErr_ResourceWarning()


Backward-compatibility problem: The C function PyErr_ResourceWarning() always call warnings.showwarning() with the keyword parameter source. If an application replaces the warnings.showwarning() function, it will probably fail because it doesn't know the source parameter.

I don't know how to handle this backward compatibility issue.


The patch is incomplete, it's not possible yet to emit a warning in pure Python with a source parameter.


x.py script used for examples below:
-----------------
import warnings
import os
import socket

def func2():
    #f=open("/etc/issue")
    #f=os.scandir('.')
    f=socket.socket()
    f=None

def func():
    func2()

func()
-----------------


Output with Python 3.5:
-----
x.py:9: ResourceWarning: unclosed <socket.socket fd=3, ...>
  f=None
-----


Output with -X tracemalloc=5 command line option and patched Python 3.6:
-----
x.py:9: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>
  f=None
Object allocated at (most recent call first):
  File "x.py", lineno 8
    f=socket.socket()
  File "x.py", lineno 12
    func2()
  File "x.py", lineno 14
    func()
-----

It's much easier to understand where the warning comes from, no? At x.py:8, line "f=socket.socket()".

Note: the traceback doesn't contain the function name, since tracemalloc only stores filename and line number.


See also the issue #26564 "Malloc debug hooks: display memory block traceback on error".


For Python < 3.6, I wrote "res_warn.py" script which monkey-patches io.FileIO and socket.socket to implement something similar. The script is a fragile hack. I would prefer to have the feature built-in Python.

https://bitbucket.org/haypo/misc/src/0a40f27360424145bad0f9b62c9e9148ffdbb169/python/res_warn.py
History
Date User Action Args
2016-03-15 13:15:17hayposetrecipients: + haypo, serhiy.storchaka, yselivanov
2016-03-15 13:15:16hayposetmessageid: <1458047716.42.0.114286591066.issue26567@psf.upfronthosting.co.za>
2016-03-15 13:15:16haypolinkissue26567 messages
2016-03-15 13:15:16haypocreate