classification
Title: FD leaks in ZipFile.read(), ZipFile.extract() and also using explicit arc_member.close()
Type: resource usage Stage:
Components: IO, Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: Valery.Khamenya, amaury.forgeotdarc, meador.inge
Priority: normal Keywords: patch

Created on 2011-10-08 18:48 by Valery.Khamenya, last changed 2012-10-15 13:23 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
zipfiletest.py Valery.Khamenya, 2011-10-08 18:48
zipfile.patch Valery.Khamenya, 2011-10-09 10:15
Messages (5)
msg145182 - (view) Author: Valery Khamenya (Valery.Khamenya) Date: 2011-10-08 18:48
The attached file reproduces 3 types of FD leaks and leads to the error like:

IOError: [Errno 24] Too many open files: '/tmp/1019'

For example if executed with pypy.
msg145185 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-10-08 20:27
I can't reproduce this problem in either 2.7.2 or 3.3.0a0.

> For example if executed with pypy.

Do you mean that this problem is only reproducible when the attached
script is run with pypy?
msg145188 - (view) Author: Valery Khamenya (Valery.Khamenya) Date: 2011-10-08 20:44
"I can't reproduce this problem in either 2.7.2 or 3.3.0a0."

You probably mean CPython implementation of Python. No, I didn't mean this implementation.

"Do you mean that this problem is only reproducible when the attached
script is run with pypy?"

I can't say "only". I just could say yes, it is reproducible with pypy. Perhaps, there are others Python implementations that will suffer from the bug in current implementation of ZipFile
msg145191 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-10-08 21:58
Yes, in 2.7 many parts of the stdlib relies on reference counting to close files.  But 3.2 introduced a ResourceWarning which is emitted (in debug mode) each time a __del__ closes a valuable resource like a file or a socket.  This was done exactly for this reason - help other implementations with a different garbage collector.

Now Lib/zipfile.py is probably much more gc-friendly: see how it uses a new member "close_fileobj", and the "with" statement in ZipFile.read().

PyPy will benefit of this when it migrates to 3.2; meanwhile, you could apply the same changes in pypy's own copy of zipfile.py.
msg145250 - (view) Author: Valery Khamenya (Valery.Khamenya) Date: 2011-10-09 10:15
Amaury, I followed your advice. All relevant changes of 3.2 are backported via the patch attached.

P.S. now I can install Twisted using pypy too. "pypy setup.py install" works fine for me.
History
Date User Action Args
2012-10-15 13:23:34serhiy.storchakasetstatus: open -> closed
2011-10-09 10:15:17Valery.Khamenyasetfiles: + zipfile.patch
keywords: + patch
status: pending -> open
messages: + msg145250
2011-10-08 21:58:42amaury.forgeotdarcsetstatus: open -> pending

nosy: + amaury.forgeotdarc
messages: + msg145191

resolution: out of date
2011-10-08 20:44:28Valery.Khamenyasetmessages: + msg145188
2011-10-08 20:27:59meador.ingesetnosy: + meador.inge
messages: + msg145185
2011-10-08 18:48:54Valery.Khamenyacreate