This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: shutil.copyfile() doesn't close the opened files
Type: behavior Stage: resolved
Components: IO Versions: Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: josh.r, serhiy.storchaka, vocdetnojz
Priority: normal Keywords:

Created on 2016-04-26 13:47 by vocdetnojz, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (6)
msg264286 - (view) Author: Vukasin Felbab (vocdetnojz) Date: 2016-04-26 13:47
shutil.copyfile() doesn't close the opened files, so it is vulnerable to IO Error 24: too many files open

actually, the src and dst files should be closed after copy
msg264294 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-04-26 14:33
Could you provide an example?
msg282042 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-11-29 20:32
As I see, shutil.copyfile() uses the "with" statements and closes files just after copying.
msg282051 - (view) Author: Josh Rosenberg (josh.r) * (Python triager) Date: 2016-11-29 22:55
Agreed. 2.7 source is definitely using with: https://hg.python.org/cpython/file/2.7/Lib/shutil.py#l82
msg282232 - (view) Author: Vukasin Felbab (vocdetnojz) Date: 2016-12-02 09:10
Okay, so the problem is that I have a command library instance which is containing a connection to a server and functions can be called to execute some queries. As I pass this instance as a parameter in a recursive chain, after a while the open files limit is reached, I got it a few times an error on file opening. It is sure that I don't leave connections and open files before calling the next recursion, only thing this command library has the connection open. Additionally, when I store the instance in a global variable and call it from there then there are no IO errors. So, it means that the python is not giving parameters by reference, but by value? Because according to the Python, if I pass a mutable object as an argument, an if I use that for querying and passing the same instance to the next recursion level it should work fine, because in this case it should pass by reference, but it is not obviously, it passes by value, the connections remain open and the io limit is reached.
msg282238 - (view) Author: Josh Rosenberg (josh.r) * (Python triager) Date: 2016-12-02 13:52
You're going to need to provide a real repro; your description is effectively useless. For the record, Python is not precisely pass by reference (it's roughly equivalent to passing a pointer in C, binding the local name to the same pointer, so if you reassign the local name, you lose the reference, but otherwise it behaves like pass-by-reference).

Are you sure the command library doesn't open new connections as needed? In a recursive call scenario, if it goes deep enough, even if it clean up with the recursive call eventually returns, you could still hit the open files limit.

The problem is not copyfile, even if you happen to see copyfile in the traceback. You've got too many open files, and copyfile is the straw that broke the camel's back, but unless you're running 1000 threads doing copyfile simultaneously, it's not responsible for the excess open file handles.

But this is all speculation; the only thing I'm confident of is that copyfile isn't ultimately responsible. We have no idea what's wrong, so you need to provide some sort of minimal repro that exhibits the problem.
History
Date User Action Args
2022-04-11 14:58:30adminsetgithub: 71048
2017-10-23 14:55:37serhiy.storchakasetstatus: open -> closed
resolution: not a bug
stage: resolved
2016-12-02 13:52:24josh.rsetmessages: + msg282238
2016-12-02 09:10:05vocdetnojzsetstatus: pending -> open

messages: + msg282232
2016-12-02 06:03:08serhiy.storchakasetstatus: open -> pending
2016-11-29 22:55:39josh.rsetstatus: pending -> open
nosy: + josh.r
messages: + msg282051

2016-11-29 20:32:13serhiy.storchakasetstatus: open -> pending

messages: + msg282042
2016-04-26 14:33:48serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg264294
2016-04-26 13:47:15vocdetnojzcreate