Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PyFile_FromString leaks file descriptors in python 2.7 #58710

Closed
doko42 opened this issue Apr 5, 2012 · 8 comments
Closed

PyFile_FromString leaks file descriptors in python 2.7 #58710

doko42 opened this issue Apr 5, 2012 · 8 comments
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) performance Performance or resource usage

Comments

@doko42
Copy link
Member

doko42 commented Apr 5, 2012

BPO 14505
Nosy @doko42, @pitrou, @vstinner

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-04-05.12:16:26.487>
created_at = <Date 2012-04-05.09:01:22.822>
labels = ['interpreter-core', 'performance']
title = 'PyFile_FromString leaks file descriptors in python 2.7'
updated_at = <Date 2012-04-05.17:25:05.928>
user = 'https://github.com/doko42'

bugs.python.org fields:

activity = <Date 2012-04-05.17:25:05.928>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2012-04-05.12:16:26.487>
closer = 'pitrou'
components = ['Interpreter Core']
creation = <Date 2012-04-05.09:01:22.822>
creator = 'doko'
dependencies = []
files = []
hgrepos = []
issue_num = 14505
keywords = []
message_count = 8.0
messages = ['157555', '157559', '157560', '157561', '157563', '157574', '157575', '157607']
nosy_count = 4.0
nosy_names = ['doko', 'pitrou', 'vstinner', 'python-dev']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue14505'
versions = ['Python 2.7']

@doko42
Copy link
Member Author

doko42 commented Apr 5, 2012

[forwarded from http://bugs.debian.org/664529]

seen with 2.7.3 rc2

File descriptors opened by PyFile_FromString don't get closed when the
reference count is decreased.

Here's my test program, pythony.c:

#include <Python.h>

int main()
{
  int i = 0;
  PyObject *obj;

  Py_Initialize();
  while (i++ < 5) {
    obj = PyFile_FromString("hello.py", "r");
    assert(obj);
    Py_DECREF(obj);
  }
  Py_Finalize();
}

hello.py is 'print("hello world")'.

I'm compiling it with both Python 2.6 and 2.7.
$ gcc pythony.c -lpython2.6 -L/usr/lib/python2.6/config -I/usr/include/python2.6/ -o pythony-2.6
$ gcc pythony.c -lpython2.7 -L/usr/lib/python2.7/config -I/usr/include/python2.7/ -o pythony-2.7

$ strace ./pythony-2.6 2>&1 | tail -n 20
ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fffb1d097b0) = -1 EINVAL (Invalid argument)
ioctl(2, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fffb1d097b0) = -1 EINVAL (Invalid argument)
open("hello.py", O_RDONLY)              = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=21, ...}) = 0
close(3)                                = 0
open("hello.py", O_RDONLY)              = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=21, ...}) = 0
close(3)                                = 0
open("hello.py", O_RDONLY)              = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=21, ...}) = 0
close(3)                                = 0
open("hello.py", O_RDONLY)              = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=21, ...}) = 0
close(3)                                = 0
open("hello.py", O_RDONLY)              = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=21, ...}) = 0
close(3)                                = 0
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7f1e1a0224f0}, {0x7f1e1a49a160, [], SA_RESTORER, 0x7f1e1a0224f0}, 8) = 0
exit_group(0)                           = ?


$ strace ./pythony-2.7 2>&1 | tail -n 20
fstat(4, {st_mode=S_IFREG|0644, st_size=1950, ...}) = 0
read(4, "", 4096)                       = 0
close(4)                                = 0
munmap(0x7fa41f10f000, 4096)            = 0
close(3)                                = 0
ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7ffff7bd33f0) = -1 EINVAL (Invalid argument)
ioctl(2, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7ffff7bd33f0) = -1 EINVAL (Invalid argument)
open("hello.py", O_RDONLY)              = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=21, ...}) = 0
open("hello.py", O_RDONLY)              = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=21, ...}) = 0
open("hello.py", O_RDONLY)              = 5
fstat(5, {st_mode=S_IFREG|0644, st_size=21, ...}) = 0
open("hello.py", O_RDONLY)              = 6
fstat(6, {st_mode=S_IFREG|0644, st_size=21, ...}) = 0
open("hello.py", O_RDONLY)              = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=21, ...}) = 0
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7fa4206e24f0}, {0x7fa420b8dd50, [], SA_RESTORER, 0x7fa4206e24f0}, 8) = 0
exit_group(0)                           = ?

The Python 2.7 version never calls close, not even at Py_Finalize().

On #d-d, jwilk suspected that this change might be the cause:
http://hg.python.org/cpython/rev/0f5b64630fda/#l4.46

@doko42 doko42 added the interpreter-core (Objects, Python, Grammar, and Parser dirs) label Apr 5, 2012
@vstinner
Copy link
Member

vstinner commented Apr 5, 2012

File descriptors opened by PyFile_FromString don't get closed
when the reference count is decreased.

Correct, PyFile_FromString() doesn't close the file descriptor, even if you call its close() method. You have to call manually fclose(file->f_fp).

Or you can use PyFile_FromFile:

fp = fopen(name, mode);
if (fp == NULL) ...
obj = PyFile_FromFile(fp, name, mode, fclose);
if (obj == NULL) {
   /* no need to call fclose(fp) here, it's done by PyFile_FromFile() */
   ...
}
...
Py_DECREF(obj);

Would you like to write a patch for the documentation?

@pitrou
Copy link
Member

pitrou commented Apr 5, 2012

Victor, that sounds like a strange behaviour to me. PyFile_FromString is a public API and maybe it shouldn't have changed between 2.6 and 2.7.

@vstinner
Copy link
Member

vstinner commented Apr 5, 2012

Victor, that sounds like a strange behaviour to me.

We may change PyFile_FromString() to call fclose() when the file is
closed explicitly (call its close() method), but it may break backward
compatibility. I prefer to document the behaviour instead.

PyFile_FromString is a public API and maybe it shouldn't
have changed between 2.6 and 2.7.

PyFile_FromString() didn't change in Python 2.7.

I changed PyFile_FromFile() in Python 2.7 to fix the issue bpo-7732.

changeset: 72456:0f5b64630fda
branch: 2.7
parent: 72450:c02e790c4535
user: Victor Stinner <victor.stinner@haypocalc.com>
date: Fri Sep 23 19:37:03 2011 +0200
files: Doc/c-api/file.rst Lib/test/test_import.py Misc/NEWS
Objects/fileobject.c Python/import.c
description:
Issue bpo-7732: Fix a crash on importing a module if a directory has the same name
than a Python module (e.g. "__init__.py"): don't close the file twice.

PyFile_FromFile() does also close the file if PyString_FromString() failed. It
did already close the file on fill_file_fields() error (e.g. if the file is a
directory).

@pitrou
Copy link
Member

pitrou commented Apr 5, 2012

Victor, what exactly are you talking about? http://hg.python.org/cpython/rev/0f5b64630fda/ *does* change PyFile_FromString.
And Matthias mentioned that the problem didn't occur with 2.6.

@python-dev
Copy link
Mannequin

python-dev mannequin commented Apr 5, 2012

New changeset 8258e5fa4a19 by Antoine Pitrou in branch '2.7':
Issue bpo-14505: Fix file descriptor leak when deallocating file objects created with PyFile_FromString().
http://hg.python.org/cpython/rev/8258e5fa4a19

@pitrou
Copy link
Member

pitrou commented Apr 5, 2012

Matthias, this should be fixed now.

@pitrou pitrou closed this as completed Apr 5, 2012
@pitrou pitrou added the performance Performance or resource usage label Apr 5, 2012
@vstinner
Copy link
Member

vstinner commented Apr 5, 2012

Victor, what exactly are you talking about? http://hg.python.org/cpython/rev/0f5b64630fda/ *does* change PyFile_FromString.

Oh. I don't know (remember) why I did this change!? I suppose that I
changed it to test something and then forget to remove the temporary
hack :-/

@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) performance Performance or resource usage
Projects
None yet
Development

No branches or pull requests

3 participants