Title: popen() slow on AIX due to large FOPEN_MAX value
Type: Stage:
Components: Library (Lib) Versions: Python 2.5
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: georg.brandl, hvbargen, johnlallen, schmir
Priority: normal Keywords: patch

Created on 2006-12-01 20:05 by johnlallen, last changed 2008-02-23 22:09 by georg.brandl. This issue is now closed.

File name Uploaded Description Edit
f_closem.patch johnlallen, 2006-12-01 20:05 AIX F_CLOSEM popen patch
Messages (4)
msg51421 - (view) Author: John L. Allen (johnlallen) Date: 2006-12-01 20:05
Experimentation revealed that the os.popen[234]() family of methods was at least 10 times slower on AIX than it was on Solaris.  It turns out that this is because of the _run_child method in, which has this definition:

    def _run_child(self, cmd):
        if isinstance(cmd, basestring):
            cmd = ['/bin/sh', '-c', cmd]
        for i in xrange(3, MAXFD):
            except OSError:
            os.execvp(cmd[0], cmd)

MAXFD is set as follows at the top of

    MAXFD = os.sysconf('SC_OPEN_MAX')
except (AttributeError, ValueError):
    MAXFD = 256

On AIX, SC_OPEN_MAX is 32767, whereas on Solaris, it is not defined, and we get the default value of 256.   So, on AIX the python code for loop is being used to close 32763 file descriptors, but only 253 on Solaris.  The slowness of this python loop is the source of the problem.

Several solutions are possible.  AIX provides a much faster way to close all file descriptors above a given one using the fcntl F_CLOSEM option.  In this case, it would be: fcntl(3, F_CLOSEM, 0).  Other OSes, like Solaris and BSD, have the closefrom() function instead.  I think ideally, we would want to have an os.closefrom() method defined in posixmodule.c, and always available on every OS, and have call that instead of doing the loop.  The closefrom() function would be defined something like this:

Close all file descriptors greater than or equal to fd (for low level IO).");

static PyObject *
posix_closefrom(PyObject *self, PyObject *args)
        int fd, maxfd, res;
        if (!PyArg_ParseTuple(args, "i:closefrom", &fd))
                return NULL;

        res = closefrom(fd);
#ifdef F_CLOSEM
        res = fcntl(3, F_CLOSEM, 0)

#if defined( HAVE_SYSCONF ) && defined( _SC_OPEN_MAX )
#  define PY_OPEN_MAX sysconf(_SC_OPEN_MAX)
#  ifdef FOPEN_MAX
#  else
#    ifdef OPEN_MAX
#      define PY_OPEN_MAX OPEN_MAX
#    else
#      ifdef _NFILE
#        define PY_OPEN_MAX _NFILE
#      else
#        define PY_OPEN_MAX 256
#      endif
#    endif
#  endif

        maxfd = PY_OPEN_MAX;

        while (fd < maxfd) {
          res = close(fd);


        if (res < 0)
                return posix_error();
        return Py_None;

While this is probably (close to) the ideal solution (since it would benefit all OSes by avoiding the close loop if possible or if not, moving it to C code), adding os.closefrom() probably needs to be discussed further before being accepted by the Python community.

Instead, I will provide a simpler patch that only benefits AIX.   It adds the F_CLOSEM attribute to the fcntl class in fcntlmodule.c, if defined, and modifies to check for it and use it if possible, instead of doing the close() loop.   See the attached patch (against the 2.5 source).  I don't believe that any documentation has to change.

John Allen
msg51422 - (view) Author: H. von Bargen (hvbargen) Date: 2007-02-19 10:22
I have submitted a new bug and referenced this patch from there.

I think should be patched in a similar way.
But, uses a "but" argument in _close_fds.
I think a correct solution should close all handles from 3 to but-1 in C using the usual "close()" API; and then close all handles from but+1 to MAX_FD by using the optimization with closefrom or fnctl.
msg62820 - (view) Author: Ralf Schmitt (schmir) Date: 2008-02-23 22:06
python 2.6 has os.closerange. however it is currently not used in popen2.
msg62821 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2008-02-23 22:09
Changed popen2 to use os.closerange() in r61019.
Date User Action Args
2008-02-23 22:09:40georg.brandlsetstatus: open -> closed
resolution: fixed
messages: + msg62821
nosy: + georg.brandl
2008-02-23 22:06:14schmirsetnosy: + schmir
messages: + msg62820
2006-12-01 20:05:34johnlallencreate