Some posix module functions unnecessarily release the GIL.
For example, posix_dup, posix_dup2 and posix_pipe all release the GIL, but those are non-blocking syscalls (the don't imply any I/O, only modifying the process file descriptors table).
This leads to the famous convoy effect (see http://bugs.python.org/issue7946).

For example:

$ cat /tmp/test_dup2.py 
import os
import threading
import sys
import time


def do_loop():
    while True:
        pass

t = threading.Thread(target=do_loop)
t.setDaemon(True)
t.start()

f = os.open(sys.argv[1], os.O_RDONLY)

for i in range(4, 1000):
    os.dup2(f, i)

Whith  GIL release/acquire:

$ time ./python /tmp/test_dup2.py  /etc/fstab 

real    0m5.238s
user    0m5.223s
sys     0m0.009s

$ time ./python /tmp/test_pipe.py 

real    0m3.083s
user    0m3.074s
sys     0m0.007s

Without GIL release/acquire:

$ time ./python /tmp/test_dup2.py  /etc/fstab 

real    0m0.094s
user    0m0.077s
sys     0m0.010s

$ time ./python /tmp/test_pipe.py 

real    0m0.088s
user    0m0.074s
sys     0m0.008s

On Windows at least, these functions call malloc() and DuplicateHandle().

I didn't even know that Windows had such calls.
But anyway, if we start releasing the GIL around each malloc call, then it's going to get really complicated:

static PyObject *
posix_geteuid(PyObject *self, PyObject *noargs)
{
 	return PyLong_FromLong((long)geteuid());
}

PyLong_FromLong -> _PyLong_New -> PyObject_MALLOC which can call malloc.

As for DuplicateHandle, I assume it's as fast as Unix's dup(2).

What are you trying to achieve? Do you have loops which contain no other syscall than os.dup2()?

Well, those are contrived examples showing the effect of the convoy effect induced by those unneeded GIL release/acquire: releasing and re-acquiring the GIL comes with a cost (e.g. under Linux, futex are really fast in the uncontended case since handled in use space but much slower when there's contention), and subverts the OS scheduling policy (forcing the thread to drop/re-acquire the GIL make the thread block after having consumed a small amount of its time slice and increases the context switching rate). I think that releasing and re-acquiring the GIL should only be done around potentially blocking calls.

> Do you have loops which contain no other syscall than os.dup2()?

No, but it's not a reason for penalizing threads that use dup, dup2 or pipe.

> Well, those are contrived examples showing the effect of the convoy
> effect induced by those unneeded GIL release/acquire: releasing and
> re-acquiring the GIL comes with a cost (e.g. under Linux, futex are
> really fast in the uncontended case since handled in use space but
> much slower when there's contention), and subverts the OS scheduling
> policy (forcing the thread to drop/re-acquire the GIL make the thread
> block after having consumed a small amount of its time slice and
> increases the context switching rate). I think that releasing and
> re-acquiring the GIL should only be done around potentially blocking
> calls.

Do you want to propose a patch?

> Do you want to propose a patch?

Sure, if removing those calls to Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS seems reasonable (I might haved missed something obvious).
Just to be clear, I'm not at all criticizing the current GIL implementation, there's been a great work done on it.
I'm just saying that releasing and re-acquiring the GIL around fast syscalls is probaly not a good idea.

> Just to be clear, I'm not at all criticizing the current GIL
> implementation, there's been a great work done on it.
> I'm just saying that releasing and re-acquiring the GIL around fast
> syscalls is probaly not a good idea.

If these syscalls aren't likely to yield control to another thread, then
I agree there's no point in releasing the GIL around them.
(but is it the case that they are always fast? for example, how about
dup() on a network file system? or is it indifferent?)

2011/3/3 Antoine Pitrou <report@bugs.python.org>:
>
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
>> Just to be clear, I'm not at all criticizing the current GIL
>> implementation, there's been a great work done on it.
>> I'm just saying that releasing and re-acquiring the GIL around fast
>> syscalls is probaly not a good idea.
>
> If these syscalls aren't likely to yield control to another thread, then
> I agree there's no point in releasing the GIL around them.
> (but is it the case that they are always fast? for example, how about
> dup() on a network file system? or is it indifferent?)

The initial open can take long, but once it's open, calling dup just
implies copying a reference to the open file (a pointer) to the file
descriptor table. No I/O is done (I tested it one a NFS mount).
Now, I don't know Windows at all, but I'm pretty sure that every
operating system does more or less the same thing, and that those
three calls (there might be others) don't block.

>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue11382>
> _______________________________________
>

Attached is a patch removing useless calls to
Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS for several posix
functions.
It's straigthforward, but since I only have Linux boxes, I couldn't
test it under Windows.

Is there anything I can do to help this move forward ?

New changeset eb7297fd5840 by Antoine Pitrou in branch 'default':
Issue #11382: Trivial system calls, such as dup() or pipe(), needn't
http://hg.python.org/cpython/rev/eb7297fd5840

Sorry for the delay, I had forgotten about this issue. Thank you for the patch!