Current pending calls limit is too small and it can be easily reached in very intensive async file io applications.

There is a little hack in pyaio[1] which sleeps if Py_AddPendingCall returns < 0 but It's not totally clear to me why the size of pendind calls array is only 32.


[1] https://github.com/felipecruz/pyaio

This array would still have a fixed size.
Could you arrange your code so that you enqueue pending calls elsewhere, and call Py_AddPendingCall only once until the pending items have been processed?

I just took a look at https://github.com/felipecruz/pyaio/commit/127372ba0a6dbca4045256dcd653789ee35f6a78 and it looks a bit silly to me: if the completion handler is called in a separate thread, then the completion handler can just take the GIL and put the results in a regular Python queue.

In short, using signals for any kind of high-speed I/O sounds completely archaic to me.

Just confirmed that signals is not a viable option. Is too slow, as Antonie already pointed. It's almost 5 times slower than with SIGEV_THREAD. 

The problem now is:

If I use Py_AddPendingCall, the tests can hang sometimes. I can still follow Amaury suggestion tough.

If I try to acquire the GIL PyGILState_Ensure() call PyObject_CallObject and release the GIL in the aio_completion_handler function a SEGFAULT occurs in 2.7.3 but not in 3.3 and newer..

This branch works only on 3.3 and newer: https://github.com/felipecruz/pyaio/tree/feature/check_no_pending_call

This other branch will segfault 2.7.3 with just acquire and release GIL on the completion handler:
https://github.com/felipecruz/pyaio/tree/feature/py27_gil_error

Since this looks very strange, can someone confirm this behavior?

Tested on a Ubuntu12 64.

> If I try to acquire the GIL PyGILState_Ensure() call
> PyObject_CallObject and release the GIL in the aio_completion_handler
> function a SEGFAULT occurs in 2.7.3 but not in 3.3 and newer..

Well, can you post a traceback?

Running test_aio_read
[New Thread 0x7ffff7ff7700 (LWP 20681)]
[New Thread 0x7ffff61ff700 (LWP 20682)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff61ff700 (LWP 20682)]
sem_post () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S:34
34	../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S: No such file or directory.
(gdb) backtrace
#0  sem_post () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S:34
#1  0x000000000051b053 in PyThread_release_lock (lock=0x0) at Python/thread_pthread.h:346
#2  0x00000000004c868f in PyEval_ReleaseLock () at Python/ceval.c:265
#3  0x0000000000501974 in PyThreadState_DeleteCurrent () at Python/pystate.c:321
#4  0x0000000000502116 in PyGILState_Release (oldstate=PyGILState_UNLOCKED) at Python/pystate.c:652
#5  0x00007ffff660ed44 in aio_completion_handler (sigval=...) at pyaio/core.c:26
#6  0x00007ffff6409cdc in notify_func_wrapper (arg=0x7ffff00008c0) at ../sysdeps/pthread/aio_notify.c:45
#7  0x00007ffff7bc4e9a in start_thread (arg=0x7ffff61ff700) at pthread_create.c:308
#8  0x00007ffff71f14bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#9  0x0000000000000000 in ?? ()

Is it with a debug Python build? `PyThread_release_lock (lock=0x0)` is a bit worrying.

Yes! 2.7.3 build with pydebug.

Without debug backtrace

#0  sem_post () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S:34
#1  0x00000000004d456e in PyGILState_Release ()
#2  0x00007ffff59b9cdc in notify_func_wrapper (arg=0x7ffff00008c0) at ../sysdeps/pthread/aio_notify.c:45
#3  0x00007ffff7bc4e9a in start_thread (arg=0x7ffff59b5700) at pthread_create.c:308
#4  0x00007ffff69b64bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()