classification
Title: Increase Py_AddPendingCall array size
Type: Stage:
Components: IO Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: amaury.forgeotdarc, felipecruz, jcea, pitrou
Priority: normal Keywords:

Created on 2012-11-27 20:37 by felipecruz, last changed 2017-09-04 23:43 by gregory.p.smith.

Messages (9)
msg176491 - (view) Author: Felipe Cruz (felipecruz) * Date: 2012-11-27 20:37
Current pending calls limit is too small and it can be easily reached in very intensive async file io applications.

There is a little hack in pyaio[1] which sleeps if Py_AddPendingCall returns < 0 but It's not totally clear to me why the size of pendind calls array is only 32.


[1] https://github.com/felipecruz/pyaio
msg176568 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2012-11-28 19:17
This array would still have a fixed size.
Could you arrange your code so that you enqueue pending calls elsewhere, and call Py_AddPendingCall only once until the pending items have been processed?
msg176569 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-11-28 19:24
I just took a look at https://github.com/felipecruz/pyaio/commit/127372ba0a6dbca4045256dcd653789ee35f6a78 and it looks a bit silly to me: if the completion handler is called in a separate thread, then the completion handler can just take the GIL and put the results in a regular Python queue.

In short, using signals for any kind of high-speed I/O sounds completely archaic to me.
msg176613 - (view) Author: Felipe Cruz (felipecruz) * Date: 2012-11-29 02:47
Just confirmed that signals is not a viable option. Is too slow, as Antonie already pointed. It's almost 5 times slower than with SIGEV_THREAD. 

The problem now is:

If I use Py_AddPendingCall, the tests can hang sometimes. I can still follow Amaury suggestion tough.

If I try to acquire the GIL PyGILState_Ensure() call PyObject_CallObject and release the GIL in the aio_completion_handler function a SEGFAULT occurs in 2.7.3 but not in 3.3 and newer..

This branch works only on 3.3 and newer: https://github.com/felipecruz/pyaio/tree/feature/check_no_pending_call

This other branch will segfault 2.7.3 with just acquire and release GIL on the completion handler:
https://github.com/felipecruz/pyaio/tree/feature/py27_gil_error

Since this looks very strange, can someone confirm this behavior?

Tested on a Ubuntu12 64.
msg176633 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-11-29 07:02
> If I try to acquire the GIL PyGILState_Ensure() call
> PyObject_CallObject and release the GIL in the aio_completion_handler
> function a SEGFAULT occurs in 2.7.3 but not in 3.3 and newer..

Well, can you post a traceback?
msg176644 - (view) Author: Felipe Cruz (felipecruz) * Date: 2012-11-29 12:30
Running test_aio_read
[New Thread 0x7ffff7ff7700 (LWP 20681)]
[New Thread 0x7ffff61ff700 (LWP 20682)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff61ff700 (LWP 20682)]
sem_post () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S:34
34	../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S: No such file or directory.
(gdb) backtrace
#0  sem_post () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S:34
#1  0x000000000051b053 in PyThread_release_lock (lock=0x0) at Python/thread_pthread.h:346
#2  0x00000000004c868f in PyEval_ReleaseLock () at Python/ceval.c:265
#3  0x0000000000501974 in PyThreadState_DeleteCurrent () at Python/pystate.c:321
#4  0x0000000000502116 in PyGILState_Release (oldstate=PyGILState_UNLOCKED) at Python/pystate.c:652
#5  0x00007ffff660ed44 in aio_completion_handler (sigval=...) at pyaio/core.c:26
#6  0x00007ffff6409cdc in notify_func_wrapper (arg=0x7ffff00008c0) at ../sysdeps/pthread/aio_notify.c:45
#7  0x00007ffff7bc4e9a in start_thread (arg=0x7ffff61ff700) at pthread_create.c:308
#8  0x00007ffff71f14bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#9  0x0000000000000000 in ?? ()
msg176649 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-11-29 13:34
Is it with a debug Python build? `PyThread_release_lock (lock=0x0)` is a bit worrying.
msg176651 - (view) Author: Felipe Cruz (felipecruz) * Date: 2012-11-29 14:01
Yes! 2.7.3 build with pydebug.
msg176659 - (view) Author: Felipe Cruz (felipecruz) * Date: 2012-11-29 16:08
Without debug backtrace

#0  sem_post () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S:34
#1  0x00000000004d456e in PyGILState_Release ()
#2  0x00007ffff59b9cdc in notify_func_wrapper (arg=0x7ffff00008c0) at ../sysdeps/pthread/aio_notify.c:45
#3  0x00007ffff7bc4e9a in start_thread (arg=0x7ffff59b5700) at pthread_create.c:308
#4  0x00007ffff69b64bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()
History
Date User Action Args
2017-09-04 23:43:42gregory.p.smithsetversions: + Python 3.7, - Python 2.7, Python 3.3, Python 3.4, Python 3.5
2012-11-29 16:08:55felipecruzsetmessages: + msg176659
2012-11-29 14:01:45felipecruzsetmessages: + msg176651
2012-11-29 13:34:45pitrousetmessages: + msg176649
2012-11-29 12:30:28felipecruzsetmessages: + msg176644
2012-11-29 07:02:10pitrousetmessages: + msg176633
2012-11-29 02:47:49felipecruzsetmessages: + msg176613
2012-11-28 19:24:43pitrousetnosy: + pitrou
messages: + msg176569
2012-11-28 19:17:06amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg176568
2012-11-28 15:56:27jceasetnosy: + jcea
2012-11-27 20:37:34felipecruzcreate