classification
Title: ctypes: segfault with large number of callback arguments
Type: crash Stage: needs patch
Components: ctypes Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: meador.inge Nosy List: amaury.forgeotdarc, belopolsky, meador.inge, vstinner
Priority: low Keywords:

Created on 2011-10-04 03:57 by meador.inge, last changed 2011-12-15 05:05 by meador.inge.

Messages (6)
msg144852 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-10-04 03:57
Reproducible in 2.7 and tip:

[meadori@motherbrain cpython]$ ./python 
Python 3.3.0a0 (default:61de28fa5537+d05350c14e77+, Oct  3 2011, 21:47:04) 
[GCC 4.6.0 20110603 (Red Hat 4.6.0-10)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes import *
>>> NARGS = 2 ** 20
>>> proto = CFUNCTYPE(None, *(c_int,) * NARGS)
>>> def func(*args):
...    return (1, "abc", None)
... 
>>> cb = proto(func)
>>> cb(*(1,) * NARGS)
Segmentation fault (core dumped)
msg145259 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-10-09 14:37
As mentioned in issue12881, this issue is a result of an unbounded 'alloca' call that trashes the stack.
msg148648 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-11-30 12:20
Right, alloca() could be replaced by some malloc(), but is it really useful?  After all, when a C function calls back to Python, all arguments needs to be pushed to the stack anyway.
msg148701 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-12-01 04:24
On Wed, Nov 30, 2011 at 6:20 AM, Amaury Forgeot d'Arc
<report@bugs.python.org> wrote:

> Right, alloca() could be replaced by some malloc(), but is it really useful?  After all, when a C function calls back to Python, all arguments needs to be > pushed to the stack anyway.

The case is somewhat pathological.  However, there are *four* 'alloca'
calls in '_ctypes_callproc', three of which are proportional to
'argcount'.  And as you point out there is an additional stack
allocation inside of 'libffi' when the callback is actually called
(also using 'alloca').

I see two reasons switching to 'malloc' might be beneficial: (1) by
shifting some of the allocation to dynamic allocation we may reduce
the chance that we blow the stack at all.  Keep in mind that this gets
more likely if the C ffi function is called from a deep in a call tree
and *not* necessarily with just a huge number of parameters.  (2) if
resources are exhausted, then we can exit more gracefully.  Out of
dynamic memory errors can actually be handled as opposed to an
'alloca' call that ends up allocating more than is left.

That being said, if this does get changed it is low priority.
msg148712 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-12-01 08:11
Is there really an use case where you need 2 ** 20 (1,048,576) arguments? If yes, I'm not against the torture in this case :-) If no, why not raising an error if there are too many arguments? E.g. limit to 1,024 arguments or maybe just 10?
msg149508 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-12-15 05:05
On Thu, Dec 1, 2011 at 2:11 AM, STINNER Victor <report@bugs.python.org> wrote:

> Is there really an use case where you need 2 ** 20 (1,048,576) arguments? If yes, I'm not against the torture in this case :-)

Not very likely :-)  However, the segfault can occur with less
arguments in low stack space situations (e.g. deep callstack).

> If no, why not raising an error if there are too many arguments? E.g. limit to 1,024 arguments or maybe just 10?

That is certainly an option.
History
Date User Action Args
2011-12-15 05:05:02meador.ingesetmessages: + msg149508
2011-12-01 08:11:10vstinnersetmessages: + msg148712
2011-12-01 04:24:55meador.ingesetpriority: normal -> low
2011-12-01 04:24:09meador.ingesetmessages: + msg148701
2011-11-30 12:20:03amaury.forgeotdarcsetmessages: + msg148648
2011-11-30 12:17:44vstinnersetnosy: + vstinner
2011-11-29 02:34:19meador.ingesetassignee: meador.inge
2011-10-09 14:37:05meador.ingesetmessages: + msg145259
2011-10-04 03:57:17meador.ingecreate