>>> from _struct import Struct
>>> s = Struct.__new__(Struct)
>>> s.unpack_from(b'asdf')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SystemError: /Objects/tupleobject.c:84: Bad argument to internal function

In Modules/_struct.c:

static PyObject *
s_unpack_internal(PyStructObject *soself, const char *startfrom) {
...
PyObject *result = PyTuple_New(soself->s_len);
// soself->s_len is -1, set in Struct.__new__

This exception goes back to at least Python 2.6 (if not older) but I'm not convinced it is a bug.

Calling __new__ alone is not guaranteed to initialise a new instance completely. The public API for creating an instance is to call the class object:

    s = Struct()


not to call __new__. You bypassed the proper initialisation of the instance, resulting in a broken, half-initialised instance. When you tried to use it, it correctly raised an exception.

If this caused a crash or a seg fault, then it would be reasonable to report it as a bug, but it looks to me that this is behaving correctly.

If you disagree, please explain why you think it is a bug.


(Also, for the record, you shouldn't be importing Struct from the private module _struct, you should import it from the public struct module.)

Well, sometimes when i do
>>> b = bytearray()
>>> s.pack_into(b)

application crashes (because it checks arg #1, which is not initialized).
Also, I imported from _struct, because it's where implementation of Struct really is.

_struct is a private implementation detail. You shouldn't use it. You shouldn't care where the implementation "really is" in your Python code, because it could move without warning. There are no backwards-compatibility guarantees for private modules like _struct.

But regardless of where you are importing it from, why are you calling Struct.__new__(Struct) in the first place? You should be calling Struct().

I still don't see any reason to consider this a bug. I can't reproduce your report of a crash:

py> from _struct import Struct
py> s = Struct.__new__(Struct)
py> b = bytearray()
py> s.pack_into(b)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SystemError: null argument to internal routine


I get an exception, which is the correct behaviour. Unless this segfaults, I don't believe this is a bug that needs fixing.

(By the way, Struct doesn't even have a __new__ method. You are calling the __new__ method inherited from object, which clearly knows nothing about how to initialise a Struct.)

(I wrote that I'm importing from _struct just for this issue.)
I've seen that tp_new of PyStructType is set to s_new in Modules/_struct.c.
And that crash is most likely caused by access to uninitialized memory, so it is not guaranteed.

I've tried running this code in Python 3.6:

from _struct import Struct
for i in range(100000):
    L = [Struct.__new__(Struct) for j in range(1000)]
    for s in L:
        try:
            x = s.pack_into(bytearray())
        except SystemError:
            pass

I've run it 6 times, for a total of 600 million calls to Struct.__new__ 
and pack_into, and I cannot reproduce any crash or segfault. An 
exception (SystemError) is the correct behaviour.

Is anyone able to try it under Python 3.7?

Unless somebody is able to demonstrate a segfault or core dump, or 
otherwise demonstrate a problem with the C code, I think this ticket 
ought to be closed.

IMHO SystemError is the wrong exception, that exception is primarily used to signal implementation errors.

BTW. I can reproduce crashes in a couple of runs of your scriptlet:

Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 23:26:24) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from _struct import Struct
>>> for i in range(100000):
...     L = [Struct.__new__(Struct) for j in range(1000)]
...     for s in L:
...         try:
...             x = s.pack_into(bytearray())
...         except SystemError:
...             pass
... 
Traceback (most recent call last):
  File "<stdin>", line 5, in <module>
TypeError: 'code' object cannot be interpreted as an integer
>>>             
>>> from _struct import Struct
>>> for i in range(100000):
...     L = [Struct.__new__(Struct) for j in range(1000)]
...     for s in L:
...         try:
...             x = s.pack_into(bytearray())
...         except SystemError:
...             pass
... 
Traceback (most recent call last):
  File "<stdin>", line 5, in <module>
TypeError: 'traceback' object cannot be interpreted as an integer
>>> 
>>> 
>>> 
>>> from _struct import Struct
>>> for i in range(100000):
...     L = [Struct.__new__(Struct) for j in range(1000)]
...     for s in L:
...         try:
...             x = s.pack_into(bytearray())
...         except SystemError:
...             pass
... 
Segmentation fault: 11

Thanks for confirming the seg fault. I've changed this to a crasher.

Should we change the exception to RuntimeError?

It's not as easy as that, the SystemError in the original report is caused by invalid use of a C-API due to partial initialisation of an _struct.Struct instance.

The solution is likely two-fold:

1) Ensure that __new__ fully initialises the fields in de C struct to some value

2) (Possibly) check that fields in the C structure have a sane value before using them. This part can have a measurable performance cost, and it would be nicer to avoid this by picking smart values in (1). 

The most important bit is the first step, even if that keeps raising SystemError when only calling Struct.__new__ because this avoid crashing the interpreter.

I think we should leave 'Extension Modules' in components field, because implementation of struct module is really written in C.

@DeKrain: I agree

Reproduced on 3.11:

>>> from _struct import Struct
>>> s = Struct.__new__(Struct)
>>> s.unpack_from(b'asdf')
Assertion failed: (self->s_codes != NULL), function Struct_unpack_from_impl, file /Users/iritkatriel/src/cpython/Modules/_struct.c, line 1603.
zsh: abort      ./python.exe