classification
Title: _struct.Struct: calling functions without calling __init__ results in SystemError
Type: crash Stage: patch review
Components: Extension Modules Versions: Python 3.9, Python 3.8, Python 3.7, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: DeKrain, ZackerySpytz, ronaldoussoren, steven.daprano
Priority: normal Keywords: patch

Created on 2018-08-29 16:18 by DeKrain, last changed 2019-07-14 22:40 by ZackerySpytz.

Pull Requests
URL Status Linked Edit
PR 14777 open ZackerySpytz, 2019-07-14 22:35
Messages (11)
msg324330 - (view) Author: (DeKrain) Date: 2018-08-29 16:18
>>> from _struct import Struct
>>> s = Struct.__new__(Struct)
>>> s.unpack_from(b'asdf')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SystemError: /Objects/tupleobject.c:84: Bad argument to internal function

In Modules/_struct.c:

static PyObject *
s_unpack_internal(PyStructObject *soself, const char *startfrom) {
...
PyObject *result = PyTuple_New(soself->s_len);
// soself->s_len is -1, set in Struct.__new__
msg324331 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2018-08-29 16:43
This exception goes back to at least Python 2.6 (if not older) but I'm not convinced it is a bug.

Calling __new__ alone is not guaranteed to initialise a new instance completely. The public API for creating an instance is to call the class object:

    s = Struct()


not to call __new__. You bypassed the proper initialisation of the instance, resulting in a broken, half-initialised instance. When you tried to use it, it correctly raised an exception.

If this caused a crash or a seg fault, then it would be reasonable to report it as a bug, but it looks to me that this is behaving correctly.

If you disagree, please explain why you think it is a bug.


(Also, for the record, you shouldn't be importing Struct from the private module _struct, you should import it from the public struct module.)
msg324335 - (view) Author: (DeKrain) Date: 2018-08-29 17:10
Well, sometimes when i do
>>> b = bytearray()
>>> s.pack_into(b)

application crashes (because it checks arg #1, which is not initialized).
Also, I imported from _struct, because it's where implementation of Struct really is.
msg324338 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2018-08-29 17:28
_struct is a private implementation detail. You shouldn't use it. You shouldn't care where the implementation "really is" in your Python code, because it could move without warning. There are no backwards-compatibility guarantees for private modules like _struct.

But regardless of where you are importing it from, why are you calling Struct.__new__(Struct) in the first place? You should be calling Struct().

I still don't see any reason to consider this a bug. I can't reproduce your report of a crash:

py> from _struct import Struct
py> s = Struct.__new__(Struct)
py> b = bytearray()
py> s.pack_into(b)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SystemError: null argument to internal routine


I get an exception, which is the correct behaviour. Unless this segfaults, I don't believe this is a bug that needs fixing.

(By the way, Struct doesn't even have a __new__ method. You are calling the __new__ method inherited from object, which clearly knows nothing about how to initialise a Struct.)
msg324341 - (view) Author: (DeKrain) Date: 2018-08-29 17:37
(I wrote that I'm importing from _struct just for this issue.)
I've seen that tp_new of PyStructType is set to s_new in Modules/_struct.c.
And that crash is most likely caused by access to uninitialized memory, so it is not guaranteed.
msg324484 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2018-09-02 23:27
I've tried running this code in Python 3.6:

from _struct import Struct
for i in range(100000):
    L = [Struct.__new__(Struct) for j in range(1000)]
    for s in L:
        try:
            x = s.pack_into(bytearray())
        except SystemError:
            pass

I've run it 6 times, for a total of 600 million calls to Struct.__new__ 
and pack_into, and I cannot reproduce any crash or segfault. An 
exception (SystemError) is the correct behaviour.

Is anyone able to try it under Python 3.7?

Unless somebody is able to demonstrate a segfault or core dump, or 
otherwise demonstrate a problem with the C code, I think this ticket 
ought to be closed.
msg324498 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2018-09-03 07:06
IMHO SystemError is the wrong exception, that exception is primarily used to signal implementation errors.

BTW. I can reproduce crashes in a couple of runs of your scriptlet:

Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 23:26:24) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from _struct import Struct
>>> for i in range(100000):
...     L = [Struct.__new__(Struct) for j in range(1000)]
...     for s in L:
...         try:
...             x = s.pack_into(bytearray())
...         except SystemError:
...             pass
... 
Traceback (most recent call last):
  File "<stdin>", line 5, in <module>
TypeError: 'code' object cannot be interpreted as an integer
>>>             
>>> from _struct import Struct
>>> for i in range(100000):
...     L = [Struct.__new__(Struct) for j in range(1000)]
...     for s in L:
...         try:
...             x = s.pack_into(bytearray())
...         except SystemError:
...             pass
... 
Traceback (most recent call last):
  File "<stdin>", line 5, in <module>
TypeError: 'traceback' object cannot be interpreted as an integer
>>> 
>>> 
>>> 
>>> from _struct import Struct
>>> for i in range(100000):
...     L = [Struct.__new__(Struct) for j in range(1000)]
...     for s in L:
...         try:
...             x = s.pack_into(bytearray())
...         except SystemError:
...             pass
... 
Segmentation fault: 11
msg324504 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2018-09-03 10:22
Thanks for confirming the seg fault. I've changed this to a crasher.

Should we change the exception to RuntimeError?
msg324505 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2018-09-03 11:05
It's not as easy as that, the SystemError in the original report is caused by invalid use of a C-API due to partial initialisation of an _struct.Struct instance.

The solution is likely two-fold:

1) Ensure that __new__ fully initialises the fields in de C struct to some value

2) (Possibly) check that fields in the C structure have a sane value before using them. This part can have a measurable performance cost, and it would be nicer to avoid this by picking smart values in (1). 

The most important bit is the first step, even if that keeps raising SystemError when only calling Struct.__new__ because this avoid crashing the interpreter.
msg324507 - (view) Author: (DeKrain) Date: 2018-09-03 11:59
I think we should leave 'Extension Modules' in components field, because implementation of struct module is really written in C.
msg324509 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2018-09-03 12:16
@DeKrain: I agree
History
Date User Action Args
2019-07-14 22:40:39ZackerySpytzsetnosy: + ZackerySpytz

versions: + Python 2.7, Python 3.8, Python 3.9
2019-07-14 22:35:08ZackerySpytzsetkeywords: + patch
stage: patch review
pull_requests: + pull_request14573
2018-09-03 12:16:18ronaldoussorensetmessages: + msg324509
components: + Extension Modules, - Library (Lib)
2018-09-03 11:59:23DeKrainsetmessages: + msg324507
2018-09-03 11:05:03ronaldoussorensetmessages: + msg324505
2018-09-03 10:22:54steven.dapranosettype: behavior -> crash
messages: + msg324504
components: + Library (Lib), - Extension Modules
2018-09-03 07:06:18ronaldoussorensetmessages: + msg324498
2018-09-02 23:27:59steven.dapranosetmessages: + msg324484
2018-08-31 14:27:15ronaldoussorensetnosy: + ronaldoussoren
2018-08-29 17:37:20DeKrainsetmessages: + msg324341
2018-08-29 17:28:21steven.dapranosetmessages: + msg324338
2018-08-29 17:10:25DeKrainsetmessages: + msg324335
2018-08-29 16:43:51steven.dapranosetnosy: + steven.daprano
messages: + msg324331
2018-08-29 16:18:52DeKraincreate