classification
Title: too small type for struct.pack/unpack in mutliprocessing.Connection
Type: crash Stage: resolved
Components: Library (Lib) Versions: Python 3.8, Python 3.7, Python 3.6, Python 3.5
process
Status: closed Resolution: duplicate
Dependencies: Superseder: problem using multiprocessing with really big objects?
View: 17560
Assigned To: Nosy List: ahcub, xtreak
Priority: normal Keywords: patch

Created on 2018-11-03 08:00 by ahcub, last changed 2018-11-03 08:21 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
error_on_bug_multiprocessing_output.py ahcub, 2018-11-03 08:00
Pull Requests
URL Status Linked Edit
PR 10305 open ahcub, 2018-11-03 08:03
Messages (2)
msg329189 - (view) Author: Oleksandr Buchkovskyi (ahcub) * Date: 2018-11-03 08:00
the problem is reproduced on big multiprocessing.Process output
when the size of the output gets bigger than signed int a struct error is raised
```
python3 test.py
Process ForkPoolWorker-1:
Traceback (most recent call last):
  File "/usr/lib64/python3.5/multiprocessing/pool.py", line 125, in worker
    put((job, i, result))
  File "/usr/lib64/python3.5/multiprocessing/queues.py", line 347, in put
    self._writer.send_bytes(obj)
  File "/usr/lib64/python3.5/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/usr/lib64/python3.5/multiprocessing/connection.py", line 393, in _send_bytes
    header = struct.pack("!i", n)
struct.error: 'i' format requires -2147483648 <= number <= 2147483647

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.5/multiprocessing/process.py", line 252, in _bootstrap
    self.run()
  File "/usr/lib64/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python3.5/multiprocessing/pool.py", line 130, in worker
    put((job, i, (False, wrapped)))
  File "/usr/lib64/python3.5/multiprocessing/queues.py", line 347, in put
    self._writer.send_bytes(obj)
  File "/usr/lib64/python3.5/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/usr/lib64/python3.5/multiprocessing/connection.py", line 393, in _send_bytes
    header = struct.pack("!i", n)
struct.error: 'i' format requires -2147483648 <= number <= 2147483647
```

this error can be fixed by changing types that are used herу for struct.pack/unpack to !Q
msg329190 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2018-11-03 08:17
Thanks for the report and PR. I think these are related issues : issue17560, issue28506 with issue17560 being open one. Correct me if I am wrong here.
History
Date User Action Args
2018-11-03 08:21:36serhiy.storchakasetstatus: open -> closed
superseder: problem using multiprocessing with really big objects?
resolution: duplicate
stage: patch review -> resolved
2018-11-03 08:17:28xtreaksetnosy: + xtreak
messages: + msg329190
2018-11-03 08:03:28ahcubsetkeywords: + patch
stage: patch review
pull_requests: + pull_request9613
2018-11-03 08:00:06ahcubcreate