classification
Title: Multiprocessing Send to Manager Fails for Large Payload
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: problem using multiprocessing with really big objects?
View: 17560
Assigned To: Nosy List: ZackerySpytz, davin, maxehr, pitrou
Priority: normal Keywords:

Created on 2017-06-16 16:12 by maxehr, last changed 2019-07-25 15:37 by pitrou. This issue is now closed.

Messages (4)
msg296210 - (view) Author: Max Ehrlich (maxehr) Date: 2017-06-16 16:12
On line 393 of multiprocessing/connection.py, the size of the payload to be sent is serialized as an integer. This fails for sending large payloads. It should probably be serialized as a long or better yet a long long.
msg297210 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-28 19:12
As the comment just above that line says, this is for wire compatibility with older Python versions.

But we could have an escape hatch: a special value of the length (which happens to be encoded as a signed integer), such as -64, which would signal a following 64-bit length field.  Would you like to submit a PR?
msg348415 - (view) Author: Zackery Spytz (ZackerySpytz) * (Python triager) Date: 2019-07-24 23:30
It seems that this was fixed in bpo-17560.
msg348439 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2019-07-25 15:37
That's right. Thanks Zackery!
History
Date User Action Args
2019-07-25 15:37:14pitrousetstatus: open -> closed
superseder: problem using multiprocessing with really big objects?
messages: + msg348439

resolution: duplicate
stage: needs patch -> resolved
2019-07-24 23:30:33ZackerySpytzsetnosy: + ZackerySpytz
messages: + msg348415
2017-06-28 19:12:58pitrousetversions: + Python 3.7, - Python 3.5
nosy: + pitrou, davin

messages: + msg297210

type: enhancement
stage: needs patch
2017-06-16 16:12:46maxehrcreate