This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: SimpleQueues put blocks if feeded with strings greater than 2**16-13 chars
Type: Stage: resolved
Components: Versions: Python 3.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: iritkatriel, rgrewe
Priority: normal Keywords:

Created on 2020-08-14 09:36 by rgrewe, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
test_simple_queue_put.py rgrewe, 2020-08-14 09:36
Messages (7)
msg375385 - (view) Author: Raphael Grewe (rgrewe) Date: 2020-08-14 09:36
Hi at all,

Maybe I am using the put function incorrectly, but 
it is not possible to add larger strings there. The programm keeps locked and is doing nothing.

I tested it on latest Debian Buster. 
Python 3.7.3 (default, Jul 25 2020, 13:03:44)

Please check the example script.
python3 test_simple_queue_put.py 2**16-13
msg375393 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2020-08-14 10:24
On windows 10 it's hanging for me from string length of 8175. 

Stepping through the code, the hang is in the call to _winapi.WaitForMultipleObjects, in PipeConnection._send_bytes, Lib/multiprocessing/connection.py:288
msg375414 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2020-08-14 15:47
I've now reproduced it on osX, with string length of 65515. It takes a different code path than the windows version, and I was able to see more. 

This seems to be the sequence of events that leads to the hang:

import multiprocessing.reduction
import struct
import os
string_length = 65515
obj = "a"*string_length
obj = multiprocessing.reduction.ForkingPickler.dumps(obj)
m = memoryview(obj)
n = 65533
mm = m[0:n]
_, writer = os.pipe()
header = struct.pack("!i", n)
os.write(writer, header)
os.write(writer, mm)


I think what's happening is that the os.pipe that the queue is writing to has filled up, and then write blocks until any of it has been consumed.  This version of your script seems to confirm that:

import multiprocessing
import sys

q = multiprocessing.SimpleQueue()

string_length = eval(sys.argv[1])
print(string_length)
long_string = "a"*(string_length//2)
q.put(long_string)
q.get()                   <---- comment this out and it hangs again
q.put(long_string)

print("Never reach this line :-(")
msg375416 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2020-08-14 16:01
You probably need to use Queue instead of SimpleQueue: 
https://docs.python.org/3.8/library/multiprocessing.html#multiprocessing.Queue
msg375675 - (view) Author: Raphael Grewe (rgrewe) Date: 2020-08-19 20:39
I used SimpleQueue on purpose because I only need the functions get and put. 
Of course I could also use Queue but that would be just a workaround for me.
msg375676 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2020-08-19 21:01
The get and put functions of Queue have the optional 'block' and 'timeout' args that you need, and SimpleQueue doesn't.
msg375693 - (view) Author: Raphael Grewe (rgrewe) Date: 2020-08-20 08:51
Soo... I think I use the functions indeed incorrectly then.

Thank you for your time.

Regards

Raphael
History
Date User Action Args
2022-04-11 14:59:34adminsetgithub: 85722
2020-08-20 08:51:26rgrewesetstatus: open -> closed
resolution: not a bug
messages: + msg375693

stage: resolved
2020-08-19 21:01:27iritkatrielsetmessages: + msg375676
2020-08-19 20:39:43rgrewesetmessages: + msg375675
2020-08-14 16:01:35iritkatrielsetmessages: + msg375416
2020-08-14 15:47:19iritkatrielsetmessages: + msg375414
2020-08-14 10:24:06iritkatrielsetnosy: + iritkatriel
messages: + msg375393
2020-08-14 09:36:36rgrewecreate