classification
Title: Pickling deadlocks in thread with python -m
Type: behavior Stage: resolved
Components: Versions: Python 3.8, Python 2.7
process
Status: closed Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Werner Smidt, iritkatriel, serakeri
Priority: normal Keywords:

Created on 2017-11-09 10:14 by Werner Smidt, last changed 2020-12-03 09:23 by Werner Smidt. This issue is now closed.

Messages (9)
msg305953 - (view) Author: Werner Smidt (Werner Smidt) Date: 2017-11-09 10:14
Hi there

I recently stumbled on an interesting behaviour.  I won't call it an error, because I think it's a mistake I made.  

BACKGROUND: I want to spawn threads that handle pickled data.  This works really well. However, I would like to execute the python script in question as a module, i.e. python -m mymodule. This is merely for aesthetic purposes.

The attached script has two functions:
1. Pickle/unpickle an instance of a `namedtuple`
2. Pickle/unpickle a string

Each of these functions are run in the main thread and then in subsequent spawned threads. 

If I run the script attached with "python testqueuepickle.py", it works fine.  I get the data pickled/unpickled in the respective functions and nothing deadlocks and everything is printed to screen. If, however, I run it with the "-m" option (python -m testqueuepickle.py) , the program deadlocks at the pickling of the "namedtuple" instance. The pickling/unpickling of the string appears to be unaffected. 

Programming practices aside, what do you think could be the cause of this?
msg382182 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2020-11-30 19:29
I don't see the hang on Linux (3.7) or windows (3.10). I think this might be a python 2-only issue.

There were problems in Python 2 with importer race conditions, and since you start threads at import time, that could be what was causing what you saw.

Python 2.7 is no longer maintained, so unless the problems is relevant to Python 3 this issue can be closed.
msg382190 - (view) Author: Werner Smidt (Werner Smidt) Date: 2020-11-30 20:30
The condition still stands.

if I execute:

python3 testqueuepickle3.py

Everything is fine. If, however I execute:

python3 -m testqueuepickle3.py

It hangs.
msg382191 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2020-11-30 20:54
Which system are seeing this on?
msg382229 - (view) Author: Sara Kelley (serakeri) Date: 2020-12-01 10:30
>python3 -m testqueuepickle3.py
>It hangs.

I see this hang too.

If you only specify the module name it does not hang.
python3 -m testqueuepickle3
msg382239 - (view) Author: Werner Smidt (Werner Smidt) Date: 2020-12-01 11:35
Sorry for being lacking in providing some OS info.  b

Opensuse Tumbleweed, Linux kernel 5.8.10-1, Intel system

I cannot explain why, but relating to Sara's answer, if you remove the .join() statements at the end, you get the following exception:

/usr/bin/python3: Error while finding module specification for 'testqueuepickle3.py' (ModuleNotFoundError: __path__ attribute not found on 'testqueuepickle3' while trying to find 'testqueuepickle3.py')
mynamedtuple(param1='INSIDE thread', param2='namedtuple')


So I guess that it gets stuck after an exception is thrown?  If I call it Sara's way, there is no exception thrown. 

Sorry, I know this is a very specific case, but I thank you both for taking the time to contribute :-)
msg382359 - (view) Author: Sara Kelley (serakeri) Date: 2020-12-03 00:07
https://docs.python.org/3/using/cmdline.html#cmdoption-m

The documentation says you must not give a file extension with the module option. Because the extension was given the thread is deadlocking in importlib._bootstrap when trying to acquire the module lock. The hang occurs in the script because join() is waiting for the deadlocked thread.
msg382370 - (view) Author: Sara Kelley (serakeri) Date: 2020-12-03 03:09
After looking a little more I don't believe the extension is the problem, but it is an easier way to see the problem. When testqueuepickle3 is run as a script you don't see the error. When it is imported as a module, even from another script with valid syntax, it does deadlock.
msg382393 - (view) Author: Werner Smidt (Werner Smidt) Date: 2020-12-03 09:23
Thanks for going to the trouble, Sara. 


Curiosity remains, but I'll mark this as closed.
History
Date User Action Args
2020-12-03 09:23:11Werner Smidtsetstatus: open -> closed

messages: + msg382393
stage: resolved
2020-12-03 03:09:14serakerisetmessages: + msg382370
2020-12-03 00:07:42serakerisetmessages: + msg382359
2020-12-01 11:35:56Werner Smidtsetmessages: + msg382239
2020-12-01 10:30:55serakerisetnosy: + serakeri
messages: + msg382229
2020-11-30 20:54:20iritkatrielsetmessages: + msg382191
2020-11-30 20:30:58Werner Smidtsetmessages: + msg382190
versions: + Python 3.8
2020-11-30 20:28:22Werner Smidtsetfiles: - testqueuepickle.py
status: pending -> open
2020-11-30 19:29:29iritkatrielsetstatus: open -> pending
nosy: + iritkatriel
messages: + msg382182

2017-11-09 10:14:19Werner Smidtcreate