This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Python 3.7 sh hangs when using in threads, forking and logging
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: cagney, ebizzinfotech, gregory.p.smith, hugh, lukasz.langa, mzbuild, ned.deily
Priority: normal Keywords:

Created on 2019-08-01 17:58 by mzbuild, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (2)
msg348869 - (view) Author: mzbuild (mzbuild) Date: 2019-08-01 17:58
Hi,
We use the sh module in a threaded context to execute shell commands. When we were migrating to python 3 from python 2 we encountered some commands hanging. We created a test script that recreates the issue we are seeing.
`import sh
import logging
import concurrent.futures.thread

def execmd(exe):
print(sh.ls())
execution_pool = concurrent.futures.thread.ThreadPoolExecutor(20)
i = 0
thread_results = []
while i<500:
i+= 1
thread_results.append(execution_pool.map(execmd, ['ls']))

execution_pool.shutdown()

When running this in python 3.7 it hangs but in python 3.6 it works fine. We think it is releated to this issue https://bugs.python.org/issue36533. Installing the latest 3.7.4 didn't fix the issue.  The sh module uses logging and forking and the top script uses threading so we think there is a locking issue with 3.7.  Any help would be great.
msg348870 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-08-01 18:16
os.fork() cannot be safely used in an application that uses threads in any manner.  This is not something Python can fix.  This is a POSIX limitation.

The "sh" module on PyPI is incompatible with threaded applications on POSIX platforms due to its use of os.fork().

https://github.com/amoffat/sh/blob/master/sh.py#L1861

Raise this issue with them.
History
Date User Action Args
2022-04-11 14:59:18adminsetgithub: 81921
2019-08-01 18:16:42gregory.p.smithsetstatus: open -> closed
resolution: not a bug
messages: + msg348870

stage: resolved
2019-08-01 17:58:33mzbuildcreate