This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: multiprocessing: Queue does not work in virtualenv but works fine in main interpreter
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Bernt.Røskar.Brenna, davin, pitrou
Priority: normal Keywords:

Created on 2019-03-22 08:51 by Bernt.Røskar.Brenna, last changed 2022-04-11 14:59 by admin.

Files
File name Uploaded Description Edit
mp_queue_example.py Bernt.Røskar.Brenna, 2019-03-22 08:51
Messages (4)
msg338591 - (view) Author: Bernt Røskar Brenna (Bernt.Røskar.Brenna) * Date: 2019-03-22 08:51
When executing mp_queue_example.py in the system interpreter:

D:\dev\eva_v_next>d:\python372\python.exe mp_queue_example.py
main, pid=892, executable=d:\python372\python.exe
process_worker, pid=12848, executable=d:\python372\python.exe
submitting...
submitting 0
submitting 1
submitting 2
Executing job 0
Executing job 1
Executing job 2

When executing mp_queue_example.py in a virtualenv:

D:\dev\eva_v_next>VENV\Scripts\python.exe mp_queue_example.py
main, pid=25888, executable=D:\dev\eva_v_next\VENV\Scripts\python.exe
process_worker, pid=28144, executable=D:\dev\eva_v_next\VENV\Scripts\python.exe
submitting...
submitting 0
submitting 1
submitting 2

## Here it hangs, Ctrl-C gives this:

Process Process-1:
Traceback (most recent call last):
  File "D:\dev\eva_v_next\mp_queue_example.py", line 13, in process_worker
    inp = input_q.get_nowait()
  File "D:\python372\lib\multiprocessing\queues.py", line 126, in get_nowait
    return self.get(False)
  File "D:\python372\lib\multiprocessing\queues.py", line 100, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\python372\lib\multiprocessing\process.py", line 297, in _bootstrap
    self.run()
  File "D:\python372\lib\multiprocessing\process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "D:\dev\eva_v_next\mp_queue_example.py", line 13, in process_worker
    inp = input_q.get_nowait()
KeyboardInterrupt
Traceback (most recent call last):
  File "mp_queue_example.py", line 43, in <module>
    main_process_experiment()
  File "mp_queue_example.py", line 39, in main_process_experiment
    p.join()
  File "D:\python372\lib\multiprocessing\process.py", line 140, in join
    res = self._popen.wait(timeout)
  File "D:\python372\lib\multiprocessing\popen_spawn_win32.py", line 80, in wait
    res = _winapi.WaitForSingleObject(int(self._handle), msecs)
KeyboardInterrupt


---

mp_queue_example.py:

import multiprocessing as mp
from queue import Empty
import os
import sys
import time


def process_worker(input_q: mp.Queue):
    print(f'process_worker, pid={os.getpid()}, executable={sys.executable}')
    while True:
        try:
            inp = input_q.get_nowait()
            if inp == 'STOP':
                break
            execute_job(inp)
        except Empty:
            pass


def execute_job(input_args):
    print(f'Executing job {input_args}')


def main_process_experiment():
    print(f"main, pid={os.getpid()}, executable={sys.executable}")
    input_q = mp.Queue()

    p = mp.Process(target=process_worker, args=(input_q, ))
    p.start()
    time.sleep(0.5)

    print('submitting...')
    for i in range(3):
        print(f'submitting {i}')
        input_q.put(i)
    input_q.put('STOP')

    p.join()


if __name__ == '__main__':
    main_process_experiment()
msg338592 - (view) Author: Bernt Røskar Brenna (Bernt.Røskar.Brenna) * Date: 2019-03-22 09:00
I just checked using 3.6. The program does NOT hang when executed from a 3.6 virtualenv. So this is a possible regression (or something could be messed up on my system).

D:\dev\eva_v_next>VENV36\Scripts\python.exe mp_queue_example.py
main, pid=28956, executable=D:\dev\eva_v_next\VENV36\Scripts\python.exe
process_worker, pid=8924, executable=D:\dev\eva_v_next\VENV36\Scripts\python.exe
submitting...
submitting 0
submitting 1
submitting 2
Executing job 0
Executing job 1
Executing job 2
msg338593 - (view) Author: Bernt Røskar Brenna (Bernt.Røskar.Brenna) * Date: 2019-03-22 09:04
Versions:

> VENV\Scripts\python.exe --version
Python 3.7.2

> VENV36\Scripts\python.exe --version
Python 3.6.8
msg338594 - (view) Author: Bernt Røskar Brenna (Bernt.Røskar.Brenna) * Date: 2019-03-22 09:10
I also checked using 3.8 (built from commit 1f58f4fa6a0e3c60cee8df4a35c8dcf3903acde8), and it works there, so it looks as if 3.7 is the problem.

> VENV38\Scripts\python.exe --version
Python 3.8.0a2+

> VENV38\Scripts\python.exe mp_queue_example.py
main, pid=8284, executable=D:\dev\eva_v_next\VENV38\Scripts\python.exe
process_worker, pid=16396, executable=D:\dev\cpython\PCbuild\win32\python.exe
submitting...
submitting 0
submitting 1
submitting 2
Executing job 0
Executing job 1
Executing job 2
History
Date User Action Args
2022-04-11 14:59:12adminsetgithub: 80580
2019-03-22 09:10:54Bernt.Røskar.Brennasetmessages: + msg338594
2019-03-22 09:04:45Bernt.Røskar.Brennasetmessages: + msg338593
2019-03-22 09:01:18SilentGhostsetnosy: + pitrou, davin
2019-03-22 09:00:35Bernt.Røskar.Brennasetmessages: + msg338592
2019-03-22 08:51:16Bernt.Røskar.Brennacreate