Title: OSError: handle is closed for ProcessPoolExecutor and run_in_executor
Type: crash Stage:
Components: asyncio Versions: Python 3.8, Python 3.7
Status: pending Resolution: duplicate
Dependencies: Superseder: ProcessPoolExecutor hangs on shutdown nowait with pickling failure
View: 39104
Assigned To: Nosy List: JoeKun, asvetlov, basnijholt, iritkatriel, yselivanov
Priority: normal Keywords:

Created on 2019-03-13 17:36 by basnijholt, last changed 2021-10-18 16:33 by iritkatriel.

Messages (3)
msg337868 - (view) Author: Bas Nijholt (basnijholt) Date: 2019-03-13 17:36
The following code in Python 3.7.1
import random
import concurrent.futures
import asyncio

executor = concurrent.futures.ProcessPoolExecutor()
ioloop = asyncio.get_event_loop()

async def func():
    result = await ioloop.run_in_executor(executor, random.random)
    executor.shutdown(wait=False)  # bug doesn't occur when `wait=True`

task = ioloop.create_task(func())

prints the following error:
Exception in thread QueueManagerThread:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/", line 917, in _bootstrap_inner
  File "/opt/conda/lib/python3.7/", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/lib/python3.7/concurrent/futures/", line 368, in _queue_management_worker
  File "/opt/conda/lib/python3.7/concurrent/futures/", line 92, in clear
    while self._reader.poll():
  File "/opt/conda/lib/python3.7/multiprocessing/", line 255, in poll
  File "/opt/conda/lib/python3.7/multiprocessing/", line 136, in _check_closed
    raise OSError("handle is closed")
OSError: handle is closed

I think this is related to and

This happens in the Adaptive package and the related issue is
msg366116 - (view) Author: Bas Nijholt (basnijholt) Date: 2020-04-10 10:58
Using `git bisect` I've discovered the commit (b713adf27a) ( that broke the code.

I've used one script:
import sys
from random import random
from process import ProcessPoolExecutor
import asyncio

ioloop = asyncio.get_event_loop()

async def func(ioloop, executor):
    result = await ioloop.run_in_executor(executor, random)
    executor.shutdown(wait=False)  # bug doesn't occur when `wait=True`

if __name__ == "__main__":
    executor = ProcessPoolExecutor()
    task = ioloop.run_until_complete(func(ioloop, executor))
and ``
import pexpect
import sys

child = pexpect.spawn("python /Users/basnijholt/Downloads/cpython/")
    child.expect(["OSError", "AssertionError"], timeout=1)
    raise Exception
except pexpect.EOF as e:

Then did
git checkout master
git reset --hard 9b6c60cbce  # bad commit
git bisect start
git bisect bad
git bisect good ad2c2d380e  # good commit
git bisect run python

I will see if I can fix it.
msg404196 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-10-18 16:33
I don't get the error on 3.11, and I think it may have been fixed under Issue39104.
Date User Action Args
2021-10-18 16:33:41iritkatrielsetstatus: open -> pending

nosy: + iritkatriel
messages: + msg404196

superseder: ProcessPoolExecutor hangs on shutdown nowait with pickling failure
resolution: duplicate
2020-10-28 21:28:50JoeKunsetnosy: + JoeKun
2020-04-10 10:58:09basnijholtsetmessages: + msg366116
2019-03-18 08:30:41basnijholtsettype: crash
2019-03-13 17:36:22basnijholtcreate