This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: compileall.py throws a traceback when using -j0 and thus 'make install' locks up
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.10
process
Status: closed Resolution: duplicate
Dependencies: Superseder: "dictionary changed size during iteration" error in _ExecutorManagerThread
View: 43498
Assigned To: Nosy List: Alexander Kanavin
Priority: normal Keywords:

Created on 2021-12-01 08:58 by Alexander Kanavin, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
log.do_install.1905494 Alexander Kanavin, 2021-12-01 09:09
Messages (2)
msg407431 - (view) Author: Alexander Kanavin (Alexander Kanavin) * Date: 2021-12-01 08:58
Hello, Yocto project has had to disable -j0 for compileall, so that it runs serially. If it doesn't, then 'make install' locks up sporadically, with hanging processes:
```
3837320 ?        SN     0:00                  \_ make -j 16 -l 52 STAGING_LIBDIR=/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280438/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/recipe-sysroot/usr/lib STAGING_INCDIR=/
 157523 ?        SNl    0:02                      \_ python3.10 -Wi -OO /home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280438/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/compileall.py -j0 -d /u
 160673 ?        SN     0:00                          \_ python3.10 -Wi -OO /home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280438/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/compileall.py -j0 -
 160677 ?        SN     0:00                          \_ python3.10 -Wi -OO /home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280438/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/compileall.py -j0 -
 160682 ?        SN     0:00                          \_ python3.10 -Wi -OO /home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280438/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/compileall.py -j0 -
 160697 ?        SN     0:00                          \_ python3.10 -Wi -OO /home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280438/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/compileall.py -j0 -
 160759 ?        SN     0:00                          \_ python3.10 -Wi -OO /home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280438/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/compileall.py -j0 -
 160816 ?        SN     0:00                          \_ python3.10 -Wi -OO /home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280438/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/compileall.py -j0 -
...
```

and installation log reveals:
```
poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/compileall.py \
        -j0 -d /usr/lib/python3.10 -f \
        -x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' \
        /home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280388/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10
Listing '/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280388/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10'...
Listing '/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280388/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/asyncio'...
Listing '/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280388/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/collections'...
Listing '/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280388/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/concurrent'...
Listing '/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280388/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/concurrent/futures'...
Listing '/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280388/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/config-3.10-x86_64-linux-gnu'...
Listing '/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280388/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/ctypes'...
Listing '/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280388/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/ctypes/macholib'...
...
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280388/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/threading.py", line 1009, in _bootstrap_inner
    self.run()
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280388/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/concurrent/futures/process.py", line 317, in run
    result_item, is_broken, cause = self.wait_result_broken_or_wakeup()
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280388/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/concurrent/futures/process.py", line 376, in wait_result_broken_or_wakeup
    worker_sentinels = [p.sentinel for p in self.processes.values()]
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280388/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/concurrent/futures/process.py", line 376, in <listcomp>
    worker_sentinels = [p.sentinel for p in self.processes.values()]
RuntimeError: dictionary changed size during iteration
```
msg407433 - (view) Author: Alexander Kanavin (Alexander Kanavin) * Date: 2021-12-01 09:09
Here's the full log where you can see what happens.
History
Date User Action Args
2022-04-11 14:59:53adminsetgithub: 90103
2022-01-25 13:51:40iritkatrielsetstatus: open -> closed
superseder: "dictionary changed size during iteration" error in _ExecutorManagerThread
resolution: duplicate
stage: resolved
2021-12-01 09:09:21Alexander Kanavinsetfiles: + log.do_install.1905494

messages: + msg407433
2021-12-01 08:58:16Alexander Kanavincreate