classification
Title: distutils.command.build_ext raises FileNotFoundError
Type: crash Stage: resolved
Components: Distutils Versions: Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: lukasz.langa Nosy List: dstufft, eric.araujo, jaraco, lukasz.langa, miss-islington, vstinner
Priority: Keywords: 3.9regression, patch

Created on 2020-07-04 01:25 by jaraco, last changed 2020-07-07 11:58 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 21359 merged jaraco, 2020-07-06 20:56
PR 21373 merged miss-islington, 2020-07-07 11:11
Messages (10)
msg372973 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-07-04 01:25
In [pypa/setuptools#2228](https://github.com/pypa/setuptools/issues/2228), by adopting the distutils codebase from a late release of CPython, Setuptools stumbled onto an API-breaking change in distutils rooted at issue39763.

Details are in the Setuptools investigation, but to summarize:

- distutils.ccompiler.CCompiler.compile declares "raises CompileError on failure" and calls `self._compile`, implemented by subclasses.
- In at least `distutils.unixcompiler.UnixCCompiler._compile`, `distutils.spawn.spawn` is called (through CCompiler.spawn).
- Since GH-18743, `distutils.spawn.spawn` calls `subprocess.Popen` which raises FileNotFoundError when the target executable doesn't exist.
- Programs trapping CompileError but not FileNotFoundError will crash where once they had error handling.

Setuptools discovered this behavior in the 48.0 release when it incorporated these distutils changes into a vendored release of Setuptools, but the failures exhibited will apply to all builds (including pyyaml) on Python 3.9.
msg373045 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-07-05 20:37
In [7aa5abeafc">pypa/distutils@7aa5abeafc](https://github.com/pypa/distutils/commit/7aa5abeafc1e0b1b351c4c8ac7eb14c310366a46), I've pushed a fix (with a repro test in the parent commit). I recommend this fix be applied to CPython 3.9.
msg373047 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-07-05 21:11
CPython should also consider [this change](https://github.com/pypa/distutils/commit/d9ba43436d), which unifies the `DEBUG` handling, consolidates the exception trapping, and uses `subprocess.check_call` to re-use exit code checking.
msg373100 - (view) Author: Ɓukasz Langa (lukasz.langa) * (Python committer) Date: 2020-07-06 11:57
Good catch! The unification change is 3.10 only but I'm totally fine accepting the _compile fix in Python 3.9.
msg373104 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-07-06 12:02
> Programs trapping CompileError but not FileNotFoundError will crash where once they had error handling.

Do you mean mean that spawn() of distutils.spawn should catch FileNotFoundError and raise a CompilerError instead?

It seems like before my commit 1ec63b62035e73111e204a0e03b83503e1c58f2e, any OSError was catched and replaced with DistutilsExecError, not only FileNotFoundError.

Do you want to propose a fix?
msg373170 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-07-06 20:26
Sure. I'll submit patches.
msg373172 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-07-06 20:53
I learned the magical incantation to port commits from pypa/distutils to CPython:

```
cpython bugfix/41207-rewrite-filenotfound $ git -C ~/p/pypa/distutils format-patch HEAD~2 --stdout | git am --directory Lib                                
Applying: Add test for spawn when exe is missing. Ref pypa/distutils#3.
Applying: Replace OSError with DistutilsExecError. Fixes pypa/distutils#3 and pypa/setuptools#2228 and bpo-41207.
```
msg373218 - (view) Author: miss-islington (miss-islington) Date: 2020-07-07 11:11
New changeset 6ae2780be0667a8dc52c4fb583171ec86067d700 by Jason R. Coombs in branch 'master':
bpo-41207 In distutils.spawn, rewrite FileNotFound (GH-21359)
https://github.com/python/cpython/commit/6ae2780be0667a8dc52c4fb583171ec86067d700
msg373219 - (view) Author: miss-islington (miss-islington) Date: 2020-07-07 11:31
New changeset 2c82628e9aa2af4b662e92e227618859675dd726 by Miss Islington (bot) in branch '3.9':
bpo-41207 In distutils.spawn, rewrite FileNotFound (GH-21359)
https://github.com/python/cpython/commit/2c82628e9aa2af4b662e92e227618859675dd726
msg373220 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-07-07 11:58
I close the issue since it's fixed. I also reset the priority.
History
Date User Action Args
2020-07-07 11:58:53vstinnersetstatus: open -> closed
priority: release blocker ->
messages: + msg373220

resolution: fixed
stage: patch review -> resolved
2020-07-07 11:31:44miss-islingtonsetmessages: + msg373219
2020-07-07 11:11:57miss-islingtonsetpull_requests: + pull_request20517
2020-07-07 11:11:44miss-islingtonsetnosy: + miss-islington
messages: + msg373218
2020-07-06 20:56:18jaracosetkeywords: + patch
stage: patch review
pull_requests: + pull_request20504
2020-07-06 20:53:00jaracosetmessages: + msg373172
2020-07-06 20:26:40jaracosetmessages: + msg373170
2020-07-06 12:02:40vstinnersetnosy: + vstinner
messages: + msg373104
2020-07-06 11:57:16lukasz.langasetmessages: + msg373100
2020-07-05 21:11:07jaracosetmessages: + msg373047
2020-07-05 20:37:12jaracosetmessages: + msg373045
2020-07-04 01:26:02jaracosetversions: + Python 3.10
2020-07-04 01:25:52jaracocreate