Title: test_peg_generator.test_c_parser emits DeprecationWarning due to distutils
Type: behavior Stage: patch review
Components: Tests Versions: Python 3.10
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: corona10, lukasz.langa, miss-islington, pablogsal, steve.dower, terry.reedy, vstinner, xtreak
Priority: normal Keywords: patch

Created on 2021-03-07 05:07 by xtreak, last changed 2021-09-15 13:15 by vstinner.

Pull Requests
URL Status Linked Edit
PR 26693 merged corona10, 2021-06-12 15:51
PR 26713 merged miss-islington, 2021-06-13 23:20
PR 26968 merged corona10, 2021-06-30 15:47
PR 26969 merged corona10, 2021-06-30 16:25
PR 26978 merged miss-islington, 2021-07-01 00:35
PR 26981 merged corona10, 2021-07-01 13:40
PR 26986 merged miss-islington, 2021-07-02 01:42
Messages (13)
msg388222 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2021-03-07 05:07
distutils was deprecated for removal in Python 3.10. It is used in test_peg_generator.test_c_parser which emits a deprecation warning. It also seems to be used in test_support for missing_compiler_executable that will emit a deprecation warning.

./python -Wall -m test test_peg_generator test_c_parser
0:00:00 load avg: 0.02 Run tests sequentially
0:00:00 load avg: 0.02 [1/2] test_peg_generator
/root/cpython/Lib/test/test_peg_generator/ DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  from import TempdirManager

Other places on grep : 

rg 'from distutils' | rg -v 'Lib/distutils|rst'   
Modules/_decimal/tests/ distutils.spawn import find_executable
Doc/includes/ distutils.core import setup, Extension
Doc/includes/ distutils.util import get_platform distutils import log distutils.command.build_ext import build_ext distutils.command.build_scripts import build_scripts distutils.command.install import install distutils.command.install_lib import install_lib distutils.core import Extension, setup distutils.errors import CCompilerError, DistutilsError distutils.spawn import find_executable
Lib/        from distutils import log
Lib/    Currently called from distutils.sysconfig
Lib/test/support/    from distutils import ccompiler, sysconfig, spawn, errors
Lib/test/test_importlib/ distutils.util import get_platform
Lib/test/test_peg_generator/ import TempdirManager
Tools/peg_generator/pegen/    from distutils.core import Distribution, Extension
Tools/peg_generator/pegen/    from distutils.command.clean import clean  # type: ignore
Tools/peg_generator/pegen/    from distutils.command.build_ext import build_ext  # type: ignore
Tools/peg_generator/pegen/    from import fixup_build_ext  # type: ignore
Tools/test2to3/ distutils.core import setup
Tools/test2to3/    from distutils.command.build_py import build_py_2to3 as build_py
Tools/test2to3/    from distutils.command.build_py import build_py
Tools/test2to3/    from distutils.command.build_scripts import build_scripts_2to3 as build_scripts
Tools/test2to3/    from distutils.command.build_scripts import build_scripts
Tools/test2to3/test/    from distutils.util import copydir_run_2to3
Misc/HISTORY:- Issue #5394: removed > 2.3 syntax from distutils.msvc9compiler.
msg388242 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-03-07 16:24
We just need to suppress the warning unless I am missing somey. Distutils will still be used internally by CPython.
msg388243 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2021-03-07 16:31
Thanks for the clarification. As per issue41282 I thought disutils will be removed. The accepted PEP 632 states this scenario so I guess silencing the warning is good enough.

> The final dependency on distutils is CPython itself, which uses it to build native extension modules in the standard library (except on Windows). Because this is a CPython build-time dependency, it is possible to continue to use distutils for this specific case without it being part of the standard library.
msg388572 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-03-13 00:39
Steve, PEP 632 lacks a reference to the SC acceptance thereof.  Could you add one?
And can you verify my interpretation below?

The warning is correct as to removal.  PEP 632 says, perhaps a bit tersely, that when the 3.11 branch is established for 3.11.0b1 and master/main becomes the 3.12.0a development branch, disutils/* be no longer be installed as part of the stdlib (ie, in /Lib).  At that point, 'import disutils' will cease working in installed CPython, outside of a cpython repository.  Although this could be worked around by catching the import error and skipping disutils-dependent tests, the dependency should better be removed, if possible, before 3.12.0a.

The PEP gives no timetable for doing so, and anticipates separate PRs like this.

It further says that when "the CPython build process no longer depends on distutils being in the standard library, the entire Lib/distutils directory and Lib/test/ file will be removed from the repository."  Leaving it deprecated but present was rejected.

Part of the extensive discussion at was about possibly putting parts of disutiles under /Tools and replacing 'import disutils' with a reference to <python dir>/Tools/distutiles/whatever.  If this is done for tests, the access and use has to be conditional.  Our Windows installer only installs 4 or 21 Tools subdirectories thought appropriate for uses and others may include less, and maybe omit Tools altogether.

Karthikeyan posted the opening of this and another issue 5 days ago on #41281, with no comment so far.
msg388780 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-03-15 22:23
The link appears to be in the Resolution header of PEP 632?

But yes, your analysis is correct. If/when we move distutils elsewhere, we'll probably have to update all of these sections.

Ideally we'd have a helper function to do the initial "import distutils",  handle failure (because it's not in Tools), and suppress the warning. But I don't know that there's anywhere we could put it reliably anyway, so probably just need to suppress the warnings directly and we'll handle failures if/when we move it into Tools.
msg395716 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2021-06-12 22:07
New changeset 736ed6f7a9f465ba728198e8bca81e5fbe71bc37 by Dong-hee Na in branch 'main':
bpo-43425: Update test_c_parser not to use TempdirManager (GH-26693)
msg395768 - (view) Author: miss-islington (miss-islington) Date: 2021-06-14 00:24
New changeset 2b57ad3f5382edb6fe2dfe3c721573fbf25a826f by Miss Islington (bot) in branch '3.10':
bpo-43425: Update test_c_parser not to use TempdirManager (GH-26693)
msg396805 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2021-07-01 00:35
New changeset c8979f780e4b7d6db5693cb26a2956cc785abb48 by Dong-hee Na in branch 'main':
bpo-43425: Update _osx_support not to use distutils.log (GH-26968)
msg396808 - (view) Author: miss-islington (miss-islington) Date: 2021-07-01 01:20
New changeset 94a4136c8eba349dc7eebe561ddaedbd0a89eb91 by Miss Islington (bot) in branch '3.10':
bpo-43425: Update _osx_support not to use distutils.log (GH-26968)
msg396837 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2021-07-02 01:41
New changeset a7e251b07aa5f1a858a3e791a30dea78e93ab728 by Dong-hee Na in branch 'main':
bpo-43425: Update not to use distutils.log (GH-26969)
msg396838 - (view) Author: miss-islington (miss-islington) Date: 2021-07-02 02:05
New changeset 956f1fc89475f6c8252a9eb0689bcf0eb69c15ea by Miss Islington (bot) in branch '3.10':
bpo-43425: Update not to use distutils.log (GH-26969)
msg397716 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-07-17 10:10
New changeset d0b2b00c7d5ac3dc18d77bf4fcf607ee98c11650 by Dong-hee Na in branch 'main':
bpo-43425: Remove test2to3 from Tools (#26981)
msg401831 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-15 13:15
test_peg_generator still emits DeprecationWarning warnings. Example:

vstinner@apu$ ./python -m test test_peg_generator -v
== CPython 3.11.0a0 (heads/main:b49263b698, Sep 15 2021, 14:44:10) [GCC 11.2.1 20210728 (Red Hat 11.2.1-1)]
== Linux-5.13.14-200.fc34.x86_64-x86_64-with-glibc2.33 little-endian
== cwd: /home/vstinner/python/main/build/test_python_193641æ
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
0:00:00 load avg: 15.94 Run tests sequentially
0:00:00 load avg: 15.94 [1/1] test_peg_generator
test_advanced_left_recursive (test.test_peg_generator.test_c_parser.TestCParser) ...

/home/vstinner/python/main/Lib/test/support/ DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  from distutils import ccompiler, sysconfig, spawn, errors

/home/vstinner/python/main/Lib/test/support/ DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
  from distutils import ccompiler, sysconfig, spawn, errors
Date User Action Args
2021-09-15 13:15:47vstinnersetnosy: + vstinner
messages: + msg401831
2021-07-17 10:10:46lukasz.langasetnosy: + lukasz.langa
messages: + msg397716
2021-07-02 02:05:33miss-islingtonsetmessages: + msg396838
2021-07-02 01:42:10miss-islingtonsetpull_requests: + pull_request25548
2021-07-02 01:41:57corona10setmessages: + msg396837
2021-07-01 13:40:19corona10setpull_requests: + pull_request25543
2021-07-01 01:20:31miss-islingtonsetmessages: + msg396808
2021-07-01 00:35:20miss-islingtonsetpull_requests: + pull_request25541
2021-07-01 00:35:18corona10setmessages: + msg396805
2021-06-30 16:25:05corona10setpull_requests: + pull_request25533
2021-06-30 15:47:41corona10setpull_requests: + pull_request25532
2021-06-14 00:24:23miss-islingtonsetmessages: + msg395768
2021-06-13 23:20:02miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request25301
2021-06-12 22:07:27corona10setmessages: + msg395716
2021-06-12 15:51:09corona10setkeywords: + patch
nosy: + corona10

pull_requests: + pull_request25278
stage: patch review
2021-03-15 22:23:30steve.dowersetmessages: + msg388780
2021-03-13 00:39:55terry.reedysetnosy: + terry.reedy, steve.dower
messages: + msg388572
2021-03-07 16:31:28xtreaksetmessages: + msg388243
2021-03-07 16:24:13pablogsalsetmessages: + msg388242
2021-03-07 05:07:09xtreakcreate