Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test_peg_generator.test_c_parser emits DeprecationWarning due to distutils #87591

Closed
tirkarthi opened this issue Mar 7, 2021 · 14 comments
Closed
Labels
3.10 only security fixes pending The issue will be closed if no feedback is provided tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error

Comments

@tirkarthi
Copy link
Member

BPO 43425
Nosy @terryjreedy, @vstinner, @ambv, @zooba, @corona10, @pablogsal, @miss-islington, @tirkarthi
PRs
  • bpo-43425: Update test_c_parser not to use TempdirManager #26693
  • [3.10] bpo-43425: Update test_c_parser not to use TempdirManager (GH-26693) #26713
  • bpo-43425: Update _osx_support not to use distutils.log #26968
  • bpo-43425: Update setup.py not to use distutils.log #26969
  • [3.10] bpo-43425: Update _osx_support not to use distutils.log (GH-26968) #26978
  • bpo-43425: Remove test2to3 from Tools #26981
  • [3.10] bpo-43425: Update setup.py not to use distutils.log (GH-26969) #26986
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2021-03-07.05:07:09.546>
    labels = ['type-bug', 'tests', '3.10']
    title = 'test_peg_generator.test_c_parser emits DeprecationWarning due to distutils'
    updated_at = <Date 2021-09-15.13:15:47.651>
    user = 'https://github.com/tirkarthi'

    bugs.python.org fields:

    activity = <Date 2021-09-15.13:15:47.651>
    actor = 'vstinner'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Tests']
    creation = <Date 2021-03-07.05:07:09.546>
    creator = 'xtreak'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 43425
    keywords = ['patch']
    message_count = 13.0
    messages = ['388222', '388242', '388243', '388572', '388780', '395716', '395768', '396805', '396808', '396837', '396838', '397716', '401831']
    nosy_count = 8.0
    nosy_names = ['terry.reedy', 'vstinner', 'lukasz.langa', 'steve.dower', 'corona10', 'pablogsal', 'miss-islington', 'xtreak']
    pr_nums = ['26693', '26713', '26968', '26969', '26978', '26981', '26986']
    priority = 'normal'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue43425'
    versions = ['Python 3.10']

    @tirkarthi
    Copy link
    Member Author

    distutils was deprecated for removal in Python 3.10. It is used in test_peg_generator.test_c_parser which emits a deprecation warning. It also seems to be used in test_support for missing_compiler_executable that will emit a deprecation warning.

    ./python -Wall -m test test_peg_generator test_c_parser
    0:00:00 load avg: 0.02 Run tests sequentially
    0:00:00 load avg: 0.02 [1/2] test_peg_generator
    /root/cpython/Lib/test/test_peg_generator/test_c_parser.py:4: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP-632 for potential alternatives
    from distutils.tests.support import TempdirManager

    Other places on grep :

    rg 'from distutils' | rg -v 'Lib/distutils|rst'
    Modules/_decimal/tests/formathelper.py:from distutils.spawn import find_executable
    Doc/includes/setup.py:from distutils.core import setup, Extension
    Doc/includes/test.py:from distutils.util import get_platform
    setup.py:from distutils import log
    setup.py:from distutils.command.build_ext import build_ext
    setup.py:from distutils.command.build_scripts import build_scripts
    setup.py:from distutils.command.install import install
    setup.py:from distutils.command.install_lib import install_lib
    setup.py:from distutils.core import Extension, setup
    setup.py:from distutils.errors import CCompilerError, DistutilsError
    setup.py:from distutils.spawn import find_executable
    Lib/_osx_support.py: from distutils import log
    Lib/_osx_support.py: Currently called from distutils.sysconfig
    Lib/test/support/init.py: from distutils import ccompiler, sysconfig, spawn, errors
    Lib/test/test_importlib/test_windows.py:from distutils.util import get_platform
    Lib/test/test_peg_generator/test_c_parser.py:from distutils.tests.support import TempdirManager
    Tools/peg_generator/pegen/build.py: from distutils.core import Distribution, Extension
    Tools/peg_generator/pegen/build.py: from distutils.command.clean import clean # type: ignore
    Tools/peg_generator/pegen/build.py: from distutils.command.build_ext import build_ext # type: ignore
    Tools/peg_generator/pegen/build.py: from distutils.tests.support import fixup_build_ext # type: ignore
    Tools/test2to3/setup.py:from distutils.core import setup
    Tools/test2to3/setup.py: from distutils.command.build_py import build_py_2to3 as build_py
    Tools/test2to3/setup.py: from distutils.command.build_py import build_py
    Tools/test2to3/setup.py: from distutils.command.build_scripts import build_scripts_2to3 as build_scripts
    Tools/test2to3/setup.py: from distutils.command.build_scripts import build_scripts
    Tools/test2to3/test/runtests.py: from distutils.util import copydir_run_2to3
    Misc/HISTORY:- Issue bpo-5394: removed > 2.3 syntax from distutils.msvc9compiler.

    @tirkarthi tirkarthi added 3.10 only security fixes tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error labels Mar 7, 2021
    @pablogsal
    Copy link
    Member

    We just need to suppress the warning unless I am missing somey. Distutils will still be used internally by CPython.

    @tirkarthi
    Copy link
    Member Author

    Thanks for the clarification. As per bpo-41282 I thought disutils will be removed. The accepted PEP-632 states this scenario so I guess silencing the warning is good enough.

    The final dependency on distutils is CPython itself, which uses it to build native extension modules in the standard library (except on Windows). Because this is a CPython build-time dependency, it is possible to continue to use distutils for this specific case without it being part of the standard library.

    @terryjreedy
    Copy link
    Member

    Steve, PEP-632 lacks a reference to the SC acceptance thereof. Could you add one?
    https://mail.python.org/archives/list/python-dev@python.org/thread/TXU6TVOMBLQU3SV57DMMOA5Y2E67AW7P/
    And can you verify my interpretation below?

    The warning is correct as to removal. PEP-632 says, perhaps a bit tersely, that when the 3.11 branch is established for 3.11.0b1 and master/main becomes the 3.12.0a development branch, disutils/* be no longer be installed as part of the stdlib (ie, in /Lib). At that point, 'import disutils' will cease working in installed CPython, outside of a cpython repository. Although this could be worked around by catching the import error and skipping disutils-dependent tests, the dependency should better be removed, if possible, before 3.12.0a.

    The PEP gives no timetable for doing so, and anticipates separate PRs like this.

    It further says that when "the CPython build process no longer depends on distutils being in the standard library, the entire Lib/distutils directory and Lib/test/test_distutils.py file will be removed from the repository." Leaving it deprecated but present was rejected.

    Part of the extensive discussion at https://discuss.python.org/t/pep-632-deprecate-distutils-module/5134/128 was about possibly putting parts of disutiles under /Tools and replacing 'import disutils' with a reference to <python dir>/Tools/distutiles/whatever. If this is done for tests, the access and use has to be conditional. Our Windows installer only installs 4 or 21 Tools subdirectories thought appropriate for uses and others may include less, and maybe omit Tools altogether.

    Karthikeyan posted the opening of this and another issue 5 days ago on bpo-41281, with no comment so far.

    @zooba
    Copy link
    Member

    zooba commented Mar 15, 2021

    The link appears to be in the Resolution header of PEP-632?

    But yes, your analysis is correct. If/when we move distutils elsewhere, we'll probably have to update all of these sections.

    Ideally we'd have a helper function to do the initial "import distutils", handle failure (because it's not in Tools), and suppress the warning. But I don't know that there's anywhere we could put it reliably anyway, so probably just need to suppress the warnings directly and we'll handle failures if/when we move it into Tools.

    @corona10
    Copy link
    Member

    New changeset 736ed6f by Dong-hee Na in branch 'main':
    bpo-43425: Update test_c_parser not to use TempdirManager (GH-26693)
    736ed6f

    @miss-islington
    Copy link
    Contributor

    New changeset 2b57ad3 by Miss Islington (bot) in branch '3.10':
    bpo-43425: Update test_c_parser not to use TempdirManager (GH-26693)
    2b57ad3

    @corona10
    Copy link
    Member

    corona10 commented Jul 1, 2021

    New changeset c8979f7 by Dong-hee Na in branch 'main':
    bpo-43425: Update _osx_support not to use distutils.log (GH-26968)
    c8979f7

    @miss-islington
    Copy link
    Contributor

    New changeset 94a4136 by Miss Islington (bot) in branch '3.10':
    bpo-43425: Update _osx_support not to use distutils.log (GH-26968)
    94a4136

    @corona10
    Copy link
    Member

    corona10 commented Jul 2, 2021

    New changeset a7e251b by Dong-hee Na in branch 'main':
    bpo-43425: Update setup.py not to use distutils.log (GH-26969)
    a7e251b

    @miss-islington
    Copy link
    Contributor

    New changeset 956f1fc by Miss Islington (bot) in branch '3.10':
    bpo-43425: Update setup.py not to use distutils.log (GH-26969)
    956f1fc

    @ambv
    Copy link
    Contributor

    ambv commented Jul 17, 2021

    New changeset d0b2b00 by Dong-hee Na in branch 'main':
    bpo-43425: Remove test2to3 from Tools (bpo-26981)
    d0b2b00

    @vstinner
    Copy link
    Member

    test_peg_generator still emits DeprecationWarning warnings. Example:

    vstinner@apu$ ./python -m test test_peg_generator -v
    == CPython 3.11.0a0 (heads/main:b49263b698, Sep 15 2021, 14:44:10) [GCC 11.2.1 20210728 (Red Hat 11.2.1-1)]
    == Linux-5.13.14-200.fc34.x86_64-x86_64-with-glibc2.33 little-endian
    == cwd: /home/vstinner/python/main/build/test_python_193641æ
    == CPU count: 8
    == encodings: locale=UTF-8, FS=utf-8
    0:00:00 load avg: 15.94 Run tests sequentially
    0:00:00 load avg: 15.94 [1/1] test_peg_generator
    test_advanced_left_recursive (test.test_peg_generator.test_c_parser.TestCParser) ...

    /home/vstinner/python/main/Lib/test/support/init.py:1713: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP-632 for potential alternatives
    from distutils import ccompiler, sysconfig, spawn, errors

    /home/vstinner/python/main/Lib/test/support/init.py:1713: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
    from distutils import ccompiler, sysconfig, spawn, errors
    (...)

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @iritkatriel
    Copy link
    Member

    test_peg_generator still emits DeprecationWarning warnings.

    Not anymore - it's skipped since 0faa0ba.

    @iritkatriel iritkatriel added the pending The issue will be closed if no feedback is provided label Apr 6, 2023
    @iritkatriel iritkatriel closed this as not planned Won't fix, can't repro, duplicate, stale Apr 25, 2023
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.10 only security fixes pending The issue will be closed if no feedback is provided tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    9 participants