classification
Title: distutils.sysconfig.parse_makefile() regression in Python 3.10
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.11, Python 3.10
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: frenzy, hroncok, miss-islington, petr.viktorin
Priority: normal Keywords: patch

Created on 2021-06-08 19:00 by hroncok, last changed 2021-06-15 12:37 by petr.viktorin.

Pull Requests
URL Status Linked Edit
PR 26637 merged frenzy, 2021-06-10 06:07
PR 26673 merged miss-islington, 2021-06-11 15:08
Messages (6)
msg395352 - (view) Author: Miro Hrončok (hroncok) * Date: 2021-06-08 19:00
Hello. I think https://github.com/python/cpython/pull/23142 changed the behavior of distutils.sysconfig.parse_makefile().

A downstream Fedora report with an affected petsc package: https://bugzilla.redhat.com/show_bug.cgi?id=1959088


Reproducers (DeprecationWarnings removed for readability):


$ cat makefile
ALL: lib

DIRS = ssls asls # isls rscs
LOCDIR = src/tao/complementarity/impls/

include ${PETSC_DIR}/lib/petsc/conf/variables
include ${PETSC_DIR}/lib/petsc/conf/rules
include ${PETSC_DIR}/lib/petsc/conf/test


$ python3.9
>>> from distutils.sysconfig import parse_makefile
>>> makevars = parse_makefile('makefile')
>>> makevars.get('DIRS','').split()
['ssls', 'asls']

$ python3.10
>>> from distutils.sysconfig import parse_makefile
>>> makevars = parse_makefile('makefile')
>>> makevars.get('DIRS','').split()
['ssls', 'asls', '#', 'isls', 'rscs']



And:

$ cat makefile
-include ../../../../petscdir.mk
ALL: lib

LIBBASE  = libpetscksp
DIRS     = cr bcgs bcgsl cg cgs gmres cheby rich lsqr preonly tcqmr tfqmr \
           qcg bicg minres symmlq lcd ibcgs python gcr fcg tsirm fetidp hpddm
LOCDIR   = src/ksp/ksp/impls/

include ${PETSC_DIR}/lib/petsc/conf/variables
include ${PETSC_DIR}/lib/petsc/conf/rules
include ${PETSC_DIR}/lib/petsc/conf/test

$ python3.9
>>> from distutils.sysconfig import parse_makefile
>>> makevars = parse_makefile('makefile')
>>> makevars.get('DIRS','').split()
['cr', 'bcgs', 'bcgsl', 'cg', 'cgs', 'gmres', 'cheby', 'rich', 'lsqr', 'preonly', 'tcqmr', 'tfqmr', 'qcg', 'bicg', 'minres', 'symmlq', 'lcd', 'ibcgs', 'python', 'gcr', 'fcg', 'tsirm', 'fetidp', 'hpddm']

$ python3.10
>>> from distutils.sysconfig import parse_makefile
>>> makevars = parse_makefile('makefile')
>>> makevars.get('DIRS','').split()
['cr', 'bcgs', 'bcgsl', 'cg', 'cgs', 'gmres', 'cheby', 'rich', 'lsqr', 'preonly', 'tcqmr', 'tfqmr', '\\']


And:

$ cat makefile
-include ../../../../petscdir.mk
ALL: lib

LIBBASE  = libpetscksp
DIRS     = jacobi none sor shell bjacobi mg eisens asm ksp composite redundant spai is pbjacobi vpbjacobi ml\
           mat hypre tfs fieldsplit factor galerkin cp wb python \
           chowiluviennacl chowiluviennaclcuda rowscalingviennacl rowscalingviennaclcuda saviennacl saviennaclcuda\
           lsc redistribute gasm svd gamg parms bddc kaczmarz telescope patch lmvm hmg deflation hpddm hara
LOCDIR   = src/ksp/pc/impls/

include ${PETSC_DIR}/lib/petsc/conf/variables
include ${PETSC_DIR}/lib/petsc/conf/rules
include ${PETSC_DIR}/lib/petsc/conf/test

$ python3.9
>>> from distutils.sysconfig import parse_makefile
>>> makevars = parse_makefile('makefile')
>>> makevars.get('DIRS','').split()
['jacobi', 'none', 'sor', 'shell', 'bjacobi', 'mg', 'eisens', 'asm', 'ksp', 'composite', 'redundant', 'spai', 'is', 'pbjacobi', 'vpbjacobi', 'ml', 'mat', 'hypre', 'tfs', 'fieldsplit', 'factor', 'galerkin', 'cp', 'wb', 'python', 'chowiluviennacl', 'chowiluviennaclcuda', 'rowscalingviennacl', 'rowscalingviennaclcuda', 'saviennacl', 'saviennaclcuda', 'lsc', 'redistribute', 'gasm', 'svd', 'gamg', 'parms', 'bddc', 'kaczmarz', 'telescope', 'patch', 'lmvm', 'hmg', 'deflation', 'hpddm', 'hara']

$ python3.10
>>> from distutils.sysconfig import parse_makefile
>>> makevars = parse_makefile('makefile')
>>> makevars.get('DIRS','').split()
['jacobi', 'none', 'sor', 'shell', 'bjacobi', 'mg', 'eisens', 'asm', 'ksp', 'composite', 'redundant', 'spai', 'is', 'pbjacobi', 'vpbjacobi', 'ml\\']
msg395385 - (view) Author: Lumír Balhar (frenzy) * Date: 2021-06-09 07:33
Thanks for the report.

The regression is caused by the fact that the old implementation of parse_makefile in distutils.sysconfig was using feature-rich class TextFile which handles all the functionalities you reported as broken - stripping of comments, joining lines, and more.

The new implementation of parse_makefile in sysconfig just reads all the lines without any processing.

The TextFile class is deprecated together with the whole distutils. The class is not used outside of distutils modules, only in these functions:

distutils.extension:read_setup_file
distutils.sdist:read_template
+ its tests in distutils/tests/test_text_file.py

If the functionality is something we want to preserve, we should find a new home for the class and then we can use it in the sysconfig module.
msg395387 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-06-09 07:56
IMO, the functionality should only be preserved until distutils is removed. So:

- distutils.sysconfig.parse_makefile should use TextFile as before, so projects that use it aren't broken *yet*
- nothing else should call distutils.sysconfig.parse_makefile, so it doesn't block removing distutils

Would that make sense?
msg395402 - (view) Author: Miro Hrončok (hroncok) * Date: 2021-06-09 09:15
Yes, for the purposes of this bug, bringing TextFile-powered parse_makefile() back to distutils (and distutils only) is the right thing to do.

Whether or not Python needs a public standard library function to parse makefiles and whether that function in sysconfig is imperfect and needs improvements, that is an entirely different discussion.
msg395638 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-06-11 15:08
New changeset fc98266ff627ba0f56f8ae241245b66bc983baa3 by Lumír 'Frenzy' Balhar in branch 'main':
bpo-44351: Restore back parse_makefile in distutils.sysconfig (GH-26637)
https://github.com/python/cpython/commit/fc98266ff627ba0f56f8ae241245b66bc983baa3
msg395873 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-06-15 12:37
New changeset 2f2ea96c4429b81f491aa1cdc4219ef2fd6d37fb by Miss Islington (bot) in branch '3.10':
bpo-44351: Restore back parse_makefile in distutils.sysconfig (GH-26637) (GH-26673)
https://github.com/python/cpython/commit/2f2ea96c4429b81f491aa1cdc4219ef2fd6d37fb
History
Date User Action Args
2021-06-15 12:37:04petr.viktorinsetmessages: + msg395873
2021-06-11 15:08:20petr.viktorinsetmessages: + msg395638
2021-06-11 15:08:13miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request25259
2021-06-10 06:07:00frenzysetkeywords: + patch
stage: patch review
pull_requests: + pull_request25223
2021-06-09 09:15:30hroncoksetmessages: + msg395402
2021-06-09 07:56:06petr.viktorinsetmessages: + msg395387
2021-06-09 07:33:26frenzysetmessages: + msg395385
2021-06-08 19:00:01hroncokcreate