classification
Title: Distribution.parse_config_files uses interpolation
Type: behavior Stage: patch review
Components: Distutils Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: alunduil, berker.peksag, eric.araujo, jaraco, noirbizarre, tarek
Priority: normal Keywords: patch

Created on 2014-02-24 00:27 by alunduil, last changed 2017-07-27 03:21 by jaraco.

Files
File name Uploaded Description Edit
disable-distutils-string-interpolation.patch noirbizarre, 2016-12-12 15:49 review
Messages (12)
msg212048 - (view) Author: Alex Brandt (alunduil) Date: 2014-02-24 00:27
I first noticed this issue when configuring nose via setup.cfg and reported the behavior in this issue on their tracker: https://github.com/nose-devs/nose/issues/733

I'll repaste the important bits here:

When using a setup.cfg with the following contents (including minor variations):

[nosetests]
logging-format = %(lineno)d: %(process)d: %(message)s
With any setup.py file the following error is raised when trying to run setup.py (python2.7 used for example, error also occurs in 3.3):

alunduil@elijah margarine % python2.7 setup.py nosetests
Traceback (most recent call last):
  File "setup.py", line 108, in <module>
    setup(**PARAMS)
  File "/usr/lib64/python2.7/distutils/core.py", line 125, in setup
    dist.parse_config_files()
  File "/usr/lib64/python2.7/distutils/dist.py", line 397, in parse_config_files
    val = parser.get(section,opt)
  File "/usr/lib64/python2.7/ConfigParser.py", line 623, in get
    return self._interpolate(section, option, value, d)
  File "/usr/lib64/python2.7/ConfigParser.py", line 669, in _interpolate
    option, section, rawval, e.args[0])
ConfigParser.InterpolationMissingOptionError: Bad value substitution:
        section: [nosetests]
        option : logging-format
        key    : pathname
        rawval : %(pathname)s:%(lineno)d: %(process)d: %(message)s

The suggested fix of using %% to pass the interpolation through to nose requires that distutils use SafeConfigParser rather than ConfigParser.

I've verified that this does indeed appear to work (at least fixing the distutils interaction).  The only place the change needs to be made is on lines 378 and 386 in distutils/dist.py.

I can attach a patch or send a pull request with this fix if there are no objections to this solution.
msg277668 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-09-28 22:07
Thanks for the report, Alex. SafeConfigParser has been renamed to ConfigParser in Python 3.2 and distutils already uses ConfigParser in Python 3:

From Distribution.parse_config_files():

    ...
    parser = ConfigParser()

For 2.7, can you try it with setuptools? Perhaps we should keep 2.7 as-is and let setuptools do its magic.
msg277669 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2016-09-28 22:10
Do the changes in issue20120 address this concern?
msg283006 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2016-12-12 14:34
Looking more closely now and as a result of the report in issue28935, I see that this issue is another separate manifestation of the issue addressed in issue20120.
msg283014 - (view) Author: Axel Haustant (noirbizarre) * Date: 2016-12-12 15:49
I just attached the patch submitted on issue28935.

The fix is to simply give interpolation=None as ConfigParser parameter (as documented in the official Python 3.x ConfigParser documentation)
msg283023 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-12-12 16:37
Thanks! We also need a test case for the new behavior.
msg283068 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2016-12-13 03:17
I forget if there was a reason for choosing RawConfigParser over ConfigParaer with interpolation=None. I'd like to know that before choosing the latter. I'd also like to see if appropriate the implementation patched in Setuptools, providing compatibility for older Pythons. 

Also, +1 for a test.
msg283194 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2016-12-14 15:06
I reviewed the docs for configparser, and I agree - using ConfigParser(interpolation=None) is preferable to the "legacy" RawConfigParser.
msg283203 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2016-12-14 16:06
I've created a backport implementation in https://github.com/pypa/setuptools/issues/889.

All that remains then is to have a test for Python, which can probably borrow from the fix for issue20120. I'll leave it to others to draft the patch for the test.
msg299287 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2017-07-27 02:40
While investigating https://github.com/pypa/setuptools/issues/1062, I discovered that the implementation in the attached patch is insufficient. The 'parse_config_files' calls ConfigParser.__init__ in two places, before the for loop and at the end of each loop iteration. The parameters to both calls needs to be the same, or the BasicInterpolation behavior is revived.
msg299291 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2017-07-27 03:20
Ack. The interpolation behavior was also present in Python 2.7. I missed that when I found this ticket and assumed it was similar to issue 20120. In this ticket, the user is requesting that distutils actually change the behavior from performing interpolation to no longer performing it. This change is apparent now that Setuptools has a proper test for the desired behavior.
msg299292 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2017-07-27 03:21
Given that it's a change in behavior and not a bugfix, I don't see how this change could go into anything but the next release.
History
Date User Action Args
2017-07-27 03:21:07jaracosetmessages: + msg299292
versions: - Python 2.7, Python 3.5, Python 3.6
2017-07-27 03:20:36jaracosettitle: Distribution.parse_config_files uses interpolation since Python 3 -> Distribution.parse_config_files uses interpolation
messages: + msg299291
versions: + Python 2.7
2017-07-27 02:40:42jaracosetmessages: + msg299287
2016-12-14 16:06:05jaracosetmessages: + msg283203
2016-12-14 15:06:55jaracosetmessages: + msg283194
2016-12-13 03:17:30jaracosetmessages: + msg283068
2016-12-12 16:37:40berker.peksagsetmessages: + msg283023
stage: resolved -> patch review
2016-12-12 16:34:26berker.peksaglinkissue28935 superseder
2016-12-12 15:49:31noirbizarresetfiles: + disable-distutils-string-interpolation.patch

nosy: + noirbizarre
messages: + msg283014

keywords: + patch
2016-12-12 14:34:21jaracosetstatus: closed -> open
versions: + Python 3.5, Python 3.6, Python 3.7, - Python 2.7, Python 3.3, Python 3.4
title: distutils should use SafeConfigParser -> Distribution.parse_config_files uses interpolation since Python 3
messages: + msg283006

resolution: out of date ->
2016-09-28 22:10:40jaracosetmessages: + msg277669
2016-09-28 22:07:20berker.peksagsetstatus: open -> closed

nosy: + berker.peksag, jaraco
messages: + msg277668

resolution: out of date
stage: resolved
2014-02-24 23:38:38Arfreversetnosy: + tarek, eric.araujo

versions: + Python 3.4
2014-02-24 00:27:50alunduilcreate