classification
Title: distutils doesn't correctly read UTF-8 content from config files
Type: behavior Stage:
Components: Distutils, Library (Lib), Windows Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: delivrance, dstufft, eric.araujo, esuarezsantana, matrixise, paul.moore, steve.dower, tim.golden, vlcinsky, zach.ware
Priority: normal Keywords:

Created on 2017-12-05 16:46 by delivrance, last changed 2019-05-06 19:00 by eric.araujo.

Files
File name Uploaded Description Edit
setup.cfg delivrance, 2017-12-05 17:10
setup.py delivrance, 2017-12-05 17:10
Pull Requests
URL Status Linked Edit
PR 4727 closed delivrance, 2017-12-05 16:46
Messages (9)
msg307668 - (view) Author: Dan (delivrance) Date: 2017-12-05 16:46
On Windows, distutils doesn't correctly read UTF-8 content from config files (setup.cfg).

Seems like the issue is located on the line reading the files via the ConfigParser; simply adding 'encoding="UTF-8"' as argument fixes the problem for me: https://github.com/python/cpython/pull/4727

On Linux it seems to be working fine.
msg307669 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2017-12-05 16:57
Can you give an example setup.cfg file, setup.py command and the full error message?
msg307670 - (view) Author: Dan (delivrance) Date: 2017-12-05 17:13
I've attached the files.

Run using 'python setup.py sdist'.
The resulting PKG-INFO will contain incorrect data:

Summary: délivrance
Author: Dan Tès

The expected output is:

Summary: délivrance
Author: Dan Tès
msg307860 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2017-12-08 17:53
`metadata` in setup.cfg is not supported directly by distutils.  Can you provide a setup.py script that shows the problem without setuptools?
msg310674 - (view) Author: Jan Vlcinsky (vlcinsky) Date: 2018-01-25 13:43
The fix shall go to

https://github.com/python/cpython/blob/2812d3d99287c50bab99625d7240bcf1c2e32369/Lib/distutils/dist.py#L406

where `parser.read(filename)`

shall be changed to `parser.read(filename, encoding="utf-8")`

This assumes that the setup.cfg shall be UTF-8 encoded what I thing is correct assumption.

Alternative assumptions are (and I do not find them good):

- assume the file is encoded as current console is (this is not deterministic and is direct cause of this issue)
- let user to specify the encoding somewhere around (this requires extra step and does not bring any value)
msg338272 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2019-03-18 17:53
Repeat: `metadata` in setup.cfg is not supported directly by distutils.  Can you provide a setup.py script that shows the problem without setuptools?
msg341139 - (view) Author: Eduardo Suarez-Santana (esuarezsantana) * Date: 2019-04-30 09:04
In my opinion, 'setup.cfg' is setuptools stuff and not part of python project. Anyway I think it is a good idea to read it as UTF-8.

Setuptools seems to be defaulting to UTF-8: https://github.com/pypa/setuptools/pull/1735
msg341553 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-05-06 16:04
Hi,

I have closed the PR because there is no feedback from deliverance and the CLA must be signed for the processing. Once the CLA will be signed, the PR could be re-opened.

Thank you so much,
msg341605 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2019-05-06 19:00
No, setup.py and setup.cfg are invented by distutils and extended by setuptools.
History
Date User Action Args
2019-05-06 19:00:12eric.araujosetmessages: + msg341605
2019-05-06 16:04:28matrixisesetnosy: + matrixise

messages: + msg341553
versions: - Python 3.6
2019-04-30 09:04:34esuarezsantanasetnosy: + esuarezsantana
messages: + msg341139
2019-03-18 17:53:12eric.araujosetmessages: + msg338272
2018-01-25 13:43:29vlcinskysetnosy: + vlcinsky
messages: + msg310674
2017-12-08 17:53:48eric.araujosetmessages: + msg307860
2017-12-05 17:13:50delivrancesetmessages: + msg307670
2017-12-05 17:10:58delivrancesetfiles: + setup.py
2017-12-05 17:10:52delivrancesetfiles: + setup.cfg
2017-12-05 16:57:45eric.araujosetversions: - Python 3.4, Python 3.5, Python 3.8
2017-12-05 16:57:32eric.araujosetmessages: + msg307669
2017-12-05 16:46:26delivrancecreate