classification
Title: Inconsistency between venv and site
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.6, Python 3.4, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, python-dev, serhiy.storchaka, vinay.sajip
Priority: normal Keywords:

Created on 2015-09-19 21:29 by serhiy.storchaka, last changed 2015-10-01 10:28 by python-dev. This issue is now closed.

Messages (7)
msg251119 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-09-19 21:29
venv writes pyvenv.cfg with utf-8 encoding, but site reads it with locale encoding.
msg251141 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-09-20 05:26
Example:

$ LC_ALL=ru_RU.cp1251 ./python -m venv ../vpy×
$ LC_ALL=ru_RU.cp1251 ./python -m venv ../vpyØ
Error: 'charmap' codec can't encode character '\udc98' in position 617: character maps to <undefined>

The first command is successful, the latter command is failed. 'Ø'.encode('utf-8') == b'\xc3\x98', and 0x98 is invalid code in CP1251.
msg251292 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2015-09-22 08:21
Seems like the right fix would be to change site.py to read it using UTF-8.
msg251776 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-09-28 17:07
I suspect that this can break backward compatibility.
msg251781 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2015-09-28 17:57
> I suspect that this can break backward compatibility.

But since venv writes it out as UTF-8 already, it should be read in UTF-8, and if it isn't, isn't that the cause of the error which led to this bug we're talking about?

If we now write it with locale encoding, anything that previously assumed we were writing it in UTF-8 might break. While I'm not aware of any specific software which reads pyvenv.cfg other than the site.py code, there could well be third party code which does so.
msg251783 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-09-28 18:26
I thought that pyvenv.cfg was used by venv's predecessor virtualenv. Now I see that I was wrong and your idea is right.
msg252016 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-10-01 10:28
New changeset d927c6cae05f by Vinay Sajip in branch '3.4':
Closes #25185: Use UTF-8 encoding when reading pyvenv.cfg.
https://hg.python.org/cpython/rev/d927c6cae05f

New changeset eaf9220bdee3 by Vinay Sajip in branch '3.5':
Closes #25185: merged fix from 3.4.
https://hg.python.org/cpython/rev/eaf9220bdee3

New changeset 69dd42cef190 by Vinay Sajip in branch 'default':
Closes #25185: merged fix from 3.5.
https://hg.python.org/cpython/rev/69dd42cef190
History
Date User Action Args
2015-11-03 09:09:20serhiy.storchakalinkissue25540 superseder
2015-10-01 10:28:53python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg252016

resolution: fixed
stage: needs patch -> resolved
2015-09-28 18:26:48serhiy.storchakasetmessages: + msg251783
stage: needs patch
2015-09-28 17:57:10vinay.sajipsetmessages: + msg251781
2015-09-28 17:07:09serhiy.storchakasetmessages: + msg251776
2015-09-22 08:21:49vinay.sajipsetmessages: + msg251292
2015-09-21 12:04:55Arfreversetnosy: + Arfrever
2015-09-20 05:26:39serhiy.storchakasetmessages: + msg251141
2015-09-19 21:29:54serhiy.storchakasetnosy: + vinay.sajip
2015-09-19 21:29:19serhiy.storchakacreate