classification
Title: venv activate.bat is UTF-8 encoded but uses current console codepage
Type: behavior Stage: patch review
Components: Library (Lib), Unicode, Windows Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: open Resolution: fixed
Dependencies: Superseder:
Assigned To: steve.dower Nosy List: Jac0, Martin Bijl-Schwab, eryksun, ezio.melotti, miss-islington, pablogsal, paul.moore, steve.dower, tim.golden, vinay.sajip, vstinner, zach.ware
Priority: normal Keywords: patch

Created on 2017-12-22 10:36 by Jac0, last changed 2018-11-08 16:38 by steve.dower.

Pull Requests
URL Status Linked Edit
PR 5757 merged steve.dower, 2018-02-19 18:11
PR 5765 merged miss-islington, 2018-02-20 01:25
PR 5765 merged miss-islington, 2018-02-20 01:25
PR 5766 merged miss-islington, 2018-02-20 01:26
PR 10295 merged python-dev, 2018-11-02 10:46
PR 10377 merged miss-islington, 2018-11-07 16:50
PR 10403 merged pablogsal, 2018-11-07 22:59
PR 10405 merged miss-islington, 2018-11-07 23:55
Messages (16)
msg308931 - (view) Author: Jaakko Roponen (Jac0) Date: 2017-12-22 10:36
Let's say I have a folder c:\test-ä in Windows

Now if I run: py -m venv env
and activate: env\scripts\activate
and check: where python

the result is incorrectly just: C:\Users\Username\AppData\Local\Programs\Python\Python36\python.exe

If I run: path 
the result is: PATH=C:\test-ä\env\Scripts;...

So clearly the encoding is broken for the folder name.

I can fix this by changing activate.bat character encoding to OEM-US and then replacing "test-├ż" by "test-ä".

If I now activate and run: where python
the result is (as should be): 
C:\test-ä\env\Scripts\python.exe
C:\Users\Username\AppData\Local\Programs\Python\Python36\python.exe

By running: path
I get: PATH=C:\test-ä\env\Scripts;...

So looks good here as well.

I suggest that what ever is creating activate.bat file, is using incorrect character encoding for the creation of the file. If this is somehow platform specific, there could be a guide in the venv documentation about how to fix this.
msg308934 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-12-22 13:24
The CMD shell decodes batch scripts using the attached console's output codepage, which defaults to OEM. OTOH, venv writes the replacement values for the template activate.bat as UTF-8 (codepage 65001), which is correct and should not be downgraded to OEM. 

Instead, the batch script could temporarily switch the console to codepage 65001. Then restore the previous codepage at the end. For example:

    @echo off
    for /f "tokens=2 delims=:" %%a in ('"%SystemRoot%\System32\chcp.com"') do (
        set "CODEPAGE=%%a"
    )
    "%SystemRoot%\System32\chcp.com" 65001 > nul

[rest of script]

    "%SystemRoot%\System32\chcp.com" %CODEPAGE% > nul
    set "CODEPAGE="
    :END
msg312356 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-02-19 16:57
Eryk's solution seems to be best, so I'll add that.
msg312389 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-02-20 01:25
New changeset 6240917b773b52f8883387b9e3a5f327a4372068 by Steve Dower in branch 'master':
bpo-32409: Ensures activate.bat can handle Unicode contents (GH-5757)
https://github.com/python/cpython/commit/6240917b773b52f8883387b9e3a5f327a4372068
msg312390 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-02-20 01:45
New changeset a3d6c1b23b8a49b5003fdbd115d3598fe3d4c4bf by Steve Dower (Miss Islington (bot)) in branch '3.7':
bpo-32409: Ensures activate.bat can handle Unicode contents (GH-5765)
https://github.com/python/cpython/commit/a3d6c1b23b8a49b5003fdbd115d3598fe3d4c4bf
msg312393 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-02-20 02:18
New changeset 8e149ff481acbb3889c825b8bf7b10aa191f09a7 by Steve Dower (Miss Islington (bot)) in branch '3.6':
bpo-32409: Ensures activate.bat can handle Unicode contents (GH-5766)
https://github.com/python/cpython/commit/8e149ff481acbb3889c825b8bf7b10aa191f09a7
msg329141 - (view) Author: Martin Bijl-Schwab (Martin Bijl-Schwab) * Date: 2018-11-02 16:41
it does not work as expected on swiss german (and likely other internationalised) windows systems. See https://bugs.python.org/issue32409
msg329142 - (view) Author: Martin Bijl-Schwab (Martin Bijl-Schwab) * Date: 2018-11-02 16:42
I meant https://bugs.python.org/issue35148
msg329425 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2018-11-07 16:49
New changeset c64583b6d3e8516a8cd2b5f84fc1e300bfac2206 by Vinay Sajip (samstagern) in branch 'master':
bpo-32409: Fix regression in activate.bat on international Windows (GH-10295)
https://github.com/python/cpython/commit/c64583b6d3e8516a8cd2b5f84fc1e300bfac2206
msg329431 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2018-11-07 19:26
New changeset 881e273c795f2f5154b8afebfa299f0e830f3712 by Vinay Sajip (Miss Islington (bot)) in branch '3.7':
bpo-32409: Fix regression in activate.bat on international Windows (GH-10295) (GH-10377)
https://github.com/python/cpython/commit/881e273c795f2f5154b8afebfa299f0e830f3712
msg329440 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2018-11-07 21:39
This commit broke a lot of the Windows buildbots:

https://buildbot.python.org/all/#/builders/32/builds/1707
https://buildbot.python.org/all/#/builders/113/builds/733
https://buildbot.python.org/all/#/builders/40/builds/1137
https://buildbot.python.org/all/#/builders/130/builds/420
https://buildbot.python.org/all/#/builders/3/builds/1634

Can someone work on a patch soon? If not we need to revert the commits in all the branches that are failed to build :(
msg329447 - (view) Author: Martin Bijl-Schwab (Martin Bijl-Schwab) * Date: 2018-11-07 22:18
I will have a look.
msg329450 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2018-11-07 23:55
New changeset 6843ffe4533e9f2cde036296fd932fef6f059687 by Pablo Galindo in branch 'master':
Revert "bpo-32409: Fix regression in activate.bat on international Windows (GH-10295)" (GH-10403)
https://github.com/python/cpython/commit/6843ffe4533e9f2cde036296fd932fef6f059687
msg329451 - (view) Author: miss-islington (miss-islington) Date: 2018-11-08 00:14
New changeset 3ba5e253de1a52e9692aa645c95072705f08b7bb by Miss Islington (bot) in branch '3.7':
Revert "bpo-32409: Fix regression in activate.bat on international Windows (GH-10295)" (GH-10403)
https://github.com/python/cpython/commit/3ba5e253de1a52e9692aa645c95072705f08b7bb
msg329464 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-08 08:48
Pablo Galindo reverted the change because it broke Windows buildbots, and we have a policy to revert a change breaking buildbots if the regression cannot be fixed "quickly" event:
https://pythondev.readthedocs.io/ci.html#revert-on-fail

Is someone working on investigating the bug? Do you need help to reproduce the bug?

Copy of the test_venv error:

======================================================================
ERROR: test_unicode_in_batch_file (test.test_venv.BasicTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\buildarea\3.x.bolen-windows8\build\lib\test\test_venv.py", line 302, in test_unicode_in_batch_file
    out, err = check_output(
  File "D:\buildarea\3.x.bolen-windows8\build\lib\test\test_venv.py", line 37, in check_output
    raise subprocess.CalledProcessError(
TypeError: __init__() takes from 3 to 5 positional arguments but 6 were given

https://buildbot.python.org/all/#/builders/32/builds/1707
msg329481 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-11-08 16:38
That error is a bug in the test, but it only shows up on an error path anyway. Without removing the extra None we don't get to see the actual error output.

I can't look into this over the next week or two, but a quick glance at the original PR looks like a lot of quotes are missing around executable paths, so maybe it was that?
History
Date User Action Args
2018-11-08 16:38:08steve.dowersetmessages: + msg329481
2018-11-08 08:48:59vstinnersetmessages: + msg329464
2018-11-08 00:14:05miss-islingtonsetnosy: + miss-islington
messages: + msg329451
2018-11-07 23:55:54miss-islingtonsetpull_requests: + pull_request9689
2018-11-07 23:55:43pablogsalsetmessages: + msg329450
2018-11-07 22:59:27pablogsalsetstage: resolved -> patch review
pull_requests: + pull_request9688
2018-11-07 22:18:18Martin Bijl-Schwabsetmessages: + msg329447
2018-11-07 21:39:10pablogsalsetstatus: closed -> open
nosy: + pablogsal
messages: + msg329440

2018-11-07 19:26:17vinay.sajipsetmessages: + msg329431
2018-11-07 16:50:26miss-islingtonsetpull_requests: + pull_request9679
2018-11-07 16:49:25vinay.sajipsetnosy: + vinay.sajip
messages: + msg329425
2018-11-02 16:42:55Martin Bijl-Schwabsetmessages: + msg329142
2018-11-02 16:41:28Martin Bijl-Schwabsetnosy: + Martin Bijl-Schwab
messages: + msg329141
2018-11-02 10:46:53python-devsetpull_requests: + pull_request9604
2018-02-20 02:18:45steve.dowersetmessages: + msg312393
2018-02-20 01:45:58steve.dowersetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2018-02-20 01:45:06steve.dowersetmessages: + msg312390
2018-02-20 01:26:32miss-islingtonsetpull_requests: + pull_request5545
2018-02-20 01:25:42miss-islingtonsetpull_requests: + pull_request5544
2018-02-20 01:25:40miss-islingtonsetpull_requests: + pull_request5543
2018-02-20 01:25:29steve.dowersetmessages: + msg312389
2018-02-19 18:11:12steve.dowersetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request5535
2018-02-19 16:57:10steve.dowersetassignee: steve.dower
messages: + msg312356
versions: + Python 3.8
2017-12-22 13:24:49eryksunsettitle: venv activation doesn't work, if project is in a Windows folder that has latin-1 supplement characters (such as ä,ö,å) in its path -> venv activate.bat is UTF-8 encoded but uses current console codepage
components: + Library (Lib), Unicode, Windows, - Extension Modules

nosy: + ezio.melotti, eryksun, paul.moore, tim.golden, vstinner, zach.ware, steve.dower
versions: + Python 3.7
messages: + msg308934
stage: needs patch
2017-12-22 10:36:15Jac0create