classification
Title: Windows: Make UTF-8 mode more accessible
Type: Stage: patch review
Components: Windows Versions: Python 3.10
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: eryksun, methane, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords: patch

Created on 2021-03-02 09:03 by methane, last changed 2021-03-10 08:55 by methane.

Files
File name Uploaded Description Edit
utf8mode-screenshot1.png methane, 2021-03-10 08:55
Pull Requests
URL Status Linked Edit
PR 24742 open methane, 2021-03-04 08:05
PR 24743 open methane, 2021-03-04 08:46
PR 24813 open methane, 2021-03-10 08:53
Messages (8)
msg387902 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2021-03-02 09:03
I have created "Make UTF-8 mode more accessible for Windows users." thread.
https://mail.python.org/archives/list/python-ideas@python.org/thread/LQVK2UKPSOI2AHYFUWK6ZII2U6QKK6BP/

I proposed per-site (e.g. install site and venv) configuration file but many core devs don't like the idea because PYTHONUTF8 envvar would be enough for many users. For example:

https://mail.python.org/archives/list/python-ideas@python.org/message/BDOV2F3LDSC2YDUR4LCPNHZFJWEZNX5U/

So I want to add "Enable UTF-8 mode" shortcut to start menu. It executes just `cmd.exe /c "SETX PYTHONUTF8 1"`.

But I am not familar with Wix. How can I add the shortcut? May I add it in Toos/msi/exe/exe.wxs ?
msg387954 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-03-02 19:09
> `cmd.exe /c "SETX PYTHONUTF8 1

You can run "[WindowsFolder]System32\setx.exe" directly. It's not a shell command.
msg387982 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-03-02 22:47
For modifying an environment variable, please clone the entire path MSI and use an Environment element to update it. Should be a little simpler than that one, because it's a constant.

I'm always very hesitant to modify system-wide (or user-wide) settings like this though. I'd be more comfortable if we made it a PYTHONUTF8_310 variable that _only_ applies to that specific version. Otherwise someone might install an update and break existing apps.

Alternatively, for this kind of policy option, it might be nice to set up some common registry keys to define it. That is a function I've been thinking about for enabling some security features (such as malware scanning), and it mimics how Windows distributes its own configuration already. That'll take a whole PEP, though, and I doubt I'll get it ready in time for 3.10 given the other stuff I've got going on right now.

So provided we adequately document that the setting may impact other versions of Python besides the one being installed, and it's not enabled by default, I'm okay with an MSI-based addable/removable setting for PYTHONUTF8.
msg387995 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2021-03-03 03:38
> You can run "[WindowsFolder]System32\setx.exe" directly. It's not a shell command.

Thank you. I will do.

> For modifying an environment variable, please clone the entire path MSI and use an Environment element to update it. Should be a little simpler than that one, because it's a constant.

How about having both? Shortcut menu is more accessible when user want to enable UTF-8 mode after they install Python. (e.g. they got troubled by UnicodeDecodeError and are adviced to enable UTF-8 mode).

> Alternatively, for this kind of policy option, it might be nice to set up some common registry keys to define it. That is a function I've been thinking about for enabling some security features (such as malware scanning), and it mimics how Windows distributes its own configuration already. That'll take a whole PEP, though, and I doubt I'll get it ready in time for 3.10 given the other stuff I've got going on right now.

Registry is also discussed in the thread, but there are no consensus about adding new way to enable UTF-8 mode.
I conclude that we should promote existing way (i.e. envvar) and get feedback from users.

> So provided we adequately document that the setting may impact other versions of Python besides the one being installed, and it's not enabled by default, I'm okay with an MSI-based addable/removable setting for PYTHONUTF8.
msg388084 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-03-04 13:14
> I'm always very hesitant to modify system-wide (or user-wide) settings 
> like this though. I'd be more comfortable if we made it a PYTHONUTF8_310
> variable that _only_ applies to that specific version. Otherwise someone
> might install an update and break existing apps.

Yes, if the installer directly sets an environment variable, then the setting should be specific to the installed version, such as PYTHONUTF8_310-32.

Maybe this could be implemented more generally by supporting a "PYTHONX_{py_version_short_plat}" environment variable that contains a list of -X implementation-specific boolean options to enable by default, e.g. set "PYTHONX_3.10-32=utf8;dev". In POSIX, name it "PYTHONX_{py_version_short}". In Windows, "{py_version_short_plat}" would be sys.winver.
msg388091 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2021-03-04 14:32
User may install Python from python.org installer, scoop, conda, and Windows Store.
So envvar is not a right tool to configure only one installation.

Anyway, adding more complex way to configure Python can be discussed later.

Paul Moore againsted adding more way to configure Python [1].
So let's focus on promoting existing way in Python 3.10.

[1] https://mail.python.org/archives/list/python-ideas@python.org/message/BDOV2F3LDSC2YDUR4LCPNHZFJWEZNX5U/

> Yes, if the installer directly sets an environment variable, then the setting should be specific to the installed version, such as PYTHONUTF8_310-32.

Then, suspend GH-24742 (adding checkbox to installer).

In GH-24743, I added the shortcut from the installer. But I am not sure it is the best way, because only Python installed from the installer have it.

I will check how conda installs Python on Windows. If it contains Tools, we may add `enable-utf8mode.bat` and `disable-utf8mode.bat` to the Tools.
msg388111 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-03-04 17:55
Maybe as a global setting, the better thing to copy is the button at the end of installation that offers to change the long path policy?

That would give a bit more room that a checkbox to explain what is being set and what the implications are, which will help users who select it when they shouldn't.

IIRC, all of that code exists in the boostrapper app (one of the .cpp files). The UI is probably in the bundle wxl/thm files.
msg388416 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2021-03-10 08:55
> Maybe as a global setting, the better thing to copy is the button at the end of installation that offers to change the long path policy?

I tried it but there is no enough space.
See GH-24813 and attached screenshot.
History
Date User Action Args
2021-03-10 08:55:08methanesetfiles: + utf8mode-screenshot1.png

messages: + msg388416
2021-03-10 08:53:49methanesetpull_requests: + pull_request23579
2021-03-05 00:33:16methanesettitle: Add shortcut to enable UTF-8 mode in start menu. -> Windows: Make UTF-8 mode more accessible
2021-03-04 17:55:37steve.dowersetmessages: + msg388111
2021-03-04 14:32:29methanesetmessages: + msg388091
2021-03-04 13:14:39eryksunsetmessages: + msg388084
2021-03-04 08:46:59methanesetpull_requests: + pull_request23513
2021-03-04 08:05:24methanesetkeywords: + patch
stage: patch review
pull_requests: + pull_request23512
2021-03-03 03:38:42methanesetmessages: + msg387995
2021-03-02 22:47:37steve.dowersetmessages: + msg387982
2021-03-02 19:09:49eryksunsetnosy: + eryksun
messages: + msg387954
2021-03-02 09:03:47methanecreate