classification
Title: site: Potential UnicodeDecodeError when handling pth file
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: closed Resolution: not a bug
Dependencies: Superseder: Use io.open_code for .pth files
View: 37364
Assigned To: Nosy List: methane, vstinner
Priority: normal Keywords: 3.8regression, easy, patch

Created on 2021-02-13 08:53 by methane, last changed 2021-03-17 10:30 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 24837 closed methane, 2021-03-13 07:09
Messages (3)
msg386916 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2021-02-13 08:53
https://github.com/python/cpython/blob/4230bd52e3f9f289f02e41ab17a95f50ed4db5a6/Lib/site.py#L160

```
        f = io.TextIOWrapper(io.open_code(fullname))
```

When default text encoding is not UTF-8 and pth file contains non-ASCII character, it will raise UnicodeDecodeError.
msg388911 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2021-03-17 08:54
locale-specific encoding is not good especially for Windows.
But we used it for a long time. Changing the encoding for pth files is breaking change.
msg388916 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-17 10:30
Since it's a Python script, the default encoding should be UTF-8, as any Python script. I guess that most pth files don't use characters outside ASCII so it's fine.

I think that distutils made a few changes to switch UTF-8 last years, so it's possible.
History
Date User Action Args
2021-03-17 10:30:46vstinnersetnosy: + vstinner
messages: + msg388916
2021-03-17 08:54:25methanesetstatus: open -> closed
resolution: not a bug
messages: + msg388911

stage: patch review -> resolved
2021-03-13 07:09:35methanesetkeywords: + patch
stage: patch review
pull_requests: + pull_request23602
2021-03-13 07:08:03methanesetsuperseder: Use io.open_code for .pth files
2021-02-13 08:55:43methanesetkeywords: + easy
2021-02-13 08:53:26methanecreate