classification
Title: Audit does not work with non-ASCII data on non-UTF-8 locale
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: miss-islington, serhiy.storchaka, steve.dower, vstinner
Priority: normal Keywords: patch

Created on 2020-06-23 20:25 by serhiy.storchaka, last changed 2020-06-24 17:16 by miss-islington. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 21095 merged serhiy.storchaka, 2020-06-23 20:32
PR 21107 closed miss-islington, 2020-06-24 05:46
PR 21108 closed miss-islington, 2020-06-24 05:46
PR 21119 merged serhiy.storchaka, 2020-06-24 16:10
PR 21120 merged miss-islington, 2020-06-24 16:12
PR 21121 merged miss-islington, 2020-06-24 16:12
PR 21124 merged miss-islington, 2020-06-24 16:46
PR 21125 merged miss-islington, 2020-06-24 16:46
Messages (10)
msg372205 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-06-23 20:25
There are issues with using PySys_Audit() with non-ASCII data on non-UTF-8 locale.

One example is with PYTHONSTARTUP. In pymain_run_startup() in Modules/main.c the value of the PYTHONSTARTUP environment variable is passed to PySys_Audit() as UTF-8 encoded data. If it contains non-ASCII characters and the locale encoding is different from UTF-8, it fails.

There are similar bugs in _Py_fopen() and _Py_fopen_obj().
msg372217 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-06-23 23:15
SGTM. Thanks for the PR!
msg372232 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-06-24 05:46
New changeset 6c6810d98979add7a89391c3c38990d0859f7a29 by Serhiy Storchaka in branch 'master':
bpo-41094: Fix decoding errors with audit when open files. (GH-21095)
https://github.com/python/cpython/commit/6c6810d98979add7a89391c3c38990d0859f7a29
msg372234 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-06-24 06:24
Oh, I made a mistake in tests (they do not test with non-ASCII data), but fixed tests fail on Windows.
msg372235 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-06-24 06:25
The problem is that they pass successfully on my local Windows, but fail on CI.
msg372259 - (view) Author: miss-islington (miss-islington) Date: 2020-06-24 16:45
New changeset 7329c8c7a263015c5dae04fc9fb7ec9d58508c76 by Miss Islington (bot) in branch '3.9':
bpo-41094: Fix decoding errors with audit when open files. (GH-21095)
https://github.com/python/cpython/commit/7329c8c7a263015c5dae04fc9fb7ec9d58508c76
msg372260 - (view) Author: miss-islington (miss-islington) Date: 2020-06-24 16:45
New changeset 1813d318fd4e517042415fa4f59fe8668c17a235 by Miss Islington (bot) in branch '3.8':
bpo-41094: Fix decoding errors with audit when open files. (GH-21095)
https://github.com/python/cpython/commit/1813d318fd4e517042415fa4f59fe8668c17a235
msg372261 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-06-24 16:46
New changeset a7dc71470156680f1fd5243290c6d377824b7ef4 by Serhiy Storchaka in branch 'master':
bpo-41094: Additional fix for PYTHONSTARTUP. (GH-21119)
https://github.com/python/cpython/commit/a7dc71470156680f1fd5243290c6d377824b7ef4
msg372267 - (view) Author: miss-islington (miss-islington) Date: 2020-06-24 17:03
New changeset 1dda40c1d2681a8f03a567b72698d88ced6bbd6c by Miss Islington (bot) in branch '3.8':
bpo-41094: Additional fix for PYTHONSTARTUP. (GH-21119)
https://github.com/python/cpython/commit/1dda40c1d2681a8f03a567b72698d88ced6bbd6c
msg372268 - (view) Author: miss-islington (miss-islington) Date: 2020-06-24 17:16
New changeset 29aa8085e95e106f4b9e312169d4af87e9b5e3e2 by Miss Islington (bot) in branch '3.9':
bpo-41094: Additional fix for PYTHONSTARTUP. (GH-21119)
https://github.com/python/cpython/commit/29aa8085e95e106f4b9e312169d4af87e9b5e3e2
History
Date User Action Args
2020-06-24 17:16:06miss-islingtonsetmessages: + msg372268
2020-06-24 17:06:51serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2020-06-24 17:03:56miss-islingtonsetmessages: + msg372267
2020-06-24 16:46:56miss-islingtonsetpull_requests: + pull_request20288
2020-06-24 16:46:47miss-islingtonsetpull_requests: + pull_request20287
2020-06-24 16:46:37serhiy.storchakasetmessages: + msg372261
2020-06-24 16:45:42miss-islingtonsetmessages: + msg372260
2020-06-24 16:45:35miss-islingtonsetmessages: + msg372259
2020-06-24 16:12:49miss-islingtonsetpull_requests: + pull_request20284
2020-06-24 16:12:40miss-islingtonsetpull_requests: + pull_request20283
2020-06-24 16:10:55serhiy.storchakasetpull_requests: + pull_request20282
2020-06-24 06:25:55serhiy.storchakasetmessages: + msg372235
2020-06-24 06:24:04serhiy.storchakasetmessages: + msg372234
2020-06-24 05:46:27miss-islingtonsetpull_requests: + pull_request20275
2020-06-24 05:46:21serhiy.storchakasetmessages: + msg372232
2020-06-24 05:46:20miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request20274
2020-06-23 23:15:43steve.dowersetmessages: + msg372217
2020-06-23 20:49:44serhiy.storchakalinkissue41069 dependencies
2020-06-23 20:32:16serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request20262
2020-06-23 20:25:16serhiy.storchakacreate