classification
Title: Non-ASCII characters in os.environ cause silent failures in test_httpservers
Type: behavior Stage: resolved
Components: Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: lukasz.langa, miss-islington
Priority: normal Keywords: patch

Created on 2021-07-15 17:50 by lukasz.langa, last changed 2021-07-16 13:55 by lukasz.langa. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 27161 merged lukasz.langa, 2021-07-15 17:53
PR 27169 merged miss-islington, 2021-07-15 19:14
PR 27170 merged miss-islington, 2021-07-15 19:14
PR 27187 merged lukasz.langa, 2021-07-16 12:07
PR 27191 merged miss-islington, 2021-07-16 13:24
Messages (6)
msg397572 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-07-15 17:50
GH-23638 introduced a new test for Accept: headers in CGI HTTP servers. This test serializes all of `os.environ` on the server side.  For non-UTF8 locales this can fail for some Unicode characters found in environment variables.

This started failing this week on Azure Pipelines with their rollout of a new Windows 2019 image version that included a "BUILD_SOURCEVERSIONAUTHOR" env variable. For me specifically it includes a leading Unicode character so all my PRs started failing on Azure Pipelines Windows 2019 alone.

The result was truncated output from the CGI HTTP server, like:

======================================================================
FAIL: test_accept (test.test_httpservers.CGIHTTPServerTestCase) [OrderedDict([('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8')])]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\1\s\lib\test\test_httpservers.py", line 860, in test_accept
    self.assertIn(expected.encode('ascii'), res.read())
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: b"'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'" not found in b''


The root cause is that the CGI script in question (`cgi_file6` in test_httpservers.py) is crashing on server-side reaching the `print(repr(os.environ))` line. However, this exception isn't visible on the client side where the test is running. The only visible issue is truncated output.

I suggest adding ENSURE_UNICODE_WORKS=Łukasz to the testing env so that this never regresses.
msg397576 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-07-15 19:14
New changeset 82b218f36ce6ef910bda5af227a9fd5be613c94f by Łukasz Langa in branch 'main':
bpo-44647: Fix test_httpservers failing on Unicode characters in os.environ on Windows (GH-27161)
https://github.com/python/cpython/commit/82b218f36ce6ef910bda5af227a9fd5be613c94f
msg397578 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-07-15 20:09
New changeset 356bdff1e952f2ab7fdebae855bd78b401b735c4 by Miss Islington (bot) in branch '3.10':
bpo-44647: Fix test_httpservers failing on Unicode characters in os.environ on Windows (GH-27161) (#27169)
https://github.com/python/cpython/commit/356bdff1e952f2ab7fdebae855bd78b401b735c4
msg397579 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-07-15 20:09
New changeset 95596d5921eeab9ae49f0dc31263a249013b6849 by Miss Islington (bot) in branch '3.9':
bpo-44647: Fix test_httpservers failing on Unicode characters in os.environ on Windows (GH-27161) (#27170)
https://github.com/python/cpython/commit/95596d5921eeab9ae49f0dc31263a249013b6849
msg397621 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-07-16 13:24
New changeset 7915c96ffd7ddc5cb6d54015ee4c31255a416892 by Łukasz Langa in branch 'main':
 bpo-44647: Add a permanent Unicode-valued env var to regrtest  (#27187)
https://github.com/python/cpython/commit/7915c96ffd7ddc5cb6d54015ee4c31255a416892
msg397630 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-07-16 13:55
New changeset 93d36a5bcec8e55026b9a749c8310b1ce490d15e by Miss Islington (bot) in branch '3.10':
bpo-44647: Add a permanent Unicode-valued env var to regrtest  (GH-27187) (#27191)
https://github.com/python/cpython/commit/93d36a5bcec8e55026b9a749c8310b1ce490d15e
History
Date User Action Args
2021-07-16 13:55:33lukasz.langasetmessages: + msg397630
2021-07-16 13:24:28miss-islingtonsetpull_requests: + pull_request25727
2021-07-16 13:24:11lukasz.langasetmessages: + msg397621
2021-07-16 12:07:04lukasz.langasetpull_requests: + pull_request25723
2021-07-15 20:13:07lukasz.langasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-07-15 20:09:59lukasz.langasetmessages: + msg397579
2021-07-15 20:09:36lukasz.langasetmessages: + msg397578
2021-07-15 19:14:55miss-islingtonsetpull_requests: + pull_request25707
2021-07-15 19:14:50miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request25706
2021-07-15 19:14:31lukasz.langasetmessages: + msg397576
2021-07-15 17:53:12lukasz.langasetkeywords: + patch
stage: patch review
pull_requests: + pull_request25705
2021-07-15 17:50:20lukasz.langacreate