classification
Title: Azure Pipelines: appx tests fail: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Type: Stage: resolved
Components: Windows Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: lukasz.langa, paul.moore, steve.dower, tim.golden, vstinner, zach.ware
Priority: release blocker Keywords: patch

Created on 2019-09-30 11:27 by vstinner, last changed 2019-09-30 20:31 by steve.dower. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 16489 merged vstinner, 2019-09-30 12:28
PR 16490 merged vstinner, 2019-09-30 12:29
Messages (9)
msg353565 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-30 11:27
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = 'D:\a\1\b\layout-appx-amd64\python.exe'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = 'D:\\a\\1\\b\\layout-appx-amd64\\.\\.\\.\\python.exe'
  sys.base_prefix = 'D:\\a\\1\\b\\layout-appx-amd64\\lib\\o'
  sys.base_exec_prefix = 'D:\\a\\1\\b\\layout-appx-amd64\\lib\\o'
  sys.executable = 'D:\\a\\1\\b\\layout-appx-amd64\\.\\.\\.\\python.exe'
  sys.prefix = 'D:\\a\\1\\b\\layout-appx-amd64\\lib\\o'
  sys.exec_prefix = 'D:\\a\\1\\b\\layout-appx-amd64\\lib\\o'
  sys.path = [
    'D:\\a\\1\\b\\layout-appx-amd64\\python38.zip',
    'D:\\a\\1\\b\\layout-appx-amd64\\lib\\o\\DLLs',
    'D:\\a\\1\\b\\layout-appx-amd64\\lib\\o\\lib',
    'D:\\a\\1\\b\\layout-appx-amd64\\.\\.\\.',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'
msg353566 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-30 12:49
New changeset dec39716ca93ee2e8d9b94915ece33014eb58e9e by Victor Stinner in branch 'master':
bpo-38322: Fix gotlandmark() of PC/getpathp.c (GH-16489)
https://github.com/python/cpython/commit/dec39716ca93ee2e8d9b94915ece33014eb58e9e
msg353567 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-30 12:49
New changeset 18c4ba9f33868761e374a725d497902863d59ea9 by Victor Stinner in branch '3.8':
bpo-38322: Fix gotlandmark() of PC/getpathp.c (GH-16490)
https://github.com/python/cpython/commit/18c4ba9f33868761e374a725d497902863d59ea9
msg353572 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-30 13:34
Facts:

* appx tests are not run on buildbots
* appx tests are not run in pre-commit CI of pull requests
* appx tests seem to only be run on Azure Pipelines of the branches (like 3.8 and master)
* the fixed bug likely existed since Python 3.6:

static int
gotlandmark(const wchar_t *landmark)
{
    Py_ssize_t n = wcsnlen_s(prefix, MAXPATHLEN);
    join(prefix, landmark);
    ...
    prefix[n] = '\0';
    ...
}

with join which uses _PathCchCombineEx() or PathCombineW()

... No idea why the bug decided to show up today, 3.8.0rc1 release day.

Anyway, the Azure Pipelines CI is back to green on the 3.8 branch.
msg353582 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-09-30 14:52
The bug was probably exposed due to a change in path initialization (there's been a few changes here recently ;) ) that stopped normalising prefix. You can see that sys.executable in your dumped path has additional "." in the path, and that's never been the case before - they're always cleaned up earlier than that.

Adding in the appx _layout_ test (it's the whole test suite in an "installed" layout) was easier to do in CI than creating a buildbot config, so I threw it in as a test and then left it on. It's certainly not been failing this whole time.
msg353614 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-30 19:46
> The bug was probably exposed due to a change in path initialization (there's been a few changes here recently ;) ) that stopped normalising prefix. You can see that sys.executable in your dumped path has additional "." in the path, and that's never been the case before - they're always cleaned up earlier than that.

Yeah, I had to modify a lot of "path config" code to fix many bugs (regressions compared to Python 3.7). I also added the first tests on the "path configuration" in test_embed.

I checked why sys.executable is not normalized in the the "appx build". PC\python_uwp.cpp calls set_process_name() which sets PyConfig.executable and the path is not normalized. If PyConfig.executable is set explicitly, the path is left unchanged: PyConfig has the highest priority.

Should the executable path be normalized in this case?

--

I looked at Python 3.6 (the version before I started to rework the Python initialization). But there is no python_uwp.cpp in this version.

In Python 3.7, python_uwp.cpp calls set_process_name() which calls _Py_SetProgramFullPath(). But config_init_path_config() seems to ignore the _Py_SetProgramFullPath() call in Python 3.7.
msg353615 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-30 19:47
> Adding in the appx _layout_ test (it's the whole test suite in an "installed" layout) was easier to do in CI than creating a buildbot config, so I threw it in as a test and then left it on. It's certainly not been failing this whole time.

I'm fine with keeping it here. I don't pay much attention to it, since I don't get an email when a Azure Pipelones job fails, and the UI is kind of "hidden" in GitHub (not as visible as CI results on a PR).
msg353618 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-09-30 20:27
> I checked why sys.executable is not normalized in the the "appx build". PC\python_uwp.cpp calls set_process_name() which sets PyConfig.executable and the path is not normalized. If PyConfig.executable is set explicitly, the path is left unchanged: PyConfig has the highest priority.

Ah, that would explain it, certainly. I don't know why the dots are ever added though.

This test also has the weird oddity that it isn't running as an actual app, so a different initialization path is taken. In this case, it should be using the final name as reported by the operating system, so the extra dots are basically unexplainable. I'll try a couple of experiments and see if I can figure it out.

> the UI is kind of "hidden" in GitHub

Yeah, but I look at it periodically (the "Commits" view is a convenient way to see recent CI and buildbot status). It's considerably simpler for me than finding out it's broken in an RC :)
msg353619 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-09-30 20:31
Either way, the paths in the latest tests [1] seem to be fine, and they all passed, so I don't think this is a release blocker anymore.

Thanks, Victor!

[1]: https://dev.azure.com/Python/cpython/_build/results?buildId=52024&view=logs&j=c8a71634-e5ec-54a0-3958-760f4148b765
History
Date User Action Args
2019-10-22 23:18:35vstinnerlinkissue31074 superseder
2019-09-30 20:31:39steve.dowersetstatus: open -> closed
resolution: fixed
messages: + msg353619

stage: patch review -> resolved
2019-09-30 20:27:41steve.dowersetmessages: + msg353618
2019-09-30 19:47:09vstinnersetmessages: + msg353615
2019-09-30 19:46:33vstinnersetmessages: + msg353614
2019-09-30 14:52:13steve.dowersetmessages: + msg353582
2019-09-30 13:34:53vstinnersetmessages: + msg353572
2019-09-30 12:49:45vstinnersetmessages: + msg353567
2019-09-30 12:49:40vstinnersetmessages: + msg353566
2019-09-30 12:29:05vstinnersetpull_requests: + pull_request16076
2019-09-30 12:28:12vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request16075
2019-09-30 11:27:09vstinnercreate