Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = 'D:\a\1\b\layout-appx-amd64\python.exe'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = 'D:\\a\\1\\b\\layout-appx-amd64\\.\\.\\.\\python.exe'
  sys.base_prefix = 'D:\\a\\1\\b\\layout-appx-amd64\\lib\\o'
  sys.base_exec_prefix = 'D:\\a\\1\\b\\layout-appx-amd64\\lib\\o'
  sys.executable = 'D:\\a\\1\\b\\layout-appx-amd64\\.\\.\\.\\python.exe'
  sys.prefix = 'D:\\a\\1\\b\\layout-appx-amd64\\lib\\o'
  sys.exec_prefix = 'D:\\a\\1\\b\\layout-appx-amd64\\lib\\o'
  sys.path = [
    'D:\\a\\1\\b\\layout-appx-amd64\\python38.zip',
    'D:\\a\\1\\b\\layout-appx-amd64\\lib\\o\\DLLs',
    'D:\\a\\1\\b\\layout-appx-amd64\\lib\\o\\lib',
    'D:\\a\\1\\b\\layout-appx-amd64\\.\\.\\.',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

New changeset dec39716ca93ee2e8d9b94915ece33014eb58e9e by Victor Stinner in branch 'master':
bpo-38322: Fix gotlandmark() of PC/getpathp.c (GH-16489)
https://github.com/python/cpython/commit/dec39716ca93ee2e8d9b94915ece33014eb58e9e

New changeset 18c4ba9f33868761e374a725d497902863d59ea9 by Victor Stinner in branch '3.8':
bpo-38322: Fix gotlandmark() of PC/getpathp.c (GH-16490)
https://github.com/python/cpython/commit/18c4ba9f33868761e374a725d497902863d59ea9

Facts:

* appx tests are not run on buildbots
* appx tests are not run in pre-commit CI of pull requests
* appx tests seem to only be run on Azure Pipelines of the branches (like 3.8 and master)
* the fixed bug likely existed since Python 3.6:

static int
gotlandmark(const wchar_t *landmark)
{
    Py_ssize_t n = wcsnlen_s(prefix, MAXPATHLEN);
    join(prefix, landmark);
    ...
    prefix[n] = '\0';
    ...
}

with join which uses _PathCchCombineEx() or PathCombineW()

... No idea why the bug decided to show up today, 3.8.0rc1 release day.

Anyway, the Azure Pipelines CI is back to green on the 3.8 branch.

The bug was probably exposed due to a change in path initialization (there's been a few changes here recently ;) ) that stopped normalising prefix. You can see that sys.executable in your dumped path has additional "." in the path, and that's never been the case before - they're always cleaned up earlier than that.

Adding in the appx _layout_ test (it's the whole test suite in an "installed" layout) was easier to do in CI than creating a buildbot config, so I threw it in as a test and then left it on. It's certainly not been failing this whole time.

> The bug was probably exposed due to a change in path initialization (there's been a few changes here recently ;) ) that stopped normalising prefix. You can see that sys.executable in your dumped path has additional "." in the path, and that's never been the case before - they're always cleaned up earlier than that.

Yeah, I had to modify a lot of "path config" code to fix many bugs (regressions compared to Python 3.7). I also added the first tests on the "path configuration" in test_embed.

I checked why sys.executable is not normalized in the the "appx build". PC\python_uwp.cpp calls set_process_name() which sets PyConfig.executable and the path is not normalized. If PyConfig.executable is set explicitly, the path is left unchanged: PyConfig has the highest priority.

Should the executable path be normalized in this case?

--

I looked at Python 3.6 (the version before I started to rework the Python initialization). But there is no python_uwp.cpp in this version.

In Python 3.7, python_uwp.cpp calls set_process_name() which calls _Py_SetProgramFullPath(). But config_init_path_config() seems to ignore the _Py_SetProgramFullPath() call in Python 3.7.

> Adding in the appx _layout_ test (it's the whole test suite in an "installed" layout) was easier to do in CI than creating a buildbot config, so I threw it in as a test and then left it on. It's certainly not been failing this whole time.

I'm fine with keeping it here. I don't pay much attention to it, since I don't get an email when a Azure Pipelones job fails, and the UI is kind of "hidden" in GitHub (not as visible as CI results on a PR).

> I checked why sys.executable is not normalized in the the "appx build". PC\python_uwp.cpp calls set_process_name() which sets PyConfig.executable and the path is not normalized. If PyConfig.executable is set explicitly, the path is left unchanged: PyConfig has the highest priority.

Ah, that would explain it, certainly. I don't know why the dots are ever added though.

This test also has the weird oddity that it isn't running as an actual app, so a different initialization path is taken. In this case, it should be using the final name as reported by the operating system, so the extra dots are basically unexplainable. I'll try a couple of experiments and see if I can figure it out.

> the UI is kind of "hidden" in GitHub

Yeah, but I look at it periodically (the "Commits" view is a convenient way to see recent CI and buildbot status). It's considerably simpler for me than finding out it's broken in an RC :)

Either way, the paths in the latest tests [1] seem to be fine, and they all passed, so I don't think this is a release blocker anymore.

Thanks, Victor!

[1]: https://dev.azure.com/Python/cpython/_build/results?buildId=52024&view=logs&j=c8a71634-e5ec-54a0-3958-760f4148b765