classification
Title: Distutils generates the wrong export symbol for unicode module names
Type: behavior Stage: resolved
Components: Distutils Versions: Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: da-woods, dstufft, eric.araujo, lukasz.langa, miss-islington, ned.deily, petr.viktorin, scoder, steve.dower, vstinner
Priority: normal Keywords: patch

Created on 2020-01-23 13:48 by da-woods, last changed 2020-02-18 13:55 by miss-islington. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 18150 merged scoder, 2020-01-23 15:10
PR 18546 merged miss-islington, 2020-02-18 11:55
PR 18548 miss-islington, 2020-02-18 13:55
Messages (11)
msg360555 - (view) Author: (da-woods) * Date: 2020-01-23 13:48
Distuitls generates "export symbols" for extension modules to help ensure that they have have the correct linkage on Windows.

https://github.com/python/cpython/blob/0d30ae1a03102de07758650af9243fd31211325a/Lib/distutils/command/build_ext.py#L692

It generates the correct symbol in most causes, but if the filename contains unicode characters then it creates the wrong symbol, causing linkage errors.

The behaviour should be updated to reflect PEP-489: https://www.python.org/dev/peps/pep-0489/#export-hook-name
msg361352 - (view) Author: miss-islington (miss-islington) Date: 2020-02-04 15:24
New changeset 9538bc9185e934bee2bd5ae2cda2b2e92a61906d by Stefan Behnel in branch 'master':
bpo-39432: Implement PEP-489 algorithm for non-ascii "PyInit_*" symbol names in distutils (GH-18150)
https://github.com/python/cpython/commit/9538bc9185e934bee2bd5ae2cda2b2e92a61906d
msg361416 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2020-02-05 08:39
Ok, this is merged into 3.9. To which versions should we backport it?
Definitely 3.8, definitely not 3.5, probably not 3.6 (since it's not a security issue). Ned, what about 3.7?
msg361434 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-05 16:52
The test fails on Windows. Example on AMD64 Windows8.1 Refleaks 3.x:
https://buildbot.python.org/all/#/builders/157/builds/76

======================================================================
FAIL: test_unicode_module_names (distutils.tests.test_build_ext.BuildExtTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\distutils\tests\test_build_ext.py", line 315, in test_unicode_module_names
    self.assertRegex(cmd.get_ext_filename(modules[0].name), r'foo\..*')
AssertionError: Regex didn't match: 'foo\\..*' not found in 'foo_d.cp39-win_amd64.pyd'

======================================================================
FAIL: test_unicode_module_names (distutils.tests.test_build_ext.ParallelBuildExtTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\distutils\tests\test_build_ext.py", line 315, in test_unicode_module_names
    self.assertRegex(cmd.get_ext_filename(modules[0].name), r'foo\..*')
AssertionError: Regex didn't match: 'foo\\..*' not found in 'foo_d.cp39-win_amd64.pyd'
msg361435 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-05 16:52
Python 3.6 doesn't accept bugfixes anymore:
https://devguide.python.org/#status-of-python-branches

The bugfix can go into 3.7 and 3.8.
msg361436 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-05 16:55
I put a breakpoint before the error:

test_unicode_module_names (distutils.tests.test_build_ext.BuildExtTestCase) ... > c:\vstinner\python\master\lib\distutils\tests
\test_build_ext.py(316)test_unicode_module_names()
-> self.assertRegex(cmd.get_ext_filename(modules[0].name), r'foo\..*')
(Pdb) p modules
[<distutils.extension.Extension('foo') at 0x274003035f0>, <distutils.extension.Extension('föö') at 0x27400303730>]
(Pdb) p modules[0].name
'foo'
(Pdb) p modules[1].name 
'föö'
msg361437 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-05 16:57
On Windows, names get a "_d" suffix for debug. Extract of build_ext.py:

    def get_libraries(self, ext):
        """Return the list of libraries to link against when building a
        shared extension.  On most platforms, this is just 'ext.libraries';
        on Windows, we add the Python library (eg. python20.dll).
        """
        # The python library is always needed on Windows.  For MSVC, this
        # is redundant, since the library is mentioned in a pragma in
        # pyconfig.h that MSVC groks.  The other Windows compilers all seem
        # to need it mentioned explicitly, though, so that's what we do.
        # Append '_d' to the python import library on debug builds.
        if sys.platform == "win32":
            from distutils._msvccompiler import MSVCCompiler
            if not isinstance(self.compiler, MSVCCompiler):
                template = "python%d%d"
                if self.debug:
                    template = template + '_d'
(...)
msg361460 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-02-05 22:29
issue39555 and PR 18357 have the fix for the buildbot.
msg361608 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-02-07 18:48
We should not be changing Distutils 3.7 behavior at this late point in its life cycle, particularly since AFAIK this issue has not come up before.  Let's see what Łukasz thinks for 3.8.
msg361800 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2020-02-11 11:35
I'm with Stefan on "Definitely 3.8". It's a bug fix (for a rarely used feature).
msg362206 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2020-02-18 13:14
New changeset 5bf58cef151249f1cca92166d1b70693348da9d8 by Miss Islington (bot) in branch '3.8':
bpo-39432: Implement PEP-489 algorithm for non-ascii "PyInit_*" symbol names in distutils (GH-18150) (GH-18546)
https://github.com/python/cpython/commit/5bf58cef151249f1cca92166d1b70693348da9d8
History
Date User Action Args
2020-02-18 13:55:42miss-islingtonsetpull_requests: + pull_request17928
2020-02-18 13:17:31scodersetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2020-02-18 13:14:47scodersetmessages: + msg362206
2020-02-18 11:55:05miss-islingtonsetstage: backport needed -> patch review
pull_requests: + pull_request17924
2020-02-11 11:35:46petr.viktorinsetmessages: + msg361800
2020-02-07 18:48:13ned.deilysetnosy: + lukasz.langa

messages: + msg361608
versions: - Python 3.7
2020-02-05 22:29:00steve.dowersetmessages: + msg361460
2020-02-05 16:57:18vstinnersetmessages: + msg361437
2020-02-05 16:55:34vstinnersetmessages: + msg361436
2020-02-05 16:52:49vstinnersetmessages: + msg361435
versions: - Python 3.5, Python 3.6
2020-02-05 16:52:10vstinnersetnosy: + vstinner
messages: + msg361434
2020-02-05 08:39:04scodersetnosy: + ned.deily

messages: + msg361416
stage: patch review -> backport needed
2020-02-04 15:24:38miss-islingtonsetnosy: + miss-islington
messages: + msg361352
2020-01-28 08:06:00scodersetnosy: + steve.dower
2020-01-23 15:16:31scodersetnosy: + petr.viktorin
2020-01-23 15:10:07scodersetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request17536
2020-01-23 14:21:29scodersetnosy: + scoder
2020-01-23 14:21:10scodersetstage: needs patch
2020-01-23 13:48:15da-woodscreate