classification
Title: distutils fails to build extension on windows when it is a package.__init__
Type: behavior Stage: resolved
Components: Distutils, Windows Versions: Python 3.8, Python 3.7, Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: dstufft, eric.araujo, evandrocoan, p-ganssle, paul.moore, ronaldoussoren, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2019-02-03 21:10 by ronaldoussoren, last changed 2021-02-03 18:10 by steve.dower. This issue is now closed.

Messages (10)
msg334800 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2019-02-03 21:10
Python supports having a C extension for the the __init__ of a package (instead of having __init__.py). This works fine on Linux, but on Windows distutils fails to build the C extension because it assumes the entry point is named PyInit___init__ while importlib expects PyInit_*package* (for a package named *package*). 

When building the extension I get the following error:

LINK : error LNK2001: unresolved external symbol PyInit___init__
build\temp.win32-3.7\Release\__init__.cp37-win32.lib : fatal error LNK1120: 1 unresolved externals


The code below can be used to reproduce the issue.

Setup.py (extracted from a larger setup.py, but should work...):

from setuptools import setup, Extension
extension3 = Extension("ext_package.__init__", sources=["init.c"])

setup(
    ext_modules=[extension3],
)

Source code for the module (init.c):

#include "Python.h"
  

static PyModuleDef mod_def = {
        PyModuleDef_HEAD_INIT,
        "ext_package.__init__",
        NULL,
        0,
        NULL,
        NULL,
        NULL,
        NULL,
        NULL
};

PyObject* PyInit_ext_package(void)
{
        return PyModule_Create(&mod_def);
}


P.S. I cannot easily debug this, I ran into this when testing one of my projects on AppVeyor and don't have a local Windows machine.
msg334801 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2019-02-03 21:18
FWIW the project in question is modulegraph2 <https://bitbucket.org/ronaldoussoren/modulegraph2/src/default/>, which almost works on Windows, except for this issue and some incorrect code on my side that assumes virtualenv on Windows behaves the same as on POSIX w.r.t. the structure of virtual environments.
msg334836 - (view) Author: Paul Ganssle (p-ganssle) * (Python committer) Date: 2019-02-04 19:16
@Ronald The module you've linked to seems to be using flit and doesn't have any C extensions. Did you change over the build process, or am I missing something?
msg334843 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2019-02-04 21:41
@Paul, modulegraph2 uses flit, but I do use setuptools to build extensions in the test suite.  The problematic setup.py file is in testsuite/nodebuilder-tree.

I do have a workaround in that setup.py:

# START
def get_export_symbols(self, ext):
    parts = ext.name.split(".")
    if parts[-1] == "__init__":
        initfunc_name = "PyInit_" + parts[-2]
    else:
        initfunc_name = "PyInit_" + parts[-1]


build_ext.build_ext.get_export_symbols = get_export_symbols
# END

Creating a PR from this would be easy, but IMHO there should be a new test case as well.
msg334845 - (view) Author: Paul Ganssle (p-ganssle) * (Python committer) Date: 2019-02-04 21:53
@Ronald Ah, interesting, I missed that.

In my experience, distutils is pretty static and it's not particularly common to merge changes into it. Whether or not this is in scope for distutils, it's definitely in scope for setuptools - do you mind opening an issue on the setuptools tracker? https://github.com/pypa/setuptools
msg334847 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2019-02-04 22:17
The bug is the stdlib, and I've checked that setuptools just calls the code in distutils in this case.

IMHO it would be better to fix this in the stdlib.

(Unselecting python 3.6 because that's in security-only mode by now)
msg334848 - (view) Author: Paul Ganssle (p-ganssle) * (Python committer) Date: 2019-02-04 22:23
Well, there's some tentative plan for `setuptools` to completely adopt distutils, so in some sense all distutils bugs are setuptools bugs as well.

That said, the reason to report it in setuptools as well is that setuptools still supports Python 2.7, 3.5 and 3.6, so even if this bug is fixed in the standard library, it still needs to be fixed in setuptools.

Of course it may not be easy or feasible to fix it in setuptools until *after* we fully adopt distutils, but ¯\_(ツ)_/¯ might as well still report it.
msg334850 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-02-04 22:45
Agreed that it should go to setuptools in order to support third-party packages.

In general, those of us who ever touch distutils these days (basically just me?) are only really maintaining those parts necessary to build Python and its own modules. Since this doesn't impact that scenario, it's not going to get much attention until there's at least a PR and tests to review.
msg342821 - (view) Author: Evandro Coan (evandrocoan) * Date: 2019-05-18 22:34
It is missing the import on:

#START
from distutils.command import build_ext

def get_export_symbols(self, ext):
    parts = ext.name.split(".")
    print('parts', parts)
    if parts[-1] == "__init__":
        initfunc_name = "PyInit_" + parts[-2]
    else:
        initfunc_name = "PyInit_" + parts[-1]

build_ext.build_ext.get_export_symbols = get_export_symbols
#END
msg386289 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-02-03 18:10
Distutils is now deprecated (see PEP 632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.

If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools
History
Date User Action Args
2021-02-03 18:10:57steve.dowersetstatus: open -> closed
resolution: out of date
messages: + msg386289

stage: resolved
2019-05-18 22:34:34evandrocoansetnosy: + evandrocoan
messages: + msg342821
2019-02-04 22:45:37steve.dowersetmessages: + msg334850
versions: + Python 2.7, Python 3.8
2019-02-04 22:23:48p-gansslesetmessages: + msg334848
2019-02-04 22:17:45ronaldoussorensetmessages: + msg334847
versions: - Python 3.6
2019-02-04 21:53:35p-gansslesetmessages: + msg334845
2019-02-04 21:41:07ronaldoussorensetmessages: + msg334843
2019-02-04 19:16:18p-gansslesetmessages: + msg334836
2019-02-04 19:08:28p-gansslesetnosy: + p-ganssle
2019-02-03 21:18:54ronaldoussorensetmessages: + msg334801
2019-02-03 21:10:11ronaldoussorencreate