classification
Title: Cygwin build broken due to use of &PyType_Type in static declaration in _abc module
Type: compile error Stage: patch review
Components: Build Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: erik.bray, jdemeyer
Priority: normal Keywords: 3.7regression, patch

Created on 2018-07-24 17:00 by erik.bray, last changed 2018-08-03 17:33 by erik.bray.

Pull Requests
URL Status Linked Edit
PR 8445 open erik.bray, 2018-07-24 17:03
Messages (5)
msg322315 - (view) Author: Erik Bray (erik.bray) * (Python triager) Date: 2018-07-24 17:00
This is essentially the same as issue21124, but introduced more recently with the addition of the _abc_data_type in the _abc module.

The workaround is simply to use PyVarObject_HEAD_INIT(NULL, 0) instead of PyVarObject_HEAD_INIT(&PyType_Type, 0); PyType_Ready should take care of the rest.

P.S. I'm trying to get going again on the project of adding an AppVeyor build, and eventually a buildbot for Cygwin.  My previous attempt was doomed because I wanted to fix all failing tests on Cygwin *first*.  This time I am going for a more "instant gratification" approach of just skipping the tests that fail (for now) so that I can at least catch new regressions, and then gradually re-enabled skipped tests as I can find fixes for them.

However, for that to happen we at least need a minimal build to work.
msg322823 - (view) Author: Jeroen Demeyer (jdemeyer) * Date: 2018-07-31 22:06
For those who are not very aware of Cygwin issues: what is wrong with

PyVarObject_HEAD_INIT(&PyType_Type, 0);
msg322857 - (view) Author: Erik Bray (erik.bray) * (Python triager) Date: 2018-08-01 12:05
> For those who are not very aware of Cygwin issues: what is wrong with
>
> PyVarObject_HEAD_INIT(&PyType_Type, 0);

I'm glad you asked, because it actually got me thinking, and since I added a fix (albeit an unsatisfying one) for issue34212, this fix is no longer strictly necessary *so long as* the _abc module is always built as a core built-in (that is, linked into libpython).

IIUC that is the case since _abc is required for the core, but I'm not sure.

The problem is explained somewhat in issue21124, but what it comes down to is that if the linker can't resolve PyType_Type at link time it will make a complaint like "can't initialize global with a non-constant".

Because of issue34212, when compiling _abc.c it was using the wrong external linkage for PyType_Type (treating it as some data that needs to be imported from an other DLL, rather than data in the same DLL).  But with a fix for issue34212 this is not a problem (again, so long as the _abc module is included in libpython).
msg322858 - (view) Author: Jeroen Demeyer (jdemeyer) * Date: 2018-08-01 12:26
Linker issues are always tricky...

I understand that there is no problem within libpython, so the questions below refer to extension modules. I am asking them from the point of view of somebody writing Python extension modules who is clueless about Cygwin.

So you're saying that it's not allowed to refer to &PyType_Type in a static struct? But it is OK to refer to &PyType_Type in code?

I assume that there is nothing special about PyType_Type and that this apply for all variables. Are functions OK? For example, in functools.c I see

static PyGetSetDef partial_getsetlist[] = {
    {"__dict__", PyObject_GenericGetDict, PyObject_GenericSetDict},
    {NULL} /* Sentinel */
};

What makes functions different from variables? Aren't they essentially just pointers?
msg323062 - (view) Author: Erik Bray (erik.bray) * (Python triager) Date: 2018-08-03 17:33
> What makes functions different from variables? Aren't they essentially just pointers?

You're on the right track by noting a difference between functions and data variables.  I can tell you off the bat that when it comes to data imported from DLLs, non-functions are handled (somewhat by necessity) quite differently from functions.

That said, since you asked, I struggled to articulate *exactly* why this exact problem occurs on Cygwin (actually on Windows in general), so I thought about it for a while and wrote up an explanation in a blog post: http://iguananaut.net/blog/programming/windows-data-import.html

The TL;DR though is that limitations of how the runtime dynamic loader on Windows works are such that it's impossible to initialize static data with a pointer to some data in an external library.  The compiler knows this and prevents you from doing it.  The workaround is simple enough for most cases: Complete the initialization at runtime.  In the case of PyType_Type objects, PyType_Ready can set their base type at runtime just fine.
History
Date User Action Args
2018-08-03 17:33:47erik.braysetmessages: + msg323062
2018-08-01 12:26:57jdemeyersetmessages: + msg322858
2018-08-01 12:05:23erik.braysetmessages: + msg322857
2018-07-31 22:06:20jdemeyersetnosy: + jdemeyer
messages: + msg322823
2018-07-24 17:03:16erik.braysetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request7967
2018-07-24 17:00:16erik.braycreate