classification
Title: python 3.7 openpty/forkpty build failure using nix package manager macOS environment
Type: compile error Stage: patch review
Components: Build, macOS Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: LnL7, ned.deily, orivej, ronaldoussoren
Priority: normal Keywords: patch

Created on 2018-07-02 21:31 by LnL7, last changed 2018-07-03 17:20 by LnL7.

Files
File name Uploaded Description Edit
darwin-libutil.patch orivej, 2018-07-03 00:34
Pull Requests
URL Status Linked Edit
PR 8056 closed LnL7, 2018-07-02 21:44
Messages (13)
msg320913 - (view) Author: Daiderd Jordan (LnL7) * Date: 2018-07-02 21:31
I can't really figure out why, but the <util.h> import in posixmodule.c seems to get skipped since python 3.7. The condition doesn't look entirely correct in case libutil.h is also available on macOS, however this has not changed since 3.6 and it worked fine there while both where available.


Part of the configure log:

checking pty.h usability... no
checking pty.h presence... no
checking for pty.h... no
checking libutil.h usability... yes
checking libutil.h presence... yes
checking for libutil.h... yes
...
checking util.h usability... yes
checking util.h presence... yes
checking for util.h... yes


Build error:

./Modules/posixmodule.c:5924:9: error: implicit declaration of function 'openpty' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
    if (openpty(&master_fd, &slave_fd, NULL, NULL, NULL) != 0)
        ^
./Modules/posixmodule.c:5924:9: note: did you mean 'openat'?
/nix/store/q819d3vjz7vswpvkrfa9gck3ys8rmvcj-Libsystem-osx-10.11.6/include/sys/fcntl.h:480:5: note: 'openat' declared here
int     openat(int, const char *, int, ...) __DARWIN_NOCANCEL(openat) __OSX_AVAILABLE_STARTING(__MAC_10_10, __IPHONE_8_0);
        ^
./Modules/posixmodule.c:5924:9: warning: this function declaration is not a prototype [-Wstrict-prototypes]
    if (openpty(&master_fd, &slave_fd, NULL, NULL, NULL) != 0)
        ^
./Modules/posixmodule.c:6018:11: error: implicit declaration of function 'forkpty' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
    pid = forkpty(&master_fd, NULL, NULL, NULL);
          ^
./Modules/posixmodule.c:6018:11: warning: this function declaration is not a prototype [-Wstrict-prototypes]
2 warnings and 2 errors generated.
make: *** [Makefile:1793: Modules/posixmodule.o] Error 1
msg320914 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-07-02 22:04
I don't see this build failure.  Can you say where you find libutil.h installed on macOS and what version of macOS this is?  It doesn't seem to be part of the standard Apple-supplied /usr/include header files installed by xcode-select --install for macOS 10.13 at least?
msg320915 - (view) Author: Daiderd Jordan (LnL7) * Date: 2018-07-02 22:16
This is a build using the nix package manager, we use all of the opensource components provided by apple instead of relying on xcode. The libutil.h that's found in this case is the one from the 10.11.6 sources.

https://opensource.apple.com/source/libutil/libutil-43/
msg320918 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-07-02 22:33
Even on macOS 10.11.6, libutil.h is not installed and posixmodule.c builds just fine.  So I don't think that the presence of something in opensource.apple.com necessarily means it was installed in place in a standard system.  I'm not saying yet that we shouldn't change the tests along the line you suggest in the PR.  But I really would like to understand first why we should make a change when as far as I recall this issue has never come up before.
msg320920 - (view) Author: Daiderd Jordan (LnL7) * Date: 2018-07-02 22:49
The headers end up in the build environment because of Libsystem which is part of our base environment. But yes, this is also the case with 3.6 and that builds just fine so I'm not sure why it's failing now.

These are the logs from our build service for reference:
python37.x86_64-darwin https://hydra.nixos.org/build/76780195/nixlog/2
python36.x86_64-darwin https://hydra.nixos.org/build/76018915/nixlog/1
msg320921 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-07-02 22:56
It's failing in 3.7 because of the change introduced in Issue27659 to prohibit implicit C function declarations by adding the -Werror=implicit-function-declaration to compiles.
msg320925 - (view) Author: Daiderd Jordan (LnL7) * Date: 2018-07-02 23:10
Oh interesting, so it only used to worked by accident. I should have looked at the 3.6 log more closely.
msg320930 - (view) Author: Orivej Desh (orivej) * Date: 2018-07-03 00:34
The explicit conditional on __APPLE__ can be avoided with the attached patch. (Tested on NixOS and macOS with Nixpkgs.)
msg320931 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-07-03 01:52
Thanks for the suggested patch but I am not persuaded that there is a problem needing fixing on the Python side.  As I see it, Python's posixmodule builds fine in the environments we claim to support, including macOS, and has for a long time.  It appears that the build failure reported here is because Nix is not properly emulating a macOS build environment and that builds under Nix have been producing warnings prior to 3.7 so it's not a new issue.  Rather than trying to patch around the Nix problem on the Python side, and potentially introducing problems on platforms that Python "officially" supports, it seems to me the better options are for either (1) Nix to better emulate a real macOS build environment by supplying a more correct set of header files (which also might help other third-party packages trying to use Nix for macOS builds); or, if that is not possible, (2) to carry a local patch for Python when building under Nix.  I'm open to persuasion but it would have to be a very compelling argument.  We have never claimed to support macOS builds on anything other than Apple-supplied macOS development environments running on macOS and I don't see that changing.
msg320933 - (view) Author: Orivej Desh (orivej) * Date: 2018-07-03 02:50
I understand the desirability of avoiding potential problems on supported platforms (given that you can not test building Python on all of them) and recognize that both options you have given are reasonable, but as I see it posixmodule.c incorrectly uses autoconf checks for the presence of header files to infer the platform: it assumes that libutil.h can not exist on macOS. It should either include all found relevant headers, as proposed in my patch or done e.g. in dropbear [1] and gnulib [2], or it should dispatch on the platform directly as LnL7 has proposed (although in this case it should no longer use the autoconf check on macOS).

[1] https://secure.ucc.asn.au/hg/dropbear/file/DROPBEAR_2018.76/includes.h
[2] http://git.savannah.gnu.org/cgit/gnulib.git/tree/lib/pty.in.h?id=281b825eba78dc403b9bf34979923bc587bc31af
msg320936 - (view) Author: Daiderd Jordan (LnL7) * Date: 2018-07-03 06:35
Even tho it's headers are not /usr/lib/libutil.dylib is available on every macOS system. Unlike on bsd this doesn't include the relevant symbols for openpty/forkpty, that's why I used an __APPLE__ condition but including both also works fine. Assuming it's headers are not available doesn't seem like a good assumption, even tho they won't be in most cases.
msg320940 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2018-07-03 07:47
I agree with Ned that we shouldn't add code to deal with non-standard compiler setups, correctly supporting macOS is hard enough as it is.

That said, the attached patch (but not PR8056) looks fine to me, it includes all possibly relevant headers files (as mentioned in msg320933) and that should be ok.  The PR is not acceptable because it unnecessarily contains macOS specific code.
msg320992 - (view) Author: Daiderd Jordan (LnL7) * Date: 2018-07-03 17:20
Either patch looks fine to me, should I close the pull request?
History
Date User Action Args
2018-07-03 17:20:44LnL7setmessages: + msg320992
2018-07-03 07:47:23ronaldoussorensetmessages: + msg320940
2018-07-03 06:35:12LnL7setmessages: + msg320936
2018-07-03 02:50:15orivejsetmessages: + msg320933
2018-07-03 01:52:47ned.deilysetmessages: + msg320931
title: python 3.7 openpty/forkpty build failure on macOS -> python 3.7 openpty/forkpty build failure using nix package manager macOS environment
2018-07-03 00:34:50orivejsetfiles: + darwin-libutil.patch
nosy: + orivej
messages: + msg320930

2018-07-02 23:10:23LnL7setmessages: + msg320925
2018-07-02 22:56:57ned.deilysetmessages: + msg320921
2018-07-02 22:49:33LnL7setmessages: + msg320920
2018-07-02 22:33:45ned.deilysetmessages: + msg320918
2018-07-02 22:16:35LnL7setmessages: + msg320915
2018-07-02 22:04:46ned.deilysetmessages: + msg320914
2018-07-02 21:44:39LnL7setkeywords: + patch
stage: patch review
pull_requests: + pull_request7664
2018-07-02 21:33:23ned.deilysetnosy: + ronaldoussoren, ned.deily
components: + macOS
2018-07-02 21:31:08LnL7create