classification
Title: Compiler failure on Mac OS X - sys/random.h
Type: compile error Stage: resolved
Components: Build, macOS Versions: Python 3.7, Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: benjamin.peterson Nosy List: Jim Nasby, Pam.McANulty, benjamin.peterson, christian.heimes, gwk, larry, mark.dickinson, ned.deily, python-dev, rhettinger, ronaldoussoren, vstinner, yan12125
Priority: release blocker Keywords: patch

Created on 2016-12-23 17:37 by Pam.McANulty, last changed 2017-01-19 23:10 by ned.deily. This issue is now closed.

Files
File name Uploaded Description Edit
Issue29057_random_include.patch Pam.McANulty, 2016-12-23 18:32 review
Issue29057_random_include-v2.patch Pam.McANulty, 2016-12-24 15:26 review
random.h Pam.McANulty, 2017-01-01 19:18
issue29057.patch yan12125, 2017-01-01 20:11 review
Pull Requests
URL Status Linked Edit
PR 52 Pam.McANulty, 2016-12-23 17:37
Messages (29)
msg283889 - (view) Author: Pam McA'Nulty (Pam.McANulty) * Date: 2016-12-23 17:37
make failed on Mac OS X including sys/random.h

It looks to be caused by this commit: https://github.com/python/cpython/commit/1958527a2c5dda766b1917ab563622434b3dad0d

Restoring the removed ifdefs solves the problem (see github PR)

Official patch to follow


Failure message.
```
In file included from Python/random.c:16:
/usr/include/sys/random.h:37:32: error: unknown type name 'u_int'
void read_random(void* buffer, u_int numBytes);
                               ^
/usr/include/sys/random.h:38:33: error: unknown type name 'u_int'
void read_frandom(void* buffer, u_int numBytes);
                                ^
/usr/include/sys/random.h:39:33: error: unknown type name 'u_int'
int  write_random(void* buffer, u_int numBytes);
                                ^
3 errors generated.
make: *** [Python/random.o] Error 1
```
msg283890 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-12-23 18:16
Pam, what version of macOS / OS X did this fail on?
msg283892 - (view) Author: Pam McA'Nulty (Pam.McANulty) * Date: 2016-12-23 18:32
Patch for fix
msg283893 - (view) Author: Pam McA'Nulty (Pam.McANulty) * Date: 2016-12-23 18:35
OSX: 10.11.6 (15G1212)
Xcode 8.2.1  Build version 8C1002
msg283896 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-12-23 19:20
Argh!  This is the third go-around on this problem, starting with the quick fix for macOS 10.12 in Issue28676 then on to Issue28932 where Benjamin added a more elaborate configure test when it broke OpenBSD, and now that causes builds on macOS versions earlier than 10.12 but newer than 10.4 or 10.5 to fail. (And, we haven't noticed it because, ATM, the only OS X buildbot active is a 10.4 one and the OpenBSD buildbot is down.)  Benjamin, if you could have a look, please, if I don't get to it first.
msg283900 - (view) Author: Pam McA'Nulty (Pam.McANulty) * Date: 2016-12-23 19:41
I will say that, though my fix seems to work, it feels that the "right" place to put the fix is in ./configure.  

Sadly, I don't speak './configure', though I apparently remember enough C from 17 years ago to get this working.
msg283920 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2016-12-24 05:21
So, if sys/random.h can't be included without error why does the configure check for it work?
msg283954 - (view) Author: Pam McA'Nulty (Pam.McANulty) * Date: 2016-12-24 15:25
Good question.

It looks like there are 3 defines getting checked:
HAVE_SYS_RANDOM_H and either HAVE_GETRANDOM or HAVE_GETENTROPY

The previous issue seemed to be that you could have HAVE_GETRANDOM and/or HAVE_GETENTROPY be set, but still not have a sys/random.h, so a check was added for HAVE_SYS_RANDOM_H.  However, on macs, you can HAVE_SYS_RANDOM_H, but still not have either HAVE_GETRANDOM or HAVE_GETENTROPY.

This _probably_ implies that the sys/random.h that's present isn't conformant enough for the compiler settings and my mac's os and/or xcode.
msg283955 - (view) Author: Pam McA'Nulty (Pam.McANulty) * Date: 2016-12-24 15:26
New patch to make consistent wrt use of "defined()"
msg284151 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2016-12-28 03:48
Right, my question is why does configure define HAVE_SYS_RANDOM_H if including sys/random.h causes an error?
msg284426 - (view) Author: Pam McA'Nulty (Pam.McANulty) * Date: 2017-01-01 19:18
I've dug into things some more.

In my mac environment, sys/random.h is pretty empty and doesn't actually seem to provide anything that Python/random.c uses.  The compile error is a type error is because 'u_int' never gets typedef'd in the rest of the compile includes.

I think the bottom line is that sys/random.h on mac os is neither useful, nor required.

Below is the error again and I've attached my sys/random.h
```
In file included from Python/random.c:16:
/usr/include/sys/random.h:37:32: error: unknown type name 'u_int'
void read_random(void* buffer, u_int numBytes);
                               ^
/usr/include/sys/random.h:38:33: error: unknown type name 'u_int'
void read_frandom(void* buffer, u_int numBytes);
                                ^
/usr/include/sys/random.h:39:33: error: unknown type name 'u_int'
int  write_random(void* buffer, u_int numBytes);
```
msg284427 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2017-01-01 20:11
Here's a how-to for reproducing this bug on Sierra:

1. Install Xcode 7.3.1 (All 7.x should be fine)
2. ./configure --enable-universalsdk=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk
3. make

On Benjamin's question:

> why does configure define HAVE_SYS_RANDOM_H if including sys/random.h causes an error?

autoconf defines several "default includes". For example in line 592 of configure:

#ifdef HAVE_SYS_TYPES_H
# include <sys/types.h>
#endif

And in line 5559:

# On IRIX 5.3, sys/types and inttypes.h are conflicting.
for ac_header in sys/types.h sys/stat.h stdlib.h string.h memory.h strings.h \
		  inttypes.h stdint.h unistd.h
do :
  as_ac_Header=`$as_echo "ac_cv_header_$ac_header" | $as_tr_sh`
ac_fn_c_check_header_compile "$LINENO" "$ac_header" "$as_ac_Header" "$ac_includes_default
"
if eval test \"x\$"$as_ac_Header"\" = x"yes"; then :
  cat >>confdefs.h <<_ACEOF
#define `$as_echo "HAVE_$ac_header" | $as_tr_cpp` 1
_ACEOF

fi

done

The check for sys/random.h starts from line 7758, so sys/types.h is already (accidentally) included when checking it.

So, here's the fix.
msg284431 - (view) Author: Pam McA'Nulty (Pam.McANulty) * Date: 2017-01-01 22:20
I can conform that Chi Hsuan Yen's patch works for me.

I'm concerned that the build includes sys/random.h despite it not actually being needed to build python on mac os x (my worry is possible "shenanigans" by Apple in future versions of their headers) 

But if this patch makes sense to those who actually speak "./configure", I'm agnostic enough.
msg284433 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2017-01-01 22:28
This is currently blocking the release of 3.5.3 rc1.
msg284458 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2017-01-02 04:31
New changeset 9ab75789c554 by Benjamin Peterson in branch '3.5':
only include sys/random.h if it seems like it might have something useful (#29057)
https://hg.python.org/cpython/rev/9ab75789c554

New changeset 74eb71b91112 by Benjamin Peterson in branch '2.7':
only include sys/random.h if it seems like it might have something useful (#29057)
https://hg.python.org/cpython/rev/74eb71b91112

New changeset 5d0cb1fd79ea by Benjamin Peterson in branch '3.6':
merge 3.5 (#29057)
https://hg.python.org/cpython/rev/5d0cb1fd79ea

New changeset bb0d145b43a3 by Benjamin Peterson in branch 'default':
merge 3.6 (#29057)
https://hg.python.org/cpython/rev/bb0d145b43a3
msg284461 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2017-01-02 04:45
Can this be marked closed now?
msg284462 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-01-02 04:54
I just did a quick build test of a few 3.5 configurations with various macOS releases and Benjamin's applied change seems to fix the previous build failures.  No idea about OpenBSD.
msg284467 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2017-01-02 05:58
I'm making an executive decision to not hold up the 3.5.3rc1 release for OpenBSD.  Hopefully the OpenBSD folks can make sure it works for them before 3.5.3 final ships in two weeks.
msg284762 - (view) Author: Jim Nasby (Jim Nasby) * Date: 2017-01-05 17:01
This is still affecting me on 2.7, even with the new changes. 3.7 seems to be OK (I'm getting a different error on 3.7, but it seems unrelated to this.)

OS X 10.11.6.
msg284765 - (view) Author: Jim Nasby (Jim Nasby) * Date: 2017-01-05 19:00
I take it back... turns out I was building with the unsupported macports gcc.
msg284766 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2017-01-05 19:04
Mr. Nasby, as long as you're in a test-reproducing mood, would you mind downloading the source to 3.5.3rc1 and confirming that it builds correctly for you?  I'd appreciate it!  (Not that I don't trust Ned et al, but independent confirmation always helps soothe these jangled nerves.)
msg285071 - (view) Author: George King (gwk) * Date: 2017-01-09 20:42
I am encountering this problem on macOS 10.12.2, with Xcode 8.2.1 (latest).

I have tried building from the following cpython branches today (using the github fork):

2.7: 13a39142c047
In file included from ../../Python/random.c:7:
/usr/include/sys/random.h:37:32: error: unknown type name 'u_int'
void read_random(void* buffer, u_int numBytes);


3.5: 35334a4d41aa
3.6: 6bf563472ccd
Both 3.x branches report a different error:

../../Python/pytime.c:567:11: warning: implicit declaration of function 'clock_gettime' is invalid in C99 [-Wimplicit-function-declaration]
    err = clock_gettime(CLOCK_REALTIME, &ts);
          ^
../../Python/pytime.c:567:25: error: use of undeclared identifier 'CLOCK_REALTIME'
    err = clock_gettime(CLOCK_REALTIME, &ts);
                        ^
../../Python/pytime.c:581:13: warning: implicit declaration of function 'clock_getres' is invalid in C99 [-Wimplicit-function-declaration]
        if (clock_getres(CLOCK_REALTIME, &res) == 0)
            ^
../../Python/pytime.c:581:26: error: use of undeclared identifier 'CLOCK_REALTIME'
        if (clock_getres(CLOCK_REALTIME, &res) == 0)
                         ^
msg285072 - (view) Author: George King (gwk) * Date: 2017-01-09 20:43
(I meant the github mirror: github.com/python/cpython)
msg285076 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-01-09 21:53
George, did you do totally clean builds, including rerunning ./configure?  If so, what ./configure and make commands did you use?  Using the current hg repo, I don't see the failures you are seeing.
msg285235 - (view) Author: George King (gwk) * Date: 2017-01-11 16:34
Still seeing this problem. Here was my exact process:

$ git clone git@github.com:python/cpython.git
$ cd cpython
$ git checkout 2.7
$ mkdir build
$ cd build
$ ../configure
$ make

In file included from ../Python/random.c:7:
/usr/include/sys/random.h:37:32: error: unknown type name 'u_int'
msg285236 - (view) Author: George King (gwk) * Date: 2017-01-11 16:36
This is using the latest apple toolchain on latest macOS 10.12.2:

$ gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 8.0.0 (clang-800.0.42.1)
Target: x86_64-apple-darwin16.3.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

$ make --version
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i386-apple-darwin11.3.0
msg285265 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-01-11 19:31
Sorry, George, but I'm unable to reproduce the failure you observe following your recipe.  The only thing I can suggest off the top of my head is that you might have out-of-date system header files installed in /usr/include.  Those files are installed with:

sudo xcode-select --install

They don't get installed or updated by just installing Xcode itself.  See if that helps.
msg285274 - (view) Author: George King (gwk) * Date: 2017-01-11 21:40
I reinstalled the command line tools by downloading from developer.apple.com/download/more and the problem went away. No idea how they broke; I had previously installed the same version. In any case, sorry for the noise.
msg285541 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2017-01-16 08:04
Releasing 3.5.3 even though technically this is an open release blocker.  IIUC the fix is checked in, and fixed the issue for OS X.  We don't know whether or not it is also fixed on OpenBSD, because we don't know anybody running OpenBSD, and nobody contacted us regarding OpenBSD during the 3.5.3 RC period.  Anyway I'm declaring victory and moving on.  Maybe we can mark this bug closed?
History
Date User Action Args
2017-01-19 23:12:34ned.deilylinkissue27596 superseder
2017-01-19 23:10:31ned.deilysetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2017-01-16 08:04:35larrysetmessages: + msg285541
2017-01-11 21:40:03gwksetmessages: + msg285274
2017-01-11 19:31:29ned.deilysetmessages: + msg285265
2017-01-11 16:36:52gwksetmessages: + msg285236
2017-01-11 16:34:51gwksetmessages: + msg285235
2017-01-09 21:53:13ned.deilysetmessages: + msg285076
2017-01-09 20:43:41gwksetmessages: + msg285072
2017-01-09 20:42:37gwksetnosy: + gwk
messages: + msg285071
2017-01-05 19:04:41larrysetmessages: + msg284766
2017-01-05 19:00:30Jim Nasbysetmessages: + msg284765
2017-01-05 17:01:29Jim Nasbysetnosy: + Jim Nasby
messages: + msg284762
2017-01-02 05:58:13larrysetmessages: + msg284467
2017-01-02 04:54:11ned.deilysetmessages: + msg284462
2017-01-02 04:45:52larrysetmessages: + msg284461
2017-01-02 04:31:05python-devsetnosy: + python-dev
messages: + msg284458
2017-01-01 22:28:28larrysetmessages: + msg284433
2017-01-01 22:20:02Pam.McANultysetmessages: + msg284431
2017-01-01 20:11:21yan12125setfiles: + issue29057.patch
2017-01-01 20:11:07yan12125setmessages: + msg284427
2017-01-01 19:18:19Pam.McANultysetfiles: + random.h

messages: + msg284426
2017-01-01 10:35:30yan12125setnosy: + yan12125
2016-12-28 03:48:58benjamin.petersonsetmessages: + msg284151
2016-12-24 15:26:59Pam.McANultysetfiles: + Issue29057_random_include-v2.patch

messages: + msg283955
2016-12-24 15:25:44Pam.McANultysetmessages: + msg283954
2016-12-24 05:21:37benjamin.petersonsetmessages: + msg283920
2016-12-24 00:01:43christian.heimessetnosy: + christian.heimes
2016-12-23 23:30:07vstinnersetnosy: + vstinner
2016-12-23 19:41:34Pam.McANultysetmessages: + msg283900
2016-12-23 19:20:04ned.deilysetpriority: normal -> release blocker

assignee: benjamin.peterson
versions: + Python 2.7, Python 3.5, Python 3.6
nosy: + larry

messages: + msg283896
stage: patch review
2016-12-23 18:35:43Pam.McANultysetmessages: + msg283893
2016-12-23 18:32:46Pam.McANultysetfiles: + Issue29057_random_include.patch
keywords: + patch
messages: + msg283892
2016-12-23 18:16:02ned.deilysetmessages: + msg283890
2016-12-23 17:43:02serhiy.storchakasetnosy: + rhettinger, ronaldoussoren, mark.dickinson, benjamin.peterson, ned.deily
components: + macOS
2016-12-23 17:37:29Pam.McANultycreate