Motivated by deprecation of distutils, I like to move more logic and checks from setup.py into configure.ac. Eventually I like to get rid of setup.py. The file contains a bunch of complicated checks and macOS-specific adjustments that I cannot verify on Linux.


1) socketmodule setup defines __APPLE_USE_RFC_3542 https://github.com/python/cpython/blob/v3.11.0a2/setup.py#L1229 Can we move the define into the __APPLE__ block of socketmodule.c?


2) -Wl,-search_paths_first linker arg, e.g. https://github.com/python/cpython/blob/v3.11.0a2/setup.py#L1616  Would it be safe to make the option default for all core extensions on $ac_sys_system = Darwin? We could add it to PY_CORE_LDFLAGS or add a new Makefile variable for core extensions.


3) detect_dbm_gdbm has about 200 lines of complicated code to detect old to ancient versions of Berkeley DB (libdb), https://github.com/python/cpython/blob/v3.11.0a2/setup.py#L1233 . Can I remove support for libdb-3 / libdb-4 and only support libdb-5 in standard locations? libdb-5.3 has been around for over a decade.

Note: libdb, gdbm, ndbm, and gdbm-compat don't provide pkg-config .pc files. We cannot use pkg-config to detect them.


4) sqlite's setup https://github.com/python/cpython/blob/v3.11.0a2/setup.py#L1531 does extra work to search for header and library files in non-standard locations. Can we replace the code with pkg-config checks in configure.ac with a fallback to AC_CHECK_LIB() and AC_CHECK_HEADERS()?


5) zlib's setup code https://github.com/python/cpython/blob/v3.11.0a2/setup.py#L1661 has a check for a CVE from 2002 (!). I think we can safely assume that everybody has upgraded to a fixed version. The check is mixed with macOS specific code.

It seems to me that some (most?) of the macOS-specific workarounds in setup.py were added to make it easy to build with the system-provided copies of the third-party libraries, like zlib and sqlite3 and openssl. At least one of the workarounds, search_paths_first, became the default ld behavior since around macOS 10.6 (Xcode 4). So when building on more recent macOS systems, either the workarounds aren't necessary any more or the third-party library is no longer provided (like openssl) or is otherwise too old (like Tk) so that you can no longer rely on building with just using system-supplied libraries to have a useful Python on macOS. And, if you are building on very old systems (because of hardware requirements or whatever), in most cases, the versions of the system-supplied libraries are so old that you shouldn't be relying on them anyway. So I think that, for 3.11, we could take a stand and say that we no longer support building with specific system-supplied third-party libraries on macOS systems older than, say, macOS 10.9; IOW, you will need to supply your own copies of those libs on older systems. That's essentially the approach the MacPorts project has taken for years; they still support providing current Pythons on old versions of macOS but they also supply current version of the third-party libs, too.

And Christian's work here for 3.11 to move away from setup.py and use pkg-config to remove the guesswork on header and lib locations should make life easier for everyone. There is a small issue here in that macOS does not supply a built-in pkg-config but there are implementations available, if you are unable or unwilling to use open source distributors like MacPorts or Homebrew, including an actively-maintained permissive-licensed version with no build dependencies that seems to work on even macOS 10.6 (I didn't try anything older than that!): https://github.com/pkgconf/pkgconf.

Ronald, what do you think?  If this sounds reasonable, we could draw up a list of libs and document it somewhere in the 3.11 code.

Thanks for the detailed explanation, Ned. Much appreciated! For non-Mac users: macOS 10.6 was released in 2009 and 10.9 in 2013. IMHO it is reasonable to ask people to provide their own copies of libraries on older system. We also require OpenSSL 1.1.1, which is not available on older systems like CentOS 7 (released 2014).

Regarding pkg-config, I like to tidy up setup.py first to get a better understanding what kind of builds and systems I need to support. Thanks for pointing out https://github.com/pkgconf/pkgconf! I keep it in mind.

For reference: https://opensource.apple.com/source/ld64/ld64-242/doc/man/man1/ld.1.auto.html

> search_paths_first
> This is now the default (in Xcode4 tools).  When processing -lx the linker now searches each directory
> in its library search paths for `libx.dylib' then `libx.a' before the moving on to the next path
> in the library search path.

According to Wikipedia, Xcode4 was released in 2011.

New changeset 24af9a40a8f85af813ea89998aa4e931fcc78cd9 by Christian Heimes in branch 'main':
bpo-45743: Move __APPLE_USE_RFC_3542 into socketmodule.c (GH-29456)
https://github.com/python/cpython/commit/24af9a40a8f85af813ea89998aa4e931fcc78cd9

1) __APPLE_USE_RFC_3542 should have been in socketmodule.c from the start, not sure why it was added in setup.py.

2) as you and Ned noticed -search_paths_first has been the default for a long while, we can just drop it and anyone building on ancient systems can add the flag to CFLAGs/LDFLAGS as needed.  IIRC we added the flag to be able to build with local copies of libraries also shipped in the OS, in particular when using static libraries for those local copies.

3) I don't know about detect_dbm_gdbm, and don't use those libraries myself. It would be nice to be able to open system files created using dbopen etc, but on the other hand there is at least one bug report complaining about data corruption when using one o the dbm extensions linked with the system version of the library.

4/5) Fine by me, although I'm slightly worried about using pkg-config because the system doesn't ship that tool. 

Something you don't mention is the logic dealing with SDK roots. I haven't checked yet if similar logic would be necessary in configure. With some luck it isn't, but that depends on what's supported by autoconf and the particular probes we want to use.

@Ned: Not being able to use system versions of libraries is a bit annoying, but that's something you already have to do due to openssl.  Maybe we should also try to clean up and refactor the build-installer.py script to do have a way to build/install those 3th-party dependencies.

> 1) __APPLE_USE_RFC_3542 should have been in socketmodule.c from the start, not sure why it was added in setup.py.

FTR, it was added by me in bpo-35569, GH-19526. I moved the check to setup.py because Ned made me do it :P https://github.com/python/cpython/pull/19526#discussion_r415637170

>> 1) __APPLE_USE_RFC_3542 should have been in socketmodule.c from the start, not sure why it was added in setup.py.

> FTR, it was added by me in bpo-35569, GH-19526. I moved the check to setup.py because Ned made me do it :P

Yeah, well, it seemed like a good thing at the time :)

> Something you don't mention is the logic dealing with SDK roots. I haven't checked yet if similar logic would be necessary in configure. With some luck it isn't, but that depends on what's supported by autoconf and the particular probes we want to use.

I haven't looked closely at it in this context but I believe most, if not all, of the SDK root shenanigans in setup.py shouldn't be necessary in autoconf tests where the Apple compiler driver handles the SDK details and the few other cases, like for when there was only a 32-bit version of Tk for macOS, are no longer relevant. In any case, that's something for us to follow up on once Christian's changes are merged and stabilized on other platforms.

> @Ned: Not being able to use system versions of libraries is a bit annoying, but that's something you already have to do due to openssl.  Maybe we should also try to clean up and refactor the build-installer.py script to do have a way to build/install those 3th-party dependencies.

Yes. I want to pull the third-party lib building out of build-installer.py into a simple separate build step, probably just using make, that can produce the minimum sets of third-party libs for use by both installer builds and by developers so that there is no dependency on other third-party distributors like Homebrew or MacPorts. With a little bit of care, it wouldn't have to be limited to just macOS builds.

New changeset 8fefaad242f45b3bd97e000a00f2aac16d935315 by Christian Heimes in branch 'main':
bpo-45743: -Wl,-search_paths_first is no longer needed (GH-29464)
https://github.com/python/cpython/commit/8fefaad242f45b3bd97e000a00f2aac16d935315

New changeset 6a1cc8bf8a0d88af9c7891c6577508ae9f70e3ef by Christian Heimes in branch 'main':
bpo-45743: Remove workaround for zlib CVE from 2002 (GH-29457)
https://github.com/python/cpython/commit/6a1cc8bf8a0d88af9c7891c6577508ae9f70e3ef