Thank you again for the suggested PR. Using "xcrun --show-sdk-path" at configure time *is* appealing.  Unfortunately, it does not cover all of the necessary use cases.

One, the --show-sdk-path option is not available on old versions of xcrun, versions we still build with to support older releases of macOS.  Second, some users build Python for macOS with compiler tool chains other than the Apple-supplied ones (for example, current gcc) that do not necessarily support the transparent selection of header and library files location via xcrun and friends.  Third, capturing the selected SDK path at configure time is no guarantee that the same SDK path will be used for extension module builds when setup.py runs.  With the Apple-supplied tools, the actual SDK path used is determined each time the compiler front-end is invoked and depends on the then-current selected values (e.g. the most recent value set by 'xcode-select --switch') and the current value of environment variables (e.g. like DEVELOPER_DIR and SDKROOT). Or a different or non-Apple compiler could now be in use by overriding CC.  In other words, lots of edge cases largely due to the inherent flexibility of Apple's compiler frontend.

The other issue here is the behavior of setup.py in trying to make reasonable default choices for finding header and library files for the extension modules it is building; to do so, it tries to guess what the compiler frontend is going to do and that's the real hack.  Life would be much simpler if Python relied on a modern autotools build setup - although some of the same issues of dynamic SDK locations would still apply but at least they would apply consistently - and not the legacy mixture of build tools we have today.  Unfortunately, it would be a big deal to replace the current build system and, while it would be desirable, that's a very big project.

One other point: while scraping the output of the compiler is hacky, it is a well-known and widely-used technique and is already in use elsewhere in setup.py; this code was adapted from that in add_multiarch_paths().