Compiler and linker flags for _decimal and internal libmpdec are currently handled by a mix of configure checks and if/else chains in setup.py. The split makes it harder to build _decimal correctly from Modules/Setup. The Modules/Setup file also does not handle --with-system-mpdec.

I have a working PR that moves all logic into configure.ac. The new system:

* sets LIBMPDEC_CFLAGS and LIBMPDEC_LDFLAGS based on --with-system-libmpdec value.

* detects libmpdec_machine by looking at ac_sys_system, MACOSX_DEFAULT_ARCH, ac_cv_sizeof_size_t, ac_cv_gcc_asm_for_x64, ac_cv_type___uint128_t, and ac_cv_gcc_asm_for_x87.

* sets libmpdec compiler args based on libmpdec_machine, have_ipa_pure_const_bug, and have_glibc_memmove_bug.

* if --with-system-libmpdec is not given, then our Makefile compiles libmpdec objects and puts them into a libmpdec.a archive.

* finally it either links _decimal with our libmpdec.a or with system's libmpdec shared library.

I went for libmpdec.a because it makes the logic for the internal path look similar to the logic with linking with an external shared library.

Modules/Setup

PS: I had to add an explicit make rule for each object file. "%.o: %c" templates are not portable.

New changeset 0486570f7b2b5a75812e5a01a8dca58bfadc2437 by Christian Heimes in branch 'main':
bpo-45798: Move _decimal build setup into configure (GH-29541)
https://github.com/python/cpython/commit/0486570f7b2b5a75812e5a01a8dca58bfadc2437

Thanks for the quick review, Mark!

I tested the --with-system-libmpdec successfully on my system. Most vendors are using the internal copy of libmpdec any way. AFAIK only Debian-based systems use their own system libmpdec.

$ ./configure -C --with-system-libmpdec
$ make
...
building '_decimal' extension
gcc -pthread -fPIC -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I./Include -I. -I/usr/local/include -I/home/heimes/dev/python/cpython/Include -I/home/heimes/dev/python/cpython -c /home/heimes/dev/python/cpython/Modules/_decimal/_decimal.c -o build/temp.linux-x86_64-3.11/home/heimes/dev/python/cpython/Modules/_decimal/_decimal.o -DCONFIG_64=1 -DASM=1
gcc -pthread -shared build/temp.linux-x86_64-3.11/home/heimes/dev/python/cpython/Modules/_decimal/_decimal.o -L/usr/local/lib -o build/lib.linux-x86_64-3.11/_decimal.cpython-311-x86_64-linux-gnu.so -lmpdec
...
$ ldd build/lib.linux-x86_64-3.11/_decimal.cpython-311-x86_64-linux-gnu.so 
        linux-vdso.so.1 (0x00007ffde21e1000)
        libmpdec.so.3 => /lib64/libmpdec.so.3 (0x00007f4f3b4cf000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f4f3b4ae000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f4f3b2df000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f4f3b19b000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f4f3b554000)

It looks like this change broke macOS universal2 builds of libmpdec. I'm looking at it now but wanted to flag it as a "release blocker" for tagging 3.11.0a3.

New changeset ddbab69b6d44085564a9b5022b96b002a52b2f2b by Ned Deily in branch 'main':
bpo-45798: Let libmpdec decide which archs to build on macOS as done previously. (GH-29949)
https://github.com/python/cpython/commit/ddbab69b6d44085564a9b5022b96b002a52b2f2b

The issue here turned out to be that the architecture selection code for libmpdec builds on macOS had been scrambled a bit on the move from setup.py to configure. To support universal builds correctly, libmpdec has code to decide itself which arch(s) to build with on macOS. PR 29949 restores that behavior and macOS universal2 builds (builds which have both native arm64 and x86_64 binaries in each executable or library file) now work again for 3.11.