building '_sqlite3' extension
creating build/temp.macosx-11.5-universal2-3.11/Users/jett/cpython/Modules/_sqlite
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -fno-semantic-interposition -flto -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -fprofile-instr-generate -I./Include/internal -IModules/_sqlite -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I./Include -I. -I/usr/local/include -I/Users/jett/cpython/Include -I/Users/jett/cpython -c /Users/jett/cpython/Modules/_sqlite/connection.c -o build/temp.macosx-11.5-universal2-3.11/Users/jett/cpython/Modules/_sqlite/connection.o
/Users/jett/cpython/Modules/_sqlite/connection.c:1179:10: error: implicit declaration of function 'sqlite3_enable_load_extension' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
    rc = sqlite3_enable_load_extension(self->db, onoff);
         ^
/Users/jett/cpython/Modules/_sqlite/connection.c:1215:10: error: implicit declaration of function 'sqlite3_load_extension' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
    rc = sqlite3_load_extension(self->db, extension_name, 0, &errmsg);
         ^
/Users/jett/cpython/Modules/_sqlite/connection.c:1215:10: note: did you mean 'sqlite3_auto_extension'?
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sqlite3.h:6551:16: note: 'sqlite3_auto_extension' declared here
SQLITE_API int sqlite3_auto_extension(void(*xEntryPoint)(void));
               ^
2 errors generated.


This is the error message from clang on macOS 11.5.1
configure options: ./configure --prefix=/Users/jett/python311 --enable-optimizations --enable-loadable-sqlite-extensions --enable-ipv6 --enable-big-digits=30 --with-lto=full --with-experimental-isolated-subinterpreters --with-static-libpython --enable-universalsdk --with-universal-archs=universal2

IIRC, macOS deprecated this functionality some releases ago. Seems like they've now removed it completely.

I guess we could wrap this functionality with some preprocessor conditionals, and/or adjust the configure script to not accept --enable-loadable-sqlite-extensions for macOS 11.5.1 (11.5.x?).

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_gdbm                 ossaudiodev           spwd               
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  pwd                   time               

running build_scripts
creating build/scripts-3.11
copying and adjusting /Users/jett/cpython/Tools/scripts/pydoc3 -> build/scripts-3.11
copying and adjusting /Users/jett/cpython/Tools/scripts/idle3 -> build/scripts-3.11
copying and adjusting /Users/jett/cpython/Tools/scripts/2to3 -> build/scripts-3.11
changing mode of build/scripts-3.11/pydoc3 from 644 to 755
changing mode of build/scripts-3.11/idle3 from 644 to 755
changing mode of build/scripts-3.11/2to3 from 644 to 755
renaming build/scripts-3.11/pydoc3 to build/scripts-3.11/pydoc3.11
renaming build/scripts-3.11/idle3 to build/scripts-3.11/idle3.11
renaming build/scripts-3.11/2to3 to build/scripts-3.11/2to3-3.11
touch profile-gen-stamp
# Next, run the profile task to generate the profile information.
/Library/Developer/CommandLineTools/usr/bin/make run_profile_task
LLVM_PROFILE_FILE="code-%p.profclangr"  ./python.exe -m test --pgo --timeout=1200 || true
0:00:00 load avg: 7.14 Run tests sequentially (timeout: 20 min)
0:00:00 load avg: 7.14 [ 1/44] test_array
0:00:00 load avg: 7.14 [ 2/44] test_base64
0:00:00 load avg: 7.14 [ 3/44] test_binascii -- test_base64 failed (env changed)
0:00:00 load avg: 7.14 [ 4/44] test_binop
0:00:00 load avg: 7.14 [ 5/44] test_bisect
0:00:00 load avg: 7.14 [ 6/44] test_bytes
0:00:02 load avg: 6.73 [ 7/44] test_bz2 -- test_bytes failed (env changed)
0:00:02 load avg: 6.73 [ 8/44] test_cmath
0:00:02 load avg: 6.73 [ 9/44] test_codecs
0:00:03 load avg: 6.73 [10/44] test_collections
0:00:03 load avg: 6.73 [11/44] test_complex
0:00:04 load avg: 6.73 [12/44] test_dataclasses
0:00:04 load avg: 6.73 [13/44] test_datetime
0:00:07 load avg: 6.27 [14/44] test_decimal
-------------------------------------------------------------------------------------------------- NOTICE --------------------------------------------------------------------------------------------------
test_decimal may generate "malloc can't allocate region"
warnings on macOS systems. This behavior is known. Do not
report a bug unless tests are also failing. See bpo-40928.
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
python.exe(70362,0x102aafd40) malloc: can't allocate region
:*** mach_vm_map(size=842105263157903360, flags: 100) failed (error code=3)
python.exe(70362,0x102aafd40) malloc: *** set a breakpoint in malloc_error_break to debug
python.exe(70362,0x102aafd40) malloc: can't allocate region
:*** mach_vm_map(size=842105263157903360, flags: 100) failed (error code=3)
python.exe(70362,0x102aafd40) malloc: *** set a breakpoint in malloc_error_break to debug
python.exe(70362,0x102aafd40) malloc: can't allocate region
:*** mach_vm_map(size=421052631578951680, flags: 100) failed (error code=3)
python.exe(70362,0x102aafd40) malloc: *** set a breakpoint in malloc_error_break to debug
python.exe(70362,0x102aafd40) malloc: can't allocate region
:*** mach_vm_map(size=421052631578951680, flags: 100) failed (error code=3)
python.exe(70362,0x102aafd40) malloc: *** set a breakpoint in malloc_error_break to debug
0:00:09 load avg: 6.27 [15/44] test_difflib
0:00:10 load avg: 6.27 [16/44] test_embed
test test_embed failed
0:00:12 load avg: 5.85 [17/44] test_float -- test_embed failed (33 failures)
0:00:12 load avg: 5.85 [18/44] test_fstring
0:00:13 load avg: 5.85 [19/44] test_functools
0:00:13 load avg: 5.85 [20/44] test_generators
0:00:13 load avg: 5.85 [21/44] test_hashlib
0:00:14 load avg: 5.85 [22/44] test_heapq
0:00:14 load avg: 5.85 [23/44] test_int
0:00:14 load avg: 5.85 [24/44] test_itertools
0:00:16 load avg: 5.85 [25/44] test_json
0:00:17 load avg: 5.54 [26/44] test_long -- test_json failed (env changed)
0:00:19 load avg: 5.54 [27/44] test_lzma
0:00:19 load avg: 5.54 [28/44] test_math
0:00:21 load avg: 5.73 [29/44] test_memoryview
0:00:21 load avg: 5.73 [30/44] test_operator
0:00:22 load avg: 5.73 [31/44] test_ordered_dict
0:00:22 load avg: 5.73 [32/44] test_patma
0:00:22 load avg: 5.73 [33/44] test_pickle
0:00:25 load avg: 5.73 [34/44] test_pprint
0:00:25 load avg: 5.73 [35/44] test_re
0:00:26 load avg: 5.35 [36/44] test_set
0:00:27 load avg: 5.35 [37/44] test_sqlite
Fatal Python error: drop_gil: drop_gil: GIL is not locked
Python runtime state: initialized

Current thread 0x0000000102aafd40 (most recent call first):
  File "/Users/jett/cpython/Lib/sqlite3/test/userfunctions.py", line 524 in test_aggr_exception_in_finalize
  File "/Users/jett/cpython/Lib/sqlite3/test/userfunctions.py", line 46 in wrapper
  File "/Users/jett/cpython/Lib/unittest/case.py", line 549 in _callTestMethod
  File "/Users/jett/cpython/Lib/unittest/case.py", line 593 in run
  File "/Users/jett/cpython/Lib/unittest/case.py", line 652 in __call__
  File "/Users/jett/cpython/Lib/unittest/suite.py", line 122 in run
  File "/Users/jett/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/Users/jett/cpython/Lib/unittest/suite.py", line 122 in run
  File "/Users/jett/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/Users/jett/cpython/Lib/unittest/suite.py", line 122 in run
  File "/Users/jett/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/Users/jett/cpython/Lib/unittest/suite.py", line 122 in run
  File "/Users/jett/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/Users/jett/cpython/Lib/test/support/testresult.py", line 169 in run
  File "/Users/jett/cpython/Lib/test/support/__init__.py", line 998 in _run_suite
  File "/Users/jett/cpython/Lib/test/support/__init__.py", line 1123 in run_unittest
  File "/Users/jett/cpython/Lib/test/libregrtest/runtest.py", line 261 in _test_module
  File "/Users/jett/cpython/Lib/test/libregrtest/runtest.py", line 297 in _runtest_inner2
  File "/Users/jett/cpython/Lib/test/libregrtest/runtest.py", line 335 in _runtest_inner
  File "/Users/jett/cpython/Lib/test/libregrtest/runtest.py", line 215 in _runtest
  File "/Users/jett/cpython/Lib/test/libregrtest/runtest.py", line 245 in runtest
  File "/Users/jett/cpython/Lib/test/libregrtest/main.py", line 437 in run_tests_sequential
  File "/Users/jett/cpython/Lib/test/libregrtest/main.py", line 535 in run_tests
  File "/Users/jett/cpython/Lib/test/libregrtest/main.py", line 708 in _main
  File "/Users/jett/cpython/Lib/test/libregrtest/main.py", line 655 in main
  File "/Users/jett/cpython/Lib/test/libregrtest/main.py", line 733 in main
  File "/Users/jett/cpython/Lib/test/__main__.py", line 2 in <module>
  File "/Users/jett/cpython/Lib/runpy.py", line 86 in _run_code
  File "/Users/jett/cpython/Lib/runpy.py", line 196 in _run_module_as_main

Extension modules: _testcapi, _testinternalcapi, _scproxy, _testbuffer (total: 4)
/bin/sh: line 1: 70362 Abort trap: 6           LLVM_PROFILE_FILE="code-%p.profclangr" ./python.exe -m test --pgo --timeout=1200
/Library/Developer/CommandLineTools/usr/bin/make build_all_merge_profile
/usr/local/bin/llvm-profdata merge -output=code.profclangd *.profclangr
warning: code-70398.profclangr: unsupported instrumentation profile format version
warning: code-70410.profclangr: unsupported instrumentation profile format version
warning: code-70418.profclangr: unsupported instrumentation profile format version
warning: code-70427.profclangr: unsupported instrumentation profile format version
warning: code-70438.profclangr: unsupported instrumentation profile format version
warning: code-70446.profclangr: unsupported instrumentation profile format version
warning: code-70394.profclangr: unsupported instrumentation profile format version
warning: code-70402.profclangr: unsupported instrumentation profile format version
warning: code-70414.profclangr: unsupported instrumentation profile format version
warning: code-70432.profclangr: unsupported instrumentation profile format version
warning: code-70422.profclangr: unsupported instrumentation profile format version
warning: code-70442.profclangr: unsupported instrumentation profile format version
warning: code-70392.profclangr: unsupported instrumentation profile format version
warning: code-70400.profclangr: unsupported instrumentation profile format version
warning: code-70412.profclangr: unsupported instrumentation profile format version
warning: code-70420.profclangr: unsupported instrumentation profile format version
warning: code-70430.profclangr: unsupported instrumentation profile format version
warning: code-70440.profclangr: unsupported instrumentation profile format version
warning: code-70396.profclangr: unsupported instrumentation profile format version
warning: code-70405.profclangr: unsupported instrumentation profile format version
warning: code-70416.profclangr: unsupported instrumentation profile format version
warning: code-70424.profclangr: unsupported instrumentation profile format version
warning: code-70434.profclangr: unsupported instrumentation profile format version
warning: code-70444.profclangr: unsupported instrumentation profile format version
warning: code-70391.profclangr: unsupported instrumentation profile format version
warning: code-70399.profclangr: unsupported instrumentation profile format version
warning: code-70419.profclangr: unsupported instrumentation profile format version
warning: code-70428.profclangr: unsupported instrumentation profile format version
warning: code-70411.profclangr: unsupported instrumentation profile format version
warning: code-70439.profclangr: unsupported instrumentation profile format version
warning: code-70395.profclangr: unsupported instrumentation profile format version
warning: code-70403.profclangr: unsupported instrumentation profile format version
warning: code-70415.profclangr: unsupported instrumentation profile format version
warning: code-70423.profclangr: unsupported instrumentation profile format version
warning: code-70433.profclangr: unsupported instrumentation profile format version
warning: code-70443.profclangr: unsupported instrumentation profile format version
warning: code-70393.profclangr: unsupported instrumentation profile format version
warning: code-70401.profclangr: unsupported instrumentation profile format version
warning: code-70413.profclangr: unsupported instrumentation profile format version
warning: code-70421.profclangr: unsupported instrumentation profile format version
warning: code-70431.profclangr: unsupported instrumentation profile format version
warning: code-70441.profclangr: unsupported instrumentation profile format version
warning: code-70397.profclangr: unsupported instrumentation profile format version
warning: code-70406.profclangr: unsupported instrumentation profile format version
warning: code-70417.profclangr: unsupported instrumentation profile format version
warning: code-70425.profclangr: unsupported instrumentation profile format version
warning: code-70435.profclangr: unsupported instrumentation profile format version
warning: code-70445.profclangr: unsupported instrumentation profile format version
error: no profile can be merged
make[1]: *** [build_all_merge_profile] Error 1
make: *** [profile-run-stamp] Error 2


build succeeded, tests are still failing without that arg. see #44998

I set fixed for this since this one issue is done, but #44998 is still a prob.

> I guess we could wrap this functionality with some preprocessor conditionals, and/or adjust the configure script to not accept --enable-loadable-sqlite-extensions for macOS 11.5.1 (11.5.x?).

FTR, this is only a problem if you link with the Apple-supplied system libsqlite3. So any change would need need to take into account the possibility of using your own or other third-party copy of libsqlite3, for example, like the python.org macOS binary installers do.

> FTR, this is only a problem if you link with the Apple-supplied system libsqlite3 [...]

Sure, but that should be relatively trivial to detect. AFAIK, the SQLite command line shell seems to be located at the same path every release, so I would guess that also applies for the library.

Alternatively, we could dump the compile options and grep for SQLITE_OMIT_LOAD_EXTENSION, but that implies locating the SQLite command line shell that corresponds to the library that we're using. Probably trickier, but would definitely be useful if/when we add support for the SQLite serialize API, RTree callbacks, etc.

...or we can just leave it as it. Seems like these reports only pops up once in a while.

Note that sqlite3_enable_load_extension was unavailable way before macOS 11.5.1, the symbol is not present on 10.15 (I used 'nm /usr/lib/libsqlite3.dylib' to check for symbol availability).

Just checking for MACOS and a system include directory should do the trick for giving a nicer error message, something like:

diff --git a/setup.py b/setup.py
index bc2ea16089..ae5d827339 100644
--- a/setup.py
+++ b/setup.py
@@ -1601,6 +1601,9 @@ def detect_sqlite(self):
             if '--enable-loadable-sqlite-extensions' not in sysconfig.get_config_var("CONFIG_ARGS"):
                 sqlite_defines.append(("SQLITE_OMIT_LOAD_EXTENSION", "1"))
 
+            elif MACOS and sqlite_incdir == os.path.join(MACOS_SDK_ROOT, 'usr/include'): 
+                raise DistutilsError("System version of SQLite3 does not support loadable extensions") 
+
             if MACOS:
                 # In every directory on the search path search for a dynamic
                 # library and then a static library, instead of first looking

With this patch I get a nice error message when trying to build with loadable sqlite extensions:

make                                                                                                                                                                      (main)cpython
 CC='gcc' LDSHARED='gcc -bundle -undefined dynamic_lookup    ' OPT='-DNDEBUG -g -fwrapv -O3 -Wall' 	_TCLTK_INCLUDES='' _TCLTK_LIBS='' 	./python.exe -E ../setup.py  build
running build
running build_ext
error: System version of SQLite3 does not support loadable extensions
make: *** [sharedmods] Error 1


My clone of the CPython repository is a bit of a mess at the moment, otherwise I'd have created a PR for this. Feel free to run with this.

Thanks, Ronald, that's a nice improvement. I'll create a PR for it.

New changeset 2ec9428e3516b6e7c6583687aa3b39fb96925577 by Erlend Egeberg Aasland in branch 'main':
bpo-44997: macOS does not support loadable SQLite extensions (GH-27979)
https://github.com/python/cpython/commit/2ec9428e3516b6e7c6583687aa3b39fb96925577

New changeset 9c74678d53847f61df255435a0e5a6e3e1d657f5 by Miss Islington (bot) in branch '3.8':
bpo-44997: macOS does not support loadable SQLite extensions (GH-27979) (GH-27996)
https://github.com/python/cpython/commit/9c74678d53847f61df255435a0e5a6e3e1d657f5

New changeset baa8d48230e3ab8a504f39ee800f8852897cb3be by Miss Islington (bot) in branch '3.10':
bpo-44997: macOS does not support loadable SQLite extensions (GH-27979)
https://github.com/python/cpython/commit/baa8d48230e3ab8a504f39ee800f8852897cb3be

New changeset 9500dd5144cb911cf4e6fb027bb25b675f0e3643 by Miss Islington (bot) in branch '3.9':
bpo-44997: macOS does not support loadable SQLite extensions (GH-27979) (GH-27995)
https://github.com/python/cpython/commit/9500dd5144cb911cf4e6fb027bb25b675f0e3643

Thanks for the fix, Ronald and Erlend! ✨ 🍰 ✨