Unfortunately, there's no way to detect R*Tree support in sqlite3.h. We could run a script that dumps the compile options, and generate a config.h file from that:

$ for L in $(sqlite3 ":memory:" "pragma compile_options"); do echo #define SQLITE_$L >> config.h; done
$ cat config.h
#define SQLITE_COMPILER=clang-12.0.0
#define SQLITE_ENABLE_DBSTAT_VTAB
#define SQLITE_ENABLE_FTS4
#define SQLITE_ENABLE_FTS5
#define SQLITE_ENABLE_GEOPOLY
#define SQLITE_ENABLE_JSON1
#define SQLITE_ENABLE_RTREE
#define SQLITE_ENABLE_STMTVTAB
#define SQLITE_THREADSAFE=1


For Windows and macOS builds, we've already got the SQLite library compile options, so we can just reuse them in CFLAGS when we're building the sqlite3 module.