classification
Title: Optimise SQLite builds on macOS and Windows
Type: enhancement Stage: resolved
Components: macOS, Windows Versions: Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: berker.peksag, erlendaasland, ned.deily, paul.moore, ronaldoussoren, steve.dower, tim.golden, zach.ware
Priority: normal Keywords: patch

Created on 2021-04-14 23:16 by erlendaasland, last changed 2021-10-19 11:02 by erlendaasland. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 25413 merged erlendaasland, 2021-04-14 23:25
PR 25414 merged erlendaasland, 2021-04-14 23:26
Messages (10)
msg391109 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021-04-14 23:16
We should apply some of the recommended "optimisation" compile-time options[1] to the SQLite builds for the macOS and Windows installers. The following options should be safe to apply:

- SQLITE_DEFAULT_MEMSTATUS=0
- SQLITE_LIKE_DOESNT_MATCH_BLOBS
- SQLITE_MAX_EXPR_DEPTH=0
- SQLITE_OMIT_DEPRECATED
- SQLITE_OMIT_AUTOINIT

I'm not sure about SQLITE_DEFAULT_WAL_SYNCHRONOUS=1.

Quoting the SQLite docs:
"So these options do not make a huge difference. But in some design situations, every little bit helps."


[1] https://sqlite.org/compile.html
msg391110 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021-04-14 23:17
BTW, is SQLITE_WITHOUT_ZONEMALLOC still needed for macOS?
msg392200 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021-04-28 14:01
I suggest applying the following safe options for the installers: 

# SQLITE_DEFAULT_MEMSTATUS
Memory status is currently not available in the sqlite3 API, so there's no need for SQLite to keep track of this. If we add such an API (not likely, IMO), we can just remove this compile time option. See also bpo-35056.

# SQLITE_OMIT_DEPRECATED
No deprecated API functions are used by the sqlite3 module.

# SQLITE_OMIT_AUTOINIT
The sqlite3 module explicitly initialises SQLite; we can safely apply this option.


I'll update the PR's.

Berker, do you have any opinion about this?
msg392204 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021-04-28 14:34
As noted in https://github.com/python/cpython/pull/25414#issuecomment-828501078, SQLITE_OMIT_DEPRECATED also leaves out PRAGMA's, which can break applications. I'll update the PR's to leave it out.
msg392206 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2021-04-28 14:38
As long as we don't introduce behavior changes between SQLite versions in mainstream Linux distributions and macOS/Windows (i.e. an application should continue working in Linux, macOS, Windows), it sounds good to me.

Maybe it's worth checking what compile options Debian (and others) are using when packaging SQLite.
msg392233 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021-04-28 16:24
Here's a list of the SQLite recommended compile-time options (only):

- macOS (v11.2.3) (SQLite v3.32.3) defines SQLITE_DEFAULT_WAL_SYNCHRONOUS=1
- Homebrew (SQLite v3.35.4) does not define any of the recommended compile-time options
- Debian Buster (SQLite v3.27.2) defines SQLITE_LIKE_DOESNT_MATCH_BLOBS
- Ubuntu 20.04 (SQLite v3.31.1) defines SQLITE_LIKE_DOESNT_MATCH_BLOBS

AFAICS, adding SQLITE_DEFAULT_MEMSTATUS=0 and SQLITE_OMIT_AUTOINIT is safe, as it only affects the C API that we use, and nothing else.
msg392235 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021-04-28 16:40
Oh, well, SQLITE_DEFAULT_MEMSTATUS=0 does in fact affect PRAGMA soft_heap_limit. Looks like I'm left with only SQLITE_OMIT_AUTOINIT, then :)
msg392395 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021-04-30 08:55
If Berker does not disapprove, I'd like to apply GH-25413 and GH-25414 to the installers, Steve & Ned.
msg392753 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2021-05-03 00:37
New changeset 555cbbe7c10c29ec4358c66c247b8d7e871a5f7b by Erlend Egeberg Aasland in branch 'master':
bpo-43851: Build SQLite with SQLITE_OMIT_AUTOINIT on macOS (GH-25413)
https://github.com/python/cpython/commit/555cbbe7c10c29ec4358c66c247b8d7e871a5f7b
msg404288 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-10-19 11:00
New changeset 8702b667d8e7da7a1888297d84f493f26c580a2d by Erlend Egeberg Aasland in branch 'main':
bpo-43851: Build SQLite with SQLITE_OMIT_AUTOINIT on Windows (GH-25414)
https://github.com/python/cpython/commit/8702b667d8e7da7a1888297d84f493f26c580a2d
History
Date User Action Args
2021-10-19 11:02:40erlendaaslandsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-10-19 11:00:02steve.dowersetmessages: + msg404288
2021-05-03 00:37:42ned.deilysetmessages: + msg392753
2021-04-30 08:55:17erlendaaslandsetmessages: + msg392395
2021-04-28 16:40:34erlendaaslandsetmessages: + msg392235
2021-04-28 16:24:37erlendaaslandsetmessages: + msg392233
2021-04-28 14:38:11berker.peksagsetmessages: + msg392206
2021-04-28 14:34:17erlendaaslandsetmessages: + msg392204
2021-04-28 14:01:45erlendaaslandsetmessages: + msg392200
2021-04-15 06:49:19erlendaaslandsetnosy: + berker.peksag
2021-04-14 23:26:03erlendaaslandsetpull_requests: + pull_request24146
2021-04-14 23:25:33erlendaaslandsetkeywords: + patch
stage: patch review
pull_requests: + pull_request24145
2021-04-14 23:17:02erlendaaslandsetmessages: + msg391110
2021-04-14 23:16:08erlendaaslandcreate