classification
Title: Update SQLite to 3.31.1 in Windows and macOS installer builds
Type: Stage: resolved
Components: Build, macOS, Windows Versions: Python 3.9, Python 3.8, Python 3.7, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Big Stone, benjamin.peterson, erlendaasland, miss-islington, ned.deily, paul.moore, ronaldoussoren, steve.dower, tim.golden, zach.ware
Priority: normal Keywords: patch

Created on 2019-10-05 22:12 by Big Stone, last changed 2020-03-04 02:45 by benjamin.peterson. This issue is now closed.

Files
File name Uploaded Description Edit
0001-bpo-38380-Update-macOS-installer-to-use-SQLite-3.30..patch erlendaasland, 2019-10-09 15:24 bpo-38380: Update macOS installer to use SQLite 3.30.0
0002-bpo-38380-Update-Windows-builds-to-use-SQLite-3.30.0.patch erlendaasland, 2019-10-09 15:25 bpo-38380: Update Windows builds to use SQLite 3.30.0
0001-bpo-38380-Update-macOS-installer-to-use-SQLite-3.30.1.patch erlendaasland, 2020-02-27 14:56
0002-bpo-38380-Update-Windows-builds-to-use-SQLite-3.30.1.patch erlendaasland, 2020-02-27 14:56
Pull Requests
URL Status Linked Edit
PR 18678 merged erlendaasland, 2020-02-27 15:00
PR 18740 merged miss-islington, 2020-03-02 12:25
PR 18741 merged miss-islington, 2020-03-02 12:25
Messages (30)
msg354023 - (view) Author: Big Stone (Big Stone) Date: 2019-10-05 22:12
there is a security fix in sqlite-3.30

https://nvd.nist.gov/vuln/detail/CVE-2019-16168#VulnChangeHistorySection

https://www.sqlite.org/releaselog/3_30_0.html
msg354025 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-10-05 22:35
I verified it is exploitable via the sqlite3 module by adapting the test case from the SQLite ticket (https://www.sqlite.org/src/info/e4598ecbdd18bd82).  But since it requires the exploiter to be able to specify raw SQL statements, it doesn't sound like it needs to be treated as a Python security issue.  We should plan to to update to the latest SQLite but it doesn't need to be a release blocker.
msg354280 - (view) Author: Erlend Egeberg Aasland (erlendaasland) * Date: 2019-10-09 15:22
I've prepared a PR for https://github.com/python/cpython-source-deps at https://github.com/erlend-aasland/cpython-source-deps/tree/upgrade-sqlite. 

Patches for Windows and macOS installer builds on 3.9 prepared at https://github.com/erlend-aasland/cpython/tree/fix-issue-38380.

If it looks ok, I'll create PR's for CPython and cpython-source-deps. I guess this should be backported to 3.8, 3.7 and 2.7.
msg354281 - (view) Author: Erlend Egeberg Aasland (erlendaasland) * Date: 2019-10-09 15:24
bpo-38380: Update macOS installer to use SQLite 3.30.0
https://github.com/erlend-aasland/cpython/commit/aa7d7b1a3bed9a6a73f0611d0542a3442e85b0b6
msg354282 - (view) Author: Erlend Egeberg Aasland (erlendaasland) * Date: 2019-10-09 15:25
bpo-38380: Update Windows builds to use SQLite 3.30.0
https://github.com/erlend-aasland/cpython/commit/e25214e6fa7a64353d9c3e16b139c41f5d62eb31
msg354284 - (view) Author: Erlend Egeberg Aasland (erlendaasland) * Date: 2019-10-09 15:56
FYI: Compiled cpython 3.9 with sqlite-3.30 on macOS 10.14.6. Make test completes without errors.
msg354300 - (view) Author: Erlend Egeberg Aasland (erlendaasland) * Date: 2019-10-09 21:53
Update: Tested on macOS 10.14.6 with make test on 2.7.17rc1, 3.7.5rc1+, 3.8.0rc1+.
msg362803 - (view) Author: Erlend Egeberg Aasland (erlendaasland) * Date: 2020-02-27 14:40
I've updated these patches to sqlite3-3.30.1 (https://www.sqlite.org/releaselog/3_30_1.html).

cpython-source-deps is updated (https://github.com/python/cpython-source-deps/pull/17).

CPython commits are updated and rebased onto current 3.9 master: https://github.com/erlend-aasland/cpython/tree/fix-issue-38380


(Since Oct 2019, SQLite 3.31 have been released, but that would probably require opening a new issue.)
msg362923 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-02-28 22:24
We still need the tag added to the cpython-source-deps repo, and I still can't complete a clone right now (something is strange with SSL to GitHub on my (temporary) internet connection).

Zach - can you tag it? https://github.com/python/cpython-source-deps/pull/17
msg362928 - (view) Author: Erlend Egeberg Aasland (erlendaasland) * Date: 2020-02-28 22:31
Yes, we need the tag for the Windows build, so the PR currently fails the Windows checks. (Tagging must be done explicitly by the maintainers, IIRC.) I also forgot to add a NEWS entry, so I'll do another push to add those (and kick off the CI) when the tag arrives.
msg362929 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-02-28 22:33
Thanks for the PRs.  If we're going to update now as we should, why not to 3.31.1 which is current?
msg362930 - (view) Author: Erlend Egeberg Aasland (erlendaasland) * Date: 2020-02-28 22:40
You're welcome. If you are ok with that, I'd be happy to prepare a PR for the source deps for sqlite3 v3.31.1, and update GH-18678 as soon as it is tagged.
msg362933 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-02-28 22:52
I would prefer to go to 3.31.1 at this point particularly given the track record of the SQLite project. It's been released for a month now.  Any objections, Steve?
msg362935 - (view) Author: Erlend Egeberg Aasland (erlendaasland) * Date: 2020-02-28 22:58
I agree. I've updated the branches for source deps and cpython. I'll wait for Steve's approval before I open a new PR over at cpython-source-deps and update GH-18678.
msg362938 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-02-28 23:18
Isn't that what we ended up merging? (Goes to check). Ah, that was 3.30.1.

Sure, go for it. We'll have RCs of everything before the next final releases go out, so provided someone double checks that it's all good before then I'm okay with it.
msg362963 - (view) Author: Erlend Egeberg Aasland (erlendaasland) * Date: 2020-02-29 05:37
Great, thanks folks! I've pushed an update to GH-18678.

(BTW, is it kosher to force push to PRs like this?)
msg362970 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-02-29 11:17
> (BTW, is it kosher to force push to PRs like this?)

We always squash merge, so you don't need to do it to hide your history. Personally, I do it occasionally when the history is very long and nobody has started reviewing.

Force pushing will reset any reviews, so you'll be forcing reviewers to start again instead of just looking at your updates. Some reviewers (e.g. me) will often defer a full rereview if we don't have a lot of time and someone just force pushed. So in general, avoid it.
msg362973 - (view) Author: Erlend Egeberg Aasland (erlendaasland) * Date: 2020-02-29 11:50
Understood; I’ll remember it for future PRs. Thanks for clarifying.
msg363144 - (view) Author: Erlend Egeberg Aasland (erlendaasland) * Date: 2020-03-02 09:42
Would you prefer that I close GH_18678 and split it up in separate RPs for macOS and Windows?
msg363156 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-03-02 11:12
No, that one is fine.

I just need someone else to jump in and tag the commit in cpython-source-deps for me (because I can't clone it successfully on my current internet connection).
msg363158 - (view) Author: Erlend Egeberg Aasland (erlendaasland) * Date: 2020-03-02 11:26
Ah, but you're all set @steve.dower, someone at cpython-source-deps already pulled my tags: https://github.com/python/cpython-source-deps/tags :)

But I guess that if I were to do the same maneouver again, it would probably be preferable if I made one PR for macOS and another for Windows, right?
msg363160 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-03-02 11:35
Nope, same PR and bug is totally fine.

Putting the NEWS items in their platform-specific sections is how we normally do it (though to be fair, I had to go look it up).
msg363161 - (view) Author: Erlend Egeberg Aasland (erlendaasland) * Date: 2020-03-02 11:36
Ok, thanks for your patience and time. I'll fix the NEWS entries right away.
msg363162 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-03-02 11:39
I marked the PR to backport to 3.7 and 3.8.

Up to Benjamin whether 2.7 gets it, but unless there's a specific and impactful CVE that's been fixed, I doubt it (the one linked at the start of this issue seems to require direct modification of the SQL statement, which would be a bug in itself if permitted, so I think it's outside of our threat model for CPython).
msg363166 - (view) Author: miss-islington (miss-islington) Date: 2020-03-02 12:25
New changeset 1382c3289bcfd34ac6811fdf9aa5bc09ca8c320e by Erlend Egeberg Aasland in branch 'master':
bpo-38380: Update macOS & Windows builds to SQLite v3.31.1 (GH-18678)
https://github.com/python/cpython/commit/1382c3289bcfd34ac6811fdf9aa5bc09ca8c320e
msg363178 - (view) Author: miss-islington (miss-islington) Date: 2020-03-02 15:02
New changeset 7ad99821d8ae75222c50e69194a39f535bb058f5 by Miss Islington (bot) in branch '3.8':
bpo-38380: Update macOS & Windows builds to SQLite v3.31.1 (GH-18678)
https://github.com/python/cpython/commit/7ad99821d8ae75222c50e69194a39f535bb058f5
msg363179 - (view) Author: miss-islington (miss-islington) Date: 2020-03-02 15:02
New changeset 7ca251bd85f1182b9734579975c17fbd0488e2a4 by Miss Islington (bot) in branch '3.7':
bpo-38380: Update macOS & Windows builds to SQLite v3.31.1 (GH-18678)
https://github.com/python/cpython/commit/7ca251bd85f1182b9734579975c17fbd0488e2a4
msg363180 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-03-02 15:03
Leaving open until someone (*cough* Benjamin) confirms whether this has to go into 2.7 or not.
msg363182 - (view) Author: Erlend Egeberg Aasland (erlendaasland) * Date: 2020-03-02 15:05
Cool. Should I remove Python-3.* from the Versions label, so it doesn't show up in searches for 3.* issues?
msg363323 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2020-03-04 02:45
Traditionally, we've taken these dependency updates in 2.7, but there doesn't seem to be a compelling reason to take this one.
History
Date User Action Args
2020-03-04 02:45:36benjamin.petersonsetstatus: open -> closed
resolution: fixed
messages: + msg363323

stage: backport needed -> resolved
2020-03-02 15:05:22erlendaaslandsetmessages: + msg363182
2020-03-02 15:03:22steve.dowersetnosy: + benjamin.peterson

messages: + msg363180
stage: patch review -> backport needed
2020-03-02 15:02:11miss-islingtonsetmessages: + msg363179
2020-03-02 15:02:11miss-islingtonsetmessages: + msg363178
2020-03-02 12:25:34miss-islingtonsetpull_requests: + pull_request18096
2020-03-02 12:25:27miss-islingtonsetpull_requests: + pull_request18095
2020-03-02 12:25:19miss-islingtonsetnosy: + miss-islington
messages: + msg363166
2020-03-02 11:39:41steve.dowersetmessages: + msg363162
2020-03-02 11:36:32erlendaaslandsetmessages: + msg363161
2020-03-02 11:35:11steve.dowersetmessages: + msg363160
2020-03-02 11:26:48erlendaaslandsetmessages: + msg363158
2020-03-02 11:12:24steve.dowersetmessages: + msg363156
2020-03-02 09:42:47erlendaaslandsetmessages: + msg363144
2020-02-29 11:50:44erlendaaslandsetmessages: + msg362973
2020-02-29 11:17:39steve.dowersetmessages: + msg362970
2020-02-29 05:38:34erlendaaslandsettitle: Update SQLite to 3.30 in Windows and macOS installer builds -> Update SQLite to 3.31.1 in Windows and macOS installer builds
2020-02-29 05:37:41erlendaaslandsetmessages: + msg362963
2020-02-28 23:18:10steve.dowersetmessages: + msg362938
2020-02-28 22:58:58erlendaaslandsetmessages: + msg362935
2020-02-28 22:52:14ned.deilysetmessages: + msg362933
2020-02-28 22:40:42erlendaaslandsetmessages: + msg362930
2020-02-28 22:33:47ned.deilysetmessages: + msg362929
2020-02-28 22:31:35erlendaaslandsetmessages: + msg362928
2020-02-28 22:24:29steve.dowersetmessages: + msg362923
2020-02-27 15:00:06erlendaaslandsetstage: needs patch -> patch review
pull_requests: + pull_request18038
2020-02-27 14:56:24erlendaaslandsetfiles: + 0002-bpo-38380-Update-Windows-builds-to-use-SQLite-3.30.1.patch
2020-02-27 14:56:08erlendaaslandsetfiles: + 0001-bpo-38380-Update-macOS-installer-to-use-SQLite-3.30.1.patch
2020-02-27 14:40:43erlendaaslandsetmessages: + msg362803
2019-10-09 21:53:22erlendaaslandsetmessages: + msg354300
2019-10-09 15:56:36erlendaaslandsetmessages: + msg354284
2019-10-09 15:25:42erlendaaslandsetfiles: + 0002-bpo-38380-Update-Windows-builds-to-use-SQLite-3.30.0.patch

messages: + msg354282
2019-10-09 15:24:50erlendaaslandsetfiles: + 0001-bpo-38380-Update-macOS-installer-to-use-SQLite-3.30..patch
keywords: + patch
messages: + msg354281
2019-10-09 15:22:43erlendaaslandsetnosy: + erlendaasland
messages: + msg354280
2019-10-05 22:35:11ned.deilysetversions: + Python 2.7, Python 3.7, Python 3.8, Python 3.9
nosy: + ned.deily, paul.moore, tim.golden, steve.dower, zach.ware, ronaldoussoren

messages: + msg354025

components: + Build, macOS, Windows
stage: needs patch
2019-10-05 22:12:20Big Stonecreate