classification
Title: expat 2.2.0: Various security vulnerabilities in bundled expat (CVE-2016-0718 and CVE-2016-4472)
Type: security Stage: resolved
Components: XML Versions: Python 3.7, Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: christian.heimes Nosy List: Chi Hsuan Yen, Natanael Copa, benjamin.peterson, christian.heimes, georg.brandl, haypo, jeremy.kloth, jkloth, larry, matrixise, ned.deily
Priority: Keywords:

Created on 2017-02-17 15:39 by Natanael Copa, last changed 2017-07-16 08:48 by ned.deily. This issue is now closed.

Files
File name Uploaded Description Edit
rebuild_expat_dir.sh haypo, 2017-06-13 14:50
Pull Requests
URL Status Linked Edit
PR 2164 merged haypo, 2017-06-13 14:45
PR 2200 merged haypo, 2017-06-14 22:58
PR 2201 merged haypo, 2017-06-14 22:59
PR 2202 merged haypo, 2017-06-14 23:05
PR 2203 merged haypo, 2017-06-14 23:13
PR 2204 merged haypo, 2017-06-14 23:21
PR 2310 merged jkloth, 2017-06-21 10:27
PR 2348 merged haypo, 2017-06-23 07:46
PR 2349 merged haypo, 2017-06-23 08:11
PR 2350 merged haypo, 2017-06-23 08:11
Messages (35)
msg288014 - (view) Author: Natanael Copa (Natanael Copa) Date: 2017-02-17 15:39
cpython bundles expat in Modules/expat/ and needs to be updated to expat-2.2.0 to fix various security vulnerabilities.

21 June 2016, Expat 2.2.0 released.
Release 2.2.0 includes security & other bug fixes.

Security fixes

CVE-2016-0718 (issue 537)
Fix crash on malformed input

CVE-2016-4472
Improve insufficient fix to CVE-2015-1283 / CVE-2015-2716 introduced with Expat 2.1.1

CVE-2016-5300 (issue 499)
Use more entropy for hash initialization than the original fix to CVE-2012-0876

CVE-2012-6702 (issue 519)
Resolve troublesome internal call to srand that was introduced with Expat 2.1.0 when addressing CVE-2012-0876 (issue 496)

Fix should be applied to all maintained python branches.
msg288016 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-02-17 15:41
> CVE-2012-6702 (issue 519)
> Resolve troublesome internal call to srand that was introduced with Expat 2.1.0 when addressing CVE-2012-0876 (issue 496)

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-6702

Extract of Modules/pyexpat.c:
---
#if ((XML_MAJOR_VERSION >= 2) && (XML_MINOR_VERSION >= 1)) || defined(XML_HAS_SET_HASH_SALT)
    /* This feature was added upstream in libexpat 2.1.0.  Our expat copy
     * has a backport of this feature where we also define XML_HAS_SET_HASH_SALT
     * to indicate that we can still use it. */
    XML_SetHashSalt(self->itself,
                    (unsigned long)_Py_HashSecret.prefix);
#endif
---

Python 2.7, 3.5, 3.6 and 3.7 have this call at least (I didn't check other versions).
msg288017 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-02-17 15:42
You may want to look also at https://pypi.python.org/pypi/defusedxml
msg288018 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2017-02-17 15:44
CVE-2016-0718 and CVE-2016-4472 might be relevant for Python. CVE-2016-5300 and CVE-2012-6702 are irrelevant. As Victor already pointed out, Python seeds libexpat from a good CPRNG.
msg288296 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-02-21 13:19
I'm working on a new documentation of Python vulnerabilities to help to handle such issue:
http://python-security.readthedocs.io/en/latest/vulnerabilities.html
msg295841 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-06-13 03:03
Note that a duplicate of this issue was opened as Issue30610 and @matrixise was working on a PR there to update the embedded expat to 2.2.0.  Since there are CVE's and a demo crash supplied in Issue30610, it seems to me we need to fix this for 3.6.2rc1 so I'm making this a "release blocker" and delaying the release.  I'm willing to be convinced otherwise.  Christian or Victor, can one of you please follow up on this for the 3.6 branch ASAP?  Thanks!
msg295924 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-13 14:50
I upgraded Modules/expat/ to expat 2.2 using attached rebuild_expat_dir.sh script:
https://github.com/python/cpython/pull/2164

TODO: Should be done later in the master branch, once the security fix is handled.

* Drop support for VMS? VMS support removed from Python 3.4. Remove Modules/expat/expat_config.h
* Drop support for the Open Watcom compiler? Compiler not supported by Python. Remove Modules/expat/watcomconfig.h
* Send Python downstream changes to expat upstream?
msg296045 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-14 22:54
New changeset 23ec4b57e1359f9c539b8defc317542173ae087e by Victor Stinner in branch 'master':
bpo-29591: Upgrade Modules/expat to libexpat 2.2 (#2164)
https://github.com/python/cpython/commit/23ec4b57e1359f9c539b8defc317542173ae087e
msg296046 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-14 23:22
Python 3.3 currently embeds a copy of libexpat 2.1.0, wheras other branches have libexpat 2.1.1:
http://python-security.readthedocs.io/vuln/issue_26556_expat_2.1.1.html
msg296047 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-14 23:26
New changeset 86b95370c45dedb8a56c9894372a43681de47a73 by Victor Stinner in branch '3.6':
bpo-29591: Upgrade Modules/expat to libexpat 2.2 (#2164) (#2200)
https://github.com/python/cpython/commit/86b95370c45dedb8a56c9894372a43681de47a73
msg296048 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-14 23:27
@Ned Deily: I removed the "release blocker" flag, since I just merged my PR to update libexpat to 2.2 in the Python 3.6 branch.
msg296050 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-15 00:16
New changeset 0e4571a68a7f48e8469ef05b04ba3463d3fd82c0 by Victor Stinner in branch '2.7':
bpo-29591: Upgrade Modules/expat to libexpat 2.2 (#2164) (#2202)
https://github.com/python/cpython/commit/0e4571a68a7f48e8469ef05b04ba3463d3fd82c0
msg296051 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-15 00:16
New changeset 8c797ed8a0fea5e3162b9415f13e270d4d5d9549 by Victor Stinner in branch '3.5':
bpo-29591: Upgrade Modules/expat to libexpat 2.2 (#2164) (#2201)
https://github.com/python/cpython/commit/8c797ed8a0fea5e3162b9415f13e270d4d5d9549
msg296067 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-06-15 05:04
Thanks, Victor, for seeing this through and thanks, everyone else, for the reviews and assistance.
msg296255 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-06-18 03:03
FYI, expat 2.2.1 has now been released.  See Issue30694 for details.
msg296537 - (view) Author: Jeremy Kloth (jkloth) * Date: 2017-06-21 10:31
Added pull_request2355 to address issues from upgrading to Expat 2.2.0 on Windows 2.7
msg296552 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-21 13:17
> Added pull_request2355 to address issues from upgrading to Expat 2.2.0 on Windows 2.7

I would prefer to first fix the new vulnerabilities, by upgrading expat to 2.2.1, and then review your change.

=> https://github.com/python/cpython/pull/2312
msg296574 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-21 19:14
New changeset ab3b0ade505ce07a3d5ec4fbc991a154242732e6 by Victor Stinner (Jeremy Kloth) in branch '2.7':
bpo-29591: Update VS project files (#2310)
https://github.com/python/cpython/commit/ab3b0ade505ce07a3d5ec4fbc991a154242732e6
msg296579 - (view) Author: Jeremy Kloth (jeremy.kloth) Date: 2017-06-21 20:08
Just a note with the PR, the changes to PCbuild\pyexpat.vcxproj and
PCbuild\_elementtree.vcxproj should probably be merged forward as
well.

On Wed, Jun 21, 2017 at 1:14 PM, STINNER Victor <report@bugs.python.org> wrote:
>
> STINNER Victor added the comment:
>
>
> New changeset ab3b0ade505ce07a3d5ec4fbc991a154242732e6 by Victor Stinner (Jeremy Kloth) in branch '2.7':
> bpo-29591: Update VS project files (#2310)
> https://github.com/python/cpython/commit/ab3b0ade505ce07a3d5ec4fbc991a154242732e6
>
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue29591>
> _______________________________________
msg296585 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-21 21:10
Jeremy Kloth added the comment:
> Just a note with the PR, the changes to PCbuild\pyexpat.vcxproj and
> PCbuild\_elementtree.vcxproj should probably be merged forward as
> well.

PR 2310. Yes, I agree. Can you please propose patches for master, and
then 3.6 and 3.5, please?
msg296689 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-23 08:09
New changeset c8fb58bd7917151e63398587a7fc2126db7c26de by Victor Stinner in branch 'master':
bpo-30726: PCbuild _elementtree: remove duplicate defines (#2348)
https://github.com/python/cpython/commit/c8fb58bd7917151e63398587a7fc2126db7c26de
msg296713 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-23 13:14
New changeset f42ce179c8aaa7e211ac4123c58fa3dd9a452004 by Victor Stinner in branch '3.5':
[3.5] bpo-30726: PCbuild _elementtree: remove duplicate defines (#2348) (#2350)
https://github.com/python/cpython/commit/f42ce179c8aaa7e211ac4123c58fa3dd9a452004
msg296716 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-23 13:21
New changeset d32a05953130fb5cc2d3c0c9fcb20ad0859353f3 by Victor Stinner in branch '3.6':
[3.6] bpo-30726: PCbuild _elementtree: remove duplicate defines (#2348) (#2349)
https://github.com/python/cpython/commit/d32a05953130fb5cc2d3c0c9fcb20ad0859353f3
msg297932 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-07-08 04:51
New changeset 5777e79ecbd1f2adf36456e09f210608ee221691 by Ned Deily (Victor Stinner) in branch '3.6':
[3.6] bpo-30726: PCbuild _elementtree: remove duplicate defines (#2348) (#2349)
https://github.com/python/cpython/commit/5777e79ecbd1f2adf36456e09f210608ee221691
msg298157 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2017-07-11 13:18
I don't quite understand what's happening on this issue.  I see that master, 3.6, 3.6, and 2.7 have been upgraded to expat 2.2.0.  This issue was created to upgrade CPython to 2.2.0.  But the PR against 3.3 and 3.4 upgrade expat to 2.2.1?!

I'm not against this change in principle, I'm just trying to understand why a) it doesn't match the issue, b) why 3.3 and 3.4 are special, c) why we don't upgrade master & 3.6 & 3.5 & 2.7 to expat 2.2.1.
msg298178 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-07-11 19:51
> I don't quite understand what's happening on this issue.  I see that master, 3.6, 3.6, and 2.7 have been upgraded to expat 2.2.0.  This issue was created to upgrade CPython to 2.2.0.  But the PR against 3.3 and 3.4 upgrade expat to 2.2.1?!
>
> I'm not against this change in principle, I'm just trying to understand why a) it doesn't match the issue, b) why 3.3 and 3.4 are special, c) why we don't upgrade master & 3.6 & 3.5 & 2.7 to expat 2.2.1.

I upgraded libexpat to 2.2.0 in this issue, and then to libexpat 2.2.1 in bpo-30694.

For 3.3 and 3.4 pull requests, I chose to use this bpo number.

3.3: https://github.com/python/cpython/pull/2204
3.4: https://github.com/python/cpython/pull/2203

So these pull requests upgrade directly to 2.2.1.
msg298179 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2017-07-11 20:06
Please instead choose to use bpo-30694 for the upgrades of 3.3 and 3.4 to expat 2.2.1.  I guess there are historical reasons why the PRs are here, but bpo stands as a historical record; let's not confuse posterity by upgrading to 2.2.1 using a bpo issue talking about--and upgrading four branches to--2.2.0.
msg298180 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-07-11 20:43
Larry: "Please instead choose to use bpo-30694 for the upgrades of 3.3 and 3.4 to expat 2.2.1.  I guess there are historical reasons why the PRs are here, but bpo stands as a historical record; let's not confuse posterity by upgrading to 2.2.1 using a bpo issue talking about--and upgrading four branches to--2.2.0."

I just updated the 3.4 PR.

In fact, the PR backports the libexpat 2.2.0 commit *and* then the libexpat 2.2.1 commit. Since it's not possible to create a "patch serie" (in GitHub, it would mean a PR which depends on another PR), I chose to stack the two commits in the same PR and reuse the existing PR to not loose context.
msg298182 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-07-11 20:44
I changed the PR title to mention the two bpo.
msg298206 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2017-07-12 12:35
Okay.  Closing this bug, because all the branches that are being upgraded to expat 2.2.*0* have already gotten their upgrades.  Job done.

The discussions for PRs 2203 and 2204 should move to Issue #30694, which is for the upgrade to expat 2.2.*1*.
msg298209 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2017-07-12 12:41
New changeset 71572bbe82aa0836c036d44d41c8269ba6a321be by larryhastings (Victor Stinner) in branch '3.4':
[3.4] bpo-29591, bpo-30694: Upgrade Modules/expat to libexpat 2.2.1 (#2164) (#2203)
https://github.com/python/cpython/commit/71572bbe82aa0836c036d44d41c8269ba6a321be
msg298210 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-07-12 12:46
> Okay.  Closing this bug, because all the branches that are being upgraded to expat 2.2.*0* have already gotten their upgrades.  Job done.

Well, technically 3.3 wasn't upgraded yet:
https://github.com/python/cpython/pull/2204
msg298215 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2017-07-12 13:22
Correct.  But technically 3.3 is being upgraded to 2.2.*1*, which is being tracked on--repeating myself here--Issue #30694.
msg298216 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2017-07-12 13:23
Here, I'll remove 3.4 and 3.5 from the versions affected.  Now everybody can be pedantic!
msg298428 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-07-16 08:48
New changeset ab90986600ba7dea2aa41e5c1773791070725453 by Ned Deily (Victor Stinner) in branch '3.3':
[3.3] bpo-29591, bpo-30694: Upgrade Modules/expat to libexpat 2.2.1 (#2164) (#2204)
https://github.com/python/cpython/commit/ab90986600ba7dea2aa41e5c1773791070725453
History
Date User Action Args
2017-07-16 08:48:07ned.deilysetmessages: + msg298428
2017-07-12 13:23:13larrysetmessages: + msg298216
versions: - Python 3.3, Python 3.4
2017-07-12 13:22:39larrysetmessages: + msg298215
2017-07-12 12:46:40hayposetmessages: + msg298210
2017-07-12 12:41:36larrysetmessages: + msg298209
2017-07-12 12:35:46larrysetstatus: open -> closed
resolution: fixed
messages: + msg298206

stage: needs patch -> resolved
2017-07-11 20:44:18hayposetmessages: + msg298182
2017-07-11 20:43:26hayposetmessages: + msg298180
2017-07-11 20:06:42larrysetmessages: + msg298179
2017-07-11 19:51:10hayposetmessages: + msg298178
2017-07-11 13:18:53larrysetmessages: + msg298157
2017-07-08 04:51:39ned.deilysetmessages: + msg297932
2017-06-23 13:21:35hayposetmessages: + msg296716
2017-06-23 13:14:20hayposetmessages: + msg296713
2017-06-23 08:11:33hayposetpull_requests: + pull_request2396
2017-06-23 08:11:19hayposetpull_requests: + pull_request2394
2017-06-23 08:09:37hayposetmessages: + msg296689
2017-06-23 07:46:56hayposetpull_requests: + pull_request2392
2017-06-21 21:10:49hayposetmessages: + msg296585
2017-06-21 20:08:10jeremy.klothsetnosy: + jeremy.kloth
messages: + msg296579
2017-06-21 19:14:55hayposetmessages: + msg296574
2017-06-21 13:17:19hayposetmessages: + msg296552
2017-06-21 12:27:54hayposettitle: Various security vulnerabilities in bundled expat (CVE-2016-0718 and CVE-2016-4472) -> expat 2.2.0: Various security vulnerabilities in bundled expat (CVE-2016-0718 and CVE-2016-4472)
2017-06-21 10:31:29jklothsetnosy: + jkloth
messages: + msg296537
2017-06-21 10:27:44jklothsetpull_requests: + pull_request2355
2017-06-18 03:03:08ned.deilysetmessages: + msg296255
2017-06-15 05:04:13ned.deilysetmessages: + msg296067
2017-06-15 00:16:40hayposetmessages: + msg296051
2017-06-15 00:16:38hayposetmessages: + msg296050
2017-06-14 23:27:17hayposetpriority: release blocker ->

messages: + msg296048
2017-06-14 23:26:19hayposetmessages: + msg296047
2017-06-14 23:22:57hayposetmessages: + msg296046
2017-06-14 23:21:41hayposetpull_requests: + pull_request2248
2017-06-14 23:19:15hayposetnosy: + georg.brandl

versions: + Python 3.3, Python 3.4
2017-06-14 23:13:20hayposetpull_requests: + pull_request2247
2017-06-14 23:05:56hayposetpull_requests: + pull_request2246
2017-06-14 22:59:56hayposetpull_requests: + pull_request2245
2017-06-14 22:58:33hayposetpull_requests: + pull_request2244
2017-06-14 22:54:38hayposetmessages: + msg296045
2017-06-13 14:50:42hayposetfiles: + rebuild_expat_dir.sh

messages: + msg295924
2017-06-13 14:45:01hayposetpull_requests: + pull_request2215
2017-06-13 03:04:45ned.deilysetnosy: + matrixise
2017-06-13 03:03:47ned.deilysetpriority: normal -> release blocker
nosy: + larry, benjamin.peterson, ned.deily
messages: + msg295841

2017-06-13 02:58:00ned.deilylinkissue30610 superseder
2017-02-21 13:19:13hayposetmessages: + msg288296
2017-02-20 09:55:38Natanael Copasettitle: Various security vulnerabilities in bundled expat -> Various security vulnerabilities in bundled expat (CVE-2016-0718 and CVE-2016-4472)
2017-02-17 15:46:00christian.heimessetassignee: christian.heimes
stage: needs patch
components: + XML
versions: - Python 3.3, Python 3.4
2017-02-17 15:44:49christian.heimessetmessages: + msg288018
2017-02-17 15:43:56Chi Hsuan Yensetnosy: + Chi Hsuan Yen
2017-02-17 15:42:34hayposetmessages: + msg288017
2017-02-17 15:41:46hayposetnosy: + christian.heimes, haypo
messages: + msg288016
2017-02-17 15:39:39Natanael Copacreate