classification
Title: expat 2.2.0: Various security vulnerabilities in bundled expat (CVE-2016-0718 and CVE-2016-4472)
Type: security Stage: needs patch
Components: XML Versions: Python 3.7, Python 3.6, Python 3.5, Python 3.4, Python 3.3, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: christian.heimes Nosy List: Chi Hsuan Yen, Natanael Copa, benjamin.peterson, christian.heimes, georg.brandl, haypo, jeremy.kloth, jkloth, larry, matrixise, ned.deily
Priority: Keywords:

Created on 2017-02-17 15:39 by Natanael Copa, last changed 2017-06-23 13:21 by haypo.

Files
File name Uploaded Description Edit
rebuild_expat_dir.sh haypo, 2017-06-13 14:50
Pull Requests
URL Status Linked Edit
PR 2164 merged haypo, 2017-06-13 14:45
PR 2200 merged haypo, 2017-06-14 22:58
PR 2201 merged haypo, 2017-06-14 22:59
PR 2202 merged haypo, 2017-06-14 23:05
PR 2203 open haypo, 2017-06-14 23:13
PR 2204 open haypo, 2017-06-14 23:21
PR 2310 merged jkloth, 2017-06-21 10:27
PR 2348 merged haypo, 2017-06-23 07:46
PR 2349 merged haypo, 2017-06-23 08:11
PR 2350 merged haypo, 2017-06-23 08:11
Messages (23)
msg288014 - (view) Author: Natanael Copa (Natanael Copa) Date: 2017-02-17 15:39
cpython bundles expat in Modules/expat/ and needs to be updated to expat-2.2.0 to fix various security vulnerabilities.

21 June 2016, Expat 2.2.0 released.
Release 2.2.0 includes security & other bug fixes.

Security fixes

CVE-2016-0718 (issue 537)
Fix crash on malformed input

CVE-2016-4472
Improve insufficient fix to CVE-2015-1283 / CVE-2015-2716 introduced with Expat 2.1.1

CVE-2016-5300 (issue 499)
Use more entropy for hash initialization than the original fix to CVE-2012-0876

CVE-2012-6702 (issue 519)
Resolve troublesome internal call to srand that was introduced with Expat 2.1.0 when addressing CVE-2012-0876 (issue 496)

Fix should be applied to all maintained python branches.
msg288016 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-02-17 15:41
> CVE-2012-6702 (issue 519)
> Resolve troublesome internal call to srand that was introduced with Expat 2.1.0 when addressing CVE-2012-0876 (issue 496)

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-6702

Extract of Modules/pyexpat.c:
---
#if ((XML_MAJOR_VERSION >= 2) && (XML_MINOR_VERSION >= 1)) || defined(XML_HAS_SET_HASH_SALT)
    /* This feature was added upstream in libexpat 2.1.0.  Our expat copy
     * has a backport of this feature where we also define XML_HAS_SET_HASH_SALT
     * to indicate that we can still use it. */
    XML_SetHashSalt(self->itself,
                    (unsigned long)_Py_HashSecret.prefix);
#endif
---

Python 2.7, 3.5, 3.6 and 3.7 have this call at least (I didn't check other versions).
msg288017 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-02-17 15:42
You may want to look also at https://pypi.python.org/pypi/defusedxml
msg288018 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2017-02-17 15:44
CVE-2016-0718 and CVE-2016-4472 might be relevant for Python. CVE-2016-5300 and CVE-2012-6702 are irrelevant. As Victor already pointed out, Python seeds libexpat from a good CPRNG.
msg288296 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-02-21 13:19
I'm working on a new documentation of Python vulnerabilities to help to handle such issue:
http://python-security.readthedocs.io/en/latest/vulnerabilities.html
msg295841 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-06-13 03:03
Note that a duplicate of this issue was opened as Issue30610 and @matrixise was working on a PR there to update the embedded expat to 2.2.0.  Since there are CVE's and a demo crash supplied in Issue30610, it seems to me we need to fix this for 3.6.2rc1 so I'm making this a "release blocker" and delaying the release.  I'm willing to be convinced otherwise.  Christian or Victor, can one of you please follow up on this for the 3.6 branch ASAP?  Thanks!
msg295924 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-13 14:50
I upgraded Modules/expat/ to expat 2.2 using attached rebuild_expat_dir.sh script:
https://github.com/python/cpython/pull/2164

TODO: Should be done later in the master branch, once the security fix is handled.

* Drop support for VMS? VMS support removed from Python 3.4. Remove Modules/expat/expat_config.h
* Drop support for the Open Watcom compiler? Compiler not supported by Python. Remove Modules/expat/watcomconfig.h
* Send Python downstream changes to expat upstream?
msg296045 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-14 22:54
New changeset 23ec4b57e1359f9c539b8defc317542173ae087e by Victor Stinner in branch 'master':
bpo-29591: Upgrade Modules/expat to libexpat 2.2 (#2164)
https://github.com/python/cpython/commit/23ec4b57e1359f9c539b8defc317542173ae087e
msg296046 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-14 23:22
Python 3.3 currently embeds a copy of libexpat 2.1.0, wheras other branches have libexpat 2.1.1:
http://python-security.readthedocs.io/vuln/issue_26556_expat_2.1.1.html
msg296047 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-14 23:26
New changeset 86b95370c45dedb8a56c9894372a43681de47a73 by Victor Stinner in branch '3.6':
bpo-29591: Upgrade Modules/expat to libexpat 2.2 (#2164) (#2200)
https://github.com/python/cpython/commit/86b95370c45dedb8a56c9894372a43681de47a73
msg296048 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-14 23:27
@Ned Deily: I removed the "release blocker" flag, since I just merged my PR to update libexpat to 2.2 in the Python 3.6 branch.
msg296050 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-15 00:16
New changeset 0e4571a68a7f48e8469ef05b04ba3463d3fd82c0 by Victor Stinner in branch '2.7':
bpo-29591: Upgrade Modules/expat to libexpat 2.2 (#2164) (#2202)
https://github.com/python/cpython/commit/0e4571a68a7f48e8469ef05b04ba3463d3fd82c0
msg296051 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-15 00:16
New changeset 8c797ed8a0fea5e3162b9415f13e270d4d5d9549 by Victor Stinner in branch '3.5':
bpo-29591: Upgrade Modules/expat to libexpat 2.2 (#2164) (#2201)
https://github.com/python/cpython/commit/8c797ed8a0fea5e3162b9415f13e270d4d5d9549
msg296067 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-06-15 05:04
Thanks, Victor, for seeing this through and thanks, everyone else, for the reviews and assistance.
msg296255 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-06-18 03:03
FYI, expat 2.2.1 has now been released.  See Issue30694 for details.
msg296537 - (view) Author: Jeremy Kloth (jkloth) * Date: 2017-06-21 10:31
Added pull_request2355 to address issues from upgrading to Expat 2.2.0 on Windows 2.7
msg296552 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-21 13:17
> Added pull_request2355 to address issues from upgrading to Expat 2.2.0 on Windows 2.7

I would prefer to first fix the new vulnerabilities, by upgrading expat to 2.2.1, and then review your change.

=> https://github.com/python/cpython/pull/2312
msg296574 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-21 19:14
New changeset ab3b0ade505ce07a3d5ec4fbc991a154242732e6 by Victor Stinner (Jeremy Kloth) in branch '2.7':
bpo-29591: Update VS project files (#2310)
https://github.com/python/cpython/commit/ab3b0ade505ce07a3d5ec4fbc991a154242732e6
msg296579 - (view) Author: Jeremy Kloth (jeremy.kloth) Date: 2017-06-21 20:08
Just a note with the PR, the changes to PCbuild\pyexpat.vcxproj and
PCbuild\_elementtree.vcxproj should probably be merged forward as
well.

On Wed, Jun 21, 2017 at 1:14 PM, STINNER Victor <report@bugs.python.org> wrote:
>
> STINNER Victor added the comment:
>
>
> New changeset ab3b0ade505ce07a3d5ec4fbc991a154242732e6 by Victor Stinner (Jeremy Kloth) in branch '2.7':
> bpo-29591: Update VS project files (#2310)
> https://github.com/python/cpython/commit/ab3b0ade505ce07a3d5ec4fbc991a154242732e6
>
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue29591>
> _______________________________________
msg296585 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-21 21:10
Jeremy Kloth added the comment:
> Just a note with the PR, the changes to PCbuild\pyexpat.vcxproj and
> PCbuild\_elementtree.vcxproj should probably be merged forward as
> well.

PR 2310. Yes, I agree. Can you please propose patches for master, and
then 3.6 and 3.5, please?
msg296689 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-23 08:09
New changeset c8fb58bd7917151e63398587a7fc2126db7c26de by Victor Stinner in branch 'master':
bpo-30726: PCbuild _elementtree: remove duplicate defines (#2348)
https://github.com/python/cpython/commit/c8fb58bd7917151e63398587a7fc2126db7c26de
msg296713 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-23 13:14
New changeset f42ce179c8aaa7e211ac4123c58fa3dd9a452004 by Victor Stinner in branch '3.5':
[3.5] bpo-30726: PCbuild _elementtree: remove duplicate defines (#2348) (#2350)
https://github.com/python/cpython/commit/f42ce179c8aaa7e211ac4123c58fa3dd9a452004
msg296716 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-23 13:21
New changeset d32a05953130fb5cc2d3c0c9fcb20ad0859353f3 by Victor Stinner in branch '3.6':
[3.6] bpo-30726: PCbuild _elementtree: remove duplicate defines (#2348) (#2349)
https://github.com/python/cpython/commit/d32a05953130fb5cc2d3c0c9fcb20ad0859353f3
History
Date User Action Args
2017-06-23 13:21:35hayposetmessages: + msg296716
2017-06-23 13:14:20hayposetmessages: + msg296713
2017-06-23 08:11:33hayposetpull_requests: + pull_request2396
2017-06-23 08:11:19hayposetpull_requests: + pull_request2394
2017-06-23 08:09:37hayposetmessages: + msg296689
2017-06-23 07:46:56hayposetpull_requests: + pull_request2392
2017-06-21 21:10:49hayposetmessages: + msg296585
2017-06-21 20:08:10jeremy.klothsetnosy: + jeremy.kloth
messages: + msg296579
2017-06-21 19:14:55hayposetmessages: + msg296574
2017-06-21 13:17:19hayposetmessages: + msg296552
2017-06-21 12:27:54hayposettitle: Various security vulnerabilities in bundled expat (CVE-2016-0718 and CVE-2016-4472) -> expat 2.2.0: Various security vulnerabilities in bundled expat (CVE-2016-0718 and CVE-2016-4472)
2017-06-21 10:31:29jklothsetnosy: + jkloth
messages: + msg296537
2017-06-21 10:27:44jklothsetpull_requests: + pull_request2355
2017-06-18 03:03:08ned.deilysetmessages: + msg296255
2017-06-15 05:04:13ned.deilysetmessages: + msg296067
2017-06-15 00:16:40hayposetmessages: + msg296051
2017-06-15 00:16:38hayposetmessages: + msg296050
2017-06-14 23:27:17hayposetpriority: release blocker ->

messages: + msg296048
2017-06-14 23:26:19hayposetmessages: + msg296047
2017-06-14 23:22:57hayposetmessages: + msg296046
2017-06-14 23:21:41hayposetpull_requests: + pull_request2248
2017-06-14 23:19:15hayposetnosy: + georg.brandl

versions: + Python 3.3, Python 3.4
2017-06-14 23:13:20hayposetpull_requests: + pull_request2247
2017-06-14 23:05:56hayposetpull_requests: + pull_request2246
2017-06-14 22:59:56hayposetpull_requests: + pull_request2245
2017-06-14 22:58:33hayposetpull_requests: + pull_request2244
2017-06-14 22:54:38hayposetmessages: + msg296045
2017-06-13 14:50:42hayposetfiles: + rebuild_expat_dir.sh

messages: + msg295924
2017-06-13 14:45:01hayposetpull_requests: + pull_request2215
2017-06-13 03:04:45ned.deilysetnosy: + matrixise
2017-06-13 03:03:47ned.deilysetpriority: normal -> release blocker
nosy: + larry, benjamin.peterson, ned.deily
messages: + msg295841

2017-06-13 02:58:00ned.deilylinkissue30610 superseder
2017-02-21 13:19:13hayposetmessages: + msg288296
2017-02-20 09:55:38Natanael Copasettitle: Various security vulnerabilities in bundled expat -> Various security vulnerabilities in bundled expat (CVE-2016-0718 and CVE-2016-4472)
2017-02-17 15:46:00christian.heimessetassignee: christian.heimes
stage: needs patch
components: + XML
versions: - Python 3.3, Python 3.4
2017-02-17 15:44:49christian.heimessetmessages: + msg288018
2017-02-17 15:43:56Chi Hsuan Yensetnosy: + Chi Hsuan Yen
2017-02-17 15:42:34hayposetmessages: + msg288017
2017-02-17 15:41:46hayposetnosy: + christian.heimes, haypo
messages: + msg288016
2017-02-17 15:39:39Natanael Copacreate