expat 2.2.0: Various security vulnerabilities in bundled expat (CVE-2016-0718 and CVE-2016-4472) #73777

NatanaelCopa · 2017-02-17T15:39:40Z

BPO	29591
Nosy	@birkenfeld, @vstinner, @larryhastings, @tiran, @benjaminp, @jkloth, @ned-deily, @matrixise, @yan12125
PRs	bpo-29591: Upgrade Modules/expat to libexpat 2.2 #2164 [3.6] bpo-29591: Upgrade Modules/expat to libexpat 2.2 (#2164) #2200 [3.5] bpo-29591: Upgrade Modules/expat to libexpat 2.2 (#2164) #2201 [2.7] bpo-29591: Upgrade Modules/expat to libexpat 2.2 (#2164) #2202 [3.4] bpo-29591, bpo-30694: Upgrade Modules/expat to libexpat 2.2.1 (#2164) #2203 [3.3] bpo-29591, bpo-30694: Upgrade Modules/expat to libexpat 2.2.1 (#2164) #2204 bpo-29591: [2.7] Update VS project files #2310 bpo-30726: PCbuild _elementtree: remove duplicate defines #2348 [3.6] bpo-30726: PCbuild _elementtree: remove duplicate defines (#2348) #2349 [3.5] bpo-30726: PCbuild _elementtree: remove duplicate defines (#2348) #2350
Files	rebuild_expat_dir.sh

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tiran'
closed_at = <Date 2017-07-12.12:35:46.638>
created_at = <Date 2017-02-17.15:39:39.676>
labels = ['type-security', 'expert-XML', '3.7']
title = 'expat 2.2.0: Various security vulnerabilities in bundled expat (CVE-2016-0718 and CVE-2016-4472)'
updated_at = <Date 2019-05-10.18:12:37.639>
user = 'https://bugs.python.org/NatanaelCopa'

bugs.python.org fields:

activity = <Date 2019-05-10.18:12:37.639>
actor = 'ned.deily'
assignee = 'christian.heimes'
closed = True
closed_date = <Date 2017-07-12.12:35:46.638>
closer = 'larry'
components = ['XML']
creation = <Date 2017-02-17.15:39:39.676>
creator = 'Natanael Copa'
dependencies = []
files = ['46949']
hgrepos = []
issue_num = 29591
keywords = []
message_count = 35.0
messages = ['288014', '288016', '288017', '288018', '288296', '295841', '295924', '296045', '296046', '296047', '296048', '296050', '296051', '296067', '296255', '296537', '296552', '296574', '296579', '296585', '296689', '296713', '296716', '297932', '298157', '298178', '298179', '298180', '298182', '298206', '298209', '298210', '298215', '298216', '298428']
nosy_count = 11.0
nosy_names = ['georg.brandl', 'vstinner', 'larry', 'christian.heimes', 'benjamin.peterson', 'jkloth', 'ned.deily', 'jeremy.kloth', 'matrixise', 'yan12125', 'Natanael Copa']
pr_nums = ['2164', '2200', '2201', '2202', '2203', '2204', '2310', '2348', '2349', '2350']
priority = None
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'security'
url = 'https://bugs.python.org/issue29591'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7']

NatanaelCopa · 2017-02-17T15:39:39Z

cpython bundles expat in Modules/expat/ and needs to be updated to expat-2.2.0 to fix various security vulnerabilities.

21 June 2016, Expat 2.2.0 released.
Release 2.2.0 includes security & other bug fixes.

Security fixes

CVE-2016-0718 (issue 537)
Fix crash on malformed input

CVE-2016-4472
Improve insufficient fix to CVE-2015-1283 / CVE-2015-2716 introduced with Expat 2.1.1

CVE-2016-5300 (issue 499)
Use more entropy for hash initialization than the original fix to CVE-2012-0876

CVE-2012-6702 (issue 519)
Resolve troublesome internal call to srand that was introduced with Expat 2.1.0 when addressing CVE-2012-0876 (issue 496)

Fix should be applied to all maintained python branches.

vstinner · 2017-02-17T15:41:47Z

CVE-2012-6702 (issue 519)
Resolve troublesome internal call to srand that was introduced with Expat 2.1.0 when addressing CVE-2012-0876 (issue 496)

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-6702

Extract of Modules/pyexpat.c:
---

#if ((XML_MAJOR_VERSION >= 2) && (XML_MINOR_VERSION >= 1)) || defined(XML_HAS_SET_HASH_SALT)
    /* This feature was added upstream in libexpat 2.1.0.  Our expat copy
     * has a backport of this feature where we also define XML_HAS_SET_HASH_SALT
     * to indicate that we can still use it. */
    XML_SetHashSalt(self->itself,
                    (unsigned long)_Py_HashSecret.prefix);
#endif

Python 2.7, 3.5, 3.6 and 3.7 have this call at least (I didn't check other versions).

vstinner · 2017-02-17T15:42:35Z

You may want to look also at https://pypi.python.org/pypi/defusedxml

tiran · 2017-02-17T15:44:49Z

CVE-2016-0718 and CVE-2016-4472 might be relevant for Python. CVE-2016-5300 and CVE-2012-6702 are irrelevant. As Victor already pointed out, Python seeds libexpat from a good CPRNG.

vstinner · 2017-02-21T13:19:14Z

I'm working on a new documentation of Python vulnerabilities to help to handle such issue:
http://python-security.readthedocs.io/en/latest/vulnerabilities.html

ned-deily · 2017-06-13T03:03:47Z

Note that a duplicate of this issue was opened as bpo-30610 and @matrixise was working on a PR there to update the embedded expat to 2.2.0. Since there are CVE's and a demo crash supplied in bpo-30610, it seems to me we need to fix this for 3.6.2rc1 so I'm making this a "release blocker" and delaying the release. I'm willing to be convinced otherwise. Christian or Victor, can one of you please follow up on this for the 3.6 branch ASAP? Thanks!

vstinner · 2017-06-13T14:50:42Z

I upgraded Modules/expat/ to expat 2.2 using attached rebuild_expat_dir.sh script:
#2164

TODO: Should be done later in the master branch, once the security fix is handled.

Drop support for VMS? VMS support removed from Python 3.4. Remove Modules/expat/expat_config.h
Drop support for the Open Watcom compiler? Compiler not supported by Python. Remove Modules/expat/watcomconfig.h
Send Python downstream changes to expat upstream?

vstinner · 2017-06-14T22:54:38Z

New changeset 23ec4b5 by Victor Stinner in branch 'master':
bpo-29591: Upgrade Modules/expat to libexpat 2.2 (bpo-2164)
23ec4b5

vstinner · 2017-06-14T23:22:57Z

Python 3.3 currently embeds a copy of libexpat 2.1.0, wheras other branches have libexpat 2.1.1:
http://python-security.readthedocs.io/vuln/issue_26556_expat_2.1.1.html

vstinner · 2017-06-14T23:26:20Z

New changeset 86b9537 by Victor Stinner in branch '3.6':
bpo-29591: Upgrade Modules/expat to libexpat 2.2 (bpo-2164) (bpo-2200)
86b9537

vstinner · 2017-06-14T23:27:17Z

@ned Deily: I removed the "release blocker" flag, since I just merged my PR to update libexpat to 2.2 in the Python 3.6 branch.

vstinner · 2017-06-15T00:16:38Z

New changeset 0e4571a by Victor Stinner in branch '2.7':
bpo-29591: Upgrade Modules/expat to libexpat 2.2 (bpo-2164) (bpo-2202)
0e4571a

vstinner · 2017-06-15T00:16:41Z

New changeset 8c797ed by Victor Stinner in branch '3.5':
bpo-29591: Upgrade Modules/expat to libexpat 2.2 (bpo-2164) (bpo-2201)
8c797ed

ned-deily · 2017-06-15T05:04:13Z

Thanks, Victor, for seeing this through and thanks, everyone else, for the reviews and assistance.

ned-deily · 2017-06-18T03:03:08Z

FYI, expat 2.2.1 has now been released. See bpo-30694 for details.

jkloth · 2017-06-21T10:31:29Z

Added pull_request2355 to address issues from upgrading to Expat 2.2.0 on Windows 2.7

vstinner · 2017-06-21T13:17:20Z

Added pull_request2355 to address issues from upgrading to Expat 2.2.0 on Windows 2.7

I would prefer to first fix the new vulnerabilities, by upgrading expat to 2.2.1, and then review your change.

=> #2312

vstinner · 2017-06-21T19:14:56Z

New changeset ab3b0ad by Victor Stinner (Jeremy Kloth) in branch '2.7':
bpo-29591: Update VS project files (bpo-2310)
ab3b0ad

jeremykloth · 2017-06-21T20:08:10Z

Just a note with the PR, the changes to PCbuild\pyexpat.vcxproj and
PCbuild\_elementtree.vcxproj should probably be merged forward as
well.

On Wed, Jun 21, 2017 at 1:14 PM, STINNER Victor <report@bugs.python.org> wrote:

STINNER Victor added the comment:

New changeset ab3b0ad by Victor Stinner (Jeremy Kloth) in branch '2.7':
bpo-29591: Update VS project files (bpo-2310)
ab3b0ad

----------

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue29591\>

vstinner · 2017-06-21T21:10:49Z

Jeremy Kloth added the comment:

Just a note with the PR, the changes to PCbuild\pyexpat.vcxproj and
PCbuild\_elementtree.vcxproj should probably be merged forward as
well.

PR 2310. Yes, I agree. Can you please propose patches for master, and
then 3.6 and 3.5, please?

vstinner · 2017-06-23T08:09:37Z

New changeset c8fb58b by Victor Stinner in branch 'master':
bpo-30726: PCbuild _elementtree: remove duplicate defines (bpo-2348)
c8fb58b

vstinner · 2017-06-23T13:14:21Z

New changeset f42ce17 by Victor Stinner in branch '3.5':
[3.5] bpo-30726: PCbuild _elementtree: remove duplicate defines (bpo-2348) (bpo-2350)
f42ce17

vstinner · 2017-06-23T13:21:36Z

New changeset d32a059 by Victor Stinner in branch '3.6':
[3.6] bpo-30726: PCbuild _elementtree: remove duplicate defines (bpo-2348) (bpo-2349)
d32a059

ned-deily · 2017-07-08T04:51:39Z

New changeset 5777e79 by Ned Deily (Victor Stinner) in branch '3.6':
[3.6] bpo-30726: PCbuild _elementtree: remove duplicate defines (bpo-2348) (bpo-2349)
5777e79

larryhastings · 2017-07-11T13:18:54Z

I don't quite understand what's happening on this issue. I see that master, 3.6, 3.6, and 2.7 have been upgraded to expat 2.2.0. This issue was created to upgrade CPython to 2.2.0. But the PR against 3.3 and 3.4 upgrade expat to 2.2.1?!

I'm not against this change in principle, I'm just trying to understand why a) it doesn't match the issue, b) why 3.3 and 3.4 are special, c) why we don't upgrade master & 3.6 & 3.5 & 2.7 to expat 2.2.1.

vstinner · 2017-07-11T19:51:10Z

I don't quite understand what's happening on this issue. I see that master, 3.6, 3.6, and 2.7 have been upgraded to expat 2.2.0. This issue was created to upgrade CPython to 2.2.0. But the PR against 3.3 and 3.4 upgrade expat to 2.2.1?!

I'm not against this change in principle, I'm just trying to understand why a) it doesn't match the issue, b) why 3.3 and 3.4 are special, c) why we don't upgrade master & 3.6 & 3.5 & 2.7 to expat 2.2.1.

I upgraded libexpat to 2.2.0 in this issue, and then to libexpat 2.2.1 in bpo-30694.

For 3.3 and 3.4 pull requests, I chose to use this bpo number.

3.3: #2204
3.4: #2203

So these pull requests upgrade directly to 2.2.1.

larryhastings · 2017-07-11T20:06:43Z

Please instead choose to use bpo-30694 for the upgrades of 3.3 and 3.4 to expat 2.2.1. I guess there are historical reasons why the PRs are here, but bpo stands as a historical record; let's not confuse posterity by upgrading to 2.2.1 using a bpo issue talking about--and upgrading four branches to--2.2.0.

vstinner · 2017-07-11T20:43:26Z

Larry: "Please instead choose to use bpo-30694 for the upgrades of 3.3 and 3.4 to expat 2.2.1. I guess there are historical reasons why the PRs are here, but bpo stands as a historical record; let's not confuse posterity by upgrading to 2.2.1 using a bpo issue talking about--and upgrading four branches to--2.2.0."

I just updated the 3.4 PR.

In fact, the PR backports the libexpat 2.2.0 commit *and* then the libexpat 2.2.1 commit. Since it's not possible to create a "patch serie" (in GitHub, it would mean a PR which depends on another PR), I chose to stack the two commits in the same PR and reuse the existing PR to not loose context.

vstinner · 2017-07-11T20:44:19Z

I changed the PR title to mention the two bpo.

larryhastings · 2017-07-12T12:35:47Z

Okay. Closing this bug, because all the branches that are being upgraded to expat 2.2.*0* have already gotten their upgrades. Job done.

The discussions for PRs 2203 and 2204 should move to Issue bpo-30694, which is for the upgrade to expat 2.2.*1*.

larryhastings · 2017-07-12T12:41:37Z

New changeset 71572bb by larryhastings (Victor Stinner) in branch '3.4':
[3.4] bpo-29591, bpo-30694: Upgrade Modules/expat to libexpat 2.2.1 (bpo-2164) (bpo-2203)
71572bb

vstinner · 2017-07-12T12:46:40Z

Okay. Closing this bug, because all the branches that are being upgraded to expat 2.2.*0* have already gotten their upgrades. Job done.

Well, technically 3.3 wasn't upgraded yet:
#2204

larryhastings · 2017-07-12T13:22:40Z

Correct. But technically 3.3 is being upgraded to 2.2.*1*, which is being tracked on--repeating myself here--Issue bpo-30694.

larryhastings · 2017-07-12T13:23:14Z

Here, I'll remove 3.4 and 3.5 from the versions affected. Now everybody can be pedantic!

ned-deily · 2017-07-16T08:48:08Z

New changeset ab90986 by Ned Deily (Victor Stinner) in branch '3.3':
[3.3] bpo-29591, bpo-30694: Upgrade Modules/expat to libexpat 2.2.1 (bpo-2164) (bpo-2204)
ab90986

NatanaelCopa mannequin added 3.7 (EOL) end of life type-security A security issue labels Feb 17, 2017

tiran added the topic-XML label Feb 17, 2017

tiran self-assigned this Feb 17, 2017

NatanaelCopa mannequin changed the title ~~Various security vulnerabilities in bundled expat~~ Various security vulnerabilities in bundled expat (CVE-2016-0718 and CVE-2016-4472) Feb 20, 2017

ned-deily added the release-blocker label Jun 13, 2017

vstinner removed the release-blocker label Jun 14, 2017

vstinner changed the title ~~Various security vulnerabilities in bundled expat (CVE-2016-0718 and CVE-2016-4472)~~ expat 2.2.0: Various security vulnerabilities in bundled expat (CVE-2016-0718 and CVE-2016-4472) Jun 21, 2017

larryhastings closed this as completed Jul 12, 2017

ezio-melotti transferred this issue from another repository Apr 10, 2022

expat 2.2.0: Various security vulnerabilities in bundled expat (CVE-2016-0718 and CVE-2016-4472) #73777

expat 2.2.0: Various security vulnerabilities in bundled expat (CVE-2016-0718 and CVE-2016-4472) #73777

Comments

NatanaelCopa mannequin commented Feb 17, 2017

NatanaelCopa mannequin commented Feb 17, 2017

vstinner commented Feb 17, 2017

vstinner commented Feb 17, 2017

tiran commented Feb 17, 2017

vstinner commented Feb 21, 2017

ned-deily commented Jun 13, 2017

vstinner commented Jun 13, 2017

vstinner commented Jun 14, 2017

vstinner commented Jun 14, 2017

vstinner commented Jun 14, 2017

vstinner commented Jun 14, 2017

vstinner commented Jun 15, 2017

vstinner commented Jun 15, 2017

ned-deily commented Jun 15, 2017

ned-deily commented Jun 18, 2017

jkloth commented Jun 21, 2017

vstinner commented Jun 21, 2017

vstinner commented Jun 21, 2017

jeremykloth mannequin commented Jun 21, 2017

vstinner commented Jun 21, 2017

vstinner commented Jun 23, 2017

vstinner commented Jun 23, 2017

vstinner commented Jun 23, 2017

ned-deily commented Jul 8, 2017

larryhastings commented Jul 11, 2017

vstinner commented Jul 11, 2017

larryhastings commented Jul 11, 2017

vstinner commented Jul 11, 2017

vstinner commented Jul 11, 2017

larryhastings commented Jul 12, 2017

larryhastings commented Jul 12, 2017

vstinner commented Jul 12, 2017

larryhastings commented Jul 12, 2017

larryhastings commented Jul 12, 2017

ned-deily commented Jul 16, 2017