cpython bundles expat in Modules/expat/ and needs to be updated to expat-2.2.0 to fix various security vulnerabilities.

21 June 2016, Expat 2.2.0 released.
Release 2.2.0 includes security & other bug fixes.

Security fixes

CVE-2016-0718 (issue 537)
Fix crash on malformed input

CVE-2016-4472
Improve insufficient fix to CVE-2015-1283 / CVE-2015-2716 introduced with Expat 2.1.1

CVE-2016-5300 (issue 499)
Use more entropy for hash initialization than the original fix to CVE-2012-0876

CVE-2012-6702 (issue 519)
Resolve troublesome internal call to srand that was introduced with Expat 2.1.0 when addressing CVE-2012-0876 (issue 496)

Fix should be applied to all maintained python branches.

> CVE-2012-6702 (issue 519)
> Resolve troublesome internal call to srand that was introduced with Expat 2.1.0 when addressing CVE-2012-0876 (issue 496)

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-6702

Extract of Modules/pyexpat.c:
---
#if ((XML_MAJOR_VERSION >= 2) && (XML_MINOR_VERSION >= 1)) || defined(XML_HAS_SET_HASH_SALT)
    /* This feature was added upstream in libexpat 2.1.0.  Our expat copy
     * has a backport of this feature where we also define XML_HAS_SET_HASH_SALT
     * to indicate that we can still use it. */
    XML_SetHashSalt(self->itself,
                    (unsigned long)_Py_HashSecret.prefix);
#endif
---

Python 2.7, 3.5, 3.6 and 3.7 have this call at least (I didn't check other versions).

You may want to look also at https://pypi.python.org/pypi/defusedxml

CVE-2016-0718 and CVE-2016-4472 might be relevant for Python. CVE-2016-5300 and CVE-2012-6702 are irrelevant. As Victor already pointed out, Python seeds libexpat from a good CPRNG.

I'm working on a new documentation of Python vulnerabilities to help to handle such issue:
http://python-security.readthedocs.io/en/latest/vulnerabilities.html

Note that a duplicate of this issue was opened as Issue30610 and @matrixise was working on a PR there to update the embedded expat to 2.2.0.  Since there are CVE's and a demo crash supplied in Issue30610, it seems to me we need to fix this for 3.6.2rc1 so I'm making this a "release blocker" and delaying the release.  I'm willing to be convinced otherwise.  Christian or Victor, can one of you please follow up on this for the 3.6 branch ASAP?  Thanks!

I upgraded Modules/expat/ to expat 2.2 using attached rebuild_expat_dir.sh script:
https://github.com/python/cpython/pull/2164

TODO: Should be done later in the master branch, once the security fix is handled.

* Drop support for VMS? VMS support removed from Python 3.4. Remove Modules/expat/expat_config.h
* Drop support for the Open Watcom compiler? Compiler not supported by Python. Remove Modules/expat/watcomconfig.h
* Send Python downstream changes to expat upstream?

New changeset 23ec4b57e1359f9c539b8defc317542173ae087e by Victor Stinner in branch 'master':
bpo-29591: Upgrade Modules/expat to libexpat 2.2 (#2164)
https://github.com/python/cpython/commit/23ec4b57e1359f9c539b8defc317542173ae087e

Python 3.3 currently embeds a copy of libexpat 2.1.0, wheras other branches have libexpat 2.1.1:
http://python-security.readthedocs.io/vuln/issue_26556_expat_2.1.1.html

New changeset 86b95370c45dedb8a56c9894372a43681de47a73 by Victor Stinner in branch '3.6':
bpo-29591: Upgrade Modules/expat to libexpat 2.2 (#2164) (#2200)
https://github.com/python/cpython/commit/86b95370c45dedb8a56c9894372a43681de47a73

@Ned Deily: I removed the "release blocker" flag, since I just merged my PR to update libexpat to 2.2 in the Python 3.6 branch.

New changeset 0e4571a68a7f48e8469ef05b04ba3463d3fd82c0 by Victor Stinner in branch '2.7':
bpo-29591: Upgrade Modules/expat to libexpat 2.2 (#2164) (#2202)
https://github.com/python/cpython/commit/0e4571a68a7f48e8469ef05b04ba3463d3fd82c0

New changeset 8c797ed8a0fea5e3162b9415f13e270d4d5d9549 by Victor Stinner in branch '3.5':
bpo-29591: Upgrade Modules/expat to libexpat 2.2 (#2164) (#2201)
https://github.com/python/cpython/commit/8c797ed8a0fea5e3162b9415f13e270d4d5d9549

Thanks, Victor, for seeing this through and thanks, everyone else, for the reviews and assistance.

FYI, expat 2.2.1 has now been released.  See Issue30694 for details.

Added pull_request2355 to address issues from upgrading to Expat 2.2.0 on Windows 2.7

> Added pull_request2355 to address issues from upgrading to Expat 2.2.0 on Windows 2.7

I would prefer to first fix the new vulnerabilities, by upgrading expat to 2.2.1, and then review your change.

=> https://github.com/python/cpython/pull/2312

New changeset ab3b0ade505ce07a3d5ec4fbc991a154242732e6 by Victor Stinner (Jeremy Kloth) in branch '2.7':
bpo-29591: Update VS project files (#2310)
https://github.com/python/cpython/commit/ab3b0ade505ce07a3d5ec4fbc991a154242732e6

Just a note with the PR, the changes to PCbuild\pyexpat.vcxproj and
PCbuild\_elementtree.vcxproj should probably be merged forward as
well.

On Wed, Jun 21, 2017 at 1:14 PM, STINNER Victor <report@bugs.python.org> wrote:
>
> STINNER Victor added the comment:
>
>
> New changeset ab3b0ade505ce07a3d5ec4fbc991a154242732e6 by Victor Stinner (Jeremy Kloth) in branch '2.7':
> bpo-29591: Update VS project files (#2310)
> https://github.com/python/cpython/commit/ab3b0ade505ce07a3d5ec4fbc991a154242732e6
>
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue29591>
> _______________________________________

Jeremy Kloth added the comment:
> Just a note with the PR, the changes to PCbuild\pyexpat.vcxproj and
> PCbuild\_elementtree.vcxproj should probably be merged forward as
> well.

PR 2310. Yes, I agree. Can you please propose patches for master, and
then 3.6 and 3.5, please?

New changeset c8fb58bd7917151e63398587a7fc2126db7c26de by Victor Stinner in branch 'master':
bpo-30726: PCbuild _elementtree: remove duplicate defines (#2348)
https://github.com/python/cpython/commit/c8fb58bd7917151e63398587a7fc2126db7c26de

New changeset f42ce179c8aaa7e211ac4123c58fa3dd9a452004 by Victor Stinner in branch '3.5':
[3.5] bpo-30726: PCbuild _elementtree: remove duplicate defines (#2348) (#2350)
https://github.com/python/cpython/commit/f42ce179c8aaa7e211ac4123c58fa3dd9a452004

New changeset d32a05953130fb5cc2d3c0c9fcb20ad0859353f3 by Victor Stinner in branch '3.6':
[3.6] bpo-30726: PCbuild _elementtree: remove duplicate defines (#2348) (#2349)
https://github.com/python/cpython/commit/d32a05953130fb5cc2d3c0c9fcb20ad0859353f3

New changeset 5777e79ecbd1f2adf36456e09f210608ee221691 by Ned Deily (Victor Stinner) in branch '3.6':
[3.6] bpo-30726: PCbuild _elementtree: remove duplicate defines (#2348) (#2349)
https://github.com/python/cpython/commit/5777e79ecbd1f2adf36456e09f210608ee221691

I don't quite understand what's happening on this issue.  I see that master, 3.6, 3.6, and 2.7 have been upgraded to expat 2.2.0.  This issue was created to upgrade CPython to 2.2.0.  But the PR against 3.3 and 3.4 upgrade expat to 2.2.1?!

I'm not against this change in principle, I'm just trying to understand why a) it doesn't match the issue, b) why 3.3 and 3.4 are special, c) why we don't upgrade master & 3.6 & 3.5 & 2.7 to expat 2.2.1.

> I don't quite understand what's happening on this issue.  I see that master, 3.6, 3.6, and 2.7 have been upgraded to expat 2.2.0.  This issue was created to upgrade CPython to 2.2.0.  But the PR against 3.3 and 3.4 upgrade expat to 2.2.1?!
>
> I'm not against this change in principle, I'm just trying to understand why a) it doesn't match the issue, b) why 3.3 and 3.4 are special, c) why we don't upgrade master & 3.6 & 3.5 & 2.7 to expat 2.2.1.

I upgraded libexpat to 2.2.0 in this issue, and then to libexpat 2.2.1 in bpo-30694.

For 3.3 and 3.4 pull requests, I chose to use this bpo number.

3.3: https://github.com/python/cpython/pull/2204
3.4: https://github.com/python/cpython/pull/2203

So these pull requests upgrade directly to 2.2.1.

Please instead choose to use bpo-30694 for the upgrades of 3.3 and 3.4 to expat 2.2.1.  I guess there are historical reasons why the PRs are here, but bpo stands as a historical record; let's not confuse posterity by upgrading to 2.2.1 using a bpo issue talking about--and upgrading four branches to--2.2.0.

Larry: "Please instead choose to use bpo-30694 for the upgrades of 3.3 and 3.4 to expat 2.2.1.  I guess there are historical reasons why the PRs are here, but bpo stands as a historical record; let's not confuse posterity by upgrading to 2.2.1 using a bpo issue talking about--and upgrading four branches to--2.2.0."

I just updated the 3.4 PR.

In fact, the PR backports the libexpat 2.2.0 commit *and* then the libexpat 2.2.1 commit. Since it's not possible to create a "patch serie" (in GitHub, it would mean a PR which depends on another PR), I chose to stack the two commits in the same PR and reuse the existing PR to not loose context.

I changed the PR title to mention the two bpo.

Okay.  Closing this bug, because all the branches that are being upgraded to expat 2.2.*0* have already gotten their upgrades.  Job done.

The discussions for PRs 2203 and 2204 should move to Issue #30694, which is for the upgrade to expat 2.2.*1*.

New changeset 71572bbe82aa0836c036d44d41c8269ba6a321be by larryhastings (Victor Stinner) in branch '3.4':
[3.4] bpo-29591, bpo-30694: Upgrade Modules/expat to libexpat 2.2.1 (#2164) (#2203)
https://github.com/python/cpython/commit/71572bbe82aa0836c036d44d41c8269ba6a321be

> Okay.  Closing this bug, because all the branches that are being upgraded to expat 2.2.*0* have already gotten their upgrades.  Job done.

Well, technically 3.3 wasn't upgraded yet:
https://github.com/python/cpython/pull/2204

Correct.  But technically 3.3 is being upgraded to 2.2.*1*, which is being tracked on--repeating myself here--Issue #30694.

Here, I'll remove 3.4 and 3.5 from the versions affected.  Now everybody can be pedantic!

New changeset ab90986600ba7dea2aa41e5c1773791070725453 by Ned Deily (Victor Stinner) in branch '3.3':
[3.3] bpo-29591, bpo-30694: Upgrade Modules/expat to libexpat 2.2.1 (#2164) (#2204)
https://github.com/python/cpython/commit/ab90986600ba7dea2aa41e5c1773791070725453