classification
Title: Update embeded copy of libexpat from 2.2.1 to 2.2.3
Type: security Stage: resolved
Components: Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, gregory.p.smith, larry, ned.deily, serhiy.storchaka, vstinner
Priority: normal Keywords:

Created on 2017-07-17 14:18 by vstinner, last changed 2017-09-25 08:51 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
cpython_rebuild_expat_dir.sh vstinner, 2017-08-16 14:35
Pull Requests
URL Status Linked Edit
PR 3106 merged vstinner, 2017-08-16 14:34
PR 3143 merged vstinner, 2017-08-18 21:49
PR 3144 closed vstinner, 2017-08-18 21:51
PR 3145 merged vstinner, 2017-08-18 21:55
PR 3352 merged vstinner, 2017-09-05 18:37
PR 3353 merged vstinner, 2017-09-05 18:41
PR 3354 merged vstinner, 2017-09-05 18:44
Messages (18)
msg298525 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-17 14:18
libexpat released a new version 2.2.2 which seems to contain 2 or 3 security fixes. I'm not sure that Python is affected by these bugs.

https://github.com/libexpat/libexpat/blob/R_2_2_2/expat/Changes#L5

Release 2.2.2 Wed July 12 2017
        Security fixes:
             #43  Protect against compilation without any source of high
                    quality entropy enabled, e.g. with CMake build system;
                    commit ff0207e6076e9828e536b8d9cd45c9c92069b895
             #60  Windows with _UNICODE:
                    Unintended use of LoadLibraryW with a non-wide string
                    resulted in failure to load advapi32.dll and degradation
                    in quality of used entropy when compiled with _UNICODE for
                    Windows; you can launch existing binaries with
                    EXPAT_ENTROPY_DEBUG=1 in the environment to inspect the
                    quality of entropy used during runtime; commits
                    * 95b95032f907ef1cd17ee7a9a1768010a825d61d
                    * 73a5a2e9c081f49f2d775cf7ced864158b68dc80
   [MOX-006]      Fix non-NULL parser parameter validation in XML_Parse;
                    resulted in NULL dereference, previously;
                    commit ac256dafdffc9622ab0dc2c62fcecb0dfcfa71fe

        Bug fixes:
             #69  Fix improper use of unsigned long long integer literals

        Other changes:
             #73  Start requiring a C99 compiler
             #49  Fix "==" Bashism in configure script
             #50  Fix too eager getrandom detection for Debian GNU/kFreeBSD
             #52    and macOS
             #51  Address lack of stdint.h in Visual Studio 2003 to 2008
             #58  Address compile warnings
             #68  Fix "./buildconf.sh && ./configure" for some versions
                    of Dash for /bin/sh
             #72  CMake: Ease use of Expat in context of a parent project
                    with multipe CMakeLists.txt files
             #72  CMake: Resolve mistaken executable permissions
             #76  Address compile warning with -DNDEBUG (not recommended!)
             #77  Address compile warning about macro redefinition

        Special thanks to:
            Alexander Bluhm
            Ben Boeckel
            Cătălin Răceanu
            Kerin Millar
            László Böszörményi
            S. P. Zeidler
            Segev Finer
            Václav Slavík
            Victor Stinner
            Viktor Szakats
                 and
Radically Open Security

--

Previous issue for expat 2.2.1: issue #30694.
msg298528 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-17 14:24
> #51  Address lack of stdint.h in Visual Studio 2003 to 2008

FYI this change only impacts Python 2.7, since Python 3.3 and newer requires Visual Studio 2010 or newer, and I already backported (cherry-picked) this specific commit in Python 2.7:
https://github.com/python/cpython/pull/2312/commits

> #58  Address compile warnings

That's my small contribution, so coming from CPython :-)
https://github.com/libexpat/libexpat/pull/58

> #76  Address compile warning with -DNDEBUG (not recommended!)

Nice contributions from Segev Finer, coming from CPython ;-)
https://github.com/libexpat/libexpat/issues/76

> #77  Address compile warning about macro redefinition

Another contribution of Segev Finer, already fixed downstream (in Python):
https://github.com/libexpat/libexpat/pull/77
msg298529 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-17 14:28
About the 3 security fixes (is the last change a security fix?).

"""
             #43  Protect against compilation without any source of high
                    quality entropy enabled, e.g. with CMake build system;
                    commit ff0207e6076e9828e536b8d9cd45c9c92069b895
"""

Since Python uses its own entropy source, I don't think that this change impacts us.

https://github.com/libexpat/libexpat/commit/ff0207e6076e9828e536b8d9cd45c9c92069b895


"""
             #60  Windows with _UNICODE:
                    Unintended use of LoadLibraryW with a non-wide string
                    resulted in failure to load advapi32.dll and degradation
                    in quality of used entropy when compiled with _UNICODE for
                    Windows; you can launch existing binaries with
                    EXPAT_ENTROPY_DEBUG=1 in the environment to inspect the
                    quality of entropy used during runtime; commits
                    * 95b95032f907ef1cd17ee7a9a1768010a825d61d
                    * 73a5a2e9c081f49f2d775cf7ced864158b68dc80
"""

I don't understand the consequence of this specific bug.

https://github.com/libexpat/libexpat/commit/95b95032f907ef1cd17ee7a9a1768010a825d61d
https://github.com/libexpat/libexpat/commit/73a5a2e9c081f49f2d775cf7ced864158b68dc80


"""
   [MOX-006]      Fix non-NULL parser parameter validation in XML_Parse;
                    resulted in NULL dereference, previously;
                    commit ac256dafdffc9622ab0dc2c62fcecb0dfcfa71fe
"""

I'm not sure that it's possible to call XML_Parse() with NULL in Python.

https://github.com/libexpat/libexpat/commit/ac256dafdffc9622ab0dc2c62fcecb0dfcfa71fe
msg300365 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-16 14:21
Expat 2.2.3 was released:

Release 2.2.3 Wed August 2 2017
        Security fixes:
             #82  CVE-2017-11742 -- Windows: Fix DLL hijacking vulnerability
                    using Steve Holme's LoadLibrary wrapper for/of cURL

        Bug fixes:
             #85  Fix a dangling pointer issue related to realloc

        Other changes:
                  Increase code coverage
             #91  Linux: Allow getrandom to fail if nonblocking pool has not
                    yet been initialized and read /dev/urandom then, instead.
                    This is in line with what recent Python does.
             #81  Pre-10.7/Lion macOS: Support entropy from arc4random
             #86  Check that a UTF-16 encoding in an XML declaration has the
                    right endianness
        #4 #5 #7  Recover correctly when some reallocations fail
                  Repair "./configure && make" for systems without any
                    provider of high quality entropy
                    and try reading /dev/urandom on those
                  Ensure that user-defined character encodings have converter
                    functions when they are needed
                  Fix mis-leading description of argument -c in xmlwf.1
                  Rely on macro HAVE_ARC4RANDOM_BUF (rather than __CloudABI__)
                    for CloudABI
            #100  Fix use of SIPHASH_MAIN in siphash.h
             #23  Test suite: Fix memory leaks
                  Version info bumped from 7:4:6 to 7:5:6

        Special thanks to:
            Chanho Park
            Joe Orton
            Pascal Cuoq
            Rhodri James
            Simon McVittie
            Vadim Zeitlin
            Viktor Szakats
                 and
Core Infrastructure Initiative
msg300367 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-16 14:30
Previous update: bpo-30694.
msg300368 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-16 14:35
cpython_rebuild_expat_dir.sh: Script used to update Modules/expat/ to 2.2.3. The script now uses the libexpat Git repository. Previously, I used tarballs.
msg300369 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-16 14:38
> #82  CVE-2017-11742 -- Windows: Fix DLL hijacking vulnerability using Steve Holme's LoadLibrary wrapper for/of cURL

https://github.com/libexpat/libexpat/issues/82

I don't think that this bug affects Python since Python sets a hash secret.
msg300414 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-08-17 12:44
Could the updating script be added into the CPython repository?
msg300437 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-17 15:24
If libexpat is upgraded in Python 2.7, the new Modules/expat/loadlibrary.c should also be added to PC/VS9.0/ project files, as I did for PCbuild.

Note: PC/VS7.1/ and PC/VS8.0/ are likely broken and don't need to be updated, right?
msg300535 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-18 21:43
New changeset 93d0cb58b4da2a88c56f472c6c19491cc7a390df by Victor Stinner in branch 'master':
bpo-30947: Update libexpat from 2.2.1 to 2.2.3 (#3106)
https://github.com/python/cpython/commit/93d0cb58b4da2a88c56f472c6c19491cc7a390df
msg300547 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-18 23:06
New changeset 83e37e16f3065086d721d4e62a3788e01db3431c by Victor Stinner in branch '3.6':
bpo-30947: Update libexpat from 2.2.1 to 2.2.3 (#3106) (#3143)
https://github.com/python/cpython/commit/83e37e16f3065086d721d4e62a3788e01db3431c
msg300548 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-18 23:06
New changeset ec4ab09b7c0b5070bdb27351f979cbecc4636245 by Victor Stinner in branch '2.7':
bpo-30947: Update libexpat from 2.2.1 to 2.2.3 (#3106) (#3145)
https://github.com/python/cpython/commit/ec4ab09b7c0b5070bdb27351f979cbecc4636245
msg301269 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-04 21:36
Expat 2.2.3 has a bug: see bpo-31170 :-(
msg301423 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-09-06 00:57
New changeset 297516ea509c72d8ebed3a9b3ce200f023aca0b7 by Ned Deily (Victor Stinner) in branch '3.3':
[3.3] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 (#3352)
https://github.com/python/cpython/commit/297516ea509c72d8ebed3a9b3ce200f023aca0b7
msg302834 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2017-09-24 08:04
New changeset 86a713cb0c110b6798ca7f9e630fc511ee0a4028 by larryhastings (Victor Stinner) in branch '3.4':
[3.4][Security] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 (#3353)
https://github.com/python/cpython/commit/86a713cb0c110b6798ca7f9e630fc511ee0a4028
msg302899 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2017-09-25 00:58
New changeset f2492bb6aae061aea47e21fc7e56b7ab9bfdf543 by larryhastings (Victor Stinner) in branch '3.5':
[3.5][Security] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 (#3354)
https://github.com/python/cpython/commit/f2492bb6aae061aea47e21fc7e56b7ab9bfdf543
msg302924 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-25 08:26
libexpat has been upgraded from 2.2.1 to 2.2.4 in 2.7, 3.4, 3.5, 3.6 and master branches.
msg302929 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-09-25 08:51
And in 3.3.
History
Date User Action Args
2017-09-25 08:51:10serhiy.storchakasetmessages: + msg302929
2017-09-25 08:26:11vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg302924

stage: resolved
2017-09-25 00:58:36larrysetmessages: + msg302899
2017-09-24 08:04:56larrysetnosy: + larry
messages: + msg302834
2017-09-06 00:57:39ned.deilysetmessages: + msg301423
2017-09-05 18:44:44vstinnersetpull_requests: + pull_request3368
2017-09-05 18:41:15vstinnersetpull_requests: + pull_request3366
2017-09-05 18:37:37vstinnersetpull_requests: + pull_request3364
2017-09-04 21:36:26vstinnersetmessages: + msg301269
2017-08-18 23:06:47vstinnersetmessages: + msg300548
2017-08-18 23:06:30vstinnersetmessages: + msg300547
2017-08-18 21:55:42vstinnersetpull_requests: + pull_request3180
2017-08-18 21:51:52vstinnersetpull_requests: + pull_request3179
2017-08-18 21:49:50vstinnersetpull_requests: + pull_request3178
2017-08-18 21:43:56vstinnersetmessages: + msg300535
2017-08-17 15:24:56vstinnersetmessages: + msg300437
2017-08-17 12:44:44serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg300414
2017-08-16 14:38:43vstinnersetmessages: + msg300369
2017-08-16 14:35:08vstinnersetfiles: + cpython_rebuild_expat_dir.sh

messages: + msg300368
2017-08-16 14:34:32vstinnersetpull_requests: + pull_request3145
2017-08-16 14:30:37vstinnersetmessages: + msg300367
2017-08-16 14:22:54vstinnersettitle: Update embeded copy of libexpat to 2.2.2 -> Update embeded copy of libexpat from 2.2.1 to 2.2.3
2017-08-16 14:21:19vstinnersetmessages: + msg300365
2017-07-17 14:28:24vstinnersetmessages: + msg298529
2017-07-17 14:24:45vstinnersetmessages: + msg298528
2017-07-17 14:18:15vstinnercreate