classification
Title: Update embedded copy of expat - fix security & crash issues
Type: security Stage:
Components: Extension Modules Versions: Python 3.2
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, Jim.Jewett, amaury.forgeotdarc, barry, benjamin.peterson, dmalcolm, georg.brandl, gregory.p.smith, pitrou, python-dev
Priority: release blocker Keywords:

Created on 2012-03-16 21:28 by gregory.p.smith, last changed 2013-02-03 19:01 by gregory.p.smith. This issue is now closed.

Messages (19)
msg156087 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2012-03-16 21:28
As pointed out in #14234, our embedded copy of expat used by pyexpat for xml parsing in Modules/expat/ is out of date.  There have been many fixes to expat that we have not applied including a few potential crash and security fixes.

We should upgrade it wholesale to the latest version for 3.3.

Someone should also audit expat changes to see if there are security fixes for expat that should be backported to 2.6/2.7/3.1/3.2 as platforms without a system expat such as Windows (and 2.6 and 3.1) will contain those problems.

I am marking this a release blocker for 3.3 to ensure expat is updated before then.  I would *not* hold up the existing round of release candidates for this, the next security+bugfix updates can contain these changes.
msg156092 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2012-03-16 22:35
What would be awesome is if we could just kill embedded versions of libraries like expat.
msg156093 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2012-03-16 22:37
Indeed!  How do we do that on windows and osx where these may not exist outside of Python?

We already require a set of external dependency libraries on windows, could we just add expat to the list?
msg156094 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-03-16 22:37
> What would be awesome is if we could just kill embedded versions of libraries like expat.

It practically wouldn't change much, since we still bundle them as part
of Windows binaries.
msg156095 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2012-03-16 22:42
It is much less effort for us to simply take a new version of an external library and recompile rather than consider it part of our code that shouldn't change within a release and manually deal with patching it and cherry picking patches onto it.
msg163208 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-06-19 20:11
Greg, are you still going to update expat in time for 3.3?  Otherwise this doesn't block 3.3 anymore.
msg163556 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-06-23 08:59
Deferring for beta1 at least.
msg163622 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2012-06-23 15:08
Thanks. I still intend to get to this.
 On Jun 23, 2012 1:59 AM, "Georg Brandl" <report@bugs.python.org> wrote:

>
> Georg Brandl <georg@python.org> added the comment:
>
> Deferring for beta1 at least.
>
> ----------
> priority: release blocker -> deferred blocker
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue14340>
> _______________________________________
>
msg164099 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-06-26 20:50
Moving back to blocker for beta2.
msg165471 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-07-14 21:12
New changeset e4dc8be9a72f by Gregory P. Smith in branch 'default':
Update the embedded copy of the expat XML parser to 2.1.0.  It brings
http://hg.python.org/cpython/rev/e4dc8be9a72f
msg165472 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2012-07-14 21:18
Updated in 3.3 for beta2.

I'll leave it up to release managers to decide if they want to apply these updates for future 2.7 and 3.2 releases (trivial, just be sure to keep our one local modification adding the "#define XML_HAS_SET_HASH_SALT" to expat.h when you do it).

I'm leaving it as a release blocker for 2.7.4 and 3.2.4 so that they see it.
msg165473 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2012-07-14 21:39
Reasons why it is a good idea to apply this change to 2.7.4 and 3.2.4:

* Memory leak in poolGrow (CVE-2012-1148)
* Resource leak in readfilemap.c (CVE-2012-1147)
* Buffer over-read and crash in big2_toUtf8 (CVE-2009-3560)
* Parser crash with special UTF-8 sequences (CVE-2009-3270)
* Dangling positionPtr after error (2855609) - http://sourceforge.net/tracker/?func=detail&aid=2855609&group_id=10127&atid=110127 - Specifically reported by a pyexpat user.
* Unitialized memory returned from XML_Parse (3206497) - http://sourceforge.net/tracker/?func=detail&aid=3206497&group_id=10127&atid=110127

The features 2.1.0 adds over 2.0.x are not exposed to pyexpat or Python users.
msg180260 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2013-01-19 20:53
If you could apply this to 2.7, that'd be great.
msg181172 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-02-02 16:10
New changeset c73a1f96dd9b by Gregory P. Smith in branch '2.7':
Update the embedded copy of the expat XML parser to 2.1.0.  It brings
http://hg.python.org/cpython/rev/c73a1f96dd9b
msg181174 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-02-02 16:17
Then I guess there is no reason not to put it in 3.2.4.
msg181193 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-02-02 18:32
Then I guess there is no reason not to put it in 3.2.4.
msg181249 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-02-03 08:55
Greg, if you are fine please apply to 3.2 or indicate if it is enough to apply the same patch as on 3.3/default.
msg181284 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-02-03 18:59
New changeset d2f6f63e73af by Gregory P. Smith in branch '3.2':
Update the embedded copy of the expat XML parser to 2.1.0.  It brings
http://hg.python.org/cpython/rev/d2f6f63e73af
msg181285 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2013-02-03 19:01
done.

btw, it looks like benjamin.peterson did it for 2.7 yesterday morning but when 'hg graft' is used to apply a change from another branch the roundup notification mentions the original commit's author, not the person who did the push of the graft.
History
Date User Action Args
2013-02-03 19:01:02gregory.p.smithsetstatus: open -> closed
resolution: fixed
messages: + msg181285
2013-02-03 18:59:05python-devsetmessages: + msg181284
2013-02-03 08:55:41georg.brandlsetmessages: + msg181249
2013-02-02 18:32:32georg.brandlsetmessages: + msg181193
2013-02-02 16:17:20georg.brandlsetmessages: + msg181174
2013-02-02 16:10:52benjamin.petersonsetversions: - Python 2.7
2013-02-02 16:10:18python-devsetmessages: + msg181172
2013-01-19 20:53:54benjamin.petersonsetmessages: + msg180260
2012-07-14 21:39:33gregory.p.smithsetmessages: + msg165473
2012-07-14 21:18:31gregory.p.smithsetmessages: + msg165472
versions: + Python 2.7, Python 3.2, - Python 3.3
2012-07-14 21:12:50python-devsetnosy: + python-dev
messages: + msg165471
2012-06-26 20:50:52georg.brandlsetpriority: deferred blocker -> release blocker

messages: + msg164099
2012-06-23 15:08:19gregory.p.smithsetmessages: + msg163622
2012-06-23 08:59:51georg.brandlsetpriority: release blocker -> deferred blocker

messages: + msg163556
2012-06-19 20:11:35georg.brandlsetmessages: + msg163208
2012-03-16 22:42:34gregory.p.smithsetmessages: + msg156095
2012-03-16 22:37:57pitrousetmessages: + msg156094
2012-03-16 22:37:26gregory.p.smithsetmessages: + msg156093
2012-03-16 22:35:23benjamin.petersonsetmessages: + msg156092
2012-03-16 21:28:26gregory.p.smithcreate