Please update bundled libexpat to 2.4.6 with security fixes (5 CVEs) #90950

hartwork · 2022-02-18T23:36:29Z

BPO	46794
Nosy	@ned-deily, @ambv, @mgorny, @mattip, @hartwork, @corona10, @miss-islington
PRs	bpo-46794: Bump up the libexpat version into 2.4.6 #31487 bpo-46794: Bump up the libexpat version into 2.4.6 #31487 [3.10] bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487) #31518 [3.9] bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487) #31519 [3.8] bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487) #31520 [3.7] bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487) #31521

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/corona10'
closed_at = <Date 2022-03-02.09:20:33.019>
created_at = <Date 2022-02-18.23:36:28.648>
labels = ['type-security', '3.8', '3.9', '3.10', '3.11', 'expert-XML', '3.7']
title = 'Please update bundled libexpat to 2.4.6 with security fixes (5 CVEs)'
updated_at = <Date 2022-03-05.16:57:31.480>
user = 'https://github.com/hartwork'

bugs.python.org fields:

activity = <Date 2022-03-05.16:57:31.480>
actor = 'mattip'
assignee = 'corona10'
closed = True
closed_date = <Date 2022-03-02.09:20:33.019>
closer = 'corona10'
components = ['XML']
creation = <Date 2022-02-18.23:36:28.648>
creator = 'sping'
dependencies = []
files = []
hgrepos = []
issue_num = 46794
keywords = ['patch']
message_count = 14.0
messages = ['413517', '413587', '413596', '413597', '413598', '413606', '413762', '413765', '413766', '413862', '414333', '414525', '414537', '414587']
nosy_count = 7.0
nosy_names = ['ned.deily', 'lukasz.langa', 'mgorny', 'mattip', 'sping', 'corona10', 'miss-islington']
pr_nums = ['31487', '31487', '31518', '31519', '31520', '31521']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'security'
url = 'https://bugs.python.org/issue46794'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']

hartwork · 2022-02-18T23:36:29Z

Thank you!

https://github.com/libexpat/libexpat/blob/97a4840578693a346e79302909b67d97492e1880/expat/Changes#L6-L35

mgorny · 2022-02-20T13:57:51Z

BTW there are test regressions with expat 2.4.5, apparently due to some test snippets now being rejected as invalid XML:

======================================================================
ERROR: test_issue3151 (test.test_xml_etree.BugsTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/mgorny/git/cpython/Lib/xml/etree/ElementTree.py", line 1718, in feed
    self.parser.Parse(data, False)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
xml.parsers.expat.ExpatError: syntax error: line 1, column 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mgorny/git/cpython/Lib/test/test_xml_etree.py", line 2196, in test_issue3151
    e = ET.XML('<prefix:localname xmlns:prefix="${stuff}"/>')
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/etree/ElementTree.py", line 1347, in XML
    parser.feed(text)
    ^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/etree/ElementTree.py", line 1720, in feed
    self._raiseerror(v)
    ^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/etree/ElementTree.py", line 1627, in _raiseerror
    raise err
    ^^^^^^^^^
xml.etree.ElementTree.ParseError: syntax error: line 1, column 0

======================================================================
ERROR: testEncodings (test.test_minidom.MinidomTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/mgorny/git/cpython/Lib/test/test_minidom.py", line 1150, in testEncodings
    self.assertRaises(UnicodeDecodeError, parseString,
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/unittest/case.py", line 734, in assertRaises
    return context.handle('assertRaises', args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/unittest/case.py", line 218, in handle
    callable_obj(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/dom/minidom.py", line 1998, in parseString
    return expatbuilder.parseString(string)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/dom/expatbuilder.py", line 925, in parseString
    return builder.parseString(string)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/dom/expatbuilder.py", line 223, in parseString
    parser.Parse(string, True)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^
xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1, column 5

======================================================================
ERROR: testExceptionOnSpacesInXMLNSValue (test.test_minidom.MinidomTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/mgorny/git/cpython/Lib/test/test_minidom.py", line 1613, in testExceptionOnSpacesInXMLNSValue
    parseString('<element xmlns:abc="http:abc.com/de f g/hi/j k"><abc:foo /></element>')
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/dom/minidom.py", line 1998, in parseString
    return expatbuilder.parseString(string)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/dom/expatbuilder.py", line 925, in parseString
    return builder.parseString(string)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/dom/expatbuilder.py", line 223, in parseString
    parser.Parse(string, True)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^
xml.parsers.expat.ExpatError: syntax error: line 1, column 0

hartwork · 2022-02-20T16:20:27Z

Hi Michal,

TL;DR would be:

There is a regression but none of these test fails are related.
There will be a release Expat 2.4.6 with the regression fixed later today.
The 3 failing tests need (small) adjustments to Expat 2.4.5
and these fails are not considered bugs in Expat.

I will demo a fix to 2 of the 3 test fails below:

# git diff -U1 | cat
diff --git a/Lib/test/test_minidom.py b/Lib/test/test_minidom.py
index 1663b1f114..38cea97a97 100644
--- a/Lib/test/test_minidom.py
+++ b/Lib/test/test_minidom.py
@@ -12,2 +12,3 @@
 from xml.dom.minidom import getDOMImplementation
+from xml.parsers.expat import ExpatError
 
@@ -1149,4 +1150,6 @@ def testEncodings(self):
         # of crashing
-        self.assertRaises(UnicodeDecodeError, parseString,
-                b'<fran\xe7ais>Comment \xe7a va ? Tr\xe8s bien ?</fran\xe7ais>')
+        self.assertRaises(ExpatError, parseString,
+                b'<fran\xe7ais></fran\xe7ais>')
+        self.assertRaises(ExpatError, parseString,
+                b'<franais>Comment \xe7a va ? Tr\xe8s bien ?</franais>')
 
@@ -1611,3 +1614,3 @@ def testEmptyXMLNSValue(self):
     def testExceptionOnSpacesInXMLNSValue(self):
-        with self.assertRaisesRegex(ValueError, 'Unsupported syntax'):
+        with self.assertRaisesRegex(ExpatError, "syntax error"):
             parseString('<element xmlns:abc="http:abc.com/de f g/hi/j k"><abc:foo /></element>')

For the third test, the key is that the closing curly brace is used as the
namespace separator in line 3660…

self->parser = EXPAT(ParserCreate_MM)(encoding, &ExpatMemoryHandler, "}");

…in file Modules/_elementtree.c (which is okay but part of the test fail).

Best

Sebastian

mgorny · 2022-02-20T16:34:16Z

Could you make a PR to fix the test failures? I suppose that could speed things up and if not, I'd at least have something to pull into Gentoo.

hartwork · 2022-02-20T16:41:58Z

I'm busy with the release upstream at the moment. I'll see what I can do.

hartwork · 2022-02-20T19:55:27Z

I have created a dedicated ticket bpo-46811 now, test suite pull request upcoming.

corona10 · 2022-02-23T01:40:46Z

New changeset 1935e1c by Dong-hee Na in branch 'main':
bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487)
1935e1c

miss-islington · 2022-02-23T02:50:45Z

New changeset 4955a9e by Miss Islington (bot) in branch '3.10':
bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487)
4955a9e

miss-islington · 2022-02-23T02:51:08Z

New changeset 87cebb1 by Miss Islington (bot) in branch '3.9':
bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487)
87cebb1

ned-deily · 2022-02-23T21:51:17Z

New changeset 15d7594 by Miss Islington (bot) in branch '3.7':
bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487) (GH-31521)
15d7594

ambv · 2022-03-02T09:19:50Z

New changeset eb6c840 by Miss Islington (bot) in branch '3.8':
bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487) (GH-31520)
eb6c840

mattip · 2022-03-04T13:31:34Z

On PyPy, the test test_issue3151 in test_xml_etree.py is failing with libexpat 2.4.6. I think the problem is connected to instantiation of the XMLParser() with parser = expat.ParserCreate(encoding, "}") where "}" is not a valid URI character. In any case, due to libexpat issue 577, libexpat/libexpat#577 they will be releasing a new version 2.4.7 soon.

hartwork · 2022-03-04T16:46:06Z

Hi mattip,

at the core the problem is not the use of non-URI character "}" for a namespace separator but the use of non-URI character "}" in a namespace URI. test_issue3151 is mistaken (meaning that non-URI characters in URIs are malformed XML) and the test has been removed in CPython pull request https://github.com/python/cpython/pull/31453/files . Expat pull request libexpat/libexpat#577 is related but it's about URI characters not about non-URI ones, so it does not change anything about test_issue3151 in PyPy. Does that make sense?

Best, Sebastian

mattip · 2022-03-05T16:57:31Z

[T]he test has been removed in CPython pull request https://github.com/python/cpython/pull/31453/files

Thanks, I missed that. Makes sense.

hartwork mannequin added 3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes topic-XML type-security A security issue labels Feb 18, 2022

hartwork mannequin changed the title ~~Please update bundled libexpat to 2.4.5 with security fixes (5 CVEs)~~ Please update bundled libexpat to 2.4.6 with security fixes (5 CVEs) Feb 20, 2022

corona10 assigned corona10 Feb 22, 2022

corona10 closed this as completed Mar 2, 2022

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Please update bundled libexpat to 2.4.6 with security fixes (5 CVEs) #90950

Please update bundled libexpat to 2.4.6 with security fixes (5 CVEs) #90950

hartwork mannequin commented Feb 18, 2022

hartwork mannequin commented Feb 18, 2022

mgorny mannequin commented Feb 20, 2022

hartwork mannequin commented Feb 20, 2022

mgorny mannequin commented Feb 20, 2022

hartwork mannequin commented Feb 20, 2022

hartwork mannequin commented Feb 20, 2022

corona10 commented Feb 23, 2022

miss-islington commented Feb 23, 2022

miss-islington commented Feb 23, 2022

ned-deily commented Feb 23, 2022

ambv commented Mar 2, 2022

mattip commented Mar 4, 2022

hartwork mannequin commented Mar 4, 2022

mattip commented Mar 5, 2022

Please update bundled libexpat to 2.4.6 with security fixes (5 CVEs) #90950

Please update bundled libexpat to 2.4.6 with security fixes (5 CVEs) #90950

Comments

hartwork mannequin commented Feb 18, 2022

hartwork mannequin commented Feb 18, 2022

mgorny mannequin commented Feb 20, 2022

hartwork mannequin commented Feb 20, 2022

mgorny mannequin commented Feb 20, 2022

hartwork mannequin commented Feb 20, 2022

hartwork mannequin commented Feb 20, 2022

corona10 commented Feb 23, 2022

miss-islington commented Feb 23, 2022

miss-islington commented Feb 23, 2022

ned-deily commented Feb 23, 2022

ambv commented Mar 2, 2022

mattip commented Mar 4, 2022

hartwork mannequin commented Mar 4, 2022

mattip commented Mar 5, 2022