This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Please update bundled libexpat to 2.4.6 with security fixes (5 CVEs)
Type: security Stage: resolved
Components: XML Versions: Python 3.11, Python 3.10, Python 3.9, Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: corona10 Nosy List: corona10, lukasz.langa, mattip, mgorny, miss-islington, ned.deily, sping
Priority: normal Keywords: patch

Created on 2022-02-18 23:36 by sping, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 31487 merged corona10, 2022-02-22 03:15
PR 31487 merged corona10, 2022-02-22 03:15
PR 31518 merged miss-islington, 2022-02-23 01:40
PR 31519 merged miss-islington, 2022-02-23 01:40
PR 31520 merged miss-islington, 2022-02-23 01:40
PR 31521 merged miss-islington, 2022-02-23 01:40
Messages (14)
msg413517 - (view) Author: (sping) * Date: 2022-02-18 23:36
Thank you!

https://github.com/libexpat/libexpat/blob/97a4840578693a346e79302909b67d97492e1880/expat/Changes#L6-L35
msg413587 - (view) Author: Michał Górny (mgorny) * Date: 2022-02-20 13:57
BTW there are test regressions with expat 2.4.5, apparently due to some test snippets now being rejected as invalid XML:

======================================================================
ERROR: test_issue3151 (test.test_xml_etree.BugsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mgorny/git/cpython/Lib/xml/etree/ElementTree.py", line 1718, in feed
    self.parser.Parse(data, False)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
xml.parsers.expat.ExpatError: syntax error: line 1, column 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mgorny/git/cpython/Lib/test/test_xml_etree.py", line 2196, in test_issue3151
    e = ET.XML('<prefix:localname xmlns:prefix="${stuff}"/>')
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/etree/ElementTree.py", line 1347, in XML
    parser.feed(text)
    ^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/etree/ElementTree.py", line 1720, in feed
    self._raiseerror(v)
    ^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/etree/ElementTree.py", line 1627, in _raiseerror
    raise err
    ^^^^^^^^^
xml.etree.ElementTree.ParseError: syntax error: line 1, column 0


======================================================================
ERROR: testEncodings (test.test_minidom.MinidomTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mgorny/git/cpython/Lib/test/test_minidom.py", line 1150, in testEncodings
    self.assertRaises(UnicodeDecodeError, parseString,
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/unittest/case.py", line 734, in assertRaises
    return context.handle('assertRaises', args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/unittest/case.py", line 218, in handle
    callable_obj(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/dom/minidom.py", line 1998, in parseString
    return expatbuilder.parseString(string)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/dom/expatbuilder.py", line 925, in parseString
    return builder.parseString(string)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/dom/expatbuilder.py", line 223, in parseString
    parser.Parse(string, True)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^
xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1, column 5

======================================================================
ERROR: testExceptionOnSpacesInXMLNSValue (test.test_minidom.MinidomTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mgorny/git/cpython/Lib/test/test_minidom.py", line 1613, in testExceptionOnSpacesInXMLNSValue
    parseString('<element xmlns:abc="http:abc.com/de f g/hi/j k"><abc:foo /></element>')
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/dom/minidom.py", line 1998, in parseString
    return expatbuilder.parseString(string)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/dom/expatbuilder.py", line 925, in parseString
    return builder.parseString(string)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/dom/expatbuilder.py", line 223, in parseString
    parser.Parse(string, True)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^
xml.parsers.expat.ExpatError: syntax error: line 1, column 0
msg413596 - (view) Author: (sping) * Date: 2022-02-20 16:20
Hi Michal,


TL;DR would be:

- There is a regression but none of these test fails are related.

- There will be a release Expat 2.4.6 with the regression fixed later today.

- The 3 failing tests need (small) adjustments to Expat 2.4.5
  and these fails are not considered bugs in Expat.

I will demo a fix to 2 of the 3 test fails below:


# git diff -U1 | cat
diff --git a/Lib/test/test_minidom.py b/Lib/test/test_minidom.py
index 1663b1f114..38cea97a97 100644
--- a/Lib/test/test_minidom.py
+++ b/Lib/test/test_minidom.py
@@ -12,2 +12,3 @@
 from xml.dom.minidom import getDOMImplementation
+from xml.parsers.expat import ExpatError
 
@@ -1149,4 +1150,6 @@ def testEncodings(self):
         # of crashing
-        self.assertRaises(UnicodeDecodeError, parseString,
-                b'<fran\xe7ais>Comment \xe7a va ? Tr\xe8s bien ?</fran\xe7ais>')
+        self.assertRaises(ExpatError, parseString,
+                b'<fran\xe7ais></fran\xe7ais>')
+        self.assertRaises(ExpatError, parseString,
+                b'<franais>Comment \xe7a va ? Tr\xe8s bien ?</franais>')
 
@@ -1611,3 +1614,3 @@ def testEmptyXMLNSValue(self):
     def testExceptionOnSpacesInXMLNSValue(self):
-        with self.assertRaisesRegex(ValueError, 'Unsupported syntax'):
+        with self.assertRaisesRegex(ExpatError, "syntax error"):
             parseString('<element xmlns:abc="http:abc.com/de f g/hi/j k"><abc:foo /></element>')
             

For the third test, the key is that the closing curly brace is used as the
namespace separator in line 3660…

  self->parser = EXPAT(ParserCreate_MM)(encoding, &ExpatMemoryHandler, "}");
  
…in file Modules/_elementtree.c (which is okay but part of the test fail).

Best



Sebastian
msg413597 - (view) Author: Michał Górny (mgorny) * Date: 2022-02-20 16:34
Could you make a PR to fix the test failures?  I suppose that could speed things up and if not, I'd at least have something to pull into Gentoo.
msg413598 - (view) Author: (sping) * Date: 2022-02-20 16:41
I'm busy with the release upstream at the moment.  I'll see what I can do.
msg413606 - (view) Author: (sping) * Date: 2022-02-20 19:55
I have created a dedicated ticket bpo-46811 now, test suite pull request upcoming.
msg413762 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2022-02-23 01:40
New changeset 1935e1cc284942bec8006287c939e295e1a7bf13 by Dong-hee Na in branch 'main':
bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487)
https://github.com/python/cpython/commit/1935e1cc284942bec8006287c939e295e1a7bf13
msg413765 - (view) Author: miss-islington (miss-islington) Date: 2022-02-23 02:50
New changeset 4955a9ed14c681ed835bc8902a9db0bcc728bdee by Miss Islington (bot) in branch '3.10':
bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487)
https://github.com/python/cpython/commit/4955a9ed14c681ed835bc8902a9db0bcc728bdee
msg413766 - (view) Author: miss-islington (miss-islington) Date: 2022-02-23 02:51
New changeset 87cebb1e69758aa8b79f8e15187b976d62cba36a by Miss Islington (bot) in branch '3.9':
bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487)
https://github.com/python/cpython/commit/87cebb1e69758aa8b79f8e15187b976d62cba36a
msg413862 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2022-02-23 21:51
New changeset 15d7594d9974cfef10e65cbb01161168c42abe9d by Miss Islington (bot) in branch '3.7':
bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487) (GH-31521)
https://github.com/python/cpython/commit/15d7594d9974cfef10e65cbb01161168c42abe9d
msg414333 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2022-03-02 09:19
New changeset eb6c840a2414dc057ffcfbb5ad68d6253c8dd57c by Miss Islington (bot) in branch '3.8':
bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487) (GH-31520)
https://github.com/python/cpython/commit/eb6c840a2414dc057ffcfbb5ad68d6253c8dd57c
msg414525 - (view) Author: mattip (mattip) * Date: 2022-03-04 13:31
On PyPy, the test `test_issue3151` in `test_xml_etree.py` is failing with libexpat 2.4.6. I think the problem is connected to instantiation of the `XMLParser()` with `parser = expat.ParserCreate(encoding, "}")` where `"}"` is not a valid URI character. In any case, due to libexpat issue 577, https://github.com/libexpat/libexpat/pull/577 they will be releasing a new version 2.4.7 soon.
msg414537 - (view) Author: (sping) * Date: 2022-03-04 16:46
Hi mattip,

at the core the problem is not the use of non-URI character "}" for a namespace separator but the use of non-URI character "}" in a namespace URI.  test_issue3151 is mistaken (meaning that non-URI characters in URIs are malformed XML) and the test has been removed in CPython pull request https://github.com/python/cpython/pull/31453/files .  Expat pull request https://github.com/libexpat/libexpat/pull/577 is related but it's about URI characters not about non-URI ones, so it does not change anything about test_issue3151 in PyPy.  Does that make sense?

Best, Sebastian
msg414587 - (view) Author: mattip (mattip) * Date: 2022-03-05 16:57
> [T]he test has been removed in CPython pull request https://github.com/python/cpython/pull/31453/files

Thanks, I missed that. Makes sense.
History
Date User Action Args
2022-04-11 14:59:56adminsetgithub: 90950
2022-03-05 16:57:31mattipsetmessages: + msg414587
2022-03-04 16:46:05spingsetmessages: + msg414537
2022-03-04 13:31:33mattipsetnosy: + mattip
messages: + msg414525
2022-03-02 09:20:33corona10setstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2022-03-02 09:19:50lukasz.langasetnosy: + lukasz.langa
messages: + msg414333
2022-02-23 21:51:16ned.deilysetnosy: + ned.deily
messages: + msg413862
2022-02-23 02:51:08miss-islingtonsetmessages: + msg413766
2022-02-23 02:50:44miss-islingtonsetmessages: + msg413765
2022-02-23 01:40:50miss-islingtonsetpull_requests: + pull_request29648
2022-02-23 01:40:46miss-islingtonsetpull_requests: + pull_request29647
2022-02-23 01:40:45corona10setmessages: + msg413762
2022-02-23 01:40:41miss-islingtonsetpull_requests: + pull_request29646
2022-02-23 01:40:37miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request29645
2022-02-22 03:15:59corona10setpull_requests: + pull_request29616
2022-02-22 03:15:59corona10setpull_requests: + pull_request29615
2022-02-22 03:09:25corona10setpull_requests: - pull_request29614
2022-02-22 03:08:52corona10setkeywords: + patch
stage: patch review
pull_requests: + pull_request29614
2022-02-22 02:31:16corona10setassignee: corona10

nosy: + corona10
2022-02-20 19:55:27spingsetmessages: + msg413606
2022-02-20 16:41:57spingsetmessages: + msg413598
2022-02-20 16:34:15mgornysetmessages: + msg413597
2022-02-20 16:20:27spingsetmessages: + msg413596
title: Please update bundled libexpat to 2.4.5 with security fixes (5 CVEs) -> Please update bundled libexpat to 2.4.6 with security fixes (5 CVEs)
2022-02-20 13:57:50mgornysetnosy: + mgorny
messages: + msg413587
2022-02-18 23:36:28spingcreate