Thank you!

https://github.com/libexpat/libexpat/blob/97a4840578693a346e79302909b67d97492e1880/expat/Changes#L6-L35

BTW there are test regressions with expat 2.4.5, apparently due to some test snippets now being rejected as invalid XML:

======================================================================
ERROR: test_issue3151 (test.test_xml_etree.BugsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mgorny/git/cpython/Lib/xml/etree/ElementTree.py", line 1718, in feed
    self.parser.Parse(data, False)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
xml.parsers.expat.ExpatError: syntax error: line 1, column 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mgorny/git/cpython/Lib/test/test_xml_etree.py", line 2196, in test_issue3151
    e = ET.XML('<prefix:localname xmlns:prefix="${stuff}"/>')
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/etree/ElementTree.py", line 1347, in XML
    parser.feed(text)
    ^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/etree/ElementTree.py", line 1720, in feed
    self._raiseerror(v)
    ^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/etree/ElementTree.py", line 1627, in _raiseerror
    raise err
    ^^^^^^^^^
xml.etree.ElementTree.ParseError: syntax error: line 1, column 0


======================================================================
ERROR: testEncodings (test.test_minidom.MinidomTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mgorny/git/cpython/Lib/test/test_minidom.py", line 1150, in testEncodings
    self.assertRaises(UnicodeDecodeError, parseString,
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/unittest/case.py", line 734, in assertRaises
    return context.handle('assertRaises', args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/unittest/case.py", line 218, in handle
    callable_obj(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/dom/minidom.py", line 1998, in parseString
    return expatbuilder.parseString(string)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/dom/expatbuilder.py", line 925, in parseString
    return builder.parseString(string)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/dom/expatbuilder.py", line 223, in parseString
    parser.Parse(string, True)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^
xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1, column 5

======================================================================
ERROR: testExceptionOnSpacesInXMLNSValue (test.test_minidom.MinidomTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mgorny/git/cpython/Lib/test/test_minidom.py", line 1613, in testExceptionOnSpacesInXMLNSValue
    parseString('<element xmlns:abc="http:abc.com/de f g/hi/j k"><abc:foo /></element>')
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/dom/minidom.py", line 1998, in parseString
    return expatbuilder.parseString(string)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/dom/expatbuilder.py", line 925, in parseString
    return builder.parseString(string)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgorny/git/cpython/Lib/xml/dom/expatbuilder.py", line 223, in parseString
    parser.Parse(string, True)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^
xml.parsers.expat.ExpatError: syntax error: line 1, column 0

Hi Michal,


TL;DR would be:

- There is a regression but none of these test fails are related.

- There will be a release Expat 2.4.6 with the regression fixed later today.

- The 3 failing tests need (small) adjustments to Expat 2.4.5
  and these fails are not considered bugs in Expat.

I will demo a fix to 2 of the 3 test fails below:


# git diff -U1 | cat
diff --git a/Lib/test/test_minidom.py b/Lib/test/test_minidom.py
index 1663b1f114..38cea97a97 100644
--- a/Lib/test/test_minidom.py
+++ b/Lib/test/test_minidom.py
@@ -12,2 +12,3 @@
 from xml.dom.minidom import getDOMImplementation
+from xml.parsers.expat import ExpatError
 
@@ -1149,4 +1150,6 @@ def testEncodings(self):
         # of crashing
-        self.assertRaises(UnicodeDecodeError, parseString,
-                b'<fran\xe7ais>Comment \xe7a va ? Tr\xe8s bien ?</fran\xe7ais>')
+        self.assertRaises(ExpatError, parseString,
+                b'<fran\xe7ais></fran\xe7ais>')
+        self.assertRaises(ExpatError, parseString,
+                b'<franais>Comment \xe7a va ? Tr\xe8s bien ?</franais>')
 
@@ -1611,3 +1614,3 @@ def testEmptyXMLNSValue(self):
     def testExceptionOnSpacesInXMLNSValue(self):
-        with self.assertRaisesRegex(ValueError, 'Unsupported syntax'):
+        with self.assertRaisesRegex(ExpatError, "syntax error"):
             parseString('<element xmlns:abc="http:abc.com/de f g/hi/j k"><abc:foo /></element>')
             

For the third test, the key is that the closing curly brace is used as the
namespace separator in line 3660…

  self->parser = EXPAT(ParserCreate_MM)(encoding, &ExpatMemoryHandler, "}");
  
…in file Modules/_elementtree.c (which is okay but part of the test fail).

Best



Sebastian

Could you make a PR to fix the test failures?  I suppose that could speed things up and if not, I'd at least have something to pull into Gentoo.

I'm busy with the release upstream at the moment.  I'll see what I can do.

I have created a dedicated ticket bpo-46811 now, test suite pull request upcoming.

New changeset 1935e1cc284942bec8006287c939e295e1a7bf13 by Dong-hee Na in branch 'main':
bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487)
https://github.com/python/cpython/commit/1935e1cc284942bec8006287c939e295e1a7bf13

New changeset 4955a9ed14c681ed835bc8902a9db0bcc728bdee by Miss Islington (bot) in branch '3.10':
bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487)
https://github.com/python/cpython/commit/4955a9ed14c681ed835bc8902a9db0bcc728bdee

New changeset 87cebb1e69758aa8b79f8e15187b976d62cba36a by Miss Islington (bot) in branch '3.9':
bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487)
https://github.com/python/cpython/commit/87cebb1e69758aa8b79f8e15187b976d62cba36a

New changeset 15d7594d9974cfef10e65cbb01161168c42abe9d by Miss Islington (bot) in branch '3.7':
bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487) (GH-31521)
https://github.com/python/cpython/commit/15d7594d9974cfef10e65cbb01161168c42abe9d

New changeset eb6c840a2414dc057ffcfbb5ad68d6253c8dd57c by Miss Islington (bot) in branch '3.8':
bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487) (GH-31520)
https://github.com/python/cpython/commit/eb6c840a2414dc057ffcfbb5ad68d6253c8dd57c

On PyPy, the test `test_issue3151` in `test_xml_etree.py` is failing with libexpat 2.4.6. I think the problem is connected to instantiation of the `XMLParser()` with `parser = expat.ParserCreate(encoding, "}")` where `"}"` is not a valid URI character. In any case, due to libexpat issue 577, https://github.com/libexpat/libexpat/pull/577 they will be releasing a new version 2.4.7 soon.

Hi mattip,

at the core the problem is not the use of non-URI character "}" for a namespace separator but the use of non-URI character "}" in a namespace URI.  test_issue3151 is mistaken (meaning that non-URI characters in URIs are malformed XML) and the test has been removed in CPython pull request https://github.com/python/cpython/pull/31453/files .  Expat pull request https://github.com/libexpat/libexpat/pull/577 is related but it's about URI characters not about non-URI ones, so it does not change anything about test_issue3151 in PyPy.  Does that make sense?

Best, Sebastian

> [T]he test has been removed in CPython pull request https://github.com/python/cpython/pull/31453/files

Thanks, I missed that. Makes sense.