classification
Title: expat parser not xml 1.1 (breaks xmlrpclib)
Type: behavior Stage:
Components: XML Versions: Python 2.7
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: Phil.Daintree, amaury.forgeotdarc, ezio.melotti, loewis, santoso.wijaya, xrg
Priority: normal Keywords:

Created on 2011-04-08 09:33 by xrg, last changed 2012-05-27 08:00 by loewis. This issue is now closed.

Files
File name Uploaded Description Edit
expat-test.py xrg, 2011-04-08 09:33 Test of expat compliance to xml 1.1
Messages (12)
msg133301 - (view) Author: Panos Christeas (xrg) Date: 2011-04-08 09:33
The expat library (in C level) is not xml 1.1 compliant, meaning that
it won't accept characters \x01-\x08,\x0b,\x0c and \x0e-\x1f .
At the same time, ElementTree (or custom XML creation, such as in xmlrpclib.py:694) allow these characters to pass through. They will get blocked on the receiving side.
Since 2.7, the expat library is the default parser for xml-rpc, so it
this is a regression, IMHO. According to the network principal, we should
accept these characters gracefully.

The attached test script demonstrates that we're not xml 1.1 compliant (but instead enforce the more strict 1.0 rule)

References:
http://bugs.python.org/issue5166
http://en.wikipedia.org/wiki/Valid_characters_in_XML
msg161341 - (view) Author: Phil Daintree (Phil.Daintree) Date: 2012-05-22 09:03
Another example - the following xml returned and displayed from verbose mode:

<?xml version="1.0"?>
<methodResponse>
<params>
<param>
<value><array>
<data>
<value><string>0001</string></value>
<value><string>001</string></value>
<value><string>002</string></value>
<value><string>100</string></value>
<value><string>121213</string></value>
<value><string>123456</string></value>
<value><string>291</string></value>
<value><string>321654</string></value>
<value><string>580</string></value>
<value><string>ABS</string></value>
<value><string>ACTIVE</string></value>
<value><string>AIRCON</string></value>
<value><string>ALIEJA</string></value>
<value><string>AMP</string></value>
<value><string>ASSETS</string></value>
<value><string>BAKE</string></value>
<value><string>BRACE</string></value>
<value><string>BYC</string></value>
<value><string>CARRO</string></value>
<value><string>CARTON</string></value>
<value><string>CO</string></value>
<value><string>COMPS</string></value>
<value><string>CULOIL</string></value>
<value><string>DECOR</string></value>
<value><string>DVD</string></value>
<value><string>E</string></value>
<value><string>FOOD</string></value>
<value><string>HDD</string></value>
<value><string>INF</string></value>
<value><string>LAB</string></value>
<value><string>LINER</string></value>
<value><string>LL</string></value>
<value><string>MCNBI</string></value>
<value><string>MEDS</string></value>
<value><string>MODEL1</string></value>
<value><string>NEM</string></value>
<value><string>PE&#195;\x87AS</string></value>
<value><string>PENS</string></value>
<value><string>PHONE</string></value>
<value><string>PLANT</string></value>
<value><string>PRJCTR</string></value>
<value><string>PROD</string></value>
<value><string>SERV</string></value>
<value><string>SOCKS</string></value>
<value><string>SS</string></value>
<value><string>SW</string></value>
<value><string>TACON</string></value>
<value><string>TEST12</string></value>
<value><string>VEGTAB</string></value>
<value><string>ZFR</string></value>
</data>
</array></value>
</param>
</params>
</methodResponse>

will not parse with the error:

File "/usr/lib/python2.7/xmlrpclib.py", line 557, in feed
    self._parser.Parse(data, 0)
xml.parsers.expat.ExpatError: not well-formed (invalid token): line 43, column 23

the following unicode characters  on that line are the trouble:

<value><string>PE&#195;\x87AS</string></value>
msg161342 - (view) Author: Phil Daintree (Phil.Daintree) Date: 2012-05-22 09:05
The xml parses happily at http://www.w3schools.com/xml/xml_validator.asp
msg161346 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2012-05-22 12:09
In sample above, is "\x87" one character, or 4 ascii characters?
msg161491 - (view) Author: Phil Daintree (Phil.Daintree) Date: 2012-05-24 09:03
The field in question contains the utf-8 text: PEÇAS
msg161503 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2012-05-24 12:50
Yes, but where does this data come from? how did you feed it to the parser? And this does not relate to xml 1.1.

BTW, I found this page about XML 1.1:
http://www.cafeconleche.org/books/effectivexml/chapters/03.html

"""
Everything you need to know about XML 1.1 can be summed up in two rules:
- Don't use it.
- (For experts only) If you speak Mongolian, Yi, Cambodian, Amharic, Dhivehi, Burmese or a very few other languages and you want to write your markup (not your text but your markup) in these languages, then you can set the version attribute of the XML declaration to 1.1. Otherwise, refer to rule 1.
"""
msg161520 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-05-24 15:58
This has nothing to do with XML 1.1 (so closing this report as "won't fix").

The UTF-8 text that you present works very well:

>>> p=xml.parsers.expat.ParserCreate(encoding="utf-8")
>>> p.Parse("<x>\xc3\x87</x", 1)
1

The character LATIN CAPITAL LETTER C WITH CEDILLA is definitely supported in XML 1.0, so there is no need for XML 1.1 here.

If this still fails to parse for you, it may be because the input is actually different, e.g.

>>> p=xml.parsers.expat.ParserCreate(encoding="utf-8")
>>> p.Parse("<x>&#195;\x87</x>", 1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1, column 9

I.e. the input might contain the character &, #, 1, 9, 5, ;, and \x87. That is ill-formed UTF-8, and the parser is right to choke on it. Even if it was declared as XML 1.1, it will still be ill-formed, because it still would be invalid UTF-8.
msg161521 - (view) Author: Panos Christeas (xrg) Date: 2012-05-24 16:07
I'm reopening the bug, as your last comment does not cover the initial report. We are not talking about invalid UTF8 here, but legal low-ASCII values.
msg161697 - (view) Author: Phil Daintree (Phil.Daintree) Date: 2012-05-27 07:09
Well maybe this should be a different bug as it is clearly not xml 1.1 related as the linue in the xml gives away :-)

<?xml version="1.0"?>

To repeat the bug ... using the webERP demo data

#!/usr/bin/env python

import xmlrpclib

x_server = xmlrpclib.Server('http://www.weberp.org/weberp/api/api_xml-rpc.php',verbose=True)
#Get the stock items defined in the demo webERP installation
StockList = x_server.weberp.xmlrpc_SearchStockItems('discontinued','0','admin','weberp')

if StockList[0]==0:
	for StockID in StockList[1]:
		print str(StockID)

The webERP xml-rpc server uses XMLRPC for PHP http://phpxmlrpc.sourceforge.net/
msg161699 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-05-27 07:38
Phil: it seems you have hijacked the bug report. Don't do that. If you want to report a bug, please create a new bug report. Structure it as follows:

1. this is what I did
2. this is what happened
3. this is what should have happened instead.
msg161700 - (view) Author: Phil Daintree (Phil.Daintree) Date: 2012-05-27 07:52
or for less data...

#!/usr/bin/env python

import xmlrpclib

x_server = xmlrpclib.Server('http://www.weberp.org/weberp/api/api_xml-rpc.php',verbose=True)
#Get the stock items defined in the webERP installation
StockList = x_server.weberp.xmlrpc_SearchStockItems('units','cm','admin','weberp')

if StockList[0]==0:
	for StockID in StockList[1]:
		print str(StockID)
msg161701 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-05-27 08:00
Panos: you are right. The original issue still exists.

However, it is not a bug in Python, but a in the expat library. So I am now closing this report as out-of-scope for Python.

There is a bug report open on expat requesting support for XML 1.1, see

http://sourceforge.net/tracker/?func=detail&atid=110127&aid=891265&group_id=10127

This bug report is open since 2004. I see little hope that expat will support XML 1.1 within the next five years.

I also fail to see the regression: expat has never supported XML 1.1.
xmlrpclib always used expat, at least since Python 2.0. In any case, this report is about expat, not xmlrpclib, so any possible regression in xmlrpclib should be reported separately.
History
Date User Action Args
2012-05-27 08:00:59loewissetstatus: open -> closed
resolution: wont fix
2012-05-27 08:00:51loewissetmessages: + msg161701
2012-05-27 07:52:45Phil.Daintreesetmessages: + msg161700
2012-05-27 07:38:33loewissetmessages: + msg161699
2012-05-27 07:09:43Phil.Daintreesetmessages: + msg161697
2012-05-24 16:07:54xrgsetstatus: closed -> open
resolution: wont fix -> (no value)
messages: + msg161521
2012-05-24 15:58:06loewissetstatus: open -> closed
resolution: wont fix
messages: + msg161520
2012-05-24 12:50:31amaury.forgeotdarcsetmessages: + msg161503
2012-05-24 09:03:17Phil.Daintreesetmessages: + msg161491
2012-05-22 12:09:20amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg161346
2012-05-22 09:05:05Phil.Daintreesetmessages: + msg161342
2012-05-22 09:03:37Phil.Daintreesetnosy: + Phil.Daintree
messages: + msg161341
2011-04-08 22:46:21ezio.melottisetnosy: + ezio.melotti
2011-04-08 18:10:27santoso.wijayasetnosy: + santoso.wijaya
2011-04-08 12:53:20pitrousetnosy: + loewis
2011-04-08 09:33:02xrgcreate