classification
Title: xmlrpclib raises when trying to convert an int to string when unicode is available
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: Nathan Williams, Zsolt Endreffy, loewis, python-dev, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2016-04-28 01:49 by Nathan Williams, last changed 2017-02-09 07:40 by Zsolt Endreffy. This issue is now closed.

Files
File name Uploaded Description Edit
pybug.txt Nathan Williams, 2016-04-28 05:51 server response
xmlrpc_unsupported.patch serhiy.storchaka, 2016-04-28 19:46 review
Messages (10)
msg264407 - (view) Author: Nathan Williams (Nathan Williams) Date: 2016-04-28 01:49
I am using xmlrpclib against an internal xmlrpc server.
One of the responses returns integer values, and it raises an exception in "_stringify"

The code for _stringify is (xmlrpclib.py:180 in python2.7):

if unicode:
    def _stringify(string):
        # convert to 7-bit ascii if possible
        try:
            return string.encode("ascii")
        except UnicodeError:
            return string
else:
    def _stringify(string):
        return string


So when "unicode" is available, .encode is called on the parameter (which are the returned objects from the server) which fails for ints.

Without the unicode path it works fine, proven with the following monkey-patch:
xmlrpclib._stringify = lambda s: s

I am using the above patch as a workaround, but a fix to the library should be straightforward, simply checking for AttributeError in the except clause would solve it while retaining the existing functionality.


The traceback:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib/python2.6/xmlrpclib.py", line 1253, in request
    return self._parse_response(h.getfile(), sock)
  File "/usr/lib/python2.6/xmlrpclib.py", line 1387, in _parse_response
    p.feed(response)
  File "/usr/lib/python2.6/xmlrpclib.py", line 601, in feed
    self._parser.Parse(data, 0)
  File "/usr/lib/python2.6/xmlrpclib.py", line 868, in end
    return f(self, join(self._data, ""))
  File "/usr/lib/python2.6/xmlrpclib.py", line 935, in end_struct
    dict[_stringify(items[i])] = items[i+1]
  File "/usr/lib/python2.6/xmlrpclib.py", line 176, in _stringify
    return string.encode("ascii")
AttributeError: 'int' object has no attribute 'encode'
msg264411 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-04-28 04:39
Could you provide a response of your server (pass verbose=True to ServerProxy)?
msg264416 - (view) Author: Nathan Williams (Nathan Williams) Date: 2016-04-28 05:51
I have attached the response.
As it is coming from our UMS, I had to redact a few values, but that shouldn't matter.

For reference, they were the host name of my email address, and the hashes of passwords etc.
Our UMS is a bit too chatty!
msg264420 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-04-28 07:08
Thank you. Your server produces a response containing nonstandard type tag "ex:nil" [1].

    <member><name>authority</name><value>ROLE_USER</value></member>
    <member><name>fromDate</name><value><ex:nil/></value></member>
    <member><name>userId</name><value><i4>15</i4></value></member>

xmlrpclib is unable to handle this tag, and error handling is poor. xmlrpclib can handle only standard types and "nil" [2].

What is worse, xmlrpclib silently ignores unsupported tag and interprets following name "userId" as a value of the name "fromDate", and following userId's value 15 as next name and so forth. Your workaround doesn't help since the result is totally broken. You get values as keys and keys as values.

What can we do with this issue?

1. Add better handling of unsupported tags. There is a bug, xmlrpc should detect this case and fail instead of producing unreliable result. This is applicable to 2.7.

2. Add support of some Apache extension tags. This is new feature and can be added in 3.6 only. Not all extension tags can be supported, "ex:serializable" is Java specific. This is separate issue.

Martin, do you agree?

[1] http://ws.apache.org/xmlrpc/types.html
[2] http://web.archive.org/web/20130120074804/http://ontosys.com/xml-rpc/extensions.php
msg264436 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-04-28 16:50
Minimal reproducer:

>>> xmlrpclib.loads('<params><param><value><struct><member><name>a</name><value><ex:nil/></value></member><member><name>b</name><value><ex:nil/></value></member></struct></value></param></params>')
(({'a': 'b'},), None)

The workaround for your particular case, Nathan:

    xmlrpclib.Unmarshaller.dispatch['ex:nil'] = xmlrpclib.Unmarshaller.dispatch['nil']
msg264444 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-04-28 19:46
Proposed patch makes xmlrpc unmarshaller to be more strong and raise ValueError instead of returning incorrect value when encounters with unsupported value type. The unmarshaller still skips unknown tags silently if they are occurred outside of the "value" element.
msg264505 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-04-29 15:34
Opened issue26885 for adding support of "ex:nil" and other types.
msg264612 - (view) Author: Nathan Williams (Nathan Williams) Date: 2016-05-01 22:45
Serhiy, that workaround worked for my needs, thanks.
msg264795 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-05-04 08:28
New changeset 0d015f6aba8b by Serhiy Storchaka in branch '3.5':
Issue #26873: xmlrpc now raises ResponseError on unsupported type tags
https://hg.python.org/cpython/rev/0d015f6aba8b

New changeset 8f7cb3b171f3 by Serhiy Storchaka in branch 'default':
Issue #26873: xmlrpc now raises ResponseError on unsupported type tags
https://hg.python.org/cpython/rev/8f7cb3b171f3

New changeset 7050c9fc1f72 by Serhiy Storchaka in branch '2.7':
Issue #26873: xmlrpclib now raises ResponseError on unsupported type tags
https://hg.python.org/cpython/rev/7050c9fc1f72
msg287385 - (view) Author: Zsolt Endreffy (Zsolt Endreffy) Date: 2017-02-09 07:40
I think this patch breaks compatibility between python 2.7 versions. Our rpc server has 2.7.10 Python version, and sends back tuples as responses (first value is a boolean, second is a string). If we connect with a computer, which has 2.7.11 or earlier Python, then it is working flawlessly. When we connect with Python 2.7.12 or later, then it sends this error:
ResponseError: ResponseError("unknown tag u'tuple'",)

As far as I understand, the xmlrpclib should be able to marshal through tuples.
History
Date User Action Args
2017-02-09 07:40:51Zsolt Endreffysetnosy: + Zsolt Endreffy
messages: + msg287385
2016-05-04 08:29:17serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2016-05-04 08:28:41python-devsetnosy: + python-dev
messages: + msg264795
2016-05-01 22:45:25Nathan Williamssetmessages: + msg264612
2016-04-29 15:34:39serhiy.storchakasetmessages: + msg264505
2016-04-29 15:33:20serhiy.storchakalinkissue26885 dependencies
2016-04-28 19:46:45serhiy.storchakasetfiles: + xmlrpc_unsupported.patch
versions: + Python 3.5, Python 3.6
messages: + msg264444

keywords: + patch
stage: needs patch -> patch review
2016-04-28 16:50:21serhiy.storchakasetmessages: + msg264436
2016-04-28 07:08:02serhiy.storchakasetnosy: + loewis
messages: + msg264420

assignee: serhiy.storchaka
type: crash -> behavior
stage: needs patch
2016-04-28 05:51:10Nathan Williamssetfiles: + pybug.txt

messages: + msg264416
2016-04-28 04:39:11serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg264411
2016-04-28 01:49:22Nathan Williamscreate