classification
Title: Support Apache extensions to XML-RPC in xmlrpclib
Type: enhancement Stage:
Components: XML Versions: Python 3.2, Python 3.3
process
Status: closed Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Adam.Bielański, amaury.forgeotdarc, bhargav, bquinlan, bra, eric.araujo, loewis, orsenthil
Priority: low Keywords:

Created on 2010-05-22 19:55 by bra, last changed 2012-10-21 09:17 by bquinlan. This issue is now closed.

Files
File name Uploaded Description Edit
dumps_with_namespace.py Adam.Bielański, 2010-11-16 13:44 Augmented xmlrpclib.dumps() function
Messages (12)
msg106322 - (view) Author: Attila Nagy (bra) Date: 2010-05-22 19:55
When talking to an Apache XML-RPC library based application via python 2.6.5 xmlrpclib, I get this exception:
Traceback (most recent call last):
  File "prb.py", line 4, in <module>
    proxy.catv.getEndpointIdByIp('1.1.1.1')
  File "/tmp/Python-2.6.5/Lib/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/tmp/Python-2.6.5/Lib/xmlrpclib.py", line 1491, in __request
    verbose=self.__verbose
  File "/tmp/Python-2.6.5/Lib/xmlrpclib.py", line 1253, in request
    return self._parse_response(h.getfile(), sock)
  File "/tmp/Python-2.6.5/Lib/xmlrpclib.py", line 1389, in _parse_response
    p.feed(response)
  File "/tmp/Python-2.6.5/Lib/xmlrpclib.py", line 601, in feed
    self._parser.Parse(data, 0)
  File "/tmp/Python-2.6.5/Lib/xmlrpclib.py", line 868, in end
    return f(self, join(self._data, ""))
  File "/tmp/Python-2.6.5/Lib/xmlrpclib.py", line 935, in end_struct
    dict[_stringify(items[i])] = items[i+1]
IndexError: list index out of range

The exception is caused by the XML response, which includes a value with "ex:i8" type. According to this: http://ws.apache.org/xmlrpc/types.html, there are a lot more types, which are not understood by python's xmlrpclib.

It's easy to fix the above by adding "ex:i8" to the list of end_int dispatcher:
    def end_int(self, data):
        self.append(int(data))
        self._value = 0
    dispatch["i4"] = end_int
    dispatch["i8"] = end_int
    dispatch["ex:i8"] = end_int
    dispatch["int"] = end_int

This makes the error disappear (and the program to work).
Of course, it would be nice to support all other types as well (in both directions).
msg106714 - (view) Author: Brian Quinlan (bquinlan) * (Python committer) Date: 2010-05-29 09:57
A few notes:
1. these types are *not* part of the XML-RPC specification, they are Apache extensions
2. these types seem designed to accommodate Java
3. some of these types would be very possible to accommodate e.g. ex:serializable which contains a serialized Java object
msg111312 - (view) Author: Bhargav (bhargav) Date: 2010-07-23 13:15
I am seeing the same exception when the server returns ex:nil, even when I enable allow_none.

The library only accepts 'nil' not 'ex:nil'

As a workaround, I have added dispatch["ex:nil"] to my local library (which is obviously not a solution)
msg111442 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-07-24 10:14
Can somebody please point to a current version of the Apache documentation? The link in msg106322 does not work anymore.

I'm tempted to close this as invalid: Apache is clearly violating the XML-RPC specification, so the bug is theirs.

As a work-around, adding this to dispatch in your own application seems like a good solution to me.
msg111443 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-07-24 10:19
I just noticed that I only need to strip the comma from the URL.

So I now see it's not an Apache bug, but explicitly marked as an extension. So I'm reclassifying this as a feature request. As a new feature, it can only go into 3.2.

Attila, Bhargav: is either of you interested in providing a patch?
msg111490 - (view) Author: Bhargav (bhargav) Date: 2010-07-24 15:49
What I do is not a possible solution, because some other machine might add e:nil, what we need is XML namespace parsing.

Also I can't really use my solution, as it makes my client non-portable, I cant move it from my dev machine to production at all.

This is just a workaround so that I am not stuck at ex:nil for my development.

I am not an expert in xmlrpc nor in python to really contribute a patch, but if I could really do think of something, I will contribute it.

Having said that, I know for sure the server works for Java clients and I have heard it works for .Net.

I know this is not a standard, but a widely used and published extension, and hence I think we should atleast port it back to the 2.7 branch as not everyone has moved to python 3 yet.
msg111513 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-07-24 23:26
Adding it to 2.7 is really out of the question. Assuming somebody would implement (which might not happen in the next three years or so), it can only go into 3.x, so you'ld have to port your code to 3.x to use it.
msg117886 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2010-10-02 16:58
It's easy enough to subclass the Transport type and add custom types to the dispatcher object, see the script below.
Attila, Bhargav, is this solution acceptable to you?


from xmlrpclib import Transport, ServerProxy

class MyTransport(Transport):
    def getparser(self):
        parser, unmarshaller = Transport.getparser(self)

        # Get the class attribute, clone it
        dispatch = unmarshaller.dispatch.copy()
        # and store it on the instance
        unmarshaller.dispatch = dispatch

        # Now we can add custom types
        dispatch["ex:i8"] = dispatch["int"]

        return parser, unmarshaller

uri = "http://time.xmlrpc.com/RPC2"
server = ServerProxy(uri, transport=MyTransport(use_datetime=True))
print server.currentTime.getCurrentTime()
msg121287 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-11-16 13:17
An addition: It was reported in #10425 that None values are incorrectly serialized as <value><nil/></value>, instead of just <nil/>, or maybe <{namespace prefix for extensions defined by Apache}:nil/>.
msg121290 - (view) Author: Adam Bielański (Adam.Bielański) * Date: 2010-11-16 13:44
To make example provided by Amaury complete I'll add that in order to support both ways you also need to replace xmlrpclib.dumps() with code from attached file.

Changes to original xmlrpclib.dumps() function are outlined with comments. In short - it adds namespace declarations to methodCall and methodResponse elements and also makes Marshaller produce <ex:nil/> instead of <nil/>
msg121463 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-11-18 15:37
> An addition: It was reported in #10425 that None values are
> incorrectly serialized as <value><nil/></value>, instead of just
> <nil/>, or maybe <{namespace prefix for extensions defined by
> Apache}:nil/>.

This is not incorrect, but follows the specification of the nil
element, see, for example

http://en.wikipedia.org/wiki/XML-RPC
msg173437 - (view) Author: Brian Quinlan (bquinlan) * (Python committer) Date: 2012-10-21 09:17
I'm closing this since the filer hasn't specified exactly what they want.
History
Date User Action Args
2012-10-21 09:17:45bquinlansetstatus: open -> closed

messages: + msg173437
2010-11-18 15:37:52loewissetmessages: + msg121463
2010-11-16 13:44:17Adam.Bielańskisetfiles: + dumps_with_namespace.py

messages: + msg121290
2010-11-16 13:17:37eric.araujosetnosy: + orsenthil, eric.araujo, Adam.Bielański
messages: + msg121287
2010-11-16 13:13:25eric.araujolinkissue10425 superseder
2010-10-02 16:58:15amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg117886
2010-07-24 23:26:31loewissetmessages: + msg111513
2010-07-24 15:49:05bhargavsetmessages: + msg111490
2010-07-24 10:19:07loewissetpriority: normal -> low
versions: + Python 3.3, - Python 2.6, Python 3.1, Python 2.7
title: xmlrpclib compatibility issues with Apache XML-RPC library -> Support Apache extensions to XML-RPC in xmlrpclib
messages: + msg111443

type: behavior -> enhancement
2010-07-24 10:14:31loewissetmessages: + msg111442
2010-07-23 13:15:35bhargavsetnosy: + bhargav
messages: + msg111312
2010-05-29 09:57:35bquinlansetnosy: + bquinlan
messages: + msg106714
2010-05-23 11:56:52pitrousetnosy: + loewis

versions: + Python 3.1, Python 2.7, Python 3.2
2010-05-22 19:55:12bracreate