I am sorry if the same issue was already considered and rejected for
some reason; quick search did not show any traces of it.

What I am going to write here is just a proof of concept, but if the
idea of the patch is acceptable, I am eager to prepare proper patches
for lib, documentation and so on.

So, my aim was to make xmlrpc module frendlier to those who want to
extend its functionality locally in their own projects. Currently there
are several problems with it:
1. Marshaller, unmarshaller and parser cannot be substituted by custom
versions easily.
2. Custom version of marshaller will look ugly (i.e., because it may
need to call Marshaller.__dump)
3. Transport, parser and unmarshaller are coupled now, though they are
completely independent.

My patch seem to eliminate these problems (see attach). Briefly, it
contains the following changes:
1. Transport, parser and unmarshaller are decoupled
2. Custom masrshaller, parser and unmarshaller classes can be passed to
client and server classes as parameters
3. Made Marshaller class easier to extend:
- __dump() renamed to _dump()
- added _add_memo() and _del_memo() (and hid .memo field)
- memo is now a set() (instead of dictionary with None values)
- dispatch table was made private

Results:
(good) This patch does not invalidate any part of documentation (but it
needs to be extended, according to new marshaller/unmarshaller/parser
parameters)
(good) test_xmlrpc still passes (with one little change to it, patch is
attached)
(good) Extension is easy now - see xmlrpc_overload.py as an example
(added bytes(), tuple() and dict() with non-string keys support)
(bad) Programs which use exported, but undocumented parts of xmlrpc can
break (though most of them can be easily fixed)

Just a few comments on the code itself:

    if type_ in self.__dispatch.keys():
should be:
    if type_ in self.__dispatch:

Previously, error reporting of recursive data stated the type of the offending value; with this patch, this hint is lost (see _add_memo)

Caching of bound methods in local variables is a common optimization (dump=self._dump); why did you remove it everywhere?

Why Marshaller.dispatch was renamed to __dispatch but Unmarshaller.dispatch stays the same? (btw, why the double underscore?)

Just a few comments on the code itself:

    if type_ in self.__dispatch.keys():
should be:
    if type_ in self.__dispatch:

Previously, error reporting of recursive data stated the type of the offending value; with this patch, this hint is lost (see _add_memo)

Caching of bound methods in local variables is a common optimization (dump=self._dump); why did you remove it everywhere?

Why Marshaller.dispatch was renamed to __dispatch but Unmarshaller.dispatch stays the same? (btw, why the double underscore?)