This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: xmlrpc.client HTTP proxy example code does not work
Type: behavior Stage: resolved
Components: Documentation Versions: Python 3.7, Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: avangel, berker.peksag, docs@python, python-dev
Priority: normal Keywords: patch

Created on 2016-10-08 11:53 by avangel, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
xmlrpc_client_http_proxy_test.py avangel, 2016-10-08 11:53 code to reproduce the problem
proxy.py berker.peksag, 2016-10-08 13:04
issue28389.diff berker.peksag, 2016-10-08 15:14 review
Messages (8)
msg278291 - (view) Author: Attila Vangel (avangel) Date: 2016-10-08 11:53
Go to https://docs.python.org/3/library/xmlrpc.client.html
Under '21.26.8. Example of Client Usage' -> 'To access an XML-RPC 
server through a HTTP proxy, you need to define a custom transport. The 
following example shows how:' copy the example code to a .py file (for 
me it is easier than REPL), e.g. xmlrpc_client_http_proxy_test.py

This is the example code:

import xmlrpc.client, http.client

class ProxiedTransport(xmlrpc.client.Transport):
    def set_proxy(self, proxy):
        self.proxy = proxy

    def make_connection(self, host):
        self.realhost = host
        h = http.client.HTTPConnection(self.proxy)
        return h

    def send_request(self, connection, handler, request_body, debug):
        connection.putrequest("POST", 'http://%s%s' % (self.realhost, handler))

    def send_host(self, connection, host):
        connection.putheader('Host', self.realhost)

p = ProxiedTransport()
p.set_proxy('proxy-server:8080')
server = xmlrpc.client.ServerProxy('http://time.xmlrpc.com/RPC2', transport=p)
print(server.currentTime.getCurrentTime())


I changed the 'proxy-server:8080' to '10.144.1.11:8080' which is a 
valid HTTP/HTTPS proxy in company I work for.

Try to run this code:

$ python3 xmlrpc_client_http_proxy_test.py 
Traceback (most recent call last):
  File "xmlrpc_client_http_proxy_test.py", line 21, in <module>
    print(server.currentTime.getCurrentTime())
  File "/usr/lib/python3.5/xmlrpc/client.py", line 1092, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python3.5/xmlrpc/client.py", line 1432, in __request
    verbose=self.__verbose
  File "/usr/lib/python3.5/xmlrpc/client.py", line 1134, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib/python3.5/xmlrpc/client.py", line 1146, in single_request
    http_conn = self.send_request(host, handler, request_body, verbose)
  File "xmlrpc_client_http_proxy_test.py", line 13, in send_request
    connection.putrequest("POST", 'http://%s%s' % (self.realhost, handler))
AttributeError: 'str' object has no attribute 'putrequest'

Personally I don't like the idea of putting this amount of code to 
documentation:
- as it seems, without automated tests running it, the code seems to rot, and gets outdated
- I need to paste this boilerplate code to my application if I want 
this functionality.

IMHO it would be much better to move this ProxiedTransport example code 
(after fixing it) to e.g. xmlrpc.client.HttpProxyTransport
(or similar name) class.

Details about python3:
$ python3
Python 3.5.2 (default, Sep 10 2016, 08:21:44) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
msg278292 - (view) Author: Attila Vangel (avangel) Date: 2016-10-08 12:02
I tested it also on Python 3.4.3. I got the same error.
msg278297 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-10-08 13:04
Thanks for the report. Can you try the attached script?
msg278300 - (view) Author: Attila Vangel (avangel) Date: 2016-10-08 13:33
Hi, thx for the quick turnaround.
I tried the proxy.py (on python 3.5) of course replacing 'YOUR_PROXY' with '10.144.1.11:8080' according to my environment.

python3 proxy.py 
Traceback (most recent call last):
  File "proxy.py", line 27, in <module>
    print(server.examples.getStateName(41))
  File "/usr/lib/python3.5/xmlrpc/client.py", line 1092, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python3.5/xmlrpc/client.py", line 1432, in __request
    verbose=self.__verbose
  File "/usr/lib/python3.5/xmlrpc/client.py", line 1134, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib/python3.5/xmlrpc/client.py", line 1147, in single_request
    resp = http_conn.getresponse()
  File "/usr/lib/python3.5/http/client.py", line 1197, in getresponse
    response.begin()
  File "/usr/lib/python3.5/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.5/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer


However, meanwhile I studied a bit the http.client API on how to use HTTP proxy, and I found set_tunnel() can do it. I had success by only overriding make_connection() in ProxiedTransport:
- copy current code of make_connection() from xmlrpc.client.Transport to ProxiedTransport (NOTE, this itself violates the DRY principle, but there is no better way to do it), change it slightly:
- create HTTPSConnection to the proxy (as I wanted to access a https URL)
- use .set_tunnel(chost) on this connection

I did not want to paste the code here, because
- I did not want to fill the 'PSF Contributor Agreement', at least yet
- it may be Python version specific solution.
msg278304 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-10-08 15:14
Thanks. I guess your solution was similar to the attached patch?
msg278305 - (view) Author: Attila Vangel (avangel) Date: 2016-10-08 15:29
It's my pleasure.
It was somewhat similar:
- the set_proxy() is the same
- for the make_connection() I gave the necessary clues, so one can create the code and you can use that in a way that I don't have to spend time on the PSF Contributor Agreement
- overriding send_request() is not necessary at all, because the HTTP proxying is done in http.client level
msg278362 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-10-09 15:19
New changeset 94c9c314f5d9 by Berker Peksag in branch '3.5':
Issue #28389: Fix ProxiedTransport example in xmlrpc.client documentation
https://hg.python.org/cpython/rev/94c9c314f5d9

New changeset 60c5c77c0190 by Berker Peksag in branch '3.6':
Issue #28389: Merge from 3.5
https://hg.python.org/cpython/rev/60c5c77c0190

New changeset 5f9351bc317e by Berker Peksag in branch 'default':
Issue #28389: Merge from 3.6
https://hg.python.org/cpython/rev/5f9351bc317e
msg278382 - (view) Author: Attila Vangel (avangel) Date: 2016-10-09 17:47
Thanks for fixing this issue. I checked the changed documentation online, and I came up with a very similar solution. One difference is that although this example overrides the make_connection() method, but omits the following lines which are present in the xmlrpc.client.Transport code (I checked that in python 3.5.3):

        # return an existing connection if possible.  This allows
        # HTTP/1.1 keep-alive.
        if self._connection and host == self._connection[0]:
            return self._connection[1]
        # create a HTTP connection object from a host descriptor
        chost, self._extra_headers, x509 = self.get_host_info(host)

Please check xmlrpc.client.Transport.make_connection().
I am not sure about the what kind of effect this may have.
I used chost as the parameter of set_tunnel(), rather than host (after briefly checking the code how it is calculated I felt that it's better to use chost than host, but I don't have a deep understanding of the http.client code).

The proxy_headers is a new thing, I guess it's a good thing if someone needs that.

I don't understand why the '*' character is needed in
connection = http.client.HTTPConnection(*self.proxy)
However I'm quite new to python.

I will try this new code tomorrow.
History
Date User Action Args
2022-04-11 14:58:38adminsetgithub: 72575
2016-10-09 17:47:25avangelsetmessages: + msg278382
2016-10-09 15:19:45berker.peksagsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2016-10-09 15:19:17python-devsetnosy: + python-dev
messages: + msg278362
2016-10-08 15:37:02berker.peksagsettype: crash -> behavior
versions: + Python 3.6, Python 3.7, - Python 3.4
2016-10-08 15:29:44avangelsettype: behavior -> crash
messages: + msg278305
versions: + Python 3.4, - Python 3.6, Python 3.7
2016-10-08 15:14:39berker.peksagsetfiles: + issue28389.diff
versions: + Python 3.6, Python 3.7, - Python 3.4
messages: + msg278304

keywords: + patch
type: crash -> behavior
stage: patch review
2016-10-08 13:33:00avangelsetmessages: + msg278300
2016-10-08 13:04:59berker.peksagsetfiles: + proxy.py
nosy: + berker.peksag
messages: + msg278297

2016-10-08 12:02:47avangelsetversions: + Python 3.4
2016-10-08 12:02:18avangelsetmessages: + msg278292
2016-10-08 11:53:34avangelcreate