Created on 2008-09-26 09:58 by firstname.lastname@example.org, last changed 2010-01-03 21:14 by gregory.p.smith. This issue is now closed.
|email@example.com, 2008-09-26 09:58||Patch file|
|firstname.lastname@example.org, 2009-12-28 22:48||patch files and test file|
|msg73840 - (view)||Author: Eldon Ziegler (email@example.com)||Date: 2008-09-26 09:58|
I updated httplib.py, python 2.4, to be able to bind to a specific IP address when connecting to a remote site. conn = httplib.HTTPConnection('220.127.116.11', 80) will connect to '18.104.22.168' using one of the local IP addresses. For example, if a machine has an primary IP address and an alias such as eth0 192.168.1.10 eth0:1 192.168.1.11 the outbound connection might use either eth0 or eth0:1. I'm not sure how it picks now. I added a bind option both for http and https so we can direct the connection through one or the other. For example, conn = httplib.HTTPConnection('22.214.171.124', 80, None, None, None, '192.168.1.10') would make sure it used 192.168.1.10 not 192.168.1.11. I ran into this on a server that is contacted by an external legacy server which requires a reverse connection over the same IP address that the original connection came in on.
|msg74184 - (view)||Author: Gregory P. Smith (gregory.p.smith) *||Date: 2008-10-02 18:17|
This is useful but as its a feature we won't be able to add it until 2.7/3.1. Assigning to me to make sure it happens. Until then, the approach I suggest is to subclass httplib.HTTPConnection and HTTPSConnection to add your own variants that do bind before connecting.
|msg74185 - (view)||Author: Eldon Ziegler (firstname.lastname@example.org)||Date: 2008-10-02 18:38|
Sounds good. Let me know if there is anything I can do to help. Eldon On Thu, 2008-10-02 at 18:17 +0000, Gregory P. Smith wrote: > Gregory P. Smith <email@example.com> added the comment: > > This is useful but as its a feature we won't be able to add it until > 2.7/3.1. Assigning to me to make sure it happens. > > Until then, the approach I suggest is to subclass httplib.HTTPConnection > and HTTPSConnection to add your own variants that do bind before connecting. > > ---------- > assignee: -> gregory.p.smith > nosy: +gregory.p.smith > priority: -> normal > versions: -Python 2.4, Python 2.5, Python 2.6, Python 3.0 > > _______________________________________ > Python tracker <firstname.lastname@example.org> > <http://bugs.python.org/issue3972> > _______________________________________
|msg96963 - (view)||Author: Greg (greg.hellings)||Date: 2009-12-28 18:56|
Did this ever happen? It seems like overkill in the non-Python sort of way to continue pointing people to over-riding classes and extending objects when such a small patch adds so powerful and useful a functionality to the library.
|msg96964 - (view)||Author: R. David Murray (r.david.murray) *||Date: 2009-12-28 19:39|
Since Gregory didn't update the ticket, it seems likely that it hasn't happened yet. If you'd care to try the patch and report on your results, that would help the process. We'll also need unit tests and documentation updates, so patches for those would also speed the process.
|msg96966 - (view)||Author: Greg (greg.hellings)||Date: 2009-12-28 20:47|
Just looking at the indicated file in the 2.6.4 release tarball, it does not seem that it would apply cleanly. The line numbers do not apply properly anymore, though the edited lines themselves still appear to be unaffected. Without context diff in the original patch, it's difficult for me to asseess exactly which lines should be the affected ones. Unfortunately, I'm not in a position right now with my job to spend the time necessary to produce this as a patch since I'm on a timescale that requires equivalent functionality by Wednesday on production systems. Modifying it to apply cleanly to the latest versions of Python appears like it would be easy, if the internals of that file have not changed drastically in structure since 2.6.4. Documentation should be relatively straightforward as well, since the functionality the patch introduces is rather transparent. Unit tests are beyond my expertise to comment on. Would have loved to have seen this in the 2.7/3.1 series, as it would make my task much easier! I'll keep it in mind for my "off time" this holiday weekend.
|msg96968 - (view)||Author: Antoine Pitrou (pitrou) *||Date: 2009-12-28 21:57|
I'm not sure why this is needed in httplib. Isn't it a matter of configuring the machine's routing tables properly?
|msg96970 - (view)||Author: Greg (greg.hellings)||Date: 2009-12-28 22:03|
For my own case, I have a machine with 50 IP addresses set and I need to run a script to grab data that will randomly select one of those IP addresses to use for its outgoing connection. That's something which needs to be selected at the socket level, as best I understand the issue, not on the routing end.
|msg96971 - (view)||Author: Antoine Pitrou (pitrou) *||Date: 2009-12-28 22:11|
> For my own case, I have a machine with 50 IP addresses set and I need to > run a script to grab data that will randomly select one of those IP > addresses to use for its outgoing connection. That's something which > needs to be selected at the socket level, as best I understand the issue, > not on the routing end. Why do you need to select manually between those 50 IP addresses? Load-balancing? Fooling the remote server so that you don't get blacklisted for opening too many connections? The problem I see with adding this to httplib is that it's not HTTP-specific. Other modules (smtplib etc.) might want to benefit, depending on the use case. By the way, what I mean by routing is your OS' internal routing table for outbound packets. For example under Linux: # route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface X.Y.223.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1 X.Y.222.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 0.0.0.0 X.Y.222.254 0.0.0.0 UG 100 0 0 eth0 The interface for each outbound route is part of the routing table.
|msg96973 - (view)||Author: R. David Murray (r.david.murray) *||Date: 2009-12-28 22:23|
You are correct that more than just httplib may need this; however, it is a real need. Consider for example a machine that has multiple IPs on the same network (perhaps there used to be two machines but the services were consolidated onto one or something), and some other server it talks to is using IP based security in its .htaccess file. So the client machine needs to bind to the IP address that the server machine has authorized to access the web service. This is why, for example, the ssh command has a '-b' option to specify the port to which the client binds. Or, for an even more apropos example, wget has a --bind-address option for this reason.
|msg96974 - (view)||Author: Giampaolo Rodola' (giampaolo.rodola) *||Date: 2009-12-28 22:30|
In case it helps, a guy recently reported the same issue for pyftpdlib: http://code.google.com/p/pyftpdlib/issues/detail?id=123
|msg96975 - (view)||Author: Eldon Ziegler (email@example.com)||Date: 2009-12-28 22:48|
The patch files for Python 2.6 and a test file are in the attached tar file. Both httplib.py and socket.py need to be patched now. httplib.HTTPConnection(host[, port[, strict[, timeout]]]) becomes httplib.HTTPConnection(host[, port[, strict[, timeout[, bindip]]]]) and httplib.HTTPSConnection(host[, port[, key_file[, cert_file[, strict[, timeout]]]]]) becomes httplib.HTTPSConnection(host[, port[, key_file[, cert_file[, strict[, timeout[, bindip]]]]]])
|msg96976 - (view)||Author: R. David Murray (r.david.murray) *||Date: 2009-12-28 23:19|
If you want the maximum chance for this to get applied, it would be best to have one or a set of patch files in 'svn diff' format against trunk (see http://python.org/dev for information on how to produce such a patch, specifically section 6.1 of the dev FAQ). The tests will need to be in unittest format (they would go in test_httplib and/or test_socket). I imagine that writing the unittests so they will run on any machine might be a tad tricky. If I were doing it I'd try binding to 127.0.0.1 and checking to see that the socket is bound to that address rather than 0.0.0.0 in the binaddr case, and that it is bound to 0.0.0.0 in the default case.
|msg96982 - (view)||Author: Gregory P. Smith (gregory.p.smith) *||Date: 2009-12-29 01:14|
yes its a real need and yes we should support this in the standard library. no i have not had time to look at it since my comment a 14 months ago. Thanks for the updated patch Eldon!
|msg97154 - (view)||Author: Gregory P. Smith (gregory.p.smith) *||Date: 2010-01-03 02:07|
|msg97162 - (view)||Author: Antoine Pitrou (pitrou) *||Date: 2010-01-03 12:57|
It seems to break at least one buildbot: ====================================================================== ERROR: testSourceAddress (test.test_socket.NetworkConnectionAttributesTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/var/lib/buildslave/trunk.murray-gentoo-wide/build/Lib/test/test_socket.py", line 120, in _tearDown self.fail(msg) AssertionError: Tuples differ: ('126.96.36.199', 40732) != ('127.0.0.1', 40732) First differing element 0: 188.8.131.52 127.0.0.1 - ('184.108.40.206', 40732) + ('127.0.0.1', 40732) ---------------------------------------------------------------------- Ran 100 tests in 11.505s By the way, does the new "source_address" parameter have to be of the same family as the target address? That is, if I pass e.g. "127.0.0.1" as source address, what happens if the target host resolves to an IPv6 address?
|msg97167 - (view)||Author: R. David Murray (r.david.murray) *||Date: 2010-01-03 15:41|
Ah, and I was even the one that suggested the bind to 127.0.0.1 strategy for the test :( My buildbots run in linux-vserver virtual hosts, and they way they handle localhost (127.0.0.1) is to alias it to the IP address assigned to the virthost (otherwise multiple virthosts opening ports on 127.0.0.1 would potentially conflict with each other). I'd consider this a bug in the virthost software (lack of feature?). Not sure what the best workaround is, but if all else fails we could put in some special case skip code. Would be good enough to have the test check that the returned IP address is *not* 0.0.0.0? Or perhaps issue a warning if the address is not 0.0.0.0 but isn't 127.0.0.1.
|msg97179 - (view)||Author: Gregory P. Smith (gregory.p.smith) *||Date: 2010-01-03 21:14|
I took the easy route and remove the test of the hostname all together. The fact that the source port was used is sufficient indication that the bind call was made.
|2010-01-03 21:14:34||gregory.p.smith||set||messages: + msg97179|
|2010-01-03 15:41:22||r.david.murray||set||messages: + msg97167|
|2010-01-03 12:57:25||pitrou||set||messages: + msg97162|
|2010-01-03 02:07:07||gregory.p.smith||set||status: open -> closed|
messages: + msg97154
|2009-12-29 01:14:43||gregory.p.smith||set||messages: + msg96982|
|2009-12-28 23:19:31||r.david.murray||set||messages: + msg96976|
messages: + msg96975
|2009-12-28 22:30:21||giampaolo.rodola||set||messages: + msg96974|
|2009-12-28 22:23:32||r.david.murray||set||messages: + msg96973|
|2009-12-28 22:11:04||pitrou||set||messages: + msg96971|
|2009-12-28 22:03:03||greg.hellings||set||messages: + msg96970|
messages: + msg96968
|2009-12-28 20:47:08||greg.hellings||set||messages: + msg96966|
+ Python 3.2, - Python 3.1|
nosy: + r.david.murray
messages: + msg96964
keywords: + easy
stage: test needed
messages: + msg96963
|2008-10-02 20:57:39||giampaolo.rodola||set||nosy: + giampaolo.rodola|
|2008-10-02 18:38:firstname.lastname@example.org||set||messages: + msg74185|
|2008-10-02 18:17:37||gregory.p.smith||set||priority: normal|
messages: + msg74184
nosy: + gregory.p.smith
versions: - Python 2.6, Python 2.5, Python 2.4, Python 3.0