New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
socket.socket.recv broken (unbearably slow) #48016
Comments
Under Linux 2.6.24-1-amd64 as well as 2.6.26 (x86-32), Python versions Some detail: For a university project, I wrote a python client for a query-server. A Finally, I stripped down the original client/server to a postable level Here is the gist of it: Sending 500 packets @ 2 tokens each (500 very short lists) takes 40 In detail: 14508 function calls in 39.980 CPU seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) Here is the same for 1 packet @ 50000 tokens (1 very long list), taking 8.51872587204 Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) I don't get the reason for the huge speed discrepancy. I include all Notably, the original software (non stripped-down version) runs without Also note that I can't get the server to shut down properly (the thread The attached archive contains all source code plus README and the |
As promised, here is the profiler output for a Mac (thanks Stefan). The $ python runme.py
500 packets @ 2 tokens each (500 very short lists)
0.0814669132233
14508 function calls in 0.082 CPU seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 1 packet @ 50000 tokens (1 very long list) Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) |
Thorben, is the problem still there if you use fork() rather than The implementation of the recv() method is straightforward and I don't |
The problem exists even if the server part is replaced by a server The C server works with reasonable speed when connected to a client Nevertheless, I will try out your suggestion. Thanks for replying, Thorben 2008/9/11 Antoine Pitrou <report@bugs.python.org>:
|
You can use setsockopt() to set the TCP_NODELAY flag and see if that As I said, socket.recv() is just a thin wrapper above the same-named C By the way, if you want to build network applications (clients and |
2008/9/11 Antoine Pitrou <report@bugs.python.org>:
You are obviously right, but IIRC, I mentioned that this is simply a now thats interesting: Do you think I could get that value down much further? I don't know
Would be interesting to examine the differences between the Perl
Thanks, I'll make sure to try it out.
|
Did you do it on both the client and server sockets?
Perhaps the Perl wrapper is less thin as the Python one. In any case, |
2008/9/11 Antoine Pitrou <report@bugs.python.org>:
Yes, obviously. Although adding it to the client socket did make no
I would greatly appreciate any help on the subject. How do *BSD
|
I've tested it on Windows XP. MSG_WAITALL is not supported, but I 500 packets @ 2 tokens each (500 very short lists) Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) None 1 packet @ 50000 tokens (1 very long list) Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) |
That's more like it. Now I'd like to see the same behavior on Linux... 2008/9/12 Gabriel Genellina <report@bugs.python.org>:
|
Well, if this precise use case is really important for you, I suggest Something else: try replacing "localhost" with "127.0.0.1", perhaps your
I don't know, but I suspect the difference is more in the TCP stack In any case, I'm gonna close this bug, as this is very likely not a |
I know this bug is closed, but I too am experiencing it under Linux I'm using urllib2, and it's just spending an obscene amount of cpu time Anyone have any ideas? Would switching to httplib2 help? |
I was able to observe the same issue: I'm using Python 2.6.5 on Ubuntu 10.0.4 LTS. My system is 64 bit Intel Core I7, Quad core, Linux kernel 2.6.32-generic x86_64, Ubuntu EGLIBC 2.11.1-0ubuntu7.5. A simple client TCP socket, which sends 35 bytes over to a server on localhost and receives 20 bytes in response, produces only 22 RPS. An identical application written in C gives me 7000 RPS. TCP_NODELAY is on on the server side. Turning on TCP_NODELAY on the client gives me ~500 RPS in Python (which I'm satisfied with, 'cause I think I then hit other bottlenecks). |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: