This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author thorben
Recipients thorben
Date 2008-09-03.21:58:44
SpamBayes Score 7.545049e-10
Marked as misclassified No
Message-id <1220479128.18.0.248146469424.issue3766@psf.upfronthosting.co.za>
In-reply-to
Content
Under Linux 2.6.24-1-amd64 as well as 2.6.26 (x86-32), Python versions
2.5.2 and 2.4.4 (on both machines), there is a huge discrepancy between
socket read latencies, depending on "code context".

Some detail:

For a university project, I wrote a python client for a query-server. A
reference implementation existed in Perl, so the job was pretty straight
forward. However, for reasons unknown to me, the Python implementation
was by several orders of magnitude slower than the reference
implementation. To track down the issue, I wrote a dummy version of the
query-server in Python, where the problem persisted. I then stripped
down both client and server to the minimal functionality and still the
problem persisted. 
I wrote a demo inside a single file using socketpair() to post here, but
the issue was not reproducible.

Finally, I stripped down the original client/server to a postable level
and ran a profiler on a testcase.

Here is the gist of it:

Sending 500 packets @ 2 tokens each (500 very short lists) takes 40
seconds on average.

In detail:

14508 function calls in 39.980 CPU seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1500   39.834    0.027   39.834    0.027 {method 'recv' of
'_socket.socket' objects}
     1500    0.024    0.000   39.877    0.027 Client.py:16(read_int)
     1500    0.020    0.000    0.020    0.000 <string>:1(sendall)
     1500    0.018    0.000    0.048    0.000 Client.py:8(send_int)
      500    0.016    0.000   39.903    0.080 Client.py:19(read_int_list)
     1500    0.015    0.000    0.019    0.000 struct.py:77(unpack)
      500    0.010    0.000    0.060    0.000 Client.py:11(send_int_list)
     1500    0.010    0.000    0.010    0.000 struct.py:54(pack)
        1    0.009    0.009   39.980   39.980 dummytest.py:12(bench)
     1000    0.007    0.000    0.007    0.000 {method 'insert' of 'list'
objects}
     1001    0.006    0.000    0.006    0.000 {range}
      500    0.005    0.000   39.968    0.080 Client.py:28(spam)
     1500    0.005    0.000    0.005    0.000 {method 'unpack' of
'Struct' objects}
      501    0.002    0.000    0.002    0.000 {len}


Here is the same for 1 packet @ 50000 tokens (1 very long list), taking
below 10 seconds on average.

8.51872587204
         400018 function calls in 8.519 CPU seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    50001    2.980    0.000    2.980    0.000 {method 'recv' of
'_socket.socket' objects}
    50000    2.980    0.000    2.980    0.000 {method 'insert' of 'list'
objects}
    50001    0.993    0.000    0.993    0.000 <string>:1(sendall)
    50001    0.410    0.000    1.558    0.000 Client.py:8(send_int)
    50001    0.334    0.000    3.581    0.000 Client.py:16(read_int)
        1    0.247    0.247    6.809    6.809 Client.py:19(read_int_list)
    50001    0.191    0.000    0.266    0.000 struct.py:77(unpack)
    50001    0.154    0.000    0.154    0.000 struct.py:54(pack)
        1    0.146    0.146    1.703    1.703 Client.py:11(send_int_list)
    50001    0.075    0.000    0.075    0.000 {method 'unpack' of
'Struct' objects}

I don't get the reason for the huge speed discrepancy. I include all
source code files for reproducing the issue.

Notably, the original software (non stripped-down version) runs without
these issues using a OS X Python version. Details may follow, I don't
own a Mac but know people who do.

Also note that I can't get the server to shut down properly (the thread
does not terminate). You have to kill the process manually and wait for
the port to be freed by the kernel. Maybe this is easily fixable but at
least I don't know how.

The attached archive contains all source code plus README and the
socketpair() version.
History
Date User Action Args
2008-09-03 21:58:48thorbensetrecipients: + thorben
2008-09-03 21:58:48thorbensetmessageid: <1220479128.18.0.248146469424.issue3766@psf.upfronthosting.co.za>
2008-09-03 21:58:47thorbenlinkissue3766 messages
2008-09-03 21:58:44thorbencreate