This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: CPU usage shoots up with asyncore
Type: Stage:
Components: Library (Lib) Versions: Python 2.4
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: akuchling Nosy List: akuchling, arigo, rhettinger, thomasabc, tim.peters
Priority: high Keywords:

Created on 2004-08-16 15:54 by thomasabc, last changed 2022-04-11 14:56 by admin. This issue is now closed.

File name Uploaded Description Edit thomasabc, 2004-08-16 15:54
asyncore_comparison.txt thomasabc, 2004-08-20 10:05 Output compared in 2.3.3 and 2.4a1
Messages (5)
msg22080 - (view) Author: Thomas (thomasabc) Date: 2004-08-16 15:54
The CPU usage of a python application program using shoot up to 99% under Python 2.4a1. After 
a comparison, it was found out that the same program 
ran without such high CPU usage under Python 2.3.3

A simple couple of programs to narrow down the problem 
showed that a single loop() was accompanied by a large 
number of poll() with 2.4a1, whereas a single loop() had 
only 3 calls to poll() with 2.3.3.

Please let me know if there is anything else I can do for 
you to analyse. A couple of python scripts are attached 
in a zip file for you as below:

- Summary of the test scripts

With these scripts, the test attempts to identify what 
function calls are called that might explain the high CPU 
usage observed.
Taken from the Python Library Reference 11.24.2 
asynchat Example, the test program starts a server 
process to simulate client request process.
This program simply establishes connection to the server 
and closes after 10 seconds.

- Procedure

1. Start
2. Start
3. Stop
msg22081 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2004-08-19 17:29
Logged In: YES 

Andrew, what this due to your updates or was Martin's loop
counter the cause?
msg22082 - (view) Author: Thomas (thomasabc) Date: 2004-08-20 10:05
Logged In: YES 

Sorry, just to let you know that you press Ctrl+C to stop the 
script above at Step 3 to get an output displayed as shown in 
the file asyncore_comparison.txt below.

msg22083 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2004-08-21 23:01
Logged In: YES 

I can confirm the primary symptom on WinXP.  For the 10 
seconds runs, CPU is nailed to the wall under 
current CVS Python, but is a trickle under 2.3.4.

asyncore appears to be irrelevant.  "The problem" is that the 
socket associated with __main__.http_request_handler 
always comes back in the w set from under 
current Python, but the select times out under 2.3.4.

More detail:  under current Python, dumping some prints in 
asyncore.poll() delivers a huge number of repetitions of this 

socket map is
{1940: <__main__.http_request_handler connected at 0x9da800>,
 1948: <__main__.AsyncHTTPServer listening :40004 at 
r, w, e passed in is [1940, 1948] [1940, 1948] [1940, 1948]
r, w, e returned is [], [1940], []

Under 2.3.4, there are only a few lines of output total -- 
select() doesn't believe the sockets passed to it are ready to 
read or ready to write.

So the difference is in socket behavior, or in how the layers 
around asyncore here are using sockets.
msg22084 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2004-09-27 17:55
Logged In: YES 

Experimenting a bit, I found out that using from Python 2.3.3 in an otherwise CVS HEAD Python solves the problem.  The problem was introduced in this file in revision 1.22 by Raymond: "Use collection.deque() instead of a list for a FIFO queue."  <Insert random rambling against this kind of optimizations here>  The problem is at line 261:  "return self.list == []"  which is a correct way to check if a list is empty, but not if a deque is empty!

Fix checked in as

The same bug doesn't seem to be present anywhere else.

A few usages of deques in the standard library seem unjustified: in, items are only added or removed at the left.
Date User Action Args
2022-04-11 14:56:06adminsetgithub: 40767
2004-08-16 15:54:15thomasabccreate