Author nirai
Recipients DazWorrall, alex, brian.curtin, carljm, coderanger, dabeaz, eric.smith, flox, jhylton, karld, kevinwatters, loewis, mahmoudimus, nirai, pitrou, rcohen, rh0dium, tarek
Date 2010-03-24.00:41:24
SpamBayes Score 3.77563e-05
Marked as misclassified No
Message-id <1269391296.08.0.748824407099.issue7946@psf.upfronthosting.co.za>
In-reply-to
Content
I upload bfs.patch

To apply the patch use the following commands on updated python 3.2:
$ patch -fp1 < bfs.patch
$ ./configure

The patch replaces the GIL with a scheduler. The scheduler is a simplified implementation of the recent kernel Brain F**k Scheduler by the Linux hacker Con Kolivas:

http://ck.kolivas.org/patches/bfs/sched-BFS.txt
"The goal of the Brain Fuck Scheduler, referred to as BFS from here on, is to
completely do away with the complex designs of the past for the cpu process
scheduler and instead implement one that is very simple in basic design.
The main focus of BFS is to achieve excellent desktop interactivity and
responsiveness without heuristics and tuning knobs that are difficult to
understand, impossible to model and predict the effect of, and when tuned to
one workload cause massive detriment to another."

Con Kolivas is the hacker whose work inspired the current CFS scheduler of the Linux Kernel. 

On my core 2 duo laptop it performs as follows compared to the other patches:

1) Florent's writenums() test: ~same
2) UDP test: x6 faster
3) cpued test: works as expected, while the other patches starve the pure python threads.

cpued test spins 3 threads, 2 of them pure python and the 3rd does time.sleep(0) every ~1ms:

import threading
import time

def foo(n):
    while n > 0:
        'y' in 'x' * n
        n -= 1

def bar(sleep, name):
    for i in range(100):
        print (name, i, sleep)
        for j in range(300):
            foo(1500)
            if sleep:
                time.sleep(0)

t0 = threading.Thread(target=bar, args=(False, 't0'))
t1 = threading.Thread(target=bar, args=(False, 't1'))
t2 = threading.Thread(target=bar, args=(True, 't2-interactive'))

list(map(threading.Thread.start, [t0, t1, t2]))
list(map(threading.Thread.join, [t0, t1, t2]))


The patch is still work in progress. In particular:
1) I still need to add support for Windows.
2) It currently requires Posix clock_gettime() and assumes good timer resolution.
3) I only verified it builds on Ubuntu Karmic 64bit.
4) I still need to optimize it and address cleanup.

The scheduler is very simple, straight forward and flexible, and it addresses the tuning problems discussed recently.

I think it can be a good replacement to the GIL, since Python really needs a scheduler, not a lock.
History
Date User Action Args
2010-03-24 00:41:36niraisetrecipients: + nirai, loewis, jhylton, pitrou, eric.smith, kevinwatters, tarek, karld, carljm, coderanger, alex, brian.curtin, flox, DazWorrall, rh0dium, rcohen, dabeaz, mahmoudimus
2010-03-24 00:41:36niraisetmessageid: <1269391296.08.0.748824407099.issue7946@psf.upfronthosting.co.za>
2010-03-24 00:41:34nirailinkissue7946 messages
2010-03-24 00:41:32niraicreate