Author kristjan.jonsson
Recipients beazley, dabeaz, flox, kristjan.jonsson, loewis, pitrou, r.david.murray, techtonik, torsten
Date 2010-04-20.22:33:22
SpamBayes Score 1.07586e-10
Marked as misclassified No
Message-id <1271802804.64.0.934502374871.issue8299@psf.upfronthosting.co.za>
In-reply-to
Content
It is interesting to see, David, the difference in the behaviour of the semaphore based and condition variable based lock on linux.  It is clear that the semaphore and the condition varable have different queuing characteristics.  I wouldn't be surprised if the semaphore were "fair" on average, but that it somehow colludes with the scheduler allow the current  thread to cut in early based on, perhaps, timeslices or something.  The condition variable seems to be much more FIFO.

It is odd to me that you don't seem to be able to shake the performance hit even with a checkinterval of 10000.  In the end, the performance hit is because of successful switches during the GIL yield in ceval.c.  You ought to be able to raise the  checkinterval up to the same level as the 'effective' one with the unfair condition variable approach and have the same sort of behaviour (inclusive IO latency).  I'll do some modifcations locally to gather stats on successful yields vs unsuccessful ones.

As for this being academic, yes it is.  As I stated early on, I never intended this to be checked in as is.  My aim was to take a step back from the radical GIL rewrite in 3.x.  Provide a simple platform for GIL experiments in the know and trusted 2.x series.  See if we could achieve the desired features without the added unknowns of the new GIL.

And in fact, I also didn't want to expend this amount of ammunition on the "ROUNDROBIN_GIL" implementation.  It was included merely as an example of one thing that was wrong with the old one and why it was so unpredictable.  What I really wanted to look at was to combine the benefit of a large checkinterval (100000, say) with a priority based mechanism giving us a low IO latency.  I've gotten so sidetracked with this whole fair/unfair business that I have neglected to provide useful performance benchmark of the PRIORITY_GIL implementation.

The fact that I had to expend so much effort explaining exactly how the gil on mac (LEGACY_GIL) was not fair, also shows how tricky working with synchronization primitives and threads can be, even using such relatively modern primitives as mutexes and condition variables.  There are many pitfalls for the unwary, and in my experience, only a lot of exposure will bend your brain into taking all those pesky race conditions into acccount.  This is why I think it is important to _understand_ the problem completely before trying to fix it, and why fixing it stepwise, with a full understanding of each step along the way, is necessary for robust behaviour.

As for missing the boat on 2.7, well, there is always 2.8 isn't there :)
History
Date User Action Args
2010-04-20 22:33:24kristjan.jonssonsetrecipients: + kristjan.jonsson, loewis, beazley, pitrou, techtonik, r.david.murray, flox, dabeaz, torsten
2010-04-20 22:33:24kristjan.jonssonsetmessageid: <1271802804.64.0.934502374871.issue8299@psf.upfronthosting.co.za>
2010-04-20 22:33:23kristjan.jonssonlinkissue8299 messages
2010-04-20 22:33:22kristjan.jonssoncreate