This is a backport of the newgil work to the 2.7 branch (otherwise known as trunk.) Patch is against r77678.

Created by diffing r76193 (last in the newgil branch)
against r76189 and applying the result. Generally applied cleanly and it looked as though only 1 reject needed to be massaged in. Passes all tests except for test_command, but the unpatched source also fails that test on my machine.

Perf numbers for linux on Intel(R) Core(TM)2 Quad CPU Q9300 @ 2.50GHz
Using benchmark from here:
http://www.mail-archive.com/python-dev@python.org/msg43407.html
-j0
2.7   : 14.93s, 15.71s, 15.03s
newgil: 15.32s, 15.45s, 15.78s

-j4
2.7   : 22.44s, 22.30s, 22.19s
newgil: 18.75s, 18.80s, 18.79s

Looks like the 2.7 patch has a spurious change to /Lib/unittest/runner.py.

A few things:

 * The ceval_gil.h file is missing from the patch.

 * Please check whether you can move these variable declarations inside the main loop function (as statics):

+/* This single variable consolidates all requests to break out of the fast path
+   in the eval loop. */
+static volatile int eval_breaker = 0;
+/* Request for droppping the GIL */
+static volatile int gil_drop_request = 0;
+/* Request for running pending calls */
+static volatile int pendingcalls_to_do = 0; 
+/* Request for looking at the `async_exc` field of the current thread state */
+static volatile int pending_async_exc = 0;

 * Instead of deprecating the set/getcheckinterval APIs, convert the arguments passed to these to the switch interval. The new APIs set/getswitchinteral are then not needed.

 * Please add the fallback solutions from the time module in case gettimeofday() is not available. You cannot assume that "all modern POSIX systems" implement that API - it was introduced in POSIX 2001 and Python 2.x still supports OSes that were released prior to that year.

 * Note that gettimeofday() et al. are rather slow APIs on most systems. It would be better to directly use a simple performance timer since we don't really need any timing accuracy in managing the GIL when run with threads. Have a look at the clock_gettime() API for ways of accessing such timers. There may also be other ways of accessing such times via pthreads.

 * I haven't looked at the timing details of the implementation, but please make sure that it works even if the clock interval is a few ms (while gettimeofday() provides us resolution, the system clock may still only tick in ms intervals - another reason to use CPU performance timers instead).

Thanks.

See Tools/pybench/systimes.py for more comments on clocks, ticks and update frequency.

>  * Instead of deprecating the set/getcheckinterval APIs, convert the
> arguments passed to these to the switch interval. The new APIs
> set/getswitchinteral are then not needed.

This is certainly not a good idea. The semantics are different, which is the reason why I created a separate API in the first place instead of recycling the old one.

Actually, there should be hardly any reason to want to change the interval with the new GIL implementation. It was required with the old GIL due to the fact that counting opcodes is a very poor way of estimating wall clock durations.

> Note that gettimeofday() et al. are rather slow APIs on most systems

In my measurements under Linux it seemed to be as fast as clock_gettime() actually. But I don't know about other systems. I didn't want to complicate the patch (with several different paths) more than seemed necessary.

Here is an updated set of patches that includes the ceval_gil.h file as well as the documentation changes.

Antoine Pitrou wrote:
> 
> Antoine Pitrou <pitrou@free.fr> added the comment:
> 
>>  * Instead of deprecating the set/getcheckinterval APIs, convert the
>> arguments passed to these to the switch interval. The new APIs
>> set/getswitchinteral are then not needed.
> 
> This is certainly not a good idea. The semantics are different, which is the reason why I created a separate API in the first place instead of recycling the old one.
> 
> Actually, there should be hardly any reason to want to change the interval with the new GIL implementation. It was required with the old GIL due to the fact that counting opcodes is a very poor way of estimating wall clock durations.

Applications currently use those APIs for two reasons:

 * they use lots of threads and need to make sure that they
   all run more often than they do with the default settings e.g.
   to make them respond faster to external inputs: in such a
   case, they use a low check interval number

 * they don't use any threads and thus don't need to check
   for other threads at all or only rarely: in such a case they
   use a large check interval number

While the semantics are different, the result is similar and
the actual numbers used are usually determined by experiment
or rough estimate - noone expects the APIs to provide any kind
of exact timing and it's not needed either.

Turning the existing APIs into no-ops is certainly not a good
idea, since that will change application performance for both
use cases.

Esp. the second use case (single threaded process) is a rather
common one in Python - using multiple processes is often the more
scalable approach for Python applications, than relying on
multiple threads.

>> Note that gettimeofday() et al. are rather slow APIs on most systems
> 
> In my measurements under Linux it seemed to be as fast as clock_gettime() actually. But I don't know about other systems. I didn't want to complicate the patch (with several different paths) more than seemed necessary.

Right, but clock_gettime() provides a better resolution.

Both APIs are relatively slow, so if you call them often, it's
better to go with a different mechanism such as one built on
CPU performance counters.

> While the semantics are different, the result is similar and
> the actual numbers used are usually determined by experiment
> or rough estimate - noone expects the APIs to provide any kind
> of exact timing and it's not needed either.

There's no reasonable formula for computing an absolute switching
interval from the current "check interval" number. Besides, as I said,
there's no demonstrated reason to change the interval with the new GIL.

> * they don't use any threads and thus don't need to check
> for other threads at all or only rarely: in such a case they
> use a large check interval number

Well, if you don't use any threads you don't have to change the setting
in the first place :-). You might want to do it in order to
"micro-optimize" your app (i.e. save 1-2% on CPU-bound code), but this
is unnecessary with the new GIL, since the interval is ignored when
there's only one thread running.

> Turning the existing APIs into no-ops is certainly not a good
> idea, since that will change application performance for both
> use cases.

The new GIL will change performance far more anyway. Trying to simulate
the old behaviour is doomed to failure. If we want to keep the current
performance characteristics, we'd better not backport the new GIL (which
I'm fine with by the way).

Antoine Pitrou wrote:
> 
> Antoine Pitrou <pitrou@free.fr> added the comment:
> 
>> While the semantics are different, the result is similar and
>> the actual numbers used are usually determined by experiment
>> or rough estimate - noone expects the APIs to provide any kind
>> of exact timing and it's not needed either.
> 
> There's no reasonable formula for computing an absolute switching
> interval from the current "check interval" number. 

Just use some conversion formula, e.g. switching interval in micro
seconds = constant * byte code check interval. We can then determine
a constant to match todays CPU performance.

Given the inexact nature of the check interval that's a very
reasonable formula.

> Besides, as I said,
> there's no demonstrated reason to change the interval with the new GIL.

Didn't I just point out two use cases ?

Perhaps I'm missing some feature of the new GIL. Is there some
documentation for it somewhere ?

>> * they don't use any threads and thus don't need to check
>> for other threads at all or only rarely: in such a case they
>> use a large check interval number
> 
> Well, if you don't use any threads you don't have to change the setting
> in the first place :-). You might want to do it in order to
> "micro-optimize" your app (i.e. save 1-2% on CPU-bound code), but this
> is unnecessary with the new GIL, since the interval is ignored when
> there's only one thread running.

Hmm, how do you determine that only one thread is running ?

What if an extension uses threads that are not under Python
control, e.g. when embedding the JVM or when using a CORBA
ORB ?

>> Turning the existing APIs into no-ops is certainly not a good
>> idea, since that will change application performance for both
>> use cases.
> 
> The new GIL will change performance far more anyway. Trying to simulate
> the old behaviour is doomed to failure. If we want to keep the current
> performance characteristics, we'd better not backport the new GIL (which
> I'm fine with by the way).

I don't follow you.

If an application currently sets the check
interval to a low value, Python should switch to other threads
more often, ie. the switch interval needs to be low.

If an application currently sets the check interval to a high
value, Python should switch to other threads less often, ie.
the switch interval needs to be higher.

Both cases can be mapped to the switch interval, so why not
do it ?

Thanks Neil for fixing up the patch.

As for Marc-Andre's comments, I'm happy to backport any further
changes which happen on the py3k branch. I'd like to keep this as a
strict backport, only changing things as necessary to get it to work
with the older codebase.

Ross

> Just use some conversion formula, e.g. switching interval in micro
> seconds = constant * byte code check interval. We can then determine
> a constant to match todays CPU performance.

Well, there are two problems:
- opcode execution cost varies so wildly that trying to pick up an
average would be meaningless (it can vary by many order of magnitudes
between e.g. a POP_TOP and a CALL_FUNCTION calling a C extension method)
- the interval doesn't have the same effect in the new GIL as it has in
the old GIL; in the old GIL, it merely gives the OS a chance to switch
threads if it wants to (there are many missed opportunities, by a
proportion which will be system-dependent); in the new GIL, thread
switching is enforced which means it is not necessary to stop so often.

So we'd have a formula like:

new GIL interval = old GIL interval * opcode duration / proportion of
missed switching opportunities in the old GIL

where two of the three factors on the right side are totally
unpredictable :)

> Perhaps I'm missing some feature of the new GIL. Is there some
> documentation for it somewhere ?

Almost nothing besides what is found in ceval_gil.h and in the following
thread:
http://mail.python.org/pipermail/python-dev/2009-October/093321.html
Dave Beazley did a high-level presentation about it:
http://www.dabeaz.com/python/NewGIL.pdf

> > Well, if you don't use any threads you don't have to change the setting
> > in the first place :-). You might want to do it in order to
> > "micro-optimize" your app (i.e. save 1-2% on CPU-bound code), but this
> > is unnecessary with the new GIL, since the interval is ignored when
> > there's only one thread running.
> 
> Hmm, how do you determine that only one thread is running ?

Actually, I don't determine it, it's a side effect of how the GIL is
implemented. If no thread is waiting for the GIL, the main thread isn't
"asked" to drop it.

> What if an extension uses threads that are not under Python
> control, e.g. when embedding the JVM or when using a CORBA
> ORB ?

These don't wait for the GIL, hopefully :-)

On second thought, you're right: while sys.setcheckinterval() could
be made to work with the switching interval, sys.getcheckinterval()
could not be made to return anything useful.

Still, please have a look at the other issues mentioned. They
apply to new GIL in general, regardless of whether implemented
in 3.x or 2.7.

It would also be interesting to see the affect on other threaded
applications, not just the "benchmark" mentioned on the python-dev
thread.

Since there aren't all that many Python 3.x applications in the wild,
it's probably a good idea to check in the patch and then run a few
Python 2.x apps with and without it.

How does pybench react to the patch ? It's single-threaded at the
moment, so a good test of how a single-threaded application would
behave with the patch.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

Antoine Pitrou wrote:
>> Perhaps I'm missing some feature of the new GIL. Is there some
>> documentation for it somewhere ?
> 
> Almost nothing besides what is found in ceval_gil.h and in the following
> thread:
> http://mail.python.org/pipermail/python-dev/2009-October/093321.html
> Dave Beazley did a high-level presentation about it:
> http://www.dabeaz.com/python/NewGIL.pdf

Thanks. Dave's slides help a lot. The logic certainly sounds like
a major improvement !

>>> Well, if you don't use any threads you don't have to change the setting
>>> in the first place :-). You might want to do it in order to
>>> "micro-optimize" your app (i.e. save 1-2% on CPU-bound code), but this
>>> is unnecessary with the new GIL, since the interval is ignored when
>>> there's only one thread running.
>>
>> Hmm, how do you determine that only one thread is running ?
> 
> Actually, I don't determine it, it's a side effect of how the GIL is
> implemented. If no thread is waiting for the GIL, the main thread isn't
> "asked" to drop it.
> 
>> What if an extension uses threads that are not under Python
>> control, e.g. when embedding the JVM or when using a CORBA
>> ORB ?
> 
> These don't wait for the GIL, hopefully :-)

Well, they might call back into Python, but I guess that's not
supported anyway, not even in 2.x, since such a thread wouldn't
have a properly setup Python thread state.

As for your other comments, Marc-Andre:

> * Please check whether you can move these variable declarations inside
> the main loop function (as statics):
> [snip]

They are used in other functions than in the main loop, so it's not possible.

> * Please add the fallback solutions from the time module in case 
> gettimeofday() is not available. You cannot assume that "all modern
> POSIX systems" implement that API - it was introduced in POSIX 2001
> and Python 2.x still supports OSes that were released prior to that
> year.

I didn't know about that. In the context of py3k, I'm not sure it is really annoying, but you're right that for 2.x a fallback might be needed.

By the way, the new GIL only works with POSIX and Windows NT threading APIs. Perhaps it can't be backported at all to 2.x, given that 2.x supports more threading APIs than py3k does?

> I haven't looked at the timing details of the implementation, but
> please make sure that it works even if the clock interval is a few ms

Well, since it works under Windows (I don't have any really reliable benchmarks, since I've tested under a VM which is not an ideal condition), I assume it's robust against poor timing precisions.
(under Windows, WaitForMultipleObjects() is used which according to the MSDN docs has worse than microsecond precision)

But, yes, testing would be welcome on such platforms.

I meant "worse than milliseconds", sorry.

On Fri, 22 Jan 2010 09:32:36 +0000
Marc-Andre Lemburg <report@bugs.python.org> wrote:

>  * Please add the fallback solutions from the time module in case gettimeofday() is not available. You cannot assume that "all modern POSIX systems" implement that API - it was introduced in POSIX 2001 and Python 2.x still supports OSes that were released prior to that year.

POSIX as a standard tends to follow, not lead. The gettimeofday() call
dates back over 20 years in BSD. time.time() falls back on ftime() and
then time(). ftime() was added to the POSIX spec at the same time as
gettimeofday() and is now deprecated. time() probably doesn't have
enough resolution.

I'd have to be pointed to a specific platform which doesn't support
gettimeofday() but which is supported by python. Otherwise, I'd be
coding blind.

Ross

On Sat, 23 Jan 2010 18:23:10 +0000
Antoine Pitrou <report@bugs.python.org> wrote:

> By the way, the new GIL only works with POSIX and Windows NT threading APIs. Perhaps it can't be backported at all to 2.x, given that 2.x supports more threading APIs than py3k does?

Looking at the Python/thread_*.h files, it looks like py3k still
supports 9 different threading models. If that's accurate, it means
py3k trunk is broken on platforms which use cthread, lwp, os2, pth,
sgi, solaris and wince threading models. The 2.x series adds atheos and
beos to that list.

I think the right way to fix this is to extend the thread_*.h files to
have a proper abstraction for conditions which can be used by the
newgil work. Then the maintainers for more obscure platforms can fix
those instead of it all turning into a big mess in ceval_gil.h.

Ross

Ross Cohen wrote:
> 
> Ross Cohen <rcohen@snurgle.org> added the comment:
> 
> On Fri, 22 Jan 2010 09:32:36 +0000
> Marc-Andre Lemburg <report@bugs.python.org> wrote:
> 
>>  * Please add the fallback solutions from the time module in case gettimeofday() is not available. You cannot assume that "all modern POSIX systems" implement that API - it was introduced in POSIX 2001 and Python 2.x still supports OSes that were released prior to that year.
> 
> POSIX as a standard tends to follow, not lead. The gettimeofday() call
> dates back over 20 years in BSD. time.time() falls back on ftime() and
> then time(). ftime() was added to the POSIX spec at the same time as
> gettimeofday() and is now deprecated. time() probably doesn't have
> enough resolution.
> 
> I'd have to be pointed to a specific platform which doesn't support
> gettimeofday() but which is supported by python. Otherwise, I'd be
> coding blind.

The point here is that we have traditionally been careful not
to break Python for platforms that don't support a certain API,
hence the different fallback solutions for getting the current
time.

gettimeofday() is indeed available on most OSes, but not all. VxWorks
is an example where it is not available and Python happily uses
alternatives.

http://www.gsalcazar.altervista.org/tools/how-to-port-python-to-vxworks/?PHPSESSID=u5do9v0lh2gqdqilmikv7nmsk1

There are also cases where gettimeofday() is buggy
(e.g. on Solaris: http://www.unix.com/sun-solaris/59187-bugs-clock.html or
Cygwin: http://old.nabble.com/About-The---GetMicroSecondTime-on-Windows---Post-td13721510.html)
or returns errors every now and then (I have observed that in mxDateTime
occasionally - it may be related to NTP doing its works).

For those situations is necessary to be able to enable a
fallback solution using other APIs.

I've also done some extra research yesterday and found that e.g.
Ruby is using gettimeofday() for thread scheduling as well. They
have observed issues with gettimeofday() causing problems due
to wallclock time not being a monotonic (e.g. due to NTP running
on the machine and the clock sometimes going backwards due to
corrections or DST changes).

http://groups.google.com/group/ruby-talk-google/browse_thread/thread/f8a616113e2eea8f

OTOH, using process timers is also not regarded as being ideal,
since on SMP systems, each CPU will have its own timer and they
are not necessarily in sync.

Other implementations tend to use the new clock_gettime() APIs
where available and then use the CLOCK_MONOTONIC timer, e.g.

http://stackoverflow.com/questions/88/is-gettimeofday-guaranteed-to-be-of-microsecond-resolution
http://www.informit.com/guides/content.aspx?g=cplusplus&seqNum=272

Here's some example code which tries to cover even more platforms
than Python 2.x:

http://www.koders.com/c/fid5A75B2B62D4024A2431D060ADD5D378DB7A1D2BD.aspx

Another aspect to consider is update frequency of these APIs.
gettimeofday()'s resolution depends on various factors and
appears to vary between 1us and 100ms (on systems running using
a 100Hz clock).

http://answers.google.com/answers/threadview/id/203397.html
http://www.informit.com/guides/content.aspx?g=cplusplus&seqNum=272
https://computing.llnl.gov/tutorials/performance_tools/#gettimeofday

clock_gettime() has similar resolution, but tends to be updated
more often:

http://forums13.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1264587819627+28353475&threadId=1230881

And finally, using wallclock time for these things is expensive
as I've already mentioned:

http://groups.google.com/group/comp.os.linux.development.apps/browse_frm/thread/dc29071f2417f75f/c46264dba0863463?lnk=st&rnum=1#c46264dba0863463
http://www.developerweb.net/forum/archive/index.php/t-4368.html
http://h21007.www2.hp.com/portal/download/files/unprot/hpux/HowToTellTheTime.pdf
(as HTML:
http://209.85.129.132/search?q=cache%3AI1ihHYBMJvgJ%3Ah21007.www2.hp.com%2Fportal%2Fdownload%2Ffiles%2Funprot%2Fhpux%2FHowToTellTheTime.pdf+gettimeofday()+support+across+platforms&hl=en&gl=de)

It appears to be better to use clock_gettime(CLOCK_MONOTONIC)
where available and only use gettimeofday() as fallback solution
together with times(), ftime() and time().

Ross Cohen wrote:
>> By the way, the new GIL only works with POSIX and Windows NT threading APIs. Perhaps it can't be backported at all to 2.x, given that 2.x supports more threading APIs than py3k does?
> 
> Looking at the Python/thread_*.h files, it looks like py3k still
> supports 9 different threading models. If that's accurate, it means
> py3k trunk is broken on platforms which use cthread, lwp, os2, pth,
> sgi, solaris and wince threading models. The 2.x series adds atheos and
> beos to that list.
> 
> I think the right way to fix this is to extend the thread_*.h files to
> have a proper abstraction for conditions which can be used by the
> newgil work. Then the maintainers for more obscure platforms can fix
> those instead of it all turning into a big mess in ceval_gil.h.

I find it rather strange that Python 3.x now only supports
NT and POSIX threads in ceval while it still supports the
whole set of other thread implementations for the _thread
module.

Has this been discussed on python-dev ?

Why can't the code necessary to get the new GIL working be
added to Sjoerd's portable thread library ?

As it stands, I think this is a bummer for both Python 2.7 and 3.2.

Le mercredi 27 janvier 2010 à 10:37 +0000, Marc-Andre Lemburg a écrit :
> 
> I find it rather strange that Python 3.x now only supports
> NT and POSIX threads in ceval while it still supports the
> whole set of other thread implementations for the _thread
> module.
> 
> Has this been discussed on python-dev ?

Yes, it was. See
http://mail.python.org/pipermail/python-dev/2009-October/093276.html
All these thread_*.h files are still there, but most of them are
deprecated (see the #error's in thread.c). It's the consensus that came
out on the mailing-list.

The one exception is OS/2, which is "supported" as long as Andrew
McIntyre (or someone else, of course) takes care about it. It was
decided that OS/2 compatibility wasn't important enough to bar adding
new features or making improvements.

> Why can't the code necessary to get the new GIL working be
> added to Sjoerd's portable thread library ?

The condition variable "emulation" used by the new GIL under Windows is
not valid as a general condition variable mechanism; this is mentioned
in ceval_gil.h, and a link is given to a much more complicated (but
valid) condition variable emulation. Therefore, I think it could be
detrimental to expose it publicly.

Le mercredi 27 janvier 2010 à 10:33 +0000, Marc-Andre Lemburg a écrit :
> 
> It appears to be better to use clock_gettime(CLOCK_MONOTONIC)
> where available and only use gettimeofday() as fallback solution
> together with times(), ftime() and time().

Thanks for the research! I will take a look at this.

Antoine Pitrou wrote:
> 
> Antoine Pitrou <pitrou@free.fr> added the comment:
> 
> Le mercredi 27 janvier 2010 à 10:37 +0000, Marc-Andre Lemburg a écrit :
>>
>> I find it rather strange that Python 3.x now only supports
>> NT and POSIX threads in ceval while it still supports the
>> whole set of other thread implementations for the _thread
>> module.
>>
>> Has this been discussed on python-dev ?
> 
> Yes, it was. See
> http://mail.python.org/pipermail/python-dev/2009-October/093276.html
> All these thread_*.h files are still there, but most of them are
> deprecated (see the #error's in thread.c). It's the consensus that came
> out on the mailing-list.
> 
> The one exception is OS/2, which is "supported" as long as Andrew
> McIntyre (or someone else, of course) takes care about it. It was
> decided that OS/2 compatibility wasn't important enough to bar adding
> new features or making improvements.

The arguments given in that thread sound a bit strange to me:
just because there were no changes to a few files, doesn't really
say anything about whether they contain working code or not.

In any case, the new GIL implementation as it stands makes
supporting any of those additional threading implementations
a moot case.

You could just as well remove them right now: if the GIL doesn't
work on OS/2, then having support for it in the _thread module
isn't really worth much, is it ?

Regardless, instead of now having two places where threading is
implemented, we should just have one: the portable thread library and
extend this as necessary.

>> Why can't the code necessary to get the new GIL working be
>> added to Sjoerd's portable thread library ?
> 
> The condition variable "emulation" used by the new GIL under Windows is
> not valid as a general condition variable mechanism; this is mentioned
> in ceval_gil.h, and a link is given to a much more complicated (but
> valid) condition variable emulation. Therefore, I think it could be
> detrimental to expose it publicly.

By moving all the thread support code to the thread library,
you don't really expose anything (the lib is only used internally
by Python), you just reorganize the code in a more natural and
easier to maintain way.

If you think that the win32 emulation shouldn't be
used for non-GIL purposes, then just add a comment to
those APIs.

With just NT and POSIX thread support, I think backporting the
new GIL implementation to 2.7 is not possible - we'd have to go
through a standard PEP 11 deprecation process and there are not
enough 2.x releases left for that. It could only be backported
as optional feature, to be enabled by a configure option.

> The arguments given in that thread sound a bit strange to me:
> just because there were no changes to a few files, doesn't really
> say anything about whether they contain working code or not.

That was a heuristic. Files which do not get any maintenance for years
while other similar files do are quite suspicious.
Given that nobody stepped up to contradict this hypothesis of mine, I
assume it was right after all ;)

More seriously, all the APIs in question (and most of their supporting
systems: IRIX etc.) seem practically dead. I don't want to rehash that
discussion here, but you can post on python-dev if you want.

> You could just as well remove them right now: if the GIL doesn't
> work on OS/2, then having support for it in the _thread module
> isn't really worth much, is it ?

Andrew told me he believed it possible to port the new GIL to OS/2. So
perhaps he'll do that before 3.2 is out.

> With just NT and POSIX thread support, I think backporting the
> new GIL implementation to 2.7 is not possible - we'd have to go
> through a standard PEP 11 deprecation process and there are not
> enough 2.x releases left for that. It could only be backported
> as optional feature, to be enabled by a configure option.

Right. That's what I think too.

Antoine Pitrou wrote:
> 
> Antoine Pitrou <pitrou@free.fr> added the comment:
> 
>> The arguments given in that thread sound a bit strange to me:
>> just because there were no changes to a few files, doesn't really
>> say anything about whether they contain working code or not.
> 
> That was a heuristic. Files which do not get any maintenance for years
> while other similar files do are quite suspicious.
> Given that nobody stepped up to contradict this hypothesis of mine, I
> assume it was right after all ;)

We'll only be able to tell for sure when it's too late: at release
time. We simply don't have any active developers working on more
exotic platforms, but that doesn't mean that Python isn't used
on those platforms.

> More seriously, all the APIs in question (and most of their supporting
> systems: IRIX etc.) seem practically dead. I don't want to rehash that
> discussion here, but you can post on python-dev if you want.

No need... I'm tired of trying to get Python devs on track with
respect to the PEP 11 process, deprecations, etc.

>> You could just as well remove them right now: if the GIL doesn't
>> work on OS/2, then having support for it in the _thread module
>> isn't really worth much, is it ?
> 
> Andrew told me he believed it possible to port the new GIL to OS/2. So
> perhaps he'll do that before 3.2 is out.
>
>> With just NT and POSIX thread support, I think backporting the
>> new GIL implementation to 2.7 is not possible - we'd have to go
>> through a standard PEP 11 deprecation process and there are not
>> enough 2.x releases left for that. It could only be backported
>> as optional feature, to be enabled by a configure option.
> 
> Right. That's what I think too.

I'll close the issue then.

Closing the issue since we can't backport to Python 2.7 due to the missing thread library support.

The patch may still be useful for experiments by users, though, so thanks to Ross and Neil for creating it.

> It appears to be better to use clock_gettime(CLOCK_MONOTONIC)
> where available and only use gettimeofday() as fallback solution
> together with times(), ftime() and time().

Ok, I've tried and it's less good than expected. Using CLOCK_MONOTONIC
absolutely kills efficiency. CLOCK_REALTIME is ok but it has no obvious
benefits (microsecond resolution as given by gettimeofday() is probably
sufficient).

The explanation AFAICT is that pthread_cond_timedwait() waits for
absolute clock values as given by CLOCK_REALTIME.
CLOCK_MONOTONIC gives other values (the man page says: "represents
monotonic time since some unspecified starting point"). These values are
probably "in the past" as seen from pthread_cond_timedwait(), which
implies a busy loop of waiting for the GIL to be released, inside of
being suspended gracefully until the timeout.

I can still produce a patch with only CLOCK_REALTIME but I'm not sure
it's worth the code complication.

Antoine Pitrou wrote:
> 
> Antoine Pitrou <pitrou@free.fr> added the comment:
> 
>> It appears to be better to use clock_gettime(CLOCK_MONOTONIC)
>> where available and only use gettimeofday() as fallback solution
>> together with times(), ftime() and time().
> 
> Ok, I've tried and it's less good than expected. Using CLOCK_MONOTONIC
> absolutely kills efficiency. CLOCK_REALTIME is ok but it has no obvious
> benefits (microsecond resolution as given by gettimeofday() is probably
> sufficient).
> 
> The explanation AFAICT is that pthread_cond_timedwait() waits for
> absolute clock values as given by CLOCK_REALTIME.
> CLOCK_MONOTONIC gives other values (the man page says: "represents
> monotonic time since some unspecified starting point"). These values are
> probably "in the past" as seen from pthread_cond_timedwait(), which
> implies a busy loop of waiting for the GIL to be released, inside of
> being suspended gracefully until the timeout.

The CLOCK_MONOTONIC timer only guarantees that you
get an accurate time count. It doesn't maintain any relationship
to the wall clock time. For the case in question you don't need
the wall clock time, though. It's more important not have the
clock go backwards or be subject to jitter.

pthreads will default to use the real time clock. In order
to have them use the monotonic timer, you have to setup
a condition variable attribute: See the man-page for
pthread_condattr_setclock().

> I can still produce a patch with only CLOCK_REALTIME but I'm not sure
> it's worth the code complication.

Even if you don't use CLOCK_MONOTONIC you should still prefer
clock_gettime() over gettimeofday() simply because it's faster.
The resolution of both will likely be the same, unless the
hardware provides a more accurate timer than kernel ticks.

The code won't get more complicated if you refactor the time
querying logic into a separate function (which the compiler can then
inline as necessary).

I am confused by this line of reasoning. Is it ok to ignore the
deprecation process in py3k but not in 2.x? Is it only ok if a core
developer does it?

If the point of 2.7 is to make it easier for apps and packages to be
ported to py3k, then what would be the point of these platforms moving
to 2.7 in the first place? It seems perfectly reasonable not to support
platforms which are never going to care the release. If the platforms
are broken for 2.7, you'll get that much more warning before 3.2 is
released so it can be fixed.

Ross

On Wed, 27 Jan 2010 14:30:20 +0000
Marc-Andre Lemburg <report@bugs.python.org> wrote:

> Antoine Pitrou wrote:
> > More seriously, all the APIs in question (and most of their supporting
> > systems: IRIX etc.) seem practically dead. I don't want to rehash that
> > discussion here, but you can post on python-dev if you want.
> 
> No need... I'm tired of trying to get Python devs on track with
> respect to the PEP 11 process, deprecations, etc.
> 
> >> You could just as well remove them right now: if the GIL doesn't
> >> work on OS/2, then having support for it in the _thread module
> >> isn't really worth much, is it ?
> > 
> > Andrew told me he believed it possible to port the new GIL to OS/2. So
> > perhaps he'll do that before 3.2 is out.
> >
> >> With just NT and POSIX thread support, I think backporting the
> >> new GIL implementation to 2.7 is not possible - we'd have to go
> >> through a standard PEP 11 deprecation process and there are not
> >> enough 2.x releases left for that. It could only be backported
> >> as optional feature, to be enabled by a configure option.
> > 
> > Right. That's what I think too.
> 
> I'll close the issue then.

> pthreads will default to use the real time clock. In order
> to have them use the monotonic timer, you have to setup
> a condition variable attribute: See the man-page for
> pthread_condattr_setclock().

I'll look at that, but I'm not thrilled at the propect of complicating
the code paths so much. There may be systems where CLOCK_MONOTONIC is
unavailable, others where pthread_condattr_setclock() is unsupported,
etc.

> The code won't get more complicated if you refactor the time
> querying logic into a separate function (which the compiler can then
> inline as necessary).

It does get more complicated, since there are several paths
(clock_gettime() and then a fallback on gettimeofday()).
I'm not talking about complexity in the executable but about maintenance
complexity.

Ross Cohen wrote:
> 
> Ross Cohen <rcohen@snurgle.org> added the comment:
> 
> I am confused by this line of reasoning. Is it ok to ignore the
> deprecation process in py3k but not in 2.x? Is it only ok if a core
> developer does it?

It's normally not ok to ignore the deprecation process for 3.x,
since the 3.x branch is in a stable state, just like the 2.x
branch (3.0 and, after some discussions, 3.1 were allowed to
break things).

I don't know why this process wasn't followed for the new GIL
implementation.

It basically breaks compatibility with the other
thread implementations, but does so without actually removing
the support for them in the thread library we're using for
the _thread module.

Perhaps this was simply not known to other developers.

There's also no PEP for the new implementation, which you'd
normally require for any such major change to the internals.

> If the point of 2.7 is to make it easier for apps and packages to be
> ported to py3k, then what would be the point of these platforms moving
> to 2.7 in the first place? It seems perfectly reasonable not to support
> platforms which are never going to care the release. If the platforms
> are broken for 2.7, you'll get that much more warning before 3.2 is
> released so it can be fixed.

Breaking existing applications and ports of Python for 2.7
certainly won't make anything easier for anyone.

For 2.7 we will certainly not allow the above to happen,
since that's the version that's going to be used by most
people for a few years to come.

What we could do for 2.7, is add Py3k deprecation warnings
to the alternative thread implementations, mentioning their
removal in 3.2.

On Fri, 29 Jan 2010 21:15:14 +0000
Marc-Andre Lemburg <report@bugs.python.org> wrote:

> Breaking existing applications and ports of Python for 2.7
> certainly won't make anything easier for anyone.
> 
> For 2.7 we will certainly not allow the above to happen,
> since that's the version that's going to be used by most
> people for a few years to come.
> 
> What we could do for 2.7, is add Py3k deprecation warnings
> to the alternative thread implementations, mentioning their
> removal in 3.2.

What if, as you proposed earlier, the patch were to leave the old
behavior if the threading model on the given platform were not
supported?

Ross

Sorry for the nosy. There is something going wrong with my Firefox caching.