Considering the number of OS X machines running Python programs, it would be good idea to get this platform into the "stable" list of buildbots so that releases are checked against it.

"stable" is also meant to mean "typically passes test suite without errors". I don't think OSX meets this criterion.

Then why bother providing binaries?

I'm trying to keep the OSX port in good shape, but at times have to hunt down issues that were introduced by other bugfixes.

A problem with adding OSX to the list of stable buildbots is IMHO that there seem to be very few core developers that care about OSX beyond it being some odd flavor of unix.

Am 22.06.2010 12:40, schrieb Ronald Oussoren:
>
> Ronald Oussoren<ronaldoussoren@mac.com>  added the comment:
>
> Then why bother providing binaries?

How is that related? There was no OSX build slave until very recently,
but binaries had been provided for years. I see no reason to stop doing
so.

 > to hunt down issues that were introduced by other bugfixes.

That's appreciated. However, I fail to see the relationship to the 
buildbot stable list. The port was in good shape, and *still* didn't
pass all tests. That's partially because the test suite tests boundary
behaviour that doesn't affect the shape of the port.

 > that there seem to be very few core developers that care about
 > OSX beyond it being some odd flavor of unix.

The problem at hand is the port *doesn't* pass the test suite on
a regular manner. If it would, it could be added, and somebody
breaking the port could be asked to revert the change, or fix it
on OSX. As it stands, changes being made cannot be easily correlated
with new failures on that platform.

Bit of a chicken/egg issue here.  Since we haven't had OS X buildbots for very long, and the ones we do have represent odd configurations, I think it's premature to say that "the port *doesn't* pass the test suite on
a regular manner".  I think it's just as reasonable to say that the developers making changes just aren't aware of bad side-effects on OS X.  A good way to remedy that would be to make those bad side-effects more apparent, for example by adding an OS X buildbot into the "stable" set.

Another issue is that none of the three OS X buildbots now running are really good representatives of the technology that most people I know who use OS X really use.  That would be an Intel Core 2 Duo machine running Snow Leopard.

> Bit of a chicken/egg issue here.  Since we haven't had OS X buildbots
> for very long, and the ones we do have represent odd configurations,
> I think it's premature to say that "the port *doesn't* pass the test
> suite on a regular manner".

And I didn't mean that literally. Rather, I meant "the build slave that 
you are proposing to add to the stable list don't pass the test suite on 
a regular manner" - it's ultimately the individual slave (OS, compiler 
installation, buildbot version, network connectivity) that can be 
questioned for stability.

So that's not really a chicken-and-egg problem. If people keep working 
on the OSX port, they might ultimately arrive in a state where even the 
odd configurations become stable.

> I think it's just as reasonable to say
> that the developers making changes just aren't aware of bad
> side-effects on OS X.  A good way to remedy that would be to make
> those bad side-effects more apparent, for example by adding an OS X
> buildbot into the "stable" set.

I am not convinced that this is a good way. First, they are all 
volunteers, so they chose to do whatever they like to do. Then, even if 
they *are* willing to fix OSX problems, they might never look at the 
buildbot results. So any estimation of the effect that the proposed 
change might have is pure guessing.

Data point: on #python-dev we get announcements when buildbots (any buildbots, not just stable ones) *change state*.  That is, when a buildbot that was passing fails, or a buildbot that was failing passes.  We do look at the failures, though not all the time (if no one is around or it is one of that flaky buildbots that change state often they won't get checked or will get checked only randomly).

So, from the #python-dev perspective, the OS X buildbots will only be useful once they start being normally green.  Then we'll notice when they go red and, if we can, fix it, or open an issue.

> I think it's just as reasonable to say that the developers making
> changes just aren't aware of bad side-effects on OS X.  A good way to
> remedy that would be to make those bad side-effects more apparent, for 
> example by adding an OS X buildbot into the "stable" set.

I can't speak for other developers, but I generally look into the "unstable" set when I fear one of my changes might break something (and, sure enough, some of the SSL changes I did had to be adapted so that test_ssl pass again on the OS X buildots).

Martin: sorry about my first question. My interpretation of your first message was that you thought that the OSX port itself wasn't stable, and you've already mentioned that you didn't mean to imply that.

I'm unassigning the issue from my as I won't be able to actually move the osx buildbots in the stable list, at least not beyond trying to get the OSX port rock solid.

Is is possible to get e-mail about changes of buildbot status? I'd be interested in two sets of mail: any buildbot failures caused by my checkins and state changes for the OSX buildbots.

BTW. None of the buildbots currently test the configuration that is used for release builds: a 2-way or 3-way universal framework build.

> Is is possible to get e-mail about changes of buildbot status? I'd be
> interested in two sets of mail: any buildbot failures caused by my
> checkins and state changes for the OSX buildbots.

Buildbot failure reports are sent to python-checkins. In theory, at 
least; I think that isn't working very well.

Sending them to individual developers might be tricky. IOW, don't expect 
to see this happen within the next months unless somebody else 
volunteers to look into the buildbot configuration.

> Buildbot failure reports are sent to python-checkins. In theory, at 
> least; I think that isn't working very well.

ISTR we disabled them because there was too much churn from both
unreliable buildbots and unreliable tests, which made the noise annoying
and useless (everyone was ignoring failure notifications, I think).

Thanks to Steven Hansen there are now OSX buildbots in the stable list.