msg94977 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-11-06 15:57 |
The x86 FreeBSD buildslave is consistently aborting the test run with a
'Signal 12' failure in test_multiprocessing. See e.g.,
http://www.python.org/dev/buildbot/builders/x86%20FreeBSD%20trunk/builds/2
756/steps/test/logs/stdio
(scroll all the way to the bottom to see the failure).
The failure occurs on 2.7, 3.1 and 3.2. On 2.6, it looks as though
test_multiprocessing is disabled: the message gives a reference to issue
3770.
Note that the maintainer of the buildbot (David Bolen) has offered to
arrange ssh access for anyone wanting to look into this. (See
http://mail.python.org/pipermail/python-dev/2009-November/093857.html
.)
|
msg94978 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-11-06 15:58 |
I mangled the buildbot results URL. Here it is again.
http://www.python.org/dev/buildbot/builders/x86%20FreeBSD%20trunk/builds/2756/steps/test/logs/stdio
|
msg94979 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-11-06 16:05 |
I'm not sure whether this is relevant, but the configure output for the
FreeBSD trunk build includes the line:
checking for broken sem_getvalue... yes
|
msg95545 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-11-20 14:01 |
I had an opportunity to play with a FreeBSD 7.2 box recently.
The diagnosis is simple. The solution may be less so...
Diagnosis: FreeBSD still considers POSIX semaphores (sem_open,
sem_close, etc.) to be experimental, so they're not enabled by default
on a standard install. So the very first call to sem_open, from
SEM_CREATE in semaphore.c (around line 439), produces the Signal 12.
Enabling POSIX semaphores (assuming that they've been built into the
kernel, which they seem to have been by default) is as simple as
executing 'kldload sem' (as root) at a shell prompt. After I did this,
test_multiprocessing ran and all tests passed.
So the question is what multiprocessing (and test_multiprocessing)
should do when POSIX semaphores aren't available. I guess the options
are: (1) fail loudly, with an error message telling the user to enable
the POSIX semaphores, or (2) try to use SysV semaphores (which are
supported out of the box) instead.
In the immediate future we should probably at least detect when POSIX
semaphores don't exist, and skip test_multiprocessing in that case.
|
msg95548 - (view) |
Author: Jesse Noller (jnoller) * |
Date: 2009-11-20 14:23 |
Thanks mark - if POSIX semaphores aren't available, we're largely dead-
on-arrival. I thought FBSD was at the point of enabling them by default
- boo on me. See also: http://bugs.python.org/issue3770#msg73958 and
http://bugs.python.org/issue3770#msg83495 - with martin's help last
pycon I cut a lot of the "HAVE_BROKEN..." values to autoconf checks in
configure.in.
I think the right way (however unpleasant) is to check to see if we have
POSIX semaphores, period - if we don't, _multiprocessing should just not
compile.
|
msg95553 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-11-20 16:18 |
Here's a patch that makes FreeBSD behave exactly as though HAVE_SEM_OPEN
is not defined, when semaphores aren't available. On my FreeBSD 7.2
test system, it results in the multiprocessing module being built
(without the contents of semaphore.c), but the test is skipped, along
with a message indicating that semaphores don't work on that system.
I added an autoconf test for the sem_open failure, and reused the
already existing pyconfig.h variable HAVE_BROKEN_POSIX_SEMAPHORES.
|
msg95555 - (view) |
Author: Jesse Noller (jnoller) * |
Date: 2009-11-20 16:58 |
Looks good so far to me - I'll apply and run the tests locally (but I
don't have a fbsd box, so I'm just checking for regressions). One question
- what's with all the
-rm -f -r conftest*
+rm -f conftest*
Lines in there?
|
msg95556 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-11-20 17:05 |
About the rm -f -r conftest stuff:
My guess is that the last person to update configure used Apple's
version of autoconf: Apple seems to have silently 'fixed' autoconf
version 2.61 to remove some (fairly benign) warnings that appear when
running the configure script. Which is fine, but it would be nice if
the version string for their autoconf gave some indication that it had
been fixed.
In any case, rm -f is what comes from standard autoconf 2.61, so I think
that's what should be there.
|
msg95557 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-11-20 17:08 |
One other thought: if this is applied, would it make sense to ask the
FreeBSD buildbot maintainer to then enable the POSIX semaphores for
FreeBSD 7.2 (but probably not for 6.4)? It looks like FreeBSD 8.0 is just
around the corner, and the rumours are that it'll have these semaphores
enabled by default; it would be nice to know about any multiprocessing
failures on FreeBSD, so that we can be reasonably sure it'll work on 8.0.
|
msg95558 - (view) |
Author: Jesse Noller (jnoller) * |
Date: 2009-11-20 17:11 |
On Fri, Nov 20, 2009 at 12:08 PM, Mark Dickinson <report@bugs.python.org> wrote:
> One other thought: if this is applied, would it make sense to ask the
> FreeBSD buildbot maintainer to then enable the POSIX semaphores for
> FreeBSD 7.2 (but probably not for 6.4)? It looks like FreeBSD 8.0 is just
> around the corner, and the rumours are that it'll have these semaphores
> enabled by default; it would be nice to know about any multiprocessing
> failures on FreeBSD, so that we can be reasonably sure it'll work on 8.0.
Agreed
|
msg95559 - (view) |
Author: Jesse Noller (jnoller) * |
Date: 2009-11-20 17:49 |
Mark, the patch looks ok on os/x and fedora core 12. Nothing jumps out at
me as incorrect. I'm for committing and watching the BSD buildbot.
|
msg95561 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-11-20 19:42 |
Thanks, Jesse. Applied in r76432 (trunk). Unfortunately the buildbots
are all purple right now; once they're building again, and the FreeBSD
buildbots have gone green (/me crosses fingers) I'll merge to py3k.
|
msg95573 - (view) |
Author: David Bolen (db3l) * |
Date: 2009-11-21 00:15 |
Looks like some sort of master side global rebuild was initiated but
without the proper SVN information. But I see a rebuild on 7.2 with
this patch revision that looks like it worked (still failed, but with a
different reason)
I'm not that familiar with the test harness, but would it be possible to
get test_multiprocessing to log an error when it has to be skipped (like
other tests that fail to find supporting modules and what not), so as to
highlight it in the log? It might even warn about what to do to fix the
behavior for someone running the tests.
In any event though, I'm fine with enabling the support on the 7.2
buildbot (I'll stick it in loader.conf so I don't have to remember after
a reboot) if we're past any point of wanting to check how the test
behaves without them. Or I guess the 6.4 buildbot can continue to serve
that purpose, right?
|
msg95581 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-11-21 14:00 |
> I'm not that familiar with the test harness, but would it be possible to
> get test_multiprocessing to log an error when it has to be skipped.
Well, there should be a skip message next to the test_multiprocessing line
in the results. I'm not sure whether that's the sort of thing you mean.
Unfortunately it looks like both runs (6.4 and 7.2) were prematurely
terminated by test_curses before they even got as far as
test_multiprocessing. I might try running them again and hope for a
different random test ordering.
I'm also seeing warnings about HAVE_BROKEN_POSIX_SEMAPHORES being
redefined, in Python/thread_pthread.h; I'll take another look at this
next week (no access to the FreeBSD machine at the moment).
|
msg95582 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-11-21 14:01 |
> In any event though, I'm fine with enabling the support on the 7.2
> buildbot (I'll stick it in loader.conf so I don't have to remember
after
a reboot)
That would be great---thank you!
> if we're past any point of wanting to check how the test
> behaves without them.
I'm not sure we are, just yet. But soon... :)
|
msg95785 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-11-28 10:27 |
Hmm. It seems that hijacking the existing HAVE_BROKEN_POSIX_SEMAPHORES
wasn't a good idea. I was surprised to find that OS X defines
_POSIX_SEMAPHORES to -1, indicating a lack of POSIX semaphore support.
|
msg95786 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-11-28 10:45 |
Variable clash should be fixed in r76558.
|
msg95787 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-11-28 12:54 |
Merged to py3k, release31-maint in r76566, r76567.
David, I think we're ready to enable POSIX semaphore support on the
FreeBSD 7.2 buildbot now, if you get the chance.
|
msg95799 - (view) |
Author: David Bolen (db3l) * |
Date: 2009-11-28 21:07 |
> David, I think we're ready to enable POSIX semaphore support on the
> FreeBSD 7.2 buildbot now, if you get the chance.
Done. I'll double check that the module remains loaded across restarts
when there's some idle time for a restart.
|
|
Date |
User |
Action |
Args |
2022-04-11 14:56:54 | admin | set | github: 51521 |
2009-11-28 21:07:20 | db3l | set | messages:
+ msg95799 |
2009-11-28 12:54:25 | mark.dickinson | set | status: open -> closed resolution: fixed messages:
+ msg95787
|
2009-11-28 10:45:03 | mark.dickinson | set | messages:
+ msg95786 |
2009-11-28 10:27:31 | mark.dickinson | set | messages:
+ msg95785 |
2009-11-21 14:01:37 | mark.dickinson | set | messages:
+ msg95582 |
2009-11-21 14:00:06 | mark.dickinson | set | assignee: mark.dickinson messages:
+ msg95581 |
2009-11-21 00:15:45 | db3l | set | messages:
+ msg95573 |
2009-11-20 19:42:26 | mark.dickinson | set | messages:
+ msg95561 |
2009-11-20 17:49:40 | jnoller | set | messages:
+ msg95559 |
2009-11-20 17:11:47 | jnoller | set | messages:
+ msg95558 |
2009-11-20 17:08:41 | mark.dickinson | set | messages:
+ msg95557 |
2009-11-20 17:05:16 | mark.dickinson | set | messages:
+ msg95556 |
2009-11-20 16:58:27 | jnoller | set | messages:
+ msg95555 |
2009-11-20 16:18:53 | mark.dickinson | set | stage: patch review |
2009-11-20 16:18:38 | mark.dickinson | set | files:
+ freebsd_multiprocessing.patch keywords:
+ patch messages:
+ msg95553
|
2009-11-20 14:23:03 | jnoller | set | messages:
+ msg95548 |
2009-11-20 14:01:58 | mark.dickinson | set | messages:
+ msg95545 |
2009-11-06 16:05:06 | mark.dickinson | set | messages:
+ msg94979 |
2009-11-06 15:58:48 | mark.dickinson | set | messages:
+ msg94978 |
2009-11-06 15:57:12 | mark.dickinson | create | |