classification
Title: Broken test_kqueue.py on OpenBSD
Type: behavior Stage: resolved
Components: Library (Lib), Tests Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: henry.precheur, neologix, python-dev, rpointel, skrah, vstinner
Priority: normal Keywords: needs review, patch

Created on 2009-07-04 21:23 by henry.precheur, last changed 2013-04-05 07:23 by neologix. This issue is now closed.

Files
File name Uploaded Description Edit
patch-Lib_test_test_kqueue_py henry.precheur, 2009-07-04 21:23 patch to fix test
kqueue_flags.diff neologix, 2013-03-03 14:22 review
Messages (12)
msg90127 - (view) Author: Henry Precheur (henry.precheur) Date: 2009-07-04 21:23
A kqueue's test doesn't pass on OpenBSD 4.6-beta, 4.4, & 4.5:

FAILED (failures=1)
Traceback (most recent call last):
  File "Lib/test/test_kqueue.py", line 186, in <module>
    test_main()
  File "Lib/test/test_kqueue.py", line 183, in test_main
    support.run_unittest(TestKQueue)
  File "/home/henry/py3k/Lib/test/support.py", line 882, in run_unittest
    _run_suite(suite)
  File "/home/henry/py3k/Lib/test/support.py", line 865, in _run_suite
    raise TestFailed(err)
test.support.TestFailed: Traceback (most recent call last):
  File "Lib/test/test_kqueue.py", line 119, in test_queue_event
    (server.fileno(), select.KQ_FILTER_WRITE, flags)])
AssertionError: Lists differ: [(6, -2, 5), (7, -2, 5)] != [(6, -2, 0),
(7, -2, 0)]

First differing element 0:
(6, -2, 5)
(6, -2, 0)

- [(6, -2, 5), (7, -2, 5)]
?          ^           ^

+ [(6, -2, 0), (7, -2, 0)]
?          ^           ^

It looks like OpenBSD behaves like Darwin. The attached patch fixes the
test.
msg105694 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2010-05-14 09:19
The same patch was applied to the Python port by OpenBSD developer Damien Miller:

http://www.openbsd.org/cgi-bin/cvsweb/ports/lang/python/2.6/patches/patch-Lib_test_test_kqueue_py


Mark, would it be ok in this situation to apply the patch even though
personally I don't know if OpenBSD kqueue behaves like Darwin kqueue?
msg105713 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2010-05-14 14:46
Yes, please do apply the patch!
msg105728 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2010-05-14 17:58
Mark, thanks. - The patch is good on OpenBSD-4.5-i386-Celeron,
but I get additional failures on OpenBSD-4.7-beta-amd64-QEMU.

This could be the result of running a beta under qemu.

Henry, could you confirm if the patch works on amd64/OpenBSD-4.7-stable?



======================================================================
FAIL: test_create_event (test.test_kqueue.TestKQueue)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/stefan/svn/py3k/Lib/test/test_kqueue.py", line 29, in test_create_event
    self.assertEqual(ev.ident, fd)
AssertionError: 562945658454018 != 2

======================================================================
FAIL: test_queue_event (test.test_kqueue.TestKQueue)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/stefan/svn/py3k/Lib/test/test_kqueue.py", line 130, in test_queue_event
    (server.fileno(), select.KQ_FILTER_WRITE, flags)])
AssertionError: Lists differ: [(1688841270329350, -2, 5), (1... != [(6, -2, 0), (7, -2, 0)]

First differing element 0:
(1688841270329350, -2, 5)
(6, -2, 0)

- [(1688841270329350, -2, 5), (1688841270329351, -2, 5)]
+ [(6, -2, 0), (7, -2, 0)]
msg105990 - (view) Author: Henry Precheur (henry.precheur) Date: 2010-05-18 17:19
The patch works well with on amd64/OpenBSD-current (CVS from May 14 or
15).

I don't have access to a 4.7-stable right now.

On Fri, May 14, 2010 at 05:58:10PM +0000, Stefan Krah wrote:
> 
> Stefan Krah <stefan-usenet@bytereef.org> added the comment:
> 
> Mark, thanks. - The patch is good on OpenBSD-4.5-i386-Celeron,
> but I get additional failures on OpenBSD-4.7-beta-amd64-QEMU.
> 
> This could be the result of running a beta under qemu.
> 
> Henry, could you confirm if the patch works on amd64/OpenBSD-4.7-stable?
> 
> 
> 
> ======================================================================
> FAIL: test_create_event (test.test_kqueue.TestKQueue)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/home/stefan/svn/py3k/Lib/test/test_kqueue.py", line 29, in test_create_event
>     self.assertEqual(ev.ident, fd)
> AssertionError: 562945658454018 != 2
> 
> ======================================================================
> FAIL: test_queue_event (test.test_kqueue.TestKQueue)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/home/stefan/svn/py3k/Lib/test/test_kqueue.py", line 130, in test_queue_event
>     (server.fileno(), select.KQ_FILTER_WRITE, flags)])
> AssertionError: Lists differ: [(1688841270329350, -2, 5), (1... != [(6, -2, 0), (7, -2, 0)]
> 
> First differing element 0:
> (1688841270329350, -2, 5)
> (6, -2, 0)
> 
> - [(1688841270329350, -2, 5), (1688841270329351, -2, 5)]
> + [(6, -2, 0), (7, -2, 0)]
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue6419>
> _______________________________________
msg141107 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2011-07-25 17:50
Stephan, did you apply the patch?

I think the failures you were experiencing on amd64 could be due to issue #12181: OpenBSD's struct kevent definition differs from FreeBSD/NetBSD, which can trigger a segfault on sparc64 (which doesn't allow unaligned access).

By the way, Henry, would you be interested in writting a patch for issue #12181?
msg141117 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2011-07-25 19:48
No, I didn't commit it. Currently I don't have an amd64 OpenBSD install to do further testing.

The problem could be #12181, it could also be qemu, which sometimes has
issues with OpenBSD.
msg142905 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2011-08-24 19:53
Rémi, can you reproduce this problem?
Could you test the patch attached?
(It's maybe already part of OPenBSD's tree, in which case it would certainly be nice to include it).
msg142957 - (view) Author: Remi Pointel (rpointel) * Date: 2011-08-25 06:57
Hi,
I think this issue is relied to issue 12181.

I have made tests.

################
1)with no patch:
################
testPair (__main__.TestKQueue) ... ok
test_create_event (__main__.TestKQueue) ... FAIL
test_create_queue (__main__.TestKQueue) ... ok
test_queue_event (__main__.TestKQueue) ... FAIL

======================================================================
FAIL: test_create_event (__main__.TestKQueue)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/ports/pobj/Python-2.7.1/Python-2.7.1/Lib/test/test_kqueue.py", line 28, in test_create_event
    self.assertEqual(ev.ident, fd)
AssertionError: 562945658454018L != 2

======================================================================
FAIL: test_queue_event (__main__.TestKQueue)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/ports/pobj/Python-2.7.1/Python-2.7.1/Lib/test/test_kqueue.py", line 129, in test_queue_event
    (server.fileno(), select.KQ_FILTER_WRITE, flags)])
AssertionError: Lists differ: [(1688841270329350L, -2, 5L), ... != [(6, -2, 0), (7, -2, 0)]

First differing element 0:
(1688841270329350L, -2, 5L)
(6, -2, 0)

- [(1688841270329350L, -2, 5L), (1688841270329351L, -2, 5L)]
+ [(6, -2, 0), (7, -2, 0)]

----------------------------------------------------------------------
Ran 4 tests in 0.002s

FAILED (failures=2)
Traceback (most recent call last):
  File "/usr/ports/pobj/Python-2.7.1/Python-2.7.1/Lib/test/test_kqueue.py", line 196, in <module>
    test_main()
  File "/usr/ports/pobj/Python-2.7.1/Python-2.7.1/Lib/test/test_kqueue.py", line 193, in test_main
    test_support.run_unittest(TestKQueue)
  File "/usr/ports/pobj/Python-2.7.1/Python-2.7.1/Lib/test/test_support.py", line 1087, in run_unittest
    _run_suite(suite)
  File "/usr/ports/pobj/Python-2.7.1/Python-2.7.1/Lib/test/test_support.py", line 1070, in _run_suite
    raise TestFailed(err)
test.test_support.TestFailed: multiple errors occurred

#####################################
2)with patch-Lib_test_test_kqueue_py:
#####################################
testPair (__main__.TestKQueue) ... ok
test_create_event (__main__.TestKQueue) ... FAIL
test_create_queue (__main__.TestKQueue) ... ok
test_queue_event (__main__.TestKQueue) ... FAIL

======================================================================
FAIL: test_create_event (__main__.TestKQueue)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/ports/pobj/Python-2.7.1/Python-2.7.1/Lib/test/test_kqueue.py", line 28, in test_create_event
    self.assertEqual(ev.ident, fd)
AssertionError: 562945658454018L != 2

======================================================================
FAIL: test_queue_event (__main__.TestKQueue)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/ports/pobj/Python-2.7.1/Python-2.7.1/Lib/test/test_kqueue.py", line 130, in test_queue_event
    (server.fileno(), select.KQ_FILTER_WRITE, flags)])
AssertionError: Lists differ: [(1688841270329350L, -2, 5L), ... != [(6, -2, 5), (7, -2, 5)]

First differing element 0:
(1688841270329350L, -2, 5L)
(6, -2, 5)

- [(1688841270329350L, -2, 5L), (1688841270329351L, -2, 5L)]
+ [(6, -2, 5), (7, -2, 5)]

----------------------------------------------------------------------
Ran 4 tests in 0.002s

FAILED (failures=2)
Traceback (most recent call last):
  File "/usr/ports/pobj/Python-2.7.1/Python-2.7.1/Lib/test/test_kqueue.py", line 197, in <module>
    test_main()
  File "/usr/ports/pobj/Python-2.7.1/Python-2.7.1/Lib/test/test_kqueue.py", line 194, in test_main
    test_support.run_unittest(TestKQueue)
  File "/usr/ports/pobj/Python-2.7.1/Python-2.7.1/Lib/test/test_support.py", line 1087, in run_unittest
    _run_suite(suite)
  File "/usr/ports/pobj/Python-2.7.1/Python-2.7.1/Lib/test/test_support.py", line 1070, in _run_suite
    raise TestFailed(err)
test.test_support.TestFailed: multiple errors occurred

#####################################################
3)with patch-Modules_selectmodule_c (see issue 12181)
#####################################################
testPair (__main__.TestKQueue) ... ok
test_create_event (__main__.TestKQueue) ... ERROR
test_create_queue (__main__.TestKQueue) ... ok
test_queue_event (__main__.TestKQueue) ... FAIL

======================================================================
ERROR: test_create_event (__main__.TestKQueue)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/ports/pobj/Python-2.7.1/Python-2.7.1/Lib/test/test_kqueue.py", line 74, in test_create_event
    ev = select.kevent(bignum, 1, 2, 3, sys.maxsize, bignum)
OverflowError: signed integer is greater than maximum

======================================================================
FAIL: test_queue_event (__main__.TestKQueue)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/ports/pobj/Python-2.7.1/Python-2.7.1/Lib/test/test_kqueue.py", line 129, in test_queue_event
    (server.fileno(), select.KQ_FILTER_WRITE, flags)])
AssertionError: Lists differ: [(6L, -2, 5L), (7L, -2, 5L)] != [(6, -2, 0), (7, -2, 0)]

First differing element 0:
(6L, -2, 5L)
(6, -2, 0)

- [(6L, -2, 5L), (7L, -2, 5L)]
?    -      ^^     -      ^^

+ [(6, -2, 0), (7, -2, 0)]
?          ^           ^


----------------------------------------------------------------------
Ran 4 tests in 0.002s

FAILED (failures=1, errors=1)
Traceback (most recent call last):
  File "/usr/ports/pobj/Python-2.7.1/Python-2.7.1/Lib/test/test_kqueue.py", line 196, in <module>
    test_main()
  File "/usr/ports/pobj/Python-2.7.1/Python-2.7.1/Lib/test/test_kqueue.py", line 193, in test_main
    test_support.run_unittest(TestKQueue)
  File "/usr/ports/pobj/Python-2.7.1/Python-2.7.1/Lib/test/test_support.py", line 1087, in run_unittest
    _run_suite(suite)
  File "/usr/ports/pobj/Python-2.7.1/Python-2.7.1/Lib/test/test_support.py", line 1070, in _run_suite
    raise TestFailed(err)
test.test_support.TestFailed: multiple errors occurred

#####################################################################
4)with patch-Modules_selectmodule_c and patch-Lib_test_test_kqueue_py
#####################################################################
BRARY_PATH=/usr/ports/pobj/Python-2.7.1/Python-2.7.1 /usr/ports/pobj/Python-2.7.1/Python-2.7.1/python /usr/ports/pobj/Python-2.7.1/Python-2.7.1/Lib/test/test_kqueue.py
testPair (__main__.TestKQueue) ... ok
test_create_event (__main__.TestKQueue) ... ERROR
test_create_queue (__main__.TestKQueue) ... ok
test_queue_event (__main__.TestKQueue) ... ok

======================================================================
ERROR: test_create_event (__main__.TestKQueue)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/ports/pobj/Python-2.7.1/Python-2.7.1/Lib/test/test_kqueue.py", line 74, in test_create_event
    ev = select.kevent(bignum, 1, 2, 3, sys.maxsize, bignum)
OverflowError: signed integer is greater than maximum

----------------------------------------------------------------------
Ran 4 tests in 0.001s

FAILED (errors=1)
Traceback (most recent call last):
  File "/usr/ports/pobj/Python-2.7.1/Python-2.7.1/Lib/test/test_kqueue.py", line 197, in <module>
    test_main()
  File "/usr/ports/pobj/Python-2.7.1/Python-2.7.1/Lib/test/test_kqueue.py", line 194, in test_main
    test_support.run_unittest(TestKQueue)
  File "/usr/ports/pobj/Python-2.7.1/Python-2.7.1/Lib/test/test_support.py", line 1087, in run_unittest
    _run_suite(suite)
  File "/usr/ports/pobj/Python-2.7.1/Python-2.7.1/Lib/test/test_support.py", line 1070, in _run_suite
    raise TestFailed(err)
test.test_support.TestFailed: Traceback (most recent call last):
  File "/usr/ports/pobj/Python-2.7.1/Python-2.7.1/Lib/test/test_kqueue.py", line 74, in test_create_event
    ev = select.kevent(bignum, 1, 2, 3, sys.maxsize, bignum)
OverflowError: signed integer is greater than maximum


Don't hesitate if you need more informations, but I think we must close issue 12181 before.

Thanks a lot,
Remi.
msg183378 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-03-03 14:22
The test assumes that the input flags (EV_ADD, EV_ENABLE...) will be returned in the output events. It's apparently not the case on OpenBSD and NetBSD (and probably on OS-X neither, because this check is disabled on this platform), and I can't see anything in the kqueue man pages hinting to this behavior (only specific flags will be set on output, like EV_ERROR). Knowing which flag was specified is not really useful actually.

The patch attached just removes this check (we keep checking the file descriptors and filter, of course).
msg185639 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-03-31 18:38
New changeset cfd4cd15809e by Charles-Francois Natali in branch '2.7':
Issue #6419: Fix a test_kqueue failure on some BSD flavors.
http://hg.python.org/cpython/rev/cfd4cd15809e

New changeset 96776fc3cbcc by Charles-Francois Natali in branch '3.3':
Issue #6419: Fix a test_kqueue failure on some BSD flavors.
http://hg.python.org/cpython/rev/96776fc3cbcc

New changeset 0c9415ddf403 by Charles-Francois Natali in branch 'default':
Issue #6419: Fix a test_kqueue failure on some BSD flavors.
http://hg.python.org/cpython/rev/0c9415ddf403
msg186065 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-04-05 07:23
test_kqueue now passes on NetBSD (can't tell for OpenBSD, all buildbots are offline).

I'm closing this, feel free to repon in case of problem.
History
Date User Action Args
2013-04-05 07:23:43neologixsetstatus: open -> closed
resolution: fixed
messages: + msg186065

stage: patch review -> resolved
2013-03-31 18:38:59python-devsetnosy: + python-dev
messages: + msg185639
2013-03-05 10:53:39mark.dickinsonsetnosy: - mark.dickinson
2013-03-03 14:22:10neologixsetkeywords: + patch, needs review
files: + kqueue_flags.diff
messages: + msg183378
2013-01-17 17:24:26neologixlinkissue16952 superseder
2011-08-25 06:57:44rpointelsetmessages: + msg142957
2011-08-24 19:53:49neologixsetnosy: + rpointel
messages: + msg142905
2011-07-25 23:22:00pitrousetversions: + Python 3.3, - Python 2.6, Python 3.1
nosy: + vstinner

components: + Library (Lib)
type: behavior
stage: patch review
2011-07-25 19:48:00skrahsetmessages: + msg141117
2011-07-25 17:50:15neologixsetnosy: + neologix
messages: + msg141107
2010-05-18 17:19:03henry.precheursetmessages: + msg105990
2010-05-14 17:58:08skrahsetmessages: + msg105728
2010-05-14 14:46:07mark.dickinsonsetmessages: + msg105713
2010-05-14 09:19:40skrahsetnosy: + mark.dickinson
messages: + msg105694
2010-05-13 14:03:34skrahsetnosy: + skrah
2009-07-04 21:40:00henry.precheursetcomponents: + Tests
2009-07-04 21:23:00henry.precheurcreate