classification
Title: Issues with kqueue
Type: behavior Stage: resolved
Components: Extension Modules, FreeBSD Versions: Python 3.7, Python 3.6, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: koobs, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2017-10-29 11:44 by serhiy.storchaka, last changed 2017-11-09 17:18 by serhiy.storchaka. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 4166 merged serhiy.storchaka, 2017-10-29 11:52
PR 4190 merged python-dev, 2017-10-31 12:01
PR 4193 merged serhiy.storchaka, 2017-10-31 12:32
PR 4196 merged serhiy.storchaka, 2017-10-31 14:40
PR 4201 merged python-dev, 2017-10-31 16:19
PR 4202 merged python-dev, 2017-10-31 16:20
PR 4219 merged serhiy.storchaka, 2017-11-01 14:37
PR 4349 merged serhiy.storchaka, 2017-11-09 15:55
Messages (20)
msg305187 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-10-29 11:44
In Modules/selectmodule.c it is assumed that the kevent structure is defined on FreeBSD and NetBSD as:

     struct kevent {
             uintptr_t ident;
             short     filter;
             u_short   flags;
             u_int     fflags;
             intptr_t  data;
             uintptr_t udata;
     };

and on OpenBSD as:

     struct kevent {
             u_int     ident;
             short     filter;
             u_short   flags;
             u_int     fflags;
             intptr_t  data;
             int       udata;
     };


Actually it is defined on FreeBSD as:

     struct kevent {
             uintptr_t ident;
             short     filter;
             u_short   flags;
             u_int     fflags;
             intptr_t  data;
             void      *udata;
     };

On OpenBSD as:

     struct kevent {
             uintptr_t ident;
             short     filter;
             u_short   flags;
             u_int     fflags;
             int64_t   data;
             void      *udata;
     };

And on NetBSD as:

     struct kevent {
             uintptr_t ident;
             uint32_t  filter;
             uint32_t  flags;
             uint32_t  fflags;
             int64_t   data;
             intptr_t  udata;
     };

Other issues are related to rich comparison. Due to integer overflows the ordering is not transitive. The rich comparison protocol is not properly supported, comparing a kevent_event object with a non-kevent_event object don't falls back to the rich comparison methods of the right arguments.
msg305290 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-10-31 11:59
New changeset b9052a0f91d2e83bbc27267247a5920c82b242a3 by Serhiy Storchaka in branch 'master':
bpo-31893: Fixed select.kqueue(). (#4166)
https://github.com/python/cpython/commit/b9052a0f91d2e83bbc27267247a5920c82b242a3
msg305294 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-10-31 12:46
New changeset f9a639b97c760f40d022223c7655053c89752850 by Serhiy Storchaka (Miss Islington (bot)) in branch '3.6':
bpo-31893: Fixed select.kqueue(). (GH-4166) (#4190)
https://github.com/python/cpython/commit/f9a639b97c760f40d022223c7655053c89752850
msg305295 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-10-31 12:48
The commit b9052a0f91d2e83bbc27267247a5920c82b242a3 broke compilation on FreeBSD:

building 'select' extension
cc -pthread -fPIC -Wno-unused-result -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -I./Include -I. -I/usr/local/include -I/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Include -I/usr/home/buildbot/python/3.x.koobs-freebsd10/build -c /usr/home/buildbot/python/3.x.koobs-freebsd10/build/Modules/selectmodule.c -o build/temp.freebsd-10.3-STABLE-amd64-3.7-pydebug/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Modules/selectmodule.o
/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Modules/selectmodule.c:1889:63: error: expected ';' at end of declaration
                FILTER_FMT_UNIT FLAGS_FMT_UNIT FFLAGS_FMT_UNIT DATA_FMT_UNIT
                                                              ^
                                                              ;
1 error generated.
*** WARNING: renaming "_asyncio" since importing it failed: No module named 'select'

http://buildbot.python.org/all/#/builders/87/builds/95/steps/3/logs/stdio
msg305300 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-10-31 14:13
New changeset 8cbf4e10646c3f5b8f0d274c2d7dea5bb6305f57 by Serhiy Storchaka in branch '2.7':
[2.7] bpo-31893: Fixed select.kqueue(). (GH-4166) (#4193)
https://github.com/python/cpython/commit/8cbf4e10646c3f5b8f0d274c2d7dea5bb6305f57
msg305305 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-10-31 14:27
Oh, the commit also broke the "x86 Tiger 3.x buildbot:

http://buildbot.python.org/all/#/builders/30/builds/93

======================================================================
FAIL: test_create_event (test.test_kqueue.TestKQueue)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_kqueue.py", line 71, in test_create_event
    self.assertEqual(ev.data, 5)
AssertionError: 25769803781 != 5
msg305309 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-10-31 14:56
Oh, I forgot that Mac OS X also in the BSD family.

Thank you Victor for signaling errors.
msg305312 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-10-31 16:18
New changeset 2298fad5ff907dd48ea0fb5c71fa22334ef28c6b by Serhiy Storchaka in branch 'master':
bpo-31893: Fix errors in b9052a0f91d2e83bbc27267247a5920c82b242a3. (#4196)
https://github.com/python/cpython/commit/2298fad5ff907dd48ea0fb5c71fa22334ef28c6b
msg305325 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-10-31 18:16
New changeset 84e252b79eed94bc9e9175f82191322c89e489ad by Serhiy Storchaka (Miss Islington (bot)) in branch '3.6':
bpo-31893: Fix errors in b9052a0f91d2e83bbc27267247a5920c82b242a3. (GH-4196) (#4201)
https://github.com/python/cpython/commit/84e252b79eed94bc9e9175f82191322c89e489ad
msg305326 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-10-31 18:16
New changeset e7531e54bf195b8d3ed35b4138901c82f7ed794c by Serhiy Storchaka (Miss Islington (bot)) in branch '2.7':
bpo-31893: Fix errors in b9052a0f91d2e83bbc27267247a5920c82b242a3. (GH-4196) (#4202)
https://github.com/python/cpython/commit/e7531e54bf195b8d3ed35b4138901c82f7ed794c
msg305329 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-10-31 19:07
Tests on all buildbots are passed.
msg305385 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-01 15:43
New changeset ce51890894be46f8f9d991a1d0ea1455fc41ccdc by Serhiy Storchaka in branch '2.7':
bpo-31893: Fix a backporting error in 8cbf4e10646c3f5b8f0d274c2d7dea5bb6305f57. (#4219)
https://github.com/python/cpython/commit/ce51890894be46f8f9d991a1d0ea1455fc41ccdc
msg305948 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-09 09:59
Test fails on x86 Tiger 2.7:

http://buildbot.python.org/all/#/builders/59/builds/17

======================================================================
FAIL: test_create_event (test.test_kqueue.TestKQueue)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/db3l/buildarea/2.7.bolen-tiger/build/Lib/test/test_kqueue.py", line 39, in test_create_event
    self.assertRaises(TypeError, cmp, ev, None)
AssertionError: TypeError not raised
msg305949 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-09 10:02
Same failure on x86-64 El Capitan 2.7/
http://buildbot.python.org/all/#builders/98/builds/17

======================================================================
FAIL: test_create_event (test.test_kqueue.TestKQueue)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/2.7.billenstein-elcapitan/build/Lib/test/test_kqueue.py", line 39, in test_create_event
    self.assertRaises(TypeError, cmp, ev, None)
AssertionError: TypeError not raised
msg305950 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-09 10:03
On also  AMD64 FreeBSD 10.x Shared 2.7:

http://buildbot.python.org/all/#/builders/97/builds/17

======================================================================
FAIL: test_create_event (test.test_kqueue.TestKQueue)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/2.7.koobs-freebsd10/build/Lib/test/test_kqueue.py", line 39, in test_create_event
    self.assertRaises(TypeError, cmp, ev, None)
AssertionError: TypeError not raised
msg305958 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-09 11:44
Oh, I didn't test on 2.7.

The old code contradicted the common comparison behavior. It allowed comparing select.kevent with other objects only for equality as for identity. `kevent == other` always returned False even if `other == kevent` returned True. `kevent < other` raised a TypeError.

The new code uses the default behavior. `kevent == other` returns the same as `other == kevent` (False by default if types are not comparable, but may return True if `other.__eq__(kevent)` returns True). `kevent < other` returns the same as `other > kevent`. E.g. raises a TypeError by default in Python 3. But in Python 2 all objects are comparable by default, and this breaks a test which expects that select.kevent is not comparable.

There are two ways to fix this:

1. Make select.kevent non-comparable again. This contradicts the default behavior and I don't know reasons why it should be non-comparable, but this was an existing behavior.

2. Remove the tests or change them to test that comparing select.kevent with other object doesn't raise an error.
msg305964 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-09 15:09
kqueue_event_richcompare() returns NotImplemented if other is not an event:

    if (!kqueue_event_Check(o)) {
        Py_INCREF(Py_NotImplemented);
        return Py_NotImplemented;
    }

So I don't understand why tests started to fail with the commit ce51890894be46f8f9d991a1d0ea1455fc41ccdc.
msg305965 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-09 15:36
Because the default behavior is different in Python 2 and Python 3. In Python 3 this is a TypeError. In Python 2 all objects are comparable by default.
msg305966 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-09 15:39
I suggest to fix the tests, the code looks good to me.
msg305971 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-09 17:18
Fixed by PR 4349 (changeset 15e14538f90cabc87473a489316fdb81af76cfb2).
History
Date User Action Args
2017-11-09 17:18:30serhiy.storchakasetstatus: open -> closed
resolution: fixed
messages: + msg305971

stage: patch review -> resolved
2017-11-09 15:55:54serhiy.storchakasetstage: resolved -> patch review
pull_requests: + pull_request4306
2017-11-09 15:39:04vstinnersetmessages: + msg305966
2017-11-09 15:36:05serhiy.storchakasetmessages: + msg305965
2017-11-09 15:09:28vstinnersetmessages: + msg305964
2017-11-09 11:44:17serhiy.storchakasetmessages: + msg305958
2017-11-09 10:03:27vstinnersetmessages: + msg305950
2017-11-09 10:02:20vstinnersetmessages: + msg305949
2017-11-09 09:59:17vstinnersetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg305948
2017-11-01 15:43:13serhiy.storchakasetmessages: + msg305385
2017-11-01 14:37:27serhiy.storchakasetpull_requests: + pull_request4187
2017-10-31 19:07:08serhiy.storchakasetstatus: open -> closed
resolution: fixed
messages: + msg305329

stage: patch review -> resolved
2017-10-31 18:16:13serhiy.storchakasetmessages: + msg305326
2017-10-31 18:16:10serhiy.storchakasetmessages: + msg305325
2017-10-31 16:20:34python-devsetpull_requests: + pull_request4171
2017-10-31 16:19:31python-devsetpull_requests: + pull_request4169
2017-10-31 16:18:24serhiy.storchakasetmessages: + msg305312
2017-10-31 14:56:12serhiy.storchakasetmessages: + msg305309
2017-10-31 14:40:29serhiy.storchakasetpull_requests: + pull_request4166
2017-10-31 14:27:33vstinnersetmessages: + msg305305
2017-10-31 14:13:54serhiy.storchakasetmessages: + msg305300
2017-10-31 12:48:18vstinnersetnosy: + vstinner
messages: + msg305295
2017-10-31 12:46:17serhiy.storchakasetmessages: + msg305294
2017-10-31 12:32:22serhiy.storchakasetpull_requests: + pull_request4163
2017-10-31 12:01:04python-devsetpull_requests: + pull_request4160
2017-10-31 11:59:57serhiy.storchakasetmessages: + msg305290
2017-10-29 11:52:58serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request4135
2017-10-29 11:44:41serhiy.storchakacreate