Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issues with kqueue #76074

Closed
serhiy-storchaka opened this issue Oct 29, 2017 · 20 comments
Closed

Issues with kqueue #76074

serhiy-storchaka opened this issue Oct 29, 2017 · 20 comments
Assignees
Labels
3.7 (EOL) end of life extension-modules C modules in the Modules dir type-bug An unexpected behavior, bug, or error

Comments

@serhiy-storchaka
Copy link
Member

BPO 31893
Nosy @vstinner, @serhiy-storchaka, @koobs
PRs
  • bpo-31893: Fixed select.kqueue(). #4166
  • [3.6] bpo-31893: Fixed select.kqueue(). (GH-4166) #4190
  • [2.7] bpo-31893: Fixed select.kqueue(). (GH-4166) #4193
  • bpo-31893: Fix errors in b9052a0f91d2e83bbc27267247a5920c82b242a3. #4196
  • [3.6] bpo-31893: Fix errors in b9052a0f91d2e83bbc27267247a5920c82b242a3. (GH-4196) #4201
  • [2.7] bpo-31893: Fix errors in b9052a0f91d2e83bbc27267247a5920c82b242a3. (GH-4196) #4202
  • [2.7] bpo-31893: Fix a backporting error in 8cbf4e10646c3f5b8f0d274c2d7dea5bb6305f57. #4219
  • [2.7] bpo-31893: Fix a test for select.kevent. #4349
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/serhiy-storchaka'
    closed_at = <Date 2017-11-09.17:18:30.162>
    created_at = <Date 2017-10-29.11:44:41.930>
    labels = ['extension-modules', 'type-bug', '3.7']
    title = 'Issues with kqueue'
    updated_at = <Date 2017-11-09.17:18:30.161>
    user = 'https://github.com/serhiy-storchaka'

    bugs.python.org fields:

    activity = <Date 2017-11-09.17:18:30.161>
    actor = 'serhiy.storchaka'
    assignee = 'serhiy.storchaka'
    closed = True
    closed_date = <Date 2017-11-09.17:18:30.162>
    closer = 'serhiy.storchaka'
    components = ['Extension Modules', 'FreeBSD']
    creation = <Date 2017-10-29.11:44:41.930>
    creator = 'serhiy.storchaka'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 31893
    keywords = ['patch']
    message_count = 20.0
    messages = ['305187', '305290', '305294', '305295', '305300', '305305', '305309', '305312', '305325', '305326', '305329', '305385', '305948', '305949', '305950', '305958', '305964', '305965', '305966', '305971']
    nosy_count = 3.0
    nosy_names = ['vstinner', 'serhiy.storchaka', 'koobs']
    pr_nums = ['4166', '4190', '4193', '4196', '4201', '4202', '4219', '4349']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue31893'
    versions = ['Python 2.7', 'Python 3.6', 'Python 3.7']

    @serhiy-storchaka
    Copy link
    Member Author

    In Modules/selectmodule.c it is assumed that the kevent structure is defined on FreeBSD and NetBSD as:

         struct kevent {
                 uintptr_t ident;
                 short     filter;
                 u_short   flags;
                 u_int     fflags;
                 intptr_t  data;
                 uintptr_t udata;
         };

    and on OpenBSD as:

         struct kevent {
                 u_int     ident;
                 short     filter;
                 u_short   flags;
                 u_int     fflags;
                 intptr_t  data;
                 int       udata;
         };

    Actually it is defined on FreeBSD as:

         struct kevent {
                 uintptr_t ident;
                 short     filter;
                 u_short   flags;
                 u_int     fflags;
                 intptr_t  data;
                 void      *udata;
         };

    On OpenBSD as:

         struct kevent {
                 uintptr_t ident;
                 short     filter;
                 u_short   flags;
                 u_int     fflags;
                 int64_t   data;
                 void      *udata;
         };

    And on NetBSD as:

         struct kevent {
                 uintptr_t ident;
                 uint32_t  filter;
                 uint32_t  flags;
                 uint32_t  fflags;
                 int64_t   data;
                 intptr_t  udata;
         };

    Other issues are related to rich comparison. Due to integer overflows the ordering is not transitive. The rich comparison protocol is not properly supported, comparing a kevent_event object with a non-kevent_event object don't falls back to the rich comparison methods of the right arguments.

    @serhiy-storchaka serhiy-storchaka added the 3.7 (EOL) end of life label Oct 29, 2017
    @serhiy-storchaka serhiy-storchaka self-assigned this Oct 29, 2017
    @serhiy-storchaka serhiy-storchaka added extension-modules C modules in the Modules dir type-bug An unexpected behavior, bug, or error labels Oct 29, 2017
    @serhiy-storchaka
    Copy link
    Member Author

    New changeset b9052a0 by Serhiy Storchaka in branch 'master':
    bpo-31893: Fixed select.kqueue(). (bpo-4166)
    b9052a0

    @serhiy-storchaka
    Copy link
    Member Author

    New changeset f9a639b by Serhiy Storchaka (Miss Islington (bot)) in branch '3.6':
    bpo-31893: Fixed select.kqueue(). (GH-4166) (bpo-4190)
    f9a639b

    @vstinner
    Copy link
    Member

    The commit b9052a0 broke compilation on FreeBSD:

    building 'select' extension
    cc -pthread -fPIC -Wno-unused-result -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -I./Include -I. -I/usr/local/include -I/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Include -I/usr/home/buildbot/python/3.x.koobs-freebsd10/build -c /usr/home/buildbot/python/3.x.koobs-freebsd10/build/Modules/selectmodule.c -o build/temp.freebsd-10.3-STABLE-amd64-3.7-pydebug/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Modules/selectmodule.o
    /usr/home/buildbot/python/3.x.koobs-freebsd10/build/Modules/selectmodule.c:1889:63: error: expected ';' at end of declaration
    FILTER_FMT_UNIT FLAGS_FMT_UNIT FFLAGS_FMT_UNIT DATA_FMT_UNIT
    ^
    ;
    1 error generated.
    *** WARNING: renaming "_asyncio" since importing it failed: No module named 'select'

    http://buildbot.python.org/all/#/builders/87/builds/95/steps/3/logs/stdio

    @serhiy-storchaka
    Copy link
    Member Author

    New changeset 8cbf4e1 by Serhiy Storchaka in branch '2.7':
    [2.7] bpo-31893: Fixed select.kqueue(). (GH-4166) (bpo-4193)
    8cbf4e1

    @vstinner
    Copy link
    Member

    Oh, the commit also broke the "x86 Tiger 3.x buildbot:

    http://buildbot.python.org/all/#/builders/30/builds/93

    ======================================================================
    FAIL: test_create_event (test.test_kqueue.TestKQueue)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_kqueue.py", line 71, in test_create_event
        self.assertEqual(ev.data, 5)
    AssertionError: 25769803781 != 5

    @serhiy-storchaka
    Copy link
    Member Author

    Oh, I forgot that Mac OS X also in the BSD family.

    Thank you Victor for signaling errors.

    @serhiy-storchaka
    Copy link
    Member Author

    New changeset 2298fad by Serhiy Storchaka in branch 'master':
    bpo-31893: Fix errors in b9052a0. (bpo-4196)
    2298fad

    @serhiy-storchaka
    Copy link
    Member Author

    New changeset 84e252b by Serhiy Storchaka (Miss Islington (bot)) in branch '3.6':
    bpo-31893: Fix errors in b9052a0. (GH-4196) (bpo-4201)
    84e252b

    @serhiy-storchaka
    Copy link
    Member Author

    New changeset e7531e5 by Serhiy Storchaka (Miss Islington (bot)) in branch '2.7':
    bpo-31893: Fix errors in b9052a0. (GH-4196) (bpo-4202)
    e7531e5

    @serhiy-storchaka
    Copy link
    Member Author

    Tests on all buildbots are passed.

    @serhiy-storchaka
    Copy link
    Member Author

    New changeset ce51890 by Serhiy Storchaka in branch '2.7':
    bpo-31893: Fix a backporting error in 8cbf4e1. (bpo-4219)
    ce51890

    @vstinner
    Copy link
    Member

    vstinner commented Nov 9, 2017

    Test fails on x86 Tiger 2.7:

    http://buildbot.python.org/all/#/builders/59/builds/17

    ======================================================================
    FAIL: test_create_event (test.test_kqueue.TestKQueue)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/Users/db3l/buildarea/2.7.bolen-tiger/build/Lib/test/test_kqueue.py", line 39, in test_create_event
        self.assertRaises(TypeError, cmp, ev, None)
    AssertionError: TypeError not raised

    @vstinner vstinner reopened this Nov 9, 2017
    @vstinner
    Copy link
    Member

    vstinner commented Nov 9, 2017

    Same failure on x86-64 El Capitan 2.7/
    http://buildbot.python.org/all/#builders/98/builds/17

    ======================================================================
    FAIL: test_create_event (test.test_kqueue.TestKQueue)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/Users/buildbot/buildarea/2.7.billenstein-elcapitan/build/Lib/test/test_kqueue.py", line 39, in test_create_event
        self.assertRaises(TypeError, cmp, ev, None)
    AssertionError: TypeError not raised

    @vstinner
    Copy link
    Member

    vstinner commented Nov 9, 2017

    On also AMD64 FreeBSD 10.x Shared 2.7:

    http://buildbot.python.org/all/#/builders/97/builds/17

    ======================================================================
    FAIL: test_create_event (test.test_kqueue.TestKQueue)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/usr/home/buildbot/python/2.7.koobs-freebsd10/build/Lib/test/test_kqueue.py", line 39, in test_create_event
        self.assertRaises(TypeError, cmp, ev, None)
    AssertionError: TypeError not raised

    @serhiy-storchaka
    Copy link
    Member Author

    Oh, I didn't test on 2.7.

    The old code contradicted the common comparison behavior. It allowed comparing select.kevent with other objects only for equality as for identity. kevent == other always returned False even if other == kevent returned True. kevent < other raised a TypeError.

    The new code uses the default behavior. kevent == other returns the same as other == kevent (False by default if types are not comparable, but may return True if other.__eq__(kevent) returns True). kevent < other returns the same as other > kevent. E.g. raises a TypeError by default in Python 3. But in Python 2 all objects are comparable by default, and this breaks a test which expects that select.kevent is not comparable.

    There are two ways to fix this:

    1. Make select.kevent non-comparable again. This contradicts the default behavior and I don't know reasons why it should be non-comparable, but this was an existing behavior.

    2. Remove the tests or change them to test that comparing select.kevent with other object doesn't raise an error.

    @vstinner
    Copy link
    Member

    vstinner commented Nov 9, 2017

    kqueue_event_richcompare() returns NotImplemented if other is not an event:

        if (!kqueue_event_Check(o)) {
            Py_INCREF(Py_NotImplemented);
            return Py_NotImplemented;
        }

    So I don't understand why tests started to fail with the commit ce51890.

    @serhiy-storchaka
    Copy link
    Member Author

    Because the default behavior is different in Python 2 and Python 3. In Python 3 this is a TypeError. In Python 2 all objects are comparable by default.

    @vstinner
    Copy link
    Member

    vstinner commented Nov 9, 2017

    I suggest to fix the tests, the code looks good to me.

    @serhiy-storchaka
    Copy link
    Member Author

    Fixed by PR 4349 (changeset 15e1453).

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life extension-modules C modules in the Modules dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    2 participants