Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test_os: test_get_set_inheritable_o_path() failed on AMD64 FreeBSD Shared 3.x #89012

Closed
vstinner opened this issue Aug 6, 2021 · 11 comments
Closed
Labels
3.9 only security fixes 3.10 only security fixes 3.11 only security fixes tests Tests in the Lib/test dir

Comments

@vstinner
Copy link
Member

vstinner commented Aug 6, 2021

BPO 44849
Nosy @vstinner, @ambv, @koobs, @pablogsal, @miss-islington, @erlend-aasland
PRs
  • bpo-44849: Fix os.set_inheritable() on FreeBSD 14 with O_PATH #27623
  • [3.9] bpo-44849: Fix os.set_inheritable() on FreeBSD 14 with O_PATH (GH-27623) #27630
  • [3.10] bpo-44849: Fix os.set_inheritable() on FreeBSD 14 with O_PATH (GH-27623) #27631
  • [3.8] bpo-44849: Fix os.set_inheritable() on FreeBSD 14 with O_PATH (GH-27623) #28978
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2021-08-06.15:14:21.237>
    created_at = <Date 2021-08-06.10:40:46.152>
    labels = ['tests', '3.9', '3.10', '3.11']
    title = 'test_os: test_get_set_inheritable_o_path() failed on AMD64 FreeBSD Shared 3.x'
    updated_at = <Date 2021-10-19.19:15:33.700>
    user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2021-10-19.19:15:33.700>
    actor = 'lukasz.langa'
    assignee = 'none'
    closed = True
    closed_date = <Date 2021-08-06.15:14:21.237>
    closer = 'vstinner'
    components = ['Tests']
    creation = <Date 2021-08-06.10:40:46.152>
    creator = 'vstinner'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 44849
    keywords = ['patch']
    message_count = 11.0
    messages = ['399065', '399067', '399069', '399085', '399088', '399089', '399167', '399168', '399169', '399264', '404340']
    nosy_count = 6.0
    nosy_names = ['vstinner', 'lukasz.langa', 'koobs', 'pablogsal', 'miss-islington', 'erlendaasland']
    pr_nums = ['27623', '27630', '27631', '28978']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue44849'
    versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

    @vstinner
    Copy link
    Member Author

    vstinner commented Aug 6, 2021

    Since build 655 (commit 6871fd0), test_os failed on AMD64 FreeBSD Shared 3.x:
    https://buildbot.python.org/all/#/builders/483/builds/655

    ======================================================================
    ERROR: test_get_set_inheritable_o_path (test.test_os.FDInheritanceTests)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_os.py", line 3898, in test_get_set_inheritable_o_path
        os.set_inheritable(fd, True)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    OSError: [Errno 9] Bad file descriptor

    @vstinner vstinner added 3.11 only security fixes tests Tests in the Lib/test dir labels Aug 6, 2021
    @vstinner
    Copy link
    Member Author

    vstinner commented Aug 6, 2021

    test_get_set_inheritable_o_path() also fails on:

    • AMD64 FreeBSD Shared 3.10
    • AMD64 FreeBSD Shared 3.9

    It smells more like a FreeBSD system update rather than a Python regression.

    @vstinner
    Copy link
    Member Author

    vstinner commented Aug 6, 2021

    On my FreeBSD 13 VM, the test is skipped:

    vstinner@freebsd$ ./python -m test -v test_os -m test_get_set_inheritable_o_path
    == CPython 3.11.0a0 (heads/main:938e84b4fa, Aug 6 2021, 10:46:32) [Clang 11.0.1 (git@github.com:llvm/llvm-project.git llvmorg-11.0.1-0-g43ff75f2c3
    == FreeBSD-13.0-RELEASE-amd64-64bit-ELF little-endian
    (...)
    test_get_set_inheritable_o_path (test.test_os.FDInheritanceTests) ... skipped 'need os.O_PATH'

    It seems like O_PATH was introduced in FreeBSD 14 CURRENT, but the test fails.

    Code of the test:

    @unittest.skipUnless(hasattr(os, 'O_PATH'), "need os.O_PATH")
    def test_get_set_inheritable_o_path(self):
        fd = os.open(__file__, os.O_PATH)
        self.addCleanup(os.close, fd)
        self.assertEqual(os.get_inheritable(fd), False)
    
            os.set_inheritable(fd, True)  # <==== OSError: [Errno 9] Bad file descriptor
            self.assertEqual(os.get_inheritable(fd), True)
    
            os.set_inheritable(fd, False)
            self.assertEqual(os.get_inheritable(fd), False)

    os.set_inheritable() calls _Py_set_inheritable() whichs has multiple implementations:

    • ioctl(fd, FIONCLEX, NULL)
    • fcntl(fd, F_GETFD) + fcntl(fd, F_SETFD, new_flags)

    Linux has a special code path:

    #ifdef __linux__
            if (errno == EBADF) {
                // On Linux, ioctl(FIOCLEX) will fail with EBADF for O_PATH file descriptors
                // Fall through to the fcntl() path
            }
            else
    #endif

    Maybe we should just enable now this code path on FreeBSD as well.

    == test.pythoninto of the last successful build 654 ==

    datetime.datetime.now: 2021-08-01 01:15:06.918625

    os.uname: posix.uname_result(sysname='FreeBSD', nodename='140-CURRENT-amd64-564d', release='14.0-CURRENT', version='FreeBSD 14.0-CURRENT #14 main-n245176-8742817ba62: Tue Mar 2 03:48:10 UTC 2021 root@140-CURRENT-amd64:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG', machine='amd64')

    == test.pythoninto of the failed build 655 ==

    datetime.datetime.now: 2021-08-04 09:56:34.287999

    os.uname: posix.uname_result(sysname='FreeBSD', nodename='140-CURRENT-amd64-564d', release='14.0-CURRENT', version='FreeBSD 14.0-CURRENT #0 main-n248401-de0c7fbe280: Mon Aug 2 02:42:31 UTC 2021 root@140-CURRENT-amd64-564d:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG', machine='amd64')

    @vstinner
    Copy link
    Member Author

    vstinner commented Aug 6, 2021

    New changeset c24896c by Victor Stinner in branch 'main':
    bpo-44849: Fix os.set_inheritable() on FreeBSD 14 with O_PATH (GH-27623)
    c24896c

    @miss-islington
    Copy link
    Contributor

    New changeset 2ae2235 by Miss Islington (bot) in branch '3.10':
    bpo-44849: Fix os.set_inheritable() on FreeBSD 14 with O_PATH (GH-27623)
    2ae2235

    @miss-islington
    Copy link
    Contributor

    New changeset 791c28a by Miss Islington (bot) in branch '3.9':
    bpo-44849: Fix os.set_inheritable() on FreeBSD 14 with O_PATH (GH-27623)
    791c28a

    @vstinner vstinner added 3.9 only security fixes 3.10 only security fixes labels Aug 6, 2021
    @vstinner vstinner closed this as completed Aug 6, 2021
    @vstinner vstinner added 3.9 only security fixes 3.10 only security fixes labels Aug 6, 2021
    @vstinner vstinner closed this as completed Aug 6, 2021
    @koobs
    Copy link

    koobs commented Aug 7, 2021

    There was a recent worker update to latest CURRENT (14). I'll try to identify recent changes or changesets that may have affected behaviour and reference them here

    @koobs
    Copy link

    koobs commented Aug 7, 2021

    O_PATH support was added in FreeBSD CURRENT April 15:

    https://reviews.freebsd.org/D29323?id=86257

    freebsd/freebsd-src@8d9ed17

    @victor If the failure in this tests suggests either a bug in our implementation, or an incompatibility, please let me know/clarify so I can get our implementation polished

    @koobs
    Copy link

    koobs commented Aug 7, 2021

    Reference for Linux behaviour:

    filedesc: don't use ioctl(FIOCLEX) on Linux bpo-62425
    rust-lang/rust#62425

    Looks like we match the behaviour intentionally (or otherwise).

    @vstinner
    Copy link
    Member Author

    vstinner commented Aug 9, 2021

    AMD64 FreeBSD Shared 3.x is back to green: test_os.test_get_set_inheritable_o_path() now pass on FreeBSD CURRENT ;-) My fix works as expected. I simply used the same fix than for Linux.

    @ambv
    Copy link
    Contributor

    ambv commented Oct 19, 2021

    New changeset 67e10be by Miss Islington (bot) in branch '3.8':
    bpo-44849: Fix os.set_inheritable() on FreeBSD 14 with O_PATH (GH-27623) (GH-28978)
    67e10be

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.9 only security fixes 3.10 only security fixes 3.11 only security fixes tests Tests in the Lib/test dir
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants