classification
Title: test_os: test_get_set_inheritable_o_path() failed on AMD64 FreeBSD Shared 3.x
Type: Stage: resolved
Components: Tests Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: erlendaasland, koobs, lukasz.langa, miss-islington, pablogsal, vstinner
Priority: normal Keywords: patch

Created on 2021-08-06 10:40 by vstinner, last changed 2021-10-19 19:15 by lukasz.langa. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 27623 merged vstinner, 2021-08-06 11:05
PR 27630 merged miss-islington, 2021-08-06 13:17
PR 27631 merged miss-islington, 2021-08-06 13:17
PR 28978 merged miss-islington, 2021-10-15 13:40
Messages (11)
msg399065 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-08-06 10:40
Since build 655 (commit 6871fd0e8e5257f3ffebd1a1b2ca50e5f494e7f6), test_os failed on AMD64 FreeBSD Shared 3.x:
https://buildbot.python.org/all/#/builders/483/builds/655

======================================================================
ERROR: test_get_set_inheritable_o_path (test.test_os.FDInheritanceTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_os.py", line 3898, in test_get_set_inheritable_o_path
    os.set_inheritable(fd, True)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 9] Bad file descriptor
msg399067 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-08-06 10:42
test_get_set_inheritable_o_path() also fails on:

* AMD64 FreeBSD Shared 3.10
* AMD64 FreeBSD Shared 3.9

It smells more like a FreeBSD system update rather than a Python regression.
msg399069 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-08-06 10:55
On my FreeBSD 13 VM, the test is skipped:

vstinner@freebsd$ ./python -m test -v test_os -m test_get_set_inheritable_o_path
== CPython 3.11.0a0 (heads/main:938e84b4fa, Aug 6 2021, 10:46:32) [Clang 11.0.1 (git@github.com:llvm/llvm-project.git llvmorg-11.0.1-0-g43ff75f2c3
== FreeBSD-13.0-RELEASE-amd64-64bit-ELF little-endian
(...)
test_get_set_inheritable_o_path (test.test_os.FDInheritanceTests) ... skipped 'need os.O_PATH'

It seems like O_PATH was introduced in FreeBSD 14 CURRENT, but the test fails.

Code of the test:

    @unittest.skipUnless(hasattr(os, 'O_PATH'), "need os.O_PATH")
    def test_get_set_inheritable_o_path(self):
        fd = os.open(__file__, os.O_PATH)
        self.addCleanup(os.close, fd)
        self.assertEqual(os.get_inheritable(fd), False)

        os.set_inheritable(fd, True)  # <==== OSError: [Errno 9] Bad file descriptor
        self.assertEqual(os.get_inheritable(fd), True)

        os.set_inheritable(fd, False)
        self.assertEqual(os.get_inheritable(fd), False)

os.set_inheritable() calls _Py_set_inheritable() whichs has multiple implementations:

* ioctl(fd, FIONCLEX, NULL)
* fcntl(fd, F_GETFD) + fcntl(fd, F_SETFD, new_flags)

Linux has a special code path:

#ifdef __linux__
        if (errno == EBADF) {
            // On Linux, ioctl(FIOCLEX) will fail with EBADF for O_PATH file descriptors
            // Fall through to the fcntl() path
        }
        else
#endif

Maybe we should just enable now this code path on FreeBSD as well.


== test.pythoninto of the last successful build 654 ==

datetime.datetime.now: 2021-08-01 01:15:06.918625

os.uname: posix.uname_result(sysname='FreeBSD', nodename='140-CURRENT-amd64-564d', release='14.0-CURRENT', version='FreeBSD 14.0-CURRENT #14 main-n245176-8742817ba62: Tue Mar  2 03:48:10 UTC 2021     root@140-CURRENT-amd64:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG', machine='amd64')

== test.pythoninto of the failed build 655 ==

datetime.datetime.now: 2021-08-04 09:56:34.287999

os.uname: posix.uname_result(sysname='FreeBSD', nodename='140-CURRENT-amd64-564d', release='14.0-CURRENT', version='FreeBSD 14.0-CURRENT #0 main-n248401-de0c7fbe280: Mon Aug  2 02:42:31 UTC 2021     root@140-CURRENT-amd64-564d:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG', machine='amd64')
msg399085 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-08-06 13:15
New changeset c24896c0e3b32c8a9f614ef51366007b67d5c665 by Victor Stinner in branch 'main':
bpo-44849: Fix os.set_inheritable() on FreeBSD 14 with O_PATH (GH-27623)
https://github.com/python/cpython/commit/c24896c0e3b32c8a9f614ef51366007b67d5c665
msg399088 - (view) Author: miss-islington (miss-islington) Date: 2021-08-06 13:40
New changeset 2ae2235c7a7627725df40c7875245cd17d90f39f by Miss Islington (bot) in branch '3.10':
bpo-44849: Fix os.set_inheritable() on FreeBSD 14 with O_PATH (GH-27623)
https://github.com/python/cpython/commit/2ae2235c7a7627725df40c7875245cd17d90f39f
msg399089 - (view) Author: miss-islington (miss-islington) Date: 2021-08-06 13:42
New changeset 791c28a56f043618111a19377f8527326037230e by Miss Islington (bot) in branch '3.9':
bpo-44849: Fix os.set_inheritable() on FreeBSD 14 with O_PATH (GH-27623)
https://github.com/python/cpython/commit/791c28a56f043618111a19377f8527326037230e
msg399167 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2021-08-07 02:35
There was a recent worker update to latest CURRENT (14). I'll try to identify recent changes or changesets that may have affected behaviour and reference them here
msg399168 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2021-08-07 02:45
O_PATH support was added in FreeBSD CURRENT April 15:

https://reviews.freebsd.org/D29323?id=86257

https://github.com/freebsd/freebsd-src/commit/8d9ed174f3afba5f114742447e622fc1173d4774

@Victor If the failure in this tests suggests either a bug in our implementation, or an incompatibility, please let me know/clarify so I can get our implementation polished
msg399169 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2021-08-07 02:54
Reference for Linux behaviour:

filedesc: don't use ioctl(FIOCLEX) on Linux #62425
https://github.com/rust-lang/rust/pull/62425

Looks like we match the behaviour intentionally (or otherwise).
msg399264 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-08-09 13:17
AMD64 FreeBSD Shared 3.x is back to green: test_os.test_get_set_inheritable_o_path() now pass on FreeBSD CURRENT ;-) My fix works as expected. I simply used the same fix than for Linux.
msg404340 - (view) Author: Ɓukasz Langa (lukasz.langa) * (Python committer) Date: 2021-10-19 19:15
New changeset 67e10be3fee7c69d4ece26e75a0b7a84dab68ce5 by Miss Islington (bot) in branch '3.8':
bpo-44849: Fix os.set_inheritable() on FreeBSD 14 with O_PATH (GH-27623) (GH-28978)
https://github.com/python/cpython/commit/67e10be3fee7c69d4ece26e75a0b7a84dab68ce5
History
Date User Action Args
2021-10-19 19:15:33lukasz.langasetmessages: + msg404340
2021-10-15 13:40:35miss-islingtonsetpull_requests: + pull_request27266
2021-08-09 13:17:18vstinnersetmessages: + msg399264
2021-08-07 02:54:37koobssetmessages: + msg399169
2021-08-07 02:45:25koobssetmessages: + msg399168
2021-08-07 02:35:35koobssetmessages: + msg399167
2021-08-06 15:14:21vstinnersetstatus: open -> closed
stage: patch review -> resolved
resolution: fixed
versions: + Python 3.9, Python 3.10
2021-08-06 13:42:58miss-islingtonsetmessages: + msg399089
2021-08-06 13:40:48miss-islingtonsetmessages: + msg399088
2021-08-06 13:17:56miss-islingtonsetpull_requests: + pull_request26125
2021-08-06 13:17:21miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request26124
2021-08-06 13:15:17vstinnersetmessages: + msg399085
2021-08-06 11:05:29vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request26117
2021-08-06 10:55:56vstinnersetmessages: + msg399069
2021-08-06 10:42:35vstinnersetnosy: + koobs
messages: + msg399067
2021-08-06 10:40:46vstinnercreate