This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Fix types for dev_t processing in posix module
Type: behavior Stage: patch review
Components: Extension Modules, FreeBSD Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: AMDmi3, koobs, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2021-11-09 17:13 by AMDmi3, last changed 2022-04-11 14:59 by admin.

Files
File name Uploaded Description Edit
posixmodule.c.patch AMDmi3, 2021-11-09 17:13 Patch
Pull Requests
URL Status Linked Edit
PR 29494 open AMDmi3, 2021-11-09 17:15
PR 31794 open serhiy.storchaka, 2022-03-10 14:51
Messages (5)
msg406030 - (view) Author: Dmitry Marakasov (AMDmi3) * Date: 2021-11-09 17:13
So, I was investigating a test failure of python 3.11 and 3.10 on FreeBSD (but it likely applies to all python versions):

======================================================================
FAIL: test_makedev (test.test_posix.PosixTester)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/ports/lang/python311/work/Python-3.11.0a1/Lib/test/test_posix.py", line 686, in test_makedev
    self.assertGreaterEqual(dev, 0)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: -5228656221359548839 not greater than or equal to 0

----------------------------------------------------------------------

The test checks that posix.stat(somepath).st_dev >= 0, but negative value was returned.

Python uses PyLong_FromLongLong to convert from dev_t C type which st_dev is:

https://github.com/python/cpython/blob/main/Modules/posixmodule.c#L2410
https://github.com/python/cpython/blob/main/Modules/posixmodule.c#L901

POSIX does not seem to define signedness of dev_t,

https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/basedefs/sys_types.h.html

only saying it shall be an "integer type". However on practice on both FreeBSD and Linux it's unsigned:

FreeBSD:
typedef __dev_t     dev_t;   // sys/types.h
typedef __uint64_t  __dev_t; // sys/_types.h

Linux (Ubuntu 18.04):
typedef __dev_t dev_t;             // sys/stat.h
__STD_TYPE __DEV_T_TYPE __dev_t;   // sys/types.h
#define __DEV_T_TYPE            __UQUAD_TYPE;  // sys/typesizes.h

So I suggest the attached patch to switch _PyLong_FromDev to PyLong_FromUnsignedLongLong which also makes it consistent with _Py_Dev_Converter which converts the other way with PyLong_AsUnsignedLongLong.




This change fixes the mentioned test, but another test failure is unmasked:

======================================================================
ERROR: test_makedev (test.test_posix.PosixTester)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/ports/lang/python311/work/Python-3.11.0a1/Lib/test/test_posix.py", line 704, in test_makedev
    self.assertEqual(posix.makedev(major, minor), dev)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
OverflowError: Python int too large to convert to C int

----------------------------------------------------------------------

This problem needs couple more trivial changes, but I'm not sure how to make them correctly (and I'm also confused by how this file relates with clinic/posixmodule.c).

The problems are that:
- os_major_impl/os_minor_impl and os_makedev_impl are inconsistent in their argument/return types for major/minor dev numbers: the former use `unsigned int`, while the latter uses `int`.
- the correct type is platform dependent, for instance Linux uses `unsigned int` and FreeBSD uses `int` (from `man makedev`).

I guess that to fix this failure one needs to add a macro/typedef for the type for minor/major dev numbers like

#if defined(__FreeBSD__)
#define DEVICE_MAJORMINOR_T int
#else
#define DEVICE_MAJORMINOR_T unsigned int
#endif

and use it in the named functions:

static DEVICE_MAJORMINOR_T os_major_impl(PyObject *module, dev_t device)
static DEVICE_MAJORMINOR_T os_minor_impl(PyObject *module, dev_t device)

static dev_t os_makedev_impl(PyObject *module, DEVICE_MAJORMINOR_T major, DEVICE_MAJORMINOR_T minor)
msg406051 - (view) Author: Dmitry Marakasov (AMDmi3) * Date: 2021-11-09 20:44
In case you're curious, here are some st_dev values which are encountered on my FreeBSD:
- 0xac2308de99d1f699 - ZFS
- 0x7100ff00 - devfs
- 0xffffffff8700ff01  - tmpfs
- 0x2900ff4e - nullfs
I suspect Linux uses smaller numbers which do not cause overflows so tese tests hasn't fired on Linux.
msg411022 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2022-01-20 12:58
Is device number -1 used in any context (for example as "unknown device number")?
msg411036 - (view) Author: Dmitry Marakasov (AMDmi3) * Date: 2022-01-20 16:49
> Is device number -1 used in any context (for example as "unknown device number")?

Yes, there's NODEV macro in both Linux and FreeBSD which expands to ((dev_t)-1).
msg414851 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2022-03-10 14:54
PR 31794 supports NODEV and checks for integer overflows.
History
Date User Action Args
2022-04-11 14:59:52adminsetgithub: 89928
2022-03-10 14:54:37serhiy.storchakasetmessages: + msg414851
2022-03-10 14:51:52serhiy.storchakasetpull_requests: + pull_request29898
2022-01-20 16:49:06AMDmi3setmessages: + msg411036
2022-01-20 12:58:22serhiy.storchakasetmessages: + msg411022
2022-01-20 09:15:11kumaradityasetnosy: + serhiy.storchaka
2022-01-16 17:47:37iritkatrielsetversions: - Python 3.6, Python 3.7, Python 3.8
2021-11-09 20:44:44AMDmi3setmessages: + msg406051
2021-11-09 17:15:03AMDmi3setstage: patch review
pull_requests: + pull_request27743
2021-11-09 17:13:09AMDmi3create