This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: FAIL: test_makedev (test.test_posix.PosixTester) on AMD64 FreeBSD CURRENT
Type: Stage: resolved
Components: Tests Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: koobs, vstinner
Priority: normal Keywords:

Created on 2017-07-26 11:46 by vstinner, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 2915 merged vstinner, 2017-07-27 14:37
PR 2917 merged vstinner, 2017-07-27 16:00
PR 2918 merged vstinner, 2017-07-27 16:00
PR 7449 merged vstinner, 2018-06-06 11:05
Messages (12)
msg299220 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-26 11:46
The test failed in the build 632 (Wed Jul 26 10:47:01 2017) for the first time.

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%20CURRENT%20Debug%203.x/builds/632/steps/test/logs/stdio


== CPython 3.7.0a0 (heads/master:ede9084476, Jul 26 2017, 20:49:33) [GCC 4.2.1 Compatible FreeBSD Clang 5.0.0 (trunk 308421)]
== FreeBSD-12.0-CURRENT-amd64-64bit-ELF little-endian
== hash algorithm: siphash24 64bit
== cwd: /usr/home/buildbot/python/3.x.koobs-freebsd-current/build/build/test_python_23542
== CPU count: 2
== encodings: locale=US-ASCII, FS=ascii
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=1, verbose=0, bytes_warning=2, quiet=0, hash_randomization=1, isolated=0)
Using random seed 9341910
...
======================================================================
FAIL: test_makedev (test.test_posix.PosixTester)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_posix.py", line 543, in test_makedev
    self.assertEqual(posix.makedev(major, minor), dev)
AssertionError: 1292118443 != 954774858155
msg299227 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-26 12:14
The test fails on Debug but also Non-Debug buildbots, master and 3.6 branches. It looks more like a change on the buildbot, maybe a FreeBSD upgrade?
msg299308 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-27 14:07
I'm able to reproduce the bug on koobs's FreeBSD CURRENT:

* stat() returns st_dev = 0xde4d0429ab
* major(0xde4d0429ab) returns 0x29
* minor(0xde4d0429ab) returns 0x4d0400ab

minor() truncates most significant bits. major/minor are defined in sys/types.h:
https://github.com/freebsd/freebsd/blob/master/sys/sys/types.h#L372

#define	major(x)	((int)(((u_int)(x) >> 8)&0xff))	/* major number */
#define	minor(x) ((int)((x)&0xffff00ff)) /* minor number */

The definition of minor() confirms that most significant bits are truncated by "& 0xffff00ff".

I'm surprised that stat().st_dev returns a device larger than INT_MAX: 0xde4d0429ab.
msg299309 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-27 14:12
os.stat() seems correct, st_dev and st_ino are the same than the system command "stat":

       
CURRENT-amd64% ./python -c 'import os; st=os.stat("setup.py"); print(st)'                  
os.stat_result(st_mode=33188, st_ino=2384528, st_dev=954774858155, st_nlink=1, st_uid=1003, st_gid=1003, st_size=99944, st_atime=1501162007, st_mtime=1501162007, st_ctime=1501162007)

CURRENT-amd64% stat setup.py                                                               
954774858155 2384528 -rw-r--r-- 1 haypo haypo 18446744073709551615 99944 "Jul 27 23:26:47 2017" "Jul 27 23:26:47 2017" "Jul 27 23:26:47 2017" "Jul 27 23:26:47 2017" 100352 79 0x800 setup.py


The used filesystem for /home is ZFS:

CURRENT-amd64% df .
Filesystem   1K-blocks     Used   Avail Capacity  Mounted on
storage/home  17041752 11205517 5836235    66%    /usr/home

CURRENT-amd64% mount|grep home
storage/home on /usr/home (zfs, local, noatime, nfsv4acls)
msg299310 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-27 14:16
The problem is more the major(), minor() and makedev() commands:

CURRENT-amd64% ./python       
Python 3.7.0a0 (heads/master:d5ed47dea2, Jul 28 2017, 00:10:54) 
[GCC 4.2.1 Compatible FreeBSD Clang 5.0.0 (trunk 308421)] on freebsd12
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> dev=954774858155; os.makedev(os.major(dev), os.minor(dev))
1292118443

"makedev, major, minor -- device number conversion" doc on FreeBSD CURRENT:

https://www.freebsd.org/cgi/man.cgi?query=makedev&apropos=0&sektion=0&manpath=FreeBSD+12-current&arch=default&format=html



SYNOPSIS
     #include <sys/types.h>

     dev_t
     makedev(int major,	int minor);

     int
     major(dev_t dev);

     int
     minor(dev_t dev);

DESCRIPTION
     The makedev() macro allows	a unique device	number to be generated based
     on	its major and minor number.  The major() and minor() macros can	be
     used to obtain the	original numbers from the device number	dev.

     In	previous implementations of FreeBSD all	block and character devices
     were uniquely identified by a pair	of major and minor numbers.  The major
     number referred to	a certain device class (e.g. disks, TTYs) while	the
     minor number identified an	instance within	the device class.  Later ver-
     sions of FreeBSD automatically generate a unique device number for	each
     character device visible in /dev/.	 These numbers are not divided in
     device classes.

     On	FreeBSD	these macros are only used by utilities	that need to exchange
     numbers with other	operating systems that may use different encodings for
     dev_t, but	also applications that present these numbers to	the user in a
     more conventional way.

RETURN VALUES
     The major() macro returns a device	major number that has a	value between
     0 and 255.	 The minor() macro returns a device minor number whose value
     can span the complete range of an int.
msg299311 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-27 14:20
Confirmation that minor() truncates high bits:

CURRENT-amd64% ./python -c 'import os; print(hex(os.minor(0xaabbccddff)))'
0xbbcc00ff
msg299314 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-27 14:34
At May 23, the dev_t type changed from 32 bits to 64 bits on FreeBSD in the kernel, but minor() wasn't updated. I reported a bug to FreeBSD:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221048
msg299315 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-27 14:55
New changeset 12953ffe12ac781332b384c36b25d12216b1db62 by Victor Stinner in branch 'master':
bpo-31044: Skip test_posix.test_makedev() on FreeBSD (#2915)
https://github.com/python/cpython/commit/12953ffe12ac781332b384c36b25d12216b1db62
msg299323 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-27 16:42
New changeset 54cb3400e500f99bd57a0273aa7ef84ffd37558e by Victor Stinner in branch '3.6':
bpo-31044: Skip test_posix.test_makedev() on FreeBSD (#2915) (#2917)
https://github.com/python/cpython/commit/54cb3400e500f99bd57a0273aa7ef84ffd37558e
msg299324 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-27 16:44
New changeset c2f7fb61511456c62877592988b31714cb8ba266 by Victor Stinner in branch '2.7':
[2.7] bpo-31044: Skip test_posix.test_makedev() on FreeBSD (#2915) (#2918)
https://github.com/python/cpython/commit/c2f7fb61511456c62877592988b31714cb8ba266
msg318820 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-06 13:28
New changeset b5d702e5e7291eed21666ed931da2a5e92c28a65 by Victor Stinner in branch 'master':
bpo-31044, test_posix: Reenable makedev() tests on FreeBSD (#7449)
https://github.com/python/cpython/commit/b5d702e5e7291eed21666ed931da2a5e92c28a65
msg318821 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-06 13:33
I enabled again the test in the master branch, the FreeBSD kernel bug has been fixed 10 months ago:

* https://svnweb.freebsd.org/base?view=revision&revision=321920
* https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221048

Technically, we could reenable the test in 3.7 as well, but I prefer to leave it disabled since I'm not 100% sure that all FreeBSD kernels in the wild are fine, and the code is already well tested by other platforms.
History
Date User Action Args
2022-04-11 14:58:49adminsetgithub: 75227
2018-06-06 13:33:08vstinnersetmessages: + msg318821
2018-06-06 13:28:56vstinnersetmessages: + msg318820
2018-06-06 11:05:51vstinnersetpull_requests: + pull_request7071
2017-07-27 16:45:12vstinnersetstatus: open -> closed
resolution: fixed
stage: resolved
2017-07-27 16:44:46vstinnersetmessages: + msg299324
2017-07-27 16:42:13vstinnersetmessages: + msg299323
2017-07-27 16:00:38vstinnersetpull_requests: + pull_request2971
2017-07-27 16:00:36vstinnersetpull_requests: + pull_request2970
2017-07-27 14:55:58vstinnersetmessages: + msg299315
2017-07-27 14:37:16vstinnersetpull_requests: + pull_request2967
2017-07-27 14:34:18vstinnersetmessages: + msg299314
2017-07-27 14:20:08vstinnersetmessages: + msg299311
2017-07-27 14:16:40vstinnersetmessages: + msg299310
2017-07-27 14:13:00vstinnersetmessages: + msg299309
2017-07-27 14:07:36vstinnersetmessages: + msg299308
2017-07-26 12:14:37vstinnersetmessages: + msg299227
title: FAIL: test_makedev (test.test_posix.PosixTester) on AMD64 FreeBSD CURRENT Debug 3.x -> FAIL: test_makedev (test.test_posix.PosixTester) on AMD64 FreeBSD CURRENT
2017-07-26 11:46:35vstinnersetnosy: + koobs
2017-07-26 11:46:30vstinnercreate