Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FAIL: test_makedev (test.test_posix.PosixTester) on AMD64 FreeBSD CURRENT #75227

Closed
vstinner opened this issue Jul 26, 2017 · 12 comments
Closed
Labels
3.7 (EOL) end of life tests Tests in the Lib/test dir

Comments

@vstinner
Copy link
Member

BPO 31044
Nosy @vstinner, @koobs
PRs
  • bpo-31044: Skip test_posix.test_makedev() on FreeBSD #2915
  • [3.6] bpo-31044: Skip test_posix.test_makedev() on FreeBSD (#2915) #2917
  • [2.7] bpo-31044: Skip test_posix.test_makedev() on FreeBSD (#2915) #2918
  • bpo-31044, test_posix: Reenable makedev() tests on FreeBSD #7449
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2017-07-27.16:45:12.041>
    created_at = <Date 2017-07-26.11:46:30.294>
    labels = ['3.7', 'tests']
    title = 'FAIL: test_makedev (test.test_posix.PosixTester) on AMD64 FreeBSD CURRENT'
    updated_at = <Date 2018-06-06.13:33:08.137>
    user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2018-06-06.13:33:08.137>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2017-07-27.16:45:12.041>
    closer = 'vstinner'
    components = ['Tests']
    creation = <Date 2017-07-26.11:46:30.294>
    creator = 'vstinner'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 31044
    keywords = []
    message_count = 12.0
    messages = ['299220', '299227', '299308', '299309', '299310', '299311', '299314', '299315', '299323', '299324', '318820', '318821']
    nosy_count = 2.0
    nosy_names = ['vstinner', 'koobs']
    pr_nums = ['2915', '2917', '2918', '7449']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue31044'
    versions = ['Python 3.7']

    @vstinner
    Copy link
    Member Author

    The test failed in the build 632 (Wed Jul 26 10:47:01 2017) for the first time.

    http://buildbot.python.org/all/builders/AMD64%20FreeBSD%20CURRENT%20Debug%203.x/builds/632/steps/test/logs/stdio

    == CPython 3.7.0a0 (heads/master:ede9084476, Jul 26 2017, 20:49:33) [GCC 4.2.1 Compatible FreeBSD Clang 5.0.0 (trunk 308421)]
    == FreeBSD-12.0-CURRENT-amd64-64bit-ELF little-endian
    == hash algorithm: siphash24 64bit
    == cwd: /usr/home/buildbot/python/3.x.koobs-freebsd-current/build/build/test_python_23542
    == CPU count: 2
    == encodings: locale=US-ASCII, FS=ascii
    Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=1, verbose=0, bytes_warning=2, quiet=0, hash_randomization=1, isolated=0)
    Using random seed 9341910
    ...
    ======================================================================
    FAIL: test_makedev (test.test_posix.PosixTester)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_posix.py", line 543, in test_makedev
        self.assertEqual(posix.makedev(major, minor), dev)
    AssertionError: 1292118443 != 954774858155

    @vstinner vstinner added 3.7 (EOL) end of life tests Tests in the Lib/test dir labels Jul 26, 2017
    @vstinner
    Copy link
    Member Author

    The test fails on Debug but also Non-Debug buildbots, master and 3.6 branches. It looks more like a change on the buildbot, maybe a FreeBSD upgrade?

    @vstinner vstinner changed the title FAIL: test_makedev (test.test_posix.PosixTester) on AMD64 FreeBSD CURRENT Debug 3.x FAIL: test_makedev (test.test_posix.PosixTester) on AMD64 FreeBSD CURRENT Jul 26, 2017
    @vstinner
    Copy link
    Member Author

    I'm able to reproduce the bug on koobs's FreeBSD CURRENT:

    • stat() returns st_dev = 0xde4d0429ab
    • major(0xde4d0429ab) returns 0x29
    • minor(0xde4d0429ab) returns 0x4d0400ab

    minor() truncates most significant bits. major/minor are defined in sys/types.h:
    https://github.com/freebsd/freebsd/blob/master/sys/sys/types.h#L372

    #define	major(x)	((int)(((u_int)(x) >> 8)&0xff))	/* major number */
    #define	minor(x) ((int)((x)&0xffff00ff)) /* minor number */

    The definition of minor() confirms that most significant bits are truncated by "& 0xffff00ff".

    I'm surprised that stat().st_dev returns a device larger than INT_MAX: 0xde4d0429ab.

    @vstinner
    Copy link
    Member Author

    os.stat() seems correct, st_dev and st_ino are the same than the system command "stat":

    CURRENT-amd64% ./python -c 'import os; st=os.stat("setup.py"); print(st)'
    os.stat_result(st_mode=33188, st_ino=2384528, st_dev=954774858155, st_nlink=1, st_uid=1003, st_gid=1003, st_size=99944, st_atime=1501162007, st_mtime=1501162007, st_ctime=1501162007)

    CURRENT-amd64% stat setup.py
    954774858155 2384528 -rw-r--r-- 1 haypo haypo 18446744073709551615 99944 "Jul 27 23:26:47 2017" "Jul 27 23:26:47 2017" "Jul 27 23:26:47 2017" "Jul 27 23:26:47 2017" 100352 79 0x800 setup.py

    The used filesystem for /home is ZFS:

    CURRENT-amd64% df .
    Filesystem 1K-blocks Used Avail Capacity Mounted on
    storage/home 17041752 11205517 5836235 66% /usr/home

    CURRENT-amd64% mount|grep home
    storage/home on /usr/home (zfs, local, noatime, nfsv4acls)

    @vstinner
    Copy link
    Member Author

    The problem is more the major(), minor() and makedev() commands:

    CURRENT-amd64% ./python       
    Python 3.7.0a0 (heads/master:d5ed47dea2, Jul 28 2017, 00:10:54) 
    [GCC 4.2.1 Compatible FreeBSD Clang 5.0.0 (trunk 308421)] on freebsd12
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import os
    >>> dev=954774858155; os.makedev(os.major(dev), os.minor(dev))
    1292118443

    "makedev, major, minor -- device number conversion" doc on FreeBSD CURRENT:

    https://www.freebsd.org/cgi/man.cgi?query=makedev&apropos=0&sektion=0&manpath=FreeBSD+12-current&arch=default&format=html

    SYNOPSIS
         #include <sys/types.h>
     dev_t
     makedev(int major,	int minor);
    
         int
         major(dev_t dev);
    
         int
         minor(dev_t dev);

    DESCRIPTION
    The makedev() macro allows a unique device number to be generated based
    on its major and minor number. The major() and minor() macros can be
    used to obtain the original numbers from the device number dev.

     In	previous implementations of FreeBSD all	block and character devices
     were uniquely identified by a pair	of major and minor numbers.  The major
     number referred to	a certain device class (e.g. disks, TTYs) while	the
     minor number identified an	instance within	the device class.  Later ver-
     sions of FreeBSD automatically generate a unique device number for	each
     character device visible in /dev/.	 These numbers are not divided in
     device classes.
    
     On	FreeBSD	these macros are only used by utilities	that need to exchange
     numbers with other	operating systems that may use different encodings for
     dev_t, but	also applications that present these numbers to	the user in a
     more conventional way.
    

    RETURN VALUES
    The major() macro returns a device major number that has a value between
    0 and 255. The minor() macro returns a device minor number whose value
    can span the complete range of an int.

    @vstinner
    Copy link
    Member Author

    Confirmation that minor() truncates high bits:

    CURRENT-amd64% ./python -c 'import os; print(hex(os.minor(0xaabbccddff)))'
    0xbbcc00ff

    @vstinner
    Copy link
    Member Author

    At May 23, the dev_t type changed from 32 bits to 64 bits on FreeBSD in the kernel, but minor() wasn't updated. I reported a bug to FreeBSD:
    https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221048

    @vstinner
    Copy link
    Member Author

    New changeset 12953ff by Victor Stinner in branch 'master':
    bpo-31044: Skip test_posix.test_makedev() on FreeBSD (bpo-2915)
    12953ff

    @vstinner
    Copy link
    Member Author

    New changeset 54cb340 by Victor Stinner in branch '3.6':
    bpo-31044: Skip test_posix.test_makedev() on FreeBSD (bpo-2915) (bpo-2917)
    54cb340

    @vstinner
    Copy link
    Member Author

    New changeset c2f7fb6 by Victor Stinner in branch '2.7':
    [2.7] bpo-31044: Skip test_posix.test_makedev() on FreeBSD (bpo-2915) (bpo-2918)
    c2f7fb6

    @vstinner
    Copy link
    Member Author

    vstinner commented Jun 6, 2018

    New changeset b5d702e by Victor Stinner in branch 'master':
    bpo-31044, test_posix: Reenable makedev() tests on FreeBSD (bpo-7449)
    b5d702e

    @vstinner
    Copy link
    Member Author

    vstinner commented Jun 6, 2018

    I enabled again the test in the master branch, the FreeBSD kernel bug has been fixed 10 months ago:

    Technically, we could reenable the test in 3.7 as well, but I prefer to leave it disabled since I'm not 100% sure that all FreeBSD kernels in the wild are fine, and the code is already well tested by other platforms.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life tests Tests in the Lib/test dir
    Projects
    None yet
    Development

    No branches or pull requests

    1 participant