Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

unittest discover order is filesystem specific - hard to reproduce #60913

Closed
rbtcollins opened this issue Dec 18, 2012 · 12 comments
Closed

unittest discover order is filesystem specific - hard to reproduce #60913

rbtcollins opened this issue Dec 18, 2012 · 12 comments
Assignees
@voidspace
Labels
easy stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@rbtcollins
Copy link
Member

BPO 16709
Nosy @pitrou, @rbtcollins, @ezio-melotti, @voidspace, @akheron
Files
  • issue16709.patch: Patch & docs update for sorting os.listdir output
  • issue16709-tests.patch: Updated mmelin's patch with test
  • issue16709-tests2.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/voidspace'
closed_at = <Date 2013-03-19.00:50:25.884>
created_at = <Date 2012-12-18.02:28:57.191>
labels = ['easy', 'type-feature', 'library']
title = 'unittest discover order is filesystem specific - hard to reproduce'
updated_at = <Date 2013-03-19.00:50:25.882>
user = 'https://github.com/rbtcollins'

    bugs.python.org fields:

    activity = <Date 2013-03-19.00:50:25.882>
actor = 'python-dev'
assignee = 'michael.foord'
closed = True
closed_date = <Date 2013-03-19.00:50:25.884>
closer = 'python-dev'
components = ['Library (Lib)']
creation = <Date 2012-12-18.02:28:57.191>
creator = 'rbcollins'
dependencies = []
files = ['29197', '29461', '29462']
hgrepos = []
issue_num = 16709
keywords = ['patch', 'easy']
message_count = 12.0
messages = ['177675', '177676', '177683', '178141', '178159', '182792', '182795', '182796', '184539', '184541', '184542', '184571']
nosy_count = 9.0
nosy_names = ['pitrou', 'rbcollins', 'vila', 'ezio.melotti', 'michael.foord', 'python-dev', 'jramnani', 'petri.lehtinen', 'mmelin']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue16709'
versions = ['Python 3.4', 'Python 3.5']
    @rbtcollins
    Copy link
    Member Author

    Openstack recently switched from nose to using discover. discover walks the filesystem using os.listdir(), and that is just a thin layer over readdir. On ext3/ext4 filesystems, readdir is in an arbitrary order dependent on file insertion into the directory if dir_index is enabled (which is the default).

    This means that files are loaded in an order that isn't reproducable by other developers, so bad tests that have isolation issues can be very tricky to track down.

    Just wrapping the os.listdir() in sorted() would be sufficient to make this robust and repeatable, and avoid the headache.

    @rbtcollins rbtcollins added the stdlib Python modules in the Lib dir label Dec 18, 2012
    @pitrou
    Copy link
    Member

    pitrou commented Dec 18, 2012

    +1. It also makes test output nicer.

    @voidspace
    Copy link
    Contributor

    Seems reasonable.

    @voidspace voidspace self-assigned this Dec 18, 2012
    @ezio-melotti
    Copy link
    Member

    Should this be considered a new feature or should it be applied to older versions as well?

    @ezio-melotti ezio-melotti added easy type-feature A feature request or enhancement labels Dec 25, 2012
    @voidspace
    Copy link
    Contributor

    It smells like a feature to me (it isn't a direct "bug" fix anyway). It can be applied to earlier versions of Python through a new unittest2 release.

    @mmelin
    Copy link
    Mannequin

    mmelin mannequin commented Feb 23, 2013

    Not sure if there was anything more to it than this, but please find an attempt to add this attached.

    @akheron
    Copy link
    Member

    akheron commented Feb 23, 2013

    The patch looks good to me.

    @voidspace
    Copy link
    Contributor

    As we're specifying this behaviour in the documentation it would be nice to test it.

    @jramnani
    Copy link
    Mannequin

    jramnani mannequin commented Mar 18, 2013

    I've added tests for this behavior by un-sorting the test inputs for test_find_tests, and adding comments that the results should be sorted for reliable test execution.

    Attaching an updated patch.

    @jramnani
    Copy link
    Mannequin

    jramnani mannequin commented Mar 18, 2013

    Fixed a spelling error.

    @jramnani
    Copy link
    Mannequin

    jramnani mannequin commented Mar 18, 2013

    Fixed a spelling error. (Part deux)

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Mar 19, 2013

    New changeset fe18f16dc2a6 by Michael Foord in branch 'default':
    Closes bpo-16709. unittest test discovery sorts test files for consistent test ordering
    http://hg.python.org/cpython/rev/fe18f16dc2a6

    @python-dev python-dev mannequin closed this as completed Mar 19, 2013
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @voidspace voidspace

    Labels
    easy stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    5 participants
    @akheron @voidspace @rbtcollins @pitrou @ezio-melotti