classification
Title: unittest discover order is filesystem specific - hard to reproduce
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.5, Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: michael.foord Nosy List: ezio.melotti, jramnani, michael.foord, mmelin, petri.lehtinen, pitrou, python-dev, rbcollins, vila
Priority: normal Keywords: easy, patch

Created on 2012-12-18 02:28 by rbcollins, last changed 2013-03-19 00:50 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
issue16709.patch mmelin, 2013-02-23 18:58 Patch & docs update for sorting os.listdir output review
issue16709-tests.patch jramnani, 2013-03-18 22:28 Updated mmelin's patch with test review
issue16709-tests2.patch jramnani, 2013-03-18 22:30 review
Messages (12)
msg177675 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2012-12-18 02:28
Openstack recently switched from nose to using discover. discover walks the filesystem using os.listdir(), and that is just a thin layer over readdir. On ext3/ext4 filesystems, readdir is in an arbitrary order dependent on file insertion into the directory if dir_index is enabled (which is the default).

This means that files are loaded in an order that isn't reproducable by other developers, so bad tests that have isolation issues can be very tricky to track down.

Just wrapping the os.listdir() in sorted() would be sufficient to make this robust and repeatable, and avoid the headache.
msg177676 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-12-18 06:52
+1. It also makes test output nicer.
msg177683 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2012-12-18 10:26
Seems reasonable.
msg178141 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2012-12-25 15:40
Should this be considered a new feature or should it be applied to older versions as well?
msg178159 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2012-12-25 18:19
It smells like a feature to me (it isn't a direct "bug" fix anyway). It can be applied to earlier versions of Python through a new unittest2 release.
msg182792 - (view) Author: Martin Melin (mmelin) * Date: 2013-02-23 18:58
Not sure if there was anything more to it than this, but please find an attempt to add this attached.
msg182795 - (view) Author: Petri Lehtinen (petri.lehtinen) * (Python committer) Date: 2013-02-23 19:08
The patch looks good to me.
msg182796 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2013-02-23 19:14
As we're specifying this behaviour in the documentation it would be nice to test it.
msg184539 - (view) Author: Jeff Ramnani (jramnani) * Date: 2013-03-18 22:28
I've added tests for this behavior by un-sorting the test inputs for test_find_tests, and adding comments that the results should be sorted for reliable test execution.

Attaching an updated patch.
msg184541 - (view) Author: Jeff Ramnani (jramnani) * Date: 2013-03-18 22:30
Fixed a spelling error.
msg184542 - (view) Author: Jeff Ramnani (jramnani) * Date: 2013-03-18 22:30
Fixed a spelling error. (Part deux)
msg184571 - (view) Author: Roundup Robot (python-dev) Date: 2013-03-19 00:50
New changeset fe18f16dc2a6 by Michael Foord in branch 'default':
Closes issue 16709. unittest test discovery sorts test files for consistent test ordering
http://hg.python.org/cpython/rev/fe18f16dc2a6
History
Date User Action Args
2013-03-19 00:50:25python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg184571

resolution: fixed
stage: test needed -> resolved
2013-03-18 22:30:35jramnanisetfiles: + issue16709-tests2.patch

messages: + msg184542
2013-03-18 22:30:01jramnanisetmessages: + msg184541
2013-03-18 22:28:18jramnanisetfiles: + issue16709-tests.patch
nosy: + jramnani
messages: + msg184539

2013-02-23 19:14:07michael.foordsetmessages: + msg182796
stage: patch review -> test needed
2013-02-23 19:08:25petri.lehtinensetnosy: + petri.lehtinen

messages: + msg182795
stage: needs patch -> patch review
2013-02-23 18:58:01mmelinsetfiles: + issue16709.patch

nosy: + mmelin
messages: + msg182792

keywords: + patch
2012-12-25 18:19:07michael.foordsetmessages: + msg178159
versions: + Python 3.5, - Python 2.7, Python 3.2, Python 3.3
2012-12-25 15:40:19ezio.melottisetkeywords: + easy
type: enhancement
messages: + msg178141

stage: needs patch
2012-12-22 01:01:50terry.reedysetversions: - Python 3.1, Python 3.5
2012-12-18 14:52:52vilasetnosy: + vila
2012-12-18 10:26:04michael.foordsetassignee: michael.foord
messages: + msg177683
2012-12-18 06:52:12pitrousetmessages: + msg177676
2012-12-18 02:28:57rbcollinscreate