This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: regrtest checks (os.environ, sys.path, etc.) are hard to use
Type: enhancement Stage: needs patch
Components: Tests Versions: Python 3.3
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: BreamoreBoy, eric.araujo, ezio.melotti, michael.foord, python-dev, r.david.murray, vstinner
Priority: normal Keywords:

Created on 2011-06-10 16:19 by eric.araujo, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Messages (9)
msg138101 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-10 16:19
When regrtest tells me that os.environ was modified by test_packaging, it’s very hard to find which test is responsible.  The -vv output is hard to read: it prints four dictionaries, which I have to visually delineate, copy, paste, pretty-print and diff to find what’s changed.  Then I edit the source code of regrtest so that it fails earlier, or I add a debug print in the packaging.util code that touches os.environ.

I’d like regrtest to tell me what exactly was changed, and where.  For the first request, I think each checked object could use a type-specific method, for example _compare_dict would see if any value was changed and if keys are different, and print only that info.
msg138109 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-06-10 16:52
Improvements are welcome, the whole thing was a QnD hack that has been evolving over time...maybe there's some way to reuse unittest's assertEqual machinery?

I also wonder if the whole environment-changed thing should be moved into unittest so it could wrap around individual test methods.
msg138110 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-10 16:56
> maybe there's some way to reuse unittest's assertEqual machinery?
Hum, I think the diff functionality is cleanly separated in a (private) method, so we could get diff for free.

> I also wonder if the whole environment-changed thing should be moved
> into unittest so it could wrap around individual test methods.
Yeah, I was thinking about a WatchfulTestRunner that would perform the same checks.  It annoys me to have two diverging sets of features to run tests: either checks with regrtest, or failfast, discovery and granularity with unittest.
msg138378 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2011-06-15 14:42
I don't think this is something that belongs in unittest - it's not something particularly useful (or at least particularly requested) outside of the python test suite. No reason that a WatchfulTestRunner couldn't live in regrtest.
msg138381 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-06-15 15:22
Sounds like a plan.
msg138673 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-06-19 23:15
> I’d like regrtest to tell me what exactly was changed, and where.

regrtests has many tests (you give some examples: os.environ, sys.path), run all tests after calling a single test function would make regrtest slower. We can add an option (e.g. --strict?) to run these sanity tests after calling a single test function.

I would also be nice to have this option for the refleak checks: run refleak checks on each function, instead of doing the checks on the whole file.
msg144276 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-09-19 14:10
New changeset 9c61f46ea6b4 by Éric Araujo in branch 'default':
Make regrtest look at internal dicts in sysconfig.
http://hg.python.org/cpython/rev/9c61f46ea6b4
msg145110 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-10-07 17:26
Idea: when a regrtest check fails, the test is run again but a WatchfulMixin is injected into the test case’s bases, so that setUp and tearDown call the right methods.  This would be more efficient than always running with a watchful test runner.
msg220508 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-06-13 22:26
@Éric do you intend following up on this?
History
Date User Action Args
2022-04-11 14:57:18adminsetgithub: 56523
2015-10-02 21:24:02vstinnersetstatus: open -> closed
resolution: out of date
2014-06-13 22:26:16BreamoreBoysetnosy: + BreamoreBoy
messages: + msg220508
2013-01-03 01:59:21vstinnerunlinkissue12309 dependencies
2011-10-07 17:26:55eric.araujosetmessages: + msg145110
2011-09-19 14:10:50python-devsetnosy: + python-dev
messages: + msg144276
2011-06-19 23:15:26vstinnersetnosy: + vstinner
messages: + msg138673
2011-06-15 15:22:46r.david.murraysetmessages: + msg138381
2011-06-15 14:42:03michael.foordsetmessages: + msg138378
2011-06-10 16:56:22eric.araujosetnosy: + ezio.melotti, michael.foord
messages: + msg138110
2011-06-10 16:52:53r.david.murraysettype: enhancement
stage: needs patch
2011-06-10 16:52:26r.david.murraysetnosy: + r.david.murray
messages: + msg138109
2011-06-10 16:19:18eric.araujolinkissue12309 dependencies
2011-06-10 16:19:09eric.araujocreate