This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author ezio.melotti
Recipients brett.cannon, eric.araujo, ezio.melotti, pitrou, r.david.murray
Date 2011-03-26.07:53:33
SpamBayes Score 2.220446e-15
Marked as misclassified No
Message-id <1301126014.82.0.935489495103.issue10966@psf.upfronthosting.co.za>
In-reply-to
Content
IMHO the current "unexpected" list is fairly pointless.  I just ran the test suite on 3.3 and I got the usual expected list of unexpected skips:
7 skips unexpected on linux2:
    test_bz2 test_dbm_gnu test_dbm_ndbm test_tcl test_tk
    test_ttk_guionly test_ttk_textonly
If these tests were really *expected to run* on my machine, having them skipped would be a failure.  They are instead just a set of tests that I *might be able to run* once I install all the right dependencies.

I think the tests can be classified in 3 categories:
 1) tests that are known to always run everywhere (e.g. test_unittest);
 2) tests that are known to always run on specific platforms and never on others (e.g. test_winreg);
 3) tests that might run on some platforms depending on some specific condition (e.g. test_zlib);

For each of these cases, regrtest should:
 1) fail if the test is not run;
 2) fail if the test is not run on the platform(s) where it's always available, always skip it on the others;
 3) skip the test conditionally without failing (e.g. checking if zlib can be imported or not);

A special case of 3) is when the skip depends on the platform name rather than the availability of some specific module/function (e.g. if  some platforms shipped with a buggy version of a library and there's no way to check if the library is buggy other than keeping a list of platforms that are known to use that version).  In this case the condition should check only for the platform where it's known to fail, and more platforms can be added if/when people report them.

I agree that the logic for this should be moved out of regrtest and added to the specific tests.

It's also possible to provide better skip messages, e.g.:
-test_winreg skipped -- No module named 'winreg'
-test_tk skipped -- No module named '_tkinter'
-test_urllibnet skipped -- Use of the `network' resource not enabled
+test_winreg skipped -- No module named 'winreg' (winreg is available on windows only)
+test_tk skipped -- No module named '_tkinter' (tcl/tk not installed)
+test_urllibnet skipped -- Use of the `network' resource not enabled (run the tests with -unetwork or -uall to enable it)
History
Date User Action Args
2011-03-26 07:53:34ezio.melottisetrecipients: + ezio.melotti, brett.cannon, pitrou, eric.araujo, r.david.murray
2011-03-26 07:53:34ezio.melottisetmessageid: <1301126014.82.0.935489495103.issue10966@psf.upfronthosting.co.za>
2011-03-26 07:53:33ezio.melottilinkissue10966 messages
2011-03-26 07:53:33ezio.melotticreate