classification
Title: Move test sub-packages to Lib/test
Type: enhancement Stage: patch review
Components: Tests Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Ido Michael, barry, belopolsky, benjamin.peterson, berker.peksag, brett.cannon, eric.araujo, ezio.melotti, ghaering, gmwils, gpolo, michael.foord, ned.deily, r.david.murray, rhettinger, tarek, theller, zach.ware
Priority: normal Keywords: easy, patch

Created on 2010-11-29 00:52 by michael.foord, last changed 2020-03-22 22:06 by Ido Michael.

Files
File name Uploaded Description Edit
issue10572-sqlite3.patch gmwils, 2013-02-23 20:10 review
issue10572-lib2to3.patch gmwils, 2013-02-23 21:04 review
issue10572-sqlite3-2.patch gmwils, 2013-02-26 19:48
issue10572-lib2to3-2.patch gmwils, 2013-02-26 20:12
issue10572-sqlite3.diff zach.ware, 2014-08-08 18:27
issue10572-ctypes.diff zach.ware, 2014-08-08 19:01
Pull Requests
URL Status Linked Edit
PR 18524 closed Ido Michael, 2020-02-16 18:07
PR 18727 open Ido Michael, 2020-03-01 22:51
Messages (50)
msg122751 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-11-29 00:52
Having tests in Lib/test instead of inside the package makes it easier to grep the unittest package without grepping the tests. The Windows installer has an "install without tests" option which is easier to honour if the tests aren't in the package.

However, currently all packages that have test *packages* have the tests in the package rather than inside Lib/test. (There are no test packages inside Lib/test.)

Examples: email, distutils, ctypes, importlib, json, lib2to3, sqlite3

I also maintain an external port of unittest from Python 3. This is unittest2-py3k. Moving the tests would make it *slightly* harder to keep this in sync. I'm moving to maintaining this port as a set of patches rather than a separate branch. These patches can be applied automatically to unittest from py3k head. unittest2-py3k will be built automatically by a script, so it isn't a big deal.
msg122815 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2010-11-29 15:34
3.1 should also be considered if the tests are moved.  In theory this is not a bug fix so it shouldn't go in 3.1, but in practice it will make merging more difficult.  This might not be a strong argument though, considering that 3.1 will accept only security fixes soon and only the few developers that work on unittest will be affected.
msg122817 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-11-29 15:42
The same is true for 2.7 though, and that is getting bug fixes. svnmerge would no longer work (and to making the change would mean moving the tests in a point release).
msg122818 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-11-29 15:52
+0, and I think we should hear from the maintainers of the affected packages first.  For packages that are also externally maintained moving tests out may cause inconvenience to the maintainer.
msg122820 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-11-29 15:57
That list of examples was non-exhaustive, there is also tkinter.
msg122828 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-11-29 16:54
For the email package I would be in favor of moving the tests to Lib/test.  I've always found it a bit inconvenient that they are in Lib/email.  After hearing of Michael's intent with unittest, and given the evolution of email5 into email5.1, I am also considering the possibility of packaging email6 (when I get to it!) as a patch set against email5, which would make this change less of an issue for email6 development.

The 2.7 sync issue is a concern, but there are certainly precedents for differing file layouts between 3.x and 2.7.  I'm willing myself to deal with this for email.

Barry may have a different opinion.

All of that said, this is a general enough issue that it may be appropriate to raise it on python-dev.  Even if exceptions are made for individual packages, it would be good to agree on a general "best practices" rule for this for the stdlib.
msg122830 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2010-11-29 17:04
grepping the code without the tests doesn't seem that compelling a use case to me, given that grep and find both provide options to prune directories.  I do think that moving the tests out of the email package will make it harder to maintain and distribute as a separate package.  However, if RDM thinks the burden won't be too high, and the advantages of a split outweigh the disadvantages, then I defer to him.  I would still make a case for distributing email6 as a package available on Cheeseshop though, otherwise it just won't get much independent use until it's in the stdlib.
msg122838 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-11-29 18:24
Yes, a cheeseshop package is definitely part of the plan, I didn't mean to imply otherwise.  It won't be hard to automate the packaging, and indeed I'll wind up doing that anyway even if the tests stay inside Lib/email.

I will say that that I'm probably only +0.5 on this change...I like it from a consistency standpoint (heading toward all stdlib tests being in Lib/test) and it seems like it would make the job of packagers who desire a 'no tests' option easier.  But things have been working fine as they are, which is why I'm not at a full +1 :).
msg122841 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-11-29 18:35
For distutils tests, I’m ±0.  I don’t see any major drawback nor any major benefit.  Tarek will decide.
msg122847 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-11-29 18:49
Of those, it makes the most sense to move the json tests to Lib/tests.  Bob is not externally maintaining the 3.x version.  It's all our now.

Also, it looks like importlib is in a maintenance mode now.

There is merit to keeping 2to3, ctypes, sqlite tests separate.

Currently all of the documentation files are still under Doc so we should keep it that way and not move them under package directory trees.
msg122852 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2010-11-29 19:11
I have no issue with moving importlib into Lib/test as long as I can still run the tests with ``python3 -m test.importlib``. I actually only put the tests in importlib.tests because that was common practice amongst newer packages in the stdlib.

And just to prevent some rumour from perpetuating, importlib is not in maintenance mode. In fact the API was heavily reworked in 3.2 and I plan on exposing more of the API publicly in 3.3 and hopefully to bootstrap as well. The only thing you could think is in maintenance mode is importlib's Chesseshop package, but that's just for 2.x compatibility and for Django's benefit.
msg127032 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-01-25 18:19
Changing the title to reflect broader scope of this issue.  Json tests were moved to Lib/test/json_tests in r86875.
msg179583 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-01-10 19:14
I believe that the complete list of test files still located outside of Lib/test are:

  tkinter/test
  distutils/tests
  ctypes/tests
  lib2to3/tests
  sqlite3/test
  unittest/test
  
That last is somewhat ironic since Michael opened the issue :).

I'm considering working on some of the uncommitted patches for sqlite3, so I may take on moving those tests.
msg179587 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-01-10 19:37
Talked to Michael in IRC.  He isn't particularly in favor of the move of the unittest tests, but doesn't object if someone else wants to do it.  So, unassigning the issue from him.
msg179588 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2013-01-10 19:45
ctypes/tests is within my area of interests and may prove to be one of the harder pieces.  I'll try to move it and report the results.  I have not worked with the rest, so unless it is truly trivial this will have to wait for another volunteer.
msg179589 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2013-01-10 19:58
I honestly don't see the point of moving tests around.
msg179596 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-01-10 20:42
Two reasons for collecting all of the tests in a single location:

1) Facilitates test discovery
2) It makes packaging of CPython easier for Linux distros that prefer to leave the tests out of the core package
msg182809 - (view) Author: Geoff Wilson (gmwils) * Date: 2013-02-23 20:10
Patch attached to move sqlite3 tests under Lib/test, and remove Lib/test/test_sqlite.py.

Naming of files has been kept the same in the move from Lib/sqlite/test, to allow for easier merging of future patches.
msg182822 - (view) Author: Geoff Wilson (gmwils) * Date: 2013-02-23 21:04
Patch attached to move Lib/lib2to3/tests to Lib/test/test_lib2to3.
msg182913 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-02-25 03:23
> Naming of files has been kept the same in the move from
> Lib/sqlite/test, to allow for easier merging of future patches.

This should be done with "hg mv" -- this will also allow to change the name while preserving the history if that's desirable.
msg182926 - (view) Author: Geoff Wilson (gmwils) * Date: 2013-02-25 07:50
The move will need to be done by someone with commit access. These patches came from using hg mv.

After doing the move, there is some cleanup needed in each. These changes are included in the attached patches.
msg182952 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2013-02-25 15:55
Mercurial’s diff formats are actually able to represent file creation, deletion and rename.
msg182954 - (view) Author: Geoff Wilson (gmwils) * Date: 2013-02-25 16:04
Odd. I must be doing something wrong. My test workflow was:
1. hg mv <a> <b>
1a. modify files to resolve issues from the move
2. hg diff > issueNNNN.patch # attached
3. hg revert --all
4. patch -p1 < issueNNNN.patch

Reading the hg docs more, I should have used 'hg patch issueNNNN.patch'. However trying that on a trivial file move, results in a delete/create still.
msg182982 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-02-25 20:51
Geoff, you need to use hg's optional "git" format diff to preserve rename info.  See "hg help diffs".
msg183074 - (view) Author: Geoff Wilson (gmwils) * Date: 2013-02-26 19:48
Thanks Ned! Attached is an update for sqlite tests with the right patch format (issue10572-sqlite3-2.patch).

May make sense to mention the hg diff --git format in:
http://docs.python.org/devguide/patch.html
msg183078 - (view) Author: Geoff Wilson (gmwils) * Date: 2013-02-26 20:12
Attach updated patch for lib2to3 (issue10572-lib2to3-2.patch)
msg183087 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-02-26 20:56
Geoff, thanks, it is documented elsewhere in the devguide but it should be mentioned there as well.  I've added a note to Issue16931.
msg224186 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-07-28 20:41
Can we get this into 3.5 for the reasons given in msg179596 ?
msg224299 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2014-07-30 11:44
I still dislike moving tests around.
msg224342 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2014-07-30 20:29
I'm in favor of a move, especially for ctypes, sqlite3, unittest, and tkinter, and don't really see a reason not to move distutils and lib2to3.  The IDLE tests make a certain amount of sense to stay in the idlelib folder, just because IDLE is a special case.  I'm pretty sure that covers all of the packages that still have tests outside of the test package.

I think the movements should only happen on 3.5; merging forward isn't nearly as much of an issue here in the days of hg, and backports to 2.7 should be becoming fewer and farther between.

I'd be happy to take care of making the moves, if we can decide that it's worth doing (or if it's left up to me :).
msg225079 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2014-08-08 18:27
Here's an updated patch to move the sqlite3 tests.  It's a large patch because it simplifies things by renaming all of the test methods from "Checkxxx" to "testxxx".  The test files are also renamed to fit the usual pattern.
msg225083 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2014-08-08 19:01
Here's a patch that moves the ctypes tests.  On this one, I remembered to patch Makefile.pre.in; my sqlite3 patch will need the equivalent change as well.
msg360661 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2020-01-24 23:26
In case anyone cares, this is still a thing.
msg361240 - (view) Author: Ido Michael (Ido Michael) * Date: 2020-02-02 14:54
Are there any conclusions? what needs to be done?
msg361325 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2020-02-03 20:09
Someone probably needs to go through the stdlib to find which modules have tests outside of Lib/test/, and then figure out if there is still any justification for that. If there isn't then they should be moved to Lib/test.
msg361401 - (view) Author: Ido Michael (Ido Michael) * Date: 2020-02-05 02:59
Thanks Brett,

I'll check this out over the week.
Let's take this module for instance: Lib/distutils.

I looked on this one test file by now, it looks decoupled (test_bdist.py), 

1.What to look for
2.Once decided it's not the right path, just migrate to Lib/tests/"module" and make sure it runs fine?
msg361446 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2020-02-05 17:43
distutils might get pulled out at some point to it's at the bottom of the priority list for now.

Basically move the tests for a package to under Lib/test under the proper subpackage name and make sure the tests didn't break due to the move. That's pretty much the extent of this work.
msg362090 - (view) Author: Ido Michael (Ido Michael) * Date: 2020-02-16 18:21
Moved all 6 modules tests under Lib/test/MODULE_NAME_test
Created this PR: GH-18524
msg362101 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-02-16 20:56
As noted in the review of the PR, keep in mind that tests are included in an installed Python instance; there is a list of test directories in the main Makefile (Makefile.pre.in) that is used by "make install".  Any movement of test directories thus needs to update the Makefile and test with an installed Python.

That said, I'm not sure what this buys us. And are any of the moved tests inherited from third-party packages that we've semi-vendorized into the cpython repo?
msg362844 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2020-02-27 21:25
@ned.deily the motivation is covered in Michael's initial comment; some vendors like to leave out the tests and not having them all under 'test' prevents that from working cleanly. Plus it makes the tests more discoverable.

As for any of the projects being vendored from outside the stdlib, none of them fall under https://www.python.org/dev/peps/pep-0360/ so it isn't a worry. Plus none of them are anymore to my knowledge anyway.
msg363106 - (view) Author: Ido Michael (Ido Michael) * Date: 2020-03-01 22:59
I've created a clean PR - will do it in 2 rounds:
This PR GH-18727, contains the modified test of ctypes, sqlite3 and tkinter modules. Once that's approved I'll deal with the other remaining 3.
msg364232 - (view) Author: Ido Michael (Ido Michael) * Date: 2020-03-15 13:50
Before I put more time into that, here's the strategy here:

1. Module by module migration.
2. By this order
 tkinter/test
  distutils/tests
  ctypes/tests
  lib2to3/tests
  unittest/test

*  sqlite3/test - excluded because needs more work.

Please let me know if there is a different order or more exclusions?
msg364233 - (view) Author: Ido Michael (Ido Michael) * Date: 2020-03-15 13:51
+ Distutils is last priority
msg364248 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2020-03-15 17:39
I would advise against this for two reasons:

- git does not track renames, contrart to Mercurial, so every log/diff command needs to analyse files to determine if something was moved rather than deleted + created; the matching can change depending on a command-line param, and git log needs a --follow option.

- github does not apply --follow, so history for moved files is not useful: https://github.com/idomic/cpython/commits/bpo-10572-testlibs/Lib/test/tkinter_test/test_tkinter/test_widgets.py (for some reason, annotate works: https://github.com/idomic/cpython/blame/bpo-10572-testlibs/Lib/test/tkinter_test/test_tkinter/test_widgets.py )
msg364312 - (view) Author: Ido Michael (Ido Michael) * Date: 2020-03-16 12:20
Hey Eric,

I'm not sure what you mean, you don't recommend the change in general?
Or you just don't recommend Git for this and saying I should use Mercurial instead?
msg364357 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2020-03-16 20:44
Yes, I recommend against this change.  Sorry for the time already spent :(

(I only named Mercurial because python-dev used it before, so I was comparing the behaviour of the two tools)
msg364358 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2020-03-16 20:58
I'm personally still in favor of the change.  This wouldn't be the first big movement of files in our repo and likely not the last, and I don't think git/GitHub's support (or lack thereof) for file moves is a good enough reason to not organize things better.
msg364359 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-03-16 21:04
> some vendors like to leave out the tests and not having them all under 'test' prevents that from working cleanly

Yeah, but presumably all the major distributors have long ago figured out how to do this (it's not *that* hard) and, by moving the tests now, would actually make work for them :) (Granted, that's also not that hard)

Losing the history of a bunch of files would be a much more serious concern, IMHO.  If the files can't be moved without losing history, I'de be a definite -1 on this change; otherwise, a meh -0.
msg364448 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2020-03-17 16:57
I'm also still in favour of the change. While people may have worked around this that doesn't mean we need to keep forcing them to do so. People worked around our lack of booleans but we chose to still fix that. ;)
msg364827 - (view) Author: Ido Michael (Ido Michael) * Date: 2020-03-22 22:06
GH-18727

Only tkinter module for start.
History
Date User Action Args
2020-03-22 22:06:01Ido Michaelsetmessages: + msg364827
2020-03-17 16:57:45brett.cannonsetmessages: + msg364448
2020-03-16 21:04:56ned.deilysetmessages: + msg364359
2020-03-16 20:58:08zach.waresetmessages: + msg364358
2020-03-16 20:44:30eric.araujosetmessages: + msg364357
2020-03-16 12:20:48Ido Michaelsetmessages: + msg364312
2020-03-15 17:39:37eric.araujosetmessages: + msg364248
2020-03-15 13:51:58Ido Michaelsetmessages: + msg364233
2020-03-15 13:50:41Ido Michaelsetmessages: + msg364232
2020-03-01 22:59:40Ido Michaelsetmessages: + msg363106
2020-03-01 22:51:24Ido Michaelsetpull_requests: + pull_request18084
2020-02-27 21:25:09brett.cannonsetmessages: + msg362844
2020-02-16 20:56:07ned.deilysetmessages: + msg362101
2020-02-16 18:21:28Ido Michaelsetmessages: + msg362090
2020-02-16 18:07:18Ido Michaelsetpull_requests: + pull_request17901
2020-02-05 17:43:26brett.cannonsetmessages: + msg361446
2020-02-05 02:59:20Ido Michaelsetmessages: + msg361401
2020-02-04 08:13:06asvetlovsetnosy: - asvetlov
2020-02-03 20:09:35brett.cannonsetmessages: + msg361325
2020-02-02 14:54:54Ido Michaelsetnosy: + Ido Michael
messages: + msg361240
2020-01-24 23:26:13brett.cannonsetversions: + Python 3.9, - Python 3.6
2020-01-24 23:26:08brett.cannonsetmessages: + msg360661
2016-09-09 00:27:31BreamoreBoysetnosy: - BreamoreBoy
2016-09-08 22:58:26christian.heimessetversions: + Python 3.6, - Python 3.5
2014-08-08 19:01:34zach.waresetfiles: + issue10572-ctypes.diff

messages: + msg225083
2014-08-08 18:27:16zach.waresetfiles: + issue10572-sqlite3.diff

messages: + msg225079
2014-07-30 20:29:02zach.waresetmessages: + msg224342
2014-07-30 11:44:50michael.foordsetmessages: + msg224299
2014-07-30 11:37:23berker.peksagsetnosy: + berker.peksag

stage: needs patch -> patch review
2014-07-28 20:41:43BreamoreBoysetnosy: + BreamoreBoy

messages: + msg224186
versions: + Python 3.5, - Python 3.4
2013-04-16 19:06:46zach.waresetnosy: + zach.ware
2013-02-26 20:56:00ned.deilysetmessages: + msg183087
2013-02-26 20:12:07gmwilssetfiles: + issue10572-lib2to3-2.patch

messages: + msg183078
2013-02-26 19:48:39gmwilssetfiles: + issue10572-sqlite3-2.patch

messages: + msg183074
2013-02-25 20:51:01ned.deilysetnosy: + ned.deily
messages: + msg182982
2013-02-25 16:04:21gmwilssetmessages: + msg182954
2013-02-25 15:55:24eric.araujosetmessages: + msg182952
2013-02-25 07:50:24gmwilssetmessages: + msg182926
2013-02-25 03:23:04ezio.melottisetmessages: + msg182913
2013-02-23 21:04:25gmwilssetfiles: + issue10572-lib2to3.patch

messages: + msg182822
2013-02-23 20:10:21gmwilssetfiles: + issue10572-sqlite3.patch

nosy: + gmwils
messages: + msg182809

keywords: + patch
2013-01-10 20:42:51brett.cannonsetmessages: + msg179596
2013-01-10 19:58:31benjamin.petersonsetmessages: + msg179589
2013-01-10 19:45:18belopolskysetmessages: + msg179588
2013-01-10 19:37:49r.david.murraysetassignee: michael.foord ->
messages: + msg179587
2013-01-10 19:14:01r.david.murraysettype: behavior -> enhancement
components: + Tests
versions: + Python 3.4, - Python 3.2
nosy: + benjamin.peterson, asvetlov

messages: + msg179583
stage: needs patch
2011-01-25 18:19:05belopolskysetnosy: barry, brett.cannon, theller, rhettinger, ghaering, belopolsky, tarek, gpolo, ezio.melotti, eric.araujo, r.david.murray, michael.foord
messages: + msg127032
title: Move unittest test package to Lib/test -> Move test sub-packages to Lib/test
2010-11-29 19:11:38brett.cannonsetmessages: + msg122852
2010-11-29 18:49:43rhettingersetmessages: + msg122847
2010-11-29 18:35:39eric.araujosetmessages: + msg122841
2010-11-29 18:24:37r.david.murraysetmessages: + msg122838
2010-11-29 17:04:28barrysetmessages: + msg122830
2010-11-29 16:54:37r.david.murraysetmessages: + msg122828
2010-11-29 16:00:39belopolskysetnosy: + gpolo
2010-11-29 15:57:04michael.foordsetmessages: + msg122820
2010-11-29 15:52:18belopolskysetnosy: + belopolsky, barry, brett.cannon, rhettinger, r.david.murray, theller, ghaering, tarek
messages: + msg122818
2010-11-29 15:42:27michael.foordsetmessages: + msg122817
2010-11-29 15:34:37ezio.melottisetnosy: + ezio.melotti
messages: + msg122815
2010-11-29 13:28:44eric.araujosetnosy: + eric.araujo
2010-11-29 00:52:57michael.foordcreate