This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Allow doctest to select tests via -m/--match option
Type: enhancement Stage: patch review
Components: Tests Versions: Python 3.10
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: costas-basdekis, steven.daprano, tim.peters
Priority: normal Keywords: patch

Created on 2020-12-31 19:18 by costas-basdekis, last changed 2022-04-11 14:59 by admin.

Pull Requests
URL Status Linked Edit
PR 24028 open costas-basdekis, 2020-12-31 19:23
Messages (3)
msg384132 - (view) Author: Costas Basdekis (costas-basdekis) * Date: 2020-12-31 19:18
Most testing frameworks allow for only a subset of tests to be run, and the reason is usually either to focus on a specific test among many failing ones, or for speed purposes if running the whole suite is too slow.

With doctests, it's usually the case that the are light and fast, but if you make a breaking change you can still have many tests failing, and you want to focus only on one.

This proposition adds an `-m`/`--match` option to the doctest runner, to allow to select 1 or more specific tests to run.

The proposed format for the parameters is :
* <test-name-suffix>:<indexes>
* <test-name-suffix> is a glob-like string where '*' characters can match any substring of a test's name, and an '*' is implicitly added to the start of the pattern: eg 'do_*_method' matches '__main__.do_a_method' and '__main__.MyClass.do_b_method'
* <indexes> is a list of numbers or ranges to match the 0-based index examples within the test:  eg '1' matches the second example, `-3` matches the first 4 examples, '4-` matches all but the first 4 examples, and `-3,5,7-10,20-` matches examples 0, 1, 2, 3, 5, 7, 8, 9, 10, 20, and the rest
msg384142 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2020-12-31 23:39
I think that -m is unacceptable as it will clash with Python's -m option.

I'm not convinced that this added complexity will provide enough benefit to make it worth while. I can see myself spending an order of magnitude more time trying to work out why my test suffixes and indexes are selecting the wrong tests than I would save.

The use of indexes to select individual tests within a single docstring is especially fragile and error-prone since it requires counting by the user. It's also likely to introduce artificial errors since tests can rely on side-effects of previous tests, e.g.

    >>> import math
    >>> math.sqrt(25)
    5.0

Selecting test number 1 alone will fail unless test number 0 is also run. For these reasons, I think that in practice, the finest selection unit we can run is a single docstring. Trying to select individual tests within a unit is likely to be far too fragile and error prone to be practical.

So by my estimation, we have the following selections:

- module docstring alone
- named function or class docstring alone
- named class.method docstring alone
- named class and all its methods
msg384158 - (view) Author: Costas Basdekis (costas-basdekis) * Date: 2021-01-01 09:40
Thanks for the thorough feedback Steven (and happy new year).

About -m flag: we could use -k as other testing frameworks use (pytest), or even -t.

About selecting multiple indexes: what you described is my exact use case, as I have a lot of failing examples in one method with verbose output, and also the examples depend on previous examples to be able to run, I'm currently debugging with `-m Ground.step:3,4,6,7`.

So I think you have very valid points, and I'd like to find a way to make the extra overhead be worth it:
* Currently the failing case output is f'File "{path}", line {line}, in {name}': we can add the index as f'File "{path}", line {line}, in {name}, example {index}', so it would be easy to know the indexes
* I think globs in the pattern have been very useful to me in the past (with frameworks that allowed it), because sometimes similarly named functions can have the same types of errors you try to focus on. In any case, it looks like less effort to allow arbitrary globs rather than matching on module name parts

I understand I'm adding complexity, and if we could support the same behaviour in a simpler way I'd be glad to attempt that. In the very least I think allowing a `finder=None` argument in `testmod` would allow more complex solutions, even if they are not accepted in `doctest`.

In the end I'd welcome even a small improvement (for my personal usecase).
History
Date User Action Args
2022-04-11 14:59:39adminsetgithub: 86963
2021-01-01 09:40:34costas-basdekissetmessages: + msg384158
2020-12-31 23:39:01steven.dapranosetnosy: + steven.daprano, tim.peters
messages: + msg384142
2020-12-31 19:23:22costas-basdekissetkeywords: + patch
stage: patch review
pull_requests: + pull_request22868
2020-12-31 19:18:13costas-basdekiscreate