classification
Title: Bring test.support docs up to date
Type: behavior Stage: resolved
Components: Documentation, Tests Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ncoghlan Nosy List: cheryl.sabella, docs@python, eli.bendersky, ezio.melotti, georg.brandl, giampaolo.rodola, ncoghlan, python-dev, sandro.tosi, serhiy.storchaka, terry.reedy
Priority: normal Keywords: easy, patch

Created on 2011-01-26 02:46 by ncoghlan, last changed 2018-02-11 19:43 by terry.reedy. This issue is now closed.

Files
File name Uploaded Description Edit
issue11015.py3k.testdoc.1.patch eli.bendersky, 2011-01-28 05:38 review
issue11015.py3k.remove_fcmp.1.patch eli.bendersky, 2011-02-25 07:36 review
issue11015.py3k.remove_fcmp.2.patch eli.bendersky, 2011-02-25 09:48 review
Pull Requests
URL Status Linked Edit
PR 5610 merged cheryl.sabella, 2018-02-10 17:50
PR 5619 merged miss-islington, 2018-02-11 13:11
Messages (30)
msg127086 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-01-26 02:46
The test.support docs are there to help CPython devs with writing good unit tests more easily. There are a few additions we've made in recent years that haven't made it into the .rst file, so it is easy to miss useful tools if you don't go looking through the module source code.

There are some other helper modules (such as test.script_helper) that could also stand to be made easier to find.

Fixing this is just a matter of doing a pass through test.support and the test directory looking for things that might be worth mentioning in the test package docs.
msg127242 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2011-01-28 03:42
I don't know if it matters much, but there's a slight mismatch in the description of test.support.verbose. The documentation says it's a boolean, while it's 0 or 1 in reality. 

Can it just be changed to True/False in the code of test.support and test.regrtest? It appears that all the other flags are True/False and there's no reason to keep this 0/1 (which is probably a relic from a distant past)
msg127244 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2011-01-28 04:19
Here's a patch fixing the 0/1 to True/False in a couple of places in test.support and test.regrtest

I ran the test suite and it passes.

A review is needed to commit. I'll keep working on the documentation itself in the meantime.
msg127247 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2011-01-28 05:38
Here's a patch to Doc/library/test.rst with additional several exported functions documented. These are the ones I found most important and clear. fcmp() is currently not documented (pending discussion in pydev).
msg127260 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-01-28 09:23
verbose isn't a boolean at all - it's an integer. You can supply it multiple times to bump the logging level up even higher than normal.

~/devel/py3k/Lib/test$ grep "verbose >" *.py
regrtest.py:                    if self.verbose > 1:
test_cmd_line_script.py:        if verbose > 1:
test_cmd_line_script.py:        if verbose > 1:
test_cmd_line_script.py:        if verbose > 1:
test_cmd_line_script.py:                if verbose > 1:
test_cmd_line_script.py:                    if verbose > 1:
test_fork1.py:                    if verbose > 1:
test_fork1.py:                    if verbose > 1:

(I didn't check explicitly, but I believe those verbose values are shorthand references to test.support.verbose)
msg127297 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2011-01-28 13:03
Nick, agreed regarding verbose. Somehow I didn't think it would be used in other modules like that, and only looked in regrtest where I didn't see anything meaningful about verbose not being binary.

It's a good thing to document it, then :)
msg129334 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2011-02-25 07:36
Following the python-dev discussion, attaching a patch for removing fcmp and replacing its uses with assertAlmostEqual when needed.

All tests pass and patchcheck is clean.

Please review before I commit.
msg129335 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2011-02-25 08:08
The divmod() part of the patch is wrong: assertAlmostEqual does not support tuple arguments.

The test succeeds because it first does an exact equality check, which apparently is true on your platform.  But if it wasn't, you'd get TypeErrors.

Also, fcmp() has a different definition of what "almost equal" means, but I assume this has been regarded in the discussion.
msg129339 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2011-02-25 09:30
Georg,

Good catch on the tuples in assertAlmostEqual, thanks! I see three methods of resolution:

1. Since the floats in this case are powers of 1/2, they could be tested for exact equality, or do you figure there are platforms where this is not guaranteed?
2. I can make a local change in the test to assertAlmostEqual on each member of the tuple
3. assertAlmostEqual on tuples can be added to the unittest module

Any ideas?

I can also commit (2) for the time being since it's the least dubious, and leave the resolution on 1 & 3 for later (and/or in separate issues).

Regarding fcmp being not exactly assertAlmostEqual - it was discussed.
msg129340 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2011-02-25 09:36
(2) would be my choice.  (1) *should* be true, but this is a change in the test semantics.  (3) would be feature creep and I don't think it's a good idea.
msg129341 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2011-02-25 09:48
Attaching a revised patch with (2) implemented.
msg129343 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2011-02-25 09:57
Yes, that looks good now.
msg129346 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2011-02-25 10:15
> Georg Brandl <georg@python.org> added the comment:
>
> Yes, that looks good now.
>

Thanks. fcmp & FUZZ removal committed in revision 88558
msg135160 - (view) Author: Sandro Tosi (sandro.tosi) * (Python committer) Date: 2011-05-04 22:14
Hi all,
IIUIC we are left with issue11015.py3k.testdoc.1.patch) since issue11015.py3k.remove_fcmp.{1,2}.patch has been already applied on default.

I just gave a look to the doc patch and it seems fine (it also applies without any warning on default).

Eli, do you want to expand this patch further (and how :) or do you think it's still the version you want to commit? Can a core devel, then, give this patch a deeper look?
msg135165 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-05-04 23:57
This is an improvement that I think should be committed before 3.2.1.
Some comments:

+.. function:: run_doctest(module, verbosity=None)
+   Run :mod:`doctest` on the given *module*.
should be, I believe,
+   Run :func:`doctest.testmod` on the given *module*.
as that is what the function actually does (I check the code).

+   If *verbosity* is :const:`None`, :meth:`doctest` is run with verbosity set
+   to :data:`verbose`.  Otherwise, it is run with verbosity set to
+   :const:`None`.

Should :meth:`doctest` be :func:`testmod` ?
Otherwise the proposed text rewrites

"    If optional argument verbosity is not specified (or is None), pass
    support's belief about verbosity on to doctest.  Else doctest's
    usual behavior is used (it searches sys.argv for -v)."

The problem with the rewrite is that the keyword param of testmod is 'verbose', not 'verbosity'. 'Verbosity' is a dummy name used to either pass support.verbose to verbose, or not. So testmod is, in net effect, run with verbose=verbose or verbose=None. My attempt to explain a bad design (with probable markup errors):

"If *verbosity* is :const:`None`, :func:`testmod` is run with verbose set to :data:`support.verbose`, which is set by :func:`regrtest`. Otherwise, it is run with verbose set to :const:`None` and subsequently replaced by :code:`'-v' in sys.argv`."


+.. function:: temp_umask(umask)
+
+   A context manager that temporarily sets the process umask to the
+   given value.

"sets the process umask to *umask*." ?


+.. function:: find_unused_port(family=socket.AF_INET, socktype=socket.SOCK_STREAM)

+   Either this method or :func:`bind_port` should be used for any tests
+   where a server socket needs to be bound to a particular port for the
+   duration of the test.
+   Which one to use depends on whether the calling code is creating a python
+   socket, or if an unused port needs to be provided in a constructor
+   or passed to an external program (i.e. the ``-accept`` argument to
+   openssl's s_server mode). 

This is copied from the doc string but does really tell me which to use in which of the two situations.

Other additions look OK to me. Some copied docstrings (or comments). Some are new. Support.py could also use a patch to add missing docstings (and turn a couple of comments into docstrings).
msg135171 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2011-05-05 03:19
> Eli, do you want to expand this patch further (and how :) or do you think
> it's still the version you want to commit? Can a core devel, then, give this
> patch a deeper look?
>

I will review this again in a couple of days and will commit.
msg135178 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2011-05-05 10:17
I left a few comments on rietveld for the testdoc and remove_fcmp patches.
Unless I'm missing something, assertEqual works just fine in all the places where fcmp was used, so there's no need to use assertAlmostEqual.
msg135262 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2011-05-06 06:28
Terry,

I've incorporated your suggestions except the new formulation for verbosity to doctest.testmod, since it's not really clear which one is more accurate. I think it's intelligible as it is now (especially given that test.support is for use of Python contributors, and not random users). If you have strong preferences about it, feel free to commit another formulation.

Ezio,

I answered your comments in the code review tool.
Regarding the use of assertEqual for the floating point results, I just didn't want to rely on the exact representation of these values. I'm not 100% sure Python ensures this is true on all platforms it supports. Recall that I was just replacing the fcmp code which did basically the same, also not comparing exactly.

----

I will commit a fixed patch to default. I see no reason to backport this since test.support is just a tool for core-devs and contributors and is mainly used for future developments.
msg135263 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-05-06 06:30
New changeset 2fd435ac3551 by Eli Bendersky in branch 'default':
Issue #11015: bring test.support docs up to date
http://hg.python.org/cpython/rev/2fd435ac3551
msg136423 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2011-05-21 07:23
Can this issue be closed?
msg136435 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2011-05-21 12:59
There are still functions that are not documented, so I think this should stay open until we documented them (unless they shouldn't be documented -- in that case it can be closed).
msg136588 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2011-05-23 04:02
Ezio, you're right, there are still a lot of undocumented symbols (functions, classes, globals). I compiled a list of those I could not find in the .rst docs, excluding ones that are only used in regrtest.py itself:

------------------------------------------

get_attribute
record_original_stdout/get_original_stdout
unload
unlink
rmtree
make_legacy_pyc
IPV6_ENABLED
TESTFN_ENCODED
TESTFN_UNENCODABLE
sortdict
check_syntax_error
open_urlresource
CleanImport
DirsOnSysPath
transient_internet
captured_output
captured_stderr
gc_collect
python_is_optimized
set_memlimit
bigmemtest
precisionbigmemtest
bigaddrspacetest
requires_resource
cpython_only
impl_detail
check_impl_detail
no_tracing
refcount_test
modules_setup / modules_cleanup
threading_setup / threading_cleanup
reap_threads
reap_children
swap_attr
swap_item
strip_python_stderr
TestHandler
Matcher
patch

------------------------------------------
msg173720 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2012-10-24 23:11
> There are some other helper modules (such as test.script_helper)
> that could also stand to be made easier to find.

Documenting script_helper.py would be good too.  I seem to remember some discussion about merging it with support.py -- if that happens it can be documented there, otherwise a separate section/file should be created.
msg311954 - (view) Author: Cheryl Sabella (cheryl.sabella) * (Python committer) Date: 2018-02-10 17:53
I have created a pull request adding documentation for all of test.support, including script_helper.py.  I hope this will be helpful.
msg311994 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2018-02-11 13:10
New changeset 988fb28431d3a3fc5dc6eb657c8a4baacc04d9ce by Nick Coghlan (Cheryl Sabella) in branch 'master':
bpo-11015: Update test.support documentation (GH-5610)
https://github.com/python/cpython/commit/988fb28431d3a3fc5dc6eb657c8a4baacc04d9ce
msg311995 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2018-02-11 13:14
I merged Cheryl's PR to the dev branch, and triggered the backport to 3.7. If nobody beats me to it, I'll merge the latter tomorrow :)
msg311999 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-02-11 16:22
I have left comments to PR.
msg312000 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2018-02-11 16:34
New changeset a0b998df314370f00502efe7ad84206f5bb782ff by Nick Coghlan (Miss Islington (bot)) in branch '3.7':
bpo-11015: Update test.support documentation (GH-5619)
https://github.com/python/cpython/commit/a0b998df314370f00502efe7ad84206f5bb782ff
msg312002 - (view) Author: Cheryl Sabella (cheryl.sabella) * (Python committer) Date: 2018-02-11 17:07
Thanks Nick for merging.  Should I open a new PR to address Serhiy's review or fix them on the original PR?
msg312011 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-02-11 19:43
Since Nick already merged and backported, at least a new PR is needed.  Since this is an old issue with a long discussion and several inactives on the nosy list, I suggest a new issue starting from the merge result as a base: "More revisions to test.support docs".  Make a list of Serhiy's questions and comments.

TESTFN_NONASCII - How different from TESTFN_UNICODE?
PGO -  value? True/False?
TEST_SUPPORT_DIR - Not used.
...

Add Nick, Serhiy, and me as nosy.  We can discuss there how to handle the additional issues.
History
Date User Action Args
2018-02-11 19:43:25terry.reedysetstatus: open -> closed
versions: + Python 3.7
messages: + msg312011

components: + Documentation, Tests
stage: backport needed -> resolved
2018-02-11 17:07:03cheryl.sabellasetmessages: + msg312002
2018-02-11 16:34:30ncoghlansetmessages: + msg312000
2018-02-11 16:22:22serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg311999
2018-02-11 13:14:14ncoghlansetassignee: docs@python -> ncoghlan
resolution: fixed
messages: + msg311995
stage: patch review -> backport needed
2018-02-11 13:11:59miss-islingtonsetpull_requests: + pull_request5428
2018-02-11 13:10:48ncoghlansetmessages: + msg311994
2018-02-10 17:53:46cheryl.sabellasetnosy: + cheryl.sabella

messages: + msg311954
versions: + Python 3.8
2018-02-10 17:50:27cheryl.sabellasetpull_requests: + pull_request5420
2012-10-24 23:11:01ezio.melottisetmessages: + msg173720
2011-05-23 04:02:42eli.benderskysetmessages: + msg136588
2011-05-21 12:59:04ezio.melottisetmessages: + msg136435
2011-05-21 07:23:56eli.benderskysetmessages: + msg136423
2011-05-06 06:30:36python-devsetnosy: + python-dev
messages: + msg135263
2011-05-06 06:28:27eli.benderskysetmessages: + msg135262
2011-05-05 10:17:51ezio.melottisetnosy: + ezio.melotti
messages: + msg135178
2011-05-05 04:10:32terry.reedysetfiles: - unnamed
2011-05-05 03:19:38eli.benderskysetfiles: + unnamed

messages: + msg135171
2011-05-04 23:57:33terry.reedysetmessages: + msg135165
2011-05-04 22:14:49sandro.tosisetmessages: + msg135160
2011-05-04 19:09:53sandro.tosisetnosy: + sandro.tosi
2011-02-25 10:15:44eli.benderskysetnosy: georg.brandl, terry.reedy, ncoghlan, giampaolo.rodola, eli.bendersky, docs@python
messages: + msg129346
2011-02-25 09:57:00georg.brandlsetnosy: georg.brandl, terry.reedy, ncoghlan, giampaolo.rodola, eli.bendersky, docs@python
messages: + msg129343
2011-02-25 09:48:52eli.benderskysetfiles: + issue11015.py3k.remove_fcmp.2.patch
nosy: georg.brandl, terry.reedy, ncoghlan, giampaolo.rodola, eli.bendersky, docs@python
messages: + msg129341
2011-02-25 09:36:52georg.brandlsetnosy: georg.brandl, terry.reedy, ncoghlan, giampaolo.rodola, eli.bendersky, docs@python
messages: + msg129340
2011-02-25 09:30:00eli.benderskysetnosy: georg.brandl, terry.reedy, ncoghlan, giampaolo.rodola, eli.bendersky, docs@python
messages: + msg129339
2011-02-25 08:08:39georg.brandlsetnosy: + georg.brandl
messages: + msg129335
2011-02-25 07:36:33eli.benderskysetfiles: + issue11015.py3k.remove_fcmp.1.patch
nosy: + terry.reedy
messages: + msg129334

2011-01-29 14:31:32giampaolo.rodolasetnosy: + giampaolo.rodola
2011-01-28 13:03:25eli.benderskysetfiles: - issue11015.py3k.1.patch
nosy: ncoghlan, eli.bendersky, docs@python
2011-01-28 13:03:20eli.benderskysetnosy: ncoghlan, eli.bendersky, docs@python
messages: + msg127297
2011-01-28 09:23:53ncoghlansetnosy: ncoghlan, eli.bendersky, docs@python
messages: + msg127260
2011-01-28 05:38:11eli.benderskysetfiles: + issue11015.py3k.testdoc.1.patch

messages: + msg127247
nosy: ncoghlan, eli.bendersky, docs@python
stage: needs patch -> patch review
2011-01-28 04:19:54eli.benderskysetfiles: + issue11015.py3k.1.patch

messages: + msg127244
keywords: + patch
nosy: ncoghlan, eli.bendersky, docs@python
2011-01-28 03:42:24eli.benderskysetnosy: ncoghlan, eli.bendersky, docs@python
messages: + msg127242
2011-01-28 03:03:33eli.benderskysetnosy: + eli.bendersky
2011-01-26 02:46:08ncoghlancreate