tests.script_helper provides various utilities for sensibly testing scripts in a subprocess. It isn't easy for CPython developers to discover, since it is isn't documented and the file is mixed in with actual tests in the main test directory.

As discussed in #15494, it should be:

1. Moved from Lib/test/ to Lib/test/support (and imports in tests adjusted accordingly)
2. Documented in Doc/library/test.rst

These changes should be made on both the 3.3 and default branches to avoid spurious merge conflicts.

Since I am already on bugs.python.org/issue18578 I will tackle this issue, if you don't mind.

I have finished the changes needed to move script_helper from Lib/test to Lib/test/support, and I have also documented it. Tests run gracefully (but be sure to check out if it works as intended on your machines too)

Two notes though:

1. I have only made the changes on the default branch. Will do 3.3 tomorrow.

2. When it came to the documentation, I had to break the 80 characters limit, as going by it was resulting in weird rendering or wrong information depicted from sphinx (for example, when showing that script_helper functions belonged to test.support instead of the correct location, test.support.script_helper).

Last but not least, this is one of my first contributions to Python, so I would appreciate your input.

I wouldn't worry about 3.3 at this point - the 3.3 test suite isn't going to see major changes for its final release, so the risk of merge conflicts is low.

Hello again.

Is everything ok with the patch? Is there something not working as expected? Perhaps an omission or something? Do I need to do something more to get it accepted?

Thanks for your time.

Please left test.script_helper as alias to test.support.script_helper.

I.e. test/script_helper.py should contains something like:

from test.support.script_helper import *

And test this with unmodifiable other tests.

Ok, here is the second (modified patch) which contains a script so that no modifications are required to existing tests.

I am uploading it as a second patch, so that the first one is left as a reference.

As a sidenote, I fail to see convincing reasons for why we would want the second solution more than the first. The first one looks cleaner to my eyes. The only downside I can see to the original approach is breaking a couple of tests, which is something that can be fixed easily without even making it to the repo (running the tests in 2-3 could help find all possible broken tests beforehand).

Could someone explain to me why the second solution is more desirable?

Third party code can use tests.script_helper.

I prefer to do these changes in several steps:

1) Move script_helper.py to Lib/test/support/, document it, and create an alias (with deprecation warning).
2) Ensure that this doesn't break any buildbot.
3) Change tests to use test.support.script_helper instead of test.script_helper.
4) One or two versions later remove an alias.

Oh, I see. Does the last patch meet your expectations, or would you rather see all these changes implemented at once?

> Does the last patch meet your expectations, or would you rather see all these changes implemented at once?

I left this for Nick.

I have added comments on Rietveld.

Taking the feedback during code review, this is a patch that has the points raised by Serhiy Storchaka fixed.

As always, please do note omission or mistakes from my part. Thanks for your help.

Note that the test.* namespace has long been declared as unstable, so we
don't worry about breaking third party code when rearranging the test suite
(as far as I am aware, most Linux distros don't even include the test suite
in their Python packages).

I'll see how Mercurial goes tracking the file move when the alias is left
behind, and decide which way to do the initial change based on whether or
not the alias breaks the history tracking.

The attached patch just moves the file without leaving an alias behind. It turns out we use this one in a fair few places (mostly for "assert_python_ok").

I'm going to sit on this change for the moment - after reading the docs, it's quite clear that the last couple of functions don't really belong in script_helper, but rather in the pkg_helper module discussed in issue 15376.

So my current plan is to factor that helper out *first*, then when this one lands, the intent of grouping more of the support code that folks working on the test suite might be interested in using together will hopefully be clearer.

script_helper was moved to the test.support package in issue 9517, so this issue is just about adding the docs now.

Also noting that my draft docs are over a year old now, so they need to be reviewed against the current state of the helper module.

I'm going to take a look at bringing these docs up to date and including the changes from issue24033.

@ncoghlan I've taken the test.rst changes from the previous diff and reviewed them against the latest script_helper (including updates from issue24033). Nothing had fallen behind, but I've added what was missing.

* I've realized that spawn_python and the helper run_python I added in issue24033 are basically the same so I will reconcile those
* Is there some way to load the docstrings from script_helper.py in test.rst instead of writing explanations for each method twice?

If you diffed your patch from a public revision in the main repository, there should be a nice “Review” link. Anyway, here are some comments on issue18576.patch:

+.. function:: assert_python_ok(*args, **env_vars)
+
+   Runs the interpreter with *args* and optional environment
+   variables, asserting that *env_vars* succeeds (``rc == 0``).

Perhaps you mean *env_vars* are the optional environment variables, and that the interpreter is asserted to have succeeded.

Also maybe clarify where the *__cleanenv* and *__isolated* keywords come from? Function signatures? Also applies to assert_python_failure(), run_python/_until_end().

Maybe it is worth mentioning that assert_python_ok/failure() strip whitespace from the stderr stream. I seem to remember being tricked by this (writing a test to ensure stderr ended in a newline or something).

+   ``stdout`` are configured as binary pipes and ``stderr`` is merged with
+   ``stdout``..

There are two full stops above here..

I encourage you to start sentences with capital letters (although I admit sometimes other people do not agree with this). Suggestions:

+   *kw* is passed through to subprocess.Popen as additional keyword
+   arguments.
=> The *kw* arguments are passed through to subprocess.Popen.

+   *source* is a Unicode string which will be written to the file as UTF-8.
=> The *source* argument is a Unicode string which will be written to the file as UTF-8.

+   *depth* controls how many deeply nested the package is.
=> The *depth* argument controls how deeply nested the package is.

+   Creates a new zip archive in the given directory, containing the specified
+   Python script/module

Needs one of those full stops I mentioned earlier :)

As far as using docstrings for the online docs goes, there are some useful Sphinx add-ons that we've never enabled for the main Python docs (like autodoc, blockdiag, seqdiag) that could be good options to have available. For autodoc in particular, we prefer to avoid it for the main docs because "good docstring" and "good prose documentation" don't necessarily look alike. For ease of maintenance of test.support though, I think the "write a good docstring, get the online docs for free" rationale is a compelling one (since these docs are core contributor facing, rather than being aimed at Python users in general)

docs@python.org would be the place to ask about the appropriateness of enabling some Sphinx add-ons. While enabling them seems like a reasonable idea to me, I don't personally know if there are actually technical issues with the idea, or if there just hasn't been a volunteer to add/enable them and update https://docs.python.org/devguide/documenting.html accordingly.

> If you diffed your patch from a public revision in the main repository, there should be a nice “Review” link.

Ah, thanks @vadmium!!! I've been confused about why some patches get review links and others do not >.<

I'll have a new patch up shortly which is diffed against a public revision and has your suggested changes.

> For autodoc in particular, we prefer to avoid it for the main docs because "good docstring" and "good prose documentation" don't necessarily look alike. 

@ncoghlan for some reason I thought that the main docs were also auto generated from docstrings in the actual modules, but I can see now that that isn't the case.

I'll reach out to docs@python.org to see how they feel about autogenerating the test.support docs.

Hey @vadmium!

I've finally responded to your review feedback. I've attached a new patch, hopefully this time actually review-able. I've removed the docs for methods I added in @24033 so that this could be merged without it.

Let me know if you have any more feedback, or if this could be merged.

Thanks!

I left some more comments on Reitveld. The main one is I think stderr gets stripped with assert_python_ok(), but I don’t think stdout is touched. Correct me if I’m wrong though.

The PR on GitHub is based on bobcatfist's patch, addressed on Martin request and some minor change.

All those functions has already been documented.