This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Speed up test_subprocess
Type: performance Stage: resolved
Components: Tests Versions: Python 3.1, Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: brian.curtin, flox, gregory.p.smith, pitrou, tim.golden
Priority: normal Keywords: patch

Created on 2010-09-18 18:24 by pitrou, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
noleaking.patch pitrou, 2010-09-18 18:24
Messages (6)
msg116821 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-18 18:24
test_subprocess is currently very slow and the main culprit is test_no_leaking (here, it takes 80s out of the 100s needed for the whole run). This tests spawns as many subprocesses as would be needed to reach the max file descriptor limit. Spawning a subprocess is a heavy operation and consequently the test is very long.

This patch adopts another approach: it opens many file descriptors (using os.open()) until the max fd limit is reached, closes some of them (10 is enough) and then launches a bunch of subprocesses in a loop to check that they succeed. Consequently, the test is very fast (even on a Windows VM in debug mode).
msg116826 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2010-09-18 21:12
Overall I like the approach.

A few questions / clarifications:

 is errno.EMFILE portable?  will that errno test work on both posix and
windows?

 should the is_resource_enabled('subprocess') stuff be preserved (why was it
there to begin with? because the old test took so long?)
msg116827 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-18 21:27
> A few questions / clarifications:
> 
>  is errno.EMFILE portable?  will that errno test work on both posix and
> windows?

It does (tested under Linux and a Win64 VM).

>  should the is_resource_enabled('subprocess') stuff be preserved (why was it
> there to begin with? because the old test took so long?)

I guess the latter: the test was so long, especially on slow machines,
that it was only enabled selectively.
msg116828 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2010-09-18 21:29
looks good to me.  commit it. :)
msg116829 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2010-09-18 21:37
+1 fo faster regrtest
msg116832 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-18 22:44
Done, thank you.
(r84883, r84884, r84885)
History
Date User Action Args
2022-04-11 14:57:06adminsetgithub: 54104
2010-09-18 22:44:57pitrousetstatus: open -> closed
resolution: fixed
messages: + msg116832

stage: patch review -> resolved
2010-09-18 21:37:05floxsetnosy: + flox
messages: + msg116829
2010-09-18 21:35:21floxsetfiles: - unnamed
2010-09-18 21:35:11floxsetfiles: - unnamed
2010-09-18 21:29:57gregory.p.smithsetfiles: + unnamed

messages: + msg116828
2010-09-18 21:27:36pitrousetmessages: + msg116827
2010-09-18 21:12:43gregory.p.smithsetfiles: + unnamed

messages: + msg116826
2010-09-18 18:24:54pitroucreate