classification
Title: Slaves don't seem to clean up their /tmp mess if a step fails.
Type: behavior Stage:
Components: Build Versions:
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: trent Nosy List: Arfrever, chris.jerdonek, pitrou, r.david.murray, trent
Priority: normal Keywords:

Created on 2012-09-18 18:57 by trent, last changed 2012-09-19 06:55 by chris.jerdonek.

Messages (7)
msg170663 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2012-09-18 18:57
All my slaves' /tmp's are polluted with regrtest fluff.  I haven't checked yet, but I presume no cleanup is done if a test/run fails.

nitrogen% find /tmp -user cpython 2> /dev/null | wc -l
197
netbsd51-x64-1$ find /tmp -user cpython 2> /dev/null | wc -l
     142
oxygen% find /tmp -user cpython 2> /dev/null | wc -l                
456

Placeholder issue until it annoys me enough to patch.
msg170667 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-09-18 19:28
Cleanup on test failure is supposed to be done.  Cleanup on crash or buildbot timeout isn't done as far as I know (and that was a concern I had with the changes made to support.TESTFN and the cwd, but I didn't articulate it very well).  

If you find tests that do no clean up on error, that's definitely a bug and we should fix them.
msg170672 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-18 19:55
To follow up on David's comment, the unit tests in the test suite aren't consistent in their treatment of temp directories (e.g. they don't use a common API or code path).  So it may be hard to address this globally short of wiping the entire temp directory (though I could be wrong).

I have a patch in issue 15415 to add a temp_dir() context manager to test.support (and consolidate with script_helper's) that would do such clean-up.  The module already has a temp_cwd(), but that context manager serves two purposes.
msg170687 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-09-18 22:36
> All my slaves' /tmp's are polluted with regrtest fluff.

Which "regrtest fluff" exactly?
msg170703 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2012-09-19 05:42
> > All my slaves' /tmp's are polluted with regrtest fluff.

> Which "regrtest fluff" exactly?

I was being lazy when I wrote that.  By "regrtest fluff" I was referring to lingering files/directories in /tmp, owned by cpython, starting with tmp*.  Few examples below.

openbsd51-x64-1% ls -l /tmp
total 32
drwxr-xr-x  2 root     wheel  512 Aug 18 05:40 aucat
drwx------  4 cpython  wheel  512 Sep 19 02:26 tmp03dyjv
-rw-------  1 cpython  wheel    3 Sep 19 02:03 tmp4t3_bk
-rw-------  1 cpython  wheel    3 Sep 18 05:02 tmp5ljha3
-rw-------  1 cpython  wheel    3 Sep 18 05:02 tmp7p3sht
-rw-------  1 cpython  wheel    0 Sep 19 05:34 tmpCFMYVJ
-rw-------  1 cpython  wheel    3 Sep 19 02:34 tmpcnwqcn
-rw-------  1 cpython  wheel    3 Sep 19 02:34 tmpk0gthu
-rw-------  1 cpython  wheel    3 Sep 19 02:03 tmpmf973r
-rw-------  1 cpython  wheel    0 Sep 19 05:34 tmpqO20FC

dragonfly30-x86-1% ls -l /tmp | grep cpython
drwx------  1 cpython  wheel     0 Aug 22 03:46 ssh-v3WDeO7FhO
-rw-------  1 cpython  wheel     3 Sep 14 16:05 tmp31idrc
-rw-------  1 cpython  wheel     3 Sep 11 15:58 tmp34z8iv
-rw-------  1 cpython  wheel     3 Sep 18 05:06 tmp43uhjt
-rw-------  1 cpython  wheel     3 Sep 19 02:03 tmp4jypkx
-rw-------  1 cpython  wheel     3 Sep 12 04:31 tmp4odqso
-rw-------  1 cpython  wheel     3 Sep 12 20:36 tmp6acej6
-rw-------  1 cpython  wheel     3 Sep 12 16:22 tmp9m5vie
-rw-------  1 cpython  wheel     3 Sep 11 16:15 tmp9oj443
-rw-------  1 cpython  wheel     3 Sep 12 20:10 tmpb086ow
-rw-------  1 cpython  wheel     3 Sep 12 14:02 tmpcm0s2g
-rw-------  1 cpython  wheel     3 Sep 11 16:15 tmpd81jie
-rw-------  1 cpython  wheel     3 Sep 12 17:05 tmpdm8vv9
-rw-------  1 cpython  wheel     3 Sep 15 23:01 tmpef4d2m
-rw-------  1 cpython  wheel     3 Sep 18 05:06 tmpel_xi_
-rw-------  1 cpython  wheel     3 Sep 12 04:31 tmpf41fj3
-rw-------  1 cpython  wheel     3 Sep 11 12:08 tmpi94ajz
-rw-------  1 cpython  wheel     3 Sep 19 02:25 tmpilpy7y
-rw-------  1 cpython  wheel     3 Sep 15 22:27 tmpiq1yo9
-rw-------  1 cpython  wheel     3 Sep 13 16:36 tmpkq6d8i
-rw-------  1 cpython  wheel     3 Sep 11 12:28 tmpkxqa4a
-rw-------  1 cpython  wheel     3 Sep 19 02:25 tmpl35290
-rw-------  1 cpython  wheel     3 Sep 11 12:08 tmpn1dq2c
-rw-------  1 cpython  wheel     3 Sep 12 17:05 tmpnxrnxp
-rw-------  1 cpython  wheel     3 Sep 19 02:03 tmpocz5ux
-rw-------  1 cpython  wheel     3 Sep 14 16:05 tmppsebwf
-rw-------  1 cpython  wheel     3 Sep 11 13:50 tmpptpcep
-rw-------  1 cpython  wheel     3 Sep 15 22:27 tmpqk1s3k
-rw-------  1 cpython  wheel     3 Sep 12 14:02 tmprk8uxy
-rw-------  1 cpython  wheel     3 Sep 11 13:50 tmprqmmfw
-rw-------  1 cpython  wheel     3 Sep 13 16:36 tmps99aky
-rw-------  1 cpython  wheel     3 Sep 11 15:58 tmpsfsmoc
-rw-------  1 cpython  wheel     3 Sep 12 20:36 tmpt8_0vu
-rw-------  1 cpython  wheel     3 Sep 15 23:01 tmpthosfe
-rw-------  1 cpython  wheel     3 Sep 11 12:28 tmpwhsp9p
-rw-------  1 cpython  wheel     3 Sep 12 20:10 tmpy5774k
-rw-------  1 cpython  wheel     3 Sep 12 16:22 tmpzgafbz


oxygen% ls -l /tmp | grep cpython
drwxr-xr-x 3 root    cpython   96 Aug 31 15:41 screens
-rw------- 1 cpython cpython    0 Sep 19 05:30 tmp249ZUA
drwx------ 2 cpython cpython   96 Sep 19 05:31 tmpAvStO1
drwx------ 2 cpython cpython   96 Sep 19 05:31 tmpFLfr2x
drwx------ 2 cpython cpython   96 Sep 19 05:31 tmpOYB_X6
drwx------ 2 cpython cpython   96 Sep 19 05:31 tmpP1Hsdf
-rw------- 1 cpython cpython    0 Sep 19 05:30 tmpTqctqQ
drwx------ 2 cpython cpython   96 Sep 19 05:31 tmpUW1Gov
drwx------ 2 cpython cpython   96 Sep 19 05:31 tmpYU0lzE
drwx------ 2 cpython cpython   96 Sep 19 05:31 tmpazia2w
drwx------ 2 cpython cpython   96 Sep 19 05:31 tmpgemyJY
drwx------ 2 cpython cpython   96 Sep 19 05:31 tmpkfg0aZ
drwx------ 2 cpython cpython   96 Sep 19 05:31 tmpt1e0Jc
msg170704 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2012-09-19 05:49
> Cleanup on test failure is supposed to be done.  Cleanup on crash or 
> buildbot timeout isn't done as far as I know (and that was a concern I
> had with the changes made to support.TESTFN and the cwd, but I didn't
> articulate it very well).  

Ah, yeah, this is almost certainly to do with crash/timeout.  I remember running into this way back in 2008 when I was running a few slaves.

Personally I think the best solution is to have the test framework allocate a single test directory, inform the parent* as to what it is, then make sure all temp files are rooted in it.  The parent* should then rm -rf the tempdir at the end of the run as a final 'catch all'.

parent*: this will mean different things in different contexts... could be the Tools/run_tests.py wrapper, buildbot slave wrapper, etc.
msg170706 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-19 06:55
> Personally I think the best solution is to have the test framework allocate a single test directory

This is partially done.  See here:

http://hg.python.org/cpython/file/19c74cadea95/Lib/test/regrtest.py#l1810

# Run the tests in a context manager that temporary changes the CWD to a
# temporary and writable directory.

regrtest sets things up such that the current working directory is supposed to be this temp directory (constructed from _make_temp_dir_for_build()).  But the tests adhere to this only weakly.  Many or most tests create their own temp directory rather than relying on the caller having set the current working directory to a temp directory.  If the tests used a common API, we could control this behavior globally.
History
Date User Action Args
2012-09-19 06:55:44chris.jerdoneksetmessages: + msg170706
2012-09-19 05:49:36trentsetmessages: + msg170704
2012-09-19 05:42:49trentsetmessages: + msg170703
2012-09-18 22:36:10pitrousetnosy: + pitrou
messages: + msg170687
2012-09-18 21:28:18Arfreversetnosy: + Arfrever
2012-09-18 19:55:06chris.jerdoneksetnosy: + chris.jerdonek
messages: + msg170672
2012-09-18 19:28:52r.david.murraysetnosy: + r.david.murray
messages: + msg170667
2012-09-18 18:57:08trentcreate