classification
Title: Convert test_mmap to use tempfile
Type: Stage: patch review
Components: Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: tim.golden Nosy List: eryksun, tim.golden
Priority: normal Keywords: patch

Created on 2018-07-26 19:22 by tim.golden, last changed 2018-07-27 07:56 by tim.golden.

Pull Requests
URL Status Linked Edit
PR 8486 open tim.golden, 2018-07-26 19:38
Messages (4)
msg322444 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2018-07-26 19:22
test_mmap currently uses the test.support.TESTFN functionality which creates a temporary file local to the test directory named around the pid. 

This can give rise to race conditions where tests are competing with each other to delete and recreate the file.
msg322456 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2018-07-27 02:25
> tests are competing with each other to delete and recreate the file

When is this an issue? Each parallel test process should have its own working directory under "{TEMPDIR}/test_python_{pid}".
msg322464 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2018-07-27 05:39
To clarify, TEMPDIR in this case is from the following code in Lib/test/libregrtest/main.py:

    # When tests are run from the Python build directory, it is best practice
    # to keep the test files in a subfolder.  This eases the cleanup of leftover
    # files using the "make distclean" command.
    if sysconfig.is_python_build():
        TEMPDIR = sysconfig.get_config_var('abs_builddir')
        if TEMPDIR is None:
            # bpo-30284: On Windows, only srcdir is available. Using abs_builddir
            # mostly matters on UNIX when building Python out of the source tree,
            # especially when the source tree is read only.
            TEMPDIR = sysconfig.get_config_var('srcdir')
        TEMPDIR = os.path.join(TEMPDIR, 'build')
    else:
        TEMPDIR = tempfile.gettempdir()
    TEMPDIR = os.path.abspath(TEMPDIR)

Then in class Regrtest we have:

    def main(self, tests=None, **kwargs):
        global TEMPDIR

        if sysconfig.is_python_build():
            try:
                os.mkdir(TEMPDIR)
            except FileExistsError:
                pass

        # Define a writable temp dir that will be used as cwd while running
        # the tests. The name of the dir includes the pid to allow parallel
        # testing (see the -j option).
        test_cwd = 'test_python_{}'.format(os.getpid())
        test_cwd = os.path.join(TEMPDIR, test_cwd)

        # Run the tests in a context manager that temporarily changes the CWD to a
        # temporary and writable directory.  If it's not possible to create or
        # change the CWD, the original CWD will be used.  The original CWD is
        # available from support.SAVEDCWD.
        with support.temp_cwd(test_cwd, quiet=True):
            self._main(tests, kwargs)
msg322471 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2018-07-27 07:56
Thanks for the information, eryksun.

For the moment, I can only say with a fair degree of certainty that using the tempfile functions as I have in test_bz2 & test_mmap appears to solve the issue which is repeatably if intermittently present without that change.

I'll be trying to narrow the issue down before making any further changes
History
Date User Action Args
2018-07-27 07:56:55tim.goldensetmessages: + msg322471
2018-07-27 05:39:18eryksunsetmessages: + msg322464
2018-07-27 02:25:07eryksunsetnosy: + eryksun
messages: + msg322456
2018-07-26 19:38:15tim.goldensetkeywords: + patch
stage: patch review
pull_requests: + pull_request8008
2018-07-26 19:22:54tim.goldencreate