classification
Title: Convert test_mmap to use tempfile
Type: Stage: patch review
Components: Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: tim.golden Nosy List: cheryl.sabella, eryksun, tim.golden, tjguk
Priority: normal Keywords: patch

Created on 2018-07-26 19:22 by tim.golden, last changed 2020-05-29 10:43 by tjguk.

Pull Requests
URL Status Linked Edit
PR 8486 closed tim.golden, 2018-07-26 19:38
Messages (7)
msg322444 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2018-07-26 19:22
test_mmap currently uses the test.support.TESTFN functionality which creates a temporary file local to the test directory named around the pid. 

This can give rise to race conditions where tests are competing with each other to delete and recreate the file.
msg322456 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2018-07-27 02:25
> tests are competing with each other to delete and recreate the file

When is this an issue? Each parallel test process should have its own working directory under "{TEMPDIR}/test_python_{pid}".
msg322464 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2018-07-27 05:39
To clarify, TEMPDIR in this case is from the following code in Lib/test/libregrtest/main.py:

    # When tests are run from the Python build directory, it is best practice
    # to keep the test files in a subfolder.  This eases the cleanup of leftover
    # files using the "make distclean" command.
    if sysconfig.is_python_build():
        TEMPDIR = sysconfig.get_config_var('abs_builddir')
        if TEMPDIR is None:
            # bpo-30284: On Windows, only srcdir is available. Using abs_builddir
            # mostly matters on UNIX when building Python out of the source tree,
            # especially when the source tree is read only.
            TEMPDIR = sysconfig.get_config_var('srcdir')
        TEMPDIR = os.path.join(TEMPDIR, 'build')
    else:
        TEMPDIR = tempfile.gettempdir()
    TEMPDIR = os.path.abspath(TEMPDIR)

Then in class Regrtest we have:

    def main(self, tests=None, **kwargs):
        global TEMPDIR

        if sysconfig.is_python_build():
            try:
                os.mkdir(TEMPDIR)
            except FileExistsError:
                pass

        # Define a writable temp dir that will be used as cwd while running
        # the tests. The name of the dir includes the pid to allow parallel
        # testing (see the -j option).
        test_cwd = 'test_python_{}'.format(os.getpid())
        test_cwd = os.path.join(TEMPDIR, test_cwd)

        # Run the tests in a context manager that temporarily changes the CWD to a
        # temporary and writable directory.  If it's not possible to create or
        # change the CWD, the original CWD will be used.  The original CWD is
        # available from support.SAVEDCWD.
        with support.temp_cwd(test_cwd, quiet=True):
            self._main(tests, kwargs)
msg322471 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2018-07-27 07:56
Thanks for the information, eryksun.

For the moment, I can only say with a fair degree of certainty that using the tempfile functions as I have in test_bz2 & test_mmap appears to solve the issue which is repeatably if intermittently present without that change.

I'll be trying to narrow the issue down before making any further changes
msg370288 - (view) Author: Cheryl Sabella (cheryl.sabella) * (Python committer) Date: 2020-05-29 10:15
@tim.golden, I closed this pull request as it was from an unknown repository and new pull request would need to be created with the changes in order to move it forward.
msg370290 - (view) Author: TJG (tjguk) Date: 2020-05-29 10:18
Thanks, Cheryl. It was getting pretty stale. If I think it's worthwhile 
I'll resurrect it.

I suspect what happened was that my cpython fork got borked somehow 
between then and now. I probably ditched the repo and re-forked. That's 
for doing the housekeeping work here.
msg370295 - (view) Author: TJG (tjguk) Date: 2020-05-29 10:43
That last should clearly have said:

Thanks for doing the housekeeping work here

:)
History
Date User Action Args
2020-05-29 10:43:18tjguksetmessages: + msg370295
2020-05-29 10:18:58tjguksetnosy: + tjguk
messages: + msg370290
2020-05-29 10:15:11cheryl.sabellasetnosy: + cheryl.sabella
messages: + msg370288
2018-07-27 07:56:55tim.goldensetmessages: + msg322471
2018-07-27 05:39:18eryksunsetmessages: + msg322464
2018-07-27 02:25:07eryksunsetnosy: + eryksun
messages: + msg322456
2018-07-26 19:38:15tim.goldensetkeywords: + patch
stage: patch review
pull_requests: + pull_request8008
2018-07-26 19:22:54tim.goldencreate