New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
race condition in pathlib mkdir with flags parents=True #73880
Comments
When pathlib mkdir is called with parents=True and some parent doesn't exists it recursively calls self.parent.mkdir(parents=True) after catching OSError. However after catching of OSError and before call to self.parent.mkdir(parents=True) somebody else can create parent dir, which will lead to FileExistsError exception. |
A different bug in the same code: if someone creates the directory itself between the two calls to Attached is a patch that fixes both cases. |
Can you provide tests? |
It's a mess to write a test, because the exact semantics of .mkdir() are not defined as far as I can tell. This patch is a best-effort attempt at making .mkdir() work in the presence of common parallel filesystem changes, that is, other processes that would create the same directories at the same time. This patch is by no means an attempt at being a complete solution for similar problems. The exact semantics have probably never been discussed at all. For example, what should occur if a parent directory is removed just after .mkdir() created it? I'm not suggesting to discuss these issues now, but to simply leave them open. I'm trying instead to explain why writing a test is a mess (more than "just" creating another thread and creating/removing directories very fast while the main thread calls .mkdir()), because we have no exact notion of what should work and what shouldn't. |
You can monkey-patch os.mkdir and pathlib._NormalAccessor.mkdir. Without tests we can't be sure that the patch fixes the issue and that the bug will be not reintroduced by future changes. Tests are required for this change. |
Changes including a test. The test should check all combinations of "concurrent" creation of the directory. It hacks around at pathlib._normal_accessor.mkdir (patching "os.mkdir" has no effect, as the built-in function was already extracted and stored inside pathlib._normal_accessor). |
I had a problem that can be solved with the presented change. The code at the end executed with Maybe this helps to write a test and shows a usecase where this fix is necessary. P.S.: I was not able to produce the error with multiprosessing. from mpi4py import MPI
from pathlib import Path
import tempfile
comm = MPI.COMM_WORLD
rank = comm.rank
size = comm.size
with tempfile.TemporaryDirectory() as tmp_dir:
tmp_dir = comm.bcast(tmp_dir, root=0)
p = Path(tmp_dir) / 'a' / 'b'
comm.barrier()
p.mkdir(parents=True, exist_ok=True) |
Maybe we should review pathlib.py for this kind of issues and first apply the fixes and new tests inside PyPy. That sounds like a better way to get things done for these rare issues, where CPython is understandably reluctant to do much changes. Note that the PyPy version of the stdlib already contains fixes that have not been merged back to CPython (or only very slowly), though so far they are the kind of issues that trigger more often on PyPy than on CPython, like GC issues. |
Update: a review didn't show any other similar problems (pathlib.py is a thin layer after all). Applied the fix and test (x2.diff) inside PyPy. |
(I fixed the problem with my CLA check. Now https://cpython-devguide.readthedocs.io/pullrequest.html#licensing says "you can ask for the CLA check to be run again" but doesn't tell how to do that, so as far as I can tell, I have to ask e.g. here.) |
I merged Armin's PR and backported tp 3.5 and 3.6. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: