classification
Title: pathlib mkdir throws FileExistsError when not supposed to
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Adam Dunlap, xtreak
Priority: normal Keywords:

Created on 2018-11-08 20:08 by Adam Dunlap, last changed 2018-11-10 00:45 by xtreak.

Files
File name Uploaded Description Edit
pymkir.py Adam Dunlap, 2018-11-10 00:20
Messages (4)
msg329484 - (view) Author: Adam Dunlap (Adam Dunlap) Date: 2018-11-08 20:08
I have 2 processes (one C, one python) that are started at the same time and both need a certain directory tree to exist to continue, so both processes create the directory tree while ignoring errors about it already existing.

In the python process, I'm using pathlib's mkdir function (https://docs.python.org/3/library/pathlib.html#pathlib.Path.mkdir) with parents=True, exist_ok=True, but this is sometimes throwing a FileExistsError. I don't expect this because the documentation says "If exist_ok is true, FileExistsError exceptions will be ignored (same behavior as the POSIX mkdir -p command), but only if the last path component is not an existing non-directory file." The last component is never a non-directory file.

I believe what is happening is that mkdir realizes that the parent doesn't exist, so it recursively tries to make the parent directory. However, when it recurses, it uses the default exists_ok value of False. Before the recursive call can make the parent directory, the other process makes the directory. This causes the inner call to throw a FileExistsError.

I believe the correct thing to do is to always pass True for exists_ok in the recursive call.
msg329495 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2018-11-09 03:45
Thanks for the report. There was a related issue : issue29694 . This was fixed in 3.7 and was backported with https://github.com/python/cpython/commit/d7abeb7024b9755c291c29bdc8c4494246e975ad (3.5.4 and above) . Can you please specify the full version of Python 3.5 you are using and if the linked issue is related ? If possible please attach a reproducer so  that this can be tested on other versions.
msg329576 - (view) Author: Adam Dunlap (Adam Dunlap) Date: 2018-11-10 00:20
Thank your for your response. I'm running python 3.5.2. The linked issue is indeed a duplicate of this one.

To reproduce, you can run two instances of the attached script at the same time, i.e. python3 pymkdir.py & python3 pymkdir.py. It is a race condition so won't happen every time, but with the number of directories created it should happen pretty frequently (remove the directories between tests with rm -rf 1)

I'm running Ubuntu 16.04.5 LTS. I don't know if you know anything about Ubuntu's release process, but it would be nice if the linked fix could be backported to work on that platform.
msg329578 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2018-11-10 00:45
> Thank your for your response. I'm running python 3.5.2. The linked issue is indeed a duplicate of this one.

Thanks for the confirmation and script.

> I'm running Ubuntu 16.04.5 LTS. I don't know if you know anything about Ubuntu's release process, but it would be nice if the linked fix could be backported to work on that platform.

I have limited knowledge of Ubuntu release process and from what I can see from https://packages.ubuntu.com/xenial/python3.5 3.5.2 is the latest release for python 3.5 ans the backport is at 3.4 and above. Core python team doesn't maintain the releases for Ubuntu and it's upto the relevant maintainer to package this. You can try a release from https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa.
History
Date User Action Args
2018-11-10 00:45:24xtreaksetmessages: + msg329578
2018-11-10 00:20:10Adam Dunlapsetfiles: + pymkir.py

messages: + msg329576
2018-11-09 03:46:00xtreaksetnosy: + xtreak
messages: + msg329495
2018-11-08 20:08:28Adam Dunlapcreate