New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
os.makedirs with exist_ok=True raises PermissionError on Windows 7^ #69769
Comments
Since Windows 7 (or even Vista), Windows gives permission error(5, ERROR_ACCESS_DENIED Here is an example session (Windows 7, admin):
d:\>IF EXIST . echo True
True
d:\>mkdir .
Access is denied.
d:\>mkdir dir
d:\>cd dir
d:\dir>mkdir .
A subdirectory or file . already exists.
d:\dir>cd ..
d:\>
d:\>py -3
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:43:06) [MSC v.1600 32 bit (In
tel)] on win32
>>> import os
>>> os.path.isdir('.')
True
>>> os.makedirs('.', exist_ok=True)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python34\lib\os.py", line 237, in makedirs
mkdir(name, mode)
PermissionError: [WinError 5] ...
>>> try:
... os.mkdir('.')
... except OSError as e:
... print(e.errno)
...
13 This means that if you want to write portable code, you still need to write like in Python 2: The actual problem is in this line (Lib/os.py#l243): Due the reasons described above, makedirs shouldn't rely on e.errno, so the right code will be: I think the issue is pretty serious to be backported. |
Probably better solution:
|
If you are trying to create a directory named '.' your code will not do anything useful, you might as well skip the call. What's the use case? That said, the fix looks reasonable. |
Of course in examples I create '.' by hand, but in real code such things are def download_file(url, save_as):
...
# Before saving, you must ensure that path exists:
os.makedirs(os.path.dirname(save_as), exist_ok=True)
... os.path.abspath is not a solution, because, as it was mentioned, mkdir with Anyway, skipping the calls must be a job of exist_ok=True, otherwise it has By the way, do not pay attention to my second message(msg254341): as I said I don't know why it was done that way, because if you try "timeit" So the best solution is the most straightforward - to replace try-except block with: |
I meant x1.3 times faster on Linux :) |
Of course, exist_ok must be taken into account: |
Daniel: your latest suggestions look like they introduce a race condition. What happens if another thread or process, perhaps also calling makedirs(), creates the directory just after isdir() says it doesn’t exist? Similar to bpo-1608579. Perhaps the existing code comment needs to clarify that the exception handling is for a real race condition, not just an excuse to “be happy” :) |
Maybe the solution is to leave OSError catching after the conditional if not (exist_ok and path.isdir(name)):
try:
mkdir(name, mode)
except OSError as e:
if not exist_ok or e.errno != errno.EEXIST or not path.isdir(name):
raise This should solve the problem. It also gives more or less guarantee |
Yes that looks like an improvement, though I wonder what’s wrong with your original proposal (performance maybe?). In any case, it definitely needs a comment explaining the first isdir() avoids competing failures that mask EEXIST, and the exception handling avoids the race to create the directory. A test case for the test suite would also be good. I understand it should be easy to do for Windows, just make a directory with an absolute path including a drive root like d:\. |
You mean msg254341? As I mentioned recently, it still will raise an exception Of course there is no practical sense to continue when, for example, FS is By the way, why 3.2 and 3.3 were removed from the list? exist_ok was introduced in 3.2. |
I don’t think we patch 3.2 or 3.3 any more unless it is a security concern. That is why I removed them. I understand 3.4 is due for its last non-security release in a couple weeks. I was actually referring to your original suggestion in <https://bugs.python.org/issue25583#msg254339\>, dressed up below: except OSError:
# Cannot rely on checking for EEXIST, since the operating system could give priority to other errors like EACCES or EROFS
if not (exist_ok and path.isdir(name)):
raise There may be practical reasons to continue if a parent directory exists on a read-only FS. Some OSes can mount writable FSes inside read-only FSes. See <https://marc.info/?l=coreutils-bug&m=124770585425870&w=2\> involving Cygwin, and <https://marc.info/?l=linux-kernel&m=120998905229849&w=2\> involving a Linux regression. Anyway, I think I am happy with either your last fix or the first, with an appropriate comment, and hopefully also a test case. |
Yes, it's probably a better solution. If had been more careful, I wouldn't have scribbled so much text here :) . But is there any sense in adding Windows-specific test with EACCES since the problem with errno may affect other platforms as well (at least in theory)? It's not a Windows-only issue. |
It is good to add a regression test for any bug if it’s not too hard. Yes this is not a Windows-only issue, but I understand it is much simpler to produce with Windows. Otherwise you need a special file system setup and a more obscure OS. Please review my patch. It would be good to confirm that the test fails on Windows if the fix is not applied. I have only tested this on Linux, and indirectly via Wine. |
New patch with simpler test case and revised NEWS entry |
Looks good to me. |
New changeset 05d6ddf2b7c2 by Martin Panter in branch '3.4': New changeset 515f76bf1254 by Martin Panter in branch '3.5': New changeset f0ad5067879b by Martin Panter in branch 'default': New changeset 6ec093f78266 by Martin Panter in branch 'default': |
None of the Windows buildbots are failing my particular test. There are other failures, but they look unrelated, so I am calling this fixed. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: