tempfile seems to treat a file as a directory when processing an exception in the onerror() #87319

mehwhatever0 · 2021-02-07T09:52:44Z

BPO	43153
Nosy	@serhiy-storchaka, @Fidget-Spinner, @akulakov, @mehwhatever0
PRs	bpo-43153: Don't mask `PermissionError` with `NotADirectoryError` during tempdirectory cleanup #29940

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-02-07.09:52:44.316>
labels = ['type-bug', 'library', '3.9', '3.10', '3.11']
title = 'tempfile seems to treat a file as a directory when processing an exception in the onerror()'
updated_at = <Date 2021-12-06.14:13:37.485>
user = 'https://github.com/mehwhatever0'

bugs.python.org fields:

activity = <Date 2021-12-06.14:13:37.485>
actor = 'kj'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-02-07.09:52:44.316>
creator = 'mehwhatever0'
dependencies = []
files = []
hgrepos = []
issue_num = 43153
keywords = ['patch']
message_count = 4.0
messages = ['386587', '397118', '407376', '407435']
nosy_count = 4.0
nosy_names = ['serhiy.storchaka', 'kj', 'andrei.avk', 'mehwhatever0']
pr_nums = ['29940']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43153'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

The text was updated successfully, but these errors were encountered:

mehwhatever0 · 2021-02-07T09:52:44Z

>>> import sys, tempfile, os
>>> sys.version
'3.9.1 (tags/v3.9.1:1e5d33e, Dec  7 2020, 17:08:21) [MSC v.1927 64 bit (AMD64)]'
>>> folder = tempfile.TemporaryDirectory()
>>> file = open(os.path.join(folder.name, "example.txt"), 'wb')
>>> folder.cleanup()
Traceback (most recent call last):
  File "C:\Users\user0\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 616, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 32] Процесс не может получить доступ к файлу, так как этот файл занят другим процессом: 'C:\\Users\\user0\\AppData\\Local\\Temp\\tmpin8xeb9b\\example.txt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\user0\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 801, in onerror
    _os.unlink(path)
PermissionError: [WinError 32] Процесс не может получить доступ к файлу, так как этот файл занят другим процессом: 'C:\\Users\\user0\\AppData\\Local\\Temp\\tmpin8xeb9b\\example.txt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\user0\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 830, in cleanup
    self._rmtree(self.name)
  File "C:\Users\user0\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 812, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\Users\user0\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 740, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\user0\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 618, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Users\user0\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 804, in onerror
    cls._rmtree(path)
  File "C:\Users\user0\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 812, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\Users\user0\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 740, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\user0\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 599, in _rmtree_unsafe
    onerror(os.scandir, path, sys.exc_info())
  File "C:\Users\user0\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 596, in _rmtree_unsafe
    with os.scandir(path) as scandir_it:
NotADirectoryError: [WinError 267] Неверно задано имя папки: 'C:\\Users\\user0\\AppData\\Local\\Temp\\tmpin8xeb9b\\example.txt'

akulakov · 2021-07-07T21:17:41Z

This is a duplicate of https://bugs.python.org/issue43219 .

akulakov · 2021-11-30T14:36:02Z

My last comment was wrong, the issue I linked is unrelated.

I think the actual issue here is the code here:

cpython/Lib/tempfile.py

Line 826 in 4b97d97

except (IsADirectoryError, PermissionError):

If path is a file that causes a permission error, rmtree is called and causes a NotADirectory error.

This can be confusing when debugging and can lead to a bug if user's code tries to catch a PermissionError it expects but instead gets NotADirectory error.

A solution is probably to check if path is a file and error is PermissionError and if ignore_errors=False, re-raise it instead of calling rmtree(). If ignore_errors=True, return instead of calling rmtree().

I don't have windows so can't test the OP code with this approach.

Serhiy: adding you since you wrote def onerror() in tempfile module, do you think that is the cause of the issue and the right solution for it?

serhiy-storchaka · 2021-12-01T09:59:22Z

I agree with Andrei. I'll test this approach on Windows.

bekieark · 2023-01-26T23:34:21Z

Hi Team, Do you have any news about an eventual patch for this? Was it solved in later Python versions?

I am running Python 3.9.1 (tags/v3.9.1:1e5d33e, Dec 7 2020, 17:08:21) [MSC v.1927 64 bit (AMD64)], on a Windows 10 64-bit machine, with 8 GB of RAM and an SSD drive. No filesystem compression. I am handling large data objects, coming from a scientific package. I am simply trying to serialize them on disk, for future use.

Smaller files serialize correctly using pickle, marshal, joblib and dill libraries. However larger files (from approx. 200 MB up, in terms of their size on the disk) cause one and the same Error all the time, as listed below. I will try also HDF (hdf5) and serpy serialization, but at this point I am sure it is simply related to large file size.

The error is identical to the one observed by You 1-2 years ago here. IMHO system tries to use temporary data in local Temp folder and something goes wrong. E.g. it attempts to unlink a file, it checks folder - yet he thinks the filename is a folder name. Or something similar.

It is quite important for me, related to my day job... Is there something I could do myself, or test? I plan to dig in the <shutil.py> library, but maybe You already made some tests... $\textcolor{green}{\text{T}}\textcolor{lime}{\text{h}}\textcolor{Orange}{\text{a}}\textcolor{BurntOrange}{\text{n}}\textcolor{red}{\text{k}}\textcolor{pink}{\text{s}}$!

>>> joblib.dump(oLarge, '20230126T2243CET_o_large.jlib')
Traceback (most recent call last):
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 616, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 32] Proces nie mo?e uzyska? dost?pu do pliku, poniewa? jest on u?ywany przez inny proces: 'C:\\Users\\myuser\\AppData\\Local\\Temp\\tmpwteculmb\\tmp.ann'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 801, in onerror
    _os.unlink(path)
PermissionError: [WinError 32] Proces nie moze uzyskac dostepu do pliku, poniewaz jest on uzywany przez inny proces: 'C:\\Users\\myuser\\AppData\\Local\\Temp\\tmpwteculmb\\tmp.ann'

<COMMENT> The above is in Polish, which is my Windows 64-bit locale. It is a generic error, related to a "Process that cannot access file, as it is used by another process". The issue is probably the filename instead of directory name and the PermissionError / NotADirectory exception </END OF COMMENT>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<pyshell#78>", line 1, in <module>
    joblib.dump(oLarge, '20230126T2243CET_o_large.jlib')
  File "C:\Users\myuser\workspace-python\Environs\mylib\lib\site-packages\joblib\numpy_pickle.py", line 553, in dump
    NumpyPickler(f, protocol=protocol).dump(value)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 487, in dump
    self.save(obj)
  File "C:\Users\myuser\workspace-python\Environs\mylib\lib\site-packages\joblib\numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 717, in save_reduce
    save(state)
  File "C:\Users\myuser\workspace-python\Environs\mylib\lib\site-packages\joblib\numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 901, in save_tuple
    save(element)
  File "C:\Users\myuser\workspace-python\Environs\mylib\lib\site-packages\joblib\numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 717, in save_reduce
    save(state)
  File "C:\Users\myuser\workspace-python\Environs\mylib\lib\site-packages\joblib\numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 997, in _batch_setitems
    save(v)
  File "C:\Users\myuser\workspace-python\Environs\mylib\lib\site-packages\joblib\numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 578, in save
    rv = reduce(self.proto)
  File "C:\Users\myuser\workspace-python\Environs\mylib\lib\site-packages\openTSNE\nearest_neighbors.py", line 353, in __getstate__
    b64_index = base64.b64encode(f.read())
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 826, in __exit__
    self.cleanup()
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 830, in cleanup
    self._rmtree(self.name)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 812, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 740, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 618, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 804, in onerror
    cls._rmtree(path)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 812, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 740, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 599, in _rmtree_unsafe
    onerror(os.scandir, path, sys.exc_info())
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 596, in _rmtree_unsafe
    with os.scandir(path) as scandir_it:
NotADirectoryError: [WinError 267] Nazwa katalogu jest nieprawid?owa: 'C:\\Users\\myuser\\AppData\\Local\\Temp\\tmpwteculmb\\tmp.ann'

<COMMENT> This one is in Polish again. It says "Folder name is not correct" </END OF COMMENT>

>>>

davetapley · 2023-08-17T20:42:06Z

I just ran in to working a separate logging issue.
Windows, Python 3.11.4. MRE here.

zenquiorra · 2023-10-12T15:50:18Z

Is there any workaround for this issue?

serhiy-storchaka · 2023-12-05T14:32:23Z

Fixed in 3.11-3.13.

…unction()

…n() (GH-112791)

…unction() (pythonGH-112791) (cherry picked from commit ba18893) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

…junction() (GH-112791) (GH-112845) (cherry picked from commit ba18893) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

…unction() (pythonGH-112791)

mehwhatever0 mannequin added 3.9 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Feb 7, 2021

akulakov added 3.10 only security fixes 3.11 only security fixes labels Jul 7, 2021

ezio-melotti transferred this issue from another repository Apr 10, 2022

davetapley mentioned this issue Aug 17, 2023

shutil.rmtree fails with NotADirectoryError #107408

Closed

serhiy-storchaka mentioned this issue Dec 5, 2023

bpo-43153: Don't mask PermissionError with NotADirectoryError during tempdirectory cleanup #29940

Merged

serhiy-storchaka closed this as completed Dec 5, 2023

serhiy-storchaka added 3.12 bugs and security fixes 3.13 new features, bugs and security fixes and removed 3.10 only security fixes 3.9 only security fixes labels Dec 5, 2023

serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue Dec 6, 2023

pythongh-87319: Simplify TemporaryDirectory cleanup using os.path.isj…

e21e11b

…unction()

bedevere-app bot mentioned this issue Dec 6, 2023

gh-87319: Simplify TemporaryDirectory cleanup using os.path.isjunction() #112791

Merged

serhiy-storchaka added a commit that referenced this issue Dec 7, 2023

gh-87319: Simplify TemporaryDirectory cleanup using os.path.isjunctio…

ba18893

…n() (GH-112791)

bedevere-app bot mentioned this issue Dec 7, 2023

[3.12] gh-87319: Simplify TemporaryDirectory cleanup using os.path.isjunction() (GH-112791) #112845

Merged

serhiy-storchaka added a commit that referenced this issue Dec 7, 2023

[3.12] gh-87319: Simplify TemporaryDirectory cleanup using os.path.is…

418adf6

…junction() (GH-112791) (GH-112845) (cherry picked from commit ba18893) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

aisk pushed a commit to aisk/cpython that referenced this issue Feb 11, 2024

pythongh-87319: Simplify TemporaryDirectory cleanup using os.path.isj…

a8ef457

…unction() (pythonGH-112791)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tempfile seems to treat a file as a directory when processing an exception in the onerror() #87319

tempfile seems to treat a file as a directory when processing an exception in the onerror() #87319

mehwhatever0 mannequin commented Feb 7, 2021 •

edited by bedevere-app bot

mehwhatever0 mannequin commented Feb 7, 2021

akulakov commented Jul 7, 2021

akulakov commented Nov 30, 2021

serhiy-storchaka commented Dec 1, 2021

bekieark commented Jan 26, 2023

davetapley commented Aug 17, 2023

zenquiorra commented Oct 12, 2023

serhiy-storchaka commented Dec 5, 2023

tempfile seems to treat a file as a directory when processing an exception in the onerror() #87319

tempfile seems to treat a file as a directory when processing an exception in the onerror() #87319

Comments

mehwhatever0 mannequin commented Feb 7, 2021 • edited by bedevere-app bot

Linked PRs

mehwhatever0 mannequin commented Feb 7, 2021

akulakov commented Jul 7, 2021

akulakov commented Nov 30, 2021

serhiy-storchaka commented Dec 1, 2021

bekieark commented Jan 26, 2023

davetapley commented Aug 17, 2023

zenquiorra commented Oct 12, 2023

serhiy-storchaka commented Dec 5, 2023

mehwhatever0 mannequin commented Feb 7, 2021 •

edited by bedevere-app bot