classification
Title: shutil.rmtree(..., ignore_errors=True) doesn't ignore errors from os.close()
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: David.Edelsohn, ZackerySpytz, andrei.avk, rabraham, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2018-11-27 19:51 by rabraham, last changed 2021-07-16 05:13 by andrei.avk.

Pull Requests
URL Status Linked Edit
PR 23766 open ZackerySpytz, 2020-12-14 16:33
Messages (5)
msg330553 - (view) Author: Ronal Abraham (rabraham) Date: 2018-11-27 19:51
These lines throw intermittently for me on AIX when removing a directory on NFS (python version 3.6.6-1), even when "ignore_errors=True":

https://github.com/python/cpython/blob/v3.6.6rc1/Lib/shutil.py#L433
https://github.com/python/cpython/blob/v3.6.6rc1/Lib/shutil.py#L492

Should there be try-except blocks around these lines and calls to onerror(...)?
msg330856 - (view) Author: Ronal Abraham (rabraham) Date: 2018-12-01 17:22
I forgot to mention: the exception raised is an OSError and the errno is 52 (ESTALE on AIX).
msg383038 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-12-15 08:20
I am wondering whether we should silence that error in os.close() unconditionally.

In what circumstances the error is raised? Can it be reproduced on Linux (by monkey-patching os.rmdir)? What can happen in worst case when the error is ignored?
msg397598 - (view) Author: Andrei Kulakov (andrei.avk) * Date: 2021-07-16 05:12
I've looked a bit into this and it seems like ESTALE can be caused in two cases:

 - Under WPAR environment
https://www.ibm.com/docs/en/aix/7.1?topic=aix-wpar-concepts

https://www.ibm.com/support/pages/apar/IV50544
> When using WPAR setup under AIX, with each WPAR configured as an NFS client, it is possible that asynchronous READDIR operations fail with ESTALE error

 - http://www.audentia-gestion.fr/IBM/PDF/basetrf2_pdf.pdf
ESTALE. The process's root or current directory is located in a virtual file system that has been unmounted.


In the first case, the solution given is to run READDIR under a different CID. I've found the definition of CID just 10 minutes ago but now can't find it again :(. Anyway, it seems to be an ID corresponding to the WPAR environment.

In the 2nd case, it's pretty clear that we shouldn't silence it: if you unmount a virtual partition while running rmtree on it, I'm 99.9% sure it was unintended!

In the first case, I don't think Python has any specific support for WPAR? Maybe? If not, we shouldn't do anything, if yes, maybe it can be fixed to handle this.

It sounds like Ronal may have ran into the first case, or something else entirely.
msg397599 - (view) Author: Andrei Kulakov (andrei.avk) * Date: 2021-07-16 05:13
May be related: https://bugs.python.org/issue43666
History
Date User Action Args
2021-07-16 05:13:34andrei.avksetmessages: + msg397599
2021-07-16 05:12:59andrei.avksetnosy: + andrei.avk
messages: + msg397598
2020-12-15 08:20:47serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg383038
2020-12-14 16:33:44ZackerySpytzsetkeywords: + patch
nosy: + ZackerySpytz

pull_requests: + pull_request22622
stage: patch review
2018-12-01 17:22:44rabrahamsetmessages: + msg330856
2018-11-28 16:18:48SilentGhostsetnosy: + David.Edelsohn
2018-11-27 19:51:53rabrahamcreate