classification
Title: OSError when using pathlib.Path.rglob() to list device files
Type: behavior Stage: patch review
Components: IO, Library (Lib), macOS Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Victor Domingos, Windson Yang, miss-islington, ned.deily, ronaldoussoren, steve.dower, xtreak
Priority: normal Keywords: patch

Created on 2018-05-24 12:15 by Victor Domingos, last changed 2018-08-27 22:37 by miss-islington.

Pull Requests
URL Status Linked Edit
PR 8542 merged python-dev, 2018-07-29 09:07
PR 8960 merged miss-islington, 2018-08-27 21:33
Messages (6)
msg317566 - (view) Author: Victor Domingos (Victor Domingos) * Date: 2018-05-24 12:15
This method fails with an error when it finds a character or block device (like those found in /dev on macOS). However, in the same machine, with the same Python version, the alternative os.walk() performs basically the same job with no errors. I am not sure if that error is the expected behaviour, but I wasn't able to find a clear explanation in the docs. Anyway, it seems to me, as a user, that the os.walk() error-less behaviour is more desirable.

```
$ python3
Python 3.7.0b4 (v3.7.0b4:eb96c37699, May  2 2018, 04:13:13) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> from pathlib import Path
>>> [f for f in Path(os.path.expanduser('/dev')).rglob("*") if f.is_file()]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in <listcomp>
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py", line 1344, in is_file
    return S_ISREG(self.stat().st_mode)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py", line 1140, in stat
    return self._accessor.stat(self)
OSError: [Errno 9] Bad file descriptor: '/dev/fd/3'
>>> 
```
msg317791 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2018-05-27 11:17
This is fairly odd behaviour of macOS, the same error can be seen from a bash session:

$ stat /dev/fd/3
stat: /dev/fd/3: stat: Bad file descriptor

The reason os.walk() works while the Path().is_file doesn't is that os.walk() explicitly guards against OSError exceptions raised by os.stat(). 

Looking at the documentation this could be seen as a bug in macOS, as the manual page for stat(2) doesn't mention EBADF as a valid error for this system call.  

I'm not sure at this point if we should add a workaround for this. An actual patch would be easy enough, "just" add EBADF to the list of ignored errno values in the implementation of is_file (and related method) in pathlib.py
msg321993 - (view) Author: Windson Yang (Windson Yang) * Date: 2018-07-20 09:44
I tried to fix this issue and I found this should be related to os.stat() in os.py. But I can't find the os.stat() API in the file, any ideas?
msg322308 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2018-07-24 15:08
Os.stat is defined in C code, in particular in Modules/posixmodule.c. 

Op 20 jul. 2018 om 10:44 heeft Windson Yang <report@bugs.python.org> het volgende geschreven:

> 
> Windson Yang <wiwindson@gmail.com> added the comment:
> 
> I tried to fix this issue and I found this should be related to os.stat() in os.py. But I can't find the os.stat() API in the file, any ideas?
> 
> ----------
> nosy: +Windson Yang
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue33635>
> _______________________________________
msg324199 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-08-27 21:33
New changeset 216b745eafa7cd4a683a8405dcfbd7f5567f504c by Steve Dower (Przemysław Spodymek) in branch 'master':
bpo-33635: Handling Bad file descriptor in Path.is_file and related. (GH-8542)
https://github.com/python/cpython/commit/216b745eafa7cd4a683a8405dcfbd7f5567f504c
msg324203 - (view) Author: miss-islington (miss-islington) Date: 2018-08-27 22:37
New changeset 2cf33161f7cb4ca726c74fcfa2d6db27073953d8 by Miss Islington (bot) in branch '3.7':
bpo-33635: Handling Bad file descriptor in Path.is_file and related. (GH-8542)
https://github.com/python/cpython/commit/2cf33161f7cb4ca726c74fcfa2d6db27073953d8
History
Date User Action Args
2018-08-27 22:37:22miss-islingtonsetnosy: + miss-islington
messages: + msg324203
2018-08-27 21:33:58miss-islingtonsetpull_requests: + pull_request8435
2018-08-27 21:33:58steve.dowersetnosy: + steve.dower
messages: + msg324199
2018-07-29 09:07:30python-devsetkeywords: + patch
stage: patch review
pull_requests: + pull_request8059
2018-07-24 15:08:49ronaldoussorensetmessages: + msg322308
2018-07-20 09:44:15Windson Yangsetnosy: + Windson Yang
messages: + msg321993
2018-07-19 11:13:05xtreaksetnosy: + xtreak
2018-07-11 07:54:21serhiy.storchakasettype: crash -> behavior
2018-05-27 11:17:24ronaldoussorensetmessages: + msg317791
2018-05-24 14:13:23Victor Domingossetnosy: + ronaldoussoren, ned.deily
components: + macOS, IO
2018-05-24 12:15:27Victor Domingoscreate