Title: OSError when using pathlib.Path.rglob() to list device files
Type: behavior Stage:
Components: IO, Library (Lib), macOS Versions: Python 3.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Victor Domingos, ned.deily, ronaldoussoren, xtreak
Priority: normal Keywords:

Created on 2018-05-24 12:15 by Victor Domingos, last changed 2018-07-19 11:13 by xtreak.

Messages (2)
msg317566 - (view) Author: Victor Domingos (Victor Domingos) * Date: 2018-05-24 12:15
This method fails with an error when it finds a character or block device (like those found in /dev on macOS). However, in the same machine, with the same Python version, the alternative os.walk() performs basically the same job with no errors. I am not sure if that error is the expected behaviour, but I wasn't able to find a clear explanation in the docs. Anyway, it seems to me, as a user, that the os.walk() error-less behaviour is more desirable.

$ python3
Python 3.7.0b4 (v3.7.0b4:eb96c37699, May  2 2018, 04:13:13) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> from pathlib import Path
>>> [f for f in Path(os.path.expanduser('/dev')).rglob("*") if f.is_file()]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in <listcomp>
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/", line 1344, in is_file
    return S_ISREG(self.stat().st_mode)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/", line 1140, in stat
    return self._accessor.stat(self)
OSError: [Errno 9] Bad file descriptor: '/dev/fd/3'
msg317791 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2018-05-27 11:17
This is fairly odd behaviour of macOS, the same error can be seen from a bash session:

$ stat /dev/fd/3
stat: /dev/fd/3: stat: Bad file descriptor

The reason os.walk() works while the Path().is_file doesn't is that os.walk() explicitly guards against OSError exceptions raised by os.stat(). 

Looking at the documentation this could be seen as a bug in macOS, as the manual page for stat(2) doesn't mention EBADF as a valid error for this system call.  

I'm not sure at this point if we should add a workaround for this. An actual patch would be easy enough, "just" add EBADF to the list of ignored errno values in the implementation of is_file (and related method) in
Date User Action Args
2018-07-19 11:13:05xtreaksetnosy: + xtreak
2018-07-11 07:54:21serhiy.storchakasettype: crash -> behavior
2018-05-27 11:17:24ronaldoussorensetmessages: + msg317791
2018-05-24 14:13:23Victor Domingossetnosy: + ronaldoussoren, ned.deily
components: + macOS, IO
2018-05-24 12:15:27Victor Domingoscreate