Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OSError when using pathlib.Path.rglob() to list device files #77816

Closed
VictorDomingos mannequin opened this issue May 24, 2018 · 6 comments
Closed

OSError when using pathlib.Path.rglob() to list device files #77816

VictorDomingos mannequin opened this issue May 24, 2018 · 6 comments
Labels
3.7 (EOL) end of life OS-mac stdlib Python modules in the Lib dir topic-IO type-bug An unexpected behavior, bug, or error

Comments

@VictorDomingos
Copy link
Mannequin

VictorDomingos mannequin commented May 24, 2018

BPO 33635
Nosy @ronaldoussoren, @ned-deily, @zooba, @miss-islington, @Windsooon, @tirkarthi
PRs
  • bpo-33635: Handling Bad file descriptor in Path.is_file and related. #8542
  • [3.7] bpo-33635: Handling Bad file descriptor in Path.is_file and related. (GH-8542) #8960
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2019-03-20.00:52:19.850>
    created_at = <Date 2018-05-24.12:15:27.517>
    labels = ['OS-mac', 'type-bug', 'library', 'expert-IO', '3.7']
    title = 'OSError when using pathlib.Path.rglob() to list device files'
    updated_at = <Date 2019-03-20.00:52:19.849>
    user = 'https://bugs.python.org/VictorDomingos'

    bugs.python.org fields:

    activity = <Date 2019-03-20.00:52:19.849>
    actor = 'cheryl.sabella'
    assignee = 'none'
    closed = True
    closed_date = <Date 2019-03-20.00:52:19.850>
    closer = 'cheryl.sabella'
    components = ['Library (Lib)', 'macOS', 'IO']
    creation = <Date 2018-05-24.12:15:27.517>
    creator = 'Victor Domingos'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 33635
    keywords = ['patch']
    message_count = 6.0
    messages = ['317566', '317791', '321993', '322308', '324199', '324203']
    nosy_count = 7.0
    nosy_names = ['ronaldoussoren', 'ned.deily', 'steve.dower', 'miss-islington', 'Victor Domingos', 'Windson Yang', 'xtreak']
    pr_nums = ['8542', '8960']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue33635'
    versions = ['Python 3.7']

    @VictorDomingos
    Copy link
    Mannequin Author

    VictorDomingos mannequin commented May 24, 2018

    This method fails with an error when it finds a character or block device (like those found in /dev on macOS). However, in the same machine, with the same Python version, the alternative os.walk() performs basically the same job with no errors. I am not sure if that error is the expected behaviour, but I wasn't able to find a clear explanation in the docs. Anyway, it seems to me, as a user, that the os.walk() error-less behaviour is more desirable.

    $ python3
    Python 3.7.0b4 (v3.7.0b4:eb96c37699, May  2 2018, 04:13:13) 
    [Clang 6.0 (clang-600.0.57)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import os
    >>> from pathlib import Path
    >>> [f for f in Path(os.path.expanduser('/dev')).rglob("*") if f.is_file()]
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "<stdin>", line 1, in <listcomp>
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py", line 1344, in is_file
        return S_ISREG(self.stat().st_mode)
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py", line 1140, in stat
        return self._accessor.stat(self)
    OSError: [Errno 9] Bad file descriptor: '/dev/fd/3'
    >>> 
    

    @VictorDomingos VictorDomingos mannequin added type-crash A hard crash of the interpreter, possibly with a core dump 3.7 (EOL) end of life stdlib Python modules in the Lib dir OS-mac topic-IO labels May 24, 2018
    @ronaldoussoren
    Copy link
    Contributor

    This is fairly odd behaviour of macOS, the same error can be seen from a bash session:

    $ stat /dev/fd/3
    stat: /dev/fd/3: stat: Bad file descriptor

    The reason os.walk() works while the Path().is_file doesn't is that os.walk() explicitly guards against OSError exceptions raised by os.stat().

    Looking at the documentation this could be seen as a bug in macOS, as the manual page for stat(2) doesn't mention EBADF as a valid error for this system call.

    I'm not sure at this point if we should add a workaround for this. An actual patch would be easy enough, "just" add EBADF to the list of ignored errno values in the implementation of is_file (and related method) in pathlib.py

    @serhiy-storchaka serhiy-storchaka added type-bug An unexpected behavior, bug, or error and removed type-crash A hard crash of the interpreter, possibly with a core dump labels Jul 11, 2018
    @Windsooon
    Copy link
    Mannequin

    Windsooon mannequin commented Jul 20, 2018

    I tried to fix this issue and I found this should be related to os.stat() in os.py. But I can't find the os.stat() API in the file, any ideas?

    @ronaldoussoren
    Copy link
    Contributor

    Os.stat is defined in C code, in particular in Modules/posixmodule.c.

    Op 20 jul. 2018 om 10:44 heeft Windson Yang <report@bugs.python.org> het volgende geschreven:

    Windson Yang <wiwindson@gmail.com> added the comment:

    I tried to fix this issue and I found this should be related to os.stat() in os.py. But I can't find the os.stat() API in the file, any ideas?

    ----------
    nosy: +Windson Yang


    Python tracker <report@bugs.python.org>
    <https://bugs.python.org/issue33635\>


    @zooba
    Copy link
    Member

    zooba commented Aug 27, 2018

    New changeset 216b745 by Steve Dower (Przemysław Spodymek) in branch 'master':
    bpo-33635: Handling Bad file descriptor in Path.is_file and related. (GH-8542)
    216b745

    @miss-islington
    Copy link
    Contributor

    New changeset 2cf3316 by Miss Islington (bot) in branch '3.7':
    bpo-33635: Handling Bad file descriptor in Path.is_file and related. (GH-8542)
    2cf3316

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life OS-mac stdlib Python modules in the Lib dir topic-IO type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    5 participants