classification
Title: stat.S_ISXXX can raise OverflowError for remote file modes
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ammar2, christian.heimes, wiggin15
Priority: normal Keywords:

Created on 2020-02-23 10:08 by wiggin15, last changed 2020-02-29 00:05 by terry.reedy.

Messages (3)
msg362499 - (view) Author: Arnon Yaari (wiggin15) * Date: 2020-02-23 10:08
The C implementation of the "stat" module on Python 3 (_stat) is using the type "mode_t" for file modes, which differs between operating systems. This type can be defined as either "unsigned short" (for example, in macOS, or the definition added specifically for Windows in _stat.c) or "unsigned long" (Linux and other Unix systems such as AIX).
This means that the "stat" module may only work with file modes that come from the same system that Python was compiled for.
It is sometimes desirable to work with file modes on remote systems (for example, when using the "fabric" module to handle remote files - https://github.com/fabric/fabric/blob/1.10/fabric/sftp.py#L42).
With the pure-python "stat" module on Python 2.7, using macros such as "stat.S_ISDIR" with any value used to work (even values that exceed "unsigned short" on macOS, for example) but with the C implementation this can result in an exception on systems with an "unsigned short" mode_t:

    >>> stat.S_ISDIR(0o240755)
    OverflowError: mode out of range

I encountered this exception when trying to "put" files from a macOS system to an AIX system with "fabric" (0o240755 is the st_mode found for "/" on AIX).

For uniform handling of file modes, modes should be handled as unsigned long instead of the system-defined "mode_t".
msg362501 - (view) Author: Ammar Askar (ammar2) * (Python triager) Date: 2020-02-23 11:08
Size of the mode aside, isn't this approach problematic anyway? One platform could have an entirely different way of signalling ISDIR vs another. In this case using the host's S_ISDIR for the remote's mode would result in possibly incorrect values.
msg362521 - (view) Author: Arnon Yaari (wiggin15) * Date: 2020-02-23 16:39
Maybe so, but IMHO it is a lesser concern - the operating systems that use "unsigned long" add bits on the right, and don't change the less significant bits that the S_ISXXX macros check (if I am not mistaken, those bits are POSIX standards).
This issue is a regression from Python 2.7, so I believe it should be addressed.
History
Date User Action Args
2020-02-29 00:05:46terry.reedysetnosy: + christian.heimes
2020-02-23 16:39:06wiggin15setmessages: + msg362521
2020-02-23 11:08:37ammar2setnosy: + ammar2
messages: + msg362501
2020-02-23 10:08:44wiggin15create