classification
Title: Python fails to read a script whose path is `/dev/fd/X`
Type: Stage:
Components: Interpreter Core, macOS Versions: Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ciprian.craciun, izbyshev, ned.deily, ronaldoussoren
Priority: normal Keywords:

Created on 2021-01-29 22:32 by ciprian.craciun, last changed 2021-01-30 12:21 by izbyshev.

Messages (3)
msg385955 - (view) Author: Ciprian Dorin Craciun (ciprian.craciun) Date: 2021-01-29 22:32
Sometimes (especially from wrapper scripts) one would like to call Python with a script that should be read from a file-descriptor, like `python3 /dev/fd/9`;  most likely the backing is a file in `TMPDIR` (perhaps unlinked, like `bash`'s "here documents").

However on OSX (I've tried it on `10.15.7` and `10.13.6`), for some reason if that script is too "large" (more than a couple of characters) it just bails out, even if that file-descriptor is in fact backed by a file.

For example:
~~~~
echo 'print("1234567890")' >/tmp/x.py && python3 /dev/fd/9 9</tmp/x.py
# correctly prints 1234567890

echo 'print("12345678901234567890")' >/tmp/x.py && python3 /dev/fd/9 9</tmp/x.py
# prints nothing, no error, no exit code, etc.

# alternative variant 1, with `bash` "here-documents"
python3 /dev/fd/9 9<<<'print("12345678901234567890")'
# still prints nothing.

# alternative variant 2, that uses `/dev/stdin` instead of `/dev/fd/N`
python3 /dev/stdin <<<'print("12345678901234567890")'
# still prints nothing.

# alternative variant 3, that uses `open` and `exec`
python3 -c 'exec(open("/dev/fd/9").read())' 9<<<'print("12345678901234567890")'
# correctly prints 12345678901234567890
~~~~

The file `/tmp/x.py` is just a simple script that prints that token.

I've tried both Python 3.9.1, 3.8.2 and even 2.7.18 and 2.7.16, all with the same results.  (This makes me think it's actually an OSX issue?)

On Linux this works flawlesly.  Furthermore if one uses something else like `cat`, `bash` or anything else it works.  Thus it is something related with Python on OSX.

Also as seen from the examples, this is not a "shell issue" or something similar;  it just seems to hit a corner case when the script path is `/dev/fd/...` or `/dev/stdin`
msg385978 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2021-01-30 10:09
I can reproduce the issue on macOS 11.1. 

As you write:
- running /dev/fd/X as a script fails silently if it refers to an smallish file
- reading /dev/fd/X referring to the same smallish files works fine (the ``open('/dev/fd/9').read()`` scenario. 

If I read the code in Modules/main.c correctly the main difference between the two scenario's is that the first scenario using the C stdio library to read the file (in pymain_run_file_obj), and the latter uses the normal Python io stack.

Reading /dev/fd/9 works fine when using either stdio or open/read in C code. 

I have not yet tried to untangle the layers of C code from pymain_run_file_obj to actually reading the script, there might be something there that sheds light on what's going on here.

I'm not yet willing to claim this is an OS bug as I've failed to reproduce this outside of Python.
msg385982 - (view) Author: Alexey Izbyshev (izbyshev) * (Python triager) Date: 2021-01-30 12:21
I would suggest to start digging from the following piece of code in `maybe_pyc_file()` (Python/pythonrun.c):

     int ispyc = 0;
     if (ftell(fp) == 0) {
         if (fread(buf, 1, 2, fp) == 2 &&
             ((unsigned int)buf[1]<<8 | buf[0]) == halfmagic)
             ispyc = 1;
         rewind(fp);
     }
History
Date User Action Args
2021-01-30 12:21:41izbyshevsetnosy: + izbyshev
messages: + msg385982
2021-01-30 10:09:55ronaldoussorensetmessages: + msg385978
2021-01-29 22:32:21ciprian.craciuncreate