classification
Title: Build does not work with closed stdin on NetBSD
Type: enhancement Stage: resolved
Components: Build Versions: Python 3.6
process
Status: closed Resolution: duplicate
Dependencies: Superseder: EBADF error on x86 Tiger 3.x buildbot
View: 30225
Assigned To: Nosy List: MartinHusemann, leot, vstinner, wiz, xdegaye
Priority: normal Keywords:

Created on 2018-04-29 10:16 by MartinHusemann, last changed 2019-10-24 10:39 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
patch-Python_pylifecycle.c leot, 2018-05-19 14:45 Use fstat(2) instead of dup(2) on NetBSD too. The behaviour are the same of the one niced in bpo-30225
is_valid_fd.c leot, 2018-05-19 14:46 is_valid_fd() from Python 3.6.5 (Python/pylifecycle.c)
Messages (9)
msg315886 - (view) Author: Martin Husemann (MartinHusemann) Date: 2018-04-29 10:16
When building python extensions in the background w/o stdin (and stderr and stdout redirected to a log file), the invocation of setup.py fails.

Normal build in a shell:

Example from pyexpat:
> /usr/pkg/bin/python3.6  setup.py  build
running build
running build_ext
building 'pyexpat' extension
creating build
creating build/temp.netbsd-8.99.14-amd64-3.6
creating build/temp.netbsd-8.99.14-amd64-3.6/Modules
gcc -DNDEBUG -O2 -pipe -D_FORTIFY_SOURCE=2 -I/usr/include -I/usr/pkg/include -O2 -pipe -D_FORTIFY_SOURCE=2 -I/usr/include -I/usr/pkg/include/python3.6 -I/usr/include -I/usr/pkg/include/python3.6 -fPIC -DHAVE_EXPAT_H -I/work/pkgobj/textproc/py-expat/work/.buildlink/include -I/usr/pkg/include/python3.6 -c Modules/pyexpat.c -o build/temp.netbsd-8.99.14-amd64-3.6/Modules/pyexpat.o
creating build/lib.netbsd-8.99.14-amd64-3.6
gcc -pthread -shared -L. -L/usr/lib -Wl,-R/usr/lib -L/usr/pkg/lib -Wl,-R/usr/pkg/lib -L/usr/X11R7/lib -Wl,-R/usr/X11R7/lib -L/usr/lib -Wl,-R/usr/lib -L/usr/pkg/lib -Wl,-R/usr/pkg/lib -L/usr/X11R7/lib -Wl,-R/usr/X11R7/lib -L/usr/lib -Wl,-R/usr/lib -Wl,-R/usr/pkg/lib -O2 -pipe -D_FORTIFY_SOURCE=2 -I/usr/include -I/usr/pkg/include/python3.6 -I/usr/include -I/usr/pkg/include/python3.6 build/temp.netbsd-8.99.14-amd64-3.6/Modules/pyexpat.o -L/work/pkgobj/textproc/py-expat/work/.buildlink/lib -L/usr/pkg/lib -Wl,-R/work/pkgobj/textproc/py-expat/work/.buildlink/lib -lexpat -lpython3.6 -o build/lib.netbsd-8.99.14-amd64-3.6/pyexpat.so


But if invoked as 

> ( sleep 15; /usr/pkg/bin/python3.6  setup.py  build ) >/tmp/log 2>&1 &
> ^D

and then watching /tmp/log from another session:

Fatal Python error: Py_Initialize: can't initialize sys standard streams
OSError: [Errno 9] Bad file descriptor

Current thread 0x000075b4e4cee800 (most recent call first):
[1]   Abort trap (core dumped) /usr/pkg/bin/pyt...
msg316183 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2018-05-04 18:24
I cannot reproduce the problem with Python 3.6.5, building pyephem and with the command:

( sleep 15; python  setup.py  build ) >foo.log 2>&1
msg317110 - (view) Author: Martin Husemann (MartinHusemann) Date: 2018-05-19 10:36
You need to exit the parent shell, to get the original stdin revoke(2)'d.
That is: the Ctrl-D in the original descritpion is not line noise. Sorry, should have been more explicit (or used "exit" or something instead)
msg317114 - (view) Author: Leonardo Taccari (leot) Date: 2018-05-19 11:19
A simpler way to reproduce that (probably this problem is not limited to
building but to sys module):

    (sleep 10; python3.6 -c 'import sys' >/tmp/log 2>&1) & exit

And in `/tmp/log':

    Fatal Python error: Py_Initialize: can't initialize sys standard streams
    OSError: [Errno 9] Bad file descriptor
     
    Current thread 0x00007226e0ab3800 (most recent call first):
msg317117 - (view) Author: Leonardo Taccari (leot) Date: 2018-05-19 13:31
And here the backtrace of the corresponding core (this is on
NetBSD/amd64 8.99.15 with lang/python36 package of today pkgsrc-current):

    % gdb -core python3.6.core `which python3.6`
    Reading symbols from /usr/pkg/bin/python3.6...done.
    [New process 1]
    Core was generated by `python3.6'.
    Program terminated with signal SIGABRT, Aborted.
    #0  0x000076255932924a in _lwp_kill () from /usr/lib/libc.so.12
    (gdb) bt
    #0  0x000076255932924a in _lwp_kill () from /usr/lib/libc.so.12
    #1  0x0000762559328e65 in abort () at /usr/src/lib/libc/stdlib/abort.c:74
    #2  0x000076255a749d33 in Py_FatalError (msg=msg@entry=0x76255a7dea20 "Py_Initialize: can't initialize sys standard streams")
        at Python/pylifecycle.c:1462
    #3  0x000076255a74af4c in _Py_InitializeEx_Private (install_sigs=1, install_importlib=1) at Python/pylifecycle.c:450
    #4  0x000076255a764902 in Py_Main (argc=argc@entry=3, argv=argv@entry=0x76255ab01060) at Modules/main.c:700
    #5  0x0000000000400e2f in main (argc=3, argv=<optimized out>) at ./Programs/python.c:69
    (gdb) f 4
    #4  0x000076255a764902 in Py_Main (argc=argc@entry=3, argv=argv@entry=0x76255ab01060) at Modules/main.c:700
    700         Py_Initialize();
    (gdb) f 3
    #3  0x000076255a74af4c in _Py_InitializeEx_Private (install_sigs=1, install_importlib=1) at Python/pylifecycle.c:450
    450             Py_FatalError(
    (gdb) list
    445         if (_PyTraceMalloc_Init() < 0)
    446             Py_FatalError("Py_Initialize: can't initialize tracemalloc");
    447
    448         initmain(interp); /* Module __main__ */
    449         if (initstdio() < 0)
    450             Py_FatalError(
    451                 "Py_Initialize: can't initialize sys standard streams");
    452
    453         /* Initialize warnings. */
    454         if (PySys_HasWarnOptions()) {

Please let me know if any further information is needed and I will try
to help!

Thank you!
msg317121 - (view) Author: Leonardo Taccari (leot) Date: 2018-05-19 14:45
After testing is_valid_fd() (from Python/pylifecycle.c) separately
(an `is_valid_fd.c' file will be attached to ease reproduction) I think
that also NetBSD is affected by bpo-30225.

Using dup(2) (at is currently done in NetBSD):

    % cc -o ivf is_valid_fd.c
    % sleep 5 && ./ivf > /tmp/log & exit
    [... the terminal is closed via ^D ...]
    % cat /tmp/log
    0: 1
    1: 1
    2: 1

Using fstat(2):

    % cc -DUSE_FSTAT -o ivf is_valid_fd.c
    % sleep 5 && ./ivf > /tmp/log & exit
    [... the terminal is closed via ^D ...]
    % cat /tmp/log
    0: 0
    1: 1
    2: 0

The possible attached patch seems to fix the problem reported by Martin.
msg355179 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-22 23:30
I'm quite that this issue has been fixed by bpo-30225 which is part Python 3.7.4, Python 3.8.0 and newer. I mark the issue as duplicate of bpo-30225. Please reopen the bug if these versions still fail on NetBSD.


commit 1c4670ea0cc3d208121af11b9b973e6bb268e570
Author: Victor Stinner <victor.stinner@gmail.com>
Date:   Thu May 4 00:45:56 2017 +0200

    bpo-30225: Fix is_valid_fd() on macOS Tiger (#1443)
    
    is_valid_fd() now uses fstat() instead of dup() on macOS to return 0
    on a pipe when the other side of the pipe is closed. fstat() fails
    with EBADF in that case, whereas dup() succeed.
msg355321 - (view) Author: Leonardo Taccari (leot) Date: 2019-10-24 10:38
Hello Victor,
I can confirm that I can no longer reproduce this problem with Python
3.7.5 on NetBSD and according a double-check to
Python/pylifecycle.c:is_valid_fd() now fstat() is always used on
NetBSD.

Thanks for the update and for looking at it!
msg355323 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-24 10:39
> I can confirm that I can no longer reproduce this problem with Python
3.7.5 on NetBSD

Great!
History
Date User Action Args
2019-10-24 10:39:12vstinnersetmessages: + msg355323
2019-10-24 10:38:04leotsetmessages: + msg355321
2019-10-22 23:30:02vstinnersetstatus: open -> closed

superseder: EBADF error on x86 Tiger 3.x buildbot
title: Build does not work with closed stdin -> Build does not work with closed stdin on NetBSD
nosy: + vstinner

messages: + msg355179
resolution: duplicate
stage: resolved
2018-05-19 14:46:48leotsetfiles: + is_valid_fd.c
2018-05-19 14:45:48leotsetfiles: + patch-Python_pylifecycle.c

messages: + msg317121
2018-05-19 13:31:32leotsetmessages: + msg317117
2018-05-19 11:19:27leotsetnosy: + leot
messages: + msg317114
2018-05-19 10:36:40MartinHusemannsetmessages: + msg317110
2018-05-04 18:24:52xdegayesetnosy: + xdegaye
messages: + msg316183
2018-04-29 10:51:01wizsetnosy: + wiz
2018-04-29 10:16:35MartinHusemanncreate