classification
Title: [2.7] test_idle fails with: /usr/bin/xvfb-run: line 181: 3617 Segmentation fault
Type: Stage:
Components: Build Versions: Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: deep42thought, eschwartz, terry.reedy, vstinner
Priority: normal Keywords:

Created on 2018-07-11 13:04 by deep42thought, last changed 2018-07-16 12:57 by deep42thought.

Files
File name Uploaded Description Edit
log.all deep42thought, 2018-07-11 13:03 log of the last command
Messages (7)
msg321447 - (view) Author: Erich Eckner (deep42thought) Date: 2018-07-11 13:03
building python 2.7.15 on archlinux32 from source succeeds, but running the testsuite segfaults with:

test_writelines (idlelib.idle_test.test_io.PseudeOutputFilesTest) ... /usr/bin/xvfb-run: line 181:  3617 Segmentation fault      (core dumped) DISPLAY=:$SERVERNUM XAUTHORITY=$AUTHFILE "$@" 2>&1

the executed commands are:

  cd Python-${pkgver}

  # Temporary workaround for FS#22322
  # See http://bugs.python.org/issue10835 for upstream report
  sed -i "/progname =/s/python/python${_pybasever}/" Python/pythonrun.c

  # Enable built-in SQLite module to load extensions (fix FS#22122)
  sed -i "/SQLITE_OMIT_LOAD_EXTENSION/d" setup.py

  # FS#23997
  sed -i -e "s|^#.* /usr/local/bin/python|#!/usr/bin/python2|" Lib/cgi.py

  sed -i "s/python2.3/python2/g" Lib/distutils/tests/test_build_scripts.py \
    Lib/distutils/tests/test_install_scripts.py

  # Ensure that we are using the system copy of various libraries (expat, zlib and libffi),
  # rather than copies shipped in the tarball
  rm -r Modules/expat
  rm -r Modules/zlib
  rm -r Modules/_ctypes/{darwin,libffi}*

  # clean up #!s
  find . -name '*.py' | \
    xargs sed -i "s|#[ ]*![ ]*/usr/bin/env python$|#!/usr/bin/env python2|"

  # Workaround asdl_c.py/makeopcodetargets.py errors after we touched the shebangs
  touch Include/Python-ast.h Python/Python-ast.c Python/opcode_targets.h

  # FS#48761
  # http://bugs.python.org/issue25750
  patch -Np1 -i ../descr_ref.patch

  export OPT="${CFLAGS}"
  ./configure --prefix=/usr \
              --enable-shared \
              --with-threads \
              --enable-optimizations \
              --with-lto \
              --enable-ipv6 \
              --enable-unicode=ucs4 \
              --with-system-expat \
              --with-system-ffi \
              --with-dbmliborder=gdbm:ndbm \
              --without-ensurepip

  make

  LD_LIBRARY_PATH="${srcdir}/Python-${pkgver}":${LD_LIBRARY_PATH} \
    xvfb-run "${srcdir}/Python-${pkgver}/python" -m test.regrtest -v -uall 2>&1 | tee log.all
msg321453 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-07-11 14:12
I don't see how test_idle runs the /usr/bin/xvfb-run command. First, it would be interesting to see what is the argument of the xvfb-run command and check if it does crash alone.

Can you please try to trace system calls using the following command and then attached the created trace file to this issue?

strace -f -o trace ./python -m test test_idle

> test_writelines (idlelib.idle_test.test_io.PseudeOutputFilesTest) ... /usr/bin/xvfb-run: line 181:  3617 Segmentation fault      (core dumped) DISPLAY=:$SERVERNUM XAUTHORITY=$AUTHFILE "$@" 2>&1

What are values of DISPLAY, SERVERNUM and AUTHFILE environment variables?
msg321470 - (view) Author: Eli Schwartz (eschwartz) Date: 2018-07-11 14:53
Note the last line, which shows that the testsuite itself is executed using xvfb-run.

The archlinux32 build script is based off the original PKGBUILD for archlinux: https://git.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/python2

xvfb-run is only involved, because the testsuite depends on being run in a GUI environment in order to test GUI components...
msg321478 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-07-11 15:10
Ah, I see. Can you try to get the Python traceback and/or C traceback using gdb?
msg321631 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-07-13 19:42
I think we should first determine whether xvfb is required to get the segfault, and second, where it fails.

Does
<python2> -m test.regrtest -v -m test_io test_idle
fail and fail in the same place, the 11th and last test run in the module? If not, what about

xvfb-run <python2> -m ...?

In either case, inserting prints in 'test_writelines', which has multiple asserts, should determine where it fails.

Test.test_idle checks that tkinter can imported, which in turn imports _tkinter, which tries to connect to a windowing system. Running with xvfb satisfies this.  Test_idle finds and run the test modules in idlelib.idle_test.  Individual modules and classes somewhat redundantly import test.support and check "requires('gui')".  So IDLE gui tests, and, I  am fairly sure, the tkinter gui tests, also require -ugui or -uall to run.  

Test_io is not a gui test, so does not need or run "requires('gui')", and hence does not need -ugui.  On the other hand, it imports PyShell, which imports tkinter and _tkinter, and hence would need xvfb-run on a headless server, as opposed to a workstation with screen.  Erich, I assume you are running on the latter.
msg321736 - (view) Author: Erich Eckner (deep42thought) Date: 2018-07-16 12:39
> Can you try to get the Python traceback and/or C traceback using gdb?
Sry, I can't - I don't know how to use gdb :-(
Is there some one-line command which makes gdb start xvfb-run and attach to it? Or do I need to start xvfb-run and separately attach gdb to it? (how do I do this?)

> ... and hence would need xvfb-run on a headless server, as opposed to a workstation with screen.  Erich, I assume you are running on the latter.

I'm running this on a headless box (packaging for archlinux32). Setting up a desktop on the test box would be some effort, but should be possible in principle.

In the original PKGBUILD which Eli posted, there is a comment about when the tests started failing - which suggests, the problem is connected to xvfb-run.
msg321737 - (view) Author: Erich Eckner (deep42thought) Date: 2018-07-16 12:57
ok, some update:

1st: running the complete test suite with xvfb fails (as already reported):
> LD_LIBRARY_PATH="${srcdir}/Python-${pkgver}":${LD_LIBRARY_PATH} \
    xvfb-run "${srcdir}/Python-${pkgver}/python" -m test.regrtest -v -uall
...
0:02:14 load avg: 0.90 [174/403/1] test_idle
test_close (idlelib.idle_test.test_io.PseudeInputFilesTest) ... ok
test_misc (idlelib.idle_test.test_io.PseudeInputFilesTest) ... ok
test_read (idlelib.idle_test.test_io.PseudeInputFilesTest) ... ok
test_readline (idlelib.idle_test.test_io.PseudeInputFilesTest) ... ok
test_readlines (idlelib.idle_test.test_io.PseudeInputFilesTest) ... ok
test_unsupported (idlelib.idle_test.test_io.PseudeInputFilesTest) ... ok
test_close (idlelib.idle_test.test_io.PseudeOutputFilesTest) ... ok
test_misc (idlelib.idle_test.test_io.PseudeOutputFilesTest) ... ok
test_unsupported (idlelib.idle_test.test_io.PseudeOutputFilesTest) ... ok
test_write (idlelib.idle_test.test_io.PseudeOutputFilesTest) ... ok
test_writelines (idlelib.idle_test.test_io.PseudeOutputFilesTest) ... ok
/usr/bin/xvfb-run: line 181: 13491 Segmentation fault      (core dumped) DISPLAY=:$SERVERNUM XAUTHORITY=$AUTHFILE "$@" 2>&1


running w/o xvfb-run succeeds:
> LD_LIBRARY_PATH="${srcdir}/Python-${pkgver}":${LD_LIBRARY_PATH} ${srcdir}/Python-${pkgver}/python -m test.regrtest -v -m test_io test_idle
== CPython 2.7.15 (default, Jul 16 2018, 12:22:58) [GCC 8.1.1 20180531]
==   Linux-4.17.4-1.0-ARCH-i686-with-glibc2.1.3 little-endian
==   /home/vagrant/python2/src/Python-2.7.15/build/test_python_14208
== CPU count: 2
Run tests sequentially
0:00:00 load avg: 0.26 [1/1] test_idle
test_close (idlelib.idle_test.test_io.PseudeInputFilesTest) ... ok
test_misc (idlelib.idle_test.test_io.PseudeInputFilesTest) ... ok
test_read (idlelib.idle_test.test_io.PseudeInputFilesTest) ... ok
test_readline (idlelib.idle_test.test_io.PseudeInputFilesTest) ... ok
test_readlines (idlelib.idle_test.test_io.PseudeInputFilesTest) ... ok
test_unsupported (idlelib.idle_test.test_io.PseudeInputFilesTest) ... ok
test_close (idlelib.idle_test.test_io.PseudeOutputFilesTest) ... ok
test_misc (idlelib.idle_test.test_io.PseudeOutputFilesTest) ... ok
test_unsupported (idlelib.idle_test.test_io.PseudeOutputFilesTest) ... ok
test_write (idlelib.idle_test.test_io.PseudeOutputFilesTest) ... ok
test_writelines (idlelib.idle_test.test_io.PseudeOutputFilesTest) ... ok

----------------------------------------------------------------------
Ran 11 tests in 0.003s

OK
1 test OK.

Total duration: 150 ms
Tests result: SUCCESS


running only these tests with xvfb-run _succeeds_:
> LD_LIBRARY_PATH="${srcdir}/Python-${pkgver}":${LD_LIBRARY_PATH} xvfb-run ${srcdir}/Python-${pkgver}/python -m test.regrtest -v -m test_io test_idle
== CPython 2.7.15 (default, Jul 16 2018, 12:22:58) [GCC 8.1.1 20180531]
==   Linux-4.17.4-1.0-ARCH-i686-with-glibc2.1.3 little-endian
==   /home/vagrant/python2/src/Python-2.7.15/build/test_python_14226
== CPU count: 2
Run tests sequentially
0:00:00 load avg: 0.15 [1/1] test_idle
test_close (idlelib.idle_test.test_io.PseudeInputFilesTest) ... ok
test_misc (idlelib.idle_test.test_io.PseudeInputFilesTest) ... ok
test_read (idlelib.idle_test.test_io.PseudeInputFilesTest) ... ok
test_readline (idlelib.idle_test.test_io.PseudeInputFilesTest) ... ok
test_readlines (idlelib.idle_test.test_io.PseudeInputFilesTest) ... ok
test_unsupported (idlelib.idle_test.test_io.PseudeInputFilesTest) ... ok
test_close (idlelib.idle_test.test_io.PseudeOutputFilesTest) ... ok
test_misc (idlelib.idle_test.test_io.PseudeOutputFilesTest) ... ok
test_unsupported (idlelib.idle_test.test_io.PseudeOutputFilesTest) ... ok
test_write (idlelib.idle_test.test_io.PseudeOutputFilesTest) ... ok
test_writelines (idlelib.idle_test.test_io.PseudeOutputFilesTest) ... ok

----------------------------------------------------------------------
Ran 11 tests in 0.004s

OK
1 test OK.

Total duration: 144 ms
Tests result: SUCCESS
History
Date User Action Args
2018-07-16 12:57:23deep42thoughtsetmessages: + msg321737
2018-07-16 12:39:09deep42thoughtsetmessages: + msg321736
2018-07-13 19:42:36terry.reedysetnosy: + terry.reedy
messages: + msg321631
2018-07-11 15:10:56vstinnersetmessages: + msg321478
2018-07-11 14:53:20eschwartzsetnosy: + eschwartz
messages: + msg321470
2018-07-11 14:12:26vstinnersetnosy: + vstinner
messages: + msg321453
2018-07-11 14:06:04vstinnersettitle: segfault in testsuite -> [2.7] test_idle fails with: /usr/bin/xvfb-run: line 181: 3617 Segmentation fault
2018-07-11 13:04:12deep42thoughtcreate