classification
Title: test_venv fails in Travis with GCC
Type: behavior Stage: resolved
Components: Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: steve.dower Nosy List: miss-islington, pablogsal, steve.dower, vstinner, xdegaye, xtreak
Priority: normal Keywords: patch

Created on 2019-02-12 17:37 by xtreak, last changed 2019-03-21 19:02 by xdegaye. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 12220 merged steve.dower, 2019-03-07 17:27
PR 12485 merged miss-islington, 2019-03-21 17:04
Messages (10)
msg335338 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2019-02-12 17:37
I noticed this while checking issue35961. test_venv is always failing on GCC which is marked as optional in Travis.

Log : https://travis-ci.org/python/cpython/jobs/492123436#L1909

0:39:35 load avg: 1.00 [390/416] test_venv
test test_venv failed -- Traceback (most recent call last):
  File "/home/travis/build/python/cpython/Lib/test/test_venv.py", line 309, in test_multiprocessing
    out, err = check_output([envpy, '-c',
  File "/home/travis/build/python/cpython/Lib/test/test_venv.py", line 37, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['/tmp/tmpg8ubeyfn/bin/python', '-c', 'from multiprocessing import Pool; print(Pool(1).apply_async("Python".lower).get(3))']' died with <Signals.SIGABRT: 6>.

Also GCC test have been timing out for at least past 4 months : https://python.zulipchat.com/#narrow/stream/116742-core.2Fhelp/topic/GCC.20build.20in.20Travis.20always.20times.20out
msg335354 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-12 19:30
Pablo: Can this issue be related bpo-35378?

commit 3766f18c524c57784eea7c0001602017d2122156
Author: Pablo Galindo <Pablogsal@gmail.com>
Date:   Mon Feb 11 17:29:00 2019 +0000

    bpo-35378: Fix multiprocessing.Pool references (GH-11627)
msg335359 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-02-12 19:39
That looks like a similar example of the problem that bpo35378 solves. But the PR is merged and the test keeps failing:

test test_venv failed -- Traceback (most recent call last):
  File "/home/travis/build/python/cpython/Lib/test/test_venv.py", line 309, in test_multiprocessing
    out, err = check_output([envpy, '-c',
  File "/home/travis/build/python/cpython/Lib/test/test_venv.py", line 37, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['/tmp/tmplplx90pf/bin/python', '-c', 'from multiprocessing import Pool; print(Pool(1).apply_async("Python".lower).get(3))']' died with <Signals.SIGABRT: 6>.

https://travis-ci.org/python/cpython/jobs/492305033


Running that command locally on master does not die with SIGABRT:

❯ ./python -c 'from multiprocessing import Pool; print(Pool(1).apply_async("Python".lower).get(3))'; echo $?
python
/home/pablogsal/github/cpython/Lib/multiprocessing/pool.py:234: ResourceWarning: unclosed running multiprocessing pool <multiprocessing.pool.Pool state=RUN pool_size=1>
  _warn(f"unclosed running multiprocessing pool {self!r}",
ResourceWarning: Enable tracemalloc to get the object allocation traceback
0
msg335361 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-12 19:54
I can reproduce the crash using the same command than the job.

Simplified commands to reproduce the bug:

./configure && make
./python -m venv venv
./venv/bin/python -m test test_venv -m test.test_venv.BasicTest.test_multiprocessing
msg335363 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-12 19:58
The Python of the virtual environment created by the test doesn't work:

$ /tmp/tmp0vybyos2/bin/python -c pass
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: initfsencoding: failed to get the Python codec of the filesystem encoding
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f2956ab8740 (most recent call first):
Aborted
msg335364 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-12 20:02
Hum, test_executable() of test_venv is skipped if run in a venv, but not test_multiprocessing():

    # If a venv is created from a source build and that venv is used to
    # run the test, the pyvenv.cfg in the venv created in the test will
    # point to the venv being used to run the test, and we lose the link
    # to the source build - so Python can't initialise properly.
    @skipInVenv
        ...
        self.run_with_capture(venv.create, self.env_dir)
        envpy = os.path.join(os.path.realpath(self.env_dir),
                             self.bindir, self.exe)
        ...

    def test_multiprocessing(self):
        ...
        self.run_with_capture(venv.create, self.env_dir)
        envpy = os.path.join(os.path.realpath(self.env_dir),
                             self.bindir, self.exe)
        ...

The test has been added by:

commit 4e02f8f8b4baab63f927cfd87b401200ba2969e9
Author: Steve Dower <steve.dower@microsoft.com>
Date:   Fri Jan 25 14:59:12 2019 -0800
msg337402 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-03-07 16:38
I guess we need a "skipInLinuxVenv", since it works fine on Windows :)
msg337407 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-03-07 16:57
I'm on this. I'll make a new (or possibly complete replacement) for @skipInVenv that is @requireVenvCreate, since that's the common piece that doesn't work in a venv (except on Windows).
msg338559 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-03-21 17:04
New changeset 8bba81fd55873148c65b7d0e6a6effbd63048c76 by Steve Dower in branch 'master':
bpo-35978: Correctly skips venv tests in venvs (GH-12220)
https://github.com/python/cpython/commit/8bba81fd55873148c65b7d0e6a6effbd63048c76
msg338560 - (view) Author: miss-islington (miss-islington) Date: 2019-03-21 17:33
New changeset b0967fe4ed2e0e15f14ea574f82970a3fd4a5556 by Miss Islington (bot) in branch '3.7':
bpo-35978: Correctly skips venv tests in venvs (GH-12220)
https://github.com/python/cpython/commit/b0967fe4ed2e0e15f14ea574f82970a3fd4a5556
History
Date User Action Args
2019-03-23 11:55:05xdegayeunlinkissue36342 dependencies
2019-03-21 19:02:33xdegayesetnosy: + xdegaye
2019-03-21 17:46:10steve.dowersetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-03-21 17:33:57miss-islingtonsetnosy: + miss-islington
messages: + msg338560
2019-03-21 17:04:44steve.dowersetmessages: + msg338559
2019-03-21 17:04:32miss-islingtonsetpull_requests: + pull_request12438
2019-03-18 15:24:27xdegayelinkissue36342 dependencies
2019-03-07 17:27:44steve.dowersetkeywords: + patch
stage: patch review
pull_requests: + pull_request12210
2019-03-07 16:57:25steve.dowersetassignee: steve.dower
messages: + msg337407
versions: + Python 3.7
2019-03-07 16:38:45steve.dowersetmessages: + msg337402
2019-02-12 20:02:15vstinnersetnosy: + steve.dower
messages: + msg335364
2019-02-12 19:58:39vstinnersetmessages: + msg335363
2019-02-12 19:54:49vstinnersetmessages: + msg335361
2019-02-12 19:39:51pablogsalsetmessages: + msg335359
2019-02-12 19:30:34vstinnersetmessages: + msg335354
2019-02-12 17:37:52xtreakcreate