classification
Title: venv and ensurepip are affected by pip environment variable settings
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ncoghlan Nosy List: christian.heimes, dstufft, haypo, jcea, larry, loewis, lukasz.langa, ncoghlan, python-dev
Priority: release blocker Keywords: buildbot

Created on 2013-11-23 13:51 by haypo, last changed 2013-12-23 08:10 by ncoghlan. This issue is now closed.

Messages (28)
msg204013 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-11-23 13:51
http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.x/builds/3178/steps/test/logs/stdio

Re-running failed tests in verbose mode
Re-running test 'test_venv' in verbose mode
test_defaults (test.test_venv.BasicTest) ... ok
test_executable (test.test_venv.BasicTest) ... ok
test_executable_symlinks (test.test_venv.BasicTest) ... ok
test_isolation (test.test_venv.BasicTest) ... ok
test_overwrite_existing (test.test_venv.BasicTest) ... ok
test_prefixes (test.test_venv.BasicTest) ... ok
test_symlinking (test.test_venv.BasicTest) ... ok
test_unoverwritable_fails (test.test_venv.BasicTest) ... ok
test_upgrade (test.test_venv.BasicTest) ... ok
test_explicit_no_pip (test.test_venv.EnsurePipTest) ... ok
test_no_pip_by_default (test.test_venv.EnsurePipTest) ... ok
test_with_pip (test.test_venv.EnsurePipTest) ... test test_venv failed
ERROR

======================================================================
ERROR: test_with_pip (test.test_venv.EnsurePipTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/python/3.x.langa-ubuntu/build/Lib/test/test_venv.py", line 288, in test_with_pip
    self.run_with_capture(venv.create, self.env_dir, with_pip=True)
  File "/opt/python/3.x.langa-ubuntu/build/Lib/test/test_venv.py", line 48, in run_with_capture
    func(*args, **kwargs)
  File "/opt/python/3.x.langa-ubuntu/build/Lib/venv/__init__.py", line 359, in create
    builder.create(env_dir)
  File "/opt/python/3.x.langa-ubuntu/build/Lib/venv/__init__.py", line 86, in create
    self._setup_pip(context)
  File "/opt/python/3.x.langa-ubuntu/build/Lib/venv/__init__.py", line 242, in _setup_pip
    subprocess.check_output(cmd)
  File "/opt/python/3.x.langa-ubuntu/build/Lib/subprocess.py", line 618, in check_output
    raise CalledProcessError(retcode, process.args, output=output)
subprocess.CalledProcessError: Command '['/tmp/tmp4_ukigy9/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 3
msg204018 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-11-23 13:57
A different error on Solaris:

http://buildbot.python.org/all/builders/AMD64%20Solaris%2011%20%5BSB%5D%203.x/builds/2202/steps/test/logs/stdio

Re-running test 'test_venv' in verbose mode
test_defaults (test.test_venv.BasicTest) ... ok
test_executable (test.test_venv.BasicTest) ... ok
test_executable_symlinks (test.test_venv.BasicTest) ... ok
test_isolation (test.test_venv.BasicTest) ... ok
test_overwrite_existing (test.test_venv.BasicTest) ... ok
test_prefixes (test.test_venv.BasicTest) ... ok
test_symlinking (test.test_venv.BasicTest) ... ok
test_unoverwritable_fails (test.test_venv.BasicTest) ... ok
test_upgrade (test.test_venv.BasicTest) ... ok
test_explicit_no_pip (test.test_venv.EnsurePipTest) ... ok
test_no_pip_by_default (test.test_venv.EnsurePipTest) ... ok
test_with_pip (test.test_venv.EnsurePipTest) ... /home/cpython/buildslave/cc-32/3.x.snakebite-solaris11-amd64/build/Lib/ctypes/util.py:179: ResourceWarning: unclosed file <_io.TextIOWrapper name=5 encoding='646'>
  for line in os.popen(cmd).readlines():
test test_venv failed
FAIL

======================================================================
FAIL: test_with_pip (test.test_venv.EnsurePipTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/cpython/buildslave/cc-32/3.x.snakebite-solaris11-amd64/build/Lib/test/test_venv.py", line 294, in test_with_pip
    self.assertEqual(err, b"")
AssertionError: b"/home/cpython/buildslave/cc-32/3.x.snake[163 chars]):\n" != b''
msg204024 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-11-23 14:06
Error similar to Solaris on OpenIndiana:

http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/6946/steps/test/logs/stdio

======================================================================
FAIL: test_with_pip (test.test_venv.EnsurePipTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/test_venv.py", line 294, in test_with_pip
    self.assertEqual(err, b"")
AssertionError: b'/tmp/tmpnv3o61s9/bin/python: No module named pip\n' != b''

----------------------------------------------------------------------
msg204038 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-23 14:53
Interesting, looks like the warnings from issue 19694 *weren't* the cause
of the bad return code - something else is failing. I should have some time
to investigate tomorrow afternoon, but it will consist of trying to improve
the diagnostic output from the tests. If we can get the buildbot owners
involved, that could help a lot (1 am here, I'm about to go to sleep).
On 24 Nov 2013 00:06, "STINNER Victor" <report@bugs.python.org> wrote:

>
> STINNER Victor added the comment:
>
> Error similar to Solaris on OpenIndiana:
>
>
> http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/6946/steps/test/logs/stdio
>
> ======================================================================
> FAIL: test_with_pip (test.test_venv.EnsurePipTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/test_venv.py",
> line 294, in test_with_pip
>     self.assertEqual(err, b"")
> AssertionError: b'/tmp/tmpnv3o61s9/bin/python: No module named pip\n' !=
> b''
>
> ----------------------------------------------------------------------
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue19734>
> _______________________________________
>
msg204146 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-24 01:20
Bumping priority and adding Larry to the nosy list

Checking the latest results from the systems listed above:

Ubuntu LTS still failing due to this: http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.x/builds/3213/steps/test/logs/stdio (added Łukasz to nosy list)

x86 OpenIndiana still failing due to this: http://buildbot.python.org/all/builders/x86%20OpenIndiana%203.x/builds/7123/steps/test/logs/stdio (added Jesús to the nosy list)

AMD64 OpenIndiana machine is still failing, but not due to this (test_tarfile failure): http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/6972/steps/test/logs/stdio

Windows 7 failure is in test_tarfile: http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/3410/steps/test/logs/stdio

Windows XP failure was a build issue for the new _opcode module: http://buildbot.python.org/all/builders/x86%20XP-4%203.x/builds/9651/steps/test/logs/stdio

Solaris (not a stable buildbot) had slightly different symptoms and a few other errors: http://buildbot.python.org/all/builders/AMD64%20Solaris%2011%20%5BSB%5D%203.x/builds/2216/steps/test/logs/stdio

AIX has lots of issues, include some where features pip is relying on don't work: http://buildbot.python.org/all/builders/PPC64%20AIX%203.x/builds/1127/steps/test/logs/stdio

System Z just isn't happy with test_gdb: http://buildbot.python.org/all/builders/System%20Z%20Linux%203.x/builds/795/steps/test/logs/stdio
msg204147 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2013-11-24 01:23
Nick: I get that gdb failure, bisecting now.  See #19743.
msg204151 - (view) Author: Roundup Robot (python-dev) Date: 2013-11-24 01:47
New changeset cb598129837c by Nick Coghlan in branch 'default':
Issue 19734: better diagnostics for test_venv failures
http://hg.python.org/cpython/rev/cb598129837c
msg204155 - (view) Author: Roundup Robot (python-dev) Date: 2013-11-24 01:53
New changeset 989de1a267b1 by Nick Coghlan in branch 'default':
Issue #19734: venv still needs isolated mode
http://hg.python.org/cpython/rev/989de1a267b1
msg204159 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-11-24 02:05
Could a deprecation warning cause the failing test?

Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\runpy.py", line 160, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\runpy.py", line 73, in _run_code
    exec(code, run_globals)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\ensurepip\__main__.py", line 66, in <module>
    main()
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\ensurepip\__main__.py", line 61, in main
    default_pip=args.default_pip,
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\ensurepip\__init__.py", line 92, in bootstrap
    _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\ensurepip\__init__.py", line 28, in _run_pip
    import pip
  File "<frozen importlib._bootstrap>", line 2147, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2136, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1178, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1140, in _load_backward_compatible
  File "C:\Users\Buildbot\AppData\Local\Temp\tmpi0kq8p59\pip-1.5.rc1-py2.py3-none-any.whl\pip\__init__.py", line 9, in <module>
  File "<frozen importlib._bootstrap>", line 2147, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2136, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1178, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1140, in _load_backward_compatible
  File "C:\Users\Buildbot\AppData\Local\Temp\tmpi0kq8p59\pip-1.5.rc1-py2.py3-none-any.whl\pip\log.py", line 8, in <module>
  File "<frozen importlib._bootstrap>", line 2147, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2136, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1178, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1140, in _load_backward_compatible
  File "C:\Users\Buildbot\AppData\Local\Temp\tmpi0kq8p59\setuptools-1.3.2-py2.py3-none-any.whl\pkg_resources.py", line 20, in <module>
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\imp.py", line 32, in <module>
    PendingDeprecationWarning)
PendingDeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
msg204161 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-24 02:16
Christian, that was my original theory in issue 19694 (since that box runs with PYTHONWARNINGS=d set), but it doesn't look like it is the problem, since running in isolated mode didn't fix the issue (although it did suppress the warnings).
msg204162 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-24 02:17
That specific stack trace is from the commit where I inadvertently reverted the isolated mode fix - the pip installation test is always run with PYTHONWARNINGS=e now to make sure it will fail in non-isolated mode.
msg204165 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-24 02:40
OpenIndiana failure looks like a resource management issue in ctypes:
======================================================================
FAIL: test_with_pip (test.test_venv.EnsurePipTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/test_venv.py", line 304, in test_with_pip
    self.assertEqual(err, "")
AssertionError: "/export/home/buildbot/32bits/3.x.cea-ind[346 chars]):\n" != ''
- /export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/ctypes/util.py:179: ResourceWarning: unclosed file <_io.TextIOWrapper name=5 encoding='646'>
-   for line in os.popen(cmd).readlines():
- /export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/ctypes/util.py:179: ResourceWarning: unclosed file <_io.TextIOWrapper name=5 encoding='646'>
-   for line in os.popen(cmd).readlines():


Ubuntu LTS failure still puzzles me:

======================================================================
FAIL: test_with_pip (test.test_venv.EnsurePipTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/python/3.x.langa-ubuntu/build/Lib/test/test_venv.py", line 289, in test_with_pip
    self.run_with_capture(venv.create, self.env_dir, with_pip=True)
subprocess.CalledProcessError: Command '['/tmp/tmpwsahapjn/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 3

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python/3.x.langa-ubuntu/build/Lib/test/test_venv.py", line 295, in test_with_pip
    self.fail(msg)
AssertionError: Command '['/tmp/tmpwsahapjn/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 3

**Subprocess Output**
Could not find an activated virtualenv (required).
msg204171 - (view) Author: Roundup Robot (python-dev) Date: 2013-11-24 02:54
New changeset 7c080ee796a6 by Nick Coghlan in branch 'default':
Issue #19734: ctypes resource management fixes
http://hg.python.org/cpython/rev/7c080ee796a6
msg204177 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-24 04:53
test_venv is now passing on 3.x OpenIndiana: http://buildbot.python.org/all/builders/x86%20OpenIndiana%203.x/builds/7129/steps/test/logs/stdio

Ubuntu LTS buildbot is still unhappy: http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.x/builds/3221/steps/test/logs/stdio

And looking at the environment armed with the improved error details, this is likely the culprit:

  PIP_REQUIRE_VIRTUALENV=true

I'll hack the test to delete it from the subprocess invocation for now, but I think there's a pip bug here: -E (and, equivalently, -I) should likely cause pip to ignore *its* environment variables as well.
msg204178 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-24 04:55
Confirmed:

$ PIP_REQUIRE_VIRTUALENV=true ./python -m test test_venv
[1/1] test_venv
test test_venv failed -- Traceback (most recent call last):
  File "/home/ncoghlan/devel/py3k/Lib/test/test_venv.py", line 298, in test_with_pip
    self.run_with_capture(venv.create, self.env_dir, with_pip=True)
subprocess.CalledProcessError: Command '['/tmp/tmpo1fj4gy6/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 3

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ncoghlan/devel/py3k/Lib/test/test_venv.py", line 304, in test_with_pip
    self.fail(msg)
AssertionError: Command '['/tmp/tmpo1fj4gy6/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 3

**Subprocess Output**
Could not find an activated virtualenv (required).


1 test failed:
    test_venv
msg204179 - (view) Author: Roundup Robot (python-dev) Date: 2013-11-24 04:59
New changeset 9186fdae7e1f by Nick Coghlan in branch 'default':
Issue #19734: Ensure test_venv ignores PIP_REQUIRE_VIRTUALENV
http://hg.python.org/cpython/rev/9186fdae7e1f
msg204182 - (view) Author: Roundup Robot (python-dev) Date: 2013-11-24 06:49
New changeset 124e51c19e4f by Nick Coghlan in branch 'default':
Issue #19734: Also run pip version check in isolated mode
http://hg.python.org/cpython/rev/124e51c19e4f
msg204188 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-24 07:48
test_venv appears to be green on all the stable buildbots now.

However, blocking PIP_REQUIRE_VIRTUALENV in the test probably isn't the right place  for workaround - resolution in venv or ensurepip would be more appropriate.
msg204313 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-25 08:41
Hmm, this may also indicate a bug in pip's "require virtualenv" handling.
Why isn't it detecting that sys.prefix and sys.base_prefix are different,
and hence it *is* running in a venv created virtual environment?
msg205106 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-12-03 12:17
Issue for the lack of attention paid to sys.flags.ignore_environment: https://github.com/pypa/pip/issues/1359
msg205107 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-12-03 12:18
Issue for the failure to detect a native venv: https://github.com/pypa/pip/issues/1358
msg205116 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-12-03 13:14
1358 (failing to detect native virtual environments) will be resolved upstream, but for 1359 (telling pip to ignore the environment variables), I've decided I agree with Paul's suggestion on the pip issue that it makes sense for venv to just strip all environment variables starting with "PIP_" from the environment passed to the subprocess.
msg206842 - (view) Author: Roundup Robot (python-dev) Date: 2013-12-23 06:16
New changeset 7a5a4d7c564d by Nick Coghlan in branch 'default':
Close #19734: ignore pip env vars in ensurepip
http://hg.python.org/cpython/rev/7a5a4d7c564d
msg206844 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-12-23 06:42
Issue 20053 covers a related issue with ensurepip and venv being affected by the user's default pip config file.
msg206847 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-12-23 07:04
The PIP_REQUIRES_VIRTUALENV buildbot still failed:

http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.x/builds/3495/steps/test/logs/stdio

This suggests both that the setting didn't get cleared as expected *and* that pip 1.5rc2 still failed to detect the pyvenv created virtual environment.
msg206848 - (view) Author: Roundup Robot (python-dev) Date: 2013-12-23 07:40
New changeset 98d8bf13a32c by Nick Coghlan in branch 'default':
Issue #19734: ignore pip env vars in ensurepip._uninstall
http://hg.python.org/cpython/rev/98d8bf13a32c
msg206849 - (view) Author: Roundup Robot (python-dev) Date: 2013-12-23 07:42
New changeset 3b3d1c312042 by Nick Coghlan in branch 'default':
Issue #19734: add missing NEWS entry
http://hg.python.org/cpython/rev/3b3d1c312042
msg206850 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-12-23 08:10
Buildbot is happy with that version:
http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.x/builds/3496/steps/test/logs/stdio
History
Date User Action Args
2013-12-23 08:10:08ncoghlansetstatus: open -> closed
type: behavior
messages: + msg206850

resolution: fixed
stage: needs patch -> resolved
2013-12-23 07:42:13python-devsetmessages: + msg206849
2013-12-23 07:40:16python-devsetmessages: + msg206848
2013-12-23 07:04:34ncoghlansetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg206847

stage: resolved -> needs patch
2013-12-23 06:42:04ncoghlansetmessages: + msg206844
2013-12-23 06:16:26python-devsetstatus: open -> closed
resolution: fixed
messages: + msg206842

stage: resolved
2013-12-21 15:31:46ncoghlansetassignee: ncoghlan
2013-12-03 13:14:48ncoghlansetmessages: + msg205116
components: + Library (Lib), - Tests
2013-12-03 12:18:23ncoghlansetpriority: deferred blocker -> release blocker
2013-12-03 12:18:14ncoghlansetmessages: + msg205107
2013-12-03 12:17:50ncoghlansetmessages: + msg205106
2013-11-25 08:41:15ncoghlansetmessages: + msg204313
2013-11-24 07:48:51ncoghlansetpriority: release blocker -> deferred blocker

messages: + msg204188
title: test_venv.test_with_pip() fails on "AMD64 Ubuntu LTS 3.x" buildbot -> venv and ensurepip are affected by pip environment variable settings
2013-11-24 06:49:32python-devsetmessages: + msg204182
2013-11-24 04:59:34python-devsetmessages: + msg204179
2013-11-24 04:55:04ncoghlansetmessages: + msg204178
2013-11-24 04:53:23ncoghlansetmessages: + msg204177
2013-11-24 02:54:04python-devsetmessages: + msg204171
2013-11-24 02:41:15ncoghlansetnosy: + dstufft
2013-11-24 02:40:42ncoghlansetmessages: + msg204165
2013-11-24 02:17:44ncoghlansetmessages: + msg204162
2013-11-24 02:16:18ncoghlansetmessages: + msg204161
2013-11-24 02:05:27christian.heimessetnosy: + christian.heimes
messages: + msg204159
2013-11-24 01:53:16python-devsetmessages: + msg204155
2013-11-24 01:47:50python-devsetnosy: + python-dev
messages: + msg204151
2013-11-24 01:23:55larrysetmessages: + msg204147
2013-11-24 01:20:53ncoghlansetpriority: normal -> release blocker
nosy: + larry, jcea, lukasz.langa, - nick
messages: + msg204146

2013-11-23 14:53:06ncoghlansetmessages: + msg204038
2013-11-23 14:06:01hayposetmessages: + msg204024
2013-11-23 13:57:08hayposetmessages: + msg204018
2013-11-23 13:55:01pitrousetnosy: + ncoghlan
2013-11-23 13:51:42haypocreate