venv and ensurepip are affected by pip environment variable settings #63933

vstinner · 2013-11-23T13:51:42Z

BPO	19734
Nosy	@loewis, @jcea, @ncoghlan, @vstinner, @larryhastings, @tiran, @ambv, @dstufft

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ncoghlan'
closed_at = <Date 2013-12-23.08:10:08.388>
created_at = <Date 2013-11-23.13:51:42.472>
labels = ['type-bug', 'library', 'release-blocker']
title = 'venv and ensurepip are affected by pip environment variable settings'
updated_at = <Date 2013-12-23.08:10:08.387>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2013-12-23.08:10:08.387>
actor = 'ncoghlan'
assignee = 'ncoghlan'
closed = True
closed_date = <Date 2013-12-23.08:10:08.388>
closer = 'ncoghlan'
components = ['Library (Lib)']
creation = <Date 2013-11-23.13:51:42.472>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 19734
keywords = ['buildbot']
message_count = 28.0
messages = ['204013', '204018', '204024', '204038', '204146', '204147', '204151', '204155', '204159', '204161', '204162', '204165', '204171', '204177', '204178', '204179', '204182', '204188', '204313', '205106', '205107', '205116', '206842', '206844', '206847', '206848', '206849', '206850']
nosy_count = 9.0
nosy_names = ['loewis', 'jcea', 'ncoghlan', 'vstinner', 'larry', 'christian.heimes', 'lukasz.langa', 'python-dev', 'dstufft']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue19734'
versions = ['Python 3.4']

vstinner · 2013-11-23T13:51:42Z

http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.x/builds/3178/steps/test/logs/stdio

Re-running failed tests in verbose mode
Re-running test 'test_venv' in verbose mode
test_defaults (test.test_venv.BasicTest) ... ok
test_executable (test.test_venv.BasicTest) ... ok
test_executable_symlinks (test.test_venv.BasicTest) ... ok
test_isolation (test.test_venv.BasicTest) ... ok
test_overwrite_existing (test.test_venv.BasicTest) ... ok
test_prefixes (test.test_venv.BasicTest) ... ok
test_symlinking (test.test_venv.BasicTest) ... ok
test_unoverwritable_fails (test.test_venv.BasicTest) ... ok
test_upgrade (test.test_venv.BasicTest) ... ok
test_explicit_no_pip (test.test_venv.EnsurePipTest) ... ok
test_no_pip_by_default (test.test_venv.EnsurePipTest) ... ok
test_with_pip (test.test_venv.EnsurePipTest) ... test test_venv failed
ERROR

======================================================================
ERROR: test_with_pip (test.test_venv.EnsurePipTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/opt/python/3.x.langa-ubuntu/build/Lib/test/test_venv.py", line 288, in test_with_pip
    self.run_with_capture(venv.create, self.env_dir, with_pip=True)
  File "/opt/python/3.x.langa-ubuntu/build/Lib/test/test_venv.py", line 48, in run_with_capture
    func(*args, **kwargs)
  File "/opt/python/3.x.langa-ubuntu/build/Lib/venv/__init__.py", line 359, in create
    builder.create(env_dir)
  File "/opt/python/3.x.langa-ubuntu/build/Lib/venv/__init__.py", line 86, in create
    self._setup_pip(context)
  File "/opt/python/3.x.langa-ubuntu/build/Lib/venv/__init__.py", line 242, in _setup_pip
    subprocess.check_output(cmd)
  File "/opt/python/3.x.langa-ubuntu/build/Lib/subprocess.py", line 618, in check_output
    raise CalledProcessError(retcode, process.args, output=output)
subprocess.CalledProcessError: Command '['/tmp/tmp4_ukigy9/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 3

vstinner · 2013-11-23T13:57:08Z

A different error on Solaris:

http://buildbot.python.org/all/builders/AMD64%20Solaris%2011%20%5BSB%5D%203.x/builds/2202/steps/test/logs/stdio

Re-running test 'test_venv' in verbose mode
test_defaults (test.test_venv.BasicTest) ... ok
test_executable (test.test_venv.BasicTest) ... ok
test_executable_symlinks (test.test_venv.BasicTest) ... ok
test_isolation (test.test_venv.BasicTest) ... ok
test_overwrite_existing (test.test_venv.BasicTest) ... ok
test_prefixes (test.test_venv.BasicTest) ... ok
test_symlinking (test.test_venv.BasicTest) ... ok
test_unoverwritable_fails (test.test_venv.BasicTest) ... ok
test_upgrade (test.test_venv.BasicTest) ... ok
test_explicit_no_pip (test.test_venv.EnsurePipTest) ... ok
test_no_pip_by_default (test.test_venv.EnsurePipTest) ... ok
test_with_pip (test.test_venv.EnsurePipTest) ... /home/cpython/buildslave/cc-32/3.x.snakebite-solaris11-amd64/build/Lib/ctypes/util.py:179: ResourceWarning: unclosed file <_io.TextIOWrapper name=5 encoding='646'>
for line in os.popen(cmd).readlines():
test test_venv failed
FAIL

======================================================================
FAIL: test_with_pip (test.test_venv.EnsurePipTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/cpython/buildslave/cc-32/3.x.snakebite-solaris11-amd64/build/Lib/test/test_venv.py", line 294, in test_with_pip
    self.assertEqual(err, b"")
AssertionError: b"/home/cpython/buildslave/cc-32/3.x.snake[163 chars]):\n" != b''

vstinner · 2013-11-23T14:06:01Z

Error similar to Solaris on OpenIndiana:

http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/6946/steps/test/logs/stdio

======================================================================
FAIL: test_with_pip (test.test_venv.EnsurePipTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/test_venv.py", line 294, in test_with_pip
    self.assertEqual(err, b"")
AssertionError: b'/tmp/tmpnv3o61s9/bin/python: No module named pip\n' != b''

ncoghlan · 2013-11-23T14:53:07Z

Interesting, looks like the warnings from bpo-19694 *weren't* the cause
of the bad return code - something else is failing. I should have some time
to investigate tomorrow afternoon, but it will consist of trying to improve
the diagnostic output from the tests. If we can get the buildbot owners
involved, that could help a lot (1 am here, I'm about to go to sleep).
On 24 Nov 2013 00:06, "STINNER Victor" <report@bugs.python.org> wrote:

STINNER Victor added the comment:

Error similar to Solaris on OpenIndiana:

http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/6946/steps/test/logs/stdio

======================================================================
FAIL: test_with_pip (test.test_venv.EnsurePipTest)
----------------------------------------------------------------------

> Traceback (most recent call last):
>   File
> "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/test_venv.py",
> line 294, in test_with_pip
>     self.assertEqual(err, b"")
> AssertionError: b'/tmp/tmpnv3o61s9/bin/python: No module named pip\n' !=
> b''
>
>

----------

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue19734\>

ncoghlan · 2013-11-24T01:20:52Z

Bumping priority and adding Larry to the nosy list

Checking the latest results from the systems listed above:

Ubuntu LTS still failing due to this: http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.x/builds/3213/steps/test/logs/stdio (added Łukasz to nosy list)

x86 OpenIndiana still failing due to this: http://buildbot.python.org/all/builders/x86%20OpenIndiana%203.x/builds/7123/steps/test/logs/stdio (added Jesús to the nosy list)

AMD64 OpenIndiana machine is still failing, but not due to this (test_tarfile failure): http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/6972/steps/test/logs/stdio

Windows 7 failure is in test_tarfile: http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/3410/steps/test/logs/stdio

Windows XP failure was a build issue for the new _opcode module: http://buildbot.python.org/all/builders/x86%20XP-4%203.x/builds/9651/steps/test/logs/stdio

Solaris (not a stable buildbot) had slightly different symptoms and a few other errors: http://buildbot.python.org/all/builders/AMD64%20Solaris%2011%20%5BSB%5D%203.x/builds/2216/steps/test/logs/stdio

AIX has lots of issues, include some where features pip is relying on don't work: http://buildbot.python.org/all/builders/PPC64%20AIX%203.x/builds/1127/steps/test/logs/stdio

System Z just isn't happy with test_gdb: http://buildbot.python.org/all/builders/System%20Z%20Linux%203.x/builds/795/steps/test/logs/stdio

larryhastings · 2013-11-24T01:23:55Z

Nick: I get that gdb failure, bisecting now. See bpo-19743.

python-dev · 2013-11-24T01:47:50Z

New changeset cb598129837c by Nick Coghlan in branch 'default':
bpo-19734: better diagnostics for test_venv failures
http://hg.python.org/cpython/rev/cb598129837c

python-dev · 2013-11-24T01:53:16Z

New changeset 989de1a267b1 by Nick Coghlan in branch 'default':
Issue bpo-19734: venv still needs isolated mode
http://hg.python.org/cpython/rev/989de1a267b1

tiran · 2013-11-24T02:05:27Z

Could a deprecation warning cause the failing test?

Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\runpy.py", line 160, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\runpy.py", line 73, in _run_code
    exec(code, run_globals)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\ensurepip\__main__.py", line 66, in <module>
    main()
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\ensurepip\__main__.py", line 61, in main
    default_pip=args.default_pip,
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\ensurepip\__init__.py", line 92, in bootstrap
    _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\ensurepip\__init__.py", line 28, in _run_pip
    import pip
  File "<frozen importlib._bootstrap>", line 2147, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2136, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1178, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1140, in _load_backward_compatible
  File "C:\Users\Buildbot\AppData\Local\Temp\tmpi0kq8p59\pip-1.5.rc1-py2.py3-none-any.whl\pip\__init__.py", line 9, in <module>
  File "<frozen importlib._bootstrap>", line 2147, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2136, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1178, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1140, in _load_backward_compatible
  File "C:\Users\Buildbot\AppData\Local\Temp\tmpi0kq8p59\pip-1.5.rc1-py2.py3-none-any.whl\pip\log.py", line 8, in <module>
  File "<frozen importlib._bootstrap>", line 2147, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2136, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1178, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1140, in _load_backward_compatible
  File "C:\Users\Buildbot\AppData\Local\Temp\tmpi0kq8p59\setuptools-1.3.2-py2.py3-none-any.whl\pkg_resources.py", line 20, in <module>
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\imp.py", line 32, in <module>
    PendingDeprecationWarning)
PendingDeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

ncoghlan · 2013-11-24T02:16:19Z

Christian, that was my original theory in bpo-19694 (since that box runs with PYTHONWARNINGS=d set), but it doesn't look like it is the problem, since running in isolated mode didn't fix the issue (although it did suppress the warnings).

ncoghlan · 2013-11-24T02:17:45Z

That specific stack trace is from the commit where I inadvertently reverted the isolated mode fix - the pip installation test is always run with PYTHONWARNINGS=e now to make sure it will fail in non-isolated mode.

ncoghlan · 2013-11-24T02:40:43Z

OpenIndiana failure looks like a resource management issue in ctypes:
======================================================================
FAIL: test_with_pip (test.test_venv.EnsurePipTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/test_venv.py", line 304, in test_with_pip
    self.assertEqual(err, "")
AssertionError: "/export/home/buildbot/32bits/3.x.cea-ind[346 chars]):\n" != ''
- /export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/ctypes/util.py:179: ResourceWarning: unclosed file <_io.TextIOWrapper name=5 encoding='646'>
-   for line in os.popen(cmd).readlines():
- /export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/ctypes/util.py:179: ResourceWarning: unclosed file <_io.TextIOWrapper name=5 encoding='646'>
-   for line in os.popen(cmd).readlines():

Ubuntu LTS failure still puzzles me:

======================================================================
FAIL: test_with_pip (test.test_venv.EnsurePipTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/opt/python/3.x.langa-ubuntu/build/Lib/test/test_venv.py", line 289, in test_with_pip
    self.run_with_capture(venv.create, self.env_dir, with_pip=True)
subprocess.CalledProcessError: Command '['/tmp/tmpwsahapjn/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 3

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python/3.x.langa-ubuntu/build/Lib/test/test_venv.py", line 295, in test_with_pip
    self.fail(msg)
AssertionError: Command '['/tmp/tmpwsahapjn/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 3

**Subprocess Output**
Could not find an activated virtualenv (required).

python-dev · 2013-11-24T02:54:05Z

New changeset 7c080ee796a6 by Nick Coghlan in branch 'default':
Issue bpo-19734: ctypes resource management fixes
http://hg.python.org/cpython/rev/7c080ee796a6

ncoghlan · 2013-11-24T04:53:23Z

test_venv is now passing on 3.x OpenIndiana: http://buildbot.python.org/all/builders/x86%20OpenIndiana%203.x/builds/7129/steps/test/logs/stdio

Ubuntu LTS buildbot is still unhappy: http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.x/builds/3221/steps/test/logs/stdio

And looking at the environment armed with the improved error details, this is likely the culprit:

  PIP_REQUIRE_VIRTUALENV=true

I'll hack the test to delete it from the subprocess invocation for now, but I think there's a pip bug here: -E (and, equivalently, -I) should likely cause pip to ignore *its* environment variables as well.

ncoghlan · 2013-11-24T04:55:04Z

Confirmed:

$ PIP_REQUIRE_VIRTUALENV=true ./python -m test test_venv
[1/1] test_venv
test test_venv failed -- Traceback (most recent call last):
  File "/home/ncoghlan/devel/py3k/Lib/test/test_venv.py", line 298, in test_with_pip
    self.run_with_capture(venv.create, self.env_dir, with_pip=True)
subprocess.CalledProcessError: Command '['/tmp/tmpo1fj4gy6/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 3

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ncoghlan/devel/py3k/Lib/test/test_venv.py", line 304, in test_with_pip
    self.fail(msg)
AssertionError: Command '['/tmp/tmpo1fj4gy6/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 3

**Subprocess Output**
Could not find an activated virtualenv (required).

1 test failed:
test_venv

python-dev · 2013-11-24T04:59:34Z

New changeset 9186fdae7e1f by Nick Coghlan in branch 'default':
Issue bpo-19734: Ensure test_venv ignores PIP_REQUIRE_VIRTUALENV
http://hg.python.org/cpython/rev/9186fdae7e1f

python-dev · 2013-11-24T06:49:32Z

New changeset 124e51c19e4f by Nick Coghlan in branch 'default':
Issue bpo-19734: Also run pip version check in isolated mode
http://hg.python.org/cpython/rev/124e51c19e4f

ncoghlan · 2013-11-24T07:48:51Z

test_venv appears to be green on all the stable buildbots now.

However, blocking PIP_REQUIRE_VIRTUALENV in the test probably isn't the right place for workaround - resolution in venv or ensurepip would be more appropriate.

ncoghlan · 2013-11-25T08:41:16Z

Hmm, this may also indicate a bug in pip's "require virtualenv" handling.
Why isn't it detecting that sys.prefix and sys.base_prefix are different,
and hence it *is* running in a venv created virtual environment?

ncoghlan · 2013-12-03T12:17:50Z

Issue for the lack of attention paid to sys.flags.ignore_environment: pypa/pip#1359

ncoghlan · 2013-12-03T12:18:15Z

Issue for the failure to detect a native venv: pypa/pip#1358

ncoghlan · 2013-12-03T13:14:48Z

1358 (failing to detect native virtual environments) will be resolved upstream, but for 1359 (telling pip to ignore the environment variables), I've decided I agree with Paul's suggestion on the pip issue that it makes sense for venv to just strip all environment variables starting with "PIP_" from the environment passed to the subprocess.

python-dev · 2013-12-23T06:16:27Z

New changeset 7a5a4d7c564d by Nick Coghlan in branch 'default':
Close bpo-19734: ignore pip env vars in ensurepip
http://hg.python.org/cpython/rev/7a5a4d7c564d

ncoghlan · 2013-12-23T06:42:05Z

bpo-20053 covers a related issue with ensurepip and venv being affected by the user's default pip config file.

ncoghlan · 2013-12-23T07:04:33Z

The PIP_REQUIRES_VIRTUALENV buildbot still failed:

http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.x/builds/3495/steps/test/logs/stdio

This suggests both that the setting didn't get cleared as expected *and* that pip 1.5rc2 still failed to detect the pyvenv created virtual environment.

python-dev · 2013-12-23T07:40:16Z

New changeset 98d8bf13a32c by Nick Coghlan in branch 'default':
Issue bpo-19734: ignore pip env vars in ensurepip._uninstall
http://hg.python.org/cpython/rev/98d8bf13a32c

python-dev · 2013-12-23T07:42:14Z

New changeset 3b3d1c312042 by Nick Coghlan in branch 'default':
Issue bpo-19734: add missing NEWS entry
http://hg.python.org/cpython/rev/3b3d1c312042

ncoghlan · 2013-12-23T08:10:08Z

Buildbot is happy with that version:
http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.x/builds/3496/steps/test/logs/stdio

vstinner added the tests Tests in the Lib/test dir label Nov 23, 2013

ncoghlan added the release-blocker label Nov 24, 2013

ncoghlan added deferred-blocker and removed release-blocker labels Nov 24, 2013

ncoghlan changed the title ~~test_venv.test_with_pip() fails on "AMD64 Ubuntu LTS 3.x" buildbot~~ venv and ensurepip are affected by pip environment variable settings Nov 24, 2013

ncoghlan added release-blocker and removed deferred-blocker labels Dec 3, 2013

ncoghlan added the stdlib Python modules in the Lib dir label Dec 3, 2013

ncoghlan removed the tests Tests in the Lib/test dir label Dec 3, 2013

ncoghlan self-assigned this Dec 21, 2013

python-dev mannequin closed this as completed Dec 23, 2013

ncoghlan reopened this Dec 23, 2013

ncoghlan closed this as completed Dec 23, 2013

ncoghlan added the type-bug An unexpected behavior, bug, or error label Dec 23, 2013

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

venv and ensurepip are affected by pip environment variable settings #63933

venv and ensurepip are affected by pip environment variable settings #63933

vstinner commented Nov 23, 2013

vstinner commented Nov 23, 2013

vstinner commented Nov 23, 2013

vstinner commented Nov 23, 2013

ncoghlan commented Nov 23, 2013

ncoghlan commented Nov 24, 2013

larryhastings commented Nov 24, 2013

python-dev mannequin commented Nov 24, 2013

python-dev mannequin commented Nov 24, 2013

tiran commented Nov 24, 2013

ncoghlan commented Nov 24, 2013

ncoghlan commented Nov 24, 2013

ncoghlan commented Nov 24, 2013

python-dev mannequin commented Nov 24, 2013

ncoghlan commented Nov 24, 2013

ncoghlan commented Nov 24, 2013

python-dev mannequin commented Nov 24, 2013

python-dev mannequin commented Nov 24, 2013

ncoghlan commented Nov 24, 2013

ncoghlan commented Nov 25, 2013

ncoghlan commented Dec 3, 2013

ncoghlan commented Dec 3, 2013

ncoghlan commented Dec 3, 2013

python-dev mannequin commented Dec 23, 2013

ncoghlan commented Dec 23, 2013

ncoghlan commented Dec 23, 2013

python-dev mannequin commented Dec 23, 2013

python-dev mannequin commented Dec 23, 2013

ncoghlan commented Dec 23, 2013

venv and ensurepip are affected by pip environment variable settings #63933

venv and ensurepip are affected by pip environment variable settings #63933

Comments

vstinner commented Nov 23, 2013

vstinner commented Nov 23, 2013

vstinner commented Nov 23, 2013

vstinner commented Nov 23, 2013

ncoghlan commented Nov 23, 2013

ncoghlan commented Nov 24, 2013

larryhastings commented Nov 24, 2013

python-dev mannequin commented Nov 24, 2013

python-dev mannequin commented Nov 24, 2013

tiran commented Nov 24, 2013

ncoghlan commented Nov 24, 2013

ncoghlan commented Nov 24, 2013

ncoghlan commented Nov 24, 2013

python-dev mannequin commented Nov 24, 2013

ncoghlan commented Nov 24, 2013

ncoghlan commented Nov 24, 2013

python-dev mannequin commented Nov 24, 2013

python-dev mannequin commented Nov 24, 2013

ncoghlan commented Nov 24, 2013

ncoghlan commented Nov 25, 2013

ncoghlan commented Dec 3, 2013

ncoghlan commented Dec 3, 2013

ncoghlan commented Dec 3, 2013

python-dev mannequin commented Dec 23, 2013

ncoghlan commented Dec 23, 2013

ncoghlan commented Dec 23, 2013

python-dev mannequin commented Dec 23, 2013

python-dev mannequin commented Dec 23, 2013

ncoghlan commented Dec 23, 2013