classification
Title: venv and ensurepip are affected by default pip config file
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: 20541 20570 Superseder:
Assigned To: ncoghlan Nosy List: dstufft, larry, ncoghlan, paul.moore, python-dev, r.david.murray
Priority: deferred blocker Keywords: buildbot

Created on 2013-12-23 06:41 by ncoghlan, last changed 2014-03-08 12:40 by ncoghlan. This issue is now closed.

Messages (28)
msg206843 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-12-23 06:41
In resolving issue 19734, I realised venv and ensurepip are actually in a similar situation with respect to the default pip configuration file as they were with respect to environment variables: those settings are unlikely to be appropriate for ensurepip, but pip will still pay attention to them during the bootstrapping process.

However, it's a bit trickier to test, since PIP_CONFIG_FILE will be ignored (due to the resolution of issue 19734).

The approach I will run with (if nobody has any better suggestions):

- create a temporary directory
- set os.environ["HOME"] to point to that directory
- create a platform appropriate file (pip\pip.ini on Windows, .pip/pip.conf elsewhere) containing the settings:

    [global]
    no-install=1

That should cause the test_venv tests to fail, just as setting PIP_NO_INSTALL in the environment caused them to fail as a test for the issue 19734 resolution.

In terms of forcing pip to *ignore* the global config file, the best option I have found is setting PIP_CONFIG_FILE=/dev/null (I believe the Windows equivalent would be PIP_CONFIG_FILE=NUL). The fact the file exists means pip uses it without falling back to the default file location, while the fact it is always read as empty means it has no effect on pip's operation.

I'm open to better suggestions on how to do that, but it seems like the best available option without an "isolated mode" equivalent in pip.
msg206846 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-12-23 06:49
pip side counterpart, suggesting an explicit "isolated mode" option: https://github.com/pypa/pip/issues/1397
msg206864 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-12-23 15:02
I don't know anything about pip configuration or the tests, but perhaps you want to use something like: 

'PIP_CONFIG_FILE={}'.format(os.devnull)
msg206867 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-12-23 16:35
Oh, I forgot about os.devnull.

ensurepip.bootstrap mutates the environment of the current process (hence the recommendation to use the CLI instead), so yes, doing "os.environ['PIP_CONFIG_FILE'] = os.devnull" before importing pip should do the trick.

And then generate a "no-install=true" config with HOME in test_venv to make sure it is properly ignored.
msg210223 - (view) Author: Roundup Robot (python-dev) Date: 2014-02-04 13:03
New changeset 1b8ba1346e67 by Nick Coghlan in branch 'default':
Close #20053: ignore default pip config settings
http://hg.python.org/cpython/rev/1b8ba1346e67
msg210311 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-02-05 13:29
*sigh*, Windows is not cooperating, suggesting it is still reading the default config file:

http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/4000/steps/test/logs/stdio

======================================================================
FAIL: test_with_pip (test.test_venv.EnsurePipTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_venv.py", line 342, in test_with_pip
    self.assertEqual(err, "")
AssertionError: 'C:\\Users\\Buildbot\\AppData\\Local\\Temp[57 chars]\r\n' != ''
- C:\Users\Buildbot\AppData\Local\Temp\tmpfiz5hrop\Scripts\python_d.exe: No module named pip
msg210313 - (view) Author: Roundup Robot (python-dev) Date: 2014-02-05 13:55
New changeset ddc82c4d1a44 by Nick Coghlan in branch 'default':
Issue #20053: new test to check an assumption
http://hg.python.org/cpython/rev/ddc82c4d1a44
msg210319 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-02-05 14:19
D'oh, still failing even though that new assumption check passes on Windows: http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/4002/steps/test/logs/stdio

Paul, any ideas?
msg210325 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2014-02-05 15:17
Nothing obvious or Windows-specific from what I can see. It does seem to me that EnvironmentVarGuard doesn't monkeypatch os.environ even though it looks like it intends to (__exit__ sets it back, but __enter__ doesn't monkeypatch it). So maybe that's doing something weird?

I'll try to have a better look later (not much spare time right now).
msg210376 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-02-06 11:35
EnvironmentVarGuard doesn't work through monkeypatching - you make your changes *through* the guard, and it undoes them in __exit__, as well as restoring the original binding (in case *other* code monkeypatched it).

This allows it to still be used when testing code that is *supposed* to make real environment changes (e.g. so they're visible in a subprocess).

Buildbot is currently down, so I can't check the logs again :(
msg210458 - (view) Author: Roundup Robot (python-dev) Date: 2014-02-07 12:28
New changeset 17bea44a9fa7 by Nick Coghlan in branch 'default':
Issue #20053: Actually test relevant assumption
http://hg.python.org/cpython/rev/17bea44a9fa7
msg210462 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2014-02-07 13:05
Sigh. Yes, I looked at that and wondered if os.devnul would show as existing, but I similarly misread the test and assumed it was testing what it was meant to test :-(

os.path.exists(os.devnull) is false on Windows (just checked) so this assumption *is* what's failing.
msg210463 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-02-07 13:06
As Paul noted, I discovered there was a bug in my assumption testing test - when I tried it directly on Windows, I discovered the culprit here is issue 20541.

So adding that as a dependency, and I'll disable that part of the test on Windows (and mark the assumption as an expected failure).
msg210470 - (view) Author: Roundup Robot (python-dev) Date: 2014-02-07 13:46
New changeset f3f92d55f942 by Nick Coghlan in branch 'default':
Issue #20053: Mark as an expected failure for 3.4
http://hg.python.org/cpython/rev/f3f92d55f942
msg210473 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2014-02-07 14:28
As noted in http://bugs.python.org/issue1311 (referenced from http://bugs.python.org/issue20541) it's not actually correct to assume that os.path.exists(os.devnull) returns true, as on Windows the "null device" is not a proper filesystem object.

So I'd suggest that the code here needs to avoid relying on this assumption.
msg210474 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-02-07 14:35
The problem is that the assumption of assumption of an exists <-> open equivalence is in pip's configuration file loading rather than in ensurepip or venv - the idea was to pass in an existing but empty file to avoid needing a change in pip to fix this.

However, it looks like we're just going to have to live with the misbehaviour on Windows until it can be fixed on the pip side for Python 3.4.1.
msg210476 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-02-07 14:40
(Note that the fix on the pip side may be something more direct like just ignoring all config files and environment variables when ENSUREPIP_OPTIONS is set, rather than changing the way it reads the config files)
msg210504 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2014-02-07 18:04
I'm not sure I grasp what the problem is
msg210515 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2014-02-07 18:37
In ensurepip, _disable_pip_configuration_settings has the line:

    os.environ['PIP_CONFIG_FILE'] = os.devnull

On Windows, os.devnull does not behave like a real file in that os.path.exists(os.devnull) is False even though opening it works fine. So that line of code is not portable to Windows.

The problem in pip us in baseparser.py in ConfigOptionParser:

    def get_config_files(self):
        config_file = os.environ.get('PIP_CONFIG_FILE', False)
        if config_file and os.path.exists(config_file):
            files = [config_file]
        else:
            files = [default_config_file]

There's no way to force pip to *not* use a config file.

A quick hack would be to respect a new environment variable (PIP_NO_CONFIG_FILE) which if set says to set files to [] here. A bigger fix is to implement a proper "isolated mode" in pip.

I could probably knock up the "quick hack" as a PR for pip reasonably easily.

This should really be discussed on the pip tracker, though. Nick - did you raise an issue on pip for this? Assuming you did, can you post the reference here?

Either way, ensurepip needs to be changed to use whatever mechanism pip implements. I am against special casing os.devnull in pip just to make this work (although it would address the issue).
msg210516 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2014-02-07 18:40
The proper fix is an isolated mode, but we could special case devnull in pip for 1.5.3 and make a proper isolated solution in 1.6.
msg210519 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2014-02-07 18:46
Maybe. I don't know what *else* might fail because devnull is special. The filename gets passed straight to configparser, for a start.

But if you want to do that I'm OK with that - I just won't make that change myself. Would the special-casing be permanent or would it be removed when isolated mode is implemented?
msg210520 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2014-02-07 18:47
I'd remove it in 1.6 with a proper isolated mode. I'm purely thinking of minimal changes to make it easier to to get it into 3.4.
msg210551 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2014-02-07 20:29
Sounds reasonable. Having thought about it a bit more, I don't think it's a big deal. I'll put a PR together for special-casing devnull.
msg210553 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2014-02-07 20:48
https://github.com/pypa/pip/pull/1543
msg210576 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2014-02-07 23:01
Fix now merged into pip 1.5.X branch
msg210718 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-02-09 04:22
I created issue 20570 to propose updating to pip 1.5.3 for 3.4rc2.

That would include the pip side workaround for this issue, also allowing this issue to be closed.
msg211789 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2014-02-21 01:58
pip 1.5.3 is released and I've requested larry cherry-pick it into 3.4.0 with issue20713
msg212927 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-03-08 12:40
pip 1.5.3 has been bundled and cherry picked to the release clone.
History
Date User Action Args
2014-03-08 12:40:42ncoghlansetstatus: open -> closed
resolution: fixed
messages: + msg212927

stage: commit review -> resolved
2014-02-21 01:58:01dstufftsetmessages: + msg211789
2014-02-09 04:22:14ncoghlansetpriority: normal -> deferred blocker

nosy: + larry
messages: + msg210718

dependencies: + Bundle pip 1.5.3 in Python 3.4rc2
2014-02-07 23:01:46paul.mooresetmessages: + msg210576
2014-02-07 20:48:51paul.mooresetmessages: + msg210553
2014-02-07 20:29:52paul.mooresetmessages: + msg210551
2014-02-07 18:47:22dstufftsetmessages: + msg210520
2014-02-07 18:46:27paul.mooresetmessages: + msg210519
2014-02-07 18:40:47dstufftsetmessages: + msg210516
2014-02-07 18:37:54paul.mooresetmessages: + msg210515
2014-02-07 18:04:45dstufftsetmessages: + msg210504
2014-02-07 14:40:09ncoghlansetmessages: + msg210476
2014-02-07 14:35:32ncoghlansetmessages: + msg210474
2014-02-07 14:28:34paul.mooresetmessages: + msg210473
2014-02-07 13:46:49python-devsetmessages: + msg210470
2014-02-07 13:06:56ncoghlansetdependencies: + os.path.exists() gives wrong answer for Windows special files
messages: + msg210463
2014-02-07 13:05:01paul.mooresetmessages: + msg210462
2014-02-07 12:28:34python-devsetmessages: + msg210458
2014-02-06 11:35:07ncoghlansetmessages: + msg210376
2014-02-05 15:17:20paul.mooresetmessages: + msg210325
2014-02-05 14:19:53ncoghlansetmessages: + msg210319
2014-02-05 13:55:12python-devsetmessages: + msg210313
2014-02-05 13:29:07ncoghlansetstatus: closed -> open

components: + Library (Lib)
type: behavior
keywords: + buildbot
messages: + msg210311
resolution: fixed -> (no value)
stage: resolved -> commit review
2014-02-04 13:03:03python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg210223

resolution: fixed
stage: resolved
2013-12-23 16:35:01ncoghlansetmessages: + msg206867
2013-12-23 15:02:34r.david.murraysetnosy: + r.david.murray
messages: + msg206864
2013-12-23 06:49:49ncoghlansetmessages: + msg206846
2013-12-23 06:41:08ncoghlancreate