classification
Title: backports import path can not be overridden in Windows (Linux works fine)
Type: behavior Stage: resolved
Components: Windows Versions: Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Chris Caron, paul.moore, r.david.murray, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2017-10-09 21:48 by Chris Caron, last changed 2017-10-12 07:45 by paul.moore. This issue is now closed.

Files
File name Uploaded Description Edit
Test.py Chris Caron, 2017-10-10 23:43 A Test script showing the PYTHONPATH issue i'm having
site-packages-Directory.png Chris Caron, 2017-10-10 23:44 C:\Python27\Lib\site-packages\backports
Test.Directory.png Chris Caron, 2017-10-10 23:46 C:\Python.Bug\Test - a test directory i created and placed the full backports.ssl_match_hostname in
Test.Script.Backports.Dir.png Chris Caron, 2017-10-10 23:47 The backports directory (in my C:\Python.Bug\Test). The contents are different then the contents in the C:\Python27\Lib\site-packages\backports
Test.Script.Output.png Chris Caron, 2017-10-10 23:48 Output of the Test.py script I already attached showing the problem
Python.Test.zip Chris Caron, 2017-10-10 23:49 The entire Test Directory (with the Test.py and Test directory) incase you want to see the entire test dev i'm sharing
Messages (13)
msg304001 - (view) Author: Chris Caron (Chris Caron) Date: 2017-10-09 21:48
The Windows install of Python 2.7.13 defaults it's install into **C:\Python27**.  It creates several subdirectories within this; one of which is **C:\Python27\lib\site-packages\backports**. So just out of the box (without this script installed):
```python
import backports
print(backports.__path__)
# C:\Python27\lib\site-packages\backports
```

I have a custom package that sets the sys.path(0, "new path") entry which works great in Linux. I ship all my modules and settings in this path.  This to work great in Windows too 'except' for the reference to `backports`.

Consider the block of code:
```python
import sys
from os import getcwd
from os.path import join
sys.path.insert(0, join(getcwd(), 'MyLibraries'))

# Now if we import backports, we should find the one in our MyLibraries directory
import backports
print(backports.__path__)    # Nope... :(
# C:\Python27\lib\site-packages\backports

# What's weird is it's not entirely broken... you can do the following:
import chardet
print(chardet.__path__)
# Prints my path to MyLibraries/chardet
# So the path is correct and other libraries are coming in correctly
```

What's really weird, is (in Windows) even if i delete the **C:\Python27\lib\site-packages\backports**, it still imports the module; so it's like part of the Python27 build in some way.

`pip install` with the backports module I want works great because it drops the correct package in this 'lib/site-packages\backports' directory which is constantly being referenced.

From what I can see the Microsoft version of Python27 can't seem to stop referencing (even when told not to) the _backports_ location. Linux works perfectly (as intended) but having customers port to this isn't an option for me.
msg304002 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-10-09 22:01
Where did you get your installer?  I don't find any references to 'backports' in our repo, other than a couple of places in the docs where the word appears.

I also don't understand the relationship between your MyPackages and a backports directory.  They seem to be unrelated in your examples.
msg304003 - (view) Author: Chris Caron (Chris Caron) Date: 2017-10-09 22:18
Thank you for replying so quickly;  the issue stems from a bug created here
which explains it a bit better:
https://github.com/caronc/nzb-notify/issues/27 (scroll to the bottom).

In my case, I had a lib\site-packages\backports in C:\Python27 that keeps
over-riding the one defined in the \MyLibraries path. But even when the
'sys.path' has been successfully overridden, the backports module (one i
provide for the package of interest) keeps reverting to the
C:\Python27\lib\site-packages\ path.  It actually just throws an exception
that the module doesn't exist (when ordering of paths would say it does).

To answer your other question: I got the installer from here:
https://www.python.org/downloads/release/python-2713/ (the MSI 64 bit
installer - second from the bottom)
msg304004 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-10-10 00:09
What is "the backports module"?  I still don't understand what a backports directory, that I don't think our installer creates, has to do with your MyLibraries directory.

I'm sorry that I'm having trouble following your explanation.  Maybe you could list out the contents of your directories and what your sys.path is?
msg304020 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2017-10-10 07:45
The "backports" module you refer to as being in site-packages is not shipped with the standard library, it's a 3rd party addon that presumably you installed yourself. You'll need to let us know what you did to install it.

You say you manipulate sys.path manually in your code. Please provide:

1. the value of sys.path before you change it.
2. The value of sys.modules.get('backports') before you change sys.path.
3. the value of sys.path after you change it.
4. The value of sys.modules.get('backports') after you change sys.path.
msg304095 - (view) Author: Chris Caron (Chris Caron) Date: 2017-10-10 23:57
I'm not doing anything unusual.

Just to recap, I installed Python27 (in Windows 7 in my case- but this problem happens in Windows 10 too).

I did use `pip` to install packages as you'll see in the screenshot.  But even if i rename the `C:\Python27\Lib\site-packages\backports` to `ignoreme-backports`.  When i change the PYTHONPATH (from within sys.path.insert(0, 'new_path/that/has/a/backports/dir/in/it');

import backports
print backports.__path__
# always prints: C:\Python27\Lib\site-packages\backports

The problem arises because the following code won't work in my program:

from import import ssl_match_hostname

^^ That throws an exception that the package doesn't exist.  It's debugging it to the __path__ being incorrect that caused me to create this ticket.
msg304115 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2017-10-11 08:10
> I did use `pip` to install packages as you'll see in the screenshot.

Sorry, I don't see any pip commands. Please could you include sample commands inline as text, and not as screenshots, as attached files aren't accessible from emails...

As I say, I suspect the backports module is being imported before you change sys.path, and hence your sys.path change isn't affecting it.

Sorry, but unless you can provide a precise set of steps to reproduce the issue, starting from the point where you install Python, I'm not sure what else we can do to help (but I will reiterate - this is *not* a problem with the Python install itself - we don't ship a site-packages/backports directory, that came with something you installed via pip).
msg304134 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-10-11 13:37
Ah, so backports is a package on pypi.  You should report this problem wherever they do their bug tracking.  It is *possible* there is some problem in 2.7 on windows, but it is much more likely to be a bug in backports on windows.  In sort, python does not do *anything* special with a directory named backports; whatever is happening is a consequence of how that package (or one of the packages you have installed that installs into backports) interacts with the generic python import machinery.

Perhaps Paul will see something of interest here, though.
msg304142 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2017-10-11 14:30
My feeling is still that it's an issue with things (i.e., imports) happening before you adjust sys.path. But without seeing actual code that reproduces the issue, there's no way of being sure. And if that is what's going on, it wouldn't be an issue with core Python.
msg304190 - (view) Author: Chris Caron (Chris Caron) Date: 2017-10-11 22:43
-- Download Instructions
* 64 Bit ver of Python v2.7 for Windows https://www.python.org/ftp/python/2.7/python-2.7.amd64.msi
* Microsoft Visual C++ Compiler for Python 2.7: https://www.microsoft.com/en-ca/download/details.aspx?id=44266. 

-- Installation Instructions
* Install Python v2.7 (run python-2.7.amd64.msi); choose default options. 
Note: In my case, i uninstalled my already installed version (for the purpose of this post) first and made sure to delete the C:\Python27 directory left behind). Then I reinstalled the package.
   Default Options:
      - Install for all users
      - Install location: C:\Python27\
      - All packages installed (5 of 5 subfeatures)
	  
** Heads up
So at this point, the C:\Python27\Lib\site-packages will contain the following: pip, setuptools,  and wheel.

-- A Starting Point
* Download the Python.Test.zip file i attached, but since you made it clear you want everything to be present in this message, i'll do my best to try to document it and paste content here: 
It's main contents is just to provide an alternative include directory called Test. In this test directory i provide six.py, odereddict, chardet, backports, socks and sockhandler.
>>>>> dir listing of Test
10/11/2017  06:04 PM    <DIR>          .
10/11/2017  06:04 PM    <DIR>          ..
10/11/2017  06:04 PM    <DIR>          backports
10/11/2017  06:04 PM    <DIR>          chardet
10/11/2017  06:04 PM             4,221 ordereddict.py
10/11/2017  06:04 PM             4,916 ordereddict.pyc
10/11/2017  06:04 PM            23,462 six.py
10/11/2017  06:04 PM            23,754 six.pyc
10/11/2017  06:04 PM            32,006 socks.py
10/11/2017  06:04 PM             2,913 sockshandler.py
10/11/2017  06:04 PM                 0 __init__.py
               7 File(s)         91,272 bytes
>>>>> End DIR Listing

Now lets introduce the second part of the zip file i provided. Test.py.  It's sitting next to (not in) the Test directory i listed above. It looks like this
>>>>> Test.py
#!/usr/bin/env python
# -*- encoding: utf-8 -*-
import sys
from os.path import join
from os.path import abspath
from os.path import dirname

print('Path Before Change: {0}'.format('\n'.join(sys.path)))
sys.path.insert(0, join(dirname(abspath(__file__)), 'Test'))
print('Path After Change: {0}'.format('\n'.join(sys.path)))

import backports
print('Backports Path: {0}'.format(backports.__path__))

import chardet
print('chardet Path: {0}'.format(chardet.__path__))

import six
print('six Path: {0}'.format(six.__file__))
>>>>> end Test.py

Those who already downloaded Test.py will notice i stripped out all of the comments.  Bear with me here, the actual code lines are still all unchanged.
I also placed the attached zip file into my root C:\ directory.  Not ideal for obvious reasons, but for this test, it makes the output small and easy to follow.

-- Script Run # 1
When i run the above script... right now... i get:
>>>>>
C:\Python.Test>python Test.py
Path Before Change: C:\Python.Test
C:\windows\system32\python27.zip
C:\Python27\DLLs
C:\Python27\lib
C:\Python27\lib\plat-win
C:\Python27\lib\lib-tk
C:\Python27
C:\Python27\lib\site-packages
Path Before Change: C:\Python.Test\Test
C:\Python.Test
C:\windows\system32\python27.zip
C:\Python27\DLLs
C:\Python27\lib
C:\Python27\lib\plat-win
C:\Python27\lib\lib-tk
C:\Python27
C:\Python27\lib\site-packages
Backports Path: ['C:\\Python.Test\\Test\\backports']
chardet Path: ['C:\\Python.Test\\Test\\chardet']
six Path: C:\Python.Test\Test\six.pyc
>>>>>

Like you said... everything is fine; it's not a python issue... but hang on...

Let's use pip and install some simple packages...
>>>>>
C:\Python.Test>pip install pylint
... lots of content flies by; but it's successful
>>>>>

Let's run our script again (same one... same content). If you're doing this from the command line, then cut and paste:
>>>>>
C:\Python.Test>python Test.py
Path Before Change: C:\Python.Test
C:\windows\system32\python27.zip
C:\Python27\DLLs
C:\Python27\lib
C:\Python27\lib\plat-win
C:\Python27\lib\lib-tk
C:\Python27
C:\Python27\lib\site-packages
Path Before Change: C:\Python.Test\Test
C:\Python.Test
C:\windows\system32\python27.zip
C:\Python27\DLLs
C:\Python27\lib
C:\Python27\lib\plat-win
C:\Python27\lib\lib-tk
C:\Python27
C:\Python27\lib\site-packages
Backports Path: ['C:\\Python27\\lib\\site-packages\\backports']
chardet Path: ['C:\\Python.Test\\Test\\chardet']
six Path: C:\Python.Test\Test\six.pyc
>>>>>

So... what should you take from this?
- backports is no longer referencing the one the PYTHONPATH (well sys.path.insert()) suggested otherwise.
- but we know it works because the other entries were loaded okay. We can see this from the 'six' and 'chardet' inport paths.

Here is where it gets really weird...
Lets install chardet:
>>>>>
C:\Python.Test>pip install chardet
... lots of content flies by; but it's successful
>>>>>

Now lets run our Test.py again:
>>>>>
C:\Python.Test>python Test.py
Path Before Change: C:\Python.Test
C:\windows\system32\python27.zip
C:\Python27\DLLs
C:\Python27\lib
C:\Python27\lib\plat-win
C:\Python27\lib\lib-tk
C:\Python27
C:\Python27\lib\site-packages
Path Before Change: C:\Python.Test\Test
C:\Python.Test
C:\windows\system32\python27.zip
C:\Python27\DLLs
C:\Python27\lib
C:\Python27\lib\plat-win
C:\Python27\lib\lib-tk
C:\Python27
C:\Python27\lib\site-packages
Backports Path: ['C:\\Python27\\lib\\site-packages\\backports']
chardet Path: ['C:\\Python.Test\\Test\\chardet']
six Path: C:\Python.Test\Test\six.pyc
>>>>>

The 'chardet' is getting imported correctly from the directory i want it to even though it's also available in the site-packages directory where the backports one is being picked up.

Here is the directory listings for you:
>>>>>
C:\Python.Test>dir C:\Python27\Lib\site-packages\backports\
 Volume in drive C has no label.
 Volume Serial Number is ACA9-DD63

 Directory of C:\Python27\Lib\site-packages\backports

10/11/2017  06:30 PM    <DIR>          .
10/11/2017  06:30 PM    <DIR>          ..
10/11/2017  06:30 PM    <DIR>          configparser
10/11/2017  06:30 PM             7,317 functools_lru_cache.py
10/11/2017  06:30 PM             7,378 functools_lru_cache.pyc
10/11/2017  06:30 PM                65 __init__.py
>>>>>

>>>>>
C:\Python.Test>dir Test\backports
 Volume in drive C has no label.
 Volume Serial Number is ACA9-DD63

 Directory of C:\Python.Test\Test\backports

10/11/2017  06:04 PM    <DIR>          .
10/11/2017  06:04 PM    <DIR>          ..
10/11/2017  06:04 PM    <DIR>          ssl_match_hostname
10/11/2017  06:04 PM                38 __init__.py
10/11/2017  06:04 PM               227 __init__.pyc
               2 File(s)            265 bytes
>>>>>

I realize i'm frustrating you with my request, but hopefully this helps explain the problem better.
msg304193 - (view) Author: Chris Caron (Chris Caron) Date: 2017-10-12 00:52
Just to point out, i forgot the instructions of installing pip (right before) the following entry in my last post:
>> ** Heads up
>> So at this point, the C:\Python27\Lib\site-packages will contain the >>> following: pip, setuptools,  and wheel.

To be thorough, i should add that that i followed these instruction: https://pip.pypa.io/en/stable/installing/

Which required me to download get-pip.py from here https://bootstrap.pypa.io/get-pip.py and run it.  The rest of the instructions (after the **heads up) comment are still all on note.
msg304204 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2017-10-12 05:05
Note that your instructions start with downloading the installer for 2.7.0, whereas the latest version of 2.7 is 2.7.14, which includes pip.

backports is a strange beast of a package, which tries to emulate Python 3's namespace packages in Python 2.  To do so, backports.__init__ must have a very specific incantation, which yours is lacking (it only contains "from __future__ import absolute_import").  See https://pypi.org/project/backports/ for more details.
msg304213 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2017-10-12 07:45
Also, one of the packages installed creates a file

configparser-3.5.0-py2.7-nspkg.pth

This has some very suspicious looking content:

import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('backports',)); importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.set default('backports', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('backports', [os.path.dirname(p)]))) ;m = m or sys.modules.setdefault('backports', types.ModuleType('backports'));mp = (m or []) and m.__dict__.setdefault('__path__',[ ]);(p not in mp) and mp.append(p)

That's likely your problem. I've no idea what it's doing, but as Zachary says, the backports module is a strange beast...
History
Date User Action Args
2017-10-12 07:45:58paul.mooresetmessages: + msg304213
2017-10-12 05:05:21zach.waresetstatus: open -> closed
resolution: third party -> not a bug
messages: + msg304204

stage: resolved
2017-10-12 00:52:08Chris Caronsetmessages: + msg304193
2017-10-11 22:43:59Chris Caronsetmessages: + msg304190
2017-10-11 14:30:51paul.mooresetmessages: + msg304142
2017-10-11 13:37:36r.david.murraysetresolution: third party
messages: + msg304134
2017-10-11 08:10:10paul.mooresetmessages: + msg304115
2017-10-10 23:57:27Chris Caronsetmessages: + msg304095
2017-10-10 23:49:53Chris Caronsetfiles: + Python.Test.zip
2017-10-10 23:48:00Chris Caronsetfiles: + Test.Script.Output.png
2017-10-10 23:47:27Chris Caronsetfiles: + Test.Script.Backports.Dir.png
2017-10-10 23:46:11Chris Caronsetfiles: + Test.Directory.png
2017-10-10 23:44:41Chris Caronsetfiles: + site-packages-Directory.png
2017-10-10 23:43:12Chris Caronsetfiles: + Test.py
2017-10-10 07:45:07paul.mooresetmessages: + msg304020
2017-10-10 00:09:22r.david.murraysetmessages: + msg304004
2017-10-09 22:18:09Chris Caronsetmessages: + msg304003
2017-10-09 22:01:35r.david.murraysetnosy: + r.david.murray
messages: + msg304002
2017-10-09 21:48:05Chris Caroncreate