Additionally, virtualenvs have the root of their directory on `sys.path` -- this is unlike posix behaviour

For example:

$ python
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD6
4)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import Lib.os
>>> Lib.os.__file__
'C:\\Python37\\Lib\\os.py'
>>> Lib.os.listdir('C:\\')
['$Recycle.Bin', 'BGinfo', 'Documents and Settings', 'pagefile.sys', 'PerfLogs',
 'Program Files', 'Program Files (x86)', 'ProgramData', 'Python27', 'Python37',
'Recovery', 'swapfile.sys', 'System Volume Information', 'Users', 'Windows']

I don't think we can change the assumption that sys.prefix is in sys.path at this point, though technically it's not required (certainly not in 3.7).

Maybe we could create a new marker file that means "never treat this directory as a namespace package"?

sys.prefix isn't on sys.path on other platforms -- why is it on windows?

Honestly, no idea. Because it always has been :)

If you can find when it was added, we may be able to answer that.

If I'm reading this correctly this is the relevant line for python3:

https://github.com/python/cpython/blob/29cb21ddb92413931e473eb799a02e2d8cdf4a45/PC/getpathp.c#L1068

it was added in 4d0d471a8031de90a2b1ce99c4ac4780e60b3bc9

python2 seems to do the same as well and I'm having a harder time tracing that code -- my guess is it comes from here: https://github.com/python/cpython/blob/103b8d9f9179089019ddb363ec0098b63816001b/PC/getpathp.c#L550

(so in that case, only present because `python.exe` is at the root of the prefix) -- but it's still showing up in the `virtualenv` case due to site manipulation (coming from `virtualenv`'s `site.py` I believe? trying to mimic that behaviour)

Hmmm not much information there but I have to get back to what I was doing before -- I'll see if I can't find more info on this later!

If .../Tools and its subdirectories had __init__.py files, the presence of sys.prefix on sys.path, on Windows, would allow imports from those subdirs.  Otherwise, it only provides a second way to import stdlib modules.  This seems like a bit of a bug.  Guido, do you remember why we have this Windows-only feature?

I doubt it's meant as a feature. I don't recall anything about it. I suspect it's got to do with limitations of the Windows installer (at the time). Maybe there are other things that are siblings of "Lib" that must be importable, e.g. DLLs?

I recommend taking this up with Steve Dower, who owns the Windows installers these days.

sys.prefix is usually argv0_path, but in a virtual environment argv0_path is set from "home" in pyvenv.cfg in order to find the standard library, and, if the site module is imported, sys.prefix is set to the virtual environment directory. Since for Windows site.getsitepackages() also includes sys.prefix, and since argv0_path and sys.prefix are different in this case, sys.path ends up with both of them. AFAIK, it needs neither of them.

Adding argv0_path to sys.path is a vestige of the NT installation scheme in the 1990s. The old scheme had a "Bin" directory that contained "python.exe" and I presume the Python DLL and standard-library extension modules. The old setup.py installer [1] had the following code to set the value of the "PythonPath" registry key:

    path = []
    pwd = nt.getcwd()
    for i in ["Bin",
          "Lib",
          "Lib\\win",
          "Lib\\tkinter",
          "Lib\\test",
          "Lib\\dos_8x3"]:
        i = pwd + "\\" + i
        path.append(i)
    sys.path[1:] = path
    pathvalue = strop.join(path, ";")
    #print "Setting PythonPath to", pathvalue
    win32api.RegSetValue(corehandle, "PythonPath", win32con.REG_SZ, pathvalue)

Note the presence of the "Bin" directory in the list.

Early NT releases used PC/getpath_nt.c [2], which was inconsistent with Modules/getpath.c in several ways. PC/getpathp.c was added in 1997. Early on, code was added to PC/getpathp.c to include the application directory in sys.path [3] in order to "make it work on NT". This was before it supported the registry "PythonPath" value that would have included the "Bin" directory.

[1] https://github.com/python/cpython/blob/8f1b6519803ffa7aef45beb58bda654533cb1fa8/PC/setup_nt/setup.py
[2] https://github.com/python/cpython/blob/8f1b6519803ffa7aef45beb58bda654533cb1fa8/PC/getpath_nt.c
[3] https://github.com/python/cpython/commit/8f1b6519803ffa7aef45beb58bda654533cb1fa8

There are certainly people relying on sys.prefix being in sys.path (for .pth files and embedding), but probably not so many that we couldn't remove it in a major release. It is a bit strange in a venv, where there's typically nothing in the root of the environment (when we discovered this, we decided we could stop copying all the modules over, as they weren't being used anyway).

I think in every supported distribution we make though, there's no need for it. But we'd probably have to just remove it and see what breaks - most of the people relying on it probably have no idea, but they're also probably bundling it in some way that makes it easy to detect and fix.

All that said, I'm not particularly offended by this. It's namespace packages working as intended, and since it seems to be a highly complex interaction of configuration settings to get sys.prefix on sys.path in a variety of circumstances (we're not relying on registry settings here any more) I'd want to see changes and tests before assuming we can do it without regressing real scenarios.