https://github.com/pypa/virtualenv/issues/979

TL;DR

py.exe does not search PATH when using `#! /usr/bin/env python3`.

And the right steps I think should be:

1. Search ./
2. Search PATH
3. Fallback to system installed Python 3

But currently py.exe seems only do Step 3 once the shebang has substring of `python3`.

1. I don't think searching . should be included - on Unix /usr/bin/env searches PATH, and I believe we should do the same here.

2. The PATH search does happen (from my reading of the code) but it looks for a "python3" command, which isn't available. Again this is the same behaviour as Unix, and so defensible, but given that Windows doesn't provide the versioned executables, it's less useful there.

The biggest problem is that with "#!/usr/bin/env python3" the user clearly expects Python 3, and without versioned executables, we can't guarantee that on Windows for a PATH search. Whether not supporting this usage is worse than supporting it without a guarantee that you'll get Python 3, I'm not sure.

But actually after I created a `python3.bat` into venv/Scripts, the shebang `#! /usr/bin/env python3` still does not take the `python3.bat`.

Then I created a hardlink from `python.exe` to `python3.exe`, the shebang still does not work.

I didn't check the code of py.exe, but it just behave like it may not search the PATH of current session correctly.

More details were updated on github:
https://github.com/pypa/virtualenv/issues/979

And during the entire testing procedure, `C:\Users\wdhwg001\study\venv\Scripts` was manually added to user `PATH` and system `PATH`, but `py.exe` cannot find the hardlinked `python3.exe` in there.

Virtual commands that are parsed to start with "python" are restricted to the list of installed versions of Python -- unless the virtual command allows searching and the Python command has no version specification (e.g.`#! /usr/bin/env python`). "python3" is versioned, so the launcher will not search for it using the default SearchPath path. 

For reference, see maybe_handle_shebang() [1], lines 1265-69:

    command += 6;   /* skip past "python" */
    if (search && ((*command == L'\0') || isspace(*command))) {
        /* Command is eligible for path search, and there
         * is no version specification.
         */

The default SearchPath path includes the py.exe application directory, the process working directory, system directories, and the %Path% directories. This default search path can be made 'safe' via SetSearchPathMode or a registry setting, but doing so just swaps the order to search system directories before the working directory. It doesn't remove the working directory, unlike the safe search used by CreateProcess when %NoDefaultCurrentDirectoryInExePath% is defined. 

I think the implementation of find_on_path() could be improved. It should always try appending .EXE, .COM, .BAT, and .CMD if the bare command isn't found, not just if the command doesn't contain a ".". It's not unheard of to have a name with mulitple dots in it, e.g. "spam4.2.exe". Also, looping over all of the extensions listed in %PathExt% is of doubtful value. We don't call ShellExecuteEx to execute arbitrary file types, such as .VBS, .VBE, .JS, .JSE, .WSF, .WSH, and .MSC.

[1]: https://hg.python.org/cpython/file/v3.6.0b3/PC/launcher.c#l1112

Okay. But somehow I still think the current way of handling shebang is confusing. That makes a python3-only script unable to find a way to use both versioned shebang and virtualenv.

Maybe it could be changed to `Command is eligible for path search, or there is no version specification`.

Then if shebang is versioned, py.exe should try to search `PATH` for a versioned executable file(though it might not exist), and fallback to the installed list.

This could be less confusing I think.

And more importantly, virtualenv cannot and shouldn't create registry keys or even hijack py.exe to fix this issue. This change provides a better way.

I agree that the docs are a little confusing on this. Having said that, though, I'm not entirely sure the behaviour needs fixing. The scenario where there's a problem is:

1. User has written a script that needs Python 3 and won't run with Python 2.
2. User does not want to force a specific interpreter, or use the system Python.
3. User wants to use virtualenvs (or otherwise manipulate PATH to switch Python versions) and will sometimes have Python 2 and sometimes Python 3 active.
4. User expects to be able to run the script with Python 2 active and in that case use the system Python 3.

That's a pretty rare situation, IMO. Add to that the fact that it's really hard to detect when the python on PATH is Python 2 (you basically need to run python -V).

The discussion seems to be veering a little too much towards "do what I mean" to me, and I'd prefer we keep the behaviour simple.

I'm happy with the idea that "/usr/bin/env python3" should search, and then fall back to the system defined Python 3. The question is what should we search *for*. There seem to me to be only 2 options:

1. python - but this could select a Python 2 interpreter, so that seems wrong.
2. python3 - that's logical but has a number of quirks
   2a. There isn't a python3 executable supplied as standard, so this behaviour is useless without user customisation.
   2b. Do we want to force searching for additional extensions like .bat? What about .ps1 for Powershell users? What do we do for the GUI version?

Overall, I think option 2 has too many grey areas to be really acceptable, and I'm inclined to say YAGNI, and we simply leave "/usr/bin/env python3" undefined.

For the OP's use case, I'm not clear precisely why "/usr/bin/env python" wouldn't be a sufficiently good approach. If there's a specific scenario in his setup that means we really do *have* to have "/usr/bin/env python3" I'd be grateful if he could clarify. Or to put it another way, is this issue simply "I was surprised it didn't work as I expected" or is it "I cannot do what I want because this doesn't work"?

> I'm inclined to say YAGNI, and we simply leave "/usr/bin/env python3" undefined. 

I can't say I'm really happy with this answer. Fact is, 
1) you almost always have to work from within a virtual env these days, 
2) you really want to have meaningful shebangs (ie, version-specific) in your scripts because, well, 2017 and the world is still split between py2 and py3, 
3) the py.exe launcher is the new shiny thing and we all just want to use it.

So, it *could* annoy some people to find out that py.exe just can't handle both venvs and shebangs in a meaningful way, and you must give up on either venvs, or shebangs, or py.exe. 

As far as I know (3.6.0 and 2.7.13), you have 3 ways to invoke a script:

1) "version-specific" launcher (eg "py -3 foo.py"): this will always pick the system interpreter, never abides neither venvs nor shebangs. A perfectly defined, utterly useless behavior. 

2) invoke interpreter (eg "python foo.py", the good old way): this will always be venvs-complaint, will never parse shebangs. And yet at the moment, it's probably your best option *when* inside a venv - and the worst when outside, of course. 

3) "version-agnostic" launcher (eg "py foo.py"). Outside a venv, this will always abide version-specific shebangs (ie, "#!python3"); the only quirk being that when facing a generic shebang instead, it will pick Python 2 (because, 2017 and Linux...). Oh well, I can live with that. 
But when you are inside a venv, then
- if it finds a "not-env" shebang (ie "python" or "/usr/bin/python"), then it will NOT abide the venv - frustrating, yet correct I suppose;
- if it finds any kind of version-specific shebang (EVEN a "env"-type of shebang!), then again it will follow the shebang but NOT the venv, and will pick up the system interpreter. That, even when you are inside a venv that perfectly matches the shebang prescription. 

Now, this can be very frustrating because a useless "/usr/bin/env python" shebang triggers the correct behavior both inside and outside a venv, BUT a much more useful "usr/bin/env python3" will fail even if we are inside a matching venv. 

I feel that all it needs to be perfect is some kind of behavior like this: dear py.exe, 
- when you are invoked in a version-agnostic way, and 
- when you find an "env"-and-version-specific shebang in the script, 
then please, 
- just look at my %path% before doing anything else. It could be that the first interpreter you find there will just match the requirement from the shebang (that is, because on occasion I just know what I'm doing, and I've just activated a matching venv, you see). If so, just give me that Python and I will be ok; 
- otherwise, my bad: feel free to resume any other search strategy you find appropriate, and I will humbly accept whatever interpreter you may fetch, even a php one. 

I think this would be just reasonable enough. What I'm missing here?

> 2) you really want to have meaningful shebangs (ie, version-specific) in your scripts because, well, 2017 and the world is still split between py2 and py3, 

This is the one I think is overspecifying. I don't see a really *good* reason for not just saying /usr/bin/env python.

What would you want to happen if you said /usr/bin/env python3 if you had Python 2 on your PATH? The only reasonable answer I can see is "give an error", so why not just put

    if sys.version_info < (3,0):
        raise RuntimeError("Needs python 3")

at the top of your script?

Add to this the fact that I don't even know how you'd check if a python interpreter that's on PATH is Python 2 or 3 without executing it (and the overhead of running it an extra time to query its version is unacceptable) and I still don't see the justification.

> - just look at my %path% before doing anything else. It could be that the first interpreter you find there will just match the requirement from the shebang (that is, because on occasion I just know what I'm doing, and I've just activated a matching venv, you see). If so, just give me that Python and I will be ok; 
> - otherwise, my bad: feel free to resume any other search strategy you find appropriate, and I will humbly accept whatever interpreter you may fetch, even a php one. 

> I think this would be just reasonable enough. What I'm missing here?

You're missing the fact that it's not possible to tell by inspection the version of a Python interpreter. On Unix, the executable name includes the version, so it's easy. If Python on Windows changed to ship python3.exe and python37.exe alongside python.exe, then it might be worth revisiting this discussion, but not as things stand (IMO).

> If Python on Windows changed to ship python3.exe and python37.exe alongside python.exe, then it might be worth revisiting this discussion

Agreed, though if we started including versioned executables wouldn't that resolve this issue naturally? (As in, we'd search for python3.exe and find it?)

FWIW, I'm not a huge fan of including versioned executables - I would much rather include versioned *launchers*, so we can put them all in the one directory and avoid cluttering the search path (i.e. rename "py.exe" to "python.exe", "python2.exe" and "python3.exe" and have it check its own name before searching).

I agree, I don't particularly want versioned executables. I'm not sure I see much point to even having versioned launchers - "py -2" and "py -3" seem fine to me if needed.

The only use cases I can see are:

1. Use the Python executable that's first on PATH: "py"
2. Use the specific system installation of Python X[.Y]: "py -X[.Y]"

For shebang usage:

1. Use the Python executable that's first on PATH: "#!/usr/bin/env python"
2. Use the specific system installation of Python X[.Y]: "#!/usr/bin/pythonX[.Y]"
3. Use a specific interpreter: "#!<absolute path>"

The Unix ability to have 2 different versions of Python on PATH and select which you use based on executable name doesn't exist on Windows, and so there's no equivalent of the Unix "#!/usr/bin/env pythonX[.Y]"

If you want your script to fail under certain conditions - whether it's that the interpreter is a version you don't support, or something else, then test for that case and fail with an error. Checking runtime conditions doesn't need to be handled by the shebang.

The only change I'd consider reasonable here would be a doc change to explain the behaviour a bit more clearly. I might try to put something together if I have the time.

Paul:
> it's not possible to tell by inspection the version of a Python interpreter. 

True, but it's an implementation detail. Couldn't be solved? Versioned interpreters a la Linux, of course, or maybe how about including some kind of manifest file?


Steve:
> including versioned executables wouldn't that resolve this issue naturally? 

Definitely. 


Paul:
> 1. Use the Python executable that's first on PATH: "py"

Except, that's currently *almost never* true when I'm inside a venv! (For this to be true, I must give up on versioned shebangs). 
Now, if only 'py' could *always* look at the PATH *only*, ignoring shebangs - well this would be at least consistent behavior. 

Now, the absolute worst scenario here is when you have a prompt like "(myenv) $>" screaming to your face that you are inside a venv, and you go "py foo.py" expecting the venv interpreter to execute foo.py, because you are inside a venv, right? 
But wait!, foo.py was actually written by some Linux hacker, therefore *versioned* shebang, therefore 'py' fetches the wrong (system) Python, therefore no dependencies found, therefore crash. 
How I'm supposed to solve this? 


Paul: 
> The only change I'd consider reasonable here would be a doc change 

Thanks, it would really help, because I'm afraid I can't make it work for me. Most of all, I would really appreciate some sort of "best practice" suggestion on how to put together 'py', venvs and shebang. 
At the moment, I'm afraid my provisional policy is as follows: 
- when outside a venv (almost never) it's ok to 'py';
- when inside a venv (almost always) go 'python' the old way, because 'py' is unreliable here, *unless* you manually check the shebang of your scripts before you execute them. 

Which practically means - almost never use 'py'.
Is this the correct way to go?

> - when inside a venv (almost always) go 'python' the old way, because 'py' is unreliable here, *unless* you manually check the shebang of your scripts before you execute them. 

No. When inside a venv:

- If you want to use the interactive interpreter, use 'py'.
- If you want to execute a script, use a shebang of #!/usr/bin/env python and then use `py myscript.py`

You should use /usr/bin/python[X[.Y]] shebangs specifically when you want to use the system Python, and bypass venvs. So that's typically for scripts you've installed in your PATH, not for working scripts in your project. You should never use versioned executable names in shebangs with /usr/bin/env.

Your issues seem to come from an insistence on using versioned interpreter names in /usr/bin/env shebangs, which does not work as you expect. (How it actually works is IMO not very helpful, but not easily fixable without shipping versioned executables, which is an entirely different debate which I don't intend to get into here).

> it's not possible to tell by inspection the version of a Python 
> interpreter.

If getting the version of python[w].exe is ever required, it should be simple for 3.5+, for which python[w].exe has standard file version information with the product version (i.e. GetFileVersionInfo, etc). 

For older versions you could walk the EXE's import directory, looking for a dependency on pythonXY.dll. First map the executable as data via CreateFileW, CreateFileMapping, and MapViewOfFile. Next get pointers to the IMAGE_NT_HEADERS and the first IMAGE_IMPORT_DESCRIPTOR via ImageNtHeader and ImageDirectoryEntryToData. Finally, walk the array of import descriptors (while the "Characteristics" field isn't 0) to get the "Name" of each DLL dependency. It's a relative address that can be converted to a char pointer via ImageRvaToVa. Using relative addresses allows this to work if a 32-bit application is inspecting a 64-bit image and vice versa.

That said, it's far simpler to just support versioned executable names (e.g. python3.exe, python3.6.exe, python3.6-32.exe, pythonw3.exe, pythonw3.6.exe, pythonw3.6-32.exe). Even if we don't install links/copies with these names, I don't see the harm in allowing the launcher to look for them. Users can create the links manually; I've seen people on SO that do this. I'm uploading a patch that implements this for "env" shebangs.

Paul: 
> When inside a venv:
- If you want to execute a script, use a shebang of #!/usr/bin/env python and then use `py myscript.py`

Yes, I'm totally on board with this - that is, as far as I run my own scripts. I just wonder, what if one day I pip install someone else's work, and then bang!, 'py' is just working differently now, for a non-local reason, and without alerting me. 

It's true that all major libraries use the "right" kind of shebang these days... Still, I find it a bit disturbing nevertheless. 

But thanks for the clarifications, anyways - now 'py' expected behavior is much more clear to me. 


Eryk:
> it's far simpler to just support versioned executable names

Keeping my fingers crossed about this.

> Even if we don't install links/copies with these names, I don't see the harm in allowing the launcher to look for them. Users can create the links manually

True, but It would allow for yet another possible source of confusion I'm afraid. No, if py were to look for versioned executables, then versioned executable should be supplied imo.

If the system doesn't have a "python3.exe" on PATH, then "env python3" will run a registered version, exactly like it currently does. IMO, this is slightly less confusing than the current behavior, which skips searching PATH in this case. We can take it a step further, however. If the launcher can't find the versioned name, then search for "python[w].exe" and determine its version using the approach I outlined above. If it doesn't match the required version, as always the launcher will default to running a registered version.

If the user is operating in a virtual environment they are ring fenced and, presumably, don't wish any other versions of python to be used, (regardless of the SheBang in the files).

Since we don't AFAIK have any possibility of mixed virtual environments ignoring SheBangs should meet the basic requirements of operating inside of a venv.

How about a simple, but brutal approach, of if VIRTUAL_ENV is set in the current environment then always use the virtual environment python and launch with the -x option (ignore first line). This is the equivalent of using a version specifier as well as having a SheBang. This way the wrong python for the script might be executed but that is the responsibility of the person who set up the venv.

> If the user is operating in a virtual environment they are ring fenced and, presumably, don't wish any other versions of python to be used

I'm not sure that's true. I have a number of tools which rely on specific libraries and so I have created venvs for them. I then link the different tool executable scripts from e.g. ~/.virtualenvs/foo_env/bin to ~/bin. Each script, because its shebang references the interpreter in its venv, runs with the correct Python version and libraries available in that venv. Once set up, these scripts are treated by me as "black boxes" in the course of my normal workflow.

When I happen to activate some other venv in the course of my development work, I don't want all these tool scripts to stop working because they try to use the interpreter and libraries installed in that venv, rather than the venv that they live in.

> Since we don't AFAIK have any possibility of mixed virtual environments ignoring SheBangs should meet the basic requirements of operating inside of a venv.

If you want a script to use the current environment, then #!/usr/bin/env python does that. There's no need for special-casing a virtual environment.

In addition, the current behaviour mirrors the Unix behaviour, which is intentional (and beneficial).

I just got hit by this bug and would like to add my thoughts on this.
If you are in an activated venv, no matter if you launch with command python or python3, it will launch the version in venv (version 3.6.1 in this case).
I expect the py-launcher and shebang to work the same way. In fact it works as expected on Linux and '#! /usr/bin/env pyton3' are in fact using the venv version.
This is a pretty major bug that needs to be fixed asap in my opinion.

Sorry I need to clarify.
On Linux both python and python3 works as there is a symlink created from python to python3 in the venv folder.
On Windows only python.exe is created. I copied it to python3.exe. Now I can use python3 script.py to start but py-launcher and shebang still didn't work with '/usr/bin/env python3'.
I expect this to work the same on Windows as it does on Linux.

@Jens Lindgren

I know, pretty annoying right? But see previous answer by Paul here http://bugs.python.org/issue28686#msg287181

> The Unix ability to have 2 different versions of Python on PATH 
> and select which you use based on executable name doesn't exist 
> on Windows, and so there's no equivalent of the 
> Unix "#!/usr/bin/env pythonX[.Y]"

Now if you ask me, I would expect py.exe to handle all common types of shebang you may find in the wild. 
But I assume that the correct answer instead is that "#!/usr/bin/env pythonX[.Y]" is not a portable shebang, and you just should stop using it. If you happen to find such a shebang in someone else's script, file a bug report there.

So we're missing two things to move this forward:

1. A clear specification (and user-facing explanation) of the behaviour of py.exe in the presence of all possible shebangs and PATH settings
2. Someone willing to update the code

Given the current behaviour is described in PEP 397, step one probably requires a new PEP (unless it turns out to be a single sentence explanation, which so far, it has not). I'd be happy to sponsor such a PEP, but I don't feel the need for it and so I'm not inclined to write it myself.

Also worth noting is that the Microsoft Store package of Python *does* include versioned executables (because we're able to provide them without causing excessive clutter on PATH), but it does not include py.exe (because the versioning of that would get broken really quickly).

Just copy/paste a related issue reported in issue39785:

When run a python script with "#!/usr/bin/python" shebang with py.exe, it will always use python2 instead of python3 on Win 10, despite the default being set to py3 already (so does the PATH).

According to https://docs.python.org/3/using/windows.html#shebang-lines, `#!/usr/bin/python` should use the default python, not just python 2.

More interestingly, I can't reproduce the same bug on my another Win 7 machine with similar setup.

Three years later, this problem seems on the way to fix itself (https://xkcd.com/1822/).

Versioned shebangs (and versioned "/env" shebangs) used to be a more prominent issue when you needed a way to tell Python 2 / Python 3 scripts apart. With the sunset of Python 2.7-3.3 almost completed, now we have a reasonably homogeneous block of inter-compatible Pythons.
True, py.exe won't cope very well with versioned shebangs, but this seems to become less and less annoying by the day.

As for the other problem from issue 39785, ie Python 2 still being selected as "default" in deference to some arcane Linux convention that we have been dragging for ten years (even on Windows!)... it would be nice to finally have *that* fixed. 
But then again, having Python 2 installed on your Windows box is becoming increasingly rare. Most users are likely never hitting this weirdness any more. 



In the meantime, now we have yet another way of installing Python on Windows... versioned executables from the Store! With no py.exe! So, who knows, maybe py.exe is the past, versioned executable are the future...

To sum up, while I would certainly add a note to the docs, to clarify that py.exe does not support versioned "/env" shebangs, IMO today the work needed to specify a behaviour for all possible cases, possibly even extending/modifying Linux conventions, is just too much to be worth the effort.