Trying to run a pyz file using the embedded distribution for 3.6.0, I get an error "Could not import runpy module".

To reproduce, see below:

>type .\__main__.py
print('Hello, world')
>zip test.pyz __main__.py
  adding: __main__.py (172 bytes security) (stored 0%)
>unzip .\python-3.6.0-embed-amd64.zip
...
>.\python.exe .\test.pyz
Could not import runpy module
ModuleNotFoundError: No module named 'runpy'

Running with the standard interpreter works fine:

>py -V
Python 3.6.0
>py .\test.pyz
Hello, world

I just checked, and 3.6.0b1 (the only prerelease version I had available) has the same problem. 3.5.2 works fine.

I thought I'd had similar code working during the beta cycle, but I can't demonstrate that any more, so maybe my recollection is wrong :-(

Confirmed that it works with alpha 2, 3 and 4. But fails with beta 1 and the release version.

Does running with -v provide any more hints?

I'm also interested in whether the alphas work when you don't have a full install on the same machine. PC/getpathp.c changed for b1. But that's probably less important.

Perhaps runpy just never made it into the python36.zip file?

Sorry I should have thought of trying -v. The output (included below) doesn't seem to offer many hints, though. runpy.pyc is in python36.zip, I checked that.

I'll see if I can find a machine without Python installed to test that case.

>.\python.exe -v .\test.pyz
import _frozen_importlib # frozen
import _imp # builtin
import sys # builtin
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'nt' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import 'winreg' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
# installed zipimport hook
# zipimport: found 609 names in 'C:\\Work\\Scratch\\test\\python36.zip'
import 'zlib' # <class '_frozen_importlib.BuiltinImporter'>
# zipimport: zlib available
# zipimport: zlib available
# zipimport: zlib available
# zipimport: zlib available
import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
import codecs # loaded from Zip C:\Work\Scratch\test\python36.zip\codecs.pyc
# zipimport: zlib available
# zipimport: zlib available
import encodings.aliases # loaded from Zip C:\Work\Scratch\test\python36.zip\encodings\aliases.pyc
import encodings # loaded from Zip C:\Work\Scratch\test\python36.zip\encodings\__init__.pyc
# zipimport: zlib available
# zipimport: zlib available
import encodings.utf_8 # loaded from Zip C:\Work\Scratch\test\python36.zip\encodings\utf_8.pyc
import '_signal' # <class '_frozen_importlib.BuiltinImporter'>
# zipimport: zlib available
# zipimport: zlib available
import encodings.latin_1 # loaded from Zip C:\Work\Scratch\test\python36.zip\encodings\latin_1.pyc
# zipimport: zlib available
# zipimport: zlib available
# zipimport: zlib available
# zipimport: zlib available
# zipimport: zlib available
# zipimport: zlib available
import _weakrefset # loaded from Zip C:\Work\Scratch\test\python36.zip\_weakrefset.pyc
import abc # loaded from Zip C:\Work\Scratch\test\python36.zip\abc.pyc
import io # loaded from Zip C:\Work\Scratch\test\python36.zip\io.pyc
Python 3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 08:06:12) [MSC v.1900 64 bit (AMD64)] on win32
# zipimport: zlib available
# zipimport: zlib available
import encodings.cp437 # loaded from Zip C:\Work\Scratch\test\python36.zip\encodings\cp437.pyc
# zipimport: found 1 names in '.\\test.pyz'
Could not import runpy module
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 948, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'runpy'
# clear builtins._
# clear sys.path
# clear sys.argv
# clear sys.ps1
# clear sys.ps2
# clear sys.last_type
# clear sys.last_value
# clear sys.last_traceback
# clear sys.path_hooks
# clear sys.path_importer_cache
# clear sys.meta_path
# clear sys.__interactivehook__
# clear sys.flags
# clear sys.float_info
# restore sys.stdin
# restore sys.stdout
# restore sys.stderr
# cleanup[2] removing builtins
# cleanup[2] removing sys
# cleanup[2] removing _frozen_importlib
# cleanup[2] removing _imp
# cleanup[2] removing _warnings
# cleanup[2] removing _thread
# cleanup[2] removing _weakref
# cleanup[2] removing _frozen_importlib_external
# cleanup[2] removing _io
# cleanup[2] removing marshal
# cleanup[2] removing nt
# cleanup[2] removing winreg
# cleanup[2] removing zipimport
# cleanup[2] removing zlib
# cleanup[2] removing encodings
# destroy encodings
# cleanup[2] removing codecs
# cleanup[2] removing _codecs
# cleanup[2] removing encodings.aliases
# cleanup[2] removing encodings.utf_8
# cleanup[2] removing _signal
# cleanup[2] removing __main__
# destroy __main__
# cleanup[2] removing encodings.latin_1
# cleanup[2] removing io
# destroy io
# cleanup[2] removing abc
# destroy abc
# cleanup[2] removing _weakrefset
# destroy _weakrefset
# cleanup[2] removing encodings.cp437
# destroy zipimport
# destroy zlib
# destroy _signal
# cleanup[3] wiping _frozen_importlib
# destroy _frozen_importlib_external
# cleanup[3] wiping _imp
# cleanup[3] wiping _warnings
# cleanup[3] wiping _thread
# cleanup[3] wiping _weakref
# cleanup[3] wiping _io
# cleanup[3] wiping marshal
# cleanup[3] wiping nt
# cleanup[3] wiping winreg
# cleanup[3] wiping codecs
# cleanup[3] wiping _codecs
# cleanup[3] wiping encodings.aliases
# cleanup[3] wiping encodings.utf_8
# cleanup[3] wiping encodings.latin_1
# cleanup[3] wiping encodings.cp437
# cleanup[3] wiping sys
# cleanup[3] wiping builtins
# destroy _imp
# destroy io
# destroy _warnings
# destroy marshal
# destroy nt
# destroy _thread
# destroy _weakref
# destroy winreg
# destroy _frozen_importlib

I just tried it and it makes no difference.

Omitting the "._pth" file seems to fix it, but apart from issue29326 (benign) sys.path is fine.

Still digging, but don't worry about testing on other machines.

Found it in Modules/main.c in RunMainFromImporter():

    /* argv0 is usable as an import source, so put it in sys.path[0]
       and import __main__ */
    sys_path = PySys_GetObject("path");
    if (sys_path == NULL) {
        PyErr_SetString(PyExc_RuntimeError, "unable to get sys.path");
        goto error;
    }
    if (PyList_SetItem(sys_path, 0, argv0)) {
        argv0 = NULL;
        goto error;
    }
    Py_INCREF(argv0);

When running with a ._pth file, we force the -I option, which removes the empty entry at sys.path[0]. Instead, it becomes the path to the .zip file with the standard library, so when RunMainFromImporter overwrites it blindly, it's cutting out the only hope it has of finding runpy.

You can see this in a normal install by doing:

    py -I -c "import sys; print(sys.path)"
    [correct output]

    py -Ii test.pyz
    [output from test.pyz]
    >>> import sys; sys.path
    [incorrect output]

So we need to stop blindly overwriting sys.path[0] here. I'm not sure what the best approach would be, but maybe Nick has a preference? Perhaps we should pass the zip file path into runpy._run_module_as_main so it can initialize __path__ with it rather than making a global change? Or maybe an insert rather than a set is the right way and I'm over-thinking this.

Nice! Thanks for finding this. I don't suppose there's any chance this would qualify as a bugfix for 3.6.1? I've been holding off on working on https://github.com/pfmoore/pylaunch until 3.6 because 3.5 doesn't handle being in a subdirectory very well. It'd be a shame if this meant I could only really support 3.7+

But it is a pretty minor use case, so I'll live either way.

FWIW, I'm inclined to think we shouldn't be blindly overwriting things, at the very least we should check that what we're overwriting is what we expect. But my knowledge of this part of the interpreter is pretty much zero, so I'll defer to the experts.

I'd say it definitely qualifies as a bug fix, even in 3.5 (which repros), assuming we don't break any existing APIs in the process.

I think this is just a straight-up interaction bug where we never accounted for the combination of -I with directory and zipfile execution back when -I was added.

From a bug fix perspective, I think the right thing to do in all affected versions would be to add a check to RunMainFromImporter such that it does an insert rather than a set when -I is used.

For 3.7+, it may make sense to change the signature of runpy._run_module_as_main to accept the path entry as Steve suggests, as that could also resolve the quirk where we'll run a __main__ module from *anywhere* on sys.path, rather than only from the given path entry.

The core of the current approach was implemented back in the 2.6 time frame, and then tweaked a bit in 2.7, and hence predates all the niceties offered by the importlib API.

However, the latter change would be a separate RFE rather than a bug fix.

New changeset c6506f759db1 by Steve Dower in branch '3.5':
Issue #29319: Prevent RunMainFromImporter overwriting sys.path[0].
https://hg.python.org/cpython/rev/c6506f759db1

New changeset 0965e2967056 by Steve Dower in branch '3.6':
Issue #29319: Prevent RunMainFromImporter overwriting sys.path[0].
https://hg.python.org/cpython/rev/0965e2967056

New changeset d0f28ee3affe by Steve Dower in branch 'default':
Issue #29319: Prevent RunMainFromImporter overwriting sys.path[0].
https://hg.python.org/cpython/rev/d0f28ee3affe

That change fixes overwriting sys.path[0], the new logic is essentially:

try:
    sys_path0 = sys.path[0]
except:
    sys.path.append(argv0)
else:
    if sys_path0:
        sys.path.insert(0, argv0)
    else:
        sys.path[0] = argv0

I'm leaving this open for the better API fix for 3.7.

New changeset baa426f44b86fb146dc17178f61b53ed5ffb08f0 by Steve Dower in branch '3.5':
Issue #29319: Prevent RunMainFromImporter overwriting sys.path[0].
https://github.com/python/cpython/commit/baa426f44b86fb146dc17178f61b53ed5ffb08f0

New changeset baa426f44b86fb146dc17178f61b53ed5ffb08f0 by Steve Dower in branch 'master':
Issue #29319: Prevent RunMainFromImporter overwriting sys.path[0].
https://github.com/python/cpython/commit/baa426f44b86fb146dc17178f61b53ed5ffb08f0

New changeset 5003ad354c6ffd33fa71537a0b33c7f0e17d0093 by Steve Dower in branch 'master':
Issue #29319: Prevent RunMainFromImporter overwriting sys.path[0].
https://github.com/python/cpython/commit/5003ad354c6ffd33fa71537a0b33c7f0e17d0093

New changeset a4c1827b15cc8309b4b375db33ee2d85c49475c8 by Steve Dower in branch 'master':
Issue #29319: Prevent RunMainFromImporter overwriting sys.path[0].
https://github.com/python/cpython/commit/a4c1827b15cc8309b4b375db33ee2d85c49475c8

New changeset baa426f44b86fb146dc17178f61b53ed5ffb08f0 by Steve Dower in branch '3.6':
Issue #29319: Prevent RunMainFromImporter overwriting sys.path[0].
https://github.com/python/cpython/commit/baa426f44b86fb146dc17178f61b53ed5ffb08f0

New changeset 5003ad354c6ffd33fa71537a0b33c7f0e17d0093 by Steve Dower in branch '3.6':
Issue #29319: Prevent RunMainFromImporter overwriting sys.path[0].
https://github.com/python/cpython/commit/5003ad354c6ffd33fa71537a0b33c7f0e17d0093

I'm wondering if I'm experiencing this same issue.
In a simple directory with a foo.py and a bar.py where foo tries to import from bar I cannot get it to work with the embeddable 3.6.0 zip, but the standard 3.6.0 that gets "installed" works fine.  Also 3.5.3 works fine

C:\Users\eric\Desktop\wtf>more foo.py
from bar import bar

print(bar('hi'))

C:\Users\eric\Desktop\wtf>more bar.py
def bar(s):
        return s.upper()

C:\Users\eric\Desktop\wtf>C:\Users\eric\Downloads\python-3.5.3-embed-amd64\python.exe foo.py
HI

C:\Users\eric\Desktop\wtf>C:\Users\eric\Downloads\python-3.6.0-embed-amd64\python.exe foo.py
Traceback (most recent call last):
  File "foo.py", line 1, in <module>
    from bar import bar
ModuleNotFoundError: No module named 'bar'

Eric - that sounds like the same issue. Can you test with 3.6.1rc1 to see if it is fixed for you?

I can confirm that this is NOT fixed in 3.6.1rc1 embeddable zip.

This is extremely easy to reproduce.  Look at the contents of foo.py and bar.py.  Just throw them in the same directory and try to run C:\path\to\extracted\python.exe foo.py

Oh, well that's by design. Neither the current working directory nor the directory of the initial script are in sys.path by default - they need to be added explicitly.

The intent of this distro is that you know exactly where relative to the executable your modules are, that won't change after you release your app, and that you want to avoid other code using your private (embedded) copy of Python.

Confirmed that 3.6.1rc1 fixes the issue in my original use case.

Issue 29723 is a follow up issue to this one, where these changes revealed some latent defects in how sys.path[0] was being initialised in general for directory and zipfile execution (those defects mean the change committed here adds an extra directory to sys.path when *not* running in isolated mode).

Paul, are you in a position to rebuild 3.6 with the changes from https://github.com/python/cpython/pull/575 and indicate whether or not that still fixes the original problem?

Nick (or Steve) can you explain how I'd do that?

I have a git checkout of the 3.6 branch that I can build. But, how do I merge in the changes from https://github.com/python/cpython/pull/575? The PR seems to be against master, so I'm not sure how to apply it to another branch (I'm not even sure how to apply it to my copy of master, TBH!) Whatever I do need to do doesn't seem to be in the devguide, although I may be missing it. Maybe there should be a section in there on how to do common tasks like this?

If you append ".patch" to a GitHub PR URL, it will give you a patch file that can be applied to any branch with "git apply".

In this case, the rendered patch is at https://patch-diff.githubusercontent.com/raw/python/cpython/pull/575.patch

OK, cool, thanks. I was sort of hoping for a way to just pull&merge direct from the PR (patches on Windows tend to be fiddly due to EOL issues), but I can go with old-style :-)

Yes, with that patch applied it still works fine (I copied python3.dll and python36.dll from the build into the existing embedded distribution, which I guess is all I needed to do - I don't know how to build a full embedded distribution file, but I can't see that would make any difference).

That will work fine. Thanks for checking

The process for pulling someone's PR into your own repo is roughly branch then pull from the repo sending the PR. Github should show instructions for this under m hidden behind a "merge manually" button (though you want to skip the final push that would complete the merge, obviously).

Thanks for that Steve. I had a recollection that there's a way of referencing the PR itself as a branch within the main repo (I guess it must *be* a branch, as how otherwise would github be able to do things like get Travis to build it?) but I don't recall the details.

Hmm, a bit of googling later I found https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch origin pull/ID/head:BRANCHNAME

And indeed that works - I checked out Nick's patch like that. I'd still need to merge it into the 3.6 branch, which is another set of git commands I don't yet know (cherry-pick, maybe?)

With the fix for issue 29723 merged, this should be properly resolved now (and the fix available with 3.6.1).

New changeset 75345c552d0889f4f63039d6063f371846c8f41f by Ned Deily (Nick Coghlan) in branch '3.6':
[3.6] bpo-29723: Consistently configure sys.path[0] (#636)
https://github.com/python/cpython/commit/75345c552d0889f4f63039d6063f371846c8f41f

New changeset c60948464fb0ec116ea227f6bce8a4bb8fb75257 by Nick Coghlan in branch '3.6':
[3.6] bpo-29723: Consistently configure sys.path[0] (#636)
https://github.com/python/cpython/commit/c60948464fb0ec116ea227f6bce8a4bb8fb75257

New changeset d2977a3ae2cc6802921b1e3b6e9d13fcfbda872d by Nick Coghlan in branch 'master':
bpo-29723: Consistently configure sys.path[0] (#575)
https://github.com/python/cpython/commit/d2977a3ae2cc6802921b1e3b6e9d13fcfbda872d

Tweaking metadata to make it clear Py3.5 was also updated.

New changeset 891c91d8d38848377a9f475242507510873eb9c3 by larryhastings (Nick Coghlan) in branch '3.5':
[3.5] bpo-32551: Consistently configure sys.path[0] (#5197)
https://github.com/python/cpython/commit/891c91d8d38848377a9f475242507510873eb9c3