pkgutil is used by runpy to run Python modules that are loaded via the -m command line switch.

Unfortunately, this doesn't work for frozen modules, since pkgutil doesn't know how to load their code object (this can be had via imp.get_code_object() for frozen modules).

We found the problem while working on eGenix PyRun (see http://www.egenix.com/products/python/PyRun/) which uses frozen modules extensively. We currently only target Python 2.x, so will have work around the problem with a patch, but Python 3.x still has the same problem.

Correction: the helper function is called imp.get_frozen_object().

Here's the fix we're applying in pyrun to make -m imports work at least for top-level modules:

--- /home/lemburg/orig/Python-2.7.3/Lib/pkgutil.py      2012-04-10 01:07:30.000000000 +0200
+++ pkgutil.py  2012-09-24 22:53:30.982526065 +0200
@@ -273,10 +273,21 @@ class ImpLoader:
     def is_package(self, fullname):
         fullname = self._fix_name(fullname)
         return self.etc[2]==imp.PKG_DIRECTORY

     def get_code(self, fullname=None):
+        if self.code is not None:
+            return self.code
+        fullname = self._fix_name(fullname)
+        mod_type = self.etc[2]
+        if mod_type == imp.PY_FROZEN:
+            self.code = imp.get_frozen_object(fullname)
+            return self.code
+        else:
+            return self._get_code(fullname)
+
+    def _get_code(self, fullname=None):
         fullname = self._fix_name(fullname)
         if self.code is None:
             mod_type = self.etc[2]
             if mod_type==imp.PY_SOURCE:
                 source = self.get_source(fullname)

This makes runpy work for top-level frozen modules, but it's really only partial solution, since pkgutil would need to get such support in more places.

We also found that for some reason, runpy/pkgutil does not work for frozen package imports, e.g. wsgiref.util. The reasons for this appear to be deeper than just in the pkgutil module. We don't have a solution for this yet. It is also not clear whether the problem still exists in Python 3.x. The __path__ attribute of frozen modules was changed in 3.0 to be a list like for all other modules, however, applying that change to 2.x lets runpy/pkgutil fail altogether (not even the above fix works anymore).

Can you confirm this problem still exists on 3.3? The pkgutil emulation isn't used by runpy any more - with the migration to importlib, the interface that runpy invokes fails outright if no loader is found rather than falling back to the emulation (we only retained the emulation for backwards compatibility - it's a public API, so others may be using it directly).

I have a feeling that there may still be a couple of checks which are restricted to PY_SOURCE and PY_COMPILED that really should be allowing PY_FROZEN as well.

Nick Coghlan wrote:
> 
> Nick Coghlan added the comment:
> 
> Can you confirm this problem still exists on 3.3? The pkgutil emulation isn't used by runpy any more - with the migration to importlib, the interface that runpy invokes fails outright if no loader is found rather than falling back to the emulation (we only retained the emulation for backwards compatibility - it's a public API, so others may be using it directly).

That's difficult to test, since the Tools/freeze/ tool no longer works
in Python 3.3. I'll open a separate issue for that.

> I have a feeling that there may still be a couple of checks which are restricted to PY_SOURCE and PY_COMPILED that really should be allowing PY_FROZEN as well.

Same here.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 25 2012)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2012-10-29: PyCon DE 2012, Leipzig, Germany ...            34 days to go
2012-10-23: Python Meeting Duesseldorf ...                 28 days to go
2012-09-25: Released mxODBC 3.2.1 ...             http://egenix.com/go31
2012-09-18: Released mxODBC Zope DA 2.1.0 ...     http://egenix.com/go32

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

FWIW Python 3.3 is almost a distant memory, but I think it works properly:

$ wine c:/Python33/python.exe -m __hello__
Hello world!
$ wine c:/Python33/python.exe -m __phello__.spam
Hello world!
Hello world!

On Python 2, it does fail. Is this what the original problem was?

$ python2.7 -m __hello__
/sbin/python2.7: No code object available for __hello__
[Exit 1]
$ python2.7 -c 'import __hello__'
Hello world...

As I see it, this only needs to be fixed in Python 2. Or is it not applicable to 2?

We're not supporting Python 3.3 with PyRun, but I can confirm that it works fine in Python 3.4 and 3.5.

In Python 2.7, it's still broken, as you can test with

pyrun2.7 -m wsgiref.simple_server

Side note, as already indicated by Nick: pkgutil may well still not support frozen modules in Python 3.4 and 3.5, since runpy, which pyrun uses, does not rely on pkgutil in Python 3.4 and 3.5.

I know I had a bit of trouble adding support to iter_modules() and related functions, see Issue 25533.

Also, Issue 21749 about ImpLoader came up in my search.

Python 2 issue.