classification
Title: pkgutil doesn't support frozen modules
Type: Stage: resolved
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, eric.snow, iritkatriel, lemburg, martin.panter, ncoghlan
Priority: normal Keywords:

Created on 2012-09-24 17:02 by lemburg, last changed 2020-11-30 10:31 by iritkatriel. This issue is now closed.

Messages (10)
msg171163 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2012-09-24 17:02
pkgutil is used by runpy to run Python modules that are loaded via the -m command line switch.

Unfortunately, this doesn't work for frozen modules, since pkgutil doesn't know how to load their code object (this can be had via imp.get_code_object() for frozen modules).

We found the problem while working on eGenix PyRun (see http://www.egenix.com/products/python/PyRun/) which uses frozen modules extensively. We currently only target Python 2.x, so will have work around the problem with a patch, but Python 3.x still has the same problem.
msg171165 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2012-09-24 17:33
Correction: the helper function is called imp.get_frozen_object().
msg171277 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2012-09-25 14:06
Here's the fix we're applying in pyrun to make -m imports work at least for top-level modules:

--- /home/lemburg/orig/Python-2.7.3/Lib/pkgutil.py      2012-04-10 01:07:30.000000000 +0200
+++ pkgutil.py  2012-09-24 22:53:30.982526065 +0200
@@ -273,10 +273,21 @@ class ImpLoader:
     def is_package(self, fullname):
         fullname = self._fix_name(fullname)
         return self.etc[2]==imp.PKG_DIRECTORY

     def get_code(self, fullname=None):
+        if self.code is not None:
+            return self.code
+        fullname = self._fix_name(fullname)
+        mod_type = self.etc[2]
+        if mod_type == imp.PY_FROZEN:
+            self.code = imp.get_frozen_object(fullname)
+            return self.code
+        else:
+            return self._get_code(fullname)
+
+    def _get_code(self, fullname=None):
         fullname = self._fix_name(fullname)
         if self.code is None:
             mod_type = self.etc[2]
             if mod_type==imp.PY_SOURCE:
                 source = self.get_source(fullname)

This makes runpy work for top-level frozen modules, but it's really only partial solution, since pkgutil would need to get such support in more places.

We also found that for some reason, runpy/pkgutil does not work for frozen package imports, e.g. wsgiref.util. The reasons for this appear to be deeper than just in the pkgutil module. We don't have a solution for this yet. It is also not clear whether the problem still exists in Python 3.x. The __path__ attribute of frozen modules was changed in 3.0 to be a list like for all other modules, however, applying that change to 2.x lets runpy/pkgutil fail altogether (not even the above fix works anymore).
msg171282 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-09-25 14:29
Can you confirm this problem still exists on 3.3? The pkgutil emulation isn't used by runpy any more - with the migration to importlib, the interface that runpy invokes fails outright if no loader is found rather than falling back to the emulation (we only retained the emulation for backwards compatibility - it's a public API, so others may be using it directly).

I have a feeling that there may still be a couple of checks which are restricted to PY_SOURCE and PY_COMPILED that really should be allowing PY_FROZEN as well.
msg171292 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2012-09-25 15:41
Nick Coghlan wrote:
> 
> Nick Coghlan added the comment:
> 
> Can you confirm this problem still exists on 3.3? The pkgutil emulation isn't used by runpy any more - with the migration to importlib, the interface that runpy invokes fails outright if no loader is found rather than falling back to the emulation (we only retained the emulation for backwards compatibility - it's a public API, so others may be using it directly).

That's difficult to test, since the Tools/freeze/ tool no longer works
in Python 3.3. I'll open a separate issue for that.

> I have a feeling that there may still be a couple of checks which are restricted to PY_SOURCE and PY_COMPILED that really should be allowing PY_FROZEN as well.

Same here.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 25 2012)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2012-10-29: PyCon DE 2012, Leipzig, Germany ...            34 days to go
2012-10-23: Python Meeting Duesseldorf ...                 28 days to go
2012-09-25: Released mxODBC 3.2.1 ...             http://egenix.com/go31
2012-09-18: Released mxODBC Zope DA 2.1.0 ...     http://egenix.com/go32

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
msg265524 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-05-14 12:18
FWIW Python 3.3 is almost a distant memory, but I think it works properly:

$ wine c:/Python33/python.exe -m __hello__
Hello world!
$ wine c:/Python33/python.exe -m __phello__.spam
Hello world!
Hello world!

On Python 2, it does fail. Is this what the original problem was?

$ python2.7 -m __hello__
/sbin/python2.7: No code object available for __hello__
[Exit 1]
$ python2.7 -c 'import __hello__'
Hello world...

As I see it, this only needs to be fixed in Python 2. Or is it not applicable to 2?
msg265527 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2016-05-14 12:47
We're not supporting Python 3.3 with PyRun, but I can confirm that it works fine in Python 3.4 and 3.5.

In Python 2.7, it's still broken, as you can test with

pyrun2.7 -m wsgiref.simple_server
msg265529 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2016-05-14 13:42
Side note, as already indicated by Nick: pkgutil may well still not support frozen modules in Python 3.4 and 3.5, since runpy, which pyrun uses, does not rely on pkgutil in Python 3.4 and 3.5.
msg265530 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-05-14 13:58
I know I had a bit of trouble adding support to iter_modules() and related functions, see Issue 25533.

Also, Issue 21749 about ImpLoader came up in my search.
msg382141 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2020-11-30 10:31
Python 2 issue.
History
Date User Action Args
2020-11-30 10:31:56iritkatrielsetstatus: open -> closed

nosy: + iritkatriel
messages: + msg382141

resolution: out of date
stage: resolved
2020-01-29 00:17:15brett.cannonsetnosy: - brett.cannon
2016-05-14 13:58:00martin.pantersetmessages: + msg265530
2016-05-14 13:42:10lemburgsetmessages: + msg265529
2016-05-14 12:47:42lemburgsetmessages: + msg265527
versions: + Python 2.7, - Python 3.4
2016-05-14 12:18:20martin.pantersetnosy: + martin.panter
messages: + msg265524
2012-11-13 06:27:34eric.snowsetnosy: + eric.snow
2012-09-25 15:41:13lemburgsetmessages: + msg171292
2012-09-25 14:29:35ncoghlansetmessages: + msg171282
2012-09-25 14:06:20lemburgsetmessages: + msg171277
2012-09-25 13:37:21pitrousetnosy: + ncoghlan
2012-09-25 13:34:25brett.cannonsetversions: + Python 3.4, - Python 3.2, Python 3.3
2012-09-25 00:05:15vstinnersetnosy: + brett.cannon
2012-09-24 18:01:56Arfreversetnosy: + Arfrever
2012-09-24 17:33:53lemburgsetmessages: + msg171165
2012-09-24 17:02:40lemburgcreate