classification
Title: pkgutil ImpLoader does not support frozen modules
Type: Stage:
Components: Distutils, Library (Lib) Versions: Python 3.5, Python 3.4, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Jim.Jewett, dstufft, eric.araujo, lemburg
Priority: normal Keywords: patch

Created on 2014-06-13 14:08 by lemburg, last changed 2014-06-13 17:34 by lemburg.

Files
File name Uploaded Description Edit
pkgutil-frozen-modules-support.patch lemburg, 2014-06-13 17:34 Updated version review
Messages (4)
msg220442 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-06-13 14:08
This leads to problems when using runpy, since it relies on pkgutil to find importers.

In Python 2, ImpLoader is used by ImpImporter which is used as fallback importer by get_importer(). get_importer() is used by runpy to implement e.g. the -m option. As a result, -m doesn't work with frozen modules.

In Python 3, ImpLoader still exists, but is deprecated. get_importer() also no longer falls back to the ImpImporter as default, so the situation is less bothersome.

Here's a patch for Python 3.4.1 which shows how to add support for frozen modules. The same patch also works in Python 2.6 and 2.7.
msg220465 - (view) Author: Jim Jewett (Jim.Jewett) Date: 2014-06-13 16:49
As best I can tell, this renames the original get_code to _get_code, and then delegates to it after handling the imp.PY_FROZEN case ... why not just add imp.PY_FROZEN to the if/elif chain in the original method?

I also note that the call to self._fix_name doesn't do anything helpful unless it is overridden in a subclass -- was this patch really intended as a recipe for subclasses, that avoided super for backwards compatibility?

If it is intended to go into the main branch, then someone has to decide whether it is OK to enhance despite being deprecated.

If it is just an example, then it should probably be closed with an appropriate status.  I'm guessing "not a bug", but that isn't quite right either.
msg220470 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-06-13 17:19
On 13.06.2014 18:49, Jim Jewett wrote:
> 
> As best I can tell, this renames the original get_code to _get_code, and then delegates to it after handling the imp.PY_FROZEN case ... why not just add imp.PY_FROZEN to the if/elif chain in the original method?

Of course, a proper patch would work that way and I can submit one
if there's demand. The attached patch was the result of a programmatic
patching system that uses search & replace. That's why it uses this
awkward style.

> I also note that the call to self._fix_name doesn't do anything helpful unless it is overridden in a subclass -- was this patch really intended as a recipe for subclasses, that avoided super for backwards compatibility?

It does: if fullname is None, it returns self.fullname.

> If it is intended to go into the main branch, then someone has to decide whether it is OK to enhance despite being deprecated.
> 
> If it is just an example, then it should probably be closed with an appropriate status.  I'm guessing "not a bug", but that isn't quite right either.

It is deprecated in 3.4, but not in 2.7 and for 2.7 I consider this
a bug, since pkgutil claims to support frozen packages in the
ImpImporter doc-string.
msg220471 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-06-13 17:34
Here's an updated patch for Python 3.4.1 that doesn't use the awkward style :-)
History
Date User Action Args
2014-06-13 17:34:58lemburgsetfiles: - pkgutil-frozen-modules-support.patch
2014-06-13 17:34:28lemburgsetfiles: + pkgutil-frozen-modules-support.patch

messages: + msg220471
2014-06-13 17:19:39lemburgsetmessages: + msg220470
2014-06-13 16:49:28Jim.Jewettsetnosy: + Jim.Jewett
messages: + msg220465
2014-06-13 14:08:31lemburgcreate