See http://mail.python.org/pipermail/python-dev/2012-May/119703.html

Two competing proposals:

- Benjamin: "Perhaps freeze_importlib.py could be rewritten in C, so
importlib could be recompiled when the compiler changes?"

- Martin: "Or we support bootstrapping from the source file, e.g. with an
environment variable BOOTSTRAP_PY which points to the _bootstrap.py
source"

I like MvL's approach if we can make it work - then we can set up the importlib.h regeneration to automatically bootstrap itself from the source file and avoid having to add another binary to the build process.

There are some additional challenges potentially posed by suggestions like http://bugs.python.org/issue10399, which would allow the compiler itself to use Python extensions.

However, those could be overcome by requiring that the compiler support running with any such extensions disabled.

Two more possible APIs:
  A -X option: "-Xbootstrap=Lib/importlib/_bootstrap.py"
  Allow frozen modules to be frozen as *source* rather than as a PYC file, then freeze importlib._bootstrap as source rather than as bytecode.

Here is a patch for the "separate C executable" approach. There are a couple of small complications in order to avoid any circular dependency (either at the Makefile level, or at the _frozen_importlib / Py_Initialize level).

The proposed "-X" or BOOTSTRAP_PY approaches would be difficult, since there doesn't appear to be an easy way to generate a given file (the Python interpreter) twice in a Makefile dependency chain: the first time using the stale importlib, the second with the fresh importlib.

Nice patch Antoine!  The method of partitioning the object sets into frozen and
non-frozen is nice.

> The proposed "-X" or BOOTSTRAP_PY approaches would be difficult, since there
> doesn't appear to be an easy way to generate a given file (the Python
> interpreter) twice in a Makefile dependency chain: the first time using the
> stale importlib, the second with the fresh importlib.

I experimented with the env var approach and came to the exact same conclusion
-- you would have to build the interpreter twice.

This patch looks good to me (I did have to rebase it against trunk, though;
it doesn't apply cleanly right now).  I did some basic regression testing and ran
it through the original scenario from issue12370 that got us thinking this problem.
I saw no issues.  Nice work.

Looks like a solid improvement to me.

One minor naming quibble is that "FROZENLESS_LIBRARY_OBJS" hurts my brain - would you mind switching it to something like "LIBRARY_OBJS_OMIT_FROZEN"?

IIUC, this patch will cause importlib.h to always be built when building from source, since _freeze_importlib will be built.

Is it then the plan to drop importlib.h from version control? If so, the Windows build process would need to be adjusted as well (with a separate project that is a prerequisite for pythoncore).

> IIUC, this patch will cause importlib.h to always be built when
> building from source, since _freeze_importlib will be built.

Yes indeed. That's not deliberate, but because of how Makefile
dependencies are specified (importlib.h needs the _freeze_importlib
executable to be rebuilt, but it should really depend on the
_freeze_importlib.c timestamp). Do you have an idea to avoid that?

> Is it then the plan to drop importlib.h from version control?

No.

> Yes indeed. That's not deliberate, but because of how Makefile
> dependencies are specified (importlib.h needs the _freeze_importlib
> executable to be rebuilt, but it should really depend on the
> _freeze_importlib.c timestamp). Do you have an idea to avoid that?

Recursive make invocation may help. Before running _freeze_importlib,
add '$(MAKE) _freeze_importlib'.

New patch incorporating Martin's and Nick's suggestions.

Is this ready to go in before beta1?

New changeset c3616595dada by Antoine Pitrou in branch 'default':
Issue #14928: Fix importlib bootstrap issues by using a custom executable (Modules/_freeze_importlib) to build Python/importlib.h.
http://hg.python.org/cpython/rev/c3616595dada

Should be ok now.