Title: Cannot build importlib.h on Windows
Type: Stage:
Components: Build, Windows Versions: Python 3.3
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: amaury.forgeotdarc, brian.curtin, jkloth, kristjan.jonsson, loewis, pitrou, python-dev, sbt
Priority: release blocker Keywords: patch

Created on 2012-07-23 06:21 by amaury.forgeotdarc, last changed 2012-07-28 20:05 by loewis. This issue is now closed.

File name Uploaded Description Edit
_freeze_importlib.patch kristjan.jonsson, 2012-07-27 11:40 review
Messages (14)
msg166209 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2012-07-23 06:21
On Windows, the _freeze_importlib tool is not built, so it's not possible to refresh the file Python/importlib.h, which makes development on the importlib very difficult on Windows.

The Makefile contains the rules below, it's probably necessary to implement a similar behavior in a new build project:

Modules/_freeze_importlib: Modules/_freeze_importlib.o $(LIBRARY_OBJS_OMIT_FROZEN)
	$(LINKCC) $(PY_LDFLAGS) -o $@ Modules/_freeze_importlib.o $(LIBRARY_OBJS_OMIT_FROZEN) $(LIBS) $(MODLIBS) $(SYSLIBS) $(LDLAST)

Python/importlib.h: $(srcdir)/Lib/importlib/ Modules/_freeze_importlib.c
	$(MAKE) Modules/_freeze_importlib
	./Modules/_freeze_importlib \
		$(srcdir)/Lib/importlib/ Python/importlib.h
msg166425 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2012-07-25 20:36
shouldn't be too hard to add, I'll give it a go soon.
msg166554 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2012-07-27 11:40
Here's a patch.  A new project, _freeze_importlib, has the necessary prerequisites and runs the executable, generating importlib.h, as a post-build step.

I'm not sure if we should make this project a pre-requisite of any other projects, though
msg166560 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2012-07-27 12:51
IMO dependencies is not a big issue here.  In the worst case, the developer can build a second time...

I think your patch won't work on Unix machines: _freeze_importlib.c is linked with all Python files *except* frozen.c which defines PyImport_FrozenModules.  #ifdef should be used IMO.

Also, I noticed that the marshalled code is different. The first difference a 32bit vs. 64bit issue: 0xFFFFFFFF is marshalled as a TYPE_INT64 on 64bit platforms, but as a TYPE_LONG on 32bit platforms.  See issue15466.
msg166582 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2012-07-27 16:54
Thank you.  I see how I must change the linkage on PC Only, although I wonder why tuen unix version isn't simply doing something similar..

The marshaled code was indeed created using 64 bits.  I was unsure whether to include that file as part of the patch, perhaps I shouldn't.

I'll wait for an update for the 64 bit issue.
msg166587 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2012-07-27 17:26
(Sorry for the previous message, it was wrong, and I also pressed the wrong button...)

A Unix Makefile cannot stand circular dependencies, so _freezeimportlib.c cannot depend on frozen.o.  OTOH it's not simple to have almost-similar list of files on VS projects (here we need all-of-python-but-frozen.c), so your solution is not bad either.
Please check that python33.dll is not always rebuilt each time you hit F7, though.
msg166589 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-07-27 17:41
> I'll wait for an update for the 64 bit issue.

I don't think it's necessary, it's a pretty minor issue since the generated bytecode is portable anyway.
msg166674 - (view) Author: Roundup Robot (python-dev) Date: 2012-07-28 18:47
New changeset fe29a657bde9 by Martin v. Löwis in branch 'default':
Issue #15431: Add _freeze_importlib project to regenerate importlib.h on Windows.
msg166675 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-07-28 18:48
Thanks for the patch, committed with slight modifications.
msg166676 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2012-07-28 18:53
Comments about PyImport_FrozenModules linkage have not been addressed.  Now the build is failing on all Unix machines...
msg166684 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-07-28 19:53
To be clear, the build fails when importlib needs to be frozen again:

$ touch Lib/importlib/ 
$ make
make Modules/_freeze_importlib
make[1] : on entre dans le répertoire « /home/antoine/cpython/default »
gcc -pthread -c -Wno-unused-result -g -O0 -Wall -Wstrict-prototypes    -I. -I./Include    -DPy_BUILD_CORE -o Modules/_freeze_importlib.o Modules/_freeze_importlib.c
gcc -pthread   -o Modules/_freeze_importlib Modules/_freeze_importlib.o Modules/getbuildinfo.o Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar.o Parser/pgen.o Parser/myreadline.o Parser/parsetok.o Parser/tokenizer.o Objects/abstract.o Objects/accu.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/capsule.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/weakrefobject.o Python/_warnings.o Python/Python-ast.o Python/asdl.o Python/ast.o Python/bltinmodule.o Python/ceval.o Python/compile.o Python/codecs.o Python/dynamic_annotations.o Python/errors.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/graminit.o Python/import.o Python/importdl.o Python/marshal.o Python/modsupport.o Python/mystrtoul.o Python/mysnprintf.o Python/peephole.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/pytime.o Python/random.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/dynload_shlib.o   Python/thread.o Modules/config.o Modules/getpath.o Modules/main.o Modules/gcmodule.o  Modules/_threadmodule.o  Modules/signalmodule.o  Modules/posixmodule.o  Modules/errnomodule.o  Modules/pwdmodule.o  Modules/_sre.o  Modules/_codecsmodule.o  Modules/_weakref.o  Modules/_functoolsmodule.o  Modules/operator.o  Modules/_collectionsmodule.o  Modules/itertoolsmodule.o  Modules/_localemodule.o  Modules/_iomodule.o Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o  Modules/zipimport.o  Modules/faulthandler.o  Modules/symtablemodule.o  Modules/xxsubtype.o -lpthread -ldl  -lutil   -lm  
Modules/_freeze_importlib.o: In function `main':
/home/antoine/cpython/default/Modules/_freeze_importlib.c:37: undefined reference to `PyImport_FrozenModules'
Python/import.o: In function `find_frozen':
/home/antoine/cpython/default/Python/import.c:978: undefined reference to `PyImport_FrozenModules'
collect2: ld a retourné 1 code d'état d'exécution
make[1]: *** [Modules/_freeze_importlib] Erreur 1
make[1] : on quitte le répertoire « /home/antoine/cpython/default »
make: *** [Python/importlib.h] Erreur 2
msg166685 - (view) Author: Roundup Robot (python-dev) Date: 2012-07-28 19:55
New changeset 7967cb63e50e by Martin v. Löwis in branch 'default':
Issue #15431: Declare PyImport_FrozenModules conditionally on Unix only.
msg166687 - (view) Author: Roundup Robot (python-dev) Date: 2012-07-28 20:00
New changeset deb421baf671 by Martin v. Löwis in branch 'default':
Issue #15431: Drop _freeze_importlib from all build configurations,
msg166688 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-07-28 20:05
The last commits should address Amaury's concerns. _freeze_importlib must now be built manually, followed by a build of pythoncore.

Interestingly, the builds failed only once. When rebuilding, the builds succeeded as no attempt to build freeze_importlib was made. After the last change to freeze_importlib, they again built it, and that seems to work now.

Tentatively closing the issue again.
Date User Action Args
2012-07-28 20:05:54loewissetstatus: open -> closed

messages: + msg166688
2012-07-28 20:00:28python-devsetmessages: + msg166687
2012-07-28 19:55:51python-devsetmessages: + msg166685
2012-07-28 19:53:49pitrousetpriority: high -> release blocker

messages: + msg166684
2012-07-28 19:36:15loewisunlinkissue14578 dependencies
2012-07-28 18:53:43amaury.forgeotdarcsetstatus: closed -> open

messages: + msg166676
2012-07-28 18:48:26loewissetstatus: open -> closed

nosy: + loewis
messages: + msg166675

resolution: fixed
2012-07-28 18:47:33python-devsetnosy: + python-dev
messages: + msg166674
2012-07-27 17:41:39pitrousetnosy: + pitrou
dependencies: - Python/importlib.h is different on 32bit and 64bit
messages: + msg166589
2012-07-27 17:26:33amaury.forgeotdarcsetmessages: + msg166587
2012-07-27 17:08:17amaury.forgeotdarcsetmessages: - msg166584
2012-07-27 17:07:56amaury.forgeotdarcsetmessages: + msg166584
2012-07-27 16:54:28kristjan.jonssonsetmessages: + msg166582
2012-07-27 12:51:36amaury.forgeotdarcsetdependencies: + Python/importlib.h is different on 32bit and 64bit
messages: + msg166560
2012-07-27 11:40:02kristjan.jonssonsetfiles: + _freeze_importlib.patch
keywords: + patch
messages: + msg166554
2012-07-25 20:36:05kristjan.jonssonsetmessages: + msg166425
2012-07-25 20:31:48pitrousetnosy: + kristjan.jonsson, sbt
2012-07-23 14:28:26jklothsetnosy: + jkloth
2012-07-23 06:22:44amaury.forgeotdarclinkissue14578 dependencies
2012-07-23 06:21:50amaury.forgeotdarccreate