Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot build importlib.h on Windows #59636

Closed
amauryfa opened this issue Jul 23, 2012 · 14 comments
Closed

Cannot build importlib.h on Windows #59636

amauryfa opened this issue Jul 23, 2012 · 14 comments
Labels
build The build process and cross-build OS-windows release-blocker

Comments

@amauryfa
Copy link
Member

BPO 15431
Nosy @loewis, @amauryfa, @pitrou, @kristjanvalur, @jkloth, @briancurtin
Files
  • _freeze_importlib.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2012-07-28.20:05:54.676>
    created_at = <Date 2012-07-23.06:21:49.930>
    labels = ['build', 'OS-windows', 'release-blocker']
    title = 'Cannot build importlib.h on Windows'
    updated_at = <Date 2012-07-28.20:05:54.675>
    user = 'https://github.com/amauryfa'

    bugs.python.org fields:

    activity = <Date 2012-07-28.20:05:54.675>
    actor = 'loewis'
    assignee = 'none'
    closed = True
    closed_date = <Date 2012-07-28.20:05:54.676>
    closer = 'loewis'
    components = ['Build', 'Windows']
    creation = <Date 2012-07-23.06:21:49.930>
    creator = 'amaury.forgeotdarc'
    dependencies = []
    files = ['26538']
    hgrepos = []
    issue_num = 15431
    keywords = ['patch']
    message_count = 14.0
    messages = ['166209', '166425', '166554', '166560', '166582', '166587', '166589', '166674', '166675', '166676', '166684', '166685', '166687', '166688']
    nosy_count = 8.0
    nosy_names = ['loewis', 'amaury.forgeotdarc', 'pitrou', 'kristjan.jonsson', 'jkloth', 'brian.curtin', 'python-dev', 'sbt']
    pr_nums = []
    priority = 'release blocker'
    resolution = 'fixed'
    stage = None
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue15431'
    versions = ['Python 3.3']

    @amauryfa
    Copy link
    Member Author

    On Windows, the _freeze_importlib tool is not built, so it's not possible to refresh the file Python/importlib.h, which makes development on the importlib very difficult on Windows.

    The Makefile contains the rules below, it's probably necessary to implement a similar behavior in a new build project:

    Modules/_freeze_importlib: Modules/_freeze_importlib.o $(LIBRARY_OBJS_OMIT_FROZEN)
    $(LINKCC) $(PY_LDFLAGS) -o $@ Modules/_freeze_importlib.o $(LIBRARY_OBJS_OMIT_FROZEN) $(LIBS) $(MODLIBS) $(SYSLIBS) $(LDLAST)

    Python/importlib.h: $(srcdir)/Lib/importlib/_bootstrap.py Modules/_freeze_importlib.c
    $(MAKE) Modules/_freeze_importlib
    ./Modules/_freeze_importlib \
    $(srcdir)/Lib/importlib/_bootstrap.py Python/importlib.h

    @amauryfa amauryfa added build The build process and cross-build OS-windows labels Jul 23, 2012
    @kristjanvalur
    Copy link
    Mannequin

    kristjanvalur mannequin commented Jul 25, 2012

    shouldn't be too hard to add, I'll give it a go soon.

    @kristjanvalur
    Copy link
    Mannequin

    kristjanvalur mannequin commented Jul 27, 2012

    Here's a patch. A new project, _freeze_importlib, has the necessary prerequisites and runs the executable, generating importlib.h, as a post-build step.

    I'm not sure if we should make this project a pre-requisite of any other projects, though

    @amauryfa
    Copy link
    Member Author

    IMO dependencies is not a big issue here. In the worst case, the developer can build a second time...

    I think your patch won't work on Unix machines: _freeze_importlib.c is linked with all Python files *except* frozen.c which defines PyImport_FrozenModules. #ifdef should be used IMO.

    Also, I noticed that the marshalled code is different. The first difference a 32bit vs. 64bit issue: 0xFFFFFFFF is marshalled as a TYPE_INT64 on 64bit platforms, but as a TYPE_LONG on 32bit platforms. See bpo-15466.

    @kristjanvalur
    Copy link
    Mannequin

    kristjanvalur mannequin commented Jul 27, 2012

    Thank you. I see how I must change the linkage on PC Only, although I wonder why tuen unix version isn't simply doing something similar..

    The marshaled code was indeed created using 64 bits. I was unsure whether to include that file as part of the patch, perhaps I shouldn't.

    I'll wait for an update for the 64 bit issue.

    @amauryfa
    Copy link
    Member Author

    (Sorry for the previous message, it was wrong, and I also pressed the wrong button...)

    A Unix Makefile cannot stand circular dependencies, so _freezeimportlib.c cannot depend on frozen.o. OTOH it's not simple to have almost-similar list of files on VS projects (here we need all-of-python-but-frozen.c), so your solution is not bad either.
    Please check that python33.dll is not always rebuilt each time you hit F7, though.

    @pitrou
    Copy link
    Member

    pitrou commented Jul 27, 2012

    I'll wait for an update for the 64 bit issue.

    I don't think it's necessary, it's a pretty minor issue since the generated bytecode is portable anyway.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Jul 28, 2012

    New changeset fe29a657bde9 by Martin v. Löwis in branch 'default':
    Issue bpo-15431: Add _freeze_importlib project to regenerate importlib.h on Windows.
    http://hg.python.org/cpython/rev/fe29a657bde9

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Jul 28, 2012

    Thanks for the patch, committed with slight modifications.

    @loewis loewis mannequin closed this as completed Jul 28, 2012
    @amauryfa
    Copy link
    Member Author

    Comments about PyImport_FrozenModules linkage have not been addressed. Now the build is failing on all Unix machines...

    @amauryfa amauryfa reopened this Jul 28, 2012
    @pitrou
    Copy link
    Member

    pitrou commented Jul 28, 2012

    To be clear, the build fails when importlib needs to be frozen again:

    $ touch Lib/importlib/_bootstrap.py 
    $ make
    make Modules/_freeze_importlib
    make[1] : on entre dans le répertoire « /home/antoine/cpython/default »
    gcc -pthread -c -Wno-unused-result -g -O0 -Wall -Wstrict-prototypes    -I. -I./Include    -DPy_BUILD_CORE -o Modules/_freeze_importlib.o Modules/_freeze_importlib.c
    gcc -pthread   -o Modules/_freeze_importlib Modules/_freeze_importlib.o Modules/getbuildinfo.o Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar.o Parser/pgen.o Parser/myreadline.o Parser/parsetok.o Parser/tokenizer.o Objects/abstract.o Objects/accu.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/capsule.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/weakrefobject.o Python/_warnings.o Python/Python-ast.o Python/asdl.o Python/ast.o Python/bltinmodule.o Python/ceval.o Python/compile.o Python/codecs.o Python/dynamic_annotations.o Python/errors.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/graminit.o Python/import.o Python/importdl.o Python/marshal.o Python/modsupport.o Python/mystrtoul.o Python/mysnprintf.o Python/peephole.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/pytime.o Python/random.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/dynload_shlib.o   Python/thread.o Modules/config.o Modules/getpath.o Modules/main.o Modules/gcmodule.o  Modules/_threadmodule.o  Modules/signalmodule.o  Modules/posixmodule.o  Modules/errnomodule.o  Modules/pwdmodule.o  Modules/_sre.o  Modules/_codecsmodule.o  Modules/_weakref.o  Modules/_functoolsmodule.o  Modules/operator.o  Modules/_collectionsmodule.o  Modules/itertoolsmodule.o  Modules/_localemodule.o  Modules/_iomodule.o Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o  Modules/zipimport.o  Modules/faulthandler.o  Modules/symtablemodule.o  Modules/xxsubtype.o -lpthread -ldl  -lutil   -lm  
    Modules/_freeze_importlib.o: In function `main':
    /home/antoine/cpython/default/Modules/_freeze_importlib.c:37: undefined reference to `PyImport_FrozenModules'
    Python/import.o: In function `find_frozen':
    /home/antoine/cpython/default/Python/import.c:978: undefined reference to `PyImport_FrozenModules'
    collect2: ld a retourné 1 code d'état d'exécution
    make[1]: *** [Modules/_freeze_importlib] Erreur 1
    make[1] : on quitte le répertoire « /home/antoine/cpython/default »
    make: *** [Python/importlib.h] Erreur 2

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Jul 28, 2012

    New changeset 7967cb63e50e by Martin v. Löwis in branch 'default':
    Issue bpo-15431: Declare PyImport_FrozenModules conditionally on Unix only.
    http://hg.python.org/cpython/rev/7967cb63e50e

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Jul 28, 2012

    New changeset deb421baf671 by Martin v. Löwis in branch 'default':
    Issue bpo-15431: Drop _freeze_importlib from all build configurations,
    http://hg.python.org/cpython/rev/deb421baf671

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Jul 28, 2012

    The last commits should address Amaury's concerns. _freeze_importlib must now be built manually, followed by a build of pythoncore.

    Interestingly, the builds failed only once. When rebuilding, the builds succeeded as no attempt to build freeze_importlib was made. After the last change to freeze_importlib, they again built it, and that seems to work now.

    Tentatively closing the issue again.

    @loewis loewis mannequin closed this as completed Jul 28, 2012
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    build The build process and cross-build OS-windows release-blocker
    Projects
    None yet
    Development

    No branches or pull requests

    2 participants