Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not embed manifest files in *.pyd when compiling with MSVC #48370

Closed
aheider mannequin opened this issue Oct 14, 2008 · 52 comments
Closed

Do not embed manifest files in *.pyd when compiling with MSVC #48370

aheider mannequin opened this issue Oct 14, 2008 · 52 comments
Assignees
Labels
build The build process and cross-build stdlib Python modules in the Lib dir

Comments

@aheider
Copy link
Mannequin

aheider mannequin commented Oct 14, 2008

BPO 4120
Nosy @malemburg, @loewis, @mhammond, @theller, @tiran, @skrah
Files
  • Python-2.6-no.manifest.in.pyd.diff: (committed)
  • msvc9compiler_stripruntimes.diff: Strip the VC90 runtimes assembly from the dependencies in the manifest
  • msvc9compiler_stripruntimes_revised.diff: revised patch
  • msvc9compiler_stripruntimes_regexp.diff
  • msvc9compiler_stripruntimes_regexp2.diff
  • testpyd.py
  • 3.1-byte-string.diff
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/loewis'
    closed_at = <Date 2009-12-03.21:03:47.161>
    created_at = <Date 2008-10-14.08:01:50.834>
    labels = ['build', 'library']
    title = 'Do not embed manifest files in *.pyd when compiling with\tMSVC'
    updated_at = <Date 2010-04-26.19:31:14.754>
    user = 'https://bugs.python.org/aheider'

    bugs.python.org fields:

    activity = <Date 2010-04-26.19:31:14.754>
    actor = 'loewis'
    assignee = 'loewis'
    closed = True
    closed_date = <Date 2009-12-03.21:03:47.161>
    closer = 'loewis'
    components = ['Build', 'Distutils']
    creation = <Date 2008-10-14.08:01:50.834>
    creator = 'aheider'
    dependencies = []
    files = ['11791', '11847', '15026', '15072', '15113', '15161', '15612']
    hgrepos = []
    issue_num = 4120
    keywords = ['patch']
    message_count = 52.0
    messages = ['74723', '74730', '74742', '74747', '74758', '74763', '74835', '74926', '74930', '74992', '74997', '75035', '75071', '75072', '75081', '75153', '75155', '75159', '75183', '75194', '75567', '75570', '75579', '80148', '80908', '80909', '80910', '93440', '93441', '93458', '93700', '93702', '93713', '93716', '93906', '93913', '93943', '94140', '94202', '94221', '94644', '94660', '94669', '94673', '95853', '95946', '96641', '96769', '104237', '104238', '104244', '104268']
    nosy_count = 14.0
    nosy_names = ['lemburg', 'loewis', 'mhammond', 'theller', 'christian.heimes', 'koen', 'aheider', 'zhirsch', 'jdetaeye', 'srid', 'cgohlke', 'skrah', 'ntouran', 'egaudry']
    pr_nums = []
    priority = 'high'
    resolution = 'accepted'
    stage = None
    status = 'closed'
    superseder = None
    type = 'compile error'
    url = 'https://bugs.python.org/issue4120'
    versions = ['Python 3.1']

    @aheider
    Copy link
    Mannequin Author

    aheider mannequin commented Oct 14, 2008

    The MSVC build process currently embeds the .manifest file, which is
    generated by the MS linker, in the following files:

    • python.exe
    • pythonXY.dll
    • *.pyd

    The latter is problematic on machines without the MS CRT redistributable
    installed (No CRT files in %WINDIR%\WinSxS). While this error won't
    occur when the python package is installed, it will in other cases like
    in ours:

    We use the python interpreter within our application and do ship
    python25.dll. We also ship the MS CRT files and place them next to the
    main application (MS calls this "private assemblies" and "xcopy
    deployment). We do not install the redistributable, because it needs
    admin rights.
    If a user tries to "import socket" on this setup it will fail, because
    the Side by Side / Fusion loader expects the CRT next to the calling
    Binary (%PYTHONHOME%\DLLs\_socket.pyd) because of the manifest.

    The solution is to not embed the manifest in the *.pyd Modules. This way
    the modules just link against msvc*.dll. This works because the windows
    loader has already mapped the CRT into memory for the process (either
    through python.exe or pythonXY.dll). This is also the only reliable way
    i could find to fix the import on machines without redist.

    Attached you'll find a patch for the MSVC90 build, apply with -p5. It
    will only patch the release file, PCbuild8/pyd_d.vsprops requires the
    same fix.

    Trolltech also uses this approach for its plugins:
    https://trolltech.com/developer/faqs/faq.2007-10-19.6185511205

    @aheider aheider mannequin added the build The build process and cross-build label Oct 14, 2008
    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Oct 14, 2008

    I'm skeptical about this patch. It seems like it could produce an
    incompatibility with existing installations or deployment procedures. In
    any case, the VS 2008 project files are not officially supported.

    If VS 2005 is any similar with VS 2008 wrt. to manifests, I think you
    can solve your problem by providing a CRT manifest in the directory that
    has _socket.pyd.

    I'd be rather interested in seeing the consequences of this approach for
    Python 2.6.

    @aheider
    Copy link
    Mannequin Author

    aheider mannequin commented Oct 14, 2008

    Yes, i know that v2.5 doesn't officially support MSVC9. But the same
    problem applies to MSVC8 and its CRT. But in contrast to MSVC9, the CRT8
    redist is installed on almost every machine, because alot of software
    installs it. But thats basically just luck ;)

    Placing just a .manifest next to *.pyd won't work either. If you're
    going this route, you have to put the 3 CRT DLLs there too (and thats
    actually the approach mentioned here:
    http://msdn.microsoft.com/en-us/library/ms235291.aspx)
    Then you've got 2 cases:

    1. The user does have the redist installed
      or
    2. He does not

    for 1) The WinSXS installation has a higher priority and the local files
    are ignored
    for 2) The files next to the binaries are used (eg c:\python and
    c:\python\DLLs)

    Sounds fine, but the latter raises another issue: Now you have 2 copies
    of the DLLs. Based on these 2 unique file names, windows decides to load
    2 copies into memory, one for python.exe|dll and one for *.pyd. As you
    might see, this gets very nasty because you have 2 different heaps now.
    Allocating memory from the one instance and freeing it in the other
    makes the whole process crash.

    This issue doesn't stop here, here's another situation:
    Python is compiled with MSVC9. A user has it and MSVC9 SP1 installed,
    which has a newer CRT version. If he builds site-packages, these files
    have an embedded manifest with another CRT version than the python
    interpreter itself. Importing these packages loads a second copy of a
    CRT into the processes memory, just like described above.

    I don't know if or for what version this should be addressed for python.
    I'm just saying that this was a huge issue the last days at the company
    i work for.

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Oct 14, 2008

    Yes, i know that v2.5 doesn't officially support MSVC9. But the same
    problem applies to MSVC8 and its CRT.

    The point is that even the MSVC8 project files are not supported.
    They have been included, but they are not actually used for anything.

    Yes, i know that v2.5 doesn't officially support MSVC9. But the same
    problem applies to MSVC8 and its CRT.

    Not necessarily. Take a look at how I deploy Python 2.6. I use a single
    copy of the DLL, but two copies of the manifest (both referring to the
    same DLL image). AFAICT, this works fine on XP (but fails on Vista SP1,
    which complains that the manifest is ill-formed).

    I'm closing this for 2.5 as rejected; it might cause more problems than
    it solves, and 2.5.3 will be the last release (i.e. with no release to
    fix it if it breaks something badly).

    If you can come up with a working patch for 2.6: that would be more
    interesting. Would the many manifest files that get generated need to be
    shipped as well?

    @loewis loewis mannequin closed this as completed Oct 14, 2008
    @aheider
    Copy link
    Mannequin Author

    aheider mannequin commented Oct 14, 2008

    Ok, point taken, so lets aim at v2.6. Should i open a new issue then?

    Attached you'll find a new diff against v2.6. This time pyd_d.vsprops
    gets patched too and the linker doesn't even generate a .manifest file.
    So no, the *.pyd files do not get any manifest info, neither a loose
    *.pyd.manifest nor an embedded one.

    I checked a per-user v2.6 install with the CRT manifest pointing at
    "..". I also tried this approach for our software, but i couldn't get it
    working on w2k3.

    The solution i am proposing doesn't need a CRT manifest file in
    "./DLLs". This approach works on w2k3, but i can not test this on vista SP1.

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Oct 14, 2008

    I'm just repurposing the issue (despite what I usually claim as a policy)

    @loewis loewis mannequin reopened this Oct 14, 2008
    @aheider
    Copy link
    Mannequin Author

    aheider mannequin commented Oct 16, 2008

    To my surprise there indeed is a vista SP1 box in our test farm. We
    tested my patch on the following windows machines (each without an
    installed CRT redist):

    • xp SP2 and SP3
    • xp64 SP2
    • server 2003 R2 SP1
    • vista with and without SP1
    • windows server 2008

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Oct 17, 2008

    For some reason, the linker *does* generate manifest files for all the
    .pyds when I try it out. Should that worry me? When I delete them, it
    still can import them just fine.

    FWIW, with the patch alone, the manifest in .DLLs will still be needed,
    to be able to load tcl85.dll.

    @koen
    Copy link
    Mannequin

    koen mannequin commented Oct 17, 2008

    In response to 74742:

    This issue doesn't stop here, here's another situation:
    Python is compiled with MSVC9. A user has it and MSVC9 SP1 installed,
    which has a newer CRT version. If he builds site-packages, these files
    have an embedded manifest with another CRT version than the python
    interpreter itself. Importing these packages loads a second copy of a
    CRT into the processes memory, just like described above.

    This is not true. For MSVC9SP (VS2008) Microsoft decided that by
    default, it will still compile against the original CRT. Only by
    explicitly adding a compiler flag can you use the updated SP1 CRT (they
    did this because the 2005SP1 caused so much grief, which did
    automatically upgrade the CRT, and subsequently companies did not roll
    out SP1 to prevent the mixing of CRTs). So you can safely compile using
    SP1 installed, and you will at least still use the same CRT version.

    @aheider
    Copy link
    Mannequin Author

    aheider mannequin commented Oct 20, 2008

    For some reason, the linker *does* generate manifest files for all the
    .pyds when I try it out. Should that worry me? When I delete them, it
    still can import them just fine.

    Hm, works for me. I am using MS SDK v6.1 and PCbuild/build.bat.

    This is not true. For MSVC9SP (VS2008) Microsoft decided that by
    default, it will still compile against the original CRT

    You're right, there are defines for that:
    http://msdn.microsoft.com/en-us/library/cc664727.aspx

    While that won't load a 2nd copy of the CRT into memory, the embedded
    manifests of site-packages still have the same problem as the .pyd files
    now. So if this gets accepted, reverting the patch of bpo-2563 should
    be considered.

    @koen
    Copy link
    Mannequin

    koen mannequin commented Oct 20, 2008

    The TCL85.dll introduces a subtlety (actually it is TK85.dll). The
    MSVCRT90 is not the only dependency that can be in the manifest. The
    Windows.CommonControls.6.0 is used by TK85.dll. So, also for all Python
    extensions which have additional dependencies beside MSVCRT90, the
    manifest should still be included.

    Situation:

    • .pyd only depends on MSVCRT90: no manifest needed (will fallback to
      Python 2.6's own manifest and to WinSXS or the installation in the main
      Python folder)
    • .pyd has more dependencies: manifest is needed. But, a wild idea,
      perhaps we could strip MSVCRT90 from the manifest? This is speculation,
      but perhaps the fallback to the manifest of Python26.exe will still
      work this way, and the runtimes will only need to be in a single place.

    Anyone know of an easy-to-build example of a Python extension with
    extra dependencies?

    @loewis loewis mannequin self-assigned this Oct 20, 2008
    @koen
    Copy link
    Mannequin

    koen mannequin commented Oct 21, 2008

    Instead of reverting the patch for bpo-2563, I propose to strip the
    VC90 runtimes from the manifest (this will preserve other dependencies
    needed). I checked this by editing tk85.dll and replacing the
    dependency there with white-space - the Windows Common Controls
    dependency still worked properly after this. I've attached a patch to
    strip the runtime from the manifest when building Python extensions
    (through search/replace).

    I think somehow the manifests of the .pyd's in DLLs should also be
    edited, because leaving them out will cause problems if this is done
    when building tk85.dll. Or perhaps only special care is needed if there
    ever is a .pyd with additional dependencies? All files with the .pyd
    extension do not rely on anything else. I don't know enough about the
    build process to suggest a change for this, except that the build
    config for the .dll files inside DLLs might also need to be changed.

    @aheider
    Copy link
    Mannequin Author

    aheider mannequin commented Oct 22, 2008

    Skipping the mt.exe call seems fine to me, but there's another solution
    which might be better:

    The automatic manifest binding comes from a "#pragma
    comment(linker,"/manifestdependency:" directive in crtdefs.h. That can
    be disabled by setting _CRT_NOFORCE_MANIFEST. This way all other
    manifest dependencies stay in tact and it could be set for *.pyd builds
    (instead of my patch).

    TCL's manifest is a different problem. I do not use tkinter, but
    according to PCbuild/readme.txt it should be possible to get rid of the
    CRT dependency the same way: just set _CRT_NOFORCE_MANIFEST in the nmake
    line.

    @aheider
    Copy link
    Mannequin Author

    aheider mannequin commented Oct 22, 2008

    .. erm, the nmake line in Tools\buildbot\external.bat and
    Tools/buildbot/external-amd64.bat ;)

    @koen
    Copy link
    Mannequin

    koen mannequin commented Oct 22, 2008

    _CRT_NOFORCE_MANIFEST sounds nice, but I'm having some trouble getting
    it to work for Python extensions. I'm still having manifests
    generated...? Can you show me how to use it? It is an undocumented
    option.

    @mhammond
    Copy link
    Contributor

    I don't see a problem with this and can see how it would help with
    private assemblies.

    @aheider
    Copy link
    Mannequin Author

    aheider mannequin commented Oct 24, 2008

    I tried to use _CRT_NOFORCE_MANIFEST but i couldn't get it working.
    There're some infos about this approach at
    http://blog.m-ri.de/index.php/2008/05/06/hotfix-fuer-usemsprivateassembliesh-und-vc-2008/
    , but even with the mentioned "__declspec(selectany) int
    _forceCRTManifest[RTM];" statement, i wasn't able to suppress the
    manifest dependency :(

    Microsoft, in its infinite wisdom, decided to compile the CRT itself
    with those #pragma's enabled. That puts the "/manifestdependency" linker
    arguments in msvcrt.lib. When you link against that file you get that
    CRT manifest entry, whether you like it or not :(
    Looks like _CRT_NOFORCE_MANIFEST only works if you build your own CRT
    without the #pragma's.
    Of course that only applies to the dynamic version of the CRT, when
    using /MT the manifest hell just vanishes.

    I'm unsure what the best way to fix this issue for python is...

    @koen
    Copy link
    Mannequin

    koen mannequin commented Oct 24, 2008

    OK, so the define is not going to work. For Python extensions built
    through distutils, the msvc9compiler_stripruntimes.diff patch will
    work: it will remove just the dependency on the VC90runtimes. It will
    leave other assembly dependencies if they are there.

    For the .pyd's and .dll's in the DLLs folder: I have opened them in a
    hex editor, and just by hand replaced the <assemblyIdentity> entry for
    the VC90runtimes in each file with spaces. I can confirm that this
    works, also for _tkinter, for which tk85.dll has an additional
    dependency on Windows Common Controls (though this one was available in
    my WinSXS). What would be the best way to integrate a search/replace
    operation like this in the Python build process? Is the manifest
    generated as a seperate file here first as well, or is it directly
    embedded?

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Oct 24, 2008

    What would be the best way to integrate a search/replace
    operation like this in the Python build process? Is the manifest
    generated as a seperate file here first as well, or is it directly
    embedded?

    What's wrong with Andre Heider's patch?

    @koen
    Copy link
    Mannequin

    koen mannequin commented Oct 24, 2008

    Ah, I wasn't thinking it through. It is fine for all .pyd of course,
    because they don't have any extra dependencies. I was thinking of DLLs
    with extra dependencies (of which there is one, TK85.dll). There,
    leaving out the manifest will break it. But indeed, the patch is fine
    for .pyd building. My mistake!

    @tiran
    Copy link
    Member

    tiran commented Nov 6, 2008

    Should this patch be applied to 3.0 before the next RC lands? Barry
    hasn't released RC2 yet.

    @tiran tiran added the build The build process and cross-build label Nov 6, 2008
    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Nov 6, 2008

    I have now committed Python-2.6-no.manifest.in.pyd.diff as r67120,
    r67121, and r67122. Thanks for the patch.

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Nov 6, 2008

    In r67125, r67126, r67127, r67128, r67129, I removed the mt.exe
    invocation from tcl and tk, and removed the DLLs CRT manifest from the
    installer.

    @zhirsch
    Copy link
    Mannequin

    zhirsch mannequin commented Jan 19, 2009

    Without msvc9compiler_stripruntimes.diff, or a patch like it, it doesn't
    seem possible to use pywin32 (or other binary extensions) with
    Python-2.6+ installed "just for me" (i.e., without the VC90 runtime
    installed globally).

    Using the python-2.6.1 MSI installed "just for me" and the pywin32-212
    installer, I get the following error when trying to import win32api:

        Python 2.6.1 (r261:67517, Dec  4 2008, 16:51:00) [MSC v.1500 32 bit
    (Intel)] on win32
        Type "help", "copyright", "credits" or "license" for more information.
        >>> import win32api
        Traceback (most recent call last):
          File "<stdin>", line 1, in <module>
        ImportError: DLL load failed: This application has failed to start
    because the application configuration is incorrect. Reinstalling the
    application may fix this problem.

    Removing the manifest from all the .pyd files installed by pywin32
    allows me to import win32api without issue.

    @jdetaeye
    Copy link
    Mannequin

    jdetaeye mannequin commented Feb 1, 2009

    Is the report posted in:
    http://groups.google.com/group/comp.lang.python/browse_thread/thread/3ec
    6af1279a162ca#
    also a symptom of the same problem?

    Johan

    @jdetaeye
    Copy link
    Mannequin

    jdetaeye mannequin commented Feb 1, 2009

    @Koen: You're aboslutely right. The issue is solved indeed by adding a
    dependency in the manifest.

    @koen
    Copy link
    Mannequin

    koen mannequin commented Oct 2, 2009

    Could msvc9compiler_stripruntimes.diff still be considered for
    inclusion in Python 2.6.3 (high priority?)?
    A lot of .pyd's of third-party extensions are getting dependencies on
    MSVCR90 added to the .pyd, which makes them not work if the runtimes
    are not installed into the WinSxS folder and/or when embedding into
    other applications. For example, PIL encountered this problem recently.
    In essence, all Windows Python extensions built using setup.py
    build_ext encounter this problem.
    What the patch does is edit the manifest file to remove only the
    dependency on Microsoft.VC90.CRT.

    @koen
    Copy link
    Mannequin

    koen mannequin commented Oct 2, 2009

    It probably won't make 2.6.3 as the final is just out. But can we have
    it for 2.6.4?
    Here's a link to the relevant discussion on ImageSIG: http://
    mail.python.org/pipermail/image-sig/2009-October/005918.html

    Looking at my local extensions installed, Matplotlib has applied this
    patch, but PyGame and PIL have embedded manifests with dependencies on
    the runtime in their .pyd's.

    The patch now does a text search/replace on an XML file, if you think
    this is bad style I could rewrit it to use proper XML parsing and
    writing.

    @cgohlke
    Copy link
    Mannequin

    cgohlke mannequin commented Oct 2, 2009

    There are two easy to fix issues with the
    msvc9compiler_stripruntimes.diff patch: 1) the dependency for 64-bit
    VC90.CRT is not removed and 2) the VC90.CRT dependency is removed also
    from executables, which will fail to run. A revised patch is attached.
    This is used to build the Matplotlib binaries for Windows and also works
    for numpy, pygame and PIL.

    @koen
    Copy link
    Mannequin

    koen mannequin commented Oct 7, 2009

    Thanks Christoph, those are two important fixes to the patch. I'm +1 on
    having this in the next 2.6 maintenance release.

    @malemburg
    Copy link
    Member

    Just a note on the style of the msvc9compiler_stripruntimes_revised.diff
    patch:

    • please try to use lines with at most 80 chars only

    • it would be better to use regexps to do the search&replace and
      perhaps add wildcards to catch future SP-versions of VC9 as well

    Added Distuils as component, since the patch is targeting distutils and
    extensions, not the Python build process.

    @malemburg malemburg added the stdlib Python modules in the Lib dir label Oct 7, 2009
    @cgohlke
    Copy link
    Mannequin

    cgohlke mannequin commented Oct 7, 2009

    The attached patch uses a regular expression.

    @malemburg
    Copy link
    Member

    Christoph Gohlke wrote:

    Christoph Gohlke <cgohlke@uci.edu> added the comment:

    The attached patch uses a regular expression.

    Much better, thanks.

    @malemburg malemburg changed the title Do not embed manifest files in *.pyd when compiling with MSVC Do not embed manifest files in *.pyd when compiling with MSVC Oct 7, 2009
    @malemburg
    Copy link
    Member

    Looking at the description of manifest files, it appears that just
    removing the assemblyIdentity-element results in an invalid manifest file:

    http://msdn.microsoft.com/en-us/library/aa374219(VS.85).aspx

    It appears that the entire dependency-element referencing the MS VC90
    CRT DLL has to be removed in order to make the manifest correct again.

    Looking at the schema, it's enough to just remove the
    dependentAssembly-element:

    http://msdn.microsoft.com/en-us/library/aa375635(VS.85).aspx

    Reading up on the manifest trouble-shooting page at (near the end of the
    page):

    http://msdn.microsoft.com/en-us/library/ms235342.aspx

    it may actually be enough to just place the Microsoft.VC90.CRT.manifest
    file into the Python folder (the one with python.exe and the CRT DLLs):

    """
    If the operating system fails to find the CRT or any other assembly as a
    shared assembly, it starts looking for the assembly as a private
    assembly. It searches for private assemblies in the following order:

    1. Check the application local folder for a manifest file with name
      <assemblyName>.manifest. In this example, the loader tries to find
      Microsoft.VC90.CRT.manifest in the same folder as appl.exe. If the
      manifest is found, the loader loads the CRT DLL from the application
      folder. If the CRT DLL is not found, load fails.
      """

    @cgohlke
    Copy link
    Mannequin

    cgohlke mannequin commented Oct 13, 2009

    This patch also removes empty dependentAssembly elements after removing
    the VC.CRT assemblyIdentity element.

    It seems not enough to just place the Microsoft.VC90.CRT.manifest and VC
    runtime DLL files into the Python folder. On a system without the VC
    runtime installed in winsxs, trying to import a pyd module build without
    the patch fails with "ImportError: DLL load failed: ...". Import works
    when 1) the manifest and VC runtime DLLs are also placed in the same
    folder as the pyd file, or 2) the Microsoft Visual C++ 2008
    redistributable package is installed, or 3) the pyd file is build with
    the patch installed. Tested with Windows XP SP3, Python 2.6.3,
    matplotlib 0.99.1, Visual Studio 2008 SP1.

    @koen
    Copy link
    Mannequin

    koen mannequin commented Oct 13, 2009

    It may actually be enough to just place the
    Microsoft.VC90.CRT.manifest file into the Python folder (the one with
    python.exe and the CRT DLLs).

    I concur with what Christoph says, that is how the embedded
    installation works. However, the .pyd files are not in the Python main
    folder. And therefore they shouldn't have a manifest for the runtimes
    (because it will look in the current folder relative to the DLL/pyd),
    because then the runtimes of the main python.exe will be used (which
    can then be in the same folder as the 'embedded' MSVCR location).

    @ntouran
    Copy link
    Mannequin

    ntouran mannequin commented Oct 16, 2009

    Just to share my recent experience with this issue: I was attempting to
    get Python 2.6 working with py4mpi, matplotlib, and pymssql on a 64-bit
    Windows Vista based HPC cluster via x-copy deployment without the charm
    of administrative privileges on the server and on my workstation (which
    is XP). It wasn't easy, but I got it working.

    I attempted to apply the patch listed here and recompile the packages
    that complained about Side-by-side configurations, but got in too deep
    while compiling matplotlib on windows (recall I don't have admin
    privileges and can't install the prereqs) so I dug deeper.

    I found this: http://blog.kalmbachnet.de/?postid=80 which suggests
    removing the publicKeyToken attribute from the manifests to force local
    DLLs. This would possibly give the same effect as not embedding the
    manifests in the first place using the patch. So I went in, using
    VS2008, and removed this attribute from the embedded manifests in
    python26.dll, python.exe, and the manifest file in C:\python26. I also
    copied msvcm90.dll, msvcp90.dll, and msvcr90.dll from the 9.0.21022.8
    folder in c:\Windows\WinSxS into c:\Python26. With that, matplotlib
    started working, but pymssql still did not.

    I then renamed _mssql.pyd to _mssql.dll so that VS2008 could recognize
    the manifest, removed the publicKeyToken attribute, and renamed it back
    to _mssql.pyd, but it still complained. Finally, using depends.exe from
    the internet, I noticed msvcr71.dll was required on my local machine, so
    I tried copying it over to the remote machine, into the site-packages
    folder. Bam. It worked. So while I don't really consider this a
    solution, it definitely worked for what I needed. Any other 3rd party
    modules that complain in the future in my weird xcopy-deployment will
    undergo manifest-stripping via VS2008 and other dependency checking via
    depends.exe. Yay.

    By the way, the pymssql was compiled with VS2005, which is why it
    requires msvcr71.dll. If I could rebuild it with VS2008, I imagine that
    step wouldn't be necessary. While I do have VS2008 on hand, I don't have
    MS SQL 2020 Super Awesome Developer Edition, or whatever 3GB package
    installs the required header.

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Oct 18, 2009

    What's a procedure for testing this patch (please be as precise as
    possible)?

    @cgohlke
    Copy link
    Mannequin

    cgohlke mannequin commented Oct 18, 2009

    There are two levels of testing.

    First, on a Windows development system with Visual Studio 2008, Python
    2.6.2+, and the msvc9compiler_stripruntimes_regexp2.diff patch applied,
    verify that the embedded manifest in distutil generated PYD extension
    files is stripped of the WinSxS VC.CRT dependency and that the
    extensions can still be imported. The attached script (testpyd.py)
    builds a minimal extension (testpyd.pyd), searches for the WinSxS VC.CRT
    dependency within the pyd file, and imports the extension. The script
    should run without exception.

    Second, verify that distutil generated PYD extensions can also be
    imported on a test system, which has no Visual Studio 2008 runtime
    installed in %WINDIR%\WinSxS:
    Install Windows XP SP3, preferably in a virtual machine. Do not install
    any additional programs or patches, which might install the Visual
    Studio 2008 runtime. Then install Python 2.6.2+ using the "Install just
    for me" option. Try import the testpyd extension built on the
    development system: run "python.exe -c 'import testpyd'" in a directory
    containing the testpyd.pyd file (not the Python installation directory
    containing python.exe). This should work without exception.

    Importing the testpyd extension built without the patch fails on the
    test system:

    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ImportError: DLL load failed: This application has failed to start
    because the application configuration is incorrect. Reinstalling the
    application may fix this problem.

    Tested with Python 2.6.3 for Windows 32-bit.

    @cgohlke
    Copy link
    Mannequin

    cgohlke mannequin commented Oct 29, 2009

    Apparently the msvc9compiler_stripruntimes_regexp2 patch causes problems
    for MinGW users. The following C program is using the Python C API to
    import the testpyd extension generated by testpyd.py. When compiled with
    MinGW, the program fails with "ImportError: DLL load failed:..." if the
    PYD extension is compiled with MSVC9 and the patch is applied. The
    program works if 1) it is compiled with MSVC9, or 2) the testpyd
    extension is build without the patch, or 3) the files
    Microsoft.VC90.CRT.manifest and msvcr90.dll are placed next to the
    executable and the manifest is also embedded into the executable (e.g.
    using mt.exe).

    /* Import the testpyd.pyd module. */
    #include <Python.h>
    int main(void) {
        Py_Initialize();
        { 
            PyObject *p = PyImport_ImportModule("testpyd"); 
        }
        Py_Finalize();
        return 0;
    }

    @malemburg
    Copy link
    Member

    Christoph Gohlke wrote:

    Christoph Gohlke <cgohlke@uci.edu> added the comment:

    Apparently the msvc9compiler_stripruntimes_regexp2 patch causes problems
    for MinGW users. The following C program is using the Python C API to
    import the testpyd extension generated by testpyd.py. When compiled with
    MinGW, the program fails with "ImportError: DLL load failed:..." if the
    PYD extension is compiled with MSVC9 and the patch is applied. The
    program works if 1) it is compiled with MSVC9, or 2) the testpyd
    extension is build without the patch, or 3) the files
    Microsoft.VC90.CRT.manifest and msvcr90.dll are placed next to the
    executable and the manifest is also embedded into the executable (e.g.
    using mt.exe).

    /* Import the testpyd.pyd module. */
    #include <Python.h>
    int main(void) {
    Py_Initialize();
    {
    PyObject *p = PyImport_ImportModule("testpyd");
    }
    Py_Finalize();
    return 0;
    }

    I'm not sure whether this is related to the problem in question.
    Does MinGW also embed a manifest in the generated executable ?

    If it doesn't, then this is more likely a problem with how
    MinGW works per default, than with Python or distutils.

    The fact that the above does work without the patch applied
    suggests that MinGW does not embed the manifest in the executable
    (since the Windows linker then uses the one from the PYD file).

    @cgohlke
    Copy link
    Mannequin

    cgohlke mannequin commented Oct 29, 2009

    My last comment was merely reporting that this patch can break MinGW
    based build processes. It did surprise some developers that their
    programs embedding matplotlib, which is now build with the MSVC9 patch,
    stopped working.

    @koen
    Copy link
    Mannequin

    koen mannequin commented Oct 29, 2009

    The MinGW breakage probably comes from the same "issue" as encountered
    in http://bugs.python.org/issue4120#msg80908 and #msg80909: the main
    application, which embeds Matplotlib, does not have a manifest for the
    MSVCR9 runtimes? That's a problem which should be addressed by adding a
    manifest in the main application (and manifests can also be separate
    files called app.exe.manifest if you do not want to embed it).

    @egaudry
    Copy link
    Mannequin

    egaudry mannequin commented Dec 1, 2009

    I came across this (very) interesting thread after experiencing some
    redistribution issue with a python(2.6)-based package built with msvc9
    compiler.

    I used to struggled with the SxS Microsoft policy in the past. After I
    finally went for the private assembly/crt runtime redistribution
    solution, I always succeeded to deliver a stand-alone package to any
    windows host around (I got full control on what's actually build and
    distribute).

    Having now (using the disutils module) the manifest file embedded in the
    python extension actually forbids this redistribution solution, for the
    reasons that have been posted here (SxS policy), unless a private
    assembly/crt runtime is provided next to each built extension. IMHO,
    this is not a convenient (and common) way to redistribute a software.

    This is why I fully agree with the propositions that were made here,
    i.e. not embedding manifest into a python extension built with the
    distutils module and msvc.

    Could anyone tell us if a decision has been made about such a change ?

    Thanks,
    Eloi

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Dec 3, 2009

    Thanks for the patch. Committed as r76651, r76652, r76653, r76654,
    r76655 and r76656.

    @loewis loewis mannequin closed this as completed Dec 3, 2009
    @skrah
    Copy link
    Mannequin

    skrah mannequin commented Dec 19, 2009

    With the latest Python3.1 svn version of msvc9compiler.py, I get a
    "can't use a string pattern on a bytes-like object" error. The
    attached diff fixes the error.

    @srid
    Copy link
    Mannequin

    srid mannequin commented Dec 21, 2009

    See bpo-7556 for the "can't use a string pattern on a bytes-like object"
    error.

    @theller
    Copy link

    theller commented Apr 26, 2010

    I'm not sure this issue is really done. The current state works fine, but there is one problem remaining:

    There are Python extensions that seem to need a manifest pointing to the MSVC runtime libs: dlls that start in-process com servers, like pythoncom.dll (from the PyWin32 extension), and also _ctypes.pyd plays this role.

    First question: Open a new issue, or discuss it in this one (and repoen it)?

    @egaudry
    Copy link
    Mannequin

    egaudry mannequin commented Apr 26, 2010

    Hi Thomas,

    I think we should open a new issue (child from bpo-4120), just to make
    sure the whole thing remains understandable.

    Regards,
    Eloi

    Thomas Heller wrote:

    Thomas Heller <theller@ctypes.org> added the comment:

    I'm not sure this issue is really done. The current state works fine, but there is one problem remaining:

    There are Python extensions that seem to need a manifest pointing to the MSVC runtime libs: dlls that start in-process com servers, like pythoncom.dll (from the PyWin32 extension), and also _ctypes.pyd plays this role.

    First question: Open a new issue, or discuss it in this one (and repoen it)?

    ----------
    nosy: +theller


    Python tracker <report@bugs.python.org>
    <http://bugs.python.org/issue4120\>


    --

    Eloi Gaudry

    Free Field Technologies
    Axis Park Louvain-la-Neuve
    Rue Emile Francqui, 1
    B-1435 Mont-Saint Guibert
    BELGIUM

    Company Phone: +32 10 487 959
    Company Fax: +32 10 454 626

    @cgohlke
    Copy link
    Mannequin

    cgohlke mannequin commented Apr 26, 2010

    I mentioned the problem with pythoncom.dll and suggested a solution in bpo-7833.

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Apr 26, 2010

    First question: Open a new issue, or discuss it in this one (and repoen it)?

    Please open a new issue. AFAICT, the original issues request (do not
    embed manifests in pyd files) is now implemented fully. Feel free to
    leave a link to the new issue here, but please start over describing the
    problem you are seeing, and in what version you are seeing it.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    build The build process and cross-build stdlib Python modules in the Lib dir
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants