classification
Title: Distutils MSVC doesn't create manifest file
Type: behavior Stage:
Components: Distutils Versions: Python 3.3, Python 3.2, Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: loewis Nosy List: DNS, David.Joy, amaury.forgeotdarc, coreypobrien, eric.araujo, gcflymoto, ixokai, lemburg, loewis, mhammond, paxan, sable, santa4nt
Priority: normal Keywords: patch

Created on 2008-11-25 20:16 by DNS, last changed 2014-02-15 05:35 by ixokai. This issue is now closed.

Files
File name Uploaded Description Edit
0001-Ensure-the-assembly-manifest-file-generation-is-gene.patch paxan, 2009-02-17 19:22 Patch that adds /MANIFEST linker option.
Messages (27)
msg76424 - (view) Author: David Schnur (DNS) Date: 2008-11-25 20:16
This is my first time submitting an issue; sorry if I am doing this 
wrong.  While attempting to build/install PyOpenSSL on Windows / MSVC, 
the mt.exe step failed because it could not find the manifest file that 
it was attempting to embed in crypto.pyd.  The problem was that link.exe 
was not creating the manifest.

The reason why is that distutils only passes link.exe the 
/MANIFESTFILE:filename parameter.  This tells it where to output the 
manifest, but not to actually create the manifest (see http://msdn.microsoft.com/en-us/library/fft52235(VS.80).aspx).  You'd 
think link could figure out that, if you use /MANIFESTFILE, you want a 
manifest, but I guess not ;)

My solution was to add this line to distutils/msvc9compiler.py:

ld_args.append('/MANIFEST')

Right beneath the existing line:

ld_args.append('/MANIFESTFILE:' + temp_manifest)

Hope that helps
msg76426 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-11-25 20:53
Since issue4120, the compilation tools precisely chose to *not* embed 
manifests in .pyd extensions.
This probably means that the "mt.exe" step should be skipped by 
distutils.
msg76429 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2008-11-25 21:17
On 2008-11-25 21:16, David Schnur wrote:
> New submission from David Schnur <david.schnur@dartware.com>:
> 
> This is my first time submitting an issue; sorry if I am doing this 
> wrong.  While attempting to build/install PyOpenSSL on Windows / MSVC, 
> the mt.exe step failed because it could not find the manifest file that 
> it was attempting to embed in crypto.pyd.  The problem was that link.exe 
> was not creating the manifest.
> 
> The reason why is that distutils only passes link.exe the 
> /MANIFESTFILE:filename parameter.  This tells it where to output the 
> manifest, but not to actually create the manifest (see http://msdn.microsoft.com/en-us/library/fft52235(VS.80).aspx).  You'd 
> think link could figure out that, if you use /MANIFESTFILE, you want a 
> manifest, but I guess not ;)
> 
> My solution was to add this line to distutils/msvc9compiler.py:
> 
> ld_args.append('/MANIFEST')
> 
> Right beneath the existing line:
> 
> ld_args.append('/MANIFESTFILE:' + temp_manifest)
> 
> Hope that helps

I'm not sure whether that's necessary. We are building pyOpenSSL
just fine against stock Python 2.6.0 in our distribution:

    http://www.egenix.com/products/python/pyOpenSSL/

and even though the command line does not include the /MANIFEST
switch, the linker does builds the .manifest file just fine.

OTOH, it probably doesn't hurt adding the switch :-)
msg76430 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2008-11-25 21:20
This is why we don't see the problem:

http://msdn.microsoft.com/en-us/library/f2c0w594.aspx

"""
The default is /MANIFEST.
"""

So it appears that you must have disabled this default somehow.
msg76432 - (view) Author: David Schnur (DNS) Date: 2008-11-25 21:39
I looked at this some more (I'm not super familiar with the use of
manifests) and I think I figured it out.  For somewhat complicated
reasons, I'm compiling with /MT rather than /MD.  Although link normally
produces a manifest, since it's unnecessary when compiling static, it
isn't created unless you specify /MANIFEST.

So I suppose the issue is that distutils is trying to embed the manifest
in cases where it may legitimately not exist.  Since /MT is not the
default (I had to make that change explicitly), this is probably not a bug.
msg76434 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-11-25 22:20
Closing it as "invalid" then.
msg82358 - (view) Author: Pavel Repin (paxan) Date: 2009-02-17 19:22
I'd like to point out that on some configurations (at least mine), you 
really need to specify /MANIFEST option to the linker, even though MSDN 
documentation seems to imply that /MANIFEST behavior is ON by default.
My config:
beta version of Windows 7
ActivePython 2.6.1.1
MSVS 2008 with 32-bit C/C++ Optimizing Compiler Version 15.00.21022.08 
for 80x86
msg82377 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2009-02-17 22:08
On 2009-02-17 20:22, Pavel Repin wrote:
> Pavel Repin <prepin+pythonbugs@gmail.com> added the comment:
> 
> I'd like to point out that on some configurations (at least mine), you 
> really need to specify /MANIFEST option to the linker, even though MSDN 
> documentation seems to imply that /MANIFEST behavior is ON by default.
> My config:
> beta version of Windows 7
> ActivePython 2.6.1.1
> MSVS 2008 with 32-bit C/C++ Optimizing Compiler Version 15.00.21022.08 
> for 80x86

Are you sure ?

We've had such a request before and the reason for MSVC not generating
a .manifest file was some setting the user had done on his system.

FWIW: distutils generates those files just fine for me.

Then again, it probably doesn't hurt just adding the option.
msg82379 - (view) Author: Pavel Repin (paxan) Date: 2009-02-17 22:34
Hi Marc,I am pretty sure it helped on my particular configuration.
I was trying to compile MySQL-python-1.2.2.tar.gz package from source and it
was failing in mt.exe step because the manifest file was not there.
I didn't do anything special on my machine. I have 3 versions of MSVS
installed cleanly side-by-side (which is a legit scenario): MSVS 2003, MSVS
2005, and MSVS 2008. All at the latest patch levels. No standalone
PlatformSDKs are installed.

Besides, if you just ignore the obscure blurb about /MANIFEST option being
default in MSDN, and read the descriptions of both /MANIFEST and
/MANIFESTFILE options, you would agree that it will not do any harm to be
explicit and always have /MANIFEST option passed to linker.

I'm going to do one more experiment with a different machine, this time it
will be Vista (not W7) with a similar MSVS setup. I will report my findings.

On Tue, Feb 17, 2009 at 2:08 PM, Marc-Andre Lemburg
<report@bugs.python.org>wrote:

>
> Marc-Andre Lemburg <mal@egenix.com> added the comment:
>
> On 2009-02-17 20:22, Pavel Repin wrote:
> > Pavel Repin <prepin+pythonbugs@gmail.com <prepin%2Bpythonbugs@gmail.com>>
> added the comment:
> >
> > I'd like to point out that on some configurations (at least mine), you
> > really need to specify /MANIFEST option to the linker, even though MSDN
> > documentation seems to imply that /MANIFEST behavior is ON by default.
> > My config:
> > beta version of Windows 7
> > ActivePython 2.6.1.1
> > MSVS 2008 with 32-bit C/C++ Optimizing Compiler Version 15.00.21022.08
> > for 80x86
>
> Are you sure ?
>
> We've had such a request before and the reason for MSVC not generating
> a .manifest file was some setting the user had done on his system.
>
> FWIW: distutils generates those files just fine for me.
>
> Then again, it probably doesn't hurt just adding the option.
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue4431>
> _______________________________________
>
msg82392 - (view) Author: Pavel Repin (paxan) Date: 2009-02-18 00:06
Marc-Andre,
I can also repro this on Vista machine with the same Python & MSVS
configuration.
This is the build error stemming from missing manifest file
(unnecessary details elided; note the absence of /MANIFEST option):
C:\src\MySQL-python-1.2.2>python setup.py -v bdist_egg
...
...\link.exe /DLL /nologo /INCREMENTAL:NO "/LIBPATH:C:\Program
Files\MySQL\MySQL Server 5.0\lib\opt" /LIBPATH:C:\Python26\libs
/LIBPATH:C:\Python26\PCbuild kernel32.lib advapi32.lib wsock32.lib
mysqlclient.lib /EXPORT:init_mysql
build\temp.win32-2.6\Release\_mysql.obj
/OUT:build\lib.win32-2.6\_mysql.pyd
/IMPLIB:build\temp.win32-2.6\Release\_mysql.lib
/MANIFESTFILE:build\temp.win32-2.6\Release\_mysql.pyd.manifest
  Creating library build\temp.win32-2.6\Release\_mysql.lib and object
build\temp.win32-2.6\Release\_mysql.exp
...
...\mt.exe -nologo -manifest
build\temp.win32-2.6\Release\_mysql.pyd.manifest
-outputresource:build\lib.win32-2.6\_mysql.pyd;2
build\temp.win32-2.6\Release\_mysql.pyd.manifest : general error
c1010070: Failed to load and parse the manifest. The system cannot
find the file specified.
error: command 'mt.exe' failed with exit status 31

The error goes away when I hack msvc9compiler.py to have:
ld_args.append('/MANIFEST')

See patch attached to the bug.

On Tue, Feb 17, 2009 at 2:34 PM, Pavel Repin <report@bugs.python.org> wrote:
>
> Pavel Repin <prepin+pythonbugs@gmail.com> added the comment:
>
> Hi Marc,I am pretty sure it helped on my particular configuration.
> I was trying to compile MySQL-python-1.2.2.tar.gz package from source and it
> was failing in mt.exe step because the manifest file was not there.
> I didn't do anything special on my machine. I have 3 versions of MSVS
> installed cleanly side-by-side (which is a legit scenario): MSVS 2003, MSVS
> 2005, and MSVS 2008. All at the latest patch levels. No standalone
> PlatformSDKs are installed.
>
> Besides, if you just ignore the obscure blurb about /MANIFEST option being
> default in MSDN, and read the descriptions of both /MANIFEST and
> /MANIFESTFILE options, you would agree that it will not do any harm to be
> explicit and always have /MANIFEST option passed to linker.
>
> I'm going to do one more experiment with a different machine, this time it
> will be Vista (not W7) with a similar MSVS setup. I will report my findings.
>
> On Tue, Feb 17, 2009 at 2:08 PM, Marc-Andre Lemburg
> <report@bugs.python.org>wrote:
>
> >
> > Marc-Andre Lemburg <mal@egenix.com> added the comment:
> >
> > On 2009-02-17 20:22, Pavel Repin wrote:
> > > Pavel Repin <prepin+pythonbugs@gmail.com <prepin%2Bpythonbugs@gmail.com>>
> > added the comment:
> > >
> > > I'd like to point out that on some configurations (at least mine), you
> > > really need to specify /MANIFEST option to the linker, even though MSDN
> > > documentation seems to imply that /MANIFEST behavior is ON by default.
> > > My config:
> > > beta version of Windows 7
> > > ActivePython 2.6.1.1
> > > MSVS 2008 with 32-bit C/C++ Optimizing Compiler Version 15.00.21022.08
> > > for 80x86
> >
> > Are you sure ?
> >
> > We've had such a request before and the reason for MSVC not generating
> > a .manifest file was some setting the user had done on his system.
> >
> > FWIW: distutils generates those files just fine for me.
> >
> > Then again, it probably doesn't hurt just adding the option.
> >
> > _______________________________________
> > Python tracker <report@bugs.python.org>
> > <http://bugs.python.org/issue4431>
> > _______________________________________
> >
>
> Added file: http://bugs.python.org/file13126/unnamed
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue4431>
> _______________________________________
msg119905 - (view) Author: David Joy (David.Joy) Date: 2010-10-29 14:59
Hi All,

I just built mysql-python against CPython2.7 MSVC2008 Express Edition and Server 2003 R2.  All were freshly built on a clean Server 2003 install.  This exact issue occurred building with pip 0.8.1 on top of distribute 0.6.14:

C:\Program Files\Microsoft SDKs\Windows\v6.0A\bin\mt.exe -nologo -manifest build\temp.win32-2.7\Release\_mysql.pyd.manifest -outputresource:build\lib.win32-2.7\_mysql.pyd;2

build\temp.win32-2.7\Release\_mysql.pyd.manifest : general error c1010070: Failed to load and parse the manifest. The system cannot find the file specified.

error: command 'mt.exe' failed with exit status 31

----------------------------------------
Command C:\Python27\python.exe -c "import  setuptools;__file__='C:\\Documents and  Settings\\Administrator\\build\\mysql-python\\setup.py';execfile(__file__)" install --single-version-externally-managed --record c:\docume~1\admini~1\locals~1\temp\pip-qqb1ax-record\install-record.txt failed with error code 1
Storing complete log in C:\Documents and Settings\Administrator\Application Data\pip\pip.log

Pavel's patch fixes my build.  Does this patch break something else?  I can reproduce this on 2.7 and 2.6.6.
msg119906 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2010-10-29 15:09
Hi David,

since both you and Pavel are building mysql-python and using
setuptools (which applies a lot of hacks on stock distutils),
could you please also try some other package from PyPI in that
same configuration and preferably one which doesn't rely on
setuptools ?

Adding the switch per default will probably not cause much harm,
except when you explicitly don't want the manifest to be created
and added to the DLL (which is needed in some situations as as well).

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
msg120123 - (view) Author: David Joy (David.Joy) Date: 2010-11-01 13:13
Hi Marc,

Well, I fried my original server install trying to trace this.  My new fresh install can still reproduce the problem with mysql-python, but I can't recreate the issue with PyOpenSSL anymore.  Grabbing packages at random from PyPi hasn't given me a single issue so far, so I'm going to say this is an issue with mysql-python only and redirect my efforts there.

Thanks for your help,
-David
msg126917 - (view) Author: Gregory Czajkowski (gcflymoto) Date: 2011-01-24 07:23
Also happening with python2.6 and building pyzmq-2.0.10 using easy_install. dschnur's suggestion fixes it.
msg132962 - (view) Author: Santoso Wijaya (santa4nt) * Date: 2011-04-04 19:34
And also with an extension module I'm trying to build with Python-2.7.1 AMD64. Schnur's suggestion fixes it.
msg132966 - (view) Author: Santoso Wijaya (santa4nt) * Date: 2011-04-04 20:25
Another workaround is by adding the linker argument to Extension() as extra_link_args:

    extra_link_args=['/MANIFEST']
msg143721 - (view) Author: Sébastien Sablé (sable) Date: 2011-09-08 09:24
I encountered the same problem while compiling the psutil package.

I used Python 3.2.2 that I compiled myself using Visual Studio 2010 SP1 in Debug mode by following the PCBuild/readme.txt documentation.

I could not compile psutil until I added in setup.py:
extra_link_args=['/MANIFEST']

I have not tried to compile any other C extension yet, but this is not mysql-python and I am not using setuptools either.
msg145390 - (view) Author: Sébastien Sablé (sable) Date: 2011-10-12 09:56
I got this problem again this morning while compiling pywin32.

This problem is not specific to me, anyone using Visual Studio 2010 to compile Python is experiencing the same issue:

http://nukeit.org/compile-python-2-7-packages-with-visual-studio-2010-express/
https://bitbucket.org/jaraco/jaraco.develop/src/f6e937d98e7f/jaraco/develop/msvc.py

Could you please reopen the issue or should I open a new one?
Thanks in advance
msg145394 - (view) Author: David Schnur (DNS) Date: 2011-10-12 14:26
I haven't commented since opening this issue, but I've been following along.  Regarding Marc-Andre's latest comment, I think whether to embed a manifest or not is a separate issue.  The current behavior is to embed a manifest, and so it should ensure that the file is created.  As others have said, given that distutils expects /MANIFEST implicitly, there seems no harm in adding it explicitly.

I agree that this issue should be re-opened, since it's apparently not restricted just to my case of using /MT.
msg145510 - (view) Author: Corey O'Brien (coreypobrien) Date: 2011-10-14 12:10
Building py2exe with VS2010 I had this same issue and the /MANIFEST fix mentioned here fixed the problem.

I also think that this issue should be re-opened.
msg145574 - (view) Author: Mark Hammond (mhammond) * (Python committer) Date: 2011-10-15 00:13
My experience is that for VS2008 at least, the /MANIFESTFILE: option seems to be ignored if there is nothing to put in the manifest, and this tends to be true if you use a static CRT instead of the DLL based one (ie, if you use /MT)

Issue 7833 has a patch designed to both (a) remove the manifest entirely if the only assembly reference is to the CRT and (b) give the setup.py author finer control over this behaviour.
msg145587 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-10-15 11:20
Corey, Sebastian: VS 2010 is not supported. So failure to work correctly is not a bug in Python.
msg145781 - (view) Author: Sébastien Sablé (sable) Date: 2011-10-18 08:29
I need Python compiled with Visual Studio 2010 because I use it embedded in an application compiled with Visual Studio 2010.

I am pretty sure there are quite a lot of people who would like to compile Python with this compiler.

I have compiled Python 2.7.2 with this version of the compiler as well as many extensions, but this required to patch quite a few things. I also intend to setup a buildbot running Python tests using this compiler.

If I can provide the patch to support Visual Studio 2010 and setup a builtbot that will pass most of the tests, could we get Python supported on this platform?

I intended to announce all of that on python-dev soon, but the buildbot is not quite ready yet.
msg145784 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-10-18 08:43
> If I can provide the patch to support Visual Studio 2010 and setup a
> builtbot that will pass most of the tests, could we get Python
> supported on this platform?

Yes. Even if VS2008 remains the preferred compiler to build Python, and the only one used for official binary distributions, there are already project files for VS8 and even VC6 (in PC/VS8.0 and PC/VC6).  They are not guaranteed to always work, but are updated from time to time by volunteers.
Please open another issue.
msg145792 - (view) Author: Mark Hammond (mhammond) * (Python committer) Date: 2011-10-18 10:21
I don't think a buildbot will be necessary - like the earlier compilers, they may have basic support but they don't all get buildbot support.  The "problem" isn't the lack of ability/will to get things working with VS2010 - it is more the lack of incentive to add another compiler to the mix.  OTOH, seeing it is nearly 2012, a case could certainly be made to move 3.3 to that compiler :)

I'm interested to see the patch required to get it working - almost every other upgrade has been almost handled completely by the VS upgrade process so it would be good to see how different this one is - so please CC me on any new issue you create.

I'm also interested to know how mixing the CRTs is causing problems - are you using the very few Python APIs impacted by such mixing, or is the issue more about needing to redistribute both, or something else entirely?
msg145804 - (view) Author: Sébastien Sablé (sable) Date: 2011-10-18 13:01
Amaury:
Great! I have opened issue 13210 to achieve that.

Mark:
Mixing the 2 CRTs results in plenty of crashes in our application. The application also uses a lot of Qt. It is not an option to use a prebuilt Python binary generated with VS2008.

I added you to issue 13210 and I will post the patches there soon.
msg211259 - (view) Author: Stephen Hansen (ixokai) Date: 2014-02-15 05:35
FYI. Windows 8.1, Visual Studio 2010 SP1 Pro just installed, Python 3.3.3; a random extension did this as a 'test' in its setup.py:

    compiler = distutils.ccompiler.new_compiler()
    if not compiler.has_function('rand', includes = ['stdlib.h']):
        ...

And this failed. Further investigation turned brought me here, and adding /MANIFEST to my ld_args as this patch does fixed it.
History
Date User Action Args
2014-02-15 05:35:15ixokaisetnosy: + ixokai
messages: + msg211259
2011-10-18 13:01:04sablesetmessages: + msg145804
2011-10-18 10:21:24mhammondsetmessages: + msg145792
2011-10-18 08:43:54amaury.forgeotdarcsetmessages: + msg145784
2011-10-18 08:29:07sablesetmessages: + msg145781
2011-10-15 11:20:07loewissetmessages: + msg145587
2011-10-15 00:13:21mhammondsetmessages: + msg145574
2011-10-14 15:55:58eric.araujosetnosy: + mhammond, eric.araujo
title: Distutils MSVC doesn't create manifest file (with fix) -> Distutils MSVC doesn't create manifest file

versions: + Python 3.3, - Python 2.6
2011-10-14 12:10:23coreypobriensetnosy: + coreypobrien
messages: + msg145510
2011-10-12 14:26:38DNSsetmessages: + msg145394
2011-10-12 09:56:34sablesetmessages: + msg145390
2011-09-08 09:24:47sablesetnosy: + sable

messages: + msg143721
versions: + Python 2.6, Python 3.2
2011-04-04 20:25:04santa4ntsetmessages: + msg132966
2011-04-04 19:34:58santa4ntsetnosy: + santa4nt
messages: + msg132962
2011-01-24 07:23:01gcflymotosetnosy: + gcflymoto
messages: + msg126917
2010-11-01 13:13:20David.Joysetmessages: + msg120123
2010-10-29 15:14:00lemburgsetfiles: - unnamed
2010-10-29 15:09:53lemburgsetmessages: + msg119906
2010-10-29 14:59:22David.Joysetnosy: + David.Joy

messages: + msg119905
versions: + Python 2.7, - Python 2.6
2009-02-18 00:06:50paxansetmessages: + msg82392
2009-02-17 22:34:26paxansetfiles: + unnamed
messages: + msg82379
2009-02-17 22:08:20lemburgsetmessages: + msg82377
2009-02-17 19:22:14paxansetfiles: + 0001-Ensure-the-assembly-manifest-file-generation-is-gene.patch
nosy: + paxan
messages: + msg82358
keywords: + patch
2008-11-25 22:20:10loewissetstatus: open -> closed
resolution: not a bug
messages: + msg76434
2008-11-25 21:39:47DNSsetmessages: + msg76432
2008-11-25 21:20:52lemburgsetmessages: + msg76430
2008-11-25 21:17:12lemburgsetnosy: + lemburg
messages: + msg76429
2008-11-25 20:53:47amaury.forgeotdarcsetassignee: loewis
messages: + msg76426
nosy: + amaury.forgeotdarc, loewis
2008-11-25 20:16:33DNScreate