diff --git a/PCbuild/readme.txt b/PCbuild/readme.txt --- a/PCbuild/readme.txt +++ b/PCbuild/readme.txt @@ -1,320 +1,317 @@ -Building Python using VC++ 10.0 -------------------------------- +Building Python using Microsoft Visual C++ +------------------------------------------ -This directory is used to build Python for Win32 and x64 platforms, e.g. -Windows 2000, XP, Vista and Windows Server 2008. In order to build 32-bit -debug and release executables, Microsoft Visual C++ 2010 Express Edition is -required at the very least. In order to build 64-bit debug and release -executables, Visual Studio 2010 Standard Edition is required at the very -least. In order to build all of the above, as well as generate release builds -that make use of Profile Guided Optimisation (PG0), Visual Studio 2010 -Professional Edition is required at the very least. The official Python -releases are built with this version of Visual Studio. +This directory is used to build CPython for Microsoft Windows NT version +5.1 or higher (Windows XP, Windows Server 2003, or later) on 32 and 64 +bit platforms. Using this directory requires an installation of +Microsoft Visual C++ 2010 (MSVC 10.0) of any edition. The specific +requirements are as follows: +Visual C++ 2010 Express Edition + Required for building 32-bit Debug and Release configuration builds. + This edition does not support "solution folders", which pcbuild.sln + uses; this will not prevent building. +Visual Studio 2010 Standard Edition + Required for building 64-bit Debug and Release configuration builds +Visual Studio 2010 Professional Edition + Required for building Release configuration builds that make use of + Profile Guided Optimization (PGO), on either platform. The official + Python releases are built with Professional Edition using PGO. -For other Windows platforms and compilers, see ../PC/readme.txt. +All you need to do to build is open the solution "pcbuild.sln" in Visual +Studio, select the desired combination of configuration and platform, +then build with "Build Solution" or the F7 keyboard shortcut. You can +also build from the command line using the "build.bat" script in this +directory. The solution is configured to build the projects in the +correct order. -All you need to do is open the workspace "pcbuild.sln" in Visual Studio, -select the desired combination of configuration and platform and eventually -build the solution. Unless you are going to debug a problem in the core or -you are going to create an optimized build you want to select "Release" as -configuration. +The solution currently supports two platforms. The Win32 platform is +used to build standard x86-compatible 32-bit binaries, output into this +directory. The x64 platform is used for building 64-bit AMD64 (aka +x86_64 or EM64T) binaries, output into the amd64 sub-directory which +will be created if it doesn't already exist. The Itanium (IA-64) +platform is no longer supported. See the "Building for AMD64" section +below for more information about 64-bit builds. -The PCbuild directory is compatible with all versions of Visual Studio from -VS C++ Express Edition over the standard edition up to the professional -edition. However the express edition does not support features like solution -folders or profile guided optimization (PGO). The missing bits and pieces -won't stop you from building Python. +Four configuration options are supported by the solution: +Debug + Used to build Python with extra debugging capabilities, equivalent + to using ./configure --with-pydebug on UNIX. All binaries built + using this configuration have "_d" added to their name: + python34_d.dll, python_d.exe, parser_d.pyd, and so on. Both the + build and rt (run test) batch files in this directory accept a -d + option for debug builds. If you are building Python to help with + development of CPython, you will most likely use this configuration. +PGInstrument, PGUpdate + Used to build Python in Release configuration using PGO, which + requires Professional Edition of Visual Studio. See the "Profile + Guided Optimization" section below for more information. Build + output from each of these configurations lands in its own + sub-directory of this directory. The official Python releases are + built using these configurations. +Release + Used to build Python as it is meant to be used in production + settings, though without PGO. -The solution is configured to build the projects in the correct order. "Build -Solution" or F7 takes care of dependencies except for x64 builds. To make -cross compiling x64 builds on a 32bit OS possible the x64 builds require a -32bit version of Python. - -NOTE: - You probably don't want to build most of the other subprojects, unless - you're building an entire Python distribution from scratch, or - specifically making changes to the subsystems they implement, or are - running a Python core buildbot test slave; see SUBPROJECTS below) - -When using the Debug setting, the output files have a _d added to -their name: python34_d.dll, python_d.exe, parser_d.pyd, and so on. Both -the build and rt batch files accept a -d option for debug builds. - -The 32bit builds end up in the solution folder PCbuild while the x64 builds -land in the amd64 subfolder. The PGI and PGO builds for profile guided -optimization end up in their own folders, too. Legacy support -------------- -You can find build directories for older versions of Visual Studio and -Visual C++ in the PC directory. The legacy build directories are no longer -actively maintained and may not work out of the box. +You can find build directories for older versions of Visual Studio and +Visual C++ in the PC directory. The legacy build directories are no +longer actively maintained and may not work out of the box. -PC/VC6/ - Visual C++ 6.0 -PC/VS7.1/ - Visual Studio 2003 (7.1) -PC/VS8.0/ - Visual Studio 2005 (8.0) -PC/VS9.0/ - Visual Studio 2008 (9.0) +Currently, the only legacy build directory is PC\VS9.0, for Visual +Studio 2008 (9.0). -C RUNTIME +C Runtime --------- -Visual Studio 2010 uses version 10 of the C runtime (MSVCRT9). The executables -no longer use the "Side by Side" assemblies used in previous versions of the -compiler. This simplifies distribution of applications. -The run time libraries are avalible under the VC/Redist folder of your visual studio -distribution. For more info, see the Readme in the VC/Redist folder. +Visual Studio 2010 uses version 10 of the C runtime (MSVCRT10). The +executables no longer use the "Side by Side" assemblies used in previous +versions of the compiler. This simplifies distribution of applications. -SUBPROJECTS ------------ -These subprojects should build out of the box. Subprojects other than the -main ones (pythoncore, python, pythonw) generally build a DLL (renamed to -.pyd) from a specific module so that users don't have to load the code -supporting that module unless they import the module. +The run time libraries are available under the VC/Redist folder of your +Visual Studio distribution. For more info, see the Readme in the +VC/Redist folder. + +Sub-Projects +------------ + +The CPython project is split up into several smaller sub-projects which +are managed by the pcbuild.sln solution file. These sub-projects fall +into a few general categories: + +The following sub-projects represent the bare minimum required to build +a functioning CPython interpreter. If nothing else builds but these, +you'll have a very limited but usable python.exe: pythoncore .dll and .lib python .exe +kill_python + kill_python.exe, a small program designed to kill any instances of + python(_d).exe that are running and live in the build output + directory; this is meant to avoid build issues due to locked files +make_buildinfo, make_versioninfo + helpers to provide necessary information to the build process + +These sub-projects provide extra executables that are useful for running +CPython in different ways: pythonw - pythonw.exe, a variant of python.exe that doesn't pop up a DOS box + pythonw.exe, a variant of python.exe that doesn't open a Command + Prompt window +pylauncher + py.exe, the Python Launcher for Windows, see + http://docs.python.org/3/using/windows.html#launcher +pywlauncher + pyw.exe, a variant of py.exe that doesn't open a Command Prompt + window + +These are miscellaneous sub-projects that don't really fit the other +categories. By default, these projects do not build in Debug +configuration: +_freeze_importlib + _freeze_importlib.exe, used to regenerate Python\importlib.h after + changes have been made to Lib\importlib\_bootstrap.py +bdist_wininst + ..\Lib\distutils\command\wininst-10.0[-amd64].exe, the base + executable used by the distutils bdist_wininst command +python3dll + python3.dll, the PEP 384 Stable ABI dll +xxlimited + builds an example module that makes use of the PEP 384 Stable ABI, + see Modules\xxlimited.c + +The following sub-projects are for individual modules of the standard +library which are implemented in C; each one builds a DLL (renamed to +.pyd) of the same name as the project: +_ctypes +_ctypes_test +_decimal +_elementtree +_hashlib +_msi +_multiprocessing +_sha3 _socket - socketmodule.c _testcapi - tests of the Python C API, run via Lib/test/test_capi.py, and - implemented by module Modules/_testcapimodule.c _testbuffer - buffer protocol tests, run via Lib/test/test_buffer.py, and - implemented by module Modules/_testbuffer.c +_testimportmultiple pyexpat - Python wrapper for accelerated XML parsing, which incorporates stable - code from the Expat project: http://sourceforge.net/projects/expat/ select - selectmodule.c unicodedata - large tables of Unicode data winsound - play sounds (typically .wav files) under Windows -Python-controlled subprojects that wrap external projects: -_sqlite3 - Wraps SQLite 3.7.4, which is currently built by sqlite3.vcproj (see below). -_tkinter - Wraps the Tk windowing system. Unlike _sqlite3, there's no - corresponding tcltk.vcproj-type project that builds Tcl/Tk from vcproj's - within our pcbuild.sln, which means this module expects to find a - pre-built Tcl/Tk in either ..\..\tcltk for 32-bit or ..\..\tcltk64 for - 64-bit (relative to this directory). See below for instructions to build - Tcl/Tk. +The following Python-controlled sub-projects wrap external projects. +Note that these external libraries are not necessary for a working +interpreter, but they do implement several major features. See the +"Getting External Sources" section below for additional information +about getting the source for building these libraries. The sub-projects +are: _bz2 - Python wrapper for the libbzip2 compression library. Homepage + Python wrapper for version 1.0.6 of the libbzip2 compression library + Homepage: http://www.bzip.org/ - Download the source from the python.org copy into the dist - directory: +_lzma + Python wrapper for the liblzma compression library, using pre-built + binaries of XZ Utils version 5.0.3 + Homepage: + http://tukaani.org/xz/ +_ssl + Python wrapper for version 1.0.1e of the OpenSSL secure sockets + library, which is built by ssl.vcxproj + Homepage: + http://www.openssl.org/ - svn export http://svn.python.org/projects/external/bzip2-1.0.6 + Building OpenSSL requires nasm.exe (the Netwide Assembler), version + 2.10 or newer from + http://www.nasm.us/ + to be somewhere on your PATH. More recent versions of OpenSSL may + need a later version of NASM. If OpenSSL's self tests don't pass, + you should first try to update NASM and do a full rebuild of + OpenSSL. - ** NOTE: if you use the Tools\buildbot\external(-amd64).bat approach for - obtaining external sources then you don't need to manually get the source - above via subversion. ** + If you like to use the official sources instead of the files from + python.org's subversion repository, Perl is required to build the + necessary makefiles and assembly files. ActivePerl is available + from + http://www.activestate.com/activeperl/ + The svn.python.org version contains pre-built makefiles and assembly + files. - A custom pre-link step in the bz2 project settings should manage to - build bzip2-1.0.6\libbz2.lib by magic before bz2.pyd (or bz2_d.pyd) is - linked in PCbuild\. - However, the bz2 project is not smart enough to remove anything under - bzip2-1.0.6\ when you do a clean, so if you want to rebuild bzip2.lib - you need to clean up bzip2-1.0.6\ by hand. + The build process makes sure that no patented algorithms are + included. For now RC5, MDC2 and IDEA are excluded from the build. + You may have to manually remove $(OBJ_D)\i_*.obj from ms\nt.mak if + using official sources; the svn.python.org-hosted version is already + fixed. - All of this managed to build libbz2.lib in - bzip2-1.0.6\$platform-$configuration\, which the Python project links in. -_lzma - Python wrapper for the liblzma compression library. - - Download the pre-built Windows binaries from http://tukaani.org/xz/, and - extract to ..\xz-5.0.3. If you are using a more recent version of liblzma, - it will be necessary to rename the directory from xz- to xz-5.0.3. - -_ssl - Python wrapper for the secure sockets library. - - Get the source code through - - svn export http://svn.python.org/projects/external/openssl-1.0.1e - - ** NOTE: if you use the Tools\buildbot\external(-amd64).bat approach for - obtaining external sources then you don't need to manually get the source - above via subversion. ** - - Alternatively, get the latest version from http://www.openssl.org. - You can (theoretically) use any version of OpenSSL you like - the - build process will automatically select the latest version. - - You must install the NASM assembler 2.10 or newer from - http://nasm.sf.net - for x86 builds. Put nasmw.exe anywhere in your PATH. More recent - versions of OpenSSL may need a later version of NASM. If OpenSSL's self - tests don't pass, you should first try to update NASM and do a full - rebuild of OpenSSL. - Note: recent releases of nasm only have nasm.exe. Just rename it to - nasmw.exe. - - You can also install ActivePerl from - http://www.activestate.com/activeperl/ - if you like to use the official sources instead of the files from - python's subversion repository. The svn version contains pre-build - makefiles and assembly files. - - The build process makes sure that no patented algorithms are included. - For now RC5, MDC2 and IDEA are excluded from the build. You may have - to manually remove $(OBJ_D)\i_*.obj from ms\nt.mak if the build process - complains about missing files or forbidden IDEA. Again the files provided - in the subversion repository are already fixed. - - The MSVC project simply invokes PCBuild/build_ssl.py to perform - the build. This Python script locates and builds your OpenSSL - installation, then invokes a simple makefile to build the final .pyd. + The ssl.vcxproj sub-project simply invokes PCbuild/build_ssl.py, + which locates and builds OpenSSL. build_ssl.py attempts to catch the most common errors (such as not being able to find OpenSSL sources, or not being able to find a Perl - that works with OpenSSL) and give a reasonable error message. - If you have a problem that doesn't seem to be handled correctly - (eg, you know you have ActivePerl but we can't find it), please take - a peek at build_ssl.py and suggest patches. Note that build_ssl.py + that works with OpenSSL) and give a reasonable error message. If + you have a problem that doesn't seem to be handled correctly (e.g., + you know you have ActivePerl but we can't find it), please take a + peek at build_ssl.py and suggest patches. Note that build_ssl.py should be able to be run directly from the command-line. - build_ssl.py/MSVC isn't clever enough to clean OpenSSL - you must do - this by hand. + The ssl sub-project does not have the ability to clean the OpenSSL + build; if you need to rebuild, you'll have to clean it by hand. +_sqlite3 + Wraps SQLite 3.7.12, which is itself built by sqlite3.vcxproj + Homepage: + http://www.sqlite.org/ +_tkinter + Wraps version 8.5.11 of the Tk windowing system. + Homepage: + http://www.tcl.tk/ -The subprojects above wrap external projects Python doesn't control, and as -such, a little more work is required in order to download the relevant source -files for each project before they can be built. The buildbots do this each -time they're built, so the easiest approach is to run either external.bat or -external-amd64.bat in the ..\Tools\buildbot directory from ..\, i.e.: + Unlike the other external libraries listed above, Tk must be built + separately before the _tkinter module can be built. This means that + a pre-built Tcl/Tk installation is expected in ..\..\tcltk (tcltk64 + for 64-bit) relative to this directory. See "Getting External + Sources" below for the easiest method to ensure Tcl/Tk is built. - C:\..\svn.python.org\projects\python\trunk\PCbuild>cd .. - C:\..\svn.python.org\projects\python\trunk>Tools\buildbot\external.bat -This extracts all the external subprojects from http://svn.python.org/external -via Subversion (so you'll need an svn.exe on your PATH) and places them in -..\.. (relative to this directory). The external(-amd64).bat scripts will -also build a debug build of Tcl/Tk; there aren't any equivalent batch files -for building release versions of Tcl/Tk lying around in the Tools\buildbot -directory. If you need to build a release version of Tcl/Tk it isn't hard -though, take a look at the relevant external(-amd64).bat file and find the -two nmake lines, then call each one without the 'DEBUG=1' parameter, i.e.: +Getting External Sources +------------------------ + +The last category of sub-projects listed above wrap external projects +Python doesn't control, and as such a little more work is required in +order to download the relevant source files for each project before they +can be built. The buildbots must ensure that all libraries are present +before building, so the easiest approach is to run either external.bat +or external-amd64.bat (depending on platform) in the ..\Tools\buildbot +directory from ..\, i.e.: + + C:\python\cpython\PCbuild>cd .. + C:\python\cpython>Tools\buildbot\external.bat + +This extracts all the external sub-projects from + http://svn.python.org/projects/external +via Subversion (so you'll need an svn.exe on your PATH) and places them +in ..\.. (relative to this directory). + +It is also possible to download sources from each project's homepage, +though you may have to change the names of some folders in order to make +things work. For instance, if you were to download a version 5.0.5 of +XZ Utils, you would need to extract the archive into ..\..\xz-5.0.3 +anyway, since that is where the solution is set to look for xz. The +same is true for all other external projects. + +The external(-amd64).bat scripts will also build a debug build of +Tcl/Tk, but there aren't any equivalent batch files for building release +versions of Tcl/Tk currently available. If you need to build a release +version of Tcl/Tk, just take a look at the relevant external(-amd64).bat +file and find the two nmake lines, then call each one without the +'DEBUG=1' parameter, i.e.: The external-amd64.bat file contains this for tcl: - nmake -f makefile.vc COMPILERFLAGS=-DWINVER=0x0500 DEBUG=1 MACHINE=AMD64 INSTALLDIR=..\..\tcltk64 clean all install + nmake -f makefile.vc DEBUG=1 MACHINE=AMD64 INSTALLDIR=..\..\tcltk64 clean all install So for a release build, you'd call it as: - nmake -f makefile.vc COMPILERFLAGS=-DWINVER=0x0500 MACHINE=AMD64 INSTALLDIR=..\..\tcltk64 clean all install + nmake -f makefile.vc MACHINE=AMD64 INSTALLDIR=..\..\tcltk64 clean all install + +Note that the above command is called from within ..\..\tcl-8.5.11.0\win +(relative to this directory); don't forget to build Tk as well as Tcl! - XXX Should we compile with OPTS=threads? - XXX Our installer copies a lot of stuff out of the Tcl/Tk install - XXX directory. Is all of that really needed for Python use of Tcl/Tk? +This will be cleaned up in the future; http://bugs.python.org/issue15968 +tracks adding a new tcltk.vcxproj file that will build Tcl/Tk and Tix +the same way the other external projects listed above are built. -This will be cleaned up in the future; ideally Tcl/Tk will be brought into our -pcbuild.sln as custom .vcproj files, just as we've recently done with the -sqlite3.vcproj file, which will remove the need for Tcl/Tk to be built -separately via a batch file. - -XXX trent.nelson 02-Apr-08: - Having the external subprojects in ..\.. relative to this directory is a - bit of a nuisance when you're working on py3k and trunk in parallel and - your directory layout mimics that of Python's subversion layout, e.g.: - - C:\..\svn.python.org\projects\python\trunk - C:\..\svn.python.org\projects\python\branches\py3k - C:\..\svn.python.org\projects\python\branches\release25-maint - - I'd like to change things so that external subprojects are fetched from - ..\external instead of ..\.., then provide some helper scripts or batch - files that would set up a new ..\external directory with svn checkouts of - the relevant branches in http://svn.python.org/projects/external/, or - alternatively, use junctions to link ..\external with a pre-existing - externals directory being used by another branch. i.e. if I'm usually - working on trunk (and have previously created trunk\external via the - provided batch file), and want to do some work on py3k, I'd set up a - junction as follows (using the directory structure above as an example): - - C:\..\python\trunk\external <- already exists and has built versions - of the external subprojects - - C:\..\python\branches\py3k>linkd.exe external ..\..\trunk\external - Link created at: external - - Only a slight tweak would be needed to the buildbots such that bots - building trunk and py3k could make use of the same facility. (2.5.x - builds need to be kept separate as they're using Visual Studio 7.1.) -/XXX trent.nelson 02-Apr-08 - -Building for Itanium --------------------- - -NOTE: -Official support for Itanium builds have been dropped from the build. Please -contact us and provide patches if you are interested in Itanium builds. - -The project files support a ReleaseItanium configuration which creates -Win64/Itanium binaries. For this to work, you need to install the Platform -SDK, in particular the 64-bit support. This includes an Itanium compiler -(future releases of the SDK likely include an AMD64 compiler as well). -In addition, you need the Visual Studio plugin for external C compilers, -from http://sf.net/projects/vsextcomp. The plugin will wrap cl.exe, to -locate the proper target compiler, and convert compiler options -accordingly. The project files require at least version 0.9. Building for AMD64 ------------------ -The build process for AMD64 / x64 is very similar to standard builds. You just -have to set x64 as platform. In addition, the HOST_PYTHON environment variable -must point to a Python interpreter (at least 2.4), to support cross-compilation. +The build process for AMD64 / x64 is very similar to standard builds, +you just have to set x64 as platform. In addition, the HOST_PYTHON +environment variable must point to a Python interpreter (at least 2.4), +to support cross-compilation from Win32. Note that Visual Studio +requires either Standard Edition or better, or Express Edition with the +Windows SDK 64-bit compilers to be available in order to build 64-bit +binaries. -Building Python Using the free MS Toolkit Compiler --------------------------------------------------- - -Microsoft has withdrawn the free MS Toolkit Compiler, so this can no longer -be considered a supported option. Instead you can use the free VS C++ Express -Edition. Profile Guided Optimization --------------------------- The solution has two configurations for PGO. The PGInstrument -configuration must be build first. The PGInstrument binaries are -lniked against a profiling library and contain extra debug -information. The PGUpdate configuration takes the profiling data and -generates optimized binaries. +configuration must be built first. The PGInstrument binaries are linked +against a profiling library and contain extra debug information. The +PGUpdate configuration takes the profiling data and generates optimized +binaries. -The build_pgo.bat script automates the creation of optimized binaries. It -creates the PGI files, runs the unit test suite or PyBench with the PGI -python and finally creates the optimized files. +The build_pgo.bat script automates the creation of optimized binaries. +It creates the PGI files, runs the unit test suite or PyBench with the +PGI python, and finally creates the optimized files. -http://msdn2.microsoft.com/en-us/library/e7k32f4k(VS.90).aspx +See + http://msdn.microsoft.com/en-us/library/e7k32f4k(VS.100).aspx +for more on this topic. + Static library -------------- -The solution has no configuration for static libraries. However it is easy -it build a static library instead of a DLL. You simply have to set the -"Configuration Type" to "Static Library (.lib)" and alter the preprocessor -macro "Py_ENABLE_SHARED" to "Py_NO_ENABLE_SHARED". You may also have to -change the "Runtime Library" from "Multi-threaded DLL (/MD)" to -"Multi-threaded (/MT)". +The solution has no configuration for static libraries. However it is +easy to build a static library instead of a DLL. You simply have to set +the "Configuration Type" to "Static Library (.lib)" and alter the +preprocessor macro "Py_ENABLE_SHARED" to "Py_NO_ENABLE_SHARED". You may +also have to change the "Runtime Library" from "Multi-threaded DLL +(/MD)" to "Multi-threaded (/MT)". + Visual Studio properties ------------------------ -The PCbuild solution makes heavy use of Visual Studio property files -(*.vsprops). The properties can be viewed and altered in the Property +The PCbuild solution makes heavy use of Visual Studio property files +(*.props). The properties can be viewed and altered in the Property Manager (View -> Other Windows -> Property Manager). +The property files used are (+-- = "also imports"): * debug (debug macro: _DEBUG) * pginstrument (PGO) * pgupdate (PGO) @@ -327,15 +324,18 @@ +-- pyproject * pyproject (base settings for all projects, user macros like PyDllName) * release (release macro: NDEBUG) + * sqlite3 (used only by sqlite3.vcxproj) * x64 (AMD64 / x64 platform specific settings) -The pyproject propertyfile defines _WIN32 and x64 defines _WIN64 and _M_X64 -although the macros are set by the compiler, too. The GUI doesn't always know -about the macros and confuse the user with false information. +The pyproject property file defines _WIN32 and x64 defines _WIN64 and +_M_X64 although the macros are set by the compiler, too. The GUI doesn't +always know about the macros and confuse the user with false +information. -YOUR OWN EXTENSION DLLs + +Your Own Extension DLLs ----------------------- -If you want to create your own extension module DLL, there's an example -with easy-to-follow instructions in ../PC/example/; read the file -readme.txt there first. +If you want to create your own extension module DLL (.pyd), there's an +example with easy-to-follow instructions in ..\PC\example\; read the +file readme.txt there first.