classification
Title: Python should use 3GB Address Space on Windows
Type: enhancement Stage:
Components: Windows Versions: Python 3.2
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: Chad.Austin, Jason.Scheirer, amaury.forgeotdarc, brian.curtin, gfe, josiahcarlson, loewis, pitrou, tim.golden, tim.peters
Priority: normal Keywords: patch

Created on 2006-03-14 11:04 by gfe, last changed 2011-05-19 21:39 by Chad.Austin. This issue is now closed.

Files
File name Uploaded Description Edit
Issue1449496.diff Jason.Scheirer, 2010-12-16 22:17
Messages (14)
msg61220 - (view) Author: Martin Gfeller Martin Gfeller (gfe) Date: 2006-03-14 11:04
Python runs fine using a 3GB address space on 32bit 
Windows.

To take advantage of this feature, Python should be 
linked using the /LARGEADDRESSAWARE flag of the 
linker. 

For details, please refer to:

http://msdn.microsoft.com/library/en-
us/memory/base/4gt_ram_tuning.asp

As most Windows users just use pre-built executables, 
this would be a worthwhile change. 

Best regards,
Martin Gfeller

msg61221 - (view) Author: Josiah Carlson (josiahcarlson) * Date: 2006-03-18 00:38
Logged In: YES 
user_id=341410

Microsoft claims that the switch does not negatively affect
users on systems without large amounts of memory, and the
extra gig of memory on platforms which could use it would be
nice.

Can you offer a patch?
msg61222 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2006-03-18 02:38
Logged In: YES 
user_id=31435

MS also says "However, the file cache, paged pool, and
non-paged pool are smaller, which can adversely affect
applications with heavy networking or I/O", so the rationale
for making this change is clear as mud.  That is, it's not a
pure win: some apps win at the expense of others.
msg61223 - (view) Author: Martin Gfeller Martin Gfeller (gfe) Date: 2006-03-20 12:08
Logged In: YES 
user_id=884167

Tim, the /LARGEADDRESSAWARE flag just specifies whether 
python.exe is enabled for #GB addresses, i.e. doesn't use 
dubious pointer manipulations that make it fail with a 3GB 
address space. 

It does not cause Windows to provide the 3GB address space, 
nor to reduce cache and pool size - that is controlled 
separately by the /3GB boot switch (and various registry 
settings for finer control of cache and pool sizes).

In other words, it is an enabling declaration, not a 
control parameter. 

Josiah, I don't compile on Windows, hence cannot provide a 
patch. What I did is applying the flag to the executable by 
using editbin.exe. 

Brgds, Martin
msg61224 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2006-03-21 00:54
Logged In: YES 
user_id=31435

Thanks, Martin.  Raymond Chen notes that some C code can
fail when this is enabled:

http://blogs.msdn.com/oldnewthing/archive/2004/08/12/213468.aspx

I doubt that core Python's C has 2GB assumptions, but
extension modules might.  Provided that whoever cares enough
to write the patch (sorry, I don't) points to that info (in
the NEWS file would be good enough), this is fine by me.
msg112751 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2010-08-04 04:09
Unless anyone plans on a patch I say we let this go. As of 2.5 we provide x64 installers, and with most users running on 64-bit OS'es I would say it's better to suggest they use a 64-bit compiled Python to obtain an even larger addressable space.

Of course, not all users are on 64-bit, but it'll take some work and tests for this to be considered for 3.2.
msg124161 - (view) Author: Jason Scheirer (Jason.Scheirer) Date: 2010-12-16 22:17
The patch is a simple tweak to the Visual Studio project's compiler settings. Seems quite innocuous, nothing broke in the Python harness or our internal harnesses (at ESRI) by enabling this.
msg124169 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-12-17 00:44
I'm rejecting this as "won't fix". Users having the underlying problem (i.e. need more than 2GB of heap) really should switch to a 64-bit installation. The risk of something breaking is just not worth it.

That testing showed no issues is not convincing at all. Existing test suites will likely not exercise critical code paths wrt. this change, which are issues caused by some objects potentially exceeding 2GB in 32-bit mode. This might not even be Python core objects, but may be objects in an extension module.
msg124198 - (view) Author: Martin Gfeller Martin Gfeller (gfe) Date: 2010-12-17 12:50
Martin, we're running with this for years and with many extensions modules, without an issue. What is 64-bit safe should be 32-bit safe, not only 31-bit safe. But you're right, this is not a proof, and we have switched to 64-bit ourselves.
msg124200 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2010-12-17 13:06
> What is 64-bit safe should be 32-bit safe, not only 31-bit safe

Not here. Python uses "signed size_t" for various lengths and sizes.
On win32 this only gives you 31 bits...
msg124262 - (view) Author: Jason Scheirer (Jason.Scheirer) Date: 2010-12-17 22:56
I would like to see this reopened: we have a very large class of users that are not ready to entirely port to 64-bit and need this now.

When running a Python script from within a desktop application (which embeds Python and has /LARGEADDRESSAWARE set) or outside in Python.exe (which does not) results in the outputs from the tools calling out to these 32 bit libraries to produce different outputs because the amount of data they are able to allocate and process at the same time. The same Python script that gives correct output on a large dataset in this desktop app will not yield the same quality of results when run overnight as a stand-alone Python script. Essentially this discounts Python scripts as an option for automation in this case.

I understand that yes, applications still cannot allocate more CONTIGUOUS memory, but this is not a regression if it continues to be so. Allowing the process to allocate more TOTAL memory is a net benefit, especially on on complex software systems that can't simply be rebuilt in 64 bit mode due to third-party restraints (tied to a specific library version, lack of access to source, lengthy software approvals processes, etc.)
msg124263 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-12-17 23:13
> When running a Python script from within a desktop application (which
> embeds Python and has /LARGEADDRESSAWARE set) [...]. Essentially this
> discounts Python scripts as an option for automation in this case.

Well, if you already embed Python in your application, you can probably also embed it in a simple console application which will enable automation, can't you?
msg124270 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-12-18 02:52
> I would like to see this reopened: we have a very large class of
> users that are not ready to entirely port to 64-bit and need this
> now.

And I remain -1 to such requests. You can appeal to that by writing a
PEP, or finding a committer who is willing to make this change.

Alternatively, just provide these users with binaries that you built
yourself, or hire somebody to build such binaries for you.

> I understand that yes, applications still cannot allocate more
> CONTIGUOUS memory, but this is not a regression if it continues to be
> so. 

It would be a regression if Python started crashing because of that
change, which is my concern.
msg136337 - (view) Author: Chad Austin (Chad.Austin) Date: 2011-05-19 21:39
IMVU's downloadable client is built atop CPython.  80% of our users use 32-bit Windows and 20% use 64-bit Windows.  We do not intend to provide 64-bit builds of our application for many of the same reasons sketched out in http://blogs.msdn.com/b/ricom/archive/2009/06/10/visual-studio-why-is-there-no-64-bit-version.aspx

Process address space is one of our key bottlenecks: limiting the amount of cache we can mmap in, various runtime heap reserves, etc.  For those 20% of users running 64-bit Windows, /LARGEADDRESSAWARE would give 32-bit Python access to 4 GB of address space, which would certainly help them.

The other reason not to use 64-bit binaries is that they consume significantly more memory and cache, especially because Python is so pointer-heavy.

In the meantime, we can use editbin /LARGEADDRESSAWARE to modify the shipped .exes.
History
Date User Action Args
2011-05-19 21:39:05Chad.Austinsetnosy: + Chad.Austin
messages: + msg136337
2010-12-18 02:52:59loewissetnosy: tim.peters, loewis, josiahcarlson, amaury.forgeotdarc, pitrou, gfe, tim.golden, brian.curtin, Jason.Scheirer
messages: + msg124270
2010-12-17 23:13:48pitrousetnosy: + pitrou
messages: + msg124263
2010-12-17 22:56:27Jason.Scheirersetnosy: tim.peters, loewis, josiahcarlson, amaury.forgeotdarc, gfe, tim.golden, brian.curtin, Jason.Scheirer
messages: + msg124262
2010-12-17 13:06:48amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg124200
2010-12-17 12:50:24gfesetnosy: tim.peters, loewis, josiahcarlson, gfe, tim.golden, brian.curtin, Jason.Scheirer
messages: + msg124198
2010-12-17 00:44:29loewissetstatus: open -> closed

messages: + msg124169
resolution: out of date -> wont fix
nosy: tim.peters, loewis, josiahcarlson, gfe, tim.golden, brian.curtin, Jason.Scheirer
2010-12-17 00:27:47pitrousetnosy: tim.peters, loewis, josiahcarlson, gfe, tim.golden, brian.curtin, Jason.Scheirer
stage: test needed ->
versions: + Python 3.2, - Python 3.1, Python 2.7
2010-12-17 00:27:28pitrousetnosy: + loewis, tim.golden
2010-12-16 22:17:36Jason.Scheirersetfiles: + Issue1449496.diff
status: pending -> open

nosy: + Jason.Scheirer
messages: + msg124161

keywords: + patch
2010-08-04 04:09:55brian.curtinsetstatus: open -> pending

nosy: + brian.curtin
messages: + msg112751

resolution: out of date
2009-03-21 01:35:50ajaksu2setstage: test needed
versions: + Python 3.1, Python 2.7
2006-03-14 11:04:09gfecreate