This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author Alex.Stewart
Recipients Alex.Stewart, ned.deily, ronaldoussoren
Date 2011-11-02.16:51:13
SpamBayes Score 1.6714408e-13
Marked as misclassified No
Message-id <1320252674.88.0.834289126042.issue13324@psf.upfronthosting.co.za>
In-reply-to
Content
----------------------------
ISSUE DESCRIPTION: 
----------------------------
In 2.6.7, 2.7.2 and also HEAD (r 73301) the fcntl module does not support the F_NOCACHE OSX specific mode.  

The affect of this is that the default behaviour (data caching enabled) is used when parsing files etc.  When data caching is enabled, OSX caches the data parsed from a file by python, keeping it available as 'inactive memory' even when it has been freed by python.  In *theory*, this should be fine as OSX *should* recycle the inactive memory as it is required.  

Unfortunately, at least under OSX 10.6.8 (and it seems 10.7) the system will do almost anything to avoid recycling inactive memory, including swallowing up all 'pure' free memory and then paging manically to the disk.  The net affect of this is significantly degraded system performance.

For most users, operating with relatively small files and a large quantity of RAM this issue is probably not that obvious.  However, for various reasons I'm working with files of 5-125+GB and thus it rapidly becomes a major issue.

----------------------------
FIX
----------------------------
Very simply, all the attached patch does it add support for F_NOCACHE to fcntl just like F_FULLFSYNC (another fcntl OSX specific flag) - it's a trivial change that allows you to control whether OSX uses data caching on a file handle, e.g, the following turns OFF data caching for the specified file:

fcntl.fcntl(theFile.fileno(), fcntl.F_NOCACHE, 1)

With this patch in place the inactive memory on my system stays (low) and flat, without it it rises until it maxes out all available memory and then starts paging.
History
Date User Action Args
2011-11-02 16:51:14Alex.Stewartsetrecipients: + Alex.Stewart, ronaldoussoren, ned.deily
2011-11-02 16:51:14Alex.Stewartsetmessageid: <1320252674.88.0.834289126042.issue13324@psf.upfronthosting.co.za>
2011-11-02 16:51:14Alex.Stewartlinkissue13324 messages
2011-11-02 16:51:13Alex.Stewartcreate