Author lorenz
Recipients lorenz
Date 2011-01-12.15:16:14
SpamBayes Score 3.66152e-13
Marked as misclassified No
Message-id <1294845379.76.0.427887707002.issue10897@psf.upfronthosting.co.za>
In-reply-to
Content
The UNIX mmap version in

  Modules/mmapmodule.c 

makes a call to dup() to duplicate the file descriptor that is passed for creating the memory-mapped region.

This way, I frequently hit the limit for the number of open file handles while creating mmap.mmap() instances, despite closing all my opened files after creating a mapping for them.

My application is scientific data (read: "large data" :-) analysis for which mmap() is very well suited together with numpy/scipy - however, the large number of files causes me to hit the resource limit on open file handles)

I propose to remove this dup(), which was apparently introduced in the process of fixing issue #728515, concerning incorrect behavior of the mmap.resize() method on UNIX, as it was feared the user could have closed the file handle already.

I think it should be the responsibility of the user not to close the file in question or either - if it needs to be closed - not to call the resize method later.

I should stress that a call to mmap(2) does not require an open file handle during the time of the mapping but that a duplicate of the file handle was only kept to allow .size() and .resize() to work. See the POSIX Programmer's Manual:

  The mmap() function shall add an extra reference to the file
  associated with the file descriptor fildes which is not removed
  by a subsequent close() on that file descriptor. This reference
  shall be removed when there are no more mappings to the file.

It would be great if this little nicety would translate better to Python!

Regards,
   Lorenz
History
Date User Action Args
2011-01-12 15:16:19lorenzsetrecipients: + lorenz
2011-01-12 15:16:19lorenzsetmessageid: <1294845379.76.0.427887707002.issue10897@psf.upfronthosting.co.za>
2011-01-12 15:16:15lorenzlinkissue10897 messages
2011-01-12 15:16:14lorenzcreate