This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: calling mmap twice fails on Windows
Type: behavior Stage: resolved
Components: Windows Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: brian.curtin, pitrou, tim.golden, vladris, zolnie
Priority: normal Keywords:

Created on 2011-07-14 18:00 by zolnie, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Messages (7)
msg140349 - (view) Author: Piotr Zolnierczuk (zolnie) Date: 2011-07-14 18:00
Hi,
I am trying to migrate from Python 2.5 to Python 2.7 I found though the mmap behaves differently on Windows XP between the two versions. It boils down to the following code:
import mmap
map1 = mmap.mmap(fileno=0, tagname='MyData', length=4096)
map2 = mmap.mmap(fileno=0, tagname='MyData', length=8192)

It runs fine (so I can "resize" shared memory) on XP with 2.5.4, but when running on 2.7.2 I get the following error

Traceback (most recent call last):
  File "D:\Workspace\memmap_test.py", line 3, in <module>
    map2 = mmap.mmap(fileno=0, tagname='MyData', length=8192)
WindowsError: [Error 5] Access is denied
msg141482 - (view) Author: Vlad Riscutia (vladris) Date: 2011-07-31 23:05
Reason you are seeing the failure for this is following change from 2.5 in mmapmodule.c (:1109):

m_obj->data = (char *) MapViewOfFile(m_obj->map_handle,
				     dwDesiredAccess,
				     0,
				     0,
				     0);

changed to mmapmodule.c (:1414 in 3.3):

m_obj->data = (char *) MapViewOfFile(m_obj->map_handle,
                                     dwDesiredAccess,
                                     off_hi,
                                     off_lo,
                                     m_obj->size);

Previously size wasn't passed to MapViewOfFile. Passing new size to MapViewOfFile greater than the size of previous map causes an error. 

This seems to be by design. From MSDN:

MapViewOfFile:

dwNumberOfBytesToMap [in]
The number of bytes of a file mapping to map to the view. All bytes must be within the maximum size specified by CreateFileMapping. If this parameter is 0 (zero), the mapping extends from the specified offset to the end of the file mapping.

CreateFileMapping:

lpName [in, optional]
The name of the file mapping object.

If this parameter matches the name of an existing mapping object, the function requests access to the object with the protection that flProtect specifies.

So on second call, CreateFileMapping will get back the previous mapping object, which has 4096 bytes of memory mapped. MapViewOfFile will try to map beyond its limit and get an error.

I am curious how "resizing" worked before. I tried passing size=0 to MapViewOfFile on second call (length=8192) then call VirtualQuery on the returned map, which can query the size of the buffer. Size is still 4096. So even if length=8192 and we call CreateFileMapping with this length, it will return the previous 4096 byte-buffer.

This looks to me like an issue which existed until Python 2.5, namely this error was silenced and returned map was still 4096 bytes, just claiming to be 8192. The way it is behaving now seems to be the correct way.
msg141508 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-08-01 13:28
Vlad, thank you for the diagnosis.
Indeed by passing a different tagname (or no tagname at all), the problem doesn't occur.
Since mmap.mmap() matches the semantics chosen by Microsoft here, I tend to think this is not a bug; besides, "fixing" it would probably be intricate.
msg141517 - (view) Author: Piotr Zolnierczuk (zolnie) Date: 2011-08-01 14:30
OK. I will work around it.
 
I was using 'mapping object of a specified size that is backed by the system paging file instead of by a file in the file system' - map->handle == INVALID_HANDLE_VALUE (-1), i.e. shared memory for inter-process communication (not my choice)

A use case was when all process were stopped and one wanted to start "fresh" using the same tag but with a bigger size. 

BTW, here's the result of the following script in Python 2.5
>>> import mmap
>>> map1 = mmap.mmap(fileno=-1, tagname='MyData', length=4096)
>>> map1[0] = 'a'
>>> map1[4095]='b'
>>> print len(map1), map1[0], map1[4095]
4096 a b
>>> map2 = mmap.mmap(fileno=-1, tagname='MyData', length=8192)
>>> print len(map2), map2[0], map2[4095]
8192 a b

which would indicate that it does 'resize' and preserves the data.



which means t
msg141519 - (view) Author: Piotr Zolnierczuk (zolnie) Date: 2011-08-01 14:38
Just looked into my "partner" C++ code and he's using it very much like in Python 2.5:

m_obj->map_handle = CreateFileMapping (INVALID_HANDLE_VALUE,
NULL,								PAGE_READWRITE,										 0,
pageSize,										 tagName);

m_obj->data = (char *) MapViewOfFile (m_obj->map_handle,								  FILE_MAP_ALL_ACCESS,										  0,										  0,									  0);
msg141520 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-08-01 14:41
Note that multiprocessing can abstract this kind of things for you:
http://docs.python.org/library/multiprocessing#sharing-state-between-processes
http://docs.python.org/library/multiprocessing#module-multiprocessing.sharedctypes
msg404856 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2021-10-23 07:15
(switching stage to resolved because it's closed/rejected; sorry for the noise)
History
Date User Action Args
2022-04-11 14:57:19adminsetgithub: 56771
2021-10-23 07:15:19tim.goldensetmessages: + msg404856
stage: resolved
2011-08-01 14:42:58pitrousetstatus: open -> closed
2011-08-01 14:41:31pitrousetmessages: + msg141520
2011-08-01 14:38:56zolniesetmessages: + msg141519
2011-08-01 14:30:01zolniesetstatus: pending -> open

messages: + msg141517
2011-08-01 13:28:36pitrousetstatus: open -> pending
resolution: rejected
messages: + msg141508

versions: + Python 3.2, Python 3.3
2011-07-31 23:05:39vladrissetnosy: + vladris
messages: + msg141482
2011-07-29 12:59:22pitrousetnosy: + pitrou, tim.golden, brian.curtin
2011-07-14 18:00:42zolniesettype: behavior
components: + Windows
2011-07-14 18:00:02zolniecreate