classification
Title: mmap resize fails on anonymous memory
Type: behavior Stage: resolved
Components: Extension Modules Versions: Python 3.11, Python 3.10
process
Status: closed Resolution: duplicate
Dependencies: Superseder: multiple problems with mmap.resize() in Windows
View: 40915
Assigned To: tim.golden Nosy List: amaury.forgeotdarc, belopolsky, kmk, ocean-city, tim.golden, trent, zolnie
Priority: normal Keywords: patch

Created on 2008-05-01 17:04 by kmk, last changed 2021-10-23 13:53 by tim.golden. This issue is now closed.

Files
File name Uploaded Description Edit
testofResizeB.txt kmk, 2008-05-01 17:04 Stripped down code to illustrate resize fail
testofResize.py.txt kmk, 2008-05-02 15:58
mmapmodule.patch tim.golden, 2009-02-16 15:36 mmap patch against r69672 review
mmapmodule.patch ocean-city, 2009-02-18 16:48 review
mmap_disable_anonymous_resize.patch ocean-city, 2009-07-05 08:27 review
Messages (16)
msg66036 - (view) Author: Kathryn M Kowalski (kmk) Date: 2008-05-01 17:04
We have a shared memory module that has been running fine on Windows 
with Active State Python 2.4.3 Build 12.  On machines with 2.5.1.1 
mmap.resize fails on an existing anonymous shared memory.  The attached 
file is a stripped down version of the code to illustrate the problem.  
Start it running in one window to create the shared memory, then in 
another window run it again to hook into existing shared memory. Result:
Testing SharedMemory
open -self.memory_size 336
Traceback (most recent call last):
  File "C:/home/weather/TESTOF~1.PY", line 164, in <module>
    example()
  File "C:/home/weather/TESTOF~1.PY", line 147, in example
    sm = SharedMemory( 'my_shared_memory')
  File "C:/home/weather/TESTOF~1.PY", line 31, in __init__
    self.__open()
  File "C:/home/weather/TESTOF~1.PY", line 94, in __open
    self.memory.resize(self.memory_size)
WindowsError: [Error 8] Not enough storage is available to process this 
command
msg66044 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-05-01 21:01
It seems that you attached the output file instead of a python script...
msg66097 - (view) Author: Kathryn M Kowalski (kmk) Date: 2008-05-02 15:58
sorry
msg82239 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2009-02-16 14:23
OK, I can see why this is happening and in fact there are two levels of
problem. The trouble is that, in my ignorance, I can't work out exactly
why the existing code is doing what it's doing.

(References to mmapmodule.c at r69666)

Problem 1: At line 456, the CreateFileMapping call is made with 0 hi/lo
size regardless of the file handle. cf line 1358 where it correctly
specifies the size because of the possibility of a -1 file handle. So we
now specify hi/lo values in this call. Without this change we get error
87 (invalid parameter).

Problem 2: The call to SetFilePointer at line 451 passes the hi/lo size
DWORDs. The hi value is in fact an input/output param which receives the
value of the new file pointer if a not-NULL pointer is passed in. Which
it is. This means that the hi value is now different from what it
previously was and causes an error if passed into the CreateFileMapping
call referred to in (1) above. So we make a copy of the value and ignore
the returned value. Without this change we get error 8 (not enough memory).

I'm not entirely sure what the SetFilePointer call is achieving at this
point but I'll put together a patch and a test case and perhaps someone
who understands this better can comment on the matter.
msg82243 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2009-02-16 15:36
Patch attached to mmapmodule.c and test_mmap.py
msg82335 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2009-02-17 13:03
Tim, I confirmed your test fails on my machine, and is fixed by your patch.
I want to commit this. Can I?
msg82336 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2009-02-17 13:12
From me, yes of course, but I assume you want another
core dev for a 2nd opinion.
msg82421 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2009-02-18 15:25
I reconsidered this issue. When mmap is anonymous, 
self->file_handle == INVALID_HANDLE_VALUE (-1), so we should not call
SetFilePointer and SetEndOfFile for this handle.

And the behavior of mmap.resize is not documented clearly though,
current behavior for anonymous mapping object is different from one for
file mapping object.

>>> import mmap
>>> m1 = mmap.mmap(-1, 10, tagname="foo")
>>> m2 = mmap.mmap(-1, 10, tagname="foo")
>>> for i in xrange(10):
...     m1[i] = str(i)
...
>>> m1[:]
'0123456789'
>>> m2[:]
'0123456789'
>>> m2.resize(5)
>>> m1[:]
'0123456789'
>>> m2[:]
'01234'

For file mapping object, mmap.resize() resizes underlying file too, but
for anonymous mapping object, doesn't resize memory region itself. I
cannot find the Win32API to resize memory region created by
CreateFileMapping. I'm not sure this difference is intended by mmap
module writer. Maybe is_resizeable(mmap_object *self) should return 0
for anonymous mapping object on windows. (Of course, such  difference
might be acceptable because mmap.size() already have such difference)
msg82422 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2009-02-18 15:37
Hirokazu Yamamoto wrote:
> Hirokazu Yamamoto <ocean-city@m2.ccsnet.ne.jp> added the comment:
> 
> I reconsidered this issue. When mmap is anonymous, 
> self->file_handle == INVALID_HANDLE_VALUE (-1), so we should not call
> SetFilePointer and SetEndOfFile for this handle.

I'm inclined to agree. I must admit, I was pushing a change
which changed as little as possible; I think, in hindsight,
it does *too* little.

> For file mapping object, mmap.resize() resizes underlying file too, but
> for anonymous mapping object, doesn't resize memory region itself. I
> cannot find the Win32API to resize memory region created by
> CreateFileMapping. I'm not sure this difference is intended by mmap
> module writer. Maybe is_resizeable(mmap_object *self) should return 0
> for anonymous mapping object on windows. (Of course, such  difference
> might be acceptable because mmap.size() already have such difference)

I have no strong opinion myself. In reality I rarely use mmaps;
I merely saw the call in the recent bug cleanup and thought I
might be of use. 

I'm not sure who's best placed to decide what
should happen, but my feeling is that altering the existing
interface by, eg, removing the ability of an anonymous Windows
mmap to resize is not a good idea. There may be code which
is already using it.
msg83201 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2009-03-05 15:27
More two cents which I noticed. (After the patch was applied)

1. On windows, resize for anonymous map can clear its contents.

>>> import mmap
>>> m = mmap.mmap(-1, 10)
>>> m[:] = "0123456789"
>>> m[:]
'0123456789'
>>> m.resize(20)
>>> m[:]
'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0
0'

2. Anonymous map on unix also has similar problem. ftruncate() fails for
fd == -1

Sorry for having clear solution for this. But I cannot say "This is what
mmap.resize should behave!".
msg83202 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2009-03-05 15:27
- Sorry for having clear solution for this.
+ Sorry for having no clear solution for this.
msg90147 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2009-07-05 08:27
I still think we should forbid to resize anonymous memory map because
this operation is really problematic. I think original poster's purpose
can be fulfilled with creation of another mmap object with same tagname.

Here is a patch for it.
msg127748 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-02-02 18:55
I don't see an actual crash reported.  An unexpected exception is not a crash.  Changing the type to "behavior".
msg141256 - (view) Author: Piotr Zolnierczuk (zolnie) Date: 2011-07-27 18:24
I wonder if this is related to the problem I reported about two weeks ago
http://bugs.python.org/issue12562?
msg404855 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2021-10-23 07:14
https://bugs.python.org/issue40915 is related
Retargetting for 3.10+
msg404874 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2021-10-23 13:37
Superseded by issue40915
History
Date User Action Args
2021-10-23 13:53:02tim.goldensetsuperseder: multiple problems with mmap.resize() in Windows
resolution: later -> duplicate
2021-10-23 13:37:44tim.goldensetmessages: + msg404874
2021-10-23 13:37:29tim.goldensetstatus: open -> closed
resolution: later
stage: resolved
2021-10-23 07:14:43tim.goldensetassignee: tim.golden
messages: + msg404855
versions: + Python 3.10, Python 3.11, - Python 2.7, Python 3.2
2011-07-27 18:24:10zolniesetnosy: + zolnie
messages: + msg141256
2011-02-02 18:55:19belopolskysettype: crash -> behavior

messages: + msg127748
nosy: + belopolsky
2009-07-05 08:27:38ocean-citysetfiles: + mmap_disable_anonymous_resize.patch

components: - Windows
title: mmap resize fails on anonymous memory (Windows) -> mmap resize fails on anonymous memory
nosy: amaury.forgeotdarc, ocean-city, tim.golden, kmk, trent
versions: + Python 3.2, - Python 2.6, Python 3.0, Python 3.1
messages: + msg90147
stage: commit review -> (no value)
2009-03-05 15:27:52ocean-citysetmessages: + msg83202
2009-03-05 15:27:06ocean-citysetmessages: + msg83201
2009-02-18 16:48:19ocean-citysetfiles: + mmapmodule.patch
2009-02-18 15:37:57tim.goldensetmessages: + msg82422
2009-02-18 15:25:09ocean-citysetmessages: + msg82421
2009-02-17 13:12:02tim.goldensetmessages: + msg82336
2009-02-17 13:03:04ocean-citysetnosy: + ocean-city
stage: commit review
messages: + msg82335
components: + Extension Modules, Windows, - Library (Lib)
versions: + Python 2.6, Python 3.0, Python 3.1, Python 2.7, - Python 2.5
2009-02-16 15:36:09tim.goldensetfiles: + mmapmodule.patch
keywords: + patch
messages: + msg82243
2009-02-16 14:23:40tim.goldensetnosy: + tim.golden
messages: + msg82239
2008-05-02 16:11:51trentsetnosy: + trent
2008-05-02 15:58:11kmksetfiles: + testofResize.py.txt
messages: + msg66097
2008-05-01 21:01:10amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg66044
2008-05-01 17:04:16kmkcreate