mmap resize fails on anonymous memory #46985

kmk · 2008-05-01T17:04:16Z

BPO	2733
Nosy	@amauryfa, @abalkin, @tjguk, @tpn
Superseder	bpo-40915: multiple problems with mmap.resize() in Windows
Files	testofResizeB.txt: Stripped down code to illustrate resize fail testofResize.py.txt mmapmodule.patch: mmap patch against r69672 mmapmodule.patch mmap_disable_anonymous_resize.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tjguk'
closed_at = <Date 2021-10-23.13:37:29.627>
created_at = <Date 2008-05-01.17:04:16.288>
labels = ['extension-modules', 'type-bug', '3.10', '3.11']
title = 'mmap resize fails on anonymous memory'
updated_at = <Date 2021-10-23.13:53:02.975>
user = 'https://bugs.python.org/kmk'

bugs.python.org fields:

activity = <Date 2021-10-23.13:53:02.975>
actor = 'tim.golden'
assignee = 'tim.golden'
closed = True
closed_date = <Date 2021-10-23.13:37:29.627>
closer = 'tim.golden'
components = ['Extension Modules']
creation = <Date 2008-05-01.17:04:16.288>
creator = 'kmk'
dependencies = []
files = ['10151', '10170', '13107', '13131', '14454']
hgrepos = []
issue_num = 2733
keywords = ['patch']
message_count = 16.0
messages = ['66036', '66044', '66097', '82239', '82243', '82335', '82336', '82421', '82422', '83201', '83202', '90147', '127748', '141256', '404855', '404874']
nosy_count = 7.0
nosy_names = ['amaury.forgeotdarc', 'belopolsky', 'ocean-city', 'tim.golden', 'kmk', 'trent', 'zolnie']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '40915'
type = 'behavior'
url = 'https://bugs.python.org/issue2733'
versions = ['Python 3.10', 'Python 3.11']

kmk · 2008-05-01T17:04:12Z

We have a shared memory module that has been running fine on Windows 
with Active State Python 2.4.3 Build 12.  On machines with 2.5.1.1 
mmap.resize fails on an existing anonymous shared memory.  The attached 
file is a stripped down version of the code to illustrate the problem.  
Start it running in one window to create the shared memory, then in 
another window run it again to hook into existing shared memory. Result:
Testing SharedMemory
open -self.memory_size 336
Traceback (most recent call last):
  File "C:/home/weather/TESTOF~1.PY", line 164, in <module>
    example()
  File "C:/home/weather/TESTOF~1.PY", line 147, in example
    sm = SharedMemory( 'my_shared_memory')
  File "C:/home/weather/TESTOF~1.PY", line 31, in __init__
    self.__open()
  File "C:/home/weather/TESTOF~1.PY", line 94, in __open
    self.memory.resize(self.memory_size)
WindowsError: [Error 8] Not enough storage is available to process this 
command

amauryfa · 2008-05-01T21:01:10Z

It seems that you attached the output file instead of a python script...

kmk · 2008-05-02T15:58:09Z

sorry

tjguk · 2009-02-16T14:23:38Z

OK, I can see why this is happening and in fact there are two levels of
problem. The trouble is that, in my ignorance, I can't work out exactly
why the existing code is doing what it's doing.

(References to mmapmodule.c at r69666)

Problem 1: At line 456, the CreateFileMapping call is made with 0 hi/lo
size regardless of the file handle. cf line 1358 where it correctly
specifies the size because of the possibility of a -1 file handle. So we
now specify hi/lo values in this call. Without this change we get error
87 (invalid parameter).

Problem 2: The call to SetFilePointer at line 451 passes the hi/lo size
DWORDs. The hi value is in fact an input/output param which receives the
value of the new file pointer if a not-NULL pointer is passed in. Which
it is. This means that the hi value is now different from what it
previously was and causes an error if passed into the CreateFileMapping
call referred to in (1) above. So we make a copy of the value and ignore
the returned value. Without this change we get error 8 (not enough memory).

I'm not entirely sure what the SetFilePointer call is achieving at this
point but I'll put together a patch and a test case and perhaps someone
who understands this better can comment on the matter.

tjguk · 2009-02-16T15:36:07Z

Patch attached to mmapmodule.c and test_mmap.py

ocean-city · 2009-02-17T13:03:04Z

Tim, I confirmed your test fails on my machine, and is fixed by your patch.
I want to commit this. Can I?

tjguk · 2009-02-17T13:12:03Z

From me, yes of course, but I assume you want another
core dev for a 2nd opinion.

ocean-city · 2009-02-18T15:25:07Z

I reconsidered this issue. When mmap is anonymous,
self->file_handle == INVALID_HANDLE_VALUE (-1), so we should not call
SetFilePointer and SetEndOfFile for this handle.

And the behavior of mmap.resize is not documented clearly though,
current behavior for anonymous mapping object is different from one for
file mapping object.

>>> import mmap
>>> m1 = mmap.mmap(-1, 10, tagname="foo")
>>> m2 = mmap.mmap(-1, 10, tagname="foo")
>>> for i in xrange(10):
...     m1[i] = str(i)
...
>>> m1[:]
'0123456789'
>>> m2[:]
'0123456789'
>>> m2.resize(5)
>>> m1[:]
'0123456789'
>>> m2[:]
'01234'

For file mapping object, mmap.resize() resizes underlying file too, but
for anonymous mapping object, doesn't resize memory region itself. I
cannot find the Win32API to resize memory region created by
CreateFileMapping. I'm not sure this difference is intended by mmap
module writer. Maybe is_resizeable(mmap_object *self) should return 0
for anonymous mapping object on windows. (Of course, such difference
might be acceptable because mmap.size() already have such difference)

tjguk · 2009-02-18T15:37:57Z

Hirokazu Yamamoto wrote:

Hirokazu Yamamoto <ocean-city@m2.ccsnet.ne.jp> added the comment:

I reconsidered this issue. When mmap is anonymous,
self->file_handle == INVALID_HANDLE_VALUE (-1), so we should not call
SetFilePointer and SetEndOfFile for this handle.

I'm inclined to agree. I must admit, I was pushing a change
which changed as little as possible; I think, in hindsight,
it does *too* little.

For file mapping object, mmap.resize() resizes underlying file too, but
for anonymous mapping object, doesn't resize memory region itself. I
cannot find the Win32API to resize memory region created by
CreateFileMapping. I'm not sure this difference is intended by mmap
module writer. Maybe is_resizeable(mmap_object *self) should return 0
for anonymous mapping object on windows. (Of course, such difference
might be acceptable because mmap.size() already have such difference)

I have no strong opinion myself. In reality I rarely use mmaps;
I merely saw the call in the recent bug cleanup and thought I
might be of use.

I'm not sure who's best placed to decide what
should happen, but my feeling is that altering the existing
interface by, eg, removing the ability of an anonymous Windows
mmap to resize is not a good idea. There may be code which
is already using it.

ocean-city · 2009-03-05T15:27:05Z

More two cents which I noticed. (After the patch was applied)

On windows, resize for anonymous map can clear its contents.

>>> import mmap
>>> m = mmap.mmap(-1, 10)
>>> m[:] = "0123456789"
>>> m[:]
'0123456789'
>>> m.resize(20)
>>> m[:]
'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0
0'

Anonymous map on unix also has similar problem. ftruncate() fails for
fd == -1

Sorry for having clear solution for this. But I cannot say "This is what
mmap.resize should behave!".

ocean-city · 2009-03-05T15:27:53Z

Sorry for having clear solution for this.
+ Sorry for having no clear solution for this.

ocean-city · 2009-07-05T08:27:38Z

I still think we should forbid to resize anonymous memory map because
this operation is really problematic. I think original poster's purpose
can be fulfilled with creation of another mmap object with same tagname.

Here is a patch for it.

abalkin · 2011-02-02T18:55:19Z

I don't see an actual crash reported. An unexpected exception is not a crash. Changing the type to "behavior".

zolnie · 2011-07-27T18:24:10Z

I wonder if this is related to the problem I reported about two weeks ago
http://bugs.python.org/issue12562?

tjguk · 2021-10-23T07:14:43Z

https://bugs.python.org/issue40915 is related
Retargetting for 3.10+

tjguk · 2021-10-23T13:37:44Z

Superseded by bpo-40915

kmk mannequin added stdlib Python modules in the Lib dir type-crash A hard crash of the interpreter, possibly with a core dump labels May 1, 2008

ocean-city mannequin added extension-modules C modules in the Modules dir OS-windows and removed stdlib Python modules in the Lib dir labels Feb 17, 2009

ocean-city mannequin removed the OS-windows label Jul 5, 2009

ocean-city mannequin changed the title ~~mmap resize fails on anonymous memory (Windows)~~ mmap resize fails on anonymous memory Jul 5, 2009

abalkin added type-bug An unexpected behavior, bug, or error and removed type-crash A hard crash of the interpreter, possibly with a core dump labels Feb 2, 2011

tjguk added 3.10 only security fixes 3.11 only security fixes labels Oct 23, 2021

tjguk self-assigned this Oct 23, 2021

tjguk closed this as completed Oct 23, 2021

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mmap resize fails on anonymous memory #46985

mmap resize fails on anonymous memory #46985

kmk mannequin commented May 1, 2008

kmk mannequin commented May 1, 2008

amauryfa commented May 1, 2008

kmk mannequin commented May 2, 2008

tjguk commented Feb 16, 2009

tjguk commented Feb 16, 2009

ocean-city mannequin commented Feb 17, 2009

tjguk commented Feb 17, 2009

ocean-city mannequin commented Feb 18, 2009

tjguk commented Feb 18, 2009

ocean-city mannequin commented Mar 5, 2009

ocean-city mannequin commented Mar 5, 2009

ocean-city mannequin commented Jul 5, 2009

abalkin commented Feb 2, 2011

zolnie mannequin commented Jul 27, 2011

tjguk commented Oct 23, 2021

tjguk commented Oct 23, 2021

mmap resize fails on anonymous memory #46985

mmap resize fails on anonymous memory #46985

Comments

kmk mannequin commented May 1, 2008

kmk mannequin commented May 1, 2008

amauryfa commented May 1, 2008

kmk mannequin commented May 2, 2008

tjguk commented Feb 16, 2009

tjguk commented Feb 16, 2009

ocean-city mannequin commented Feb 17, 2009

tjguk commented Feb 17, 2009

ocean-city mannequin commented Feb 18, 2009

tjguk commented Feb 18, 2009

ocean-city mannequin commented Mar 5, 2009

ocean-city mannequin commented Mar 5, 2009

ocean-city mannequin commented Jul 5, 2009

abalkin commented Feb 2, 2011

zolnie mannequin commented Jul 27, 2011

tjguk commented Oct 23, 2021

tjguk commented Oct 23, 2021