msg66036 - (view) |
Author: Kathryn M Kowalski (kmk) |
Date: 2008-05-01 17:04 |
We have a shared memory module that has been running fine on Windows
with Active State Python 2.4.3 Build 12. On machines with 2.5.1.1
mmap.resize fails on an existing anonymous shared memory. The attached
file is a stripped down version of the code to illustrate the problem.
Start it running in one window to create the shared memory, then in
another window run it again to hook into existing shared memory. Result:
Testing SharedMemory
open -self.memory_size 336
Traceback (most recent call last):
File "C:/home/weather/TESTOF~1.PY", line 164, in <module>
example()
File "C:/home/weather/TESTOF~1.PY", line 147, in example
sm = SharedMemory( 'my_shared_memory')
File "C:/home/weather/TESTOF~1.PY", line 31, in __init__
self.__open()
File "C:/home/weather/TESTOF~1.PY", line 94, in __open
self.memory.resize(self.memory_size)
WindowsError: [Error 8] Not enough storage is available to process this
command
|
msg66044 - (view) |
Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * |
Date: 2008-05-01 21:01 |
It seems that you attached the output file instead of a python script...
|
msg66097 - (view) |
Author: Kathryn M Kowalski (kmk) |
Date: 2008-05-02 15:58 |
sorry
|
msg82239 - (view) |
Author: Tim Golden (tim.golden) * |
Date: 2009-02-16 14:23 |
OK, I can see why this is happening and in fact there are two levels of
problem. The trouble is that, in my ignorance, I can't work out exactly
why the existing code is doing what it's doing.
(References to mmapmodule.c at r69666)
Problem 1: At line 456, the CreateFileMapping call is made with 0 hi/lo
size regardless of the file handle. cf line 1358 where it correctly
specifies the size because of the possibility of a -1 file handle. So we
now specify hi/lo values in this call. Without this change we get error
87 (invalid parameter).
Problem 2: The call to SetFilePointer at line 451 passes the hi/lo size
DWORDs. The hi value is in fact an input/output param which receives the
value of the new file pointer if a not-NULL pointer is passed in. Which
it is. This means that the hi value is now different from what it
previously was and causes an error if passed into the CreateFileMapping
call referred to in (1) above. So we make a copy of the value and ignore
the returned value. Without this change we get error 8 (not enough memory).
I'm not entirely sure what the SetFilePointer call is achieving at this
point but I'll put together a patch and a test case and perhaps someone
who understands this better can comment on the matter.
|
msg82243 - (view) |
Author: Tim Golden (tim.golden) * |
Date: 2009-02-16 15:36 |
Patch attached to mmapmodule.c and test_mmap.py
|
msg82335 - (view) |
Author: Hirokazu Yamamoto (ocean-city) * |
Date: 2009-02-17 13:03 |
Tim, I confirmed your test fails on my machine, and is fixed by your patch.
I want to commit this. Can I?
|
msg82336 - (view) |
Author: Tim Golden (tim.golden) * |
Date: 2009-02-17 13:12 |
From me, yes of course, but I assume you want another
core dev for a 2nd opinion.
|
msg82421 - (view) |
Author: Hirokazu Yamamoto (ocean-city) * |
Date: 2009-02-18 15:25 |
I reconsidered this issue. When mmap is anonymous,
self->file_handle == INVALID_HANDLE_VALUE (-1), so we should not call
SetFilePointer and SetEndOfFile for this handle.
And the behavior of mmap.resize is not documented clearly though,
current behavior for anonymous mapping object is different from one for
file mapping object.
>>> import mmap
>>> m1 = mmap.mmap(-1, 10, tagname="foo")
>>> m2 = mmap.mmap(-1, 10, tagname="foo")
>>> for i in xrange(10):
... m1[i] = str(i)
...
>>> m1[:]
'0123456789'
>>> m2[:]
'0123456789'
>>> m2.resize(5)
>>> m1[:]
'0123456789'
>>> m2[:]
'01234'
For file mapping object, mmap.resize() resizes underlying file too, but
for anonymous mapping object, doesn't resize memory region itself. I
cannot find the Win32API to resize memory region created by
CreateFileMapping. I'm not sure this difference is intended by mmap
module writer. Maybe is_resizeable(mmap_object *self) should return 0
for anonymous mapping object on windows. (Of course, such difference
might be acceptable because mmap.size() already have such difference)
|
msg82422 - (view) |
Author: Tim Golden (tim.golden) * |
Date: 2009-02-18 15:37 |
Hirokazu Yamamoto wrote:
> Hirokazu Yamamoto <ocean-city@m2.ccsnet.ne.jp> added the comment:
>
> I reconsidered this issue. When mmap is anonymous,
> self->file_handle == INVALID_HANDLE_VALUE (-1), so we should not call
> SetFilePointer and SetEndOfFile for this handle.
I'm inclined to agree. I must admit, I was pushing a change
which changed as little as possible; I think, in hindsight,
it does *too* little.
> For file mapping object, mmap.resize() resizes underlying file too, but
> for anonymous mapping object, doesn't resize memory region itself. I
> cannot find the Win32API to resize memory region created by
> CreateFileMapping. I'm not sure this difference is intended by mmap
> module writer. Maybe is_resizeable(mmap_object *self) should return 0
> for anonymous mapping object on windows. (Of course, such difference
> might be acceptable because mmap.size() already have such difference)
I have no strong opinion myself. In reality I rarely use mmaps;
I merely saw the call in the recent bug cleanup and thought I
might be of use.
I'm not sure who's best placed to decide what
should happen, but my feeling is that altering the existing
interface by, eg, removing the ability of an anonymous Windows
mmap to resize is not a good idea. There may be code which
is already using it.
|
msg83201 - (view) |
Author: Hirokazu Yamamoto (ocean-city) * |
Date: 2009-03-05 15:27 |
More two cents which I noticed. (After the patch was applied)
1. On windows, resize for anonymous map can clear its contents.
>>> import mmap
>>> m = mmap.mmap(-1, 10)
>>> m[:] = "0123456789"
>>> m[:]
'0123456789'
>>> m.resize(20)
>>> m[:]
'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0
0'
2. Anonymous map on unix also has similar problem. ftruncate() fails for
fd == -1
Sorry for having clear solution for this. But I cannot say "This is what
mmap.resize should behave!".
|
msg83202 - (view) |
Author: Hirokazu Yamamoto (ocean-city) * |
Date: 2009-03-05 15:27 |
- Sorry for having clear solution for this.
+ Sorry for having no clear solution for this.
|
msg90147 - (view) |
Author: Hirokazu Yamamoto (ocean-city) * |
Date: 2009-07-05 08:27 |
I still think we should forbid to resize anonymous memory map because
this operation is really problematic. I think original poster's purpose
can be fulfilled with creation of another mmap object with same tagname.
Here is a patch for it.
|
msg127748 - (view) |
Author: Alexander Belopolsky (belopolsky) * |
Date: 2011-02-02 18:55 |
I don't see an actual crash reported. An unexpected exception is not a crash. Changing the type to "behavior".
|
msg141256 - (view) |
Author: Piotr Zolnierczuk (zolnie) |
Date: 2011-07-27 18:24 |
I wonder if this is related to the problem I reported about two weeks ago
http://bugs.python.org/issue12562?
|
msg404855 - (view) |
Author: Tim Golden (tim.golden) * |
Date: 2021-10-23 07:14 |
https://bugs.python.org/issue40915 is related
Retargetting for 3.10+
|
msg404874 - (view) |
Author: Tim Golden (tim.golden) * |
Date: 2021-10-23 13:37 |
Superseded by issue40915
|
|
Date |
User |
Action |
Args |
2022-04-11 14:56:33 | admin | set | github: 46985 |
2021-10-23 13:53:02 | tim.golden | set | superseder: multiple problems with mmap.resize() in Windows resolution: later -> duplicate |
2021-10-23 13:37:44 | tim.golden | set | messages:
+ msg404874 |
2021-10-23 13:37:29 | tim.golden | set | status: open -> closed resolution: later stage: resolved |
2021-10-23 07:14:43 | tim.golden | set | assignee: tim.golden messages:
+ msg404855 versions:
+ Python 3.10, Python 3.11, - Python 2.7, Python 3.2 |
2011-07-27 18:24:10 | zolnie | set | nosy:
+ zolnie messages:
+ msg141256
|
2011-02-02 18:55:19 | belopolsky | set | type: crash -> behavior
messages:
+ msg127748 nosy:
+ belopolsky |
2009-07-05 08:27:38 | ocean-city | set | files:
+ mmap_disable_anonymous_resize.patch
components:
- Windows title: mmap resize fails on anonymous memory (Windows) -> mmap resize fails on anonymous memory nosy:
amaury.forgeotdarc, ocean-city, tim.golden, kmk, trent versions:
+ Python 3.2, - Python 2.6, Python 3.0, Python 3.1 messages:
+ msg90147 stage: commit review -> (no value) |
2009-03-05 15:27:52 | ocean-city | set | messages:
+ msg83202 |
2009-03-05 15:27:06 | ocean-city | set | messages:
+ msg83201 |
2009-02-18 16:48:19 | ocean-city | set | files:
+ mmapmodule.patch |
2009-02-18 15:37:57 | tim.golden | set | messages:
+ msg82422 |
2009-02-18 15:25:09 | ocean-city | set | messages:
+ msg82421 |
2009-02-17 13:12:02 | tim.golden | set | messages:
+ msg82336 |
2009-02-17 13:03:04 | ocean-city | set | nosy:
+ ocean-city stage: commit review messages:
+ msg82335 components:
+ Extension Modules, Windows, - Library (Lib) versions:
+ Python 2.6, Python 3.0, Python 3.1, Python 2.7, - Python 2.5 |
2009-02-16 15:36:09 | tim.golden | set | files:
+ mmapmodule.patch keywords:
+ patch messages:
+ msg82243 |
2009-02-16 14:23:40 | tim.golden | set | nosy:
+ tim.golden messages:
+ msg82239 |
2008-05-02 16:11:51 | trent | set | nosy:
+ trent |
2008-05-02 15:58:11 | kmk | set | files:
+ testofResize.py.txt messages:
+ msg66097 |
2008-05-01 21:01:10 | amaury.forgeotdarc | set | nosy:
+ amaury.forgeotdarc messages:
+ msg66044 |
2008-05-01 17:04:16 | kmk | create | |