classification
Title: memoryview.format is corrupted due to dangling shared pointer
Type: crash Stage: resolved
Components: Interpreter Core Versions: Python 3.5, Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Knio, pitrou, python-dev, r.david.murray, skrah
Priority: normal Keywords: patch

Created on 2014-10-18 22:13 by Knio, last changed 2015-01-29 18:20 by skrah. This issue is now closed.

Files
File name Uploaded Description Edit
memoryview_bug.py Knio, 2014-10-18 22:13 test case
22668patch.diff Knio, 2014-10-18 22:15 patch review
issue22668-2.diff skrah, 2015-01-27 16:16
Messages (7)
msg229664 - (view) Author: Tom Flanagan (Knio) * Date: 2014-10-18 22:13
When slicing or cloning a memoryview object that has been previously cast to change its format string, the new memoryview's format shares a pointer to the parent's format string, which may be deleted at any time.

This manifests as a corrupt format when using the new memoryview object, causing crashes or unexpected behavior.

Tested on:
Python 3.4.0 (default, Apr 11 2014, 13:05:11) [GCC 4.8.2] on linux
Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 10:45:13) [MSC v.1600 64 bit (AMD64)] on win32
Python 3.5.0a0 (default:cb8606fc84df, Oct 18 2014, 14:55:44) [GCC 4.8.2] on linux
msg229665 - (view) Author: Tom Flanagan (Knio) * Date: 2014-10-18 22:15
Fix memoryview object allocations to copy format string
msg234826 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2015-01-27 16:16
Thanks for the detailed report!  Making a private copy of 'format' for
each memoryview generally sounds like the best solution.  However, format
strings can be arbitrarily large, so we'd need to store the copy in the
ob_array after shape, strides and suboffsets.

This of course would slow down memoryview creation in the general case.

Given that the disappearing format strings are only created during casting,
I think we can get away with a local solution (see patch).
msg234972 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-01-29 13:34
New changeset e9c1fca50b46 by Stefan Krah in branch '3.4':
Issue #22668: Ensure that format strings survive slicing after casting.
https://hg.python.org/cpython/rev/e9c1fca50b46

New changeset 37112bd3dfb3 by Stefan Krah in branch 'default':
Closes #22668: Merge from 3.4.
https://hg.python.org/cpython/rev/37112bd3dfb3
msg234976 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-01-29 15:42
The buildbots aren't happy.  This one errors in the added test:

http://buildbot.python.org/all/builders/System%20Z%20Linux%203.x/builds/2595/steps/test/logs/stdio

Another one had a MemoryError in lib2to3 tests.  Not sure what is going on, but the one above at least is related to this issue.

(That's not a very descriptive name for a test, by the way, though it did allow me to find the issue easily ;)
msg234978 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-01-29 16:41
New changeset 9a4af12dcc9d by Stefan Krah in branch '3.4':
Issue #22668: Remove endianness assumption in test.
https://hg.python.org/cpython/rev/9a4af12dcc9d

New changeset da0ca7b1351f by Stefan Krah in branch 'default':
Issue #22668: Merge from 3.4.
https://hg.python.org/cpython/rev/da0ca7b1351f
msg234984 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2015-01-29 18:20
Thanks, David.  The tests assumed little-endian, which is now fixed.

The MemoryError is sporadic and unrelated -- the OpenIndiana bot
often has system load issues.

I don't see any more related failures, please reopen if you do. :)
History
Date User Action Args
2015-01-29 18:20:24skrahsetstatus: open -> closed

messages: + msg234984
2015-01-29 16:41:56python-devsetmessages: + msg234978
2015-01-29 15:42:52r.david.murraysetstatus: closed -> open
nosy: + r.david.murray
messages: + msg234976

2015-01-29 13:34:54python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg234972

resolution: fixed
stage: patch review -> resolved
2015-01-27 16:16:56skrahsetfiles: + issue22668-2.diff

messages: + msg234826
2014-11-04 23:11:18berker.peksagsetstage: patch review
versions: - Python 3.6
2014-10-18 22:15:23Kniosetfiles: + 22668patch.diff
keywords: + patch
messages: + msg229665
2014-10-18 22:13:53Kniocreate