classification
Title: Crash when deleting slices from duplicated bytearray
Type: crash Stage: resolved
Components: Interpreter Core Versions: Python 3.5, Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: alexei.romanov, johan, martin.panter, ned.deily, pitrou, python-dev, serhiy.storchaka, terry.reedy, tim.golden, wolma, zach.ware
Priority: high Keywords: patch

Created on 2015-04-17 06:37 by johan, last changed 2015-05-21 18:01 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
bytearray_bug.py johan, 2015-04-17 06:37 Program demonstrating the problem
linux-x86-64.py martin.panter, 2015-04-17 23:30
bytearray-fix.patch martin.panter, 2015-04-18 05:48 Fixes the bug review
bytearray-resize.patch martin.panter, 2015-04-18 05:54 Stop del expanding buffer review
bytearray-fixes.v2.patch martin.panter, 2015-05-19 14:30 Both fixes + tests review
Messages (20)
msg241316 - (view) Author: Johan Dahlberg (johan) Date: 2015-04-17 06:37
Python 3.4.3 crashes after some time when running the attached program under Windows 7.

The program appends a fixed bytes "string" to two independent bytearray buffers.
If slices are removed from the beginnging of the two buffers and the two buffers are print:ed, the program will crash at some random occation.
msg241321 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2015-04-17 08:16
Reproduced with top of trunk default branch on OS X (so, not Windows-specific).  With debug enabled:

$ ./bin/python3.5 /tmp/b/bytearray_bug.py
buf2: 13 5 bytearray(b'1234567890123')
buf1: 13 3 bytearray(b'1234567890123')
buf2: 21 2 bytearray(b'678901231234567890123')
buf1: 23 5 bytearray(b'45678901231234567890123')
buf2: 32 28 bytearray(b'89012312345678901231234567890123')
buf1: 31 21 bytearray(b'9012312345678901231234567890123')
buf2: 17 10 bytearray(b'01231234567890123')
buf1: 23 9 bytearray(b'45678901231234567890123')
buf2: 20 3 bytearray(b'78901231234567890123')
buf1: 27 14 bytearray(b'312345678901231234567890123')
buf2: 30 22 bytearray(b'012312345678901231234567890123')
buf1: 26 17 bytearray(b'12345678901231234567890123')
buf2: 21 1 bytearray(b'678901231234567890123')
buf1: 22 12 bytearray(b'5678901231234567890123')
buf2: 33 13 bytearray(b'789012312345678901231234567890123')
buf1: 23 4 bytearray(b'45678901231234567890123')
buf2: 33 23 bytearray(b'789012312345678901231234567890123')
buf1: 32 4 bytearray(b'89012312345678901231234567890123')
buf2: 23 19 bytearray(b'45678901231234567890123')
buf1: 41 36 bytearray(b'23123456789012312345678901231234567890123')
buf2: 17 3 bytearray(b'01231234567890123')
buf1: 18 3 bytearray(b'901231234567890123')
buf2: 27 23 bytearray(b'312345678901231234567890123')
buf1: 28 22 bytearray(b'2312345678901231234567890123')
buf2: 17 2 bytearray(b'01231234567890123')
buf1: 19 5 bytearray(b'8901231234567890123')
buf2: 28 10 bytearray(b'2312345678901231234567890123')
buf1: 27 2 bytearray(b'312345678901231234567890123')
buf2: 31 7 bytearray(b'9012312345678901231234567890123')
buf1: 38 2 bytearray(b'23456789012312345678901231234567890123')
buf2: 37 23 bytearray(b'3456789012312345678901231234567890123')
buf1: 49 2 bytearray(b'4567890123123456789012312345678901231234567890123')
buf2: 27 21 bytearray(b'312345678901231234567890123')
buf1: 60 1 bytearray(b'678901231234567890123123456789012312345678901231234567890123')
Debug memory block at address p=0x10bd238e0: API 'o'
    61 bytes originally requested
    The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected.
    The 8 pad bytes at tail=0x10bd2391d are not all FORBIDDENBYTE (0xfb):
        at tail+0: 0x33 *** OUCH
        at tail+1: 0x00 *** OUCH
        at tail+2: 0xfb
        at tail+3: 0xfb
        at tail+4: 0xfb
        at tail+5: 0xfb
        at tail+6: 0xfb
        at tail+7: 0xfb
    The block was made by call #44842 to debug malloc/realloc.
    Data at p: 34 35 36 37 38 39 30 31 ... 35 36 37 38 39 30 31 32
Fatal Python error: bad trailing pad byte

Current thread 0x00007fff70e18300 (most recent call first):
  File "/tmp/b/bytearray_bug.py", line 25 in <module>
Abort trap: 6
msg241323 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2015-04-17 08:24
Above was in 64-bit mode (x86_64).  Same run in 32-bit (i386):

$ ./bin/python3.5-32 /tmp/b/bytearray_bug.py
buf2: 13 2 bytearray(b'1234567890123')
buf1: 13 11 bytearray(b'1234567890123')
buf2: 24 16 bytearray(b'345678901231234567890123')
buf1: 15 4 bytearray(b'231234567890123')
buf2: 21 16 bytearray(b'678901231234567890123')
buf1: 24 3 bytearray(b'345678901231234567890123')
buf2: 18 1 bytearray(b'901231234567890123')
buf1: 34 28 bytearray(b'6789012312345678901231234567890123')
buf2: 30 21 bytearray(b'012312345678901231234567890123')
buf1: 19 8 bytearray(b'8901231234567890123')
buf2: 22 6 bytearray(b'5678901231234567890123')
buf1: 24 11 bytearray(b'345678901231234567890123')
buf2: 29 23 bytearray(b'12312345678901231234567890123')
buf1: 26 15 bytearray(b'12345678901231234567890123')
buf2: 19 15 bytearray(b'8901231234567890123')
buf1: 24 4 bytearray(b'345678901231234567890123')
buf2: 17 7 bytearray(b'01231234567890123')
buf1: 33 21 bytearray(b'789012312345678901231234567890123')
buf2: 23 13 bytearray(b'45678901231234567890123')
buf1: 25 20 bytearray(b'2345678901231234567890123')
buf2: 23 8 bytearray(b'45678901231234567890123')
buf1: 18 7 bytearray(b'901231234567890123')
buf2: 28 3 bytearray(b'2312345678901231234567890123')
buf1: 24 8 bytearray(b'345678901231234567890123')
buf2: 38 1 bytearray(b'23456789012312345678901231234567890123')
buf1: 29 7 bytearray(b'12312345678901231234567890123')
buf2: 50 38 bytearray(b'34567890123123456789012312345678901231234567890123')
buf1: 35 7 bytearray(b'56789012312345678901231234567890123')
buf2: 25 4 bytearray(b'2345678901231234567890123')
buf1: 41 20 bytearray(b'23123456789012312345678901231234567890123')
buf2: 34 24 bytearray(b'6789012312345678901231234567890123')
buf1: 34 19 bytearray(b'6789012312345678901231234567890123')
buf2: 23 1 bytearray(b'45678901231234567890123')
buf1: 28 18 bytearray(b'2312345678901231234567890123')
buf2: 35 20 bytearray(b'56789012312345678901231234567890123')
buf1: 23 13 bytearray(b'45678901231234567890123')
buf2: 28 16 bytearray(b'2312345678901231234567890123')
buf1: 23 19 bytearray(b'45678901231234567890123')
buf2: 25 16 bytearray(b'2345678901231234567890123')
buf1: 17 4 bytearray(b'01231234567890123')
buf2: 22 18 bytearray(b'5678901231234567890123')
buf1: 26 18 bytearray(b'12345678901231234567890123')
buf2: 17 14 bytearray(b'01231234567890123')
buf1: 21 18 bytearray(b'678901231234567890123')
buf2: 16 14 bytearray(b'1231234567890123')
buf1: 16 11 bytearray(b'1231234567890123')
buf2: 15 10 bytearray(b'231234567890123')
buf1: 18 2 bytearray(b'901231234567890123')
buf2: 18 2 bytearray(b'901231234567890123')
buf1: 29 3 bytearray(b'12312345678901231234567890123')
buf2: 29 11 bytearray(b'12312345678901231234567890123')
buf1: 39 9 bytearray(b'123456789012312345678901231234567890123')
buf2: 31 23 bytearray(b'9012312345678901231234567890123')
buf1: 43 31 bytearray(b'0123123456789012312345678901231234567890123')
Debug memory block at address p=0x7aec88: API 'o'
    49 bytes originally requested
    The 3 pad bytes at p-3 are FORBIDDENBYTE, as expected.
    The 4 pad bytes at tail=0x7aecb9 are not all FORBIDDENBYTE (0xfb):
        at tail+0: 0x31 *** OUCH
        at tail+1: 0x32 *** OUCH
        at tail+2: 0x33 *** OUCH
        at tail+3: 0x00 *** OUCH
    The block was made by call #44809 to debug malloc/realloc.
    Data at p: 31 32 33 34 35 36 37 38 ... 33 34 35 36 37 38 39 30
Fatal Python error: bad trailing pad byte

Current thread 0xa0ddc1d4 (most recent call first):
  File "/tmp/b/bytearray_bug.py", line 25 in <module>
Abort trap: 6
msg241325 - (view) Author: Wolfgang Maier (wolma) * Date: 2015-04-17 08:32
Also happening with Python 3.4.0 on Ubuntu 14.04 (after ~ half a minute and A LOT of output):

[skipping lots of lines]
buf2: 29 13 bytearray(b'12312345678901231234567890123')
buf1: 25 9 bytearray(b'2345678901231234567890123')
buf2: 29 2 bytearray(b'12312345678901231234567890123')
buf1: 29 5 bytearray(b'12312345678901231234567890123')
buf2: 40 8 bytearray(b'3123456789012312345678901231234567890123')
buf1: 37 10 bytearray(b'3456789012312345678901231234567890123')
buf2: 45 31 bytearray(b'890123123456789012312345678901231234567890123')
buf1: 40 7 bytearray(b'3123456789012312345678901231234567890123')
buf2: 27 3 bytearray(b'312345678901231234567890123')
buf1: 46 9 bytearray(b'7890123123456789012312345678901231234567890123')
buf2: 37 6 bytearray(b'3456789012312345678901231234567890123')
buf1: 50 15 bytearray(b'34567890123123456789012312345678901231234567890123')
buf2: 44 5 bytearray(b'90123123456789012312345678901231234567890123')
buf1: 48 27 bytearray(b'567890123123456789012312345678901231234567890123')
buf2: 52 3 bytearray(b'1234567890123123456789012312345678901231234567890123')
buf1: 34 1 bytearray(b'6789012312345678901231234567890123')
buf2: 62 16 bytearray(b'45678901231234567890123123456789012312345678901231234567890123')
buf1: 46 13 bytearray(b'7890123123456789012312345678901231234567890123')
*** Error in `python3': realloc(): invalid pointer: 0x00007f2a45580000 ***
msg241327 - (view) Author: Wolfgang Maier (wolma) * Date: 2015-04-17 10:12
Surprisingly, a much simpler version with just one bytearray seems to run stably (for several minutes at least), but when you wait a while then hit Ctrl-C, you are getting a Segmentation fault:

Python 3.4.0 (default, Apr 11 2014, 13:05:11) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import random
>>> buf1 = bytearray()
>>> data = b"1234567890123"
>>> 
>>> while True:
...     buf1 += data
...     l = len(buf1)
...     n = random.randrange(1, l-1)
...     del buf1[:n]
... 

^CSegmentation fault (core dumped)

The same code crashes spontaneously (without attempting a keyboard interrupt) when run in IDLE.
msg241330 - (view) Author: Alexei Romanov (alexei.romanov) Date: 2015-04-17 13:39
No problem with python 2.7.6 on Ubuntu 14.04.2 LTS amd64. Attached script was working about 2 hours - no crash.
msg241376 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-04-17 23:07
Win7, 64 bit, 3.5.0a3: crash in about 10 seconds.
msg241382 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-04-17 23:30
Here is a reduced version without using random numbers that reliably crashes for me on 64-bit Linux. Hopefully it also crashes for others, and they can investigate further. Crashes for me with the standard Arch Linux binary:

Python 3.4.3 (default, Feb 26 2015, 23:01:07) 
[GCC 4.9.2 20150204 (prerelease)] on linux

and my recently built v3.5 version:

Python 3.5.0a3+ (default:0b3027a2abbc, Apr 11 2015, 23:27:07) 
[GCC 4.9.1] on linux

It does not crash on 32 bit computer I tried though.
msg241399 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-04-18 05:48
After cleaning my build and rebuilding with “./configure --with-pymalloc --with-pydebug”, I reduced my script to these four lines:

b1 = bytearray(b"abcdefghij")  # 10 bytes
del b1[:1]
del b1[:1]
b1 += b"klmnopq"  # 7 bytes

Patch bytearray-fix.patch fixes the bug by taking account fact that ob_start is offset into the allocated memory buffer when checking if a reallocation is necessary.

Explanation with the unpatched code and my four-line script above:

1. First line allocates 11 bytes of memory (10 for the byte string + 1 NUL terminator)

2. First “del” reduces the bytearray length to 9 bytes, but actually reallocates an expanded memory buffer of 16 bytes! (quirky but not the bug)

3. Second “del” reduces the bytearray length to 8 bytes, and increments an internal ob_start offset without any memory copying or reallocation. (Fine.)

4. Appending step needs to add 7 bytes to the 8-byte array. 7 + 8 + 1 is 16 bytes total required for bytearray and NUL terminator, but since ob_start is offset from the start of the allocated memory buffer, we overwrite past the end of the buffer.

Memory debugging output and extra debug printfs of my own:

Resizing to size 10 (current log. offset 0 alloc 0)
=> Major upsize, new alloc = 11
Assigning to linear slice
- Shifting ob_start for negative growth -1
Resizing to size 9 (current log. offset 1 alloc 11)
=> Moderate upsize, new alloc = 16
- Done assigning to linear slice
Assigning to linear slice
- Shifting ob_start for negative growth -1
Resizing to size 8 (current log. offset 1 alloc 16)
=> Minor downsize
- Done assigning to linear slice
Debug memory block at address p=0x7f1af630a0d0: API 'o'
    16 bytes originally requested
    The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected.
    The 8 pad bytes at tail=0x7f1af630a0e0 are not all FORBIDDENBYTE (0xfb):
        at tail+0: 0x00 *** OUCH
        at tail+1: 0xfb
        at tail+2: 0xfb
        at tail+3: 0xfb
        at tail+4: 0xfb
        at tail+5: 0xfb
        at tail+6: 0xfb
        at tail+7: 0xfb
    The block was made by call #32897 to debug malloc/realloc.
    Data at p: 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70 71
Fatal Python error: bad trailing pad byte
msg241400 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-04-18 05:54
Posting bytearray-resize.patch which stops “del” from expanding the allocated buffer. This one is not necessary to fix the reported bug, but avoids the separate quirk identified in step 2.
msg241402 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-04-18 06:18
This bug might have been caused by the changes for Issue 19087, so I left a note there. It looks like that issue added the ob_start field to bytearray() objects, so that deleting from the start does not require memory copying.
msg241594 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-04-20 03:32
A test case for this that would trigger when memory debugging is enabled could look something like the following. Would it be appropriate to add it to the test suite?

a = bytearray(10)
size = sys.getsizeof(a)
a.pop()  # Defeat expanding buffer off-by-one quirk
self.assertEqual(sys.getsizeof(a), size, "Quirk not defeated")
del a[:1]
# Or a.pop(0)  # Does not trigger bug
# Or a[:1] = ()  # Triggers bug
self.assertEqual(sys.getsizeof(a), size, "Test assumes buffer not resized")
a += bytes(2)  # Add exactly the number of free bytes in buffer
# Or a.extend(bytes(2))  # Unaffected
# Or a.append(0); a.append(0)  # Unaffected
# Or a[8:] = bytes(2)  # Unaffected
del a  # Trigger memory buffer to be freed, with verification
msg241611 - (view) Author: Johan Dahlberg (johan) Date: 2015-04-20 06:18
Thank you all for working really fast on this issue!
I'm happy to see that a fix is already being tried out.
msg243161 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-05-14 08:07
Antoine, would you have a chance to review my patches? I assume you were responsible for adding the ob_start field.

It would be nice to see this bug fixed in the next 3.4 and 3.5 releases. As well as the original poster’s problem, I suspect this bug may be the cause of some occasional strange behaviour I have seen in my own bytearray() FIFO type code.
msg243162 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-05-14 08:08
Sorry. I'll take a look!
msg243596 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-05-19 14:30
Posting a new patch which combines both fixes and adds some test cases. However the test needs Python to be built with “./configure --with-pydebug” to detect the buffer overrun; without this the test will probably silently pass.

I removed the offending buffer space check, and let it always call PyBufferArray_Resize(). I also looked around the bytearray module for similar errors for other operations but I couldn’t find any. The other cases already tend to always call PyByteArray_Resize().
msg243618 - (view) Author: Roundup Robot (python-dev) Date: 2015-05-19 18:55
New changeset 98c1201d8eea by Antoine Pitrou in branch '3.4':
Issue #23985: Fix a possible buffer overrun when deleting a slice from the front of a bytearray and then appending some other bytes data.
https://hg.python.org/cpython/rev/98c1201d8eea

New changeset 06fab9093973 by Antoine Pitrou in branch 'default':
Issue #23985: Fix a possible buffer overrun when deleting a slice from the front of a bytearray and then appending some other bytes data.
https://hg.python.org/cpython/rev/06fab9093973
msg243619 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-05-19 19:06
I've committed the patch. Thanks, Martin!
msg243768 - (view) Author: Roundup Robot (python-dev) Date: 2015-05-21 17:52
New changeset 274c1b0a2494 by Serhiy Storchaka in branch '2.7':
Issue #23985: Fixed integer overflow in iterator object.  Original patch by
https://hg.python.org/cpython/rev/274c1b0a2494

New changeset 5b86a1abc8c3 by Serhiy Storchaka in branch '3.4':
Issue #23985: Fixed integer overflow in iterator object.  Patch by
https://hg.python.org/cpython/rev/5b86a1abc8c3

New changeset 9f2a1d9d7164 by Serhiy Storchaka in branch 'default':
Issue #23985: Fixed integer overflow in iterator object.  Patch by
https://hg.python.org/cpython/rev/9f2a1d9d7164
msg243772 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-05-21 18:01
Sorry, it was related to issue22939.
History
Date User Action Args
2015-05-21 18:01:32serhiy.storchakasetmessages: + msg243772
2015-05-21 17:52:29python-devsetmessages: + msg243768
2015-05-19 19:06:47pitrousetstatus: open -> closed
resolution: fixed
messages: + msg243619

stage: patch review -> resolved
2015-05-19 18:55:51python-devsetnosy: + python-dev
messages: + msg243618
2015-05-19 14:30:45martin.pantersetfiles: + bytearray-fixes.v2.patch

messages: + msg243596
2015-05-14 15:19:56steve.dowersetnosy: - steve.dower
2015-05-14 08:08:08pitrousetmessages: + msg243162
2015-05-14 08:07:46martin.pantersetmessages: + msg243161
2015-04-20 06:18:38johansetmessages: + msg241611
2015-04-20 03:32:08martin.pantersetmessages: + msg241594
2015-04-18 06:18:45martin.pantersetmessages: + msg241402
2015-04-18 05:54:23martin.pantersetfiles: + bytearray-resize.patch

messages: + msg241400
stage: needs patch -> patch review
2015-04-18 05:48:47martin.pantersetfiles: + bytearray-fix.patch
keywords: + patch
messages: + msg241399
2015-04-17 23:30:56martin.pantersetfiles: + linux-x86-64.py
nosy: + martin.panter
messages: + msg241382

2015-04-17 23:07:12terry.reedysetnosy: + terry.reedy
messages: + msg241376
2015-04-17 13:39:19alexei.romanovsetnosy: + alexei.romanov
messages: + msg241330
2015-04-17 12:09:58serhiy.storchakasetpriority: normal -> high
nosy: + pitrou, serhiy.storchaka
2015-04-17 10:12:10wolmasetmessages: + msg241327
2015-04-17 08:32:02wolmasetnosy: + wolma
messages: + msg241325
2015-04-17 08:24:09ned.deilysetmessages: + msg241323
2015-04-17 08:16:55ned.deilysetversions: + Python 3.5
nosy: + ned.deily

messages: + msg241321

components: + Interpreter Core, - Windows
stage: needs patch
2015-04-17 06:37:48johancreate