classification
Title: stringlib_bytes_join doesn't raise MemoryError on allocation failure
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, pitrou, python-dev, serhiy.storchaka
Priority: normal Keywords: easy

Created on 2012-12-02 01:27 by pitrou, last changed 2012-12-02 10:33 by pitrou. This issue is now closed.

Messages (5)
msg176767 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-12-02 01:27
>>> l = [b''] * (100*1024*1024)
[104918914 refs]
>>> d = b''.join(l)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SystemError: error return without exception set

(you'll have to adjust the list size based on your system memory size)
msg176770 - (view) Author: Roundup Robot (python-dev) Date: 2012-12-02 06:56
New changeset 9af5a2611202 by Christian Heimes in branch 'default':
Issue #16592: stringlib_bytes_join doesn't raise MemoryError on allocation failure
http://hg.python.org/cpython/rev/9af5a2611202
msg176771 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-12-02 06:59
Antoine, on Unix you can restrict the address space of a program to test the issue without almost crashing and OOMing your box. ;)

>>> import resource
>>> resource.setrlimit(resource.RLIMIT_AS, (1024*1024*100, 1024*1024*100))
>>> l = [b''] * (100*1024*70)
>>> d = b''.join(l)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError

I wonder why I don't see a memory error in Python 3.3 or earlier. Any idea?
msg176778 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-12-02 09:49
> I wonder why I don't see a memory error in Python 3.3 or earlier. Any idea?

See issue #15958. Now join() accept arbitrary buffer objects, which can be modified during iteration, and therefore need a temporary list of Py_buffers.
msg176779 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-12-02 09:54
> Antoine, on Unix you can restrict the address space of a program to
> test the issue without almost crashing and OOMing your box. ;)

Ah, thanks, but I didn't crash and OOM it, since it has no swap: memory errors get raised quickly here :)
History
Date User Action Args
2012-12-02 10:33:08pitrousetstatus: open -> closed
stage: needs patch -> resolved
2012-12-02 09:54:51pitrousetmessages: + msg176779
2012-12-02 09:49:07serhiy.storchakasetstatus: pending -> open
nosy: + serhiy.storchaka
messages: + msg176778

2012-12-02 06:59:25christian.heimessetstatus: open -> pending

nosy: + christian.heimes
messages: + msg176771

resolution: fixed
2012-12-02 06:56:52python-devsetnosy: + python-dev
messages: + msg176770
2012-12-02 01:27:30pitroucreate