classification
Title: Memory overallocation crash and keyboard interrupt stops working
Type: Stage:
Components: Versions: Python 3.6, Python 3.5
process
Status: closed Resolution: duplicate
Dependencies: Superseder: Crash On bytearray()
View: 27559
Assigned To: Nosy List: martin.panter, pabstersac, xiang.zhang, ztane
Priority: normal Keywords:

Created on 2016-07-18 09:08 by pabstersac, last changed 2016-07-18 10:44 by martin.panter. This issue is now closed.

Files
File name Uploaded Description Edit
crash.py pabstersac, 2016-07-18 09:08
Messages (6)
msg270732 - (view) Author: pablo sacristan (pabstersac) Date: 2016-07-18 09:08
zlib.compress crashes when you put a lot of stuff into it and even when you try keyboard interrupts it doesn't work for some reason, and Python gets killed because trying to compress so much data gets the Python process get killed by the kernel. If you put a large amount of data in it will get killed quicker. A problem is that keyboard interrupts stop working while the program is running, which is usually around 5 - 15 seconds during which python stops working (including interrupts) until it gets killed by the kernel. This may be used to crash python remotely if a server takes user input and runs it through zlib.compress(b'variable' * 2**32) which is not very likely. You probably should set a limit for what you compress and how much it is.
The output is:
Killed: 9
Even though you try doing keyboard interrupt it doesn't work.
msg270738 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-07-18 09:38
There is a good chance crashes and related 2 GiB and 4 GiB limits in zlib will be solved by Issue 27130. Do you want to see if the patch improves the situation? (I cannot test it directly because I don’t have enough memory.)
msg270739 - (view) Author: pablo sacristan (pabstersac) Date: 2016-07-18 09:42
I'd like to but I don't have enough memory either, sorry.
msg270740 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2016-07-18 09:46
Compressing large data is born to be slow and it's highly possible to cause the kernel halt then due to resource limit. 

> This may be used to crash python remotely if a server takes user input and runs it through zlib.compress(b'variable' * 2**32) which is not very likely. You probably should set a limit for what you compress and how much it is.

This is the responsibility of application code not library code.
msg270741 - (view) Author: Antti Haapala (ztane) * Date: 2016-07-18 09:48
I am pretty sure **it never calls zlib.compress**. I get memory error from that argument alone, on Linux with overcommit memory enabled, 16G ram, swap.
msg270747 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-07-18 10:44
Yes you are probably right Antti. The input to compress would need 31 TB of memory, and is a text string, while Python 3’s zlib.compress() requires bytes. In other words, this will be the same problem of overallocating memory as Issue 27559.

Pablo: I suggest you carefully look at the code, try to understand what it is doing, eliminating the unnecessary parts, etc. Especially when so many of your reports today involve similar techniques.
History
Date User Action Args
2016-07-18 10:44:06martin.pantersetstatus: open -> closed
title: zlib.compress() crash and keyboard interrupt stops working -> Memory overallocation crash and keyboard interrupt stops working
superseder: Crash On bytearray()
messages: + msg270747

resolution: duplicate
2016-07-18 09:48:18ztanesetnosy: + ztane
messages: + msg270741
2016-07-18 09:46:44xiang.zhangsetnosy: + xiang.zhang
messages: + msg270740
2016-07-18 09:42:11pabstersacsetmessages: + msg270739
versions: + Python 3.5, Python 3.6
2016-07-18 09:38:06martin.pantersetnosy: + martin.panter
messages: + msg270738
2016-07-18 09:08:54pabstersaccreate