classification
Title: IDLE Doc: Text consumes unlimited RAM, consoles likely not
Type: resource usage Stage: resolved
Components: Documentation, IDLE Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: Big Stone, jbrearley, roger.serwy, terry.reedy
Priority: normal Keywords: patch

Created on 2018-03-05 16:30 by jbrearley, last changed 2018-11-07 06:05 by terry.reedy. This issue is now closed.

Files
File name Uploaded Description Edit
Python_IDLEX.png jbrearley, 2018-03-06 20:53
Pull Requests
URL Status Linked Edit
PR 10373 merged terry.reedy, 2018-11-07 04:12
PR 10374 merged miss-islington, 2018-11-07 04:55
PR 10375 merged miss-islington, 2018-11-07 04:55
Messages (15)
msg313263 - (view) Author: John Brearley (jbrearley) Date: 2018-03-05 16:30
While running a tensorflow script in the IDLEX GUI that runs for 8 million steps and produce 2 lines stdout per step, my PC used all 16GB RAM and crashed the python process, not to mention messed up other apps, like Firefox & Norton AntiVirus. While the RAM was recovered, Firefox started responding, but Norton Antivirus didn’t, so the PC had to be rebooted. 
 
The issue is easily reproduced with the short print loop that dumps 20K lines of stdout, at 171 characters / line on the IDLEX GUI window. When the script is run in the IDLEX GUI, the Windows Task Manager shows the python process start at 19MB RAM consumption, then grows to 569MB RAM consumption. If I run the script a second time in the same IDLEX GUI window, it grows to 1.1GB RAM consumption. 
 
So 20K lines off output at 171 characters / line (“i: nnnnn” prefix + 2 * 80 byte string + newline) = 3.4M total characters stored in the scrollback buffer. The delta memory consumed was 569MB – 19MB = 550MB. The RAM consumed / character is 550MB / 3.4M = 161 bytes / character. This seems excessively inefficient.
 
I now understand how the tensorflow script would stop after 550K iterations and the 550K lines of stdout in the IDLEX GUI would consume all 16GB RAM on my PC.
 
BTW, when I run the same test script in the WinPython command prompt window, it only consumes 4MB RAM while it runs. However the scrollback buffer is limited to 10K lines, wrapped at the 80 character mark, so much less data saved.
 
I haven’t found any options in IDLEX GUI to limit the scrollback buffer size.
 
My request is to review the scrollback memory storage algorithms. If nothing can be done to improve them, then please add a circular buffer to limit the memory consumption.
 
# Print loop to test memory consumption in Python IDLEX GUI.
s1 = "0123456789"
s2 = s1+s1+s1+s1+s1+s1+s1+s1
for i in range(20000):
   print("i:", i, s2, s2)

I am using Python 3.6.4 on Windows 7 PC, Intel i7-4770S, 3.1GHz, 16GB RAM.
msg313272 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-03-05 17:50
IDLEX is an independently developed and distributed set of IDLE eXtensions.  We have nothing to do with it.

IDLE, as the name suggests and the doc spells out, is an Integrated  Development and Learning Environment.  It is not a production run environment.  For interactive development, one can close the shell window and reclaim its memory while keeping an editor window open.

16 million lines of screen output way outside of IDLE's intended use.  Why are you (mis)using it this way?

As you already discovered, an effectively infinite output stream can be sent to a console.  Command Prompt has a similar behavior.  If you want more saved, output to disk.

Or write your own gui for your simulations.  IDLE (and hence IDLEX) uses the tkinter wrapper of the tcl/tk GUI framework.  tcl/tk is also separately developed and distributed.  We just use it.  Its text widget does not come with any automatic size management.  But you could write your own insert wrapper that would delete a line for each line added after you deem the widget 'full'.

When I ran your sample with 64 bit 3.7.0b2 on Win10, the memory reported by Task Manager increased from 30 to 195 Mb or 165 MB net, much less than you report.  I once printed about 500,000 60 char lines.  I had either 12 or 24 GB memory at the time.

I am inclined to close this issue as 'third party' (IDLEX and tcl/tk) but I will let you respond first.
msg313327 - (view) Author: John Brearley (jbrearley) Date: 2018-03-06 14:56
Hi Terry: I am exploring the value of a language specific editor and runtime environment. Its definitely a large step up from Windows Notepad and Gnome Gedit. Perhaps some notes in the IDLEX documentation regarding the development versus production runtime usages would be in order?

Perhaps in your Win10 environment you get the memory back when you close the shell window. In Win7, you dont get the memory back until you close both windows.
msg313348 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-03-06 18:09
I did not close either window before looking at the Task Manager number.

Once your program is debugged, I cannot think of any advantage of running it through IDLE instead of directly with the Python interpreter.

Since you repeat 'IDLEX' I still do not know whether you are running IDLEX or IDLE itself.
msg313364 - (view) Author: John Brearley (jbrearley) Date: 2018-03-06 20:53
Hi Terry: The icon on my Win 7 desktop points to: "C:\WinPython\IDLEX (Python GUI).exe".

This was part of the 430MB installer file WinPython-64bit-3.6.4.0Qt5b4.exe from https://sourceforge.net/projects/winpython.

I attached a screen shot of IDLEX window & help about.
msg313394 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-03-07 19:16
I don't know whether memory difference between us is due to IDLEX or Windows version or something else, but I don't really care.  The IDLE doc has a section "IDLE-console differences".  It should gain a paragraph explain the difference between memory-limited tk.Text-based Shell and limited-lines consoles.  For Command Prompt, the buffer defaults to a pathetic 300 lines, which is not enough for the CPython test suite output.  The max is 9999 for those who discover how to enlarge it.  No limit is an advantage for most uses, but not for one who fills up memory.

I will also think about a sentence to add to the introductory section about development versus production runs.
msg315806 - (view) Author: Big Stone (Big Stone) Date: 2018-04-26 16:42
WinPython-64bit-3.6.4.0Qt5b4 doesn't in fact includes IDLEX: IDLEX is not compatible with Python-3.6+

The "IDLEX" icon of WinPython switches back to "normal" IDLE, when IDLEX is not detected. 

As WinPython-3.5 is being deprecated, the icon should be renamed IDLE.

(see also https://bugs.python.org/issue17535 quest)
msg315810 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-04-26 18:19
Big Stone: thank you for the clarification that IDLEX, and its patching of idlelib, is not involved.   WinPython is an independent augmented distribution. Your suggestion about its IDLEX icon should be sent to its authors.
msg315821 - (view) Author: Big Stone (Big Stone) Date: 2018-04-26 21:40
done, sorry for the (now) missleading icon name.
msg329094 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2018-11-02 01:39
Big Stone: Yes, IDLEX does have a slow memory leak. Please check if this bug is happening with IDLE itself.

Terry: Thanks for responding to this. I suggest this issue can be closed.
msg329111 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-11-02 06:55
Roger, John may have run IDLE rather than IDLEX.  This issue is about documenting the fact that tk Texts do not throw away lines, and will accept text until there are memory problems.  (Command Prompt, for instance, only keeps the last N lines, where N defaults to 300 (not enough to see a test suite run) and is max 9999.)
msg329146 - (view) Author: Big Stone (Big Stone) Date: 2018-11-02 17:48
for the record, since summer 2018, idlex-1.18 is compatible again with Python-3.6+: the author simply forked the old IDLE library IDLEX was depending on.

so IDLEX icon in recent Winpython is"truly" IDLEX again.
msg329403 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-11-07 04:55
New changeset 76cd0c30d60961d1a10e2673834a455d2b51f695 by Terry Jan Reedy in branch 'master':
bpo-33000: Document that IDLE's shell has no line limit. (#10373)
https://github.com/python/cpython/commit/76cd0c30d60961d1a10e2673834a455d2b51f695
msg329404 - (view) Author: miss-islington (miss-islington) Date: 2018-11-07 05:06
New changeset 2b2a8c130ceb47842cfbd0c725b5b6599b26cf27 by Miss Islington (bot) in branch '3.7':
bpo-33000: Document that IDLE's shell has no line limit. (GH-10373)
https://github.com/python/cpython/commit/2b2a8c130ceb47842cfbd0c725b5b6599b26cf27
msg329405 - (view) Author: miss-islington (miss-islington) Date: 2018-11-07 05:18
New changeset 25bd1073996f26ad4895d3eb2d09315361c3cc84 by Miss Islington (bot) in branch '3.6':
bpo-33000: Document that IDLE's shell has no line limit. (GH-10373)
https://github.com/python/cpython/commit/25bd1073996f26ad4895d3eb2d09315361c3cc84
History
Date User Action Args
2018-11-07 06:05:44terry.reedysetstatus: open -> closed
nosy: - miss-islington

resolution: fixed
stage: patch review -> resolved
2018-11-07 05:18:40miss-islingtonsetmessages: + msg329405
2018-11-07 05:06:58miss-islingtonsetnosy: + miss-islington
messages: + msg329404
2018-11-07 04:55:51miss-islingtonsetpull_requests: + pull_request9676
2018-11-07 04:55:38miss-islingtonsetpull_requests: + pull_request9675
2018-11-07 04:55:11terry.reedysetmessages: + msg329403
2018-11-07 04:12:59terry.reedysetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request9674
2018-11-06 22:11:37terry.reedylinkissue23220 dependencies
2018-11-06 02:50:38terry.reedyunlinkissue35099 dependencies
2018-11-02 17:48:11Big Stonesetmessages: + msg329146
2018-11-02 06:55:14terry.reedysetmessages: + msg329111
2018-11-02 01:39:08roger.serwysetnosy: + roger.serwy
messages: + msg329094
2018-10-29 05:06:26terry.reedylinkissue35099 dependencies
2018-04-26 21:40:08Big Stonesetmessages: + msg315821
2018-04-26 18:19:03terry.reedysetmessages: + msg315810
2018-04-26 16:42:24Big Stonesetnosy: + Big Stone
messages: + msg315806
2018-03-07 19:16:09terry.reedysetversions: + Python 3.7, Python 3.8
title: IDLEX GUI consumes all RAM for scrollback buffer, uses 161Bytes / character stored -> IDLE Doc: Text consumes unlimited RAM, consoles likely not
messages: + msg313394

components: + Documentation
stage: needs patch
2018-03-06 20:53:57jbrearleysetfiles: + Python_IDLEX.png

messages: + msg313364
2018-03-06 18:09:36terry.reedysetmessages: + msg313348
2018-03-06 14:56:58jbrearleysetmessages: + msg313327
2018-03-05 17:50:24terry.reedysetmessages: + msg313272
2018-03-05 16:30:03jbrearleycreate