I wrote a little python script that prints a large
dictionary to stdout (simply using 'print
mydictionary').

In fact, the type is irrelevant, what matters is that
the resulting output had approx. 200,000 characters.

The shell prints the dictionary into a single line,
which causes the window to be almost non-responding,
e.g. when I try to scroll the window.

Even on a high-end PC it takes a minute or even
longer to react to anything.

I use Python 2.4.2 on Windows XP SP2.

I am aware that it is not exactly wise to print such
large objects, but I usually print return values to
stdout when I debug a script, and I do not always
expect an object to be that large.

The average text editor handles such long lines much
better.

A quick workaround might be to break very long lines
automagically (perhaps at around column 1000).

PS: I already observed the bug some years ago. I
think I even submitted it to python or idlefork a
long time ago but I was unable to find it in the
buglist.

Logged In: YES
user_id=593130

I verified this with print 100000*'a', also XP (home) sp2.
The sluggishness continued after getting the prompt back
and trying to do something simple, like 2+2, taking maybe
1/2 minute to print 4 and then the >>> prompt again.
The sluggishness also continued after restarting the
shell (^F6). This indicates that the problem is with the
window, not with IDLE. Hope someone can try same on *nix
system to see if general with TKinter or specific to Win
systems.

Logged In: YES
user_id=341410

Generally speaking, most wrapping text controls have issues
with wrapping long lines. It would seem reasonable to get
the width of the text control in characters, and manually
wrap all incoming lines regardless. If the existance or not
of real line endings are important, one could mark which
lines are manually wrapped and remove the line endings on
copy (edit->copy, ctrl+c, etc.).

Logged In: YES
user_id=149084

This is a known problem with Tk/Tcl/tkinter - large output
scrolls slowly. It's not something that can be fixed in IDLE. I
tried it on Arch Linux - IDLE 2.5a0 - Tk 8.4. 250,000 character
output not too bad , 25 sec, but 10,000 lines of 25 char takes
over twice that long, so breaking the lines doesn't help.

I don't see any response problem once the output completes. The
situation is exponentially worse at 500,000 char.

What is your use case? IDLE is designed to be an IDE. Why
output such massive data?

You may be interested in Squeezer, a Noam Raphael extension to
IDLEfork.

http://sourceforge.net/tracker/index.php?
func=detail&aid=704316&group_id=9579&atid=309579

I haven't tried it myself, but it might be what you're looking
for.

Logged In: YES
user_id=865975

Hi kbk,

well, my use case is debugging. I write a script and run it
with IDLE. It doesn't behave as expected, so I insert a
print statement. Next time I run it, IDLE hangs. Oops, it
was a long array; it should have been an int. Line too long.
Duh.

OK, I wait through it, correct the bug, run it again. What
happens? IDLE hangs again, trying to scroll a long line (of
the previous run).

Never mind, I can always kill the process... Dammit, I
can't! It eats 100% CPU; task manager doesn't respond.

IMHO his takes away one of python's strengths, which is
making quick hacks really quick.

Would you suggest redirecting this issue to tkinter? You
don't seem to consider this an issue at all.

I will give squeezer a try. Or maybe PyDev?

Logged In: YES
user_id=341410

You can close the window which includes the Shell that has
the huge output, or even reduce the priority of your Idle
shell (you can make it automatic by mucking about with the
shortcut; see the 'start' command).

Logged In: YES
user_id=149084

It's not that I don't consider it an issue, but I can't
do anything to improve the performance of the Tk text
widget. IDLE is pure Python.

One thing that comes to mind is to set a maximum line
length. If the line exceeds that, print line(:(MAX -100)
+ '...' + line(:-100) instead of printing the whole thing
which no one wants to look at anyway.

Another thing I've wanted to do is provide the ability
to clear the shell window when it gets too full, w/o
restarting IDLE.

Yes, you might ask the tkinter guys on their mail list,
I'd be interested in hearing their reply.

Logged In: YES
user_id=1330769

The Squeezer extension works like a charm! It's also been
thoroughly tested by tens of users over the past several years.

Why not include it as one of the default extensions, and
have it enabled by default?

BTW I have a tweaked version of Squeezer (I fixed the line
counting code), if you're interested.

Logged In: YES
user_id=149084

Sure, please open a patch and supply a diff against
Noam's version.

Logged In: YES
user_id=1330769

Done.

Logged In: YES
user_id=149084

Patch 1529353 by Tal Einat

Patch review in bpo-1529353.

print(100000*'a') is still a problem in 3.3.

In addition to squeezing, it would be nice (and easy) to add a menu option (and hotkey) to clear the text pane.

@Raymond, see bpo-6143 for an (outdated) extension to clear the shell window with a hotkey.

The only reason that the IDLE shell is slow is due to the shell's text widget being configured to have wrap="char". If we manually wrapped the output then the shell responds very quickly to rendering really long strings.

The attached proof-of-concept patch (against 2.7 tip) implements manual wrapping. You can type "print('a' * 10**6)" and the shell responds almost instantly when using no-subprocess mode. (The RPC overhead becomes readily apparent when using a subprocess, introducing a large uninteruptable delay. That's another issue.)

I left text wrapping enabled in the shell since the user may be using a variable-spaced font. A possible compromise would be to increase the wrap_index to a large number, like 32768, before IDLE inserts a '\n' into the output. This would mimic the wrapping behavior of the original shell, but keep the shell responsive when you write a very long string to the output.

manual_wrap.patch patches OutputWindow (2.7) to arbitrarily and blindly wrap at 80 chars. OutputWindow is used directly for grep output, as well as the base for PyShell. The grep output format is currently "'{}: {}: {}'.format(filepath, linenum, codeline)". Output lines are typically (for me) > 80 chars and I would not want a fixed-column wrap. With a right click, one can go to the file and line and this should not be disabled. Ditto for tracebacks, where code lines are pre-wrapped (with an added indent) onto a second physical line. Wrap at 80 would wrap lines that were originally 80 before having the traceback indent added. Autowrap should only be applied to user stdout sent to Shell.

Perhaps wrapping should have a window (80-100?) within which we look for a space. I have about concluded that we should add horizontal scrollbars anyway, since Python and IDLE output lines longer than 80 chars.

To evaluate the patch further, I want to look at how the socket stream is being read.

As long as we are modifying user output before inserting into the text widget, astral chars should be expanded into their unicode escapes. (There are multiple issue for astral chars. Tk 8.7 reportedly will handle them.) The replacement text should be tagged and colored as such. Wrapping should not break replacements. The same could be done for control chars to make them visible. (Astral char handling is needed for paths also!).

Besides warping text, there has a performance issue inside the RPCServer and Client.

The (console, write, (text, file), {}) command is sent by server asynccall->putmessage. It should be sent by chunk size to client, and render on IDLE shell.

The result is performed as sent by chunk size, but the client will gather all chunk until it receives all data, then render on IDLE shell.

This cause the shell seems like hanging there, and doing nothing (in REPL, it will output the long string to stdout and so on).

We can manually detect this then manully chunk out (console, write, args, kwargs) command's args size, so that it will look like not hanging there.

The attach patch is a PoC about this.

-----

For the text widget performance, I dislike the wrap method, it shouldn't be a limit to the user on IDLE (GUI IDE), even it can be set to 80 or 100 or whatever.

Squeezer was added last summer, and definitely helps, but I still intend to consider other points raised here.

This is the oldest performance-labeled bug in the tracker. @terryjreedy Are you still planning to do something about it?

Squeezer is now integrated into IDLE and enabled by default. That has solved the painfully low responsiveness of IDLE's shell when displaying very long lines of text, which was the original issue raised here, and was a common occurrence.

The other points raised here should be considered separate issues IMO:

1. There is already an open issue about being able to clear IDLE's shell output: IDLE - clear and restart the shell window #50393.
2. We should create a separate issue about possibly improving IDLE's internal RPC communication to better handle large outputs and avoid long periods where no output is shown despite the sub-process already generating output.

Would you be a dear and create the issue for (2), link it here, and then close this issue? (Or I can close it, if you don't have the powers -- but I don't feel I'm the right person to file the issue, since I haven't thought about or used IDLE in decades.)

CC: @terryjreedy

@gvanrossum, I have the powers, but in all things IDLE I'm deferring to @terryjreedy who has taken the lead on it.

This case does seem rather clear-cut though, so I'll go ahead and do as you suggest.

See issue #96560.

Thanks, and sorry! I wasn’t sure if you’d made transition to GitHub with us. :)

No worries :) Unfortunately I've been extremely inactive recently, but hope to change that!

Here’s to welcome back!