Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IDLE: Have the shell mimic terminal handling of \r and \b control characters in outputs #82008

Closed
taleinat opened this issue Aug 12, 2019 · 8 comments
Assignees
Labels
3.9 only security fixes topic-IDLE type-feature A feature request or enhancement

Comments

@taleinat
Copy link
Contributor

BPO 37827
Nosy @rhettinger, @terryjreedy, @taleinat, @csabella
PRs
  • bpo-37827: IDLE shell handling of \r and \b control chars #15211
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/terryjreedy'
    closed_at = <Date 2020-09-19.19:10:07.559>
    created_at = <Date 2019-08-12.06:52:53.755>
    labels = ['expert-IDLE', 'type-feature', '3.9']
    title = 'IDLE: Have the shell mimic terminal handling of \\r and \\b control characters in outputs'
    updated_at = <Date 2020-09-19.19:57:03.203>
    user = 'https://github.com/taleinat'

    bugs.python.org fields:

    activity = <Date 2020-09-19.19:57:03.203>
    actor = 'taleinat'
    assignee = 'terry.reedy'
    closed = True
    closed_date = <Date 2020-09-19.19:10:07.559>
    closer = 'taleinat'
    components = ['IDLE']
    creation = <Date 2019-08-12.06:52:53.755>
    creator = 'taleinat'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 37827
    keywords = ['patch']
    message_count = 8.0
    messages = ['349440', '349443', '350474', '350599', '350639', '356591', '377182', '377191']
    nosy_count = 4.0
    nosy_names = ['rhettinger', 'terry.reedy', 'taleinat', 'cheryl.sabella']
    pr_nums = ['15211']
    priority = 'normal'
    resolution = 'rejected'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue37827'
    versions = ['Python 3.9']

    @taleinat
    Copy link
    Contributor Author

    IDLE's shell doesn't currently handle \r and \b in any special way; they are written the the Tk Text widget which displays them in strange, system-dependent ways.

    These are often used to show continuously updated progress, e.g. in text-based progress bars, without flooding the output, since they allow overwriting previously written output. If we implement handling for \r and \b, progress indicators such as these could finally work properly in IDLE's shell.

    To make things worse, Tk's Text widget becomes increasingly slow when it wraps very long lines. IDLE's shell must wrap lines, and is therefore prone to such slowdowns. Attempting to show updating progress info using \r or \b results in such increasingly long lines of output, eventually slowing IDLE's shell down to a crawl. (The recent addition of squeezing of long outputs help for the case of single, very long outputs, but not with many short strings written on a single line.)

    As a recent example, the basic Tensorflow tutorial shows such progress information for several of its stages. Due to the lack of handling for these control characters, it is practically unusable in the IDLE shell due to this. See issue bpo-37762 about this.

    Since the shell aims to closely emulate using an interactive terminal session, and since showing progress is so common in interactive work, I propose to add special handling of these two control characters in outputs written to the shell window.

    Related issues:

    bpo-23220: Documents input/output effects of how IDLE runs user code (originally titled "IDLE does not display \b backspace correctly")

    bpo-24572: IDLE Text Output With ASCII Control Codes Not Working (marked as duplicate of bpo-23220)

    bpo-37762: IDLE very slow due a super long line output in chunks

    StackOverflow: what is the difference between cmd and idle when using tqdm? (https://stackoverflow.com/questions/35895864/what-is-the-difference-between-cmd-and-idle-when-using-tqdm)

    @taleinat taleinat added the 3.9 only security fixes label Aug 12, 2019
    @taleinat taleinat added topic-IDLE type-feature A feature request or enhancement labels Aug 12, 2019
    @taleinat
    Copy link
    Contributor Author

    See PR #59416 with a working implementation.

    @gvanrossum
    Copy link
    Member

    I agree -- this was added to Emacs a long time ago and it makes a big difference for people (like myself) who do a lot of work in Emacs shell windows. I imagine it's the same for IDLE.

    @terryjreedy
    Copy link
    Member

    In this post, I slightly expand the terminal mode proposal and define the context in which I will accept it. The context will be the subject of other issues. In the next post (tomorrow) I will try to define a reviewable specification for the terminal mode.
    ---

    The proposal that IDLE's should *always* respond to \b and \r "properly
    , like terminals" is the combined proposal of bpo-24572 and the original proposal of bpo-23220. The combined proposal was rejected in the latter, for reasons best given by Serhiy in msg246602. More is given below. bpo-23220 was then redefined as a doc issue.

    I noted in bpo-23220 that one solution for people who wanted terminal-like behavior would be to add an option to execute code being edited to the system terminal. However, this a) throws away the benefits of running in Shell and b) would not apply to interactive exploration. While running in a terminal is needed for certain use cases, it is not needed for handling control of line output.

    What Shell should do with control characters in output depends on the users goal. I propose that Shell should have 3 display modes that treat ascii control codes differently for different use cases.

    1. Raw mode (the current mode). IDLE inserts all BMP chars into the tk Text. This is needed if one is using a font that has glyphs associated with control codes. On Windows 10, the 8514oem font (the MSDOS font) has symbols for all control codes except NUL and newline. This is a rare need; this mode should not be the default.

    2. Development mode (my proposed new default). To the extent possible, developers should be able to see the characters their program produces without adding repr() to all their print and write statements (and later deleting them). Control chars would be replaced with unique glyphs, such as 'Control Pictures', \u2400-\u241f,
      ␀ ␁ ␂ ␃ ␄ ␅ ␆ ␇ ␈␉␊␋␌␍␎␏␐ ␑ ␒ ␓ ␔ ␕ ␖ ␗ ␘ ␙ ␚ ␛ ␜ ␞ ␟.
      or maybe circled letters, \u24d0-\u24df,\u24b6-\u24c5, ⓐ-ⓟ, Ⓐ-Ⓟ. Replacements would be tagged as such and get distinct highlight colors. The purpose of a visible tag is to differentiate replacements from characters actually output by the users program.

    To fulfill the goal of Development mode, more is needed. Astral chars, instead of causing an exception, should be replaced by a string consisting of their \U escape and tagged as a replacement. This is needed anywhere astral chars can appear. It would also be nice to get the codepoint and name of any displayed character. If implemented, this should be added to the base editor class.

    1. Terminal mode. Tk interprets \n and \t. Also interpret (to begin with) \a, \b, and \r. The motivation for the latter two has already been given. As said above, the specification is deferred to another post.

    @terryjreedy terryjreedy changed the title IDLE: Have the shell mimic terminal handling of \r and \b control characters in outputs IDLE Shell: add a terminal mode that responds to \a, \b, and \r Aug 27, 2019
    @taleinat
    Copy link
    Contributor Author

    Terry, thanks for the detailed writing of your thoughts on the matter and their context.

    Serhiy's argument (in msg246602) is that different terminals interpret different control characters in different ways, and that we have no way of unifying their behavior. This is true, and I will add that we should also not aim to fully emulate any single terminal nor multiple types of terminals.

    On the other hand, there is a common ground where the vast majority of terminals do, in fact, behave very similarly WRT control characters. For example, IDLE already does interpret '\n' in a special way (partially because the underlying Tk text widget does). The same goes for '\t'.

    I argue that we should instead *slightly* expand the set of control characters which IDLE interprets, to include a few more which are universally treated in a consistent manner.

    Looking at the list of ASCII control characters on Wikipedia[1], I don't think any beyond \a, \b, \n, \r and \t are universal enough and in common enough use to merit inclusion. (For reference, \a is for "bell".)

    ..[1]: https://en.wikipedia.org/wiki/C0_and_C1_control_codes

    I find the suggestion to have more than a single "mode" for the IDLE shell contrary to the "simple and novice friendly" design principle that we are aiming for. I also think that it would bring little added benefit to the great majority of our users.

    As for other control characters and astral characters, I very much agree that we could do better than to have them often "garbled" as done by the Tk text widget. I think this should be dealt with as a separate issue.

    @taleinat
    Copy link
    Contributor Author

    Note that most of the issues with special characters described in Terry latest comment have been addressed by the fix for bpo-13153.

    Can we continue discussing just properly rendering \r and \b, e.g. as done in PR #59416? Let's just decide whether we want this or not.

    My main point in favor is that we already have special support for two similar control characters, \n and \t. IMO we should expand this to the other very standardized ones, \r and \b, and perhaps \a ("bell") as well.

    @taleinat
    Copy link
    Contributor Author

    As the creator of this issue, I'm reverting the name back to the original, since I did not propose to add a new "mode" to the IDLE shell.

    Such a proposal could be a separate issue.

    I'm unfortunately going to mark close this as "rejected". If in the future there is interest in this again, it may be reopened.

    @taleinat taleinat changed the title IDLE Shell: add a terminal mode that responds to \a, \b, and \r IDLE: Have the shell mimic terminal handling of \r and \b control characters in outputs Sep 19, 2020
    @taleinat
    Copy link
    Contributor Author

    I will highlight the fact that IDLE's shell currently allows no way of showing a progress indicator, beyond extremely simple bar that only fill up from left to right, without any indication of the rightmost limit, a percentage indicator, or a rotating progress indicator. Specifically, all common progress indicator libraries do not work properly in IDLE.

    This is a true detriment to interactive work.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.9 only security fixes topic-IDLE type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    3 participants