classification
Title: Windows: Python not using ANSI compatible console
Type: enhancement Stage: resolved
Components: Windows Versions: Python 3.7
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: eryksun, joseph.hackman, methane, paul.moore, steve.dower, terry.reedy, tim.golden, v+python, zach.ware
Priority: normal Keywords: patch

Created on 2016-12-24 01:33 by joseph.hackman, last changed 2017-04-21 23:43 by terry.reedy. This issue is now closed.

Files
File name Uploaded Description Edit
issue29059.patch joseph.hackman, 2016-12-28 05:12 review
Messages (13)
msg283913 - (view) Author: Joseph Hackman (joseph.hackman) * Date: 2016-12-24 01:33
On windows, Python does not request that Windows enable VT100 for console output for Python.

That is to say that ANSI codes such as \033[91m will function on Mac and Linux Pythons, but not Windows.

As there is no good interface in core Python to the kernel32 console operations (and there probably shouldn't be, it would be better to be consistent), I suggest just flipping the bit at startup on Windows.

I would be happy to submit a patch, but seek guidance on the best location for addition of code to 'at startup iff a tty is attached'.

The following code solves the issue:
import platform
if platform.system().lower() == 'windows':
    from ctypes import windll, c_int, byref
    stdout_handle = windll.kernel32.GetStdHandle(c_int(-11))
    mode = c_int(0)
    windll.kernel32.GetConsoleMode(c_int(stdout_handle), byref(mode))
    mode = c_int(mode.value | 4)
    windll.kernel32.SetConsoleMode(c_int(stdout_handle), mode)

Please see:
https://msdn.microsoft.com/en-us/library/windows/desktop/ms686033(v=vs.85).aspx (Search for ENABLE_VIRTUAL_TERMINAL_PROCESSING)
https://msdn.microsoft.com/en-us/library/windows/desktop/ms683231(v=vs.85).aspx (As for why stdout is -11 on Windows)
msg283979 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2016-12-25 01:25
Is it a global state, or application specific state?
In other words, if Python enables VT100, doesn't it remain after os._exit(1)?

If it is global state of console, why Python should change it?
Shouldn't user who want to use VT100 enable it before starting Python?
msg283982 - (view) Author: Joseph Hackman (joseph.hackman) * Date: 2016-12-25 02:58
The flag is application specific. I.e. a python program that writes to console  using ansi codes, when used on windows, will just display the codes. If the Output is redireced to file and then the file is printed to console using a console tool, the colors will show instead.
msg283998 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-12-25 16:02
Great suggestion. If someone is willing to contribute the work then we can consider it for 3.7.

If you need this functionality sooner, either ctypes or colorama should allow you to use it on existing releases.
msg283999 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-12-25 16:05
Oh, and it should go in the WinConsoleIO class, needs to be written in C, and probably needs a short PEP explaining why it has to be on by default and can't be turned on by programs that need it. I'm sure the discussion will lead to other ideas as well (perhaps posting on python-ideas is a good start? Though there may not be a lot of discussion for a Windows-centric suggestion like this.)
msg284153 - (view) Author: Joseph Hackman (joseph.hackman) * Date: 2016-12-28 05:12
Thanks for the tip! If you hadn't said that, I probably would have written it into the init scripts.

I'll try to write something for python-ideas / PEP tomorrow, but have attached a quick patch here so I can link to this issue for an example implementation.
msg284178 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-12-28 15:51
Sounds good. Don't focus on ease of implementation, BTW. The post and eventual PEP need to cover the benefits, risks and alternatives more than anything else.
msg284220 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2016-12-29 01:11
> As there is no good interface in core Python to the kernel32 console
> operations (and there probably shouldn't be, it would be better to be 
> consistent), I suggest just flipping the bit at startup on Windows.

I don't follow your statement about consistency. Python could adopt a built-in wincon module to partially support the console API, just like winreg partially supports the registry API. 

When _WindowsConsoleIO was being developed, I mentioned the idea of enabling VT mode a couple of times, but there was no interest that I could see. I don't think Windows devs really care much about this feature, especially since it's only available in Windows 10. colorama has been the popular way to get cross-platform support for supporting text color in terminals and the Windows console. There's also pyreadline, which supports VT escapes and much more. Enabling the console's VT mode won't simplify the implementation of colorama or pyreadline so long as Windows 7 and 8 are supported, which will be for several more years. 

One problem with enabling VT mode at startup is that it could cause compatibility problems with child processes. cmd faced this problem and has since resolved it [1] by reverting to the original console mode before executing a program. If Python enables VT mode at startup, there should be a sys variable (based on a C global) that has the original mode value. Applications would be able to restore the original mode to either opt out entirely or before executing a child process. Py_FinalizeEx would also have to be modified to restore the original mode before shutting down the interpreter.

[1]: http://preview.tinyurl.com/jgtcgfs
msg284363 - (view) Author: Glenn Linderman (v+python) * Date: 2016-12-31 04:57
Nice idea, but not by default. An easy way to switch back and forth, and to be sure the original mode is restored on process exit would be a win.

Most windows users want a real GUI, not curses, but compatibility with VT escape codes for cross-platform semi-GUI stuff could be useful... and could enable the stdlib curses to be included in Python for windows also? I'm not sure why it isn't, and "ANSI" a.k.a. VT escape codes have been available in Windows for a long time... but until Python 3.6.0 (thanks again Steve, and others), Windows console handling in Python has been extremely limited.

Also note that UniCurses and Pygcurse exist and provide similar capabilities.
msg284365 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-12-31 05:21
I suspect curses could be updated to fully support Windows, even without the VT100 mode (all the functionality has existed for years, just through a different interface). It really just needs someone to take on the project, and there are already better options out there (I forget the name, but the IPython team are very happy with whatever library they're using these days), so possibly it's more of an awareness problem that doesn't need to be solved in the core distro.
msg284369 - (view) Author: Glenn Linderman (v+python) * Date: 2016-12-31 06:49
Re: curses... maybe that becomes a DOCS issue, to mention the available packages. But it would be easier, no doubt to port curses to a known existing escape sequence control set, than to use a bunch arcane, foreign-to-the-Unix-porter-that-wants-curses obscure and seldom discussed Windows APIs.

IIRC, ANSI is somewhat incompatible with sending random binary gibberish to the screen, as people accidentally do with TYPE sometimes :) But the random binary gibberish may contain ANSI control sequences... That's why I'm negative on making it a default.
msg284459 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-01-02 04:40
> IIRC, ANSI is somewhat incompatible with sending random binary gibberish to the screen, as people accidentally do with TYPE sometimes :) But the random binary gibberish may contain ANSI control sequences... That's why I'm negative on making it a default.

I don't actually know how big a deal this would be. I seem to get identical results from "print(''.join(chr(x) for x in range(32)))" both with and without the VT100 flag set, though of course "print('\033[91m')" behaves differently.

But given "on by default" isn't popular, and "off by default" implies adding new public API that is already available either as a short ctypes snippet or a number of 3rd-party libraries, I think we should consider this rejected.

For future reference, the python-ideas thread starts with https://mail.python.org/pipermail/python-ideas/2016-December/044033.html
msg292084 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-04-21 23:43
In #30075, Tithen Firion reports that "subprocess.call('', shell=True)" leaves the console in ANSI mode.  In Eryk Sun says that this is likely a bug in cmd.exe and gives improved code for putting the console in VT mode, when this is possible.  I closed that issue as a duplicate of this.
History
Date User Action Args
2017-04-21 23:43:37terry.reedysetnosy: + terry.reedy

messages: + msg292084
stage: resolved
2017-04-15 22:50:43martin.panterlinkissue30075 superseder
2017-01-02 04:40:03steve.dowersetstatus: open -> closed
resolution: rejected
messages: + msg284459
2016-12-31 06:49:59v+pythonsetmessages: + msg284369
2016-12-31 05:21:10steve.dowersetmessages: + msg284365
2016-12-31 04:57:26v+pythonsetnosy: + v+python
messages: + msg284363
2016-12-29 01:11:01eryksunsetnosy: + eryksun
messages: + msg284220
2016-12-28 15:51:24steve.dowersetmessages: + msg284178
2016-12-28 05:12:51joseph.hackmansetfiles: + issue29059.patch
keywords: + patch
messages: + msg284153
2016-12-25 16:05:51steve.dowersetmessages: + msg283999
2016-12-25 16:02:58steve.dowersettype: behavior -> enhancement
messages: + msg283998
versions: + Python 3.7, - Python 2.7, Python 3.3, Python 3.4, Python 3.5, Python 3.6
2016-12-25 02:58:08joseph.hackmansetmessages: + msg283982
2016-12-25 01:25:19methanesetnosy: + methane
messages: + msg283979
2016-12-24 01:33:10joseph.hackmancreate