Title: OSError: raw write() returned invalid length on latest Win 10 Consoles
Type: behavior Stage: resolved
Components: IO, Windows Versions: Python 3.6, Python 3.5
Status: closed Resolution: third party
Dependencies: Superseder:
Assigned To: Nosy List: Segev Finer, Simon Depiets, eryksun, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2017-12-07 15:21 by Simon Depiets, last changed 2018-09-18 20:28 by eryksun. This issue is now closed.

Messages (8)
msg307811 - (view) Author: Simon Depiets (Simon Depiets) Date: 2017-12-07 15:21
A couple of users have been having issues on console output since the Fall 2017 Creator Update on Windows 10

An OSError is triggered randomly when rewriting data on the console (typically with progress bars, for instance when you install a module with pip), this only happens with the Microsoft Console (within Powershell or cmd.exe).

It seems the windows stdout console stream returns a length double what python expects. I don't have the skills to go deeper than the bufferedio.c method _bufferedwriter_raw_write to diagnostic the issue, so I've made a very dirty fix (which I do not recommend)

Different unrelated use cases where an error is triggered :
msg307918 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-12-09 21:23
Does this only affect Python 3.5? We're highly unlikely to take a fix for that version.

This code was rewritten for 3.6, so it wouldn't surprise me if there is no longer an issue.
msg307920 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2017-12-09 21:30
To confirm what Steve said: we no longer accept bug fixes for Python 3.5 (or 3.4).  We only accept security fixes for 3.5 (and 3.4).
msg307926 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-12-09 22:15
We need a test that reproduces this problem on a vanilla installation of Python 3.5. Include the system locale context as well, i.e. the ANSI codepage, OEM codepage, and active console output codepage. A reliable test will help determine whether this problem also affects legacy console I/O in Python 3.6. However, even if the problem affects 3.6, that doesn't mean there's anything we can reasonably do about it if the fault is the console host process (i.e. conhost.exe, running either ConhostV1.dll or the new ConhostV2.dll implementation). 

A possible workaround in Python 3.5 would be to install and enable the win_unicode_console package. This package uses the console's native Unicode API instead of the legacy codepage API.
msg308012 - (view) Author: Simon Depiets (Simon Depiets) Date: 2017-12-11 03:37
The issue doesn't seem to happen on either 3.6 (with the new stdio mode) or with win_unicode_console enabled.

I was able to reproduce it on 3.6 with the PYTHONLEGACYWINDOWSSTDIO flag enabled, it's easier to trigger when using directional keys or mouse clicks within the console.

chcp : Active code page : 437

System locale : en-us
msg308088 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-12-12 05:30
I was able to reproduce this problem in 3.6 in Windows 10 (1709), but only for code paths that call WriteFile, i.e. os.write and legacy standard I/O mode.

Writing to the console will block if there's an active text selection. The operation completes once the user copies the selection to the clipboard (e.g. by pressing enter). In the case of WriteFile, when the blocked call finally completes, the console mistakenly returns the number of internally written UTF-16 bytes. 

This bug is not present with WriteConsoleA, which means there's a simple workaround. _Py_write could detect a console handle and call WriteConsoleA instead of _write. 

The following test executes a write on a separate thread, after a timed delay that allows selecting text beforehand. Text can be selected via Ctrl+A, Shift+Up, or the mouse (the latter requires quick-edit mode or toggling mark mode). 

    import os
    import sys
    import ctypes
    import msvcrt
    import threading

    kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
    fd_stdout = sys.stdout.fileno()
    h_stdout = msvcrt.get_osfhandle(fd_stdout)
    n = ctypes.c_ulong()

    def test_write():
        n.value = os.write(fd_stdout, b'spam\r\n')

    def test_WriteFile():
        kernel32.WriteFile(h_stdout, b'spam\r\n', 6, ctypes.byref(n), None)

    def test_WriteConsoleA():
        kernel32.WriteConsoleA(h_stdout, b'spam\r\n', 6, ctypes.byref(n), None)

For example, given manual text selection after starting the timer:

    >>> threading.Timer(5, test_write).start()
    >>> spam

    >>> n

    >>> threading.Timer(5, test_WriteFile).start()
    >>> spam

    >>> n

    >>> threading.Timer(5, test_WriteConsoleA).start()
    >>> spam

    >>> n

This test could be completely automated by combining SendInput (to set the session control-key state), GetConsoleWindow, and PostMessageW (WM_KEYDOWN, WM_KEYUP).
msg308595 - (view) Author: Segev Finer (Segev Finer) * Date: 2017-12-19 00:13
Upstream issue, also has links to similar issues this caused in other projects.
msg321350 - (view) Author: Segev Finer (Segev Finer) * Date: 2018-07-09 21:04
This should be fixed by the Windows 10 April Update (build 1803) according to
Date User Action Args
2018-09-18 21:42:20eryksunlinkissue34727 superseder
2018-09-18 20:28:31eryksunsetstatus: open -> closed
resolution: third party
stage: test needed -> resolved
2018-07-09 21:04:52Segev Finersetmessages: + msg321350
2017-12-19 00:13:25Segev Finersetnosy: + Segev Finer
messages: + msg308595
2017-12-12 05:30:57eryksunsetmessages: + msg308088
versions: + Python 3.6
2017-12-11 03:37:31Simon Depietssetmessages: + msg308012
2017-12-10 03:24:34larrysetnosy: - larry
2017-12-09 22:15:36eryksunsetnosy: + eryksun
messages: + msg307926
2017-12-09 21:30:05larrysetmessages: + msg307920
2017-12-09 21:23:15steve.dowersetnosy: + larry
messages: + msg307918
2017-12-09 20:36:23eryksunsetnosy: + paul.moore, tim.golden, zach.ware

components: + Windows
stage: test needed
2017-12-09 07:07:35berker.peksagsetnosy: + steve.dower
2017-12-07 15:21:22Simon Depietscreate