New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
print(s) fails on Windows with long strings #55604
Comments
Python 3.2 fails when printing long strings. C:\Python32>python
Python 3.2 (r32:88445, Feb 20 2011, 21:30:00) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> print("a"*66000)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IOError: [Errno 12] Not enough space
>>> Some observations:
|
Likewise, this fails with 3.2:: |
It's probably a Windows limitation regarding the number of bytes that can be written to stdout in one write. |
no, it works with 3.2b2 (r32b2:87398), and fails with 3.2 final (r32:88445) |
Extract of issue bpo-1602: I suppose that os.write(1) does indirectly write into the Windows console which has this limit. |
Extract of the WriteConsole Function: << The storage for this buffer is allocated from a shared heap for the process that is 64 KB in size. The maximum size of the buffer will depend on heap usage. >> Ah ah, that's funny, "depend on heap usage" :-) |
This changed with r87824 |
Yes, I changed Python to open all files in binary mode. With Python < 3.2, you can open sys.std* streams in binary mode using -u command line option (u like unbuffered, not Unicode ;-)). |
Indeed, Python3.1 fails with the -u option. Before r87824, the C call to write() performed CRLF conversion. In the implementation of MSVCRT, a local buffer is used (1025 chars in vs8.0, 5*1024 in vs10.0), so WriteFile is called with small sizes. |
Anyway, use os.write() to write unicode into the Windows console is not the right thing to do. We should use WriteConsoleW(): bpo-1602 is the correct fix for this issue. |
I'm writing bytes here: os.write(1, b"b" * 66000) |
print() (sys.stdout and sys.stderr) should use WriteConsoleW() and use small chunks (smaller than 64 KB, I don't know the safest size). |
IIUC, this is a Windows bug? Is there any easy workaround for us? |
It may be a windows bug, but it's also an a python regression! =================================================================== --- D:/py3k/Modules/_io/fileio.c (revision 87824)
+++ D:/py3k/Modules/_io/fileio.c (copie de travail)
@@ -712,6 +712,8 @@
errno = 0;
len = pbuf.len;
#if defined(MS_WIN64) || defined(MS_WINDOWS)
+ if (len > 32000 && isatty(self->fd))
+ len = 32000;
if (len > INT_MAX)
len = INT_MAX;
n = write(self->fd, pbuf.buf, (int)len); On my system, errors start at ~52200 (why?). I hope that 32K is low enough... MSVCRT's write() (version vs10.0) uses a buffer of 5K. |
print("a"*66000) works (after some delay) running from an IDLE edit window (but see bpo-144249). Works means that I get a working prompt back with no errors. Unlike IDLE, the Command Prompt Windows keeps a limited number of lines in its buffers (default: 4x50), and may have a total size limit. |
I'm adding a test that will reproduce the crash. |
And a patch for the test + fix. |
I'm also attaching another test to reproduce the crash with '-u' option. |
I did some tests: os.write(1, b'X'*length) does always fail with length >= 63842. It does sometimes fail with length > 35000. The maximum looks completly random: as written in Microsoft documentation, "The maximum size of the buffer will depend on heap usage"... 32000 looks arbitrary. I would prefer 2^15-1 (32767) because it looks less magical :-) |
Remarks about test_wconsole_binlarge.patch:
|
Other remarks about test_wconsole_binlarge.patch:
Suggestion for the comment in fileio.c:
|
Thanks for the comment. It's my first patch. :-)
That latest patch file I generated against the tip of 3.1 branch. Should I create two separate patches for 3.1 and 3.2+ (which will apply on 3.3, as well)? Actually, this crash will reproduce on (from my testing) 2.7 with "-u" option on, as well...
I have a few ideas to work around this and still have a unit test... |
Attached a modified patch that should work against 3.2+ heads:
These changes are conditionally compiled on Windows only. Should a similar patch be made for 2.7+ (maybe earlier)? |
On Windows, isatty() is a cheap call: a simple lookup in the _ioinfo structure. And dup2() can still change the destination of a file descriptor, so the new attribute can be out of sync... |
FWIW, here's the Microsoft's source for isatty (in VC\crt\src\isatty.c): /***
*Entry:
*Exit:
*Exceptions: int __cdecl _isatty (
int fh
)
{
#if defined (_DEBUG) && !defined (_SYSCRT)
/* make sure we ask debugger only once and cache the answer */
static int knownHandle = -1;
#endif /* defined (_DEBUG) && !defined (_SYSCRT) */
#if defined (_DEBUG) && !defined (_SYSCRT)
if (knownHandle == -1) {
knownHandle = DebuggerKnownHandle();
}
if (knownHandle) {
return TRUE;
}
#endif /* defined (_DEBUG) && !defined (_SYSCRT) */
} |
Attached a version of the last patch without |
I tried to commit io_write.patch, but I had problems with Mercurial :-) I will commit it later. |
This last patch looks good, except that the comments "if stdout mode is binary (python -u)" are incorrect: since r87824, all files are opened in binary mode. |
I plan to commit the patch to 3.1 and then forward port to 3.2 and 3.3. Yes, I will adapt the comment (remove "(python -u)") for 3.2 and 3.3. |
New changeset 8939a21bdb94 by Victor Stinner in branch '3.2': New changeset 4b3472169493 by Victor Stinner in branch 'default': |
I realized that it was a little more difficult to port the fix on 3.1 because 3.1 doesn't have the fix for Windows 64 bits. So I only fixed Python 3.2 and 3.3, also because nobody reported failure for Python 3.1 on Windows with -u flag. I tested my fix: the test fails without the fix, and it pass correctly with the fix. So let's close the last regression that I introduced in Python 3.2! |
If I understand the bug in the Windows console functions correctly, a limit of 32767 bytes might not always be small enough. The problem is that if two or more threads are concurrently using any console functions (which all use the same 64 KiB heap), they could try to allocate up to 32767 bytes plus overhead at the same time, which will fail. I wasn't able to provoke this by writing to sys.stdout.buffer (maybe there is locking that prevents concurrent writes), but the following code that calls WriteFile directly, does provoke it. GetLastError() returns 8 (ERROR_NOT_ENOUGH_MEMORY; see http://msdn.microsoft.com/en-us/library/ms681382%28v=vs.85%29.aspx), indicating that it's the same bug. # Warning: this test may DoS your system. from threading import Thread
import sys
from ctypes import WINFUNCTYPE, windll, POINTER, byref, c_int
from ctypes.wintypes import BOOL, HANDLE, DWORD, LPVOID, LPCVOID
GetStdHandle = WINFUNCTYPE(HANDLE, DWORD)(("GetStdHandle", windll.kernel32))
WriteFile = WINFUNCTYPE(BOOL, HANDLE, LPCVOID, DWORD, POINTER(DWORD), LPVOID) \
(("WriteFile", windll.kernel32))
GetLastError = WINFUNCTYPE(DWORD)(("GetLastError", windll.kernel32))
STD_OUTPUT_HANDLE = DWORD(-11)
INVALID_HANDLE_VALUE = DWORD(-1).value
hStdout = GetStdHandle(STD_OUTPUT_HANDLE)
assert hStdout is not None and hStdout != INVALID_HANDLE_VALUE
L = 32760
data = b'a'*L
def run():
n = DWORD(0)
while True:
ret = WriteFile(hStdout, data, L, byref(n), None)
if ret == 0 or n.value != L:
print(ret, n.value, GetLastError())
sys.exit(1) [Thread(target=run).start() for i in range(10)] |
The buffer size only needs to be capped if WINVER < 0x602. This issue doesn't apply to Windows 8 since it uses the ConDrv device driver instead of LPC. Prior to Windows 8, WriteFile redirects to WriteConsoleA when passed a console handle. This makes an LPC call to conhost.exe (csrss.exe before Windows 7), which copies the buffer to a shared heap. But a Windows 8 console process instead has actual File handles provided by the ConDrv device:
For File handles, ReadFile and WriteFile simply call the NT system functions NtReadFile and NtWriteFile. The buffer size is only limited by available memory. |
This issue is closed. You should reopen it or open a new one. |
I've been hit by this issue recently. On my configuration, print("a" * 10215) fails with an infinite loop of OSErrors (WinError 8). This even cannot by interrupted with Ctrl-C nor the exception can be catched.
My configuration is Python 3.7.3 64 bit on Windows Vista 64 bit. I wonder if anyone is able to reproduce this on their configuration. |
The .py file association is probably using a different version of Python, or it's associated with the py launcher and there's a shebang in the file that runs 2.7. Verify the version that's executing the script. If it's running in 3.7, follow up in a new issue or on python-list. Please do not follow up on this resolved issue. Also, please try to reproduce the issue on a supported, updated system. Windows Vista is not supported by 3.7, and the latest 3.7 release is 3.7.7. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: