Test failures with non-ascii character in hostname on Windows #70414
Comments
|
Compiled latest master and ran test suite (Py_DEBUG build). A few failures here and there, and some tests skipped. I haven't yet looked into getting the proper libraries to make some of the skipped tests execute, I'm trying to make the whole test suite pass first. Attached file includes verbose output for all failed tests. The end of the file and the traceback at the end of the message here happen, then Python hangs and never resumes (tested for ~20 minutes). Python 3.6.0a0 (default, Jan 27 2016, 10:49:09) [MSC v.1900 32 bit (Intel)] on w
in32
Type "help", "copyright", "credits" or "license" for more information.
>>> import test.regrtest
>>> test.regrtest.main_in_temp_cwd() # same as 'py -m test'
== CPython 3.6.0a0 (default, Jan 27 2016, 10:49:09) [MSC v.1900 32 bit (Intel)]
== Windows-7-6.1.7601-SP1 little-endian
== hash algorithm: siphash24 32bit
== E:\GitHub\cpython\build\test_python_13304
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0,
dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0,
bytes_warning=0, quiet=0, hash_randomization=1, isolated=0)
[ 1/400] test_grammar
[ 2/400] test_opcodes
[ 3/400] test_dict
[ 4/400] test_builtin
[ 5/400] test_exceptions
[ 6/400] test_types
[ 7/400] test_unittest
[ 8/400] test_doctest
[ 9/400] test_doctest2
[ 10/400] test_support
[ 11/400] test___all__
[ 12/400] test___future__
[ 13/400] test__locale
[ 14/400] test__opcode
[ 15/400] test__osx_support
[ 16/400] test_abc
[ 17/400] test_abstract_numbers
[ 18/400] test_aifc
[ 19/400] test_argparse
[ 20/400] test_array
[ 21/400] test_asdl_parser
[ 22/400] test_ast
[ 23/400] test_asynchat
[ 24/400] test_asyncio
E:\GitHub\cpython\lib\asyncio\sslproto.py:327: ResourceWarning: unclosed transport
<asyncio.sslproto._SSLProtocolTransport object at 0x05EA8BD0>
warnings.warn("unclosed transport %r" % self, ResourceWarning)
test test_asyncio failed -- multiple errors occurred; run in verbose mode for details
[ 25/400/1] test_asyncore
[ 26/400/1] test_atexit
[ 27/400/1] test_audioop
[ 28/400/1] test_augassign
[ 29/400/1] test_base64
[ 30/400/1] test_bigaddrspace
[ 31/400/1] test_bigmem
[ 32/400/1] test_binascii
[ 33/400/1] test_binhex
[ 34/400/1] test_binop
[ 35/400/1] test_bisect
[ 36/400/1] test_bool
[ 37/400/1] test_buffer
[ 38/400/1] test_bufio
[ 39/400/1] test_bytes
[ 40/400/1] test_bz2
[ 41/400/1] test_calendar
[ 42/400/1] test_call
[ 43/400/1] test_capi
[ 44/400/1] test_cgi
[ 45/400/1] test_cgitb
[ 46/400/1] test_charmapcodec
[ 47/400/1] test_class
[ 48/400/1] test_cmath
[ 49/400/1] test_cmd
[ 50/400/1] test_cmd_line
[ 51/400/1] test_cmd_line_script
[ 52/400/1] test_code
[ 53/400/1] test_code_module
test test_code_module failed -- multiple errors occurred; run in verbose mode for details
[ 54/400/2] test_codeccallbacks
[ 55/400/2] test_codecencodings_cn
[ 56/400/2] test_codecencodings_hk
[ 57/400/2] test_codecencodings_iso2022
test test_codecencodings_iso2022 failed -- multiple errors occurred; run in verbose mode for details
[ 58/400/3] test_codecencodings_jp
[ 59/400/3] test_codecencodings_kr
[ 60/400/3] test_codecencodings_tw
[ 61/400/3] test_codecmaps_cn
[ 62/400/3] test_codecmaps_hk
[ 63/400/3] test_codecmaps_jp
[ 64/400/3] test_codecmaps_kr
[ 65/400/3] test_codecmaps_tw
[ 66/400/3] test_codecs
[ 67/400/3] test_codeop
[ 68/400/3] test_collections
[ 69/400/3] test_colorsys
[ 70/400/3] test_compare
[ 71/400/3] test_compile
[ 72/400/3] test_compileall
[ 73/400/3] test_complex
[ 74/400/3] test_concurrent_futures
[ 75/400/3] test_configparser
[ 76/400/3] test_contains
[ 77/400/3] test_contextlib
[ 78/400/3] test_copy
[ 79/400/3] test_copyreg
[ 80/400/3] test_coroutines
[ 81/400/3] test_cprofile
[ 82/400/3] test_crashers
[ 83/400/3] test_crypt
test_crypt skipped -- No module named '_crypt'
[ 84/400/3] test_csv
[ 85/400/3] test_ctypes
[ 86/400/3] test_curses
test_curses skipped -- Use of the 'curses' resource not enabled
[ 87/400/3] test_datetime
[ 88/400/3] test_dbm
[ 89/400/3] test_dbm_dumb
[ 90/400/3] test_dbm_gnu
test_dbm_gnu skipped -- No module named '_gdbm'
[ 91/400/3] test_dbm_ndbm
test_dbm_ndbm skipped -- No module named '_dbm'
[ 92/400/3] test_decimal
[ 93/400/3] test_decorators
[ 94/400/3] test_defaultdict
[ 95/400/3] test_deque
[ 96/400/3] test_descr
[ 97/400/3] test_descrtut
[ 98/400/3] test_devpoll
test_devpoll skipped -- test works only on Solaris OS family
[ 99/400/3] test_dictcomps
[100/400/3] test_dictviews
[101/400/3] test_difflib
[102/400/3] test_dis
[103/400/3] test_distutilsE:\GitHub\cpython\build\test_python_13304>exit 1 E:\GitHub\cpython\build\test_python_13304>exit 0
Warning -- files was modified by test_distutils
test test_distutils failed -- multiple errors occurred; run in verbose mode for details
[104/400/4] test_docxmlrpc
[105/400/4] test_dummy_thread
[106/400/4] test_dummy_threading
[107/400/4] test_dynamic
[108/400/4] test_dynamicclassattribute
[109/400/4] test_eintr
[110/400/4] test_email
[111/400/4] test_ensurepip
[112/400/4] test_enum
[113/400/4] test_enumerate
[114/400/4] test_eof
[115/400/4] test_epoll
test_epoll skipped -- test works only on Linux 2.6
[116/400/4] test_errno
[117/400/4] test_exception_variations
[118/400/4] test_extcall
[119/400/4] test_faulthandler
[120/400/4] test_fcntl
test_fcntl skipped -- No module named 'fcntl'
[121/400/4] test_file
[122/400/4] test_file_eintr
[123/400/4] test_filecmp
[124/400/4] test_fileinput
[125/400/4] test_fileio
[126/400/4] test_finalization
[127/400/4] test_float
[128/400/4] test_flufl
[129/400/4] test_fnmatch
[130/400/4] test_fork1
test_fork1 skipped -- object <module 'os' from 'E:\\GitHub\\cpython\\lib\\os.py'> has no attribute 'fork'
[131/400/4] test_format
[132/400/4] test_fractions
[133/400/4] test_frame
[134/400/4] test_fstring
[135/400/4] test_ftplib
[136/400/4] test_funcattrs
[137/400/4] test_functools
[138/400/4] test_future
[139/400/4] test_future3
[140/400/4] test_future4
[141/400/4] test_future5
[142/400/4] test_gc
[143/400/4] test_gdb
test_gdb skipped -- Couldn't find gdb on the path
[144/400/4] test_generators
[145/400/4] test_genericpath
[146/400/4] test_genexps
[147/400/4] test_getargs2
[148/400/4] test_getopt
[149/400/4] test_getpass
[150/400/4] test_gettext
[151/400/4] test_glob
[152/400/4] test_global
[153/400/4] test_grp
test_grp skipped -- No module named 'grp'
[154/400/4] test_gzip
[155/400/4] test_hash
[156/400/4] test_hashlib
[157/400/4] test_heapq
[158/400/4] test_hmac
[159/400/4] test_html
[160/400/4] test_htmlparser
[161/400/4] test_http_cookiejar
[162/400/4] test_http_cookies
[163/400/4] test_httplib
test test_httplib failed -- multiple errors occurred; run in verbose mode for details
[164/400/5] test_httpservers
Exception in thread Thread-857:
Traceback (most recent call last):
File "E:\GitHub\cpython\lib\threading.py", line 916, in _bootstrap_inner
self.run()
File "E:\GitHub\cpython\lib\test\test_httpservers.py", line 42, in run
self.server = HTTPServer(('localhost', 0), self.request_handler)
File "E:\GitHub\cpython\lib\socketserver.py", line 443, in __init__
self.server_bind()
File "E:\GitHub\cpython\lib\http\server.py", line 140, in server_bind
self.server_name = socket.getfqdn(host)
File "E:\GitHub\cpython\lib\socket.py", line 662, in getfqdn
hostname, aliases, ipaddrs = gethostbyaddr(name)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc9 in position 0: invalid
continuation byte |
|
For the test_httpservers failure, can you please try the following commands on your PC? >>> socket.gethostname()
'selma'
>>> socket.gethostbyaddr(socket.gethostname())
('selma', [], ['2a01:e34:ec8d:4c70:3ea9:f4ff:fe65:c0c', 'fe80::3ea9:f4ff:fe65:c0c']) |
|
Well, it has a non-ASCII character in it, so I wouldn't be surprised if this was the issue :) Latest master (3.6): >>> import socket
>>> socket.gethostname()
'Émanuel-PC'
>>> socket.gethostbyaddr(socket.gethostname())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
socket.gaierror: [Errno 11004] getaddrinfo failedPython 2.7.11: >>> import socket
>>> socket.gethostname()
'\xc9manuel-PC'
>>> socket.gethostbyaddr(socket.gethostname())
('\xc9manuel-PC.home', [], ['fe80::c9b7:5117:eea4:a104']) |
|
I'm surprised by the test_codecencodings_iso2022 failures. Example: ====================================================================== Traceback (most recent call last):
File "E:\GitHub\cpython\lib\test\multibytecodec_support.py", line 208, in test_incrementaldecoder
self.assertEqual(ostream.getvalue(), self.tstring[1])
AssertionError: b'\xe[334 chars]\x80\n\xed\x9a\xa8\xec\x9c\xa8\xec\xa0\x81\xec[1668 chars]4.\n' != b'\xe[334 chars]\x80\r\n\xed\x9a\xa8\xec\x9c\xa8\xec\xa0\x81\x[1682 chars]\r\n'It looks like Python uses the wrong encoding to decode stdout of child processes: ====================================================================== Traceback (most recent call last):
File "E:\GitHub\cpython\lib\distutils\tests\test_build_ext.py", line 61, in test_build_ext
cmd.run()
File "E:\GitHub\cpython\lib\distutils\command\build_ext.py", line 338, in run
self.build_extensions()
File "E:\GitHub\cpython\lib\distutils\command\build_ext.py", line 447, in build_extensions
self._build_extensions_serial()
File "E:\GitHub\cpython\lib\distutils\command\build_ext.py", line 472, in _build_extensions_serial
self.build_extension(ext)
File "E:\GitHub\cpython\lib\distutils\command\build_ext.py", line 532, in build_extension
depends=ext.depends)
File "E:\GitHub\cpython\lib\distutils\_msvccompiler.py", line 306, in compile
self.initialize()
File "E:\GitHub\cpython\lib\distutils\_msvccompiler.py", line 199, in initialize
vc_env = _get_vc_env(plat_spec)
File "E:\GitHub\cpython\lib\distutils\_msvccompiler.py", line 92, in _get_vc_env
universal_newlines=True,
File "E:\GitHub\cpython\lib\subprocess.py", line 636, in check_output
**kwargs).stdout
File "E:\GitHub\cpython\lib\subprocess.py", line 705, in run
stdout, stderr = process.communicate(input, timeout=timeout)
File "E:\GitHub\cpython\lib\subprocess.py", line 1062, in communicate
stdout = self.stdout.read()
File "E:\GitHub\cpython\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 49: character maps to <undefined>It would help to add a print() to lib\distutils\_msvccompiler.py:92 to get the executed command, then run manually the command, and see non-ASCII characters in the output. |
|
Python 2.7.11: >>> import socket
>>> socket.gethostname()
'\xc9manuel-PC'This one works on Python 3 because the Python function is implemented with a call to the Windows native API. >>> socket.gethostbyaddr(socket.gethostname())
('\xc9manuel-PC.home', [], ['fe80::c9b7:5117:eea4:a104'])This one fails on Python 3 because it uses the gethostbyaddr() C function and then decodes the hostname from UTF-8, whereas the hostname looks more to be encoded to ISO 8859-1 or something like that. IMHO it should be decoded from the ANSI code page. I opened the issue bpo-26227 to track this bug. |
====================================================================== Traceback (most recent call last):
File "E:\GitHub\cpython\lib\test\multibytecodec_support.py", line 208, in test_incrementaldecoder
self.assertEqual(ostream.getvalue(), self.tstring[1])
AssertionError: b'\xe[334 chars]\x80\n\xed\x9a\xa8\xec\x9c\xa8\xec\xa0\x81\xec[1668 chars]4.\n' != b'\xe[334 chars]\x80\r\n\xed\x9a\xa8\xec\x9c\xa8\xec\xa0\x81\x[1682 chars]\r\n'I run into this problem during a sprint, with a machine that was also running Windows and cloned from the git repo. Those bytes are read from txt files in the test/cjkencodings dir, and they are marked as binary files in .hgeol. Since git ignores .hgeol and treats them as text files, it also changes the newline to \r\n causing the failure. |
|
Regarding the files getting messed up in Lib/test/cjkencodings, IMO the solution is to not configure Git to mess with newlines in files. ;) But perhaps a reasonable workaround might be to rename the relevant files to something like *.txt.bin. Perhaps that will bypass whatever logic Git uses. |
|
It looks like all issues are now fixed in the default branch, no? Can we close this issue? |
|
It looks like this issue can be closed, but a new one should be opened for dealing with the newline issue under git on windows. However, I'm surprised there weren't failures in the email tests. Although I've been moving the email tests away from depending on data on the disk being in DOS format, I think there are some compat32 tests that still do so. |
|
Some of the tests that used to fail are now passing, however some are still failing (and new ones are also failing now). Tests that are still failing: test_code_module (TestInteractiveConsole; test_ps1 and test_ps2 failed) New failures: test_capi (_testcapi doesn't have test_buildvalue_N) Tests previously hung up after that point. The tests continue now, so here are more failures: test_logging (socket timeout) I'll spawn new issues for the more serious/specific ones, but unless you'd rather have one issue per failure, I'd like to keep this one open (for now at least). @r. David: 4 skips, but no failures in the email tests, so at least there's that! |
|
About the strptime failure - it seems time.strftime returns "Est (heure d?été)" while _strptime.LocaleTime().timezone has "est (heure d\x92été)". I don't know why it's like that, but neither are correct - the character that goes there is a single quote ('). I'm also puzzled by the fact that 'é' appears properly despite not being ASCII. |
|
test_distutils: Is the failure the same as before, or changed? Are you testing with new code that includes the bpo-27048 fix? It looks like test_distutils doesn’t support running directly via “unittest”. Perhaps try “python -m test -W test_distutils” or similar if you were previously only trying unittest. Some of your new failures look like symptoms of running the test suite on out-of-date compiled modules. For test_ctypes, see bpo-27343; test_getargs2: bpo-26282; test_capi: bpo-26168. strptime: Maybe the test suite is trying to report a string that contains both U+0092 and “é”, but cannot encode U+0092, and uses backslashreplace error handler. |
|
time.strftime calls the CRT's strftime function, which the Windows universal CRT implements by calling wcsftime and encoding the result. The timezone name is actually stored as a char string (tzname), so wcsftime has to decode it via mbstowcs. The problem is that in the C locale tzname is an ANSI (1252) string while mbstowcs simply casts to wchar_t, which is the same as decoding as Latin-1. This works fine for "é" (U+00E9). But the right single quote character (U+2019) is "\x92" in 1252, and a simple cast maps it to the non-character U+0092. When the CRT's strftime encodes this back as an ANSI string, it maps U+0092 to the replacement character for 1252, a question mark. Similarly, time.tzname decodes the tzname ANSI strings using mbstowcs, with the same mismatch between LC_CTYPE and LC_TIME, resulting in the string "Est (heure d\x92été)" In summary, the problem is that LC_TIME uses ANSI in the C locale, while LC_CTYPE uses Latin-1. A workaround (in most cases) is to delay importing the time module until after setting LC_CTYPE (also setting LC_TIME should cover all cases). For example: >>> import sys, locale
>>> 'time' in sys.modules
False
>>> locale.setlocale(locale.LC_CTYPE, '')
'French_France.1252'
>>> import time
>>> time.tzname
('Est', 'Est (heure d’été)')
>>> time.strftime('%Z')
'Est (heure d’été)'Note that Unix Python 3 sets LC_CTYPE at startup, so doing the same on Windows would actually improve cross-platform consistency. |
|
That is good to know, thanks Eryk for the explanation! I also think making Python set LC_CTYPE on startup on Windows would be good. Martin's comment got me wondering, so I'm going to try to see if all modules properly got re-compiled, and re-run tests. I'll also run test_distutils through the command line to see if it passes. Following that, I'll branch off the relevant issues; and hopefully close this one too (I realize that it started going in all directions, it's hard to keep track even for me now). |
pppery
mannequin
changed the title
|
So I checked to make sure everything re-compiled properly and re-ran tests. Now, only a very few fail, and I've opened separate issues for each of them. test_os: bpo-27423 I'm keeping this one open specifically for the test_uuid failure, which seems to be caused by my hostname having non-ascii characters (and this was more or less the focus of this issue for a while). |
Vgr255
mannequin
changed the title
|
Since non-ASCII characters are not really supported in hostnames, I changed mine to a saner alternative. This issue can be closed then, since any test failure I encounter can no longer be because of this. One last thing: is it safe to say "Don't use non-ASCII in hostnames" and not bother supporting such edge cases? Whichever way we end up going, I think this should be documented somewhere (although I don't really have an idea of where). I can submit a doc patch for this. |
|
Please keep it ok. I don't say that we are going to fix all issues, but it's good to have an issue which collects as much as possible information about the bug and how we can fix it. |
|
Python not working.. |
|
I'm not sure that all issues have been fixed, but almost all of them are. I prefer to close this issue which has a long history and mentions very different bugs. If someone wants to continue the effort for better Unicode support on Windows with non-ASCII hostname, please open a new specific issues. Thanks Emanuel Barry for your bug report! |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: