classification
Title: Windows: socket.gethostbyaddr(name) fails for non-ASCII hostname
Type: Stage:
Components: Unicode, Windows Versions: Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: abarry, eryksun, ezio.melotti, paul.moore, python-dev, serhiy.storchaka, steve.dower, tim.golden, vstinner, zach.ware
Priority: normal Keywords: patch

Created on 2016-01-28 00:58 by vstinner, last changed 2016-01-29 16:07 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
gethostbyaddr_encoding.patch vstinner, 2016-01-28 00:59
gethostbyaddr_encoding-2.patch vstinner, 2016-01-28 08:49 review
gethostbyaddr_encoding-3.patch vstinner, 2016-01-28 09:41 review
Messages (18)
msg259078 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-01-28 00:58
On Windows, socket.gethostbyaddr() must decode the hostname from the ANSI code page, not from UTF-8. See for example this issue:
https://bugs.python.org/issue26226#msg259077

Attached patch changes the socket module to decode hostnames from the ANSI code page on Windows.

See also issues #9377, #16652 and #5004.
msg259085 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-01-28 02:34
Might be nice to switch the socket APIs to the Unicode ones universally. That would also clear up a range of deprecation warnings on build.
msg259092 - (view) Author: Anilyka Barry (abarry) * (Python triager) Date: 2016-01-28 05:23
FWIW this patch doesn't fix the test_httpservers failure (or any other) in #26226
msg259106 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2016-01-28 07:55
The patch is missing the "errors" parameter of PyUnicode_DecodeLocale. But it should call PyUnicode_DecodeMBCS instead. In the "C" locale, PyUnicode_DecodeLocale is Latin-1 because the CRT mbstowcs just casts the values to wchar_t.

socket_getnameinfo also decodes as UTF-8:

    >>> socket.getnameinfo(('127.0.0.1', 20), 0)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 0: invalid start byte

Steve, does your suggestion include reimplementing socket.gethostbyaddr and socket.gethostbyname_ex using GetNameInfoW and GetAddrInfoW? gethostbyaddr and gethostbyname are deprecated and lack a Unicode implementation.
msg259108 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-01-28 08:48
> The patch is missing the "errors" parameter of PyUnicode_DecodeLocale.

Woops, I shouldn't write patch in the middle of the night :-) Hopefully, I didn't push it :-) PyUnicode_DecodeLocale() should only be used when the encoding depends on the *currenet* value of LC_CTYPE.

Here, the ANSI code page is fine, and so PyUnicode_DecodeFSDefault() should be used instead.

> socket_getnameinfo also decodes as UTF-8

Hum, let met try a new patch. It decodes hostname from the ANSI code page on Windows for:

* socket.getnameinfo()
* socket.gethostbyaddr()
* socket.gethostbyname_ex()

The behaviour on other platforms is unchanged.
msg259111 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-01-28 09:29
Added comments on Rietveld.
msg259112 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-01-28 09:41
> Added comments on Rietveld.

Crap. It's easy to miss a compilation error on extensions :-/

I used "make && ./python -m test -v test_socket" to validate  gethostbyaddr_encoding-2.patch and it succeded.

Maybe we should setup.py to *fail* if an extension failed to be compiled?

New patch should have less typos :-) I also checked for reference leak using ./python -m test -R 3:3 test_socket => no leak.


> Why not use PyUnicode_DecodeFSDefault on all platforms? It is used in
gethostname() on Unix.

I don't know which encoding is the best choice on UNIX. I prefer to move step by step and fix an obvious bug on Windows blocking Émanuel (see his issue #26226). (Émanuel uses Émanuel-PC for its hostname, an non-ASCII hostname ;-))

I guess that UTF-8 works in most cases on UNIX, whereas using the locale encoding can introduce regressions if the hostname is non-ASCII. For example, decoding non-ASCII hostname would fail with LANG=C which forces an ASCII locale encoding.

The issue #9377 proposes a more advanced code to choose the encoding to decode hostnames. Sorry, I didn't follow this issue recently, so I don't know if it proposes to use surrogateescape and/or IDNA.

I prefer to discuss the encoding used on UNIX in a new issue (or better continue the existing discussion on issue #9377?).
msg259113 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-01-28 09:42
By the way, thanks for your reviews. Code review rocks ;-)
msg259129 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-01-28 14:05
I couldn't remember the names of the alternate functions Windows provides to do the encoding for you, but yes. There are socket APIs there that do encoding and handle memory allocation more safely.

Apart from bugs like this, it's not really urgent and it requires someone motivated to do it. Might be a good project for someone at the PyCon sprints.
msg259131 - (view) Author: Anilyka Barry (abarry) * (Python triager) Date: 2016-01-28 14:12
Yes, it's not all that urgent. And Victor's latest patch doesn't work, either :(

I wonder if there's a way to (temporarily) modify the output of ``socket.gethostname()`` to be able to test such weird corner cases.
msg259133 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-01-28 14:23
> Yes, it's not all that urgent. And Victor's latest patch doesn't work, either :(

Could you please elaborate? The patch applies cleanly? You rebuild the socket module? Which error message do you get?
msg259134 - (view) Author: Anilyka Barry (abarry) * (Python triager) Date: 2016-01-28 14:41
Oh, sorry. The patch applies without any problem, then I re-compile everything and run, and the same error happens. I re-compiled just now to make double sure.
msg259135 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-01-28 14:45
New changeset 0681f0a1fe6e by Victor Stinner in branch '3.5':
Windows: Decode hostname from ANSI code page
https://hg.python.org/cpython/rev/0681f0a1fe6e

New changeset 26f6d8cc2749 by Victor Stinner in branch 'default':
Merge 3.5: Issue #26227
https://hg.python.org/cpython/rev/26f6d8cc2749
msg259136 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-01-28 14:48
> Oh, sorry. The patch applies without any problem, then I re-compile everything and run, and the same error happens. I re-compiled just now to make double sure.

I tested my patch on Windows. I called my computer héça (3 non-ASCII letters!). Without the patch, I get the UTF-8 decoding error, as expected. With the patch, it gets the nice "héça" Unicode string, correctly decoded. I tested socket.getfqdn().

My patch will not fix all your issues at once :-) In the issue #26226, I saw at least 3 different bugs. But I'm now sure that my patch fixes a real bug, so I pushed it to Python 3.5 and default (3.6).

Thanks for the bug report Emanuel ;-)
msg259137 - (view) Author: Anilyka Barry (abarry) * (Python triager) Date: 2016-01-28 14:51
If it worked for you, I assume it's fine and I probably did something wrong on my side. Thanks for the fix!
msg259138 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-01-28 14:51
Steve:
"""
I couldn't remember the names of the alternate functions Windows provides to do the encoding for you, but yes. There are socket APIs there that do encoding and handle memory allocation more safely.

Apart from bugs like this, it's not really urgent and it requires someone motivated to do it. Might be a good project for someone at the PyCon sprints.
"""

Yeah, using the native Windows API is better, it gives access to the full Unicode character set. But it requires to spend time on the C code, and *I* am not interested to work on such project.

If you are motived, please open a new issue for that. If you are not motivated, I'm not sure that it's worth to open a bug report. Using an hostname not encodable to the ANSI code page would probably cause serious issues (not in Python, but in other applications).

When I played with filenames non-encodable to the ANSI code page, I also get errors from multiple applications, whereas Python now uses the native Windows API to access the filesystem. So sometimes Python is better than some other applications, sometimes it's as good :-)
msg259208 - (view) Author: Anilyka Barry (abarry) * (Python triager) Date: 2016-01-29 13:25
For future reference, Victor's patch does fix it, I was checking the wrong thing when testing.
msg259212 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-01-29 16:07
He he, no problem. Thanks again for the bug report. I'm surprised that
nobody reported it before.
History
Date User Action Args
2016-01-29 16:07:49vstinnersetmessages: + msg259212
2016-01-29 13:25:45abarrysetmessages: + msg259208
2016-01-28 14:51:42vstinnersetmessages: + msg259138
2016-01-28 14:51:39abarrysetmessages: + msg259137
2016-01-28 14:48:17vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg259136
2016-01-28 14:45:53python-devsetnosy: + python-dev
messages: + msg259135
2016-01-28 14:41:39abarrysetmessages: + msg259134
2016-01-28 14:23:18vstinnersetmessages: + msg259133
2016-01-28 14:12:10abarrysetmessages: + msg259131
2016-01-28 14:05:35steve.dowersetmessages: + msg259129
2016-01-28 09:42:57vstinnersetmessages: + msg259113
2016-01-28 09:41:03vstinnersetfiles: + gethostbyaddr_encoding-3.patch

messages: + msg259112
2016-01-28 09:29:11serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg259111
2016-01-28 08:49:24vstinnersetfiles: + gethostbyaddr_encoding-2.patch
2016-01-28 08:48:27vstinnersetmessages: + msg259108
2016-01-28 07:55:38eryksunsetnosy: + eryksun
messages: + msg259106
2016-01-28 05:23:51abarrysetmessages: + msg259092
2016-01-28 02:34:48steve.dowersetmessages: + msg259085
2016-01-28 00:59:13vstinnersetfiles: + gethostbyaddr_encoding.patch
keywords: + patch
2016-01-28 00:58:50vstinnercreate