classification
Title: Empty response from http.server when directory listing contains invalid unicode
Type: behavior Stage: resolved
Components: Library (Lib), macOS Versions: Python 3.5, Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: demian.brecht, durin42, hynek, jleedev, martin.panter, ned.deily, orsenthil, python-dev, r.david.murray, ronaldoussoren, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2014-08-07 15:06 by jleedev, last changed 2015-01-05 09:06 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
issue22165.patch serhiy.storchaka, 2014-08-07 18:24 review
issue22165_2.patch serhiy.storchaka, 2014-08-16 19:24 review
test_undecodable_filename.diff serhiy.storchaka, 2014-08-17 10:17 review
Messages (16)
msg225016 - (view) Author: Josh Lee (jleedev) Date: 2014-08-07 15:06
While SimpleHTTPServer from Python2 would happily spit out whatever bytes were in the directory listing, Python3's http.server logs an error and closes the connection without responding to the HTTP request.

$ mkdir $'\xff'
$ ls
\377/
$ python3 -m http.server 
Serving HTTP on 0.0.0.0 port 8000 ...
----------------------------------------
Exception happened during processing of request from ('74.125.59.145', 19648)
Traceback (most recent call last):
  File "/home/josh/local/lib/python3.5/socketserver.py", line 321, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/home/josh/local/lib/python3.5/socketserver.py", line 347, in process_request
    self.finish_request(request, client_address)
  File "/home/josh/local/lib/python3.5/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/home/josh/local/lib/python3.5/socketserver.py", line 681, in __init__
    self.handle()
  File "/home/josh/local/lib/python3.5/http/server.py", line 398, in handle
    self.handle_one_request()
  File "/home/josh/local/lib/python3.5/http/server.py", line 386, in handle_one_request
    method()
  File "/home/josh/local/lib/python3.5/http/server.py", line 677, in do_GET
    f = self.send_head()
  File "/home/josh/local/lib/python3.5/http/server.py", line 716, in send_head
    return self.list_directory(path)
  File "/home/josh/local/lib/python3.5/http/server.py", line 772, in list_directory
    % (urllib.parse.quote(linkname), html.escape(displayname)))
  File "/home/josh/local/lib/python3.5/urllib/parse.py", line 688, in quote
    string = string.encode(encoding, errors)
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcff' in position 0: surrogates not allowed
----------------------------------------
msg225030 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-08-07 16:53
It should return a server error, I think.
msg225031 - (view) Author: Augie Fackler (durin42) * Date: 2014-08-07 16:54
Why not treat the filename as opaque bytes, and let the client fetch it anyway?
msg225035 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-08-07 17:50
Because http traffic is supposed to be either latin-1 or whatever charset is specified (at least, to my understanding that is the case), so sending incorrectly encoded data seems wrong.

On the other hand, we support unix files systems not having well defined charsets, so extending this to directory listings in http isn't crazy.  That does raise the question, though, of passing the bytes through python3's string model without breaking anything, so some careful thought may be required.  I haven't looked at the details, though, so it might well be pretty simple.
msg225036 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-08-07 18:24
Here is a patch which fixes handling of undecodable paths in SimpleHTTPRequestHandler.
msg225407 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2014-08-16 17:49
Attached patch looks good to me. If an unittest can be provided for this situation in test_httpservers.py it will be comprehensive and will be good to go.
msg225409 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-08-16 19:24
Here is a patch with a test.
msg225426 - (view) Author: Roundup Robot (python-dev) Date: 2014-08-17 05:25
New changeset f180a9156cc8 by Serhiy Storchaka in branch '3.4':
Issue #22165: SimpleHTTPRequestHandler now supports undecodable file names.
http://hg.python.org/cpython/rev/f180a9156cc8

New changeset 3153a400b739 by Serhiy Storchaka in branch 'default':
Issue #22165: SimpleHTTPRequestHandler now supports undecodable file names.
http://hg.python.org/cpython/rev/3153a400b739
msg225427 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-08-17 05:31
Thank you for the review Senthil.
msg225428 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2014-08-17 07:59
Looks like we hit with an encoding issue, which is due to way os.fsdecode() and os.listdir() decode the filenames.


>>> support.TESTFN_UNDECODABLE
b'@test_99678_tmp\xe7w\xf0'
>>> dir_list = os.listdir(self.tempdir)
>>> dir_list
['@test_99678_tmp%E7w%F0.txt', 'test']
>>> filename = os.fsdecode(support.TESTFN_UNDECODABLE) + '.txt'
>>> filename
'@test_99678_tmp\udce7w\udcf0.txt'


======================================================================
FAIL: test_undecodable_filename (test.test_httpservers.SimpleHTTPServerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.4.murray-snowleopard/build/Lib/test/test_httpservers.py", line 282, in test_undecodable_filename
    .encode('utf-8', 'surrogateescape'), body)
AssertionError: b'href="%40test_62069_tmp%ED%B3%A7w%ED%B3%B0.txt"' not found in b'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">\n<html>\n<head>\n<meta http-equiv="Content-Type" content="text/html; charset=utf-8">\n<title>Directory listing for tmp0asrs9ei/</title>\n</head>\n<body>\n<h1>Directory listing for tmp0asrs9ei/</h1>\n<hr>\n<ul>\n<li><a href="%40test_62069_tmp%25E7w%25F0.txt">@test_62069_tmp%E7w%F0.txt</a></li>\n<li><a href="test">test</a></li>\n</ul>\n<hr>\n</body>\n</html>\n'
msg225430 - (view) Author: Roundup Robot (python-dev) Date: 2014-08-17 09:22
New changeset a894b629bbea by Serhiy Storchaka in branch '3.4':
Issue #22165: Fixed test_undecodable_filename on non-UTF-8 locales.
http://hg.python.org/cpython/rev/a894b629bbea

New changeset 7cdc941d5180 by Serhiy Storchaka in branch 'default':
Issue #22165: Fixed test_undecodable_filename on non-UTF-8 locales.
http://hg.python.org/cpython/rev/7cdc941d5180
msg225431 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-08-17 10:17
Oh, I missed that os.listdir() on Mac returns really strange result. Thank you Senthil.

Here is a patch which try to workaround this. I'm not sure that it is enough. May be we should fix os.listdir(). Or conclude that this issue can't be fixed on Mac OS.
msg225433 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2014-08-17 10:51
OSX returns a strange value in os.listdir because the HFS+ filesystem itself has unicode filenames and transforms byte strings that are assumed to contain UTF-8 into something the filesystem can handle (and seems to replace bytes that aren't valid UTF-8 into a percent-encoded value).
msg225435 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-08-17 11:58
Well, then the workaround should work.
msg225444 - (view) Author: Roundup Robot (python-dev) Date: 2014-08-17 13:59
New changeset b05d4f3ee190 by Serhiy Storchaka in branch '3.4':
Issue #22165: Fixed test_undecodable_filename on Mac OS.
http://hg.python.org/cpython/rev/b05d4f3ee190

New changeset 58e0d2c3ead8 by Serhiy Storchaka in branch 'default':
Issue #22165: Fixed test_undecodable_filename on Mac OS.
http://hg.python.org/cpython/rev/58e0d2c3ead8
msg233449 - (view) Author: Roundup Robot (python-dev) Date: 2015-01-05 09:06
New changeset 1bc41bbbe02d by Ned Deily in branch '3.4':
Issue #22165: Skip test_undecodable_filename on OS X prior to 10.5.
https://hg.python.org/cpython/rev/1bc41bbbe02d

New changeset 85258e08b69b by Ned Deily in branch 'default':
Issue #22165: merge from 3.4
https://hg.python.org/cpython/rev/85258e08b69b
History
Date User Action Args
2015-01-05 09:06:09python-devsetmessages: + msg233449
2014-08-19 03:19:15orsenthilsetresolution: fixed
stage: resolved
2014-08-19 03:19:01orsenthilsetstatus: open -> closed
2014-08-17 13:59:13python-devsetmessages: + msg225444
2014-08-17 11:58:25serhiy.storchakasetmessages: + msg225435
2014-08-17 10:51:55ronaldoussorensetmessages: + msg225433
2014-08-17 10:17:45serhiy.storchakasetfiles: + test_undecodable_filename.diff

components: + macOS

nosy: + ronaldoussoren, vstinner, ned.deily, hynek
messages: + msg225431
resolution: fixed -> (no value)
stage: resolved -> (no value)
2014-08-17 09:22:49python-devsetmessages: + msg225430
2014-08-17 08:00:18orsenthilsetstatus: closed -> open
2014-08-17 07:59:13orsenthilsetmessages: + msg225428
2014-08-17 05:31:53serhiy.storchakasetstatus: open -> closed
resolution: fixed
messages: + msg225427

stage: patch review -> resolved
2014-08-17 05:25:54python-devsetnosy: + python-dev
messages: + msg225426
2014-08-16 19:24:52serhiy.storchakasetfiles: + issue22165_2.patch
2014-08-16 19:24:20serhiy.storchakasetassignee: serhiy.storchaka
messages: + msg225409
2014-08-16 17:49:01orsenthilsetnosy: + orsenthil
messages: + msg225407
2014-08-09 01:35:31martin.pantersetnosy: + martin.panter
2014-08-07 19:34:43demian.brechtsetnosy: + demian.brecht
2014-08-07 18:24:50serhiy.storchakasetfiles: + issue22165.patch

nosy: + serhiy.storchaka
messages: + msg225036

keywords: + patch
stage: needs patch -> patch review
2014-08-07 17:50:24r.david.murraysetmessages: + msg225035
2014-08-07 16:54:29durin42setmessages: + msg225031
2014-08-07 16:53:17r.david.murraysetversions: + Python 3.4
nosy: + r.david.murray

messages: + msg225030

type: behavior
stage: needs patch
2014-08-07 15:06:33jleedevcreate