This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: "python -m pydoc -w" fails in nondecodable directory
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, martin.panter, orsenthil, pitrou, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2015-09-19 21:07 by serhiy.storchaka, last changed 2022-04-11 14:58 by admin.

Files
File name Uploaded Description Edit
pydoc_undecodabple_path.patch serhiy.storchaka, 2015-09-21 13:37 review
pydoc_undecodabple_path_2.patch serhiy.storchaka, 2015-09-27 19:56 Using pathlib review
pydoc_iri.patch martin.panter, 2015-10-01 00:49 IRI file: link review
Messages (9)
msg251117 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-09-19 21:07
$ pwd
/home/serhiy/py/cpy�thon-3.5
$ ./python -m pydoc -w pydoc
Traceback (most recent call last):
  File "/home/serhiy/py/cpy\udcffthon-3.5/Lib/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/serhiy/py/cpy\udcffthon-3.5/Lib/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/serhiy/py/cpy\udcffthon-3.5/Lib/pydoc.py", line 2648, in <module>
    cli()
  File "/home/serhiy/py/cpy\udcffthon-3.5/Lib/pydoc.py", line 2611, in cli
    writedoc(arg)
  File "/home/serhiy/py/cpy\udcffthon-3.5/Lib/pydoc.py", line 1642, in writedoc
    page = html.page(describe(object), html.document(object, name))
  File "/home/serhiy/py/cpy\udcffthon-3.5/Lib/pydoc.py", line 370, in document
    if inspect.ismodule(object): return self.docmodule(*args)
  File "/home/serhiy/py/cpy\udcffthon-3.5/Lib/pydoc.py", line 651, in docmodule
    url = urllib.parse.quote(path)
  File "/home/serhiy/py/cpy\udcffthon-3.5/Lib/urllib/parse.py", line 706, in quote
    string = string.encode(encoding, errors)
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcff' in position 19: surrogates not allowed
msg251212 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-09-21 07:50
Seems to be caused by the Python directory being non-decodable; the current working directory does not matter. What is going on is “pydoc” is trying to make a link to a module’s source code, such as

<a href="file:/usr/lib/python3.4/pydoc.py">/usr/lib/python3.4/pydoc.py</a>

For non-decodable paths, the following would work in Firefox:

<a href="file:/home/serhiy/py/cpy%FFthon-3.5/Lib/pydoc.py">/home/serhiy/py/cpy�thon-3.5/Lib/pydoc.py</a>

but since URL percent encoding already uses UTF-8, this scheme isn’t foolproof (e.g. a UTF-8 sequence when the locale is ASCII would be ambiguous). A simpler and more consistent way forward would be an error handler substituting something like this, decoding the surrogate escape code with the “replace” handler, with the HTML link suppressed:

/home/serhiy/py/cpy�thon-3.5/Lib/pydoc.py (invalid filename encoding)
msg251213 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-21 08:00
Technically, I think that it's possible to put bytes in an URL using %HH format. I didn't check if we can retrieve the "raw bytes".
msg251226 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-09-21 13:37
We could use url = urllib.parse.quote_from_bytes(os.fsencode(path)) on Posix systems, but I heart that on Windows os.fsencode() can irreversible spoil file names (replace unencodable characters with '?'). On other side, I'm not sure that Windows unicode path can't contain lone surrogates. In this case we should use the 'surrogatepass' error handler (at least it allow to restore the path in principle).

Here is a patch that tries to handle undecodable and unencodable paths. Need to test it on Windows.
msg251271 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-09-21 22:31
Serhiy’s patch essentially uses the local filesystem encoding and then percent encoding, rather than the current behaviour of strict UTF-8 encoding and percent encoding. This is similar to what the “pathlib” make_uri() methods do, so maybe we could let “pathlib” do the work instead.

This draft RFC discusses encoding “file:” URLs:

https://tools.ietf.org/html/draft-ietf-appsawg-file-scheme-03#section-4

It suggests leaving Unicode characters alone (in IRIs) if possible, or using UTF-8 and percent encoding even if the filesystem uses a non-UTF-8 encoding. Perhaps we could leave the filename in the HTML as Unicode characters without percent encoding, and only percent encode the undecodable (surrogate-escaped) bytes.

This “IRI” scheme is also recommended by <http://blogs.msdn.com/b/ie/archive/2006/12/06/file-uris-in-windows.aspx>, which says on Windows, “in file URIs, percent-encoded octets are interpreted as a byte in the user’s current codepage”. This contradicts the draft RFC and the “pathlib” implementation, which both use UTF-8.
msg251721 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-09-27 19:56
Yes, perhaps using pathlib is more correct. Updated patch uses pathlib. Can you please test it on Windows?

Can filenames on Windows contain lone surrogates?
msg251731 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-09-27 22:26
I don’t have much to do with Windows, but I understand we don’t support surrogate-escaped bytes there. E.g. see <https://docs.python.org/dev/library/os.html#os.fsdecode> and sys.getfilesystemencoding(). However I suspect your first patch would have failed on Windows doing os.fsencode(TESTFN_UNENCODABLE); apparently it cannot represent all possible file names in bytes. The second patch doesn’t call fsencode() so this shouldn’t be a problem.

Your tests do not test that the output is valid Unicode without surrogates. With your first patch applied, when pydoc wrote the HTML to a UTF-8 disk file, I got the error:

  File "/media/disk/home/proj/python/cpy\udcffthon/Lib/pydoc.py", line 2626, in cli
    writedoc(arg)
  File "/media/disk/home/proj/python/cpy\udcffthon/Lib/pydoc.py", line 1659, in writedoc
    file.write(page)
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcff' in position 674: surrogates not allowed

I have been working on an alternative patch using my IRI (Unicode URLs) proposal for “file:” links, and “surrogatepass” for HTTP links. But I am also trying to fix some related problems with the built-in HTTP server, and the unit tests are a bit tricky.
msg251985 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-10-01 00:49
Here is a patch that implements my IRI (Unicode URL) proposal. Differences compared to Serhiy’s patches:

* “file:” URLs use Unicode if possible. Percent encode encoding only used for reserved ASCII characters and undecodable bytes.

* HTTP URLs use UTF-8 and retain surrogate escaping. This means that the links generated by “getobj” pages to “getfile” pages work with troublesome paths.

* Displayed file names are get an ASCII question mark thanks to the “replace” error handler. This means that the generated HTML is now encodable to UTF-8. Serhiy’s path would be displayed something like:

/home/serhiy/py/cpy?thon-3.5/Lib/pydoc.py (invalid filename encoding)

* Added some missing html.escape() calls.
msg332221 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-12-20 12:28
Could you please create a PR for your path Martin?
History
Date User Action Args
2022-04-11 14:58:21adminsetgithub: 69371
2018-12-20 12:28:32serhiy.storchakasetversions: + Python 3.7, Python 3.8, - Python 3.4, Python 3.5, Python 3.6
2018-12-20 12:28:17serhiy.storchakasetmessages: + msg332221
2015-10-01 00:49:41martin.pantersetfiles: + pydoc_iri.patch

messages: + msg251985
2015-09-27 22:26:29martin.pantersetmessages: + msg251731
2015-09-27 19:57:17serhiy.storchakasetnosy: + pitrou
2015-09-27 19:56:00serhiy.storchakasetfiles: + pydoc_undecodabple_path_2.patch

messages: + msg251721
2015-09-21 22:31:22martin.pantersetmessages: + msg251271
2015-09-21 13:37:36serhiy.storchakasetfiles: + pydoc_undecodabple_path.patch
keywords: + patch
messages: + msg251226

stage: patch review
2015-09-21 12:03:14Arfreversetnosy: + Arfrever
2015-09-21 08:00:28vstinnersetnosy: + vstinner
messages: + msg251213
2015-09-21 07:50:32martin.pantersetnosy: + martin.panter
messages: + msg251212
2015-09-19 21:13:23serhiy.storchakalinkissue25181 dependencies
2015-09-19 21:07:25serhiy.storchakacreate