classification
Title: test_undecodable_filename() in Lib/test/test_httpservers.py broken on APFS
Type: behavior Stage: resolved
Components: macOS, Tests Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: howarthjw, mattbillenstein, ned.deily, ronaldoussoren, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2017-09-07 09:58 by howarthjw, last changed 2017-12-05 05:09 by ned.deily. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 4720 merged ned.deily, 2017-12-05 04:25
PR 4721 merged python-dev, 2017-12-05 04:43
Messages (9)
msg301580 - (view) Author: Jack Howarth (howarthjw) Date: 2017-09-07 09:58
The Python 3.x test suite produces a new regression on macOS 10.13 under the new APFS filesystem when executing the test_undecodable_filename() test from Lib/test/test_httpservers.py. The error appears as...

======================================================================
ERROR: test_undecodable_filename
(test.test_httpservers.SimpleHTTPServerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/howarth/Python-3.6.2/Lib/test/support/__init__.py",
line 601, in wrapper
    return func(*args, **kw)
  File "/Users/howarth/Python-3.6.2/Lib/test/test_httpservers.py",
line 380, in test_undecodable_filename
    with open(os.path.join(self.tempdir, filename), 'wb') as f:
OSError: [Errno 92] Illegal byte sequence:
'/var/folders/7g/1x2rsy3j40n1pydq931hzlkm0000gn/T/tmpomp1r36b/@test_58317_tmp\udce7w\udcf0.txt'

----------------------------------------------------------------------
Ran 59 tests in 3.207s

See https://developer.apple.com/library/content/documentation/FileManagement/Conceptual/APFS_Guide/FAQ/FAQ.html

"""
APFS accepts only valid UTF-8 encoded filenames for creation, and preserves both case and normalization of the filename on disk in all variants..

Some differences between how APFS and HFS+ handle filenames include the following:
...
        • APFS doesn’t allow files to be created with filenames that contain unassigned codepoints in the Unicode 9.0 standard, whereas HFS+ does.
msg306006 - (view) Author: Matt Billenstein (mattbillenstein) Date: 2017-11-10 06:20
Reflected on the High Sierra buildbot now:  http://buildbot.python.org/all/#/builders/14/builds/162
msg307106 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-11-28 02:41
More background: APFS became available in macOS 10.12 although not used by default.  But with the current macOS 10.13 (High Sierra), root file systems on certain devices, such as SSDs, are automatically migrated from HFS+ to APFS so are much more likely to be seen starting in 10.13 and, in general, there is no easy way of knowing what type of FS a particular file resides on.

# TMPDIR on APFS volume
$ ./bin/python3.7 -m test test_httpservers
Run tests sequentially
0:00:00 load avg: 1.43 [1/1] test_httpservers
test test_httpservers failed -- Traceback (most recent call last):
  File ".../lib/python3.7/test/support/__init__.py", line 598, in wrapper
    return func(*args, **kw)
  File ".../lib/python3.7/test/test_httpservers.py", line 395, in test_undecodable_filename
    with open(os.path.join(self.tempdir, filename), 'wb') as f:
OSError: [Errno 92] Illegal byte sequence: '/var/folders/sn/0m4rnbyj2z1byjs68sz838300000gn/T/tmpqeg89k76/@test_64005_tmp\udce7w\udcf0.txt'

test_httpservers failed

1 test failed:
    test_httpservers

Total duration: 3 sec
Tests result: FAILURE
# TMPDIR redirected to HFS+ volume
$ TMPDIR=/Volumes/euterpe/a ./bin/python3.7 -m test test_httpservers
Run tests sequentially
0:00:00 load avg: 1.28 [1/1] test_httpservers
1 test OK.

Total duration: 3 sec
Tests result: SUCCESS

@vstinner, as the expert in this area, what do you suggest as the best way to deal with this?  In support.__init__.py?  Or test_httpservers itself?
msg307109 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-28 07:38
I suggest to catch an error and skip the test if OSError with the specific errno has been raised.
msg307114 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-28 09:54
> @vstinner, as the expert in this area, what do you suggest as the best way to deal with this?  In support.__init__.py?  Or test_httpservers itself?

It's tricky when a filename is valid or not depending on the path, depending on the filesystem. The tempfile module checks if O_TMPFILE is supported: same issue, it depends on the path, and the user can configure the path. So we check each time if O_TMPFILE works or not.
msg307116 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2017-11-28 10:25
Another option is to skip this test unconditionally on macOS when using APFS, the test is already skipped on win32 and tests a generic codepath where only testing on Linux should be fine.

BTW. It is fairly easy to detect if a path is on APFS:

```
import subprocess
  
def is_apfs(path):
    lines = subprocess.check_output(['df', path]).decode('utf-8').splitlines()
    mountpoint = lines[1].split(None, 1)[0]

    lines = subprocess.check_output(['mount']).decode('utf-8').splitlines()
    for ln in lines:
        path = ln.split(None, 1)[0]
        if path == mountpoint:
            return '(apfs' in ln

    return False
```
msg307641 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-12-05 04:42
New changeset b3edde8dd44c878e9f039a2165d00ff075157d4b by Ned Deily in branch 'master':
bpo-31380: Skip test_httpservers test_undecodable_file on macOS. (#4720)
https://github.com/python/cpython/commit/b3edde8dd44c878e9f039a2165d00ff075157d4b
msg307643 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-12-05 05:03
New changeset d9cadc5f597e5966132c9249f6c7ff0ed1eba0cb by Ned Deily (Miss Islington (bot)) in branch '3.6':
[3.6] bpo-31380: Skip test_httpservers test_undecodable_file on macOS. (GH-4720) (#4721)
https://github.com/python/cpython/commit/d9cadc5f597e5966132c9249f6c7ff0ed1eba0cb
msg307644 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-12-05 05:09
I've chosen to unconditionally skip test_undecodable_filename on macOS; it was already skipped for older macOS systems and I think it would be a mistake to test for a specific error code that could change in later releases.  As Ronald points out, testing on Linux should be sufficient.  Thanks everyone for your input.  Fixed in 3.6.4 and 3.7.0.
History
Date User Action Args
2017-12-05 05:09:03ned.deilysetstatus: open -> closed
resolution: fixed
messages: + msg307644

stage: patch review -> resolved
2017-12-05 05:03:31ned.deilysetmessages: + msg307643
2017-12-05 04:43:12python-devsetpull_requests: + pull_request4630
2017-12-05 04:42:04ned.deilysetmessages: + msg307641
2017-12-05 04:25:25ned.deilysetkeywords: + patch
stage: patch review
pull_requests: + pull_request4629
2017-11-28 10:25:36ronaldoussorensetmessages: + msg307116
2017-11-28 09:54:15vstinnersetmessages: + msg307114
2017-11-28 07:38:06serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg307109
2017-11-28 02:41:30ned.deilysetmessages: + msg307106
2017-11-17 14:51:49vstinnersetnosy: + vstinner
2017-11-17 14:51:44vstinnerlinkissue32061 superseder
2017-11-10 09:46:45serhiy.storchakasettype: crash -> behavior
components: + Tests
versions: - Python 3.3, Python 3.4, Python 3.5
2017-11-10 09:45:27serhiy.storchakalinkissue32000 superseder
2017-11-10 06:20:37mattbillensteinsetnosy: + mattbillenstein
messages: + msg306006
2017-09-07 09:58:29howarthjwcreate