classification
Title: Regression: os.walk now using os.scandir() breaks bytes filenames on windows
Type: behavior Stage: needs patch
Components: Library (Lib), Windows Versions: Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: SilentGhost, benhoyt, db3l, eryksun, ideasman42, mont29, paul.moore, python-dev, serhiy.storchaka, steve.dower, tim.golden, vstinner, zach.ware
Priority: high Keywords: 3.5regression, patch

Created on 2015-12-19 15:39 by mont29, last changed 2017-06-28 17:13 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
walk_bytes.patch serhiy.storchaka, 2015-12-23 09:33 review
walk_bytes_2.patch serhiy.storchaka, 2016-01-13 08:34 review
test_walk_bytes_deprecate.patch serhiy.storchaka, 2016-02-11 13:17 review
Pull Requests
URL Status Linked Edit
PR 2475 vstinner, 2017-06-28 17:13
Messages (37)
msg256731 - (view) Author: Bastien Montagne (mont29) Date: 2015-12-19 15:39
In py3.4 and below we used to be able to use bytes filenames with os.walk(), it’s now impossible under windows due to the limitation of os.scandir().

This issue was reported to Blender tracker (https://developer.blender.org/T47018).

I assume this should be considered as a regression? At least, this should be mentioned in the documentation…

Thanks,
Bastien
msg256732 - (view) Author: SilentGhost (SilentGhost) * (Python triager) Date: 2015-12-19 15:57
Perhaps, I'm looking in the wrong place, but there doesn't seem be any test for bytes filenames.
msg256755 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-12-20 09:13
Regressions are not cool, but it was a deliberate choice to not support
bytes filenames in os.scandir(). If you use bytes, you can get filenames
which are invalid: you will be unable to use the filename with open() to
read its content for example.

Bytes filename are deprecated since python 3.2 if i recall correctly. It's
time to use the right time, it's also simpler to use on Python 3.

I suggest to document the regression rather than adding bytes support to
os.scandir or don't use scandir in os.walk().
msg256757 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-12-20 09:45
For softer transition we can return old os.walk() implementation for bytes (may be with a deprecation warning). Using os.scandir() is just an optimization, and this shouldn't break existing program.
msg256785 - (view) Author: Campbell Barton (ideasman42) Date: 2015-12-21 07:37
@haypo, I checked available info online and couldn't find any reference to byte file-paths were deprecated since Python3.2.

On the contrary, the 3.2 release notes [0] state:

 "countless fixes regarding bytes/string issues; among them full support for a bytes environment (filenames, environment variables)"

----
Also docs for open 3.2 [1] and 3.5 [2] say that byte filenames are supported with no mention of deprecation.

Since this is already working properly in 3.5 for other systems besides ms-windows, and worked in 3.4x.
Dropping support on a single platform seems a rather problematic regression.

----

[0]: https://www.python.org/download/releases/3.2/
[1]: https://docs.python.org/3.2/library/functions.html#open
[2]: https://docs.python.org/3.5/library/functions.html#open
msg256787 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2015-12-21 09:31
The ANSI API is problematic because it returns a best-fit encoding to the system codepage. For example:

    >>> os.listdir('.')
    ['ƠƨưƸǀLjǐǘǠǨǰǸ']

    >>> os.listdir(b'.')
    [b'O?u?|?iu?Kj?']

To somewhat work around this problem, listdir and scandir could return the cAlternateFilename of the WIN32_FIND_DATA struct if present. This is the classic 8.3 short name that Microsoft file systems create for MS-DOS compatibility. For NTFS it can be disabled in the registry, or per volume, but I assume whoever does that knows what to expect. 

Also, since Python wouldn't need the short name for a wide-character path, there's no point in asking for it. (For NTFS it's a separate entry in the MFT. If it exists, which is known ahead of time, finding the entry requires a second lookup.) In this case it's better to call FindFirstFileExW and request only FindExInfoBasic. Generally the difference is inconsequential, but in a contrived example with 10000 similarly-named files from "ĀāĂă0000" to "ĀāĂă9999" and short names from "0000~1" to "9999~1", skipping the short name lookup shaved about 10% off the total time. For this test, I replaced the FindFirstFileW call in posix_scandir with the following call:

    iterator->handle = FindFirstFileExW(path_strW,
                                        FindExInfoBasic,
                                        &iterator->file_data,
                                        FindExSearchNameMatch,
                                        NULL, 0);
msg256804 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2015-12-21 17:20
I like Serhiy's suggestion to use the old walk implementation on Windows if the argument is bytes, with a DeprecationWarning (like all other bytes operations in os on Windows).  There was no warning that bytes paths would stop working with os.walk, so we should fix it and I'm bumping up the priority.

bytes filenames have been deprecated since Python 3.3 (#13374); I don't think scandir (new in 3.5) should support bytes.
msg256812 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-12-21 21:55
Python 3.5.0 and 3.5.1 are out. I don't think that it's worth to add back
bytes support is os.walk() in 3.5.2. I would prefer to do the opposite and
really drop bytes support for all filenames in the whole stdlib in 3.6.

It's really a bad idea to use bytes on Windows. With Python 3, it becomes
possible and simple to handle filenames as Unicode on all platforms and in
all cases (see PEP 393 for undecodable filenames).
msg256820 - (view) Author: Ben Hoyt (benhoyt) * Date: 2015-12-22 02:31
Just to clarify what Victor and Zachary mentioned: bytes filenames have been deprecated only *on Windows* since Python 3.3; bytes filenames are still fine on POSIX.
msg256904 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-12-23 09:33
Here is a patch that restores the ability of os.walk() to work with bytes filenames on Windows.
msg256908 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-12-23 11:35
Did we want a deprecation warning too? I don't see it in the patch.

Otherwise, LGTM.
msg256909 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2015-12-23 13:14
Considering there's no plan to implement bytes paths for scandir on Windows, then the following line in the os docs needs to be modified: "All functions accepting path or file names accept both bytes and string objects, and result in an object of the same type, if a path or file name is returned." It should be changed to acknowledge that some functions on Windows do not support bytes paths, and that existing support is deprecated and may be removed in a future release.

Also, the docs for listdir should warn that using bytes returns invalid results on Windows when filenames contain characters that aren't mapped in the system locale codepage.
msg256911 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-12-23 13:29
There is open issue16700 for the documentation.
msg258133 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-01-13 08:34
Fixed a typo found by Eryk Sun.
msg258134 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-01-13 08:39
> Did we want a deprecation warning too?

I didn't tested on Windows. If bytes paths considered deprecated on Windows, os.listdir() should emit a warning. Otherwise I don't think we should special case os.walk().
msg258378 - (view) Author: Campbell Barton (ideasman42) Date: 2016-01-16 10:13
A related question, (realize this isn't a support forum), but what is the equivalent API calls in Python3.5 so we can update scripts.

Should we be doing: os.walk(os.fsdecode(path)) ?

From Python's documentation, os.fsdecode is using 'strict' handler which may raise exceptions.
I'm mainly concerned that changes here will introduce bugs where bytes previously resolved the paths (albeit working in an imperfect way).

If this is not the best place to ask about this, then can post elsewhere.
msg259846 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-02-08 14:24
New changeset 5310f94772f4 by Serhiy Storchaka in branch '3.5':
Issue #25911: Restored support of bytes paths in os.walk() on Windows.
https://hg.python.org/cpython/rev/5310f94772f4

New changeset b060af2a58b6 by Serhiy Storchaka in branch 'default':
Issue #25911: Restored support of bytes paths in os.walk() on Windows.
https://hg.python.org/cpython/rev/b060af2a58b6
msg259847 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-02-08 14:33
> New changeset 5310f94772f4 by Serhiy Storchaka in branch '3.5':
> Issue #25911: Restored support of bytes paths in os.walk() on Windows.
> https://hg.python.org/cpython/rev/5310f94772f4
> 
> New changeset b060af2a58b6 by Serhiy Storchaka in branch 'default':
> Issue #25911: Restored support of bytes paths in os.walk() on Windows.
> https://hg.python.org/cpython/rev/b060af2a58b6

While this change is ok for Python 3.5, I would prefer to *drop* support for bytes filenames on Windows. I started a thread on python-dev for that:
https://mail.python.org/pipermail/python-dev/2016-February/143150.html
msg259848 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-02-08 14:34
Bastien:
> In py3.4 and below we used to be able to use bytes filenames with os.walk(), it’s now impossible under windows due to the limitation of os.scandir().
>
> This issue was reported to Blender tracker (https://developer.blender.org/T47018).

Again, why do you use bytes? Blender only supports Python 3. You must use Unicode on all platforms. On Windows, it gives you support of the full Unicode character set for free.
msg260103 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-02-11 13:17
New tests emit deprecation warnings on Windows and failed.

http://buildbot.python.org/all/builders/AMD64%20Windows8%203.x/builds/1753/steps/test/logs/stdio
test_rmdir_on_directory_link_to_missing_target (test.test_os.Win32SymlinkTests) ... D:\buildarea\3.x.bolen-windows8\build\lib\os.py:447: DeprecationWarning: The Windows bytes API has been deprecated, use Unicode filenames instead
  for name in listdir(dir):
D:\buildarea\3.x.bolen-windows8\build\lib\os.py:441: DeprecationWarning: The Windows bytes API has been deprecated, use Unicode filenames instead
  return path.isdir(self.path)
D:\buildarea\3.x.bolen-windows8\build\lib\ntpath.py:249: DeprecationWarning: The Windows bytes API has been deprecated, use Unicode filenames instead
  st = os.lstat(path)
D:\buildarea\3.x.bolen-windows8\build\lib\test\test_os.py:2688: DeprecationWarning: The Windows bytes API has been deprecated, use Unicode filenames instead
  func(name, *func_args)
D:\buildarea\3.x.bolen-windows8\build\lib\unittest\case.py:176: DeprecationWarning: The Windows bytes API has been deprecated, use Unicode filenames instead
  callable_obj(*args, **kwargs)
D:\buildarea\3.x.bolen-windows8\build\lib\test\test_os.py:1881: DeprecationWarning: The Windows bytes API has been deprecated, use Unicode filenames instead
  sorted(os.listdir(path)),
D:\buildarea\3.x.bolen-windows8\build\lib\test\test_os.py:1867: DeprecationWarning: The Windows bytes API has been deprecated, use Unicode filenames instead
  sorted(os.listdir(os.fsencode(support.TESTFN))),
test test_os failed
skipped 'currently fails; consider for improvement'

======================================================================
FAIL: test_walk_bottom_up (test.test_os.BytesWalkTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\buildarea\3.x.bolen-windows8\build\lib\test\test_os.py", line 894, in test_walk_bottom_up
    self.sub2_tree)
AssertionError: Tuples differ: ('@test_4312_tmp\\TEST1\\SUB2', ['broken_link', 'link'], ['tmp3']) != ('@test_4312_tmp\\TEST1\\SUB2', ['link'], ['broken_link', 'tmp3'])

First differing element 1:
['broken_link', 'link']
['link']

- ('@test_4312_tmp\\TEST1\\SUB2', ['broken_link', 'link'], ['tmp3'])
?                                                 ----------

+ ('@test_4312_tmp\\TEST1\\SUB2', ['link'], ['broken_link', 'tmp3'])
?                                 ++++++++++


======================================================================
FAIL: test_walk_prune (test.test_os.BytesWalkTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\buildarea\3.x.bolen-windows8\build\lib\test\test_os.py", line 874, in test_walk_prune
    self.assertEqual(all[1], self.sub2_tree)
AssertionError: Tuples differ: ('@test_4312_tmp\\TEST1\\SUB2', ['broken_link', 'link'], ['tmp3']) != ('@test_4312_tmp\\TEST1\\SUB2', ['link'], ['broken_link', 'tmp3'])

First differing element 1:
['broken_link', 'link']
['link']

- ('@test_4312_tmp\\TEST1\\SUB2', ['broken_link', 'link'], ['tmp3'])
?                                                 ----------

+ ('@test_4312_tmp\\TEST1\\SUB2', ['link'], ['broken_link', 'tmp3'])
?                                 ++++++++++

Proposed patch should silence warnings. Could anyone please test it on Windows?
msg260787 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-02-24 11:05
New changeset 9bffe39e8273 by Serhiy Storchaka in branch '3.5':
Fixed a bug in os.walk() with bytes path on Windows caused by merging fixes
https://hg.python.org/cpython/rev/9bffe39e8273
msg261360 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-03-08 14:13
New changeset 19a3e0e664af by Serhiy Storchaka in branch '3.4':
Issues #23808, #25911: Trying to fix walk tests on Windows.
https://hg.python.org/cpython/rev/19a3e0e664af

New changeset f9e22717722d by Serhiy Storchaka in branch '3.5':
Issues #23808, #25911: Trying to fix walk tests on Windows.
https://hg.python.org/cpython/rev/f9e22717722d

New changeset da020e408c7f by Serhiy Storchaka in branch 'default':
Issues #23808, #25911: Trying to fix walk tests on Windows.
https://hg.python.org/cpython/rev/da020e408c7f
msg261370 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-03-08 19:32
New changeset 757159fb4847 by Serhiy Storchaka in branch '3.5':
Issue #25911: Tring to silence deprecation warnings in bytes path walk tests.
https://hg.python.org/cpython/rev/757159fb4847

New changeset ecf4e51c222f by Serhiy Storchaka in branch 'default':
Issue #25911: Tring to silence deprecation warnings in bytes path walk tests.
https://hg.python.org/cpython/rev/ecf4e51c222f
msg261735 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-03-14 09:06
Deprecation warnings are suppressed, but my attempt to fix test failures failed.

http://buildbot.python.org/all/builders/AMD64%20Windows8%203.x/builds/1832/steps/test/logs/stdio
======================================================================
FAIL: test_walk_bottom_up (test.test_os.BytesWalkTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\buildarea\3.x.bolen-windows8\build\lib\test\test_os.py", line 894, in test_walk_bottom_up
    self.sub2_tree)
AssertionError: Tuples differ: ('@test_4060_tmp\\TEST1\\SUB2', ['broken_link', 'link'], ['tmp3']) != ('@test_4060_tmp\\TEST1\\SUB2', ['link'], ['broken_link', 'tmp3'])

First differing element 1:
['broken_link', 'link']
['link']

- ('@test_4060_tmp\\TEST1\\SUB2', ['broken_link', 'link'], ['tmp3'])
?                                                 ----------

+ ('@test_4060_tmp\\TEST1\\SUB2', ['link'], ['broken_link', 'tmp3'])
?                                 ++++++++++


======================================================================
FAIL: test_walk_prune (test.test_os.BytesWalkTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\buildarea\3.x.bolen-windows8\build\lib\test\test_os.py", line 874, in test_walk_prune
    self.assertEqual(all[1], self.sub2_tree)
AssertionError: Tuples differ: ('@test_4060_tmp\\TEST1\\SUB2', ['broken_link', 'link'], ['tmp3']) != ('@test_4060_tmp\\TEST1\\SUB2', ['link'], ['broken_link', 'tmp3'])

First differing element 1:
['broken_link', 'link']
['link']

- ('@test_4060_tmp\\TEST1\\SUB2', ['broken_link', 'link'], ['tmp3'])
?                                                 ----------

+ ('@test_4060_tmp\\TEST1\\SUB2', ['link'], ['broken_link', 'tmp3'])
?                                 ++++++++++


----------------------------------------------------------------------

In some cases a broken link is considered as a link to regular file, but in other cases  is considered as a link to directory.

It is hard to fix this Windows issue without Windows. Needed a help of Windows experts.
msg262174 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-03-22 10:56
Windows buildbots still fail (sometimes?) because of this issue :-/

http://buildbot.python.org/all/builders/AMD64%20Windows8%203.x/builds/1874/steps/test/logs/stdio
msg262330 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-03-24 08:21
New changeset d54ee39b061f by Victor Stinner in branch 'default':
Fix DeprecationWarning on Windows
https://hg.python.org/cpython/rev/d54ee39b061f
msg262332 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-03-24 11:25
New changeset eb91d0387d59 by Victor Stinner in branch 'default':
Enhance os._DummyDirEntry
https://hg.python.org/cpython/rev/eb91d0387d59
msg262333 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-03-24 11:31
I faile to reproduce the bug by running test_os alone on Windows 8.

I wrote a script to try to isolate a sequential list of tests which reproduce the bug on Windows, but I failed to find such list after 2 hours:
https://bitbucket.org/haypo/misc/src/3e2e6138798c6cec061544ff116e50a8d1d6d2b8/python/isolate.py?fileviewer=file-view-default

So I'm unable to reproduce the bug, I tried a change to enhance os._DummyDirEntry, it should now mimick better the C implementation of os.DirEntry. Let's see if the test still fails randomly...
msg262474 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-03-26 00:12
Update: test_os pass on all Windows buildbots except one!

I don't understand why the test fails on this buildbot.

http://buildbot.python.org/all/builders/AMD64%20Windows8%203.x/builds/1906/steps/test/logs/stdio

======================================================================
FAIL: test_walk_bottom_up (test.test_os.BytesWalkTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\buildarea\3.x.bolen-windows8\build\lib\test\test_os.py", line 895, in test_walk_bottom_up
    self.assertEqual(len(all), 4)
AssertionError: 5 != 4
msg262475 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-03-26 00:13
New changeset 82da02b5bf22 by Victor Stinner in branch 'default':
Issue #25911: more info on test_os failure
https://hg.python.org/cpython/rev/82da02b5bf22
msg262485 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-03-26 06:14
Some tests were failed on Windows 8 even in 3.4 (issue23808).
msg262594 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-03-29 09:29
New changeset 2b25fa7e3b7a by Victor Stinner in branch 'default':
Fix os._DummyDirEntry.is_symlink()
https://hg.python.org/cpython/rev/2b25fa7e3b7a
msg262599 - (view) Author: David Bolen (db3l) Date: 2016-03-29 10:36
I'm including some comments here from an email thread I had with Victor about this issue on the Win8 buildbot, which led to his recent changeset 2b25fa7e3b7a.

The Win8 3.x failure (the 5 != 4 length error) was due to the revision of os._DummyDirEntry (introduced I believe in eb91d0387d59) using stat() rather than lstat() to check for a symbolic link, which couldn't work, which in turn failed to exclude the symbolic directory link from the walk in the test.

The 3.5 branch failure about the mis-matched tuple values from the walk is a separate issue, where the broken directory link is still counted as a directory (rather than a file).  It seems due to os.path.isdir() - in the older os._DummyDirEntry implementation - returning True for the broken link in the test, rather than False as on Unix.  That's because on Unix the underlying os.stat call fails, but on Windows isdir is replaced with nt._isdir which avoids os.stat but is just a shim for.  The success on Windows sort of makes sense since the target_is_directory parameter to os.symlink() was used to create the broken link in the test, so Windows knows it's a directory, even if the link target is missing.

If the bytes walk implementation needs to treat broken links like files on Windows (matching the non-bytes version), then it can't depend on isdir() failing.  The non-bytes version (using scandir()) works even on Windows since internally scandir appears to build up mode bits out of os.stat/os.lstat calls and never uses isdir.

Back-porting the 3.x implementation of DummyDirEntry would be one quick way to fix the 3.5 issue as it also avoids isdir().

BTW, with respect to changeset 2b25fa7e3b7a, I'm not sure it has quite the right semantics for is_dir, at least not if it's supposed to parallel os.path.isdir().  I believe that should return True for a symbolic link to a directory, which it doesn't look like this change would, if is_symlink happened to have been called first.  It's possible the semantics of how _DummyDirEntry is used precludes that scenario, but it seems a little fragile.  I'd probably just use lstat() for is_symlink, but otherwise not touch is_dir.

Finally, as to why this only showed up on the Win8 buildbot - it turns out that's the only machine where the tests could create a symbolic link and thus encounter the scenario - they were prevented on Win7 and Win10 under the normal buildbot user due to lack of privileges (which was a surprise to me).  That's also what was happening on Victor's local Win8 VM.  So the failing conditions were just being skipped.  The Win8 buildbot was itself sort of pure luck, as early on I must have set it up to run under an elevated (administrative) command prompt, probably due to trying to handle some other issue.  I've now done the same thing for Win10 so it began seeing the same failures in the last few tests.
msg262605 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-03-29 11:47
New changeset c38ac7ab8d9a by Victor Stinner in branch '3.5':
Issue #25911: Backport os._DummyDirEntry fixes
https://hg.python.org/cpython/rev/c38ac7ab8d9a
msg262606 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-03-29 11:58
> BTW, with respect to changeset 2b25fa7e3b7a, I'm not sure it has quite the right semantics for is_dir, at least not if it's supposed to parallel os.path.isdir().  I believe that should return True for a symbolic link to a directory, which it doesn't look like this change would, if is_symlink happened to have been called first.  It's possible the semantics of how _DummyDirEntry is used precludes that scenario, but it seems a little fragile.  I'd probably just use lstat() for is_symlink, but otherwise not touch is_dir.

I think that you misunderstood the code. The "use cache lstat" path is only taken if the file is *not* symbolic link.

I tested manually _DummyDirEntry on a symbolic link to a directory: _lstat is filled by the constructor, is_dir() *and* is_symlink() returns True. Both are not exclusive, is_dir() works as follow_symlinks=True, whereas is_symlink() works as follow_symlinks=False.
msg262607 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-03-29 12:02
Since all test_os now pass again on 3.x buildbots (after a few months of red buildbots), I backported the _DummyDirEntry fixes to 3.5.

David:
> The 3.5 branch failure about the mis-matched tuple values from the walk is a separate issue, (...)

I'm sorry, I don't understand you. The test_os failure was the same on Python 3.5 and 3.6:


- ('@test_4312_tmp\\TEST1\\SUB2', ['broken_link', 'link'], ['tmp3'])
?                                                 ----------
+ ('@test_4312_tmp\\TEST1\\SUB2', ['link'], ['broken_link', 'tmp3'])
?                                 ++++++++++

This error should now be fixed on Python 3.5 and 3.6.

Is there a remaining open issue related to os.scandir() (bytes or Unicode)?
msg262632 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-03-29 21:57
The issue looks to be fixed in Python 3.5 (future version 3.5.2) and 3.6, I close the issue.

I repeat myself: On Python 3, don't store filenames as bytes, it's a bad practice. Use unicode, it works on all platforms and gives access to the full Unicode Character Set!
History
Date User Action Args
2017-06-28 17:13:22vstinnersetpull_requests: + pull_request2536
2016-03-29 21:57:59vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg262632
2016-03-29 12:02:49vstinnersetmessages: + msg262607
2016-03-29 11:58:44vstinnersetmessages: + msg262606
2016-03-29 11:47:29python-devsetmessages: + msg262605
2016-03-29 10:36:34db3lsetnosy: + db3l
messages: + msg262599
2016-03-29 09:29:25python-devsetmessages: + msg262594
2016-03-26 06:14:05serhiy.storchakasetmessages: + msg262485
2016-03-26 00:13:02python-devsetmessages: + msg262475
2016-03-26 00:12:48vstinnersetmessages: + msg262474
2016-03-24 11:31:41vstinnersetmessages: + msg262333
2016-03-24 11:25:57python-devsetmessages: + msg262332
2016-03-24 08:21:48python-devsetmessages: + msg262330
2016-03-22 10:56:00vstinnersetmessages: + msg262174
2016-03-14 09:07:23serhiy.storchakasetstage: patch review -> needs patch
2016-03-14 09:06:53serhiy.storchakasetassignee: serhiy.storchaka ->
messages: + msg261735
2016-03-08 19:32:06python-devsetmessages: + msg261370
2016-03-08 14:13:10python-devsetmessages: + msg261360
2016-02-24 11:05:05python-devsetmessages: + msg260787
2016-02-11 13:17:24serhiy.storchakasetresolution: fixed -> (no value)
stage: resolved -> patch review
2016-02-11 13:17:12serhiy.storchakasetstatus: closed -> open
files: + test_walk_bytes_deprecate.patch
messages: + msg260103
2016-02-08 18:52:02serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2016-02-08 14:34:51vstinnersetmessages: + msg259848
2016-02-08 14:33:39vstinnersetmessages: + msg259847
2016-02-08 14:24:40python-devsetnosy: + python-dev
messages: + msg259846
2016-02-08 14:07:39serhiy.storchakasetassignee: serhiy.storchaka
2016-01-16 10:13:37ideasman42setmessages: + msg258378
2016-01-13 08:39:08serhiy.storchakasetmessages: + msg258134
2016-01-13 08:34:56serhiy.storchakasetfiles: + walk_bytes_2.patch

messages: + msg258133
2015-12-23 13:29:57serhiy.storchakasetmessages: + msg256911
2015-12-23 13:14:47eryksunsetmessages: + msg256909
2015-12-23 11:35:36steve.dowersetmessages: + msg256908
2015-12-23 09:33:47serhiy.storchakasetfiles: + walk_bytes.patch
keywords: + patch
messages: + msg256904

stage: test needed -> patch review
2015-12-22 02:31:38benhoytsetmessages: + msg256820
2015-12-21 21:55:53vstinnersetmessages: + msg256812
2015-12-21 17:20:27zach.waresetpriority: normal -> high

messages: + msg256804
2015-12-21 09:31:47eryksunsetnosy: + eryksun
messages: + msg256787
2015-12-21 07:37:12ideasman42setnosy: + ideasman42
messages: + msg256785
2015-12-20 09:45:14serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg256757
2015-12-20 09:13:48vstinnersetmessages: + msg256755
2015-12-19 18:11:28serhiy.storchakasetnosy: + vstinner, benhoyt
2015-12-19 15:57:14SilentGhostsettype: behavior
versions: + Python 3.6
keywords: + 3.5regression
nosy: + SilentGhost

messages: + msg256732
stage: test needed
2015-12-19 15:39:38mont29create