This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: finalizer
Type: Stage:
Components: Library (Lib) Versions: Python 3.0, Python 2.5
process
Status: closed Resolution: duplicate
Dependencies: Superseder: os.listdir can return byte strings
View: 3187
Assigned To: Nosy List: Ilya.Kulakov, pitrou, vstinner
Priority: normal Keywords:

Created on 2008-08-20 09:00 by vstinner, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Messages (6)
msg71521 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-08-20 09:00
If the directory contains invalid filenames (invalid in the system 
charset), an exception is raised by os.path.join() used by 
shutil.rmtree():

    fullname = os.path.join(path, name)
  File "/home/haypo/prog/py3k/Lib/posixpath.py", line 64, in join
    if b.startswith('/'):
TypeError: expected an object with the buffer interface

name is an bytes object, not a str object. My system charset is utf-8.

Example to reproduce the problem:

   mkdir x
   # create a file with an invalid name
   touch -- $(echo -e 'x/--\0250--')
   python -c "import shutils; shutil.rmtree('x')"

=> TypeError: expected an object with the buffer interface
msg71524 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2008-08-20 09:36
This is certainly a consequence of #3187, which is very high priority
but also very difficult to find a satisying solution to.
msg72084 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-08-28 11:47
Python 2.5 has the same problem (at least, on Linux). rmtree(<unicode 
directory name>) fails if the directory contains invalid unicode 
string. Backtrace:
---
File "shutil.py", line 163, in rmtree
   fullname = os.path.join(path, name)
File "posixpath.py", line 65, in join
   path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xac in position 
3: ordinal not in range(128)
---

The filename:
   $ ls /tmp/run-3/exitcode1/run-4/run-1/session-2/
   ?I?#??????????|?*?  Pum

The instruction was 
rmtree(u"/tmp/run-3/exitcode1/run-4/run-1/session-2").
msg72088 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2008-08-28 13:49
Selon STINNER Victor <report@bugs.python.org>:
>
> Python 2.5 has the same problem (at least, on Linux). rmtree(<unicode
> directory name>) fails if the directory contains invalid unicode
> string. Backtrace:

Well, I'm not sure we should call it the same problem, although the roots are
the same. The difference is that in 2.x, using bytes strings for file and
directory names is quite normal, especially under Linux where they are just
bytes at the OS level. In 3.0 though, those strings are supposed to be unicode
at the Python level, which makes the problem much more critical.
msg246888 - (view) Author: Ilya Kulakov (Ilya.Kulakov) * Date: 2015-07-18 07:13
This issue is marked as closed as a duplicate without a reference to the original task.

I still have this issues on Python 3.4.2, on Windows when shutil.rmtree fails to
msg246889 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-07-18 07:43
Hi, the issue was fixed in Python 3. On Windows, you must use Unicode.
Otherwise, you can get errors like that. On other platforms, Unicode is now
also the best choice on Python 3.

See the second message for the superseder issue.
History
Date User Action Args
2022-04-11 14:56:37adminsetgithub: 47866
2015-07-18 07:43:06vstinnersetmessages: + msg246889
2015-07-18 07:13:20Ilya.Kulakovsetnosy: + Ilya.Kulakov

messages: + msg246888
title: shutil.rmtree() fails on invalid filename -> finalizer
2008-08-28 13:49:50pitrousetmessages: + msg72088
2008-08-28 11:47:00vstinnersetmessages: + msg72084
versions: + Python 2.5
2008-08-20 09:37:30pitrousetsuperseder: os.listdir can return byte strings
2008-08-20 09:36:39pitrousetstatus: open -> closed
resolution: duplicate
messages: + msg71524
nosy: + pitrou
2008-08-20 09:00:58vstinnercreate