classification
Title: Don't accept bytearray as filenames, or simplify the API
Type: Stage:
Components: Interpreter Core, Library (Lib), Unicode Versions: Python 3.2
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: lemburg, loewis, pitrou, vstinner
Priority: normal Keywords: patch

Created on 2010-04-21 12:23 by vstinner, last changed 2010-04-22 12:11 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
no_bytearray_filename.patch vstinner, 2010-04-21 12:28
Messages (6)
msg103829 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-04-21 12:23
r72313 (PEP 383) created the PyUnicode_FSConverter() function: encode an object to a byte string using the default file system encoding. PyBytes and PyByteArray are leaved unchanged (just increment the reference counter), PyUnicode is encoded to PyBytes (if the encoder produces something else, an error is raised).

In my opinion, a file name is a character string (Windows) or a byte string (POSIX) and a bytearray object is unexpected. Only few function support this type: no function of os.path accept them. In the Python test suite, no function use bytearray for filenames.

It's already complex to support 2 types (bytes and str) for filenames in os.path, I think that a third type is too much and has no real world use case (the module manipuling filenames is os.path and it doesn't support bytearray and nobody noticed that).

Suppport bytearray is complex because we need to acquire a lock (using PyObject_GetBuffer()) and release the lock (o->ob_type->tp_as_buffer->bf_releasebuffer(o, 0)... that's not really intuitive...). posixmodule.c uses functions bytes2str() and realease_bytes() to handle bytearray easily. But these functions are static and other modules have to reimplement them.

I propose the reject bytearray in PyUnicode_FSConverter(), or to simplify the API and fix Python to accept bytearray filename everywhere especially in os.path.

***

Reject bytearray in PyUnicode_FSConverter() is trivial only there is only one test that have to be changed in the whole Python3 test suite: test_empty_bytearray in test_bytes.py. This function shows that bytearray is complex and introduce subtle bugs (the lock/GIL issue). All code using PyUnicode_FSConverter() would become simpler.

Example:

----
/* Release the lock, decref the object. */
static void
release_bytes(PyObject* o)
{
        if (PyByteArray_Check(o))
                o->ob_type->tp_as_buffer->bf_releasebuffer(o, 0);
        Py_DECREF(o);
}

...
realease_byte(path);
----

becomes "Py_DECREF(path);" and release_bytes() can be removed.

And
------------
static char*
bytes2str(PyObject* o, int lock)
{
        if(PyBytes_Check(o))
                return PyBytes_AsString(o);
        else if(PyByteArray_Check(o)) {
                if (lock && PyObject_GetBuffer(o, NULL, 0) < 0)
                        /* On a bytearray, this should not fail. */
                        PyErr_BadInternalCall();
                return PyByteArray_AsString(o);
        } else {
                /* The FS converter should have verified that this
                   is either bytes or bytearray. */
                Py_FatalError("bad object passed to bytes2str");
                /* not reached. */
                return "";
        }
}

...
path = bytes2str(opath);
------------

becomes "path = PyBytes_AS_STRING(opath);" (or maybe "path = PyBytes_AsString(opath);" if you don't trust PyUnicode_FSConverter)  and bytes2str() can be removed.

***

Simplify the API means that bytes2str() and release_bytes() should become protected functions (not static, but prefixed by "_Py_") or maybe public ("Py_" prefix).

But the most complex part is to modify os.path to support bytearray, and this task would not be funny :-)
msg103832 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-04-21 12:28
Patch removing bytearray filename support: it mostly removes code.
msg103833 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-04-21 12:28
Or perhaps the bytearray can be converted to a bytes object. This is not optimal performance-wise but is unlikely to make a difference in real-world code (if you are passing a filename to an external API, chances are some IO will occur which will dwarf the cost of creating a separate bytes object).

But I agree that supporting bytearrays in filename-taking functions, while "nice" from a consistency point of view, isn't really useful in practice. So I would be ok to remove that support if it simplifies (or avoids complexifying) the logic for those functions.
msg103883 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-04-21 18:43
I'm in favor of removing that support, too, but please check with python-dev whether anybody thinks a proper deprecation cycle (deprecated in 3.2, removed in 3.3) is needed.
msg103910 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2010-04-21 20:44
Antoine Pitrou wrote:
> 
> Antoine Pitrou <pitrou@free.fr> added the comment:
> 
> Or perhaps the bytearray can be converted to a bytes object. This is not optimal performance-wise but is unlikely to make a difference in real-world code (if you are passing a filename to an external API, chances are some IO will occur which will dwarf the cost of creating a separate bytes object).
> 
> But I agree that supporting bytearrays in filename-taking functions, while "nice" from a consistency point of view, isn't really useful in practice. So I would be ok to remove that support if it simplifies (or avoids complexifying) the logic for those functions.

+1

bytearrays are basically the remains of the attempt to use mutable
byte string objects in Python 3.x. They may gain some usefulness
in the future, but I doubt that this will be in the area of filenames.
msg103958 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-04-22 12:11
MvL, MaL and Antoine Pitrou agreed to drop support of bytearray filenames in Python. I choosed to remove it directly in Python 3.2 because it wasn't really used, open() and os.path never supported bytearray filenames, and I will simplify the other issues related to surrogates in filenames.

Fixed: r80361 (py3k), don't backport to 3.1.
History
Date User Action Args
2010-04-23 11:38:00vstinnerlinkissue8242 dependencies
2010-04-22 12:11:16vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg103958
2010-04-21 20:44:54lemburgsetnosy: + lemburg
messages: + msg103910
2010-04-21 18:43:17loewissetmessages: + msg103883
2010-04-21 12:28:55pitrousetnosy: + pitrou
messages: + msg103833
2010-04-21 12:28:20vstinnersetfiles: + no_bytearray_filename.patch
keywords: + patch
messages: + msg103832
2010-04-21 12:24:32vstinnersetnosy: + loewis
2010-04-21 12:23:35vstinnercreate