classification
Title: Inadequate C-API to Python 3 I/O objects
Type: enhancement Stage:
Components: Extension Modules Versions: Python 3.2
process
Status: closed Resolution: postponed
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, pitrou, pv
Priority: normal Keywords:

Created on 2010-09-12 13:23 by pv, last changed 2013-07-08 15:40 by christian.heimes. This issue is now closed.

Messages (3)
msg116188 - (view) Author: Pauli Virtanen (pv) * Date: 2010-09-12 13:23
The C-API exposed by the `io` module on Python 3.1/3.2 is very limited, and makes interfacing with Python file objects in extension modules difficult.

In more detail:

1) Because the Python layer has buffering etc., the file handle returned by `PyObject_AsFileDescriptor` is not usable as-is. It requires flush and seek before use, every time there is a chance that the file object has been accessed on the Python side. 

2) There are no C-API functions such as the minimal set of `PyFile_Write(buf, length)`, `PyFile_Read(buf, length)`, `PyFile_Seek(pos, whence)`, `PyFile_Tell()`.

Instead, every call must go through PyObject_CallMethod, and the file objects only handle `PyBytes` and `PyByteArray` which are cumbersome and inefficient to use in extension modules.
msg116192 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-12 13:47
> Instead, every call must go through PyObject_CallMethod, and the file
> objects only handle `PyBytes` and `PyByteArray` which are cumbersome
> and inefficient to use in extension modules.

Because of the generic nature of the 3.x I/O stack, even shortcuts such as the proposed PyFile_Write will still have to use PyObject_CallMethod(obj, "write", ...) under the hood.

As for the types handled by file objects, you should be able to use a PyMemoryViewObject, which allows you to create a memory buffer without copying it (if that's what you're after).
You can also pass your own objects provided they support the new buffer API: http://docs.python.org/dev/c-api/buffer.html#bufferobjects
(I agree this documentation is not very well written, though)

So, bottom line, we could create a set of PyFile_* wrappers (or, rather, PyStream_*), but they wouldn't be much more efficient that what you can write by hand. Do you still think it's worth it? If so, I think you should float the idea on python-dev (the mailing-list).
msg192662 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-07-08 15:40
I agree with Antoine. If you are still interested to improve the situation then please start a discussion on the python-ideas mailing list.
History
Date User Action Args
2013-07-08 15:40:30christian.heimessetstatus: open -> closed

nosy: + christian.heimes
messages: + msg192662

resolution: postponed
2010-09-12 22:19:03eric.araujosetversions: - Python 3.1
2010-09-12 13:47:24pitrousetnosy: + pitrou
messages: + msg116192
2010-09-12 13:23:45pvcreate