New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Raw I/O writelines() broken for non-blocking I/O #70480
Comments
Copy of Antoine Pitrou's email (sent in 2012 ;-): Hello, I was considering a FileIO.writelines() implementation based on (it's probably broken with non-blocking streams too, for the same In the spirit of RawIO.write(), I think RawIO.writelines() could return Regards Antoine. -- |
"(it's probably broken with non-blocking streams too, for the same Yes it is :-) We hitted this issue on eventlet when I changed the socket.send() method in eventlet 0.18 to stop sending data on partial write, whereas eventlet < 0.18 used a loop to ensure that all bytes are written. The side effect of my change is that (indirectly) socket.makefile().writelines() may also use partial write if the underlying send() only writes partially data. The problem is that writelines() has *no* result, so the caller cannot be aware of the partial write and data is lost... eventlet:
"In the spirit of RawIO.write(), I think RawIO.writelines() could return I don't know yet what is the best option for eventlet, but we should probaly enhance the Python stdlib io module to return something on writelines() and *document* the behaviour on partial write. The problem writelines() API is that writelines() not only takes a single text/bytes string, but a *list* of text/bytes strings. Another option is to modify to retry on partial write to ensure that all data is written. If the underlying write() method returns None (blocking I/O error), we must raise an error. It doesn't make sense to retry writing on a non-blocking I/O error. In this case, we must document that writelines() *must not* be used on non-blocking I/O (ex: non-blocking socket, pipe, whatever). |
Is deprecating RawIOBase.writelines() an option, and only recommending BufferedIOBase.writelines() and TextIOBase.writelines()? Otherwise, I think it would make most sense to keep retrying until all the data is written. This mirrors how I understand readline() and readall() work (keeps reading until it gets as much as necessary). For non-blocking mode, readline() does not support that (see bpo-13858). It does not make much sense to me to have writelines() support non-blocking mode either. |
IMHO the problem is the same for other classes. |
For BufferedIOBase, the documentation says write() will not return a short number of bytes, so I don’t see how writelines() is a problem there. For TextIOBase, the documentation is not so clear <https://docs.python.org/release/3.4.3/library/io.html#io.TextIOBase.write\>, but I understand the standard library implementations do not do short writes. |
Victor, why did you change the title to specify non-blocking mode? I think blocking mode can also be handled at the same time. I propose:
BufferedIOBase.writelines() could be fixed to report the correct BlockingIOError.characters_written value, but that could be handled as a separate bug if anyone cares. |
Here is a patch documenting that RawIOBase.writelines() is undefined for partial writes and blocking errors, as mentioned in (1) above. The alternative of requiring write() to be retried would IMO be unfair to existing writelines() implementations. It also seems out of place considering we don’t even have a RawIOBase.writeexactly() type of method. |
I close this issue as duplicate of bpo-13322 which is older and has a longer history. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: