Title: .readline() has bug WRT nonblocking files
Type: enhancement Stage:
Components: Library (Lib) Versions: Python 2.4, Python 2.3, Python 2.6, Python 2.5
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: Yuri.Bochkarev, ajb, gvanrossum, jafo
Priority: low Keywords:

Created on 2007-09-18 10:37 by ajb, last changed 2013-11-07 13:01 by Yuri.Bochkarev. This issue is now closed.

Messages (7)
msg55993 - (view) Author: Alex Burr (ajb) Date: 2007-09-18 10:37
If you have made a file nonblocking using fcntl.fcntl, .readline() will
discard  the start of a line if you get EGAIN. It should attach the
partial line to the exception somehow - or at least warn the user.

I observe this on 2.3.5, but the same code exists in TRUNK
msg55996 - (view) Author: Sean Reifschneider (jafo) * (Python committer) Date: 2007-09-18 15:11
Doesn't the exception count as warning the user?

We probably don't want to change readline to return a partial line in
this case.  An exception could be added for EGAIN that includes the
partial line.  Another option would be to just document the behavior for
readline or fcntl.

What is the right behavior for a non-blocking readline?  Obviously, it
can't return a None and buffer the line.  Another option would be to
define readline as a blocking operation and enable buffering before
starting the readline, and possibly revert it when done.

msg55998 - (view) Author: Alex Burr (ajb) Date: 2007-09-18 15:40
The exception would count as a warning if it wasn't EGAIN. One expects
to catch EGAIN and try again.

The current situation is unfortunate because it *nearly* works. My
scenario is: I'm driving GDB/MI via popen2.Popen3 ( gdbCommand, False,1).
It works for most GDB commands, but sometimes GDB returns a huge line
and I get EGAIN in the middle. (For my purposes, I've worked round it,
by avoiding the command which generates a long line)
msg55999 - (view) Author: Sean Reifschneider (jafo) * (Python committer) Date: 2007-09-18 15:46
Why are you putting the file in non-blocking mode?  Why not just reading
in blocking mode?  If you want to do other work when a line is not
available, you could use select to check to see if there's data ready
via a small or 0 timeout.
msg56000 - (view) Author: Alex Burr (ajb) Date: 2007-09-18 16:25
I don't need to change my code as I have now worked round the problem,
but for an example of why someone might want nonblocking mode:

It was not actually in order to do any more work, but because I was
managing both input and output to gdb in one thread (for simplicity). I
therefore needed to collect everything gdb had printed out, but not try
to read any more and cause a block because then I could not give GDB its
next command, and gdb would not give me any more until I did.

I think I did try select, but if I recall correctly, that doesn't work
well with .readline() either. I can't remember why, though.
msg56015 - (view) Author: Sean Reifschneider (jafo) * (Python committer) Date: 2007-09-18 21:24
Arguably, you should be using "select" and "read" (instead of readline)
for this operation.  That's what I've done in the past when doing
something similar.  Specifically, I believe I have looped reading into a
buffer with read, and using select with a timeout (even a fraction of a
second can work) until the timeout triggers.  This way I don't block,
but I also don't run into problems with the system call returning an error.

I avoid using readline in this case because I don't expect it to work
well on the stdout of an interactive command.

You may also want to look at the pty module.

I guess I'd say this should probably go to the python mailing list for
further discussion.  Can you post a message there about it?
msg56041 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2007-09-19 17:22
readline() goes through C stdio which makes it impossible to get
non-blocking I/O right. You should be using raw calls (until
python 3000 which will remove Python's reliance on C stdio).
Date User Action Args
2013-11-07 13:01:02Yuri.Bochkarevsetnosy: + Yuri.Bochkarev
2007-09-19 17:22:28gvanrossumsetstatus: open -> closed
resolution: wont fix
messages: + msg56041
nosy: + gvanrossum
2007-09-18 21:24:29jafosetmessages: + msg56015
2007-09-18 16:25:13ajbsetmessages: + msg56000
2007-09-18 15:46:52jafosetmessages: + msg55999
2007-09-18 15:40:36ajbsetmessages: + msg55998
2007-09-18 15:11:19jafosetversions: + Python 2.6, Python 2.5, Python 2.4
nosy: + jafo
messages: + msg55996
priority: low
components: + Library (Lib), - Interpreter Core
type: behavior -> enhancement
2007-09-18 10:37:34ajbcreate