classification
Title: subprocess.Popen with universal_newlines and nonblocking streams fails with "can't concat NoneType to bytes"
Type: behavior Stage:
Components: IO Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: martin.panter, sambayer
Priority: normal Keywords:

Created on 2019-01-17 15:22 by sambayer, last changed 2019-01-19 07:42 by martin.panter.

Messages (2)
msg333883 - (view) Author: Samuel Bayer (sambayer) Date: 2019-01-17 15:22
This bug is probably related to issue 24560.

This:

>>> import subprocess, fcntl, os
>>>> p = subprocess.Popen(["python", "-c", 'import time; time.sleep(5)'], stdin = subprocess.PIPE, stdout = subprocess.PIPE, stderr = subprocess.PIPE, universal_newlines= True)
>>> fcntl.fcntl(p.stderr.fileno(), fcntl.F_SETFL, os.O_NONBLOCK | fcntl.fcntl(p.stderr.fileno(), fcntl.F_GETFL))
>>> p.stderr.read()

causes this:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/codecs.py", line 321, in decode
    data = self.buffer + input
TypeError: can't concat NoneType to bytes

I'm assuming the problem is that the underlying unbuffered stream returns None and the incremental byte decoder that's induced by universal_newlines = True isn't expecting it.
msg334045 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2019-01-19 07:42
Yes, universal newlines mode uses the TextIOWrapper class to read the pipe, which isn’t really designed for non-blocking mode. This is the same problem described by Izbyshev at <https://bugs.python.org/issue13322#msg307763>.

Raising TypeError isn’t ideal. IMO it would be better to raise BlockingIOError, which is what basic calls like “os.read” would do, and what “BufferedReader.read” is supposed to do according to the documentation. (However the documentation and implementation don’t match; that is what Issue 13322 is about.)
History
Date User Action Args
2019-03-15 08:20:42martin.panterlinkissue36293 superseder
2019-01-19 07:42:50martin.pantersettype: crash -> behavior

messages: + msg334045
nosy: + martin.panter
2019-01-18 11:16:17nanjekyejoannahsettitle: subprocess.Popen with universal_newlines and nonblocking streams failes with "can't concat NoneType to bytes" -> subprocess.Popen with universal_newlines and nonblocking streams fails with "can't concat NoneType to bytes"
2019-01-17 15:22:48sambayercreate