Title: [3.7] Popen Control Characters in stdout affect shell session
Type: Stage: resolved
Components: Versions: Python 3.7
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: San, steven.daprano, terry.reedy
Priority: normal Keywords:

Created on 2021-07-22 16:00 by San, last changed 2021-07-23 21:17 by terry.reedy. This issue is now closed.

File name Uploaded Description Edit
shell.png San, 2021-07-22 16:00
Messages (5)
msg397986 - (view) Author: San (San) Date: 2021-07-22 16:00
I was trying to wrap an old game server executable with a built-in console using Popen.

class ServerWrapper(Thread):
    def __init__(self, pathToExecutable: str, statusMonitor: Popen = None, active: bool = False):
        self.pathToExecutable = pathToExecutable
        self.statusMonitor = statusMonitor = active

    def run(self):
        self.statusMonitor = Popen(['./bf1942_lnxded', "+statusMonitor", '1'], encoding='latin_1', stdout=PIPE,
                                   stdin=PIPE, cwd=self.pathToExecutable)
            currentLine = self.statusMonitor.stdout.readline()
            if currentLine:

    def start(self): = True

    def stop(self): = False

I expected to be able to read the output line-by-line using enter, in a 'normal fashion'.
After importing this from a terminal and setting it up somewhat like so:

wrapper = ServerWrapper('bf1942')

However, once the server process started and thereby started writing to stdout, weird stuff started to happen to my shell with the python interpreter.

Apparently, there are control characters and ansi-escape codes sent from the process, that somehow manage to manipulate my shell if I specify an encoding. Consequentially the lines output with 'print(currentLine)' are reduced by these chars.

Is this the desired behaviour of the decoder? If so then I think this should potentially be further clarified in the documentation of Popen. I would have expected the chars to be escaped because the worst thing is, this happens even if you don't actually read from stdout at all. Seemingly the decoder executes incoming control sequences immediately (probably for the buffer?), regardless of if you actually bother to read the stdout or not.

The paragraph in sounds like this shouldn't be happening if 'shell' is not set to 'True' at least.
msg398018 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2021-07-23 00:48
I might be missing something here, but if you send arbitrary binary data to stdout, where it gets written to the terminal, strange things are likely to happen. The terminal interprets the shell characters before Python gets to see them.

I think that this is pretty much unavoidable.

The Security Considerations section in the docs is about avoiding shell code injection attacks by sending unescaped data to the shell, it isn't relevant to the reverse situation where the process sends terminal control codes back.
msg398035 - (view) Author: San (San) Date: 2021-07-23 08:36
I get your point. But I thought setting 'stdout=PIPE' should not send the stdout directly to the terminal but rather through the pipe first?!

Also what you're saying seems to contradict that this doesn't happen if I don't set the encoding and let it be a byte-stream. In that case you get the full byte-stream without any control-characters being executed.

In my opinion it is unintuitive that this is happening. 

For instance if I open a file using io, that contains the same control characters like so:
f = open('cc.txt', 'r', encoding='latin_1')

Then gladly this doesn't happen and the control chars get escaped using \x1b which is what I would also expect to happen when using Popen.
msg398037 - (view) Author: San (San) Date: 2021-07-23 08:55
I just realised that this is not a problem of python but rather a problem of the shell that I was using. I was testing the io in a different environment. Now I retried in the same environment that the wrapper was operating and the behaviour is the same using io.

Sorry for the confusion and thanks for nudging me the right direction.
msg398097 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-07-23 21:17
Interpreting the last post ...
Date User Action Args
2021-07-23 21:17:36terry.reedysetstatus: open -> closed

nosy: + terry.reedy
messages: + msg398097

resolution: not a bug
stage: resolved
2021-07-23 08:55:27Sansetmessages: + msg398037
2021-07-23 08:36:52Sansetmessages: + msg398035
2021-07-23 00:48:21steven.dapranosetnosy: + steven.daprano
messages: + msg398018
2021-07-22 16:00:26Sancreate