This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: pdb.set_trace() crashes with UnicodeDecodeError when binary data is input via stdin
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.4
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: ned.deily, silverbacknet
Priority: normal Keywords:

Created on 2014-07-19 01:01 by silverbacknet, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (4)
msg223453 - (view) Author: Silverback Networks (silverbacknet) Date: 2014-07-19 01:01
sample.py:
import sys, pdb
infile = sys.stdin.buffer
pdb.set_trace()

command line:
python -m pdb sample.py < binary.exe

output:
> c:\users\me\dropbox\sample.py(1)<module>()
-> import sys, pdb
(Pdb) Traceback (most recent call last):
  File "C:\dev\Python\x86\lib\pdb.py", line 1661, in main
    pdb._runscript(mainpyfile)
  File "C:\dev\Python\x86\lib\pdb.py", line 1542, in _runscript
    self.run(statement)
  File "C:\dev\Python\x86\lib\bdb.py", line 431, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "c:\users\me\dropbox\test.py", line 1, in <module>
    import sys, pdb
  File "c:\users\me\dropbox\test.py", line 1, in <module>
    import sys, pdb
  File "C:\dev\Python\x86\lib\bdb.py", line 48, in trace_dispatch
    return self.dispatch_line(frame)
  File "C:\dev\Python\x86\lib\bdb.py", line 66, in dispatch_line
    self.user_line(frame)
  File "C:\dev\Python\x86\lib\pdb.py", line 259, in user_line
    self.interaction(frame, None)
  File "C:\dev\Python\x86\lib\pdb.py", line 346, in interaction
    self._cmdloop()
  File "C:\dev\Python\x86\lib\pdb.py", line 319, in _cmdloop
    self.cmdloop()
  File "C:\dev\Python\x86\lib\cmd.py", line 126, in cmdloop
    line = input(self.prompt)
  File "C:\dev\Python\x86\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 72: character maps to <undefined>
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> c:\dev\python\x86\lib\encodings\cp1252.py(23)decode()
-> return codecs.charmap_decode(input,self.errors,decoding_table)[0]
(Pdb) Traceback (most recent call last):
  File "C:\dev\Python\x86\lib\pdb.py", line 1661, in main
    pdb._runscript(mainpyfile)
  File "C:\dev\Python\x86\lib\pdb.py", line 1542, in _runscript
    self.run(statement)
  File "C:\dev\Python\x86\lib\bdb.py", line 431, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "c:\users\me\dropbox\test.py", line 1, in <module>
    import sys, pdb
  File "c:\users\me\dropbox\test.py", line 1, in <module>
    import sys, pdb
  File "C:\dev\Python\x86\lib\bdb.py", line 48, in trace_dispatch
    return self.dispatch_line(frame)
  File "C:\dev\Python\x86\lib\bdb.py", line 66, in dispatch_line
    self.user_line(frame)
  File "C:\dev\Python\x86\lib\pdb.py", line 259, in user_line
    self.interaction(frame, None)
  File "C:\dev\Python\x86\lib\pdb.py", line 346, in interaction
    self._cmdloop()
  File "C:\dev\Python\x86\lib\pdb.py", line 319, in _cmdloop
    self.cmdloop()
  File "C:\dev\Python\x86\lib\cmd.py", line 126, in cmdloop
    line = input(self.prompt)
  File "C:\dev\Python\x86\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 72: character maps to <undefined>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\dev\Python\x86\lib\runpy.py", line 171, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\dev\Python\x86\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\dev\Python\x86\lib\pdb.py", line 1685, in <module>
    pdb.main()
  File "C:\dev\Python\x86\lib\pdb.py", line 1677, in main
    pdb.interaction(None, t)
  File "C:\dev\Python\x86\lib\pdb.py", line 346, in interaction
    self._cmdloop()
  File "C:\dev\Python\x86\lib\pdb.py", line 319, in _cmdloop
    self.cmdloop()
  File "C:\dev\Python\x86\lib\cmd.py", line 126, in cmdloop
    line = input(self.prompt)
  File "C:\dev\Python\x86\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 57: character maps to <undefined>

python version:
Python 3.4.0 (v3.4.0:04f714765c13, Mar 16 2014, 19:24:06) [MSC v.1600 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
msg223455 - (view) Author: Silverback Networks (silverbacknet) Date: 2014-07-19 01:18
Just verified the problem on 3.4.1 as well.
msg223460 - (view) Author: Silverback Networks (silverbacknet) Date: 2014-07-19 02:54
OK, now that I've screwed my thinking cap back on, it's obvious that pdb in the example is pulling from stdin. However, that isn't be the case in my original problem, so modify the example to:

sample.py:
import sys, pdb
infile = sys.stdin.buffer.read()
pdb.set_trace()

Same error despite nothing being left in stdin's buffer.
msg223494 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-07-20 02:02
The problem here is that you are trying to use sys.stdin for two different purposes: 1. as a data input source for your program using stdin redirected to a disk file; and 2. for pdb's source of debugging commands, expected to be a terminal file.  By specifying "-m pdb", you are telling pdb to start an interactive debugging which it does, paused at the first statement in sample.py ("-> import sys, pdb").  It (actually the cmd standard library module which pdb uses) then tries to read stdin for command input but, because of the command line redirect (" < binary.exe"), reads the binary file instead.  Note this happens before sample.py has done anything.  If you just ran the command "python sample.py < binary.exe", then pdb would not be involved initially, the script would run, read the redirected file until EOF, and then invoke pdb via set_trace.  But, at that point, the redirected sys.stdin is at EOF so pdb immediately exits after trying to read a command.  The doc page for pdb does not specifically warn against redirecting stdin but it does document that the pdb.Pdb instance passes stdin arguments to a cmd.Cmd instance which defaults to using sys.stdin.  I think that's warning enough.

There is an open pdb feature request, Issue20061, to make it more convenient to use pdb with a separate terminal window.  That feature would also cover the rare use case here of having a redirected stdin with pdb.  Otherwise, you need to be careful how you use stdin while using pdb.
History
Date User Action Args
2022-04-11 14:58:06adminsetgithub: 66208
2014-07-20 02:03:01ned.deilysetstatus: open -> closed

type: crash ->
components: + Library (Lib), - Windows

nosy: + ned.deily
messages: + msg223494
resolution: not a bug
stage: resolved
2014-07-19 02:54:26silverbacknetsetmessages: + msg223460
2014-07-19 01:18:17silverbacknetsetmessages: + msg223455
title: pdb.set_trace() crashes when binary data is input via stdin -> pdb.set_trace() crashes with UnicodeDecodeError when binary data is input via stdin
2014-07-19 01:01:50silverbacknetcreate