This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Misleading TypeError when pickling bytes to a file opened as text
Type: enhancement Stage:
Components: IO Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: alexandre.vassalotti, boris, jaraco, joncle, pitrou, serhiy.storchaka, stein-k
Priority: low Keywords:

Created on 2015-05-10 14:34 by jaraco, last changed 2022-04-11 14:58 by admin.

Messages (3)
msg242858 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2015-05-10 14:34
I had a piece of code which I distilled to this:

import pickle
with open('out.pickle', 'w') as out:
    pickle.dump(out, b'data')

Running that code raises this error:

TypeError: must be str, not bytes

The error is raised at the dump call with no additional context. Based on the error, my reaction is to think that pickled doesn't support bytes objects in pickles.

On further examination, it's not actually that the bytes cannot be pickled, but that the 'dump' call requires that the file be opened in binary mode ('wb'), but because of the error message essentially says "expecting a text string" and because it's unclear that the error is raised during the write to the stream and because the JSON library expects an output stream to be opened in text mode, the error message is misleading.

At least [two other people think the behavior could be clearer](http://stackoverflow.com/questions/13906623/using-pickle-dump-typeerror-must-be-str-not-bytes#comment43761528_13906715).

Would it be possible and reasonable to trap a TypeError at the call to `.write` and replace or augment the message with something like "file must be opened in binary mode"?

On second thought, perhaps the culprit isn't pickle here, but the stream writer. Perhaps the `.write` method should provide a clearer message indicating the context at which it's expecting str and not bytes.
msg288157 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-02-19 19:24
I think it is worth to improve the error message in the write() method of binary files.

>>> sys.stdout.write(b'')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: write() argument must be str, not bytes
>>> sys.stdout.buffer.write('')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: a bytes-like object is required, not 'str'
>>> sys.stdout.buffer.raw.write('')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: a bytes-like object is required, not 'str'

But this is large issue. Other file-like objects (GzipFile, ZipExtFile etc) should be updated too.
msg355816 - (view) Author: Борис Верховский (boris) * Date: 2019-11-01 13:38
As I said in issue38226, the error message you get when you try to pickle.load() a file opened in "r" mode instead of "rb" mode,

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

is also confusing and can be improved.
History
Date User Action Args
2022-04-11 14:58:16adminsetgithub: 68347
2019-11-02 21:39:44stein-ksetnosy: + stein-k
2019-11-01 13:38:36borissetnosy: + boris
messages: + msg355816
2018-02-19 12:13:24serhiy.storchakasetversions: + Python 3.8, - Python 3.7
2017-02-19 19:24:35serhiy.storchakasetversions: + Python 3.7, - Python 3.5
nosy: + serhiy.storchaka

messages: + msg288157

type: behavior -> enhancement
2015-05-11 21:34:10ned.deilysetnosy: + pitrou, alexandre.vassalotti
2015-05-10 17:08:15jonclesetnosy: + joncle
2015-05-10 14:34:19jaracocreate