Author jtaylor
Recipients jtaylor
Date 2017-04-14.14:18:56
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1492179536.71.0.785460457662.issue30073@psf.upfronthosting.co.za>
In-reply-to
Content
Probably a case of 'don't do that' but reading lines in a compressed files in binary mode produces bytes with invalid newlines in encodings that where '\n' is encoded as something else:

with lzma.open("test.xz", "wt", encoding="UTF-32-LE") as f:
    f.write('0 1 2\n3 4 5');

lzma.open("test.xz", "rb").readlines()[0].decode('UTF-32-LE')

Fails with:
UnicodeDecodeError: 'utf-32-le' codec can't decode byte 0x0a in position 20: truncated data

as readlines() produces:
b'0\x00\x00\x00 \x00\x00\x001\x00\x00\x00 \x00\x00\x002\x00\x00\x00\n'
The last newline should be '\n'.encode('UTF-32-LE') == b'\n\x00\x00\x00'
History
Date User Action Args
2017-04-14 14:18:56jtaylorsetrecipients: + jtaylor
2017-04-14 14:18:56jtaylorsetmessageid: <1492179536.71.0.785460457662.issue30073@psf.upfronthosting.co.za>
2017-04-14 14:18:56jtaylorlinkissue30073 messages
2017-04-14 14:18:56jtaylorcreate