Author Nathan.Goldbaum
Recipients Nathan.Goldbaum, pitrou
Date 2020-02-19.21:44:28
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1582148668.53.0.428067131933.issue39681@roundup.psfhosted.org>
In-reply-to
Content
So I *think* I've pieced together what caused the user crash that originated in the flair library. It turns out that pickle.load, via torch.load, is getting passed an mmap.mmap. 

https://github.com/flairNLP/flair/blob/1d44abf35f1122d1e146c9a219b2cb5f98b41b40/flair/file_utils.py#L25-L36

https://github.com/flairNLP/flair/blob/1d44abf35f1122d1e146c9a219b2cb5f98b41b40/flair/nn.py#L85-L86

Since mmap doesn't implement readinto, pickle.load objects as of Python 3.8. This is new behavior in Python3.8, it used to be possible to load a memory-mapped pickle file.

Short repro script:

import pickle
import mmap
data = "some data"

with open('my_data.pkl', 'wb') as f:
    pickle.dump(data, f)

with open("my_data.pkl", "r+b") as f_in:
    mm = mmap.mmap(f_in.fileno(), 0)

print(pickle.load(mm))

On Python3.8, this script prints an error, on Python3.7 it prints "some data".
History
Date User Action Args
2020-02-19 21:44:28Nathan.Goldbaumsetrecipients: + Nathan.Goldbaum, pitrou
2020-02-19 21:44:28Nathan.Goldbaumsetmessageid: <1582148668.53.0.428067131933.issue39681@roundup.psfhosted.org>
2020-02-19 21:44:28Nathan.Goldbaumlinkissue39681 messages
2020-02-19 21:44:28Nathan.Goldbaumcreate