classification
Title: Race condition during pickle.load()
Type: crash Stage: resolved
Components: Library (Lib) Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: papad, pitrou, ronaldoussoren, serhiy.storchaka, taleinat
Priority: normal Keywords:

Created on 2019-05-02 08:24 by papad, last changed 2019-08-05 08:26 by taleinat. This issue is now closed.

Files
File name Uploaded Description Edit
reproduce.tar.gz papad, 2019-05-02 08:24
Messages (5)
msg341257 - (view) Author: papad (papad) Date: 2019-05-02 08:24
There seems to be a race condition when unpickling the same object from different threads (My guess is it's related to imports that are related to the unpickled object). 

I have used the following files for reproduction (File contents are in the attached archive, also pasted at the bottom of the issue report):
import_me.py - Containing the object we will pickle
pickle_object.py - the code to pickle the object
trigger.py - a piece of code to unpickle the object from different threads simultaneously and trigger the crash

I have used the files in the attached archive (containing the 3 files mentioned above)

to reproduce run the following commands:

pickle_object.py
trigger.py

Running trigger.py will crash on about 50% of the runs with the following error:

Traceback (most recent call last):
  File "./trigger.py", line 16, in pickle_load_thread
    pickle.load(h)
AttributeError: Can't get attribute 'PickleMe' on <module 'import_me' from '/data/import_me.py'>


I have tested this on the following software stacks:

1. python:3.7.3 docker
    - linux version: 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 GNU/Linux
    - python: 3.7.3
    - distro: Debian GNU/Linux 9 (stretch)

2. my laptop
    - linux version: 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
    - python: 3.6.7
    - distro: Ubuntu 18.04.2 LTS


I'm uncertain if this is a bug related to pickle or python module importing.
Similar issues I've found: https://bugs.python.org/issue12680 and https://bugs.python.org/issue30891.


---------------------------------------------------------------------------------------
filename: import_me.py
#! /usr/bin/python3


class PickleMe(object):

    def __init__(self):
        self.a = "a"

---------------------------------------------------------------------------------------
filename: pickle_object.py
#! /usr/bin/python3

import pickle
from import_me import PickleMe

p = PickleMe()

with open('pickled', 'wb') as h:
    pickle.dump(p, h)

---------------------------------------------------------------------------------------
filename: trigger.py
#! /usr/bin/python3

import threading
import pickle

import logging

threads = []

def pickle_load_thread():

    logging.error("Thread %d loading", threading.get_ident())

    try:
        with open('pickled', 'rb') as h:
            pickle.load(h)
    except:
        logging.exception("Exception in loading")


def start_pickle_load_thread():

    for x in range(2):
        load_thread = threading.Thread(target=pickle_load_thread)
        load_thread.daemon = True
        threads.append(load_thread)

    for x in threads:
        x.start()


if __name__ == '__main__':

    start_pickle_load_thread()

    for t in threads:
        t.join()
msg341265 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2019-05-02 13:09
This could be a duplicate of Issue34572
msg341424 - (view) Author: papad (papad) Date: 2019-05-05 08:49
In what version should this be fixed? I see https://bugs.python.org/issue34572 says the fix is in version 3.7, while I'm experiencing this on python 3.7.3, which version should I check?
msg341431 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-05-05 11:50
It should be fixed in 3.7.3.

https://docs.python.org/3.7/whatsnew/changelog.html#python-3-7-3-release-candidate-1
msg349035 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2019-08-05 08:26
On Windows 10, this reproduces consistently with 3.6 and 3.7.0, but not with 3.7.4, 3.8.0b3 and current master. So this definitely seems to be fixed.
History
Date User Action Args
2019-08-05 08:26:43taleinatsetstatus: open -> closed

nosy: + taleinat
messages: + msg349035

resolution: fixed
stage: resolved
2019-05-05 11:50:52serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg341431
2019-05-05 08:49:31papadsetmessages: + msg341424
2019-05-02 13:09:00ronaldoussorensetnosy: + ronaldoussoren
messages: + msg341265
2019-05-02 11:17:28xtreaksetnosy: + pitrou
2019-05-02 08:24:59papadcreate