classification
Title: multithreading traceback KeyError when modifying file
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Michael Allen, Michael Graczyk, akuchling, cheryl.sabella, miss-islington, rahul-kumi, xtreak
Priority: normal Keywords: patch

Created on 2015-12-15 17:21 by Michael Allen, last changed 2020-05-29 12:17 by miss-islington.

Pull Requests
URL Status Linked Edit
PR 12061 closed cunkel, 2019-02-27 00:40
PR 17360 closed python-dev, 2019-11-23 02:07
PR 18007 merged Michael Graczyk, 2020-01-15 04:24
PR 20079 closed miss-islington, 2020-05-14 16:37
PR 20092 merged akuchling, 2020-05-14 19:19
PR 20511 merged miss-islington, 2020-05-29 11:59
Messages (7)
msg256469 - (view) Author: Michael Allen (Michael Allen) Date: 2015-12-15 17:21
Modifying a file while getting a stacktrace across multiple threads causes linecache's cache to bust and del to be called on the global cache variable. This is not thread safe and raises a KeyError.

Reproducible with,

import threading
import traceback

def main():
    with open(__file__, 'a') as fp:
        fp.write(' ')
        traceback.format_stack()

threads = [
    threading.Thread(target=main)
    for i in range(100)
]
map(lambda t: t.start(), threads)
map(lambda t: t.join(), threads)

I see the following error,

Exception in thread Thread-56:
Traceback (most recent call last):
  File "/Users/me/.pyenv/versions/2.7.10/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/Users/me/.pyenv/versions/2.7.10/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "test.py", line 7, in main
    traceback.format_stack()
  File "/Users/me/.pyenv/versions/2.7.10/lib/python2.7/traceback.py", line 279, in format_stack
    return format_list(extract_stack(f, limit))
  File "/Users/me/.pyenv/versions/2.7.10/lib/python2.7/traceback.py", line 305, in extract_stack
    linecache.checkcache(filename)
  File "/Users/me/.pyenv/versions/2.7.10/lib/python2.7/linecache.py", line 69, in checkcache
    del cache[filename]
KeyError: 'test.py'

Possible solution is to ignore KeyError on del cache[filename].
msg323180 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2018-08-06 07:41
It seems there was a major refactor in traceback module with 6bc2c1e7ebf359224e5e547f58ffc2c42cb36a39 where this was fixed in Python 3. Ignoring the KeyError seems reasonable to me.

Thanks
msg359390 - (view) Author: Cheryl Sabella (cheryl.sabella) * (Python committer) Date: 2020-01-06 00:27
As @xtreak said, this looks like it was fixed for Python 3 and was only an issue for 2.7, so I'm closing the issue.
msg359392 - (view) Author: Michael Graczyk (Michael Graczyk) * Date: 2020-01-06 00:50
This issue still exists in Python 3. The repro just needs to be changed so that the threads are actually started.

- map(lambda t: t.start(), threads)
- map(lambda t: t.join(), threads)
+ [t.start() for t in threads]
+ [t.join() for t in threads]

My fix is linked.
msg359427 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2020-01-06 14:38
Thanks Michael, reopening. I was wrong while trying the reproducer since map is lazy in Python 3 and threads were not executed.
msg370301 - (view) Author: miss-islington (miss-islington) Date: 2020-05-29 11:59
New changeset b86636bff4b29ce23c886df079715dd951f13a07 by Andrew Kuchling in branch '3.8':
[3.8] bpo-25872: Fix KeyError in linecache when multithreaded (GH-18007) (GH-20092)
https://github.com/python/cpython/commit/b86636bff4b29ce23c886df079715dd951f13a07
msg370302 - (view) Author: miss-islington (miss-islington) Date: 2020-05-29 12:17
New changeset 852e8a7ed4d3d48e5c1c8120cfc932eb6a84bb8e by Miss Islington (bot) in branch '3.7':
[3.8] bpo-25872: Fix KeyError in linecache when multithreaded (GH-18007) (GH-20092)
https://github.com/python/cpython/commit/852e8a7ed4d3d48e5c1c8120cfc932eb6a84bb8e
History
Date User Action Args
2020-05-29 12:17:46miss-islingtonsetmessages: + msg370302
2020-05-29 11:59:57miss-islingtonsetpull_requests: + pull_request19755
2020-05-29 11:59:47miss-islingtonsetmessages: + msg370301
2020-05-18 16:49:34rahul-kumisetnosy: + rahul-kumi
2020-05-14 19:19:52akuchlingsetnosy: + akuchling
pull_requests: + pull_request19398
2020-05-14 16:37:08miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request19394
2020-01-15 04:24:26Michael Graczyksetstage: patch review
pull_requests: + pull_request17408
2020-01-06 14:38:49xtreaksetstatus: closed -> open
versions: + Python 3.9, - Python 2.7
messages: + msg359427

resolution: wont fix ->
stage: resolved -> (no value)
2020-01-06 00:50:41Michael Graczyksetnosy: + Michael Graczyk
messages: + msg359392
2020-01-06 00:27:17cheryl.sabellasetstatus: open -> closed

nosy: + cheryl.sabella
messages: + msg359390

resolution: wont fix
stage: patch review -> resolved
2019-11-23 02:07:22python-devsetpull_requests: + pull_request16845
2019-02-27 00:40:02cunkelsetkeywords: + patch
stage: patch review
pull_requests: + pull_request12085
2018-08-06 07:41:56xtreaksetnosy: + xtreak
messages: + msg323180
2018-07-11 07:41:59serhiy.storchakasettype: crash -> behavior
2015-12-15 17:21:04Michael Allencreate