classification
Title: multithreading traceback KeyError when modifying file
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Michael Allen, Michael Graczyk, akuchling, cheryl.sabella, iritkatriel, miss-islington, python-dev, rahul-kumi, uniocto, xtreak
Priority: normal Keywords: easy, patch

Created on 2015-12-15 17:21 by Michael Allen, last changed 2021-05-18 14:27 by iritkatriel. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 12061 closed cunkel, 2019-02-27 00:40
PR 17360 closed python-dev, 2019-11-23 02:07
PR 18007 merged Michael Graczyk, 2020-01-15 04:24
PR 20079 closed miss-islington, 2020-05-14 16:37
PR 20092 merged akuchling, 2020-05-14 19:19
PR 20511 merged miss-islington, 2020-05-29 11:59
PR 25913 merged python-dev, 2021-05-05 09:39
PR 26208 closed miss-islington, 2021-05-18 08:57
PR 26211 merged iritkatriel, 2021-05-18 10:38
PR 26212 merged iritkatriel, 2021-05-18 10:52
Messages (12)
msg256469 - (view) Author: Michael Allen (Michael Allen) Date: 2015-12-15 17:21
Modifying a file while getting a stacktrace across multiple threads causes linecache's cache to bust and del to be called on the global cache variable. This is not thread safe and raises a KeyError.

Reproducible with,

import threading
import traceback

def main():
    with open(__file__, 'a') as fp:
        fp.write(' ')
        traceback.format_stack()

threads = [
    threading.Thread(target=main)
    for i in range(100)
]
map(lambda t: t.start(), threads)
map(lambda t: t.join(), threads)

I see the following error,

Exception in thread Thread-56:
Traceback (most recent call last):
  File "/Users/me/.pyenv/versions/2.7.10/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/Users/me/.pyenv/versions/2.7.10/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "test.py", line 7, in main
    traceback.format_stack()
  File "/Users/me/.pyenv/versions/2.7.10/lib/python2.7/traceback.py", line 279, in format_stack
    return format_list(extract_stack(f, limit))
  File "/Users/me/.pyenv/versions/2.7.10/lib/python2.7/traceback.py", line 305, in extract_stack
    linecache.checkcache(filename)
  File "/Users/me/.pyenv/versions/2.7.10/lib/python2.7/linecache.py", line 69, in checkcache
    del cache[filename]
KeyError: 'test.py'

Possible solution is to ignore KeyError on del cache[filename].
msg323180 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2018-08-06 07:41
It seems there was a major refactor in traceback module with 6bc2c1e7ebf359224e5e547f58ffc2c42cb36a39 where this was fixed in Python 3. Ignoring the KeyError seems reasonable to me.

Thanks
msg359390 - (view) Author: Cheryl Sabella (cheryl.sabella) * (Python committer) Date: 2020-01-06 00:27
As @xtreak said, this looks like it was fixed for Python 3 and was only an issue for 2.7, so I'm closing the issue.
msg359392 - (view) Author: Michael Graczyk (Michael Graczyk) * Date: 2020-01-06 00:50
This issue still exists in Python 3. The repro just needs to be changed so that the threads are actually started.

- map(lambda t: t.start(), threads)
- map(lambda t: t.join(), threads)
+ [t.start() for t in threads]
+ [t.join() for t in threads]

My fix is linked.
msg359427 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2020-01-06 14:38
Thanks Michael, reopening. I was wrong while trying the reproducer since map is lazy in Python 3 and threads were not executed.
msg370301 - (view) Author: miss-islington (miss-islington) Date: 2020-05-29 11:59
New changeset b86636bff4b29ce23c886df079715dd951f13a07 by Andrew Kuchling in branch '3.8':
[3.8] bpo-25872: Fix KeyError in linecache when multithreaded (GH-18007) (GH-20092)
https://github.com/python/cpython/commit/b86636bff4b29ce23c886df079715dd951f13a07
msg370302 - (view) Author: miss-islington (miss-islington) Date: 2020-05-29 12:17
New changeset 852e8a7ed4d3d48e5c1c8120cfc932eb6a84bb8e by Miss Islington (bot) in branch '3.7':
[3.8] bpo-25872: Fix KeyError in linecache when multithreaded (GH-18007) (GH-20092)
https://github.com/python/cpython/commit/852e8a7ed4d3d48e5c1c8120cfc932eb6a84bb8e
msg381409 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2020-11-19 12:52
The issue was fixed but a unit test for this still needs to be added.
msg393187 - (view) Author: So Ukiyama (uniocto) * Date: 2021-05-07 14:46
I apologize if this is rude, as I am not familiar with this method.
I created a following PR to add unit tests about this issue.

https://github.com/python/cpython/pull/25913

I would be happy to receive feedback on the PR.
msg393872 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-05-18 13:54
New changeset 373741a97c9f6ffee427c2b4eaccb74347af228a by Irit Katriel in branch '3.10':
[3.10] bpo-25872: Add unit tests for linecache and threading (GH-25913) (GH-26212)
https://github.com/python/cpython/commit/373741a97c9f6ffee427c2b4eaccb74347af228a
msg393874 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-05-18 14:26
New changeset c05d8a6b67785450b1fec0d30fe26d5478bc4f0b by Irit Katriel in branch '3.9':
bpo-25872: Add unit tests for linecache and threading (GH-25913) (GH-26211)
https://github.com/python/cpython/commit/c05d8a6b67785450b1fec0d30fe26d5478bc4f0b
msg393875 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-05-18 14:27
@uniocto - thank you for the tests.
History
Date User Action Args
2021-05-18 14:27:03iritkatrielsetstatus: open -> closed
resolution: fixed
messages: + msg393875

stage: patch review -> resolved
2021-05-18 14:26:07iritkatrielsetmessages: + msg393874
2021-05-18 13:54:06iritkatrielsetmessages: + msg393872
2021-05-18 10:53:06iritkatrielsetcomponents: + Library (Lib), - Interpreter Core
2021-05-18 10:52:06iritkatrielsetpull_requests: + pull_request24829
2021-05-18 10:38:48iritkatrielsetpull_requests: + pull_request24828
2021-05-18 08:57:09miss-islingtonsetpull_requests: + pull_request24825
2021-05-07 14:46:44unioctosetnosy: + uniocto
messages: + msg393187
components: + Interpreter Core, - Library (Lib)
2021-05-05 09:39:14python-devsetkeywords: + patch
nosy: + python-dev

pull_requests: + pull_request24582
stage: test needed -> patch review
2020-11-19 12:52:17iritkatrielsetnosy: + iritkatriel
messages: + msg381409

keywords: + easy, - patch
stage: patch review -> test needed
2020-05-29 12:17:46miss-islingtonsetmessages: + msg370302
2020-05-29 11:59:57miss-islingtonsetpull_requests: + pull_request19755
2020-05-29 11:59:47miss-islingtonsetmessages: + msg370301
2020-05-18 16:49:34rahul-kumisetnosy: + rahul-kumi
2020-05-14 19:19:52akuchlingsetnosy: + akuchling
pull_requests: + pull_request19398
2020-05-14 16:37:08miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request19394
2020-01-15 04:24:26Michael Graczyksetstage: patch review
pull_requests: + pull_request17408
2020-01-06 14:38:49xtreaksetstatus: closed -> open
versions: + Python 3.9, - Python 2.7
messages: + msg359427

resolution: wont fix -> (no value)
stage: resolved -> (no value)
2020-01-06 00:50:41Michael Graczyksetnosy: + Michael Graczyk
messages: + msg359392
2020-01-06 00:27:17cheryl.sabellasetstatus: open -> closed

nosy: + cheryl.sabella
messages: + msg359390

resolution: wont fix
stage: patch review -> resolved
2019-11-23 02:07:22python-devsetpull_requests: + pull_request16845
2019-02-27 00:40:02cunkelsetkeywords: + patch
stage: patch review
pull_requests: + pull_request12085
2018-08-06 07:41:56xtreaksetnosy: + xtreak
messages: + msg323180
2018-07-11 07:41:59serhiy.storchakasettype: crash -> behavior
2015-12-15 17:21:04Michael Allencreate