This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: "RuntimeError: dictionary changed size during iteration" using trace.py module
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Delgan, belopolsky, paul.moore, pitrou, pmpp, rhettinger, steve.dower, terry.reedy, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2018-03-11 17:19 by Delgan, last changed 2022-04-11 14:58 by admin.

Files
File name Uploaded Description Edit
minitrace.py Delgan, 2018-03-17 13:00
Messages (6)
msg313604 - (view) Author: Delgan (Delgan) * Date: 2018-03-11 17:19
Hello.

I am strangely encountering an error whil trying to run "python -m trace -c script.py" on this simple code:

> import multiprocessing
> queue = multiprocessing.Queue()
> queue.put("a")

Which raises on Windows 10 using Python 3.6.3:

> Traceback (most recent call last):
>   File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
>     "__main__", mod_spec)
>   File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
>     exec(code, run_globals)
>   File "/usr/lib/python3.6/trace.py", line 742, in <module>
>     main()
>   File "/usr/lib/python3.6/trace.py", line 739, in main
>     results.write_results(opts.missing, opts.summary, opts.coverdir)
>   File "/usr/lib/python3.6/trace.py", line 258, in write_results
>     for filename, lineno in self.counts:
> RuntimeError: dictionary changed size during iteration

Fixing it seems straightforward, but I do not know what is causing the bug internally.
msg313789 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-03-13 22:29
I cannot reproduce this on Ubuntu 16.04.
msg313978 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-03-16 22:31
I copied the three lines into a file, deleted '> ', added  print('done'), and saved.  I ran the file, on Windows 10, with installed 2.7, 3.5, 3.6, and repository 3.6, 3.7, and 3.8.  No failures.
  
Adrien, please retest by copying what you posted, and if you still get a failure, upgrade to 3.6.4 and try again.  Unless we see a failure on another machine, this should be closed.
msg313986 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2018-03-17 04:46
Running on MacOs, I can't reproduce the failure.
msg314003 - (view) Author: Delgan (Delgan) * Date: 2018-03-17 13:00
Thanks for your attention to this issue.

I am not surprised that you was not able to reproduce it as it seems deeply related to multiprocessing and threads. 

I tested it on 3 others completely different computers, and I was able to reproduce the error for 2 of them which were running Ubuntu 16.04 and  Windows 10. I made a totally fresh install of Python 3.6.4, the Windows machine never had Python installed before. 
The only common point between computers where the error occurs compared to the third one, whether running Windows or Linux, is that they have a fairly old or weak hardware, and therefore are slower.

Fortunately, the error was systematic on my machine. So I was able to study the trace.py code to try to isolate the problem. Piece by piece, I simplified trace.py to get a small snippet that reproduces the bug (I joined the file to this message).
Sometimes, there is also a "BrokenPipeError" which pop-out. I do not know if this is related.

The error seems to come from the fact that "counts" start to be iterated while compiled code execution is not fully terminated, and so "localtrace_count" may add another item to the dict. Move the "time.sleep()" before the loop and the error should gone. This is most likely related to the internal thread used by Queue as using a SimpleQueue doesn't raise an exception.

As I do not have access for now to a computer where the error does not occur, I can not continue my investigations. However I hope that I give a little more information so that you maybe succeed to reproduce it.
msg314010 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-03-17 18:28
Thank you for the improved example.  From Command Prompt, I now see
f:\dev\37>python -m trace -c f:/python/a/tem2.py
Running Debug|Win32 interpreter...
Traceback (most recent call last):
  File "f:\dev\37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "f:\dev\37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "f:\dev\37\lib\trace.py", line 737, in <module>
    main()
  File "f:\dev\37\lib\trace.py", line 725, in main
    t.runctx(code, globs, globs)
  File "f:\dev\37\lib\trace.py", line 463, in runctx
    exec(cmd, globals, locals)
  File "f:/python/a/tem2.py", line 31, in <module>
    main()
  File "f:/python/a/tem2.py", line 24, in main
    for filename, lineno in counts:
RuntimeError: dictionary changed size during iteration
History
Date User Action Args
2022-04-11 14:58:58adminsetgithub: 77228
2018-03-17 18:28:51terry.reedysetmessages: + msg314010
2018-03-17 13:00:56Delgansetfiles: + minitrace.py

messages: + msg314003
2018-03-17 06:02:06pmppsetnosy: + pmpp
2018-03-17 04:46:01rhettingersetnosy: + rhettinger
messages: + msg313986
2018-03-16 22:31:24terry.reedysetnosy: + terry.reedy
messages: + msg313978
2018-03-13 22:29:15pitrousetnosy: + pitrou, belopolsky, paul.moore, tim.golden, zach.ware, steve.dower
messages: + msg313789
2018-03-11 17:19:59Delgancreate