classification
Title: functools.lru_cache raises KeyError
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: 24483 Superseder:
Assigned To: Nosy List: Aaron.Meurer, Peter Brady, rhettinger, serhiy.storchaka
Priority: normal Keywords:

Created on 2015-10-02 03:58 by Peter Brady, last changed 2016-02-10 22:25 by Peter Brady. This issue is now closed.

Messages (7)
msg252083 - (view) Author: Peter Brady (Peter Brady) Date: 2015-10-02 03:58
The SymPy project (https://github.com/sympy/sympy) makes heavy use of caching to speed up the creation of symbols and expressions.  If available, we make use of the fastcache library (https://github.com/pbrady/fastcache) - an lru_cache written in C.  Otherwise we use lru_cache provided by functools.  When testing with 3.5, we started to observe `KeyError`s coming from the new lru_cache implementation in _functoolsmodule.c.  There was some discussion on this on the SymPy mailing list here https://groups.google.com/forum/#!topic/sympy/AnwYTJGRBB4.

Here's an example of the failure in the SymPy test suite (Note that this failure does not occur if we use the lru_cache in 3.4 or fastcache or if we use the lru_cache in 3.5 with no size limit)

$ /usr/local/opt/python-3.5/bin/python3
Python 3.5.0 (default, Oct  1 2015, 21:51:43) 
[GCC 5.1.1 20150618 (Red Hat 5.1.1-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from sympy import test
>>> test("test_frame.py")
============================= test process starts ==============================
executable:         /usr/local/opt/python-3.5/bin/python3  (3.5.0-final-0) [CPython]
architecture:       64-bit
cache:              yes
ground types:       python 
random seed:        7625225
hash randomization: on (PYTHONHASHSEED=2787593824)

sympy/physics/vector/tests/test_frame.py[4] .EE.                          [FAIL]

________________________________________________________________________________
____________ sympy/physics/vector/tests/test_frame.py:test_ang_vel _____________
  File "/home/peter/.local/lib/python3.5/site-packages/sympy/physics/vector/tests/test_frame.py", line 66, in test_ang_vel
    B = A.orientnew('B', 'Axis', [q2, A.x])
  File "/home/peter/.local/lib/python3.5/site-packages/sympy/physics/vector/frame.py", line 640, in orientnew
    newframe.orient(self, rot_type, amounts, rot_order)
  File "/home/peter/.local/lib/python3.5/site-packages/sympy/physics/vector/frame.py", line 523, in orient
    [-axis[1], axis[0], 0]]) * sin(theta) + axis * axis.T)
  File "/home/peter/.local/lib/python3.5/site-packages/sympy/core/cache.py", line 93, in wrapper
    retval = cfunc(*args, **kwargs)
  File "/home/peter/.local/lib/python3.5/site-packages/sympy/core/function.py", line 375, in __new__
    result = super(Function, cls).__new__(cls, *args, **options)
  File "/home/peter/.local/lib/python3.5/site-packages/sympy/core/cache.py", line 93, in wrapper
    retval = cfunc(*args, **kwargs)
  File "/home/peter/.local/lib/python3.5/site-packages/sympy/core/function.py", line 199, in __new__
    evaluated = cls.eval(*args)
  File "/home/peter/.local/lib/python3.5/site-packages/sympy/functions/elementary/trigonometric.py", line 459, in eval
    if arg.could_extract_minus_sign():
  File "/home/peter/.local/lib/python3.5/site-packages/sympy/core/expr.py", line 2047, in could_extract_minus_sign
    negative_self = -self
  File "/home/peter/.local/lib/python3.5/site-packages/sympy/core/expr.py", line 113, in __neg__
    return Mul(S.NegativeOne, self)
  File "/home/peter/.local/lib/python3.5/site-packages/sympy/core/cache.py", line 93, in wrapper
    retval = cfunc(*args, **kwargs)
KeyError: (<class 'sympy.core.mul.Mul'>, 1, B_y, sin(2*q(t)), <class 'sympy.core.assumptions.ManagedProperties'>, <class 'sympy.core.numbers.One'>, <class 'sympy.physics.vector.frame.CoordinateSym'>, sin)
________________________________________________________________________________
______________ sympy/physics/vector/tests/test_frame.py:test_dcm _______________
  File "/home/peter/.local/lib/python3.5/site-packages/sympy/physics/vector/tests/test_frame.py", line 131, in test_dcm
    B = A.orientnew('B', 'Axis', [q2, A.x])
  File "/home/peter/.local/lib/python3.5/site-packages/sympy/physics/vector/frame.py", line 640, in orientnew
    newframe.orient(self, rot_type, amounts, rot_order)
  File "/home/peter/.local/lib/python3.5/site-packages/sympy/physics/vector/frame.py", line 523, in orient
    [-axis[1], axis[0], 0]]) * sin(theta) + axis * axis.T)
  File "/home/peter/.local/lib/python3.5/site-packages/sympy/core/cache.py", line 93, in wrapper
    retval = cfunc(*args, **kwargs)
  File "/home/peter/.local/lib/python3.5/site-packages/sympy/core/function.py", line 375, in __new__
    result = super(Function, cls).__new__(cls, *args, **options)
  File "/home/peter/.local/lib/python3.5/site-packages/sympy/core/cache.py", line 93, in wrapper
    retval = cfunc(*args, **kwargs)
  File "/home/peter/.local/lib/python3.5/site-packages/sympy/core/function.py", line 199, in __new__
    evaluated = cls.eval(*args)
  File "/home/peter/.local/lib/python3.5/site-packages/sympy/functions/elementary/trigonometric.py", line 459, in eval
    if arg.could_extract_minus_sign():
  File "/home/peter/.local/lib/python3.5/site-packages/sympy/core/expr.py", line 2050, in could_extract_minus_sign
    (negative_self).extract_multiplicatively(-1) is not None)
  File "/home/peter/.local/lib/python3.5/site-packages/sympy/core/expr.py", line 1859, in extract_multiplicatively
    quotient = self / c
  File "/home/peter/.local/lib/python3.5/site-packages/sympy/core/decorators.py", line 77, in __sympifyit_wrapper
    return func(a, b)
  File "/home/peter/.local/lib/python3.5/site-packages/sympy/core/decorators.py", line 118, in binary_op_wrapper
    return func(self, other)
  File "/home/peter/.local/lib/python3.5/site-packages/sympy/core/expr.py", line 161, in __div__
    return Mul(self, Pow(other, S.NegativeOne))
  File "/home/peter/.local/lib/python3.5/site-packages/sympy/core/cache.py", line 93, in wrapper
    retval = cfunc(*args, **kwargs)
KeyError: (<class 'sympy.core.mul.Mul'>, B_x, -cos(2*q(t)) + 1, <class 'sympy.core.assumptions.ManagedProperties'>, <class 'sympy.physics.vector.frame.CoordinateSym'>, <class 'sympy.core.add.Add'>)

=========== tests finished: 2 passed, 2 exceptions, in 1.57 seconds ============
DO *NOT* COMMIT!
False
>>> 



Sadly, I don't have a simplified test case to reproduce the error.  I would be happy to try some debugging suggestions.

Thanks,
Peter.
msg252097 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-10-02 09:42
Issue24483 fixes this issue.
msg252104 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-10-02 10:18
I don't know what was the actual cause of the error and if issue24483 fixed it or mask.
msg252126 - (view) Author: Peter Brady (Peter Brady) Date: 2015-10-02 15:10
As a sanity check I removed the stored hashvalue in Raymond Hettinger's backported lru_cache (which we use to support 2.6-3.2) and end up with errors as well.  So it seems like 24483 is the appropriate fix to restore the old behavior.  Thanks for looking into this.
msg252134 - (view) Author: Aaron Meurer (Aaron.Meurer) Date: 2015-10-02 15:56
Does this mean that some SymPy object is giving different hash values on successive calls to hash()? We definitely need to look into this on the SymPy side.
msg259797 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-02-07 18:25
Seems this issue can be closed now.
msg260047 - (view) Author: Peter Brady (Peter Brady) Date: 2016-02-10 22:25
Yes. The patch you suggested restores the old behavior of the lru_cache.
History
Date User Action Args
2016-02-10 22:25:40Peter Bradysetstatus: open -> closed
resolution: fixed
messages: + msg260047
2016-02-07 18:25:18serhiy.storchakasetmessages: + msg259797
2015-10-02 15:56:14Aaron.Meurersetmessages: + msg252134
2015-10-02 15:10:54Peter Bradysetmessages: + msg252126
2015-10-02 10:18:39serhiy.storchakasetmessages: + msg252104
2015-10-02 09:42:50serhiy.storchakasetdependencies: + Avoid repeated hash calculation in C implementation of functools.lru_cache()
messages: + msg252097
2015-10-02 08:14:41serhiy.storchakasetnosy: + rhettinger, serhiy.storchaka

type: behavior
versions: + Python 3.6
2015-10-02 05:13:48Aaron.Meurersetnosy: + Aaron.Meurer
2015-10-02 03:58:34Peter Bradycreate