This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Immutable types inplace operations work incorrect in async
Type: behavior Stage: resolved
Components: asyncio, Interpreter Core Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, ay2306, bryzgaloff, rhettinger, yselivanov
Priority: normal Keywords:

Created on 2019-11-15 19:56 by bryzgaloff, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
scratch_1.py bryzgaloff, 2019-11-15 19:56 Example code
inplace_bug_threading.py bryzgaloff, 2019-11-15 20:11 Example code (threading version)
Messages (5)
msg356710 - (view) Author: Anton Bryzgalov (bryzgaloff) Date: 2019-11-15 19:56
Example code (also is attached to the issue):


import asyncio

async def count_smth(seconds: int) -> int:
    await asyncio.sleep(seconds)
    return seconds * 3

async def small_job(d: dict, key: str, seconds: int) -> None:
    d[key] += await count_smth(seconds)  # <-- strange happens here

async def main() -> None:
     d = dict(key=0)
     await asyncio.gather(
           small_job(d, 'key', 1),
           small_job(d, 'key', 2),
     )
     print(d['key'])  # expected: 0 + 1 * 3 + 2 * 3 = 9, actual: 6

if __name__ == '__main__':
    asyncio.run(main())


Expected output: 0 + 1 * 3 + 2 * 3 = 9. Actual: 6. Seems to be a race condition.

Same happens for other immutable types: str, tuple (when used as values of dict instead of int). But works correctly with list and custom class with __iadd__ method.
msg356714 - (view) Author: Anton Bryzgalov (bryzgaloff) Date: 2019-11-15 20:11
Example code (also is attached to the issue):


import asyncio

async def count_smth(seconds: int) -> int:
    await asyncio.sleep(seconds)
    return seconds * 3

async def small_job(d: dict, key: str, seconds: int) -> None:
    d[key] += await count_smth(seconds)  # <-- strange happens here

async def main() -> None:
     d = dict(key=0)
     await asyncio.gather(
           small_job(d, 'key', 1),
           small_job(d, 'key', 2),
     )
     print(d['key'])  # expected: 0 + 1 * 3 + 2 * 3 = 9, actual: 6

if __name__ == '__main__':
    asyncio.run(main())


Expected output: 0 + 1 * 3 + 2 * 3 = 9. Actual: 6. Seems to be a race condition.

Same happens for other immutable types: str, tuple (when used as values of dict instead of int). But works correctly with list and custom class with __iadd__ method.

Same effect for treading version (output is the same):


import time
import threading

def count_smth(seconds: int) -> int:
    time.sleep(seconds)
    return seconds * 3

def small_job(d: dict, key: str, seconds: int) -> None:
    d[key] += count_smth(seconds)  # <-- strange happens here

def main() -> None:
    d = dict(key=0)
    t1 = threading.Thread(target=small_job, args=(d, 'key', 1))
    t2 = threading.Thread(target=small_job, args=(d, 'key', 2))
    t1.start()
    t2.start()
    t1.join()
    t2.join()
    print(d['key'])  # expected: 0 + 1 * 3 + 2 * 3 = 9, actual: 6

main()


Inplace operation is expected to occur only at the time when the second operand is evaluated.
msg356729 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-11-15 23:44
> Inplace operation is expected to occur only at the time when 
> the second operand is evaluated.

That expectation does not match the implementation and it likely isn't possible to get that to work.   In-place operations on immutable objects necessarily create a new object, so the operation isn't as atomic as one might expect (the lookup and the assignment are two separate steps).
msg357588 - (view) Author: Ayush Mahajan (ay2306) * Date: 2019-11-28 00:35
Can I work on this issue?
msg357615 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-11-28 11:27
Raymond correctly spotted the problem.

I think we should just close the issue.
History
Date User Action Args
2022-04-11 14:59:23adminsetgithub: 82998
2019-11-28 11:27:46asvetlovsetstatus: open -> closed
resolution: not a bug
messages: + msg357615

stage: resolved
2019-11-28 00:35:44ay2306setnosy: + ay2306
messages: + msg357588
2019-11-15 23:44:49rhettingersetnosy: + rhettinger
messages: + msg356729
2019-11-15 20:11:08bryzgaloffsetfiles: + inplace_bug_threading.py

messages: + msg356714
2019-11-15 19:56:47bryzgaloffcreate