classification
Title: Allow retrieve the number of waiters pending for most of the asyncio lock primitives
Type: enhancement Stage: resolved
Components: asyncio Versions: Python 3.7
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: pfreixes, yselivanov
Priority: normal Keywords:

Created on 2017-05-24 14:48 by pfreixes, last changed 2017-10-19 17:39 by yselivanov. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 1786 closed pfreixes, 2017-05-24 14:52
Messages (6)
msg294357 - (view) Author: pfreixes (pfreixes) * Date: 2017-05-24 14:48
Currently, there is no way to access to the number of waiters pending to be woken up. This information can be useful for those environments which create and delete asyncio primitives instances depending if there are waiters still to be processed.

The following example shows an example of the DogPile solution that uses the Event lock mechanism. Each time that there is a miss in the cache, a new Event is created and it will be removed by the last waiter.


import asyncio

cache = {}
events = {}

async def get(process, key):
    try:
        return cache[key]
    except KeyError:
        try:
            await events[key].wait()
            if len(events[key]._waiters) == 0:
                events.pop(key)
            return cache[key]
        except KeyError:
            events[key] = asyncio.Event()
            # simulates some IO to get the Key
            await asyncio.sleep(0.1)
            cache[key] = "some random value"
            events[key].set()


async def main():
    tasks = [get(i, "foo") for i in range(1, 10)]
    await asyncio.gather(*tasks)

asyncio.get_event_loop().run_until_complete(main())
msg304093 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2017-10-10 23:09
Isn't this code equivalent to yours:

async def get(process, key):
    try:
        return cache[key]
    except KeyError:
        if key in events:
            await events[key].wait()
        else:
            events[key] = asyncio.Event()

            # simulates some IO to get the Key
            await asyncio.sleep(0.1)
            cache[key] = "some random value"
            event.set()

        return cache[key]


?
msg304109 - (view) Author: pfreixes (pfreixes) * Date: 2017-10-11 06:31
Not really, your code never removes the key from the events. If the cache is cleaned all further executions will keep forever in the `wait` statement. It is needed to create the Event again to perform at least one query to retrieve the cache value.
msg304110 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2017-10-11 06:35
> Not really, your code never removes the key from the events. If the cache is cleaned all further executions will keep forever in the `wait` statement. It is needed to create the Event again to perform at least one query to retrieve the cache value.

Right.  Well, that should be solved by adding "events.pop(key)" right after "events[key].set()"?
msg304111 - (view) Author: pfreixes (pfreixes) * Date: 2017-10-11 07:36
You are right, having the `pop` after the `set` it would remove the key from the events. Despite the workaround that you proposes is quite clean, it is kinda implicit.

The nature of the `pending` method is to give to the developer a way to check how many waiters are still pending. This not only helps to make this code more explicit also other pieces of code that might need access to the length of the waiters.
msg304624 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2017-10-19 17:39
> The nature of the `pending` method is to give to the developer a way to check how many waiters are still pending. This not only helps to make this code more explicit also other pieces of code that might need access to the length of the waiters.

This also is a perfect way to introduce races in your code.  I'm going to reject this, sorry.  I've demonstrated that it's possible to do what you wanted to do without adding new functionality.
History
Date User Action Args
2017-10-19 17:39:56yselivanovsetstatus: open -> closed
resolution: rejected
messages: + msg304624

stage: resolved
2017-10-11 07:36:55pfreixessetmessages: + msg304111
2017-10-11 06:35:02yselivanovsetmessages: + msg304110
2017-10-11 06:31:23pfreixessetmessages: + msg304109
2017-10-10 23:09:30yselivanovsetmessages: + msg304093
2017-05-24 14:52:49pfreixessetpull_requests: + pull_request1868
2017-05-24 14:48:25pfreixescreate