This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Add `pop` function to remove context manager from (Async)ExitStack
Type: enhancement Stage:
Components: Library (Lib) Versions: Python 3.11, Python 3.10, Python 3.9, Python 3.8, Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: andreash, ncoghlan, serhiy.storchaka, vbrozik, yselivanov
Priority: normal Keywords:

Created on 2021-09-13 14:40 by andreash, last changed 2022-04-11 14:59 by admin.

Messages (5)
msg401703 - (view) Author: Andreas H. (andreash) * Date: 2021-09-13 14:40
Currently it is not possible to remove context managers from an ExitStack (or AsyncExitStack). 


Workarounds are difficult and generally do accesses implementation details of (Async)ExitStack.
See e.g. https://stackoverflow.com/a/37607405. It could be done as follows:


class AsyncExitStackWithPop(contextlib.AsyncExitStack):
    """Same as AsyncExitStack but with pop, i.e. removal functionality"""
    async def pop(self, cm):
        callbacks = self._exit_callbacks
        self._exit_callbacks = collections.deque()
        found = None
        while callbacks:
            cb = callbacks.popleft()
            if cb[1].__self__ == cm:
                found = cb
            else:
                self._exit_callbacks.append(cb)
        if not found:
            raise KeyError("context manager not found")
        if found[0]:
            return found[1](None,None,None)
        else:
            return await found[1](None, None, None)

The alternative is re-implementation of ExitStack with pop functionality, but that is also very difficult to get right (especially with exceptions). Which is probably the reason why there is ExitStack in the library at all.


So I propose to augment (Async)ExitStack with a `pop` method like above or similar to the above.


Use-Cases:

An example is a component that manages several connections to network services. 
During run-time the network services might need to change (i.e. some be disconnected and some be connected according to business logic), or handle re-connection events (ie. graceful response to network errors).
It is not too hard to imagine more use cases.
Essentially every case where dynamic resource management is needed and where single resources are managable with python context managers.
msg401708 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-13 16:22
If you can remove context managers in arbitrary order it is no longer a stack.

I think that it is just a wrong tool for your problem. Just maintain a collection of opened connections. Set or dict can be better that list or deque.
msg401722 - (view) Author: Andreas H. (andreash) * Date: 2021-09-13 19:42
I see your point. But even with `pop` or `remove` it is still a stack or stack-like. In the normal case the context managers are still released in reverse order as they were added. Order cannot be changed arbitrarily.

There is just the additional function of removing a single context manager prematurely(e.g. for graceful error recovery and such). 

I would perhaps say that a stack is the "wrong" solution to the problem of "programmatically combining context managers" [this is from the official documentaion] in the first place. I write wrong in quotes because it is of course not really wrong, as one wants the reverse exit order. But to adequately address the dynamic case one needs in my opinion the ability to prematurely remove context managers. Otherwise the use is limited.

Reimplemeting the desired functionality with dicts or lists does not seem appealing to me as the code will be 90% the same to ExitStack. It will then also make ExitStack obsolete. So why not integrate it there?

The unsymmetry of being able to add context managers but not being able to remove them also seems odd to me.
msg413477 - (view) Author: Václav Brožík (vbrozik) Date: 2022-02-18 11:08
There was an interesting discussion about this problem in 2015-12:
https://mail.python.org/archives/list/python-ideas@python.org/thread/Y3SLBJHJHAMBEZJWRXEJ5LE5JBVNE7XS/
(https://groups.google.com/g/python-ideas/c/l2FcErYbjfo - for the case the link above rots)

Certainly there are use cases for this functionality when you need to dynamically open multiple resources and close them when they are not needed any more. Keeping resources open till the very end of ExitStack is a bad practice which can lead to locking problems or overuse of resources.
msg413806 - (view) Author: Andreas H. (andreash) * Date: 2022-02-23 15:24
Inside the discussion an ExitPool class is sketched (https://mail.python.org/archives/list/python-ideas@python.org/message/66W55FRCYMYF73TVMDMWDLVIZK4ZDHPD/), which provides this removal of context managers.

What I learned is that this would have different cleanup mode (atexit style), as compared to present ExitStack cleanup (nested style). 

So contrary to what I was originally thinking ExitPool functionality would be close to, but not a strict superset of ExitStack functionality. Still such an ExitPool functionality would be extremely useful.
History
Date User Action Args
2022-04-11 14:59:49adminsetgithub: 89347
2022-02-23 15:24:26andreashsetmessages: + msg413806
2022-02-18 11:08:40vbroziksetmessages: + msg413477
2022-02-18 10:38:17vbroziksetnosy: + vbrozik
2021-09-13 19:42:26andreashsetmessages: + msg401722
2021-09-13 16:22:04serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg401708
2021-09-13 14:40:02andreashcreate