classification
Title: Add a "no-op" (null) context manager to contextlib
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ncoghlan Nosy List: Albert.Zeyer, Alexander.Jones, DLitz, Jesse Bakker, Martin Blais, blais, daniel.urban, eric.araujo, georg.brandl, giampaolo.rodola, hniksic, michael.foord, ncoghlan, piotr.dobrogost, pitrou, r.david.murray, rhettinger, vstinner
Priority: normal Keywords: patch

Created on 2010-10-08 11:13 by hniksic, last changed 2017-11-23 00:27 by ncoghlan. This issue is now closed.

Files
File name Uploaded Description Edit
fl hniksic, 2010-10-08 11:13
nullcontext.patch hniksic, 2010-10-09 14:36
Pull Requests
URL Status Linked Edit
PR 4464 merged Jesse Bakker, 2017-11-19 17:46
Messages (43)
msg118181 - (view) Author: Hrvoje Nikšić (hniksic) Date: 2010-10-08 11:13
I find that I frequently need the "null" (no-op) context manager.  For example, in code such as:

with transaction or contextlib.null():
    ...

Since there is no easy expression to create a null context manager, we must resort to workarounds, such as:

if transaction:
    with transaction:
        ... code ...
else:
    ... duplicated code ...

(with the usual options of moving the duplicated code to a function—but still.)

Or by creating ad-hoc null context managers with the help of contextlib.contextmanager:

if transaction is None:
    transaction = contextlib.contextmanager(lambda: iter([None])()
with transaction:
    ...

Adding a "null" context manager would be both practical and elegant.  I have attached a patch for contextlib.py
msg118183 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-10-08 11:20
+1

Looks like a reasonable use case.
msg118184 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-10-08 11:21
Patch is missing tests and documentation.
msg118185 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-10-08 11:21
About your patch:
 - __enter__() might return self instead of None... i don't really know which choice is better. "with Null() as x:" works in both cases
 - __exit__() has no result value, "pass" is enough
 - I don't like "Null" name, I prefer "Noop" (NoOperation, NoOp, ...) or something else
msg118186 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-10-08 11:32
I also find the Null/_null/null affair confusing.
msg118187 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-10-08 11:33
@contextlib.contextmanager
def null():
    yield

Do we really need to add this to the stdlib?
Previous proposals to add an "identity function" or "no-op function" have always be refused. This one seems even less useful.
msg118189 - (view) Author: Hrvoje Nikšić (hniksic) Date: 2010-10-08 11:52
Thank you for your comments.

@Michael: I will of course write tests and documentation if there is indication that the feature will be accepted for stdlib.

@Antoine: it is true that a null context manager can be easily defined, but it does requires a separate generator definition, often repeated in different modules.  This is markedly less elegant than just using contextlib.null() in an expression.

I'm not acquainted with the history of identity function requests, but note that the identity function can be defined as an expression, using simply lambda x: x.  The equivalent expression that evaluates to a null context manager is markedly more convoluted, as shown in my report.

@Éric: The Null/_null/null distinction is an optimization that avoids creating new objects for something that is effectively a singleton.  It would be perfectly reasonable to define contextlib.null as Antoine did, but, this being stdlib, I wanted the implementation to be as efficient as (reasonably) possible.
msg118190 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-10-08 11:54
> @Antoine: it is true that a null context manager can be easily
> defined, but it does requires a separate generator definition, often
> repeated in different modules.  This is markedly less elegant than
> just using contextlib.null() in an expression.

But you can use mymodule.null() where mymodule is a module gathering
common constructs of yours.
msg118193 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2010-10-08 12:03
The difference here is the one pointed out in the original post: for a function, you usually only care about having a value, so if you don't want to call it, you can just swap in a None value instead. If you need an actual callable, then "lambda:None" fits the bill.

The with statement isn't quite so forgiving. You need a genuine context manager in order to preserve the correct structure in the calling code. It isn't intuitively obvious how to do that easily. While not every 3-line function needs to be in the standard library, sometimes they're worth including to aid discoverability as much as anything else.

However, I don't see the point in making it a singleton and the name should include the word "context" so it doesn't becoming ambiguous when referenced without the module name (there's a reason we went with contextlib.contextmanager over contextlib.manager).

Something like:

class nullcontext():
    """No-op context manager, executes block without doing any additional processing.

    Used as a standin if a particular block of code is only sometimes
    used with a normal context manager:

      with optional_cm or nullcontext():
          # Perform operation, using the specified CM if one is given
    """
    def __enter__():
        pass
    def __exit__(*exc_info):
        pass
msg118194 - (view) Author: Hrvoje Nikšić (hniksic) Date: 2010-10-08 12:04
That is what we are using now, but I think a contextlib.null() would be useful to others, i.e. that its use is a useful idiom to adopt.  Specifically I would like to discourage the "duplicated code" idiom from the report, which I've seen all too often.

The "closing" constructor is also trivial to define, but it's there for convenience and to promote the use of with statement over try/finally boilerplate.  The same goes here: you don't miss the null context manager when you don't have it; you invent other solutions.  But when it's already available, it's an elegant pattern.  In my experience, if they have to define it to get it, most people won't bother with the pattern and will retain less elegant solutions.
msg118195 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2010-10-08 12:09
Actually, the singleton idea isn't a bad one, but I'd go one step further and skip the factory function as well. So change that to be:

class NullContext():
   ... # as per nullcontext in my last message

nullcontext = NullContext()

(with the example in the docstring adjusted accordingly)
msg118196 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2010-10-08 12:13
If you can supply a full patch before the end of the month, we should be able to get this in for 3.2beta1 (currently scheduled for 31 October)
msg118200 - (view) Author: Hrvoje Nikšić (hniksic) Date: 2010-10-08 12:48
I considered using a variable, but I went with the factory function for two reasons: consistency with the rest of contextlib, and equivalence to the contextmanager-based implementation.

Another reason is that it leaves the option of adding optional parameters at a later point.  I know, optional parameters aren't likely for a "null" constructor, but still... it somehow didn't feel right to relinquish control.
msg118274 - (view) Author: Hrvoje Nikšić (hniksic) Date: 2010-10-09 14:36
Here is a more complete patch that includes input from Nick, as well as the patch to test_contextlib.py and the documentation.

For now I've retained the function-returning-singleton approach for consistency and future extensibility.
msg118406 - (view) Author: Hrvoje Nikšić (hniksic) Date: 2010-10-12 07:34
Is there anything else I need to do to have the patch reviewed and applied?

I am in no hurry since we're still using 2.x, I'd just like to know if more needs to be done on my part to move the issue forward.  My last Python patch was accepted quite some years ago, so I'm not closely familiar with the current approval process.
msg118409 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-10-12 10:40
Unless Nick has further feedback I think you've done all you need to, thanks.
msg118448 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2010-10-12 17:49
I'm with Antoine. Why not just do this in the context function itself?
I think it's more explicit and easier than reading the doc to figure out what nullcontext is supposed to do:


from contextlib import contextmanager

CONDITION = False

@contextmanager
def transaction():
    if not CONDITION:
        yield None
    else:
        yield ...

with transaction() as x:
    ...
msg118450 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-10-12 18:03
Because hardcoding a particular condition into a context manager is less flexible?  (I'm +0 on this thing myself, by the way.)
msg118453 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-10-12 18:24
Are you sure that this is useful enough to warrant inclusion in the standard lib?  I don't know of anyone else who has used the same idiom.  It seems crufty to me -- something that adds weight (mental burden and maintenance effort) without adding much value. I don't know that anyone actually needs this.
msg118599 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2010-10-13 22:49
To me, this is more a matter of conceptual completeness than one of practical utility (ala fractions.Fraction). That said, I *have* personally encountered the "I only sometimes want to wrap this code in a CM" situation, so it isn't completely impractical, either. Those two factors are enough to reach my threshold for it being worthwhile to declare "one obvious way to do it" through the contextlib module.

There is a possible alternative approach that may be more intuitive to use and read than nullcontext() though:

@contextmanager
def optional_cm(cm, *, use_cm=True): # See naming note below
    if cm is None or not use_cm:
        yield
    else:
        with cm:
            yield

The OP's original example would then look like:

with optional_cm(transaction):
    ...

I suspect readers would find it far easier to remember what optional_cm does than to learn to recognise the "or nullcontext()" idiom. It also plays better with nested context managers:

with optional_cm(sync_lock), optional_cm(db_transaction), \
     open(fname) as f:
    ...


Naming Note: I nearly suggested "optional_context" as a name for this, but realised that would be subtly misleading (suggesting PEP 377 style functionality that potentially skipped the statement body, rather than the intended semantics of skipping use of the CM)
msg118616 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2010-10-14 06:11
I like your latest suggestion, except for the name.  Given that we also have the (quite generic) "closing", what about just "optional"?
msg118626 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-10-14 07:00
> To me, this is more a matter of conceptual completeness 
> than one of practical utility ...

Nick, you don't seem to be truly sold on the need.
I'm -1 on adding this.  It's basically cruft.  If
it were published as an ASPN recipe, its uptake
would be nearly zero.

We need to focus on real problems in the standard 
library and provide solid solutions.  If weight
gets added to the standard lib, it needs to be
selective.
msg119514 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2010-10-24 13:52
I find Raymond's perspective persuasive in this case. Feel free to post either the original idea or my later variant as an ASPN cookbook recipe. (you could actually combine the two, and use NullContext as an implementation detail of an optional_cm() function)
msg152909 - (view) Author: Alexander Jones (Alexander.Jones) Date: 2012-02-08 21:05
Not having this as a standard idiom makes it very tempting to just do copy-paste coding as in hniksic's example. Who likes to invent their own library for generic language-supporting idioms?

What about an alternative of giving NoneType empty enter and exit methods? So instead of a 'null' CM you can just use "with None"?
msg153046 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-02-10 13:58
FWIW, it's likely I'll be adding contextlib.ContextStack (see [1]) for 3.3. While it's far from the primary use case, that API also serves as a "no-op" context manager (if you never register any contexts or callbacks, the __exit__ impl does nothing).

[1] http://contextlib2.readthedocs.org/en/latest/index.html#contextlib2.ContextStack
msg153049 - (view) Author: Alexander Jones (Alexander.Jones) Date: 2012-02-10 14:45
That's very reassuring. Thanks, Nick!
msg175656 - (view) Author: Dwayne Litzenberger (DLitz) Date: 2012-11-15 23:23
After seeing a context manager named like "TempfileIfNeeded(..., cond)", whole sole purpose is to handle the conditional case, I'm firmly +1 on this proposal.

It's much easier to just read "with Tempfile() if cond else nullcontext():" than to read through another level of indirection every time someone wanted some conditional logic on a context manager.

Is there any chance that this issue could be reopened?

Perhaps a more elegant solution would be to modify the "with" statement so that any object can be given to it (then we could just use None directly), but I suspect that would be a tad more controversial. ;)
msg175662 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-11-16 00:34
No, an empty ExitStack() instance already works fine as a no-op context manager in 3.3: http://docs.python.org/3/library/contextlib#simplifying-support-for-single-optional-context-managers

We're not going to add a dedicated one under a different name.
msg281177 - (view) Author: Martin Blais (Martin Blais) Date: 2016-11-18 22:59
I've been looking for this today; I would have used it.

Instead of an obvious (and explicit) null context manager, I had to read through this entire thread to eventually find out that I can use something called ExitStack(), which is designed for another use case.

When many users have to replicate the same boilerplate code time and time again, it's not cruft, it's just something that ought to be part of the stdlib. There are a number of such cases in the stdlib. I think nullcontext should be part of the included batteries Python aims to provide.
msg281180 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2016-11-18 23:30
ExitStack() already covers the "null ctx mgr" use case described in the first message. Original example:


with transaction or contextlib.null():
    ...


By using ExitStack:


with transaction or ExitStack():
    ...


You can push this further and do this, which is even more flexible:


with ExitStack() as stack:
    if condition:
        stack.enter_context(transaction)
    ...


So ExitStack really is better than the original proposal which could have made sense 6 years ago but not anymore.
msg281195 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-11-19 03:17
The problem Martin is referring to is the SEO one, which is that the top link when searching for either "null context manager python" or "no-op context manager python" is this thread, rather than the "Use ExitStack for that" recipe in the docs: https://docs.python.org/3/library/contextlib.html#simplifying-support-for-single-optional-context-managers

We unfortunately have exactly zero SEO experts working on the CPython documentation, so even when we provide specific recipes in the docs for solving particular problems, they aren't always easy for people to find.

I've at least added the "use contextlib.ExitStack()" note to the issue title here, so folks can find that without having to read through the whole comment thread.
msg281200 - (view) Author: Martin Blais (blais) * (Python committer) Date: 2016-11-19 05:51
Adding nullcontext = ExitStack in the source file would solve this problem in a single line of code.
msg281215 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-11-19 12:23
No, it wouldn't, as ExitStack() does far more than merely implement a null context.

It would be like adding "nulliterable = ()" as a builtin, rather than just telling people "If you need a null iterable, use an empty tuple".
msg281234 - (view) Author: Martin Blais (blais) * (Python committer) Date: 2016-11-19 22:30
Well that just echoes exactly what I originally thought, but somebody else said it was not needed because ExitStack already exists and could be used for that purpose.

If this were at work and/or it were all just to me, I'd just implement a brand new nullcontext and move on.
msg281264 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-11-20 14:16
Unfortunately, the redundancy doesn't buy enough to justify the permanent documentation and style guide cost of providing two ways to do exactly the same thing.
msg281556 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-11-23 12:17
It turns out that there's a variant on the "null context manager" idea that may *not* be redundant with ExitStack(), and hence could potentially counter the current rationale for not adding one.

Specifically, it relates to context managers like click.progressbar() that are designed to be used with an "as" clause:

    with click.progressbar(iterable) as myiter:
        for item in myiter:
            ...

At the moment, making that optional is a bit messy, since you need to do something like:

    with click.progressbar(iterable) as myiter:
        if not show_progress:
            myiter = iterable # Don't use the special iterator
        for item in myiter:
            ...

or:

    with ExitStack() as stack:
        if show_progress:
            myiter = stack.enter_context(click.progressbar(iterable))
        else:
            myiter = iter(iterable)
        for item in myiter:
            ...

or:

    @contextmanager
    def maybe_show_progress(iterable, show_progress)
        if show_progress:
            with click.progressbar(iterable) as myiter:
                yield myiter
        else:
            yield iter(iterable)

    with maybe_show_progress(iterable, show_progress) as myiter:
        for item in myiter:
            ...

The problem is that there's no easy way to say "return *this* value from __enter__, but otherwise don't do anything special".

With a suitably defined NullContext, that last approach could instead look more like:

    if show_progress:
       ctx = click.progressbar(iterable)
    else:
       ctx = NullContext(iter(iterable))

    with ctx as myiter:
        for item in myiter:
            ...
msg305951 - (view) Author: Albert Zeyer (Albert.Zeyer) Date: 2017-11-09 10:04
Note that this indeed seems confusing. I just found this thread by search for a null context manager. Because I found that in TensorFlow they introduced _NullContextmanager in their code and I wondered that this is not provided by the Python stdlib.
msg305952 - (view) Author: Hrvoje Nikšić (hniksic) Date: 2017-11-09 10:13
For what it's worth, we are still using our own null context manager function in critical code. We tend to avoid contextlib.ExitStack() for two reasons:

1) it is not immediately clear from looking at the code what ExitStack() means. (Unlike the "contextmanager" decorator, ExitStack is unfamiliar to most developers.)

2) ExitStack's __init__ and __exit__ incur a non-negligible overhead compared to a true do-nothing context manager.

It doesn't surprise me that projects like Tensor Flow introduce their own versions of this decorator. Having said that, I can also understand why it wasn't added. It is certainly possible to live without it, and ExitStack() is a more than acceptable replacement for casual use.
msg306001 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-11-10 01:35
Reopening this based on several years of additional experience with context managers since I wrote https://bugs.python.org/issue10049#msg119514 when originally closing it.

The version I'm now interested in adding is the one from https://bugs.python.org/issue10049#msg281556 - rather than being completely without behaviour, the null context manager should accept the value to be returned from the call to __enter__ as an optional constructor parameter (defaulting to None). That allows even context managers that return a value from __enter__ to be made optional in a relatively obvious way that doesn't involve fundamentally rearranging the code.

I think the overhead argument against the use of ExitStack() for this purpose also has merit (so I'd be curious to see relative performance numbers collected with perf), but it's not my main motive for changing my mind.
msg306002 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-11-10 01:39
Reverting to "Needs patch", as the currently attached patch is for the "No behaviour" variant that always returns None from __enter__().

(hniksic, would you still be willing to sign the Python CLA? If so, then your patch could be used as the basis for an updated implementation. Otherwise I'd advise anyone working on this to start from scratch)
msg306266 - (view) Author: Hrvoje Nikšić (hniksic) Date: 2017-11-15 12:21
I am of course willing to sign the CLA (please send further instructions via email), although I don't know how useful my original patch is, given that it caches the null context manager.
msg306770 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-11-23 00:23
New changeset 0784a2e5b174d2dbf7b144d480559e650c5cf64c by Nick Coghlan (Jesse-Bakker) in branch 'master':
bpo-10049: Add a "no-op" (null) context manager to contextlib (GH-4464)
https://github.com/python/cpython/commit/0784a2e5b174d2dbf7b144d480559e650c5cf64c
msg306771 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-11-23 00:27
Thanks to Jesse Bakker for the PR implementing this for 3.7!
History
Date User Action Args
2017-11-23 00:27:28ncoghlansetstatus: open -> closed
versions: + Python 3.7
messages: + msg306771

resolution: fixed
stage: patch review -> resolved
2017-11-23 00:23:31ncoghlansetmessages: + msg306770
2017-11-19 17:47:15Jesse Bakkersetnosy: + Jesse Bakker
2017-11-19 17:46:11Jesse Bakkersetstage: needs patch -> patch review
pull_requests: + pull_request4397
2017-11-19 16:59:08Jesse Bakkersetversions: + Python 3.8, - Python 3.2
2017-11-15 12:21:09hniksicsetmessages: + msg306266
2017-11-10 01:39:04ncoghlansetmessages: + msg306002
stage: patch review -> needs patch
2017-11-10 01:35:42ncoghlansetstatus: closed -> open
resolution: rejected -> (no value)
messages: + msg306001

title: Add a "no-op" (null) context manager to contextlib (Rejected: use contextlib.ExitStack()) -> Add a "no-op" (null) context manager to contextlib
2017-11-09 10:13:48hniksicsetmessages: + msg305952
2017-11-09 10:04:29Albert.Zeyersetnosy: + Albert.Zeyer
messages: + msg305951
2016-11-23 12:17:44ncoghlansetmessages: + msg281556
2016-11-20 14:16:55ncoghlansetmessages: + msg281264
2016-11-19 22:30:05blaissetmessages: + msg281234
2016-11-19 12:23:06ncoghlansetmessages: + msg281215
2016-11-19 05:51:44blaissetnosy: + blais
messages: + msg281200
2016-11-19 03:17:48ncoghlansetmessages: + msg281195
title: Add a "no-op" (null) context manager to contextlib -> Add a "no-op" (null) context manager to contextlib (Rejected: use contextlib.ExitStack())
2016-11-18 23:30:19giampaolo.rodolasetmessages: + msg281180
2016-11-18 22:59:00Martin Blaissetnosy: + Martin Blais
messages: + msg281177
2013-10-04 09:24:19piotr.dobrogostsetnosy: + piotr.dobrogost
2012-11-16 00:34:26ncoghlansetmessages: + msg175662
2012-11-15 23:23:47DLitzsetnosy: + DLitz
messages: + msg175656
2012-02-10 14:45:24Alexander.Jonessetmessages: + msg153049
2012-02-10 13:58:29ncoghlansetmessages: + msg153046
2012-02-08 21:05:15Alexander.Jonessetnosy: + Alexander.Jones
messages: + msg152909
2010-10-24 13:52:35ncoghlansetstatus: open -> closed
resolution: rejected
messages: + msg119514
2010-10-14 07:00:03rhettingersetmessages: + msg118626
2010-10-14 06:11:10georg.brandlsetnosy: + georg.brandl
messages: + msg118616
2010-10-13 22:49:10ncoghlansetmessages: + msg118599
2010-10-12 18:24:28rhettingersetnosy: + rhettinger
messages: + msg118453
2010-10-12 18:03:03r.david.murraysetmessages: + msg118450
2010-10-12 17:49:40giampaolo.rodolasetnosy: + giampaolo.rodola
messages: + msg118448
2010-10-12 10:40:01r.david.murraysetnosy: + r.david.murray
messages: + msg118409
2010-10-12 07:34:32hniksicsetmessages: + msg118406
2010-10-10 04:06:58ncoghlansetassignee: ncoghlan
2010-10-09 14:36:46hniksicsetfiles: + nullcontext.patch
keywords: + patch
messages: + msg118274
2010-10-08 18:26:41daniel.urbansetnosy: + daniel.urban
2010-10-08 12:48:32hniksicsetmessages: + msg118200
2010-10-08 12:13:27ncoghlansetmessages: + msg118196
2010-10-08 12:09:42ncoghlansetmessages: + msg118195
2010-10-08 12:04:38hniksicsetmessages: + msg118194
2010-10-08 12:03:20ncoghlansetmessages: + msg118193
2010-10-08 11:54:18pitrousetmessages: + msg118190
2010-10-08 11:52:03hniksicsetmessages: + msg118189
2010-10-08 11:33:54pitrousetnosy: + pitrou
messages: + msg118187
2010-10-08 11:32:13eric.araujosetversions: + Python 3.2
nosy: + eric.araujo

messages: + msg118186

stage: patch review
2010-10-08 11:21:49vstinnersetnosy: + vstinner
messages: + msg118185
2010-10-08 11:21:00michael.foordsetmessages: + msg118184
2010-10-08 11:20:27michael.foordsetnosy: + ncoghlan, michael.foord
messages: + msg118183
2010-10-08 11:17:05vstinnersettitle: Add the null context manager to contextlib -> Add a "no-op" (null) context manager to contextlib
2010-10-08 11:14:22hniksicsettype: enhancement
components: + Library (Lib)
2010-10-08 11:13:44hniksiccreate