I would like to see this posted as a recipe before being put in the standard library.  It needs a chance to mature and to demonstrate that people will want to use it.

FWIW, the example is problematic in a couple of ways.  The registration of shutil.rmtree(tmpdir) will run *before* mngr register is called.  

Also, it doesn't take advantage of any of the with-statement features.  It doesn't show any advantage over a standard try/finally which is arguably cleaner:

    tmpdir = tempfile.mkdtemp()
    try:
        # do stuff with tmpdir
    finally:
        shutil.rmtree()

Also, I suspect that the CleanupManager would be an error-prone construct because the registration occurs somewhere after the with-statement is set-up, possibly resulting in errors if there is an early, pre-registration failure.