Dear lazy web...

I've had this code sitting around as a wtf.py for a while. I've been meaning to understand what's going on and write a blog post about it for a while, but I'm lacking the time. Now that I have a few minutes, I actually sat down to look at it and I think I figured it out:

from contextlib import contextmanager


@contextmanager
def bad():
    print('in the context manager')
    try:
        print("yielding value")
        yield 'value'
    finally:
        return print('cleaning up')


@contextmanager
def good():
    print('in the context manager')
    try:
        print("yielding value")
        yield 'value'
    finally:
        print('cleaning up')


with bad() as v:
    print('got v = %s' % v)
    raise Exception('exception not raised!')  # SILENCED!

print("this code is reached")
with good() as v:
    print('got v = %s' % v)
    raise Exception('expection normally raised')

print("NOT REACHED (expected)")

For those, like me, who need a walkthrough, here's what the above does:

  1. define a bad context manager (the things you use with with statements) with contextlib.contextmanager) which:

    1. prints a debug statement
    2. return a value
    3. then returns and prints a debug statement
  2. define a good context manager in much the same way, except it doesn't return, it just prints statement

  3. use the bad context manager to show how it bypasses an exception

  4. use the good context manager to show how it correctly raises the exception

The output of this code (in Debian 11 bullseye, Python 3.9.2) is:

in the context manager
yielding value
got v = value
cleaning up
this code is reached
in the context manager
yielding value
got v = value
cleaning up
Traceback (most recent call last):
  File "/home/anarcat/wikis/anarc.at/wtf.py", line 31, in <module>
    raise Exception('expection normally raised')
Exception: expection normally raised

What is surprising to me, with this code, is not only does the exception not get raised, but also the return statement doesn't seem to actually execute, or at least not in the parent scope: if it would, this code is reached wouldn't be printed and the rest of the code wouldn't run either.

So what's going on here? Now I know that I should be careful with return in my context manager, but why? And why is it silencing the exception?

The reason why it's being silenced is this little chunk in the with documentation:

If the suite was exited due to an exception, and the return value from the exit() method was false, the exception is reraised. If the return value was true, the exception is suppressed, and execution continues with the statement following the with statement.

This feels a little too magic. If you write a context manager with __exit__(), you're kind of forced to lookup again what that API is. But the contextmanager decorator hides that away and it's easy to make that mistake...

Credits to the Python tips book for teaching me about that trick in the first place.

return print??
I knew about the context manager behavior, what I don't know how to make heads of tails of is what do you expect from returning print("...")!
Comment by glacierre
comment 2
Frankly, I don't remember. The point is not really the return value of print here, the point is the return. I could have return "kumbaya" and it would have worked, and a plain return and it wouldn't have, the trick with return print() is that it's not obvious what the return value is for a newbie (it's None, of course, which is the whole point here).
Comment by anarcat
it's not about __exit__

Actually the returned value doesn't matter; I don't think contextmanager uses it at all (it uses its own exception suppression logic).

What actually suppresses the exception is this (from 8.4. The try statement):

If the finally clause executes a return, break or continue statement, the saved exception is discarded:

And this is why at least on my system nothing changes when I use return False or just return (and put the print on the line before).

Comment by stbuehler
comment 4

See this is exactly why this is a horrible gotcha and, to a certain extent, an API failure. Even looking at the docs, the two of us, we can't quite make up our minds as to what exactly is going on, because it's kind of a tangled mess.

I think you're right here: the contextmanager object doesn't change the try/except/finally semantics and return in the finally discards the exception.

But somehow because this is inside a contextmanager, I am led to imagine things act differently. I think part of the problem here is that the contextmanager docs do not explicitly state what ends up in the __exit__() code. It just says that you don't have to explicitly make it.

But yeah, you're right: the return value doesn't matter.

Comment by anarcat
Created . Edited .