Exceptions and Context Managers

The first version of a file reader looks like this: open the file, parse the numbers, compute something, print the result, and wrap the whole thing in try with except Exception at the bottom so that nothing crashes. It works, in the sense that the program keeps running. It also reports a typo in your own code as a problem with the file, and the first person to debug it starts from nothing.

Exceptions are not a feature you add at the end to stop crashes. They are how Python reports that a function could not do what its name says, and catching one is a decision to handle that failure here, with information the caller has and the library does not. The machinery is small: exception classes form a tree, except matches a branch of it, and with is a protocol with two methods that the interpreter calls at the start and end of a block.

Exception types are a hierarchy, and except matches a whole branch

Every exception class inherits from another one, which is what lets a single except clause cover a useful category of failures:

print(issubclass(KeyError, LookupError), issubclass(LookupError, Exception))
print(issubclass(FileNotFoundError, OSError))
print(KeyError.__mro__)
print(issubclass(KeyboardInterrupt, Exception))
print(issubclass(KeyboardInterrupt, BaseException))

True True
True
(<class 'KeyError'>, <class 'LookupError'>, <class 'Exception'>, <class 'BaseException'>, <class 'object'>)
False
True

KeyError inherits from LookupError, so except LookupError catches a missing dictionary key and a bad list index in one clause. FileNotFoundError inherits from OSError, so except OSError is the honest way to say "the operating system refused this" and covers permissions, missing paths and full disks together. That is the vocabulary worth learning: OSError for system calls, ValueError for a value of the right type with the wrong content, TypeError for the wrong type entirely, LookupError for a failed lookup.

Clauses are tested in order and the first match wins, so a subclass has to come before its parent. except Exception before except OSError makes the second clause unreachable, and Python does not warn you.

A bare except catches the signals that stop the program

except: with no type catches everything reachable, and the tree explains why that is worse than it sounds. KeyboardInterrupt and SystemExit do not inherit from Exception, so except Exception lets them through while a bare except swallows them. Press Ctrl-C during a long loop and the difference is whether the program stops:

try:
    raise KeyboardInterrupt
except Exception:
    print("caught by except Exception")
print("this line never runs")

Traceback (most recent call last):
  File "interrupt.py", line 2, in <module>
    raise KeyboardInterrupt
KeyboardInterrupt

The interpreter exits with status 130 and no frame from inside your handler, because no handler ran. Change one line to a bare except: and the same program prints its message and continues with status 0. A tool that catches Ctrl-C is a tool you cannot stop, and a service that catches SystemExit ignores the request to shut down.

except Exception is the broadest clause that still behaves, and it is still too broad for most code: it catches an AttributeError from your own typo as readily as a genuine network failure, and the last section shows how that goes wrong. Use it at the boundary of a long-running loop where one bad item must not kill the process, log the exception with its traceback, and name specific types everywhere else.

raise ... from ... keeps the original failure visible

Replacing a low-level error with a domain error is usually the right thing to do, because the caller knows what a config file is and has never heard of errno. What is not right is throwing the original away, because that error is the only record of what actually happened:

import json


class ConfigError(Exception):
    """A configuration file could not be loaded."""


def load_config(path):
    try:
        with open(path, encoding="utf-8") as handle:
            return json.load(handle)
    except FileNotFoundError as exc:
        raise ConfigError(f"no config file at {path}") from exc


load_config("missing.json")

Traceback (most recent call last):
  File "config.py", line 10, in load_config
    with open(path, encoding="utf-8") as handle:
         ~~~~^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'missing.json'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "config.py", line 16, in <module>
    load_config("missing.json")
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "config.py", line 13, in load_config
    raise ConfigError(f"no config file at {path}") from exc
ConfigError: no config file at missing.json

Two tracebacks, two jobs. The first names the file, the errno and the line that tried to open it. The second names your domain error and where your code raised it. Without from exc the two are joined by "During handling of the above exception, another exception occurred", which says only that the second error happened while the first was being handled. State the relationship instead: from exc sets __cause__ and prints "the direct cause", while raising without it sets __context__ and prints the vaguer message.

exc.__cause__ is the original FileNotFoundError, so a handler can still read what went wrong underneath. When the original is noise, raise ConfigError(...) from None removes it, deliberately and rarely.

Custom exceptions carry data, and one base class gives callers one name

A custom exception earns its place when it carries the data a handler needs. A message alone forces the caller to parse a string, which is how error handling turns into string matching:

class TaskLogError(Exception):
    """Base class for everything this module raises."""


class ParseError(TaskLogError):
    def __init__(self, line_number, line):
        super().__init__(f"line {line_number}: cannot parse {line!r}")
        self.line_number = line_number
        self.line = line


class UnknownOwnerError(TaskLogError):
    def __init__(self, owner):
        super().__init__(f"unknown owner {owner!r}")
        self.owner = owner


def parse_line(number, text):
    if "@" not in text:
        raise ParseError(number, text)
    title, owner = text.split("@", 1)
    if owner not in {"ana", "bo"}:
        raise UnknownOwnerError(owner)
    return title.strip(), owner


for number, text in enumerate(["ship it@ana", "no owner here"], start=1):
    try:
        print(parse_line(number, text))
    except TaskLogError as exc:
        print(type(exc).__name__, exc)
        if isinstance(exc, ParseError):
            print("  offending line:", exc.line)

('ship it', 'ana')
ParseError line 2: cannot parse 'no owner here'
  offending line: no owner here

Three decisions are worth copying. super().__init__(message) runs first, so a handler can print exc without touching the attributes. The base class lets a caller treat every task-log failure the same way with one except TaskLogError, while a caller that cares about validation names ParseError and reads exc.line_number. And the message names the failing input: 'no owner here' is more useful in a log than "unexpected token".

Keep the hierarchy shallow. One base class and a handful of concrete types is enough for most modules.

finally runs on every path out of the block

Cleanup that must happen whether the block succeeds, fails or returns early belongs in finally. The notes.txt below is a two-line file written earlier in the session:

def read_first_line(path):
    handle = open(path, encoding="utf-8")
    try:
        return handle.readline().strip()
    finally:
        handle.close()
        print("closed in finally; handle closed:", handle.closed)


def failing():
    try:
        raise ValueError("bad value")
    finally:
        print("finally still runs while the exception is in flight")


print("returned:", read_first_line("notes.txt"))
try:
    failing()
except ValueError as exc:
    print("caught outside:", exc)

closed in finally; handle closed: True
returned: first
finally still runs while the exception is in flight
caught outside: bad value

The first line printed before the returned value, which is the whole point: finally runs after return has decided what to return and before control leaves the function, so a break, a continue or a return inside the try does not skip it. The same happens while an exception is travelling outward, which is why the message appears before the exception is caught outside. One caution: a return inside a finally discards any exception in flight and overrides the earlier return value. The interpreter warns while compiling the file:

finally_return.py:5: SyntaxWarning: 'return' in a 'finally' block
  return "from finally"

That warning is easy to lose in a noisy test run, and the resulting function looks like it never raises. Keep finally for cleanup and put returns in the try.

with is a protocol, and __exit__ decides what happens to an exception

with exists because the try/finally above is easy to forget. Any object that implements two methods works with it: __enter__ returns what the as name binds to, and __exit__ is called when the block ends, including when it ends with an exception:

import time


class Timer:
    def __init__(self, label):
        self.label = label

    def __enter__(self):
        self.start = time.perf_counter()
        return self

    def __exit__(self, exc_type, exc, tb):
        self.elapsed = time.perf_counter() - self.start
        print(f"{self.label}: {self.elapsed:.3f}s, exc_type={exc_type}")
        return False


with Timer("sleep") as timer:
    time.sleep(0.05)
print("timer object survived the block:", round(timer.elapsed, 3))


class IgnoreMissing:
    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc, tb):
        return exc_type is not None and issubclass(exc_type, FileNotFoundError)


with IgnoreMissing():
    open("absent.txt")
print("carried on after a suppressed FileNotFoundError")

sleep: 0.054s, exc_type=None
timer object survived the block: 0.054
carried on after a suppressed FileNotFoundError

__exit__ receives the exception type, the exception instance and the traceback, or three Nones when the block completed. Its return value is a decision: a falsy value lets the exception continue, and a truthy value tells Python the exception has been handled, so execution resumes after the with statement. IgnoreMissing returns True only for FileNotFoundError, which is why the misspelled file did not raise while anything else would.

Two rules follow. __exit__ must not raise for the ordinary success case, and it must not return True carelessly, because a manager that swallows everything is a bare except with a friendlier face. Before writing one, check the standard library: contextlib.suppress is IgnoreMissing for a caller-chosen type.

contextlib.contextmanager builds that protocol from a generator

Most context managers have one cleanup step and no need for a class. contextlib.contextmanager turns a generator into a context manager: everything before yield is __enter__, the yielded value is what as binds to, and the code after yield is __exit__:

import contextlib


class FakeConnection:
    def __init__(self):
        self.actions = []

    def begin(self):
        self.actions.append("BEGIN")

    def commit(self):
        self.actions.append("COMMIT")

    def rollback(self):
        self.actions.append("ROLLBACK")

    def execute(self, statement):
        if "DROP" in statement:
            raise RuntimeError("statement rejected")
        self.actions.append(statement)


@contextlib.contextmanager
def transaction(conn):
    conn.begin()
    try:
        yield conn
    except Exception:
        conn.rollback()
        raise
    else:
        conn.commit()


conn = FakeConnection()
with transaction(conn) as active:
    active.execute("INSERT INTO tasks VALUES ('ship it')")
print("committed:", conn.actions)

conn = FakeConnection()
try:
    with transaction(conn) as active:
        active.execute("DROP TABLE tasks")
except RuntimeError as exc:
    print("failed with:", exc)
print("rolled back:", conn.actions)

committed: ['BEGIN', "INSERT INTO tasks VALUES ('ship it')", 'COMMIT']
failed with: statement rejected
rolled back: ['BEGIN', 'ROLLBACK']

The try/except/else inside the generator is not decoration. The exception is thrown into the generator at the yield point, so the except clause sees it, rolls back, and re-raises so the caller still finds out. The else clause runs only when the block completed without an exception, which is what makes "commit on success" a single place in the code rather than a flag the caller has to remember to set. Writing conn.commit() after the try would commit after a rollback as well.

One limitation: the generator body runs only when with is reached, so a manager that must do work at definition time has to be a class. contextmanager is also not reentrant, so one instance cannot serve two overlapping blocks.

A try block that is too large hides the bug it should report

This is the failure the lesson opened with. The reader below looks careful: it handles a missing file and computes something from the words it read.

import statistics


def describe(path):
    try:
        with open(path, encoding="utf-8") as handle:
            text = handle.read()
        words = text.split()
        lengths = [len(word) for word in words]
        return f"{path}: {len(words)} words, mean length {statistics.mean(lenghts):.1f}"
    except Exception:
        return f"{path}: unreadable"


print(describe("notes.txt"))

notes.txt: unreadable

The file exists and is readable, and notes.txt is not the problem. The miscount is a typo, lenghts instead of lengths, which raises NameError inside the try block, where except Exception is waiting. The error message says the file is unreadable, which sends the reader to the file system instead of the function.

Narrow the block to the two lines that can actually fail, and the typo reports itself:

def describe(path):
    try:
        with open(path, encoding="utf-8") as handle:
            text = handle.read()
    except OSError:
        return f"{path}: unreadable"
    words = text.split()
    lengths = [len(word) for word in words]
    return f"{path}: {len(words)} words, mean length {statistics.mean(lenghts):.1f}"


print(describe("notes.txt"))

Traceback (most recent call last):
  File "describe.py", line 15, in <module>
    print(describe("notes.txt"))
          ~~~~~~~~^^^^^^^^^^^^^
  File "describe.py", line 12, in describe
    return f"{path}: {len(words)} words, mean length {statistics.mean(lenghts):.1f}"
                                                                      ^^^^^^^
NameError: name 'lenghts' is not defined. Did you mean: 'lengths'?

The rule that produces this shape: the try block contains the smallest set of lines that can raise the exception you are catching, and the except names that type. Everything else moves out, where its failures are visible. The interpreter even underlines the exact expression these days, which is worth nothing if a broad handler stops the traceback from being printed at all.

The cost of a tight block is more of them, and that is the right trade: three small try blocks with named types describe failures better than one large block that catches everything.

from contextlib import contextmanager


class DataError(Exception):
    def __init__(self, path, line_number, line):
        super().__init__(f"{path} line {line_number}: not a number: {line!r}")
        self.path = path
        self.line_number = line_number
        self.line = line


@contextmanager
def opened(path):
    handle = open(path, encoding="utf-8")
    try:
        yield handle
    finally:
        handle.close()


def load_numbers(path):
    lines = []
    try:
        with opened(path) as handle:
            lines = handle.read().splitlines()
    except OSError as exc:
        raise DataError(path, 0, "") from exc

    numbers = []
    for number, line in enumerate(lines, start=1):
        try:
            numbers.append(int(line))
        except ValueError as exc:
            raise DataError(path, number, line) from exc
    return numbers

python3 lab12.py

good: [10, 20, 30]
bad failed: bad.txt line 2: not a number: 'twenty'
line: 2 of bad.txt
caused by: ValueError("invalid literal for int() with base 10: 'twenty'")