Modules, Packages, and Laying Out a Project

The single file works. A month later it is nine hundred lines, the functions at the top cannot be tested without running everything below them, and the sensible move is to split it. That is where the errors start: a ModuleNotFoundError, an ImportError naming a module you did not import, and a program that runs from one directory and not from another.

None of those are random. The import system is a short set of rules: a module is a file, importing executes it once and caches the result, and the interpreter searches a list of directories in order. Every import error is one of those rules reporting a disagreement between what you wrote and what is on disk.

A module is a file, and importing it executes it once

The module name is the filename without the .py suffix. import tasklog.store makes the interpreter look for tasklog/store.py, execute its top-level statements, and keep the resulting module object in sys.modules under the key "tasklog.store". A second import of the same name finds the cached object and returns it without executing anything again.

You can watch the execution happen. Put a print at the top of src/tasklog/__init__.py:

print("tasklog/__init__.py ran")

python3 -c "import tasklog.report"

tasklog/__init__.py ran

One line of output for an import of a different file, because importing tasklog.report imports the tasklog package first. Remove the print and run it again, and the import is silent.

The practical rule is that top-level code is code that runs whenever anything touches the module. Definitions, constants and imports belong there. File reads, network calls and prints belong in functions that the caller chooses to run.

__name__ decides whether a module is a program or a library

The interpreter sets one variable in every module it loads. For the module it was told to run, __name__ is the string "__main__". For a module reached through an import, it is the module's own name. The CLI in this project ends with the guard:

import sys

from .report import count_by_owner
from .store import load


def main(argv):
    tasks = load(argv[0])
    for owner, count in count_by_owner(tasks).items():
        print(f"{owner}: {count}")


if __name__ == "__main__":
    main(sys.argv[1:])

python3 -m tasklog.cli tasks.txt

ana: 2
bo: 1

python3 -c "import tasklog.cli; print(tasklog.cli.__name__)"

tasklog.cli

The first command runs the module, so the guard is true, so main runs. The second imports it, so __name__ is "tasklog.cli", so the guard is false and nothing is printed. The same file is a program and a library depending on how it was reached.

In a package, the guard answers a second question as well: which module owns main. Put main in cli.py with the guard, add a two-line __main__.py that calls it, and python3 -m tasklog works as well as python3 -m tasklog.cli. Without a __main__.py the package itself is an import target with nothing to run.

sys.path is the search list behind almost every import failure

The interpreter does not search the whole disk. It searches a list of directories in order, called sys.path, and the first match wins. Printing it is the fastest diagnostic you have.

python3 /home/ana/tasklog/show_path.py

/home/ana/tasklog

Where show_path.py prints sys.path[0]. For a script, the first entry is the directory containing that script, not the directory you are standing in. The rest of the list follows a fixed order:

PYTHONPATH=/home/ana/tasklog/src python3 -c "import sys; print(sys.path[:3])"

['', '/home/ana/tasklog/src', '/usr/lib/python3.14/python314.zip', ...]

Entry zero is the current directory when you run with -c or -m (as an empty string or an absolute path respectively), and the script's directory when you name a file. Then come the PYTHONPATH entries in the order given, then the standard library, then site-packages where installed packages live.

Two consequences are worth keeping. Installed packages lose to anything in your working directory, so a file called json.py next to your script shadows the standard library module. And a path that is not on the list cannot be found, however correct the import statement is. Printing the list answers most import questions in one line, and when your package's directory is missing from it, the fix belongs in the environment or the layout rather than in the import statement.

A module inside a package cannot be run as a script

This is the error that follows a successful split. The file works when imported and fails when named on the command line:

python3 src/tasklog/cli.py tasks.txt

Traceback (most recent call last):
  File "/home/ana/tasklog/src/tasklog/cli.py", line 3, in <module>
    from .report import count_by_owner
ImportError: attempted relative import with no known parent package

Both halves of the import system explain it. The script's directory, src/tasklog, became sys.path[0], so the search root is the package directory rather than the directory containing tasklog. And the module was run, so its __name__ is "__main__", which has no parent package for the leading dot to resolve against.

Run the same code as a module instead, from the project root, with the package importable:

PYTHONPATH=src python3 -m tasklog.cli tasks.txt

ana: 2
bo: 1

Nothing about the code changed. The module now has a package, so .report resolves, and the package is found through PYTHONPATH rather than through the working directory. Install the package and the same command works with no prefix at all. The habit worth forming: entry points are run with -m, and files inside a package are not scripts.

__init__.py no longer decides what can be imported

Since Python 3.3, a directory with no __init__.py is still importable. It becomes a namespace package, and the interpreter reports that clearly:

python3 -c "import plain; print(plain.__file__); print(list(plain.__path__))"

None
['/home/ana/tasklog/plain']

__file__ is None because there is no file to execute, while __path__ still lists the directory, so plain.mod imports normally. __init__.py is therefore not what makes a directory importable. What it does now is run once when the package is first imported, which makes it the place for package-level setup and a deliberate import surface:

"""Task log: count who owns what in a plain-text task list."""

from .store import load

__all__ = ["load"]

That gives callers from tasklog import load instead of from tasklog.store import load, which pays off when the internal layout changes. __all__ is what from tasklog import * reads; it does not restrict named imports, so it documents intent more than it enforces it.

One caution. The file runs for import tasklog.report as well as for import tasklog, so work placed there is paid on every import, and importing submodules in it is a common way to build the circular import below.

Absolute imports at the edges, relative imports inside

Absolute imports name the package, so they keep working when a file moves to a different depth, and they work from tests and scripts that sit outside the package. Relative imports say "the sibling module in whatever package I am part of", so the package can be renamed without editing them. Inside the package I use relative imports and keep them to one dot: a from ..other import thing that climbs a level usually means a layer is importing from the layer above it. In tests/ and in anything that might be run as an entry point I use absolute imports, because those files have no package context.

Circular imports are a dependency pointing both ways

Two modules that import each other produce a failure that looks unrelated to imports:

Traceback (most recent call last):
  File "/home/ana/circ/a.py", line 1, in <module>
    from b import render
  File "/home/ana/circ/b.py", line 1, in <module>
    from a import format_line
ImportError: cannot import name 'format_line' from 'a' (consider renaming '/home/ana/circ/a.py' if it has the same name as a library you intended to import)

Here is the sequence. a starts executing and imports b on line 1. b imports a, which is already in sys.modules but only half built, because its line 1 never finished, so there is no format_line on it yet. The parenthetical hint varies between releases, in this case suggesting a name clash that is not happening. Read the frames: the two files import each other, and the name was missing because its definition had not run.

Two fixes are standard. Move what both modules need into a third module that imports neither of them:

python3 -c "import a, b; print(a.show('hello')); print(b.show('hello'))"

* hello
[* hello]

Where formatting.py holds format_line and render, and both modules import from it. This removes the cycle instead of hiding it. The second fix moves the import inside the function that needs it, so the module finishes loading before the name is requested:

def show(text):
    from a_fixed import format_line

    return render(format_line(text))

That works and it is legitimate when the dependency genuinely runs one way at call time. It also makes the coupling invisible at the top of the file, so it is the fix to use when the alternative is a large refactor, not the fix to reach for first.

A layout that survives: src, tests and pyproject.toml

tasklog/
├── pyproject.toml
├── src/
│   └── tasklog/
│       ├── __init__.py
│       ├── __main__.py
│       ├── cli.py
│       ├── report.py
│       └── store.py
└── tests/
    └── test_report.py

The importable package lives under src/ so that nothing in the project root is importable by accident. Tests then have exactly two ways to find it: install the package, or tell the test runner where it is. Both force the import to happen the way it happens on a machine that installed your package, which is how a module left out of the build gets caught before a user finds it.

[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"

[project]
name = "tasklog"
version = "0.1.0"
requires-python = ">=3.12"

[tool.setuptools.packages.find]
where = ["src"]

[tool.pytest.ini_options]
pythonpath = ["src"]
testpaths = ["tests"]

One file carries the package metadata, the build backend and the tool settings. where = ["src"] tells the build which directory holds packages, so tests/ is never installed. pythonpath is read by pytest, which adds its paths to the front of sys.path; that is the alternative to pip install -e . while a project is small, and if your pytest does not read the key, use the editable install or PYTHONPATH instead.

python3 -m pytest -q

..                                                                       [100%]
2 passed in 0.00s

That run is CPython 3.14.7 with pytest 9.1.1 in a virtual environment, on the two tests in tests/test_report.py, which import tasklog.report and assert on count_by_owner. Module 5 covers what those tests should be. The layout is what makes them possible: the test imports the package by its real name, the way an installed copy would be imported.

What this layout does not do is pin dependency versions or decide how the service starts. Module 6 handles both. What it gives you now is a project where a new checkout needs one command to install and one to test, and where the import rules are the same in both.

import sys


def load(path):
    with open(path, encoding="utf-8") as handle:
        return [line.strip() for line in handle if line.strip()]


def count_by_owner(tasks):
    counts = {}
    for task in tasks:
        owner = task.rsplit("@", 1)[-1]
        counts[owner] = counts.get(owner, 0) + 1
    return counts


tasks = load(sys.argv[1])
for owner, count in count_by_owner(tasks).items():
    print(f"{owner}: {count}")

# src/tasklog/__main__.py
import sys

from .cli import main

main(sys.argv[1:])

# src/tasklog/cli.py
import sys

from .report import count_by_owner
from .store import load


def main(argv):
    tasks = load(argv[0])
    for owner, count in count_by_owner(tasks).items():
        print(f"{owner}: {count}")


if __name__ == "__main__":
    main(sys.argv[1:])

# src/tasklog/__init__.py
"""Task log: count who owns what in a plain-text task list."""

from .store import load

__all__ = ["load"]

python3 -m tasklog tasks.txt
python3 -m tasklog.cli tasks.txt

ana: 2
bo: 1
ana: 2
bo: 1