Migrations & Upgrades

You change your models; you never write a migration. Running fullfinity-server -u all upgrades a database across any number of versions safely (it never loses data and never half-applies) and painlessly (no hand-written migration scripts for the common case, and custom modules survive core changes).

The guarantee is mechanical, not a matter of discipline: the schema diff is incapable of destroying data, a gate blocks silent breaking changes before they merge, and every upgrade runs in one transaction. (For the full rationale, see MIGRATION_UPGRADE_DESIGN.md at the repo root.)

What -u all does

A single -u all runs one atomic transaction — everything below commits together or rolls back together, so a failure never leaves a half-migrated database:

1. Additive schema sync. New columns, tables, and indexes are added; safe in-place ALTERs are applied. The diff never drops a column or table — so a field that merely disappeared from the code can’t silently destroy data.
2. Ledger replay. Recorded schema changes are applied in order, per installed module: a rename copies the old column’s data to the new column, then drops the old column; a delete drops its column; a complex change runs its transform hook.
3. Commit (or roll back the whole thing on any error).

So a rename neither loses data nor leaves clutter behind: it carries, then drops, in the same command. There is no separate “cleanup” step to run.

The append-only rule (and the gate)

Existing stored fields may not be renamed, retyped, or removed without recording the change. This is enforced by a static gate, so you can’t ship a silent break:

fullfinity-server --check-schema

--check-schema composes the product’s field surface from code (no database needed) and diffs it against the committed baseline (schema_baseline.json). It exits non-zero if any existing field was removed, renamed, or retyped — or if a complex change ships without a fixture test. It runs in CI and as a pre-commit hook:

git config core.hooksPath .githooks   # enable the local pre-commit gate

You never have to remember any of this — if you rename a field, the gate fails the build and tells you to resolve it.

Recording a change: --resolve-schema

When the gate flags a change, record it:

fullfinity-server --resolve-schema            # interactive
fullfinity-server --resolve-schema --assume-renames   # auto-record obvious renames

For each flagged field it asks one question — rename, delete, or complex — and writes an entry to the change ledger (schema_changes.yaml), then re-snapshots the baseline so the gate goes green:

• rename → the new field name; replay will carry the data.
• delete → the column is dropped on upgrade (the pre-upgrade backup + the atomic transaction are the safety net).
• complex → a transform hook (module.path:fn) plus a fixture test that proves it. The gate refuses a complex change without its fixture.

There is also a data entry for a one-time data pass that isn’t triggered by a schema change (a backfill, a storage reshape) — a hook (module.path:fn, e.g. in modules/<m>/migrations.py) + a fixture, replayed once per DB via the id clock. This is the home for what used to be a post_upgrade hook. There is one mechanism: the version-keyed upgrades/v{X}_{Y}.py / pre_upgrade/post_upgrade path has been retired, and fullfinity-server upgrade is just an alias for -u all.

Writing a complex transform

A complex hook is async def fn(env) that replay calls inside the upgrade transaction. For the common cases — changing a column’s type, or remapping values / a foreign key — use the set-based helpers in fullfinity.engine.migration_helpers (they scale to large tables and are one line, instead of hand-written DDL):

# invoicing/migrations.py  (a complex/data ledger entry references it as "module.path:fn")
from fullfinity.engine.migration_helpers import retype_column, remap_column

async def quantity_to_integer(env):
    # Change a column's type, transforming existing values (ALTER ... TYPE ... USING).
    await retype_column(env, "InvoiceLine", "quantity", "integer",
                        using="round(NULLIF(quantity, ''))::integer")

async def retarget_partner_fk(env):
    # Re-map a foreign key by matching a natural key (set-based UPDATE).
    await remap_column(env, "Order", "partner",
        using="(SELECT id FROM contact c WHERE c.legacy_ref = \"order\".partner_legacy)")

Then record it and write the fixture:

fullfinity-server --resolve-schema   # choose 'complex', point at the hook + a fixture test

The fixture seeds representative rows, runs the transform, and asserts the output — the input→output oracle the gate requires. (See fullfinity/modules/core/tests/test_schema_replay.py for worked examples of both helpers.)

using/where expressions are interpolated into SQL and must be trusted, code-authored — never user input.

The ledger is keyed by a monotonic change id, not a version — so a field renamed several times across many commits replays correctly, and a database jumping many versions at once applies exactly the changes it’s missing, in order. Each database tracks how far it has applied per module (Module.schema_applied_id), so changes are never re-applied, and a module that isn’t installed simply skips its entries.

Model-level changes

Whole-model changes are handled the same way as fields, one level up. The gate detects a model removed from the code (or a table rename) and routes it through --resolve-schema:

• rename a model → rename_model: replay runs ALTER TABLE old RENAME TO new. The data and every foreign key pointing at the table follow automatically — no copy needed. (If the model’s _table_name is explicit and unchanged, only the Python name changed and there’s nothing to do at the DB level.)
• delete a model → delete_model: the table is dropped (plain DROP, not CASCADE — if other tables still reference it the upgrade fails and rolls back, surfacing the dependency rather than cascading loss).
• split / merge → resolve as complex: a hook apportions or combines rows across tables (use remap_column and raw INSERT ... SELECT as needed), with a fixture.

--resolve-schema suggests a rename target by matching the removed model’s field set against newly-added models. As with fields, the diff itself never drops a table — model deletes happen only through a recorded ledger entry.

Scale (large tables)

Most upgrade operations are metadata-only and instant regardless of row count: adding a column, dropping a column, and renaming a table (ALTER TABLE … RENAME, used for model renames) don’t touch rows. The one operation that does touch rows is a field rename’s data carry — a single set-based UPDATE to copy the old column into the new one, then a metadata-only DROP. On a table with millions of rows that UPDATE is the cost (minutes + WAL, inside the upgrade transaction — fine for a maintenance window).

We deliberately do not rename the column in place (RENAME COLUMN), even though it would be O(1): renaming changes the column’s identity and invalidates the connection’s cached prepared statements, and inside the atomic upgrade transaction that error cannot be retried — it would abort the upgrade. Carry-then-drop keeps the new column’s identity stable, so it is cache-safe. For genuinely huge tables the scalable and cache-safe optimization is to batch the carry UPDATE into chunks (not yet implemented; add when a deployment’s table is large enough to need it). Type changes (ALTER COLUMN TYPE) rewrite the table and are inherently heavy at scale — they go through complex hooks, where the author can choose a batched or online strategy.

Custom modules

Because core never removes the fields a custom module depends on (the gate enforces it), and renames are recorded and replayed, a custom module’s references keep resolving across upgrades. View inheritance is anchored to field names and stable element anchors, so a reflowed core layout doesn’t orphan an extension.

Command reference

Command	Purpose	DB needed
fullfinity-server -u all	Upgrade: additive sync + ledger replay, atomic	yes
fullfinity-server --check-schema	Gate: fail on a breaking field-surface change	no
fullfinity-server --resolve-schema	Record a rename/delete/complex change	no
fullfinity-server --snapshot-schema	Regenerate the committed baseline	no

For framework contributors

When you change a model field:

• Add a field → nothing to do; -u adds the column.
• Rename / remove / retype a field → the gate fails; run --resolve-schema and commit the resulting ledger entry + updated baseline alongside your model change.
• Reshape data (split a field, change units) → resolve it as complex, write the transform hook and a fixture test asserting its input → output.

A data-survival test suite (fullfinity/modules/core/tests/test_schema_replay.py) proves on a real database that a rename carries data and drops the old column, that a delete drops, and that replay advances correctly — run with fullfinity-server test --module core -k replay.