Migrations

A library's tables live in a database the library does not control. The host may have added columns, indexes, triggers or foreign keys to them; it may use binary_id keys, a schema prefix, several prefixes, MySQL or SQLite, a table of fifty million rows, and rolling deploys where old and new code run against the same schema. PhoenixMountable.Migrations is built for that.

It needs ecto_sql, and no Phoenix: a core package uses it on its own.

The library side

defmodule MyLib.Migrations do
  use PhoenixMountable.Migrations,
    otp_app: :my_lib,
    versions: [MyLib.Migrations.V1, MyLib.Migrations.V2, MyLib.Migrations.V3],
    install_options: [primary_key: [type: {:in, [:id, :binary_id]}, default: :id]]
end

defmodule MyLib.Migrations.V1 do
  use PhoenixMountable.Migration, description: "Items"

  def up(ctx) do
    create table(:my_lib_items, primary_key: primary_key(ctx), prefix: ctx.prefix) do
      add :title, :string
    end

    create index(:my_lib_items, [:title], prefix: ctx.prefix)
  end

  def down(ctx), do: drop(table(:my_lib_items, prefix: ctx.prefix))
end

Each version is a PhoenixMountable.Migration, with the Ecto.Migration DSL. It receives a PhoenixMountable.Migrations.Context: the repo, the adapter (:postgres, :mysql, :sqlite), the prefix, and the install options. Match on the adapter where the SQL differs:

def up(%{adapter: :sqlite} = ctx), do: ...
def up(ctx), do: ...

The host side

The host owns one pinned line per upgrade:

defmodule MyApp.Repo.Migrations.AddMyLib do
  use Ecto.Migration

  def up, do: MyLib.Migrations.up(version: 3)
  def down, do: MyLib.Migrations.down(to: 0)
end

mix phoenix_mountable.gen.migration MyLib.Migrations writes it (or a library's own mix my_lib.gen.migration): it reads the version the host's migrations pin, and writes the next migration with the latest version, the attributes it needs, and notes about the versions. The file is created by mix ecto.gen.migration, so it follows the host's conventions.

Pinning keeps history reproducible: a developer or CI running every migration on an empty database gets the same sequence production did. An unpinned up() changes meaning when the dependency updates.

Tracking

The version is kept in a phoenix_mountable_schema table in the library's prefix, written in the transaction of each version. It works on every database (table comments do not exist on SQLite and are lost by pg_dump --no-comments), per prefix (two instances in two schemas track separately), and records the install options.

Install options

Options chosen once, at install, that later versions need:

MyLib.Migrations.up(version: 3, primary_key: :binary_id)

They are stored, and every later version sees them in ctx, so a version written years later adds a foreign key of the right type (key_type(ctx)). A :primary_key option defaults to the repo's :migration_primary_key.

Existing installs

A library moving onto PhoenixMountable declares how to read the version of a database migrated before:

use PhoenixMountable.Migrations, ..., legacy: MyLib.Migrations.Legacy

defmodule MyLib.Migrations.Legacy do
  alias PhoenixMountable.Migrations.SQL

  # From Oban-style table comments, or from the columns present.
  def detect(ctx) do
    if SQL.table_exists?(ctx.repo, ctx.adapter, ctx.prefix, "my_lib_items"), do: 2
  end
end

On the first run the detected version is recorded and only later versions run.

Baselines

A library with many versions can ship a baseline creating the latest schema in one step, for fresh installs:

defmodule MyLib.Migrations.Baseline do
  use PhoenixMountable.Migration, baseline: 12
  def up(ctx), do: ...
end

PhoenixMountable.Migrations.Test.assert_baseline_matches/3 proves it gives the same schema as running every version.

Rolling deploys

Each version is safety: :expand (the default: new tables, nullable columns, new indexes) or safety: :contract (dropping, renaming, changing types). A compile-time lint warns about destructive operations in an expand version.

compatible_with: n is the oldest schema version the library's code still runs against. A rename is three releases: add the new column and write both, backfill, then drop the old one in a contract version. The generator notes contract versions, so hosts deploy the code first.

Large tables

create_concurrently(index(...)) builds an index without locking writes on Postgres (and normally elsewhere). Such a version cannot run in a transaction; the generator adds @disable_ddl_transaction true and @disable_migration_lock true to the host migration, and running it in a transaction fails with those lines.
lock_timeout ("5s" by default) makes a version fail fast rather than queue behind a long transaction and block the app.
Backfills move data in batches, apart from the schema change:

defmodule MyLib.Backfills.Slugs do
  use PhoenixMountable.Backfill,
    table: "my_lib_items",
    where: "slug IS NULL",
    set: "slug = LOWER(title)",
    batch_size: 1_000
end

def up(ctx) do
  alter table(:my_lib_items, prefix: ctx.prefix), do: add(:slug, :string)
  backfill(ctx, MyLib.Backfills.Slugs)
end

Backfills run inline by default. With mode: :deferred, or up(version: n, backfills: :deferred) from the host, they run after the deploy in PhoenixMountable.Backfill.Runner (or mix phoenix_mountable.backfill), batch by batch, recording a cursor after each, so they resume after a restart. A version that needs the data declares requires_backfills: [MyLib.Backfills.Slugs] and refuses to run before the backfill finishes.

Host changes

Indexes have deterministic names; create_index_if_missing/1, drop_index_if_present/1 and add_column_if_missing/5 work on every adapter (MySQL has no IF NOT EXISTS for them), so a version runs again after a partial failure (MySQL DDL is not transactional).
SQLite cannot alter a column, so rebuild_table/3 recreates the table, copying every column, index and trigger, including the host's, and refusing tables with CHECK constraints rather than losing them.
verify/1 compares the tables with a snapshot written by the test kit, ignoring what the host added.

Checks

With schema: MyLib.Migrations in the definition module, the first use of each mount compares the database with the library:

below compatible_with: raises PhoenixMountable.Migrations.OutdatedError with the command to run;
between: logs a warning once, or raises with config :phoenix_mountable, pending_migrations: :raise (set it in the host's test config so CI catches a forgotten upgrade).

mix phoenix_mountable.check checks every library in CI.

Testing

test "migrations" do
  PhoenixMountable.Migrations.Test.run_suite(MyLib.Repo, MyLib.Migrations, prefix: "my_lib_test")
end

runs every version up, down and up again, checks the baseline, upgrades from each version, and adds a column and an index to every table after each version (as a host might) to check that later versions keep them. Run it against each database the library supports.

← Previous Page Assets

Next Page → Processes