Migrations¶
Generating the schema from the entities is enough for a prototype, but an
application that lives for a while needs a history: which changes were
applied, in what order, and how to apply the next one on a database that
already has data. That is what turso-orm-migration provides. This page
explains how migrations are declared, how the migrator applies them, why
each one is atomic, and how to run them.
How it works¶
The migrator keeps a bookkeeping table in your database,
turso_migrations(version TEXT PRIMARY KEY, applied_at INTEGER), with
one row per applied migration; migration_table_name() on the migrator
renames it, for a database shared by several migrators or migrated by
another tool before. Every operation starts by making sure that
table exists and reading it; then it walks your declared migrations in
order and runs the ones whose state does not match: up applies the
pending ones, down reverts the applied ones, newest first.
Each migration runs inside its own BEGIN IMMEDIATE transaction, and the
bookkeeping insert or delete goes through that same transaction before
the commit. Two consequences:
- Taking the write lock up front means a busy database fails at
BEGIN, before any schema change, rather than halfway through. - SQLite DDL is transactional, so if a migration fails, its tables, its indexes and its version row are all rolled back together. The database is left exactly as before, and the migrator can be run again after the fix. There is never a "version 3 applied but table missing" state.
Declaring a migration¶
A migration is a module whose name carries its version, holding a struct
that implements MigrationTrait. The DeriveMigrationName derive takes
the migration name from the module path, so every struct can simply be
called Migration and the file name is the single source of truth for
ordering:
//! Creates the `post` table.
use turso_orm_migration::prelude::*;
/// The migration.
#[derive(DeriveMigrationName)]
pub struct Migration;
#[async_trait]
impl MigrationTrait for Migration {
async fn up(&self, manager: &SchemaManager<'_>) -> Result<(), DbErr> {
manager
.create_table(
Table::create()
.table("post")
.if_not_exists()
.col(ColumnDef::integer("id").primary_key().auto_increment())
.col(ColumnDef::text("title").not_null())
.col(ColumnDef::text("text").not_null()),
)
.await
}
async fn down(&self, manager: &SchemaManager<'_>) -> Result<(), DbErr> {
manager.drop_table(Table::drop().table("post")).await
}
}
The naming convention is m<yyyymmdd>_<seq>_<what>: a date, a sequence
number for several migrations on the same day, and a description. Keep
the three parts: the migrator compares names as strings to know which
ones ran.
up receives a SchemaManager, a thin view over the migration's
transaction. It offers the DDL operations and a few catalog lookups:
| Method | What it does |
|---|---|
create_table(Table::create()...) |
Runs a CREATE TABLE built with the SQL builder. |
alter_table(Table::alter()...) |
ALTER TABLE, including Turso's ALTER COLUMN ... TO and column renames. |
drop_table(Table::drop()...) |
DROP TABLE. |
create_index(CreateIndex::new()...), drop_index(...) |
Indexes, including Turso's USING fts. |
has_table(name), has_column(table, column), has_index(table, name) |
Inspect the catalog through sqlite_schema and pragma_table_info. |
get_connection() |
The transaction itself, for anything else. |
down reverts what up did. It has a default implementation that fails
with DbErr::Migration, so a migration that cannot be reverted says so
explicitly instead of silently doing nothing; a down that is wrong is
worse than none.
Data migrations¶
Because get_connection() hands out the migration's transaction, a
migration can also move or seed data, and it benefits from the same
atomicity. The entity API works on it as on any connection:
//! Seeds two posts so that the API has something to list on first start.
use entity::post;
use entity::prelude::Post;
use turso_orm::prelude::*;
use turso_orm_migration::prelude::*;
/// The migration.
#[derive(DeriveMigrationName)]
pub struct Migration;
#[async_trait]
impl MigrationTrait for Migration {
async fn up(&self, manager: &SchemaManager<'_>) -> Result<(), DbErr> {
// The connection is the migration's own transaction, so the seed is
// rolled back together with the schema change if anything fails.
Post::insert_many([
post::ActiveModel {
title: Set("Hello, Turso".to_owned()),
text: Set("The first post, inserted by a migration.".to_owned()),
..Default::default()
},
post::ActiveModel {
title: Set("Second post".to_owned()),
text: Set("Also seeded.".to_owned()),
..Default::default()
},
])
.exec(manager.get_connection())
.await?;
Ok(())
}
async fn down(&self, manager: &SchemaManager<'_>) -> Result<(), DbErr> {
Post::delete_many()
.filter(post::Column::Title.is_in(["Hello, Turso", "Second post"]))
.exec(manager.get_connection())
.await?;
Ok(())
}
}
Two details are worth noting. The seed uses insert_many through the
entity, so the column list comes from the model and cannot drift from the
code. And down deletes exactly the rows up inserted, filtered by
title, rather than emptying the table, which may hold user data by then.
A migration and the entity it uses can drift
A migration refers to the entity as it is today, but the migration
describes the schema as it was then. If a later migration adds a
required column to post, this seed would start failing on a fresh
database. Data migrations through the entity are convenient early in a
project; for a schema that keeps changing, prefer explicit SQL or a
copy of the model frozen inside the migration module.
The migrator¶
The migrator is a type that lists the migrations, oldest first. That
list is the only thing to implement; up, down, status and the rest
come from MigratorTrait:
//! The migrator of the axum example and its migrations, oldest first.
//!
//! Each migration lives in a module named `m<date>_<seq>_<what>`; the
//! `DeriveMigrationName` derive takes the migration name from that module,
//! so every struct is simply called `Migration`.
pub use turso_orm_migration::prelude::*;
mod m20240101_000001_create_post_table;
mod m20240101_000002_seed_posts;
/// The migrator; `migrations` is the only method to implement.
pub struct Migrator;
#[async_trait]
impl MigratorTrait for Migrator {
fn migrations() -> Vec<Box<dyn MigrationTrait>> {
vec![
Box::new(m20240101_000001_create_post_table::Migration),
Box::new(m20240101_000002_seed_posts::Migration),
]
}
}
| Method | Effect |
|---|---|
up(db, None) / up(db, Some(n)) |
Applies every pending migration, or the first n. |
down(db, None) / down(db, Some(n)) |
Reverts every applied migration, or the last n, newest first. |
status(db) |
Each declared migration with an applied flag, in order. |
refresh(db) |
down everything, then up: rebuilds the schema through the migrations. |
fresh(db) |
Drops every table including the bookkeeping one, then up. For development databases. |
reset(db) |
down everything and stop. |
A migration that is in the bookkeeping table but no longer in the list is left alone; one that is in the list but not in the table is pending.
Running migrations¶
The migrator is a library: how and when it runs is your decision. The web
examples call Migrator::up(&db, None) at start-up, before the server
binds its port, which is right for a service that owns its database. They
also ship a small binary for operating by hand:
//! A minimal migration CLI: `up`, `down`, `status`, `fresh`, `refresh`, `reset`.
//!
//! The database path comes from `DATABASE_URL` in the environment or in the
//! `.env` file next to the workspace root, like the server.
//!
//! ```text
//! cargo run -p migration -- status
//! ```
#![allow(clippy::print_stdout, reason = "a CLI reports on stdout")]
use migration::{Migrator, MigratorTrait};
use turso_orm_migration::prelude::*;
/// Parses the single command argument and runs it against the database.
#[tokio::main]
async fn main() -> Result<(), DbErr> {
dotenvy::dotenv().ok();
let url = std::env::var("DATABASE_URL").unwrap_or_else(|_| "axum_example.db".to_owned());
let db = Database::connect(ConnectOptions::new(url)).await?;
let command = std::env::args().nth(1).unwrap_or_else(|| "up".to_owned());
match command.as_str() {
"up" => Migrator::up(&db, None).await?,
"down" => Migrator::down(&db, Some(1)).await?,
"fresh" => Migrator::fresh(&db).await?,
"refresh" => Migrator::refresh(&db).await?,
"reset" => Migrator::reset(&db).await?,
"status" => {}
other => {
eprintln!(
"unknown command `{other}`; expected up, down, fresh, refresh, reset or status"
);
std::process::exit(2);
}
}
for status in Migrator::status(&db).await? {
let mark = if status.applied { "applied" } else { "pending" };
println!("{mark:8} {}", status.name);
}
Ok(())
}
Running migrations from a dedicated binary rather than at start-up is the
better fit when several instances share one database file, or when a
human should see the status before anything changes.
Starting from the entities¶
The DDL builders and the entity schema generator produce the same statements, so the first migration of a project can be written in terms of the entities:
/// Creates the three tables and the index declared on `cake`, in dependency order.
///
/// Foreign keys are derived from the `belongs_to` relations and the DDL is
/// `STRICT`-compatible, so the generated statements are what a migration
/// would contain.
async fn create_schema(db: &Database) -> Result<(), DbErr> {
let schema = Schema::new();
for stmt in [
schema.create_table_from_entity(Bakery).to_statement(),
schema.create_table_from_entity(Cake).to_statement(),
schema.create_table_from_entity(Fruit).to_statement(),
] {
println!("{}", stmt.sql);
db.execute(stmt).await?;
}
for idx in schema.create_index_from_entity(Cake) {
let stmt = idx.to_statement();
println!("{}", stmt.sql);
db.execute(stmt).await?;
}
println!();
Ok(())
}
In a migration, each to_statement() would be executed through
manager.get_connection(). From the second migration on, write the
change explicitly: the entity now describes the target state, not the
step from the previous one.
Turso specifics
CREATE VIEW IF NOT EXISTS is not idempotent in Turso 0.8, and
PRAGMA defer_foreign_keys and foreign_key_check are unsupported.
Create referenced tables before the tables that reference them, and
drop them in the reverse order in down.