migrate

Database migrations written in Go. Use as CLI or import as library.

• Migrate reads migrations from sources and applies them in correct order to a database.
• Drivers are "dumb", migrate glues everything together and makes sure the logic is bulletproof. (Keeps the drivers lightweight, too.)
• Database drivers don't assume things or try to correct user input. When in doubt, fail.

Forked from mattes/migrate

forked repository

This is a fork from the original golang-migrate/migrate project, maintained by Lightning Labs.

The project was forked to speed up development for features required by Lightning Labs. All changes in this fork will also be suggested upstream as Pull Requests, so there is a chance the fork will eventually not be needed.

As long as the fork exists, the following procedure is proposed: New feature:

• Create a PR against the ll-fork branch.
• Once merged, a new tag should be pushed (see "tag naming" section below), updating the PR number in the tag name.

Rebasing with upstream:

• The master branch of the fork can always be fast-forwarded to match the upstream's master branch.
• If a new feature or bugfix is added to upstream's master, we'll want to rebase the ll-fork branch on top of the latest master branch.
• After rebasing, a new tag should be pushed (see "tag naming" section below), updating the first part of the tag name (upstream tag and commit).

tag naming

To make it very obvious what a tagged version of this fork contains in terms of the upstream changes (independent of any rebases of the ll-fork branch), the tags pushed from the ll-fork branch should be named as follows:

  v4.18.2-9023d66-fork-pr-1

Which consists of the following elements:

  v<last_upstream_tag>-<actual_upstream_commit_in_master>-fork-pr-<last_merged_pr_in_fork>

This naming schema allows us to track both rebases of the ll-fork as well as new features added to the fork.

Databases

Database drivers run migrations. Add a new database?

• PostgreSQL
• PGX v4
• PGX v5
• Redshift
• Ql
• Cassandra / ScyllaDB
• SQLite
• SQLite3 (todo #165)
• SQLCipher
• MySQL / MariaDB
• Neo4j
• MongoDB
• CrateDB (todo #170)
• Shell (todo #171)
• Google Cloud Spanner
• CockroachDB
• YugabyteDB
• ClickHouse
• Firebird
• MS SQL Server
• rqlite

Database URLs

Database connection strings are specified via URLs. The URL format is driver dependent but generally has the form: dbdriver://username:password@host:port/dbname?param1=true&param2=false

Any reserved URL characters need to be escaped. Note, the % character also needs to be escaped

Explicitly, the following characters need to be escaped: !, #, $, %, &, ', (, ), *, +, ,, /, :, ;, =, ?, @, [, ]

It's easiest to always run the URL parts of your DB connection URL (e.g. username, password, etc) through an URL encoder. See the example Python snippets below:

$ python3 -c 'import urllib.parse; print(urllib.parse.quote(input("String to encode: "), ""))'
String to encode: FAKEpassword!#$%&'()*+,/:;=?@[]
FAKEpassword%21%23%24%25%26%27%28%29%2A%2B%2C%2F%3A%3B%3D%3F%40%5B%5D
$ python2 -c 'import urllib; print urllib.quote(raw_input("String to encode: "), "")'
String to encode: FAKEpassword!#$%&'()*+,/:;=?@[]
FAKEpassword%21%23%24%25%26%27%28%29%2A%2B%2C%2F%3A%3B%3D%3F%40%5B%5D
$

Migration Sources

Source drivers read migrations from local or remote sources. Add a new source?

• Filesystem - read from filesystem
• io/fs - read from a Go io/fs
• Go-Bindata - read from embedded binary data (jteeuwen/go-bindata)
• pkger - read from embedded binary data (markbates/pkger)
• GitHub - read from remote GitHub repositories
• GitHub Enterprise - read from remote GitHub Enterprise repositories
• Bitbucket - read from remote Bitbucket repositories
• Gitlab - read from remote Gitlab repositories
• AWS S3 - read from Amazon Web Services S3
• Google Cloud Storage - read from Google Cloud Platform Storage

CLI usage

• Simple wrapper around this library.
• Handles ctrl+c (SIGINT) gracefully.
• No config search paths, no config files, no magic ENV var injections.

CLI Documentation (includes CLI install instructions)

Basic usage

$ migrate -source file://path/to/migrations -database postgres://localhost:5432/database up 2

Docker usage

$ docker run -v {{ migration dir }}:/migrations --network host migrate/migrate
    -path=/migrations/ -database postgres://localhost:5432/database up 2

Use in your Go project

• API is stable and frozen for this release (v3 & v4).
• Uses Go modules to manage dependencies.
• To help prevent database corruptions, it supports graceful stops via GracefulStop chan bool.
• Bring your own logger.
• Uses io.Reader streams internally for low memory overhead.
• Thread-safe and no goroutine leaks.

Go Documentation

import (
    "github.com/golang-migrate/migrate/v4"
    _ "github.com/golang-migrate/migrate/v4/database/postgres"
    _ "github.com/golang-migrate/migrate/v4/source/github"
)

func main() {
    m, err := migrate.New(
        "github://mattes:personal-access-token@mattes/migrate_test",
        "postgres://localhost:5432/database?sslmode=enable")
    m.Steps(2)
}

Want to use an existing database client?

import (
    "database/sql"
    _ "github.com/lib/pq"
    "github.com/golang-migrate/migrate/v4"
    "github.com/golang-migrate/migrate/v4/database/postgres"
    _ "github.com/golang-migrate/migrate/v4/source/file"
)

func main() {
    db, err := sql.Open("postgres", "postgres://localhost:5432/database?sslmode=enable")
    driver, err := postgres.WithInstance(db, &postgres.Config{})
    m, err := migrate.NewWithDatabaseInstance(
        "file:///migrations",
        "postgres", driver)
    m.Up() // or m.Steps(2) if you want to explicitly set the number of migrations to run
}

Getting started

Go to getting started

Tutorials

• CockroachDB
• PostgreSQL

(more tutorials to come)

Migration files

Each migration has an up and down migration. Why?

1481574547_create_users_table.up.sql
1481574547_create_users_table.down.sql

Best practices: How to write migrations.

Coming from another db migration tool?

Check out migradaptor. Note: migradaptor is not affiliated or supported by this project

Versions

Version	Supported?	Import	Notes
master	✅	import "github.com/golang-migrate/migrate/v4"	New features and bug fixes arrive here first
v4	✅	import "github.com/golang-migrate/migrate/v4"	Used for stable releases
v3	❌	import "github.com/golang-migrate/migrate" (with package manager) or import "gopkg.in/golang-migrate/migrate.v3" (not recommended)	DO NOT USE - No longer supported

Development and Contributing

Yes, please! Makefile is your friend, read the development guide.

Also have a look at the FAQ.

---

Looking for alternatives? https://awesome-go.com/#database.