Implementing migrations into your Golang application?
This can be a source of some information while you search around for implementation details on how-to tackle it in Go systems.
Tools
Golang migrate tool: golang-migrate
PGX - driver for postgres: pgx
GoDotEnv - reading .env: godotenv
Project structure
cmd/
migrate/
main.go
migrations/
<generated migration files>
How to create a new migration file
While checking out golang-migrate repo you probably noticed that we need to install a CLI tool first so we have access to the tool from your local machine.
Lets break down what the migrate create does by reading the official help docs:
create [-ext E] [-dir D] [-seq] [-digits N] [-format] [-tz] NAME
Create a set of timestamped up/down migrations titled NAME, in directory D with extension E.
Use -seq option to generate sequential up/down migrations with N digits.
Use -format option to specify a Go time format string. Note: migrations with the same time cause "duplicate migration version" error.
Use -tz option to specify the timezone that will be used when generating non-sequential migrations (defaults: UTC).
-digits int
The number of digits to use in sequences (default: 6) (default 6)
-dir string
Directory to place file in (default: current working directory)
-ext string
File extension
-format string
The Go time format string to use. If the string "unix" or "unixNano" is specified, then the seconds or nanoseconds since January 1, 1970 UTC respectively will be used. Caution, due to the behavior of time.Time.Format(), invalid format strings will not error (default "20060102150405")
-seq
Use sequential numbers instead of timestamps (default: false)
-tz string
The timezone that will be used for generating timestamps (default: utc) (default "UTC")
Important one is
seq- this has some implications in how you want to track your migration sequence. If you are doing a solo project it doesn't matter that much... but with a team it actually does matter.If you for example deploy a migration with seq number 10 and teammate working in another branch deployed a migration yesterday with the same number 10... your migration wont run.
Then we can actually start with creating a new up and down migration sql files.
migrate create -ext sql -dir cmd/migrate/migrations -seq create_users
This will create 2 files:
cmd/migrate/migrations/000001_create_users.down.sql
cmd/migrate/migrations/000001_create_users.up.sql
How is migrations storing the data
Migrations when running the first time will create it's own table in the database called schema_migrations with 2 columns version(bigint) and dirty(boolean).
Version holds the information which version was last ran.
Dirty holds information about current state of the last migration ran.
You could make a sql mistake and you ran the migration... happens right?
Lets say you ran a migration with number 4 and you had a mistake in the sql.
The single row in the table would change to 4 (version) and true (dirty).
You would then have to manually change the version back to 3 and dirty to false. And retry! This time without any mistake hopefully :)
Running the migrations
Well we created the migrations - we know how the tool is tracking it - now we just need to write the script that will actually handle the changes.
For this example lets just create UP and DOWN migrations.
We need to edit our cmd/migrate/main.go which will contain all the logic.
You need to have defined DB_USER, DB_USER, DB_PASSWORD, DB_HOST, DB_PORT and DB_NAME in your .env file.
package main
import (
"context"
"database/sql"
"errors"
"fmt"
"log"
"os"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database/postgres"
_ "github.com/golang-migrate/migrate/v4/source/file"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/joho/godotenv"
)
func main() {
if err := godotenv.Load(); err != nil {
fmt.Println("No .env file found")
}
ctx := context.Background()
fmt.Println("Starting database migration process")
connectionString := fmt.Sprintf("postgres://%s:%s@%s:%s/%s?sslmode=disable",
os.Getenv("DB_USER"),
os.Getenv("DB_PASSWORD"),
os.Getenv("DB_HOST"),
os.Getenv("DB_PORT"),
os.Getenv("DB_NAME"))
dbConfig, err := pgxpool.ParseConfig(connectionString)
if err != nil {
log.Fatalf("Failed to parse database config: %v", err)
}
pool, err := pgxpool.NewWithConfig(
ctx,
dbConfig,
)
if err != nil {
log.Fatalf("Failed to create database connection pool: %v", err)
}
defer pool.Close()
fmt.Println("Database connection established successfully")
db, err := sql.Open("postgres", connectionString)
if err != nil {
log.Fatalf("Unable to connect to database: %v", err)
}
// Create migration instance
driver, err := postgres.WithInstance(db, &postgres.Config{})
if err != nil {
log.Fatal(err)
}
// Point to your migration files. Here we're using local files, but it could be other sources.
m, err := migrate.NewWithDatabaseInstance(
"file://cmd/migrate/migrations",
"postgres",
driver,
)
if err != nil {
log.Fatal(err)
}
cmd := os.Args[len(os.Args)-1]
if cmd == "up" {
if err := m.Up(); err != nil && !errors.Is(err, migrate.ErrNoChange) {
log.Fatalf("Migration up failed: %v", err)
}
fmt.Println("Migration up completed successfully")
}
if cmd == "down" {
if err := m.Down(); err != nil && !errors.Is(err, migrate.ErrNoChange) {
log.Fatalf("Migration down failed: %v", err)
}
fmt.Println("Migration down completed successfully")
}
}
That is it!
You can run it with the following commands:
Up:
go run cmd/migrate/main.go up
Down:
go run cmd/migrate/main.go down
You can put it in the makefile if you want to:
migrate-up: ## Database migration up
@go run cmd/migrate/main.go up
migrate-down: ## Database migration down
@go run cmd/migrate/main.go down
Hope it helps someone out!
Top comments (0)