Forem: Rotem Tamir

Creating Terraform-like configuration languages with HCL and Go

Rotem Tamir — Sat, 06 Aug 2022 20:58:51 +0000

In the past year and a half, I've been working on Atlas, a database schema management tool that we're developing at Ariga. As part of this effort, I worked on implementing the infrastructure for the Atlas DDL, a data definition language that is the basis for Atlas's declarative style workflow for managing database schemas.

The Atlas language is based on HCL, a toolkit for creating configuration languages with a neat and simple information model and syntax. HCL was created at HashiCorp and used in popular tools such as Terraform and Nomad. We chose HCL as the basis of our configuration language for multiple reasons:

It has a base syntax that is clear and concise, easily readable by humans and machines.
Popularized by Terraform and other projects in the DevOps / Infrastructure-as-Code space, we thought it would feel familiar to practitioners which are the one of the core audiences for our tool.
It's written in Go, making it super easy to integrate with the rest of our codebase at Ariga.
It has great support for extending the basic syntax into a full-blown DSL using functions, expressions and context variables.

PCL: The Pizza Configuration Language

In the rest of this post, we will demonstrate how to create a basic configuration language using HCL and Go. To make this discussion entertaining, let's imagine that we are creating a new PaC (Pizza-as-Code) product that lets users define their pizza in simple HCL-based configuration files and send them as orders to their nearby pizza place.

Orders and Contacts

Let's start building our PaC configuration language by letting users define where they want their pizza delivered and who is the hungry contact waiting for the pizza to arrive. We're aiming for something like:

contact {
  name  = "Sherlock Holmes"
  phone = "+44 20 7224 3688"
}
address {
  street  = "221B Baker St"
  city    = "London"
  country = "England"
}

To capture this configuration, we will define a Go struct Order with sub-structs for capturing the Contact and Address:

type (
    Order struct {
        Contact *Contact `hcl:"contact,block"`
        Address *Address `hcl:"address,block"`
    }
    Contact struct {
        Name  string `hcl:"name"`
        Phone string `hcl:"phone"`
    }
    Address struct {
        Street  string `hcl:"street"`
        City    string `hcl:"city"`
        Country string `hcl:"country"`
    }
)

The Go HCL codebase contains two packages with a fairly high-level API for decoding HCL documents into Go structs: hclsimple (GoDoc)
and gohcl (GoDoc). Both packages rely on the user supplying Go struct field tags to map from the configuration file to the struct fields.

We will start the example by using the simpler one, with the surprising
name, hclsimple:

func TestOrder(t *testing.T) {
    var o Order
    if err := hclsimple.DecodeFile("testdata/order.hcl", nil, &o); err != nil {
        t.Fatalf("failed: %s", err)
    }
    require.EqualValues(t, Order{
        Contact: &Contact{
            Name:  "Sherlock Holmes",
            Phone: "+44 20 7224 3688",
        },
        Address: &Address{
            Street:  "221B Baker St",
            City:    "London",
            Country: "England",
        },
    }, o)
}

Pizza sizes and toppings (using static values)

Next, let's add the ability to order actual pizzas in our PaC application. To describe a pizza in our configuration language users should be able to do
something like:

pizza {
  size = XL
  count = 1
  toppings = [
    olives,
    feta_cheese,
    onions,
  ]
}

Notice that to make our API more explicit, users do not pass string values to the size or toppings field, and instead they use pre-defined, static identifiers (called "variables" in the HCL internal API) such as XL or feta_cheese.

To support this kind of behavior, we can pass an hcl.EvalContext (GoDoc),
which provides the variables and functions that should be used to evaluate an expression.

To construct this context we'll create this ctx() helper function:

func ctx() *hcl.EvalContext {
    vars := make(map[string]cty.Value)
    for _, size := range []string{"S", "M", "L", "XL"} {
        vars[size] = cty.StringVal(size)
    }
    for _, topping := range []string{"olives", "onion", "feta_cheese", "garlic", "tomatoe"} {
        vars[topping] = cty.StringVal(topping)
    }
    return &hcl.EvalContext{
        Variables: vars,
    }
}

To use it we need to add the pizza block to our top level Order struct:

type (
    Order struct {
        Contact *Contact `hcl:"contact,block"`
        Address *Address `hcl:"address,block"`
        Pizzas  []*Pizza `hcl:"pizza,block"`
    }
    Pizza struct {
        Size     string   `hcl:"size"`
        Count    int      `hcl:"count,optional"`
        Toppings []string `hcl:"toppings,optional"`
    }
    // ... More types ...
)

Here's our pizza block read using ctx() in action:

func TestPizza(t *testing.T) {
    var o Order
    if err := hclsimple.DecodeFile("testdata/pizza.hcl", ctx(), &o); err != nil {
        t.Fatalf("failed: %s", err)
    }
    require.EqualValues(t, Order{
        Pizzas: []*Pizza{
            {
                Size: "XL",
                Toppings: []string{
                    "olives",
                    "feta_cheese",
                    "onion",
                },
            },
        },
    }, o)
}

How many pizzas to order? (Using functions in HCL)

The final conundrum in any pizza delivery order is of course, how many pizzas to order. To help our users out with this riddle, let's level up our DSL and add the for_diners function that will take a number of diners and calculate for the user how many pizzas should be ordered. This will look something like:

pizza {
  size     = XL
  count    = for_diners(3)
  toppings = [
    tomato
  ]
}

Based on the universally accepted heuristic that one should order 3 slices per diner and round up, we can register the following function into our EvalContext:

func ctx() *hcl.EvalContext {
    // .. Variables ..

    // Define a the "for_diners" function.
    spec := &function.Spec{
        // Return a number.
        Type: function.StaticReturnType(cty.Number),
        // Accept a single input parameter, "diners", that is not-null number.
        Params: []function.Parameter{
            {Name: "diners", Type: cty.Number, AllowNull: false},
        },
        // The function implementation.
        Impl: func (args []cty.Value, _ cty.Type) (cty.Value, error) {
            d := args[0].AsBigFloat()
            if !d.IsInt() {
                return cty.NilVal, fmt.Errorf("expected int got %q", d)
            }
            di, _ := d.Int64()
            neededSlices := di * 3
            return cty.NumberFloatVal(math.Ceil(float64(neededSlices) / 8)), nil
        },
    }
    return &hcl.EvalContext{
        Variables: vars,
        Functions: map[string]function.Function{
          "for_diners": function.New(spec),
        },
    }
}

Testing the for_diners function out:

func TestDiners(t *testing.T) {
    var o Order
    if err := hclsimple.DecodeFile("testdata/diners.hcl", ctx(), &o); err != nil {
      t.Fatalf("failed: %s", err)
    }
    // For 3 diners, we expect 2 pizzas to be ordered.
    require.EqualValues(t, 2, o.Pizzas[0].Count)
}

Wrapping up

With these features, I think we can call it a day for this prototype of the world's first Pizza-as-Code product. As the source code for these examples is available on GitHub under an Apache 2.0
license, I truly hope someone picks this up and builds this thing!

In this post we reviewed some basic things you can do to create a configuration language for your users using HCL. There's a lot of other cool features we built into the Atlas language (such as input variables, block referencing and block polymorphism), so if you're interested so if you're interested in reading more about it feel free to ping me on Twitter.

Announcing Atlas Project Files

Rotem Tamir — Mon, 23 May 2022 20:35:17 +0000

A few days ago we released v0.4.1 of Atlas. Along with a multitude of improvements and fixes, I'm happy to announce the release of a feature that we've been planning for a while: Project Files.

Project files provide a way to describe and interact with multiple environments while working with Atlas. A project file is a file named atlas.hcl that contains one or more env blocks, each describing an environment. Each environment has a reference to where the schema definition file resides, a database URL and an array of the schemas in the database that are managed by Atlas:

// Define an environment named "local".
env "local" {
  // Declare where the schema definition file resides.
  src = "./schema/project.hcl"

  // Define the URL of the database which is managed in
  // this environment.
  url = "mysql://localhost:3306"

  // Define the URL of the Dev Database for this environment.
  // See: https://atlasgo.io/dev-database
  dev = "mysql://localhost:3307"

  // The schemas in the database that are managed by Atlas.
  schemas = ["users", "admin"]
}

env "dev" {
  // ... a different env
}

Project files arose from the need to provide a better experience for developers using the CLI. For example, consider you are using Atlas to plan migrations for your database schema. In this case, you will be running a command similar to this to plan a migration:

atlas migrate diff --dev-url mysql://root:password@localhost:3306 --to file://schema.hcl --dir file://migrations --format atlas

With project files, you can define an environment named local:

env "local" {
    url = "mysql://root:password@localhost:3306"
    dev = "mysql://root:password@localhost:3307"
    src = "./schema.hcl"
    migration_dir {
        url = "file://migrations"
        format = atlas
    }
}

Then run the migrate diff command against this environment using the --env flag:

atlas migrate diff --env local

Alternatively, suppose you want to use Atlas to apply the schema on your staging environment. Without project files, you would use:

atlas schema apply -u mysql://root:password@db.ariga.dev:3306 --dev-url mysql://root:password@localhost:3307 -f schema.hcl

To do the same using a project file, define another env named staging:

env "staging" {
  url = "mysql://root:password@db.ariga.dev:3306"
  dev = "mysql://root:password@localhost:3307"
  src = "./schema.hcl"
}

Then run:

atlas schema apply --env staging

Passing credentials as input values

Similar to schema definition files, project files also support Input Variables. This means that we can define variable blocks on the project file to declare which values should be provided when the file is evaluated. This mechanism can (and should) be used to avoid committing to source control database credentials. To do this, first define a variable named db_password:

variable "db_password" {
  type = string
}

Next, replace the database password in all connection strings with a reference to this variable, for example:

env "staging" {
  url = "mysql://root:${var.db_password}@db.ariga.dev:3306"
  dev = "mysql://root:${var.db_password}@localhost:3307"
  src = "./schema.hcl"
}

If we run schema apply without providing the password input variable, we will receive an error message:

Error: missing value for required variable "db_password"

To provide the input variable run:

atlas schema apply --env staging --var db_password=pass

Input variables can be used for many other use cases by passing them as input values to schema files.

What's next

In this post, I presented Project Files, a new feature recently added to Atlas to help developers create more fluent workflows for managing changes to their database schemas. In the coming weeks we will be adding a few more improvements to the dev flow, such as support for marking a specific environment as the default one (alleviating the need to specify --env in many cases) and multi-file schema definitions.

Have questions? Feedback? Find our team on our Discord server.

Programmatic inspection of databases in Go using Atlas

Rotem Tamir — Fri, 11 Feb 2022 11:35:02 +0000

Database inspection is the process of connecting to a database to extract metadata about the way data is structured inside it. In this post, we will present some use cases for inspecting a database, demonstrate why it is a non-trivial problem to solve, and finally show how it can be solved using Atlas, an open-source package (and command-line tool) written in Go that we are maintaining at Ariga.

As an infrastructure engineer, I have wished many times to have a simple way to programmatically inspect a database. Database schema inspection can be useful for many purposes. For instance, you might use it to create visualizations of data topologies, or use it to find table columns that are no longer in use and can be deprecated. Perhaps you would like to automatically generate resources from this schema (such as documentation or GraphQL schemas), or to use to locate fields that might carry personally-identifiable information for compliance purposes. Whatever your use case may be, having a robust way to get the schema of your database is the foundation for many kinds of infrastructure applications.

When we started working on the core engine for Atlas, we quickly discovered that there wasn't any established tool or package that could parse the information schema of popular databases and return a data structure representing it. Why is this the case? After all, most databases provide some command-line tool to perform inspection. For example, psql, the standard CLI for Postgres, supports the \d command to describe a table:

postgres=# \d users;
                       Table "public.users"
 Column |          Type          | Collation | Nullable | Default
--------+------------------------+-----------+----------+---------
 id     | integer                |           | not null |
 name   | character varying(255) |           |          |
Indexes:
    "users_pkey" PRIMARY KEY, btree (id)

So what makes inspection a non-trivial problem to solve? In this post, I will discuss two aspects that I think are interesting. The first is the variance in how databases expose schema metadata and the second is the complexity of the data model that is required to represent a database schema.

How databases expose schema metadata

Most of the SQL that we use in day-to-day applications is pretty standard. However, when it comes to exposing schemametadata, database engines vary greatly in the way they work. The way to retrieve information about things like available schemas and tables, column types and their default values and many other aspects of the database schema lookscompletely different in each database engine. For instance, consider this query (source) which can be used to get the metadata about table columns from a Postgres database:

SELECT t1.table_name,
       t1.column_name,
       t1.data_type,
       t1.is_nullable,
       t1.column_default,
       t1.character_maximum_length,
       t1.numeric_precision,
       t1.datetime_precision,
       t1.numeric_scale,
       t1.character_set_name,
       t1.collation_name,
       t1.udt_name,
       t1.is_identity,
       t1.identity_start,
       t1.identity_increment,
       t1.identity_generation,
       col_description(to_regclass("table_schema" || '.' || "table_name")::oid, "ordinal_position") AS comment,
       t2.typtype,
       t2.oid
FROM "information_schema"."columns" AS t1
         LEFT JOIN pg_catalog.pg_type AS t2
                   ON t1.udt_name = t2.typname
WHERE table_schema = $1
  AND table_name IN (%s)
ORDER BY t1.table_name, t1.ordinal_position

As you can see, while it's definitely possible to get the needed metadata, information about the schema is stored in multiple tables in a way that isn't particularly well documented, and often requires delving into the actual source code to understand fully. Here's a query to get similar information from MySQL (source):

SELECT `TABLE_NAME`,
       `COLUMN_NAME`,
       `COLUMN_TYPE`,
       `COLUMN_COMMENT`,
       `IS_NULLABLE`,
       `COLUMN_KEY`,
       `COLUMN_DEFAULT`,
       `EXTRA`,
       `CHARACTER_SET_NAME`,
       `COLLATION_NAME`
FROM `INFORMATION_SCHEMA`.`COLUMNS`
WHERE `TABLE_SCHEMA` = ?
  AND `TABLE_NAME` IN (%s)
ORDER BY `ORDINAL_POSITION`

While this query is much shorter, you can see that it's completely different from the one we ran to inspect Postgres column metadata. This demonstrates just one way in inspecting Postgres is difference from inspecting MySQL.

Mapping database schemas into a useful data structure

To be a solid foundation for building infrastructure, inspection must produce a useful data structure that can betraversed and analyzed to provide insights, in other words, a graph representing the data topology. As mentioned above, such graphs can be used to create ERD (entity-relation diagram) charts, such as the schema visualizations on the Atlas Management UI:

Let's consider some aspects of database schemas that such a data structure should capture:

Databases are split into logical schemas.
Schemas contain tables, and may have attributes (such as default collation).
Tables contain columns, indexes and constraints.
Columns are complex entities that have types, that may be standard to the database engine (and version) or custom data types that are defined by the user. In addition, Columns may have attributes, such as default values, that may be a literal or an expression (it is important to be able to discern between now() and "now()").
Indexes contain references to columns of the table they are defined on.
Foreign Keys contain references to column in other tables, that may reside in other schemas.
...and much, much more!

To capture any one of these aspects boils down to figuring out the correct query for the specific database engine you are working with. To be able to provide developers with a data structure that captures all of it, and to do it well across different versions of multiple database engines we've learned, is not an easy task. This is a perfect opportunity for an infrastructure project: a problem that is annoyingly complex to solve and that if solved well, becomes a foundation for many kinds of applications. This was one of our motivations for creating Atlas (GitHub) - an open-source project that we maintain here at Ariga.

Using Atlas, database schemas can be inspected to product Go structs representing a graph of the database schema topology. Notice the many cyclic references that make it hard to print (but very ergonomic to traverse :-)):

&schema.Realm{
    Schemas: {
        &schema.Schema{
            Name:   "test",
            Tables: {
                &schema.Table{
                    Name:    "users",
                    Schema:  &schema.Schema{(CYCLIC REFERENCE)},
                    Columns: {
                        &schema.Column{
                            Name: "id",
                            Type: &schema.ColumnType{
                                Type: &schema.IntegerType{
                                    T:        "int",
                                    Unsigned: false,
                                },
                                Null: false,
                            },
                        },
                    },
                    PrimaryKey: &schema.Index{
                        Unique: false,
                        Table:  &schema.Table{(CYCLIC REFERENCE)},
                        Attrs:  nil,
                        Parts:  {
                            &schema.IndexPart{
                                SeqNo: 0,
                                Desc:  false,
                                C:     &schema.Column{(CYCLIC REFERENCE)},
                            },
                        },
                    },
                },
                &schema.Table{
                    Name:    "posts",
                    Schema:  &schema.Schema{(CYCLIC REFERENCE)},
                    Columns: {
                        &schema.Column{
                            Name: "id",
                            Type: &schema.ColumnType{
                                Type: &schema.IntegerType{
                                    T:        "int",
                                    Unsigned: false,
                                },
                                Null: false,
                            },
                        },
                        &schema.Column{
                            Name: "author_id",
                            Type: &schema.ColumnType{
                                Type: &schema.IntegerType{
                                    T:        "int",
                                    Unsigned: false,
                                },
                                Null: true,
                            },
                        },
                    },
                    PrimaryKey: &schema.Index{
                        Unique: false,
                        Table:  &schema.Table{(CYCLIC REFERENCE)},
                        Parts:  {
                            &schema.IndexPart{
                                SeqNo: 0,
                                Desc:  false,
                                C:     &schema.Column{(CYCLIC REFERENCE)},
                            },
                        },
                    },
                    ForeignKeys: {
                        &schema.ForeignKey{
                            Symbol:  "owner_id",
                            Table:   &schema.Table{(CYCLIC REFERENCE)},
                            Columns: {
                                &schema.Column{(CYCLIC REFERENCE)},
                            },
                            RefTable:   &schema.Table{(CYCLIC REFERENCE)},
                            RefColumns: {
                                &schema.Column{(CYCLIC REFERENCE)},
                            },
                            OnDelete: "SET NULL",
                        },
                    },
                },
            },
        },
    },
}

Inspecting databases in Go using Atlas

While Atlas is commonly used as a CLI tool, all of Atlas's core-engine capabilities are available as a Go module that you can use programmatically. Let's get started with database inspection in Go:

To install Atlas, use:

go get ariga.io/atlas@master

Drivers

Atlas currently supports three core capabilities for working with SQL schemas.

"Inspection" - Connecting to a database and understanding its schema.
"Plan" - Compares two schemas and produces a set of changes needed to reconcile the target schema to the source schema.
"Apply" - creates concrete set of SQL queries to migrate the target database.

In this post we will dive into the inspection with Atlas. The way inspection is done varies greatly between the different SQL databases. Atlas currently has four supported drivers:

MySQL
MariaDB
PostgreSQL
SQLite

Atlas drivers are built on top of the standard library database/sql package. To initialize the different drivers, we need to initialize a sql.DB and pass it to the Atlas driver constructor. For example:

package tutorial
import (
    "database/sql"
    "log"
    "testing"
    _ "github.com/mattn/go-sqlite3"
    "ariga.io/atlas/sql/schema"
    "ariga.io/atlas/sql/sqlite"
)
func Test(t *testing.T) {
    // Open a "connection" to sqlite.
    db, err := sql.Open("sqlite3", "file:example.db?cache=shared&_fk=1&mode=memory")
    if err != nil {
        log.Fatalf("failed opening db: %s", err)
    }
    // Open an atlas driver.
    driver, err := sqlite.Open(db)
    if err != nil {
        log.Fatalf("failed opening atlas driver: %s", err)
    }
    // ... do stuff with the driver
}

Inspection

As we mentioned above, inspection is one of Atlas's core capabilities. Consider the Inspector interface in the sql/schema package:

package schema
// Inspector is the interface implemented by the different database
// drivers for inspecting multiple tables.
type Inspector interface {
    // InspectSchema returns the schema description by its name. An empty name means the
    // "attached schema" (e.g. SCHEMA() in MySQL or CURRENT_SCHEMA() in PostgreSQL).
    // A NotExistError error is returned if the schema does not exists in the database.
    InspectSchema(ctx context.Context, name string, opts *InspectOptions) (*Schema, error)
    // InspectRealm returns the description of the connected database.
    InspectRealm(ctx context.Context, opts *InspectRealmOption) (*Realm, error)
}

As you can see, the Inspector interface provides methods for inspecting on different levels:

InspectSchema - provides inspection capabilities for a single schema within a database server.
InspectRealm - inspects the entire connected database server.

Each database driver (for example MySQL, Postgres or SQLite) implements this interface. Let's see how we can use this interface by inspecting a "dummy" SQLite database. Continuing on the example from above:

package tutorial
func TestInspect(t *testing.T) {
    // ... skipping driver creation
    ctx := context.Background()
    // Create an "example" table for Atlas to inspect.
    _, err = db.ExecContext(ctx, "create table example ( id int not null );")
    if err != nil {
        log.Fatalf("failed creating example table: %s", err)
    }
    // Open an atlas driver.
    driver, err := sqlite.Open(db)
    if err != nil {
        log.Fatalf("failed opening atlas driver: %s", err)
    }
    // Inspect the created table.
    sch, err := driver.InspectSchema(ctx, "main", &schema.InspectOptions{
        Tables: []string{"example"},
    })
    if err != nil {
        log.Fatalf("failed inspecting schema: %s", err)
    }
    tbl, ok := sch.Table("example")
    require.True(t, ok, "expected to find example table")
    require.EqualValues(t, "example", tbl.Name)
    id, ok := tbl.Column("id")
    require.True(t, ok, "expected to find id column")
    require.EqualValues(t, &schema.ColumnType{
        Type: &schema.IntegerType{T: "int"}, // An integer type, specifically "int".
        Null: false,                         // The column has NOT NULL set.
        Raw:  "INT",                         // The raw type inspected from the DB.
    }, id.Type)
}

The full source-code for this example is available in the atlas-examples repo.

And voila! In this example, we first created a table named "example" by executing a query directly against the database. Next, we used the driver's InspectSchema method to inspect the schema of the table we created. Finally, we made some assertions on the returned schema.Table instance to verify that it was inspected correctly.

Inspecting using the CLI

If you don't want to write any code and just want to get a document representing your database schema, you can always use the Atlas CLI to do it for you. To get started, head over to the docs.

Wrapping up

In this post we presented the Go API of Atlas, which we initially built around our use case of building a new database migration tool, as part of the Operational Data Graph Platform that we are creating here at Ariga. As we mentioned in the beginning of this post, there are a lot of cool things you can build if you have proper database inspection, which raises the question, what will you build with it?

Getting involved with Atlas

Follow the Getting Started guide on the Atlas website.
Join our Discord Server.
Follow us on Twitter.