Forem: Emil Privér

Running schema/database migrations using Geni

Emil Privér — Wed, 17 Apr 2024 13:53:01 +0000

A while ago, I developed Geni, a CLI database migration tool. The goal of the app was to make migrations to Tursos databases easier. When I developed Geni, I also decided to add support for Postgres, MariaDB, MySQL, and SQLite. The goal of this post is to describe what database/schema migrations are and how to perform them via Geni.

A schema within a database describes its current structure, which includes elements such as tables, indexes, and constraints. When you connect to a PostgreSQL database and run \\d+, you'll see something like this:

The image above shows the current structure of my job table. Essentially, migrations are used to modify the database structure. In other words, a migration is a set of instructions for making changes to your database.

Suppose you receive a new task at work that requires you to add a table and a new column to another table. One approach is to connect to the database and make these changes manually. Alternatively, you could use a tool, such as Geni, for this task.

Using Geni offers several advantages. It provides a reproducible database structure, useful for deploying to different environments or running tests within a CI. Because they are just normal SQL files, you can version control them within your repository. Additionally, Geni allows for programmatically making changes to your database without human input.

I've applied this within a project running on Kubernetes. When we released a new version of our app within the Kubernetes cluster, we spun up a geni container in the same Kubernetes namespace. This container checked for any migrations and, if found, ran them before terminating itself.

You can utilize migrations in your integration tests to evaluate the entire application against its actual structure.

There are specific requirements for using this type of tool. Firstly, each migration can only be executed once; if a migration has been applied, it cannot be reapplied. Secondly, migrations need to be run in the correct sequence; for instance, migration 3 cannot be executed before migration 2.

Geni handles this by checking if the table schema_migrations exists in your database before executing each migration. If it doesn't, Geni creates the table and inserts the migration id. This table is used to keep track of which migrations have been applied.

When Geni creates migrations, it uses timestamp as the ID, followed by the name of the migration. The format for a migration is {TIMESTAMP}_{NAME}.sql. Geni orders each migration based on the ID because it's incremental, preventing migrations from being run in the wrong order.

This is why Geni scales effectively with your projects. It allows you to track each migration within your version control. Simultaneously, you can view the current database structure by reading the schema.sql file that Geni also produces.

Installation

There are several ways to install Geni:

Github

You can download the official binaries directly from Github here: https://github.com/emilpriver/geni/releases

Homebrew

You can install Geni using the Homebrew package manager with the following command:

brew install emilpriver/geni/geni

Cargo

You can also install Geni using Cargo with the following command:

cargo install geni

PKGX

Alternatively, you can run Geni using PKGX with the following command:

pkgx geni up

Creating a Migration

Creating a migration is straightforward. Start by running the following command in your repository:

geni new new_table

Geni will create two new files in the ./migrations folder. These files end with .up.sql and .down.sql respectively.

The .up.sql file is where you write new changes. For instance, if you want to create a new users table, you could add:

create table users (
    id int primary key,
    name varchar(255)
);

The .down.sql file is for rolling back changes. Ideally, the SQL code in the .down.sql file should revert the changes made in the .up.sql file. An example is:

drop table users;

Running the migration

After creating your new migration, execute geni up to apply the migrations. This command will prompt geni to read the migrations folders and apply them.

You can also run Geni directly via GitHub Actions as part of your CI flow:

- uses: emilpriver/geni@main
  with:
    migrations_folder: "./migrations"
    wait_timeout: "30"
    migrations_table: "schema_migrations"
    database_url: "<https://localhost:3000>"
    database_token: "X"

Alternatively, you can include it as a container in your docker-compose file:

version: "3.8"

services:
  luigi-database:
    image: postgres:15.6
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: development
    profiles: [dev, test]
    ports:
      - "7432:5432"
    restart: unless-stopped

  migration-postgres:
    image: ghcr.io/emilpriver/geni:latest
    environment:
      - DATABASE_URL=postgresql://postgres:postgres@luigi-database:5432/development
    volumes:
      - ./migrations:/migrations
    command: up
    profiles: [dev, test]

I use Geni as a container for projects where I want others to easily set up and run an environment.

The end

I hope this article has inspired you to use migrations for your databases, and perhaps even try Geni.

If you're interested in tracking the development of Geni, I recommend starring it on GitHub: https://github.com/emilpriver/geni

For more of my content, consider following me on Twitter: https://twitter.com/emil_priver

Geni, A database migration CLI tool

Emil Privér — Thu, 28 Dec 2023 08:34:58 +0000

I'm excited to introduce Geni, a user-friendly CLI migration tool for databases written in Rust. It currently supports LibSQL, Postgres, MariaDB, MySQL, and SQLite, and I have plans to add support for more databases in the future. The concept behind Geni is simple: it can seamlessly integrate with your code as a plugin for your CI, or as a sidebar in Kubernetes, without causing any disruptions or requiring you to use a specific programming language.

Personally, I'm a big fan of dbmate and I rely on it for both my professional and personal projects to handle migrations. It's an excellent tool. However, when they declined my PR to add LibSQL support, I decided to develop my own CLI tool. I couldn't find any satisfactory migration tools, although I did come across Atlas. However, Atlas handles migrations in a way that doesn't align with my preferences, and it also appears to be closely tied to Kubernetes.

Geni became my Christmas project with the goal of being a small component for your workspaces. One common issue with migrations is not keeping track of which migrations have been executed. Geni (and dbmate, for that matter) solves this problem by storing all migrated migrations in the database. This ensures that developers won't overwrite or re-run the same migrations.

The application is developed using Rust and relies on the libsql-client-rs library for SQLite and LibSQL. Moreover, it makes use of SQLX to support Postgres, MariaDB, and MySQL databases. As this is written in rust is it lighting fast, blazingly fast, tiny, ultra fast and memory safe.

Features

Databases:
- Postgres
- MariaDB
- MySQL
- SQLite
- LibSQL
Generating migrations using geni new name
Migrating using geni up
Rollback using geni down
Create database using geni create
Dropping database using geni drop
Timestamp based migrations

Installation

Github

$ sudo curl -fsSL -o /usr/local/bin/geni <https://github.com/emilpriver/geni/releases/latest/download/geni-linux-amd64>
$ sudo chmod +x /usr/local/bin/geni

Homebrew

TBA(check github for more information). Will be added when we are out of beta

Scoop

TBA(check github for more information). Will be added when we are out of beta

PKGX

Run using PKGX

pkgx geni up

Docker

Docker images are published to GitHub Container Registry (ghcr.io/emilpriver/geni).

$ docker run --rm -it --network=host ghcr.io/emilpriver/geni --help

If you wish to create or apply migrations, you will need to use Docker's bind mount feature to make your local working directory (pwd) available inside the geni container:

$ docker run --rm -it --network=host -v "./migrations:./migrations" ghcr.io/emilpriver/geni new hello`

Commands

geni new    # Generate a new migrations file
geni up     # Run any pending migration
geni down   # Rollback migrations, use --amount to speify how many migrations(default 1)
geni create # Create the database, only works for Postgres, MariaDB and MySQL. If you use SQLite will geni create the file before running migrations if the sqlite file don't exist. LibSQL should be create using respective interface.
geni drop   # Remove database
geni help   # Print help message

Example usage

First, make sure you have installed Geni CLI using one of the installation methods mentioned above.

Creating a New Migration

To create a new migration, run the following command:

DATABASE_URL="x" geni new hello_world

This will generate two files, one ending with .up.sql and the other with .down.sql. The .up.sql file is used for creating migrations, while the .down.sql file is used for rolling back migrations. The path to the generated files will be displayed in the console.

Adding SQL to the Migration

Open the .up.sql file and add your SQL code to create the desired database table or make any other changes you need. For example, to create a table named Persons, you can add the following code:

CREATE TABLE Persons (
    PersonID int
);

Make sure to include the corresponding rollback code in the .down.sql file. In this case, the rollback code would be:

DROP TABLE Persons;

Running the Migration

To run the migration and apply the changes to the database, use the following command:

geni up

This will execute all pending migrations and update the database accordingly.

That's it! You have now successfully created a migration, added SQL code to it, and executed the migration using Geni CLI.

End

That’s it. Hope this CLI might be useful to you :D If you find any bugs then I would love to hear about them here https://github.com/emilpriver/geni/issues

Don't forget to follow me on twitter: https://twitter.com/emil_priver

Elixir: Introduction

Emil Privér — Tue, 13 Dec 2022 11:58:16 +0000

I did recently visit a small Elixir and Erlang talk in Varberg, Sweden and I learned some stuff that is something I want to share in this article. The first impression I got was that elixir is a language that is made to be easily understood by developers (short time to get started) but is also built quite smart when it comes to stuff such as spreading running code over multiple cores. But also do elixir have support for releasing a new update to the server without needing to restart the server (hot swap).
Erlang also comes with soft real-time computing meaning that when Elixir is computing real-time data does Elixir allows tasks to be sent after the task's deadline without the system handling the tasks as a failure as long as the task is completed.

What is Elixir

Elixir is a functional, concurrent, fault-tolerance and general-purpose programming language running in Erlang VM, built to make it easy to develop scalable and maintainable applications, also a language that gives a guarantee to know if another process is killed or stuck. The language comes with support for hot-swapping code, meaning that you can swap out running code and use the new functionality without interrupting the service. Ericsson did build the language erlang and erlang-VM to be able to hot swap code which is running without interrupting or closing phone calls. With other languages could this mean downtime which is something that telecom can't have as important calls could be active when a bug needs to be fixed urgently.

Elixir runs your code in lightweight threads when your code is executed, called processes. Each process is isolated but talks (exchange information) to another process via what Elixir calls "Messages". It's not uncommon that Elixir is running 100 thousands of processes concurrently without impacting another process. As all the processes are running isolated, each process has its garbage collection, meaning that 1 process doesn't need to stop other processes from running. In languages such as JavaScript could this mean that the whole script is not running as Node needs to handle its garbage, in elixir case would this mean that the code you are serving your users won't stuck or be slow. This is generally called the actor model. The Actor model is a mathematical model of concurrent computation that treats the actor as the universal primitives of concurrent computation. When an Actor receives a message is it able to make its own decisions such as creating more actors, sending more messages, and telling the system how to respond to the next message. An actor can modify its state but no other process states directly, but can modify other processes states by messaging. As each process can communicate with another process does also Elixir provide fault-tolerance functionality by creating what Elixir calls "supervisors". The supervisor is used to telling the system how to restart parts of your system when things go bad and revert it to its initial state which is guaranteed to work, this is also how Elixir can guarantee fault tolerance. To make Elixir able to run concurrently does it need BEAM or Bogdan Erlang Abstract Machine which is Erlangs Virtual Machine.

Erlangs OTP could you say is a part of the code of Erlang which is providing modules, libraries, and behaviors that represent standard implementations of common practices like message passing or spawning tasks. This means that instead of inventing the wheel again for common services like GenServer, Elixirs OTP provides these tools for you to just grab and use. BEAM uses 1 OS thread per core and runs a scheduler on every os thread to schedule tasks to be executed on that core.

An OTP that Erlang provides which can be used in Elixir is called GenServer. A library that Erlang provides which can be used in Elixir is called GenServer which is a long-running stateful process allowed to send and receive messages. As GenServer or generic server process is that you can keep GenServer always up to date but also execute code async, create a standard set of functions that can be re-used, and also a good way to log errors via GenServer. But it does also provide a good way to handle monitoring events and system messages. GenServer is part of a supervisor and works as a client-server relation between your code and the GenServer. But GenServer is not the only way to work with a state in your application, Erlang also provides ETS or Erlang Term Storage which is a robust in-memory storage store built into OTP which makes it usable within Elixir. ETS can store large amounts of data available to processes.

Outside of platform and language features does also Elixir provide good tooling features which are built to help you develop your application. The mix is the tool used to create, build, manage tasks, and run tests in your application. But Elixir does also provide tooling to interact with your code, the same way Django has interactive tooling. Here are you able to run different functionality such as code reloading, reading formatted docs, or debugging a problem by executing functions in a shell environment. But these interactive tools are also reachable from a website that enables you to run functionality in your code from a website called Livebook.

Atoms

An atom is a constant whose value is its name with no static type assigned to the atom as Elixir is a dynamic language. Some of those are defined by the system to create a standard for how the code is running. One of these atoms is ":ok" which can be used to tell a function that the execution of the function was ok and no error happened. To be able to create an atom in another language would you need to create a constant and assign a value to it with a static type. Meaning that a language that is strict on types would probably create problems as you could check if a return value which is a string is the same as a constant which is a number, but with the same data in both the return and the constant.

iex>:apple
:apple

iex>:orange
:orange

iex>:apple == :apple
true

iex>:apple == :orange
false

Phoenix

Phoenix is a web framework built on Elixir which implements a server-sided Model View Controller. For people that know what Pythons Django is, then Phoenix is the similarity but for Elixir with some extra features such as LiveView. But it also comes with features such as PubSub and Swoosh which are 2 great libraries providing value when developing your Phoenix application. PubSub is used to develop real-time functionality on your website and Swoosh is a feature to help you compose, deliver and test emails sent from your application. As Phoenix comes with a lot out of the box it makes you able to get up and running fast but also that you don't need to reinvent the wheel as Phoenix has a lot of tools already created for you.

More info about Phoenix here

Phoenix's LiveView

The first thought I had when I first did see Phoenix's live view was that it looked like React Server Components which is not the case. It's more like Blazor's live view. Phoenix live view works by listening to a WebSocket which broadcasts messages to the socket telling it to re-render parts of the HTML/template. This means that if you make a change to the server side for example, that an SMS is sent into the system and your users need this information without reloading the page then phoenix can show that message without reloading the page by sending the message into the socket.

Livebook

Livebook is Elixir's version of Jupyter but with some extra cool functions. Livebook is a library of markdown files where you can add executable Elixir code which can for example be used to discover more about your program or work with data but they also give you LiveView and Beam which allows you to write code in 1 place and re-use it in another place. But also to collaborate in a team within your
Livebook. Livebook is both available to run locally on your machine and also in the cloud to enable multiple people to use the same Livebook server and work together.

Livebook is also able to let you investigate problems within your running project. This means that if your customers are experiencing problems and you want to find out what the problem is, then you can within Livebook test certain functionality even if the application is still serving your users. But also a good way to document your application to enable others in your team to understand and work with your application easier.

More info about Livebook here

Pattern Matching

I've heard that pattern matching was something good within Elixir, so I looked into it. In elixir is = a match operator which you can compare to the equal sign in algebra, where the goal is that the value on the left side of = is the same as the value on the right side of =. And if the values match does Elixir return the value, or it throws an error if it fails.

iex> x = 1
1

iex> 1 = x
1

iex> 2 = x
** (MatchError) no match of right hand side value: 1

In this example, we see that in the first 2 examples both values did match as 1 = x and x = 1 but when we tried to check if 2 = x then the execution failed as 2 != 1. Now, this is a basic example that in many other languages works the same, but pattern matching in elixir also works with for example tuples.

iex> {a, b, c} = {:hello, "world", 42}
{:hello, "world", 42}

iex> a
:hello

iex> b
"world"

iex> {a, b, c} = {:hello, "world"}
** (MatchError) no match of right hand side value: {:hello, "world"}

Meaning that we can use this when we for example want to match the return value of a function. We are also able to pattern match on lists.

iex> [a, b, c] = [1, 2, 3]
[1, 2, 3]

iex> a
1

And we can use a pipeline to add the rest of the values to 1 variable

iex> [head | tail] = [1, 2, 3]
[1, 2, 3]
iex> head
1
iex> tail
[2, 3]

And append values to an array in an easy way with the pipeline

iex> list = [1, 2, 3]
[1, 2, 3]

iex> [0 | list]
[0, 1, 2, 3]

But Elixir also supports the pin operator which is used to do pattern matching to a variable with an existing variable rather than rebinding the variable.

iex> x = 1
1

iex> ^x = 2
** (MatchError) no match of right hand side value: 2

As we did earlier pinned value of x to 1 is the equivalent of the following code

iex> 1 = 2
** (MatchError) no match of right hand side value: 2

Use-cases of Elixir or Erlang in production

There are plenty of use cases for Elixir or Erlang by companies but most of the companies I've seen using the languages in production use it for real-time communication or messages queues and some of these companies are Spotify, Discord, Klarna, Ericsson, and WhatsApp. But Elixir is also popular in e-commerce, banking, advertising, and IoT maybe because it has plenty of libraries and frameworks built on top of it, such as Phoenix. And one of the main reasons is that Elixir handles a lot of traffic concurrently and scales easily due to Erlang's VM. Elixir, on the other hand, will probably not be the fastest choice, but it handles tons of traffic on a high level.

More use-cases at Elixirs website

Discord

Discord has been using Elixir since the beginning and handling the core of its products which is the chat infrastructure. Discord is a real-time communication service with both voice and messaging support which as of 8 October 2020 handles 100 million active users each month. Elixir was picked to power relaying messages, real-time applications, and Discord's WebSocket which is used for real-time communication within all apps they provide. While Discord uses Python to power its API. While the Python part of the infrastructure is a monolith while the Elixir part has multiple services distributed to multiple servers running each task concurrently with GenServer and GenStage.

Cars.com

Cars.com is one of the leading platforms for selling cars and buying cars online serving more than 400 million visitors annually. Adopting Elixir as the primary language for its application enabled developers to get started faster and understand the code faster helping cars.com to grow faster and enabling 0 downtimes with a scalable and reliable platform. Cars.com uses Phoenix for its front end enabling Cars.com to show real-time data for its users.

All Aboard

All Aboard is a Swedish company selling train tickets all over the world. They use Elixir to run their backend in a pretty cool way. As All Aboard needs to fetch information from multiple APIs could this mean that they need to way a long time for the response they need for their users, and sometimes they fetch APIs returning links to other APIs which need to also be fetched. They solved this by creating a struct that returns

{:async, fn -> fetch_the_other_thing() end}

When the whole struct is built do they go through each value and put them into a queue in GenServer which creates tasks with the respective function and executes it and as fast as a new task is done will a new task start. And as fast as they have a new response, they add a new message to their WebSocket which updates user's view.
In this case, does Elixirs Concurrency work really well as some responses could take a really long time and still don't interrupt other tasks that are running.

I did ask Patrik at All Aboard for a comment on what he thought about Elixir and how they use Elixir. Patrik told that as he comes from Python and Ruby and thinks this quite magical because it's easy to understand Elixir, and it's model, but also works really well.

Thanks to Patrik at All Aboard for wanting to share his experience and how he use Elixir. You can find more information about All Aboard on their website

My thoughts

I think Elixir is a great language that can handle tons of work at the same time due to its concurrent model but also how easy it is to get started with Elixir and get up and running. I don't think programming should be hard, and many languages make it hard to get up and running quickly, but also hard to write your program in general. This is something that Elixir doesn't have due to the tooling they have created and the way the language is written. Something I don't like is that it doesn't come as required, mostly because I think types help prevent problems and enable us to remove some tests that we don't need to write as we typed them and tested all cases by using the types. On the other hand are you able to use types in Elixir and could set a standard for the project to type the code.

I think the way I would use Elixir is with real-time data or message queues. For example, building a notification queue for apps and using Elixir to concurrently send all the messages to the devices as I think this would be the best use-case for it, but I want to explore it in more areas than queues.

Thanks

Thanks to Lars Wikman for the great Elixir talk he had, But also for wanting to read this post and give feedback. Lars is doing great things for the Elixir community. I recommend checking him out at https://underjord.io/ or his YouTube

Also want to thank the community at Kodsnack for answering my questions regarding the language and sharing information on how they use it. Great community with great people.

End

Do you have any input or stuff I could add or change? I would love feedback! I am using a Twitter post for comments here: https://twitter.com/emil_priver/status/1589632749446438912

Deploy Django to AWS Lamda with Zappa

Emil Privér — Fri, 09 Oct 2020 22:28:44 +0000

Originally posted at https://emilpriver.com/posts/deploy-django-to-aws-lamda-with-zappa

Information

Django is a popular CMS for building a website. Many companies and people use Django to power big APIs for their services, Ex Instagram. This article helps you as a developer to set up Django on AWS Lambda.

Django

Django is a python based web application framework that comes with a lot of stuff out of the box, ex Security or Object-relational mapping. Django’s main goal is to ease the creation of complex, database-driven websites. Django is a popular framework used by many computes, for example, Instagram. Django is maintained by DSF (Django Software Foundation).

What is AWS Lambda

AWS Lambda is serverless(Function as a service) computing and makes you able to run code without thinking about servers and downtime. Pretty much deploy code to AWS Lambda with a small config and you are ready to use it. More info about AWS Lambda here: https://aws.amazon.com/lambda/.

Zappa

Zappa is a Python package that bundles up your code written in Django or Flask and with a command deploys the package to AWS Lambda and handling everything from creating a zip of your code to deploy it to Amazon S3 and then to AWS Lambda. Zappa only requires an AWS config on the machine that is used to deploy the code.

What you need to get started

Python installed on your computer (https://www.python.org/downloads/)
Virtualenv installed (https://docs.python.org/3/library/venv.html)
Have AWS CLI configured on your computer with a profile that is allowed to configure Lambda and S3 ( https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html)
A Django application ready to use (https://docs.djangoproject.com/en/1.8/intro/tutorial01/)

Beginning

Start off by entering a virtualenv and enter it. Then install all needed packages.

vitualenv venv
source venv/bin/activate

Then run

zappa init

Zappa will then ask you for an environment name. Example production or dev. Those environments are you then able to use in a CI/CD or something similar to test your code.

Your Zappa configuration can support multiple production stages, like 'dev', 'staging', and 'production'.
What do you want to call this environment (default 'dev'): 
Answer with your desired environment name. You will be able to add more environments later on.

Zappa will then ask you for the profile that will be used to push and deploy the lambda function. The profile needs to be the profile that is allowed to create lambda functions and upload to and create an S3 bucket.

AWS Lambda and API Gateway are only available in certain regions. Let's check to make sure you have a profile set up in one that will work.
We found the following profiles: default, and hdx. Which would you like us to use? (default 'default'): 
Zappa will now ask you which S3 bucket that will be used to upload your code and then deploy it to lambda.

Your Zappa deployments will need to be uploaded to a private S3 bucket.

If you don't have a bucket yet, we'll create one for you too.
What do you want call your bucket? (default 'zappa-108wqhyn4'): django-zappa-sample-bucket
Answer with the name of the bucked you will use.

The next question will either be that Zappa asks for the Django settings path or the main function path depending on if Zappa finds a Django installation in the folder you typed "Zappa init" in. If Zappa doesn't find a Django installation is it not a big deal, you can change to Django set up after the "Zappa init" questions is finished. If Zappa doesn't find a Django installation then type the path to your Django main settings file.

The next questions will be if you want to deploy this Lambda function worldwide.

If you are using Zappa for the first time, you probably don't want to do this!
Would you like to deploy this application globally? (default 'n') [y/n/(p)rimary]: 
If the config looks good then press y and Zappa will generate a Zappa_settings.json file.

If Zappa didn't find a Django installation then change "app_function" to "django_settings" in your zappa_settings.json file.

Deploying your AWS Lambda function

Deploying your AWS Lambda function is not hard. You only need 1 command to deploy the code.

Zappa deploy $ENV

** $ENV **, in this case, is the environment you created earlier when running the Zappa init command.

Zappa will not make a zip of your code, Send it to s3, and then deploy it to AWS Lambda and return a URL with the deployed and running code. You can now try your code by accessing the full URL including /admin at the end (If you run you Django admin at /admin".

You will probably not be able to access your website as you need to configure a remote database as your code is running in a Lambda function. Connecting Django to a database as AWS RDS is a good move that will work with your Lambda Django code.

Uploading updates of your Django code

Doing updates with Zappa is easy as you only need 1 command as well to deploy your code.

zappa update $env

$env is the environment name you want to update.
Zappa will not make a zip of your code, Send it to s3, and then deploy it to AWS Lambda and return a URL with the deployed and running code.

Connect a custom domain to your AWS Lambda function

Start off by setting up a certificate for your custom domain at AWS Certificate Manager and verify your domain. You will, later on, use this certificate you created to connect to your API gateway. Now go you AWS API Gateway and create an API if Zappa has not created an API gateway for you. Now go to "Custom domain names" in API Gateway and press create. Add the domain you want to use. For example api.doimain.tld, and select TLS 1.2 and the endpoint type which depends on the settings you made when running "Zappa init" and press "Create domain name". You should now be able to see a generated "API Gateway domain name" which you point the domain to with a CNAME. Go to the tab "API mappings" and press "Configure API mappings" and map the paths you need for your application. For example /admin and /graphql.

If everything is set up correctly would you now be able to enter the domain with the added mapping and use it for your service.

My thoughts on using Django with AWS Lambda.

Using AWS Lambda with Django has only been great for me so far. Its cheap, and fast. The startup time when the application goes from frozen to warm is fast as well. Most of the request I make is bellow 300ms when the function is warm and when the function is going from cold to warm are the requests served under 800ms. The current setup I have includes Lambda for running the code, RDS for database, and S3 with Cloudfront for all the media. What I like about Lambda is the part of not thinking about scaling and downtime as Lambda scale my application automatic and start my application if is stops(unless the downtime is caused by something else)

I am using a CI/CD which deploys my code to AWS Lambda and handles all the migrations. And if I need to roll back my code am I able to do that with Zappa as well. Zappa has thought of a lot of stuff, for example, CRON jobs(Zappa call CRON jobs "schedule"), Being able to roll back to older versions of the code, and more. More information and commands can be found here: https://github.com/Miserlou/Zappa.

The only "problem" I have with Zappa and AWS, in general, is that Zappa is building for AWS if I want to use most of the tools which Zappa gives me. For example rollback or schedule. And that I don't know a tool which works pretty much the same with another cloud platform example Google Cloud.

Happy coding!

Twitter if anything: https://twitter.com/emil_priver

Sitemap with Nextjs after 9.4 update

Emil Privér — Sun, 09 Aug 2020 18:02:26 +0000

Nextjs have done some changes since last post that I wrote. Since then was getStaticProps, getStaticPaths and getServerSidedProps introduced(You can read more about that here: https://nextjs.org/blog/next-9-4)

One way to generate a sitemap with nextjs after the update on SSG and SSR with a dynamic output based on the pages the application have is to create an API route that fetches data and return it as XML which could be used to read what pages you have on your application.

Create the code that generate the XML feed:

// import functions from the package
import { SitemapStream, streamToPromise } from "sitemap";
// Fetch data from a source which will be used to render the sitemap.
const { posts } = await graphlqlFetch(`
    query getSitemapData {
      projects: allWorks {
        slug {
          current
        }
        publishedAt
      }
    }
  `);
// Create the a stream to write to with a hostname which will be used for all links
// Your are able to add more settings to the stream. I recommend to look a the npm package for more information.
const smStream = new SitemapStream({
  hostname: "https://priver.dev",
});
// Add frontpage
smStream.write({
  url: "/",
});
// Add a static url to ex: about page
smStream.write({
  url: "/about",
});
// add all dynamic url to the sitemap which is fetched from a source.
posts.forEach((element) => {
  smStream.write({
    url: `/${element.slug.current}`,
    lastmod: element.publishedAt,
  });
});
// tell sitemap that there is nothing more to add to the sitemap
smStream.end();
// generate a sitemap and add the XML feed to a url which will be used later on.
const sitemap = await streamToPromise(smStream).then((sm) => sm.toString());

They way I generated my sitemap is with a package named "sitemap" which is an awesome package to generate a sitemap with.

The generation of the sitemap looked like this:

// import functions from the package
import { SitemapStream, streamToPromise } from "sitemap";
// Fetch data from a source which will be used to render the sitemap.
const { posts } = await graphlqlFetch(`
    query getSitemapData {
      projects: allWorks {
        slug {
          current
        }
        publishedAt
      }
    }
  `);
// Create the a stream to write to with a hostname which will be used for all links
// Your are able to add more settings to the stream. I recommend to look a the npm package for more information.
const smStream = new SitemapStream({
  hostname: "https://priver.dev",
});
// Add frontpage
smStream.write({
  url: "/",
});
// Add a static url to ex: about page
smStream.write({
  url: "/about",
});
// add all dynamic url to the sitemap which is fetched from a source.
posts.forEach((element) => {
  smStream.write({
    url: `/${element.slug.current}`,
    lastmod: element.publishedAt,
  });
});
// tell sitemap that there is nothing more to add to the sitemap
smStream.end();
// generate a sitemap and add the XML feed to a url which will be used later on.
const sitemap = await streamToPromise(smStream).then((sm) => sm.toString());

But this is only the code that generates the sitemap.
They sitemap const will later on be used to send data to the user or an service which want to read the sitemap/xml feed.

Send the sitemap to the user:
Nextjs have a build in way to return data to a user or service when an api route is generated. Which means we dont need any external packages like express.

They way I returned data to the user or service when using an api route is this code:

export default async (req, res) => {
  // here is the generation of the sitemap happening

  // tell the output that we will output XML
  res.setHeader("Content-Type", "text/xml");
  // write the generate sitemap to the output
  res.write(sitemap);
  // end and send the data to the user or service.
  res.end();
};

Now the user will have a sitemap that is dynamic generated and outputed in a way that a service like Goole Search Engine is able to read it.

Want to use domain.ltd/sitemap.xml as url for your sitemap?

To use domain.ltd/sitemap.xml in this solution do you basic add a rewrite from /sitemap.xml to /api/sitemap in the next.config.js. More about rewrites here: https://nextjs.org/docs/api-reference/next.config.js/rewrites
Like this:

module.exports = {
  async rewrites() {
    return [
      {
        source: "/sitemap.xml",
        destination: "/api/sitemap",
      },
    ];
  },
};

The full code:

This is the whole code I used to generate a dynamic sitemap:

// import functions from the package
import { SitemapStream, streamToPromise } from "sitemap";
// A custom function I use to fetch data from a backend. I will keep the import to make it more clear why "graphlqlFetch" is used in the code
import graphlqlFetch from "lib/apollo"
export default async (req, res) => {
  // Fetch data from a source which will be used to render the sitemap.
  const { posts } = await graphlqlFetch(`
    query getSitemapData {
      projects: allWorks {
        slug {
          current
        }
        publishedAt
      }
    }
  `);
  // Create the a stream to write to with a hostname which will be used for all links
  // Your are able to add more settings to the stream. I recommend to look a the npm package for more information.
  const smStream = new SitemapStream({
    hostname: "https://priver.dev",
  });
  // Add frontpage
  smStream.write({
    url: "/",
  });
  // Add a static url to ex: about page
  smStream.write({
    url: "/about",
  });
  // add all dynamic url to the sitemap which is fetched from a source.
  posts.forEach((element) => {
    smStream.write({
      url: `/${element.slug.current}`,
      lastmod: element.publishedAt,
    });
  });
  // tell sitemap that there is nothing more to add to the sitemap
  smStream.end();
  // generate a sitemap and add the XML feed to a url which will be used later on.
  const sitemap = await streamToPromise(smStream).then((sm) => sm.toString());
  // here is the generation of the sitemap happening
  // tell the output that we will output XML
  res.setHeader("Content-Type", "text/xml");
  // write the generate sitemap to the output
  res.write(sitemap);
  // end and send the data to the user or service.
  res.end();
};

Github gist with the same code: https://gist.github.com/emilpriver/d037fa56ddb3f450132ba6efe7b0c217

Hope this help you to generate a dynamic sitemap for your Nextjs application.

Twitter if anything: https://twitter.com/emil_priver

What is your go to CSS-in-JSS library?

Emil Privér — Sat, 30 May 2020 11:26:22 +0000

There is many great CSS-in-JS libraries on the internet, Which one do you use and why?

Sitemap with Next.js

Emil Privér — Mon, 03 Feb 2020 19:29:34 +0000

OUTDATED! NEW WAY: https://emilpriver.com/blog/sitemap-with-nextjs-after-9-4-update

Originally written at: My website

So, Nextjs is really hyped and a really good framework for developing. Nextjs is shiped with alot of stuffs, but not an generated sitemap (Which i understand is not added by default in Nextjs).

This is 1 way to create an dynamic sitemap without creating a server to handle Nextjs and use getInitialProps instead as normal pages does.

URL

I wanted to be able to use /sitemap.xml as its a standard for sitemaps.

By testing out different type of filename did i later on understand that you are able to use

sitemap.xml.js

as name and it will be rendered when entering /sitemap.xml in your browser.

Something we will use in this small article.

Developing the XML feed.

Create a new file in pages/ folder in your projects map, the name of the file will be used when you want to access your sitemap.

import React from "react";
import axios from "axios";

const sitemapXML = data => {
  let latestPost = 0;
  let projectsXML = "";

  data.map(post => {
    const postDate = Date.parse(post.modified);
    if (!latestPost || postDate > latestPost) {
      latestPost = postDate;
    }

    const projectURL = `https://domain.ltd/project/${post.slug}/`;
    projectsXML += `
      <url>
        <loc>${projectURL}</loc>
        <lastmod>${postDate}</lastmod>
        <priority>0.50</priority>
      </url>`;
  });

  return `<?xml version="1.0" encoding="UTF-8"?>
    <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
      <url>
        <loc>https://domain.ltd/</loc>
        <lastmod>${latestPost}</lastmod>
        <priority>1.00</priority>
      </url>
      <url>
        <loc>https://domain.ltd/about/</loc>
        <priority>0.80</priority>
      </url>
      ${projectsXML}
    </urlset>`;
};

First part of the code.
Here we create the XML feed with dynamic data.
We have some statics paths which is / and /about which we add manually.
but we also have some dynamic data, in my case is the dynamic data all my projects. So in the code do we loop thrue all my projects from an external source and add xml data to a variable which we later on uses to display the data.

class Sitemap extends React.Component {
  static async getInitialProps({ res }) {
    const data = await axios
      .get(
        "https://domain.ltd/some/url/"
      )
      .then(response => response.data);

    res.setHeader("Content-Type", "text/xml");
    res.write(sitemapXML(data));
    res.end();
  }
}

export default Sitemap;

In the last part do we use axios to fetch data from an api. This works exactly the same way we do normally fetch data when using SSR with Next.JS.
But instead of using render() or return() do we directly send the XML feed to the client. So now we have working XML feed which is rendered SSR and ready when you need it.

And exempel using the method is my website: https://priver.dev/sitemap.xml

Github GIST if you want to copy the code

Hope this helps you :D

Working with Django in Docker

Emil Privér — Sun, 12 Jan 2020 20:36:38 +0000

Originally written at: My website

Django is an awesome framework which powers alot of website, ex Instagram. Working with Django inside an docker image is like heaven. This small article will tell you how to install, build, and run a docker image o your machine. Both production and for development with Docker-compose.

Information:

Django

Django is an pytho based web application framework that comes with alot of stuffs out of the box, ex Security or Object-relational mapper. Djangos main goal is to ease the creation of complex, database-driven websites. Django is a popular framework used by many computes, for exempel Instagram. Django is maintained by DSF(Django Software Foundation).

Docker

Docker is an set coupled SaaS(Software-as-a-service) and PaaS(Platform-as-a-service) products that are using Operating-System-Level Virtualization. This systems or projects that are hosted as a package inside docker are called containers. Multiple Containers are able to to run on the same machine and share the OS kernel that the operativ system provide. Container compared to and Virtual Machine use less space and can handle more applications and require fewer VMs and Operating systems.

What you need in order to start:

Docker
Django installed on your computer

Folder structure:

Install django

To create a new django application do you need to have Django installed. If you dont have django installed run:

python -m pip install Django

To be able to run django inside our Docker image do we need to start a django project. In the folder run:

django-admin startproject backend

We use "backend" as our project name, you can change that to anything you want. But later in this text do you need to change backend to the name you provided when creating the project. This will create a folder named "backend" or the name you provided.

Dockerfile:

# pull docker image 
FROM python:3.7.4-alpine3.10

# add need modules
RUN apk add --no-cache --virtual .build-deps gcc musl-dev
RUN apk add --no-cache mariadb-connector-c-dev gettext python-dev py-pip jpeg-dev zlib-dev libpq postgresql-dev

# Define env:
# PYTHONDONTWRITEBYTECODE tels python to not write .pyc files on the import of source modules
ENV PYTHONDONTWRITEBYTECODE=TRUE

# If this is set to a non-empty string it is equivalent to specifying the -u option.
ENV PYTHONUNBUFFERED=TRUE

# copy our code to docker images path
COPY /backend /usr/app/backend

# tell docker we work in our codes map
WORKDIR /usr/app/backend

# Install all dependencies our image require
RUN pip install -r requirements.txt

# remove uneedeed build dependencies
RUN apk del .build-deps

Docker-compose:

Docker-compose in this case is good for development

# define docker-compose cersion
version: "3"
services:
    django:
        build: .
        command: python /usr/app/backend/manage.py runserver 0.0.0.0:8080
        volumes:
        - ./backend:/usr/app/backend
        ports:
        - "8080:8080"

You are now able to start building :D If you use docker-compose just run "docker-compose up" and the image will be build and start.

Browse Django by open localhost:8080 in your browser

Good luck :) Hit me up if you have any questions.

Docker(Docker-compose) + Wordpress/Bedrock

Emil Privér — Tue, 06 Aug 2019 17:36:40 +0000

Originally written at: My website

This is a small article on how you can work with Docker and Wordpress(bedrock) to be able to host an wordpress site inside a docker machine.

Information

Wordpress:
Wordpress are one of the most used CMS system for websites. Wordpress is baded on PHP and MySQL. Wordpress was released fist time May 27 2003 by WordpressFoundation. As Wordpress are a big CMS system does a lot of people create plugins, themes and so on for Wordpress which later on, Other people can use.

Bedrock:
Bedrock are an "WordPress boilerplate with modern development tools, easier configuration, and an improved folder structure" - Roots. Bedrock makes it possible to easy handle a Wordpress installation and plugins through an PHP dependency manager named Composer. Bedrock is using Dotenv to be able to easy work with different environments for different purpose. This makes you able to work with production, staging and dev environments etc.
Bedrock is also providing security by isolate web root to limit access to non-web files and also more secure passwords by working with wp-password-bcrypt.

Docker:
Docker is an set coupled SaaS(Software-as-a-service) and PaaS(Platform-as-a-service) products that are using Operating-System-Level Virtualization. This systems or projects that are hosted as a package inside docker are called containers. Multiple Containers are able to to run on the same machine and share the OS kernel that the operativ system provide. Container compared to and Virtual Machine use less space and can handle more applications and require fewer VMs and Operating systems.

Composer:
Composer is a Dependency Manager(Not a Package Manager). Composer makes it possible, in this scenario, to be able to install plugins through command line. More information about composer can you read here

Bedrock and Wordpress

What you need in order to start

Docker
Composer
A MySQL database setup.

Setup

From a folder you like run the

composer create-project roots/bedrock

This command will create a Bedrock folder with necessary files and folders.

Files and folder inside the Bedrock folder:

Information about the folders

Config:
Inside the config folder do you have application.php and env files. The application.php have the same functions as wp-config.php has in an normal wordpress installation. Inside the application.php do you for exempel define database information or auth keys and much more. The env files are used to override some of the code inside application.php file or add more code to the project which you for exempel will use when your developing.

Web:
The web folder have 2 sub folders, index.php and wp-config.php. The app folder contains are the only folder that needed when developing. App folder contains themes, plugins, mu(Most use)-plugins and uploads.

Working with environments

As Bedrock supports use of env, is it really simple to use different .env files. If you add an environments folder with different .env are you able to load different .env files depending of which docker-compose file you will choose(Use of different docker-compose file will come later in this post).

Exempel of .env file:


DB_NAME=database_name
DB_USER=database_user
DB_PASSWORD=database_password

# Optionally, you can use a data source name (DSN)
# When using a DSN, you can remove the DB_NAME, DB_USER, DB_PASSWORD, and DB_HOST variables
# DATABASE_URL=mysql://database_user:database_password@database_host:database_port/database_name

# Optional variables
# DB_HOST=localhost
# DB_PREFIX=wp_

WP_ENV=development
WP_HOME=http://example.com
WP_SITEURL=${WP_HOME}/wp

# Generate your keys here: https://roots.io/salts.html
AUTH_KEY='s]Ae(9f!]KK.hYej(:m6qQ.S[xYQAoA&1IJ1S2P3$pU(5.OQxR*7t3Zcm+0;bjEX'
SECURE_AUTH_KEY='N#1;&qR+eESza?AHsE`:}vHpitkG5|x<{bprQNS3%x;Q*hti=}Li?_Z@?%X;w*E4'
LOGGED_IN_KEY='%-SrP^&lx{#;{,m]2(;J!O^E4Noi{TRf9L_CY-B2XeOB7[D>gh6v&q.Z4YHit%mv'
NONCE_KEY='8ncSeT1eS5X,q>a4y`*=XvczTi>&P92fK4naUmvxEM66;(X1_3kE7qYkRnxoFxFN'
AUTH_SALT='(JGfH8,6P}+>]I+3gyo-Q(KpFN_T&fe[]K+5X!jq}]Cw3l'
SECURE_AUTH_SALT='j>O{0B(94Cy_d!=oLJC$1t&T&7z5A>5&kV:]EdN,!ZNOH>QdVLq`!QTDj&VUj1gG'
LOGGED_IN_SALT='X2WjCA1.?>S+8KT9M19fxD4c#TQ0W>Mdn9`P2X1p)NZ%lS+RuXlI$N5323,?}>L}'
NONCE_SALT='fGcv0C0wk3}+L<&`}{?O;&=$!J3_By$WfnH[Iankm4M*q=)1x/%f5;b(K#^sogUl'

This file are you able to clone into for exempel .env.staging or .env.production to be able to work with different environments. You can add any env variable and load this variable into application.php and the project by defining an variable in the application.php and use the env variable as value. Exempel:

Inside .env.production:

TWITTER_URL = https://twitter.com/emil_priver


And load the env variable inside application.php with:
 Config::define('TWITTER_URL', env('TWITTER_URL')); 

Then you are able to call the new defined variable with TWITTER_URL

Creating the theme.

Developing your theme are the same as with an normal installation. Create your folder inside themes folder and code your theme.

I recommend to work with Roots Sage9 which are an theme creator framework that are using Laravel Blade

Working with plugins

As you use composer to create the Bedrock/Wordpress installation, is it an great idea to work with composer to install plugins aswell. When you create an bedrock installation does bedrock provide a composer.json file inside the installation. This composer.json file does already have WordPress Packagist which are a great repository to use to install and download plugins to Wordpress.

WordPress Packagist

Wordpress Packagist are a Composer repository build to be able to install plugins from command line. Wordpress Packagist makes it possible to install plugins easy via an CI/CD and version controll.

So you know.

Bedrock ads an .gitignore file that are excluding all folder and files inside the plugins, mu-plugins, uploads folders. If you adding custom plugins or files then you need to add that folder to not be excluded inside the .gitignore.

Docker setup

Nginx

Lets go thrue the nginx folder we have in the root directory. This folder have a nginx.conf file and host.conf file. Nginx.conf are a settings file for nginx where you can add custom settings and host.conf are the host file for the site to use. This folder also have an Dockerfile that tells docker which image it should use, but also we're to mount the .conf files.

Exempel of Dockerfile:

 
FROM nginx:latest
COPY /vhost.conf /etc/nginx/conf.d/default.conf
COPY /nginx.conf /etc/nginx/nginx.conf

Exempel of nginx.conf:

 
user nginx;
worker_processes auto;

pid /var/run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    include /etc/nginx/conf.d/*.conf;
}

Exempel of host.conf:


server {
    listen 80;
    server_name localhost;
    charset utf-8;
    index index.php;
    root /var/www/html/web;

    location / {
        try_files $uri $uri/ /index.php?$args;
    }

    location ~ \.php$ {
        try_files $uri =404;
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_pass php-fpm:9000;
        fastcgi_index index.php;
        fastcgi_read_timeout 300s;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param PATH_INFO $fastcgi_path_info;
    }
}

PHP

In order to execute our Wordpress PHP code do we need an PHP image. As this is an basic guide will we keep it simple. Inside our docker-compose file can we easy add

  image: php:7.3.6-fpm-alpine3.9

which will add the php library we need, but this is not the only thing we need to do. If we start our app with docker-compose and going to localhost:8080 will we probably see an 404 page. why? As we haven't mounted the code into PHP so PHP can't access the code. How to fix? By adding volume which mounts the code into docker, like this:

  
volumes:
      - ./website/bedrock:/var/www/html

The PHP part of the docker-compose file should look like this now:

 
 php:
    image: php:7.3.6-fpm-alpine3.9
    volumes:
      - ./website/bedrock:/var/www/html

Docker

Docker are an awesome tool to use and makes alot possible while developing with Bedrock and Wordpress. With an CI/CD make this setup it possible to easy deploy multiple container to different servers and use the same database and so on. So how does our final docker-compose file look? Like this:


version: '3'
services:
  nginx:
    build: ./nginx
    volumes:
      - ./website/bedrock:/var/www/html
    ports:
      - '8080:80'
    depends_on:
      - php
  php:
    image: php:7.3.6-fpm-alpine3.9
    volumes:
      - ./website/bedrock:/var/www/html

What if we want to use an different .env file then the one we have inside our bedrock folder? No problem, this is really simple:


version: '3'
services:
  nginx:
    build: ./nginx
    volumes:
      - ./website/bedrock:/var/www/html
    ports:
      - '8080:80'
    depends_on:
      - php
  php:
    image: php:7.3.6-fpm-alpine3.9
    env_file:
      - ./path/to/custom/env-file/.env

    volumes:
      - ./website/bedrock:/var/www/html

By adding env_file into PHP settings of the docker-compose file are we able mount an env file into Bedrock.
If you want to use different env for different enviornments(exempel production or development). Then copy the docker-compose file we have into something else, ex: docker-compose.production.yml and start docker-compose using this file, like this:

  docker-compose -f docker-compose.production.yml up --build

Or if you want to start the normal docker-compose file use:

  docker-compose up --build

End

This should be all the stuffs needed to get started with Docker + Bedrock/Wordpress. As this is an basic guide did I want to have everything at a basic level. Docker and Bedrock are great and with adding some information to all this are you able to use different enviornments, use different php.ini and so on.

An exempel of an site using this setup are Leagueshop. This website have Gitlabs CI/CD connected so we never need to login to the server to build the website. The main reason we are using this setup are to be able to scale when more users comes. If many users visits the website, are we able to connect the server we have to an Load Balancer and then startup more servers to be able to run many Wordpress projects on many servers to work against downtime but also have redundancy when alot of users are visiting the site.

DM me for any questions.

Github project that you can use as template: https://github.com/emilpriver/wordpress-docker-bedrock-starter

Twitter for DMs: https://twitter.com/emil_priver