Forem: Michael Rätzel

Database Functions in Elm-Time - Easy Database Updates in Production

Michael Rätzel — Tue, 09 May 2023 22:08:32 +0000

Originally published at https://michaelrätzel.com/blog/database-functions-in-elm-time-easy-database-updates-in-production

Last week, I added this new feature in Elm-Time, helping you manage and update production databases using custom Elm functions.

It's called 'Database Functions' and allows you to apply Elm functions on your database.

You can see it in action in this demo video:

https://youtu.be/9mFjdf_ABNM

The Problem of Updating a Database Online in Production

Elm-Time is the runtime for the Elm programming language that radically simplifies the development and operation of web services and full-stack web apps.

Although running a backend on Elm-Time is generally a smooth experience, I was not satisfied with the options for manually editing the Elm application state.

For example, I sometimes add items to user accounts as special rewards or delete an account. This means changing the database that is running online in production. The database transactions to make these changes should be quick, as I want to avoid service interruptions. And I also want to avoid a complex preparation: The smaller the project, the more likely I prefer manually entering these changes instead of wiring it up in the application program code.

What had been our options to update a database online in production?

In general, the admin interface is the way to make changes without involving the app's main update function. In earlier versions, the options here were the backup/restore APIs or a migration.

Using the backup and restore APIs for such an update is impractical: If we don't pause the processing of application events, we would effectively erase updates between reading the old state and setting the new state.

Updates via migrations, on the other hand, guarantee isolation and consistency. As we discovered earlier, we support applying a migration function on any deployment, not only when the state type changes.

So yes, we can use a migration function to update the database correctly. But this is unwieldy as it requires running a deployment. The idea for today's solution has been around for some time. And this month, I felt frustrated enough to prioritize it.

So how does the new feature make this easier?

It enables us to apply update functions with custom arguments using the admin interface.

Using Database Functions in Elm-Time

To make a function available on the admin interface, we place it in an Elm module named Backend.ExposeFunctionsToAdmin.

Let's look at an example that exposes three function declarations this way. Below is an excerpt from an example application at https://github.com/elm-time/elm-time/blob/b0062e0c43250b5b49c82fbc4c740ccccffe9bb2/implement/example-apps/database-demo/src/Backend/ExposeFunctionsToAdmin.elm

module Backend.ExposeFunctionsToAdmin exposing (..)

import Backend.State
import Dict
import UserAccount


listNewUserAccounts :
    { accountMaximumAgeInDays : Int }
    -> Backend.State.State
    -> List { accountId : String, accountAgeInDays : Int }
listNewUserAccounts { accountMaximumAgeInDays } state =
[...]


deleteUserAccountById : String -> Backend.State.State -> Backend.State.State
deleteUserAccountById userAccountId stateBefore =
    { stateBefore
        | usersAccounts = stateBefore.usersAccounts |> Dict.remove userAccountId
    }


userAccountAddCreditBalanceEvent :
    { userAccountId : String, addedCredits : Int, reasonText : String }
    -> Backend.State.State
    -> Backend.State.State
userAccountAddCreditBalanceEvent { userAccountId, addedCredits, reasonText } stateBefore =
[...]

Since we have placed them into the Elm module with that magic name, Elm-Time makes these functions available via the admin interface. That means whether you use the command-line or graphical interface, the authorization works the same as for deployments.

In the command-line interface, we can use the new commands list-functions and apply-function.

In the graphical admin interface, we find the exposed functions in the new 'Database Functions' section.

Update the Database - Apply a Function

If you want to follow along, you can use this command to run the same demo app:
elm-time  run-server  --admin-password=test  --deploy=https://github.com/elm-time/elm-time/tree/b0062e0c43250b5b49c82fbc4c740ccccffe9bb2/implement/example-apps/database-demo

Let's take a closer look and apply the simplest function in our example, deleteUserAccountById. This function has two parameters, the userAccountId and the stateBefore of the database.

The second parameter bound to stateBefore is special because its type equals our application state type. The runtime understands this and automatically fills in the current application state as the argument value. That leaves the first parameter, userAccountId, for us to supply manually.

As we can see in the type annotation, the return type also equals the app state type. That means applying this function has the potential to change the application state. Sometimes, we want to do a dry run before changing our database's main branch. For that reason, the interface lets us discard the resulting value instead of committing it to the database.

To make the change go live, we enable the checkbox 'Commit resulting state to database'

In the CLI, we set the corresponding flag using the --commit-resulting-state on the apply-function command.

After using the 'Apply Function' button in the GUI or the apply-function command in the CLI, we get a report about the results of the function application.

Conclusion

In summary, we can now quickly make a manual update to an online database without the need to go through the main update function of the Elm app.

Since this feature is brand-new, I assume the interfaces will evolve and look different soon. We will see. The functionality I showed here has been implemented and is available since version 2023-05-07. For me, this solution made operating backends much more relaxed.
I am looking forward to hearing your thoughts!

Elm Silent Teacher - An Interactive Way to Learn Elm

Michael Rätzel — Sun, 23 Apr 2023 20:37:42 +0000

Originally published at https://michaelrätzel.com/blog/elm-silent-teacher-an-interactive-way-to-learn-elm

Let me introduce Elm Silent Teacher, an educational game designed to help you learn the Elm programming language through interactive exercises.

As I assist beginners in getting started with Elm, I often wonder how we can make learning this programming language easier. This way, I also discovered the 'Silent Teacher' shared by Martin Janiczek at https://discourse.elm-lang.org/t/silent-teacher-a-game-to-teach-basics-of-programming/1490

These compact, interactive learning experiences are an excellent resource for students, so I expanded on the approach and developed it further.

The core loop - learning through bite-sized exercises

In the core interaction loop, the app presents an Elm expression and prompts us to evaluate it.
Of course, evaluating an expression in our brain requires some knowledge of the programming language. So we enter our best guess and initially often get it wrong.
After submitting an answer, the system checks it for correctness. If the answer is incorrect, the Silent Teacher points out the correct answer. Once we are ready for the next challenge, we can continue with a new, similar exercise on the same topic.
And as we get more exercises right, the system shows more advanced challenges. The user gradually learns about programming language elements and functions from the core libraries through many small repetitions.

Elm Silent Teacher - evolved

Compared to the earlier implementation, I adopted a different approach for encoding exercises, simplifying the authoring process. Course authors no longer need to provide a function that computes the correct answer. An interpreter now takes care of this part automatically, running the evaluation on the user's device in the web browser.

Since the interpreter is readily available, users can also experiment with additional expressions. After submitting their answer, they can access a REPL-like sandbox initialized with the exercise's expression. In this exploration mode, users can modify the expression and observe how the evaluation changes accordingly.

To try an example course, visit https://silent-teacher.netlify.app/
Elm Silent Teacher is open-source; you can find the code at https://github.com/elm-time/elm-time/tree/main/implement/elm-time/ElmTime/learn-elm
You can modify the exercises and customize the learning path in the Exercise module

If you took the trip, let me know how it went. I'd love to hear your thoughts.

Introducing Elm Editor - a web-based IDE for Elm programs

Michael Rätzel — Sat, 28 May 2022 17:14:46 +0000

(Original publication at https://michaelrätzel.com/blog/introducing-elm-editor-a-web-based-ide-for-elm-programs)

About one year ago, I started working on Elm Editor, a web app for developing Elm programs.

As an integrated development environment, it assists us in reading, writing, and testing Elm programs and in collaborating with other developers.

This project minimizes the friction for newcomers to get started with programming. Since the front-end is entirely web-based, it requires practically no setup. Any modern web browser is enough to use it.

To see Elm Editor in action, check out the public instance at https://elm-editor.com

This video is a short demonstration of a code editing cycle in Elm Editor:

Features in 2021

Some of the features come for free with the Monaco Editor. Others required more effort and glue code in javascript to communicate with and synchronize with the code editor. (Learned quite a bit about javascript in this project)

Viewing and editing all Elm module files as well as other text files in your project.
Checking programs for problems such as Elm compiler errors.
Saving and sharing the current state of a project, including all files.
Importing complete projects from public git repositories.
For front-end web apps, viewing and testing the app in an iframe.
Visual markers in the code to quickly find locations of problems.
Showing error descriptions on mouse hover.
Completion suggestions to discover available declarations and explore useful codes.
Showing documentation and details when hovering the mouse cursor over a part of the code.
Command palette to discover new functionality and keyboard shortcuts.
Text search with options for case sensitivity, regular expressions, and replacing matches.

Besides these features also found in traditional big desktop-based IDEs, there are more innovative features like the differential project state model.

Project Organization and Implementation

Elm Editor is an open-source project organized for easy customization and deployment of custom instances. The source code lives at https://github.com/elm-fullstack/elm-fullstack/tree/main/implement/example-apps/elm-editor

Most of the action happens in the front-end. The primary role of the back-end is integrating tools like the Elm and elm-format executable files and interfacing with Git hosting services like GitHub and GitLab.

The front-end is mainly written in Elm and integrates the Monaco Editor from the VS Code project. The Elm app implements ports with the javascript-based Monaco Editor. The Elm side also implements language services that power editor features that require understanding the syntax and semantics of the Elm programming language.

Outlook

Development continues to make reading and developing Elm programs even more efficient. Integration of a test runner seems one of the more obvious things to add soon. Adding a REPL seems another great addition. A REPL is a more interactive alternative for testing and exploring the behavior of the code in the current workspace.

Then there are more minor improvements like a preview for Markdown files and viewers for images and audio files (for those working on video games).

There are tons of ideas for new features; I could probably go on for hours. As with my other open-source projects, prioritization depends in large part on your feedback. Besides the Elm discourse, a good place for discussion and feedback is on GitHub at https://github.com/elm-fullstack/elm-fullstack/discussions or in the issues section.

Design Report - Migrations in Elm Fullstack Deployments

Michael Rätzel — Mon, 11 Oct 2021 11:54:27 +0000

This post was originally published at https://michaelrätzel.com/blog/design-report-migrations-in-elm-fullstack-deployments

Elm Fullstack is a tool for developing web services and full-stack web applications. It leverages the Elm programming language and its ecosystem of libraries. As it evolves, Elm Fullstack automates more and more activities in software development that humans had performed in the past.

Some of the tasks it automates are related to maintaining an application's program state. This 'state' contains all information a web service should remember. It is the 'database' of the application. Maintaining this program state is one of the features of Elm Fullstack that shields app developers from the complexity of lower-level concepts in software development.

One common machine-level concept is the distinction between volatile and persistent memory. Many frameworks for software development reflect this distinction, delegating responsibility to handle this complexity to the app developer. The first feature I implemented in Elm Fullstack resolves this distinction so that we don't need to consider it when programming apps.
After persistence, I took on Migrations - another aspect of state management. This post overviews exploration of this area and the design I discovered for migrations in deployments.

Migrations

What are migrations, and why do we need them?

Successful software applications evolve. This evolution manifests in changes to our code and our data. After working on and testing functionality in a development environment, we eventually make our changes available to users and customers. To do this, we update a production system with the new program code. This process is what we call a 'deployment'.

In many cases, we can deploy an Elm app simply by pushing the code to a production system. But sometimes, a change affects the state of our backend or how we represent it in the code. Changing the type of the persistent state presents a new challenge. Since we maintain the backend state across deployments, there is a type mismatch between our app's old and new versions. To resolve the type mismatch, we need to transform that data, map it from the old to the new type.
Elm Fullstack manages these changes through a mechanism called migration.

The example of a type mismatch illustrates why we need migrations. If considering only type-mismatches, I might have picked an API that expects migrations only when changing the state type.
But, looking closer at app development, I found more uses for migrations.

The meaning of the state value not only depends on type declarations but also on functions. Functions in the program code define how our service responds to clients and how these responses depend on the current state of the service. When we deploy a new program version, these dependencies of responses on the state can change, effectively changing the meaning of the service state, even if its type declaration stays the same.
Sometimes when testing during development, I change the app state or edit the database directly. The migration functionality would help here, packaging such an edit in an atomic transaction.
To support these scenarios, I went with a design that, by default, requires us to declare a migration function for every deployment explicitly.

Application Programming Interface

How do migrations work from the app developers' perspective?

Migrations happen as part of deployments, so we do not trigger them explicitly. By default, every deployment implies a migration. But sometimes, we prefer to throw away the current backend state instead of coding a migration. For these cases, the API offers an option to take the app state from an init function instead of looking for a migration. In the command-line interface, we see this option surface on the deploy command.

When deploying a new version of our app, we describe how to map values from the current to the new state type. Elm Fullstack then does the type-checking across these versions and rejects the deployment in case of a mismatch. If we provided a matching migration function with our deployment, Elm Fullstack performs the migration in an atomic transaction.

Ok, now we know the admin interface will yell at us if we don't supply a migration with the next deployment. But how do we declare the migration?

The Elm programming language already provides the elements to code a migration. A function with one argument is enough to describe the transformation. When applying the migration, the type of that argument needs to match the backend state type found in the currently running version. We need to maintain the declaration of that old type somewhere in the new program code so that standard Elm tooling continues to work. Compared to type-checking in Elm, the rules for custom types are more relaxed for migrations. As long as all tags are the same, we can substitute one custom type with another. In other words, type-checking in migrations does not consider the module names of custom types. This way, we can conveniently place all custom types we keep around for migrations under a common prefix.

We code the migration as an Elm function that takes the old state as input and returns the new state with the new backend state type. We place it into the deployed code in an Elm module with a particular name to declare the migration.

For optimal consistency and familiarity, the type of the migrate function is the same as in other update functions used on the backend, minus the event-specific first argument. We might see apps that should issue commands on migration, so the return type supports that, analogous to the update functions for application events. The difference to update functions for application events is that the state type on the input side can differ from the output side.

To summarize, here is the Elm syntax with the type annotation of a migration function:

migrate : OldBackendTypes.State -> ( Backend.Main.State, ElmFullstack.BackendCmds Backend.Main.State )

Conclusion

What is the state of migrations now?

Migrations are implemented and available with the latest releases.
To declare the migration function, we place it in the module Backend.MigrateState like this:

module Backend.MigrateState exposing (migrate)

import Backend.Main
import ElmFullstack


migrate : Backend.Main.State -> ( Backend.Main.State, ElmFullstack.BackendCmds Backend.Main.State )
migrate state =
    ( state, [] )

In this example, we see the trivial case where we don't need any transformation.

While introducing type-checked migrations was significant progress, we can still improve the developer experience around migrations.

An apparent future step is to add automation for coding the migration function. Extracting the backend type declarations from a previous version into a dedicated Elm module is the least we can do here.

BMP Image File Decoding In Elm

Michael Rätzel — Sun, 22 Sep 2019 13:37:03 +0000

Recently I implemented a BMP image decoder in Elm. This Elm function takes an image as byte-array (Bytes.Bytes) and returns the values of the pixels in this image.

For a current project, I was looking for a way to decode 2-dimensional raster images in Elm to get values of individual pixels out of an image. I googled and searched the Elm forum for existing solutions but did not come up with anything. So I built a BMP image file decoder from scratch.

So what is the scenario exactly? The user loads an RGB image file containing a screenshot into our program. The program then needs to read the intensities of the individual pixels' red, green, and blue components for further processing. What's more, the user also should be able to load the same image file into popular raster graphics editors like MS Paint. Following this constraint, I could not use the simplest possible image file format. Instead, I needed to support one of the popular file formats like PNG, BMP, or JPEG.

Looking a bit closer at those image file formats, it seemed that BMP would be the simplest one to implement a decoder for. I found the Wikipedia article on the BMP file format explains it well. In that article, we can also see that BMP supports diverse kinds of pixel formats. But for our application, we only need to support the 24-bit pixel (24bpp) format.

Even with this relatively limited set of supported formats, I expected plenty of opportunities to introduce bugs while writing a decoder implementation. So it was clear from the beginning that I would need to run some tests on the decoder to trust it works correctly. I began with coding tests for some image files. These tests are written in Elm, so we can run them with the elm-test tool to see if the decoding function works correctly. The tests model the flat file on one side and the pixels values on the other side. The reference image files are encoded as base64 strings to get them into the Elm source code. The testing module is published at https://github.com/Viir/bots/blob/82ef37f3089a3e40bd496f507b691a5c25011b33/implement/devtools/read-from-image/tests/DecodeBMPImageTest.elm

To figure out the decoding, I found the Wikipedia article easy enough to follow. One thing which took me by surprise was the behavior of Bytes.Decode.decode in the Elm libraries. This function can return different results for the same inputs since it suppresses stack overflows.

The final image decoding function to get the RGB pixel values out of an image file is published at https://github.com/Viir/bots/blob/82ef37f3089a3e40bd496f507b691a5c25011b33/implement/devtools/read-from-image/src/DecodeBMPImage.elm

(This post was originally published at https://michaelrätzel.com/blog/bmp-image-file-decoding-in-elm)

Persistence in Kalmit - April 2019 Edition

Michael Rätzel — Sun, 28 Apr 2019 14:42:36 +0000

This post was originally published at https://michaelrätzel.com/blog/persistence-in-kalmit---april-2019-edition
It reflects the state of Kalmit in April 2019. As the project is young, I would not be surprised to find substantial changes later.

This guide assumes you already know how state changes work in Elm applications. To learn about the role of the update function as used here, have a look at The Elm Architecture (TEA) animation by lucamug. Note that his illustration contains 'view' and 'DOM' parts which are specific to client-side applications and not used here.

Persistence for development and operation

Immediate automatic persistence is one of the main features of the Kalmit web host. Every update of the application state is immediately persisted.

What does 'immediate automatic persistence' mean exactly? Let's break this down and define these terms.

Why persistence? After the server application has confirmed or reported a transaction to an external party, it should remember the resulting state. Kalmit records every change of the application state, to support consistent continuation even after sudden shutdown of the hosting machine. Since server applications run for months and years, such an interruption is bound to happen sooner or later. When the system starts up again, the application state is restored automatically.
Immediately means before other effects from an update become observable. The commands emitted by the update function are only applied after the state from the same update is saved. For example, when a client receives an HTTP response from your server, you know the new state of the server is already persisted.
I use the attribute automatic to clarify that the implementation of this persistence requires no attention by the app developer, as the Kalmit web host takes care of it. Since the app state itself is persistent, considering a database in the application code is not necessary.

To learn about configuration and operation, see the guide on How to Configure and Deploy Your Kalmit Web App.

Under the hood - how it works

After clarifying what guarantees the runtime offers to support development and operation, the remainder of this guide illustrates what happens under the hood to make this possible and performant.

Recovering After Unexpected Shutdown

During operation of the process, the system records the sequence of events (sometimes called 'messages' in Elm lingo) entering the process. Events are appended to a composition log, contained in composition records. Each composition record also contains the SHA256 hash of the previous composition record. These references from one record to another form a chain enabling complete traceability. A reader can determine the original order of the processed events using the hash references, even if the records have been shuffled around in the meantime.

On startup, the Kalmit web host uses the composition log to restore the process state. It reads the sequence of events and replays the updates to compose the process state again.

For an automated test covering the restoration of the process state, see the method Restore_process_state_over_compositions in the implementation.

Optimizing for Faster Restore

For real-world applications, it is not practical to depend on replaying the whole history every time we want to restore a process state. Even more so since restore is not only used for disaster recovery but also for experimentation and performance profiling.
Besides the time it takes for replaying the composition, a growing composition chain takes up more and more storage space.

To solve both the restore time and storage space problems, Kalmit also stores reductions of the composition chain at regular intervals. The reduced value equals the process state derived from the reduced composition chain. When performing a restore, the composition chain is not anymore read back to the beginning of history, but only up to the last written and available reduction record. Since the earlier part of the composition log is not needed anymore for restoration, it can be truncated, freeing up storage space.

The automated test Restore_process_state_from_combination_of_reduction_and_compositions in the implementation demonstrates the recovery of process states from truncated composition logs, with the help of reductions.