Forem: Miguel Cobá

Editing forms with live_file_input

Miguel Cobá — Wed, 24 May 2023 16:03:22 +0000

Imagine you are writing an app to handle products for a marketplace. Each Product has up to 3 images associated with it. When editing a Product, you should be able to remove any of the already associated images and add new images to the Product, always respecting the limit of 3 images per product.

Quite common requirements, right?

And given that Phoenix has pretty good code generators and LiveView now has an amazing live_file_input component that automates uploading files in an interactive way, it should be pretty easy to implement.

Well...

Keep reading to discover what I learned in the last couple of days trying to code this simple task.

The initial project and schema

Let's start by creating a new project with the latest Phoenix version. I use asdf for this:

# Add plugins
asdf plugin-add erlang
asdf plugin-add elixir
# Install them
asdf install erlang 25.3
asdf global erlang 25.3
asdf install elixir 1.14.4-otp-25
asdf global elixir 1.14.4-otp-25
# Install Phoenix 1.7
mix local.rebar --force
mix local.hex --force
mix archive.install hex phx_new 1.7.2 --force

We can now generate a project. I'll name mine mercury:

mix phx.new mercury --install

We'll use a simple migration and schema for a Product with two attributes: name and a list of images, stored as an array of strings.

Phoenix code generators

Our first attempt is to use the Phoenix code generators:

mix phx.gen.live Products Product products name:string images:array:string

We'll see that the generated code for the images attribute is an input field with type of select (lib/mercury_web/live/product_live/form_component.ex):

<.input
  field={@form[:images]}
  type="select"
  multiple
  label="Images"
  options={[{"Option 1", "option1"}, {"Option 2", "option2"}]}
/>

That's reasonable because the code generators have no way to know that we intended the array of strings to be image URLs and that the input should have a type of file and be supported by the live_file_input component.

As powerful as the generators are for CRUD pages, in this case, we need to do it by ourselves.

(Find the code for this part on the phoenix-code-generator branch of the git repo)

Official live_file_input docs

Let's write the code for file uploads manually following the official docs for live_file_input. We'll start from the generated code and we'll adapt it by adding the code from the official docs.

Add the following routes to router.ex:

  scope "/", MercuryWeb do
    pipe_through :browser

    get "/", PageController, :home

    live "/products", ProductLive.Index
    live "/products/new", ProductLive.New, :new
    live "/products/:id", ProductLive.Show
    live "/products/:id/edit", ProductLive.Edit, :edit
  end

Replace the contents or create the following files.

lib/mercury_web/live/product_live/index.ex:

defmodule MercuryWeb.ProductLive.Index do
  use MercuryWeb, :live_view

  alias Mercury.Products

  @impl true
  def mount(_params, _session, socket) do
    socket =
      socket
      |> assign(:page_title, "Listing Products")
      |> stream(:products, Products.list_products())

    {:ok, socket}
  end
end

lib/mercury_web/live/product_live/index.html.heex:

<.header>
  Listing Products
  <:actions>
    <.link navigate={~p"/products/new"}>
      <.button>New Product</.button>
    </.link>
  </:actions>
</.header>

<.table
  id="products"
  rows={@streams.products}
  row_click={fn {_id, product} -> JS.navigate(~p"/products/#{product}") end}
>
  <:col :let={{_id, product}} label="Name"><%= product.name %></:col>
  <:action :let={{_id, product}}>
    <div class="sr-only">
      <.link navigate={~p"/products/#{product}"}>Show</.link>
    </div>
    <.link navigate={~p"/products/#{product}/edit"}>Edit</.link>
  </:action>
</.table>

lib/mixquic_web/live/artisan_courses_live/new.ex:

defmodule MercuryWeb.ProductLive.New do
  use MercuryWeb, :live_view

  alias Mercury.Products.Product

  @impl true
  def mount(_params, _session, socket) do
    socket =
      socket
      |> assign(:page_title, "New Product")
      |> assign(:product, %Product{})

    {:ok, socket}
  end
end

lib/mercury_web/live/product_live/new.html.heex:

<.live_component
  module={MercuryWeb.ProductLive.FormComponent}
  id={:new}
  title={@page_title}
  action={@live_action}
  product={@product}
  navigate={~p"/products"}
/>

lib/mercury_web/live/product_live/edit.ex with this:

defmodule MercuryWeb.ProductLive.Edit do
  use MercuryWeb, :live_view

  alias Mercury.Products

  @impl true
  def mount(_params, _session, socket) do
    {:ok, socket}
  end

  @impl true
  def handle_params(%{"id" => id}, _url, socket) do
    {:noreply,
     socket
     |> assign(:page_title, "Edit Course")
     |> assign(:product, Products.get_product!(id))}
  end
end

lib/mixquic_web/live/artisan_courses_live/edit.html.heex with:

<.live_component
  module={MercuryWeb.ProductLive.FormComponent}
  id={@product.id}
  title={@page_title}
  action={@live_action}
  product={@product}
  navigate={~p"/products"}
/>

lib/mercury_web/live/product_live/show.ex:

defmodule MercuryWeb.ProductLive.Show do
  use MercuryWeb, :live_view

  alias Mercury.Products

  @impl true
  def mount(_params, _session, socket) do
    {:ok, socket}
  end

  @impl true
  def handle_params(%{"id" => id}, _, socket) do
    {:noreply,
     socket
     |> assign(:page_title, "Show course")
     |> assign(:product, Products.get_product!(id))}
  end
end

lib/mixquic_web/live/artisan_courses_live/show.html.heex:

<.header>
  Product <%= @product.id %>
  <:actions>
    <.link navigate={~p"/products/#{@product}/edit"} phx-click={JS.push_focus()}>
      <.button>Edit product</.button>
    </.link>
  </:actions>
</.header>

<.list>
  <:item title="Name"><%= @product.name %></:item>
</.list>
<div class="mt-4">
  <.label>Images</.label>
  <div class="grid justify-center md:grid-cols-2 lg:grid-cols-3 gap-5 lg:gap-7 my-10">
    <figure
      :for={image <- @product.images}
      class="rounded-lg border shadow-md max-w-xs md:max-w-none"
    >
      <img src={image} />
    </figure>
  </div>
</div>
<.back navigate={~p"/products"}>Back to products</.back>

lib/mercury_web/live/product_live/form_component.ex:

defmodule MercuryWeb.ProductLive.FormComponent do
  use MercuryWeb, :live_component

  alias Mercury.Products

  @impl true
  def render(assigns) do
    ~H"""
    <div>
      <.header>
        <%= @title %>
        <:subtitle>Use this form to manage product records in your database.</:subtitle>
      </.header>

      <.simple_form
        for={@form}
        id="product-form"
        phx-target={@myself}
        phx-change="validate"
        phx-submit="save"
      >
        <.input field={@form[:name]} type="text" label="Name" />
        <.label for="#images">Images</.label>
        <div id="images">
          <div
            class="p-4 border-2 border-dashed border-slate-300 rounded-md text-center text-slate-600"
            phx-drop-target={@uploads.images.ref}
          >
            <div class="flex flex-row items-center justify-center">
              <.live_file_input upload={@uploads.images} />
              <span class="font-semibold text-slate-500">or drag and drop here</span>
            </div>

            <div class="mt-4">
              <.error :for={err <- upload_errors(@uploads.images)}>
                <%= Phoenix.Naming.humanize(err) %>
              </.error>
            </div>

            <div class="mt-4 flex flex-row flex-wrap justify-start content-start items-start gap-2">
              <div
                :for={entry <- @uploads.images.entries}
                class="flex flex-col items-center justify-start space-y-1"
              >
                <div class="w-32 h-32 overflow-clip">
                  <.live_img_preview entry={entry} />
                </div>

                <div class="w-full">
                  <div class="mb-2 text-xs font-semibold inline-block text-slate-600">
                    <%= entry.progress %>%
                  </div>
                  <div class="flex h-2 overflow-hidden text-base bg-slate-200 rounded-lg mb-2">
                    <span style={"width: #{entry.progress}%"} class="shadow-md bg-slate-500"></span>
                  </div>
                  <.error :for={err <- upload_errors(@uploads.images, entry)}>
                    <%= Phoenix.Naming.humanize(err) %>
                  </.error>
                </div>
                <a phx-click="cancel" phx-target={@myself} phx-value-ref={entry.ref}>
                  <.icon name="hero-trash" />
                </a>
              </div>
            </div>
          </div>
        </div>
        <:actions>
          <.button phx-disable-with="Saving...">Save Product</.button>
        </:actions>
      </.simple_form>
    </div>
    """
  end

  @max_entries 3
  @max_file_size 5_000_000

  @impl true
  def update(%{product: product} = assigns, socket) do
    changeset = Products.change_product(product)

    {:ok,
     socket
     |> assign(assigns)
     |> assign_form(changeset)
     |> allow_upload(:images,
       accept: ~w(.png .jpg .jpeg),
       max_entries: @max_entries,
       max_file_size: @max_file_size
     )}
  end

  @impl true
  def handle_event("validate", %{"product" => product_params}, socket) do
    changeset =
      socket.assigns.product
      |> Products.change_product(product_params)
      |> Map.put(:action, :validate)

    {:noreply, assign_form(socket, changeset)}
  end

  def handle_event("save", %{"product" => product_params}, socket) do
    images =
      consume_uploaded_entries(socket, :images, fn meta, entry ->
        filename = "#{entry.uuid}#{Path.extname(entry.client_name)}"
        dest = Path.join(MercuryWeb.uploads_dir(), filename)

        File.cp!(meta.path, dest)

        {:ok, ~p"/uploads/#{filename}"}
      end)

    product_params = Map.put(product_params, "images", images)

    save_product(socket, socket.assigns.action, product_params)
  end

  def handle_event("cancel", %{"ref" => ref}, socket) do
    {:noreply, cancel_upload(socket, :images, ref)}
  end

  defp save_product(socket, :edit, product_params) do
    case Products.update_product(socket.assigns.product, product_params) do
      {:ok, _product} ->
        {:noreply,
         socket
         |> put_flash(:info, "Product updated successfully")
         |> push_navigate(to: socket.assigns.navigate)}

      {:error, %Ecto.Changeset{} = changeset} ->
        {:noreply, assign_form(socket, changeset)}
    end
  end

  defp save_product(socket, :new, product_params) do
    case Products.create_product(product_params) do
      {:ok, _product} ->
        {:noreply,
         socket
         |> put_flash(:info, "Product created successfully")
         |> push_navigate(to: socket.assigns.navigate)}

      {:error, %Ecto.Changeset{} = changeset} ->
        {:noreply, assign_form(socket, changeset)}
    end
  end

  defp assign_form(socket, %Ecto.Changeset{} = changeset) do
    assign(socket, :form, to_form(changeset))
  end
end

lib/mercury_web.ex:


  def static_paths, do: ~w(assets fonts images uploads favicon.ico robots.txt)
  def uploads_dir, do: Application.app_dir(:mercury, ["priv", "static", "uploads"])

lib/mercury/application.ex:

  @impl true
  def start(_type, _args) do
    MercuryWeb.uploads_dir() |> File.mkdir_p!()
    # ...
  end

Run migrations and start the server to try it:

mix ecto.migrate
mix phx.server

If you go now to http://localhost:4000/products

Create a new product:

Save it

and go to the show.html.heex:

We are now able to create a product, add images to it, and display them. So far so good.

(Find the code for this part on the official-live-file-input-docs branch of the git repo)

Editing the images of the product

Let's try to edit the product and change the images:

Now we see the issue. The existing images are not being shown the same way the product name is. If you select one image and save it, this will replace the three ones currently associated with the product. If you don't have the original images available to upload them again when editing the product, they will be replaced and lost.

That's not good. We need to:

show the existing images if they exist
be able to remove some of the existing images if we want to
add new images to the product and save it
ensure that we don't exceed the max number of images allowed.

Let's implement each of those points

Showing existing images

Let's show the images of the product. The images to show are the ones currently associated to the product, not the entries in the :images uploads map.

<.label for="#images">Images</.label>
<div id="images">
  <div class="flex flex-row flex-wrap justify-start items-start text-center text-slate-500 gap-2 mb-8">
    <div :for={image <- @product.images} class="border shadow-md pb-1">
      <figure class="w-32 h-32 overflow-clip">
        <img src={image} />
      </figure>
    </div>
  </div>

Now we can see the images in the edit form:

Removing existing images and adding new images

To remove the existing images from the product we need to take into consideration a couple of things. First, when we remove them, we won't show them on the edit form anymore, but we don't really apply the changes to the product until we click the Save Product button. We'll need to keep this in memory but not persist it until told so by the user.

We can use an removed_images assign to hold the images the user has removed so far. It will start as an empty list and, because it is the initial state, we'll assign it on the mount callback instead of the update callback. Let's add it:

  @impl true
  def mount(socket) do
    socket =
      socket
      |> assign(:removed_images, [])

    {:ok, socket}
  end

Now in the update handler we can calculate which images to show:

  @impl true
  def update(%{product: product} = assigns, socket) do
    changeset = Products.change_product(product)
    images_to_show = product.images -- socket.assigns.removed_images

    {:ok,
     socket
     |> assign(assigns)
     |> assign_form(changeset)
     |> assign(:images_to_show, images_to_show)
     |> allow_upload(:images,
       accept: ~w(.png .jpg .jpeg),
       max_entries: @max_entries,
       max_file_size: @max_file_size
     )}
  end

We add a new handler for the remove event, that will be sent each time we remove an existing image from the Product:

  @impl true
  def handle_event("remove", %{"image" => image}, socket) do
    product = socket.assigns.product
    removed_images = [image | socket.assigns.removed_images]
    images_to_show = product.images -- removed_images

    socket =
      socket
      |> assign(:removed_images, removed_images)
      |> assign(:images_to_show, images_to_show)

    {:noreply, socket}
  end

Let's tweak the template to correctly show the images and allow removing them:

<div :for={image <- @images_to_show} class="border shadow-md pb-1">
  <figure class="w-32 h-32 overflow-clip">
    <img src={image} />
  </figure>
  <a
    phx-click="remove"
    phx-target={@myself}
    phx-value-image={image}
    class="hover:text-slate-800"
  >
    <.icon name="hero-trash" />
  </a>
</div>

We can now remove images and if we reload without saving the changes, the original images are shown.

Let's modify the save event to correctly save the new set of images:

  def handle_event("save", %{"product" => product_params}, socket) do
    images_to_show = socket.assigns.images_to_show

    images =
      consume_uploaded_entries(socket, :images, fn meta, entry ->
        filename = "#{entry.uuid}#{Path.extname(entry.client_name)}"
        dest = Path.join(MercuryWeb.uploads_dir(), filename)

        File.cp!(meta.path, dest)

        {:ok, ~p"/uploads/#{filename}"}
      end)

    product_params = Map.put(product_params, "images", images_to_show ++ images)

    save_product(socket, socket.assigns.action, product_params)
  end

If we now remove an image and add a new one, the Product will correctly be saved

Nice, right?

Well, not completely. If you play around with the current code, you'll see that you can add more images than the maximum allowed by our requirements. In fact, you can go to the edit page and add 3 more images every time you save the form.

(Find the code for this part on the show-remove-add-images branch of the git repo)

Ensure we don't exceed the allowed number of images

Now we have reached the part where the problem is. As it currently is, the live_file_input has no way of knowing additional information other than the one you provide when you call the allow_update function. It is also not possible to modify the configuration after initialization unless you destroy it and start over. live_file_input doesn't know how many files to allow at any given moment because that depends on external information not available to it.

We can try some ways to address this issue.

Replacing the live_file_input configuration dynamically

We want the live_file_input component to allow only enough files so that they, when added to the images that remain in the product after any removal by the user, don't exceed the maximum number of allowed files. For example, if we have a restriction of 3 files maximum, the product has already 2 images associated, and the user removes 1 in the edit page, then the live_file_input component should allow 2 additional files to be uploaded.

This suggests we should be able to reconfigure the live_file_input with the correct value for :max_entries.

Something like this:

  @impl true
  def handle_event("remove", %{"image" => image}, socket) do
    product = socket.assigns.product
    removed_images = [image | socket.assigns.removed_images]
    images_to_show = product.images -- removed_images
    max_entries = @max_entries - length(images_to_show)

    socket =
      socket
      |> assign(:removed_images, removed_images)
      |> assign(:images_to_show, images_to_show)
      |> allow_upload(:images,
        accept: ~w(.png .jpg .jpeg),
        max_entries: max_entries,
        max_file_size: @max_file_size
      )

    {:noreply, socket}
  end

If we now go to the edit form and remove an image we see that, if we start with 3 images associated to the product and we remove 1, the live_file_input component allows us to upload only 1 file. If we remove 2, it will allow us to add 2 new files.

Trying to add 3 files, it shows the error message:

Removing one upload and trying to upload only two images, the error disappears.

It looks good!

Let's remove the last associated image and try to upload 3 completely new images:

Something went wrong.

  Parameters: %{"image" => "/uploads/4c8cb89e-a9ef-4bb6-8672-960570e79fce.jpg"}
[error] GenServer #PID<0.2173.0> terminating
** (ArgumentError) cannot allow_upload on an existing upload with active entries.

Use cancel_upload and/or consume_upload to handle the active entries before allowing a new upload.

    (phoenix_live_view 0.18.18) lib/phoenix_live_view/upload.ex:19: Phoenix.LiveView.Upload.allow_upload/3

There is a problem when we try to reconfigure the live_file_input if there are active entries (those are the files ready to be uploaded and that we are seeing the preview on the screen).

As the message states, one way to avoid this error is to not have active entries when removing images. We can check for active entries and cancel them before trying to reconfigure the live_file_input.

Phoenix has a method to do that. Let's use it:

  @impl true
  def handle_event("remove", %{"image" => image}, socket) do
    product = socket.assigns.product
    removed_images = [image | socket.assigns.removed_images]
    images_to_show = product.images -- removed_images
    max_entries = @max_entries - length(images_to_show)

    socket =
      socket
      |> assign(:removed_images, removed_images)
      |> assign(:images_to_show, images_to_show)
      |> maybe_cancel_uploads()
      |> allow_upload(:images,
        accept: ~w(.png .jpg .jpeg),
        max_entries: max_entries,
        max_file_size: @max_file_size
      )

    {:noreply, socket}
  end

  defp maybe_cancel_uploads(socket) do
    {socket, _} = Phoenix.LiveView.Upload.maybe_cancel_uploads(socket)
    socket
  end

This does the work, but not perfectly. When we cancel the active uploads in the remove handler, they disappear from our preview section.

Before removing the remaining associated image

After removing it, the previews of the two active uploads are gone. :(

We can mitigate this a little by showing an alert to the user informing her that the pending uploads will be removed.

<a
  phx-click="remove"
  phx-target={@myself}
  phx-value-image={image}
  class="hover:text-slate-800"
  data-confirm="Are you sure? The pending uploads will be removed and you need to select them again."
>
  <.icon name="hero-trash" />
</a>

Now a warning is shown when you try to remove an existing image and there are active uploads.

Still, I don't quite like this approach.

(Find the code for this part on the replace-live-file-input-config branch of the git repo)

Updating the max_entries option of live_file_input

Another way I tried was to change just the :max_entries option of the allow_update() call instead of completely replacing the configuration. Something like this:

  @impl true
  def handle_event("remove", %{"image" => image}, socket) do
    product = socket.assigns.product
    removed_images = [image | socket.assigns.removed_images]
    images_to_show = product.images -- removed_images
    max_entries = @max_entries - length(images_to_show)

    socket =
      socket
      |> assign(:removed_images, removed_images)
      |> assign(images_to_show: images_to_show)
      |> maybe_update_upload_config(max_entries)

    {:noreply, socket}
  end

  defp maybe_update_upload_config(socket, max_entries) do
    images_config = Map.get(socket.assigns.uploads, :images)

    new_uploads =
      Map.put(socket.assigns.uploads, :images, %{images_config | max_entries: max_entries})

    assign(socket, :uploads, new_uploads)
  end

But Phoenix wasn't happy about it, telling me that the :uploads is a reserved assign and I can't set it directly.

  Parameters: %{"image" => "/uploads/c121a0d2-2724-4c03-8f7f-dc7c78c07ccc.jpg"}
[error] GenServer #PID<0.2770.0> terminating
** (ArgumentError) :uploads is a reserved assign by LiveView and it cannot be set directly

A dead-end path.

(Find the code for this part on the update-live-file-input-config branch of the git repo)

Leveraging changeset validations

One way to completely work around the constraints of the live_file_input config is to leave the configuration as it is and, instead, rely on the changeset validations to apply the business logic checks.

Let's start with the Product's changeset. We can set a validation there to check the max amount of images allowed:

defmodule Mercury.Products.Product do
  use Ecto.Schema
  import Ecto.Changeset

  schema "products" do
    field :images, {:array, :string}
    field :name, :string

    timestamps()
  end

  @max_files 3

  @doc false
  def changeset(product, attrs) do
    product
    |> cast(attrs, [:name, :images])
    |> validate_required([:name, :images])
    |> validate_length(:images, max: @max_files, message: "Max number of images is #{@max_files}")
  end
end

The validate_length will check that we put at most @max_files elements on the images array.

Now we need to ensure the validate and save events put all the images the user wants to save in the images attribute and the changeset will do the rest.

Let's first create a helper function that does the validation of the current data in the socket: the @images_to_show and the entries in the live_file_input config:

  defp validate_images(socket, product_params, images_to_show) do
    images =
      socket.assigns.uploads.images.entries
      |> Enum.map(fn entry ->
        filename = "#{entry.uuid}#{Path.extname(entry.client_name)}"
        ~p"/uploads/#{filename}"
      end)

    product_params = Map.put(product_params, "images", images_to_show ++ images)

    changeset =
      socket.assigns.product
      |> Products.change_product(product_params)
      |> Map.put(:action, :validate)

    assign_form(socket, changeset)
  end

This helper doesn't really consume the entries, but does iterate over them to generate the filenames so that it can add them to the images that remain associated to the product. That's the new set of images that the user is trying to associate to the product. Then delegates to the Products.change_product/2 (that then calls the Product.changeset/2) to apply the validations. Finally it replaces the changeset in the socket so that the form in the browser shows the new errors if necessary.

With this in place the validate handler is now:

  @impl true
  def handle_event("validate", %{"product" => product_params}, socket) do
    images_to_show = socket.assigns.images_to_show

    socket =
      socket
      |> validate_images(product_params, images_to_show)

    {:noreply, socket}
  end

The save handler doesn't need changes as it already performs the same logic in addition to consuming the uploads.

There are other places where we need to do these checks, for example, when we remove an image from the product.

The remove handler is now this:

  @impl true
  def handle_event("remove", %{"image" => image}, socket) do
    product = socket.assigns.product
    removed_images = [image | socket.assigns.removed_images]
    images_to_show = product.images -- removed_images

    socket =
      socket
      |> assign(:removed_images, removed_images)
      |> assign(:images_to_show, images_to_show)
      |> validate_images(%{}, images_to_show)

    {:noreply, socket}
  end

In this case, as we are not validating the full form, we can just pass an empty product_params to the validate_images function so that the changeset can validate if the images are within the limits.

The cancel event, triggered when we remove one active upload from the form, becomes:

  def handle_event("cancel", %{"ref" => ref}, socket) do
    images_to_show = socket.assigns.images_to_show

    socket =
      socket
      |> cancel_upload(:images, ref)
      |> validate_images(%{}, images_to_show)

    {:noreply, socket}
  end

Here again, we're not validating the full form, and we can just pass an empty product_params to get the validations applied.

Finally, we show the error message in the template, if present:

<.label for="#images">Images</.label>

<.error :for={{msg, _} <- @form[:images].errors}><%= msg %></.error>

<div id="images">

With that in place, we can now try it.

We start with 3 images already in the Product:

If we try to upload an extra image, the validation shows the error:

We can now either remove one of the uploads or one of the images associated to the Product.

Removing one of the uploads:

Removing one of the associated images:

In both cases, the validations are reapplied and the error message is correctly handled.

I think this has better UX than the other options but still is not perfect. If you play around with the form you'll soon notice that the validations on other parts of the form are removed when we pass %{} as the product_params to the validate_images function. Sigh!

(Find the code for this part on the changeset-validations branch of the git repo)

Last resort

One final approach is to completely abandon live_file_input and changesets and perform all the logic in the form_component.ex module:

Use a boolean assign to toggle the showing/hiding of the max files error message in the template
Set/unset the toggle in each of the remove, cancel, validate, and save handlers
Check the assign before saving so that we don't save the form if the toggle is set.

This will work and won't mess with the form's internal errors or with the live_file_input configuration.

But also it feels like is not in line with the "Elixir way" of doing things. But hey, it works, right?

Final considerations

This post is different than the ones I usually write, where I end with a happy feeling of accomplishment. This one feels like I personally couldn't find a way to make it work in an elegant way.

One thing is for sure, live_file_input is amazing for the use case it was created. I am sure it will evolve to support or at least facilitate other use cases like the one discussed here.

Judging by the lightning speed the Phoenix core team and contributors gift us with amazing tools, I'm sure minds more brilliant than mine will find a better way to handle scenarios like this one.

And just to avoid any misunderstanding, I'm not complaining about live_file_input, the bittersweet conclusions of this post fall totally on me and my limited experience with the way LiveView works.

What do you people think? Do you have another solution to this scenario that you want to share? I for sure want to learn it!

You can find the code for this article in this github repo

Photo by Leon Dewiwje on Unsplash

Elixir metrics and StatsD

Miguel Cobá — Sun, 05 Jun 2022 19:15:36 +0000

Last time we saw how to send our metrics to Prometheus. This time, we'll send them to a StatsD server.

What's StatsD?

StatsD is a network daemon that listens for statistics, like counters and timers, sent over UDP or TCP and sends aggregates to one or more pluggable backend services.

StatsD does aggregation of data.

To visualize the data you need another tool. For example, to create graphs from the aggregated data you can use Graphite.

StatsD forwards aggregated data to

What's Graphite?

Graphite does two things:

Store numeric time-series data
Render graphs of this data on demand

In summary, StatsD collects and aggregates metrics and then sends them to Graphite to be rendered in nice graphs.

In this tutorial we'll send our Elixir metrics to StatsD and then we'll use the Graphite dashboard to see the graphs generated from our metrics.

Install StatsD and Graphite

There is a way to install separately StatsD server and Graphite but for this tutorial, I'll just use the official Graphite docker image that also includes the StatsD server.

In a terminal run:

docker run -d \
 --name graphite \
 --restart=always \
 -p 80:80 \
 -p 2003-2004:2003-2004 \
 -p 2023-2024:2023-2024 \
 -p 8125:8125/udp \
 -p 8126:8126 \
 graphiteapp/graphite-statsd

This will start the StatsD on port 8125 and the Graphite dashboard on port 80.

Let's add some sample data just to check that the graphs render correctly.

Evaluate this on another terminal:

echo "foo:2|c" | nc -u -w0 127.0.0.1 8125

Wait at least 10 seconds and then run this:

echo "foo:4|c" | nc -u -w0 127.0.0.1 8125

We have sent two counter values separated by at least 10 seconds.

Let's open a browser and go to http://localhost. You'll see something like this:

On the left panel expand the tree (Metrics -> stats_count) and select the foo entry.

You'll see the graph on the main panel populated with some data. But it is too difficult to see the data. Let's change the resolution of the graph.

Click on the clock icon:

And select a time range of 10 minutes. Now you can see clearly the two data points we send to StatsD:

Why we had to wait 10 seconds in the previous examples? It is because that's the default flush interval timeout that StatsD waits before aggregating the stats and sending them to Graphite.

Try sending this data one after the other without any delay:

echo "foo:3|c" | nc -u -w0 127.0.0.1 8125
echo "foo:5|c" | nc -u -w0 127.0.0.1 8125

If you go to the graph and click the Auto-Refresh button, you'll see that a new value appears on the graph, but the value is 8, that, as you can tell is the sum of 3 and 5 that we sent. StatsD aggregated all the values inside a flush interval and sent the result to Graphite.

Ok, enough with StatsD and Graphite. Let's do some Elixir.

Send Elixir metrics to StatsD

Let's continue using the project we created for this series. Clone it from here if you don't have it. We'll start working on the main branch

Add the :telemetry_metrics_statsd dependency to mix.exs:

  defp deps do
    [
      {:telemetry, "~> 1.0"},
      {:telemetry_metrics, "~> 0.6.1"},
      {:telemetry_metrics_statsd, "~> 0.6.0"}
    ]
  end

and run mix deps.get

Now edit the telemetry.ex file and add the TelemetryMetricsStatsd reporter to the list of children:

  def init(_arg) do
    children = [
      {TelemetryMetricsStatsd, metrics: metrics()}
      {Metrics.Telemetry.CustomReporter, metrics: metrics()}
    ]

    Supervisor.init(children, strategy: :one_for_one)
  end

Now our metrics will be also sent to the StatsD server. By default the TelemetryMetricsStatsd reporter sends the data to 127.0.0.1:8125, but you can configure the host and port passing additional :host and :port options.

One thing to know is that the reporter doesn't aggregate metrics. It just forwards them to StatsD and it is StatsD the one that will do the aggregation before sending the data to Graphite.

Let's generate some metrics. Start the application:

iex -S mix

and run this:

1..80 |> Enum.each(&Metrics.emit(&1))

Wait at least 10 seconds and then run:

1..100 |> Enum.each(&Metrics.emit(&1))

If you navigate the left pane and visualize the stats_count.metrics.emit.value you'll see a graph like the following, showing that the aggregated values in two subsequent intervals are 80 and 100. This is because we generated 80 metric events in the first flush interval and after 10 seconds we generated another 100 events in the second flush interval:

If you now visualize the stats.gauges.metrics.emit.value you'll see the sum metric values.

You'll see that in the first interval the value is 3240: the sum of the first 80 events.

In the second interval, the value is 8290: the sum of the next 100 events.

Now our application is using a new reporter to send the metrics to a StatsD server and we are using Graphite to visualize the metrics in configurable graphs.

There is no limit to the amount of data you can send to StatsD and visualize it with Graphite. You can also configure Graphite and save graphs arranged in dashboards so that you can see your data all together in a simple place

You can find the code for this article in the metrics-statsd branch in the github repo.

About

I'm Miguel Cobá. I write about Elixir, Elm, Software Development, and eBook writing.

Follow me on Twitter
Subscribe to my newsletter
Read all my articles on my blog
Get my books:
- 100 Elixir Tips [FREE]
- Deploying Elixir [FREE]
- Deploying Elixir: Advanced Topics
- eBook Writing Workflow for Developers

Elixir, Kubernetes, and minikube

Miguel Cobá — Sun, 29 May 2022 23:03:41 +0000

A couple of weeks ago I presented my talk:

Elixir, Kubernetes, and minikube

in the Elixir Meetup #5 organized by the fine people on curiosum.com.

This talk is based on chapter 4 of my book "Deploying Elixir: Advanced Topics".

If you haven't gotten the book, this is an excellent way to see what the book is about.

Watch the recorded talk here:

About

I'm Miguel Cobá. I write about Elixir, Elm, Software Development, and eBook writing.

Follow me on Twitter
Subscribe to my newsletter
Read all my articles on my blog
Get my books:
- 100 Elixir Tips [FREE]
- Deploying Elixir [FREE]
- Deploying Elixir: Advanced Topics
- eBook Writing Workflow for Developers

"Deploying Elixir: Advanced Topics" eBook

Miguel Cobá — Tue, 03 May 2022 22:19:57 +0000

I have written a new Elixir book:

Deploying Elixir: Advanced Topics

It is based on the feedback and suggestions I got from the readers of my previous book "Deploying Elixir".

They asked me to write more about clustering in Elixir and deploying to Kubernetes.

I'll take you from zero to deploying a Kubernetes cluster in the three big Cloud Providers.

In this book:

I'll show you how to create an Elixir Release from your application and Dockerize it
Learn what an Erlang cluster is and how to create one
We explore the different strategies to create clusters using the libcluster library
I'll show you how to deploy it to a Kubernetes cluster using minikube
When you feel comfortable scaling your app I'll show you how to deploy to Amazon Web Service, Microsoft Azure, and Google Cloud Platform

You can download a sample chapter here:

Download sample chapter

If you prefer a video version, I presented part of chapter 4 in the this Elixir meetup talk:

Watch it here

Get the full book here:

Deploying Elixir: Advanced Topics

Elixir Wizards Podcast S8E1

Miguel Cobá — Fri, 15 Apr 2022 17:28:21 +0000

A couple of weeks ago I was invited to talk on the Elixir Wizards podcast by Smartlogic.

We talk about Elixir in polyglot environments (that is, where you have a lot of languages used at the same time), my beginnings in the software field, my books, and other sideprojects.

I had a lot of fun chatting with Sundi Myint and Owen Bickford who were amazing in guiding my ramblings to a coherent discussion.
Thank you so much for having me.

You can listen to the podcast here:

Elixir Wizards S8E1

About

I'm Miguel Cobá. I write about Elixir, Elm, Software Development, and eBook writing.

Follow me on Twitter
Subscribe to my newsletter
Read all my articles on my blog
Get my books:
- 100 Elixir Tips [FREE]
- Deploying Elixir [FREE]
- Deploying Elixir: Advanced Topics
- eBook Writing Workflow for Developers

[PRESALE] Deploying Elixir: Advanced Topics

Miguel Cobá — Thu, 07 Apr 2022 12:50:20 +0000

I have a new Elixir book for you:

"Deploying Elixir: Advanced Topics"

What is it about?

I'll show you how to deploy to:

AWS
Azure
GCP

You'll deploy like a PRO using:

clustering
Kubernetes orchestration

Tell me more

This book expands on my previous book "Deploying Elixir".

Many of you asked me to write about common scenarios used in the Enterprise! 🏭

Well, I did it! 💪

If you want to deploy your application like the PROS in the Industry do, this book is for you! 🎯

When it will be released?

The book will be published on May 3rd, 2022! 🗓️

But as a SPECIAL, TIME-LIMITED OFFER you can get 35% off the final price if you purchase it on PRESALE.

⚡ Get it on PRESALE and save 35% off the final price ⚡

Elixir, Telemetry, and Prometheus

Miguel Cobá — Mon, 21 Feb 2022 23:51:59 +0000

Yes, just as you read it in the title. I'm going to show you how to send your Elixir telemetry data to Prometheus.

First, let's recapitulate. In the previous article, I showed you how to create an Elixir Metrics Reporter to do the aggregation of your telemetry data and convert it to meaningful metrics.
Now I am going to show you how to use the telemetry_metrics_prometheus library to expose this data so that Prometheus can retrieve it and process it.

Prerequisites

We are going to use the repo we did for the previous article. Clone it from here.

Install the telemetry_metrics_prometheus library

Add the telemetry_metrics_prometheus to your mix.exs dependencies.

  defp deps do
    [
      {:telemetry, "~> 1.0"},
      {:telemetry_metrics, "~> 0.6.1"},
      {:telemetry_metrics_prometheus, "~> 1.1.0"}
    ]
  end

Then get the new dependency:

mix deps.get

Configuring the Prometheus reporter

Now we need to add the reporter to telemetry.ex. Change the init/1 function to this:

  def init(_arg) do
    children = [
      {TelemetryMetricsPrometheus, metrics: metrics()},
      {Metrics.Telemetry.CustomReporter, metrics: metrics()}
    ]

    Supervisor.init(children, strategy: :one_for_one)
  end

And that should be all we need to do.

Unfortunately, it is not. The current version of the library doesn't allow several metrics to have the same metric name. Right now, we have this:

  defp metrics do
    [
      counter("metrics.emit.value"),
      sum("metrics.emit.value")
    ]
  end

We need to use different metric names but still handle the same events. We can take advantage of the second parameter to the metric functions that allow us to be explicit about events and measurements. Let's change it to this:

  defp metrics do
    [
      counter("metrics.emit.value.counter", event_name: [:metrics, :emit], measurement: :value),
      sum("metrics.emit.value.sum", event_name: [:metrics, :emit], measurement: :value)
    ]
  end

You can see that now the metric names are different but the event names and measurements are correct. With this, telemetry_metrics_prometheus will work correctly.

Testing the metrics generated

The Prometheus library includes a web server that exposes a /metrics endpoint on port 9568. We can use it to check the metrics collected by Prometheus. Let's try it. Start your app with iex:

iex -S mix

and go to http://localhost:9568/metrics

And you'll see an empty page. Disappointing. But that's because we haven't yet generated any event.

Let's emit some events. In your just opened iex session write this:

iex -S mix
iex(1)> 1..100 |> Enum.each(&Metrics.emit(&1))
Metric: Elixir.Telemetry.Metrics.Counter. Current value: {1, 0}
Metric: Elixir.Telemetry.Metrics.Sum. Current value: {1, 1}
...
Metric: Elixir.Telemetry.Metrics.Counter. Current value: {100, 4950}
Metric: Elixir.Telemetry.Metrics.Sum. Current value: {100, 5050}
:ok
iex(2)>

As you can see, I am emitting 100 events, with values from 1 to 100. As we have count and a sum metrics defined, we should get the count of events (100) and the sum of all those events (the sum from values from 1 to 100 is 5050).

From the iex output, at least we can affirm that our CustomReporter works. What about the Prometheus reporter?

Go again to http://localhost:9568/metrics and you'll see this:

Yay, the output of the /metrics endpoint is correctly counting and adding the values we emit with our events. Good.

We now need Prometheus to ingest this information. So far, our app is only exposing that endpoint but if nobody access it, is not really useful.

Install Prometheus

Let's install Prometheus. You can go to the prometheus page and follow the install instructions. Or, if you are in mac, use brew to install it.

For this tutorial, I am assuming you used brew.

brew install prometheus

This will install it locally and you can either start it manually or as a service. But first, we need to configure it.
The output of brew install will show where the configuration files are. You should see messages similar to these:

==> Caveats
When run from `brew services`, `prometheus` is run from
`prometheus_brew_services` and uses the flags in:
   /usr/local/etc/prometheus.args

To restart prometheus after an upgrade:
  brew services restart prometheus
Or, if you don't want/need a background service you can just run:
  /usr/local/opt/prometheus/bin/prometheus_brew_services

Be sure to look at that output to know where it is. On my laptop, the config file is in /usr/local/etc/prometheus.yml

Open it and add a new section for our elixir app:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
    - targets: ["localhost:9090"]
  - job_name: "telemetry_metrics_prometheus"
    static_configs:
    - targets: ["localhost:9568"]

I only added the last job_name named "telemetry_metrics_prometheus". You can see that I specified the 9568 port and left out the /metrics part. Prometheus by default tries to access the /metrics endpoint of the target.

Save it and now we can start the Prometheus server:

brew services restart prometheus
Stopping `prometheus`... (might take a while)
==> Successfully stopped `prometheus` (label: homebrew.mxcl.prometheus)
==> Successfully started `prometheus` (label: homebrew.mxcl.prometheus)

Prometheus runs on port 9090 so open http://localhost:9090/ and you'll see something like this:

Not much to see yet, but that's because we haven't specified what metric we want to see. If you remember the metrics we expose on /metrics have the names metrics_emit_value_sum and metrics_emit_value_counter. Let's use metrics_emit_value_sum to see the values Prometheus scrapped from our Elixir app. Put it in the "Expression" input field and click on the "Execute" button.

You'll see now the last value Prometheus scrapped from our Elixir app:

Nice. If you go to the graph tab you'll see the value too.

As we only have a single data point, we see a simple horizontal line. Let's add more values, but let's wait 15 seconds between each one (that's the interval Prometheus waits between scraps).

1..80 |> Enum.each(&Metrics.emit(&1))
1..110 |> Enum.each(&Metrics.emit(&1))
1..50 |> Enum.each(&Metrics.emit(&1))
1..170 |> Enum.each(&Metrics.emit(&1))

After that, if you refresh the graph, you'll see something like this:

That's it. Prometheus is now scraping the values from the /metrics endpoint in our Elixir application.

What's next

There is a lot more to explore from here. You could for example integrate with Grafana to create stunning dashboards for your metrics. If you go that way, don't forget to take a look at the prom_ex library, because it automates exposing your Prometheus data and creating Grafana dashboards on application start.

Summary

We learned:

what the telemetry_metrics_prometheus library is
how to scrap our metrics with Prometheus

Source code

You can find the source code for this article in this GitHub repository. Check the metrics-prometheusbranch.

About

I'm Miguel Cobá. I write about Elixir, Elm, Software Development, and eBook writing.

Follow me on Twitter
Subscribe to my newsletter
Read all my articles on my blog
Get my books:
- 100 Elixir Tips [FREE]
- Deploying Elixir [FREE]
- Deploying Elixir: Advanced Topics
- eBook Writing Workflow for Developers

Telemetry and Metrics in Elixir

Miguel Cobá — Mon, 14 Feb 2022 16:22:47 +0000

In the previous article, I introduced you to Telemetry. The telemetry library provides a way to:

generate events
create handler functions that will receive the events
register which handlers will be called when an event occurs

The principal advantage of telemetry is a standardized way to emit events and collect measurements in Elixir. This way, third-party libraries and code that we write ourselves can use it to generate events in a consistent way.

Let's now take a step back and consider a couple of things. An event is just a measure of something that happened at a given point in time. By itself doesn't tell us much. We might need more data to compare it against or to infer some trend or some rate of change. But the measure by itself is not very useful.

With that in mind, the telemetry developers created a library called telemetry_metrics. According to their documentation, telemetry_metrics is:

Common interface for defining metrics based on :telemetry events.

Ok, but what are metrics, and how it is different from measurements taken with events? Well, again according to their docs:

Metrics are aggregations of Telemetry events with specific name, providing a view of the system's behavior over time.

Ok, it seems clear, right? Not really.

If you start using the telemetry_metrics library you'll notice that in fact DOESN'T AGGREGATE anything!!!

What?

Exactly that. Doesn't aggregate anything.

If you generate 1000 events with a measurement each, this library is not going to keep, manipulate, calculate, summarize or do anything by itself.

So, what is useful for, you might ask?

Its aim is to define what kind of metrics you're going to create from events. That's it. Just a declaration, a definition, a contract if you will. Nothing else.

So, how do you get the real metrics, the processed results of your carefully collected measurements?

For that, you need something else. You need a reporter.

The reporter's responsibility is to aggregate and process each event received so that it conforms to the specified metric it handles.

telemetry_metrics ships with a simple reporter called ConsoleReporter that doesn't do much. It only outputs the measurement and metric details to the console. Not much, but allows you to verify that everything works and you're collecting data to generate metrics correctly.

For more useful reporters you should either write them yourself or get them from someone that has already written it.

In this article, I am going to show you how to use the default ConsoleReporter and how to create a custom reporter.

Prerequisites

Use asdf and install these versions of elixir and erlang:

asdf install erlang 24.2.1
asdf global erlang 24.2.1
asdf install elixir 1.13.3-otp-24
asdf global elixir 1.13.3-otp-24

Create an Elixir app

mix new metrics --sup

This will create an Elixir app with an application callback where we can attach our telemetry events and configure our metrics.

Install dependencies

Add telemetry and telemetry_metrics to your mix.exs:

  defp deps do
    [
      {:telemetry, "~> 1.0"},
      {:telemetry_metrics, "~> 0.6.1"}
    ]
  end

Create an event emitter

Let's create a simple event emitter that we can use to test our metrics. Open the lib/metrics.ex file that mix created for you and replace its contents with this:

defmodule Metrics do
  def emit(value) do
    :telemetry.execute([:metrics, :emit], %{value: value})
  end
end

As you can see it uses :telemetry.execute to emit an event passing the value we provide.

Define some metrics

Ok, now the interesting part: let's define some metrics we want to collect. telemetry_metrics allows to create several types of metrics:

counter/2 which counts the total number of emitted events
sum/2 which keeps track of the sum of selected measurement
last_value/2 holding the value of the selected measurement from the most recent event
summary/2 calculating statistics of the selected measurement, like maximum, mean, percentiles, etc.
distribution/2 which builds a histogram of selected measurement

We are going to start with the basics and define a counter metric, assuming we want to count how many times the event has happened.

Create a new file lib/metrics/telemetry.ex and put this in it:

defmodule Metrics.Telemetry do
  use Supervisor
  import Telemetry.Metrics

  def start_link(arg) do
    Supervisor.start_link(__MODULE__, arg, name: __MODULE__)
  end

  def init(_arg) do
    children = [
      {Telemetry.Metrics.ConsoleReporter, metrics: metrics()}
    ]

    Supervisor.init(children, strategy: :one_for_one)
  end

  defp metrics do
    [
      counter("metrics.emit.value")
    ]
  end
end

Let's analyze it. First of all, it is a simple Supervisor that we need to start from some other process in order to work. We'll attach it to our main application supervisor tree. More on that later.

Then, the init function is configuring the children for this supervisor. We have a single child here, the ConsoleReporter. And we are passing a list of metrics to it as initial arguments.

Here is where it becomes interesting. Telemetry.Metrics.counter is the definition of the metric we want to use. We pass a string: "metrics.emit.value". This string will be split using the period as separator and everything but the last part (metrics.emit), will be the event to attach to. The last part (value) will be taken from the measurement passed to the telemetry event handler.

Let's explain it. We have this call in Metrics.emit/1:

    :telemetry.execute([:metrics, :emit], %{value: value})

and we have a metrics definition in Metrics.Telemetry.metrics/0:

      counter("metrics.emit.value")

As you see, the counter is going to attach a handler for the [:metrics, :emit] event and get the :value attribute from the second argument to telemetry.execute call (the measurement):

event: "metrics.emit" -> [:metrics, :emit]
measurement attribute: "value" -> :value

If we were collecting the :query_time of [:my_app, :repo, :query] event, we would write "my_app.repo.query.query_time".

Ok, let's continue.

We are using the ConsoleReporter to collect our metrics. We defined the counter metric. We also said that ConsoleReporter do nothing but output the values received. That's is enough, for now, to check that we are in fact collecting metrics with our ConsoleReporter.

But one thing is missing. We need to attach our Metrics.Telemetry supervisor to the application supervisor, otherwise, nothing will happen. Open application.ex and change the start/2 function to this:


  def start(_type, _args) do
    children = [
      Metrics.Telemetry
    ]

    opts = [strategy: :one_for_one, name: Metrics.Supervisor]
    Supervisor.start_link(children, opts)
  end

As you see we are putting our Metrics.Telemetry as a child of the application Supervisor.

Test it

Let's try it with iex. Open a shell terminal, get the dependencies and open an iex session:

mix deps.get
iex -S mix

Emit an event:

iex(1)> Metrics.emit(4)
[Telemetry.Metrics.ConsoleReporter] Got new event!
Event name: metrics.emit
All measurements: %{value: 4}
All metadata: %{}

Metric measurement: :value (counter)
Tag values: %{}

:ok
iex(2)>

Yay, it works. Although we didn't attach any handler to the [:metrics, :emit] event manually, the ConsoleReporter handled it for us thanks to telemetry_metrics. As I said, it only echoes what is passed in, but the important thing here is that it works.

If the reporter were a little more advanced it would do something with the passed value.

CustomReporter

Let's create a simple CustomReporter by ourselves to see how this works. One thing to notice is that the reporter needs to have a memory of previous values in order to do its calculations. In our case, we need to remember how many events we have received in order to increment it every time we receive a new event. We could use :ets or a GenServer or Agent depending on how complex our implementation needs to be. For this tutorial, I am going to use an Agent as it is very simple to use to keep some persistent value.

Create a file called lib/metrics/telemetry/reporter_state.ex and write this:

defmodule Metrics.Telemetry.ReporterState do
  use Agent

  def start_link(initial_value) do
    Agent.start_link(fn -> initial_value end, name: __MODULE__)
  end

  def value do
    Agent.get(__MODULE__, & &1)
  end

  def increment do
    Agent.update(__MODULE__, &(&1 + 1))
  end
end

Let's attach it as a child to our main application supervisor so that it is started when the app starts. Edit application.ex and change the start/2 function to this:

  def start(_type, _args) do
    children = [
      {Metrics.Telemetry.ReporterState, 0},
      Metrics.Telemetry
    ]

    opts = [strategy: :one_for_one, name: Metrics.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

Now the application will automatically start a counter agent with an initial value of 0 that we can use from every other part of the app by using its name Metrics.Telemetry.CounterAgent. Let's try it:

iex -S mix
iex(1)> Metrics.Telemetry.ReporterState.value()
0
iex(2)> Metrics.Telemetry.ReporterState.increment()
:ok
iex(3)> Metrics.Telemetry.ReporterState.value()    
1

Nice, we have our agent working. We can use it to maintain the state of our custom reporter.

Let's write our reporter. Create a file called lib/metrics/telemetry/custom_reporter.ex:

defmodule Metrics.Telemetry.CustomReporter do
  use GenServer

  alias Metrics.Telemetry.ReporterState
  alias Telemetry.Metrics

  def start_link(metrics: metrics) do
    GenServer.start_link(__MODULE__, metrics)
  end

  @impl true
  def init(metrics) do
    Process.flag(:trap_exit, true)

    groups = Enum.group_by(metrics, & &1.event_name)

    for {event, metrics} <- groups do
      id = {__MODULE__, event, self()}
      :telemetry.attach(id, event, &__MODULE__.handle_event/4, metrics)
    end

    {:ok, Map.keys(groups)}
  end

  def handle_event(_event_name, measurements, metadata, metrics) do
    metrics
    |> Enum.map(&handle_metric(&1, measurements, metadata))
  end

  defp handle_metric(%Metrics.Counter{} = metric, _measurements, _metadata) do
    ReporterState.increment()

    current_value = ReporterState.value()

    IO.puts("Metric: #{metric.__struct__}. Current value: #{current_value}")
  end

  defp handle_metric(metric, _measurements, _metadata) do
    IO.puts("Unsupported metric: #{metric.__struct__}")
  end

  @impl true
  def terminate(_, events) do
    for event <- events do
      :telemetry.detach({__MODULE__, event, self()})
    end

    :ok
  end
end

A lot of things happening here, but don't get distracted by those. What you should notice is the handle_event function that distributes the event to all the specific metric handlers to, well, handle the event.

The handle_metric function uses ReporterState to keep the current state and increments the running counter that we use to track how many times we have received this event. It also prints the new current state to the console.

Let's use our CustomReporter instead of the ConsoleReporter. Change the init/1 function in telemetry.ex to be like this:

  def init(_arg) do
    children = [
      {Metrics.Telemetry.CustomReporter, metrics: metrics()}
    ]

    Supervisor.init(children, strategy: :one_for_one)
  end

As you see, now we use our CustomReporter and we pass the metrics to it just as we did with the ConsoleReporter. Let's try it. Open the console and start iex:

iex -S mix
iex(1)> Metrics.emit(4)
Metric: Elixir.Telemetry.Metrics.Counter. Current value: 1
:ok
iex(2)> Metrics.emit(5)
Metric: Elixir.Telemetry.Metrics.Counter. Current value: 2
:ok
iex(3)> Metrics.emit(2)
Metric: Elixir.Telemetry.Metrics.Counter. Current value: 3
:ok

And there it is, now the counter is correctly keeping track of how many times we have received the event.

Let's review a few more things about the CustomReporter. Most of them are better explained in the official docs here.

It traps the exit signal to detach the handlers when the app terminates or the process is about to finish. It also groups together similar events that require several metrics tracking. For example, suppose you want to collect counter and sum metrics for the event "my_app.repo.query.query_time". This code gets the event once and then calls handle_metric two times, one with the %Telemetry.Metrics.Counter{} as the first argument and one with %Telemetry.Metrics.Sum{} as the first argument. And for both of those calls, the second argument is the measurement we just received.

Add support for more metrics

Ok, so far so good. But if you notice we are just handling one metric. Let's add support for the sum metric.

First, to sum all the event measurements, we need to also keep track of the running sum so far. Our ReporterState is only handling a single integer as the state. Let's change it to now store a tuple, with the first element being the count and the second element being the sum.

Change ReporterState to this:

defmodule Metrics.Telemetry.ReporterState do
  use Agent

  def start_link(initial_value) do
    Agent.start_link(fn -> initial_value end, name: __MODULE__)
  end

  def value do
    Agent.get(__MODULE__, & &1)
  end

  def increment do
    Agent.update(__MODULE__, fn {count, sum} -> {count + 1, sum} end)
  end

  def sum(value) do
    Agent.update(__MODULE__, fn {count, sum} -> {count, sum + value} end)
  end
end

Now the agent has a composite state and a function to increment the count part and a function to add to the total part. This is a naive implementation, of course, and doesn't even care about both parts of the state getting out of sync, but for our purposes, it will suffice. We needed a way to store state, we have it.

Let's change our CustomReporter. Remove the old handle_metric functions and put these ones instead:

  defp handle_metric(%Metrics.Counter{} = metric, _measurements, _metadata) do
    ReporterState.increment()

    current_value = ReporterState.value()

    IO.puts("Metric: #{metric.__struct__}. Current value: #{inspect(current_value)}")
  end

  defp handle_metric(%Metrics.Sum{} = metric, %{value: value}, _metadata) do
    ReporterState.sum(value)

    current_value = ReporterState.value()

    IO.puts("Metric: #{metric.__struct__}. Current value: #{inspect(current_value)}")
  end

  defp handle_metric(metric, _measurements, _metadata) do
    IO.puts("Unsupported metric: #{metric.__struct__}")
  end

Again, nothing fancy. Now we handle a second type of metric, the %Metrics.Sum{} and, similarly to the count one, we use the ReporterState to keep track of the event measurements so far.

Let's tell telemetry_metrics to also handle this type of metrics. Update the metrics/0 function of telemetry.ex to this:

  defp metrics do
    [
      counter("metrics.emit.value"),
      sum("metrics.emit.value")
    ]
  end

Now the sum metric is also being collected.

Finally, change the start/2 function in application.ex to this:

 def start(_type, _args) do
    children = [
      {Metrics.Telemetry.ReporterState, {0, 0}},
      Metrics.Telemetry
    ]

    opts = [strategy: :one_for_one, name: Metrics.Supervisor]
    Supervisor.start_link(children, opts)
  end

We are set. Let's try it. In the shell:

iex -S mix
iex(1)> Metrics.emit(4)
Metric: Elixir.Telemetry.Metrics.Counter. Current value: {1, 0}
Metric: Elixir.Telemetry.Metrics.Sum. Current value: {1, 4}
:ok
iex(2)> Metrics.emit(3)
Metric: Elixir.Telemetry.Metrics.Counter. Current value: {2, 4}
Metric: Elixir.Telemetry.Metrics.Sum. Current value: {2, 7}
:ok
iex(3)> Metrics.emit(2)
Metric: Elixir.Telemetry.Metrics.Counter. Current value: {3, 7}
Metric: Elixir.Telemetry.Metrics.Sum. Current value: {3, 9}
:ok
iex(4)> Metrics.emit(1)
Metric: Elixir.Telemetry.Metrics.Counter. Current value: {4, 9}
Metric: Elixir.Telemetry.Metrics.Sum. Current value: {4, 10}
:ok

And that's it. Our CustomReporter is now capable of tracking two different metrics for us. Of course, a production system will have a better way to store and keep track of the series of measurements so that it has more guarantees about the data ingested. But that is out of the scope of this article.

Summary

We learned that telemetry_metrics:

offers a way to define 5 types of metrics in a standard way.
doesn't really care or dictates how the measurements should be stored, aggregated, or manipulated
the reporter responsibility is to ingest and implement the real manipulation of the data

There are several open-source implementations of reporters that allow us to make our measurements available to tools like Prometheus or StatsD servers.

In a future article, I'll talk about integrating with them.

Source code

You can find the source code for this article in this github repository.

About

I'm Miguel Cobá. I write about Elixir, Elm, Software Development, and eBook writing.

Follow me on Twitter
Subscribe to my newsletter
Read all my articles on my blog
Get my books:
- 100 Elixir Tips [FREE]
- Deploying Elixir [FREE]
- Deploying Elixir: Advanced Topics
- eBook Writing Workflow for Developers

Introduction to Telemetry in Elixir

Miguel Cobá — Thu, 03 Feb 2022 11:16:03 +0000

Maybe you, like me, have found Telemetry while reading Elixir but have had no time to learn more details about it. I recently put some time aside to learn it and decided to write about it.

In this article I am going to show you what Telemetry is, why is needed, and how to use it in your own Elixir project.

Software Telemetry

First of all, what is telemetry?

Wikipedia explains it very clearly:

Telemetry is the [...] collection of measurements [...] and their automatic transmission [...] for monitoring

We can apply that to software systems, too, by collecting measurements while a software system is running and keeping track of that data.

Why would we do this, you may ask? Glad you ask. There are several reasons, but the main one is to discover what is happening inside our systems and then optimize them based on our findings.

What problem does telemetry solve for us? It gives us visibility into things that are important to us to measure and monitor.

Maybe we want to keep track of CPU utilization to see why our AWS bill is so high, or we need to see how many times a query is executed and how long it took to finish in our PostgreSQL database.

Measuring things and monitoring is not enough. If we don't use the collected data to make decisions and change or optimize the system measured then we might as well not measure anything at all.

By measuring our systems we get several benefits:

we can look for parts of the system that are slow and can be improved.
we could discover parts of the system that are not used frequently enough and that maybe can be scaled down to reduce our infrastructure bill.
we could aggregate how many resources a customer is using every month and generate utilization graphs and other insights to make better use of our system. This would allow us to invite the customer to upgrade to a higher plan that includes this kind of data analysis.

In summary, software telemetry allows us to measure and act upon the collected data of a running system for our own benefit.

Telemetry in Elixir

There are several ways to take measures from a running Elixir app.

For example, the following fragment would take the duration of a function execution:

def some_function do
    start = System.monotonic_time()
    # do something that you want to measure
    stop = System.monotonic_time()
    record_measure("operation took: ", stop - start)
    # ...
end

In the strict sense, that is telemetry. But is not the best approach. It has several drawbacks:

It is repetitive
You have to reimplement it in every app you want to use it
You have no way to disable measuring, for example, during unit tests
more things that I can't think right now

Elixir Telemetry library

Given that telemetry is something that many people want, a library was created to address this need. It is called telemetry.

Advantages:

provides a standard interface to measure and emit telemetry data
it is small and simple to use

How does it work? It is very simple:

you execute a measurement
you create a function that will be invoked when the measure is taken
you attach the function to the event measured

Using telemetry

Let's try a simple example to see how easy is to start collecting telemetry data.

I am going to create a very simple Elixir for a fictional grocery store.
We want to measure each sale.

First, create a simple Elixir project:

mix new telemetry_intro
cd telemetry_intro

Add telemetry to the list of dependencies in mix.exs:

defp deps() do
  [
    {:telemetry, "~> 1.0"}
  ]
end

Download the dependencies:

mix deps.get

Now our business logic module. Create a file named lib/grocery/store.ex and put this inside:

defmodule TelemetryIntro.Grocery.Store do
  def sale(product, quantity, amount) do
    total = quantity * amount

    "Sold #{product}: #{quantity} units at #{amount} each. Total #{total}"
  end
end

As you see it is very simple. It just receives some parameters, does something, and then returns a value. Doesn't have any measure code yet.

Let's just try it with iex. In the shell, write:

iex -S mix
iex(1)> TelemetryIntro.Grocery.Store.sale("apple", 4, 0.5)
"Sold apple: 4 units at 0.5 each. Total 2.0"

Nothing unexpected here. Let's add some measurements. Say we want to keep track of the total and also to track what product was sold.

Change the sale function to be like this:

defmodule TelemetryIntro.Grocery.Store do
  def sale(product, quantity, amount) do
    total = quantity * amount

    :telemetry.execute(
      [:grocery, :store, :sale],
      %{total: total},
      %{product: product}
    )

    "Sold #{product}: #{quantity} units at #{amount} each. Total #{total}"
  end
end

Good. Now we are doing something. We are using the execute function of the telemetry module and we are passing three things:

the event name as a list of keywords
the measurement we are interested in (sale total in our example)
additional metadata we might need (here I am just putting the product name)

The metadata could be anything we want: our users' IP address, some tracking header from a web request, the user id, etc.

The measurement is what we are interested in. This is what we'll collect and analyze and then use to decide things. It can be anything, but it must be a measurement of something.

Finally is the event name. You can put anything here. Most people put a hierarchy of words to easily identify the events. What is important to keep in mind is that this event name will be used later to match which function will handle this event.

Let's see how the counterpart, the event handler, looks.

Create a file lib/metrics/instrumenter.ex

defmodule TelemetryIntro.Metrics.Instrumenter do
  require Logger

  def handle_event([:grocery, :store, :sale], measurements, metadata, _config) do
    Logger.info("[Sale telemetry: #{measurements.total}] total for #{metadata.product}")
  end
end

It is very simple. It has a function (the name doesn't matter but the convention is to name it handle_event) that has as the first parameter the keyword list we are going to match against. Then the measurements and metadata. The last parameter doesn't matter right now. Just ignore it.

This function is going to be called every time the execute function is called and the event name matches. That means that this function must avoid blocking execution for much time as it will be evaluated synchronously.

Finally, we do a lame thing and just pass the received info to Logger.info. Ideally, we'll send that telemetry data to some repository for further processing and analysis but that is outside of the scope of this article (I'll cover it in a future one, tho).

What you need to notice here is that we decoupled the emitting of the telemetry data with the handling of the data collected. Our app can use :telemetry.execute everywhere with the same standard interface and, as long as we have a corresponding function to handle that event, we'll be collecting that data.

Even more important, if we don't have an event handler for that event, our emitting code will still work. It won't even notice there is nothing collecting data.

We are only missing one thing. We need to specify which function will handle a given event. We use the :telemetry.attach function to do that.

Create a function called setup in the Instrumenter module:

defmodule TelemetryIntro.Metrics.Instrumenter do
  require Logger

  def setup do
    events = [
      [:grocery, :store, :sale]
    ]

    :telemetry.attach_many("telemetry-intro-instrumenter", events, &__MODULE__.handle_event/4, nil)
  end

  def handle_event([:grocery, :store, :sale], measurements, metadata, _config) do
    Logger.info("[Sale telemetry: #{measurements.total}] total for #{metadata.product}")
  end
end

As you can see, it is passing to :telemetry.attach_many a list of event names to monitor and the function that will handle all those event names.

By using pattern matching, the same handle_event function can handle all the necessary events in your app.

Now we need to call this setup function when the app starts.

Let's change the way our elixir app starts by modifying the mix.exs file and changing the application function inside it so that it is like this:

  def application do
    [
      extra_applications: [:logger],
      mod: {TelemetryIntro, []}
    ]
  end

We added the mod: to specify a module that will be called when the application starts to execute some custom code. Our custom code will be the telemetry configuration.

Now replace the contents of the lib/telemetry_intro.ex module with this:

defmodule TelemetryIntro do
  use Application

  def start(_type, _args) do
    TelemetryIntro.Metrics.Instrumenter.setup()

    children = []
    Supervisor.start_link(children, strategy: :one_for_one)
  end
end

This module implements the start callback of the Application behavior. This function's responsibility is to spawn and link a supervisor and return {:ok, pid} or {:ok, pid, state}. Here we just start a supervisor with no children. That will suffice for this example.

Let's try it. Start again the application with iex:

iex -S mix
iex(1)> TelemetryIntro.Grocery.Store.sale("apple", 4, 0.2)

00:18:01.867 [info]  [Sale telemetry: 0.8] total for apple
"Sold apple: 4 units at 0.2 each. Total 0.8"

Great, it works. The first output line is the result of handle_event call. The second one is the return value of the sale function.

Let's see what happens if we don't configure a handler for the event.

Comment the TelemetryIntro.Metrics.Instrumenter.setup() in telemetry_intro.ex

defmodule TelemetryIntro do
  use Application

  def start(_type, _args) do
    # TelemetryIntro.Metrics.Instrumenter.setup()

    children = []
    Supervisor.start_link(children, strategy: :one_for_one)
  end
end

And try again with iex:

iex -S mix
iex(1)> TelemetryIntro.Grocery.Store.sale("apple", 4, 0.2)
"Sold apple: 4 units at 0.2 each. Total 0.8"

As you can see, the code still works, even if there is no handler configured for that event. Our app still calls the :telemetry.execute function but nothing breaks when nothing is configured to handle that event. How cool is that?

We learned how to emit events and handle them. You can send this info to a third-party error monitoring service or ingest it to a big data store. From here, the sky is the limit.

Summary

In this article we learned:

what is telemetry
how to add it to an Elixir project
how to measure and collect data
how telemetry works

As we have seen, telemetry is a powerful tool to inspect the behavior of a running system.

As long as we use the collected data for making decisions to improve our system or its behavior, data collection through telemetry is an investment that every system must seriously consider making.

If you want to learn more about telemetry, you can read the full documentation here

About

I'm Miguel Cobá. I write about Elixir, Elm, Software Development, and eBook writing.

Follow me on Twitter
Subscribe to my newsletter
Read all my articles on my blog
Get my books:
- 100 Elixir Tips [FREE]
- Deploying Elixir [FREE]
- Deploying Elixir: Advanced Topics
- eBook Writing Workflow for Developers

"100 Elixir Tips" eBook

Miguel Cobá — Wed, 05 Jan 2022 14:15:24 +0000

I wrote a book with 100 short Elixir samples

100 knowledge pills for you
100 curated examples created for you
100 ways to master Elixir
100 opportunities to increase your knowledge

IT IS FREE TO DOWNLOAD!!!

Download it now

About

I'm Miguel Cobá. I write about Elixir, Elm, Software Development, and eBook writing.

Follow me on Twitter
Subscribe to my newsletter
Read all my articles on my blog
Get my books:
- 100 Elixir Tips [FREE]
- Deploying Elixir [FREE]
- Deploying Elixir: Advanced Topics
- eBook Writing Workflow for Developers

New Elixir Book: "100 Elixir Tips"

Miguel Cobá — Mon, 27 Dec 2021 17:59:34 +0000

For the last three months, I have been publishing a daily Elixir tip on my Twitter account.

As the Twitter search interface isn't good I decided to compile them in an ebook so that you can download it and have it as a reference that is easy to search.

It contains:

75 tips already published in my Twitter account
25 new, unpublished tips

All of them were carefully curated, edited, and selected by me to show the expressive power and elegance of the Elixir programming language.

The best part: it is FREE!!!

You can get it in pre-order now and you'll receive it in your email inbox on January 5th, 2022.

Pre-order it now here 👇🏼

100 Elixir Tips

About

I'm Miguel Cobá. I write about Elixir, Elm, Software Development, and eBook writing.

Follow me on Twitter
Subscribe to my newsletter
Read all my articles on my blog
Get my books:
- 100 Elixir Tips [FREE]
- Deploying Elixir [FREE]
- Deploying Elixir: Advanced Topics
- eBook Writing Workflow for Developers

Elixir Mix podcast

Miguel Cobá — Mon, 20 Dec 2021 18:16:55 +0000

A couple of weeks ago I had the honor to be a guest in the "Elixir Mix" podcast. Sascha Wolf and Allen Wyma did an amazing job in driving the conversation and not letting me ramble endlessly too much.

We talked about deployment in the Elixir world.

I was super nervous and really suffering an episode of imposter syndrome, but I think in the end it was not so bad.

You can listen to the discussion here

https://elixirmix.com/deploying-elixir-with-miguel-cob-emx-155

About

I'm Miguel Cobá. I write about Elixir, Elm, Software Development, and eBook writing.

Follow me on Twitter
Subscribe to my newsletter
Read all my articles on my blog
Get my books:
- 100 Elixir Tips [FREE]
- Deploying Elixir [FREE]
- Deploying Elixir: Advanced Topics
- eBook Writing Workflow for Developers