Forem: Erik Guzman

Avoid Trips To The Database With Nebulex - Phoenix Series

Erik Guzman — Fri, 01 Dec 2023 23:30:01 +0000

Recently, I have been looking for ways to optimize my Stream Closed Captioner application. During one of my typical scroll-throughs of my Following lists and other sites, I discovered Nebulex.

Nebulex, a local and distributed caching toolkit for Elixir.

Like with many other caching toolkits, Nebulex covers all the basics.

This article is not a deep dive into Nebulex features but an example of how I used it in my own application.

Inserting (with time to live):

Blog.Cache.put(key, data, ttl: :timer.hours(1))

Retrieving:

Blog.Cache.get(key)

Counters:

Blog.Cache.incr(:my_counter)

Blog.Cache.incr(:my_counter, 5)

Deleting:

Blog.Cache.delete(1)

Now, this is a partial list of Nebulex capabilities. There are several more functions to interact with the cache, covering many more use cases you might have.

But Why Choose Nebulex?

While looking at Nebulex, two things caught my eye that would make Nebulex easily fit into any Phoenix implication (in my opinion).

The easy setup using a mix task
Nebulex implementation of Cache as System of Record

Easy Setup

Starting with Nebulex is extremely easy because of their handy dandy mix task. Just run:

mix nbx.gen.cache -c Blog.Cache

This command will generate the cache configuration in your config.exs and create a cache module. The last step is to add the cache module to your supervision tree. This step is called out in the docs and task output.

Cache as System of Record (Cache-as-SoR)

This feature caught my eye and is perfect for how I want to use the cache.

If you need to learn what Cache as System of Record is, here is an excerpt from the Nebulex docs.

The cache-as-SoR pattern implies using the cache as though it were the primary system-of-record (SoR).

The pattern delegates SoR reading and writing activities to the cache, so that application code is (at least directly) absolved of this responsibility. To implement the cache-as-SoR pattern, use a combination of the following read and write patterns:

Read-through

Write-through

Advantages of using the cache-as-SoR pattern are:

Less cluttered application code (improved maintainability through centralized SoR read/write operations)

Choice of write-through or write-behind strategies on a per-cache basis

Allows the cache to solve the thundering-herd problem

To better understand what this means, here is a code example of implementing a read-through pattern in my project:

  @decorate cacheable(
              cache: Cache,
              key: {StreamSettings, user_id}
            )
  def get_stream_settings_by_user_id(user_id) do
    Repo.get_by(StreamSettings, user_id: user_id)
  end

In my Settings module, the get_stream_settings_by_user_id is called every time a caption payload is broadcasted to viewers to check if the broadcaster wants to add additional profanity filters or other text modifiers. Nebulex offers the cacheable decorator function that will cache the result of my get_stream_settings_by_user_id function using a specified key. If data already exists in the cache and hasn't expired, then it will return the cached result the next time this function is called.

In the above code, I am caching StreamSettings by the user_id and saving time by avoiding trips to the database till the cache entry expires. I love how easy it is to set up caching on any function I am using.

But what happens if the user's StreamSettings are updated? That's where the write-through pattern's decorator comes in! Let's look at some code again:

@decorate cache_put(
            cache: Cache,
            key: {StreamSettings, stream_settings.user_id}
          )
def update_stream_settings(%StreamSettings{} = stream_settings, attrs) do
  stream_settings
  |> StreamSettings.update_changeset(attrs)
  |> Repo.update()
end

cache_put is the magic function here. It will update the cache with the result of update_stream_settings using the key specified, {StreamSettings, stream_settings.user_id}. I like how Nebulex makes it simple to wrap your existing functions with cache reads and writes.

But the value you are caching in your read-through is different from the value you are caching in your write-through function!?!?

Good Catch! Well, Nebulex also has a function for that. If the value you are caching using the cacheable function is a different shape from the value that the cache_put function, you can use the cache_evict function. Let's go through an example:

# Cache - key:{StreamSettings, 123}, value: nil
get_stream_settings_by_user_id(123) 
# Return: %StreamSettings{some_data}
# Cache - key:{StreamSettings, 123}, value: %StreamSettings{some_data}

Calling get_stream_settings_by_user_id places takes the returned value and places it in the cache, going from nil --> %StreamSettings{some_data}

update_stream_settings(stream_settings, %{update_data}) 
# Return: {:ok, %StreamSettings{some_updated_data}}
# Cache - key: {StreamSettings, 123}, value: {:ok, %StreamSettings{some_updated_data}}

But the problem arises with the return value of update_stream_settings. Instead of the result being a StreamSettings struct, it instead returns a tuple {:ok, %StreamSettings{}}. But cache_evict is here to save the day!

@decorate cache_evict( # <------- The change -------
            cache: Cache,
            key: {StreamSettings, stream_settings.user_id}
          )
def update_stream_settings(%StreamSettings{} = stream_settings, attrs) do
  stream_settings
  |> StreamSettings.update_changeset(attrs)
  |> Repo.update()
end
# Cache - key:{StreamSettings, 123}, value: nil

Instead of caching the new result from update_stream_settings this will remove the value from the cache, removing the worry or mismatched cached data and instead letting the read-through function, aka cacheable write to the action on the following invocation.

Conclusion

Nebulex's Cache as System of Record is a feature that fits right into my mental model of how I would like to work with a cache. Just decorate the functions I use throughout my applications and BAM caching with minimal configuration. Nebulex has more features; look at Nebulex if you're in the market for a caching toolkit.

Using Phoenix Channels? High Memory Usage? Save Money With ERL_FULLSWEEP_AFTER

Erik Guzman — Sat, 18 Nov 2023 16:27:49 +0000

Phoenix Channels are amazing. They make using WebSocket connections a breeze. They are even more lovely using them with Absinthe, a GraphQL server implementation, and GraphQL Connections.

But over time, I discovered Phoenix Channels has a downside. If you use long-lived connections, memory usage on your server will remain unusually high.

How I Use Phoenix Channels

Now for a bit of background on how I use Phoenix Channels. My Stream Closed Captioner application is a free Closed Caption service I offer for Twitch streamers and Zoom meetings. Users log into a dashboard on the website and turn on Speech-to-text. Then a Phoenix Channel sends Text from the front end to the back end, where it goes through a couple of filters like profanity filtering, among other things.

If a Twitch streamer is using the service, it will broadcast the resulting text over another Phoenix Channel for that user's viewers to receive. If they use Zoom, it will send an HTTP request to the Zoom meeting's Closed Caption endpoint.

On the Twitch viewer side, the app communicates to the server via GraphQL. When a Twitch streamer goes live, the viewer client will initialize a GraphQL Connection (aka WebSocket/Phoenix Channel) to receive broadcaster captions from the server.

In other words, each streamer will have a connection to send captions to the backend. Each viewer is another connection that lasts for the live broadcast. For example, if a live stream has 200 viewers, that makes for 200 + 1 connections.

What Does That Mean For Memory Consumption?

Typically, Twitch live streams last at least 3 hours, and I quickly discovered these connections can take up a lot of memory. 6-7 gigabytes of memory to support the number of connections necessary for all streams... YIKES

But why? I spent a little time with my nose in some books and found this passage in

Real-Time Phoenix:

Short-Life Versus Long-Life Processes

Real-time applications rely on long-lived processes in order to have a direct connection to clients—this allows them to send data immediately when the data is available. There is a downside to this, though. It’s possible that processes get stuck in a state where garbage collection doesn’t occur, but memory isn’t being used. This can cause large memory bloat when amplified across thousands of processes.

If a piece of memory makes it past generational garbage collection, it lives until a full-sweep garbage collection occurs. This happens, by default, every 65535 generational passes or when the process is close to using its available memory. It’s possible for a Channel, Socket, or other long-lived process to get stuck in a state where there is plenty of free memory, but not enough work to trigger a generational pass. A process in this state will live without memory being collected, and it will potentially take up more memory than necessary.

Processes that are created and terminated quickly (short-lived processes) do not get into this state because their memory is reclaimed when the process is terminated. While all applications use long-life processes to some degree, real-time apps use them in the thousands. This amplifies the impact of memory bloat.

You can fix this problem by forcing a full-sweep garbage collection to occur...

Why memory consumption is so high now makes a lot more sense. Every connection that is being initialized is long-lived, and the garage collector won't clean up the memory of these processes until 65k garbage collection sweeps have happened. Now what?

But There Is A Solution!

Luckily, Real-Time Phoenix also recommends a few things you can do to force garbage collection sooner.

One such setting is ERL_FULLSWEEP_AFTER.

Understanding ERL_FULLSWEEP_AFTER

ERL_FULLSWEEP_AFTER is a setting related to the Erlang Virtual Machine’s (VM) garbage collection mechanism. It allows you to specify the maximum number of generational collections before forcing a full-sweep, a type of garbage collection that cleans up memory that is no longer in use. By default, this is set to 65535.

ERL_FULLSWEEP_AFTER 20

Real-Time Phoenix author mentioned that they had success setting this value to 20 for their application, so this is what I set my environment variable, but it can be any value that works for you.

The Result!

Within minutes of setting the ERL_FULLSWEEP_AFTER environment variable to 20, I saw results in my Gigalixir memory consumption dashboard. I went from using 6 gigabytes of memory down to around 3.5 gigabytes in a dramatic fashion. Unfortunately, I forgot to snap a screenshot of the dashboard, but the screenshots below still show the updated setting at work.

The above image shows memory consumption after two deployments. You can see the memory consumption jump up to my historical average of around 6 gigabytes after all existing users on the application reconnect to the new instance. After the 20 minor garbage collections kick in, the major garbage collector is triggered, and you see a significant drop.

In this image, those spikes are Twitch streamers going live and connections for all their viewers being made. You can quickly see how much memory connections can take up. But after 20 minor garbage collections, the major garbage collector starts cleaning things up, and you see a significant drop. Amazing! With one simple setting, I solved my memory consumption woes without issues.

Conclusion

If you're building a real-time application using Phoenix Channels, it's essential to be mindful of memory usage, especially if you're dealing with long-lived connections. While Phoenix Channels are fantastic for using WebSocket connections, they can cause memory bloat when used in thousands of processes. However, with a bit of knowledge and tweaking of the ERL_FULLSWEEP_AFTER setting, you can force garbage collection sooner and dramatically reduce memory consumption. So, if you're experiencing a similar issue with your Phoenix Channels application, give this solution a try and see the results for yourself.

I would also like to make a plug for Real-Time Phoenix by Stephen Bussey. It's an awesome book, and you can find the passage referenced in the chapter Manage Your Application’s Memory Effectively, along with more goodies.

Happy coding!

What!? ReactJS isn’t for SPAs anymore?

Erik Guzman — Mon, 20 Mar 2023 01:47:15 +0000

ReactJS has been a popular choice for building single-page applications (SPAs) for years. However, with the release of new documentation, they stripped away almost all mention of SPAs to some people's confusion. Instead, they are pushing the broader idea of “React Architecture.” But what is “React Architecture”? To summarize it:

React architecture is flexible and does not enforce any particular pattern to write a complex application. Developers are free to choose the design pattern of their choice. React architecture is built on a solid foundation and is simple, flexible, and extensible.

In other words, React is a library that can be used to build complex applications with the design pattern of your choice. The primary piece to this change is React Server Components, which were teased a couple of years ago. React Server Components are components that can be rendered on both the server and the client. They allow developers to build applications that span both sides of the application, combining the rich interactivity of client-side apps with the improved performance of traditional server rendering. The new documentation now reflects React as a library to use on both the client and server sides. It pushes single-page apps as a footnote and now recommends using a framework like Next.js that incorporates React Server Components instead of create-react-app.

The React team has admitted that they need to do a better job of adequately communicating the initiatives they have going on. We were teased with React Server Components and other features like Suspense a couple of years ago, but there was mostly radio silence after that until the new docs were released, making it appear like React took a hard right turn off the SPAs highway. But React has been silently heading in this new "direction" for a while now; it just wasn't noticeable.

Is React no longer for SPAs? NO, it’s not only for SPAs but also for the server. The React team wants you to start considering React in that way so you can get the best performance possible for your users by first considering frameworks that use the newer features of React to their fullest extent. But in the end, the beauty of “React Architecture” is that you can choose how to use React.

Ruby Turns 30!

Erik Guzman — Sat, 25 Feb 2023 19:01:14 +0000

30 years ago Ruby programming language was born. Interesting to read that it could have been named coral instead by Ruby. But "coral has worse impression than perl" and Ruby is Matsumoto's birthstone.

Ruby is a programming language I love because of its simplicity and community. It doesn't hinder me from building what I want, and I can do it easily.

Check out the original conversation Matz and Keiju had when deciding the code name for what will be know as Ruby today.

https://blade.ruby-lang.org/ruby-talk/88819

How has your experience been with Ruby?

GitHub Copilot Labs blew my mind with the Custom brush

Erik Guzman — Sun, 05 Feb 2023 09:04:06 +0000

Sorry about the mouse cursor not showing up in this video! If you're not already using the GitHub Copilot Labs extension for VSCode then you missing out on some awesome features. In this video I show you one feature that blew my mind.

You can install Github Copilot Labs here https://marketplace.visualstudio.com/items?itemName=GitHub.copilot-labs

Find out all the details about Github Copilot Labs Here: https://githubnext.com/projects/copilot-labs/#how-to-use-copilot-labs

Fix iPhone not showing as webcam | Continuity Camera

Erik Guzman — Sat, 07 Jan 2023 21:22:15 +0000

youtube.com

So if you're like me, you've been frustrated with trying to use your iPhone as a webcam on Mac Os using the continuity feature.
Sometimes you load up Chrome, and it just doesn't appear. You can use it in FaceTime, but Chrome doesn't have it as a selection. Well, I figured out a workaround, and all you need is one application.

OBS Studio! https://obsproject.com/

All you have to do is

Go and download OBS Studio for your Mac Os.
Open up, and you're presented with two sections at the bottom of the window called “Scenes” and “Sources”. You will want to add a new source.
Click the “+” at the bottom of the “Sources” window
Select “Video Capture Device”, and click Ok.
Select your iPhone in the “Device” selection list. Click Ok.

You should see your webcam taking up the largest preview area. Everything you see in the preview screen will show up as your webcam.

Next, click “Start Virtual Camera”

Now go back to whatever website or application you want to use your webcam. Go into the camera selection. And now you have OBS Virtual Camera as a selection. You hopefully have a reliable way to get your iOS webcam showing up consistently.

You have to check out Github Copilot Labs

Erik Guzman — Sat, 31 Dec 2022 02:07:44 +0000

If you are familiar with the awesomeness that is Github Co-Pilot then you need to check out the Github copilot labs visual studio code extensions. This extension comes packed with loads of cool AI features to help you understand, document, improve, and make your code more robust. Here is how you can install and use it.

Installing Github Co-Pilot Labs Visual Studio Code Extension

Open Visual Studio Code.
Click the Extensions icon in the left sidebar.
Search for Github Co-Pilot Labs in the search bar.
Select the extension and click the Install button.
Once the installation is complete, click the Reload button.
You should now see the Github Co-Pilot Labs extension in the left sidebar.
Click on it and highlight any piece of your code and see what it can do

Microservices to a Monolith, Making a case with Elixir Phoenix

Erik Guzman — Thu, 22 Dec 2022 00:20:52 +0000

Increasing productivity, affordability, scalability, and even less coding (gasp!) are the goals for any web application. Add on top unpredictable server traffic, then a solution you might consider is going serverless and throwing in one of those nifty NoSQL database solutions. Easy enough, but at what cost? You can unintentionally suffer vendor lock-in, deal with cold starts, architect your application improperly, and miscalculate cost structure at scale. But there is an even better solution you might want to consider, Elixir. Elixir and Phoenix can help developers solve all these technical challenges, allowing you to focus on the end-user experience.

Check out out my tech at Elixir Conf 2022 talking about how I made the switch.

TIL: Disabling introspection in prod for Absinthe

Erik Guzman — Sat, 03 Dec 2022 18:22:21 +0000

Recently I had the opportunity to study security best practices for GraphQL. One security recommendation that stood out as low-hanging fruit was disabling GraphQL introspecting for your API in production. GraphQL libraries I have used for Ruby and JavaScript made doing this very easy, so I was expecting the same thing with the Absinthe library in Elixir. It then surprised me that disabled introspection in Absinthe was documented, and every more confusing to figure out how.

Luckily after some web searching, I stumbled upon a Gist made by the community with middleware to disable introspection. So for my own implementation, I decided to make it simpler and disable introspection entirely if you are not in the development environment. Check it out.

defmodule MyAppWeb.Schema.Middleware.AuthorizedIntrospection do
  @moduledoc """
  Disable schema introspection outside of development
  """
  @behaviour Absinthe.Plugin

  @impl Absinthe.Plugin
  def before_resolution(exec) do
    if Enum.find(exec.result.emitter.selections, fn %{name: field_name} ->
         Macro.underscore(field_name) == "__schema" && Mix.env() != :dev
       end) do
      %{
        exec
        | validation_errors: [
            %Absinthe.Phase.Error{message: "Unauthorized", phase: __MODULE__}
          ]
      }
    else
      exec
    end
  end

  @impl Absinthe.Plugin
  def after_resolution(exec), do: exec

  @impl Absinthe.Plugin
  def pipeline(pipeline, _exec), do: pipeline
end

Making RedwoodJS Easier with VSCode Dev Containers

Erik Guzman — Fri, 08 Jul 2022 17:01:00 +0000

In this short (but wordy) guide, I would like to show you how you can set up a RedwoodJS application using Postgres to run in a VSCode Dev Container or Github Codespaces.

But first, if you're not sure what RedwoodJS is, it's an open-source full-stack web framework that makes building applications extremely easy. Think of Ruby on Rails, but all in JavaScript using libraries you might already be familiar with and extremely easy to set up. You can check it out here https://redwoodjs.com/docs/introduction.

RedwoodJS is easy to set up and start, but what would make it even better? Making it extremely easy to get your developer environment set up and running with a couple of clicks and compatible with Github Codespaces. This short guide will give you a template needed to do just that!

We will use the Visual Studio Code Dev Container extension/feature to accomplish this goal. It's a fantastic tool I use almost daily to run my projects in what I like to think of as a development sandbox. Since each project runs in its own sandbox, it doesn't have to worry about installing different Ruby, Rails, NodeJS, Elixir, or Erlang to get my project up and running on my computer. So let's get to it.

Required To Get Started

Docker for Desktop
Visual Studio Code

Set Up Dev Container!

Alright, to get started, open up your RedwoodJS project in VSCode. If you don't have one, it's as simple as.

yarn create redwood-app ./super-awesome-project-name

Now let's install the Remote Development extension. It should look like the following image.

The Remote Development extension package contains everything we would need and more to use Docker as a remote development environment.

Now let's start generating our development container configuration. Bring up the command palette using Command + Shift + P on Mac and Ctrl + Shift + P on Windows. Type "add dev" and select Add Development Container Configuration Files.

You will see a list of pre-built definitions. Well, go with the Node.js & PostgreSQL since it is already almost everything we need to get RedwoodJS up and running.

You will next be asked which version of Node.js you would like your sandbox to run. Let's stick with the default.

The last window allows you to install additional packages/features, but clicking "OK" in the top right will be good enough.

You should now see a new folder in your project called .devcontainer. Inside it will be 4 files, devconatiner.json, docker-compose.yml, create-db-user.sql and Dockerfile.

Well, ignore create-db-user.sql since it's unimportant for this guide. We also won't be touching the Dockerfile but it's what Docker will use to build a development environment with all bells and whistles we need to run Node.js and more.

Let's open up docker-compose.yml file. We will need to make a minor tweak to it. If you dont know what docker-compose is, it's a tool to tell Docker how to step up containers that will run services we would like. Usually, most web development projects need 2-3 services. One service is your web server, RedwoodJS, for us, and another service would be a database like Postgres.

Out of the box, RedwoodJS will use SQLite, but most of the time, I find myself using Postgres in production, and wouldn't it be cool to emulate production? Using Development Container makes it super easy to use Postgres locally, also. So let's try making our local environment run.

Our new docker-compose.yml already comes pre-configured with an app container we will use to develop inside. Looking closely, you will see this container is based on our Dockerfile image. It also has a db container to run our development Postgres database. We need to add one more thing, a container to run our test database. I have annotated the code below to add a db-test container and corresponding volumes. You might be able to copy and paste the whole thing.

version: '3.8'

services:
  app:
    # Using a Dockerfile is optional, but included for completeness.
    build:
      context: .
      dockerfile: Dockerfile
      args:
        # Update 'VARIANT' to pick an LTS version of Node.js: 18, 16, 14.
        # Append -bullseye or -buster to pin to an OS version.
        # Use -bullseye variants on local arm64/Apple Silicon.
        VARIANT: 16-bullseye

    volumes:
      # This is where VS Code should expect to find your project's source code and the value of "workspaceFolder" in .devcontainer/devcontainer.json
      - ..:/workspace:cached

      # Uncomment the next line to use Docker from inside the container. See https://aka.ms/vscode-remote/samples/docker-from-docker-compose for details.
      # - /var/run/docker.sock:/var/run/docker.sock

    # Overrides default command so things don't shut down after the process ends.
    command: sleep infinity

    # Runs app on the same network as the service container, allows "forwardPorts" in devcontainer.json function.
    network_mode: service:db

    # Use "forwardPorts" in **devcontainer.json** to forward an app port locally.
    # (Adding the "ports" property to this file will not forward from a Codespace.)

    # Uncomment the next line to use a non-root user for all processes - See https://aka.ms/vscode-remote/containers/non-root for details.
    # user: vscode

    # Uncomment the next four lines if you will use a ptrace-based debugger like C++, Go, and Rust.
    # cap_add:
    #   - SYS_PTRACE
    # security_opt:
    #   - seccomp:unconfined

  # You can include other services not opened by VS Code as well
  db:
    image: postgres:latest
    restart: unless-stopped
    volumes:
      - postgres-data:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: postgres
      POSTGRES_DB: development_db
      POSTGRES_PASSWORD: postgres
  ###### START ADDED SECTION: Create a container to run the test database
  db-test:
    image: postgres:latest
    restart: unless-stopped
    volumes:
      - postgres-test-data:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: postgres
      POSTGRES_DB: test_db
      POSTGRES_PASSWORD: postgres
  ###### END ADDED SECTION
volumes:
  postgres-data:
  postgres-test-data:#### ADDED LINE

Now let's open up your .devcontainer.json file. This file is specifically for VSCode. It tells it how to run our development container. More specially, the docker-compose file to use, which container we will use to do development inside, and the extension we would like to install by default. The extension configuration is super lovely, or anyone who works with you will have the same extension automatically installed to make their dev life more effortless. Finally, you can copy and paste the code below into the file.

{
    "name": "RedwoodJS & Postgres (Community)",

    // Update the 'dockerComposeFile' list if you have more compose files or use different names.
    "dockerComposeFile": "docker-compose.yml",

    // The 'service' property is the name of the service for the container that VS Code should
    // use. Update this value and .devcontainer/docker-compose.yml to the real service name.
    "service": "app",

    // The optional 'workspaceFolder' property is the path VS Code should open by default when
    // connected. This is typically a volume mount in .devcontainer/docker-compose.yml
    "workspaceFolder": "/workspace",

    // Configure tool-specific properties.
    "customizations": {
        // Configure properties specific to VS Code.
        "vscode": {
            // Add the IDs of extensions you want installed when the container is created.
            "extensions": [
                "redwoodjs.redwood",
                "dbaeumer.vscode-eslint",
                "ofhumanbondage.react-proptypes-intellisense",
                "mgmcdermott.vscode-language-babel",
                "editorconfig.editorconfig",
                "prisma.prisma",
                "graphql.vscode-graphql"
            ]
        }
    },

    // Uncomment the next line if you want to keep your containers running after VS Code shuts down.
    // "shutdownAction": "none",

    // Uncomment the next line to use 'postCreateCommand' to run commands after the container is created.
    // "postCreateCommand": "uname -a",

    // Comment out to connect as root instead. To add a non-root user, see: https://aka.ms/vscode-remote/containers/non-root.
    "remoteUser": "node"
}

Postgres In Development

The last phase of our changes is going to be with RedwoodJS. Since we're setting up our local development to run Postgres, we need to update some things in our RedwoodJS.

First, let's update our schema.prisma file to run Postgres. Then, update your provider, like in the example below.

datasource db {
  provider = "postgresql" // Switched from MySQL to Postgres for local development
  url      = env("DATABASE_URL")
}

generator client {
  provider      = "prisma-client-js"
  binaryTargets = "native"
}

// Define your own datamodels here and run `yarn redwood prisma migrate dev`
// to create migrations for them and apply to your dev DB.
// TODO: Please remove the following example:
model UserExample {
  id    Int     @id @default(autoincrement())
  email String  @unique
  name  String?
}

Next, we need to open and update our .env file. Since we changed to using Postgres, we need to change DATABASE_URL and TEST_DATABASE_URL to point to our Postgres database containers and use the credentials that are set in the docker-compose.yml. Looking back at the docker-compose.yml file, you will see where we got the username, password, and database name. Another important bit is to set the RWJS_DEV_API_URL variable so routing happens properly from within our container to the GQL server.

# THIS FILE SHOULD NOT BE CHECKED INTO YOUR VERSION CONTROL SYSTEM
#
# Environment variables set here will override those in .env.defaults.
# Any environment variables you need in production you will need to setup with
# your hosting provider. For example in Netlify you can add environment
# variables in Settings > Build & Deploy > environment
#
DATABASE_URL=postgres://postgres:postgres@localhost:5432/development_db
TEST_DATABASE_URL=postgres://postgres:postgres@localhost:5432/test_db
#
# Sets an app-specific secret used to sign and verify your own app's webhooks.
# For example if you schedule a cron job with a signed payload that later will
# then invoke your api-side webhook function you will use this secret to sign and the verify.
# Important: Please change this default to a strong password or other secret
# WEBHOOK_SECRET=THIS_IS_NOT_SECRET_PLEASE_CHANGE
# Set API base URL to be localhost, not 0.0.0.0
RWJS_DEV_API_URL=http://localhost

Start It Up!

Now we're in the home stretch. Let's get your project running in its Development Container. Bring up the command palette using Command + Shift + P on Mac and Ctrl + Shift + P on Windows. Type in "reopen in container" and select "Reopen in Container" option.

This last step will take a little while. It will be downloading the Docker images for your development environment and Postgres, then building it. So sit tight, grab a cup of tea, and get a stretch break-in.

Once done, your VSCode should open up to what looks like a typical project, but with one secret. Your VSCode is connected to your docker container and running inside. The area you will notice the most significant difference will be the terminal.

It says you are inside the workspace folder, and your user is node. If you were to change directories and move up one, you would see completely different folders on your regular computer. You are now in your developer sandbox. Now run yarn redwood dev to start your server and see the magic happen. A bonus to all this work is that it will also work in Github Codespaces.

If there is anything still fuzzy or like me to elaborate further, let me know. I will make sure to do so.

Creating Your Own Custom StreamElements Widgets

Erik Guzman — Tue, 21 Jun 2022 17:58:03 +0000

This is a video walk-through explaining how you can create your very own Custom Stream Element Widgets.

Links:
https://streamelements.com
https://github.com/StreamElements/widgets
https://github.com/StreamElements/widgets/blob/master/CustomCode.md
https://github.com/StreamElements/widgets/tree/master/CustomChat

Deploying Elixir: Creating Your Own Elixir Package

Erik Guzman — Sun, 06 Mar 2022 18:22:57 +0000

During your journey as an Elixir developer, there is going to come to a point where you might want to publish your own package. This has happened to me a couple of times already, here are a couple of packages I have published.

Equip with the knowledge of what I needed to do to publish my own package, I thought I would take you on the same journey. Let's get started.

Our first stop is going to be the Hex website. Hex has a great guide for publishing your own package that we will be following.

https://hex.pm/docs/publish

Registering a Hex User

Before we can do anything, we will need to register our own user. When registering a user, you will be prompted for a username, your email, and a password. The email is used to confirm your identity during signup, as well as to contact you in case there is an issue with one of your packages. The email will never be shared with a third party.

$ mix hex.user register

Now it’s time to decide on a name for your package. In this guide I will be creating a new Ueberauth package. If you were to go on http://hex.pm and look at other Ueberauth packages, you notice there is a certain pattern followed. This will make the decision easy for us on what to call our Uberauth package that will implement the Patreon OAuth flow.

uberauth_patreon

Generating Our Project

Now that we know the name of our package let’s generate the project

If you’re curious how to generate a standard mix project, please visit the Elixir site here: https://elixir-lang.org/getting-started/mix-otp/introduction-to-mix.html#our-first-project

mix new ueberauth_patreon

We should expect to see:

* creating README.md
* creating .formatter.exs
* creating .gitignore
* creating mix.exs
* creating lib
* creating lib/ueberauth_patreon.ex
* creating test
* creating test/test_helper.exs
* creating test/ueberauth_patreon_test.exs

Your Mix project was created successfully.
You can use "mix" to compile it, test it, and more:

    cd ueberauth_patreon
    mix test

Run "mix help" for more commands.

Let’s change the directory to our new project directory and let’s open the code in your editor of choice. For me I’ll be using Visual Studio Code, this is important later in the guide when I am making a test app.

Next, we will update the mix.exs file to have all the settings we know to start things off. I follow this section of the Hex publishing guide https://hex.pm/docs/publish#adding-metadata-to-code-classinlinemixexscode

def project do
    [
      app: :ueberauth_patreon,
      description: "Ueberauth strategy for Patreon OAuth.",
      version: "1.0.0",
      elixir: "~> 1.13",
      source_url: "https://github.com/talk2MeGooseman/ueberauth_patreon",
      homepage_url: "https://github.com/talk2MeGooseman/ueberauth_patreon",
      start_permanent: Mix.env() == :prod,
      deps: deps(),
      package: [
        links: %{"GitHub" => "https://github.com/talk2MeGooseman/ueberauth_patreon"},
        licenses: ["MIT"],
      ]
    ]
  end

I decided to set my package version at 1.0.0 but feel free to start it at 0.0.1 since its your first version ever.

Adding Documentation Generation

If you follow along in the Hex guide, they will suggest using ex_doc and there is no reason not to. It generates some beautiful documentation.

Following the HexDoc’s guide https://hexdocs.pm/ex_doc/readme.html. Let’s add it to our project dependencies in our mix.exs file.

def deps do
  [
        ## Other deps ...
    {:ex_doc, "~> 0.27", only: :dev, runtime: false},
  ]
end

Dont forget to install our new dependency.

mix deps.get

Now let's generate our docs to make sure everything is set up correctly

> mix docs
Generating docs...
View "html" docs at "doc/index.html"
View "epub" docs at "doc/ueberauth_patreon.epub"

If you have the latest NPM installed you can run the following command to see your docs in the browser:

npx serve doc/

This will kick off a little webserver without having to go install anything locally.

Let’s Get Coding

When creating a Ueberauth package, the project needs to be structured in a particular way. If you click on any of the listed strategies Ueberauth has you will get a lot of good examples of what you need to do. You can structure your project any way you like but in the case of Ueberauth it’s best we follow their expected structure.

https://github.com/ueberauth/ueberauth/wiki/List-of-Strategies

Inside of lib, we need to create several files and folders.

mkdir lib/ueberauth
mkdir lib/ueberauth/strategy
touch lib/ueberauth/strategy/patreon.ex
mkdir lib/ueberauth/strategy/patreon
touch lib/ueberauth/strategy/patreon/oauth.ex

Since we're making an Ueberauth strategy we need to make sure to install Ueberauth and OAuth2 packages. The mix.exs file should start looking like this now.

defp deps do
    [
      {:ueberauth, "~> 0.7"},
      {:oauth2, "~> 2.0"},
      {:ex_doc, "~> 0.27", only: :dev, runtime: false}
    ]
end

Now let’s install those new dependencies.

mix deps.get

Now it's time to implement what we want our package to do. I am not going to explain how to implement an Uberauth package in this article, you can read my tutorial Creating Your Own Ueberauth Strategy on how. Instead, we'll just be copying and pasting some code in.

Inside lib/ueberauth/strategy/patreon.ex add the following:

defmodule Ueberauth.Strategy.Patreon do
  use Ueberauth.Strategy,
    oauth2_module: Ueberauth.Strategy.Patreon.OAuth

  alias Ueberauth.Auth.Info
  alias Ueberauth.Auth.Credentials
  alias Ueberauth.Auth.Extra

  @doc """
  Handles the initial redirect to the patreon authentication page.

  To customize the scope (permissions) that are requested by patreon include
  them as part of your url:

      "https://www.patreon.com/oauth2/authorize"
  """
  def handle_request!(conn) do
    scopes = conn.params["scope"] || option(conn, :default_scope)

    params =
      [scope: scopes]
      |> with_state_param(conn)

    module = option(conn, :oauth2_module)
    redirect!(conn, apply(module, :authorize_url!, [params]))
  end

  @doc """
  Handles the callback from Patreon.

  When there is a failure from Patreon the failure is included in the
  `ueberauth_failure` struct. Otherwise the information returned from Patreon is
  returned in the `Ueberauth.Auth` struct.
  """
  def handle_callback!(%Plug.Conn{params: %{"code" => code}} = conn) do
    module = option(conn, :oauth2_module)
    token = apply(module, :get_token!, [[code: code]])

    if token.access_token == nil do
      set_errors!(conn, [
        error(token.other_params["error"], token.other_params["error_description"])
      ])
    else
      fetch_user(conn, token)
    end
  end

  @doc false
  def handle_callback!(conn) do
    set_errors!(conn, [error("missing_code", "No code received")])
  end

  @doc """
  Cleans up the private area of the connection used for passing the raw Notion
  response around during the callback.
  """
  def handle_cleanup!(conn) do
    conn
    |> put_private(:patreon_token, nil)
    |> put_private(:patreon_user, nil)
  end

  @doc """
  Fetches the uid field from the Twitch response. This defaults to the option `uid_field` which in-turn defaults to `id`
  """
  def uid(conn) do
    %{"data" => user} = conn.private.patreon_user
    user["id"]
  end

  @doc """
  Includes the credentials from the Patreon response.
  """
  def credentials(conn) do
    token = conn.private.patreon_token

    %Credentials{
      token: token.access_token,
      token_type: token.token_type,
      refresh_token: token.refresh_token,
      expires_at: token.expires_at
    }
  end

  @doc """
  Fetches the fields to populate the info section of the `Ueberauth.Auth`
  struct.
  """
  def info(conn) do
    %{ "data" => %{
      "attributes" => %{
        "full_name" => full_name,
        "first_name" => first_name,
        "last_name" => last_name,
        "about" => about,
        "image_url" => image_url,
        "url" => url,
        "email" => email
      }
    }} = conn.private.patreon_user

    %Info{
      email: email,
      name: full_name,
      first_name: first_name,
      last_name: last_name,
      description: about,
      image: image_url,
      urls: %{
        profile: url
      }
    }
  end

  @doc """
  Stores the raw information (including the token) obtained from the Patreon
  callback.
  """
  def extra(conn) do
    %Extra{
      raw_info: conn.private.patreon_user
    }
  end

  defp fetch_user(conn, token) do
    conn = put_private(conn, :patreon_token, token)

    case Ueberauth.Strategy.Patreon.OAuth.get(
           token.access_token,
           "https://www.patreon.com/api/oauth2/v2/identity?fields%5Buser%5D=full_name,email,first_name,last_name,about,image_url,url"
         ) do
      {:ok, %OAuth2.Response{status_code: 401, body: _body}} ->
        set_errors!(conn, [error("token", "unauthorized")])

      {:ok, %OAuth2.Response{status_code: status_code, body: user}}
      when status_code in 200..399 ->
        put_private(conn, :patreon_user, user)

      {:error, %OAuth2.Error{reason: reason}} ->
        set_errors!(conn, [error("OAuth2", reason)])

      {:error, %OAuth2.Response{body: %{"message" => reason}}} ->
        set_errors!(conn, [error("OAuth2", reason)])

      {:error, _} ->
        set_errors!(conn, [error("OAuth2", "uknown error")])
    end
  end

  defp option(conn, key) do
    Keyword.get(options(conn), key, Keyword.get(default_options(), key))
  end
end

Inside lib/ueberauth/strategy/patreon/oauth.ex paste the following code:

defmodule Ueberauth.Strategy.Patreon.OAuth do
  @moduledoc """
  An implementation of OAuth2 for Patreon.

  To add your `:client_id` and `:client_secret` include these values in your
  configuration:

      config :ueberauth, Ueberauth.Strategy.Patreon.OAuth,
        client_id: System.get_env("PATREON_CLIENT_ID"),
        client_secret: System.get_env("PATREON_CLIENT_SECRET")

  """
  use OAuth2.Strategy

  @defaults [
    strategy: __MODULE__,
    site: "https://www.patreon.com/",
    authorize_url: "https://www.patreon.com/oauth2/authorize",
    token_url: "https://www.patreon.com/api/oauth2/token",
    token_method: :post
  ]

  @doc """
  Construct a client for requests to Patreon.

  Optionally include any OAuth2 options here to be merged with the defaults:

      Ueberauth.Strategy.Patreon.OAuth.client(
        redirect_uri: "http://localhost:4000/auth/patreon/callback"
      )

  This will be setup automatically for you in `Ueberauth.Strategy.Patreon`.

  These options are only useful for usage outside the normal callback phase of
  Ueberauth.
  """
  def client(opts \\ []) do
    config =
      :ueberauth
      |> Application.fetch_env!(Ueberauth.Strategy.Patreon.OAuth)
      |> check_credential(:client_id)
      |> check_credential(:client_secret)
      |> check_credential(:redirect_uri)

    client_opts =
      @defaults
      |> Keyword.merge(config)
      |> Keyword.merge(opts)

    json_library = Ueberauth.json_library()

    OAuth2.Client.new(client_opts)
    |> OAuth2.Client.put_serializer("application/json", json_library)
    |> OAuth2.Client.put_serializer("application/vnd.api+json", json_library)
  end

  @doc """
  Provides the authorize url for the request phase of Ueberauth.

  No need to call this usually.
  """
  def authorize_url!(params \\ [], opts \\ []) do
    opts
    |> client
    |> OAuth2.Client.authorize_url!(params)
  end

  def get(token, url, headers \\ [], opts \\ []) do
    client()
    |> put_header("authorization", "Bearer " <> token)
    |> put_header("accept", "application/json")
    |> OAuth2.Client.get(url, headers, opts)
  end

  def get_token!(params \\ [], options \\ []) do
    headers = Keyword.get(options, :headers, [])
    options = Keyword.get(options, :options, [])

    client_options = Keyword.get(options, :client_options, [])

    client = OAuth2.Client.get_token!(client(client_options), params, headers, options)

    client.token
  end

  # Strategy Callbacks
  def authorize_url(client, params) do
    OAuth2.Strategy.AuthCode.authorize_url(client, params)
  end

  def get_token(client, params, headers) do
    client =
      client
      |> put_param("grant_type", "authorization_code")
      |> put_header("Accept", "application/json")

    OAuth2.Strategy.AuthCode.get_token(client, params, headers)
  end

  defp check_credential(config, key) do
    check_config_key_exists(config, key)

    case Keyword.get(config, key) do
      value when is_binary(value) ->
        config

      {:system, env_key} ->
        case System.get_env(env_key) do
          nil ->
            raise "#{inspect(env_key)} missing from environment, expected in config :ueberauth, Ueberauth.Strategy.Patreon"

          value ->
            Keyword.put(config, key, value)
        end
    end
  end

  defp check_config_key_exists(config, key) when is_list(config) do
    unless Keyword.has_key?(config, key) do
      raise "#{inspect(key)} missing from config :ueberauth, Ueberauth.Strategy.Patreon"
    end

    config
  end

  defp check_config_key_exists(_, _) do
    raise "Config :ueberauth, Ueberauth.Strategy.Patreon is not a keyword list, as expected"
  end
end

It’s always a great idea to add doc headers to all your public functions you expect your package users to be using. This will help them immensely in figuring out what each function does and seeing examples of input and output. For this package, a user won't be using it directly but instead following specific setup steps, so our documentation reflects that.

Generate our documentation to see how things look now.

mix docs

And spin up a server to see how things.

npx serve docs/

Bonus: Add the README

This is more of a personal preference than anything else. I'm not sure if you noticed when browsing the Hex Docs for various packages. But sometimes you will come across a project that has pretty nice docs but the setup steps will be missing. If you actually visit the packages repository, you will find a great README with all the information on how to setup up the project.

GAHHHH I also need those steps in those docs! Well, you easily can tell ex_doc to include the README. Let’s update our project settings to do just that.

In our mix.exs file add the docs field, it should look something like this now.

def project do
    [
      app: :ueberauth_patreon,
      description: "Ueberauth strategy for Patreon OAuth.",
      version: "1.0.0",
      elixir: "~> 1.13",
      source_url: "https://github.com/talk2MeGooseman/ueberauth_patreon",
      homepage_url: "https://github.com/talk2MeGooseman/ueberauth_patreon",
      start_permanent: Mix.env() == :prod,
      deps: deps(),
      package: [
        links: %{"GitHub" => "https://github.com/talk2MeGooseman/ueberauth_patreon"},
        licenses: ["MIT"],
      ],
      docs: [ #### New docs field
        extras: ["README.md"] ### This is going to include the readme in the docs
      ]
    ]
  end

Now let’s go to our README and update it with the project setup steps.

# Überauth Patreon

[![Hex Version](https://img.shields.io/hexpm/v/ueberauth_patreon.svg)](https://hex.pm/packages/ueberauth_patreon)

> Patreon OAuth2 strategy for Überauth.

## Installation

1. Setup your application in Patreon Development Dashboard https://www.patreon.com/portal/registration/register-clients

1. Add `:ueberauth_patreon` to your list of dependencies in `mix.exs`:

    ```
{% endraw %}
elixir
    def deps do
      [{:ueberauth_patreon, "~> 1.0"}]
    end
{% raw %}

    ```

1. Add Patreon to your Überauth configuration:

    ```
{% endraw %}
elixir
    config :ueberauth, Ueberauth,
      providers: [
        patreon: {Ueberauth.Strategy.Patreon, [default_scope: "identity[email] identity"]},
      ]
{% raw %}

    ```

1.  Update your provider configuration:

    ```
{% endraw %}
elixir
    config :ueberauth, Ueberauth.Strategy.Patreon.OAuth,
      client_id: System.get_env("PATREON_CLIENT_ID"),
      client_secret: System.get_env("PATREON_CLIENT_SECRET"),
      redirect_uri: System.get_env("PATREON_REDIRECT_URI")
{% raw %}

    ```

1.  Include the Überauth plug in your router pipeline:

    ```
{% endraw %}
elixir
    defmodule TestPatreonWeb.Router do
      use TestPatreonWeb, :router

      pipeline :browser do
        plug Ueberauth
        ...
       end
    end
{% raw %}

    ```

1.  Add the request and callback routes:

    ```
{% endraw %}
elixir
    scope "/auth", TestPatreonWeb do
      pipe_through :browser

      get "/:provider", AuthController, :request
      get "/:provider/callback", AuthController, :callback
    end
{% raw %}

    ```

1. Create a new controller or use an existing controller that implements callbacks to deal with `Ueberauth.Auth` and `Ueberauth.Failure` responses from Patreon.

    ```
{% endraw %}
elixir
      defmodule TestPatreonWeb.AuthController do
        use TestPatreonWeb, :controller

        def callback(%{assigns: %{ueberauth_failure: _fails}} = conn, _params) do
          conn
          |> put_flash(:error, "Failed to authenticate.")
          |> redirect(to: "/")
        end

        def callback(%{assigns: %{ueberauth_auth: auth}} = conn, _params) do
          case UserFromAuth.find_or_create(auth) do
            {:ok, user} ->
              conn
              |> put_flash(:info, "Successfully authenticated.")
              |> put_session(:current_user, user)
              |> configure_session(renew: true)
              |> redirect(to: "/")

            {:error, reason} ->
              conn
              |> put_flash(:error, reason)
              |> redirect(to: "/")
          end
        end
      end
{% raw %}

    ```

## Calling

Once your setup, you can initiate auth using the following URL, unless you changed the routes from the guide:

    /auth/patreon

## Documentation

The docs can be found at [ueberauth_patreon][package-docs] on [Hex Docs][hex-docs].

[hex-docs]: https://hexdocs.pm
[package-docs]: https://hexdocs.pm/ueberauth_patreon

Generate our documentation to see how things look now.

mix docs

And spin up a server to see how things.

npx serve docs/

Our new package is code complete but we don't know if it actually works.

Testing Our Package In A Project

I think I write half-decent code, but not enough to trust that it just works so I want to try testing it in an actual Phoenix project. Let’s spin one up to test it LIVE.

mix phx.new test_patreon

Short Detour: Using VS Code Run Our Project

We will need a database for our project, the easiest way I like to do this is using Visual Studio Code’s Remote - Containers feature. Using Docker and an existing template, I can spin up a dev environment that all the fixings I need to work on a Phoenix project.

Start VS Code, run the Remote-Containers: Open Folder in Container... command from the Command Palette (F1) or quick actions Status bar item, and select the project folder you would like to set up the container for.
Now select Show All Definitions... > Elixir, Phoenix, Node.js & PostgresSQL (Community)
After picking the starting point for your container, VS Code will add the dev container configuration files to your project (.devcontainer/devcontainer.json).
The VS Code window will reload and start building the dev container. A progress notification provides status updates. You only have to build a dev container the first time you open it; opening the folder after the first successful build will be much quicker.
After the build completes, VS Code will automatically connect to the container.

Now your project is all set up with everything you need to run the project. You will most likely need to go into the docker-compose.yml file and update the database name to the name your project will be using.

Resuming: Testing Our Package In A Project

Let make sure our project setup works.

mix setup

In order to test our project, we don’t need to worry about publishing it to Hex. We can install it locally or install it through git. I have already pushed my project to Github so I want to show you how you can install any package hosted on Git or Github really easily.

Add the following to your deps. Your name or URL would be different if your project.

{:ueberauth_patreon, github: "talk2MeGooseman/ueberauth_patreon"}

As a personal rule of thumb, make sure to go through the setup guide you have for your package VERBATIM. This ensures you didn't miss any steps as part of the setup guide, because were the author sometimes we can accidentally gloss over an important.

Now install your package and test it out.

mix deps.get

Here is to hoping your project just works during testing it!

UH OH, My Package Has A Bug!

When the inevitable and your find a bug in your code there are a couple of things you can do.

Make updates to your package project code and push the changes to Git. If you do this, in your test project make sure to run the following command to install the latest updates.

mix deps.update ueberauth_patreon

Modify the package code directly inside your test project. All your installed packages can be found inside your deps/ folder and any changes you make to them can be recompiled so you can see how they affect your project. This can save you the trouble of having to go back to your package project, make code changes, and then push the code. Just run the following command to recompile your package.

mix deps.compile ueberauth_patreon

Make sure you copy the fixes you make back to your package project and commit them!

Submitting The Package

Now a package is complete and is ready to publish. Hex again does a great job detailing the steps you need to go through here https://hex.pm/docs/publish#submitting-the-package

Just run the command and confirm when you verify everything looks good.

> mix hex.publish
Building ueberauth_patreon 1.0.0
  Dependencies:
    ueberauth ~> 0.7 (app: ueberauth)
    oauth2 ~> 2.0 (app: oauth2)
  App: ueberauth_patreon
  Name: ueberauth_patreon
  Files:
    lib
    lib/ueberauth
    lib/ueberauth/strategy
    lib/ueberauth/strategy/patreon
    lib/ueberauth/strategy/patreon/oauth.ex
    lib/ueberauth/strategy/patreon.ex
    lib/ueberauth_patreon.ex
    .formatter.exs
    mix.exs
    README.md
  Version: 1.0.0
  Build tools: mix
  Description: Ueberauth strategy for Patreon OAuth.
  Licenses: MIT
  Links: 
    GitHub: https://github.com/talk2MeGooseman/ueberauth_patreon
  Elixir: ~> 1.13
Before publishing, please read the Code of Conduct: https://hex.pm/policies/codeofconduct

Publishing package to public repository hexpm.

Oh No I Messed Up My Docs!

Free not, for Hex has a way to directly publish doc updates. No need to make a new release just to update docs. Once you have made your docs updates just run the following command and you are good.

mix hex.publish docs