Forem: Noel Worden

Don't nuke your test ENVs

Noel Worden — Wed, 05 Nov 2025 02:50:19 +0000

NOTE: I know that modifying environment variables in tests is a controversial technique, but we do it on this project, so we have to do it properly.

This week I learned that Application.put_env/4 can cause flaky tests, both when running sync and async.

I was building out a test for a module that pulled data from config.exs. It looked like this:

and I set data as module attributes:

When I started running the tests, I was getting intermittent failures. After some digging, I found that when the tests failed, the returned config list was not what I expected, it only contained a single key/value pair:

I was using a new (to me) mocking library for my tests, so my first thought was that I was somehow overwriting the configs, or that the mocks were using a different set of test-specific configs. I tinkered with that, but it wasn't the case. Then, in the process of trying to narrow down the variables, I changed the test from async: true to async: false. I have had experiences in the past where async tests have data read/write conflicts, which I thought setting my test to async: false would eliminate. But that didn't improve the flakiness, in fact, it made it worse. With no luck fixing the flake, I pinged my team, and got a great piece of advice:

"...any other test does an Application.put_env and you're in for trouble"

Lightbulb moment.

I noticed a private function in another test that was modifying this same config. At first glance, it looked like it reset the config to its original state. The culprit:

To me, what it looked like that private function was doing was:

Grab that particular key/value pair
Update the value
Return that updated key/value
After the test completes, reset the key/value to its original state and insert it back into the list of MyApp.Example configs.

But it's a bit deceiving, because the call:

Does not merge into the existing config, it overwrites the entire config with only the key/value passed.

So, the on_exit function reset the entire MyApp.Example config to:

Which meant that if my test ran after the test containing this private function, mine would fail because the necessary config key/values did not exist. And, my debugging attempt to change my test to async: false didn't help, because our non-async tests generally run at the end of the test suite. The fix was to refactor that private function to restore the entirety of the original config in the on_exit block, like this:

After I refactored that private function, I ran a script to execute the test suite 100x, and I saw no failures. Flake fixed!

The lesson learned: If you must modify envs in tests, isolate them carefully or restore the full config explicitly to avoid cross-test pollution.

Elixir/Phoenix Project: Import CSV & Seed Data

Noel Worden — Tue, 01 Dec 2020 15:32:27 +0000

Goals

Seed the destinations table

I finished setting up the first schema with everything needed to CRUD (create / edit / update / delete) a record in the destinations table. The views and forms are functional, but they definitely need some love, particularly the index view. But before I dive into any styling, it would be beneficial to get some data into the index table. I could have used the app's newly created html forms and generated the data one record at a time, but its far more efficient -and beneficial down the road- to utilize the seeds.exs file. Using this method, the database can be populated quickly and consistently.

Seed the `destinations` table

Phoenix ships with a seed file, located at:

priv/repo/seeds.exs

Functions in the seed file don't have to be complex, it's simply an automated way to insert data into the table. A lot of times the data being inserted is fake, and primarily used as a kind of placeholder. But in my case I already had a spreadsheet with all the data and figured that since I was going through this process to inject data, it might as well be real data. Taking the time to write functionality to ingest data from a spreadsheet would also allow me to batch upload data through the browser at some point, which could be handy. Lastly, this kind of spreadsheet ingestion was handled in my last work project, so I had some experience with it, as well as some existing functions to guide me through the process.

Because of the potential dual use of the ingestion code I would be writing, I created a new CSVUtil.ex file, and housed it under the Mapping context. I did a bit of digging to see what libraries were available for processing a CSV (that's the format I exported the spreadsheet to for ingestion), and NimbleCSV was the clear front runner. I wasn't surprised, as that's what we used on my work project.

Knowing that I would use the same library as I already had experience with, I took an iterative approach to writing the functionality. Basically, what happened over the course of a day was:

Starting with the work project functions as a rough scaffold, I implemented my own version of CSV ingestion.
My initial version was a little verbose and clunky, but functional.
I started refactoring my work and seeing how and where it was similar to the work project code.
Ultimately I ended up with a good bit of similarity to the work project code, but with a much better understanding of what was happening from start to finish.

I will be discussing the end result of that process shortly. But first things first, the NimbleCSV library needed to be added to the dependencies in mix.exs:

Then I fetched the new dependency with:

mix deps.get

Once that finished, I rebuilt the docker image:

docker-compose build

And, finally, spun it up again:

docker-compose up

With the library added to the project, I added an alias of the implementation of NimbleCSV I planned on using in CSVUtil.ex:

This allows me to just call CSV.<function> instead of NimbleCSV.RFC4180.<function> every time.

Pulling from my previous experience, I knew I needed to either hardcode the CSV headers or grab them from the file. Initially I hardcoded them, but for future versatility I went with this function that extracts the headers and maps them to an index:

The breakdown of that function:

This takes in the CSV file and streams it line by line, which is perfect creating a record from each row of the CSV. This function alone only returns a struct with data about the file, its best utilized when another function is piped into it.

This tells File.stream!() to break that file down line by line, then parse it into a stream of rows. The default action of parse_stream is to exclude the headers, so I need to explicitly state that I want to include the headers here. The rows aren't user-facing until they are looped over.

This is returning only the first row of the stream, and dropping the rest. At this point the results start to take shape into something usable:

Next is:

This takes that list of strings and creates a list of tuples, the string as the first element and its index as the other, like this:

And lastly:

This converts the list of tuples into a map, using the index value as the key and the string as the value:

The need for the indexing and conversion to key/value pairs will reveal itself in the next step.

The next function, csv_row_to_table_record/1 calls the first, sets it as a variable, and then works through the rest of the CSV file, to insert it into the destinations table:

I'll skip the first two pipes, since they were covered above, and start with:

That is taking each row, and adding an index to each field in that row. A small sample of the output looks like this:

Then there is:

Which is taking each of those rows, creating a new map from it, and using the indexing to match the header value from column_names to the row value, the result of which is:

And, finally, there is a custom function to use that map to insert the data into the table:

The goal with this function was to be able to update that CSV, execute the seeding file, and only have the new files be added to the database, so the standard create function:

Wouldn't work here, because it would add the entirety of the CSV to the table each time, instead of only the new records. This is what the custom function looks like:

The argument row is the map created in the pipe above the calling of this function. The case is searching the destination table for the lat/long coordinates created in the call above. If that search returns nil, the map and Mapping.create_destination are utilized to create and insert a new record. Because of the header/value pairing that happened in the creation of the map, I can reference the value of the fields with this syntax:

If those coordinates do exist, it will return a tuple instead of creating a new record.

This logic might be easier to parse when seeing everything at one time, here is the file in its entirety:

The spreadsheet I've been using to hold my data up to this point is a Google Sheet. I exported it out as a CSV, and created a new folder at:

priv/repo/data

To store this file an any potential future files of this nature.

With all that in place, the code in seeds.exs is actually only one line:

And, as the notes in the seed file mention, that can be executing by running:

Closing Thoughts

This was definitely a more complex data seeding than is typically performed as the first seed of a new project. It would have taken a good amount of time to manually create dummy records, and if I ever needed to drop the database and rebuild it, I would have had to manually insert records again. Instead I took a bit more time and created reusable logic that can seed the database with actual records in quite literally the blink of an eye. I also now have the foundation to do more work with CSVs, which seems like a feasible option for mass uploading in the future.

As always, here is my repo at the end of this block of work.

References

This post is part of an ongoing series. I encourage any critique, feedback, or suggestions in the comments.

Elixir/Phoenix Project: Create First Schema & Run Tests in Docker

Noel Worden — Tue, 10 Nov 2020 15:50:58 +0000

Goal

Create Destinations schema under Mapping context

Create Destinations schema under Mapping context

The goal for this next step was to establish the first schema in the project, with the intention to then write the tests, and finally the views. In my previous projects I had always done this by hand, starting with the migration, then the schema, and then the context; and that was the plan for this work as well. But, plans changed, pretty dramatically. Having the time to explore and hopefully improve on my workflows and knowledge base, I spent some time researching before diving into the code. That research -unsurprisingly- led me to the the Phoenix Context docs, which, led me to the phx.gen.html generator. I've known about generators, but I had no idea that a generator existed that did so much.

Initially I was on the fence about using the generator, on one hand it is incredibly convenient for the framework to generate every file necessary to get a schema running, tested, and browser-ready. On the other hand, having all the work done automatically can lead to problems down the road, like debugging, because I'm not as intimately familiar with generated files as I would be with files I typed out by hand. Similar to the common idea that most people learn coding principles much better if they type it all out instead of copy/pasting it from the source. Ultimately I decided to use the generator. I have enough experience with the files being generated that I didn't feel the need to write them all out by hand. I also felt that it would be beneficial to know all the necessary bases were covered, and have that as a pull request to utilize as a reference for future work that I may write without a generator.

The phx.gen.html generator takes four arguments. The first is the context module name, then the schema module name, third is the schema module's plural name (to be used as the table name), and, optionally, the last is the list of fields and corresponding types.

The schema I wanted to created had a good number of fields, my generation looked like this:

In that generation command, Mapping is the context, Destination is the schema module, destinations the table name, and lastly the list of fields and their types.

side note: That command could have been typed out as one long string, like:

But by using the \ as an escape character, the command can be broken out over several lines to enhance readability.

This one generation command created the context, schema, migration, all tests, and all views. The output showed all the files created and then instructions on two more steps needed to complete the process:

Executing the first step from the instructions created all the default the routes for this schema. For this case, the entirety of the default routes works because I intended on utilizing them all. I confirmed the routes generated as expected with this command:

docker-compose exec web mix phx.routes

Which returned:

The second step in the instructions was to execute the migration created in the generation, creating the destinations table in the database.

With the exception of including null:false to a few the fields in the migration, all files created by the generator were good to go. I have some work to do to the generated html files, but that will be covered by a ticket down the road. After executing the migration command, I can navigate to localhost:4000/destinations and I see the index page of that schema, with the ability to create a new record, as well as all the other record manipulating functionality I expect. It's actually pretty amazing how much was accomplished with that phx.gen.html command. There are several mix.phx commands, all of course do something different, and save a lot of typing. They can be found in the Mix Tasks section of the Phoenix documentation, or by typing mix phx in the terminal.

Bonus - Running tests in the Docker container

My initial plan was to separate out this work, but since the tests were included in the files produced by the generator I decided to tackle this and include it in the pull request. I hit some bumps in the road when changing the database configurations in test.exs, to get the tests to run within the container. The process was involved enough that it warranted its own post. The high-level overview is that the hostname of the testing database needs to be "db" to run the tests in a container, but "localhost" for the tests to run in CI. The solution was to use System.get_env/2 in the database configuration logic, and add an env field in the CI testing configuration.

The update to the elixir.yml file took this step:

And added an environmental variable of DB_HOSTNAME:

Then the hostname field of the testing database configuration found at test.exs went from this:

To this:

Where System.get_env/2 is looking for the environmental variable set in the elixir.yml CI file. When that environmental variable is present (when tests are being run in the CI) it sets hostname to the variable's value, "localhost", allowing the tests to run in the CI build. And if the environmental variable is not found, the hostname defaults to "db", allowing for the test to properly run in a container.

Closing Thoughts

With the completion of this work, I can now Create, Read, Update, and Delete (CRUD) records in the destination table. The index view is currently a little rough, because of the size of the table that is being displayed, but that task is for another time. I can now also execute tests in the Docker container with this command:

docker-compose exec web mix test

And the tests also run as part of the CI check.

Having a working schema sets me up well for my next task, which is to utilize the seeds.exs file to populate the new table with sample data.

This is what my repo looks like at the end of this block of work.

References

This post is part of an ongoing series. I encourage any critique, feedback, or suggestions in the comments.

Elixir/Phoenix Project: Dockerizing

Noel Worden — Wed, 14 Oct 2020 15:23:09 +0000

Goals:

Dockerize project
Update README with new instructions on launching app with Docker

Dockerize project

Although this is implied with all of my writing, it's especially true with Docker, so it bears to be said explicitly: I am by no means a Docker expert. What I will be talking about in this post are the steps I took to get my project running in Docker from online resources and my past experience. If there is any aspect of what I'm about to talk about that could be improved, I would greatly appreciate any feedback. With that caveat out of the way, here is what I did to build my app in Docker images.

Actually, before I dive into the files, here's a quick refresher on whats what in the world of Docker:

A Dockerfile builds an image
A container is started from that image
docker-compose.yml defines which containers (aka services) make up a complete app

The first step was to create a Dockerfile, which is a file of specifications that Docker uses to build an image. For my Dockerfile I used an image from Docker Hub that I have used in the past, alpine-elixir-phoenix. I have been happy with it, and I'm also familiar with how to setup a Dockerfile with it. I used the image docs as a starting point for my Dockerfile but ended up simplifying it a bit for my needs:

Heres the breakdown of the file:

This is declaring which version of elixir will be used.

This is executing the RUN command to create a new directory ,/app. Then WORKDIR is setting that newly created folder as the working directory.

This is executing the COPY command to move the mix.exs and mix.lock files from the local directory into the image directory, then using RUN to execute a mix command in the image directory.

Similar to the previous command, this is executing COPY to move the necessary assets from the local directory to the image, then RUN to execute an npm install --prefix into the folder on the image.

This copies the rest of the files and folders from the root of the local directory to the root of the image.

Lastly, an execution of CMD to start the server

With the Dockerfile complete, the next step was to create a docker-compose.yml file. Like I said earlier, this file defines the containers that will make up the complete app. There's a good amount of content, so I'll utilize comments in the example code to explain what is what, instead of listing everything out individually:

One last thing that I needed to do before taking this for a test run was to change to the dev.exs file. This change was not on any of the tutorials I used, and it caused me enough trouble that when I found the solution I actually wrote up a standalone post about it. The change is pretty painless, the hostname field of the Repo config needs to be updated from localhost to db to match the name of the container running the database:

With the two docker files created, the update to dev.exs, and Docker for Mac installed an running on my machine, the app is up and running with three commands (I recommend executing in two tabs of the terminal):

Terminal Tab 1
- docker-compose build
- docker-compose up

Once its spun up, the db container shows the following error:

Although the db container is up and running, the app is looking for the database that is declared in dev.exs, in this case atlas_dev. The third command will take care of this:

Terminal Tab 2
- docker-compose exec web mix ecto.setup

That command uses exec web to execute the mix command in the currently running web container. The ecto.setup command is from the mix.exs file, and runs a list of commands:

Although I don't have any migrations or seeds at the moment, I still use that command, it's mostly a muscle memory thing at this point.

With that command executed in a second tab, the logs in the first tab no longer display an error, and everything is ready to go. The app can now be viewed at localhost:4000. Of course, since I have yet to actually code anything in this new app, it's still the generic Phoenix startup page, but that will change soon.

Update README with new instructions on launching app with Docker

With the project Dockerized, I also updated the README with new instructions on how to spin up the app:

Spinning up project with Docker

Install Docker for Mac
In one terminal tab:
- docker-compose build
- docker-compose up
In a second tab, once the two commands in the first tab are completed:
- docker-compose exec web mix ecto.setup
If everything spins up with no errors, site will be live at localhost:4000

It's essentially the steps from above, but I believe that keeping docs updated is important, so it's worth noting as part of my process

Bonus - Ignoring files for a faster Docker build

A file can be created that is similar to what .gitignore does for git, but for Docker. This file is .dockerignore and Docker looks for this file before each build to see if there are files that can be skipped in the execution of the build. Skipping files creates a smaller image and therefore a faster build. I pulled the contents of my .dockerignore from the Distillery hexdocs:

Adding this file was one of my last steps in the process, and I kind of regret that, as the build is noticeably faster with Docker ignoring these files. Adding this earlier in the process would have saved me a lot of time while experimenting with the proper Dockerfile and docker-compose.yml configurations.

Closing thoughts

Docker will be the last devops aspect I am going to setup. I am ready to do some feature coding and get that default Phoenix splash page out of there. But, I am very happy that I have taken the time to set my project up with these tools and make the process as streamlined as possible right out of the gate. From here on out I can basically just concern myself with features and whatever random thing I want to try, and all the checks I have in place will keep my code tidy and easy to deploy.

This is my repo at the end of this block of work.

References

This post is part of an ongoing series. I encourage any critique, feedback, or suggestions in the comments.

Elixir/Phoenix Project: Establish Hosting Service

Noel Worden — Wed, 07 Oct 2020 14:24:44 +0000

Goals:

Establish a hosting service
Update README with deploy steps
Setup custom domain to point to new environment

Establish a hosting service

A project running locally on my machine is great, but it doesn't mean much if that's the only place it exists. It might seem a bit premature, considering that at this point my entire project is still the default Phoenix splash page. But getting a hosting environment setup early means that new features will be able to go live as soon as they are merged. I started the process by digging around the web to see what others had done (noticing a pattern yet?). And I was a bit surprised that there is a whole service targeted specifically to hosing Elixir/Phoenix projects that I had not heard of: Gigalixir. What I was not surprised by was that there is an entire Deployment section of the Phoenix hexdocs, and in that section there is a whole subsection of how to deploy to Gigalixir.

But, before I go any further, I'm sure someone out there is thinking "why not Heroku?". In a nutshell, Heroku is not setup to give access to all of the features of an Elixir/Phoenix app, like node clustering. Not being partial to any particular hosting service, I figured why not setup my hosting with a service that is made for Elixir/Phoenix?

I ended up walking through the Gigalixir Getting Started Guide, and I have to say, the process was pretty painless. The Gigalixir docs are very detailed, and searchable. Because of the level of detail in their docs, I'm not going to write out my entire process, because I literally just followed the docs through step by step. I will note that I got a bit hung up in the Specify Versions section, because my local versions of elixir and erlang were newer than what were specified in the docs.

My local:
elixir -> 1.10.4
erlang -> 23

Docs specs:
elixir -> 1.10.3
erlang -> 22.3

I attempted to declare these new versions in the elixir_buildpack.config file that Gigalixir uses to set environment versioning, but when I tried to push up to the service was giving an error stating that those versions couldn't be found. I tinkered with things for a bit, but the only way I could get a push to this new staging environment to be successful was to use the environment specs given in the docs. It's definitely less than desirable to have a local version be different than in a staging environment, but it's a minor version difference, and I know that my next step in this app is to dockerize everything, and there I will take the steps to ensure the versions match.

Another thing to note is that while setting up my Github repository I switched my primary branch name from master to main. It seems that Gigalixir has not quite caught up with this optional change, and in the Deploy step it has this:

git push gigalixir master

But, thats basically shorthand, stating that master from Github should be pushed to master in the staging environment. Since I don't have a master branch, I get the following error:

The fix is pretty simple, it just requires a more verbose command:

git push gigalixir main:master

That command is stating that the main branch of Github should be pushed to the master branch of staging

One last note that was not noted in the Getting Started Guide is that the remote can be renamed. By default Gigalixir named the remote gigalixir but I am more used to referencing remotes by their purpose, like staging or production. So I ran this command to update the name of my remote:

git remote rename gigalixir staging

Although there were a few small hiccups, all in all the process to setup a staging environment for the project was relatively painless. I have setup a few Heroku environments back in the day, and this process seems remarkably similar. I would strongly recommend at least skimming through the entirety of the Gigalixir docs as they are chock full of useful information.

Update README with deploy steps

In the vein of trying to keep within the ream of best practices I felt like it would be beneficial to document the deploy steps in the README. I've been on more than a few projects where I was the second or third engineer onboarded and there was no documentation on how to perform relatively essential tasks. This is what I included in the project's README:

Deploying to staging

Hosted by Gigalixir, docs can be found here

It should be noted that the docs are very good, and searchable
Once the CLI is installed, gigalixir help is also very detailed

Project URLs

Install Gigalixir CLI (with homebrew)

brew tap gigalixir/brew && brew install gigalixir

Log into Gigalixir

gigalixir login

Verify login was successful

gigalixir account

Setup git remote

Navigate to project folder
gigalixir git:remote atlas-staging

Confirm remote

git remote -v

If desired, rename remote (which defaulted to `gigalixir`) to `staging`

git remote rename gigalixir staging

Push to staging

Any branch can be pushed to the staging site
Branches need to be pushed to Github before being pushed to staging
- git push staging <branch name>:master

Main vs Master

This project's main branch is named main, but Gigalixir is expecting it to be named master
- Because of this, when pushing main to staging, it cannot be done with the shorthand: git push staging master
- Until a fix is made by Gigalixir, the following syntax must be used to push: git push staging main:master

Executing `mix` Commands in Staging

The app is currently setup to be able to run mix commands on the staging environment

The `run` command

This can be used to run a shell command as a job in a separate container
gigalixir run ...

Using `run` and `mix`

Any mix command available locally can be executing on staging utilizing run
gigalixir run mix <command here>

Executing a migration

gigalixir run mix ecto.migrate
The logs do not automatically show, to view the logs
- gigalixir logs

Viewing the app

To open the app in a browser

gigalixir open

With this information, along with the slightly tweaked default Spin up the Project steps, a new developer should be able to pull down the project, get it spun up, and push work to staging all from the docs. I don't really plan on anyone besides me working on this in the foreseeable future, but solid documentation should be the goal of all projects, right?

Setup custom domain

Before I even started this project I dug around for a domain that I felt would be appropriate. It would be a bummer if I got 3 months into this project and I couldn't acquire a domain that had a name is a similar vein. I was able to purchase anglingatlas.com, so I named the project and repo atlas (I didn't want to type out AnglingAtlas every time I had to reference something in the project).

Gigalixir makes it pretty easy to assign a custom domain to the staging (or any other) environment. The one-liner can be found here. I should also point out that the Gigalixir web interface is pretty user-friendly as well. Now my staging environment can be found at https://staging.theanglingatlas.com/

Closing thoughts

I kind of gravitated to all of these devops tasks because it's an aspect of projects that I haven't dealt with directly in quite some time. I was curious what it actually took to get all of this stuff in place. And since I was so far down this road I figured I might as well get it all taken care of. Once this is all squared away, I'll be able to just worry about the features. The last piece of this (I think) is going to be dockerizing the project.

There aren't a ton of changes in the repo since a lot of this section was configuring Gigalixir, but here is what it looks like after all of this configuration.

References

This post is part of an ongoing series. I encourage any critique, feedback, or suggestions in the comments.

Elixir/Phoenix Project: Ticket Workflow & CI Integration

Noel Worden — Tue, 29 Sep 2020 14:11:17 +0000

Goals:

Create ticket workflow, convert notes to tickets
Integrate CI into the project

Create ticket workflow, convert notes to tickets

Although I am working on this new project solo and for the foreseeable future, I still wanted to have a way to create and track tasks. Back in the day, I used waffle.io but that project appears to have been axed. I liked waffle.io because of its Github integration and because it was relatively lightweight. I've used Jira and Atlassian, but those project managers seemed a bit heavy-handed for what I needed. After digging around a bit and asking some colleagues I settled on Github's own Issues and Projects.

It seemed like a logical choice, the biggest win is that they are already a part of the repo, so its one less service I need to keep track of. It's also not any more complex than I need it to be, I can keep it lightweight and nimble. My current workflow is to first create an Issue ticket, mostly just a title, to get my thoughts documented. If I have notes at the time of creation I add them, if not I leave it blank. The order of the issue tickets is trivial, that's where Projects come in. Projects is basically a kanban board, like Trello, Jira, etc. My flow with the Project board is that I pull in 4-8 issue tickets at a time, so I can see a short runway, then move the ticket across the board as the task progresses.

I recognize that I'm using both Issues and Projects at their minimum capacity, but at the moment that serves my needs. After all, I can always expand and modify my workflow, but this is a good starting point. The list of Issues for my project can be found here, and the Project board here

Integrate CI into the project

This was a big one for me. A lot of the tutorials and walk-throughs I came across in Part 1 jumped from the creation of the project right into something like tweaking the CSS setup or generating the first migration. The longer a project goes on without incorporating some kind of CI (continuous integration), the bigger headache it is going to be to get the tests to pass. I feel like it's much better to get integration dealt with right out of the gate and hopefully never have to think about it again.

Github has made it a lot easier than I had remembered to setup CI. The entire process can be done in the through Github web interface by clicking Actions > New workflow. Github suggests an Elixir-based workflow, clicking Set up this workflow brought me to the Github editor, where it has a file named elixir.yml setup to be added to a new folder structure:

./.gihhub/workflows/elixir.yml

This is what the default file looks like:

Referencing a tutorial, I went ahead and added a formatting check to the end of the file:

And committed the file via the Github web interface. At that point this was the entire elixir.yml

On the first run of that CI configuration I got an error that my tests weren't passing. I did some research and found that I needed to add some code to include a database for the tests to run on:

So, now the updated elixir.yml looks like this:

With that database setup, the tests passed, and I had that very satisfying green dot next to the pull request. Currently the CI was confirming that my tests pass and that the code was formatted properly. I was also interested in a service to analyze the code for consistency. I accomplished this by adding Credo to the project. Once included in the mix.exs file, Credo can be run locally with mix credo and it can be incorporated into the CI by adding the following line to elixir.yml:

I added the --strict flag because I figure if I'm going to use a tool like this, I might as well use it to the fullest extent. With that addition, this is my final elixir.yml file:

To my surprise, Credo actually found a few syntax errors in the Phoenix-generated code, so I ended fixing those and pushing up a small refactor commit to get the CI green again. It's worth noting that the compile time of Credo has slowed significantly with modern versions of Elixir, as noted in this issue. I'm not downgrading my Elixir version or doing anything to handle this, as it still meets my needs, and I have the time to wait for the checks to complete.

Bonus - Include CI Status Badge in README

One of the articles I was referencing through this CI work also included an example of how to include the CI badge in the repo's README. I mean, why not? It will be pretty gratifying to see that green CI tag every time I'm in my repo. As far as the markdown goes, it's basically an image tag, being linked to a Github-generated svg. This is what my my image tag looks like:

I'll break it down, since it took me two commits to get it right:

A basic image tag in markdown is:

So, in the case of CI badge, the first chunk:

Is declaring that its an image, and the alt text is Elixir CI. For what its worth, it would also work if the alt text were blank:

And the second chunk:

Is the source of the image file. The URL breaks down as such:

https://github.com/noelworden -> my Github account
/atlas -> the project repo
/workflows -> the folder where the CI file is contained
/Elixir%20CI -> the CI file I'd like to see the status of
- It should be noted that its necessary to use URI-encoding to include the space in the name field of the file

I didn't realize this until I had already coded it wrong once and merged it (🤦‍♂️), but that image link can be tested and confirmed in any markdown previewer.

Bonus - Create Pull Request Description Template

A pull request template wasn't any part of my original plan for this block of work, but I'm still figuring out my commit and pull request workflow. I had merged a handful of pull requests to get all of this CI work up and running correctly and realized that I should have been including the Issue number in the pull request descriptions. Knowing that I would forget to include that detail in future pull requests I figured it would be worth my time to incorporate a pull request template.

The file is a markdown file, that Github will look for when generating a new pull request. For Github to recognize it, the new file needed to be created and named:

PULL_REQUEST_TEMPLATE.md

This file can be stored in any of the following places:

the project root -> atlas > PULL_REQUEST_TEMPLATE.md
in a docs folder -> atlas > docs > PULL_REQUEST_TEMPLATE.md
in a hidden .github folder -> atlas > .github > PULL_REQUEST_TEMPLATE.md

In my original implementation of this template file I put it in the docs folder. But, as I write this and am thinking about it more, Im going to move it to the .github folder. The .github folder already exists because of the implementation of the CI workflow, and it feels more appropriate to have this Github-specific file there than in a docs folder.

In my case, I just wanted to ensure I always included the Issue number, so my template for the moment is simply one line:

But, of course, a template can as detailed as necessary for the given project.

Closing thoughts

With these four items complete, I feel like my project is in the realm of best practices. I can create and track tasks, have continuous integration setup, and have the ability to run a linter locally. I can also see the status of the project's continuous integration at a glance, and have a simple pull request template setup. Since I'm in the mode of laying the groundwork for the project, the next step will be setting up a hosting environment.

This is what my repo looks like at the end of this block of work.

References

This post is part of an ongoing series. I encourage any critique, feedback, or suggestions in the comments.

Elixir/Phoenix Project: Project Setup

Noel Worden — Wed, 23 Sep 2020 15:04:45 +0000

Goals:

Create new Elixir/Phoenix project and get it running on my local machine
Initialize git and push to Github

Before I took the dive in creating my new Elixir/Phoenix project I did what I bet most people would do, and typed something like Creating Elixir Phoenix Project into my favorite search engine. That produced plenty of hits, and I looked at the first steps of a handful of them. Then I took a look at hexdocs. What is hexdocs? Hexdocs is the source of documentation for all the hex packages for Erlang (which is the base language for Elixir). Pheonix is a package for Elixir, and is managed by hex, and as a result has amazing documentation.

Create new project and get it running on my local machine

I took a look at the other blog posts and tutorials to just double check there wasn't something new or improved from the official docs that I should incorporate into my new project. It appears that everyone is going through the same steps as is recommended in the documentation, so thats what I did. If you are new to Elixir/Phoenix you might be surprised by just how good the hexocs are. In fact, they are so good that I'm not going to rewrite the step-by-step instructions, but instead simply include the links for installation and how to get a project up an running.

Initialize git and push to Github

With the new project working locally, I wanted to get it pushed up to Github. Again, great documentation for this whole process. First, I needed to initialize git in my newly created project folder. The documentation for that can be found here. Once that was setup, I needed to create a new Github repo to push this project to. Being that I was already signed into Github, I navigated to https://github.com/new. There are a few options when setting up a new repo, but I was fine just entering the new repo name and keeping it public. The Phoenix generator already created a README and .gitignore file, and if I decide I want a license I can add it later. Once the repo was created, Github provided instructions on how to link this new repo with the newly created project, I think this documentation is a bit more verbose in what needs to be done and why. Following those instructions, all of my Phoenix-generated files are in my newly created repo.

There isn't a ton to see, but this is what my repo looks after completing the goals of this post.

Closing thoughts

Now, I know it might seem to be a bit lazy to write an entire post about how I started my new project and basically rely on the language and service's documentation. But this particular documentation is so good and detailed, I don't see much benefit in me just rewriting it. And, if there's one piece of advice I would give a new engineer, it's that he or she should get comfortable reading and implementing from documentation. The next few parts of this project will be less out of the box, influenced by multiple sources of inspiration, and have more written guidance.

References

This post is part of an ongoing series. I encourage any critique, feedback, or suggestions in the comments.

New Side Project, New Blog Series

Noel Worden — Wed, 23 Sep 2020 15:04:10 +0000

I am currently in between projects at work, and I decided a good way to spend my time would be to spin up a side project. For nearly a year I have been compiling a dataset of my favorite GPS coordinates along with other attributes of the location, knowing that at some point, when I had the time, I could use it as a jumping off point for a project. There are some aspects of my recent work project I wanted to dig further into, as well as some things that didn't exist at all and I want to see what it would take to implement. The end goal of this project is more a sandbox to experiment with than a polished app. But, if at some point I get something that is worth sharing with the world, that would be great as well.

As with my other blog series This Week I Learned, I know that I learn a lot when I write about my process, so writing my way through this side project was an easy decision. In addition to expanding my knowledge, I hope that writing about my process can help someone else's process be a little bit smoother.

This side project and the articles that accompany it are all about learning, as such, I would greatly appreciate any feedback on how I might improve on the code I implement.

A Few Caveats to Running Elixir Tests in Containers and CI

Noel Worden — Thu, 17 Sep 2020 14:18:08 +0000

As a fitting follow up to last week's post dealing with proper database configuration and Docker containers, this week I ran into a related snag while trying to get my tests to run in the app container. When I ran the mix test command through docker-compose:

docker-compose exec web mix test

It failed with the following error:

Although this wasn't the exact same error from my previous experience, it's in a similar vein. I checked out the test.exs configuration, and low and behold, in the Repo configuration I found:

I updated hostname from "localhost" to "db" -the name of the database container- and re-ran:

docker-compose exec web mix test

It successfully compiled, ran the tests, and everything was good to go, or so I thought. After creation of the pull request the CI spun up a new check... and it failed. To my surprise, it was failing at the same step and showing the same error as I discussed in last weeks post, only this time instead of the error happening on my local machine, it was happening on the CI build:

After changing the test.exs configuration and pushing that up to the pull request to confirm, I concluded that the CI workflow needed the testing database hostname to be "localhost" while it also needed to be "db" for the tests to run in the app container.

The solution I found for this utilized System.get_env/1. I found a post that shows how to set configurations based on environment variables. When the app is being run in the CI workflow, there is a GITHUB_ACTION environmental variable set, so a conditional can be built around that:

Placing that conditional under the initial configuration will override the line hostname: "db" when the app is running in a CI workflow. The entire setup looks like this:

With that conditional set, the tests successfully run in the local app container using "db" and in the CI workflow using "localhost".

But there's more, a potential for refactoring! While researching and digging deeper into this, I came across a post that utlized environment variables in a slightly different way, which I actually preferred and refactored my work to emulate. He suggested setting an environmental variable in the CI workflow file itself, then injecting that variable as a conditional in the original database configuration. My original mix test command for the CI workflow was this:

Adding an environmental variable looks like this:

Now, in test.exs, instead of that entire conditional under the original configuration, the hostname field can be set conditionally as a one-liner:

But that one-liner can be refactored even further. Theres also a System.get_env/2 that allows for two arguments to be passed, the first is the environmental variable, and the second is a default value if that environmental variable cannot be found. So, instead of using || to set the hostname to "db" if System.get_env("DB_HOSTNAME") returns nil, it can be all be done in the single function call:

With that refactor both testing environments run error free, and the codebase is just a touch cleaner.

If you'd like to see the complete files, feel free to explore my public repo, linked to the point where this was work was merged in.

This post is part of an ongoing This Week I Learned series. I welcome any critique, feedback, or suggestions in the comments.

Docker: Connecting an Elixir/Phoenix App Container and Database Container

Noel Worden — Wed, 09 Sep 2020 14:54:56 +0000

While spinning up an Elixir/Phoenix side project I ended up being blocked for longer than I'd like to admit trying to get my app container and database containers to communicate with each other. The fix ended up to not having anything to do with the Dockerfile or docker-compose.yml, but in the dev.exs config file.

The default database configuration from the phx.new generation looked like this:

Which works fine when running the app locally, without Docker. But, when starting the app and database containers in Docker, I would get the following error:

What's happening is that the project container is looking for a database named localhost, but the database is no longer running locally nor by the name localhost. It is now running as a container by the name declared in the docker-compose.yml file. In my case I named it db, so I needed to update the hostname field of the database config to this:

Now that I have solved the problem, and know what to look for, I see that it is mentioned in the Docker docs. In hindsight I may have been too rabbit-holed in finding the solution to this problem in blog posts or step-by-steps that were specific to Elixir/Phoenix and Docker, although I am a bit surprised that this step was not mentioned in any of those sources. But, it is documented now, so hopefully I can help save someone else a a bit of time debugging their dockerized Elixir/Phoenix app.

This post is part of an ongoing This Week I Learned series. I welcome any critique, feedback, or suggestions in the comments.

The F*ck - A Great Little Productivity App

Noel Worden — Tue, 01 Sep 2020 14:21:52 +0000

Welp, things have been a bit slow on the project. The last few weeks have been a lot of rinse and repeat kind of stuff, but alas, that's how it goes sometimes. All that to say that there wasn't much in the way of This Week I Learned.

However, I do want to share what is probably my favorite productivity-related app: The Fuck. Now is probably a good time to mention that the 'F Bomb' will be dropped a few more times in this post, but it's all in good fun, so I hope it's not too offensive. Now, what is The Fuck, and why do I like it so much? From the docs:

The Fuck is a magnificent app, inspired by a @liamosaur tweet, that corrects errors in previous console commands

So, basically, if you drop a typo in a command, say, like:

git chekout develop

You would get a a nice error message from git, even suggesting the correction. But instead of having to type it out again, you can simply type:

fuck

And you are presented with what The Fuck assumes you meant to type initially:

git checkout develop [enter/↑/↓/ctrl+c]

From there you can either hit enter to use that suggested command, use the up/down arrows to cycle through the options, or ctrl c to abort.

But that is just one of the fairly large list of commands that The Fuck recognizes and can fix, a comprehensive list can be found on the incredibly well documented README.

Because the README is so detailed, I'm not going to retype all the intricacies here. I will say that the install is very simple:

brew install thefuck

And it is highly customizable. There is even an experimental instant mode, if that's your cup of tea.

This little app gets me to chuckle out loud just about every time I use it. It's kind of perfect, since I have been known to swear under my breath when frustrated, now I can take that energy and actually fix the problem! And, if you are someone who isn't a fan of typing out the 'F word' on the regular, you could always utilize this powerful tool from an alias, like, I dunno, duck?

PSA: Take Some Time for Yourself

Noel Worden — Tue, 25 Aug 2020 18:54:43 +0000

I was on PTO most of last week, so I don't have much of a This Week I Learned kind of thing. But, since I'm trying to write every week I thought it would be worthwhile to type out a friendly reminder:

take time for yourself

My supervisor said it best, "you're not working from home, you're living at work". Whether it is during the regular work day, or during a side project, remember that it's ok to take a break. Go for a stroll around the block, around the neighborhood, or until your feet are sore!

I promise you will be happier and more productive after you allow yourself some time to breath.

Forem: Noel Worden

Don't nuke your test ENVs

Elixir/Phoenix Project: Import CSV & Seed Data

Goals

Seed the destinations table

Closing Thoughts

References

Elixir/Phoenix Project: Create First Schema & Run Tests in Docker

Goal

Create Destinations schema under Mapping context

Bonus - Running tests in the Docker container

Closing Thoughts

References

Elixir/Phoenix Project: Dockerizing

Goals:

Dockerize project

Update README with new instructions on launching app with Docker

Spinning up project with Docker

Bonus - Ignoring files for a faster Docker build

Closing thoughts

References

Elixir/Phoenix Project: Establish Hosting Service

Goals:

Establish a hosting service

Update README with deploy steps

Deploying to staging

Hosted by Gigalixir, docs can be found here

Project URLs

Install Gigalixir CLI (with homebrew)

Log into Gigalixir

Verify login was successful

Setup git remote

Confirm remote

If desired, rename remote (which defaulted to gigalixir) to staging

Push to staging

Main vs Master

Executing mix Commands in Staging

The run command

Using run and mix

Executing a migration

Viewing the app

To open the app in a browser

Setup custom domain

Closing thoughts

References

Elixir/Phoenix Project: Ticket Workflow & CI Integration

Goals:

Create ticket workflow, convert notes to tickets

Integrate CI into the project

Bonus - Include CI Status Badge in README

Bonus - Create Pull Request Description Template

Closing thoughts

References

Elixir/Phoenix Project: Project Setup

Goals:

Create new project and get it running on my local machine

Initialize git and push to Github

Closing thoughts

References

New Side Project, New Blog Series

A Few Caveats to Running Elixir Tests in Containers and CI

Docker: Connecting an Elixir/Phoenix App Container and Database Container

The F*ck - A Great Little Productivity App

PSA: Take Some Time for Yourself

take time for yourself

Seed the `destinations` table

If desired, rename remote (which defaulted to `gigalixir`) to `staging`

Executing `mix` Commands in Staging

The `run` command

Using `run` and `mix`