Forem: Corey Alexander

Introducing Byte Code Review Challenges

Corey Alexander — Thu, 05 Sep 2024 14:04:04 +0000

Hey Team!

Today I'm excited to launch Bytes!

Interactive coding games, with the first ones focusing on Code Review, which I think is a super important if under-appreciated part of software development

Here is the first game if you are too impatient to read farther: https://coreyja.com/bytes/level-0-0

This is a Bug Hunt, where you are presented with a code diff that contains in this case 2 bugs that you need to find and leave comments on! You'll be scored based on the bugs you find and how fast you are! There is even a leaderboard to see how you stack up against your friends!

Check out my introduction blogpost for the backstory and to learn more! https://coreyja.com/posts/introducing-bytes/

Introducing Bytes

Corey Alexander — Tue, 03 Sep 2024 00:00:00 +0000

Introduction

Hey Team! 👋

Been awhile; too long really! Life got in the way, but I’m back and ready to do more streams and write more posts.

And today I’m super excited to introduce one of the things that got be back and excited about making content again!

Today I’m launching Byte Challenges, or Bytes for short! These are going to be short ‘games’, designed to test your programming and code review skills. We are launching with a “Bug Hunt” but hope to have different formats in the future.

If you want to skip right to the game you can find the first Byte here! Be sure to checkout the leaderboard too, and see how you stack up.

Background and Concept

These challenges are using a platform one of my friends, Ali, is working on called, cookd!cookd as a platform currently focuses on code review and reviewing diffs. It’s built as an interviewing tool, and it thats interesting to you definitely reach out to Ali, he’d love to give you a demo! But the interviewing angle isn’t as fun to me personally, I’m here for the games!

I’ve always said that one of the most important skills for a software developer working with a team, is the ability to read code well and provide code review to their teammates. As I’ve become a more senior engineer I’ve seen how good Pull Request reviews can be a tool to teach and mentor others. But the flip side to that is reading code and reviewing others work, is also a skill that needs practice to get better.

So when Ali reached out to show me cookd and it’s focus on Pull Request reviews, I knew he was onto something and I wanted to integrate it into my site!

The Birth of Bytes

Ali explained the platform to me, but it wasn’t until I was solving my first puzzle that I fell in love with the format.

We were on a video call chatting about what each of us had been up to recently, and Ali shared a link and told me to share me screen when I clicked on it. He wanted to see my live reaction to what we’d been cooking up, and I don’t think I disappointed him. The premise of the game was simple, he had taken a bit of open source code me and our mutual friend Seif had been working on and hidden a few bugs in it.

Ali wanted to get my take on the platform, and see what I thought of everything. But I was way too distracted by the puzzle to care about anything else, it was a perfect Nerd Snipe for me. Especially with the bit of gamification thrown in; the counter on the side telling me how many bugs I had left to find and the timer constantly ticking down were all I could pay attention to. I turned on my internal ‘streamer mode’, and started thinking out loud trying to track down the third and final bug that was hidden. I was instantly hooked and after I failed to find the third bug we chatted about how I could start building these on my own and even integrate them into my site!

We talked about the types of bug that I thought were good for this format, and the ones I thought were less fun and more annoying. For example that third bug that I was searching for, was a syntax error with the Rust turbo fish operator. The code read something like collect::Vec<i32>() when it should have been collect::<Vec<i32>>(), note the missing <> around the Vet<i32>. While that was a bug in the code, it wasn’t one I would expect a human reviewer to catch. That’s the compilers job after all!

Ali had been working on AI assisted tools to help create these games at scale, but I knew I was more interested in making hard crafted artisanal challenges for all of you, and with the Bytes were born!

Introducing Level 0-0

Today I am very excited to introduce to you all the first Byte challenge! The first few of these will be in Rust, but I plan to branch out to other languages soon. Reach out and let me know what languages and types of puzzles you are interested in seeing!

The first puzzle is titled Level 0-0 and I think its a great into to the platform and concept. It is in Rust, but I promise none of the bugs will be syntax errors, and I tried my best to make it pretty language agnostic as well. Even if you aren’t super familiar with Rust I encourage you to give this one a try, I think you’ll be surprised how approachable it is!

I don’t want to spoil too much about this puzzle, you’ll just have to play it for yourself!

You can try your Bug Hunting skills on this first challenge here: https://coreyja.com/bytes/level-0-0And after you’ve given it your best shot, check out the Leaderboard to see how others have done!

Plans for “Bytes”

The plan is to publish these one a week, on Monday’s. (This one is coming out on Tuesday, because it was a US holiday yesterday and definitely not because I hadn’t written this post yet!)

On Monday’s a new Byte will be released, and I’ll spam it across my social media accounts!

To follow up on Friday’s I’ll post a video ‘wrap-up’ talking about this weeks challenge, and showing the answers. For the one’s my friends write, I’ll do a live attempt at trying to solve it myself as part of the wrap-up!

If you can’t wait till Friday to get the solution, consider sponsoring my work on Github Sponsors and you’ll get private access to the solution video as soon as the challenges come out on Monday.

The leaderboards will based on the honor system. If you’ve watched the solution videos, please don’t go solve the challenge to get a perfect score on the leaderboard. Don’t be the reason I need to implement safe guards to keep this fun for everyone!

Call to Action

And this is the part of the blog where I kinda want you to click away.

To try this first challenge click here: https://coreyja.com/bytes/level-0-0

Or to view the leaderboards head over here:https://coreyja.com/bytes/level-0-0/leaderboard

If you want to sponsor my check out my Github Sponsors Profile: https://github.com/sponsors/coreyja

And if you have feedback reach out! You can join my Discord sever and we can chat live: https://coreyja.club

Or you can send me an email or send a toot via one of my social media channels. I’ll be most active on Mastodon, but will do my best to respond on all platforms! https://toot.cat/@coreyja

And don’t forget to come back for the wrap-up video this Friday! If you want to be notified for each future Byte challenge, consider subscribing to my newsletter and each new challenge will land right in your inbox every Monday morning!

I’m really excited to see what you all think about these challenges! Please do reach out and share your thoughts.

In the meantime, I’m doing to be watching the leaderboard to see how everyone does! Good luck!

Easily use Github Avatars

Corey Alexander — Wed, 21 Aug 2024 00:00:00 +0000

tldr: You can use avatars from Github by adding .png to the end of a profile url. For instance this will show you my Github Avatar: https://github.com/coreyja.png

On stream on Sunday I was working on adding a Leaderboard for some coding 'games' that I'm going to be adding to the site soon! It's all based on Github usernames, and thats what I am displaying on the Leadboard. And I wanted to also include a Users github avatar!

I had assumed I was going to need to hit the API to get an avatar for the user, and probably cache that somewhere on my side to avoid needing to make an API call for each page view.

But it turns out there is an easier way!

Github makes it really easy to grab a Users avatar, no API calls needed! You can just append .png to a users profile URL to get their avatar. This worked great in an img HTML tag!

Here is an HTML img tag that will display my Github Profile avatar!

<img src="https://github.com/coreyja.png">

Make the Change Easy, Then Make the Easy Change

Corey Alexander — Fri, 26 Jul 2024 00:00:00 +0000

A manta I’ve repeated for a while in my software career has been Make the Change Easy, Then make the Easy Change.

This phrase is about adding new features to your code, but what it really gets at is the constant need to refactor your code to make it more amendable to the changes you want to make.

When you go to add a new feature to your codebase, you are often faced with a decision:

Hack in the needed functionality to the existing structure
Refactor some stuff to make the new feature fit nicely

Me and this mantra are advocating for the second option here. And that’s not to say that there aren’t times when the first “hacky” option is the right choice. But more often than not doing one hacky thing is a slippery slope to continuing to hack in new features, since each time you are in those code paths they are worse than the time before.

By refactoring BEFORE you add a new feature your codebase is getting cleaner and cleaner as you go! As opposed to creating more tech debt with each PR, you can actually clean up your codebase as you go! This will pay for itself in the long run, AND has the benefit of making the work more “predictable”. In a clean, well organized code base there are less chances of running into something unexpected that drastically shifts the timelines for a project. In a messy codebase it’s common to find issues buried, or things that take longer to untangle than expected.

Not only does cleaning up the codebase allow you to be more productive, it also helps be more predictable! And predictability is sometimes even more important than raw speed. Sometimes being finished a day or two earlier is not a deal breaker, but missing an agreed upon deadline because of a spagehti code bug put you off timeline can cause issues for the organization.

I'm a firm believer that in 90% of circumstances its better to refactor the code ahead of an expected change, instead of hacking it in and just kicking the cleanup can down the road.

Fallback Routing with Axum

Corey Alexander — Tue, 27 Feb 2024 00:00:00 +0000

Yesterday I ran into a small problem with my Axum routing for my blog. So today I wanted to take 10 minutes and do quick write up about it!

My blog uses Axum as its Web Framework of choice. I was looking through some old emails and found one where a reader reached out about a post I made! (It was this one about FZF Spell Checking in VIM)

But I realized that it was linking to an older URL format, The link was to https://coreyja.com/blog/2018/11/10/vim-spelling-suggestions-fzf.html!

And you can see I was using a different URL structure then, and my current blog didn't have a redirect setup!

And that was our goal! I wanted https://coreyja.com/blog/2018/11/10/vim-spelling-suggestions-fzf.html to redirect to /posts/vim-spelling-suggestions-fzf. And I also decided that I wanted blog/anything-else-here to redirect to /posts

Simple enough!

I added these two routes to my Axum Router function:

.route("/blog/:year/:month/:date/:slug", get(redirect_to_new_post_url))
.route("/blog/*catchall", get(redirect_to_posts_index))

And I expected this to work out of the box! But it didn't :sadnerd:

Instead I got an error about conflicting routes. Luckily the answer is relatively easy, at least in my case! I just needed to use nest to target all the /blog urls and then use a fallback. Here is what I ended up with!

.nest(
  "/blog",
  Router::new()
    .route("/:year/:month/:date/:slug", get(redirect_to_new_post_url))
    .fallback(redirect_to_posts_index)
)

And now I was able to get all the redirects working as I wanted! Go check out that 'old' URL above now and with any luck you should be redirected to the right post!

Video from Screenshots in Git History

Corey Alexander — Sat, 13 Jan 2024 00:00:00 +0000

Here is a quick view of this Rust powered site as its been built! It does that annoying jumping in the middle when I didn't have consistent font loading, all fixed now!

I've been using shot-scraper by simonw to take screenshots of my blog and save them to my repo.

That means I have a history of what my site has been looking in prod!

So I throw together a small Rust script calls out to ffmpeg that compiles all the screenshots I have saved into a short movie

I'll try to post a new one at the end of the year and we can see how it changes!

coreyja weekly - January 12th

Corey Alexander — Fri, 12 Jan 2024 00:00:00 +0000

Hey Team, let’s see if we can’t get back to doing these weekly!

[

Week Recap

](#week-recap)[

Three Pillars of a Web App

](#three-pillars-of-a-web-app)

I don’t think this was technically this week, but I hadn’t mentioned it here so might as well now!

I published a new Blog Post titled “Three Pillars of a Web App” of a web app, where I talk about my ideal web app structure. And the three processes to go into that: Server, worker and cron.

Check out the full post on my blog!

[

Sunday Stream

](#sunday-stream)

Watch the full stream on YouTube here

I did my first stream of the New Year last Sunday the 7th! Was good to get back to streaming more ‘normal’ projects after doing Advent of Code in December and then taking a bit of a break for the Holidays.

On this stream we worked on creating a small “framework” for my Cron and Worker processes in Rust. It’s following the same patterns from my “The Pillars of a Web App” post that I mentioned above.

We broke out a small crate called cja, which holds the code for our custom Cron and Job framework!

SideNote: I’m really happy with the cja name for that crate! Because it stands for “Cron, Jobs and Axum” and just so happens to perfectly match my initials!

We used this new cja framework to add Cron and Jobs to a new app I’m working on called status. That app isn’t ready for prime time yet, but it will be an uptime monitoring service that will hopefully grow into a more full features monitoring platform!

[

Things I Saw This Week

](#things-i-saw-this-week)

I’m still experimenting with the format of this blog, and this week I want to try a small section about news or projects that I came across recently that looked interesting!

`quickwit`

Quickwit is a project that came across my feeds this week, and I immediately knew I was going to have to check it out. They advertise themselves as: “Quickwit is the fastest search engine on cloud storage. It's the perfect fit for observability use cases”

This sounded perfect to me! Back many moons ago now I was working on a small side project to make log data cheaper to store for my scale of projects. It used S3 as a storage backend, and I think that direction is really powerful in terms of controlling cost especially!

And quickwit takes the same approach but are MUCH farther along than I ever made it!

It provides a pretty all included DB, specifically tailored to Logs and Traces. I’m pretty excited about it but haven’t tried it out at all yet. I’m thinking I might try to integrate it with status up above once that is off the ground a bit.

`Challenging projects every programmer should try`

https://austinhenley.com/blog/challengingprojects.html

This is a blog post from a couple years back now, and it’s simply a list of “challenging” projects for you to try and experiment with! I’m not sure I agree with the ‘every programmer should try’ bit, but if you are interested in them the challenges do seem fun!

I’m going to keep them in my back pocket for a rainy day when I’m bored. Definitely let me know if you try any out!

[

Github Sponsors

](#github-sponsors)

If you enjoy this newsletter and my other content, please consider sponsoring me on Github Sponsors!

Every sponsor helps me be able to continue to working on all these projects I love, and creating posts and videos for all of you!

I also want to give all my Sponsors a bit of a bonus for helping support me and my content. Right now if you sponsor me you’ll get access to a super special private channel in my Discord, where you’ll be able to interact with me directly and ask questions or get help!

And in the future I plan on giving Sponsors access to my side projects! status that I mentioned earlier might be the first example of this, and I will likely go back and get caje ready for users too and add that.

So, sponsor me on Github to have access to all my projects and apps! I haven’t decided on a price point for the “all inclusive” sponsorship yet, but if you sponsor before I get that added you’ll be grandfathered into “all access” for the life of your subscription!

Sponsor now!

Why can't I mutate this HashMap - References in Rust

Corey Alexander — Sat, 06 Jan 2024 00:00:00 +0000

Note: This is an older video I made for TikTok and never posted as a TIL so posting now

In a discord I'm in someone just posted this question about Rust.

I'm trying to iterate through the keys of a hash map in rust. And then if a certain thing is true, change the value of that key. I don't see why we need to clone the keys. Since I'm only changing values. Is there a way to get around cloning? And here's the code that they left in that snippet?

And the answer to the specific question was to you, something like iter_mut() to loop over the iterator and still be able to edit the data underneath. But that alone doesn't really explain why, why do we need to clone the keys before we can change the values of the map? If you don't do this, Rust will complain about having an immutable borrow to keys when trying to create a mutable borrow for the insert method call. And if you're not really used to references and Rust, that can sound like a bunch of gibberish.

So let's try an analogy. We have some data, let's say it's a list of numbers and we want to remember it. So we take our own personal whiteboard and we write down our list of numbers. And now we want to share this list with Frank. We don't want to give Frank our whole whiteboard. We just want them to have the list of numbers. So we take a piece of paper out and we write down the list of numbers. And this is our reference. Frank can take this piece of paper and with just that he can read the list of numbers that we have on our whiteboard.

And Frank can now share this with Kim. He can take a photocopy of this piece of paper, give it to Kim. And now Kim can see the list of numbers that we have on our whiteboard to. But what if we want to add a number to this list? We could just walk to our whiteboard and write a new number down. But now Kim and Frank's papers, aren't going to be right. They're not going to know about this extra number we added. And so Rust is going to stop us Rust. Isn't going to let us write on our whiteboard just yet. We have to make sure all of the pieces of paper, all of the references are cleaned up first. So if all of those pieces of paper are thrown away and Rust doesn't see any pieces of paper anywhere, now we can finally write on our whiteboard again and change the data.

So if we go back to the sample code that we were looking at before, when we asked for the keys that was creating a reference, we've got a piece of paper with all the keys in our hash map. So since we have that reference, that piece of paper, we can't add new things to the hash map and we can't change the values of anything. Because we have a reference to it. So, what we need to do is we need to get rid of that reference and how we are doing that is we're cloning it. So we take that list. That's on our piece of paper and basically we rewrite it down on our whiteboard and now it's a separate piece of data we could add to it separately than our hash. And then things like might be out of whack. We might have something in one list that isn't in the other, but as far as Rust is concerned, those are two completely separate pieces of data. So we took the reference and put it on our whiteboard, and then we threw away that piece of paper and got rid of the reference. And now we're allowed to mutate our hash map again, because there are no references hanging around. There are no pieces of paper. We just have the one data structure and we can mutate it as we, as we want.

Hopefully that analogy was helpful to someone. And this helps explain Rust references to you. Let me know in the comments, if this helped you out or if it didn't, I'd like to know how we can do something better next time. Thanks.

coreyja - 2023 Review and 2024 Preview

Corey Alexander — Sat, 06 Jan 2024 00:00:00 +0000

Hey Team!

It's the first week of 2024 and I wanted to write something to kick off the New Year! I'd also like to try and get one of these newsletters out each month, so let's call this the January Edition!

[

2023 In Review

](#2023-in-review)

2023 was an awesome year!

[

Lavender Iguana

](#lavender-iguana)

Brandi and I launched Lavender Iguana, our Design and Development business. And through that we've been producing our first podcast, which has been a learning experience and a great time! If anyone reading this is looking to make a podcast and doesn't know where to start, reach out and I'll connect you with Brandi to get started!

caje

caje was one of the most comprehensive projects we tackled this year! It all started with a comment on an existing video, asking about making a CDN in Rust.

YouTube Playlist

And from there we made eight full length streams where we build up caje from scratch in Rust!

We work on sitting in the middle between users and the origin servers, and saving any request we see to a file system based cache. But from there we had to make it a bit more global, after all this is supposed to be a Content Distribution Network, we can't be only in a single region! So we expanded caje to run in multiple regions, and share the state of all files in the cache with SQLite. We are using LiteFS to make our SQLite file distributed across our caje cluster!

Moodboard App

At the end of the year we started on this new app that Brandi is helping plan out and design!

The idea is to create an app to help designers and their clients get on the same page for imagery on the site! The goal is to build a "dating-app" like experience where the Client can swipe through a set of photos and choose if they do or do not like them for the aesthetic of their site or brand!

The YouTube Playlist for the build is here.

[

Advent of Code

](#advent-of-code)

And in December this year I started Advent of Code again! I say 'started' because, as is tradition, I only got part of the way through with doing these as streams. You can see my solutions for the first half-ish of December before I ran out of steam on YouTube.

[

Looking forward to 2024

](#looking-forward-to-2024)

While I'm sure I'll find many small projects to fill my time, there is one big one on the horizon I'm really excited to get started on and share more about!

In 2024 I'm going to be working on my video course currently titled "Writing a Web Framework in Rust"! The goal is to be a course for people who may not know Rust, but are interested in learning both about Rust and seeing how to build a Web framework from the ground up! We are going to only use the Standard Library and build everything up for ourselves. The first chapter starts by opening a TcpStream and looping over the lines until we can parse an HTTP message, I think you all are going to love it!

If you want to get a sneak peek at that course be sure to Subscribe to my Newsletter and/or Join my Discord.

I'll be giving my Sponsors a sneak peek while I record episodes so if you don't want to miss anything, be sure to help support me in making these videos and courses and sign up for my Github Sponsors!

My Three Pillars of a Web App

Corey Alexander — Wed, 03 Jan 2024 00:00:00 +0000

When I'm writing a web app there is an architecture I always gravitate too. It's simple yet scalable, and fits a wide array of use cases. It involves three distinct processes and one queue, in addition to whatever datastore you want to use. You can reduce the number of moving pieces by using your primary datastore as your queue, I often do this with Postgres.

As for the processes I like to call them server, worker and cron.

`server`

server is responsible for serving your web requests. This is probably the most familiar of the three services, and is what all web frameworks will provide. It listens for new HTTP requests, and returns responses to the browser.

You want to keep those web requests as fast as possible for a good user experience. And to do that you'll often have some work that you don't want to do during a web request. The common example is sending an email after signup. We want this to happen, but we don't necessarily want to make the user wait for the email to be sent to see the next page.

`worker`

And this is where the worker process comes in! This process is responsible for working of a queue of jobs. Each job should define its type and any arguments that are needed to run the job. The worker process sits in a loop looking for new things to be added to the queue and process them.

I often find it useful to give jobs a priority and work them off in some way based on that, but we'll call that an optimization. A first in first out queue is where I would start with if building a worker from scratch. Which is exactly what I'm doing for this site!

`cron`

And the final process is cron. Cron's job is simple, to put new jobs in the queue based on some schedule. I often have jobs that run on some interval, like every hour. And then some that run on a specific schedule, like every morning at 6am.

[

Horizontal Scaling

](#horizontal-scaling)

One thing I like about this approach is that is horizontally scales well! Both server and worker scale horizontally to fit demand. If you have more web requests, or more jobs you can simply spin up more server and worker processes. And importantly, you scale these two processes independently! If you have more jobs than web requests, you can keep your server process count consistent and increase your worker processes. Both of these also work well with auto-scaling if that suits the use case.

cron is the exception here. You do not want to naively horizontally scale your cron process, or else you'll end up with duplicate jobs in the queue. Luckily cron doesn't often have scaling concerns, as it's "only" responsible for looping and enqueuing jobs when needed. If you do need to horizontally scale your cron, you can find a way to partition the jobs and assign a portion to each process you spin up.

[

Guarantees

](#guarantees)

Another cool aspect of this is that you can have different durability guarantees, depending on exactly how you implement each process and which datastore you use.

For instance if you need guarantees about Cron jobs definitely being enqueued, you can write the state of the cron to your favorite data store and make sure that even if your cron goes down it can enqueue any jobs it misses when it comes back up.

You can use a durable store for your queue too, so you don't risk losing jobs. This is one reason I like Postgres queue. They are durable 'by default.'

[

Simple

](#simple)

These three processes are relatively easy to manage, and can be applied to just about every project. I've found that basically every web app I've needed to build fits nicely into this mold, and most don't require any additional infrastructure.

My goal with setting up these three processes early, is that it reduces decision making in the future. If you already have a job queue, it's easy to take advantage of it. You never have to rely on 'hacks' like sending emails in web requests, since it's easy to offload that work to the worker process.

Like I mentioned earlier I'm currently building out these three processes for this site! It's been a lot of fun to stand up these processes from first principles, instead of using frameworks. If you want to follow along, be sure to subscribe to my newsletter or checkout my Discord!

Autoformat in VSCode on Auto-Saves

Corey Alexander — Fri, 08 Sep 2023 00:00:00 +0000

I've had VS Code setup to format my code on save for a while now, and I love it!

Recently though I had a few files slip through my auto-formatting and be caught in CI. When I investigated it turns out that my VS Code settings were prevening auto-formatting from running on auto-save.

I had files.autoSave set to afterDelay and files.autoSaveDelay set to 300. Which meant that after 300ms my code would auto-save. However, auto-formatting would only run on manual saves, not auto-saves triggered by afterDelay.

The fix was pretty simple! I switched from afterDelay to onFocusChange! Now my files get changed anytime I switch to a different window or file, and even better auto-formatting runs as well!

Here is a snippet from my new config with a comment I left for future me:

"files.autoSave": "onFocusChange",
// By setting files.autoSave to `onFocusChange` the delay below isn't needed
// We need to use `onFocusChange` (or `onWindowChange`) since auto formatting isn't
// applied when using `afterDelay` :sadnerd:
// "files.autoSaveDelay": 300,

Sept 1st Update - Lockfiles

Corey Alexander — Fri, 01 Sep 2023 00:00:00 +0000

Hey Team! Its September now, so how was everyones August?

Today I want to talk a little about lock files, especially since the Cargo Team just updated their guidance about them in this blog post: Change in Guidance on Committing Lockfiles

This Week

We had two good stream this week!

Sunday, Aug 27th

Link: https://youtu.be/EBmDL9ZYI_Y

On Sunday we worked more on our Bot Byte! We fixed some issues I had introduced since the previous stream, and got Byte able to listen to my audio again nicely!

After that we made use of our new neon.tech powered DB to save and store chatters preferred nicknames. We also use this to customize how Byte pronounces your name! For instance I set my nickname to Corey Jay Aye so that it would pronounce my username correctly.

I'm really excited to keep improving Byte and using them as I stream! Let me know what else we should teach Byte to do on upcoming streams!

Wednesday Aug 30th

Link: https://youtu.be/oTS7LB2ChK8

On Wednesday we got back to my most requested topic recently, Building a CDN in Rust!

Previously we had gotten our server stood up to Proxy requests to our origin server, and cache each request without checking its cache ability, And on Wesnesday we fixed that! Now we check the Cache-Control and other relevant headers to determine if a resource is cachable. If it isn't we always make the live request from the origin server!

While streaming we found out that Chromium in developer mode always adds Max-Age=0 as a header, which was forcibly 'shutting down' my caching proxy! We switched over to Firefox and were better able to test the caching behavior.

The brains of our header checking were done by thehttp-cache-semantics crate.

We wrapped up this stream by making a plan for next time! We still want to add a 'manifest' of cached pages, and a way to share this manifest between nodes so that different nodes can cache the content without it being requested through them!

coreyja's Ramblings

Lockfiles

Today I want to talk about package manager lockfiles! In particular I'm going to focus on the two languages and package managers I have the most experience with Ruby and Rust.

Ruby with Bundler and Rust with Cargo have very similar philosophies. And that makes sense knowing that Cargo was written by some of the same people who developed Bundler for Ruby! Here is the announcement post for Cargo (on archived version) which mentions the dev Bundler roots.

Lockfiles are an important part of the Cargo and Bundler designs. In Ruby we have the Gemfile.lock and in Rust we have the Cargo.lock file. These files specify the exact version of your dependencies that you have installed.

When you add a dependency to Cargo or Bundler you specify a range of versions that your code can work with. In Bundler you might add some a range like this ~> 1.1.0. This range means that we need AT LEAST version 1.1.0, and that we also allow any version where the last digit is higher. So ~> 1.1.0 allows version 1.1.1 or even 1.1.99. Cargo does a very similar thing, but you don't need the ~> operator as this is Cargo's default behavior!

These version ranges live in either the Cargo.toml or Gemfile, and when you install the dependencies for the first time a lock file is created, that lists the exact version of the dependency that was installed. So if your version range was ~> 1.1.0 the lock file might specify 1.1.99 if that was the latest matching version at the time you installed.

If you committed your lock file to version control, any other developers on your project would get the EXACT same version of the dependencies. Even if 1.1.100 comes out, without updating the lock file everyone will still get 1.1.99 when they install dependencies.

And this can be really powerful! It allows all the developers to have the same version of every dependency so as to avoid issues cause by differences in versions. Or cases where something works for one developer but not for another. Having and committing a lock file can eliminate a whole class of issues in dependency management.

And when you want to start using a new version, you can have your tool of choice update the lock file to a new compatible version! And every developer will install that new version when they pull that lock file from version control.

What I described above is considered the "best practice" for applications, where the final result is an application to run and not a library.

Libraries are a bit different.

See in a library you don't have as much control over what versions of dependencies are installed. This is because these package managers do not take lockfiles into account when installing a library. This is often something that trips people up at first when they haven't run into it before. But libraries only use the version ranges in your Cargo.toml or Bundler file to decide what version of dependencies to install. This is because both of these package managers try to install dependency versions that work for your whole tree of dependencies. This is more exaggerated in Ruby and Bundler, where you can only have a single version of any dependency.

Let's say we have 3 gems (what Ruby calls packages). base_library has no dependencies. lib_a depends on base_library ~> 1.0 and lib_b depends on base_library <= 1.2. Since we can only have one version of base_libary installed we have to find one that matches both ~> 1.0 and <= 1.20. This might be 1.2.0 which satisfies both ranges. If thought lib_b depended on base_library < 1.0 we'd have a problem. There isn't any version that can match both ~> 1.0 and < 1.0 at the same time. Here Bundler will blow up, saying it couldn't find matching dependencies.

Cargo is a little more lenient here as it allows multiple versions of the same dependency to be installed. Though you might run into issues later if you try to pass Structs from one version to methods from a different version.

Since libraries don't use the lock file for resolving dependency trees, there is debate on what you should do with lockfiles in libraries.

Until recently the Cargo teams advice has been to commit lockfiles for applications, but NOT for libraries.

This is so that when CI runs, it will always pull the latest possible version of your dependencies. This lets you know what a user of you package might get if they installed it for the first time. It can help find incompatibilities and bugs with the newest versions of your deps.

But of course, this does mean that different developers working on the same project won't have the same exact versions of things. Each individual will have their own local lock file.

And Cargo recently amended their stance to say that it's ok to commit the lock file for libraries too, and to consider it for your project. This is a very reasonable answer because each project is different. On some projects keeping all the developers on the same version might outweigh the benefits of potentially finding issues earlier in CI. The "don't commit" for libraries advice also kinda implies that you have a working CI that could potentially catch issues early. If you don't then some of the benefits of not committing your lock file are lost. And on the flip side it's possible to get the benefits of not committing your lock file in other ways. I've recently heard of a project that commits their lock file, but has an automation that periodically updated the lock file with the latest releases. Or a different project that has two CI jobs setup, one that uses the committed lock file and one that does NOT and grabs the latest compatible versions.

I like this new advice in that it acknowledges that there are tradeoffs that might not apply to everyone. However I do like that the old advice as a bit more universal. Universal standards are nice in that they can reduce the number of things someone starting out has to think about.

And I'm really glad to hear from different people around the internet, and hear their tips and tricks for dealing with dependency versioning!

I was firmly in the "don't commit for libraries" camp, and think I still lean that way. But I am very intrigued by the idea of committing the lock file and having automation to update it on some cadence. I think I'll play around with this more sometime!

And finally there is one piece to this puzzle that I didn't get today and that is Cargo Workspaces. These allow you to bundle a bunch of crates together into a workspace, and have a single lock file for the whole workspace! This is really nice and has quickly become my default for Rust projects. But it is very interesting as it relates to this topic, cause now you have a single lock file potentially for both apps AND libraries. And I think in this case the tradeoffs are different. So far I've always committed lockfiles in workspaces, and I think that makes the most sense. Since there are applications in my workspaces, it makes sense to lock the deps for those versions. This works great for me because most of the library style crates in my workspaces aren't published externally so their versioning isn't super important. But I think I want to expand on this topic focusing on workspaces for an upcoming newsletter, so be on the lookout for that!

And with that I'm going to wrap up for the day. Thanks for reading team, and see ya online this week!