Forem: Kevin Cox

Losslessly Changing Video Framerate

Kevin Cox — Wed, 06 Aug 2025 00:05:00 +0000

I’ve been taking a few timelapse videos recently and as an ammeter I’ve just been using the default phone camera apps. Google Camera defaults to “Auto” for the speedup, which keeps the resulting video between 15 and 30s no matter how long you record for (with minimum of 5x speed). It also offers explicit 5, 10, 30 and 120x speeds. The iPhone forgoes explicit options and only offers an auto mode with 20-40s output target and a 20x minimum speed.

Both of these apps have a limitation that they can only output 30 fps video. The seems like a missed opportunity since the faster than real time playback already tends to produce jumpy video. Getting 60 fps (or even 120 fps) makes a huge improvement to smoothness. The hardware can easily support these output framerates as the capture framerate will still be lower than regular speed videos.

My preferred approach is to record the video at “twice the duration” as I actually want, then to speed it up after the fact. Recording at a higher input framerate always gives you more flexibility as you can always speed it up later, and if you carefully pick your input framerate you can do this losslessly to avoid degrading the video quality.

For example, if I want to end up with a 1 minute 60 fps video covering 1 hour of real time. I need to record at 60 s×60 Hz÷1 h60\u{s} \times 60\u{Hz} \div 1\u{h}which is 1 fps. Google Camera doesn’t show you the framerate directly, but you can easily convert the input speeds to framerates. Since it always expects to output 30 fps the calculation is just 30 fps÷speed30\u{fps} \div \v{speed}, so to capture at 1 fps I need to select 30x speed.

To losslessly convert the video you can use ffmpeg bitstream filters.

ffmpeg
    -i input-video.mp4 # Input file.
    -c copy # Don't transcode, remain lossless.
    -r 60 # Output framerate, this is mostly informational for the whole-video metadata.
    -fps_mode vfr # Don't drop frames to match the expected framerate, just preserve all input frames.
    -bsf:v "setts=ts=PTS/2" # Update the presentation time to 1/2 the previous one (play twice as fast).
    output.mp4 # Output file.

This can easily be adjusted to other framerates by changing the -r and -bsf:v values.

Note

The ffmpeg wiki has a page How to speed up / slow down a video. However this page only has a lossless method for h264 and h265. Furthermore this method makes it very difficult to preserve metadata, don’t handle variable frame rate properly and is multiple steps. I have found my method to be much easier and more effective.

I Don't Like Imports

Kevin Cox — Sun, 20 Jul 2025 22:15:00 +0000

I think this is an unusual opinion, so I thought I would share.

I prefer not to import external symbols into the local scope. For example, I prefer:

let body = reqwest::get("https://kevincox.ca/feed.atom").await?.text().await?;

over

use reqwest::get; // At the top of the file.

// Inside some function.
let body = get("https://kevincox.ca/feed.atom").await?.text().await?;

Unambiguous

There are various reasons for this, but all the most important ones boil down to being unambiguous. When I am reading code I see library::Foo and know that it comes from library. I see crate::Foo and know it comes from elsewhere in my code and if I see just Foo I know it is from the current file (and possibly private, so I could be a bit more careful with how I use the API). I don’t need to guess or lookup what Foo I am working with. (And when I do inevitably guess I don’t need to have that background process asking “did I guess wrong?”)

I find when I am working on codebases where the convention is to import things I am very frequently tracking down the definition of some type or function as I don’t know whether it is local or from which library. This is especially painful when reading or reviewing code in a web interface. If you are reading a patch the imports are likely not expanded (if they didn’t change) and even if you have the whole file expanded you need to check if the referenced value is the global import from the top of the file or some local shadow.

Copy + Paste

Another thing I love about fully qualified names is that I can just copy and paste code around or move it from file to file without needing to figure out which imports are needed in the new file (and inevitably importing the wrong Error at least a few times). For the most part code in other files and examples online can just be pasted into the file and work.

It’s not some dramatic productivity boon, but it certainly makes cleanups and refactors much smoother and a frustrating tedious step that is necessary when working with import-heavy code.

This would be a great editor feature. If I am copying from one file to another in the same editor it should be able to also pull in the required imports. Technically it should be fairly straightforward to any editor with enough language services for precise autocomplete. The only real challenge would be coming up with a new name in the case of a identifier conflict.

But The Names Are So Long!

In some cases they are. But more often than not this is just a poor choice by the library author (in my opinion). For example in Rust the standard hash-map is std::collections::HashMap. But even that is just an alias. If you want to work on an Entry you are now typing std::collections::hash_map::Entry. I see no reason it couldn’t have been std::HashMap and std::HashMapEntry or std::HashMap::Entry. (I don’t mind a small amount of nesting when it is very strongly associated, but still don’t really see much point.)

I much prefer C++‘s style where almost everything is just under std. For example std::unordered_map. An iterator is slightly nested at std::unordered_map::iterator but if you ask me that is still a pretty reasonable name. And I would much rather see that in code than wonder what iterator we are working with.

If it hurts stop hitting yourself, don’t find a way to numb the pain.

Conflicts

I see this come up as an argument against, and I am pretty dumbfounded. Some people see that you have std::io::Error and std::fmt::Error. So they can’t both be std::Error. But that is kind of my point! When I see code referencing Error I don’t know which of error type from the dozens libraries that the code might be using it is! That is what makes the {library}::{name} pattern so great, it is conflict free. If I use that throughout my code I never have to worry if Write is std::fmt::Write, std::io::Write or crate::Write because it is spelt out! I would much rather the types be actually named std::FmtWrite and have everyone “agree” on the name (because they have no choice) than half the time having it called Write and half the time having it called FmtWrite, WriteFmt or DebugWrite (because some other Write was in scope and the programmer needed to come up with their own name).

This isn’t an imagined problem. I regularly find myself reading rustdoc documentation and needing to hover over a type name so that my browser displays the fully qualified path in the link target! I feel that which Error or which Write is being used is important enough that it be immediately visible, in both code and generated documentation.

Avoiding imports means that the library authors are responsible for solving conflicts. And we are back to the fact that when I see a type or function it is unambiguous and I don’t have to waste brain cycles figuring out what Error I am looking at.

But Sometimes They Are Really Long

I’m not a stickler to this rule. There are some cases where I will use imports. Sometimes the names are just too long and repetitive. In that case I strongly prefer a local use with a short scope. This ensures that I don’t need to look far to resolve the ambiguity.

use sentry::integrations::tracing::EventFilter;
match *meta.level() {
    tracing::Level::ERROR => EventFilter::Event,
    tracing::Level::WARN => {
        if module.starts_with("hickory_proto::") {
            EventFilter::Ignore
        } else {
            EventFilter::Event
        }
    }
    tracing::Level::INFO => EventFilter::Ignore,
    tracing::Level::DEBUG => EventFilter::Ignore,
    tracing::Level::TRACE => EventFilter::Ignore,
}

(And more often than not when I run into this case it isn’t because the name is intrinsically high-entropy, but because someone got carried away with hierarchy. Stop hitting yourself.)

Rust-Specific Notes

I’ve been using Rust as an example but been trying to avoid being Rust-specific. But it is my primary language, so I do have some thoughts to share. If you don’t care about Rust you can stop reading now.

Avoiding imports in Rust is going against the grain. I would recommend that all crates provide a flat API. However, for many projects avoiding use isn’t the right choice. Just follow the common coding convention and be happy. To see a real-world example of both a flat API and avoiding use you can check out rl-core which is a small project where I am quite glad I chose to break convention. A lot of my “for-me” or “internal” code is written in a similar style.

Provide a Flat API

By default, Rust wants to ship your file structure as your exports (Conway’s law). So for example mylib::Foo needs to be defined in src/lib.rs and mylib::submodule::Bar needs to be defined in src/submodule.rs or src/submodule/mod.rs.

With a single re-export per module this can be solved!

mod config; pub use config::*;
mod tracker; pub use tracker::*;

One extra statement to free you from hierarchy. Re-exports are well-supported by Rust so rustdoc and the compiler will pick mylib::Config as the path to show users rather than mylib::config::Config. (I don’t know the exact heuristics but when working like this there will be a single public export, so there isn’t much cleverness needed.)

pub use also only sets the maximum export visibility. This means that pub, pub(crate) and private things in the module will have the expected visibility. So the definitions in the files look the same as they normally would and the export lines only need to be added when you add or remove modules.

The only real downside of this approach is the src/lib.rs will see everything as local and can reference Config rather than crate::Config. But this is a very minor issue and the solution is not to put anything into lib.rs. (Keeping only module definitions and re-exports in lib.rs is a very common pattern anyways.)

The nice thing about just re-exporting everything is that it also frees you from deciding where to put things based on what you want the exported name to be. You don’t need to put std::collections::hash_map::HashMap and std::collections::hash_map::Entry in the same file to have them appear at the same place, you can put them into separate files if you wish or the same file if you need access to non-public APIs. (The Rust std implementation does actually re-export these from various places, re-exports are very common and not something to be feared.)

Personally I just put every single type and most functions into separate files. I find that it keeps everything simple and organized and I prefer switching between different files than different parts of a file when going back and forth. So for example in rl-core I define rl_core::Config in src/config.rs and rl_core::Tracker in lib/tracker.rs.

Traits

Traits are where things start to get hairy. To call a trait method as a method the trait needs to be in scope. So if you want to call foo.a_method() where a_method is somelib::TheType::a_method you need to use somelib::TheType.

However you always have the option to call the method using a fully-qualified path. For example somelib::TheType::a_method(&foo) or <Foo as somelib::TheType>::static_func(). For the occasional method call I often do this because I do like seeing where .a_method() comes from, it isn’t always clear if it is a method directly on foo or which trait defined it. However nesting a few calls with this quickly becomes messy and sometimes I make a practical choice to just use the use. I prefer to do this just for a short scope to help keep the origin information close to the code, but for very common traits I will sometimes add a global import like use std::io::Write as _ (make the trait methods available but don’t actually pull the name into scope).

`std`

The elephant in the room is that in Rust std (and core) love love love hierarchy and lots of names are super long. I wish I could have a long talk with whoever decided it was really important that std::sync::atomic really needed to be nested under sync. I think we would have been fine to reserve the top level std::atomic for these operations even though they are synchronization related.

It is what it is. I’m not going to advocate for The Great std Naming Reform even if I would prefer it. We have a pattern, and we can deal with it. Most of the names aren’t so long that they need to be imported so most of the time I just spell them out. Occasionally I find myself repeating a long name frequently and just import it. I have yet to be stricken by a bolt of lightning from the heavens.

Dead Bird Features

Kevin Cox — Sat, 03 May 2025 12:00:00 +0000

Anyone who has had an outdoor cat will have experienced a thoughtful gift left on your doorstep in the morning, usually a dead bird or mouse. These gifts are made with love, and take effort to acquire and return to your doorstep. However, us humans generally do not appreciate them.

I often find similar features in software. For example, the wiki that our company uses recently released a new feature: when you copy from a document it would format the copied text as Markdown. For example if I copied part of a code snippet inside a second-level list it would be formatted as:

1.
    1. 

```bash
       -funsafe-math-optimizations
       ```

Whoever implemented this feature clearly worked really hard to ensure that parent context like the two lists above were included in this copying. However ultimately it isn’t what users want. When copying the entire list keeping the formatting could be useful, however when copying just one word the context is typically not desired. In this case the result is that I can’t paste it into my terminal.

Often these features are technically interesting. It is a fun coding project to take a snippet of the document and turn it into accurate Markdown. It is easy to forget to take a step back and think if the user actually wants this.

I find that these features often occur when overriding default behaviours. Web apps are a common culprit. Overriding copying is a common example, but also overriding spell checking, form filling, link following or undo. Many times there is a key feature that the developer wanted to provide, and they made impressive effort re-implementing browser behaviour. At best the feature is useful, but the reimplemented browser behaviour falls short of native. But very often the feature that was provided isn’t actually even helpful.

CORS is Stupid

Kevin Cox — Sat, 24 Aug 2024 20:00:00 +0000

CORS, and the browser’s same-origin policy are often misunderstood. I’m going to explain what they are and what you need to do to stop worrying about them.

Note: I’m going to talk about CORS and the same-origin policy as one thing and use the terms mostly interchangeably. This is because they are basically one system, they work together to decide what you can do with what cross-origin resources. Basically if your requests are not same-origin then you need to deal with CORS rules, policies and mechanisms.

First and foremost CORS is a giant hack to mitigate legacy mistakes. It provides both opt-out protections as an attempt to mitigate CSRF attacks against unaware or unmodified sites and opt-in protections for sites to actively protect themselves. But none of these protections are actually sufficient to solve the intended problem. If your site uses cookies you must take action to be safe. (Ok, not every site, but you better not rely on it. Carefully audit your site or follow these simple steps. Very reasonable seeming patterns can expose you to CSRF vulnerabilities.)

The Problem

The key problem is how implicit credentials are handled in the web. In the past browsers made the disastrous decision that these credentials could be included in cross-origin requests. This opened up the following attack vector.

Log in to https://your-bank.example.
Visit https://fun-games.example.
https://fun-games.example runs fetch("https://your-bank.example/profile") to read sensitive information about you like your address and current balance.

This worked because when you logged into your bank it issued you a cookie to access your account details. While fun-games.example can’t just steal that cookie, it could make its own requests to your bank’s API and your browser would helpfully attach the cookie to authenticate you.

The Solution

This is where CORS comes in. It describes a policy for how cross-origin requests can be made and used. It is both incredibly flexible and completely insufficient.

The default policy allows making requests, but you can’t read the results. So fun-games.example is blocked from reading your address from https://your-bank.example/profile. It can also use side channels such as latency and if the request succeeded or failed to learn things.

But despite being incredibly annoying this doesn’t actually solve the problem! While fun-games.example can’t read the result, the request is still sent. This means that it can execute POST https://your-bank.example/transfer?to=fungames&amount=1000000000 to transfer one billion dollars to their account.

This must be one of the biggest security compromises ever made in the name of backwards compatibility. The TL;DR is that the automatically provided cross-origin protections are completely broken. Every single site that uses cookies needs to explicitly handle it.

Yes, every single site.

Actually Solving the Problem

The key defence against these cross-site attacks is ensuring that implicit credentials are not inappropriately used. It is best to start by ignoring all implicit credentials on cross-site requests, then you can add specific exceptions as required.

Warning: There is no combination of Access-Control-Allow- headers that you can set that solves simple requests, they are made before any policy is checked. You need to handle them in another way. Do not try to fix this by setting a CORS policy.

The best solution is to set up server-wide middleware that ignores implicit credentials on all cross-origin requests. This example strips cookies, if you use HTTP Authentication or TLS client certificates be sure to ignore those too. Thankfully the Sec-Fetch- headers are now available on all modern browsers. This makes cross-site requests easy to identify.

def no_cross_origin_cookies(req):
    if req.headers["sec-fetch-site"] == "same-origin":
        # Same origin, OK
 return

    if req.headers["sec-fetch-mode"] == "navigate" and req.method == "GET":
        # GET requests shouldn't mutate state so this is safe.
 return

    req.headers.delete("cookie")

This provides a safe baseline. If needed you can add specific exceptions for endpoints that are prepared to handle cross-origin implicitly authenticated requests. I would strongly recommend against wide exceptions.

Defence in Depth

Explicit Credentials

One of the best ways to avoid this whole problem is by not using implicit credentials. If all authentication is done via explicit credentials then you don’t need to worry about when the browser may add a cookie that you didn’t expect. Explicit credentials can be obtained by signing up for an API token or via an OAuth flow. But either way the most important thing is that logging into one site won’t allow other sites to use these credentials.

The best way to do this is to pass an authentication token in the Authorization header.

Authorization: Bearer chiik5TieeDoh0af

Using the Authorization header is a standardized behaviour and will be handled well by many tools. For example this header is likely redacted from logs by default.

But most importantly it must be set explicitly by all clients. This not only solves the CSRF problem but makes multi-account support a breeze.

The major downside is that explicit credentials are unsuitable for server-rendered sites as they aren’t included in top-level navigation. Server-rendering is great for performance so this technique is often unsuitable.

`SameSite` Cookies

Even though our server should be ignoring cookies in cross-origin requests it is good to practice avoid including them in the requests in the first place. You should set the SameSite=Lax attribute on all of your cookies which will cause the browser to omit them for cross-origin requests.

Note: When I say “top-level” I am talking about the URL that you see in the address bar. So if you load fun-games.example in your URL bar and it makes a request to your-bank.example then fun-games.example is the top-level site.

It is important to remember that cookies are still included in top-level navigation including form posts. You can use SameSite=Strict which avoids this, but will make the user appear logged out for the first page load after following a cross-origin link (as that request will lack cookies).

CORS Policy

A simple copy-pastable policy is this:

Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: *

That’s it, you’re done.

Warning: This policy is different than mirroring the Origin header in the Access-Control-Allow-Origin header. The * is more than just a wildcard, it also disables implicit credentials. This ensures that other origins can’t make authenticated requests except via an explicit flow such as attaching an Authorization header.

The effect of this policy is that other sites can only make anonymous requests. This means that you are just as secure as if these requests were made via a CORS Proxy.

Shouldn’t I be more Specific?

Probably not. There are a few reasons for this:

You may create a false sense of security. Just because a different webpage running in a “correctly functioning” browser can’t make these requests doesn’t mean that they can’t be made. For example CORS proxies are very common.
It prevents read-only access to your site. This can be useful for URL previews, feed fetching or other features. This results in more CORS proxies being run which just harms performance and user privacy.

Remember, CORS is not about blocking access, it is about preventing the accidental reuse of implicit credentials.

Rant

Why do I need to know all of this stuff, why isn’t the web safe by default? Why do I have to deal with an ineffective policy that makes everything annoying by default without actually solving anything?

IDK, it’s quite annoying. I think most of the reason goes back to backwards compatibility. Sites built features on these security holes, so browsers tried to close them as much as they could without breaking existing sites.

Luckily there may be some sanity on the horizon, with browsers finally willing to break some sites for the good of the user. Major browsers are moving toward top-level domain isolation. They call it different things, Firefox calls it State Partitioning, Safari calls it Tracking Prevention and Google likes cross-site tracking cookies, so they have implemented an opt-in CHIPS system.

The main problem is that these approaches are being implemented as privacy features, not security features. This means that they can’t be relied on as they use heuristics to sometimes allow cross-origin implicit credentials. CHIPS is actually better in this respect as it is reliable in supporting browsers, but it only supports cookies.

So it does seem like browsers are moving away from cookies that span top-level contexts, but it is a uncoodinated bumbling. It also isn’t clear if this will be by blocking third-party cookies (Safari), partitioning (Firefox, CHIPS).

Sementic Versioning Doesn't Support Rolling Deprecation

Kevin Cox — Sat, 17 Aug 2024 17:40:00 +0000

I like SemVer. However, there is one important use case that I wish it supported better. This is what I’ll call “rolling deprecation”.

The idea is simply that instead of removing APIs in a single compatibility-breaking version, you first deprecate an API in one version, then remove it in a later version. This gives time to migrate off of the deprecated API without being cut off from new features. Let’s look at a quick example:

You start a new project and release a public API that contains awesome_function(). All excited you cut a release.
You realize that awesome_function() has a few flaws. Maybe it silently ignores an important error, or it would work much better with an extra argument. You don’t want to break backwards compatibility, so you release the improved version as awesome_function_2() but leave awesome_function() around. (Maybe it is reimplemented using awesome_function_2(), but your users don’t need to care.) You mark awesome_function() as deprecated and release.
We add some unrelated feature and make a new release.
A some point in the future you decide that keeping awesome_function() around isn’t a good idea anymore. Maybe it adds an internal requirement that is expensive to maintain, or you just want that wart gone. You’ve given people enough time to move off. You delete it and cut a new release.

So what does SemVer say about this?

v1.0.0 - First release.
v1.1.0 - Added functionality, backwards compatible.
v1.2.0 - Extra feature, backwards compatible.
v2.0.0 - Breaking change.

Pretty straightforward. Users have between release 2 and release 4 to move off of the deprecated functionality. But it isn’t optimal.

What if a user created their project with release 2 (v1.1.0 according to SemVer) and didn’t use the deprecated functionality? What they upgraded from v1.0.0 and moved away from all deprecated APIs? In both of these cases their code is compatible with the latest release, but SemVer can’t express this.

Wouldn’t it be great if you could easily maintain compatibility with multiple “major” versions? This means that an ecosystem could gradually migration from one major version to the next without any incompatibilities during the transition period.

This is a real problem with SemVer. Every major release is a “flag day”. Sure, maybe your ecosystem of choice will let your dependency tree use both v1 and v2 at the same time (for example Cargo does this) but that doesn’t mean that the experience is seamless. It is common to have one dependency using foo v1 and one dependency using foo v2. Interacting with both of these dependencies at the same time can be difficult because your top-level code can only use one version of foo (without getting creative). It gets especially painful if you need to move data of type foo::Data between the two (not possible unless you can convert it).

Some ecosystems support expressing compatibility as a range. For example Cargo lets you specify >=1.4.0, <3.0.0, but this moves the complexity back to the users of the API. One of the great things about SemVer is that it moves most of the complexity onto the package author. The author has a responsibility to indicate their compatibility via the version number and clients (mostly package managers) can calculate if they are compatible. If Cargo sees foo = "1.2.3" it assumes foo v1.6.2 will be compatible. Specifying dependency ranges pushes that back onto the user. Every user will need to specify the range which requires understanding the versioning scheme that every dependency uses. Most importantly if any library you depend on gets it wrong you will have a version conflict, one mistake ruins the whole system.

Specification

I have an example versioning scheme here to illustrate how the problem could be solved. It isn’t intended as a real proposal. I honestly think SemVer is pretty good and close enough for most things. But it is interesting to see what the simplest form of this versioning scheme would look like.

Let’s stick with dotted numbers for familiarity. We also won’t worry about patch versions for now. We’ll just have two components, the base version and the best version.

You bump the best version every time you make a release. Any time you make a backwards incompatible API change you set the base version to the best version of the first release in which that API was marked as deprecated.

How this looks with our example:

v1.1 - First release.
v1.2 - Fully compatible, marks awesome_function() as deprecated.
v1.3 - Fully compatible.
v2.4 - Breaks API that was marked deprecated in v1.2.

One interesting is that both components of the version only go up. There is no “reset” for the best version when the base is incremented. (Although there probably would be if we added patch versions.) If we wanted to release a completely incompatible version it would be v5.5.

Let’s clarify how to check compatibility. Software that doesn’t use any deprecated APIs on vA.B is compatible with version vX.Y if X≤B≤YX \le B \le Y. (A is irrelevant)

So v1.1 would be compatible with v1.3 because 1≤1≤31 \le 1 \le 3but not v2.4 because 2≰1≤42 \nleq 1 \le 4. v1.2 is compatible with v1.3 because 1≤2≤31 \le 2 \le 3and v2.4 because 2≤2≤42 \le 2 \le 4.

SemVer Compatibility

Interestingly this scheme is backwards compatible with SemVer! This means that you could start publishing using this scheme and software that expects to receive SemVer would do something reasonable. Most notably it would never consider two versions compatible that aren’t. (However it will tell you that some versions are not compatible when they are.) This is obviously true because the “major” version is bumped on any change that could break anyone and the “minor” version still requires ordering. The main difference is that a “major” version bump doesn’t mean breaking everyone, it specifies which versions were broken. The only downside for using this scheme for SemVer clients is that this scheme would encourage more frequent “major” releases, which would be annoying for SemVer users.

Patterns

Last N Compatibility

Some projects have “deprecation compatibility” for a fixed number of releases. In this scheme it would just look like the base version trailing the best version by that fixed amount.

For example for “last 3 versions” compatibility.

v1.1
v1.2
v1.3
v2.4
v3.5
v4.6

…and so on.

No Compatibility

For projects wishing to express no backwards compatibility the base and best versions would always be the same.

v1.1
v2.2
v3.3

Private Internet

Kevin Cox — Fri, 16 Aug 2024 23:55:00 +0000

Many widespread internet protocols were written at a time when internet security wasn’t much of a consideration. From things like lack of From address verification in email to NTP reflection there are lots of protocols that are now considered badly designed. When they were authored there was a lot of implicit trust (For example because there were just a few universities connected to your network and you could just kick off any bad actors.) but now the internet is full of bad actors and these designs are harming us.

I think IP is generally considered well-designed. IPv4 refuses to die and IPv6 is working great. But I think that these protocols have some fundamental flaws. Or at the very least they need more layers on top to work well on today’s internet.

The most obvious sign of this is the many non-technical people know what an IP address is. I don’t think the average user should need to care how their networking works, but due to issues with the protocol many people are aware that their IP is sensitive information and that they need to protect it.

Two of the biggest flaws of IP are privacy concerns and DoS susceptibility. These were likely not even considered when designing the protocol. But much like From address forgery was not a concern when designing SMTP, I think these privacy issues should be considered a deficiency that needs addressing.

Metadata absolutely tells you everything about somebody’s life. If you have enough metadata you don’t really need content.

Stewart Baker, NSA General Counsel

Valuable Properties

Here I tried to brainstorm a few properties that I would like to see in a private and DoS resistant network protocol.

Non-sensitive Client Addresses

If I make two connections, the remote end should not be able to tell if I am the same person. This property should hold whether I connect to the same host twice, or two hosts that coordinate.

DoS Resistant

All methods of connecting to me must be revocable. That is at some point they stop functioning and you need new information to connect. Ideally this could be done at any time, immediately invalidating an address, but could be time based (address expire after a period of time).

This revocation must not just cause the endpoints to ignore traffic. The traffic should be dropped as close to the source as possible. (For example the first “well behaved” network. On the internet this would at least be a tier-1 ISP, if not the attacker’s own ISP)

No Leak of Outgoing Connection Server Address

If I open a connection, a local observer must not be able to identify which server I am connecting to. This property should continue for as many hops as possible. (At some point the traffic will have to reach the server and someone who can observe every hop on the path will be able to correlate it.)

Independent Services

A server should be able to host any number of public services and no client should be able to tell if these services are co-located.

No Leak of Incoming Connection Server Address

If I receive a connection a local observer must not be able to identify which address was used to create it. (If the address is publicly known they can likely probe this by making connections and comparing traffic, but I’ll consider that out of scope of this feature.)

Stretch Goals

Encryption

I didn’t require encryption as while it would be nice (my default stance is to encrypt everything, more encryption never hurts) it may make the protocol significantly more expensive. Furthermore, fundamental protocols like IP tend to evolve very slowly, so when the encryption included in the protocol is no longer state-of-the-art it will be very hard to update. For these reasons encryption is likely best managed by a higher layer.

That being said it would still be a great stretch goal. Especially if the traffic was differently encrypted on each hop. This would prevent attacks where traffic is sniffed at multiple locations on the network to try to identify different ends of a connection. This is a feature that could be hard to implement just on the endpoints.

Possible Implementations

I haven’t actually designed a protocol. These requirements are more of a discussion point. I’m sure other properties will be desirable as well. But I did think of ways to accomplish some of them.

Onion Routing

My first thought would be some form of onion routing. The client builds a route and encrypts the packet for each layer in turn. Each router then decrypts a layer and forwards it. This has worked well for Tor but seems to have some fundamental downsides.

Some concerns with this approach:

The client needs to know about various routers along the path. On the public internet this is a lot of data that is frequently changing.
Every layer needs metadata on where to forward the packet next, eating into the MTU.
Encrypting data many times is expensive.
How is the connection initially routed while keeping the server private?

The first 3 could be mitigated by reducing the number of hops.

Individual ISPs could act as one large hop, rather than multiple routers. This could reduce the number of hops for each connection and total number of logical routers that clients need to know about.
Instead of having a layer for every hop the packet can be wrapped only until large networks. For example major ISPs. At this point they can absorb any attacks and your traffic is well mixed with other sources.

I don’t know how to solve 4. Tor Onion Services solve this by having the server maintain a few fixed routes leading to it. The client then adds one of these (encrypted) routes to the end of its route. However, this results in a non-optimal path which may not be a good tradeoff for general internet use. Maybe the client can evaluate them all use use whichever one is fastest?

Fine-grained Routing

Another option may be implementing something like BGP but on a mine narrower scale. Servers generate random values as addresses and announce them. These are then broadcast across routers like BGP routes are today. When the client connects it generates a random “return” address and sends its traffic. Routers along the route also remember the return address to establish a connection.

There are also concerns with this approach.

How can we avoid leaking the server address?
Internet routing tables are already huge. To help combat this there are minimum network sizes that are publicly routable. Having many unique routes to each host would explode routing table size.

We can avoid leaking the client address by having each router along the path transform it. (Example: generate a new random address and remember the mapping) However, this trick doesn’t work for the server address because every client needs to be able to connect to the same server address.

I have no idea how to solve 2. If addresses are random it is very hard to be able to route to any random address from anywhere else.

Time Sharing in Instant Messaging

Kevin Cox — Fri, 05 Jul 2024 19:55:00 +0000

A seemingly-obvious feature that I haven’t seen in any instant messenger yet. I want to be able to share a timestamp in a way that can be computer-understood and presented to the user with a good UX.

For example, I want to say:

Let’s have a call about this at 11:30.

But that “11:30” is actually a rich object that encodes an exact timestamp. If your IM application uses HTML formatting for messages it may look something like this:

<time datetime=2024-07-05T11:30:00-04:00>2024-07-05 at 11:30</time>

The time is machine-readable (with some fallback text). Clients would show the users the time converted to their local time zone! Or maybe a relative time like “in 32 minutes” if they prefer. Maybe they can click it and see that time in their calendar, or see what time that is in different time zones. The possibilities are endless, but the most important thing is that the time is correct, because computers are much better at time math than humans are.

Right now I frequently see people just mention a local time and you need to have their time zone memorized (and know if and where they are travelling). If you are lucky they say something like “11:30 IST”, then you just need to guess if they mean Indian Standard Time or Irish Standard Time. Having a rich “timestamp” object in instant messages would be easier for both the sender and receiver.

Announcing git bisect-find

Kevin Cox — Sun, 19 May 2024 17:50:00 +0000

This is a small utility that I wrote to compliment git bisect. git bisect is a fantastic tool. In the most basic usage you give it one “good” commit (a commit that doesn’t yet include some property) and at least one “bad” commit (one that does have it). git bisect will then guide you through the search, picking commits to test that cut the search space in half. This allows you to efficiently identify the first commit with a particular feature.

However one of the premises of git bisect is that you know a good commit. Often times you notice a bug but don’t yet know a good version. Was that broken in the last release? Or is it new? Maybe it was broken unnoticed in the past couple of releases? Sure, you could start checking out various revisions to see if the bug is there, but why do this manually?

This is where git bitsect-find comes in. It will take just a bad commit, and step back looking for the first good commit. Basic usage looks like this:

Basic Usage

First check out your bad commit:

$ git checkout main

Then start bisecting with git bisect-find bad.

You will be asked to start a regular git bisect run, type y.

Then a new candidate commit will be checked out. If the candidate commit is bad run git bisect-find bad and a new candidate will be checked out. Repeat the test and mark process until a good commit is found.

Once you have a good commit found you are done with git bisect-find. Just run git bisect good and continue with the regular bisection process.

There are more complete docs in the README. There are also more features to add (like automatic bisection). However the basics have already been quite useful to me so I figured I would share.

Mechanism

There really isn’t anything too special about the way it works. It just jumps back twice as far each time (following first parents). All state is logged via regular git bisect so once you find a good commit all of your git bisect-find bad (and git bisect-find skip) commands are remembered as regular git bisect bad and git bisect skip would be.

I do wonder if jumping back by twice as much is optimal. Obviously jumping right to the first commit would reduce the number of git bisect-find steps. But would result in more git bisect. But git bisect is quite efficient. (Not to mention that the first commit is likely not particularly interesting.) At some point I should probably work out which growth pattern results in the shortest number of total steps. However I think the optimal number also depends on the distribution of the age of the target commit. But either way doubling works well enough for this tool to be very helpful for me.

My Favourite Feeds

Kevin Cox — Thu, 29 Feb 2024 23:10:00 +0000

I am an avid user of RSS feeds. Checking my list reveals that I am subscribed to 530 feeds! In the spirit of sharing I thought it would be interesting to share a snapshot of my current reading list. This may prove an interesting resource for someone looking to get started with feeds or just to grow their existing subscription list.

I’ve tried to focus on feeds that are generally interesting and have high value per-post. So these should get you some interesting content without exploding your reading queue. I’ve also preferred feeds that contain the full article content so that you don’t need to leave your reader (although articles that require interactivity or are worth the click are still included). I have cut out feeds about specific software that I use or games I play as I don’t think most of these changelogs are very interesting unless you use that software. I’ve also cut out most of the higher-volume feeds and feeds that were specific to me (like a feed of outdated packages I maintain).

I’ve managed to cut down the list to 50 items. Still a good chunk, but I encourage you to read a post or two from each before subscribing and take the ones that interest you. Or just subscribe to them all then unsubscribe if you don’t find the posts interesting. That is one of the great things about RSS, subscribing costs nothing and unsubscribing is reliable. No one will save your email to spam you later.

If you just want to grab them all you can download this OPML file which can be imported into your feed reader. Or you can just follow along and subscribe to your favourites.

Below I’ve provided links to each site, as well as a direct link to the feed. I’ve also added a small summary which is typically copied from the site or feed itself, but may be added or edited by me if it didn’t seem very helpful. Items are just in alphabetical order.