Forem: Ron Newcomb

Git is Elegant

Ron Newcomb — Sat, 04 May 2024 22:43:36 +0000

Enough with the fear of the bad rebase. Let's learn how git works under the hood.

Files and Folders

For each of our source files, a SHA-1 hash is created from its complete contents, and then the contents are compressed with zlib. The hash becomes git's "filename" for that compressed version.

For each folder in our project, a tree file is created. It's a text file of four columns, one row per file or sub-folder. The four columns are: the "git filename" hash; the user's original filename/foldername; the file system permissions as a bitfield; and a type which specifies if the row is representing a file ("blob") or a sub-folder ("tree").

100644 blob 0155eb4229851634a0f03eb265b69f5a2d56f341      README.md
040000 tree d8329fc1cc938780ffdd9f94e0d364e0ea74f579      src
100644 blob fa49b077972391ad58037050f2a75f74e3671e92      tsconfig.json
100644 blob 1f7a7a472abf3dd9643fd615f6da379c4acb3e3a      package.json

This tree file is itself treated like a source file, being both compressed and then named by its own contents-hash. This allows tree files to recurse just like folders do.

All of the above live in the /.git/objects/ folder.

Commits

Most tree files are pointed to by other tree files, but the top-most tree is pointed to by a commit. A commit is a text file with five pieces of information: the "git filename" of a top-most tree file, the parent (previous, older) commit, the author (from user.name and user.email git config settings), the datetime (as seconds since epoch – 1 January 1970, midnight UTC), and a commit message following a blank line.

tree a1e672eba0d552e704f294625bab398f0e4bef70
parent 5938f809b156e743aa47075befb5ff414ad791ff
author Ron Newcomb <pscion@yahoo.com> 1713826102 -0700
committer Ron Newcomb <pscion@yahoo.com> 1713826102 -0700

wip - it still doesn't build

Most commits are pointed to by the next-most recent commit. In other words, each commit's pointer points backward in time. The newest commit points to the second-newest commit. But what points to the newest commit?

Branches

The top-most commit is pointed to by a branch. A branch is a text file in /.git/refs/heads/ whose filename is the branch name and whose contents is just the "git filename" of that newest commit.

~/myproject/.git/refs/heads> cat my-temporary-branch
6862ff2bdfdc4953f86cd3ba345e058a01cc0952

So far, we have branches which point to commits which point to (other commits and) tree files which point to (other tree files and) compressed file blobs.

The Word "HEAD"

Some coders consider the "head" of a linked list to be the oldest entry, where newer entries are appended to the "tail" end. An example is a list containing ABCD where A is the oldest entry and the "list pointer" points at A, and D points to nothing. But in git as in functional programming it's the other way around. The same list is represented DCBA where the "pointer" points at newest entry D while A points at nothing. This arrangement allows "the older part of the list" to remain immutable.

For example, if an unknown, second pointer points at DCBA, and the first pointer prepends E to the list, the first pointer can point at E and therefore EDCBA while the second pointer still points at D and hence DCBA. From the second pointer's point of view the list hasn't been modified out from underneath itself. The list DCBA is still there completely untouched and unaware of E.

With that in mind, the reason branches are stored in a folder called /heads/ might make more sense. Each file is the name of a branch and is pointed at a list of commits, DCBA. An older branch on the same path can point to it's older commit C and it still sees the same CBA history as it always did even as the first branch adds commits over time FEDCBA.

HEAD as a Cursor

So, one definition of the word HEAD is the most recent commit in a string of commits. But the other definition, "the" head, is more like a cursor or current pointer. This is the concept of the current working branch, the head. There is a singular file, /.git/HEAD, which is a one-line text file that points at a branch.

ref: refs/heads/my-temporary-branch

Think of the file HEAD as a cursor. If we're on branch main then the HEAD file has the path to the file refs/heads/main. Switching to branch my-temporary-branch means changing the /.git/HEAD file contents to refs/heads/my-temporary-branch.

Also, HEAD can point directly to a commit instead of indirectly through a branch. This is called the detached HEAD state. For example, git checkout HEAD^^ will point HEAD to a commit located two commits back from wherever the current branch points. No branch or tag in the whole system points directly at this commit. But we can still create a new branch to point to it, and create commits that point to that commit. It's just a directed acyclic graph after all.

Tags and Annotations

As an aside there's tags, of which are two types. The lightweight tag is effectively identical to a branch but with the expectation that it is immutable – it never changes which commit it points to. Heavyweight tags point to an annotation which then point to the commit. Annotations are text files that hold the tag author, tag creation datetime, the tag name, a tag message, and points to a commit. Tags are stored in /.git/refs/tags/ and the tag name is also the filename.

object 6862ff2bdfdc4953f86cd3ba345e058a01cc0952
type commit
tag v2.0
tagger Ron Newcomb <pscion@yahoo.com> 1714360500 -0700

typescript added and builds without error; can now turn on ci/cd checks

File System Allowances

As nice as the above setup is, there's a problem using it on most operating systems' file systems. One, the above scheme means the folder /.git/objects gets a very large number of files in it very quickly. Most general-purpose file systems do not perform well on that. Second, a zlib-compressed text file is very small, sometime just a hundred bytes, and there will be a lot of these very small files. Most general-purpose file systems do not perform well on that, either. So git does two things to mitigate these problems.

For the first problem, the first two or three characters of the "git filename" is broken off to become the name of a subfolder. So instead of /.git/objects/bd37ea4bf8876c306771c8666ec92f4d582135bd we'll have /.git/objects/bd/37ea4bf8876c306771c8666ec92f4d582135bd. (Note the added slash between objects/bd and 37ea.) This helps bring down the number of items directly contained in /objects to a reasonable number.

For the second problem there are packfiles. Once a set of those tiny compressed files are old enough and numerous enough, git will concat them into a larger file, called a packfile, and maintain a corresponding index file to remember which file begins where. When packing, git can also use diffs if two files are very similar, further saving space at the cost of increasing time to access a very particular old version of a file.

These mitigations don't fundamentally change any of the concepts in play. They are merely optimizations that the git tools already step around for we.

Losing Work

By now we understand that git rarely deletes anything. Every modified file is hashed, compressed, and stored in its entirety, every time, with no direct reference to its older version. "Diffs" are not stored except for much older work in packfiles, and maybe not even then. But, sometimes a rebase goes wrong, and work goes missing. What now?

The first thing to remember is our work's missing modifications are still in git somewhere, but they are almost definitely orphaned. Although git does delete orphans eventually, they must be a few months old so last week's rebase nightmare can still be overcome.

The second thing to remember is due to the above immutable-full-content-hash way that git works, if we have 4 commits in a row DCBA, and we squash CB into X, DXA can't just magically happen. Commit D still has commit C as its parent, so it must be "moved" so commit X is its new parent. (This is rebasing: giving it a new "base" to work from.) But if we move commit D to a different parent, then line two of the text file representing D is changed, which changes its contents and therefore its hash and therefore its git filename. (So technically speaking rebasing D doesn't actually change D but creates a new commit Y. )

Anyway, when we compare a branch-to-be-rebased to the new main, we'll see all of the changes in CB on both branch and main, and they conflict, and it's very confusing.

The first trick is in the start of the (interactive) rebase -i main, where we see a screen like this.

pick c6a71d7 commit B
pick 898a9b5 commit C
pick c6729d2 commit D

we can change pick to s for squash (or not), but in the case of CB already being squashed and merged into main as commit X, we actually want to completely delete the lines for those commits, leaving only our commit(s) for D, like so.

pick c6729d2 commit D

After saving the file, most times, the rebase will immediately succeed. we completely deleted two commits from our branch and it improves things remarkably.

The other side of this, if we didn't already know this and are now in the thick of it with detached HEAD states and a boatload of conflicts, try to --abort and start over, knowing this. While we can't completely avoid conflicts, merely understanding the linked list nature of commits, the branches-as-named-pointers that point to a few of them, and why squashing is causing the issue not rebasing, can help give we a clear idea of why it's all going pear-shaped. That is always more powerful than a magical incantation to recover orphaned commits. (But for expediency: create a new backup branch to point to the same commit we're about to rebase so nothing gets orphaned, or, git log --graph --reflog afterward to browse around for orphans.)

Gamers Need Not Apply

Git is specialized for text files and immutability. The one thing that git does not do well is incremental changes to large binary files. Binary doesn't compress nearly as well as text does, and git's philosophy of whole-content hashing and storage doesn't work well for incremental changes to large binaries. So images, videos, skeletal animations, tesselated meshes, audio samples and so on would take a very large amount of disk space and time under git's implementation. This is one reason the videogame industry tends to use Perforce over git, which diffs a binary against its old version and merely stores the latest copy with a reverse-diff to re-create the older version. Older source control systems also used a diff philosophy, and some poorer ones stored forward diffs from the original, with occasional checkpoints. This is obviously slower and very brittle in the case of one stray bit flip on disk instantly corrupting large portions of a repository downwind of itself. Combined with the tendency of older source control systems only giving a local machine the minimum it needs for a branch and one can see why they failed so often, and have been replaced so totally by git.

Authentication Orientation

Ron Newcomb — Thu, 17 Aug 2023 23:03:38 +0000

At its most basic, private communication between two persons' devices involve one function, encrypt(msg, key), to create an encrypted version of the message, e_msg, and another function, decrypt(e_msg, key), to recover the message.

The mathematicians who create these two functions tell us that the key is an astronomically large number with certain properties that matter to them but not to us. We're promised that only people or devices that possess the key can participate in any conversation piped through those functions.

And if the two devices in question are both servers which were created together and deployed securely, this is enough. The key is also called a secret and is never transmitted between devices or anywhere else. It sits in each device's config file, permanently hidden from the world.

But for the common case of a web browser wanting to speak with a new web server, this won't work. The browser is on a device completely ignorant of any keys the server uses. So, a new class of encryption algorithms was created.

Asymmetric key algorithms, unlike the above symmetric key algorithms, have two keys. The public key is used in encrypt(msg, publicKey) while the private key is used in decrypt(e_msg, privateKey). The public key is shared freely with anyone and anything, while the private key remains a hidden secret.

This does mean that communication is only one-way with asymmetric key algorithms. If a device generates a pair of keys (i.e. with the command ssh-keygen) then it can only listen to private messages specifically sent to it, never sending messages back. To send, the other party must generate its own pair of keys, sharing the public key, to set up a second one-way channel moving in the opposite direction.

Example algorithms:

Symmetric-key algorithms: DES, 3DES, RC4, AES
Asymmetric-key algorithms: RSA, DSA, EdDSA, ECDSA

Protocols

Asymmetric keys form the basis of web browsers talking to unknown web servers. The browser can send its own public key to the server, perhaps getting the server's public key back in the response, and then private communication can happen. Exchanges of key information as a prelude to what they actually want to talk about is called a protocol.

However, when the mathematicians said these encrypted messages cannot be broken, they put a few astericks on it. A million computers given a million years can brute-force find what astronomically large number was used. If the message being sent is somehow already known it's possible to bring down those millions to something practical. In lieu of the message being completely known, if a part of it is known and we have many examples of messages it's also possible to triangulate the position of the secret number.

And so, a protocol will often only use the devices' permanent keys just long enough to create new, temporary keys, defeating any sampling attempts. It may also attach an expiration date to the temporary keys in which case it will create even newer temporary keys on a periodic basis.

Example Protocols:

PGP - Pretty Good Protection
SSL - Secure Sockets Layer
TLS - Transport Layer Security (successor to SSL)
HTTPS - literally http plus either SSL or TLS
SSH - Secure Shell (favorite of ftp, telnet, and github)
PPP - Point-to-Point Protocol (modem screech)
IPsec - you might remember IP from such hits as "TCP/IP" and "IPv6: The Enlargening"
Kerberos - someone was triple-dog dared to use symmetric-key over UDP/IP
OIDC - OpenID Connect (implemented by companies like Auth0, Keycloak, Okta...)
SAML - Security Assertion Markup Language (not Sad At Much xmL)
and many more

Signatures

Next is signatures. Given an official document for, say, purchasing a house, the buyer will be asked to digitally sign the document. Here the buyer's public and private keys are used in reversed roles. The buyer will encrypt(document, privateKey) and the result is appended to the document.

This means anyone with the buyer's public key (i.e. literally anyone) can decrypt that portion of the document: decrypt(e_document, publicKey). This gives no new information of course since they can already read the unencrypted copy, but the very fact that the public key does in fact reproduce the document instead of gibberish means that The Buyer Was Here^tm. It also allows seeing if any changes to the unencrypted document have happened after the signing.

Certificates

Lists of public keys and their holders are kept by a certificate authority. At registriation time the CA may require forms of identification from the requestee ranging in importance from government IDs to dot-com domains to just an email address and name, depending upon the type of certificate (and hence, the type of guarantees) the requestee needs. The money and time required for the process vary accordingly.

Once satisfied the CA puts this identifying information with the public key, an expiration date, the requestee's server's domain name (if applicable), and information about the CA itself in a standard X.509 document and digitally signs it. This document is called a public key certificate, or cert.

Now when a browser contacts that server, it can verify the server's cert by using the CA's public key to decrypt the signature and get back a copy of the cert instead of gibberish. After checking the expiration date and the domain name in the cert matches the domain name navigated to, the browser knows it's safe to proceed.

Example Certificate Authorities:

VeriSign
VISA
Microsoft Azure
Google Cloud
Cloudflare
Amazon Web Services
... your browser or operating system has a long but incomplete list

Authentication

Certs authenticate servers so clients like web browsers know they're safe, but the reverse isn't yet true. The server needs to authenticate the person on the client device.

In the specific and usual case of the web, once the cert is verified, the HTTPS protocol (specifically the SSL or TLS protocol underneath) takes over, creating a private channel with which to speak to each other. Then javascript or even plain HTML can form submit a login/password combination.

It's worth noting that the password is plaintext in the request's body, but the SSL/TLS connection means it cannot be seen. However, the headers of a HTTP request is outside of the SSL/TLS tunnel, so never place the plaintext password in the URL. It must be in the body, hence the request must also be POST or PUT never GET.

If encrypting HTTP headers is desired, investigate IPsec.

The ol' localhost on HTTPS

When developing a web app, typically http://localhost is the protocol and domain in use. While it's easy to find the magic incantations that allow https://localhost to work, the reason for each of those steps should hopefully be much clearer.

Create a self-signed certificate.
Tell your webserver (IIS, nginx, NodeJS something) to use that cert for that domain.
Put that cert into your browser or operating system's list of trusted certificate authorities, meaning that your laptop now stands beside AWS and VeriSign as the arbiter of good taste... but only for browsers on your laptop.
If using an IDP like Keycloak that redirects back to your site after login, you might need to inform it that https prefix on your origin is OK for callback.

Closing Remarks

This little article is just the barest brush against this very large and ever-expanding subject. The amount of details left out could easily fill a bookshelf, and it contains very little practical and immediately actionable advice on how to lock something down. But, many practical books on computer security read like a random grab-bag of do's and don't's with very little rhyme or reason as to why things must be this way or that. I hope this brief article will help ground you with a basic foundation so you can better understand those books.

Choosing Technologies as a Senior Dev

Ron Newcomb — Sun, 17 Jul 2022 05:22:45 +0000

I found an interesting change in my perspective upon becoming an actual Senior Developer. Of course my responsibilities increased: I'm expected to mentor lesser experienced devs and new employees, and talk with PMs, BAs, and upper management more. But of interest is deciding on some of the tools and technologies we use going forward. And sometimes, selling that to a team that may not be all that thrilled with my choices.

Formerly my choice of technology came down to one of two things. First, what was new, what was hot, what was growing in popularity with the speed of a rocket. I wanted in on those bandwagons and a lot of my peers did too. Second was technical specs. What was more performant, what elided boilerplate, what had good tooling.

But now I have not only newer concerns but larger ones and they are this: what does the team already know? What could the team learn quickly based on analogous things to technologies we already use? What is the expected lifespan of any new tech we bring into our business? What are the security and licensing ramifications? What other tools exist in its ecosystem, and are they mature?

These new concerns pretty much obliterate my previous measures of desirability. I'm more concerned with the long-term stability of software and the short-term training of people than I have ever been in my life. I'm concerned with budget, with time and money, despite that I'm NOT the one who wields the schedule.

I used to be the outspoken one that wasn't afraid to express his dissatisfaction with management for their clueless tech takes.

Now I am the boss monster. Now I feel I owe an apology to twenty or thirty years' worth of my former managers.

Modal Dialogs as a Promise Versus Inline

Ron Newcomb — Tue, 27 Oct 2020 04:02:24 +0000

There are two major ways a UI framework implements modal dialogs. One is writing the modal into the component template that will pop it, and adding show/hide logic into that component's code.

<InvoicesPage>
  <ActionsRow>
    <CopyButton />
    <DeleteButton />
  </ActionsRow>
  <InvoicesList />
  <ConfirmDeleteModal show={prop} />
</InvoicesPage>

Sometimes the entire contents of the modal are also inline, such as with a tag that promotes a piece of template into being a modal by adding the necessary CSS.

<InvoicesPage>
  <ActionsRow>
    <CopyButton />
    <DeleteButton />
  </ActionsRow>
  <InvoicesList />
  <AsModal show={prop}>
    <h3>Are you sure you want to delete?</h3>
    <button>OK</button> <button>Cancel</button>
  </AsModal>
</InvoicesPage>

The other way uses an imperative function that returns a Promise, where the promised value is in most cases the button that was pressed to dismiss it.

const onDeleteClick = async itemId => {
  const clickedOKbutton = await askIf('Are you sure?');
  if (!clickedOKbutton) return;
  /* else they are sure */

Sometimes a modal's job is to create an object, say, from a form. If so then that is the promised value.

const onCreateClick = async ownerId => {
  const newOrder = await createOrderModal(ownerId);
  if (!newOrder) return; /* cancelled */
  /* else we have a valid object */

Typically, React uses the former method and Angular uses the latter method. But there are clear use-cases for each method, and sticking to either method exclusively brings unnecessary complexity when using the wrong method for the other's best case.

Let's look at some of these use-cases which favor one method over the other.

Use-case #1: Zooming In

Sometimes you just need extra space. Most apps have a main menu, at least one sidebar, and possibly a footer eating up a portion of the screen. The main content area frequently has a heading or breadcrumbs or action buttons taking up another row or two of space.

Let's say our user is filling in a form where one of the form fields requires a lot of space, say, which email template to use for a scheduled customer communication. Email templates start with words like, "Dear CUSTOMERNAME" and go on for three to five paragraphs of just the right tone and messaging. We want our user to be able to see and read the templates before committing to one, so a simple <select> of template titles won't do. Rather, we want to pop a large modal that shows the complete email template in a box of significant width and height, and allow the user to flip through them reading each.

This is a poor use-case for promise-based modals. Despite being presented as a modal, we're still in an actual <form> with live validation, and that validation may change which email templates we're allowed to choose or vice-versa. Keeping that validation intact when the modal contents are elsewhere would be obnoxious. By defining the modal inline, the selection remains part of the form at all times.We're just using the language of modals to "zoom-in" on the details of one field while our user manipulates it.

An even simpler example: let's say we have a dashboard with several charts and graphs. Each chart is rather small to see much detail on it, so we decide that clicking a chart brings it up in a modal that's a good deal larger. The chart library will re-render itself when it detects the resize, no longer omitting labels since it now has the space, and making smaller pie slices visible.

The implementation of such is trivial for an inline modal. Just wrap the chart's <div> with something like <ClickForZoom isPoppedOut={prop}>. Modals don't get much simpler than toggling the size and position of a div on each click. The contents of the div don't change at all in this case.

Use-case #2: Simple Questions

The other kind of very simple modal has more to do with its job than its implementation. Asking simple questions for simple answers is one of the commonest use-cases for modal dialogs, so much so that a page with many action buttons might be able to pop three or four different questions.

Writing such modals inline means writing their show/hide logic in that same page, and more logic to set the question and available answers each time. For a busy page the lines of code required for multiple inline modals can begin to mask the page's own logic through sheer amount.

Here the promise-based modal is a natural fit. Confirmation as a function which accepts the question and returns the promised boolean answer gets the job done with minimal intrusiveness.

const onDeleteClick = async () => {
  const ok = await ask("Are you sure you wish to delete this?");
  if (!ok) return;
  /* else delete it... */

Multiple-choice instead promises one of the passed-in strings (or its numeric index, per your taste).

const onCustomizeClick = async () => {
  const theme = await ask("Choose a theme", ["Light", "Dark", "High Contrast"]);
  if (!theme) return;
  /* else apply the choice */

Here the promise setup puts the logic and everything in a re-usable manner elsewhere so it cannot clutter the page with show/hide logic, nor duplicate <ConfirmModal/> tags all over the codebase's various consumer components.

Use-case #3: Modeless Dialogs

Modal dialogs are called such because they put the software into a special "mode" which persists, with explicit entry and exit. But there are modeless dialogs, which float around like a persistant toolbar.

One case is a CRM that assists our user, a customer support representative, with making phone calls to her customers. While she's on the phone a modeless dialog appears with the customer's name, info and quick-links to their recent orders and invoices. She can roam all over our app while this dialog shows, and can click links in the dialog to make the rest of the app navigate there without affecting the dialog. Only by clicking its Hang Up button will the dialog then dismiss itself.

The primary difference between the modeless and modal dialogs is that the modeless isn't trying to gather an answer for any particular reason. Since there's no answer to wait upon, an awaitable function returning the Promise of an answer wouldn't have much purpose here.

A Hybrid Approach

Consider an awaitable function which accepts elements to be inserted in the modal's body.

const theme = await ask(<BlinkingText>Choose any of our three themes</BlinkingText>, ["Light", "Dark", "High Contrast"]);

This hybrid approach is a mixed bag of limitation and feature. It encapsulates the show/hide state away from the calling page but not the details of the modal body. If the passed elements are more than half a line of code it will look out of place in an event handler code block. Also, someone will eventually stretch that too far and ask how to put state into it.

We can try to salvage the idea with a version which accepts a single element, the component-as-promise.

const theme = await ask(<BlinkingThemeSelectModal />);

The issue with this is it would be fewer import statements and fewer keystrokes if BlinkingThemeSelectModal wrapped the above in a dedicated function and exported that instead.

const theme = await askBlinkingThemeSelectModal();

But that brings us back to ordinary modals-as-a-service again. Hence I wouldn't recommend the hybrid approach of passing elements to the promise function.

Stacking Contexts and Position Unfixed

Sometimes you can't use inline modals when you want to.

First of all, z-index isn't global, it's relative to the current stacking context. In each stacking context, the whole z-index numberline starts over. This is the case where a z-index of 1 is still shown on top of a z-index of ninety bagillion. Although you can try to put the entire app in a single stacking context by adding z-index:1 to document.body, you'll then find yourself trawling through your third party libraries for tooltips, drag-drops, leaflet maps, and other popovers and discovering that they some use z-index values of tens while others use thousands, so they won't place nicely together anymore. The usual solution instead is to use document source order, which means placing the modal's HTML near the end of the document, which is how the promise-modal method works anyway.

Second is the way the modal itself is constructed. Nowadays we use position:fixed to pull a <div> out of document flow to center it onscreen, usually with left:50%; top:50%; transform: translate(-50%, -50%);. If our app also has, say, a side drawer that slides out from the right side of the screen, we might position and move it in a similar way. And if we did, then one day we discover that if we try to do an inline modal from within the drawer, position:fixed doesn't work. The modal and its backing overlay only cover the drawer, and is centered within the drawer. This is because transform creates a new context which acts as a viewport itself. The usual solution is, again, document source order.

Finally, modals aren't always the top element anyway. Small, non-blocking notifications that self-dismiss after a few seconds like toastr should appear on top of modals. Auto-logout warnings which give the user 15 seconds to press the supplied "I'm still here" button should appear on top of them as well. A good promise-modal system allows one to put a placeholder near the end of the root <App> so the placeholders for these other things can be put immediately after.

<App>
  /* most of the app here... */
  <ModalDialogsContainer />
  <AutoLogoutWarning />
  <NotificationsContainer />
</App>

Abort, Retry, Ignore?

I hope you found this long deliberation on the simple modal dialog enlightening. The imperative promise-based method is best for some use cases and the declarative inline method is best for others, so I would expect both methods to appear in a codebase of any complexity. For the middle ground where either work, go with the framework's or the team's favorite. Just be wary of anyone demanding the two methods can't or shouldn't coexist.

Missing the Message Bus in HTML Custom Elements (and Micro-Frontends in General)

Ron Newcomb — Thu, 17 Sep 2020 20:31:45 +0000

How to create a frontend with de-coupled HTML Custom Components wasn't the most obvious thing to me. Did they communicate by a parent component calling a function on a child component, or by sending a custom event from one to the other? What if the two components weren't on the same ancestor-descendant line? What role do attributes play? How to avoid hard-coding function names or event names in both components?

Before I understood the intended way, I created a "message bus" script that caught custom events and re-sent them directly to the subscribed components. This was a mess. Not only are events hard to work with due to all the readonly properties and the still-amazing inability for vanilla javascript to deep-clone something, but I still had one component's custom event names hard-coded into all receiving components.

The correct solution is actually asymmetrical. One component publishes a custom event, which unlike element attributes, accepts complex payloads like arrays and objects. The receiver component exposes a javascript method that accepts a payload through ordinary parameters. Receiver components do NOT addEventListener and sender components do NOT call methods on other elements.

This setup is perfectly de-coupled, so it requires glue code at the top level. I went through a few iterations of this glue code, starting with the most minimal amount of code that would make things work, then immediately found trade-offs no matter what is done.

Here's the absolute minimal code I could come up with to make the components work together. It begins with an array of all custom event types, because there is no way to query a browser for a list nor any way to add a listener for "all custom events". (I'll come back to that little problem shortly.)

['drilldown', 'pieChartInit', 'showTip', 'hideTip']
    .forEach(et => 
        document.addEventListener(et, ev => 
            document.querySelectorAll(`[${ev.type}]`).forEach(el =>
                el?.[el.getAttribute(ev.type)]?.bind(el)(ev))));

It reads, "Add an event listener to the top-level for every custom event type. When one of these events happens, the handler will find all elements with an attribute of the same name as the event type, and the string value which that attribute is equal to shall be the method name to call on that element. The parameters to the method will always be the event itself, which includes its payload."

For example, for custom event type "drilldown", the code looks for all elements with a "drilldown" attribute, <some-component drilldown="method">, and will use the value of that attribute, "method", as the method to call: someComponent.method(event).

I previously had a backup plan when getAttribute found an empty string, as in the case of <some-component drilldown>, where it would use "drilldown" as the method to call as well, but that again couples the two components too tightly. Plus, attributes are always lowercased / case-insensitive while javascript is not, meaning method "drillDown" won't be found, leading to confusion.

For a component to use the code, it had to create a custom event which bubbles. This is extra typing at every dispatch call, which is tiresome. So I added the third parameter "true" to addEventListener so it catches during the Capture phase of an event, which tidies up all dispatch code everywhere.

['drilldown', 'pieChartInit', 'showTip', 'hideTip']
    .forEach(et => 
        document.addEventListener(et, ev => 
            document.querySelectorAll(`[${ev.type}]`).forEach(el =>
                el?.[el.getAttribute(ev.type)]?.bind(el)(ev)), true));

The other tack I tried was not going with minimal code at all, and putting said code into a "library.js" file. With less emphasis on minimalism, I added a few quality-of-life features. I prepended "on" to the event type, so the event "drilldown" would be caught with attribute "onDrilldown", leading to more readable HTML: <some-component onDrilldown="method">. I added a "register" function which avoided the need to list all possible custom event types in the app in the initial string array. Finally, I added a "say" function which the components could use to dispatch a custom event more succinctly than the native syntax, which also registered the event type if it hadn't been already, avoiding the need for the components to call register at startup.

const registeredCustomEventTypes = {};

export function say(element, customEventType, detail) {
    if (!registeredCustomEventTypes[customEventType])
        register(customEventType);
    element.dispatchEvent(new CustomEvent(customEventType, { detail }));
}

export function register(customEventType) {
    document.addEventListener(customEventType, go, true);
    registeredCustomEventTypes[customEventType] = true;
}

function go(event) {
    const onEventType = 'on' + event.type;
    document.querySelectorAll(`[${onEventType}]`).forEach(element => 
        element?.[element.getAttribute(onEventType) || onEventType]?.bind(element)(event));
}

I think using this library code may be a bad idea. One advantage of HTML Custom Elements is creating a true single-file component: zero dependencies, not even on a library. We lose that advantage by including the library. Secondly, it requires each component that is used with it to be aware of this library code and interact with at least its register function, which starts coupling components together again, albeit through a common intermediary. Although that could be avoided by putting all the require calls in the top-level glue code, that's just a more verbose version of the string array in my first code sample.

I find it odd that browsers have no built-in Event Registry for custom events. Even the minimal code still slightly couples components together by passing the raw event to the handler, instead of picking out the necessary information from the event's payload and giving the method only what it needs in whatever format it wants it. That kind of minor data transform, which perfectly preserves the components' individuality, makes every event-to-method tie unique.

And if every event-to-method tie is unique, then every handler is unique, which means every addEventListener on the root is unique, which means there can be no general message bus at all. Whatever form such a bus would take, it implicitly demands all components conform to its interface, just as my library.js did. There's no way to win here.

document.addEventListener('drilldown', e => {
    document.querySelectorAll('chart-legend').repopulate(e.detail.fieldName, e.detail.newData);
});    

document.addEventListener('drilldown', e => {
    document.querySelectorAll('field-selector').refresh({
        Old: e.detail.fieldName, 
        New: e.detail.meta.fields.map(f=>f.name),
    });
});    

// etc., forever

The Many Ways of Templates in HTML Custom Elements

Ron Newcomb — Sun, 13 Sep 2020 04:40:45 +0000

Already having a couple years each in React and Angular I decided to learn HTML Custom Elements, the browser's native component framework. "This should be easy," I thought, "Just tell me where to put the {{substitutions}} and I'll go from there."

Ha.

Point #1: such substitutions are part of a templating engine, which the native framework lacks. In this article I'll detail many choices in handling the HTML portion of a component, such as what format to keep it in, how to parameterize it, when to use the <template> tag, how to render and then re-render it, and how to know when to re-render it.

In these examples I'll be using single-file components with zero dependencies, so no fetching HTML separately from the imported code. I'll use a little bit of Typescript for documentation purposes. I'll also hold the preference of always using the Shadow DOM, and always keeping the HTML/CSS at the bottom of the code file, not because it is easy but because it is hard.

Our example is a <flex-row> component, which merely styles its children in a flexbox's row alignment. Since it accepts child nodes as input, we'll see the <slot> tag instructing the native framework where to put those children. Example usage:

<flex-row>
  <div>Left sidebar.</div>
  <div>Main content, containing a very long text that wraps around the viewport multiple times. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</div>
</flex-row>

Simplest Example

customElements.define('flex-row', class extends HTMLElement {
  connectedCallback() {
    this.attachShadow({ mode: 'open' }).innerHTML = /*html*/`<slot style="display: flex; flex-direction: row; flex: 1 1 auto; align-items: flex-start;"></slot>`;
  }
})

By inlining the class definition right inside the final .define call, we don't actually need to name the class. .attachShadow creates the Shadow DOM, and returns a reference to it which we can immediately .innerHTML it with a string which is our initial template.

The /*html*/ comment is a hint to our IDE that the backtic string should be color-syntax highlighted as HTML. CSS is inlined but a <style></style> after the </slot> works just as well. But if our template is large we'd probably move it out of the code to the bottom of the file...

Put Template At Bottom

customElements.define('flex-row', class extends HTMLElement {
  connectedCallback() {
    this.attachShadow({ mode: 'open' }).innerHTML = template();
  }
})

function template() {
  return /*html*/`<slot style="display: flex; flex-direction: row; flex: 1 1 auto; align-items: flex-start;"></slot>`;
}

Moving the template to the end of the file looks cleaner. But, calling .define so early can error with template() is undefined if we used const template = () => .... So here I must use function so hoisting happens. Or I could name the class and move just the .define call to the very bottom... where it's easy to forget it.

Anyway, let's add a parameter <flex-box wrap='nowrap'> for our template to wrangle.

Add Parameter for `flex-wrap`

customElements.define('flex-row', class extends HTMLElement {
  connectedCallback() {
    const wrap = this.getAttribute('wrap');
    this.attachShadow({ mode: 'open' }).innerHTML = template(wrap);
  }
);

function template(flexWrap: string) {
  return /*html*/`<slot style="display: flex; flex-direction: row; flex: 1 1 auto; align-items: flex-start; ${flexWrap ? `flex-wrap: ${flexWrap};` : ''}"></slot>`;
}

One difference in attaching the Shadow in connectedCallback instead of the constructor is that the element's attributes are ready. Our template function is parameterized like any function.

Now, using this feature is weird. Why <flex-box wrap='nowrap'> instead of just <flex-box nowrap> or <flex-box wrap>? It also never re-renders. Let's add both of these features.

Render Method and Property Watching

customElements.define('flex-row', class FlexRow extends HTMLElement {
  connectedCallback() {
    this.wrap = typeof this.getAttribute('wrap') === 'string' ? 'wrap' : typeof this.getAttribute('nowrap') === 'string' ? 'nowrap' : '';
    this.attachShadow({ mode: 'open' }).innerHTML = template(this.wrap);
  }

  #wrap = '';
  get wrap() { return this.#wrap; }
  set wrap(v) { this.#wrap = v; FlexRow.render(this); };

  static render({ shadowRoot, wrap }: FlexRow) {
    if (shadowRoot) shadowRoot.innerHTML = template(wrap);
  }
});

function template(flexWrap: string) {
  return /*html*/`<slot style="display: flex; flex-direction: row; flex: 1 1 auto; align-items: flex-start; ${flexWrap ? `flex-wrap: ${flexWrap};` : ''}"></slot>`;
}

There's a lot to unpack here.

1) We check both attributes wrap and nowrap on init.
2) We store the appropriate string in a javascript private variable this.#wrap. (Typescript shims this since browser support isn't yet complete, but my later examples won't use it anyway.)
3) We create a getter/setter pair so when outside code changes the property we can react.
4) We declare a static render function for the re-render.

Now the consumer can initialize our component with short attributes like <flex-row wrap> and <flex-row nowrap> but then change them anytime with the property ourElement.wrap = 'nowrap';.

If we didn't make the backing field #wrap a private member, then the consumer could just hit it directly with ourElement._wrap = ... and be left wondering why nothing works, and why is there both a wrap and a _wrap property visible in the browser devtools.

Similarly, we hide our render function by making it static. This does require naming the class, but in return we can destructure the parameter to avoid a lot of this. prefixes.

Point #2: attributes and properties don't automatically mirror each other in HTML Custom Elements. Think of the .value of <input type=text value='initialVal' />: the attribute has the initial value and is effectively a constant, while only the property updates according to the user. Everything works that way with HTML Custom Elements.

But what if we wanted some mirroring, especially so CSS could alter itself based on whether the flex-row was currently wrapping or not? CSS only looks at attributes not properties...

Render Method and Property Watching II

class FlexRow extends HTMLElement {
  connectedCallback() {
    this.wrap = typeof this.getAttribute('wrap') === 'string' ? 'wrap' : typeof this.getAttribute('nowrap') === 'string' ? 'nowrap' : '';
    this.attachShadow({ mode: 'open' }).innerHTML = template(this.wrap);
  }

  get wrap() { return this.getAttribute('wrap'); }
  set wrap(v) { this.setAttribute('wrap', v || ''); render(this); };
}

const render = ({ shadowRoot, wrap }: FlexRow) => shadowRoot ? shadowRoot.innerHTML = template(wrap) : null

const template = (flexWrap: string | null) =>
  /*html*/`<slot style="display: flex; flex-direction: row; flex: 1 1 auto; align-items: flex-start; ${flexWrap ? `flex-wrap: ${flexWrap};` : ''}"></slot>`;

customElements.define('flex-row', FlexRow);

A couple of things here.

Since we need to update the attribute whenever the property changes, the attribute might as well be the backing field.

For the sake of example I moved .define to the very bottom, allowing us to use arrow functions for template(). Since that's a pretty one-liner we'll do render() the same way as an alternative to the static.

The new features come with a couple of issues.

The nowrap attribute lacks a matching property, and the wrap property accepts the string "nowrap", which can confuse our users. I'll defer on this design issue since it's unrelated to templates but did want to call it out.

More importantly, later updates to the attributes via ourElement.attributes['wrap'] = 'wrap' or ourElement.setAttribute('wrap','wrap') will do nothing, even though the attributes worked initially, potentially disappointing our users.

Property AND Attribute Watching

class FlexRow extends HTMLElement {
  connectedCallback() {
    this.wrap = typeof this.getAttribute('wrap') === 'string' ? 'wrap' : typeof this.getAttribute('nowrap') === 'string' ? 'nowrap' : '';
    this.attachShadow({ mode: 'open' }).innerHTML = template(this.wrap);
  }

  get wrap() { return this.getAttribute('wrap'); }
  set wrap(v) { this.setAttribute('wrap', v || ''); render(this); };

  static observedAttributes = ['wrap', 'nowrap'];

  attributeChangedCallback(name: string, oldValue: string, newValue: string) {
    if (oldValue !== newValue) render(this);
  }
}

const render = ({ shadowRoot, wrap }: FlexRow) => shadowRoot ? shadowRoot.innerHTML = template(wrap) : null

const template = (flexWrap: string | null) =>
  /*html*/`<slot style="display: flex; flex-direction: row; flex: 1 1 auto; align-items: flex-start; ${flexWrap ? `flex-wrap: ${flexWrap};` : ''}"></slot>`;

customElements.define('flex-row', FlexRow);

Adding attributeChangedCallback, and listing the possible values for its first argument in the static (and required) observedAttributes allows us to react to attribute changes. Normally we need a this[name] = newValue; in that function to copy the value from the attribute to the property, but since our property uses a getter/setter already pointed to the attribute, there's no need. We of course must check for infinite loop.

I said at the start of this article there's no simple substitutions like in React and Angular. We've been using template string literals and .innerHTML to parse the string for us. This is fine if the component is never re-rendered, being both permanent and one-use, like a main-menu bar. But when you introduce re-rendering, the re-parsing of strings has a hefty performance hit.

Let's fix our performance problem with a single string parse for the initial render, and a more surgical update for re-rendering. We'll need the <template> HTML tag and the .cloneNode method for this one.

Precompile Template, Minimal Re-render

const template = /*html*/ `
<template>
    <slot style="display: flex; flex-direction: row; flex: 1 1 auto; align-items: flex-start;"></slot>
</template>`;

let fragment: DocumentFragment;

class FlexRow extends HTMLElement {
  static observedAttributes = ["wrap", "reverse"] as const;

  #elementsCache: {
    rootElement?: CSSStyleDeclaration;
  } = {};

  connectedCallback() {
    if (!fragment) fragment = new DOMParser().parseFromString(template, "text/html").head.querySelector("template")!.content;
    const myFragment = fragment.cloneNode(true) as DocumentFragment;
    this.#elementsCache.rootElement = (myFragment.firstElementChild as HTMLElement).style;
    for (const attribute of FlexRow.observedAttributes) this.attributeChangedCallback(attribute, "", this.getAttribute(attribute));
    this.attachShadow({ mode: "closed" }).appendChild(myFragment);
  }

  // reflect properties to attributes
  get wrap() {
    return this.getAttribute("wrap");
  }
  set wrap(value: string | null) {
    this.setAttribute("wrap", value || "");
  }

  get reverse() {
    return this.getAttribute("reverse");
  }
  set reverse(value: string | null) {
    if (value == null) this.removeAttribute("reverse");
    else this.setAttribute("reverse", value);
  }

  // reflect attributes to properties
  attributeChangedCallback(name: (typeof FlexRow.observedAttributes)[number], oldValue: string | null, newValue: string | null) {
    if (oldValue === newValue) return; // prevent infinite loop
    //console.log("name", name, "old", oldValue, "new", newValue);
    switch (name) {
      case "wrap":
        this.#elementsCache.rootElement!.setProperty(name, newValue);
        return;
      case "reverse":
        // newValue = null means attr is now missing;  newValue == "" means attribute there
        this.#elementsCache.rootElement!.setProperty("flex-direction", newValue == "" || (!!newValue && newValue != "false") ? "row-reverse" : "row");
        return;
    }
  }
}

customElements.define("flex-row", FlexRow);

Our template() function is now just a variable no longer accepting parameters because it's only going to be used once. Its contents are wrapped in a <template> tag because now the DOMParser parses the text and returns the disconnected node tree. We then immediately querySelector it for the <template> tag, and put the content into a variable for use by every instance of <flex-row> afterward. It will never be re-parsed again.

Given the template that this particular instance of flex-row will use, we then find and cache each element or piece that changes at runtime in response to attribute/property changes. Here it's just the .style of the root element.

When a property changes, we reflect its new value into the corresponding attribute. When an attribute changes, the standard attributeChangedCallback is called. It's here where we update our component's HTML. We also call this from a loop in connectedCallback on initialization to remove all dummy values that were cloned from the static template.

Note that if oldValue or newValue are null, this means the attribute didn't exist at all, which is a distinct case from it existing but with an empty string.

This way is a bit more typing and complexity, but on re-render, that sole setProperty is astronomically faster than creating and parsing strings, especially if said string invokes other custom components.

I wrote a fake UI framework in 3 lines of code

Ron Newcomb — Sun, 06 Sep 2020 02:20:28 +0000

I wanted one of my new personal UI projects to be a back-to-basics affair. After six years of Angular/React I decided I'd keep my simple project simple and see if I still could, for example, make main menu dropdowns in pure CSS. Do things like the good old days but with CSS Grid and arrow functions.

I got an "append-only" CSS monstrosity in a single day. My index.html was only better in that it might take a whole week before sprouting tentacles in that same disturbing shade of purple.

When I broke apart the large files into smaller ones, I found myself using a structure that closely mirrored web components. The fancy popovers, for example, required a lot of CSS with a touch of HTML and JS. Dropdown menus and layout helpers etc. all required most if not all three of HTML, CSS, and JS. Folders were rearranged. File-naming conventions happened.

I also missed having a readable DOM. I missed looking at useful HTML tags instead of an endless parade of div and span. I still didn't need a reactive framework that re-renders custom tags on-the-fly in response to auto-diffing of JS model data, and .innerHTML was working fine for what little wasn't static, but I did want something better than the, ahem, bad old days. I wanted readable HTML and good code structure even in this mostly static site. This is what I wrote.

 <script type="module">
      /* the "render once" UI framework */

      const loadHtmls = element =>
        Array.from(element.children)
          .map(child => (child.tagName.includes("-") ? loadHtml : loadHtmls)(child));

      const loadHtml = el =>
        fetch("html/" + el.tagName + ".html")
          .then(response => response.text())
          .then(html => {
            if (html.match(/{{innerHTML}}/)) html = html.replace(/{{innerHTML}}/g, el.innerHTML);
            Array.from(el.attributes).forEach(attr => (html = html.replace(new RegExp("{{" + attr.name + "}}", "g"), attr.value)));
            el.innerHTML = html;
          })
          .then(_ => loadHtmls(el))
          .then(_ =>
            Array.from(el.querySelectorAll("script")).forEach(old => {
              const replacement = document.createElement("script");
              replacement.setAttribute("type", "module");
              replacement.appendChild(document.createTextNode(old.innerHTML));
              old.replaceWith(replacement);
            })
          );

      loadHtmls(document.body);
    </script>
  </body>
</html>

The last line invokes the first line on the displayable bits of index.html in which this snippet lives.

The first line looks through the immediate children of the passed-in element. If the element's tag contains a hyphen <like-this></like-this> then the element is passed to the second line. Else the element is passed to the first line for recursion.

The HTML5 spec states that a tag containing a hyphen is guaranteed to always be UnknownElement. I have no way to tell a built-in div from a custom foobar, and that's an easy restriction to live with to keep this snippet low effort.

Another restriction tradeoff, this time for the second line, requires all "components" to be in the same folder, named after the tag they implement. Even though the single-folder rule won't scale well, the filename convention doubles as a wonderful code organization tenet. <some-component> is implemented by some-component.html, which in turn uses <style>, <script> or <link> to involve CSS and JS. Easy.

The second line fetches the component file from the server, replaces substitutions {{likeThis}} in its HTML for the values of the same-named attribute: <some-component likeThis='42'>. It also replaces {{innerHTML}} with the element's pre-render innerHTML which is useful for components like <flex-row> which wrap what goes into the flexbox.

    <flex-row wrap="nowrap">
      <named-panel name="Sidebar"></named-panel>
      <named-panel name="Main Content"></named-panel>
    </flex-row>

Finally, we recurse into the newly-updated DOM so the custom components can invoke other custom components. Only after we finish the DOM do we run scripts, which might addEventListener DOM elements. (HTML5 proscribes running scripts automatically, and I've heard bad things about eval even though it's dramatically less typing.)

This "framework" obviously lacks too much to be a framework. The whole thing could probably be ran and rendered server-side into a nearly completely static site since vanilla JS does any actual runtime changes. And yet like most personal projects, I'm kind of fond of it, in the same way I might be fond of a little pet dog that's maybe kind of ugly to most people. But he's my pet dog, and he looks a lot better since we removed the tentacles.

A Secret to Writing Nice Code

Ron Newcomb — Sat, 02 May 2020 23:23:55 +0000

We all know the basics of writing nice, readable, maintainable code. Shorter code is usually better, so omit needless lexemes. Use meaningful names because heavy commenting is a hack. Parallelism (the literary kind) trumps all. But I find one of the unsung heroes of nice code is choosing to create just the right helper function.

Here's a react example.

{items.length < 2 ? (
  <>{items.map(item => renderItem(item, allOtherVars)}</>
) : (
  <ItemGroup>{items.map(item => renderItem(item, allOtherVars)}</ItemGroup>
)}

We want the ItemGrouping wrapper if the array has more than one item. The renderItem function is non-trivial and only exists because we don't want to duplicate all the code inside it. But declare a helper like so.

export const ItemGroupIf = ({ if: ig, children, ...otherProps }: Props & { if: boolean }) =>
  ig ? <ItemGroup {...otherProps}>{children}</ItemGroup> : <>{children}</>;

This allows us to rewrite the client code as this.

<ItemGroupIf if={items.length > 1}>
  {items.map(item => {
    /* the body of renderItem */
  })}
</ItemGroupIf>

This allows us to avoid declaring the workaround function renderItem, avoid passing around several variables to it, to almost hide the condition (possibly reducing cyclomatic complexity depending on how its counted), and primarily just removes the one-off "wiring" in favor of a re-usable component that provides these same benefits wherever else in the codebase this issue occurs.

Yet the helper in question doesn't do anything complex or "worth saving" in a dedicated utility function, so if you're just browsing the Utility class or file or wherever you keep these things and happen upon it, it seems like a stupid function.

Here's another example in C#.

The following function, which is one of many just like it, is getting information from several API calls and amalgamating them into a larger object. A few of the API calls are required and a few are optional. Each call returns a wrapper object containing either the result or an error object. So we use a lot of if statements checking return values.

public FinalResult GetAggregateResult(int id)
{
    var primary = await FirstCall(id);
    if (primary.Problem != null) 
    {
       logger.Error(primary.Problem);
       return new FinalResult(primary.Problem);
    }

    var secondary = await SecondCall(primary.secondId);
    if (secondary.Problem != null) 
    {
       logger.Error(secondary.Problem);
       return new FinalResult(secondary.Problem);
    }

    var optional1 = await OptionalCall1(id);
    if (optional1.Problem != null) 
    {
       logger.Info(optional1.Problem);
    }

    var optional2 = await OptionalCall2(primary.pieceId);
    if (optional2.Problem != null) 
    {
       logger.Info(optional2.Problem);
    }

    return new FinalResult() {
      x = primary.result;
      y = secondary.result.moneyShot;
      a = optional1.Problem != null ? optional1.result : null;
      b = optional2.Problem != null ? optional2.result?.piece?.subpiece : null;
    };
}

The try-catch was invented so we could separate the happy and unhappy path, simplifying both. But none of our calls can actually throw on error because the caller might decide its optional. Exceptions throws are expensive and should be avoided for normal processing, and in many of our use-cases a 404 is perfectly normal. Yet the if statements dominate the conversation the function is trying to have with its upper and lower levels.

So here's a few stupid functions. The first two are instance methods on the returned wrappers, the second two are C# extension method on C#'s awaitable Task type. (Define ResultException as a simple wrapper if not already done so.)

public T ThrowIfError()
{
    if (Problem != null)
        throw new ResultException(this);
    else
        return Value;
}

public T LogIssue(Logger logger)
{
    if (Problem != null)
        logger.Info(Problem);
    return Value;
}

public static async Task<T> ThrowIfError<T>(this Task<LowerResult<T>> gettingResult) 
    => (await gettingResult).ThrowIfError();

public static async Task<T> LogIssue<T>(this Task<LowerResult<T>> gettingResult, Logger logger) 
    => (await gettingResult).LogIssue(logger);

Now we write our caller code to use exception handling, but we, not the lower levels, decide if a call throws or not.

public FinalResult GetAggregateResult(int id)
{
  try 
  {
    var primary = await FirstCall(id).ThrowIfError();

    var secondary = await SecondCall(primary.secondId).ThrowIfError();

    var optional1 = await OptionalCall1(id).LogIssue(logger);

    var optional2 = await OptionalCall2(primary.pieceId).LogIssue(logger);

    return new FinalResult() {
      x = primary;
      y = secondary.moneyShot;
      a = optional1;
      b = optional2?.piece?.subpiece;
    };
  } 
  catch (ResultException rex)
  {
    logger.Error(rex.Problem);
    return new FinalResult(rex.Problem);
  }
}

The instance method ThrowIfError either unwraps and returns the Result, or unwraps the Error and throws an exception, separating the unhappy path. No changes happened to the lower level calls. The extension method allows us to avoid extra parenthesis in (await Call(x)).ThrowIfError() which are easy to forget and hard on the eyes.

Similarly the LogIssue unwraps the Result and always returns it, even if null, but quietly logs any issue so we don't need to clutter the caller with checks and logs.

We all like code where it's easy to see what's going on, and we almost always prefer less code over more. But in all the conversation I hear about naming conventions and architecture and "how we do things here", I rarely hear anyone championing the humble helper function as a strong tool for producing nice code.

Why can't Form Validation be nice?

Ron Newcomb — Wed, 01 Jan 2020 22:23:03 +0000

I've used five different ways of doing form validation now, two in React and two in Angular and one with just jQuery, and I hate them all.

Why is form validation, especially in the age of Typescript, so long-winded, so convoluted, so many lines of code? Why isn't form validation nice? It's just a few boolean functions, right? So why does it require 50 lines of code for a simple email-and-password form, let alone anything complex?

A model has properties that are string, number, boolean, Date, enum, and arrays or sub-objects also comprised of same. But HTML forms largely have only one type, that of string. So a form validation library must perform model mapping to turn non-strings into strings on form init, and then map them all back to non-strings on blur (or even more often) to validate. So form validation implies a layer of model mapping, even though 95% of the time it's so straightforward it basically goes unused.

Model mapping itself is a bit of a headache. Although a native datatype, dates lack a HTML input element dedicated to them so we'll always need some custom datepicker component that works who-knows-how. Enums have two elements, the dropdown and the radio button set, even though javascript lacks a native enum type. And then you get a work ticket wanting to represent an expiration date with a lone checkbox element which "means 60 days into the future if checked, null if unchecked, or preserving the same value it had on form init no matter how many times the box is unchecked then rechecked, unless it was left unchecked on submit in which case undefined is fine."

Form fields don't map to validation functions 1-to-1. The same field will have multiple constraints like required and range and length, but combining them into a single function limits re-use. But if we library-ize them, then our fields must work with the standard functions as well as custom functions, and syntax is almost always nicer for one set than the other.

Form fields need outside information to validate. AngularJS learned this lesson the hard way when validating a field depended on the value in another field. (Angular2+ then explicitly added ways for cross-field validation.) Some validations require an async call to see what the valid values even are. For example, the province/state field relies on the value of the country field above it, but there are a whole lot of provinces in the world so wait on fetching the list until after country is chosen.

Mapping itself can throw surprise validation errors of the is-it-plugged-in variety. How many times have we tried to use a new DatePicker but it fails either on the initial date-to-string conversion, or the string-to-Date submit conversion? It's a silently understood validation rule that a date must be a valid date, a number a valid number, yet we're surprised when a datepicker passes all its Required and Less-Than-Expiration rules, but fails anyway on an unwritten rule because of parsing.

The final format of the Errors object can cause lines of code. If Errors is an array of keywords then showing lastName's Required message involves a lengthy array.find invocation. If Errors is an object then asking how many involves a nested Object.keys invocation. Nested, because the Errors object can't have a simple flat structure. Multiple fields can fail Required, and one field can fail all of its validations simultaneously.

When it comes to showing the error messages there's also several fine ways of doing so. Add or remove CSS classes that control a div's visibility, or pass something from Errors to a component as in <Err show={errors.lastName.required}>Last Name is required</Err>. Sometimes a pre-existing form validation library doesn't intersect well with a pre-existing UI Elements kit and that causes a lot of boilerplate: the validation gives an Errors object but the UI kit wanted it to toggle a particular classname on a particular element, so we have to glue it together.

And I haven't even mentioned forms which have an array of stuff in them, like an array of addresses. Validation functions need to know if their result and the field they are attached to are irrelevant because the user deleted the third address. Dangling references annoy.

Many nice solutions for mapping fail when the model has a sub-object, even one as simple as three properties year/month/day.

But when I see every single <input/> element on every single form in the whole app possess a filled-in name, type, id, value, onBlur, onChange, onInit and various data-* properties, sixteen React Hooks or a page of Angular's type-unsafe FormBuilder, regexes embedded within HTML, and onSubmit pointed to the validation library's onSubmit handler which then takes another onSubmit handler as a parameter which is our actual onSubmit handler, which can fail form validation after submission because the server said so and how do we even, I just gotta ask: why can't form validation be nice?