Forem: Sean Allin Newell

Effect in React

Sean Allin Newell — Fri, 24 May 2024 13:44:20 +0000

Emerging from ZIO in the lands of Scala, there is a new ecosystem of functional programming edging its way into TypeScript - Effect.

ZIO: For more about this ZIO and Scala lineage, check out this video from Effect Days 2024.

Behold the power! The following are adaptations taken from the Effect home page, showing examples of what you can do with Effect.

Want to fetch some JSON from an endpoint, handling 200 OK and valid JSON errors, boom.

const getUser = (id: number) =>
  Http.request.get(`/users/${id}`).pipe(
    Http.client.fetchOk,
    Http.response.json,
  )

Want to retry that endpoint if something goes wrong? Did somebody order a one line fix?

--- http.ts 2024-05-21 19:23:24.194145996 +0100
+++ http-with-retry.ts  2024-05-21 19:23:33.524145427 +0100
@@ -3,4 +3,5 @@
     Http.client.fetchOk,
     Http.response.json,
+    Effect.retry({ times: 3 })
   )

How about a more controlled timeout, let's say 3 seconds? Another one liner.

--- http-with-retry.ts  2024-05-21 19:45:49.874052498 +0100
+++ http-with-retry-and-timeout.ts  2024-05-21 19:45:41.354053136 +0100
@@ -3,5 +3,6 @@
     Http.client.fetchOk,
     Http.response.json,
+    Effect.timeout("3 seconds"),
     Effect.retry({ times: 3 }),
   )

Do you want more control on the retry, like an exponential back off? Effect has you covered😎

--- http-with-retry-and-timeout.ts  2024-05-21 19:45:41.354053136 +0100
+++ http-with-retry-timeout-backoff.ts  2024-05-21 19:47:56.084044043 +0100
@@ -4,5 +4,9 @@
     Http.response.json,
     Effect.timeout("3 seconds"),
-    Effect.retry({ times: 3 }),
+    Effect.retry(
+      Schedule.exponential(1000).pipe(
+        Schedule.compose(Schedule.recurs(3)),
+      ),
+    ),
   )

Want a schema to parse the JSON into? Effect.schema. Want to throw in an abort controller? Http takes in a signal.

These are all composable additions to the program we started with, and each of these pieces can be transparently reused and mixed into other Effects. Kind of like how you snap legos together, or how types help us reason about our program. The Effect type is core to this composition, so it's worth a bit of an introduction

The Effect Type

The core of Effect lies in theEffect type, defined here. The docs linked say this about the type parameters:

type Effect<TSuccess, TError, TRequirements> = ...

Success : Represents the type of value that an effect can succeed with when executed. If this type parameter is void, it means the effect produces no useful information, while if it is never, it means the effect runs forever (or until failure).

Error : Represents the expected errors that can occur when executing an effect. If this type parameter is never, it means the effect cannot fail, because there are no values of type never.

Requirements : Represents the contextual data required by the effect to be executed. This data is stored in a collection named Context. If this type parameter is never, it means the effect has no requirements and the Context collection is empty.

We're mostly going to focus on the first two for this first blog post, but don't sleep on Context (aka requirements), as it is how Effect can do Type-first Dependency Injection, nice! ✨

So in addition to the semantic meaning we give the three parameters, the other truly important thing to know about Effect is that it is lazy. The JSDoc block describing the type from the source code says:

The Effect interface defines a value that lazily describes a workflow or job.

This is critical, especially when we bridge the Effectual world to the normal world.

Let's do that now, with a small React app!

✋ We're not going to lay down patterns I'm happy with to ship to production (yet!). I'm still learning Effect after all. If you go on the Effect Discord's react channel, there is chatter about a library called effect-rx which is in active development. I'd encourage you to try to use it if you want to be a pioneer, a potential contributor, or want to see what it has to offer.

In the next blog post I do on Effect, I'll share another stepping stone, and before too long I'll be confident enough to ship to production with Effect in the client 💪! If you're more interested in server side work, you should check out the examplesand the talks, as it is far more "off the shelf" I would say.

useEffectEffect?

💡 Yes, the library is, unfortunately, called Effect. And in React, we already have a concept with that name - the useEffect hook. It's worth repeating that the react docs highlight useEffect as a way to "...connect to and synchronize with external systems" for "...non react code...". Some words I've been playing around with to differentiate the two worlds:

ReEffect

Coalesce

Collapse

Bridge

Right now, I hate all of them. We'll see what sticks! 😅

If we fire up a quick vite app and get the famous counter going, we can begin by installing effect (I use pnpm, feel free to use your favorite package manager):

pnpm create vite my-vue-app --template react-ts
# Make sure your tsconfig is strict!

pnpm add effect

💡 If you are worried about bundle size, I wouldn't be (yet). At least for our small example things are all still gzipped under 100Kb. The first screenshot below is the default, single chunk of the app we're building today, and the second is with two chunks, a ui chunk for react+react-dom, and a 2nd chunk just for effect. I did not see any difference in bundle size with more specific imports.

Before we do anything useful, let's test our sanity by doing something we think should be trivial, like saying hello world. In Effect, the simplest thing you can do is succeed.

Effect.success("Hello world!")

Great! If we look at the type signature, it describes the program we created by using success: Effect<string, never, never> which means this is a program that when run will yield a string, never error, and requires no context. Cool. 🍨

This is a lazily evaluated program (sort of... we did not pass in a lazy value, so the string is evaluated, we'll get to that), so we now need to execute or resolve it. Let's do that with runSync

const HiEff = Effect.success("Hello world!");
const hi = Effect.runSync(HiEff);

console.log(hi);

Now we see our glorious "Hello world!" in the console, so let's wrap this up in a component and spit it out on the browser

// declare Effects _outside_ of React.
const HiEff = Effect.success("Hello world!");

export const HelloEff = () => {
  // execute Effects once, on mount
  const hi = useMemo(() => Effect.runSync(HiEff), []);

  return <h1>{hi}</h1>;
}

// elsewhere, in App.tsx
return <HelloEff />

Nice! Feel free to change the text and ensure all the react-refresh / hot module reloading / vite / fast go-go juice all works, it does for me!

🤔 I've explicitly said to put the Effect outside of React - very purposefully. We'll have to create or leverage Effects that take in user input eventually, but as much as possible put code outside of the render function of a component. That's generally true with and without Effect btw.

Now you may be asking... okay, that's nice, but what about promises and all that async stuff? Well my friend, there's a run promise as well! Keeping our sanity still, let's just try to lift the runSync to promised land and see if we can get it into react, as that part is odd. Since we're going to run our Effect in a Promise, we can't just use the promise in our markup, unless we're on the server and can await or can suspend somehow. The more direct solution is to have some state, just like we would with fetch (or if we were to reach for react-query, just like how react-query stores the response of our n/w call, in state). That looks like this:

// declare Effects _outside_ of React.
const HiEff = Effect.success("Hello world!");

export const HelloEffAsync = () => {
  // Double up as our loading state, and storing the result.
  const [result, setResult] = useState<string>("...");

  useEffect(() => {
    // Behaves the same as Promise.resolve
    Effect.runPromise(HiEff).then(setResult);
  }, []);

  return <h1>{result}</h1>;
};

This looks like it synchronously resolves, so let's add a bit of a delay, with Effect of course, for that one line goodness.

// Leverage Effect.delay instead of timeouts
const HiEff = Effect.delay(Effect.success("Hello world!"), "2 seconds");

export const HelloEffAsync = () => {
  const [result, setResult] = useState<string>("...");

  useEffect(() => {
    Effect.runPromise(HiEff).then(setResult);
  }, []);

  return <h1>{result}</h1>;
};

look ma', no (explicit) setTimeout or whacky promise shenanigans!

Now, with the delay, we actually see the "..." loading string that we used as the initializer to useState first, then the promise resolves. OK - sanity confirmed, async experienced.

❌ Errors & Effect & React

Our sync and async workflows were simple Success Effects, now let's Fail and see what happens. In Effect, you can 'throw' with Effect.fail(...), so let's do that:

const FailEff = Effect.fail("not feeling it...");

Let's see what happens if we runSync a failure:

(FiberFailure) Error: Error: Not feeling it

Oof. An exception, what is this, try-catch town clown 🤡 city?

const FailEff = Effect.fail("Not feeling it");

let result;
try {
    result = Effect.runSync(FailEff);
} catch (e) {
    result = "Exception: " + String(e)
}

console.log(result); // => Exception: (FiberFailure) Error: Not feeling it

At least our program didn't crash, but is there a better way to statically match over the error we know will happen? After all, the type signature of FailEff describes a program that never returns, errors with a string, and needs no context.

There is! There is a type Effect provides called Exit and it has async and sync versions of the run methods, so we wrap our result in a structure we can use. Check out the docs here, and let's take it for a spin:

🧠 Effects are composable, and you don't need to use what you don't understand. If you see something useful, just start with that and see if it helps you make apps with more confidence.

import { Cause, Effect, Exit, identity } from "effect";

function handleLeftRight(left: string, right: string) {
    return `Unexpected exit: (${left}, ${right})`;
}

const FailEff = Effect.fail("Not feeling it");
const ResultExit = Effect.runSyncExit(FailEff);

// Effect provides these handy match functions to map over all possibilities of a type
const result = Exit.match(ResultExit, {
  onSuccess: () => "Were we secretely feeling it?",
  onFailure: (cause) =>
    // The type of cause is Cause, which can
    // encapsulate many kinds of failures.
    Cause.match(cause, {
        // We expect this to be the only path taken, given that our program
        // is Effect<never, string, never>
        onFail: identity,
        onDie: (_defect) => "Unexpected die from a defect\n" + cause.toJSON(),
        onInterrupt: (fiberId) =>
            `Unexpected interrupt on fiber ${fiberId}\n` + cause.toJSON(),
        onParallel: handleLeftRight,
        onSequential: handleLeftRight,
        onEmpty: "Empty"
    })
});

console.info(result); // => "Not feeling it"

Awesome! You may have experienced one of two emotions:

Wow - Effect really makes you cross your ts and dot your is.
That was annoying inside the failure case, if I console log out the cause, I see that it has a failure / error I can just pull out!

By convention (or maybe coercion) there is a _tag field on many (all?) types that support this .match behaviour, so we can do our own kind of collapsing with a switch if we wanted to. This would look like this:

const result = Exit.match(ResultExit, {
  onSuccess: () => "Were we secretely feeling it?",
  onFailure: (cause: Cause.Cause<string>) => {
    // We can do our own pattern matching
    switch (cause._tag) {
        case "Fail":
            return cause.error;
        default:
            return "Unexpected failure";
    }
  }
});

This is still type safe, as _tag is discriminating the union and we still get result: string - but word to the wise, try to leverage match when possible, as it does some narrowing for you (notice the identity we were able to use in the nested match above), which is quite nice to "zoom" in to the underlying type of each case, when we build the switch ourself, TypeScript has narrowed the type, but we have to do the selection and mapping ourself ( cause.error ) - which isn't a big deal in this case, but could be for others.

runPromiseExit: Try to switch to an async version of the sync console program shown above.

Hint - if you are using node v20/bun you can use top level awaits, and the code changes remarkably little!

Now let's bring this home into react! We can start by just trying to do it all in the component, but pretty soon a wee custom hook will "pop" out of this work:

const FailEff = Effect.delay(Effect.fail("I can't even"), "2 seconds");

type States = "processing" | "success" | "failure";

export const HelloEffAsync = () => {
  const [effState, setEffState] = useState<States>("processing");
  const [result, setResult] = useState<string | null>(null);

  useEffect(() => {
    Effect.runPromiseExit(FailEff).then((FailEffExit) => {
      Exit.match(FailEffExit, {
        onSuccess: () => {
          setResult("I CAN even!");
          setEffState("success");
        },
        onFailure: (cause) => {
          Cause.match(cause, {
            onFail: (error) => setResult(error),
            onDie: (_defect) =>
              setResult("Unexpected defect: " + cause.toJSON()),
            onInterrupt: (fiberId) => setResult("Interrupted: " + fiberId),
            onParallel: (_l, _r) => setResult("Unexpected parallel error"),
            onSequential: (_l, _r) => setResult("Unexpected sequential error"),
            onEmpty: null,
          });

          setEffState("failure");
        },
      });
    });
  }, []);

  if (effState === "processing") {
    return <h1>Processing...</h1>;
  }

  if (effState === "success") {
    return <h1 className="text-green-500">{result}</h1>;
  }

  return <h1 className="text-red-500">{result}</h1>;
};

Let's wrap up the reusable promise and exit code, tidy it up a bit, and then we should have halfway decent API to work with:

type States = "processing" | "success" | "failure";

// name pending... naming is hard!
const useEff = <TSuccess, TError>(
  eff: Effect.Effect<TSuccess, TError, never>
) => {
  const [effState, setEffState] = useState<States>("processing");
  const [result, setResult] = useState<TSuccess | null>(null);
  const [error, setError] = useState<TError | string | null>(null);

  // still going to opt for a useEffect mount hook to _run it once per mount_
  // but we will take care of aborting
  useEffect(() => {
    const controller = new AbortController();

    Effect.runPromiseExit(eff, { signal: controller.signal }).then((exit) => {
      Exit.match(exit, {
        onSuccess: (resolvedValue) => {
          setResult(resolvedValue);
          setEffState("success");
        },
        onFailure: (cause) => {
          let setToFailure = true;
          Cause.match(cause, {
            onFail: (error) => setError(error),
            onDie: (_defect) =>
              setError("Unexpected defect: " + cause.toJSON()),
            onInterrupt: (fiberId) => {
              setToFailure = false;
              console.warn(`Interrupted [${fiberId}] - expecting retry`);
            },
            onParallel: (_l, _r) => setError("Unexpected parallel error"),
            onSequential: (_l, _r) => {
              setToFailure = false;
              console.warn("Sequential failure, expecting retry");
            },
            onEmpty: null,
          });

          if (setToFailure) {
            setEffState("failure");
          }
        },
      });
    });

    return controller.abort.bind(controller);
  }, []);

  return {
    result,
    error,
    state: effState,
  };
};

The react code then becomes quite clean, akin to a react-query or similar feel:

const FailEff = Effect.delay(Effect.fail("I can't even"), "1 seconds");

export const HelloEffAsync = () => {
  const { result, error, state } = useEff(FailEff);

  if (state === "processing") {
    return <h1>Processing...</h1>;
  }

  if (state === "success") {
    return <h1 className="text-green-500">{result}</h1>;
  }

  return <h1 className="text-red-500">{error}</h1>;
};

Try switching between fail and succeed, changing the delay, and whatever else comes to mind. You'll see warnings in the console, mostly because React strict mode will mount, unmount, then mount components again which exercises that abort controller we added.

🕰 Full Example

Let's take that hook and stick it into a folder with whatever of those names you like most (I think I'll go with re-effect for now), and boom we've started a react effect library. Now let's really exercise what we have more with some shenanigans around Dates and Time.

We're going to do something simple for now, just something to make it easier for Hobbits to see when their next meal is and how to plan accordingly. They'll need to know what day it is, the time (in 12 hour format, because 'murica), the season (for what to wear), and their next meal. Let's add some chaos with "business" rules.

We cannot display a time with an even second, only odd. This is because we like jazz too much and cannot live on the down beat, only the off beat (🎺would the off beat be even? or odd? in 'music' beats the first beat is 1 - the down beat, and we emphasize 2 and 4... but in raw timestamps would that be offset? #tangent 🎷). So a timestamp of 00:00:03 is fine (HH:mm:ss format), but 00:00:02 is NOT okay. We must show an error showing the offending, hideous even second.
We cannot display a time in the winter, hobbits are not monsters and cannot go out when it is too cold, naturally. Their hair is a fashion statement, not a coat.
We cannot display a time that is too early, when hobbits should be sleeping (before 5am).

Other than that, nothing else can go wrong. Because Effect! (And because we're ignoring timezones, for now. 🤡)

All of our rules can be determined from a valid Date object in JS, and we can get the current time with new Date() - there are other variants of this with libraries and feel free to install and use them - but we should wrap that with Effect so we can compose our program together, and sync is the perfect tool for that.

// Effect<Date, never, never>
const GetToday = Effect.sync(() => new Date());

I don't think Date can fail like this, but it certainly can fail if we pass something to it that is invalid, so keep that in mind for any future feature requests or explorations you may elect to do on your own (cough user input cough 💡).

With the power of Effect, we can pretend stuff exists that we know we can build later - like functions - and start with the modelling of our expected return type and errors. Let's do that now:

// we want something like this, so we can show everything we need to
// the hobbitses
export interface Today {
  monthName: Month; // union of strings January -> December
  ordinalDate: string;
  year: string;
  /** 24 hour time */
  hourNum: number;
  /** 12 hour time */
  hour: string;
  minute: string;
  second: string;
  meridiem: "pm" | "am";
  season: Season; // union of season names
  nextMeal: Meal; // Enough info for name + time of a hobbit meal
}

// we can use simple strings for errors, feel free to use a class
// or anything you'd like: https://effect.website/docs/guides/error-management/expected-errors
type EvenSecond = "SECOND_EVEN";
type TooCold = "TOO_COLD";
type TooEarly = "TOO_EARLY";

type TodayErrors = EvenSecond | TooCold | TooEarly;

// we need this type
type TodayEff = Effect<Today, TodayErrors, never>;

There's a lot of little formatting bits here and there, but the gist is we have that Effect sync that lifts up a Date into the effect world, we have our interface we want React to consume, and we have our errors - now we just need to compose together a new function with our initial function and we should be golden - for now let's just use FlatMap, which will be able to handle the errors / success duality, and return what we need. Oh yeah, and all that stuff I invented like Season/Meal should be made to be real, else it can't compile. 😘

function processDate(date: Date): TodayEff {
  // use Date functions
  // raise invalid states with Effect.fail
  // map to final format of Today with Effect.succeed
}

const GetToday = TodayEff.pipe(Effect.flatMap(processDate));

I'm going to use tailwind to make my app look not like plain html, otherwise the hobbits will laugh at me, but try your hand at it and compare my solution on github to yours! (apologies for the bright light)

A success state:

A few fail states:

And some extra conditions if the hobbit might be running late:

Some notes:

You can put a button on the page to remount your component which will re-run your effect, just use key and increment a number or something (see my linked source code)
You can use an input and its value existing or not to switch between a TodayEff or an ArbitraryDayEff program, but you'll need to be sure to handle how the Date constructor can fail and create an "Invalid Date" object, yay browsers!

Here is the source code - I will continue to update it as I explore more, the commit where this blog was published was ea158.

What's next?

I don't quite feel satisfied, so I'm going to do two things next:

Interact with an API, coming soon
Investigate a managed runtime to provide context (like whatthis video shows in Remix, code)

Once these two are done, I'll want to try out lots of different kinds of Effects being consumed in React to see where this falls down (as I suspect it will, hence all the work being done in effect-rx). For example, you can declare a Schedule for an Effect, which I think would still work, despite the code we wrote in this post being very "one shot" oriented. And should an async Effect suspend, and how should that work? I'll mostly be looking at react-query and effect-rx / rx-react for inspiration as I move forward.

I'm off work until July this year, so it's a good chunk of time to explore new things! 😁

Don't Fear the Rebase 🐮🔔

Sean Allin Newell — Fri, 29 Mar 2024 11:14:46 +0000

I'm ashamed to report that this title is not unique, but I felt it in my soul, so I needed to keep it. Please forgive me. 😅

If you use git, you likely have experienced the pain of merge conflicts. Due to the decentralised nature of git, merge conflicts are inevitable given that two people work on the same file and area of the codebase.

There are ways to address merge conflicts, prepare for them, or outright avoid them, so let's try to use these tools and strategies to improve our collaboration with others 💪.

One of these tools is git rebase.

I want this blog post to address one primary thing:

🙈 Rebase is Scary

This blog post won't leap frog your git skills to that of the master of ancients, but it will hopefully give you a mental model to use and inject some confidence to start failing with safety and fallbacks, moving from the above to:

💡 I can try Rebasing

Aborting

Many operations in git can be aborted and you will go back to where you were. If you attempt to git merge and something scary happened, just abort! Then you can prepare, backup (pushing to a new branch, for example), and collect yourself. git merge --abort

The Reflog

We're going to look at the reflog first. It is an invaluable tool whenever you have "lost" some file or changes, because, oftentimes, you haven't! As long as you have added and committed something, the reflog will be able to find it, even if you deleted the branch.

reflog of a small project

If you type git reflog in your terminal, (press q to escape! Like git log) you will see the reference log. The docs in the 2nd / 3rd sentence give a good summary of what this is, and why it is so, so, useful:

Reflogs are useful in various Git commands, to specify the old value of a reference. For example, HEAD@{2} means "where HEAD used to be two moves ago"

If you haven't directly seen it yet, HEAD is just a reference to the tip of the branch you are on. You can act on HEAD with commands like git show HEAD which prints the commit and diff of your last commit, and you can git diff HEAD to show what changes on your local filesystem there are, or even git diff HEAD --cached to see what is the diff between the staged files (the ones that are green in git status) and the last commit. You can even list all the commits between two references with git log and the .. syntax with git log trunk..HEAD or git log main..HEAD depending on what you name your main branch. 💆‍♂️

The reference log records all changes to HEAD _ and other references _ . It's basically the audit log of git. And we can access and copy SHAs from the audit log by using git reflog and copying the SHAs we need to restore our HEAD back, potentially to a time before we attempted a merge or rebase - which is a neat party trick if anything. Well, a special, nerdy, git party. 🥳

It's worthwhile to mention that reflog has a lot of information, so sometimes it is easier to reason about a log or a diff command. Even if the log command is a bit more complicated, it reads nicely with that .. syntax we discussed above. That extra bit of information a well specified log or diff command has is just the extra bit you need to realise why something is conflicting:

# Show me a list of commits on a branch
git log main..feat-2

# show me a list of commits to where i am now
git log main..HEAD

# show me a diff between two branches
git diff main feat-1
git diff feat-1 feat-2
git diff main feat-2

Fast Forward Merges

Our overall goal is to be able to use git with more confidence , and hopefully less merge conflicts! So to that end, before rebasing, what if we could just use git merge and/or git pull in a 'safer' way that wouldn't cause merge conflicts?

Enter - Fast Forwarding ⏩

Fast Forward Merges are a merge strategy to constrain what git is allowed to do to your working tree when merging. A "fast forward" means there is a linear path without any branching or conflicts, so the whole lineage of changes can be "fast forwarded" to the new end state. You may have seen this when git is warning you about it's merge strategy, some message like this.

warning: Pulling without specifying how to reconcile divergent branches is discouraged. You can squelch this message by running one of the following commands sometime before your next pull:

git config pull.rebase false # merge (the default strategy)

git config pull.rebase true # rebase

git config pull.ff only # fast-forward only

You can replace "git config" with "git config --global" to set a default preference for all repositories. You can also pass --rebase, --no-rebase, or --ff-only on the command line to override the configured default per invocation.

This is nothing to be afraid of! It actually has both of the main concepts we'll cover in this blog post (fast forward and rebase), but in general, this is just asking you how do you want pull to work? (maybe we should land a PR to simplify that whole message to that one liner question 😜)

The default strategy is pull.rebase false. It may have been the cause of that 11:59pm assignment upload to turn it in .com (am I dating myself?) where you were fighting a merge conflict and just gave up and sent your own local copy of the code instead of what was merged into main. 💀

Whenever you run git pull, the pull.rebase false strategy will merge the changes you have fetched from the origin (ie github) into the changes you've made, it is no different than if the remote branch and your local branch were entirely different branches. It has its uses, and is the default most folks (including myself) are used to. The reasoning being, when executing git pull, I fully expect git to first fetch all the changes, and then attempt to merge them in. It's what it's done since forever. But it is important to recognize that pull.rebase false means git pull does two operations. git fetch then git merge.

If you want to avoid conflicts with git pull, consider turning on the pull.ff only config flag, as that will make pulling disallow conflicts.

git config pull.ff only

If you try to pull something that would cause a conflict, it will now fail, as you have told git to only do fast forward merges. This is the same as if you did a git merge and then a git merge --abort.

Resolving Impossible Fast Forwards

If this happens, what do you do? This is my recommendation:

Branch off of your branch and push to a new remote branch (take note of the commit, or if multiple this branch name)
Check out main
Delete your original branch
Checkout the remote branch (dbl check the SHA on github/gitlab/origin match your local copy)
Use the reflog to cherry pick commit(s) on top of this branch, resolving smaller conflicts one by one. (alternatively, you can manually re apply the changes using your new temporary branch as a reference if it small)

❔ A cherry-pick is a way to apply just a single commit to HEAD - it's kind of like a surgical merge of a single commit to the branch you are currently on. 🔪

For example, the above aborted feat-2 example has one commit that conflicts, but going through the process looks like

# backup our changes before merging
git checkout -b feat-2-san-local-backup
git push -u origin feat-2-san-local-backup:feat-2-san-local-backup

# align our local with what our coworkers pushed to feat-2
# this should get feat-2 to match the remote origin (github/gitlab)
git checkout main
git branch -D feat-2
git pull
git checkout feat-2

# get the commit we need
git reflog

# induce a focused conflict we can fix with cherry-pick
git chery-pick fff265694ace4d75f4c9366df29b4ff648f34405
vim src/pages/index.astro # fix the conflict

Instead of reflog, you could also just show the commits on the branch with a log command: git log main..feat-2-san-local-backup which will have a lot less output and sometimes just has the one commit you need.

Instead of inducing a conflict with cherry-pick , you can elect to open up your change in a UI like github, for example you could have GitHub make a diff between any two branches like this, and then manually do you changes on the shared branch, and then push a fresh commit - avoiding merge conflicts entirely - at the expense of you copying/pasting or redoing a bit of work.

Rebase

Rebasing means to change the base of your branch to some other starting point. Rebasing rewinds all your changes (storing them safely in a stack), then checks out to move to the new base (usually a newer version of main / trunk), and then re-applies your local rewound changes onto the new base.

💡 If you notice, the resolution suggestion I gave above for the fast-forward only strategy does something very similar, if not identical!

This is useful because it can pre-empt any merge conflicts, prevent merge conflicts by preparing a way for fast forward merge, and helps keep main tidy. Pre-empting merge conflicts doesn't mean there won't be any, but if you rebase often, the scope of the merge conflict is contained.

You don't even need a new branch, if you git fetch and your git status is reporting conflicts, you can use the origin/feat-2 style of reference to rebase your local changes on top of the branch from your origin (ie your latest local copy of the state of the branch that github/gitlab etc have). It's a good idea to add the -i flag to make it an interactive rebase if you are using origin/ style references for branches that have more than 1 / 2 commits, for reasons we will dive into in the next section.

This may still incur conflicts, but you will be much more in control of what is happening, and they will happen one at a time, commit by commit, which often means they are easier to reason about, even if it feels like an onslaught of conflict after conflict (and can be a bit repetitive).

Interactive Rebase

Rebase has an option -i to go into interactive mode, which allows you to tell git what to do when applying each of your rewound changes. This is very powerful as it allows you to re-order, re-word the commit body or message, and squash your commits.

When you submit the command, you are presented with a text file to edit - when you save+close this file, git will execute the rebase based on this list of instructions. You are kind of programming git here, in rebase syntax. The commented out section tells you about the commands, and you can edit the first word of each line with a single letter or the full command word. The command letter or word plus the git SHA is all git needs, the title is there for us humans.

Especially if you use the origin/ reference style I strongly recommend you use an interactive rebase. This is because if you share commits you likely want to drop them before inducing needless merge conflicts. Many times the rebase rewinding and finding the common ancestor works flawlessly, and you won't see duplicate commits - but if someone else has force pushed or rebased or the lineage has differentiated, then it's a good idea to at least review what the interactive rebase lists as the commits to apply after switching to the new base, and looking at the new base on your remote server (ie GitHub) to see if anyone has squashed things that you still have expanded.

drop the first commit, the new base already includes this

In the above screenshot, I had checked GitHub and saw that the branch I was rebasing onto squashed its commits after rebasing onto main, I had already squashed my commits onto main in a different way, so I now need to drop my first squash, to not conflict with the origin's squash, and edit my second squash to only include the non-conflicting parts.

To be clear - this kind of situation is likely to be quite rare for the latest versions of git, as most of the time git can tell what has happened, but it can save a lot of pain by doing that extra step when you see a divergence in git status after a fetch or failed pull.

When you tell git rebase to do something like an edit/squash/fixup command, git may find conflicts or may pause to let you do that action. The simplest one is just rewording a commit, which you will be presented with a git commit like screen and move on. But for editting or fixup-ing, you will have to tell git rebase when you are done working on the commit.

You do this by doing whatever the rebase or git status message says, which is often just a git add / git commit / git rebase --continue.

Conclusion

Feel free to drop a question wherever you found this post, happy to discuss and amend! Happy git-ing 💻, here is a quick reference of some of the commands we used:

# the reflog, git's audit trail
git reflog

# log .. syntax with HEAD, and branches
git log main..HEAD
git log main..feat-1
git log feat-1..feat-2
git log origin/feat-1..feat-1

# diffs
git diff feat-1..feat-2
git diff origin/feat-1..HEAD
git diff main..HEAD

# fetch stuff
git fetch

# change strategies
git config pull.ff only
git config pull.rebase false
git config pull.rebase true

# rebasing
git rebase
git rebase -i
git rebase -i origin/feat-1
git rebase -i main
git rebase -i feat-2

git rebase --continue

# may be necessary - push force with lease
git push --force-with-lease

## BONUS shell + git aliases!!

alias gs='git status'
alias gc='git checkout'

# git rebase shell aliases
alias gbase='git rebase'
alias gbae='git rebase'
alias grc='git rebase --continue'

# get current branch
alias gcb='git branch | grep "*" | cut -d" " -f2'

# push to the origin and track a new remote branch
function gpuocb () {
    local branch=$(gcb)
    git push -u origin $branch:$branch
}

# don't miss out on things!
alias gfomo='git fetch origin main:main'

# Bae Git Aliases for typos
git config --global alias.bae rebase
git config --global alias.rebae rebase

GraphQL & TypeScript Magic

Sean Allin Newell — Sat, 16 Mar 2024 16:41:02 +0000

If you haven't heard, there's a new player in town when working with a GraphQL Schema in TypeScript.

🪄 gql.tada (repo)

🌟

"...with gql.tada and GraphQLSP you get on-the-fly, automatically typed GraphQL documents with full editor feedback, auto-completion, and type hints!"

- gql.tada readme

What magic is this? Let's see!

We can use an existing GraphQL API and Schema, so we can focus on just a GQL Client workflow. There's a wonderful list of many GQL apis, so pick your favorite, we're using Trevor Blade's Countries API.

You can find the final code here.

Setup

Create a new astro site, then add some relevant libraries following the gql tada instructions and urql instructions, and our editor and code boilerplate will be ready to go.

mkdir gql-tada-explore
cd gql-tada-explore
pnpm create astro@latest . # choose strictest TS
pnpm astro add react
pnpm add lodash.debounce gql.tada urql @urql/exchange-graphcache
pnpm add -D @0no-co/graphqlsp @types/lodash.debounce

setup instructions

Update your tsconfig with the following plugin configuration:

 {
   "plugins": [
      {
        "name": "@0no-co/graphqlsp",
        "schema": "https://countries.trevorblades.com/graphql",
        "tadaOutputLocation": "./src/graphql-env.d.ts"
      }
    ]
}

tsconfig.json

Create a client.ts file to setup urql (cache is optional, but I'd recommend it)

import { Client, fetchExchange } from 'urql';
import { cacheExchange } from '@urql/exchange-graphcache';

export const client = new Client({
  url: 'https://countries.trevorblades.com',
  exchanges: [cacheExchange({}), fetchExchange],
});

src/client.ts

In src/component folder, create a react component App.tsx. We will be using <App /> as a pure SPA so we can get going quickly. We will pull in the client we declared above to provide our gql client to all useQuery calls.

import { Provider } from 'urql';
import { client } from './client';

export const App = () => {
  return <Provider value={client}>
    <h1>Hello from React</h1>
  </Provider>
}

src/App.tsx

Use our App as a client SPA in src/pages/index.astro as a client only react component.

---
import { App } from "../components/App";
---
<html lang="en">
  <head>
    <meta charset="utf-8" />
      <link rel="icon" type="image/svg+xml" href="/favicon.svg" />
      <meta name="viewport" content="width=device-width" />
      <meta name="generator" content={Astro.generator} />
      <title>Country Search</title>
  </head>
  <body>
    <App client:only="react" />
  </body>
</html>

src/pages/index.astro

Now, to see the magic of our setup, we're going to create a placeholder SearchBar component in src/components and use it in our App.tsx - it won't do anything yet, but that's where we'll start issuing GQL Queries and use the data.

Magic

Before proceeding - try to do the following task on your own

✍️

Load all continents from one query, so we have each Continent's code and name available. There are not that many Continents on the planet so it we can eager load them all and do our searches in memory. Try your hand at implementing a search by name in memory and display matching continents in a list.

Now that we are all set up, we can see the 🪄 magic in action. In our SearchBar, import gql from gql.tada, and start authoring a new query in the tagged template literal and you will now get end to end autocomplete and types. Beautiful.

type safe completions at query author time

Now that we are authoring queries it's time to see how this works, and how far the intelligence goes.

Lints

If we query for continents, including their code and name, but only access the name - we get a TypeScript warning telling us we are, essentially, over fetching!

All The Types

When a project gets going, you often want to create components to display child nodes of a large query, gql.tada gives us some type helpers so we can cut through the GQL and get to the useful TypeScript bits quickly, look at this code example, taken from src/queries.ts in the codebase

import { graphql, type ResultOf } from "gql.tada";

export const getTopLevel = graphql(`
query getTopLevel {
  continents {
    code
    name
  }

  countries {
    code
    name
    continent { code }
    emoji
    capital
    currencies
  }
}`);

type TopLevel = ResultOf<typeof getTopLevel>;

export type Continents = TopLevel["continents"];
export type Countries = TopLevel["countries"];

export type Continent = Continents[0];
export type Country = Countries[0];

extract types

Make an App

Now that we've explored some of the powers and how to make use of gql.tada - the full app in the repo has all the code, but take the time to play with it yourself and fulfill the following requirements:

Search all continents by Code or Name.
Implement tab and enter and click to select a Continent.
Show a Continent Card that displays all Countries on the selected Continent.
When you select a Country, show a Country Card that displays information on the selected Country.
Construct wikipedia links to relevant articles on Countries and Capitals
Construct financial links to relevant currency comparisons.

Some further ideas and explorations you may want to consider:

Find a GQL API you can locally host, to test how gql.tada + urqlg handles mutations as I have largely skipped over that here.
Find a large GQL API and see if the latest optimizations will still retain the snappiness (consider pulling the schema into a local file)

4 Day work week remote opening

Sean Allin Newell — Wed, 20 Jan 2021 05:01:09 +0000

About Us

We're a Scottish tech firm in EdTech. We're breaking into the enterprise market in NA right now and winning bigger clients. We have a four day work week and a strong culture of lived out values.

Our product offerings include a tms, lms, and customized integrations.

About The Opportunity

Administrate is looking to gear up more experienced devs to help mature our platform and take us to the next level. There's never a dull moment! Many systems are in flight to be deprecated, containerized, and migrated to continuous deployment. We have most of our main services containerized and deployed to AWS. We still need a wide range of product work, and are working with enterprise clients in an agile process.

Skill & Qualifications

This is a senior position, so we'll be expecting a broad range of skills in web development; full stack experience, knowledge of modern front end development, domain driven design, aws, docker, python, typescript, react, etc. All helps.

How To Apply

Apply on hire hive and say Sean Newell sent ya 👍.

https://my.hirehive.io/administrate/jobs/67536/senior-software-engineer-remote?source=CareerSite

Please feel free to leave questions in the comments section.

Unity 2020 - pipe dream or reasoned alternative?

Sean Allin Newell — Fri, 21 Aug 2020 00:48:44 +0000

What are your thoughts on #ticket2unity and the unity 2020 idea in general? Would it have any significant impacy on things in the tech world like data privacy, net neutrality, and others?

Overall platform: https://articlesofunity.org/

Dockerizing our F# SQLProvider App

Sean Allin Newell — Fri, 21 Feb 2020 20:37:00 +0000

Photo by Moritz Kindler / Unsplash

This post continues our journey of statically checked SQL schemas in F# land. We're still using SQLProvider, still on .NET Core 3.1, and still living in CLI world with our console app. All the code lives in a public GitLab repo here. This post starts from this commit, and all the work is contained in MR#3. My previous post (F# SQLProvider Pipeline on GitLab Kubernetes Runners) refined our CI pipeline to run on a private k8s cluster instead of GitLab's shared runners which will let us iterate faster in this post.

So, for today, we want to dockerize our application and build out our pipeline to run docker build on every commit. Once the pipeline succeeds, we should merge in and tag the freshly minted code to cut a full release.

Now, because we strictly enforce our schema in a docker container - with our glorious SQLProvider - we should be aware that we may have some bumps in the road. The command we'll be issuing that could cause such bumps is dotnet publish since that runs the compiler and subsequently our SQLProvider. Let's just try a simple Dockerfile and iterate - #agile.

# use 3.1 LTS dotnet SDK image
FROM mcr.microsoft.com/dotnet/core/sdk:3.1-alpine AS compiler
WORKDIR /app

# add everything; .dockerignore will filter files and folders
ADD . .

# run publish!
RUN dotnet publish -f netcoreapp3.1 -o dist

# switch to a runtime image w/out full sdk
FROM mcr.microsoft.com/dotnet/core/runtime:3.1-alpine AS runtime
WORKDIR /app

# copy only the compiled dlls
COPY --from=compiler /dist .

# setup entrypoint to run the app
ENTRYPOINT ["dotnet", "test-sql-provider.dll"]

If you run this, you'll get an error along the lines of Exception while connecting which basically means our dotnet publish command, in our local docker daemon, could not find a postgre db to connect to. Luckily the docker build docs ( docker build --help ) show us a neat little option: --network. We want to use host networking to expose our postgres db or service. So now, locally, we can run docker build --network host -t test . and we'll get a docker image built! Yay!

Now, in order to run our container locally, you'll also need that --network host option: docker run -it --rm --network host your-image-tag. Now your docker container will be able to connect to your local database!

CI - The Real Deal

So it works locally - will it work in GitLab's CI?

Well... almost - in order to get a docker image built in our GitLab CI, we'll need to choose how to do so. In this post I'm opting for the Kaniko route but there is also the Docker-In-Docker route, although that route requires privileged containers. If at all possible, you should avoid using privileged containers.

Now, even after choosing kaniko, going through the docs, and getting valid yaml - the road was uh, a little bumpy:

If you don't at first succeed...

So I'll spare you my pain and jump to explaining Kaniko.

Kaniko is a google project for building OCI images with a plain binary instead of the docker daemon. It's an attempt at being a portable docker build. The reason this is useful is exactly our use case - we have a private CI fleet in k8s and would like to build docker images in a CI job without a privileged container.

That all sounds great - but the one gotcha I ran into is no matter what I did, _ I couldn't run _ .sh _ scripts in a kaniko container _. If you try, you'll likely get something like this:

docker run --entrypoint "" -it --rm gcr.io/kaniko-project/executor:debug sh

$ vi test.sh
$ chmod +x test.sh
$ cat test.sh
#!/bin/sh

echo 'hey'
# ./test.sh
sh: test.sh: not found
$ sh -c test.sh
sh: test.sh: not found

If you know anything about this... drop a comment or @ me on twitter - I would very much like to know why kaniko can't handle shell scripts. I know that the debug tag of the kaniko image has a shell and otherwise it just drops you into the kaniko executor, so it could have something to do with that - but I'm at a loss.

Why would we need to execute a shell script you ask? Well, to seed our database of course. But where there's a will there's a way! Instead of seeding our database in our CI pipeline with a script like we did , we could just create our own image that puts our schema into the containerized postgres db on startup. So that's what we'll do!

FROM postgres:12.1-alpine

RUN echo "CREATE SCHEMA IF NOT EXISTS test AUTHORIZATION test_app;" > /docker-entrypoint-initdb.d/setup.sql
RUN mkdir /sql
ADD schema.sql /sql/schema.sql
RUN cat /sql/schema.sql >> /docker-entrypoint-initdb.d/setup.sql

It's a pretty simple Dockerfile, basically just puts our schema in a special init folder. Postgres has documented environment variables to set up the test_app user and password, so make sure you set those if you try to pull this image down.

Now that we have a 'seeded' db of sorts (not seeded with data of course, just the schema) - we can use it in our pipelines!

Passed Kaniko Pipeline

Kaniko Step

GitOps Release

Eventually, we will want a release pipeline, and versioning, so we can go ahead and get that set up with tag builds for both the db image and our main app image. I also added some sweet tests so the build would be doing a little bit more than just compilation. I decided to squash the dotnet build and dotnet test steps into one build to shave some time. In this case, I think it makes sense to have them as steps in the same job as opposed to different jobs, but your case may vary.

Tagged release of App's docker image
Tagged release of db docker image
We also have pre-releases, tagged with the commit SHAs

This was far trickier than I thought at first, and if you look at the whole pipeline history on my GitLab repo and my commit history, you'll see the experiments I was playing with (multi-lines, buildah, getting kaniko to work right, services, seeding the db...). But I sure did learn a lot! Before this adventure I didn't know about the network flag, kaniko, buildah, or how an F# SQLProvider project could be dockerized.

Conclusion

I think what I have now is quite stable. I feel comfortable with the tests I banged out in this merge request, and now that we are dockerized we can deploy our cat db app anywhere. Also, this MR landed a DB_CONN_STR environment variable, so as long as we add that --network host to docker run, we can pass this env var with the -e flag to connect our image to any database with the schema it was compiled against - and if we wanted to just pull that down too we have an image with the schema pre-baked. Nice!

In the next few posts we can now explore:

Adding an API layer on top of our core logic
Adding a front end maybe?
Building out the pipeline to put the pre-release docker images through the paces (actually run integration tests on the docker image)
Switch from rebuild to retag on a release build somehow
Add more advanced db features to showcase the power of F#'s SQLProvider
Add more useful tests (FsCheck tests are already very powerful for our helper layer)
Migrate this kind of strategy to a 'real' project (like my trakr project)

See y'all soon!

F# SQLProvider in GitLab Kubernetes Runners

Sean Allin Newell — Tue, 11 Feb 2020 05:08:42 +0000

Photo by Fran . on Unsplash.

My previous post dove into using GitLab's Shared Runners to verify, in an F# SQLProvider data layer, that the code and schema agree. In this post we'll explore getting off the shared runner infrastructure, and using our own Kubernetes cluster for private CI. I happen to have a Digital Ocean Kubernetes cluster (DOK) - so that's what where my k8s cluster will live, however I'll be sticking with normal helm and kubectl commands in this post, so hopefully it'll apply to any k8s installation.

In this post, I'll be mostly focused on the moving from shared runners to dedicated private CI in our own k8s cluster for our SQLProvider F# project - but that doesn't mean you can't use this post to guide your own CI/CD infrastructure setup for your own project, so cheers 🍻!

Getting a Cluster

First thing's first, get a k8s cluster up and running. Using a cloud provider makes this easy-peasy, but there are tons of way to stand up a cluster using various k8s distributions. The key is to make sure you have an up to date kubernetes version, I'll be using v1.16.2, but anything that supports helm v3 and GitLab's Runners should work.

In order to verify your cluster is up, try to connect to with some standard tooling like helm and kubectl - if you don't have those tools follow the links and get installing ⏬!

$ kubec version
$ helm version

Connecting the Cluster

Your k8s provider, or your custom setup, will have instructions on how to connect kubectl and helm on your local. For Digital Ocean, setting up an API key in your Digitial Ocean account and using the doctl CLI sets up OAuth tokens for kubectl to authenticate with the k8s cluster. Once you have that in place, you can run a quick kubectl cluster-info and kubectl get all --all-namespaces to see if your local CLI tooling can connect to the cluster.

Once that's settled, we can install our target workload on the cluster - GitLab Runners that support _ services _. That'll be the key for us, since our SQLProvider based CI setup requires a postgres service.

Runner Setup

We'll be following these docs for the k8s runner setup insetad of the integrated setup. The integrated setup uses helm v2, which needs to install tiller on the cluster which I (and others) don't recommend due to its security implications. We can still use helm charts, but we have helm v3 because we live in the future 🔮.

First we'll need to add GitLab's helm repo:

$ helm repo add gitlab https://charts.gitlab.io

Now we can mess with some yaml, the docs give us the basic picture, but I had to play with it a bit to get the services part working correctly. Here are the fruits of my labor with explainers in comments

values.yaml

gitlabUrl: https://gitlab.com/

# Can set with -set runnerRegistrationToken=$CI_TOKEN
# runnerRegistrationToken: ""

# how many jobs should be able to be ran concurrently
concurrent: 3

# how often we check, in seconds, for work
checkInterval: 5

# Role Based Access Control - #veryMuchGoodSecure ?
rbac:
  create: true

runners:
  # default image for jobs
  image: alpine:3.11.3
  locked: false

# This set an env var that presumably allows the postgres service to be used
envVars:
  - name: DOCKER_ALLOWED_SERVICES
    value: "postgres:12-alpine"

Now that we have this yaml in place we can use helm to deploy the runner

# pull this from settings -> CI -> runner token
export CI_TOKEN=runner-registration-token-from-gitlab

# install!
helm install -n YOUR_NAMESPACE \
  YOUR_RELEASE_NAME \
  -f values.yaml \
  gitlab/gitlab-runner \
    --set runnerRegistrationToken=$CI_TOKEN

Note!

I found this stellar SO answer that informs us about the differences between docker networking and kubernetes networking as it pertains to the executors in GitLab's runners. If you, for example, had a postgres service accessed via the postgres hostname in a Shared Runner context (docker executor), then you would need to switch to using 127.0.0.1:5432. That change will handle the could not translate host name. This commit is an example refactor of scripts+code that would be necessary to move from shared runners to kubernetes runners.

Done!

Now that you have the runners deployed, you can disable shared runners for your GitLab project and push commits and get blazing fast, private CI for just your projects 🎉! This is an example merge request for our ongoing fsharp project that shows all the work we did in this post.

Now that we have our own private CI, it's time to think about building out a web API layer for our cat app, so we can have something more robust than a CLI app to deploy to. Before that, we'll need dockerize too. Onwards and upwards!

Type Safe DB Access checked on every Commit

Sean Allin Newell — Sun, 09 Feb 2020 04:20:03 +0000

Photo by Ryan Quintal on Unsplash.

The previous post had a local setup for type safe SQL access with F#'s Type Provider - SQLProvider. In this post we're going to move into CI territory from this commit in our repo; enforcing that every commit has to be working with the DB Schema in a valid way. In order to do that - in general for any CI provider - we're going to have to duplicate the steps we did locally on our machines

We'll track our work in a branch feat/ci-capabilities and land it in a Merge Request.

General CI

So, in general, in order to compile our F# project with SQLProvider, we need:

A running database
- With a schema loaded
A valid connection string to the running database
Nuget's restore to pull down the right dlls
- We have a pre build script that can move those dlls into a spot for the F# compiler.

Depending on your CI provider, there are a few different ways to approach getting a db, loaded with a schema, into your pipeline.

Have a 'stable' external environment accessible to your builds
Use a feature from your provider (ex GitLab services)
Leverage docker/custom-images/nested-docker/docker compose to set things up and orchestrate (containerization solves everything right?)

In this post we'll explore GitLab's postgres service solution on shared runners.

GitLab Shared Runners

Note - We'll explore how to leverage a personal k8s cluster of runners in the next post.

GitLab offers free CI/CD for all of their repositories on their shared runner infrastructure. They have a feature called services (linked above as option 2) that we can leverage to have a postgres db loaded for a job.

One of the annoying things about CI/CD (this goes for any CI provider) is trying to test out your pipeline/job before just pushing and seeing if it works. That can be... a not-so-great experience. It's one thing to test a shell script locally, but it's another to:

Have valid yaml
Have a valid sequence of steps
Use a feature (aka service) correctly

And if you try to do anything complicated that propogate artifacts or has dependent jobs, it becomes a lot harder to test the pipeline before git push and pray.

Luckily, GitLab has a nice way to install a tool to help you test locally before pushing. They have a nice set of instructions on their docs, and on my debian buster system it was fairly straight forward:

curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh \
  | sudo bash \
  && sudo apt-get install gitlab-runner \
  && which gitlab-runner

In order to see if this works, let's just do the simplest build step:

build:
  script:
    - echo "Hello World"

Now we can run this locally with the shell executor:

▶ gitlab-runner exec shell build
Runtime platform arch=amd64 os=linux pid=6518 revision=003fe500 version=12.7.1
Running with gitlab-runner 12.7.1 (003fe500)
Using Shell executor...
executor not supported job=1 project=0 referee=metrics
Running on flantop...
Fetching changes...
Initialized empty Git repository in /home/sean/repos/fsharp-sql-provider/builds/0/project-0/.git/
Created fresh repository.
From /home/sean/repos/fsharp-sql-provider
 * [new branch] master -> origin/master
Checking out 5829e774 as master...
Skipping Git submodules setup
$ echo "Hello World"
Hello World
Job succeeded

OH WOWEE.

If you add an image to the yaml, you can use the docker executor (assuming you have docker setup locally).

  build:
+ image: alpine:3.11.3
  script:
    - echo "Hello World!"

Now you can use the docker executor instead of the shell executor, which will be 'more real' since we'll need to have the dotnet core sdk available for our F# builds: $ gitlab-runner exec docker build.

OH NICE!

Okay, that's all fine and dandy, but we're business-business and need to build enterprising enterprise apps (which of course means we need the dotnet core sdk).

  build:
- image: alpine:3.11.3
+ image: mcr.microsoft.com/dotnet/core/sdk:3.1-alpine
  script:
    - dotnet build

Unfortunately that won't work. We need a database too - so let's set that up with some services! But wait... we need to load our schema into this database, and our current run-postgres-container.sh script assumes we're using docker to exec psql - when we really should be executing psql in our job, connecting to the service that gitlab will set up for us. We'll have to write a custom script the ci will run, since the service will setup the user for us, but not the schema. Let's call that ci-pg-setup.sh and it'll look like:

#!/bin/sh

echo "CREATE SCHEMA IF NOT EXISTS test AUTHORIZATION test_app;" \
  | PGPASSWORD=pass psql -h postgres -U test_app -d postgres -f -

cat schema.sql | PGPASSWORD=pass psql -h postgres -U test_app -d postgres -f -

We'll also need to make a change to our prebuild.sh to make it POSIX compliant, using sh instead of bash - see this commit for deets.

Now before we update the CI yaml we'll need to update our code to conditionally use the postgres host (due to docker networking in the docker executor). We can do that by conditionally creating a debug symbol in our fsproj file based on the presence and value of an environment variable, and using that symbol to switch our compile time constant ConnString in DbAccess.fs. That's done in this commit. It's important to note that we're depending on the psql tool in our ci setup script, so we need to make sure we add that (apk add postgres-client) in our job.

Our full yaml is setup in one commit here - copied here for your full pleasure:

build:
  image: mcr.microsoft.com/dotnet/core/sdk:3.1-alpine
  services:
    - postgres:12.1-alpine
  variables:
    POSTGRES_DB: postgres
    POSTGRES_USER: test_app
    POSTGRES_PASSWORD: pass
  script:
    - apk add postgresql-client
    - ./ci-pg-setup.sh
    - CI=TRUE dotnet build

And voila! We have a passing job that checked our data layer!

Break it

Let's break our build!

One way to break our build would be to write the code without updating our schema script (maybe some dev manually changes their local schema without committing the schema changes) - or to change the schema in a way that causes F# to blow up. Let's do the latter.

schema.sql

@@ -20,6 +20,7 @@ CREATE TABLE IF NOT EXISTS test.breed_attributes(
 CREATE TABLE IF NOT EXISTS test.cats(
   id SERIAL PRIMARY KEY,
   name TEXT NOT NULL DEFAULT '',
+ age INTEGER NOT NULL DEFAULT 1,
   breedId INTEGER NOT NULL DEFAULT 1,
   FOREIGN KEY (breedId) REFERENCES test.breeds
 );

With this we should expect an error like this (emphasis added by mua):

 error FS0039:
   The field, constructor or member 'Create(breedid, name)' is not defined.
   Maybe you want one of the following:
+ Create(age, breedid, name)

And the proof is in the pudding! A failed build in our pipeline!

Let's fix it. Now that Cat's require an age as per our schema, we should update our data layer, and our domain type too. Those changes will cascade down to the loader and cli programs, but after that the compiler has done all the hunting for us, and if it compiles, the program will run soundly.

Merging it in

There are a few tweaks I've made before merging:

Bundle psql into the image we're using
Fix typos here and there
Use the module alias like we did in Loader in Cli

And that's a wrap!

Future Work

There's still a lot to do to get to a a point where this is workable for a modern CI/CD web app, in the future I'll put up posts to:

Instead of shared runners, use a dedicated k8s cluster
Dockerize the F# application
Introduce a web layer for API
Create a deployment
Automate the deployment

Stay tuned!

F# SQL Provider in .NET Core 3.1

Sean Allin Newell — Sat, 08 Feb 2020 04:46:44 +0000

Photo by Ryan Quintal on Unsplash.

Preface

Type providers have long fascinated me as a killer feature of F#, but I haven't gone whole hog since I was more focused on wanting to use .NET Core F# since .NET Core 2.1 LTS was released. I didn't want to have to finangle with mono or framework on either my work windows 10 box or my linux laptop at home. Luckily, the SQL Provider type provider that lets F# programmers write F# flavored LINQ with strong typings right from the database does work! It works with MySQL and Postgres, on .NET Core 3.1, today! (and probably yesterday and tomorrorw as well).

Setting up

You'll need to install a few things:

.NET Core 3.1 SDK
Local Db server (or use Docker!)
- Ex: Postgres in a container
Editor with F# intellisense
- Ex: VSCode + Ionide
Ability to run scripts
- Ex: .sh or .ps1
Terminal

I'm just going to use VSCode here, but in theory this works in Visual Studio as well - I've tested this on a windows 10 box connected to a remote MySQL server, and on a debian laptop connected to a local postgres server running in a docker container. We'll just be making a small console app, but this works for apps written with Suave, Giraffe, Saturn, and in any netstandard project (although for simplicity sake we'll be making netcore3.1 projects).

If you're unfamilar with all the hooplah microsoft is throwing our way with netframework, netstandard, netcore, and the lts releases and f# versions I'd encourage you to peruse this blog post and these docs. Something that confused me recently was the shift from nuget metapackages to bundled packages shipped in the runtime. The gist is:

.NET 5 will unify everything into .NET Core
Netstandard will likely not apply when everything is net5
.NET 3 changed how nuget metapackages are handled, and bundles a lot more dlls into the runtime

These changes are pertinent to F# Type Providers since (especially the SQL Provider) the libraries we use often need to be pointed to a place to pull in compile-time dependency libraries to connect to databases.

Setup

Setup is pretty easy once you have an editor and the dotnet sdk since it comes with a nice CLI tool.

mkdir test-sql-provider
dotnet net console -lang f#

# make sure this runs
dotnet run

# and if you have VSCode + Ionide go ahead and fire it up!
code .

Now you have a nice starting point.

Note - on linux you may want to ensure the .fs and .fsproj file are in plain utf-8 (without bom) and with lf line endings. ;)

Type Provider Setup

In order to work with the SQL Provider, we'll have to create a stable location for the compiler to look for libraries as it is compiling to use in the type provider (ie make connections to the database to generate the types from our sql database).

I like having conventions enforced by tools, and what better tool than the dotnet build tooling! You can create a 'hook' of sorts that runs an arbitrary command - and that command can populate a well known place (./lib/) with the appropriate dlls required by the F# type provider so the compiler can consume the types it creates. Nice!

Let's first create an executable script that just prints something out and wire up the 'hook' in the .fsproj file. touch prebuild.sh && chmod +x prebuild.sh to get the file and put an echo in there:

#!/bin/sh

echo "Ahoy!"

This should work in windows if windows understands how to run .sh files - but your mileage may vary, so feel free to swap out the shell file with powershell or even use a node script if you want to.

Now open the .fsproj file in your editor and add the following.

   </PropertyGroup>

+ <Target Name="TypeProviderSetup" **BeforeTargets="Build"** >
+ <Exec Command="./prebuild.sh"/>
+ </Target>
+
   <ItemGroup>

To see if it works run a quick dotnet build to see your echo!

▶ dotnet build
Microsoft (R) Build Engine version 16.4.0+e901037fe for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Restore completed in 42.01 ms for /home/sean/repos/test-sql-provider/test-sql-provider.fsproj.
  test-sql-provider -> /home/sean/repos/test-sql-provider/bin/Debug/netcoreapp3.1/test-sql-provider.dll
  Ahoy!

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:03.53

Now it's time to add packages and figure out how to get the necessary dlls from nuget into the lib folder. I'll just use the dotnet CLI to add the packages for Postgres and SQLProvider:

# gotta do em one at a time
dotnet add package SQLProvider
dotnet add packages Npgsql

Here is the nuget page for the postgres Npgsql package. We are particularly interested in the 3.0 dependencies this package requires:

So this tells us that for apps that target netcoreapp3.x we need System.Runtime.CompilerServices.Unsafe (>= 4.6.0). Since we added the package to our project with dotnet add package (which uses nuget under the hood) - nuget added the dependnecies to our machine's global nuget cache already. So we actually already have the dll on our machine, and our script can locate it and copy it for us.

In order to do this in a somewhat robust way, we can leverage nuget to figure out where it stores packages, but then we'll have to figure out the directory structure once we're in the package.

To get nuget to tell us where it puts things, run dotnet nuget locals global-packages -l. To figure out where System.Runtime.CompilerServices.Unsafe lives you can explore the directory structure to reveal this path:

▶ ls ~/.nuget/packages/system.runtime.compilerservices.unsafe/4.6.0/lib/netstandard2.0/
System.Runtime.CompilerServices.Unsafe.dll System.Runtime.CompilerServices.Unsafe.xml

Now that we have these two pieces we can finish our bash script (we actually need to use bash if we want to use arrays - you could read from a txt file that lists the same info out to loop over by reading line by line):

-#!/bin/sh
+#!/bin/bash

-echo "Ahoy!"
+# Run a restore to be sure we have all the dlls on the system.
+dotnet restore web > /dev/null
+
+# Find nuget's global directory
+nugetRootPath=$(dotnet nuget locals global-packages -l | cut -d' ' -f4)
+nugetRootPath="${nugetRootPath%?}"
+
+# array of ${nameOfDep}|${pathToDllFolder}
+packagesAndPaths=("System.Runtime.CompilerServices.Unsafe|4.6.0/lib/netstandard2.0")
+
+mkdir -p lib
+
+for i in "${packagesAndPaths[@]}"
+do
+ # all these vars split out for clarity
+ name=$(echo "$i" | cut -d'|' -f1)
+ path=$(echo "$i" | cut -d'|' -f2)
+ nugetDll="$name.dll"
+ pathFromNugetRoot="$(echo "$name" | awk '{print tolower($0)}')/$path"
+ fullPath="$nugetRootPath/$pathFromNugetRoot/$nugetDll"
+
+ cp "$fullPath" "./lib/$nugetDll"
+ echo " > Copied $nugetDll into lib for type provider."
+done

Run this manually to verify that it populates the lib folder, than rm -rf lib and run dotnet build to see...

▶ dotnet build
Microsoft (R) Build Engine version 16.4.0+e901037fe for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Restore completed in 32.34 ms for /home/sean/repos/test-sql-provider/test-sql-provider.fsproj.
  test-sql-provider -> /home/sean/repos/test-sql-provider/bin/Debug/netcoreapp3.1/test-sql-provider.dll
    > Copied System.Runtime.CompilerServices.Unsafe.dll into lib for type provider.

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:02.03

SQL Time

Let's write a fun SQL Schema! (Did'ya ever think SQL Schemas could be fun? 😹)

-- pre-sql - have these run as the postgres user
CREATE USER test_app WITH PASSWORD 'pass';
CREATE SCHEMA IF NOT EXISTS test AUTHORIZATION test_app;

-- schema sql - run these as the test_app user
CREATE TABLE IF NOT EXISTS test.attributes(
  id SERIAL PRIMARY KEY,
  name TEXT UNIQUE,
  description TEXT
);

CREATE TABLE IF NOT EXISTS test.breeds(
  id SERIAL PRIMARY KEY,
  name TEXT UNIQUE
);

CREATE TABLE IF NOT EXISTS test.breed_attributes(
    attributeId INTEGER,
    breedId INTEGER,
    FOREIGN KEY (attributeId) REFERENCES test.attributes,
    FOREIGN KEY (breedId) REFERENCES test.breeds
);

CREATE TABLE IF NOT EXISTS test.cats(
  id SERIAL PRIMARY KEY,
  name TEXT,
  breedId INTEGER,
  FOREIGN KEY (breedId) REFERENCES test.breeds
);

CREATE TABLE IF NOT EXISTS test.owners(
    id SERIAL PRIMARY KEY,
    name TEXT,
    age INTEGER
);

CREATE TABLE IF NOT EXISTS test.owner_cats(
    ownerId INTEGER,
    catId INTEGER,
    FOREIGN KEY (ownerId) REFERENCES test.owners,
    FOREIGN KEY (catId) REFERENCES test.cats
);

Maybe I went too far... Let's load this up in a docker container with a new volume and then execute the script with psql - all wrapped up nice and neat in a script.

./run-postgres-container.sh

Now normally at this point you'd copy some huge SQL file to load data... but let's use the SQL Provider to do that! Who wants to write or copy inserts anyway.

F# |> SQL

Following the getting started and the postgres instructions for the Type Provider should yield boilerplate like this:

open FSharp.Data.Sql

[<Literal>]
let DbVendor = Common.DatabaseProviderTypes.POSTGRESQL

[<Literal>]
let ConnString = "Host=localhost;Port=5432;Database=postgres;Username=test_app;Password=pass"

[<Literal>]
let Schema = "test"

[<Literal>]
let ResPath = __SOURCE_DIRECTORY__ + @"./lib"

[<Literal>]
let IndivAmount = 1000

[<Literal>]
let UseOptTypes = true

type DB =
    SqlDataProvider<DatabaseVendor=DbVendor, ConnectionString=ConnString, ResolutionPath=ResPath, IndividualsAmount=IndivAmount, UseOptionTypes=UseOptTypes, Owner=Schema>

let ctx =
    DB.GetDataContext(selectOperations = SelectOperations.DatabaseSide)

let catDb = ctx.Test

In VSCode+Ionide that looks like:

Now let's load some cats in our database - with only F# code and the schema loaded!

First let's write a sort of 'domain layer' - this is just normal F# modelling with records and such:

// Domain modelling - normal F#!

/// A type of cat (ignoring attributes for now)
type Breed = {
    id : int option
    name : string
}

/// KITTEH
type Cat = {
    id : int option
    name : string
    breed : Breed
}

/// Make a cat with just a name and breed name, with None for ids
let mkCat catName breedName =
    { id = None; name = catName; breed = { id = None; name = breedName } }

/// Turn a cat to a string - my how the turn tables!
let catToStr cat =
    sprintf "%s(%d)" cat.name (Option.defaultValue -1 cat.id)

Second, let's write some db F# code as 'our db layer' using the SQLProvider boilerplate we wrote above:

// plain tables - query on these or get other functions
let Attributes = catDb.Attributes
let Cats = catDb.Cats
let Breeds = catDb.Breeds
let Owners = catDb.Owners

// join tables
let BreedAttrs = catDb.BreedAttributes
let OwnerCats = catDb.OwnerCats

// creating entity functions
let createBreed = Breeds.``Create(name)``
let createAttribute = Attributes.``Create(description, name)``
let createCat = Cats.``Create(breedid, name)``
let createOwner = Owners.``Create(age, name)``
let createBreedAttr = BreedAttrs.``Create(attributeid, breedid)``
let createOwnerCat = Owners.``Create(age, name)``

let findBreedIdByName bn =
    query {
        for b in Breeds do
        where (b.Name = bn)
        take 1
        select b.Id
    }

// DB functions
let insertBreed breedName () =
    let breedEntity = createBreed (breedName)
    ctx.SubmitUpdates()
    breedEntity.Id

let insertCat cat =
    let breedId =
        findBreedIdByName cat.breed.name
        |> Seq.tryHead
        |> Option.defaultWith (insertBreed cat.breed.name)

    let catEntity = createCat (breedId, cat.name)
    ctx.SubmitUpdates()

    { cat with id = Some catEntity.Id; breed = { cat.breed with id = Some breedId } }

So far so good, all the types line up - let's compose this into our main program with some helpers along the way to make it super 'fsharpy' and nice:

// Helpers

/// Take a function, and a tuple, and call the function with the destructured tuple
let withTuple f (a, b) =
    f a b

/// Take a tuple and make a cat in the db, then turn it into a string
let tupleToCatStr =
    withTuple mkCat
    >> insertCat
    >> catToStr

/// Take anything, and give back a zero - a success exit code
let returnZero _ = 0

let rawCatData =
    [
        ("Denton", "Gray Tabby")
        ("Mitzie", "Tuxedo")
        ("Saphire", "Blue Russian")
        ("Ailee", "Siamese")
        ("Oreo", "Tuxedo")
        ("Frisky", "Tuxedo")
    ]

[<EntryPoint>]
let main argv =
    rawCatData
    |> List.map tupleToCatStr
    |> String.concat ", "
    |> printfn "Made a bunch o cats!\n ***\n%s\n*** \n"
    |> returnZero

Now that we have it all setup, let's run it and see our database in action!

dotnet run && docker exec cat-postgres-db psql -U test_app -d postgres -c "SELECT c.name, b.name FROM test.cats c JOIN test.breeds b ON b.id = c.breedId"

Refactorings

That was a little messy, we can probably extract a lot of reuse from the functions we wrote so create a full fledged CLI cat catalog, so let's refactor before we really crank out some code.

Try your hand at refactoring from this commit if you want to try your refactor F# chops!

Here ate the modules I split out (these are in compile order):

The best name in all the land ;P

Helpers.fs

/// Abstract functional heplers that don't go anywhere particularly specific
module Helpers

Domain.fs

/// Domain modelling - normal F#!
module Domain

DbAcces.fs

/// Domain modelling - normal F#!
module DbAccess

Loader.fs

/// Load data into the db the easy way!
module Loader

Program.fs

/// It just calls whatever we're doin (in this case the Loader program)
module Program

All wrapped up in a beautiful commit.

Profit

Let's make a Cat Manager CLI app with the easiest data layer evaaar!

We'll need to make a new Cli module for all the... well the cli stuff! There'll be a loop in there and reading/parsing inputs, and then we can write any new db functions in our data layer and do any new modelling we need in the domain layer. If anything gets a little out of hand we can throw something into the helpers to help us out. We won't add any new functionality to the loader for this (in fact we will be removing the call to the Loader module, so it'll be dead weight for now).

The main CLI loop is as simple as:

let start() =
    let latestCatNumber = countNumOfCats()
    printfn "Cat Manager CLI 😺: Currently tracking %d cats" latestCatNumber

    let mutable keepGoing = true
    while keepGoing do
        keepGoing <-
            ()
            |> prompt
            |> getInput
            |> handleInput

    printfn "You're purrfect! 😽"
    0

Then you expand on handleInput:

/// Handles input - matches on lowered string and returns to keep going or not
let private handleInput (str: string) =
    let lowered = str.ToLowerInvariant()
    match lowered with
    | "?"
    | "help" ->
        printfn "Available Commands are:\n\t%s" commandList
        true
    | "c"
    | "count" ->
        let latestCatNumber = countNumOfCats()
        printfn "Currently tracking %d cats" latestCatNumber
        true
    | "a"
    | "add" -> handleAdd()
    | "q"
    | "quit" -> false
    | _ -> true

Here's the full commit.

Here's a screen shot of the program in action:

For the next part, we need to set up GitLab CI, I'll show you how to do it with GitLab's shared runners and your own runners (on say some kubernetes cluster) - there's some setup you'll have to do to ensure that your F# programs have access to a db with at least the schema preloaded even in the CI environment. We'll then talk about runtime configuration for production environments (often with secret passwords specified with environment variables). And after that, we'll be ready to create a full on web api project in F# using a modern CI/CD pipeline, and even do some clean architecture! Stay tuned!

Why I Program

Sean Allin Newell — Fri, 22 Dec 2017 21:47:27 +0000

Or, maybe more accurately, why I am a programmer.

People program for many reasons. If your reason differs from my or others' reasons, that's totally cool! We need diversity in this industry, and having differing reasons will broaden our holistic approach to not only problem solving, but to the underlying philosophies that our tools, frameworks, languages, and companies are built on. I'm hopeful that this diversity will grow as our industry continues to mature.

So, why does Sean Newell program?

My Reasons

To Learn
To solve problems
To help people
To excercise rigorous creativity
To explore

Boom. Blog post over.

Let's start a discussion on each of these reasons since discourse is how we refine one another.

To Learn

I am a voracious learner. A livid learner. (Insert your profound two-word phrase here). In fact, it gets me into trouble sometimes. Case in point, refer to the previous blog post - I probably was minutes away from getting my nginx config to work, but then I learned about Caddy, then traefik... When does it stop? When do I stop hopping the tool or framework train and ship? Here's the deal though.

When I have to ship, I don't stop learning.

Shipping doesn't mark an end to the cycle of exploration and discovery, but rather shipping itself provides a rich set of instructive experiences that can direct my learning further.

"Experience is the best teacher."

-Cicero, Oration for Rabirius

This cycle is inherent in programming. And not just because of tooling / framework churn, but also because of the nature of our craft. In the book "The Mythical Man Month", the author characterizes programming as dealing with "pure thought stuff". This means we can learn as we work. Not just through mastery of our skill, but by thinking and iterating on our thoughts. Composing new ideas, and seeing how functions combine in new and interesting ways.

That's exciting! And it drives me deeper into nooks and crannies of programming to shed light on the parts that I don't quite understand fully yet.

To Solve Problems

At my core being I'm probably three things. A sleepy man, a learning man, and a problem solver. It's my default when I'm presented with emotional, relational, or analytical problems. It's just what I do.

And programming, specifically being a programmer, demands that I have my problem solver cap on all day.

Sometimes the problems are rote, have been solved before, or have already been solved and I simply need to implement the solution. Other times, there are interesting twists, constraints, or other circumstances that drives me to think a little deeper, or ask more questions to get to the heart of the problem.

Either way, a problem is solved, and my brain has to flex. Whether that's categorizing the problem into an existing box, or transforming and composing a problem domain into a workable solution.

This is where my exploration of functional programming has paid large dividends, because the 'known problem space' now, for me, keeps expanding as I learn about problems other people have been solving for the last ~50 years in category theory, general maths, and other programming communities.

To Help People

Helping people is a core component of relating to others in a healthy and productive way, and teaching is often the best teacher, as it forces one to refine ideas and find concise language to deliver an idea well. So being a programmer allows me to flex these relational area well, whether that's teaching or exploring with a co-worker or delivering a feature or product to a customer.

It's important to note that this is what drives UX and other user-centric methodologies, that's what makes those careers so rewarding. It's a great feeling to know you made a site more accessible, reliable, or usable for real people.

To Exercise Rigorous Creativity

Programming, math, and engineering are a special kind of art. These disciplines all deal with certain laws, invariants, or conditions and constraints, but require some desired output. It is then our job to massage the inputs and get the desired output, or prove some desired property (like "this bridge will hold X cars").

We have to apply a rigor and care to the laws and maths that define our environment, but we have freedom in how we mold and use those laws to bend to our will. It takes practice, discipline, and a certain bull-headed-ness that I find in many programmers. We just don't give up. We press into the discomfort of a tangled problem.

Then magic happens, a la a creative leap or insight, or a plain educated (or not) guess. Then an idea becomes a reality.

That's beautiful!

To Explore

This may seem redundant - and if so then all I have to say is that repitition is the friend of the adult learner. But really, I think exploration warrants a different discussion.

You see, it's expensive to experiment in most industries. That's why there are RnD departments. In software, going back to that pure thought stuff idea, we can iterate and experiment as cheaply as we can type git checkout -b new-experiment.

Or rather, it can be cheap to experiment in our craft. As long as we are responsible and aware of how our experiment affects the project and team. Often an experiment can improve an existing tool chain or arcane build script, or provide a new, useful abstraction around some mundane work.

Fin

So hopefully some of that resonates with you - feel free to drop a comment via disqus down below and tell me some of your perspectives.