Forem: Ryland G

shrun: A modern CLI testing framework

Ryland G — Mon, 13 Apr 2020 18:27:35 +0000

TL;DR: Test your CLI commands in isolated docker containers using the Jest test environment you already love.

A few years ago, I was working as lead architect for a startup building a high-performance competitor to AWS Lambda. One of my responsibilities was maintaining a standalone CLI and SDK for the functions backend. The CLI/SDK was built with Node and commander (eventually yargs), and had very similar structure/usage to the popular Serverless framework.

A while after I built out this initial SDK/CLI we started having some internal frustrations regarding the process user-facing features would go through before eventually reaching the CLI. We realized that we would often design a backend feature only to later realize that the CLI interface/API would need to be quite nasty to satisfy it. This frustration had a measurably negative effect on both the quality of new features and the velocity in which they could be released. Many readers might assume that we simply had bad communication and planning, and while there was definitely room for improvement in that area, it didn't help that our team was separated by a 10-11 hour time difference. Regardless of the cause, at some point one of my coworkers started a conversation with me to explore ways we could make our process more declarative and reliable. After a especially frustrating day, he came to me with an amazing idea. He suggested that we create a "spec" format that would allow us to both test the CLI and propose new user-facing features in a concrete way. I perfectly understood the direction he was going, so I immediately started building a prototype. A day later I had a MVP version of the tool which consumed yaml based spec tests and ran them automatically against our open source CLI. Below is an example to show you the format of the spec (testing the npm init --help command):

- test: Test init help output
  setup:
    - "curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -"
    - "sudo apt install nodejs"
  steps:
    -   in: npm init --help
        out: |-
          npm init [--force|-f|--yes|-y|--scope]
          npm init <@scope> (same as `npx <@scope>/create`)
          npm init [<@scope>/]<name> (same as `npx [<@scope>/]create-<name>`)
          aliases: create, innit

Spec format

test: string - each spec test must have a test stanza with a unique name. For those who are familiar with Jest/Ava/Mocha, this maps directly to the test("someName", () => {}) format used by those frameworks.

setup?: string[] - the setup section allows you to run a series of shell commands before the test itself runs. This is convenient for tests that rely on a specific set of environment variables, need iptables configured etc. For those who are familiar with Jest/Ava/Mocha, this partially maps to the beforeEach (more like beforeThis since you specify it per test) construct.

steps: Step[] - steps are where the bulk of your test logic is defined and there is no limit to the number you can have per test. All steps must have an in entry, this is what will actually be run against the containers internal shell. If a step is expected to succeed, it is a PassStep and must have an out entry. in and out map to actual and expected in traditional testing frameworks. If a test is not expected to succeed (not 0 exit code), it must either have an err or exit entry. err is similar to out but is checked against stderr as opposed to stdout. exit makes it possible to specify the expected exit code that resulted from running the tests in statement.

There are also two other stanzas not show by the above spec:

cleanup?: string[] - the exact same as setup but runs after the test has finished. Useful for resource cleanup. Maps to the afterEach/afterThis construct in traditional testing frameworks.

foreach: Map<string, string>[] - allows a single test to be run multiple times with different input values.

Why shrun?

Some of you may think a dockerized solution like this is overkill. I understand that sentiment but there are convincing reasons why shrun brings value:

Each test runs in it's own isolated environment. CLI testing is unique in the sense that it's often the ultimate point of contact between your product and the user. Ensuring that a set of steps runs from start to finish on X environment is paramount.
Tests have minimal ability to interfere with each other. There are still issues such as noisy neighbors and throttling by external services, but generally speaking parallel test runs will not degrade the reliability of the tests.
The containers of troublesome failing tests can be sent to other developers and debugged quickly.
You can run shrun on any platform that supports Docker (basically all of them)

Conclusion

This is the initial release of shrun so don't expect things to be perfect. In the future I hope to improve the framework and add all relevant but missing Jest flags. Contributors and feedback are welcome and desired, so I'd love to hear how shrun could be improved to better fit your needs. If you like what you saw, please star the project on GitHub so it can be useful to a wider audience.

A Product Story

Ryland G — Thu, 01 Aug 2019 14:17:24 +0000

Cover Credit

Setup

Seven months ago, the idea for a “FaaS webpack integration” emerged at a team brainstorming session. As with most great ideas, it was initially misunderstood, miscommunicated and therefore, dismissed.

Three months ago, it became apparent that some changes were needed. We had a few great customers with some specific use cases, but in order to continue growing, we needed something more. We still believed strongly in our core product value (scalable and easy to use cloud compute), and therefore weren’t looking to create something entirely new. Instead, we wanted to take the product we loved and shift it, so more developers could love it too.

Over the following month, all energy was dedicated towards understanding our options. We revisited past ideas, came up with some new ones and did as much market research as possible. The outcome: a narrow list of 4-5 ideas for repurposing our existing product to broaden developer appeal. On paper, a few of the possibilities seemed to have real potential, but the excitement wasn’t there. As a business, not choosing a potentially lucrative path due to lack of internal excitement is one of the hardest choices to make, but almost always the right one. Great products are equal parts understanding your users, solid engineering, and passion. It’s possible to have a successful product without passion, but not a truly great one.

Two months ago, a reimagining of the webpack idea popped into my head, and I started to feel the excitement. After spending some serious time researching the front-end ecosystem, it became clear that there was still no adequate solution for full-stack development. There were a myriad of offerings, but none of them seemed to address the crux of the problem.

// Detect dark theme var iframe = document.getElementById('tweet-1123232047441633282-927'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1123232047441633282&theme=dark" }

Product Definition

Using the webpack plugin as a basis, I began imagining a platform that would enable front-end developers to focus only on the things that bring them value. Armed with this idea and the approval of Avner (our CEO), I quickly began work on a demo version of the system with the help of Vladimir (one of our senior engineers).

Two days, a few thousand lines of code and too many cups of coffee later, the demo system was working. As an engineer, I can proudly say it was one of the most hacked together, janky pieces of software that I’d ever been a part of, but it worked. I was also quite nervous, I felt very passionate about the idea and was worried others might not like my interpretation of it. Avner was the first to try the demo, he instantly dispelled any worries I had when he said “Where has this been all my life”.

As the rest of the team heard about the idea, excitement started to grow again. Not wanting to lose momentum, I began pushing for an initial “demo release”. The demo release would only include a video and a blog post, showcasing this new product. It’s always important to validate your assumptions and beliefs, before investing dev resources in a full fledged product effort. The demo proposal was quickly approved. So I began working on a product document, that I could use to align with the rest of the team. During this time, Roey (another senior engineers) had played with the demo system, and started making some feature additions. Within a day he released a highly improved version of the original demo system. Working closely with him, I was able to quickly iterate on an initial proposal that could be shared with the team.

After collecting more feedback from the team, I set to work preparing for the release. It was Monday, and I had agreed to Friday for the release date. There was now a deadline looming. I needed to make the videos, get the demo system and code presentable enough for the video and prepare the material that I planned to release on social platforms. The first thing I handled, was setting up accounts for social sharing, those can take days to become usable (for example reddit). Next priority, was getting the code presentable, as this was a blocker for the video and other content. This process took a few iterations, which is a challenging feat, considering our 10 hour time gap. Everyone pushed extra hard and stayed in constant communication, allowing us to avoid basic misunderstandings. During the periods of downtime, I made the necessary modifications to the demo system so it would be presentable for the video.

The Final Push

It was Wednesday by the time the code had reached an agreeable state. I now had two days to publish the blog post and make the accompanying video. After a 14 hour push on Wednesday, I was able to finish the initial draft of the blog post and video. For our team in Israel, weekend begins on Friday, making “Israeli Thursday” the “American Friday”. This meant, it was the last real opportunity to get feedback from the team. Unfortunately, when you do 14 hours of work without communicating, there’s a high chance of misalignment. So when I woke up Thursday morning, it was to a daunting number of comments, suggestions and fixes on my post and video. It was obvious that my document and video didn’t meet the team’s expectations. So instead of trying to “put a bandaid on a broken leg", I decided to start from scratch, using the feedback I had received as a guide. After another seriously long day, V2 of the blog post and video were complete. At this point I was feeling very stressed. Release was set for the next day, and both components of the release (blog post and video), were entirely unreviewed.

I woke up the next day, fearing the worst. My phone was full of notifications from Google docs. I immediately jumped on the computer, and was incredibly relieved to see that this time I've hit home. While there were considerable comments, they were much less fundamental compared to the previous day. All of the actionable feedback could easily be completed within a few hours. This was still far from optimal, as it’s never a good feeling to be working on a release the day of release. It was also Friday in the states, which is deep into the Israeli weekend. Fortunately, Michael (our CTO) sacrificed his weekend and supported me in every way he possibly could. This allowed me to quickly address the feedback on the post and video, making them release ready.

As I went to publish the post on our blog, disaster struck. The npm package ecstatic which our blog depends on, had been unpublished entirely from npm and removed from github, just 1 hour before. I first attempted to look for an older version, but the author had been incredibly diligent about wiping everything ecstatic related. Luckily, Michael had a cached version of the blog on his computer. Using him as a proxy, I was able to make the final formatting changes and corrections, allowing me to release the blog and video.

Gathering Feedback

It was time for the hard part, getting people interested enough to give feedback. There’s absolutely no easy way to go about this, and usually brute force is the most viable option. I knew that if I limited the marketing effort to a single platform, we would receive nowhere near enough feedback to reach a reliable conclusion about the product. To increase our chances of success, I opted for a breadth approach. This meant posting on countless subreddits, Slack, Discord and Gitter channels, Spectrum chat, Twitter, YCombinator, Dev.to and many many more. Even then, things were difficult. Getting people to give feedback is much harder than getting them to retweet, like or upvote, the conversion rate is incredibly low.

But it seemed my efforts were not in vain. After a few hours, our post on /r/reactjs hit the front page. The comments we received were incredibly insightful, but there were only a handful. This isn’t surprising considering we only had a video and blog post, meaning that potential users had “nothing to try”. In fact, the number one criticism we received across all platforms was, “it’s hard for me to judge it without trying it”.

Once it became clear that the reddit posts were not generating enough feedback, I shifted efforts elsewhere. I knew that the only reliable way to get more feedback was by starting 1:1 discussions on a live chat platform. So I began posting in every Discord, Slack and Gitter channel related to our space, immediately responding to anyone who showed interest. This is an incredibly uncomfortable undertaking for many people (myself included), but resulted in the most (and highest quality) feedback by a large margin. It’s also very time consuming, so the rest of my day was spent strictly DMing potential users.

By the end of the day, I had managed to collect a substantial amount of feedback. My hundreds of conversations allowed me to produce a list of 40-50 tangible criticisms and observations, which is exactly what I set out to collect.

Takeaways

There are definitely some learnings from the experience, I’ve shared those below

Communication is critical. Without daily syncs and continuous alignment checks, the release would have never been possible. Teams that are constantly communicating are usually constantly productive. This is especially important with a remote team spread over different time zones.

Prepare in advance. Because I didn’t fully prepare for the social marketing effort, a few channels weren’t available to us. Remember, each platform has its own requirements and etiquette for posting. You definitely want to get those things underway far before release day.

It’s never too early for feedback. We didn’t have a usable product, demo or even solidified APIs and we were still able to get the validation we needed. The top priority for any new product should always be validation.

Initiate dialogue. It’s always uncomfortable to start a conversation with a stranger. Most times, if you’re willing to initiate, people are more than happy to give their opinions. The majority of the positive interactions I had, stemmed from me cold starting a conversation.

Overall, it was an incredibly valuable and defining experience for us. We gained the confidence needed to feel comfortable moving forward with a new product.

Updated (July 31 2019): Over the last few months we've been hard at work to provide something that will change developers lives. Binaris is now ShiftJS. If you've ever thought that full stack development isn't quite as easy as it should be, ShiftJS is for you. We have a killer team, and with the addition of Amir Shevat (previously VP Product Twitch/Amazon) as CPO, we are now full fledged ninja assassins. Join us!

We are now in private beta. If you’d like to give ShiftJS a try, sign up for the waitlist here.

https://shiftjs.com/

For those interested in seeing the original blog post and video in question.

https://blog.binaris.com/cloudless/

Practical Ways to Write Better JavaScript

Ryland G — Thu, 25 Jul 2019 14:08:56 +0000

I don't see enough people talking about practical ways to improve at JavaScript. Here are some of the top methods I use to write better JS.

Use TypeScript

The number one thing you can do to improve your JS, is by not writing JS. For the uninitiated, TypeScript (TS) is a "compiled" superset of JS (anything that runs in JS runs in TS). TS adds a comprehensive optional typing system on top of the vanilla JS experience. For a long time, TS support across the ecosystem was inconsistent enough for me to feel uncomfortable recommending it. Thankfully, those days are long behind us and most frameworks support TS out of the box. Now that we're all on the same page about what TS is, let's talk about why you would want to use it.

TypeScript enforces "type safety".

Type safety describes a process where a compiler verifies that all types are being used in a "legal" way throughout a piece of code. In other words, if you create a function foo that takes a number:

function foo(someNum: number): number {
  return someNum + 5;
}

That foo function should only ever be called with a number:

good

console.log(foo(2)); // prints "7"

no good

console.log(foo("two")); // invalid TS code

Aside from the overhead of adding types to your code, there are zero downsides to type-safety enforcement. The benefit on the other hand, is too large to ignore. Type safety provides an extra level of protection against common errors/bugs, which is a blessing for a lawless language like JS.

starring shia lebouf

Typescript types, make refactoring larger applications possible.

Refactoring a large JS application can be a true nightmare. Most of the pain of refactoring JS is due to the fact that it doesn't enforce function signatures. This means, a JS function can never really be "misused". For example, if I have a function myAPI that is used by 1000 different services:

function myAPI(someNum, someString) {
  if (someNum > 0) {
    leakCredentials();
  } else {
    console.log(someString);
  }
}

and I change the call signature a bit:

function myAPI(someString, someNum) {
  if (someNum > 0) {
    leakCredentials();
  } else {
    console.log(someString);
  }
}

I have to be 100% certain, that every place where this function is used (1000's of places), I correctly update the usage. If I even miss 1 my credentials could leak. Here's the same scenario with TS:

before

function myAPITS(someNum: number, someString: string) { ... }

after

function myAPITS(someString: string, someNum: number) { ... }

As you can see, the myAPITS function went through the same change as the JavaScript counterpart. But instead of resulting in valid JavaScript, this code results in invalid TypeScript, as the 1000's of places it's used are now providing the wrong types. And because of the "type safety" we discussed earlier, those 1000 cases will block compilation, and your credentials don't get leaked (that's always nice).

TypeScript makes team architecture communication easier.

When TS is setup correctly, it will be difficult to write code without first defining your interfaces and classes. This also provides a way to share concise, communicative architecture proposals. Before TS, other solutions to this problem existed, but none solved it natively, and without making you do extra work. For example, if I want to propose a new Request type for my backend, I can send the following to a teamate using TS.

interface BasicRequest {
  body: Buffer;
  headers: { [header: string]: string | string[] | undefined; };
  secret: Shhh;
}

I already had to write the code, but now I can share my incremental progress and get feedback without investing more time. I don't know if TS is inherently less "buggy" than JS. I do strongly believe that forcing developers to define interfaces and API's first, results in better code.

Overall, TS has evolved into a mature and more predictable alternative to vanilla JS. There is definitely still a need to be comfortable with vanilla JS, but most new projects I start these days are, TS from the outset.

Use Modern Features

JavaScript is one of the most popular (if not the most) programming languages in the world. You might expect that a 20+ year old language used by 100s of millions of people would be mostly "figured out" by now, but the opposite is actually true. In recent times, many changes and additions have been made to JS (yes I know, technically ECMAScript), fundamentally morphing the developer experience. As someone who only started writing JS in the last 2 years, I had the advantage of coming in without bias or expectations. This resulted in much more pragmatic, non-religious choices about what features of the language to utilize and which to avoid.

async and await

For a long time, asynchronous, event driven callbacks were a unavoidable part of JS development:

traditional callback

makeHttpRequest('google.com', function (err, result) {
  if (err) {
    console.log('Oh boy, an error');
  } else {
    console.log(result);
  }
});

I'm not going to spend time explaining why the above is problematic (but I have before). To solve the issue with callbacks, a new concept, "Promises" were added to JS. Promises allow you to write asynchronous logic, while avoid the nesting issues that previously plagued callback-based code.

Promises

makeHttpRequest('google.com').then(function (result) {
  console.log(result);
}).catch(function (err) {
  console.log('Oh boy, an error');
});

The biggest advantage of Promises over callbacks is readability and chainability.

While Promises are great, they still left something to be desired. At the end of the day, writing Promises still didn't feel "native". To remedy this, the ECMAScript comittee decided to add a new method of utilizing promises, async and await:

async and await

try {
  const result = await makeHttpRequest('google.com');
  console.log(result);
} catch (err) {
  console.log('Oh boy, an error');
}

The one caveat being, anything you await must have been declared async:

required definition of makeHttpRequest in prev example

async function makeHttpRequest(url) {
  // ...
}

It's also possible to await a Promise directly since an async function is really just a fancy Promise wrapper. This also means, the async/await code and the Promise code, are functionally equivalent. So feel free to use async/await without feeling guilty.

let and const

For most of JS's existence, there was only one variable scope qualifier var. var has some pretty unique/interesting rules in regards to how it handles scope. The scoping behavior of var is inconsistent and confusing, and has resulted in unexpected behavior and therefore bugs , throughout the lifetime of JS. But as of ES6, there is an alternative to var, const and let. There is practically zero need to use var anymore, so don't. Any logic that uses var, can always be converted to equivalent const and let based code.

As for when to use const vs let, I always start by declaring everything const. const is far more restrictive and "immutablish" which usually results in better code. There aren't a ton of "real scenarios" where using let is necessary, I would say 1/20 variables I declare with let. The rest are all const.

I said const is "immutablish" because it does not work in the same way as const in C/C++. What const means to the JavaScript runtime, is that the reference to that const variable will never change. This does not mean the contents stored at that reference will never change. For primitive types (number, boolean etc), const does translate to immutability (because it's a single memory address). But for all objects (classes, arrays, dicts), const does not guarantee immutability.

Arrow => Functions

Arrow functions are a concise method of declaring anonymous functions in JS. Anonymous functions, describe functions that aren't explicitly named. Usually, anonymous functions are passed as a callback or event hook.

vanilla anonymous function

someMethod(1, function () { // has no name
  console.log('called');
});

For the most part, there isn't anything "wrong" with this style. Vanilla anonymous functions behave "interestingly" in regards to scope, which can/has result in many unexpected bugs. We don't have to worry about that anymore, thanks to arrow functions. Here is the same code, implemented with an arrow function:

anonymous arrow function

someMethod(1, () => { // has no name
  console.log('called');
});

Aside from being far more concise, arrow functions also have much more practical scoping behavior. Arrow function inherit this from the scope they were defined in.

In some cases, arrow functions can be even more concise:

const added = [0, 1, 2, 3, 4].map((item) => item + 1);
console.log(added) // prints "[1, 2, 3, 4, 5]"

Arrow functions that reside on a single line, include a implicit return statement. There is no need for brackets or semi-colons with single line arrow functions.

I want to make it clear. This isn't a var situation, there are still valid use cases for vanilla anonymous functions (specifically class methods). That being said, I've found that if you always default to an arrow function, you end up doing a lot less debugging as opposed to defaulting to vanilla anonymous functions.

As usual, the Mozilla docs are the best resource

Spread Operator ...

Extracting key/value pairs of one object, and adding them as children of another object, is a very common scenario. Historically, there have been a few ways to accomplish this, but all of those methods are pretty clunky:

const obj1 = { dog: 'woof' };
const obj2 = { cat: 'meow' };
const merged = Object.assign({}, obj1, obj2);
console.log(merged) // prints { dog: 'woof', cat: 'meow' }

This pattern is incredibly common, so the above approach quickly becomes tedious. Thanks to the "spread operator" there's never a need to use it again:

const obj1 = { dog: 'woof' };
const obj2 = { cat: 'meow' };
console.log({ ...obj1, ...obj2 }); // prints { dog: 'woof', cat: 'meow' }

The great part is, this also works seamlessly with arrays:

const arr1 = [1, 2];
const arr2 = [3, 4];
console.log([ ...arr1, ...arr2 ]); // prints [1, 2, 3, 4]

It's probably not the most important, recent JS feature, but it's one of my favorites.

Template Literals (Template Strings)

Strings are one of the most common programming constructs. This is why it's so embarassing that natively declaring strings is still poorly supported in many languages. For a long time, JS was in the "crappy string" family. But the addition of template literals put JS in a category of its own. Template literals natively, and conveniently solve the two biggest problems with writing strings, adding dynamic content, and writing strings that bridge multiple lines:

const name = 'Ryland';
const helloString =
`Hello
 ${name}`;

I think the code speaks for itself. What an amazing implementation.

Object Destructuring

Object destructuring is a way to extract values from a data collection (object, array, etc), without having to iterate over the data or access it's key's explicitly:

old way

function animalParty(dogSound, catSound) {}

const myDict = {
  dog: 'woof',
  cat: 'meow',
};

animalParty(myDict.dog, myDict.cat);

destructuring

function animalParty(dogSound, catSound) {}

const myDict = {
  dog: 'woof',
  cat: 'meow',
};

const { dog, cat } = myDict;
animalParty(dog, cat);

But wait, there's more. You can also define destructuring in the signature of a function:

destructuring 2

function animalParty({ dog, cat }) {}

const myDict = {
  dog: 'woof',
  cat: 'meow',
};

animalParty(myDict);

It also works with arrays:

destructuring 3

[a, b] = [10, 20];

console.log(a); // prints 10

There are a ton of other modern features you should be utilizing. Here are a handful of others that stand out to me:

Always Assume Your System is Distributed

When writing parallelized applications your goal is to optimize the amount of work you're doing at one time. If you have 4 available cores, and your code can only utilize a single core, 75% of your potential is being wasted. This means, blocking, synchronous operations are the ultimate enemy of parallel computing. But considering that JS is a single threaded language, things don't run on multiple cores. So what's the point?

JS is single threaded, but not single-file (as in lines at school). Even though it isn't parallel, it's still concurrent. Sending an HTTP request may take seconds or even minutes, if JS stopped executing code until a response came back from the request, the language would be unusable.

JavaScript solves this with an event loop. The event loop, loops through registered events and executes them based on internal scheduling/prioritization logic. This is what enables sending 1000's of "simultaneous" HTTP requests or reading multiple files from disk at the "same time". Here's the catch, JavaScript can only utilize this capability if you utilize the correct features. The most simple example is the for loop:

let sum = 0;
const myArray = [1, 2, 3, 4, 5, ... 99, 100];
for (let i = 0; i < myArray.length; i += 1) {
  sum += myArray[i];
}

A vanilla for loop is one of the least parallel constructs that exists in programming. At my last job, I led a team that spent months attempting to convert traditional R lang for-loops into automagically parallel code. It's basically an impossible problem, only solvable by waiting for deep learning to improve. The difficulty of parallelizing a for-loop comes from a few problematic patterns. Sequential for-loops are very rare, but they alone make it impossible to guarantee a for-loops separability:

let runningTotal = 0;
for (let i = 0; i < myArray.length; i += 1) {
  if (i === 50 && runningTotal > 50) {
    runningTotal = 0;
  }
  runningTotal += Math.random() + runningTotal;
}

This code only produces the intended result if it is executed in order, iteration by iteration. If you tried to execute multiple iterations at once, the processor might incorrectly branch based on inaccurate values, which invalidates the result. We would be having a different conversation if this was C code, as the usage is different and there are quite a few tricks the compiler can do with loops. In JavaScript, traditional for loops should only be used if absolutely necessary. Otherwise utilize the following constructs:

map

// in decreasing relevancy :0
const urls = ['google.com', 'yahoo.com', 'aol.com', 'netscape.com'];
const resultingPromises = urls.map((url) => makHttpRequest(url));
const results = await Promise.all(resultingPromises);

map with index

// in decreasing relevancy :0
const urls = ['google.com', 'yahoo.com', 'aol.com', 'netscape.com'];
const resultingPromises = urls.map((url, index) => makHttpRequest(url, index));
const results = await Promise.all(resultingPromises);

for-each

const urls = ['google.com', 'yahoo.com', 'aol.com', 'netscape.com'];
// note this is non blocking
urls.forEach(async (url) => {
  try {
    await makHttpRequest(url);
  } catch (err) {
    console.log(`${err} bad practice`);
  }
});

I'll explain why these are an improvement over traditional for loops. Instead of executing each "iteration" in order (sequentially), constructs such as map take all of the elements and submit them as individual events to the user-defined map function. This directly communicates to the runtime, that the individual "iterations" have no connection or dependence to each other, allowing them to run concurrently. There are many cases where a for-loop would be just as performant (or maybe more) in comparison to a map or forEach. I would still argue that losing a few cycles now, is worth the advantage of using a well defined API. That way, any future improvements to that data access patterns implementation will benefit your code. The for-loop is too generic to have meaningful optimizations for that same pattern.

There are a other valid async options outside of map and forEach, such as for-await-of.

Lint Your Code and Enforce a Style

Code without a consistent style (look and feel), is incredibly difficult to read and understand. Therefore, a critical aspect of writing high-end code in any language, is having a consistent and sensible style. Due to the breadth of the JS ecosystem, there are a LOT of options for linters and style specifics. What I can't stress enough, is that it's far more important that you are using a linter and enforcing a style (any of them), than it is which linter/style you specifically choose. At the end of the day, no one is going to write code exactly how I would, so optimizing for that is an unrealistic goal.

I see a lot of people ask whether they should use eslint or prettier. For me, they serve very different purposes, and therefore should be used in conjunction. Eslint is a traditional "linter", most of the time, it's going to identify issues with your code that have less to do with style, and more to do with correctness. For example, I use eslint with AirBNB rules. With that configuration, the following code would force the linter to fail:

var fooVar = 3; // airbnb rules forebid "var"

It should be pretty obvious how eslint adds value to your development cycle. In essence, it makes sure you follow the rules about what "is" and "isn't" good practice. Due to this, linters are inherently opinionated. As with all opinions, take it with a grain of salt, the linter can be wrong.

Prettier is a code formatter. It is less concerned with "correctness", and far more worried about uniformity and consistency. Prettier isn't going to complain about using var, but it will automatically align all the brackets in your code. In my personal development process, I always run prettier as the last step before pushing code to Git. In many cases, it even makes sense to have Prettier run automatically on each commit to a repo. This ensures that all code coming into source control has consistent style and structure.

Test Your Code

Writing tests, is an indirect but incredibly effective method of improving the JS code you write. I recommend becoming comfortable with a wide array of testing tools. Your testing needs will vary and there's no single tool that can handle everything. There are tons of well established testing tools in the JS ecosystem, so choosing tools mostly comes down to personal taste. As always, think for yourself.

Test Driver - Ava

AvaJS on Github

Test drivers are simply frameworks that give structure and utilities at a very high level. They are often used in conjunction with other, specific testing tools, which vary based on your testing needs.

Ava is the right balance of expressiveness and conciseness. Ava's parallel, and isolated architecture is the source of most my love. Tests that run faster save developers time and companies money. Ava boasts a ton of nice features, such as builtin assertions, while managing to stay very minimal.

Alternatives: Jest, Mocha, Jasmine

Spies and Stubs - Sinon

Sinon on Github

Spies give us "function analytics" such as how many times a function was called, what they were called by, and other insightful data.

Sinon is a library that does a lot of things, but only a few super well. Specifically, sinon excels when it comes to spies and stubs. The feature set is rich but the syntax is concise. This is especially important for stubs, considering they partially exist to save space.

Alternatives: testdouble

Mocks - Nock

Nock on Github

HTTP mocking is the process of faking some part of the http request process, so the tester can inject custom logic to simulate server behavior.

Http mocking can be a real pain, nock makes it less painful. Nock directly overrides the request builtin of nodejs and intercepts outgoing http requests. This in turn gives you complete control of the response.

Alternatives: I don't really know of any :(

Web Automation - Selenium

Selenium on Github

Selenium is one I have mixed emotions about recommending. As it's the most popular option for web automation, it has a massive community and online resource set. Unfortunately, the learning curve is pretty steep, and it depends on a lot of external libraries for real use. That being said, it's the only real free option, so unless you're doing some enterprise grade web-automation, Selenium will do the job.

Two Other Random JS Things

Very rarely should you use null, poor null
Numbers in JavaScript just suck, always use a radix parameter with parseInt

Conclusion

Draw your own.

Things All Developers Should Learn In College

Ryland G — Mon, 15 Jul 2019 14:16:00 +0000

Forget About "Lines of Code"

[Source](https://images.techhive.com/images/article/2015/09/historic_loc-chart-100615917-large.idge.jpg)

As a developer, you'll hear a lot of crazy, unbelievable theories about what "lines of code" signify. Believe none of them. Lines of code is a ridiculous metric to base decisions on. In very rare cases it tells you something, in all the other cases, it tells you nothing. Using lines of code to make decisions is like rating book quality by number of pages.

Some might make the case that the fewer the lines of code there are in an application, the easier it is to read. This is only partially true, my metrics for readable code are:

Code should be consistent
Code should be self descriptive
Code should be well documented
Code should utilize stable modern features
Code shouldn't be unnecessarily complex
Code shouldn't be un-performant (don't write intentionally slow code)

The moment reducing the number of lines of code interferes with any of those, it becomes a problem. In practice, it will almost always interfere and thus is nearly always a problem. But here's the thing, if you strive to meet the above criteria, your code will be the perfect number of lines, no need for counting.

Languages aren't necessarily "Good" or "Bad"

_Except for php, just kidding. [Source](https://stackoverflow.blog/wp-content/uploads/2017/10/languages-1-900x675.png)_

I see people say stuff like this, all the time:

C is better than X, because performance

Python is better than X, because conciseness

Haskell is better than X, because aliens

The notion, that a language comparison could be reduced down to a single sentence is almost insulting. They're languages, not Pokemon.

Don't get me wrong, there are definitely differences between languages. It's just that, there are very few "unusable" languages (although there are many outdated/dead languages). Each language brings it's own unique set of tradeoffs. In that regard, languages are similar to tools in a toolbox. A screwdriver can do what a hammer can't, but would you ever say a screwdriver is better than a hammer?

obviously the hammer is better

Before I talk about how I evaluate languages, I want to make something very clear. There are very few cases where the language choice actually matters. There are things you can obviously not do in some languages. If you write frontend code, you don't get a language choice. But in general, language choice is usually one of the least important issues for a project.

Here are the core aspects (ordered), that I believe should dictate your choice of language (these are it's Pokemon stats)

Density of available online resources (StackOverflow density)
Development Velocity (vroom vroom)
Bug proneness (eeek)
Quality and breadth of package ecosystem (yea npm, it says quality)
Performance characteristics (more dots)
Hirability (sorry COBOL)

There are also some strong couplings that are out of your hands. If you're working in data science, you realistically need to use Python, R or Scala (maybe Java). If it's a hobby project, use whatever will make you happiest. There's only one non-negotiable rule I have. I refuse to use languages that don't have most of the problems I will encounter, directly solved on StackOverflow. It's not that I can't solve it, it's just not worth the time.

Reading Other People's Code is Hard

[Source](http://www.sph.as/why-bible-reading-can-be-hard-for-kids-and-what-to-do-about-it/)

Reading other peoples code is difficult. Robert C. Martin talks about this in "Clean Code":

Indeed, the ratio of time spent reading versus writing is well over 10 to 1. We are constantly reading old code as part of the effort to write new code. ...[Therefore,] making it easy to read makes it easier to write.

For a long time, I assumed that I just sucked at reading other peoples code. Over time, I realized that it's something almost every programmer struggles with on a daily basis. Reading other people's code almost feels like reading a foreign language. Even if you're comfortable with the programming language choice of the writer, you still have to adjust to differing styles and architecture choices. This also assumes that the author wrote consistent and reliable code, which can be hit or miss. This is a really difficult one to overcome. There are a few things I've found helped a LOT.

Reviewing other peoples code will improve your code reading skills immensely. In the past two years, I've reviewed quite a few Github PR's. With each PR, I feel slightly more comfortable reading other peoples code. Github PR's are especially great for these reasons

Can be practiced anytime, just find an open source project that you feel like you want to contribute to.
Practice reading in a scoped context (driving feature or bug of PR).
Attention to detail required, which will force you to evaluate each line.

The second hack which can help you read other peoples code is a bit more unique. It's a technique I came up with, and it's really reduced the amount of time it takes for me to feel comfortable in a foreign codebase. After looking at the style of the code I want to read, I first open up vi and starting writing code in the style used by the project. When you write code in the new style, it will also improve your reading skills. The style will feel less foreign, as you've actually experienced it. Even if I'm just browsing a random project on Github, I'll quickly do this. Try it out.

You'll Never Write "Perfect" Code

[Source](https://www.youtube.com/watch?v=WPoQfKQlOjg)

I was a "lone wolf" developer for 4 years before I started working on a team. For most of that time, I just had this assumption that every programmer in the industry wrote perfect code. I figured it was just a matter of time and effort before I also wrote "perfect" code.
It's something I used to feel really anxious about. Once I joined a team, it quickly became clear that no one was writing "perfect" code. But the code going into the system was almost always "perfect", what gives? The answer, code reviews.

I work with a team of really brilliant engineers. These are some of the most competent, confident programmers money can buy. Every single member of our team (including me) would have a full blown panic-attack if someone ever suggested committing un-reviewed code. Even if you think you're the next Bill Gates, you will make mistakes. I'm not even talking logical mistakes, I'm talking typos, missing characters. Things that your brain tunes out and will never pick up on. Things you need another set of eyes for.

Strive to work with others that are attentive to detail and willing to criticize your work. Hearing criticism is difficult at first, but it's the only consistent way to improve. Do your best to not become defensive during a code review, and never take any comments personally. You're not your code.

When I'm reviewing someone else's code, I just Google search every choice they make and see if it differs from strong popular opinion. Often times, looking at the same problem with a "beginners mind", can catch things the person would have never gone back and looked at.

Working as a Programmer Doesn't Mean 8 Hours of Programming a Day

This is a very common question, but people never seem to give a clear answer.

Very few people are writing code for more than 4 hours a day

People who disagree with this are either the exception to the rule or work at companies that should treat them better. Programming is a mentally draining, focus intensive task. It's entirely unreasonable to expect anyone to write code for 8 hours a day, 5 days a week. There will be rare cases where you need to meet a deadline or pull a little extra for a stretch, but those are few and far between. When I say "rare", I mean almost never. Do not tolerate a workplace that abuses you and makes you work overtime due to poor planning/under-hiring.

For the record, it's not even in your companies best interest for you to be actively programming for 8 hours a day. Your boss might think that's the case, but it's shortsighted and ignores the longterm implications, on productivity and mental health.

Just to be clear, I'm not suggesting that you only work four hours a day. The other four hours are usually best spent:

Researching, both work related and non-work related topics
Discussing your work with a colleague
Helping coworkers who are struggling with their own work
Planning your future work
Code Reviews
Meetings

I also highly recommend taking regular breaks during the day, and exercising (even if only briefly). The benefits of exercise on mental fatigue are well documented. I've personally found that exercise especially helps if I'm having trouble focusing.

Update: /u/terriblesarcasm suggested that I add what a developer should be doing in the 4 hours they’re not programming. I added a section addressing that.

Conclusion

These are a few of the things I wish they were teaching at university instead of pure theory. In the process of writing this, I came up with a ton of other items but those will have to come in the next post!

How to Write a Good Blog Post

Ryland G — Mon, 08 Jul 2019 15:11:32 +0000

Cover Image

In my first month of blogging, I learned a ton of lessons, that have helped me improve as a writer, as a communicator and as a person.

Last week, I published a post titled "How I Started Existing on the Internet". The post primarily discussed my social media efforts, but I also tried to include some practical advice about writing successful blogs. I got a lot of feedback based on that post, and the most overwhelming request was for more meta content about the blogging process. When I received the following comment yesterday, it was the inspiration I needed to write this post.

Thanks Tim!

How to Write a Good Blog Post

Set the Right Expectations

The first, and most important tip to writing blogs, is to calm down. Out of all the conversations I've had with people who want to blog, but haven't yet, the only common factor is anxiety.

I believe that the common advice/support given to these people is toxic and only contributes to that anxiety. I can't even count the number of times I've seen comments such as:

Oh you're just in your head, your post is going to do very well

Great content speaks for itself, if it's quality people will read it

Look, it sucks that I have to be the one to tell you this, but that's all bullshit. I'm not saying those aren't genuine comments driven by a desire to positively impact others, they're just not factual.

No matter how amazing your writing/technical skills are, some of your content will do bad, and that's ok.

This is exactly what I mean by setting the right expectations for yourself. I see a lot of new bloggers spin a fantasy in their head of how things will play out. And even when things go well, if they don't match the fantasy exactly, those people feel disappointed. It's critical that you accept right now that you will produce a ton of flukes throughout your blogging career. It's normal, and it happens to EVERYONE! In fact, I would bet every dollar I own that INSERT-YOUR-FAVORITE-BLOGGER-HERE has written many posts that tanked, you just obviously never heard about those ones.

Source

Another aspect of this is, what really constitutes a "bad blog post". For most people, a "bad blog post" is any blog post that doesn't do well. With a polarizing definition like that, it's not hard to see why people feel very anxious about writing blogs. You have to remember, there are so many factors that go into whether a blog post (or any content on the internet) succeeds or fails. Unfortunately, the quality of that content is not even the primary determining factor.

Accepting this as truth brings a ton of benefits. If you internally believe that a great post can do bad, you won't set unrealistic expectations that only serve to disappoint you. Keep in mind, just because you wrote a good post and it didn't do well, doesn't mean you can't improve. As I said, there are many aspects to producing a successful blog post outside of the writing quality. Here are a few...

You distributed the post to the wrong audience.
You didn't distribute the blog post at all.
The formatting and organization of the post turned people off
The title wasn't catchy enough
You posted at the wrong time of day
There was similar content out that hurt your visibility

"There was similar content out that hurt your visibility" is especially great, because this is a problem that also affects box office movies. Would you say that every movie that does bad at the box office is a "bad movie"? I wouldn't, in fact, some of the most critically acclaimed movies bombed at the box office.

By all standard measures, this movie bombed

Personal Note: I said that everyone writes unsuccessful posts and that definitely includes me. Out of all the posts I've written, the one I spent the most time on did the worst. Over the few weeks it took to write the post, I learned that it might not work well with my audience. Learning that ahead of time made it a lot easier to see the post not succeed. In the end, I enjoyed writing it and it helped me improve so it was just as worthwhile as a successful post.

Have a Goal

Having a goal when you start, will tremendously improve your writing quality, and increase the value that others distill from your work. The biggest advantage of being goal driven when blogging is the effect it has on the "focus" of your content.

"Content focus" is something that many people are aware of, but never think about. Have you ever read an article that was allegedly about X and somehow turned into a discussion of Y. A lot of the time, I'll see comments on one of those posts accusing the author of being "clickbaity". It's funny, because half the time they aren't even trying to write clickbait, they just got distracted when writing the article and lost focus on their goal. Remember, if someone clicked on your article, it was because they were interested in the value alleged by the title. The only way your post can be "bad" is if it doesn't fulfill the expectation you set.

Let's take this post for example. The title is "How to Write a Good Blog Post", which is a pretty lofty goal to deliver on. Keep in mind that you're not expected to please everyone. There's a good chance that someone will read this post and not get anything meaningful out of it, and again, that's ok. As long as I'm honest, on topic and striving not to waste the readers time, I've done my part.

On a more meta level, goals tie back into setting realistic expectations. I've seen many people set their high level goal as

Write my first blog post, it should also be successful.

How many times have you tried something for the first time and done it perfectly, how about even well? It's not just you, things take practice. A much more realistic goal would be

I will write 10 blog posts and one will succeed.

Lastly, it doesn't have to be externally validated goal, just a purposeful one.

Know Your Audience

Having an accurate "audience profile" is key to writing a good blog post. An audience profile is the set of defining characteristics of your average reader. Even if you have 0 followers, and have never written a blog post, you still have an audience profile. Although in that case, it would be mostly defined by audience profile of the sites where you post.

Audience Profile of a Few Sites

Site	Profile (Heavily Opinionated)
Devto	programming, tech, mostly beginners, discussion, friendly, very liberal
Reddit (programming subs)	experienced, critical, attentive to detail, political (moderator based),
Hacker News	tech, negative, overly critical, bitter, highly intelligent
LinkedIn	professional, no non-sense, critical, mixed experience

Although it may not feel the most natural, writing content with your audience in mind will produce a more relatable and empathetic result. It helps me to think of each individual blog post as a full fledged product. This allows me to manage them like I would any other product, by prioritizing features (content) that solves a user need.

Knowing your audience allows you to focus your writing and deliver on the the core value of your post. For example, if you're explaining technical concepts, how much can you expect your average reader to know coming in? This is something I see a lot of writers struggle with, including myself. I'll see a post that is seemingly aimed at a highly technical crowd, but starts off teaching people about rudimentary programming concepts. Don't do that!

There are a lot of other potential ways to alienate your audience. The unfortunate reality is that your content will never be able to satisfy everyone, so your goal should just be to satisfy one group very well. There are some very common things that alienate audiences that I do my best to avoid.

Memes that are only funny with context
Using unnecessarily complex words when simpler alternatives are possible
Title doesn't convey value of the post to the audience
Excess references to pop culture or other time sensitive constructs

irony.jpg

The last point is pretty easy to understand because you've all experienced it before watching older T.V. shows. Some shows are timeless and others become very dated very quickly.

Personal Note: I struggle to not alienate my audience more than anything else discussed in this post. I involve myself with a wide range of topics, so it's very hard for me to not rely on references to other topics and fields outside of audience comfort zone. I learn very relationally, so often times I struggle to explain a concept if it's not in the context of something else.

Create an Outline

The biggest improvement to my blogging came from outlines, and by quite a margin. I wrote quite a few posts before I started using outlines, and it was always a big time-wasting mess. I would often spend 1-2 hours writing before I realized that the pieces of my idea didn't really connect, or that the information didn't have a natural way to fit together.

Source

Your brain also hates that empty space staring back at you. Intros are so crucial, so it's pretty daunting to start a post if the intro is the first task you have to tackle. To combat this, start your post by just jotting down a ton of topics. I usually try to organize these in logical sections, that way the outline provides starter info and structure.

In my experience, outlines also have a tremendously positive effect on "Decision Fatigue". Decision Fatigue is a scientifically studied phenomenon that describes how each decision we make (over the course of a day) inherently reduces the quality of future decisions we make. When it comes down to it, writing a blog post involves making a ton of micro and macro decisions. I have found writing an outline to be the most effective method of combating decision fatigue.

Personal Note: Here's the actual outline I used for this post.

As you may have noticed, some stuff changes. That's ok, the outline still did its job.

Start Writing

I actually have the least to say about this part of the process. Being a good writer is an entirely separate skill from being a good blogger. I'm not a bad writer, but I'm not a great writer. I do have some general tips that I find very useful.

Don't be afraid to move on. There is no required linear path when writing a blog post, lol.
If you can handle reading other similar posts to yours first without getting discouraged, do it. It will help you write more confidently and know which areas to spend more time addressing.
Use a spell checker.
Have a quiet place to work.
Set aside enough time to not feel rushed.

Learning From Twitter

// Detect dark theme var iframe = document.getElementById('tweet-1137658720698228736-847'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1137658720698228736&theme=dark" }

This tweet-storm is a great example of concise effective communication

I've started using Twitter recently (just since last month). Twitter provides a great opportunity to practice succinct and effective communication. I tend to be "unnecessarily verbose", and Twitter has really made me aware of the words I choose and the way I phrase things.

My learnings on Twitter eventually led to a strategy that I like to call "lossless blog compression". The goal is to reduce the length of a blog post as much as possible, without increasing the barrier of entry or losing meaning. This isn't an excuse to pull out the thesaurus and start swapping out words for their shorter, obscure counterparts. This is a casual, passive habit for me, so once or twice per blog post I read it over and put everything to the test.

Aim for a Peer Review

Having another person read your post before you publish is really helpful and can really boost your confidence. Unfortunately, this one is really hard to accomplish and most of the time it's not even possible. I would encourage you to send your post to personal friends and family before you release. Even if the readers aren't the intended audience, they can still provide valuable feedback on things like typos and grammar.

Conclusion

Writing a good blog post requires you to become a good blogger. As with everything, it takes practice. Go in with the right expectations and with the understanding that you'll need to walk before you run. Anyone can write good blog posts as long as they stay focused and work hard. The most important step (and most difficult) is to get yourself out there. I promise it's not as bad as you think.

For the record, there are a ton of other aspects of writing a good blog post such as

Be passionate
Be positive
Have a unique voice
Don't be religious about tooling and technology
Use lots of headers
Don't use un-cited data (or unfairly present the data like I did in this post)
Etc

I ran out of room for this post, but I might even make a part two covering some topics I didn't address. Either this week or next week I will release a meta-article covering a blog post that I never released. In the article, I will discuss how I identified that the piece was bad and how I plan to fix it.

Looking For Feedback

Ryland G — Sat, 06 Jul 2019 15:32:49 +0000

Recently, I've been writing a lot of content on Dev.to. Obviously I know how many people read each post, but it's not necessarily the best indicator of what content I should be making.

Instead, I want to know what type of content you guys will appreciate seeing the most in the future. My skillset is pretty general, so don't feel obliged to suggest topics that I've already covered, anything is fair game!

This should also be considered an open forum to give me feedback about my writing (style, content, etc) including constructive criticism (don't hold back).

I also want to just thank everyone for the support in the ~1.5 months I've been on Dev.to, this site has a really unique community.

TL;DR

What topics do you want to see more of?
What topics do you want to see less of?
How can my writing be improved?
How can I be improved?
Anything else?

The Best Book to Read as a Developer

Ryland G — Tue, 02 Jul 2019 14:59:02 +0000

I was able to determine that the book is now open sourced, here is a pdf available online. If it's between reading the book and reading my post, read the book.

One of the best choices I ever made as a software developer was to read "Inside the Machine", a book that's almost entirely about hardware. As someone who is allergic to "textbooks", this book is an oasis in a sea of poorly worded, misguided attempts at explaining hardware to software developers.

Inside the Machine covers nearly the entire spectrum of the software/hardware relationship. The book starts, by introducing you to incredibly primitive constructs of micro-processors, such as an ALU. By the end, you will be learning and understanding incredibly complex microprocessor design and architecture.

Fun Fact: Inside the Machine is written by Jon Stokes, founder of Ars Technica and briefly editor of Wired!

Why This Book?

It's Straightforward

This book cuts the bullshit. The information is presented with the minimum amount of fluff to make it digestible. It's also incredibly consistent and uses a few base analogies to explain almost every concept. Inside the Machine, does what every great instructional book should do, cater to the reader. The intro of the book explains it best:

Inside the Machine is an introduction to computers that is intended to fill the gap that exists between classic but more challenging introductions to computer architecture, like John L. Hennessy’s and David A.
Patterson’s popular textbooks, and the growing mass of works that are simply too basic for motivated non-specialist readers. Readers with some experience using computers and with even the most minimal scripting or programming experience should finish Inside the Machine with a thorough and advanced understanding of the high-level organization of modern computers. Should they so choose, such readers would then be well equipped to tackle more advanced works like the aforementioned classics, either on their own or as part of formal curriculum.

The book’s comparative approach, described below, introduces new design features by comparing them with earlier features intended to solve the same problem(s). Thus, beginning and intermediate readers are encouraged to read the chapters in order, because each chapter assumes a familiarity with the concepts and processor designs introduced in the chapters prior to it.

TL;DR: If you're not a great programmer I recommend this book even more. I read this book very early on in my dev career (when I was bad) and I believe it heavily shaped my future growth.

It Gives You Respect For What's Happening Under the Hood

One of the greatest things about this book, is that by the end, you'll really understand the "why" of modern programming. Even for most experienced developers, how the underlying computer works is essentially magic. Using that analogy, this book is goddamn Hogwarts. You might go in not being able to cast a single spell, but by the end you'll be full-fledged witch/wizard. For some, it might seem strange that I'm suggesting that learning about hardware will help you write software. But you have to remember, at the end of the day, software is just a hardware abstraction.

It's Sexy

The book is beautiful. As someone who is a bit of a perfectionist, I really appreciate the effort that not only went into the illustrations, but the overall style and design of the book. The illustrations really provide meaningful value, instead of the often contrived "figures" provided by normal text books. For example, here is one of the illustrations the author uses when teaching the reader about caching.

Another great example used to explain pipelining.

It's Aware

The amount of self awareness imparted into the book is one of my favorite aspects. The author, predicting that there might be some confusion about the previous picture, actually provides a very human rationalization of his choice.

It's Riveting At Points

I won't lie and say the entire book is a "thrill ride", but there are sections that cover the history of processors that are incredibly thrilling. Reading about PowerPC and x86 politics is surprisingly intriguing. To quantify it, I would watch the HBO series.

Conclusion

Whether you're just getting into software development, or already an experienced developer this book has something for you. Information is presented joyfully so you'll be dying (instead of dreading) to read the next chapter.

Update: I wanted to add a comment I received on Reddit from the /u/DingBat99999 because they had some amazing insight I wish I had written myself.

The suggested book looks great. But it illustrated an issue that I hadn't considered before now.

I've been working in software for about 40 years now. When I started assembly language was unavoidable. There just wasn't as much between you and the bare metal in those days. As a result, everyone kind of got an idea of how a computer worked at low levels naturally, even without hardware focused courses.

It's been a long time since I went to university so I'm not sure what they teach these days, but given the kind of new grads I see, at least some of them are graduating with no exposure to hardware at all. And that's perfectly fine. You can do quite well in the industry without it. But, as the OP's book demonstrates, it can be interesting and fun to learn about it anyway.

To wrap this up, for me, the have been a couple of books that have affected my view of creating software. They are "Clean Code" by Bob Martin, and "Working Effectively with Legacy Code" by Michael Feathers.

Free PDF of the book

The author even started porting the book to Github!

How I Started Existing on the Internet

Ryland G — Mon, 01 Jul 2019 14:52:54 +0000

Warning, this is a detailed post. If you want to just see the results click here

Intro

For those of you who don't know me. Hello, my name is Ryland. I'm an engineer/product manager working for Binaris (a startup) in the Bay area.

About a month and a half ago, my company was going through some significant changes. We launched an effort to get feedback about a new idea I was pushing through the company. As part of this effort, I had to create some content (blog, video) and then subsequently evangelize that content across social media.

Not wanting to lose momentum that the work event had given me, I decided it was the right time to start working on my social presence.

Getting Acquainted

Most people who know me also know that I'm not a fan of social media. I really believe that current platforms are designed to manipulate and hook users, instead of encouraging genuine social interactions. Although I feel very strongly about this, I also strive to be a pragmatic person. And since social media is such a key aspect of modern day business, I knew that I had to get with the program.

Until about a month ago I was completely off social media. I had a Facebook account in high school, but it's been mostly deactivated for the last 6 years. I do use Reddit constantly, but mostly as an informational tool, not for social interaction. I originally created my twitter account two years ago, but until ~1.5 months ago, I had only 40 followers (entirely made up of personal friends) and I hadn't tweeted (anything at all) since making the account originally. Here is my non-activity from April of this year.

After doing some research, I decided that twitter was the best bet for establishing myself. Even though I didn't understand the culture or the platform, I believed Twitter had the strongest demographics relevant to me (tech and lifestyle). I started by following a number of top influencers and quickly started to learn what type of content works well on Twitter.

From May 15 to May 30, I gained ~600 followers purely through tweeting arbitrary thoughts, posting joke tweets and replying to interesting content on my feed. Having done my research, I knew that hashtags were critical for tweet visibility, so I used them generously. During this time my average tweet was doing pretty decently, but the content wasn't represenative of "me".

// Detect dark theme var iframe = document.getElementById('tweet-1130505297024315392-889'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1130505297024315392&theme=dark" }

My most successful tweet of May

Towards the end of May I started to realize that although people were following me, I wasn't establishing any credibility. If you're relatively unknown on Twitter, establishing credibility is incredibly difficult. I strongly believe that this aspect of twitter contributes to the incredible toxicity of the platform and really hurts meaningful discourse. Trying to stay pragmatic, I looked for other avenues I might use to expedite my growth.

MyOS

Based on my observations from Twitter, I hypothesized that creating an open-source project was a pretty reliable way to get traction. Many social media tech influencers were creators of popular open-source projects, giving them both technical credibility and an already involved audience that was likely to follow them outside of Github.

I had been working on a personal project for a few months, and it was something that I was eager to share with others. I realized that it would be very difficult to spread the word through Twitter alone, and that a blog post seemed like a more appropriate medium. I'm very comfortable with webdev, so I spent a day or two building a blog using Gridsome.

I then moved onto cleaning up my code and writing the accompanying blog post. I really believed in the value of the project (it's something I had already been using every day), so I decided that I would do everything I could to make it visible. When everything was ready, I published the project to Github and started posting the corresponding blog post everywhere I could.

Someone gave me platinum on /r/vim

MyOS Reddit posts

During my quest to post MyOS across the internet, I heard about a newer, community-driven content platform, Dev.to. I liked what I saw after an initial visit, and decided to cross post my blog there too.

I also considered cross posting to Medium as well, but it was then that I discovered the lack of canonical_url support on Medium.

// Detect dark theme var iframe = document.getElementById('tweet-1134175295349567489-189'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1134175295349567489&theme=dark" }

MyOS was more of a success than I initially expected. As of today the project has ~120 stars and 16 contributors, not bad for a first try. Unfortunately, the MyOS model doesn't scale. It took me months to get the project into a place I was willing to share, there was no way that a multi-month process was an efficient (not to mention the risk) method of gaining followers.

But something else came out of the MyOS effort, I now had a personal blog.

Twitter stats for May (includes MyOS traffic)

The Best Medium

A week before the MyOS release, I started seeing a lot of negative attention directed at Medium on Twitter. After doing some research, I realized that most of the anger was misguided so I did my best to combat the disinformation.

// Detect dark theme var iframe = document.getElementById('tweet-1130586919014764544-681'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1130586919014764544&theme=dark" }

But then, in early June, a leaked email from the Free Code Camp founder Quincy popped up in my feed. It was obvious that Quincy was unhappy with the leak, and because the tweet was relatively unseen, I figured it would dissipate.

A few days later, articles started appearing referencing the leaked email and I no longer felt morally/ethically inclined to keep quiet about it. I was already frustrated from my experience with Medium, and I felt that the actions alleged in the email crossed the line. I quickly wrote a post discussing the leaked email, along with my personal issues with Medium. I knew the post was controversial, which is almost always a good thing if you can handle it. I once again posted across many "mediums" (I'm funny), this time immediately posting to Dev.to.

I also decided to post to hacker news, which I had never done before. Many will probably argue with me, but I think Hacker News is one of the most negative platforms in existence. And because my story was negative in message, the post took off and was #1 on Hacker News for most of the day.

I accidentally posted to /r/javascript thinking it was /r/technology and was shocked to come back and see that it also did quite well.

At this point, I was primarily linking to my personal domain instead of Dev.to, here's the google analytics.

Dev.to

The Medium post did pretty well, but more importantly, it showed me the potential of Dev.to. From just the MyOS and Medium post I had nearly 500 followers on Dev.to. I'll be honest, my first guess was that Dev.to was inflating my follower count for some sort of social psychology.

I started looking into it and realized very quickly that nothing malicious was going on, in fact it was the opposite. Because Dev.to does not publicly display follower counts, readers tend to follow based on content they like, not who they think is popular. This resonated with me deeply, as it perfectly solves the issue I have with platforms like Twitter. On Dev.to what is being said matters much more than who is saying it.

At this point I was still very uncomfortable writing blogs and getting "out there". It may seem hard to believe, but I'm not a very extroverted person. Having overcome personal deficits like these in the past, I knew total immersion was a quick and reliable way to normalize yourself to an uncomfortable experience. In light of this, I decided I would do my best to post something every weekday for the next month. This was a serious undertaking considering I work fuller time at a startup already.

Blog Blog Blog Blog

Week One

The first week was easy, there was only one day left after the Medium post. I hadn't fully committed to the plan, so the post was essentially a discussion thread.

Week Two

Things ramped up during the second week. I was posting everything to my personal blog, but I was only giving out links to the Dev.to versions. This was a hard choice but it mostly stemmed from users existing familiarity with the Dev.to platform as opposed to my personal site. I predicted that people would be more willing to engage/interact if they already had an account.

I started experimenting with different types of content during this week. It was obvious that I had the technical chops to produce highly technical content, but it was much easier to produce high quality non-technical content with my limited time. You have to understand these are each full-fledged blog posts, they take a lot of time to write. I also started to get familiar with my content pipeline, each day I would post my new content to the relevant platforms.

// Detect dark theme var iframe = document.getElementById('tweet-1139572089440968705-314'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1139572089440968705&theme=dark" }

As you can see from the above tweet, my content start getting more engagement. This was a combination of a growing twitter following (was up to nearly 800 followers) and the added Dev.to traffic that I previously wasn't receiving.

Week Three

I was getting into a real rhythm by this point. As they say, the best way to get good at something is by doing it. Writing became less of a painful experience which translated into more time and better content.

The biggest win was the first post of the week which actually ended up being one of my best posts yet. I definitely attribute some of its success to my corresponding Reddit posts.

Note: If you try and self-promote on Reddit, you're going to have a bad time. I don't post everything to Reddit, and when I do I make sure it's in a digestible Reddit friendly format. I also made a rule that for every piece of content I posted to Reddit, I needed to comment on 5 other peoples posts and help/respond to them.

The post about disagreement also did quite well, I assume mostly due to this fortunate tweet by Dev.to on Twitter.

// Detect dark theme var iframe = document.getElementById('tweet-1141458887817777157-52'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1141458887817777157&theme=dark" }

The biggest surprise of that week was actually my post about K-Means. I really felt that it was higher quality writing/content than anything else so far, which led me to realize that there is essentially not a data-science community on Dev.to.

For those wondering why I did an AMA instead of a proper post on Jun 21, it's because it was a Friday. After looking over general statstics of content posted to Dev.to, I realized that Friday was practically a dead day, and therefore I could be more lax with what I posted. I also felt that the AMA was a nice way to establish credibility.

Week Four (The Final Week)

Week four started very strong. My first post of the week was about Elon Musk, Elon Musk isn't related to either of the topics that had been working for me so far. Regardless, I knew that it was niche content and had a lot of appeal because of how popular Tesla was.

A lot of writing successful content comes from understanding the topology of society and the internet. For example, because I knew Elon Musk has a semi-cult level following, content that positively depicted him might do quite well. Especially when posted to communities dedicated to that type of content.

I also learned that when someone is controversial like Musk, writing content about them will also bring that controversy back home.

// Detect dark theme var iframe = document.getElementById('tweet-1143270598799327233-477'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1143270598799327233&theme=dark" }

A Response to My Elon Musk Blog Post Tweet

I really was short on time for Tuesdays post. I actually ended up scrapping my entire post about 1 hour before I went to sleep and then writing something entirely new in a panicked rush. That post, "How to Write a Good Piece of Code" ended up being my most successful post by a HUGE margin. I attribute this almost entirely to the fact that it sat on the frontpage of /r/programming for over an entire day.

// Detect dark theme var iframe = document.getElementById('tweet-1143546719117824000-899'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1143546719117824000&theme=dark" }

The Tweet Also Did Very Well

Wednesdays post was special, or so I thought. It was a project I had been working on for a very long time, and walked through the process of building a simple AI (Multilayer Perceptron) only using Google sheets formulas. I use Google sheets all the time, and I figured it was something that nearly everyone can relate to. I was wrong. The post didn't do terrible, but not good enough to justify the number of hours I invested. This also confirmed my previous suspicion that there isn't a data science community on Dev.to.

Even by Thursday, "How to Write a Good Piece of Code" was still dominating my social media. Every morning I would wake up to tons of notifications from people sharing it. Partially due to this (and just being so exhausted), for the last 3 posts of the month (Thursday, Friday and Saturday) I kept things very casual.

June Twitter Stats

Outcome

The story I told is great, but what about the results? We'll start with Twitter, as the initial goal was to get more Twitter followers.

Over the course of 1.5 months I essentially went from 0 twitter followers to ~1250 Twitter followers. I have no real way to gauge this but I'm personally pretty satisfied. I also think, because I shifted my focus more towards blogging, it slowed my twitter growth.

The biggest gains were actually on Dev.to. In exactly 30 days of having a Dev.to account, I gained 3500 followers and accumulated ~54k views from scratch.

For any of you still wondering if Dev.to is a good place to get started, wonder no more. Lastly, this is the karma I gained on Reddit from posting my content and helping random redditors with tech questions.

Disclosure: I had never purchased "ads on the internet" before and was curious, so I spent ~$200 across Reddit, Twitter and Google trying out the respective ad platforms. Either ads suck or I suck at ads, they did very poorly and resulted in almost no increased traffic.

Takeaways

Some content isn't going to do well This is something that I 100% expected going into the process, which made it really easy to stay productive throughout. The biggest benefit of making yourself do this every day, is that you'll eventually post so much content that a few poorly received pieces aren't as meaningful. Often, those who are "afraid" to blog have a lot of anxiety that what they make will not be well received. Posting frequently allows you to solve this problem with scale. Yes some content won't work, no it's not because it sucks (maybe wrong audience, maybe wrong format).
It's not as scary as you think I won't lie and say that the blogging/social media process is something I enjoy. But I definitely don't feel the same stomach knotting I did when I started a month ago. I know a lot of people out there struggle like I did, but I promise its all in your head. As long as you go out there with a self-improvement mentality, and are willing to fix your mistakes, it will be a smooth ride.
Write Outlines: I recommend that you never start writing a blog post without at least a rough outline. Not only do outlines make it easier to write content, it also usually makes the final product a lot more concise and focused. I wasn't using outlines at the beginning of the month and things were rough.
You have to evangelize your content: I see so many great posts that never gain traction because the author doesn't share it adequately. I know a lot of people are uncomfortable with this (including me at the beginning of last month), but you really shouldn't be. If you spent the effort to make a quality product it's illogical to not at least put some effort into sharing it. You can see how much extra traction my posts got from me making sure it was visible in multiple platforms.
Dev.to is hot: If you're not already considering writing content on the site, you should be. The amount of growth I've seen in the month I've been here is astounding. I've also had many interactions with the Dev.to team and can safely say that they are some of the most reasonable, positive, level headed people you'll meet. I probably get on their nerves constantly and they've never complained. The community is also pretty non-toxic, but I fear that's simply a factor of its size.
Engage back: If you just write posts and don't respond to comments/reactions, you're missing out. I did my best to give a genuine response to every reasonable/non-troll comment I received. Yes, it's time consuming, but it also is highly visible and helps build a positive image of you in the community.
I can't post every day The month of June was ROUGH for me. I didn't have a single moment of free time. I was either working at my startup or writing content for the next days post. I would say that I was averaging 12-16 hours of just working, writing and editing, every day (including weekends). And that's almost entirely productive time. I eat only 1 meal a day, and I live 5 minutes from work, so very little overhead.

Wake up -> Work -> Go Home -> Work -> Go to Bed -> Repeat

What Dev.to Theme Are You Using?

Ryland G — Sat, 29 Jun 2019 15:30:09 +0000

Did you know Dev.to has themes? Until recently, I definitely didn't. Now I can't live without Dev.to night theme, it even turns the "Write A Post" button into an awesome street sign.

How to Use Dev.to Themes

Go to Settings
Go to Settings->Misc
Scroll Down and Choose Your Theme
Even Transfers to Mobile!

What Theme Are You Using?

How to Make a Button on Dev.to

Ryland G — Fri, 28 Jun 2019 14:32:54 +0000

If you've read any of my last few posts you may have noticed that I started including a twitter image at the end. You might not have realized that it was actually a fully functional button.

I hadn't seen anyone do this before. Luckily, Dev.to is open source, so it was easy enough to see that the markdown parser permitted a simple image button. I've included an example below.

<a href="https://twitter.com/<YOUR_TWITTER_HERE>" rel="some text">
![Foo]
(https://thepracticaldev.s3.amazonaws.com/i/gmrz82bjwhej1f1iqb1e.png)</a>

or (as suggested by @artemix)

[![](https://thepracticaldev.s3.amazonaws.com/i/gmrz82bjwhej1f1iqb1e.png)](https://twitter.com/<YOUR_TWITTER_HERE>)

That code (with your own twitter URL added) will display a follow button for your twitter in a post. Example below

My Dog is a Therapist

Ryland G — Thu, 27 Jun 2019 15:26:14 +0000

Cover Credit

Significantly more personal post today. Wanted to change things up.

This is an experience we had with our 1.5 year old border collie Audrey. It might be insightful if you struggle with anxiety.

Audrey and Anxiety

Audrey

We've put a ton of effort into training Audrey. She's had ~40 minutes of quality 1:1 training/games, nearly every day since we got her. She also gets 2 walks a day, plus a ton of other random attention. So you can understand our confusion when Audrey started developing anxiety issues a few months ago, especially while we were on walks.

Jeanie (my girlfriend) and I are both very anxious people. It's something we've both struggled with our entire lives. Obviously we're both aware and try very hard to improve the situation, but sometimes your mind has abstract ways of being anxious that are not apparent to "you".

We were devastated to watch as Audrey's anxiety worsened over a period of weeks. We had been passively working on it, but decided to start taking real conscious action to combat the problem. I noticed that if you were fully engaged with Audrey while you walked, her anxiety would dissipate. But if you were distracted on your phone or having a conversation, she would regress again.

I made a new rule for our walks with Audrey, if you were walking her, you needed to be fully engaged with her. I hoped, that given enough time, this would dampen the negative association her brain currently had with walking. Due to the nature of this rule, I spent a lot of time watching her over the next few weeks.

After a few weeks, I started to think there might be a pattern to her anxiety on the walks. Because she's a collie, she has a strong herding mentality. It's obvious to see how upset she gets when the group "isn't together". In fact, if the group separates, she refuses to lose sight of the other group members. So when I would be walking her, she would be constantly watching Jeanie to make sure we stayed together (and vice versa).

This made me curious. I decided to start observing Jeanie and Audrey's behavior simultaneously. It didn't take long to see that Jeanie's body language was incredibly indicative of if she was feeling anxious. She read like a bad poker player. And because dogs (as a species) have become incredibly adept at reading human body language, Audrey was seeing Jeanie from a similar perspective that I was. All of a sudden, I got it. Audrey's anxiety on the walks was a reflection of our anxiety on the walks.

If it isn't clear yet, here's an example. I noticed that Jeanie would walk through crosswalks much faster than she walked on the sidewalk. She was subconsciously anxious about the cars waiting to turn (cars held up by us pedestrians), and would increase her pace to satisfy that anxiety. Audrey, unable to understand the context of the anxiety, only saw that Jeanie was discontent and alert. We cross the street 20+ times per walk, and crossing the street is just one of many anxiety inducing scenarios. Compound all of these micro experiences over each walk twice a day and you have a dog with a serious anxiety issue.

Credit

Conclusion

After having this revelation, Jeanie and I started putting a ton of effort into reducing our own anxiety on the walks. This drastically improved the walks, nearly immediately. It also forced Jeanie and I to become far more aware of our own anxiety, and how we express it externally.

I strongly believe, that we pick up on those subtle indicators from other people, without even knowing it. This may drastically change the way you perceive someones behavior. Maybe there's someone who you just "don't like" and it's because your brain is subconsciously picking up on their anxiety. Maybe there's someone who doesn't like you for a similar reason.

Worry not, this is something that everyone can improve. Be mindful, and try to imagine someone is only able to understand things based on your body language. Would that be a pleasant experience? If not, how would you change to make the experience more pleasant?

And just in case you wanted to see Audrey in action, here's a video.

Spreadsheet Powered AI

Ryland G — Wed, 26 Jun 2019 16:03:42 +0000

TL;DR Multilayer perceptron that can be used to solve XOR problems only using spreadsheet formulas

How To Use the AI Sheet

Get Your Own Copy

Straight to business, here's a link that automatically makes a copy of the sheet, I've had issues with this method, so I'm still including the manual steps below.

Get Your Own Copy (Manual Method)

In case that link doesn't work, here's the main sheet (not a copy).. For obvious reasons, the main version is readonly. To play around with it, select File->Make a Copy.

That will give you your own version of the sheet.

Iteration Issues

Sometimes the iterations setting for sheets won't carry over to copies. To enable this setting go to File->Spreadsheet settings

Next, navigate to the Calculation tab and pick settings similar to these (feel free to play with max iterations, prepare for potential lag).

Running the Network

To get the network working, no ML knowledge is needed. The sheet works like a for loop, the "Current Iteration" cell D1 increments after each iteration of the network. Before incrementing, it checks the value of the "Target Iterations" cell B5. This is equivalent to the following code.

for (int current_iterations = 0; current_iterations < target_iterations; current_iterations++) {
  // run the network
}

So if you want the network to run, simply increase the value of target iterations. I would reccommend doing this in small increments (10 - 50) or the network can get out of wack. If the network does get out of wack, I've found copying cells B69:E78 then deleting and immediately pasting the cells back to be particularly effective.

I've also provided an "Update" button which simply increases the "Target Iterations" cell value by 10. You might need to enable the script on your copy of the sheet.

"Success" is when the "Current Predictions" G9:G15 match the "target result" E9:E15. At that point it has 100% accuracy with any XOR input (try moving the inputs around).

Explanation/Precursor Knowledge

If You Already Know About Neural Networks, Feel Free to Skip Ahead

Linear Separability

Data is said to be "linearly separable" if a single straight line can cleanly separate that data's features on a graph. For example, here is a graph of two distinct groups (red and blue) that are linearly separable (green line).

To separate data linearly, you would normally turn to an equation such as linear regression. But what if we wanted to work with data that can't be separated linearly (shown below)?

Algorithms such as SVM handle this problem by artificially adding dimensions to your data. This is done in the hope, that at higher dimensions, there exists a single line that can separate the data. But there's also another option, neural networks!

Perceptron

The simplest form of a neural network is the Perceptron (AKA the "artificial neuron"). Perceptrons take in n inputs, and give a single binary output, indicating whether the provided input was below or above a predefined threshold. Because of these principles, perceptrons are often used as binary linear separators.

But didn't we just say that we wanted to solve the separation problem for non-linear data? How can a perceptron be of use to us?

Multilayer Perceptron

Although a single perceptron can only separate data in a linear fashion, a few groups of perceptrons working together are able to accomplish the task of separating non-linear data. We refer to this collection of perceptrons working together as a MLP(multilayer perceptron). A multilayer perceptron consists of at least three layers (sets of perceptrons), the input layer, the hidden layer and and the output layer.

MLP is a supervised algorithm, which means it needs to be "told" when it correctly or incorrectly predicts something. MLP accomplishes this by having two distinct phases.

Feed-forward Stage: During this stage, the input perceptrons of the network are fed training input data. The input passes through each layer of perceptrons, eventually resulting in a single binary output that represents the "prediction" made by the network.

Back Propagation Stage: The output prediction from the feed-forward stage is compared to the expected output defined by the training set. If the predictions are correct, there is very little to be done. But if the predictions are incorrect, the network "propagates" the difference between the expected and actual results back through the network. This process gives the network a chance to "correct" the mistakes that caused the failed prediction, hopefully increasing the accuracy of subsequent runs. Usually, this is accomplished through the use of an optimization function such as the SGD (Stochastic Gradient Descent) algorithm.

XOR Problem

We need a non-linear problem that we can try and use a network to solve. To keep things simple, we'll attempt to build an "AI" that can correctly predict XOR when given a set of inputs. For those unfamiliar, XOR is a simple bitwise operator that is defined by the following truth table.

Input Bit 0	Input Bit 1	Input Bit 2	Expected Output
1	0	0	1
0	1	0	1
0	0	1	1
0	1	1	0
1	1	0	0
1	0	1	0
1	1	1	0

If there is more than 1 bit in a row of the XOR table, the expression is false. If there exists only a single bit (as with rows 1, 2, 3), the expression is true.

Getting the Sheet Ready

To start building the spreadsheet XOR predictor, we need to first copy the XOR training set from our table (above).

B9:E15

We also need to define some stateful variables to make sure our network can persist and learn. Specifically, there variables represent the perceptrons "weights" at each layer of the network.

B74:E78

Here's the formula from one of the above cells.

IF(ISNUMBER(B69), B69, (RAND() * 2) - 1)

The weights of the perceptrons need to start out as random values in a defined range of -1 to +1. This is easily accomplished using the (RAND() * 2) - 1 formula (RAND() generates a random number from 0 - 1).

The problem arises once we need to update the weights. We don't want them to be randomly set and therefore override the progress made by our network. IF allows us to start with random values, but once cell B69 has content, the formula will use the value of that cell (instead of generating randomly).

Feed Forward Stage

Input Activation Potentials B17:E23

Next, we will begin implementing the feed-forward stage of our network. The first step is to calculate "Input Activation Potentials" by taking the dot product of each row of the training input data B9:D15 and each column of the hidden perceptron weights B76:E78.

=ARRAYFORMULA(
 IF(
 AND(current_iteration < target_iterations,
  ISNUMBER(current_iteration)),
    { 
      MMULT({training_inputs, ones},
        TRANSPOSE(hidden_perceptrons)),
      ones
    },
    INDIRECT("RC", FALSE)))

The IF portion simply stops the network from looping infinitely, by checking the cell holding the current number of iterations.

IF(
AND(current_iteration < target_iterations,
 ISNUMBER(current_iteration)),

The arguments following the IF are evaluated based on the result of the branch. If...

current iterations == target iterations

we use INDIRECT("RC", FALSE), which is a comparable to using this in modern programming languages. Otherwise, we calculate the dot products using MMULT (matrix multiplication is just multiple dot products).

Hidden Layer Activators B25:E31

Next we need to calculate the activators of the hidden layer of the network. To compute the activators, we run a "activation function" on the potentials B17:E23. In practice there are many choices for your activator, in this case we will use the Sigmoid function.

Sigmoid Function 1 / (1 + exp(-x))

={
  ARRAYFORMULA(1 / ( 1 + EXP(-potentials))),
  potentials_bias
 }

Output Layer Activators B33:B39

Calculating the activations of our outputs is next. Unlike the hidden activations, we don't need to run our output potentials through the activator function. This makes calculating our output activations much easier.

=MMULT(
  hidden_activators, TRANSPOSE(output_perceptron)
 )

Just like we did for the hidden potentials, we take the dot product of the output perceptrons B74:E74 with the "inputs" (hidden activators B25:E31) to generate the initial output of the network.

Output Binary D33:D39

We now need to take the "raw" values emitted from the output layer B33:B39 and convert them into a binary representation. This is required, because we are trying to solve a XOR problem, where only boolean values are considered valid.

=IF(B33 < 0.5, 0, 1)

Each cell of the "Output Binary" D33:D39 checks it's partner cell in the "Output Activations" (what was calculated in the last step, cells B33:B39). If the output activation value is below 0.5, the binary result is 0. Otherwise, the binary result is 1.

Congratulations! Even though the current prediction is almost statistically guaranteed to be wrong, you can at least say a you got to the point where a prediction was made!

Backward Propagation Stage

This stage is where real "learning" takes place. Using the predictions and expected results, we calculate derivatives to adjust our weights.

Output Layer Delta F33:F39

To continue, we need to calculate the amount the output differed from the target results. This is accomplished by subtracting each target result (E9:E15) from it's corresponding output activator (B33:B39). The 2 is simply a scalar used to increase the significance of the difference.

=(B33 - E9) * 2

Hidden Layer Sum B41:E47

This next step is very similar to a portion of the feed-forward stage. We're just calculating dot products between the output deltas previously calculated and the current "Output Perceptron" weight values.

=MMULT(
  output_delta, output_perceptron
 )

Hidden Layer Deltas B49:E55

Next, we determine how much the hidden layer weights should be updated. This is calculated using our hidden activators B25:E31 and the derivative of the sigmoid function (the derivative of our activator in the feed forward stage).

Derived Sigmoid Function x * (1 - x)

=ARRAYFORMULA(
  (hidden_activators * (1 - hidden_activators)) * hidden_sum
 )

Output Change B57:E63, Output Change Average B65:E65 and Updated Output B67:E67

We want to calculate how much our output should change based on "Output Delta" F33:F39, our "Hidden Activators" B25:E31 and the "Learning Rate" B3. Due to a limitation of sheets, we have to do this in three discrete steps.

Output Change B57:E63

We want to calculate how much the output perceptrons should be changed. The "Output Delta" represents how much our prediction differed from the expected results. We multiply by the "Hidden Activators", because they represent the state that produced the most recent predictions.

"Learning rate" B3 is a scalar that controls how much each iteration influences the "learning" of the network. A high learning rate may sound enticing, but it introduces the possibility of overstepping a minimum and missing a convergence. Feel free to play around with learning rate, but take small steps.

=ARRAYFORMULA(
  ((output_delta * hidden_activators) * learning_rate)
 )

Output Change Avg B65:E65

For both clarity and practicality, it was easier to separate out the averaging of our changes into a separate step. Averaging is necessary because we are actually feeding the network 7 inputs at once B9:E15. That means we're actually dealing with 7 outputs at once, and averaging provides a way to consolidate that into a single value.

=SUM(dim0_output_change)

Updated Output B67:E67

Finally, subtract the calculated "Output Change Average" B65:E65 from the current values of the "Output Perceptron" B74:E74.

=ARRAYFORMULA(
  output_perceptron - output_weights_avg_change
 )

Updated Hidden B69:E72

We just updated the weights of our "Output Perceptrons" so now we need to update the weights for our "Hidden Perceptrons". Mirroring the structure used in the previous step, we use the "Hidden Deltas" B49:E55 the "Learning Rate" B3 and the "Test Input" B9:E15, to calculate how much our "Hidden Perceptrons" should change, based on the previous iteration of the network.

The TRANSPOSE({1,1,1,1,1,1,1}) represents our bias neurons, which function similarly to b in y = mx + b.

=ARRAYFORMULA(
  hidden_perceptrons - ARRAYFORMULA(
    (learning_rate * (
        MMULT(
          TRANSPOSE(hidden_delta),
          { 
            training_inputs, TRANSPOSE({1,1,1,1,1,1,1})
          }
        )
      )
    )
  )
 )

Instead of the 3 distinct steps we used to calculate the "Output Change", we keep this condensed to a single step.

Loop

At the beginning of the explanation, we described how our weights are initialized to random values (using the RAND formula). Specifically, the IF conditional referred to a set of cells, and if those cells were empty, RAND was the fallback. Now that we have calculated valid values for the "Updated Output" and "Updated Hidden", the IF no longer evaluates the RAND branch. Instead, it proxies those "Updated" values as its own.

Output Perceptron

=IF(ISNUMBER(B67), B67, (RAND() * 2) - 1)

Hidden Perceptron

=IF(ISNUMBER(B69), B69, (RAND() * 2) - 1)

And because our "Activation Potentials" B17:E23 depend on the "Hidden Perceptrons", any updates to the "Hidden Perceptrons" force a recalculation of the "Activation Potentials". This means, as long as "Current Iteration" D1 is less than "Target Iterations" B5, the network will loop.

Idea Origin

At a previous company, I led a technical team implementing high performance neural networks on top of our core product (scale-out distributed compute platform). During this time, I quickly learned, that for most people (including upper-level management), neural networks are indistinguishable from alien technology.

It became clear, that I needed a way to explain the basic mechanics behind deep learning, without explaining the intricacies of implementing a high performance neural network. At first I tried using simple network engines written in python, but this only made sense to those who were already python programmers. But then, I had an epiphany! I realized that it might be possible to build a simple neural network, entirely with spreadsheet formulas.

My Blog