Forem: Tommy May III

Typescript + ESLint and StandardJS

Tommy May III — Fri, 02 Aug 2019 03:16:18 +0000

Update 12-9-2019 - I have created an ESLint plugin that does the same thing as this article but should make your ESLint config super simple.
The package can be found here. Just follow the README instructions and you will be setup with the config described in this article.

Now that Typescript linting is moving from TSLint to ESLint it is time to adjust our configuration. If you are like me you probably love using StandardJS as your official style when writing Typescript/Javascript.

When I was using TSLint I was using a package that made using StandardJS easy called tslint-config-standard. Unfortunately, the setup for Typescript with ESLint is a little more involved so I thought I would share my config in hopes to save you a few hours of fumbling.

npm install --save-dev @typescript-eslint/eslint-plugin @typescript-eslint/parser eslint eslint-config-standard eslint-plugin-import eslint-plugin-node eslint-plugin-promise eslint-plugin-standard

This is a lot of dependencies so I'll explain each one.

@typescript-eslint/eslint-plugin - An ESLint-specific plugin which, when used in conjunction with @typescript-eslint/parser, allows for TypeScript-specific linting rules to run.
@typescript-eslint/parser - An ESLint-specific parser which leverages typescript-estree and is designed to be used as a replacement for ESLint's default parser, espree.
eslint - The actual tool that runs the linting rules.
eslint-config-standard - Configuration for StandardJS rules.
- eslint-plugin-import
- eslint-plugin-node
- eslint-plugin-promise
- eslint-plugin-standard All of the eslint-plugins under eslint-config-standard are required by that package, and I'm not sure why the package doesn't include them as dependencies... 😅

.eslintrc.js

module.exports = {
  'parser': '@typescript-eslint/parser',
  'parserOptions': {
    'project': './tsconfig.json', // Required to have rules that rely on Types.
    'tsconfigRootDir': './'
  },
  'extends': [
    'plugin:@typescript-eslint/recommended', // Out of the box Typescript rules
    'standard' // Out of the box StandardJS rules
  ],
  'plugins': [
    '@typescript-eslint' // Let's us override rules below.
  ],
  'rules': {
    '@typescript-eslint/no-use-before-define': 'off', // Allows us to hoist variables and functions which I am a fan of, functions not variables that is.
    '@typescript-eslint/no-explicit-any': 'off', // Too strict for my case, sometimes I need an any type
    '@typescript-eslint/member-delimiter-style': ['error', { // Prevents us from using any delimiter for interface properties.
      'multiline': {
        'delimiter': 'none',
        'requireLast': false
      },
      'singleline': {
        'delimiter': 'comma',
        'requireLast': false
      }
    }],
    '@typescript-eslint/indent': 'off', // This is the job of StandardJS, they are competing rules so we turn off the Typescript one. 
    'no-unused-vars': 'off', // On the fence about using this one, sometimes we import a package that is never used directly. 
    'node/no-unsupported-features/es-syntax': 'off' // Allows us to use Import and Export keywords.
  }
}

For the curious here is my tsconfig.json

{
  "compilerOptions": {
    "alwaysStrict": true,
    // TODO Needed because of Apollo https://github.com/apollographql/apollo-server/issues/1182, this should be considered a bug with Apollo
    "allowSyntheticDefaultImports": true,
    "skipLibCheck": true,
    "outDir": "dist",
    "module": "commonjs",
    "declaration": true,
    "sourceMap": true,
    "target": "es6",
    "rootDir": "./src",
    "lib": [
      "es5",
      "es6",
      "es7",
      "esnext.asynciterable"
    ],
    "experimentalDecorators": true,
    "emitDecoratorMetadata": true,
    "strict": true
  },
  "include": [
    "src/**/*.ts"
  ],
  "exclude": [
    "node_modules",
    "**/*.spec.ts",
    "dist"
  ]
}

Hope this config helps or saves you some time, any follows on twitter is much appreciated @itmazyiii

Removing React from Gatsby

Tommy May III — Tue, 02 Jul 2019 02:01:25 +0000

Recently with help from my coworkers I released 2 new Gatsby plugins that make removing React from Gatsby a lot easier. Your probably screaming through your monitor right now WHY WOULD YOU REMOVE REACT and I want to point you to another post of mine. This article explains a lot of trouble we went through to make the bundle sizes smaller for our App but at the end of the day these optimizations were not enough for our requirements. The company that I work for demands a very fast website that performs well on older mobile devices and React is simply too much Javascript for our needs.

Anyways here is the 2 plugins.

Remove Javascript Plugin - Does what it says and removes the Javascript files that Gatsby generates but not all javascript.
Add Webpack Entry - Adds a webpack entry point so that you can still include some javascript on your page that is not React. This entry point goes through the Gatsby webpack configuration as normal and will be auto hashed and added to your HTML.

I would encourage anyone who wants to try these plugins out to first read the NPM page. With the combination of these 2 plugins we were able to reduce our Javascript footprint on our pages from 190KB to around 20KB which resulted in a huge performance boost.

Sidenote

One nice benefit of removing React is that your React code simply becomes your view layer that runs only on the server and all your components become simple functions with no state, lifecycle, or event management. I do not recommend using these plugins unless your javascript bundle size is super important to you These plugins go against how Gatsby wants you to use Gatsby (although these plugins are not hacks and simply implement Gatsby's APIs).

Better Performance using Dynamic Code Splitting in Gatsby with loadable-components

Tommy May III — Thu, 06 Jun 2019 03:50:57 +0000

Preface

I use Gatsby at work and in my personal projects because I believe it is the best tool out there right now in terms of efficiency as a developer and value added to my clients. The thing that keeps me using Gatsby is they really focus on performance and we all know performance matters when it comes to retaining users. As amazing as Gatsby is, it does not fully take performance off of our plate so we never have to worry about it again. As developers we should be testing the speed of our websites after every code and content change because no tool is going to handle every edge case in the world for us. Gatsby and websites in general are fast out of the box, but it is our job not to mess it up. In this post I want to share with you a case where Gatsby itself was not enough to handle our performance requirements and how we tackled the issue by constantly testing and making incremental changes.

The Performance Issue We Were Facing

At my work we primarily use 2 testing tools to measure our website performance.

In Lighthouse our website was scoring in the mid 70s (out of 100) and two of the things that were pointed out to improve were

Reduce JavaScript execution time
Minimize main-thread work

In Web Page Test our website had a very high time until the page was considered fully loaded and high load times are bad. I say "high" subjectively compared to the performance we were accustomed to seeing for the same exact website. An interesting thing about this Web Page Test tool is that you can block certain HTTP requests from happening which is a really handy way to test whether or not the presence of a certain request is the cause of performance issues. It turns out after blocking the gatsby generated javascript files on the page our website load time was cut in half!

The conclusion we drew from both of these testing tools was that the downloading, parsing, and execution time for our javascript scripts was too high.

Understanding Why Gatsby was Failing Us

In truth Gatsby did not fail us, but the out of the box solution that Gatsby provides for code splitting did. Gatsby provides a very in depth article to how they handle code splitting here so I'm not going to spend a lot of time going over it.

Dynamic Pages are the Real Issue

We are using Gatsby I believe in a very unique way where we have a custom CMS / design system feeding Gatsby data to create static pages with. Our CMS breaks up pages into different sections that we call modules.

The red lines separate what we call a module on our website and content writers in our CMS can compose a page of any of these modules which means on the Gatsby side we have to have code like this:

export default function Page ({pageFromCMS}) {
  return pageFromCMS.modules.map((module) => {
    const Module = findModuleComponent(module.id)
    return <Module module={module}/>
  })
}

This is not the real code but it very much illustrates what we are trying to accomplish. The idea is that we just want to take the modules that the CMS has for any given page and loop over them to dynamically put them on the page.

The issue with this code is that inside the function above called findModuleComponent we have to do something like:

import ModuleOne from './module-one'
import ModuleTwo from './module-two'

const modules = {
  'moduleOne': ModuleOne,
  'moduleTwo': ModuleTwo
}

export function findModuleComponent (moduleId) {
  if (!modules.hasOwnProperty(moduleId)) {
    throw new Error(`Module ${moduleId} does not exist`)
  }

  return modules[moduleId]
}

Do you spot the issue here and how it relates to code splitting from the title of this article?

Basic Understanding Code Splitting

If you have two import statements at the top of a file Gatsby / Webpack is going to bundle those imports into one javascript file during the build, and make something like https://www.dumpsters.com/component---src-templates-page-js-123eb4b151ebecfc1fda.js.

Bringing It All Together

Our requirements for our CMS to have any module on any page forces us to dynamically render the modules on the Gatsby side. In order to dynamically render any module we have to have a map of module names to react components which forces us to import all of our react components in the same file. The act of having all of these imports in the same file makes Gatsby/Webpack think that every module/import is needed on every single page so there is essentially no code splitting at all for our page specific code. This is a real problem because we could easily have 100 total modules and any given page probably only uses 10 of them so we have a lot of unneeded javascript on our pages.

Solving the Issue

We need a way to only import the modules that we need for any given page without sacrificing the dynamic nature of our CMS. Introducing dynamic imports mentioned by react and also Webpack. The issue with the dynamic imports right now is that it relies on React.lazy which does not support server side rendering. We absolutely need server side rendering, it is another big reason we chose to use Gatsby to statically render our HTML pages. React themselves acknowledges this limitation of React.lazy and they recommend using loadable components to address the issue for now.

Implementing Loadable Components in Gatsby

If you follow the documentation for loadable components you will probably quickly get confused when you get to the third step which is about how to set up the server side of your application. This step is confusing because Gatsby already takes care of these things for you! Gatsby itself is in charge of doing the server rendering and you will not need to override it to make loadable components work. Instead if you just follow the first 2 steps in the documentation then it will be enough to get started.

Step 1

You will need to use a custom babel plugin so you need to overwrite the Gatsby default one as described here.

.babelrc

{
  "plugins": [
    "@loadable/babel-plugin"
  ],
  "presets": [
    [
      "babel-preset-gatsby",
      {
        "targets": {
          "browsers": [">0.25%", "not dead"]
        }
      }
    ]
  ]
}

make sure to install @loadable/babel-plugin and babel-preset-gatsby

Step 2

You will need to add a custom webpack plugin.

gatsby-node.js

const LoadablePlugin = require('@loadable/webpack-plugin')
exports.onCreateWebpackConfig = ({ stage, getConfig, rules, loaders, plugins, actions }) => {
  actions.setWebpackConfig({
    plugins: [new LoadablePlugin()]
  })
}

again make sure to install @loadable/webpack-plugin and @loadable/component

Changing Our Code

Now that we have loadable components lets use its dynamic import abilities.

import loadable from '@loadable/component'

export default function Page ({pageFromCMS}) {
  return pageFromCMS.modules.map((module) => {
    const moduleFileName = findModuleFileName(module.id)
    const ModuleComponent = loadable(() => import(`../modules/${moduleFileName}`))
    return <ModuleComponent module={module}/>
  })
}

If we stopped now we would be most of the way there with code splitting happening at the module level and therefore we are not including a bunch of unneeded javascript on our pages. There is an issue with code like this though.
What will happen is:

The static HTML will render to the user.
React will hydrate itself onto the static HTML
Your current DOM will be destroyed by React because it takes time for the dynamic import to resolve
The modules will be added back to the page once the dynamic import actually loads the javascript file it needs.

This has a nasty effect of having content on the screen, it disappearing, and then reappearing which is a terrible UX. In order to solve this issue we did something clever/hackish (I'll let you decide). Essentially the loadable components library allows you to specify fallback content as a prop until it is able to load the javascript file. We don't want to use a loading spinner because that is still going to flash content, instead we know the HTML is already statically rendered on the page so we grab the HTML for that module with a document.querySelector and then specify it as the fallback content until the module's javascript has loaded.

This post is getting sort of long so I'm going to share will you some psuedo code / real code of the final solution.

import loadable from '@loadable/component'

return page.modules.map((module, index) => {
  const { moduleFileName, shouldLoadJavascript } = retrieveModulePath(module.id)
  if (isServer()) {
    // The server should always render the module so we get the static HTML.
    // RENDER YOUR MODULE
  }

  const wasUserPreviouslyOnSite = window.history.state
  const htmlEl = document.querySelector(`[data-module-index="${index.toString()}"]`)
  if (htmlEl && !shouldLoadJavascript && !wasUserPreviouslyOnSite) {
    // These modules do not require javascript to work, don't even load them
    // RENDER THE STATIC HTML ONLY HERE - something like <div dangerouslySetInnerHTML={{ __html: htmlEl.outerHTML }}></div>
  }

  const fallback = htmlEl && htmlEl.outerHTML ? <div dangerouslySetInnerHTML={{ __html: htmlEl.outerHTML }}></div> : null
  // RENDER THE MODULE NORMALLY HERE WITH THE FALLBACK HTML SPECIFIED
})

The above code accomplishes a lot of different things for us:

Dynamic importing code for better code splitting
Allows us to opt into not importing code at all for modules that don't need JS to work.
Prevents any flash of content from happening.

Conclusion

Sometimes you have to go beyond what our tools offer us out of the box and that is okay. Gatsby is an excellent tool I plan on using for a long time but it needed some super powers added to it with loadable components. We saw a total of about 200KB of javascript removed from our site when we implemented something like this code and yes we have seen improvements in our page speed when using lighthouse and web page test.

I know I left some of the code above open ended but I really can't share much more since it is a company project. Feel free to reach out to me though if you have questions and I will guide you as much as I can without handing you the word for word solution.

Any follows on dev.to and twitter are always appreciated!

Cover Photo by José Alejandro Cuffia on Unsplash

How to Process Epic Amounts of Data in NodeJS

Tommy May III — Wed, 30 Jan 2019 03:57:20 +0000

Preface

If you are like me then you like to use NodeJS for a lot of different things like handling HTTP requests, making CLI tools, IoT, and so on. You have likely ran into situations where you needed to send off multiple HTTP requests at the same time, and if you haven't then don't worry because one day you will. Using javascript's asynchronous nature, most experienced developers will be able to send off a couple of HTTP requests at the same time. BUT what happens when you need to send millions of HTTP requests? This problem is likely to trip up even seasoned javascript developers because it touches a problem that most people don't have to deal with too often which is handling big data.

You may have guessed that if you try to asynchronously send 1 million HTTP requests then your program is going to crash and your guess would be correct. In fact your program would likely crash way before 1 million HTTP requests. Just because something is asynchronous doesn't mean it can handle an infinite amount of data. In the rest of this article I hope to show you how to handle data of any size in an efficient manner that will never cause you to run out of system resources. We will be using NodeJS Streams which is our secret sauce, so if you need a guide to streams then this is my favorite article. Unlike that article I don't plan on diving into how streams work except at a high level, instead my goal is to give you a practical example of handling big data using streams.

Straight To The Finished Code

If your in a hurry or don't care to read then here is the finished Github Repository of what we will be building.

itmayziii / dev-to-nodejs-epic-data

dev.to article - How to Process Epic Amounts of Data in NodeJS

View on GitHub

What We Will Build

We will be reading a list of Github usernames from a file
With each Github username we want to call the github API and get a list of their repositories. We will only be working with a smaller list of 12 usernames because I don't want a bunch of readers spamming Github's APIs and because the concept is the same for any amount of data.
Write this data to our database, but to avoid setup complexity for this step we will just be writing the data to a file.
Finally we will refactor to make things more performant.

We will be doing all of this using NodeJS Streams, which if done correctly have the concept of backpressuring which is what helps us use NodeJS resources in a way that won't run out of memory.

1. Reading the File Of Github Usernames

You can find the file in the example file in the repository

src/main.js

const fs = require('fs')
const path = require('path')
const CsvParser = require('csv-parse')

const readGithubUsernamesStream = fs.createReadStream(path.resolve(__dirname, '../github-usernames.csv'))
const csvParser = new CsvParser({})

let githubUsernames = ''
readGithubUsernamesStream
  .pipe(csvParser)
  .on('data', (data) => githubUsernames += data)
  .on('end', () => console.log(githubUsernames))

// Outputs - itmayziii,dhershman1,HetaRZinzuvadia,joeswislocki,justinvoelkel,mandarm2593,mfrost503,wmontgomery,kentcdodds,gaearon,btholt,paulirish,ryanflorence

2. Get List of Repositories From Github

It was nice that NodeJS provided us with createReadStream to read our file as a stream, but now we will need our own stream to take that list of usernames, read it, and transform it into github results.

For this step we will be using axios to make HTTP requests

src/transform-username-to-github-repos.js

const axios = require('axios')
const stream = require('stream')

module.exports = class TransformUsernameToGithubRepos extends stream.Transform {
  constructor (options = {}) {
    super({ ...options, objectMode: true })
  }

  _transform (chunk, encoding, callback) {
    const username = chunk
    this.getGithubRepositoriesForUser(username)
      .then((response) => {
        let repositories = []
        if (response.data) {
          repositories = response.data.map((repository) => repository.name)
        }

        this.push(JSON.stringify({
          username,
          repositories
        }))
        callback()
      })
      .catch(callback)
  }

  getGithubRepositoriesForUser (username) {
    return axios.get(`https://api.github.com/users/${username}/repos`, {
      headers: {
        Authorization: `Token ${process.env.GITHUB_ACCESS_TOKEN}`
      }
    })
  }
}

and change our src/main.js

const fs = require('fs')
const path = require('path')
const CsvParser = require('csv-parse')
const TransformUsernameToGithubRepos = require('./transform-username-to-github-repos')

const readGithubUsernamesStream = fs.createReadStream(path.resolve(__dirname, '../github-usernames.csv'))
const csvParser = new CsvParser({ columns: false })
const transformUsernameToGithubRepos = new TransformUsernameToGithubRepos()

let githubUserRepositories = []
readGithubUsernamesStream
  .pipe(csvParser)
  .pipe(transformUsernameToGithubRepos)
  .on('data', (data) => githubUserRepositories.push(data))
  .on('end', () => console.log(githubUserRepositories))

We changed a lot of things right there so let's unpack. We created a Transform stream that has a _transform method on it. When we pipe our CSV file to this transform stream this _transform method will be called. Once the _tranform method has been called with the username passed to it, we take the username and ask github for all the repositories for that user. We then send our results to the next thing in the stream with this.push(...). We don't have a next step in the steam pipeline yet so we start listening to the data event that where we collect the data and log in main.js.

3. Write Our User Repositories to a File

src/main.js

const fs = require('fs')
const path = require('path')
const CsvParser = require('csv-parse')
const TransformUsernameToGithubRepos = require('./transform-username-to-github-repos')

const readGithubUsernamesStream = fs.createReadStream(path.resolve(__dirname, '../github-usernames.csv'))
const csvParser = new CsvParser({ columns: false })
const transformUsernameToGithubRepos = new TransformUsernameToGithubRepos()
const writeStream = fs.createWriteStream(path.resolve(__dirname, '../github-user-repositories.txt'))

let githubUserRepositories = []
readGithubUsernamesStream
  .pipe(csvParser)
  .pipe(transformUsernameToGithubRepos)
  .pipe(writeStream)
  .on('end', () => process.exit())

This was an easy step, we just created a write stream write things into a txt file.

4. Refactor

What we have works, but it is far from ideal. If you look at the code it is terribly inefficient.

It only does one HTTP request at a time, just because we can't do 1 million HTTP requests at the same time doesn't mean we can't do maybe one hundred. For this example we will limit to 5 per pipeline walkthrough for demonstration purposes.
Code also has poor error handling

Lets go ahead and fix these things, starting with multiple HTTP requests per pipeline walkthrough

src/transform-username-to-github-repos.js

const axios = require('axios')
const stream = require('stream')

module.exports = class TransformUsernameToGithubRepos extends stream.Transform {
  constructor (options = {}) {
    super({ ...options, objectMode: true })
    this.requests = []
  }

  _transform (chunk, encoding, callback) {
    const username = chunk[0]
    const githubRequest = this.getGithubRepositoriesForUser(username)
    this.requests.push(this.prepareGithubRequest(username, githubRequest))
    if (this.requests.length < 5) {
      return callback()
    }

    this.processRequests(callback)
  }

  _flush (callback) {
    this.processRequests(callback)
  }

  getGithubRepositoriesForUser (username) {
    return axios.get(`https://api.github.com/users/${username}/repos`, {
      headers: {
        Authorization: `Token ${process.env.GITHUB_ACCESS_TOKEN}`
      }
    })
  }

  prepareGithubRequest (username, githubRequest) {
    return githubRequest
      .then((response) => {
        let repositories = []
        if (response.data) {
          repositories = response.data.map((repository) => repository.name)
        }

        return {
          username,
          repositories
        }
      })
  }

  processRequests (callback) {
    return Promise.all(this.requests)
      .then((responses) => {
        this.requests = []

        this.push(responses.reduce((accumulator, currentValue) => {
          return accumulator + JSON.stringify(currentValue)
        }, ''))
        callback()
      })
      .catch(callback)
  }
}

Again we just did a lot so let's upack what happened. We changed our _tranform method to call the Github API and then shove the promise into an array, we then continue on if the total accumulated promises are less than 5. Basically we are calling Github 5 times before we ever tell the transform to push data through the stream which can be found in the method processRequests. We have successfully changed our pipeline to process 5 requests each time instead of 1 which is a huge performance gain.

We could imagine if we were processing 1 million records and instead of 5 as our number we used 100, so we would send 100 HTTP requests near the same time and wait for them all to resolve before sending 100 more. This is a really efficient / resource saving way of processing large amounts of data.

We are not done though, we still need to have better error handling, for this we will take advantage of the native NodeJS pipeline function.

pipeline - A module method to pipe between streams forwarding errors and properly cleaning up and provide a callback when the pipeline is complete.

src/main.js

const fs = require('fs')
const path = require('path')
const CsvParser = require('csv-parse')
const TransformUsernameToGithubRepos = require('./transform-username-to-github-repos')
const stream = require('stream')

const readGithubUsernamesStream = fs.createReadStream(path.resolve(__dirname, '../github-usernames.csv'))
const csvParser = new CsvParser({ columns: false })
const transformUsernameToGithubRepos = new TransformUsernameToGithubRepos()
const writeStream = fs.createWriteStream(path.resolve(__dirname, '../github-user-repositories.txt'))

stream.pipeline(
  readGithubUsernamesStream,
  csvParser,
  transformUsernameToGithubRepos,
  writeStream,
  (error) => {
    if (error) {
      console.error('error ', error)
      return process.exit(1)
    }

    process.exit()
  }
)

Conclusion

NodeJS streams allow us to effectively have a pipeline from which data starts at one point and flows through until the end. Using backpressuring, which comes from simply implementing NodeJS's already built streams, we efficiently use a computer's resources while processing very large sets of data. I know methods like this work because I've used it to process over 10 million records from a CSV, call an API to get additional data, and then store the results in a database, much like we did in this article. Streams are effective on their own, but if you really wanted to speed things up I would consider combining child processes with streams for maximum efficiency.

Cover photo Credit - Jonathan Kemper on unsplash

My 4 Year Journey With Testing Software

Tommy May III — Tue, 27 Nov 2018 03:25:31 +0000

I just want to preface this post with saying that this whole post is just my experience and will give you no practical advice or opinions.

4 Years Ago I Could Not Assert 2 + 2 = 4

Four years ago when I was 20 years old I was given my first software development job. Not only was this my first big job but it was at a fortune 500 company where I was surrounded by experts so I never felt like I really knew anything in comparison. Luckily for me these experts were amazing mentors and I owe most of what I know to these people, BUT this didn't stop me from always feeling like I didn't know enough. I remember the first time one of the senior devs paired with me and showed me his tests It felt like this already massive amount of knowledge I was lacking tripled in size. I had a lot of questions

Why write tests? They seem like they take a long time.
How do you know what to test?
What in god's name is a mock or a stub?
Why would I need to mock something, doesn't that defeat the purpose of a test?
What is the difference between unit, behavior, and integration tests?
What is black/white box testing?
Now I'm testing a react component, is it inappropriate to simulate a click event in a unit test?
I'm scared I will spend all this time writing tests that no one will run except me, should I still write them?

This list could go on forever really. The idea of writing tests seemed so foreign, I mean I was still learning how to write code, I didn't have time to learn this nuance that is testing.

3 Years Ago I Avoided Tests Like the Plague

Having spent many hours of many days trying to learn how to write tests I concluded that testing was not for me and that it slowed down the software development process too much to be valuable. My team didn't have any hard requirements like the code must have 70% test coverage or anything silly like that so it was fairly easy to avoid. Although I was happy avoiding testing I couldn't help but feel that every time QA would find a bug that maybe I could have avoided that bug if I had written tests. It was worse when something I knew to be working before would get a bug filed a couple months down the road saying it no longer worked. It felt like I would write a feature and break 2 others, and I wouldn't know it until QA or a user would tell me about it. Having broken many things I really started to fear making changes. Not feeling confident in my code was and is the worst feeling in software I have had.

2 Years Ago I Wrote Tests But They Sucked

I had come a long way in my journey to learn how to test. Let me try and rephrase that... I had come a long way in my journey to write tests that were fragile. I was no longer scared to write tests, especially unit tests. I knew how mocks worked and I would mock anything in the world if it meant I could isolate my test in it's own little tiny sandbox. My solution was to unit test everything in order to conquer my fear of making changes. This led to me wasting a lot of time refactoring tests over and over because an implementation would change and all of a sudden I had to change 6 test files because they relied on that implementation. My productivity was at an all time low, even though I was writing more tests and having less bugs, but there had to be a better way.

1 Year Ago I Stopped Having Hard Opinions And My Life Was Blissful

I had given up on the idea that everything needed to be unit tested. In fact my opinions were quite the opposite, and instead I looked for the right test for the job. There was no need to unit test a controller that relied on 8 other classes to be injected in the constructor, this would only lead to a fragile test, instead this would be better suited for an integration test. I was beginning to instinctively understand what kind of tests would be most appropriate.

0 Years Ago (Today 😉)

I teach other developers how to test, and I feel extremely confident in the tests that I write. My tests rarely break for the wrong reasons, and I spend more time developing than worrying about how I'm going to test this feature. My biggest problem is implementing build systems that make sure tests pass before code is merged.

I wrote this article because I want junior developers to know that your not alone if you find testing difficult for a long time. Software development is hard and anyone that says otherwise is either a genius or naive (most likely the latter.) It doesn't help that everyone has their own opinions when it comes to testing and it seems that every article you read contradicts the last article. Just know that your not alone if you feel this way, and you will one day conquer those tests and feel proud!

Maybe one day I'll write about my own opinions when it comes to testing.

Step-by-Step Guide to Setup Node With Docker

Tommy May III — Sun, 18 Nov 2018 17:59:59 +0000

Learn how to setup Node JS inside of a Docker container

Goals of this article

Have a working NodeJS application
Make the Node app resilient by ensuring the process does not exit on error
Make the Node app easy to work with by auto restarting the server when code changes
Utilize Docker to:
- Quickly set up a development environment that is identical to production.
- Easily be able to switch Node versions both locally and on a server
- All the other benefits of Docker

Prerequisites

Docker already installed
At least entry level Knowledge of Node and NPM

If you are the type of person that just wants to see the end result then maybe the github repo will suit you better

itmayziii / dev-to-node-docker-setup

Repo for the dev.to article https://dev.to/itmayziii/node--docker-setup--3nc5-temp-slug-2056229

View on GitHub

1. Get a simple Node app in place

We are going to use Express because of how easy it is to set up and the popularity of the framework.

In a clean directory let's start by initializing NPM, go ahead and run this command and follow the prompts (what you put in the prompts is not that important for this guide)

npm init

Install Express

npm install --save-prod express

Setup basic express server. The file below simply says to start a Node process that listens to port 3000 and responds with Hello World! to the / route.

src/index.js

const express = require('express')
const app = express()
const port = 3000

app.get('/', (req, res) => res.send('Hello World!'))

app.listen(port, () => {console.log(`Example app listening on port ${port}!`))

2. Setup Docker to run our Node app

We will be using a docker-compose.yml file in order to start and stop our Docker containers as opposed to typing long Docker commands. You can think of this file as a config file for multiple Docker containers.

docker-compose.yml

version: "3"
services:
  app:
    container_name: app # How the container will appear when listing containers from the CLI
    image: node:10 # The <container-name>:<tag-version> of the container, in this case the tag version aligns with the version of node
    user: node # The user to run as in the container
    working_dir: "/app" # Where to container will assume it should run commands and where you will start out if you go inside the container
    networks:
    - app # Networking can get complex, but for all intents and purposes just know that containers on the same network can speak to each other
    ports:
    - "3000:3000" # <host-port>:<container-port> to listen to, so anything running on port 3000 of the container will map to port 3000 on our localhost
    volumes:
    - ./:/app # <host-directory>:<container-directory> this says map the current directory from your system to the /app directory in the docker container
    command: "node src/index.js" # The command docker will execute when starting the container, this command is not allowed to exit, if it does your container will stop

networks:
  app:

Now we have our config in place, let's start the docker container with this command. This just means start the containers defined in our config file and run them in the background (-d)

docker-compose up -d

Now you should be able to go to localhost:3000 in your browser and see Hello World!

You should also be able to verify that your container is running by running

docker ps

which should output the list of your running docker containers, something like

Useful docker commands for managing this container

List all running containers

docker ps

List all containers regardless of if they are running

docker ps -a

Start containers from a docker-compose.yml file in the same directory

docker-compose up -d

Stop containers from a docker-compose.yml file in the same directory

docker-compose stop

Restart containers from a docker-compose.yml file in the same directory

docker-compose restart

See the log files from your docker container

docker-compose logs -f

3. Make our application resilient

If you have worked with Node before then you probably know that if an error occurs in your application, like an uncaught exception, then it will shut down that Node process. That is *really bad news for us because we are bound to have a bug in our code and can not ever guarantee our code is 100% error free. The solution to this problem is usually another process that watches our Node app and restarts it if it quits. With so many solutions out there like linux's supervisord, the NPM package forever and PM2, etc... we will just need to pick one for this guide.

I am going to focus on PM2 since I am most familiar with it and it also comes with some other features besides process management such as file watching which will come in handy for our next section.

Install PM2

npm install --save-prod pm2

PM2 can be used through the command line but we are going to set up a simple config file much like we did with the docker-compose.yml file in order to prevent us from typing long commands repeatedly

ecosystem.config.js

const path = require('path')

module.exports = {
  apps: [{
    name: 'app',
    script: 'src/index.js', // Your entry point
    instances: 1,
    autorestart: true, // THIS is the important part, this will tell PM2 to restart your app if it falls over
    max_memory_restart: '1G'
  }]
}

Now we should change our docker-compose.yml file to use PM2 to start our app instead of starting it directly from index.js.

docker-compose.yml (Only changed the command option)

version: "3"
services:
  app:
    container_name: app # How the container will appear when listing containers from the CLI
    image: node:10 # The <container-name>:<tag-version> of the container, in this case the tag version aligns with the version of node
    user: node # The user to run as in the container
    working_dir: "/app" # Where to container will assume it should run commands and where you will start out if you go inside the container
    networks:
    - app # Networking can get complex, but for all intents and purposes just know that containers on the same network can speak to each other
    ports:
    - "3000:3000" # <host-port>:<container-port> to listen to, so anything running on port 3000 of the container will map to port 3000 on our localhost
    volumes:
    - ./:/app # <host-directory>:<container-directory> this says map the current directory from your system to the /app directory in the docker container
    command: "npx pm2 start ecosystem.config.js --no-daemon" # The command docker will execute when starting the container, this command is not allowed to exit, if it does your container will stop

networks:
  app:

It should be noted that changing your docker-compose.yml file will not affect already running containers. In order for your changes to take place you should restart your containers

docker-compose restart

Great we should now be back to a working app at locahost:3000 but now our app will not fall over when we have errors.

4. Make our application easy to develop on

You may have noticed that once a Node process has started then changing the code does not actually do anything until you restart that Node process, and for us that would involve restarting our Docker containers every time we make a change. Ewwwwwwwww that sounds awful. It would be ideal if we could have our Node process restarting for us automatically when we make a code change. In the past I have done things like bring in a file watching utility and using that file watching utility to restart Docker on file changes, or I would use Nodemon but that comes with some caveats when using Docker. Recently I have been using PM2 to restart my Node process when a file changes, and since we already have it pulled in from the previous step we won't have to install another dependency.

ecosystem.config.js (only added the watch option)**

const path = require('path')

module.exports = {
    apps: [{
        name: 'app',
        script: 'src/index.js',
        instances: 1,
        autorestart: true,
        watch: process.env.NODE_ENV !== 'production' ? path.resolve(__dirname, 'src') : false,
        max_memory_restart: '1G'
    }]
}

The config file above will now watch the src directory if we do not have the NODE_ENV environment variable set to production. You can test it out by changing your index.js file to print something else to the browser besides Hello World!. Again before this can work you need to restart your Docker containers, since you changed how PM2 is running the containers

docker-compose restart

It should be noted that restarting the Node process may take a second to finish up, if you want to watch to see when it is finished you could watch your Docker logs to tell when PM2 is done restarting your Node Process.

docker-compose logs -f

You will see something like this when your process has restarted

Wrapping Up

One of our goals was to be able to easily change Node versions, you can do this by changing the image option in the docker-compose.yml file.
Installing dependencies locally is done with your local NPM and Node version which can cause conflicts sometimes if your local versions are different than Dockers. It is safer to use the same Docker container to install your dependencies. You can use this command which will use that container to install dependencies and then remove it

docker run --rm -i -v <absolute-path-to-your-project-locally>:/app -w /app node:10 npm install

As mentioned above having a different local version of Node than what Docker is running could be problematic. It is best to run commands inside of your container for consistency. You can go inside a container with

docker exec -it app bash

The above command will put you inside the container so you can continue to run your commands from there i.e. npm run start or npm run test

If you prefer not to go inside the container you could run the commands like this

docker exec -t app bash -c "npm run start"