Forem: DEBOJIT ROY

Why do we need a Software Engineering Process

DEBOJIT ROY — Fri, 26 Jun 2020 01:19:51 +0000

This was originally posted here.

Why do I need a Software Engineering Process

I remember when I started as a developer, I used to get so bugged writing unit tests and filling documents. I was one of the developers in a team of hundreds of developers, building a mammoth software which needed a monster server to run. For even a few line of changes, I had to write lengthy discourse about what was the change, the thinking behind the change and how I believed this will not impact the performance of the application. Then I had to add unit tests, followed by a series of regression tests. And before even sending the change for review, had to run regression tests for my part of the application and smoke tests for the whole application. What an overkill !!! At least that's what I used to think. But later I realised how all those steps were a necessary evil, and understood the importance of the process when I had my first production bug 🤦

Building software is hard, if anyone think differently I would like you to reconsider. There are many good tools out there to make our life easy as developers, but sadly we are left on our own when it comes to using the tools. There are guidelines and documentation on how to use the tools, but there are very few guidelines on how to build software the right way. Most of us learn by trial and error, learning from our mistakes. But I want you to learn from my mistakes and maybe have less heartburn next time you build software.

What is Engineering Process

Building software is an engineering process. Though we don’t relate to it, but building software is equally complex and challenging as manufacturing a car or building a skyscraper.

Any engineering process needs a set of guidelines that needs to be followed while we engineer a solution. A solution engineered without sound principles is inconsistent and tends to be brittle over time.

Engineering Principles are ideas, rules, and concepts which guides our engineering process. Though they may seem like an overkill, but it pays in the long run. Think of Engineering Process as taking your car out of parking, you will make sure the path is clear, check your blind spots and move on to the road.

Below is a sample Engineering Process which works for me. Tweak it as per your needs.

Engineering Process that works for me

Step 1: I will Design my solution before Implementation

Design before you build. This is a known concept in every engineering trade but somehow this eludes us when it comes to Software Engineering.

While building a house, we will never allow our contractor to start building unless he gets the designs approved. A car company will never start a production line unless every stakeholder agrees on the features and approves the design. Then why should it be any different when it comes to Software Engineering.

Prototyping a concept or writing quick and dirty code to learn a new language / technology is fine as either one can be discarded without any cost. But when it comes to production code, we should exactly know what we are doing. You will never drill a hole in your living room till you are 100% sure about what you are going to do with that hole.

Another problem that arises when we jump from requirement to code is, when the code goes for review. A typical story takes 1 - 2 working days of an engineer’s time. When the code goes for peer review, the reviewer is expected to provide feedback in few hours time. So there is added onus on the reviewer to understand the requirement, understand the engineer’s solution / thought process, apply the same to code and finally give a verdict all in few minutes of time, as engineers are measured on the amount of code they write and not the code they review. If we spend time collectively coming to a design solution, during review the reviewer only has to verify the solution against the approved design. This will reduce the review time as well as the chances of missed out logic or corner cases in the review.

Every Feature / Bug fix needs to designed, and the designs must be approved by your peers.

Step 2: I will Document my solution before Development

As a product grows, the code base grows and becomes more complex. Growing teams add engineers and engineers add more and more code. After sometime looking at code written a year back looks like mangled mess 🍝

Another challenge with ever evolving products is “Tribal Knowledge”. Before information could be passed down in written format, all useful information will be converted to poems and designated people in the tribe will spend their life memorising them and the saga continues for generations. Even with the advancement of technology, evolving product still are crippled by this problem. Every product has a go to guy / gal, who has all the poems memorised.

Only way to break this process is to document stuff. Not everyone is well versed in writing down their thoughts in a concise manner. Documentation doesn’t need to be written in fancy poetic English. Write documentation in the simplest consumable format. If words are too difficult to express the solution, use your drawing skills.

Every change to product must reflect in the living HLD (High Level Document).

Step 3: I will write code and explain

At the time of writing code, it makes perfect sense to us. The problem starts when revisit the same thing after sometime. We try looking at the code from all directions ⬆ ⬅ ↕ 🙃 😒 and start thinking why on earth did I write this and then we create a tech debt to clean up.

Continuous Improvement is good, but not every old code needs to be re-written. Knowing why you did something or why something needs to be changed will save you lot of heartburn and frustration 🤯

Get into the habit of writing Readme for every feature added or shortcuts taken which needs to be revisited. This will become the LLD (Low Level Document) of your code base which will benefit you or anyone who looks at it later.

Every change to code should reflect in the ReadMe of the project (Low Level Documentation).

Step 4: I will test my code

Whenever we buy any electronic good, most of them will have a small sticker which says “QC passed“.

All code is only as good as the tests which verifies it.

We all trust our code, as we have written it ourselves. But how does it hold good against the requirement? A function to add 2 numbers may compile and run fine in production, but it is of no value if the original requirement was to multiply 2 numbers 😊

Only unit testing your code shows your logic is solid and it works as expected. But code is part of an engineering solution which is expected to do a specific job. You can build the world’s best exhaust system but if it doesn’t fit in your final car’s model it is of no use. Hence it is important to test your code against the requirements for which it was written.

Every change must be Unit tested for logic, and Integration Tested for business requirements.

Step 5: I will track my changes

All effort put in engineering a solution is for a specific reason. We develop feature to be working in an expected manner. But how do we confirm if it is working as expected?

Product metrics are for measuring business success and cannot be a proxy for engineering success. A poorly engineered product can achieve business targets.

Every change we make is for a reason and the same reasoning must be reflected in tracking. If the change was engineered to improve the page load performance, we should be able to have enough tracking to prove we have improved the page load performance. If the feature release was done for user interaction, then we should have enough tracking to prove that the feature is working as expected.

Tracking should also be proactive in detecting system anomalies. It should inform the engineer when unexpected errors are occurring and collect relevant data to reproduce the error. It should also warn the engineer about possible system degradation (when user is having visible errors but performance is taking a hit). In a nutshell, every change should have enough metrics to give a detailed picture of the overall product.

Every change must define its tracking metrics and have enough trace to find anomalies.

Engineering Process is NOT a silver bullet which will solve all your problems. Having an Engineering Process will not make your bugs disappear or suddenly make your application run faster.

Engineering Process will bring sanity in your chaos filled process. It will help you build software products which can be reasoned, tracked and measured.

React Project - Idea to Production - Part Four - Hosting the Movie app and setting up CI/CD

DEBOJIT ROY — Thu, 18 Jun 2020 04:16:05 +0000

This was originally posted here

This is the last post in the series. You can find the first post here

Where are we

Ok so till now we have

Brainstormed on our brilliant idea to build a Movie App.
We have decided what features are needed as part of the MVP.
Our design team has given us the wireframes.
We have setup our project as a Monorepo.
We have setup linting rules, code formatter and commit hooks.
We have setup our our component library
We added support for Typescript in our component library
We have setup Storybook
We added our components to the component library
We have added unit tests for our components
We can see our components showcased in Storybook
We extracted our core services in core project
We setup our webapp project for the web application
We added webpack configs for our dev build
We developed our Web Application
We made sure everything runs on our local development machine

What are we going to do now

So right now we have our web application running on our local system. So the next step is to host the application on a server and make sure it is accessible on the internet. We also would like to setup Continuous Integration (CI) to make sure every-time some one makes any changes, everything is still working fine. And we would also like Continuous Delivery (CD) to push our application to production every-time we merge new changes. We will use AWS as our cloud service. We will use Github Actions for our CI/CD pipeline.

TL;DR

This is a 4 part post

Part One : Wireframes and Project Setup
Part Two : Setting up a Component Library
Part Three : Building the Movie App using component library
Part Four: Hosting the Movie app and setting up CI/CD

Source Code is available here

Component Library Demo is available here

Movie App Demo is available here

Setting up Production Build

We already have our development server running perfectly, but we can't publish development build to production. We will have to configure our production build.

We start by creating a separate config for production. Create a new config file webapp/config/webpack.prod.js.

What we need in our production build

We want to type check our code before building (as we are using babel-loader, we have to do type check separately).
We want production quality JS code
We want our chunks to be content hashed
We want our CSS to be minified
We would want to split our vendor files into separate chunks. (This is controversial but a very good reason for doing this is explained here).

So we end up having a prod config which looks like this

const path = require("path")
const merge = require("webpack-merge")
const webpack = require("webpack")
const TerserJSPlugin = require("terser-webpack-plugin")
const MiniCssExtractPlugin = require("mini-css-extract-plugin")
const OptimizeCSSAssetsPlugin = require("optimize-css-assets-webpack-plugin")
const ForkTsCheckerWebpackPlugin = require("fork-ts-checker-webpack-plugin")
const common = require("./webpack.common.js")

module.exports = merge(common, {
  mode: "production",
  output: {
    path: path.join(__dirname, "../dist"),
    filename: "[name].[contenthash].js",
    publicPath: "/",
  },
  plugins: [
    new ForkTsCheckerWebpackPlugin(),
    new webpack.HashedModuleIdsPlugin(), // so that file hashes don't change unexpectedly
    new MiniCssExtractPlugin({
      // Options similar to the same options in webpackOptions.output
      // all options are optional
      filename: "[name].[contenthash].css",
      chunkFilename: "[id].[contenthash].css",
      ignoreOrder: false, // Enable to remove warnings about conflicting order
    }),
  ],
  optimization: {
    minimize: true,
    minimizer: [new TerserJSPlugin({}), new OptimizeCSSAssetsPlugin({})],
    runtimeChunk: "single",
    splitChunks: {
      chunks: "all",
      maxInitialRequests: Infinity,
      minSize: 0,
      cacheGroups: {
        vendor: {
          test: /[\\/]node_modules[\\/]/,
          name(module) {
            // get the name. E.g. node_modules/packageName/not/this/part.js
            // or node_modules/packageName
            const packageName = module.context.match(
              /[\\/]node_modules[\\/](.*?)([\\/]|$)/
            )[1]

            // npm package names are URL-safe, but some servers don't like @ symbols
            return `npm.${packageName.replace("@", "")}`
          },
        },
      },
    },
  },
})

Now with our config in place, let's add our production build script.

"build:webapp:prod": "cross-env production=true webpack --config config/webpack.prod.js"

Now if we run yarn build:webapp:prod we should see our compiled files under webapp/dist folder.

Preparing our Cloud Environment for hosting

Once we have our production build, there are many ways we can host our application. If you google hosting a static site there will be thousands of good ways to do it.

For our use case we will use Amazon S3 for hosting our site, and Amazon Cloudfront as our CDN.

Setting up S3

We will first setup our S3 Bucket for storybook, then we can repeat the same steps for webapp.

Create the Bucket

Allow `Public` Access

For hosting static site, people should be able to access your S3 bucket. So we need to allow public access.

Make `Read` public on the bucket

S3 will show warning when we do this change, but we need public read for our website.

Enable `Static Website` hosting for S3 bucket

Create a test `index.html`

Lets create a test index.html file.

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta http-equiv="x-ua-compatible" content="ie=edge" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <title>S3 Temp File</title>
  </head>
  <body>
    <h1>This is a temp file</h1>
  </body>
</html>

Uploading the `index.html`

Couple of things to note here. We need to set the Content-Type to text/html otherwise S3 will default to binary and we will see gibberish on the browser.

And very important thing, we need to make sure browser never caches our index.html. Hence we need to set the Cache-Control header correctly.

Testing our S3 website

If everything went well, you should be seeing this on our browser.

Setting up Cloudfront

With S3 Bucket setup, next thing we want to do is setup our CDN. We will use Cloudfront for our CDN.

Why do we need a CDN

We can work directly with our S3 Bucket, but it is always good to serve your content through CDN. Not only CDN makes the content delivery faster, it also adds an added layer of security.

Creating Cloudfront Distribution

We will be creating a standard Cloudfront Distribution with S3 origin. We will redirect all our traffic to https and will respect origin headers.

Adding rules for React Router

As we are using React Router for routing, we want all our requests to be handled by index.html. So even if a user refreshes the page with some route, we want to tell Cloudfront to redirect them back to index.html and not throw errors.

We do this by adding custom error rules.

Verifying the changes

Once we have setup the distribution, it will take some time to deploy. And even when it is deployed, if we try hitting the endpoint we may get weird errors.

This is expected behaviour as it takes time for the change to get propagated across all the Edges. Ideally wait for at least 20 minutes after deploying to verify the distribution.

Why no custom domain

If you want, you can add a CNAME to the distribution. But if we choose to do that, we need to upload our own certificates or in case you are using Route 53 for your DNS, you can use your AWS certificates. As I am not using Route 53 and my hosting is not on AWS, I have skipped this step.

If you want to add a CNAME, please check out the steps

Setting up Continous Integration

Now we have our Hosting sorted out, next step is to get our CI working.

CI Expectation

Before we even start to define our pipeline, we should first map the steps we want the CI to follow. This is a very important step as pipelines can get complicated and deciphering yaml files are not the easiest way to figure out what is happening.

We can write it like this,

Branch: master

Condition: When a Pull Request is created / updated

Exclusions: None

Tags: Any

Docker: Yes

Docker Image: unohomeloans/endcustomerportal:node12.8.1-chrome78-ff70-cypress-robot (Reusing an image which I created for running my pipelines)

Steps::

Checkout Code
List Files
Sanity Check node
Sanity Check yarn
Pull Dependencies
Build Components
Build Storybook
Build Core
Build WebApp
Run Components Tests
Run Core Tests
Archive Components Build Files
Archive Storybook Build Files
Archive Components Coverage
Archive Core Build Files
Archive Core Coverage
Archive WebApp Build Files

This is a simple workflow. Complicated workflows can have multiple branches with parallel and sequential executions.

We will use Github Actions to configure our CI for Pull Requests.

We will create a .github folder at the root of our project and add pull-requests.yml file. Our CI steps converted will look something like this

name: Pull Requests
on:
  pull_request:
    branches:
      - master
jobs:
  build:
    name: Build and Test Pull Request
    # This job runs on Linux
    runs-on: ubuntu-latest
    container:
      image: unohomeloans/endcustomerportal:node12.8.1-chrome78-ff70-cypress-robot
    steps:
      - name: Checkout code
        uses: actions/checkout@v2
      - name: List files
        run: "ls -ltr"
      - name: Check Node Version
        run: "node --version"
      - name: Check Yarn Version
        run: "yarn --version"
      - name: Pull Dependencies
        run: "yarn"
      - name: Build Components
        run: |
          cd packages/components
          yarn build-js:prod
      - name: Build Storybook
        run: |
          cd packages/components
          yarn build:storybook
      - name: Build Core
        run: |
          cd packages/core
          yarn build-js:prod
      - name: Build WebApp
        env:
          API_URL: ${{ secrets.API_URL }}
          API_KEY: ${{ secrets.API_KEY }}
        run: |
          cd packages/webapp
          yarn build:webapp:prod
      - name: Run Components Tests
        run: |
          cd packages/components
          yarn test
      - name: Run Core Tests
        run: |
          cd packages/core
          yarn test
      - name: Archive Components Build Files
        uses: actions/upload-artifact@v1
        with:
          name: components
          path: packages/components/lib
      - name: Archive Storybook Build Files
        uses: actions/upload-artifact@v1
        with:
          name: storybook
          path: packages/components/dist
      - name: Archive Components Coverage
        uses: actions/upload-artifact@v1
        with:
          name: components-coverage
          path: packages/components/coverage
      - name: Archive Core Build Files
        uses: actions/upload-artifact@v1
        with:
          name: core
          path: packages/core/lib
      - name: Archive Core Coverage
        uses: actions/upload-artifact@v1
        with:
          name: core-coverage
          path: packages/core/coverage
      - name: Archive WebApp Build Files
        uses: actions/upload-artifact@v1
        with:
          name: webapp
          path: packages/webapp/dist

We are configuring our build environment variables using Github Secrets. Once the Secrets are setup, we can use it anywhere in our workflow. For more details check out their documentation.

Once we have set this up and raise a Pull Request, Github will run this pipeline everytime someone updates the pull request. We can check out the run under Actions tab.

Setting up Continuous Delivery

Now with Continuous Integration (CI) we are confident that every changes will be tested and application build is working. Next step is Continuous Delivery (CD).

As part of CD, we want to make sure every-time any approved change when merged to master, we would like to release our Component Library and Movie App.

Preparing the Deployment script

After the build is done, we would like to upload the files to S3 Bucket and Invalidate our Cloudfront Distribution Cache. Also I don't want the browser to cache my index.html and I would like the browser to cache all my other files for longer time as they are content hashed.

We can use any of the available NPM packages for this purpose, but I prefer using AWS CLI for the purpose. I have bundled aws cli in my docker image so I don't have to download it for every run.

So we create deployApp.sh

#!/usr/bin/env bash
# Usage deployApp.sh <folder> <bucket_name> <cloudfront_id>
deploy_folder="$1"
s3_bucket_name="$2"
cloudfront_dist_id="$3"

echo "Deploying app...";

if [ ! $deploy_folder ]; then
echo "Deployment folder cannot be empty."
echo "Usage: deployApp.sh <folder> <bucket_name> <cloudfront_id>"
exit 2;
fi

if [ ! $s3_bucket_name ]; then
echo "S3 Bucket Name cannot be empty."
echo "Usage: deployApp.sh <folder> <bucket_name> <cloudfront_id>"
exit 2;
fi

if [ ! $cloudfront_dist_id ]; then
echo "Cloudfrint Distribution Id cannot be empty."
echo "Usage: deployApp.sh <folder> <bucket_name> <cloudfront_id>"
exit 2;
fi



# Upload all files except index.html to S3 Bucket
# Add 1 year cache as everything is content hashed
aws s3 sync $deploy_folder s3://$s3_bucket_name/ \
  --cache-control max-age=31556952 \
  --acl public-read \
  --exclude index.html

# Upload index.html with no-cache
aws s3 cp $deploy_folder/index.html s3://$s3_bucket_name/index.html \
  --metadata-directive REPLACE \
  --cache-control max-age=0,no-cache,no-store,must-revalidate \
  --content-type text/html \
  --acl public-read

# Invalidate Cloudfront Distribution
aws cloudfront create-invalidation \
  --distribution-id $cloudfront_dist_id \
  --paths /*


echo "Finished deploying app...";

Important note regarding AWS Credentials

For the release workflow, we will need to set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in Secrets. Easiest way to set this up is to copy your Admin user credentials and paste them. But I would suggest strongly NOT to do it.

I would suggest creating separate users with limited access to only the required S3 Buckets and Cloudfront distributions for the application. I would even recommend not reusing users across applications.

Though it adds an overhead, it is a better approach to minimize the risk of misuse of credentials. Also AWS makes it easy for setting up roles and access by making it declarative using the UI wizard.

Defining Release Workflow

The steps for release is pretty much the same with one addition of calling deploy script.
We create release.yml and add it to the .github folder.

name: Release
on:
  push:
    branches:
      - master
jobs:
  build:
    name: Build, Test, Release Movie App
    # This job runs on Linux
    runs-on: ubuntu-latest
    container:
      image: unohomeloans/endcustomerportal:node12.8.1-chrome78-ff70-cypress-robot
    steps:
      - name: Checkout code
        uses: actions/checkout@v2
      - name: List files
        run: "ls -ltr"
      - name: Check Node Version
        run: "node --version"
      - name: Check Yarn Version
        run: "yarn --version"
      - name: Pull Dependencies
        run: "yarn"
      - name: Build Components
        run: |
          cd packages/components
          yarn build-js:prod
      - name: Build Storybook
        run: |
          cd packages/components
          yarn build:storybook
      - name: Build Core
        run: |
          cd packages/core
          yarn build-js:prod
      - name: Build WebApp
        env:
          API_URL: ${{ secrets.API_URL }}
          API_KEY: ${{ secrets.API_KEY }}
        run: |
          cd packages/webapp
          yarn build:webapp:prod
      - name: Run Components Tests
        run: |
          cd packages/components
          yarn test
      - name: Run Core Tests
        run: |
          cd packages/core
          yarn test
      - name: Deploy Storybook
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: |
          cd packages/components
          yarn deploy:storybook
      - name: Deploy WebApp
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: |
          cd packages/webapp
          yarn deploy:webapp
      - name: Archive Components Build Files
        uses: actions/upload-artifact@v1
        with:
          name: components
          path: packages/components/lib
      - name: Archive Storybook Build Files
        uses: actions/upload-artifact@v1
        with:
          name: storybook
          path: packages/components/dist
      - name: Archive Components Coverage
        uses: actions/upload-artifact@v1
        with:
          name: components-coverage
          path: packages/components/coverage
      - name: Archive Core Build Files
        uses: actions/upload-artifact@v1
        with:
          name: core
          path: packages/core/lib
      - name: Archive Core Coverage
        uses: actions/upload-artifact@v1
        with:
          name: core-coverage
          path: packages/core/coverage
      - name: Archive WebApp Build Files
        uses: actions/upload-artifact@v1
        with:
          name: webapp
          path: packages/webapp/dist

Now whenever a Pull Request is merged, Github will make sure that the Release Pipeline is run.

Checking everything works fine

If everything went well, we should be able to access our websites.

Component Library

Awesome Movie App

That was one long post !!! I hope next time you have an idea you will be able to easily take it from an idea to production.

If you have any doubts or just want to say hello, feel free to drop me a note on hello@debojitroy.com

React Project - Idea to Production - Part Three - Building the Movie App using component library

DEBOJIT ROY — Thu, 18 Jun 2020 03:47:10 +0000

This was originally posted here

This is the third post in the series. You can find the first post here

Where are we

Ok so till now we have

Brainstormed on our brilliant idea to build a Movie App.
We have decided what features are needed as part of the MVP.
Our design team has given us the wireframes.
We have setup our project as a Monorepo.
We have setup linting rules, code formatter and commit hooks.
We have setup our our component library
We added support for Typescript in our component library
We have setup Storybook
We added our components to the component library
We have added unit tests for our components
We can see our components showcased in Storybook

What are we going to do now

Ok so the next step is to build the movie app using the component library. We will be using TMDB for fetching our movie details. We will maintain our application state using Redux. We will use Webpack to bundle our application. At the end of this post we should have converted our wireframes to an actual working website.

TL;DR

This is a 4 part post

Part One : Wireframes and Project Setup
Part Two : Setting up a Component Library
Part Three : Building the Movie App using component library
Part Four: Hosting the Movie app and setting up CI/CD

Source Code is available here

Component Library Demo is available here

Movie App Demo is available here

Extracting common functionality in `core`

It is always advisable to extract common services to keep it DRY. As we extracted common components in our previous post, we will extract common functionality in core.

What resides in `core`

The definition of common functionality is very broad and there are more than one way to skin the chicken 🐔 For our project we will extract our api calls in core

Setting up `core`

Move to the packages folder

cd packages

Create a new folder for our core

mkdir core
cd core

Initialise the yarn project

yarn init

Following the steps for naming, as we did in our previous post, our package.json looks like

{
  "name": "@awesome-movie-app/core",
  "version": "1.0.0",
  "description": "Core Services for Awesome Movie App",
  "main": "index.js",
  "repository": "git@github.com:debojitroy/movie-app.git",
  "author": "Debojit Roy <debojity2k@gmail.com>",
  "license": "MIT",
  "private": true
}

Building `core`

Adding `axios`

We will be making a lot of XHR calls to fetch data. We can choose to use browser's native AJAX functionality or the shiny new fetch api. With so many browsers and different implementation of fetch it is safer not to use fetch. If we choose to include fetch we will have to add the required polyfills.

So it is much better to go ahead with axios which will make sure our network calls work correctly irrespective of the user's browser.

Initialising `config` variables

As core is a common library, we don't want to hardcode, nor dictate how the environment variables are set. We would like to delegate it to the calling project to decide.

So we will create a bootstrap file which will be used to initialise the config.

let config: { url: string; apiKey: string } = { url: "", apiKey: "" }

export const setConfig = (incomingConfig: { url: string; apiKey: string }) => {
  config = incomingConfig
}

export const getConfig = () => config

Adding `search service`

One of the first things as per our requirement was to add a search service. We are going to use the Search Endpoint

After mapping the response, the functionality looks something like this

import axios from "axios"
import isNil from "lodash/isNil"
import { getConfig } from "./bootstrap"

export interface SearchResult {
  popularity: number
  vote_count: number
  video: boolean
  poster_path: string
  id: number
  adult: boolean
  backdrop_path: string
  original_language: string
  original_title: string
  genre_ids: number[]
  title: string
  vote_average: number
  overview: string
  release_date: string
}

export interface SearchResponse {
  page: number
  total_results: number
  total_pages: number
  results: SearchResult[]
}

export const searchMovie = async (
  queryString?: string
): Promise<SearchResponse> => {
  const config = getConfig()

  if (isNil(queryString) || queryString.trim() === "") {
    return new Promise(resolve => {
      resolve({
        page: 1,
        total_pages: 1,
        total_results: 0,
        results: [],
      })
    })
  }

  const encodedQuery = encodeURI(queryString)

  const result = await axios.get(
    `${config.url}/3/search/movie?api_key=${config.apiKey}&query=${encodedQuery}`
  )

  return result.data
}

We will continue mapping rest of the functionality, the complete code is available here

Setting up Web Application

Now with the required services mapped out, we will focus on building the actual web application.

Splitting out code in this way helps to re-use functionality without copy pasting things over and over again.

Major parts of our webapp will be

Public Files
Webpack config
Common parts
Feature specific segregation

WebApp Project setup

Move to the packages folder

cd packages

Create a new folder for our webapp

mkdir webapp
cd webapp

Initialise the yarn project

yarn init

Following the steps for naming, as we did in our previous post, our package.json looks like

{
  "name": "@awesome-movie-app/webapp",
  "version": "1.0.0",
  "description": "Web Application for Awesome Movie App",
  "main": "index.js",
  "repository": "git@github.com:debojitroy/movie-app.git",
  "author": "Debojit Roy <debojity2k@gmail.com>",
  "license": "MIT",
  "private": true
}

Setting up `public` assets

So for the React project to mount, we need a DOM element, where React can take over and inject the elements. For this purpose we need a index.html file which will be served by the server before React takes over.

We will keep this index.html in our public folder, but feel free to choose any other name.

You can find the file here Feel free to name the folder and files as you want, but make sure to update the same in the webpack config in the next step.

Setting up Webpack

We will use webpack to package our application. You can choose any other packager for your project and make changes accordingly.

Prepare the config folder

mkdir config

Setting up shared configuration

For our local development we will be using webpack dev server and production build and minification for production build. But some of the steps will be common for both, we will extract those in our common config.

So our common config looks something like this

// webpack.common.js
const path = require("path")
const MiniCssExtractPlugin = require("mini-css-extract-plugin")
const { CleanWebpackPlugin } = require("clean-webpack-plugin")
const HtmlWebPackPlugin = require("html-webpack-plugin")

const isEnvDevelopment = process.env.NODE_ENV === "development"
const isEnvProduction = process.env.NODE_ENV === "production"

module.exports = {
  entry: { main: "./src/entry/index.tsx" },
  resolve: {
    extensions: [".ts", ".tsx", ".js", ".jsx"],
  },
  node: {
    fs: "empty",
  },
  module: {
    rules: [
      {
        test: /\.(js|jsx|mjs|ts|tsx)$/,
        exclude: /node_modules/,
        use: {
          loader: "babel-loader",
        },
      },
      {
        test: /\.css$/,
        use: [
          "style-loader",
          {
            loader: MiniCssExtractPlugin.loader,
            options: {
              hmr: isEnvDevelopment,
            },
          },
          "css-loader",
          {
            loader: "postcss-loader",
            options: {
              ident: "postcss",
              plugins: () => [
                require("postcss-flexbugs-fixes"),
                require("postcss-preset-env")({
                  autoprefixer: {
                    flexbox: "no-2009",
                  },
                  stage: 3,
                }),
                require("postcss-normalize"),
              ],
              sourceMap: isEnvProduction,
            },
          },
        ],
        // Don't consider CSS imports dead code even if the
        // containing package claims to have no side effects.
        // Remove this when webpack adds a warning or an error for this.
        // See https://github.com/webpack/webpack/issues/6571
        sideEffects: true,
      },
      {
        test: /\.(png|svg|jpg|gif)$/,
        use: ["file-loader"],
      },
      {
        test: /\.(woff|woff2|eot|ttf|otf)$/,
        use: ["file-loader"],
      },
    ],
  },
  plugins: [
    new CleanWebpackPlugin(),
    new HtmlWebPackPlugin({
      title: "Awesome Movie App",
      template: "./public/index.html",
      filename: "./index.html",
      favicon: "./public/favicon.ico",
    }),
  ],
}

Most of the things are self explanatory. If you are new to webpack, I would suggest checking out their awesome documentation

Setting up the `dev` config

With common config setup, we would like to setup our dev config. We want to use webpack dev server and hmr with routing fallback.

Our dev config looks like

//webpack.dev.js
const path = require("path")
const merge = require("webpack-merge")
const MiniCssExtractPlugin = require("mini-css-extract-plugin")
const common = require("./webpack.common.js")

module.exports = merge(common, {
  mode: "development",
  devtool: "inline-source-map",
  output: {
    path: path.join(__dirname, "../../dist/dist-dev"),
    filename: "[name].[contenthash].js",
    publicPath: "/",
  },
  devServer: {
    contentBase: "./dist-dev",
    historyApiFallback: true,
    allowedHosts: [".debojitroy.com"],
  },
  plugins: [
    new MiniCssExtractPlugin({
      filename: "[name].css",
    }),
  ],
})

Building the `common` parts

Common parts are feature agnostic pieces which have the cross cutting functionality.

Common - Components

These are the common components which will be used across the features.

Common - Config

Configurations for applications which are defined here.

Common - Redux

Redux specific files will be stored here.

Common - Routes

Routing specific files will be stored here.

Common - Utils

Common utilities will be added here.

Building Features

Features is where the actual features of the application will be kept. Think of each feature as a standalone piece of the application. Each feature in itself should be able to stand apart.
For demonstration purpose we will look into SiteHeader feature.

SiteHeader - Components

This part will contain all our React components as the name suggests. Based on the functionality required we will break down our feature in components.

SiteHeader - Redux

This is where all Redux related files will be added.

I am skipping over these sections fast as they are standard React / Redux stuff which are better explained in many other places.

Getting the webapp running

Adding `.env`

We need to declare the config variables for running our application. In our production step we will be doing it differently. For local development let's add .env file and add it to .gitignore so that it doesn't get checked in.

Go to webapp

cd packages/webapp

Create a .env file

vim .env

Add the config values

API_URL=https://api.themoviedb.org
API_KEY=<Replace with actual key>

Preparing launch script

Now once we have .env setup, last thing we need to do is add the start script.

Open package.json inside webapp and add this under scripts

"start": "cross-env development=true webpack-dev-server --config config/webpack.dev.js --open --port 8000"

Running Webapp locally

Once we are done setting up webapp, lets try to run it locally.

First, build your components

cd packages/components
yarn build-js:prod

Second, build your core

cd packages/core
yarn build-js:prod

Finally start your webapp

cd packages/webapp
yarn start

If everything went well, you should see something like this

Phew!!! That was a long one.

Now, the final step is to configure Continuous Integration and Deployment to make sure every-time we make changes, it gets deployed seamlessly. You can read about it in the last instalment of this series.

React Project - Idea to Production - Part Two - Setting up a Component Library

DEBOJIT ROY — Thu, 18 Jun 2020 02:21:01 +0000

This was originally posted here

This is the second post in the series. You can find the first post here

Where are we

Ok so till now we have

Brainstormed on our brilliant idea to build a Movie App.
We have decided what features are needed as part of the MVP.
Our design team has given us the wireframes.
We have setup our project as a Monorepo.
We have setup linting rules, code formatter and commit hooks.

What are we going to do now

Ok so the next step is to break down our wireframe into components. We will build a component library which can be used across various projects. Finally we will setup storybook to showcase our component library.

TL;DR

This is a 5 part post

Part One : Wireframes and Project Setup
Part Two : Setting up a Component Library
Part Three : Building the Movie App using component library
Part Four: Hosting the Movie app and setting up CI/CD

Source Code is available here

Component Library Demo is available here

Movie App Demo is available here

Setting the Component Library

Now let's move ahead by setting up our component library.

Move to the packages folder

cd packages

Create a new folder for our components

mkdir components
cd components

Initialize the yarn project

yarn init

Naming is important here as we will be referring our projects in our workspace using the name. I prefer organization scoped name to avoid naming conflicts. So for our example I will be using @awesome-movie-app as our organization name. Feel free to replace with your organization scope.

Next thing to keep in mind is how you want to publish your packages to npm. If you would like to publish packages to npm, then make sure the version is semantic and let lerna handle the publishing to packages.

If you have a restricted / private NPM organization, make sure to add publishConfig with restricted access in your package.json to avoid accidental publishing of the packages to public npm.

"publishConfig": {
    "access": "restricted"
}

As for the purpose of this post, we will not be publishing our packages to npm, so we will skip defining the publishConfig.

So our package.json looks like

{
  "name": "@awesome-movie-app/components",
  "version": "1.0.0",
  "description": "Component Library for Awesome Movie App",
  "main": "index.js",
  "repository": "git@github.com:debojitroy/movie-app.git",
  "author": "Debojit Roy <debojity2k@gmail.com>",
  "license": "MIT",
  "private": true
}

Defining the requirements

Our project is now setup, lets define our requirements before we move further.

Our components will be React components
We will use TypeScript to build our components
We want to showcase our components using Storybook
We will use Bootstrap for base styles
We will adopt CSS-in-JS and use StyledComponents
We will transpile our code using Babel

Why no Webpack

In a ideal world we will be publishing our packages to npm. Before publishing our packages to npm we would want to transpile and package them nicely. For this my ideal choice will be webpack.

But one very important feature for libraries is the package should support Tree Shaking. Tree Shaking is a fancy word for trimming excess fat i.e. eliminating code that is not used in the importing library. Due to this known webpack issue, sadly it makes it impossible right now.

To work around the problem we can use Rollup, but as we are not interested right now in publishing our package to npm, we will use babel to transpile our components. I will cover how to use Rollup and tree shake your library in another post.

Preparing the project

Ok that was way too much theory, now let's move on to setup our project.

Last bit of theory before we move ahead. As we are using lerna as our high-level dependency manager, we will use lerna to manage dependencies. Which means to add a new dependency we will use this format

lerna add <dependency-name> --scope=<sub-project-name> <--dev>

dependency-name: Name of the npm package we want to install
sub-project-name: This is optional. If you omit this then the dependency will be installed across all the projects. If you want the dependency to be installed only for a specifc project, then pass in the name of the project from individual package.json
--dev: Same as yarn options. If you want to install only dev dependencies, pass in this flag.

Adding Project Dependencies

Usually I will go ahead and add most of the dependencies in one command. But for this post I will go verbose explaining each of the dependencies I am adding and the reasoning behind it.

Note: We will be adding everything from the root folder of the project i.e. the root folder of movie-app (one level above packages folder)

Adding React

lerna add react --scope=@awesome-movie-app/components --dev
lerna add react-dom --scope=@awesome-movie-app/components --dev

Why one dependency at a time

Sadly due to this limitation of lerna 😞

Why is React a dev dependency 🤔

This part is important. As this library will be consumed in other project, we don't want to dictate our version of React, rather we want the consuming project to inject the dependency. So we are going to add common libraries as dev dependencies and mark them as peer dependencies. This is true for any common libraries you may want to build.

We will be adding React in our peer dependencies of @awesome-movie-app/components

"peerDependencies": {
    "react": "^16.13.1",
    "react-dom": "^16.13.1"
  }

Adding TypeScript

lerna add typescript --scope=@awesome-movie-app/components --dev

Adding types for React

lerna add @types/node --scope=@awesome-movie-app/components
lerna add @types/react --scope=@awesome-movie-app/components
lerna add @types/react-dom --scope=@awesome-movie-app/components

Adding tsconfig for TypeScript

{
  "compilerOptions": {
    "outDir": "lib",
    "module": "commonjs",
    "target": "es5",
    "lib": ["es5", "es6", "es7", "es2017", "dom"],
    "sourceMap": true,
    "allowJs": false,
    "jsx": "react",
    "moduleResolution": "node",
    "rootDirs": ["src"],
    "baseUrl": "src",
    "forceConsistentCasingInFileNames": true,
    "noImplicitReturns": true,
    "noImplicitThis": true,
    "noImplicitAny": true,
    "strictNullChecks": true,
    "suppressImplicitAnyIndexErrors": true,
    "noUnusedLocals": true,
    "declaration": true,
    "allowSyntheticDefaultImports": true,
    "experimentalDecorators": true,
    "emitDecoratorMetadata": true,
    "esModuleInterop": true
  },
  "include": ["src/**/*"],
  "exclude": ["node_modules", "build", "scripts"]
}

Adding Storybook

lerna add @storybook/react --scope=@awesome-movie-app/components --dev

Adding some cool add-ons

lerna add @storybook/addon-a11y --scope=@awesome-movie-app/components --dev
lerna add @storybook/addon-actions --scope=@awesome-movie-app/components --dev
lerna add @storybook/addon-docs --scope=@awesome-movie-app/components --dev
lerna add @storybook/addon-knobs --scope=@awesome-movie-app/components --dev
lerna add @storybook/addon-viewport --scope=@awesome-movie-app/components --dev
lerna add storybook-addon-styled-component-theme --scope=@awesome-movie-app/components --dev
lerna add @storybook/addon-jest --scope=@awesome-movie-app/components --dev

Adding Test Libraries

We will be using jest for unit testing

lerna add jest --scope=@awesome-movie-app/components --dev
lerna add ts-jest --scope=@awesome-movie-app/components --dev

We will be using enzyme for testing our React Components

lerna add enzyme --scope=@awesome-movie-app/components --dev
lerna add enzyme-adapter-react-16 --scope=@awesome-movie-app/components --dev
lerna add enzyme-to-json --scope=@awesome-movie-app/components --dev

Adding jest-styled-components for supercharing jest

lerna add jest-styled-components --scope=@awesome-movie-app/components --dev

Configure enzyme and jest-styled-components to work with jest. We will add setupTests.js

require("jest-styled-components")
const configure = require("enzyme").configure
const EnzymeAdapter = require("enzyme-adapter-react-16")

const noop = () => {}
Object.defineProperty(window, "scrollTo", { value: noop, writable: true })
configure({ adapter: new EnzymeAdapter() })

Configure jest.config.js

module.exports = {
  preset: "ts-jest",
  // Automatically clear mock calls and instances between every test
  clearMocks: true,

  // Indicates whether the coverage information should be collected while executing the test
  collectCoverage: true,

  // An array of glob patterns indicating a set of files for which coverage information should be collected
  collectCoverageFrom: [
    "src/**/*.{ts,tsx}",
    "!src/**/index.{ts,tsx}",
    "!src/**/styled.{ts,tsx}",
    "!src/**/*.stories.{ts,tsx}",
    "!node_modules/",
    "!.storybook",
    "!dist/",
    "!lib/",
  ],

  // The directory where Jest should output its coverage files
  coverageDirectory: "coverage",

  // An array of regexp pattern strings used to skip test files
  testPathIgnorePatterns: ["/node_modules/", "/lib/", "/dist/"],

  // A list of reporter names that Jest uses when writing coverage reports
  coverageReporters: ["text", "html", "json"],

  // An array of file extensions your modules use
  moduleFileExtensions: ["ts", "tsx", "js", "jsx"],

  // A list of paths to modules that run some code to configure or set up the testing framework before each test
  setupFilesAfterEnv: ["./setupTests.js"],

  // A list of paths to snapshot serializer modules Jest should use for snapshot testing
  snapshotSerializers: ["enzyme-to-json/serializer"],
}

Adding Styled Components and BootStrap

lerna add styled-components --scope=@awesome-movie-app/components --dev
lerna add react-bootstrap --scope=@awesome-movie-app/components --dev
lerna add bootstrap --scope=@awesome-movie-app/components --dev

lerna add @types/styled-components --scope=@awesome-movie-app/components

Adding Babel

As we will be using babel to transpile everything. Its important we configure Babel properly.

Adding Babel Dependencies

lerna add @babel/core --scope=@awesome-movie-app/components --dev
lerna add babel-loader --scope=@awesome-movie-app/components --dev
lerna add @babel/cli --scope=@awesome-movie-app/components --dev
lerna add @babel/preset-env --scope=@awesome-movie-app/components --dev
lerna add @babel/preset-react --scope=@awesome-movie-app/components --dev
lerna add @babel/preset-typescript --scope=@awesome-movie-app/components --dev
lerna add core-js --scope=@awesome-movie-app/components --dev

A bit on the babel components we added

@babel/core : Core babel functionality
babel-loader : Used by storybook webpack builder
@babel/cli : Will be used by us to transpile files from command line
@babel/preset-env : Environment setting for transpiling
@babel/preset-react : React setting for babel
@babel/preset-typescript : TypeScript settings for babel
core-js : Core JS for preset-env

Now let's add our .babelrc file

{
  "presets": [
    "@babel/preset-typescript",
    [
      "@babel/preset-env",
      {
        "useBuiltIns": "entry",
        "corejs": "3",
        "modules": false
      }
    ],
    "@babel/preset-react"
  ]
}

Bringing it all together

Important Note

The steps below may differ based on which version of Storybook and Jest you are using. The below steps are written for Storybook v5.3+ and Jest v26.0+

Setting up our theme

First step will be to setup our theme. We can start with a blank theme and fill it up as we go.

cd packages/components
mkdir theme

Defining the Theme

export interface Theme {
  name: string
  color: {
    backgroundColor: string
    primary: string
    secondary: string
  }
}

Defining Light theme

import { Theme } from "./theme"

const lightTheme: Theme = {
  name: "LIGHT",
  color: {
    backgroundColor: "#fff",
    primary: "#007bff",
    secondary: "#6c757d",
  },
}

export default lightTheme

Defining Dark theme

import { Theme } from "./theme"

const darkTheme: Theme = {
  name: "DARK",
  color: {
    backgroundColor: "#000",
    primary: "#fff",
    secondary: "#6c757d",
  },
}

export default darkTheme

Setting up Storybook

To configure storybook, we need to setup the configuration folder first. We will use the default .storybook folder, but feel free to use folder name.

mkdir .storybook

Now inside .storybook folder we will create the configuration files needed for storybook

main.js

This is the main configuration file for storybook. We will configure the path for stories, register our addons and override webpack config to process typescript files.

// .storybook/main.js

module.exports = {
  stories: ["../src/**/*.stories.[tj]sx"],
  webpackFinal: async config => {
    config.module.rules.push({
      test: /\.(ts|tsx)$/,
      use: [
        {
          loader: require.resolve("ts-loader"),
        },
      ],
    })
    config.resolve.extensions.push(".ts", ".tsx")
    return config
  },
  addons: [
    "@storybook/addon-docs",
    "@storybook/addon-actions/register",
    "@storybook/addon-viewport/register",
    "@storybook/addon-a11y/register",
    "@storybook/addon-knobs/register",
    "storybook-addon-styled-component-theme/dist/register",
    "@storybook/addon-jest/register",
  ],
}

manager.js

Here we configure the Storybook manager. There are many options that can be overriden, for our project we want the add-ons panel to be at the bottom (default is right)

// .storybook/manager.js

import { addons } from "@storybook/addons"

addons.setConfig({
  panelPosition: "bottom",
})

preview.js

Finally we will configure the Story area. We intialize our add-ons and pass global configurations.

// .storybook/preview.js
import { addParameters, addDecorator } from "@storybook/react"
import { withKnobs } from "@storybook/addon-knobs"
import { withA11y } from "@storybook/addon-a11y"
import { withThemesProvider } from "storybook-addon-styled-component-theme"
import { withTests } from "@storybook/addon-jest"
import results from "../.jest-test-results.json"
import lightTheme from "../theme/light"
import darkTheme from "../theme/dark"

export const getAllThemes = () => {
  return [lightTheme, darkTheme]
}

addDecorator(withThemesProvider(getAllThemes()))

addDecorator(withA11y)
addDecorator(withKnobs)

addDecorator(
  withTests({
    results,
  })
)

addParameters({
  options: {
    brandTitle: "Awesome Movie App",
    brandUrl: "https://github.com/debojitroy/movie-app",
    showRoots: true,
  },
})

Creating React Components

Now we can create our very first react component.

Our first button

We will first create a src folder

mkdir src && cd src

Then we will create a folder for our component. Let's call it Sample

mkdir Sample && cd Sample

Now let's create a simple styled button and pass some props to it.

// styled.ts
import styled from "styled-components"

export const SampleButton = styled.button`
  background-color: ${props => props.theme.color.backgroundColor};
  color: ${props => props.theme.color.primary};
`

// Button.tsx
import React from "react"
import { SampleButton } from "./styled"

const Button: React.FC<{
  value: string
  onClickHandler: () => void
}> = ({ value, onClickHandler }) => (
  <SampleButton onClick={onClickHandler}>{value}</SampleButton>
)

export default Button

Awesome !!! We have our first component finally !!!

Adding unit tests

Now let's add some tests for our new button.

mkdir tests

// tests/Button.test.tsx

import React from "react"
import { mount } from "enzyme"
import { ThemeProvider } from "styled-components"
import lightTheme from "../../../theme/light"
import Button from "../Button"

const clickFn = jest.fn()
describe("Button", () => {
  it("should simulate click", () => {
    const component = mount(
      <ThemeProvider theme={lightTheme}>
        <Button onClickHandler={clickFn} value="Hello" />
      </ThemeProvider>
    )
    component.find(Button).simulate("click")
    expect(clickFn).toHaveBeenCalled()
  })
})

Adding stories

Now with the new button in place, lets add some stories

mkdir stories

We will use the new Component Story Format (CSF)

// stories/Button.stories.tsx

import React from "react"
import { action } from "@storybook/addon-actions"
import { text } from "@storybook/addon-knobs"
import Button from "../Button"

export default {
  title: "Sample / Button",
  component: Button,
}

export const withText = () => (
  <Button
    value={text("value", "Click Me")}
    onClickHandler={action("button-click")}
  />
)

withText.story = {
  parameters: {
    jest: ["Button.test.tsx"],
  },
}

Time to check if everything works

Transpiling our code

As we discussed in the beginning, we will be using babel to transpile our code and let the calling projects take care of minification and tree-shaking.

So going ahead with that, we will add some scripts and test they are working.

Typecheck and Compilation

We will first use TypeScript's compile to compile our code.

"js:build": "cross-env NODE_ENV=production tsc -p tsconfig.json"

If everything is fine, we should see an output like this

$ cross-env NODE_ENV=production tsc -p tsconfig.json
✨  Done in 5.75s.

Transpiling with Babel

Next step will be to transpile our code with babel

"build-js:prod": "rimraf ./lib && yarn js:build && cross-env NODE_ENV=production babel src --out-dir lib --copy-files --source-maps --extensions \".ts,.tsx,.js,.jsx,.mjs\""

If everything is fine, we should see an output like this

$ rimraf ./lib && yarn js:build && cross-env NODE_ENV=production babel src --out-dir lib --copy-files --source-maps --extensions ".ts,.tsx,.js,.jsx,.mjs"
$ cross-env NODE_ENV=production tsc -p tsconfig.json
Successfully compiled 4 files with Babel.
✨  Done in 7.02s.

Setting up watch mode for development

During development, we would like incremental compilation every time we make changes. So let's add a watch script.

"js:watch": "rimraf ./lib && cross-env NODE_ENV=development concurrently -k -n \"typescript,babel\" -c \"blue.bold,yellow.bold\"  \"tsc -p tsconfig.json --watch\" \"babel src --out-dir lib --source-maps --extensions \".ts,.tsx,.js,.jsx,.mjs\" --copy-files --watch --verbose\""

We should see output like this

Starting compilation in watch mode...
[typescript]
[babel] src/Sample/Button.tsx -> lib/Sample/Button.js
[babel] src/Sample/stories/Button.stories.tsx -> lib/Sample/stories/Button.stories.js
[babel] src/Sample/styled.ts -> lib/Sample/styled.js
[babel] src/Sample/tests/Button.test.tsx -> lib/Sample/tests/Button.test.js
[babel] Successfully compiled 4 files with Babel.
[typescript]
[typescript] - Found 0 errors. Watching for file changes.

Running Unit Tests

Once we are sure our compilation and transpiling works, lets make sure our tests work.

"test": "jest"

Running our tests should show an output similar to this

We are getting there slowly 😊

Now we need to generate json output for storybook to consume and show next to our stories. Let's configure that as well.

"test:generate-output": "jest --json --outputFile=.jest-test-results.json || true"

Running Storybook

Finally we want to run storybook with our stories. Lets run storybook in dev mode.

"storybook": "start-storybook -p 8080"

If everything was configured properly, we should see the storybook in our Browser

We will add couple of more commands for building storybook for deployment. We will be using these when we configure Continuous Deployment in our last post - Part Four: Hosting the Movie app and setting up CI/CD

"prebuild:storybook": "rimraf .jest-test-results.json && yarn test:generate-output",
"build:storybook": "build-storybook -c .storybook -o dist/"

After this we can start splitting our wireframes into components. I will not go into the details of that as there are much better posts out there which do a better job of explaining the process. You can find the code that we complete till now here

In the next part we will setup and build our movie app, continue to Part Three : Building the Movie App using component library

React Project - Idea to Production - Part One - Wireframes and Project Setup

DEBOJIT ROY — Thu, 18 Jun 2020 01:05:15 +0000

This was originally posted here

What are trying to achieve

We as developers like writing code. Given a choice I will write code whole day 💻. But writing code is only part of the job description. As engineers we are expected to build stuff and provide solutions. Writing code is only a part of it. I have met amazing developers who can write React way better than me. I have met designers and product guys who can come up with innovative ideas and design it to pixel perfect detail.

All of them being good in what they do, struggle in understanding how all this fits together in a software.

This is NOT a frontend tutorial about React, Design Systems or JavaScript. There are smarter people with better tutorials than this post.

What we are trying to do is to see how to take an idea from wireframe to production with automated tests and Continuous Integration and Continuous Deployment (CI / CD). 🚀

TL;DR

This is a 4 part post

Part One : Wireframes and Project Setup
Part Two : Setting up a Component Library
Part Three : Building the Movie App using component library
Part Four: Hosting the Movie app and setting up CI/CD

Source Code is available here

Component Library Demo is available here

Movie App Demo is available here

The Idea 💡

Lets say we came up the brilliant idea to build a movie app. We have no idea something like IMDB exists. So we get together an awesome team to build this Web application.

The Requirements

So the team wants to start simple, so we agree on building only 3 features for the MVP.

The Landing Page

This will be the face of our product. The user should be able to see these features

Search Functionality: User should be able to search any movie
Featured Movies: App should be able to promote some movies
Popular People: Show the most popular people this week in the entertainment industry

The Movie Profile Page

The user can navigate to this page to see the details of the movie. This page should highlight these basic details about the movie

Movie Details and Metadata
Movie Rating
Movie Poster
Movie Trailer
Movie Cast
Similar Movies

The Actor Bio Page

This user can navigate to this page to see the details of an Movie Actor. This page should highlight the basic details of the actor.

Actor Details and Info
Actor Photo
Actor Bio
Movie Acted

The Wireframes

Before we start hacking code, a lot of work goes in design phase where our awesome UX designers come up the designs which we will be building. As I am not a UX designer, I will spare you from my horrible sketch skills and proceed with wireframes for now.

Landing Page Wireframe

Movie Profile Page Wireframe

Actor Bio Page Wireframe

Project Setup

Now we have the wireframes ready, let's get our project setup.

Create a Github Project

Setting up Project on our development machine

So there are many ways to skin the chicken. I personally prefer a Monorepo. Some of you may hate monorepo and some of you may love it, the war is never ending. Lengths of blogs and documents are written on the benefits and perils of using a monorepo. Sometimes pros outweigh the cons, and sometime it is the other way round. For your next company project, I would suggest to figure out whether mono or multi repo fits your requirement.

For the sake of this demo let's go ahead with a monorepo.

For our monorepo we are going to use Lerna for high-level management, and yarn workspaces for low level package management. Again there is pretty good literature available for this setup which I would strongly suggest to read.

If shell>yarn workspaces are not enabled, lets first enable it.

yarn config set workspaces-experimental true

Lets clone the repository

git clone git@github.com:debojitroy/movie-app.git
cd movie-app

Lets not pollute the master and move to a separate branch

git checkout -b setting-it-up

Initialise the project

yarn init

Add Lerna as a dev dependency

yarn add lerna --dev

Once Lerna is added, it is time to initialise Lerna

lerna init

This should create packages folder and lerna.json

Now as we don't want to publish everything to npm simultaneously and would prefer yarn as our default client. We would also be using yarn workspaces, so we need to make some changes to lerna.json

{
  "packages": ["packages/*"],
  "version": "independent",
  "npmClient": "yarn",
  "useWorkspaces": true
}

Now we gotta enable yarn workspaces, so we need to add these few lines to our root package.json

"workspaces": [
     "packages/*"
  ]

Point to note, packages is the default name for the folder and you can name it anything you want. If you are changing the name, then make sure to update both lerna.json and package.json

Ok so our root project is setup now, let's commit our work and sync it to github. Always a good idea to do it.

git add .
git commit -m "Setting up root project"
git push --set-upstream origin setting-it-up

Setting up Linting and Commit Hooks

Before we move ahead and start writing code, we should take care of one more necessary evil - Linting and Commit Hooks.

We always end up adding lint rules and commit hooks way down the line in our development cycle and then spend ages fixing the errors. So in my recent projects I have decided to turn the tables around and set it up before I start writing code. You can always add more checks as the project progresses, but based on what we know, we should have the baseline rules in place.

So what we know about our Application

It will be a React Application
We will be using TypeScript
We may have a mix of JS and TS Files

So with the starting knowledge of how our application will look like, we can decided on our tools.

Tools we are going to use

As with any frontend project, there are so many ways to implement coding standards across the team. Tools for the job also differ based on the requirements. I have found a combination of tools to work best for me, and as I have the luxury to choose my tools, I will be using these

ESLint for enforcing lint rules
Prettier for code formatting
Husky for commit hooks

Now lets go ahead and add these tools

yarn add eslint --dev

When you will run this command, you will see this error

error Running this command will add the dependency to the workspace root rather than the workspace itself, which might not be what you want - if you really meant it, make it explicit by running this command again with the -W flag (or --ignore-workspace-root-check).

This is yarn warning us that we should not be adding dependencies in the root project. The warning is perfectly valid as we shouldn't add packages to the root project. But this a special case, where we want to enforce project level rules, so it is safe to override the warning and go ahead with -W flag.

So running it again

yarn add eslint --dev -W

Adding Prettier and Husky as well

yarn add prettier husky --dev -W

Adding ESLint Plugins

Now as we want to use ESLint with custom setup, we need to add plugins.

We want to run linting whenever some file is staged.

yarn add lint-staged --dev -W

We want to ESLint to behave nicely with Prettier.

yarn add eslint-config-prettier eslint-plugin-prettier --dev -W

We want to ESLint to understand React syntax and JSX format.

yarn add babel-eslint eslint-plugin-react eslint-plugin-react-hooks --dev -W

Finally, we want TypeScript and ESLint to understand our TypeScript syntax.

yarn add typescript @typescript-eslint/eslint-plugin @typescript-eslint/parser --dev -W

Nearly there...

Now lets configure our linter and formatter

Configuring Prettier

Now to configure Prettier, let's create a .prettierrc file and add these rules.
Again, these are my rules, feel free to configure it as per your requirement.

{
  "bracketSpacing": true,
  "jsxBracketSameLine": false,
  "printWidth": 80,
  "singleQuote": true,
  "tabWidth": 2,
  "trailingComma": "es5",
  "useTabs": true
}

Configuring ESLint

Re-visiting the rules again

Our code base will be mostly TypeScript
We will be writing React Code
We will be using latest React stuff including Hooks
We may have some .js and .jsx files as well
We want Prettier to work with ESLint.

With these rules in place, let's go ahead an create a .eslintrc.js file.

module.exports = {
  parser: "@typescript-eslint/parser",
  extends: [
    "plugin:react/recommended",
    "plugin:@typescript-eslint/recommended",
    "prettier/@typescript-eslint",
    "plugin:prettier/recommended",
  ],
  parserOptions: {
    ecmaVersion: 2018,
    sourceType: "module",
  },
  plugins: ["react-hooks"],
  rules: {
    "react/prop-types": [0],
    "react/forbid-prop-types": [0],
    "react-hooks/rules-of-hooks": "error",
    "react-hooks/exhaustive-deps": "warn",
    "@typescript-eslint/explicit-function-return-type": "off",
    "@typescript-eslint/no-explicit-any": "off",
  },
  settings: {
    react: {
      version: "detect",
    },
  },
  overrides: [
    {
      files: ["**/*.js", "**/*.jsx"],
      parser: "babel-eslint",
      extends: ["plugin:react/recommended", "plugin:prettier/recommended"],
      plugins: ["react-hooks"],
      rules: {
        "react-hooks/rules-of-hooks": "error",
        "react-hooks/exhaustive-deps": "warn",
      },
      settings: {
        react: {
          version: "detect",
        },
      },
    },
  ],
}

We have turned off some of the TypeScript lint checks for our project. This is optional and you can remove them if you want.

Now we dont want ESLint to lint our CSS and compiled files. So we will add .eslintignore file and add ignored paths.

dist
lib
cdn
src/styles
*.d.ts

Getting our lint together

Ok we are nearly there. Final step is to get everything together.
Now moving on to our package.json

Let's add our rules for staged files. Add a section to package.json

"lint-staged": {
    "packages/**/*.{js,ts,jsx,tsx}": [
      "eslint --color --fix"
    ]
  }

Finally adding our pre-commit hook using husky

"husky": {
    "hooks": {
      "pre-commit": "lint-staged"
    }
  }

Testing it all works together

Now is the time of truth. Lets see if everything works together.

git add .
git commit -m "Setting up Code Standards and Lint Rules"

If everything was setup correctly, you should be seeing a message like this

husky > pre-commit (node v12.11.1)
ℹ No staged files match any configured task.

If you see this message, congratulations you have setup linting and code formatting with commit hooks correctly 🎉 🎉 🎉

Push your changes to Github and let's continue with writing some code.

To find out what happens next, continue to Part Two : Breaking Wireframes to Components and setting up a Component Library

Forem: DEBOJIT ROY

Why do we need a Software Engineering Process

Why do I need a Software Engineering Process

What is Engineering Process

Engineering Process that works for me

Step 1: I will Design my solution before Implementation

Step 2: I will Document my solution before Development

Step 3: I will write code and explain

Step 4: I will test my code

Step 5: I will track my changes

React Project - Idea to Production - Part Four - Hosting the Movie app and setting up CI/CD

Where are we

What are we going to do now

TL;DR

Setting up Production Build

Preparing our Cloud Environment for hosting

Setting up S3

Create the Bucket

Allow Public Access

Make Read public on the bucket

Enable Static Website hosting for S3 bucket

Create a test index.html

Uploading the index.html

Testing our S3 website

Setting up Cloudfront

Why do we need a CDN

Creating Cloudfront Distribution

Adding rules for React Router

Verifying the changes

Why no custom domain

Setting up Continous Integration

CI Expectation

Setting up Continuous Delivery

Preparing the Deployment script

Important note regarding AWS Credentials

Defining Release Workflow

Checking everything works fine

React Project - Idea to Production - Part Three - Building the Movie App using component library

Where are we

What are we going to do now

TL;DR

Extracting common functionality in core

What resides in core

Setting up core

Building core

Adding axios

Initialising config variables

Adding search service

Setting up Web Application

WebApp Project setup

Setting up public assets

Setting up Webpack

Setting up shared configuration

Setting up the dev config

Building the common parts

Common - Components

Common - Config

Common - Redux

Common - Routes

Common - Utils

Building Features

SiteHeader - Components

SiteHeader - Redux

Getting the webapp running

Adding .env

Preparing launch script

Running Webapp locally

React Project - Idea to Production - Part Two - Setting up a Component Library

Where are we

What are we going to do now

TL;DR

Setting the Component Library

Defining the requirements

Why no Webpack

Preparing the project

Adding Project Dependencies

Adding React

Why one dependency at a time

Why is React a dev dependency 🤔

Adding TypeScript

Allow `Public` Access

Make `Read` public on the bucket

Enable `Static Website` hosting for S3 bucket

Create a test `index.html`

Uploading the `index.html`

Extracting common functionality in `core`

What resides in `core`

Setting up `core`

Building `core`

Adding `axios`

Initialising `config` variables

Adding `search service`

Setting up `public` assets

Setting up the `dev` config

Building the `common` parts

Adding `.env`