Forem: Asjad Anis

Retrieval Augmented Generation 101

Asjad Anis — Fri, 01 Sep 2023 00:37:43 +0000

I recently came across RAG (Retrieval Augmented Generation), at first it looked like a very fancy tech jargon but after learning more about it I found it to be a very interesting and clever technique to supplement an LLM in order to perform well on your custom data without hallucinating and going through the process of fine-tuning the model.

What Is Hallucination

Hallucination is a behavior exhibited by Large Language Models where the model generates a response that is incorrect and the model makes up that information. You might have observed this with chat-gpt when asking for very specific details about some library it might end up making up documentation and GitHub links for specific ways of doing things that are not supported by the library. For instance, I was searching on Google on how to extend the express request object type in typescript and it told me about some made-up extend method that it didn't even use

// Detect dark theme var iframe = document.getElementById('tweet-1694857443665125885-554'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1694857443665125885&theme=dark" }

How To Avoid Hallucination

There are a couple of techniques to prevent hallucinations when dealing with LLMs, we will be looking into two specific techniques for the sake of this post.

Fine Tuning
Retrieval Augmented Generation (RAG)

Fine Tuning

When fine-tuning the model you basically use the pre-trained model and then re-train additional layers or the last layers at the end of the model on the target data, keeping the model's pre-trained weights so the model's general understanding could be fine-tuned to the custom data set. However, this is computationally more expensive, time-consuming, and requires more expertise in the field of ML.

Steps Involved In Fine Tuning

Select a pre-trained model
Prepare your custom data
Tune hyperparameters
Fine-tune model using transfer learning
Evaluate the model

RAG Approach

RAG takes a different approach to augment the generative model output by supplementing the process with a retrieval system, since LLMs are already trained on a huge corpus of text they already have a very well understanding of the contextual and semantic meaning of the words, what they lack is the additional information from a custom knowledge base in order to come up with an educated response.

This is where RAG comes in and augments the LLM to prevent hallucinations by providing it with essential information in order to answer the query. Instead of passing the user query directly to the LLM it first goes through a retrieval system to fetch the relevant documents based on the user query which are then augmented on top of the user’s prompt and passed to the LLM, This makes it much more dynamic and flexible to provide external information to the model that it wasn’t aware of before in order to produce a much better response.

This approach also has the benefit of updating your repository of documents without having to retrain the entire model as in the case of fine tuning. You can also employ modern semantic search techniques and Vector DB to augment the retrieval system to get the best results from your knowledge base that can then be passed to the LLM. You can see the high-level overview of a RAG system in the below diagram

If you have any questions or feedback, feel free to reach out!

Connect with me on Twitter, Github, and LinkedIn

Optimizing React App Performance

Asjad Anis — Tue, 28 Sep 2021 12:39:06 +0000

Quick Summary

This article aims to explain how to write efficient and performant React components along with some common profiling techniques at our disposal that we can use to figure out unoptimized rendering behaviors in our app and improve the performance.

Audience

The targeted audience for this article is mid-senior level React engineers who are well familiar with the library and have a good understanding of how the library works especially the Virtual DOM, Reconciliation, and how the library renders and updates the actual DOM.

React is a great library that allows you to write applications declaratively, this approach is great because it abstracts away all the functional and internal details on how the library achieves a specific UI state and ensures to keep the DOM in sync with the state you describe. This is achieved by keeping a Virtual DOM and the process of reconciliation. Let's look at these two terms in order to understand them properly

Virtual DOM and Reconciliation

The Virtual DOM as the word itself suggests is basically a virtual representation of the UI, you can think of it as a blueprint that contains all the necessary details required to construct an actual DOM. React relies on the Virtual DOM to efficiently render only the components that have been updated. Any interaction with the application that is associated with some state will likely cause the application to trigger a re-render, but React does this part efficiently by only updating the Virtual DOM first instead of the actual DOM and then applying a comparison algorithm on the new and old Virtual DOM's to detect if an actual DOM update is required.
This diffing algorithm is basically what enables React to determine what DOM elements or attributes to update and makes it efficient.

You can read more about the diffing algorithm on the official React docs.

One of the main reasons for performance issues in a React app is some faulty implementation on our end or unnecessary re-rendering especially when its resource-intensive and doing some expensive calculations that causes repeated triggering of this diffing and rendering cycle and triggering updates to the actual DOM which can result in a performance hit and a slow experience.

In order to achieve good performance on our application, we need to ensure that React is only updating the components that are affected by the state change and ideally ignore all other components, this will result in saving up the wasted CPU cycles and resources that went in for re-rendering the unaffected components and give us a performance boost in our application.

Optimizing a React application without profiling or benchmarking won't do us much good as there is a cost associated with the optimization techniques and if not done correctly the performance gains might not be worth the complexity introduced in the codebase and could potentially affect the performance.

Let's get started with a very simple application that I have created and profile it along the way to see if the optimizations are doing us any good

// Clone the repo and switch to profiling branch

git clone https://github.com/asjadanis/react-performance-tutorial
git checkout profiling

Install node modules by running yarn and then start the application by running yarn start you should see something like below in your browser.

Now open up the browser console and play with the application add some books, courses and if you see some odd rendering behavior great, if you can't figure it out I'll break it down for you when you add a book you will notice that the courses list gets rendered too and vice versa. This is sub-optimal and not the desired behavior and we will be optimizing our components to ensure that only those components get rendered that are affected by the state change. Before we dive into profiling let us quickly go through the code so we know what we are dealing with.

// App.js

import { useState } from "react";
import List from "./List";
import "./styles.css";

function App() {
  const [books, setBooks] = useState([]);
  const [courses, setCourses] = useState([]);

  const onAddBook = (item) => {
    const updatedItems = [...books, { item, id: `book-${books.length + 1}` }];
    setBooks(updatedItems);
  };

  const onAddCourse = (item) => {
    const updatedItems = [
      ...courses,
      { item, id: `course-${courses.length + 1}` },
    ];
    setCourses(updatedItems);
  };

  return (
    <main className="App">
      <section>
        <h3> Books </h3>
        <List onAddItem={onAddBook} items={books} listKey="books" />
      </section>
      <section>
        <h3> Courses </h3>
        <List onAddItem={onAddCourse} items={courses} listKey="courses" />
      </section>
    </main>
  );
}

export default App

// AddItem.js

import { useState } from "react";

const AddItem = (props) => {
  const [item, setItem] = useState("");

  const onChange = (e) => {
    setItem(e.target.value);
  };

  const addItem = () => {
    if (!item) {
      return;
    }
    props.onAddItem(item);
    setItem("");
  };

  return (
    <>
      <input
        value={item}
        onChange={onChange}
        type="text"
        placeholder={props.placeholder}
      />
      <button onClick={addItem}> Add </button>
    </>
  );
};

export default AddItem;

// List.js

import AddItem from "./AddItem";

const List = (props) => {
  const { items } = props;

  console.log("List rendered: ", props.listKey);

  return (
    <>
      <AddItem onAddItem={props.onAddItem} placeholder="Add book" />
      <ul>
        {items.map((item) => {
          return <li key={item.id}>{item.item}</li>;
        })}
      </ul>
    </>
  );
};

export default List;

Our app is composed of three components first up is App.js that is our main component it contains the logic for adding books and courses and it passes down the handlers and books/courses states as props to the List component.
The List component provides input control to add books or courses using the AddItem component and maps over the list of books and courses to render them.

That's pretty straightforward, every time we add a book or a course we are updating the state in our App.js component causing it to render and its children. So far so good now we can jump straight into our IDE and fix this behavior but in this article, we are going to take a step back and first profile our application to see what's happening.

I pre-configured the repo with a nice package why-did-you-render which basically enables you to see any avoidable re-renders in your application during the development mode.

You can check out the package docs to see how you can configure it with your setup.

Note: Don't use this package in your production build, this should only be used in the development mode and should be in your devDependencies.

Profiling

First of all, you need to set up the React Developer Tools which is available as a browser extension and allows us to profile our React applications. You'll need to set it up for your browser in order to follow along with the profiling section, once you have it set up head over to the application on http://localhost:3000/ and open developer tools.

Now head over to the profiler tab and you should be able to see something like the below screenshot in your dev-tools

In order to profile our application for performance hits and see how rendering is happening we need to record our application while using it, let's do that. Hit the record button and then interact with the application add some books and courses and then stop recording. You should be able to see a flame graph of your app components and how long each component took to render out of the total render duration. The greyed-out components indicate that they did not render during that commit.

From here onwards you can step through the various commits in the graph and take notes of which components are taking the most time to render and if there are any wasted renders. The bars peak is a quick visual indicator for which commit took the most time to render and then you can click on it to further see each of the components that caused it. In our case, we can see a yellow peak followed by a couple of green peaks which indicate the rendering happening when we add a book or course.

Here we can see that our App component is rendering which makes sense as we are updating the state. Rendering both lists is although unoptimized as we can only update one list at a given time and we only want the respective list to render but in our case, both lists are re-rendering along with the AddItem component they compose of. Now that we have a clear picture of what's happening let's fix this behavior by wrapping our List component in React.memo which is a higher-order component that enables React to skip rendering for a specific component given the new props are the same as old props. Note that React.memo only compares props so if your wrapped component includes internal state, updating that will still cause the component to re-render which is desired.

Optimizing the Components

In order to fix this behavior head over to the List component and import memo from React and wrap the default export with memo

// List.js
import { memo } from "react";

const List = (props) => {
 ...
 ...
}

export default memo(List);

Looks good now let's give it a try keep your browser console open and add a book to the list you should notice that even after wrapping our component in React.memo both our lists are still rendering sound strange right? You should also notice some additional console logs telling us why the List component re-rendered as below

These console logs are coming from the why-did-you-render package that we talked about earlier which enables us to see any avoidable re-renders in our React app. Here it's telling us that the component re-rendered because of props changes and specifically the onAddItem function. This is happening because of the referential equality in JavaScript, every time our App component renders it will create new functions for our handlers and the referential equality will fail as both functions won't be pointing to the same address in memory this is just how JavaScript works. You should read more about referential equality in JavaScript in order to get a good grasp of this concept.

To fix this behavior in React what we can do is wrap our handlers in a useCallback hook which basically returns a memoized version of our handlers and it will change only if one of the provided dependencies changes. This will ensure that the new instance of our functions are not created and will prevent the re-rendering. Note here that memoization is not something specific to React but a general optimization technique used in programming to store results of expensive computations and return cached results when already computed.

Let's wrap our handlers in a useCallback

import { useCallback } from "react";

const onAddBook = useCallback((item) => {
    setBooks((books) => [...books, { item, id: `book-${books.length + 1}` }]);
  }, []);

const onAddCourse = useCallback((item) => {
  setCourses((courses) => [
    ...courses,
    { item, id: `course-${courses.length + 1}` },
  ]);
}, []);

We are passing an empty dependency list as we don't want our handlers to be re-initialized on every render but if required you can add the dependencies there, Let's run the application now and see how it behaves, if you add any book or a course now you will immediately notice that only the respective list gets re-rendered which is great but let's also profile it to see if we have gained any significant performance boost although our sample application is very simple and straight forward but if it were to be a bit complex consider each list item had a children array on it which can be further listed down and included some logic that was resource intensive and so on you can imagine in such a scenario the re-rendering would definitely be a problem. Below are the results after profiling you can give it a try yourself too.

We can see above that after memoization the total render duration for the highest peak in our flame-graph took about 2.8ms compared to 7.3ms before and our second List component did not render, this sounds great we have successfully saved around 4.5ms wasted in rendering time by putting in about 15-20 mins debugging, profiling, optimizing and the performance benefits in our case don't make any visual difference as the app is quite simple and doesn't take a lot of resources in re-rendering, but this doesn't mean we did this all for nothing the purpose was to understand the behavior and reasoning behind re-rendering and objectively approaching at optimizing the application instead of randomly wrapping everything in React.memo and React.useCallback. Now we have developed a basic mental model that we can use when approaching performance-related issues in a React app.

Another thing to keep in mind here is that React is smart enough to determine which DOM nodes to actually update, in our example above even though our List component is unnecessarily re-rendering React isn't triggering actual DOM updates unless necessary you can verify this is in your browser dev-tools and since React is taking care of the more expensive part i.e DOM updates in our simple example above we probably don't even need to optimize our components. Such optimizations are more fruitful when our components are expensive to render or include some expensive calculations during the rendering phase that is just wasting CPU cycles and is not required.

General Guidelines

Keep the below points in mind when using React.memo

React.memo does shallow comparison of props by default
You can pass in a custom function as a second argument to React.memo adding your custom logic to compare props.
If you need to do a deep comparison of props then keep in mind that it comes with additional cost depending on the complexity of your props.
It makes sense to use React.memo when your component renders the same thing when it's given the same props or if it's computationally expensive to render unlike our list component above.

If you are working with React class components you can use shouldComponentUpdate life-cycle method or React.PureComponent to achieve the same behavior but make sure you aid it with profiling.

You can use the useMemo hook to memoize any computationally expensive calculations on every render, make sure to provide a dependency array in case the memoized value is dependent on some other fields and needs to be recalculated if any of those fields is changed.

Conclusion

The objective of this blog was to build a mental model when approaching optimization problems in React applications and emphasize on profiling techniques to objectively achieve it. Optimization techniques come with a cost if not used properly and wrapping everything in memo or useCallback won't magically make your apps fast, but using them properly and profiling along the way could definitely be a lifesaver.
As always feel free to share your thoughts with me in the comment section or connect with me on Twitter.

TypeScript: Custom Type Guards

Asjad Anis — Tue, 07 Sep 2021 08:46:50 +0000

Did TypeScript ever annoy you with the error Object is possibly undefined even if you filter out all the undefined values, if yes then this post is for you.

Code Setup

Let's quickly setup a very basic example to see how we can get to such a situation.

interface Item {
  id: string;
  name: string;
  description: string;
}

const myItems: Array<Item> = [
  { id: "1", name: "item-1", description: "bla bla bla" },
  { id: "2", name: "item-2", description: "yada yada yada" },
  { id: "3", name: "item-3", description: "bla bla bla" },
  { id: "4", name: "item-4", description: "yada yada yada" }
];

const itemIds: Array<string> = ["4", "5", "2", "6"];

const getItems = (id: string): Item | undefined => {
  return myItems.find((item) => item.id === id);
};

const items = itemIds.map(getItems);

console.log(items);

In the above example we have an array of Items and list of itemIds , and we are trying to get the metadata of the itemId . When you run this code and hover over items , TypeScript will tell you that the variable items is of type (Item | undefined)[] which makes sense as some id's won't match and return undefined so far so good, now the undefined items in the array are of no use to us and we don't want to deal with the undefined errors when further referring the items variable in our codebase, so lets quickly fix this by filtering out all the undefined values.

const items = itemIds.map(getItems).filter(item => !!item);
console.log(items) // No undefined values in items

Great now we can easily loop over the items array and process it without having to deal with undefined values, let's quickly try it.

const processItem = (item: Item) => {
  return {
    ...item,
    name: item.name.toUpperCase()
  }
}

const processedItems = items.map(item => {
   return processItem(item) // Argument of type 'Item | undefined' is not assignable to parameter of type 'Item'
})

Wait but why? we already handled this above, let's hover over items and see it's type

const items: (Item | undefined)[] // Wait whaaat

The problem is that the .filter method will always return an array of the type with which you initially started so in our case it was Item | undefined which makes sense now but how do we fix this 🤔

Type Guards to the Rescue

This is where TypeScript custom type guards come to rescue, a custom type guard is basically a function that's return type is a type predicate. It's a very simple and intuitive approach for fixing the above problem properly, let's see it in action.

// Our type-guard
const isItem = (item: Item | undefined): item is Item => {
  return typeof item !== "undefined"
}

const items = itemIds.map(getItems).filter(isItem); //items: Item[]

And that's how you can filter out the undefined values properly in TypeScript with type guards and keeping the types intact in your codebase. This same approach can be extended to a use-case where you are dealing with a Union type and you want to make sure you are accessing the right properties of the instance of that type.

As always do share your thoughts on this approach in the comments below and feel free to connect with me on Twitter.

Essential JavaScript Tooling

Asjad Anis — Fri, 03 Sep 2021 21:45:30 +0000

Quick Summary

Software development is great and fun but it can be a mess and a very bad influence if the developer experience is not given the importance and prioritized. Some standard development practices along with proper tooling can greatly improve this experience. This also aids in keeping the codebase clean and the repository in a good health. In this article, we will cover some development practices and essential tools to help you improve the development experience.

Audience

This article is essentially for JS developers and covers tooling around the JS ecosystem, however, the same concepts can be applied in other languages with the tooling available there. We will cover the below topics in this article.

Code Linting using ESLint.
Code Formatting using Prettier.
Setting up Git Hooks for linting and code-formatting.
Conventional Commits using Commitizen.
Github Actions for CI/CD

Code-Linting

Code linting is very essential and important especially when you are working with an interpreted language like JavaScript. Linters are essentially static code analyzers it scans through your code without running it and flags any programmatic errors, stylistic errors, and any suspicious contracts. Properly setting up a linter in your codebase can help catch errors early on and ensure that some standard practices are being followed. This makes the codebase clean and helps in code reviews.

Linters can enforce code styling and rules like no unused variables or no console.log statements or no unused imports in fact it's a very powerful tool and can do much more than that.

Setting up Linters can be tedious and tiring but it brings great results once you adopt it and start using it properly in your team. I personally believe that linters and such tools should come with the language itself and developers shouldn't have to worry about setting one up. This experience is much more improved with deno (A secure runtime for JavaScript and TypeScript) which ships with a built-in linter and formatter for JavaScript and TypeScript.

Now that we have a basic understanding of linters let's see how we can set one up in a TypeScript project. We will be using ESLint which is very popular in the JS ecosystem and is fully pluggable. Every single rule in ESLint is a plugin, this means you can start off with a base configuration and then extend the rules to your needs by adding in more rules as a plugin.

Create a new folder and run npm init -y to initialize your project, this will create a package.json in the root directory with some stub fields.

Next, install the required dev-dependencies into the project.

yarn add -D nodemon ts-node eslint typescript @typescript-eslint/eslint-plugin @typescript-eslint/parser

Let's first quickly setup TypeScript by running tsc --init which will initialize the tsconfig.json file in the project root, the only thing we are going to change is uncommenting outdir option and change it to dist

"outDir": "dist"

Next, add a .eslintrc file in the project's root directory. This is going to be the configuration file for eslint where we can customize our rules. To quickly get started let's add a bare minimum configuration to the file.

{
  "root": true,
  "parser": "@typescript-eslint/parser",
  "plugins": ["@typescript-eslint"],
  "extends": [
    "eslint:recommended",
    "plugin:@typescript-eslint/recommended",
  ]
}

Let's look at some of these options briefly, the root key basically tells eslint to stop looking for configuration files in the parent directories. By default, eslint looks for config files in all parent folders until the root directory which can lead to unexpected results, plus this option can be very handy when you have a monorepo in place and each project uses a different configuration.

The extends key lets you add the configuration you want to use for your project this can be a string that specifies a configuration or a path to a configuration file, here we are using two configurations first one is the eslint-recommended, and the next up is typescript-eslint recommended. You can also use airbnb-typescript config which is also very popular.

Now let's add some scripts in our package.json to run our project in dev mode and a script for linting.

"dev": "nodemon --watch '**/*.ts' --exec 'ts-node' src/index.ts",
"lint": "eslint . --ext .ts"

And for the sake of testing our eslint setup let's create an index.ts file and very simple for loop with an intended unused variable.

//src/index.ts

const range = 10;

for (let i = 0; i < 10; i++) {
  console.log("i : ", i);
}

Now let's run yarn lint and we will get a warning on our console.

This is great our eslint setup is working, but let's say we want to be more strict on our code base and want this to be an error and cause lint to fail, to do that head over to your .eslintrc file and add the below rule.

{
  "root": true,
  "parser": "@typescript-eslint/parser",
  "plugins": ["@typescript-eslint"],
  "extends": ["eslint:recommended", "plugin:@typescript-eslint/recommended"],
  "rules": {
    "no-unused-vars": "error"  // "error" | "warn" | "off"
  }
}

Now if you run yarn lint your console will display this as an error and lint won't succeed, also if you have the eslint extension installed on vs code it will highlight this issue.

We can also add another script to auto-fix the lint issues, note that not all issues can be auto-fixed and you can get a list of all the rules that can be auto-fixed here.

"lint-fix": "eslint . --ext .ts --fix"

That's pretty much for the linting setup, I hope you can now create a mental model for yourself and see how useful this tool can be if used properly.

Code Formatting

Let's accept it, seeing different code styles in a repo is a mess and if not controlled you can see all sorts of code conventions being followed in the repo. This is where code formatters come to our rescue, but before we jump into setting one up in our project we first need to understand that code formatters and linters are essentially not the same things, although there is a very thin line between them but they both serve a different purpose. Code-formatters essentially deal with the formatting of code and apply the code styling you have specified, they don't check code for potential bugs or issues and adhere to the styling.

Prettier is a very famous tool in the JS ecosystem and we will be setting it up in our project, with prettier we can enforce style rules like max-length, tab width, single-quotes, semicolon, etc. It's basically an opinionated code formatter that takes your code and parses it into an AST (Abstract Syntax Tree) discarding the original styling, after that it just pretty prints the AST with the code-styling you have specified.

Let get started, first we need to install the dev dependencies

yarn add -D prettier eslint-config-prettier eslint-plugin-prettier

Let's quickly explain the above dev-dependencies and what purpose do they serve.

prettier: opinionated code-formatter.
eslint-config-prettier: used to disable all eslint rules that might conflict with prettier.
eslint-plugin-prettier: runs prettier as an eslint rule

Now create a .prettierrc file in the root of your directory and add the below config.

//.prettierrc

{
  "semi": true,
  "trailingComma": "none",
  "singleQuote": false,
  "printWidth": 120,
  "tabWidth": 2,
  "arrowParens": "always",
  "bracketSpacing": true
}

These rules specify to put semicolons at the end, remove trailing commas, use double quotes for strings, etc, you can read more about the rules here and set them up to your preference.

Now that we have the rules set up, let's add a script to format our code.

"format": "prettier --config .prettierrc 'src/**/*.ts' --write"

That's it we have configured prettier in our project, now whenever you run this script it will format all your source code in the src directory according to the config you defined. Give it a test, use some single quotes or remove semicolons and then run yarn format. You can also install the prettier vs code extension and set it to format on save.

Now that we have prettier setup let's configure it with eslint, update your .eslintrc as below and you'll have prettier working with eslint and configured as a plugin in eslint config.

{
  "root": true,
  "parser": "@typescript-eslint/parser",
  "plugins": ["@typescript-eslint", "prettier"],
  "extends": [
    "eslint:recommended",
    "plugin:@typescript-eslint/recommended",
    "prettier"
  ],
  "rules": {
    "no-unused-vars": "error",
    "prettier/prettier": "error"
  }
}

Git Hooks

Cool, if you have been following till here this is where it gets the most exciting, you must be wondering it's great that we have set these tools up, but someone can still commit to the repo without running these scripts, this is where git hooks come to our rescue. Git hooks are just scripts that run automatically every time an event occurs in the repository. We will use a tool called husky that makes working with git-hooks easy.

Let's quickly install it as a dev dependency.

yarn add -D husky

// Also add a prepare script to your package.json as below.

"prepare": "husky install"

// Now run

yarn prepare

prepare is an npm lifecycle script that will run on npm install and this will ensure that whenever node modules are installed husky is installed too. Read more on life-cycle scrips here.

That's it, husky is now set up in our project, and we can configure it to run our lint and format scripts before commit, for that we can use the pre-commit hook that runs before a commit. Let's add that hook now, head over to the terminal, and run the below command.

npx husky add .husky/pre-commit "yarn lint && yarn format"

You will notice that now there's a .husky folder in the root of your repo and it contains a file pre-commit with the below contents. This hook will run every time you commit to your repo.

#!/bin/sh
. "$(dirname "$0")/_/husky.sh"

yarn lint && yarn format

Try committing something in the repo now, and you will notice that lint and format scripts are executed first. If everything looks good, your commit will be added successfully, in case of issues it won't be committed to the repo and you'll have to fix those issues in order to make a commit. You should be able to make a mental model of this now, this ensures that no code with potential bugs or styling issues gets committed to the repo. This can help you save a lot of time in code reviews if these conventions are properly followed.

This configuration is great but if you notice there is one problem with this approach, if we have a very large codebase and we run linting and formatting on every commit in the code-base it can take up much longer and that's not the intended use case, we only want to run it on the files that have been staged for the commit because ideally only those files should be checked that are being committed to the repo, that's where another tool lint-staged comes to our rescue and it will ensure that our pre-commit hooks only run on staged files. This can be configured very quickly too.

yarn add -D lint-staged

// Add below config to your package.json

"lint-staged": {
  "src/**/*.ts": [
    "yarn lint",
    "yarn format"
  ],
},

And update your pre-commit hook to run npx lint-staged instead of running yarn lint && yarn format

#!/bin/sh
. "$(dirname "$0")/_/husky.sh"

npx lint-staged

That's it, now lint will only run against the staged files, you can quickly test it by creating a file in the repo with some lint-issues and don't stage it while staging some other files that are compliant with the lint rules and you should notice that git commit will run fine, but if you stage the file with the lint issues, it will block the commit and give you the lint errors on the terminal that should be fixed. If you want to lint the entire project you can still do it by running yarn lint. Now our tooling is set up well enough to ensure that the whole team can follow it and adhere to the same coding standards and style guides.

Conventional Commits

Our development setup is pretty much complete but there's still one area where the team can adopt different conventions, that will be the commit messages yes that's right people can have different preferences when adding commit messages and we need to ensure that the team conforms to a standard convention. We will be adopting the conventional commits specification in our project and ensure it using a tool named commitizen

The general syntax for this specification is as follow

<type>[optional scope]: <description>

[optional body]

[optional footer(s)]

// Example

feat(landing-page): add new landing page

A new landing page for the website...

Closes #<github-issue-number>

Alright now that we have an understanding of the specification let's configure our project to use this tool.

yarn add -D commitizen cz-conventional-changelog

And add this to your pacakge.json

"config": {
  "commitizen": {
    "path": "cz-conventional-changelog"
  }
}

"scripts": {
  "cz": "cz",
    ... 
}

Note: If you end up naming your script commit then it might run twice because of this issue mentioned on the repo.

Now try committing to the repo by running yarn cz and you will be prompted a couple of questions on the terminal to make the proper commit. Great now our conventional-commits are set up we can also set it up as a git hook by running the below command and running commitizen with our prepare-commit-msg hook which is invoked by git commit however there's an issue with this approach which triggers the git commit twice when running yarn cz. The issue is opened on the repo here, I would advise not to use this approach until this issue is fixed and rely on the previous script yarn cz

npx husky add .husky/prepare-commit-msg "exec < /dev/tty && node_modules/.bin/cz --hook || true"

Github Actions

In the last part, we will focus on setting up a GitHub action to ensure that our lint and format jobs are run on every commit and how Github actions can help in our CI/CD pipeline. Setting up a good and fast CI/CD pipeline is very essential in modern software development now. As your software evolves it might become a very tedious task to compile the builds manually and if it needs to be compiled on multiple platforms you can imagine how time taking this could be.

Github actions are a handy tool to automate software workflows and provides CI/CD right out of your Github code repo. Github actions are event-driven and basically require a .yaml file where you can provide your configuration in steps.

Some key terminologies to keep in consideration when working with Github Actions include

Workflows: automated work procedures that contain one or more jobs.
Events: an activity that triggers a workflow (e.g push to a branch)
Jobs: a set of steps that execute on the same runner.
Step: a task that can execute commands on a job.

You can read more about these on the official docs.

First, initialize a git repo in your project if you haven't already by running git init and commit your changes to the repo. Make sure you add a .gitignore file and add node_modules and dist to it so they don't get committed to the repo.

Next, create a repo on your Github account and copy the remote's origin url, now head over to your local repo and run

git remote add origin <remote-repo-origin>
git push -u origin <branch-name>

Next head over to your Github repo and click on the Actions tab and select set up a workflow yourself

In the editor name the file lint.yml and clear the default action and replace it with.

name: Lint

on: [push]

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Setup Node.js environment
        uses: actions/setup-node@v2.4.0
        with:
          node-version: '14.16.1'

      - name: Install Node Modules
        run: npm install

      - name: Lint and format
        run: |
          npm run lint
          npm run format

This action file is pretty straight forward and you should be able to navigate through it very easily, we are declaring our workflow name as Lint that has a single job lint. A workflow can have multiple jobs and by default, jobs run in parallel but can be configured to make them run sequentially by using the needs <job-id> syntax in the .yml file

Next, we are specifying to run the job on Ubuntu's latest version and set up Node v14.16.1 using a marketplace action. After that, we are just installing node modules and running our lint/format scripts.

Note this is a very simple example but you can extend it to your needs, for example, let's say you have set up tests in your repo then you can define another test job that runs after linting, and if the test job succeeds, you can run a build job to compile a build and deploy to staging. You can basically configure it to adapt to your needs and automate your development workflow. Github actions are indeed a powerful tool and you should definitely explore them.

You can see all of this in action on my GitHub repo here

Conclusion

The idea with this blog was not to go into the depths of each tool but rather give you an overview of all these tools that can help you in our development process and ensure a better development experience. Setting up these tools can be a very boring and cumbersome task but these tools are your friends and once you adopt them properly in your code-base you won't regret it. Feel free to share your thoughts with me in the comment section or connect with me on Twitter.

Parsing .env with TypeScript

Asjad Anis — Sat, 28 Aug 2021 12:39:08 +0000

When working with Node.js its a common practice to keep our credentials and secrets in a separate .env file, that's never pushed to our repo. In order to access these variables in our code, we simply use dotenv package to parse the .env file and load our env variables into process.env.

Example

Here's a quick example on how to do this in plain JavaScript.



// .env

API_KEY = <YOUR_API_KEY>
DB_URI = <YOUR_DB_URL>
....

Now to access these variables we can do something like



// index.js

const dotenv = require('dotenv');

dotenv.config()

const connectDB = () => {
  ...
  mongooose.connect(process.env.DB_URI)
  ...
}

Since JavaScript doesn't care much about the type so we can pretty much access any property on process.env and our code won't give us any red signs untill we run it and find out that process.env.DB_URI is undefined.

Now let's say we had to do the same thing in TypeScript, and keep it fully typed and have all the benefits that come with a typed system. In order to keep it fully typed we will have to first extend the ProcessEnv Interface to have our custom env variables available on the interface. For this we will have to use Declaration-Merging which is a TypeScript feature for combining two or more declarations with the same name.

You can have a quick overview of this feature and its use case on this twitter thread

// Detect dark theme var iframe = document.getElementById('tweet-1431000236269477889-457'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1431000236269477889&theme=dark" }

We will use the same feature to extend the ProcessEnv interface with our custom env varibales.

Create a file named global.d.ts with the below contents



namespace NodeJS {
  interface ProcessEnv {
    NODE_ENV: string;
    PORT: string;
    MONGO_URI: string;
  }
}

Above we are just extending the ProcessEnv Interface that's accessible under NodeJS namespace and specifying our custom env varibales types.

Now if you write process.env in your code, intellisense would autosuggest you the fields avaialable on process.env which is great process.env.MONGO_URI is no longer a mystery in our code.

Make sure you specify the path to this file in your tsconfig.json

A better approach would be to have a folder lets say extended-types and keep all your extended types there and just specify the path to that folder here instead of a single file.



"typeRoots": ["./global.d.ts", "./node_modules/@types"]

Now let's create a config.ts file that will encapsulate the logic for parsing .env file and converting it into our own custom Config type that we want to use in the code.



import path from "path";
import dotenv from "dotenv";

// Parsing the env file.
dotenv.config({ path: path.resolve(__dirname, "../config/config.env") });

// Interface to load env variables
// Note these variables can possibly be undefined
// as someone could skip these varibales or not setup a .env file at all

interface ENV {
  NODE_ENV: string | undefined;
  PORT: number | undefined;
  MONGO_URI: string | undefined;
}

interface Config {
  NODE_ENV: string;
  PORT: number;
  MONGO_URI: string;
}

// Loading process.env as ENV interface

const getConfig = (): ENV => {
  return {
    NODE_ENV: process.env.NODE_ENV,
    PORT: process.env.PORT ? Number(process.env.PORT) : undefined,
    MONGO_URI: process.env.MONGO_URI
  };
};

// Throwing an Error if any field was undefined we don't 
// want our app to run if it can't connect to DB and ensure 
// that these fields are accessible. If all is good return
// it as Config which just removes the undefined from our type 
// definition.

const getSanitzedConfig = (config: ENV): Config => {
  for (const [key, value] of Object.entries(config)) {
    if (value === undefined) {
      throw new Error(`Missing key ${key} in config.env`);
    }
  }
  return config as Config;
};

const config = getConfig();

const sanitizedConfig = getSanitzedConfig(config);

export default sanitizedConfig;

Now you can simple import config.ts in your code and access the fields and it would be super cool to have the intelllisense give you type inference about the fields present on config object.



import mongoose from "mongoose";
import config from "./config";

const connectDB = async () => {
  const connection = await mongoose.connect(config.MONGO_URI);
  console.log(`🟢 Mongo db connected:`, connection.connection.host);
};

I hope this post helped you and you can extend the same concept to other similar use cases when working with TypeScript.

Feel free to drop any suggestions or improvements on my approach for this, and you can always connect with me on twitter

Why you should always use virtualenv when working in Python

Asjad Anis — Sun, 13 Dec 2020 21:49:29 +0000

If you have been developing in Python, you must have had a tough time with dependency conflicts when working on multiple projects, and it's kind of inevitable you have to face this issue at least once in your python development career before you look for solutions and come across the term virtual-environments or virtualenv.

If you have faced such issues then this post is exactly for you to give an understanding of how python run time works and why virtual-environments are needed.

Pip's Default Choice

Pip is the default package manager for python just like npm is for Node, and by default, pip will install any package globally and this is where things start to get a little out of hands. Now imagine you are working on two projects both of these projects have a common dependency, and for the sake of simplicity, we will call it dependency A. Project 1 requires version 1.0.0 of dependency A and Project 2 requires version 2.0.0 of dependency A and since at any given time the python interpreter can only maintain one version of the dependency A we won't be able to run these two projects without having to manually install the correct version every time we switch between projects which will soon become a hindrance in our development flow. This was a very basic example now imagine a specific version of dependency A is dependent on a specific version of dependency B and you can kind of get an idea on how this could potentially break other projects.

Virtual-Environments to Rescue

This is where Virtual Environments come to play and facilitate us in creating isolated environments for each project. Virtual Environment is basically a fancy term for a directory within your project that encapsulates executables and packages that are required to run the project. Virtual Environment contains links to the python interpreters that are installed on the disk and this essentially means that by tweaking this property you can define which python-runtime you want to run your project with, sounds cool right let's see how we can set up a virtual environment in our project.

Steps

pip3 install virtualenv.
cd < project-path >.
virtualenv < name-of-your-env >. (Generally a good practice to name it venv)
source ./< name-of-your-env >/bin/activate. (To activate your environment)
deactivate (To deactivate)
virtualenv -p < path-to-python-interpreter > < name-of-your-env > (To switch to python interpreter of your choice)

And that's it once the virtual environment is created you can install any package and that will be installed locally within that project without affecting your global packages and creating dependency conflicts.

Now if you use a modern IDE like PyCharm you would have noticed that every time you create a new project PyCharm creates a virtual environment for you and configures it, and this is exactly what's going on behind the scenes. Now that you know what virtualenv is why its a handy tool when developing in python you should always try to use virtual environments, this would greatly increase your productivity and save you a lot of time that is spent figuring out dependency conflicts.

Food for thought

If you have had the chance of working in Node or Javascript you must have noticed that npm deals with this very nicely by default installation in the local space rather than global-space plus the package.json file helps maintain the dependencies and differentiates between dev-dependencies and production dependencies in an elegant manner. Managing dependencies for a python project with a requirements.txt file can become very tricky when your dependencies have sub-dependencies etc. Although both have their pro's and con's I have always wondered why pip installs packages globally and why there isn't any project configuration file when working in Python. Let me know in the comments if you know why thing's happen like this in python.

If you like this post do share it with your friends.

Follow me on Twitter
Add me on LinkedIn