Forem: Nishu Goel

The ü/ü Conundrum

Nishu Goel — Mon, 12 Feb 2024 01:23:23 +0000

I implemented search and filtering for entities on our product at epilot. The users were heavily using the feature, however, a unique issue surfaced: difficulties in filtering file names containing diacritical marks like umlauts (Äpfel, über, schön, etc.).

Intrigued, I delved into the logic to investigate. Initially, everything seemed in order. For instance, a file named “blöb” was saved precisely as “blöb” — no bizarre encoding alterations were apparent.

Can you spot any difference between “blöb” and “blöb”?

This time instead of typing the word in the search bar, I copied it from the uploaded and saved filename, and and voilà, the search functioned perfectly! 🤯. This definitely meant the file name was subtly altered upon upload, hinting at a background encoding transformation.

So, as anyone would, I employed the power of encodeURIComponent to see for myself what really was different.

There it was! The different types of encoding forms that led to this. Now you might be wondering (I was wondering too!):

As if worrying about encoding wasn’t enough, now we’ve got to stress over its different types too 🤦‍♀ and wait, its 2024, and we are still grappling with Unicode character encoding problems?

The culprits in this case were these two contrasting encoding types: %C3%BC and %CC%88, leading to the search malfunction. This is a well-known issue with NFD/NFC encoding forms.

There is a detailed explanation here.

Now imagine explaining this technical jargon that to a user! 😃

To fix it, I aligned the encoding type of the searched filename with that of the saved filename.
On MacOS, the encoding was NFD (for ö: 111 + 776), whereas on Linux, it switched to NFC (ö: 246).

The solution?

Post-upload, before saving the file as an entity, I applied .normalize() to the filename, ensuring it was uniformly searchable.
Additionally as a next cleanup step — to write a migration script to normalize the names of all existing files with non-ASCII characters.

A simple string normalise proved effective in this case. ✅

Read more about String.prototype.normalize() here. There is also specific normalization option to pass to ensure canonically equivalent normalization — for example: string.normalize("NFC")

Here’s a visual representation of the problem with the solution output:

files umlaut.mov

Hope it helps!

How using Server-Timing API helped bring > 70% perf improvement

Nishu Goel — Tue, 23 Jan 2024 00:27:12 +0000

(Originally posted on Web performance Calendar)

When working on the web, we all end up coming across situations where the experience for the user seems a bit janky. This happens either when you are dogfooding, or the users actually highlight facing that slowness, or you see that for yourself in your performance metrics or tools.

What one’d do in such a situation is look at the obvious things first: are the images loaded lazily? Is the bundle size too big? Is the main thread blocked? How is the performance panel recording looking? Is the data coming from the backend taking more time than it should?

If it’s one of the first problems, we’d fire up our favorite devtools and try to find out the root cause. If it’s the data being slow, we could see how long that is taking in the network requests duration. And, if it does end up being slow from the backend, the next step would be to dive into some logs and checking what piece of code in the request endpoint takes the time, or embedding some performance information in the response, or using other performance tools etc.

We had a similar situation at epilot where we use Elasticsearch to look for all related products of an order for example, and also all the other entities the order itself might be a relation to.

This meant, looking for direct relations of the order but also the reverse relations of that order.

Now in the Elasticsearch world, this meant:

Getting the list of related products of the order.
Searching through the documents in elasticsearch for all those related IDs.
Also, searching all other docs for having this one order as a relation to this (the reverse relation example, I shared above).
Finally returning all the found relations.

Showcasing two products as relations of an order

This worked great for the first few years and got us exactly what we needed quickly. However, as we scaled, the amount of documents indexed in ES increased too, and that meant, the step 3 above started getting expensive and slow! The logic was to look for an entity ID in all the indexed documents and find if it is a relation or not. For orders with a lot of products, the endpoint would respond in seconds.

We had to fix the logic/ update the design asap.

The first solution was to update the ES query to make it more specific about what is being looked up, and then check if it improved with that query change and how the results looked for the entities with many relations. This meant looking into a crazy amount of Cloudwatch logs and looking at the total time taken and if it was any better.

We made a change to search query to be an IDs specific query which would look through only the IDs in the indexed docs, and not through all the fields in those ES documents.

GET /_search
{
  "query": {
    "ids" : {
      "values" : ["1", "4", "100"]
    }
  }
}

The expectation was this would completely solve the problem and be super quick. This wasn’t the case unfortunately.

What we were missing was to see which part of the code was really adding to the time taken by the endpoint. We needed to observe the different steps in the process and see all the sections where the time taken could be reduced. We needed to see if it was really ES that was the culprit.

This is where Server-Timing API came into picture. Something that would break down the request into pieces and give us an idea about what’s expensive. The API allowed to pass the timings specific to a request from the backend to the browser. This meant that we could now see in the network requests, how the response times looked like, which part of the function took more time and so on.

To support this, we need to send a Server-Timing response header from the backend and time the methods that we needed. Something like this:

HTTP/1.1 200 OK

Server-Timing: db;dur=53, app;dur=47.2

The Server-Timing header can send three bits of information:

metric name
duration
description of the metric
In the above example:
db — metric name
dur=53 — the time in milliseconds the db metric took (This is for instance, the time to fetch some data from the Database)
app — another metric name and dur=47.2 being the time taken for the same.

The header can take multiple metrics separated by commas delivering great information and is super lightweight. Although it is recommended to keep the metric names as small as possible.

An example of what the returned Server-Timing header looks like:

db; dur=142.715967; desc="getRelationsForEntity",flt; 
dur=36.777609; desc="filterEntityListByAccess",pgn; 
dur=22.96549; desc="getPaginatedRelations",hyd; 
dur=0.64605; desc="hydrateRelations",ddp; 
dur=0.13311699999999999; desc="dedupeRelations",total; dur=721.388338

And this information then translates into a more visual and helpful view as a part of the network timings tab of the request OR available via the PerformanceServerTiming interface.

To access the three available properties, they correspond in the PerformanceServerTiming as:

“name” -> PerformanceServerTiming.name
“dur” -> PerformanceServerTiming.duration
“desc” -> PerformanceServerTiming.description

Here’s the working draft with more information.

These are not available as a series of timings as such and just simple metrics and therefore, do not show in a waterfall pattern.

This means we can now tackle both the frontend performance problems and the backend slowness problems all in one place. No jumping to the logs to find a response was slow or heading backing again to the devtools to find the frontend problem.

We could also read the data now available using the navigation (PerformanceNavigationTiming) and resource timing (PerformanceResourceTiming) APIs and send it our analytic tools to create metrics/monitors.

const performanceObserver = new PerformanceObserver((list) => {
  for (const entry of list.getEntries()) {
    log('Server Timing', entry.serverTiming);
  }
});


performanceObserver.observe({type: 'navigation', buffered: true});

Below is one of the examples from a method taking different amounts of time at different steps in the request. We will set up together the whole setting up and measuring the server response times later in this article.

This was especially helpful for the Elasticsearch example that I shared earlier as we needed to know which change was improving what part of the implementation and if it was moving us closer to improved response times. The Server-Timing header does not directly improve the timings but helps find out what could be worked towards.

The uniqueness of the API is that its super flexible and allows the server to communicate important information to the browser, even other than the server timings. It would be amazing to have this supported on Safari as well, which currently lacks the API support.

Coming back to how we came down to the API response in 400ms seconds as compared to 1.8s earlier

These are the two versions of the API. When we measured the performance with server-timings on the ES implementation after switching to the IDs search query on Elasticsearch, the performance seemed to be taking all the time, especially on this function searchRelationsForEntity.

Now we knew this was to do with ES but where exactly. We timed specifics inside this method.

On measuring further, it was entitySearch.searchEntities that was taking up the most time so we needed to find a solution that would not go through all the ES documents at all to get the relation entities.

This is where changing the design made sense, and we moved to a graph table design with Dynamo, which would store all the relationships as mappings, and return all the entity IDs whether as direct relations or reverse.

Now we could choose if we wanted to search ES with all the IDs that we have or just batch get the entities from the db, and support pagination (this was an added advantage with ES) etc. separately.

Now, this complete dynamo approach saved us looking through all the indexed documents for the IDs that we need, and also cut some slack on Elasticsearch which would have this unnecessary load, eventually affecting other straightforward searches as well.

This is how the server timings looked with the DynamoDB approach:

This was an amazing improvement, clearly visible with this significant performance information right in the browser devtools for us. This also ended up helping improve the code in general, to gain some more milliseconds by refactoring, and using performant ways to filter/hydrate/paginate things etc.We love to see the output of what’s in the works, and this was a great example to look at and feel a sense of achievement for.

Implementation of Server-Timing as a middleware for AWS Lambda

Now if you look up the existence of libraries supporting server-timing header, there are so many (server-timing, server-timings etc.) but most of them for Express and NodeJS.

We needed to create one for the serverless use case which could be used for AWS Lambda, lets say to use with middy for example.

Step 1. Create a withServerTimings() middleware to use for AWS lambda

import middy from '@middy/core';
import * as Lambda from 'aws-lambda';

export const withServerTimings = () => {

  return {
    // add Server-Timing header to response
    after: (handler: middy.Request) => {
      const response = handler.response as Lambda.APIGatewayProxyStructuredResultV2;
      try {
        const headers: unknown[] = [];
        /**
          This is where the headers with different metrics are be saved
          and retrieved from
        */
        const timings = getServerTimingHeader(headers);
        response.headers = {
          ...response.headers,
          'Server-Timing': timings,
        };
      } catch (e) {
        Log.debug(`Middleware Error: Could not record server timings`, e);
      }
    },
  };
};

This creates a usable withServerTimings() middleware to then be used as:

Step 2. Apply the middleware to the request handler

return middy(handler)
  .use(...)
  .use(withCors())
  .use(withServerTimings())
  .use(...)

This already ensures that your backend will now be sending a Server-Timing header in the request endpoints.

Next step would be to actually time our operations and store them as a common timing variable which can then be sent to the getServerTimingHeader method for example.

Step 3. Implement timing for your methods

const times = new Map<string, Record<string, unknown>>();

// starting the timer on a method
const startTimer = (name: string, description?: string) => {
  times.set(name, {
    name,
    description: description || '',
    startTime: process.hrtime(),
  });
};

// ending the timer and recording the duration of the method
const endTimer = (name: string, description?: string) => {
  const timeObj = times.get(name);
  if (!timeObj) {
    return console.warn(`No such name ${name}`);
  }

  const duration = process.hrtime(timeObj.startTime as [number, number]);
  const value = duration[0] * 1e3 + duration[1] * 1e-6;

  timeObj.value = value;
  times.delete(name);
  return timeObj;
};

// finally setting the metric value received from endTimer object
const metric =
  typeof description !== 'string' || !description
    ? `${name}; dur=${dur ?? 0}`
    : `${name}; dur=${dur}; desc="${description}"`;

This would then be used as:

Step 4. Usage of timers of the methods

startTime('db', 'getRelationsForEntity');
const relations = await getRelationsForEntity({
  ...
});
endTime('db', 'getRelationsForEntity');
  .
  .
  .
startTime('pgn', 'getPaginatedRelations');
const { relations: paginatedRelations, hits } = getPaginatedRelations({
  ...
});

endTime('pgn', 'getPaginatedRelations');

And there we go, we will have the timed methods and the server headers looking like:

db; dur=139.99; desc="getRelationsForEntity",flt; dur=36.68; desc="filterEntityListByAccess",
pgn; dur=39.37; desc="getPaginatedRelations",hyd; dur=0.61;
desc="hydrateRelations",ddp; dur=0.130; desc="dedupeRelations",
total; dur=741.53

All the timing implementation and setting up the middleware can be a hassle, and should be a one-time job. I created a package out of this implementation called lambda-server-timing:

lambda-server-timing (npmjs.com)

GitHub – NishuGoel/lambda-server-timing (github.com)

withServerTimings to use a middleware
startTime to start measuring a function/set of functions
endTime to end the timing of the function/set of functions and get the metric set.

import {withServerTimings, startTime, endTime } from 'lambda-server-timing';

Let’s time our functions down!

References

Working draft
MDN Server Timing
Navigation Timing
Resource Timing

Spin up a new Elasticsearch cluster?

Nishu Goel — Tue, 08 Aug 2023 12:53:24 +0000

This post explains how to setup a new elasticsearch cluster from scratch and make it ready to be used for indexing data into, communicating to/from as a remote cluster, along with setting up monitoring for the new cluster.

Problem 1: Create a new elasticsearch cluster on elastic.co based on the required configuration for your use case.

To create a new cluster from scratch:
Create a main.tf file with following resources:

a. ec_deployment - The main cluster deployment resource
b. aws_secretsmanager_secret- The cluster secrets
c. aws_secretsmanager_secret_version - The cluster secret version needed for the secrets
d. elasticstack_elasticsearch_cluster_settings - To add default settings to the cluster like cluster nodes etc.

Example: ec_deployment

resource "ec_deployment" "<cluster-name>" {
  name = "<cluster-name>"

  region                 = "aws-eu-central-1"
  version                = var.elasticsearch_version
  deployment_template_id = standard/arm-based (decide based on the requirement)

  elasticsearch {
    autoscale = "false"
    topology {
      id   = "hot_content"
      size = "15g" // size of the cluster
      zone_count = 3 // number of nodes 
    }
  }
  kibana {}

  // important to include this in the cluster config to ensure to NOT override any existing remote cluster related config
  # ignore changes to remote cluster config
  lifecycle{
    ignore_changes = [
      elasticsearch[0].remote_cluster
    ]
  }
}

Few important bits to note in this:

Ensure the region and ES version are consistent for your cluster.

By default, deployment_template_id should be defined as standard aws-storage-optimized-v3 but if your are creating the cluster for memory-intensive work, consider creating it as arm-based aws-general-purpose-arm-v6

The size and number of nodes vary based on the need, so decide it accordingly from the Elastic Cloud UI.

An important bit to always add in your cluster deployment is to ignore changes to the remote cluster config. This ensures the remote communication that is configured among clusters doesn’t get overriden by this update.

ignore_changes = [
      elasticsearch[0].remote_cluster
    ]

Example: Cluster Secrets

# Create the AWS Secret Manager secret
resource "aws_secretsmanager_secret" "<cluster-secret-name>" {
  name = "/elasticsearch/<cluster-name>/credentials"
  description = "Elasticsearch credentials for <cluster-name> cluster"
}

# Create the AWS Secret Manager secret version with JSON value
resource "aws_secretsmanager_secret_version" "<cluster-secret-name>" {
  secret_id     = aws_secretsmanager_secret.<cluster-secret-name>.id
  secret_string = <<JSON
{
  "endpoint": "${ec_deployment.<cluster-name>.elasticsearch.0.https_endpoint}",
  "username": "${ec_deployment.<cluster-name>.elasticsearch_username}",
  "password": "${ec_deployment.<cluster-name>.elasticsearch_password}"
}
JSON
}

Example: cluster_settings

resource "elasticstack_elasticsearch_cluster_settings" "<cluster-settings>" {
  elasticsearch_connection {
    username = ec_deployment.<cluster-name>.elasticsearch_username
    password = ec_deployment.<cluster-name>.elasticsearch_password
    endpoints = [ec_deployment.<cluster-name>.elasticsearch.0.https_endpoint]
  }

  persistent {
    setting {
      name  = "cluster.max_shards_per_node"
      value = "2500"
    }
    setting {
      name  = "cluster.routing.allocation.cluster_concurrent_rebalance"
      value = "30"
    }
    setting {
      name  = "cluster.routing.allocation.node_concurrent_recoveries"
      value = "30"
    }
  }
}

Three important settings that would make sense to add as default 👆

max_shards_per_node the value would vary based on the cluster size and requirement
cluster_concurrent_rebalance To allow correct rebalancing of shards to all available nodes (20-30 is a good value for this)
node_concurrent_recoveries To allow correct rebalancing of shards to all available nodes (20-30 is a good value for this)

Above steps will create a cluster with the defined config. ✅

Always good to ensure and check what our changes are adding/updating/deleting, to be extra cautious to NOT end up deleting/updating some existing resource. 😨

Problem 2: Add the newly created elastic cluster to the POOL of clusters in the DynamoDB Cluster assignment table

Now that the new cluster is and available to view on elasticsearch as a healthy deployment, we can make the next move to include it in the list of available clusters, to be able to then use it for our ES needs.

example:

To add it to the pool, you can assign it to the table where cluster assignment is defined, in our case: ClusterAssignmentTable

To add your new cluster to this pool, use:

json=$(cat <<EOF
{
  "pk": {"S": "POOL"},
  "sk": {"S": "CLUSTER#${cluster_name}"},
  "cluster": {"S": "${cluster_name}"},
  "endpoint": {"S": "${endpoint}"},
  "username": {"S": "${username}"},
  "password": {"S": "${password}"}
}
EOF
)

echo "===> Storing cluster $cluster_name in table $table_name..."
aws dynamodb put-item \
    --table-name $table_name \
    --item "${json}"

As a result of this, the new cluster should be found in the cluster assignment table.

Problem 3: Allow it the access to communicate to/from other existing ES clusters.
When dealing with multiple ES clusters, there is the requirement to query data from one cluster to another. In the current implementation, we already have use cases of doing multi-org queries where those orgs could belong to different clusters.

Usage looks like:

To allow this, we need to ensure, every time a new cluster is created, its also set up for the remote cluster communication.

To do this, you could do something like for all your clusters in the pool:

payload=$(cat <<EOF
{
  "persistent": {
    "cluster": {
      "remote": {
        "${remote_cluster}": {
          "mode":"proxy",
          "server_name": "${remote_server_name}",
          "proxy_address": "${remote_proxy_address}",
          "skip_unavailable": true
        }
      }
    }
  }
}
EOF
)

curl -sS -X PUT "$endpoint/_cluster/settings" \
  -u "$username:$password" \
  -H 'Content-Type: application/json' \
  -d "$payload" | jq

This will ensure all the clusters for that environment are setup for remote communication to one another.

Also, remember we had that extra step to ignore changes to the remote cluster config? This is why! We DO NOT want to override any remote config changes when updating clusters.

At this point, you have your cluster ready on elasticsearch, cluster allocation table, and also able to talk to other clusters 💪

And then there are the Not-so-good times when clusters are not doing that great and healthy.

With these many clusters (even more probably in the future), we want to know the whereabouts of each of them, to be better at managing these.
So a cluster specific monitoring is a good idea. We do that using Datadog. Datadog provides a Elastic Cloud integration that can be used for the same.

Something like above 👆

Hope this helps!

Matching faces to Twitter DPs #ngIndia

Nishu Goel — Fri, 27 May 2022 05:23:06 +0000

Guess what I am talking about?

Year 2022, this was after more than 2 years that we got to be at an in-person conference, and gosh, the experience! I am forced to accept that we were simply compromising when being a part of the online conferences.

I think conferences were anyway never about only consuming the content delivered from the sessions. It was majorly about the networking, the warmth from meeting people. So when we we went to ngIndia 2022 event this May, it was about meeting those people from online bonds from the past two years and connecting NOT from behind the screens.

So, 21st May, I got the opportunity to speak at #ngIndia on the topic ‘Microfrontends’. Now I chose this topic because I am currently working with micro-frontends at @epilotGmbH, and what could be better than talking about the success and challenges of using micro-frontends for larges scale apps than this.

So my preparation did include solving issues related to, for example, cross-micro-frontend communication AT WORK and coming back home and trying to explain the problem and its solution from the day through my slides.

Finally I decided to do my talk in the form of a role-play with Srashti Jain to help audience understand better.

Me — I wish I could work for this company but I haven’t used X framework for a while now.

Srashti Jain (as a recruiter) — oh that’s not a problem at all! Our product is based on a micro-frontend architecture. We’d love to hire you to work on your choice of framework and build our awesome product.

Me — Wow that sounds interesting! So I don’t need to skill up in another framework (X framework) now to contribute to this product. Awesome!

So this is how I began my talk to explain how micro-frontends made scaling up teams and projects framework-agnostic. But thats not it, I am sharing my slides here to get a sense of what was discussed, however, we will soon have the recorded talk up soon and ready to be shared right here.

mfe_ngIndia - Google Slides

docs.google.com

My experience speaking about micro-frontends was beyond my expectations, and I had such important conversations with many engineers from the audience. Our discussions revolved around how developers could use single-spa framework to deal with their micro-frontend architecture as the top level router, around how developers faced routing issues in their current single-spa MFEs, how some developers desired to switch to micro frontends to migrate from AngularJS etc.

The conference had more than 500 people joining in the fun, with the right element of over-coffee networking, over-lunch chit-chat, after-talk questions.

Speakers from all over the world shared interesting findings during their talks. The conference was followed by an after party where the speakers and the volunteers partied together with food, drinks, music, dance and what not!
During the pre-conference dinner, the speakers were felicitated with gifts and swags from ngIndia organiser, Dhananjay Kumar.

I always love to speak at ngIndia and looking at the love from people who attend this event yearly, I am proud to be a home grown speaker of ngIndia, saying this since this is where my foray into speaking at events to share my learning started.

Here are some of the captured moments from the conference.

Thank you!!!

Stay tuned for the talk recordings :)

https://www.ng-ind.com/

svelte-i18next

Nishu Goel — Fri, 08 Apr 2022 13:32:09 +0000

Presenting the svelte wrapper for i18next.
This library, based on i18next, wraps an i18next instance inside a svelte store and observes the i18next events like languageChanged, resource added, etc. to be able to render our svelte components.

Package:

svelte-i18next - npm

Svelte wrapper for i18next. Latest version: 2.0.0, last published: 3 months ago. Start using svelte-i18next in your project by running `npm i svelte-i18next`. There are no other projects in the npm registry using svelte-i18next.

npmjs.com

GitHub:

NishuGoel / svelte-i18next

Internationalization for svelte framework. Based on i18next ecosystem

svelte-i18next

Svelte wrapper for i18next

npm i svelte-i18next i18next

Implementation

This library wraps an i18next instance in a Svelte Store to observe i18next events so your Svelte components re-render e.g. when language is changed or a new resource is loaded by i18next.

Quick Start

i18n.js:

import i18next from "i18next";
import { createI18nStore } from "svelte-i18next";

i18next.init({
 lng: 'en',
 resources: {
    en: {
      translation: {
        "key": "hello world"
      }
    }
  },
  interpolation: {
    escapeValue: false, // not needed for svelte as it escapes by default
  }
});

export const i18n = createI18nStore(i18next);

App.svelte:

<script>
  import i18n from './i18n.js';
</script>

<div>
    {$i18n.t('key')}
</div>

See full example project: Svelte example

View on GitHub

The implementation looks like this:
The package creates a Svelte writable of i18next and listens to the events triggered by any updates on the i18next instance.
For example, if an application loads a new namespace for their translations, the svelte_i18next package will trigger the
i18next.on(‘loaded’, function(loaded) {}) event.
Check the i18next events here.

Usage:
i18n.js

import i18next from "i18next";
import { createI18nStore } from "svelte-i18next";

i18next.init({
   lng: 'en',
   resources: {
     en: {
       translation: {
         "key": "hello world"
       }
     }
   }
});

export const i18n = createI18nStore(i18next);

App.svelte

<script>
  import i18n from './i18n.js';
</script>

<div>
    {$i18n.t('key')}}
</div>

See a example usage here: Svelte example
You could also use namespaces as you would using i18next and the svelte wrapper will re-render the translations based on the namespace provided.

import { createI18nStore } from 'svelte-i18next'
const i18n = createI18nStore(i18n_instance) 
i18n.loadNamespaces(['example']) // this triggers the re-render and loads translations from the provided namespace.
<div>
    {$i18n.t(key)}
</div>

Into svelte? Dealing with localisation?

Show some love @svelte-i18next ❤

dabbling @epilot-dev with ______

Nishu Goel — Mon, 21 Mar 2022 08:59:12 +0000

Year 2022 marked an amazing start for me with a new stint with epilot GmbH based out of Cologne, Germany. I joined the team to be working on their micro-frontend architecture, with my primary focus going to be on the performance aspects of the applications.
Like the first recent addition we did after I started was to add brotli compression to each of our micro-frontend (MFEs) bundles.

Let’s look at the tech stack involved.

Micro services

The main epilot app is built using microfrontends with each entity as its own application. There is a login application that takes care of the user login state (built using React + TypeScript), a topbar application that has features like Global search, notifications (React + TypeScript), a sidebar application that navigates the user to different entities based on user permissions (Svelte + TypeScript + Tailwind), and so on.
The component library of the product is based on material UI, built with React.

https://www.npmjs.com/package/@epilot/base-elements

Due to the open source experience of many engineers in the team, most of the non-confidential stuff is kept public on GitHub and npm.

One of the very interesting things I got to do to ramp up was to build a part of the product as a Svelte MFE. With most of the existing MFEs built using react, this was quite interesting and challenging to do.

We use single-spa framework to build our microfrontends and use different frameworks per the ease and knowledge of an engineer. To make the development journey easier for a new developer starting up with microfrontends, we have a mfe-app-generator package that creates a boilerplate application with the required config.

https://www.npmjs.com/package/@epilot360/mfe-app-generator

Due to performance reasons, one of the goals was to switch to rollup for bundling all our applications, however, we use webpack due to rollup’s known issues with aws-sdk.

We use serverless microservices architecture exposed via APIs. Most of the backend services are written in TypeScript using AWS lambda functions, APIs, Cloudfront distribution etc. More on that here.

With Svelte added to our microfrontend stack, we plan to further diversify the frameworks to be used, ensuring to not lose focus on a performant application.
Here’s a list of support frameworks by single-spa.

I also plan to share our journey from a high LCP (largest contentful paint) to a much reduced one, but I’d keep it for another blog post.

When working with custom implementation and bundling, one of the challenging tasks is to configure your bundler and ensure it is consistent throughout all the applications (MFEs in this case). We have defined a @epilot360/webpack-config-epilot360 to base one’s webpack config on top of the existing basic config defined.

https://www.npmjs.com/package/@epilot360/webpack-config-epilot360

I will share more on the webpack configuration for microfrontends and cache invalidation strategies in another post but cache invalidation is definitely my major ongoing focus area.

Epilot SDK

All public epilot services are available through an epilot-sdk (written in JS/TS) that allows developers to interact with epilot APIs. Internally as well, the recommended way to use the services or APIs using directly the SDK instead of going the granular way.
For e.g.,

import { getClient } from 'epilot-sdk/entity-client'

Want to scratch your itch? Find it here:

https://www.npmjs.com/package/@epilot/epilot-sdk

Communicating between microfrontends

One of the interesting challenges I recently faced was to communicate between two applications without any shared library. One was the sidebar built using Svelte and another the topbar built with React. Now we usually tend to avoid any such communication between our MFEs but this was a state clear event to happen before a routing change in any of the two MFEs.

I used single-spa’s custom events to do this. The idea was to clear the selected state of an icon in the topbar when an icon in the sidebar was selected, and vice versa.
To do this, I used the single-spa:before-routing-event custom event to check if the active URL in the current application was undefined, we could clear the state, if however, the URL existed, that becomes the active state.

Checking the URL history without communicating between two applications was not possible due to two separate routers for each application. Svelte app using the svelte-routing package where with React, the react-router.

Here’s a detailed view of the tech stack used at epilot:

https://docs.epilot.io/techradar

I have shared more info on my journey to epilot as a full-stack developer in my Spotify Podcast here:

Interested to know more about how we do things at epilot? Reach out to me here: https://twitter.com/TheNishuGoel

Creating framework-agnostic web components with Angular

Nishu Goel — Thu, 12 Dec 2019 20:33:51 +0000

Wait, framework-agnostic web components? What is that!
Let us try to understand the meaning of this term by an example, apparently my own example.

The setting in which I work at my company is based on a Garage concept. This translates to not having a single account which you work for, and instead getting to work on many short-duration, faster-delivery projects which keep flowing in and out of the garage. And the technology/framework for the deliverable is arrantly based on the requirement of the project.
Therefore, this means that one can get to work on Angular for let’s say one 7-month project whereas React/Vue for another project for some amount of time.
Now in this kind of setting where I have created lets say, a loader component for one of my projects in Angular, I wouldn’t want to redo the effort of creating a similar component for another project now that it is a react-based project let’s say. I would want something reusable, wouldn’t I?
So this small story was to set the context of why would one want to create a Web Component instead of a regular component in any of the frameworks that they code in.

Web components though?

Does it mean components coming from the web?
Well, the best way to explain web components would be to say that you can create a reusable custom element in any technology and use its functionality inside your web-apps by using the element.
As per the MDN web docs, Web components are defined as:

Web Components is a suite of different technologies allowing you to create reusable custom elements — with their functionality encapsulated away from the rest of your code — and utilize them in your web apps.

The amazing thing about web components is that they only work on the web standards without the involvement of third-party libraries.
Now to understand this term web component which has a bigger meaning in a simple way, lets try to understand what it actually contains.
Like I mentioned before, web components follow certain web standards and work as per those. These web component specifications are:

Custom Elements
Shadow DOM
HTML Template
ES Module i.e. HTML Imports

Each one of these standards are independently capable of being used. To leverage Web Components, we combine these four web standards and use and re-use our created custom elements built with the functionality of the holding framework in any of the frameworks/libraries, vanilla JavaScript etc.

Let us look at each one of these one by one.

First Spec: Custom Elements

This is the first standard of Web component spec which allows us to create our own tags just like another HTML element but we decide which behavior should be contained inside that tag.
A Custom Element can be created as:

class MyElement extends HTMLElement {
  […]
}
customElements.define("my-element", MyElement);

More about Custom Elements and the two types of custom elements that can be created can be read here.

Second Spec: Shadow DOM

The shadow DOM specification for web components basically means that the custom element would have supported style encapsulation and it wouldn’t conflict with the ID or class of your other elements in the DOM. It does so by creating a sub-DOM tree of the element.
These elements enters as children of the element and have their own encapsulated styles.
This works by using element as a placeholder for the the custom element.

More on the DOM API, Shadow DOM, slots etc. can be read here in this amazing article.

Third Spec: HTML Template

This third standard of the web component basically helps it to be have some chunk to the template to be loaded at runtime. This is achieved by using the tag.
Whatever is placed inside these template tags can be rendered at runtime by cloning and inserting them though JS.
After retrieving the template tag in JS, you can activate it using:

var clonedObj = document.importNode(templateRef.content, true);

We can then append this using to the DOM as a sub-DOM tree.

Final Spec: HTML Import i.e. ES Module Specification

This specification helps you integrate your custom element with the different JS apps by defining the interface using the script.

<script type=”module”> import {example} from ‘@example/example’; </script>

This will then allow us to use the custom tag inside our app like:

<example>…</example>

These four web component specifications help us understand how and why the components are to be used in which ever framework/ JS app as you wish to.
Now that we have an understanding of Web components, in general, custom elements, Shadow DOM etc. We can proceed to create one custom element inside the Angular framework and be able to use it inside other library projects.

Creating a Web Component in Angular

Starting Angular v6, we can use regular Angular components as web components and load these components inside any of the frameworks/libraries or a JavaScript code.
This is achieved using Angular Elements. Angular Elements is an Angular package that helps us create regular Angular components as Web components (custom elements) and use them in other technologies.

As Rob Wormald mentioned in his talk,

To start with this, the first step would be to install the package @angular/elements .

ng add @angular/elements

npm install --save @webcomponents/webcomponentsjs

import ‘@webcomponents/custom-elements/src/native-shim’; import ‘@webcomponents/custom-elements/custom-elements.min’;

The installation of @angular/elements also adds polyfills which are lighter in size but the support for some browsers is quite limited. Therefore, these two above-mentioned polyfills come to be of best use and thus we can avoid installing the polyfills of @angular/elements by simply doing:

npm install @angular/elements

The next step is to create a traditional Angular component (to be used as a web component) which looks like this:

@Component({
  templateUrl: './custom-elements.component.html',
  styleUrls: ['./custom-elements.component.scss']
})
export class ComponentName {
  title = 'custom-elements-demo';
}

The selector name will be the one used while creating the web-component. Also, you would want to give it a generic name as this web component is going to be a re-usable one which you’d definitely want to use in your other projects as well.
The next step then is to create the custom Element inside the module of your app. To do this, we first need to declare it inside the entryComponents array inside NgModule.

Note: Starting Angular v9, this would not be required as with Ivy, we do not need to explicitly specify a component as an entryComponent.

Inside the module class, define the custom element component now as:

//Inside NgModule  
  
  entryComponents: [CustomElementDemoComponent],

  })
  
  export class FeatureModule {
  constructor(private injector: Injector) {
  }

  ngDoBootstrap() {
      const customElement = createCustomElement(CustomElementDemoComponent, { injector: this.injector });
      customElements.define('custom-element', customElement);
  }
  }

Inside the module, We use the injector service to basically wrap this Angular component as a web component using the createCustomElement method from Angular Elements. customElements.define registers this component on to the browser. This is done inside the ngDoBootstrap method as you want to manually bootstrap the component instead of asking Angular to do it using the bootstrap array.

<custom-element></custom-element>

Once this is done, you can use it on an Angular component’s template as above and to make sure that the Angular component recognizes this custom Element, we add CUSTOM_ELEMENTS_SCHEMA in the schemas array in NgModule inside AppModule.
You custom Element is now ready to be used and is accessible inside any of your Angular projects.

Wait, just Angular projects?

You now might be wondering that I mentioned about using these anywhere across apps created using different frameworks or in JavaScript. Well, officially Angular does not yet support stand-alone web components which can be used outside the Angular projects, however, you can still use these custom elements in other frameworks by a variety of ways. Let us look at that.

The community project, ngx-build-plus by Manfred Steyer, allows us to create the required polyfills and the optional --single-bundle switch, helps create a single bundle which then exposes our web component.

There is an amazing series of blog posts about Angular elements in detail by Manfred himself here.

To add it to your web component project,

ng add ngx-build-plus

This will update the build target in your angular.json to ngx-build-plus:build and update scripts related to polyfills.

Also, remember we manually added polyfills to our web component project? We can avoid doing that as well with the help of ngx-build-plus!

We can do this by using the command: ng g ngx-build-plus:wc-polyfill

Now, to use the single-bundle flag, we would update our scripts in the package.json for building the web component.

“build-my-comp”: “ng build --prod --single-bundle true --keep-polyfills”

The single-bundle flag defaults to false so we set it to true explicitly in the build script. When you try to build your component now using npm run build-my-comp , you might get to see another error saying:

Schema validation failed with the following errors: Data path “.budgets[1].type” should be equal to one of the allowed values.

To solve this, remove the following lines of code from angular.json

{
“type”: “anyComponentStyle”,
“maximumWarning”: “6kb”,
“maximumError”: “10kb”
}

Finally, you will get a main bundle created now that can be exported to a Vanilla JS project, or any other framework and can be used as a web component, with, ofcourse, polyfills in place.

To test this, in a basic project inside index.html, I included the file as:

<script src=”main.js”></script> and used the custom element

<custom-element></custom-element>
To run this to be able to see the output now, you can install static-server from npm
npm i -g static-server , open the app on the port specified, and there you go with own web component in a non-Angular project (where the web component was created.)

The web component can now be published to npm and used in other frameworks as well.
For this blog post, I will be publishing this login form to the npm registry and using it in a React project.
After building the web component using npm run build-my-comp , you get a single bundle generated for your web component, looking like this:

You can now publish it to npm by creating a package.json file in the dist directory. This needs only the name and version of the package that you will be publishing. To avoid the deletion of package.json every time you build the component, you can place it inside another folder.
The package.json looks like:

{
    "name" : "@nishugoel/prod-card",
    "version": "0.0.5"
}

We can now cd into the built package and pack it into a package to be published to npm using npm pack. This will generate .tgzfile for you. To publish it to npm, make sure you are logged in to npm, if not, use npm login.

Finally publish using,

npm publish packagedfile.tgz --access public

Our web component is finally on npm and can be used in other frameworks now :)

Using my web component in React

Create a new react app and inside the index.html, use the reference to your main.js script from npm.

//Inside index.html

        <script src="https://unpkg.com/@nishugoel/prod-card@0.0.5/main.js"></script>

//Use the custom element in the App.js file

        <prod-card username='Enter your name' password='Enter new password' btnname='Sign in'></prod-card>

Install webcomponents.js

npm install --save @webcomponents/webcomponentsjs

Put the required polyfills for custom-elements-es5-adapter.js and webcomponents-bundle.js

And finally use the custom element in your app.js file with your own property values to the component.

And here we go!
In the upcoming articles, I will be writing more on binding the data i.e. properties, events to the web components made in Angular.

Custom validation for your reactive forms?

Nishu Goel — Sat, 27 Jul 2019 19:03:15 +0000

When working with forms in Angular, we are provided with a choice. The choice is similar to the one between choosing tea or coffee. This choice is about choosing which types of forms we should use as per the requirement of our solution.
And the two options that we get are:

Template-driven forms
Model-driven forms a.k.a Reactive Forms

Now each of these have their own advantage over another and it totally depends on the developer that which one he prefers to use for this set of requirement. Is he looking to quickly get an input from the user without much interest in all the fancy validation and stuff OR is she wanting to leverage all the possibilities that a form is capable of providing.
One of the reasons somebody would choose to work with template-driven forms can be to get the work done without really wanting to have a form model inside the component, leaving alone the unit testing bit which gets difficult to do against the DOM.

One of the reasons somebody would choose to work with model-driven forms (Reactive forms) can be to do most of the task from the component class only putting the instances of the input fields on the template, making it easier to unit test and use up other great features of reactive forms, for example, Custom validation.

In this blog post, we will see how we can use custom validation inside our forms and make the best use of reactive forms.
Now a validator is nothing but a function. To create this validator fucntion, the syntax goes like this:

function classValidator(control: AbstractControl) : {[key : string] : boolean} | null { return null; }

Let us break this down now and understand.
Since our custom validator is a function, we use the function keyword followed by the name of our validator that we want to create. As an argument comes the FormControl or FormGroup for which use the base class i.e., AbstractControl. The next part in it means what type of value will be returned by the function. So it there is something wrong with the input field, it returns an object in the form of a key-value pair, where the value is of type boolean and the key is of type string. If everything works fine with the input field, it returns null.
So lets create a simple range validator for age here:

function ageValidator (control: AbstractControl):{[key: string]: boolean} | null { if( control.value !==null && (isNaN(control.value) || control.value <20 || control.value> 70)){ return {'ageValidator': true} } return null; };

To apply this to your input field, use the name of your custom validator on the form control name inside the form group,

this.customerForm = this.fb.group({ firstname: ['', [Validators.required, Validators.minLength(5)]], email: ['', Validators.required], age: [null, ageValidator] })

On the template,

<span *ngIf= “customerForm.get(‘age’).errors?.ageValidator”>Only for age group 20 to 70</span>

Let us look at the results in the browser now.

Lazy Loading in Angular 8

Nishu Goel — Sun, 09 Jun 2019 15:08:59 +0000

We all know lazy loading is one of the most useful concepts of Angular Routing, and for those of us who have been working with Angular, we know how it brings down the size of large files. This is done by lazily loading the files that are required occasionally.

To start with lazy loading by asynchronously loading the feature module for routing whenever required, we go to the route configuration and use the property loadChildren. Let us see what this property does.

    {path: ‘user’, loadChildren: ‘./users/user.module#UserModule’}

This property loadChildren is used for lazily loading the routes and is not related to child routes or such.

Let us break down what the property’s value means. The loadChildren property accepts a string value which contains the route to the feature module followed by a hash symbol and then the name of the feature module.

Now when the route gets activated, this loadChildren property will get activated and load the requested module. It will then load the requested component and display that component’s template.

Once we have configured this property, we go to the console to see which files are generated.

We will see an extra bundle file generated.
Now if you go to the network tab in the console to see the files generated on routing to the UserModule, you will see one extra file created with some numeric value which might look something like this:

This is how lazy loading gets implemented using the loadChildren feature in the route configuration for the specific feature module. And this creates another bundle file only when that route is navigated to and the data is requested.

This is how we have been working with lazy loading till now, right?

But…
If we look at the route config again,

  {path: ‘user’, loadChildren: ‘./users/user.module#UserModule’}

the loadChildren property accepts a string which means that even if there is a wrong module name or a typo while writing the code, Angular would not know there is something wrong and accept whatever is there as a string until we try building it.

So, let us say we write the config as :

   {path: ‘user’, loadChildren: ‘./users/user.module#UserModulee’},

with an extra ‘e’, it will not throw any error considering it a part of the string.

Therefore,
Angular 8 comes up with support for dynamic imports in our router configuration. This means that we use the import statement for lazy loading the module and this will be understood by the IDEs, webpack, etc.

So when you update to Angular 8, this will automatically accommodate the changes in your application.

Now if you look at how lazy loading is done in this new route config, you will see:

 {path: ‘user’, loadChildren: () => import(‘./users/user.module’).then(m => m.UserModule)};

Now your editor, let’s say VSCode understands what is this syntax and will recognize if there is some mistake and we won't have to wait till build time to realize about an error.

This new syntax now means that loadChildren is a function which will execute when it tries to access the user module. This will asynchronously load the import statement and implement the module.

This new lazy loading is explained by Stephen Fluin in this video: https://www.youtube.com/watch?v=jPXl7sCPCOA

Angular Library with ng-packagr

Nishu Goel — Mon, 03 Jun 2019 06:31:58 +0000

When you have a small application with three or more components, and they do not share a relationship among themselves when it comes to sharing data through component communication. For such components, we have seen the usage of services. Services come to be of great help when we have to use the same data in more than one components. This data is then used inside the component by using dependency injection.

This is when we have the components inside one module which contains the service. However, if we want to use the data from the service in multiple modules, a better idea is to put this data inside the service somewhere, from where you can actually import it in whichever module and component you wish to.

Now, this is about one application, what if you want maximum applications to use the same data and you definitely do not want to copy paste the code for each of your applications. Also, you do not want any of the applications to communicate with each other, but just share some data from an external source to all the applications.

That is when Libraries and packages come into the picture. In this blog post, we will see how to create an Angular library and what are the requirements to create one. When researching more about Angular libraries, I came across this great talk Create and publish Angular libs like a Pro by Juri Strumpflohner wherein he explains the need for a library with this image.

Angular Library with ng-packagr

Nishu Goel — Wed, 29 May 2019 05:50:17 +0000

NOTE: The article is relevant to the earlier versions of Angular and with the version 7 and above, we can create a library in Angular using the command ng generate library . For more on this, refer to the documentation here: https://next.angular.io/guide/creating-libraries
When you have a small application with three or more components, and they do not share a relationship among themselves when it comes to sharing data through component communication. For such components, we have seen the usage of services. Services come to be of great help when we have to use the same data in more than one components. This data is then used inside the component by using dependency injection.
This is when we have the components inside one module which contains the service. However, if we want to use the data from the service in multiple modules, a better idea is to put this data inside the service somewhere, from where you can actually import it in whichever module and component you wish to.
Now, this is about one application, what if you want maximum applications to use the same data and you definitely do not want to copy paste the code for each of your applications. Also, you do not want any of the applications to communicate with each other, but just share some data from an external source to all the applications.
That is when Libraries and packages come into the picture. In this blog post, we will see how to create an Angular library and what are the requirements to create one. When researching more about Angular libraries, I came across this great talk Create and publish Angular libs like a Pro by Juri Strumpflohner wherein he explains the need for a library with this image.
It explains how our applications comprise of different javascript files and other external libraries and tools and how they are assembled together.
To start with, let us look at the points to consider when creating a library.
A library should be:
Distributed
Platform-independent
Bundled
Supporting Ahead-of-time compilation
Work with TypeScript doing type checking, autocompletion, etc.

All these requirements together make a perfect Angular library, and it has come up with a solution called Angular Package format to support building the Angular libraries. This format is a recommended way of building Angular libraries and distributing the packages. Libraries which are built using this format will be handled properly by the Angular applications, node packages, npm.
It is a set of guidelines which Angular team uses to distribute packages and if we follow those guidelines while developing an Angular library, we put ourselves on a safe side because then our library is supported by a lot of tools that are used in the ng ecosystem and the packages then integrate easily with these tools. It will also have an optimized bundle size and a build time which are some of the very important factors.
Steps to produce an Angular library
Since the library should be bundled all in once place, the first step should be to inline all the external templates.
The next step is to compile this with the ngc compiler which is sort of a wrapper around the tsc. The ngc will produce the javascript files out of the typescript ones and also produce the the d.ts files to do the Ahead-of-time compilation by creating the metadata for this library.
Build different ESM2015, UMD formats.

ng-packagr
We can developer libraries using the ng-packagr, which is a project on nom and Github. These libraries can then be put inside the Angular Package Format. The libraries create all the bundles for us. It is consumed by different module loaders, inline the templates by itself, creates the type definitions (.d.ts files)
To install ng-packagr using npm, so that it can then be used globally from the command line,

IWD 2019 — GDG Makurdi

Nishu Goel — Fri, 05 Apr 2019 21:06:38 +0000

In the March of 2019, Pius contacted me to share my knowledge about the basic concepts of Angular with the audience of Makurdi, Nigeria. It was organised as a celebration of International Women’s Day 2019.

On the 31st of March 2019, I joined the audience in Makurdi via a Hangout session and discussed the topics starting from the basic set up required to work on an Angular application to component communication, directives etc.

This was my first online session teaching an international audience and there was no less energy in the crowd than that in an offline/physical session.

https://thepracticaldev.s3.amazonaws.com/i/igz0frk6oi7u5hx7929y.jpeg

https://thepracticaldev.s3.amazonaws.com/i/8ho0soo6rbtgoebmrdei.jpeg

However, the session could not be recorded due to network issues.
The topics taught in the session are covered in this Beginners course on Angular by me here: https://www.udemy.com/angular-for-beginners-l/

Contact me at: nishu0505@gmail.com