Forem: Eyal Lapid

Converting Extension from JS to JSX

Eyal Lapid — Sat, 25 May 2024 06:44:21 +0000

This article is part of the Converting Large Codebase Project to Vite series.

All files that contains JSX code and had ".js" extension failed in the browser.

This happened because the react swc vite plugin does not compile ".js" files. Instead it send them to the browser as is.

https://github.com/vitejs/vite-plugin-react-swc/blob/b829b03f6476bed60ff5535fe883abc1b0b4e095/src/index.ts#L231

const transformWithOptions = async (
  id: string,
  code: string,
  target: JscTarget,
  options: Options,
  reactConfig: ReactConfig,
) => {
  const decorators = options?.tsDecorators ?? false;
  const parser: ParserConfig | undefined = id.endsWith(".tsx")
    ? { syntax: "typescript", tsx: true, decorators }
    : id.endsWith(".ts")
    ? { syntax: "typescript", tsx: false, decorators }
    : id.endsWith(".jsx")
    ? { syntax: "ecmascript", jsx: true }
    : id.endsWith(".mdx")
    ? // JSX is required to trigger fast refresh transformations, even if MDX already transforms it
      { syntax: "ecmascript", jsx: true }
    : undefined;
  if (!parser) return;

  let result: Output;
  try {
    result = await transform(code, {
      filename: id,
      swcrc: false,
      configFile: false,
      sourceMaps: true,
      jsc: {
        target,
        parser,
        experimental: { plugins: options.plugins },
        transform: {
          useDefineForClassFields: true,
          react: reactConfig,
        },
      },
    });

That is an explicit decision made by the plugin authors.

Chosen Solution

So in order to have the plugin take all those files and transpile them with SWC, we had to change the extension all of "js/ts" files that contains JSX to "jsx/tsx" equivalent extension.

Without a transpiler in the process, the jsx left as is:
https://github.com/vitejs/vite/discussions/3448

The conversion process can be done in several ways. One of them is using a script to go though all project "js/ts" files, detect if they have any JSX syntax and change their extension.

An example of such script taken from comment in one of the issues:

find src -type f | grep "\.[jt]s$" | xargs -n1 grep -HE "(<\/)|(\/>)" | cut -d: -f1 | uniq | awk '{print "mv "$1" "$1"x"}' | sh

Breakdown of Script

list All Files Nested in a Target Folder

find src -type f

find - This command searches for all files (-type f) within the src directory and its subdirectories.

The output is a list of the filenames, include their paths.

./docs/.eslintrc.cjs
./docs/tsconfig.json
./docs/.storybook/main.js
./docs/package.json
./docs/stories/button.stories.ts

Filter Out Only Needed Files By Extension

| grep "\.[jt]s$"

The pipe | takes the output of the find command and passes it to grep.

grep - grep ".[jt]s$" filters the files, keeping only those whose names that end with .js or .ts (JavaScript or TypeScript files). for the grep patterns basic regex can be used as default.

the output is the same format of file paths and names, but filtered only to the files end with js/ts extensions.

./docs/.storybook/main.js
./docs/stories/button.stories.ts

Filter Out Files by Existence of Jsx Content Inside Them

| xargs -n1 grep -HE "(<\/)|(\/>)"

xargs -n1 takes each file name from the previous output one by one and executes the following grep command.

grep -HE "(<\/)|(\/>)"

searches for the pattern </ or /> in each file.
-H includes the filename of the file grep searched in, in the output,
-E allows for extended regular expressions syntax.

./docs/stories/button.stories.ts:    </Button>

Split the Output and Keep Only the File Names

| cut -d: -f1

cut - remove sections from each line of files

-d: splits the output by the : delimiter. The previous grep output it to separate the file and the content found.
-f1 keeps only the first field, which is the filename.

This removes the line number and matched text from the grep output, leaving only the unique filenames.

./docs/stories/button.stories.ts

Reduce the File List to Unique File Names

| uniq

uniq filters out any duplicate filenames, so each filename appears only once.

./docs/stories/button.stories.ts

Construct a `Rename` Command for the Specific File Name

| awk '{print "mv "$1" "$1"x"}'

awk constructs a mv (move) command for each file.

It renames each file by appending an "x" to its name.

mv ./docs/stories/button.stories.ts ./docs/stories/button.stories.tsx

Execute the `Rename` Command

| sh`

The final | sh executes the constructed mv commands in the shell, renaming the files.

Process

On a large project, you will want to do this conversion in several PRs and not one big chunk, so not to create large conflicts with other people ongoing work.

Converting Large Codebase Project from Webpack to Vite

Eyal Lapid — Sat, 25 May 2024 05:24:41 +0000

Photo by Roberto Nickson on Unsplash

In the company I work at, our frontend codebase consists of couple of thousands files. The framework is React 16 and the bundler used was webpack 4.

Reasons

We wanted to change the bundler to vite for two main reasons:

DX - development experience - Faster boot time from starting the development server to seeing the project on the browser and start coding.
DX - Faster production build time for qa envionments.

Process

So we took the opportunity in the company hosted hacketon and built a POC that worked. The POC was mapped after that to separated individual tasks.

This to tackle each migration step more cleanly without disrupting existing features development.
Have the developers and interested parties participate in the CRs and give their takes and experience.

Steps

This article summarize the issues that rose and how they were handled.

Converting couple thousands Js/Ts files that contains JSX content to jsx extension
Js Errors - Using variables before declaration
lingui swc plugin replacement for babel macros - does not work for now on .js files
Third party libraries unconventional import - react-d3-tree
Css mixin implicit import
Unsupported js decorator used in .js files
Need for async/await transpilation which use bluebird cancelable feature
Third party node modules that import sub third party node modules needs to prebundle explicitly
Unconventional node modules import - "../../../node_modules/date-fns"
Code that use global, process and other node environment variables needs polyfill or mock
Code that use require instead of import
webpack only syntax for hmr needs to be converted to vite syntax
Webpack multiple import statement needs to be converted to vite supported syntax
Webpack raw loader needs to be converted to vite raw loading
Converting index.html dynamic generation instead to work also with vite
Mock process.cwd for react-markdown vfile

Nest IOC DI (Dependency Injection) as a Seeds Tasks Runner

Eyal Lapid — Fri, 09 Dec 2022 10:23:41 +0000

Nest IOC DI (Dependency Injection) as a Seeds Tasks Runner

Using Nest DI as a task runner for seeding the DB.

The example code can be found on Github.

Why?

Nest and the Prisma example provides only a rudimentary examples for seeding. No options to provide dependencies between the seeding tasks and no factories for the seeding.

seed.ts

import { PrismaClient } from '@prisma/client';
const prisma = new PrismaClient();
async function main() {
  await prisma.user.deleteMany();
  await prisma.post.deleteMany();

  console.log('Seeding...');

  const user1 = await prisma.user.create({
    data: {
      email: 'lisa@simpson.com',
      firstname: 'Lisa',
      lastname: 'Simpson',
      password: '$2b$10$EpRnTzVlqHNP0.fUbXUwSOyuiXe/QLSUG6xNekdHgTGmrpHEfIoxm', // secret42
      role: 'USER',
      posts: {
        create: {
          title: 'Join us for Prisma Day 2019 in Berlin',
          content: 'https://www.prisma.io/day/',
          published: true,
        },
      },
    },
  });

  const user2 = await prisma.user.create({
    data: {
      email: 'bart@simpson.com',
      firstname: 'Bart',
      lastname: 'Simpson',
      role: 'ADMIN',
      password: '$2b$10$EpRnTzVlqHNP0.fUbXUwSOyuiXe/QLSUG6xNekdHgTGmrpHEfIoxm', // secret42
      posts: {
        create: [
          {
            title: 'Subscribe to GraphQL Weekly for community news',
            content: 'https://graphqlweekly.com/',
            published: true,
          },
          {
            title: 'Follow Prisma on Twitter',
            content: 'https://twitter.com/prisma',
            published: false,
          },
        ],
      },
    },
  });

  console.log({ user1, user2 });
}

main()
  .catch((e) => console.error(e))
  .finally(async () => {
    await prisma.$disconnect();
  });

In contrast, in other frameworks, for example in ruby rake you can define tasks and dependencies between those tasks.

task name: [:prereq1, :prereq2] do |t|
  # actions (may reference t)
end

Another example is in gulp where compoosed tasks can be used to build a tree of dependencies between tasks.

const { series, parallel } = require('gulp');

function clean(cb) {
  // body omitted
  cb();
}

function css(cb) {
  // body omitted
  cb();
}

function javascript(cb) {
  // body omitted
  cb();
}

exports.build = series(clean, parallel(css, javascript));

This article shows a way to achieve the same pattern of tasks and tasks dependencies by utilizing Nest base app DI mechanism for tasks execution system.

Architecture

The DI (dependency injection) system of Nest, allows declaring services and in those services, declare their dependency on other services.

This means if service A lifecycle of initialization (OnInit, constructor..) is used for performing a task, then service B that depends on it, can be sure that when it itself is initialized, then service A has finished its initialization and task.

That is the main solution. Using DI to declare services that depends on other services. A service get initialize only after its dependencies finished to be initialized also.

Solution Walkthrough

seed.ts

In the root file, you create a standalone Nest app. Then seeding services can be declared on the app module or be divided to sub modules (Dev, Stage, Common...).

import { NestFactory } from "@nestjs/core";
import { DevelopmentSeederModule } from "./seeders/development/DevelopmentSeederModule";

async function main() {
  await bootstrap();
}

async function bootstrap() {
  const seederModule = DevelopmentSeederModule;

  const app = await NestFactory.createApplicationContext(seederModule, {
    logger: [
      "error",
      "warn",
      "log",
      // "debug",
      // "verbose"
    ],
  });

  // Starts listening for shutdown hooks
  app.enableShutdownHooks();

  await app.close();
}

main()
  .catch((e) => console.error(e));

DevelopmentSeederModule

Each seeding module declares a seeding services. The provideSeederService utility is used here to declare an async factory seeder service.

import { Logger, Module, OnModuleDestroy, OnModuleInit, Scope } from '@nestjs/common';
import { CommonSeederModule } from '../common/CommonSeederModule';
import { provideSeederService } from '../common/utils/provideSeederService';
import { UsersSeeder } from './UsersSeeder.service';
import { UserFactory } from './UserFactory.service';
import { PrismaService } from '../common/Prisma.service';

@Module({
  imports: [CommonSeederModule],
  providers: [
    {
      provide: UserFactory,
      useFactory: ({ prisma }: PrismaService) => {
        const instance = new UserFactory(prisma);
        return instance;
      },
      inject: UserFactory.$inject,
    },
    provideSeederService(UsersSeeder),
  ],
})
export class DevelopmentSeederModule implements OnModuleInit, OnModuleDestroy {
  private readonly logger = new Logger(DevelopmentSeederModule.name);

  onModuleInit() {
    this.logger.log('Module init ...');
  }

  onModuleDestroy() {
    this.logger.log('Module destroy ...');
  }
}

provideSeederService.ts

Since seeding the DB is asynchronous work, we can not rely on the constructor to be the seeding point of work.

Instead, Nest asynchronous providers can be used to mark when the service has finished its initialization - its seeding work.

Each seeding service, needs to implement an interface consist of a async seed function that the factory activate in the initialization process.

import { PrismaService } from '../Prisma.service';
import { SeederClassType } from './SeederClassType';

export function provideSeederService<T extends SeederClassType>(
  seederClassType: T
) {
  const factory = async ({ prisma }: PrismaService, ...rest: any[]) => {
    const seederService = new seederClassType(prisma, ...rest);
    await seederService.seed();
    return seederService;
  };

  return {
    // the class will be used as DI token
    provide: seederClassType,
    useFactory: factory,
    inject: seederClassType.$inject,
  };
}

UserSeeder.service.ts

A seeding service consists of a custom $inject property that the factory use for declaring its dependencies and an async seed function that do the seeding work.

import { Logger } from '@nestjs/common';
import { PrismaClient } from '@prisma/client';
import { PrismaService } from '../common/Prisma.service';
import { faker } from '@faker-js/faker';
import { hash } from 'bcrypt';
import { UserFactory } from './UserFactory.service';

export class UsersSeeder {
  static $inject = [PrismaService, UserFactory];

  private readonly logger = new Logger(UsersSeeder.name);

  constructor(private prisma: PrismaClient, private userFactory: UserFactory) {}

  async seed() {
    this.logger.log('Seeding users...');

    await this.prisma.user.deleteMany();

    //const { BCRYPT_SALT } = process.env;
    // const salt = parseSalt(BCRYPT_SALT);

    await this.userFactory.create();

    this.logger.log('Seeding users done');
  }
}

UserFactory.service.ts

A model factory service is used as an encapsulation of the create logic for an entity. This includes generating random values and/or use other factories to create dependencies of the model.

import { Logger } from "@nestjs/common";
import { PrismaClient } from "@prisma/client";
import { PrismaService } from "../common/Prisma.service";
import { faker } from "@faker-js/faker";

export class UserFactory {
  static $inject = [PrismaService];

  private readonly logger = new Logger(UserFactory.name);

  constructor(private prisma: PrismaClient) {}

  async create() {
    const user = await this.prisma.user.create({
      data: {
        email: 'lisa@simpson.com',
        firstname: 'Lisa',
        lastname: 'Simpson',
        password: '$2b$10$EpRnTzVlqHNP0.fUbXUwSOyuiXe/QLSUG6xNekdHgTGmrpHEfIoxm', // secret42
        role: 'USER',
      },
    });

    return user;
  }
}

CommonSeederModule

import { Logger, Module, OnModuleDestroy, OnModuleInit } from "@nestjs/common";
import { provideSeederService } from "./utils/provideSeederService";
import { CurrenciesSeeder } from "./CurrenciesSeeder.service";
import { PrismaService } from "./Prisma.service";

@Module({
  providers: [
    PrismaService,
    provideSeederService(CurrenciesSeeder),
  ],
  exports: [
    PrismaService,
    CurrenciesSeeder,
  ],
})
export class CommonSeederModule implements OnModuleInit, OnModuleDestroy {
  private readonly logger = new Logger(CommonSeederModule.name);

  onModuleInit() {
    this.logger.log("Module init ...");
  }

  onModuleDestroy() {
    this.logger.log("Module destroy ...");
  }
}

Prisma.service.ts

import { OnModuleDestroy } from "@nestjs/common";
import { PrismaClient } from "@prisma/client";

export class PrismaService implements OnModuleDestroy {
  prisma: PrismaClient;

  constructor() {
    this.prisma = new PrismaClient();
  }

  onModuleDestroy() {
    this.prisma.$disconnect();
  }
}

SeederClassType.ts

export type SeederClassCtor = new (...args: any[]) => any;

export type SeederClassType = SeederClassCtor & {
  $inject: any[];
};

Summary

As you can see, Nest base app with the DI mechanism can be utilizes as a task runner - service initialization - with the ability to define dependencies between the tasks - injection of services into other services.

By using this pattern, you can achieve an asynchronous task runner system with dependencies capability.

react-i18next useTranslation Does not Get Same Provider

Eyal Lapid — Wed, 09 Feb 2022 07:11:12 +0000

^{Photo by Amador Loureiro on Unsplash}

Scenario

Using react-i18next with I18nextProvider instead of one global provider.

In our case, in a hybrid app with Angular and React, the i18next instance was generated in the Angular app. The React sub app needed that instance instead of its own.

js monorepo using yarn workspaces and yarn pnp
angular app with webpack bundler - bundle all including libraries and external third parties
react sub apps with rollup bundler - leaves out external to be bundle by the end app

Case Problem

At first, it seems that the provider was receiving the instance correctly.

But the useTranslation hook, when asking for the react i18next conext, always received and empty one. Not the instance that was set on the provider

  const { i18n: i18nFromContext, defaultNS: defaultNSFromContext } = useContext(I18nContext) || {};

Solution - TLDR

Make the js bundler provider the same react-i18next library instance for both apps/libraries.

For webpack and yarn pnp, we use the reslove alias configuration:

   resolve: {
      alias: {
        'react-i18next': require.resolve('react-i18next'),
      },
      extensions: ['.ts', '.js'],
    },

Solution Walkthrough

To debug the situation, the context can be marked by adding a property guid to the object.

import * as rx from '@proftit/rxjs';
import { I18nextProvider, I18nContext } from 'react-i18next';
import { TranslationsContext } from './TranslationsContext';
import { useObservable, useObservableState } from 'observable-hooks';

export function TranslationsProvider({ value, children }) {
  const i18n = useMemo(() => {
    value.__guid = 'translation-provider-top';
    return value;
  }, [value]);

  return (
    <TranslationsContext.Provider value={value}>
      {i18n && <I18nextProvider i18n={i18n}>{children}</I18nextProvider>}
    </TranslationsContext.Provider>
  );
}

Then, in the useTranslation that does not work at the further down component, we can step into the useTranslation code and see if the useContext gives the same instance with the __guid property on it.

Since it does not, this indicate that the I18nContext given as a token for the useContext to search the correct context in its storage, is not the same.

To prove it, again, we can import the I18nContext in the TranslationProvider and tag it also. Then in the useTranslation we see it is not the same instance

import { getI18n, getDefaults, ReportNamespaces, I18nContext } from './context';

// ....
  const { i18n: i18nFromContext, defaultNS: defaultNSFromContext } = useContext(I18nContext) || {};

This phenomenon is explained in more details here

In some js bundling scenarios, one library gets a different instance of the js module of third party library like react-i18next then the other library.

Since React useContext works on reference equality for the tokens for its dependency injection like mechanism, it does not find the correct context.

This can happen more generally for other libraries that stumble on the same use case situation. The solution for webpack is the same

    resolve: {
      alias: {
        'react-i18next': require.resolve('react-i18next'),
        '@emotion/react/jsx-runtime': emotionJsxPath,
        '@emotion/react': require.resolve('@emotion/react'),
      },
   }

Storybook Solution

This also happens in a monorepo with storybooks packages as self packages.

The solution is the same. You need to configure storybook main.js webpack settings to resolve the the import to the same package instance.

main.js

module.exports = {
  resolve: {
      alias: {
        'react-i18next': require.resolve('react-i18next'),
      },
      extensions: ['.ts', '.js'],
    }
}

References

Debug Jest Spec In VsCode - Error missing ) after argument list

Eyal Lapid — Sat, 29 Jan 2022 18:52:08 +0000

^{Photo by Ferenc Almasi on Unsplash}

Scenario

Trying to debug a Jest spec in Vscode, leads to an error:

basedir=$(dirname "$(echo "$0" | sed -e 's,\\,/,g')")
          ^^^^^^^
SyntaxError: missing ) after argument list

Solution

There is a problem in the configuration. The default config tries to run the sh shell script of jest as a node program.

Instead, we can skip the sh jest script and run directly the jest nodejs script.

Instead of:

"${workspaceRoot}/node_modules/.bin/jest"

We replace it with:

"${workspaceRoot}/node_modules/jest/bin/jest.js",

   {
      "name": "vscode-jest-tests",
      "type": "node",
      "request": "launch",
      "runtimeArgs": [
        "--inspect-brk",
        "${workspaceRoot}/node_modules/jest/bin/jest.js",
        "--runInBand"
      ],
      "console": "integratedTerminal",
      "internalConsoleOptions": "neverOpen",
      "port": 9229
    }

Solution Walkthrough

In this case, I'm using pnpm. pnpm is creating a bin that is a bash script that calls node to run the actual bin nodejs script.

node_modules/.bin/jest

#!/bin/sh
basedir=$(dirname "$(echo "$0" | sed -e 's,\\,/,g')")

case `uname` in
    *CYGWIN*) basedir=`cygpath -w "$basedir"`;;
esac

if [ -z "$NODE_PATH" ]; then
  export NODE_PATH=".........."
else
  export NODE_PATH="$NODE_PATH:/........"
fi
if [ -x "$basedir/node" ]; then
  exec "$basedir/node"  "$basedir/../jest/bin/jest.js" "$@"
else
  exec node  "$basedir/../jest/bin/jest.js" "$@"
fi

pnpm linkbin package

async function linkBin (cmd: CommandInfo, binsDir: string, opts?: { extendNodePath?: boolean }) {
  const externalBinPath = path.join(binsDir, cmd.name)

  try {
    let nodePath: string[] | undefined
    if (opts?.extendNodePath !== false) {
      nodePath = await getBinNodePaths(cmd.path)
      const binsParentDir = path.dirname(binsDir)
      if (path.relative(cmd.path, binsParentDir) !== '') {
        nodePath = union(nodePath, await getBinNodePaths(binsParentDir))
      }
    }
    await cmdShim(cmd.path, externalBinPath, {
      createPwshFile: cmd.makePowerShellShim,
      nodePath,
      nodeExecPath: cmd.nodeExecPath,
    })
  } catch (err: any) { // eslint-disable-line
    if (err.code !== 'ENOENT') {
      throw err
    }
    globalWarn(`Failed to create bin at ${externalBinPath}. The source file at ${cmd.path} does not exist.`)
    return
  }

cmd-shim

  let sh = `\
#!/bin/sh
basedir=$(dirname "$(echo "$0" | sed -e 's,\\\\,/,g')")
case \`uname\` in
    *CYGWIN*) basedir=\`cygpath -w "$basedir"\`;;
esac

So the bin that the jest vscode launch configuration is targeting the bash script. The runner is set as a nodejs runner script. The script that it is trying to run is a sh script. That is the problem.

launch.json

   "type": "node",
    ...
    "${workspaceRoot}/node_modules/.bin/jest"

We can find the atrribute for the launch.json here

So, instead of running the sh script, we redirect the launcher to run directly the jest nodejs script.

VsCode Shortcut - Open New Terminal Here

Eyal Lapid — Wed, 19 Jan 2022 06:02:57 +0000

^{Photo by Yancy Min on Unsplash}

Scenario

Sometimes you want to open a terminal at the place of the current file folder you are working on, and do it fast and easily.

In vscode, you needed to go to the folder at the tree view and select the option Open in integrated terminal.

This invloves mouse work.

Solution

Instead, VsCode is exposing a new command newWithCwd that allow to bind keys to open new terminal in the current file path.

  {
    "key": "ctrl+shift+t",
    "command": "workbench.action.terminal.newWithCwd",
    "args": {
      "cwd": "${fileDirname}"
    }
  }

References

stackoverflow

Using NX Distributed Execution In Github Actions CI

Eyal Lapid — Fri, 07 Jan 2022 18:28:36 +0000

^{Photo by Ruslan Bardash on Unsplash}

Scenario

In building and testing a monorepo in CI, there are problems regarding splitting the tasks in order to lower the time of the CI run.

NX Solution

Nx has a feature - distributed task execution to run tasks parallel in CI that helps speed up building and testing a monorepo.

Unlike manually splitting the build and tests into different jobs, in NX distributed execution the tasks are distributed by a automated manager. The user decided the number of workers to give and the manager utilizes them.

TLDR

`CI.yaml` - Main Section

name: CI
on:
  push:
    branches:
      - main
  pull_request:
  workflow_dispatch:

env:
  NX_CLOUD_DISTRIBUTED_EXECUTION: true

`CI.yaml` - Job - Main

jobs:
  main:
    runs-on: ubuntu-latest
    if: ${{ github.event_name != 'pull_request' }}
    steps:
      - uses: actions/checkout@v2
        name: Checkout [main]
        with:
          fetch-depth: 0

      - name: Derive appropriate SHAs for base and head for `nx affected` commands
        uses: nrwl/nx-set-shas@v2

      - uses: pnpm/action-setup@v2.0.1
        with:
          version: 6.22.2

      - uses: actions/setup-node@v2
        with:
          node-version: '17'
          cache: 'pnpm'
      - run: pnpm install
      - run: pnpm exec nx-cloud start-ci-run
      - run: pnpm exec nx affected --target=build --parallel --max-parallel=3
      - run: pnpm exec nx affected --target=lint --parallel --max-parallel=2
      - run: pnpm exec nx affected --target=test --parallel --max-parallel=2

      - name: Clean Agents
        if: ${{ always() }}
        run: pnpm exec nx-cloud stop-all-agents

`CI.yaml` - Job - PR

  pr:
    runs-on: ubuntu-latest
    if: ${{ github.event_name == 'pull_request' }}
    steps:
      - uses: actions/checkout@v2
        with:
          ref: ${{ github.event.pull_request.head.ref }}
          fetch-depth: 0

      - name: Derive appropriate SHAs for base and head for `nx affected` commands
        uses: nrwl/nx-set-shas@v2

      - uses: pnpm/action-setup@v2.0.1
        with:
          version: 6.22.2

      - uses: actions/setup-node@v2
        with:
          node-version: '17'
          cache: 'pnpm'

      - run: pnpm install
      - run: pnpm exec nx-cloud start-ci-run
      - run: pnpm exec nx affected --target=build --parallel --max-parallel=3
      - run: pnpm exec nx affected --target=lint --parallel --max-parallel=2
      - run: pnpm exec nx affected --target=test --parallel --max-parallel=2
      - name: Clean Agents
        if: ${{ always() }}
        run: pnpm exec nx-cloud stop-all-agents

`CI.yaml` - Job - Agents

  agents:
    runs-on: ubuntu-latest
    name: Agent - General
    timeout-minutes: 15
    strategy:
      matrix:
        agent: [1, 2]
    steps:
      - uses: actions/checkout@v2'

      - uses: pnpm/action-setup@v2.0.1
        with:
          version: 6.22.2

      - uses: actions/setup-node@v2
        with:
          node-version: '17'
          cache: 'pnpm'

      - run: pnpm install
      - name: Start Nx Agent ${{ matrix.agent }}
        run: npx nx-cloud start-agent

Solution Walkthrough

Activating the Distributed Execution Manager

In order for the NX commands to know this they should use the manager for work dispatching, they have to have the environment variable NX_CLOUD_DISTRIBUTED_EXECUTION to true.

We put this variable environment wide for all jobs and steps.

env:
  NX_CLOUD_DISTRIBUTED_EXECUTION: true

If we want to disable the usage of distribute tasks for a specific NX command we can set this variable to false temporarily for a specific step for instance.

      - name: Publish artifacts
        run: pnpm exec nx affected --target=custom-publish --parallel=2 --commitid=$GITHUB_SHA
        env:
          NX_CLOUD_DISTRIBUTED_EXECUTION: false

NX distributed execution can only run commands that are specified as cachable in the NX comnfig. Commands that are not specified as cachable gives error.

Checking out Branch

To be able to determine which packages were changed - meaning needing build and test - on the main and pr branches, we want to retrieve all the repo commits. Otherwise, NX Affected command will not be able to compare commits regarding packages changes.

We do this by specifying to download all commits in the checkout action.

fetch-depth: 0 - Specify to download all commits

For determine changes of pr branches, we want to checkout not the special branch Github creates which is the merge between the PR branch and the base it is against, but the PR branch head only.

This is done by specifying ref: ${{ github.event.pull_request.head.ref }}

The event.pull_request.head.ref contains the branch name which is the reference to the head commit

The tests and builds themselves will be performed in the agents which will checkout the merged PR commit as usual.

Job - main

      - uses: actions/checkout@v2
        name: Checkout [main]
        with:
          fetch-depth: 0

Job - pr

      - uses: actions/checkout@v2
        with:
          ref: ${{ github.event.pull_request.head.ref }}
          fetch-depth: 0

Helping NX Affect Command Comparison

For PR's we want to compare the head of the PR branch against the base of the PR.

For main branch commits, the desire is to find changes against the latest commit that was succesful.

NX provide an action utility that calculate those head and base commits and set those to environment variables - NX_BASE and NX_HEAD - used by the NX Affected command.

      - name: Derive appropriate SHAs for base and head for `nx affected` commands
        uses: nrwl/nx-set-shas@v2

Starting the Distributed Execution Manager

To start the manager daemon, we execute nx-cloud utility with start-ci-run command.

      - run: pnpm exec nx-cloud start-ci-run

Dispatching a Distributed Task

The NX Affect command run without customization.

The parallel/max-parallel does not specify the number of workers/agents but specify the number of processes to run parallel on each agent and on the manager.

      - run: pnpm exec nx affected --target=build --parallel --max-parallel=3

Cleanup Agent After Success and Failure

After the job is done or after a build error, we want to signal the agents to shutdown and not keep waiting for future tasks to execute.

This is done by activating nx-cloud stop-all-agents.

In order to do this step even on error, the step needs to specify that it will run always and not just on success.

if: ${{ always() }} - Do the step always

      - name: Clean Agents
        if: ${{ always() }}
        run: pnpm exec nx-cloud stop-all-agents

Agent Job Configuration

The Agent has one shared configuration. To parallel and multiple the number of agents node, we use the matrix feature of the job.

It will create parallel worker nodes, each with different agent name variable value.

    strategy:
      matrix:
        agent: [1, 2]

Rest of the agent job configuration is standard. Checking out the repo merged commit, setting up nodejs, and starting up the agent server that listen for the tasks manager task dispatches.

      - name: Start Nx Agent ${{ matrix.agent }}
        run: npx nx-cloud start-agent

Conclusion

For monorepo scalable testing and building, checkout NX and NX Cloud

It provide an interesting solution for efficient parallel task execution to speedup monorepos CI workflows.

Linux - Find in Multiple Folders and Delete Files Within

Eyal Lapid — Sun, 19 Dec 2021 08:05:07 +0000

This is a worknote.

Scenario

In a Javascript monorepo, there was a need to:

delete all dist folders files, not the folders themselfs
but without touching the .gitignore in the folders

TLDR

dir=$(find ./packages -name "dist" -type d);

for i in $dir; do find $i -type f \( -iname "*" ! -iname ".gitignore" \) -exec rm {} +; done

Solution

Find all dist folders in the monorepo
Iterate over the folders set and collect all files within them, excluding .gitignore
Delete all sets of found files.

Solution Walkthrough

Find all folders with specific name

find ./packages -name "dist" -type d

find - find - find files. Allow filtering.

./packages - target root folder to start the search from.

-name "dist" - filter only object with the name "dist".

-type d - filter only object with type of directory.

Execute Bash Expression

Exectute an expression in bash and put the results in a local variable.

dir=$()

Find all relevant files in a folder

find $i -type f \( -iname "*" ! -iname ".gitignore" \)

$i - Variabe containing the folder name from the for iteration.

-type f - filter object only of type of file.

-iname - From the docs: Like -name, but the match is case insensitive.

! - From the docs: ! expr True if expr is false. This character will also usually need protection from interpretation by the shell.

$ - From the docs: ( expr ) Force precedence. Since parentheses are special to the shell, you will normally need to quote them. Many of the examples in this manual page use backslashes for this purpose: \(...$ instead of (...)

for i in $dir; do find $i -type f \( -iname "*" ! -iname ".gitignore" \); done

Run through folders set and execute find on them

for i in $dir; do find $i; done

for - for loop

$i - iteration variable

Execute delete on found set

find $i -exec rm {} +

-exec command {} + - From the docs: his variant of the -exec action runs the specified command on the selected files, but the command line is built by appending each selected file name at the end.

The string {} is replaced by the current file name being processed everywhere it occurs in the arguments to the command, not just in arguments where it is alone, as in some versions of find.

RxJS in React - From Class to Functional - Part 1 Class Components

Eyal Lapid — Sat, 25 Sep 2021 07:07:29 +0000

^{^{Image is from Pixabay user kordi_vahle}}

React functional component and hooks pose a slight different model of operation then the standard component lifecycle model.

In order to use RxJS effectively in React functional components with hooks, we need to adjust some of the pattern we use in regular components to the new paradigm.

This is part 1 of the series. It deals with RxJS patterns for React Class components.

Part 2 will handle React functional components patterns and what is the logic behind them.

RxJS in Regular Class Components

Component Lifecycles

In regular components frameworks, there is an on init lifecycle that runs once, on change lifecycle that runs on each component's input change, and on destroy lifecycle that get called when the component is removed and discarded.

The analog to those lifecycles in React are:

componentDidMount for onInit functionality,
componentDidUpdate for onChange functionality, and
componentWillUnmount for onDestroy functionality.

class Calculator {
  constructor(props) {
    // ... 
  }

  componentDidMount() {
    // ...
  }

  componentDidUpdate(prevProps, prevState, snapshot) {
    // ... 
  }

  componentWillUnmount() {
    // ...
  }
}

Using RxJS with Class Component Lifecycles

Initialization

The componentDidMount callback and the constructor allow to create the main streams that will guide the component logic and bring them to life - to subscribe to them.

class Calculator {
  constructor(props) {
    super(props);

    /*
     * Initialization of callbacks fucntions 
     * to send stream events
     */
    this.onNumber = (num) => this.numAction$.next(num);
    this.onMathOperation = (operation) => this.mathOperation$.next(operation);

    /*
     * Initialize reflected state of the main streams values.
     * To be used in the view layer.
     */
    this.state = {
      result: "",
      expr: []
    };
  }

  componentDidMount() {
    /* 
     * Setting up the event Rx subjects. Will be used in 
     * the main streams.
     */
    this.onUnmount$ = new Subject();
    this.numAction$ = new Subject();
    this.mathOperation$ = new Subject();

    /* 
     * Setting up the main logic streams.
     */
    this.mathExpression$ = this.streamMathExpression();
    this.result$ = this.streamResult();

    /* 
     * 1. Reflecting the main streams values into
     *    regular state to be used in the view.
     * 2. Subscribing to the main streams to bring
     *    them to life for the duration of the 
     *    component.
     */
    merge(
      this.result$.pipe(
        tap((result) => this.setState((state) => ({ ...state, result })))
      ),
      this.mathExpression$.pipe(
        tap((expr) => this.setState((state) => ({ ...state, expr })))
      )
    )
      .pipe(takeUntil(this.onUnmount$))
      .subscribe();
  }

 /* 
  * Main stream expression
  */
 streamMathExpression() {
    return pipe(
      () =>
        merge(
          this.numAction$.pipe(
            map((num) => ({ action: "number", value: num }))
          ),
          this.mathOperation$.pipe(
            map((op) => ({ action: "operation", value: op }))
          )
        ),
      scan((expr, { action, value }) => {
        // reducer logic for the math expression
      }, [])
    )(null);
  }
}

Here, in the init phase of the component, several things happening:

Main observables values reflection into React component state.
Creating Rx Subjects to be used and notify the main streams of changes and events.
Creating and wiring up the main streams that handle the component behavior and state management.

Using Values in the View

This is needed for using the observable values in the view jsx layer. In Angular for instance, this is done with the async pipe that subscribe and provide the observable value automatically in Angular HTML template expressions.

Initialization of React component state in the constructor:

constructor() {
  this.state = {
      result: "",
      expr: []
    };

Reflecting and updating the main observables values into the React component states. This is done with subscribing to the observables and updating the React component state.

componentDidMount() {
merge(
      this.result$.pipe(
        tap((result) => this.setState((state) => ({ ...state, result })))
      ),
      this.mathExpression$.pipe(
        tap((expr) => this.setState((state) => ({ ...state, expr })))
      )
    )
      .pipe(takeUntil(this.onUnmount$))
      .subscribe();
}

Creating Rx Subjects to be used and notify the main streams of changes and events

The main streams subscribe to the subjects that are used in the view or in other stream to notify of changes or events such as click and timer events.

constructor(props) {
    super(props);

    this.onNumber = (num) => this.numAction$.next(num);
    this.onMathOperation = (operation) => this.mathOperation$.next(operation);
}
 componentDidMount() {
    this.onUnmount$ = new Subject();
    this.numAction$ = new Subject();
    this.mathOperation$ = new Subject();
}

Main Streams Logic

The main streams of the component. Those needs to be created and brought to life - subscribed to - only once and live for the rest of the component life until it destroyed.

componentDidMount() {
    this.mathExpression$ = this.streamMathExpression();
    this.result$ = this.streamResult();

    merge(
      this.result$.pipe(
        // ...
      ),
      this.mathExpression$.pipe(
        // ...
      )
    )
      .pipe(takeUntil(this.onUnmount$))
      .subscribe();
}

Input Change Detection

The onChange callback allows to push detected input changes into RxJS subjects. The main streams of the component then can listen on those subject and react to component input changes.

Gathering the changes events:

componentDidMount() {
  this.propsChanges$ = new Subject();
}

componentDidUpdate(prevProps) {
    this.propsChanges$.next({ prev: prevProps, current: this.props });
}

Listening and notifying relevant prop changes:

 componentDidMount() {
    this.inputMode$ = handlePropChange("mode", this.propsChanges$);

    merge(
      this.inputMode$,
    )
      .pipe(takeUntil(this.onUnmount$))
      .subscribe();
  }

function handlePropChange(propName, propsChanges$) {
  return propsChanges$.pipe(
    filter(({ prev, current }) => prev[propName] !== current[propName]),
    map(({ current }) => current[propName]),
    shareReplay({ bufferSize: 1, refCount: true })
  );
}

Cleanup

The componentWillUnmount callback allows to cleanup subscriptions of the main logic streams so no dangling subscriptions will be left after the component is destroyed.

This is done through another subject - onUnmount - that all other main streams subscription depends on so when it emit and complete, the subscriptions of the main streams complete as well.

The takeUntil operator subscribe to the unMount$ subject and complete the main subscriptions when unMount complete.

componentDidMount() {
    this.onUnmount$ = new Subject();
    this.mathExpression$ = this.streamMathExpression();
    this.result$ = this.streamResult();

    merge(
      this.result$.pipe(
        // ...
      ),
      this.mathExpression$.pipe(
        // ...
      )
    )
      .pipe(takeUntil(this.onUnmount$))
      .subscribe();
  }

 componentWillUnmount() {
    this.onUnmount$.next();
    this.onUnmount$.complete();
  }

Full example code can be found on github;

Conclusion

This part dealt with RxJS patterns for React class components. Specifically we saw:

How to create main streams for the duration of the component life.
Use the streams values in the view layer.
Listen on component input props changes.
Cleanup subscriptions when the component is destroyed.

Next part will show how to update those patterns to React functional components with hooks and the logic behind those patterns.

Disable Domain Bare Request HTML Cache for SPA

Eyal Lapid — Mon, 13 Sep 2021 07:51:21 +0000

Cover image is taken from pixabay/chanwity

This is a #worknote.

Scenario

A SPA app that is served by a server (static Nginx or Nodejs ssr)
A reverse proxy that direct traffic from outside to the app server
All SPA assets are generated with fingerprints in the file names. Only the main index.html stays as is with no fingerprint.

When deploying a new version of the SPA app, clients need to manually clear browser cache, because they go to the index.html which stays with the same name for each version. The browsers do not know no to cache index.html coming from domain request.

We want to disable cache for the Bare domain response which returns the index.html, but keep the other cache mechanism in place for all other assets.

Solution

The kind global community provides some suggestions and solutions.

The solution that worked for us was using the expires Nginx directive with conditions to only disable cache for html documents.

nginx.conf

map $sent_http_content_type $expires {
  default                    off;
  text/html                  epoch; # means no-cache
}

server { 
 ...
 expires $expires;
 ...

map $sent_http_content_type $expires {
  default                    off;
  text/html                  epoch; # means no-cache
}

server { 
 listen 80;
 server_name frontend;
 expires $expires;

 location / {
   proxy_set_header X-Real-IP $remote_addr;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_set_header X-NginX-Proxy true;
   proxy_pass http://localhost:8082;
   proxy_ssl_session_reuse off;
   proxy_set_header Host $http_host;
   proxy_cache_bypass $http_upgrade;
   proxy_redirect off;
 }
}

Solution Walkthrough

Nginx Map Directive

The ngx_http_map_module module creates variables whose values depend on values of other variables.

The map directive can be thought of as a function that wrap a switch-case statement. The function accept an input parameters and return the result of the switch-case for that input value.

In this case, the expression $expires in the statement expires $expires in converted to a function that gets the content-type header when the expires directive comes to action.

Pseudo Javascript code representing the expansion.

import { expires } from 'nginx';

const $expires = ({ contentType }) => {
  if (contentType === 'text/html') return 'epoch';
  return 'off';
}

const siteConfig = {
  // ...
  expires: (response) => expires($expires(response)),
  // ...
};

$sent_http_content_type

We can access the response header content type using this variable.

From the docs of Nginx core Embedded Variables:

$sent_http_name
arbitrary response header field; the last part of a variable name is the field name converted to lower case with dashes replaced by underscores

Other References

Oreilly Nginx Http Server Docs

NGinx Expires Directive

The main purpose of the directive is to add headers to the response related to cache and cache expiration.

From the docs

Enables or disables adding or modifying the “Expires” and “Cache-Control” response header fields

The off value

The off parameter disables adding or modifying the “Expires” and “Cache-Control” response header fields.

The epoch value

The epoch parameter sets “Expires” to the value “Thu, 01 Jan 1970 00:00:01 GMT”, and “Cache-Control” to “no-cache”.

Other References

https://developer.mozilla.org/en-US/docs/Web/HTTP/Caching

https://www.docker.com/blog/how-to-use-the-official-nginx-docker-image/

Using Jest With AngularJS

Eyal Lapid — Thu, 09 Sep 2021 08:14:54 +0000

This is a summary of the exeperiences I had intergrating Jest into an AngularJs project.

This is a living document - work in progress.

angular.mock.module is not a function

This took sometime to investigate. There are some info on the web regarding this:

Mismatch of angular and angular-mock versions.
Order of loading angular and angular-mocks regarding the test framework.

In my case it was something more sinister. AngularJs itself skip and does not load the nessasery logic for angular.mock.module to work.

In angular-mocks.js file, Angular load the mock.module logic only if it detects Jamsine or Mocha test framework presents.

(function(jasmineOrMocha) {

  if (!jasmineOrMocha) {
    return;
  }

...

})(window.jasmine || window.mocha);

They describe it in a oneliner of the api site also:

NOTE: This function is declared ONLY WHEN running tests with jasmine or mocha

Solution

Before loading angular-mocks, set the global jasmine or mocha to truthy value.

setup-jest.js

global.mocha = true;
...

jest-config.js

...
setupFiles: [
  './src/setup-jest.js',
],
...

Activating Outbound Attribute Expression in AngualrJs Directive

Eyal Lapid — Thu, 07 Jan 2021 05:55:42 +0000

This is a #worknote.

Scenario

In cases where an attribute directive is needed and a component cannot be used and the desire is for the directive to pass some data to the using parent component.

In a component, we can use & binding. But in a directive, we need to do the & work ourselves.

Solution

Directive Controller

function ($scope, $attrs, $parse) {
  const exprFn = $parse($attrs.myDirective);

  const newScope = $scope.$new(false, $scope);
  newScope.variableToPass = 1234;

  exprFn(newScope);

  newScope.$destroy();
}

Parrent Component

<div my-directive="$ctrl.setValue(variableToPass)"></div>

$attrs is a special object that for each injected component/directive contains the attrs of the attached element.
If myDirective is registered as myDirective, it will also be in $attrs object as myDirective by the same name.
Angular $parse service allows us to compile a string representing an AnuglarJs javascript expression into an activation function. That function can be activated which in turn runs that expression in the context of the params given to that function
We create a new scope so the outside user of the directive can use the parent scope variables also in the provided expression in case needing them. Otherwise we would just do exprFn({ someVar: 1234 })
new scopes need to be disposed of to prevent leak.

Forem: Eyal Lapid

Converting Extension from JS to JSX

Chosen Solution

Breakdown of Script

list All Files Nested in a Target Folder

Filter Out Only Needed Files By Extension

Filter Out Files by Existence of Jsx Content Inside Them

Split the Output and Keep Only the File Names

Reduce the File List to Unique File Names

Construct a Rename Command for the Specific File Name

Execute the Rename Command

Process

Converting Large Codebase Project from Webpack to Vite

Reasons

Process

Steps

Nest IOC DI (Dependency Injection) as a Seeds Tasks Runner

Nest IOC DI (Dependency Injection) as a Seeds Tasks Runner

Why?

Architecture

Solution Walkthrough

Summary

react-i18next useTranslation Does not Get Same Provider

Scenario

Case Problem

Solution - TLDR

Solution Walkthrough

Storybook Solution

References

Debug Jest Spec In VsCode - Error missing ) after argument list

Scenario

Solution

Solution Walkthrough

VsCode Shortcut - Open New Terminal Here

Scenario

Solution

References

Using NX Distributed Execution In Github Actions CI

Scenario

NX Solution

TLDR

CI.yaml - Main Section

CI.yaml - Job - Main

CI.yaml - Job - PR

CI.yaml - Job - Agents

Solution Walkthrough

Activating the Distributed Execution Manager

Checking out Branch

Helping NX Affect Command Comparison

Starting the Distributed Execution Manager

Dispatching a Distributed Task

Cleanup Agent After Success and Failure

Agent Job Configuration

Conclusion

Linux - Find in Multiple Folders and Delete Files Within

Scenario

TLDR

Solution

Solution Walkthrough

Find all folders with specific name

Execute Bash Expression

Find all relevant files in a folder

Run through folders set and execute find on them

Execute delete on found set

RxJS in React - From Class to Functional - Part 1 Class Components

RxJS in Regular Class Components

Component Lifecycles

Using RxJS with Class Component Lifecycles

Initialization

Using Values in the View

Creating Rx Subjects to be used and notify the main streams of changes and events

Main Streams Logic

Input Change Detection

Cleanup

Conclusion

Disable Domain Bare Request HTML Cache for SPA

Scenario

Solution

Solution Walkthrough

Nginx Map Directive

Construct a `Rename` Command for the Specific File Name

Execute the `Rename` Command

`CI.yaml` - Main Section

`CI.yaml` - Job - Main

`CI.yaml` - Job - PR

`CI.yaml` - Job - Agents