Forem: vorillaz

Netflix But better

vorillaz — Thu, 06 Feb 2025 15:22:14 +0000

Attention: This project is not affiliated with Netflix, it's just a fun experiment.

Why We Need a Better Netflix

Netflix is the undisputed leader in streaming, with over 200 million subscribers. It offers a library of more than 17000 movies and TV shows from across the globe, and this catalog continues to expand.

One of the major features of the platform is its recommendation system. According to Netflix, the platform suggests new content based on your viewing history, similar shows, trends from other users, and popular titles. But what if we could replicate this system without relying too heavily on extensive data sets or complex algorithms?

In this project, we'll try to build a simple movie recommendation system that includes direct search capabilities, a vector-based recommendation engine, and a way to make recommendations using AI-generated content.

The Tech Stack

For this application, the foundation is built on Next.js, TailwindCSS, and Shadcn UI. The idea for the project came from a talk by Tejas Kumar, where he developed a movie recommendation system using Datastax. Our tech stack, however, draws significant inspiration from the book inventory project by Lee Robinson.

To ensure fast performance, we'll utilize Next.js's SSR capabilities, along with its Image component to pre-render images and aggressive prefetching of each page.

Data storage will be handled by a Neon cloud PostgreSQL database, while the Vercel AI SDK will power our embeddings and search queries. Finally, Drizzle will serve as the ORM to interact with the database.

Feeding the Data

We’ll start by using an open-source dataset containing over 8,000 movies and TV shows, available on Kaggle. The dataset is in CSV format, which we can import directly into the database to get started. Additionally, the OMDB API is used to fetch poster images and IMDb scores for each title.

The demo csv file uses cast as a column name, but in the database we'll use _cast as a column name to avoid conflicts with the cast function.

COPY tmp_titles(show_id, title, description, director, genres, _cast, production_country, release_date, rating, duration)
FROM '/data/netflix_titles.csv'
DELIMITER ','
CSV HEADER;

We’ll need to establish relationships between our tables, so we’ll create a few additional tables alongside a shows table to serve as the main source of data for our application.

Here's how we can define the shows table:

CREATE TABLE "shows" (
    "id" serial PRIMARY KEY NOT NULL,
    "title" text NOT NULL,
    "poster" text, -- url to the poster image
    "description" text, -- short description of the show
    "release_year" integer, -- year of release
    "duration" text, -- duration of the show
    "imdb_score" numeric, -- imdb score of the show
    "imdb_id" text, -- imdb id of the show
    "content_type" text -- type of the show (movie or tv show)
);

Next, we'll create a table for actors and set up many-to-many relationships between the shows and actors tables:

CREATE TABLE "actors" (
    "id" serial PRIMARY KEY NOT NULL,
    "name" text NOT NULL,
    CONSTRAINT "actors_name_unique" UNIQUE("name")
);

CREATE TABLE "show_actors" (
    "show_id" integer NOT NULL,
    "actor_id" integer NOT NULL,
    CONSTRAINT "show_actors_show_id_actor_id_pk" PRIMARY KEY("show_id", "actor_id")
);

This allows us to establish connections between shows and their actors. We can continue by recursively linking actors to the shows they've appeared in. This is done by matching the shows from the tmp_titles table with the shows table and parsing the _cast column to associate actors with their respective shows.

-- Insert actors from tmp_titles._cast into actors table
WITH RECURSIVE split_cast AS (
    SELECT DISTINCT -- Trim whitespace and split the _cast string on commas
        trim(unnest(string_to_array(_cast, ','))) as actor_name
    FROM tmp_titles
    WHERE _cast IS NOT NULL
)
INSERT INTO actors (name)
SELECT actor_name
FROM split_cast
WHERE actor_name != '' -- Skip empty strings
    ON CONFLICT (name) DO NOTHING;

This same process can be applied to the categories, directors, and countries tables.

Full Text Search

To improve search capabilities, we’ll add a search_text column to the shows table. This column will store a combination of text from the title, description, content_type, actors, directors, and categories columns, making it easier to perform full-text search queries.

We’ll utilize PostgreSQL’s pg_trgm extension to improve text search performance, which breaks down text into three-character sequences for more efficient matching.

-- Create the extension for text search
CREATE EXTENSION IF NOT EXISTS pg_trgm;
-- Add the search_text column
ALTER TABLE "shows"
ADD COLUMN "search_text" text;
-- Create the index for the search_text column

CREATE INDEX "idx_shows_search_text_tsv" ON "shows" USING gin (to_tsvector('english', "search_text"));

Once this migration is completed, we can populate the search_text column using a simple script:

import { db } from '@/lib/db';

const shows = await db.select().from(shows);

for (const show of batch) {
    const searchText = `
      ${show.title} 
      ${show.contentType}
      ${show.actors || ""} 
      ${show.directors || ""} 
      ${show.categories || ""}
    `
      .toLowerCase()
      .replace(/\n/g, " ")
      .replace(/\s+/g, " ");

    await db.update(shows).set({ searchText }).where(eq(shows.id, show.id));
  }

Now, with everything set up, we can perform full-text searches using the search_text column on the server side:

const exactMatches = await db
    .select({
      id: shows.id,
      title: shows.title,
      poster: shows.poster,
      imdbScore: shows.imdbScore,
    })
    .from(shows)
    .where(
      sql`to_tsvector('english', search_text) @@ to_tsquery('english', ${freeTextQuery})`
    )
    .orderBy(desc(shows.imdbScore))
    .limit(10);

The tsvector and tsquery functions are used to match the vector and the query against the search_text column.

Vector Search with AI Embeddings

While full-text search provides powerful functionality, it doesn’t fully meet our needs. We also want to support semantic searches, where users can find shows that are similar to a given title or based on abstract criteria.

This is where the Vercel AI SDK comes in. With the AI SDK, we can easily create a client that generates embeddings—vector representations of the metadata—enabling semantic search capabilities.

For the object detection model, we’ll use Mistral:

import { mistral } from "@ai-sdk/mistral";
export const objectModel = mistral("mistral-large-latest", {});

Next, we define the Zod schema for show metadata that the AI will use to generate embeddings:

const showMetadataSchema = z.object({
  mood: z.array(z.string()).optional(),
  targetAudience: z.array(z.string()),
  themes: z.array(z.string()),
  characters: z.array(z.string()),
  setting: z.array(z.string()),
  viewerEngagement: z.array(z.string()),
  plotComplexity: z.array(z.string()).optional(),
  plot: z.string().optional(),
  eras: z.array(z.string()).optional(),
});

Using this schema, we can ask the AI model to analyze a show’s information and generate metadata in the required format.

const result = await generateObject({
    // @ts-ignore
    model: objectModel,
    schemaName: "showMetadata",
    schemaDescription: "Information for semantic search",
    schema: showMetadataSchema,
    prompt: `
          Analyze the following show information and generate metadata, return the metadata as a JSON object.
          plotComplexity should be a list of  keywords that describe the plot complexity of the show.
          characters should be a list of character names.
          setting should be a list of keywords that describe the setting of the show. Use countries, places, cities or other locations that are relevant to the show.
          eras should be a list of keywords that describe the eras of the show.
          viewerEngagement should be a list of single keywords that describe the viewer engagement of the show, how the show sticks with the viewer.
          targetAudience should be a list of single keywords that describe the target audience of the show.
          mood should be a list of single keywords that describe the mood of the show.
          plot should be a single sentence that describes the plot of the show.

          Here is the show information:
          Title: ${show.title}
          Description: ${show.description || "N/A"}
          IMDB Rating: ${show.imdbScore || "N/A"}
          Type: ${show.contentType || "N/A"}
          Actors: ${show.actors || "N/A"}
          Categories: ${show.categories || "N/A"}
          Directors: ${show.directors || "N/A"}
          Duration: ${show.duration || "N/A"}
          Genres: ${show.genres || "N/A"}
          Based on this information, provide the metadata as specified in the schema.
        `,
  });

The result could look like this for Breaking Bad:

{
  "mood": [
    "Dark",
    "Intense",
    "Drama"
  ],
  "targetAudience": [
    "Adult",
    "Crime Enthusiasts",
    "Drama Lovers"
  ],
  "themes": [
    "Crime",
    "Drugs",
    "Family",
    "Morality"
  ],
  "characters": [
    "Walter White",
    "Jesse Pinkman",
    "Skyler White",
    "Hank Schrader",
    "Saul Goodman",
    "Gustavo Fring",
    "Mike Ehrmantraut",
    "Walter White Jr.",
    "Marie Schrader",
    "Steven Gomez"
  ],
  "setting": [
    "Albuquerque",
    "New Mexico",
    "United States"
  ],
  "viewerEngagement": [
    "Binge-worthy",
    "Edge of your seat",
    "Thought-provoking"
  ],
  "plotComplexity": [
    "Character-driven",
    "Moral dilemmas",
    "Twists and turns"
  ],
  "plot": "A high school chemistry teacher dying of cancer teams with a former student to secure his family's future by manufacturing and selling crystal meth.",
  "eras": [
    "2000s",
    "Contemporary"
  ]
}

With this metadata in hand, we can store the embeddings for each show in the shows table and enable vector search. First, we create the necessary extension and add a new column to the table.

The embedding column is a vector of size 1024, which is the default size for the Mistral model.

-- Create the extension
CREATE EXTENSION IF NOT EXISTS vector;
-- Add the embedding column
ALTER TABLE "shows"
ADD COLUMN "embedding" vector(1024);
-- Create the index for the embedding column
CREATE INDEX "embeddingIndex" ON "shows" USING ivfflat ("embedding" vector_cosine_ops);

Once the extension is set up, we generate and store the embeddings using the AI SDK:

import { mistral } from "@ai-sdk/mistral";
const embeddingModel = mistral.textEmbeddingModel("mistral-embed");

const shows  = await db.select().from(shows).limit(1);
const show = shows[0];
const embedding = await embed(`
    Title: ${show.title}
    Description: ${show.description || "N/A"}
    IMDB Rating: ${show.imdbScore || "N/A"}
    Type: ${show.contentType || "N/A"}
    Actors: ${show.actors || "N/A"}
    Categories: ${show.categories || "N/A"}
    Directors: ${show.directors || "N/A"}
`);

await db.update(shows).set({ embedding }).where(eq(shows.id, show.id));

With the embeddings in place, we can query the nearest neighbors for any given show:

const query = "A show with strong female characters";
const embedding = await embed(query);

const results = await db
    .select({
      id: shows.id,
      title: shows.title,
      poster: shows.poster,
      imdbScore: shows.imdbScore,
    })
    .from(shows)
    .where(sql`vector_cosine_ops(embedding, vector(${embedding})) > 0.5`);

    // Returns Borgen, Gilmore Girls, La Femme etc.

Moving Forward

If you want to try this out for yourself, you can clone the repository and run the project locally. You can find the repository here and the live demo here.

Try to add a new feature to the project, for example add a new type of recommendation or improve the search functionality using different models like OpenAI or Gemini. Additionally you can also expand the dataset to include more shows and movies as well as try to add more metadata to the shows.

Final Thoughts

In conclusion, by leveraging simple yet effective technologies like PostgreSQL, Next.js, and AI-driven embeddings, we can build a highly functional and scalable recommendation engine for a any data heavy platform.
The combination of full-text search and semantic AI search provides a more robust, personalized experience, allowing users to discover content that resonates with them based on both explicit and inferred preferences.

Next.js with Globally Accessible Variables

vorillaz — Mon, 21 Oct 2024 12:31:10 +0000

Next.js has been infamously criticized for its handling of environment variables. While it has been improved over the years, it still has some quirks that can be confusing especially when it comes to public variables exposed through the process.env object.

All the NEXT_PUBLIC_ prefixed variables are client-side accessible, but as mentioned in the official documentation, they are only available at build time, thus not being able to change at runtime when used with Docker.

Although I've covered this issue in the past, I've found a another approach that might be interesting to explore, using globally shared variables that can be changed at runtime.

Setup The Polyfill

Global variables are partially supported by Next.js, but we can use a polyfill to make them available using a small injected script.

// app/layout.tsx
<script
  dangerouslySetInnerHTML={{
    __html: `!function(t){function e(){var e=this||self;e.globalThis=e,delete t.prototype._T_}"object"!=typeof globalThis&&(this?e():(t.defineProperty(t.prototype,"_T_",{configurable:!0,get:e}),_T_))}(Object);`,
  }}
/>

The original code was taken from this comment on GitHub, and it basically creates a global globalThis object that can be used to share variables between the client and the server. The polyfill is coming with Next.js 14.x, but it seems to break the globalThis object in some browsers, that's why we are using the __html property to inject the code directly into the page.

Using Zod To Validate The Variables

Next up, we can use Zod to validate the variables at runtime, and throw an error if the variables are not valid. This step is crucial to ensure that the variables are always available and valid, avoiding runtime errors as well as exposing the application to security issues.

npm install zod

We then create a variables.ts file that will contain some utility functions to grab the variables from process.env and safely cast them to the expected type.

// lib/env.ts
import { z } from 'zod';

// Load the variables
export const load = () => {
  return process.env;
};

// Parse or throw the variables
export function parseOrThrow(schema: z.Schema, data: unknown, error: Error) {
  const parsed = schema.safeParse(data);
  // Log the errror
  if (parsed.success) return parsed.data;
  console.error(parsed.error);
  throw error;
}

// Some zod helpers to use
export const port = z
  .string()
  .refine(
    (port) => parseInt(port) > 0 && parseInt(port) < 65536,
    'Invalid port number'
  );

export const str = z.string();
export const url = z.string().url();
export const num = z.coerce.number();
export const bool = z.coerce.boolean();

load is a simple function that returns the process.env object, while parseOrThrow is a utility function that parses the variables using a Zod schema and throws an error if the variables are not valid.

Finally, we can create a variables.ts file that will contain the variables schema and the utility functions to load and parse the variables.

// lib/vars.ts
import { z } from 'zod';
import { load, parseOrThrow } from './env';
import { parseOrThrow, load, str, num, bool, port } from './env';

// Define the variables schema
const schema = z.object({
  PUBLIC_VARIABLE: str.optional(),
  PUBLIC_MY_NUM: num,
  PUBLIC_BOOL: bool,
  PUBLIC_PORT: port,
});

export const loadEnv = () => {
  const env = load();
  const parsed = parseOrThrow(schema, env, new Error('Invalid variables'));
  for (const key of Object.keys(parsed)) {
    if (!globalThis[key]) {
      globalThis[key] = parsed[key];
    }
  }
};

Using The Variables

In order to use the variables, we need to load them first. We can do that in the app/layout.tsx file or any other layout file to gradually expose them to the rest of the application, both on the client and the server.

// app/layout.tsx
import { loadEnv } from '@/lib/vars';

loadEnv();

export default function RootLayout({
  children,
}: {
  children: React.ReactNode;
}) {
  return (
    <html lang="en">
      <body>{children}</body>
    </html>
  );
}

References

Tauri vs. Electron: A Technical Comparison

vorillaz — Wed, 24 Jul 2024 14:18:29 +0000

For web technologies in desktop application development, two of the most promising entrants to the game are Tauri and Electron. It is worthy to note that each of the projects aims to provide tools for building cross-platform desktop applications with the use of web technologies, but at the same time, they do so in ways that are quite different from one another. This post will try to elaborate on a technical comparison between Tauri and Electron in terms of architecture, performance, security, development experience, and community support.

1. Architecture

Electron Architecture:

Core components: Electron is a combination of Chromium, an open-source web rendering engine, and Node.js, an open-source JavaScript runtime. It produces a runtime that has all the features to develop desktop GUIs via web technologies.

Process model: An Electron app runs a single Node.js process named the main process. This process manages life-cycle events in the application and can spawn multiple renderer processes (one per application window) to handle the GUI.
Bundling: Because Electron bundles the entirety of Chromium and Node.js, applications tend to be relatively large in size.

Tauri Architecture:

Core Components: Tauri is quite a lightweight framework; it uses just a webview component of the OS (uses WebView2 on Windows, WKWebView on macOS, WebKitGTK on Linux) for rendering the UI and utilizes Rust for the backend logic.

Process Model: Tauri splits apart the frontend (running inside the webview) from the backend (written in Rust), improving security and performance.

Bundling: The native webview components that Tauri applications use mean they bundle much smaller compared to applications bundling a full browser engine.

2. Performance

Electron:

Memory Usage: Electron applications tend to consume more memory because of the bundled Chromium engine and the overhead of running multiple processes.

Startup Time: Startup time for an Electron app tends to be slow because it first has to start up the entire Chromium engine.

Runtime Performance: Electron applications sometimes run less efficiently because of the sometimes resource-heavy Chromium engine. This is the case, especially for apps with many UI components or complex rendering needs.

Tauri:

Memory Usage: Tauri applications generally consume memory in smaller quantities since it is powered by the native webview and lightweight Rust backend.

Startup Time: In general, Tauri applications will start up quicker thanks to a smaller application size and use of native webview components.

Runtime Performance: Tauri will thus be able to take advantage of Rust's performance and features with respect to safety—putting it in good stead as a powerhouse in its own right for use in resource-constrained environments and applications demanding high performance.

3. Security

Electron:

Sandboxing: While Electron renderer processes are sandboxed, the main process has complete access to the system and introduces some security concerns if not handled properly.

Vulnerability Surface: The full Chromium engine adds to the attack surface. This makes it imperative to frequently update and upgrade against newly found vulnerabilities.

IPC (Inter-Process Communication): The Inter-process communication model has to be mindfully implemented to enable the main and renderer processes to communicate securely without any security glitches.

Tauri:

Sandboxing: The architecture of Tauri sandbox's away the webview/UI from the backend/logic. This inherently decreases risk concerning security bugs.

Vulnerability Surface: The vulnerability surface is lower as it uses native webview components in the system when compared to a whole browser engine that is built and bundled in.

IPC: Tauri has a more secure Inter-Process Communication mechanism, thanks to Rust's strong typing and memory safety guarantees that lessen the risks of security issues.

4. Development Experience

Electron:

Ecosystem: Electron has a mature ecosystem with extensive documentation, countless plugins, and a lots of community-developed modules.

Tooling: Shared tools and libraries for web development make the job easier for developers.

Debugging: Tools like Chrome DevTools come bundled with Electron, and it's pretty easy to debug.

Tauri:

Ecosystem: Tauri is an ecosystem, growing every day around it; the support and documentation are increasing day by day as well, but it is still in its early stages. We can compare it with an earlier version of the Electron ecosystem.

Tooling: Tauri has nice integration with today's modern web dev tools, and its Rust backend can be managed using Rust's powerful tooling.

Debugging: Debugging techniques include both webview debugging tools and Rust debugging tools that are richer in nature but can sometimes be more complicated to set up.

5. Community Support

Electron:

Adoption: Electron has high adoption as compared to others with many high-profile apps (Slack, Visual Studio Code, Discord).

Community: The large and active community around Electron makes for a robust support network with frequent updates and a host of third-party resources.

Tauri:

Adoption: Tauri is newer but in the process of very rapid adoption because of its light nature and the security benefits it brings.

Community: The size of the community of Tauri is growing with the amount of new contributions; indeed, the project is more than active, but surely not as big as the Electron one.

Conclusion:

Choose between Tauri and Electron depending on the necessities and limitations of your project:

Electron will be suitable for developers who target a mature ecosystem, common web development practices, using JavaScript alround and vast community support. The mentioned advantages, however, result in high memory consumption and app sizes.

Tauri — use when you will care about performance, security, and small application sizes by taking full advantage of Rust and native webview components. The ecosystem is moving fast, but it might require a greater learning curve if the developers are new to Rust.
Both frameworks can provide a good cross-platform desktop application, and the best choice will depend on your project's priorities and the expertise of the development team.

Next.js with Public Environment Variables in Docker

vorillaz — Mon, 01 Apr 2024 13:16:12 +0000

In my previous guide on containerizing Next.js apps, we encountered a notable challenge. Next.js offers two ways to handle environment variables:

Variables accessible in the browser, prefixed with NEXT_PUBLIC_, are bundled during the build process, inlined and included in the client-side code.
Variables without this prefix, such as MYVAR=1, are only accessible to server-rendered components and API routes.

This distinction complicates the creation of a Docker image for Next.js apps that require dynamic, browser-accessible environment variables. When using an .env file during the Docker build, the variables are statically included in the image, losing their dynamic nature. This is a limitation coming up from the deployment models of platforms like Vercel, which build and deploy code on-the-fly, unlike the pre-packaged approach required for Docker.

The Quick and Dirty Solution.

Given a .env.local file with several variables, we can expose these through an API route. This method bypasses the need for the NEXT_PUBLIC_ prefix convention.

# .env.local
DOCKER_MY_ENV=hello
DOCKER_ANOTHER_ENV=world

We can then create an API route to serve these values:

// app/env/route.ts
// Disable caching
export const dynamic = "force-dynamic";
export async function GET(request: Request) {
  const env = {
    DOCKER_MY_ENV: process.env.DOCKER_MY_ENV,
    DOCKER_ANOTHER_ENV: process.env.DOCKER_ANOTHER_ENV,
  };
  return Response.json(env);
}

Client components can fetch and display these values as follows:

// components/fetch-env.tsx
"use client";
import { useState, useEffect } from "react";

export const MyFetchComponent = () => {
  const [env, setEnv] = useState({});

  useEffect(() => {
    fetch("/env")
      .then((res) => res.json())
      .then((data) => setEnv(data));
  }, []);

  return (
    <div>
      <pre>{JSON.stringify(env, null, 2)}</pre>
    </div>
  );
};

A More Consistent Approach.

While the above solution is functional, it includes an additional network request and it's not that versatile. A more seamless integrations involves passing variables from server components to client React components through a context provider.

First, define and export the environment variables using a helper function:

// env/env.ts
"use server";
export const getEnv = async () => ({
  DOCKER_MY_ENV: process.env.DOCKER_MY_ENV,
  DOCKER_ANOTHER_ENV: process.env.DOCKER_ANOTHER_ENV,
  DOCKER_MY_INT: process.env.DOCKER_MY_INT,
  DOCKER_MY_URL: process.env.DOCKER_MY_URL,
  NEXT_PUBLIC_ANALYTICS_ID: process.env.NEXT_PUBLIC_ANALYTICS_ID,
});

The "use server" directive will mark the call as a server-side function, in order to consume the helper in a Next.js layout the call should be asynchronous.

Next, create a context provider component:

// env/provider.tsx
"use client";
import { createContext, useContext, useEffect, useState } from "react";
import { getEnv } from "./env";

export const EnvContext = createContext({});

export const EnvProvider = ({
  children,
}: Readonly<{
  children: React.ReactNode;
}>) => {
  const [env, setEnv] = useState({});

  useEffect(() => {
    getEnv().then((env) => {
      setEnv(env);
    });
  }, []);
  return <EnvContext.Provider value={env}>{children}</EnvContext.Provider>;
};

export const useEnv = () => {
  return useContext(EnvContext);
};

The "use client" directive is mandatory in order to keep track of any state changes. using the useState hook. Upon mounting the provider will retrieve the values from the getEnv() method and initialise the context.

Finally wrap your page contents with the <EnvProvider /> component:

// app/layout.tsx
import { EnvProvider } from "@/env/provider";

export default async function RootLayout({
  children,
}: Readonly<{
  children: React.ReactNode;
}>) {
  return (
    <html lang="en">
      <body>
        <EnvProvider>{children}</EnvProvider>
      </body>
    </html>
  );
}

Client components can now access the environment variables:

// components/example.tsx
"use client";
import { useEnv } from "@/env/provider";

export const MyClientComponent = () => {
  const env = useEnv();

  return (
    <div>
      <h1>This is rendered in a client component.</h1>
      <pre
        style={{
          padding: "1rem",
          backgroundColor: "#f4f4f4",
          border: "1px solid #ccc",
          borderRadius: "5px",
          margin: "1rem 0",
        }}
      >
        {JSON.stringify(env, null, 2)}
      </pre>
    </div>
  );
};

Building and Running the Docker Image.

Modify the Dockerfile from the previous tutorial by removing the environment variable bundling. Here's how to build and run the image:

FROM node:20-alpine AS base

# --- Dependencies ---
### Rebuild deps only when needed ###
FROM base AS deps
RUN apk add --no-cache libc6-compat git

RUN echo Building nextjs image with corepack

# Setup pnpm environment
ENV PNPM_HOME="/pnpm"
ENV PATH="$PNPM_HOME:$PATH"
RUN corepack enable
RUN corepack prepare pnpm@latest --activate

WORKDIR /app

COPY package.json pnpm-lock.yaml ./
RUN pnpm install --frozen-lockfile --prefer-frozen-lockfile

# --- Builder ---
FROM base AS builder
RUN corepack enable
RUN corepack prepare pnpm@latest --activate

WORKDIR /app

COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN pnpm build

# --- Production runner ---
FROM base AS runner
# Set NODE_ENV to production
ENV NODE_ENV production

# Disable Next.js telemetry
# Learn more here: https://nextjs.org/telemetry
ENV NEXT_TELEMETRY_DISABLED 1

# Set correct permissions for nextjs user
# Don't run as root
RUN addgroup nodejs
RUN adduser -SDH nextjs
RUN mkdir .next
RUN chown nextjs:nodejs .next

# Automatically leverage output traces to reduce image size
# https://nextjs.org/docs/advanced-features/output-file-tracing
COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./
COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static
COPY --from=builder --chown=nextjs:nodejs /app/public ./public

USER nextjs

# Expose ports (for orchestrators and dynamic reverse proxies)
EXPOSE 3000
ENV PORT 3000
ENV HOSTNAME "0.0.0.0"

HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 CMD [ "wget", "-q0", "http://localhost:3000/health" ]

# Run the nextjs app
CMD ["node", "server.js"]

Here's how to build and run the image:

~ echo Building image
~ docker build --tag my-app-docker-env .

Run the application with a local .env file:

docker run -p 3000:3000 --env-file .env.local my-app-docker-env

 ▲ Next.js 14.1.4
   - Local:        http://localhost:3000
   - Network:      http://0.0.0.0:3000

 ✓ Ready in 34ms

Test by stopping the container, modifying .env.local, and restarting the application.

Enhancing Security.

To further secure your environment variables, consider using envsafe for validation and type safety. Install the package and adjust the env/env.ts file accordingly along with the variable types:

pnpm add envsafe

// env/env.ts
"use server";
import { envsafe, str } from "envsafe";

export const getEnv = async () =>
  envsafe({
    DOCKER_MY_ENV: str(),
    DOCKER_ANOTHER_ENV: str(),
    DOCKER_MY_INT: str(),
    DOCKER_MY_URL: str(),
  });

Wrap up.

In conclusion, managing public environment variables in a Dockerized Next.js application requires a little effort to keep up with their dynamic nature. We explored two primary methods: serving variables through a Next.js API route and a more seamless integration using a context provider to pass variables from server components to client components. The latter approach, while slightly more complex, offers a more consistent and versatile solution. Finally, we covered the necessary changes to the Dockerfile to accommodate both strategies and highlighted the importance of security and type safety with tools like envsafe.

References

How to Build a Views Counter with Astro DB

vorillaz — Tue, 26 Mar 2024 15:17:20 +0000

This is cross-post from my blog: https://www.vorillaz.com/astro-views

In my early days of browsing, I can remember one of the most iconic widgets on the web: the views counter at the bottom of a blog post. It was a clear way to showcase the popularity of a post, and it was always fun to see how many people had read it. Platforms like Geocities and MySpace had this functionality long before the birth of Google Analytics or any basic analytics, for that matter.

In early March 2024, Astro DB was launched, introducing a new database service built atop Turso, LibSql, and Astro. It's a simple, fast, and reliable database service, perfectly suited for content-related applications. This inspired me to create a views counter for my blog using Astro DB since I am already using Astro.

Getting Started

First, you need to have an Astro Studio account. You can sign up for a free account at studio.astro.build. Once you have an account, you can create a new database and obtain your API key.

Upgrade your Astro project to the latest version (later than 4.5.0) and install the @astrojs/db package. This can be done through the Astro CLI helper, as mentioned in the Astro DB documentation.

npx astro add db

Define Your Database Schema

After setting up, you can create your database and define your schema. Astro DB schemas are configured through a single TypeScript file. Define your schema in a file named db/config.ts:

// db/config.ts
import { defineDb } from 'astro:db';

export default defineDb({
  tables: { },
});

Now, it's time to define your Views schema. For this example, we will create a table named Views with a count column and a slug column. The slug column will identify the post associated with the view count.

// db/config.ts
import { defineDb, defineTable, column } from "astro:db";

const Views = defineTable({
  columns: {
    slug: column.text({ primaryKey: true }),
    count: column.number({
      default: 1,
    }),
  },
});

export default defineDb({
  tables: { Views },
});

Creating the ViewCounter Component.

With our database set up, we can now create a view counter for our blog. We'll develop an Astro component to display and increment the view count when the post is viewed.

---
// components/ViewCounter.astro
import { db, Views, sql } from "astro:db";
const { slug } = Astro.props;
let item;

try {
  item = await db
    .insert(Views)
    .values({
      slug,
      count: 1,
    })
    .onConflictDoUpdate({
      target: Views.slug,
      set: {
        count: sql`count + 1`,
      },
    })
    .returning({
      slug: Views.slug,
      count: Views.count,
    });
} catch (error) {
  console.error(error);
  item = { slug, count: 1 };
}
---

<div>
  <p>Views: {item.count}</p>
</div>

The <ViewCounter /> component utilizes the db object to interact with the database. The Astro DB package includes Drizzle ORM, which provides a typesafe API for database transactions. In the <ViewCounter /> component, we inject the slug prop to identify the associated post. We then attempt to insert a new row into the Views table with the slug and a count of 1. If the row already exists, we increment the count by 1 using the onConflictDoUpdate method and return the updated row, this is how upserts are handled by Drizzle.

Using the ViewCounter Component.

With our ViewCounter component ready, we can add it into our blog post template. We use the slug from the URL as a key to identify the associated post.

---
// pages/blog/[slug].astro
import ViewCounter from '@/components/ViewCounter.astro';
const slug = Astro.params.slug;
---

<article>
  <h1>{slug}</h1>
  <ViewCounter slug={slug} />
  <p>Post content goes here</p>
</article>

Creating an Article Listing Component.

For post listing we can also create a small utility component for showing the views per entry in the list. This time instead of incrementing the view count we will just read the count from the database.

---
// components/Views.astro
import { db, Views, eq } from "astro:db";
interface Props {
  slug: string;
}

const { slug } = Astro.props;
const found = await db
    .select({
      count: Views.count,
    })
    .from(Views)
    .where(eq(Views.slug, slug ));
const count = found[0]?.count || 0;
---

<span>Views: {count}</span>

And then we can use this component in our listing page.

---
// pages/blog.astro
import { getCollection } from 'astro:content';
import Views from '@/components/Views.astro';

// Get all `src/content/blog/` entries
const allBlogPosts = await getCollection('blog');

const posts = await Astro
---
<ul>
  {posts.map((post) => (
    <li>
      <h2><a href={`/blog/${post.slug}`}>{post.title}</a></h2>
      <Views slug={post.slug} />
    </li>
  ))}
</ul>

Client Side Rendering & API Routes.

Since Astro components do not render in the client and our blog is mostly static, we need to use an API route to increment the view count in order to avoid server-side rendering delays. We can easily create an API route to handle the view count increment and return the updated count for the specified slug.

// api/views.ts
import type { APIRoute } from "astro";
import { db, Views, sql, eq } from "astro:db";

export const GET: APIRoute = async ({ request }) => {
  const url = new URL(request.url);
  const params = new URLSearchParams(url.search);

  const slug = params.get("slug");

  if (!slug) {
    return new Response("Not found", { status: 404 });
  }

  let item;
  try {
    const views = await db
      .select({
        count: Views.count,
      })
      .from(Views)
      .where(eq(Views.slug, slug));

    item = await db
      .insert(Views)
      .values({
        slug: slug,
        count: 1,
      })
      .onConflictDoUpdate({
        target: Views.slug,
        set: {
          count: sql`count + 1`,
        },
      })
      .returning({
        slug: Views.slug,
        count: Views.count,
      })
      .then((res) => res[0]);
  } catch (error) {
    item = { slug, count: 1 };
  }

  return new Response(JSON.stringify(item), {
    status: 200,
    headers: {
      "content-type": "application/json",
      "cache-control": "public, s-maxage=60, stale-while-revalidate=25",
    },
  });
};

The API route increments the view count for the specified slug and returns the updated count in a similar way to the ViewCounter component created earlier. We can now create a client-side component to fetch the view count from the API route using the fetch API and a framework integration like Svelte.

We can set up Astro along with Svelte to fetch the view count from the API route using the official Svelte integration.

Once set up, we can create a new Svelte component to fetch the view count from the API route.

<!-- src/components/ViewCounter.svelte -->
<script>
  export let slug;
  const fetchImage = (async () => {
    const response = await fetch("/api/views?" + new URLSearchParams({ slug }));
    return await response.json();
  })();
</script>

{#await fetchImage then data}
  <span>{data.count}</span>
{:catch}
  <span>1</span>
{/await}

Finally, you can use the ViewCounter component in your Astro template.

---
// pages/blog/[slug].astro
import ViewCounter from '@/components/ViewCounter.svelte';
const slug = Astro.params.slug;
---

<article>
  <h1>{slug}</h1>
  <ViewCounter client:only slug={slug} />
  <p>Post content goes here</p>
</article>

The client:only directive ensures that the component is only rendered on the client side, allowing the view count to be updated without delays in server-side rendering.

Wrap Up.

As you can see, it's very easy to build data driven components with Astro DB. The ViewCounter component is a simple example of how you can use Astro DB to build more interactive and dynamic components without relying on a third-party service like third part CMS solutions.
You can use the same approach to build other data-driven components, such as comments, likes, or social interaction counters widgets and more.

Scale from One to Millions of Users

vorillaz — Tue, 30 Jan 2024 19:24:12 +0000

This is cross-post from my blog: https://www.vorillaz.com/scale/.

When it comes to hot topics, there is always a huge question that strikes controversy among the development community: Does the X/Y/Z framework scale properly?

In a world where how we develop web applications changes rapidly, the way we are also treating scalability has also changed along. New frameworks have emerged in the JavaScript community trying to simplify the process of publishing a web application, but they are also coming along with compromises, tons of assumptions, and they might not always be a good fit for every single use case.

In this detailed series of articles, I will try to provide a generic guide for quantifying a web application from one up to a few million users in a linear and progressive way, in a way that is affordable and actually makes sense. We are going to document each step, use open-source technologies, and create a plan for scaling without driving you bankrupt.

Choosing the best framework for the Job

For the purpose of this series, we are going to build a Next.js application. Next.js has been a widely adopted framework, built and maintained by Vercel. The reason for choosing Next.js is its popularity, along with the vision that we can actually decouple the deployment and scaling process without relying on a "major" deployment provider, still providing the same fundamentals.

While this guide offers generic guidance, you can use also Remix, Nuxt, Sveltekit, Blitz, or any other framework of your choice. The real advantage of picking Next.js or any other mentioned framework lies in its versatility.

It allows the creation of web applications with SSR (Server Side Rendering) along with API routes, making it a genuinely good candidate to demonstrate how we can quantify an application for both the frontend and backend. Furthermore, since Vercel provides A-class support for building, deploying, monitoring and effectively scaling, it's also a great case study on how we can build applications without solely relying on "the cloud", or at least "the cloud"m provided by a single vendor.

Leaving the Cloud?

In 2023, 37Signals began documenting their journey of leaving the cloud. The idea was pretty simple yet bold — they moved their main products away from AWS, deciding to purchase and deploy their own infrastructure on-premises, utilizing dedicated servers for deployment, monitoring and scaling on demand.
Of course, this approach may not be viable for everyone, as it demands a good investment in both money and resources that may not be available for you or your team since 37signals is a well established company with a some millions of dollars in revenue.

The economics might not be a good fit at first, especially for a project that is in its early days, considering that most major hosting vendors offer generous free plans that can keep you up and running for some time. Still, we can provide a few strong arguments for not choosing a major "provider" like Vercel, GCP, Azure or AWS and instead move along with a vendor that it offers more "generic" solutions, call me virtual or dedicated servers. And the benefits can be quite obvious.

It's Cheaper.

Vercel, fly.io, Render, or Netlify might not be good candidates for one-to-one comparison as they are services built around AWS, GCP, Azure, or Cloudflare. They function as "middlemen" companies, providing a better developer experience (DX) using resources with "better deals" on infrastructure than what you can get directly from the major cloud providers.

Vercel, for example, heavily invests in their scalability around the serverless ecosystem from AWS. Although the comparison might not be ideal, it's definitely cheaper to use a cloud provider like Hetzner, Scaleway, or DigitalOcean (and many others) providing a dedicated server or a virtual machine.

As of the time of writing this series (January 2024), for a naive comparison, the cost of an EC2 server in AWS (t3a.large) is almost $630 per month, while on Hetzner the a VM with the same specs, it's just $15.48. Considering there are also offers and discounts running throughout the year, you and your team can save even more money. Furthermore, providing a portable and versatile solution for deploying your web applications can be a lifesaver if the hosting providers of your choice decide to start price gouging.

There might be hidden costs that you might not even consider when talking about side projects. Due to limitations of platforms like Vercel, simple DevOps operations like hosting a MySQL database, running a few cron jobs, or using Redis can actually accumulate tons of hidden costs.

Also, the "infinite scalability" mantra that these platforms advertise might also lead to infinite costs. There are horrific stories out there from people who got billed with huge amounts of money from major vendors when they introduced a long-standing job or function. Managing your own deployments can be far more predictable and progressive, and that's also the main purpose of creating sustainable businesses on the web. Assuming that your traffic or users are in perfect analogy with your income, you can actually invest more money in your infrastructure needs as your business grows.

It's Versatile.

Since we have talked about the economics around bringing your own cloud, there is also a counterargument that comes up most of the time: Using AWS/GCP/you-name-it is a rigid and battle-tested platform. That might be true in most cases; still, you have to build your web applications around the ecosystem of the provider, which can be quite restraining. Even when we are talking about Vercel, Netlify, or Cloudflare, you have to shape your code, your architecture, and even your deployment decisions around the caveats, the compromises and the ecosystem provided by your provider.

On the other hand, a solid deployment solution using an industry standards, like deploying a Docker image, is so versatile and, at some point, a no-brainer solution. In the long run, if for some reason you are not satisfied with your provider, moving along to another provider requires minimal effort. You can even bring up a "multicloud" architecture where you use different cloud providers, actually reducing the cost more and increasing the availability of your system.

It's Privacy First.

Privacy might not be the first priority for most companies or solo developer, but it can also be quite a deal-breaker. When we started building Proxima, our business model required us to avoid overseas data transfers for our customers and all the data they were collecting. Thus, we chose to use infrastructure from companies not only physically located in EU soil but also owned by European companies. For that reason, we chose to move along with Bunny, Scaleway, and Hetzner.

Is It Secure Enough, Though?

Security is a main concern when it comes to deploying web applications. Considering that most of the major providers overpromote their security policies and safeguards, it might sound pretty unsafe rolling your own deployments. Still, we can counter-argue that most hosting providers are certified themselves with well-established web and physical security standards like ISO/IEC 27001. Furthermore, extra security services like the DDoS mitigation service from AWS cost extra and they might add up to your deployment budget.

You're Helping the Web Move Forward

Once again, this might not be a good argument for most people reading this article and might be a hot take of mine, but monopolies are not moving the industry forward. When an open-source project like Next.js is built around a deployment platform as a service, most of the work invested is actually trying to make money. Web frameworks, especially the most popular ones, should get shaped and move forward by the community. Once again, all the big companies are hosting and heavily investing in the big 4 cloud computing platforms (Azure, AWS, GCP, and Cloudflare).

As a logical assumption, most of these platforms are shaping their strategies around the needs and the investment of their bigger customers, and most of the time, your infrastructure needs won't be the same as the ones that a Fortune 500 company has. That's why these platforms tend to be so bloated for the common user and that's also why probably if someone has to deploy a service to AWS, it's almost certain that they might need to hire "AWS expert." Complexity and corporate bureaucracy have created a whole new industry.

Finally, we could also argue that since even for a small project, the bill can get up to a few thousand dollars per month, an equal distribution to the smaller hosting companies can push forward better services and a more flexible and sustainable ecosystem.

You Can Get as Complex as You Want

Developers tend to be attracted to complexity like moths to flames. It requires lots of effort to keep a system, and especially a deployment infrastructure with lots of moving parts, lean, well-documented, and established in a way that makes sense, especially as it evolves over time. In a nutshell, you don't have to overcomplicate your infrastructure setup from day one. As you get more users, you might need to reconsider your needs and your deployment and scalability strategy.

On the other hand, it's hardly unlikely to build an application that's super trivial to deploy; the "Hello world" applications are for demo purposes only. Using an architecture that progressively grows with your user base and effectively with your revenue stream can actually grow in a linear and projectable fashion.

Ok, What's the Catch?

There isn't any, actually. I am not trying to sell you anything, nor promoting one hosting provider or a framework over another. While I think a self-deployment platform can be quite an investment, especially for small teams or products that are starting now, it might not be a viable solution for everyone.

It can get quite tricky to work along with systems, infrastructure, and networking. It requires lots of reading, experimenting, asking hard questions, spending money, and time.

So, this plan might not be the perfect fit, especially if you're lacking time. Some teams can't afford spending too many resources on DevOps and system management, and that's totally acceptable. Use whatever tool, methodology, and act with whatever makes sense for you at the time being; just make sure you're aware of all the options available in the wild.

Moving Along

In this first part of our series, I've tried to sum up the good and not-so-good aspects of setting up your own web applications without sacrificing quality and staying within budget.

In the upcoming episodes, we'll break down the costs of our setup, find the best tools for each job, and kick off our deployment plan based on our userbase.

How to Automatically Consume RESTful APIs in Your Frontend

vorillaz — Thu, 25 Jan 2024 18:31:00 +0000

This is cross-post from my blog: https://www.vorillaz.com/consume-apis/.

The Problem.

One of the tasks that I've worked on extensively in the past is designing, developing, deploying, and finally consuming numerous RESTful APIs, and I can tell you, this is one of the hardest tasks a web developer has to deal with.

Well, REST is Painful.

I wouldn't complain, argue, or compare the pros and cons of RESTful APIs in this article, but I can tell you that a REST API can be really painful. We have to tackle a lot of tricky points like:

API changes.

When something changes in the backend, the frontend always has to keep everything in sync. A simple change in the API call site or the returned payload can cause the frontend to break, and unit testing it is almost impossible to catch any bugs associated with the contract between the two.

app.post(
  "/create",
  {
    schema: {
      body: {
        type: "object",
        properties: {
          name: { type: "string" },
          // This parameter addition will break the frontend.
          surname: { type: "string" },
        },
        required: ["name", "surname"],
      },
    },
  },
  (request, reply) => {
    reply.send({ data }); 
  }
);

Documentation.

Offering detailed documentation for each endpoint in our APIs is a common and always a good practice, although this task can be quite tedious and time-consuming to maintain. We have to keep the documentation in sync with the actual code changes. Way too often, the documentation is outdated, and it doesn't reflect the actual API contract, leading to confusion and bugs.

Type Safety

When consuming APIs in the frontend, we have to be extremely careful about the data types we are receiving and sending over the wire. Type safety is imperative in the frontend, both for calling the API and for rendering the data.

API Consumption

RESTful APIs often involve a significant amount of boilerplate code. We must address tasks such as calling the API, handling responses, managing errors, and more. Even with the help of libraries like axios, SWR, and TanStack Query, we still require a custom abstraction layer to simplify client's calls. However, as the API grows, the abstraction layer will become more complex and challenging to maintain.

// Using SWR to fetch the data for each endpoint
import useSWR from 'swr'

const fetcher = (url: string) => fetch(url).then((res) => res.json());
// Using SWR to fetch the data for users.
export const useUsers = () => {
  const { data, error } = useSWR("/api/users", fetcher);
  return {
    users: data,
    isLoading: !error && !data,
    isError: error,
  };
};
// More API calls...
export const useArticles = () => {};
export const useProjects = () => {};

GraphQL or tRPC to the Rescue?

Both GraphQL and tRPC offer excellent alternatives to RESTful APIs, providing lots of benefits and effectively addressing many of the challenges associated with "traditional" REST. However, what if we can't "technically" afford using them?

Both of these solutions require learning a new syntax and a new way of thinking, effectively adding complexity to our stack.
Additionally, particularly with tRPC, it is tightly integrated with the Node.js ecosystem and is not available for other languages like Python, Ruby, or Go. This limitation can be problematic, especially when a Node.js backend API isn't always a viable option.

So, given these constraints, what alternatives do we have?

Swagger and OpenAPI.

In order to automatically connect our frontend code with our backend APIs, we need to establish a common language between the two. Our backend APIs must be documented in a way that our frontend can understand and consume, and this is where Swagger and OpenAPI come into play.

Swagger is an open-source software framework that implements the OpenAPI Specification—an API description format for REST APIs. The OpenAPI Specification defines a standard, language-agnostic interface to HTTP APIs, enabling both humans and computers to discover and understand the capabilities of the API.

Using the specification, we can describe our API's parameters, responses, errors, and document evertyhing, all in one place, all through our codebase.

Building the API.

In order to demonstrate how to automatically generate a client SDK in the frontend, we will develop a dead simple API. We will use fastify, but you can choose any framework you prefer. Swagger and OpenAPI are baked by a really active community, and there are numerous integrations available for most popular frameworks and languages.

We can start by installing the required dependencies for our backend.

❏ ~ npm i fastify fastify-swagger @fastify/swagger-ui

Then we are going to create a simple fastify instance, attach the fastify-swagger and fastify-swagger-ui modules as plugins, and configure the OpenAPI specification metadata.

import fastify from "fastify";
import ui from "@fastify/swagger-ui";

const start = async () => {
  const server = fastify();
  await server.register(swagger, {
    openapi: {
      info: {
        title: 'Test OpenAPI',
        description: 'How cool is that?',
        version: '0.1.0',
      },
      servers: [
        {
          url: 'http://localhost:3000',
          description: 'Development server',
        },
      ],
    },
  });
  await server.register(ui, {});

  server.listen({ port: 3001 }, (err, address) => {
    onsole.log(`Server listening at ${address}`);
  });
};

start();

Now, we can start our server and visit the Swagger UI at http://localhost:port/documentation. The documentation will be empty for now, but we will add our endpoints later on. You can also have a look at how the Swagger UI looks in the Swagger's online demo.

Finally, upon starting the server in development mode, we will export the OpenAPI specification in JSON format, save it in a local file called openapi.json and we are almost done with our backend setup.

if (process.env.NODE_ENV === "development") {
  const spec = "./openapi/openapi.json";
  const specFile = path.join(__dirname, spec);
  server.ready(() => {
    const apiSpec = JSON.stringify(server.swagger() || {}, null, 2);

    writeFile(specFile, apiSpec).then(() => {
      console.log(`Swagger specification file write to ${spec}`);
    });
  });
}

This openapi.json specification file can be safely sourced and committed to our repository, and it will be used by our frontend to autodiscover the API and consume it.

In the meantime, we are going to expand our backend with two endpoints: one for fetching data and another one for creating data.
Fastify provides out-of-the-box support for API serialization and validation through its schema-based approach built on top of JSON Schema.
Through the schema option, we can attach a schema definition to each route.

const retrieveSchema = {
  response: {
    200: {
      type: "object",
      properties: {
        data: {
          type: "array",
          items: {
            type: "object",
            properties: {
              id: { type: "number" },
              name: { type: "string" },
            },
          },
        },
      },
    },
  },
};

const createSchema = {
  body: {
    type: "object",
    properties: {
      name: { type: "string" },
    },
    required: ["name"],
  },
  response: {
    200: {
      type: "object",
      properties: {
        data: {
          type: "object",
          properties: {
            id: { type: "number" },
            name: { type: "string" },
          },
        },
      },
    },
  },
};

const data = [
  { id: 1, name: "John" },
  { id: 2, name: "Jane" },
  { id: 3, name: "Jack" },
];

app.post("/users", { schema: createSchema }, (request, reply) => {
  reply.send({ users: data });
});

app.get("/users", { schema: retrieveSchema }, (request, reply) => {
  reply.send({ users: data });
});

The code above, defines two users endpoints, one for creating a user and another one for retrieving all the available users in our system.
The retrieveSchema definition will validate the response and the createSchema will serialize and validate the request and the response. We can also add an operationId and additional tags to our endpoints, making them more descriptive and reference them later on when building the API client.

const retrieveSchema = {
  operationId: "retrieveUsers",
  tags: ["users"],
  // ...
};

const createSchema = {
  operationId: "createUser",
  tags: ["users"],
  // ...
};

Finally, as we're shaping our API, we can see that the output at topenapi/openapi.json is also changing, reflecting the updates we make to the endpoints' schema definitions. We can also add additional information to each route, such as a description, a summary, or example values, according to the specification.

Consuming the API.

Now, it's time to consume our API. We'll use React for this tutorail, but feel free to use any other framework you prefer; the process remains the same. Additionally, we'll utilize SWR to fetch data from the API and TypeScript to ensure type safety.

Generating the API Client.

In order to generate the API client, there are a few options available, but we are going to use (Orval)[https://orval.dev]. Orval is a CLI tool that generates API clients based on an OpenAPI specification. It supports TypeScript, JavaScript, Axios, React, Vue, Angular and Svelte and it's highly customizable.

As an alternative, you can also use the official OpenAPI Generator, which is a more generic tool supporting a wide range of languages and frameworks.

Installing and Using `orval`.

In our frontend codebase, we will install orval using the following command:

❏ ~ npm i -D orval

Up next, we will create a configuration file called orval.config.js in the root of our project and add the following configuration:

const path = require("path");

const input = path.join(__dirname, "server", "openapi", "openapi.json");
const output = path.resolve(__dirname, "client", "src", "sdk");

module.exports = {
  sdk: {
    output: {
      clean: true,
      prettier: true,
      mode: "tags-split",
      target: path.join(output, "api"),
      schemas: path.join(output, "schemas"),
      client: "swr",
      // ... more opts
    },
    input: {
      target: input,
    },
  },
};

Orval supports splitting the generated code into multiple files, organized by the tags we have defined earlier in our schema definitions. Additionally, we will need to configure the input and output paths for the generated API client, as well as the framework we want to use. In our case, we will use the SWR client for React.

Then, we will add a script in our package.json file to generate the API client and we are ready to go.

{
  "scripts": {
    "generate:api": "orval"
  }
}

Running npm run generate:api from the command line will generate the client SDK in the client/src/sdk folder.

Working with the SDK.

The autogenerated API client will contain all the endpoints we have declared in our OpenAPI specification, the createUser and retrieveUsers endpoints in our case, as they were defined through the createSchema and retrieveSchema definitions.

Orval wraps the API calls in a custom hook that we can then import and use to fetch the data. The hook returns the users, the isLoading and isError flags, and it also provides type safety for the data we are receiving over the network.

// Importing the generated API client
import { useUsers } from "sdk/api/users";

export const Users = () => {
  const { users, isLoading, isError } = useUsers();

  if (isLoading) {
    return <div>Loading...</div>;
  }

  if (isError) {
    return <div>Error...</div>;
  }

  return (
    <ul>
      {users?.map((user) => (
        <li key={user.id}>{user.name}</li>
      ))}
    </ul>
  );
};

Up next, we will also import the createUser function which calls the POST /users endpoint, once again the name of the function is based on the operationId we have defined in the createSchema definition.
Keep in mind that the passed parameters are type safe and the function will throw an error if we pass an invalid payload.

import { useUsers, createUser } from "sdk/api/users";

export const Users = () => {
  const { users, isLoading, isError } = useUsers();
  const handleCreateUser = () => {
    createUser({ name: "John" });
  };

  if (isLoading) {
    return <div>Loading...</div>;
  }

  if (isError) {
    return <div>Error...</div>;
  }

  return (
    <div>
      <button onClick={handleCreateUser}>Create User</button>
      <ul>
        {users?.map((user) => (
          <li key={user.id}>{user.name}</li>
        ))}
      </ul>
    </div>
  );
};

Testing the Client.

With orval, we can also integrate the API client in our unit tests. Orval provides first class support for mocking through the (Mock Service Worker)[https://mswjs.io/] library, and it can automatically generate the MSW handlers for testing server.

Setting up the handlers is fairly simple, we just need to import the getUsersMSW autogenerated function from the API client and pass it to the setupServer function from MSW.

import { setupServer } from 'msw/node';
import { getUsersMSW } from 'sdk/api/users/users.msw';
export const server = setupServer(...getUsersMSW());

Then we can import and use the mocking server instance in our tests.

import { render, screen } from '@testing-library/react';
import { server } from 'setupTests';
import { Users } from './Users';

describe('Users', () => {
  beforeAll(() => server.listen());
  afterEach(() => server.resetHandlers());
  afterAll(() => server.close());

  it('should render the users', async () => {
    render(<Users />);
    expect(await screen.findByText('John')).toBeInTheDocument();
  });
});

Isomorphic Data Validation.

Since the OpenAPI specification is language-agnostic, we can use the output to validate data in both our frontend and backend. This approach makes total sense since we can provide uniform validation rules on both sides of our stack. There are a few libraries out there that can help us with this task, such as typed-openapi or openapi-zod-client, which can take an OpenAPI definition as an input and transform it to zod schema definitons.

Automating the SDK Generation.

Our backend and frontend code might not be in the same repository, thus we need a way to automatically detect any specification changes and sync the client SDK. This can be easily achieved by using a CI/CD pipeline. We can integrate GitHub actions, triggered by any backend code commited, to produce the client SDK and commit it to our frontend repository.

Furthermore, since we can split the generated code into multiple parts based on tag filtering, we can also create different SDKs from different resources or even publicly available APIs. There is an extensive list of publicly available OpenAPI specifications on SwaggerHub, RapidAPI and APIs.guru.

Running the Client without Any Backend.

Since the OpenAPI can effectively describe our resources, we can reuse it to generate a dummy server that can be later used for development and testing purposes without bootstrapping any actual services. There some tools available that can help us with this task, such as Prism, OpenAPI Mock, OpenAPI Backend and the MSW library we have already seen.

Additionally, using the OpenAPI Generator we can spawn a fully functional backend in a few seconds from a given specification.

# Generate a php laravel server
❏ ~ npx @openapitools/openapi-generator-cli generate -i ./openapi/openapi.json -g php-laravel -o ./server

# Or a nodejs express server
❏ ~ npx @openapitools/openapi-generator-cli generate -i ./openapi.yaml -g nodejs-express-server -o ./another-server

Conclusion.

In this article, we've explored automating the consumption of RESTful APIs in our frontend using Swagger and OpenAPI. Working with a stack of widely available tools, we can significantly reduce the boilerplate code, improve code efficiency, and cut down the time spent on API development. A streamlined API development process between the frontend and backend teams can also lead to better communication and collaboration between the two, and ultimately, to better code and product quality.

How to Dockerize a Next.js application

vorillaz — Wed, 17 Jan 2024 18:31:00 +0000

This is cross-post from my blog: https://www.vorillaz.com/nextjs-docker/.

Next.js is like the Swiss Army knife for creating React applications. It makes the process smooth, offering outstanding performance and a top-notch developer experience. But, when it comes to deploying an application you've built outside of a major deployment provider, it can get a bit tricky. Managing dependencies, configuring the environment, and making sure everything behaves consistently across different platforms .

Now, here's where Docker comes into play. We're going to wrap up your Next.js application, its environment, and all its dependencies into a small little container. Think of a container as your application running in a stripped-down version of Linux. The brain and hearts of your application t's called a Docker image – the blueprint for your container. The container itself? That's your application running based on that master plan.

To make this magic happen, we use a declarative method through something called a Dockerfile. It's like giving Docker a to-do list. Docker then reads and executes the instructions in this file, building and deploying your application.

Why You Should Containerize Your Next.js App with Docker

There are several reasons why you might want to consider containerizing your Next.js application with Docker. Here's a list of advantages:

Consistent behavior across environments: Docker containers ensure your Next.js app behaves consistently across different platforms and environments.
Easy Deployment and Scaling: Docker enables straightforward deployment and scaling of your Next.js app to multiple instances. This makes it easier to handle traffic spikes and meet high availability requirements.
Portablability: Docker containers are portable and shareable. You can collaborate seamlessly with team members or deploy your app to different environments without manual configuration headaches.
Cost-Effective Infrastructure: Managing your own deployment environment can be cost-effective. Docker allows you to self-host multiple isolated Next.js applications on a single machine or move across different hosting vendors, saving you time and resources.
Streamlined Dependency Management: Bundling all your app's dependencies within a Docker container eliminates the need for manual installation and management on the host environment.

Containerizing Your Next.js App: A Step-by-Step Guide

Install Docker.

First things first, you need to ensure that you have Docker installed on your machine. Once done, run the Docker daemon and ensure that everything works as expected. You can check by running a simple command in your terminal.


❏ ~ docker info

Client:
    Version: 24.0.7
    Context: orbstack
    Debug Mode: false
Plugins:
    buildx: Docker Buildx (Docker Inc.)
    Version: v0.12.0

Edit the `next.config.js` File

In order to build your Next.js app as a container, you need to change the configuration of your project.

Next.js can automatically create a standalone folder that copies only the necessary files for a production deployment including all your node_modules.


/** @type {import('next').NextConfig} */
module.exports = {
  output: "standalone",
};

Create a `.dockerignore` file

We need to create a .dockerignore file as we don't want to increase the size of our docker image with unnecessary files tranfered to the container. You can create a .dockerignore file in the root of your project and add the following content:


.dockerignore

# Not needed as we are bundling the whole app
node_modules
# Log files
logs
*.log
npm-debug.log*
pnpm-debug.log*
yarn-debug.log*
yarn-error.log*
.next
.git

Create the Dockerfile

You can now create a Dockerfile that describes how your image gets built. We are going to use the pnpm package manager, but you can also use npm or yarn instead. The Dockerfile will be placed on the root of your project and it will look like this:

FROM node:20-alpine AS base



### Dependencies ###
FROM base AS deps
RUN apk add --no-cache libc6-compat git



# Setup pnpm environment
ENV PNPM_HOME="/pnpm"
ENV PATH="$PNPM_HOME:$PATH"
RUN corepack enable
RUN corepack prepare pnpm@latest --activate

WORKDIR /app

COPY package.json pnpm-lock.yaml ./
RUN pnpm install --frozen-lockfile --prefer-frozen-lockfile

# Builder
FROM base AS builder

RUN corepack enable
RUN corepack prepare pnpm@latest --activate

WORKDIR /app

COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN pnpm build


### Production image runner ###
FROM base AS runner

# Set NODE_ENV to production
ENV NODE_ENV production

# Disable Next.js telemetry
# Learn more here: https://nextjs.org/telemetry
ENV NEXT_TELEMETRY_DISABLED 1

# Set correct permissions for nextjs user and don't run as root
RUN addgroup nodejs
RUN adduser -SDH nextjs
RUN mkdir .next
RUN chown nextjs:nodejs .next

# Automatically leverage output traces to reduce image size
# https://nextjs.org/docs/advanced-features/output-file-tracing
COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./
COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static
COPY --from=builder --chown=nextjs:nodejs /app/public ./public

USER nextjs

# Exposed port (for orchestrators and dynamic reverse proxies)
EXPOSE 3000
ENV PORT 3000
ENV HOSTNAME "0.0.0.0"
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 CMD [ "wget", "-q0", "http://localhost:3000/health" ]

# Run the nextjs app
CMD ["node", "server.js"]

Let's break this down a bit. We will divide our building process into multiple steps, allowing us to utilize Docker's caching mechanism. Our base image is node:20-alpine, which is relatively small in size. For each step, we will instruct PNPM to cache the module installation.

On the deps step, we will bundle our dependencies. We need to move the package.json and pnpm-lock.yaml lock file into our working directory and install all the project's dependencies. Since Next.js will bundle the production dependencies, we don't need to install devDependencies and the production ones separately.

This building step will significantly speed up our building process, as it is internally cached by the Docker engine. Additionally, pnpm's symlinking strategy and internal caching will install our modules at an extremely fast pace.

Next, we have the building step. Once we switch to our working directory, we will copy our node_modules from the previous step, then we will copy the entire project and run pnpm build. This action will trigger the script defined in our package.json, after which we can proceed to the final step.

The runner step is where we will ultimately create our Docker image. We need to set up a user and a group with limited permissions, as we don't want to run our application as root within our container. Once done, we will copy the /.next/standalone, .next/static, and /public folders from our builder step. Next.js' standalone mode bundles everything into an isolated bundle, so we don't need to worry about copying any other folders. Note that there is also a .next/BUILD_ID file that you can use to differentiate your builds. We also need to set up our NODE_ENV environmental variable to production, as many npm modules heavily rely on this setting.

Finally, we will expose port 3000, as this is the default port Next.js listens to. As an added little extra, we have included a healthcheck, pinging our /health API route. This step will ensure that the container is ready before launching the web server. After that, we will run our server by executing CMD ["node", "server.js"].

Build the image and run the container

If you followed along this tutorial, you are now ready to build your Next.js application.

Head to your terminal and build your image by running:

❏ ~ docker build -t my-nextjs-docker .

Once done, you may try to rerun the building process and obtain the building logs, you will see that all the steps are internally cached and the building process just takes a few seconds.

❏ ~ docker build -t my-nextjs-docker .

[+] Building 2.6s (25/25) FINISHED docker:orbstack
=> [internal] load build definition from Dockerfile 0.0s

=> transferring context: 225B 0.0s
=> [internal] load metadata for docker.io/library/node:20-alpine 2.6s

=> CACHED [runner 1/7] RUN addgroup nodejs 0.0s
=> CACHED [runner 2/7] RUN adduser -SDH nextjs 0.0s  
=> CACHED [runner 3/7] RUN mkdir .next

You can also find out the building details about your image by using the docker images command.


❏ ~ docker images my-nextjs-docker

REPOSITORY TAG IMAGE ID CREATED SIZE
--
my-nextjs-docker latest 09462a81711b 14 hours ago 155MB

Now it's time to finally start our container, we need to bind port 3000 to our container and spin up the image.

❏ ~ docker run -p 3000:3000 my-nextjs-docker

The output in your terminal will show you that the application is running.


❏ ~ docker run -p 3000:3000 my-nextjs-docker

  ▲ Next.js 14.0.4
  - Local: http://localhost:3000
  - Network: http://0.0.0.0:3000
  ✓ Ready in 39ms

You can now head over to http://localhost:3000 and you will get prompted by the Next.js default homepage.

Envinromental Variables

Next.js offers a reliable way for managing environmental variables through .env files. You can create separate .env files, and the suffix determines the setup environment. Keep in mind that the deployment environment is not actually bound to the NODE_ENV environmental variable. Even when running our container into the staging the NODE_ENV variable is set to production.

So, first things first, we need to create three sample files, one for each deployment envinronment (dev, staging, production).

❏ ~ touch .env.local.sample .env.production.sample .env.staging.sample

You can now add a variable inside each file.

MY_ENV=This is my production envinronment variable

And display that value in your Next.js landing page:


export default function Home() {
  return (
    <div className={styles.description}>
      <div>Coming from `.env`: {process.env.MY_ENV}</div>
    </div>
  );
}

Then we need to copy the sample .env files into our image. To achieve this, we will pass the building environment as an argument through the build command, so we need to modify our Dockerfile accordingly.

For the first step, let's declare an argument called SETUP_ENVINROMENT. This argument is set to production by default.

FROM node:20-alpine AS base
ARG SETUP_ENVINROMENT=production

Then for the builder step we will use the SETUP_ARGUMENT, copy and rename the appropriate .env file.


FROM base AS builder
RUN corepack enable
RUN corepack prepare pnpm@latest --activate
WORKDIR /app

# Copy the .env{SETUP_ENVINROMENT} file to the container
# This will copy the .env.production.sample -> .env file if the SETUP_ENVINROMENT is production
COPY .env.$SETUP_ENVINROMENT.sample .env

You can try to build separate images for each deployment envinroment, try to also change the MY_ENV variable and rerun each container.

❏ ~ echo Building staging env
❏ ~ docker build --build-arg SETUP_ENVINROMENT=staging --tag my-app-docker-staging .

Create a CI/CD Pipeline

The process of creating and deploying a Docker image is referred to as a pipeline. A pipeline is a sequence of steps that are executed one after the other to build and deploy your application. You can use a CI/CD tool to automate this process and simplify your deployment management.

There are several CI/CD tools available for automating Docker deployments, such as Jenkins, CircleCI, and Travis CI. In this tutorial, we will utilize GitHub Actions to automate our Docker deployments.

Create the Release Workflow

GitHub Actions is a CI/CD tool that enables you to automate your software development workflows. It offers a collection of predefined workflows that you can use to streamline your deployments and it's almost free for hobby and open source projects.

In order to design a custom workflow for our project, we need to create a YAML file within our repository. This file contains the steps that will be executed to build and deploy our project.

In order to publish our image on Docker Hub as a publicly accessible image, we need to generate a token in Docker Hub. You can create a token by navigating to your Docker Hub account and then to Account Settings -> Security -> New Access Token. You can label your token and grant it with read and write permissions.

Next, add the token to your GitHub repository as a secret accessible to GitHub actions. You can do so by visiting your repository and then heading to Settings -> Secrets and Variables -> Actions. Add two new repository secrets named DOCKER_USER and DOCKERHUB_TOKEN, and set their values to your Docker username and the token you just created.

Once completed, you can proceed to your GitHub repository and create a new file in the .github/workflows directory. You can name the file release.yml and insert the following content:

name: Build and Publish Docker image

on:
  push:
    branches: [main]

jobs:
  docker:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v3.5.3

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2.9.1

      - name: Login to DockerHub
        uses: docker/login-action@v2.2.0
        with:
          username: ${{ vars.DOCKER_USER }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Docker Hub Description
        uses: peter-evans/dockerhub-description@v3
        with:
          username: ${{ vars.DOCKER_USER }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
          repository: ${{ vars.DOCKER_USER }}/my-nextjs-docker
          readme-filepath: ./README.md

      - name: Build and push
        uses: docker/build-push-action@v4.1.1
        with:
          context: .
          file: ./Dockerfile
          push: true
          tags: ${{ vars.DOCKER_USER }}/my-nextjs-docker:latest
          platforms: linux/amd64,linux/arm64/v8

The configuration might initially appear somewhat complex, but let's break it down into smaller components.

First, we need to define the triggering actions for our workflow. In our case, we want to initiate the workflow upon every push to the main branch. Next, we need to define the jobs that will be executed.

We also have a single job called docker that will run on the most recent version of Ubuntu. The job consists of a single step that checks out the code, followed by the setup of the Docker Buildx plugin. Buildx is a Docker CLI plugin that extends the docker command with the complete support of the features provided by the Moby BuildKit builder toolkit.

Aftewards, we need to log in to Docker Hub using the docker/login-action action. Then, we need to set up the description of our Docker image using the peter-evans/dockerhub-description action. Finally, we need to build and push our image using the docker/build-push-action action and we are basically done.

Wrapping Up

In this tutorial, we explored how to containerize a Next.js application using Docker. We also delved into automating our Docker deployments using GitHub Actions. We hope that this tutorial will serve as a helpful introduction to Docker and Next.js. If you have any questions or suggestions, please do not hesitate to get in touch.
Additionally, you may want to check out the complete example of this tutorial.

References

The Art of State Management in React.

vorillaz — Thu, 26 Mar 2020 22:20:56 +0000

This is cross-post from my blog tutorial: https://www.vorillaz.com/the-art-of-state-management/.

Over the past few years, I've had the privilege (or perhaps the curse) of implementing various state management solutions recommended by the React community in production environments. From Flux and Redux to prop drilling and the Context API, I can totally brag,I've seen it all.

Crafting a scalable and efficient state management architecture, particularly for applications with extensive stores, can be challenging. In this guide, I'll walk you through using React Context alongside hooks effectively. We'll create a straightforward Todo application, available on CodeSandbox and GitHub.

Key Principles

To ensure our application is both performant and scalable, we'll adhere to these principles:

Transparency: Maintain control over state changes without side effects.
Component-Centric: Components are responsible for consuming and updating state within their lifecycle.
Minimal Rerenders: Components should only rerender when their specific slice of state changes.
Code Reusability: Easily create and integrate new components with minimal boilerplate.

Understanding Selectors

Selectors are pure functions that compute derived data, inspired by the reselect library often used with Redux. They can be chained to manipulate or retrieve parts of the state.

Consider this simple example where our state stores a list of todo tasks:

const state = ['todo1', 'todo2'];

const getTodos = todos => todos;
const getFirstTodo = todos => todos[0];
const addTodo = todo => todos => [...todos, todo];

getFirstTodo(getTodos(state)); // => 'todo1'
addTodo('todo3')(getTodos(state)); // => ["todo1", "todo2", "todo3"]

To improve readability when chaining selectors, we can use a wrapper function:

const noop = _ => _;

const composeSelectors = (...fns) => (state = {}) =>
  fns.reduce((prev, curr = noop) => curr(prev), state);

composeSelectors(getTodos, getFirstTodo)(state); // => 'todo1'
composeSelectors(getTodos, addTodo('todo3'))(state); // => ["todo1", "todo2", "todo3"]

Libraries like Ramda, lodash/fp, and Reselect offer additional utility functions for use with selectors. This approach allows for easy unit testing and composition of reusable code snippets without coupling business logic to state shape.

Integrating Selectors with React Hooks

Selectors are commonly used with React hooks for performance optimization or as part of a framework. For instance, react-redux provides a useSelector hook to retrieve slices of the application state.

To optimize performance, we need to implement caching (memoization) when using selectors with hooks. React's built-in useMemo and useCallback hooks can help reduce the cost of state shape changes, ensuring components rerender only when their consumed state slice changes.

Context Selectors

While selectors are often associated with Redux, they can also be used with the Context API. There's an RFC proposing this integration, and an npm package called use-context-selector that we'll use in this guide. These solutions are lightweight and won't significantly impact bundle size.

Setting Up the Provider

First, install use-context-selector:

npm install use-context-selector
# or
yarn add use-context-selector

Create a Context object with a default value in context.js:

import {createContext} from 'use-context-selector';
export default createContext(null);

Next, create the TodoProvider in provider.js:

import React, {useState, useCallback} from 'react';
import TodosContext from './context';

const TodoProvider = ({children}) => {
  const [state, setState] = useState(['todo1', 'todo2']);
  const update = useCallback(setState, []);
  return <TodosContext.Provider value={[state, update]}>{children}</TodosContext.Provider>;
};
export default TodoProvider;

Implementing the Main Application

Wrap your application with the TodosProvider:

import React from 'react';
import TodosProvider from './provider';
import TodoList from './list';

export default function App() {
  return (
    <TodosProvider>
      <TodoList />
    </TodosProvider>
  );
}

Creating the Todo List Component

The main component renders a bullet list of todo items and includes a button to add new items:

import React, {useCallback} from 'react';
import Ctx from './context';
import {useContextSelector} from 'use-context-selector';

export default () => {
  const todos = useContextSelector(Ctx, ([todos]) => todos);
  const update = useContextSelector(Ctx, ([, update]) => update);
  const append = todo => update(state => [...state, todo]);

  const add = useCallback(e => {
    e.preventDefault();
    append('New item');
  }, [append]);

  return (
    <div>
      <ul>
        {todos.map(todo => (
          <li key={todo}>{todo}</li>
        ))}
      </ul>
      <button onClick={add}>Add</button>
    </div>
  );
};

Enhancing Selectors

We can use the composeSelectors helper to leverage the power of composition:

const getState = ([state]) => state;
const getUpdate = ([, update]) => update;

const todos = useContextSelector(Ctx, composeSelectors(getState));
const update = useContextSelector(Ctx, composeSelectors(getUpdate));

Optimizing the useContextSelector Hook

For an extra performance boost, implement a wrapper around the original useContextSelector hook:

import {useRef} from 'react';
import identity from 'lodash/identity';
import isEqual from 'lodash/isEqual';
import {useContextSelector} from 'use-context-selector';

export default (Context, select = identity) => {
  const prevRef = useRef();
  return useContextSelector(Context, state => {
    const selected = select(state);
    if (!isEqual(prevRef.current, selected)) prevRef.current = selected;
    return prevRef.current;
  });
};

This implementation uses useRef and isEqual to check for state updates and force updates to the memoized composed selector when necessary.

Creating Memoized Selectors

Add an extra memoization layer for selectors using the useCallback hook:

const useWithTodos = (Context = Ctx) => {
  const todosSelector = useCallback(composeSelectors(getState), []);
  return useContextSelector(Context, todosSelector);
};

const useWithAddTodo = (Context = Ctx) => {
  const addTodoSelector = useCallback(composeSelectors(getUpdate), []);
  const update = useContextSelector(Context, addTodoSelector);
  return todo => update(todos => [...todos, todo]);
};

Testing

Testing becomes straightforward with this approach. Use the @testing-library/react-hooks package to test hooks independently:

import {renderHook} from '@testing-library/react-hooks';
import {createContext} from 'use-context-selector';
import {useWithTodos} from './todos';

const initialState = ['todo1', 'todo2'];

it('useWithTodos', () => {
  const Ctx = createContext([initialState]);
  const {result} = renderHook(() => useWithTodos(Ctx));
  expect(result.current).toMatchSnapshot();
});

Handling Async Actions

To integrate with backend services, pass a centralized async updater through the TodoProvider:

const TodoProvider = ({children}) => {
  const [state, setState] = useState(['todo1', 'todo2']);
  const update = useCallback(setState, []);
  const serverUpdate = useCallback(
    payload => {
      fetch('/api/todos', {
        method: 'POST',
        body: JSON.stringify(payload)
      }).then(data => {
        // Optionally update the state here
        // update(state => [...state, data])
      });
    },
    [update]
  );
  return (
    <TodosContext.Provider value={[state, update, serverUpdate]}>{children}</TodosContext.Provider>
  );
};

Advanced Techniques

In rare cases, you might need to combine data from multiple providers. While this approach is generally discouraged due to potential performance issues, here's how it can be implemented:

export const useMultipleCtxSelector = ([...Contexts], selector) => {
  const parseCtxs = useCallback(
    () => Contexts.reduce((prev, curr) => [...prev, useContextSelector(curr)], []),
    [Contexts]
  );
  return useContextSelector(createContext(parseCtxs()), selector);
};

Note that this technique violates the hooks concept by using useContextSelector inside a loop.

Conclusion

While these techniques may seem complex, especially compared to Redux, they offer significant benefits for production-grade projects where state management grows over time. Selectors allow for isolation, composition, and minimal boilerplate code, making components aware of state changes efficiently.

This approach can be particularly effective for creating large forms with controlled inputs without side effects. The main principles can be summarized as:

Actions are only triggered through components.
Only selectors can retrieve or update the state.
Composed selectors are always hooks.

While this method lacks some features like time traveling and labeled actions, it provides a solid foundation for state management. It can save time, effort, and boost productivity and performance in your React applications.

You can find the complete demo application on CodeSandbox and GitHub.

Thank you for your time and attention.

Glossary and Links.

How to Detect Idle Browser Tabs with JavaScript

vorillaz — Sun, 22 Mar 2020 16:44:25 +0000

In some cases, we can found ourselves doing lots of intensive, CPU hungry tasks while the users are interacting with our end products or applications.
Firing pollers, establishing WebSocket connections or even loading media like videos or images could become performance dreadlocks especially if these tasks are consuming resources while there is no need to. It's a really good and meaningful practice to release the main thread from unnecessary workloads or network requests while the users are not actively interacting with the interface. In another manner, in an industry where most hosting providers are introducing quota-based pricing models reducing the network request could also reduce the costs for running your application or services.

The Page Visibility API

All the modern web browsers have incorporated the Page Visibility API which allows us to detect when a browser tab is hidden, moreover, we can also register an event listener in order to detect signals upon visibility changing.

document.visibilityState

The document.visibilityState could either be visible while the page is in a foreground
tab of a non-minimized window or hidden while the page is not actually visible to the user.

We can directly access the document.visibilityState as:

console.log(document.visibilityState);
// => It could be `visible` or `hidden`

visibilitychange Event

We also can easily detect changes in the visibility property using an event listener.

const onVisibilityChange = () => {
  if (document.visibilityState === 'hidden') {
    console.log('> The window is hidden.');
  } else {
    console.log('> The window is visible.');
  }
};
document.addEventListener('visibilitychange', onVisibilityChange, false);

An Example with Polling

Consider a scenario where we are polling our API for updates and we want to avoid making unnecessary calls for idle users. A simplified example would look like this:

const poll = () => {
  const interval = 1500;
  let _poller = null;
  const repeat = () => {
    console.log(`~ Polling: ${Date.now()}.`);
  };

  return {
    start: () => {
      _poller = setInterval(repeat, interval);
    },
    stop: () => {
      console.log('~ Poller stopped.');
      clearInterval(_poller);
    }
  };
};

const poller = poll();
poller.start();

const onVisibilityChange = () => {
  if (document.visibilityState === 'hidden') {
    poller.stop();
  } else {
    poller.start();
  }
};

document.addEventListener('visibilitychange', onVisibilityChange, false);

Asynchronously Loading in the Background

But sometimes we can accelerate our users' end experience by following the other way around. Instead of canceling all jobs and requests we can asynchronously load external dependencies or assets. In that way, users' end experience would be more "contentful" and rich when they come back.

Webpack

Using ES2015 dynamic imports proposal along with the appropriate Webpack configuration manifest we can easily load additional modules or assets in the background

let loaded = false;
const onVisibilityChange = () => {
  if (document.visibilityState === 'hidden') {
    // Aggresively preload external assets ans scripts
    if (loaded) {
      return;
    }
    Promise.all([
      import('./async.js'),
      import('./another-async.js'),
      import(/* webpackChunkName: "bar-module" */ 'modules/bar'),
      import(/* webpackPrefetch: 0 */ 'assets/images/foo.jpg')
    ]).then(() => {
      loaded = true;
    });
  }
};

document.addEventListener('visibilitychange', onVisibilityChange, false);

Rollup

Rollup does also support dynamic import out of the box.

let loaded = false;
const onVisibilityChange = () => {
  if (document.visibilityState === 'hidden') {
    // Aggresively preload external assets ans scripts
    if (loaded) {
      return;
    }
    Promise.all([
      import('./modules.js').then(({default: DefaultExport, NamedExport}) => {
        // do something with modules.
      })
    ]).then(() => {
      loaded = true;
    });
  }
};

document.addEventListener('visibilitychange', onVisibilityChange, false);

Preload with Javascript

Besides using a bundler we can also preload static assets like images using just a few lines of JavaScript.

let loaded = false;

const preloadImgs = (...imgs) => {
  const images = [];
  imgs.map(
    url =>
      new Promise((resolve, reject) => {
        images[i] = new Image();
        images[i].src = url;
        img.onload = () => resolve();
        img.onerror = () => reject();
      })
  );
};

const onVisibilityChange = () => {
  if (document.visibilityState === 'hidden') {
    // Aggresively preload external assets ans scripts
    if (loaded) {
      return;
    }
    Promise.all(
      preloadImgs(
        'https://example.com/foo.jpg',
        'https://example.com/qux.jpg',
        'https://example.com/bar.jpg'
      )
    )
      .then(() => {
        loaded = true;
      })
      .catch(() => {
        console.log('> Snap.');
      });
  }
};

document.addEventListener('visibilitychange', onVisibilityChange, false);

Micro-interactions

Finally, a neat approach for grabbing users' attention is dynamically changing the favicon, using just a few pixels you can retain interaction.

const onVisibilityChange = () => {
  const favicon = document.querySelector('[rel="shortcut icon"]');
  if (document.visibilityState === 'hidden') {
    favicon.href = '/come-back.png';
  } else {
    favicon.href = '/example.png';
  }
};

document.addEventListener('visibilitychange', onVisibilityChange, false);

References

You can also find this post on vorillaz.com

We Want Your Help to Learn Together

vorillaz — Sun, 20 Oct 2019 20:34:07 +0000

For the past couple of months, I had an idea about creating a new open source project for string operations. After lots of procrastination, Plexis is published in GitHub and it is still the most incomplete library ever made because I want to create a community-driven solution that can help us learn and grow together.

The Idea

Plexis is a utility library made for string, something like Lodash, Underscore or Ramda. The basic idea is to pack small utility functions into a single project. Every method lives like a single package inside the registry and it is ready for use.

As I was preparing and documenting the Plexis API, getting myself into coding more or less 50 packages/methods, I had an epiphany.
Instead of creating the entire library at once, I decided to try a different path. The thing is, I want to offer a chance through open source contributions and let us grow together. Therefore, I created a modern JavaScript library with minimal setup and carefully documented how you can make your first open-source contribution. Each new feature is also documented as a GitHub issue. There, you will find descriptions, examples, and even hints about creating a new package.
Also, labels can help you distinguish and pick your very first open-source contribution.

The Vision

Through Plexis we can learn together. I can still recall my very first semester in college learning about string operators and how this knowledge helped me understand the basics of C. I wanted to provide the very same experience for anyone wanting to get involved with a modern, open-source JavaScript library. I've always believed that providing people with an inclusive, friendly and open environment will help us thrive. Newcomers can learn through the process of writing code and understanding the basics of the language. Senior developers, on the other hand, can give us their expertise via reviewing pull requests or open issues,suggesting alternative solutions and best practices or even providing guidance. Finally, collaborating is a valuable asset for every single one of us, especially when it comes to exchanging knowledge and spreading ideas.

What You Can Learn

Plexis will give you an idea about a huge spectrum of technologies, design patterns and will also help you understand JavaScript fundamentals.
You will learn more about string operations, array operations, and regular expressions.
You will find yourself using and understanding closures, pure functions or even how functional programming paradigms work.
You will also have a taste of ES6, like a lot.
You will find out how monorepos and packages live together.
You will experience how a modern library is made using Lerna, Babel, Yarn, and Jest
You will learn how a CI/CD environment works through the GitHub / CircleCI / NPM integration.

Your Help is Crucial

You can start right away by providing feedback, opening issues, suggesting features, starring the project, getting through our documentation and finally making your first contribution to Plexis.

You may also get a chance to win a T-shirt since Hacktoberfest is running till the end of the month.

Thank you so much. Hope to see you soon in our pull requests.

You can also find this post on vorillaz.com

Getting started with Espruino and ESP8266

vorillaz — Tue, 16 Jul 2019 17:17:10 +0000

I have written a series of posts about using JavaScript for embedded development using Johnny-Five.

While it is a great library for programming using Arduino or other popular microcontrollers, it lacks one basic feature, portability.
For the majority of the supported platforms, JavaScript is executed on a host machine that runs Node.js. Johnny-Five transmits basic I/O instructions to the connected board via a USB serial, which acts as a thin client. Basically, we can just control the board by executing commands, thus creating a standalone prototype is not possible.

Say Hello to Espruino

Espruino is a firmware with a JavaScript interpreter integrated right into it. It was founded and developed by Gordon Williams, while it’s coming with an IDE as a Chrome extension, a vivid community and lots of tutorials and examples. Espruino is more like an interpreter, which means we can use JavaScript which gets compiled and executed on the target board.

ESP8266 and Espruino

In the Espruino e-shop, there are a few boards available for just a couple of British pounds. But you can also use any of the other options available. You can get up and running using Espruino with the ESP boards. ESP8266 is a great option, as it's affordable, expandable and there are lots of examples, tutorials, and projects built with it.

Flashing Espruino

For the purpose of this simple tutorial we are going to use the NodeMcu ESP8266 Lua board, which comes with a micro USB port for power, programming and debugging as well as a bunch of soldered pin headers for quick prototyping. Flashing Espruino is a lengthy process and may get quite tricky, especially for newcomers to embedded development. Luckily though we can use Flasher.js, a command-line tool that allows flashing Espruino without any hassle. You can run Flasher.js on Linux, Mac or Windows and there is also support for a bunch of other boards besides NodeMcu.

Download and Install Flasher.js

You can find a binary for your operating system in the Flasher.js releases page which you can download and install. ou may need to download and install some additional drivers in order to allow Flasher.js to communicate with your device. There is also a list with the required drivers per device and operating system available in the documentation page.

Flash Your Device

Open Flasher.js app and connect your device to your development machine. For the NodeMcu ESP8266 Lua board you need to use a USB to Micro USB cable. Flasher will try to auto detect the available port. On Windows, these are going to look like COM3, while on Mac and Linux they will be more like /dev/ttyUSB1 or /dev/cu.usbmodem150. Double-check that the port is the appropriate one, pick the latest Espruino binary and hit the Flash button.

While it's flashing, you should see your onboard LED blinking. When the process is done you can immediately start using your device with Espruino.

Setup the Espruino IDE

Head over to the Espruino Web IDE (page)[https://www.espruino.com/Web+IDE] and install the Chrome extension application. Connect your device and launch the app. Click the "Connect" button on the upper left corner and pick the appropriate port. You are directly connected to your board. You can verify your board, as well as the Espruino build using the Web IDE's console like this:

~ print(JSON.stringify(process.env));

Conclusion

Using Espruino for embedded development can open a whole new world for web developers. ESP8266 has been around for a while and you can start prototyping right away. In a series of upcoming tutorials, I am going to show you how you can use Espruino in order to create lots of interesting experiments and projects.

Further resources

You can also find this post on vorillaz.com

Forem: vorillaz

Netflix But better

Why We Need a Better Netflix

The Tech Stack

Feeding the Data

Full Text Search

Vector Search with AI Embeddings

Moving Forward

Final Thoughts

Further reading

Next.js with Globally Accessible Variables

Setup The Polyfill

Using Zod To Validate The Variables

Using The Variables

References

Tauri vs. Electron: A Technical Comparison

1. Architecture

Electron Architecture:

Tauri Architecture:

2. Performance

Electron:

Tauri:

3. Security

Electron:

Tauri:

4. Development Experience

Electron:

Tauri:

5. Community Support

Electron:

Tauri:

Conclusion:

Next.js with Public Environment Variables in Docker

The Quick and Dirty Solution.

A More Consistent Approach.

Building and Running the Docker Image.

Enhancing Security.

Wrap up.

References

How to Build a Views Counter with Astro DB

Getting Started

Define Your Database Schema

Creating the ViewCounter Component.

Using the ViewCounter Component.

Creating an Article Listing Component.

Client Side Rendering & API Routes.

Wrap Up.

Scale from One to Millions of Users

Choosing the best framework for the Job

Leaving the Cloud?

It's Cheaper.

It's Versatile.

It's Privacy First.

Is It Secure Enough, Though?

You're Helping the Web Move Forward

You Can Get as Complex as You Want

Ok, What's the Catch?

Moving Along

How to Automatically Consume RESTful APIs in Your Frontend

The Problem.

Well, REST is Painful.

API changes.

Documentation.

Type Safety

API Consumption

GraphQL or tRPC to the Rescue?

Swagger and OpenAPI.

Building the API.

Consuming the API.

Generating the API Client.

Installing and Using orval.

Working with the SDK.

Testing the Client.

Isomorphic Data Validation.

Automating the SDK Generation.

Running the Client without Any Backend.

Conclusion.

How to Dockerize a Next.js application

Why You Should Containerize Your Next.js App with Docker

Containerizing Your Next.js App: A Step-by-Step Guide

Installing and Using `orval`.

Edit the `next.config.js` File

Create a `.dockerignore` file