Forem: Uploadcare

Vitest: Blazing fast unit test framework

Uploadcare — Tue, 31 Jan 2023 14:37:06 +0000

If you have heard of Vite, then you have probably heard of Vitest, the fast unit test framework built from it. In this article, let’s explore what Vitest is, how to use it and why it can be the next test framework for your apps.

What is Vitest

Before we get into Vitest, let’s note that Vite is a build tool that allows for faster server starts and updates thanks to its native ESM-based method to serve code on demand:

If you want to get more, feel free to read my Introduction to Vite article.

So Vitest is the unit testing framework built on top of Vite and is an excellent unit test framework with many modern features such as:

Component testing for Vue, React, Svelte, Lit and more.
Out-of-the-box TypeScript and JSX support.
ESM first, top level await.
Multithreading workers.
Filtering, timeouts, concurrent for suite and tests.
Jest-compatible snapshot testing.
Chai built-in for assertions + Jest expect compatible APIs.
Designed with a Jest compatible API.

Let’s get started with a simple example on how to use Vitest in a Vite-powered project.

Example usage: writing unit tests in Vitest

Step 1. Create a Vite project

It’s simple, just type:

npm create vite .

Note: Vite requires Node.js version 14.18+ or 16+.

This will start the process to create a Vite project. You can name your project and choose a template appropriately. For this example, I will be choosing the Vanilla template and TypeScript:

$ npm create vite .
✔ Package name: … vitest
✔ Select a framework: › Vanilla
✔ Select a variant: › TypeScript

Scaffolding project in /tmp/vitest...

Done. Now run:

  npm install
  npm run dev

As you see, after the project is created, you can install the necessary dependencies by running:

npm install

Now, you should have the following project structure:

Finally, to install Vitest, run:

npm install -D vitest

Alternative for non-Vite powered projects

It is just as easy to setup Vitest in a non-Vite powered projects.
Say we have a CRA project and want to use Vitest for testing.

All we need to do is to run:

npm install -D vitest @vitejs/plugin-react

And then add a vite.config.js in the root folder of the project:

/// <reference types="vitest" />
import { defineConfig } from 'vitest/config';
import react from '@vitejs/plugin-react';

export default defineConfig({
  plugins: [react()],
  server: {
    open: true,
  },
  build: {
    outDir: 'build',
    sourcemap: true,
    commonjsOptions: {
      include: [],
    },
  },
  optimizeDeps: {
    disabled: false,
  },
  test: {
    globals: true,
    environment: 'jsdom',
    setupFiles: '@testing-library/jest-dom',
    mockReset: true,
  },
});

Step 2. Write some simple tests

Now that our project is set up, let us write some simple tests.

Vitest has been designed based on Jest, so it shares a lot of similarities. One of them is that it can automatically detect your test files as long as you name it in any of the 3 following formats:

A .js file in a folder named __tests__.
A file with a name like [name].spec.js.
A file with a name like [name].test.js.

So let’s create a new folder in our example project called tests and add a basic.test.js file:

tests/
    |- basic.test.js

In this file, let’s add some simple tests:

import { describe, it, assert } from 'vitest';

describe('Math.sqrt() test', () => {
  it('SQRT4', () => {
    assert.equal(Math.sqrt(4), 2);
  });

  it('SQRT144', () => {
    assert.equal(Math.sqrt(144), 12);
  });

  it('SQRT2', () => {
    assert.equal(Math.sqrt(2), Math.SQRT2);
  });
});

If you have worked with other testing libraries such as Jest, Mocha or Jasmine, you should be familiar with the BDD (Behavior Driven Development) pattern. It describes a function, explains what it does and uses test case(s) to assert that it works as intended.

Step 3. Run tests

To run the tests, the command is:

npx vitest

Or you can configure your package.json to run the command with npm test:

"scripts": {
  "test": "vitest"
}

More Vitest CLI commands can be found at the official documentation.

By default, the tests run in watch mode so if you make any file changes, they will re-run immediately. If the tests are successful, you should see the terminal outputting:

$ npx vitest

 DEV  v0.27.2 /tmp/vitest

 ✓ tests/basic.test.js (1)

 Test Files  1 passed (1)
      Tests  3 passed (3)
   Start at  15:05:07
   Duration  933ms (transform 379ms, setup 0ms, collect 13ms, tests 2ms)


 PASS  Waiting for file changes...
       press h to show help, press q to quit

Vitest VS Code extension

To speed up testing and make debugging easier,
there is a VS Code extension for Vitest available in the Marketplace.

This extension can help to:

filter tests by status,
debug easier,
inspect console output,
rerun tests faster.

Vitest vs Other test frameworks

Vitest is often compared to Jest, another popular test framework.
It is because it is built on top of Jest, making it a more modern and improved version. Also, it offers compatibility with most of the Jest API and ecosystem libraries, making it simple to migrate by following their official guide here.

Just like other widely used test frameworks such as Mocha and Jasmine, Vitest follows a simple describe-it-assert or describe-it-expect pattern. The advantage of using Vitest over them is that it is fast to set up and does not require installing a separate assertion library.

The most convenient advantage of using Vitest is that it requires minimal configuration and can be used with any bundler. You can define the configuration for your dev, build and test environments as a single pipeline under vite.config.js.

A simple example would be setting up a Jest and Babel environment for a React app. Often, you would need to install additional packages besides the ones that come with CRA:

babel-jest,
@babel/core,
@babel/preset-env,
@babel/preset-react,
@babel/preset-typescript,
@types/jest.

After that, you would need a jest.config.js and a babel.config.js to complete setting up the configuration.

With Vitest, you don’t have to install those extra dependencies. All you need is a vite.config.js or vitest.config.js file. Even for non-Vite projects, it is a single file to configure. For example:

import { defineConfig } from 'vitest/config';

export default defineConfig({
  test: {
    environment: 'jsdom',
  },
});

Learn more about configuration in the official documentation.

Here is a summarized comparison of different popular test frameworks with Vitest:

	Pros	Cons
Jest	Compatible with most frameworks like React, Vue, and Babel-based projects. Performance is good. Supports asynchronous code and other wide range of tests. Extended API, can be optionally included when needed. Auto mock any libraries imported for easy testing. Highly active community support.	Auto mocking can slow down performance. When used with third-party libraries or tools, it may not be supported or difficult to configure. Can require a lot of dependencies to set up (i.e. Babel projects).
Mocha	Simple, lightweight and easy to get into. Flexible to configure and include different libraries with. Active community support.	Does not come with a lot of features and need to install other libraries separately. Can be hard to configure as different dependencies has different requirements. Can be complex to include auto mocking or snapshot testing.
Jasmine	Very flexible and is compatible with most frameworks and libraries. Built-in assertion library, so it doesn’t require additional libraries. Easy to read and clean syntax.	Difficult to test asynchronous code. Not as widely used as newer test frameworks.
Vitest	Native ESM and out-of-box TypeScript support. Multithreaded and lightweight. Minimal configuration and setup needed. Build upon Jest API, includes all Jest testing features. Performance is fast, live reloads are faster than Jest. Highly compatible with Vite projects.	Still in early phase, may not have active community support. Does not use a global module like Jest and isolates each test to run, may affect performance when scaling.

More features. Test coverage report

Let’s see a simple example how we can output a test coverage report. First, let’s create a simple counter function in counter.js:

export const setCounter = (count) => {
  return count + 1;
};

Then, let’s create a test for this function in the counter.test.js:

import { describe, it, expect } from 'vitest';
import { setCounter } from '../src/counter';

describe('setCounter', () => {
  it('returns 1', () => {
    expect(setCounter(0)).toBe(1);
  });
});

Now if we run vitest run --coverage, we can see how much of our code has been tested:

$ vitest run --coverage

 RUN  v0.27.2 /tmp/vitest
      Coverage enabled with c8

 ✓ tests/counter.test.js (1)

 Test Files  1 passed (1)
      Tests  1 passed (1)
   Start at  15:19:08
   Duration  2.12s (transform 390ms, setup 0ms, collect 40ms, tests 2ms)

 % Coverage report from c8
------------|---------|----------|---------|---------|-------------------
File        | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
------------|---------|----------|---------|---------|-------------------
All files   |     100 |      100 |     100 |     100 |
 counter.ts |     100 |      100 |     100 |     100 |
------------|---------|----------|---------|---------|-------------------

Let’s say we add a new function in our counter.js:

// added another function
export function setupCounter(element) {
  element.addEventListener('click', () => {
    setCounter(counter++);
    element.innerHTML = `count is ${counter}`;
  });
}

If we run vitest run --coverage, we can see that not all our code has been tested:

$ vitest run --coverage

 RUN  v0.27.2 /tmp/vitest
      Coverage enabled with c8

 ✓ tests/counter.test.js (1)

 Test Files  1 passed (1)
      Tests  1 passed (1)
   Start at  15:31:52
   Duration  2.03s (transform 480ms, setup 0ms, collect 40ms, tests 2ms)

 % Coverage report from c8
------------|---------|----------|---------|---------|-------------------
File        | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
------------|---------|----------|---------|---------|-------------------
All files   |   54.54 |      100 |      50 |   54.54 |
 counter.ts |   54.54 |      100 |      50 |   54.54 | 7-11
------------|---------|----------|---------|---------|-------------------

This report ensures that most of your project’s code is tested.

By default, Vitest uses the c8 package to run coverage reports. You may install it manually with the command:

npm i -D @vitest/coverage-c8

Or install it when prompted:

$ vitest run --coverage
 MISSING DEP  Can not find dependency '@vitest/coverage-c8'

 ✔ Do you want to install @vitest/coverage-c8? …

Even more features. Component testing

Another common testing we can do with Vitest is component testing, which verifies the functionality of individual components.

Let’s add an App.test.jsx to a React App. For this example, we’re using a non-Vite project (i.e. basic Create React App template) to show that Vitest works just as fine.

In this test, we want to test only the App component and here’s what our App.test.jsx will look like:

import { render, screen } from '@testing-library/react';
import { describe, it, expect } from 'vitest';
import App from './App';

describe('App Component Test', () => {
  it ('renders without crashing', () => {
    const { container } = render(<App />);
    expect(container).toBeInTheDocument();
  })

  it('renders learn react link', () => {
    render(<App />);
    const linkElement = screen.getByText(/learn react/i);
    expect(linkElement).toBeInTheDocument();
  });
});

Now we run the tests with npm test or npx vitest. The 2 tests should pass successfully, as shown below:

$ npx vitest

 DEV  v0.27.2 /tmp/vitest

 ✓ tests/App.test.jsx (1)

 Test Files  1 passed (1)
      Tests  2 passed (2)
   Start at  16:12:12
   Duration  1.94s (transform 685ms, setup 126ms, collect 356ms, tests 37ms)


 PASS  Waiting for file changes...
       press h to show help, press q to quit

Conclusion

In this article, we learned about Vitest, a fast and modern unit testing framework powered by Vite. Thanks for reading, I hope it has been a helpful article in getting you started with Vitest. Please refer to the References section below if you would like to read more about Vitest and Vite. Cheers!

References

Originally published by Victoria Lo in Uploadcare Blog

What is NaN in JavaScript?

Uploadcare — Wed, 25 Jan 2023 15:56:50 +0000

This article is a part of the ReturnTrue series. Here we solve ReturnTrue puzzles and try to explain how we find solutions and why exactly they work.

The goal is not to show the answers but to dive deep into the elaborate internals of JavaScript and browsers. Yet, we recommend you try to find solutions yourself. It would be more interesting this way, be sure.

Now let’s continue our journey and move to the 2nd task and the main topic of this article — NaN.

Reflexive

function reflexive(x) {
  return x != x;
}

If you are an experienced developer, you know the answer:

reflexive(NaN)

But there is something to discuss.

First, why is it “reflexive”?

In a nutshell, a binary relation across a set is reflexive if each element of the set is related to itself.

Math definitions from Wikipedia, brrr. Let’s explain it another way.

Imagine a set of natural numbers. You can take any number and check whether it equals itself or not:

> 1 == 1
true

If this check returns true for all the natural numbers, then equality is “reflexive” for them.

The same goes for inequality. Inequality is “anti-reflexive” because it’s false for all the natural numbers:

> 1 != 1
false

But NaN is not equal to itself. It means that for NaN equality is anti-reflexive, while inequality is reflexive. Hence the name.

> NaN == NaN
false

> NaN != NaN
true

Second, how to get NaN in JavaScript?

If we try to make a number from a non-numeric string, JavaScript will not throw an exception. Instead, it will return NaN. It is, well, understandable. But JavaScript is one of the few languages that returns NaN in such a common operation.

E.g. Python throws an exception:

int("a")

$ python nan.py
Traceback (most recent call last):
  File "./nan.py", line 1, in <module>
    int("a")
ValueError: invalid literal for int() with base 10: 'a'

PHP returns zero:

<?
$str = "a";
$num = (int)$str;
echo $num;
?>

$ php -f nan.php
0

But JavaScript says:

> Number('a')
NaN

That’s why every JavaScript newbie knows about NaN. Usually, they do not go further than "NaN is, eh, when a string to number conversion failed". But we will go deeper.

Going deeper

As you know, when parseInt can not do its job, it returns NaN:

> parseInt('a')
NaN

So, we have to check the result of such parsing for NaN. There is a global function isNaN, that is available since the first edition of ECMAScript. This function is suitable for this check, but you should not trust it all the time. isNaN tries to convert the passed argument to a number before the very check. That’s why isNaN returns true for any value that it can’t convert to a number:

> isNaN({})
true

With this in mind, the standard authors should have named isNaN — canNotBeConvertedToNumber. To fix this issue, they introduced Number.isNaN in ES6, which works as expected:

> Number.isNaN({})
false

Unclear browser API is not the only problem in JavaScript related to NaN. As you also may know, NaN as an operand transforms any arithmetic operation result into NaN:

> 1 + NaN
NaN

Plus, NaN does not equal anything, even itself:

> false == NaN
false

> NaN != NaN
true

Such behavior is so unique that the spec suggests checking NaN using a strict inequality operation. Like, duck testing:

A reliable way for ECMAScript code to test if a value X is a NaN is an expression of the form X !== X. The result will be true if and only if X is a NaN.

— 18.2.3 isNaN

It’s important to note here that this situation is not the same as we have with objects in JavaScript:

> {} !== {}
true

But in this case we’re comparing two different objects, while with NaN we’re comparing the same value.
To make it clear, consider these examples:

> const obj = {}
> obj !== obj
false

> const nan = NaN
> NaN !== NaN
true

Anyway, aside from those described above, there are plenty of other ways to get NaN in JavaScript. Most of them are not related to arithmetic operations at all. E.g., you can try to get a code of the character of the string outside of the length of this string:

> ''.charCodeAt(1)
NaN

Or parse an invalid date string:

> Date.parse('a')
NaN

From some perspective, the fact that we get NaN in such cases is okay, but is it? If you can’t get a code of the character of the string, you would like to get an exception signaling the error. But in JavaScript, access to any non-existing element of an array does not lead to an error:

> [1, 2][3]
undefined

> ({})['a']
undefined

> ''[0]
undefined

That’s why in our case JavaScript tries to return a “thing”, which is a number and an error at the same time. You know, NaN is a number:

> typeof NaN
"number"

Shameless plug you may want to click on:

Going even deeper

The NaN weirdness in JavaScript does not end here.

If we try to get the NaN-th character of the non-empty string, we’ll get it:

> 'a'.charAt(NaN)
"a"

It works this way because sometimes JS converts NaN to zero behind the scenes. Surprise!

There are lots of different type conversions described in the spec. One of them is ToInteger. JavaScript uses it when a standard function expects an integer argument. Before the use of the passed argument, JavaScript applies ToInteger to the value of that argument. If the value is NaN, then ToInteger replaces it with zero:

2. If number is NaN, +0, or -0, return +0.

— 7.1.5 ToInteger

Aside from that, there are more curious cases with NaN. Check this out:

> [1, 2, NaN].indexOf(NaN)
-1

> [1, 2, NaN].includes(NaN)
true

indexOf can’t find NaN, while includes can. Why? Because indexOf uses strict equality checks during the search. NaN does not equal itself, so indexOf can’t find it.

includes search is a bit more complicated. It has an explicit check for NaN for the argument and array values.

Another example is exponentiation which goes nuts when we pass NaN. Yeah, we know that “NaN turns any arithmetic operation into NaN” but for the sake of God, what is going on here?

> NaN ** 0
1

It’s a tricky question. Exponentiation in JavaScript does not care what exactly you are trying to raise to the power 0. The result will always be the same:

> ({ wtf: true }) ** 0
1

> (function orly() {}) ** 0
1

Honestly, the exponentiation algorithm is not that simple at all:

> (-10) ** 1.2
NaN

Here we get NaN not because of JavaScript weirdness, but that’s the story for the next time.

Wait, but what is NaN, once again?

Oh, glad you asked!

JavaScript implementations are trying to be ECMAScript compliant, right? Let’s check what the spec says about NaN:

Number value that is a IEEE 754 “Not-a-Number” value.

— 4.3.24 NaN

sigh

Alright, what is that IEEE 754 thing?

IEEE stands for “Institute of Electrical and Electronics Engineers”. It is a professional association that develops engineering standards. In particular, they create standards for telecommunications.

There are IEEE standards for the many things we use daily. E.g. IEEE 1003 is POSIX, while IEEE 802 is a set of standards describing different networks. In particular, 802.11 describes wireless communications, in other words — Wi-Fi. Hence the “versions” of Wi-Fi: 802.11a, 802.11b, and so on. All of them are different versions of that standard.

So, IEEE 754 is a standard for floating-point arithmetic. It describes the implementation of floating point numbers and operations for them what those numbers are we will see in the next articles

Among other things, the standard defines NaN as a special numeric value. Computers should return NaN when they can’t find the operation result. For example, when it’s impossible to compute or the operation itself is invalid.

There are lots of examples of such operations. There are the ones the standard describes:

0 by 0 or ∞ by ∞ division;
0 by ∞ multiplication;
getting a reminder of ∞ by 0 division;
subtraction of infinities of the same sign;
and so on.

There is NaN in most programming languages because they follow the standard. But not all the operations listed above return NaN in those languages. Some may throw exceptions if the language developers have decided so. Otherwise, NaN is usually used.

For instance, you can not divide by zero in Python:

0/0

$ python zero.py
Traceback (most recent call last):
  File "./zero.py", line 1, in <odule>
ZeroDivisionError: division by zero

But you can get NaN by trying to add +∞ to −∞:

float('inf') + float('-inf')

$ python inf.py
nan

Go will do the same, while in PHP you may get NaN by trying to get an arccosine of the value out of [−1; 1]:

<?
echo acos(2);
?>

$ php -f acos.php
NaN

The same is true for C.

As you see, programming languages work with NaN in varied ways. They may not even use NaN in arithmetic. They may throw exceptions in “critical” cases and return defaults in the others.

But some languages use NaN for all the awkward cases. JavaScript is one of those languages.

NaN and floating-point arithmetic are extensive and intriguing topics. But that’s all for today, folks!

If you wish to continue the journey, here is the cliffhanger. There is not one NaN for “all the weirdness”. IEEE 754 defines two of them: quiet NaN and signaling NaN. Happy research!

Originally published by Igor Adamenko in Uploadcare Blog

TCP & UDP, or the two pillars of the Internet

Uploadcare — Sun, 22 Jan 2023 16:30:08 +0000

As a company who offers a CDN service we know a lot about this technology. In this series of articles we’re trying to describe how a CDN works and what are the things that make the existence of CDN possible.

In the previous article we discussed the structure of the Internet,
the differences between mediums, and the problems that users encounter trying to connect to servers far away from them. Now it’s time to go a few levels up to see what is “the language” computers use to communicate with each other.

The plan

There is no way we can even briefly discuss everything that relates to the Internet. Thus, in this and the following articles we are going to look at the exact subset of the technologies — the part laying between physical connection and applications.

We’re not going to discuss how computers use electricity to represent and transfer data. This topic is fascinating, but it is too low-level for us.

At the same time we won’t discuss in detail how your browser loads pages: no HTTP requests, no compression, no programming languages evaluation, and no rendering.

What we are going to discuss is how the data from your computer finds its way to another computer.

Switching

If you’re old enough, you might remember the landlines we had back in the days at home. Or maybe you still have one! If so, you probably know that a hundred years ago they worked in a semi-manual mode.

First, you had to ask a switchboard operator (who was a real human) to connect you with your friend from another city. The operator said “OK,” hung up, and then later they called you back with your friend “on line.”

What you may not know is the procedure this operator had to do for you. They switched cables from one port to another to create the connection:

This operation is called circuit switching. Circuit switching is a method of establishing connection between two nodes of a network, when they have a dedicated channel to communicate through. It means that when you dial a friend, there is someone or something that literally builds up a cable path from you to your friend.

As you might guess, this type of commutation does not suit the Internet well, because we connect all the time to servers all over the world. Implementing physical cable switching for this purpose would be too crazy. It would be too expensive to create and even more expensive to maintain.

Instead, the Internet is built upon packet switching. In this type of communication there is no physical cable switching. When we send a message from our computer to our friend’s computer, it goes through different networks, each of those transferring the message to the path that it thinks would be the best.

This may sound mad, but most of the time there is no “main coordinator” of the Internet who decides where every packet must go. Our computer simply “throws” the packet to the network, and all the devices in this network are trying to decide where this packet should travel. Surprisingly, almost all the time these devices do their job pretty well. All thanks to the protocols!

Protocols

A protocol is a set of rules defined to allow nodes of a network to transmit data properly. You probably know some high-level protocols such as HTTP, SSH, IMAP and DHCP.

There are tons of protocols in the communication area that are used for different purposes. Due to the huge amount, there are even special models defining “layers of protocols.” We might imagine these layers as a collection of different candy wrappers.

Let’s say you have ordered a bag of delicious handmade chocolate from a local manufacturer to make a gift for your friend.

First, a cook wraps a chocolate bar with its own wrapper. Then a packer put the bar into a branded bag. After that somebody from the delivery department uses a piece of wrapping paper to make it look like a gift. Finally, they pass the gift to the postal service.

When the postal service gets the gift, they put it into a box and write the address of your friend on this box. Then they load a truck with the boxes like yours and go to deliver.

The process goes vice versa on your friend’s side.

This wrapping on each step is literally how protocols work on the Internet:

OSI & TCP/IP history

Historically there are two main models that describe protocol layers. One of them is theoretical — the OSI model, another is practical — TCP/IP model.

There were a lot of possibilities and a lot of problems to solve in the telecommunication area in the 70-80s. There was no Internet as a thing, but there were a bunch of networks created by different companies. Those networks needed a common language to communicate. Therefore many people wanted to create this language. Roughly speaking these people were split into two groups.

The first group believed that the development of the language should be an open process, with all the criticism discussed and all the issues resolved. This group named the standard they were trying to create as OSI, standing for Open Systems Interconnection.

Another group chose the different way, they started to develop and test protocols on the network they already had. Thereby, they created TCP — Transmission Control Protocol, and IP — Internet Protocol. Initially, these protocols were the parts of one big protocol, but later they were divided into two to take the maintenance easier.

This group successfully tested and moved the network they had to this TCP/IP stack. This happened on January 1st, 1983. This is truly the day when the Internet was born.

After that TCP/IP started to gain popularity, because it simply worked. Also it was a free of charge model, while authors of OSI wanted to get money for each usage of their standards.

So, what happened with OSI? The initiative failed. Due to the idea of “openness”, the people, who made decisions back then, spent a lot of time in discussions and disputes. Some of these people tried to lobby for ideas that were beneficial to large corporations. Others forced everybody to discuss small, not important things.

Even though the OSI model didn’t find it’s way to the real world, it’s still used as a good theoretical model of protocol layers. But the model that is actually implemented is TCP/IP.

Here’s how the layers of the models look:

TCP/IP

Usually when people discuss OSI and TCP/IP they try to compare these models and match their layers. Even though the pic above looks like a comparison, we are not going to compare the models, because matching of the layers is not so important. Even more, the layering itself is “considered harmful” according to RFC 3439.

The key protocols of TCP/IP are obviously TCP and IP. Everything around them is not so smooth, so even authors of the textbooks about network communication cannot agree with each other about the exact number of layers or their names.

Is there a Physical Layer below the Network Access Layer? Should we even call Network Access Layer so, or it’s better to call it Link Layer? There are many questions like these. However, all of these questions do not matter, because the main idea of TCP/IP stack is pretty simple.

Let’s say there are two applications on the network that want to communicate. One of them generates a message. This message may be formatted using a protocol like HTTP, or not, it does not matter. The message is generated on the Application Layer.

This message is passed to the Transport Layer, where it’s prepended with a TCP or a UDP header. The header contains information about the ports those applications are using and some other important information which we will describe later. The result of this prepending may be called a datagram or a segment.

Then, the segment is passed to the Internet Layer. It’s pretended with an IP header, which contains the addresses of the computers that run the communicating applications. The combination of a segment and an IP header is usually called a packet.

There is no big difference between datagrams, segments and packets. Most of the time all of them are called “packets.” The terms “datagram” and “segment” are used when it’s important to highlight the type of the protocol handling the entity.

Finally, the packet is passed to the Link Layer, where it’s encoded and transmitted over the network. There are protocols that also add their own headers to each packet, and the result of this addition is a frame.

Frames usually contain not only the header, but also the footer. E.g. here’s how an Ethernet frame for our message may look like:

When the frame is passed from one intermediate node to another during the communication, these nodes disassemble the frame, check the packet’s header, decide what they should do with the packet, then create a new frame for the packet and pass it to the next node.

So, when the message is passed through nodes like this:

The underlying data flow looks like this:

Alright, now we know that TCP/IP specifies the protocols of the Transport and Internet Layers, but does not really care about the protocols of the Application and Link Layers. Let’s see what’s going on on the Transport layer in detail.

TCP & UDP

The Transport Layer of TCP/IP stands on two main protocols: Transmission Control Protocol and User Datagram Protocol. There are other protocols on this layer (e.g. QUIC) but they are not so commonly used yet.

The protocols of the Transport Layer are used to address the packet from the source port to the destination port. Moreover, the protocols do not know about the differences between nodes on the used network. Everything the protocols know about addressing is the fact that the source application is bound to a port, and the destination application is also bound to a port. The actual “Internet addressing” is done by Internet Protocol which we will describe in the next article.

UDP is much “lighter” and simpler than TCP, but at the same time UDP is not reliable and does not do much work compared to TCP. To see the difference between these protocols, let’s start with the User Datagram Protocol.

User Datagram Protocol

This protocol is used for “connectionless communication,” which means that the connection is not set up prior to the actual sending. One node sends packets to the network addressing them to another node. The sender node does not ensure that the recipient node exists or is ready to receive the packets. The sender node also does not wait for confirmation that the recipient node has got the previous packets.

The packets handled by UDP are sometimes called datagrams. The term usually describes the packets sent using any connectionless communication.

UDP header contains 8 bytes of information that includes:

Source port
Destination port
Length
Checksum

As simple as it sounds. A typical UDP datagram looks like this:

Field test

It’s easy to set up a UDP server on your own. Here is an example written in Node.js right from the official docs:

const dgram = require('dgram');
const server = dgram.createSocket('udp4');

server.on('error', (err) => {
  console.log(`error:\n${err.stack}`);
  server.close();
});

server.on('message', (msg, rinfo) => {
  console.log(`got a message from ${rinfo.address}:${rinfo.port}: ${msg}`);
});

server.on('listening', () => {
  const address = server.address();
  console.log(`listening ${address.address}:${address.port}`);
});

server.bind(8082);

Let’s start the server:

$ node udp.js
server listening 0.0.0.0:8082

Now we can use netcat to send datagrams from another terminal window:

$ nc -u 127.0.0.1 8082
Hello, server!
Do you have a minute to talk about UDP?

Our server successfully handles the datagrams we’ve sent:

$ node udp.js
server listening 0.0.0.0:8082
server got a message from 127.0.0.1:55823: Hello, reader!

server got a message from 127.0.0.1:55823: Do you have a minute to talk about UDP?

As you see netcat has bound itself to the port 55823 to send the packets. If we use Wireshark to intercept the packets, we will see this:

Quick Wireshark tutorial. At the top there is a log of the packets. Below there is a parsed data of the chosen packet. Note that all the data surrounded by square brackets is calculated by Wireshark and does not exist in the packet “as is”. Finally, at the bottom there is a text representation of the packet.

By the way, if you don’t want to set up Node.js server and capture the communication yourself, feel free to use our dump file.

So, what do we see here? The source port is 55823 as netcat picked. The destination port is 8082 as we set it. The length is 48 bytes because the first 8 bytes are occupied by the header and the rest is the message we sent. Finally, there is even some checksum set by netcat.

As you see, there is no answer from the UDP server in the Wireshark log, so the client can’t be sure that the server actually got the datagrams. That’s why UDP is used when it’s fine to lose messages, e.g. for video or audio streaming. It’s possible to use TCP for these purposes, but as you will see in a moment the usage of TCP slows down media streaming.

Well, UDP does not have confirmations but we still can detect unreachable destinations using ICMP. Try to send UDP requests using netcat without the started server and check the Wireshark logs. Here’s what you may see there; here’s the dump.

Shameless plug you may want to click on:

Transmission Control Protocol

While UDP is simple and connectionless, TCP is complex and connection-oriented. It means that prior to sending the data, client and server establish the connection — they agree on some sort of settings they will use to communicate.

Unlike datagrams in UDP, packets handled by TCP are called segments.

Here’s what TCP does during the communication:

It ensures the order of the packets.
It ensures the delivery.
It controls the flow of packets to avoid network nodes congestion.

That’s a lot of work, so the TCP header’s size is at least 20 bytes. The header includes:

Source port
Destination port
Sequence number
Acknowledgment number
Header length (or “Data offset”)
9 control bits (or “Flags”)
Window size
Checksum
Urgent pointer
Options

Here’s a picture to visualize the segment:

The source port, the destination port and the checksum are the same as the UDP header has. To explain the purpose of the rest header fields let’s create a test server.

Field test

Let’s set up a simple TCP server using Node.js (again, using the code from the official docs):

const net = require('net');

const server = net.createServer(connection => {
  console.log('client connected');

  connection.on('end', () => {
    console.log('client disconnected');
  });

  connection.write('Hello, client!\n');

  // 'connection' is a stream which we can pipe

  // first we pipe to stdout to print messages from the client
  connection.pipe(process.stdout);

  // then we pipe to itself to force this stream to read itself
  // making the incoming messages outcoming
  connection.pipe(connection);
});

server.on('error', (err) => {
  console.log(`server error:\n${err.stack}`);
  server.close();
});

server.listen(8082, () => {
  const address = server.address();
  console.log(`server listening ${address.address}:${address.port}`);
});

The code above looks like the UDP server we’ve created earlier, but instead of a straightforward handling of incoming messages, we operate a connection object. This connection object allows us not only to print the messages we get, but also to detect when a client is connecting or disconnecting, and to send some responses back. If you set up HTTP servers earlier you should see the similarities.

Let’s start the server using this code:

$ node tcp.js
server listening :::8082

Double colon instead of a “usual” IP means that the server is bound to the localhost using IPv6, but it does not matter for now.

Now use netcat to start a TCP connection to the server from another terminal window:

$ nc 127.0.0.1 8082
Hello, client!

The string we’ve got here has come from the server when the connection has been successfully established. Here’s what we have in the server logs:

$ node tcp.js
server listening :::8082
client connected

As you see the server got a notification from the client and it knows that the client exists, and this client is going to send something. So let’s send:

$ nc 127.0.0.1 8082
Hello, client! # message from the server
Hello, server! # our message
Hello, server! # reply from the server
Hey, stop it!  # our message
Hey, stop it!  # reply from the server

We’ve written a simple echo server, which sends back everything it gets from us.

Close netcat by pressing Ctrl+C and check the server logs:

$ node tcp.js
server listening :::8082
client connected
Hello, server!
Hey, stop it!
client disconnected

There are two messages we’ve sent earlier and the notification about the disconnection.

So, how is it possible for the server to know about the connection state while we haven’t sent anything yet? To figure it out let’s use Wireshark again.

Here’s how all the connection looks there:

Again, if you can’t set up a Node.js server by yourself, but want to dive into the logs, feel free to use our dump.

That’s a lot! Let’s investigate!

Handshake

When a client wants to send data to a server, first they both should establish a connection. This “establishing” is usually called handshaking.

The handshake consists of three steps:

A client sends a packet called SYN to the server.
The server sends a packet called SYN-ACK to the client.
The client sends a packet called ACK to the server.

The names are derived from the flags that are set in TCP headers of these packets. Here’s how the first packet looks in Wireshark:

As you see, one of the bit flags is set to 1. This flag is called “Syn,” standing for “Synchronization”.

TCP is a reliable protocol, which means that the sender is always notified whether or not the recipient has got the packet. To implement this feature TCP specifies special types of packets (such as SYN, ACK, etc) and every TCP header contains two numbers: Sequence number and Acknowledgment number.

Sequence number refers to the amount of data that the sender of the packet has sent so far. The sequence number is increased by 1 when SYN or FIN flags are set, it’s also increased by the payload size when the payload exists. The number does not include the current packet. Thus, the first packet sent by the client always has a sequence number set to 0.

Acknowledgment number refers to the amount of data that the sender of the packer has received for far. It’s a mirroring of the sequence number from the opposite side of the connection, but with one step ahead. Thus, the first packet sent by the server has an acknowledgement number set to 1.

Actually, the starting values of those numbers are picked up randomly. But it’s easier to operate on relative values as you may see on the Wireshark screenshot above.

Those numbers are useful because they help to maintain the state of the communication on both sides. Both the server and the client expects to get exact acknowledgement and sequence numbers from the other side. When the actual numbers are not the same as the expected, it means that there is an error somewhere and someone should retransmit the data.

Here’s how the handshake looks in our case:

When the handshake is done, our server knows that there is a client ready to communicate, and the server waits for this communication.

Data transfer

When the handshake is done, it is time to send a payload.

As you see in the log above, each piece of information sent by one end is ACK-ed by the other using the separate segment.

It looks simple and straightforward, but that’s because our example is simple. In the real world there are a lot of problems: packages may be lost, the channel may be congested, an error might occur, and so on. TCP handles all these situations. How? This is a story for another long series of articles. Let us know if you’re interested and we may write about TCP in detail.

By the way, every packet in the log above has the flag PSH set. This is unusual, and in real life this flag is not so commonly used. This is probably netcat specific behavior.

Termination

When one of the nodes is going to close the connection, it initiates a termination. The process itself looks similar to the connection initiation, but instead of using the SYN flag the ends use the FIN flag.

Here is the termination in our Wireshark log:

In our case the termination has taken four segments, which is not like the handshake. It works this way because when the connection is terminated, the other party (the server in our case) must first notify the Application Layer about the termination. When the app that is using the connection is ready to close it, then the TCP of the other party sends its FIN packet.

Due to the fact that nodes “agree” on the termination we are able to listen for such an event in our code above.

By the way, sometimes termination is triggered by the RST flag instead of FIN. It’s kind of hacky, but it is possible. Also, the RST flag is used for the attack which prevents TCP connection between the nodes. If it sounds interesting to you, check out this article by Robert Heaton.

Summary

In this article we’ve found out that the Internet is built around packet switching, and the core protocol stack is TCP/IP. We have also briefly looked at the two main protocols of the Transport Layer of TCP/IP: the very TCP and UDP.

In the next article we will dive into Internet Protocol, and we will see how it glues every computer in the world into the biggest network.

Originally published by Igor Adamenko in Uploadcare Blog

A useful function that does nothing

Uploadcare — Thu, 19 Jan 2023 17:12:48 +0000

This article is a part of the ReturnTrue series. Here we solve ReturnTrue puzzles and try to explain how we find solutions and why exactly they work.

Alright, the disclaimer is over. Let’s continue with an introduction explaining what ReturnTrue is.

Introduction

ReturnTrue is a set of puzzles. Each of them is a tricky function.
You should pass arguments to such a function, and the result of its evaluation should be true. Not “truthy value”, but exactly true.

Erling Ellingsen, a programmer from Norway, created the game more than six years ago. Kudos to him and all the contributors!

Finally, let’s start.

ID

function id(x) {
  return x;
}

Easy. The main goal of this task is to explain the rules.

The solution is obvious:

id(true)

The game assumes that the shorter solution the better. What is shorter than true? Not-zero!

id(!0)

Two symbols. Not bad.

But wait, why is it called “id”?

Glad you asked!

The title “id” here is not by accident. There is a mathematical term called identity function. It is a function that returns its argument without changing it.

To explain when we may use it, let’s assume we want to go through an array of numbers and increase each odd value by 1. Here is how it may look in JavaScript:

arr.map(x => x % 2 ? x + 1 : x)

But even though we use map here, the code is not “functional” enough. In functional programming, it is more common to use functions, not statements. In other words, programmers say what to do, not how. They are trying to be declarative, not to be imperative.

So, if our code was truly functional, it would look like this:

map(arr, when(isOdd, inc, id))

Here you can see a typical functional way to process data.
We pass data to a pipeline, which transforms it, and returns the result. No more of those nasty statements, my precious!

Usually, a functional programming language has the functions like those we used. For instance, Clojure has identity, F# has id, Elixir has identity/1, and so on.

But, JavaScript was not designed as a pure functional language. So, their implementation may look like this:

const isOdd = x => x % 2 === 1;
const inc = x => x + 1;
const id = x => x;
const when = (cond, thenFn, elseFn) => x => cond(x) ? thenFn(x) : elseFn(x);
const map = (xs, fn) => xs.map(fn);

And now you see when the function returning its argument may be handy.

Do we need such a function?

Yeah, it may look weird in the context of front-end development. One may even say that id function is a kludge from FP world.

Sure, it may sound like a joke or a “bug” of programming language design. First, you define functions that can not be evaluated without passed arguments (e.g. when function we defined above). Then, you define more functions to use them as a sticky patch for the previously defined.

However, the functional programming paradigm is trying to be as close as possible to math. id function is a normal thing for this field of science.

It may not be obvious why exactly we need such a function. But, when we work with numbers, we use some special ones like 0 or 1.
When we work with arrays, the special one is an empty array. When we work with functions, we need a “special” one too. It is id.

It may look like overhead sometimes, and you may think: "Why not define when function another way, where elseFn will not be required?". Sure, it’s possible. But explicit is better than implicit.

Originally published by Igor Adamenko in Uploadcare Blog

We asked ChatGPT how to use our service. Here is what AI answered

Uploadcare — Thu, 12 Jan 2023 05:49:48 +0000

Welcome to 2023! Here we have chatbots that can easily replace real humans!

OpenAI launched ChatGPT in November 2022. We played with this bot, discovered some appealing applications, and forgot about it.

But then we thought: “What if we ask ChatGPT to build a file uploader page using Uploadcare?” Why not? It should be entertaining.

Instead of providing the context to the bot at the start, we decided to play a role of a junior developer. Our “developer” heard something about frontend, HTML, CSS & JS, but they do not know much. This is why they ask a lot of newbie questions. To make it easier to read: our questions are headlines, and ChatGPT’s answers are the normal text. Italic text is our comments on the whole play.

We used the bot’s answers as is. The only thing we removed is the ubiquitous reminders that the bot is ready for any questions. The bot says it at the end of each reply.

Let’s get started, huh?

How to create a simple HTML webpage with a file input?

Here is a simple HTML webpage with a file input field that allows users to select a file to upload:

<html>
  <body>
    <form>
      <label for="file">Select a file:</label><br>
      <input type="file" id="file" name="file"><br>
      <input type="submit" value="Submit">
    </form>
  </body>
</html>

This HTML code creates a form with a file input field and a submit button. When the user selects a file and clicks the submit button, the form will be submitted and the selected file will be uploaded.

How to serve a simple HTML file on localhost to be able to open it in a browser?

To serve an HTML file on localhost, you will need to use a web server. Here are the steps to serve an HTML file on localhost using a simple Python web server:

Install Python: Make sure you have Python installed on your computer. You can check if Python is already installed by opening a terminal and entering the command python --version. If Python is not installed, you can download and install it from the Python website.
Start the web server: Open a terminal and navigate to the directory that contains your HTML file. Then start the Python web server by entering the following command:

   python -m http.server 8000

This will start a web server on your local machine that listens on port 8000.

View the HTML file in your browser: Open a web browser and enter the following URL:

   http://localhost:8000/your_html_file.html

Replace “your_html_file.html” with the name of your HTML file. This will open the HTML file in your browser.

Well it works for now:

Give me code that installs Uploadcare widget on my website, please

We are trying to be polite. Never know when AI becomes Skynet!

To install the Uploadcare widget on your website, you will need to add the following code to your HTML:

<script src="https://ucarecdn.com/libs/widget/3.x/uploadcare.full.min.js" charset="utf-8"></script>

This will include the Uploadcare widget library in your website.

Once you have added this script to your HTML, you can then use the widget to allow users to upload files from your website. Here is an example of how you can do this:

<!-- HTML for file input -->
<input type="hidden" role="uploadcare-uploader" name="my_file" />

<!-- JavaScript to initialize the widget -->
<script>
  UPLOADCARE_PUBLIC_KEY = 'your_public_key';
  var widget = uploadcare.Widget('[role=uploadcare-uploader]');
</script>

In this example, the input element with the uploadcare-uploader role serves as a placeholder for the widget. The JavaScript code initializes the widget and specifies your Uploadcare public key, which you can find in your account settings.

Again, it works:

We tested the widget, it works properly. But this is an old way to install our uploader. Currently we have a beta version of File Uploader, that is based on Uploadcare Blocks. Let’s find out what the bot knows about it!

I see on their website, that now they offer something called “Blocks”. How do I create a file uploader using these?

Uploadcare Blocks is a new way to build file uploaders and file handling workflows using pre-built UI components. It allows you to easily add file uploading and file handling functionality to your website or application with minimal code.

To create a file uploading form using Uploadcare Blocks, you will need to do the following:

Sign up for an Uploadcare account: Go to the Uploadcare website and sign up for an account. You will need to provide your email address and create a password.
Install the Uploadcare Blocks library: To use Uploadcare Blocks, you will need to include the Blocks library in your HTML code. You can do this by adding the following script tag to your HTML:

   <script src="https://ucarecdn.com/libs/blocks/2.x/uploadcare.min.js" charset="utf-8"></script>

Add a file input element to your HTML: To create a file uploading form using Uploadcare Blocks, you will need to add a file input element to your HTML code. You can do this by using the following HTML:

   <lr-file-uploader-regular name="my_file"></lr-file-uploader-regular>

This will create a file input element that will be used to select and upload files.

Initialize the Uploadcare Blocks library: To initialize the libaby set the Uploadcare public key when using the lr-file-uploader-regular element. You can use the --cfg-pubkey attribute as a CSS context property. For example:

   <style>
     lr-file-uploader-regular {
       --cfg-pubkey: "your_public_key";
     }
   </style>

Replace “your_public_key” with your actual Uploadcare public key, which you can find in your account settings. This will set the public key for the file uploader and allow it to connect to the Uploadcare API.

Ha! This time it does not work!

Let’s ask the robot why!

The link you provide, that should install the library, is broken

I apologize for the broken link. The correct URL for the Uploadcare Blocks library is https://unpkg.com/@uploadcare/blocks@0.14.3/web/blocks-browser.min.js. You should use this URL to include the Blocks library in your HTML code.

To include the Blocks library in your HTML, you will need to add the following script tag to your HTML:

<script src="https://unpkg.com/@uploadcare/blocks@0.14.3/web/blocks-browser.min.js" charset="utf-8"></script>

This will include the Blocks library in your HTML, allowing you to use the Blocks library to build file uploaders and file handling workflows.

Well, mister smarty-pants, it looks like you’ve forgot something:

Thanks, now the script works, but it looks like styles are not loaded. Here’s what I see in the console: “CSS Data error: --icon-upload”. How to fix the styles?

To fix the CSS errors that you are seeing in the console, you will need to include the CSS styles for the Uploadcare Blocks library in your HTML.

You can do this by adding the following link element to the head of your HTML:

<link href="https://unpkg.com/@uploadcare/blocks@0.14.3/web/file-uploader-regular.min.css" rel="stylesheet">

This will include the CSS styles for the Blocks library in your HTML, and should fix the CSS errors that you are seeing in the console.

Alternatively, you can use the css-src attribute on the lr-file-uploader-regular element to specify the URL of the CSS styles for the Blocks library. For example:

<lr-file-uploader-regular
  name="my_file"
  css-src="https://unpkg.com/@uploadcare/blocks@0.14.3/web/file-uploader-regular.min.css"
></lr-file-uploader-regular>

It works!

We’ve managed to upload a file using the code the bot suggests:

Alright. When I uploaded an image to Uploadcare, how do I inject it to the webpage? I also want to resize it to 256×256px

To display an image that you have uploaded to Uploadcare in your webpage, you will need to use an img element and set the src attribute to the URL of the image on Uploadcare’s servers.

For example:

<img src="https://ucarecdn.com/image_id/">

Replace “image_id” with the actual ID of the image that you want to display. You can find the ID of the image in the response that the Blocks library returns when you upload the image.

To resize the image to a specific size, you can use the -/resize/ operation in the URL of the image. For example, to resize the image to 256×256 pixels, you can use the following URL:

<img src="https://ucarecdn.com/image_id/-/resize/256x256/">

This will display a 256×256 pixel version of the image. You can specify any size that you want by changing the values in the resize operation.

Easy-peasy:

Now, let’s sum up our experiment without italic.

Conclusion

To be honest, when we started the experiment we did not believe that ChatGPT could deal with it. Sure, it’s a cool toy, and it even knows how to generate code examples related to your problem. But we are surprised that the bot knows a lot about our new features, recent documentation, and so on.

It feels like you’re talking to an assistant who can google for you.

After the first rapture had passed, we asked ChatGPT more programming-related questions. We would say that it generates code that looks like it should work. If you do not know how to solve the problem, the code you get can mislead you because it is too convincing.

Besides, ChatGPT makes mistakes. In this article, we’ve encountered only one — the broken link. But the more questions you ask, the more connected the bot’s answers are. Sometimes they become too connected and start to affect each other. E.g. ChatGPT adapts replies to the language or programming paradigm you prefer. This “feature” sometimes leads to broken or unoptimized code.

Anyway, ChatGPT is still a handy tool to use. There are some use cases:

To collect a broad overview of some topics. E.g. you may ask about scientific concepts or discoveries. When the bot answers, you may ask about relations to other concepts, key people in the field, and so on. It’s like reading those Wikipedia snippets in Google, but you can get such a snippet for any of your questions. Try: "Tell me, please, what Alan Turing’s most popular paper was about?"
To handle interactive chats on your website or app (no way!). Or to help a human in a call center to find an answer in a bunch of documentation.
To generate blocks of boring text, like legal documents or official letters. Sure, a lawyer should review the output, but the result is not that bad already. Try: "I’m creating an Android game that does not collect any data. It is not even connected to the Internet. Do you mind generating the privacy policy for the game?"
To extract SEO keywords from the provided text. But we will see what Google thinks about it.
To simplify the description of a complex thing to the desired level. E.g. try to copy-paste a science term definition from Wikipedia and ask ChatGPT to simplify it for a second-grade student.
To solve puzzles together. It can solve crosswords with ease. If you try hard, you can even explain the bot Wordle rules and get help with today’s word!

Finally, if you have a tech company, it’s interesting to see whether the bot can use your service. The result of such a check may trigger you to improve the docs or to clarify the purpose of your product.

Give it a try, and, who knows, you may find ChatGPT a place in your toolset.

P.S. By the way, we generated an OpenGraph image and the image you see below via Midjourney, while ChatGPT helped us to write the text on that images. Gosh, what a time to be alive!

Originally published by Igor Adamenko in Uploadcare Blog

Import maps 101

Uploadcare — Tue, 10 Jan 2023 20:15:04 +0000

As web developers, the use of external libraries saves a lot of time and effort, allowing us to build apps faster and more efficiently. In this article, I will be introducing you to a spec called import maps, which we can use to control the behavior of importing these libraries.

Introduction

Typically, using external libraries in browsers requires bundlers/build tools such as webpack or browserify, in order to parse the code and allow the use of import statements. The modules which are imported without an absolute or relative URL are known as bare modules. Such modules cannot be imported in the browser without bundle tools. For example, importing the lodash package would be written as:

import _ from 'lodash';

This bare import specifier makes it easier to write and manage code, without having to write absolute or relative URLs of the library.

What exactly does import maps solve?

With bundlers and build tools solving the issue with bare imports, you may now wonder what problem import maps actually solves.

The short answer is: import maps can be used to load bare modules without the need for bundlers.

The longer explanation is that normally, when code gets bundled together into a large file, import maps can be used to control the behavior of importing libraries. Examples of such behaviors we might want to control include remapping of URL-like specifiers, scoping, dynamic imports, etc. This control allows better caching for module script graphs and better developing performance.

Example usage

Let’s take a look at an example implementation of import maps.
In this tutorial, we will learn how to import a bare module like dayjs without the use of a bundler.

Step 1. Create HTML and JS files

First, let’s create a new HTML page and a script file.

In the HTML file, let’s import a JavaScript file (still empty though) as a module like so:

<!doctype html>
<html>
<head>
  <meta charset="utf-8">
  <title>Import maps demo</title>
  <script type="module" src="./index.js"></script>
</head>
<body>
</body>
</html>

Step 2. Write a simple script in the JavaScript file

Now let’s write something simple that uses dayjs.
For this example, I am writing a script that prints out the current datetime every second:

import dayjs from 'dayjs';

const el = document.createElement('h1');

document.body.appendChild(el);

setInterval(() => {
  el.innerHTML = dayjs(Date.now()).format('YYYY-MM-DD HH:mm:ss');
}, 1000);

Step 3: Add an import map to import the library

If you try running the page now, it would not work because the dayjs we imported in the JavaScript file is a bare module.
You will see the following error:

Uncaught TypeError: Failed to resolve module specifier "dayjs". Relative references must start with either "/", "./", or "../".

So let’s use import maps to resolve this error.
Change the HTML file in this way:

<!doctype html>
<html>
<head>
  <meta charset="utf-8">
  <title>Import maps demo</title>

  <!-- Add this importmap before index.js -->
  <script type="importmap">
    {
      "imports": {
        "dayjs": "https://cdn.skypack.dev/dayjs@1.11.5"
      }
    }
  </script>

  <script type="module" src="./index.js"></script>
</head>
<body>
</body>
</html>

Note: When implementing import maps, ensure that they are added before the first module load is started, in this example, index.js.

Now let’s run the page, and our script should be running with no issues:

Scopes

Another important aspect to know about Import Maps is scoping, which allows the use of multiple versions of the same package in a specified scope.

For example, the code below specifies that any module in /src path will use the https://cdn.skypack.dev/dayjs@1.10.7 package, while anything outside that path will use https://cdn.skypack.dev/dayjs@1.11.5.

<script type="importmap">
  {
    "imports": {
      "dayjs": "https://cdn.skypack.dev/dayjs@1.11.5"
    },
    "scopes": {
      "/src": {
        "dayjs": "https://cdn.skypack.dev/dayjs@1.10.7"
      }
    }
  }
</script>

This is useful in the case where you need to work with multiple versions of the same package simultaneously for various reasons,
such as compatibility with other libraries, ensuring legacy systems can still run and so on.

VS Code extension

Alternatively, instead of writing our import maps script, we could use the VS Code extension which can automatically generate, and inject import maps for modules and HTML pages.

Other Usages

There are also many other uses of import maps besides allowing bare module imports such as:

importing modules within modules;
allow extension-less imports;
remapping imports (mapping away hashes).

You may learn more about these useful features in the official documentation.

Browser Support

In terms of browser support, at the time of this article, it is supported in Chromium-based browsers only (i.e. Chrome, Edge). More of this information can be found on the image below:

Polyfills

If you want import maps to be supported in any browser, there is an ES Module Shims polyfill which is compatible with any browser that has baseline ES Module Support (i.e. Edge 17+, Firefox 60+, Safari 10.1+, and Chrome 61+).

To use it, simply add the polyfill script with an async attribute, and add an import map or module script below:

<!-- import polyfill here -->
<script async src="https://ga.jspm.io/npm:es-module-shims@1.5.16/dist/es-module-shims.js"></script>

<script type="importmap">
  {
    "imports": {
      "react": "https://ga.jspm.io/npm:react@18.0.0-rc.0/index.js"
    },
    "scopes": {
      "https://ga.jspm.io/npm:react@18.0.0-rc.0/": {
        "object-assign": "https://ga.jspm.io/npm:object-assign@4.1.1/index.js"
      }
    }
  }
</script>

<script type="module">
  import react from 'react';
  console.log(react);
</script>

The code example is taken from the shim repo.

Conclusion

In this article, we learned about import maps, what problems they can solve, and how to implement them. Thanks for reading, I hope it has been a helpful article in getting you started with import maps!

You can read more about import maps in the References section below. Please share this article if it is insightful! Cheers!

References

Originally published by Victoria Lo in Uploadcare Blog.

Basics of the Internet

Uploadcare — Wed, 28 Dec 2022 16:20:16 +0000

There is a great book called “High Performance Browser Networking” by Ilya Grigorik that contains basic knowledge about networking and data transfer optimizations. In this book author writes the following about CDNs:

Content delivery network (CDN) services provide many benefits, but chief among them is the simple observation that distributing the content around the globe, and serving that content from a nearby location to the client, enables us to significantly reduce the propagation time of all the data packets.

We may not be able to make the packets travel faster, but we can reduce the distance by strategically positioning our servers closer to the users! Leveraging a CDN to serve your data can offer significant performance benefits.

When you think about it, you probably agree with the fact that CDNs make the user experience better, because they speed up website loading. But... wait a minute! How does the CDN do it?

You might say, “Well, when a user loads files from CDN, they are loaded from the nearest server to this user. Easy-peasy.” Yeah, but how does it happen if we use the one and only domain name for the content?

In Uploadcare we offer our clients a CDN along with other optimizations, and we know a lot about the technologies that are used to make the content delivery fast, reliable, and secure. In this small series of articles we are going to share this knowledge with you and explain the magic behind the CDN.

Wiring the things up

Let us start with basic networking. Let’s say we have two computers that should communicate with each other:

The solution is simple — wire them and the job is done:

Yeah, it works. When computer A wants to send a message to computer B, it transfers the data through the port where the cable is connected to. It’s easy to detect when the communication channel is occupied, because there are only two users of this channel. There’s no problem with addressing due to the same reason.

However, this solution is hard to scale. What should we do when we have five computers instead of two? Sure, we can connect them to each other, but...

You see that communication quickly becomes a problem. It’s easier to summon Satan with these wires rather than build a clear network. Not only do we need a lot more cables now, but also each computer should have four ports. Moreover, when we want to add a new computer to this network, we have to find a way to add a new port to each existing computer! The topology of this network is way too complex.

In communication networks topology is the arrangement of the elements. There are physical and logical topologies. Physical topology is the placement of the various components of the network, while logical topology is how data flows within a network.

In our case, every device is connected to each other, which means that we have a fully connected network, physically and logically. To reduce the number of cables we may choose a different topology. For example, each computer may be connected to its neighbors on the left and on the right only:

This is called a ring network. As you see, this type of network does not require too many cables and each computer may have two ports only, but the tradeoff is the necessity of transferring data through the intermediate computers while trying to reach the destination.

Even though using intermediate computers isn’t that bad, there is a much more important flaw in a ring network. The connection relies on the availability of the nodes. If one node of the ring is broken, then the signal can’t be transferred properly.

There are different topologies which solve different problems. E.g. instead of using a ring network we might use one central wire to which all the computers are connected. That is, we would get a bus network:

Now our network does not rely on computers, but it will be broken if the central wire (usually called a bus, a backbone or a trunk) does not work properly. This type of network is harder to maintain, because you have to be able to check and repair every inch of the backbone.

To solve the maintenance problem described above, we may use a “helper device” which will receive data from a computer and distribute it through the network. The device is called a hub, while the topology of this kinds of network is called a star:

Even though this network may be broken by shutting down the hub, it’s much easier to maintain — instead of controlling a long cable, we have to control only one device. Also, it’s much easier to connect new nodes — just plug the cable into the hub, and it works.

In real life we use star networks at home. All the devices are usually connected to one router that receives messages and distributes them when necessary. The Internet itself is a hybrid network, which is combined from the myriads of small networks, each one of those may use any physical or logical topology.

The Internet hierarchy

Alright, we have a network at home that may look like this:

But what happens on the next level? What is the way of your request to Google? Let’s see.

First of all, your request will land on the computer that will process it — a Google server:

But the request doesn’t go straight to the Google server.

Your router connects you with a larger network owned by your Internet Service Provider (ISP). Same goes with the Google’s computer:

Each of those ISPs maintains its own network for many of the local networks, not only for yours and Google’s.

If you are connected to the same ISP as your Google server — you’re lucky, because the data may be transferred inside the provider’s network which is fast, secure and reliable:

But in the real world it is not possible to connect everyone using only one ISP. There are hundreds and thousands of providers. Each ISP has to communicate with each other to make the Internet work properly.

So, the closest to real life example may look like this:

You might see the problem here. If all the ISPs should connect to each other directly then we may encounter the same situation we got at the start of this article — tons of wire! That’s why they are connected using hubs called Internet Exchange Points, or IXPs.

Our network is already complicated, but there is one more problem — geography.

What if you live in Spain, while the Google computer “lives” in the USA? Well, they should be connected in some way, right?

This way is an underwater cable! You may think that this is a joke, but no, it is not. The continents are connected to each other using undersea cables. It may seem unreliable, and in fact sometimes it is. But it’s still the best way to connect everything:

Now it looks like we know how to connect two computers using the Internet. But here’s where things become really mad.

First, there are lots of ISPs.

Second, there are lots of IXPs.

Third, there are lots of undersea cables! Check the map of them dated 2021:

Finally, not every ISP is connected directly to IXP. Sometimes they are connected to “parent” ISPs.

Even this is not the end, the cables are different: some of them transmit the signal faster, others do it slower. What’s more important, some of them cost more than others to use. You don’t think that the connection between ISPs and IXPs is free of charge, do you? ISPs pay to each other and IXPs to use the network each of them have. Some providers charge more, others charge less.

That’s why the signal your computer sends usually does not go by the shortest path. The signal goes by the cheapest one.

Transmission medium

A transmission medium is a system or substance we use to transfer the signal. When devices are connected to each other using Wi-Fi, the medium is air. When your PC is connected to your router, the medium is usually a copper wire. The backbone of the Internet — the undersea cables — is made of optic fiber.

The medium defines the conditions in which the communication can be done.

For instance, copper wires are affected by any electrical noise. That’s why the twisted pair is actually “twisted” — to prevent electromagnetic interference. Such cables also may be covered with a special material which “shields” the transmitted signal.

At the same time, optical fiber does not care about electrical noise, because it’s just a “light pipe.” Even more, due to the techniques used to transfer the data, the speed of transmission may be much higher in optical fiber than in copper wire!

Now you may say, “The Internet is based on undersea cables and this is kinda sick. Why not use satellites if we can transmit data through the air?”

Sure, it is possible to connect computers using satellites. E.g. GPS uses satellites and it works. First of all, it’s cheaper to simply wire the computers up. Besides, the network laying on the ground (or in the sea) is much easier to repair.

Moreover, let’s do the quick math. Let’s recall the connection we described earlier. Let’s say that the orange dot was Nashville, USA, while the white dot was Zaragoza, Spain:

The distance between these cities is 7200 kilometers. If we connect them using a direct wire, the signal transmitted from Zaragoza will reach Nashville in 36 milliseconds, because the speed of light in optic fiber is around 200 000 km/s (it’s 1.5 lower than in the vacuum).

At the same time, there might be a satellite somewhere in the middle of the sky, between Zaragoza and Nashville. Historically, communication satellites were deployed on geostationary orbit, the altitude of which is 35 786 kilometers. Our signal should go from Zaragoza to the satellite and then from the satellite to Nashville.

Let’s omit trigonometry and say that the total distance is 2 × 35 786 kilometers, which is 71 572 km.

At the same time, the speed of the signal that goes to the satellite and back is approximately 300 000 km/s, because radio waves are used for transferring the signal. The total trip time is 238 ms.

So, the time that signal goes from Zaragoza to Nashville by direct wire is 36 milliseconds. For the satellite this time is 238 ms.

We have not said anything about the fact that all those intermediate points, such as ISPs, IXPs and the rest of them, slows the signal down, making the real transferring time bigger than 36 ms. But at the same time, we haven’t said anything about the quality of the signal, which is usually not so good for transmissions over the air.

So, you see why the world is still using cables. In the foreseeable future, the status quo may change, due to massive deployment of satellites (e.g. by Starlink) to a low Earth orbit. Lower altitude allows the satellites to maintain a connection with much less latency.

However, it’s highly unlikely that the current wired infrastructure disappears, which means that if you maintain an Internet service you have to use services like CDNs to serve data closer to users.

In the next article of this series we will see how the signal is encoded during the transfer and figure out how to intercept requests to inspect them.

Originally published by Igor Adamenko in Uploadcare Blog

Service workers tutorial

Uploadcare — Mon, 26 Dec 2022 00:19:12 +0000

Web developers or anyone involved in web development can tell you they know of “workers” in browsers. However, the problem is that most folks think of either web workers in general, or service workers (or even shared workers) when they say “workers.” These terms can be confusing, especially if you didn’t come across them.

That is why today, we will dive into service workers. We’ll see why you’d want to use service workers, what they are, how they work and explain the difference between service workers and other workers in the browser.

Before we start, let’s dive in and see what the web workers are in general.

What are web workers?

A web worker is a background task defined via script — a JavaScript file. Web workers can communicate back and forth with their creator. A web worker runs in a separate thread from the main JavaScript thread. To see how they work and get a small introduction to the world of web workers, let us go through a simple example.

You can create a web worker in the application code like so:

const myAwesomeWorker = new Worker('/path/to/worker.js');

Then, you can communicate with the worker:

myAwesomeWorker.postMessage('Hi there, love you');

Inside the web worker, you can accept the message and respond back to the caller:

// /path/to/worker.js

onmessage = function (e) {
  // do some magic here
  postMessage('I am all done, back to you, main application code.');
};

Then, the caller can listen to the messages from the web worker:

myAwesomeWorker.onmessage = function (e) {
  console.log('Message received from myAwesomeWorker');
  console.log(e.data);
};

Above is a trivial example of how web workers work, but it should be a great start to our topic where we will explore a specialized worker — service worker. To summarize, a web worker is a JavaScript file that runs in a different thread, but still can communicate with our application code. We can communicate with the web worker back and forth, and it can be a great helper when we want to delegate work away from the main thread of our application.

Now that we got over the basics, let’s see what service workers are.

What are service workers?

A service worker is a mediator between the web application, the browser, and the network. The service worker is like a proxy server that sits between your application and the network, intercepting requests and serving the corresponding responses.

You can also envision a service worker as a middleware for your site. Any request from the site and any response it gets goes through the service worker. Besides being a middleware, service workers also have access to a cache, where they can save responses and assets.

All this makes a perfect scenario if you’re planning to add exceptional offline support to your website. A service worker can decide if it should serve a resource from the cache or the network, as it would happen without a service worker. However, it is important to note that not all browsers support service workers. We should make service workers an enhancement rather than a requirement for our website.

One more use case can be the ability to use Push API to receive notifications from the server. This API provides the option for developers to deliver asynchronous messages and updates to users who opt in, resulting in better engagement with timely new content. For example, you can build a service worker that will update users when a new app is available.

OK, now that we understand the concept of service workers, let’s see what a typical lifecycle of a service worker is.

The service worker lifecycle

To fully understand how service workers behave, we must understand the different states they can exist in. To use a service worker, we have to register it on the client side. Here’s how we can do it:

if ('serviceWorker' in navigator) {
  navigator.serviceWorker.register('/myServiceWorker.js');
}

First, we do a simple check to see if service workers are supported in the user’s browser. Then, we call navigator.serviceWorker.register with the path to our service worker JavaScript file.

Then, we have to distinguish two use cases:

When we register a service worker for the first time
When we update the existing service worker

1. Registering a service worker for the first time

In the first scenario, where the page doesn’t have any service workers, the client will attempt to download the service worker we registered. After the download, the browser will try to install the service worker. After successful installation, the service worker is then activated and ready to perform.

To sum up the first case, the lifecycle looks like this:

Download (parse, and execute)
Install
Activate

2. Updating existing service worker

Now, in the second use case where we want to update the existing service worker, we might get into an extra state between installing and activating. If an existing service worker is available, the installation for the new worker is done in the background. Once the installation is complete, the new worker is not yet activated.

The worker now needs to wait for when there are no longer any pages loaded that are still using the old service worker. As soon as there are no more pages to be loaded, the new service worker activates (becoming the active worker). You can skip the waiting for the new worker using ServiceWorkerGlobalScope.skipWaiting().

So when the service worker is updated, this is how its lifecycle looks:

Download
Install
Wait
Activate

We talked about how to register a service worker, and how browsers later activate it. But how does the lifespan of a service worker end?

It’s important to note that service workers don’t live indefinitely. While exact timings differ between browsers, service workers will be terminated if they’ve been idle for a few seconds, or if they’ve been busy for too long. If a service worker has been terminated and an event occurs that may start it up, it will be restarted.

All in all, the service worker lifecycle can be shown in one diagram:

OK, now that we got through the basics on how to register a service worker and its whole lifecycle in the browser, let’s go on and see how it works. We’re going to create a service worker that caches our requests.

Using service worker to cache requests

One of the most common use cases for service workers is that they are used to cache assets and requests on your website. They are a fundamental part if you’re looking into making a Progressive Web App (PWA). As we mentioned before, when registered, the service worker acts as a layer between your website and the network. The worker intercepts every request from your website out to the world.

In addition to being a middleware between your website and the world, a service worker can utilize the Cache Storage API. In this example, we will use just that — a simple service worker that caches requests in the browser’s cache. This API works similarly to the browser’s standard cache, but it is specific to your domain. You have full control over the API, meaning you can control when the keys expire and add new keys. It’s important to mention that the service worker’s storage is independent of the browser HTTP cache to ensure we don’t mix the two.

Without further ado, let us go into how we can cache our resources with service workers.

Caching assets on service worker’s installation

Before our service worker is fired, we need to register it on a page. We will use a similar code we used in the lifecycle section:

const registerWorker = async () => {
  if ('serviceWorker' in navigator) {
    try {
      const registration = await navigator.serviceWorker.register('/worker.js');
      console.log('Service worker registration succeeded:', registration);
    } catch (error) {
      console.error(`Registration failed with ${error}`);
    }
  }
};

registerWorker();

First, we will check whether service workers are supported in the context where our code runs with 'serviceWorker' in navigator check. After that, we’ll do navigator.serviceWorker.register('/worker.js') which will register the worker.js file as our service worker. After that, we’ll write to the browser console, to ensure everything gets set up correctly. If there were an error somewhere, our try/catch block would take care of that.

Also, it is good to note that every service worker has a configurable scope. You can set scope when registering a script to be a service worker like so:

navigator.serviceWorker.register('worker.js', { scope: '/some-scope' });

If you omit scope, it will use the default value. The default value of the scope depends on where the service worker got registered. If you register a service worker under domain.com/index.html, the worker will control domain.com/index.html and all pages underneath. If you change the scope to something else like below:

navigator.serviceWorker.register('worker.js', { scope: '/blog/' });

And, if you register the service worker in the domain.com/index.html, then it will control only the domain.com/blog/ portion of it.

Great, now that we registered our service worker and understand the concept of a worker’s scope — let us define some logic in the service worker file worker.js. Here’s how it will look:

const version = 'v1';

const addResourcesToCache = async (resources) => {
  const cache = await caches.open(version);
  await cache.addAll(resources);
};

self.addEventListener('install', (event) => {
  console.log(`${version} installing...`);

  event.waitUntil(
    addResourcesToCache([
      '/',
      '/index.html',
      '/styles.css',
      '/script.js',
      '/jungle.png',
    ])
  );
});

At the top, we first declare the version to be v1 — we’ll get to this in a second. Then, we define the addResourcesToCache function that opens a Cache Storage instance and writes to it with:

const cache = await caches.open(version);
await cache.addAll(resources);

The caches.open receives a string that represents a cache name. Later on, as we change our service worker, we will have to change the name of the cache and remove the old one, but we’ll get to that in a second.

So if we open our page now, the service worker defined in worker.js will install and cache our resources. To be sure it happened, we can check the “Applications” tab in Google Chrome. If we open the “Storage” section there, we will see how much storage our website is taking:

Also, we can go into “Cache Storage” and check the v1 cache we just created in our service worker. In “Cache Storage,” you should see the resources we specified:

Cool, we got our website’s assets in the service worker cache, safely away from the standard browser cache. But, what now, you must be asking? Now, we need to tell the browser to serve those assets from the cache. And a great way to do that is to use our service worker.

Serving assets from the cache

We will listen for a fetch event in our worker, and intercept requests going from our website to the network. Here’s how it looks:

self.addEventListener('fetch', (event) => {
  event.respondWith(caches.match(event.request));
});

With this code, we are hijacking each request firing from our site. Then, we call respondWith and respond with a cached value in our cache. If we add this code, close and then reopen our website, and open the “Network” tab in a browser. We will see that the resources are cached in by the service worker:

This is all fine and dandy, but if we add a resource to our site or remove a resource from the cache manually, our site can break.

For example, I tried to remove styles.css from the cache in “Application” → “Cache Storage,” and the CSS went missing the next time I opened the page. Neither worker nor the browser fetched styles.css again because of the logic in the worker. To mitigate this, we have to make sure that a request gets passed through our service worker if there’s no matching resource in our cache. Here’s how we can do that:

const cacheFirst = async (request) => {
  const responseFromCache = await caches.match(request);

  if (responseFromCache) {
    return responseFromCache;
  }

  return fetch(request);
};

self.addEventListener('fetch', (event) => {
  event.respondWith(cacheFirst(event.request));
});

With the new cacheFirst function, we are sure that a request that is not in the cache will get sent out to the network with fetch(request).

We can go a step further, and cache requests that are not in the cache. That way, we will fetch a resource from the network, and store it into the cache. If our app goes offline for some reason, that resource will still be available, yay! This is what our fetch event handler should look like:

const putInCache = async (request, response) => {
  const cache = await caches.open(version);

  if (request.method !== 'GET') {
    console.log('Cannot cache non-GET requests');
    return;
  }

  await cache.put(request, response);
};

const cacheFirst = async (request) => {
  const responseFromCache = await caches.match(request);

  if (responseFromCache) {
    return responseFromCache;
  }

  const responseFromNetwork = await fetch(request);

  // we need to clone the response because the response stream can only be read once
  putInCache(request, responseFromNetwork.clone());

  return responseFromNetwork;
};

self.addEventListener('fetch', (event) => {
  event.respondWith(cacheFirst(event.request));
});

Here, we define the putInCache function that puts the request and its response into the cache.

If you notice, we also check for a request method. By design, we cannot cache non-GET requests in the Cache Storage API. After that, we changed the cacheFirst function to call the putInCache function with the request object and the clone of a response. We have to clone the response, because request and response streams can only be read once. Then, we return the original response to the browser.

And that is it, this is the basic service worker that will cache all GET requests that go from your website to the network. This setup can be a great first step if you plan to make your app work offline. You can play around with different caching strategies, let’s quickly go over them.

Cache-First

We implemented here a cache-first approach, where we first try to serve a request from the cache if it is available.

Network-First

You can try and implement a network-first approach, where we normally send out the request to the network and cache it. Then, when the request can’t be made (the site lost connectivity, for example) — we serve the failed request from the cache.

Stale-While-Revalidate

There’s also a stale-while-revalidate approach that will use a cached version if available, but it will fetch an update and put it in the cache for next time.

We could also go on and make sure that a user gets a fallback response, in case the resource is not in the cache and it can’t be fetched over the network. But this should be enough to get you started on a service worker. Instead, I’d like us to compare the difference between service workers and workers in general. Let’s dive into that in the next section.

Service workers versus other workers

Now that we have an understanding of service workers, we can look over other types of web workers, and how they differ from each other. Let’s start with how similar web workers and service workers are.

Web workers and service workers are two different types of background scripts available to websites in the browser. Both web workers and service workers have some things in common:

They both run in a different thread without blocking the main JavaScript thread and the user interface.
They can’t interact with the DOM directly, and they have limited access to browser APIs.
They both are web workers. But service workers are just a specialized version of web workers.

And the differences between them:

As we mentioned before, service workers allow you to intercept network requests (via the fetch event), and to listen for Push API events in the background (via the push event). Web workers can’t do that.
A page can spawn multiple web workers, but only one service worker controls all the active tabs under the scope it was registered with.
The lifespan of the web worker is tightly coupled to the tab it belongs to, while the service worker’s lifecycle is independent of it. When you close the tab where a web worker is running, the worker will be terminated. A service worker can continue running in the background, even when the site that registered it doesn’t have any active tabs open.
Service workers only run over HTTPS for security reasons. Being able to modify network requests, they are wide open to man-in-the-middle attacks, which is why they are only allowed on secure connections. In Firefox, Service Worker APIs are also hidden, and cannot be used when the user is in private browsing mode. There’s an open bug for that if you want to track it.

There are also shared workers. They are workers that can be utilized by multiple scripts running in different windows, iframes, and similar, as long as they are in the same domain as the worker. The scripts must communicate via an active port, so they are more complex than the standard workers or service workers.

Besides these, you can also utilize worklets. A worklet interface is a lightweight version of web workers, and gives developers access to low-level parts of the rendering pipeline. Worklets can be used to run JavaScript and WebAssembly code to perform graphics rendering or audio processing, where high performance is required.

Summing up

Phew, what a ride. We learned the basics of service workers, but let’s go over it once again. Here are things to remember:

A service worker is a specialized worker that performs away from the main JavaScript thread.
A service worker can act as middleware between your site and the network.
To have a service worker on your website, it has to be first downloaded, installed, and then activated.
If you’re updating an existing service worker, the previous service worker must be unloaded from the page (or you can instead skip waiting for that).
Service workers only run over HTTPS for security reasons.
One of the main use cases for a service worker is to cache resources.

As far as web workers are concerned, they are mostly used to delegate work from the main thread while the page is open. As soon as the page is closed, the classic worker is terminated as well.

If you’re interested in the code from this blog post, I pushed it to a repository on GitHub. You can check it out there, it has a small index.html page that showcases how you can cache resources.

That’s all folks, next time, we will go into how to debug service workers (even though we showed a glimpse of where you can do that), and we can show how to use service workers in some other use case.

Until then, thanks for reading, and catch you in the next one.

Originally published by Nikola Đuza in Uploadcare Blog

Creating responsive images with image-set

Uploadcare — Thu, 22 Dec 2022 00:05:19 +0000

image-set is a CSS function that allows you to make an image responsive, showing variations of the same picture for different screen resolutions when needed. This function is almost the same as the HTML srcset attribute for img tags.

What are responsive images?

Simply put, images are responsive when they display the information they are intending to regardless of the screen size or pixel resolution. This is to say; these images automatically adjust themselves to fit the size of the container they’re placed into, and load according to the resolution of the user’s screen. This process improves the user experience, as images load faster and look crisper to the human eye.

Why responsive images?

Why not? Here are three essential reasons why we should always make our web images responsive:

To render high-quality images on different devices. Also known as resolution switching.
To load the right images on different screen sizes. Also known as art direction.
For faster loading of web pages.

Let’s discuss a little about each of them.

Resolution switching

A digital screen uses pixels to display images. These pixels are arranged horizontally and vertically in a grid. The number of pixels both horizontally and vertically, is known as the screen resolution.

For example, a “screen with a 1024×768 resolution” means that the screen has 1024 pixels horizontally, and 768 pixels vertically. An image with a 1024×768 resolution also has 1024 pixels horizontally and 768 pixels vertically, making them have the same width and height.

If we insert such an image into the page, and set its width and height to 100% of the user’s screen, it will be nice looking on 1024×768 screen. However, the same image would be stretched on a screen with a 2048×1536 resolution, because the screen’s width and height are higher, and we would be showing fewer image’s pixels in a larger space. So on a smaller screen, the image will look sharper because you are showing many pixels in a smaller space.

Image pixel resolutions also partly determine the image size, and the size of an image can affect how fast a browser loads a web page and the amount of data the user uses to download it. For example, a 150×100 image worth 17.7 Kb would load faster, and save internet data than a 3500×2321 image worth 2.4 Mb.

Therefore, smaller images should be used on smaller screens where possible. So, we should somehow switch image sources to adapt the image to the current user’s screen.

Here’s a live example and its source code. Feel free to view and play with them for a more practical understanding of resolution switching. Continue reading to understand how this demo works.

Art direction

We know different images are displayed differently on various screen sizes. However, art direction focuses on the details of the image content shown to the user, not the image resolution. Art direction might not matter for images that don’t pass any information to the user, but it counts for images that give the user information.

For example, we can have a landscape orientation image with a person standing at the center of the image as the primary focus. We can easily view the person’s face on large screens. However, if we let the image have the same aspect ratio and orientation on smaller screens, we won’t be able to view the person easily,
let alone their face.

But if we crop the image, thereby changing its aspect ratio and letting it have a portrait orientation, we get to view the person clearly on smaller screens. But when we switch back to larger screens with the new image, it becomes too big.

Art direction ensures that regardless of the screen size, the user has smooth access to whatever details or content they are supposed to get from an image.

For a more practical understanding of art direction, you can view the live example and play with the source code.

Speed up loading of web pages

Nowadays, people tend to access the Internet through their mobile devices, so it is essential to consider this when adding images to your web pages.

Users should not have to download overly large images for their mobile screens, or wait for larger images to load if they have slow Internet. Instead, you should be able to provide the images suitable for their screen sizes and Internet connection. This customization reduces the time it takes for the web pages to load, the user’s bandwidth and money, and overall improves the user experience of your website.

So, how to implement all those features?

Use image-set function

As I said earlier, the image-set is a CSS function that allows web developers to turn an image responsive, showing variations of the same image for different screen resolutions when needed.

The idea is simple. We list a set of images within the image-set function. The browsers have full access to pick the most appropriate image from this list for different screen resolutions,
with the assurance that the browser would make the best decision about which one to use.

How to create responsive images with image-set

Syntax

image-set() = image-set(<image> [<resolution>] [<type>])

The <image> represents the URL to where the image is stored. It is required.

The <resolution> represents resolution units, x or dppx, for dots per pixel unit, dpi, for dots per inch, and dpcm for dots per centimeter. Every image within an image-set must have a unique resolution. The parameter itself is optional, because it’s possible to omit it for the one “default” image of the set.

The <type> specifies the image’s file format — JPEG, PNG, GIF, AVIF, etc. It is optional.

Example 1

.picture {
  background-image: image-set(
    url('small-car.jpg') 1x,
    url('large-car.jpg') 2x
  );
}

In Example 1 1x identifies the image the browser should use if the image resolution is higher or matches the screen, making the pixel density equal to 1. If the image resolution is lower than the screen’s, the 2x image should be used. You can add more if you want to.

Don’t forget to check current browser support to add vendor prefixes if your project needs them. Also, a fallback can be added, so browsers not supporting image-set can have an image to show. We add the fallback before the background-image property with the image-set function.

.picture {
  background-image: url('small-car.jpg');  /* fallback */
  background-image: image-set(
   url('small-car.jpg') 1x,
   url('large-car.jpg') 2x
  );
}

Example 2

.picture {
  background-image: image-set(
    url('car.avif') type('image/avif'),
    url('car.jpg') type('image/jpeg')
  );
}

For Example 2, we have specified two format types for the same image; AVIF and JPEG. The widely supported format should always come last, in this case, the JPEG, while the newer formats should go before it.

The browser will download only one image. The browser checks the first image specified and uses this image if it supports this one. If the browser doesn’t, it ignores the image and moves on to the next.

Example 3

.div {
  background-image: image-set(
    url('small-car.avif') type('image/avif') 1x,
    url('large-car.avif') type('image/avif') 2x,
    url('small-car.jpg') type('image/jpeg') 1x,
    url('large-car.jpg') type('image/jpeg') 2x
  );
}

For Example 3, we specify the resolution along with the image format. This gives the browser even more variations on which image to use.

How does the image-set function differ from the srcset attribute?

Although the image-set function, and the srcset attribute set out to do the same thing, there are a few differences between them.

Obviously, the significant difference is that image-set function can be used only in CSS, while srcset attribute can be used only in HTML.

Another difference, is that unlike with the srcset attribute, where you can also give image options based on the width of the browser (when art direction is required), you cannot do that with the image-set. However, art direction can still be achieved in CSS using media queries!

View the live example on responsive images using the resolution switching method, and the art direction method and play with the source code.

Conclusion

Responsive images are vital in creating a good user experience, and building faster web pages. Now, you better understand what a responsive image should be, and how we can make better-performance images with the image-set CSS function.

Originally published by Pelumi Akintokun in Uploadcare Blog

ECMAScript modules. Going beyond the ordinary

Uploadcare — Tue, 22 Nov 2022 16:35:13 +0000

Let’s say you’re working on a static site generator, and while implementing new features you decide to import text files (HTML, MD, CSS, SVG, etc) as dependencies into JavaScript files. Sure, you may use a bundler with some loader, but what if your bushidō mandates you not to type npm install something every time you have to solve a problem? Well, there is a solution.

But first, make sure that:

You know what ESM is about.
You have Node.js installed, and this Node.js is not outdated.
You’re not using .mjs as a file extension for your JavaScript code, browsers do not like them, even though the syntax of the files are the same as .js. However, it’s important only if you want to write isomorphic code.

Now, let’s continue.

Exercise one. Node.js. Custom loader

Create a loader:

// loader.js

import { URL } from 'url';
import { readFile } from 'fs/promises';

function isAllowedURL(url) {
  return ['.html', '.htm', '.md', '.css', '.svg', '.json'].some(x => {
    return url.endsWith(x);
  });
}

export async function load(url, context, defaultLoad) {
  if (!isAllowedURL(url)) {
    return defaultLoad(url, context, defaultLoad);
  }

  const content = (await readFile(new URL(url))).toString();
  const json = url.endsWith('.json') ? content : JSON.stringify(content)

  return {
    format: 'module',
    source: `export default ${json};`,
    shortCircuit: true,
  };
}

Then create a file where you want to import some non-JS stuff:

// app.js

import html from './index.html';
import doc from './doc.md';
import css from './styles.css';
import svg from './image.svg';
import data from './data.json';

console.log(html, doc, css, svg, data);

Now, boom:

$ node --loader ./loader.js ./app.js

Huh?

import { URL } from 'url';
^^^^^^

SyntaxError: Cannot use import statement outside a module

Oh, yeah, don’t forget to set "type": "module" in the package.json of your project — otherwise imports don’t work.

But with this small key in the package.json everything just works, without the need for additional packages, bundlers, etc.

As you see on the screenshot, you may get this warning:

(node:22662) ExperimentalWarning:
Custom ESM Loaders is an experimental feature.
This feature could change at any time

That’s because the feature is still experimental, but, well, it works.

If you want to read more about plans of Node.js team related to loaders, check nodejs/modules#351 issue.

Exercise two. Browser. Request interception

Now move to a browser. The task remains the same. We want to load modules, but not JS modules, rather we want static files with styles, templates and plain text. Browsers do not support special “hooks” for ES Modules. What’s the plan?

Well, let’s create a dragnet using Service Workers.

The tools are different, but the approach is the same.

Create a Service Worker file:

// sw.js

function isAllowedURL(url) {
  return ['.html', '.htm', '.md', '.css', '.svg', '.json'].some(x => {
    return url.endsWith(x);
  });
}

async function handleRequest(req) {
  const content = await (await self.fetch(req.url)).text();
  const json = req.url.endsWith('.json') ? content : JSON.stringify(content);

  return new Response(
    `export default ${json};`,
    {
      headers: {
        'Content-Type': 'text/javascript'
      },
    },
  );
}

self.addEventListener('fetch', e => {
  if (e.request.destination === 'script' && isAllowedURL(e.request.url)) {
    e.respondWith(handleRequest(e.request));
  }
});

<!-- index.html -->

<script>
  navigator.serviceWorker.register('./sw.js');
</script>

Use the same imports as in the Node.js example above:

// module.js

import html from './index.html';
import doc from './doc.md';
import css from './styles.css';
import svg from './image.svg';
import data from './data.json';

console.log(html, doc, css, svg, data);

But this time, instead of marking package.json as a “module,” do the same on the script tag:

<!-- index.html -->

<script src="./module.js" type="module"></script>

Now serve the page, open DevTools and check the logging:

No bundlers, no additional stuff. Just pure JavaScript and browser API. Cool, isn’t it?

If you decide to use this solution in a real project, read Service Worker’s docs carefully. There are some important limitations and nuances of compatibility with different web browsers. For instance, currently Firefox does not run Service Workers in private browsing mode. This is by design, but probably will be changed in the future.

Exercise three. Node.js + Browser. Request parameters

Okay, now it’s time for sick stuff. It’s just an experiment and I don’t encourage you to try it, but why not try when you can?

If you don’t know, ES Modules support top level await. It means that before a module “returns” its export, you can do something asynchronous in it. For instance, make a network call and get a response. Where does this lead us? Right.

Create a loader module:

// load.js

let result = null;

const path = import.meta.url.split('?')[1];

if (path) {
  let content;

  if (typeof window === 'object') {
    content = await (await fetch(path)).text();
  } else {
    const fs = (await import('fs')).default;
    content = fs.readFileSync(path).toString();
  }

  result = path.includes('.json') ? JSON.parse(content) : content;
}

export default result;

Import everything else using this module as a proxy:

// module.js

import html from './load.js?./index.html';
import doc from './load.js?./doc.md';
import css from './load.js?./styles.css';
import svg from './load.js?./image.svg';
import data from './load.js?./data.json';

console.log(html, doc, css, svg, data);

Even though the example is simplified, it’s already isomorphic. Which means you can use the code above in a browser and in Node.js without any changes. Try it.

I don’t think that anyone will seriously use this method to load modules, but the approach itself is at least interesting to think about.

An important thing to note here is that each module’s exports are cached. The cache is based on the module’s URI. Therefore, if the imported files are changed dynamically, their URIs should be changed too. That means that it’s possible to use dynamic imports only in this kind of situation. But when we have a top level await support, it’s not a huge problem.

Conclusion. Kind of

You may ask, “Is there any value in this at all? It’s easier to use fs.readFile or fetch instead of importing a module!”

I would like not to answer. You never know when the weird knowledge you have will help you in the future.

Anyway, I hope it was a fun ride! Maybe you’ve learned something new and useful.

Use more ESM and less CJS. Peace!

Originally published by Sergey Matiyasevich in Uploadcare Blog

A guide to Intersection Observer

Uploadcare — Mon, 21 Nov 2022 19:06:27 +0000

At some point during your frontend career, you probably had to or wanted to, figure out where an element is related to the scrolling position of the user. If not, don’t worry, I believe this will still be interesting to you, because it might pop some ideas and give you inspiration for your projects. Today, we will go through the Intersection Observer API — an easy way to determine whether the DOM element somehow intersects with the viewport, or another element.

Why Intersection Observer?

Throughout the frontend history, detecting when something is intersecting with another thing was tricky. Often, hand-crafted solutions were unreliable, to the point it caused some visible impact on users’ browsers and websites. Today, we are lucky to have the Intersection Observer API, but why? What were the main drivers behind this development? Let’s look at some examples of how things were done before.

The lazy loading technique is one example of where developers want to know when a DOM element is entering the viewport. To lazy load an image, you have to know whether a user is relatively close to the image position on the page as they are scrolling. You could achieve this by hooking up the scroll event, and calling getBoundingClientRect method.

The getBoundingClientRect would return an object providing information about the size of an element and its position relative to the viewport. Then, you could figure out whether to trigger the download of the image your user wants to see. Or, you could trigger a request to notify that the user just viewed an image or specific element on the page.

This solution was used over and over in the past. However, this solution was often sluggish because calling getBoundingClientRect forces a reflow of the page. The reflow is a process when the browser needs to re-calculate the position, and dimensions of the elements on the page. If called often, browsers and computers can only take so much heat, and will eventually start to become laggy.

Here’s a small Codepen example of how you can tap into the scroll listener and get the data about the desired element:

For this exact reason, the Intersection Observer API was introduced in Chrome 51. What makes this API better than the approach with the getBoundingClientRect, is that it provides a way to asynchronously observe changes in the intersection of a target element with another element. That way, when you use Intersection Observer, you’re not stressing the browser and the page, as much as you are with the getBoundingClientRect approach.

Awesome, now that we went through the backstory of how Intersection Observer came to be, let’s dive in and see how we can use it properly.

How to use the Intersection Observer?

To get the best idea of how to use an intersection observer, let’s dive right into how to call it in code:

const options = {
  root: document.getElementById('some-id'),
  rootMargin: '0px',
  threshold: 1.0,
};

const observer = new IntersectionObserver(callback, options);

The IntersectionObserver constructor receives two arguments:

callback — a callback function that gets called whenever a threshold is crossed in any direction.
options — an object with additional configuration.

The options object contains following fields:

root — it pinpoints an element that serves as a viewport, and must be an ancestor of the target element (we will show later how to target an element). If you leave this option out, the observer will default to the browser viewport.
rootMargin — it tells the observer when to trigger the intersection between the root and the target element. Here, you can put values like you usually do for a margin in CSS.
threshold — this can be a single number, or array of numbers. The threshold indicates at what percentage of the target element’s visibility you want the provided callback to get called. So, if you want to lazy load an image only when a user scrolls 25% of it, then you can put the value of 0.25 there. If you want some logic to be triggered, both at 25% and 50% of visibility of the target, then you put [0.25, 0.5] array here.

Cool, now we know what the constructor receives, but how do we actually target an element to be observed? Let’s find out through an example:

const target = document.getElementById('target-element');
observer.observe(target);

Nice, now when our element with the ID target-element intersects with the root element we set, the callback will get called. Naturally, we should look into how to write a proper callback for the Intersection Observer:

const intersectionCallback = (entries, observer) => {
  entries.forEach(entry => {
    // you can find out easily if target is intersecting by inspecting `isIntersecting` property
    if (entry.isIntersecting) {
      console.log(`Hey, I'm intersecting`)
    }
  });
}

The callback receives a list of entries that are of IntersectionObserverEntry type. Having a list as the first argument means that one observer can observe multiple target elements. Each entry has a set of fields, but one that’s most often used is the isIntersecting. In the example above, we log to the browser console whenever our target is intersected.

The second argument passed to the callback is the observer itself. This can be useful when you want to stop observing the target after an intersection — then you call observer.unobserve() in the callback.

Great, now that we went through some basics on using the Intersection Observer, let’s put it to use and build something.

Using Intersection Observer

Let’s build a working example where a video starts playing as you scroll into it, and it pauses as you scroll away. To achieve this, we’ll use the powerful Intersection Observer we just showcased in the section above.

To start off, we’ll need a video inside HTML. I will share the important HTML, and full JS code if you want to try it out. There will also be an interactive demo you can try out at the bottom. Here’s the simplified HTML:

<video muted controls>
  <source
    src="https://ucarecdn.com/33fa1b5d-a164-4178-908c-5a51f872fcef/video.webm"
    type="video/webm"
  />
  <source
    src="https://ucarecdn.com/1b63a65c-7796-4b23-a6fc-bb751f1221ed/video.ogg"
    type="video/ogg"
  />
  <source
    src="https://ucarecdn.com/ec3f39c9-be9f-4231-a4db-d7fcbd209e71/video.mp4"
    type="video/mp4"
  />
</video>

In the HTML we simply put a video tag with several sources. And, finally, our Intersection Observer code:

let video = document.querySelector('video');

let observer = new IntersectionObserver(
  (entries, observer) => {
    entries.forEach((entry) => {
      if (entry.intersectionRatio !== 1 && !video.paused) {
        video.pause();
      } else {
        video.play();
      }
    });
  },
  { threshold: 1 }
);

observer.observe(video);

Here, we are getting the video element using document.querySelector('video'). Then, we define our Intersection Observer instance with a callback and a threshold of 1. The threshold of 1 means that our callback will trigger when the video is fully visible, or it stops being fully visible.

In our callback, we check whether the video is not fully visible, with the entry.intersectionRatio !== 1 check. The intersectionRatio indicates which portion of the element is visible, and it goes from 0 to 1 — where 1 means it is fully visible. Everything between 0 and 0.99 means that the element, video element in our case, is hidden a bit. There, we also check whether the video is paused. If the video is playing and it is hidden a bit — we pause it.

Now, if the video’s intersectionRatio is 1, we fall into the else branch and we play the video.

This is a cool trick you can try out on your project. It doesn’t require a lot of code, and can be a good user experience. But remember to not autoplay videos by default — it can be annoying to users. Here’s a demo you can play around with:

Summing up

I hope you had a great time reading, and trying out the Intersection Observer API. If you learned a thing or two, then I call this blog post a success. Let’s go over the things we learned today:

Intersection Observer can be used to detect intersections between elements, or between an element and the viewport.
You use the API by instantiating the observer with new IntersectionObserver(callback, options).
The callback parameter is a method that receives an array of entries, and the observer itself.
The options parameter is an object that can consist of root, rootMargin, and threshold properties. This parameter is optional.
The instantiated observer can observe an element you pass to it with observer.observe(element).

That’s it! You’ve learned the basics. If you’re looking for something more advanced, you can always play around with the threshold option that Intersection Observer receives. Or, you can dabble with the IntersectionObserverEntry object for extra information when the intersection happens. If we tried to cover that, this blog post would be a lengthy one, so we will save it for another time.

Until then, thanks for reading, and catch you in the next one.

Originally published by Nikola Đuza in Uploadcare Blog

Different Types of Video File Formats and Why WebM and MP4 Beat Them All

Uploadcare — Tue, 19 Apr 2022 05:00:10 +0000

Which video formats are the best in terms of size/quality and are compatible with most players on the web, Android & iOS? The short answer for 2021 is: WebM backed up by MP4 is the winning stack to consider. Glance through this article for all the why's and get some encoding automation options.

💡 This content will be helpful for developers who are working on video sharing apps, CMSs, and ecommerce platforms. We didn’t cover the part about streaming in much detail, but ping us in the comments if you’re interested.

Video File Formats, Codecs, and Containers: What’s the Difference?

You’ve probably come across such acronyms as AVI, MPEG-4, WMV, DivX, WebM, or MP4. For non-savvy users, they stand for different types of video files. But the truth is, some of them are indeed file formats defined by their multimedia containers, and others are codecs that help to shrink video and audio data inside those containers.

There are three main reasons to start differentiating formats, codecs, and containers:

Video file formats and extensions don’t tell much about file characteristics. Containers and codecs do.
Containers and codecs define whether a file will be played on a particular device.
Containers and codecs influence how the data is stored and compressed, hence have a direct impact on video size and quality.

Let’s look at video file formats, containers and codecs from a developer’s perspective and distill their main characteristics.

What’s a multimedia container?

The word container speaks for itself. It’s like a bucket that may contain:

Encoded video and audio tracks (can be several in different languages)
Subs and interactivity elements like chapters and menus
Synchronization information to make it all play together
Metadata for SEO purposes

'Video containers are also called video wrappers and can store various types of data.'

Each multimedia container stores different types of data and is compatible with particular codecs and players. Here is an overview of the containers you’ve probably heard about:

Container	Video file extensions	Perks	Limitations	Quality and compression	Best for
Audio Video Interleave, AVI	.avi	Supports many different codecs, can run both on Mac and Windows High quality of video	No video chapters or interactive menus Doesn’t support some modern compression techniques	Low degree of compression and relatively large file sizes with possible quality losses	DVD
MPEG-4 Part 14, MP4	.mp4	Browser compatible Was designed for low bit rates and is bandwidth friendly Supports chapters and interactivity	Relatively high loading time Losses of video and audio quality after compression Codec dependent Non loyalty-free	High degree of compression and smaller file sizes	Web, mobile, social media, streaming
Matroska	.mkv .mk3d	Supports all codecs Open-source and loyalty free	Is not natively supported on mobile devices	Different levels of lossy and lossless compression	DVD, Web
Ogg	.ogg .ogv .ogx	Open-source and free Is designed to provide efficient video streaming	Not supported by Safari and Internet Explorer Codec dependent	Lossy and lossless compression, relatively small file size	Streaming
QuickTime File Format	.mov .qt	Good for video editing (does it in place, without rewriting the data)	Is supported by QuickTime Player and just a few other players	High quality with substantial file sizes	Apple devices
Flash Video, FLV	.flv	Supports all types of interactive content	No chapters or interactive elements Not supported by iOS or web High video making cost	No severe loss of quality, relatively big file sizes	Web, Interactive content
WebM	.webm	Open source and loyalty free Based on Matroska container profile and supports most codecs	Relatively low video quality No support for some browsers and devices No interactive elements	High video quality with relatively small file sizes	Web

What’s a codec?

Codecs compress (en*code) or decompress (dec*ode) video and audio files. As video data is always massive, codecs shrink it to make storage and delivery easier.

Video content is booming these days, so it’s hard to underestimate the role of codecs: thanks to encoding and decoding, anyone can create or watch videos all around the globe.

“Imagine that all image files have to be put in an archive. You start emailing people your photos inside .zip or .rar files, in accordance with this new law. Although both WinZip and WinRar have different features, at the end of the day, it's either a JPEG, PNG, GIF or BMP inside that file and that's what really makes the image. You'd say "The codec is JPEG, in a ZIP container.“

– a simple explanation by a Reddit user.

👉 Learn about the Most Common Image File Extensions

What codecs actually do is:

Compress a video file and make it fit the multimedia container format
Decompress the file and make it ready to play if the destination device supports the codec

Here is a list of popular codecs with supported multimedia container formats.

Some popular video codecs	Highlights	Supported container formats	Best for
H.264/AVC	The most commonly used nowadays, supports video streaming and is also used on cable broadcasting.	MP4 MOV F4V 3GP TS	Online HD videos and streaming. H.265 is the best in terms of video quality. H.264 is better for streaming thanks to high compatibility.
H.265/HEVC	Offers 50% better compression than H.264/AVC. Supports 8K resolution.	TS MP4 3GP MKV
AV1	Open-source and loyalty-free codec, competing with H.265.	MP4 3GP MKV	Those who need H.264 and H.265 for free.
VP9	A free codec developed by Google. Supported by Android, Chrome, and YouTube but not supported by Apple.	WEBM MKV	Audiences on Google, Android, or YouTube.

MPEG-4 and DivX are legacy codecs that can still be seen nowadays but are outshined by their widespread competitors, H.264 and H.265. They already lag behind them in terms of efficiency and mobile delivery.

💡 Codecs and containers are frequently confused. I bet you can’t tell the difference between MPEG-4 and MP4 even after reading the paragraphs above. Cheat sheet: MPEG is a compression method, and MPEG-4 is one of its versions which is designed for low bandwidth audio/video encoding. MP4, abbreviated from MPEG-4 Part 14, is a media container.

If you see an .mp4 file, it doesn’t mean it was encoded with the MPEG-4 codec. As the MP4 container is compatible with a number of popular codecs, it can be any of them. On the other hand, videos encoded in MPEG-4 can be a part of any supported container like Matroska or Ogg.

What’s a video file format?

A video file format is one that stores digital video data. As we already know, the way the data is stored is defined by the multimedia container; that’s why video file formats and extensions are usually named after them, causing lots of confusion.

Here’s a list of the most used video file formats with their corresponding containers, codecs, and file extensions.

Video file format	Video filename extension	Multimedia container	Video codec
Matroska	.mkv	Matroska	any
Ogg Video	.ogv .ogg	Ogg	Theora, Dirac
WebM	.webm	WebM based on Matroska	VP8, VP9, AV1
MP4	.mp4 .m4p (with DRM) .m4v	MPEG-4 Part 14 (MP4)	H.265, H.264, MPEG-4 Part 2, MPEG-2, MPEG-1

And here is how it looks in the real world:

How to Pick Video File Formats and Why WebM and MP4 Are Worth Considering

When choosing from different types of video file formats, there are four things you should think about:

What kind of players and devices should the video be compatible with?
Does the size matter, and how much video data are you ready to store?
What level of video playback quality do you expect?
Will you need the video to be livestreamed?

If you need a video file format that’s compatible with most devices, platforms, and browsers, including low bandwidth cases, look at MP4. It provides the best balance between quality and size, is compatible with several of the most popular codecs, and is widely supported.

💡 To further improve web performance, encode MP4 videos to WebM and use them together. WebM files are smaller in size and are the best fit for most web browsers. Force WebM every time a browser supports it, and if not, fall back to MP4. This can be done via video source tags. Check out this setup guide from Google.

Video File Processing, Reformatting and Automation

To prepare your video files for the web or mobile apps, convert them to the formats you chose. The best practice is to use FFmpeg. It’s a free and open-source tool designed for command-line-based video and audio file processing. It can easily convert your files to WebM and MP4 containers and even allows you to specify codecs.

Apart from reformatting files and choosing codecs, you might want to process them to fit your website. Such processing can include: cropping, trimming, adjusting resolution, and generating thumbnails.

All these operations can be performed on autopilot if you adopt Uploadcare, an out-of-the-box infrastructure for file uploads, processing, and delivery. Apart from reformatting files to web-friendly WebM, Ogg, and MP4, Uploadcare lets you standardize video assets in terms of size and quality. In fact, it’s a single robust API that allows you to:

Run URL-based video transformations in the browser
Set up a background workflow with automated delivery
Process and standardize user-generated video content

👉 Learn more about image and video transformations with Uploadcare

Encoding and Handling User-Generated Video Content

Let’s say you’re building a platform for pasta lovers who can share their favourite recipes or host live workshops. Asking users to upload video files of certain types can create friction. On the other hand, delivering files as they are can affect your web performance at low bit rates and grossly inflate traffic costs.

That’s why the best practice is to receive files as they are and encode them programmatically upon delivery. This can be done by setting up file encoding functionality, but that’s obviously not the best option in terms of time and money. Instead, you can take a buy-don’t-build approach and use a third-party image CDN with video encoding support.

Uploadcare has a built-in image CDN that takes on all the video handling, including video storage, editing, and reformatting, so you can focus on your core product.

Case study: Polished is a marketplace that offers job seekers to get video feedback from hiring professionals. The team had to implement video uploading, processing, and delivery to get the service up and running.

If done in-house, such infrastructure would’ve taken considerable time and resources, so they integrated Uploadcare: an out-of-the-box API-based solution. As a result, they launched the product in just two months and now handle video files up to 400 MB with no sweat.

👉 Read the full story

So, WebM, MP4, or Ogg? Choose Your Fighter!

Now you know the difference between popular video file formats and what actually makes that difference. Hint: codecs and containers do! Choose your file stack wisely and consider delegating file handling to a trusted provider. Reinventing the wheel when it comes to video may turn out to be a great waste of time and resources.