Forem: Josh Carvel

Understanding JS Build Tools (Part 2: Bundling and Transforming)

Josh Carvel — Mon, 31 Jul 2023 11:38:15 +0000

Intro 🔨

In Part 1, we talked about the benefits of modular JavaScript and package managers, and how this led to the widespread practice of bundling JS before it hits the browser.

In Part 2, we’re exploring the process of bundling and transforming JavaScript. These days, we tend to use a single tool to do all this work, known as a build tool. We’ll discuss a variety of build tools and the core features they provide, so you can understand how your projects are built and be able to choose and configure a build tool if you need to.

Prerequisites

Part 1, or good understanding of JavaScript modules and npm

Choosing your tool 🛒

There are many build tools in the JavaScript community, and they come in many forms. Some are simpler but more ‘opinionated’ in how they are configured, and others are more complicated to set up, but have a lot of flexibility. There’s no perfect tool for your project, unless you make a new one yourself (and now you know why there are so many JavaScript tools!).

You can see usage statistics and ratings of these tools in the annual State of JavaScript survey. As of now, webpack is still by far the most used build tool. It falls into the category of being very flexible and potentially very complicated, which has earned it a divisive reputation. It’s also been around for 11 years now, which is a long time in the JavaScript world.

A few years after webpack debuted, a couple of other build tools also became quite popular. Rollup, from the author of Svelte, was designed as a minimal tool which focuses only on bundling JS and allows you to specify plugins for everything else. Meanwhile, a tool called Parcel sold itself on a zero-config approach, to make it even easier to get started.

But the biggest change in recent years has been the move away from the idea that JS build tools must be written in JavaScript. Though they’re available to install via npm, the new generation of build tools uses low-level languages to perform build tasks more efficiently. This can streamline the deployment process, but perhaps the biggest advantage is speed of build during development, which can make a huge difference to productivity in big projects.

One of the pioneers of this approach is esbuild. It’s still a fairly new and quite minimal project, but it has a major focus on bundling JS very fast, using the Go language and various performance techniques. The benchmark on the esbuild homepage shows it bundling 100x faster than webpack.

Other tools are trying to cater for needs that esbuild can’t meet. The author of Vue.js has written Vite. Its main selling point is around reloading the page during development, which we’ll talk about later. Parcel, meanwhile, was rewritten in Rust for its version 2 - specifically, it makes use of a Rust framework called SWC (Speedy Web Compiler), which we’ll also mention later. And the author of webpack has been busy working with the creators of Next.js to produce Turbopack, which they bill as “The Rust-powered successor to Webpack”. It aims to provide an even better, faster development experience than Vite.

Finally, a couple of very new tools are trying to revolutionise front-end tooling with an all-in-one approach, for a more convenient development experience. Bun is designed as a faster replacement for Node.js and npm, but also comes with a transpiler (I’ll explain that later), a test runner, and a bundler which they have measured as even faster than esbuild. A project called ‘Rome’ had similar ideas - unfortunately the startup behind it closed in August 2023, but it has been forked as Biome. It doesn’t replace Node or npm, but plans to do all of the rest, along with formatting and linting (only the formatter and linter are released at the time of writing). It will be interesting to see if this approach takes off in the JS community once the tools mature.

Ultimately, what build tool you end up using very much depends on the needs of your project, such as its size and complexity, and what front-end library/framework (React, Vue, etc.) and rendering framework (Next.js, Astro etc.) you use, if any. There are initialisers for these frameworks that are built on top of a specific build tool - for example, the create-vue initialiser is built on Vite, since they have the same author.

In this article I’ll demonstrate Parcel, but only because it lends itself well to simple examples, so don’t be unduly influenced by this choice.

Command line JS 👨‍💻

Before we get to discussing their features, it’s worth taking a minute to understand how build tools run with npm.

Executable JS files

Most of the modules we install from npm are imported into our code to eventually be executed (run) in the browser. However, authors of Node.js files can also write their code to be executed on the command line. These are often referred to as executables (though strictly speaking, only code the computer can run directly without an interpreter meets this definition). By tradition in Unix-like systems (e.g. Linux, mac), executable files are held in a folder called bin (for binaries, not ‘bin’!).

It is possible to run Node.js executables using the command node in the terminal followed by the file name. This tells whatever shell program you use such as Bash, Zsh (mac) or PowerShell (Windows), to run the code via Node.js rather than trying to interpret it. However, in any Unix-like system the file author can give the file executable permission, then add an interpreter directive to make the shell run it in the program specified (Node.js in this case): #!/usr/bin/env node.

So let’s say I have installed Parcel and I want to run their command line code. In the ‘parcel’ folder in node_modules I can see they’ve added it in lib/bin.js with an interpreter directive. This means I can actually run this code with node_modules/parcel/lib/bin.js in my project’s directory in a Bash terminal. However, JS devs are lazy folks and we don’t like writing things like paths to files.

npm makes this easier. Firstly, it creates a single folder at the top of node_modules called .bin where it stores symbolic links (sort of like a shortcut) to the actual files. The author must list their executable JS files with names in the “bin” property of their package.json, or if it’s a single file the name is taken by default from the package.json “name” property - ‘parcel’ in this case. This means our command can just be node_modules/.bin/parcel.

Of course this is still too much for JS devs! npm actually provides even more of a shortcut with npx (mentioned in the previous article). npx already knows the node_modules/.bin path, so we can just write npx parcel. I’ve taken you the scenic route to get here, but I think it’s nice to understand what’s really going on under the hood.

Executing non-JS files

One more thing. I mentioned that some bundlers contain code written in languages like Rust or Go, yet they’re available via npm to run in Node.js. What’s the deal?

Well, we are not actually running the Rust or Go code directly. That has already been compiled into various different binaries suitable for execution in particular operating systems. Often these are published to npm separately as scoped packages (each npm user or organisation has a scope starting with @ and can publish related packages under it, e.g. @esbuild/linux-x64), and then added as optionalDependencies in the package.json of the main project, to be installed if possible in the user’s OS.

These binaries can then be imported and used directly in Node.js code because Node.js exposes an API for running natively executable files.

Bundling code 🧺

Ok, now let’s look at how we use a build tool for bundling.

Bundling is essentially performing an action on multiple source files in order to output one (or more, if specified) file(s). The source files are left intact for you to continue working on, and the output can be run in the target environment (the browser, in our case).

To bundle your code, you need to specify an input file that is not imported by other files, from which the bundler will start tracing the chain of imported files. You may also need to give the name of the output file to be created, and the path to the folder where you want it to live.

Your input file may be called something like index.js. The output folder is often called dist (for ‘distribution’), and we typically create a separate folder named src to house the source files. How you set up the HTML file(s) depends on the build tool and configuration, but it could for example live in src and contain a script tag pointing to index.js.

We can then ask the bundler to bundle the files. With Parcel, a basic example looks like:

npx parcel src/index.html

We passed the input file as an argument to the parcel command. The output folder was assumed to be dist, which Parcel creates for us and adds the bundled files into, including a copy of my HTML file with a script tag pointing to the outputted JS file. If we want to change the name or location of the output folder, in Parcel we can use the command line flag --dist-dir, or “targets” in package.json.

Now we have an HTML file which is ready to be served to the browser. All the build tools I’ve mentioned have a simple command like this to bundle JS files. But we can use something called npm scripts to make this even simpler.

npm scripts

npm scripts allow us to specify a shorthand for running commands in the terminal, similar to aliases. These shorthands can be anything you like, but some of them are considered standard, which is nice because you can run the same command for basic tasks no matter what project you’re working on.

The scripts are specified in the “scripts” property of the package.json file, like so:

"scripts": {
    "start": "parcel src/index.html"
  }

Like with npx, we don’t need to specify the path to the .bin folder here. To execute the command, you call npm run (an alias for npm run-script) followed by the name you assigned the script. However, the “start” script is so commonly used that npm have aliased it to just npm start.

Config files

Sometimes it is inconvenient or not possible to specify all options to the build tool on the command line, for example when using plugins. In this case we can use a configuration file, written with a name and format the build tool is expecting. Because these files will be run in Node.js, they are often written using CommonJS syntax. However, since 2020 Node.js has had built-in support for ESM syntax - one way to enable it is adding the .mjs extension to your files rather than .js.

We won’t use a config file in this article because our examples are quite simple, but if you need one the build tool docs should explain everything you need.

Bundling CSS

Ever since webpack came along, build tools (or their plugins) have been able to bundle CSS files too. Since every linked CSS file requires a network request, bundling them in advance helps the page load quicker. It also makes it easier to locate your styles next to the JS module that uses them, if you prefer this approach. Rather than having to add a link tag with a path to the file, you can specify an import from the same directory in the JS file, using ESM import syntax. For example, in index.js:

import "./index.css";

Bundling CSS requires a specific loader, which may be a plugin, or built into the bundler. Typically a bundled CSS file is produced and linked in the outputted HTML via a link tag. In Parcel this works without any extra config. A different approach is to add all the CSS rules to the document’s internal stylesheet (the <style> tag in the HTML file’s <head>, which has the same specificity as an external stylesheet), meaning no network requests are needed to load the CSS. webpack’s style loader uses this approach.

Loading assets

Other assets such as images can also be loaded by build tools from JavaScript. There are a few ways to import these assets, but the simplest syntax looks like this:

import image from "./image.jpeg"

The import which I’ve called ‘image’ here is a URL which can be assigned to an img element’s src attribute. The build tool’s relevant loader will handle getting the asset linked correctly in your output folder. Parcel, esbuild and webpack have loaders built in, though you need a little configuration in webpack and esbuild, and there is a plugin for Rolllup.

Bundling for development

So now we know how to make a basic bundle, but we still need to see our work in the browser before this is useful. We could just open our HTML file in the browser, but that’s not very convenient. npm projects typically use a server, even for local development. Some build tools have this built in, and for others you need a plugin package.

Using a development server

When you serve files, you serve to a particular host (computer), which has a hostname, such as the name of a website. When you tell your browser to get a file from that host, it translates that into the IP address of the computer where that file can be found.

But it is also possible to serve files to the same computer you are serving from. In that case, the hostname is always localhost. The computer reserves at least one specific IP address to do this, which in the IPv4 system is the address 127.0.0.1. (The 127 may seem a bit random, but it’s just half the maximum of an 8-bit block in binary).

When we ask to go to localhost or 127.0.0.1, the computer knows it can skip the actual networking and instead use something called the loopback interface, so we don’t need to specify the HTTP protocol. But we do need to specify a port so the browser knows which program to ask for the file.

There are 16 bits free for port numbers, which is over 65,000 numbers, though some are reserved for particular things. A running instance of a program on the host machine that wants to access the network can use any port which is not reserved or currently being used. Port 3000 is the default port for Node.js apps, so you will often see this used by JS build tools. Other round numbers in the thousands are often used too. Another standard is 8080, based on the fact that the standard HTTP port is 80. The port number is added after a colon, e.g. localhost:3000.

In Parcel, there is a built-in server which by default serves the code on localhost:1234 when you run the parcel command. You can also add the --open flag at the end to get the server to open up that page in your browser. Then you will see the contents of your HTML file on the page.

Rebuilding and reloading

Having the project served is one thing, but during development, we also want to make lots of changes to our source files and see the results reflected in the browser. A typical dev server can watch development files and detect when a file is changed. When this happens, it can trigger a rebuild, so that if you refresh your browser page, the new code is served.

This is all well and good, but JS devs are impatient as well as lazy. We scoff at having to refresh the browser page when we make a code change!

One solution is to get the build tool to reload the page for you, often called live reloading. In esbuild, if you’ve enabled file watching you can add some JS that listens to server events using the browser’s EventSource API and calls the browser’s reload function when a change event occurs.

new EventSource('/esbuild').addEventListener('change', () => location.reload())

“Still not good enough!”, the JS devs cry.

In fairness though, if your project contains a lot of JS and other assets that need loaded up-front, a page reload can be quite slow, which hurts productivity. So modern JS build tools (or their plugins) often have functionality to reload only part of the page content during development without a page refresh, which is referred to as hot reloading or Hot Module Replacement (HMR). This basically involves the build tool sending updated chunks of content to the browser to replace the outdated parts only.

The new tools on the block focus heavily on the HMR experience. Vite figured out it would be faster in development mode to actually serve native ES Modules to the browser (whereas in production it uses Rollup to bundle the files). When there is a code change it serves new modules, but lets the browser handle caching everything that hasn’t changed. Turbopack, in contrast, does not use ESM but claims to have achieved a faster refresh speed than Vite by leveraging WebSockets (a fast connection between browser and server).

esbuild, however, has gone in the opposite direction and doesn’t support HMR for JavaScript. In their words:

… JavaScript is stateful so you cannot transparently implement hot-reloading for JavaScript like you can for CSS. Some other development servers implement hot-reloading for JavaScript anyway, but it requires additional APIs, sometimes requires framework-specific hacks, and sometimes introduces transient state-related bugs during an editing session.

Bundling for production

When we bundle for production, we will need a slightly different process. In Parcel, we can use the parcel build command. We can add this command to an npm script called build, so my npm scripts now looks like this:

"scripts": {
    "start": "parcel src/index.html --open",
    "build": "parcel build src/index.html"
  },

Then to create my production bundle, I would run npm run build. How and where this is run depends on your deployment process, which is outside the scope of this article. But it is important to have a separate production build so that the development-only features we’ve talked about are turned off.

The packages you use also sometimes want to know if they are running in production mode, so they can remove development-friendly code like log messages. By convention this is done via an environment variable, which is just a variable that is stored outside of the process (running instance of the program) and can be set by the user. In Node.js, environment variables are accessed via the global process object, on an object called env. The variable NODE_ENV is used for toggling production mode.

Handily, most build tools set this variable for you based on your configuration, so you don’t need to worry about it. For example, Parcel sets process.env.NODE_ENV to ‘production’ when the parcel build command is used, and ‘development’ otherwise. You can always override the variable by setting it via the command line, or alternatively, any Node.js environment variables can be listed in a file called .env.

As well as turning off features, in production mode build tools can also apply optimisations to the bundling process for better performance.

Tree shaking

Tree shaking is a term popularised by Rollup which means removing unused imports from the outputted bundle. As described in the webpack docs:

You can imagine your application as a tree. The source code and libraries you actually use represent the green, living leaves of the tree. Dead code represents the brown, dead leaves of the tree that are consumed by autumn. In order to get rid of the dead leaves, you have to shake the tree, causing them to fall.

Tree shaking works best with ESM import statements, which as we mentioned last article are static in nature, meaning you can know for sure what code will be imported without running it. So build tools usually tree shake unused imports so long as they are not CommonJS imports or being converted to that format (though Parcel can tree shake CommonJS).

Note that we are referring only to unused imports, not unused code in general. For example, if you import a package that exports a single class and you only use some of the methods, the whole package will still end up in your bundle. But many popular libraries split up their exports, so you can just import the parts you need.

Code splitting

Code splitting refers to any means of splitting up your code into smaller parts, known as chunks, in the interests of better performance.

One natural way your code might be split is in a multi-page application with separate HTML and associated JS files. Build tools allow you to specify these files as separate entry points, which results in separate bundles for each page, so the browser only loads the content it needs for each page. However, these pages may use some of the same code, e.g. third-party packages. There is no need for the browser to re-download that code if it already has it cached, which is why build tools enable splitting up a bundle further into chunks (separately loaded files) so parts of it can be shared between pages.

Chunks can also be used in single-page applications, via ESM dynamic imports in the form import(moduleName). Dynamic imports return a Promise that resolves when the module and its dependencies have loaded. These can be used inside conditions or callbacks to implement lazy loading for modules, so they are only loaded if and when they are needed (e.g. see React.lazy in React 18). The build tool then handles creating separate chunks for dynamically imported modules and any shared dependencies. As the name suggests, dynamic imports are dynamic expressions, so they are not as easily compatible with tree-shaking (though again, Parcel supports it).

Note that code splitting support differs across build tools. Parcel and Rollup do a lot automatically, but esbuild support is still in progress. webpack requires you to specify dependencies when splitting chunks for multiple entry points, but has automatic support for dynamic imports via its SplitChunksPlugin. SplitChunksPlugin also by default splits out any node modules code into a separate chunk (known as the ‘vendor chunk’). This is handy for example if you ship new application code but your dependencies haven’t changed, because the vendor chunk does not need re-downloaded if the user’s browser already has it cached.

Content hashing

We just mentioned how the browser can cache our JS files so it doesn’t need to re-download them. This works great unless we do actually need them re-downloaded because the content has changed. For this reason, build tools support content hashing, either via configuration or by default. Hashing is the process of applying a function to some content that outputs a unique string, and if the content has not changed, the string will always be the same. In this case the outputted ‘hash’ string is added to the filename so the browser knows when the file has changed.

Transforming code 🤖

Now that we know how to create a basic bundle for development and production, let’s move on to understanding transformations. The high-level source code that we write can be transformed in a number of ways before it is run by the browser. This process is known as transpilation, i.e. converting code to a different high-level form, as opposed to compilation, which is converting high-level to very low-level code.

We already know that bundling usually involves some conversion of ESM or CommonJS syntax. For example when we bundle for the browser, module syntax is stripped away and often replaced with plain functions that wrap each module.

But there are additional transformations we can do for various reasons. Previously, task runners such as Gulp or Grunt were used for many of these operations, but using the bundler to do them keeps things simpler.

Transforming code for development

Certain tools are used by developers to write code more productively, but need transpiled for the browser to run them.

CSS transforms

Tools known as CSS preprocessors allow you to write custom CSS syntax, then transpile it to regular CSS. The most popular one (as per the 2022 State of CSS survey) is called Sass. Its syntax provides for CSS rule nesting and some programming concepts like variables and functions. How you transform the syntax into CSS depends on what bundler you use. Parcel automatically installs its own transformer for you - other bundlers require adding plugins and in some cases also installing the sass package from npm.

Another tool you should know about is PostCSS. PostCSS is not a preprocessor or even a tool you use directly. It provides some core logic that parses non-standard CSS and can run a chain of plugins to do the transformations. There are hundreds of these plugins such as autoprefixer, which automatically adds vendor prefixes to CSS selectors where necessary, and preset-env, which can transpile modern CSS for older browsers. There are also plugins for pretty much all Sass functionality, so it’s a viable and more flexible alternative. You can use PostCSS plugins by configuring them for your bundler as required.

Finally, it’s worth mentioning Lightning CSS, a fast CSS transform tool written in Rust by the author of Parcel. It’s built into Parcel and also available as a standalone tool. It implements the functionality of some of the most popular PostCSS plugins and can also support the draft CSS specification on nesting if you add the config for this.

Templating languages

Most devs writing for the browser use some kind of library or framework to make the job of writing JS for a system devised in the 90’s a bit more manageable. These provide some form of templating to make it easier to display different bits of UI conditionally with JS. However, this special syntax needs transformed to plain JavaScript for the browser.

The most popular of these languages is probably JSX, the lovechild of JavaScript and HTML which was created for React and can be optionally used in Vue. This is traditionally transpiled by a tool called Babel, which can translate the JSX markup into JavaScript functions to create UI elements.

However, there are now alternatives to Babel. SWC is a tool written in Rust to achieve tens of times faster performance than Babel, and is built into Parcel and Turbopack. esbuild comes with its own JSX transpilation, which is also used by Vite. And the Typescript compiler, which we’ll cover in a second, can also transform JSX to React code. With webpack and Rollup, you can specify your transpiling plugin. All these build tools also support many other templating languages.

Some frameworks come with their own compiler, which just needs configured with the build tool. For example, Svelte uses its own templating language which it compiles to JavaScript, and Solid.js also uses this approach, but with JSX syntax.

TypeScript

TypeScript is an extension to JavaScript with types. VSCode comes with Typescript, so if you write Typescript in a .ts file in VSCode, it understands the syntax, shows type information and errors on hover in the file, and lists errors and warnings from all files in the ‘Problems’ tab. This is a fairly convenient way to use Typescript in your project. However, because the browser doesn’t understand Typescript, it needs to be transpiled to JavaScript.

Many bundlers now come with Typescript transpilation built in, either using SWC (Parcel, Turbopack) or esbuild (esbuild, Vite), so you should be able to get started straight away. There is also an official Typescript compiler called tsc. This is handy when you want the type errors to stop your build from completing, rather than causing an error in the browser console. This is most important for production builds, but can be handy in development too. You can run it from the command line any time you like, or add it to a npm script, e.g. run tsc and then (&& operator) run parcel. We use the --noEmit flag so that tsc only does type checking, not emitting the transpiled JS files.

"start": "tsc --noEmit && parcel index.html",

Getting type errors to block the build on an ongoing basis as you edit files is more complicated because tsc has to run in the background, though Parcel has an experimental plugin for this which you can try. It can also be achieved by bundlers that run tsc via plugins such as webpack.

Bear in mind that tsc uses configuration defined in a tsconfig.json file, which is documented on the Typescript website. Explaining this config is outside the scope of this article, but you will need to configure at least a few properties here if you want to use tsc directly.

Transforming code for production

Serving code in a production environment brings its own challenges which further code transformations can help with.

Browser compatibility

During development you probably use the latest version of your preferred browser, but your users may be using an older browser in which newer JavaScript syntax doesn’t work. To tackle this problem, build tools can transform the newer code into the equivalent older syntax, via any of the transpilers mentioned above - Babel, SWC, esbuild or tsc.

To do this, you need to specify what level of compatibility you need. Many transpilers allow you to specify a range of browsers, from which they figure out which syntax needs transformed. This is done via a popular npm package called browserslist, which accepts a plain English syntax for specifying a range of browser versions and returns the full list. This configuration can be added in a “browserslist” property in package.json, or a .browserslistrc file (a filename ending in ‘rc’ is standard for a config file in any Unix-like system, and can contain whatever format the parser expects). You can see the configuration details on their website.

Note that tsc uses a different approach - you specify the lowest JS version you need to support in the “target” property of tsconfig.json. esbuild’s transpiler (also used by Vite) also uses a “target” property which takes JS versions, but it does accept a list of browser versions as well, which can be generated by running browserslist separately if necessary.

All of this is great for transforming newer syntax, such as the nullish coalescing operator, for example, which can be fairly easily written in older JS. However, there is a separate problem of newer APIs, e.g. Promises, which have more complicated implementations inside the browser (typically in C++). Writing another implementation of an API to fill a gap in browser support is known as a polyfill (a term coined back in 2009). Transpilers don’t take on the task of writing polyfills, but luckily there is an incredibly popular npm library called core-js that does, though it’s mostly maintained, with much frustration, by just one guy in Russia (yup, welcome to the JS ecosystem!).

core-js polyfills can be configured with Babel and SWC. They are not included as part of esbuild or Typescript, but you can import the polyfills you need directly from core-js as described in the core-js docs.

Finally, it’s worth noting that adding extra syntax and polyfills makes your bundles larger, especially when supporting ES5, which is a bit of a waste in up-to-date browsers. One approach to tackle this is known as differential serving/bundling. Two scripts are output - one with ES6 code, linked with type=module in the script tag, the other with ES5 code, with a nomodule attribute in the script tag, which tells the browser to ignore it if it supports modules (an ES6 feature). This is built into Parcel and can be configured in some other bundlers such as webpack.

Minification

Production is all about getting the code to the user as fast as possible, so the less code there is, the better. This also means as few characters as possible, why is why developers use minification to remove any characters that are not necessary for the code to run. Whitespace and comments are removed, variable names are shortened to one character, and syntax may be rewritten in a more concise way (you can usually configure these operations). This can be used on CSS, HTML, SVG and other files as well as JS.

The standard tool for minification is called terser. This became the most popular tool over the older projects uglify-js and babel-minify which are no longer maintained. It’s used by webpack when you specify ‘production mode’, and there is a plugin for Rollup. However, by now you will probably not be surprised to hear that SWC and esbuild use their own, speedier minification process.

Scope hoisting

There's one more little optimisation you might hear about. As mentioned earlier, bundlers need to maintain each module's scope to avoid naming conflicts once the code is all bundled together. In development mode, they create plain wrapping functions to achieve this, but for production, there is a newer approach called scope hoisting. Any top-level variables are given unique names, which avoids the need for wrapper functions, so the code runs quicker. This is built in with Parcel and Rollup, and available for webpack via a plugin.

Debugging code 🔧

Bundling and transforming code has lots of benefits, but it does lead to an issue with debugging. You are probably familiar with debugging a JavaScript error in devtools by clicking on the link to the source file in the top-right of the error. However, when we bundle JS, the source is now one giant file which has probably been transformed in a number of ways from the original code, making debugging much more difficult.

Luckily, there is a solution to this, called source maps. These are files ending in .map that are generated by the build tool, and they enable devtools to point you back to your original source file instead of the bundle. Unless you are writing your own library, you don’t need to worry about these maps too much aside from making sure they are generated, which is sometimes done by default (e.g. by Parcel), and sometimes requires a small amount of configuration.

Conclusion

Hopefully these explanations have given you a much better understanding of various build tools and what they do. Let’s recap some key points:

Modern build tools can handle both bundling and running various transformations on our code, and the needs of the development environment are very different from production.
There are many tools to choose from, with different pros and cons, including speed of development experience, ease of use and flexibility. Always keep in mind the needs of your particular project rather than getting carried away with the latest trend.
Build tools are complicated, but we are in an exciting era where they are also faster, more all-encompassing and perhaps easier to get started with than ever before.

I definitely haven’t covered all the ins and outs of build tools here, so if you want to know more about any of the tools I’ve mentioned, their documentation (listed below) is a good place to start. There are also plenty of discussions in blogs and social media about all these tools and users’ experiences and frustrations when it comes to the fine details.

Read these if you wish, though your main focus should always just be achieving reasonable productivity for your work. After all, tools are just tools - it’s what we build with them that matters!

Footnote: yet more build tools?!

If you looked at the State of JS 2022 survey, you’ll notice a couple of build tools I didn’t mention: Snowpack and WMR. As explained on Vite’s comparisons page, Snowpack is no longer maintained, and WMR is geared towards Preact, so it’s a more niche solution. You may also be aware of Deno, which is the follow-up runtime from the author of Node.js. It used to have a bundle option built in, but this was deprecated in favour of letting other tools handle bundling instead.

Sources

I cross-reference my sources as much as possible. If you think some information in this article is incorrect, please leave a polite comment or message me with supporting evidence 🙂.

* = particularly recommended for further study

Understanding JS Build Tools (Part 1: Modules and Packages)

Josh Carvel — Mon, 31 Jul 2023 11:37:58 +0000

Intro 🏗️

When you learn front-end development for the first time, you learn a simple fact: to run JavaScript in the browser, you add a <script> tag in an HTML file. You can set the src attribute to link to a JS file, or just write the JS inside the tag, and if you’re not using much JavaScript, you may not need to know much more than this to run it.

But these days, web applications and sites tend to use a lot of JavaScript. JS is gaining more and more responsibility in the browser, and on the server too. For this reason, we also have a lot of techniques and tools to help us construct our JS projects. And that can mean a lot of wrestling with build tools you don’t fully understand, even if you’re an experienced dev.

Of course, every layer of complexity or configuration is meant to improve either the developer or end user’s experience. But over time, it adds up to quite a heap of technical knowledge. And while we have tools and frameworks that can abstract away these details for you, it always pays to know what’s really going on, so you can take greater control of your projects.

In this two-part article, I’ll help you do that. We will demystify the key methods and tools commonly used to set up modern JS projects, examining what they do, why they exist, and their pros and cons. By the end, you should be able to make intelligent decisions about build tools and feel confident in tackling their configuration.

In Part 1, we’ll ground ourselves with a solid understanding of JavaScript modules and packages.

Prerequisites

Basic understanding of HTML, JavaScript and JSON
Basic understanding of Git

Modules 🗃️

Modules are at the heart of modern front-end development. This is because the JavaScript for our project is likely to be spread across different files. Even if we wrote all the JS ourselves, a single large file is not very nice to manage. And we definitely need other files if we’re using code other people wrote.

As a starting point, we could split up our JS files into separate scripts. Perhaps one for each page of the website or application, for example. But when you add multiple <script> tags in HTML, by default the JavaScript runs as if it was still all in a single file. If I declare a function or a variable globally (not inside a function), I can use it later on, even in a different file.

So if I have two scripts like this:

    <script src="./script1.js"></script>
    <script src="./script2.js"></script>

With a function like this in script1:

function sayHi() {
  alert("hi");
}

I can use the function in script2:

sayHi();
// the "hi" alert shows in the browser

This behaviour probably seemed OK back when JS was a novelty that could add a sprinkling of interactivity to mostly ‘static’ websites. But for today’s complex sites and applications, it’s far from ideal.

Firstly, we have to manage dependency order. Because script2 depends on a function from script1, the <script> tag for script2 must come after script1 in the HTML. The scripts are run in order, and you can't use a function or variable before it is defined. This may not seem like a big deal here, but the problem grows with the amount of scripts you have. Keeping track of what script needs to go where is manual work that we don’t want to do.

Secondly, the relationships between parts of our code are implicit, which can break our code in unexpected ways. If there is another sayHi function somewhere after script1, and I call it later expecting to get the script1 version, actually it will have been overwritten by the other implementation. This happens for function declarations and variables defined with var. It’s even worse for any variables (including function expressions) defined with const or let - these cause an error if another already exists with the same name. For code that uses a lot of libraries and frameworks, this is simply unworkable.

Finally, global variables and functions create a security issue. Anyone running your code could run a script to access your code’s inner workings and even change what it does.

So how do we deal with this? The answer is modules. But before proper module systems were used in JavaScript, there was a way to reduce the issues of global scope a little, sometimes referred to as the ‘module pattern’.

The 'module pattern'

Before modules, library authors would wrap their code in a single function, so the inner workings were not in global scope and couldn't conflict with anything. Inside that function, they would add all of their library’s public API to a single object (e.g. the $ object in the famous example of jQuery), and add it to the window object in order to make it globally accessible. Then finally they would call the wrapping function to make it work.

The catch here is that the name of your wrapping function would still end up on the global scope. This problem was sidestepped with an obscure technique. We can define functions without names in JavaScript if they are part of an expression (a line of code that resolves to a value). Usually expressions are assigned to a variable, but they don’t have to be - just writing a value such as 3 is a perfectly valid line of JavaScript.

However, a function not assigned to a variable is treated as a function declaration by the JavaScript compiler, so it will expect a function name. Putting the expression inside parentheses convinces the compiler it’s not a function declaration. We then need to call the function, and since we can't do it later (there’s no name to reference), we have to do it immediately. Hence we get this weird looking construction called an immediately invoked function expression (IIFE):

(function() {
    const library = {}
    // my library code...
    window.library = library;
})();

This pattern was very widely used in JavaScript code, but definitely wasn’t ideal. Each ‘module’ still exposes a public object to all other scripts, and we still have to manage the dependency order of scripts.

So let’s look at how a module system can solve these problems for us.

ECMAScript Modules (ESM)

There is a module syntax described in the ECMAScript specification (which defines JavaScript). It uses the keywords import and export to link dependencies.

For example, in script1 I could have:

export function sayHi() {
  alert("hi");
}

Then, in script2:

import { sayHi } from "./script1.js";

sayHi();
// the "hi" alert shows in the browser

We can immediately see how much nicer this is - the relationships between different files are now explicitly defined. And the code inside each module is scoped to the module, not globally scoped. So sayHi can only be used where it is imported. If I don't import it, I get an error:

sayHi();
// Uncaught ReferenceError: sayHi is not defined

Though I could have a different definition of sayHi if I wanted, and both would work fine in their respective files. If I both import sayHi and define a new version with the same name, I will get a (different) error when running the program, but at least the conflict of names is obvious in the file, not implicit.

import { sayHi } from "./script1.js";

function sayHi() {}

sayHi();
// Uncaught SyntaxError: Identifier 'sayHi' has already been declared

One way to avoid naming conflicts is using the module object syntax, which adds any exports from the imported file to an object with whatever name you assign it.

import * as Script1 from "./script1.js";

Script1.sayHi()
// the "hi" alert shows in the browser

Using Modules in the Browser

Now comes the complicated bit. ESM has become the standard for modern web applications, but there is a bit of history.

Firstly, this system was only added to the ECMAScript specification in the 2015 release commonly referred to as ES6 (the 6th release). Before this, browsers simply did not understand ESM syntax or have the functionality to wrap up scripts into isolated modules.

The first JS module system was actually created for running JavaScript outside the browser, by the CommonJS project in 2009. It was then implemented in Node.js, the popular JS runtime environment.

CommonJS modules use a different syntax to ESM. For example, here is a named export in script1:

module.exports.sayHi = function sayHi() {
  console.log("hi");
}

And here’s the import in script2:

const { sayHi } = require('./script1')

sayHi()
// "hi" is logged to the console

This system made it easy for authors of server-side and command-line JS projects to split up their code and use other people’s modules. But there was an obstacle to implementing it in the browser.

In Node.js, a JS file can be located on the machine and loaded immediately, so the code can run synchronously - line by line. The browser environment is different. HTML was not designed with modules in mind. You can’t just hand the browser a bunch of modules and get it to run them in the right order. You have to specify individual script tags, and each one is a network request that takes time to complete. So unless we want to block the rendering of the page while we wait for each script, that makes the process asynchronous.

Thus began the tradition of using tools to transform JavaScript modules into code the browser can run correctly. Early approaches to this transformed the code at the time of execution. There was a popular specification called Asynchronous Module Definition (AMD), where your scripts defined a function which wrapped the code to be executed and listed the dependencies required, but did not call that function. At runtime, a third-party library known as a script loader retrieved the script elements from the DOM, sorted out the required order and added the module code into the DOM dynamically.

Other libraries were available as command-line build tools. These compiled the CommonJS code before it was delivered to the browser. A popular tool called Browserify was created to handled the dependencies between modules and bundle all the code inside a single script tag. It was later overtaken by webpack and other bundlers that could handle ESM syntax and provide other features we’ll talk about in the next article.

Unfortunately, before ESM there was no consensus on the best module approach, so library authors needed their code to work for users using either CommonJS or AMD. This led to Universal Module Definition (UMD), which basically took authors back to using an IIFE, then adding code to export the module in whatever way was being used at runtime.

You won’t need to use these approaches in your projects, but you will hear them mentioned in bundler documentation. The bundler needs to handle different types of exports and imports, because many popular libraries are still not exported using ESM (for example, React still exports CommonJS modules). And when we bundle for the browser, we usually output our code in an IIFE so it can run in older browsers, while still being isolated from any other JS that may be running on the page.

Default exports

Another legacy of previous module systems is that exporting a single object containing all the library’s functionality was very common. ESM has a syntax that allows for this called default exports.

For example, let’s say I have a file called MyLibrary.js, and all I want to export from it is a class called MyLibrary. If I export it with export default, it can then be imported without curly brackets:

import MyLibrary from "./MyLibrary.js"

To bundle or not to bundle?

Ok, we’re done with the history lesson and we want to use ESM syntax. But we still have a shoot-out between bundling the modules before they hit the browser, and just using the browser implementation. If you want to use import statements without bundling, the <script> tag for each module must have its type attribute set to “module”. With our example scripts that looks like this:

<script type="module" src="./script1.js"></script>
<script type="module" src="./script2.js"></script>

When the browser reads these files, it figures out the dependencies before any of the code is executed, so we don’t have to worry about the order. If script1 imports something from script2, that tells the browser to execute script2 first, even though it comes later in the HTML.

Note that this makes the imports static - they’re not allowed to change during execution. So you can't use an import or export statement inside a function or condition - it must be at the 'top level'. (There is an alternative called ‘dynamic imports’ however, which we’ll cover in Part 2).

So now this exists in the browser, why bother using build tools instead? Well, there are a number of reasons why bundling JavaScript has become so popular. One is that it cuts down on network requests - why make the browser request multiple files when we can send just one? Another is that build tools can handle other useful operations before outputting our code, which we’ll cover in Part 2.

But perhaps the biggest reason relates to how we use other people’s code.

Sharing code 🎁

We often want to use other people's libraries and frameworks, known as dependencies, in our JS. What's the best way to do this?

You could go to the website or GitHub repository of the library you want to use and download the code, then add it to your project. But that’s slow, and you would have to also download all of the libraries that it depends on. Since JS projects these days have so many dependencies, that’s not really feasible. And this is just for one version of the library, never mind dealing with updates.

A slightly more common approach is linking in a <script> tag to the URL of a server that the library code (and its dependencies) is stored on. To do this, the author must have first publicised this link and the server must allow cross-origin requests (requests coming from other URLs). Additionally, in order for the code to get to the browser quickly, they will most likely use a content delivery network (CDN).

CDN Links

A CDN is a network of servers across the world. Companies such as Cloudflare, Akamai, Amazon, Google, Microsoft and many others provide these networks, which code authors pay to handle serving their resource, e.g. a website, application or library.

The author usually puts their code on their own server (known as the origin server), but either sets up a redirect or alters their urls to point to the CDN instead. The CDN servers pull that resource from the origin and cache it so wherever the user is, there should be a copy nearby which can reach them quickly. Periodically, the CDN servers request the resource from the origin server to ensure it is up-to-date, or the author can manually invalidate the cache.

The advantage of using a library via a CDN is that you don't have to download the code or its dependencies yourself, and it's generally fast and reliable. Plus it can be better for the user, because since multiple sites can access a library from the same CDN, the user’s browser might already have cached the resource you’re linking to, in which case it doesn’t need to fetch it again over the network. So why not use CDN links for everything?

Firstly, there are some small drawbacks that might be an issue, such as not being able to work offline or not having control of serving the resources needed for your project. Another simple one is that there might not be a CDN link available for the library you want to use.

But there are bigger reasons too:

Admin. During development, you need to add all the <script> tags in the right place. Then as time goes on, you need to manually manage what version of the library you need when third party packages are updated.
Performance. This is the same problem we mentioned earlier - the browser has to make a separate request for each <script> tag (unless the resource is in the browser cache), which will slow down load times in production.

For a very small project, these downsides might be no problem. But for everything else, this is why package managers are so popular. Package managers provide a system for managing dependencies. All the code is stored on the developer's computer, so the application can be developed offline if necessary. And you can use a bundler to output only one <script> tag (or however many you prefer).

The downside of packages? They come from the Node.js world. You have to run Node.js on your computer, and a bundler to join all the modules together. This can be a hurdle for junior developers, and adds some complexity to getting started with JS projects. But for most JS devs this isn’t a big deal - we’ve accepted the inconvenience of setting up Node and a bundler config every once in a while, in return for the benefits it provides.

It does mean, however, that native ESM in the browser was somewhat dead on arrival in 2015. It has its use cases, but people were already used to using packages and a bundler, and they had lots of reasons to stick with that.

So let’s dive into how packages work.

Packages

A package is a library of code together with information about the code, such as the version number. A package manager is the tool that lets you download, manage, and if necessary create those packages. It has a command-line interface which communicates with the registry, which is the database of all the packages that have been published by users. These aren't all public (access can be scoped to a particular organisation, for example), but many are.

We’ll be looking at the most popular package manager for JS, npm. In the early years of npm, there were some issues that led some developers at Facebook and elsewhere to create yarn, a similar alternative. However, npm is generally considered to have caught up with the main yarn improvements. There is also pnpm, which offers some speed and disk space improvements, but is less widely used. As a beginner, you should get used to npm for the foreseeable, since the others are similar to use anyway.

To use npm, you just need to download Node.js from the Node.js website - npm is included with it because it’s so central to JS development these days. You can find the list of npm packages on the npm website. To install a package, navigate to your project’s directory and run npm install followed by the package name, e.g. npm install react. There are built-in aliases (names for the same command) such as npm i, which you can use if your prefer. If you need to uninstall a package, you can use npm uninstall, which also has aliases such as npm remove or npm r.

The install command downloads a package from the registry. The installation is local by default - you can only use the package in the directory you installed it in. This is the standard approach nowadays and is useful when you need to use different versions of a package for different projects. However, you can install a package globally if you need to, using the global flag (option) -g, e.g. npm install -g react. Then it will be installed somewhere accessible everywhere (the exact location depends on the operating system).

Packages are downloaded to a folder npm creates in your project called node_modules, with a subfolder for each package. Most packages in turn have their own dependencies (sub-dependencies), which are also installed to your machine in the node_modules folder. This means the node_modules folder gets very big very quickly! You definitely don't want to commit this large folder into version control, so add the node_modules folder to your .gitignore file.

Security and npm

Please note that anyone can upload code to npm, and it relies on user reports to take down malicious code. In particular there is a practice called typosquatting, where malicious code is uploaded with a slightly different name from a popular npm package so users might install it by accident. So be careful what you install, and check the package information on the npm website, including the ‘weekly downloads’ figure, to determine if a package seems legitimate and reliable.

Using a package

Once you have installed a package, you can import it in any JS file. You might expect you need to specify the path to the folder in node_modules, but luckily bundlers allow bare specifiers, for example:

import React from "react"

When we do this, the bundler (or a plugin) knows to look for the dependency in node_modules and import it from there.

package.json

When you install a package, an entry is added to a file called package.json, which npm creates for you if you haven’t created it already. This lists your dependencies in the dependencies property, and the file should be committed to version control. That way, when someone else clones your project to work on it, they can just run npm install and all the dependencies listed in package.json will be installed for them.

The package.json is also used by package authors to specify information about their package. This includes the package name, description, version number, and devDependencies, which is dependencies they used to create the package, but the package users don’t need installed. There is also peerDependencies, which should be specified by plugin authors who are adding functionality to another package without using that package directly.

You will often see package users installing packages that aren’t used in the application code, such as testing libraries, with the --save-dev flag, so they are listed in devDependencies. This is not strictly necessary, since bundlers usually eliminate code that is not imported by your application (covered in the next article). However, it has become something of a standard practice, which you can follow if you like.

Peer dependencies are also worth understanding because since npm 7, npm automatically installs peer dependencies by default, meaning you may see an error relating to conflicting versions of a package being installed in your project. This error can be ignored with npm install --legacy-peer-deps, though it’s better in the long run to resolve the conflict.

Finally, you might see some custom properties in package.json that are not documented by npm. These can be used to specify configuration to build tools, which we’ll cover in Part 2.

Package versions

Now we can get to one of the main advantages and complications of package managers - managing package versions.

When packages are published, they are classified by a version number. The traditional approach in software is that any release has three aspects to it, represented by three numbers separated by dots: the 'major' release version for major feature additions (1.x.x), the ‘minor’ release version for minor feature additions (x.1.x), and the 'patch' release version for small bug fixes (x.x.1). These numbers are incremented whenever necessary, so a package could skip straight from version 1.1.0 to version 1.2.0 for example, if the change made was a minor feature.

npm packages are recommended to follow a strict, systemised interpretation of this approach called semantic versioning (often abbreviated to 'semver'). This aims to make working with dependencies more reliable. The main rule is that minor and patch releases cannot contain breaking changes, i.e. changes that would break code currently using the package.

When you install a npm package, you can specify a version using @ if that version is available, e.g. npm install react@16.14.0. But if no version is specified, and we don’t already have the package installed, npm will always install the latest version. In the example of npm install react, that may look like this in the dependencies property of package.json:

"react": "^18.2.0"

OK, so version 18.2.0 was installed. But what's that funny thing before the number? Well that's called a caret, and it complicates things a little.

In theory, every package user and author could always specify exact versions of packages. This can be configured for all installations by running npm config set save-exact=true, or for a particular package with npm install --save-exact. You then need to update packages individually when you are sure it's safe to do so (minor and patch releases aren’t supposed to break stuff - that doesn't mean they can't!). With this approach, package versions are specified in package.json with no caret.

There are couple of downsides to this: firstly, you have to track and manage all the versions of your dependencies yourself. Secondly, consider you install package A and package B in your project, but those packages depend on package C, and have specified slightly different versions of it. npm will have to install both versions of package C in your node_modules folder, which could be a waste of space - the packages probably would have worked fine with the same version of package C.

So npm allows some flexibility - we can specify acceptable ranges of package versions. The default is that new patch or minor versions of the package will be tolerated (since they should not break existing code). This is what the caret in package.json indicates. I can therefore run npm update (or npm install) with a particular package name, and the package will be updated to the latest non-major version. Or I can run npm update without arguments to update all my packages at once.

You may also see a tilde before the package version, e.g. "react": "~17.0.0". This means that only updated patch versions are allowed, so React 16.3.1 could be installed but not React 16.4.0. This can be configured with npm config set save-prefix='~'.

Locking package versions

Unfortunately there is a side-effect of allowing version ranges. Say someone else clones my repository and runs npm install, and in the time since I installed the packages myself, there have been some minor or patch releases. The package.json file stays exactly the same, but the other user gets slightly different versions of the packages, which could lead to different behaviour.

To get around this, npm introduced another file called package-lock.json, which is also created when you install packages. This file is sort of like a historical record: it shows the exact version of all the dependencies and sub-dependencies that were installed using npm install, and it should be committed to version control along with package.json.

When another user runs npm install in your project, npm will usually install the exact versions specified in package.lock-json rather than the more recent minor or patch versions that could be installed according to package.json. This ensures consistency of package installations while still maintaining the benefits of specifying dependency ranges.

For example, your package.json might list "react": "^18.1.0". Without a package-lock.json, the other user could get React 18.2.0 installed instead. But if the package-lock.json file says 18.1.0, that is what the other user will get.

There is one exception, in the scenario that package.json has been edited manually to specify a different version. If it was edited to say 18.2.0, for example, the other user will get that when running npm install, even though package-lock.json said 18.1.0. The installation causes package-lock.json to be updated to list 18.2.0.

Note: package-lock.json used to always win out in this case, but since users tended to see package.json as the 'source of truth' of their package versions, npm altered the behaviour in 2017.

Initialisers

So now you understand packages and the purpose of module bundling, but you still might be unsure how to get started building a JS application. One way to do this is by using a type of npm package known as an initialiser.

Initialisers are designed to configure your local environment for building a certain type of app, including setting up the bundler. The name of these packages begins with ‘create’, and they contain code that can be run on the command line to install certain packages and set up config files for you.

You can run these packages with the npm command npx, which temporarily installs and runs the package specified in your local directory, then removes the installation once its work is done. This makes it ideal for initialisers, since they are only needed once in a project. So an example would be npx create-react-app. You could also run npm init react-app (no ‘create’ in the package name), which is a slightly newer equivalent (init is also aliased to create).

Initialisers like this make it quicker to get up and running with a project. You can then to refer to their documentation to get an idea of what they are doing for you and how to continue. Usually you can also customise the setup to some degree to suit your needs.

One downside is that you need a fairly clear idea of your app’s requirements before you start, unless you want to spend some additional time untangling the config later. Another is that outsourcing configuration means you don’t really need to understand it. Usually in development this works fine until something doesn’t work as you want it to, at which point it causes you a headache. In this series we’re diving into the details that some people ignore, so you can make a properly informed decision of how to set up your project, whether you choose to use an initialiser or not.

Conclusion

We’ve covered lots of information about modules and npm here, but we didn’t get to how build tools work. I’ll cover that in the next article because it’s a big topic. First, let’s recap the key points:

Modules are key to modern JS, but the systems we use today are heavily influenced by the under-powered original design of HTML and the ‘historical’ developments since then. This is why we ended up using ESM, but not necessarily the browser implementation.
Using packages via Node.js is so popular partly because it allows us to bundle our code to be fetched with a single network request, as well as managing different versions of our dependencies more easily. However, it is not without its own complications and drawbacks.
Even though certain approaches have become very popular in the JavaScript community, you should always look to use the right tool for the job. That requires a solid understanding of the tools, which is what this series is all about.

Footnote: CDNs and npm

You may have heard of an open-source project called unpkg.com which uses Cloudflare to provide a CDN link for every package available on npm. This is a free service relying on sponsors, and it doesn’t provide any uptime or support guarantees. So the project author doesn’t recommend relying on it for anything critical to a business. However, it can be very useful for demos, personal projects and trying out packages without a local environment set up.

Sources

I cross-reference my sources as much as possible. If you think some information in this article is incorrect, please leave a polite comment or message me with supporting evidence 🙂.

* = particularly recommended for further study

How to Get Your First Dev Job: A Complete Guide

Josh Carvel — Sun, 20 Dec 2020 16:47:07 +0000

Intro 🚪

If you are reading this article, you've probably heard of a strange phenomenon.

I'm here to tell you that yes, it's true: the vast and mysterious realm of developers is accessible to you, the humble sales assistant, accountant, fast-food worker or philosophy graduate, and the price of admission is not (necessarily) four or more years of gruelling and expensive education. In fact, you need only to ask for entry.

I know, because I did it. I taught myself to code online and landed a good job at a software company.

But you may also know that the road to the door can be long, winding, and full of confusing words. You're probably wondering how best to avoid the trapdoors, navigate the mazes and tangled forests, and appear wise and knowing to the keepers of the gate.

Well, I'm going to tell you that no matter how long and hard the journey, you just need to focus on three factors. I'll tell you exactly how to do that, and also give you three secret weapons to help you along the way.

The factors 📊

The three factors you need to work on to get your first developer job are (in no particular order):

Your technical ability and knowledge, in the context of the job you are applying for
Your general employability, i.e. non-technical skills, traits and experience
The number of positions you are considered for

Of course there are other factors that affect whether someone is hired, such as luck, systemic issues or any number of prejudices or irrelevant considerations on the part of the employer. However, this article is about what you can do to get a job, right now. In that context (and I mean only in that context), every moment spent dwelling on these other factors is a wasted one.

A game of percentages

You need to understand the three factors above for what they are: things that increase your chances of getting a job as a developer. Being employable is not a binary state. You are not 'unemployable' as a developer, and then at some point emerge from your cocoon into an 'employable' developer butterfly.

The choice to hire you is an individual one, made by an individual company. You could be hired with next to no technical skills, or you could do everything right to get a particular job, but still not get it.

In these circumstances, the only thing you can do is anything that will positively affect one of the three factors. All other things being equal, if there is a chance to increase your technical ability, you'll take it. If there is a chance to make yourself more 'employable', you'll take it, no matter how hard it is to judge yourself, or how much you hate promoting yourself. If there is another application to be made, you'll make it, no matter how afraid of rejection you are.

The above may sound a little obvious, but if your journey to getting a developer job is like most people's (and mine), there will be plenty of opportunities for you to bemoan something outside of your control rather than taking a targeted, positive action. This is natural and only human, but you are going to need to keep picking yourself up and refocusing, time and time again.

Priorities

The caveat to the above is that you only have so much time. In this article we'll look at how to prioritise in each of the three areas so you can maximise your efficiency.

You may also be wondering, how important is each factor? There is no exact answer to this, but neglecting any one of these factors is damaging to your chances of getting a job, and it's far better to have a good level in all the factors than to have disproportionate focus on one.

Many people focus far too much on factor 1, usually because they are most comfortable with working on technical skills. But virtually no company hires someone that did not present themselves in a good light through their communication with the company, and who doesn't seem like someone who can be worked with. And no company hires someone who they haven't heard of because they never applied, or who sent one email that got lost in a packed inbox.

Some people go all in on factor 3, neglecting the other two, most commonly factor 2. But the fact is that if there is something fundamental holding you back, like poor communication skills, it's not going to matter how many companies you apply to - they are all going to react the same way. You need to be very aware of whether you have the fundamental qualities required for the positions you are applying for. This can be difficult because the companies you apply for will usually not tell you this, and mostly just ignore you. But we'll look at some ways you can improve these skills later.

How long will it take?

You may have seen articles in the form of 'how I became a developer in [x] months', with the only constraint on x seemingly being that it is greater than 0. One such article may have even inspired you to become a developer, and they often have lots of useful information in them.

However, aside from the fact that they usually focus mainly on factor 1 and neglect the other two, they also tend to obscure the fact that transitioning into a coding job is a personal journey that depends very much on your own circumstances, traits, and the amount of time you have available to dedicate to it.

The truth is that you should try to get your job as fast as possible, but no faster. Which sounds obvious, but when comparing yourself to others, you'll often feel pressure to go faster than you can. You will find it difficult, and chalk up your failings to personal deficiencies rather than seeing them as an inevitable part of the journey. You may face financial and personal pressures that make it seem like you needed to get your new job yesterday. But all you can do is keep plugging away at the three factors.

To give some kind of indication, yes, it is possible to get a developer job in a matter of months, given full-time study and efficient, focused learning. Less than 3 months is incredible, less than 6 is still remarkable and usually requires the help of a bootcamp. A more normal timescale for full-time study is 6-12 months. For anything less than full-time study, 1-2 years would be a more common timescale. All of this also depends on the requirements of the job you get at the end of it as well - there are many kinds of developer roles.

A wake-up call?

If you are already some way on the journey to your first dev job, just understanding and acting on the advice in this section will take you a long way. If you already know the best way to gain technical skills, become employable or get your foot in the door with a company, but have been neglecting one of these factors, your main take-away from this should be to just get on with working on it.

But I'll now go into a lot more detail on how to work on the 3 factors. First, I'm going to tell you your secret weapons.

Your secret weapons ⚔️

Many people know some of the things I'll tell you in the next few sections, such as where to find coding materials, what to put on your CV, and so on. What they struggle with is some more fundamental problems.

They lack confidence, they don't see themselves as good enough to be a developer. They struggle with keeping discipline and good habits. They give up too early.

But you are going to overcome those challenges. You have some secret weapons.

1. Vision

The first step to achieving anything should always be honing your vision of where you are going and how you see yourself. Below, we'll talk about getting clarity on the job you want. But ideally, you should be looking even further beyond that.

Where do you want to be, ultimately? What would you be doing if money was no object, really? It doesn't even matter if the answer is not 'being a developer', if becoming a good, well-paid developer is how you get there. It also doesn't matter if the answer is just a guess (you're not a soothsayer), so long as it helps to focus and motivate you.

Supercharged motivation

Once you have a long-term vision, you should already be even more excited about your journey. Work backwards from there to the short-term. Roughly plot out some points on the journey - where will you be in ten years, five years, one year. Just guesses that don't seem totally unreasonable. This should lead you to a better idea of the first developer job you want to get.

Once you've done this, everything you do from now on should have a clear why behind it. When you have to do hard things like sit down to code, make another job application or attend a networking event, and you wonder why bother?, your answer is right there staring at you: because it will directly lead you to the thing you most want to do in the world. Then you have a simple choice: do something a bit hard now for limitless gain in the future, or give up on what you really want.

You 2.0

Having a clear vision also goes to the core of how you see yourself. If you can see and believe (even just a little bit) a vision of yourself achieving something you would be really proud of, you can start to see how you could change and become the sort of person that can achieve that result. Then simply start trying to be that person a little more every day - the more you practise, the better you get.

Start thinking of yourself as whatever you want to be - an entrepreneur or freelancer, a problem-solver, a leader - and let your actions slowly start to fit the role a little better. And, of course, start thinking of yourself as a developer. As soon as you write some code that does something, you're not an aspiring developer, you are a developer, by definition. Just not a professional one. So embrace it. Put it on your CV. This is the new you!

2. Habit

Habit is the key to achieving any goal. Once you have a big, long-term vision, you need to link it to a small, focused, consistent habit.

Scientifically speaking, a habit is more of an automatic reaction. It doesn't cover a conscious and deliberate action like coding for an hour. But the secret is to master the automatic behaviours that put you in the right place, with the right environment, at the right time, to do the activity you want to do. Once you practise this for a while, nine or more times out of ten, you will do the activity you intend.

Let's say you want to practise sitting down to code at a certain time. Here are some factors you need to think about:

You need to be in the right frame of mind for focusing. Carving out some time at the beginning of the day rather than leaving it to the end will probably be better for most people. There is evidence that your willpower and ability to focus is sapped as the day goes on (see sources at the end of this article).
Your environment needs to be as comfortable as possible, e.g. a proper desk with good ergonomics, preferably clean and tidy - a place that makes you want to spend time there.
Minimise distractions. Just leave your phone in another room, seriously. It's the biggest productivity killer imaginable. If there are other people around, they need to know not to disturb you if possible.
Have a clear time period with breaks. E.g. study in 20 minute bursts with breaks, or 45 minutes, whatever works for you. Use a timer for best results.

Slow and steady

When you start a new habit, you have to start small. You need small wins. If you say you're going to fit in 4 hours a day of coding that you barely have time for, you're setting yourself up to miss your target, and the negative self-talk most of us are very familiar with will happily step in.

Instead, focus on that small behaviour. It could be as simple as just sitting down at the computer at the right time. Every day, for a month. Or studying without getting distracted for an hour (breaks not included). Simple, quantifiable actions. It doesn't really matter if the study session went great, or it was really hard and confusing, and you didn't feel like you made progress. What matters is that you did what you intended, and that's all you can do. It will all add up.

Other tips

There's lots more I could say about habits, but here's just a few more tips to follow if you can:

Accountability is a great tool. Tell one or two people you respect what you are going to do (don't only announce it on social media - no one there will chase you up if you don't follow through). For best results, write it down in the form of a promise.
Be flexible. If something's not working, tweak it. You don't want to change your habit all the time, but you don't want to stick with something that's not achievable. Find what works for your schedule, your life, your personality, so you can do it consistently.
Keep a chart on your wall of your progress. Set mini-goals and celebrate when you achieve these landmarks. It doesn't matter how much of the mountain is still ahead of you - always remind yourself of how far you've come and be proud.

3. Persistence

Most of us are guilty of a poor understanding of results. We expect results to happen steadily as a direct proportion of effort - a perfectly diagonal line on a graph. In most cases, it's actually a curve that starts off very gentle for a long time and then starts to curve sharply upwards.

Once we accept this (and it is very hard to accept), we have to commit to simply doing the actions, and have faith that the results will come later. If you've ever learnt a musical instrument, you know that in the beginning, after the initial rush of your first few little achievements, you start to feel like you've plateaued at a really low level, and that you're not making any progress despite your practise. Then, one day, like magic, you suddenly realise you can play something that you've been struggling with for ages.

Learning to code is no different. Getting a job is no different. These things are hard for a long time, until suddenly, they're not. You finally understand a concept. You finally get to grips with a language. You finally get the knack of job applications. You finally get the job.

It's easy to get discouraged in the thick of it. But if you have your clear vision, you have your simple, well-defined habit, and you have the commitment to just seeing it through until you are successful, then there will be no stopping you.

With all that said, let's finally look more closely at the three factors I mentioned at the beginning.

Factor 1: Technical ability 👨🏽‍💻

Factor 1 is: "Your technical ability and knowledge, in the context of the job you are applying for."

Note the second part - in the context of the job you are applying for. If your goal is to get hired, and time is of the essence, then everything you learn needs to be relevant to getting hired.

Naturally, if you don't know what sort of job you are looking for, you will struggle with knowing what to learn. So step 1 is to target a specific kind of job - the more specific, the better.

Targeting a job

How do you know what job you want? First you need to do a little research on what is out there. Google and YouTube are your friends here. Also try to speak directly to developers if possible, for example at the company where you work, or through social media like Twitter.

You don't want to drag out this stage too long - get some understanding of web dev vs. software dev, front end vs. back end and so on, just enough to start to feel some kind of preference, then choose. The important thing is just to make a decision. It's not for life. After you've gotten your first dev job, you can branch out or switch tracks - it happens all the time.

If you are really struggling with the decision, you may want to go further into a few different paths, for example by trying out a few different programming languages and seeing what you prefer. It certainly is never going to hurt to be learning something about coding. However, the longer you put off deciding what to focus on, the less efficient your journey will be.

Narrowing it down further

Once you've chosen your path, you will need to do some further research to understand the technologies involved, the type of jobs available, and to narrow your focus down even further. For example, you may have decided to become a front-end web developer. Even within this path, there is a lot of variety.

Do you want to build websites, for example as a freelancer or at a web design agency, focusing mostly on HTML and CSS, maybe doing some design work, and probably working with a content management system? Or do you want to work on web applications, focusing more on JavaScript and using a JavaScript framework? (for related discussion see: https://css-tricks.com/the-great-divide/). And which framework - React, Vue, Angular?

Look at job listings in your area to get a better idea of this. You should narrow it down to the point where there is a just a handful of job titles you are looking for (although some companies will use titles inaccurately) and you can instantly recognise the sort of job description that suits you.

When we talk about narrowing down the jobs you are eligible for, you may be concerned that you are reducing your chances of getting a job. The opposite is true. You are applying for fewer jobs, but have a much better chance of getting them, because everything you have been learning is relevant to those jobs. If you cast your net wider, you will be overlooked in favour of people with more relevant skills.

A note about job adverts

It's well known that many developer job adverts are not very realistic or helpful. They may list skills that are not really required, overemphasise certain aspects or ask for a computer science degree regardless of whether the role requires it.

Companies often advertise for their ideal candidate, and there can be quite a gap between the job description and the skills of the person they will actually hire. In other cases, the advert may be written by someone that doesn't fully understand the role.

It's very important to bear this in mind to avoid being discouraged. Focus on whether the main gist of the role suits you. Then apply.

Learning

This is the best part of the journey. If you don't enjoy learning, you probably shouldn't become a developer.

But how should you do it? First, keep in mind two main principles, which we'll come back to:

Learn from a reputable source of information
Make sure you are getting value for money

Two options

Let's assume you are not going to do a computer science degree, since the main topics of study are not immediately relevant to the typical new developer.

You then have the choice between some other paid course, most commonly at a coding 'bootcamp', or 'teaching yourself' with online resources. As someone who did the latter, I'm a bit biased towards it, but you have to choose the option that's right for you.

Teaching yourself is very difficult - you have fewer people to go to for help, it can be hard to learn the right topic at the right time, and you have to be very driven and self-motivated. You are alone with the mountain of information on the internet. On the other hand, you'll save a hefty amount of money, and if you get through it, you'll also come out of it with bucket-loads of resilience, research skills and problem-solving ability to brag to employers about.

If you feel like you probably need a bit more of a push and a guiding hand, a bootcamp could be for you. Your learning will be efficiently packaged up for you, you will have a cohort to get through it with, teachers to ask questions of, and probably be given the opportunity to network with some local companies. It's not uncommon for people to go straight from a bootcamp into a job, though it depends on the individual.

Just bear in mind that bootcamps are not currently regulated, and they usually ask for a lot of money for a relatively short period of education, so you should do thorough research and make your own judgement, keeping in mind the two principles mentioned above.

Teaching yourself

We'll now talk about how to go about teaching yourself, but this is not just for the non-bootcamp route. As a developer you will always be teaching yourself to some extent. Technology is always changing and the web is your instruction manual.

Here's three points to consider.

1. What you learn

This part should be fairly straightforward for getting your first dev job - you should know from your research where you need to start, and you can add some things to the list as you go.

However, the aim is to keep the list as short as possible to get your first developer job. As you start learning you will come across many, many different technologies and aspects of development. There will be a lot of temptations to stray off course. You need to build the habit of sticking with what is on your list.

2. Where you learn

There is no shortage of resources online to learn development, but they do vary in quality and price. You need to again bear in mind the two principles above.

I won't list them all here, because there is plenty of info out there if you google what resources are best in your chosen area of development. But I'll explain the types of content you will see:

Interactive tutorial websites

Websites such as freeCodeCamp (free!) and codecademy (free/paid) offer tutorials where you can code in your browser, along with other kinds of content. You will probably want to make use of one of these kinds of courses at least once in your journey.

Video tutorials

Video tutorials are great resources which I personally like a lot, and they're especially useful for beginners because you can see how professional developers write code and learn a lot from them. As well as teaching particular topics, there are lots of videos teaching you to build a particular project which you can code along with.

There are lots of great programming video creators, and I really strongly advise that you do some googling to find out who they are first, to avoid missing out on content you might miss if you just search for content on a particular topic. Many of them have free videos on YouTube, either on a single topic or a free sample of a paid-for course, so make use of these.

You can also buy courses on sites like Udemy (just be aware of Udemy's marketing tactics of hiking up prices then having an almost constant 'sale'). Don't shy away from buying a course from a reputable creator, because you will usually get what you pay for and get a lot more value than free resources. However, this shouldn't be your default option and you shouldn't need more than a handful of paid courses. Personally, I got a job after spending a grand total of £45 on Udemy courses (I bought three).

Plain text information

You will find a lot of good content in the form of technical documentation, books and online references. For example, if you are learning JavaScript there is the MDN documentation and tutorials, sites like javascript.info and books like You Don't Know JS. Personally, I didn't use any physical books to learn to code - there was enough info available free online. But if you prefer textbooks, there are plenty, such as O'Reilly books.

The stuff you find when you google a question...

Here is where you really need to pay special attention to the reputable source of information point. When you google a programming question you will usually find lots of articles on it, and lots of answers on the programming Q&A site StackOverflow. Usually the cream rises to the top. Just be careful. Make a habit of cross-checking information with documentation where possible and don't just believe someone because they used some big words.

Audio

There are quite a lot of podcasts related to development which you may find useful when you are away from your computer. Because of the nature of the medium, they often aren't focused on strictly technical information but on related discussions.

My experience was that early on, I found it useful and encouraging to listen to podcasts such as CodeNewbie podcast and hear stories of people who transitioned into tech. But afterwards, I wanted technical information and found it hard to find. I also found that a lot of podcasts were aimed at more experienced developers and I found the discussions alienating and discouraging because I didn't understand them.

I did find a couple of podcasts that were right for me, and I encourage you to do some digging to find the ones that are relevant to you. But I don't subscribe to the theory that you should be trying to absorb as much development information as possible, regardless of whether you understand it - I think this can often do more harm than good.

3. How you learn

Most people don't think about this factor, but it's vitally important. It's a huge topic in itself, on which there is decades of scientific research, but I will mention just a few key principles that will drastically improve your learning efficiency. (Sources to back up these points are listed at the end of this article).

1. The holy grail: Active learning

This is a very simple idea, but it takes effort, so it's often ignored. The truth is, if you sit and passively watch a video or scroll through some information, you are not going to remember most of it. We are exposed to a cacophony of stimulus every second of our lives - if we remembered everything, we just wouldn't be able to function. Our brain is designed to discard most information.

So if you want to remember something, you need to make your brain know it's important. Anything you can do to get your brain working will help you learn, such as:

Testing yourself. For example, make something by following a coding tutorial, then try to make it without the tutorial. It doesn't really matter how often you have to check the tutorial for answers - just trying to think for yourself will still be beneficial to your retention of the information.
Asking questions ('what does that mean?', 'is that what I expected?', 'what do I think is going to be the answer here?', etc.)
Coding along with a video (but trying to fully understand the information rather than just copying it out)
Being creative - applying your knowledge in a new context by making something new or customising something that exists
Cross-checking information
Trying to explain the information to someone else, e.g. in a blog post (one of my personal favourites!)

You don't have to wait until halfway through learning to do these things, you can do them from the beginning. For example, it has been shown that testing yourself on a topic you know nothing about improves your subsequent retention of information on that topic.

Bear in mind that active learning is harder. It will feel uncomfortable. In learning science this is often called 'desirable difficulty'. You will come to recognise the right level of discomfort, which is somewhere between comfort and feeling completely clueless. If you are feeling completely clueless you need to break the task down into smaller pieces or try something a bit simpler first.

Because it's harder, you need to take regular breaks. Breaks have been shown to reduce stress and improve productivity anyway, so it's all good. And active learning with breaks will still be more efficient in the long-term than passive learning without breaks.

2. Rehearsal

Even taking all the advice above, your retention of the information will still degrade naturally over time, for the reason mentioned above (we can't remember everything). You need to rehearse information to make it stick.

The maximally efficient method is to rehearse information just before you are about to forget it, but this is very difficult, as the timing of that point has been shown to depend on a lot of factors. So don't worry too much about this. When you are learning to code on a need-to-know basis, there are natural points for rehearsal: when you need to use something you learned and you realise you've forgotten it. In my experience, I usually learn the info quicker the second time, so the time I originally spent wasn't wasted, and that's good enough.

Just be aware that if you are taking the time to study certain concepts, it could be worth also going over some that you learned a week or a month ago. The main thing is, don't feel bad about forgetting things. It's not a personal failing - it's just how the brain works.

3. Variety

Variety is good for a few reasons. First, doing relatively short bursts of varied activities (called interleaving in cognitive science) has been shown across many fields of study to be more effective than studying one concept for a longer period. So if you were planning for example to learn about loops in JavaScript for an hour, you will probably be better splitting the time between loops, if statements and scope etc. instead, even if it doesn't feel as helpful at the time.

Second, you want to get a diversity of perspectives. In an age of free-flowing but not necessarily correct information, you need to make sure the info you hear is actually correct. And when it comes to matters of opinion, the world of development is rife with these, so all you can do is hear a variety of perspectives and make up your own mind. So vary your learning materials as well as your activities. If you always stick with the same teacher or resource, you are missing the opportunity to become a better developer.

Finally, varying the context of learning has been shown to improve learning. Learn in different places, in different ways, with different music, and so on, and you have a better chance of it sticking.

Troubleshooting

Almost as important as your ability to learn is your ability to find solutions when things go wrong.

Writing code for computers is hard, because they do exactly what we wrote, not just what we meant. An unintended effect of your code is called a bug. Debugging is a huge time sap and a crucial skill for developers, which is often neglected. Many developers don't think too much about their approach and just try lots of things until it works. Here's a quick summary of some steps I usually take.

Googling

The web is full of answers to development problems. People have written about them in countless blog articles or discussed them in forums. All developers, from junior to senior, google stuff all the time. There is simply far, far too much to know and no point reinventing the wheel when so many have been there before you.

For example, if you have a problem with a specific library, you may find answers or at least better explanation of the problem on the Issues page of the library's repository on GitHub. But your most common source for answers will probably be StackOverflow, an invaluable Q&A site for development questions.

When googling, you need to bear a few things in mind:

Phrasing is crucial. Don't ask the question one way then give up. Try a number of different possible keywords. Sometimes you will not know the word for the issue, but it will pop up in one of the search results. Then use that word in your search, and voilà! You find what you were looking for.
Don't expect to find the exact answer you need for your use case. Often you can get what you need from the answer to a somewhat similar problem, which will inspire you to try something that solves your problem, or you come across some code that can be adapted a bit to suit your use case.
As mentioned already, think critically about the answer you find. For example, don't just find one answer on StackOverflow and immediately copy some code. Look at the other answers and reaction. Cross-check what the person is saying and see if it is reliable, or if there are problems with the solution. If you don't, you will probably just cause yourself more headaches in the future.

While Google is often your first port of call, it isn't always. Some problems simply require basic programming knowledge to fix, or are very specific to your use case, and Google will be of less help here. I've found that when my code doesn't work and I find nothing via Google, it's either because I've just made some basic error somewhere, or my case is so niche that no-one has written about it.

Check the obvious things

Checking you haven't done something really stupid before you proceed will save you a lot of time as a developer. Check for typos. Check little things like whether your greater-than/less-than signs and true and false conditions are the wrong way around. Always assume there could be something really simple going wrong because this is quite often the case!

The next thing to check is the few things you think are most likely to be causing the problem. Just jump straight in investigating what is going on in those areas and you might get lucky.

In-depth debugging

If you've exhausted your other options, I find the best approach is a very methodical one. This is time-consuming but usually gets to the root of the problem. Problems usually arise in complicated, niche use cases. So first, create a more simplified version of the scenario where everything does work as expected. Then add things one by one, to get it closer to your real-life use case. Sooner or later you'll identify the thing that causes the issue.

Factor 2: Employability 👩‍💼

'Soft' skills are not really any less important for a developer than any other job, despite the stereotype of the unsociable programmer. True, you will probably escape being tested on these aspects, but you may be asked questions about them, and it will also greatly affect your application and how you come across at interview.

Let's look at how to develop and use these skills before and during interviews.

Getting an interview

Before you get an interview, you have one, simple goal: to get an interview. Forget about whether you'll get the job, forget about how the interview might go, forget about technical questions. Just do everything you can to make yourself stand out in a positive light - that means your CV, portfolio site and customised cover letter.

Of course, you don't technically have to do any of this (though you'll probably always need a CV). For example, you could get lucky and land an interview through someone you know. But if you don't, then you have no excuse not to do everything you can to sell yourself.

CV

CVs are still very relevant for developer jobs, and you should assume the employer will look carefully at it even if you have an online portfolio. You want your CV to be a simple, at-a-glance reflection of your best qualities and technical skills. Keep it to one page.

If you are a design-focused developer you can design something nice, but otherwise, don't fret about details like layout - it's window dressing. Use a nice, clean template that displays the information clearly and has a bit of colour.

Assuming you don't have much or any work experience that is relevant to development, I suggest you do a skills-based CV that has the following sections:

Title

The title can just be your name, and the subtitle is the job title you are after, or the specific one you are applying to. For example, my subtitle was usually Front-End Developer.

Don't say 'junior' developer. The company may or may not end up labelling you that way, but there's no point lowering your status yourself. Just calling yourself a 'developer' shows more confidence and is by no means inaccurate.

Profile

This is a bullet-point style tagline reflecting the key selling points - a couple of sentences at most. Here is the one I used: "Highly motivated, self-disciplined, self-taught developer. I create modern, responsive and accessible websites and applications."

Skills

List the technologies you are familiar with and other desirable skills e.g. test-driven development, responsive design etc.

Experience

Employers want to see experience that is relevant to a developer job. That means writing about the app you taught yourself to make is more relevant than some unrelated paid job. There's no rule that says you have to list all your paid work in your experience section - it's your CV, you can do what you want.

So create a sub-section for your self-teaching or bootcamp and summarise the key things you did. Any extra things you did like making something at a hackathon or contributing to open-source are good here too. Then you can add a brief sub-section on your most recent job, and put the rest of your work history on your LinkedIn so you aren't hiding anything.

Education

This can go last - just your most recent qualifications.

Finally, you will need your contact information somewhere, other website links (covered below) and a link to your portfolio.

Portfolio

An online portfolio is a chance to stand out a bit more to the employer, and a good place to show off the stuff you've built. It doesn't have to be complicated.

If your main focus is building websites (as opposed to web applications, apps or backend programming), you should arguably make the whole thing yourself, or at least heavily customise it using the CMS you intend to mainly work with professionally.

If not, I'd recommend using something a little more off-the-shelf and doing a bit of customisation, since you are going to need most of your time for working on the stuff to showcase. For example, I made a WordPress site for mine and integrated a paid theme. There was some hassle involved in learning to use WordPress, but I still ended up with an attractive, reliable site with a functioning blog in a relatively short space of time.

You want your portfolio to do three main things - emphasise your key selling points, showcase your projects and host your blog. This can be done in three sections or pages (depending on how your portfolio is laid out). Remember to also include a contact form or contact details.

Home / About

This will rehash some of the key points from your CV, but hopefully with a bit more style. So highlight who you are, your main attributes and skills, but add some nicer design into the mix. Look at other developer portfolios for some inspiration. Bear in mind, if you google for this you'll find some insanely good ones which aren't the most realistic examples for you - it's better to just find some good developers - blog authors, devs on Twitter etc. and look at their personal site.

You can also add a (brief) element of story about who you are to make it more interesting, e.g. briefly explaining what you used to do and why and how you are transitioning to tech - but remember to sell yourself along the way - emphasise why your skills suit tech and the characteristics and skills that helped you learn.

Projects

Secondly, you want a page to show off your projects. As well as linking to the deployed project and the code, talk a little about the project, why you made it, why you made the choices you did and what you learned. This will really show you can think and communicate intelligently about tech and give insight into your learning process, which is important to employers.

Blog

Thirdly, you ideally want a blog where you can write a little about tech and things you've learned. Yes, this is even more work, but it will show employers that you are enthusiastic, knowledgeable and eager to share that knowledge, which is a triple win.

Of course, you can and should blog on other platforms. For example, Dev, freeCodeCamp news and hashnode are popular platforms for free tech articles, and Medium is also still popular, though they have a paywall system. However, the best strategy is to have any articles you publish on these platforms be copies that link back to the original your own website via the canonical url section ('This article was originally published on...'). That way employers can see your writing on your own site and your site gets a SEO boost, plus you have full control of your original articles.

Many newcomers, and even experienced developers feel self-conscious about posting an article on technology. You could get something wrong, and some people might not like your article. However, if you use reliable sources of information and are not pretending to be an expert in the subject matter, you should publish it anyway. If something needs corrected you can make a correction, and anyone who responds with unconstructive criticism can safely be ignored.

Remember, this is helping you get a job, and getting a job is about putting yourself out there. The greater the exposure, the greater the chance of reward. And the majority of people in the developer community will be supportive of you sharing your knowledge.

Cover Letter

Cover letters are important - they highlight why you are right for this specific job, unlike your CV or portfolio. It's fine to have a rough template, but you need to tailor it for every application you make. Here are some key points to consider.

Format

Many people use the traditional Word-processed document approach, and so did I, but you may prefer to just write your pitch in the body of your email. It does seem a little antiquated to produce a 'letter' with the company address when you aren't actually posting it, though on the other hand it does show some effort and give it a little gravitas. Different employers will have different preferences so you just need to pick one approach - either will probably do fine.

Style

I see no reason to be formal in the content of the letter and I think an informal, natural style works best in this day and age. I start my letters with 'Hi,' and end with something like 'Thanks for your time', sidestepping those awkward old-fashioned introductions like 'To whom it may concern', or worse, 'Dear Sir/Madam'.

Content

You can find plenty of good pointers and inspiration online for this, and hopefully you already understand the main points (sell yourself, be persuasive, use evidence etc.) so I just want to emphasise some key points:

Be specific. It's pretty obvious when you've copied a template and changed the company name. Yes, you may have never heard of the company before but you need to mention something that attracts you to the company and shows you do know who they are.
Pitch you, not just your knowledge. Yes, you should mention that you tick off the main requirements for the job, but you are not going to be able to compete with someone with more experience on a purely technical level. You may as well acknowledge this, because you can bet the employer will already be thinking it. You have to argue that your drive, ability to learn and other personal attributes are what set you apart and will add more value to the company.
Be real and honest. You are asking the employer to take a punt on someone with no professional experience - it's not unusual in the world of development, but it's still a big ask. Your letter really needs to get across your genuine enthusiasm for coding, and hopefully remind the employer of when they were first starting out. Just using lots of buzzwords and jargon won't cut it. You need to get your story into their head.

Other websites

Since you have been creating personal projects, you should have a GitHub account set up so that employers can see the code. Ideally, the code will be clean and tidy and the commits will have sensible descriptions.

As for social media, at a minimum you should be on LinkedIn as this will expose you to some opportunities, though not all good quality ones by any means. The setup is pretty simple - add the info from your CV, add a slightly longer work history if you want, and make sure to check the option for showing recruiters you're open to work.

Other social media is optional but may be useful. Yes, building a following on a site like Twitter takes time and effort, but just being on there, making some connections and following some good people may still expose you to opportunities and useful information. Many people on there are doing the #100DaysOfCode challenge so you may find it useful to do that or follow some people who are.

Commercial experience

Some of you may wonder if you should take an internship of some kind to gain extra experience to bolster your application for a full-time job. I'm not going to say don't do this, especially if the opportunity presents itself to you, but I do think you should focus your effort on getting a full-time job straight away, since it is possible.

The position you start off with will affect how employers see you, and if you start off at a very low level, it might be quite difficult to advance quickly to a higher level. Focusing all your efforts on building very in-demand skills instead might take a bit longer, but you improve your chances of getting a good-quality job at the end of it.

That said, if freelancing is something you are interested in and you have an opportunity to do it, getting some paid development work under your belt will certainly improve your standing in the eyes of an employer.

Interviews

The interview process is different for each company, but there are some key things you'll definitely need to do and prepare.

Coding exercises

You may be asked to build something according to a set of instructions and then submit the code. This could be later on in the interview process but in my experience it was quite early on, to weed out candidates.

On paper, this is great for you because you have the chance to prove your skills, despite your lack of professional experience. In reality, you may not actually have the time to do the assignment to the best of your ability, or the requirements may be unreasonable.

Nevertheless, I suggest you treat these exercises as good experience at professional development tasks and give it your best shot. Submit it unfinished if you have to, and always ask for feedback (though you might not get it). With any luck, you'll get a task you can complete well in time, and pass this stage.

Company research

You will have done a little of this already, but obviously, you need to do some more before interview. Look at the company website, social media, linkedIn if possible. You know the drill.

Preparing answers

The main part of your interview preparation should be preparing answers to general interview questions about your skills and experience. How you communicate in general is so much more important than whether you know the answer to some technical trivia.

On paper

I like to get my talking points for a potential interview question down on paper first.

You probably want to start by preparing some raw material that might be useful for a variety of questions. For example, the story of the main application or website that you worked on - what was the task, what were the specific actions that you took, what was the result, plus any lessons learned.

You probably also want to look at common questions and make sure you have something for it. So you could cover all the main topics likely to come up, and allocate specific examples and talking points to them. This also ensures you don't end up talking about the same example over and over.

Topics that might come up are things like communication, teamworking, working under pressure and time management. Remember these 'soft' skills are still important for dev jobs and you need to be able to talk about them.

Out loud

This is really useful and I suspect a lot of people overlook it: you should practise actually saying the stuff you've prepared. First of all, because you need to make sure you actually remember it and don't end up rambling in interview, but also, you need to make sure it comes across well - your body language as well as your words. Saying answers you've prepared that promote your positive qualities probably doesn't come naturally to you, so you need to practise.

While practising alone or with someone you know (or better, don't know very well) is good, I find it really helpful to record myself practising on my laptop, so I can watch it back and see exactly what impression the employer will get of me. This is definitely uncomfortable, but I find it helps me identify lots of things I can improve on.

Preparing questions

There is no easier thing you can do to improve your performance at interview than asking lots of questions. It will show that you are enthusiastic, that you understand and are interested in the company, and you are thorough and focused on details. So once you've done your company research, jot down lots of questions.

You can get a lot of inspiration for this online. Asking about the tech is definitely worthwhile, and the opportunities for your development. There are also plenty of things you can ask about the company values, practices and strategy.

Be genuine with this - if you are going to start your dev career with this company, you do need to make sure they are a good fit and will support you and help you grow - otherwise you are hurting your long-term potential for short-term gain.

Preparing for technical questions

Full disclosure - in the four proper interviews I did before getting my first dev job, I wasn't really asked any 'quiz' or 'whiteboarding' questions, as such. Yeah, I got a bit lucky.

There is plenty of material online to help you prepare for these questions. But I think the best preparation is actually knowing your stuff by just coding regularly, reading around the subject and trying to improve as a programmer.

Yes, you may gain a short-term advantage by practising specific algorithm exercises and looking up commonly asked technical questions, but I still think your main focus as a new developer should be learning in the context of building something, rather than in the abstract.

The interview itself

There are 101 interview tips that people give, but let's focus on a few key things.

Confidence

Confidence is so crucial. The employer is really looking for someone that tells them with their tone and body language, "Yes, I could definitely do this job well, you don't need to worry about me". How do you get this confidence if you don't naturally feel that way?

Firstly, remind yourself of your positive traits. The things you've overcome. If you're coming into development from an unrelated field and stuck with it, then you've got a lot of things going for you and a lot of reasons to feel confident.

Secondly, imagine that you're going to get the job. If you don't believe that can happen, you probably shouldn't be going for the interview. Then keep that frame of mind throughout the interview - think of the interview almost like a formality before you will be given the job (without being arrogant, obviously).

Positivity

This is also crucial - employers want someone who will contribute positively to the team and relishes challenges. I find the above trick - imaging you're going to get the job - works really well for this too, because it allows your genuine excitement about the job to trump your nerves about the interview.

Also remember that you can learn anything. Whatever doubts you have about your ability, whatever tough questions you might be asked, you have a safety net, which is that with a positive attitude, you could learn whatever you need to know, even if you don't know it right now. And that is more valuable to most employers than someone who knows quite a lot but seems difficult to work with.

Salesmanship

At the end of the day, the interview is an exercise in persuasion. There's no scope for being modest or reticent. Drive the key points home. Add an extra sentence at the end of your examples that highlights how your unique strengths were crucial.

The best opportunity for salesmanship is the beginning of the interview, especially when a 'tell us about yourself' style question is asked. This is your chance to highlight your key selling points. Have something prepared that you know by heart. First impressions are very important. Nail it, and you'll set yourself up for success.

Factor 3: Number of applications 🧮

Now you have the technical skills, and you know how to impress employers. What next?

The only remaining variable at this point is how many shots you take. In general, more applications = greater chance of success. But, you have a problem. Applications take time, and it's not as if you can't also still improve your technical and soft skills at this point. Therefore, you need to prioritise your applications and give the ones you do make the greatest chance of success.

Good quality jobs

You should already have a narrow idea of the job you're looking for, as discussed above. You also want to make sure that the prospective job is worth your time.

Many developer job listings come through recruitment agencies, and they barely even tell you anything about the job. My advice is don't waste your time on these. If you don't know what you're applying to, you can't make a good application. Personally, I found that Glassdoor Jobs and LinkedIn Jobs were good sites for better quality job adverts.

Find a job advert that you can apply to directly and has a clear, relatively short list of requirements where it sounds like the company knows what they are talking about. Ideally, they sound like they have a good company ethos or even specify that self-taught developers are welcome. You want to spend a lot of time on this application and make sure it's very personalised and as good as it can be.

If you find the job advert on a third-party website, see if the job is advertised on the company's site and apply there instead, ideally via an email address. As a general rule of thumb, the more direct to the company your application is, the better.

Of course, most job adverts won't be ideal. Sometimes you have to play the numbers game with jobs that don't seem quite as promising - they might actually turn out to be good opportunities once you get further along in the process. And even if you don't think you want the job, you should never turn down interview process experience at this stage in your development career - the practise will help you get the job you do want.

Following up

The advantage of applying very directly to the company is that you can follow up. Following up is key. An application doesn't count for much if it is ignored. But don't take it personally. Often you just need to give the employer a nudge.

If you can find the email address for the company's HR department or general enquiries, follow up your application, say a week later, asking if they received it and re-emphasising how keen you are on the opportunity. If you still get no reply, there is no harm in trying again after another week or so has passed.

Personally, I only got my current job because I followed up on my application submission with an email. If you don't ask, you don't get, and if you don't get, ask again!

Networking

Using your network is a great way to hear about good quality jobs, and being referred for a job by someone else is a great way to ensure your application isn't ignored.

Most of us aren't very comfortable with networking, but since it can help you get a job, you should give it your best shot. At the most basic level, it just means letting your family and friends, or really just as many people as possible, know that you are looking for a specific type of position.

You should also be able to find various tech events and meetups in your area (less so during the pandemic, but there should be some online events at least). These can be useful just for your general knowledge of tech and the tech industry, but with the added bonus of making connections with people. The worst that can happen is you have wasted an hour or so.

Also try to make the most of sites like LinkedIn and Twitter for making connections - you never know how it might pay off, so it's worth putting a bit of work in.

The obstacles 🌩️

You should know now what you need to do to get your first developer job. But every journey has obstacles. You should be prepared for the long haul.

Articles like this one lay things out in a straightforward way, but the journey of learning to code is rarely straightforward and linear. Inefficiency is inevitable, especially if you are completely self-taught. You'll sometimes try to learn something at the wrong time and not succeed. You'll 'waste' time on silly mistakes or trying to get something set up that should have been simple. You'll seem to 'fail' countless times and probably feel 'not cut out for this'.

I can't tell you what other global or personal calamities or inconveniences might befall you along your journey. You probably won't have any control over them, and you'll almost certainly experience certain 'negative' emotions in response.

Luckily, there is usually something you can do about your internal world. You can train your mindset and responses to the emotional challenges of a big task like this one. The key here is that you've made a commitment to achieving the goal, and to continuing with the actions you need to do until you succeed, whenever that may be. Your secret weapons: vision, habit and persistence, can see you through.

Of course, it's much easier to give up and say 'I'm no good'. It's your brain's defence mechanism against having to do hard and challenging things. But there's no use in letting one bad day turn into two. You have to get up the next day and try again. Because eventually you'll see that "I'm no good" was never the reality of the situation. The reality was: "this is a learning process".

Overwhelm and stress

Programming for modern computers dates back to the 1950s and the technology has developed and changed at breakneck speed since then. Even in the field of web development, which is under 30 years old, enough technology has developed that you could spend a lifetime learning it.

The fact that most of the information (and misinformation) on this, plus millions of people's thoughts and opinions on it, is all on the web, unfiltered, for you to find with a simple google search, can be just as much a curse as a blessing.

You will encounter more information than you've ever found on a topic before, and more topics than you ever imagined there would be. It's natural to be daunted by the sheer amount of information, and the content of it, especially when much of it is aimed at more experienced developers and uses a lot of terminology you don't understand.

Combine this with the pressure and difficulties of building new habits, job-seeking, the interview process and any time and financial pressures you have, and you have a recipe for potential stress, overwhelm and burnout.

The best thing you can do is just try to look after yourself, listen to your needs, take breaks, keep doing things you like, try to stay healthy, and exercise. Not ground-breaking advice, but sensible nonetheless. You will get to the end of the path in your own time, and it will be worth it.

I can also give you a few words of comfort as you start out on your journey:

You will find good teachers and resources that explain things in terms you can understand, and make you feel at home in the world of development. You will also find friendly and welcoming people in the development community that want to help and mentor you, if you look for them.
If you accept that there will always be a vast amount of things that you don't know in development, but find your own corners of knowledge that you can master and feel comfortable in, you'll be happier.
When you do finally find yourself in the right developer job, a lot of the pressure will be relieved, and you'll be able to enjoy the fruits of your labour, safe in the knowledge that despite all the obstacles, you finally made it!

Thanks for reading, and feel free to contact me here or via Twitter if you have a question about this that I haven't covered here 🙂

Sources

Some of the tips on habit are derived from information in The Power of Habit by Charles Duhigg (Random House, 2012) and The Chimp Paradox by Prof Steve Peters (Vermilion, 2012).

The idea that you lose willpower as the day goes on is known as ego depletion and various studies have shown this effect, though the exact causes and nature of the effect are still up for debate. This article provides a good overview:

What Is Ego Depletion? - verywellmind

The information on learning science comes from the book How We Learn by Benedict Carey (Macmillan, 2014). But I've included some online articles on the relevant topics here if you want to explore the ideas further:

Properly Understanding the DOM

Josh Carvel — Sat, 12 Dec 2020 11:12:39 +0000

Intro 🧰

If you're a front-end developer, you've probably heard of the DOM, or used some DOM methods in JavaScript. However, you may not know exactly what it is, or how it works.

This article will give you a solid understanding of the DOM and how it fits in with the rendering of webpages on the screen. Along the way, we'll cover some crucial concepts to do with JavaScript objects, the browser and rendering. This will help develop your expertise in web development and make you more productive with the tools that the DOM provides, even if you are using a JavaScript library or framework.

Prerequisites

Some familiarity with HTML, CSS and JavaScript

The browser 💻

Firstly, we need to understand the web browser on your device a little better. In this article I'll refer to three core components of browsers.

The first is the rendering engine (also called the browser engine), which reads HTML and CSS files and renders (outputs) the content on the screen. This is the component that creates the DOM! It can actually be used outside the browser, for example email clients use a rendering engine to display HTML email. You may have heard of the rendering engines used in popular browsers - Blink (Chromium browsers, i.e. Chrome, recent versions of Microsoft Edge and many more), Gecko (Firefox) and Webkit (Safari).

The second component is the JavaScript engine, which reads and runs any JavaScript files given to it. Again, this is a standalone component that can be run outside the browser. The most popular one is Google's V8, used in Chromium browsers and by NodeJS/Deno. Firefox uses SpiderMonkey and Safari's is called JavaScriptCore.

The third is the JavaScript runtime environment. This is some code that allows the JavaScript engine to access features relevant to the environment it is running in. So in a web browser, it provides functionality specific to the browser, such as interacting with the DOM. NodeJS, for comparison, provides a different runtime environment for the JavaScript engine that is specific to non-browser environments such as a server or the command line.

These components work together inside your browser to produce webpages. They tend to be written mainly in the programming language C++.

The core functionality that browsers provide is, like the web itself, not centralised, but based on certain standards. When referring to the features browsers make available to developers, I'll refer to the Mozilla Developer Network web docs rather than the actual standards, because they are a more accessible guide to the tools available to us and how they are implemented in different browsers.

The global object 🌍

Another thing it's important to understand properly is objects in JavaScript. In programming, we describe the world with objects - little containers of data that link to other data.

Let's imagine for a moment we wanted to describe the whole world. That object would have a lot of things on it, i.e. properties. Things that exist in nature like trees, human inventions like the mobile phone, and things you can do like 'eat cake'. The last one would be a function in JavaScript, and the property is called a method in that case.

In our example, the world object is the 'place we put all the stuff'. JavaScript also has a place like this, and it's called the global object. Assuming my JavaScript is running in the browser, the global object contains properties and methods related to the browser and the webpage.

It's quite hard to define what the global browser object actually represents. Your webpage runs in a tab, with unique elements and events happening. A page in another tab is separate, running different JavaScript with its own global object. So we might call the global object the 'tab' object. But you also have access to browser properties, like browser history and storage for example. So what should we call it?

Well, the browser provides it in a variable called window. But it doesn't exactly represent a user interface window. It's just a label for the 'place we put all the stuff'. JavaScript makes it easy to access this place - we don't need to specify window to access the stuff on it, just saying someProperty is the same as saying window.someProperty (in most cases).

The definition of what the browser should provide on the window object has been standardised, using interfaces. This is an object-orientated programming term which refers to the description of an object, rather than the object itself. Though an interface is generally a point of interaction, here it means the description of an object, because that enables the interaction of objects to happen smoothly, since they know what properties and methods another object has.

Here's two things we should know about interfaces:

The interface name is written in PascalCase as a convention.
Interfaces can take properties and methods from other interfaces, by inheriting them from an ancestor interface, or getting them from an unrelated interface called a mixin. We'll see this later.

Web APIs 💬

Here's MDN's documentation on the interface for the window object: Window.

Have a look and you'll see there's quite a lot on there. The functionality the browser gives us to communicate with it is known as Web APIs.

API stands for application programming interface. In other words, someone wrote an application, in this case the browser, and they also wrote a set of features and rules so you could interface (interact) with it using programming.

For example, let's say you use fetch() in your JavaScript code to get a resource from the internet. That's not part of the JavaScript language - you couldn't use it in JavaScript not being run by a browser. But in a browser you can use it, because the browser attached the fetch method to the window object when it created it.

When you call fetch(), or any other Web API method, you are making use of the runtime environment provided by the browser. The main difference with these methods is that they are asynchronous, which means they don't necessarily run immediately after the previous command in your JS code - you make a request for an action, which is queued up and runs when possible. For example in the case of fetch(), there will be a delay while it gets the requested resource.

The Web APIs make use of objects with properties and methods, just like the window object. In the fetch API, one of these is the Response object. The API defines exactly what the structure of the object should be.

But we're not going to talk about all the weird and wonderful APIs available to us in the browser: we want to know what the DOM is. There's just one more thing to look at first: a property of the window object called document.

Documents and trees 🌲

Just like how the window object is the container for almost all of the 'global' stuff (console, scrollbars, window dimensions etc.) in your browser, the document is a container for the content, i.e. the webpage itself. It represents what you give the browser, not what's already there. This can be an HTML, XML or SVG document, but we're just going to talk about HTML.

You can give your browser a HTML file by asking it to open one stored locally on your device, or you can request to view a website, causing the browser to retrieve the file from that website's server via the internet. The browser's rendering engine (mentioned at the beginning) then does two things: parse the HTML (read the code line by line), then create a tree of elements.

When I say create a tree, I'm not talking about planting. It's one way of storing data with a programming language, by creating objects that have 'family' relationships between them. These 'family' relationships are the same you create in an HTML document.

The relationships are defined by edges (which clearly ought to be called 'branches', but never mind...). The objects at the end of an edge are known as nodes, because this means the place where lines join (it's also the place where a leaf and stem join on a plant, so it's a bit closer to the tree metaphor). But remember, a node is still just a type of object.

The node at the very top of the tree is called the root. Visually, the structure would be sort of like a tree. What the browser creates is known as a document tree: a node tree where the root node is a document. It stores information about the document in that root node, and each HTML element on the page and any text inside them also has its own node.

Enter the DOM 📄

Let's finally talk about the DOM.

The DOM, technically, is not the document tree, i.e. the data structure itself. It's the model that describes how the data should be stored and interacted with. However, you will often hear people saying things like 'manipulating the DOM', which is simpler to say than 'manipulating the document tree'. I'll use DOM in this sense too, for convenience.

The technical term for it is an 'object model', which means it defines some objects and how they can be manipulated, but we don't need to worry about that. Just know that's what DOM stands for: Document Object Model.

The key thing is that the DOM is one of the browser's Web APIs. We can get information about (read) DOM nodes and change them (write) using JavaScript. We know how to do this because it's described in the interfaces for the DOM API.

To be clear, the DOM is a generic API for manipulating documents. There is a specific offshoot for HTML called the HTML DOM API (remember that other types of documents can be modelled by the DOM). But this distinction doesn't really affect us practically.

We can see the interfaces we need in MDN's documentation on the DOM and HTML DOM. (The 'official' description is currently WHATWG's DOM Living Standard, and the HTML DOM is defined in WHATWG's HTML Living Standard.)

Using the DOM 👩‍💻

Let's use an example to understand interfaces.

In my JavaScript (which the browser's rendering engine discovered in my HTML document via the <script> tag, and the browser's JavaScript engine is running with window as the global object), I have access to the document object, as discussed.

It's described by the Document interface. On the list of methods, you will see Document.querySelector(). This lets me use CSS selector syntax to get an element from the document - in this case, an HTML element, because our document is HTML.

Now say I have an <input> element in my HTML file with an id my-input. I write the following in my JavaScript:

const input = document.querySelector('#my-input');

When the JavaScript engine parses my code, it will need to work out the value of the input variable. The querySelector() call triggers the runtime environment to go find the right element (C++ object) in the document tree (provided by the rendering engine), convert it to a JavaScript object, then give it to the JavaScript engine. If it doesn't find one, it returns null, a primitive value in JavaScript essentially meaning 'no value'.

In my example, I now have a variable pointing to the element object. Specifically, it's a HTML input element, described by the HTMLInputElement interface (part of the HTML DOM). You can see from the properties listed that I can access the value (the text) in the input and read/write it. Pretty useful.

Now looking at the methods, you'll see things like blur() and focus(). Very useful too. But look at where they come from - they are inherited from HTMLElement. My input is a type of HTMLElement, so it gets properties and methods shared by all HTML elements.

The inheritance doesn't stop there - HTMLElement is a type of Element (now we're back in the generic DOM API). There's some useful stuff there too, like setAttribute(), so I could add, say, a class on my input field in certain circumstances.

Let's keep moving on up. An element is a type of Node. We know what those are. Element isn't the only type of node - Document is also, of course, a type of Node, since it's the root node of the tree. And we mentioned before that the text inside an element gets its own node, Text, which you can read/write from the node with the textContent property.

Note: we may be confused here because there is also an HTMLElement.innerText and an Element.innerHTML property. As MDN explains, these properties have poorer performance and innerHTML can leave you vulnerable to cross-site scripting (e.g. I get the value from my input and set the innerHTML of a div somewhere else to whatever it is - someone could have written a <script> tag with malicious JavaScript code that will be run on my page). So if I just want to add text to an element, textContent is the better property to use.

Now we get to the top of our chain of our inheritance - all of these are a type of EventTarget. And so is Window. This allows me to add or remove event listeners, which allow me to respond to page events (like clicks) with a JavaScript function.

One last thing to discuss here: let's say we used Document.querySelectorAll() to get all inputs of a particular type. Note that it returns a NodeList. That's annoying, why not a JavaScript array? Well, remember that the DOM isn't part of JavaScript - it's language-independent. You could use DOM methods in Python, for example. That means working with DOM objects in JavaScript isn't quite like working with any other kind of object.

The DOM in DevTools 🔨

Handily, browsers give us some nice tools that help us view and interact with the DOM.

Here I opened Chrome developer tools on the Google homepage and inspected their festive logo img element:

The Elements tab shows us the image tag and its place in the document. It looks like it's just an HTML tag, but it's not. We could see the original HTML by right-clicking the page and selecting 'view page source'.

In fact, the Elements tab is a visual representation of the DOM, and the elements in it are objects.

Let's prove this by going to the Console tab. If we enter $0 (the Console shortcut for logging the element currently selected in the Elements tab) this will just show us the same representation. But if I use console.dir I can see the object:

Here we can see all the object's properties, including those inherited properties.

In JavaScript, the object an object inherits from is called its prototype, i.e. the thing you base something else on. Our image element inherits properties and methods from its prototype, 'HTMLImageElement', which in turn inherits from its prototype, 'HTMLElement', and so on. This is a prototype chain.

We can see the prototype object by expanding the __proto__ property. If we kept following the chain up we'd end up at Object, which is the object that contains the properties and methods all JavaScript objects inherit. This is just for demonstration - you won't need to do this.

All of these objects in the chain, except the actual image element, already existed on the window object of the JavaScript engine. If you did console.log(window) on a blank HTML page you could still find them. When I accessed the logo img element using the DOM and it became a JavaScript object, its prototype chain was set with those objects.

The property values were either provided as attributes in the HTML image tag, set using the DOM API in JavaScript, just known by the browser e.g. properties relating to dimensions, or have remained as default values since the object was created. If you just create a plain image element without any further information, the values are all defaults.

Hopefully you now have a better idea of what DOM objects are and how to inspect them. If you want to learn more about inspecting the DOM with Chrome devtools, Google provides a guide here.

Rendering 🎨

Now we understand the DOM and how to use it, let’s look more closely at the process of rendering a page, so we can think more carefully about how we use the DOM.

Any site you visit is essentially an HTML file (the 'document'), with references to other files (HTML, CSS or JavaScript) which are all stored on a server and sent to the browser via the internet. The browser parses the HTML and starts constructing the DOM.

However, JavaScript can affect the parsing process. If the browser gets to a <script> tag in the HTML, it will pause DOM construction by default while the JavaScript code in the <script> tag is executed, because the JavaScript might alter the HTML content by using the DOM API.

This is why it's often advised that you put the <script> tag at the bottom of your HTML, so the HTML can be loaded first. Alternatively, you can change the default behaviour by using the defer or async attributes on the script tag.

The browser also creates a CSS Object Model (CSSOM). This is similar to the DOM, but instead of representing your HTML document, it represents your CSS style sheets and their content with interfaces.

It’s an API, so you could interact with it to alter your styles, but you are usually better off defining all the styles you will need in your stylesheet first, then if necessary changing what they apply to using the DOM, by altering the class names on your elements (or using the style attribute on the elements if you prefer).

To get ready for rendering, the DOM and CSSOM are combined to create another tree, the render tree. Anything that won’t be displayed on the page, e.g. the <head> element, is excluded. The render tree contains all the information the browser needs to display the webpage.

The browser assembles the layout of elements on the page (like doing a pencil sketch before a painting), then paints the elements to the screen.

This means that if we respond to user interaction on the page by changing the DOM, the browser will have to do some work to re-layout and repaint items on the page. This has a performance cost, and could be what we would call expensive in performance terms. However, the browser responds to events efficiently as possible, only doing as much re-layout and repainting as necessary. This is explained in Tali Garsiel's research on how browsers work.

Bear that in mind, because there is sometimes a misconception that the reason we have fancy front-end frameworks is that the DOM itself is slow. That wouldn't make sense - frameworks still have to use the DOM, so they couldn't possibly make it faster. Really, it's all down to how you use the DOM.

Let's look briefly at the history and present of DOM manipulation to understand this.

Libraries, frameworks and plain JS 📚

You will often hear about JavaScript libraries and frameworks. A library gives you additional methods written by other developers, and you can call those methods whenever you want. A framework has more control of your application architecture, so it calls the functions in your code when appropriate, not the other way around.

For a long time, jQuery was the standard way to write JavaScript. It's a library that was created in 2006 to make DOM manipulation easier at a time when the DOM API was limited and very inconsistently implemented by browsers. It's still used today and some people like using its concise syntax, but its core functionality can now be achieved in modern browsers using plain JavaScript.

Modern libraries and frameworks don't need to tackle deficiency in the DOM, but they do aim to improve your efficiency and productivity in using it. It's not the sole reason they exist, but it's a big one.

If you are writing a simple website with limited user interaction, you probably won't run into the efficiency problem, provided you're not doing something very silly performance-wise with your DOM manipulation. But simple sites are not all we have on the web today – web applications such as Facebook are very common.

These applications contain dynamic, constantly changing content that heavily rely on user input and pulling new data from the server. JavaScript pulls the strings of these changes and is central to the operation of the application. This is a big departure from what the whole infrastructure of serving webpages to the browser was originally designed for. But the problem is not that lots of changes need to be made, it's how to tell the browser exactly which bits need to change, so you're not re-rendering more than necessary, and to do so without causing any bugs.

The core front end libraries and frameworks most used today are React, Angular and Vue.js. These aim to take efficient DOM manipulation off your hands, so there is more emphasis on what you want the page to look like, not how this should be achieved. If you want to make web applications professionally, your best bet is to simply pick one of these frameworks and learn it (you don't have to, but most companies use one of them or one like them).

If you are making simpler websites or are just curious to learn the DOM API, there are lots of guides to plain JavaScript DOM manipulation, like this one by MDN.

Conclusion

Let's recap the key points:

The DOM is an API provided by browsers, but the term is also often used to refer to the document tree. The document tree is a model of your HTML document created by the browser's rendering engine.
The browser window is the global object in the browser's JavaScript engine. This gives you access to JavaScript runtime environment functionality, including a JS implementation of the DOM API. The DOM API allows you to interact with document tree objects, which are described by interfaces.
Front end libraries and frameworks can help improve your productivity with the DOM, but you should be aware of why you are using them to ensure you get the best out of them.

Thanks for reading and happy DOM manipulation! 🙂

Sources

I cross-reference my sources as much as possible. If you think some information in this article is incorrect, please leave a polite comment or message me with supporting evidence 🙂.

* = particularly recommended for further study

Document tree image credit: Birger Eriksson, CC BY-SA 3.0, via Wikimedia Commons (side banner removed)

This article was updated on 24 April 2021, mainly to include mention of the JavaScript runtime environment.

Networks, the Internet and the Web Demystified

Josh Carvel — Sat, 05 Dec 2020 12:24:40 +0000

Intro 💡

Computer networking is a confusing topic. So much jargon. So many complicated technologies. As a new developer, it's easy to get overwhelmed by it.

In this article I'll demystify the fundamentals by tackling them one at a time, in simple language. You'll understand the different types of computer networks, and exactly how the internet and the web work (and if you don't know the difference between those two, this article is definitely for you!).

We'll start from the beginning of computer networking. As a developer in the 2020s you might not care about computing in the 1960s, but it's going to help you build up concepts one at a time and not get overwhelmed. It's also a good reminder of the incredible progress we've made, from networking a few huge mainframe computers for government research, to you reading this article on a web browser from anywhere in the world!

Prerequisites

Basic knowledge of computing concepts such as bits and operating systems - see my How Computers Work series.

The ARPANET 📦

Computer networking began with the Advanced Research Projects Agency (ARPA), an agency of the US Department of Defense, established as part of its efforts in the space race in the 60s. ARPA funded computers at various universities, at a time when computers were rare, large and expensive.

Within those institutions, there was a system of timesharing, where multiple users could connect to the computer with separate physical terminals containing a monitor, and the computer was powerful enough to serve all the users. Users could share files and even send what could be described as email as long as they were connected to the same computer.

But the US government wanted ARPA researchers to be able to connect to the computers and allow results to be shared easily. Initially, an ARPA director called Bob Taylor was connected to three separate university computers via three separate terminals. He wanted to access them all from a single terminal, and secured the funding for what became the precursor to the internet, the ARPANET.

Data would be sent over phone lines, since the infrastructure already existed. However, in a phone call, you intend to speak to each other continuously for a set period, during which a physical line of communication is reserved, and then stop, at which point the line is freed up. Computers, on the other hand, should be able to send messages to any other computer at any time. A Welsh computer scientist called Donald Davies had already considered this and the ARPANET used a system he had devised: packet-switching.

The user-facing computers on the network, known as hosts, would communicate in short chunks of information known as packets. A long message could be split into multiple packets which could travel down whichever phone line was available. So a single line could be handling packets from various different hosts, one after another. In between all the host computers were reliable, single-purpose computers which directed the traffic. They read the destination address from the packet and passed it on until it got to the destination host. Today we use routers for this.

The other advantage of this system was that it provided protection from lines going down - packets had many, many ways of getting to their destination. This was important for the US military in the 60s, and the implementation of packet-switching was strongly influenced by the work of Paul Baran, who described a communication system that could survive a nuclear strike!

The first ARPANET message was sent in 1969 and the network developed throughout the 70s.

The Internet 🌐

Packet-switching established the system for sending messages, but ARPANET also needed a set of rules about how to use that system: a protocol. The protocol included that packets would conform to a standard format, with the data to be transmitted, called the payload, plus a header, containing the address it was going to.

However, from the early 1970s onwards, other computer networks began to emerge that used different protocols. You couldn't send a message from one network to another - the other network wouldn't understand it! This problem eventually led to the development of a way of inter-networking, or 'internetting'. All these networks working together became what we call the internet.

The internet started to really get going in the late 1980s. Cisco Systems began popularising the modern router - a device that could connect your network to other networks. Each network just had to know how to talk to the router, not the computers on the other network.

There are two important protocols which allow this happen.

Internet Protocol (IP)

This is a fairly straightforward protocol derived from the original ARPANET protocols. It requires that a packet has a header with an IP address, which indicates where it's going. An IP address is a unique sequence of binary digits. Addresses are managed globally by the Internet Assigned Numbers Authority (IANA) and assigned to your device by your internet service provider. It is dynamic i.e. not permanent, so addresses can be reused when possible. Despite this, in the 1990s the world began to realise it would run out of IP addresses.

The old system, IPv4, used a 32-bit number, separated by dots into four 8-bit numbers displayed in their decimal equivalent, e.g. 172.217.7.238. We officially ran out of unique IPv4 addresses in 2019 and this system is very slowly being phased out.

The new system, IPv6, uses a 128-bit number, separated by colons into eight 16-bit numbers, displayed in their hexidecimal equivalent (the base 16 number system, which uses the digits 1-10 plus the characters a-f, so it's shorter to write), e.g. 2001:cdba:0000:0000:0000:0000:3257:9652. This system was chosen for various reasons, but we can also safely say we won't run out, given that it can generate over 340 trillion trillion trillion unique addresses!

Just type 'my IP address' into google to see your current IP address.

Transmission Control Protocol (TCP)

While IP is used by routers, TCP is used by the host computers on either end of the transmission of data - the routers in between don't read the TCP information on the packet.

A TCP header contains a lot of useful information. It contains a sequence number which is used to determine the position of the data in the completed message, so if packets arrive out of order, the part of the destination host's operating system that knows how to deal with TCP will reassemble them.

It also has a system to ensure data is not lost, known as the TCP handshake or three-way handshake. It uses flags, i.e. a bit in a certain position in the header which is allocated a specific meaning and set to 1 or 0. In the first transmitted message, a flag referred to as SYN (synchronisation) is set to 1. The receiving host responds with a message with the ACK (acknowledgement) flag set to 1. Then the first host also sends a message with ACK set to 1.

After three messages, the hosts are confident of a good connection. Ideally, the fourth packet sent contains some actual data. However, if acknowledgements were not received, attempts will be made to retransmit the data (subject to a maximum number of attempts). Any duplicates received at the destination will be discarded.

TCP also enables a receiving host's operating system to know what program the data is meant for. Like a ship arriving at a harbour, TCP packets arrive at ports. This is simply a unique identifying number, and like IP addresses, IANA maintains an official list of ports for specific uses. A TCP packet specifies the number of the destination port as well as the source port it has come from. Note that the combination of an IP address and port is generally referred to as a socket or endpoint and represents one end of a TCP connection.

Internet protocol suite

TCP and IP, designed by Vint Cerf and Bob Kahn and finalised in 1978, form the basis of what we today call the Internet protocol suite, often referred to simply as TCP/IP, though it also contains many other protocols.

One such protocol which is commonly used is User Datagram Protocol (UDP). Invented in 1980, this allowed for messages to be exchanged without the initial handshake, meaning messages are transmitted more quickly, but with greater unreliability. Today it's used by applications such as video chat apps, where speed of transmission is key.

Types of network 🏢

Let's now zoom in to the kinds of networks that were being linked together by the internet.

There are many different classifications of network, mostly based on their geographical extent. For example, ARPANET was a Wide Area Network (WAN). A small, local network on the other hand, is known as a Local Area Network (LAN).

One of the first LANs was established in 1973 at Xerox's research centre, the famous Xerox PARC. At Xerox PARC they used personal computers before most of the rest of the world had even seen one, and a fast, scalable way of connecting them was invented, called ethernet.

Every computer on the network had a unique media access control (MAC) address. Packets of data were sent along a cable containing what's called an ethernet frame, which includes a header with the source and destination MAC addresses, and a data payload. The data was passed to all connected computers, but the only intended recipient processed it.

Only one computer could send a message at one time, otherwise there would be a collision. If the computers detected the network was busy, they waited for a period of time before attempting retransmission of that data - the period of time was increased exponentially each time the network was busy, which is called exponential backoff.

The configuration of the connections between computers is known as the network topology. The ethernet at Xerox PARC connected computers with a single cable, which is called the bus topology. Later, it became possible to plug several cables into a device called a hub, which could relay the signals as if the computers were all connected, but one cable going down wouldn't crash the whole network. Nowadays it is more common to use a switch in place of a hub. A switch learns the MAC addresses of the devices on the network so it only sends data to the computers that need it, allowing multiple communications between different computers to happen at once.

Using a hub or switch in this way is an example of the star topology (the hub or switch is the centre and the other cables feed out from it). There are many types of topologies, each with their own pros and cons, but star is the most common these days. For example, your home network probably uses a star topology, with the router as the central point and your devices the points on the star.

Ethernet spread to many offices in the 80s so users could do things like share files and connect to printers on the network. For this to work, the computers needed a network interface controller (NIC, also known as a network adapter). These days they are built into motherboards. The operating system of the computer also needed facilities for networking built in - again, this is now standard.

At this time servers, such as file servers, mail servers and print servers, also came into widespread use in offices. The term server usually refers to a computer that has the dedicated purpose of providing some service to another computer, known as the client, although it may also refer to server software, which today can run on pretty much any computer. Most servers are dedicated machines that have no monitor, are in constant use and have a specific operating system and hardware optimised for their serving function.

Having a server-based network allows a network administrator to manage things like data storage, security and access to files, though it comes with its fair share of expenses. The opposite of a server-based network is a peer-to-peer network. This means the computers are connected together on an equal footing - there is no administrative control. We could say it's more 'democratic'.

The ARPANET and early internet were all about peer-to-peer networking, where files could be shared freely. But as businesses adopted networking and security concerns became more apparent, things became more siloed. Files could be restricted and you could set up a firewall to block certain traffic to your network. Later on, when sites like Napster emerged, peer to peer file sharing went mainstream again, although the market for sharing copyrighted material was quickly shot down by legal challenges. Nowadays, we see new types of peer to peer networks facilitating cryptocurrencies like Bitcoin.

Modern networking

Internet connections originally used phonelines in the traditional way with a dial-up connection that hogged the phoneline and was pretty slow for data transfer. They were used with a modem that modulated digital data to analogue for the phone line, and demodulated it back to digital for the computer (hence, mo-dem).

In the early 2000s, broadband came into widespread use. It divided the phone line into several channels, allowing more data to travel down the lines (greater bandwidth) and enabling multiple connections at once, specifically using digital subscriber line (DSL) technology. This was usually asymmetric (ADSL), which is optimised for downloading information more than uploading, which suits how most people use it and reduces the scope for interference on the line.

However, most of the internet is now connected with fiber-optic cables. They go all over the world and under the oceans, carrying digital information at incredible speeds using light. ASDL is still often used, but only at the last leg of connection to the user (the 'last mile'), where it is less cost-effective to install fiber-optic cables. In either case, some kind of modem is used to translate the signal, but this is often built into the same device as the router.

The router allows access for multiple devices on the local network, using the MAC addresses of the devices - assigned by the device manufacturer - to identify where to send the data. Most routers can send data wirelessly over short distances using radio waves, i.e. Wi-Fi. The other devices have a wireless NIC built in so they can understand the signals. Ethernet can still be used and is often used by businesses for a fast, reliable connection.

Outside of local networks, we can also get wireless internet access from our mobile network operator. The same towers that connect our devices for calls are used to send and receive data to and from our devices using radio waves, and are connected to the internet via cables. This is quite cost-effective to implement and has been improved over a number of 'generations' of technology, the latest being 5G (fifth generation).

But there's one development that has impacted us more than any other, and it's what you're reading this article on right now.

The World Wide Web 🕸️

The World Wide Web and internet are often conflated, but as we know, the internet came first.

The idea for the web originated with the idea of hypertext. Inspirations for it date back to the 40s and the term was coined in the 60s. 'Hyper' comes from the Greek for 'beyond', and the idea was having text with hyperlinks that take you beyond that text, to related information. Some systems were developed to implement this on computers, including one written by scientist Tim Berners-Lee while working at CERN in 1980. It was a web of sorts, but certainly not world-wide: the information was accessible to just him.

Over the course of the 1980s, CERN became the largest node (point of connection) on the internet. Berners-Lee came up with the big idea that a hypertext system could exist on the internet, so everyone could access the same information from their own computer, without having to log into another system or ask someone else for it.

In 1991, Berners-Lee wrote the world's first web server. A web server is a server that serves files known as webpages.

Let's look at the technologies that make browsing webpages possible.

Domains

Since the birth of the internet, there have been domain names. A domain represents a computer or group of computers on the internet, and domain names provide a more user-friendly description of a location on the internet. Since 1985 there has been a dedicated system to match domain names to IP addresses, called the Domain Name System (DNS).

This means a request for 'google.com' will go to a DNS server, and that server may pass the request on to another DNS server, until one that actually knows the address for that domain name is found, and that server will pass the request to the google server that has the page. In this example, there are many servers across the world that can serve you 'google.com', even though the domain name is the same. Once your computer has the IP address, it will remember (cache) it for a while, so it doesn't have to ask again.

The top-level domain is at the end of the domain name, for example .com (commercial), or .org (organisation). Management of these is overseen by IANA. Any name preceding this last dot is a subdomain. So 'google' is a subdomain of '.com', and in 'play.google.com' (Google play store), 'play' is a subdomain of google.

'www' is in fact just another subdomain, originally just added for extra clarity. It became a standard by accident. You don't have to include that subdomain when you register your domain name, and you can redirect requests using 'www' to the domain without 'www'. Because the web is so dominant now, people generally avoid specifying 'www', and browsers usually add it automatically in the address bar.

As you know, we refer to the collection of webpages at a particular domain name as a website. A website is just a hierarchical collection of files (the webpages) on the web server. The server serves the homepage when no specific page is requested. To access a different page, the file path identifying that directory on the server is added after the domain name.

HTTP

Domains provided the infrastructure to make the web possible, but to specify the rules of requesting webpages, Berners-Lee needed a new protocol. He and his team came up with Hypertext Transfer Protocol (HTTP). Today, it works as follows.

First, the client and server establish a reliable connection using the TCP protocol. Then, the client makes a specific type of HTTP request. For example, a GET request, which requests a page (originally this was the only request available). The client can specify headers with additional information, one of which must be the location of the resource, which looks like this:

Host: google.com

The server then sends a response which includes a status code, a number from one of the following groupings (only a handful of numbers in each group are used as status codes):

Informational responses (100–199)
Successful responses (200–299)
Redirects (300–399)
Client errors (400–499)
Server errors (500–599)

The most common status codes you will have seen as a user are 404: Not Found (the resource just isn't on the server) and 500: Internal Server Error (an error occurred on the server). You are probably less familiar with 200: OK, because this means the request was successful, and in the browser you will see the webpage that was returned in the body of the server's response, rather than a status code.

There are also other types of request available, such as POST, so the client can send data the user has inputted in a form to the server. For example, if you request to log into a website, the browser sends a POST request to the web server with your credentials. If your credentials are valid and the web server uses cookies as authentication, its response would contain the Set-Cookie header with an access token. The browser stores the token and sends it in the Cookie header on future requests so the server knows you are logged in.

HTTPS

As the web grew it quickly (within a few years) became apparent that greater security was needed. Simply requesting webpages isn't much of a problem, but sending user data is. One such problem is known as a man-in-the-middle-attack - in other words, someone could intercept communication between you and the web server without you knowing.

So HTTPS (Hypertext Transfer Protocol Secure) was developed to accommodate the use of encryption. Encryption is where data is encoded into a scrambled form known as a ciphertext. An encryption key is a string of digits computers can apply in a mathematical way to decipher the message again.

The client and server cannot make the encryption key public, as it would defeat the point. However, they do need to know what key the other is using. This is achieved with mathematics, by combining a public key with a private key in such a way that both client and server generate the same number to use as a key, and use it privately to decipher the encrypted messages they send to each other. Anyone trying to intercept the messages can't reverse-engineer the process using the publicly available variables, because they are missing one variable: a private key. (For further explanation see the Computerphile video in Sources below).

The encryption protocol used by HTTPS is now Transport Layer Security (TLS) but you have more likely heard of its predecessor Secure Sockets Layer (SSL). A TLS certificate, still widely referred to as a SSL certificate, certifies the ownership of a public key, indicating to the client that HTTPS can be used (browsers usually display a padlock icon in the address bar to indicate this). All websites these days are advised to have one, and Google ranks sites that do more favourably in its search algorithms.

URLs

We haven't yet mentioned the Uniform Resource Locator (URL). This brings together a domain name with a scheme - a lowercase value indicating what protocol is being used, of which http and https are two possibilities.

What other types of scheme are there? Well, a common one is ftp, for File Transfer Protocol. FTP has been around since the early days of the internet and requires you to login to a server to upload or remove files. HTTP was a kind of adaption of FTP to optimise it for web requests. FTP is still used in the context of the web - the author of a website would use it to upload a website to a server, or remove it from one.

Other schemes you may have seen are mailto, which allows you to send email from a webpage, or file, which allows you to open a local file in a browser.

The scheme is followed by a colon to separate it, and the domain name is prefaced with two forward slashes, which indicates a path to a computer, as opposed to a path to a file on a particular computer (usually a single forward slash).

HTML

To enable a webpage to be created, Berners-Lee needed a markup language. A markup language allows the user to provide the computer with text along with annotations that indicate the display and function of that text. When the text is then displayed to the intended user, the annotations are hidden. Berners-Lee wrote his own markup language, based heavily on one in use at CERN at the time, but tailored for the web. It's called Hypertext Markup Language (HTML).

HTML provides elements that go on the webpage - headings, paragraph elements, lists and so on. An element is produced by using tags, in most cases by writing an opening tag, followed by the text to display, followed by the closing tag. If I wanted a main heading that read 'My webpage', I would write that element like this:

<h1>My webpage</h1>

But back in 1991, the star of the show was the anchor tag, <a> which creates a hyperlink and makes hypertext possible. To tell the page where you want it to go, you have to add an attribute. Each element has a valid set of attributes you can use for more control over how that element is used. The attribute to add a hyperlink is href (hypertext reference) and is used with the relevant url as the value, like so:

<a href="https://www.google.com">Go to google</a>

If the link is to another page on the same site, a relative file path could be used, e.g.

<a href="/shop.html">Shop</a>

Later, form elements would be added which allow other kinds of interactivity.

Each element has a very plain default display style, and the elements follow each other vertically down the page in the order they were written in the HTML. Originally, these default display styles couldn't be changed, but now they can. However, you can still often see them on older websites or even notice them for a split second on some modern websites before the stylesheets have loaded! 😉

Browsers

The final piece of the puzzle was the web browser - a program designed for requesting files from web servers (principally HTML files) and displaying them. Berners-Lee wrote the first browser, but it didn't take long for things to get pretty complicated.

The technologies that underpin the web are open - that's the whole idea, and the main reason that it went global. In 1994, Berners-Lee founded the World Wide Web Consortium (W3C) to establish some standards for web technologies, such as the HTML specification.

The problem is, it's up to the companies and organisations that produce browsers to actually implement the standards, and there's nothing to really stop them from doing their own thing for any technical or business reason. In the early days of the web, websites tended to be designed specifically for one type of browser (and browser version!) because the browsers differed from each other so much. This competition did at least establish the precedent of browsers being free, however.

By the end of the first 'browser war' in 2001, Microsoft had triumphed over a company called Netscape by establishing Internet Explorer as the dominant browser. Netscape later became the non-profit Mozilla foundation, and they made their browser source code open-source and released it as Firefox in 2004. Firefox is still used today, with a usage share of around 5-10% on desktop browsers.

Also in 2004, individuals from Mozilla, Opera (another browser provider) and Apple formed, wait for it... the Web Hypertext Application Technology Working Group (WHATWG). WHATWG was born from dissatisfaction with W3C's approach, and it is now instrumental in driving web standards forward. Mozilla also began what is now the Mozilla Developer Network (MDN) web docs, a hugely valuable resource for web developers which provides a reliable picture of the various web standards and their actual implementation by browsers, which these days is, thankfully, a lot more consistent.

The period since then has seen Google's browser, Chrome (first released in 2008), win the second 'browser war' and rise to huge prominence. Not only does it have around 70% of usage share, almost all other notable browsers besides Firefox and Safari (Apple's browser) use Chromium, the core part of the browser, which is open-source. This includes Microsoft Edge, the successor to Internet Explorer, as of version 79. Google (and Microsoft) are also now part of the WHATWG's Steering Group.

Layers 🍰

Before we finish, let's take a step back again.

Most explanations of computer networking inevitably reference the Open Systems Interconnection (OSI) model. This is a general-purpose concept that defines the different layers of features or services in a network, and it will help us recap.

It defines 7 layers that work independently of the other layers. This is helpful because when something goes wrong, you can first narrow the problem down to a specific layer of the network.

1. Physical Layer

This one is simple: networks rely on electricity, radio waves and the physical components that carry them to function. All of this makes up the physical layer.

2. Data Link Layer

This layer covers the connection between two nodes on a network, such as the transmission of data frames linked by two computers via ethernet. Error correction is done at this layer to reduce corruption of data caused by things like interference on the line.

Technologies at the physical and data link layer are standardised by the Institute of Electrical and Electronics Engineers (IEEE), in the IEEE 802 standards. This is a grouping of standards for things like Ethernet, WiFi, and so on.

3. Network Layer

The network layer refers to the forwarding of packets between networks. This is is the layer that routers and the Internet Protocol operate at.

4. Transport Layer

This is about the delivery of the data between computers at either end of the connection, and getting data to the right software application. TCP falls under this layer.

Layers 5 and 6 are the Session Layer and the Presentation Layer, but we won't discuss them because they are not that relevant to the Internet Protocol Suite, which predates the OSI model.

7. Application Layer

This refers to the data the user sees and interacts with in an application such as a web browser, and the protocols that are closest to it like HTTP. If you're writing a web application and send an incorrect HTTP request, then the network problem is at the application layer.

Conclusion

Let's recap the key takeaways.

Networks come in many forms and sizes, and are built on layers of technology working together.
The internet brings different computer networks together and allows a variety of services such as email, file transfer, and of course, the web, to exist.
The internet and the web were created with the advantages in mind, and the disadvantages were discovered later. We're only now just starting to get a grip on the monumental ways they can affect our lives. That the web will be free, secure and a force for good is far from a given, and achieving it takes a lot of work from people that believe in those ideals.

What is made of the Web is up to us. You, me, and everyone else — Tim Berners-Lee

Sources

I cross-reference my sources as much as possible. If you think some information in this article is incorrect, please leave a polite comment or message me with supporting evidence 🙂.

How Computers Work (Part 3): Software

Josh Carvel — Fri, 06 Nov 2020 08:19:59 +0000

Intro 👋

In Parts 1 and 2, we learned the fundamentals of how computers are able to do what they do, and the hardware of computers.

In this final part, we'll look more closely at computer instructions: software.

Prerequisites

Parts 1 & 2, or basic understanding of computing concepts and hardware

What is software? 💿

A computer can be thought of in terms of hardware, the physical components, and software, which is a term for computer programs. Programs are just specific instructions to achieve certain tasks. Writing these instructions is programming.

The computer can only process binary instructions, but they can be written in various other ways and converted to binary. In the early days of digital computing, there was a system involving punching holes in cards that would be fed into the computer. The computer would run the instructions (that did some mathematical calculation, for example), you would get some output, then you would give it another program, and so on.

These days, things are a lot more sophisticated. For one thing, we can make the computers do the work of making computers understand us! We can write instructions using high-level programming languages, which are abstracted away from the low-level details of how computers work. Instructions are written in more human-readable form, and other programs do the conversions that eventually turn our instructions into binary.

We can also run many programs at once, handle multiple users, and much more, thanks to operating systems.

Let's look at the different types of modern software.

BIOS 🥾

Your computer can't do much without BIOS (Basic Input/Output System).

BIOS is some instructions that run when the computer first powers on. As mentioned in Part 2, the instructions are stored in flash memory on a chip attached to the motherboard. You may be familiar with configuring some BIOS settings, which in many older computers was accessible to the user only by pressing a certain key very quickly when the first screen came up.

A key BIOS task is loading the operating system from memory. Because the computer loads its own instructions without any external input, this is known as booting, after the old phrase 'to pull oneself up by one's own bootstraps'.

BIOS is also a go-between for the CPU and hardware, including external devices (hence the name input/output system). Part of this role involves testing that all the hardware components are working when the computer switches on, known as a power-on self-test (POST). This is a task BIOS does as part of the boot sequence: the steps it takes when it starts up.

BIOS is a type of firmware. This means it is software that controls the low-level details of hardware on a device, and is specifically configured for that device. In the past, the firmware of a device was rarely replaced, although nowadays firmware can be updated and even physically replaced.

Operating systems 📃

An operating system (OS) is just a computer program, but it has privileged access to the hardware and handles a lot of fundamental operations, like running all the other programs you want to run. It has a number of elements to it.

The kernel

The core part of any OS is called the kernel. The kernel code is always present in RAM while your computer is on. It handles a number of essential tasks.

Process management

To run a program, the CPU needs to be told to move the program data from the disk (or other storage device) into RAM and start running it. The kernel handles this.

This 'running' of the program is called a process. If you've ever opened up Task Manager on Windows, or the equivalent on other OSs, you will have seen these processes listed, and noticed that while some relate to programs you opened yourself, most are background processes that the OS initiated. You may also notice that you can open up a program more than once, and see multiple processes, one for each instance of the program.

A process needs space in RAM to hold various values that it needs to do its work, which the kernel gives it. However, the process also needs to do things that only the kernel is trusted to do (because if they go wrong, the whole computer could crash), like interact with input/output devices, access files, start new processes, and so on.

So the process asks the kernel to do it with a magic word (not 'please', just an instruction name the kernel understands). In programming, a request for an action is known as a call, in the sense that you are calling on something to be done. In this case, the call is to the operating system and it's known as a system call.

The kernel handles sensitive access sort of like a vault. When no special access is needed, the kernel runs a process's code in what's called user mode, where access is locked down. When greater access is needed, the kernel switches to kernel mode (also called 'supervisor mode') to open things up.

Another important aspect of process management is that processes are frequently interrupted. Interrupts may be generated by hardware e.g. keyboard input, software e.g. a request for a system call, or the CPU, e.g. indicating an error.

Interrupts may be responded to immediately if serious, otherwise they are scheduled according to priority. The CPU stops the process it is running, saves its status in memory, switches to kernel mode and executes a response (these are pre-set responses which map to specific types of interrupts, just like with system calls).

Most operating systems are also designed to run processes 'simultaneously', i.e. multitasking. It does this pretty much how we would do multitasking (but much, much quicker) - kicking off one task, then another, then switching back to the first one when it needs attention, and so on. As noted in Part 2, it can also make use of the multiple cores of the CPU. The code that handles all this is called the scheduler, and it runs for very short periods between the chunks of process code.

Memory management

The kernel also handles memory management, using the memory management unit (MMU). It makes sure the data used by one program is not overwritten by data used by another. At its core, this just means allocating specific blocks of memory for a specific process, which belong to that process only while it is running.

This also provides a level of memory protection: programs should not be able to read or modify the memory of other programs. This helps limit the damage done if one program starts doing undesirable things with memory, either unintentionally or intentionally (malicious software).

Computers usually use an abstraction called virtual memory, where the computer's memory is represented in a more ideal way to processes than it actually is in reality. One problem this solves is that a process's data may become scattered across memory addresses over time, which is difficult for the process to work with. The process can instead be given a nice, consecutive block of logical addresses (also known as virtual addresses) which are mapped to physical memory addresses by the MMU. Each block is known as a page.

Another benefit is that the computer can optimise the space available in RAM, by keeping some process data that isn't currently needed in disk storage temporarily, a technique known as paging (if the entire process's memory is moved to storage, it's called swapping). Paging causes some delay due to moving data back and forth between RAM and storage, but the trade-off is seen to be worth it.

File management

Another crucial role of the kernel is handling files, i.e. data that we want to be self-contained. The kernel provides a file system so we can work with data easily. File systems help with moving data into storage, where data is just a long sequence of binary values. To know where one file begins and another ends, we need a file that stores this information: a directory.

On our desktop we have the more abstract concept of 'folders', and think of them as places our files live. But most of the time the folder just refers to a directory listing the file information. For each file this contains details of the address in storage, its length, name, creation and last modification dates, the file owner and read/write access of the file. This is updated whenever there is a change.

A particular file has a particular file format, which indicates to the operating system how to decode it and what sorts of applications it should be opened in. The file extension at the end of the name indicates the format, e.g. a plain text file has extension .txt. The file itself also begins with some metadata (data about data) which helps the computer interpret the data in the file.

Operating systems such as Windows and macOS hide the file extension in their file explorer, because simply changing the extension doesn't change the data, so the file may be interpreted incorrectly (at best, the incorrect extension will just be ignored). But you can change the encoding by saving it to a different file format, when possible.

Modern file systems store files in blocks so that there is extra space for the file to become bigger. If the file exceeds its block space however, the rest of the file may be stored somewhere else. This is known as fragmentation, and the directory has to keep track of where all the fragments are in storage. Reading a fragmented file can cause delay in hard disk drives (not so much in SSDs), which can be mitigated by running a process called defragmentation, where the file fragments are put back together. Most OSs now do this automatically for you.

If you 'delete' a file, the record is removed from the directory, but the data in storage isn't overwritten until something else needs that block. This is why you are advised to clear your computer's disk drive when getting rid of an old computer - some 'deleted' data may still be recoverable.

When you have lots of files, you need a hierarchical file system, where files are grouped in directories and subdirectories, to keep things organised. At the top of the hierarchy is the root directory, which references other files, including directories. We can describe the route to those files with a file path. Moving a file's location is as simple as changing the two affected directories - the data itself doesn't move.

On Windows the root directory represents the storage disk drive as a whole, which for historical reasons (drives A & B were floppy disk drives), is known as the C: drive. On other systems the root directory is usually just '/'. OSs that allow multiple users can deal with this by assigning users their own subdirectory of files they have read/write access to, and not showing them other user's files, while allowing an administrator to access all files.

A file system is implemented in what's known as a partition, i.e. a segment of the disk. Often a disk is divided into multiple partitions with separate file systems, for example to allow the user to have different OSs in separate partitions on the same computer.

Device management

We can think of software as something that communicates with the hardware. However, not all computer hardware is the same: there are lots of different configurations. A certain hardware component may expect instructions in a different form than a similar one created by someone else for a different computer. So it seems you would have to write different instructions for each version of the component.

This is the problem device drivers aim to solve. They are a middleman, a translator. The software asks for a particular action in a standard language, and the device driver translates the request to account for hardware differences. These days many hardware components, such as keyboards, tend to conform to a generic standard for their type, so the OS can include generic drivers for these components. The OS also allows you to install more specific device drivers where necessary.

Networking and security

Aside from the tasks mentioned above, the kernel does some other things. It provides the ability for networking i.e. accessing the resources of other computers, and various security features.

User interfaces

The main OS feature outside of the kernel is user interfaces, which you use to interact with the computer.

Command line interface (CLI)

The CLI is the interface at which you enter commands with the keyboard.

There has to be a graphical element to this interface for the user to see the input and output, but it's very basic and doesn't accept mouse input. We can call this part the terminal.

Beneath that is a program that receives certain commands, interprets them, executes the instruction then sends the output back to the screen. This program is not part of the kernel, it just communicates with it. It's known as the shell (the kernel of a nut or seed is surrounded by its shell), or in this context could also be called a command line interpreter.

Operating systems usually have a default command line interface, and they can vary in various ways, such as the commands they understand. But on modern computers you can download and use different types.

Text-based user interface (TUI)

A TUI is a less common interface that was seen in older computers for certain purposes and you might still see sometimes, for example on the BIOS screen. It's just a much more basic graphical interface where you select text options.

Graphical user interface (GUI)

The GUI (pronounced 'gooey') dates back to 1973 when Xerox developed the Alto computer at their research centre Xerox PARC. The computer featured a mouse and the GUI had windows, icons and drop-down menus, and began the trend of the desktop metaphor that is still used today (the desktop environment).

The Alto wasn't a commercial product, but the GUI began to go mainstream after Steve Jobs visited Xerox PARC in 1979 and saw the potential. Apple's Lisa computer in 1983 featured a GUI, though the Macintosh in 1984 was the first commercially successful usage.

The desktop environment on modern computers is actually made up of a number of components, such as the windowing system, file system and graphical theme, which are customisable to a greater or lesser extent depending on the operating system. Beneath them there is still a shell that processes the instructions.

Though GUIs revolutionised how we use computers, and made many tasks simpler, they didn't replace CLIs. Although these days there are GUIs for a great many things, CLIs still give you the flexibility to achieve any computing task, which is not quite true of GUIs.

Utility programs

Utility programs are what they sound like: programs to support the computer infrastructure. This includes things like backup, anti-virus utilities and disk defragmentation. These are considered part of the operating system, though the user may install replacements or additional utilities.

Modern OSs in context 🖥️

Now we'll look at the different operating systems we have today and how they came about.

Unix

Most operating systems around today derive from an OS called Unix, developed at the famous Bell Labs in the 1970s.

AT&T, which ran Bell Labs, was legally restricted from selling software, and required to cheaply grant licences for the Unix source code to other educational institutions. In 1982, AT&T became legally free to sell Unix. But by then, the University of California, Berkeley, had already developed a slightly different version - the Berkeley Software Distribution (BSD) - based on the source code they had been licensed in the 1970s.

By the 1990s, Berkeley made the BSD source code freely available to study and use. Today we would call this open-source software. It contained no source files from the AT&T version (though a legal battle went on for years). OSs like this are known as Unix-like operating systems and are not AT&T copyright. Though Berkeley stopped working on BSD, others took on the mantle.

One of the derivatives is known as FreeBSD. Like BSD, it uses a permissive software license. This means you can modify the software and claim full copyright rights over the derivative product. This is one reason FreeBSD is used in some way by many companies. It is a key part of the PS4 operating system and in OSs created by Apple.

Apple OSs

Apple became a successful company after releasing the Apple II in 1977, one of the first highly successful, mass-produced personal computers. However, they failed to produce a personal computer and operating system to match their initial success in the following two decades.

Since the early 2000s, Apple computers have used macOS (previously known as Mac OS X and OS X). The kernel of macOS, called Darwin, used many elements of Free BSD, and Darwin is the basis for all the other Apple OSs - iOS, iPadOS, watchOS and tvOS.

Linux

In 1991, a Helsinki University student called Linus Torvalds released a Unix-like kernel he had made called Linux (a combination of his name and the 'x' from Unix, pronounced 'lin-ucks'). However, he hadn't written the other software needed to create a fully-fledged OS, such as a shell and a program to compile (translate) the code he had written.

As it happened, there was a project that had begun in the 1980s and had already made that other software, but didn't yet have a working kernel. It was called the GNU Project, with GNU standing for 'GNU's not Unix!' (yes, really!). GNU's aim was to freely distribute a Unix-like operating system and lots of software to go with it. So Linux and GNU developers began co-operating. GNU developers made their software compatible with Linux so a fully-fledged OS could be produced, and Torvalds started releasing Linux versions under a GNU license.

GNU is about free software (in the sense that it is not owned by any individual). It's more ethically opinionated than open-source (though definitions of the two movements are disputed). While BSD was one stop short of public domain, more or less just requiring a copyright notice, GNU software is distributed under a 'copyleft' scheme, meaning the work can never be redistributed under a more restrictive scheme.

OSs that derive from the Linux kernel and other software are known as Linux distributions, and there are hundreds of them, for all sorts of different needs and preferences. At one time, distributions using GNU software were referred to as 'GNU/Linux' distributions. Now they're usually just called Linux, a point of controversy to this day.

Linux systems are everywhere. Though not exactly a Linux distribution, Google's Android, the OS that runs on roughly 3/4 of the world's smartphones and tablets, is based on a modified version of the Linux kernel, as is the Chrome OS used in its Chromebooks.

You will find it running on mainframe computers, supercomputers, routers, smart TVs, smart home technology, smartwatches, video game consoles and much more. By most estimates, most web servers are run on Linux, and it continues to be popular among programmers and those who enjoy its high configurability.

Windows

"But what about Microsoft Windows?!", I hear you cry. Well, Windows is the big exception.

As it happens, Microsoft had acquired a licence for a version of Unix from AT&T in the late 1970s, which they would sell to other companies, under the name Xenix for legal reasons. Xenix actually became the most popular Unix variant of the time, and Microsoft believed it would be the standard operating system of the future.

In 1980, IBM began to compete in the personal computer market, and went to Microsoft for various things, notably their programming language Microsoft BASIC, and an operating system. Xenix was not suitable for the hardware of the PC, so IBM considered CP/M, a common operating system of the time for business computers, but couldn't reach an agreement with its creator. Microsoft jumped in, buying the rights to an OS initially known as QDOS (Quick and Dirty Operating System), essentially a CP/M clone, for just $50,000. Their version was called MS-DOS (Microsoft Disk Operating System).

Microsoft was paid a fixed fee by IBM for MS-DOS, but retained the rights to sell MS-DOS to other companies, which IBM didn't think was important. It turned out to be very important indeed when lots of companies began reverse-engineering and selling clones of the IBM PC (using a legal loophole where an engineer who has seen the copyrighted source code can write instructions, which will be used by another engineer in a 'clean room' who has never seen the source code). Microsoft sold MS-DOS to many of these companies, while IBM got pushed out of the PC market.

When AT&T became free to sell Unix and enter the PC market in 1982, Microsoft decided that pursuing Xenix was not a profitable strategy. MS-DOS was successful, so they worked on a follow up, Microsoft Windows, which is used on at least 3/4 of all personal computers today.

Applications 📱

User-facing programs are known as applications or apps. An app communicates with what we might call the platform or environment it will be run on, using the application programming interface (API) that the environment makes available.

The API is the code that understands a particular set of instructions. The instructions are documented somewhere so the programmer can use them without knowing the details of the hardware. For example, OS system calls are implemented via an API (though they vary depending on the OS). Other environments such as mobile devices and web browsers have different APIs that allow software to be written for them.

An app that is run directly by a particular device's OS is called a native application. More recently, web applications have become more popular. In a web app, the code is stored on a server and the device's web browser downloads the code via the internet, so the user doesn't have to download any code themselves. There are trade-offs to both approaches.

Applications for different platforms tend to have different programming languages that they require the software to be written in, for various reasons such as suitability as well as business reasons. For example, even to specifically write for mobile devices you have to write in Java for Android, but in Swift for iOS. This is one advantage that web applications have - they can be written in one language, JavaScript, and be run on any device so long as it has a web browser and an internet connection.

Conclusion

Let's briefly recap:

Software is just instructions the computer understands, but today it can be written in a variety of high-level programming languages for a variety of different platforms.
Applications are not the only type of software, and modern applications couldn't be run without the OS kernel managing things like processes and memory.
There are various financial, political and technical reasons for the different forms of software we have today.

That concludes this three-part series on computers. I hope it has provided a solid introduction and a good jumping-off point for further exploration.

Thanks for reading! 😀

Sources

I cross-reference my sources as much as possible. If you think some information in this article is incorrect, please leave a polite comment or message me with supporting evidence 🙂.

* = particularly recommended for further study

* Crash Course Computer Science Episodes 18 and 20
* Operating System Basics - Brian Will
* Understanding the Shell - dwmkerr.com
How BIOS works - howstuffworks
Operating system (and associated pages) - Wikipedia
Virtual Memory in Operating System - GeeksforGeeks
Hard Disk Partitions Explained - How-To Geek
Shell (computing) - Wikipedia
What Is Unix, and Why Does It Matter? - How-To Geek
Unix vs Linux - Gary Explains
Quora question on origin of Windows
Usage share of operating systems - Wikipedia
Triumph of the Nerds - documentary (1996)
Revolution OS - documentary (2001)

How Computers Work (Part 2): Hardware

Josh Carvel — Fri, 30 Oct 2020 13:56:00 +0000

Intro 🛠️

In Part 1, we learned the basics of how computers store and process data.

In this part we'll look at the components that handle these operations: hardware. Specifically, the sort of hardware found in personal computers.

Prerequisites

Part 1, or basic understanding of computing fundamentals such as binary, bits and logic gates

The motherboard 👩

Inside a modern personal computer you will find many things, including:

A power supply, batteries and cables to connect various parts
Devices for cooling (since using electricity generates heat), usually a 'heat sink' and fan
Drives (usually in desktop PCs) that accept discs or disks with stored data
Internal storage (hard disk or SSD)
(In laptops) built-in speakers

But most of the action takes place on the motherboard, i.e. the main circuit board of the computer, so that will be our focus here.

In early computers, different electrical components were just wired together in a complicated fashion. Two innovations changed this.

Chips (integrated circuits)

The first is the integrated circuit, commonly known as a chip or microchip. This allows lots of tiny transistors to be connected on one small, flat piece (or 'chip') of silicon.

The type of chip seen today, which uses photolithography (using light to imprint circuit designs in silicon) was invented by Robert Noyce in 1959. In the early 1960s, an integrated circuit might hold around 5 transistors, but thanks to advances in science, the number of transistors that could fit doubled roughly every two years, known as 'Moore's law' after an engineer called Gordon Moore.

In 1968, Moore and Noyce formed a company that became the world's largest chip manufacturer. Since it specialised in integrated electronics, they called it Intel. Today, Intel chips can hold billions of transistors.

Printed circuit boards

Integrated circuits dramatically reduce the number of components needed, but the components still need to be wired together. Printed circuit boards (PCBs) helped with this.

Wires (usually made from copper) that connect components are arranged in an intricate design for maximum efficiency, and placed on a board usually made of fibreglass (plastic reinforced with glass).

You can see the outline of these wires on a circuit board, but that's not all of them! The insulating material may have wires on both sides, and there may be multiple layers, like a sort of electrical lasagne, with connections being made vertically (via small holes) as well as horizontally on the surface.

Being able to put chips on PCBs revolutionised what computers could do. This is how we get to the modern motherboard.

Motherboard components

The motherboard is a PCB which the main components of a computer are connected to. Here are some notable ones (not all of them):

The central processing unit (CPU), which connects to a socket on the motherboard and controls operations
The chipset, which connects the CPU to the other components
RAM: 'sticks' of memory (chips on a PCB) which connect into slots on the motherboard, and are the main place data is held while the computer runs
BIOS chip, which stores the instructions the CPU uses to start up the computer
Ports, which connect devices known as peripherals that provide some input (e.g. keyboard), output (e.g. monitor), or both (e.g. USB stick)
Expansion cards: PCBs that connect to an expansion slot and enhance the functionality of the computer, for example a graphics card or sound card (used in desktop computers)

We'll focus on the CPU, because it's the most crucial to understand.

The CPU 👑

Computer instructions need to be interpreted and carried out, i.e. processed. This mainly happens in the CPU.

The CPU is a type of processor, of which there are others, such as the graphics processing unit (GPU). It's usually made using integrated circuits (usually a CPU is a single chip), in which case it's referred to as a microprocessor.

The CPU, quite simply, is in charge of the computer. It has a few main components which we'll talk about, though the actual implementation is dependent on the type of CPU chip.

The control unit

The CPU is in charge of the computer, but the control unit is in charge of the CPU! It directs the processing of instructions.

No matter how complicated computer instructions are, they always involve basic actions that the CPU can perform, such as:

Getting data in and out of memory
Mathematical and logical operations
Inputting and outputting data from external devices such as a keyboard or monitor

Each of these instructions is given a code in binary, like how a particular video game action such as 'jump' is associated with a particular button. (Jump is actually a computer instruction too! We'll look at it shortly).

This is an opcode (operation code). It is supplied along with the memory address of some data to do the operation on (called the operand).

These codes are not universal - it depends on the particular instruction set architecture (ISA) that the CPU chip has been built with, i.e. the way the circuits have been configured.

The most common series of ISAs was developed by Intel and is called x86 architecture (because Intel produced a number of chips that used it whose name ended with 86).

RAM, caches & registers

To perform operations, the control unit works with RAM, a large bank of memory which is separate from the CPU but nearby on the motherboard, connected with wires. RAM connects to the motherboard via slots and you can add more to your computer. The most common type is called dynamic RAM (DRAM), and it's cheaper than the other main type, static RAM (SRAM).

The wires that transfer data on the motherboard are called buses, because they transport information. They are grouped together to carry bits, for example you might have 64 wires together to transport a 64-bit number.

The CPU is connected to RAM by an address bus, which the CPU uses to tell RAM the address of some data it wants to access. It is also connected by a data bus, through which data can be transferred back and forth.

For any given computer program, the data it wants to perform operations on and the instructions it wants the CPU to carry out are both stored in RAM. Each instruction contains an opcode and operand, together in a single binary number.

Sending data down wires is quick, but it still takes time, so location matters. For this reason, there is an area of memory within the CPU for temporarily holding values - the registers. Each register can hold a single value and like the buses, this is usually between 8 and 64 bits.

The registers are used to hold data needed for operations, such as the current instruction. But there is also some memory kept for values that are not being used right now, but are likely to be needed soon: the caches. These are divided into 'levels' and there are often three of them. Level 1 cache, the closest and smallest, is usually on the CPU chip itself. The others are bigger but further away, though not as far as RAM.

The instruction cycle

Each instruction is fetched from RAM by the control unit, using the buses, one at a time. Each half of the instruction (opcode and operand) is read separately, so the control unit knows the action to perform, and it can get the data to perform it from RAM using the buses.

Once the control unit receives an instruction, it decodes it. This means it runs it through a circuit containing logic gates, which triggers binary values to be sent on a certain path of wires depending on what the binary opcode is. This is all just determined by the instruction set that has been hardwired into the chip.

Now the CPU is ready to execute, i.e. perform the operation. Again, this is a case of the CPU's circuits doing the job they have been wired to do - it's just the data they do it with that changes.

The whole cycle is known as the instruction cycle (also known as the fetch-decode-execute cycle or fetch-execute cycle).

Optimisation

A simple CPU (as used in early digital computers) would wait to go through the whole cycle before moving on to the next instruction, but modern CPUs do multitasking, so when one instruction is being decoded, the next one is already being fetched, and so on.

In addition, since some operations take longer to perform, modern processors can split the circuitry that executes instructions into separate units specialised for different instructions (still on the same processor) and process them at the same time, allowing it to get more done quicker.

The clock

The control unit directs the CPU, but it relies on the clock to function.

The clock generates the consistent electrical pulses that are sent down the wires and received by transistors. It does this in the same way a digital watch keeps time - using a quartz crystal that vibrates at a fixed interval when electricity is applied.

The clock rate is the speed this happens, measured in clock cycles per second. A clock 'cycle' is not the same as the instruction cycle.

In early computers, an instruction would actually take multiple clock cycles, because there are a few steps to carry out each instruction, each requiring a pulse of electricity. The process has been refined since then using optimisations such as those described above, so the opposite is now true - multiple instructions can be carried out per clock cycle.

This and many other factors affect how productive the computer actually is. But to keep things simple, the clock rate (also known as clock speed or frequency) is often used as a rough indication of how fast the computer can work. It's measured in gigahertz, i.e. billion cycles per second!

The ALU

The component that makes operations possible is the arithmetic and logic unit (ALU). The ALU takes two pieces of data and returns one output.

Maths

In Part 1 we looked at how logic gates are used with true and false values. Computers can do maths the same way, but we think of the values as 0 and 1 instead.

When it does maths, the ALU uses a type of logic gate called 'exclusive or', abbreviated to XOR. For the output to be 1, one of the inputs must be 1, and the other must be 0.

Here are the four possible variations and outcomes:

0 + 0 = 0
0 + 1 = 1
1 + 0 = 1
1 + 1 = 0

We are essentially doing binary maths here! You may notice the sums all look correct except for the last one. But it's actually just the answer for the '1s' column - a remainder 1 is carried to the next column.

In this simple sum there would just be 0s in the '2s' column for both numbers being added. 0 + 0 = 0, plus the 1 remainder = 1 in the '2s' column. So the complete answer is 10, which is binary for two.

For this to work, computers have to be wired for the largest length of number they are expecting to deal with - they need the physical wires to represent each column, so the calculation can be performed. So a computer designed to deal with 8-bit numbers (like early computers), would represent the number 1 as 00000001.

Doing different types of calculations usually requires more logic gates, dealing with bigger numbers requires more wires, and there are some adjustments needed for negative numbers, but any mathematical operation can be achieved which this approach.

Logic

Let's look at an example of a logical operation now.

A common logical operation is for the control unit to 'jump' to a different instruction in the instruction list in RAM, if a certain condition is true. Imagine we have a simple 'guess the secret number' game, and the instructions tell the computer to keep prompting the user to guess again on the screen when a guess is made.

But, if the user's guess is equal to the secret number, the computer should jump to a different address in RAM to get an instruction for congratulating the user.

To do this, the ALU can subtract number A from number B, e.g. 5 - 5. If the result is 0, the numbers must be the same! (This would work the same way with words, which are just encoded as binary numbers).

In this case, we don't want the output of the calculation, we just want a true or false answer: is the result 0? ALUs have a wire specifically for this, along with some other useful indicator wires, which just output a 1 or 0: true or false. These are known as the flags.

So if the zero flag indicates true, the control unit can 'jump', by fetching a different instruction at the specified RAM address. This is just one example - all manner of logical instructions can be achieved using the ALU and its flags.

Cores

For quite a while, computers did 'multitasking' in pretty much the same way you would - just switching back and forth between multiple tasks, and often completing each task slower than if you had spent all the time on it.

But if you had some other people to do the other tasks for you, you would get more done, and this is how modern CPUs tend to work. There are essentially multiple CPUs, known as cores, usually all on the same chip. Different sets of instructions can be carried out on different cores at the same time (as long as they don't depend on each other), allowing more work to be done quicker.

Computer specifications 💻

We'll finish by looking at some key hardware specifications (specs). Hardware is a key factor in choosing a computer, especially if you are using the computer for programming.

Let's say you are choosing a laptop for typical programming, or you just want some solid performance. Here are three respectable specs to consider (as of time of writing):

1. A mid-range CPU chip

CPU chips are named quite confusingly. Here is an example: the 'Intel Core i5-8250U'.

To find a respectable mid-range Intel chip, focus on three things:

The brand: look for 'Intel Core', other brands are 'budget' chips
The model number (i3, i5, i7 etc.): look for i5 or higher
The 'generation': this number comes immediately after the dash following the model number, look for 7 or above

The suffix at the end is also informative, because it tells you what the processor is optimised for, e.g. the 'U' in the example above means it is optimised for saving battery power.

The 3 digits in between can be ignored as they are just for Intel's stock-keeping purposes, like a barcode. The naming convention is explained in more detail on Intel's website.

The main alternative to Intel is AMD, and the AMD Ryzen series competes with the Intel Core. The model numbers match, so AMD Ryzen 5 is roughly equivalent to i5, and there are fewer generations, so 2nd generation is fine. Here is some further explanation of their naming.

Though you don't need to, if you look in more detail at the specs of a chip, you will see the number of cores that it has (our example chip has 4) and the 'base' (usual) clock speed (1.60GHz) and a 'max' clock speed that it might be able to reach if it needs to (3.40GHz in our example).

While I have given a rough guide here, it is worthwhile doing further research on this, because exactly how you use your computer and what your priorities are will greatly affect the type of CPU you should look for.

2. 8GB RAM or more

RAM is a much simpler matter. You will want 8GB unless you are on a very tight budget, and 16GB if you are doing a lot of intensive work or gaming etc. You might need to stretch to 32GB but usually not.

3. A solid-state drive (SSD)

We mentioned this in Part 1 of this series. These allow much faster access to the stored data, although you get less storage for the price.

Aside from these three specs, there are many other factors to consider. For the screen you want at least full HD resolution (1920x1080 pixels) and an IPS panel, not TN (the ones where the colours look weird when you change the screen angle). Then you have battery life, ports, weight, and more to consider.

Conclusion

That concludes our look at hardware. In the final part of this series we'll look at the world of operating systems and software 😀. First, a brief recap:

The key component of a computer is the motherboard, a printed circuit board containing the computer's main memory and components, including the CPU.
The CPU is directed by the control unit, driven by the clock, and performs operations on data from RAM using the ALU.
The type of CPU, amount and type of memory/storage and other factors have a big impact on the performance of the computer.

Sources

I cross-reference my sources as much as possible. If you think some information in this article is incorrect, please leave a polite comment or message me with supporting evidence 🙂.

* = particularly recommended for further study

How Computers Work (Part 1)

Josh Carvel — Thu, 29 Oct 2020 08:36:24 +0000

Intro ☕

As a new developer with no computing background, it's possible to jump into coding without really knowing anything else about computers, besides being a typical user of them. But it can be confusing, and a little embarrassing, to be completely clueless on this topic.

This three-part series takes an enormous topic and gives you the absolute fundamentals that you could learn over a cup of coffee, using language you can actually understand, so you have the confidence to make your home in the world of computers.

This part looks at the fundamentals, Part 2 covers hardware and the CPU, and Part 3 covers software and operating systems.

Let's get started!

Using electricity ⚡

As you know, modern computers are powered using electricity, which is conducted along copper wires in the computer's circuitry. When the computer receives some input (keyboard press, mouse click, voice command etc.) that information is converted to electrical signals, and when we want some output, the electrical signals are converted into something like light (displaying text and images on the monitor) or sound (via speakers).

Measuring the voltage (pressure) of an electrical signal over time gives you lots of different values. A device that uses those values exactly as they are for the output is called analogue, from a Greek word meaning 'proportionate', i.e. the processed values are proportional to the voltage. Old TVs and radios are an example.

Modern computers are digital. They don't just relay information, they give electrical signals meaning. A signal under a certain threshold of voltage can be thought of as 'off', and over a certain threshold is 'on'. The data is reduced to just two values. We can also think of these values as the digits 0 and 1 (hence the term 'digital').

It is possible to use more values in computers, so you may wonder why we just use two. For one thing, it is easier to identify the signals reliably when there are just two. But it also makes it easier to write logic - we'll get to this shortly.

Almost unbelievably, with just two values we can do all the things modern computers can do. But how?

Encoding data 📖

The first thing you can do with 0 and 1 is represent data.

There is a number system based on these two numbers: binary (from 'bi', meaning two). We use binary as a code for all the data we have. We can take words, numbers, audio, images - you name it - and convert it to binary, process it in a computer then convert it back again for the output.

Numbers

We use the decimal number system (from 'deci', meaning ten), which has 10 digits (0 - 9). When we get to 9, we run out of digits. So we imagine columns of digits, that increase in multiples of ten each time (remember long addition/subtraction from school?). If we have '1' in our tens column, and '0' in our 1s column, we call that 'ten' and write it as 10.

In binary, you run out of digits after 1! How do we count to 2?

Well, we still have columns, but they are in multiples of two. We can have 1 in the 2s column, and 0 in the 1s column, which is written 10. It looks the same as ten in decimal, but it's the number two in binary.

So you can represent decimal numbers in binary, it just takes a lot more columns, i.e. digits, to do it. This is how computers process numbers.

Characters

What about words, not to mention punctuation and symbols?

All we need is a code where characters are swapped out for numbers. As computers developed, there were lots of different (incompatible!) ways of doing this.

Thankfully, we now we have a universal standard, Unicode. It's a standard with a governing body (the Unicode Consortium) whose mission is to encode all of the world's languages and symbols, old and new, so all computers can understand and process them the same way.

Side-note: this is how we ended up with emojis. An engineer in Japan in the 90s encoded certain numbers as little pictures for some Japanese phones, so it eventually made its way into Unicode. Years later Apple added them to the iPhone, they became popular and now year after year, the Unicode Consortium begrudgingly adds new emojis to the standard to make them more universal and representative! 😃

Unicode assigns a unique identifier (a number prefixed with U+) to every character in use, such as a lowercase 'a', for example. Computers encode that number in binary (usually using an encoding called UTF-8), and the number is decoded on the way out so an 'a' can be shown on the screen.

Pictures

Digital pictures are divided into tiny picture elements. Merging 'pics' and 'el' (elements), gave rise to the term pixel.

We will leave aside how screens actually display pixels, which can vary. When we talk about a pixel in a digital image, we're talking about an individual element of colour.

Various shades of a primary colour such as red can be encoded as a number between 0 and 255, and thought of as one pixel. Since all colours can be made from a combination of primary colours (red, blue and green), if you combine the 256 possible values for each primary colour you get over 16 million possible colours! (256 x 256 x 256). Adding another value between 0-255 for transparency gives you over 4 billion.

So computers just split images into a lot of pixels. The numerical values for each colour in each row can be stored in a list. The dimensions of the image are also stored so the computer knows where each row ends, and therefore can recreate the image.

The more pixels are stored, the more realistic the image will look at its intended size, though you have to store more numbers so it takes up more memory space. Hence, the graphics on early computers with less memory space looked very 'pixelated', i.e. they had fewer pixels so the pixels were more noticeable to the naked eye.

Bear in mind that with pictures and other types of files, computers can use various shortcuts and tricks to optimise the memory space they have available and make file sizes smaller. This is called compression. For example, an image of a plain black square doesn't need the colour value of black repeated over and over in memory: the fact that it has 'x' many black squares could be encoded instead.

Audio

A sound wave can be recorded digitally by measuring by its amplitude at a fixed interval (the sample rate): a common method takes 44,100 samples per second. That gives you enough data points to recreate the sound wave. The amplitude values are of course stored in binary.

Raw, uncompressed audio data is often stored in a specific format known as a .wav file. The audio can also be compressed to various degrees, for example to create an .mp3 file. mp3 files use lossy compression - this means it discards and approximates some parts of the audio data to make the file smaller. Often the aim is to reduce the parts of the sound wave that are considered beyond the range of human hearing.

Video

Video is basically a sequence of images and can be stored as such. However, if left uncompressed the file size would be impractically large for the average video. Almost all video uses lossy compression for this reason. For example an .mp4 file may be up to 200 times smaller than the original file.

The compression usually works by identifying parts of the image that haven't changed and only transmitting information about the parts that have changed. In many cases it will simply describe transformations that should be applied to these changed parts, rather than describing the whole image over and over. However, if this goes wrong and the transformations are applied over the top of the wrong part of the video, it can look very glitchy!

Bits, bytes, and so on 🗄️

Even at the low levels of computing, humans don't think about data in binary form. This is where units of information come in.

In computing, 1 binary digit (a 1 or 0) is called a bit. It represents one bit of data.

It's not very useful on its own, so you will hear much more often about bytes. Bytes are 8 bits. With that you can represent any number from 0 to 255, which is enough to store a standard English character or a colour value. That's one of the main reasons the number 8 was settled on for a byte.

It's also convenient because it is a multiple of two, specifically 2³ (2 x 2 x 2 = 8). Multiples of two are used because binary is based on two digits. The next unit that engineers use is 2¹⁰, which is 1,024. For convenience, it came to be known as a kilobyte, since a kilo is 1,000 of something in the metric system and it was pretty close to that.

However, people started using kilobyte to mean 1,000 bytes. This generated a debate that goes on to this day. And with the growth of computing capabilities, larger and larger units were needed. In the traditional definitions, we had 1,024² bytes: a megabyte, 1,024³ bytes: a gigabyte, and now we even talk about terabytes of data on storage devices (1,024⁴ bytes).

But people, including technical people, still often use these unit names to mean multiples of 1,000 instead, which makes a big difference in how many bytes we are talking about for the larger units. Alternative terms have been suggested for the binary versions (e.g. 'kibibyte' for kilobyte), but it hasn't exactly caught on.

If you think this is confusing, that's because it is. Sorry! Let's move on.

Instructions 📜

We now know how data is encoded as bytes of information, but none of that explains how computers do anything with it. We need instructions.

To do this, we need a slight shift in thinking. Let's think of the value above a certain voltage (1) as true instead, and the other value (0) as false. With that simple shift, we can write logic using a branch of mathematics called boolean algebra, after mathematician George Boole, which is all about performing operations on true and false values.

We can do this because the flow of electric charge (current) in a computer's circuitry can be manipulated depending on whether a true or false value is received by components called transistors. Transistors are made from silicon (hence 'Silicon Valley'), a metal that is a semi-conductor, which can be made to either conduct or not conduct electricity. We can think of it like a switch that is on or off. We pass a value from an input wire into the transistor, and get a value from an output wire.

Clever placement of transistors in a circuit can be used to create components called logic gates. The basic building blocks of computer logic are the following three logic gates:

NOT: output is true if input is false.

This is the simplest boolean operation: NOT true means false, and vice versa.

It uses just one transistor in a way that acts like a reversed switch. When the transistor is off, current flows along its normal path to the output wire. But if it's on, current is directed away from the output wire.

AND: output is true if both inputs are true.

Here one bit AND another bit must both be true. Two inputs are fed into two transistors that interrupt the flow of current towards the output. Both transistors must be on for current to flow through.

OR: output is true if either input is true.

Two transistors are used in parallel: just one needs to be on for current to flow through to the output.

We can use logic gates to make statements like: "if this bit of data is true, AND this other bit of data is NOT true, then send a true value to some other component". The value can be carried to the other component because the output wire would be fed into it.

That particular sort of statement is quite common and so the arrangement of transistors it uses actually form its own logic gate, the XOR ('exclusive or'), which we come to in Part 2.

Of course the computer doesn't 'understand' the logic being expressed or the information passing through, but a human can design the circuitry in a way that is useful to us.

Now imagine you have billions of microscopic transistors in the circuitry. It sounds crazy, but that is exactly what is in your phone, computer or device right now. That's one reason it can carry out so many instructions so fast.

Memory 🧠

Computer logic isn't all that useful if you can't keep values in memory and come back to them later. This is why your computer has memory.

Memory is built from lots and lots of memory cells that can hold onto a 0 or a 1. There are many ways this can be done, all using pretty complicated science, which I'll only briefly attempt to explain.

Each method also has its own trade-offs. Here's some key characteristics memory can have:

Performance/speed: Memory which allows you to access values quickly is more expensive, and vice versa.
Volatility: If the memory needs a power source to hold values, it is volatile. If not, it's called non-volatile or persistent memory. Confusingly, volatile memory may also just be called 'memory', and non-volatile memory 'storage'.
Access method: The main distinction is between random access (access any value at any time) and serial access (go through the sequence to get to a certain bit of data). For example, you can't jump to a certain section on a tape or CD, you have to fast-forward through the other bits first.

Computers use a range of different types of memory internally to balance these factors, and we also have external forms of storage for convenience.

Primary memory

We can call memory that the CPU accesses directly primary memory (there are other definitions of 'primary' or 'main' memory but we'll stick to this distinction here).

First we have the registers built into the computer's processor chip (we cover the processor in Part 2), holding values such as the code corresponding to the next instruction to be processed. Then there are some caches, either within the processor or a bit further away, storing values that the processor is likely to need.

These types of memory are fast, random access and volatile. They use a circuit known as a latch (it can latch onto one of two states). A latch uses logic gates and feeds outputs back into inputs in such a way that we can set and reset the value and the circuit will hold onto it. But even though the circuit can 'hold' a 1 or 0, i.e. a certain level of voltage, it can't do so if no current is supplied, hence it is volatile memory.

Further away from the processor still, we have RAM. This stands for Random Access Memory, though that doesn't really distinguish it from the types of memory just mentioned. Though one type of RAM does use latches, the most common type instead uses a device called a capacitor which maintains electrical charge, though the charge needs to be regularly refreshed. It's not as quick to retrieve values, but still fast enough to run software.

RAM uses a grid with intersecting wires. 'Write enable' wires can be activated for a particular row and column in the grid: a memory address. Then a 1 or 0 can be sent along a 'data' wire to be stored at those coordinates. Storing 1 bit at a time isn't very useful, but lots of grids can be used together, and the same address (i.e. coordinates) can be passed to each grid at once to store a value of say, 32 or 64 bits. The computer uses a different memory location to store all the addresses for when it needs to access the data again.

There is also some form of non-volatile memory which contains important code the computer needs to start up (BIOS, covered in parts 2 and 3). Originally, this was stored with ROM (read-only memory, in its original form), which couldn't be changed at all. Many variants of ROM were produced over time which allowed data to be deleted and written over in some way. The culmination of these efforts is flash memory, invented at Toshiba, in which data is deleted and replaced in blocks, but fewer transistors are used so it is relatively cheap. This is still technically in the family of 'read-only' memory because you can't write an individual bit with one action.

Secondary memory

Your computer needs some non-volatile memory to hold all your data and programs, which it loads into RAM every time computer starts up for quicker access. To this day, many computers use hard disk drives, also just called hard drives. It's a magnetic disk sort of like a CD, in a case with an arm that reads data from the disk sort of like a vinyl record player. The data is stored on the disk using magnetism, where the 1s and 0s are represented by magnetic grains which are polarised one way or the other.

Note that 'discs' (with a 'c' not a 'k') like CDs and DVDs, are similar but use light, not magnetism, to store 1s and 0s, and are known as optical storage. CD-ROMs (read-only CDs) were commonly used to distribute software programs before we could download them over the internet.

More recently, solid-state drives were invented: SSDs. These use flash memory and have no moving parts - they can access and delete data quicker than hard drives and run more smoothly, with less noise. On the other hand, they are more expensive.

You can also get external hard disk or solid state drives for storing a lot of data externally. If you have a smaller amount of data to store, you would probably use a flash drive (a.k.a. 'USB stick' - USB describes the type of connection to the computer it uses). They use flash memory too and are an inexpensive and common form of external storage. The precursor to the flash drive was the floppy disk - a sort of mini hard disk read by a floppy disk drive in the computer. Though now out of use, they are immortalised in the form of the humble 'save' icon.

Conclusion

Hopefully you now have a much better idea of how computers work. Part 2 will look at little closer at computer hardware and how instructions are carried out by the CPU.

First, a brief recap:

Our data is represented by binary states which we produce using physical phenomena such as voltage. We encode it then decode it again as output that has meaning to us.
Computers use billions of transistors to process these binary values in circuitry we design to implement logic using boolean operations.
We can record the values in memory, but there are many types with different characteristics - computers use a variety of types to achieve value for money.

Sources

I cross-reference my sources as much as possible. If you think some information in this article is incorrect, please leave a polite comment or message me with supporting evidence 🙂.

* = particularly recommended for further study