Forem: Reece Charsville

How to optimise a Next.js site for maximum performance

Reece Charsville — Tue, 31 Jan 2023 12:24:32 +0000

This post was originally posted on my blog

Recently at Forward Digital, we have had a big push on SEO. A key part of this is the performance of your website. There are a lot of resources out there supplied by Google, that go into detail on how performant your website should be, and what metrics are used to track performance.

So, we have spent a lot of time optimising our site for maximum performance, and we thought we would share some useful tips and how we did it.

To give some context, our website is built using Next.js and hosted on Vercel.

Tools for tracking performance

The process of optimising your website is incremental. It is important to track how each change is affecting performance. By doing this you can gain a better understanding of where to prioritise your efforts. Thankfully, Chrome Dev Tools is a great tool for doing this, it has Lighthouse built into it which enables you to quickly run reports and calculate a performance score.

This is what we used to track our progress optimising our site, and what I will keep referring to in this post.

Now some important things to note. The first is that you should track your performance based on your mobile performance.

Generally speaking, desktop performance is much better than mobile, and so you will notice that when analysing sites via Lighthouse the performance score on mobile is much lower than on desktop. Therefore if we can get good mobile performance then we know we should have a really good desktop performance.

The next important thing to note is that your lighthouse score can change each time you run a report, even though you may have made no changes. If you run your website locally, open localhost and run the lighthouse report you will notice the score is significantly lower than when you run lighthouse on your live site. Then if you try opening an incognito window and running the reports again, the score may have changed again.

This is because of two things. Firstly when you are running your code locally with next dev, you are not running it in production mode. This means that there are lots of extra debugging and dev tools included in the bundle and files are not being minified. This causes slower performance.

Secondly, when you are running your browser outside of an incognito window you will have all of your browser extensions. Some browser extensions will require extra resources and inject things onto the page. A good example of this is React Dev Tools. This then gives you are worse score than other users may experience who do not have any extensions installed.

Process for tracking performance

In our case, to try and make sure we had a good understanding of our current performance we always ran our lighthouse checks in an incognito window. We also ran two reports after each change. We ran one on localhost, and then we would push our changes to a branch which had a Vercel preview deployment attached to it and run a report on the preview URL in an incognito window. This helps give a clear understanding of how our code changes have affected performance, and what we can expect our performance to be on a live deployment.

Before you start making changes to your code, run these two reports and make note of the mobile performance scores. Each time we make a change we can run the reports again and compare the results.

It can become a tedious task making changes, pushing to preview and then waiting for it to deploy, just to run the performance report again. So when you run the report on your localhost, if your performance score has not increased by at least 2, then I would not bother pushing it, as the change is unlikely to have made much difference.

However, it is important to note that even though your score on localhost may only have increased by 2, it does not mean that the preview deployment will have only increased by 2. In our case, we would sometimes see a change of 2 on localhost, which was more like 8/10 when on the preview URL.

How does Google measure performance

There are 6 key metrics that are measured, and these are:

First Contentful Paint (FCP)
Time to Interactive (TTI)
Speed Index
Total Blocking Time
Largest Contentful Paint (LCP)
Cumulative Layout Shift (CLS)

I won’t go into too much detail on what each of these mean, but Google actually supply us with numbers that they see as Good, Needs Improvement or Poor. We can use these as our targets.

You can read more about what each of these metrics means and how they are measured here. I have attached some more resources at the end of this post if you want to dive in further on each of them.

You will notice in these metrics they show FID, which is not on the Lighthouse metrics. FID stands for First Input Delay, and it is very similar to Total Blocking Time. So, we should aim to get our Total Blocking Time to less than 100ms.

These targets are NOT for our localhost reports. These are the targets for the preview deployment report. It will be almost impossible to achieve these on our localhost report, and the preview deployment will be much closer to what the live deploy report will be. The ultimate goal is to achieve these targets with the live URL.

Now let’s get into actually improving our scores!

Image Optimisation

Next.js includes image optimisation via the next/image component. If you deploy your site using next export then you may want to skip this section, as you will probably be aware that the component cannot be used the same with exported sites.

Try to make sure that you are using this component instead of <img> tags everywhere.

You can read the docs here

It may cause some slight pains with sizing your images on the page but it is worth it for the benefits. You can also use the fill prop with a relatively positioned container to help make this easier.

When using the Image component there is a priority prop. This can help improve your FCP and LCP scores. If you have any images “above the fold” when your site first loads, then these should all have this set to true. It ensures that these images are loaded first.

You can also use the quality prop to set a number between 1 and 100. It defaults to 75 but can be used to help make larger images load faster.

Here is an example:

<Image
    src="me.png"
    alt="Picture of the author"
    width={500}
    height={500}
    quality={60}
    priority
/>

Custom Fonts and Google Font Optimisation

Whether you are loading your own font files or loading from a CDN like Google Fonts you can optimise this. Loading in fonts can increase your Total Blocking Time and your Time to Interactive.

In our case, we use a custom font that is included in our website bundle.

Next.js has a package called @next/font that allows you to optimise your fonts. It is not included with Next so you must install this package separately. You can read the documentation here.

As we host our own font files we needed to use the Local Fonts method of optimising fonts. This required us to define the paths to our files in an object, and inject them into the main of our app.

The process is similar for Google Fonts, however, you can import these directly from the package.

As the fonts are injected via className it means that you can only include fonts for specific pages or components where the font is used.

Example:

import { Roboto } from '@next/font/google'

const roboto = Roboto({
  weight: '400',
  subsets: ['latin'],
})

export default function MyApp({ Component, pageProps }) {
  return (
    <main className={roboto.className}>
      <Component {...pageProps} />
    </main>
  )
}

Lazy Loading Content

When a page is first loaded we want to prioritise the content above the fold. The LCP is referring to the largest element that the user will see first. In order to get this time down we want it to be one of the first things to load. This is where next/dynamic can help. You can read the documentation here.

In our case, our pages are broken down into smaller React components. Each section of our homepage is its own component and all of the content above the Fold is wrapped in a Fold component. This approach makes using the dynamic import packages really easy.

We wrapped all of the components below the fold in dynamic imports. This means that all of the content in the FCP and LCP is prioritised, reducing those all-important times.

Here is a quick code snippet to show you how it looks:

import dynamic from 'next/dynamic';
import { Fold } from 'Components';

const WhatWeDo = dynamic(() => import('../components/what-we-do'));
const AboutUs= dynamic(() => import('../components/about-us'));
const Services= dynamic(() => import('../components/services'));

export default function Home() {
  return <>
    <Fold />
    <WhatWeDo />
    <AboutUs />
    <Services />
  </>
}

3rd Party Scripts

You may use 3rd party scripts for certain libraries or things like Google Analytics. These need to be loaded into the page, and if they are required then they can increase your TTI and Total Blocking Time.

You can use the next/script components to optimise these imports (Documentation is here).

Some scripts do not need to be loaded in before the page renders and so you can utilise the strategy prop on the Script component. By default the strategy is afterInteractive this means that these are not loaded until the TTI has finished.

You can also use lazyOnLoad, this is useful for any scripts that are for content below the fold or that can be loaded while the user is already using the site.

If there are certain scripts that must be loaded before you can use beforeInteractive but you should try to do this as little as possible as it will have a negative impact on the performance.

Google Analytics can actually be rendered later. However, one thing to note is that your analytics will not track users who did not complete loading the page.

Here is an example of the script component in action:

<Script src="https://example.com/script.js" strategy="lazyOnload" />

Don’t use Babel compiler, use Next.js compiler

In version 12 of Next.js they introduced the new compiler, that would replace the Babel compiler. The new compiler is much more performant and effective at optimising your final bundle.

When we first built our website we were using version 11 of Next which came with the babel compiler. Even after upgrading to the latest version we still had a custom .babelrc file in our project. Just by having this file it meant that when we compiled our website it would use the babel compiler instead of the Next.js one. So we removed the file entirely and re-compiled and saw a huge increase in performance.

As of Next 13 the Next.js swc compiler removed the use of Terser for minification, and the swc compiler handles all minification which is 7x faster. So make sure you are running the latest version of Next without any babel config.

You can read up more about the compiler here.

Another thing to check is that in your next.config.js you do not have the swcMinifier disabled.

For example, this is bad as it will use the slower Terser minifier:

// next.config.js

module.exports = {
  swcMinify: false,
}

Bundle size

Here is the big one! Most of the previous tips try to reduce the number of render-blocking resources and optimise what gets loaded and when. However, ultimately one of the most important things you need to maximise performance is a smaller bundle size.

Most of us know what it’s like to be waiting for downloads, and the dreaded feeling when you start a download and see almost 100GB+. Websites are not so different. The smaller the bundle, the faster it downloads. The main difference when it comes to websites is that we are working with much smaller numbers.

If you were using the Babel compiler and moved to the Next.js compiler then you will have probably already managed to get your bundle down somewhat. I will run you through some more ways of getting the bundle size down.

next bundle analyzer

The first step to reducing the bundle size is analysing what you currently have. This way you can see what are the biggest contributors to your bundle, and look for ways to reduce them. In some cases, you may even want to remove them.

Follow the instructions to install @next/bundle-analyzer here. Then run a build. You will see a few windows open in your browser.

The first one to look at is the client.html. This shows you a visual representation of all the files included in your bundle. The size of the file or directory on the screen is proportionate to the size of it in the bundle.

Take a look at this example:

There are a few that stand out to me. These are:

react-fullpage.js
react-dom.production.min.js
data-protection-policy.mdx
prism.js

You will notice that 3 of these are wrapped in node_modules. This means that they are some of the libraries that we have installed in our project.

react-dom there is not much we can do about that, as we need React installed.

However, react-fullpage.js is a 3rd party library we installed for some fancy slide animations. When I hover it I can see that it has a parsed size of 73.36KB. This is significantly larger than most of the other files and directories in the bundle. Therefore, this would be a great place to start.

The first step is to see if I can remove it entirely. If I remove it then I will be saving 73.36KB which would reduce my total bundle size from 836.78 KB to 763.42 KB, which equates to around 11.4% reduction. This is a significant reduction in size.

How do I know if I can remove it? This is where depcheck comes into play.

depcheck

depcheck is a package that analyses your code and helps determine if you have any packages installed that are not being used. You can read the docs here.

You do not need to install the package, you can run this command in the root of your project

npx depcheck

This will output to the console a list of your dependencies. There is a section with the heading *******************************Unused dependencies*******************************. This is where you should focus your attention.

According to depcheck the packages under this section are not being used in your app, so ******in theory,****** you can remove all of these packages.

However, be careful. Just because depcheck can’t find any reference to them it doesn’t mean they are not being used. I would recommend uninstalling them one by one and re-running your app to check it still builds and works as expected.

Removing any unused packages can help reduce bundle size without having any effect on the app, but in our example, we are using react-full-page which means we can’t just remove the dependency. So the next step is to see if we can reduce it.

Light versions

If you have a package that is taking up most of your bundle size then it may be a good idea to look for an alternative. There are often a lot of packages out there that do similar jobs but have different bundle sizes. A good example of this is Lottie animations.

lottie-web is the official package that is used, however, this has a massive bundle size. We were using react-lottie-player which had a dependency on lottie-web and it took up almost 40% of our total bundle size when we ran the analyser.

After looking through the docs, we found that there was a light version of Lottie that react-lottie-player provided in the distribution. So we updated all of our references to Lottie to use the light version. This reduced the bundle size a lot and got it down from 40% to around 20%.

This was a big performance boost, and after re-running our performance reports we saw a big difference. But even at 20% it was still one of our biggest contributors to the bundle size. So we pushed on to see if there were any other ways we could reduce it.

Introducing dotlottie. Dotlottie is a new format and allows you to convert Lottie .json into the new and smaller .lottie format. We can then install the @dotlottie/player-component package instead of the react-lottie-player one. This removed the dependency on lottie-web entirely and got the bundle size of the Lottie animations so small I can’t even see it on the analyser!

Not all packages have light versions, so you may need to try something else. What next?

Tree shaking

Tree shaking is the term used to describe when dead code is removed from a bundle. Lots of packages will implement tree shaking, and in this context, it means that when your app is compiled it will only include the specific files and assets that you are using in that package.

Some packages will not have a good implementation of tree shaking. An example of this (at the time of writing this post) is react-icons.

We used react icons across our website and used various icons from the material design icon set. Across the entire site, we probably used about 10 different icons. Overall it is a very small subset of a large icon set.

When we ran the bundle analyser we could see that react-icons was including the entire material design icon set in the final bundle. This is an example of failed tree shaking. We should only be including the 10 icons we used.

After looking around online it seemed that this was a known issue with react-icons and Next.js. The solution was to install a new package that provided all of the individual icon files (@react-icons/all-files). Using this package we could import all of our icons by specifying the exact paths to the file.

import { MdMail } from '@react-icons/all-files/md/MdMail';

This then meant that only the 10 icons we used were included in the final bundle.

When you are looking at some of the larger packages in your bundle, you should try to see if it uses an effective tree shaking approach.

Last resort

If you find that you are using a package that has no light version or alternative, then you may want to consider writing your own implementation.

In our case react-fullpage-js is being used for some nice CSS animations. It includes a lot of other functionality but we are only using a small part of it. So it wouldn’t be too much of a big job to write our own CSS animations to replace it and remove the dependency.

However, this may be too much of an undertaking for some more complex third-party packages, and it also becomes more code for you to maintain yourself. So tread lightly with this one!

Summary

After implementing all of the above techniques on our site, we were able to see massive performance gains. Check out our website, and you can see what I mean forward.digital.

I have gathered a list of some of the resources that I found useful when researching and optimising our website performance.

Useful links

Web app vs mobile app: Which should I use?

Reece Charsville — Tue, 17 Jan 2023 09:26:02 +0000

This post originally appeared on my blog

When working with a client one of the most frequent questions we get asked is “should I use a web app or a mobile app?”. There isn't a one-size-fits-all answer. The answer depends on a few factors specific to your project and its requirements. I will outline some of the key differences between the two, the benefits of each and the cost implications.

Push Notifications

A push notification is a message that is sent to a user's mobile device by a mobile application that the user has installed. These notifications can be used to alert users to new content or updates within the app, such as a new message, breaking news, or a sale. Push notifications are often delivered in real-time and can be customized to include things like images, sounds, or vibrations to grab the user's attention. However, push notifications are not available on web applications.

One of the main benefits of push notifications is that they can be used to re-engage users with an app. When users receive a push notification, they are more likely to open the app and interact with the content or feature that the notification is promoting. This can be particularly useful for businesses, as it can lead to increased sales or engagement. Additionally, push notifications can also help to keep users informed of important updates or news, without the need for them to open the app or check for updates manually. Furthermore, they increase retention and can help to improve brand loyalty.

The catch to this is that push notifications must be used carefully. If a user is frequently receiving notifications that are not relevant then they may start to find them annoying and this could lead to the user uninstalling your app entirely.

Push notifications are often not a requirement for your application but can be a great addition to help improve user experience. Although they are not available on web applications, there are alternative solutions that often suffice. For example email alerts.

Offline Capability

A key benefit of a mobile app is its offline capabilities. When working with a web app you can only access it when you have an internet connection. This can be a particular problem if a user is trying to use your app when they are in an area with bad service or is using a device that relies on WiFi and does not have internet access.

Mobile applications are installed onto the device. This means that they can still run and perform certain actions without connecting to the internet. If your app is able to still function without an internet connection then this can be a very powerful tool. It is also possible to queue requests. This means that even if your mobile app does require an internet connection, requests can be queued and sent off when the device regains internet access.

However, there are some caveats to this. Mobile app development is not straightforward and so integrating a queue system into your mobile app is going to take an app developer some time. This can therefore have some implications on the total cost.

Is it a big deal? More often than not whether its a web app or a mobile app they require an internet connection to store data, so it is not uncommon to see applications not working without it. Not only that, but internet access is becoming much more widespread anyway. Sometimes a simple indication to the user that they are offline is enough.

Personally, I would say offline capability is only a real benefit if your app can perform its duties without internet access. The exception to this would be that you have the budget to implement some more advanced functionality to handle an offline mode. If you don't need an offline mode then this should not sway you to a mobile app instead of a web app.

Device Capabilities

Mobile apps are developed to run on specific platforms, such as iOS or Android, and have access to the device's hardware, software and capabilities like the camera, GPS, sensors, microphone, and more. This allows them to provide a more engaging experience to users. For example, a mobile camera app can directly access the device's camera and take photos, a navigation app can access the GPS to provide turn-by-turn directions, and a fitness app can access the device's accelerometer to track movement and activity.

Web applications, on the other hand, are limited to the features and capabilities provided by the browser. For example, a web application that is designed to take photos would need to use the device's camera through the browser's built-in features, which would not provide the same level of control and functionality as a mobile app. Similarly, a web application can access the device's location through the browser by using the Geolocation API, but the level of accuracy, granularity and details available is not comparable with a mobile app.

Access to device capabilities is important when you are trying to decide between a mobile app and a web application. For some apps, access to the mobile device functionality is integral to the functionality and therefore it makes sense for it to be a mobile app. For others, the device functionality can be used to improve the user experience or add to the overall seamlessness of the app, but it is not required or always worth the extra cost and app development time.

User Experience

The user experience (UX) of a mobile app and a web app can be quite different, due to the way they are designed and the platforms on which they run.

Mobile apps are specifically designed for smaller screens and touch-based interfaces. They are typically optimized for use on a single device, such as a smartphone or tablet. This allows for a more seamless and intuitive experience, as the design and layout of the app can be tailored to the specific device and its capabilities. Mobile applications also tend to be faster, smoother, and more responsive than web apps, as they are developed to take advantage of the device's hardware.

Web apps, on the other hand, are designed to be accessed on a wide variety of devices and screen sizes. They are typically optimized for use with a mouse and keyboard, and the layout and design must be able to adapt to different screen sizes and resolutions. They can also bean less performant as they rely on the internet connection, and the experience may vary depending on the quality of the connection.

In terms of software development, a mobile app can be much more difficult, especially if building a native app. It requires in-depth knowledge of the differences between Android and iOS and how to handle that. Hybrid applications can help solve some of these issues but there is still a lot that needs to be done as part of the mobile app testing phase to ensure you create the best UX for your cross platform app. Web app development is much easier as it relies on more widely used web technologies like HTML, CSS and Javascript. There are also a lot of resources and libraries out there to aid with UX design and development, making it much easier for a web developer. This can mean a better user experience can be developed in a shorter amount of time.

Play Store/Apple App Store

Mobile apps are typically distributed through app stores, such as the Apple App Store or Google Play Store. These app stores give developers a way to reach a large audience of potential users. When an app is made available on an app store, it can be easily discovered by users through search and browsing. Additionally, the app store's ranking and curation system can be leveraged to promote popular or well-reviewed apps, which can give them greater visibility.

App stores can also provide other features that improve discoverability, such as recommendations, featured apps and charts that show the top apps based on downloads or user ratings. This can make it easier for users to discover and download new apps, even if they are not actively looking for them. Additionally, App stores also provide reviews, ratings and app descriptions to help users decide if the app is worth downloading or not.

However the stores are very crowded places, so it is still a very competitive market when trying to appear higher up in the stores. It is also a lot more work that you need to consider, there are things like creating assets for store pages, writing store page listings, curating required policy documents for each app store and ensuring your app can make it through Apple's strict review system. These additional things take more time and therefore cost more money.

Web applications need to be accessed via a web browser and require the user to know the URL of the application. Unlike mobile apps, web apps don't have a centralized distribution point, which means they aren't discoverable through a single platform or marketplace. Users have to find the web application by searching the web, or through promotions, advertising or word of mouth. This can make it more challenging for web applications to reach a large audience and acquire new users. This is where SEO comes into play. There is an entire industry built around getting websites and web apps higher up in search and making them more discoverable. Meaning there are a lot of options out there for aiding in discoverability, whether it's online tools or SEO professionals. In some cases, you may find that good SEO is more powerful than just being on the app stores, and you don't have to worry about all of the extra work involved in getting your mobile app on the app store.

Another benefit of using a web app is that it is a much easier process to push out updates. Mobile apps require you to generate new builds for both your android app and your ios app, which must then be uploaded to the stores. You then need to create the new release on the stores and submit them for review before they can be pushed out to users. Once they pass review it is not going to be instantaneous that every user is on the latest version, as it is down to the user when they install the updates. On the other hand, when you want to update your web app it is just a case of pushing the update to your hosting service. Once the update is deployed anyone who accesses your web app will be on the latest version.

Cost

In the previous sections, we have touched on a few of the factors that can cause mobile applications to be a little more expensive than web applications. It is not always the case but I will reiterate and outline the differences.

The first thing to consider is complexity. Does your app require the use of device hardware? If it is a necessity then you may find that it is more cost-effective to build a mobile application. This is because it is easier for a mobile application to access device hardware (e.g camera or GPS). But, generally speaking, mobile app development is harder than web app development, so if you do not require access to specific mobile device functionality then it will most likely be cheaper to develop a web app. This does not just apply to the initial development phase either, software needs to be maintained and due to mobile app development being more complex, it can mean maintenance costs are also higher.

Hosting costs are another big thing to consider. When you build a mobile app you have the additional costs incurred by the Apple App Store and Google Play Store. This can mean more time is spent pushing updates, curating store assets and making updates to get your app through app store reviews. Whereas a web app is hosted with a provider of your choice. There are countless hosting providers out there so it is a competitive market. This makes it easier to get better deals on hosting. That's not to say web apps are always cheaper to host, as some web apps can become expensive if you are handling very high volumes of traffic, but it is easy to keep track of your traffic with tools like google analytics.

If you do decide to go down the mobile platform route, then you must also decide whether you would like to have a native mobile app or a hybrid mobile app. In this case, a hybrid mobile app is significantly more cost-effective. A native application requires you to essentially build two apps, each with its own challenges and knowledge requirements.

Using an S3 object storage provider in Node.js

Reece Charsville — Wed, 14 Jul 2021 09:21:51 +0000

This post originally appeared on my blog here.

Introduction

Today I am going to be running through how to use an S3 object storage provider.

(Just want to see the code? The GitHub is here)

For those who don't know S3 object storage is a cloud service for hosting files. It is accessible via an API, which means it can easily be integrated into your projects. There are hundreds of uses cases but some of the most common involve hosting user-generated content and allowing users to upload profile images.

Some of the most popular providers for S3 storage include Amazon AWS, Vultr and Digital Ocean. They all provide the same service but have a few differences when it comes to price, locations, capacities and bandwidths, so it's worth looking around to see which one suits your needs best.

My first experience with S3 was using AWS. AWS is great.....but its also very confusing, especially for a backend developer like me who tries to stay clear of DevOps as much as he can. I trawled through the AWS documentation trying to understand how to implement the S3 service and after many hours of playing with buckets, policies and IAM roles I got it working. After my ordeal I decided to try other providers to see how the implementations differ (In the hope of finding a simpler solution). It turns out that implementations are the same across providers!

So, I will run you through a very simple example of how to implement a basic S3 object storage in Nodejs. The example I am going to give uses Express and multer for the file upload, however the object storage code is framework agnostic and only requires the aws-sdk.

Preparing our Node.js project

Before we can connect to our S3 provider there are 4 things you will need. These are:

The buckets endpoint URL
The bucket name
Access key
Secret access key

These should be provided to you once you have set up your bucket through your chosen providers dashboard. You will want to ensure that your keys are kept private and securely. So in this example we will use dotenv environment variables.

Firstly, lets create our .env file in our project root:

S3_BUCKET_NAME=your_bucket_name // e.g my-bucket
S3_ENDPOINT_URL=your_endpoint_url // e.g https://eu.amazons3.com/
S3_ACCESS_KEY=your_access_key
S3_SECRET_KEY=your_secret_access_key

Now we have the information for creating a connection, lets go ahead and install the packages for initialising a connection.

The first thing we need is the aws-sdk this is the npm package used for connecting and interacting with an S3 storage. Run the following command to install:

npm install aws-sdk

In this example we are using TypeScript so we can also install some type definitions. If you are using JavaScript then you can ignore this step.

npm install --save-dev @aws-sdk/types

Setting up the connection

Once installed we can create our connection.ts :

import * as S3 from 'aws-sdk/clients/s3';

export default function Connect(path: string | null = ''): S3 {
    return new S3({
        apiVersion: 'latest',
        endpoint: `${process.env.S3_ENDPOINT_URL}${path}`,
        credentials: {
            accessKeyId: process.env.S3_ACCESS_KEY,
            secretAccessKey: process.env.S3_SECRET_KEY,
        },
    });
}

Lets go through this code line by line. So firstly we import the S3 client from the aws-sdk. The aws-sdk includes a lot of features, so we only need to import the S3 client for this implementation.

Next we create our Connect function. This function will new up an S3 client using the credentials that we stored in our environment variables.

Our connect function takes in an optional path parameter. When this is set we can specify the path that we want to upload our file to. For example we may want to upload an image to a subdirectory called images. So we would set the path to 'images'. This path is then appended to the endpoint URL. So as an example our endpoint now becomes https://eu.amazons3.com/images. If we don't set the path parameter the connection will default to the buckets root.

In our configuration we also provide an S3 API version. In this example I will use latest but you may want to pick a version that works for you. You can read up more about API versions and why you should pick one here.

Uploading a file

Now we have a working S3 client instance we can use it to upload files. Lets create a function for uploading a file. For this example we are using multer, so TypeScript users you can install the types with npm i --save-dev @types/multer.

Our upload.ts will look like this:

import { PutObjectOutput, PutObjectRequest } from 'aws-sdk/clients/s3';
import {AWSError} from 'aws-sdk/lib/error';
import * as S3 from 'aws-sdk/clients/s3';
import Connect from './connection';

export default async function Upload(bucket: string, file: Express.Multer.File, objectName: string, path: string | null = null): Promise<string> {
    return new Promise<string>((resolve, reject) => {
        const s3: S3 = Connect(path);
        const params: PutObjectRequest = { Bucket: bucket, Key: objectName, Body: file.buffer, ACL: 'public-read', ContentType: file.mimetype };
        s3.putObject(params, (err: AWSError, data: PutObjectOutput) => {
            if (err) reject(err);
            resolve(`${process.env.S3_ENDPOINT_URL}${bucket}/${path}/${objectName}`);
        });
    });
}

In our Upload function we are passing in 4 parameters:

Parameters	Description
bucket	This is the name of the bucket you set up with the provider and what we have stored in our environment variable (e.g my-bucket).
file	This is the actual file that we are uploading.
objectName	This is the name that we would like to use when we store the file in the cloud. This name should include your file extension. If you are uploading a gif then this should be `image.gif` as oppose to just `image`.
path	(Optional) This is passed straight through to the connection we made previously. So by default it is set to null, which would mean the file is uploaded to the root of the bucket. If you supply `'images'` to this parameter then the file you upload will be stored in a subdirectory called images.

Our Upload function will return a Promise. This will resolve the URL of our uploaded file once the S3 client has finished uploading.

Inside our new Promise, we first use our Connect function to get an initialised S3 client, passing through our optional path parameter.

Then we create our S3 request parameters. In the parameters we set 5 options:

Parameters	Description
Bucket	This is the name of the bucket. We set this using our bucket parameter.
Key	This is the name that is used when the file is stored in the bucket. We use our objectName parameter here.
Body	This is the file we are uploading. This option takes a file buffer. So we use our parameter `file.buffer`
ACL	This option is used to specify the access of the file we are uploading. In this example we are using `'public-read'`. This means that anyone who has the URL of the file we upload can read it. If you want to read more about the different ACL types then read here.
ContentType	This is used to tell S3 the type of file we are uploading. It takes in a file mime type. We pass this in using our file parameters `file.mimetype`.

Next we call the putObject method on the S3 client. We pass in our request parameters above, and define a callback. The callback will give us an error if the upload fails. So we can check if this has a value in our callback and reject our Promise if there is an error. If there is no error then we can resolve our promise with the URL of our object. We construct the URL of our uploaded object using the endpoint URL, bucket name, path and object name. So as an example if uploaded image.gif to an images folder inside our my-bucket, then the URL would be https://eu.amazons3.com/my-bucket/images/image.gif

Deleting a file

When it comes to deleting a file the process is very similar to upload.

We can create a delete.ts:

import {DeleteObjectOutput, DeleteObjectRequest} from 'aws-sdk/clients/s3';
import {AWSError} from 'aws-sdk/lib/error';
import * as S3 from 'aws-sdk/clients/s3';
import Connect from './connection';

export default async function Delete(bucket: string, objectName: string, path: string | null = null): Promise<DeleteObjectOutput> {
    return new Promise<DeleteObjectOutput>((resolve, reject) => {
        const s3: S3 = Connect(path);
        const params: DeleteObjectRequest = { Bucket: bucket, Key: objectName };
        s3.deleteObject(params, (err: AWSError, data: DeleteObjectOutput) => {
            if (err) reject(err);
            resolve(data);
        });
    });
}

This function takes in 3 of the parameters we have seen before:

Parameters	Description
bucket	The name of our bucket we created with the provider and stored in the environment variables.
objectName	The name that we used when storing the object. E.g `image.gif`
path	The path to the object. E.g `'images'` would delete the object with the objectName supplied inside the images subdirectory. If null this defaults to the root of the bucket.

Inside our Promise we use our Connect function to get an initialised S3 client.

We create our request parameters. Setting the Bucket and Key options using our functions parameters.

Then we use the deleteObject method on the client, passing in our request parameters and defining a callback. Just like before we check if the callback has errored and reject the promise if an error occurs.

If no error occurs then we resolve the deleteObject response.

Setting up our Express endpoints

We have defined some functions to connect to our S3 provider, upload objects and delete objects. The next question is how do we use them?

We will use Express and Multer as an example to demonstrate how to use them.

Using our Express app we can define a POST endpoint like the following:

app.post(
    '/upload',
    multer().single('formFile'),
    async (req, res) => {
        if(!req.file) res.status(400).send('Bad Request: No file was uploaded');
        // If you want to retain the original filename and extension just use originalname like below
        // const filename: string = req.file.originalname;
        const fileExtension: string = req.file.originalname.split('.').pop();
        const filename: string = `my-custom-filename.${fileExtension}`;
        const url: string = await Upload(process.env.S3_BUCKET_NAME, req.file, filename, 'images/logo');
        res.status(201).send(url);
    });

This creates an endpoint called /upload which accepts multi-part form data. We use the multer middleware with this endpoint. The multer middleware will look in the submitted form data for the field with the key formFile. This key should be paired with a file. The middleware then attaches the file object to the request under the property file .

In our handler we check that a file has been supplied and throw a Bad Request response if none was sent.

In the example I have shown how to use a custom filename. We read the file extension from our files original name first. Then we create a new filename, appending the original file extension e.g my-custom-filename.gif .

Next we call our Upload function. We pass in the bucket name stored in our environment variables; the file in the request; our custom filename; and in the example I am uploading to the subdirectory images/logo.

After awaiting our Upload we will have the URL of the uploaded file and we can send this in our endpoints response object.

If you would like to see how to use the delete function with an Express endpoint then take a look at the example project.

Example project

I have created a full working example project on GitHub which uses the code we have gone through today. Check it out here.