Forem: Nehal

AWS S3 Pre-Signed URLs: Your Key to Secure Media Uploads

Nehal — Mon, 11 Mar 2024 13:21:15 +0000

Introduction

AWS S3 Pre-Signed URLs offer a powerful solution for managing media file uploads securely and efficiently. In this article, we'll discuss what pre-signed URLs are, why they're beneficial for uploading media files, how the upload process works, and key code best practices to keep in mind when implementing this approach.

What is a Pre-Signed URL?

A pre-signed URL is a time-limited URL granting temporary access to a specific AWS resource. Typically generated by AWS services or SDKs, it allows clients to perform specific actions on a resource without requiring permanent credentials.

Why Choose Pre-Signed URLs for Media File Uploads?

Reduced Load on Your App Server:

Allowing users to upload files directly to Amazon S3 without going through your application server. When using pre-signed URLs, the file upload is handled directly between the client and Amazon S3, bypassing your server. This can reduce the load on your server, making it more scalable, especially in scenarios with a large number of file uploads.

Security:

Providing time-limited access to resources for sharing purposes. Granting temporary access to private objects in Amazon S3 for downloading or viewing.

Now, let's explore what sets pre-signed URLs apart from other methods of uploading files directly to S3, such as multer-s3. Now, let's explore what sets pre-signed URLs apart from other methods of uploading files directly to S3, such as multer-s3.

Multer is a middleware for handling multipart/form-data , which is typically used for file uploads.

Comparison with multer-s3

Server-Side Handling:

When using multer-s3 or a similar library, the file upload is handled directly by your server. This can increase the load on your server, especially if you have a large number of file uploads.

Security Concerns:

multer-s3 involves passing the file through your server first, potentially exposing it to additional security considerations. While Pre-signed url enable direct client-side uploads to S3, bypassing your server.

Media Upload with Pre-Signed URLs

User requests a pre-signed URL through API Gateway.
API Gateway triggers an AWS Lambda function.
Lambda generates a pre-signed URL and a unique key for the object.
User makes a PUT request including the file to upload to S3.
The unique key is then stored in the database.

Role of API Gateway and Lambda

API Gateway:

Acts as a centralized access point
Triggers Lambda upon a URL request
API Gateway can enforce authentication, authorization, and other security policies before allowing clients to generate pre-signed URLs.

AWS Lambda:

Implements business logic for generating pre-signed URLs.
Dynamically sets expiration time based on business rules.
Handles scaling automatically based on demand.

Code best practices

ContentType:

Define Content Type in the URL query parameters. If it is not defined, the object will be uploaded with the default content type, which may lead to incorrect interpretation or processing of the file.
For Example, image/jpeg or image/png for images and video/mp4 or video/mp3 for videos and audios.

Unique Object Name:

Ensure each object has a unique key in the specified bucket. If any duplicate name is used, it will result in overwriting the existing object with the same name in the s3 bucket. We can use a library such as uuid to generate unique id for every object.

Expiration Time:

It's crucial to set an expiration time to enhance security. The expiration time determines the period during which the URL is valid, and after this duration, the URL becomes inactive.

Here's a sample TypeScript code for a Lambda function that retrieves a pre-signed URL from Amazon S3.



import { APIGatewayEvent, Callback, Context } from 'aws-lambda';
import AWS from 'aws-sdk';
import { v4 as uuidV4 } from 'uuid';
import mime from 'mime-types';
import { ACL, GET_PRE_SIGNED_URL_OP } from '../lib/s3/enum';
import { GetSignedUrlParams } from '../lib/s3/interface';
import { messages } from '../messages/en';
import { Body, Headers, Response } from '../lib/response/interface';

const s3 = new AWS.S3();

const getExtensionFromContentType = (mimeType: string): string | false => {
  return mime.extension(mimeType);
};

export const getSignedUrl = (
  operation: string,
  { bucket, key, expires, ACL: acl, contentType }: GetSignedUrlParams,
): string => {
  let params: any = {
    Bucket: bucket,
    Key: key,
    Expires: expires,
  };

  if (operation === GET_PRE_SIGNED_URL_OP.PUT_OBJECT) {
    params = { ...params, ACL: acl, ContentType: contentType };
  }

  return s3.getSignedUrl(operation, params);
};

export const createResponse = (
  statusCode: number,
  body: Body,
  extraHeaders?: Headers,
): Response => {
  const headers = {
    'Content-Type': 'application/json',
  };
  return {
    statusCode,
    body: JSON.stringify(body),
    headers: { ...headers, ...extraHeaders },
  };
};

export const handler = (event: APIGatewayEvent, __: Context, callback: Callback): void => {
  const contentType = event.queryStringParameters?.contentType as string;
  const extension = getExtensionFromContentType(contentType);
  if (!extension) {
    const response = createResponse(400, {
      error: 'Bad request',
      message: messages.error.badContentType,
    });
    callback(null, response);
  }
  const key = `${uuidV4()}.${extension}`;

  const params: GetSignedUrlParams = {
    bucket: process.env.IMAGE_BUCKET as string,
    key,
    ACL: ACL.PUBLIC_READ,
    expires: Number(process.env.S3_SIGNED_IMAGE_URL_EXPIRY),
    contentType,
  };
  const s3ImageUrl = getSignedUrl(GET_PRE_SIGNED_URL_OP.PUT_OBJECT, params);

  const response = createResponse(200, {
    key,
    url: s3ImageUrl,
  });
  callback(null, response);
};

That's all for today. Thanks for reading.

Ehsaan Technologies is a Software Development & Consultancy firm providing drone inspection services to its customers across the Globe.

From Testing to Automation: NestJS and GitHub Actions

Nehal — Sun, 01 Oct 2023 16:32:52 +0000

Introduction

In the world of software development, we're on a mission to make our code rock-solid and reliable. In my previous article, we explored how to test our code in NestJS using Jest. But here's the catch: testing manually can be time-consuming. That's where automation swoops in to save the day.

In this article, we're going to learn a super cool trick: how to make our tests run automatically using GitHub Actions. Think of it like having your own coding assistant that checks your work whenever you make changes. But why do we need this? Well, that's where Continuous Integration (CI) comes in.

Continuous integration refers to the build and unit testing stages of the software release process. Every revision that is committed triggers an automated build and test.

Automation makes our system reliable, resilient and gives us confidence not only to the development team but also the leadership if all tests are passing in CI.

Buckle up, then! As NestJS and GitHub Actions work together to make your coding life easier and your program more dependable, we're going to enter into the world of automation. Let's get started on this exciting journey!

Prerequisites

Before diving into this tutorial, make sure you have a basic understanding of Git/GitHub.

Putting Unit Test Automation into Action

We'll start by setting up a new repository on your GitHub account and initializing Git for your project. Follow these straightforward steps:

Open your project folder in your terminal and run git init
Stage your project files with git add .
Commit your staged changes with git commit -m "commit-name"
Connect your local project to GitHub repository using git remote add origin "your_repository_origin"
Push your changes to GitHub with git push origin master

Now, your project is on GitHub, and you're ready to set up GitHub Actions for automated testing.

In the root folder of your project, create a .github/workflows directory if it doesn't already exist.

Inside the workflows directory, create a test.yml file. This file will contain the configuration for your GitHub Actions workflow.

Next, copy and paste this code into your test.yml file:

test.yml



name: NestJS-UnitTest

on:
  push:
    branches:
      - master
  pull_request:
    branches:
      - master

jobs:
  build:
    runs-on: ubuntu-latest

    strategy:
      matrix:
        node-version: [16.x, 18.x]

    steps:
      - name: Checkout Repository
        uses: actions/checkout@v2

      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v2
        with:
          node-version: ${{ matrix.node-version }}

      - name: Install Dependencies
        run: npm ci

      - name: Run Tests
        run: npm test
        working-directory: src/book

Event Trigger: This workflow is triggered whenever there is a push or pull request to the "master" branch of your GitHub repository.

Node.js Versions: It tests your code on different Node.js versions (16.x and 18.x).

Checkout Repository: This step checks out your code repository, making it available for further actions.

Set Up Node.js: It configures the Node.js environment by using the specified Node.js version.

Install Dependencies: This step installs project dependencies using npm ci to ensure a clean and consistent environment.

Run Tests: It executes your unit tests using "npm test"

The Run Tests task specifies its working directory as src/book because it contains the book.service.spec.ts file where all the tests are located. In contrast, the working directory for the Install Dependencies task is configured as the root directory. This is because the package.json file, which holds the project's dependencies, is located in the root folder.

Finally, after making these configuration changes, you should commit and push them to your GitHub repository. Once you've done that, GitHub will automatically start running the tests associated with your commits. You can easily track the progress and results of these tests by navigating to the GitHub Actions tab in your repository. This way, you can keep an eye on how your automated tests are performing. This ensures that your code is continuously checked for errors and bugs, preventing issues from going unnoticed.

That's all for today. Thanks for reading.

Here you'll find the source code for the implementation mentioned above.

Ehsaan Technologies is a Software Development & Consultancy firm providing customized Web development and Mobile App development services to its customers across the Globe.

Mastering Unit Testing With NestJS

Nehal — Wed, 13 Sep 2023 13:00:03 +0000

Introduction

Unit testing is a software development process in which the smallest testable parts of an application, called units, are individually scrutinized for proper operation. In this guide, we will walk through an example of unit testing in Node.js using the NestJS framework. We will focus on testing a BookService class, covering its methods for creating, retrieving, updating and deleting books while demonstrating best practices for testing.

Prerequisites

Before diving into unit testing, make sure you have a basic understanding of Node.js, NestJS, Jest and MongoDB.

Setting up the Environment

To begin, let's set up the environment by creating a book.service.ts file. Additionally, we'll install and import all necessary dependencies and construct a BookService class.

Next, proceed to develop a function for generating a new book entry using MongoDB and the Mongoose library.

Then, develop a function to find a book by its id.

Similarly, create a function to update a book.

Finally, create a function to delete a book.

Unit Testing the BookService

Now, as we delve into the unit testing process for the BookService class, our aim is to ensure that each function operates precisely as intended.

Initializing the Testing Environment

First create a book.service.spec.ts file in the same directory as the book.service file.
In unit testing for NestJS, the Test and TestingModule classes from @nestjs/testing establish the testing environment. To simulate database interactions, the getModelToken function from @nestjs/mongoose is employed. Through the jest.fn() function, a mock BookService instance is created to track method calls. This mock service and a simulated database model are provided by configuring the testing module using getModelToken(Book.name). After compilation, the BookService instance and model are accessed using the module.get() method.

Testing the 'create' Function of BookService

In this test scenario, we create a mock object that simulates a new book. We then set up a mock implementation of the create function within the model to resolve with our predefined mock book. By calling the create function of our service with the mock book data, we ensure that the service responds as anticipated. The expect statement confirms that the result corresponds to our predefined mock book, thereby validating the correctness of the create function. Also initialize all the required dependencies.

Testing the 'findById' Function of BookService

For this test scenario, we begin by importing essential dependencies and creating a mockBook object. This object will serve as a placeholder for simulating a book's attributes and properties during our tests.

Then we thoroughly test findById function. We have declared three test cases.

The first test case ensures that the function correctly finds and returns a book by its ID. We achieve this by mocking the model's findById function to resolve with our predefined mock book, and then verifying if the result matches the expectation.

The subsequent test cases handle error scenarios. In the second test case, we simulate an invalid ID and assert that calling findById with this ID triggers a BadRequestException. We achieve this by mocking the isValidObjectId function to return false and then using expect to confirm the thrown error.

The third test case tests the condition where the findById function cannot find the requested book. We mock the model's findById function to return null, simulating a scenario where the book is not found. As expected, we then verify if calling findById with this ID raises a NotFoundException.

By testing these different scenarios, we ensure that our findById function behaves correctly and handles exceptions as intended.

Testing the 'updateById' Function of BookService

In this test scenario, we verify the functionality of updating and returning a book's information. The test scenario involves simulating the update of a book's title and confirming that the updated book object is returned as expected.

During the test execution, the service's 'updateById' function is called with the relevant book ID and the updated book data. The mock implementation ensures that the service interacts with the model's 'findByIdAndUpdate' method as intended.

Testing the 'deleteById' Function of BookService

The test ensures that the "deleteById" function successfully deletes a book and returns the expected result. By mocking the database operation using "findByIdAndDelete," we verify that the function interacts correctly with the data model.

Running Test Scenarios

After completing our test scenarios to cover various aspects of our application's functionality, it's time to see them in action.

Must ensure that you are in the root directory of your Nest.js project, follow these steps:

Open your terminal or command prompt.
Enter the following command: npm run test

This command initiates the execution of your unit tests using the testing framework we set up earlier.

Running the tests is a crucial step in ensuring the reliability and correctness of your code base.

Conclusion

Throughout this guide, we explored the core concepts of unit testing in NestJS, including the setup of testing modules, the creation of mock services, and the execution of test cases.

Ehsaan Technologies is a Software Development & Consultancy firm providing customized Web development and Mobile App development services to its customers across the Globe.

Forem: Nehal

AWS S3 Pre-Signed URLs: Your Key to Secure Media Uploads

Introduction

What is a Pre-Signed URL?

Why Choose Pre-Signed URLs for Media File Uploads?

Reduced Load on Your App Server:

Security:

Comparison with multer-s3

Server-Side Handling:

Security Concerns:

Media Upload with Pre-Signed URLs​

Role of API Gateway and Lambda

API Gateway:

AWS Lambda:

Code best practices

ContentType:

Unique Object Name:

Expiration Time:

From Testing to Automation: NestJS and GitHub Actions

Introduction

Prerequisites

Putting Unit Test Automation into Action

test.yml

Mastering Unit Testing With NestJS

Introduction

Prerequisites

Setting up the Environment

Unit Testing the BookService

Initializing the Testing Environment

Testing the 'create' Function of BookService

Testing the 'findById' Function of BookService

Testing the 'updateById' Function of BookService

Testing the 'deleteById' Function of BookService

Running Test Scenarios

Conclusion

Media Upload with Pre-Signed URLs