Forem: Kevin Lactio Kemta

How to combine SQS and SNS to implement multiple Consumers (Part 2)

Kevin Lactio Kemta — Tue, 15 Oct 2024 04:30:00 +0000

Day 020 - 100DaysAWSIaCDevopsChallenge

In my previous article, Why you Shouldn't Use Amazon SQS for multiple Consumers—Choose Amazon SNS Instead! (Part 1), I discussed the differences between Amazon SQS and Amazon SNS from my perspective. The key takeaway was that Amazon SQS is better suited for scenarios where messages need to be processed by a single consumer, whereas Amazon SNS excels at broadcasting notifications to multiple systems simultaneously.

Today, I'll demonstrate how to combine SNS and SQS when you have a message producer that needs to communicate with multiple consumers/subscribers.

Scenario

Let's get straight to the point. The goal of this article is to demonstrate how to create a simple and effective Pub/Sub system. The concept revolves around a basic user registration system. Her's an overview:

A SpringBoot Application exposes an API that allows users to be created.
Once a user is created, a message containing the user's information is sent to an SNS Topic.
Two other applications consume the messages published to this topic throught SQS Queue:
- the first indexes the user (for future search functionality, e.g: with OpenSearch)
- the second generates a user code (similar to a PIN) that con be used by the user to access certain services or applications.

Below is the complete scenario illustrated.

+--------------------------+         +------------------------+
|  User Management App     |         |  Amazon SNS Topic      |
|  (Producer)              |-------->|  "UserCreatedTopic"    |
|  Spring Boot             |         |  (Broadcast messages)  |
+--------------------------+         +------------------------+
                                        /           \
                                       /             \
                                      /               \
                  +-----------------------+    +----------------------+
                  |   Amazon SQS Queue 1  |    |  Amazon SQS Queue 2  |
                  |   "SearchQueue"       |    |  "CodeGenQueue"      |
                  +-----------------------+    +----------------------+
                             |                                   |
                             v                                   v
              +--------------------------+      +--------------------------+
              |  Search Indexing App     |      |  Code Generation App     |
              |  (Consumer)              |      |  (Consumer)              |
              |  Spring Boot             |      |  Spring Boot             |
              +--------------------------+      +--------------------------+

Producer & Consumers

To keep things simple and avoid complicating my life, I will use Spring Cloud AWS Docs[↗]

Spring Cloud AWS simplifies using AWS managed services in a Spring Framework and Spring Boot applications. It offers a convenient way to interact with AWS provided services using well-known Spring idioms and APIs.

To configure the SNS and SQS services, this is the beans configuration class for every application:

@Configuration
public class ApplicationConfiguration {
    @Bean
    public AwsRegionProvider customRegionProvider() {
        return new InstanceProfileRegionProvider();
    }
    @Bean
    public AwsCredentialsProvider customInstanceCredProvider() {
        return  InstanceProfileCredentialsProvider.builder()
                .build();
    }
}

User Management App (`producer`)

package com.nivekaa.msproducer.infra.messaging.producers;
import com.nivekaa.msproducer.core.domain.User;
import com.nivekaa.msproducer.core.interactors.IndexingUserInteractor;
import io.awspring.cloud.sns.core.SnsNotification;
import io.awspring.cloud.sns.core.SnsOperations;
import lombok.RequiredArgsConstructor;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;

@Component
@RequiredArgsConstructor
public class IndexingUserProducer implements IndexingUserInteractor {
  private final SnsOperations snsOperations;
  private @Value("${sns.topic.users}") String userTopic;
  @Override
  public void pushUserAsMessage(User payload) {
    SnsNotification<User> notification = SnsNotification.builder(payload)
        .build();
    snsOperations.sendNotification(userTopic, notification);
  }
}

## resources/application.properties
sns.topic.users=UserCreatedTopic

The code above is the crucial part, as it handles the publication of the message after a user is created. Given that the application is fairly large and spans multiple files, I’ve decided to provide a link to the full project on GitHub for a more comprehensive view and better context. GitHub repo: User Management[↗]

Search Indexing App (`consumer`)

package com.nivekaa.sqsconsumer.infra.messaging.consumers;
import java.time.LocalDateTime;
import io.awspring.cloud.sqs.annotation.SqsListener;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Component;

@Slf4j
@Component
public class ExampleSQSConsumer {
    @SqsListener(queueNames = { "${sqs.queue.index-user}" })
    public void listen(String payload) {
        log.info("""
        *******************  SQS Payload ***************
        * User Info: {}
        * Received At: {}
        ************************************************
        """, payload, LocalDateTime.now());
    }
}

## resources/application.yml
sqs:
  queue:
    index-user: SearchQueue-main

Follow the full project GitHub repo[↗]

Code Generation App (`consumer`)

@Slf4j
@Component
@RequiredArgsConstructor
public class UserRegisterSQSConsumer {
    private final LocalStorageInteractor storageInteractor;
    @SqsListener(queueNames = { "${sqs.queue.gen-code}" })
    public void listen(String payload) {
        log.info("""
        *******************  SQS Payload ***************"
        * Message Content: {}
        * Received At: {}
        ************************************************
        """, payload, Date.from(Instant.now()));
    // Unwanted code👇🏻👇🏻...
}

## resources/application.yml
sqs:
  queue:
    gen-code: CodeGenQueue-main

Follow the full project GitHub repo[↗]

Build infrastructure using CDK with java as language


// MayStack.java
public class MyStack extends Stack {
  public MyStack(final Construct scope, final String id) {
    this(scope, id, null);
  }
  public MyStack(final Construct scope, final String id, final StackProps props) {
    super(scope, id, props);
    // Create a VPC (Virtual Private Cloud) using a custom NetworkConstruct class.
    IVpc vpc = new NetworkContruct(this, "NetworkResource", props).getVpc();
    // Define the first SQS queue for user code generation, using a custom QueueConstruct.
    IQueue genUserCodeQueue =
        new QueueConstruct(
                this, "GenerateUserCodeQueueResource", new CustomQueueProps("CodeGenQueue"), props)
            .getQueue();
    // Define the second SQS queue for user indexing, also using a custom QueueConstruct.
    IQueue indexingUserQueue =
        new QueueConstruct(
                this, "IndexingUserQueueResource", new CustomQueueProps("SearchQueue", true), props)
            .getQueue();
    // Create an SNS topic named "UserCreatedTopic" using a custom TopicConstruct class.
    TopicConstruct topic =
        new TopicConstruct(this, "TopicResource", new CustomTopicProps("UserCreatedTopic"), props);

    // Subscribe both SQS queues to the SNS topic, so they will receive messages when published.
    topic.subscribeQueue(genUserCodeQueue);
    topic.subscribeQueue(indexingUserQueue);

    // Define an EC2 instance (Compute) for the code generation service.
    ComputerConstruct genCodeService =
        new ComputerConstruct(
            this,
            "GenCodeServerResource",
            ComputerProps.builder()
                .allowSSHConnection(true)
                .vpc(vpc)
                .instanceName("GenCodeServer")
                .volumeSize(8)
                .enableKeyPair(true)
                .hostedAppPort(8081)
                .bootstrapScript("./gen-code-webserver-startup.sh")
                .build(),
            props);
    // Grant the EC2 instance permissions to consume messages from the SQS queue for code generation.
    genCodeService.addPolicyToComputer(
        PolicyStatement.Builder.create()
            .sid("AllowConsumingSQSMessage")
            .actions(
                List.of(
                    "sqs:DeleteMessage",
                    "sqs:ReceiveMessage",
                    "sqs:GetQueueAttributes",
                    "sqs:GetQueueUrl"))
            .resources(List.of(genUserCodeQueue.getQueueArn()))
            .effect(Effect.ALLOW)
            .build());

    // Define an EC2 instance for the user management service (message producer).
    ComputerConstruct userManagementService =
        new ComputerConstruct(
            this,
            "ProducerComputerResource",
            ComputerProps.builder()
                .volumeSize(10)
                .enableKeyPair(true)
                .allowSSHConnection(true)
                .hostedAppPort(8080)
                .vpc(vpc)
                .instanceName("MSProducer")
                .bootstrapScript("./producer-webserver-startup.sh")
                .build(),
            props);

    // Grant the user management EC2 instance permissions to publish messages to the SNS topic.
    userManagementService.addPolicyToComputer(
        new PolicyStatement(
            PolicyStatementProps.builder()
                .effect(Effect.ALLOW)
                .actions(List.of("sns:Publish", "sns:ListTopics", "sns:CreateTopic"))
                .resources(List.of(topic.getTopic().getTopicArn()))
                .build()));

    // Define an EC2 instance for the indexing service (message consumer).
    ComputerConstruct indexingComputer =
        new ComputerConstruct(
            this,
            "IndexingComputerResource",
            ComputerProps.builder()
                .bootstrapScript("./indexing-webserver-startup.sh")
                .vpc(vpc)
                .instanceName("indexing-webserver")
                .enableKeyPair(true)
                .allowSSHConnection(true)
                .volumeSize(8)
                .build(),
            props);

    // Grant the indexing EC2 instance permissions to consume messages from the SQS queue for indexing.
    indexingComputer.addPolicyToComputer(
        PolicyStatement.Builder.create()
            .effect(Effect.ALLOW)
            .sid("AllowInstanceToReceiveSQSMessage")
            .actions(
                List.of(
                    "sqs:DeleteMessage",
                    "sqs:ReceiveMessage",
                    "sqs:GetQueueAttributes",
                    "sqs:GetQueueUrl"))
            .resources(List.of(indexingUserQueue.getQueueArn()))
            .build());
  }
}


// MyApp.java
public class Day020App {
  public static void main(final String[] args) {
    App app = new App();
    new MyStack(
        app, "Day020Stack",
        StackProps.builder()
            .env(Environment.builder()
                    .account(System.getenv("CDK_DEFAULT_ACCOUNT"))
                    .region(System.getenv("CDK_DEFAULT_REGION"))
                    .build())
            .build());
    app.synth();
  }
}

🎯NetworkContruct.java - [↗]
🎯ComputerConstruct.java - [↗]
🎯QueueConstruct.java - [↗]
🎯TopicConstruct.java - [↗]

Deployment

⚠️⚠️ Before run the deployment command ensure that you have java installed on your host machine. I used Java 21 under MacOs to build this insfrastructure.

Open the terminal anywhere and run the following commande:

git clone https://github.com/nivekalara237/100DaysTerraformAWSDevops.git
cd 100DaysTerraformAWSDevops/day_020
cdk bootstrap --profile cdk-user
cdk deploy --profile cdk-user Day020Stack

You can find the full project in my GitHub repo[↗]

Why you Shouldn't Use Amazon SQS for multiple Consumers—Choose Amazon SNS Instead! (Part 1)

Kevin Lactio Kemta — Mon, 14 Oct 2024 04:30:00 +0000

Day 019 - 100DaysAWSIaCDevopsChallenge

Recently, I’ve come across questions in communities like Stack Overflow and Reddit, where users are asking how to enable multiple applications (acting as consumers) to process a single message from an Amazon SQS Queue. While it is technically possible to consume or read an SQS message multiple times, in this article, I will walk you through the best practices for handling situations where you have one queue serving multiple consumers.

Amazon SQS vs Amazon SNS

Points of comparison	Simple Queue Service	Simple Notification Service
Communication model	`Queue based`, SQS is used for Point-to-Point communication. The producer pushes a message to the `queue`, and only one consumer retrieves and processes it. Messages are garanteed to be processed only one time(idempotent).	`Pub/sub`, SNS is used to broadcast messages to multiple subscribers simultaneously. The message is published on a `topic` and all subscribers receive it in parallel.
Destination	Only one destination per message. Once the message is consumed, it is desappeared/deleted from the queue. It is suitable for systems where a single service must process the message	Designed for multiple destinations. The message is sent to many consumers like applications (HTTP/S), SMS, SQS or Lambda Function. It is suitable for the broadcast at scale
Ordonnancing and duplication	There are two types of SQS: `STANDARD` and `FIFO` (First-In-First-Out). In the `STANDARD` version, messages can be delivered in any order and can be duplicated. In the `FIFO` version, messages are delivered exactly in the order they are sent and no duplication is allowed.	The order is no guaranteed and no protection against duplication. Each consumer receives the message at the same time and in any order.
Message retention	Messages are retained inside the queue until they are consumed or until the retention period expires (the maximun retention period is 14 Days)	Message are not retained. They are immediately sent to the subscribers as soon as they published. ⚠️⚠️ If the subscriber is not available, the message is lost(only if he is attached to the SQS Queue)
Use Case	Buffering 📬	Broadcasting 🛗
Pricing	Based on the number of sent and receive requests and messages deletions	Based on publising and noticications sent

There might be other elements of comparison that I haven't mentioned here. But for now we'll stop at the most important ones here. If you think of anything else, please leave a comment, it could also help me😇.

Which one to choose ?

Now that we've established the differences between SQS and SNS, the question arises: when should you use one over the other?

You can use SQS when:

you wish to control the order of message processing using SQS FIFO
you need to handle a high volume of messages or process messages in batches
you have an asynchronous workflow where only one consumer should process each message.

You can use SNS when:

you need to send the same message to multiple consumers simultanously.
you're to implementing a Pub/Sub(Publish/Subscribe) model
you require real-time proccessing, such as push notifications
you want to alert multiple systems in real time (e.g. Email, SMS, monitoring, SQS Queue, etc.).

To summarize:
✅✅ SQS is suitable for systems where messages need to be processed by a single consumer.
✅✅ SNS is perfect for broadcasting notifications to multiple systems simultaneously.

In practice, you can combine SNS and SQS, where SNS sends notifications to multiple SQS Queues, allowing each queue to have its dedicated consumer. This will be covered in the next article (How to combine SQS and SNS to implement multiple Consumers). stay turned!

Conclusion

The right choice between Amazon SQS and SNS can elevate your application’s messaging capabilities. If you have any experiences, questions, or insights to share, feel free to leave a comment below 💬.

Let’s learn and grow together!😎

Create an AWS SNS Topic Using CDK and consume messages with a Spring Boot Microservice

Kevin Lactio Kemta — Sun, 13 Oct 2024 18:59:04 +0000

Day 018 - 100DaysAWSIaCDevopsChallenge

Amazon SNS (Simple Notification Service) is a fully managed Amazon messaging service that allows large number of subscribers using different delivery protocoles, such as HTTP/HTTPS, email, SQS, SMS and AWS Lambda.
It designed for scalable, high-throughput, push-bash messaging. It enables applications, microservices, and systems to communicate with each other asynchroniously in real time.
The core concept of SNS revolvers around topicsand subscriptions.

Subscribe and Consume a SNS Messages using Springboot application

This guide demonstrates how to create a Spring Boot application that subscribes to and processes messages from an SNS topic. The infrastructure, built using AWS CDK (in Java), includes the following components:

A VPC with a public Subnet to host an EC2 instance where the Spring Boot application will run.
An Internet Gateway to provide the EC2 instance with internet access for downloading dependencies.
An SNS Topic for publishing messages.
An EC2 Instance for hosting the Spring Boot application.
An IAM Role to grant the EC2 instance permissions to receive messages from the SNS topic (critical for secure communication).

Create the Infrastructure

Set up the necessary infrastructure using CDK (Java)

VPC & Subnet + Internet Gateway

// constructs/NetworkConstruct.java
public class NetworkContruct extends Construct {
  private final IVpc vpc;
  public NetworkContruct(Construct scope, String id, StackProps props) {
    super(scope, id);
    this.vpc =
        new Vpc(
            this,
            "VpcResource",
            VpcProps.builder()
                .vpcName("my-vpc")
                .enableDnsHostnames(true)
                .enableDnsSupport(true)
                .createInternetGateway(true)
                .ipProtocol(IpProtocol.IPV4_ONLY)
                .ipAddresses(IpAddresses.cidr("10.0.0.1/16"))
                .maxAzs(1)
                .subnetConfiguration(
                    List.of(
                        SubnetConfiguration.builder()
                            .name("Public-Subnet")
                            .mapPublicIpOnLaunch(true)
                            .subnetType(SubnetType.PUBLIC)
                            .build()))
                .build());
  }
  public IVpc getVpc() {
    return vpc;
  }
}

// MyStatck.java
public class MyStack extends Stack {
  public MyStack(final Construct scope, final String id, final StackProps props) {
    super(scope, id, props);
    IVpc vpc = new NetworkContruct(this, "NetworkResource", props).getVpc();
  }
}

The above code will create:

A VPC named my-vpc and enable DNS hostname enabled.
A public subnet named Public-Subnet which allows resources to attach a public IP (if configured with one).
An Internet Gateway to enable internet traffic.

SNS Topic

// MyStatck.java
public class MyStack extends Stack {
  public MyStack(final Construct scope, final String id, final StackProps props) {
    super(scope, id, props);
    //...
    String topicName = "example-topic-main";
    ITopic topic = new Topic(
        this, "TopicResource", TopicProps.builder()
        .topicName(topicName)
        .fifo(false)
        .build());
  }
}

The above code creates an SNS Topic named example-topic-main. This topic will be used for publishing messages, which the Spring Boot application will subscribe to and process.

EC2 Instance for Hosting the Spring Boot Application

// MyStatck.java
public class MyStack extends Stack {
  public MyStack(final Construct scope, final String id, final StackProps props) {
    super(scope, id, props);
    //...
    IVpc vpc; // previously instanciated
    String topicName;
    int port = 8089;
    ITopic topic; // previously instanciated

    ComputerConstruct webserver =
            new ComputerConstruct(
                this, "ComputerResource", ComputerProps.builder().vpc(vpc).port(port).build(), props);
    IInstance instance = webserver.getComputer();
    webserver.addPolicyToComputer(
        PolicyStatement.Builder.create()
            .effect(Effect.ALLOW)
            .resources(List.of(topic.getTopicArn()))
            .actions(
                List.of(
                    "sns:ConfirmSubscription",
                    "sns:Subscribe",
                    "sns:GetTopicAttributes",
                    "sns:ListTopics"))
            .build());
    ITopicSubscription endpointSubscription =
        new UrlSubscription(
            "http://%s:%d/topics/%s"
                .formatted(instance.getInstancePublicDnsName(), port, topicName),
            UrlSubscriptionProps.builder()
                .rawMessageDelivery(false)
                .protocol(SubscriptionProtocol.HTTP)
                .build());
    topic.addSubscription(endpointSubscription);
  }
}

ComputerConstruct.java [↗]
The above CDK construct will create the following resources:

A Security Group named Webserver-security-group that allows inbound traffic on Port 22 for SSH connections and allows inbound traffic on Port 8089, which is the application connection port.
A Key Pair named ws-keypair that will be used to connect to the app host via SSH. Since we are using CDK to build the infrastructure, if you need to download the private key (PEM file) after deployment, refer to my previous article on How the retrieve the private key file PEM after Cloudformation or CDK stack creation[↗].
An Ec2 Instance named Webserver-Instance.
An IAM Role for the Ec2 Instance named webserver-role, which allows the spring Boot application hosted on the Ec2 Instance to establish connections with the Amazon SQS Queue (already created) and perform actions: sns:ConfirmSubscription, sns:Subscribe, sns:GetTopicAttributes and sns:ListTopics.
An SNS Subscription for the HTTP endpoint, allowing the Spring Boot application hosted on the EC2 instance to receive and process messages from the SNS topic.

Create the stack

// Day17App.java
public class Day017App {
  public static void main(final String[] args) {
    App app = new App();
    new MyStack(app,"Day017Stack",
        StackProps.builder()
                .env(
                    Environment.builder()
                        .account(System.getenv("CDK_DEFAULT_ACCOUNT"))
                        .region(System.getenv("CDK_DEFAULT_REGION"))
                        .build())
                .build());
    app.synth();
  }
}

Create SpringBoot Subscriber Application

To keep things simple and avoid complicating my life, I will use Spring Cloud AWS Docs[↗]

Spring Cloud AWS simplifies using AWS managed services in a Spring Framework and Spring Boot applications. It offers a convenient way to interact with AWS provided services using well-known Spring idioms and APIs.

To configure the SNS service, add the following beans in the configuration class:

@Configuration
public class ApplicationConfiguration {
    @Bean
    public AwsRegionProvider customRegionProvider() {
        return new InstanceProfileRegionProvider();
    }
    @Bean
    public AwsCredentialsProvider customInstanceCredProvider() {
        return  InstanceProfileCredentialsProvider.builder()
                .build();
    }
}

Finally, add a controller designed to handle incoming notifications from an AWS SNS (Simple Notification Service) topic. This controller allows the application to subscribe to the SNS topic and receive messages sent to it. It utilizes annotations from the Spring Cloud AWS library to manage SNS message mappings effectively.

@RestController
@RequestMapping("/topics/${sns.topic.name}") // ${sns.topic.name} will return example-topic-main
@Slf4j
public class ConsumeNotificationResource {

  @NotificationSubscriptionMapping
  public void confirmSubscription(NotificationStatus status) {
    status.confirmSubscription();
    log.info("Subscription confirmed");
  }

  @NotificationMessageMapping
  public void receiveMessage(@NotificationSubject String subject, @NotificationMessage String message) {
    log.info(
        """
        ************************* SNS Notification ***************
        * Subject : {}
        * Content: {}
        * Date : {}
        **********************************************************
        """,
        subject,
        message,
        LocalDateTime.now());
  }

  @NotificationUnsubscribeConfirmationMapping
  public void unsubscribe(NotificationStatus status) {
    status.confirmSubscription();
    log.info("Unsubscription confirmed");
  }
}

You can find the full project in my GitHub repo[↗]

Deployment

⚠️⚠️ Before run the deployment command ensure that you have java installed on your host machine. I used Java 21 under MacOs to build this insfrastructure.

Open the terminal anywhere and run the following commande:

git clone https://github.com/nivekalara237/100DaysTerraformAWSDevops.git
cd 100DaysTerraformAWSDevops/day_018
cdk bootstrap --profile cdk-user
cdk deploy --profile cdk-user Day018Stack

Resut

Your can find the full source code here #day_18

nivekalara237 / 100DaysTerraformAWSDevops

Thriving with AWS and Terraform

articles about my work 👉🏽 https://dev.to/nivekalara237/series/28339

View on GitHub

Building a Spring Boot Consumer Application with Amazon SQS: Setup Infrastructure Using Cloud Development Kit (CDK)

Kevin Lactio Kemta — Mon, 07 Oct 2024 12:59:45 +0000

Day 017 - 100DaysAWSIaCDevopsChallenge

Today in my series of 100 days of code challenge, I will show you how to decouple microservices developed with springboot using Amazon SQS.

What is Amazon SQS ?

Amazon SQS (Simple Queue Service) is a cloud service that helps applications communicate by sendyg, storing and receiving messages in a queue. It's like a waiting line where are stored until a consumer is ready to process them. This prevents systems from getting overhelmed and make sure no message is lost.

Consume a SQS Messages using Springboot application

To demonstrate how to consume SQS messages by creating a Spring Boot app that processes each message from an SQS queue. The infrastructure, built using CDK (Java), will include:

a VPC with a public Subnet to host an EC2 instance where the Spring Boot app will run.
An Internet Gateway for the EC2 instance to access the internet and download dependencies.
A SQS Queue and Dead Letter Queue for message storage.
An EC2 Instance for hosting SpringBoot App.
An IAM Role to allow the EC2 instance to retrieve messages from the SQS queue (very important).

Create the Infrastructure

Set up the necessary infrastructure using CDK (Java)

VPC & Subne + Internet Gateway

public class NetworkContruct extends Construct {
  private final IVpc vpc;
  public NetworkContruct(Construct scope, String id, StackProps props) {
    super(scope, id);
    this.vpc =
        new Vpc(
            this,
            "VpcResource",
            VpcProps.builder()
                .vpcName("my-vpc")
                .enableDnsHostnames(true)
                .enableDnsSupport(true)
                .createInternetGateway(true)
                .ipProtocol(IpProtocol.IPV4_ONLY)
                .ipAddresses(IpAddresses.cidr("10.0.0.1/16"))
                .maxAzs(1)
                .subnetConfiguration(
                    List.of(
                        SubnetConfiguration.builder()
                            .name("Public-Subnet")
                            .mapPublicIpOnLaunch(true)
                            .subnetType(SubnetType.PUBLIC)
                            .build()))
                .build());
  }
  public IVpc getVpc() {
    return vpc;
  }
}

This cdk construct will create:
👉🏽 A VPC named my-vpc and enable DNS hostname enabled.
👉🏽 A public subnet named Public-Subnet which allows resources to attach a public IP (if configured with one).
👉🏽 An Internet Gateway to enable internet traffic.

SQS Queue & Dead Letter Queue

public class QueueConstruct extends Construct {
  private final IQueue queue;
  public QueueConstruct(Construct scope, String id, IVpc vpc, StackProps props) {
    super(scope, id);
    IQueue dlq =
        new Queue(
            this,
            "DeadLetterQueue",
            QueueProps.builder()
                .deliveryDelay(Duration.millis(0))
                .retentionPeriod(Duration.days(10))
                .queueName("my-queue-dlq")
                .build());
    DeadLetterQueue deadLetterQueue = DeadLetterQueue.builder()
        .queue(dlq)
        .maxReceiveCount(32)
        .build();

    this.queue =
        new Queue(
            this,
            "SQSQueueResource",
            QueueProps.builder()
                .queueName("my-queue")
                .retentionPeriod(Duration.minutes(15))
                .visibilityTimeout(Duration.seconds(90))
                .deadLetterQueue(deadLetterQueue)
                .build());
  }

  public IQueue getQueue() {
    return queue;
  }
}

The above CDK construct will create the following resources:

A Queue named my-queue, which will be used in the Spring Boot app.
A DeadLetter Queue named my-queue-dlq which captures failed messages so they can be analysed and fixed later, without blocking the main queue.

EC2 Instance for Hosting the Spring Boot Application

// ComputerProps.java
public record ComputerProps(IVpc vpc, String sqsQueueArn) {}

// ComputerConstruct.java
public class ComputerConstruct extends Construct {
  private final IInstance computer;
  public ComputerConstruct(
      Construct scope, String id, ComputerProps computerProps, StackProps props) {
    super(scope, id);
    SecurityGroup securityGroup =
        new SecurityGroup(
            this,
            "WebserverSGResource",
            SecurityGroupProps.builder()
                .allowAllOutbound(true)
                .securityGroupName("Webserver-security-group")
                .disableInlineRules(true)
                .vpc(computerProps.vpc())
                .description("Allow trafic from/to webserver instance")
                .build());

    securityGroup.addIngressRule(Peer.anyIpv4(), Port.SSH, "Allow ssh traffic");
    securityGroup.addIngressRule(Peer.anyIpv4(), Port.tcp(8089), "Allow traffic from 8089 port");

    KeyPair keyPair =
        new KeyPair(
            this,
            "KeyPairResource",
            KeyPairProps.builder()
                .keyPairName("ws-keypair")
                .account(Objects.requireNonNull(props.getEnv()).getAccount())
                .type(KeyPairType.RSA)
                .format(KeyPairFormat.PEM)
                .build());

    new CfnOutput(
        this, "KeyPairId", CfnOutputProps.builder().value(keyPair.getKeyPairId()).build());

    Instance ec2Instance =
        new Instance(
            this,
            "WebServerInstanceResource",
            InstanceProps.builder()
                .securityGroup(securityGroup)
                .keyPair(keyPair)
                .instanceName("Webserver-Instance")
                .machineImage(
                    MachineImage.lookup(
                        LookupMachineImageProps.builder()
                            .name("*ubuntu*")
                            .filters(
                                Map.ofEntries(
                                    Map.entry("image-id", List.of("ami-0e86e20dae9224db8")),
                                    Map.entry("architecture", List.of("x86_64"))))
                            .windows(false)
                            .build()))
                .vpc(computerProps.vpc())
                .role(buildInstanceRole(computerProps))
                .instanceType(InstanceType.of(InstanceClass.T2, InstanceSize.MICRO))
                .associatePublicIpAddress(true)
                .blockDevices(
                    List.of(
                        BlockDevice.builder()
                            .mappingEnabled(true)
                            .deviceName("/dev/sda1")
                            .volume(
                                BlockDeviceVolume.ebs(
                                    10,
                                    EbsDeviceOptions.builder()
                                        .deleteOnTermination(true)
                                        .volumeType(EbsDeviceVolumeType.GP3)
                                        .build()))
                            .build()))
                .userDataCausesReplacement(true)
                .vpcSubnets(SubnetSelection.builder().subnetType(SubnetType.PUBLIC).build())
                .build());

    UserData userData = UserData.forLinux();
    userData.addCommands(readFile("./webserver-startup.sh"));

    ec2Instance.addUserData(userData.render());

    this.computer = ec2Instance;
  }

  public IInstance getComputer() {
    return computer;
  }

  private String readFile(String filename) {

    InputStream scriptFileStream = getClass().getClassLoader().getResourceAsStream(filename);

    try {
      assert scriptFileStream != null;
      try (InputStreamReader isr = new InputStreamReader(scriptFileStream, StandardCharsets.UTF_8);
          BufferedReader br = new BufferedReader(isr)) {
        StringBuilder content = new StringBuilder();
        String line;
        while ((line = br.readLine()) != null) {
          content.append(line).append("\n");
        }
        return content.toString();
      }
    } catch (IOException e) {
      throw new RuntimeException(e.getMessage());
    }
  }

  private IRole buildInstanceRole(ComputerProps props) {
    return new Role(
        this,
        "WebserverInstanceRoleResource",
        RoleProps.builder()
            .roleName("webserver-role")
            .assumedBy(new ServicePrincipal("ec2.amazonaws.com"))
            .path("/")
            .inlinePolicies(
                Map.ofEntries(
                    Map.entry(
                        "sqs",
                        new PolicyDocument(
                            PolicyDocumentProps.builder()
                                .assignSids(true)
                                .statements(
                                    List.of(
                                        new PolicyStatement(
                                            PolicyStatementProps.builder()
                                                .effect(Effect.ALLOW)
                                                .actions(
                                                    List.of(
                                                        "sqs:DeleteMessage",
                                                        "sqs:ReceiveMessage",
                                                        "sqs:SendMessage",
                                                        "sqs:GetQueueAttributes",
                                                        "sqs:GetQueueUrl"))
                                                .resources(List.of(props.sqsQueueArn()))
                                                .build())))
                                .build()))))
            .build());
  }
}

The above CDK construct will create the following resources:

A Security Group named Webserver-security-group that allow inbound traffic on Port 22 for SSH connections and allow inbound traffic on Port 8089, which is the application connection port.
A Key Pair named ws-keypair that will be used to connect to the app host via SSH. Since we are using CDK to build the infrastructure, if you need to download the private key (PEM file) after deployment, refer to my previous article on How the retrieve the private key file PEM after Cloudformation or CDK stack creation[↗].
An Ec2 Instance named Webserver-Instance.
An IAM Role for the Ec2 Instance named webserver-role, which allows the spring Boot application hosted on the Ec2 Instance to establish connections with the Amazon SQS Queue (already created) and perform actions: Delete Message, Receive Message, Send Message, Get Queue Attributes and Get Queue Url.

Create the stack

// MyStack.java
public class MyStack extends Stack {
  public MyStack(final Construct scope, final String id, final StackProps props) {
    super(scope, id, props);
    IVpc vpc = new NetworkContruct(this, "NetworkResource", props).getVpc();
    IQueue queue = new QueueConstruct(this, "QueueResource", vpc, props).getQueue();
    IInstance webserver =
        new ComputerConstruct(
                this, "ComputerResource", new ComputerProps(vpc, queue.getQueueArn()), props)
            .getComputer();
  }
}

// Day17App.java
public class Day017App {
  public static void main(final String[] args) {
    App app = new App();
    new MyStack(app,"Day017Stack",
        StackProps.builder()
                .env(
                    Environment.builder()
                        .account(System.getenv("CDK_DEFAULT_ACCOUNT"))
                        .region(System.getenv("CDK_DEFAULT_REGION"))
                        .build())
                .build());
    app.synth();
  }
}

Create SpringBoot Consumer Application

To keep things simple and avoid complicating my life, I will use Spring Cloud AWS Docs[↗]

Spring Cloud AWS simplifies using AWS managed services in a Spring Framework and Spring Boot applications. It offers a convenient way to interact with AWS provided services using well-known Spring idioms and APIs.

To configure the SQS service in my project, I will add the following beans in the configuration class:

@Configuration
public class ApplicationConfiguration {
    @Bean
    public AwsRegionProvider customRegionProvider() {
        return new InstanceProfileRegionProvider();
    }
    @Bean
    public AwsCredentialsProvider customInstanceCredProvider() {
        return  InstanceProfileCredentialsProvider.builder()
                .build();
    }
}

And the listener that captures all new message and prints their content.

@Slf4j
@Component
public class ExampleSQSConsumer {
    @SqsListener(queueNames = { "my-queue" }) // 👈👈
    public void listen(String payload) {
        log.info("*******************  SQS Payload ***************");
        log.info("Message Content: {}", payload);
        log.info("Received At: {}", Date.from(Instant.now()));
        log.info("************************************************");
    }
}

You can find the full project in my GitHub repo[↗]

Deployment

⚠️⚠️ Before run the deployment commands ensure that you have java installed on your host machine. I used Java 21 under MacOs to build this insfrastructure.

Open the terminal anywhere and run the following commands:

git clone https://github.com/nivekalara237/100DaysTerraformAWSDevops.git
cd 100DaysTerraformAWSDevops/day_017
cdk bootstrap --profile cdk-user
cdk deploy --profile cdk-user Day017Stack

Resut

Your can find the full source code on GitHub Repo↗

Deploying SonarQube on AWS EC2 with Cloud Development Kit(CDK): Extending the Database to Amazon RDS PostgreSQL

Kevin Lactio Kemta — Mon, 23 Sep 2024 13:17:54 +0000

Day 016 - 100DaysAWSIaCDevopsChallenge

In my previous post, Step-by-Step Guide to Setting Up and Deploying SonarQube on an AWS EC2 Instance Using AWS Cloud Development Kit (CDK)[↗], I demonstrated how to deploy a SonarQube Server on an EC2 instance. As the database were self-hosted in EC2 Instance, today, I will show you how to extend the database by integrating it with an external service like Amazon RDS or Aurora, rather than keeping it self-hosted within the server.

Why did I need to extend the SonarQube Database to a RDS PostgreSQL Instance?

Extending SonarQube's database to an external service like Amazon RDS (Relational Database Service) or Aurora (Amazon Severless Database managed) instead of keeping it inside the self-hosted SonarQube sever offers several advantages:

Scalability: Managed database like RDS and Aurora can handle inscreased workloads, provide more robust storage and offer auto-scaling feature by ensuring best permformance as your SonarQube instance grows.
Reliability: Provide high availability through automatic backups, replication and failover mechanisms. They reduce the risk of data loss and downtime in case of hardware failure.
Security: Offer built-in security features, including encryption at rest and in transit, network isolation through VPC and private subnet making your data more secure than self-managed (self-hosted) databases.
Maintainance: AWS handles regular maintenance tasks, such as backups, patching, and updates, reducing the burden on your infrastructure team.

Update the previous infrastructure to setting up a RDS PostgreSQL Instance for SonarQube

The infrastructure consists of the following components:

VPC and two (02) Subnets(private and public subnet) to manage network traffic within the server and database instance.
Internet Gatewayto allow internet communication between our secured cloud environment and external networks.
Security group to control access to the server and database based on IP addresses and ports connections.
EC2 Instance to host the SonarQube server (including the Web Server, ElasticSearch, and Postgres Database).
Key Pair the private key that allows external hosts to connect securely to the instance.
Secret Manager to securely store the database user password.
IAM Role to allow EC2 Instance to connect to Secret Manager and retrieve database password
Elastic IP (EIP) - The static IP to attach to the EC2 Instance. It allows to maintain a fixed IP address aven if the instance is stopped of restarted.
Nat Gateway to allow instance (here database instance) in a private subnet to connect to the internet or other AWS Service without exposing then to internet traffic.
RDS PostgreSQL the database instance to host the sonarqube data.

Create the database Construct

interface DatabaseProps extends CustomProps {
  vpc: IVpc,
  sg?: ISecurityGroup // the security group for SonarQube Server Instance
}
export class Database extends Construct {
  private readonly _dbInstanceArn: string
  private readonly _passwordSecretArn: string
  private readonly _instanceEndpointUrl: string
  constructor(scope: Construct, id: string, props: DatabaseProps) {
    super(scope, id)
    const secret: ISecret = new aws_secretsmanager.Secret(this, 'SecretResource', {
      secretName: 'database-instance-secret',
      generateSecretString: {
        secretStringTemplate: JSON.stringify({ username: props.databaseUsername }),
        generateStringKey: 'password',
        excludeCharacters: '/@"`()[]\'',
        includeSpace: false
      }
    })
    const securityGroup = new SecurityGroup(this, 'DatabaseSecurityGroupeResource', {
      securityGroupName: 'sq-database-sg',
      disableInlineRules: false,
      allowAllOutbound: true,
      vpc: props.vpc,
      description: 'Database instance security group'
    })
    securityGroup.addIngressRule(
      Peer.securityGroupId(props.sg!.securityGroupId),
      Port.tcp(props.databasePort ?? 5432), 'Allow EC2 to connect to the database instance')
    const db = new rds.DatabaseInstance(this, 'DatabaseInstanceResource', {
      engine: rds.DatabaseInstanceEngine.postgres({
        version: rds.PostgresEngineVersion.VER_12_16
      }),
      vpc: props.vpc,
      networkType: rds.NetworkType.IPV4,
      vpcSubnets: props.vpc.selectSubnets({ subnetType: SubnetType.PRIVATE_WITH_EGRESS }),
      instanceType: InstanceType.of(InstanceClass.BURSTABLE3, InstanceSize.MICRO),
      publiclyAccessible: false,
      storageType: StorageType.GP3,
      credentials: rds.Credentials.fromSecret(secret, props.databaseUsername),
      allocatedStorage: 20,
      databaseName: props.databaseName,
      instanceIdentifier: 'sonarqube-db-1',
      port: props.databasePort ?? 5432,
      securityGroups: [
        securityGroup
      ]
    })
    this._dbInstanceArn = db.instanceArn
    this._passwordSecretArn = secret.secretArn
    this._instanceEndpointUrl = db.instanceEndpoint.socketAddress
  }
  get dbInstanceArn(): string {
    return this._dbInstanceArn
  }
  get passwordSecretArn() {
    return this._passwordSecretArn
  }
  get instanceEndpointUrl(): string {
    return this._instanceEndpointUrl
  }
}

The important thing to know here is the snippet code:

securityGroup.addIngressRule(
    Peer.securityGroupId(props.sg!.securityGroupId),
    Port.tcp(props.databasePort ?? 5432), 'Allow EC2 to connect to the database instance')

This code allows communication between the database and the EC2 instance server through the specified port.

Update the Stack

This is the update for the stack created in the previous article [↗].

export class SonarServerInfrastructureStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props: CustomProps) {
    super(scope, id, props)
    ....
    const vpc = new ec2.Vpc(this, 'PublicVpcResource', {
      ...
      maxAzs: 2,
      subnetConfiguration: [{
        subnetType: ec2.SubnetType.PUBLIC,
        name: 'sq-public-subnet-webserver',
        mapPublicIpOnLaunch: true
      }, {
        subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS,
        name: 'sq-private-subnet-database'
      }]
    })

    sonarQubeSG.addEgressRule(ec2.Peer.anyIpv4(), ec2.Port.tcp(props.databasePort ?? 5432), 'Allow outbound traffic to database instance')

    // set up the database instance
    const db = new Database(this, 'DatabaseResource', {
      ...props,
      vpc: vpc,
      sg: sonarQubeSG
    })

    const sonarQubeServer = new ec2.Instance(this, 'SonarQubeServerInstanceResource', {
      ...
      role: new iam.Role(this, 'Ec2RoleResource', {
        roleName: 'ec2-role-and-permission',
        assumedBy: new iam.ServicePrincipal('ec2.amazonaws.com'),
        path: '/',
        inlinePolicies: {
          secretsManager: new iam.PolicyDocument({
            assignSids: true,
            statements: [
              new iam.PolicyStatement({
                effect: iam.Effect.ALLOW,
                actions: [
                  'secretsManager:GetSecretValue',
                  'secretsManager:DescribeSecret'
                ],
                resources: [db.passwordSecretArn],
                conditions: {}
              })
            ]
          })
        }
      })
    })

    const userDataEncoded = ec2.UserData.forLinux()
    userDataEncoded.addCommands(fs.readFileSync('./assets/sonarqube-server.sh', 'utf-8'))
    sonarQubeServer.addUserData(userDataEncoded.render()
      .replace('{{DB_NAME}}', props.databaseName || 'sonarqube')
      .replace('{{DB_USER}}', props.databaseUsername || 'sonar')
      .replace('{{SECRET_ARN}}', db.passwordSecretArn)
      .replace('{{DB_SOCKET_ADDRESS}}', db.instanceEndpointUrl))
  }
}

PRIVATE_WITH_EGRESS - In the vpc configuration, we had added a new entry of subnet of type PRIVATE_WITH_EGRESS, meaning that it can access the internet but is not exposed to inbound internet traffic.
Egress rule - Allows the SonarQube server to send traffic to the database instance on the specified port (default 5432 for PostgreSQL)

New User data scripts As we no longer need the self-hosted database, we have to replace the database connection properties with the new ones:

## Update package index
sudo apt update

## Install OpenJDK 17
sudo apt install -y openjdk-17-jdk zip unzip
## Set up the JAVA_HOME environment variable
echo "export JAVA_HOME=$(dirname $(dirname $(readlink -f $(which java))))" | sudo tee /etc/profile.d/jdk.sh
source /etc/profile.d/jdk.sh

## download sonarqube server - community version
wget https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-9.9.6.92038.zip

## unzip content
unzip sonarqube-*.zip
sudo mv sonarqube-9.9.6.92038 sonarqube
sudo mv sonarqube /opt/

## Create sonarqube user and its group
sudo groupadd sonarqube
sudo useradd -d /opt/sonarqube -g sonarqube sonarqube
sudo chown -R sonarqube:sonarqube /opt/sonarqube

## Insert user runner after `APP_NAME="SonarQube"` line
sudo sed -i 's/APP_NAME="SonarQube"/APP_NAME="SonarQube"\n\nRUN_AS_USER=sonarqube/' /opt/sonarqube/bin/linux-x86-64/sonar.sh

## create sonarqube launcher service
sudo tee /etc/systemd/system/sonarqube.service <<EOF
[Unit]
Description=SonarQube service
After=syslog.target network.target
[Service]
Type=forking
User=sonarqube
Group=sonarqube
PermissionsStartOnly=true
ExecStart=/opt/sonarqube/bin/linux-x86-64/sonar.sh start
ExecStop=/opt/sonarqube/bin/linux-x86-64/sonar.sh stop
StandardOutput=journal
LimitNOFILE=131072
LimitNPROC=8192
TimeoutStartSec=5
Restart=always
SuccessExitStatus=143

[Install]
WantedBy=multi-user.target
EOF

## Installing AWS-CLI
sudo snap install aws-cli --classic

sudo tee /opt/sonarqube/conf/sonar.properties <<EOF
sonar.web.port=9000
sonar.jdbc.username={{DB_USER}}
sonar.jdbc.password=$(aws secretsmanager get-secret-value --secret-id {{SECRET_ARN}} | jq --raw-output '.SecretString' | jq -r .password)
sonar.jdbc.url=jdbc:postgresql://{{DB_SOCKET_ADDRESS}}/{{DB_NAME}}
EOF

## Start SonarQube Server
sudo systemctl enable sonarqube.service
sudo systemctl start sonarqube.service

Deploy the stack

const app = new cdk.App()
new SonarServerInfrastructureStack(app, 'Day016Stack', {
  cidr: '10.0.0.1/16',
  databaseName: 'sonarqubedb',
  databaseUsername: 'sonar',
  databasePort: 5000,
  env: {
    region: process.env.CDK_DEFAULT_REGION,
    account: process.env.CDK_DEFAULT_ACCOUNT
  }
})
app.synth()

Open the terminal anywhere and run the following commande:

git clone https://github.com/nivekalara237/100DaysTerraformAWSDevops.git
cd 100DaysTerraformAWSDevops/day_016
cdk deploy --profile cdk-user Day016Stack

After the deployment, open the browser and type this url http://PUBLIC_IP_or_DNS:9000

🥳✨
We have reached the end of the article.
Thank you so much 🙂

Your can find the full source code on GitHub Repo↗

Step-by-Step Guide to Setting Up and Deploying SonarQube on an AWS EC2 Instance Using AWS Cloud Development Kit (CDK)

Kevin Lactio Kemta — Sat, 21 Sep 2024 13:17:02 +0000

Day 015 - 100DaysAWSIaCDevopsChallenge

Today in my serie of 100 days of code challenge, I am going to show you step by step how to deploy SonarQube Server into an AWS EC2 Instance using AWS Cloud Development Kit (CDK).

What is SonarQube ?

SonarQube is an open-source application that automates the process of code quality analysis. It scans your codebase to detect issues such as bugs, security vulnerabilities, code smells (maintainability issues), and technical debt. It provides comprehensive reports and metrics on your code quality, helping teams identify and fix problems early in the development process.

Why do you need to use it in your application ?

SonarQube helps developers write clean, secure, and maintainable code by offering real-time insights and automated feedback during the software development lifecycle. Using SonarQube in your application development provides several critical benefits:

Code Quality Checks: Detects issues related to code quality, including bugs and inefficient code practices.
Security Vulnerabilities: Identifies potential security risks such as SQL injections and cross-site scripting (XSS).
Reduces Technical Debt: Measures code complexity, duplication, and other factors that affect the maintainability of your project.
Supports Continuous Integration: Can integrate with CI/CD pipelines (e.g., Jenkins, GitLab, Github-Actions) for automatic code scanning during development.

how to install SonarQube on an AWS EC2 Instance?

To achieve this, we will use AWS CDK to create the infrastructure required. The infrastructure consists of the following components:

VPC and a Public Subnetto manage network traffic within the server.
Internet Gatewayto allow internet communication between our secured cloud environment and external networks.
Security group to control access to the server based on IP addresses and ports connections.
EC2 Instance to host the SonarQube server (including the Web Server, ElasticSearch, and Postgres Database).
Key Pair the private key that allows external hosts to connect securely to the instance.

Create the CDK Stack

VPC, Subnet and Internet Gateway

interface CustomProps extends StackProps {
  cidr: string
}

export class SonarServerInfrastructureStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props: CustomProps) {
    super(scope, id, props)
    const vpc = new ec2.Vpc(this, 'PublicVpcResource', {
      vpcName: 'sonarqube-server-vpc',
      enableDnsHostnames: true,
      enableDnsSupport: true,
      ipAddresses: ec2.IpAddresses.cidr(props.cidr),
      createInternetGateway: true,
      subnetConfiguration: [{
        subnetType: ec2.SubnetType.PUBLIC,
        name: 'sq-public-subnet',
        mapPublicIpOnLaunch: true
      }]
    })
  }
}

enableDnsHostnames/enableDnsSupport - These two options ensure that instances launched in the VPC will have DNS resolution capabilities.
ipAddresses - The VPC's CIDR block. It is provided as props.cidr, making it configurable.
subnetConfiguration - Configures a public subnet. The configuration creates a public subnet ('sq-public-subnet'), with the option to map public IP addresses to instances on launch
createInternetGateway - Automatically creates an Internet Gateway for external communication

Security group

const sonarQubeSG = new ec2.SecurityGroup(this, 'SonarQubeSecuGroupResource', {
    securityGroupName: 'SonarSecurityGroup',
    allowAllIpv6Outbound: false,
    allowAllOutbound: true,
    description: 'Inbound and outbound traffic to sonarqube server',
    disableInlineRules: false,
    vpc: vpc,
})
sonarQubeSG.addIngressRule(ec2.Peer.anyIpv4(), ec2.Port.SSH, 'Allow ssh traffic')
sonarQubeSG.addIngressRule(ec2.Peer.anyIpv4(), ec2.Port.HTTPS, 'Allow https traffic')
sonarQubeSG.addIngressRule(ec2.Peer.anyIpv4(), ec2.Port.tcp(9000), 'Allow custom port traffic')

To allow any IP address (i.e., 0.0.0.0/0) to connect to the SonarQube server on port 9000, we have defined an ingress rule in the Security Group that is attached to your EC2 instance. This allows traffic to the SonarQube application, which runs on port 9000.

SonarQube Web Server (EC2 Instance) + KeyPair

const keypair = new ec2.KeyPair(this, 'KeyPairResource', {
    keyPairName: 'sonarqube-server-keypair',
    type: ec2.KeyPairType.RSA,
    format: ec2.KeyPairFormat.PEM
})
const sonarQubeServer = new ec2.Instance(this, 'SonarQubeServerInstanceResource', {
    instanceName: 'Sonarqube-server',
    vpc,
    securityGroup: sonarQubeSG,
    instanceType: ec2.InstanceType.of(
    ec2.InstanceClass.T2, ec2.InstanceSize.MEDIUM),
    keyPair: keypair,
    associatePublicIpAddress: true,
    userDataCausesReplacement: true,
    machineImage: ec2.MachineImage.lookup({
        name: '*ubuntu*',
        filters: {
            'image-id': ['ami-0e86e20dae9224db8'],
            'architecture': ['x86_64']
        },
        windows: false
    }),
    vpcSubnets: vpc.selectSubnets({ subnetType: ec2.SubnetType.PUBLIC }),
    blockDevices: [
        {
            deviceName: '/dev/sda1',
            mappingEnabled: true,// to override the default Volume mounted automatically on creation
            volume: ec2.BlockDeviceVolume.ebs(30, {
            volumeType: ec2.EbsDeviceVolumeType.GP3,
            deleteOnTermination: true
            })
        }
    ]
})

const userDataEncoded = ec2.UserData.forLinux()
userDataEncoded.addCommands(fs.readFileSync('./assets/sonarqube-server.sh', 'utf-8'))
sonarQubeServer.addUserData(userDataEncoded.render())

// display the ARN of private key in AWS Parameter Store
new CfnOutput(this, 'SonarqubeKeyPairParamArn', {
    value: keypair.privateKey.parameterArn
})
// Optionally store the private key locally
fs.writeFileSync('./assets/private_key.pem', keypair.privateKey.stringValue, {
    flag: 'w', encoding: 'utf-8'
})

Key Pair Creation - We create an RSA key pair that will be used to securely connect to the EC2 instance. The private key is stored either in AWS Parameter Store, retrieved using keypair.privateKey.parameterArn or locally as a PEM file private_key.pem.
EC2 Instance Configuration - we create a new EC2 instance named Sonarqube-server. The instance is provisioned in the public subnet of the VPC and is attached to the security group that allows access to port 9000. The AMI is filtered using its ID and architecture (ami-0e86e20dae9224db8), and the instance type is t2.medium. The instance has a block device configured with a 30Go GP3 EBS volume that will be deleted upon termination.

User Data Configuration - The instance is initialized with a user data script from assets/sonarqube-server.sh, which is executed upon instance launch to configure SonarQube on the server. The bolow code is the content of user data bootstrap source:

# Update package index
sudo apt update

# Install OpenJDK 17, zip and unzip
sudo apt install -y openjdk-17-jdk zip unzip

# Set up the JAVA_HOME environment variable
echo "export JAVA_HOME=$(dirname $(dirname $(readlink -f $(which java))))" | sudo tee /etc/profile.d/jdk.sh
source /etc/profile.d/jdk.sh

# Installing Postgres
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ `lsb_release -cs`-pgdg main" /etc/apt/sources.list.d/pgdg.list'
wget -q https://www.postgresql.org/media/keys/ACCC4CF8.asc -O - | sudo apt-key add -
sudo apt install postgresql postgresql-contrib -y

# Initiate the database server to begin operations
sudo systemctl enable postgresql
sudo systemctl start postgresql

# Create database and its owner
sudo -u postgres psql -c "CREATE DATABASE sonarqubedb;"
sudo -u postgres psql -c "CREATE USER sonar WITH ENCRYPTED PASSWORD 'sonar';"
sudo -u postgres psql -c "GRANT ALL ON SCHEMA public TO sonar;"
sudo -u postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE sonarqubedb TO sonar;"
sudo -u postgres psql -c "ALTER DATABASE sonarqubedb OWNER TO sonar;"

# download sonarqube server - community version
# shellcheck disable=SC2317
wget https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-9.9.6.92038.zip

# unzip content
# shellcheck disable=SC2317
unzip sonarqube-*.zip
sudo mv sonarqube-9.9.6.92038 sonarqube
sudo mv sonarqube /opt/

# Create sonarqube user and its group
sudo groupadd sonarqube
sudo useradd -d /opt/sonarqube -g sonarqube sonarqube
sudo chown -R sonarqube:sonarqube /opt/sonarqube

# Insert user runner after `APP_NAME="SonarQube"` line
sudo sed -i 's/APP_NAME="SonarQube"/APP_NAME="SonarQube"\n\nRUN_AS_USER=sonarqube/' /opt/sonarqube/bin/linux-x86-64/sonar.sh

# create sonarqube launcher service
sudo tee /etc/systemd/system/sonarqube.service <<EOF
[Unit]
Description=SonarQube service
After=syslog.target network.target
[Service]
Type=forking
User=sonarqube
Group=sonarqube
PermissionsStartOnly=true
ExecStart=/opt/sonarqube/bin/linux-x86-64/sonar.sh start
ExecStop=/opt/sonarqube/bin/linux-x86-64/sonar.sh stop
StandardOutput=journal
LimitNOFILE=131072
LimitNPROC=8192
TimeoutStartSec=5
Restart=always
SuccessExitStatus=143

[Install]
WantedBy=multi-user.target
EOF

# Configuring the database
sudo tee /opt/sonarqube/conf/sonar.properties <<EOF
sonar.web.port=9000
sonar.jdbc.username=sonar
sonar.jdbc.password=sonar
sonar.jdbc.url=jdbc:postgresql://localhost:5432/sonarqubedb
EOF

# Start SonarQube Server
sudo systemctl enable sonarqube.service
sudo systemctl start sonarqube.service

For better security, it is recommended not to hard-code sensitive data such as database usernames and passwords. Instead, use services like AWS Systems Manager Parameter Store, AWS Secrets Manager, or alternatives such as HashiCorp Vault to securely manage and access sensitive information.

Deploy the stack

const app = new cdk.App()
new SonarServerInfrastructureStack(app, 'Day015Stack', {
  cidr: '10.0.0.1/16',
  env: {
    region: process.env.CDK_DEFAULT_REGION,
    account: process.env.CDK_DEFAULT_ACCOUNT
  }
})
app.synth()

Open the terminal anywhere and run the following commande:

git clone https://github.com/nivekalara237/100DaysTerraformAWSDevops.git
cd 100DaysTerraformAWSDevops/day_015
cdk deploy --profile cdk-user Day015Stack

After the deployment, open the browser and type this url http://PUBLIC_IP_or_DNS:9000

🥳✨
We have reached the end of the article.
Thank you so much 🙂

Your can find the full source code on GitHub Repo↗

Create Test Coverage Visualizer and Deploy to AWS S3 with Cloud development Kit (CDK) and GitHub Actions

Kevin Lactio Kemta — Mon, 02 Sep 2024 13:40:11 +0000

Day 014 - 100DaysAWSIaCDevopsChallenge

In my previous article, Configuring CI/CD for NPM Library Publishing with GitHub Actions, I explored automating various tasks with GitHub Actions. In this follow-up, I'll focus on how to integrate the automation of test coverage uploads to an S3 bucket.

Why Centralize Test Coverage Reporting?

In the software development, maintaining high code quality is crucial for delivering reliable and efficient products. One of the key aspects of ensuring code quality is monitoring code coverage, the measure of how much of your code is tested. However, as projects grow in complexity, keeping track of coverage metrics across multiple components and teams can become challenging. This is why centralizing your test coverage becomes more valuable by bringing all your coverage data into a single, unified platform. You can streamline analysis, enhance collaboration, and ensure that your entire codebase meets the necessary standards. In this article, I will show you how I centralized the tests coverage of my NPM library.

The process will involve the following steps:

Create and AWS infrastructure for hosting the coverage visualizer - Set up the necessary AWS resources to host and securely serve coverage visualizer on the internet. This involves using AWS S3 for storage and AWS CloudFront as a Content Delivery Network (CDN) to improve availability and security.
Create user with less privilege to update files to the bucket - Enhance security by creating an AWS IAM user with limited permissions to upload files to the S3 bucket for security reason. This user will be used by your GitHub Actions workflow to push the coverage reports to S3.
Set Up the GitHub Actions job to upload test coverage file to S3 Bucket - Automate the process of uploading test coverage reports to the S3 bucket after every successful release using GitHub Actions. This ensures that the latest coverage data is always available on the visualizer. Also give the ability to run this job manually.

Flow diagram

1. Create and AWS infrastructure for hosting the coverage visualizer

In a previous segment of the 100 Days of Code Challenge series, I demonstrated how to create and host an Angular App in an S3 Bucket as a static website. In this section, I will use the same AWS resources to host the coverage visualizer. This will involve the following:

S3 Bucket We'll use S3 to store the visualizer files.
AWS CloudFront for Content Distribution - CloudFront will distribute the content globally, just like an S3 static website. We will secure it with HTTPS by using SSL at this layer. Additionally, we'll configure S3 to accept traffic only from CloudFront and enforce that all traffic must be secure.
Route 53 for Subdomain Configuration We will configure a subdomain for your visualizer using Route 53.
IAM User We'll create an IAM user that GitHub Actions will use to upload files to the S3 Bucket. This IAM user will have limited permissions, specifically only to PutObject inside the bucket, ensuring a minimal access for better security need.

1.a S3 Bucket

interface S3WbesiteProps extends StackProps {
  origins?: string[]
}
export class S3Website extends Construct {
  private readonly _bucket: s3.IBucket
  private readonly _baseSegment: string
  constructor(scope: Construct, id: string, private props: S3WbesiteProps) {
    super(scope, id)
    const bucketName = 'coverage-visualizer-sws-o8stnnkqmos1v'
    const defSeg = 'co2visualizer'
    const wsBucket = new s3.Bucket(this, 'WebsiteBucket', {
      bucketName: bucketName,
      versioned: false,
      blockPublicAccess: {
        blockPublicPolicy: true,
        blockPublicAcls: false,
        ignorePublicAcls: false,
        restrictPublicBuckets: true
      },
      objectLockEnabled: true,
      objectOwnership: s3.ObjectOwnership.BUCKET_OWNER_PREFERRED,
      websiteIndexDocument: `index.html`,
      websiteErrorDocument: `error.html`,
      autoDeleteObjects: true,
      removalPolicy: RemovalPolicy.DESTROY
    })
    wsBucket.addCorsRule({
      allowedMethods: [
        s3.HttpMethods.HEAD,
        s3.HttpMethods.GET
      ],
      allowedOrigins: props.origins || ['*'],
      maxAge: Duration.minutes(5).toSeconds()
    })
    wsBucket.addLifecycleRule({
      enabled: true,
      expiration: Duration.days(31),
      transitions: [{
        storageClass: s3.StorageClass.ONE_ZONE_INFREQUENT_ACCESS,
        transitionAfter: Duration.days(60)
      }]
    })
    this._bucket = wsBucket
    this._baseSegment = defSeg
  }
  get bucket() {
    return this._bucket
  }
  get baseSegment() {
    return this._baseSegment
  }
}

versioned - is set to false to disable versioning of objects in the bucket. This configuration ensures that only the latest version of each file is retained, which is ideal for scenarios where you only need the most recent file.
blockPublicAccess.blockPublicPolicy - to enforcing a critical security measure that prevents the bucket from being publicly accessible via a bucket-wide public policy.
objectLockEnabled - Enables object lock for data protection
objectOwnership- is setted to BUCKET_OWNER_PREFERRED, to ensure that the bucket owner retains control over all objects, making access management easier, more secure, and consistent. This setting is particularly useful in environments where multiple users or automated processes are uploading objects to the bucket.
autoDeleteObjects - automatically deletes all objects when the bucket is destroyed.

Lifecycle Rules - Manages the storage of objects in the bucket

wsBucket.addLifecycleRule({
    enabled: true,
    expiration: Duration.days(90),
    transitions: [{
        storageClass: s3.StorageClass.INFREQUENT_ACCESS,
        transitionAfter: Duration.days(30)
    }]
})

expiration - deletes objects after 90 dayls
transitionAfter - Moves Obejects to Infrequent Access storage after 30 days.

This CDK Construct automates the setup of an S3 bucket configured as a static website, with additional features like CORS, object lifecycle management, and security controls. It abstracts the complexity of configuring these AWS services, providing a reusable and secure way to host static content.

1.b. CloudFront & Route 53 Subdomain configuration

export interface WebsiteDistributionProps extends StackProps {
  websiteBucketName: string;
  mainDomain: string,
  acm: {
    certificateArn: string
  },
  distribution: {
    subDomain: string
  }
}
export class WebsiteDistribution extends Construct {
  private readonly _distributionUrl: string
  private readonly _distributionArn: string
  private readonly _viewersUrl: string
  constructor(scope: Construct, id: string, props: WebsiteDistributionProps) {
    super(scope, id)
    const bucket = s3.Bucket.fromBucketName(this, `BucketName_${id}`, props.websiteBucketName!)
    const certificate = acm.Certificate.fromCertificateArn(this, `Certificate_${id}`, props.acm?.certificateArn!)
    const cachePolicy = new cf.CachePolicy(this, `Cache-${id}-Policy`, {
      cachePolicyName: `Cache-${id}-Policy`,
      enableAcceptEncodingGzip: true,
      enableAcceptEncodingBrotli: true,
      queryStringBehavior: cf.CacheQueryStringBehavior.all(),
      cookieBehavior: cf.CacheCookieBehavior.none(),
      defaultTtl: Duration.seconds(30),
      headerBehavior: cf.CacheHeaderBehavior.allowList(
        'Origin',
        'Accept',
        'Access-Control-Request-Method',
        'Access-Control-Request-Headers'
      )
    })
    const originRequestPolicy = new cf.OriginRequestPolicy(this, `Origin-Request-${id}-Policy`, {
      originRequestPolicyName: `Origin-Request-${id}-Policy`,
      queryStringBehavior: cf.CacheQueryStringBehavior.all(),
      cookieBehavior: cf.CacheCookieBehavior.none(),
      headerBehavior: cf.CacheHeaderBehavior.allowList(
        'Origin',
        'Accept',
        'Access-Control-Request-Method',
        'Access-Control-Request-Headers'
      )
    })
    const distribution = new cf.Distribution(this, `SSLCertificate_${id}`, {
      enabled: true,
      defaultBehavior: {
        origin: new origins.HttpOrigin(bucket.bucketWebsiteDomainName, {
          protocolPolicy: cf.OriginProtocolPolicy.HTTP_ONLY,
          httpPort: 80,
          httpsPort: 443,
          connectionTimeout: Duration.seconds(10)
        }),
        allowedMethods: cf.AllowedMethods.ALLOW_ALL,
        cachedMethods: cf.CachedMethods.CACHE_GET_HEAD_OPTIONS,
        cachePolicy,
        originRequestPolicy
      },
      certificate,
      httpVersion: cf.HttpVersion.HTTP2,
      domainNames: [props.distribution?.subDomain].filter(value => !!value)
    })
    const hostedZone = route53.HostedZone.fromLookup(this, `HostedZone_${id}`, {
      domainName: props.mainDomain!
    })
    const cnameRecord = new route53.CnameRecord(this, `DomainCNAME_${id}`, {
      recordName: props.distribution?.subDomain + '.',
      domainName: distribution.distributionDomainName,
      zone: hostedZone,
      deleteExisting: true,
      ttl: Duration.minutes(10),
      comment: `RecordSet to send traffic from ${props.distribution?.subDomain} to ${distribution.distributionDomainName}`
    })
    this._distributionUrl = distribution.distributionDomainName
    this._viewersUrl = cnameRecord.domainName
    this._distributionArn = `arn:aws:cloudfront::${props.env?.account}:distribution/${distribution.distributionId}`
  }
  get distributionArn(): string {
    return this._distributionArn
  }
  get distributionUrl(): string {
    return this._distributionUrl
  }
  get viewsUrl(): string {
    return this._viewersUrl
  }
}

The WebsiteDistribution construct efficiently sets up a secure, high-performance website delivery system using CloudFront, S3, ACM, and Route 53. It automates the infrastructure needed to serve static websites with custom domains and SSL/TLS encryption.

1.c. Visualizer Stack

interface CustomProps extends cdk.StackProps {
  domain: string
  certificateArn: string
}
export class CoverageVisualizerStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props: CustomProps) {
    super(scope, id, props)
    const bucket = new S3Website(this, 'Coverage-Visualizer-Bucket', {
      origins: [`https://visualizer.${props.domain}`]
    })
    const distribution = new WebsiteDistribution(this, 'CloudFront', {
      mainDomain: props.domain,
      websiteBucketName: bucket.bucket.bucketName,
      distribution: {
        subDomain: `visualizer.${props.domain}`
      },
      acm: {
        certificateArn: props.certificateArn
      }
    })
    const cloudFrontPolicy = new aws_iam.Policy(this, 'S3PolicyForCloudFront', {
      roles: [
        new aws_iam.Role(this, 'S3PolicyRoleForCloudFront', {
          assumedBy: new aws_iam.ServicePrincipal('s3.amazonaws.com'),
          path: '/',
          inlinePolicies: {
            s3: new aws_iam.PolicyDocument({
              assignSids: true,
              statements: [
                new aws_iam.PolicyStatement({
                  effect: Effect.ALLOW,
                  actions: ['s3:GetObject'],
                  resources: [bucket.bucket.bucketArn + '/*'],
                  conditions: {
                    Bool: { 'AWS:SecureTransport': true },
                    StringEquals: { 'AWS:SourceArn': distribution.distributionArn }
                  }
                })
              ]
            })
          }
        })
      ]
    })
    const user = new aws_iam.User(this, 'S3Uploader', {
      userName: 'S3-Uploader',
      passwordResetRequired: false
    })
    user.addToPolicy(new aws_iam.PolicyStatement({
      actions: ['s3:PutObject','s3:PutObjectAcl'],
      effect: Effect.ALLOW,
      resources: [bucket.bucket.bucketArn + '/' + bucket.baseSegment + '/*']
    }))
  }
}

CloudFront Distribution
- mainDomain - The primary domain (e.g., example.com).
- websiteBucketName - The name of the S3 bucket where the visualizer is hosted.
- distribution.subDomain - The subdomain (e.g., visualizer.example.com) for the CloudFront distribution.
- acm.certificateArn - The ARN of the SSL certificate used to secure the CloudFront distribution.
IAM Policy for CloudFront to Access S3 - A policy is created that allows CloudFront to access the S3 bucket
- AWS:SecureTransport- Ensures that only secure requests (HTTPS) are allowed
- AWS:SourceArn - Restricts access to requests coming from the specified CloudFront distribution
IAM User for S3 Uploads An IAM user is created specifically to upload files to the S3 bucket
- Policy Attached - The user is granted permissions to perform the s3:PutObject action, limited to a specific path in the S3 bucket (defined by bucket.baseSegment).

This stack sets up an infrastructure to securely host a coverage visualizer on an S3 bucket and distribute it via CloudFront. It includes the following:

S3 Bucket: For hosting static files.

CloudFront Distribution: For secure, high-performance delivery of content.

IAM Policies: For secure access control, ensuring that only authorized entities (like CloudFront and a specific IAM user) can interact with the S3 bucket.

This setup ensures that the visualizer is securely accessible over HTTPS, with controlled access to the underlying S3 bucket.

1.e. Deploy infrastructure

git clone https://github.com/nivekalara237/100DaysTerraformAWSDevops.git

cd 100DaysTerraformAWSDevops/day_014
export DOMAIN="yourdomain.com"
export CERT_ARN="arn:aws:acm:us-east-1:xxxx:certificate/xxxx-xxxx-xxxx-xxxxxxxx"

cdk deploy --profile cdk-user --all

2. Set Up the GitHub Actions job to upload test coverage file to S3 Bucket

To automate the process of uploading test coverage files to an S3 bucket using GitHub Actions, you'll need to configure the workflow with the necessary AWS credentials. First, log in to the AWS Management Console and navigate to the IAM service. Then, generate the access keys for the S3-Uploader user that was previously created after the infrastructure was deployed.

name: "Deploy Test Cov to S3 Visualizer"
on:
  workflow_run:
    types:
      - completed
    workflows:
      - Release
  workflow_dispatch:
    inputs:
      environment:
        default: ENV
        type: environment
        description: 'Pick Environment'

jobs:
  deploy-to-s3-bucket:
    name: 'Upload Test Result'
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 18.x
      - name: Run test
        run: |
          npm ci && npm test
      - name: Upload Files to S3
        uses: shallwefootball/s3-upload-action@master
        id: S3
        with:
          aws_bucket: ${{ secrets.BUCKET_NAME }}
          aws_key_id: ${{ secrets.AWS_ACCESS_KEY }}
          aws_secret_access_key: ${{ secrets.AWS_SECRET_KEY }}
          source_dir: 'coverage'
          destination_dir: 'co2visualizer/${{ github.ref_name }}'
      - name: Display URL
        run: |
          echo "${{ env.VISUALIZER_URL }}/${{ steps.S3.outputs.object_key }}/index.html"

If you want to know how to configure GitHub Actions Workflow please follow my previous article here

on:
  workflow_run:
    types:
      - completed
    workflows:
      - Release

Specify that the workflow is triggered only when a release workflow is created, as defined in release.yml.

on: 
  workflow_dispatch:
    inputs:
      environment:
        default: ENV
        type: environment
        description: 'Pick Environment'

Specify that the workflow can also be triggered manually.

And the Test Coverage Visualizer

Happy Coding!!

Your can find the full source code on GitHub Repo↗

Effortless Automation: Configuring CI/CD for NPM Library Publishing with GitHub Actions

Kevin Lactio Kemta — Sun, 01 Sep 2024 18:23:50 +0000

Day 013 - 100DaysAWSIaCDevopsChallenge

Recently, I developed an npm package for JavaScript and TypeScript projects. Initially, I manually handled the publishing process, which involved testing, compiling, and publishing the library using various npm commands. This approach was time-consuming, required significant effort, and left little room for error—I had to remember the exact sequence of commands and their specific options. Recognizing the inefficiency, I decided to automate the publishing process.

In this article, I’ll walk you through the steps I took to automate the process and streamline my workflow.

By the end, you’ll have a clear understanding of how to efficiently automate your npm package publishing with GitHub Actions. The process will involve the following steps:

Create a GitHub Workflow to automate Linting, building, and Testing process for every push events - We'll start by setting up a GitHub Actions workflow that automatically runs linting, builds your project, and executes tests each time code is pushed to the repository. This ensures that any issues are caught early in the development process, maintaining the integrity of your code source.
Create the second Workflow to automate the Release creation and Publishing proccess for every tag push events - Next, we’ll create a dedicated workflow to handle the release process. This workflow will trigger when a new tag is pushed, automatically compiling the code, updating version numbers, and publishing the package to npm. This step removes the manual intervention from the release cycle, making it more efficient and less prone to errors.
Configure Github Actions (Setting up NPM Access Tokens) - To enable GitHub Actions to publish your package to npm, we’ll configure the necessary secrets within your repository. This involves generating an npm access token and securely storing it in your GitHub repository settings, allowing for seamless and secure automated publishing.
Update README.md by adding the build badges status - Finally, we’ll update your project’s README.md file to include build status badges. These badges provide instant visual feedback on the status of your project’s workflows, showing whether your builds are passing, if tests are successful, and the overall health of your code source.

Flow diagram

Create a GitHub Workflow to automate `Linting`, `building`, and `Testing`

This step consists of creating a workflow that allows me to automate the linting, testing, and building processes for every push event. By doing this, we ensure that any potential errors or overlooked issues are caught early in development, ensuring that the code is clean and ready before considering deployment.

Create a new file named ci.yml in the following directory: .github/workflows, and add this content:

Github Actions trigger



name: CI-Build
on:
  push:
    branches:
      - "**"
    tags-ignore:
      - "v*"
  pull_request:
    types:
      - closed
    branches:
      - master

name - The name of the pipeline
on - The configuration of event trigger. In my case,
- The workflow is triggered by any push event to any branch (branch: "*"), *except for tags starting with "v" (tags-ignore: "v*").
- It is also triggered when a pull request is closed on the master branch.

Jobs: linting code



# ....
    jobs:
    eslint:
        name: Check Syntax with ESLint
        runs-on: ubuntu-22.04
        if: github.event_name == 'push' && github.actor.name != 'github-actions[bot]'
        strategy:
          matrix:
            node-version: [ 18.x, 20.x, 22.x ]
        steps:
        - name: Checkout code style
            uses: actions/checkout@v4
        - name: Use Node.js ${{ matrix.node-version }} to check Lint
            uses: actions/setup-node@v4
            with:
            always-auth: 'false'
            cache: 'npm'
            node-version: ${{ matrix.node-version }}
        - name: Install Dependencies
            run: npm ci
        - name: Run ESLint
            run: |
            npm run lint

This job defines an automated process to check the code's syntax using ESLint↗ whenever certain conditions are met (like a push event). It runs on multiple Node.js versions (strategy.matrix.node-version: [ 18.x, 20.x, 22.x ]) to ensure compatibility and helps maintain code quality by catching issues early.
if: github.event_name == 'push' && github.actor.name != 'github-actions[bot]' - This condition checks if the event that triggered the workflow is a push, and if the person who triggered the event is not the github-actionsbot.

Jobs: Testing



  tests:
    if: github.event_name == 'push' && github.actor.name != 'github-actions[bot]'
    name: Run Tests
    needs: eslint
    runs-on: ubuntu-22.04
    strategy:
      matrix:
        node-version: [ 18.x, 20.x, 22.x ]
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      - name: Use Node.js ${{ matrix.node-version }} to run Tests Covering
        uses: actions/setup-node@v4
        with:
          always-auth: 'false'
          node-version: ${{ matrix.node-version }}
          cache: "npm"
      - name: Install Dependencies
        run: npm ci
      - name: Run Tests Command (Jest-CI)
        run: npm test

This job, tests, is designed to run the project's tests across multiple Node.js versions (18.x, 20.x, and 22.x). The job is executed after the eslint job completes successfully, ensuring that only clean, lint-free code is tested. The job checks out the code, sets up the appropriate Node.js environment, installs dependencies, and then runs the test suite.

Jobs: Building



jobs:
  # {...} <-- Linting job
  buildDist:
    if: github.event_name == 'push' && github.actor.name != 'github-actions[bot]'
    name: Build-to-JS
    needs: [ eslint ]
    runs-on: ubuntu-22.04
    strategy:
      matrix:
        node-version: [ 18.x, 20.x, 22.x ]
        arch:
          - x64
          # - x86
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      - name: Use Node.js ${{matrix.node-version}} to run build dist on ${{ matrix.arch }}
        uses: actions/setup-node@v4
        with:
          cache: "npm"
          node-version: ${{ matrix.node-version}}
      - name: Install dependencies
        run: npm ci
      - name: Run Build Dist
        run: npm run build

This job, buildDist, is designed to compile the project's distribution files (dist) after linting and testing have successfully completed. It does so across different Node.js versions (18.x, 20.x, and 22.x). The job uses caching for npm dependencies to improve speed and reliability.
The job runs conditionally, only on push events and only if the workflow wasn't triggered by GitHub's automated bot account, ensuring that it only executes in relevant contexts.

Full code



name: CI-Build
on:
  push:
    branches:
      - "**"
    tags-ignore:
      - "v*"
  pull_request:
    types:
      - closed
    branches:
      - master
jobs:
  eslint:
    name: Check Syntax with ESLint
    runs-on: ubuntu-22.04
    if: github.event_name == 'push' && github.actor.name != 'github-actions[bot]'
    strategy:
      matrix:
        node-version: [ 18.x, 20.x, 22.x ]
    steps:
      - name: Checkout code style
        uses: actions/checkout@v4
      - name: Use Node.js ${{ matrix.node-version }} to check Lint
        uses: actions/setup-node@v4
        with:
          always-auth: 'false'
          cache: 'npm'
          node-version: ${{ matrix.node-version }}
      - name: Install Dependencies
        run: npm ci
      - name: Run ESLint
        run: |
          npm run lint
  buildDist:
    if: github.event_name == 'push' && github.actor.name != 'github-actions[bot]'
    name: Build-to-JS
    needs: [ eslint, tests ]
    runs-on: ubuntu-22.04
    strategy:
      matrix:
        node-version: [ 18.x, 20.x, 22.x ]
        arch:
          - x64
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      - name: Use Node.js ${{matrix.node-version}} to run build dist on ${{ matrix.arch }}
        uses: actions/setup-node@v4
        with:
          cache: "npm"
          node-version: ${{ matrix.node-version}}
      - name: Install dependencies
        run: npm ci
      - name: Run Build Dist
        run: npm run build
  tests:
    if: github.event_name == 'push' && github.actor.name != 'github-actions[bot]'
    name: Run Tests
    needs: eslint
    runs-on: ubuntu-22.04
    strategy:
      matrix:
        node-version: [ 18.x, 20.x, 22.x ]
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      - name: Use Node.js ${{ matrix.node-version }} to run Tests Covering
        uses: actions/setup-node@v4
        with:
          always-auth: 'false'
          node-version: ${{ matrix.node-version }}
          cache: "npm"
      - name: Install Dependencies
        run: npm ci
      - name: Run Tests Command (Jest-CI)
        run: npm test

Create the second Workflow to automate the `Release creation` and `Publishing proccess`

This step consists of creating a workflow that automates the release and publishing processes for every tag push event. Let's dive into the code and explore how I automated these processes.

First, create a new file named release.yaml in the .github/workflows directory.

Github Action trigger



name: Release
on:
  create:

permissions:
  contents: write

on.create: - The workflow is triggered by the creation of a tag, branch, or release.
permissions.contents: write - The workflow is allowed to write to the repository's contents, which is essential for automating tasks like tagging, updating files, or committing changes as part of the release process. In this case we need the content write permission to create a new Github Release.

Jobs: Github releasing



jobs:
  release:
    if: ${{ startsWith(github.ref, 'refs/tags/v') }}
    name: Release co2mjs
    runs-on: ubuntu-latest
    steps:
      - name: Create release
        env:
          TAG_NAME: ${{ github.ref_name }}
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          gh release create "$TAG_NAME" \
             --repo="$GITHUB_REPOSITORY" \
             --title="${TAG_NAME#v}" \
             --generate-notes

This job automates the creation of a release in a GitHub repository whenever a new tag starting with "v" is pushed.

if: ${{ startsWith(github.ref, 'refs/tags/v') }} - This line ensures that the job only runs if the event is triggered by pushing a tag that starts with "v" (e.g., v1.0.0). This line very important, because remember that the workflow is triggered by the creation of a tag, branch, or release.
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} - This sets the GitHub token required to authenticate the gh command. GitHub automatically provides this token, and it is securely stored in secrets.GITHUB_TOKEN
The job uses the gh (GitHub CLI) command to create the release, setting the title and generating release notes based on the tag.

Jobs: Github publishing



jobs:
  publish-npm:
    if: ${{ startsWith(github.ref, 'refs/tags/v') }}
    name: "Publish the new version to the npmjs"
    runs-on: ubuntu-22.04
    needs: release
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4
      - name: Use Node.js
        uses: actions/setup-node@v4
        with:
          cache: "npm"
          node-version: 18.x
          architecture: x64
          registry-url: https://registry.npmjs.org/
      - name: Run NPM Install
        run: npm ci && npm run build
      - name: Publish
        run: |
          cd dist
          npm publish --access public
        env:
          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

The publish-npm job automates the process of publishing a new version of the package to npm whenever a new tag starting with "v" is pushed. It relies on the successful completion of the release job and performs the following actions: checks out the code, installs dependencies and builds the project, and finally publishes the built package to npm.

⚠️⚠️

NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }} - Provides the authentication token for npmjs.com. This token is stored in GitHub secrets (⚠️repository⚠️ section) to securely authenticate the npm publish command.
registry-url: https://registry.npmjs.org/ - Sets the npm registry URL to the default public npm registry.

Full code



name: Release
on:
  create:
permissions:
  contents: write
jobs:
  release:
    if: ${{ startsWith(github.ref, 'refs/tags/v') }}
    name: Release co2mjs
    runs-on: ubuntu-latest
    steps:
      - name: Create release
        env:
          TAG_NAME: ${{ github.ref_name }}
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          gh release create "$TAG_NAME" \
             --repo="$GITHUB_REPOSITORY" \
             --title="${TAG_NAME#v}" \
             --generate-notes
  publish-npm:
    if: ${{ startsWith(github.ref, 'refs/tags/v') }}
    name: "Publish the new version to the npmjs"
    runs-on: ubuntu-22.04
    needs: release
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4
      - name: Use Node.js
        uses: actions/setup-node@v4
        with:
          cache: "npm"
          node-version: 18.x
          architecture: x64
          registry-url: https://registry.npmjs.org/
      - name: Run NPM Install
        run: npm ci && npm run build
      - name: Publish
        run: |
          cd dist
          npm publish --access public
        env:
          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

Configure Github Actions

To enable GitHub Actions to publish your package to npm, you'll need to configure the necessary access tokens.

1. Create an NPM Access Token

Log in to your NPM account
Navigate to your account settings
1. Access Tokens
2. click Create New Token
3. choose Automation as the token type

copy the generated token

2. Add the Token to Github Secrets

Click on the Settings tab in your repository.
In the left sidebar, select Secrets and variables and then Actions.
Click New repository secret
Name the secret NPM_TOKEN and paste the npm access token you copied earlier into the Value field.

Update `README.md` by adding the build badges status

To provide visibility into the status of your GitHub Actions jobs, you can add build status badges to your README.md file. Insert the following block into your README to display the status of the build and release workflows:



##### 🚦 Build Status
![CI workflow](https://github.com/<OWNER>/<REPO>/actions/workflows/ci-pipeline.yml/badge.svg?branch=master)
![Release workflow](https://github.com/<OWNER>/<REPO>/actions/workflows/release.yml/badge.svg)
___

Now that everything is set up, you can test the workflow by following these steps:

Create a New Branch and Push Changes



git checkout -b feature/your-branch-name # Create and switch to a new branch
touch ./nothing.txt && echo "some text here" >> ./nothing.txt # Create a new file and add some content
git add ./nothing.txt # Stage the new file
git commit -m "my changes" # Commit the changes
git push origin feature/your-branch-name # Push the branch to the remote repository

Monitor the Actions Tab:
- Go to the Actions tab of your repository to observe the CI and release workflows in action.
- Result

Create and Merge a Pull Request
Create and Push a New Tag



git checkout master # Switch to your default branch
npm version patch # this command update package.json and package-lock.json, commit the changes and then, create a new tag locally.
# output : v1.0.1 - or a different version number based on your package.json
git push origin v1.0.1 # Push the new tag to the remote repository

This will create new release and publish a new version to npmjs.com.

Github release ==> https://github.com/nivekalara237/co2mjs/releases
NPM Package ==> https://www.npmjs.com/package/co2m.js

🥳✨
We have reached the end of the article.
Thank you so much 🙂

Your can find the full source code on GitHub Repo↗

Setting up Github Integration with AWS CodeBuild using Cloud Development Kit(CDK) : Automating Angular App Deployment to S3

Kevin Lactio Kemta — Mon, 26 Aug 2024 18:18:10 +0000

Day 012 - 100DaysAWSIaCDevopsChallenge

In this article, I am going to create a new stack to create a AWS CodeBuild for building and deploying an Angular Application from a GitHub repository. Since the the application will be deployed as a static website within an S3 Bucket, configure AWS CodeBuild tp clear the entire bucket before deploying the latest version of the app.

The process will involve the following steps:

Configure the GitHub Repository to trigger CodeBuid on each event in the defined branch
Create a Personal Access in Github concerned project which will be use by AWS CodeBuild project to create and configure a webhook, whenever there is a new commit or pull request to the defined branch.
Set Up AWS Codebuild with Build and Deployment stages
Create a CodeBuild project that compiles and builds the Angular applications. After the build process, we'll include a deployment stage where the built files are uploaded tp the designated S3 Bucket. During this setup, we will also configure an AWS Lambda function that will handle the cleanup of the S3 Bucket before deploying the new version of app.
Deploy the stack
Finally, we will deploy the stack using CDK command line, will will provision the resources (CodeBuild, Lambda, CloudWatch, Role and Permission)

Infrastructure Diagram (`updated`)

Configure the GitHub Repository

Create Personal access token

AWS CodeBuild requires a GitHub Personal Access Token to establish a connection between CodeBuild and the GitHub project.

To create the token, follow these steps:.

go to Github Personal Access Tokeb page↗
Enter a name for the token
Under Select scopes, choose Only select repositories to limit the token's access to specific repositories.
In the Repository permissions section, grant the appropriate permissions to the token, such as Metadata,Webhooks, etc.
Click Generate token

Secrets Manager

Now that the Personal Access Token is generated, it’s best practice and more secure to avoid hardcoding sensitive information such as passwords or secret keys directly in the code. AWS recommends storing secrets in Secrets Manager or Parameter Store. In this case, we will store the GitHub Personal Access Token in AWS Secrets Manager for secure and convenient retrieval when needed.

To store your GitHub Token in the Secrets Manager AWS using console, follow these steps

click on Store a new secret
Select type Other type of secret
In the Key/Value pairs section, fill github as the key and paste de Github Token Previously generated as the value
click Next
In the Configure secret fill enter a name for the secret
click Next then Next, and finally Store

Set Up AWS Codebuild with Build and Deployment stages

Load Github credential

interface DeveloperToolsProps extends StackProps {
  git: {
    repositoryName: string,
    owner: string,
    branchRef?: string,
    accessTokenSecretArn?: string
  },
  build: {
    todoAppBucketName: string
  }
}
export class DeveloperToolsStack extends BaseStack {
  constructor(scope: Construct, id: string, props: DeveloperToolsProps) {
    super(scope, id, props)
    const secret = secrets.Secret.fromSecretAttributes(this, 'Github-Personal-Access-Token', {
        secretCompleteArn: props.git.accessTokenSecretArn
    })
    new codebuild.GitHubSourceCredentials(this, 'LoadGitHubCredentials', {
      accessToken: secret.secretValueFromJson('github')
    })
    // setup stack..
  }
}

secret - The Github personal access token
GitHubSourceCredentials - Credentials used to authenticate with GitHub within the current AWS account and region.

Create LogGroup for CodeBuild reports logs

export class DeveloperToolsStack extends BaseStack {
  constructor(scope: Construct, id: string, props: DeveloperToolsProps) {
    super(scope, id, props)
    ...
    const cloudwatch = new logs.LogGroup(this, 'CodeBuildLogGroup', {
      logGroupName: 'CodeBuildLogGroup',
      retention: logs.RetentionDays.ONE_DAY,
      logGroupClass: logs.LogGroupClass.STANDARD,
      removalPolicy: RemovalPolicy.DESTROY
    })
  }
}

Set up AWS CodeBuid Project

export class DeveloperToolsStack extends BaseStack {
  constructor(scope: Construct, id: string, props: DeveloperToolsProps) {
    super(scope, id, props)
    ...
    const emptyBucketLambda = new LambdaFunction(...);

    const todoBuildProject = new codebuild.Project(this, 'CodeBuild', {
      projectName: 'TodoAppCodeBuildProject',
      buildSpec: this.buildSpec(),
      logging: {
        cloudWatch: {
          logGroup: cloudwatch,
          enabled: true
        }
      },
      source: codebuild.Source.gitHub({
        repo: props.git.repositoryName,
        reportBuildStatus: true,
        owner: props.git.owner,
        webhook: true,
        webhookFilters: [
            codebuild.FilterGroup.inEventOf(codebuild.EventAction.PUSH, codebuild.EventAction.PULL_REQUEST_MERGED)
            .andBranchIs(props.git.branchRef ?? 'master')
            .andFilePathIs('apps')
        ]
      }),
      visibility: codebuild.ProjectVisibility.PUBLIC_READ,
      environment: {
        buildImage: codebuild.LinuxBuildImage.AMAZON_LINUX_2_5,
        computeType: codebuild.ComputeType.SMALL,
        privileged: true
      },
      environmentVariables: {
        BUCKET_NAME: {
          type: codebuild.BuildEnvironmentVariableType.PLAINTEXT,
          value: props.build.todoAppBucketName
        },
        EMPTY_BUCKET_FUNCTION_ARN: {
          type: codebuild.BuildEnvironmentVariableType.PLAINTEXT,
          value: emptyBucketLambda.functionArn
        }
      },
      badge: false,
      role: new iam.Role(this, 'BuildCodeRole', {
        path: '/',
        assumedBy: new iam.ServicePrincipal('codebuild.amazonaws.com', {
          region: props.env?.region
        }),
        inlinePolicies: {
          logging: new iam.PolicyDocument({
            assignSids: true,
            statements: [
              new iam.PolicyStatement({
                effect: iam.Effect.ALLOW,
                actions: [
                  'logs:CreateLogGroup',
                  'logs:PutLogEvents',
                  'logs:CreateLogStream'
                ],
                resources: [cloudwatch.logGroupArn]
              })
            ]
          }),
          s3: new iam.PolicyDocument({
            assignSids: true,
            statements: [
              new iam.PolicyStatement({
                effect: iam.Effect.ALLOW,
                actions: ['s3:PutObject'],
                resources: [
            `arn:aws:s3:::${props.build.todoAppBucketName}`,
            `arn:aws:s3:::${props.build.todoAppBucketName}/*`
                ]
              })
            ]
          }),
          secrets: new iam.PolicyDocument({
            assignSids: true,
            statements: [
              new iam.PolicyStatement({
                effect: iam.Effect.ALLOW,
                resources: [props.git.accessTokenSecretArn!],
                actions: ['secretsmanager:GetSecretValue']
              })
            ]
          }),
          lambda_function: new iam.PolicyDocument({
            assignSids: true,
            statements: [
              new iam.PolicyStatement({
                effect: iam.Effect.ALLOW,
                actions: [
                  'lambda:InvokeFunction'
                ],
                resources: [emptyBucketLambda.functionArn]
              })
            ]
          })
        }
      })
    })

  }

  buildSpec = (): codebuild.BuildSpec => {
    return codebuild.BuildSpec.fromObjectToYaml({
      version: '0.2',
      env: {
        shell: 'bash'
      },
      phases: {
        pre_build: {
          commands: [
            'echo Build started on `date`',
            'npm install -g @angular/cli',
            'cd apps/todo-app && npm install'
          ]
        },
        build: {
          commands: [
            'ng build --configuration production',
            `aws lambda invoke --function-name \${EMPTY_BUCKET_FUNCTION_ARN} --cli-binary-format raw-in-base64-out --payload '{ "payload": {"bucketName": "\${BUCKET_NAME}"} }' response.json`,
            'aws s3 cp dist/todo-app/browser s3://${BUCKET_NAME}/ --recursive'
          ]
        }
      },
      artifacts: {
        files: ['**/*']
      },
      cache: {
        paths: ['node_modules/**/*', 'dist/**']
      }
    })
  }
}

This snippet code sets up AWS CodeBuild to automatically build the application when specific events occur in a GitHub repository. It listens for pushes or merged pull requests on a specified master branch, but only triggers a build if changes are made in the apps directory. Additionally, it reports the build status back to GitHub to provide feedback directly within the GitHub interface.

source - Defines the source of the CodeBuild project : It specifies that the build source will be a Github repository
- repo - Specifies the name of the GitHub repository that CodeBuild will use as the source
- reportBuildStatus - Enables reporting the build status back to GitHub. GitHub will display the build status (e.g., success, failure)
- webhook - Enable a webhook in the Github repository to automatically trigger the Codebuild project whenever some events occurs (PUSH and PULL REQUEST). This ensures continuous integration.
- webhookFilters - Speficied the condidions under which the webhook will trigger the CodeBuild project.
Environment - Defines the environment in which CodeBuild will run (e.g: Operating system, computer resources, etc.)
- buildImage - The build Image that CodeBuild will use, needs for Operating System.
- computeType - The type of compute resources (CPC, Memory) allocated to the build environment.
- privileged - Allows the build container to perform certain operations that require elevated permissions. It is require by Docker.

Setup Lambda to delete all objects in the bucket

As for new deployment, we need to clean up the bucket for a new version of code. This is the function code to clean a specified bucket.

// ./apps/functions/todo-app/empty-bucket.function.ts
const client = new S3Client({
  region: process.env.REGION ?? 'us-east-1'
})
const repo = new ObjectRepository(client)
export const handler: Handler = async (event: {
  payload: { bucketName: string }
}, context: Context, callback: Callback) => {
  if (!event.payload?.bucketName) {
    callback(new Error('Invalid payload'))
  } else {
    try {
      const response = await repo.deepGetAllObjects(event.payload.bucketName)
      await repo.deleteBucketObjects(event.payload.bucketName, response.objects.map(value => value.key))
      callback(null, response)
    } catch (e) {
      callback(e, null)
    }
  }
}

Find the full components of the function here↗

export class DeveloperToolsStack extends BaseStack {
  constructor(scope: Construct, id: string, props: DeveloperToolsProps) {
    super(scope, id, props)
    ...
    const emptyBucketLambda = new LambdaFunction(this, 'EmptyBucketFunction', {
      entryFunction: './apps/functions/todo-app/empty-bucket.function.ts',
      functionName: 'Empty-Bucket-Function',
      env: {
        REGION: props.env?.region!
      },
      role: {
        name: 'EmptyBucketFunctionRole',
        inlinePolicies: {
          s3: new iam.PolicyDocument({
            assignSids: true,
            statements: [
              new iam.PolicyStatement({
                effect: iam.Effect.ALLOW,
                actions: [
                  's3:GetBucket',
                  's3:GetObject',
                  's3:ListObjects',
                  's3:DeleteObject'
                ],
                resources: [
                  `arn:aws:s3:::${props.build.todoAppBucketName}`,
                  `arn:aws:s3:::${props.build.todoAppBucketName}/*`
                ]
              })
            ]
          })
        }
      },
      logged: true
    });
    emptyBucketLambda.resource.applyRemovalPolicy(RemovalPolicy.DESTROY)
    ...
  }
}

Allow AWS CodeBuild to invoke Lambda

emptyBucketLambda.grantInvoke({
    grantPrincipal: new iam.ServicePrincipal('codebuild.amazonaws.com')
}, todoBuildProject.projectArn, props.env?.account)

Deploy the stack

const app = new cdk.App()
const devtoolsStack = new DeveloperToolsStack(app, 'Day012DevToolsStack', {
  env,
  git: {
    repositoryName: '100DaysTerraformAWSDevops',
    branchRef: 'master',
    owner: '<GITHUB_USERNAME>',
    accessTokenSecretArn: process.env.PERSONAL_ACCESS_SECRET_ARN
  },
  build: {
    todoAppBucketName: "<YourBucketName>"
  }
})
app.synth()

git clone https://github.com/nivekalara237/100DaysTerraformAWSDevops.git

cd 100DaysTerraformAWSDevops/day_012
export PERSONAL_ACCESS_SECRET_ARN="arn:aws:secretsmanager:<REGION>:<ACCOUNT>:secret:<SECRET_ID>"

cdk deploy --profile cdk-user Day012DevToolsStack

Below are the results after two pushes to the master branch:

🥳✨
We have reached the end of the article.
Thank you so much 🙂

Your can find the full source code on GitHub Repo↗

Setting Up and Securing CloudFront for S3 Static Sites with Custom Subdomains Using AWS Cloud Development Kit(CDK)

Kevin Lactio Kemta — Sat, 24 Aug 2024 13:43:22 +0000

Day 011 - 100DaysAWSIaCDevopsChallenge

Recently on Day 010 for my 100 Days of Code Challenge, I demonstrated how to create a custom domain name for API Gateway to make API endpoints more human-readable. Today, I will address setting up a secure subdomain for an Angular app deployed in an S3 bucket to avoid using the insecure default S3 URL http://<BUCKET_NAME>.website-static.s3.amazonaws.com. To achieve this, I will use CloudFront Distribution, a Content Delivery Network (CDN) service provided and managed by AWS. It is designed to deliver a content such as Images, Video Streams and static webside (like ours) quickly and securely to users around the world. CloudFront enhances the performance and security of websites and applications by delivering content closer to end users.

To reach out the goal, I will follow these steps:

Set Up a New CDK Construct for CloudFront Distribution with Subdomain Configuration
Create a new Construct that defines the CloudFront distribution. This will include setting up the distribution to serve content via a subdomain, specifying the origin, and configuring any necessary settings like request and caching policies and SSL certificates.
Update the CORS Configuration of the Website Bucket: Modify the CORS configuration of the S3 bucket hosting the website. This involves adding the newly created CloudFront distribution as an allowed origin, ensuring that the content can be accessed correctly through the subdomain.
Update the existing stack
Incorporate the new CloudFront distribution and any associated changes into the existing infrastructure stack.
Deploy the new infrastructure
Deploy the updated infrastructure, including the new CloudFront distribution and any changes to the existing stack.

Diagram of infrastructure

Set Up a New CDK Construct for CloudFront Distribution with Subdomain Configuration

I chose to create a new CDK construct rather than simply updating the stack with the necessary resources. Since this setup requires multiple resources that need to work together cohesively, creating a custom construct provides a more organized and modular approach. This allows for better management, easier reuse, and a cleaner integration of the CloudFront distribution with the subdomain configuration without forgeting the SSL configuration.

// The construct properties
interface WebsiteDistributeionProps extends CustomStackProps {
  websiteBucketName: string;
  domain: string; // the base domain
  acm: {
    certificateArn: string
  };
  distribution: {
    domainName: string
  }
}
// the custom construct
export class WebsiteDistribution extends Construct {
    constructor(scope: Construct, id: string, props: WebsiteDistributeionProps) {
    super(scope, id);
    // retrieve or add resources here
}

Lookup the necessaries resources

To create a secure distribution that points to the S3 static website, we need to retrieve the certificate and the bucket resource.

const bucket = s3.Bucket.fromBucketName(this, `BucketName_${id}`, props.websiteBucketName!)
const certificate = acm.Certificate.fromCertificateArn(this, `SSLCertificate_${id}`, props.acm?.certificateArn!)

Create CloudFront Distribution

const cachePolicy = new cf.CachePolicy(this, `Cache-${id}-Policy`, {
    cachePolicyName: `Cache-${id}-Policy`,
    enableAcceptEncodingGzip: true,
    enableAcceptEncodingBrotli: true,
    queryStringBehavior: cf.CacheQueryStringBehavior.all(),
    cookieBehavior: cf.CacheCookieBehavior.none(),
    defaultTtl: Duration.seconds(30),
    headerBehavior: cf.CacheHeaderBehavior.allowList(
        'Origin',
        'Accept',
        'Access-Control-Request-Method',
        'Access-Control-Request-Headers'
    )
})
const originRequestPolicy = new cf.OriginRequestPolicy(this, `Origin-Request-${id}-Policy`, {
    originRequestPolicyName: `Origin-Request-${id}-Policy`,
    queryStringBehavior: cf.CacheQueryStringBehavior.all(),
    cookieBehavior: cf.CacheCookieBehavior.none(),
    headerBehavior: cf.CacheHeaderBehavior.allowList(
        'Origin',
        'Accept',
        'Access-Control-Request-Method',
        'Access-Control-Request-Headers'
    )
})
const distribution = new cf.Distribution(this, `SSLCertificate_${id}`, {
    enabled: true,
    defaultBehavior: {
        origin: new origins.HttpOrigin(bucket.bucketWebsiteDomainName, {
            protocolPolicy: cf.OriginProtocolPolicy.HTTP_ONLY,
            httpPort: 80,
            httpsPort: 443,
            connectionTimeout: Duration.seconds(10),
            originId: generateResourceID()
        }),
        allowedMethods: cf.AllowedMethods.ALLOW_ALL,
        cachedMethods: cf.CachedMethods.CACHE_GET_HEAD_OPTIONS,
        cachePolicy,
        originRequestPolicy
    },
    certificate,
    httpVersion: cf.HttpVersion.HTTP2,
    // @ts-ignore
    domainNames: [props.distribution?.domainName].filter(value => !!value)
})

certificate - We need this to secure the traffic in and out of our system.
httpVersion - Refer to the documentation: For viewers and CloudFront to use HTTP/ 2, viewers must support TLS 1.2 or later, and must support server name identification (SNI).
domainNames - Since we don't want to use the default URL provided by AWS (somthing like 12345abcdef.cloudfront.net), we want inbound traffic to come from the specified subdomain.
defaultBehavior
- origin - Where the cloudFront will route the traffic to. S3 static website in our case.
- allowedMethods - HTTP methods to allow for this behavior. In this case GET, POST, PUT, HEAD, DELETE and OPTIONS
- cachedMethods - HTTP methods to cache for this behavior. In this case CloudFront will cache only the responses of GET and OPTIONS request according to the cachePolicy.
- cachePolicy - Determines what values are included in the cache key, and the time-to-live (TTL) values for the cache.
- originRequestPolicy - Determines which values (e. g., headers, cookies) are included in requests that CloudFront sends to the origin.

This is the minimal configuration that we need to create a new cloudFront distribution.

Create a subdomain - RecordSet of type `CNAME`

Now that the CloudFront distribution is properly configured, let can proceed to create the new subdomain and attach it to the distribution. Let's begin by retrieving the HostedZone:

const hostedZone = route53.HostedZone.fromLookup(this, `HostedZone_${id}`, {
    domainName: props.domain!
})

⚠️⚠️ Note that domainName refers to the main domain, not the subdomain that we are going to create.

const cnameRecord = new route53.CnameRecord(this, `DomainCNAME_${id}`, {
    recordName: props.distribution?.domainName + '.',
    domainName: distribution.distributionDomainName,
    zone: hostedZone,
    deleteExisting: true,
    ttl: Duration.minutes(10),
    comment: `RecordSet to send traffic from ${props.distribution?.domainName} to ${distribution.distributionDomainName}`
})

recordName - The subdomain nomination.
domainName - The target resource (URL) to which the traffic will be routed.
zone - The HostedZone to which the subdomain will be attached.

The complete source code of the custom construct:

import { Construct } from 'constructs'
import {
  aws_certificatemanager as acm,
  aws_cloudfront as cf,
  aws_cloudfront_origins as origins,
  aws_route53 as route53,
  aws_s3 as s3,
  Duration
} from 'aws-cdk-lib'
import { CustomStackProps } from '../custom-stack.props'
import { generateResourceID } from './utils'

interface WebsiteDistributeionProps extends CustomStackProps {
  websiteBucketName: string;
  acm: {
    certificateArn: string
  },
  distribution: {
    domainName: string
  }
}
export class WebsiteDistribution extends Construct {
  private readonly _distributionUrl: string
  private readonly _viewersUrl: string
  constructor(scope: Construct, id: string, props: WebsiteDistributeionProps) {
    super(scope, id)
    const bucket = s3.Bucket.fromBucketName(this, `BucketName_${id}`, props.websiteBucketName!)
    const certificate = acm.Certificate.fromCertificateArn(this, `Certificate_${id}`, props.acm?.certificateArn!)
    const cachePolicy = new cf.CachePolicy(this, `Cache-${id}-Policy`, {
      cachePolicyName: `Cache-${id}-Policy`,
      enableAcceptEncodingGzip: true,
      enableAcceptEncodingBrotli: true,
      queryStringBehavior: cf.CacheQueryStringBehavior.all(),
      cookieBehavior: cf.CacheCookieBehavior.none(),
      defaultTtl: Duration.seconds(30),
      headerBehavior: cf.CacheHeaderBehavior.allowList(
        'Origin',
        'Accept',
        'Access-Control-Request-Method',
        'Access-Control-Request-Headers'
      )
    })
    const originRequestPolicy = new cf.OriginRequestPolicy(this, `Origin-Request-${id}-Policy`, {
      originRequestPolicyName: `Origin-Request-${id}-Policy`,
      queryStringBehavior: cf.CacheQueryStringBehavior.all(),
      cookieBehavior: cf.CacheCookieBehavior.none(),
      headerBehavior: cf.CacheHeaderBehavior.allowList(
        'Origin',
        'Accept',
        'Access-Control-Request-Method',
        'Access-Control-Request-Headers'
      )
    })
    const distribution = new cf.Distribution(this, `CfDistribution_${id}`, {
      enabled: true,
      defaultBehavior: {
        origin: new origins.HttpOrigin(bucket.bucketWebsiteDomainName, {
          protocolPolicy: cf.OriginProtocolPolicy.HTTP_ONLY,
          httpPort: 80,
          httpsPort: 443,
          connectionTimeout: Duration.seconds(10),
          originId: generateResourceID()
        }),
        allowedMethods: cf.AllowedMethods.ALLOW_ALL,
        cachedMethods: cf.CachedMethods.CACHE_GET_HEAD_OPTIONS,
        cachePolicy,
        originRequestPolicy
      },
      certificate,
      httpVersion: cf.HttpVersion.HTTP2,
      // @ts-ignore
      domainNames: [props.distribution?.domainName].filter(value => !!value)
    })
    distribution.metric5xxErrorRate({
      label: `website-distribution-${id}-5xxError`,
      color: '#e93d1a'
    })
    const hostedZone = route53.HostedZone.fromLookup(this, `HostedZone_${id}`, {
      domainName: props.domain!
    })
    const cnameRecord = new route53.CnameRecord(this, `DomainCNAME_${id}`, {
      recordName: props.distribution?.domainName + '.',
      domainName: distribution.distributionDomainName,
      zone: hostedZone,
      deleteExisting: true,
      ttl: Duration.minutes(10),
      comment: `RecordSet to send traffic from ${props.distribution?.domainName} to ${distribution.distributionDomainName}`
    })
    this._distributionUrl = distribution.distributionDomainName
    this._viewersUrl = cnameRecord.domainName
  }
  get distributionUrl(): string {
    return this._distributionUrl
  }
  get viewsUrl(): string {
    return this._viewersUrl
  }
}

website-distribution.construct.ts[↗]

Update the CORS Configuration of the Website Bucket

export interface WebsiteStackProps extends StackProps {
  ...
  origins?: string[] // the list of allowed origins
}
const wsBucket = new s3.Bucket(...)

wsBucket.addCorsRule({
    allowedMethods: [
        s3.HttpMethods.GET, s3.HttpMethods.HEAD,
        s3.HttpMethods.DELETE, s3.HttpMethods.PUT,
        s3.HttpMethods.POST
    ],
    allowedOrigins: (props.origins ?? ['*']).filter(value => !!value && value.trim().length > 0),
    maxAge: Duration.minutes(10).toSeconds(),
    allowedHeaders: ['*']
})

website-stack.ts[↗]

allowedOrigins - Includes the subdomain that was previously created.
allowedHeaders - Allows all headers coming from CloudFront.

Upload the website files to the S3 Bucket

new s3deploy.BucketDeployment(this, 'DeployTodoApp', {
    destinationBucket: wsBucket,
    sources: [
        s3deploy.Source.asset('../apps/todo-app/dist/todo-app/browser', {
            exclude: ['node_modules']
        })
    ]
})

website-stack.ts[↗]

Update the existing stack

Find the full stack class 👉🏽👉🏽 cdk-stack.ts[↗]


export class CdkStack extends BaseStack {
  constructor(scope: Construct, id: string, private props: CustomStackProps) {
    ...
    const distribution = new WebsiteDistribution(this, 'TodoCloudfront', <WebsiteDistributeionProps>{ ...props })
    new CfnOutput(this, 'TodoCloudfrontUrl', {
        value: distribution.distributionUrl,
        key: 'CfURL'
    })
    new CfnOutput(this, 'TodoCloudfrontViewUrl', {
        value: distribution.viewsUrl,
        key: 'ViewUrl'
    })
    new CfnOutput(this, 'ApiGatewayUrlOutput', {
        value: restApi.url,
        key: 'ApiGatewayUrl'
    })
  }
}

Deploy the new infrastructure

git clone https://github.com/nivekalara237/100DaysTerraformAWSDevops.git

cd 100DaysTerraformAWSDevops/apps && ng build --configuration production && cd ../..

cd 100DaysTerraformAWSDevops/day_011
export MAIN_DOMAIN="yourdomain.com"
export STAGE_NAME="dev" # needed by api gateway staging
export CERTIFICATE_ARN="arn:aws:acm:us-east-1:xxxx:certificate/xxxx-xxxx-xxxx-xxxxxxxx"

cdk deploy --profile cdk-user --all

After a few minutes, CloudFront will complete the deployment and distribution of the content. Here is the result:

🥳✨
We have reached the end of the article.
Thank you so much 🙂

Your can find the full source code on GitHub Repo↗

Configuring a Custom Domain for API Gateway with AWS Cloud Development Kit (CDK): SSL Certificate Use and Route 53 Integration

Kevin Lactio Kemta — Thu, 22 Aug 2024 11:37:16 +0000

Day 010 - 100DaysAWSIaCDevopsChallenge

Recently in the Day 009 of my 100 Days of Code Challenge, I created and deployed an infrastructure for securing API Gateway methods using AWS Cognito. Since these APIs are used by an Angular app, it is necessary to update the environment.ts file with the new backend endpoint after every API Gateway ID renewest. To avoid this repetitive update with each infrastructure modification, today I am going to configure a domain name in front of our API Gateway, so that we no longer need to change the environment.ts file for every API Gateway destruction.

To achieve this, we will follow these steps:

Set Up a New CDK Construct for Domain Name Configuration
Create a new CDK construct to handle the domain name configuration for the API Gateway.
Update the Existing Stack
Modify the existing stack by instantiating the new construct with the appropriate parameters.
Deploy the New Infrastructure
Deploy the updated infrastructure and update the endpoint values in the environment.ts file accordingly.

Set Up a New CDK Construct for Domain Name Configuration

To make the infrastructure stack more readable and maintainable, I prefer to create a custom construct for setting up a domain name for the API Gateway. This is necessary because creating a domain name for the API Gateway involves configuring multiple AWS resources. Within this custom construct, the following resources will need to be configured:

Retrieve Certificate by ARN To customize a domain name for our API Gateway, AWS requires us to prove ownership of the domain. The SSL/TLS certificate, typically provided by AWS Certificate Manager (ACM), verifies that we control the domain associated with the custom API Gateway domain name. Since I have already requested my certificate in ACM, the following code snippet retrieves the existing certificate:



const existingCertificate = acm.Certificate.fromCertificateArn( this, 
    `ApiDomainCertificate_${id}`, 
    "<CERTIFICATE_ARN>")

The Custom domain name



// const props: {api: IRestApi, apiDomain: string} = {...}
const domain = new api.DomainName(this, `CustomApiDomainName`, {
    domainName: props.apiDomain,
    certificate: existingCertificate,
    endpointType: api.EndpointType.EDGE,
    securityPolicy: api.SecurityPolicy.TLS_1_2,
    mapping: props.api
})

domainName - The domain name to be used. In my case, it is a subdomain like todo-api.mydomain.com.
certificate - The previously retrieved certificate, looked up by its ARN.
mapping - The API Gateway that will be mapped to the domain name.
endpointType - Since we are using us-east-1 to create/import the certificate, we intend to create an edge-optimized API Gateway. For this type, the certificate must be created in us-east-1. If you choose to create a regional API Gateway, the certificate must reside in the region where the API Gateway is created.

Hosted Zone & RecordSet

Since the hosted zone has already been created, we just need to find it by using the main domain name.



// const props: {api: IRestApi, apiDomain: string, domain: string} = {...}
const hostedZone = route53.HostedZone.fromLookup(this, 
`HostedZoneLookup_${generateResourceID()}`, {
    domainName: props.domain
})

Then, create an A record for the subdomain that will route all traffic to the API Gateway domain name.



// const props: {api: IRestApi, apiDomain: string, domain: string} = {...}
const a_record = new route53.ARecord(this, `A_Record`, {
    recordName: props.apiDomain.concat('.'),
    zone: hostedZone,
    target: route53.RecordTarget.fromAlias(new targets.ApiGatewayDomain(domain)),
    ttl: Duration.minutes(2),
    comment: `The Record “A“ to route traffic from subdomain ${props.apiDomain} to the api gateway #${props.api.restApiId}`,
    region: props.env?.region,
    deleteExisting: true
})
a_record.applyRemovalPolicy(RemovalPolicy.DESTROY)

recordName - The subdomain name for this record. Note that the value ends with a dot (.). If you don't include the dot, CDK will treat it as a segment rather than a fully qualified domain name.

recordName	result
foo	foo.mydomain.com
zuzu.mydomain.com	zuzu.mydomain.com.mydomain.com
bar.mydomain.com.	bar.mydomain.com

The full code:



import { Construct } from 'constructs'
import { CustomStackProps } from '../custom-stack.props'
import {
  aws_apigateway as api,
  aws_certificatemanager as acm,
  aws_route53 as route53,
  aws_route53_targets as targets,
  Duration,
  RemovalPolicy
} from 'aws-cdk-lib'
import { generateResourceID } from './utils'

interface ApiDomainNameProps extends CustomStackProps {
  apiDomain: string,
  api: api.RestApi,
  certificateArn: string
}

export class ApiDomainName extends Construct {
  constructor(scope: Construct, id: string, private props: ApiDomainNameProps) {
    super(scope, id)
    if (!props.domain) {
      throw new Error('The domain parameter must be specified as the ApiDomainName has been instantiated')
    }
    if (!this.props.apiDomain.endsWith(props.domain)) {
      throw new Error(`The apiDomain parameter must be a subdomain of ${this.props.domain}`)
    }
    const existingCertificate = acm.Certificate.fromCertificateArn(this, `ApiDomainCertificate_${id}`, props.certificateArn)
    const domain = new api.DomainName(this, `CustomApiDomainName_${id}`, {
      domainName: props.apiDomain,
      certificate: existingCertificate,
      endpointType: api.EndpointType.EDGE,
      securityPolicy: api.SecurityPolicy.TLS_1_2,
      mapping: props.api
    })
    domain.node.addDependency(props.api)

    const hostedZone = route53.HostedZone.fromLookup(this, `HostedZoneLookup_${generateResourceID()}`, {
      domainName: props.domain
    })

    const a_record = new route53.ARecord(this, `A_Record_${id}`, {
      recordName: props.apiDomain.concat('.'),
      zone: hostedZone,
      target: route53.RecordTarget.fromAlias(new targets.ApiGatewayDomain(domain)),
      ttl: Duration.minutes(2),
      comment: `The "A" record is used to route traffic from the subdomain ${props.apiDomain} to the API Gateway #${props.api.restApiId}`,
      region: props.env?.region,
      deleteExisting: true
    })
    a_record.applyRemovalPolicy(RemovalPolicy.DESTROY)
  }
}

Update the Existing Stack



export class CdkStack extends BaseStack {
  constructor(scope: Construct, id: string, private props: CustomStackProps) {
    ....
    new ApiDomainName(this, 'TodoListApiDomainName', {
        ...props,
      apiDomain: props.route53?.apigw?.subdomain ?? 'api.' + this.props.domain,
      api: restApi,
      certificateArn: props.route53?.apigw?.certificateArn!
    });
    ....
  }
}

Deploy the entire infrastructure



git clone https://github.com/nivekalara237/100DaysTerraformAWSDevops.git

cd 100DaysTerraformAWSDevops/day_010

export STAGE_NAME="dev" # needed by api gateway staging
export CERTIFICATE_ARN="arn:aws:acm:us-east-1:xxxx:certificate/xxxx"

cdk deploy --profile cdk-user --all

Re-deploy app as S3 static website

To deploy the Angular app to the S3 bucket as a website, follow the instructions in my previous article 👉🏽👉🏽 Deploying a REST API and Angular Frontend Using AWS CDK, S3, and API Gateway↗
⚠️⚠️ Make sure to update the API Gateway URL in the src/environment.ts file with the correct API Gateway endpoint before building and deploying the application.



export const environment = {
  production: true,
  apiUrl: 'https://todo-api.mydomain.com/'
}

🥳✨
We have reached the end of the article.
Thank you so much 🙂

Your can find the full source code on GitHub Repo↗

Building a Secure Serverless Angular App with AWS CDK, Cognito, Lambda, and API Gateway

Kevin Lactio Kemta — Tue, 20 Aug 2024 15:55:22 +0000

Day 009 - 100DaysAWSIaCDevopsChallenge

Recently, on day 008 of my 100 Days of Code challenge, I created an infrastructure to deploy a REST API behind an API Gateway and an Angular app to consume the exposed APIs. Today, I am going to secure some of those APIs with AWS Cognito. To access these APIs, the user will need to provide an ACCESS or ID token in the header of every requests made.

To achieve this, we will follow the steps below:

Set Up AWS Cognito User Pool: Create an User Pool to handler users, and then create the App Client for authentication needs.
Create Lambda functions for authentication process: Develop the necessary Lambda functions to support the authentication process.
Create Api Gateway Authorizer: Set up an API Gateway authorizer of type Cognito.
Update the Rest Api Methods: Secure the methods that need protection by applying the Cognito authorizer.
Updated the Angular App with authentication proccess: Implement the authentication process within the Angular app.
Re-deploy the app as S3 static website

Prerequises

AWS Cloud Developement Kit (CDK)
Cognito, Secrets Manager, Lambda, API Gateway, IAM
Typescript, Angular, esbuild

Infrastructure Diagram

Set Up AWS Cognito User Pool

I chose to create my own CDK construct to achieve this. In my construct, I first create the User Pool and then associate an App Client with it. Before these two resources are created, I will store the raw app client secret in an AWS Secrets Manager resource for added security, and I will retrieve it directly inside the Lambda by key.

Cogniton Construct



import { Construct } from 'constructs'
import { aws_cognito as cognito, aws_secretsmanager as sm, Duration, RemovalPolicy, StackProps } from 'aws-cdk-lib'

interface CognitoProps extends StackProps {
  verificationEmailUrl: string,
  fromEmail: string
}

export class Cognito extends Construct {
  private readonly _defaultAppClientId: string
  private readonly _secretValueArn: string
  private readonly _userPoolArn: string
  private readonly _userPoolId: string
  private readonly _userPool: cognito.IUserPool
  constructor(scope: Construct, id: string, props: CognitoProps) {
    super(scope, id)
    const stdRequired: cognito.StandardAttribute = {
      required: true
    }
    const userPool = new cognito.UserPool(this, 'CognitoUserPoolResource', {
      removalPolicy: RemovalPolicy.DESTROY,
      accountRecovery: cognito.AccountRecovery.EMAIL_ONLY,
      userPoolName: 'awschallenge-userpool-2',
      selfSignUpEnabled: true,
      userInvitation: {
        emailSubject: 'An account has been created for you',
        emailBody: 'Your Cognito account has been created by an admin. Here is your username: {username} and your temporary password: {####}'
      },
      signInCaseSensitive: false,
      signInAliases: {
        email: true,
        username: true
      },autoVerify: {
        email: true,
        phone: true
      },keepOriginal: {
        email: true
      },standardAttributes: {
        email: { ...stdRequired, mutable: true }
      },
      customAttributes: {
        'domain': new cognito.StringAttribute(),
        'created_at': new cognito.StringAttribute(),
        'last_updated_at': new cognito.StringAttribute(),
        'verified_at': new cognito.StringAttribute(),
        'first_name': new cognito.StringAttribute(),
        'last_name': new cognito.StringAttribute()
      },
      passwordPolicy: {
        tempPasswordValidity: Duration.days(14),
        minLength: 6,
        requireDigits: true,
        requireUppercase: true,
        requireLowercase: true,
        requireSymbols: true
      },
      email: cognito.UserPoolEmail.withCognito(props.fromEmail)
    })
    userPool.addTrigger(cognito.UserPoolOperation.CUSTOM_MESSAGE, this.customEmailMessageLambda())
    const userPoolClient = userPool.addClient('todo-app-client', {
      authFlows: {
        userPassword: true,
        adminUserPassword: true,
        custom: true,
        userSrp: true
      },
      disableOAuth: true,
      preventUserExistenceErrors: true,
      generateSecret: true,
      userPoolClientName: 'Todo-App-Client',
      accessTokenValidity: Duration.minutes(30),
      idTokenValidity: Duration.minutes(30),
      refreshTokenValidity: Duration.days(30)
    })
    const clientSecret = new sm.Secret(this, 'SecretManagerResource', {
      secretName: `${userPool.userPoolId}/${userPoolClient.userPoolClientId}`,
      generateSecretString: {
        secretStringTemplate: JSON.stringify({
          appClientSecret: userPoolClient.userPoolClientSecret.unsafeUnwrap()
        }),
        generateStringKey: `${userPoolClient.userPoolClientName}-AppSecret`
      }
    })
    this._defaultAppClientId = userPoolClient.userPoolClientId
    this._secretValueArn = clientSecret.secretArn
    this._userPoolArn = userPool.userPoolArn
    this._userPoolId = userPool.userPoolId
    this._userPool = userPool
  }
  get defaultAppClientId() {
    return this._defaultAppClientId
  }
  get secretValueArn() {
    return this._secretValueArn
  }
  get userPoolArn() {
    return this._userPoolArn
  }
  get userPoolId() {
    return this._userPoolId
  }
  get userPool() {
    return this._userPool
  }
  customEmailMessageLambda = () => {
    return new LambdaFunction(this, 'CustomVerificationMessage', {
        functionName: 'CustomMessageFunction',
        entryFunction: './apps/functions/auth/custom-confirm-message.function.ts',
        env: {
            CONFIRM_URL: this.props.verificationEmailUrl
        },
        logged: true
        }).resource
    }
}

The Lambda construct



import * as cdk from 'constructs'
import { Construct } from 'constructs'
import { Aws, aws_apigateway, aws_iam as iam, aws_lambda as lambda, Duration, StackProps } from 'aws-cdk-lib'
import { NodejsFunction, SourceMapMode } from 'aws-cdk-lib/aws-lambda-nodejs'
import { generateResourceID } from './utils'
interface Props {
  entryFunction: string;
  functionName: string;
  env?: Record<string, string>;
  layersArns?: string[];
  externalDeps?: string[];
  logged?: boolean;
  memory?: number;
  role?: {
    name?: string,
    inlinePolicies?: Record<string, iam.PolicyDocument>
  };
}

export class LambdaFunction extends cdk.Construct {
  private readonly _functionArn: string
  private readonly _resource: NodejsFunction
  constructor(scope: Construct, id: string, private param: Props) {
    super(scope, id)
    const policies: Record<string, iam.PolicyDocument> = param.role?.inlinePolicies ?? {}
    if (param.logged) {
      policies['logging'] = new iam.PolicyDocument({
        assignSids: true,
        statements: [
          new iam.PolicyStatement({
            actions: [
              'logs:CreateLogGroup',
              'logs:PutLogEvents',
              'logs:CreateLogStream'
            ],
            effect: iam.Effect.ALLOW,
            resources: ['*']
          })
        ]
      })
    }
    const lamnbdaRole = this.createLambdaRole(
      param.role?.name ?? `Role${generateResourceID()}`,
      param,
      policies
    )

    const func = new NodejsFunction(this, `Lambda${generateResourceID()}Resource`, {
      entry: param.entryFunction,
      handler: 'index.handler',
      timeout: Duration.seconds(10),
      functionName: param.functionName,
      environment: param.env,
      runtime: lambda.Runtime.NODEJS_20_X,
      memorySize: param.memory ?? 128,
      role: lamnbdaRole,
      bundling: {
        externalModules: [
          ...(param.externalDeps || []),
          'utils'
        ],
        sourceMap: true,
        sourceMapMode: SourceMapMode.BOTH
      },
      layers: [
        ...((param.layersArns || [])
          .filter(value => !!value && value.length)
          .map(value => lambda.LayerVersion.fromLayerVersionArn(this, `LayerVersion${generateResourceID()}Resource`, value)))
      ]
    })

    this._functionArn = func.functionArn
    this._resource = func
  }

  get functionArn() {
    return this._functionArn
  }

  get resource() {
    return this._resource
  }

  grantApi = (restApi: aws_apigateway.RestApi) => {
    this._resource.addPermission(`TodoAppLambdaPermissionResource-${randomUUID()}`, {
      action: 'lambda:InvokeFunction',
      principal: new iam.ServicePrincipal('apigateway.amazonaws.com'),
      sourceAccount: restApi.env?.account,
      sourceArn: `arn:${Aws.PARTITION}:execute-api:${restApi.env?.region}:${restApi.env?.account}:${restApi.restApiId}/*/*/*`
    })
  }

  private createLambdaRole(roleName: string, props: StackProps, inlinePolicies?: Record<string, iam.PolicyDocument>): iam.Role {
    return new iam.Role(this, `${roleName}Resource`, {
      roleName: roleName,
      assumedBy: new iam.ServicePrincipal('lambda.amazonaws.com', {
        region: props.env?.region
      }),
      path: '/',
      inlinePolicies: {
        ...inlinePolicies
      }
    })
  }
}

The important things here to know is:

accountRecovery Specifies the method for users to recover their account if they lose their password. In our case, it's set to email only, so the recovery link will be sent to the user via email provided by him.
selfSignUpEnabled Give the habillity to the users (unknown user) to use the Sign-Up functionality to create their accounts.
userInvitation The message template (subject and body) sent to the user when their account is created by an admin. In the emailBody, {username} and {####} will be replaced by the user's username and the verification code, respectively.
signInAliases A list of methods by which a user can register or sign in to a user pool. It allows either username with aliases or sign-in with email, phone, or both. In our case, the user can sign in using either their email or username.
customAttributes Additional attributes for the user. Note that each attribute here will be represented in the Cognito schema as custom:<ATTR_NAME>, so there's no need to prefix these attributes with custom.
email The email settings for the User Pool. There are two options: Cognito or SES. Since we chose the Cognito option, we provide the no-reply address (sender name )

In the code above, we added a Lambda Trigger of type CUSTOM_MESSAGE to customize the email content for a more appealing appearance. This trigger activates when the verification code email is sent to the user after sign-up.



userPool.addTrigger(
    cognito.UserPoolOperation.CUSTOM_MESSAGE,
    this.customEmailMessageLambda()
)

The rendering will be hardcoded in the provided Lambda and will look like this.

You can find the entire source code for the function here custom-confirm-message.function.ts↗

Now let's talk about the app client. The important points to note are:

disableOAuth set to false to desable the OAuth interaction.
generateSecret set to true to generate the app client secret.
authFlows Specifies the set of OAuth authentication flows to enable on the client

Note that for security purposes, it is recommended to retrieve the secret at Lambda runtime logic to avoid exposing the secret in the CloudFormation template. For this, pass the secret ARN as an environment variable to the lambda function instead of passing the secret value directly.

As you can see, we also store the app client secret using AWS Secrets Manager with the following snippet of code



const clientSecret = new sm.Secret(this, 'SecretManagerResource', {
      secretName: `${userPool.userPoolId}/${userPoolClient.userPoolClientId}`,
      generateSecretString: {
        secretStringTemplate: JSON.stringify({
          appClientSecret: userPoolClient.userPoolClientSecret.unsafeUnwrap()
        }),
        generateStringKey: `${userPoolClient.userPoolClientName}-AppSecret`
      }
    })

Create Lambda functions for authentication process

In this section, I will create three (03) Lambda functions for Sign-In, Sign-Up, and Confirm Email.

Lambda function for user Sign-Up

apps/functions/auth/register.function.ts



import { APIGatewayProxyEvent, Context, Handler } from 'aws-lambda'
import { LambdaResponse } from '../../src/infra/dto/lambda.response'
import {
  CognitoIdentityProviderClient,
  SignUpCommand,
  SignUpCommandOutput
} from '@aws-sdk/client-cognito-identity-provider'
import moment from 'moment'
import { computeSecretHash, DATETIME_FORMAT, responseError, responseOk, toPayload } from 'utils'
import { SecretsManagerClient } from '@aws-sdk/client-secrets-manager'
import { SecretsManagerRepository } from '../../src/infra/storage/secrets/secrets-manager.repository'

const client = new CognitoIdentityProviderClient({})
const smClient = new SecretsManagerClient({
  region: process.env.REGION || 'us-east-1'
})
const CLIENT_ID = process.env.APP_CLIENT_ID
const SECRET_VALUE_ARN = process.env.SECRET_VALUE_ARN

export const handler: Handler = async (event: APIGatewayProxyEvent, context: Context) => {
  const payload = toPayload(event)
  let responseHandler: LambdaResponse
  if (!validatePayload(payload)) {
    responseHandler = responseError('some Input(s) are invalid', 403)
  } else {
    const attributes = [{
      Name: 'email',
      Value: payload.email
    }, {
      Name: 'custom:domain',
      Value: 'nivekaa.com'
    }, {
      Name: 'custom:first_name',
      Value: payload.firstName
    }, {
      Name: 'family_name',
      Value: payload.lastName
    }, {
      Name: 'given_name',
      Value: payload.firstName
    }, {
      Name: 'custom:last_name',
      Value: payload.lastName
    }, {
      Name: 'custom:created_at',
      Value: moment().format(DATETIME_FORMAT)
    }, {
      Name: 'custom:last_updated_at',
      Value: moment().format(DATETIME_FORMAT)
    }, {
      Name: 'name',
      Value: (!!payload.lastName || !!payload.firstName) ? `${payload.firstName} ${payload.lastName}`.trim() : null
    }]
    let messageError = null
    try {
      const clientSecret = await new SecretsManagerRepository(smClient)
        .getCognitoAppClientSecretValue(SECRET_VALUE_ARN!)
      const command = new SignUpCommand({
        ClientId: CLIENT_ID,
        Username: payload.username,
        Password: payload.password,
        SecretHash: computeSecretHash(CLIENT_ID, clientSecret, payload.username),
        UserAttributes: attributes,
        UserContextData: {
          IpAddress: event.requestContext?.identity?.sourceIp
        }
      })
      const response: SignUpCommandOutput = await client.send(command)
      if (response.$metadata.httpStatusCode !== 200) {
        messageError = `Something wrong: StatusCode=${response.$metadata.httpStatusCode}`
      }
    } catch (e) {
      console.error(e)
      messageError = e.message
    }
    if (messageError !== null) {
      return responseError(messageError)
    } else {
      return responseOk({}, 'User successfully created')
    }
  }
  return responseHandler
}
const validatePayload = (body: any): boolean => {
  const email = body.email
  const password = body.password
  const username = body.username
  const areNotNull = [email, password, username].every(value => !!value)
  const isEmailValid = /^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(email)
  const isValidPassword = /^(?=.*[0-9])(?=.*[a-z])(?=.*[A-Z])(?=.*\W)(?!.* ).{8,16}$/.test(password)
  return areNotNull && isEmailValid && isValidPassword
}

Considering that the SECRET_VALUE_ARN and APP_CLIENT_ID were passed to the Lambda resource during the infrastructure construction, remember that in the previous section, I stored the user pool app client secret in Secrets Manager. Below is the code to retrieve it:



// app/src/infra/storage/secrets/secrets-manager.repository.ts
import { GetSecretValueCommand, SecretsManagerClient } from '@aws-sdk/client-secrets-manager'
export class SecretsManagerRepository {
  constructor(private client: SecretsManagerClient) {
  }
  getCognitoAppClientSecretValue = async (secretArn: string): Promise<string> => {
    const command = new GetSecretValueCommand({
      SecretId: secretArn
    })
    try {
      const response = await this.client.send(command)
      if (response.$metadata.httpStatusCode === 200) {
        const json = JSON.parse(response.SecretString!)
        return json.appClientSecret + ''
      }
    } catch (e) {
      console.error(e)
    }
    throw new Error('Technical error. Please Contact the admin or try again later')
  }
}

And below is the content of function to compute the SecretHash parameter



import { createHmac } from 'node:crypto'
....
export const computeSecretHash = (clientId: string, clientSecret: string, username: string): string => {
  const hasher = createHmac('sha256', clientSecret + '')
  hasher.update(`${username}${clientId}`)
  return hasher.digest('base64')
}

For more information about the client-secret hash value see aws docs↗

Lambda function for user Sign-In

apps/functions/auth/login.function.ts



import { APIGatewayProxyEvent, Context, Handler } from 'aws-lambda'
import { computeSecretHash, responseError, toPayload } from 'utils'
import {
  AuthFlowType,
  CognitoIdentityProviderClient,
  InitiateAuthCommand
} from '@aws-sdk/client-cognito-identity-provider'
import { SecretsManagerClient } from '@aws-sdk/client-secrets-manager'
import { SecretsManagerRepository } from '../../src/infra/storage/secrets/secrets-manager.repository'
const client = new CognitoIdentityProviderClient({
  region: process.env.REGION || 'us-east-1'
})
const smClient = new SecretsManagerClient({
  region: process.env.REGION || 'us-east-1'
})
const CLIENT_ID = process.env.APP_CLIENT_ID
const SECRET_VALUE_ARN = process.env.SECRET_VALUE_ARN

export const handler: Handler = async (event: APIGatewayProxyEvent, context: Context) => {
  const payload = toPayload(event)
  if (!validatePayload(payload)) {
    return responseError('Invalid input(s).', 403)
  }
  try {
    const clientSecret = await new SecretsManagerRepository(smClient)
      .getCognitoAppClientSecretValue(SECRET_VALUE_ARN!)
    const command = new InitiateAuthCommand({
      ClientId: CLIENT_ID,
      AuthFlow: AuthFlowType.USER_PASSWORD_AUTH,
      AuthParameters: {
        SECRET_HASH: computeSecretHash(CLIENT_ID!, clientSecret, payload.username),
        USERNAME: payload.username,
        PASSWORD: payload.password
      },
      UserContextData: {
        IpAddress: event.requestContext?.identity?.sourceIp
      }
    })
    const response = await client.send(command)
    if (response.$metadata.httpStatusCode !== 200) {
      return responseError(`Something wrong!! Status=${response.$metadata.httpStatusCode}`)
    }
    return {
      isBase64Encoded: false,
      statusCode: 200,
      headers: {
        Authorization: `Bearer ${response.AuthenticationResult?.IdToken}`
      },
      body: JSON.stringify({
        message: 'user logged!!',
        idToken: response.AuthenticationResult?.IdToken,
        accessToken: response.AuthenticationResult?.AccessToken,
        tokenType: response.AuthenticationResult?.TokenType,
        expiredIn: response.AuthenticationResult?.ExpiresIn,
        refreshToken: response.AuthenticationResult?.RefreshToken
      })
    }
  } catch (e) {
    console.error(e)
    return responseError(e.message)
  }
}

const validatePayload = (payload: any): boolean => {
  return (payload.hasOwnProperty('username') && payload.hasOwnProperty('password'))
    && [payload.username, payload.password].every(value => !!value)
}

This function initiates the authentication process by generating the Id and Access tokens that will be used by external systems to access API resources.

Lambda function for Confirm Email

This function will be called when the user clicks on theActivate Account button inside the verification email. We need the code (generated by Cognito during the sign-up process) and username for this process.



import { APIGatewayProxyEvent, Context, Handler } from 'aws-lambda'
import { CognitoIdentityProviderClient, ConfirmSignUpCommand } from '@aws-sdk/client-cognito-identity-provider'
import { computeSecretHash, responseError, responseOk, toPayload } from 'utils'
import { SecretsManagerClient } from '@aws-sdk/client-secrets-manager'
import { SecretsManagerRepository } from '../../src/infra/storage/secrets/secrets-manager.repository'
const client = new CognitoIdentityProviderClient({})
const smClient = new SecretsManagerClient({
  region: process.env.REGION || 'us-east-1'
})
const CLIENT_ID = process.env.APP_CLIENT_ID
const SECRET_VALUE_ARN = process.env.SECRET_VALUE_ARN
const DOMAIN = process.env.DOMAIN

export const handler: Handler = async (event: APIGatewayProxyEvent, context: Context) => {
  const payload = toPayload(event)
  if (!validatePayload(payload)) {
    return responseError('Invalid input(s)', 403)
  }
  if (payload.hasOwnProperty('email') && payload.email && DOMAIN !== payload.email.split('@')[1]) {
    return responseError('The email is wrong.')
  } else {
    try {
      const clientSecret = await new SecretsManagerRepository(smClient)
        .getCognitoAppClientSecretValue(SECRET_VALUE_ARN!)
      const cmd = new ConfirmSignUpCommand({
        ConfirmationCode: payload.code + '',
        ClientId: CLIENT_ID,
        Username: payload.username,
        SecretHash: computeSecretHash(CLIENT_ID!, clientSecret, payload.username)
      })
      const response = await client.send(cmd)
      console.log(response)
      if (response.$metadata.httpStatusCode === 200) {
        return responseOk('Email confirmed')
      }
      return responseError('Something unexpected wrong!', response.$metadata.httpStatusCode)
    } catch (e) {
      console.error(e)
      return responseError(e.message)
    }
  }
}

const validatePayload = (payload: any): boolean => {
  return (payload.hasOwnProperty('username') && payload.hasOwnProperty('code'))
    && [payload.username, payload.code].every(value => !!value)
}

Add methods for authentication process

Now that all the functions for the authentication process are ready, let's add methods for them to the API Gateway.

stacks/cdk-stack.ts



const cors: api.CorsOptions = {...}
const layerArn = ...
...

const rootResource = restApi.root.addResource('todo-app-api')
const authRoot = restApi.root.addResource('auth')
// Create the authentication endpoints
const loginResource = authRoot.addResource('login', {
    defaultCorsPreflightOptions: cors
})
const registerResource = authRoot.addResource('register', {
    defaultCorsPreflightOptions: cors
})
const confirmResource = authRoot.addResource('confirm-email', {
    defaultCorsPreflightOptions: cors
})
// Add authentication methods with each lambda integration
const cognitoInlinePolicy = {
    cognito: new iam.PolicyDocument({
    assignSids: true,
    statements: [
        new iam.PolicyStatement({
        effect: iam.Effect.ALLOW,
        actions: [
            'cognito-idp:ConfirmDevice',
            'cognito-idp:ChangePassword',
            'cognito-idp:ConfirmForgotPassword',
            'cognito-idp:ChangePassword',
            'cognito-idp:ConfirmSignUp',
            'cognito-idp:ForgotPassword',
            'cognito-idp:GetUser',
            'cognito-idp:GetUserAttributeVerificationCode',
            'cognito-idp:ListUsers',
            'cognito-idp:SignUp',
            'cognito-idp:UpdateUserAttributes'
        ],
        resources: [
            cognito.userPoolArn
        ]
        })
    ]
    }),
    secret: new iam.PolicyDocument({
    assignSids: true,
    statements: [
        new iam.PolicyStatement({
        effect: iam.Effect.ALLOW,
        actions: ['secretsmanager:GetSecretValue'],
        resources: [cognito.secretArn]
        })
    ]
    })
}

const loginFunction = new LambdaFunction(this, 'LoginFunction', {
    functionName: 'Auth-Cognito-Login-NodejsFunction',
    entryFunction: './apps/functions/auth/login.function.ts',
    env: {
        USER_POOL_ID: cognito.userPooId,
        SECRET_VALUE_ARN: cognito.secretArn,
        APP_CLIENT_ID: cognito.clientId,
        REGION: this.props.env?.region!
    },
    logged: true,
    role: { inlinePolicies: cognitoInlinePolicy },
    layersArns: [layerArn],
    externalDeps: Object.keys(dependencies)
})

const registerFunction = new LambdaFunction(this, 'RegistrationFunction', {
    functionName: 'Auth-Cognito-Register-NodejsFunction',
    entryFunction: './apps/functions/auth/register.function.ts',
    env: {
        USER_POOL_ID: cognito.userPooId,
        SECRET_VALUE_ARN: cognito.secretArn,
        APP_CLIENT_ID: cognito.clientId,
        REGION: this.props.env?.region!
    },
    role: {
        name: 'AuthRegistrationRole',
        inlinePolicies: cognitoInlinePolicy
    },
    layersArns: [layerArn],
    externalDeps: Object.keys(dependencies)
})

const confirmUserFunction = new LambdaFunction(this, 'ConfirmUserFunction', {
    functionName: 'Auth-Cognito-ConfirmUser-NodejsFunction',
    entryFunction: './apps/functions/auth/confirm-user.function.ts',
    env: {
        USER_POOL_ID: cognito.userPooId,
        SECRET_VALUE_ARN: cognito.secretArn,
        APP_CLIENT_ID: cognito.clientId,
        DOMAIN: this.props.cognito?.domain!,
        REGION: this.props.env?.region!
    },
    logged: true,
    role: {
        name: 'AuthConfirmRole',
        inlinePolicies: cognitoInlinePolicy
    },
    layersArns: [layerArn],
    externalDeps: Object.keys(dependencies)
})

// Allow each lambda to be invoked by API Gateway
Array.of(loginFunction, registerFunction, confirmUserFunction)
      .forEach(func => {
    func.resource.addPermission(`APIGW-Permission-${randomUUID()}`, {
        principal: new iam.ServicePrincipal('apigateway.amazonaws.com'),
        sourceArn: `arn:${Aws.PARTITION}:execute-api:${this.props.env?.region}:${this.props.env?.account}:${restApi.restApiId}/*/*/*`,
        sourceAccount: this.props.env?.account,
        scope: this,
        action: 'lambda:InvokeFunction'
    })
})

const loginMethod = auth.loginResource.addMethod(MediaType.POST, new api.LambdaIntegration(loginFunction.resource, {
    proxy: true
}), {
    authorizationType: api.AuthorizationType.NONE,
    methodResponses
})

const registerMethod = auth.registerResource.addMethod(MediaType.POST, new api.LambdaIntegration(registerFunction.resource, {
    proxy: true
}), {
    authorizationType: api.AuthorizationType.NONE,
    methodResponses
})

const confirmUserMethod = auth.confirmResource.addMethod(MediaType.POST, new api.LambdaIntegration(confirmUserFunction.resource, {
    proxy: true
}), {
    methodResponses,
    authorizationType: api.AuthorizationType.NONE
})

⚠️⚠️ Pay attention to the Lambdas set up above. Since they call Cognito and Secrets Manager, it is necessary to grant them the appropriate permissions to perform their operations without causing issues. Additionally, you’ll notice that API Gateway is granted permission to invoke all of these Lambda functions.

Create Api Gateway Authorizer

Before creating the Cognito Authorizer, it is imperative to instantiate our construct created in the User Pool section, because the authorizer needs its ARN.



const cognitoUserPool = new Cognito(this, 'CustomCognitoResource', {
    verificationEmailUrl: 'https://localhost:4200/auth/confirm-email',
    fromEmail: props.cognito?.verificationFromEmail!
})

Now that the user pool is constructed, we can create the Cognito authorizer and attach it directly to the User Pool we previously created.



const cognitoAuthorizer = new api.CognitoUserPoolsAuthorizer(this, 'AuthorizerResource', {
    authorizerName: 'todo-app-authorizer',
    cognitoUserPools: [cognitoUserPool.userPool],
    identitySource: api.IdentitySource.header('authorization')
})

api.IdentitySource.header('authorization') will return the string 'method.request.header.Authorization'

Update the Rest Api Methods

In my last article (Day 008↗), I had created insecure APIs. Now, let's update these resources by securing them with the Cognito authorizer that we previously created.



const getTodoListsMethod = getTodoListsResource.addMethod(MediaType.GET, new api.LambdaIntegration(lambdaFunction.resource, {
    connectionType: ConnectionType.INTERNET,
    proxy: true
}), {
    authorizationType: api.AuthorizationType.COGNITO,
    authorizer: cognitoAuthorizer,
    methodResponses
})

const createTodoListMethod = createTodoListResource.addMethod(MediaType.POST, new api.LambdaIntegration(lambdaFunction.resource, {
    proxy: true,
    connectionType: ConnectionType.INTERNET
}), {
    authorizer: cognitoAuthorizer,
    authorizationType: api.AuthorizationType.COGNITO,
    methodResponses
})
const updateTodoListMethod = updateTodoListResource.addMethod(MediaType.PUT, new api.LambdaIntegration(lambdaFunction.resource, {
    proxy: true,
    connectionType: api.ConnectionType.INTERNET
}), {
    authorizer: cognitoAuthorizer,
    authorizationType: api.AuthorizationType.COGNITO,
    methodResponses
})
const deleteTodoListMethod = deleteTodoListResource.addMethod(MediaType.DELETE, new api.LambdaIntegration(lambdaFunction.resource, {
    connectionType: api.ConnectionType.INTERNET,
    proxy: true
}), {
    authorizer: cognitoAuthorizer,
    authorizationType: api.AuthorizationType.COGNITO,
    methodResponses
})

Now let's test one of these APIs to verify that the security integration is working correctly.

I received a 401 Unauthorized error, indicating that I need to provide either the Access or ID token.

I copied the idToken, included it in the header of my request, and tried again. This time, I received:

Updated the Angular App with authentication proccess

The app integration involves passing the IdToken generated during the sign-in process in the header of each request, formatted as Authorization: Bearer <ID_TOKEN>



export const authorizerInterceptor = (req: HttpRequest<unknown>, next: HttpHandlerFn): Observable<HttpEvent<unknown>> => {
  const tokenAuth = inject(MEMORY_TOKEN_REPOSITORY)
  const url = req.url
  if (!url.includes('/auth/') || url.endsWith('/auth/details')) {
    if (tokenAuth.hasToken()) {
      const reReq = req.clone({
        setHeaders: {
          Authorization: `Bearer ${tokenAuth.getToken()}`
        }
      })
      return next(reReq)
    }
  }
  return next(req)
}

You can find the complete code for the Angular app here↗

Deploy the entire infrastructure



git clone https://github.com/nivekalara237/100DaysTerraformAWSDevops.git

export STAGE_NAME=dev # needed by api gateway staging

cd 100DaysTerraformAWSDevops/day_009
cdk deploy --profile cdk-user --all

Re-deploy app as S3 static website

To deploy the Angular app to the S3 bucket as a website, follow the instructions in my previous article 👉🏽👉🏽 Deploying a REST API and Angular Frontend Using AWS CDK, S3, and API Gateway
⚠️⚠️ Make sure to update the API Gateway URL in the src/environment.ts file with the correct API Gateway endpoint before building and deploying the application.



export const environment = {
  production: true,
  apiUrl: 'https://{{API_GATEWAY_ID}}.execute-api.us-east-1.amazonaws.com/{{STAGE_NAME}}/'
}

🥳✨
We have reached the end of the article.
Thank you so much 🙂

Your can find the full source code on GitHub Repo

Forem: Kevin Lactio Kemta

How to combine SQS and SNS to implement multiple Consumers (Part 2)

Day 020 - 100DaysAWSIaCDevopsChallenge

Scenario

Producer & Consumers

User Management App (producer)

Search Indexing App (consumer)

Code Generation App (consumer)

Build infrastructure using CDK with java as language

Deployment

Why you Shouldn't Use Amazon SQS for multiple Consumers—Choose Amazon SNS Instead! (Part 1)

Day 019 - 100DaysAWSIaCDevopsChallenge

Amazon SQS vs Amazon SNS

Which one to choose ?

Conclusion

Create an AWS SNS Topic Using CDK and consume messages with a Spring Boot Microservice

Day 018 - 100DaysAWSIaCDevopsChallenge

Subscribe and Consume a SNS Messages using Springboot application

Create the Infrastructure

VPC & Subnet + Internet Gateway

SNS Topic

EC2 Instance for Hosting the Spring Boot Application

Create the stack

Create SpringBoot Subscriber Application

Deployment

Resut

nivekalara237 / 100DaysTerraformAWSDevops

Thriving with AWS and Terraform

Building a Spring Boot Consumer Application with Amazon SQS: Setup Infrastructure Using Cloud Development Kit (CDK)

Day 017 - 100DaysAWSIaCDevopsChallenge

What is Amazon SQS ?

Consume a SQS Messages using Springboot application

Create the Infrastructure

VPC & Subne + Internet Gateway

SQS Queue & Dead Letter Queue

EC2 Instance for Hosting the Spring Boot Application

Create the stack

Create SpringBoot Consumer Application

Deployment

Resut

Deploying SonarQube on AWS EC2 with Cloud Development Kit(CDK): Extending the Database to Amazon RDS PostgreSQL

Day 016 - 100DaysAWSIaCDevopsChallenge

Why did I need to extend the SonarQube Database to a RDS PostgreSQL Instance?

Update the previous infrastructure to setting up a RDS PostgreSQL Instance for SonarQube

Create the database Construct

Update the Stack

Deploy the stack

Step-by-Step Guide to Setting Up and Deploying SonarQube on an AWS EC2 Instance Using AWS Cloud Development Kit (CDK)

Day 015 - 100DaysAWSIaCDevopsChallenge

What is SonarQube ?

Why do you need to use it in your application ?

how to install SonarQube on an AWS EC2 Instance?

Create the CDK Stack

VPC, Subnet and Internet Gateway

Security group

SonarQube Web Server (EC2 Instance) + KeyPair

Deploy the stack

Create Test Coverage Visualizer and Deploy to AWS S3 with Cloud development Kit (CDK) and GitHub Actions

Day 014 - 100DaysAWSIaCDevopsChallenge

Why Centralize Test Coverage Reporting?

Flow diagram

1. Create and AWS infrastructure for hosting the coverage visualizer

1.a S3 Bucket

1.b. CloudFront & Route 53 Subdomain configuration

1.c. Visualizer Stack

1.e. Deploy infrastructure

2. Set Up the GitHub Actions job to upload test coverage file to S3 Bucket

Effortless Automation: Configuring CI/CD for NPM Library Publishing with GitHub Actions

Day 013 - 100DaysAWSIaCDevopsChallenge

Flow diagram

Create a GitHub Workflow to automate Linting, building, and Testing

Github Actions trigger

Jobs: linting code

Jobs: Testing

Jobs: Building

Create the second Workflow to automate the Release creation and Publishing proccess

Github Action trigger

Jobs: Github releasing

Jobs: Github publishing

Configure Github Actions

User Management App (`producer`)

Search Indexing App (`consumer`)

Code Generation App (`consumer`)

Create a GitHub Workflow to automate `Linting`, `building`, and `Testing`

Create the second Workflow to automate the `Release creation` and `Publishing proccess`

Update `README.md` by adding the build badges status

Infrastructure Diagram (`updated`)

Create a subdomain - RecordSet of type `CNAME`