Forem: Piero Bozzolo

Configuring API Gateway for Secure S3 File Uploads: A Step-by-Step Guide

Piero Bozzolo — Mon, 26 Aug 2024 13:42:34 +0000

This is a reviewed version of the same previously published article on my medium page

To allow an API Gateway to upload files directly to S3, you need to create an IAM role that can be assumed by the API Gateway and has the necessary permissions to write objects to S3.

Step 1: Create the IAM Role

Open the IAM service in your AWS console.
Navigate to “Roles” and click “Create role.”
Select “AWS service” under “Trusted entity type,” then choose “API Gateway” under “Use cases for other AWS services.”
Click “Next” to proceed.

Step 2: Set Permissions for the Role

After creating the role, name it (e.g., “api-gateway-s3-upload”) and click “Create role.”
To add permissions, open the newly created role from the IAM Roles list.
Under the “Permissions” tab, click “Add permissions” -> “Attach policies.”

Filter by “S3” and select AmazonS3FullAccess.

Note: While AmazonS3FullAccess grants full S3 access, it does not adhere to the principle of least privilege. Consider creating a custom policy that only allows the PutObject action on the specific S3 bucket.

Step 3: Fine-Tuning the Permissions

To ensure compliance with the least privilege principle:

Remove the AmazonS3FullAccess policy by selecting it and clicking “Remove.”
Add a new inline policy by clicking “Add Permissions” -> “Create inline policy.”

In the JSON tab, insert the following policy:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "s3:PutObject",
    "Resource": "arn:aws:s3:::your-bucket-name/*"
  }]
}

Review the policy, assign it a unique name, and create it.

Conclusion

You've successfully created an IAM role with a custom policy that allows API Gateway to upload files to S3 securely. In the next part of this series, we'll explore how to configure an API that leverages this role for S3 operations.

Extreme Programming Meets the Cloud: How Serverless Would Have Been XP's BFF

Piero Bozzolo — Thu, 15 Aug 2024 15:33:34 +0000

Last weekend, I was getting ready for a trip in my camper, knowing I'd be without an internet connection. As I passed by my bookshelf, I quickly grabbed an old book to have something to read, and I happened to pick up "Extreme Programming Explained" by Kent Beck. Over the weekend, I reread after almost 15 years the first few chapters and couldn't help but think about how perfectly the serverless world aligns with an XP approach.

Why Serverless is Perfect for Extreme Programming

In recent years, Extreme Programming (XP) has become a cornerstone of agile development, thanks to its focus on code quality and its ability to quickly adapt to changes. But if you want to truly maximize the benefits of XP, choosing the right infrastructure is key. And that's where serverless comes in.

Rapid Feedback: One of the core aspects of XP is getting continuous feedback through frequent, short release cycles. With serverless, you can deploy new features in real-time and get immediate feedback from users. You don’t have to worry about the infrastructure, which means you can release updates more often and at a lower cost.

Focus on Code: XP promotes simple design and constant code refactoring. Serverless lets you focus entirely on developing features without thinking about the underlying infrastructure. This reduces complexity, keeping the code clean and easy to manage.

Continuous Integration Made Easier: Continuous integration is a cornerstone of XP, and serverless is the perfect ally here. Thanks to the modularity of serverless functions, you can isolate and test each part of the code independently, reducing the risk of bugs. Moreover, many serverless platforms come with built-in support for CI/CD (Continuous Integration/Continuous Deployment) pipelines, allowing you to automate testing and code deployment. This speeds up development and ensures consistent quality over time. The best part is you don't have to worry about managing the infrastructure—serverless takes care of it all, leaving you free to focus on features and business logic. This makes for a smoother workflow and lets you quickly adapt to changes, in true XP fashion.

Dynamic Scalability: One of XP's strengths is its ability to quickly adapt to change. With serverless, your applications automatically scale based on demand, with no manual intervention needed, which translates to tremendous flexibility in tackling new challenges or adding features.

Reduced Complexity: XP loves simplicity, and serverless is perfect for that. The modular solutions provided by serverless reduce architectural complexity, improve code quality, and make it easier to continuously enhance the system.

In conclusion, if you're using Extreme Programming, serverless is the ideal companion. It helps you stay agile, improve code quality, and respond quickly to changes, perfectly embodying the principles of XP. And let’s be honest, if serverless had been available during XP's heyday, I'm sure it would have become one of the favorite tools of early XP enthusiasts, thanks to its ability to support a fast, iterative development approach.

Dynamically Choosing Origin Based on Host Header and Path with AWS CloudFront and Lambda@Edge

Piero Bozzolo — Thu, 27 Jun 2024 17:41:42 +0000

In a recent project for our customer Skillgym, I faced the challenge of serving content from different origins based on specific conditions, like the host subdomain. The goal was to leverage AWS CloudFront to distribute content efficiently while ensuring the flexibility to choose between multiple origins based on the host header and request path. This post will walk you through the solution, highlighting the differences between CloudFront Functions and Lambda@Edge, and how we used Lambda@Edge to dynamically select origins based on subdomains.

The Challenge

The primary challenge was to configure CloudFront to serve content from different origins based on:

Specific domains or subdomains
Different paths within each domain/subdomain

For example, we needed to route traffic to an S3 bucket for static assets on a specific path for a particular domain. This means that requests to www.example.com/images/test.png should be directed to the S3 object at s3://my-bucket/example.com-content/public/images/test.png. At the same time, we used an Application Load Balancer (ALB) to handle dynamic content. So, requests to www.company-website.com/images/dynamic-image.png should be directed to the API served by the ALB.

Solution Overview

The solution involves:

Configuring an S3 bucket and CloudFront distribution.
Writing Lambda@Edge functions to dynamically choose the origin.
Setting up CloudFront Functions for lightweight request manipulation.

CloudFront Functions vs. Lambda@Edge

CloudFront Functions are designed for lightweight, short-duration tasks such as simple HTTP header manipulation. They execute very quickly and are cost-effective for simple tasks. CloudFront Functions are priced based on the number of function invocations.

As of the current pricing model:

First 2,000,000 invocations each month are free.
$0.10 per 1,000,000 invocations thereafter.

They are limited to basic transformations and do not support accessing external services or executing complex logic.

Lambda@Edge, on the other hand, offers more power and flexibility. It allows for more complex logic and can interact with external services, making it suitable for tasks such as dynamically choosing origins based on request attributes.

Lambda@Edge functions are priced based on:

Invocations: $0.60 per 1,000,000 invocations.
Duration: $0.00000625125 for every GB-second used.

If the requested content is already cached and the cache is still valid, the origin-request Lambda@Edge function will not be invoked. This means Lambda@Edge only executes when CloudFront needs to go to the origin to fetch the content.

Implementation Details

We will define two origins, but only one cache behavior. In the function associated with the viewer-request trigger, we will manipulate the incoming request. Then, in the function associated with the origin-request trigger, we will change the origin based on the host header. This setup allows us to dynamically select the appropriate origin for each request while maintaining a consistent caching strategy.

Terraform Configuration

Here's the Terraform configuration to set up the necessary AWS resources.

It’s important to note that the provided Terraform code is far from complete. Additional configurations are required to fully set up the S3 bucket, Origin Access Control (OAC), and other distribution parameters. Specifically, detailed bucket definitions, complete OAC settings, and comprehensive CloudFront distribution parameters must be included to ensure a fully functional and secure content delivery setup. These additional configurations are crucial for properly managing permissions, securing access, and optimizing the performance of the CloudFront distributions.

Since the origin request function is executed on the S3 cache behaviour, CloudFront automatically replaces the host header with the S3 host header. To preserve the original host header, we need to save it in a custom header, such as x-original-host. When the origin request function determines that the origin should be the ALB, we can use the value stored in x-original-host to restore the original host header. This allows us to continue using the host header as a rule to select the target group in the ALB, ensuring proper routing and origin selection based on the original request.


resource "aws_cloudfront_distribution" "default" {
  depends_on = [aws_s3_bucket.distribution_bucket]

  origin {
    domain_name              = aws_s3_bucket.distribution_bucket.bucket_regional_domain_name
    origin_access_control_id = aws_cloudfront_origin_access_control.default.id
    origin_id                = local.s3_origin_id
  }

  origin {
    domain_name = var.default_target_origin
    origin_id   = local.main_origin
    custom_origin_config {
      origin_protocol_policy = "match-viewer"
      http_port              = 80
      https_port             = 443
      origin_ssl_protocols   = ["TLSv1.2"]
    }
  }

  enabled         = true
  is_ipv6_enabled = true

  aliases = var.host_subdomains

  default_cache_behavior {
    // ... 
    target_origin_id = local.s3_origin_id
    cache_policy_id = var.default_cache_policy

    viewer_protocol_policy = "redirect-to-https"

    function_association {
      event_type   = "viewer-request"
      function_arn = aws_cloudfront_function.replace-s3-view-path.arn
    }

    lambda_function_association {
      event_type   = "origin-request"
      lambda_arn   = aws_lambda_function.select-origin-function.qualified_arn
      include_body = false
    }
  }


}

Lambda@Edge and CloudFront Functions

Viewer Request Function: This CloudFront function rewrites the URL path based on subdomain mappings and saves the original host header in the x-original-host header.

exports.handler = async (event) => {
    const request = event.Records[0].cf.request;
    const uri = request.uri;
    const host = request.headers.host.value;
    request.headers['x-original-host'] = {
        'value': host
    };

    // Add logic to manipulate URI based on subdomain
    if (uri.startsWith('/old-path')) {
        request.uri = uri.replace('/old-path', '/new-path');
    }

    return request;
};

Origin Request Function: This Lambda@Edge function routes requests to different origins based on the subdomain and path.

exports.handler = async (event) => {
    const request = event.Records[0].cf.request;
    const headers = request.headers;

    const hostHeader = headers.host[0].value;
    const originalHost = request.headers['x-original-host'][0].value;

    const subdomain = hostHeader.split('.')[0];
    // You can add your logic here
    if (subdomain === 'static') {
        request.headers['host'] = [{ key: 'Host', value: 'your-s3-bucket.s3.us-east-1.amazonaws.com'}]


    } else {
        // Store the original host header in a custom header

        const originalHost = request.headers['x-original-host'][0].value;

        request.origin = {
            custom: {
                domainName: 'your-alb.amazonaws.com',
                port: 443,
                protocol: 'https',
                path: '',
                sslProtocols: ['TLSv1.2'],
                readTimeout: 5,
                keepaliveTimeout: 5,
                customHeaders: {}
            }
        };
        // Set the host header to the original request host
        request.headers['host'] = [{ key: 'host', value: originalHost }];
    }

    return request;
};

Bucket security

This approach leverages the Origin Access Control (OAC) feature, as it is the only method that works with this configuration (during the implementation I tested OAI without success).

Conclusion

Selecting the right CDN origin and using Lambda@Edge with CloudFront Functions gives you the flexibility to select the origin with complex logic. Lambda@Edge enables advanced logic execution closer to users, while CloudFront Functions handle lightweight tasks like URL rewrites efficiently.

The Secrets of AWS API Gateway

Piero Bozzolo — Tue, 14 May 2024 15:07:24 +0000

The Secrets of AWS API Gateway

How can an API be secured and exposed to the internet? How the underlying API resource access can be controlled? Is it possible to make an AWS service accessible to external users? Let’s see how API Gateway helps us answer those questions.

Generally speaking, an API Gateway is a management service that lies between an API client and a bunch of backend services. That service can be a container running NGINX, a Lambda function, a general HTTPS server running on EC2, or a group of legacy APIs.

Although the service can centralize and manage many APIs under a unique endpoint, somehow like Application Load Balancer, API Gateway can solve many other contour problems. Let’s see some examples.

Authentication

API Gateway can be configured to filter out unauthorized requests. API Gateway-managed API can be connected to AWS Cognito to handle authentication. This method involves the use of a bearer token included in every HTTP request.

Cognito is not the only option for authentication. A lambda function that checks the token and allows or denies every request can be set as a custom authenticator.

Throttling

How the number of API invocations can be limited per user or per region? Throttling is the right feature and put a limit on invocations preventing overwhelming the application. Also, different thresholds can be set for every consumer using usage plans and API keys.

Data Transformations

Sometimes, when a legacy or a 3rd-party service is used, the request body must be remapped in another format before processing. The same can happen backward: the answer sent from the service to the customer should be changed for compatibility purposes. Using the Velocity Template Language engine, mapping templates are responsible to reconstruct and transform request/response data. Request parameters can be transformed too.

Expose HTTP APIs.

Exposing HTTP API is the core feature of API Gateway. In the microservice era, this is the fundamental feature of a modern cloud-native application, where the architecture consists of lowly coupled and highly specialized services. Every application service can be written in different languages, by different teams or vendors, with custom development and release cycle. API Gateway exposes a unique endpoint for all of them. Moreover, every AWS service can be consumed as an HTTP request, so many of them can be served from API Gateway to your customer, even Lambda Function!

Body validations

One rule exists from ancient times: never trust inputs. When APIs are implemented an input validation technology must be chosen. In the previous paragraph, we highlighted that a modern application can be written in many languages with its validations library. API Gateway can validate inputs on your behalf, keeping the validations in one single point, using one only syntax.

Other features

API Gateway can work also as a cache for HTTP APIs and can mock API for development purposes.

AWS API Gateway types

There are three variants of API Gateway available, what is the right choice for your application?

HTTP API
If a gateway is needed without special features the HTTP API is the right choice. It doesn't support i.e. body transformation, throttling, and many other features BUT is the lowest latency and cheapest solution.

WebSocket API
If a full-duplex communication between the application and clients is needed, WebSocket is the cutting-edge technology that permits that.

REST API
This is the most used variant. Has all the features listed and more. When the developer wants to test API Gateway functionality this can be a good choice due to the advanced testing capability and the ability to import OpenAPI specifications.
REST API can be deployed in three modes:

Edge endpoint. Every request to the gateway will be routed to CloudFront Point of Presence making the application available with low latency all around the world.
Regional endpoint (you can start with this option). The endpoint will be deployed in your application region
Private regional endpoint. Same as Regional Endpoint, but the application will be available only in your Virtual Private Cloud (your network inside AWS)

For more information on what kind of API Gateway should be chosen this documentation page contains a sum of all differences.

In the following post, we will see how to integrate API Gateway with an AWS Service like S3.

Accelerate serverless development cycle with SAM Accelerate

Piero Bozzolo — Tue, 14 May 2024 15:06:52 +0000

Accelerate serverless development cycle with SAM Accelerate

One of the most common objections about serverless is that the development cycle and the test release can be extremely slow.

When you start with a serverless test project the first releases are fast and streamlined, but one may encounter problems once resources are added.

As the project expands we will probably start to need a CI / CD environment. This is an excellent method to release the application code in a staging or production environment. Still, when we have to run the pipeline just for test or to update one single function it can be uncomfortable, awkward, and a time-wasting activity. The pipeline steps can be executed locally from your machine, but you will only save a few seconds. The deployment time grew with the project.

We would need a feature in serverless frameworks that streamlines this process in the development phase.

For some time this feature has existed in the AWS Serverless Application Framework (hereinafter AWS SAM) framework in the form of a beta feature and is now available to everyone.

Let’s take a short step back and try to understand what SAM is and how it works. If you already know SAM you can go directly to the next section.

Serverless Application Model

A serverless application is made up of code and configurations. How can we manage the project and how can we describe my applications so that can be released in multiple environments? We need a tool that packages my code and all his stuff and releases it in the cloud.

In AWS one tool is AWS SAM. AWS SAM is strictly related to AWS CloudFormation with whom it shares syntax and introduces an easy way to create and configure serverless resources (i.e. Lambda Function, Bucket S3, or DynamoDB tables).

Let’s see an example: We need a lambda function invoked when an object is created on an S3 bucket.

Our code folder will contain the following files:

template.yaml #application definitions
onupload_function/app.js #application code

The content of the *template.yam*l will be:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31Resources:ArticlesS3Bucket:
  Type: AWS::S3::BucketOnUploadFunction:
  Type: AWS::Serverless::Function
  Properties:
    Handler: app.lambdaHandler
    Runtime: nodejs14.x
    CodeUri: onupload_function/
    Events:
      OnFileCreated:
        Type: S3
        Properties:
          Bucket: !Ref ArticlesS3Bucket
          Events: s3:ObjectCreated:*

In the onupload_function/app.js we will write a function that reacts to the upload event on the S3 bucket. In the example, we will just log the event payload:

exports.lambdaHandler = async (event, context) =>{
 console.log(JSON.stringify(event));
};

To release our application it will be enough from the console to run the following commands:

$ sam build
$ sam deploy

As you can see many seconds are needed before having the whole infrastructure released. In the deployment phase, you will also choose a CloudFormation stack name to which the application belongs. Keep it noted.

This is a mandatory step that must be completed one time also with the new feature.

How to streamline the deployment phase

As we saw the deployment phase is easy and not complex, but it can only become slow. Every time we add or make changes in the infrastructure the whole code is packaged, uploaded, and released. The build and deployment process can last 20 minutes.

For speeding up the deployment process we can use the new “sam sync” feature which, unlike “sam deploy”, can release only the differences that we have done to the code and to the infrastructure. If we want to use that feature we must complete at least one time the “sam deploy” phase.

Sync also comes with a very handy file watcher that is able to monitor our template and apply the changes when we save the file.

To use sync and its watcher file we need to run this command:

sam sync --stack-name sam-app --watch

the stack name will be the name we gave to the stack in the deployment phase.

Our application will be updated in seconds and resources such as Lambda function, API definitions in API Gateway, Step Function and other infrastructure components will be ready to be truly tested in the AWS infrastructure.

If you are interested in the serverless world, don’t miss the Claranet live shows on Twitch https://www.twitch.tv/claranet_it and the videos on my YouTube channel https://www.youtube.com/channel/UCSC2n9Q4fRacTr0jGOdcO0g

Accelerare il ciclo di sviluppo serverless con SAM Accelerate

Piero Bozzolo — Tue, 14 May 2024 15:04:01 +0000

Accelerare il ciclo di sviluppo serverless con SAM Accelerate

Una delle più importati obbiezioni che mi vengono fatte quando parlo di serverless è che il ciclo di sviluppo e rilascio in test delle app può risultare piuttosto lento.

Spesso si parte con le prime prove nel serverless con un progetto di piccole dimensioni che é semplice e veloce da rilasciare, magari anche dalla propria macchina.

Man mano che il progetto cresce sorge la necessità di fare build e implementare pipeline secondo i principi di CI/CD. Questo è un ottimo modo per rilasciare il codice e l’applicativo, ma può risultare scomodo se devo fare test continui sul codice o se devo validare le modifiche fatte all’infrastruttura. Questo perchè devo attendere che la pipeline compia i passi di test, build, package e deploy dell’applicativo serverless e della sua infrastruttura. Anche senza passare da una pipeline questi passi possono essere eseguiti localmente dalla nostra macchina se stiamo rilasciando l’ambiente di sviluppo. Purtroppo più il progetto cresce più lenta diventa la fase di deploy.

Servirebbe una feature nei framework serverlesss che snellisca questo processo nella fase di sviluppo.

Da qualche tempo questa feature esiste nel framework AWS SAM sotto forma di beta feature e da oggi è disponibile a tutti.

Facciamo un breve passo indietro e cerchiamo di capire cos’è e come funziona SAM. Se già conosci SAM puoi andare direttamente alla sezione successiva.

Serverless Application Model

Un’applicazione serverless è costituita generalmente da codice e configurazioni.
Come gestisco il progetto e come posso descrivere il mio applicativo in modo che possa essere rilasciato su più ambienti? Ho bisogno di uno strumento che pacchettizza il mio codice e tutto il suo contorno e che lo rilasci nel cloud.

In AWS uno degli strumenti a disposizione è SAM o Serverless Application Model. SAM è fortemente imparentato con AWS CloudFormation che ne condivide la sintassi ed introduce la possibilità di configurare risorse serverless (es. Lambda Function, Bucket S3 o tabelle DynamoDB) in modo più semplice.

Facciamo un esempio: vorrei che al caricamento di un file su un bucket S3 venga invocata una funzione lambda.

La cartella del nostro codice sarà popolata con due file:

template.yaml #definizione dell'applicativo
onupload_function/app.js #codice applicativo

Il contenuto del file template.yaml è il seguente:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Resources:

ArticlesS3Bucket:
  Type: AWS::S3::Bucket

OnUploadFunction:
  Type: AWS::Serverless::Function
  Properties:
    Handler: app.lambdaHandler
    Runtime: nodejs14.x
    CodeUri: onupload_function/
    Events:
      OnFileCreated:
        Type: S3
        Properties:
          Bucket: !Ref ArticlesS3Bucket
          Events: s3:ObjectCreated:*

il contenuto di onupload_function/app.js è una funzione che risponde all’evento di caricamento di un file sul bucket S3. Nell’esempio loggeremo semplicemente il contenuto dell’evento:

exports.lambdaHandler = async (event, context) =>{
  console.log(JSON.stringify(event));
};

Per rilasciare il nostro applicativo basterà da console impartire il comando:

$ sam build
$ sam deploy

Come potrete vedere sono necessari diversi secondi prima di aver il codice e l’infrastruttura rilasciata. Nella fase di deploy sceglierete anche il nome dello stack CloudFormation a cui apparterrà l’applicativo. Segnamolo da parte perchè ci servirà dopo.
Questo è un passaggio obbligatorio anche in presenza della nuova feature.

Come velocizzare il processo di deploy

Come abbiamo visto il processo di rilascio non è complicatissimo, ma può rallentare quando aggiungo molte risorse perchè, ad esempio, vengono pacchettizzate ogni volta tutte le funzioni lambda e ne viene creata una nuova versione. Spesso possono trascorrere anche alcuni minuti prima di poter provare il nuovo codice sul cloud.

Per velocizzare il processo possiamo utilizzare la feature sync che a differenza di deploy è in grado di rilasciare solo le modifiche che noi abbiamo fatto al codice o all’infrastruttura. Per poter utilizzare sync dobbiamo aver portato a termine un deploy con successo almeno una volta.

Sync ha anche in dotazione un comodissimo file watcher che è in grado di monitorare il nostro template e applicare le modifiche quando salviamo il file.

Per utilizzare sync e il suo file watcher dobbiamo dare questo comando:

sam sync --stack-name sam-app --watch

lo stack name sarà il nome che abbiamo dato allo stack nella fase di deploy.

Il nostro applicativo sarà aggiornato in pochi secondi e risorse come Lambda function, definizioni di API in API Gateway, Step Function e altri componenti infrastrutturali saranno pronti per essere testati realmente nell’infrastruttura AWS.

Se ti interessa il mondo serverless non perderti le live di Claranet su Twich
https://www.twitch.tv/claranet_it
i video su
https://www.youtube.com/channel/UCSC2n9Q4fRacTr0jGOdcO0g