Forem: Sebastien Napoleon

From cloud noob to AWS re:Invent speaker : my story

Sebastien Napoleon — Tue, 19 Dec 2023 15:10:26 +0000

It's November 29, 2023, 2:30 PM in Las Vegas. I've just had a tattoo done, and here's what I hear: "Please let me welcome Sébastien on the stage, here at re:Invent." I'm not sure if a translation is needed, but here it is anyway: "Thank you for welcoming Sébastien to the stage, here at re:Invent." Before delving into this magical moment, let me tell you how I transitioned back into the cloud and secured a speaking role at the prestigious AWS re:Invent in less than 2 years.

I won't go through the entire history since my inception in the cloud; I've already described that in this article. Let me start a year ago.

The beginning of the story

A year ago, a flame began to grow within me: the flame of sharing. I've always enjoyed sharing, whether through articles, with my colleagues on assignments, or during presentations. But everything started to accelerate from December 2022. While scrolling through LinkedIn, I came across a post promoting a program of a group of tech enthusiasts specializing in AWS. This program is called AWS Community Builder. Since registrations were not yet open, I inquired about the requirements, and bingo, I ticked all the boxes:

Participated in the AWS community (blogging, open-source, presentations,...) - DONE
Have a passion for sharing - DONE
Have a specific expertise - DONE (serverless in my heart)
Passionate about AWS - DONE DONE DONE

So, it was time to join the waiting list to receive a notification when registrations opened. One fine morning, I received an email saying that registrations were open. After filling out a form with a short letter of motivation, my application was submitted! Now, all that was left was to wait because not everyone is accepted as a Community Builder; you have to prove yourself.

A few weeks later, another email arrived: congratulations, you are now a Community Builder! If only I knew that at that precise moment, my career would take a turn. From that point on, you're entitled to several benefits:

A Slack channel with over 3000 enthusiasts, including AWS employees and AWS Heroes
$500 AWS credit just for you
Goodies (swag, as we like to say)
Access to the Community Builders group on dev.to to publish under this banner
And many more!

The most significant aspect here is access to a community of enthusiasts. Picture a Slack filled with 3000 individuals producing content on AWS. It's a real goldmine, but not just that, it's also a fantastic way to expand your network.

So, here I am, a Community Builder. With a bit of time on my hands during my assignments, I take advantage of it to write several articles, prepare Call for Papers (CFP) for the upcoming conference season, and create materials for presentations. I can already deliver these presentations at Ippon and local AWS User Group events.

When speaking becomes a passion

I'm starting to gain some experience with AWS User Groups. I've given 2 talks in Nantes, one for the opening of the Ippon agency in Tours, and 3 internally at Ippon. I feel more and more comfortable speaking in public. One fine morning, I received a message on Slack. It's from Brian Tarbox, an AWS Community Hero and leader of the bAWSton user group. He saw one of the articles I wrote about the new Verified Access service. He asks if I'd be interested in presenting it in a talk for the AWS User Group in Boston (virtually, of course). The format? A 1-hour presentation with 5-10 minutes for questions, all with a little time difference, starting the presentation at 11:00 PM French time. Without hesitation, I accepted the offer, and here I am, a speaker... in English this time!

It's not an easy challenge; you have to ignore your "French accent," get through the first 5 minutes, and then everything flows smoothly. You don't even realize you're giving the presentation in English (except when you can't find your words, of course...). However, this presentation has given me confidence in giving talks in English.

The time for Call for Papers (CFP) arrives, it's the moment to send our famous "abstracts" everywhere we want to go. DevFest Nantes, Lille, ServerlessDay Zurich, Cloud Nord, AWS Community Day Hungary... Then comes the waiting time. The wait is long, but the results come in: rejection, rejection, rejection, rejection... Except for one, I got a slot for the AWS Community Day Hungary, the only conference in English! No problem, this one is also online, with a little twist; the sessions are pre-recorded (I found out on the day of the conference, too bad...). But another valuable experience to add to my speaker's CV. Unfortunately, no "major" French conferences wanted my topics. Too bad, but well, maybe next year!

The selection and the three months before re:Invent

A few weeks pass, and then Jason Dunn, the leader of the AWS Community Builder program, posts a message on our Slack:

"AWS re:Invent is coming! We are fortunate to have slots open for you builders (as well as AWS heroes) this year. Some of you will have the opportunity to have a talk at the Dev Lounge, right in the middle of the expo hall! We will open a Call for Papers (CFP) shortly. Choose up to three topics that you can propose; we will then make a selection and retain a little over a dozen. The format is 20 minutes with or without questions.
What an announcement! I think I've never spent so much time on an abstract; I wanted to make it perfect. In addition to that, the abstract came with a letter of motivation. Why should we send you and not someone else? Because being selected here means you will be invited by AWS to re:Invent, no ticket to buy (except, of course, for the plane, hotel...). Here I am spending almost a day selling myself to the fullest and making this CFP a real teaser to spark interest in hearing the rest of the talk.

The talk selection process then left us waiting for over a month. A month during which I completely forgot that I had participated. It was unthinkable for me that my topic would be selected by AWS. Me, the little cloud newbie, speaking at one of the world's biggest tech conferences? Never, it will never happen. And then, one beautiful evening in the middle of the week, I receive an email:

I'll spare you the details of the following two hours; I called everyone, I was over the moon. So, here I am, a speaker at re:Invent 2023! BOOM! As a bonus, I get to present the topic that interests me the most: event-driven architectures.

The talk preparation

Now that we are speakers, what exactly happens? Well, we begin a lengthy process with AWS internal teams. The first step involves administrative formalities, personal information, and so on.

Nothing too complicated, but it needs to be done. We have access to a timeline to know the key dates. One month before having to submit my presentation, I received the official AWS template for the slides. There's no deviation from the template, no room for variations with copyrighted images (farewell to memes...). However, the template is really nice; I like the colors and fonts used.

One month before departure, it's time to submit the finalized presentation. Quite pleased with my slides, it's now time to practice. The 20-minute format makes training much easier. You don't end up as exhausted as after a 1-hour presentation. After a few solo practice sessions, I extended an invitation to my colleagues in the company for an internal run-through and feedback. Everything goes smoothly; I receive great feedback, and my narrative about a bartender engages well, which is cool!

As a speaker, we're invited to a tips & tricks session on how to capture an audience and ensure that our talk is a true experience for the listener. I learned a lot of great tips, with the best one being to always use the word "YOU." It should be everywhere to ensure that the person listening can relate. Pay attention to posture, gestures...

We also have the opportunity to have a session with a coach, this time personalized. I jumped at the opportunity without knowing exactly what to expect. We then have a 50-minute slot with someone more accustomed to the stage. The experience was really cool; I could do a live demo of my presentation with their highly relevant feedback. One of the feedback points that helped me the most was to engage the audience during the presentation. Asking questions, getting people to raise their hands in the room. Especially at the beginning of the presentation, it helps to hook the audience.

Arriving in Las Vegas and the Pre-Talk Experience

The big day arrives, time to depart. Well, the journey from Nantes to Vegas was long, but after more than 16 hours of travel, I finally made it to Las Vegas. On the Sunday just before the conference kicks off, it's advisable to pick up your badge. So, I set foot for the first time in the Venetian convention center. Armed with my retrieved badge and a little AWS re:Invent speaker and AWS Community Builder pin in my pocket, I headed back to the hotel for some rest before the first day of the conference.

The first day of the conference was fantastic. I won't go into a recap of all the sessions I attended (however, my colleagues have written two great articles on the subject, check them out: recap 1 and recap 2), but one thing struck me: I have never seen an event so well organized. You cannot get lost at re:Invent; it's impossible. Despite being 60,000 people strong, I never had to queue once for lunch. The only line I saw in Vegas was the one to attend my talk (just kidding, although I did have to wait a long time for tattoos!).

Furthermore, on the second day, I had to go register as a speaker. This is a process that needs to be done at least 24 hours before our presentation, and at the latest, up to 4 hours before our slot. It's a large room (there are actually several, one in each major hotel), where you confirm your presence. You also have the opportunity to make 2-3 modifications to your slides. That worked out well; I had a few adjustments to make. After a 30-minute wait, it was my turn. 10 minutes later, I was done, perfect!

Now, I'm all set for my talk tomorrow (theoretically). I had the chance to do 3 more rehearsals these days. I hadn't really practiced standing up without having to touch the mouse or pad to advance the slides. So, I downloaded a little app to simulate a mouse and be able to click from my phone while standing, as if I were addressing a crowd (well, in reality, I was talking to my bed and the 4 pillows in my room at the Horseshoe...).

By the way, if like me, you have problems with jet lag (at least in this direction), the melatonin remedy works wonders for me. From that moment on, I slept like a baby (unlike waking up every hour without it...).

Show time !

What better way to relax than getting a tattoo? So, there are three good hours spent – arriving an hour before the opening to wait in line, waiting to get your slot, and finally, getting the tattoo.

Then we go on to shoot some videos for Ippon and social media, and finally, we head to the meal.

One hour before: we make our way to the Dev Lounge to watch the ongoing sessions. I meet the person accompanying the speakers; he tells me to meet behind the booth 10 minutes before the start of my talk. I comply. Here I am, backstage! The person just before me hands over, and now I have a magnificent microphone around my neck, getting acquainted with the environment (screens down here, on the stand, big presentation screen, the remote control, the stage as seen from here). Backstage again, the person helps us relax, we chat about our tattoos, we talk about everything and nothing, we almost forget that we're about to step on stage (or not)!

And then suddenly, you hear, "We are now pleased to welcome Sebastien on stage, here at re:Invent." It's time to enter the stage. You are gently pushed onto the stage, you grab the remote control, and it's showtime, baby.

I won't lie to you, the first 2 minutes, I was completely disoriented. Some slightly approximate English, and then I got into the groove. I managed to forget the present world (and there were a lot of people, about a hundred, it seems that event-driven architectures attract a crowd). Once that little moment passes, you find your bearings, you relax, you look at people in the audience, you think about articulation, gestures... And 20 minutes later, it's already done. Then, you go back through the backstage, remove the microphone, and decompress. It's time to savor it; we did it! We take the opportunity to grab a beer on the way out because it's now time for the happy hour in the expo hall. For the rest of the evening, as they say, what happens in Vegas stays in Vegas!

To conclude my experience as a speaker and my little journey in the world of AWS cloud: the watchword for me is as follows: try. Nothing ventured, nothing gained as they say! If I had listened to myself, I would never have submitted my CFP for re:Invent. I'm currently writing this conclusion on the return flight from re:Invent, and thankfully, I tried. I undoubtedly had one of the most enriching experiences of my life thanks to my boldness. So, if you also enjoy sharing, if you also want to "accelerate your career," TRY. Find a company that can support you in this path, as Ippon has done. Try to enter a program that suits you (go for the Community Builder program, there aren't enough French participants). You are then the master of your own destiny.

I'll finish with a photo of all the speakers at this year's Dev Chat. Congratulations to everyone; it's certainly an adventure we won't forget. I hope through these few lines, I've also inspired you to try.

Simplified Retrieval of Terraform Output for Ansible via AWS SSM Parameter Store 🚀

Sebastien Napoleon — Tue, 07 Nov 2023 13:24:17 +0000

I recently had to work with Ansible for the first time. I needed to install software on several EC2 instances, and I was given an Ansible playbook that I needed to execute.

When examining the structure of the Ansible folder, I noticed that I had to update certain variables within a file named 'specifics.yml.' These variables were formatted as follows:

nfs_mounts:
  - src: "myNfsDNSname:/"

my_super_infra:
  hosts:
    10.10.10.10:

I initially began working with it. However, as I continued testing, I found myself making numerous deployments with Terraform. Each time, I had to modify my code to include the new EFS DNS name, which proved to be quite a hassle.

For context, I had established a CI/CD pipeline using Gitlab-CI for this project, which had the following structure:

From that point on, I needed to discover a way to dynamically obtain values generated during the Terraform 'apply' stage. Initially, I considered using Terraform to create a file at the end of the 'apply' stage containing the output of my modules. While this approach would work, it would also make the execution of the Ansible job dependent on the completion of the Terraform 'apply' stage. This was not what I wanted; I aimed to run the Ansible playbook without having to go through Terraform each time.

So, I came up with this idea: introduce a Terraform module that stores the necessary variables in the AWS SSM Parameter Store. This module takes certain variables as input, which are the outputs of my modules that I wish to retrieve in Ansible, such as the EFS DNS name. To achieve this, you only need to include the following code snippet for each of your variables:

resource "aws_ssm_parameter" "efs_dns_url" {
  name  = "OUTPUT_EFS_DNS_URL"
  type  = "String"
  value = var.efs_dns_url
}

You can also use count if you have a list (for example a list of IPs) :

resource "aws_ssm_parameter" "my_ip" {
  count = length(var.my_ip)
  name  = "OUTPUT_MY_IP_${count.index}"
  type  = "String"
  value = var.my_ip[count.index]
}

With this step, the initial part of my strategy was set up. The next challenge was to retrieve these values within my Ansible 'specifics.yml' file. As I delved into the documentation, I came across a page in the Ansible documentation that was precisely what I needed! The first step was to include the requirements in the 'requirements.yml' file:

  - name: amazon.aws
    version: 6.5.0

Then, you remember the specifics.yml ? it now looks like this :

nfs_mounts:
    - src: "{{ lookup('amazon.aws.aws_ssm', 'OUTPUT_EFS_DNS_URL', region='eu-central-1' ) }}:/"

It's as straightforward as that! You can now fetch specific values from Terraform directly into your Ansible playbook. This is incredibly convenient. The only drawback I've encountered here is the output when you execute the playbook, particularly when you utilize it for the host section, like:

my_block:
  hosts:
    "{{ lookup('amazon.aws.aws_ssm', 'OUTPUT_MY_IP_0', region='eu-central-1' )}}":
    "{{ lookup('amazon.aws.aws_ssm', 'OUTPUT_MY_IP_1', region='eu-central-1' )}}":

When you run the playbook with this kind of setup, you get an output which looks like this :

This is not ideal, you can change it using variables, but I haven’t found a way yet to get the IP here instead of a placeholder. If you have an idea, feel free to reach out to me via the comments or on my socials.

Automatically start and stop EC2 instance leveraging EventBridge Scheduler

Sebastien Napoleon — Wed, 16 Aug 2023 12:30:25 +0000

FYI: All of the code mentioned in this article can be found here

During one of my recent assignments, I faced a requirement to schedule the start and stop of EC2 instances. The challenge was to design a mechanism that would be user-friendly and require minimal maintenance.

The specifications outlined that users should be able to initiate the starting and stopping of a specific EC2 instance through an API, based on provided parameters. A route like /provision using the HTTP POST method was chosen for this purpose. Parameters would be passed in the request body in JSON format:

start_time (timezone-less datetime): The date and time when the specified action should start for the EC2 instance, without considering timezones.
end_time (timezone-less datetime): The date and time when the specified action should end for the EC2 instance, without accounting for timezones.
instance_id (EC2 instance identifier): The unique identifier assigned to the EC2 instance.

The AWS API is sufficient for starting and stopping EC2 instances. This task can be easily achieved with a Lambda function written in Python, using the Boto3 SDK.

Now, triggering these Lambdas at specific dates and times is the next challenge. For this, the EventBridge scheduler is the perfect fit.

To implement this, our API route will create two rules within EventBridge. The first rule will activate a Lambda function to start the EC2 instance, and the second to stop it. However, there's a potential problem with this approach: What happens to the rules we create that may have already been triggered?

Not long ago, AWS introduced a solution to this dilemma – the capability to automatically delete rules post-trigger. You can read more about it in this article: https://aws.amazon.com/blogs/compute/automatically-delete-schedules-upon-completion-with-amazon-eventbridge-scheduler/

Having conceptualized our architecture, it's time to dive in. Here's a brief outline, represented as a diagram:

In this scenario, we'll primarily use AWS and Python as our programming language. For deploying our infrastructure, we'll use code, specifically Terraform.

Hands-on

Let's begin with the first step - two Lambdas: one to start and the other to stop.

Here's the Lambda function to start one or more EC2 instances:

import boto3

def lambda_handler(event, _context):
    region = 'eu-west-3'
    ec2 = boto3.client('ec2', region_name=region)
    ec2.start_instances(InstanceIds=event["instances"])
    for instance in event["instances"]:
        print('Started your instances: ' + instance)

And here's the function to stop one or more EC2 instances:

import boto3


def lambda_handler(event, _context):
    region = 'eu-west-3'
    ec2 = boto3.client('ec2', region_name=region)
    ec2.stop_instances(InstanceIds=event["instances"])
    for instance in event["instances"]:
        print('Stopped your instances: ' + instance)

Once our instance management code is set up, the next task is creating our rules in EventBridge. We'll utilize a third Lambda function for this. This function will generate these rules using the AWS SDK, eliminating the need for manual intervention.

import json
import os
import boto3

def lambda_handler(event, _context):
    body = json.loads(event["body"])
    region = os.environ["REGION"]
    start_time = body["start_time"]
    end_time = body["end_time"]
    ec2_server_id = body["ec2_instances_id"]
    scheduler = boto3.client('scheduler', region_name=region)
    scheduler.create_schedule(
        ActionAfterCompletion="DELETE",
        FlexibleTimeWindow={"Mode": "OFF"},
        Name="ec2-schedule-start-" + str(ec2_server_id),
        ScheduleExpression="at(" + start_time + ")",
        ScheduleExpressionTimezone="Europe/Paris",
        Target={
            "Arn": os.environ["START_EC2_LAMBDA_ARN"],
            "RoleArn": os.environ["EXECUTE_SCHEDULE_ROLE_ARN"],
            "Input": "{\"instances\": [\"" + ec2_server_id + "\"]}"
        },

    )

    scheduler.create_schedule(
        ActionAfterCompletion="DELETE",
        FlexibleTimeWindow={"Mode": "OFF"},
        Name="ec2-schedule-stop-" + str(ec2_server_id),
        ScheduleExpression="at(" + end_time + ")",
        ScheduleExpressionTimezone="Europe/Paris",
        Target={
            "Arn": os.environ["STOP_EC2_LAMBDA_ARN"],
            "RoleArn": os.environ["EXECUTE_SCHEDULE_ROLE_ARN"],
            "Input": "{\"instances\": [\"" + ec2_server_id + "\"]}"
        },
    )

    return {
        'statusCode': 200,
        'headers': {
            'Content-Type': 'application/json',
            'Access-Control-Allow-Origin': '*'
        },
        'body': json.dumps({
            'success': True
        }),
        "isBase64Encoded": False
    }

For the Terraform code, please refer to this GitHub link where I've made my code public. Using this, you can deploy all components to your AWS account (follow the README for instructions).

After deployment, you'll have an API in the API Gateway with a /provision route. You can access this with a POST request, using the following body:

{
    “ec2_instances_id“: “xxxxxxxxxx“,
    “start_time”: “2023-04-01T05:00:30”,
    “end_time”: “2023-04-01T07:00:30” 
}

This API call facilitates the creation of two rules within the EventBridge Scheduler. One rule will start your application at the time specified in the start_time field, while the other will halt your instance at the time indicated in the end_time field.

This streamlined setup makes orchestrating one or many EC2 instances incredibly straightforward. API-based interaction is immensely convenient, and the automated rule cleanup post-execution is a significant advantage.

However, exercise caution. Currently, this stack doesn't manage "concurrency". For instance, if one event starts at 3:00 PM and ends at 6:00 PM, you can still have another event that begins at 5:00 PM and concludes at 8:00 PM. This overlap could result in the instance stopping at 6:00 PM during the second event. Pairing this stack with an app like a scheduler or calendar would be optimal (spoiler alert: this integration is part of my ongoing project).

And there you have it! I hope you found this article helpful!

How to optimize your lambda functions with AWS Lambda power tuning

Sebastien Napoleon — Thu, 13 Apr 2023 09:53:41 +0000

Whether you're a beginner or experienced with AWS, optimizing lambda functions is usually not the starting point for a serverless project. That's normal - at the beginning of a project, the priority is to get something functional, not necessarily optimized (at least, not right away). As a result, you often end up with one or more lambda functions that run, but could run either faster or for less cost!

I have the opportunity to work on a personal project with a 100% serverless architecture. In this project, I have several functions that make API calls in parallel. We're talking about functions that make thousands, if not tens of thousands of calls. At the very beginning, we were close to the time quotas for a lambda in terms of execution time (15 minutes). So, we first added multithreading to manage the API calls (instead of doing them one by one, we now do them in parallel). We were able to cut the time in half, or even by a factor of three for certain functions. And with lambdas, time is money.

Being satisfied with the improvement, we decided to leave the functions as they were and focus on other features. After a few months, I noticed that we had the same default memory on each of our lambdas (1024 MB), and I asked myself the following question: is it possible to increase or decrease this memory on the "big" lambdas and have a decrease in cost (and even an increase in performance)?

That's when I remembered an tool I had heard about in an episode of the AWS podcast in French: AWS Lambda Power Tuning. This is the utility that I'm going to talk about today, the one that allowed me to save over 30% on my AWS Lambda bill.

AWS Lambda Power Tuning: How does it work?

This tool, which is open source and available here, takes the form of a Step Function that is deployed on your AWS account. The purpose of this Step Function is to run your lambda with different memory configurations several times and output a comparison in the form of a graph (or JSON) to try to find the optimal balance between cost and execution time. There are three possible optimization modes: cost, execution time, or a "balanced" mode where it tries to find a balance between the two.

To deploy it on your account, there are six different ways to do it:

Using AWS SAR (Serverless Application Repository), simply go to this link: https://serverlessrepo.aws.amazon.com/applications/arn:aws:serverlessrepo:us-east-1:451282441545:applications~aws-lambda-power-tuning and install the application.
With the SAM CLI.
With AWS CDK.
Via the open source AWS Power Tuner UI project (https://github.com/mattymoomoo/aws-power-tuner-ui).
Via Terraform by deploying the SAR application.
Via Terraform natively (without going through the SAR application).

You can find all the tutorials for deploying the application here: https://github.com/alexcasalboni/aws-lambda-power-tuning/blob/master/README-DEPLOY.md

The tool is then very simple to use. You need to go to the AWS Step Functions service, find the step function with a name that starts with "powerTuningStateMachine-", and press the "start execution" button.

Once you have pressed the button, you will be presented with this interface:

The part that interests us is the input part, where we will set up what we want to test. Here is the list of possible inputs for our execution:

lambdaARN: ARN of the lambda to execute
powerValues: Array representing the memory configurations to test
num: The number of lambda invocations to perform for each configuration (minimum 5, recommended between 10 and 100)
payload: Input parameters of the Lambda function
payloadS3: Input parameters of the Lambda function (if too large for the lambda)
parallelInvocation: Allows invoking the lambda in parallel (watch out for throttling if enabled...)
strategy: The optimization strategy I mentioned earlier (cost, time or balanced)
balancedWeight: Parameter representing what you want to optimize (0 corresponds to the cost strategy and 1 to the time strategy)
autoOptimize: Automatically applies the best configuration at the end of the step function execution
autoOptimizeAlias: Creates or updates the alias with the new configuration
dryRun: Allows testing the operation and call to the lambda function (IAM permission for example)
preProcessorARN: ARN of a lambda to execute before each execution of the lambda to be tested
postProcessorARN: ARN of a lambda to execute after each execution of the lambda to be tested
discardTopBottom: Allows removing the top and bottom 20% (to have a reliable representation of reality => remove cold starts, for example)
sleepBetweenRunsMs: Time between each execution of the function to be tested.

There is a lot of information here, but we can focus on lambdaARN, num, and strategy to launch our first invocation. The rest can be explored after the first optimization if you are not yet satisfied.

For example, an input would look like this :



{
  "lambdaARN": "arn:aws:lambda:eu-west-1:xxxxxxxx:function:lambda-power-tuning-snapoleon-article",
  "powerValues": [
    700,
    1000,
    1500,
    2000,
    2500,
    3000
  ],
  "num": 30,
  "payload": {},
  "parallelInvocation": false,
  "strategy": "balanced"
}

If everything goes well, you should have a nice state graph for the step function like this one:

If you click on the last step "Optimizer", you will have access to the output of the step with, among other things, the results of the step function (also available under the Execution input and output panel). Here, in JSON format, you will have the output with the results for each configuration, as well as a link to a website to visualize the results in the form of a graph. Each result for a given configuration will look like this:



{
      "averagePrice": 0.00003504375000000001,
      "averageDuration": 2135.3594444444443,
      "totalCost": 0.0010591546875000002,
      "value": 1000
}

and the graph will look something like this :

It is now time to show you the power of the tool with a concrete example.

Optimizing your Lambda function in practice

A good use case for optimization would be for pure computation, using a library like Pandas in Python. So we will try to optimize a Lambda function that uses the Pandas library. The code is very simple:



import json
import pandas as pd
import numpy as np


def lambda_handler(event, context):
    # generate random data
    data = np.random.randn(1500000, 10)
    df = pd.DataFrame(data)

    # apply compute onto data
    df = df.apply(lambda x: x**2)
    df = df.apply(lambda x: x + 10)

    df = df.apply(lambda x: x**2)
    df = df.apply(lambda x: x + 10)

    df = df.apply(lambda x: x**2)
    df = df.apply(lambda x: x + 10)

    # print results
    print(df)
    return {
        'statusCode': 200,
        'body': json.dumps('Please optimize me !')
    }

If you want to perform the test by yourself, you need to deploy the function and don't forget to add the layer containing the required libraries (Pandas and Numpy). Here is the ARN of the layer used here: arn:aws:lambda:eu-west-1:336392948345:layer:AWSSDKPandas-Python39:5.

Once the function is deployed and assuming that you have already deployed AWS Lambda Power Tuning on your account, all you have to do is to set up the execution of the function :



{
  "lambdaARN": "arn:aws:lambda:eu-west-1:xxxxxxx:function:lambda-power-tuning-snapoleon-article",
  "powerValues": [
    700,
    1000,
    1500,
    2000,
    2500,
    3000
  ],
  "num": 30,
  "payload": {},
  "parallelInvocation": false,
  "strategy": "balanced"
}

Here's what I used for our example. Please note that most of the time you will be limited to 3008MB of memory, which is a default "soft limit" for all AWS accounts. You can request an increase (up to 10,000 MB), but you will need to make a request to the support.

Once the parameters are filled in, we launch the execution and all we have to do is wait.
Two minutes later, everything is done and we have the results available :

We notice that if we allocate 700 MB, the execution time is 3097 ms for a cost of $0.000036. By increasing the memory to 1000 MB, we can reduce the execution time to 2135 ms for a similar cost of $0.000035. If we increase to 1500 MB, the execution time drops to 1422 ms and the cost remains the same. Starting from 2000 MB, we reach a plateau in terms of execution time (we won't go below 1149 ms). Since the time won't decrease anymore from this point, the cost will only increase. For 2000 MB, we have a cost of $0.000038, $0.000049 for 2500 MB and $0.000059 for 3000 MB.
We can conclude from this execution that the sweet spot seems to be around 1500 or 2000 MB depending on what you're looking to optimize. If you want to save money, 1500 MB seems like a good candidate. On the other hand, if you're looking to optimize execution time, 2000 MB seems like the best choice.

To conclude

This tool is essential if you want to optimize the cost or execution time of your Lambda functions. It will allow you to make considerable savings on your AWS bill if you frequently use the Lambda service. It is also important to note that this tool can be integrated into a CI/CD and executed with each deployment of your application. This way, you can continuously update your Lambda functions through DevOps processes (but beware of drifts in the IaC).

What if I were to tell you that a VPN is no longer useful

Sebastien Napoleon — Mon, 03 Apr 2023 09:28:34 +0000

Let's start with a scenario. It's a Monday morning, and we begin the week by connecting to our VPN. We can then access our emails, Teams channels, log in to the AWS console, etc. Just a typical Monday morning, right? An hour passes, and the connection is lost. "Well, I must have been disconnected from the VPN...," I'm sure that phrase sounds familiar to you.

Yes, the VPN, this tool that everyone uses. But what if I told you that it was no longer useful using AWS ? Imagine a world where we could securely connect to our applications without going through a heavy client? What if, on top of that, we applied the principle of zero trust? And even further, if we could put restrictions on the type of device we use to connect and the geographic location?

This is now possible. AWS announced its new AWS Verified Access service a few months ago, and today I will try to break it down for you.

VPN, what is it and why does everyone use it?

A VPN (Virtual Private Network) is a service that routes a user's web traffic to an internet destination via a secure remote server. This remote, secure server acts as an intermediary between the user and the internet network.

The operation of a VPN is not very complicated (unless you really get into the details).

source : https://blog.360totalsecurity.com/en/how-vpn-works/

During VPN activation, a connection to the VPN server is established. All incoming and outgoing data from your device then passes through an encrypted tunnel via a process of encapsulation. Encryption secures the connection by making your traffic unreadable. Data is effectively indecipherable - except for those who possess the encryption key.

No company wants unrestricted access to their internal applications. The https protocol is not sufficient here, so an additional layer of security is desired. Moreover, when someone now arrives on the Internet, the IP address of the person using the VPN will not be their ownIP address. Indeed, the person's request goes through the VPN server, so there has been a change of IP address indicating that the person went through the VPN.

It is therefore possible to restrict access to our applications by limiting incoming IP addresses to that of the VPN server(s).

We can see that having a VPN today is super interesting. However, there are still some disadvantages. Indeed, once connected to the VPN, it is much easier to harm an infrastructure, although there are still safeguards, but not always (authentication via user/password or via an SSO). If a malicious person manages to intrude on one of your servers, what about the rest?

In addition, it is quite annoying to have to start your VPN client all the time, connect to it. Not to mention the moment when this VPN no longer works or randomly disconnects (session timeout on the server side, any network issue, etc).

The solution offered by AWS: Verified Access.

Verified Access works by evaluating access requests against access policies attached to either an application or an application group. If all access policies are met, the user's request is accepted and sent to the application through a Verified Access-managed access point.

The notable difference between Verified Access and a VPN is that each request is evaluated independently against access policies defined by the security team.

Verified Access is composed of several components. Here is a diagram from the service's documentation that describes a high-level view of the service :

Verified Access instance (1) - An instance that evaluates application requests and authorizes or denies access based on defined security criteria.

Verified Access endpoint (2) - Each endpoint represents an application. There are two possible endpoint types: a Load Balancer or an Elastic Network Interface (ENI).

Verified Access group (3) - A collection of Verified Access endpoints. It is recommended to group applications with similar security criteria into a single group. For example, applications for marketing in a marketing group.

Access Policies (4) - A set of access policies defined by the client that determines whether access to an application is granted or denied. This can be a combination of things like the identity of the user trying to access the application, the properties of the device they are using (OS, browsers, updates, etc.), the location of the person, etc. An access policy can be assigned to a group to encompass a set of applications and, then a policy can be applied to a specific application within that group for more granularity.

Trust Providers (5) - A service that handles user identity (such as IAM identity center, for example, or your on-premises identity provider) or device security compliance. AWS works with its own services, of course, but also with third-party services (including you, possibly as an IdP) and others, notably for metadata related to device compliance. It is mandatory to have a Trust Provider attached to a Verified Access instance to use the service.

Trust data (6) - The data that passes between the Trust Provider and AWS Verified Access. This data can be an email address, an IP address, the browser used, your location, etc.

Verified Access will then confront this data with the access policies defined by the user when it receives an access request for an application.

For the rest of the article, I would like to dig into four important points from the perspective of a company that wants to migrate to AWS Verified Access: Access Policies, Trust Providers, Monitoring, and Security. Since VPNs are already often in place in companies, we would like to know how to integrate what already exists with the service. Additionally, a few lines are also necessary on how to write access policies, their formalism, as well as monitoring and security.

It is true that I have not even talked about infrastructure management. You create a Verified Access Instance, and then what? Well, afterwards, AWS takes care of everything behind the scenes. It is a managed service, which is why I will not discuss this topic.

You only pay for what you consume. As for the endpoints, I will not dwell on them. AWS manages the redirection to your application from the service. A CNAME will be created by AWS internally to redirect the endpoint to your application if the request is authorized.

Trust Providers: How to Migrate Legacy?

We have seen that a trust provider is mandatory to use the Verified Access service. This trust provider can take two forms: one for user identity (Identity Provider, IdP for short) or one for managing the devices used to access our applications (computers, tablets, smartphones, etc.).

Let's focus first on the identity of the users, as it is often the most relevant to us.

Three different cases may exist when it comes to user identity to use the Verified Access service:

You already use IAM Identity Center (successor to AWS Single Sign-on) and use this service as an IdP. You just need to connect IAM Identity Center with Verified Access and you're good to go.
You don't use IAM Identity Center and already have an on-prem IdP. It's simple, you'll need to create your space under IAM Identity Center. You can then easily connect the service with your on-prem IdP. It's also possible to use an Active Directory as an IdP. You just need to connect Verified Access with IAM Identity Center, and you're good to go.
You don't want to use the IAM Identity Center service and already have an on-prem IdP? No problem, if this IdP is compatible with the OpenID Connect (OIDC) protocol, you can directly connect the Verified Access service with your on-prem IdP via the OIDC protocol. It requires a bit of configuration, but it's possible.

What about support management now?

Verified Access allows support management using external services. At the time of writing this article, 2 services are supported: Jamf and CrowdStrike.

This list will likely expand, as a highlight of the service AWS wanted to emphasize at Re:Invent (https://www.youtube.com/watch?v=Kkxn-bAIlnI) is that they want to make it an "open" service. That is, to integrate as many reliable external solutions as possible, as this need is often raised with AWS. We will revisit this later, but the monitoring section is filled with integration with external solutions used everywhere.

Access Policies, what do they look like?

The access policies used in the Verified Access service are written in AWS's proprietary language: Cedar (https://www.cedarpolicy.com/).

Even before seeing examples of access policies, it is important to note that if you create an application and start using Verified Access, the default rule is the Zero Trust Policy: everything is denied by default. Similar to the principle of least privilege in IAM policies, we want our users to only have access to what they need and nothing more. We do not want someone to escalate certain privileges because we have overly permissive access policies.

Now that the basics are laid out, it's time to discover how an access policy written in Cedar is structured.

Effect - Allows or denies access based on the policy that follows.

Scope - Specifies the principal, actions, and resources that the policy applies to.

Condition clause - Specifies the context in which the effects apply.

It is important to note that in the context of access policies, the scope field should always be left blank. All that we are interested in is in the "condition clause" part, where we have access to the user's identity and the support they are using.

We end up with an access policy that looks like this:

permit(principal,action,resource)

when{
 context.<policy-reference-name>.<attribute> &&
 context.<policy-reference-name>.<attribute2>
};

We notice the possibility of using operators, here is the list of those available:

!, ==, !=, <, <=, >, >=, in, &&, ||, .exists(), has, like, .contains(), .containsAll(), .containsAny()

If one wishes to access certain information, particularly about the support used by the user, it is necessary for the user to have a browser extension to transmit the information to Verified Access. However, for any information related to the user's identity, there is no need for this browser extension.

Let's see what we can do through two examples taken from the AWS documentation.

permit(principal,action,resource)
when {
    context.identity.groups.contains("finance") &&
    context.identity.email_verified == true &&
    ip(context.http_request.client_ip).isInRange(ip("10.0.0.0/8"))
};

Here, we access the interesting data using the keyword "context". It is important to note that anything added to the "http_request" field will always be present. It contains certain information about the HTTP request such as the client's IP address, the HTTP method used, etc. (documentation on this subject can be found here: https://docs.aws.amazon.com/verified-access/latest/ug/trust-data-default-context.html).

The "identity" field allows us to know that the data comes from the IAM Identity Center service (and thus, that you are using it as an IdP). If you are not using this IdP, the documentation does not mention how to retrieve the fields (except for services that manage supports such as Jamf and CrowdStrike).

Returning to our example, we see that we will verify that the person's group (retrieved through IAM Identity Center) is indeed the "finance" group, then ensure that the email they are using is verified, and finally, that their IP address is in the range 10.0.0.0/8. If all this information is evaluated positively, the person's request will be accepted (thanks to the "permit" keyword at the top).

permit(principal,action,resource)
when {
     context.crwd.assessment.overall > 50 
     && context.jamf.risk == "LOW"
};

Second example which is actually a combination of two examples from the documentation. Here, we see the use of the keywords "crwd" and "jamf". These correspond to the two third-party services authorized by AWS for support management.

The first one is CrowdStrike, characterized by the keyword "crwd". In this example, we will verify that the overall rating of the support used is greater than 50. For the second one, which is none other than Jamf, characterized by "jamf", we will verify that the risk associated with the support is equal to "LOW".

We can see that the service allows for a large part of what businesses want in terms of options to secure access to their applications.

How do we monitor all of this? And what about security?

There is still a very important point that we have not yet discussed. Let's talk about monitoring and security. As a VPN is often a bottleneck and the gateway for all traffic to the internal network of a company, it is necessary that it be well monitored with a hermetic security.

For security, all data at rest is encrypted by KMS with a key held and managed by the service itself. We can use CloudTrail to audit the use of the service. For data in transit, the service uses TLS 1.2 for encryption.

As for traffic between multiple networks, it is possible to configure Verified Access to restrict access to specific resources in a Virtual Private Cloud (VPC). It is also possible to restrict access to portions of the network based on the user's group.

Let's now talk about monitoring. In August 2022, AWS released the Open Cybersecurity Schema Framework (OCSF). In collaboration with various industry players, they defined a standard log schema to enable those who produce logs to have the same view as those who consume them. AWS Verified Access currently uses this log format to allow for simplified integration with globally recognized monitoring services, including the following list of partner solutions (at present):

Datadog
IBM
netskope
new relic
RAPID7
sumo logic
Trellix
Splunk

How much does it cost?

Certainly, one of the sensitive issues concerning this service is its price. The bill is divided into two parts: hourly charges for each application, and data that has passed through during the use of the service.

The hourly price of the service per application, in the Ireland region (eu-west-1), amounts to $0.31/hour per application up to 148,800 hours per application (which corresponds to approximately 200 applications per month, running all the time). The rate is lower for hours per application above 148,800 and costs $0.23/hour thereafter.

Processed data costs $0.02/GB.

If we take an example with 210 applications that process 1 GB of data each for 31 days, we arrive at the following calculation:

148,800 * 0.31 + 7,400 * 0.23 + 1 * 210 * 0.02 = $47,834.2

To conclude,

To add a personal opinion about this service, I think it is a super useful service. I believe there is a good chance that this kind of service will replace the VPN solutions we currently know. Given the complexity of maintaining a VPN today, there is a big gain to be had here (because yes, when the VPN goes down, it's a big part of the day's revenue that goes up in smoke for many companies with the democratization of teleworking). Here, the VPN is managed by AWS, we no longer have to worry about its robustness. In addition, the access policy part seems to be very user-friendly, it is much easier to centralize policies and manage access to them using IAM. So, of course, all these advantages come with a cost.

Can we say that the VPN is no longer useful? Not really. Indeed, some use cases will still be relevant, I think. What about SSH access to a database (a very good alternative being AWS SSM with the session manager if the resource is on AWS)? Access to a NAS? The service doesn't seem to be made for that yet. It remains to be seen what the future holds.

If you are interested in setting up this service, I found this article to be very well detailed: https://medium.com/@chaim_sanders/a-visual-guide-to-setting-up-aws-verified-access-1333466f7222

What is an event-driven architecture and why storing events is important ?

Sebastien Napoleon — Mon, 06 Mar 2023 13:31:39 +0000

In this post, we are going to see how to create a serverless analytics architecture based off an event-driven architecture. This is made possible with the help of services like Kinesis Data Firehose, EventBridge, Athena, Glue, Lambda and S3.

To avoid rewriting yet another definition of this type of architecture, let's turn to AWS's definition: "An event-driven architecture relies on events to trigger communication between decoupled services. This type of architecture is common in modern applications based on microservices. An event can be a state change or update, such as adding an item to the cart on an e-commerce site. Events can refer to a state (purchased item, modified price, and entered delivery address) or consist of identifiers (shipment notification of an order).

Event-driven architectures have three key elements:

Event producers
Event routers
Event consumers

A producer sends an event to the router, which filters the events and then passes them on to consumers. The producer and consumer services are decoupled, which allows them to be scaled, updated, and deployed independently." (source: https://aws.amazon.com/event-driven-architecture/)

Now that you are familiar with this type of architecture (or at least understand the principle), it is important to understand that events become the heart of our application. Without them, the application no longer makes sense, and it stops functioning as it is no longer receiving input. We quickly realize that storing these events is essential. The marketing team would like to know how many products were sold in the last few hours, days, months, or years. On the other hand, architects would like to create a new service that would consume one or more existing events. They would therefore like to know the number of events produced in x time.

If you are also convinced that storing and analyzing these events is essential, you're lucky because you've come to the right place.

The complete code presented in this article is available here : https://gitlab.com/bootstrap-stack/serverless-analytics

Producing events on AWS

The central service of our architecture is AWS EventBridge, the event bus service released by AWS in July 2019. This service is widely used and in fact, used internaly by other AWS services. Config rules, for example, are triggered by events passing through AWS event buses. Each service has its own events that are emitted into EventBridge (from the success/failure of a Lambda function to a scale up or down in an auto scaling group), and a lot of information passes through these buses (example for auto scaling events: https://docs.aws.amazon.com/autoscaling/ec2/userguide/automating-ec2-auto-scaling-with-eventbridge.html)

But how do we create our own data bus in EventBridge and then send events into? It's simple - you can do it directly in the console or using your favorite infrastructure-as-code tool. In this article, I will use AWS Cloud Development Kit (CDK) to deploy my infrastructure (the code used will be in TypeScript)




const businessEventsBus = new EventBus(app, ‘IpponBlogDemo’)

Now that we have created our bus, let me tell you about the formalism to use when sending events to EventBridge. An event consists of 5 fields:

time: the time and date of our event that we are producing.
source: the source corresponds to the domain from which your event was sent (e.g. sales domain).
detailType: defines the action related to the event. If we stick to the perspective of an e-commerce store, we could have "item.sold" for example in this field.
detail: the content of the event, this is where we will put specific fields (product name, price, etc.).
event bus name: the name of the bus in which we want to send the event.

All of these fields are encapsulated in a JSON object and in our case, sent via a TypeScript class that uses the AWS SDK to call the EventBridge API.




import { EventBridgeClient, PutEventsCommand } from '/opt/nodejs/node_modules/@aws-sdk/client-eventbridge'

export default class EventSenderEventBridge {
    private client = new EventBridgeClient({})
    private busName = ‘myBusName’

    send = async () => {
        const query = new PutEventsCommand({
            Entries: [{
                EventBusName: this.busName,
                Source: “sales”,
                DetailType:”item.sold”,
                Detail: {“name”: “toothbrush”, “price”: 9.99, “quantity”: 1},
            }],
        })
        await this.client.send(query)
    }
}

How to store our data?

As I am a huge fan of the serverless world and managed services (yes, it does make life so much easier), having a small wallet, I turned to a flexible stack.

We will use S3 to store our data. This support is perfect for our use case. It's cheap, ultra-flexible, and 100% managed by AWS.

We have the support for our data, we know how to easily produce events. Now, we need that little glue between our receptacle and the events.

This glue will be provided by the Kinesis service, particularly Kinesis Data Firehose. Indeed, there is an existing integration between Kinesis Data Firehose and EventBridge with S3 bucket as the destination!

First, you may not know the Kinesis Data Firehose service. Here's the AWS definition: Amazon Kinesis Data Firehose is an Extract, Transform, and Load (ETL) service that captures, transforms, and reliably delivers streaming data to data lakes, data stores, and analytics services. (https://aws.amazon.com/kinesis/data-firehose/)

It may not be very clear yet, so here's a diagram illustrating how the service works:

We can see on the left, the "20+ AWS Services" that contains the EventBridge service. So, our events will be transmitted to KDF (Kinesis Data Firehose), where we can easily transform them using a lambda or tools provided by AWS. The result will finally be transmitted to the destination, in our case, an S3 bucket.

How to create and configure a Kinesis Data Firehose stream

Now, let's move on to the creation of our infrastructure with CDK, and I'll try to explain the different fields required when creating our stream.

First, you need to create an IAM role with certain permissions. To avoid writing too much code, I will only create the role and add an IAM policy to it to show you how to do it:




const deliveryStreamRole = new Role(
    construct, props.eventTypeUppercased + 'DeliveryStreamRole', {
        assumedBy: new ServicePrincipal('firehose.amazonaws.com'),
    },
)

deliveryStreamRole.addToPolicy(new PolicyStatement({
    resources: ['*'],
    actions: [
        'logs:CreateLogGroup',
        'logs:PutLogEvents',
        'logs:CreateLogStream',
    ],
}))

Now, you need to add certain IAM policies (depending on your needs):

For the S3 service: AbortMultipartUpload, GetBucketLocation, GetObject, ListBucket, ListBucketMultipartUploads, PutObject.
For the Lambda service (if you want to transform your input objects into a format different from that of EventBridge): InvokeFunction, GetFunctionConfiguration.
For the Glue service (we will see the usefulness of this service later): GetDatabase, GetTable, GetPartition*, GetTableVersions.

Permissions are now created. It's time to create our S3 bucket to store our data:




const destinationBucket = new Bucket(
    construct,
    ‘BlogIpponDemoBucket’,
    { bucketName: ‘blog-ippon-destination-repository’ }
)

Let's move on to the big part, the creation of our Kinesis Data FireHose (KDF) stream. I will break down the creation into several steps to explain what each of them corresponds to.

First, in the CDK document for the KDF service, we need to choose a "destination configuration" to pass as an object in the constructor. Here is a link to the documentation: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_kinesisfirehose.CfnDeliveryStream.html#construct-props

There are different destinations that can be found there, but the one we are interested in is S3, more specifically the extendedS3DestinationConfiguration object. The following code will be the configuration of this object.

For readability reasons, we will add intermediate variables for certain parts. Two variables will be needed, the first concerns the processingConfiguration field and the second concerns the dataFormatConversionConfiguration field.




const processingConfiguration = {
    enabled: true,
    processors: [
        {
            type: 'Lambda',
            parameters: [
                {
                    parameterName: 'LambdaArn',
                    parameterValue: ‘BlogIpponDemoLambdaTransformationArn’,
                },
            ],
        },
        {
            type: 'AppendDelimiterToRecord',
            parameters: [
                {
                    parameterName: 'Delimiter',
                    parameterValue: '\\n',
                },
            ],
        }
    ],
}

We are now in the processingConfiguration section. This section comes into play once the data has passed through the buffer (which I will explain later). Either we do nothing, in which case the data goes directly to our destination, or we decide to transform it before storing it. In our case, our source data is an EventBridge event.

We would like to be able to transform the source event into something more meaningful, something that will make sense when we come to analyze it. In this case, we will use a small lambda that we will have built. It's a very basic lambda that takes JSON as input and transforms it into another JSON as output (our business event).

As a result, we have streamlined the source data by removing unnecessary fields. The second processor is just there to add a delimiter between each of our records. The data is now ready to go to our S3 bucket.




const dataFormatConversionConfiguration = {
    enabled: true,
    inputFormatConfiguration: { deserializer: { openXJsonSerDe: {} } },
    outputFormatConfiguration: { serializer: { parquetSerDe: {} } },
    schemaConfiguration: {
        catalogId: props.glueDatabase.catalogId,
        roleArn: deliveryStreamRole.roleArn,
        databaseName: props.glueDatabaseName,
        tableName: props.glueTableName,
    },
}

Our second variable is dataFormatConversionConfiguration. It allows us to configure the format part of data conversion (input and output) and to define a schema for our data (this is where Glue comes in).

Here, our input events use the openXJsonSerDe serializer and for the output, in order not to store the data as is, we will store it in Apache Parquet format, which is a columnar format (which will reduce costs). We will then use the parquetSerDe deserializer.

Now it's time to define a schema for our data. This schema is contained in a Glue table, which itself is contained in a database. Here we only specify where our schema is stored (part schemaConfiguration).

It is now time to assemble everything in our configuration object by adding the other fields.




const deliveryStream = new CfnDeliveryStream(construct, props.eventTypeUppercased + 'DeliveryStream', {
        extendedS3DestinationConfiguration: {
            bucketArn: props.destinationBucket.bucketArn,
            roleArn: deliveryStreamRole.roleArn,
            bufferingHints: {
                intervalInSeconds: 60,
            },
            cloudWatchLoggingOptions: {
                enabled: true,
                logGroupName: 'kinesis-log-group',
                logStreamName: 'kinesis-logs',
            },
            prefix: 'year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}custom_partition=!{partitionKeyFromLambda:custom_partition}/',
            errorOutputPrefix: 'errors/!{firehose:error-output-type}/',
            processingConfiguration: processingConfiguration,
            dataFormatConversionConfiguration: dataFormatConversionConfiguration,
            dynamicPartitioningConfiguration: { enabled: true, retryOptions: { durationInSeconds: 10 } },
        },
    }
)

The first two fields correspond to the ARN of the destination S3 bucket and the ARN of the IAM role to use.

The bufferingHints field is interesting as it allows us to configure the management of our stream buffer. The buffer corresponds to either the data retention time or the minimum amount of data to be sent.

Two fields are available, intervalInSeconds which corresponds to the time part (how long to wait). The value of this field can range from 60s to 900s and defaults to 300s (we wait at least 1 minute between each send and up to 15 minutes). Then we have the sizeInMBs field, which corresponds to the amount of data we should wait for before sending. This amount varies between 1 and 128 with a default value of 5.

We now move on to the part concerning the logs. Here we have a cloudWatchLoggingOptions object that allows us to activate logs in CloudWatch by setting the name of the CloudWatch Group and Stream.

We now have one of the crucial parts to define our stream, the prefix to use (prefix) to know where to send our data in our S3 bucket (and especially, with which partitions). Here, I wanted to have a separation by datetime with a compartment per year/month/day and then by a custom partition that we will see later. We use a partitioning mode based on Hive to take advantage of dynamic partitioning. It's just necessary to know that we need to define our partition keys with this formalism "partition_key=!{We retrieve the key here}". We do the same for the output of processing or sending errors with the errorOutputPrefix keyword. The fact that errors are sent directly to the S3 bucket is super convenient, we can easily see if there are errors. We can also automate the detection of these errors by building a lambda that will be triggered when a file is added to this error compartment.

To finish creating our stream, we will activate and configure dynamic partitioning using the dynamicPartitioningConfiguration keyword. Dynamic partitioning is recent and offers many advantages. The first is to be able to query our data according to keys that we define ourselves. These keys will allow us to build SQL queries with WHERE clauses. This will allow Athena (the service we will use later to query) to only pick the data we really care about (and therefore not scan unnecessary data). So we will have an improvement in performance, cost, and the number of scanned data. The downside is that this option is recent and not necessarily well documented at the moment.

https://docs.aws.amazon.com/firehose/latest/dev/dynamic-partitioning.html

How to create and configure a Glue table to use with KDF

Here, the use of Glue that we make is based on the definition of a data schema in a database. It should be noted that Glue is a serverless data integration service that allows analytics users to easily discover, prepare, move, and integrate data from multiple sources. So it is much more complete and complex than the use we make of it here.

We are almost there, just two more steps and we can finally query our business events from EventBridge. Let's move on to the creation of a Glue database and table:




const glueDatabase = new CfnDatabase(scope, 'glue-database', {
        catalogId: ‘IpponBlogDemoAwsAccountId’,
        databaseInput: {
            name: ’IpponBlogDemoGlueDatabase’,
        },
    }
)

There's no need to explain this step, as the code is pretty basic with what you have seen in the article by now.

For the creation of the table, the interesting configuration will be located in the tableInput object.

Since this object is quite large, we will have to use a few intermediate variables. These variables will be partitionKeys and columns.




const partitionKeys = [

    {
        'name': 'year',
        'type': 'string',
    },
    {
        'name': 'month',
        'type': 'string',
    },
    {
        'name': 'day',
        'type': 'string',
    },
    {
        ‘name’ : ‘custom_partition’,
        ‘type’: ‘string’,
    }
]

This first object allows us to define partition keys. These keys correspond to what we put in the prefix section when configuring our stream earlier. The "custom_partition" field here is just to remind us of the partition that we added thanks to the transformation lambda.




const columns = [
    {
        ‘name’: ‘name’,
        ‘comment’: 'Name of the item’,
        ‘type’: 'string'
    },
    {
       ‘name’: ‘price’,
       ‘comment’: 'Price of the item’,
       ‘type’: 'string'
    },
    {
        ‘name’: ’quantity’,
        ‘comment’: ’Quantity of item’,
        ‘type’: 'string'
    },
]

This field will allow us to define the schema that our data will have in our S3 bucket. Here we will have a name, a price, and a quantity for each item.

We can now assemble everything in our CDK object.




const glueTable = new CfnTable(scope, 'glue-table-for-athena', {
    databaseName: ’IpponBlogDemoGlueDatabase’’,
    catalogId: ‘IpponBlogDemoAwsAccountId’,
    tableInput: {
        name: ‘IpponBlogDemoGlueTable’,
        tableType: 'EXTERNAL_TABLE',
        partitionKeys: partitionsKeys,
        storageDescriptor: {
            compressed: true,
            inputFormat: 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat',
            outputFormat: 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat',
            serdeInfo: {
                serializationLibrary: 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe',
            },
            columns: columns,
            location: ‘IpponBlogDemoS3BucketUrl’,
        },
    },
})

The storageDescriptor field will allow us to describe how the data will be stored. Here, we will specify that our data will be compressed, our input format will be Apache Parquet, and our output format will also be Apache Parquet. As I am not an expert in this area, I will not go into the details of this configuration, but it has the merit of working. We will simply need to define the location of our data (where it is stored) using the location keyword (we will pass in the URL of our S3 bucket).

We still need to describe to Glue what these partition keys correspond to (i.e. the associated type, possible values, and the number of digits possible, for example). To do this, we will use the addPropertyOverride() function on the table we created earlier.




glueTable.addPropertyOverride('TableInput.Parameters.projection\\.enabled', true)
glueTable.addPropertyOverride('TableInput.Parameters.projection\\.year\\.type', 'integer')
glueTable.addPropertyOverride('TableInput.Parameters.projection\\.year\\.range', '2022,2050')
glueTable.addPropertyOverride('TableInput.Parameters.projection\\.year\\.digits', '4')
glueTable.addPropertyOverride('TableInput.Parameters.projection\\.month\\.type', 'integer')
glueTable.addPropertyOverride('TableInput.Parameters.projection\\.month\\.range', '1,12')
glueTable.addPropertyOverride('TableInput.Parameters.projection\\.month\\.digits', '2')
glueTable.addPropertyOverride('TableInput.Parameters.projection\\.day\\.type', 'integer')
glueTable.addPropertyOverride('TableInput.Parameters.projection\\.day\\.range', '1,31')
glueTable.addPropertyOverride('TableInput.Parameters.projection\\.day\\.digits', '2')
glueTable.addPropertyOverride('TableInput.Parameters.projection\\.custom_partition\\.type', 'injected')
glueTable.addPropertyOverride('TableInput.Parameters.storage\\.location\\.template', 's3://ippon-blog-demo-analytics-repository/year=${year}/month=${month}/day=${day}/custom_partition=${custom_partition}')

In our case, we specify that the year partition is a 4-digit integer between 2022 and 2050. We repeat the process for months and days and let Glue define the type of our custom partition itself using the injected keyword. It's very convenient but has a big disadvantage: we can only perform equalities in SQL queries on this field afterwards (which can be annoying when we want to retrieve information for a specific period and not just a fixed date, for example).

And there you have it, the job is done.

How to query data in S3?

We now have a functional stack that allows us to retrieve events sent to EventBridge, transform them into a business-oriented format, and send them to an S3 bucket with beautiful partitions. The only thing left to do is to query all this data. When it comes to analytics and S3, we often think of the AWS Athena service. Athena is a service that allows you to easily query an S3 bucket (and many other data sources), using SQL language and Apache Spark under the hood. In addition to that, Athena is a completely managed service by AWS, no infrastructure management required and that's awesome.

The best part of the story? You don't need to do anything: once you get to the Athena console, simply choose your Glue database, select the table in question, configure the destination for query results and you'll be ready to query your data.

What's magical is that thanks to storing our data in the parquet format combined with dynamic data partitioning, this entire stack will cost you very little money. Unless, of course, you produce a huge number of events (because yes, Athena can be expensive). And when I say huge, I really mean a lot. On a personal project, I'm at around 100,000 events per day and the stack doesn't cost me a penny with AWS's free plan. The query over 10 days, or ~1m events, represents 80MB of scanned data. Considering that Athena's price is €5 per terabyte of scanned data, this query represents a little less than 0.0005 cents. Of course, it all depends on the size of your events.

In conclusion, it is possible to optimize performance when querying with Athena. Indeed, the fewer files we have in our S3 destination bucket, the faster Athena will execute the query. It is therefore possible, with the help of an EMR (Elastic MapReduce) job, to join the files together to have fewer, but larger files.

In terms of costs, you also need to be careful with the Athena query result bucket and not hesitate to set lifecycle configurations on the buckets to expire objects that you no longer need (queries can quickly reach gigabytes of results). You should also think carefully about what you want to store. Storing just for the sake of storing is not very useful (either for the business or for the wallet). Only store what can add value to the queries you are going to make, the rest is certainly not useful for analytics.