Forem: Andra Somesan (she/her)

What is AWS Community Builders and what's in it for you

Andra Somesan (she/her) — Fri, 14 Jan 2022 07:00:52 +0000

A few days ago the application for the AWS Community Builders program has opened and I happily shared the news on Twitter (@ AndraSomesan) and LinkedIn. Since then, I keep getting DMs like "What exactly is AWS Community Builders program?" and "Could you help me with my application?" and I am delighted to help.
A few of my colleagues have already written some great articles on this topic, but I would like to highlight what the program is about and what you can get from it ✨.

❗The application is open until January 23rd EOD Pacific Time.

What is AWS Community Builders Program

You can read everything about it on the main page, but basically it is a great way to connect the people from all around the globe (94 countries now) who are activly involved in the AWS community, all in one place. Of course, it will be impossible to manage a program like this if everyone in the AWS space gets in, and this is why only a small percent of the applicants receive the great news. In the last round of applications - October 2021, only 10% of the applicantions were approved.
To stand out you have to really be involved, write more articles, do some Meetup presentations, organize Meetups, have an YouTube channel, a Podcast, be very active on Twitter. You don't have to be doing all of them, you just have to stand out among the applicants. And it's not all about that, the questions cover your motivation for joining, what is your unique perspective, and what are your future plans.

Eager to find out more

To know more about who should apply and how, you can read this article by @pawelpiwosz
For a more in depth information about the program and some of its benefits, you can read this article by @andrewbrown

What's in it for you

It is all about the community ❤️ and it comes with some extra benefits 🎁.

AWS Commnity Builders is a great community - I've met some of my collegues for the first time ~~this~~ last year at re:Invent, and it was amazing to start feeling comfortable with people I have actually never met before. Being in the Slack workspace, you get a lot of help and support, but also a chance to do some networking and to make new friends, or just finding out about things you never heard about before 😄.
Free stuff - Everybody loves free stuff, especially when they are branded with the cloud you love or with the great community logo. You get an Welcome Kit, AWS Credits, exam voucher and some extra things that usually vary.
Exclusiveness - There are exclusive events with people from AWS and sessions with the AWS Heroes. This is more amazing than I can describe. The chance to do some networking with the Heroes and to be able to attend a presentation from Jeff Barr itself is just beyond awesome 🦄.

Eager to find out more

Then this article on 10 benefits of the program by @vattybear is the right place.

Conclusion

I hope this was a quick read that covered the basics of what the AWS Community Builders program is and what benefits it can bring you if you will join. Keep being great and keep trying! If you don't get in easily, doesn't mean you won't get in at all! 🎓

How to prepare for AWS re:Invent 2021

Andra Somesan (she/her) — Tue, 19 Oct 2021 06:08:31 +0000

Photo: screenshot from re:Invent page

For the past few days I've been searching for ways to prepare for the AWS re:Invent 2021 in person conference and I started to feel a bit overwhelmed at first. I have started to search and read more about it and in this post I would like to share with you what helped me out so far in making a plan for the event.

This will be my first time ever attending the conference (hopefully in person), and as we get closer to the event and also to the reserved seating opening up (today!) I had to try and make up my mind quickly.
I will start by sharing a few links I found useful and at the end I will talk a bit about the tips and tricks.

Useful links to prepare for AWS re:Invent 2021

AWS re:Invent FAQs page - here we can find all we need about topics such as health and safety, registration, accessibility, and so on.

AWS How to re:Invent page - this is an AMAZING page! Here Annie Hancock and Kelley Schultz from AWS are sharing everything we need to know about the conference and how to approach and plan it in order to make the most out of it. The first episode you'll see is the latest one, scroll down and start with episode 1. It will take you about 45 minutes to watch all 3 episodes published by now (October 18, 2021).
The highlights for me here were:
1. Getting an overview over the session types and which ones have reserved seating, which ones don't.
2. Make a goal for what you want to achieve at this conference, something like: see a certain talk, get a certification, learn more about a topic or work to improve another. Arrange your calendar with this goal in mind, but always listen to your body and have a Plan B because this week is going to be a LOT to handle. The final goal will be to enjoy it nevertheless, don't forget that 😄
AWS Breakout Content page - here we can find the session names explained, more details about what a breakout session or a chalk talk is.
AWS Attendee Guides page - here I will try to follow the most relevant guides for me from the AWS Hero Guides part. It's nice that we can chose a domain, like Serverless and have a look at what Emrah Samdan will pick and chose for this event. Every guide has at the end the same few tips for the conference and explains how to reserve seats.
AWS re:Invent 2021 Agenda - as the name implies, here we can find a high-level agenda of the event.

Tips & tricks

There is so much to do in so little time, so my approach was to first see which kind of sessions will be available online after the event, this way I can go if I really want to see that session live, or I can skip if I really can't make it and see it online later on. I feel this takes a bit the stress of "I have to see them all" thought that crossed my mind 😅 .

Breakout sessions, leadership sessions, and keynotes will be available online later on.

About the reserved seating, if you don't have the time to watch the session on How to re:Invent here are my takes:

Reserve first seats for chalk talks and smaller session because they are usually filling up faster than others.
Leadership sessions have reserved seating but the Keynotes don't, you have to be there sooner to get your spot. There are ways to see the Keynotes from outside of the room in places specially prepared for this.

Some tips I have received over LinkedIn and also found in this nice article:

Wear comfy shoes
Pack a bit of water and some snacks
Have a backpack (keep it lightweight if possible)

Let me know if this was useful to you and please share other tips and tricks if you have some 🌟 .

How to configure AWS news RSS feed for Microsoft Teams

Andra Somesan (she/her) — Wed, 18 Aug 2021 14:42:07 +0000

This is my second time configuring an RSS news feed, and to be honest, the first time was one month ago.
What made me write this article was the fact that I didn't remember exactly how to do it and finding all the information in one place was not that easy.
Working in IT is challenging and one of the causes is staying up to date with the latest releases and news. By being up to date with the announcements will make any developer's life easier and will also bring business value to the company. For example, if AWS is launching a new instance type that will reduce the costs and increase the performance, we will want to know that.
This is why I find it extremely important while working with AWS to have a channel dedicated to the latest news.
Let's see how we can configure this in Microsoft Teams and stay until the end for an extra tip.

1. Create a new channel

To start we have to first go to our team group in Microsoft Teams and click on the 3 dots in the right, then click Add Channel.

2. Configure the new channel

In the new window we can add a name for the new channel and optionally a description.
We can also choose for it to be either Public or Private.

I like to start with it as private, do all the configuration and then add people or make it Public if it will make sense for the team. So, in the next step we can skip adding new members for now.

Congrats! Our new private channel is up and running 🎉.

3. Add connectors to the new channel

Connectors in Microsoft Teams help us by delivering content and updates from services into the channel. To add one, let's click on the 3 dots in the upper right corner of our newly created channel and then select Connectors.

Search for RSS if you don't see it on the first page, and click to add it.

This is what we will see next and we have to click again the Add button.

4. Configure the RSS connector

Now that the RSS connector is added, the only thing left to do is to configure it. Let's click again on the 3 dots of our new channel and then click on Connectors as seen in step 3. The option to configure the added RSS will be available now.

Next up we need to give a name to our new RSS connection, add the link address for the RSS feed (see this for the AWS News RSS feed), and select the frequency you would like to be notified by any changes.

And that's all. This is a pretty easy thing to do, but it is nice to find all these steps in one place, especially if you are switching from Slack to Microsoft Teams and the later is a bit of a mystery for you at the beginning 😃

Now, as I promised in the beginning, I will shortly show how to delete an RSS news feed, if you accidentally added the wrong one to the channel (you know why I know this 🙈).

Go again on the 3 dots of the channel and click on Connectors like we did in step 3. There will be two or more configured RSS feeds, click x configured to expand like in the picture below.

And we'll see this:

Click on Remove.

Remove need to be clicked again.
Now our AWS news channel is nice and clean, with no other feeds in it. Having separate channels for every RSS news feed is a good idea.

I hope this will be useful and please let me know if you have any tips and tricks for doing this.

Cheers,
Andra 👻

Automated database update and restore with AWS Lambda functions for AWS DocumentDB

Andra Somesan (she/her) — Sat, 14 Aug 2021 07:37:01 +0000

Disclaimer: This article was first published here

In my previous article I talked about my last challenge, namely the automation of an update and restore process for the solution that was already migrated to the cloud. We discussed there the advantages of cloud adoption and we have focused on the Aurora DB. We will discuss in this article how we have reused the same solution and what modifications we have made to make it suitable for DocumentDB, because another great thing about the cloud is that we can reuse and adapt solutions to meet different requirements easly.

Amazon DocumentDB has MongoDB compatibility and “is a database service that is purpose-built for JSON data management at scale, fully managed and integrated with AWS, and enterprise-ready with high durability.” as AWS states in the official documentation.

Scenario (recap)

Our customer has 2 different AWS accounts:

Acceptance
Production

All changes made in the production account should be (every two weeks) synchronised to their acceptance account for regression testing in the acceptance account.
The old way of doing this with the on-prem infrastructure was to manually run some commands and update the databases.
The process should be done in the maintenance window, which implies working in the night/early morning when nobody is using the databases.
The big advantage of having the solution in the cloud is that it can easly be automated using services that bring low or no cost at all, and that this solution needs no human interaction.

Solution overview (recap)

The solution we agreed on was to split the process in 2 main parts having 2 AWS Lambda functions:

first one, db-update-latest-snapshot-id, will mainly focus on preparing the environment:
- copy the shared DocumentDB snapshot (because we can’t restore from it directly, see this article)
- check the latest copied snapshot for DocumentDB
- update the latest snapshot id parameters in AWS SSM Parameter Store
the second one, db-restore-from-snapshot, will mainly focus on triggering the update and restore process:
- will copy the latest snapshot ID value to the current snapshot ID and then trigger the main pipeline to deploy the changes and restore the databasess from the latest snapshots I took advantage of the already existing AWS Lambda functions that are sharing the database snapshots from one account to the other. The sharing part is not in the scope of this article but it was ilustrated for a better view of the solution. Having two lambda functions with a small defined taks, is not only best practise of AWS, but gives also the flexibility to copy/share more frequently without a real restore being done.

In AWS SSM Parameter Store, I have created 2 more parameters:

documentdb_current_snapshot_id
documentdb_latest_snapshot_id

and I have reused one:

main_cicd_name

The current value will be the one from which the databases are restored from and the latest value will be the value of the most recent snapshot that exists in the account for DocumentDB or Aurora.
We need to have 2 different parameters, one for the current and one for the latest snapshot, to be able to restore the databases from a specific snapshot ID outside of this process and not directly trigger a restore when the pipeline is running but restore on a pre-defined time/date.
There are 2 pipeline in the account, so the name of the main pipeline, the one that is deploying the infrastructure, was needed.

The entire solution was written using IaC in AWS CDK with Python.

How does it work

Let’s have a look at the code and you can see that a lot of it was reused. We have been using environment variables and a custom function that gets the environment variable value get_optional. There is an existing process of restoring the database from a specific snapshot ARN from the environment variables, and this is why we use the check for the current_docdb_backup_snapshot parameter. If it exists it will update the parameter with this value. For the ssm_docdb_latest_snapshot_name a dummy value was used because you can’t create an SSM parameter with an empty value.

current_docdb_backup_snapshot = get_optional(
    'DOCDB_BACKUP_SNAPSHOT_ID_ARN',
    None
)

ssm_docdb_current_snapshot_name = get_optional(
    'SSM_DOCDB_CURRENT_SNAPSHOT_NAME',
    None
)

ssm_docdb_latest_snapshot_name = get_optional(
    'SSM_DOCDB_LATEST_SNAPSHOT_NAME',
    None
)
resources = []
if current_docdb_backup_snapshot:
    current_arn = ssm.StringParameter(
        scope,
        f'{config["stack_name"]}DocDbCurrentDSnapshotArn',
        parameter_name=ssm_docdb_current_snapshot_name,
        string_value=current_docdb_backup_snapshot
    )
    resources.append(current_arn.parameter_arn)

main_cicd_ssm = ssm.StringParameter.from_string_parameter_name(
    scope,
    f'{config["stack_name"]}MainPipelineName',
    string_parameter_name=main_cicd_name,
)
main_cicd_name_value = main_cicd_ssm.string_value
resources.append(main_cicd_ssm.parameter_arn)
resources.append(f'arn:aws:codepipeline:*:*:{main_cicd_name_value}')

All the code from above was reused from the previous task that had as target the AWS Aurora DB. The change and adaptation were needed more in the way that the function work, as you can see below.
Using a cronjob the update function (db-update-latest-snapshot-id) will run every 2 weeks at 2 AM.

This function has the following tasks/functions:

copy_docdb_snapshot - checks for the latest DocumentDB shared and manual snapshot, and it takes the latest value for both of them. It then checks if the latest shared snapshot is already copied and if not, it will copy that one, if yes, it will return a nice message. The process was not needed for Aurora DB and had to be investigated and added to the db-update-latest-snapshot-id Lambda function. This copy process has added a new required check to be done for the database. The search we have used for the shared snapshot is still needed, but a new search process needs to be in place because once the shared snapshot is copied, this one becomes a manual snapshot and we want to check if the latest shared snapshot was copied or needs to be copied. See the code below:

 # Check if the latest shared snapshot is already copied and if not
    # take the latest shared snapshots and copy it

    logger.info('Copying the latest shared snapshot for DocDB')
    latest_shared_snapshot = f"copy-{docdb_shared_snapshots_source[0]['DBClusterSnapshotIdentifier'].split(':')[-1]}"
    for snapshot in docdb_manual_snapshots:
        if latest_shared_snapshot == snapshot['DBClusterSnapshotIdentifier'].split(':')[-1]:
            logger.info('Latest snapshot already copied')
            return snapshot
    docdb_copied_snapshot = {}
    try:
        docdb_copied_snapshot = docdb_client.copy_db_cluster_snapshot(
            SourceDBClusterSnapshotIdentifier=(
                docdb_shared_snapshots_source[0]['DBClusterSnapshotIdentifier']
            ),
            TargetDBClusterSnapshotIdentifier=(
                latest_shared_snapshot
            ),
            KmsKeyId=kms_key
        )
    except Exception as err:
        logger.error(
            f'ERROR copying snapshot: '
            f"{docdb_shared_snapshots_source[0]['DBClusterSnapshotIdentifier']}")
        logger.error('ERROR: ' + str(err))

    return docdb_copied_snapshot

update_ssm - it will update the SSM Parameter store with the new values of the latest snapshots in documentdb_latest_snapshot_id.

And this is it for the update process.
Now for the restore is even simpler. Using another cronjob the restore function (db-restore-from-snapshot) will run every 2 weeks, 2 hours later than the update process.
The tasks of this functions are:

update_current - it gets the value from the latest parameters and copies it to the current one. Now the value of the documentdb_current_snapshot_id will be updated with the value of documentdb_latest_snapshot_id. The pipeline will check the value of the current parameters. We have split this process with latest and current parameters to facilitate the restore from a snapshot in other cases than the ones described in this article as menioned above.
trigger the deployment pipeline using the main_cicd_name from the SSM Parameter Store

# Trigger a new pipeline deployment
    cicd_client.start_pipeline_execution(
        name=cicd_name
    )

Small bumps

Given the fact that we were able to reuse a lot of the code already written, the only small bump we had was the copy process. A new method of search and a new comparison were also needed in order to have a clean copy process.

Conclusion

Automated processes in the cloud are very flexible and easly reusable. To adapt the existing solution for the automated update and restore process of Aurora DB for DocumentDB aproximately 80% of the code was reused.

Automated database update and restore with AWS Lambda functions for AWS Aurora

Andra Somesan (she/her) — Sat, 14 Aug 2021 07:14:10 +0000

Disclaimer: This article was first published here

The roadmap to cloud adoption can be difficult to set and implement, but once it is completed it offers a lot of flexibility, a lot of space for continuous improvement and a lot of room for creativity to build new solutions.
Having automated processes helps the company to focus on what is important for the business and lets the developers experiment more and efficiently optimize their work.
One of the latest things I’ve been working on involves a task that needed to be continued after the client’s solution was already moved to the cloud. What I liked about it was that it is not only taking the advantage of being in the cloud, it also involves serverless technology that I consider to be the next level of cloud solutions. The task involved both AWS Aurora and AWS DocumentDB databases, and I will make a separate article about the second one since there are some differences to be presented and discussed.

Scenario

Our customer has 2 different AWS accounts:

Acceptance
Production

Solution overview

The solution we agreed on was to split the process in 2 main parts having 2 AWS Lambda functions:

first one, db-update-latest-snapshot-id, will mainly focus on preparing the environment:
- check the latest shared snapshot for Aurora
- update the latest snapshot id parameter in AWS SSM Parameter Store
the second one, db-restore-from-snapshot, will mainly focus on triggering the update and restore process:
- will copy the latest snapshot ID value to the current snapshot ID and then trigger the main pipeline to deploy the changes and restore the database from the latest snapshots I took advantage of the already existing AWS Lambda functions that are sharing the database snapshots from one account to the other. The sharing part is not in the scope of this article but it was illustrated for a better view of the solution.

In AWS SSM Parameter Store, I have created 3 parameters:

ssm_rds_current_snapshot_name
ssm_rds_latest_snapshot_name
main_cicd_name

The current value will be the one from which the database is restored from and the latest value will be the value of the most recent snapshot that exists in the account.
We need to have 2 different parameters, one for the current and one for the latest snapshot, to be able to restore the database from a specific snapshot ID outside of this process and not directly trigger a restore when the pipeline is running but restore on a pre-defined time/date.
There are 2 pipeline in the account, so the name of the main pipeline, the one that is deploying the infrastructure, was needed.

The entire solution was written using IaC in AWS CDK with Python.

How does it work

First lets look at how the parameters are defined in code. We have been using environment variables and a custom function that gets the environment variable value get_optional. There is an existing process of restoring the database from a specific snapshot ARN from the environment variables, and this is why we use the check for the current_rds_backup_snapshot parameter. If it exists it will update the parameter with this value. For the ssm_rds_latest_snapshot_name a dummy value was used because you can’t create an SSM parameter with an empty value.

current_rds_backup_snapshot = get_optional(
    'DATABASE_BACKUP_SNAPSHOT_ID_ARN',
    None
)

ssm_rds_current_snapshot_name = get_optional(
    'SSM_RDS_CURRENT_SNAPSHOT_NAME',
    None
)

ssm_rds_latest_snapshot_name = get_optional(
    'SSM_RDS_LATEST_SNAPSHOT_NAME',
    None
)

main_cicd_name = get_optional(
    'SSM_MAIN_CICD_NAME',
    None
)

resources = []
if current_rds_backup_snapshot:
    current_arn = ssm.StringParameter(
        scope,
        f'{config["stack_name"]}RdsCurrentSnapshotArn',
        parameter_name=ssm_rds_current_snapshot_name,
        string_value=current_rds_backup_snapshot
    )
    resources.append(current_arn.parameter_arn)

rds_ssm = ssm.StringParameter(
    scope,
    f'{config["stack_name"]}RdsLatestSnapshotArn',
    parameter_name=ssm_rds_latest_snapshot_name,
    string_value='dummyvalue'
)
resources.append(rds_ssm.parameter_arn)

main_cicd_ssm = ssm.StringParameter.from_string_parameter_name(
    scope,
    f'{config["stack_name"]}MainPipelineName',
    string_parameter_name=main_cicd_name,
)
main_cicd_name_value = main_cicd_ssm.string_value
resources.append(main_cicd_ssm.parameter_arn)
resources.append(f'arn:aws:codepipeline:*:*:{main_cicd_name_value}')

Using a cronjob the update function (db-update-latest-snapshot-id) will run every 2 weeks at 2 AM.
This function has the following tasks/functions:

get_latest_rds_snapshots - checks for the latest shared snapshot for Aurora DB and it gets the latest one.
update_ssm - it will update the SSM Parameter store with the new values of the latest snapshots in aurora_latest_snapshot_id.

update_current - it gets the value from the latest parameters and copies it to the current one. Now the value of the ssm_rds_current_snapshot_name will be updated with the value of ssm_rds_latest_snapshot_name. The pipeline will check the value of the current parameters. We have split this process with latest and current parameters to facilitate the restore from a snapshot in other cases than the ones described in this article as menioned above.
trigger the deployment pipeline using the main_cicd_name from the SSM Parameter Store

# Trigger a new pipeline deployment
    cicd_client.start_pipeline_execution(
        name=cicd_name
    )

Small bumps

The process of finding out a solution was challenging in each phase of implementation. At the beginning we tried to find a solution that will come on top of the existing one and work to improve it. In the process of implementing the solution with Lambda functions there was the problem of identifying the exact snapshot we need, which of course was just a matter of knowing what you want to extract and reading the documentation. For example, the process of checking the available shared snapshot is looking like this:

rds_shared_snapshots = rds_client.describe_db_cluster_snapshots(
        SnapshotType='shared',
        IncludeShared=True
    )

    # Check available RDS shared snapshots from the source account
    rds_snapshots_source = []
    for snap in rds_shared_snapshots['DBClusterSnapshots']:
        source_account_id = snap['DBClusterSnapshotIdentifier'].split(':')[4]
        if snap['Status'] == 'available' and \
           source_account_id == source_account and \
           snap['EngineVersion'] == rds_engine_version:
            rds_snapshots_source.append(snap)

So we had to select from all the snapshots the ones that are having a specific source account ID, that are available and that are specific for Aurora engine (MySQL or PostgreSQL).

Conclusion

Even with the small bumps we had, this solution is pretty easy to implement in the AWS cloud. The solution not only saves time and money, but also protects the environment against the human errors that can easily happen at that time at night when databases operations are implemented. With the manual task out of the way the developers can focus on other important tasks and also on bringing more automation to the solution as well. Stay tunned for the DocumentDB scenario to see what are the differences and how can you make it work.

Terraform S3 Cross Region Replication: from an unencrypted bucket to an encrypted bucket

Andra Somesan (she/her) — Sat, 24 Jul 2021 09:25:22 +0000

I've wrote this article on 14th of December last year and I thought to share it here as well. I hope it will help :)
Disclaimer: This article was first published here

I’ve been working with Terraform for a few months now, and one of the scenarios that I’ve encountered, that put me in trouble was this:
New client wants to migrate several buckets from the existing account, Ohio region, to the new account, Frankfurt region. This is, of course, no problem for AWS, and this type of migration can be found in a lot of scenarios already explained on the internet. But what was new was that some of the buckets were not encrypted at the source, and at the destination everything must be encrypted to comply with security standards.

What is the issue?

For the Cross Region Replication (CRR) to work, we need to do the following:

Enable Versioning for both buckets
At Source: Create an IAM role to handle the replication
Setup the Replication for the source bucket
At Destination: Accept the replication

If both buckets have the encryption enabled, things will go smoothly. Same way it goes if both are unencrypted.
But if the Source bucket is unencrypted and the Destination bucket uses AWS KMS customer master keys (CMKs) to encrypt the Amazon S3 objects, things get a bit more interesting.

What is the solution?

One of the best advices I have received while working with software for infrastructure as code in AWS, was that if I am going to deploy something new and have troubles with it, one good way to solve it is to go into the AWS console, and try to manually create what I need. This makes things clearer and helps to understand better what it’s needed and how it needs to be modified in order to make it work.
This was the process I followed, and after a few hours of trials and a support ticket with AWS, this was solved with the feedback that, this scenario is ‘tricky’.
The 2 things that must be done, in order to make the CRR work between an unencrypted Source bucket to an encrypted Destination bucket, after the replication role is created, are:

1.In the Source account, get the role ARN and use it to create a new policy. This policy needs to be added to the KMS key in the Destination account.
2.Modify the role to add a new policy to it, to be able to use the KMS key in the Destination account. For this, the KMS key ARN is needed and the policy will look like this:

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Sid": "VisualEditor0",
           "Effect": "Allow",
           "Action": [
               "kms:Decrypt",
               "kms:Encrypt",
               "kms:GenerateDataKey*",
               "kms:ReEncrypt*",
               "kms:DescribeKey"
               ],
           "Resource": "arn:aws:kms:[aws-region]:[account-id]:key/1234abcd-12ab-34cd-56ef-1234567890ab"
       }
   ]
}

How do I put it in code?

Let’s say that the bucket to be replicated is called: source-test-replication, and it is in the Source account, in the Ohio region. The versioning is enabled, and the default encryption is disabled. The bucket in the Destination account is destination-test-replication.

The Terraform code for the normal replication, that creates a KMS key for the new bucket, includes these KMS resources:

resource "aws_kms_key" "replication_s3_kms_key" {
 description = "s3 encryption key"
}

resource "aws_kms_alias" "replication_s3_kms_alias" {
 name          = "alias/replication-s3-key"
 target_key_id = aws_kms_key.replication_s3_kms_key.key_id
}

For this scenario to work, the code needs to me modified and the following information need to be added:

data "aws_iam_policy_document" "kms_policy" {
  statement {
   sid = "Enable IAM User Permissions"
   effect = "Allow"
   principals {
     type = "AWS"
     identifiers = [
       "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"
       ]
   }
   actions = [
     "kms:*"
   ]
   resources = [
     "*"
   ]
 }
 statement {
   sid = "Allow use of the key"
   effect = "Allow"
   principals {
     type = "AWS"
     identifiers = [
       "arn:aws:iam::<Replication role ARN>”
       ]
   }
   actions = [
     "kms:Encrypt",
     "kms:Decrypt",
     "kms:ReEncrypt*",
     "kms:GenerateDataKey*",
     "kms:DescribeKey"
   ]
   resources = [
     "arn:aws:s3:::destination-test-replication"
   ]
 }
}

resource "aws_kms_key" "replication_s3_kms_key" {
 description = "s3 encryption key"
 policy = data.aws_iam_policy_document.kms_policy.json
}

resource "aws_kms_alias" "replication_s3_kms_alias" {
 name          = "alias/replication-s3-key"
 target_key_id = aws_kms_key.replication_s3_kms_key.key_id
}

Both statements are needed, and if you are getting any errors saying something like this:

Error: MalformedPolicyDocumentException: The new key policy will not allow you to update the key policy in the future.

it means that the first statement is missing.

This is all that needs to be done in code, but don’t forget about the second requirement: the policy in the Source account to add to the replication role. For this we need to create this new policy, chose a name, and attach it to the replication role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt",
                "kms:Encrypt",
                "kms:GenerateDataKey*",
                "kms:ReEncrypt*",
                "kms:DescribeKey"
                ],
            "Resource": "arn:aws:kms:us-east-2:<Source Account ID>:key/523b2035-e947-4c71-8690-db6b43589c34"
        }
    ]
}

To wrap it up, for the replication to work in this scenario, the KMS key in the Destination account needs to have a policy to allow the replication IAM role to use it, and the replication role needs to have a policy to use the KMS key in the destination account.

Looking forward

This year at re:Invent, a lot of great things were announced for S3 and I am looking forward to seeing which one will facilitate the automated deployments and which one will be tricky to play with.

How to use AWS Lambda to create CI/CD dependencies with CodePipeline

Andra Somesan (she/her) — Fri, 23 Jul 2021 06:44:50 +0000

Disclaimer: This article was first published here

By trying to fully automate the solutions we work on we are makeing everybody's life much easier. Using the proper tools and services to do so, including AWS CDK and AWS CodePipeline really simplifies the process. When we have started to offer Kubernetes solutions to our customers at my old company, one CodePipeline pipeline was not enough to do the job, because one pipeline with multiple purposes will add complexity to the solution and it will create unwanted dependencies.

Separating the pipelines per purpose, will remove complexity from the pipeline design, will create separate processes and will decouple the components, will be easier to debug, and it will improve the change process by only changing the part you need.

In our case, we wanted to separate the infrastructure deployments, from the applications (pods) deployments for K8s, because we wanted to have all the infrastructure up to date with the latest changes, before we deploy the application. And, if the only modification made is at the application level, there is no need to update the infrastructure as well.

Although having different pipelines for different purposes brings many benefits, it does come with some challenges as well. The main challenge is managing all the pipelines. This is because some of them will require more time to run, and some of them will require less. And, usually when this happens, you might need them to run in a specific order, so the result of one pipeline can be used as an input for the next one, and so on. In order to obtain this, some pipeline dependencies need to be created. To do this in a professional and cost-effective manner, following the recommended best practices from AWS, leads us to use AWS Lambda functions.

Back to our use case, we have the main pipeline, the one responsible for deploying all the underlying infrastructure, and needs more time to complete than the K8s pipeline that is used for the pods deployment. The first one is responsible for updates of the underlying infrastructure, including K8s infrastructure and it should finish to update successfully all the stages before the second one starts running. This way, the K8s pipeline will have the latest updates on the infrastructure and can run safely.

The plan is to create a new stage in the K8s pipeline and add an action to trigger one lambda function. This first lambda function is the main lambda and will be responsible to check if the main pipeline is running, and if it isn’t running, let the process continue.

To control the ScheduledExpression parameter, a cron expression is being used to schedule an execution 5 minutes after the current time.

If the main pipeline is running, it will trigger a CloudWatch Event that will have another lambda function as a target, let’s call it K8s Lambda. The CloudWatch Event will have a rule that is being triggered by the scheduled expression that we talked about above, and it has as target the K8s Lambda function. This function will have as scope to first delete the event, and then to trigger the K8s pipeline again. In order to delete the event, it must first delete the events targets. The event deletion will ensure that we can always trigger the K8s Lambda, 5 minutes after the event was created.

These checks will be done until the main pipeline has finished running (first scenario is being valid), and the Main Lambda will allow: the stage to be validated and the CodePipeline to move to the next stage in the K8s pipeline.

The decision tree looks like this:

Having separate pipelines for different purposes, simplifies any complex solution and brings automation and flexibility to it. You can now achieve decoupled infrastructure, simplified debug process, more control over the changes, to run only what you need, when you need it. As any change, this comes with some challenges. But when you have the right tools and some creativity, these challenges can be transformed in great solutions for the future.