Forem: StintriLamah

Leveraging IRSA with AWS SES To Send Emails from EKS Pods

StintriLamah — Sun, 25 Aug 2024 08:16:27 +0000

By the end of the article, readers will have a clear understanding of how to configure their EKS (Elastic Kubernetes Service) clusters to send emails using Amazon SES (Simple Email Service) in a different AWS account, leveraging the power of IRSA (IAM Roles for Service Accounts) to manage permissions securely and efficiently.

Introduction
IRSA and its Benefits
IRSA Configuration
Cross-Account Access
SES Configuration
Testing and Validation
Best Practices and Security Considerations

Introduction

Managing email communications from Kubernetes pods within Amazon EKS can be challenging, especially when Amazon SES is located in a different AWS account. Traditionally, managing IAM credentials and securely configuring permissions across accounts involves complex setups and potential security risks of using Access Keys that can be compromised.

The problem intensifies when developers need to ensure that their applications can send emails efficiently and securely without compromising on the principles of least privilege and access management.

This guide will help developers and DevOps engineers simplify their setup, enhance security, and streamline their email-sending workflows from Kubernetes pods. This is not only limited to SES but other AWS services that permissions can be managed by roles.

IRSA and its Benefits

What is IAM Roles for Service Accounts (IRSA)

IRSA allows Kubernetes service accounts to assume IAM roles, similar to the way that Amazon EC2 instance profiles provide credentials to Amazon EC2 instances. Instead of creating and distributing your AWS credentials to the containers or using the Amazon EC2 instance's role, you associate an IAM role with a Kubernetes service account and configure your Pods to use the service account. This enables pods running on EKS to interact with AWS services securely without embedding AWS credentials in the pods. Applications in a Pod's containers can use an AWS SDK or the AWS CLI to make API requests to AWS services using AWS Identity and Access Management (IAM) permissions.

The following steps explains how IRSA works to assign the pod temporary credentials.

A reference between the EKS Cluster and IAM is established via OIDC. This is a one-time setup per cluster.
A reference between a Kubernetes service account and an IAM Role has to be created.
The Kubernetes resource is configured with an appropriate service account annotation.
As soon a Pod with a service account annotation comes up, the Pod Identity Webhook will be triggered and reconfigure (mutate) the Pod to use IRSA.
The Pod assumes the specified IAM Role and connects to the AWS Security Token Service.
AWS STS verifies the request by contacting AWS IAM.
If the request could be verified and is valid, AWS STS assigns temporary credentials.

Benefits of using IRSA for managing permissions in EKS

Security: No need to store long-term AWS credentials in pods by using Access Keys.
Fine-Grained Access: Assign least privilege permissions to specific workloads. You can scope IAM permissions to a service account, and only Pods that use that service account have access to those permissions.
Credential isolation: A Pod's containers can only retrieve credentials for the IAM role that's associated with the service account that the container uses. A container never has access to credentials that are used by other containers in other Pods.
Auditability: Access and event logging is available through AWS CloudTrail to help ensure retrospective auditing.

IRSA Configuration

Before configuring IRSA, ensure that the EKS cluster has OIDC enabled.

The following configurations will be done on the account where the EKS Cluster and pods are running.

Create IAM Policy

Create an IAM policy that allows sending emails using SES.

{
  "Version": "2012-10-17",
  "Statement": [
    {
            "Effect": "Allow",
            "Action": [
                "ses:SendEmail",
                "ses:SendRawEmail"
            ],
            "Resource": [
                "arn:aws:ses:<REGION>:<SES_ACCOUNT_ID>:identity/<SENDER_EMAIL>",
                "arn:aws:ses:<REGION>:<SES_ACCOUNT_ID>:identity/<RECEIVER_EMAIL>"
            ]
        }
  ]
}

Since we are using the SES Sandbox environment, both the Sender and Receiver needs to be verified on SES.

This is different on the Production environment where the receivers will not need any verification hence on the IAM Policy the arn of the receiver won't be needed.

Create an IAM Role with Trust Policy

Create an IAM role with a trust policy that allows the EKS service account to assume the role.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<EKS_ACCOUNT_ID>:oidc-provider/<OIDC_PROVIDER>"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "<OIDC_PROVIDER>:sub":"system:serviceaccount:<NAMESPACE>:<SERVICE_ACCOUNT_NAME>",
          "<OIDC_PROVIDER>:aud":"sts.amazonaws.com"
        }
      }
    }
  ]
}

Associate the IAM Role with a Kubernetes Service Account

Create a Kubernetes service account and annotate it with the IAM role ARN created.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: <SERVICE_ACCOUNT_NAME>
  namespace: <NAMESPACE>
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<EKS_ACCOUNT_ID>:role/<ROLE_NAME>

Cross-Account Access

Accessing AWS resources across different accounts can be complex due to different security boundaries and policies. The key challenge is to securely grant permissions to SES resources from the EKS cluster in another account.

Cross-Account Trust Policies: Establish trust policies that allow IAM roles in EKS Account to access SES in the different account.
IRSA Configuration: Use IRSA to link EKS service accounts with IAM roles, enabling secure communication.

SES Configuration

Verify Your Domain or Email Address.

In SES we have two environments (Sandbox and Production). All new accounts come configured with the SES Sandbox and you need to request it to be converted to the SES Production. (How to Request SES Production Access)

For the demo I will be using SES SandBox and email address to configure the email identities. I will create two identities one for the sender and the other for the receiver. In SES Sandbox, you need both sender and recipient emails to be verified. This restriction is in place to prevent spam and misuse of the service while you're testing.

When your account has moved out of the sandbox and into production, you can send email to any recipient, regardless of whether the recipient's address or domain is verified. However, you still have to verify all identities that you use as "From", "Source", "Sender", or "Return-Path" addresses.

Create an Authorization policy for Sending Email Identity

Create an authorization Policy for the Sender Email Address identity to allow the Service Account role created to send mails

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<EKS_ACCOUNT_ID>:role/<SERVICE_ACCOUNT_NAME_ROLE>"
            },
            "Action": [
                "ses:SendEmail",
                "ses:SendRawEmail"
            ],
            "Resource": "arn:aws:ses:<REGION>:<SES_ACCOUNT_ID>:identity/<SENDER_EMAIL>"
        }
    ]
}

Testing and Validation

Deploy a sample application in your EKS cluster that uses the service account to send emails via SES.

Use the following deployment file to create a pod that uses an image with AWS CLI so that you can test sending the mail via CLI.

This will create a deployment with one replica in the default namespace and use the Service Account created with the IAM role configured

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ses-test
  labels:
    app: ses-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ses-test
  template:
    metadata:
      labels:
        app: ses-test
    spec:
      serviceAccountName: <SERVICE_ACCOUNT_NAME>
      containers:
      - name: ses-test
        image: amazon/aws-cli
        command: [ "sh", "-c", "sleep 3600" ]

Confirm the pod is running

kubectl get pods | grep ses-test

Verify that your application can send emails by sending a test email

Once the pod is running, execute the below commands to confirm that you can send an email from the pod.

kubectl exec -it <POD_NAME> -- /bin/sh

aws ses send-email \
  --region <REGION> \
  --from "<SENDER_EMAIL>" \
  --source-arn "arn:aws:ses:<REGION>:<SES_ACCOUNT_ID>:identity/<SENDER_EMAIL>" \
  --destination "ToAddresses=[\"<RECEPIENT_EMAIL>\"]" \
  --message "Subject={Data=Test email from EKS pod,Charset=utf-8},Body={Text={Data=This is a test email sent from a pod in EKS Account using SES in a different account.,Charset=utf-8}}"

Best Practices and Security Considerations

Limit Permissions: Assign only the necessary permissions to your IAM roles to follow the principle of least privilege.
Monitor Access: Regularly monitor access and review IAM policies and roles for any changes or anomalies.

AWS Automated Centralized Multi Account Patching

StintriLamah — Mon, 06 May 2024 16:50:28 +0000

AWS Automated Centralized Multi Account Patching

This is a walkthrough on the steps needed to set up the Systems Manager Management Account and Target accounts for Patching.

Setup Resource Groups to Logically Group your Managed Instances
Setup the Required IAM permissions on Management account
Setup the Required IAM permissions on Target account
Create an Automation Document to Execute Patch Baselines
Execute Automation to Patch Multi Account Resources
Schedule Event Bridge Invocation during Patching Window

Many organizations struggle with effectively managing vulnerabilities and patching across their various environments, such as Production, UAT, and Staging. AWS Systems Manager Automation offers a solution by supporting multi-account and multi-Region actions, allowing centralized management of AWS resources. This capability streamlines configuration, operational tasks, and compliance efforts across the enterprise.

In this article, I'll illustrate how to utilize Resource Groups for organizing instances for patching purposes. For instance, you can create Resource Groups for different environments like development, test, and production. Additionally, I'll demonstrate the creation of a custom Automation Document that harnesses Patch Manager. Finally, I'll guide you through executing this custom Automation Document to install patches on your managed instances, which can be scheduled to run during a designated patching Maintenance Window.

Begin by selecting a single account to serve as your management account, alongside a designated AWS Region to act as your management Region. With this chosen management setup, you'll be able to schedule Automation tasks from this central location, directing them towards other AWS accounts and Regions as needed.

Setup Resource Groups to Logically Group your Managed Instances

Resource groups serve as a helpful tool for organizing your AWS resources, providing a streamlined approach to managing and automating tasks across numerous resources simultaneously. By categorizing resources based on their function, such as distinguishing between web servers and databases, resource groups simplify operations and reduce the risk of applying patches incorrectly.

Open your Target account and navigate to the service Resource Groups & Tag Editor.
Create a new Resource Group using tags associated with your managed instances. It's essential to have previously tagged the instances you wish to manage.
Specify the resource type as AWS::EC2::Instance.
For the tags, ensure you've tagged your instances using the designated Tag Key (e.g., PatchGroup) and Tag Value (e.g., 1). This allows for accurate identification and grouping of instances within the resource group.

Preview the Resources and Create the Resource Group

Setup the Required IAM permissions on Management account

From the selected Management account, you need to provision the administrator automation role that will assume the execution roles on the Targeted accounts.

Navigate to CloudFormation Console and Create a stack from AWS-SystemsManager-AutomationAdministrationRole.

Ensure the user you are logged in with should have AmazonSSMAutomationRole attached, iam:PassRole and resource-groups:ListGroupResources actions in order to target Resource Groups and pass the role to the Automation Administrator Role created. Here's an example of an inline policy you can create to achieve this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "resource-groups:ListGroupResources"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:PassRole"
            ],
            "Resource": "arn:aws:iam::<ManagementAccountId>:role/AWS-SystemsManager-AutomationAdministrationRole"
        }
    ]
}

Make sure to replace <ManagementAccountId> in the policy below with the account ID of the management account.

Setup the Required IAM permissions on Target account

To provision the Execution IAM role in the account you wish to target for Automation tasks, you can use the provided CloudFormation template.

Go to CloudFormation Console and Create a stack from the AWS-SystemsManager-AutomationExecutionRole

Provide a name and the <Management Account ID> in the Parameters

Create an Automation Document to Execute Patch Baselines

To create an automation document for executing the AWS-RunPatchBaseline command, follow these steps in the Management Account:

Go to Systems Manager and open Documents section from the left Navigation Menu.
Click on the Create document button to Start Creating an automation document and Select Automation as the document type.

Provide a unique name for the Automation Runbook.
From Automation-RunPatchBaseline, copy the JSON content and replace the default content with JSON content copied.
Create runbook to create the Automation document

Execute Automation to Patch Multi Account Resources

To execute the Automation Runbook on the Management Account:

From the Left Navigation pane on Systems Manager, select Automation
Click on Execute Automation to initiate the execution process.
Choose the Automation Runbook you previously created (e.g., Automation-RunPatchBaseline) and click Execute Automation
Select Multi account and Region

In the Target accounts and Regions section:
- Provide the Account IDs of the targeted accounts and specify the Region where EC2 instances are located
- Specify the Automation Execution Role Name created in the target accounts
- Optionally, specify the number or percentage of locations (account-Region pairs) on which to execute the task simultaneously.
- Optionally, set an error threshold to stop the task after it fails on a specific number or percentage of locations.

In the Targets Section:
- Choose InstanceID as the parameter.
- Select Resource Group as the targets.
- Provide the name of the Resource Group you created earlier as the Resource group

In the Input parameters section:
- For the parameter AutomationAssumeRole, provide the IAM role "AWS-SystemsManager-AutomationAdministrationRole" that you previously created.
- Specify Install as the operation to Scan and Install.
- Optionally, you can provide an "InstallOverrideList" as a list of patches to be installed. This list will override the patches specified by the default patch baseline.
- Optionally, if needed, you can specify the snapshot ID to use for retrieving a patch baseline snapshot.

In the Rate control section:
- Specify the number or percentage of target instances on which to execute the task simultaneously
- Set an error threshold, which will halt the task after it fails on a specific number or percentage of target instances.

Execute the automation.

You can follow these execution steps using the AWS CLI:

Make sure you have the AWS CLI installed and configured with the necessary permissions.

Run the following command to execute the Automation task:

  aws ssm start-automation-execution 
  --document-name "Automation-RunPatchBaseline" 
  --target-parameter-name InstanceId 
  --document-version "\$DEFAULT"
  --parameters '{"AutomationAssumeRole":["arn:aws:iam::<Management Account ID>:role/AWS-SystemsManager-AutomationAdministrationRole"],"Operation":["Install"],"SnapshotId":[""],"InstallOverrideList":[""]}'
  --targets '[{"Key":"ResourceGroup","Values":["Test_Patching"]}]' 
  --target-parameter-name InstanceId 
  --max-errors "1" 
  --max-concurrency "1" 
  --target-locations '[{"Accounts":["<TargetAccountIDs>"],"Regions":["<Region>"],"ExecutionRoleName":"AWS-SystemsManager-AutomationExecutionRole"}]'
  --region <Region>

Replace Management Account ID, TargetAccountIDs, and Region with your actual values.

On the Management account, you can monitor the execution progress of the Automation task. This allows you to track the status of the task and any associated errors or issues. By keeping an eye on the execution progress, you can ensure that the patching process is proceeding as expected and take necessary actions if any problems arise.

On the Target Account, you can observe the execution of the AWS-RunPatchBaseline command triggered by the Automation Document created in the Management Account. This visibility allows you to monitor the progress of the patching operation directly within the targeted environment.

Schedule Event Bridge Invocation during Patching Window

To enhance automation further, you can set up scheduled Patching Maintenance Windows for your organization. This can be achieved by creating a Scheduled EventBridge Rule that triggers a Lambda function. The Lambda function, in turn, initiates the Multi-Account Centralized Automation Flow, automating the patching process at designated times. It will ensure timely and consistent patching across your AWS environment.

This is the flow of tasks:

In the management account, the EventBridge rule is triggered based on the cron or rate-based expression specified.
The EventBridge rule then invokes a Lambda function, which, in turn, initiates a multi-account and multi-Region Automation workflow.
The Systems Manager administrator role assumes the execution role in each target account and Region.
The execution role initiates a Run Command task for AWS-RunPatchBaseline. This command scans for, or installs missing updates on target managed instances based on membership in the provided AWS Resource Group.

You need to deploy the CloudFormation Stack from Scheduled-Patch-Automation to create the following resources: EventBridge rule, IAM service role for Lambda, Lambda function and Automation document to invoke the Command document AWS-RunPatchBaseline.

In the Management account, navigate to the CloudFormation console and create a stack.
Upload the Scheduled-Patch-Automation.yaml template file, and then choose Next.
Specify the stack details:
- For EventBridgeRuleSchedule, enter a cron or rate-based expression for the schedule of the EventBridge rule. For example, cron(30 22 ? * SAT *) schedules the rule to initiate patching on Saturdays at 22:30 UTC. Choose a cron that matches your patching window.
- Optionally modify the ExecutionRoleName to match the Automation execution role in target accounts.
- Specify MaximumConcurrency and MaximumErrors as needed. You can specify a number, such as 10, or a percentage, such as 10%. The default value is 10%.
- Provide the ResourceGroupName that includes the resources you want to target.
- Optionally enter an HTTPS or S3 URL for RunPatchBaselineInstallOverrideList.
- For RunPatchBaselineOperation, choose Scan to scan for missing updates only or Install to scan and install missing updates based on the rules of the patch baseline.
- For RunPatchBaselineRebootOption, choose the reboot behavior for the patching operation. The valid options are RebootIfNeeded and NoReboot.
- Enter the list of TargetAccounts as comma-separated AWS account IDs (for example, 012345678901, 987654321098).
- Optionally modify TargetLocationMaxConcurrency and TargetLocationMaxErrors. The default value is 1.
- Enter the list of TargetRegionIds as comma-separated AWS Region names. (for example, us-east-1, eu-west-1).

Review the stack details, then click Next. Finally, acknowledge that CloudFormation might create IAM resources and click "Create stack".

Once the scheduled time is reached, you'll observe the automation being executed in the AWS Systems Manager Automation console. Here, you'll notice the latest automation document triggered by Lambda and the previous executions initiated manually.