Forem: Alexander Hose

Understanding AWS Instance Metadata Service

Alexander Hose — Thu, 03 Apr 2025 21:21:24 +0000

Introduction

In AWS, roles have become an essential part of securely granting permissions to services without the need for manual key management. Traditionally, in legacy systems or on-premises environments, services like virtual machines would rely on static access keys or service accounts for authentication. But with AWS roles were introduced to simplify permission management and to ensure better security and scalability.

At the heart of AWS's identity and access management for services like EC2 is the Instance Metadata Service (IMDS). IMDS allows an EC2 instance to securely query metadata about itself, including information about its assigned role. Instead of embedding credentials within your application or instance, IMDS allows instances to retrieve temporary, dynamically generated credentials associated with their role - making it a more secure and manageable approach.

What Happens Behind the Scenes?

When you launch an EC2 instance with an attached role, the instance doesn’t need any access keys stored within it. Instead, it can use IMDS to retrieve metadata, including the temporary credentials associated with its IAM role. This is how it works:

1. The EC2 Instance Role

When you create an EC2 instance, you can assign an IAM role to it. This role will define what AWS services the instance can interact with. For instance, if your EC2 instance needs to access S3, you might assign it a policy that grants read or write permissions to specific S3 buckets.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*",
                "s3:Describe*",
                "s3-object-lambda:Get*",
                "s3-object-lambda:List*"
            ],
            "Resource": "*"
        }
    ]
}

2. IMDS Endpoint

IMDS operates over the link local IP address 169.254.169.254. This endpoint is only accessible from within the EC2 instance, ensuring that no external parties can query it. Through this endpoint, the instance can retrieve metadata, such as:

Instance ID
Public and private IP addresses
Region
Availability zone
IAM role credentials

3. Fetching Temporary Credentials

When an EC2 instance needs to interact with AWS services (like S3 or DynamoDB), it does not use hardcoded credentials. Instead, it requests temporary credentials from IMDS. Using the AWS CLI or AWS SDKs (like boto3 for Python), the instance automatically queries IMDS.

Step 1: Checking for Credentials

When running an AWS command (e.g., aws s3 ls), the AWS SDKs and CLI check for available credentials in multiple locations:

2024-12-30 16:50:34,631 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env  
2024-12-30 16:50:34,631 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role  
2024-12-30 16:50:34,631 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file  
2024-12-30 16:50:34,632 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: iam-role

Step 2: Querying IMDS for Credentials

Once AWS determines that an IAM role is available, it makes a request to IMDS for security credentials:

2024-12-30 16:50:34,634 - MainThread - urllib3.connectionpool - DEBUG - http://169.254.169.254:80 "GET /latest/meta-data/iam/security-credentials/" HTTP/1.1" 200 24  
2024-12-30 16:50:34,636 - MainThread - urllib3.connectionpool - DEBUG - http://169.254.169.254:80 "GET /latest/meta-data/iam/security-credentials/EC2SSMSessionManagerRole HTTP/1.1" 200 1582  
2024-12-30 16:50:34,637 - MainThread - botocore.credentials - DEBUG - Found credentials from IAM Role: EC2SSMSessionManagerRole

Step 3: IMDS Returns Temporary Credentials

IMDS responds with temporary credentials assigned to the IAM role:

{
  "Code": "Success",
  "AccessKeyId": "ASIAMFKOAUSJ7EXAMPLE",
  "SecretAccessKey": "UeEevJGByhEXAMPLEKEY",
  "Token": "TQijaZw==",
  "Expiration": "2024-12-30T22:31:56Z"
}

These credentials include:

✅ Access Key ID

✅ Secret Access Key

✅ Session Token (required for authentication)

✅ Expiration Timestamp (after which the credentials must be refreshed)

4. Using Credentials

Once the instance has retrieved the temporary credentials, it automatically uses them to sign API requests. For example, when listing S3 buckets:

2024-12-30 16:55:43,775 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://s3.us-east-2.amazonaws.com/, headers={'User-Agent': b'aws-cli/2.17.18 md/awscrt#0.19.19 ua/2.0 os/linux#6.1.119-129.201.amzn2023.x86_64 md/arch#x86_64 lang/python#3.9.20 md/pyimpl#CPython cfg/retry-mode#standard md/installer#source md/distrib#amzn.2023 md/prompt#off md/command#s3.ls', 'X-Amz-Date': b'20241230T165543Z', 'X-Amz-Security-Token': b'TQijaZw==', 'X-Amz-Content-SHA256': b'e3b0c44206fc1c199afbf4c8996fb92427ae41e4692b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=ASIAMJJSIOAJMJ7EXAMPLE/20241230/us-east-2/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=1b0f7131b8626b61b192f14fdccd5768560b8f3fefca0f60ec9bd2d6cb6b18ad'}>

Here, the request contains:

X-Amz-Security-Token: The session token needed to authenticate API calls
Authorization: A signed request using temporary credentials

Since these credentials expire periodically , the EC2 instance must re-query IMDS to get fresh credentials before making additional API calls.

Why is IMDS Important?

IMDS is a crucial part of how EC2 instances interact securely with AWS services. Without IMDS, you would have to embed access keys within the instance or application, which is error-prone and increases the risk of credentials being exposed. By using IMDS, AWS ensures that credentials are temporary, automatically rotated, and never hard-coded into your applications.

Additionally, since the metadata service is only accessible from the EC2 instance itself (via the link local address 169.254.169.254), this significantly reduces the risk of external attacks trying to access sensitive metadata.

Centrally Managing AWS GuardDuty for an AWS Organization

Alexander Hose — Sat, 22 Mar 2025 18:25:00 +0000

Introduction

Imagine you’re responsible for securing an AWS environment with dozens or even hundreds of accounts. Each account has its own users, permissions, and workloads. One day, you find out that a compromised IAM key in a sandbox account was used to exfiltrate data from an S3 bucket.

The problem?

❌ That account had no GuardDuty enabled — so you never got an alert.
❌ You only found out after the damage was done.

This is exactly why centralized GuardDuty management is critical. Instead of manually enabling GuardDuty for each account, AWS provides a way to automatically monitor security threats across your entire organization.

In this guide, we’ll set up AWS GuardDuty centrally so that every account in your AWS Organization is protected without manual intervention.

Step 1: Enable GuardDuty for AWS Organizations

Before we do anything, we need to allow AWS GuardDuty to integrate with AWS Organizations. This step must be performed in the management account of your organization.

Run this AWS CLI command:

~ $ aws organizations enable-aws-service-access --service-principal guardduty.amazonaws.com

At this point, AWS GuardDuty gains the necessary permissions to manage security across all AWS accounts in your organization. By enabling this service access, you allow GuardDuty to function centrally rather than being configured manually for each account.

To confirm that GuardDuty has been granted access, you can check the list of enabled services with:

~ $ aws organizations list-aws-service-access-for-organization
{
    "EnabledServicePrincipals": [
        {
            "ServicePrincipal": "guardduty.amazonaws.com",
            "DateEnabled": "2025-03-21T22:28:11.629000+00:00"
        }
    ]
}

If guardduty.amazonaws.com appears in the response, GuardDuty now has the necessary permissions and is ready to be configured.

Step 2: Appoint a Security Command Center (Delegate Account)

Now, you need to designate an account as the central security administrator for GuardDuty. AWS requires one account to act as the GuardDuty administrator , where all security findings will be collected.

Typically, organizations use a Security Tooling account for this purpose - a dedicated account that manages security services across all AWS accounts.

To set this up, run the following command in the management account :

~ $ aws guardduty enable-organization-admin-account --admin-account-id 123456789012

By running this command, you specify that the Security Tooling account will now serve as the central location for all GuardDuty alerts and findings. Instead of having GuardDuty findings scattered across multiple accounts, this setup ensures that security teams have a single, consolidated view of all threats.

Step 3: Auto-Enroll All Accounts in GuardDuty

With the administrator account set up, the next step is to ensure that all AWS accounts in your organization are automatically enrolled in GuardDuty.

Switch to the Security Tooling account , then run:

~ $ aws guardduty update-organization-configuration --detector-id 12abc34d567e8fa901bc2d34e56789f0 --auto-enable-organization-members=ALL

This command enables GuardDuty across all existing accounts in the AWS Organization. Additionally, it ensures that any newly created AWS accounts will automatically be added to GuardDuty without requiring manual configuration. This step eliminates the risk of forgetting to enable GuardDuty in a new account, ensuring continuous security coverage across your organization.

💡

It took about 12 hours for me until all my accounts were enrolled

Step 4: Fine-Tune What GuardDuty Detects

GuardDuty offers various security detection features, but not all of them are enabled by default. By default, only standard GuardDuty protection is turned on. You can choose to enable additional features such as S3 Protection, Kubernetes Protection, or Malware Protection.

Step 5: Confirm Everything is Working

Before assuming that everything is configured correctly, it’s important to verify that GuardDuty is monitoring all AWS accounts in the organization.

To check if all accounts have been added to GuardDuty, run the following command in the Security Tooling account or use the UI:

~ $ aws guardduty list-members --detector-id 12abc34d567e8fa901bc2d34e56789f0
{
    "Members": [
        {
            "AccountId": "987654321000",
            "DetectorId": "f1e2d3c4b5a67890abcdef1234567890",
            "MasterId": "111222333444",
            "RelationshipStatus": "Enabled",
            "UpdatedAt": "2025-03-22T00:31:51.728Z",
            "AdministratorId": "111222333444"
        },
        {
            "AccountId": "987654321001",
            "DetectorId": "0a9b8c7d6e5f4321abcdef9876543210",
            "MasterId": "111222333444",
            "RelationshipStatus": "Enabled",
            "UpdatedAt": "2025-03-22T00:31:51.728Z",
            "AdministratorId": "111222333444"
        }
    ]
}

If the output lists all accounts in your organization, this confirms that GuardDuty has successfully enrolled all member accounts and is actively monitoring security threats. Suspended accounts may not be shown.

Step 6: Reviewing GuardDuty Findings in the Central Dashboard

Now that GuardDuty is enabled across all member accounts, we can verify that everything is working as expected by reviewing the findings in the central dashboard.

GuardDuty continuously analyzes activity within AWS accounts and detects potential threats based on anomaly detection, threat intelligence, and machine learning models.

Setting up GuardDuty across all accounts is just the first step. To make the most out of it, you need to centralize logging and alerts to detect and respond to threats effectively.

AWS Cross-Account Roles with Lambda

Alexander Hose — Sun, 16 Mar 2025 20:42:20 +0000

Introduction

When managing an AWS Organization , you often need to execute tasks across multiple AWS accounts. Instead of manually configuring credentials for each account, AWS provides cross-account IAM roles that allow services in one account (such as a Lambda function in the central account) to assume a role in another (the member account). This approach enhances security by avoiding static credentials and provides better control through IAM policies.

In this guide, we will set up a cross-account IAM role in a member account and configure a Lambda function in the central account to assume that role and perform actions, such as listing S3 buckets.

Creating the Lambda Function in the Central Account

To interact with AWS accounts in an AWS Organization , we will deploy a Lambda function in the central account. This function will assume a cross-account IAM role in a member account and execute actions like listing S3 buckets.

1️⃣ Creating the Lambda Function

First, create a new Lambda function using Python in the central account. The default execution role will be automatically generated.

2️⃣ Adjusting the Execution Role

When you create a Lambda function, AWS automatically generates an IAM execution role. By default, this role includes basic Lambda permissions , but it does not have permissions to assume roles in other accounts. To fix this, update the role’s policy to allow sts:AssumeRole:

Navigate to IAM > Roles and find the execution role created for your Lambda function.
Attach the following inline policy to allow assuming roles:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "*"
        }
    ]
}

This policy grants the Lambda function permission to assume any role. For security reasons, it’s best to restrict this to specific roles by replacing "Resource": "*" with the exact ARN of the role in the member account.

3️⃣ Adding an Environment Variable

Now, let's head back to the Lambda console. To make our Lambda function more flexible and easier to maintain, we'll define the IAM role name of the member account as an environment variable , rather than hardcoding it directly in the function. This approach ensures that if we need to update the role in the future, we can do so without modifying the function code itself. It's a simple yet effective way to streamline updates and improve long-term manageability.

4️⃣ Writing the Lambda Function

Now, let’s write the Python code for our Lambda function. This function will:

✅ Retrieve the role name from environment variables.

✅ Assume the cross-account IAM role in the member account.

✅ Use temporary credentials to list S3 buckets in the member account.

Here’s the full code:

import boto3
import os

def lambda_handler(event, context):
    role_name = os.environ['CROSS_ACCOUNT_ROLE_NAME']
    account_id = "123456789012"

    role_arn = f"arn:aws:iam::{account_id}:role/{role_name}"
    session_name = 'LambdaSession'

    sts_client = boto3.client('sts')
    assumed_role_object = sts_client.assume_role(
        RoleArn=role_arn,
        ExternalId='0B7C3A30-4713-4825-8A83-64A15B3B3483',
        RoleSessionName=session_name
    )

    credentials = assumed_role_object['Credentials']

    # Use the temporary credentials to create a new client for the service you want to access
    s3_client = boto3.client(
        's3',
        aws_access_key_id=credentials['AccessKeyId'],
        aws_secret_access_key=credentials['SecretAccessKey'],
        aws_session_token=credentials['SessionToken']
    )

    # Example: List S3 buckets
    response = s3_client.list_buckets()
    if 'Buckets' in response:
        print('Existing buckets:')
        for bucket in response['Buckets']:
            print(f' {bucket["Name"]}')
    else:
        print('No S3 buckets found.')

🔹 Fetching Role Details – The function retrieves the role name from the environment variable and constructs the IAM role ARN dynamically.

🔹 Assuming the Role – Using AWS STS , the function assumes the cross-account role and receives temporary credentials.

🔹 Using Temporary Credentials – These credentials allow the function to interact with AWS services (e.g., S3) in the member account.

🔹 Interacting with AWS Services – It creates an S3 client and lists all available buckets in the member account.

Setting Up the IAM Role in the Member Account

To allow the Lambda function in the central account to access resources in the member account , we need to create an IAM role that grants the necessary permissions.

1️⃣ Generate a Unique External ID

For additional security, AWS recommends using an External ID when allowing cross-account access. This ensures that only your Lambda function can assume the role.

Run the following command to generate a UUID (Universally Unique Identifier):

alexanderhose@Alexanders-Air ~ % uuidgen
0B7C3A30-4713-4825-8A83-64A15B3B3483

We’ll use this as the External ID when creating the IAM role.

2️⃣ Creating the IAM Role

Now, let's create the IAM role in the member account :

For this example, we grant read-only access to S3 in the member account:

In the Permissions policies step, search for AmazonS3ReadOnlyAccess.
Select the policy and click Next.

🚨 Security Tip: In production, create a custom policy with only the necessary permissions instead of using a broad managed policy.

5️⃣ Configuring the Trust Policy

The trust policy is the key component that defines who can assume this role. While it's possible to allow any service or user in the central account to assume the role, a more secure approach is to restrict access to a specific IAM role - such as the service-role/CrossAccountLambda-role-xq53wb4g used by our Lambda function.

This ensures that only the intended function has permission to assume the role, reducing the risk of unauthorized access.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789012:role/service-role/CrossAccountLambda-role-xq53wb4g"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "0B7C3A30-4713-4825-8A83-64A15B3B3483"
                }
            }
        }
    ]
}

Principal – This should point to the Lambda execution role in the central account.
Action – Allows the role to be assumed using sts:AssumeRole.
Condition – Requires the correct External ID to be provided for security.

🎯 Running the Lambda Function in the Central Account

Navigate to AWS Lambda in the central account , select the Lambda function we created earlier, and click Test. You can use an empty test event since our function doesn't rely on any input.

Once the function runs, check the logs in Amazon CloudWatch or the Lambda console output. If everything is configured correctly, you should see a list of S3 buckets from the member account , similar to this:

This confirms that our cross-account IAM role setup is working as expected! 🎉

Common Issues and Troubleshooting

Invalid principal in policy: "AWS":"arn:aws:iam::123456789012:role/CrossAccountLambda-role-xq53wb4g"

Cause: The IAM role’s trust policy has an incorrect principal.

Solution: Ensure that the role ARN includes /service-role/, like this:

"AWS": "arn:aws:iam::123456789012:role/service-role/CrossAccountLambda-role-xq53wb4g"

AccessDenied: User is not authorized to perform sts:AssumeRole

Cause: The Lambda function’s IAM policy does not allow sts:AssumeRole.

Solution: Ensure the Lambda function’s IAM role includes a policy that explicitly grants permission to assume the target role.

{
   "Effect":"Allow",
   "Action":"sts:AssumeRole",
   "Resource":"*"
}

An error occurred (AccessDenied) when calling the ListBuckets operation

Cause: The assumed role does not have permission to list S3 buckets.

Solution: Attach an S3 policy to the assumed role in the member account, such as:

{
    "Effect": "Allow",
    "Action": "s3:List*",
    "Resource": "*"
}

How to Use AWS Inspector and Lambda to Create Scheduled SBOMs 📄

Alexander Hose — Thu, 31 Aug 2023 20:28:00 +0000

Do you want to keep your AWS workloads secure? 🔐

If so, you need to know what software components are running on your workloads. This is where software bills of materials (SBOMs) come in. SBOMs are a comprehensive list of the software components used in a software application. They can be used to track the provenance of software, identify security vulnerabilities , and comply with security regulations.

In this blog article, we will explore how to use Amazon Inspector and Lambda to create scheduled SBOMs. This will allow you to generate SBOMs for your workloads regularly. Afterward, you can incorporate analytics 📊 and alerting ⏰.

Prerequisites 🏗️

Ensure you have the Inspector agent installed on all workloads, which are in scope for SBOM creation.

Create KMS key 🔑

First, you need to create a new KMS key and allow Inspector to use it. Inspector will use this key to encrypt the data. Make sure you create a customer-managed key.

{
   "Sid":"Allow Amazon Inspector",
   "Effect":"Allow",
   "Principal":{
      "Service":"inspector2.<region>.amazonaws.com"
   },
   "Action":[
      "kms:Decrypt",
      "kms:GenerateDataKey*"
   ],
   "Resource":"*",
   "Condition":{
      "StringEquals":{
         "aws:SourceAccount":"<accountID>"
      },
      "ArnLike":{
         "aws:SourceArn":"arn:aws:inspector2:<region>:<accountID>:report/*"
      }
   }
}

You only need to add the region of the principal if you are using a manually activated Inspector. Otherwise, you can use inspector2.amazonaws.com

Create S3 Bucket and adjust bucket policy 🗄️

You want to store your SBOM in a S3 bucket. To do this, you need to create a new bucket and adjust the bucket policy. In the newly created bucket, you need to go to permissions and then click on edit bucket policy. Make sure the allow-inspector section is available in the policy and adjust it based on your settings.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ForceSSLOnlyAccess",
            "Effect": "Deny",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::<bucketName>",
                "arn:aws:s3:::<bucketName>/*"
            ],
            "Condition": {
                "Bool": {
                    "aws:SecureTransport": "false"
                }
            }
        },
        {
            "Sid": "allow-inspector",
            "Effect": "Allow",
            "Principal": {
                "Service": "inspector2.<region>.amazonaws.com"
            },
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:AbortMultipartUpload"
            ],
            "Resource": "arn:aws:s3:::<bucketName>/*",
            "Condition": {
                "StringEquals": {
                    "aws:SourceAccount": "<accountID>"
                },
                "ArnLike": {
                    "aws:SourceArn": "arn:aws:inspector2:<region>:<accountID>:report/*"
                }
            }
        }
    ]
}

You also only need to add the region to the principal if you are using a manually activated Inspector. Otherwise, you can use inspector2.amazonaws.com

Create Lambda function ⚡️

Now we have covered the prerequisites and can move to the fun part. Let's develop the Lambda function. You need to create a new Python function and add the following code:

import json
import boto3
client = boto3.client('inspector2')

def lambda_handler(event, context):
    response = client.create_sbom_export(
        reportFormat='CYCLONEDX_1_4',
        resourceFilterCriteria={
            'ec2InstanceTags': [
                {
                    'comparison': 'EQUALS',
                    'key': 'Criticality',
                    'value': event['Criticality']
                },
            ]
        },
        s3Destination={
            'bucketName': '<bucketName>',
            'kmsKeyArn': 'arn:aws:km:<region>:<accountID>:key/<keyID>'
        }
    )

    return {
        'statusCode': 200,
        'body': json.dumps(response)
    }

We are first choosing the reportFormat. This can be either CYCLONEDX_1_4 or SPDX_2_3. To filter certain instances, we will check the Criticality tag of the instances. Later we can create different schedules for the criticality of the asset. In the last step, we define our s3Destination and need to choose the KMS key we created earlier.

In the permission configuration of your Lambda role, ensure that you allow the inspector2:CreateSbomExport action:

{
   "Sid":"Allow Lambda to create SBOM",
   "Effect":"Allow",
   "Action":[
      "inspector2:CreateSbomExport"
   ],
   "Resource":[
      "*"
   ]
}

Configure Amazon EventBridge Scheduler 🔄

Amazon EventBridge Scheduler is a serverless scheduler that allows you to create, run, and manage tasks from one central, managed service. It is a great way to automate tasks that need to be performed regularly.

To use Amazon EventBridge Scheduler, you first need to create a schedule. A schedule defines the time and frequency at which a task should be run. In your case, I would recommend for example every 7 days.

You can also specify a target for the task. Here you need to choose Lambda and choose your function.

During the invocation of the function, you can also configure a payload. In the Lambda function, we are checking the variable event['Criticality'] which you can define here. Now we can create different schedulers for the different criticality levels of our assets. You could for example check all critical assets daily and non-critical ones every week.

You will now regularly get your SBOM export in the configured S3 bucket 🎉

The Advantages of Scheduled SBOMs 💡

Embracing the synergy of Inspector and Lambda to create Scheduled SBOMs presents benefits to your cybersecurity strategy:

Continuous Compliance: By automating the SBOM generation process, you ensure consistent compliance with security best practices.
Resource Efficiency: AWS Lambda's serverless architecture ensures efficient utilization of resources, optimizing costs while maintaining security standards.
Analytics: You can analyze your SBOM exports and create compliance dashboards with Amazon QuickSight or Amazon Athena.

In Conclusion 🎓

Inspector and Lambda provide an innovative, automated solution to enhance your defense against potential vulnerabilities. By following the steps outlined in this guide, you'll be on your way to creating scheduled SBOMs that bring visibility to your application ecosystem. Take charge of your cybersecurity journey today and ensure your digital assets remain secure tomorrow. 🔐

How To Safeguard Your AWS S3 Bucket Files From Malicious Intruders With VirusTotal

Alexander Hose — Thu, 01 Jun 2023 22:29:31 +0000

AWS S3 is a great storage platform for hosting data. However, it's also a target for malicious intruders. 🎯

Most applications allow the user to upload their own data on S3 buckets. For example, to store images 📷, videos 📹, or any other custom data.

How can you safeguard your AWS S3 bucket from attackers uploading malicious files?

One way to safeguard your AWS S3 bucket files is to use VirusTotal to scan them for malware. 🔍 This free online service allows you to quickly scan files for viruses and other malicious content.

In this article, we will create lambda functions who are performing all the logic to scan the files and merge everything together in a Step Function.

Create S3 bucket 🚧

In the first step, we create a new bucket or choose an existing one. The bucket configuration doesn't matter, but we need to ensure that the Amazon EventBridge notifications are enabled. We later use this to trigger our Step Function. You can enable this in the properties of your bucket underneath Event notifications.

Generate VirusTotal API key ✨

Next, we go to https://www.virustotal.com/ and create an account. Please make sure that the free account is only a limited version and that you are not allowed to use it for business workflows, commercial products, or services.

After the successful registration, we can navigate to our profile https://www.virustotal.com/gui/user/username/apikey and obtain the API key.

Add VirusTotal API key to AWS Secrets Manager 🔑

We take the API key from the previous step and create a new AWS Secrets Manager secret. For later reference, we need to note down the secret name. The key itself should be named api_key and the secret virustotal_api_key.

Create IAM role 🛡️

We need to create an IAM role with access to the encryption keys for the AWS Secrets Manager and access to the S3 bucket. For the S3 access, I have attached the AmazonS3FullAccess role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt",
                "secretsmanager:GetSecretValue",
                "ssm:GetParameter"
            ],
            "Resource": "*"
        }
    ]
}

🚨 Make sure to adjust the policies to reflect the least privilege principle

As we attach the role to Lambda, we need to enable a trusted relationship:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "lambda.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Create lambda functions 💡

We need to create two Lambda functions. The first one will upload the file to VirusTotal and the second one will get the results of the scan. Please make sure that both functions have the AWS-Parameters-and-Secrets-Lambda-Extension layer added. You can do that from the bottom of the Code section in your lambda function.

VirustotalScan

import json
import os
import urllib3
import time
import boto3

s3Client = boto3.client('s3')

url = "https://www.virustotal.com/api/v3/files"

def lambda_handler(event, context):
    secret_id = "virustotal_api_key"
    secret_key = "api_key"
    auth_headers = {"X-Aws-Parameters-Secrets-Token": os.environ.get('AWS_SESSION_TOKEN')}

    http = urllib3.PoolManager()
    r = http.request("GET", "http://localhost:2773/secretsmanager/get?secretId=" + secret_id, headers=auth_headers)

    parameter = json.loads(r.data)
    VIRUSTOTAL_API_KEY = json.loads(parameter["SecretString"])[secret_key]
    headers = {
        "accept": "application/json",
        "x-apikey": VIRUSTOTAL_API_KEY,
    }

    bucketName = event['detail']['bucket']['name']
    objectName = event['detail']['object']['key']
    fileName = event['detail']['object']['key'].split('/')[-1]
    s3file = s3Client.get_object(Bucket=bucketName, Key=objectName)
    s3ContentType = s3file['ContentType']
    s3file = s3file['Body'].read()

    files = {"file": (fileName, s3file, s3ContentType)}
    response = http.request('POST', url, headers=headers, fields = files)
    analysisURL = json.loads(response.data)["data"]["links"]["self"]
    return {
        'analysisURL': analysisURL,
        'bucket': bucketName,
        'key': objectName
    }

This function will first get the API key from the AWS Secrets Manager.

To call the VirusTotal API, we need to always add the API key into the header of the HTTP call:

headers = {
    "accept": "application/json",
    "x-apikey": VIRUSTOTAL_API_KEY,
}

Next, we want to get the file that was uploaded to S3 and save it in the s3file variable. We can retrieve this information from the event variable. This variable is populated by the service triggering the lambda function. Additionally, we want to save the file type in the s3ContentType variable.

bucketName = event['detail']['bucket']['name']
objectName = event['detail']['object']['key']
fileName = event['detail']['object']['key'].split('/')[-1]
s3file = s3Client.get_object(Bucket=bucketName, Key=objectName)
s3ContentType = s3file['ContentType']
s3file = s3file['Body'].read()

Now we can send the file to VirusTotal for the scan.

files = {"file": (fileName, s3file, s3ContentType)}
response = http.request('POST', url, headers=headers, fields = files)
analysisURL = json.loads(response.data)["data"]["links"]["self"]

As the scan can take some time, we return the unique URL of the scan and the S3 file information. This information will be passed to the next lambda function.

return {
    'analysisURL': analysisURL,
    'bucket': bucketName,
    'key': objectName
}

VirustotalResults

import json
import os
import urllib3
import time
import boto3

s3Client = boto3.client('s3')

def lambda_handler(event, context):
    secret_id = "virustotal_api_key"
    secret_key = "api_key"
    auth_headers = {"X-Aws-Parameters-Secrets-Token": os.environ.get('AWS_SESSION_TOKEN')}

    http = urllib3.PoolManager()
    r = http.request("GET", "http://localhost:2773/secretsmanager/get?secretId=" + secret_id, headers=auth_headers)

    parameter = json.loads(r.data)
    VIRUSTOTAL_API_KEY = json.loads(parameter["SecretString"])[secret_key]
    headers = {
        "accept": "application/json",
        "x-apikey": VIRUSTOTAL_API_KEY,
    }

    response = http.request('GET', event['analysisURL'], headers=headers)
    response = json.loads(response.data)
    status = response['data']['attributes']['status']
    if status != "queued":
        suspicious = response['data']['attributes']['stats']['suspicious']
        malicious = response['data']['attributes']['stats']['malicious']
        if(malicious > 0):
            s3Client.put_object_tagging(
                Bucket= event['bucket'],
                Key= event['key'],
                Tagging={
                    'TagSet': [
                        {
                            'Key': 'malicious',
                            'Value': 'true'
                        },
                    ]
                }
            )
        else:
            s3Client.put_object_tagging(
                Bucket= event['bucket'],
                Key= event['key'],
                Tagging={
                    'TagSet': [
                        {
                            'Key': 'malicious',
                            'Value': 'false'
                        },
                    ]
                }
            )
        return {
            'status': 'sucess',
            'suspicious': suspicious,
            'malicious': malicious
        }
    else:
        return {
            'status': 'failed'
        }

This lambda function will retrieve the results of the VirusTotal scan. First, we take the analysis URL from the previous function. It is stored in the event variable. If we call the analysis URL we get information about the status of the analysis and the results of the different scanners.

response = http.request('GET', event['analysisURL'], headers=headers)
response = json.loads(response.data)
status = response['data']['attributes']['status']

The response from VirusTotal will look like this:

{
   "meta":{
      "file_info":{
         ...
      }
   },
   "data":{
      "attributes":{
         "date":1684306866,
         "status":"completed",
         "stats":{
            "harmless":0,
            "type-unsupported":16,
            "suspicious":0,
            "confirmed-timeout":0,
            "timeout":0,
            "failure":0,
            "malicious":0,
            "undetected":59
         },
         "results":{
            "Bkav":{
               "category":"undetected",
               "engine_name":"Bkav",
               "engine_version":"2.0.0.1",
               "result":"None",
               "method":"blacklist",
               "engine_update":"20230516"
            },
            "Lionic":{
               ...
            }
         }
      },
      "type":"analysis",
      "id":"YjdkNjE5MTM4MDViNGNlMmFhZTZkYmE2MzJmMDg1ZjM6MTY4NDMwNjg2Ng==",
      "links":{
        ...
      }
   }
}

If the scan is not finished yet, the status message will be queued otherwise it will state completed ($.data.attributes.status). If the status is queued we will exit the function and return the status as failed.

if status != "queued":
    ...
else:
    return {
        'status': 'failed'
    }

If the scan is completed we will tag the S3 object. To identify how to tag the S3 object, we take the number of scanners that flagged the file as suspicious or malicious. In my case, I just check if the malicious count is higher than 0 and would flag the file as malicious. You can adjust the logic to fit your business purpose.

suspicious = response['data']['attributes']['stats']['suspicious']
malicious = response['data']['attributes']['stats']['malicious']
if(malicious > 0):
    s3Client.put_object_tagging(
        Bucket= event['bucket'],
        Key= event['key'],
        Tagging={
            'TagSet': [
                {
                    'Key': 'malicious',
                    'Value': 'true'
                },
            ]
        }
    )
else:
    s3Client.put_object_tagging(
        Bucket= event['bucket'],
        Key= event['key'],
        Tagging={
            'TagSet': [
                {
                    'Key': 'malicious',
                    'Value': 'false'
                },
            ]
        }
    )
return {
    'status': 'sucess',
    'suspicious': suspicious,
    'malicious': malicious
}

Create Step Functions 🔄

Now we can put everything together and create the Step Function logic. The Workflow editor makes it very easy. Just drag and drop the correct functions to reflect the below flow. You can adjust the wait time accordingly. Usually, a wait time of 60 seconds is sufficient.

Create EventBridge rule ⏰

Let's create the EventBridge rule now. First, we need to define the event pattern. You can define the detail type and bucket:

{
  "source": ["aws.s3"],
  "detail-type": ["Object Created"],
  "detail": {
    "bucket": {
      "name": ["virustotal"]
    }
  }
}

Next, we define the previously created Step Function as our target.

Every time we upload a new object to our S3 bucket the file will be scanned via VirusTotal and tagged accordingly.

Conclusion 🎉

Integrating VirusTotal into your AWS S3 bucket security strategy can provide an additional layer of protection. 🔒 VirusTotal's comprehensive threat detection capabilities, combined with the step-by-step guide we have provided, can help you detect and mitigate potential risks effectively. 🔍

Remember to configure access control and permissions properly, implement encryption, enable logging and monitoring, conduct regular audits and vulnerability assessments, implement two-factor authentication, utilize AWS security services, educate users, and follow security best practices. These measures will enhance the overall security of your AWS S3 bucket files and reduce the risk of malicious intrusions. 🚧

By prioritizing data security and leveraging tools like VirusTotal, you can have peace of mind knowing that your AWS S3 bucket files are well-protected from potential threats.