Forem: Jacco Kulman

Safely scripting with boto3

Jacco Kulman — Wed, 22 Nov 2023 13:14:28 +0000

It is very common to set up the AWS credentials used by boto3 and the aws client in ~/.aws/config and ~/.aws/credentials. There are some security risks you need to be aware of though. In this article I will explain risks and mitigation when scripting against your AWS account(s).

The risks involved in working this way are:

long-term credentials eventually leak
the accidental running destructive code
generic roles are overly permissive for specific task
scripting is not infrastructure as code
without mfa this is still single factor auth

I will go into each of these risks below.

Compromised credentials

The credentials often used in this scenario are valid indefinitely. The longer you keep a secret unchanged, the higher the chance of it being compromised. Lately Center for Internet Security (CIS) lowered the required minimum rotation frequency of credentials and secrets from 90 to 45 days in its AWS benchmarks! If you do not rotate the credentials periodically yourself, they will be working as long as your account exists (put a periodic reminder in your calendar!). If you left your laptop unlocked unattended for just a one minute, you should consider the contents of your ~/.aws/credentials compromised, especially if you hang out in places with many cloud engineers 😉.

It is much safer to not use long-term credentials at all. Instead use the short-term credentials you can can on the sso-pages once you set that up:

You can just click the credential exports and paste them in the console you are using to execute the script. These credentials are valid for a limited time and they are only known to the console you paste them in. This is equivalent to using get_session_token in your code.

Be aware that this copy-and-pasting credentials does introduces a new risk in a multi-account scenario. Maybe you have different environments in different accounts. You have no way of seeing for which account you currently pasted the credentials. This means you best to copy-and-paste them with every script execution to prevent a mistake. Otherwise, you might accidentally run a script on the production environment rendering it broken.

Accidentally running destructive code

Coincidentally Joris Conijn mentions this risks in his blog post. It is about accidentally running code that does something destructive without any warning prompt. It doesn't even have to be your own code, any downloaded code, for example in a package initialization, employing boto3 will just try and execute its payload. To mitigate this risk just don't use the default profile, but set up the credentials in a named profile. That way any script not knowing about the profile name will not use it and it will try use the default profile and render an error. (A really nasty script could parse the credentials file though and try to find a working one. The boto3 library will even help the malicious code by supplying boto3.Session().available_profiles).

It could also be your own code that is malicious. I once had a script that would remove the ingress rules of the default SecurityGroup of every Vpc in all the accounts in my organization (this is a good practice). The code contains a condition to limit its destructive payload to only the SecurityGroup that has IsDefault=True. But one commenting # before this if would arm this code to become a threat of immense magnitude to the organization! (I would not even know where to begin recovering if this armed code would accidentally run...)

This also shows the next risk over overly permissiveness.

Overly permissive sessions

Most of the times the permissions you get from the profile or SSO are very generic or maybe even admin. Once you have a session using those credentials you basically script around with a fully loaded rifle to shoot in your own foot. I have found it to be a good practice to scope down the permissions using a session.

I usually have a function in my code for getting the boto3 session like this:

def get_session(role_name: str) -> boto3.Session:
    sts = boto3.Session(region_name='us-east-1').client('sts')
    account_id = sts.get_caller_identity()['Account']
    creds = sts.assume_role(
        RoleArn=f'arn:aws:iam::{account_id}:role/{role_name}',
        RoleSessionName='my_session',
        Policy=json.dumps({ 
            "Version": "2012-10-17", 
            "Statement": [{
                "Effect": "Allow",
                "Action": [
                    "organizations:DescribeOrganization"
                ],
                "Resource": "*"
            }]
        })
    )["Credentials"]
    return boto3.Session(
        aws_access_key_id=creds['AccessKeyId'],
        aws_secret_access_key=creds['SecretAccessKey'],
        aws_session_token=creds['SessionToken']
    )

The only thing required is an IAM role that you can assume with enough permissions for your purpose.

In larger scripts I often even pass in the policy I want for that section of the code. This way each section has its own scoping down to further reduce the risks.

When you start adding code to your script that uses other boto3 calls and you run the code you usually get an error elaborate error message mentioning the action and resource missing in the scoping down policy. You just evaluate them, add them to the policy and run the script again. This way you are constantly aware that you are doing things safely.

Infrastructure as Code

Infrastructure as Code (IaC) is the way to go when creating or changing things hosted in the cloud. There should be source code in a repository indicating the desired state of your infrastructure. Usually people consider CloudFormation a good format to express the desired state in. And ideally you would only want one format. But with CloudFormation you are limited to what it supports, unless you are willing to code or introduce some CustomResources into your infrastructure project. Coding CustomResources is not requires a fair amount of experience though. It might seem easy, but surfing around all the edge cases the CloudFormation process has is sometimes not so intuitive. And the CustomResources come with some responsibility of maintenance as well. The Lambda functions involved often sit around in your AWS account for years. The runtime it uses probably no longer supported when the time comes to destroy your resources. You might need to roll out updates to CustomResources and test wether they still cover your use-case.

That why I often compromise the one format rule. You can add a yaml file to your repo containing some desired state and a post deploy script in python that updates the target AWS account(s) to reflect the state from the yaml file. You set up this script to run in the same pipeline where your CloudFormation is deployed. This can be a lot simpler than juggling with custom resources and I think it is still called "Infrastructure as Code".

Here are some examples where I chose to go for the yaml-script approach (sometimes without yaml):

delete all rules from default security groups in vpcs
(re)setting IAM password policy
configuring the state of SecurityHub controls to ENABLED or DISABLED
updating confluence page with information about your infrastructure
account creation bootstrapping / destruction

I think as long as you put the actual state in separate files (like the yaml) you can still consider it to be Infrastructure as Code.

How to share your organizations ID

Jacco Kulman — Sun, 12 Nov 2023 14:57:48 +0000

As a cloud engineer you sometimes find yourself in a rabbit hole. Typically you fall from one problem into the next, and then the next and so on. Personally I quite enjoy the experience because you never know what you'll learn. The only real problem is that I sometimes forget why I got there in the first place. This is such an occasion.

Here is my reconstruction of how I think I got here: I was trying to implement SSM parameter replication. At a certain point I needed to navigate the Organization tree. Then I found out that I have to create a resource policy on the Organization in the management account. And - if you want to do that using Infrastructure as Code - you can use the AWS::Organizations::ResourcePolicy resource type for that. The AWS documentation mentions the attributes of this resource includes its arn and the arn has the format arn:aws:organizations::111111111111:resourcepolicy/o-exampleorgid/rp-examplepolicyid111. And because this includes the o-exampleorgid which is a good candidate for such replication.

Then I stumbled over a github issue where a feature is discussed for CloudFormation to have an additional pseudo parameter for the Id of the Organization. I decided to dig a little further. Although rabbit holes can be quite deep your are never alone...

In the github issue they are asking for an {AWS::OrgId} pseudo parameter to be implemented. Having seen the ResourcePolicys attribute Arn I thought I had a perfect solution for the stated problem so I decided to share that as a comment on the github issue. Here is my initial take at solving the problem:

(for readers only interested in the final solution and not so much for the narrative: skip this one and scroll down)

Resources:
  OrganizationsResourcePolicy:
    Type: AWS::Organizations::ResourcePolicy
    Properties:
      Content:
          Version: 2012-10-17
          Statement:
            - Sid: AllowReadonlyNavigateOrganization
              Effect: Allow
              Principal:
                AWS: '*'
              Action:
                - organizations:ListRoots
                - organizations:ListOrganizationalUnitsForParent
                - organizations:ListAccountsForParent
              Resource: '*'
              Condition:
                StringEquals:
                  aws:PrincipalOrgID": "${aws:PrincipalOrgID}"
  OrganizationParameter:
    Type: AWS::SSM::Parameter
    Properties:
      Name: /organizations/id
      Value: !Select
        - 1
        - !Split 
          - /
          - !GetAtt OrganizationsResourcePolicy.Arn
      Type: String

With a nifty Select on a Split intrinsic I yank out the Id of the organization and create an SSM parameter of it. If you bootstrap your management account with this you'll have this ssm parameter available for use as parameter default or dynamic parameter in CloudFormation. But... sadly... only in the same region and only in this account. But maybe my upcoming SSM parameter replication feature will save the day! Well... maybe there is an easier solution.

While reading the documentation I saw that AWS::Organizations::Organization is a better candidate to use when solving this because apart from a direct Id attribute it also has the RootId attribute which can be very useful. But this resource cannot be simply used when you already have an Organization in your account. (See how we fell into the next problem here?) To solve this you have to import the Organization resource. To do that you need to create a template where it is the only resource, add a DeletionPolicy and use the import feature.

Resources:
  Organization:
    DeletionPolicy: Retain
    Type: AWS::Organizations::Organization
    Properties:
      FeatureSet: ALL

Once you have done the initial import you can add more resources and update the stack. In this stack you can do all you want with all the attributes the Organization has: Arn, Id, ManagementAccountArn, ManagementAccountEmail, ManagementAccountId and RootId.

So the goal is to let other accounts/regions know about the Id. Going through the list of CloudFormation resources I also encountered StackSet and though that might be a good candidate to do the distributing. After fiddling around a bit I came up with this:

(For the skipping reader you can't use this template directly. You'll need to start with importing the template right above this one)

Parameters:
  Regions:
    Type: CommaDelimitedList
    Default: us-east-1,eu-central-1,eu-west-1,eu-west-2,eu-west-3
Resources:
  Organization:
    DeletionPolicy: Retain
    Type: AWS::Organizations::Organization
    Properties:
      FeatureSet: ALL
  ParameterStackSet:
    Type: AWS::CloudFormation::StackSet
    Properties:
      StackSetName: OrganizationParameterStack
      PermissionModel: SERVICE_MANAGED
      AutoDeployment:
        Enabled: True
        RetainStacksOnAccountRemoval: True
      ManagedExecution:
        Active: true
      StackInstancesGroup:
        - DeploymentTargets:
            OrganizationalUnitIds: 
              - !GetAtt Organization.RootId
          Regions: !Ref Regions
      TemplateBody: !Sub
        - |
          {
            "AWSTemplateFormatVersion": "2010-09-09",
            "Resources": {
              "AtLeastOneResource": {
                "Type": "AWS::CloudFormation::WaitConditionHandle"
              }
            },
            "Outputs": {
              "OrganizationId": {
                "Description": "Id of the Organization",
                "Value": "${OrganizationId}",
                "Export": {
                  "Name": "organizationId"
                }
              }
            }
          }
        - OrganizationId: !GetAtt Organization.Id

It works!!

I decided to go for an output instead of a ssm parameter. Because outputs are a little stricter (better?) You can't change or remove the while they are imported. (This is a feature I am also looking for my ssm parameter replication feature.) Internally CloudFormation does reference counting on them.

A problem with this solution is that the StackSet will not be distributed to the management account itself. Maybe you can get it to do that if you are not using SERVICE_MANAGED but I refused to fall further down ;-)

Note: I did experimented Fn::ToJsonString and have it all in yaml which works but you also need the extra Transform: 'AWS::LanguageExtensions'. Which somehow needs all the IAM warning checkboxes with every deploy. So I decided not to use that. No more time for additional holes to fall into because it is almost time to cook dinner.

You could add many parameters here and they would be shared with the whole organization across regions. Is my ssm parameter replicator feature obsolete now? I think not because this solution still cannot handle my use case: create resource in one account, share an attribute of that resource across (part) of the organization, have the use of it reference counted, handle race conditions correctly. So I guess I will still be stuck implementing it if I can keep out of rabbit holes long enough.

Using undocumented AWS APIs

Jacco Kulman — Sun, 22 Oct 2023 13:04:13 +0000

TL;DR just give me the code

While evaluating some existing IAM policies in a codebase, I found myself repeating the same steps over and over again: navigate Google and search iam actions servicename and look up information about the actions used.

Pro tip: it is much easier to just bookmark this one: https://docs.aws.amazon.com/service-authorization/latest/reference/reference_policies_actions-resources-contextkeys.html

The information I needed for my policy validation work is quite simple:

what are all the services?
what are all the actions a service has?
what actions match a pattern like "Desc*" for a service?
what are mandatory and optional resources for each action?
what condition keys can be used?
is the action readonly, readwrite or something else?

The first place to go to find out if this information is somehow exposed would be the AWS SDKs. I looked through the boto3 documentation on the iam service and came up empty.

After doing a lot of policies using the manual process, I remembered that AWS has a policy-editing tool in the console that seems to be using the information I was looking up manually. So I started my adventure by trying to automate my struggles.

I decided to invest some time in the policy editor, which was using some kind of API I could use to automate some things. So with the Chrome Inspect pane using the network tab, I saw a lot of http requests to:

https://us-east-1.console.aws.amazon.com/iamv2/api/iamv2

After some experimentation and fiddling with cookies and CSRF tokens, I found out how this undocumented API worked. So I cooked up a little Python to automate that. Since it is only 80 lines of code, I'll share it here. I will probably make an installable Python package of it soon. The repository with the code and some examples is https://github.com/binxio/aws-iamv2.

import requests
import json
import boto3
from bs4 import BeautifulSoup

# composePolicy, decomposePolicy, checkMultiMFAStatus, createX509, cuid, generateKeyPairs
methods = { "services"                    : lambda p: None,
            "actions"                     : lambda p: { "serviceName": p, "RegionName": "eu-central-1" },
            "resources"                   : lambda p: None,
            "contextKeys"                 : lambda p: None,
            "globalConditionKeys"         : lambda p: None,
            "getServiceLinkedRoleTemplate": lambda p: { "serviceName": p },
            "policySummary"               : lambda p: { "policyDocument": p },
            "validate"                    : lambda p: { "policy": json.dumps(p), "type": "" } }

class ConsoleSession:
    def __init__(self, boto3_session):
        self._credentials = boto3_session.get_credentials()
        self._signed_in = False
        self._csrf_token = None
        self._cache = {method: {} for method in methods}
        self._rsession = requests.Session()

    def __getattribute__(self, name):
        if name in methods:
            def make_lambda(method, converter):
                return lambda param=None: self.get_api_result(method, converter(param))
            return make_lambda(name, methods[name])
        else:
            return object.__getattribute__(self, name)

    def signin(self):
        token = json.loads(self._rsession.get(
            "https://signin.aws.amazon.com/federation", 
            params={
                "Action": "getSigninToken",
                "Session": json.dumps({
                    "sessionId": self._credentials.access_key,
                    "sessionKey": self._credentials.secret_key,
                    "sessionToken": self._credentials.token
                })
            }
        ).text)["SigninToken"]
        self._rsession.get(
            "https://signin.aws.amazon.com/federation",
            params={
                "Action": "login",
                "Issuer": None,
                "Destination": "https://console.aws.amazon.com/",
                "SigninToken": token
            }
        )
        for m in BeautifulSoup(self._rsession.get(
            "https://us-east-1.console.aws.amazon.com/iamv2/home#",
            params={ "region": "eu-central-1", "state": "hashArgs" }
        ).text, "html.parser").find_all("meta"):
            if m.get("name") == "awsc-csrf-token":
                self._csrf_token = m["content"]
        self._signed_in = True

    def get_api_result(self, path, param=None):
        not self._signed_in and self.signin()
        params = json.dumps(param)
        if self._cache[path].get(params, None):
            return self._cache[path][params]
        self._cache[path][params] = json.loads(self._rsession.post(
            "https://us-east-1.console.aws.amazon.com/iamv2/api/iamv2",
            headers={
                "Content-Type": "application/json",
                "X-CSRF-Token": self._csrf_token,
            },
            data=json.dumps({
                "headers": { "Content-Type": "application/json" },
                "path": f"/prod/{path}",
                "method": "POST",
                "region": "us-east-1",
                "params": {},
                **({ "contentString": params } if params else {})
            })
        ).text)
        return self._cache[path][params]

A few things about this code:

The ConsoleSession class takes a boto3.Session as input. This session needs no actual rights to AWS.
The code might look a bit strange because I wanted to make it as a dynamic class so I only had to add one line to implement an extra API endpoint. This uses the __getattribute__ override and the methods object.
I use the requests module and start a requests.Session() that does much of the heavy lifting handling cookies needed for the http requests to succeed.
To fetch the awsc-csrf-token from the page I use BeautifulSoup
The method in methods were not all discover by me, I did a search on iamv2 on github and found some json files that were already detailing this API.
Since the information retrieved for the API calls I needed does not change per request I implemented a simple caching feature. For some methods like policySummary and validate this might not be optimal.

The actual signing in is done using three HTTP requests:

Getting a SigninToken from signin.aws.amazon.com/federation
Use the token to login to https://console.aws.amazon.com/
Retrieve https://us-east-1.console.aws.amazon.com/iamv2/home# to get the awsc-csrf-token from the page.

Example usage

The code below demonstrates example usage:

import boto3
from iamv2 import ConsoleSession
import re

awssvcs = {}
console_session = None

def get_iam_info():
    global console_session
    boto_session = boto3.Session(region_name='us-east-1')
    console_session = ConsoleSession(boto_session)

    services = console_session.services()
    for service in services:
        name = service["serviceName"]
        if name not in awssvcs:
            awssvcs[name] = { "parts": [] }
        awssvcs[name]["parts"].append(service)

def get_statement_actions(statement):
    result = []
    actions = statement.get("Action") or statement.get("NotAction")
    reverse = "NotAction" in statement
    reverse = not reverse if statement["Effect"] == "Deny" else reverse
    actions = [actions] if isinstance(actions, str) else actions
    for action in actions:
        service, act = action.split(':')
        if "Actions" not in awssvcs[service]:
            awssvcs[service]["Actions"] = console_session.actions(awssvcs[service]["parts"][0]["serviceKeyName"])
        actrgx = act.replace('*', '[A-Za-z]+')
        for svc_action in awssvcs[service]["Actions"]:
            if bool(re.match(actrgx, svc_action["actionName"], flags=re.IGNORECASE)) ^ reverse:
                result.append(svc_action)
    return result

def get_policy_actions(policy):
    for statement in policy["Statement"]:
        yield get_statement_actions(statement)

get_policy_actions will simply list all the actions allowed by the statements in the policy. Here it is in action:

    policy = {
        "Version": "2012-10-17", 
        "Statement": [{
            "Sid": "ReadOnlyCloudTrail",
            "Effect": "Deny", 
            "NotAction": "cloudtrail:De*", 
            "Resource": "*"
        }]
    }

    get_iam_info()
    for statement_actions in get_policy_actions(policy):
        statement_actions = sorted(statement_actions, key=lambda x: x["actionName"])
        for action in statement_actions:
            print(f'{action["actionName"]:40} {", ".join(action["actionGroups"])}')

This will give the following list:

DeleteChannel                            ReadWrite
DeleteEventDataStore                     ReadWrite
DeleteResourcePolicy                     ReadWrite
DeleteServiceLinkedChannel               ReadWrite
DeleteTrail                              ReadWrite
DeregisterOrganizationDelegatedAdmin     ReadWrite
DescribeQuery                            ReadOnly, ReadWrite
DescribeTrails                           ReadOnly, ReadWrite

You can see that this policy allows some actions that are not read-only. You could use this code in tests being evaluated in your pipeline to make sure you never accidentally allow non-readonly actions in this policy.

Future

I will make an installable Python package from the API part. Also, I have some ideas for some neat policy tools:

check for common mistakes in policies.
generate readonly service statements for in a permissions boundary

Regain control over orphaned resources with CDK

Jacco Kulman — Wed, 08 Mar 2023 22:23:37 +0000

Sometimes AWS resources are orphaned. This means the CloudFormation stack is destroyed but the resource is retained. This can happen either because the CloudFormation template specified this wish in the DeletionPolicy or after receiving an error from CloudFormation during the destruction and you decided to delete the stack anyway awknowledging that the resource in question will be retained.

It is entirely possible to work with the orphaned resource because you can always refer to it using its Arn. In CDK you use the from... static methods for doing that. But this will not give you back the control over the properties of the orphaned resource.

Orphaned resources become troublesome to deal with in practice though. One example is that, when the maturity level of you organisation rises, you get confronted with a new rule saying that all your S3 buckets should have certain security measures in place like enforcing SSL access or KMS encryption. Now if also click-ops (managing resources using the AWS console) is banished by your Security Officer, you need a way to regain control over the orphaned buckets via code.

CloudFormation has this feature for a while already, but for CDK project you have to use an experimental feature. Here are the steps you have to take to adopt an ophaned resource:

Decide which application / stack should become the new parent for your resource
Create the resource in that stack using CDK code (the exact properties do not matter for now)
Synth your project and lookup the logical id of the resource in the generated stack in CDK.OUT
Create mapping.json in which you link the logical id to the physical id of the existing resource
Run cdk import -m mapping.json
Lookup the stack in the console and perform a drift detection
Now adjust the code in you CDK project so the resource will match the drift
Run cdk deploy

It is very important to understand that when running step 5 nothing other than the resource addition may have changed in your stack! If more things changed you will get an error message.

You can also work without the mapping.json file, but the cdk import process becomes interactive then, making it unsuitable for in a CI/CD pipeline. The json file should look like this:

{
    "LogIdOfTheResourceYouLookedUp": {
        "BucketName": "my-existing-bucket-name"
    }
}

I only tested this with a bucket, the actual link to the physical id property might differ per resource type.

In a CI/CD pipeline automation you might choose do an cdk import step in stead of a cdk deploy whenever the mapping.json is around. Another thing to note is that this currently only works if you either have a project with one stack or you specify that stack you want to perform the cdk import with. I imagine you could create a stackname-mapping.json and have your pipeline perform the stack selecting.

Joris Conijn already wrote a nice article about migrating resources between CDK stacks without using the experimental feature here.

The experimental cdk feature is described here. The rfc is still open so you might be able to track or influence how it will stabilize.

re:Invent Queueing Guide

Jacco Kulman — Thu, 08 Dec 2022 18:56:10 +0000

Having a great live re:Invent has a lot to do with your success rate at getting to see the sessions you want.

I was quite disappointed when I discovered that all the sessions I wanted to see were already fully booked 6 hours after reserving seats was possible for re:Invent 2022. I felt "lucky" that I could get on the "waiting-list" for 3 sessions...

But on Monday morning freshly arrived in Las Vegas I decided to use my "Lowlands-strategy". Lowlands is a popular music festival in the Netherlands with around 60.000 visitors and 5 big stages and a few smaller ones. If you want to see a good band play there, you have to go at least one hour before the starting time to the venue and camp there with the groupies until the show starts.

On re:Invent this strategy worked out perfectly for me. I never had a full room, because I was always near the front of the walk-up queue. Being one hour in the line did not feel like a huge sacrifice either. I often had my laptop with me and otherwise there was always someone to have a nice "preview"-chat with about the session we were waiting for.

The "downside" of the strategy is that the achieved number sessions per day could be a bit lower than you anticipated. I achieved 3-4 sessions per day this way. But hey I never expected to see any of them :-)

Details of the queue:

60 min before session - start forming 2 queues "walk-up" and "reserved". Try find a spot near a wall so you can give your back a rest.

30 min before session - start of allowing reserved seats

10 min before session - start of allowing in walk-ups

Note: After being let in, you can leave for a toiletbreak and come back in :-) Before being let in you can of course agree with your neighbours that you can have your spot back.

Note: Being on the "waiting-list" for a session does not mean you will have any priority when doors to the room are opened (30 minutes before the start of the session).

Note: If you have a reserved seat and come after 10 minutes before the start of the session: you will be sent to the end of the "walk-up" queue and probably not make it in.

Note: I hope this blog does not inspire everyone to adopt this strategy for next year's re:Invent, because I guess it doesn't work then....

Anyway hope to see you next year on re:Invent ;-)

AWS StepFunction Distributed Map

Jacco Kulman — Thu, 08 Dec 2022 15:57:12 +0000

The last day of re:Invent 2022 still had some nice talks about the newly released features. This one is an extension of the existing Map step that StepFunctions already has.

Why

The existing Map step has some limitations regarding input/output size (256KB), logging the execution history (25000), parallelism (max 40). It is good enough to execute smaller tasks in parallel. If you want larger tasks you can of coarse use S3 as a storage and implement the loading and saving of data yourself in the Lambda function(s) but since this is a quit common pattern AWS decided the extend the functionality of the Map step.

What

When using the new distributed processing mode you can specify a S3 source (either on file, objects with a prefix or the whole bucket). The internals of the Map step will then be treated a a separate workflow.

At the start of the execution of the parent StepFunction the information from S3 is gathered using ListObjectsV2 and then, using the parallellism and batching you specify, the child StepFunction is called. All attributes from ListObjectsV2 will be passed into the child StepFunction.

The maximum parallelism is increased up to 10000 parallel executions of the child StepFunction. The input and output limitations are no longer there. And because the inner StepFunction has its own execution history also this limitation is mitigated.

Other sources than S3

Other sources are currently not supported. In order to use other sources you have to set up some steps in the parent StepFunction to prepare some S3 objects to use as input.

Features

Because the amount of executions can be very large you can specify an error threshold. If the number of errors is lower then the threshold the Map step will still succeed.

In the Map step you can also specify an export location in S3. The results of running the Map step will be reported there (manifest.json, Succeeded.json, Failed.json, Pending.json). You can use a step after the Map step in the parent StepFunction to process these output files to the result you require.

In the console you can follow the progress of your distributed map step.

Use cases

When doing map-reduce like operations on large sets of data this can be a very nice architecture. No need for spark or haddoop instances but a completely serverless solution.

To get a grasp of the size oof the workload you can process using this functionality. In the demo AWS presented a case where over 500000 S3 objects were processed in batches of 250 objects with a paralellism of 3000 parallel lambda executions, each lambda would take 16 seconds to process. The total execution time was under 2:30 minutes.

Kinesis streamed to S3 has a directory structure very suitable to process in this way. If the S3 objects are too large to handle in the lambda function you can use for example the AWS Lambda Powertools to have the lambda process consequtive parts of the objects.

Picture is from a hike we took on the day of arrival in Las Vegas. The location is called Valley of Fire.

AWS Application Composer

Jacco Kulman — Wed, 07 Dec 2022 05:48:38 +0000

On re:Invent 2022 Werner Vogels announced the availability of the preview of AWS Application Composer. I am a big fan of Scratch, a visual programming language to teach young children how to program, so I was very interested to see what this new feature looked like.

Why

According to the presenters a lot of people struggle learning CloudFormation and Infrastructure as Code frameworks in general. This new tool should make it easier for beginning cloud engineers to get started.

What

AWS Application Composer is basically a smart designer canvas on which resources can be dropped and connected. While designing Composer keeps a CloudFormation template in sync on you local development setup using your browsers capability to manage files.

After dropping a resource on the canvas it will automatically be configure with "sane" settings. Also in some cases more than one resource will be created under the hood. The UI hides some more complicated configuration options to not complicate things too much. You can however also go into the template using your favorite code editor and add or change more complicated stuff. Composer will notice changes and update the template accordingly.

Connections between resources take care of several things:

set default IAM policies
set environment variables (for Lambna)
manage event subscriptions

The CloudFormation generated uses the SAM format (Serverless Application Model). Ryan Coleman, who also is the Product Manager for SAM. SAM's smart parameterizable policies are used to keep also the generated CloudFormation readable.

The philosophy seems to be to keep the CloudFormation deployable while designing. That is also why lambda's will be created with some template code already. SAM will take care of packaging and deploying the code.

Demo awesomeness

During the demo at re:Invent Ryan had setup SAM to "watch" the template he was working on. So while dragging and dropping en designing SAM would behind the scene already validate and deploy the design using the generated CloudFormation. This is really great for these kind of demo's if you quickly want to show how things work without going to all the consoles needed to configure the necessary resources.

Current features

import existing CloudFormation templates
grouping resources into named groups (stored in metadata)
lambda scaffolding
managing event subscriptions

Future

The following features are not yet in the product but when asked Ryan said they could be prioritized based on customer feedback:

support for other languages than CloudFormation (CDK, terraform)
working with multiple stacks and the interdependencies
importing SSM parameters and other "lookup"
availability as a plugin for VSCode (and others)
copy/paste on the designer
support for more resources

Conclusion

This tool in its current state is a great way to get started with learning CloudFormation. For quickly demonstrating or prototyping of a serverless architecture the tool is really great.

Probably for maintaining production CloudFormation stacks the product is currently too limited.

I was very impressed though with how carefully this product has been set up. Handling any template including ones with yet unsupported resource types (those will show as read-only resources) and keeping all manual CloudFormation edits in the design seems to be a promise that some day this might become the mature enough to manage all your CloudFormation stacks with.

AWS Inspector for AWS Lambda

Jacco Kulman — Tue, 06 Dec 2022 12:26:35 +0000

Fresh from re:Invent 2022. Just left the pop-up session organized just after the announcement by the AWS Inspector team Rick Anthony and Kashish Wadhwa. After the session was able to ask Rick Anthony a few questions.

Why is this important

More and more customers are moving to serverless workloads on AWS, moving away from EC2 instances and containers. Lambda usage is increasing every year and there are hardly any native security features for this service. This is why it becomes critical to address these vulnerabilities.

What does inspector do

Inspector can already scan EC2 instances and ECR images for vulnerabilities. Now scanning Lambdas is also supported. Security Hub can be configured to accept the findings from Inspector so you can have one place where all security findings will be aggregated.

Inspector decides what and when to scan instead of you scheduling. Also brand new vulnerabilities can be reason for AWS Inspector to decide to rescan. In the case of Lambda this means whenever the function is created or updated a scan will be performed. When a lambda is not called the last 90 days it will no longer be scanned (mothly cost will then also no longer apply). Currently scanning for the following run-times is supported: Python, nodejs and Java. Also scanning layers is supported, findings from layers will be reported for all functions where the layer is used and the finding will indicate that the cause of the vulnerability is from a layer.

Vulnerabilities

Vulnerabilities in Lambda are not related to the operating system (unless you are using custom run-times, which are not supported by the new feature) but come from the packages used for the programming language chosen. If, for example, you are using Java run-time and are using the log4j package you could have a vulnerability if you are using the unpatched version of the library. Inspector scans the artifacts the package tooling leaves behind (package.json for nodejs projects for example).

Exploitability

Not all vulnerabilities in Lambda can be exploited easily because the OS firecracker is well hardened. Also the attack plane is limited because calling a lambda requires IAM authorization when calling it the regular way. If you are using the new function URL functionality though without authentication the attack plane is a little larger. Also if called through API Gateway and you are not using the request validation you could be vulnerable to an attack. Inspector takes into account the network reachability of the lambda in the score of the finding. The resulting score gives an indication of the priority with which this finding should be treated. Currently the AssumeRole policy of the Lambda execution role is not taken into account for the scoring and also the fact that FunctionUrls are used or not. AWS monitors 50 sources that publish vulnerabilities and they add some intelligence to it. The score which also takes into account the exploitabilty. A network scan is also done where the VPC edges are inspected for network reachability. It is correlated with CVE data. If the CVE is only exploitable remotely but the resource is local, the score is lowered.

Live runtimes

Lambda containers are recycle after a period of time so when successfully attacked the attack needs to be re-executed after a recycle. But if the vulnerability involves code injection an attacker can basically infect each newly spun up lambda execution environment. The running lambda environments themselves are not scanned. Like with ECR this is an agentless solution. This also means that if you install extra python packages using a sub-command in python these will not be detected. (You shouldn't do that).

What should you do if there are findings

If you have a vulnerability in a Lambda the following steps should be taken:

inspect the Lambda code if the vulnerable part of the library is actually used. If not you could suppress the finding.
If you find that the vulnerability can be exploited. The lambda needs to be redeployed using patched libraries and packages.

Enabling for the Organization

Enabling Lambda scanning by Inspector for your whole organization is just a few clicks: In the delegated admin account check the checkbox to include lambda scanning (also check the box for auto-enabling for new accounts). Be aware though that the pricing if $0.30 per function per month so you might want to exclude some functions before enabling this for all your accounts.

Current features

nodejs, python and java runtimes
lastest version scanning on update or create
rescanning when there are new threats
AWS Organizations support
network reachability included in score
scoring the vulnerabilities (priority)

Future

For the future the Rick from the Inspector team said they are looking into supporting the following features in the future:

add an option to scan all versions of the Lambda (currently only latest is scanned)
adding lambda accessability (AssumeRolePolicy of the execution role or FunctionUrls) might be included in the scoring calculation
making the stale function scanning period adjustable