Forem: James Monek

Securing Data Lake in AWS

James Monek — Tue, 14 Mar 2023 23:47:04 +0000

Cloud security has always been a challenge for moving resources to the cloud. Particularly with constant negligence in leaving S3 buckets and data open to the public. With massive amounts of data available and the need to make data driven decisions, Amazon offers many services to build and utilize a data lake in the cloud. How can one make the move to building a data lake in the cloud while still maintaining security? Fortunately, Amazon has a service for that. Well, actually multiple services.

For the purpose of this article, we are going to assume that you will create and store your data in S3 buckets.

Getting some of the basics out the way

For starters, let's get some of basic best practices out of the way.

Key Management Service

Absolutely encrypt everything, encryption at rest and encryption in transit. You'll want to utilize Key Management Service to generate keys to be used for your data lake. I suggest creating operational keys for data in CloudTrails, SNS notifications, RDS, and S3 buckets for services such as AWS Config and VPC Flow Logs. This ensures that if sensitive data is accidentally placed in logs and communications, it is still encrypted. Use different KMS keys for data in the lake. Yes, rotate them at least annually if not more frequently.

S3

Misconfigured S3 buckets being left open to the world is the root of many breaches. Until recently, the best practice is to force blocking all public access at the account level.

aws s3control put-public-access-block \
--account-id AccountId \
--public-access-block-configuration '{"BlockPublicAcls": true, "IgnorePublicAcls": true, "BlockPublicPolicy": true, "RestrictPublicBuckets": true}'

Amazon recently announced making the default setting for S3 buckets is deny public access, https://aws.amazon.com/blogs/aws/heads-up-amazon-s3-security-changes-are-coming-in-april-of-2023/. I still recommend the using the S3 control and absolutely monitor for changes in those settings which I will cover later.

While Server Side Encryption works fine, I would ensure that all your buckets are encrypted using KMS for extra protection for your lake.

Also, ensure you are enforcing secure connections only (SSL). More information at https://repost.aws/knowledge-center/s3-bucket-policy-for-config-rule

EC2

If you are using EC2 services as part of your lake, I recommend taking measures to ensure all EBS volumes are encrypted by default.

aws ec2 enable-ebs-encryption-by-default

Check for encryption by default on other services you may use such as AWS RDS.

CloudTrails

Everything you do in AWS, whether it's in the console, command line interface, or CloudFormation templates, you are accessing an API. All API calls get logged to CloudTrails. My suggestion here is to create CloudTrails to be sent to S3 buckets and CloudWatch log group. If you have AWS Organizations, then use Organization CloudTrail.

I'm suggesting S3 bucket to allow for your Security Information and Event Management (SIEM) tool to be able pull from the bucket and then you can easily search for, develop dashboards, and alarms based on that data. Prime tools in this area are Elastic Search-Logstash-Kibana (ELK Stack), Splunk, and AWS Opensearch.

Sending them to CloudWatch allows for you to create CloudWatch Alarms based on events. There are a bunch of events you can trigger on such as unauthorized login attempts, changes to security groups, etc. One I like to monitor for is using the root account, which should never be used. More information can be found at https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudwatch-alarms-for-cloudtrail.html.

Guard Duty

One of the easiest services to enable is Guard Duty and it provides intelligent threat detection. It constantly monitors your buckets, workloads, and accounts for anomalies. Findings can also be sent to Security Hub.

AWS Config

AWS Config assess, audits, and evaluates configurations for your resources. It can detect when there are configuration drifts and alert you. At the core, Security Hub compliance packs use AWS Config to check for hundreds of compliance rules. In combination with Security Hub, I've found the tool helpful in getting your account set up securely based on your rules and then audit when configurations change. Also, remember the settings for blocking public access to S3 bucket? Set up configs to monitor changes in these settings.

Cloudwatch

Earlier in the article, I mentioned enabling CloudTrails and sending this to CloudWatch log group. The reason why I suggest doing so, is that you can create alarms based on data in these logs. Unauthorized access to resources, logins, AWS config compliance changes, policy changes, and so on. More information on creating these types of alarms is at https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Create_alarm_log_group_metric_filter.html.

Security Hub

The best way I can describe Security Hub is a central location for the state of securing your account. It contains pre-built compliance packs with rules to secure your resources. Other AWS services such as Guard Duty, AWS Config, Firewall Manager sends findings to security hub. 3rd party tools such as Prowler can also send findings to Security Hub.

When you first start with Security Hub, you'll be overwhelmed with findings. You'll want to assess those findings and correct the ones that makes sense. If you discover ones that you are comfortable with, you can suppress them. You'll want to suppress with a note and the way I handle this is through the CLI, which can be automated, https://docs.aws.amazon.com/cli/latest/reference/securityhub/batch-update-findings.html

Security Hub re-evaluates on its schedule so there is no way to know how long it will take. Handy tip is to use the CLI to force a manual check. Security Hub will throttle you on the number of manual checks in a timeframe. More information is at https://docs.aws.amazon.com/config/latest/developerguide/evaluating-your-resources.html#evaluating-your-resources-console.

Once you get Security Hub in a cleaned state, you'll want to start with setting up notifications using your preferred channel whether that is email or Slack channels, https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub-cloudwatch-events.html

Prowler

I can't speak about AWS security without mentioning Open Source Prowler which is a tool you can run against your account to generate security findings. It has hundreds of compliance rules. I have a write-up on how to use Prowler along with a CloudFormation script to deploy. The article is at https://dev.to/aws-builders/automating-prowler-for-compliance-checking-in-aws-3oef

Wrapping it all up

By no means is this article a complete list of what you can do to secure your data lake in the cloud. I haven't covered all the services I used or even touched on listed of compliance checks you can deploy. Use this as a starting point.

One final comment about securing your environment. Always use CloudFormation templates, CDK, or CLI to develop your Infrastructure as Code. Avoid doing any of this in the console. This allows for all your work to be portable to other accounts. Trust me, it's worth the investment up front.

Lambda with Oracle Layers

James Monek — Tue, 24 Jan 2023 00:29:58 +0000

Overview

Recently, I came across a need to finally use Lambda in AWS. I was looking to query an Oracle database and export the Oracle table to a CSV file. While using Python I realized the Oracle client is not available in Lambda so I needed a way to add libraries and dependancies in order to execute my code. Lambda layers allows you to do this. You can add several layers to your Lambda function. This article documents how to create the Oracle Layer for Python in Lambda with the required libraries.

The main limitation that I discovered using Lambda layers is the size limit of 250 megs for all your code and libraries. Fortunately, I was just under this limitation but there are several ways to get around this if you run into this problem. One is to break your code into various micro services. The other is to to use EFS and place all of your libraries in the EFS volume and mount the volume on your Lambda function.

Building the Oracle Layer

Optional, but preferred, commission an EC2 instance so you can start from a clean environment.

Verify the Python version

python3 --version

Create your project directory

mkdir project

Create a directory called python

mkdir python

Make library directory

mkdir lib

Install cx_Oracle Python library

pip3 install cx_Oracle -t python/

Download the latest Oracle Client from https://www.oracle.com/database/technologies/instant-client/linux-x86-64-downloads.html

wget https://download.oracle.com/otn_software/linux/instantclient/218000/instantclient-basic-linux.x64-21.8.0.0.0dbru.zip -O oracli.zip

Unzip the Oracle Client into the library folder

unzip -j oracli.zip -d lib/

Ensure Libaio is insalled on the EC2 instance (it should be)

sudo yum install libaio -y

Copy Libaio to library

cp /lib64/libaio.so.1 lib/libaio.so.1

Zip up all your libraries to be used for the Lambda Layer

zip -y -r oracletable.zip python/ lib/

Copy your zipped library file to a S3 bucket so that the Lambda function can access it.

Adding Layer into Lambda

Copy the url to be use when creating the layer
Go to lambda and create a layer using AWS Lambda Layer documentation.
Create the Lambda function using AWS Lambda Documentation.
Add your code
Add the layer your created
Run the Lambda function
Using CloudFormations

Below is an example of the CloudFormation code to create a Lambda Layer and Lambda function.

Note: Amazon Python Power Tools Layer documentation is at https://awslabs.github.io/aws-lambda-powertools-python/2.6.0/

Role:
  Type: 'AWS::IAM::Role'
  Properties:
    RoleName: lambda-role-name
    AssumeRolePolicyDocument:
      Version: "2012-10-17"
      Statement:
        - Effect: Allow
          Principal:
            Service:
              - lambda.amazonaws.com
              - s3.amazonaws.com
          Action:
            - 'sts:AssumeRole'
    Path: /
    Policies:
      - PolicyName: lambda-policy
        PolicyDocument:
          Version: "2012-10-17"
          Statement:
# Add access to secrets manager to give access to Lambda function to use
            - Effect: Allow
              Action: 
                - secretsmanager:GetResourcePolicy
                - secretsmanager:GetSecretValue
              Resource: 'arn:aws:secretsmanager:<region>:<account>:secret:<secretname>'
            - Effect: "Allow"
              Action:
                - s3:ListBucket
              Resource:
                - !Sub 'arn:aws:s3:::<S3 Bucket>'
            - Effect: "Allow"
              Action:
                - s3:GetObject
                - s3:PutObject
                - s3:PutObjectAcl
              Resource:
                - !Sub 'arn:aws:s3:::<S3 Bucket>/*'
                - !Sub 'arn:aws:s3:::<S3 Bucket>/*'
    ManagedPolicyArns:
      - 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'
      - 'arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole'

# Create the Oracle Layer for Python
OrcaleLayer:
  Type: AWS::Lambda::LayerVersion
  Properties:
    CompatibleRuntimes:
      - python3.7
    Content:
      S3Bucket: <S3 Lambda Bucket with code>
      S3Key: <layer ip file>
    Description: "Oracle Layer"
    LayerName: oracle-layer

# Create Lambda Function
OracleFunction:
  Type: AWS::Lambda::Function
  Properties:
    FunctionName: oracle-lambda
    Handler: oraclelambda.lambda_handler
    Layers:
# Oracle Client Library Layer
      - !Ref OracleLayer
# Add Python Power Tools
      - 'arn:aws:lambda:us-east-1:017000801446:layer:AWSLambdaPowertoolsPythonV2:13'
    Role:
      Fn::GetAtt:
        - Role
        - Arn
    Environment:
#Pass Secret Name, Target Bucket, and S3 Prefix to the Python code
      Variables:
        SecretName: <secret name>
        TargetBucket: <target bucket>
        S3Prefix: <target prefix>
# Location of the Python Code
    Code:
      S3Bucket: <S3 Lambda Bucket with code>
      S3Key: <pthon code zip file>
    Runtime: python3.7
    Timeout: 90
    TracingConfig:
      Mode: Active
    VpcConfig:
      SecurityGroupIds: <security group id>
        - !Ref SecurityGroupId
      SubnetIds:
        - <subnet 1>
        - <subnet 2>

What’s next?

Currently, the process for updating the layers and code is very manual. I’m looking to find ways to automate getting the CloudFormation templates and Lambda to get updated when new changes are made.

Automating Prowler for Compliance Checking in AWS

James Monek — Sun, 27 Mar 2022 18:15:23 +0000

Whether you are looking to improve your AWS security posture or checking compliance against cybersecurity frameworks, Prowler is an amazing open source tool developed by Toni de la Fuente. Toni has created a tool to check over 200 security controls in AWS ranging from ensuring S3 buckers are not publicly accessible to encryption everywhere.

Toni's Github portal provides extensive documentation on how to use the tool, but I wanted to share a CloudFormation template that I created to automate the deployment in AWS to run compliance checks and then decommission the stack and remove all resources.

Launching EC2 Instance for Prowler

First, we will want launch an EC2 instance and run a bash script to download the necessary software, install, and configure Prowler.

  ProwlerInstance:
    Type: 'AWS::EC2::Instance'
    Properties:
      ImageId: !Ref ImageId
      InstanceType: !Ref InstanceType
      SubnetId: !Ref SubnetId
      SecurityGroupIds:
        - !Ref InstanceSecurityGroup
      KeyName: !Ref KeyName
      IamInstanceProfile: !Ref ProwlerInstanceProfile
      Tags:
        -
          Key: Name
          Value: Prowler
      BlockDeviceMappings:
        - DeviceName: /dev/xvda
          Ebs:
            VolumeSize: 8
            Encrypted: true
# Run bash to install and configure Prowler
      UserData:
        Fn::Base64:
          !Sub |
            #!/bin/bash -xe
            sudo yum update -y
            sudo yum remove -y awscli
            cd /home/ec2-user
            curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "/home/ec2-user/awscliv2.zip"
            unzip /home/ec2-user/awscliv2.zip
            sudo /home/ec2-user/aws/install
            sudo yum install -y python3 jq git
            sudo pip3 install detect-secrets==1.0.3
            git clone https://github.com/prowler-cloud/prowler /home/ec2-user/prowler
            chown -R ec2-user:ec2-user /home/ec2-user/prowler

Create an instance profile

Create an instance profile tied to a role with necessary permissions to run the audit.

  ProwlerInstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      InstanceProfileName: prowler-ec2-instance-profile
      Path: /
      Roles:
       - !Ref ProwlerEc2InstanceRole

Provide access to run Prowler

Next we will want to generate a role that has view-only and security audit permission that is required by Prowler to run compliance checks.

  ProwlerEc2InstanceRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: prowler-ec2-instance-role
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          -
            Effect: Allow
            Principal:
              Service:
                - ec2.amazonaws.com
            Action:
              - sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/SecurityAudit
        - arn:aws:iam::aws:policy/job-function/ViewOnlyAccess
      Path: /

Security Group

We'll want to create a security group to only allow SSH access into the EC2 instance.

  InstanceSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
        GroupDescription: Allow ssh from specific host
        GroupName: ProwlerSecurityGroup
        VpcId: !Ref VpcId
        SecurityGroupIngress:
          - IpProtocol: 'tcp'
            FromPort: '22'
            ToPort: '22'
            CidrIp: !Ref CidrIp

Parameters

Lastly, to improve automation, we will pass parameters into the CloudFormation template. If you launch the template via the console, some of these settings will be selected via a dropdown. For launching via the command-line interface, pass the parameters through a JSON file.

ImageId : Default is AWS Linux 2 ami-0e1d30f2c40c4c701

InstanceType : Default is t3.micro

VpcId : VPC to launch EC2 instance into

SubnetId : Subnet for EC2 instance

KeyName : Keypair to use

CidrIp : CIDR range for SSH x.x.x.x/x

Parameters:
  ImageId:
    Type: String
    Description: AMI - Linux 2
    Default: 'ami-0e1d30f2c40c4c701'
  InstanceType:
    Type: String
    Description: Instance type to be used - t3.micro default
    Default: t3.micro
  VpcId:
    Type: AWS::EC2::VPC::Id
    Description: VPC to be used
  SubnetId:
    Type: AWS::EC2::Subnet::Id
    Description: Subnet to be used
  KeyName:
    Type: AWS::EC2::KeyPair::KeyName
    Description: Keyname
  CidrIp:
    Type: String
    Description: CidrIp to be used to connect from x.x.x.x/x
Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      -
        Label:
          default: "Network Configuration"
        Parameters:
          - ImageId
          - InstanceType
          - VpcId
          - SubnetId
          - KeyName
          - CidrIp

Final YAML Script

After putting all this together. The final YAML scripts looks like the following. The code is also available at Github.

AWSTemplateFormatVersion: "2010-09-09"
Description: "Create EC2 instanace with Prowler pre-configured and tied to roles to run"
# Template Parameters
# ImageId : Default is AWS Linux 2 ami-0e1d30f2c40c4c701
# InstanceType : Default is t3.micro
# VpcId : VPC to launch in
# SubnetId : Subnet to connect
# KeyName : Keypair to use
# CidrIp : CIDR range for SSH x.x.x.x/x
Resources:
# Create Prowler Instance - Parameters for ImageId, InstanceType, SubnetId, SecurityGroupIds, and KeyName
  ProwlerInstance:
    Type: 'AWS::EC2::Instance'
    Properties:
      ImageId: !Ref ImageId
      InstanceType: !Ref InstanceType
      SubnetId: !Ref SubnetId
      SecurityGroupIds:
        - !Ref InstanceSecurityGroup
      KeyName: !Ref KeyName
      IamInstanceProfile: !Ref ProwlerInstanceProfile
      Tags:
        -
          Key: Name
          Value: Prowler
      BlockDeviceMappings:
        - DeviceName: /dev/xvda
          Ebs:
            VolumeSize: 8
            Encrypted: true
# Run bash to install and configure Prowler
      UserData:
        Fn::Base64:
          !Sub |
            #!/bin/bash -xe
            sudo yum update -y
            sudo yum remove -y awscli
            cd /home/ec2-user
            curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "/home/ec2-user/awscliv2.zip"
            unzip /home/ec2-user/awscliv2.zip
            sudo /home/ec2-user/aws/install
            sudo yum install -y python3 jq git
            sudo pip3 install detect-secrets==1.0.3
            git clone https://github.com/prowler-cloud/prowler /home/ec2-user/prowler
            chown -R ec2-user:ec2-user /home/ec2-user/prowler
  ProwlerInstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      InstanceProfileName: prowler-ec2-instance-profile
      Path: /
      Roles:
       - !Ref ProwlerEc2InstanceRole
# Create Security Group
  InstanceSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
        GroupDescription: Allow ssh from specific host
        GroupName: ProwlerSecurityGroup
        VpcId: !Ref VpcId
        SecurityGroupIngress:
          - IpProtocol: 'tcp'
            FromPort: '22'
            ToPort: '22'
            CidrIp: !Ref CidrIp
# Create EC2 Instance Role to run security checks and attach to instance
  ProwlerEc2InstanceRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: prowler-ec2-instance-role
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          -
            Effect: Allow
            Principal:
              Service:
                - ec2.amazonaws.com
            Action:
              - sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/SecurityAudit
        - arn:aws:iam::aws:policy/job-function/ViewOnlyAccess
      Path: /
# Parameters for cloudformation template with some defaults
Parameters:
  ImageId:
    Type: String
    Description: AMI - Linux 2
    Default: 'ami-0e1d30f2c40c4c701'
  InstanceType:
    Type: String
    Description: Instance type to be used - t3.micro default
    Default: t3.micro
  VpcId:
    Type: AWS::EC2::VPC::Id
    Description: VPC to be used
  SubnetId:
    Type: AWS::EC2::Subnet::Id
    Description: Subnet to be used
  KeyName:
    Type: AWS::EC2::KeyPair::KeyName
    Description: Keyname
  CidrIp:
    Type: String
    Description: CidrIp to be used to connect from x.x.x.x/x
Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      -
        Label:
          default: "Network Configuration"
        Parameters:
          - ImageId
          - InstanceType
          - VpcId
          - SubnetId
          - KeyName
          - CidrIp
Conditions: {}

Running Prowler

After launching the CloudFormation template, simply sign into the EC2 instance and change into the /home/ec2-user/prowler directory.

To start, I recommend running Prowler with the HTML output file option. This provides a dynamic HTML file that you can review all the findings.

./prowler -M html

You can run direct output to multiple formats at once such as csv and json

./prowler -M csv,json,html

Decommissioning the resources

After you run Prowler, copy the output files to another system or S3 for review and record keeping. Go back into CloudFormation and delete the stack to remove all the resources that were generated.

Next steps

Prowler is also supported by AWS Security Hub, so you can send your findings directly to Security Hub. There's also a workshop available to build security dashboards in Quicksight from Prowler data. Details for this integration can be found at Building Prowler into a QuickSight powered AWS Security Dashboard.

Compliance Checking in AWS

James Monek — Sun, 13 Mar 2022 23:24:48 +0000

Storing protected data such as HIPAA in the cloud requires extra attention to ensure the data is not compromised. Using the NIST 800-171 framework provides guidance for controls that can help facilitate the protection of HIPAA and other protected data sets. AWS Audit Manager is a good starting point for auditing your environment against NIST 800-171 but I wanted to learn more about NIST and building controls in AWS to protect data. To do this, I wanted to use AWS Config service to monitor for compliance for controls and to use AWS Simple Notification Service to send alerts to via email and Slack. To accomplish this, AWS provides operational best practices for using AWS Config service to continuous monitor many NIST 800-171 controls. From this, I pulled controls that are applicable for services being utilized to host and analyze data in AWS.

AWS Config Service

AWS Config provides services to assess and audit configurations for AWS resources. We all have read articles about S3 buckets being misconfigured and exposing data to the public. AWS Config can monitor for buckets that are configured for public access. If bucket should be misconfigured or changed to allow public access, AWS Config will report this compliance violation and you can set up remediation to automatically change the settings back.

Lets build out Cloudfomation template that checks for compliance for blocking public access and for EBS volumes that are not encrypted.

Check for the S3 bucket public access

# S3 Compliance checks

  ConfigRuleBucketReadProhibited:
    Type: "AWS::Config::ConfigRule"
    Properties:
      ConfigRuleName: "s3-bucket-public-read-prohibited"
      Scope:
        ComplianceResourceTypes:
          - "AWS::S3::Bucket"
      Description: "A Config rule that checks that your Amazon S3 buckets do not allow public read access. If an Amazon S3 bucket policy or bucket ACL allows public read access, the bucket is noncompliant."
      Source:
        Owner: "AWS"
        SourceIdentifier: "S3_BUCKET_PUBLIC_READ_PROHIBITED"
  ConfigRuleBucketWriteProhibited:
    Type: "AWS::Config::ConfigRule"
    Properties:
      ConfigRuleName: "s3-bucket-public-write-prohibited"
      Scope:
        ComplianceResourceTypes:
          - "AWS::S3::Bucket"
      Description: "A Config rule that checks that your Amazon S3 buckets do not allow public write access. If an Amazon S3 bucket policy or bucket ACL allows public write access, the bucket is noncompliant."
      Source:
        Owner: "AWS"
        SourceIdentifier: "S3_BUCKET_PUBLIC_WRITE_PROHIBITED"
  ConfigRuleS3PublicAccess:
    Type: "AWS::Config::ConfigRule"
    Properties:
      ConfigRuleName: "s3-account-level-public-access-blocks"
      Scope:
        ComplianceResourceTypes:
          - "AWS::S3::AccountPublicAccessBlock"
      Description: "A Config rule that checks whether the required public access block settings are configured from account level. The rule is only NON_COMPLIANT when the fields set below do not match the corresponding fields in the configuration item."
      InputParameters:
        IgnorePublicAcls: "True"
        BlockPublicPolicy: "True"
        BlockPublicAcls: "True"
        RestrictPublicBuckets: "True"
      Source:
        Owner: "AWS"
        SourceIdentifier: "S3_ACCOUNT_LEVEL_PUBLIC_ACCESS_BLOCKS"
  ConfigRuleBucketPublicAccessProhibited:
    Type: "AWS::Config::ConfigRule"
    Properties:
      ConfigRuleName: "s3-bucket-level-public-access-prohibited"
      Scope:
        ComplianceResourceTypes:
          - "AWS::S3::Bucket"
      Description: "Checks if Amazon Simple Storage Service (Amazon S3) buckets are publicly accessible. This rule is NON_COMPLIANT if an Amazon S3 bucket is not listed in the excludedPublicBuckets parameter and bucket level settings are public."
      Source:
        Owner: "AWS"
        SourceIdentifier: "S3_BUCKET_LEVEL_PUBLIC_ACCESS_PROHIBITED"

Check that EBS Volumes are encrypted

#EC2 and EBS Compliance Checks
  ConfigRuleEncryptedVolumes:
    Type: "AWS::Config::ConfigRule"
    Properties:
      ConfigRuleName: "encrypted-volumes"
      Scope:
        ComplianceResourceTypes:
          - "AWS::EC2::Volume"
      Description: "A Config rule that checks whether the EBS volumes that are in an attached state are encrypted."
      Source:
        Owner: "AWS"
        SourceIdentifier: "ENCRYPTED_VOLUMES"

Notification for compliance changes

Now that we have some compliance checking rules, we’ll want to notify us when resources are out of compliance. To do this, we will use a combination of AWS EventBridge and Simple Notification Service.

EventBridge is a serverless event bus that we will use to detect changes to AWS Config. To get started, we will create an event rule to detect when a compliance changes. Portion of the cloud formation template is below and the particular section to note is the EventPattern.

  EventRule:
    Type: "AWS::Events::Rule"
    Properties:
      Name: "detect-config-changes"
      Description: "A CloudWatch Event Rule that detects changes to AWS Config and publishes change events to an SNS topic for notification."
      State: "ENABLED"
      Targets:
        - Arn:
            Ref: "SnsTopic"
          Id: "target-id1"
      EventPattern:
        detail-type:
          - "Config Rules Compliance Change"
        source:
          - "aws.config"

Next, we will want to build out the SNS Topic to send us an email to alert us of compliance changes. The SNS Topic section creates the subscription, where as, the TopicPolicy section provides permissions to publish the notification.

  SnsTopic:
    Type: "AWS::SNS::Topic"
    Properties:
      Subscription:
        - Endpoint: "aws-notifications-list@example.com"
          Protocol: "email"
      TopicName: "sns-topic"
  TopicPolicy:
    Type: AWS::SNS::TopicPolicy
    Properties:
      PolicyDocument:
        Id: HDWTopicPolicy
        Statement:
          - Sid: AllowServices
            Effect: Allow
            Principal:
              Service: events.amazonaws.com
            Action: 'sns:Publish'
            Resource:
              - !Ref SnsTopic

Putting this all together

The final CloudFormation templates are below. The first stack creates the AWS Config rules and the other creates the EventBridge and SNS notifications.

AWS Config CloudFormation Template

AWSTemplateFormatVersion: "2010-09-09"
Description: "AWS Config Checks"
Resources:
#EC2 and EBS Compliance Checks
  ConfigRuleEncryptedVolumes:
    Type: "AWS::Config::ConfigRule"
    Properties:
      ConfigRuleName: "encrypted-volumes"
      Scope:
        ComplianceResourceTypes:
          - "AWS::EC2::Volume"
      Description: "A Config rule that checks whether the EBS volumes that are in an attached state are encrypted."
      Source:
        Owner: "AWS"
        SourceIdentifier: "ENCRYPTED_VOLUMES"
# S3 Compliance checks
  ConfigRuleBucketReadProhibited:
    Type: "AWS::Config::ConfigRule"
    Properties:
      ConfigRuleName: "s3-bucket-public-read-prohibited"
      Scope:
        ComplianceResourceTypes:
          - "AWS::S3::Bucket"
      Description: "A Config rule that checks that your Amazon S3 buckets do not allow public read access. If an Amazon S3 bucket policy or bucket ACL allows public read access, the bucket is noncompliant."
      Source:
        Owner: "AWS"
        SourceIdentifier: "S3_BUCKET_PUBLIC_READ_PROHIBITED"
  ConfigRuleBucketWriteProhibited:
    Type: "AWS::Config::ConfigRule"
    Properties:
      ConfigRuleName: "s3-bucket-public-write-prohibited"
      Scope:
        ComplianceResourceTypes:
          - "AWS::S3::Bucket"
      Description: "A Config rule that checks that your Amazon S3 buckets do not allow public write access. If an Amazon S3 bucket policy or bucket ACL allows public write access, the bucket is noncompliant."
      Source:
        Owner: "AWS"
        SourceIdentifier: "S3_BUCKET_PUBLIC_WRITE_PROHIBITED"
  ConfigRuleS3PublicAccess:
    Type: "AWS::Config::ConfigRule"
    Properties:
      ConfigRuleName: "s3-account-level-public-access-blocks"
      Scope:
        ComplianceResourceTypes:
          - "AWS::S3::AccountPublicAccessBlock"
      Description: "A Config rule that checks whether the required public access block settings are configured from account level. The rule is only NON_COMPLIANT when the fields set below do not match the corresponding fields in the configuration item."
      InputParameters:
        IgnorePublicAcls: "True"
        BlockPublicPolicy: "True"
        BlockPublicAcls: "True"
        RestrictPublicBuckets: "True"
      Source:
        Owner: "AWS"
        SourceIdentifier: "S3_ACCOUNT_LEVEL_PUBLIC_ACCESS_BLOCKS"
  ConfigRuleBucketPublicAccessProhibited:
    Type: "AWS::Config::ConfigRule"
    Properties:
      ConfigRuleName: "s3-bucket-level-public-access-prohibited"
      Scope:
        ComplianceResourceTypes:
          - "AWS::S3::Bucket"
      Description: "Checks if Amazon Simple Storage Service (Amazon S3) buckets are publicly accessible. This rule is NON_COMPLIANT if an Amazon S3 bucket is not listed in the excludedPublicBuckets parameter and bucket level settings are public."
      Source:
        Owner: "AWS"
        SourceIdentifier: "S3_BUCKET_LEVEL_PUBLIC_ACCESS_PROHIBITED"
Parameters: {}
Metadata: {}
Conditions: {}

EventBridge and SNS Notifications

AWSTemplateFormatVersion: "2010-09-09"
Description: "Create SNS topic to send alerts and notifications to aws-notifications-list@example.com"
# Create SNSTopic to send email to aws-notifications-list@example.com
Resources:
  SnsTopic:
    Type: "AWS::SNS::Topic"
    Properties:
      Subscription:
        - Endpoint: "aws-notifications-list@example.com"
          Protocol: "email"
      TopicName: "sns-topic"
  TopicPolicy:
    Type: AWS::SNS::TopicPolicy
    Properties:
      PolicyDocument:
        Id: HDWTopicPolicy
        Statement:
          - Sid: AllowServices
            Effect: Allow
            Principal:
              Service: events.amazonaws.com
            Action: 'sns:Publish'
            Resource:
              - !Ref SnsTopic
      Topics:
        - !Ref SnsTopic
# Create CloudWatch Event Rule to notify of config changes
  EventRule:
    Type: "AWS::Events::Rule"
    Properties:
      Name: "detect-config-changes"
      Description: "A CloudWatch Event Rule that detects changes to AWS Config and publishes change events to an SNS topic for notification."
      State: "ENABLED"
      Targets:
        - Arn:
            Ref: "SnsTopic"
          Id: "target-id1"
      EventPattern:
        detail-type:
          - "Config Rules Compliance Change"
        source:
          - "aws.config"
Parameters: {}
Metadata: {}
Conditions: {}

What’s next?

There are two additional items I’ll be looking to expand on. One is to add functionality for SNS to post the event in a Slack Channel when a resource comes out of compliance. Lastly, I’ll look to clean up JSON output to be more readable for the notifications.

Don’t bring your network to the cloud

James Monek — Sun, 02 Jan 2022 23:11:52 +0000

I recently had the opportunity to facilitate a workshop at the annual Cloud Forum hosted by Cornell. The purpose of the workshop was to discuss whether to bring your networking to the cloud, meaning develop a hub and spoke architecture in AWS using transit gateways and then connecting to your campus network using VPN or Direct Connect. Or is it best to use cloud native networking and connect through the internet.

Why would you bring your networking to the cloud?

Let’s explore some of the reasons why you may consider bringing your network to the cloud. Some being very legitimate reasons, while others are perceived benefits.

Funneling all network traffic through your campus network and then through VPN is considered more secure. Initially, this sounds logical, forcing all your network traffic through your networking and firewalls allows you utilize existing security tools, risk assessments, monitoring, filtering, and SIEM. With this architecture, you need to ensure you lock down all cloud services for being exposed to the internet and force all services through your hub and spoke model. Your attack vector may be reduced but if your on-premise infrastructure is compromised, the attacker will still be able to access your cloud resources. Seems like a win-win scenario and it may be.

One downside of this architecture is the limitations of bandwidth of the VPN tunnel and the cost of direct connections. You also need to account for redundancy in this model, which will increase costs. Many higher education institutions have multiple high speed ISP connections for performance and redundancy. By not using the commodity ISP, you lose redundancy and high bandwidth.

Having your data travel across wild west of the internet adds a layer of complexity when debugging performance issues with applications and services. Whereas, the Direct Connect model comes with expectation of performance.

DDoS attacks are real and can bring the institution’s access to the internet to a crawl or be decommissioned altogether. This mean that during an attack, access to cloud services through the ISPs will be inaccessible. Campus resources will continue to function and more than likely the Direct Connect will be unaffected by the attack, so cloud direct services will be accessible. Of course, the simple answer is to connect to those cloud resources from an alternative location such as remote home office, but this can be disruptive for higher education institutions.

Argument for not bringing your networking to the cloud

As noted, most higher education institutions have multiple high speed ISPs. By not forcing traffic through your network, you can take advantage of those high bandwidth connections and inherent redundancies. You lose the additional data transfer costs and connection fees associated with VPN and Direct Connect.

By relying solely on cloud services, you gain the benefit of building redundancy and resilience for your applications and services by using elastic load balancers, Route 53, auto-scaling, and design architectures across availability zones and regions. Many cloud networking services have solid integration with other services you will be utilizing.

Bringing traditional networking to the cloud can add complexity and, in some regards, less secure. It may mean adding 3rd party appliances or services to match on-premise architecture. Whereas, using cloud native networking, you utilize cloud services to the fullest, improve security within your architecture, and is often more cost efficient. Just as in lift and shift model of moving servers into the cloud, you lose the value of cloud services. Don’t do the same with your networking.

No one solutions for every use case

Something to consider is Software Defined Networking (SDN) and how it can be used to bridge the gaps between on-premise networks and the cloud by providing a central view of all networks, allowing for agility, virtualization, and portability.

The important takeaways from this are that you don’t need to limit your networking architect to one model or path, you can bring both your network to the cloud and use cloud native networking. You’ll need to consider security, costs, performance, supportability, and user experience when selecting the path to the cloud. Selecting the right networking architecture can be challenging but keep in mind some of these points and use cases.

Special thanks and credits to John Bailey. Asst. Director, Cloud Systems at Washington University in St. Louis. John initially held a lightning round session and lead the workshop design for this topic.