Forem: Monica Escobar

Bridging the Gap: Solving Common Business Connectivity Challenges with Hybrid DNS Architecture

Monica Escobar — Sun, 07 Jul 2024 19:32:42 +0000

Before we dive in, I’d like to extend my special thanks to Adrian Cantrill for his exceptional training materials. More details at the end of the article.

A4L's Mission

Animals 4 Life (A4L) is a fictional but passionate animal charity dedicated to protecting animals around the world. Founded in Australia, A4L has expanded its reach globally, leveraging technology to enhance its conservation efforts. With operations spanning multiple regions, A4L’s commitment to wildlife protection drives its need for a robust and interconnected IT infrastructure. This hybrid DNS architecture plays a crucial role in supporting their mission, ensuring that both their on-premises and cloud environments can work together seamlessly to safeguard wildlife everywhere.
In the dynamic metropolis of A4L, two distinct domains existed: the boundless cloud of AWS and the steadfast on-premises data centers. These domains, though powerful, struggled with communication. AWS-hosted applications needed access to on-premises data and vice versa, and they had to keep some data on premises for regulatory compliance. The inability to resolve domain names across these environments created silos (and if you follow Scrum, this ain’t good), hindering the seamless integration vital for A4L's operations.

Have a peek at the initial architecture:

Key Challenges presented by this architecture:

Isolation: AWS resources couldn’t locate on-premises resources.
Complexity: Manual DNS configurations were error-prone and inefficient.
Performance: Inconsistent connectivity led to delays and reliability issues.

Here we can see how when trying to connect from the AWS side with the on premises side, no connection was established:

And the same when trying the other way around:

The Bridge which came to the rescue

Enter the hybrid DNS architecture, a powerful solution designed to unite these disparate worlds. Central to this architecture were AWS Route 53 Resolver and Direct Connect (I used VPC peering as Direct Connect is hard to set up and not worth for a fictional company), forming the bridge for seamless communication.

The Architecture Unfolds:

Establish Direct Connect:

Direct Connect provides a dedicated, high-speed network link between AWS and the on-premises data center, ensuring reliable and secure dedicated communication. Although expensive, and with high overhead requirements, seemed to be the perfect solution for joining this two worlds, as it provides a constant connection between our two worlds providing a consistent connection, which was one of our key issues. Therefore, going back to our initial key challenges, two of them were removed by the Direct Connect connection, leaving us to focus only on the DNS configurations:
  
- Isolation: AWS resources couldn’t locate on-premises resources. - Fixed
- Complexity: Manual DNS configurations were error-prone and inefficient.
- Performance: Inconsistent connectivity led to delays and reliability issues. - Fixed

However, after doing this, we can still see there is no DNS resolution:

Configure AWS Side:

Route 53 Private Hosted Zone: Managed internal DNS records within AWS.
Route 53 Resolver Endpoints:
- Inbound Endpoints: Inbound endpoints are crucial because they allow on-premises DNS servers to send DNS queries to AWS. This means that resources within the on-premises environment can resolve domain names for AWS-hosted services. For example, our on- premises application will need to access an AWS-hosted database or web service. Without inbound endpoints, these queries would fail, isolating the on-premises environment from the cloud.

Outbound Endpoints: they serve the opposite purpose. They enable AWS resources to query the on-premises DNS servers. This is essential for cloud-hosted applications that need to interact with on-premises services. For instance, our web application running on an EC2 instance will require data from an on-premises database. Outbound endpoints ensure these DNS queries can be resolved, allowing seamless communication between AWS and on-premises resources.

Integrate On-Premises Side:

As a result of the previous step, Our Application Servers can now directly query AWS services, bridging the gap. In this architecture, both inbound and outbound endpoints work in tandem to create a bidirectional flow of DNS queries. This unified approach ensures that applications and services, regardless of their location, can resolve each other’s domain names, facilitating seamless integration and operation. This hybrid DNS setup is a cornerstone in achieving a truly interconnected and efficient IT ecosystem. Therefore, going back to our initial key challenges, the last remaining one was removed by the endpoint resolvers.

Isolation: AWS resources couldn’t locate on-premises resources. - Fixed
Complexity: Manual DNS configurations were error-prone and inefficient. - Fixed
Performance: Inconsistent connectivity led to delays and reliability issues. -Fixed

Leaving us with this final architecture:

Unified Operations, two Kingdoms working as one:

With the hybrid DNS architecture in place, A4L saw transformative changes, some of these were:

Seamless Integration: AWS and on-premises resources communicated effortlessly.
Centralised Management: Streamlined DNS management reduced errors and administrative burdens.
Enhanced Performance: Direct Connect ensured low-latency, high-reliability connections.
Scalability and Flexibility: Supported dynamic and scalable DNS query handling.
Security and Compliance: Ensured sensitive data remained protected through private connections.

The Happy Ending: Who Benefits?

Wildlife Protectors:
- A4L’s web servers in AWS could now seamlessly access on-premises databases, ensuring real-time updates on wildlife conservation efforts, improving chances on saving all animals, yay!
Disaster Recovery Champions:
- Ensured consistent DNS resolution across environments, enabling robust failover strategies.
Data Residency Stewards:
- Maintained compliance by keeping sensitive data on-premises while leveraging cloud resources.
Innovators with Complex Networks:
- Facilitated seamless communication across microservices and hybrid deployments.

Conclusion

The story of A4L's hybrid DNS architecture is one of bridging gaps and fostering unity between cloud and on-premises environments. With AWS Route 53 Resolver and Direct Connect as pivotal characters, the tale of seamless integration and operational excellence unfolds, promising a future where cloud and local resources work harmoniously together. This architecture not only solves the problem of fragmented networks but also paves the way for a new era of interconnected possibilities.

Special thanks to Adrian Cantrill for his exceptional training materials. Adrian is a top-notch learning provider who excels in teaching not just theory but also hands-on application. His courses are packed with practical projects that allow learners to apply what they've learned and navigate real-world business scenarios.

Deploy an Amazon Lex Chatbot to your own website.

Monica Escobar — Fri, 14 Jun 2024 15:33:57 +0000

Why a chatbot?

For professionals, integrating a chatbot into your website is more than just a cool tech feature; it’s a way to showcase your commitment to innovation and user experience. It reflects a forward-thinking approach and shows that you value your visitors’ time and needs. By offering instant, personalised interactions, a chatbot makes your website more engaging and user-friendly.

Incorporating a chatbot into your website is a smart move that demonstrates a proactive approach to communication and technology. It’s about making connections easier, information more accessible, and experiences more enjoyable for your visitors. In essence, it’s a reflection of who you are as a professional – someone who values innovation, accessibility, and excellence in every interaction.

These are just some of the reasons I chose to build and deploy my own chatbot, and I ended up liking it so much that I wanted to share the steps with everyone else in case you found it useful or beneficial for any of your projects.

Stack/resources used:

-Amazon Lex
-CloudFront
-My own website

Below are the steps I followed to create the chatbot using AWS Lex:

When creating a chatbot in AWS Lex, you have several options:
- Descriptive Bot Builder: Automatically generates intents, utterances, and slots based on your use case but you need to use BedRock for this, make sure you check the fees before you do this. Also, you will need to apply to have access granted if you have never used it before.
- Create a Blank Bot: Start from scratch and define your own intents, utterances, and slots. This is the one I chose.
- Start with a Transcript: Upload a JSON file with the conversation flow. Bear in mind, that if you decide to upload the json file you will need to provide 1000 lines as a minimum.

* Using the Visual Editor: *
- Create a blank bot and proceed to the Visual Editor.
- Define intents, slots, and conversation flows using a visual interface.
- Add intents, slots, prompts, and responses to script the conversation flow.

 Testing and Building the Chatbot:

- After defining the conversation flow, save the bot.
- Build the bot and test it within the AWS Lex console to ensure it functions correctly.

 Integrating the Chatbot:

- Once the chatbot is built successfully, create an alias for the bot.
- Integrate the chatbot with your website by deploying a stack (you can get it here: https://aws.amazon.com/blogs/machine-learning/deploy-a-web-ui-for-your-chatbot/)that includes CloudFront, web UI artifacts, and authentication using Amazon Cognito (if required, I personally did not include authentication).

 Deployment and Configuration:
- Launch the stack with the necessary parameters like Bot ID, Alias ID, and other configurations.
- Copy the snippet URL provided after the stack creation to integrate the chatbot into your website.

_ Finalising Integration: _

- Update your website's HTML code with the provided snippet URL to embed the chatbot.
- Upload the updated HTML file to your hosting platform (e.g., S3 bucket).
- Invalidate the CloudFront cache to reflect the changes on your website.

 Testing the Integrated Chatbot:
- Access your website and interact with the chatbot using text or voice commands.
- Validate that the chatbot functions correctly and responds to user inputs as expected.

*Future enhancements: *

Looking ahead, my journey with AI and automation is far from over. I have exciting plans to further enhance the capabilities of the chatbot and, in turn, the overall user experience of my portfolio.
One of the key areas I’m focusing on is harnessing user behaviour insights. The chatbot has the potential to track common questions and interactions, revealing what visitors find most interesting or important, this constant feedback can help me enhance the user experience.

If you got this far, thank you so much and happy building!

Using AI to improve security and learning in your AWS environment.

Monica Escobar — Mon, 10 Jun 2024 13:57:00 +0000

Why We're Building a Chatbot: Empowering Our Platform Team

Our Platform engineers are the backbone of our secure development process. However, they often face hurdles that slow them down and hinder their ability to deliver top-notch work. On top of this, there are different levels of expertise within the team and consulting all the available documentation can be a daunting and time consuming task. This chatbot could potentially reduce the coaching workload from the senior members to the junior members of the team.

To address these challenges and empower the teams, I chose to build a chatbot specifically designed to assist them.

Our team’s stack is composed of Airflow and Spark mainly, as well as the infrastructure hosted in AWS.

Here's a closer look at the pain points we're aiming to solve:

Streamlining Security Testing: Finding potential security vulnerabilities in code can be a time-consuming task. The chatbot will help engineers identify issues quickly and efficiently, it can trigger automated security scans through Spark or Airflow jobs, analyze scan results from S3 buckets, and present findings to engineers in a user-friendly way. This has become even more prominent and needed now with GDPR regulations. It can become a good tool for detecting PII data especifically tailored to our environment, if for whatever reason, Amazon Macie is not on the cards.
Enhancing Security Visibility: the team needs a clear picture of potential software vulnerabilities across all applications, like out of support versions. The chatbot can leverage data from various sources, like security reports stored in S3 buckets, and use this information to identify trends and potential vulnerabilities across applications managed by your EC2 instances within VPCs.
Heavy teaching workload: having an easy accesible but secure chatbot to consult all the platform’s documentation, fully tailored to our environment, will positively impact everyone in the team. Junior members will become more independent and confident in what they do and senior members will have more time to spend working on their own tickets.
Simplifying Security Practices: Developing strong threat models and security policies is crucial, but it can be complex. The chatbot will offer guidance and best practices, making it easier for engineers to implement these essential safeguards.
Boosting Infrastructure Security: Infrastructure as Code (IaC) plays a vital role in our development process.The chatbot can leverage tools like CloudFormation or Terraform to identify potential security risks within IaC templates before deployment to EC2 instances.
Enforcing Security Pipelines: Integrating security checks seamlessly into our development pipelines is essential. The chatbot will help enforce these checks, guaranteeing that security is never an afterthought by triggering security scans within Airflow pipelines, ensuring vulnerabilities are identified and addressed before code is deployed to production instances behind your ELBs.
Ensuring Quality Control: We are committed to delivering high-quality solutions. The chatbot will provide us with valuable insights and data, enabling us to maintain a high level of control over the development process.

By building this chatbot, we're investing in building secure and reliable software, as well as promoting a continuous learning culture and self development.

Here's a detailed breakdown of the steps I followed to build our chatbot:

Step 0:

Use the following CloudFormation template to deploy all the resources.

Thetechyteacher / ai-chatbot

ai-chatbot

View on GitHub

Step 1: Adding Documents to Amazon Simple Storage Service (S3)

What it is: Amazon S3 is a secure, scalable object storage service that acts as the foundation for our chatbot's knowledge base.
The Why: You'll store essential documents like security best practices, threat model templates, and IaC security guidelines in S3. This documents will be specific to your own production environment, so the answers can be fully tailored to our requirements. I personally chose to feed it with Spark and Airflow documentation, as well as general security docs.
How to do it:
1. Create an S3 bucket: This acts as a virtual folder where you'll store your documents.
2. Upload relevant documents: Upload security policies, best practice guides, and any other resources you might need.
3. Configure access permissions: Ensure the chatbot has the necessary permissions to access and retrieve information from the S3 bucket.

Step 2: Searching with Amazon Kendra

What it is: Amazon Kendra is an intelligent search service that allows users to easily find relevant information across various sources like S3 buckets.
The Why: Kendra will be crucial for your chatbot to efficiently search the vast amount of security information stored in S3.
How to do it:
1. Create a Kendra index: This index tells Kendra where to look for information, in this case, your S3 bucket containing security documents.

2. Add an s3 connector and link it to the s3 bucket which contains the data.

3. Sync now.

Step 3: Setting Up Access to Amazon Bedrock

What it is: Amazon Bedrock is a security configuration language that helps enforce security best practices throughout your infrastructure.
The Why: The chatbot can leverage Bedrock (AI) to guide chatbot users (our engineers) in developing secure IaC configurations and identify potential security risks in their code.
How to do it:
1. You will need access to Anthrotopic (Claude) and Titan Express. If you don’t have it granted yet, you’ll need to request it now.

Step 4: Using SageMaker Studio IDE to Build Your Chatbot

What it is: SageMaker Studio IDE is a cloud-based integrated development environment (IDE) designed for building and deploying machine learning models.
The Why: SageMaker Studio provides the tools and resources you need to develop your chatbot's core functionality, including natural language processing (NLP) and dialogue management capabilities.
How to do it:
1. Choose an appropriate Large Language Model (LLM): LLMs are AI models trained on massive amounts of text data, forming the foundation for your chatbot's ability to understand and respond to user queries. We will be using Claude and Titan Express.
2. Train the LLM on your security knowledge base: This involves feeding the LLM with the documents stored in S3, allowing it to learn the specific language and concepts related to DevSecOps security.
3. From the terminal, git clone the following repo: git clone https://github.com/aws-samples/generative-ai-to-build-a-devsecops-chatbot/
4. From the terminal, you will have to export Kendra’s Index ID, like this:  export KENDRA_INDEX_ID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
5. Install requirements: pip install -r requirements.txt pip install -U langchain-community
6. And use streamline to design your script’s interface by running the following command: streamlit run app.py titan
7. Get the URL from this page and remove the /lab path. Instead, add this: /proxy/8501/
8. Navigate to that URL and you will see your own chatbot live and running, how exciting!

NOTE: If you want to use Claude run this command instead: streamlit run app.py claudeInstant

Why would you want to test different LLM? Different LLMs have varying strengths and weaknesses. Testing with multiple options helps identify the LLM that best understands the terminology and delivers the most accurate and helpful responses.

Step 5: Remember to clean up your resources to avoid any potential charges.

Set up an automated incident management response using AWS

Monica Escobar — Sun, 09 Jun 2024 19:05:27 +0000

Incident management

Excited to share my latest AWS project focusing on automating an incident response following what I found to be a really engaging AWS made workshop.

The project involves setting up a core configuration of 3 EC2 instances with its corresponding security groups and a VPC in us-east-1 through a CloudFormation template.

To enhance security and ensure prompt incident response, a pipeline has been implemented. This pipeline starts with GuardDuty constantly monitoring the environment. When an anomaly is detected, an Amazon EventBridge Rule triggers a Lambda function.

The Lambda function plays a crucial role in securing the EC2 instances by restricting access to only ports 3389 and 22. Additionally, it takes an EC2 snapshot and stores it in an S3 bucket to prevent any data loss. The architecture is as follows:

To initiate the scenario and create the infrastructure for the automated incident response, I followed these steps to deploy the CloudFormation template:

Deploy the CloudFormation template

Review the CloudFormation Template
- Before deploying, you can review the template to understand its components and configurations.

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "Security Automations Workshop template.  Sets up VPC, EC2 instances, turns on GuardDuty and GuardDuty-Tester",
    "Metadata": {
        "AWS::CloudFormation::Interface": {
            "ParameterGroups": [
                {
                    "Label" : {"default": "Workshop Service Configuration"},
                    "Parameters": ["EnableGuardDuty"]
                },
                {
                    "Label" : {"default": "Workshop Parameters"},
                    "Parameters": ["LatestAMZNLinuxAMI", "LatestAMZNLinux2AMI", "LatestWindows2016AMI"]
                }
            ],
            "ParameterLabels": {
                "EnableGuardDuty": {"default" : "Automatically enable GuardDuty?"}            
            }
        }
    },
    "Parameters": {
        "LatestAMZNLinuxAMI": {
            "Description": "DO NOT CHANGE: The latest AMI ID for Amazon Linux",
            "Type": "AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>",
            "Default": "/aws/service/ami-amazon-linux-latest/amzn-ami-hvm-x86_64-gp2"
        },
        "LatestAMZNLinux2AMI": {
            "Description": "DO NOT CHANGE: The latest AMI ID for Amazon Linux2",
            "Type": "AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>",
            "Default": "/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2"
        },
        "LatestWindows2016AMI": {
            "Description": "DO NOT CHANGE: The latest AMI ID for Windows 2016",
            "Type": "AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>",
            "Default": "/aws/service/ami-windows-latest/EC2LaunchV2-Windows_Server-2016-English-Full-Base"
        },
        "EnableGuardDuty": {
            "Description": "Choose Yes if GuardDuty is not yet enabled in the account and region this template will be deployed to, otherwise choose No.",
            "Type": "String",
            "AllowedValues": ["Yes-Enable GuardDuty", "No-GuardDuty is already enabled"],
            "Default": "Yes-Enable GuardDuty"
        }
    },
    "Mappings": {
        "AWSRegionAMIMap": {
            "ap-south-1": {"HVM64": "ami-b46f48db"},
            "eu-west-3": {"HVM64": "ami-cae150b7"},
            "eu-west-2": {"HVM64": "ami-c12dcda6"},
            "eu-west-1": {"HVM64": "ami-9cbe9be5"},
            "ap-northeast-3": {"HVM64": "ami-68c1cf15"},
            "ap-northeast-2": {"HVM64": "ami-efaf0181"},
            "ap-northeast-1": {"HVM64": "ami-28ddc154"},
            "sa-east-1": {"HVM64": "ami-f09dcc9c"},
            "ca-central-1": {"HVM64": "ami-2f39bf4b"},
            "ap-southeast-1": {"HVM64": "ami-64260718"},
            "ap-southeast-2": {"HVM64": "ami-60a26a02"},
            "eu-central-1": {"HVM64": "ami-1b316af0"},
            "us-east-1": {"HVM64": "ami-467ca739"},
            "us-east-2": {"HVM64": "ami-976152f2"},
            "us-west-1": {"HVM64": "ami-46e1f226"},
            "us-west-2": {"HVM64": "ami-6b8cef13"}
            }
    },

    "Conditions": {
        "EnableGuardDuty": {"Fn::Equals": [{"Ref": "EnableGuardDuty"}, "Yes-Enable GuardDuty"]}
    },

    "Resources": {
        "SSMInstanceRole": {
            "Type": "AWS::IAM::Role",
            "Properties": {
                "AssumeRolePolicyDocument": {
                    "Version": "2012-10-17",
                    "Statement": [
                        {
                            "Effect": "Allow",
                            "Principal": {
                                "Service": [
                                    "ssm.amazonaws.com",
                                    "ec2.amazonaws.com"
                                ]
                            },
                            "Action": "sts:AssumeRole"
                        }
                    ]
                },
                "Policies": [
                    {
                        "PolicyName": "S3andSSMAccess",
                        "PolicyDocument": {
                            "Statement": [
                                {
                                    "Effect": "Allow",
                                    "Action": [
                                        "ssm:DescribeAssociation",
                                        "ssm:GetDeployablePatchSnapshotForInstance",
                                        "ssm:GetDocument",
                                        "ssm:DescribeDocument",
                                        "ssm:GetManifest",
                                        "ssm:GetParameters",
                                        "ssm:GetParameter",
                                        "ssm:ListAssociations",
                                        "ssm:ListInstanceAssociations",
                                        "ssm:PutInventory",
                                        "ssm:PutComplianceItems",
                                        "ssm:PutConfigurePackageResult",
                                        "ssm:UpdateAssociationStatus",
                                        "ssm:UpdateInstanceAssociationStatus",
                                        "ssm:UpdateInstanceInformation"
                                    ],
                                    "Resource": "*"
                                },
                                {
                                    "Effect": "Allow",
                                    "Action": [
                                        "ssmmessages:CreateControlChannel",
                                        "ssmmessages:CreateDataChannel",
                                        "ssmmessages:OpenControlChannel",
                                        "ssmmessages:OpenDataChannel"
                                    ],
                                    "Resource": "*"
                                },
                                {
                                    "Effect": "Allow",
                                    "Action": [
                                        "ec2messages:AcknowledgeMessage",
                                        "ec2messages:DeleteMessage",
                                        "ec2messages:FailMessage",
                                        "ec2messages:GetEndpoint",
                                        "ec2messages:GetMessages",
                                        "ec2messages:SendReply"
                                    ],
                                    "Resource": "*"
                                },
                                {
                                    "Effect": "Allow",
                                    "Action": [
                                        "cloudwatch:PutMetricData"
                                    ],
                                    "Resource": "*"
                                },
                                {
                                    "Effect": "Allow",
                                    "Action": [
                                        "ec2:DescribeInstanceStatus"
                                    ],
                                    "Resource": "*"
                                },
                                {
                                    "Effect": "Allow",
                                    "Action": [
                                        "ds:CreateComputer",
                                        "ds:DescribeDirectories"
                                    ],
                                    "Resource": "*"
                                },
                                {
                                    "Effect": "Allow",
                                    "Action": [
                                        "logs:CreateLogGroup",
                                        "logs:CreateLogStream",
                                        "logs:DescribeLogGroups",
                                        "logs:DescribeLogStreams",
                                        "logs:PutLogEvents"
                                    ],
                                    "Resource": "*"
                                },
                                {
                                    "Effect": "Allow",
                                    "Action": [
                                        "s3:GetBucketLocation",
                                        "s3:PutObject",
                                        "s3:GetObject",
                                        "s3:GetEncryptionConfiguration",
                                        "s3:AbortMultipartUpload",
                                        "s3:ListMultipartUploadParts",
                                        "s3:ListBucket",
                                        "s3:ListBucketMultipartUploads"
                                    ],
                                    "Resource": "*"
                                },
                                {
                                    "Sid": "S3ListBuckets",
                                    "Effect": "Allow",
                                    "Action": [
                                        "s3:ListAllMyBuckets"
                                    ],
                                    "Resource": "arn:aws:s3:::*"
                                },
                                {
                                    "Sid": "S3GetObjects",
                                    "Effect": "Allow",
                                    "Action": [
                                        "s3:ListBucket",
                                        "s3:GetBucketLocation",
                                        "s3:GetObject"
                                    ],
                                    "Resource": {
                                        "Fn::Join": [
                                            "",
                                            [
                                                "arn:aws:s3:::",
                                                "agentbucket-",
                                                {
                                                    "Ref": "AWS::AccountId"
                                                },
                                                "/*"
                                            ]
                                        ]
                                    }
                                }
                            ]
                        }
                    }
                ],
                "Path": "/"
            }
        },
        "VPC": {
            "Type": "AWS::EC2::VPC",
            "Properties": {
                "CidrBlock": "10.0.0.0/16",
                "EnableDnsSupport": "true",
                "EnableDnsHostnames": "true",
                "Tags": [
                    {
                        "Key": "Application",
                        "Value": {
                            "Ref": "AWS::StackName"
                        }
                    },
                    {
                        "Key": "Name",
                        "Value": {
                            "Fn::Join": [
                                "-",
                                [
                                    "VPC",
                                    {
                                        "Ref": "AWS::StackName"
                                    }
                                ]
                            ]
                        }
                    }
                ]
            }
        },
        "PublicSubnet": {
            "Type": "AWS::EC2::Subnet",
            "Properties": {
                "VpcId": {
                    "Ref": "VPC"
                },
                "CidrBlock": "10.0.1.0/24",
                "MapPublicIpOnLaunch": "true",
                "AvailabilityZone": {
                    "Fn::Select": [
                        "0",
                        {
                            "Fn::GetAZs": {
                                "Ref": "AWS::Region"
                            }
                        }
                    ]
                },
                "Tags": [
                    {
                        "Key": "Name",
                        "Value": {
                            "Fn::Join": [
                                "-",
                                [
                                    "Pub1",
                                    {
                                        "Ref": "AWS::StackName"
                                    }
                                ]
                            ]
                        }
                    }
                ]
            }
        },
        "IGW": {
            "Type": "AWS::EC2::InternetGateway",
            "Properties": {
                "Tags": [
                    {
                        "Key": "Application",
                        "Value": {
                            "Ref": "AWS::StackName"
                        }
                    }
                ]
            }
        },
        "AttachGateway": {
            "Type": "AWS::EC2::VPCGatewayAttachment",
            "Properties": {
                "VpcId": {
                    "Ref": "VPC"
                },
                "InternetGatewayId": {
                    "Ref": "IGW"
                }
            }
        },
        "PublicRouteTable": {
            "Type": "AWS::EC2::RouteTable",
            "Properties": {
                "VpcId": {
                    "Ref": "VPC"
                },
                "Tags": [
                    {
                        "Key": "Application",
                        "Value": {
                            "Ref": "AWS::StackName"
                        }
                    },
                    {
                        "Key": "Network",
                        "Value": "Public"
                    }
                ]
            }
        },
        "PublicRoute": {
            "Type": "AWS::EC2::Route",
            "DependsOn": [
                "AttachGateway"
            ],
            "Properties": {
                "RouteTableId": {
                    "Ref": "PublicRouteTable"
                },
                "DestinationCidrBlock": "0.0.0.0/0",
                "GatewayId": {
                    "Ref": "IGW"
                }
            }
        },
        "PublicSubnetRouteAssociation": {
            "Type": "AWS::EC2::SubnetRouteTableAssociation",
            "Properties": {
                "SubnetId": {
                    "Ref": "PublicSubnet"
                },
                "RouteTableId": {
                    "Ref": "PublicRouteTable"
                }
            }
        },
        "GDdetector": {
            "Type": "AWS::GuardDuty::Detector",
            "Condition": "EnableGuardDuty",
            "Properties": {
                "Enable": true,
                "FindingPublishingFrequency": "FIFTEEN_MINUTES"
            }
        },
        "GuardDutyTesterTemplate": {
            "Type": "AWS::CloudFormation::Stack",
            "Properties": {
                "TemplateURL":{
                    "Fn::Join": [
                            "",
                            [
                                "https://sa-security-specialist-workshops-",
                                {
                                    "Ref": "AWS::Region"
                                },
                                ".s3.",
                                {
                                    "Ref": "AWS::Region"
                                },
                                ".amazonaws.com/security-hub-workshop/templates/guardduty-tester-template.json"
                            ]
                        ]
                    },
                "Parameters": {
                    "InstanceSubnetId": {
                        "Ref": "PublicSubnet"
                    },
                    "DeployVPC": {
                        "Ref": "VPC"
                    },
                    "DeployVPCCidr": {
                        "Fn::GetAtt": [
                            "VPC",
                            "CidrBlock"
                        ]
                    },
                    "LatestWindows2012R2AMI":  "/aws/service/ami-windows-latest/EC2LaunchV2-Windows_Server-2016-English-Full-Base"
                }
            }
        }
    }
}

Deploy the Template
- Select aregion, for instance, I will be using us-east-1 (N. Virginia).
Specify Stack Details
- Enter the following parameters:
  - Stack name: AutomatedIncidentResponseWorkshop
  - Enable GuardDuty: Yes
- After filling in the parameters, click Next.
Configure Stack Options
- Click Next again on the following page, leaving all options at their default values.
Acknowledge and Create Stack
- Scroll down to the bottom of the page, check the box acknowledging that the template will create IAM roles, and click Create stack.

Setting up a security group

Create Security Group
- Create a new security group named ForensicsSG.
- Remove all outbound rules and set the following inbound rules:
  - RDP: Protocol TCP, Port 3389, Source (your IP), Description: RDP for IR team
  - SSH: Protocol TCP, Port 22, Source (your IP), Description: SSH for IR team

Creating and Attaching Policies

Create a New IAM Policy
- Create a policy called ec2instance-containment-with-forensics-policy with the following JSON to deny termination of isolated instances:  {
- "Version": "2012-10-17",
- "Statement": [
- {
- "Sid": "VisualEditor0",
- "Effect": "Deny",
- "Action": [
- "ec2:TerminateInstances",
- "ec2:DeleteTags",
- "ec2:CreateTags"
- ],
- "Resource": "*",
- "Condition": {
- "StringEquals": {
- "aws:ResourceTag/status": "isolated"
- }
- }
- }
- ]
- }
-  
Create the Execution role for the Lambda function:
 Create a role called ec2instance-containment-with-forensics-role with Lambda as a trusted entity in Trust Relationships
Create a User Group
- Create a group named ec2-users.

Attach Policies to the Group
- Attach the following policies to the ec2-users group:
  - AmazonEC2FullAccess (AWS Managed Policy)
  - Deny-termination-of-isolated-instances (Custom policy created above).
Create a New User
- Create an IAM user named testuser
- Add this user to the ec2-users group.

Configuring the Lambda Function

Create IAM Policy for Lambda
- Create an IAM policy and attach it to the IAM role that the Lambda function will assume for automated responses.
Create the Lambda Function
- Develop and deploy the Lambda function that will handle automated incident responses. Change the timeout to 15 minutes and select ec2instance-containment-with-forensics-role as the execution role. Select Python as runtime.

Add the following environmental values:
Key: ForensicsSG
Value: sg-...(the ID of your Forensics SG)

Include the following code:  import boto3 import time from datetime import date from botocore.exceptions import ClientError import os

def lambda_handler(event, context):
# Copyright 2022 - Amazon Web Services

# Permission is hereby granted, free of charge, to any person obtaining a copy of this
# software and associated documentation files (the "Software"), to deal in the Software
# without restriction, including without limitation the rights to use, copy, modify,
# merge, publish, distribute, sublicense, and/or sell copies of the Software, and to
# permit persons to whom the Software is furnished to do so.

# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
# INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
# PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
# SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

# print('## ENVIRONMENT VARIABLES')
# print(os.environ)
# print('## EVENT')
# print(event)
response = 'Error remediating the security finding.'
try:
    # Gather Instance ID from CloudWatch event
    instanceID = event['detail']['resource']['instanceDetails']['instanceId']
    print('## INSTANCE ID: %s' % (instanceID))

    # Get instance details
    client = boto3.client('ec2')
    ec2 = boto3.resource('ec2')
    instance = ec2.Instance(instanceID)
    instance_description = client.describe_instances(InstanceIds=[instanceID])
    print('## INSTANCE DESCRIPTION: %s' % (instance_description))

    #-------------------------------------------------------------------
    # Protect instance from termination
    #-------------------------------------------------------------------
    ec2.Instance(instanceID).modify_attribute(
    DisableApiTermination={
        'Value': True
    })
    ec2.Instance(instanceID).modify_attribute(
    InstanceInitiatedShutdownBehavior={
        'Value': 'stop'
    })

    #-------------------------------------------------------------------
    # Create tags to avoid accidental deletion of forensics evidence
    #-------------------------------------------------------------------
    ec2.create_tags(Resources=[instanceID], Tags=[{'Key':'status', 'Value':'isolated'}])
    print('## INSTANCE TAGS: %s' % (instance.tags))

    #------------------------------------
    ## Isolate Instance
    #------------------------------------
    print('quarantining instance -- %s, %s' % (instance.id, instance.instance_type))

    # Change instance Security Group attribute to terminate connections and allow Forensics Team's access
    instance.modify_attribute(Groups=[os.environ['ForensicsSG']])
    print('Instance ready for root cause analysis -- %s, %s' % (instance.id,  instance.security_groups))

    #------------------------------------
    ## Create snapshots of EBS volumes 
    #------------------------------------
    description= 'Isolated Instance:' + instance.id + ' on account: ' + event['detail']['accountId'] + ' on ' + date.today().strftime("%Y-%m-%d  %H:%M:%S")
    SnapShotDetails = client.create_snapshots(
        Description=description,
        InstanceSpecification = {
            'InstanceId': instanceID,
            'ExcludeBootVolume': False
        }
    )
    print('Snapshot Created -- %s' % (SnapShotDetails))

    response = 'Instance ' + instance.id + ' auto-remediated'        

except ClientError as e:
    print(e)

return response

Test the Lambda Function
- Perform tests to ensure the Lambda function works as expected. You can use the following code (remember to change some of the variables): import boto3, json import time from datetime import date from botocore.exceptions import ClientError import os

def lambda_handler(event, context):
# Copyright 2022 - Amazon Web Services

# Permission is hereby granted, free of charge, to any person obtaining a copy of this
# software and associated documentation files (the "Software"), to deal in the Software
# without restriction, including without limitation the rights to use, copy, modify,
# merge, publish, distribute, sublicense, and/or sell copies of the Software, and to
# permit persons to whom the Software is furnished to do so.

# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
# INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
# PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
# SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

# print('## ENVIRONMENT VARIABLES')
# print(os.environ)
# print('## EVENT')
# print(event)
response = 'Error remediating the security finding.'
try:
    # Gather Instance ID from CloudWatch event
    instanceID = event['detail']['resource']['instanceDetails']['instanceId']
    print('## INSTANCE ID: %s' % (instanceID))

    # Get instance details
    client = boto3.client('ec2')
    ec2 = boto3.resource('ec2')
    instance = ec2.Instance(instanceID)
    instance_description = client.describe_instances(InstanceIds=[instanceID])
    print('## INSTANCE DESCRIPTION: %s' % (instance_description))

    #-------------------------------------------------------------------
    # Protect instance from termination
    #-------------------------------------------------------------------
    ec2.Instance(instanceID).modify_attribute(
    DisableApiTermination={
        'Value': True
    })
    ec2.Instance(instanceID).modify_attribute(
    InstanceInitiatedShutdownBehavior={
        'Value': 'stop'
    })

    #-------------------------------------------------------------------
    # Create tags to avoid accidental deletion of forensics evidence
    #-------------------------------------------------------------------
    ec2.create_tags(Resources=[instanceID], Tags=[{'Key':'status', 'Value':'isolated'}])
    print('## INSTANCE TAGS: %s' % (instance.tags))

    #------------------------------------
    ## Isolate Instance
    #------------------------------------
    print('quarantining instance -- %s, %s' % (instance.id, instance.instance_type))

    # Change instance Security Group attribute to terminate connections and allow Forensics Team's access
    instance.modify_attribute(Groups=[os.environ['ForensicsSG']])
    print('Instance ready for root cause analysis -- %s, %s' % (instance.id,  instance.security_groups))

    #------------------------------------
    ## Create snapshots of EBS volumes 
    #------------------------------------
    description= 'Isolated Instance:' + instance.id + ' on account: ' + event['detail']['accountId'] + ' on ' + date.today().strftime("%Y-%m-%d  %H:%M:%S")
    SnapShotDetails = client.create_snapshots(
        Description=description,
        InstanceSpecification = {
            'InstanceId': instanceID,
            'ExcludeBootVolume': False
        }
    )
    print('Snapshot Created -- %s' % (SnapShotDetails))

    response = 'Instance ' + instance.id + ' auto-remediated'        

except ClientError as e:
    print(e)

return response

**IMPORTANT: **Verify status after execution: check on the EC2 console what is the current status of the instance "BasicLinuxTarget”. You will see that the security group has changed to the one we configured only for the IR team. You will also see new snapshots have been created. You are now seeing our automated response in live action!

Before testing, the security group in the instance was:

After testing, note the change in security group. Our IR security group took over.

*You can check the GuardDuty Dashboard as well to see the threats it detected and the lambda logs, as well as the snapshots created. *

Create EventBridge Rule
- Create a rule in EventBridge that triggers the Lambda function based on findings from GuardDuty. On creation method, select custom pattern and use the following code:

{
  "source": ["aws.guardduty"],
  "detail": {
    "type": ["UnauthorizedAccess:EC2/TorClient", "Backdoor:EC2/C&CActivity.B!DNS", "Trojan:EC2/DNSDataExfiltration", "CryptoCurrency:EC2/BitcoinTool.B", "CryptoCurrency:EC2/BitcoinTool.B!DNS"]
  }
}

As target, select the lambda you previously created. Click on create.

This is one way to manage incident responses automatically in the cloud.

Personal learnings from this project:

•To perform automated basic incident response tasks for containment and gathering data for analysing cyber threats.
•To understand possible actions to take and how to prevent such threats affecting a production environment.

How could this be improved?

As a reflective professional, I like to spend some time after finishing projects thinking how they can be further improved.

For future enhancements, I plan to integrate an SNS topic to notify the incident response team. This will enable manual checks in case of any damage and facilitate reverting to the original state when the situation is under control.

There is also the downside of a 15 minute maximum time for the lambda function. If needed, some environments will probably benefit more of using a different architecture, one that relies on step functions to avoid the time restriction.

 Excited to continue optimising this project for enhanced incident response capabilities. Thanks for reading and happy deploying if you want to give this a go!

Terraform pipeline (IaC for AWS)

Monica Escobar — Mon, 27 May 2024 17:42:57 +0000

Automating Infrastructure Deployment for Static Websites with Terraform in AWS

We'll leverage Terraform, an infrastructure as code (IaC) tool, to orchestrate the process. Here's a breakdown of the key steps involved:

1. Terraform Setup:

Ensure Terraform is installed and configured locally.

2. Configuring the AWS Provider:

Terraform requires an AWS provider configuration file specifying the region and credentials for provisioning resources. We will create a provider.tf file for this.

3. Defining the VPC Network:

We'll define the Virtual Private Cloud (VPC)in the vpc.tf file where the server on the EC2 instance will reside. This includes creating a public subnet for internet access.

Creating Internet Connectivity and Security:

An internet gateway is established (in network.tf) to enable internet access for the VPC. We'll also define a custom route table and associate it with the public subnet for proper routing. Finally, a security group is created to control inbound and outbound traffic for the Jenkins EC2 instance.

4. Building the Terraform Configuration:

The core Terraform configuration file (main.tf) defines the provisioning of our EC2 instance with through a user data script. Additionally, it creates an S3 bucket for hosting the static website and configures it for website hosting.

5. Terraform Workflow:

Once the configuration is complete, we'll use Terraform commands to initialize, plan, and apply the changes. This will provision the resources in our AWS environment.

terraform init terraform plan terraform apply

6. Validating Resources (Optional):

You can now validate the created AWS resources like the EC2 instance, security group, VPC, and S3 bucket to ensure everything is set up correctly.

AWS Code Pipeline - CloudFront - S3 CI/CD Pipeline

Monica Escobar — Sat, 25 May 2024 17:30:46 +0000

Steps to create an automated pipeline in AWS without losing the plot

Resources used:

S3 bucket
CloudFront Distribution
GitHub
Route 53
Domain name
SSL Certificate
Code Pipeline

The Big Picture

Steps followed:

Step 1: Setting Up Your AWS Environment

1.1 Create an AWS Account

If you don’t already have an AWS account, sign up for AWS.

1.2 Set Up IAM User

Create an IAM user with the necessary permissions to access the services you'll be using:

Navigate to the IAM Console.
Create a new user and attach the policies for CodePipeline, S3, CloudFront, Route 53, and Certificate Manager.

Step 2: Purchase a Domain and obtain an SSL Certificate

2.1 Purchase a Domain via Route 53

Go to the Route 53 Console.
Click on Domains -> Register Domain.
Search for your desired domain name and follow the prompts to purchase it.

2.2 Request an SSL Certificate via AWS Certificate Manager

Navigate to the AWS Certificate Manager (ACM) Console.
Click on Request a certificate.
Select Request a public certificate and enter your domain name.
Follow the steps to validate your domain ownership (via DNS is much faster than via email, remember to create a CNAME record for verification purposes).

Step 3: Set Up S3 Bucket for Static Hosting

Go to the S3 Console.
Create a new bucket (e.g: my-portfolio).
Enable static website hosting in the Properties tab (optional tip: add max-age=0 to the header for faster updates).
Set up the bucket policy to allow public read access (or configure CloudFront for secure access).

Step 4: Configure CloudFront for CDN

Navigate to the CloudFront Console.
Create a new distribution.
Set the origin to the S3 bucket you created.
Configure the distribution settings, ensuring to set up the SSL certificate you requested.

Step 5: Configure Route 53 to Point to CloudFront

In the Route 53 Console, navigate to Hosted Zones.
Select your domain and create a new A record.
Set the alias target to your CloudFront distribution.

Step 6: Set Up GitHub Repository

Create a new repository on GitHub and push your application code to it.

Step 7: Create the CI/CD Pipeline with AWS CodePipeline

7.1 Set Up CodePipeline

Go to the CodePipeline Console.
Click Create pipeline and give it a name.
Select New service role to create a new IAM role for CodePipeline.

7.2 Add Source Stage

In the Source stage, select GitHub2 as the source provider.
Connect your GitHub account and select the repository and branch you want to use.

7.3 Add Build Stage (OPTIONAL)

In the Build stage, leave empty or choose CodeBuild (I personally left it empty).

7.4 Add Deploy Stage

In the Deploy stage, choose Amazon S3 as the deploy provider.
Select the S3 bucket you created for static hosting.

Step 8: Test the Pipeline

Review the pipeline configuration and click Create pipeline.
Commit a change to your GitHub repository to trigger the pipeline.
Verify the build and deployment process.
Access your application using the domain name configured in Route 53 or if the cache is at its default setting it will take longer to show in your domain, but you can check the S3 URL.

And this is all you need to create your own automated pipeline from your main branch using AWS and GitHub.

Happy deploying!