Forem: Kenaz Kwa

Terminate Unused EC2 Instances with Python!

Kenaz Kwa — Thu, 11 Jun 2020 22:11:42 +0000

Automatically terminate EC2 instances with configurable lifetimes

This workflow looks at all of the EC2 instances in a given account and region and selects a subset of those to terminate. The termination criteria are:

Not tagged with a termination date or lifetime after 4 minutes
The termination_date or lifetime tags are present but cannot be parsed
The termination_date or lifetime tags indicate that the instance hasexpired

An instance may be configured to never terminate if its lifetime tag has thespecial value indefinite.

Prerequisites

Before you run this workflow, you will need the following:

An AWS account.
An AWS IAM user with permissions to list and terminate EC2 instances (if notrun in dry run mode).
One or more running EC2 instances that are configured to use thetermination_date or lifetime tags.

Run the workflow

Follow these steps to run the workflow:

Add your AWS credentials as secrets:

Click Edit > Secrets.
Click Define new secret and use the following values:
KEY : aws.accessKeyID
VALUE : Enter your AWS access key id associated with the account
KEY : aws.secretAccessKey
VALUE : Enter your AWS secret access key associated with the account
1. Click Run workflow and wait for the workflow run page to appear.
2. Warning: If you run the workflow with the dryRun parameter set tofalse, instances not in compliance with this workflow policy willimmediately be terminated.

Automatically running on a schedule

Policy-driven workflows are best run on a recurring schedule. To set up aschedule trigger for this workflow:

Click Edit > Triggers.

Click Define new trigger and use the following values:

Trigger type : Schedule
Trigger integration : System
Interval : Intervals follow the ISO 8601 repeatingintervalformat. To run this workflow every 5 minutes indefinitely from now on,enter: R/2020-01-01T00:00:00Z/PT5M. You can configure the interval at theend of this string to change the execution frequency.
1. Enter values for the parameter bindings that match your environment.
2. Click Add trigger.

Within the next 5 minutes, you should see the workflow run automatically for thefirst time.

Secure your AWS Account by Removing Unused EC2 Key Pairs with Python

Kenaz Kwa — Fri, 22 May 2020 00:00:00 +0000

Cloud security is top of mind right now, given the various high-profile security breaches today. One overlooked source of potential vulnerabilities is unused EC2 Key Pairs. EC2 Key Pairs are used to configure an EC2 instance with SSH access and provide a convenient way to manage instances. However, when was the last time you performed an audit to make sure that the only key pairs in your account are given to active employees who have proper authorization to connect to instances? Are you sure all of those keys are even being used?

In this blog post, we’ll use a simple Python script to perform an audit of all EC2 key pairs in the account and determine which of those keys are not being used and delete them.

First, let’s find all the keys.

First, we configure the boto3 client to connect to our AWS account and allow us to start listing EC2 Key Pairs. We’ll also create 3 lists - one for storing all key pairs, one for used key pairs, and one for unused key pairs.

import boto3

sess = boto3.Session(
  *# TODO: Supply your AWS credentials & specified region here*
  aws_access_key_id=' MYAWSACCESSKEYID',
  aws_secret_access_key='MYSECRETACCESSKEY',
  region_name='us-east-1', *# Or whatever region you want*
)

# Creating lists for all, used, and unused key pairs
all_key_pairs = []
all_used_key_pairs = []
all_unused_key_pairs = []

Next, we make a call to get all the key pairs and filter for the key pair names:

# List the key pairs in the selected region
ec2 = sess.client('ec2')
all_key_pairs = list(map(lambda i: i['KeyName'], ec2.describe_key_pairs()['KeyPairs']))

Second, let’s find all the key pairs in use.

In order to find all the key pairs currently in use, we first list the EC2 instances and then inspect those instances for their key pair.

# Each EC2 reservation returns a group of instances.
instance_groups = list(map(lambda i: i['Instances'], ec2.describe_instances()['Reservations']))

# Create a list of all used key pairs in the account based on the running instances
for group in instance_groups:
  for i in group:
    if i['KeyName'] not in all_used_key_pairs:
      all_used_key_pairs.append(i['KeyName'])

Next, compare the lists.

We compare each key pair in all_key_pairs to the list of used key pairs. If the key pair is not being used, we add it to the list of unused key pairs.

# Now compare the two lists
for key in all_key_pairs:
  if key not in all_used_key_pairs:
    all_unused_key_pairs.append(key)

Finally, we delete the unused key pairs.

We delete the unused key pairs by iterating over the list of unused key pairs and calling the ec2.delete_key_pair() function:

# Delete all unused key pairs
print('Deleting unused key pairs:')
for key in all_unused_key_pairs:
  print(key)
  ec2.delete_key_pair(KeyName=key)

Find the whole Python script here

Conclusion

Deleting unused key pairs is a good practice to ensure that no one is granted unauthorized access to your instances. Want to run this workflow on a schedule? Invite other people on your team to kick off workflows? Notify your team in Slack when key pairs are deleted? That’s why we built Relay.

We built this workflow out for you so you don’t have to – try it out here.

To learn more about the product, sign up for our updates on relay.sh. Our mission is to free you of tedious cloud-native workflows with event-driven automation! For more content like this, please follow our blog.

Delete Unattached EBS Volumes with Python

Kenaz Kwa — Wed, 06 May 2020 00:00:00 +0000

In our continuation of ways to reduce your AWS bill, one source of unexpected charges are unattached and forgotten EBS volumes. While not typically the most expensive line item on the bill, all of those little volumes can add up when deploying across multiple accounts and regions.

For months, I remember getting bills for nominal amounts of money due to unattached EBS volumes, going into the AWS console to delete them and take care of it, only to realize the next month I’ve forgotten one.

My sad AWS bill

Stop it before it starts - `DeleteOnTermination`

One proactive way of ensuring that EBS volumes are deleted when your EC2 instance is terminated is by ensuring that the DeleteOnTermination attribute is set to true for all volumes that you don’t want to stick around and add up.

To change the DeleteOnTermination attribute, use the following command:

# TODO: Insert your instance id
$ aws ec2 modify-instance-attribute —instance-id i-1234567890abcdef0 —block-device-mappings file://mapping.json

With the following mapping.json:

[
  {
    “DeviceName”: “/dev/sda1”,
    “Ebs”: {
      “DeleteOnTermination”: false
    }
  }
]

For more information, check out the AWS EC2 docs here. By default, the root volume is already configured for DeleteOnTermination to be true.

Running a Python script to audit and delete unattached EBS volumes

Ok, that’s great. But what about all the EBS volumes I have laying around my AWS account right now? Here’s a Python script that solves this problem in 3 parts:

Listing and describing each of your EBS volumes in a given AWS account and region
Filtering the volumes for those that don’t have any attachments
Deleting the unattached EBS volumes

Let’s get started.

Step 1: Listing and describing EBS volumes

First, we configure the boto3) client to connect to our AWS account and allow us to start listing EBS volumes.

import boto3

sess = boto3.Session(
    # TODO: Supply your AWS credentials & specified region here
  aws_access_key_id='MYAWSACCESSKEYID',
  aws_secret_access_key='MYSECRETACCESSKEY',
  region_name='us-east-1', # Or whatever region you want
)

Next, we make a call to get all EC2 volumes:

ec2 = sess.resource('ec2')
volumes = ec2.volumes.all()

This will return the list of all EC2 volumes in the specified region and account. This will return a collection of EBS volume objects.

Step 2: Filtering unattached EBS volumes

Next, we can filter this list and add the ones that are unattached to a termination list to_terminate. Check out the code below:

to_terminate=[]
for volume in volumes:
    print('Evaluating volume {0}'.format(volume.id))
    print('The number of attachments for this volume is {0}'.format(len(volume.attachments)))

    # Here's where you might add other business logic for deletion criteria
    if len(volume.attachments) == 0:
        to_terminate.append(volume)

Why not just delete the volume in the same step? By decoupling the logic of filtering from the deletion logic, we can take different types of actions for volumes with different attributes.

For example, we might delete all the unattached volumes, but we might want to send a Slack notification for all of the volumes that are not encrypted (out of scope for this post).

Step 3: Deleting the volumes

The deletion of the logic is simple. Each volume has a delete() method to delete the volume. We check for the empty condition, then delete all the volumes that are on the termination list.

if len(to_terminate) == 0:
    print ("No volumes to terminate! Exiting.")
    exit()

for volume in to_terminate:
    print('Deleting volume {0}'.format(volume.id))
    volume.delete()

You can find the whole Python script here.

Conclusion

That’s it! Want to run this workflow on a schedule? Invite other people on your team to kick off workflows? Notify your team in Slack when volumes have been deleted? That’s why we built Relay.

To learn more about our mission and product, sign up for our updates on relay.sh. Our mission is to free you of tedious cloud-native workflows with event-driven automation! For more content like this, please follow our blog.

How to Cut Unused EC2 instances with AWS Lambda

Kenaz Kwa — Fri, 13 Mar 2020 17:41:30 +0000

Originally posted on our blog.

Uh... Why is this month's AWS bill so high?

Forgotten AWS EC2 instances have made everyone’s pockets hurt. If you don’t proactively clean up unused EC2 instances, cloud spending can quickly get out of control. However, it can be tedious to routinely check which EC2 instances are still in use, track down the old ones, and remove them. Luckily — we know how to automate these tasks!

This post walks you through de-provisioning unused EC2 instances by using AWS Lambda and CloudFormation to deploy an EC2 reaper that uses simple Tags to cut down on spending.

To see the full code, check out this repo.

AWS EC2 Reaper overview

The AWS Reaper works by checking and enforcing tags that are set on the EC2 instances. All EC2 instances must be tagged with a lifetime or a termination_date. The termination_date defines a future date after which the EC2 instance will be terminated. Alternatively, the Reaper looks for a lifetime tag–if found, it calculates a new future date and adds that date as the termination_date tag for the EC2 instance.

First, let’s look at the reaper.py. The main reaper logic for handling instances is in the terminate_expired_instances function which lists instances and looks up the termination date tag for each instance:

instances = ec2.instances.filter(
        Filters=[{'Name': 'instance-state-name', 'Values': ['running']}])
        print(instances)
        for instance in instances:
          ec2_termination_date = get_tag(instance, 'termination_date')

Improperly Tagged Instances

If we find an instance that doesn’t have a termination_date or we find the tag can’t be parsed, we stop it:

if ec2_termination_date is None:
            print("No termination date found for {0}".format(instance.id))
            stop_instance(instance, "EC2 instance has no termination_date")
            improperly_tagged.append(instance)
            continue

This enables us to stop the b̶l̶e̶e̶d̶i̶n̶g̶ billing while we contact the instance owner to see if it should still be kept around.

Expired Instances

For all instances we find that are expired, we destroy:

if ec2_termination_date != INDEFINITE:
  try:
    if dateutil.parser.parse(ec2_termination_date) > timenow_with_utc():
       ttl = dateutil.parser.parse(ec2_termination_date) - timenow_with_utc()
       print("EC2 instance will be terminated {0} seconds from now, roughly".format(ttl.seconds))
    else:
       terminate_instance(instance, "EC2 instance is expired")
       deleted_instances.append(instance)

Deploying the EC2 reaper

Now, we could just run this python script against different AWS regions and we’d already be better off than doing this manually. However, we would rather not spend time babysitting scripts at all. We’re going to deploy this into AWS using CloudFormation Stacks.

Deploying the AWS reaper has two parts:

deploy_to_s3.yaml AWS CloudFormation template that places the lambda zip resources in S3 buckets in every region so that the deploy_reaper template can read them for Reaper deployment.
deploy_reaper.yaml AWS CloudFormation template that installs the reaper creates the IAM role and deploys the lambda function to perform the instance reaping.

deploy_to_s3 template

In order to use this template, you must first manually create an S3 bucket that contains the resources to copy across all regions. You will need to do this once per region; S3 resources can be read between accounts but not between regions for AWS Lambda. This only needs to be done one time for the administrative account.

Manually create an S3 bucket accessible from the administrative account. Zip up the two python reaper files, reaper.py and slack_notifier.py and place them in the bucket, naming them reaper.zip and slack_notifier.zip.
From the administrative account, create a new stack set and use the deploy_to_s3 template. An example CLI invocation would look like:

$ aws cloudformation create-stack-set --stack-set-name reaper-assets --template-body 
file://path/to/deploy_to_s3.yaml --capabilities CAPABILITY_IAM --parameters
ParameterKey=OriginalS3Bucket,ParameterValue=reaperfiles

Deploy stack-set-instances for this stack set, one per region in the administrative account. Check the Amazon documentation for the most up-to-date region list. For example:

$ aws cloudformation create-stack-instances --stack-set-name --accounts 123456789012
--regions "us-west-1" "us-west-2" "eu-west-1" ...

deploy_reaper template

After the resources for the reaper have been distributed, you can use the deploy_reaper CloudFormation template to deploy the reaper into an account.

You will need to follow the steps below for each account you are deploying the reaper to.

First, create a stack set representing the account you wish to run the reaper in. Example invocation:

$ aws cloudformation create-stack-set --stack-set-name reaper-aws-account --template-body
file://path/to/deploy_reaper.yaml --capabilities CAPABILITY_IAM --parameters
ParameterKey=SLACKWEBHOOK,ParameterValue=1234567 ...

Deploy the reaper into the account.

$ aws cloudformation create-stack-instances --stack-set-name reaper-aws-account --accounts
098765432109 --regions "us-west-1" "us-west-2" "eu-west-1" ...

Turning on the EC2 Reaper

Once deployed, the EC2 Reaper will not reap anything unless the environment variable LIVEMODE is set to TRUE. It will only report what it would have done to Slack.

When you want to activate the Reaper, update the parameter value LIVEMODE to "TRUE"(the regex is case-insensitive).

$ aws cloudformation update-stack-set --stack-set-name reaper-aws-account
--use-previous-template --parameters ParameterKey=LIVEMODE,ParameterValue=TRUE --capabilities CAPABILITY_IAM

Conclusion

Now you have learned how to control costs on AWS by reaping old EC2 instances. To learn more about our mission and product, sign up for our updates on https://relay.sh. Our mission is to free you of tedious cloud-native workflows with event-driven automation!

How to Cut Unused EC2 instances with AWS Lambda

Kenaz Kwa — Wed, 11 Mar 2020 21:20:20 +0000

How to Cut Unused EC2 Instances with AWS Lambda

Uh… Why is this month’s AWS bill so high?

Forgotten AWS EC2 instances have made everyone’s pockets hurt (including Puppet!). Take it from us (relay.sh team) — if you don’t proactively clean up unused EC2 instances, cloud spending can quickly get out of control. However, it can be tedious to routinely check which EC2 instances are still in use, track down the old ones, and remove them. Luckily — we know how to automate these tasks!

Our mission is to free you to do what robots can’t.

This post walks you through de-provisioning unused EC2 instances by using AWS Lambda and CloudFormation to deploy an EC2 reaper that uses simple Tags to cut down on spending.

To see the full code, check out this repo.

puppetlabs/aws_resource_reaper

AWS EC2 Reaper overview

The AWS Reaper works by checking and enforcing tags that are set on the EC2 instances. All EC2 instances must be tagged with a lifetime or a termination_date. The termination_date defines a future date after which the EC2 instance will be terminated. Alternatively, the Reaper looks for a lifetime tag– if found, it calculates a new future date and adds that date as the termination_date tag for the EC2 instance.

First, let’s look at the reaper.py. The main reaper logic for handling instances is in the terminate_expired_instances function which lists instances and looks up the termination date tag for each instance:

Improperly Tagged Instances

If we find an instance that doesn’t have a termination_dateor we find the tag can’t be parsed, we stop it:

This enables us to stop the b̶l̶e̶e̶d̶i̶n̶g̶ billing while we contact the instance owner to see if it should still be kept around.

Expired Instances

For all instances we find that are expired, we destroy:

Deploying the EC2 reaper

Deploying the AWS reaper has two parts:

deploy_to_s3.yaml AWS CloudFormation template that places the lambda zip resources in S3 buckets in every region so that the deploy_reaper template can read them for Reaper deployment.
deploy_reaper.yaml AWS CloudFormation template that installs the reaper creates the IAM role and deploys the lambda function to perform the instance reaping.

deploy_to_s3 template

Manually create an S3 bucket accessible from the administrative account. Zip up the two python reaper files, reaper.py and slack_notifier.py and place them in the bucket, naming them reaper.zip and slack_notifier.zip.
From the administrative account, create a new stack set and use the deploy_to_s3 template. An example CLI invocation would look like:

$ aws cloudformation create-stack-set --stack-set-name reaper-assets --template-body 
file://path/to/deploy\_to\_s3.yaml --capabilities CAPABILITY\_IAM --parameters
ParameterKey=OriginalS3Bucket,ParameterValue=reaperfiles

Deploy stack-set-instances for this stack set, one per region in the administrative account. Check the Amazon documentation for the most up-to-date region list. For example:

$ aws cloudformation create-stack-instances --stack-set-name --accounts 123456789012
--regions "us-west-1" "us-west-2" "eu-west-1" ...

deploy_reaper template

After the resources for the reaper have been distributed, you can use the deploy_reaper CloudFormation template to deploy the reaper into an account.

You will need to follow the steps below for each account you are deploying the reaper into.

First, create a stack set representing the account you wish to run the reaper in. Example invocation:

$ aws cloudformation create-stack-set --stack-set-name reaper-aws-account --template-body
file://path/to/deploy\_reaper.yaml --capabilities CAPABILITY\_IAM --parameters
ParameterKey=SLACKWEBHOOK,ParameterValue=1234567 ...

Deploy the reaper into the account.

$ aws cloudformation create-stack-instances --stack-set-name reaper-aws-account --accounts
098765432109 --regions "us-west-1" "us-west-2" "eu-west-1" ...

Turning on the EC2 Reaper

Once deployed, the EC2 Reaper will not reap anything unless the environment variable LIVEMODE is set to TRUE. It will only report what it would have done to Slack.

When the time comes to activate the Reaper, update the parameter value LIVEMODE to "TRUE"(the regex is case-insensitive).

$ aws cloudformation update-stack-set --stack-set-name reaper-aws-account
--use-previous-template --parameters ParameterKey=LIVEMODE,ParameterValue=TRUE --capabilities CAPABILITY\_IAM

Conclusion

Now you have learned how to control costs on AWS by reaping old EC2 instances. To learn more about our mission and product, sign up for our updates on relay.sh. Our mission is to free you of tedious cloud-native workflows with event-driven automation! For more content like this, please follow our medium page at https://medium.com/relay-sh.

Building the future of DevOps automation

Kenaz Kwa — Wed, 04 Mar 2020 00:00:00 +0000

Today, we’re starting Relay by Puppet, a new initiative to build the future of cloud-based DevOps automation. This post captures our musings about the evolving world of cloud-native infrastructure and how Relay addresses this evolution.

Human-triggered operations

Working with many of the largest companies in the world, we’ve seen server virtualization increase the number of things that operators have to manage from 1x to 10x. The automation we’ve built today enables us to scale to large infrastructure estates and eliminate this manual work. We’ve made one-click automation possible.

However, in moving to public cloud, the number of things that operators have to manage has gone from 10x to 100x. A single VM running in a public cloud comprises a set of cloud resources that include the VM itself, the network and subnet it resides in, the IP address, the storage account, the disks, the security rules, the roles and permissions, and more. When you start to layer on container orchestration, the number of things to manage increases dramatically.

Similarly, the number of tools that operators use every day and tie together to solve automation challenges has exploded in number and complexity. Operators must learn infrastructure provisioning with Terraform, configuration management with Bolt, Kubernetes deployments with Helm, building pipelines with Jenkins, maintain a whole flock of public cloud services — all in addition to responding to the endless queue of ServiceNow tickets, demanding that someone restart their server for them.

the now infamous Cloud Native Landscape

In this world, human-triggered operations can no longer keep up. When you’re dealing with hundreds of tasks, even one-click automation is one click too many. Operators today need a different category of automation that responds to monitoring alerts, new service request tickets, cloud events and more. This kind of automation eliminates the low-value tasks operators spend their time doing today so that they can focus on the high-value work for their teams.

When you’re dealing with hundreds of tasks, even one-click automation is one click too many.

Meet Relay

Relay is an event-driven automation service for eliminating low-value DevOps tasks. Relay makes it easy to pull together the tools you use today into a single workflow and automate these workflows with an event-based trigger.

We think there are a few useful scenarios here that either started out as problems for us or the DevOps engineers we talked to about Relay:

Incident triggers: Enabling PagerDuty or VictorOps incident to trigger creating a war room in Slack
Ticket triggers: Using a Jira ticket to trigger infrastructure provisioning of development environments with Terraform
Git triggers: Deploying new versions of a containerized application to Kubernetes when a Github PR is merged
Monitoring triggers: Clearing out /var/log when SignalFx throws a disk utilization alert
Cloud event triggers: Making sure new S3 buckets are configured with the appropriate permissions when created

We’re not trying to boil the ocean here. Instead, we’re building a small set of use cases for DevOps teams that will make life way better for us (and hopefully you too).

Get involved

We’re hard at work on our public beta launch (coming soon!) and we’d love to hear from you. What are the things that you’d like to automate today? Let us know. What’s a terrible idea (hello, storage policies should be separate from alerting)? Tell us in exchange for free swag!

Over the next couple months, we’ll talk through our thinking, our bets, our progress, and what we’re hearing from folks like you.

Let us know what to automate. We’ll give you free stuff!

Interested in following along? Join our mailing list and we’ll certainly offer our earliest supporters perks along the way (Kickstarter-style vague promises — check!).

Thanks

Good ideas (and products) don’t happen in a vacuum. And really, no ideas are new. We’re inspired by the work that teams at Zapier, Stackstorm, Github Actions, and others have created before us. In the next few months, we’ll talk about how we see Relay being similar or different from many of these tools.

Forem: Kenaz Kwa

Terminate Unused EC2 Instances with Python!

Automatically terminate EC2 instances with configurable lifetimes

Prerequisites

Run the workflow

Automatically running on a schedule

Secure your AWS Account by Removing Unused EC2 Key Pairs with Python

First, let’s find all the keys.

Second, let’s find all the key pairs in use.

Next, compare the lists.

Finally, we delete the unused key pairs.

Conclusion

Delete Unattached EBS Volumes with Python

Stop it before it starts - DeleteOnTermination

Running a Python script to audit and delete unattached EBS volumes

Step 1: Listing and describing EBS volumes

Step 2: Filtering unattached EBS volumes

Step 3: Deleting the volumes

Conclusion

How to Cut Unused EC2 instances with AWS Lambda

AWS EC2 Reaper overview

Improperly Tagged Instances

Expired Instances

Deploying the EC2 reaper

deploy_to_s3 template

deploy_reaper template

Turning on the EC2 Reaper

Conclusion

How to Cut Unused EC2 instances with AWS Lambda

How to Cut Unused EC2 Instances with AWS Lambda

Uh… Why is this month’s AWS bill so high?

AWS EC2 Reaper overview

Improperly Tagged Instances

Expired Instances

Deploying the EC2 reaper

deploy_to_s3 template

deploy_reaper template

Turning on the EC2 Reaper

Conclusion

Building the future of DevOps automation

Human-triggered operations

When you’re dealing with hundreds of tasks, even one-click automation is one click too many.

Meet Relay

Get involved

Let us know what to automate. We’ll give you free stuff!

Thanks

Stop it before it starts - `DeleteOnTermination`