Forem: Relay

How to Remediate Unencrypted S3 buckets

Relay — Tue, 26 Jan 2021 19:29:05 +0000

Introduction

Cloud environments are always susceptible to security issues. A significant contributor to this problem is misconfigured resources.

Traditional IT Infrastructure was somewhat static; server hardware only changed every few years. With few changes occurring, security was also more static. The modern cloud environment is a much different challenge. In cloud environments, servers, services, and storage are created with automation, resulting in a dynamic and potentially ever-changing server environment.

Standardized policies and regular enforcement of best practices are key to reducing security risks. New automation can be created to enforce these policies on an ongoing basis. Even if the configuration drifts, automation can pull systems back into compliance.

Unencrypted S3 buckets are an example of a configuration setting that could expose enormous quantities of sensitive data. In this article, we will look at using Relay to enforce encryption that is easy to set up and monitor.

Setting Up Relay for AWS

Relay is an online platform with Graphical User Interface (GUI) and Command Line Interface (CLI) for all your cloud automation use cases. In our example, we will be using the GUI web interface.

You will need to set up a Relay account at https://app.relay.sh/signup to create and run a workflow. The workflow will force encryption on specified S3 buckets in your Amazon Web Services (AWS) account.

The Relay AWS connection requires creating an Identity and Access Management (IAM) user with permissions to edit S3 buckets. In your AWS console, go to the IAM dashboard and set up an IAM user. For example, under “Access management,” I set up a group called “BucketGroup” and a user named “relaydemouser”. Make sure you keep a copy of the “Access key ID” and “Secret access key” as you will need these. The “Secret access key” is only shown once.

The user/group will need permissions to access the S3 buckets. For the example above, I added the policy “AmazonS3FullAccess” to the permissions of the BucketGroup. Do the same with the user/group you create.

This policy gives the IAM user, or group of users, full access to the S3 buckets, including checking configuration, listing buckets, and encrypting S3 buckets.

Once you have a user with the required permissions, you can set up the Relay connection configuration. Open the Connections section of Relay and add a connection to AWS with this IAM user.

You can choose any name for the connection but will need to use the AWS “Access key ID” and “Secret access key” for your IAM user as shown below:

Creating a Workflow

Relay has many sample workflows, including one to remediate unencrypted S3 buckets. Use this link to find it, or, from inside the GUI, click on the workflow icon and then Explore workflows. Under the Security heading, you will see “Remediate unencrypted S3 buckets.”

Looking at a new AWS S3 Storage bucket’s properties page, you will see that encryption is disabled by default.

The workflow will enable server-side encryption. After running the workflow, your default encryption settings will look like this:

You can review the workflow contents (code, graph, and so on) then click on the “Use this workflow” button.

Clicking Try this workflow will bring you to a new workflow dialog with a suggested name. Note that workflow names must be unique within your account. Click on Create workflow and Relay will display the workflow graph:

To run the workflow, click Run in the top right corner of the workflow page. Running the workflow will bring up a dialog showing dryRun = true. Doing a dry run will test the logic but will not make any changes. Go ahead and run the workflow in dryRun mode. Visit the Using Workflows section of the documentation for more information on workflows.

The workflow will stop at the approval step and will not execute the action step to encrypt the S3 bucket. In dryRun mode, the Yes and No approval buttons are disabled. If you rerun the workflow and set the dryRun dialog to false, Relay will stop the workflow at the approval step and allow you to click on “Yes” or “No.” If you click on “Yes,” the flow will continue and encrypt the S3 buckets.

After the workflow is complete, go back to AWS and check the encryption status on your S3 bucket. The buckets that had previously been unencrypted should now be encrypted.

Setting up a Trigger

Relay supports multiple ways to trigger a workflow. We have already walked through the manual trigger, running a workflow manually. You can also schedule a trigger to run automatically, using a process similar to that of a Linux cron job. Relay also supports webhook and REST API triggers. Services which support webhooks can post a JSON payload to Relay when an event happens. The Relay REST API receives JWT-authenticated requests from a remote system and runs a workflow in response. Read all about triggers in the Using Triggers section of the documentation.

For this example, we’ll use a simple schedule trigger to run the workflow at midnight every day. Click on the “Add trigger” button in the first block of the graph to display the Add trigger dialog.

Clicking on “Run a trigger every day” will bring up a code snippet that will be added to your workflow.

Once you save the modified code, you will see that the trigger block in the graph now says it will run every day. A summary of the five digits of Cron time setting ‘* * * * *’ is shown below—an asterisk (*) means no specific value.

Position	Time Unit	Possible Values
1	Minute	0 to 59 or *
2	Hour	0 to 23 or *
3	Day of Month	1 to 31 or *
4	Month	1 to 12 or *
5	Day of Week	0 to 7 or * Both 0 and 7 represent Sunday.

When you run the workflow, it will run immediately, plus run automatically every day. You would need to remove or modify the approval step if you want the workflow to run continuously.

You can remove the schedule trigger by deleting the triggers section in the code and saving changes. The graph button will change back to “Add trigger.”

Next Steps

You have configured a workflow to add encryption to AWS S3 storage buckets, and learned to set up a scheduled trigger. As described in this article, many preconfigured workflows make it easy to get started with Relay. Plus, the code is readily available for modification and expansion outside the GUI interface.

Other example Relay flows are available for many Azure, AWS, and GCP tasks.

Check out Relay and start automating your DevOps maintenance today.

Save Time and Money by Automatically Deleting Unused Azure Load Balancers

Relay — Tue, 12 Jan 2021 00:00:00 +0000

Using the cloud reduces on-premises infrastructure costs and related maintenance. Instead of deploying more servers, storage, and networking components to your own datacenter, you are now deploying these as cloud resources. Using the cloud is supposed to reduce infrastructure and maintenance costs. However, deploying cloud resources also risks over-commissioning, under-usage, and keeping resources running that are not always needed or, even worse, no longer in use.

To help you avoid wasting these unused resources, Puppet created Relay. This tool enables you to automate DevOps cloud maintenance, including automatically cleaning up resources you no longer need. This reduces resource waste while saving DevOps time, helping your team focus on delivering exciting new product features.

Cleaning up Azure

In this article, we walk you through a common scenario. You may be using Azure Infrastructure components, like Virtual Machines and related Virtual Networking resources, together with Azure Load Balancers. When you no longer need the Virtual Machine, and delete it, you may forget about the Azure Load Balancer. The Load Balancer then continues to use your valuable cloud computing resources. Relay workflow helps you clean up.

To use Relay:

Activate your Relay account at relay.sh.
Define a connection between Relay and Azure, using an Azure service principal.
Create your Relay workflow, which then performs the cleanup.

Let’s examine each of these steps with step-by-step guidance. If you already have an Azure subscription with administrative access, you can follow these steps in your own environment.

Setting Up Relay

Setting up a Relay account is rather straightforward. It is a hosted-cloud service with nothing to download, install, update, or maintain.

First, create a new Relay account by completing the required fields on the signup page.

After accepting the activation link in your email, create a complex password, confirm it, and that’s all it takes.

Then, after successfully logging on to the Relay platform, you are ready to start. From the Relay portal, browse sample workflows or create a new one from scratch.

There is also an option to define connections, where you specify the service account of your cloud platform. Relay supports many different public and private cloud environments, including Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform (GCP), and Kubernetes. In our example, we will use Azure, but the process is similar in all environments.

Configuring Access to Azure

Before creating a workflow, let’s start by defining a connection for Azure. This relies on an Azure Active Directory service principal. Think of this object as just another user account, used by an application and similar to an Azure administrative account.

Once you create the service principal, apply Azure Role Based Access Control (RBAC) permissions, limiting this account’s administrative capabilities to keep your production environment secure. Optionally, you can read more about service principal objects.

There are several ways to create a service principal, such as Azure Portal, Azure Cloud Shell, PowerShell, ARM Template, or REST API. We will show you how to create the service principal using Azure Cloud Shell.

First, navigate to the Azure Portal, and open Azure Cloud Shell from the top right menu. Note: if this is the first time you use Azure Cloud Shell, it will ask you to create an Azure Storage Account and Azure FileShare – complete this step to continue. Select Bash as the interface.

Type the following Azure command-line interface (CLI) command:

az ad sp create-for-rbac -n "Relaysp"

This creates the service principal in your Azure Active Directory, and shows the actual account credential details as Shell output. Copy this information for later use.

Go to the Relay portal and select “Connections”. Next, click the “Add Connection” button and choose Azure from the list. This opens the “Set up your Azure Connection” window. Complete the fields, copying the information from the Cloud Shell output, as shown in the below example:

Relay Names Azure Service Connection Names

Subscription ID - Subscription
Client ID - appId
Tenant ID - tenant
Secret - password

Save the information. This creates your connection.

We are now ready to create our workflow.

Creating a Workflow

There are a couple of different ways to create the workflow. Remember, it is a YAML dialect, allowing us to author the file in any intelligent text editor, like Visual Studio Code or similar.

You could write the YAML from scratch, but Relay provides an extensive open source library of sample workflows on GitHub, integrated into the Relay portal. So, let’s have a look.

First, from the Relay portal, navigate to Workflows. Select “Explore our workflow library_”._

From the category list, select “Cost optimization”.

Locate the “Delete empty Azure Load Balancers” workflow.

Click “Use this workflow”, provide a unique name, and confirm by pressing “Create workflow”.

This imports the workflow into your dashboard. From here, you need to complete some minimal configuration settings.

The first warning we get is “missing required connection”. You need to specify which Azure Connection the workflow should use, namely, the one you created in the previous step. To fix this error, click the “Fill in missing connections” link.

Here, it shows a default Connection of “my-azure-account”. You could add a new connection here, however, we already created one. To use it, we need to switch back to the Code view of the workflow and make a change in the YAML.

In the code editor, search for the keyword “connection” (line 27 in the sample file), and replace the name “my-azure-account” with the name of the Azure Connection you created earlier (Azure-Relay in our setup). Make sure you save the changes.

Notice how the connection is ‘recognized’ under the settings pane to the right.

From here, let’s test the workflow to ensure it can successfully connect to our Azure subscription and detect Load Balancers. Click “Run” and confirm by pressing “run workflow” in the popup window. Also, notice this flow can run in dryRun mode, which means it won’t actually change anything in our environment.

The workflow kicks off and displays the step-by-step sequence:

The dry run workflow worked fine. To look at each step in a bit more detail, hover over the step, for example, “list-azure-load-balancers”, and select view logs.

The following output is displayed:

As you can see, the workflow lists the valid load balancers. Note: we didn’t deploy a Load Balancer yet, so it is normal that the result is zero, but at least it ran fine.

Remember, this workflow is checking for “empty load balancers”, which means it looks for Azure Load Balancers without any endpoint connection parameters. To make this a more viable test, let’s deploy a Load Balancer in Azure. If you need some assistance on how to do this, you might use this sample from Azure QuickStart Templates.

This deploys a Standard-type Azure Load Balancer, together with three Virtual Machines (VMs) as a back-end pool. If you want, you can update the Azure deployment template to deploy only a single VM if you don’t want to test three.

After deploying this template, the setup looks similar to this:

Let’s go back to Relay and run our workflow once more. Notice that, this time, the Load Balancer is actually detected.

Next, select the next step in the flow, filter-loadbalancers, and view logs. This reveals that the detected Load Balancer is not empty.

It is interesting to know how Relay identifies if the Load Balancer configuration is empty or not. Let’s have a look at the actual YAML code.

The script includes a pointer to a Python script, which looks like this:

You can see it is checking for any Load Balancers with empty “backend_address_pools”.

To run a more valid test, let’s go back to the Azure Load Balancer, and clear the backend pool configuration. The easiest way to remove this is by using the following PowerShell cmdlet:

Remove-AzureRMLoadBalancerBackendAddressPoolConfig

For more information on doing this, refer to the following Microsoft Doc:

Remove-AzureRmLoadBalancerBackendAddressPoolConfig

Note: if you used the default settings from the Azure Quickstart Template sample deployment earlier, use the following PowerShell script:

Get-AzureRmLoadBalancer -Name "MyLB-lb" -ResourceGroupName "MyLBRG" | Remove-AzureRmLoadBalancerBackendAddressPool -Name "LoadBalancerBackendPool" | Set-AzureRmLoadBalancer

The Load Balancer configuration shows an empty BackEndPool now:

Run the Relay workflow once more to see what happens with the Azure Resource. To test, make sure you keep the DryRun value set as True. The result is the following:

This time, the empty Load Balancer is detected for removal.

Next, trigger the workflow once more, this time setting the DryRun to False, which means it will actually force the removal of the Azure Load Balancer resource. You will also need to confirm “Yes” on the approval step in the workflow.

After about a minute, we get positive confirmation saying all specified load balancers are deleted. Validating from the Azure Portal confirms this.

As a last step in this process, I want to highlight the trigger option at the beginning of the workflow. Up till now, we kicked off the workflow manually. This works fine for testing, but doesn’t work well in production. You may want to schedule this cleanup, validating your environment every week or maybe every night.

To schedule your load balancer cleanup, first, select the first step in the workflow, “Add trigger”.

This opens a list with different triggers to choose from, like running a “per day” job or triggering the workflow using an HTTP interaction. This could be interesting if you want to automate the process from your systems management tool. Whenever you connect to the trigger-HTTP-URL, the workflow executes. Let’s go back to the day-trigger scenario for now.

Select the “Run a trigger every day” option, which reflects the corresponding code. The most important setting here is “schedule”, using the standardized cron notation to define the scheduling format.

For more information on the cron definition, I can recommend this helpful link. You now have one less thing to monitor. You can check it off your busy DevOps team’s to-do list.

Next Steps

In this article, we introduced you to Relay, a product Puppet created to automate cloud management tasks. From first configuring the cloud connection, we explored an example scenario, cleaning up empty Azure Load Balancers. We then learned how to trigger the workflow as a daily scheduled task.

Other example workflows enable you to easily set up maintenance of VMs, network interface controllers (NICs), and Disks.

While this example detailed how to clean up Azure, Relay also works with other cloud environments. You can adjust the example above to develop workflows for any Azure, AWS or GCP resources.

Automating your load balancer cleanup and other DevOps tasks saves you time and money as your team instead focuses on creating great new features for your software applications. Check out Relay to automate your DevOps maintenance today.