Forem: Vantage

Using CloudFormation StackSets to Onboard Hundreds of Member AWS Accounts

Everett — Fri, 17 Jun 2022 21:34:20 +0000

This is the second post in a series of two on how Vantage securely connects to user AWS accounts. Start by reading the first post on how Vantage uses cross-account IAM roles.

In the previous post we talked about how Vantage quickly and securely connects to users' AWS accounts. Immediately after connecting the root account the user can see all their cost data and begin exploring Cost Reports due to the fact that AWS root accounts have access to all member account cost data.

However, in order to access resource level information - for instance seeing the full list of currently running EC2 instances - users have to connect one or more member accounts where those resources reside. This post outlines the reasoning for utilizing member accounts in the first place and how we use CloudFormation StackSets to automate the connection to these member accounts.

Using Multiple AWS Accounts

It is considered a best practice to set up multiple member accounts as part of an AWS organization. It allows for better security practices, simplified cost allocation and for teams to move more independently. It is common for organizations to create AWS accounts per environment, team or project which can lead to dozens or even hundreds of AWS member accounts. AWS automatically aggregates cost data from member accounts up to the root account in order to provide unified billing reports, however access to service APIs, which Vantage uses to collect data about individual resources, requires specifically defined permissions for that account.

While it takes less than a minute to connect a single AWS account to Vantage, doing this process for each member account was cumbersome for our customers and took a long time. We wanted to provide as close to the "1-click" experience we have for connecting a single account and make it as seamless as possible to connect any number of member accounts. After a bit of research we decided to leverage CloudFormation StackSets.

CloudFormation StackSets

CloudFormation StackSets are an extension of CloudFormation Stacks which allow you to run the same Stack across multiple AWS accounts. If you are unfamiliar with CloudFormation, it is an AWS service which provisions and configures AWS services based on an inputted template. With StackSets you also provide a single template along with a configuration of which accounts it should be deployed in and what permission mode should be used.

Vantage currently uses the "service-managed" permission mode which requires AWS Organizations to be set up and configured. The other option is the "self-managed" permission mode which requires more setup and a role to be created in each member account. With the service-managed mode it requires no additional setup if your organization is already configured and it will automatically run the stack when you create a new member account. This is great because with no additional work new member accounts will automatically be connected to Vantage.

Connecting to Vantage with StackSets

The one downside to StackSets is there is currently no console UI we can integrate with to make it easy for the user to complete this process through the AWS UI. When not using StackSets we use the "quick-create links" functionality of CloudFormation to pass in required information via URL parameters.

Using the "quick-create links", the user just has to click through the wizard to create the connection to Vantage. With StackSets, we instead give the user a set of AWS CLI commands to run in order to kick off the process. (P.S. If anyone from the CloudFormation team is reading, it would be great if AWS added the quick-create functionality for StackSets.)

The first command creates the Stack in the root AWS account, but does not yet run anything. Vantage passes a series of parameters (which are the same parameters that are passed to the regular CloudFormation quick-create interface) and a couple configuration options for StackSets. There are two configuration options you need to pass when creating a service-managed StackSet. AutoDeployment dictates whether new member accounts should automatically run this stack when they are created which the user can set to false if they would like to disable this functionality. RetainStacksOnAccountRemoval will automatically remove any created resources if the account is ever removed from the Organization.

aws cloudformation create-stack-set \

  --auto-deployment Enabled=true,RetainStacksOnAccountRemoval=true \

  --permission-mode SERVICE_MANAGED \

  --stack-set-name <VANTAGE_GENERATED_NAME> \

  --template-url <PUBLIC_URL_FOR_STACK_TEMPLATE> \

  --parameters <SERIES_OF_VANTAGE_SPECIFIC_PARAMETERS>

The second command actually starts the process of running the Stacks for a StackSet. This command requires the name of the stack which was just created, the region and accounts in which you would like to deploy the stacks. For the accounts we use the Root organization ID since we want to deploy this to all member accounts, but this can be changed to use the Accounts parameter instead to limit the deployment to specific accounts.

aws cloudformation create-stack-instances \

  --stack-set-name <VANTAGE_GENERATED_NAME> \

  --regions us-east-1 \

  --deployment-targets OrganizationalUnitIds=<ROOT_ORGANIZATION_ID>

Previous to this process we were not aware of StackSets. Overall they provide a very easy way for us to automate the connection to any number of member accounts and a great user experience for Vantage users. Users looking to find these commands with all of the correct parameters filled in can visit the console settings page.

CloudWatch Metrics Pricing Explained in Plain English

Jono Bergquist — Wed, 18 Aug 2021 21:14:34 +0000

Amazon CloudWatch is an observability tool that gives you the power to monitor your cloud infrastructure and know when there is a service issue. CloudWatch achieves this by aggregating logs and metrics in a central datastore and providing visualization tools for the data. Observability is commonly defined as having "3 Pillars'' or primary components to achieve full observability into a system. Metrics and logs are two of these pillars; the third is traces. The core CloudWatch service offers metrics and logs out of the box. Traces are offered as a separate service called AWS X-Ray which can be integrated into CloudWatch via ServiceLens. Since AWS X-Ray is billed as a separate service we won't cover pricing in this article.

CloudWatch Pricing

Generally, observability tools are made up of 3 primary architectural components; a data ingest pipeline, a datastore and a management console. Observability tools tend to price their service based on these 3 components; CloudWatch is no different. There will be costs for ingesting data into the datastore, costs for retaining data in the datastore and costs for the visualization/management tools needed to work with the data in order to derive insights.

Part 1 of this series will discuss CloudWatch Metrics. Part 2 of this series will discuss CloudWatch Logs and Dashboards.

CloudWatch Free Tier

CloudWatch offers a free tier which will always be applied to your service before you receive any charges based on the paid tier. The free tier has small allowances for each CloudWatch service (Metrics, Logs, Dashboards, etc.) You can view all of the free limits by visiting the CloudWatch pricing page.

CloudWatch Metrics Pricing

CloudWatch prices its Metrics service per metric that is sent to CloudWatch and how frequently the API is called to send or pull a metric. The more metrics you send to CloudWatch and the higher the frequency you call the API, the higher your bill. This is important to keep in mind because the more metrics that you track the easier it is to diagnose specific problems in your system and the higher the rate you send metrics the more granular and precise you can be when solving service issues. Essentially, CloudWatch is priced on fidelity; the higher fidelity the data that is tracked and stored, the higher the cost.

Different than CloudWatch Logs, CloudWatch Metrics do not incur a separate ingest and storage fees. Instead, Metrics combines the fee for ingest and storage into a single fee based on the number of custom metrics tracked. CloudWatch Metrics is able to prevent abuse of storage costs by having an additional cost per API call (which could be considered a data ingest fee) and having a data retention policy that rolls up high resolution metrics into less granular metrics over time. See the section below on data retention below for full details on how metrics are aggregated over time.

CloudWatch Metrics offers volume discounting as you track more custom metrics. Here is the pricing table for Custom Metrics in US East 2 (Ohio):

Tiers	Cost (metric/month)	Cost of Fully Utilized Tier (month)
First 10,000 metrics	$0.30	$3,000
Next 240,000 metrics	$0.10	$24,000
Next 750,000 metrics	$0.05	$37,500
Over 1,000,000 metrics	$0.02	--

Volume discounts are great for customers that spend tens of thousands of dollars a month on CloudWatch. In order to benefit from the best unit price discount, you'll need to be paying $64,500 per month in CloudWatch Metrics costs. Once your CloudWatch Metrics specific costs hit that level you will be able to take advantage of the $0.02 unit cost for all metrics over 1,000,000. This pricing tier is really only meant for large enterprises, smaller businesses aren’t likely to be able to benefit from the higher discounting tiers.

Lastly, pricing is dependent on the region where you store your metrics¹ and metrics exist only in the region in which they are created. Due to this, the costs associated with CloudWatch will vary depending on the distributed nature of your AWS service architecture.

What is a metric?

A metric is a specific value over time. For example, if I want to track the memory utilization for a specific EC2 instance this would be counted as one metric. If I have a group of 10 EC2 instances and I want to track the memory utilization for each of these instances, the memory utilization metric per instance would count as a metric for billing purposes; 10 total in this example. Costs for CloudWatch Metrics scale based on the number of unique metrics (e.g. memory utilization, requests per second, etc.) you want to track and the number of resources (count of EC2 instances, etc.) where each unique metric needs to be collected from.

CloudWatch Metrics Free Tier Allowances

CloudWatch Metrics offers 3 different allowances on its free tier.

Basic Monitoring (see below)
10 Custom Metrics (EC2 Detailed Monitoring metrics count toward this quota)
1 Million API requests (not applicable to GetMetricData and GetMetricWidgetImage)²

Basic monitoring provides users a few core metrics per service to ensure that users are able to monitor a particular AWS service for availability and high-level performance characteristics. Most AWS Services (EC2, EBS, RDS, S3, Kinesis, etc.) offer basic monitoring.³ None of the metrics tracked under basic monitoring are billed to a customer. The resolution of basic monitoring is dependent on the service, some default to 1-second, others to 5-second. You can consult the full documentation to understand the specifics of basic monitoring metrics

Basic Monitoring for EC2

Basic monitoring of an EC2 instance includes CPU load, disk I/O, and network I/O metrics. You probably noticed that one metric that is normally considered a baseline metric to track is missing; EC2 does not expose metrics related to memory. You will need to implement a custom metric to track this. By default, Amazon EC2 sends metric data to CloudWatch in 5-minute intervals. If this level of monitoring isn't sufficient for your needs AWS offers a higher fidelity level of monitoring called detailed monitoring.

Detailed Monitoring for EC2

Detailed monitoring delivers metrics in 1-minute intervals, rather than 5-minute intervals. This provides higher fidelity resolution of metrics so that you will be able to know and act on service issues quicker. There is a pricing implication to this though. If you enable detailed monitoring, you will be charged for all of the metrics that were previously included as part of basic monitoring. Detailed monitoring essentially charges all of the basic monitoring metrics as custom metrics from a pricing perspective.

For example, if you are running 10 EC2 instances with detailed monitoring enabled. Your CloudWatch Metric bill will be $18 per month. This is because by default, for the most commonly used EC2 instance types, there are 7 built-in metrics tracked. 7 metrics per instance * 10 instances = 70 metrics total but you get 10 metrics free as part of the CloudWatch free tier, so only 60 metrics will be charged. The cost per metric is $0.30 per month for the first 10,000 metrics. In this scenario we only have 60 metrics which is well below 10,000 which means none of the volume pricing discounts will apply. 60 metrics * $0.30 = $18 per month.

Custom Metrics

Custom metrics are metered differently than any of the built-in metrics. You are charged for both the number of custom metrics that are tracked and the number of API calls (i.e. PutMetricData API calls) that are made.

If you want to be able to monitor memory utilization at a resolution faster than 1 minute then you'll see your API calls scale in a corresponding fashion. For example, if you want to track memory utilization at a 15-second resolution then you will pay 4x the amount in API costs because the API call will be made 4 times per minute. High-resolution metrics like this are only maintained at this level of granularity for a period of time after which they are rolled up into a less granular time period.

Dimensions

Every metric has specific characteristics that describe it, and you can think of dimensions as categories for those characteristics. For example, a metric might be CPU utilization but a dimension might be CPU core. For multi-core machines this could lead to many CPU metrics being tracked by CloudWatch. Dimensions are part of the unique identifier for a metric, whenever you add a unique name/value pair to one of your metrics, you are creating a new variation of that metric. CloudWatch treats each unique combination of dimensions as a separate metric, even if the metrics have the same metric name. In the example above, if the CPU utilization metric is tracked on a 4-core CPU machine then it would create 4 unique metrics and be charged as 4 Custom Metrics rather than one.

It is important when deciding to track specific dimensions of a metric that the cardinality of the dimension is considered. High-cardinality metrics like IP Address or a unique identifier can cause the number of CloudWatch metrics tracked to explode which will have an equally large impact on your AWS bill.

Data Retention

CloudWatch Metrics can receive metrics with a variety of different periods. As a data point ages, it will be “rolled up” or averaged into a less granular data point that aggregates a number of surrounding values. If you create a custom metric with a 15-second resolution and want to look at that metric 4 hours into the last, you will only be able to see the metric at a 1-minute resolution. After 3 hours, CloudWatch will aggregate your 15-second metric into a 1-minute metric thereby reducing the resolution of the high-resolution you pay extra for. Full details on how CloudWatch rolls up metrics below:

Data points with a period of less than 60 seconds are available for 3 hours. These data points are high-resolution custom metrics.
Data points with a period of 60 seconds (1 minute) are available for 15 days
Data points with a period of 300 seconds (5 minute) are available for 63 days
Data points with a period of 3600 seconds (1 hour) are available for 455 days (15 months)⁴

CloudWatch Metrics Best Practices

TL;DR

Remove unneeded custom metrics
Reduce the resolution of metrics that do not need high resolution
Be careful with setting dimensions for a metric, high-cardinality metrics can have a large impact on the total number of metrics that are tracked
Take advantage of free basic monitoring on AWS services that are not high priority

Metrics are a balance between data fidelity and cost. The more data that you collect, the more likely you will have the data necessary to pinpoint exactly what went wrong when a service has an issue. Not all service issues are weighted equally, internal services or less important external services can tolerate some level of service disruption and not materially impact your business. Correctly, prioritizing the importance of a service will help in determining which services deserve more monitoring costs associated with them. Mission-critical services should have the most metrics tracked with the highest metric resolution while less important services should only utilize the bare minimum amount of metrics and resolution to keep costs down.

Custom Metrics and EC2 Detailed Monitoring are both premium monitoring tools that should be evaluated using the above framework. Unnecessary or overly detailed metrics will not help a system perform any better but will accrue costs. Consider whether a metric will help diagnose a specific issue before monitoring it or if a metric is already tracked, ask whether the metric has been useful in previous analysis and should be continued to be monitored. If a service is not a high priority, will Basic Monitoring suffice?

The resolution of a metric should also be considered, high-resolution metrics can be vital during an analysis to help find a need in a haystack but they carry a cost. If a service is so important that knowing there is an issue within seconds is important then utilize a high-resolution metric. If on the other hand, having a 5 minute delay on metric delivery and not being able to increase granularity on the metric is okay for the service, there is a cost savings to reducing metric resolution.

These are the most important questions to ask while architecting a well observed system that is also cost effective.

Conclusion

CloudWatch Metrics are a large and complex piece of the entire puzzle of managing costs in AWS. The goal of this series is to help demystify that complexity, surface relevant ways to think about CloudWatch costs and present a list of best practices on how to manage these costs. In the next post in the series, we’ll dive deep on the second major component of CloudWatch costs; CloudWatch Logs.

The examples in the post are great for illustrating hypothetical situations to better discuss the topics. If you’re looking to see how these concepts apply to your Amazon CloudWatch bill with your data, Vantage has created a Cost Transparency platform to help users better understand their AWS costs and do automated analysis to surface when an organization isn’t applying CloudWatch cost best practices. Take a look at your data by creating a Vantage account and connecting your AWS account to it. Users typically find 20% cost savings within the first 5 minutes of signing up.

Interested in this content? Join our Slack Community

Or at least this is what the CloudWatch pricing page states. If you click through all of the regions the prices are all the same for custom metric storage as of July 28, 2021. There is a slight pricing variation for API calls in GovCloud. ↩
There isn't an official explanation as to why GetMetricData and GetMetricWidgetImage isn't included as part of the free tier allowance for CloudWatch Metrics but it probably related to the fact that 3rd party CloudWatch competitors utilize these API calls in order to pull data from CloudWatch into their systems. ↩
Request metrics for Amazon Simple Storage Service (Amazon S3) and Amazon Simple Email Service (Amazon SES) events sent to CloudWatch incur charges. Amazon Kinesis Data Streams enhanced (shard-level) metrics and AWS Elastic Beanstalk enhanced health reporting metrics sent to CloudWatch incur charges. https://aws.amazon.com/premiumsupport/knowledge-center/cloudwatch-understand-and-reduce-charges/ ↩
Data retention policy. https://aws.amazon.com/cloudwatch/faqs/ ↩

Securely connecting to third party AWS accounts using cross account roles

Brooke McKim — Mon, 16 Aug 2021 16:11:05 +0000

From the beginning of Vantage we have had the need to connect to our customer's AWS accounts to interact with AWS services, ingest cost data and potentially take actions on resources within their account. Developing a product which has direct access to a company's infrastructure requires diligence and we take this responsibility very seriously. We’ve invested considerable time and effort into ensuring the mechanics of our integration is secure and we thought the community would benefit from us sharing our approach.

During our invite-only, early-access period nearly a year ago we started down the “easy” path of having the customer create an IAM User in their account and then giving Vantage the access key and secret for that IAM User. While this was very quick for us to implement in a time where we needed customer feedback, it required additional effort from our customers and did not follow AWS security best practices. After doing some additional research and talking with AWS representatives we changed our approach and implemented the Cross-Account IAM Role methodology.

Connecting to a customer's AWS account via a Cross-Account IAM Role requires more effort on the part of the provider (in this case, Vantage), but ends up being more convenient for our users and provides a much higher level of security. This method works by creating an IAM Role in the customer's AWS account with a specific set of permissions (in the case of Vantage, a set of read-only permissions to a variety of AWS services) and then specifying which other IAM Roles have the permissions to "assume" that role. While Vantage uses this method to interact across AWS accounts from different companies, AWS recommends using this same method for interactions between AWS accounts within your own organization as well.

So what are the benefits of Cross Account IAM Roles? It is very common for companies to have multiple AWS accounts. These accounts have completely separate users, roles and permissions. To make it easier to maintain and reduce the blast radius of a security incident, best practices dictate maintaining your IAM users in a management account and specifically granting permissions for those users to assume predefined roles present in the other accounts in the organization.

There are a few benefits with this approach:

If an IAM user’s password is compromised you only have to reset it in one place (the management account).
When Vantage assumes a role, the access to that role is temporary and the credentials expire after a specified period of time - which is defined via a TTL.
The customer ultimately can add or remove IAM permissions on their IAM Role very easily.
The customer can revoke access to Vantage very easily.

Vantage’s Cross Account IAM Role Architecture

In the Vantage AWS account we create an IAM User and IAM Role per customer. This IAM User has access to assume that IAM Role and that role is what is configured to assume the role in the customer’s account. This process is called Role Chaining and again creates a model where the user is maintained separately from the role. This allows Vantage to make API calls on behalf of the customer.

One more layer of security we employ is the use of an External ID. This is an unique ID we generate per customer account and supply in the creation of the IAM Role in the customer’s account. Only if you supply that specific ID when assuming the role does it actually grant you access. The primary function of the External ID is to address and prevent the "confused deputy problem”. This prevents malicious users from guessing the ARN of another Vantage user's IAM Role and submitting a connection to Vantage claiming to be the owner of that ARN. Because the External IDs won't match between accounts, the connection to assume this role will fail. Without the use of an External ID, a malicious user could get all the same privileges of the compromised account.

Heightened Security Doesn’t Mean Sacrificing Developer Experience

At Vantage we strongly believe in providing an intuitive developer experience in all aspects of our product. While the details mentioned above are obviously complex, we believe it's our responsibility to ensure our self-service onboarding process is as easy as possible. In this case, we wanted to maintain a “one-click” (or copy and paste CLI command) experience for our customers onboarding to Vantage without the need to manually research and configure a multitude of thing

As a result, we turned to AWS CloudFormation to automate the creation of the IAM Role in the customer's account and also communicate back to Vantage to hand us the information necessary to complete the connection. In order to communicate back to Vantage we took advantage of the AWS SNS Custom Resources functionality which allows you to specify an SNS topic and the data which will be sent to that topic inside of a CloudFormation template.

Conclusion

Overall we thought this was a great process built on primitives provided by AWS. AWS representatives also told us that not everyone was utilizing this method to connect to third-party AWS accounts. In light of that, we thought it would be useful to share our implementation with the community to ensure as many products as possible were utilizing the cross-account method.

This blog post details how we connect to a single AWS account. Our larger customers have requirements for connecting dozens or maybe hundreds of AWS accounts. The next post in this series will cover our implementation of CloudFormation StackSets which make it very easy for our users to connect all of their organizational member accounts in two simple commands in five minutes or less.

Want to see the cross account IAM role in action? Sign up for Vantage here.
Interested in solving problems like this? We’re hiring! See our open roles.
Want to discuss this topic and more? Join our Slack Community.