Forem: Ross Wickman

Automating Contact Updates Across Accounts In Your Organization

Ross Wickman — Fri, 23 Jan 2026 19:35:00 +0000

Standardizing contact information across a growing AWS footprint is a critical, yet often overlooked, part of cloud governance. Whether it’s ensuring the billing department receives invoices or making sure security notifications reach your SOC, manual updates are not scalable.

Today, we’re sharing a unified automation approach to manage both Primary and Alternate contacts across an entire AWS Organization using Python and Boto3.

The Challenge

As organizations scale, member accounts often end up with stale or inconsistent contact metadata. This can lead to missed security alerts or operational bottlenecks. AWS now provides APIs through the account and organizations services to handle these updates programmatically from the Management account.

The Solution: Automated Contact Management

We have published a comprehensive script on our Cacher Snippets library that handles two core workflows:

Alternate Contact Automation: Targets specialized roles (Security, Operations, and Billing).
Primary Contact Standardization: Mass-updates corporate address and identity details.

Access the full code here: https://snippets.cacher.io/snippet/78a7e179bfb70e1be449

Step-by-Step Implementation Guide

Follow these steps to deploy the scripts safely within your environment:

1. Configure IAM Permissions

The execution environment (Local machine, Lambda, or CloudShell) must be authenticated to the Management Account (or a delegated administrator account) with the following permissions:

organizations:ListAccounts
account:PutContactInformation
account:PutAlternateContact

2. Set Up Your Exclusion List

To prevent overwriting the Management account or specific "Break Glass" accounts, update the exclusion logic in the script:

Python

# Skip/Exclude sensitive accounts
if account_id != '111111111111' and account_id != '222222222222':
    put_contact_information(account_id)

3. Define Your Standard Metadata

Populate the ContactInformation and AlternateContact dictionaries in the script with your organization’s standard data (e.g., your centralized SOC email for Security contacts and corporate headquarters for the address).

4. Execution

Run the script using Python 3. The process will:

Paginate through your entire AWS Organization.
Identify member accounts while respecting your exclusion list.
Update the Primary and Alternate contacts, providing a console log for success or failure for each account.

Why This Matters

Automating these details ensures that AWS can always reach the right people during a security event or billing inquiry. By utilizing the organizations paginator, you ensure no new accounts are missed, keeping your governance posture consistent as you grow.

*For more AWS automation tips and cloud security insights, stay tuned to the Sentri Cloud Blog.

import boto3

# Assuming org_client is already initialized
org_client = boto3.client('organizations')

# Collect all accounts using paginator
accounts = []
paginator = org_client.get_paginator('list_accounts')
for page in paginator.paginate():
    accounts.extend(page['Accounts'])

# Update the alternate contact details for each account in the organization
for account in accounts:
    account_id = account['Id']

    # Initialize a session using the AWS account
    account_session = boto3.Session()
    account_client = account_session.client('account')

    # Now you can use account_client to update alternate contact details or perform other actions
    try:
        # Update alternate contact details for the account
        account_client.put_alternate_contact(
            AccountId=account_id,
            AlternateContactType='SECURITY',
            Title='',
            EmailAddress='',
            Name='',
            PhoneNumber=''
        )
        account_client.put_alternate_contact(
            AccountId=account_id,
            AlternateContactType='OPERATIONS',
            Title='',
            EmailAddress='',
            Name='',
            PhoneNumber=''
        )
        account_client.put_alternate_contact(
            AccountId=account_id,
            AlternateContactType='',
            Title='',
            EmailAddress='',
            Name='',
            PhoneNumber=''
        )

        print(f"Updated alternate contact details for account: {account_id}")
    except Exception as e:
        print(f"Failed to update alternate contact details for account: {account_id}. Error: {e}")

import boto3

def put_contact_information(account_id):
    account_client = boto3.client('account')
    response = account_client.put_contact_information(
      AccountId=account_id,
      ContactInformation={
        'FullName': '',
        'AddressLine1': '',
        'City': '',
        'CountryCode': '',
        'CompanyName': '',
        'FullName': '',
        'PhoneNumber': '',
        'PostalCode': '',
        'StateOrRegion': '',
        'WebsiteUrl': '',
      }
  )

    if response['ResponseMetadata']['HTTPStatusCode'] == 200:
        print('Successfully updated contact information for account ID: {}'.format(account_id))
    else:
        print('Failed to update contact information for account ID: {}'.format(account_id))


def get_account_name(account_id):
    """
    Retrieve the name of the AWS account.

    :param account_id: ID of the AWS account
    :return: Name of the AWS account
    """
    client = boto3.client('organizations')
    response = client.describe_account(AccountId=account_id)
    return response['Account']['Name']

def main():
    """Loops through all accounts in an AWS organization and updates primary account information."""

    organizations_client = boto3.client('organizations')
    response = organizations_client.list_accounts()

    for account in response['Accounts']:
        account_id = account['Id']
        # Skip/Exclude the master account(s)
        if account_id != '222222222222' and account_id != '111111111111':
            put_contact_information(account_id)

if __name__ == '__main__':
    main()

Quiverstone: The Single Pane of Glass for AWS Multi-Account Chaos

Ross Wickman — Mon, 12 Jan 2026 17:00:46 +0000

Managing a single AWS account is a full-time job. Managing five? It’s a challenge. Managing five hundred? It’s a nightmare of lost visibility, security "hoop-jumping," and inventory sprawl.

For the last year, we’ve been building the solution to this exact problem under the name Quiverstone.

Introducing Quiverstone: The multi-account management platform built specifically for MSPs, Cloud Consultants, and Enterprises who need to own their AWS fleet, not just survive it.

Why we built Quiverstone (The "Why Now?")

Standard AWS management tools assume you are managing one giant organization. But for MSPs and consultants, the reality is different. You have dozens of customers, hundreds of stand-alone accounts, and a constant need to jump between them without losing your security posture.

The status quo involves brittle spreadsheets, manual IAM role switching, and "shadow" accounts that go unnoticed until the bill arrives. We knew there had to be a better way to automate the high-stakes work that runs your business.

The "Big Three" Features

Quiverstone isn't just a dashboard; it’s an operational engine for your cloud practice.

1. Unified Inventory Discovery (The "Stop Searching" Feature)

Stop manually tracking accounts in Excel. Quiverstone automatically discovers and imports your AWS Organization member accounts while allowing you to manage stand-alone accounts in the same view.

The Benefit: Total visibility of every asset across your entire customer base in seconds.
The Difference: Other tools still require individual authentication in the AWS Organization owners Identity Provider or User Pool (e.g. MS Entra, Google Workspace, Okta),but not Quiverstone.
The Edge: One or One-Hundred Organizations, Quiverstone has you covered. Tiered support teams come back after a long weekend and need to support an account that your customer spawned over the weekend? Not a problem. Quiverstone has already inventoried it and your support team has the access they need.

2. Secure, Chained Role Assumption

Accessing accounts shouldn't feel like a security risk. We’ve built a fine-grained Access Control system that supports direct and chained IAM role assumption with External ID support.

The Benefit: One-click access to any account with a full CloudTrail audit trail, ensuring zero permanent credentials.

Does that sound scary? This is the same process all enterprise tools use. AWS Partners meet all requirements by requiring and IAM role with trust and External ID. Quiverstone is no different.

3. Multi-Tenant Organization Management

Designed for the "Professional Services" reality. Categorize your accounts by Customer, Organization, or Project. Whether it's Resale, Managed Services, or a quick Professional Services engagement, Quiverstone organizes your workflow.

The Benefit: Manage 1,000+ accounts with the same effort it takes to manage one.

Quiverstone: Why the name?

Everyone has the tools they created and use to manage their environments. These tools are Arrows in your team’s quiver. When a team member leaves, they often take their quiver with them and are seldom left behind for use by others in the consulting firm. Quiverstone represents something different: The Foundation. Just as a quiver holds the tools for a precision strike, the ‘stone’ is the foundation on which your AWS accounts, access, and data reside, so you can execute your AWS cloud management strategy with precision. We are 100% focused on being the SaaS platform that empowers your services, not competing with them.

The Roadmap & Our Vision

We are just getting started. Our vision is to become the definitive operating system for AWS power users. In the coming months, we are shipping multiple features that consolidate many of the public tools and resources provided by community members and the AWS teams into a single dashboard for quick access.

We’re building this in the open, and as a SaaS founder, I want your feedback to shape our next sprint.

A Special Founder's Offer

To celebrate the general availability of Quiverstone and our official SaaS launch in 2026, we’re offering a "Founder’s Deal" for our first 50 users.

Try Quiverstone for Free

Found a bug? Have a feature request? Drop a comment below—our engineering team is standing by to answer everything!

Scaling Performance Testing: Leveraging the AWS Distributed Load Testing Solution

Ross Wickman — Fri, 09 Jan 2026 21:21:00 +0000

In modern cloud architecture, "it works on my machine" isn't enough. When your application scales to millions of users across the globe, you need a testing strategy that reflects that reality. However, building a home-grown distributed testing infrastructure can be a massive engineering undertaking.

This is where the AWS Distributed Load Testing (DLT) solution comes in. In this post, we’ll explore what this solution offers, how it works, and how you can use it to simulate massive, multi-region traffic.

What is the AWS Distributed Load Testing Solution?

The AWS Distributed Load Testing solution is an AWS Solutions Implementation that automates the testing of software applications at scale and at load. Instead of managing a fleet of EC2 instances manually, this solution provides a pre-built framework to:

Generate Massive Scale: Simulate thousands of concurrent users.
Go Global: Launch test runners across multiple AWS regions to simulate realistic geographic traffic patterns.
Visualize Results: A built-in web console provides real-time metrics and historical data for every test run.

Core Architecture

The solution leverages Amazon ECS on AWS Fargate to run containerized test engines. This serverless approach means you only pay for the compute while the test is running. It supports popular open-source engines like JMeter, Locust, and K6.

Why Use It?

While many developers use simple tools for local benchmarking, the DLT solution is designed for high-concurrency scenarios that local machines cannot handle.

CloudFront & CDN Validation: By downloading large assets through various edge locations, you can validate CDN caching behavior and performance under stress.
Multi-Region Simulation: Test how your application handles latency and traffic spikes originating from different parts of the world simultaneously.
Predictive Cost Modeling: By generating realistic egress patterns, teams can better estimate data transfer costs before a major product launch.

Automation & Scheduling

One of the most powerful features of the DLT solution is the ability to move beyond manual execution. Performance testing is most effective when it is consistent and predictable.

Scheduled and Recurring Tests

You don't need to be at your desk to trigger a massive load test. The DLT console allows you to:

Run Once at a Specific Time: Schedule a test to run during off-peak hours or ahead of a planned maintenance window.
Recurring Schedules: Set up tests to run daily, weekly, or monthly to ensure no performance regressions have been introduced into your codebase.
CRON Job Integration: For advanced automation, the solution supports standard CRON expressions. This is ideal for engineering teams who want to align performance tests with specific deployment cycles or complex internal schedules.

How to Get Started

1. Deployment

The quickest way to get started is using the AWS CloudFormation templates provided in the official documentation.

2. Setting Up Your First Test

Once the stack is deployed, you will receive a URL for your private DLT Web Console.

Authenticate: Log in using the credentials created during the CloudFormation deployment.
Define the Endpoint: Enter the target URL (e.g., https://dlt1.csne.io/test-5gb.bin).
Configure Parameters: * Tasks: Number of Fargate containers (e.g., 100).
- Concurrency: Sessions per task (e.g., 5).
- Regions: Select the regions for load generation (e.g., us-east-1, us-east-2, us-west-1, us-west-2).
Set the Schedule: Choose "Run Now" or configure your Schedule/CRON settings for later execution.
Monitor Results: Track response times, success rates, and bandwidth in real-time or review the reports later.

Example Configuration Spotlight

To simulate realistic traffic, your configuration should look similar to the following:

Parameter	Configuration
Method	GET
Endpoint	`https://cdn.example.com/assets/video-part-1.bin`
Tasks	100
Concurrency	5
Geographic Scope	All 4 US Regions

Power at Scale: Real-World Performance

When scaled appropriately, the AWS DLT solution can generate massive throughput. In a recent baseline test using 100 tasks with 5 concurrent sessions distributed across all 4 US regions, we observed the solution's true potential.

By targeting a series of 5GB dummy files across multiple CloudFront distributions, the test infrastructure successfully pushed the boundaries of standard egress:

Peak Performance: CloudFront egress reached 50+ TB per 5-minute interval.
Sustained Load: The system maintained a steady-state egress of 20TB for the duration of the test.

This level of performance is critical for validating that your CDN configurations, origin shielding, and bandwidth quotas are sufficient for high-traffic events like global product launches or major media broadcasts.

Example Use Cases

Scenario A: High-Bandwidth Egress Testing

If your product serves large binaries (e.g., video chunks or software installers), you can use DLT to pull files of varying sizes (1GB to 5GB) through Amazon CloudFront. This allows you to measure the impact of sustained high-bandwidth egress on your infrastructure and budget.

Scenario B: API Stress Testing with JMeter/K6

Beyond simple GET requests, you can upload custom scripts. For instance:

JMeter: Upload a .jmx file to simulate complex user flows involving authentication and database writes.
K6: Use JavaScript-based scripts to define "Sustained Load" vs. "Spike" scenarios, allowing you to see how your auto-scaling groups react to sudden surges.

Summary

The AWS Distributed Load Testing solution takes the heavy lifting out of performance engineering. By moving from local "one-off" tests to a distributed, automated, and scheduled framework, you ensure that your application is ready for the real world—before the real world arrives.

For more detailed configuration templates and advanced deployment options, check out the AWS DLT Documentation.

Streamlining Security Management

Ross Wickman — Thu, 18 Jan 2024 17:18:22 +0000

AWS re:Invent 2023 had very little for announcements around Management and Governance. My core competency and concern.

One announcement; however, struck deep in solving an issue I've faced since taking on my recent role managing the security and governance of four greenfield AWS Organizations.

That announcement... New from AWS: You can now customize security controls in AWS Security Hub

From the title, it doesn't sound like much. This could already be done on a per-account basis. My challenge was setting the standard for developers and operators to live by within many accounts across multiple organizations.

I set the requirements of:

All critical and high controls are required
- of those outside of the control of the individual account owners and team, I would disable them with a comment
- disabling of controls, to the developers meant, "I got this one", or that it is inherited
Mediums owned by developers unless could be inherited
Lows mostly taken care of by globally deployed management solutions and standards.

Up until re:Invent 2023, the solution to do that very thing required a custom solution not native to the AWS Security Hub service itself. This seemed counterintuitive. An AWS Service that could already be delegated for Organizational level management seemed like it should already do this, but it took a few years for the functionality to be part of the service itself. I'm sure all of this is much more complicated on the back end than it seems and for that, I give a lot of grace.

In the interim for deploying four new AWS Organizations, the solution was to deploy an AWS-Samples solution which allowed cross-account control disabling. I wrote about this solution in an earlier issue of my AWS Management & Governance Newsletter - Unlimited Leave. You can read that issue here:

https://newsletter.unlimitedleave.com/p/security-hub-integrates-with-control-tower

The implementation of that solution required me to build a personally managed project with the AWS Samples solution(s) as a submodule. This wasn't a big deal but it did add some additional complexity. Some of which are self-inflicted due to having multiple AWS Organizations.

A topic I cover here.

Regardless of the complexities, this sample solution got the job done. Albeit not very intuitively.

Challenges of the Old Process

While effective, this approach had several drawbacks:

Complex Setup and Maintenance: Implementing the custom solution required considerable effort, including setting up Lambda and Step functions and ensuring they worked seamlessly across all accounts.
Additional Documentation Needed: Deploying a solution not native to the AWS Service requires faith in the referenced solution itself, how well it is documented, as well as custom documentation for how the solution is deployed for your use case.
Manual Updates and Scalability Issues: Any changes in Security Hub controls necessitated manual updates to the script. This was not only time-consuming but also prone to errors, especially in large, dynamic environments. The consolidation of Control IDs across standards had a major impact on this.
Limited Visibility and Control: The custom solution provided basic functionality but lacked the sophistication to offer granular control or insights into security posture across accounts.
Core functionality not obvious: even though the roll-up of controls for the whole organization could be in the central delegated account, and across regions, the solution needed to be deployed to every active region to work. Or, worse yet, at the time Security Hub and AWS Config, didn't honor the fact that "Global" resources and controls, were only enabled in the "Home Region" of your Control Tower Deployment. This led to much back and forth with the SH team and ultimately temporarily turning off SH in secondary regions.

The Game-Changer

The landscape of cloud security management took a significant turn with the announcement at AWS re:Invent 2023. AWS unveiled enhanced central configuration capabilities in Security Hub, marking a pivotal advancement in how security controls are managed.

Key Features of the Updated Security Hub

Centralized Control Management: Users can now enable or disable specific security standards and controls across all accounts from a single, central location.
Automated and Scalable: The new features automate what was previously a manual and error-prone process, facilitating scalability and consistency across large AWS environments.
Enhanced Visibility and Compliance: With centralized control, it's easier to maintain a consistent security posture and compliance status, as changes are propagated automatically across all accounts.

Comparing Old and New: A Leap in Efficiency

The contrast between the old and new methods of managing security controls in AWS is stark:

Ease of Use: The updated Security Hub eliminates the need for custom scripts and manual intervention, offering a straightforward, user-friendly interface.
Time and Resource Efficiency: What used to take hours of scripting and testing can now be accomplished in minutes with a few clicks.
Consistency and Reliability: The central configuration feature ensures consistent application of security controls, reducing the risk of misconfigurations and compliance issues.
Most importantly: True AWS Support. The second you deploy an AWS-Samples solution, you're pretty much on your own. While the community on some of these projects is strong, the gist of the sample solutions seems to be side (or passion) projects asked of team members by AWS leadership which help with performance reviews and level increases. Don't get me wrong, every one of these solutions or reference architectures is great, but if they aren't part of the core service, you never know how long support is going to last.

Core Functionality

The evolution of AWS Security Hub, especially with the latest updates announced at AWS re:Invent 2023, represents a significant stride in cloud security management. The shift from relying on custom solutions to leveraging integrated, centralized features not only simplifies security control management but also enhances overall security and compliance.

Anytime a custom implementation is required, specifically regarding security, you increase the risk of security issues. The solution may stop working or not be reliable (in the case of Control IDs being consolidated). The implementation of security protocols many times can be a vulnerability all by itself. This core service feature helps minimize that vulnerability.

If you enjoy learning about AWS Cloud Management and Governance, please follow along by subscribing to my newsletter.

Until next time, Cloud Securely.

- Ross

Why You Should Use AWS Control Tower

Ross Wickman — Thu, 23 Feb 2023 21:47:57 +0000

I have an email newsletter called Unlimited Leave where I write about AWS Management and Governance at scale.

The idea behind the name is to build systems, processes, and policies in AWS that enable the autonomy of developers, reporting for InfoSec, and scalability of work. Things that allow you to take carefree time off and reduce work on your nights and weekends.

Among all of the AWS Announcements, I primarily focus on the ones that come off of the Cloud Operations and Migrations Blog.

My shtick is all about how to manage, govern, and secure many AWS accounts at scale across multiple AWS Organizations.

If you are interested, you can subscribe here: https://newsletter.unlimitedleave.com/subscribe.

A couple of weeks ago I received a question from a reader regarding the use of the Control Tower. With the reader's permission, here is his question:

Hi,

Thank you for your newsletter, I have been finding it useful.

I see you are asking us dear readers a question, but instead, I'd like to ask you a question. I have been looking at implementing Control Tower off and on again, but I've never really found it compelling enough to put it in the good 'ole backlog. Maybe I am missing something here. My organization is using AWS SSO with an external identity provider and also at this point has a pretty static number of AWS accounts. We are also using Terraform to deploy a good amount of our infra. Could you say what benefit Control Tower brings compared to, say, just putting the controls in Terraform? Are there controls that there are no APIs? Maybe this would be covered in your Terraform module, but it's definitely something I'm interested in learning about.

I responded directly in the email but promised a more detailed write-up of Control Tower and provide a compelling reason why an organization should use it (or not). Here it is.

Why use Control Tower?

A very solid question and a very polarizing one in certain circles in fact. I have, and still do share the sentiment on Is Control Tower right for me/us.

The quickest response to that question is - it depends.

The benefit derived depends squarely on what your organization is hosting and who/what you are beholden to when it comes to the data you possess.

The question isnt necessarily about Controls but more about Governance. Who has a thumb on you, and how do you keep a thumb on developers or teams in your AWS Organization(s)?

In other words, do you report to or care about any multi-letter agency's opinion of your environment?

Do industry standards and security frameworks matter when it comes to the workloads and data you are hosting?

Do any of these acronyms cause your hair to stand on end GDPR, PCI, SOC, HIPPA, FedRAMP, and so on?

Governance is about allowing and enabling when total control isnt the desired outcome or is impractical. Configuring your AWS Organization with Control Tower helps accomplish this.

So what is AWS Control Tower

It is hard to argue that AWS Control Tower (CT) is a traditional AWS service. CT as a service isnt a service at all. It is a Click-Next style wizard for setting up accounts using a Vending Machine Model.

The service is just an administrative dashboard that incorporates multiple AWS Management and Security services into a single Console and has very little API/CLI functionality itself.

Think of an Air Traffic Control Tower orchestrating everything going on down on the airfield (Landing Zone), taxiway, and all of their responsible airspace. The need to organize, secure, deploy, track, authenticate, and comply.

With CT, the backend is a series of API calls to all of those services:

Organizations
Service Catalog
CloudFormation StackSets
Lambda
Config
Security Hub
S3
Identity Center (formerly AWS SSO)

This is important to realize when deciding if you should start with, or migrate to CT. Your organization will not gain any functionality that you couldn't implement directly with any of the services listed above or that you already have implemented.

Marketing Speak (as written by AI)

Managing multiple AWS accounts can be a daunting task, especially when it comes to maintaining consistent security and compliance standards. AWS Control Tower is a powerful solution that helps you govern your multi-account AWS environment, providing a central place to set up and manage your accounts, resources, and policies. Here Ill discuss the top three reasons why you should consider using AWS Control Tower.

Simplified Account Management

AWS Control Tower simplifies the management of multiple AWS accounts by providing a single place to set up and manage accounts, users, and policies. With AWS Control Tower, you can quickly create new accounts with pre-configured resources and permissions, allowing you to deploy new workloads faster. Additionally, AWS Control Tower provides a consistent account structure across your organization, ensuring that all accounts are set up with the same security and compliance standards.

Enhanced Security and Compliance

AWS Control Tower helps organizations achieve better security and compliance by providing a set of pre-configured guardrails that automatically enforce security best practices and compliance standards. These guardrails include policies for identity and access management, logging and monitoring, networking, and data protection. With AWS Control Tower, you can rest assured that your AWS environment is configured to meet best practices and compliance standards, reducing the risk of security breaches and compliance violations.

Cost Optimization

AWS Control Tower provides cost optimization features that help you manage your AWS environment more efficiently. For example, AWS Control Tower provides a pre-configured AWS Landing Zone that includes a set of pre-approved AWS services that are optimized for cost and performance. You can use this Landing Zone to quickly deploy new workloads, ensuring that they are set up with the right resources and configurations to meet your performance and cost requirements. Additionally, AWS Control Tower provides cost optimization guardrails that automatically monitor and optimize your AWS environment to reduce costs and improve performance.

What is life like outside of the CT brochure page? (as written by me)

Im pretty opinionated when it comes to how teams should manage their AWS Organizations. First, one AWS Organization is never enough. Two is the minimum for reasons Ive laid out in My Case for Multi-Org.

Beyond that argument, an AWS Organization should be your boundary. Even if not all accounts in the org are considered in scope. Think of the structure as a vein diagram.

How to accomplish this is by using Control Tower.

Advantages of Control Tower

The advantages of CT are many. One is that CT in a brand-new organization will set up central logging of all accounts to a protected location just by enabling CT. All new accounts will get a CloudTrail Trail enabled and offloaded to an S3 bucket in a separate account that can be protected.

In addition, CT will configure AWS Config to monitor core best practice configuration and give you a dashboard to enable preventative and detective controls across all accounts in an organization. Beyond that, CT will protect those configurations so even Administrators at the individual account level cannot modify them.

These things of course can all be done without Control Tower:

the delegation of Security Services
enabling Config and Central Aggregator
implementation of SCPs
New Account Creation
Bucketing (OUs) of account types

Some of these still need to be done with API calls or in the Console but for the most part, the major items are done with Control Tower.

You could do all of this on your own, but youd have to build that all yourself, and it's not supported except by you. AWS versions all of this in what is called the Landing Zone version.

Is there a downside?

There is a downside to using Control Tower natively.

That is, you have to understand what the service is doing when engaging with all the other backend services. Meaning, if you want to keep all of your Infrastructure and Policy as Code, you shouldnt just be using the Control Tower Console.

You would use the Control Tower API for managing Controls / SCPs. That is if you dont have any custom SCPs to deploy.

You would use the Service Catalog API for managing your accounts via CT Account Factory.

By using these things directly, your management Infrastructure and policies are not stored anywhere unless you document the API calls and controls youve enabled with them.

Are there tools?

There are tools to help you manage these items that require CT to be deployed first. My two favorites are:

What if you are like the reader with the question?

There is a point where you might have so many accounts and a process already, that you just write off ever using Control Tower. That is ok.

The best case is you have a small enough footprint where you can start with a Greenfield Organization and migrate accounts into the new Organization.

If you are sitting at a static number of accounts, you would have to decide as an organization if the juice is worth the squeeze for enrolling all of your existing accounts into a CT Landing Zone, only doing net-new accounts, or not bothering altogether.

When you (or the reader with the question) say just putting the controls in Terraform that is a whole lot different than a top-down governance model that leverages CT and Management account policies.

This places the controls external to the team's visibility and workload.

Sure, you can deploy IAM policies and permissions boundaries at the account level, but then that opens the door for deviation from the standard and configuration of other like accounts. CT can help deploy account types with compliance applied at the top.

Unless you are deploying SCPs with your Terraform to the Management Account (or by using a delegated account) you are managing all of your controls on a per-account configuration. Which can get messy and prone to error, inadvertent privilege escalation, or even self-induced DoS.

Going forward

AWS Control Tower is indeed a powerful solution that helps organizations govern their multi-account AWS environment with greater ease, security, and cost efficiency.

Side note on Cost efficiency

If you don't properly configure your Landing Zone and the Default VPC settings, you are spending way more money by default because you are probably deploying a new VPC per account, and in every region placed under governance. In each of those regions, in every new account created by the account factory, NAT Gateways are getting deployed and sitting Idle.

Be sure to only enable regions you will have workloads and opt out of deploying the default VPC.

Control Tower costs about $50 a month by default before any additional configuration.

Just deploy Control Tower

With simplified account management, enhanced security and compliance, and cost optimization features, AWS Control Tower provides a comprehensive solution for managing your AWS environment.

If you're looking to gain better visibility and control over your AWS accounts, AWS Control Tower is worth considering. Your workload and future compliance woes will benefit from it.

Hands down you will benefit from Control Tower if you ever have to worry about a 3rd party audit or any of the compliance frameworks regarding data storage and transmission.

Without actually seeing any of this in practice, it is really hard to see where you could benefit.

Where to get help

To help assist with this, Im officially creating a course that will walk through how to configure a brand-new Greenfield Organization with Control Tower.

For more information on gaining early access to the course as well as getting more details about AWS Management and Governance, please subscribe to Unlimited Leave.

If you want to research more AWS-specific literature about why and how to deploy AWS Control Tower, take a look at the Cloud Foundation Team Guidance for Establishing an Initial Foundation using control Tower AWS. Ill be talking more about this in the newsletter in the future.

If you have any questions or need direct assistance with your next AWS Control Tower deployment, please contact me.

The case for AWS Multi-Org

Ross Wickman — Tue, 22 Nov 2022 15:15:00 +0000

Be in any game long enough and you start to notice patterns. The pendulum begins to swing in different or new directions altogether. This is especially true in technology. I haven’t been in AWS all that long and I already can see areas where things begin to drift.

This time - Multi Organization configurations

It is hard to explain the feeling you get when you just know something is about to happen or is already in the works. I could be way off on this. To those already struggling to maintain their AWS Architectures, I may look like a lunatic. Instead, I hope I lay out a new idea and ways to consider building out cloud architectures.

I also want to get this thought down and public prior to AWS re:Invent 2022. Just in case things really start moving in this direction, I can have a little proof that I knew what I was talking about.

I have no inside information. This is all based on assumptions and derived from my opinions about some of the most recent service and service feature announcements.

If you are paying attention and care enough about a certain area you begin to see patterns emerge. For me, that is AWS management and governance. My career in AWS has revolved almost entirely around the management of environments. So anytime a new service announcement relates to managing AWS Accounts as a whole, I take notice.

What I’m seeing

The delegation of AWS services to a central account has proliferated. This has a tremendous impact on securing critical AWS accounts.

Protecting the core (Management) account of the organization from inadvertent access or access by excessive individuals is critical. The management account really is the key to the kingdom. Getting administrators out of that account is a huge step.

The next step isn’t a big leap. Architects can already delegate main services administered for any entire AWS Organization to a single account. In the case of CloudFormaiton StackSets, you can delegate up to 5 accounts. This process leverages the AWS Organization and service role trusts. I don’t know all the deep interworking of those service trusts.

I can make an educated guess, that it wouldn’t take too much effort to elevate that trust one more degree higher to accounts outside of an Organization to accounts that are members of a different AWS Organization.

An Organization of delegated administrator accounts

Imagine a new AWS Organization that isn’t configured like your current organizations with workloads, Control Tower, Service Control Policies, and shared networking.

Image this organization having central control across multiple other organizations for the following:

Access Control (i.e Identity Center, formerly SSO)
Service Catalog
CloudFormation StackSets
Security Hub
Inspector
SSM
Billing
etc.

Just as you can invite accounts into an AWS Organization, at some point I believe AWS will enable the ability to invite delegated service administrator accounts to manage other organizations.

You may or may not have to first delegate to a central account INSIDE of an organization. I think you could and should. Administrators and security teams can have a view of their landscape from inside the individual organization. Global admins could see everything from the Org of Orgs.

Break Glass Access Control

AWS should have an external service that can be protected with MFA, like AWS QuickSight or Amplify, for example, that has its own external login portals outside of the AWS Console. This service could be used to manage break glass access to any account through the use of assumable roles.

Until there is an external service like this, there isn’t a great way to set up secure emergency access to accounts in an organization.

The most basic configuration of a multi-org strategy you could start with today is setting up a new organization just for external break glass access.

This model of access would be an Identity Center configuration in a region separate from the deployed regions of your other organization(s). In some cases, but not all, this could prevent access issues when there are problems in the region where you have your Identity Center configuration deployed. Since almost all authentication services run through us-east-1 anyway, your results on this one may vary. At the very least you have a separate AWS SSO/Identity Center solution in the event something bad happens with your External IDP. For instance Azure AD or Okta.

How do admins gain access to any accounts if your External IDP is down or broken?

Out of the box, this is likely the OrganizationAccountAccessRole from your Management account. But you shouldn't use your Management account for break glass access. This likely means you have root user access or IAM users for your Management account. You shouldn't use the Management account for much of anything really.

The best option is, in your existing workload Organization(s) you have a StackSet that deploys an assumable role to all accounts. This role is then accessed from an account or accounts in your new organization used just for break glass access.

You can have a specific permission set dedicated to only being able to assume a specific role in all, or even some, of your critical accounts across your other organizations.

Break glass access from a special account isn’t a novel idea. We’ve all needed it at some point. While this post about having an AWS account just for getting into other AWS accounts doesn’t meet exactly what I am talking about, it gives you the right idea.

Workload Access control

Think about the number of people (developers, administrators, security professionals, newly hired, novices, etc.) that will be accessing your single AWS Organization. Now think about how easy it is for someone to be improperly added to the wrong Identity Center Group or individually added to a permission set on a very critical workload account.

If you are using SAML or some other solution for access, the problems compound with the more accounts you have. There aren't many solutions that allow the nesting of groups of users that port nicely into access within AWS.

Having multiple organizations allows you to have multiple Identity Center configurations and therefore multiple External IDP configurations. You can have dedicated access to your organization where users simply don’t have access to a specific organization because their workload accounts don’t exist there.

Billing & Cost Allocation

In any organization, there are two types of development.

There is the development directly related to service and/or feature enhancements of production products. This type of work supports build releases and bug fixes of revenue-generating solutions.

Then there is the R&D or dinking around your developer teams conduct to test and learn a new feature or service. Here services may get left on or accidentally introduce vulnerabilities into your sanctioned product development accounts.

A solution…

Give all of your developers an account in a Sandbox/R&D Organization. Lock it down to one region. Give every account a monthly budget and alarm and let your developers tinker in there. This way R&D cost is separate (and consolidated) inherently from development that is tied directly to your production product or service. No additional tags or access restrictions are required. Easily terminate accounts or nuke services in R&D accounts.

All of which can be automated nightly as well. Automated without the concern of hitting anything in production or critical development.

You might also consider creating an organization just for suspending accounts or transferring ephemeral accounts.

Product, Service, or Department Segmentation

With access control sort of figured out, suddenly managing multiple organizations doesn't seem so icky. Bucketing your accounts into different AWS Organizations instead of just Organizational Units has a lot of benefits as you scale out your architecture.

I don't personally know your organization's departments or structures. However, there are many places where this might work for you.

One example is anywhere your company might sell, cut ties, or have services that can operate on their own without global governance or management being tied to another group of services.

Do you have a business unit that operates self-sufficiently and has its own cost structure? Even if you need to have shared networking you can have these solutions in their own organizations. With a dedicated invoice, credits, and access controls.

Could a whole arm of your organization undergo acquisition by another company? Maybe that should be in its own AWS Organization.

Are you in the business of acquiring part of or other businesses entirely? How can you keep those accounts in a different organization indefinitely or at least until you have a good consolidation strategy?

The answer to all of these could very well be unique AWS Organizations for each situation.

Compliance

Security Hub checks, workload security, and compliance frameworks can be managed pretty easily across a single organization. But what happens when you have multiple subsets of accounts that all require an exemption to some framework or benchmark standard?

Wouldn't it be nice to look at a single pane of glass that is all in one compliance context?

If you have dozens or hundreds of accounts that fall under the same standard(s), anomalies stand out. When all of your Security Hub checks or Inspector results line up across all accounts of the same compliance standard, you have a true understanding of your security posture.

When you have to account for the small number of accounts that are allowed to have EC2 instances directly accessible from the internet, those results distract you and can catch your eye every time you are reviewing your dashboard.

This makes you more susceptible to missing a true configuration issue when it pops up. Since you are already expecting to see red.

Where would this setup be helpful?

Don’t get me wrong. There are caveats and nuances to all of this.

Especially concerning billing. Any credits you may get from AWS, things like Reserved Instances (RIs), Savings Plans (SPs), and Enterprise Support are all factors. I have solutions for some of that today. For instance, creating accounts just for the purchase of RIs or SPs that can be moved from one organization to another if needed.

I’m certain this will be taken care of by AWS as this all evolves. Again. I could be completely wrong about all of it.

A relatable example

I get it. This all seems like it could be considerably more complicated than it is worth. You are not wrong. The initial setup takes a little bit of extra work.

Once the initial foundation is laid and automation is in place, this all just makes a lot more sense. This solution may resonate with some of you.

The solution I have recently been deploying is as follows:

A management organization
- primed and ready for the central delegation of services I mentioned above
- central S3 bucket(s) with permissions for all other organizations to access global packages and CloudFormation Templates
- Identity Center with dedicated permissions for assuming all other accounts across all other organizations
Networking - something I will get into deeper in a future post but think…
- Direct connect terminated in networking account
- Transit gateway connections to a Shared VPC in a networking account in each organization
- VPC shared subnets to all accounts within the organization using RAM in the Organization
automation account
- a single entry point with an IAM trust to automate across N other organizations
- this prevents unique credentials in automation for each organization.
- a single set of credentials for one account that can run automatic assumptions across all other organizations.
Workload Organization 1
- a specific product/service
- a very specific team of developers that needs access
Workload organization N
- any number of additional organizations
- no cross-team access is required - different dedicated developers
Back office services, administration, and automation
- Finance services
- HR
- etc.

All of this is being set up with automation in place using GitHub Actions or GitLab-CI, I deploy identical resources to all accounts across all organizations. I can do so with the same number of operations as using a single Organization. It's just a small tweak to the automation.

At a larger scale

I have over a decade of experience in the federal system. I am going to use that to expand this entire idea to a much larger scale. This way you can see where this might apply anywhere from a smaller local business with a heavy technical workload to a much larger enterprise scale.

The government has many ‘Departments’. Any time you see a department, that means there are smaller bureaus or branches within the department.

For instance, the Department of Defence (DOD) has the Army, Marines, Air Force, Navy, Coast Guard, and Space Force.

The Department of Interior (DOI) is a whole hodgepodge of bureaus that honestly didn’t fit very well anywhere else.

The model of multi-organization I discuss here would work great for this style of hierarchy. Anywhere central governance and compliance all matter and looks a specific way. Everything rolls down from the top including funding, but yet each individual branch or sub-division operates very differently.

At the top, there is global administration and compliance. Each with their core services, monitoring, logging, etc. delegated and transferred to the mothership AWS Organization.

Inside each organization are the unique access requirements for each sub-division. Since they all have their own Enterprise Directory systems anyway, they can manage their own Identity Centers. The teams of people doing the work within their organization manage the compliance up and down the chain.

This is probably the most extreme of the use cases but it helps bookend the unlimited options in between.

Wrapping up

Finally, the driving factor behind all of these options is enablement.

Why did I even start to think this way in the first place? I asked myself these questions:

How can you get the biggest bang for your buck from a management standpoint where you don't have to constantly be correcting mismanaged resources, or mistagged resources?

What is the least common denominator where a set of accounts belong together?

Where can I reduce risk and increase autonomy by letting the platform do my job for me?

Breaking out account types and/or products into their own Organizations seemed like the most logical solution. However, I have over the last to years focused on deploying many new AWS Organizations. All with Control Tower, Customizations, and more recently Account Factory for Terraform.

What are your thoughts on all of this?

Where do you see gaps or concerns in my architecture?

I would love to hear from you.

If you enjoyed this post or would like to learn more about how I look at AWS architectures, Management, and Governance please subscribe to my weekly newsletter - https://unlimitedleave.com.

Use AWS CloudFront Functions for URI Rewrites

Ross Wickman — Thu, 06 May 2021 19:27:21 +0000

With just over one week to move a customer application from one data center into our managed service environment, I was provided with a new (to me) requirement. The requirement was to configure URI redirects for their web application. This didn't seem like a huge concern at the immediate request. I figured I'd just put a couple of listener rules on the Application Load Balancer (ALB) and call it good doing what all DevOps engineers do...

That was before I received the spreadsheet with almost 60 (known) redirects. More potentially to come.

What to do?

I didn't want to make an already complicated solution more complicated.

I knew I had limits to the number of rules allowed in Load Balancer Listeners.

I wasn't interested in adding or modifying any Lambda@Edge functions managing my Content Security Policies.

Fortunately for me, right at the time of receiving this new requirement, AWS Announced CloudFront Functions.

What is/are CloudFront Functions?

...a new serverless edge compute capability. You can use this new CloudFront feature to run JavaScript functions across 225+ CloudFront edge locations in 90 cities across 47 countries.... ( read the full announcement here ).

How could CloudFront Functions help me?

Aside from the rule limits imposed on me by ALB Listeners such as:

100 total rules per ALB
5 conditions per rule
5 wildcards per rule
5 weighted target groups per rule

I would have had to use wildcards to make some of the rewrites work. You can use your imagination and the following example to see how this may not produce a favorable outcome.

Use case example

Imagine a website like AllRecipies wanting to redirect certain recipes to new or improved names.

Consider Homemade Mac and Cheese served here:
https://www.allrecipes.com/recipe/11679/homemade-mac-and-cheese/

Let's say for example sake the 11679 in that URI is a catalog of recipes. AllRecipies may want to redirect anything that isn't a successful hit to a different catalog of recipes. So they use a wildcard like:
https://www.allrecipes.com/recipe/11679/* to push all requests to https://www.allrecipies.com/summer21/

What happens is now https://www.allrecipes.com/recipe/11679/homemade-MOUSE-and-cheese/ successfully redirects with a 200 response instead of going to a 404 Page Not Found.

NOTE: oddly enough, AllRecipes is actually doing some magic to forward that URL to the correct recipe.

Using CloudFront Functions for URI Rewrites

Being under the gun to come up with a solution, I immediately took a stab at CloudFront Functions. I'm honestly not sure how I got this to work after the first attempt, so I'm interested in getting anyone's feedback on this implementation.

To accommodate an unknown amount of future requests for rewrites, I tweaked and implemented the example provided in the CloudFront Functions Introduction Blog.

My implementation

function handler(event) {
  var request = event.request;
  var rewrites = [
    ['/summer21','/recipies?year=2021&season=summer'],
    ['/recipies/homemade-mouse-and-cheese/', '/recipies/homemade-mac-and-cheese/'],
    ['/recipies/camping/grilling', '/recipies?activities=camping&with=grill']
  ]

  for (var arrayIndex in rewrites){
    if (request.uri == rewrites[arrayIndex][0]) {
      var newUri = rewrites[arrayIndex][1];

      var response = {
        statusCode: 301,
        statusDescription: 'Permanently moved',
        headers: {
          "location": { "value": newUri }
        }
      }
      return response;
    }
  }
  return request;
}

Making the solution more robust

This CloudFront Function appears to be working as expected. This example allows you to redirect any URI pattern and forward it to a new path or include a search query for client or server-side functionality. All while using JavaScript at the Edge. Without a doubt, this could be modified to further accommodate query strings in the request.

I typically like to deploy my solutions via CloudFormation specifically because this is for a customer; however, at the time of this post and the function implementation, the CloudFormation Team has not released an update to the User Guide for creating this as a managed resource in CloudFormation. I am told it should be released soon. Once I have the information, I will try to get back here and provide a CloudFormation Template in an update.

Until then, if you are looking to try this out or implement CloudFront Functions in your own environment, I encourage you to checkout this blog post by a fellow AWS Community Builder.

Please do not hesitate to comment below how you decide to implement CloudFront Functions, this specific function, or better yet how I can make this specific function even better.

Automate AWS Resource Review and Deployments with GitHub Actions

Ross Wickman — Sat, 20 Feb 2021 14:22:58 +0000

Deploying AWS resources with GitHub Actions can help save you a lot of time and ensure you are deploying quality templates. Beyond quality, you can be sure you are keeping your Infrastructure-as-Code (IaC) in line with what is ACTUALLY deployed in your environments.

There is a time and a place to not use CICD to deploy AWS Resources, but most of the time you can run your pipelines or workflows to deliver your resources directly into your AWS Account knowing that what is deployed is exactly what is represented in your Master branch.

Don't need the 30,000ft. view? Jump right to the code here:

https://github.com/rosswickman/aws-automation-workflows

The Setup

You will need:

A Clone or Fork of the repo linked above
S3 Bucket to store CFN and TF templates. Possibly your TF state. (recommended but not required)
IAM User with permissions to deliver templates to S3
IAM User with permissions to deploy resources

NOTE: You could do this with a single IAM User but I recommend minimizing blast radius with least-privilege wherever possible. In the ideal production situation, you should configure a GitHub Actions Runner in your environment with the permissions required.

The Configuration

Deploy the three stacks in the requirements/ directory of the shared repo. The workflows are configured for region us-west-2, so deploy there.

Configure your repository secrets by getting the IAM Access Keys from AWS Secrets Manager in your account.

Create a 'feature' or 'update' branch in your project that you cloned from above or created on your own.

If you cloned/forked the project to start, you will see there is a cloudformation and a terraform directory in the project. Each directory has a respective template that deploys a security group in your configured AWS Account. We used a security group because they don't cost anything. You will need to update the VPC ID (from us-west-2) in each template for the resource to properly deploy in your account, but that is all you should need to do.

The Process

Before deploying anything to (specifically) production, the templates should be reviewed for security issues and quality. For this, we use tools like SuperLinter, CFN-NAG, CFN-Lint, and Checkov.

Each tool is represented by a workflow that completes a respective process. These workflows can be organized to your liking. However, as they are configured in this project, this process will run after you push any changes to your feature branch.

Once our templates check out and pass those basic checks, we can deliver the templates to an S3 bucket. This allows the templates to be referenced within the AWS Console or stored for backup but is not required.

To prepare the resources for delivery and deployment, you will need to create a Pull Request. The pull request will run the Terraform Plan and show you what will happen if the Terraform template is deployed.

Delivery happens upon Merge to the Master Branch. You can configure the deployment of resources to be automatic or by a manual trigger. In some cases, manual deployment makes a lot of sense. I'll let you be the judge of what should be a manual trigger.

The Outcome

Once your Pull Request was merged into the Master Branch, the templates were Delivered to the S3 bucket you configured using the instructions in the Git Project.
If you left the Workflows alone, the CloudFormation Deploy and Terraform Deploy required manual action to trigger.

Once triggered and completed successfully, you should see two new Security Groups attached to your VPC. One was deployed via the CloudFormation template and the other with Terraform.

There you have it. You have just configured a CICD/D workflow to deliver source-controlled AWS Resource Templates directly to your Account.

In future posts, I will show how to deploy AWS Organization StackSets as well as deploying StackSets to a Multi-Org Configuration sourced from a single repo.

I know there are many different ways to configure this process, this demo uses a single repository for both CFN and TF templates. That may not always be the desired solution. If you are similarly deploying resources, I'd love to know what you are doing differently or if you have any recommendations on what I should do differently.

Lessons learned around multi-account architecture

Ross Wickman — Mon, 30 Mar 2020 12:30:00 +0000

This is post 9 of 9 in a multi-part series (hosted here) discussing the advantages, pitfalls, deployment methodology, and management of a multi-cloud account architecture. For this series, we are focusing strictly on using AWS as the Cloud Service Provider (CSP) but the concepts discussed port well to any provider or even to on-premise operations.

Wear a helmet

No matter how prepared you are to start this process, you’re going to take some lumps along the way. No amount of technical skill or understanding of the provider services will prevent you from going deep in a direction only to complete it just before the release of a service enhancement that may have done all of the work for you. Fortunately, service enhancements or additional features that might be responsible for something like this happening have significantly slowed down as the providers mature. This is why we stated earlier on that it is important where possible to use the provider recommended solution over a different solution. Wait if you can. It is likely something on the roadmap.

There will be times when you hit an un- (or not so well) documented limitation that will require your organization to get creative to meet requirements. This is almost completely unavoidable. The important thing is to stay flexible, design with the future in mind, document as described in a previous post, and automate as much as possible.

Think of this entire process with regard to software development. The deployment model for your architecture is essentially backed by code. Being ‘agile’ and leveraging brief release cycles and make adjustments over time. Rather you’ve landed in a single cloud provider or are spread out in hybrid-cloud, think about the ‘software development lifecycle‘ and treat your architecture the same way. Every few years or even months there will come a time to re-evaluate and re-deploy services leveraging the latest and greatest services and standards based on current industry best practice.

Start small

Consider compartmentalizing some of the most basic functionality. It is possible to de-couple too much creating more work for yourself, but it is much easier to add in than remove. It is important to iterate slowly over time. Or if you need to move a little bit faster, include many small calculated changes in your infrastructure that don’t create a whole bunch of dependencies if you don’t have to.

Automation enables small iterative change to many solutions at scale. If your deployment process is part of a CICD pipeline you have the ability to easily make multiple changes a day and rollback to previous configurations if needed at any time.

Working in small iterative chunks allows you to better keep an eye out for potentially destructive operations. In many situations, especially with AWS, certain modifications of resources actually result in a total replacement of the resource changing unique identifiers and/or breaking dependencies for other deployed resources. More often than not these operations will fail as a result of being dependent or depended upon but this is why you should be very considerate of everything you deploy, in what order, and with what other resources.

Test everything

In the solid foundation that you should have already laid out to this point, there should have been included a test environment. This is the account where you keep a working copy of everything that is deployed into your mission-critical accounts. If you can keep an up-to-date version of every mass-deployed resource in this account, you should be able to avoid major mishaps when it comes time to deploy and terminate solutions across your MAA. This account is critical for testing the removal of resources and creating procedures or scripts that will help you clean up or recover from a situation where you didn’t have your resources as de-coupled as you had hoped.

Plan for scale

Rather you are shooting for 4 accounts or 100, it is important to design solutions with reuse in mind. This means parameterizing as much as possible. Hardcoding names, identifiers, and other unique variables will lead to additional and unnecessary work.

Don’t build something yourself if there is already a solution available. This is important to consider for smaller organizations thinking they are going to save a few bucks. Your custom solutions most likely won’t scale the way you need when growth starts happening organically. Once your foundation is laid and systems in place, you will be surprised how quickly your architecture will scale. Maintaining and updating home-bake solutions to meet your architecture needs will quickly become very cumbersome if not impossible.

Expect the unexpected

Regardless of the CSP you go with, there will be nuance upon nuance with how the provider operates. Things you couldn’t possibly have planned for or thought of will creep up causing you to reevaluate your entire deployment model. Expect the unexpected. Treat your architecture like software and create newer versions of it over time to correct for the things you didn’t realize at the time were going to be an issue.

Feedback and flexibility

Communicate openly and honestly about the pros and cons of a particular deployment decision with all stakeholders. It is likely the person responsible for the creation of a new process or solution isn’t seeing everything from all angles. No single developer or operator in your organization can be responsible for accounting for every gotchya.

Everyone has blindspots. If you are the one responsible for the deployment of a new process or solution that is supposed to optimize ‘X’, be ready for your team to look at your solution critically and provide feedback or criticism. It is everyone’s responsibility to do what is in the best interest of the organization. With regard to cloud technology, there are at least a half dozen ways to accomplish anything. Be flexible and accepting of their feedback and do not take it personally.

Wrapping up

In this series, we’ve covered a whole lot. Everything from why an organization would even want to bother with all the extra work of establishing a multi-account cloud architecture, what one would look like, when the right time to implement is, how to best support MAA at scale, and now some of the lessons learned.

We sincerely hope you’ve enjoyed this series and we look forward to your feedback. We would also appreciate you sharing your opinions and stories regarding your current or projected future cloud architecture.

AWS Control Tower v. Landing Zones

As we were preparing this series, it was requested we provide a comparison between two AWS native service offerings designed to help streamline and manage this entire process. We have that in the works currently so check back often for that post.

Content & Resources

We are also in the process of polishing up an online course that better details all of the steps for creating the AWS Organizations foundation for a successful multi-account architecture that we have discussed in this series.

Series parts are as follows:

Support and maintenance of multi-account architecture

Ross Wickman — Mon, 23 Mar 2020 12:30:00 +0000

This is post 8 of 9 in a multi-part series (hosted here) discussing the advantages, pitfalls, deployment methodology, and management of a multi-cloud account architecture. For this series, we are focusing strictly on using AWS as the Cloud Service Provider (CSP) but the concepts discussed port well to any provider or even to on-premise operations.

All aboard!

This journey cannot begin until you have everyone on the same page. By this point in the process, the gears should have really started to turn. Your organization can see how a multi-account architecture can really benefit everyone. We’re talking from customer to developer, operations and security, finance as well as stakeholders.

However, ideas can start to run rampant. Egos can go unchecked. Having a solid foundation with extreme flexibility can lead to a lot of carelessness. Now is more important than ever to make sure everyone is following procedure in order to properly support and maintain this new large-scale architecture.

From my experience, working with enterprise networks and leading teams that manage large-scale architecture, success can be distilled down to these 4 components:

Baseline
Documentation
Collaboration
Automation

Out of the box, this all just looks like work. A lot of work. Work that no one wants to take on. It will take time, but this is the under-the-hood work that needs to be done to make your architecture run well for the long haul.

Baseline

Having a defined starting point is critical. There could be many baselines within a fine-tuned architecture. A baseline in this context is really a defined set of instructions, values, conditions, and resources that will be used to deploy a new cloud account.

An organization could maintain a different baseline for the deployment of different types of accounts. For example, a different set of instructions for a department-specific account verses an application account. The baseline for what a development account looks like in comparison to a production account should definitely be different.

The key here with a baseline is they are fixed. Not that they cannot change and evolve over time, but once you’ve initiated the deployment of an account or process, the current baseline version will be followed through until completion. Organizations will evolve, grow, and prune their baselines over time. They are not set in stone.

Benefits of baselines

A well-defined baseline helps everyone within the team or organization know and understand the definite starting point for a project or resource. Baselines also allow developers and operators to set expectations on what is or should be present in every account allowing for automation and scalability. This four-letter acronym, RECD, helps describe the benefits.

Repeatability

When creating a baseline, deploying the exact same thing every time inherently becomes the norm. This allows solutions that are developed for a single account to scale to many. With the concept of repeatability, you have a cookie-cutter deployment model where operations know exactly how to deploy or maintain an account so developers can find or use whatever they need in the exact same place no matter what account they are in.

Evolution

Because architecture baselines should be deployed with coded templates, your organization will have a documented history of how the environment has evolved over time. This is important to help the organization understand where they may have gained or lost efficiencies and be able to adjust accordingly in the next iterations. If a key component to your baseline is removed because it may be viewed as no longer necessary, you will have a historical record to reference if that component ever needs to be reintroduced.

Compliance

It’s hard to imagine a space any more where there isn’t some level of regulation dictating what or how you should be doing something. Even if it is only from your internal security team, baselines allow operations to better provide compliance information as well as maintain a more secure architecture. Critical security configurations should be near-identical across all accounts as a result of being deployed with a baseline. As stated earlier, having confidence that something is configured in a specific way in a specific location is paramount to being able to gather compliance information as well as make bulk modifications.

Destruction

Knowing that the steps to recreate something identically if need significantly reduces re-work. Having a baseline means you can (in most cases) completely tear-down something that seems to be causing issues. Maybe someone made a configuration change out-of-band. Being able to recreate and environment the exact way it’s supposed to be in short order increases the ability to simply cut-bait when something isn’t working as it should.

It’s cyclical

There are many benefits to having baselines with regards to your environment. We really can’t come up with a scenario where not having a baseline would be a net positive. Even baselining a development sandbox enables cost reduction and more secure testing practices.

Creating baselines, however, does come at a cost. Documentation is critical to maintaining robust baselines.

Documentation

It should come as no surprise that in order to support and maintain a large-scale multi-account cloud architecture requires documentation. But it might not be the type of documentation you are thinking. Granted the skill level, experience, and tenure of your operators and developers will at times determine how detailed your documentation needs to be.

Because of the vast majority of your architecture and deployment of that architecture should be in coded templates, there is no need to create seemingly endless numbers of detailed SOPs containing hundreds of steps and dozens of screenshots. Your documentation for this type of architecture should be very basic and enable autonomy. The templates and scripts should be well commented on. The logical and agreed-upon naming convention goes a long way. Lastly, less documentation of higher caliber enables more reuse. If a significant amount of context is required to understand a process, your repeatability is reduced greatly.

Naming convention & standards

Determining upfront what the labels, names, and variables for resources and processes within your environment should look like is critical to maintenance success. Having a system for naming ‘things’ is not only important for information gathering but it is a necessity in order to make mass modifications at scale.

Have documentation that lays out what names of resources should look like. Agree on what Snake case, Camel case, Kebab, and many others should look like for each category. Use different standards for different resource types.

Avoid redundancy. There are some (very few) cases where (especially in AWS) you need to include the type of the resource in the name of the resource. This is especially true when the people caring the most about the resource works mainly in the console. The is no need to name your Development VPC – DevVPC.

Layout the key labels required. What are the minimum pieces of information a resource should have to tell an admin, developer, or operator what the resource if for, what level of protection it needs, and how long it should be allowed to live?

Procedural

Mapping out the steps to deploy a solution template does not need to include every input for every field. In fact, it shouldn’t include any. Your steps should be type-o proof. We’ll discuss in a different series about automation. Anyone present in your environment after 6 months of spinup time should be able to glean the information required to conduct any process with the vaguest of steps.

Any process should have the minimum amount of detail required to successfully conduct. Only add clarification and additional details where required AFTER a step in the process has failed or been misinterpreted by more than one individual. This is an indication that more clarity may be required. However, do not add this information to the master procedure. When possible, abstract additional detailed steps to a separate document and reference that document from the primary instructions.

The key to solid documentation for maintaining a robust baseline and deploying secure and manageable architecture is simplicity. Maintaining up-to-date documentation can be very difficult. The information may be continually changing. Get as close to the process without introducing confusion and misinformation. No information is much better than incorrect information. Especially with regards to deploying critical infrastructure.

Location of information

Documentation that describes mindset, mentality, and the logic behind an organization’s process is more important in most cases than the information itself. Create resources that enable autonomy and decision making at the lowest level. When there are already agreed-upon naming conventions and procedural understandings within the organization, support and maintenance move faster.

Share instructions on how to use the organizational standards to get the information that is required. Documentation for the sake of having a process documented is worthless if no one is going to follow the steps. Document only the situations where manual intervention is required and could most likely cause a discrepancy. This would be ‘custom settings’, one-off configurations, or dependencies. The bulk of your documentation should represent where and how to find the required information when it is needed – not demonstrate the precise steps required. Parameterize and use variables in your documentation as much as possible to scale with your architecture.

Collaboration

Communicate between teams the standards, conventions, and procedures frequently. At a reasonable frequency for your organization, get together and discuss what is/not working for all stakeholders.

Any opportunity to reduce the back and forth between developers, administrations and operators will significantly impact the supportability and maintainability for the positive. If teams become annoyed or upset about something, it is important to understand why and agree on corrective action before groups begin drifting from the standard or coming up with their own processes.

Collaboration enables team cohesion, builds comradery, and further reduces the requirement for over detailed documentation. The idea is to build a culture that supports large-scale architecture. Not one that sabotages it.

Automation

About as obvious as documentation is automation. If manual intervention is a requirement for many steps in an architecture deployment, one could imagine at some point in the scaling process, systems begin to break down.

Automate anything that you can that happens more than once. Automation is a form of documentation. The appropriate process for automating something should include commented code and the ability to logically follow a process in case that process were to ever break down.

Automation does so much more than shave seconds or minutes off the front end of a process. As we stated earlier, it is important to enable autonomy. Automation does this. The more cogs you have in a process, the more potential for things to go wrong or steps to get missed.

Even a simple process, that only takes a minute or two, conducted weekly could have a profound impact on the amount of rework it could cause if the process is completed incorrectly and deploys or configures a resource incorrectly. Remember, the naming convention is critical for the ability to make a mass modification at scale. If the individual responsible for the process does so incorrectly or misses the process entirely due to a busy schedule or absence, considerable harm to your architecture could result.

Automation doesn’t completely eliminate human error; however, it can significantly reduce it.

Tools, 3rd-party solutions, & systems

As a bonus or honorable mention, it is important to state that these four components for successfully supporting and maintaining MMA have some sort of tool, service, or system behind them.

How you conduct your automation and where you store your documentation is considerably less important than actually having these components as a solid part of your foundation. However, it would be impossible to support and manage this type of architecture without an array of different tools. This series and this specific post is not the place to share the many options available.

Each organization will also be very different than the next with regards to their preferred tools and systems for deployment. Budget, technical competency, time resources, etc. all factor into what should be considered a viable solution for each architecture.

Series parts are as follows:

How to accomplish a successful multi-account architecture

Ross Wickman — Mon, 16 Mar 2020 12:30:00 +0000

This is post 7 of 9 in a multi-part series (hosted here) discussing the advantages, pitfalls, deployment methodology, and management of a multi-cloud account architecture. For this series, we are focusing strictly on using AWS as the Cloud Service Provider (CSP) but the concepts discussed port well to any provider or even to on-premise operations.

It’s all about the numbers

Once you’ve made it this far — you’ve laid a solid foundation with your organizational structure — it’s all about turning out ‘widgets.’ Widgets being new AWS accounts that is. What you’ve built up until now is just the starting point of your assembly line. Now you need to make it run like a fine-tuned system. From here on out, it shouldn’t matter if you are creating a new AWS account per week, month, quarter, or once per year. Your growth trajectory doesn’t matter with regards to the work up to this point. Again, you need a solid foundation to build off of, but the real work hasn’t even started yet. Once complete, 3 accounts or 100 accounts shouldn’t give any more or less heart-burn. That goes for Ops, Support, and Security teams.

Getting to the starting line

If you are coming at this from an already ‘well-established’ cloud environment, you might be thinking to yourself “how in the hell…” I don’t blame you. It can look like a really daunting task. It is. Unfortunately, a single blog post can’t help you that much.

If you have a clean slate and are just starting out, you are in a very good position. We will cover the steps for a successful MMA here, starting with your preexisting condition.

Somebody’s baby

No matter how you go about it, someone is likely to get their feelings hurt when you take on such an initiative of correcting your current state of affairs in your cloud architecture. The biggest thing to remember moving forward is — it got this way using the tools, services, talents, and knowledge available at the time. ‘Don’t throw the baby out with the bathwater.’

There is nothing wrong with how it is or how we got to where we are. It is just time to make some changes. Use this opportunity to learn and grow technically (whoever is going to die on the hill about not changing what they created because it is working). Not even a technologist with a crystal ball would have been able to predict the rapid rate of change and service offerings of any cloud provider. It’s pretty amazing you’ve done as well as you have.

If you are in this predicament, here are the abbreviated steps I recommend to get to a good starting point. It is important to realize that the following process will look very different for every organization. Use these steps as a guidepost for the direction you want to head and not as instruction etched in stone.

First, a gif to illustrate all the steps:

Take inventory of your current (legacy) architecture
- The current number of accounts
- Services represented in each account (AWS and Application Tiers)
Deploy a new account an promote it to the AWS Organization Master if you do not already have one
- Create OUs for your desired structure
- Enable inherited features/services you know you will use across all account
- Prepare Billing alerts and quotas
- Establish potential SCPs but do not deploy them over your OUs yet
Stand up your Orchestration and Administration tier
- create a security account for a logging and security functionality
- depending on your organization and needs, this may be multiple accounts
Deploy a research & development tier
- If you are developing in a different VPC but in the same account, now is the time to stop
- Create a sandbox (dirty) environment that has significantly more lax permissions for service so developers can freely test things without mucking up development
- Dev/Test/QA, depending on the number of accounts you want to manage, should be a 99% replica of your production environment
Here there is a fork in the road.
1. Establish a new production to be a clean slate.
2. This allows dev to be more in line with the previous bullet.
3. Hard to move enterprise tools/config can remain in place and move through attrition
4. Begin pulling out all other administration and enterprise tooling from your existing production account
5. Establish share/brokered services accounts
6. Deploy authentication and networking in their own managed accounts
7. This allows for better delegation and separation of responsibility as well as the ability to more easily leverage new services and features released in the future
By here you’ve determined where production is going to live
- New account
- legacy account cleaned out
- Either way now is the opportunity to clean house and lockdown ingress/egress points. I know how hard this step can be, so if at all possible I recommend a clean environment for production
Transition your deployment of new releases to your new production environment
- If you’ve taken this opportunity to establish new roots for production, it’s time to cut over. Leave behind your legacy setup for roll-back
- If you’re living in your legacy production environment still, you’ve completed moving as much administration and enterprise resources as possible out of this account
Terminate legacy or wash – rinse – repeat.

That’s it. You’ve done it!

Well, most of it. We haven’t talked about how to deploy all the additional accounts you might want going forward.

If fact, some of that information could be very beneficial back around Step 3. Luckily for you, we’ll try to cover that in the next post.

Clean slate

If you’re just starting out, you don’t actually get to jump all the way down here. You need to go back and review what life could be like if you don’t properly organize from the get-go.

You see, you have a lot of benefits in being a little tardy to the party. But don’t count your chickens before they hatch. Hindsight is 20/20. Coming into this clean means you don’t know what you don’t know. You don’t have the ability to know how you would have done things differently until after you’ve already deployed a solution.

The upside is with the solid foundation established upfront, ‘back to the beginning’ can be as simple as hitting the delete key.

Just because you have a clean canvas for your architecture doesn’t mean you won’t still find yourself pinned in a corner as a result of some undocumented limitation or you didn’t totally account for the entire cost of implementing a solution.

Use the previous steps to whiteboard a few potential deployments. Ask yourself:

Who needs to access what services? And for what?
What security implications are there when putting to services together?
Which services will be brokered or shared?
Where will we fail-over to? Is DR in a different availability zone, region, or combination?

Playing the long game

Still, at this point, the work has barely even started. Going forward you will need the tools and resources to deploy rapidly on top of your foundation. While the battle starts here with the planning phase, the war is won by implementing sound processes, standardization, convention, and lots of rigor.

We cover that in the next post.

Series parts are as follows:

What a multi-account architecture might look like

Ross Wickman — Mon, 09 Mar 2020 12:30:00 +0000

This is post 6 of 9 in a multi-part series (hosted here) discussing the advantages, pitfalls, deployment methodology, and management of a multi-cloud account architecture. For this series, we are focusing strictly on using AWS as the Cloud Service Provider (CSP) but the concepts discussed port well to any provider or even to on-premise operations.

Choose your own adventure

There’s really no ‘one-size-fits-all’ solution when it comes to deploying accounts within your organizational structure. In our previous post about ‘end-state’, we discuss very briefly a few different types of accounts you’d want to include in your architecture and why they might be beneficial. We also shared this diagram:

In this diagram, we show at the top a pretty standard foundation for managing a cloud architecture of any size that should support nearly any scale.

The AWS Organizations structure works by implementing inheritance through the use of containers called Organizational Units. These containers can be nested and used to organize different accounts based on an infinite configuration scheme that best suits your needs. Once the management structure is determined, the organization of sub-accounts can take place.

At the top

In our ‘End-State’ post, we already discussed the key foundational accounts recommended for creating a robust MAA. The number of those types of accounts and for what service they provide should be determined by organizational needs and available resources. There can be a fine line between logical separation of services and total chaos.

At the very least you have your AWS Organizations account that is used to spawn subsequent accounts. In a later series or individual post, we will share how this process takes place. It is in this account you enable the sharing of other AWS Services across all accounts linked in the Organization. This is an important component.

Alongside your Organization account, an architecture should at the very least have a single account for security and enterprise services as well as an account for shared resources.

However ya wanna slice it

Once you’ve established the Root Organizations account and any other management accounts, you can begin placing the accounts into Organizational Units (OU) or ‘functionality buckets.’

These buckets allow you to apply the maximum level of permissions as well as available AWS Services allowed to be used in each account. Controlling the permissions allowed in an account is done through the use of Service Control Policies (SCPs). Too much to cover here; however, these policies allow you to completely disable services in all accounts within the OU they are placed.

One especially beneficial use-case for SCPs is implementing available services within AWS GovCloud Regions. For example, when working with the DOD and their Cloud Computing Security Requirments, you will learn that different AWS services are allowed for different information types or ‘Impact Levels’. However, there a not different types of AWS GovCloud account to accommodate the different impact levels. All services regardless of their accreditation for use are enabled in the GovCloud Regions. You can use OUs and SCP to disable the use of services that have not been vetted for use within a specific impact level. This will prevent even administrators from inadvertently leveraging a service within an environment that it has not been accredited for.

Below is an example of an Organizational Hierarchy for such a situation. Within our testing organization, we have 6 AWS accounts: Orgs/Master, Dev, Prod, Security, Management, and Release.

Prod and Release are in a higher impact level OU, while Dev, is in the lower to enable R&D. The Orchestration Accounts can be excluded from either to enable the use of shared services.

Be logical

The following images are taken directly from a presentation we’ve given on creating and managing via code a multi-account architecture. You can use them to spark ideas on the best way to structure your account hierarchy.

It is important to note that the shaded and unshaded boxes do not need to denote an actual Account or an OU. Depending on the situation, they could both be OUs where the lowest level OU contents all the accounts for that category or service.

Project

A sub OU structure can be created to organize all projects into their own Organizational Units where the first OU is the for the whole project, and all accounts can fall under that single OU, or additional sub-OUs can be containers for like account types.

Billing

Each initial level OU in your structure could be for all accounts that are related to the same cost-center or source of funding. Many organizations have different ‘colors of money’ where lumping like accounts into the same OU structure could enable the disabling of services. For now, you must still use tags to allocate the cost of all services across all accounts in a structure like this, but expect AWS to enhance AWS Organizations to accommodate more granular billing views in the future.

Isolation

If you’ve properly configured your networking and other application-wide services within shared services accounts, it might be logical to completely break out your OU structure to tier specific roles. Not only can you segregate database admins from the developers and their services, but you can also completely disable services that have nothing to do with hosting data from consuming cost in the respective accounts.

Failover & High-availability

This is a concept that we have not personally tested but it seems like, in the right circumstance, it could prove to be very beneficial. There are many different ways to implement blue/green or virtually zero down-time deployments in the cloud. One solution could be to dedicate full accounts to releases of your application or service.

This solution provides many benefits. Depending on your release cycle and how long a release is live, you could have unique billing information for every release. This would give you the ability to more easily see if new changes to your infrastructure, scaling, and/or code reduced the cost of resources or increased.

This solution also more easily gives you the ability to give weighted access to the new release for a small percentage of customers and increase over time. Since nothing at all should have changed in a completely separate account with regards to the development and deployment of new code, you can very easily revert back to the old release if something starts going wrong.

You get to decide

As you can see, there is virtually an infinite number of accounts and OU configurations with regard to the hierarchy of your multi-account architecture. Each business case could serve different needs.

The benefit of this flexibility increases the manageability and scalability of whatever solution you are working to deploy. However, you logically decide to layout your accounts doesn’t have to be concrete either. This can evolve over time. With the proper configuration of shared services, it may even be possible to introduce accounts or services from entirely different organizational structures. But that’s enough to cover for one post.

Series parts are as follows: