Forem: Derek Sedlmyer

Maintaining and Governing Developer Accounts with AWS Control Tower, Part 2

Derek Sedlmyer — Fri, 12 Jun 2020 18:10:22 +0000

I work at a consulting company where there are numerous developers and consultants that require a sandbox environment in AWS. Developers may be working in isolation or they may be collaborating with other developers on a shared project.

At present, everyone shares a single AWS account and it becomes challenging to govern. We routinely have to prune resources no longer in use when our monthly bill becomes out of control. This requires an admin to look through the resources in numerous regions and reach out to developers to figure out who owns a resource.

To help manage and govern our AWS usage, I'm going to use AWS Control Tower to allow admins and developers to provision isolated accounts at the developer or project scope and to provide a set of guardrails to enforce AWS best practices.

In the second part of this series, I'm going to describe how the Landing Zone is configured for my team and the customizations added to support some of our policies.

User Management

The first administrative task is to create User accounts in AWS Single Sign-On so that developers can access appropriate AWS accounts in the organization.

Control Tower configures an AWS Single Sign-On directory in the Master account. This directory can be configured to use a user identity store in Active Directory, an external identity provider, or through the built-in identity store in AWS SSO. For my team, we're going to use the built-in identity store.

The next step is to begin inviting the users into the Organization. The users will be added to the AWSAccountFactory group initially. This will give them permissions to use Account Factory to create new AWS accounts within the Organization. Upon create a new user, an invitation will be sent via email where the user can set their password and login.

A recommended practice is to create groups that model your organization's roles and responsibilities. Since groups in AWS SSO can't be members of other groups, assign users to groups managed by your organization and the AWS-managed groups.

I'm going to create a Developers group within AWS SSO and assign the developer users to that group.

There are a few of us on the team that will be administrators on Control Tower and the Organization. These users will have administrator access to the Master account and full access to AWS Control Tower. For them, I'm going to create an Admins group and assign their user accounts to that group.

In AWS SSO, the current list of groups:

For my organization-managed Admins group, I'm going to assign it the AWSAdministratorAccess Permission Set to the Master, Log archive, and Audit AWS accounts.

Permission Sets are backed by an IAM policy. Use Permission Sets to enforce permissions on a User or Group for an AWS Account.

Landing Zone Configuration

A common issue with using a single-shared AWS account is that the account becomes like the wild west. Resources are created but never terminated, team members come and go, and it is tedious to determine who created a resource when it comes time to prune resources when the AWS bill comes due.

Another issue with the single-shared AWS account is the blast radius if the account were ever to become compromised. When an account is compromised AWS disables the account until all issues are resolved. This could have a wide impact on other developers and projects.

To improve management and governance, each developer will have a dedicated AWS account. Additionally, there are shared projects which numerous developers work and collaborate. Each shared project will so have a dedicated AWS account.

Each developer AWS account and project AWS account will be added to the Landing Zone configured by Control Tower. The account will inherit the guardrails assigned to the account's OU. All API activities and resource configurations will also be logged.

Two Organizational Units (OUs) will be added as well. One for developers and one for projects. Using separate OUs will allow us to manage and govern the accounts with different guardrails and policies.

This diagram illustrates the layout of the OUs within the Landing Zone.

When creating OUs for Control Tower, it is best practice to create them through Control Tower and not through AWS Organizations. Using Control Tower to add OUs to the Landing Zone will ensure that they display within Control Tower and have guardrails applied to them. Control Tower will manage the lifecycle of an OU within AWS Organizations.

OUs within Control Tower should be parented to the Root OU and not nested in another OU. Nested OUs are not accessible to Control Tower as it only displays the top-level OUs.

After creating the developers and projects OUs within Control Tower, the Organizational Units page looks like this:

Examining the developers OU, it will show details on the OU including accounts assigned to the OU and guardrails.

At this time, the OU is ready to be used and new accounts can be assigned to the OU. There are further customizations required to enforce some custom policies which will be covered next.

Landing Zone Customizations

While AWS Control Tower provides a good base set of guardrails on member accounts, my team would like to add further policies. For instance for developer accounts, initially we would like to enforce these policies:

Limit EC2 Instance Types to nano, micro, small, or medium instance types
Limit RDS Instance Types to micro, small, or medium instance types

To enforce these policies, Service Control Policies (SCPs) can be configured within AWS Organizations and attached to an OU or an individual account.

Additionally, we would like to add a resource to each developer AWS account:

Set up an initial budget of $25 per month and send notifications at configured thresholds

Resources can be added to AWS accounts using CloudFormation and by using StackSets to deploy to the member AWS accounts from the Master account.

Configuring Service Control Policies and CloudFormation stacks is straightforward, it can be difficult to do at scale. Managing SCPs at the OU level in AWS Organizations makes it easier to enforce policies, the same is not true for CloudFormation stack sets. After accounts are creating through the Account Factory, an admin will have to remember to set up the appropriate CloudFormation stacks on the new account. Also, if new resources and stacks need to be added to exist accounts, then an admin must manually configure the stacks on each AWS account. This can be very tedious and error prone.

AWS provides a custom solution named Customizations for AWS Control Tower.

From the Customizations for AWS Control Tower Deployment Guide:

The Customizations for AWS Control Tower solution combines AWS Control Tower and other highly-available, trusted AWS services to help customers more quickly set up a secure, multi-account AWS environment using AWS best practices. Before deploying this solution, customers need to have an AWS Control Tower landing zone deployed in their account. This solution enables customers to easily add customizations to their AWS Control Tower landing zone using an AWS CloudFormation template and service control policies (SCPs). You can deploy the custom template and policies to individual accounts and organizational units (OUs) within your organization. This solution integrates with AWS Control Tower lifecycle events to ensure that resource deployments stay in sync with the customer's landing zone. For example, when a new account is created using the AWS Control Tower account factory, the solution ensures that all resources attached to the account's OUs will be automatically deployed.

Information on Customizations for AWS Control Tower can be found here: https://aws.amazon.com/solutions/implementations/customizations-for-aws-control-tower/.

Customizations for AWS Control Tower configures the following deployment architecture in the Master account to deploy SCPs and CloudFormation Stack Sets on existing member accounts and new member accounts.

Deploying the Customizations for AWS Control Tower Solution

To configure Customizations for AWS Control Tower in the Master account use the CloudFormation template that's included. It can be found at: https://s3.amazonaws.com/solutions-reference/customizations-for-aws-control-tower/latest/custom-control-tower-initiation.template.

There are 2 parameters for the CloudFormation stack -- PipelineApprovalEmail and PipelineApprovalStage.

Set PipelineApprovalStage to Yes if you would like to approve any changes to SCPs or CloudFormation StackSets before deployment. An email will be sent to email address set in the PipelineApprovalEmail parameter to notify when there is a pending approval.

Setup Customization Configuration Package

The Customizations for AWS Control Tower will deploy SCPs and CloudFormation StackSets using policy files and templates contained in a Configuration Package. AWS CodePipeline in the Customizations for AWS Control Tower will create SCPs using the policy files and create CloudFormation StackSets from the templates and apply them to the appropriate accounts at the OU or account levels.

The developers guide for the Configuration Package and Customizations for AWS Control Tower can be found at: https://s3.amazonaws.com/solutions-reference/customizations-for-aws-control-tower/latest/customizations-for-aws-control-tower-developer-guide.pdf.

The Configuration Package is uploaded into an S3 bucket which will trigger the deployment process in AWS CodePipeline.

I created a GitHub repo with a sample Configuration Package that contains Service Control Policies for limiting instance types for EC2 and RDS and a CloudFormation template for configuring a Budget in each member account.

DerekSedlmyer / aws-landing-zone-sample

Sample Configuration Package for the Customizations for AWS Control Tower solution.

The manifest file in the Configuration File controls the deployment of SCPs and CloudFormation templates. In the manifest file, the list of Organization Policies (SCPs) and CloudFormation Resources is configured to include the OUs or individual member accounts to target. The configuration of the manifest file will be described in the following sections.

Configuring Service Control Policies

In our team's member accounts, we want to limit the size of the EC2 and RDS instances in order to save costs. This can be controlled via Service Control Policies (SCPs). SCPs are defined as IAM policies. The SCPs will be configured within AWS Organizations via the Customizations for AWS Control Tower solution.

Service Control Policies (SCPs) offer central control over the maximum available permissions for all accounts in your organization. SCPs help you to ensure your accounts stay within your organization’s access control guidelines.

The following is a sample SCP for limiting EC2 instance types to only nano, small, micro, or medium. In this policy there is an explicit deny to Run EC2 Instances that don't match the pattern of *.nano, *.small, *.micro, or *.medium.

custom-control-tower-configuration/policies/ec2-instance-types.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "LimitEC2InstanceType",
      "Effect": "Deny",
      "Action": "ec2:RunInstances",
      "Resource": "arn:aws:ec2:*:*:instance/*",
      "Condition": {
        "ForAnyValue:StringNotLike": {
          "ec2:InstanceType": [
            "*.nano",
            "*.small",
            "*.micro",
            "*.medium"
          ]
        }
      }
    }
  ]
}

The following is a sample SCP for limiting RDS instance types to only small, micro, or medium. In this policy there is an explicit deny to Create or Start RDS databases that don't match the pattern of *.small, *.micro, or *.medium.

custom-control-tower-configuration/policies/rds-instance-types.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "LimitRDSInstanceType",
      "Effect": "Deny",
      "Action": [
        "rds:CreateDBInstance",
        "rds:StartDBInstance"
      ],  
      "Resource": "arn:aws:rds:*:*:*",
      "Condition": {
        "ForAnyValue:StringNotLike": {
          "rds:DatabaseClass": [
            "db.*.micro",
            "db.*.small",
            "db.*.medium"
          ]
        }
      }
    }
  ]
}

With the SCPs defined in the two separate files, the next step is to configure them within the manifest so they are applied appropriately. It will apply the SCP to all current and future accounts within the developers OU.

This section of the manifest file will apply the SCPs to the appropriate member accounts in the landing zone.

custom-control-tower-configuration/manifest.yaml:

organization_policies:
  - name: limit-ec2-instance-types
    description: Limit EC2 instance type in member accounts
    policy_file: policies/ec2-instance-types.json
    apply_to_accounts_in_ou:
      - developers
  - name: limit-rds-instance-types
    description: Limit RDS instance type in member accounts
    policy_file: policies/rds-instance-types.json
    apply_to_accounts_in_ou:
      - developers

After deployment, we can see the SCP configured in AWS Organizations for the developers OU.

In any AWS account within the developers OU, the creation of EC2 or RDS instances outside of the policy will not be allowed.

Configuring CloudFormation Resources

In our team's member accounts, we want to create a budget for $25 per month (for example) and then notify once budget thresholds of 75/90/100/110% thresholds. This will help us govern costs so that we can stop unused resources before the bill becomes out of control.

To set up the Budget, a CloudFormation template is needed. A CloudFormation StackSet will be created within the Master account using the template and will create individual CloudFormation stacks in the appropriate member AWS accounts.

AWS CloudFormation StackSets extends the functionality of stacks by enabling you to create, update, or delete stacks across multiple accounts and regions with a single operation. Using an administrator account, you define and manage an AWS CloudFormation template, and use the template as the basis for provisioning stacks into selected target accounts across specified regions.

The CloudFormation template to define the Budget is shown here.

custom-control-tower-configuration/templates/budget.template

AWSTemplateFormatVersion: 2010-09-09
Description: Configures Budget Notifications for member accounts at 75/90/100/110% thresholds
Parameters:
  BudgetName:
    Description: Name to assign budget
    Type: String
  BudgetAmount:
    Description: Budget Limit for one month 
    Type: Number
    Default: 25
  NotificationEmailAddress:
    Description: Email Address to notify when 75/90/100/110% thresholds are reached
    Type: String
Resources: 
  AccountBudget:
    Type: AWS::Budgets::Budget
    Properties:
      Budget:
        BudgetName: !Ref BudgetName
        BudgetLimit:
          Amount: !Ref BudgetAmount
          Unit: USD
        TimeUnit: MONTHLY
        BudgetType: COST
      NotificationsWithSubscribers:
        - Notification:
            NotificationType: ACTUAL
            ComparisonOperator: GREATER_THAN
            Threshold: 75
            ThresholdType: PERCENTAGE
          Subscribers:
            - SubscriptionType: EMAIL
              Address: !Ref NotificationEmailAddress
        - Notification:
            NotificationType: ACTUAL
            ComparisonOperator: GREATER_THAN
            Threshold: 90
            ThresholdType: PERCENTAGE
          Subscribers:
            - SubscriptionType: EMAIL
              Address: !Ref NotificationEmailAddress
        - Notification:
            NotificationType: ACTUAL
            ComparisonOperator: GREATER_THAN
            Threshold: 100
            ThresholdType: PERCENTAGE
          Subscribers:
            - SubscriptionType: EMAIL
              Address: !Ref NotificationEmailAddress
        - Notification:
            NotificationType: ACTUAL
            ComparisonOperator: GREATER_THAN
            Threshold: 110
            ThresholdType: PERCENTAGE
          Subscribers:
            - SubscriptionType: EMAIL
              Address: !Ref NotificationEmailAddress

The next step is to configure the CloudFormation template within the manifest so the CloudFormation resources are created appropriately and targets the right accounts. The CloudFormation template to define a Budget will be configured to be created in all current and future AWS accounts in the developers OU.

This section of the manifest file will apply the CloudFormation resources to the appropriate member accounts in the landing zone.

custom-control-tower-configuration/manifest.yaml

cloudformation_resources:
  - name: budget-small
    template_file: templates/budget.template
    parameter_file: parameters/budget.small.json
    deploy_method: stack_set
    deploy_to_ou:
      - developers
    regions:
      - us-east-1

The parameters file will define the parameter values to be used when creating the CloudFormation StackSets. In this case, the values for a small budget will be used. You will notice that BudgetAmount is $25

custom-control-tower-configuration/parameters/budget.small.json

[
  {
    "ParameterKey": "BudgetName",
    "ParameterValue": "budget-small"
  },
  {
    "ParameterKey": "BudgetAmount",
    "ParameterValue": "25"
  },
  {
    "ParameterKey": "NotificationEmailAddress",
    "ParameterValue": "name@email.com"
  }
]

After deployment, we can see the CloudFormation StackSet created in the Master account in the landing zone with a few instances in the member accounts:

If we look in a member account, we can see the instance of the CloudFormation stack that was created from the StackSet in the Customizations for AWS Control Tower solution in the Master account.

Conclusion

This post shows how to configure the Landing Zone created by AWS Control Tower and also how to deploy custom policies and resources to AWS accounts in the Landing Zone using the Customizations for AWS Control Tower.

Stay tuned for additional posts in this series on AWS Control Tower and governance.

Maintaining and Governing Developer Accounts with AWS Control Tower, Part 1

Derek Sedlmyer — Fri, 12 Jun 2020 18:10:03 +0000

In the first part of this series, I'm going to walkthrough the process of configuring a Landing Zone in AWS Control Tower. Further articles will dive deeper into the specific configuration for my team.

AWS Control Tower Initial Setup

At my company we already have an existing AWS account that's acting as a shared space for all developers. I'm going to use this account as the AWS Control Tower landing zone. A landing zone is a is a well-architected, multi-account AWS environment that's based on security and compliance best practices. This is the enterprise-wide container that holds all of your organizational units (OUs), accounts, users, and other resources that you want to be subject to compliance regulation.

To start configuring AWS Control Tower:

Log in with a IAM user that has AdministrativeAccess in the account
Navigate to the AWS Control Tower console https://console.aws.amazon.com/controltower/home/dashboard?region=us-east-1
At the console, if AWS Control Tower has not been configured, the following screen will be shown, click Set up landing zone to begin
At the Set up landing zone screen, the following information is required:
- Master account: This is the email address used to set up your master account. This account will be used as the master account for the landing zone and will be the consolidated billing account (each member account managed by AWS Control Tower will have bills consolidated under the master account).
- Log archive account: This is the email address used to create the AWS account that contains a log archive of all API activities and resource configurations from the master account and all member accounts managed by Control Tower. This email address must be unique. I had to work with my company's system administrator to create an email alias for this account.
- Audit account: This is the email address used to create the AWS account that strictly used by security and compliance teams. These teams will have read/write access to all accounts managed by Control Tower. This email address also has to be unique.
Creating the landing zone takes about 60 minutes or less. The status can be monitored from the Control Tower console
When complete, the Control Tower dashboard will indicate success and provide a summary of the items that were configured.

At this point, there is a base-level landing zone configured and ready to be used. In the next section, I'll walk through the items that were configured by Control Tower.

What was created

A Landing Zone in AWS Control Tower creates items in other AWS services such as AWS Organizations, AWS Single Sign On, CloudTrail, Config and others.

AWS Organizations

Control Tower creates an organization in AWS Organizations to create and centrally manage multiple AWS accounts. Accounts in AWS Organization can be organized into groups and policy-based controls can be attached to those accounts.

Within Control Tower, Organizations help centrally manage billing. The bills for the member accounts will be billed under the Control Tower master account as a single bill with detailed views for each member account.

Organizations also helps control access, compliance and security across the member accounts. Access, compliance and security policies can be mapped to groups of AWS accounts called Organizational Units to efficiently manage the policies at scale.

Other AWS resources can also be shared across accounts in an Organization. One example, is sharing a VPC across all accounts in an Organization. This allows the networking team to centrally manage network policies, gateways, and VPN/Direct Connect connections.

Organizational Units

While AWS Organizations provide the capabilities to organize AWS accounts into Organizational Units (OUs), Control Tower provides the blueprint on what the pre-configured OUs should be and the policies attached to those OUs.

In addition to the Root OU, which contains the master account, Control Tower configures these additional OUs:

Core: Contains the Audit and Log Archive shared AWS accounts that were created when Control Tower was configured
Custom: Contains the member accounts for users to perform their workloads

Guardrails are applied at the OU scope for all accounts that are a member of that OU.

Shared Accounts

Control Tower creates three shared AWS accounts in the Landing Zone.

Master Account: This is the account where the AWS Control Tower Landing Zone was first configured. All billing will be consolidated to this account for all accounts within the Landing Zone. OUs and Guardrails are configured in the Master account and the Account Factory uses the Master account to provision member accounts
Log Archive Account: This account maintains a log of all of API activities and resource configurations. This account has CloudTrail, CloudWatch Logs and S3 buckets for storing the logs files.
Audit Account: Restricted account for audit and compliance teams. Those teams will have read/write access to all member accounts in the Landing Zone. Only programmatic access from the audit account to other member accounts is allowed.

Guardrails

Guardrails are high-level rules for providing governance across all accounts in the Landing Zone. Guardrails can be expressed in simple language to convey their goal clearly. Guardrails can be mandatory, strongly recommended, and elective. Mandatory guardrails cannot be disabled.

Guardrails are attached to OUs within the Organization and not specific accounts.

Guardrails can either be preventive or detective. An example of a preventative guardrail is Disallow policy changes to log archive. This prevents any IAM policy changes in the log archive shared account. Any attempt to perform a prevented action is denied and logged in CloudTrail. The resource is also logged in AWS Config.

A detective guardrail detects an event and logs the event in CloudTrail. An admin can view resources that are out-of-specification with the guardrail. For example, the Enable encryption for EBS volumes attached to EC2 instances detects if an unencrypted Amazon EBS volume is attached to an EC2 instance in your landing zone.

Guardrails are implemented with Service Control Policies (SCPs)in AWS Organizations for Preventative guardrails and AWS Config rules and Lambda functions for Detective guardrails.

At this time, Guardrails are provided by AWS and controlled via Control Tower versions. Admins of a Landing Zone cannot create their own Guardrails and apply them to OUs. Later in this article, I'll show how to create custom Service Control Policies and apply them to OUs within Control Tower.

Account Factory

There are three methods to creating a member account within AWS Control Tower:

Account Factory console (AWS Service Catalog)
Enroll Account (AWS Control Tower)
Lamdba with IAM roles in Landing Zone Master Account

The recommended approach is through the Account Factory console. Users will the appropriate permissions can launch a new member account within the Landing Zone maintained by AWS Control Tower. The new member account will assigned to an OU and will inherit the enabled guardrails of that OU.

When launching a new account from the Account Factory console, end users won't know they are working within Control Tower. They will simply use the console to create and terminate an AWS account.

AWS Single Sign-On (SSO)

An AWS Single Sign-On directory is created in the Master account when creating a new Landing Zone with Control Tower. AWS Single Sign-On allows cloud administrators and end users to manage access to multiple AWS accounts.

Within AWS SSO, Control Tower configures multiple user groups that have various permissions to perform operations within the Landing Zone. The following user groups are created:

Group name	Description
AWSAccountFactory	Read-only access to account factory in AWS Service Catalog for end users
AWSAuditAccountAdmins	Admin rights to cross-account audit account
AWSControlTowerAdmins	Admin rights to AWS Control Tower core and provisioned accounts
AWSLogArchiveAdmins	Admin rights to log archive account
AWSLogArchiveViewers	Read-only access to log archive account
AWSSecurityAuditors	Read-only access to all accounts for security audits
AWSSecurityAuditPowerUsers	Power user access to all accounts for security audits
AWSServiceCatalogAdmins	Admin rights to account factory in AWS Service Catalog

AWS SSO also defines a list of Permission Sets to define permissions that a User/Group have within an AWS Account. Permission Sets are created with an IAM Policy that defines the Allow and Deny permissions to be given to a user/group.

The following Permission Sets are created by default:

Permission set	Description
AWSReadOnlyAccess	This policy grants permissions to view resources and basic metadata across all AWS services
AWSServiceCatalogAdminFullAccess	Provides full access to AWS Service Catalog admin capabilities
AWSServiceCatalogEndUserAccess	Provides access to the AWS Service Catalog end user console
AWSPowerUserAccess	Provides full access to AWS services and resources, but does not allow management of Users and groups
AWSAdministratorAccess	Provides full access to AWS services and resources
AWSOrganizationsFullAccess	Provides full access to AWS Organizations

Control Tower admins can create additional permission sets within the Landing Zone to meet their needs.

Conclusion

This article gave a high-level overview of AWS Control Tower and a walkthrough on configuring a Landing Zone using it. In the next article of this series, I'll discuss how I configured the Landing Zone for our group of consultants and developers.

Preparing for SnowPro Core Certification

Derek Sedlmyer — Tue, 28 Apr 2020 02:36:54 +0000

Recently I sat for the SnowPro Core Certification exam via online proctoring by Kryterion. I wanted to share my exam preparation and exam experience in order to help fellow colleagues interested in obtaining this certification.

About Snowflake

Snowflake is a cloud data platform built from the ground up for the cloud. Snowflake is built on a patented, multi-cluster, shared data architecture created for the cloud to revolutionize data warehousing, data lakes, data analytics and a host of other use cases.

Snowflake is a single platform comprised of storage, compute, and services layers that are logically integrated but scale infinitely and independent from one another.

Snowflake is a SaaS solution and is maintained by Snowflake and not the customer. Snowflake manages the services, compute, and storage layers. Data is only accessible via SQL in Snowflake. Snowflake can be deployed within AWS, Azure or GCP although not in customer controlled accounts.

About SnowPro Core Certification

The SnowPro Core Certification demonstrates an individual's knowledge to apply specific core expertise implementing and migrating to Snowflake.

The subject areas covered includes:

Snowflake Architecture
Snowflake Client Interfaces and Connectivity
Data Loading/Unloading
Semi-Structured Data
Working with Snowflake
Account and Resource Management
Performance Optimization
Data Sharing
Security & Access Control

Exam Format:

100 multiple select and multiple choice
Length: 2 hours
Registration fee: $175 USD
Currently only available in English

Exam Preparation

To prepare for the exam, I utilized the following resources:

Hands-On Lab Guide
Training videos and Practice Exams hosted on Udemy and developed by Hamid Qureshi
Snowflake University
Snowflake documentation

I estimate that I spent about 8-10 hours on exam preparation.

Hands-On Lab Guide

The Hands-On Lab Guide is a great place to become familiar with Snowflake. This will require a trial account to complete. Snowflake provides a 30-day trial of the Standard edition that can be used for this tutorial.

The Hands-On Lab Guide tutorial takes a 1-2 hours to complete.

Udemy Courses

There are 2 Udemy courses that I found valuable in preparing for the exam, both of them developed by Hamid Qureshi.

Snowflake Decoded - Fundamentals and hands on Training contains 60 video lectures with hands-on labs and quizzes which took about 4 hours to complete. The short videos covered all the major topics of Snowflake. The questions in the quizzes are a lot easier than the questions in the exam. I purchased this training for $11.99 from Udemy using a promotion they were running at the time.

Tackling Snowflake Certification - Practice Questions contains 5 practice exams that really do a good job of helping you prepare for the exam. Each exam contains 56-74 questions that you will find on the exam. This course was also $11.99 which was a great value.

Snowflake University

Snowflake University contains a number of free courses to help you learn Snowflake and help with certification exam preparation. Snowflake University has the following courses:

I used the Level Up Series to help fine tune some areas where I needed some improvement. The Level Up Series has 8 videos with quizzes dedicated to various topics included first concepts and performance topics. The performance topics were really beneficial in learning about caching and query and results history.

Additionally, Snowflake University provides a study guide for the exam. This has valuable links to the documentation and other exam resources like videos and blogs.

Snowflake Documentation

The Snowflake Documentation is the source for all content required to pass the exam. If there is a single piece of advice I could give is to study the documentation. Use the practice exams to determine areas of improvement and use the documentation to study in further depth.

Exam

The exam itself is proctored by Kryterion. Due to COVID-19, I used online proctoring since onsite testing centers were closed. When using online proctoring, software will be required to deliver the exam to your computer and enable the webcam and audio for proctoring.

To schedule, go to WebAssessor and find a time slot that will fit your schedule. At the time I scheduled, the available time slots were very limited.

Prior to your exam, be sure to install the Sentinel software that will deliver the exam to your computer. This software will block you from accessing other software on your computer during the duration of the exam. The software will also require access to your webcam and microphone. After installing the Sentinel software, a biometric profile will require setup. The biometric profile setup will prompt you to enter your full name multiple times and it will perform facial recognition.

The exam itself will contain 100 multiple choice and multiple select questions. You will have 2 hours to complete the exam. At the end of the exam, your results will be provided.

Conclusion

Overall the exam is straightforward. By doing the hand-on lab tutorial, reading the documentation in depth and utilizing some existing free resources provided by Snowflake you have a good start to passing the exam. Using the Udemy courses, especially the practice exams, will provide a great foundation.

Getting Started with Amazon WorkSpaces

Derek Sedlmyer — Mon, 06 Apr 2020 18:50:35 +0000

As I write this, the world is dealing with the COVID-19 pandemic. A majority of the world's citizens are under some sort of strict lockdown or stay-at-home orders by their government in an effort to slow the spread or "flatten the curve" in order to not overwhelm the hospital system. This has brought an immediate challenge to organizations to quickly enable a remote workforce in order to achieve business continuity.

One of the challenges facing organizations in this crisis is connecting remote workers in order to maintain productivity and data security. Organizations are faced with a number of issues at this time of crisis including hardware and network constraints, desktop software patching, endpoint security and others. The crisis is forcing many organizations to adapt to a new reality of remote workers.

With the influx of a large number of remote workers, on-premise networks including VPNs are unable to meet the new demands. Workers may have limited to no connectivity if corporate VPNs are out of capacity. Additionally, remote workers using legacy desktop applications on their laptops may face networking issues due to increased latency and limited bandwidth due to unplanned usage scenarios of legacy 2-tier desktop apps.

Some organizations may have workers that depend on higher performance workstation to support more demanding workloads. When workers are suddenly forced to work remote their productivity will be impacted due to the lack of access to the higher-performance hardware. They may be relegated to use underpowered laptops which can severely affect their productivity.

A solution to these issues is to use Desktop-as-a-Service (DaaS). DaaS is the next generation of Virtual Desktop Infrastructure (VDI). Previous VDI implementations required complicated capacity planning, large capital expenditures for hardware, complicated licensing agreements, long implementation schedules and scalability limitations that prohibit dynamic scaling for an increased remote workforce.

AWS offers Amazon WorkSpaces as a DaaS solution. Amazon WorkSpaces simplifies desktop delivery, keeps your data secure, reduces costs and allows an organization to centrally manage and scale global desktop deployments. Using Amazon WorkSpaces an organization can launch Windows or Linux desktops in a matter of minutes as well as scale to thousands of desktops to support workers across the globe.

In response to increased demand for virtual desktops due to COVID-19, Amazon Web Services recently announced a new offer for organizations to use Amazon WorkSpaces for up to 50 users at no charge beginning on April 1, 2020 and running through June 30, 2020. For more details refer to this blog post: https://aws.amazon.com/blogs/desktop-and-application-streaming/new-offers-to-enable-work-from-home-from-amazon-workspaces-and-amazon-workdocs/

Creating WorkSpaces

I have deployed Amazon WorkSpaces for a few organizations and found them to be very beneficial and easy to use. In this blog post, I'm going to walkthrough a quick start to stand up an Amazon WorkSpaces environment. This should allow an organization to begin using the offer from Amazon for 50 free WorkSpaces.

Create an AWS account if you don't have one already. New accounts are eligible for the WorkSpaces free offer. Existing accounts are also eligible provided they haven't used WorkSpaces prior to the offer.
Open the Amazon WorkSpaces console at https://console.aws.amazon.com/workspaces/
In the upper-right hand corner, be sure to choose a region that is closest to your users. WorkSpaces require minimal latency between the client and AWS. WorkSpaces is available in the following regions:
- US East (N. Virginia): us-east-1
- US West (Oregon): us-west-2
- Asia Pacific (Seoul): ap-northeast-2
- Asia Pacific (Singapore): ap-southeast-1
- Asia Pacific (Sydney): ap-southeast-2
- Asia Pacific (Tokyo): ap-northeast-1
- Canada (Central): ca-central-1
- Europe (Frankfurt): eu-central-1
- Europe (Ireland): eu-west-1
- Europe (London): eu-west-2
- South America (São Paulo): sa-east-1
At the console, click the Get Started Now button. The Getting Started Now button will be displayed if you haven't used WorkSpaces in the account.
At the Get Started screen, click Launch next to Quick Setup
At the Get Started with Amazon WorkSpaces screen, in the Bundles section choose an appropriate bundle. I chose Standard with Windows 10 and Office 2016 which will provide 2 vCPU and 4GiB of Memory. I'll write more about Bundles in another post.
In the Enter User Details section, add the list of users to create WorkSpaces. Required fields are Username, First Name, Last Name, and Email. Only one user is required at this time. Additional users can be added later.
Once complete, click the Launch Workspaces button. This will create a WorkSpace for each user. Once the WorkSpace is created, each user will receive an email message providing instructions on accessing their WorkSpace.
After WorkSpaces are launched, go back to the WorkSpaces console. It may take around 20 minutes to launch the WorkSpace. After the WorkSpace is successfully created, you will see the WorkSpace listed with a status set to AVAILABLE.

At this point, the user will have received an email notifying them that the WorkSpace is ready for use. It will provide instructions on how to install the WorkSpace client app on their device, register the WorkSpace in the client app and use the WorkSpace.

Configuring WorkSpaces Client

At this point, a system administrator has created a WorkSpace for a user. The user received an email from AWS notifying them that a WorkSpace has been created. The next steps are for the user to install the client app on their device and register the WorkSpace in the client app.

The email received by the user looks something like this:

From the email follow the link in Step 1 to complete user profile and download a client app. The following screen will be displayed:
Complete the form, by entering and confirming the password, then click Update User. Note that passwords are case-sensitive and must be between 8 and 64 characters in length, inclusive. Passwords must contain at least one character from three of the following categories: lowercase letters (a-z), uppercase letters (A-Z), numbers (0-9), and the set ~!@#$%^&*_-+=`|(){}[]:;"'<>,.?/.
You will then be sent to the WorkSpaces Client Download page which will allow you to download the WorkSpaces client app for your device.
Download and install the client app appropriate for your device. Supported devices are:
- Windows
- iPad
- MacOS X
- Android Tablet
- Chromebook
- Fire Tablet
- Web Access
- Linux
The next step is to register the WorkSpace with the client app. Go back to the email and find Step 2. Copy the registration code to the clipboard.
Open the WorkSpaces client app on your device. The following screen will be displayed.
Paste your registration code in the text box and click Register
After successful registration, the login screen is displayed:
At the login screen, enter your username and password (set in Step 2) and click Sign In

What's Created

So far, I have shown how a WorkSpace is provisioned by a system administrator and how a user accesses the WorkSpace from their device. Next, I'll dive a little deeper to show the AWS resources that were created during this process.

Virtual Private Cloud (VPC) was created during the Quick Start. The VPC has 2 public subnets, each residing in a different availability zone.
Simple AD Directory created in the VPC. The Simple AD Directory is a basic Active Directory-compatible directory used to store user and WorkSpace information. The Simple AD is deployed across 2 availability zones for high-availability and redundancy.
User Account in the Simple AD Directory, john.doe in this case.
WorkSpace instance. The instance is associated with an elastic network interface in the VPC and the network interface is assigned a public IP address to provide internet access.

Summary and Next Steps

This blog post showed a quick start for standing up an Amazon WorkSpaces environment. While this is beneficial for training and sandbox purposes, deploying to WorkSpaces in a production environment for an organization requires more planning and architecture.

In future blog posts, I'll write about various WorkSpace features, pricing and best practices.

Stay tuned. In the meantime, please be safe.

Using AWS Batch to Generate Image Thumbnails for Voyager

Derek Sedlmyer — Tue, 01 Aug 2017 01:53:09 +0000

In the 1.9.9 release of Voyager, some exciting new capabilities were added. Most notably are Voyagers new support for cloud deployments and AWS S3. Many organizations, including a lot of our customers, are increasingly deploying applications, services and data in public clouds, which is one of the reasons why they have asked us to deploy Voyager in the cloud and index data stored in cloud repositories.

For general users, Voyagers easy-to-use interface and its ability to find any piece of data within their organizations are key benefits. System administrators like Voyager for its enterprise integration and data management features; while developers like Voyager because of its REST APIs and extensibility for creating custom locations, extractors and pipeline steps.

This blog post will highlight a new way for developers to build on top of Voyager and use some elasticity features that public cloud providers offer and focus on how AWS Batch can help developers horizontally scale out processes that will push data into Voyager.

AWS and NAIP Content

As part of our recent VoyagerODN release, we are including NAIP content from the NAIP on AWS hosted dataset. NAIP on AWS contains leaf-on imagery for the continental United States for the prior two collection years and is hosted in Amazon S3 in a Requester Pay bucket in us-east-1 (Northern Virginia) with over 1.3 million images accumulated between 2012 and 2015.

With VoyagerODN, we wanted to index NAIP content while providing users with visually appealing thumbnails in the search results. Adding the new S3 capabilities enables us to easily index NAIP content, but in order to generate thumbnails, we needed a solution to scale out beyond a single server so it could be completed quickly. Given the NAIP imagery was over 90TB in volume, processing would take weeks to build thumbnails. We needed to build the thumbnails faster.

Enter AWS Batch. AWS Batch was announced at AWS re:Invent 2016. AWS Batch can run hundreds of thousands of batch computing jobs by dynamically provisioning compute resources based on the volume and specific resource requirements of the batch jobs.

The key components of AWS Batch are jobs, job definitions, job queues and compute environments. Jobs are executed as commands in a Docker container, and the job definitions define the Docker container to use, the command to execute, the resource requirements, and other configurations for a job. Jobs are then submitted to a job queue. Compute environments define the cluster of servers to execute the jobs, pulling them from one or more job queues and executing them. AWS Batch manages the execution, provides up-to-date status, and writes console output to CloudWatch for monitoring. The only real work is developing the scripts for actually processing the data.

For VoyagerODN, the plan for building thumbnails is:

Develop a script to build thumbnails for a set of NAIP images. In this case, well use State Name and Year combination prefix to define a set. Use GDAL in the script to build the thumbnails After building thumbnails for the set, push the thumbnails to S3 In Voyager, when indexing NAIP data, turn off thumbnail generation, and add a Pipeline step to set the external thumbnail URL.

Step 1: Build Docker image

In this step, I need to build a Docker image based on Ubuntu with GDAL and Python installed and configured.

The full Docker file shows how the Docker image is built. The aws-cli, python, and gdal software packages will be installed via apt. Additionally, Python libraries will be installed via pip. The script to build thumbnails will be copied into the image as well.

FROM ubuntu:trusty

# Define Python required libraries in requirements.txt
# Copy requirements file into the Docker image
COPY requirements.txt /tmp/requirements.txt

# Install required software via apt and pip
RUN sudo apt-get -y update && \
      apt-get install -y \
    awscli \
    python \
    python-pip \
    software-properties-common \
 && add-apt-repository ppa:ubuntugis/ppa \
 && apt-get -y update \
 && apt-get install -y \
      gdal-bin \
 && pip install --requirement /tmp/requirements.txt

# Copy Build Thumbnail script to Docker image and add execute permissions
COPY build-thumbnails.py build-thumbnails.py

RUN chmod +x build-thumbnails.py

The Docker image is built and tagged (vg-gdal) with the following command:

$docker build -t vg-gdal .

The benefit of using Docker is the same image can be used in development, test, and production environments. No more works on my machine. Lets test this new Docker image on a developer machine by running the script in a Docker container with the Docker image:

$docker run -e AWS_ACCESS_KEY_ID=<ACCESSKEY> -e AWS_SECRET_KEY=<SECRETKEY> -it vg-gdal /usr/bin/python build-thumbnails.py aws-naip al/2013/ voyager-aws-naip

This command does the following:

Runs the build-thumbnails script in a container running the previously built Docker image named vg-gdal
Specifies environment variables for AWS credentials (this wont be needed when running in AWS Batch, more on that later)
Build thumbnails from images in the aws-naip bucket with key that begin with al/2013/ (All images collected over Alabama in 2013)
Stores the thumbnails in the voyager-aws-naip bucket (key names will be in sync between aws-naip and voyager-aws-naip)

After successful execution, thumbnails will be built and stored in the voyager-aws-naip S3 bucket. From here, the Docker image needs to be pushed to an image repository, either Docker Hub or Amazon ECR. In this case, Ill use a Docker Hub repository. The following commands will login to Docker Hub, re-tag the Docker image appropriately and push the Docker image to the repository:

$docker login
$docker tag vg-gdal dsedlmyervg/vg-gdal:latest
$docker push dsedlmyervg/vg-gdal:latest

Step 2: Configure AWS Batch

Now that I have a working script to build thumbnails running inside a Docker container and with the Docker image pushed to an image repository, lets configure AWS Batch to run batch jobs.

First step is to configure the Compute Environment. This will configure the instance type, security groups, networking, scaling policies and provisioning model.

To build NAIP thumbnails, Im going to configure the compute environment with:

Managed Compute Environment. AWS will scale and configure the cluster instances for me.
On-demand provisioning model. If I wanted to run this on SPOT that is another option, but I prefer to run this on-demand for now.
M4 instance family. M4 instances are good general purpose instance types and will fit fine for building NAIP thumbnails.
96 Maximum vCPUs. I chose 96 because there are 48 states with NAIP imagery and 2 years of data for each year, so I can process the entire dataset in parallel.
I also assigned an Instance Role to the instances. This will give the instances access to only write thumbnails to a specific S3 bucket. The instance profile will not grant access for the instances to access any other AWS service. Now I dont have to specify an AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY in my scripts, credential files or environment variables.

The next step is to create the job queue and connect it to the previously created Compute Environment. Ill assign a priority of 1 to the job queue so its jobs are given higher priority for compute resources.

Finally, its time to create a job definition to build thumbnails using the Docker image built in Step 1.

I created this job definition with:

Name is set to vg-aws-naip-thumbs
Using my previously created Docker image, dsedlmyervg/vg-gdal, as the Container Image
The command is defined as:

This is a similar command that I used when testing the Docker image in Step 1. The difference is Im specifying the input_prefix a Parameter in the Job Definition. Because its a Parameter, I can pass the actual input prefix when submitting Jobs.

vCPUs is set to 1. The job doesnt need multiple vCPUs
Memory (MiB) is set to 512. 512MiB of RAM will be enough for the OS and the Job to run.
Job Attempts is set to 2. In case of failure, the entire job will automatically be restarted. The build-thumbnails script will look to see if a thumbnail already exists before building it. This saves processing time.

The Compute Environment, Job Queue and Job Definition are configured. Next, its time to submit jobs to be executed.

Step 3: Submit Jobs

With AWS Batch configured to build thumbnails, I can now submit jobs to the job queue. AWS Batch will handle the execution and status updates for the jobs.

I decided to define a job using a State and Year. For example, Alabama has data for 2013 and 2015, so Ill submit two jobs for Alabama, one for 2013, and a second for 2015. I wrote a simple Python script to submit the jobs:

import json
import boto3

batch = boto3.client('batch')
s3 = boto3.client('s3')

states = s3.list_objects(Bucket='aws-naip',
      Delimiter="/",
      RequestPayer="requester")

for state_prefix in states['CommonPrefixes']:
  if state_prefix['Prefix'] == ".misc/":
    continue

  years = s3.list_objects(Bucket='aws-naip',
      Prefix="{0}".format(state_prefix['Prefix']),
      Delimiter="/",
      RequestPayer="requester")

  if 'CommonPrefixes' not in years:
    continue

  for year_prefix in years['CommonPrefixes']:
    input_prefix = year_prefix['Prefix']
    jobName = "NAIP_{0}".format(input_prefix.replace("/", "_"))
    print(jobName)

    try:
      response = batch.submit_job(jobQueue='vg-naip-thumbs',
        jobName=jobName,
        jobDefinition='vg-aws-naip-thumbs',
        parameters={"input_prefix":input_prefix })

      print ("Response: " + json.dumps(response, indent=2))
    except Exception as e:
      print ("Error submitting Batch Job")
      print (e)

I can run this script from my developer machine and will need to set the AWS credentials before running the script:

$export AWS_ACCESS_KEY_ID=<ACCESSKEY>
$export AWS_SECRET_ACCESS_KEY=<SECRETKEY>
$python submit-aws-naip-thumb-jobs.py

After all jobs are submitted, I my submitted jobs in the AWS Batch console:

I can monitor these jobs move through the Pending, Runnable, Startable, Running, and ultimately Succeeded states.

Step 4: Index NAIP

The final step is to configure Voyager to index NAIP and set the thumbnail appropriately. To index the NAIP data, create a new Cloud Storage location and set the Thumbnails Strategy to Do Not Build Images. Since I built thumbnails externally, I dont want Voyager to build the thumbnails.

Now I can use a Pipeline step to set the external URL for NAIP thumbs. Voyager's Indexing Pipeline provides functions to transform and manipulate the properties (metadata) of data records as it adds them to the Index. Pipeline steps can easily be developed using Python.

The Python code for the external_naip_thumbs step is:

import sys
import json

def run(entry):
    """
    Python step to compute the Thumb URL for a NAIP on AWS record. Thumbs are located in a
    publicly accessible AWS bucket. Thumbs have the same key as the full resolution image only
    ending in .thumb.jpg instead of .tif
    :param entry: a JSON file containing a voyager entry.
    """
    new_entry = json.load(open(entry, "rb"))
    key = new_entry['entry']['fields']['fs_key']
    thumb_key = key.replace(".tif", ".thumb.jpg")
    thumb_url = "https://voyager-aws-naip.s3.amazonaws.com/{0}".format(thumb_key)
    new_entry['entry']['fields']['image_url'] = thumb_url

    sys.stdout.write(json.dumps(new_entry))
    sys.stdout.flush()

The external_naip_thumbs.py script should be copied to the /app/py/pipeline/steps directory.

The external_naip_thumbs script does the following:

Defines a run method that accepts an entry parameter. The run method is required for a pipeline step. The entry parameter is a file path that contains the documents JSON representation.
The file path is opened and parsed as JSON into new_entry. new_entry is a Python dict that represents the indexed document.
The key (fs_key) field is retrieved. The key field was populated as a result of indexing S3 content. It contains the key name of the NAIP image in S3.
Using the key value, a URL to the Thumbnail is computed. The thumbnail is stored in S3 from running the jobs through AWS Batch.
The Thumbnail URL is stored in a separate field in the indexed document named image_url. Voyager will interpret image_url as an absolute URL to a thumbnail for the document.
As the final action in the pipeline step, the indexed document is serialized to JSON to stdout. The Voyager Pipeline process will read the indexed document from stdout and continue the indexing process.

Results

Using AWS Batch, I was able to build thumbnails for over 1.3 million NAIP images with data volume over 90TB over a weekend. That is a far better improvement compared to running on a single server and taking weeks to execute.

Below is the indexed content in Voyager. You will notice the thumbnails generated in AWS and hosted in S3.

Conclusion

This is a simple example of how developers can integrate a custom solution to push content into Voyager and still leverage Voyager Searchs capabilities within their organization. Future blog posts will build on this sample and show more advanced custom solutions.

For those interested in viewing source code, visit: https://github.com/voyagersearch/voyager-samples/tree/master/aws-batch-naip-thumbs