Forem: Ryan Cartwright

Resourcely adds Atlantis Support

Ryan Cartwright — Sat, 03 Aug 2024 02:40:27 +0000

You can integrate Resourcely with Atlantis to automatically evaluate your Terraform plans on pull requests. The Resourcely guardrail evaluation will result in findings that help developers address the violations.

In order to set up Resourcely with Atlantis, you must perform the following steps:

Verifying Prerequisites
Change management
Setup Resourcely with Custom workflows

Verifying Prerequisites

Before adding Resourcely to existing workflows, please verify that your Atlantis server environment:

Has internet egress access to download the Resourcely CLI binary or container (e.g., through a NAT Gateway).
Is configured to allow custom workflows.
Is used with GitHub as a VCS.

Change Management

This setup assumes you have already completed the integration of Source Code Management (SCM). If you have not, please follow this guide to complete the SCM integration.

Setup Resourcely with Custom workflows

This requires an Atlantis server-side workflow written in Atlantis YAML. Create a new file called repos.yaml or update your existing YAML and add the following content:

repos:
  - id: /.*/
    workflow: resourcely_guardrails
    allow_custom_workflows: true
    policy_check: false
    pre_workflow_hooks:
      # Install resourcely cli, use location `/opt/resourcely-cli` to run the CLI
      - run: |
            LATEST_RELEASE_TAG=$(curl -s -I <https://github.com/Resourcely-Inc/resourcely-container-registry/releases/latest> | awk -F '/' '/^location/ {print  substr($NF, 1, length($NF)-1)}')
            curl -s -L -O <https://github.com/Resourcely-Inc/resourcely-container-registry/releases/download/$LATEST_RELEASE_TAG/resourcely-cli-${LATEST_RELEASE_TAG}-linux-amd64.tar.gz> > /dev/null && tar xvzf resourcely-cli-${LATEST_RELEASE_TAG}-linux-amd64.tar.gz && mv resourcely-cli /opt/resourcely-cli
workflows:
  resourcely_guardrails:
    plan:
      steps:
        - env:
            name: RESOURCELY_API_TOKEN
            value: '<RESOURCELY_API_TOKEN>' # get a token from https://portal.resourcely.io/settings/generate-api-token
        - init
        - plan
        - show 
        # Run Resourcely 
        - run: /opt/resourcely-cli --log debug --api_host https://api.resourcely.io evaluate --change_request_url $PULL_URL  --change_request_sha $HEAD_COMMIT --format plain --plan $SHOWFILE
        description: Running Resourcely Guardrails

The resourcely-cli command in your repos.yaml evaluates your Terraform plans by downloading policies from Resourcely, assessing them, and submitting the results to Resourcely. These findings will be displayed on the pull request associated with the Atlantis run.

Note that the server needs to run with --repo-config=repos.yaml

atlantis server \\
...
--repo-config=repos.yaml \\
...
...

Atlantis should now run the Resourcely CLI on every pull request whenever new code is created or updated.

Why we built Resourcely

Ryan Cartwright — Thu, 18 Jul 2024 01:30:11 +0000

Written by: Travis McPeak
Published Date: July 16, 2024

Three primary trends have influenced the software industry over the last 15 years: the rise of the cloud, a shift from waterfall to continuous development, and DevOps practices.

These trends have all shifted more responsibility to the developer: deployment, security, and configuration are now the responsibility of software engineers. Software teams who were once only responsible for designing and creating applications are now expected to be infrastructure and deployment experts as well. You can read more about the history of security and configuration here.

Developers owning infrastructure inevitably results in rampant misconfiguration: improper parameters for cloud infrastructure that are costing companies millions. When a database, virtual machine, or SaaS application is configured incorrectly, there are two results:

incidents, breaches, and outages
wasted developer time

We built Resourcely to eliminate misconfiguration, stemming the tide of needless incidents and tackling the hours of waste we observed throughout the industry.

Frankly, we were frustrated by the status quo of cloud infrastructure configuration, and we decided to do something about it. Our experiences at top-performing cloud-based companies like Netflix, Robinhood, and Databricks taught us how to solve this problem, and Resourcely is a product of those learnings.

It allows security, ops, and platform teams to give developers a paved road to production, generating secure and compliant Terraform that is deployed as part of your existing change management tooling. We didn’t stop at bespoke, customizable templates that give developers an easy button to deploy infrastructure. Resourcely also features a novel cloud resource policy language that acts as a safety net: making sure that any Terraform deployed meets the standards that you define.

With these tools, developers no longer loathe deploying infrastructure, and security teams look like superheroes by empowering secure defaults built into developers.

Status quo of configuration

Cloud resources are misconfigured across all software organizations every day. Developers who aren’t Terraform experts are asked to deploy heterogeneous services that they are unfamiliar with. These developers are faced with a trade-off: ship misconfigured infrastructure, or ship slowly. In many cases, businesses are pushing developers to ship faster at the expense of misconfiguration.

What does that developer experience look like? Developers are forced to research the individual cloud services they want to deploy, learn about configuration best practices for them, write Terraform that matches their mental model, and then push that Terraform in a PR. Often they are asked to use a specialized, internally-developed tool to accomplish this. While many individuals start offthinking that only a single resource will be required, they slowly realize that multiple cloud resources are always required (IAM, networking, storage, etc.). At every step along this path there is friction and opportunity for misconfiguration.

Security teams are only compounding the problem with long lists of vulnerabilities flagged by CSPMs and a reactive modus operandi. These vulns end up on a long list, handed to developers, who never get around to fixing them anyway.

So why does it matter?

These dynamics result in two primary issues costing companies millions of dollars each year:

Incidents

Improper configuration inevitably results in incidents, vulnerabilities, and breaches. Let’s consider two scenarios: a security vulnerability, and an internal application that goes down.

Security vulnerabilities can threaten the very lifeblood of a business. Consider the most simple of examples, where an S3 bucket is left accessible to public access. This is actually somewhat tricky to get right, as we’ve written about before.

Inadvertent exposure of data, or server access, or firewalls, or anything sensitive can result in public-facing breaches that cost upwards of millions of dollars. As of 2023, the average cost of a data breach in the US was around $9.5M.

Misconfiguration doesn’t only have to be about security and data breaches. Consider an internal application that is used to support customer service agents for an airline. Let’s say you have a database misconfiguration that results in queries to that database failing for several hours during peak travel season. This has directly impacted your customer satisfaction, retention, and future revenue prospects.

Developer productivity

In the same way that incidents have a quantifiable impact on your business, so does developer productivity - especially for businesses with software as a core product. Deploying and configuring infrastructure is a significant impediment to the speed at which developers move. To successfully do this they must:

Research cloud resources to use
Figure out the correct configuration, which is challenging to get completely right without being an expert
Write Terraform or use a Terraform module
If they use a module, they’re often fitting a square peg into a round hole
Push that Terraform to production

Frequently, the Terraform they’ve written is bounced back to them by a reviewer, which further compounds the developer productivity problem. Maintaining, updating, triaging, and changing their infrastructure can result in even more hours than they spent on the initial project.

Let’s say you have just 100 developers pushing infrastructure to production, with an average infrastructure deployment that make infrastructure changes 2/week. If your developers spend 4 additional hours each time they deploy researching Terraform and (trying) to get the configuration right, and they cost $275,000 fully loaded, you’re looking at $5.5M in annual cost and an incalculable amount of opportunity cost (from projects they could be working on instead).

👉 (100 developers) * (2 infra deployments/week) * (52 weeks) * (4 hours/deployments) * (275,000/year / 2,080 hours in a year) = $5.5M annually.

Secure defaults and the Resourcely answer

So you’ve got a problem costing you millions of dollars a year, and your cost almost triples each time you have an incident. We had these same issues at Netflix, Robinhood, Databricks, and other companies that our founding team worked at.

We solve this problem is by removing humans from the loop as much as possible. We implemented secure-by-default frameworks across the business, and gave developers a paved road to production that would allow them to get what they need without friction.

Asking humans to configure infrastructure, when we already know what best practices and company-specific requirements should be, is an outmoded practice rooted in years of on-premise and legacy security practices. Instead, we should be asking machines to make configuration decisions for humans. (Check out this episode of the a16z podcast where our CEO Travis discussed this concept)

Resourcely tackles misconfiguration at the source: turning security teams into proactive superheroes, and unleashing developers from DevSecOps hell of the past. How do we do it?

Guardrails

Guardrails govern how infrastructure can be created or changed, preventing inadvertent misconfiguration. These are policies, managed and written by security, ops, or platform teams, and enforced as part of your existing change management/GitOps workflows.

Imagine you write the following policy:

GUARDRAIL "Require prefix on all S3 buckets"
   WHEN aws_s3_bucket
       REQUIRE bucket STARTS WITH "resourcely-"
 OVERRIDE WITH APPROVAL @securityops

Resourcely will:

Let the developer know they are violating a guardrail if they don’t start an S3 bucket with resourcely-
Guide the user about how they should be configuring the resource
If the developer still chooses to continue, Resourcely will route the PR to the appropriate approver specified in the guardrail
The PR will always fail without approval, if the S3 bucket is not appropriately named Guardrails limit accidental public exposure, oversized infrastructure, inadvertent deletion, and more - all in an easy-to-read policy language remarkably similar to SQL.

Blueprints

Resourcely Blueprints are configurable templates, customized by security, ops, or platform teams, that give your developers a paved road to cloud infrastructure resources. They provide a UI with context, selection criteria, lists, and more.

Once published, blueprints are available in a service catalog that developers can use to bundle the infrastructure they want to deploy. Guardrails can be attached to blueprints, ensuring that your policies are enforced.

After submission, Terraform is generated and deployed through your existing change management and review processes.

Configuration platform
Over the next few weeks, we’ll be announcing the General Availability of Resourcely, covering how our customers are using it to build a secure-by-default platform for deploying infrastructure, and showing you how anyone can do the same.

The misconfiguration problem is widespread and hugely impactful. Adopt the premier configuration platform to eliminate misconfiguration, reducing millions in developer waste and bidding goodbye to incidents & breaches that average $9.5M of impact. Get started with Resourcely today, and follow along on our GA launch journey over the next few weeks!

Death of DevSecOps, Part 3

Ryan Cartwright — Fri, 05 Jul 2024 16:12:42 +0000

Written by Travis McPeak, Resourcely CEO.

In part 2 of this series, I explored the promises of DevSecOps and where they went wrong. To wrap up this series, we’ll propose how to solve the current problems in security and software development and highlight some early success cases using this approach.

DevSecOps has two primary problems: we asked developers to be the primary owners of security configuration at the expense of their primary responsibilities, and we haven’t provided automation tools that can take SecOps off their plate.

The result? Developers are burning down never-ending tickets, going through tedious threat modeling exercises across all of their applications, and undergoing hours of training for all vulnerability classes.

Secure-by-default

The solution is secure-by-default: an approach that shifts responsibility onto systems, not people. Secure defaults integrate security and configuration guidelines into tools that developers are using, leveraging new libraries that make security the default, all supported by a new security team. In short, systems ****should be responsible for security, not people.

Secure-by-default can help developers move faster and reduce incidents by automatically taking care of secure configuration without requiring developers to make complex, nuanced decisions - and stepping in to help them, when they make incorrect ones.

New technologies

The past ~10 years of DevSecOps have taught us some valuable lessons about developer behavior: they are not security experts, and they don’t like to leave their standard development and CI workflow.

To accomplish secure-by-default, any automated tooling needs to be embedded into existing developer workflows. This ranges from auto-suggesting security best practices within IDEs, to embedded context wherever configuration occurs, to using systems that make good security choices for you.

Some great examples of secure default libraries and systems are:

Netflix’s Lemur: makes it easy for a developer to get a TLS certificate for a microservice, without having to deal with cryptography, manage private keys securely, and remember to rotate certs before they expire
Google’s Safe Golang Libraries: protect against common issues such as YAML injection (see also Google’s Secure by Design)
See Clint Gibbler’s full list of secure by default libraries here

The second critical part of a secure-by-default platform is guardrails: policies and rules that proactively prevent misconfiguration, again embedded into the developer’s workflow. These are backstops that prevent developers from deploying vulnerable software, while allowing them to follow their existing workflow: developing locally, pushing to the cloud, using version control, and leveraging automated deployment tooling.

These embedded secure-by-default practices combine with guardrails that keep developers in track - resulting in a paved road to production. There should be paved roads across a variety of fields: infrastructure, application development, CI, and more.

The new security team

Automated tools that can take cognitive load off of developers are only possible with a savvy security team that is willing to truly embed security where developers are. This team should:

Shift from a reactive, issue-centric view of the world to a proactive, preventative strategy
Aggressively embed linters, context, and other in-IDE tech to help developers deploy securely at development time
Utilize secure by default libraries and frameworks that make classes of vulnerabilities impossible
Implement backstops and guardrails that preserve optionality by developers, while preventing incorrect configuration The foundational work of security should be done BY a security team, FOR a developer team - shifting the burden of security decision-making from developers onto systems, and making the last mile work for developers painless.

These new automation technologies will allow a security team to become extensible, scaling with a development team by embedding into their workflows without having to add additional security resources and burning out developers.

DevSecOps: Can it be saved?

Security teams have lagged their developer counterparts over the past 20 years, as cloud computing and dev practices have revolutionized the tech industry. While DevSecOps held great promise, it has resulted in the worst of both worlds: slow development, and frustrated security teams dealing with constant misconfiguration.

The next generation of security is secure-by-default. We have the tech, and we know what it takes to accomplish it - the only thing left is committed security teams helping embed secure-by-default into developer workflows.

Resourcely is working hard on this problem! To make your organization secure-by-default, get started with Resourcely and give your developer teams the security capabilities they need without leaving the tools they love.

Originally Published Resourcely Blog.

Resourcely founder-led in person or virtual hands-on workshop

Ryan Cartwright — Tue, 25 Jun 2024 22:10:06 +0000

Join Resourcely for a free founder-led in person or virtual hands-on workshop.

Learn how easy it is to enable cloud infrastructure paved roads to prevent misconfigurations for your organization.

In this session, you’ll learn how to:

✅ Navigate Resourcely user interface, and connection options.

✅ Integrate your SSO provider

✅ Integrate your VCS provider

✅ Understand what Resourcely Blueprints and Guardrails are in our catalog out of the box

✅ Understand the importance of Global Contexts

✅ Understand configuration options for Blueprints

✅ Understand configuration options for Really, our policy as code language

✅ Understand how Resourcely can integrate into your existing cicd process

✅ Import your existing terraform modules to transform them into module backed blueprints

✅ Understand Resourcely usage metrics available

✅ Learn how to configure the resourcely.yaml to manage different environments

✅ Create new Blueprints and Guardrails using Foundry for your specific use cases

Elevate your infrastructure as code maturity today!

Request workshop

Introducing Really: A New Policy as Code DSL… that doesn't suck

Ryan Cartwright — Wed, 29 May 2024 03:48:00 +0000

Using Rego for cloud configuration is awful. Use Really: policy-as-code built for humans.

Announcing Really

When we started building Resourcely, the global configuration engine for cloud infrastructure, we slowly realized that the status quo of policy-as-code was broken. Writing our first Resourcely guardrails in Rego took hours to create and even more time to maintain. Writing new policies was extremely tedious and time-intensive, especially given the fact that we wanted to make them flexible. To help achieve our mission of making infrastructure more secure, it became evident that a new policy language would be needed that allowed policy to be written and maintained without headaches.

The Resourcely team has spent the last 6 months working on developing that new policy language, and today we are introducing Really: a policy language and enforcement engine that is built to be written and read by humans - not machines. With the introduction of Really, policy can be created in minutes compared to hours, with a human-readable syntax that looks remarkably like SQL. The readability of our structured policy language means that writing, deciphering, and adjusting policy isn’t a nightmare, and your policy team can take vacation without worrying about their pager going off.

Yet Another Config Language (YACL)

Why did we invest millions of dollars in time to build Yet Another Config Language (YACL)?

Read full blog here.

Taming misconfiguration chaos with Resourcely

Ryan Cartwright — Mon, 15 Apr 2024 01:40:57 +0000

Misconfiguration of cloud infrastructure is the root cause of slow developer velocity and prolific incidents causing millions of dollars in damage.

Resourcely is a cloud resource configuration platform, built to give developers a paved road for deploying quickly and securely. tl;dr Resourcely generates secure-by-default Terraform code for anything that can be managed with a terraform provider.

Join Resourcely's CEO, Travis McPeak, on April 16th as he walks through why misconfiguration exists, the impact it has on businesses around the world, and how Resourcely can help increase developer velocity and mitigate incident risk.

https://app.livestorm.co/resourcely/resourcely-webinar-taming-misconfiguration-chaos

Resourcely exists to make cloud resources more secure and make developers' lives easier. In a DevOps world, developers have too many responsibilities, including writing code, testing code, deploying their services, responding to outages, provisioning cloud services, and keeping their services secure. With so much on their plate, developers need tools that give them best practices in these areas without thinking about them.

The Ultimate List of Resources to Migrate off of Terraform Cloud

Ryan Cartwright — Wed, 24 May 2023 22:55:02 +0000

If you are looking to migrate away from HashiCorp Terraform Cloud or Terraform Enterprise, here is a growing list (in no particular order) of possible options and specific migration resources to assist in your journey.

There isn’t much need to go into the back story on the inspiration for this post, but I hope at least one person or company finds it helpful.

Disclaimer: If you want to be added to the list and I missed an option, send me a note.

Self-managed Infrastructure as Code (IaC) Deployment: Set up your own infrastructure and deployment pipeline using self-managed tools like Terraform CLI, GitLab, Jenkins, or AWS CloudFormation.
GitOps: Adopt a GitOps approach where infrastructure changes are versioned and applied automatically based on changes to a Git repository using tools like Flux, Argo CD, or Jenkins X.
Cloud Provider-Specific Tools: Utilize cloud provider-specific infrastructure management tools like AWS CloudFormation, Azure Resource Manager, or Google Cloud Deployment Manager to transition away from Terraform.
Ansible: Use Ansible, an automation tool, for infrastructure provisioning and configuration management. Ansible allows you to define and apply infrastructure changes using declarative YAML-based playbooks.
Pulumi: Consider migrating to Pulumi, an open-source infrastructure as code tool that allows you to define and manage infrastructure using familiar programming languages like JavaScript, Python, or Go.
Cloud Native Tools: Embrace cloud-native tools and services offered by your cloud provider, such as AWS Cloud Development Kit (CDK), Azure Resource Manager templates, or Google Cloud Deployment Manager.
Vendor-Specific Migration Tools: Explore vendor-specific migration tools that help transition infrastructure from one IaC tool to another, such as the Terraform Import tool provided by HashiCorp or similar tools offered by other vendors.
Manual Configuration: Transition to manually configuring your infrastructure using web-based consoles or command-line tools provided by your cloud provider.
Infrastructure Automation Frameworks: Investigate other infrastructure automation frameworks like Chef, Puppet, or SaltStack, which offer configuration management and infrastructure provisioning capabilities.
Custom Scripting: Develop custom scripts using your preferred programming language to automate infrastructure provisioning and configuration changes based on your specific requirements.
Continuous Delivery Platforms: Companies battling the CICD space such as Harness, Armory, and OpsMx also offer ways to deploy Terraform.
Terra* Options: There is a growing list Terra______ (fill in the blank) that don’t find themselves to fit in any of the above categories such as Terragrunt, Terrateam, Terramate, Terrakube, Terraspace.
Open Source Options: One option has been around for a while called Atlantis, a terraform pull request automation platform and another is newcomer Digger, an open source GitOps tool for Terraform.
Terraform Automation & Collaboration Software (TACOS): TACOS like Spacelift, Scalr, and Env0 provide a framework for solving the problems with operating IaC at scale.

Here are a few recent blogs to assist with your migration journey:

How to Migrate From Terraform Cloud to Spacelift https://spacelift.io/blog/how-to-migrate-from-terraform-cloud

Recommendations for Migrating from Terraform Cloud
https://www.env0.com/blog/migrate-from-terraform-cloud-to-env0

More Recommendations for Migrating from Terraform Cloud
https://www.env0.com/blog/recommendations-for-migrating-from-terraform-cloud

Migrating Terraform state from Terraform Cloud to S3
https://blog.marcolancini.it/2023/blog-migrate-terraform-state-from-terraform-cloud-to-s3/

Remote State Migration with Terrakube
https://docs.terrakube.org/user-guide/remote-state-migration

Terraspace Cloud vs Terraform Cloud
https://terraspace.cloud/docs/vs/tsc-vs-tfc/

Migrating State Data Off Terraform Cloud

https://nedinthecloud.com/2022/03/03/migrating-state-data-off-terraform-cloud/

Migrating to Terrateam

https://terrateam.io/docs/migrating/terraform-cloud

System Integrator & Services:

CloudPosse
https://cloudposse.com/jumpstart-devops-accelerator/

https://cloudposse.com/faqs/why-do-you-recommend-spacelift/

GitHub Repos:

https://github.com/spacelift-io/spacelift-migration-kit

https://github.com/Scalr/terraform-scalr-migrate-tfc

https://github.com/env0/customer-tools/tree/main/migrations/tfc-to-env0-migration/s3-backed-state

https://github.com/cloudposse

It's essential to carefully plan and evaluate the migration strategy based on your existing infrastructure, requirements, and preferences. Consider the learning curve, migration effort, and compatibility of the chosen alternative with your existing environment.

PS. I’ll add more examples and resources as I discover them or as people send them my way.

Good news for Terraform Cloud & Terraform Enterprise Users

Ryan Cartwright — Sun, 21 May 2023 03:24:20 +0000

Announcing Spacelift’s price match guarantee for anyone migrating from Terraform Cloud or Terraform Enterprise!

I understand pricing and packaging pages cannot satisfy everyone’s requirements on the surface.

Some individual or companies want to..

🙋‍♂️ pay per user

🏃 pay per run (or apply)

🟰 pay per parallel execution

🥞 pay per workspace (or stack)

☁️ pay per cloud resource

⏱️ pay per minute

🎗️ pay per support type

♾️ pay for unlimited usage

Rigid pricing models create a lot of controversy and discussion. I’ve seen them incentivize bad engineering practices while others incentivize bad business processes, and others than slow down innovation and add more TOIL.

What if you could mix and match the models?

What if you could allow the customer to opt in to the model based on what works for them, what they understand, and what they can properly communicate internallly to other stakeholders in leadership, finance, procurement.

Most companies can’t satisfy everyone. At the end of the day, we are all human. We are the same and different all at the same time.

If you are genuinely interested in using Spacelift, I’m here to facilitate that conversation. On top of the flexibility that we offer, I will go one step further.

I will provide a price match guarantee for any user that wants to migrate away from Terraform Cloud or Terraform Enterprise.

I promise and commit to beat any price by 10% and will increase how much I beat them based on the size of your current contract.

For clients paying <$100K, I’ll beat their price by 10%.

For any clients paying over $100K, I’ll be their price by 25%.

For any clients paying over $1M, I’ll beat their price by 50%.

If you want to take advantage of this, I’ll throw in free migration services at no added cost to hand hold you through the process. If you don’t want our assistance, here is a link to our migration utility and blog.

https://github.com/spacelift-io/spacelift-migration-kit

https://spacelift.io/blog/how-to-migrate-from-terraform-cloud

Happy migrating!

Do I need another CI/CD for my infrastructure?

Ryan Cartwright — Mon, 07 Feb 2022 04:19:06 +0000

Yes, we believe it's a good idea. While in an ideal world one CI system would be enough to cover all use cases, we don't live in an ideal world. Regular CI tools can get you started easily, but Terraform has a rather unusual execution model and a highly stateful nature. Also mind the massive blast radius when things go wrong. We believe Spacelift offers a perfect blend of regular CI's versatility and methodological rigor of a specialized, security-conscious infrastructure tool - enough to give it a shot even if you're currently happy with your infra-as-code CI/CD setup.

There are many challenges of running Terraform in a general purpose CI system. At the end of the day, it's mostly about two things - collaboration and security.

Can I into my existing CICD process or system?

The short answer is Yes.

We can integrate with most of those tools. Although, your mileage may vary.

We realize that you may live and work in a world where you have to use existing tools and can't start from scratch. Or you may be forced to integrate with an existing CICD process or system.

Option 1

You can use the spacectl command line tool to easily call out to Spacelift from other CI/CD systems.

Option 2

In a similar way, you can call out to other CI/CD systems from Spacelift in a Stack's after_apply hooks.

Option 3

Spacelift can optionally be set to send webhooks - POST requests about run state changes - to an HTTP endpoint of your choice.

You can find documentation on how to access the #graphql API here.

You can see the details of available queries and mutations with their detailed documentation by using a client with native graphql support.

Hello, Spacelift!

Take your infra-as-code to the next level!

Spacelift is a specialized, Terraform-compatible continuous integration and deployment (CI/CD) platform for infra-as-code. It's designed and implemented by long-time DevOps practitioners based on previous experience with large-scale installations - dozens of teams, hundreds of engineers and tens of thousands of cloud resources.

At the same time, Spacelift is super easy to get started with - you can go from zero to fully managing your cloud resources within less than a minute, with no pre-requisites. It integrates nicely with the large players in the field - notably GitHub and AWS.

Introduction to Main Concepts

We will briefly introduce some key concepts that you need to know to work with Spacelift. These concepts will be followed by detailed instructions to help you create and configure your first run with Spacelift.

Stacks

A stack is a central entity in Spacelift. It connects with your source control repository and manages the state of infrastructure. It facilitates integration with cloud providers (AWS, Azure, Google Cloud) and other important Spacelift components. You can learn more about Stacks in Spacelift detailed documentation.

State Management

State can be managed by your backend or can be imported into Spacelift for Terraform projects. It is not required to let Spacelift manage your infrastructure state.

Worker Pools

Spacelift provides public and private worker pools that execute Spacelift workflows. Public worker pools are managed by Spacelift whereas private pools are hosted by you. Due to security and compliance requirements, several of our customers choose private pools to manage their infrastructure. You can learn more about worker pools here.

Policies

Spacelift policies provide a way to express rules as code, rules that manage your Infrastructure as Code (IaC) environment, and help make common decisions such as login, access, and execution. Policies are based on the Open Policy Agent project and can be defined using its rule language Rego. You can learn more about policies here.

Cloud Integration

Spacelift provides native integration with AWS, Azure and Google Cloud (GCP). Integration with other cloud providers is also possible via programmatic connection with their identity services. You can learn more about cloud provider integration in Spacelift detailed documentation.

Change Workflow

Spacelift deeply integrates with your Version Control System (VCS). Pull requests are evaluated by Spacelift to provide a preview of the changes being made to infrastructure; these changes are deployed automatically when PRs are merged. You can learn more about VCS integration here.

Step-by-step

This section provides step-by-step instructions to help you set up and get the most out of Spacelift. If you want to learn about core concepts, please have a look at the main concepts section.

You can get started with either forking our Terraform Starter repository and testing all Spacelift capabilities in under 15 minutes or you can explore Spacelift on your own by adding your own repository and going from zero to fully managing your cloud resources.

Free Trial

Our free trial provides 30 days of our enterprise tier if you want to kick the tires on Spacelift. If you get stuck or need advice, the Spacelift team is available to help.

How To: Create & Manage Preview Environments

Ryan Cartwright — Tue, 11 Jan 2022 00:50:00 +0000

Thanks to Jakub Martin, Software Engineer at Spacelift for this contribution.

Source: https://github.com/spacelift-io/demo-preview-environments-manager

This collection of repositories is a demo, as well as starting point, for a preview environments setup on top of Spacelift. It features a declarative, VCS-driven, very flexible, infa-as-code like approach which embraces the philosophy of managing Spacelift with Spacelift itself.

Whether you need per Pull Request development environments, QA environments created on demand, or any other form of dynamically allocated environment, the approach represented here will be able to cover all of them. Most imaginable preview environment lifecycles should be achievable with this setup.

No matter if your preview environment comprises a single Stack, or a hundred, a single microservice, or a hundred, including queues, databases, and other accompanying infrastructure, you'll be able to adapt the solution presented here to suit your needs.

The best way to get an overview is to watch the demo video: https://www.loom.com/share/a3427a529eb74b97994668fabc71b969

You can also look at the Pull Requests on the Service and Infra repositories to see the existing preview environments and play with them, i.e. curling https://475ed74.hello-service.preview-environments.liftspace.net/hello

Birds-eye view

The described setup is one of the more conceptually complex ones, with separate repositories describing the artifact for a microservice, and the infrastructure which hosts this artifact.

Preview environments are created for all Pull Requests in the Service and Infra repositories, are kept up to date with changes to those Pull Requests, and get cleaned up after Pull Request deletion.

The following repositories are used:

Service - AWS Lambda representing our application code, the resulting artifact is a zipfile in S3.
Infra - Sets up all the AWS resources the AWS Lambda needs, including the Lambda itself as well as API Gateway resources.
Manager - Manages resources representing preview environments. Acts as a repository for all existing preview environments.
Setup - Sets up the manager, with all its environment variables and policies.
Update File Action - Can be used to create/update/delete files in a different repository. The Service and Infra repository workflows use this to create preview environments in the Manager repository.

Detailed Descriptions

Manager

The root of the manager repository contains one directory per service, for which preview environments are created. In this case, there's one directory, hello-service, which is a module instantiated by the main.tf file.

Looking at the hello-service directory we can see multiple things:

policies - Fixtures for policies used by preview environment Stacks.
template - Describes a single hello-service preview environment as a Terraform module. In this case this is a single Spacelift Stack with accompanying environment variables, though you could easily extend this to be i.e. multiple Stacks connected by Trigger Policies. This Stack also has a spacelift_stack_destructor attached, which will make sure to destroy all resources of this Stack before deleting it. This way all resources of a preview environment are cleaned up on deletion.
environments - List of active preview environments, one file per preview environment. Each file is just an instantiation of the template module.
main.tf - Creates dependencies common to all preview environments, in this case it creates record for the wildcard domain name *.hello-service.preview-environments.liftspace.net. Each preview environments AWS Lambda will be accessible through this URL, with the environment ID in place of the wildcard.
For each preview environment there are multiple variables which need to be specified, with the interesting being:

code_version - The version of the artifact zipfile to use.
environment - The ID of the preview environment. Should be reasonably unique. In this case we'll be using 8 character prefixes of the hash of the creator repositories owner, name and pull request number.
infra_repository_branch - Which branch of the infra repo to base the preview environment Stack on. The Stacks also have a Push Policy attached to ignore any other branches, so that they don't create any proposed changes.
The manager Stack based on this repository also has an interesting Trigger Policy attached:

trigger[stack.id] {
    change := input.run.changes[_]
    change.phase == "apply"
    change.entity.type == "spacelift_environment_variable"
    stack := input.stacks[_]
    sanitized(stack.id) == change.entity.data.values.stack_id
}

Whenever it finishes execution, it will trigger any stacks it controls, for which environment variables have been changed. This way, if only the code_version for a preview environment changes, which only results in an environment variable change, the Stack representing this preview environment will be triggered.

Service

The service is code for an AWS Lambda responding with Hello World and the current commit SHA.

Whenever a commit to a non-master branch is pushed, it creates a zipfile with the source code, and uploads it to an S3 bucket with the object name being .zip.

Whenever a commit to the master branch is pushed, it does the same, but uploads the resulting artifact as latest.zip.

For each PR, it will create a file in the hello-service/environments directory, with the code_version filled with the head commit SHA of the Pull Request, and the infra_repository_branch filled with master. It updates the files on each commit added to the PR, and deletes the file when the Pull Request is closed.

Infra

The infra repository creates the actual AWS Lambda and API Gateway resources.

For each PR, it will create a file in the hello-service/environments directory, with the code_version filled with latest, and the infra_repository_branch filled with the source branch of the PR. It updates the files on each commit added to the PR, and deletes the file when the Pull Request is closed.

Setup

This just creates the manager Stack with the required Trigger Policy attached.

Update File Action

You can use this action to operate on files in a different repository. A GitHub token with suitable permissions has to be provided.

Usage

You can copy all those repositories to your organization and create a Stack based on the Setup repository to get started.

Free Trial

If you'd like to play around with Spacelift, you can just sign up on the homepage, and it'll automatically provision a trial account for you with all the features of Spacelift available.

If you have any compliance requirements, we're SOC2 Type 2 certified.

Book a Product Demo

Book a free demo with one of our engineers to understand Spacelift capabilities, ask questions and see whether we’re the right fit for you.

Solving the Diamond Problem with a Spacelift Trigger policy

Ryan Cartwright — Thu, 09 Dec 2021 06:20:36 +0000

The "diamond problem" (sometimes referred to as the "Deadly Diamond of Death") is an ambiguity that arises when two classes B and C inherit from A, and class D inherits from both B and C.

It is called the "diamond problem" because of the shape of the class inheritance diagram in this situation.

Before we solve this problem, let's walk through some building blocks so we can create complex workflows using trigger policies.

Purpose

Frequently, your infrastructure consists of a number of projects (stacks in Spacelift parlance) that are connected in some way - either depend logically on one another, or must be deployed in a particular order for some other reason - for example, a rolling deploy in multiple regions.

Enter trigger policies. Trigger policies are evaluated at the end of each stack-blocking run (which includes tracked runs and tasks) and allow you to decide if some tracked Runs should be triggered. This is a very powerful feature, effectively turning Spacelift into a Turing machine.

All runs triggered - directly or indirectly - by trigger policies as a result of the same initial run are grouped into a so-called workflow. In the trigger policy you can access all other runs in the same workflow as the currently finished run, regardless of their Stack. This lets you coordinate executions of multiple Stacks and build workflows which require multiple runs to finish in order to commence to the next stage (and trigger another Stack).

Data input

The schema of the data input that each policy request will receive can be found here.

Use Cases

Since trigger policies turn Spacelift into a Turing machine, you could probably use them to implement Conway's Game of Life, but there are a few more obvious use cases. Let's have a look at two of them - interdependent Stacks and automated retries.

Interdependent stacks

The purpose here is to create a complex workflow that spans multiple Stacks. We will want to trigger a predefined list of Stacks when a Run finishes successfully. Here's our first take:

package spacelift

trigger["stack-one"]   { finished }
trigger["stack-two"]   { finished }
trigger["stack-three"] { finished }

finished {
  input.run.state == "FINISHED"
  input.run.type == "TRACKED"
}

Here's a minimal example of this rule. But it's far from ideal. We can't be guaranteed that stacks with these IDs still exist in this account. Spacelift will handle that just fine, but you'll likely find if confusing. Also, for any new Stack that appears you will need to explicitly add it to the list. That's annoying.

We can do better, and to do that, we'll use Stack labels. Labels are completely arbitrary strings that you can attach to individual Stacks, and we can use them to do something magical - have "client" Stacks "subscribe" to "parent" ones.

So how's that:

package spacelift

trigger[stack.id] {
  stack := input.stacks[_]
  input.run.state == "FINISHED"
  input.run.type == "TRACKED"
  stack.labels[_] == concat("", ["depends-on:", input.stack.id])
}

The benefit of this policy is that you can attach it to all your stacks, and it will just work for your entire organization.

Can we do better? Sure, we can even have stacks use labels to decide which types of runs or state changes they care about. Here's a mind-bending example:

package spacelift

trigger[stack.id] {
  stack := input.stacks[_]
  input.run.type == "TRACKED"
  stack.labels[_] == concat("", [
    "depends-on:", input.stack.id,
    "|state:", input.run.state],
  )
}

Now, how cool is that?

Automated retries

Here's another use case - sometimes Terraform or Pulumi deployments fail for a reason that has nothing to do with the code - think eventual consistency between various cloud subsystems, transient API errors etc. It would be great if you could restart the failed run. Oh, and let's make sure new runs are not created in a crazy loop - since policy-triggered runs trigger another policy evaluation:

package spacelift

trigger[stack.id] {
  stack := input.stack
  input.run.state == "FAILED"
  input.run.type == "TRACKED"
  is_null(input.run.triggered_by)
}

Note that this will also prevent user-triggered runs from being retried. Which is usually what you want in the first place, because a triggering human is probably already babysitting the Stack anyway.

Diamond Problem

The diamond problem happens when you're stacks and their dependencies form a shape like in the following diagram:

          ┌────┐
       ┌─►│ 2a ├─┐
       │  └────┘ │
┌───┐  │         │   ┌───┐
│ 1 ├──┤         ├──►│ 3 │
└───┘  │         │   └───┘
       │  ┌────┐ │
       └─►│ 2b ├─┘
          └────┘

Which means that Stack 1 trigger both Stack 2a and 2b, and we only want to trigger Stack 3 when both predecessors finish. This can be elegantly solved using workflows.

First we'll have to create a trigger policy for Stack 1:

package spacelift

trigger["stack-2a"] {
  tracked_and_finished
}

trigger["stack-2b"] {
  tracked_and_finished
}

tracked_and_finished {
  input.run.state == "FINISHED"
  input.run.type == "TRACKED"
}

This will trigger both Stack 2a and 2b whenever a run finishes on Stack 1.

Now onto a trigger policy for Stack 2a and 2b:

package spacelift

trigger["stack-3"] {
  run_a := input.workflow[_]
  run_b := input.workflow[_]
  run_a.stack_id == "stack-2a"
  run_b.stack_id == "stack-2b"
  run_a.state == "FINISHED"
  run_b.state == "FINISHED"
}

Here we trigger Stack 3, whenever the runs in Stack 2a and 2b are both finished.

You can also easily extend this to work with a label-based approach, so that you could define Stack 3's dependencies by attaching a depends-on:stack-2a,stack-2blabel to it:

package spacelift

# Helper with stack_id's of workflow runs which have already finished.
already_finished[run.stack_id] {
  run := input.workflow[_]
  run.state == "FINISHED"
}

trigger[stack.id] {
  input.run.state == "FINISHED"
  input.run.type == "TRACKED"

  # For each Stack which has a depends-on label,
  # get a list of its dependencies.
  stack := input.stacks[_]
  label := stack.labels[_]
  startswith(label, "depends-on:")
  dependencies := split(trim_prefix(label, "depends-on:"), ",")

  # The current Stack is one of the dependencies.
  input.stack.id == dependencies[_]

  finished_dependencies := [dependency | 
                                       dependency := dependencies[_]
                                       already_finished[dependency]]

  # Check if all dependencies have finished.
  count(finished_dependencies) == count(dependencies)
}

Are you ready to create complex workflows using trigger policies?

Start your 30 day free trial of Enterprise today or book a demo if you need to dive in deeper with our engineers.

Original Source: https://docs.spacelift.io/concepts/policy/trigger-policy

Spacelift Policy-as-code Introduction

Ryan Cartwright — Tue, 07 Dec 2021 08:58:30 +0000

Policy-as-code is the idea of expressing rules using a high-level programming language and treating them as you normally treat code, which includes version control as well as continuous integration and deployment. This approach extends the infrastructure-as-code approach to also cover the rules governing this infrastructure, and the platform that manages it.

Spacelift as a development platform is built around this concept and allows defining policies that involve various decision points in the application. User-defined policies can decide:

who gets to log in to your Spacelift account and with what level of access;
who gets to access individual Stacks and with what level of access;
how Git push events are interpreted;
which Runs and Tasks can be started;
which changes can be applied;
which one-off commands can be executed;
what happens when blocking runs terminate;

You can refer to this section to learn more about commonalities and differences between these policies, or to the dedicated article about each policy to dive deep into its details.