Forem: Sam Nixon

Understanding the Backstage System Model

Sam Nixon — Thu, 12 Dec 2024 14:21:20 +0000

The Backstage Internal Developer Portal is, at its heart, a software catalog. As a catalog, Backstage relies on a structured System Model to represent and organize individual items, in order to make it easier to find the information development teams need. When you are setting up or running Backstage you’ll often want to tweak this Model (or make wholesale changes to it) to make it fit your organization.

In this blog we’ll explore the Backstage System Model and how you can extend it if you need to.

Why do we need a system model?

Catalogs require at least some structure. If you don’t have a common taxonomy for how to describe each element inside it then it lacks coherence, like a library with no labels on the shelves (or worse yet, contradictory labels). You could pour in all of your various repositories, components, gateways, resources and clusters into a catalog and it will closely resemble a giant blob of nothing.

In a Catalog, information needs to be sorted to have value. Decisions need to be made about what gets included and what does not, and you need an idea of what goes where - how things are categorized now and how they should be categorized in the future.

The Basics

The Backstage data model is made up of nodes ("entities") and edges ("relationships").

Entities

The Backstage data model is built around "entities." Entities are the core units within the Backstage catalog that represent various elements of your software ecosystem.

Each entity is defined via metadata (name, description, labels etc), spec (custom properties), and relations (connections with other entities). In OSS Backstage this information is often piped into Backstage via YAML files that adhere to Backstage's entity specification. Sometimes entities can also come from "Providers" which provide the entity from some source of truth (i.e. Users and Group entities from Okta)

This model allows teams to maintain a structured, discoverable Catalog by distributing the load across every team who owns part of the Catalog.

Friction Warning:

Backstage advocates for distributed ownership (i.e. each team owns the information in the Catalog that represents the software that it owns) so it can be tricky to update your model and change it over time. For example, if you wanted to replace a Kind all of the various teams would need to update their catalog files. To get around this, a lot of self-hosted Backstage users have built API-based methods for mass updates.

Kinds

Entities are grouped into Kinds. Kinds are like a aisle at a supermarket - everything within it is broadly cohesive and organised around similar principles.

Kinds have a schema and they require a processor to correctly ingest them into the Catalog.

You get some core Kinds out-of-the-box with Backstage, like:

Domain: Defines larger business domains, organizing systems and components
System: Higher-level abstraction representing a collection of components working together
Component: Represents deployable units like services, websites, or libraries)

Friction Warning:

In OSS Backstage you can extend existing Kinds or write new Kinds to include whatever you’d like, but you need to build or modify a processor each time. That means writing code.
You will also need consider the long-term impact of a new Kind. You’ll likely be supporting that Kind for a long time unless you want to deprecate it and force entities that use that Kind to fail.

Types

Kinds have Types, allowing grouping within these larger buckets.

Types can be defined on-the-fly. Nothing special is needed to make Types work, any team can create a new Type just by articulating it in their catalog-info.yaml file.

Friction Warning:

This can lead to a Cambrian explosion of Types, so you may want to introduce some constraint there. Validation of Types is common.
Annoying errors can creep into Types (i.e.Website and Wesbite) unless you’re validating them in some way.

Relationships

Relationships exist between entities to provide the connective tissue of the Backstage Catalog.

Each Kind has a preset series of permissible relationships that are built when the processor runs for that Kind.

For example, a simple Component might have some API relationships and dependencies defined:

```apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
name: artist-web
description: The place to be, for great artists
spec:
type: website
lifecycle: production
owner: artist-relations-team
system: artist-engagement-portal
dependsOn:
- resource:default/artists-db
dependencyOf:
- component:default/artist-web-lookup
providesApis:
- artist-api




## The Core System model

Out of the box, Backstage comes with a lot of built-in Kinds with attendant relationships so you can get started as quickly as possible. 

Some Kinds, like software templates and Locations are effectively atomic and compartmentalised away from other Kinds. The remainder are tied to how the Catalog is built and used to represented entities.

![Backstage System Model](//images.ctfassets.net/hcqpbvoqhwhm/2c89CI4rKHDNmWvCM2IHhM/37022568f4168c497956da8d9615511a/software-model-entities.drawio-3ce7f43dd236c3934209fde8f21a4d9e.svg)

They in effect represent "The Spotify Way" to model software. That’s not for everyone and won’t necessarily work perfectly for you. 

If that’s the case, you have two options:

- `Force it a little`: aka shoehorn your existing concepts into Spotify’s version. This works in a lot of cases, but is necessarily a compromise.
- `Re-model`: if that doesn’t do the trick, you need to get to work remodeling Backstage entity Kinds and types to fit your needs. Some can be done without code changes, but some need you to get your hands dirty.

# Going beyond the basics and extending the Backstage System Model

The Backstage framework is designed to be highly extensible, allowing you to modify or add new Kinds, Types, and Relationships based on the requirements of your organisation. 

That said, there are a few things you need to think about when extending the model:

### 1. No code extensibility

Backstage has flexibility baked on for a large degree of software definition. Using Types or built-in relationships handles for most situations when you want to model your software inside the Backstage System model. 80-90% of the time this will do the trick, but will often come with some degree of compromise. For example, let’s say you want to articulate `Value Streams` as a top level concept, but have to make do with `Value Streams` being a Type associated to the `Domain` Kind. It’s imperfect, but it’ll do in a pinch.

At Roadie, we evaluate and extend the System Model for our customers regularly. That works a lot of the time, but sometimes customers have niche requests that we don’t feel would benefit all our users. This is non-optimal. We want to customers the freedom to extend the model without talking to us or writing code. To achieve that we’re building a fully self-serve, no-code UI for dynamically generating Kinds and defining a system model that can be as arbitrary as you’d like: if you want a Kind called `purple-monkey-dishwasher` you should be able to have one. 

### 2. Extending the framework using code

Backstage is built around [providers](https://backstage.io/docs/features/software-catalog/external-integrations/) and [processors](https://backstage.io/docs/features/software-catalog/external-integrations/#custom-processors). Providers pull data in, processors manipulate and validate that data to build the Catalog entities and relationships. 

You can create wholly new providers to handle the ingestion of data from sources not currently handled by Backstage. The Backstage community has built  a lot of Providers over the years, but they may require tweaks to fit your specific use-case. For example, Roadie has rebuilt the GitHub provider to use webhook-based ingestion because the size of Catalog we habitually deal with break the GitHub rate limits

You can also modify processors for existing Kinds. For example to extend the list of allowed relationships between Kinds you need to tweak those processors.

You can also create wholly new processors to define new business logic or processes for manipulating and validating that data when you create a new Kind. Going back to the Value Stream example, now you can differentiate `Value Stream` from `Domain` and allow the Kinds to deviate usefully from one another. Maybe they each need different allowed relationships, or they’ll build their entities differently: the choice is yours.

### 3. Data

In the [out-of-the-box OSS Backstage model](https://backstage.io/docs/features/software-catalog/system-model/) the data for the system model comes from yaml files. This follows the GitOps model, where changes are made in git-tracked repositories and then ingested by other systems (in this case, the Backstage Catalog).

That means if you want to change or update your model you need to change all those files. That in turn means that opening PRs against every repos which contain a relevant yaml file. This is often a large undertaking, adding significant friction. That’s why most high-volume users of OSS Backstage have built API- and database-based mechanisms to do mass updates. Roadie has two: the Decorator UI and APIs to do a variety of different update patterns (idempotent updates to sync data from a source of truth into Backstage, or just pushing in whole entities via the Roadie Entities API).

# Levers to pull when extending the model

Below are some common methods for extending the Backstage data model:

### 1. **Custom Annotations**

Difficulty: Trivial

- **Why**: If you need to add metadata specific to your organization (like security labels, compliance levels, etc.), you can define custom annotations.
- **How**: Annotations are added as key-value pairs within the `metadata.annotations` field in your YAML definitions. These annotations can be used to enhance search functionality, create custom views, or provide additional context.
- **Example**: Adding `security-level: high` as an annotation for services that handle sensitive data allows you to quickly filter and prioritize compliance and monitoring for these services.

**References**:

- [Backstage Annotations Documentation](https://backstage.io/docs/features/software-catalog/well-known-annotations/#annotations): Documentation on creating custom annotations to extend metadata.



    ```yaml
    apiVersion: backstage.io/v1alpha1
    kind: Component
    metadata:
      name: fraud-detection-model
      description: "AI model for fraud detection"
      annotations:
        security-level: high

    ...
    ```



### 2. **Custom Types**

Difficulty: Easy

- **Why**: In cases where the existing entity types (Component, API, etc.) do not fit your specific resources, you can create custom entities.
- **How**: Define a new entity type in any valid catalog-info.yaml. This simple involves adding a new type to the `spec.type` in the YAML file.
- **Example**: Suppose you have machine learning models as a core resource in your project. You could define a new `model` type.

**References**:

- [Roadie Kinds and Types documentation](https://roadie.io/blog/kinds-and-types-in-backstage/) talks a lot about how to use Types without introducing problems



    ```yaml
    apiVersion: backstage.io/v1alpha1
    kind: Component
    metadata:
      name: fraud-detection-model
      description: "Machine learning model for fraud detection"
      annotations:
        security-level: high
    spec:
        type: model
      version: "1.0"
      trainingDataset: "transactions-v1"
      accuracy: "95%"
    ```



### 3. **Modifying Existing Kinds to Add Custom Relations**

Difficulty: Normal

- **Why**: Relationships between entities help you capture dependencies, ownership, and team structures within your catalog. If your use case involves additional relationship types, custom relations can improve representation.
- **How**: Modify the relevant processor for a given Kind to enable new types of relationships to be built for that kind. Then define relations within the `spec.relations` section of the YAML file.
- **Example**: Suppose you want to track models associated with data sources. You could create a custom relation `usesDataFrom`, linking ML models to the Resource entities that document data sources they rely on.

**References**:

- [Roadie Kinds and Types Documentation](https://roadie.io/blog/kinds-and-types-in-backstage/): Provides practical examples of defining and extending Kinds.



```yaml
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: fraud-detection-model
  description: "Machine learning model for fraud detection"
  annotations:
    security-level: high
spec:
    type: model
  version: "1.0"
  trainingDataset: "transactions-v1"
  accuracy: "95%"
  relations:
  - type: usesDataFrom
    targetRef: resource:exampleorg/some-data-source
    target:
      kind: resource
      namespace: exampleorg
      name: some-data-source

4. Creating Entirely New Custom Kinds

Difficulty: Normal / Hard

Why: When the System Model cannot adequately encapsulate how you build software or the relationships between various parts of your organisation, you will need to build a custom Kind.
How: Write a new processor for that Kind and define a custom schema for that Kind. This ensures all entities adhere to required fields, valid types, and constraints, providing an additional layer of validation. Then add new catalog-info.yaml files for the new Kind to relevant resources, or modify existing catalog-info.yaml files.
Example: For the MLModel entity, you could create a new Kind to represent that in your System model. Using that new Kind you could then model relationships as version, trainingDate, and accuracy.

References:

Backstage JSON Schema Documentation: Explains how to define and enforce custom schemas.

Conclusion

Backstage is an extremely flexible framework for modelling software and once the building blocks and options are understood it’s simple enough to fully customise the model.

Useful links:

Backstage System Model: official docs and a good starter diagram for how entities in the Catalog interact.
Backstage Entities: official docs on the lifecycle of entities
Backstage Relationships: official docs on how relationships work inside Backstage
Modelling software in Backstage: Roadie blog from 2021 about how to model software in Backstage using the core system model. This still represents a great primer on the out-of-the-box system model and how you could use it.

How to Define Engineering Standards (with Backstage)

Sam Nixon — Sat, 28 Sep 2024 16:12:11 +0000

At Roadie we don’t claim to be experts in writing software standards - we’re done it ourselves, both for Roadie and before as part of other startups, scaleups and large companies, but we don’t claim to be world-beaters at it. What we can claim is that we’ve seen many, many companies go through the journey to create standards and then apply them and we have seen what works.

This post originally appeared on the Roadie blog over at https://roadie.io/blog/how-to-define-engineering-standards/

Why standardise?

Defining and adopting engineering standards is essential for organizations as they scale.

Teams involved in Platform initiatives normally find themselves as the first intrepid explorers in this territory for larger organizations.

Building tooling for multiple different teams and departments requires consistency and a coherent set of practices. Only then can teams coordinate, share and build scalable, maintainable, and secure software together.

Without standardization, answering basic questions becomes impossible and progress is painfully slow, a problem that often hits home at times of peak stress. How can you know which teams operate publicly facing software that has critical vulnerabilities if only half of the teams are using a dependency scanning tool? How can you roll out a new security requirement when engineering teams are each using one of 5 different security tools?

Defining Engineering Standards

Engineering standards are formal guidelines that outline how code should be written, how systems should be designed, and how processes should be executed. These standards ensure that all engineers are working toward the same quality benchmarks.

That means you need consensus between teams about what exactly should be in those standards. In order to do that, even before you decide what the standards are that you’d like to focus on, it’s good to have a plan for how you can agree together on what they should be.

Strategies for Agreeing on Engineering Standards

Getting a team to agree on a set of engineering standards can be challenging but is crucial for their success. Here are some strategies to help facilitate agreement:

Start with your own SDLC - if you have one - and/or Industry Best Practices:
- Software Development Lifecycle and Production Readiness documents often effectively contain a lot of standards recommendations. They’re extremely useful as an input into a formal set of software standards and the two should be synced closely together.
- Use industry standards like OWASP for security or WCAG for accessibility as a baseline. This helps reduce subjective debate by relying on well-known benchmarks. For example, it’s hard to argue that secure logging isn’t important when it shows up prominently in the OWASP table.
Collaborate across teams and functions:
- Involve engineers, product managers, and operations teams in defining the standards.
- Run workshops where everyone can voice their opinions, then converge on a decision.
Appoint a champion:
1. Usually this is a member of engineering leadership who is responsible for driving this process and the eventual rollout.
2. Occasionally this can be a group though, like an Architecture guild.
Gradually implement:
- Start small, either:
  - A minimal set of standards and build from there as the team gets more comfortable. For example, you can start by enforcing code formatting standards and then gradually add performance or security checks.
  - A full set of standards with only a few initial checks so that the team can get comfortable with the whole suite of standards
- These strategies allow teams to give feedback and for the standards to evolve before they are fully enforced
Regularly review and update:
- An engineering standards document should never stand still.
- Once the standards are set, encoded them into your systems.
- Hold regular reviews (e.g., fortnightly or monthly) to review progress of each team
- Create a regular cadence of review for the standards themselves, based on team feedback and new technology trends and recommit. Once a year is often enough here.
- As part of these reviews, use data to demonstrate the value of each standard (e.g., reduced production errors, improved system uptime).

Common Standards

To make this concept concrete, let’s consider some common standards teams might define (nb: this is by definition a non-exhaustive list):

Logging and Monitoring: Log levels, message formats, error tracking, use of a centralized tool, use of alerts.
Security: Authentication, authorization, encryption, secure coding practices, and dependency vulnerability management.
Performance: Response times, load management, and scalability.
Reliability: Redundancy strategies, failure handling, backup and recovery.
Code Quality: Style guides, review processes, formatting rules, and readability standards.
Documentation: API documentation, code comments, and README files.
Testing: Code coverage, test automation, and test environment standards.
Version Control: Branching strategies, commit message guidelines, and pull request processes.
Deployment: Continuous Integration/Continuous Deployment (CI/CD) pipelines, rollback procedures, and environment configurations.
Accessibility: WCAG 2.2 guidelines, color contrast, text-to-speech, keyboard navigation

You don't want to over-elaborate at this point. It is important to end up with ~8-10 different areas to focus on.

Nice vocabulary to use when setting standards

Must. Used to define mandatory items. i.e. A service must use a logger
Should. Used to define items which are reasonably expected to exist. If a services choses not to adopt this standard, the expectation is that they justify why not.
May / Could / Will. Used to define items which are more aspirational or for services that are consider mature.

An example: AcmeCorp.com

Let’s imaginee AcmeCorp.com are a well-known platform selling books, clothes, food, laptops, paddling pools and power tools around the world. They’re an anything store, if you will.

Availability and reliability are key to their business, so they spend a lot of time thinking about how to measure and improve that for the software they build.

Previously, teams would simply assert that their service was reliable, performance, secure etc, but aside from anecdotally looking at the past weeks/months/years to validate that assertion, it was hard to prove or disprove. It was also hard to compare across services.

To help that, a cross-functional group at AcmeCorp.com have agreed a series of standards that they believe will ensure their service stand up to considerable load during peak periods, and that if outages or incidents do happen that they’ll recover quickly.

Area	Standard
Monitoring	Health checks for critical components must be defined and an ideal state determined.
	Service state must be constantly observed and recorded and dashboards should be created to show this data.
	Monitoring should have metrics that describe how effective a service is. These metrics are available and easily viewable on a dashboard.
	Events should be exported and/or sampled and collected in addition to other metrics
Availability	Service availability must be determined programmatically.
	Expected and unexpected behavior for a given service must be defined in tests and alerts.
	Basic SLIs should be defined and used to calculate SLO targets. This should include the number of number of good events / total number of events are being recorded
	SLOs should be actively measured, calculated, and displayed in a dashboard
	Error budgets may established and a policy outlining what to do when service runs out of budget is established.
	SLOs (and error budget policy where appropriate) should be documented in a prominent location where teams and stakeholders can easily review.
Logging	Logs must show startup, shutdown, and errors.
	Logs must have have a rotation and retention policy defined.
	Logs from all hosts must be sent to a centralized location.
	Logging pipeline must be resilient to transient failures and should be fully recoverable when ingestion returns to a healthy state.
Alerting	Basic health checks must be attached to alerts when failing.
	A dashboard must display all alerts currently firing.
	The body of any alert must contain information that is be needed to diagnose and fix the problem.
	An official on-call rotation for high-priority alerts must be configured and activated.
	High-priority alerts should tuned such that they don't fire outside of business hours unless necessary. If resolution of an issue can wait until business hours, it should not page the on- call engineer.
	High-priority alerts should be triggered only for urgent, actionable issues that require a human's intervention.
Scalability	Operating manuals for service scaling must be up to date and consumable by newly onboarded or tangentially-familiar engineers.
	Service must handle unexpected increases in load without manual effort, up to a known threshold.
	Unexpected increases and decreases in load must be handled automatically.
	Unexpected increases in load above a known threshold may be handled automatically.
	Owners of a service may run regular scaling exercises to test scaling assumptions.
	Service may be able to deprioritize features and load when needed.
Resiliency and Recovery	Run books must exist that outlines steps of recovering from loss of capacity.
	Owners should have conducted testing on outages to validate recovery run books and quantify performance degradation.
	Owners should demonstrate manual recovery is possible with minimal performance degradation (within established threshold)
	Owners may demonstrate automatically recovery is possible with minimal performance degradation (within established threshold)

Breaking Standards Down Into Scorecards and Checks

Once the standards are defined, they need to be actionable, measurable, and concisely group so that teams can understand them. This is where checks and scorecards come into play.

Scorecards

Scorecards allow teams to measure in only a few data points how well they are adhering to their engineering standards across a project or organization.

Scorecards should flow naturally from your standards and be fairly simple to define. Name them things that align with those standards and are comprehensible as a bucket of actionable checks against those standards.

For example:

Security is a good, simple, easy to understand name. If you wanted to create levels for your scorecards to have some sense of progression, you could say Security - Level 1
Secure Coding Standards might be a good option if you wanted to go to a more granular level with your scorecards.

Try and end up with ~10.

Checks

For each scorecard/standard, you need to break it down into one or more specific checks. A "check" is a verifiable condition that can be automated or manually enforced.

Just like Scorecards, Checks should be named things that are comprehensible but crucially should also be actionable.

For example:

Node version should be >18 is a clear true/false statement about what the expectation is for a given service that uses Node.js. It is also clear what needs to be done in order to pass that check.
Similarly CODEOWNERS should be enabled draws an even more direct line to what needs to be done for a given service to pass a check.

At Roadie we use our Tech Insights plugin to build these checks - the backend for which we also open sourced.

Whether you’re using Roadie, OSS or hand-rolling these checks, it’s important to have an idea of what a computationally enforceable check would look like for each of your standards.

Automating and visualising standards

Last but not least, you need some way to repeatedly check standards are being adhered to.

This usually comes in two forms:

You can programatically check to see whether documentation exists. This is often in the form of a runbook at a given path, i.e. /docs/runbooks/recovery
A third party tool is used to capture data that can then be interrogated programmatically. For example, this can be something as simple as an SLO existing in Datadog for a given service.

Both data sources can be used to confirm at scale whether or not a given service is correctly adhering to a given standard.

Teams no longer have to assert compliance, they simply need to ensure that the evidence is currently surfaced to prove that they comply.

Many scorecarding solutions do exactly this for you, without the need for teams to individual wire up different systems. For example, Roadie customers use Tech Insights plugin to provide standardised, automated checks across their entire software catalog with minimal or no intervention from individual teams.

Returning to AcmeCorp

Using the example of AcmeCorp.com again, let’s take one of their areas are turn it into a Scorecard with a series of checks. They use Datadog for their dashboards and Sentry for their logging so they can both provide sources of truth for their checks.

Scorecard	Underlying Standard	Checks	Data source
Monitoring	Health checks for critical components must be defined and an ideal state determined.	Service has >1 health checks defined	Repo file that contains healthcheck test results
	Health checks for critical components must be defined and an ideal state determined.	Healthcheck test results return current status codes	Repo file that contains healthcheck test results
	Service state must be constantly observed and recorded and dashboards should be created to show this data.	Service has a Datadog dashboard to record service health	Datadog
	Monitoring should have metrics that describe how effective a service is. These metrics are available and easily viewable on a dashboard.	>1 service metric is defined in Datadog	Datadog
	Monitoring should have metrics that describe how effective a service is. These metrics are available and easily viewable on a dashboard.	Datadog metric monitor is configured	Datadog
	Events should be exported and/or sampled and collected in addition to other metrics	>1 event has been sent to Sentry in the last day	Sentry

Making a fix easy to implement

The final stage of implementing engineering standards is to make adherence as simple and easy as possible. If a check or scorecard failure is hard to achieve then teams will take longer to resolve it, the rate at which they resolve it will be lower, and standards will ultimately suffer.

Ask yourself:

How many steps does it take to bring a service into compliance for a given standard? How can that be reduced?
How can common or shared systems be leverage to help multiple teams into compliance? Using the AcmeCorp example above, a templated Dashboard for Datadog could help all teams skip design and production steps when setting up a Monitoring dashboard.
How can the overall cycle time from error to resolution be reduced? Can common fixes be added to shared repositories or How To guides be written to help teams?

Where possible, implement quick fix options

At Roadie we use the Backstage scaffolder to automate many of the fixes for our scorecards. To take a simple example, one of our engineering standards is that branch protection must be enabled on any repository we create. If a service is linked to a repository without branch protection it fails that check. To resolve it, we have a 3-second Scaffolder template that can modify the GitHub settings associated with the repository. The only thing the team needs to do is look at the check and click a button.

Conclusion

Defining engineering standards is critical for ensuring that software systems are built and maintained with quality, security, and scalability in mind. Breaking these standards down into checks and scorecards allows teams to monitor compliance and ensure continuous improvement. By following a collaborative approach to defining standards and making them measurable, teams can streamline their development processes and produce more reliable software.

Whatever you are focusing on, the key to success lies in making standards actionable, measurable, and adaptable.