Forem: David Tuite

Why Hybrid is Best for IDPs

David Tuite — Wed, 04 Dec 2024 12:01:18 +0000

The Internal Developer Portal (IDP) market is only 5 or 6 years old, but we’ve seen a huge amount of growth in that time, and many different options emerging.

According to Gartner, IDPs are reported as the most frequently piloted technology in their 2022 - 2024 Technology Adoption Roadmap Survey, and 75% of organizations with platform engineering teams will provide IDPs by 2026.

With this growth, 3 main categories of Internal Developer Portal products have emerged:

Open-source IDPs are exactly what you think. The code is open-source and you host and maintain it yourself.
Proprietary IDPs are typically software as a service startups funded by venture capital. You purchase access to them just like you purchase access to PagerDuty or GitHub Enterprise.
Hybrid IDPs are a combination of open-source and proprietary. They’re built on an open-source foundation, but come with commercial support and proprietary features.

In this article, I’m going to dive into each category one by one, and learn the pros and cons of each.

I’ll also explain why I believe the Hybrid model brings the best of both worlds. The community, extensibility and lack of vendor lock-on of the open-source model, alongside the low cost, ease of use, and fast deployment of the proprietary model.

Open-source

While technically not an IDP, and technically not the only open-source IDP, you can’t talk about open-source IDPs without the conversation focussing on Backstage.

Backstage was open-sourced in early 2020 by Spotify. It was a rewrite of their internal IDP that they’d been using for years. It’s not an IDP because it’s actually a set of TypeScript libraries that developers can combine together to build an IDP, but it has still captured a huge amount of attention in the IDP space.

When Backstage launched, it was often compared to Hygieia from Capital One, which predates it by almost 4 years. Even after Backstage appeared, a third contender emerged with a strong start. Ride-hailing company Lyft threw their hat into the ring when they released Clutch a few months after Backstage was open-sourced. Clutch describes itself as "an extensible platform for infrastructure management”.

in 2024, Clutch popularity seems to have stalled, Hygieia is deprecated, and the reality is that Backstage dominates the open-source IDP market. Although not a perfect metric by any means, GitHub stars show the difference in popularity of each tool.

Benefits of the open-source model for IDPs

In many ways the open-source model is perfect for building an Internal Developer Portal. IDPs are a window into all of the tools that developers use. They’re the venerable “single pane of glass”.

Developers use a lot of tools though, and the developer tool landscape is thousands of tools strong and growing. Even the Cloud Native Landscape alone has more than 1,000 tools in it’s ecosystem.

How can a single IDP integrate with and plug-in to all these tools? Well, by creating an open-source community who will help with the effort.

Huge community

This is exactly what Backstage has accomplished. The Backstage community is huge and engaged.

Backstage was the top end-user contributed CNCF project of 2023, with more than 4,000 contributions from end user companies.

Benefits of the open-source model for IDPs

Developers use a lot of tools though, and the developer tool landscape is thousands of tools strong and growing. Even the Cloud Native Landscape alone has more than 1,000 tools in it’s ecosystem.

How can a single IDP integrate with and plug-in to all these tools? Well, by creating an open-source community who will help with the effort.

Huge community

This is exactly what Backstage has accomplished. The Backstage community is huge and engaged.

Backstage was the top end-user contributed CNCF project of 2023, with more than 4,000 contributions from end user companies.
There are 17,000 people in the Backstage Discord channel, with dozens and dozens of questions being asked or answered every day.
Backstage has a deep partner ecosystem. Whether you’re a Fortune 50 enterprise or a Series C startup, there’s a partner who can help you deploy and use Backstage.
Backstage has official support from leading DevTools companies. Companies like Snyk, PagerDuty, Dynatrace and even AWS have released officially supported plugins for Backstage.

This vibrant community means that you can always get help with Backstage. The other benefit is the extensibility and the number of integrations available.

Huge number of integrations

The result of this large community is that Backstage is installed and adopted at thousands of organizations. These organizations use a diverse set of engineering tools, and many adopters have built and open-sourced Backstage plugins for these tools.

The Backstage plugins directory has more than 200 plugins available, so you can be fairly confident that there are plugins available for the tools you use.

Highly extensible

Underlying all these plugins is a framework which is designed from the ground up to be maximally extensible. The docs say:

Backstage is a single-page application composed of a set of plugins.

Our goal for the plugin ecosystem is that the definition of a plugin is flexible enough to allow you to expose pretty much any kind of infrastructure or software development tool as a plugin in Backstage.

Practically, this means you can customize and extend your Backstage install to your specific requirements. If you need to build your catalog in Gerrit rather than GitHub, then you can swap out the GitHub integrations and replace them. The same concept applies to most parts of Backstage.

By coupling this technical extensibility with a permissive Apache 2.0 license and a mature governance model, adopters can make wholesale changes to Backstage if they like.

The result is that many organizations have customized Backstage heavily. Take a look at this screenshot of Sunrise, the Backstage-based IDP that Zalando created, as an example.

This completely custom UI shows the progress of changes as they flow through CICD on their way to production. Very little of what you see here comes out of the box with Backstage. Instead, it’s wired together from the basic building blocks of Backstage plugins.

Lack of vendor lock-in

Because Backstage is open-sourced and backed by large end user companies and the CNCF, adopters can feel comfortable that it will be around for a long time, and will continue to meet their needs into the future.

The IDP market is still very early. The majority of the players have been founded in the past 5 years. We will likely see some consolidation of this market as players get acquired and wound down. Any customers of these companies will be forced to migrate to a different solution.

We’ve seen this play out in the CICD market already. There was a time when Travis CI was a dominant provider of continuous integration. In 2019, Travis CI was acquired by private-equity firm Idera. One month later, layoffs began. In early 2020, Travis CI stopped providing free support for open-source products, and in late 2020 the pricing changes began.

Companies can avoid this fate in the IDP market by building on an open-source project instead of a commercial vendor with it’s own data model and lock-in.

Drawbacks of the open-source model of IDPs

It’s expensive to build

Backstage is a larger undertaking than most people realize before they get into it. To quote Gartner in their Innovation Insight for Internal Developer Portals report:

Gartner inquiries indicate that Backstage implementations may require substantial effort in standing up the service.

In order to replicate something approaching the Sunrise portal mentioned above, companies should expect to allocate 2 to 5 engineers to the project for a number of years.

Don’t believe me? Zalando said it themselves in this presentation they did at the Autodesk Developer Productivity Summit.

All that I’ve showed you was done through the contributions of many teams, but the core team was three engineers and a engineering manager.

That team has been working on Sunrise since 2020.

And that’s where my team is focussed on, actually like, the discovery part of our application landscape, and where we actually decided to, a few years ago in the beginning of the Backstage era in 2020 when Backstage was released, to actually onboard ourselves into Backstage.

Let’s do the math. Assume an engineer or engineering manager costs $250k per year, therefore the team of 4 costs $1 million per year. 4 years have passed since 2020, so they’ve spent $4 million in total on their core Backstage team. Add in the “contributions of many teams” and we could easily be talking about a $6 million outlay.

Now you’re probably saying “yeah but we’re not as big as Zalando”, but trust me, even small companies will spend a half a million dollars pretty easily.

It requires uncommon skills

Backstage is typically owned and managed by platform teams or DevOps teams. These teams are probably skilled in YAML and kubernetes-native languages like Go.

Backstage is written in TypeScript. The backend runs on NodeJS, and the frontend is written in React.

Organizations who are considering self-hosted Backstage should consider whether or not they have the skills to develop in these languages. As mentioned above, Backstage is a set of libraries that users combine together to make a developer portal. That means you need to write TypeScript to make use of it. You need to understand frontend development, HTML and CSS to customize it. It may not make sene make sense to build up these skills in a Platform and Infrastructure organization because they will not be easily transferrable to other projects.

The second set of uncommon skills that a Backstage deployment needs are project management and developer relations. Once it’s live, you need to evangelize it. You have to go out to the rest of the organization and teach them about it and how to use it. Are your platform developers really going to want to do this work?

It’s got a long time to value

The typical Backstage rollout has a few phases. It starts with an initial deployment, which typically takes a few months to get a basic Backstage instance into production. Some parts of Backstage will work straight out of the box at this point. You’ll be able to use the scaffolder to create new software projects from predefined templates, for example.

Other parts of Backstage require an adoption phase. The software catalog is usually one of them. In order to have a complete software catalog, teams must first put their software into it.

In 2023, at BackstageCon North America, I spoke to the experiences of some Backstage adopters who had attempted this feat. I first explained that 60% of the Backstage adopters I had interviewed were attempting to populate their catalog by writing YAML files that contain metadata about their services (these are called catalog-info.yaml files).

I then shared some quotes from adopters who had followed this path

The catalog never really worked for us. We struggled with adoption and rate limiting. Half the info in the catalog was wrong when they first pushed adoption.
2,000 engineers, 2 years experience

The thing we struggled with the most was getting teams to create the YAML file. We spent 6 months actively encouraging teams to register their components with Backstage. After 1 year, only 30 or 40% of the active repos were there.
120 engineers, 1 year experience

Without a lot of product development and project management, this catalog population process takes a long time. There are many Backstage adopters who are at less than 50% catalog completeness after two years with the tool. Even Spotify admit that Backstage adoption rate often stagnates at 10%. You may have dreams of your software catalog becoming a single pane of glass for all of the software development in your organization, but it simply cannot fulfill its purpose if the software is not in there.

Proprietary

At the other end of the spectrum, there are a number of proprietary developer portals like Cortex and Port.

The concept of proprietary developer portals is largely the same as Backstage. They have similar features and target a similar user in the engineering organization. The main differences are in their extensibility. Users can’t edit the code of a proprietary developer portal, and there won’t be a large community who are producing open-source plugins.

Benefits of the proprietary model for IDPs

While a proprietary developer portal may never be as extensible as Backstage, they certainly do have some benefits.

Quicker to get started

Proprietary IDPs will mostly just work out of the box. You don’t have to plan sprints and do research and write code to get some value. They come with admin interfaces where you can set some properties, connect up your tools, and starting using the product straight away. This is likely a better experience for platform engineers who don’t have the time or expertise to deploy and customize Backstage from scratch.

When you do need to configure something, the documentation provided by a proprietary developer portal will likely be more comprehensive and up to date. They’re simply more motivated and able to produce high quality documentation than an open-source community. This makes setup faster and less confusing, and helps to prevent misconfigurations.

User-friendly interface

Proprietary IDPs bring their own user interface (UI). They’re developed from the ground up with design systems in place to help to ensure consistency across integrations and pages. They also have designers and UI engineers working to implement design practices in a coherent way.

While the Backstage core team have done a lot to put the tool in a good place, the reality is that Backstage’s default interface leaves a lot to be desired.

This reality is brought on by the combination of overly familiar Material UI libraries and community contributed code that can be hit-or-miss in terms of how it looks and feels.

Backstages plugins can be contributed by anyone, so there’s no guarantee that they will look good, or look consistent. It’s very likely that different plugins will work completely differently even though they are installed in the same Backstage instance.

More guidance

Proprietary IDPs are more opinionated in how they should be used. When you work with a vendor you should get access to solutions engineering and customer success resources who can help to share lessons from other customers.

Working with the open source can sometimes feel like being handed a box of car parts and asked to build a car. Sure you can figure it out eventually, but you’re going to make mistakes along the way, and it probably won’t be the most efficient path.

Having a partner to work alongside and share lessons is worth a lot, and helps to cut the time to value for your IDP project.

Drawbacks of the proprietary model for IDPs

Vendor lock-in

Proprietary IDPs have their own in-house APIs, their own data model, and their own way of doing things. Many of the options on the market are early stage startups. There’s a good chance some of them will disappear over the next few years. If you’re a customer, your IDP will go with them.

When they do survive, you’ll likely be forced to choose between paying up for price hikes, or completing an expensive migration project to a different solution.

Limited extensibility

It’s not possible to change the code of a proprietary IDP. If there’s something about the way it works that doesn’t suit the needs of your company, your best hope is to contact support and hope that they have the desire and the roadmap space to change it.

Most development organizations have some homegrown tools that have sprung up over the years internally. These tools only exist inside the company they were created, and no IDP vendor will have an out-of-the box integration for them. For these tools, you need to create your own plugins if you wish them to be part of your single pane of glass IDP. Some proprietary IDPs do support custom plugin development, but you’ll need to do so using only the concepts and libraries that the vendor provides, and the plugins you create will not be transferrable to any other solution.

Proprietary IDPs have fewer integrations than the massive Backstage community. Even the most well funded proprietary vendor has 48 plugins listed on their integrations page, less than a quarter of the number that are available for Backstage. No matter how big this vendor grows their team, they won’t be able to compete with the 500+ companies who have contributed to Backstage.

Limited speed of execution

The tech landscape is constantly shifting and evolving. Before serverless we had Kubernetes, before Kubernetes we had virtualization, before virtualization we had bare metal. In between and around these major platform shifts we had and continue to have a massive number of different types of serverless, containerization and virtualization options.

A developer portal is supposed to wrap all of these shifts and technologies in order to give the best possible coverage over your internal engineering landscape. Proprietary IDPs may struggle to capture all of these technology shifts into the future, leading to gaps in the portal that platform teams put in front of their internal users.

Hybrid

Last, but not least, we have the Hybrid model. This model takes an open-source foundation with hundreds of plugins and integrations available, and makes it available in an out-of-the-box format which is easy to get started with.

This is what Roadie is. It’s the only standalone IDP based on Backstage, and it offers the best of both worlds - extensibility and speed of execution.

Huge number of integrations

The vast majority of plugins that have been created for Backstage can work on Roadie. Our integrations library contains 70+ plugins and integrations today, and we’re adding more all the time, in line with customer requests. Once a new open-source plugin is created, it takes us about a day to make it available in Roadie.

The only reason we wouldn’t add a Backstage plugin are that it’s of such low quality that it doesn’t work or provide any value.

This means that Roadie customers can effectively choose from 200+ plugins and integrations that the Backstage community has created and use them on our platform. In fact, we’ve created some of the most popular plugins ourselves and open-sourced them.

Highly extensible

Roadie strives to support as much customization of our IDP as possible. In addition to theming and branding, we support a custom data model, custom Backstage plugins, custom proxies, layouts, scaffolder actions and on and on. You name it, we can probably let you customize it.

This means that you can customize Roadie to meet the specifics of your organization, improving adoption and facilitating ease of use.

No vendor lock-in

Roadie is API compatible with Backstage. The same software metadata schema that works on Backstage will work on Roadie. Custom frontend plugins written for Backstage will also work on Roadie. We’re just supporting and extending the same APIs.

This has two benefits for customers.

They can migrate off self-hosted Backstage onto Roadie very easily. Numerous organizations have already done this. Simply connect us to your source code management tool and we’ll automatically ingest any software metadata files you’ve created.
They can migrate back to self-hosted Backstage very easily. In fact, our Terms of Service dictate that we will give you your data back within 30 days of termination.

User friendly interface

Roadie’s interface is vastly improved compared to the barebones Backstage experience. We’ve worked with our users to streamline how people use the product. This means better out of the box defaults and easier customization.

Inexpensive

In many cases, Roadie is 5 times cheaper than self-hosting Backstage.

People tend not to realize this up front, but the most highly developed Backstage deployments in the world have cost millions of dollars. We talk to companies all the time who have built a team of 5+ engineers around Backstage and run those teams for 3+ years. The most expensive Backstage deployment we’ve seen cost $7.5 million dollars over 3 years.

Roadie doesn’t require such an extensive team to be built around it. We take away all of the operations and deployment effort, and much of the customer support that has to happen. We also give you out-of-the box features like Scorecards and Role-Based-Access-Control that you would otherwise have to build yourself to realize the potential of your IDP.

Lastly, we charge based on usage. You only pay for the engineers who write code which is tracked in the catalog. Everyone else gets to log in for free. This means you can ramp gradually throughout your org, rather than paying for the developement of the entire IDP up front.

Requires few skills

Roadie doesn’t require any TypeScript skills. You just drag and drop plugins where you want them, and configure the product in our administration panels. It’s dead simple. This is perfect for platform teams who are trained on cloud-native technologies. We even have no-code plugins that you can use to display lists or charts inside the UI without writing any code.

If you do need to make something completely custom and you don’t want to write a Backstge plugin yourself, we can sort you out with professional services to suit your needs.

Short time to value

We’ve built a ton of features into Roadie to help shorten the time to value. For example, we help teams build out their catalog quickly and we see customers reaching a high level of catalog completeness within 4 months.

We also provide customer success and guidance to help organizations effect change with their IDP. We take the lessons we’ve learned over the years and use them to guide you to success in as short a time as possible.

Conclusion

We’ve designed Roadie from the ground up with flexibility and easy adoption in mind. We believe engineering organizations should not waste their resources on undifferentiated heavy lifting projects like deploying an open-source IDP. At the same time, they shouldn’t be locked into a proprietary data model or limited in terms of what they can build or the integrations that are available.

It turns out that you can have your cake and eat it too. An Internal Developer Portal can be both easy to use and supported by a massive and growing open-source community. This means that you can focus on unlocking productivity in your engineering organization while knowing you have a rock-solid and flexible foundation beneath.

I’ve also written about the difference between IDPs and the IDP space in general over on The New Stack

Image by MSTORK from Pixabay

The Power of Customization: Making Backstage Work for You with Roadie

David Tuite — Thu, 17 Oct 2024 08:48:37 +0000

Backstage is the most flexible way to build an Internal Developer Portal that exists. The downside of this flexibility is that it can take a lot of effort to use. Roadie’s mission is to take the hassle out of Backstage, but we want to deliver on this goal without preventing our users from customizing Backstage to the fullest extent possible.

This article explains many of the customization options available in Roadie today. From look-and-feel customization like theming, to deep data model flexibility, custom scaffolder actions, and completely custom plugins, Roadie has the flexibility you need.

Backstage customizability

Backstage is not a Developer Portal. It’s actually a framework for building developer portals. You can think of it as being more like a Software Development Kit for building your own developer portal. Each company is expected to mix & match and extend the libraries that the Backstage community make available, so that they end up with a completely customized and unique IDP.

This fact is evident from day one of using Backstage. To get started, you don’t download and run a Docker container like you might expect. Instead, you run a command line tool to scaffold your own Backstage instance. Once that’s done, you write your own code to customize it.

-> npx @backstage/create-app@latest
? Enter a name for the app [required]: acme-corp-idp

Creating the app....

Another example of this flexibility is evident in the plugin architecture. The Backstage community has created hundreds of open-source plugins that can be installed into a Backstage-based IDP in order to provide visibility into the many many tools that might be used. Backstage is designed in a way that makes these plugins easy to install and configure.

From authentication hook points to source code management tool integrations to self-service scaffolder templates and UI components…. each and every part of Backstage can be ripped out and replaced and customized to your needs and desires. This is part of the philosophy of Backstage.

Roadie customizability

Despite being built on Backstage, Roadie is a complete, out-of-the-box, developer portal. We take away all of the work of building and maintaining your IDP, so that you can focus on getting value from the tool.

We believe that most organizations shouldn’t need to build a team of 4 or 5 engineers around Backstage. You should want most of your engineering efforts focussed on initiatives that deliver direct customer value, rather than building internal tools from scratch.

This is one of the reasons Roadie is delivered as Software as a Service. We want you to show up on day one and just start using it.

At the same time, we also want to retain the philosophy of Backstage. You need to be able to customize Roadie just as much as you would customize Backstage. You should to be able to connect your own authentication providers, integrate your homegrown tools, and mix and match the plugins that make sense to you. For this reason, we strive to make Roadie as customizable and extensible as we possibly can.

We believe that by combining Roadie’s ease of use with the flexibility and scope of the Backstage ecosystem, Roadie offers the best of both worlds, and is the best IDP on the market.

This post explores many of the major ways you can customize Roadie to meet your needs. Here are your options.

Theming and UI

Your IDP should look and feel familiar to your users, and it should focus their workflows in order to improve efficiency.

Logos and branding

Roadie let’s you replace the logos and branding across the site so that you can expose an IDP that matches your brand and themes.

This image shows the dark-mode version of a theme for a fictional company called BeautyBox.

Sidebar

Roadie’s sidebar configuration lets you cut the navigation down to focus it on the most important use cases in your company. If you don’t need self-service automation capabilities in your IDP then you can easily remove them to streamline things.

We also support reordering of sidebar sections, custom plugins in the sidebar, and deep links to important documentation.

Catalog customization

The Roadie catalog offers control over how the catalog is collected in the first place, and how it is represented to end users who need to consume information in the catalog.

Data model

Many engineering organizations have custom terminologies they use to describe their particular software development life cycle. If your company uses Domain Driven Design, you might want top level concepts like “Value Streams” in your catalog. Other companies might want to support groups of people called “Tribes”. Probably every company will want a list of “Products”.

Roadie supports a flexible and customizable data model that lets you rename existing Kinds and create your own types of entities.

The screenshot below shows:

Backstage’s Group kind renamed to “Teams”,
A custom entity type called “Products” has been created,
Unused kinds like Locations and Domains have been hidden from users.

Table columns

Different IDP users have different jobs, and that means they need different views. Roadie’s tables can be customized on a per user basis, so that everyone can streamline the UI to match the workflows they need.

Users can turn columns on and off, resize columns and change the density of catalog tables. The best part is that these settings are persisted so you can easily dive back into your workflow.

The launch announcement covered the benefits of Roadie’s catalog UI in more depth.

Entity providers

Custom entity providers are a way to provide entities into the catalog from external systems and existing data sources. They’re a critical integration point that helps ensure that your catalog will automatically stay up to date with external systems.

Roadie makes it easy to use custom entity providers via the Roadie Agent. The agent is a library that you can dump entities into in order to have them appear in the catalog.

Here’s a simple example where we use the Roadie Agent to dump an array of hardcoded entity metadata.

const { RoadieAgent, createRoadieAgentEntityProvider } = require('@roadiehq/roadie-agent');

const fakePayload = {
  type: 'full',
  entities: [
    {
      entity: {
        // Standard Backstage entity metadata omitted for brevity
      },
    },
  ],
};

const myEntityProviderHandler = async (emit) => {
  await emit(fakePayload);
}

RoadieAgent.fromConfig()
  .addEntityProvider(
    createRoadieAgentEntityProvider({
      name: 'testprovider',
      handler: myEntityProviderHandler
    }),
  )
  .start();

Auto-discovery settings

Any software metadata in YAML files should be auto-discovered by Roadie so that engineers don’t need to remember to register it via the UI.

Roadie supports custom auto-discovery settings, including glob matching.

Group and User ingestion

Roadie needs an organization chart in order to ensure that ownership of software can correctly be assigned to teams (aka. Groups), and to ensure that scorecard data is correctly rolled up through the org chart.

The best way to make sure the org chart is up-to-date and correct is to automatically sync it from a HR tool. To facilitate this, Roadie supports built in integrations for Microsoft Graph (also known as Microsoft Entry ID, or Azure Active Directory) and Okta. Other solutions can easily push Users and Groups into the catalog via our API.

Plugins

The real power of the Backstage ecosystem comes from its plugins. Roadie supports more than 70 Backstage plugins out of the box at the time of writing (September 2024). It typically takes us about a day to integrate a new open source plugin once it’s created by the community.

Plugin configuration

Most plugins have configuration options that Roadie admins need to be able to configure to suit their particular setup. This is easily done via admin panels we have built into the product.

Take the Argo CD plugin for example. Roadie supports two completely separate ways of integrating with Argo CD:

The app locator method allows you to dynamically search and identify Argo CD registered applications from multiple Argo CD instances.
The proxy method is used to construct links to individual Argo CD applications.

You can choose the best one for your needs, and even configure minute details like Namespace and Resource allowlists and blocklists.

The APIs of your Argo CD instances are probably not exposed on the public internet, but it’s still easy for Roadie to access them securely. We provide an open-source broker that runs in your infrastructure and makes an outbound connection to Roadie.

For other plugins, like the Kubernetes plugin and EKS, we support AWS Role Sharing as the preferred configuration method. Whatever your connectivity needs, we probably have a solution for it.

Of course Roadie’s responsibilities don’t begin and end with plugin configuration. We’re also:

Vetting plugins for quality issues as we integrate them,
Scanning plugins for licensing issues (which your legal team will be happy about),
Documenting plugins so people know how to use them,
Updating plugins as their authors release new features.

Frontend plugins

If there isn’t a community created plugin that suits your needs, or you need a plugin for a homegrown tool that only exists in your company, Roadie can support that too.

Users can write native Backstage plugins that run on Roadie and securely connect back to your private internal APIs via the broker.

The plugin development workflow supports the ability to locally host custom plugins and run them right on Roadie during development. This basically means that you can make a change to your plugin code in your IDE, and simply refresh Roadie to see your changes live.

We also support running production and development versions of the same custom plugin in Roadie at the same time, so you can iterate on your plugins without disrupting your users.

Productionizing a plugin is as simple as running the Roadie CLI against it to package it up and push it to Roadie.

roadie plugin:build --location /my-custom-plugin-folder/ --host https://static-assets.roadie.so/<my-tenant>/myCustomPlugin

The full range of Backstage frontend APIs are available to your custom plugins. You can query the catalog, check Role Based Access Control permissions, and even push analytics to see how people are using your plugin.

Proxies

Plugins frequently make use of proxies to upgrade requests with token authentication before forwarding them on to a backend service to retrieve data. Roadie users can create their own proxies inside the application. These proxies can be used to connect to third-party SaaS APIs like PagerDuty, or private internal APIs.

Here’s a user created proxy for SonarCloud.

Entity Pages

Entity pages are the pages in the catalog which pull together information on a particular piece of software, a team, or an infrastructure resource.

UI layouts

Adding plugins to Roadie interfaces is a simple matter of picking them off a list. This process is the same for both community created plugins (including plugins created by Roadie, plugins created by Spotify, and official plugins from companies like PagerDuty and Snyk) and for custom plugins you create yourself.

Once you’ve chosen your plugin, simply drag-and-drop it on the page, and resize it to suit your needs.

Different types of entities need different layouts. The Backstage Lighthouse plugin makes sense for a website, but not for a Kubernetes cluster. Roadie organizes the plugins in this way so that it’s easy to create custom views without too much setup. Our built in role based access control ensures that admins have the power to set up layouts, without overloading regular users with too many options. Team pages are customizable in the same way, they just typically have different cards and widgets.

Props on plugins

Sometimes individual cards will have their own editable properties. Roadie supports that too. For example, this card displays a list of recent GitHub Actions CI runs and their status. But how many recent CI runs should be displayed, and what layout should be used? Well, you can set that with the editable props.

This screenshot shows the edit mode of the Dynatrace Backstage plugin.

Metadata cards

We’re aware that many platform teams wish to display custom information in Roadie, but don’t necessarily have the TypeScript skills on hand to create custom plugins from scratch. To make this easier, Roadie includes no-code cards that can display information from custom metadata on the Entity.

Imagine we have a Backstage entity with some custom properties in its metadata.

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: sample-service
spec:
  type: service
  owner: group:roadiehq/engineering
  lifecycle: production
  custom:
    engineeringManager: user:dtuite
    productManager: user:samnixon87

How would we display this in the catalog? On Roadie, it’s as simple as adding the EntityMetadataCard and configuring it in a couple of clicks. The card even auto-links the Engineering Manager and Product Manager to their User pages in the catalog.

Home page

Of course it’s not just pages in the software catalog that can be customized. Roadie users can also customize the Home page plugin to display key information that’s relevant to them. This is the place to display your open pull requests, community news, your calendar, and other useful info that’s tailored to the logged in user’s experience.

Scaffolder

Roadie scaffolder supports the full range of customization that you would expect from the Backstage scaffolder.

Actions

Custom scaffolder actions are a way to execute homegrown CLI’s or arbitrary code as part of scaffolder templates. These are executed within your own infrastructure, which provides an added benefit in that it makes it easy to send internal network requests.

Users can use the Roadie Agent (the same library used in the Custom Entity Providers section above) to register custom scaffolder actions with Roadie.

RoadieAgent.fromConfig(config)
  .addScaffolderAction(
    createRoadieAgentScaffolderAction({
      name: 'custom-action', // The name of the action as defined in Roadie
      handler: async (ctx) => {
        try {
          fs.writeFileSync(
            `${ctx.workspacePath}/test.txt`,
            'new file with new contents',
          ); 
          // Writing a new file into the shared workspace
        } catch (err) {
          console.error(err);  // Local logging on the Roadie Agent process
        }

        let count = 0;
        while (count < 5) {  // Additional other actions that is wanted to be taken. This time looping for 5 seconds
          await new Promise((resolve) => setTimeout(resolve, 1000));
          count++;
          await ctx.log(`hello world`); // Sending a log message to be displayed to the end user
        } 
      },
    }),
  )
  // Add a second custom scaffolder action
  // .addScaffolderAction(...) 
  .start();

These can then be used in your templates like this:

steps:

  # This step executes on Roadie
  - id: fetchTemplate
    name: Fetch file
    action: fetch:template
    input:
      url: ./skeleton

  # This sends the workspace context to a self-hosted scaffolder runner
  # and executes the custom action we defined above. The payload passes
  # values to the self-hosted scaffolder runner and logs are written
  # back to Roadie.
  - id: invokeCustomAction
    name: Invoke a custom self-hosted action
    action: roadie:agent
    input:
      action: custom-action
      shareWorkspace: true
      payload:
        name: someValue

Field extensions

Custom field extensions give you the ability to build your own inputs and UI components for the scaffolder. You write them using the normal Backstage API, and push them into Roadie using the custom plugins pipeline.

Once registered, they can be used in Scaffolder templates by matching the name of the ui:field prop.

parameters:
  - title: Fill in some steps
    required:
      - name
    properties:
      name:
        title: Name
        type: string
        description: My custom name for the component
        ui:field: MyCustomInput

Authentication & Authorization

Single-Sign-On (SSO) setups

Roadie provides native support for any SSO provider you can imagine. Okta, Microsoft Entra ID, Google, AWS, Ping Identity…. you name it, we’re probably already using it in production.

Setting this up is a simple matter of sending our docs to your IT team, or whoever controls your SSO setup. We will then work directly with that team to get your authentication provider configured.

Once set up, your IT team can grant and revoke access to Roadie without being blocked by us. They have full control over the process.

Roles

Customers who have purchased our role-based access control add-on can define custom roles which can then be assigned to groups of users.

In this example, you can create a role which can only read the catalog. It wouldn’t be able to execute scaffolder templates or register anything in the catalog.

Roles can also be assigned via your Identity Provider like Okta or Microsoft Entra ID. This is especially useful when your IT team wishes to manage access and roles in a single place.

Permissions/policies (coming soon)

When role-based access control gets really powerful is with the ability to define custom permissions which can then be combined in roles. For example, perhaps you want to hide Component entities which are tagged with sensitive from everyone except the security team.

We’re working on this ability at Roadie and expect it to roll out in 2024.

Tech Insights (Scorecards)

Custom data sources

Data Sources are recurring jobs which gather data that can then be used to write checks against.

For example, Roadie comes with a built-in GitHub Settings Data Source which records facts about each repository in the catalog, like whether or not branch protection is turned on, or whether or not force pushes are allowed.

Users can also create their own Data Sources from scratch. At the time of writing, 7 different types of Data Sources are supported. Here are some examples:

HTTP Data Sources hit third-party APIs and pick values from the JSON response
Component Repository File Data Sources inspect files in the repository associated with each Component and record values from them.
Push Based Data Sources receive webhook events and store them for processing.

Creating Data Sources is a simple matter of filling out a few fields. You don’t need to write any TypeScript or YAML. Data Sources support advanced features like metadata variables, JSONata processing, inline testing, and error handling.

Custom checks

Checks inspect a value created by a Data Source and give it a pass or fail for each entity it applies to.

This check ensures there is a README.md in the repository associated with each piece of software.

Advanced features like boolean logic and live testing of the check are supported. Checks can link to documentation when they fail, or even link to a scaffolder template that can help the owner of a service to fix the failure.

Custom scorecards

Users can create their own scorecards that apply to the software in the catalog in order to communicate best practices and expectations to teams.

Each scorecard is a user defined collection of checks that shows up in reporting, in the catalog, and on team pages. Scorecard data is rolled-up through the catalog so that you can dip in at any level of the org chart and see aggregated data from below that point.

And there’s more coming

We’re constantly adding to the ways you can customize Roadie. If you want to stay up to date with the latest customizations, subscribe to our newsletter to hear what’s coming.

Image by GrumpyBeere from Pixabay

The Ultimate Guide to Backstage Software Catalog Completeness

David Tuite — Wed, 09 Oct 2024 10:10:00 +0000

Internal developer portals (IDPs) like Backstage and Roadie are, at their core, software catalogs. The software catalog is the backbone upon which much of the other functionality hangs, and building a complete catalog is vital to the success of the IDP project.

Without a complete catalog, an IDP cannot fulfill it’s primary purpose: improving developer productivity. It cannot drive improved discoverability if the software that teams are trying to discover is not in there. It cannot help to measure the standardization or security posture of software it does not know about.

This article is a comprehensive guide to catalog completeness in Backstage. Roadie is built on top of Backstage, and so all of the same lessons apply. We cover why it’s important, how to measure it, and how to achieve it.

First, let’s learn why you want to build a complete catalog in the first place.

Why building a complete catalog is important

The primary reason companies deploy an IDP is because they want to make developers more efficient. A big part of this is helping teams answer basic questions about the software around them. They want to make it easy to answer questions like “do we have a geocoding service?”, “which team owns the checkout service?” and “where is the API spec for the users service?”. These questions can only be answered by the IDP if the geocoding service, checkout service, and users service are registered there in the first place.

A complete catalog is essential when using an IDP to measure the maturity of the software that teams are producing. Applying scorecards to software in the catalog of the IDP can be a great way to determine the security posture of each production service. But scorecards can only apply to software which is actually in the catalog.

The software catalog is also the fundamental unit of navigation in the IDP. The lack of a software catalog is the reason that wikis like Confluence or Notion cannot solve the discoverability problem that IDPs solve. In a wiki, content is organized into pages, which are intentionally unstructured and flexible. In an IDP, content is organized by Component (think “piece of software”) and is structured so that the same information is available for each Component.

How to measure catalog completeness

Having a complete catalog doesn’t necessarily mean that every piece of software is in the catalog. Most organizations have their share of abandoned code. It may have been created during hackathons or for brief experiments that never saw production usage. Having all of this stuff in the catalog can create clutter and noise.

There may also be parts of the organization for whom it doesn’t make sense to be in the catalog. We work with companies which have large hardware divisions who write code for embedded devices. They don’t necessarily follow the DevOps lifecycle and thus are sometimes intentionally omitted.

Production software is the most important stuff to have in the catalog. It’s the software that most engineers work with most of the time. Production software will have the most frequently referenced APIs and docs and it’s much more important to have an understanding of the maturity of the software that runs in production environments, since it has the most attack vectors.

For this reason, we frequently see customers create a list of software which is deployed to production by referencing ArgoCD or some other deployment tool. They then compare this against software in the catalog, frequently by accessing the Roadie APIs or by using our CSV export functions.

The next best bet is to compare number of pieces of software (aka. Components) in the catalog against the number of active, unarchived repositories in your source code solution. This will never give you a perfect answer, because “pieces of software” don’t necessarily map perfectly one-to-one to repositories (monorepos etc), but it’s a good start.

How Roadie Helps

At Roadie we give every customer a chart which shows the percentage of their active (non-archived and received a commit in the past 12 months) repositories against the number of Components in the catalog.

Roadie customers can achieve high catalog completeness

Achieving a high level of catalog completeness is definitely possible.

The majority of established Roadie customers are happy with their level of catalog completeness and we have many customers who have a catalog completeness level (measured as a percentage of active repositories which are registered in the catalog) which is above 80%.

Here’s a chart showing catalog completeness for Uplight who onboarded in September 2023. Four months later they were at 88% catalog completeness, with more than 600 components in their catalog.

Here’s another Roadie customer who increased their catalog completeness from 40% to 90% over a period of four months. They now have more than 750 components registered.

There are countless more examples like this amongst Roadie customers. The goal of this post is to help all companies achieve the same results.

Strategies for building a complete catalog

There are multiple strategies that can be used to build a complete catalog. The strategy that is right for your company can depend on factors like the size of the company and the enthusiasm of the developers to have an IDP.

You may also want to run multiple strategies in parallel, or start with one in order to get the bulk of software into the catalog, and switch to another to get closer to full catalog completeness.

Think about expanding catalog completeness like the layers of an onion. Start with the most enthusiastic early adopters and get them onboard. Then use them as an example as you expand out to the rest of the organization.

We recommend you import users (aka. employees) and teams (called Groups in Backstage nomenclature) into the catalog before creating any software components. You will ideally want to assign ownership of each component to a team as you import it. This task is much easier if all of your teams already exist in the catalog.

If you don’t know which strategy to choose…

Experiment! You don’t need to roll out to the whole organization in one go. Pick some friendly teams, run one strategy on each team, and analyze the results. Did you end up with software in the catalog? What is the feedback from each team? Where did they get stuck? Take this feedback and use it to improve the process before expanding out to more teams.

Strategy 1: Centralized automated

This strategy involves connecting to a chokepoint in the organization in order to push software metadata into the catalog programatically. It does not require anyone to write catalog-info.yaml files.

Frequently, the initial chokepoint will not have all of the information required to build a useful catalog. When this happens, the software metadata must be enriched after the fact.

Depending on your tech stack and permission settings, following options may be viable:

An ArgoCD instance which does deployments to the production environment. It has knowledge of the most critical software in the org (software that goes to production) and can thus can be a good starting point for the catalog.
A Helm chart which is used by a large percentage of the deployable software in the organization.
A centrally owned CI job or build tool which can be written by the centralized team and applied to software which is owned by other teams in the organization. For example, Lunar Bank have talked about how they populate their catalog from their build tool called shuttle.
A legacy software catalog, developer portal or spreadsheet.

Pros

The software catalog can quite quickly be bootstrapped to a highly complete state.

Cons

The ownership link between software and teams is not immediately established.
Companies with no source of truth or a heavily fractured deployment ecosystem may not be able to implement this strategy.
The product teams will be less well educated on the value of the IDP when the process is finished.

Tactics to prioritize in order to succeed with this strategy

Use custom entity providers
Make catalog presence a requirement for deployment

Strategy 2: Centralized manual

This strategy tries to avoid asking the individual product teams to do work. Instead, the centralized team takes it upon themselves to populate the catalog. They may do this in collaboration with the product teams, but they likely won’t ask them to write any YAML.

Pros

This strategy is likely to be faster than the distributed manual strategy, especially for companies which don’t have thousands of microservices.

Cons

The product teams will be less well educated on the value of the IDP when the process is finished.
It may be difficult for the centralized team to gather enough data about each individual software component.
It becomes the centralized teams job to manually keep the catalog up to date.

Tactics to prioritize in order to succeed with this strategy

Store software metadata in a single repository
Open scripted pull requests
Customize the catalog nomenclature

Strategy 3: Distributed manual

This strategy involves asking all of the individual product teams to register the software that they own in the catalog.

Typically, the central team who own the IDP will meet with and educate the product teams on the value of the IDP, either on an individual basis, or in larger groups. The central team will provide tools and materials to the product teams in order to teach them what they need to do to get their software into the catalog (typically create a catalog-info.yaml file), and give them clear steps to take in order to achieve catalog completeness.

Pros

Product teams are more likely to feel a sense of ownership over their software in the catalog because they put it there in the first place.
Product teams have an opportunity to learn about features of the IDP as they are registering their software. They may then choose to use features in the IDP such as technical documentation.

Cons

This strategy requires a lot of work from the central team in order to yield high catalog completeness. It will take time. They will need to educate and continually follow up with product teams throughout the company.

Tactics to prioritize in order to succeed with this strategy

Give teams a scaffolder template to register software
Share catalog completeness numbers publicly
Tie catalog completeness to a wider initiative
Write custom plugins to create value

Tactics for building a complete catalog

The tactics you need to use will depend on the strategy you are implementing. This is a full list of all the tactics we know. Please refer to the strategies above in order to know which tactics to choose.

Some tactics will apply regardless of the strategy that is chosen. They are:

Customize the catalog taxonomy
Move onboarding docs into Roadie
Present on the value of Roadie to people managers
Tie catalog completeness to a wider initiative

Each of the tactics mostly fall into one of the following categories:

Reduce friction for developers who want to get their software into the catalog.
Create incentive for developers to put their software into the catalog.
Educate developers on how to use the catalog.

Give teams a scaffolder template to register their software

Category: Reduce friction

This involves writing a scaffolder template that teams can use inside Backstage in order to create a catalog-info.yaml file in their repositories.

The scaffolder template will ask the user to fill out some information about their software, typically by picking from pre-defined values, and will open a pull request against a repository when finished. Once the user reviews and merges the pull request, auto-discovery will pick up the catalog-info.yaml file and populate the catalog.

Benefits

Individual developers don’t need to understand the (long) Backstage YAML API spec in detail.
The owners of the IDP can use the template to constrain the “type”, “lifecycle” and other properties that is assigned to each software component. This puts guardrails in place that will help create more consistency in the catalog. Catalog consistency is important, and will help you avoid problems in future.
The form can integrate with external APIs to pull in sensible options. For example, in the screenshot above, the “Component Owner” is a selection of all the engineering teams in the company. The user doesn’t need to type an exact string.

How Roadie helps

Roadie provides a starting point for a software registration template in the getting-started repo. Customers can fork it into their own GitHub org, edit it to meet their needs and import it into their own Backstage instance.

Archive unused repositories

Category: Reduce friction

If our measure of catalog completeness is:

then we can increase catalog completeness by archiving old repositories that nobody is using.

It may sound somewhat silly, but every organization has abandoned hackathon projects and old test repos that are unused and simply causing clutter. By archiving them, we clean up our source code management tool while improving catalog completeness.

Believe it or not, Roadie has one customer who increased catalog completeness from 45% to 75% just by archiving repositories.

How Roadie helps

Our catalog implementation ensures that Components are removed from the catalog when the repo they reference is archived.

Customize the catalog nomenclature

Category: Reduce friction

By mapping the Kinds of entity that show up in the catalog to familiar concepts in your company, you can create instant recognition for developers who land in the catalog.

For example, if your company has a concept of “Valuestream” then make this front and center in the catalog so that users instantly understand what they’re looking at.

Benefits

Users can orient themselves quickly and get value rapidly.

How Roadie helps

Roadie customers can use our admin interfaces to customize and rename the core catalog concepts, and create completely new ones.

Write custom plugins to create value

Category: Create incentive

Custom plugins provide value for product teams by giving them faster ways to perform bespoke workflows inside the catalog. By creating custom plugins, a central team can incentivize developers to add their software into the component.

For example, Lunar Bank have custom plugins for dealing with dead letters in RabbitMQ. These plugins are regularly useful for developers. This causes them to visit the catalog to use the plugin.

Benefits

Custom plugins are quite simple to produce and can quickly create value for software engineers.
Custom plugins unlock tailored value for engineers to help them do things more quickly or more easily than they otherwise could. They sometimes even allow them to perform a task that they cannot do at all outside of Roadie or Backstage.

How Roadie Helps

Roadie provides an interface for registering and managing custom plugins. It also facilitates live reloading of custom plugins, the ability to run multiple versions of a plugin side by side, and a scaffolder template to bootstrap a custom plugin monorepo. Custom plugins on Roadie can securely connect back to a private network to load data from internal APIs. Check out our docs to learn more.

Move onboarding docs into Roadie

Category: Educate

Engineer onboarding docs are typically used to help a new engineer to set up their environment and get to productivity quickly. Expedia Group, Spotify and other Backstage adopters have successfully used TechDocs and scaffolder templates to speed up engineer onboarding and help new engineers become familiar with the IDP on day one. The exact same tactic can be deployed on Roadie.

Expedia Group put 850+ engineers through their Backstage based bootcamp in 2022. They discuss it in their case study on the Backstage website.

Benefits

Newly onboarded engineers start using Roadie on day one. They get used to it and understand how to come back.
Engineers learn how to use a scaffolder template to create a new service during onboarding. This service is added to the catalog, and they learn how the catalog works.

How Roadie Helps

Roadie supports standalone TechDocs that are not tied to a particular software component in the catalog. These are perfect for onboarding docs.

Present on the value of Roadie to people managers

Category: Educate

Roadie provides specific value to managers, directors and VPs that is different than the value that developers might care about. By educating managers on the value they will receive, you can encourage them to work with their teams to get their software into the catalog.

For example, did you know that frequent Backstage users at Spotify are 5% more likely to be at the company one year later. Retention is important for managers, so they need to know about this.

How Roadie Helps

Roadie provides Scorecards which can help managers ensure that their teams are producing mature and high quality software. This feature is not available for open-source Backstage.
Roadie gives customers value calculators to help them estimate the dollar value they can expect to receive.

Open scripted pull requests

Category: Reduce friction

Opening an automated pull request containing a catalog-info.yaml file is a good way to ease the burden on developers who want to get their software into the catalog. All they need to do is edit the pull request a little bit, review it and merge it.

Keep in mind that your script will need permissions to open a pull request against a majority of repositories in your source code management tool. Depending on your security model, this may not be possible.

This method can work especially well in companies that operate out of large monorepos. A monorepo setup allows the generation of a single pull request that can add many catalog-info.yaml files in one go. It can also be reviewed and merged by a single person with elevated permissions.

While this is a tempting option to quickly build a catalog with YAML files, we have seen customers experience issues with catalog correctness when they use this method. Some teams may blindly merge the pull request without validating the information that it contains. This tactic should be executed alongside an education program to teach teams what to do. Go slowly and experiment.

How Roadie Helps

Roadie provides a token authenticated API which the centralized team can use to tell which repositories are already in the catalog. A simple script can consume this to open a PR against the repos which are not already accounted for.

Our solutions engineering team can work with you to write a simple script that will open a pull request containing a YAML file into each repository.

Make catalog presence a requirement for deployment

Category: Create incentive

By making catalog presence mandatory for deployment, platform teams can be confident that they have all of the important software in the catalog.

In the early days of Backstage at Spotify, teams could not SSH into their machines unless their services were in the catalog. The catalog owner was referenced to determine who was and was not allowed to access the machine.

This tactic works best when the IDP is orchestrating a new greenfield platform that other teams are migrating onto. Adding the catalog-info.yaml file can be one simple step in what is likely a series of steps that teams have to do to migrate. Outside of this situation, it can be politically problematic to block deployments due to a missing YAML file.

Automate catalog collection with custom entity providers

Category: Reduce friction

If developers find writing YAML files tedious, potentially the best thing to do is to make them optional. Backstage’s custom entity providers allow adopters to programmatically shovel software entries into the catalog. Custom entity providers are a great way to connect Backstage to an existing source of truth for catalog data, such as a legacy IDP, an ArgoCD instance, a kubernetes cluster, or a CICD tool.

How Roadie helps

Roadie gives customers the roadie-agent. This wraps the custom entity provider concept with a secure connection to Roadie so that customers can easily dump software metadata into the agent and have it appear in the catalog.

Roadie has an Entity Provider API. Simply push an array of software metadata to this endpoint and it will appear in the catalog. To update the metadata, simply push again.

Frequently the programatic source of truth will have some but not all of the metadata that the catalog needs. For example, it may be missing the name of the team who owns the software. Roadie allows users to decorate software in the catalog with extra metadata within the UI. This ensures that the catalog can become complete and rich over time.

Share catalog completeness numbers publicly

Category: Create incentive

One great way to get people bought into the idea of building a complete catalog is to make it a group effort. By transparently reporting on the completeness and “health” of the catalog, a shared sense of ownership over the goal can be created.

Twilio explained how they do this in their catalog at the Autodesk Developer Productivity Summit.

How Roadie Helps

Roadie gives all Tech Insights customers an out-of-the-box measurement of catalog completeness.

We also report on important aspects of catalog richness, like the percentage of pieces of software in the catalog which have an owner assigned. Customers can use these building blocks to measure other attributes of the health of their catalog.

Tie catalog completeness to a wider initiative

Category: Create incentive

Companies usually want a complete catalog so that they can accomplish some wider engineering goal. By promoting catalog completeness in service of this wider goal, teams can better understand why it is important to be in the catalog, and why they should help.

For example, we recently worked with a customer who needed a complete and correct list of all software that had access to their users personally identifiable information (PII). By attaching to this company wide goal and leveraging the influence of the company CTO, the company was able to rapidly label hundreds of software components in Roadie with their PII status.

How Roadie Helps

We run regular customer success meetings with customers to help them identify wider engineering initiatives where Roadie can help accomplish the goal more quickly or more easily. We will then work with teams in order to project manage the delivery of a solution.

Do a whiteboarding session

I did some eduction with a team at [company] where we brainstormed the services they wanted to cover and their relationships, this was a whiteboard exercise and I then created the PR's and had them review.

When Paddle started with Backstage, they realized that they didn’t have an existing source of truth they could lean on to populate their catalog, and they would have to collate it manually.

They started with a whiteboarding exercise so they could iterate quickly. Meeting with each team in small groups, Ioannis Georgoulas (Director of SRE), led the process. He first spent time brainstorming the services that the groups wanted to catalog, and defining their boundaries and the relationships between them. This information was all collected in a simple document to start.

Once he had a solid understanding of the service map, Ioannis opened pull requests against each repository with the catalog-info.yaml file that was needed. All the teams had to do was review and merge it. Because they had participated in the process to gather this information, they were already bought in and could understand the value of it.

How Roadie Helps

We provide frequent customer success calls with every customer through the initial stages of implementation. We’ll partner with your implementors to run these whiteboarding sessions and gather the information you need to be successful.

Key takeaways/Conclusion

Building a high level of catalog completeness in Backstage need not be intimidating. By carefully choosing the strategy that will work at your company, expanding outwards from the most eager adopters first, and communicating widely as you go, you can reach a high level of catalog completeness in a short amount of time.

Two topics we have not discussed yet, are catalog richness and completeness. These areas go hand in hand with completeness, and work together to ensure that your IDP has the answers that developers need, when they need them.

We’ll be covering richness and completeness in another article. If you want to be among the first to read it, make sure to subscribe to the newsletter below.

Backstage Weekly 22 - Progress made on Backstage search

David Tuite — Tue, 06 Apr 2021 12:57:12 +0000

Backstage Weekly is a roundup of recent features and community news for the OSS project Backstage.

Merged in the last 2 weeks

106 pull requests were merged in the last 2 weeks. Here are the highlights.

Search foundations 🔍

The TechDocs team from Spotify added a new search API to the backend. Until now, all searching happened in the browser and thus was limited to small datasets. This move is one of the first steps towards offloading document indexing and querying to third-party tooling such as Elasticsearch.

Two new concepts are introduced in this PR (#4515). Collators are responsible for providing the IndexableDocument for a given type of document and Decorators are responsible for providing additional attributes on existing documents.

There are no user facing changes yet but it's great to see this progress.

Delete items from catalog

Backstage now has more a natural deletion flow for catalog items. Previously you could "unregister" items which had been hardcoded into the Backstage config. This was confusing because they would magically reappear a few minutes later. The UI now handles the various different cases a little more elegantly.

For example, if you add an entity via the Backstage configuration file, you will be warned that you're trying to perform an action that doesn't make sense and given an advanced option to proceed regardless. #5043

Read-only mode

Lunar architect Bjørn Sørensen added readonly mode to the Backstage catalog in #5034.

This feature is useful for organizations who don't want to use the Backstage scaffolder, perhaps because they have existing tooling in place or use a strict GitOps flow to create new components.

By setting readonly: true in the Backstage config, all UI interactions which would mutate the catalog are disabled.

OwnerPicker and GitHub Deployments plugin

Fintech company GoCardless have picked up the pace of their open-source development work with 2 notable merges recently.

The catalog importer now has UI for assigning ownership of a component to a given team. Previously you would have to exactly type the name of the team who owned the component. Now you can pick it from a select menu. #5086

GoCardless also added a GitHub deployments plugin which displays a table of the most recent deployments, showing the status and environment of each deployment, along with the commit which was deployed.

Roadie news

Plugin upgrades

We have upgraded and released new versions of most of our plugins lately. If you previously hit any issues because of mis-matched Backstage versions, please upgrade and try again.

We have a dedicated open-source engineer starting in mid-April so the pace of development in this area should increase shortly.

New Roadie feature

We've rolled out a new Dynamic UI feature to all of our customers. This feature allows users to reorganize their component overview pages by dragging and dropping plugin widgets around and add new tabs to Backstage components without editing the code.

Backstage Weekly 21 - Spotify story - Why OSS Backstage succeeded

David Tuite — Thu, 25 Mar 2021 08:14:32 +0000

Last week marked the 1 year anniversary of Backstage being announced to the world on the Spotify Engineering blog.

Last year, a small team of Spotifiers had a hunch about our homegrown developer portal: if Backstage could help our 1,600+ engineers manage the 14,000+ software components we use at Spotify, then couldn’t it do the same for other growing tech companies, too?

Backstage was released with a very limited feature set that has improved rapidly over the past few months. The service catalog and scaffolder, which didn't exist in that first release, have reached beta status, the Kubernetes plugin has developed into a useful window into production deployments, and the plugin ecosystem has begun to flourish.

Even more important than the technical achievements, the community is bonding, growing and collaborating and the number of non-Spotify contributors is increasing over time.

Of course, the community must keep one eye on the past and one on the future in order to thrive. Tyson Singer, the VP of Technology and Platform at Spotify, took 10 minutes of the latest Backstage Community Session to talk through Spotify's vision for Backstage and their history and relationship with the open source community at large. Tyson's org builds Backstage so he is in an influential and important position when it comes to its development.

Tyson talked through Spotify's strategic approach to open source over the past 2 or 3 years and shared incredible learnings for anyone trying to drive large scale change in an engineering organization.

Spotify followed a 3 step strategy to improve the impact of their open source development.

Clearly define the goal and the reasons for investing in open source.
Build new organizational capability and nurture it internally.
Unleash this capability on the world outside to achieve the goals.

Spotify identified 3 key reasons to improve their open source offering.

They wanted to attract and challenge the best technical talent and become known as a top 10 technical employer.
They wanted to gain influence over their cloud vendors.
They wanted to end the cycle of Spotifiers building great internal technology, only to see it replaced by open source offerings because they hadn't released it to the world.

Once they had the goal clearly defined, they built new 3 internal capabilities to help achieve it.

They aligned the organization around a customer-centric view which took focus away from infrastructure mechanics and placed it on the customer — other developers inside Spotify.
They spent more time with their customers by embedding in their teams, and encouraged their customers to do the reverse.
They built product marketing skills so they could communicate the value of their work effectively.

Once they had exercised this new organizational muscle internally and decided that Bacsktage would be their first Tier 1 Open Source project they released it early, iterated quickly and built in the open. The rest, as they say, is history.

You can see the rest of the meetup on YouTube.

Merged last week

58 pull requests were merged last week. Here are the highlights.

Todo Plugin

Part of me feels like I could dedicate a whole newsletter issue to this neat idea.

The todos plugin finds TODO comments in the source code of each component in the catalog and creates a table inside Backstage where they can be seen by other developers.

To me, this simple idea represents everything that Backstage is about. TODO comments are typically buried in the code because they're only visible to the engineer who happens to have that code in front of them. This plugin makes TODOs visible to everyone in the company who may be curious, elevating them to a altogether different level of usefulness.

The plugin was contributed by Backstage core team member Patrick Oldsberg in #4911.

Powered by Backstage badges

Rolling out Backstage isn't simply a matter of deploying the technology. Backstage is at it's best when it is regularly used by a wide range of personas in an organization. During the rollout, it can be helpful to nudge towards Backstage by making it present inside their existing routines.

To facilitate one aspect of this, Andreas Stenius from Svenska Spel contributed badge generation for Backstage components. When you use this feature you get a copy-pastable markdown snippet which you can add to the GitHub readme of any component. #4581

Once added, people can more easily navigate to Backstage from GitHub repos and discovery should increase across your org.

Roadie news

The first ever Backstage Open Mic will take place on the 31st of March. This is a community organized Backstage event where people share the experiences with Backstage and help each other out. There will be two talks:

Roadie engineer Iain Billett will present on how we're deploying Backstage on Kuberentes using flux and Terraform.
SDA SE regular Dominik Henneke will show how to use the Backstage API docs plugin to discover APIs.

This was advertised as the 25th of March in a previous newsletter. It was rescheduled after publishing.

Backstage Weekly 20 - Upcoming meetups and Kubernetes CRD support

David Tuite — Mon, 15 Mar 2021 16:02:17 +0000

Believe it or not there are two Backstage meetups happening over the next 2 weeks! 📅

The next official Backstage Community Session will take place on March 17th. As usual, you can suggest topics for the session and vote on your favorite topics using the linked GitHub issue. The current top suggestion is a presentation on how Expedia Group are using Backstage internally. Expedia is a huge organization with many engineers employed so this could be very interesting indeed.

Roughly a week later, on March 25th, the first ever Backstage Open Mic will take place. This is a community event where anyone can talk about technical topics or their experiences with Backstage. We at Roadie will present on how we're deploying Backstage on Kubernetes using terraform and flux. RSVP here.

Merged last week

61 pull requests were merged last week. Here are the highlights.

CRD Support for the Kubernetes plugin

Spotify engineer Matthew Clarke, who you may have recently come across on the Kubelist podcast, contributed CRD support for the Backstage Kubernetes plugin. So far it supports ArgoCD Canary rollouts, with more rollout types in the pipeline. #4892

If you use ArgoCD Canary rollouts you'll get a view like this inside Backstage and you'll be able to see your rollout progress over time.

Bitrise plugin

SDA SE contributed a plugin for mobile app CI/CD platform Bitrise this week. It can quickly show you when your mobile builds are failing and even allow you to download the artefact generated by the builds. #4746

Backstage errors package

The Backstage team have started an effort to standardize error reporting in Backstage in order to improve consistency when things go wrong. There is now a @backstage/errors package and an error panel which shows the most pertinent information to the user immediately and can also expand to show more details. #4903

I've seen a number of discussions about when and how to display errors to the user in Backstage so it's good to see standardization begin in this area.

That's all for this week. Get upgrading folks!

Backstage Weekly 19 - Custom actions for the scaffolder

David Tuite — Mon, 08 Mar 2021 11:54:53 +0000

The Backstage scaffolder is an easy way for engineering organisations to bake best practices into a set of software templates and encourage their use for creating new services and components. This helps improve production consistency and reduce lead time for new services.

Each template is a YAML file which Backstage uses like a recipe for scaffolding a new application and performing actions on it.

Once templates have been defined and imported into Backstage, they appear in the template directory.

Engineers who wish to start a new project can choose a template from the directory, enter some customization parameters, and submit the form to Backstage for scaffolding to begin.

At this point, Backstage runs the steps defined in the template in order to do some work. As it runs, Backstage reports progress and logs to the user so they can understand what is happening. Once finished, the user is presented with some links to the newly created component.

Out of the box, Backstage supports useful scaffolding steps such as:

Fetch a cookiecutter template from a remote URL and run it with the parameters provided by the user.
Add the TechDocs framework to the templated application.
Publish the templated application to GitHub as a new repository.
Register the templated application in the Backstage catalog.

Last week, Spotify engineer Johan Haals added the ability for Backstage to support custom scaffolder actions (#4718). This opens up a whole world of new potential for the scaffolder.

For example, you could use the custom actions feature to hook your newly scaffolded application up to your CI system, configure it for monitoring, or even deploy it to a staging environment. Perhaps a whole marketplace like the GitHub actions marketplace will spring up around this feature.

Whatever way this goes, it's great to see the Backstage core team iterating on core features and constantly improving them. The scaffolder docs are here.

Merged last 2 weeks...

There was no newsletter last week so we're covering 2 weeks and 142 merged pull requests here.

OpenStack swift support for TechDocs

Swift is a distributed blob store for the popular open-source virtualization platform OpenStack. In pull request #4714, Trendyol engineer Erdogan Oksuz added the ability to use it as a store for published TechDocs.

This is the first time I've seen OpenStack referenced in a Backstage pull request and it's a nice sign of the portability and broad platform support that the core team are baking in.

Extended support button config

Andrew Thauer from Wealthsimple added a top level config to Backstage to help define ways that you can get help inside Backstage. When rolling out Backstage inside your org, it's important that users have a clearly defined way to get help while they are learning to use Backstage. This work from Andrew makes that a little easier to do. #4517

Other news

Backstage team manager Lee Mills and Kubernetes plugin developer Matthew Clarke appeared on the latest episode of the popular Kubelist Podcast where they talked about the inspiration for Backstage and direction of the project.

The Backstage team also participated in a wildly popular Ask Me Anything on Reddit a few days ago. Beyond the plugin, they talked about their philosophy for exposing developers to Kubernetes, how they hire people to work on Backstage and how they implement multi-tenancy in Kubernetes. Check it out here.

That's all for this week, get upgrading folks! 🚀

Backstage Weekly 18 - How Spotify drove Backstage adoption

David Tuite — Mon, 22 Feb 2021 16:40:12 +0000

The first Backstage community session took place last Wednesday and was a resounding success. 117 people joined the core team on Zoom to hear Spotify explain the drivers behind the huge success of Backstage there.

We also got an update on recent progress against the roadmap and SDA SE spoke about how they are implementing Backstage to improve information sharing between insurance companies in Germany.

How Spotify drove Backstage adoption

Spotify's Backstage journey began about 4 and a half years ago, in 2016. At the time, the company was growing rapidly and experiencing productivity issues as a result. Many new teams were formed and and a large number of new services were created. All this growth meant that discoverability was at an all time low.

Some engineers got together and asked themselves "how do we help and aid discoverability while empowering teams to move quickly". From this question, Backstage was born.

Once released, a number of key features drove adoption inside Spotify.

The service catalog became a central place to track ownership and simply look up contact details of teams.
The plugin model unlocked rapid growth in the number of ways that Backstage could be used and the features it had.
Building engineer onboarding flows into Backstage meant that new engineers got used to using it quite quickly.
Growing usage triggered a flywheel effect where more plugins brought more usage which caused more plugins to be added and even more usage.

Eventually enough gravity built up to ensure that Backstage became the default place to build any internal UI.

Today, an internal Bacskstage core team with only 4 engineers drives productivity improvements across the entire 2,000 person engineering organization. Only 15% of the code that makes up Backstage is contributed by this team. The other 85% comes in the form of plugins created by Spotify engineers who use it every day.

You can hear this story, and more recent adoption stories from SDA SE and Zalando by checking out full recording on YouTube.

The next community session will take place on March 17th. You can suggest topics via GitHub.

Merged last week

57 pull requests were merged last week. Here are the highlights...

Splunk On-Call plugin

Splunk On-call is an incident management tool for DevOps teams. It helps with on-call scheduling, incident response and analysis. If you've used PagerDuty before you'll get the idea.

In addition to creating this plugin and opening a pull request packed with screenshots and details, Rémi Doreau also documented the entire process on his blog. #4416

Search and filter for templates

Organizations who make heavy use of the Backstage scaffolder may find that they build up a large number of software templates. Pull request #4479 from Oscar D. Hernandez adds the ability to search, filter and star templates so it's easier to find the exact one you need.

Customize entity links via annotations

James Turley from GoCardless added the ability to customize the entity links on the about card by setting certain annotations in your catalog-info.yaml file. This seems trivial but makes configuration much easier for Backstage users who want to populate the catalog from an existing service registry or custom backend. #4575

Roadie news

We have published a humongous in-depth guide to deploying Backstage on Fargate. It covers everything... setting up an RDS PostgreSQL database, configuring Backstage with parameter store, load balancing, deployment... you name it. Check it out if you're looking for a straightforward guide to deploying Backstage. Thanks to Roadie engineer Jussi Hallila for writing this one.

We've paused work on the Backstage Datadog plugin. It seems that Datadog charts are either private on Datadog or public to the entire internet. There is no mode which allows Backstage to have secure access without making the chart completely public. We've posted our findings in the GitHub issue.

Backstage Weekly 17 - Next steps for the scaffolder

David Tuite — Tue, 16 Feb 2021 18:06:42 +0000

Engineering organisations often face a dilemma. When there is a lot of freedom to choose technologies and tools, the developer ecosystem ends up with high variability and few opportunities to optimize developer flows. On the other hand, if too much constraint is applied then innovation is stifled and creativity suffers.

Spotify addressed this problem by ensuring that the path to production consistency was also the path of least resistance for engineers who were just trying to get things done.

As part of this work, they developed the Scaffolder in Backstage. The scaffolder is a UI which collects metadata describing a new service and runs automation to create it with your organizations best practices baked in.

The existing alpha version of the open source Scaffolder can collect metadata and template a new GitHub repo into your account. This is great as an MVP, but it's about to get a lot more powerful.

Last week, Johan Haals from Spotify merged the 5th pull request on the path towards getting the scaffolder out of alpha.

You can read about the planned features and developer experience in the RFC. The plans include:

A declarative, GitHub-Actions-like language for defining templates.
Template composability so users can use the TechDocs or monitoring setup steps in their Python backend API template and in their React website template.
Hooks for setting up CI pipelines, configuring projects or automatically setting branch protection rules.

Getting the scaffolder out of alpha is a Q1 goal for Spotify so we're looking forward to trying out these features soon.

Merged last week

71 pull requests were merged last week. Here are the highlights...

Rework the catalog import plugin

This pull request has 16 emoji's on it so you know it's a good one! ❤️

Dominik Henneke from SDA SE took some early work that Roadie did to make it easier to add GitHub repos to Backstage and supercharged it so it's easier to use and covers more cases. The importer now has better monorepo support, a confirmation step to prevent mistakes and a refreshed UI which feels more natural. #4271

Here's a demo showing how it looks when adding a monorepo:

Add entity links card component

This one seems simple, but can actually be quite powerful. We've all been in scenarios where something is down in production and there is a pertinent Grafana chart which could unlock the problem but nobody knows how to find it. 🚨

Entity links provide a simple solution to this problem by providing you with a way to tag your services with links to other locations. The next time the service goes down the relevant charts should be linked off it in Backstage. This could save vital minutes in an outage.

To use it, simply add some links to your catalog-info.yaml file and they will show up in Backstage.

Props to Andrew Thauer from Wealthsimple for delivering this feature in chunks over the past few weeks.

Roadie news

We are currently migrating all of our Backstage plugins to the new plugin composability API.

The AWS Lambda, ArgoCD, GitHub Insights, Security Insights, Buildkite and Firebase Functions plugins are done.

The Pull Requests, JIRA and TravisCI plugins are in progress and should be finished any day now.

Backstage Weekly 16 - Upcoming Backstage community meetup 👯

David Tuite — Mon, 08 Feb 2021 09:59:14 +0000

The first Backstage community meetup of 2021 has been announced on the new Backstage community GitHub repo.

The meetup will take place on February 17th (just over a week from now) at 15:00 GMT on Zoom.

So far over 20 people have indicated their intention to come along. You can find Calendar links, the zoom meeting URL and other information on this GitHub issue. You can also suggest and vote on agenda items.

Roadie will be there in force and we'd ❤️ to see you there too.

Merged last week

111 pull requests were merged in the past two weeks (there was no newsletter last Monday). Here are the highlights.

Ecosystem explorer

In mid-January we mentioned that Systems, Domains and Resources had been added to Backstage as kinds of entities which could be modeled.

Last week, Oliver Sand from SDA-SE expanded on this feature to make Domains accessible through the Backstage UI. When you group your services into business domains, you should be able to step through them from this page to understand which services contribute to each business function. #4165

This PR also brings back the old Explore page in the form of Tools. Backstage users can use this to understand the various plugins that are available.

Azure Storage support for TechDocs

Vitor Grenzel from SSYS Sistemas added #4017 to allow Backstage users to back their technical documentation off to Azure Storage. All the major cloud providers are now supported, Google, AWS and Azure.

Using external storage for TechDocs, rather than the local filesystem, means you can move towards a compile on write (rather than read) architecture and unlocks faster load times for documentation in Backstage.

Team page is integrated

This may have happened at some point in the past and I just didn't notice it 😳 . Regardless, this is worth mentioning.

Catalog item ownership is now linked to groups of Backstage users, so you can click the owner of an entity to get some information about the team that created it.

In the screenshot below, you can see that team-a owns 4 services, 3 APIs and 1 website. There are 4 people on the team and their parent organisation is the Backstage org.

Roadie news

We published a new blog post which runs though a simple deployment of Backstage on Kubernetes.

I demoed an upcoming Roadie feature in this tweet. We're building the ability to dynamically add and remove plugins to Backstage (this will include bespoke custom plugins in the future). Our goal is to make it possible to get set up with Backstage in less than 5 minutes. As the tweet says, feedback and suggestions are welcome.

// Detect dark theme var iframe = document.getElementById('tweet-1356644531832250369-860'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1356644531832250369&theme=dark" }

Happy 1 year anniversary to Backstage 🎂

David Tuite — Tue, 26 Jan 2021 16:36:26 +0000

Today marks the one year anniversary of the very first commit to open source Backstage 🍾 Since then it has been... busy, with as many as 99 pull requests merged each week.

With more adoption, a bigger community and a strong direction, 2021 looks set to be an even bigger year. We can't wait to see what this awesome community produces.

Merged last week...

82 pull requests were merged last week. Here are the highlights.

New plugins: Snyk, Kafka and Opsgenie

This was a big week for new plugins with three additions to the plugin directory. Interestingly, all three plugins were contributed by non-Spotify engineering teams. More evidence that this community has critical mass.

Snyk, the developer-first security company, released their own Backstage plugin this week. As far as I can tell, this is the first official corporate plugin to be built for Backstage.

The Snyk plugin lets you quickly see the number of vulnerabilities present in your services so you can start to understand the security story across your fleet. #4213

Nir Gazit from Fiverr merged the first parts of a Kafka plugin. The functionality is simple for now, the plugin lists the offset status of consumer groups for services that have a Kafka annotation. It's a welcome start and it provides a good foundation for others to build on. #3985

Last, but not least, Kévin Gomez from electric scooter sharing company VOI Technology contributed an Opsgenie plugin. It supports tabs for alerts and incidents and an overview widget to show the latest alerts for a particular service. #4227

New starts: GitHub Apps, Scaffolder V2 and Search roadmap

This week also brought some exciting new beginnings in the Backstage codebase.

GitHub Apps will make most GitHub <> Backstage integrations simpler and support more granular permissions. Thankfully, Johan Haals from Spotify has begun the work in #4058.

One of the Backstage features I'm most frequently asked about is the Scaffolder and the functionality it supports for creating new services. In #3900, some parts of the existing Backstage Scaffolder have been deprecated to lay groundwork for Scaffolder V2. It's a small step but definitely pointed in the right direction.

A new roadmap for Search in Backstage has been added to the docs along with a proposal for production ready search architecture. Search improvements feature on the core roadmap for Backstage so we're expecting big improvements in this area #4030.

Roadie news

We got a shoutout on a recent episode of the Kubernetes Podcast thanks to friend of Roadie and CTO of the CNCF, Chris Aniszcyzk.

In the episode, Chris shares his technology journey and Cloud Native predictions for 2021.

// Detect dark theme var iframe = document.getElementById('tweet-1351645570100822016-839'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1351645570100822016&theme=dark" }

Roadie is hiring a remote JavaScript engineer who will work on our Backstage plugins and on Backstage directly. If you or someone you know would like to be paid to work on open source software then please, apply here.

Backstage Weekly - CNCF Sandbox ✅

David Tuite — Mon, 28 Sep 2020 14:04:12 +0000

💡 Backstage is a CNCF sandbox service catalog project. It helps companies to catalog their microservices and productize their path to production. Each week, we round up the latest Backstage community news.

Backstage has been accepted into the CNCF Sandbox. 🎉

The Sandbox is an entry point into the CNCF ecosystem. The goals of the sandbox include encouraging public visibility of projects and helping them grow to the point where they can move on to the next CNCF level. In addition to Backstage, the Sandbox holds projects such as chaos engineering Kubernetes operator Chaos Mesh and GitOps operator Flux.

This can only mean good things for those of us who are investing in Backstage and should help ease the minds of many organizations who were considering adopting it, either by deploying internally or by using a hosted Backstage provider such as Roadie.

The Backstage team wrote about the implications of this on their official blog post, and on the Spotify engineering blog.

The Backstage community is healthy and growing quickly. Over 130 people have contributed to the project, and roughly 40% of pull requests are now coming from external, non-Spotify, contributors. With companies now deciding to adopt Backstage we are also seeing a shift in the kinds of contributions we are getting from the community.

Merged last week...

70 pull requests we merged in the last week. Here are the highlights:

Matthew Clarke, who works on infrastructure at Spotify, had more than 7,000 lines of code merged for his Backstage Kubernetes plugin. This is going to be such an important plugin for cloud-native companies who want to use Backstage and could really ease onboarding in future. It's great to see progress being made. You can now list Kubernetes clusters and pods in Backstage. #2618, #2571 and others.
Fredrik Adelöw (Spotify) added a concept of Users to Backstage. #2587 and Althaf Hameez (Grab) added a concept of Groups #1838. Groups will associate users with teams, departments and organizations. The association between services and the teams that own them will be core to the value of Backstage and allow many powerful ownership oriented features in future.