Forem: Mark Roberts

Deployment Notes: How we document changes to production

Mark Roberts — Wed, 17 May 2023 05:00:00 +0000

At Propel, our continuous integration (CI) and continuous delivery (CD) processes enables us to deliver features rapidly, multiple times during a single workday. With each release to production, we’ve found it useful to document

customer-facing changes contained within the release,
non-customer-facing changes within the release, and
our strategy for rolling back the release.

We call this documentation “deployment notes” or “release notes” and we publish these notes on every release to an internal Slack channel. For example, you can see a screenshot of deployment notes for one of our services below:

Why do we do this?

Customer communication

For changes that are not gated behind a feature flag, the deployment notes provide a record of when the changes were actually shipped to production, and for customers, this is what matters. Even with continuous delivery, there will be a little bit of delay between merging to main and releasing to production. So as soon as the deployment notes go out, we know we can reach out to customers and let them know that a new feature or bug fix is available.

Additionally, we often refer to deployment notes when we put together our customer-facing changelog entries for the month. The deployment notes are the record of truth, so we don’t have to ask engineers, “Did feature X ship already?” We just go to our internal Slack channel.

Internal communication

Occasionally, larger changes will require multiple deployments of one or more services. These changes may not be immediately customer-facing; however, we still want to document and communicate where we are in the process, so that there’s never any doubt about the state of production.

For example, Propel’s GraphQL API is actually a GraphQL supergraph combining multiple subgraphs using Apollo Router. In order to extend our API with a new subgraph X, we would

Deploy a new service for subgraph X.
Update and deploy Apollo Router to include subgraph X.

The deployment in step 1 contains no customer-facing changes; however, we still write the deployment notes. The deployment notes serve as a record that step 1 was completed, allowing us to continue to step 2.

Externalizing this knowledge means no single engineer has to keep the state of production in their head and everyone can see what was deployed. And that can be really freeing when engineers go on vacation or when collaborating with teammates.

Incident response

Bugs are a fact of life, and the only way to eliminate them is to stop writing software. But we’re not in the business of not writing software. We’re in the business of shipping features that delight customers! That’s why we invest so heavily in our testing and continuous integration processes, in order to mitigate the likelihood of shipping unintended behavior changes to production.

Despite this, there will be behavior changes, bugs, and even serious incidents that automated testing doesn’t catch. When this happens, it’s the responsibility of an on-call engineer to remediate the issue as quickly as possible in order to restore the customer experience.

Rollbacks

Most of the time, pagers go off and incidents are discovered shortly after a deployment, and the quickest path to remediation is a rollback. By “rollback”, we mean rolling back to the previous known working version of the software. This is such a common remediation strategy that we emphasize it in our deployment notes. The instructions we usually give are

Re-deploy the last successful release.

Of course, there are some rollbacks that could be more complicated than a re-deploy, like a schema change; but, as a rule, we try design all of our changes to be easy to roll back.

Reconstructing timelines

Sometimes a bug or behavior change is discovered which does not correspond to a recent deployment. In these cases, accurate deployment notes have been essential for tracking down the source of the change and reconstructing the incident timeline. Having this clarity allows us to fix issues and resolve incidents with confidence. This is a crucial part of our incident response process that we will describe in a future blog post.

How do we do this?

Deployment notes are a core feature of our CI/CD pipelines. Once we merge a change to main, our pipeline deploys the change to our development environment, where it runs our cluster tests. If the cluster tests succeed, it posts to Slack requesting deployment notes and approval for deploying to production.

Our pipelines are currently running as GitHub Actions and using GitHub environments, but they used to run on CircleCI, so you could build a similar solution on any provider.

Curation vs. Automation

There are many tools that can automate the creation of changelogs and release notes. For example, we follow the Conventional Commits specification, which means we prefix our commit messages with “fix” or “feat” to indicate a bug fix or new feature, and we include “BREAKING CHANGE” if we need to ship a breaking change.

To be clear, we consider Conventional Commits a best practice! We even automatically bump version numbers according to the commit history. But we’ve found the automated changelogs derived from Conventional Commits to be too low-level for use in deployment notes. And when it’s handled automatically, we’re not necessarily “thinking” about the changes going out.

That’s why, on every release, we have the engineer in charge of the release pause and write the deployment notes. This allows the engineer to communicate at a higher level the changes going out and to review once more the changes being released.

Template

We always follow the same Markdown template, which you are free to re-use for yourself. If one of the sections is not applicable for a particular deployment, we write “N/A”.

# Customer-facing changes

- Cool new feature X
- Cool new feature Y

# Non-customer-facing changes

- Refactored Z

# Rollback strategy

Re-deploy the last successful release.

Conclusion

Deployment notes are an effective tool for communicating customer-facing and non-customer-facing changes. They’re also useful for incident response, where the on-call engineer needs to know how to rollback changes. Although tools exist to automate the creation of changelogs, we’ve found it best to explicitly author our deployment notes and require human approval before releasing them to production.

How we migrated to Apollo Server 4

Mark Roberts — Wed, 22 Feb 2023 16:44:00 +0000

At Propel, it’s no secret we’re fans of GraphQL: our entire product is centered around serving in-product analytics and customer dashboards using this flexible and performant technology.

We chose GraphQL first because it empowers clients to fetch exactly the data they need, usually in a single round-trip to the server. This is great for building customer dashboards, which may include multiple time series, counters, and leaderboards in a single page. It’s part of what makes using Propel so snappy.

Second, the developer experience around GraphQL is amazing, and we’ve been fortunate to use some great tools from The Guild and Apollo in building our product. For example, we publish our GraphQL schemas to Apollo Studio, we embed the Apollo Studio Explorer in our docs, and our GraphQL API is actually built on top of Apollo Server.

I want to take a moment to share part of our journey on the server-side, since we just recently upgraded our GraphQL APIs from Apollo Server 3 to 4. Apollo did a great job documenting the expected effort in their “Migrating to Apollo Server 4” guide, but we still encountered (and solved!) some unexpected differences in Apollo Server 4.

💡 Apollo Server 3 is officially deprecated and will reach end-of-life October 22, 2023. Read along to see how we at Propel migrated to Apollo Server 4.

Basic changes

The apollo-server library is now @apollo/server

In Apollo Server 3, you depended on version 3.x of the apollo-server library and any supporting libraries, like apollo-server-errors or apollo-server-core.

In Apollo Server 4, everything is published in a single, namespaced library. You only have to depend on version 4.x of @apollo/server.

New `startStandaloneServer` method

In Apollo Server 3, the apollo-server library wrapped apollo-server-express and offered a “batteries-included” HTTP server for handling GraphQL requests.

In Apollo Server 4, the “batteries-included” approach is now startStandaloneServer. Migrating to this new method is straightforward and well-described in the migration guide. At Propel, we tried to use this new method; however, for reasons described below, we ultimately needed to use expressMiddleware.

The `gql` template literal tag has been removed

In Apollo Server 3, you could import the gql template literal tag directly from the apollo-server library. This template literal tag is provided by the graphql-tag library and allows parsing a GraphQL query string to an AST that can be used by Apollo and other GraphQL libraries.

import { gql } from 'apollo-server'

In Apollo Server 4, this template literal tag is no longer exported. In fact, it’s no longer depended on by the @apollo/server library at all. At Propel, we had a few usages of gql in our backend, so we had to add a dependency on graphql-tag and change our import in order to fix this. Easy fix. 👍

import { gql } from 'graphql-tag'

ApolloError is replaced by GraphQLError

In Apollo Server 3, the apollo-server-errors library defined an ApolloError class. At Propel, we sub-classed ApolloError in our backend to create a hierarchy of errors. Any time we needed to throw an error from a resolver and have it appear in the GraphQL response’s errors array, we would use this class. For example, one of our NotFoundErrors looked like this:

import { ApolloError } from 'apollo-server-errors'

export class NotFoundError extends ApolloError {
  readonly name = 'NotFoundError'
  readonly code = 'NOT_FOUND'

  constructor (resourceName: string) {
    super(`${resourceName} not found`, this.code)
    Object.defineProperty(this, 'name', { value: this.name })
  }

  // Members omitted…
}

In Apollo Server 4, ApolloError is removed. Instead, we should now use the GraphQLError class from the graphql library. We already had a dependency on graphql, so we just needed to update our sub-classes. After migrating, our NotFoundError looked like this:

import { GraphQLError } from 'graphql'

export class NotFoundError extends GraphQLError {
  readonly name = 'NotFoundError'
  readonly code = 'NOT_FOUND'

  constructor (resourceName: string) {
    super(`${resourceName} not found`, {
      extensions: {
        code: this.code
      }
    })
    Object.defineProperty(this, 'name', { value: this.name })
  }

  // Members omitted…
}

Additionally, Apollo Server 4 adds the ability to control HTTP response status codes by including an http extension in the GraphQLError. For example, throwing a GraphQLError like the following will cause the HTTP server to respond with 418 I’m a teapot. 🫖

throw new GraphQLError("I'm a teapot", {
  extensions: {
    code: 'TEAPOT',
    http: { status: 418 }
  }
})

Plugin changes

Plugin error-handling has changed

In Apollo Server 3, plugins could receive the incoming request object in their requestDidStart method. At Propel, we used this to build an authentication and authorization plugin (our “AuthPlugin”).

Our AuthPlugin determines if incoming requests are authentic and if they are authorized. If an incoming request is inauthentic or unauthorized, we throw an HttpQueryError with status code 403. This would cause Apollo Server to return a 403 Unauthorized HTTP response back to the user. It looked something like this:

import { ApolloServer } from 'apollo-server'
import { HttpQueryError } from 'apollo-server-core'

class AuthPlugin {
  requestDidStart ({ request, context }) {
    if (!isAuthenticated || !isAuthorized) {
      throw new HttpQueryError(403, 'Unauthorized')
    }
    context.authCtx = authCtx
  }
}

const server = new ApolloServer({
  typeDefs,
  plugins: [new AuthPlugin()],
  context: () => ({})
})

In Apollo Server 4, not only is HttpQueryError removed, but throwing an error from a plugin’s requestDidStart method results in a 500 Internal Server Error. We already knew we would need to migrate from HttpQueryError to GraphQLError, but how could we control the HTTP status code if Apollo Server was re-throwing plugin errors as 500s?

After some head-scratching, I opened an issue on Apollo Server’s GitHub repository. There, Apollo Server contributor @glasser shared a helpful suggestion: why not invoke our AuthPlugin from Apollo Server’s context function? Throwing from context would ensure we can control the HTTP status response without having to introduce more methods and error checks to our AuthPlugin (like unexpectedErrorProcessingRequest). With that suggestion in mind, we rewrote our AuthPlugin as follows:

import { ApolloServer } from '@apollo/server'
import { startStandaloneServer } from '@apollo/server/standalone'
import { GraphQLError } from 'graphql'

class AuthPlugin {
  getAuthContext (req) { // ← Method renamed (not a true plugin anymore).
    if (!isAuthenticated || !isAuthorized) {
      throw new GraphQLError('Unauthorized', {
        extensions: {
          code: 'UNAUTHORIZED',
          http: { status: 403 }
        }
      })
    }
    return authCtx
  }
}

const server = new ApolloServer({
  typeDefs,
  plugins: [] // ← AuthPlugin is omitted here.
})

const authPlugin = new AuthPlugin()

startStandaloneServer(server, {
  context: async ({ req }) => {
    // Plugin invoked inside of `context`. Read on for details.
    return { authCtx: await authPlugin.getAuthContext(req) }
  }
})

It’s a few extra lines, but it works well. 👍

Plugins no longer receive the request URL

In Apollo Server 3, the incoming request object passed to requestDidStart included a URL property. Our AuthPlugin would read the request’s URL and headers to determine if the request was authentic and authorized.

In Apollo Server 4, this URL property has been removed. I was curious about this change, so I asked about it on Apollo Server’s GitHub repository. Apollo Server contributor @glasser helpfully pointed out that the URL was still accessible in Apollo Server’s context function. Because we had already moved our AuthPlugin inside of context, we could accept the request object provided by startStandaloneServer and extract the URL from there. The fix was straightforward:

startStandaloneServer(server, {
  context: async ({ req }) => {
    // `req` here is an `http.IncomingMessage` with `url`.
    return { authCtx: await authPlugin.requestDidStart(req) }
  }
})

Server changes

HTTP-level health check endpoint removed

In Apollo Server 3, there was an HTTP-level health check endpoint included at /.well-known/apollo/server-health. This would return 200 OK for any GET request. While too simple to evaluate if the server process was truly healthy, it did indicate whether the Node.js process and HTTP server were running. At Propel, we deploy our GraphQL API servers as Fargate tasks behind Application Load Balancers (ALBs), so this was a natural choice to configure as our target groups’ health check endpoint:

https://your.server/.well-known/apollo/server-health

In Apollo Server 4, the HTTP-level health check endpoint has been removed. Apollo Server recommends issuing a simple GraphQL query instead. For example,

https://your.server/?query=%7B__typename%7D

Indeed, this is a more sophisticated health check than the HTTP-level health check in Apollo Server 3: by executing a GraphQL query, we can ensure that Apollo Server is working in addition to the Node.js process and HTTP server.

At Propel, two issues prevented us from adopting this:

Our entire GraphQL API is protected by our AuthPlugin, so by default the ALBs’ health check requests would be rejected with status code 403.
Apollo Server’s CSRF protection would require the ALB to send an “Apollo-Require-Preflight” header with each health check request, which is unsupported.

Instead, we followed Apollo Server’s “Swapping to expressMiddleware” guide. By swapping to expressMiddleware, we would get access to the underlying express HTTP server. This way, we can install a health check endpoint at a new URL, /.well-known/server-health. This health check endpoint is completely private, and so it’s safe to leave unauthenticated.

const app = express()

// Setup Apollo Server with `expressMiddleware`…

app.get('/.well-known/server-health', (_, res) => {
  // Response follows https://tools.ietf.org/html/draft-inadarei-api-health-check-01
  res.type('application/health+json')
  res.json({ status: 'pass' })
})

We followed the same response format as Apollo Server 3. Indeed, this is a very simple check; however, we plan to incorporate more checks in the future.

Conclusion

Migrating from Apollo Server 3 to Apollo Server 4 took us less than a week, and we received a lot of help from Apollo’s own migration guide as well as on Apollo Server’s GitHub issues tracker. If any of this interests you — be it GraphQL or in-product analytics — reach out, follow us on Twitter @PropelDataCloud, or subscribe to our email newsletter.

If you’d like to learn more about our GraphQL API, check out our GraphQL API documentation or request access!

Forem: Mark Roberts

Deployment Notes: How we document changes to production

Why do we do this?

Customer communication

Internal communication

Incident response

Rollbacks

Reconstructing timelines

How do we do this?

Curation vs. Automation

Template

Conclusion

How we migrated to Apollo Server 4

Basic changes

The apollo-server library is now @apollo/server

New startStandaloneServer method

The gql template literal tag has been removed

ApolloError is replaced by GraphQLError

Plugin changes

Plugin error-handling has changed

Plugins no longer receive the request URL

Server changes

HTTP-level health check endpoint removed

Conclusion

New `startStandaloneServer` method

The `gql` template literal tag has been removed