Forem: Schaeffer

Distributed Tracing 101 for Full Stack Developers

Schaeffer — Thu, 12 Aug 2021 21:02:50 +0000

Welcome to Part 1 of our multipart series on Distributed Tracing for Full Stack Developers. In this series, we’ll be learning the ins-and-outs of distributed tracing and how it can assist you in monitoring the increasingly complex requirements of full stack applications.

In the early days of the web, writing web applications was simple. Developers generated HTML on the server using a language like PHP, communicated with a single relational database like MySQL, and most interactivity was driven by static HTML form components. While debugging tools were primitive, understanding the execution flow of your code was straightforward.

In today’s modern web stack it’s anything but. Full stack developers are expected to write JavaScript executing in the browser, interop with multiple database technologies, and deploy server side code on different server architectures (e.g. serverless). Without the right tools, understanding how a user interaction in the browser cascades into a 500 server error deep in your server stack is nigh-impossible. Enter: distributed tracing.

Me trying to explain a bottleneck in my web stack in 2021.

Distributed tracing is a monitoring technique that links the operations and requests occurring between multiple services. This allows developers to “trace” the path of an end-to-end request as it moves from one service to another, letting them pinpoint errors or performance bottlenecks in individual services that are negatively affecting the overall system.

In this post, we’ll learn more about distributed tracing concepts, go over an end-to-end tracing example in code, and see how to use tracing metadata to add valuable context to your logging and monitoring tools. When we’re done, you’ll not only understand the fundamentals of distributed tracing, but how you can apply tracing techniques to be more effective in debugging your full stack web applications.

But first, let’s go back to the beginning: what’s distributed tracing again?

Distributed tracing basics

Distributed tracing is a method of recording the connected operations of multiple services. Typically, these operations are initiated by requests from one service to another, where a “request” could be an actual HTTP request, or work invoked through a task queue or some other asynchronous means.

Traces are composed of two fundamental components:

• A span describes an operation or “work” taking place on a service. Spans can describe broad operations – for example, the operation of a web server responding to an HTTP request – or as granular as a single invocation of a function.

• A trace describes the end-to-end journey of one or more connected spans. A trace is considered to be a distributed trace if it connects spans (“work”) performed on multiple services.

Let’s take a look at an example of a hypothetical distributed trace.

The diagram above illustrates how a trace begins in one service – a React application running on the browser – and continues through a call to an API web server, and even further to a background task worker. The spans in this diagram are the work performed within each service, and each span can be “traced” back to the initial work kicked off by the browser application. Lastly, since these operations occur on different services, this trace is considered to be distributed.

Aside: Spans that describe broad operations (e.g. the full lifecycle of a web server responding to an HTTP request) are sometimes referred to as transaction spans or even just transactions. We’ll talk more about transactions vs. spans in Part 2 of this series.

Trace and span identifiers

So far we’ve identified the components of a trace, but we haven’t described how those components are linked together.

First, each trace is uniquely identified with a trace identifier. This is done by creating a unique randomly generated value (i.e. a UUID) in the root span – the initial operation that kicks off the entire trace. In our example above, the root span occurs in the Browser Application.

Second, each span first needs to be uniquely identified. This is similarly done by creating a unique span identifier (or span_id) when the span begins its operation. This span_id creation should occur at every span (or operation) that takes place within a trace.

Let’s revisit our hypothetical trace example. In the diagram above, you’ll notice that a trace identifier uniquely identifies the trace, and each span within that trace also possesses a unique span identifier.

Generating trace_id and span_id isn’t enough however. To actually connect these services, your application must propagate what’s known as a trace context when making a request from one service to another.

Trace context

The trace context is typically composed of just two values:

• Trace identifier (or trace_id): the unique identifier that is generated in the root span intended to identify the entirety of the trace. This is the same trace identifier we introduced in the last section; it is propagated unchanged to every downstream service.

• Parent identifier (or parent_id): the span_id of the “parent” span that spawned the current operation.

The diagram below visualizes how a request kicked off in one service propagates the trace context to the next service downstream. You’ll notice that trace_id remains constant, while the parent_id changes between requests, pointing to the parent span that kicked off the latest operation.

With these two values, for any given operation, it is possible to determine the originating (root) service, and to reconstruct all parent/ancestor services in order that led to the current operation.

A working example with code

To understand this all better, let’s actually implement a bare-bones tracing implementation, using the example we’ve been returning to, wherein a browser application is the initiator of a series of distributed operations connected by a trace context.

First, the browser application renders a form: for the purposes of this example, an “invite user” form. The form has a submit event handler, which fires when the form is submitted. Let’s consider this submit handler our root span, which means that when the handler is invoked, both a trace_id and span_id are generated.

Next, some work is done to gather user-inputted values from the form, then finally a fetch request is made to our web server to the /inviteUser API endpoint. As part of this fetch request, the trace context is passed as two custom HTTP headers: trace-id and parent-id (which is the current span’s span_id).

// browser app (JavaScript)
import uuid from 'uuid';

const traceId = uuid.v4();
const spanId = uuid.v4();

console.log('Initiate inviteUser POST request', `traceId: ${traceId}`);

fetch('/api/v1/inviteUser?email=' + encodeURIComponent(email), {
   method: 'POST',
   headers: {
       'trace-id': traceId,
       'parent-id': spanId,
   }
}).then((data) => {
   console.log('Success!');
}).catch((err) => {
   console.log('Something bad happened', `traceId: ${traceId}`);
});

Note these are non-standard HTTP headers used for explanatory purposes. There is an active effort to standardize tracing HTTP headers as part of the W3C traceparent specification, which is still in the “Recommendation” phase.

On the receiving end, the API web server handles the request and extracts the tracing metadata from the HTTP request. It then queues up a job to send an email to the user, and attaches the tracing context as part of a “meta” field in the job description. Last, it returns a response with a 200 status code indicating that the method was successful.

Note that while the server returned a successful response, the actual “work” isn’t done until the background task worker picks up the newly queued job and actually delivers an email.

At some point, the queue processor begins working on the queued email job. Again, the trace and parent identifiers are extracted, just as they were earlier in the web server.

// API Web Server
const Queue = require('bull');
const emailQueue = new Queue('email');
const uuid = require('uuid');

app.post("/api/v1/inviteUser", (req, res) => {
  const spanId = uuid.v4(),
    traceId = req.headers["trace-id"],
    parentId = req.headers["parent-id"];

  console.log(
    "Adding job to email queue",
    `[traceId: ${traceId},`,
    `parentId: ${parentId},`,
    `spanId: ${spanId}]`
  );

  emailQueue.add({
    title: "Welcome to our product",
    to: req.params.email,
    meta: {
      traceId: traceId,

      // the downstream span's parent_id is this span's span_id
      parentId: spanId,
    },
  });

  res.status(200).send("ok");
});

// Background Task Worker
emailQueue.process((job, done) => {
  const spanId = uuid.v4();
  const { traceId, parentId } = job.data.meta;

  console.log(
    "Sending email",
    `[traceId: ${traceId},`,
    `parentId: ${parentId},`,
    `spanId: ${spanId}]`
  );

  // actually send the email
  // ...

  done();
});

If you’re interested in running this example yourself, you can find the source code on GitHub.

Logging with distributed systems

You’ll notice that at every stage of our example, a logging call is made using console.log that additionally emits the current trace, span, and parent identifiers. In a perfect synchronous world – one where each service could log to the same centralized logging tool – each of these logging statements would appear sequentially:

If an exception or errant behavior occurred during the course of these operations, it would be relatively trivial to use these or additional logging statements to pinpoint a source. But the unfortunate reality is that these are distributed services, which means:

• Web servers typically handle many concurrent requests. The web server may be performing work (and emitting logging statements) attributed to other requests.

• Network latency can cloud the order of operations. Requests made from upstream services might not reach their destination in the same order they were fired.

• Background workers may have queued jobs. Workers may have to first work through earlier queued jobs before reaching the exact job queued up in this trace.

In a more realistic example, our logging calls might look something like this, which reflects multiple operations occurring concurrently:

Without tracing metadata, understanding the topology of which action invoked which action would be impossible. But by emitting tracing meta information at every logging call, it’s possible to quickly filter on all logging calls within a trace by filtering on traceId, and to reconstruct the exact order by examining spanId and parentId relationships.

This is the power of distributed tracing: by attaching metadata describing the current operation (span id), the parent operation that spawned it (parent id), and the trace identifier (trace id), we can augment logging and telemetry data to better understand the exact sequence of events occurring in your distributed services.

Tracing in the real world

Over the course of this article, we have been working with a somewhat contrived example. In a real distributed tracing environment, you wouldn’t generate and pass all your span and tracing identifiers manually. Nor would you rely on console.log (or other logging) calls to emit your tracing metadata yourself. You would use proper tracing libraries to handle the instrumentation and emitting of tracing data for you.

OpenTelemetry

OpenTelemetry is a collection of open source tools, APIs, and SDKs for instrumenting, generating, and exporting telemetry data from running software. It provides language-specific implementations for most popular programming languages, including both browser JavaScript and Node.js.

Sentry

Sentry is an open source application monitoring product that helps you identify errors and performance bottlenecks in your code. It provides client libraries in every major programming language which instrument your software’s code to capture both error data and tracing telemetry.

Sentry uses this telemetry in a number of ways. For example, Sentry’s Performance Monitoring feature set uses tracing data to generate waterfall diagrams that illustrate the end-to-end latency of your distributed services’ operations within a trace.

Sentry additionally uses tracing metadata to augment its Error Monitoring capabilities to understand how an error triggered in one service (e.g. server backend) can propagate to an error in another service (e.g. frontend).

You can learn more about Sentry and distributed tracing here.

Next: Span trees, OpenTelemetry, and Sentry

Congratulations: you’ve graduated from part 1 of our series on distributed tracing for full stack developers. At this point you should understand what a trace is, how trace information is propagated through services via tracing context, and how trace context can be used to connect logging statements between multiple distributed services.

Stay tuned for the next post in this series, where we’ll expand on the OpenTelemetry set of tracing tools, and take a further dive into how Sentry incorporates tracing context to augment error and performance monitoring telemetry.

Instrumenting Our Frontend Test Suite (...and fixing what we found)

Schaeffer — Tue, 27 Jul 2021 18:39:32 +0000

By: Billy Vong & Scott Cooper

Adding Instrumentation to Jest

Here at Sentry, we like to dogfood our product as much as possible. Sometimes, it results in unusual applications of our product and sometimes these unusual applications pay off in a meaningful way. In this blog post, we’ll examine one such case where we use the Sentry JavaScript SDK to instrument Jest (which runs our frontend test suite) and how we addressed the issues that we found.

We have high-level metrics for how well (or not) our CI is performing. This is useful to get a general sense of the health of the system, however, it does not help when doing a deeper dive into why a system is slow. We decided to instrument our CI jobs at the test runner level in order to get insights into the performance of the individual tests. We needed to be able to see if our test suite was slowing down because we were adding more tests or if it was because we had poor-performing tests.

As mentioned above, we use Jest as our test runner. It’s important to note, that our instrumentation method requires using jest-circus test runner. This is the default for Jest version 27, however, it can be used with earlier versions of Jest. jest-circus is required because it allows us to listen to events from Jest by using a custom environment and defining a handleTestEvent method. Below is a basic test snippet annotated with an approximation of where jest-circus’s events are fired.

// --> setup
// --> add_hook<beforeEach>
// --> start_describe_definition
// --> add_hook<beforeAll>
// --> add_hook<beforeEach>
// --> add_hook<afterEach>
// --> add_hook<afterAll>
// --> add_test
// --> finish_describe_definition
// --> run_start
describe('describe', function () {
  // --> run_describe_start
  // --> hook_start<beforeAll>
  // --> hook_success<beforeAll>

  beforeAll(function () {});
  beforeEach(function () {});
  afterEach(function () {});
  afterAll(function () {});

  // --> test_start
  // --> hook_start<beforeEach>
  // --> hook_success<beforeEach>
  it('test', function () {
    // --> test_fn_start
    expect(true).toBe(true);
  }); // --> test_fn_success
  // --> hook_start<afterEach>
  // --> hook_success<afterEach>
  // --> test_done

  // --> hook_start<afterAll>
  // --> hook_success<afterAll>
}); // --> run_describe_finish
// --> run_finish
// --> teardown

Sentry’s Performance Monitoring instrumentation is tree-like in structure with a transaction as the root node, and spans as child nodes. We can use a combination of the event name and the test name to determine if we should either create a transaction, create a child span from an existing transaction or end a span/transaction. Here’s an example of how we implemented it.

Fixing Performance Issues

Starting an investigation into a slow Jest test is a bit like peering into a black box because there’s so much abstraction. So the first steps are - find out which tests to look at and then find out what they’re spending time doing. The slowest tests were all larger page views with many subcomponents and components that access data from our data stores. Bigger components should be slower, right?

Looking at the “alert rules index” test we can see that the beforeEach function is getting slower after every test, yet the code in the beforeEach block remains the same and should take just as much time on the first test as it does on the last test. Looking at what is in the beforeEach function wasn’t enough to decide what was slowing it down. beforeEach contained a few api mocks and a call to ProjectsStore.loadInitialData([]); that was initializing the project store with an empty array. But none of that should be slow, so let’s have node tell us what is happening.

Start jest using the node debugger on just the one test file. The --inspect-brk flag tells node to wait until we have attached our profiler.

node --inspect-brk node_modules/.bin/jest --runInBand --no-cache tests/js/spec/views/alerts/rules/index.spec.jsx

After profiling the test and zooming into the area where the tests are running, we located the code that runs during the beforeEach. In this screenshot, we’ve renamed the anonymous beforeEach function to a named function badBeforeEach so it can be found easier. Below it is one call to ProjectsStore.loadInitialData followed by several calls to setState and a bunch of react work being kicked off. This shouldn’t be happening as there shouldn’t be a component listening to the store yet.

This led us to check that the component was being unmounted after each test. We added a componentWillUnmount to the class component being tested with a console.log inside to see if it was being called. When we did not see the console.log, we tested it manually by unmounting the component after every test.

The result: our beforeEach function takes a fraction of a second every iteration instead of running change detection in components that should have been unmounted and our test has its overall total time cut nearly in half. The enzyme docs warn you that calling mount instead of shallowMount will persist components in the DOM and the reason we’re seeing a larger impact on this test is because the store is still triggering change detection in components that should’ve been destroyed.

Using Sentry’s Performance Trends feature, we can confirm the performance impact these changes made to the tests.

Instrumenting your own tests

We provided an example Jest environment that you can use if you’re interested in instrumenting your own Jest tests. You’ll need to install the jest-sentry-environment package and update your Jest configuration to use the fore-mentioned package, as well as supplying your Sentry DSN. For further instructions, please see the repository.

How we updated our Content Security Policy using our own product

Schaeffer — Thu, 27 May 2021 19:12:12 +0000

By: Mark Story

Everyone talks about creating content. But what about controlling it? In this Dogfooding Chronicle, I’ll walk you through how we updated our Content Security Policy using Discover — and how we’re keeping watch with Dashboards.

Defining Content Security Policies

As the Internet evolves, so must our security mechanisms. It’s why same-origin policies (a Netscape-era directive in which resources from one origin are unable to modify or read data from another origin) put your application at risk for clickjacking and cross-site scripting attacks. That’s because same-origin policies have a weakness that enables third-party entities to have the same access rights as trusted entities. Content Security Policies (CSPs) address this blind spot by approving specific content types — everything from JavaScript sources to CSS fonts, HTML frames and embeddable Java applets.

Put simply, a CSP lets a browser know which content sources are to be trusted — and which aren’t. And when there is a CSP violation, browsers can submit the error to a report-uri. What’s more, if you have Sentry, these violation reports are integrated into your application’s monitoring dashboards.

During our most recent penetration test, we found that some of the CDN domains in our allow list were hosting potentially dangerous scripts. One of our recommendations from this audit was to refine and improve our CSP rules.

Discover-ing ‘report-only’ mode

Content Security Policies have two modes. The first mode enforces and actively blocks resource loading and execution, while the second collects the errors that would happen if the rules were active. This mode is set via the invaluable Content-Security-Policy-Report-Only header that defines your CSP rules.

When I combined this ‘report-only’ feature with Discover queries, I was able to visualize all the errors coming from report-only mode. Now I could view the impact of fixes in real time, and see which rules were being broken — without disrupting customers.

Sussing out Safari

While report-only mode worked well, I did encounter some hiccups in Safari, as it’s been known to jumble its report-only and enforced modes. This meant I was getting errors for sources that should be allowed. To verify that these errors were caused by Safari bugs, I used our staging environment to test Safari with only the new rules being enforced. And using Dashboards, I was able to fully view Safari’s impact on our CSP errors.

A glance at the solution

Like reading a good detective novel, once I solved this problem, another, more confounding problem bubbled up to take its place. Our Jira integration was having trouble integrating with our CSP. Because our Jira integration includes an embedded ‘glance view’, I could view Sentry data alongside other fields like assignee, priority and labels. After reviewing these embedded views, I discovered the issue: outdated libraries that relied on eval() expressions which are not allowed in our CSP rules.

While upgrading these libraries to newer versions resolved a large number of errors, it didn’t solve them all. And though I was unable to reproduce these last few errors, I was able to use Discover’s tag breakdowns to narrow things back down to — you guessed it — a Safari-specific issue:

CSP 3: Coming to a browser near you

The newest iteration of CSP has been in production for years, and is about to be fully released. Which is why we have defined rules for all three levels to get the broadest — and sturdiest — coverage possible. Doing so has both simplified our CSP configuration and improved our resilience to cross-site scripting. Which brings me to my last tip: it’s important to define directives in the order of their levels. Browsers will often stop parsing a directive when they find something they don’t know, so it’s important that there are specific directives for all three CSP levels.

Developing a Content Security Policy isn’t about being fussy — it’s about being functional. Data and code from untrusted sources should not have the same privileges as the trusted programmer’s code. By implementing a well-defined CSP, you can mitigate attacks, control assets, and breathe easier.

Great Moments in Application Monitoring

Schaeffer — Fri, 21 May 2021 19:53:17 +0000

The consequences of poor application performance are both real and terrifying: lost customers, lost trust from your team, and lost confidence in your abilities. It’s why every developer remembers those moments where their solution improves both process and performance. We’ve seen this happen firsthand with our customers, and want to share their Great Moments in Application Monitoring:

DoorDash

When you’re connecting hungry customers with hurried drivers, you can’t afford to have debugging eat up development time. Unfortunately, whenever a DoorDash developer tried something new without informing the larger team, resources would be wasted debugging their unknown logic. With Sentry’s automated stack traces, however, DoorDash developers could finally — and clearly — understand their colleagues’ code. By including $request_id in their access logs, passing that value downstream to their application servers, and then modifying the application’s entry point, DoorDash was able to utilize $request_id as a logging variable. Now, whenever there’s an error, DoorDash can search $request_id and immediately see the exact issue and exception. The result? Scalable production, the ability to trace every request end‑to‑end, and a process that’s as cool as a cucumber bisque.

"As the one service that’s consistently able to catch these types of NGINX variables, Sentry is really valuable at tracing exceptions of errors. And by providing automatic feedback, we’re able to see things clearly and gain a better understanding of what’s going wrong."

- Zhaobang Liu, DevOps Engineer, DoorDash

Eventbrite

For event marketer Eventbrite, their errors were talking over one another, as their team was constantly being distracted by alerts on duplicate issues. So they hired Sentry to emcee their errors. After instrumenting Sentry’s grouping and merging features, the Eventbrite team was able to create a list of errors prioritized by user impact and frequency. Unlike triaging in the past — where the Eventbrite team had to parse through countless duplicate issues — now they were only focused on solving what mattered most. What’s more, using Release Health, Eventbrite improved the quality of their releases by gauging adoption levels as well as the percentage of both crash-free sessions and crash-free users.

"Sentry helps our team fix the most important issues in each release. We can track how a release is trending by percent of crash-free sessions. With this data, we can remediate issues that impact the most users and move on to building more features."

- Jaylum Chen, Staff Software Engineer at Eventbrite

Cloudflare

The year was 2019. And Cloudflare’s release process consisted of manually tagging a release and then manually running through their dashboard to find errors. Not only was this process tedious, it made the Cloudflare team susceptible to errors and redundant work. But with Sentry’s Releases, the Cloudflare team was able to part the skies of gloomy manual work. By setting up an automated release plan, the Cloudflare team could now safely deploy code, monitor every step of their deployment process, and link all releases back to the Releases page. When automated tools — not manual brains — do the work, both Cloudflare’s code and their customers win.

"Sometimes errors on the front-end have roots on the backend. We use Sentry’s tags and metadata about a request that comes in to pass along a version of distributed tracing to link these back."

- Tony Stuck, Engineering Manager, Cloudflare

Volley

With trivia games like Song Quiz, Yes Sire, and Are You Smarter Than a 5th Grader?, Volley entertains millions of families on smart speaker devices every month. But with all those players blurting out answers at their voice-controlled devices, Volley engineers knew that they needed complete visibility into their code so they could properly scale their product.

Using our performance monitoring offering, the Volley team observed that their voice-controlled LaunchRequest was waiting for an API call to update the user’s subscription status. Once Volley moved this call to only those sections of the code which needed it, they noticed an improved Apdex score, faster user interactions, and their time to launch was cut in half.

"Sentry has become part of my daily routine. Every morning, I check to see if there are any new issues, and look at yesterday’s performance to see if any of our latest changes affect it negatively or positively."

- Devin Ozel, Sr. Software Engineer, Volley

With Great Moments in Application Monitoring, we recognize those intrepid developers who were able to get their application to perform the way it was intended - and the way their customers demand.

Supporting Native Android Libraries Loaded From APKs

Schaeffer — Thu, 13 May 2021 21:37:41 +0000

By: Arpad Borsos

Like mechanics who restore their own cars or plastic surgeons who self-rhinoplasty, our developers put their skills to interesting uses during their free time. Here, Native Platform Engineer Arpad Borsos breaks down how memory mappings and dynamic library loading works and how it relates to native Android libraries loaded from APKs.

Libraries are key to modular programming, as they offer functionality in a single unit which can be shared with other developers. As you’re no doubt aware, there are two types of libraries: static and dynamic. Static libraries are directly embedded into your application at build time, while dynamic libraries are linked when starting the application, or at any later point.

This concept is important because dynamic libraries can be updated without modifying the application itself for things like fixing security issues or improving performance. Applications can also be split into multiple, dynamically-loaded libraries for organizational reasons, or when an application consists of a User Interface and a separate service running in the background.

When working with the Android NDK, native libraries written in C, Rust, or similar low-level languages are loaded dynamically by the Java layer. That is how sentry-native gets loaded into Android Apps that use the NDK.

Usually, these libraries are individual files on disk. You might have seen a few of these “.dll” files next to Windows Applications, which are similar on other operating systems. The dynamic loader on Android, which is itself a system library, has the ability to load libraries directly from Android “.apk” packages without needing to first extract them to disk. This is quite beneficial, as it saves precious disk space on your mobile device.

So far, sentry-native, and in turn, NDK support for our Android SDK, relied on these files being extracted to disk. This has created a lot of friction — especially for new customers — as newer Android Versions no longer extract “.apk” packages by default. To get around this, application developers had to set some explicit configuration flags, many of which were frequently overlooked when setting up Sentry for Android.

Starting with version 5 of our Android SDK, we added support for libraries loaded from “.apk”, which will remove the need to change configuration flags while also improving disk space usage of Apps that use the Sentry SDK.

Let us take a deeper dive into how all of this works, and what we need to change in order to support this use case.

Which libraries are loaded?

On most platforms, you can query the list of loaded libraries directly from the dynamic loader via an API. For example, Windows has the Tool Help Library, and on Apple platforms there are some dyld functions available. Unfortunately, Linux has no standardized userspace tools. While GNU/Linux has the dl_iterate_phdr function, it is notably not available on older Android systems (Bionic Status lists the API as available starting with API 21, aka Android 5, released at the end of 2014). That means in order to support ancient Android versions, you will need to get the list of loaded libraries from somewhere else.

Standard practice here is to find the mapped ELF files by parsing the memory map info from /proc/XXX/maps. This is what Breakpad (in two places), Crashpad, Android’s libunwindstack, and LLDB do. In my opinion, they take this approach because they’re outside observers in the sense that they just can’t query the dynamic loader from inside the process.

This approach makes sense: it’s how I approached loading libraries for sentry-native. That said, you have to be careful to cover all cases — notably .so files loaded directly from inside Android .apk packages. And so, I went about finding ways to support these .apk cases.

The /proc/X/maps format

In Linux, all the executables and libraries are ELF files. Cloudflare has a great tutorial on how a loader parses and processes these ELF files. The documentation for the /proc/X/maps output format is described in this manpage. The format includes the start/end of the virtual address space, permission information, information about the inode (file), and the offset inside that file.

While there are cases when a library needs just one mapping, most of the time, it’s split into two or more mappings. Usually that consists of a read-only mapping that includes the ELF headers and metadata, and an executable mapping that holds the actual program code. On my Linux system, I saw up to six mappings for a single file:

7f8cd3467000-7f8cd3475000 r--p 00000000 00:1c 7597971                    /usr/lib/libcurl.so.4.7.0
7f8cd3475000-7f8cd34da000 r-xp 0000e000 00:1c 7597971                    /usr/lib/libcurl.so.4.7.0
7f8cd34da000-7f8cd34f6000 r--p 00073000 00:1c 7597971                    /usr/lib/libcurl.so.4.7.0
7f8cd34f6000-7f8cd34f7000 ---p 0008f000 00:1c 7597971                    /usr/lib/libcurl.so.4.7.0
7f8cd34f7000-7f8cd34fa000 r--p 0008f000 00:1c 7597971                    /usr/lib/libcurl.so.4.7.0
7f8cd34fa000-7f8cd34fc000 rw-p 00092000 00:1c 7597971                    /usr/lib/libcurl.so.4.7.0

The interesting case here is that the fourth mapping is not readable, and basically creates a gap in the address space.

These two mappings load the exact same libraries, once extracted to disk, once directly from the apk:

```77a85dbda000-77a85dbdd000 r-xp 00000000 fd:05 40992 /data/app/x/y/lib/x86_64/libsentry-android.so
77a85dbdd000-77a85dbde000 ---p 00000000 00:00 0
77a85dbde000-77a85dbdf000 r--p 00003000 fd:05 40992 /data/app/x/y/lib/x86_64/libsentry-android.so
77a85dc15000-77a85dd6c000 r-xp 00000000 fd:05 40991 /data/app/x/y/lib/x86_64/libsentry.so
77a85dd6c000-77a85dd6d000 ---p 00000000 00:00 0
77a85dd6d000-77a85dd79000 r--p 00157000 fd:05 40991 /data/app/x/y/lib/x86_64/libsentry.so
77a85dd79000-77a85dd7a000 rw-p 00163000 fd:05 40991 /data/app/x/y/lib/x86_64/libsentry.so






```77a85dbf0000-77a85dbf3000 r-xp 00001000 fd:05 40977                      /data/app/x/y/base.apk
77a85dbf3000-77a85dbf4000 ---p 00000000 00:00 0
77a85dbf4000-77a85dbf5000 r--p 00004000 fd:05 40977                      /data/app/x/y/base.apk
77a85dc15000-77a85dd6c000 r-xp 00006000 fd:05 40977                      /data/app/x/y/base.apk
77a85dd6c000-77a85dd6d000 ---p 00000000 00:00 0
77a85dd6d000-77a85dd79000 r--p 0015d000 fd:05 40977                      /data/app/x/y/base.apk
77a85dd79000-77a85dd7a000 rw-p 00169000 fd:05 40977                      /data/app/x/y/base.apk

The mappings are basically the same — it’s just that in the case of the base .apk, the file offsets are different. And the Android loader, again, inserts a non-readable gap in between.

So how do we get the library list from there?

So far, the sentry-native modulefinder implementation has been a bit too conservative. Because of concerns reading arbitrary memory, we mmap-ed the file into memory and tried to extract ELF headers. Unfortunately, that approach doesn’t work with those libraries that are loaded directly from apk files, as the ELF headers are at a certain offset in that file. Plus, as we demonstrated above, some issues related to non-contiguous mappings and double mappings caused problems in the old implementation, as it worked based on the filename that it saw.

So my new approach is to keep track of readable mappings, their file offsets, and the gaps between them. For each readable mapping, I am looking for the magic ELF signature. If I find one, I process the previously saved mappings, while also taking care of possible duplicates.

This approach still has unanswered questions. One is trying to read arbitrary memory. I think I’m pretty safe as I’m only considering readable mappings, but one improvement would be to use process_vm_readv here. That said, I have also seen problems with using that mapping on Android. Another potential issue is how to correctly deal with mappings which have gaps in them, or even ones that appear multiple times. The ELF file might instruct the loader to load the executable code at a different offset to the ELF header in RAM than the offset on disk — or it might not. It very much depends on how well we use this information to post-process crash reports.

This problem is not unique to the way that sentry-native used to read the list of libraries. We also saw some breakpad tools get this wrong by creating minidumps with invalid mappings that fail later on in the post-processing pipeline.

Loading libraries is a non-trivial problem, and I am confident that I’m not the only one struggling with it. And make no mistake: it’s some work to investigate failures and patch the relevant code for such a specific use case. But with Android adoption picking up more and more traction, it’s necessary work that will save space for your user - and stress on yourself.

Slow and Steady: Converting Sentry’s Entire Frontend to TypeScript

Schaeffer — Tue, 13 Apr 2021 19:59:24 +0000

Recently, Sentry converted 100% of its frontend React codebase from JavaScript to TypeScript. This year-long effort spanned over a dozen members of the engineering team, 1,100 files, and 95,000 lines of code.

In this blog post, we share our process, techniques, challenges, and ultimately, what we learned along this journey.

The pitch

Back in 2019, we were shipping more frontend bugs than what was acceptable. After looking at the underlying causes of these incidents, it became clear that many of these bugs could have been prevented by static analysis and type checking.

During that year’s Hackweek event, Lyn Nagara, Alberto Leal, and Daniel Griesser pitched introducing TypeScript to the Sentry frontend. This team bootstrapped the TypeScript compiler to our build process as well as converted a few non-trivial views — and their related components — to TypeScript.

Hackweek is an event that takes place once a year, giving all Sentry employees the opportunity to set aside their usual work to focus solely on innovative projects and ideas. Hackweek has given birth to numerous applications and tools that are now important parts of our product, like the recently launched Dark Mode project.

Typescript project presentation during Hackweek 2019

After considering the presentation, we felt Typescript was a strong fit for Sentry because:

Several classes of bugs could be detected — and eliminated — during compilation.
We could improve the developer experience through editor integrations such as auto-completion, faster code navigation, and inline compiler feedback.
We could reduce the need for API documentation, as type annotations help produce self-describing code.
TypeScript has an active community with a clear and maintained development roadmap in addition to rapid releases.
Many of the libraries we use (including React) already have type definitions available.
TypeScript can be adopted incrementally. That meant we can start writing new code with TypeScript and incrementally convert over time.

There were some potential drawbacks of adopting TypeScript, though:

It’s a large time investment. Our frontend code is non-trivial in scope, so it would take significant effort to convert it. That complexity meant additional build time.
We would need to educate the frontend team in TypeScript and support them as they learned.
TypeScript and JavaScript would need to coexist in the code base for a significant period of time.

Maturing the prototype

Shortly after Hackweek, excitement was high and a more formal proposal was brought to our Frontend Technical Steering Committee (TSC). This group meets every two weeks to guide our frontend architecture. While TypeScript wasn’t among the “winning” projects for Hackweek, we were confident that it would be a worthwhile investment that would ultimately pay off in the long run.

Sentry’s first Typescript Pull Request: https://github.com/getsentry/sentry/pull/13786

Overall strategy

We broke our high-level strategy into several phases:

Educate. In this phase, we needed to let people know that TypeScript was coming, and provide the right learning resources to help folks onboard.
New code in TypeScript. In this phase, we needed to have all new development being done in TypeScript. If we continued to create new JavaScript, we would never finish the conversion phase.
Conversion. In this phase, all new work would be done in TypeScript, giving us a finite number of files to convert. Then it is “just work”™️.
Our most controversial decision was agreeing to not undergo any other major refactors until the code base was converted 100% to TypeScript. This meant we would not take on other quality-of-life improvements — things like upgrading our state-management library or introducing React hooks — until the TypeScript conversion was complete.

Educating the team

Early on, we recognized that the broader development team at Sentry would need additional resources and materials to learn TypeScript. To help folks who were new to TypeScript, we shared a list of introductory articles and resources for configuring various editors.

Additionally, members of the TSC took the time to review code and help educate those folks eager to learn TypeScript. Having this support system in place helped create more TypeScript “believers” who would, over time, write new code in TypeScript.

Example of a TypeScript code review.

Slack shoutout for TypeScript conversion.

Taking root in green fields

While we were educating the broader team, folks who were keen on TypeScript not only began building out their new feature work in TypeScript, but also found opportunities to convert files which overlapped with new features. This approach let us build up our type definitions and gain more experience writing Typescript in lower-risk parts of the product that were not exposed to customers.

As the broader team gained more experience and found value in what TypeScript provides, they naturally stopped creating more JavaScript. While we never used tooling to stop people from creating new JavaScript, our education efforts and social agreements helped prevent new JavaScript from being created.

Work out from the core — and in from the leaves

Once TypeScript had a firm foothold, we needed a strategy to work through the 1,100+ files that needed conversion. Here, we audited our imports, ordering them by how frequently each module was imported. We used this list to prioritize which modules were converted first. By converting frequently used modules, we would be able compound our gains as we converted files.

The list we used to prioritize and assign conversion work based on import frequency.

This approach worked well in the beginning, as some modules have dramatically more imports than others. But because most of our modules have fewer than 10 imports, we quickly plateaued. Our next approach was starting from “leaf node” modules that are imported in one place. Converting these files enabled us to accumulate progress more quickly.

All we have to do now is convert 1,100+ files…

Like many software projects, our initial rollout plans were overly ambitious. We started off by retroactively calculating a timeline where we completed within 2019. With approximately 15 weeks before the end of the year, that meant would need to convert approximately 74 files per week. This assumed that we would not accumulate any additional JavaScript files (we did) and that we could sustain that effort (we didn’t). After eight weeks, we checked in on our progress.

Despite promising progress during our first 8 weeks, we knew we weren’t going to finish in 2019.

It was obvious that we were not going to complete it in 2019. Given projections from the current effort, a more likely completion date would be mid-2020.

During the fall and winter of 2019, progress was slow. People were focusing on meeting product goals and didn’t have as much time to devote to TypeScript conversion. In February of 2020, we reached equilibrium. We were no longer making new JavaScript and our backlog of conversion work became fixed.

Our conversion progress over time. Note there are several lulls as well as periods of renewed activity.

Challenges encountered

Although the introduction of TypeScript was definitely a game-changer, we also faced a few challenges during the conversion process. Most of these were due to interoperability issues between TypeScript and React:

1.Default Props

When using defaultProps on classes, TypeScript is able to correctly infer that the props are not required when using the component, but when using Higher Order Components, the types for defaultProps generally don’t work, and the previously optional properties would become required.

An example of how defaultProps interacts poorly with Higher Order Components is:

const defaultProps = {
  statsPeriod: DEFAULT_STREAM_GROUP_STATS_PERIOD,
  canSelect: true,
  withChart: true,
  useFilteredStats: false,
};

type Props = {
  id: string;
  selection: GlobalSelection;
  organization: Organization;
  displayReprocessingLayout?: boolean;
  query?: string;
  hasGuideAnchor?: boolean;
  memberList?: User[];
  onMarkReviewed?: (itemIds: string[]) => void;
  showInboxTime?: boolean;
  index?: number;
} & typeof defaultProps

type State = {...};

class StreamGroup extends React.Component<Props, State> {
  static defaultProps = defaultProps;
  ...
}

export default withGlobalSelection(withOrganization(StreamGroup));

Normally, TypeScript would be able to use the defaultProps attribute of our class component to infer that those properties are not required. However, when wrapped in a Higher Order Component, TypeScript displays the following errors:

Here our solution was to use Partial on the defaultProps and rely on React to fill in the default values.

const defaultProps = {
      statsPeriod: DEFAULT_STREAM_GROUP_STATS_PERIOD,
      canSelect: true,
      withChart: true,
      useFilteredStats: false,
    };

    type Props = {
      id: string;
      selection: GlobalSelection;
      organization: Organization;
      displayReprocessingLayout?: boolean;
      query?: string;
      hasGuideAnchor?: boolean;
      memberList?: User[];
      onMarkReviewed?: (itemIds: string[]) => void;
      showInboxTime?: boolean;
      index?: number;
    } & Partial<typeof defaultProps>

    type State = {...};

    class StreamGroup extends React.Component<Props, State> {
      static defaultProps = defaultProps;
      ...
    }

    export default withGlobalSelection(withOrganization(StreamGroup));

You can find a more complete implementation of this approach here.

2.Libraries adding incorrect types

One downside to relying on the type definitions in DefinitelyTyped is that occasionally the library types are not written by the maintainers. Instead, community users contribute types, and because of that some types are missing or incorrectly defined. We encountered this with the versions of ECharts and Reflux we were using. Our solution here was to add additional type definitions in our code.

3.React.forwardRef is not compatible with generics

Using generic types with React.forwardRef is not directly possible, as it requires concrete types. In more detail, the forwardRef function has only one parameter named render. The type of this parameter is ForwardRefRenderFunction, which is not a generic function declaration, so higher order function type inference cannot propagate free type parameters on to the calling function React.forwardRef. We had to make compromises and use “any” when this situation arose.

Sustaining motivation and energy

Toward the end of the conversion, many contributors were feeling the burning toil this project was having.

In the summer of 2020 – a full year after this project began – we crossed the 70% threshold. This revitalized folks, as we knew the end was near. We were able to sustain that energy and focus though the summer and fall by using part of our TSC meeting as a check in and collecting “conversion pledges” for the next meeting. This introduced a light-hearted social game that helped us stay focused.

In addition, our fantastic tools team introduced a slackbot that would allow us to track progress on demand. Seeing the number go up every day was a big motivator in the final stages, so much so it’s something we’ll likely to use again. You can find the early versions of that bot here.

The end, finally

After 18 months of migrating our frontend code base to TypeScript, the day everyone at Sentry had been working toward had finally arrived. When we started on our TypeScript journey, we had 1,100+ files to convert. Now, we have over 1,915 Typescript files. It’s worth mentioning that at no time was a GitHub check added to block new JavaScript files. After developers saw the benefits that TypeScript would bring, writing new code in TypeScript was an organic choice.

With TypeScript, we now have an extra layer of protection in our code, which means we’re able to ship with greater confidence, higher productivity, and most importantly, fewer bugs. Some of our newer frontend developers have never seen a production incident caused by a frontend change.

Looking back

Like everything in life, we also learned a few things along this journey.

1.Incremental conversion is key

Our strategy to gradually migrate our files to TypeScript worked out well. We were able to balance converting our code to TypeScript, without delaying important product work. It’s important to highlight that from the beginning, we were not in a hurry to achieve our goal, but instead we wanted to proceed carefully and do a great job.

2.Stay current with TypeScript releases

During our conversion process, several new releases of TypeScript came out. Each one helped us refine our types further with new features like optional chaining, nullish coalesce, named tuples, and more. While upgrading did take additional effort, the benefits were well worth it. It’s why we recommend staying as current as you can with TypeScript releases.

3.Gradually build complex types

At the beginning of the migration, it was impossible to know the correct type of everything. After all, Sentry possesses a large code base, and not everyone is familiar with all parts of the application. Which meant we had to build our more complex types incrementally. As we were converting files, we became more familiar with their types and as we converted related files, we were able to better identify whether the types we had previously defined were updated with the new insights.

4.Use TODO Comments to note future work

In Sentry, we generally use TODO comments in the code to help us track something we need to review later. This approach proved to be very useful during our migration to TypeScript. When we encountered an unclear type, or problematic component we would leave a TODO(ts) for later review. We are now incrementally reviewing the TODO list and further refining and improving our types.

import {Frame} from 'app/types';

// TODO(ts): define correct stack trace type
function getRelevantFrame(stacktrace: any): Frame {
  if (!stacktrace.hasSystemFrames) {
    return stacktrace.frames[stacktrace.frames.length - 1];
  }
  for (let i = stacktrace.frames.length - 1; i >= 0; i--) {
    const frame = stacktrace.frames[i];
    if (frame.inApp) {
      return frame;
    }
  }
  // this should not happen
  return stacktrace.frames[stacktrace.frames.length - 1];
}

export default getRelevantFrame;

Moving forward

Migrating to TypeScript was just the beginning. The frontend team at Sentry will continue to gradually improve types, making sure they are correct as possible, including the removal of all React PropTypes.

We’re also seriously considering introducing end-to-end type safety, so that a backend engineer can make changes in the API without unknowing breaking clients, and frontend engineers can be confident in the data that will be coming back from the server.

This important achievement would not have been possible without the patience, persistence, attention to detail, passion and hard work of everyone involved. A big thank you to all the Sentaurs who contributed to this enormous effort.

Eager for a challenging project like this one? Then join us at Sentry. We are hiring!