Forem: onesignaldevs

Fixing Memory Leaks in Rust

onesignaldevs — Mon, 23 May 2022 23:35:29 +0000

At OneSignal, we love Rust. We have blogged before about pivoting some of our core business systems to Rust, how the language has changed over the past several years, and interesting things we’ve learned about the thread safety model. Other than Onepush, the core of our push notification delivery system, we also use Rust for a gRPC service and multiple Kafka consumers. In this article, we’re going to talk about a stumbling block that we ran into with one of our newest Rust projects.

The Challenge

Near the end of February, we announced the general availability of our Journeys feature. Journeys allows customers to easily build complex messaging workflows with a no-code UI. It is powered by a few gRPC services and a Kafka consumer written in Rust, called JourneyX. Journeys is an event-driven system, so everything that happens within a journey workflow is triggered by an event on a Kafka stream. These events are consumed by JourneyX (journey executor).

A few weeks ago, as Journeys adoption started to increase and JourneyX started to process more events, we began to notice a disturbing pattern in its memory usage. The most active processes were consistently utilizing lots of memory, then getting killed by the kernel. The Linux kernel has a piece of functionality called the OOM (out-of-memory) killer, which will automatically kill processes when they consume too much of the system memory. This prevents the system from becoming unstable or locking up due to resource starvation. In our case, the JourneyX processes were constantly being killed, restarted, and killed again by the OOM killer. This showed up on a graph as a sawtooth pattern of rapid allocation and near-instant deallocation when the process was killed. Memory usage would spike up to 17 GiB in the busiest processes, but we expect one of these worker processes to need under 1 GiB.

Graph of memory usage for a single JourneyX process over 24 hours

This caused a few issues for us. Obviously from a theoretical perspective, we didn’t want a system that was constantly being killed by the OS. The operations it performed were idempotent, so we weren’t necessarily worried about sending multiple notifications due to this problem. The issue did create alert spam for us, which led us to over-provision on memory. The constant crashing also caused concern about the long-term health of the service.

BUT WAIT! I hear you say — doesn’t Rust’s borrow checker prevent memory errors? Isn’t Rust supposed to be “safe?” It turns out that, according to Rust’s rules, leaking memory is completely safe! In fact, we can purposely leak as much memory as we want using the function std::mem::forget. The only thing “unsafe” about memory leaks is that they might eventually result in your program being killed by the kernel (as JourneyX does in this case). A program ending in a predictable way is also considered safe behavior. Rust’s safety guarantees exist to protect us from invalid memory access, not resource starvation.

Troubleshooting with Distributed Tracing

In addition to Rust, we’re big fans of distributed tracing. If you’re not familiar with tracing concepts, check out this document from Honeycomb's Introduction to Distributed Tracing. All of our services report trace data and send W3C trace propagation headers to one another so that we can tie operations together through our system. Tracing data in Honeycomb is now our first stop when our on-call team gets paged. Here is an example trace from our Journeys processing system.

In some languages, tracing instrumentation can be accomplished largely through automated means. In our Ruby on Rails codebase, the Honeycomb and OpenTelemetry libraries use ActiveSupport hooks to add trace spans around operations like HTTP handlers and ActiveRecord queries. Rust is a lot of things, but it's not really “runtime configurable.” There are libraries that allow you to write Rust code that can be configured at runtime, but most Rust code is not written in this way. Because of this, we need to add a lot of manual instrumentation to our Rust apps. We need to manually delineate operations into spans and add fields on those spans.

We have a few choices for adding these data to our codebases. We could use the Rust opentelemetry library, which is generic and standardized according to the OpenTelemetry specification. Instead, we use the tracing library, which is a façade crate, similar to log. It allows us to connect and configure multiple tracing backends, and it captures data from the log crate straight out of the box.

If we had some Rust code like this:

fn http_handler() {
    let data = get_data();
    let result = perform_expensive_computation(data);
}

And you wanted to put a span around the perform_expensive_computation function, you could do so like this:

use tracing::info_span;

fn http_handler() {
    let data = get_data();
    let span = info_span!(?data, "perform_expensive_computation");
    let _guard = span.enter();
    let result = perform_expensive_computation(data);
}

The enter function returns a guard that marks the time in which the span is active. It begins when you first call enter, and ends when the guard value is dropped as it goes out of scope. Rust’s Drop trait allows it to easily provide the functionality to run at the time it’s deallocated.

Within JourneyX, surrounding the “send a notification” HTTP request, we had the following snippet:

let (res, body) = {
    let query_span = tracing::info_span!("send HTTP request");
    let _guard = query_span.enter();
    self.client.request(body).await?.into_parts()
};

Now experienced users of the tracing library may already know the issue, but to those of us who aren’t intimately familiar with the API, this looks quite reasonable based on what I’ve already told you. If we look at the documentation for the enter method though, we may notice something that calls this into question.

Warning : in asynchronous code that uses async/await syntax, Span::enter should be used very carefully or avoided entirely. Holding the drop guard returned by Span::enter across .await points will result in incorrect traces.

Notice the last line of the block above uses .await when calling the request method. This is going to cause a problem for us. To see why, we need to look at the conceptual models for tracing and async Rust.

The .enter method from tracing returns what is conventionally referred to as a “guard” type. One of the great benefits of Rust’s ownership and lifetime system is that it provides the ability to write destructors for any type and to be sure about when those destructors are going to run. We can do this by implementing the Drop trait for any type. The interface looks (very, very roughly) like this:

struct EnterGuard {
}

impl Span {
  fn enter(&self) -> EnterGuard {
    begin_span(self.id);
    EnterGuard { id: self.id }
  }
}

impl Drop for EnterSpan {
  fn drop(&mut self) {
    end_span();
  }
}

Because of Rust’s destructor rules, the EnterSpan type will call the end_span function whenever the variable it’s attached to goes out of scope. Here is an annotated example:

fn do_something() -> i32 {
  let x = {
    let span = Span::new();
    let _guard = span.enter(); // begin_span called
    do_expensive_computation()
  }; // end_span called by the Drop of `_guard`

  let span = Span::new();
  let _guard = span.enter(); // begin_span called
  more_compute(x)
} // end_span called by the Drop of `_guard`

This is great because it means that most of the time, we just don’t need to worry about remembering to mark a span as complete. It just works (tm). For synchronous code, that’s definitely the case, but it falls apart for async code. To see why that's the case, let’s look at the pseudocode for begin_span and end_span.

thread_local! {
  static ACTIVE_SPANS: RefCell<Vec<SpanId>> = RefCell::new(Vec::new());
}

fn begin_span(id: SpanId) {
  ACTIVE_SPANS.with(|s| s.borrow_mut().push(id));
}

fn end_span() {
  ACTIVE_SPANS.with(|s| s.borrow_mut().pop());
}

Now this is incredibly simplified (please don’t get mad at me, tracing maintainers) but the general idea is that each thread maintains a stack of span IDs that represents the hierarchy of spans currently active. This works great for synchronous code, when each OS thread can maintain this state independently. When we use async code, however, we have multiple tasks running concurrently on top of the same OS thread, sharing thread locals. And the Drop handler for our EnterGuard won’t be called when context switches happen. That means if we were to write code like this:

let mut tasks = futures::stream::FuturesUnordered::new();
for id in 0..5 {
  tasks.push(async move {
    let span = info_span!(id, "task");
    let _guard = span.enter();
    tokio::time::sleep(std::time::Duration::from_millis(100)).await;
  });
}
futures::future::join_all(tasks).await;

Let’s consider what happens to our thread-local stack at each stage in the task execution. For simplicity's sake, we’ll assume for now that these five tasks are executed in the order that they were enqueued, although this typically isn't the case.

Notice that at both the beginning and end of the spans there are large discrepancies in the state of the stack. When spans are beginning, we create a hierarchy on the stack that indicates all of the tasks have a parent/child relationship with one another, but this is not accurate. When spans end, we pop the wrong task span off of the stack, because the state of the stack is inconsistent with reality.

In our production code, this issue caused memory usage to skyrocket, because our span tree was essentially expanding endlessly and never able to shrink. Because of how quickly concurrent operations were occurring, the stack was continuously having new spans added to it, and previous spans (now with tens of thousands of children) would never be dropped.

The Solution

So what’s the fix? It’s very simple. Recall that the original code looked like this:

let (res, body) = {
    let query_span = tracing::info_span!("send HTTP request");
    let _guard = query_span.enter();
    self.client.request(body).await?.into_parts()
};

We need to update it to this:

let (res, body) = {
    let query_span = tracing::info_span!("send HTTP request");
    self.client.request(body).instrument(query_span).await?.into_parts()
};

It’s a subtle change, but absolutely critical. Let’s see how this works. This is once again simplified for the sake of brevity.

pub trait Instrument {
  fn instrument(self, span: Span) -> Instrumented<Self> {
    Instrumented { span, inner: self }
  }
}

pub struct Instrumented<T> {
  span: Span,
  inner: T,
}

impl<T> Future for Instrumented<T> where T: Future {
  type Output = T::Output;

  fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
      let _enter = self.span.enter();
      self.inner.poll(cx)
  }
}

There are a few things going on here. First off, we have the Instrument trait. This is an extension trait that exists only to allow us to build an Instrumented struct from an arbitrary Future. Instrumented is a wrapper future that allows us to have tight control over the polling logic. In this case, we use .enter to begin the span while the underlying future is polled, and end the span when the context switches.

This works because poll is the context-switching boundary of a future. While poll is being called, this future and its children are the only things burning CPU time on the thread. Other futures may be sleeping in the background, but nothing else can preempt the currently running future’s poll.

Tracing’s .enter method isn’t the only instance of this. There are many types that were not designed to be held across await points. Mutex, RwLock, and RefCell for example all have guard types that hold a lock on a resource and could result in deadlocks if held across await points.

So where do we go from here? There’s a subtle code difference that results in both incorrect observability results and potentially extreme memory usage. Normally, issues like this could be caught by the Rust compiler. There is an RFC to add a lint to the compiler called must_not_suspend which will allow developers to mark types as being unsafe to hold across await points. But there is still a lot of work remaining before this will land, so what should we do in the meantime?

Rust’s standard linting tool, Clippy, has existing lints for the Mutex and RefCell types, but we don’t want to have to create a custom Clippy lint for every type like this. A few weeks ago, I submitted rust-lang/clippy#8707 which allows Clippy users to provide a list of types that are not allowed to be held across await points. Once this makes it to Clippy’s stable release branch, users will be able to add the following to their clippy.toml files:

await-holding-invalid-types = [
  "tracing::trace::Entered",
  "tracing::trace::EnteredSpan",
]

This will provide users with a warning if a span’s enter guard is held across an await point when they run cargo Clippy. The warnings will look like this:

error: `tracing::trace::Entered` may not be held across an `await` point per `clippy.toml`
  --> main.rs:10:9
   |
LL | let _guard = span.enter();
   | ^^^^^^

This provides some amount of safeguarding, though the must_not_suspend lint will be a welcome improvement, as it will not require an explicit configuration in order to work.

Thank you for coming with me on this journey of memory leak discovery! I’ll leave you with a four-day graph of the memory usage of our JourneyX processes. It’s quite clear when this change was deployed that we instantly fixed the issue with this very subtle code change.

A graph of the memory usage of our JourneyX processes over four days.

If you’re interested in working on using Rust in production to create high-throughput evented systems, distributed systems, or any other areas of engineering, please check out the OneSignal careers page!

>> Browse OneSignal Careers

How to Add Push Notifications to a Flutter App

onesignaldevs — Mon, 14 Mar 2022 18:35:30 +0000

Push notifications have proved to be one of the most effective ways to enhance your user experience, improve app engagement, and boost retention. In this tutorial, we’ll show you how to integrate push notifications in your iOS and Android apps with minor configuration changes using the OneSignal Flutter SDK. In no time, you'll be able to send out transactional and promotional push notifications to users for free!

If you are a Flutter developer and want to leverage push notifications to engage with your users, follow along to add this amazing functionality to your apps.

Guide Overview:

Part 1: Set Up Your OneSignal Account
Part 2: Creating a Flutter App

Part 1: Set Up Your OneSignal Account

To begin, log in to your OneSignal account or create a free account. Then, click on the blue button entitled _ New App/Website _ to configure your OneSignal account to fit your app or website.

Add your app name, select _ Google Android _ if the target is Android devices or select _ Apple iOS _ if the target is iOS devices. Then click, Next: Configure Your Platform.

For iOS, you will need to add an iOS push certificate. If you're not sure about this step, read OneSignal's instructions on how to generate an iOS push certificate.

Apple iOS (APNs) Configuration in OneSignal

For Android, you will need to enter Firebase Server Key and Server ID that you can get by creating a new project on Firebase Console.

Google Android (FCM) Configuration in OneSignal

To find the Firebase service key, log in to the Firebase console and click on Add Project.

Add the name of your project and turn off Enable Google Analytics for this project if you want, then click on Create Project.

Click _ Create project _ to save the new entry, then click _ continue _.

You’ll be directed to the project dashboard where you have to click on the Setting icon next to _ Project Overview _ and click on _ Project Settings _ from the menu that appears.

Navigate to the project settings menu in the Firebase console.

In the Cloud Messaging tab, you’ll be able to find the Firebase Server Key and Server ID. Navigate back to your OneSignal dashboard and copy and paste those values into the appropriate fields under OneSignal Google Android(FCM) Configuration like this:

Input your Firebase Server Key and Sender ID in OneSignal.

After clicking on Save & Continue , you’ll be asked to select your target SDK. Choose Flutter.

Choose Flutter as your target SDK.

Click Save & Continue again and then click Done at the bottom of the screen.

View your app ID in OneSignal to finish the Google Android FCM configuration process.

Now, you should see your app dashboard on OneSignal under the _ Settings > Platforms _ tab. On this page under _ Native App _ _ platforms , you should see an **_Active** tag next to _ Google Android, _ which means you’re ready to send notifications to users using the Android version of your Flutter app. If you followed the iOS setup, you should see an _ Active _ tag next to _ Apple iOS _.

An "Active" tag is displayed next to Google Android in the OneSignal dashboard.

Part 2: Creating a Flutter App

So the first thing you need to do after creating a new Flutter project is to add the onesignal_flutter dependency. To do that, visit pub.dev and search for onesignal_flutter.

Shows onesignal_flutter 3.2.7 in pub.dev.

Add the latest version of onesignal_flutter to pubspec.yaml under the dependencies section with the version number from pub.dev.

Now open main.dart and remove MyHomePage() class (or just its contents).

Remove the content of MyHomePage() from main.dart.

Create a new dart file named home.dart (or continue in the main.dart file) and create a stateful widget by the name Home like this:

class Home extends StatefulWidget {
const Home({Key? key}) : super(key: key);
@override
_HomeState createState() => _HomeState();
}
class _HomeState extends State<Home> {
@override
Widget build(BuildContext context) {
return Container();
}
}

Next, build a simple UI by adding a few widgets like Container, Column and Text in the Home class.

@override
Widget build(BuildContext context) {
return Scaffold(
body: Container(
child: Column(
mainAxisAlignment: MainAxisAlignment.center,
crossAxisAlignment: CrossAxisAlignment.center,
children: const [
Text("Hello"),
],
),
),
);
}

Now you can add the OneSignal App ID which you can find in your OneSignal dashboard under the Keys & IDs tab.

Copy the OneSignal App ID and add it into your Flutter app in main.dart as a constant.

static final String oneSignalAppId = "16090413-4b70-4c0b-a9f4-dd43c445ccee";

Then add SDK initialization code to your XXX.

Future<void> initPlatformState() async {
   OneSignal.shared.setAppId(oneSignalAppId);
   OneSignal.shared
       .promptUserForPushNotificationPermission()
       .then((accepted) {});
 }

Also, initialize state.

@override
 void initState() {
   super.initState();
   initPlatformState();
 }

Complete Code(main.dart)

import 'package:flutter/material.dart';
import 'package:onesignal_flutter/onesignal_flutter.dart';

class Home extends StatefulWidget {
 const Home({Key? key}) : super(key: key);

 @override
 _HomeState createState() => _HomeState();
}

class _HomeState extends State<Home> {
 @override
 void initState() {
   super.initState();
   initPlatformState();
 }

 static final String oneSignalAppId = "16090413-4b70-4c0b-a9f4-dd43c445ccee";
 Future<void> initPlatformState() async {
   OneSignal.shared.setAppId(oneSignalAppId);
   OneSignal.shared
       .promptUserForPushNotificationPermission()
       .then((accepted) {});
 }

 @override
 Widget build(BuildContext context) {
   return Scaffold(
     body: Container(
       child: Column(
         mainAxisAlignment: MainAxisAlignment.center,
         crossAxisAlignment: CrossAxisAlignment.center,
         children: const [
           Text("Hello"),
         ],
       ),
     ),
   );
 }
}

Run your Flutter app on your physical device or emulator.

In case you recieve compileSdkVersion error then go to your app-level build.gradle file and update compileSDKVersion to 31.

defaultConfig {
// TODO: Specify your own unique Application ID (https://developer.android.com/studio/build/application-id.html).
applicationId "com.example.flutter_onesignal"
minSdkVersion 16
compileSdkVersion 31
targetSdkVersion 30
versionCode flutterVersionCode.toInteger()
versionName flutterVersionName
}

If you get Kotlin Version error then go to your build.gradle file in the root of your Android directory and upgrade your Kotlin_version to the latest version (ext.kotlin_version = '1.5.10')

For the iOS version, there’s are some additional steps that need to be completed. The first one is to add an iOS service extension.

The OneSignalNotificationServiceExtension allows your application (in iOS) to receive rich notifications with images and buttons, along with badges and Confirmed Deliveries.

The first step is to navigate to your Flutter project's iOS folder and open the .xcworkspace file in Xcode. You can use the open ios/Runner.xcworkspace command in the terminal to do the same.

Then select File > New > Target.

Select Notification Service Extension and press Next.

Add the product name as OneSignalNotificationExtension and change the language to _ Object-C _ or Swift according to your needs. The team account should be your account or your organization’s account.

Add the product name as OneSignalNotificationExtension in Flutter.

Click the _ Finish _ at the bottom right corner of the window, but when prompted to “Activate” scheme click on _ Cancel _.

Click "Cancel" when prompted to activate the scheme.

By canceling, you are keeping Xcode set to debug your app rather than using the extension. If you select _ Activate _ by accident, you can simply switch back to debug your app in Xcode (next to the Play button).

Now, open the Xcode project settings and select the _ OneSignalNotificationServiceExtension _ target. Under the _ Deployment info _ section on this page, select iOS 10.0 as the target unless there are any reason for using higher versions.

Now, close the Xcode project and go back to your Flutter development IDE. In the /ios directory of your project, open the Podfile and add the following lines outside of the main target (they should be at the same level as your main target):

target 'OneSignalNotificationServiceExtension' do
use_frameworks!
pod 'OneSignalXCFramework', '>= 3.4.3', '< 4.0'
end

Also, make sure to uncomment the platform line at the top of the Podfile. It can be iOS version 9.0 or higher according to your needs.

Now Open the terminal, cd to the ios directory, and run pod install.

If you see the error below, remove # from the above in front of use_frameworks! and try again.

- Runner (true) and OneSignalNotificationServiceExtension (false) do not both set use_frameworks!.

Open the <project-name>.xcworkspace file. In your project, in the OneSignalNotificationServiceExtension/ folder, open NotificationService.m.

Replace the whole file contents with the following code for Objective-C:

#import <OneSignal/OneSignal.h>
#import "NotificationService.h"
@interface NotificationService ()
@property (nonatomic, strong) void (^contentHandler)(UNNotificationContent *contentToDeliver);
@property (nonatomic, strong) UNNotificationRequest *receivedRequest;
@property (nonatomic, strong) UNMutableNotificationContent *bestAttemptContent;
@end
@implementation NotificationService
- (void)didReceiveNotificationRequest:(UNNotificationRequest *)request withContentHandler:(void (^)(UNNotificationContent * _Nonnull))contentHandler {
   self.receivedRequest = request;
   self.contentHandler = contentHandler;
   self.bestAttemptContent = [request.content mutableCopy];

   //If your SDK version is < 3.5.0 uncomment and use this code:
   /*
   [OneSignal didReceiveNotificationExtensionRequest:self.receivedRequest
                      withMutableNotificationContent:self.bestAttemptContent];
   self.contentHandler(self.bestAttemptContent);
   */

   /* DEBUGGING: Uncomment the 2 lines below and comment out the one above to ensure this extension is excuting
                 Note, this extension only runs when mutable-content is set
                 Setting an attachment or action buttons automatically adds this */
   // NSLog(@"Running NotificationServiceExtension");
   // self.bestAttemptContent.body = [@"[Modified] " stringByAppendingString:self.bestAttemptContent.body];

   // Uncomment this line to set the default log level of NSE to VERBOSE so we get all logs from NSE logic
   //[OneSignal setLogLevel:ONE_S_LL_VERBOSE visualLevel:ONE_S_LL_NONE];
   [OneSignal didReceiveNotificationExtensionRequest:self.receivedRequest
                      withMutableNotificationContent:self.bestAttemptContent
                                  withContentHandler:self.contentHandler];
}
- (void)serviceExtensionTimeWillExpire {
   // Called just before the extension will be terminated by the system.
   // Use this as an opportunity to deliver your "best attempt" at modified content, otherwise the original push payload will be used.

   [OneSignal serviceExtensionTimeWillExpireRequest:self.receivedRequest withMutableNotificationContent:self.bestAttemptContent];

   self.contentHandler(self.bestAttemptContent);
}
@end

For Swift, replace it with:

import UserNotifications

import OneSignal

class NotificationService: UNNotificationServiceExtension {

   var contentHandler: ((UNNotificationContent) -> Void)?
   var receivedRequest: UNNotificationRequest!
   var bestAttemptContent: UNMutableNotificationContent?

   override func didReceive(_ request: UNNotificationRequest, withContentHandler contentHandler: @escaping (UNNotificationContent) -> Void) {
       self.receivedRequest = request
       self.contentHandler = contentHandler
       self.bestAttemptContent = (request.content.mutableCopy() as? UNMutableNotificationContent)

       if let bestAttemptContent = bestAttemptContent {
           //If your SDK version is < 3.5.0 uncomment and use this code:
           /*
           OneSignal.didReceiveNotificationExtensionRequest(self.receivedRequest, with: self.bestAttemptContent)
           contentHandler(bestAttemptContent)
           */

           /* DEBUGGING: Uncomment the 2 lines below to check this extension is excuting
                         Note, this extension only runs when mutable-content is set
                         Setting an attachment or action buttons automatically adds this */
           //OneSignal.setLogLevel(.LL_VERBOSE, visualLevel: .LL_NONE)
           //bestAttemptContent.body = "[Modified] " + bestAttemptContent.body

           OneSignal.didReceiveNotificationExtensionRequest(self.receivedRequest, with: bestAttemptContent, withContentHandler: self.contentHandler)
       }
   }

   override func serviceExtensionTimeWillExpire() {
       // Called just before the extension will be terminated by the system.
       // Use this as an opportunity to deliver your "best attempt" at modified content, otherwise the original push payload will be used.
       if let contentHandler = contentHandler, let bestAttemptContent = bestAttemptContent {
           OneSignal.serviceExtensionTimeWillExpireRequest(self.receivedRequest, with: self.bestAttemptContent)
           contentHandler(bestAttemptContent)
       }
   }

}

You also need to enable an App Group to use Confirmed Deliveries and increment/decrement Badges through push notifications. For that, you can follow the iOS SDK App Groups setup guide to add the OneSignal App Group in your app.

Now you need to enable push capability for iOS apps. Open your .xcworkspace file in Xcode.

In the main app target, select Signing & Capabilities > All > + Capability and then search "push." Double-click on _ Push Notifications _ to enable it.

Next, enable Background Modes and check Remote Notifications.

Now you are ready to send notifications from the OneSignal dashboard!

Back in your OneSignal dashboard, click on Messages in the top navigation menu and select _ Push _ from the sub-menu. Click the +New Push button to create your first notification.

Note: Notifications are enabled on Android devices by default if you have disabled your notifications, make sure you enable them again. If you're sending a notification to an iOS device, you will need to opt-in to notifications in your app settings.

Click the "+New Push" button in your OneSignal dashboard.

You will be redirected to a new window that will allow you to customize your push notification. Under _ Audience _, make sure that _ Send to Subscribed Users _ is selected. Then, create your message by adding your message title, content, and image. Because this is the first notification your subscribers will receive, you may choose to craft a simple welcome message to confirm that they've been subscribed and reinforce the value that notifications will provide.

Under the _ Delivery Schedule _ section, select _ Immediately _ and _ Send to everyone at the same time _ to send to all your current push subscribers. If you have just finished setting up your OneSignal account, chances are you're the first and only subscriber. If your app or website is heavily trafficked and other users have already opted in to receive push notifications, you may want to select Send to a particular segment(s) to test your message out on a specific audience.

Specify the audience for your push notification in OneSignal.

Add your message title and copy and preview how your message looks on your target device.

When you're ready to send your message, click on the blue _ Review and Send _ button at the bottom of the screen.

A small popup will appear for you to review your message. Once you are satisfied, click on the blue _ Send Message _ button. You should receive a push notification on your device! 🚀

You’re done 🎊!

You’ll receive the notification on your physical device that you used to run your Flutter app or on the emulator.

After the message sends, you'll be directed to the Delivery page where you can view the send report and track engagement.

Still have questions?

Check out our complete OneSignal Flutter SDK Documentation.

This post was guest authored by Akanksha Singh.

Thread safety and Learning in Rust

onesignaldevs — Fri, 02 Jul 2021 17:00:13 +0000

Programming interviews can be a great learning experience for interviewers as well as interviewees. In fact, I recently had the opportunity to learn something new about Rust while interviewing an engineering candidate.

Background About our Interview Process

At OneSignal, we don't believe in handing someone a problem and watching them flounder with zero autocomplete and zero docs. Engineers who apply to join our team are allowed to use their own environments, access documentation, and treat the interviewer as a resource. If a candidate can solve a problem with some gentle assistance from the interviewer or by asking a few questions, then they're much closer to the real development experience. Senior engineers on our teams frequently act as resources for other developers, and if a candidate can get to a solution by treating the interviewer like a colleague then they're halfway home. Candidates often learn something new from the interviewer, and we've had great feedback on our process.

The Learning

The candidate was working on a problem involving concurrency and needed to share the same std::sync::mpsc::Sender with multiple threads concurrently. They wrote code analogous to the following (extremely simplified) example:

use std::sync::{RwLock, Arc, mpsc::channel};

pub fn main() {
    let (tx, rx) = channel();

    let x = Arc::new(RwLock::new(tx));

    std::thread::spawn(move || {
       x.read().unwrap().send(4u8).unwrap(); 
    });

    dbg!(rx.recv().unwrap());
}

Code that attempts to share a Sender<u8> across threads using an RwLock

Now you may initially be thinking that it's just wrong to put a channel sender inside of an RwLock — and you'd be right. Both of these types are useful for cross-thread synchronization, and it's odd to use both of them together. Each Sender is intended to be owned by a single thread, and if you need to coordinate between multiple threads you should be calling Sender::clone(). However, I urge you to recall that this was taking place during an interview, and concurrency was not the only thing going on in this program. The candidate had made other architectural decisions that led them here, and this small example has removed all of that context. Rust is a complex language with many types of synchronization primitives available, and during the time constraints of an interview, it's not unreasonable for someone to combine them in unusual ways.

The Rust compiler had a much stricter and less egalitarian view on this combination of types, however. When we tried to compile it, we were greeted with this error:

error[E0277]: `Sender<u8>` cannot be shared between threads safely
   --> src/main.rs:8:5
    |
8 | std::thread::spawn(move || {
    | ^^^^^^^^^^^^^^^^^^ `Sender<u8>` cannot be shared between threads safely
    |
    = help: the trait `Sync` is not implemented for `Sender<u8>`
    = note: required because of the requirements on the impl of `Sync` for `RwLock<Sender<u8>>`
    = note: required because of the requirements on the impl of `Send` for `Arc<RwLock<Sender<u8>>>`
    = note: required because it appears within the type `[closure@src/main.rs:8:24: 10:6]`

Compiler error that is generated by the above program

This confused both of us a lot. I know that Sender<T> can be sent across threads safely — that's the whole point of it! We tried a few things (using .write() on the RwLock instead of .read()) but we couldn't get it to compile. Eventually, I wondered if the code would work if we used the simpler Mutex instead of an RwLock. I had the candidate try this instead, and the program works. Here's what that program looks like:

use std::sync::{Mutex, Arc, mpsc::channel};

pub fn main() {
    let (tx, rx) = channel();

    let x = Arc::new(Mutex::new(tx));

    std::thread::spawn(move || {
       x.lock().unwrap().send(4u8).unwrap(); 
    });

    dbg!(rx.recv().unwrap());
}

Program which uses Mutex<Sender<u8>> instead of RwLock<Sender<u8>>

Running this program produces the output that we expected from the first one:

[src/main.rs:12] rx.recv().unwrap() = 4

It's worth pointing out that if you find yourself putting a Sender inside some type of Mutex or locking data structure, then you may want to reconsider your program's structure. But why does the RwLock version fail, and the Mutex version work?

To answer this question, we need to consider the meaning of the Send and Sync traits, the Sender, RwLock and Mutex types, and look closely at the implementations of Send and Sync for all of the types in question. Let's start with RwLock and Mutex.

RwLock vs. Mutex

Let's look at the differences between the two synchronization types that were used in this example, RwLock and Mutex. They are conceptually very similar — both allow you to guard a resource to ensure that it is accessed from multiple threads in a safe way. They have one important difference, however. Mutex holds a lock for both reads and writes, whereas RwLock treats reads and writes differently, allowing for multiple read locks to be taken in parallel but requiring exclusive access for write locks. The way that RwLock enforces single writer, multiple reader rules is the same as the single &mut, multiple & reference rules for single-threaded code. Here's a visual of how the same sequence of reads & writes occurs with a Mutex and an RwLock.

Visual on read vs write scheduling with Mutex and RwLock

As the diagram shows, when you have a read-heavy workload it is preferable to use an RwLock to avoid blocking reader threads when it's not required. The same sequence of events takes much longer with a Mutex than it does with an RwLock.

We thought that we could get away with using an RwLock around our Sender because the method that we needed to call (Sender::send(&self, value: T)) requires an immutable (read) reference to the underlying Sender rather than a mutable &mut self reference. This implies that we could use RwLock and RwLock::read() to guard calls to Sender::send(). This would allow us to spend much less time waiting on synchronization than if we used a Mutex around the Sender.

Send

The Send trait is the simpler of the two traits to explain conceptually. A type is Send if it is safe to transfer it across thread boundaries. This means that you can construct an instance of the type on one thread, transfer ownership to another thread, and Drop it on that other thread. Most types in Rust are Send, but there are a few exceptions. The most common non-Send types are Rc and all raw pointer types. The raw pointer types are not Send because Rust cannot make any guarantees about the synchronization mechanisms that are in the underlying types, or when the pointers will be accessed. Rc is not Send because it uses a non-atomic reference counter which is not safe to send between threads.

Sync

Sync is a bit trickier to explain. A type T is Sync if and only if &T is Send. This means that it must be safe to share immutable references of T between threads. A simpler way to think about this is to imagine if it would be safe to have multiple immutable references of type T being read from in parallel from multiple threads. The most common non-Sync types are UnsafeCell, Cell, and RefCell. These types all rely on unsynchronized mutation, so they are inherently not thread-safe.

Sender

The std::sync::mpsc::Sender type is the producer-half of a multi-producer, single-consumer channel. It's intended to be used by calling Sender::clone and using a single owned Sender per-thread. This pattern permits Sender to be non-Sync because each individual instance of Sender does not need to synchronize with itself across multiple threads. It only needs to be Send so that instances can be moved from thread to thread with ownership staying consistent.

Send + Sync in our programs

Let's look at the Send & Sync implementations for all of the types that are in our example programs. Our programs used Arc<Mutex<Sender<u8>>> (works) and Arc<RwLock<Sender<u8>>> (does not work).

impl<T: ?Sized + Send> Send for RwLock<T>
impl<T: ?Sized + Send + Sync> Sync for RwLock<T>

impl<T: ?Sized + Send> Send for Mutex<T>
impl<T: ?Sized + Send> Sync for Mutex<T>

impl<T: Send> Send for Sender<T>
impl<T> !Sync for Sender<T>

impl<T: ?Sized + Sync + Send> Send for Arc<T>
impl<T: ?Sized + Sync + Send> Sync for Arc<T>

The !Sync syntax indicates that Sender<T> is explicitly not Sync. The ?Sized syntax is not relevant to this discussion.

There are a few important things to note here:

Sender<T> is not Sync. It's not safe to have multiple immutable references to a Sender live across multiple threads at the same time. Considering that Sender::send(&self) requires only an immutable reference, there must be some kind of interior mutability going on.

Mutex<T> is both Send and Sync if T inside of it is Send. If something is inside of a Mutex, it will never have multiple immutable references live at the same time since Mutex does not allow multiple locks (read or write) to be taken at the same time. Because of this, Mutex<T> effectively bypasses the restrictions of Sync.

RwLock<T> is Send where T is Send, but it is only Sync if T is both Send + Sync. Since most types are Send + Sync, this is a non-issue. It only causes problems because the Sender<T> within it is not Sync. Recall from our explanation that RwLock allows multiple read locks to be open in parallel.

Arc<T> is only Send and Sync if the underlying T is both Send + Sync, meaning that you cannot send an Arc<T> across thread boundaries where T: !Sync.

These things all cascade to cause the compiler error that we saw in the first program. Sender<T> is not Sync meaning that RwLock<Sender<T>> is not Sync, meaning that Arc<RwLock<Sender<T>>> is neither Send nor Sync. That means that this type, despite containing three different supposedly thread safe synchronization types is not thread safe at all.

So what?

I think that this compiler error and the resulting exploration of the lock types and Send & Sync was a fantastic demonstration of the power of Rust's type system. Coming from other systems programming languages, these ideas are often much more nebulous. A type is often labeled (only via documentation) as "thread-safe" or "not thread-safe." In Rust, the Send and Sync traits allow us to express much more granular ideas about thread safety, and communicate them much more clearly. A Sender<T> is "thread safe" in that it can be sent between threads, but it cannot be shared between threads. That's what Send and !Sync tells us. There are so many layers of safety to multithreaded programming, and those layers exist in other systems languages, but in Rust we have the compiler to help us through them instead of relying on our own memory of how they work.

In an interview setting, would the un-safety of Arc<RwLock<Sender<u8>>> have caused program crashes? Maybe, maybe not. It would have caused issues under a production workload for sure, and I'm glad that we have Rust to point these things out to us for those times.

I really enjoyed going on this dive into thread safety in Rust, and it was all due to collaborating with an interview candidate on solving an interesting problem. If you like solving interesting problems, check out our jobs board and join our team!

>> View Job Openings at OneSignal

Converting GitHub Repositories to “main”

onesignaldevs — Tue, 29 Jun 2021 18:38:45 +0000

Overview

Developers use a “version control system” to both track their changes to code, and to allow sharing those changes with other developers. Over my career, I’ve used many proprietary and free systems, but in the past decade or so, most of the industry has settled on the tool “git,” very often hosted on sites like GitHub.com.

If you’ve never used a version control system, you can think of it sort of like an extra-powerful shared document. Our projects have thousands of files in them, and multiple developers may be working on the same parts of a project at once, and making many interdependent changes to multiple files simultaneously. To facilitate this, they will “branch” the code, make changes, and then subsequently merge it back into the main line.

Over the years, various systems I’ve used have had different names for this main branch - “trunk,” “base,” “main.” Unfortunately, git’s default name was “master,” which is not inclusive. As bad as this usage is, it gets even worse with “master/slave” for a running system and its backup - thankfully the industry is moving to replace those with “primary/secondary.” Google has put together a useful resource for developers about inclusion in technical naming.

In fall of 2020, GitHub added support for renaming the base branch on an existing repository, and made it so that the default for new repositories was “main.” I’d been idly meaning to look into this functionality, but one of our developers, Isaiah, prompted me on our plans for it. I went and researched the status out at GitHub and discovered that, while it was possible to do, it was still difficult - you needed to do a bunch of manual steps on the repository before the rename. However, they promised they’d have a one-step tool to do it for you released “by the end of the year.” We have approximately 250 repositories, so I decided we’d just wait for the tool to get it done more easily.

2020 came and went with no tool. I don’t know exactly what date they released it, but by the time I thought to look towards the end of the first quarter, we were in our last push to get our quarterly commits completed, and I decided to put it off until the second quarter. Once we got through planning for the quarter, though, I got to work trying to figure out exactly what needed to happen.

At the high level, it’s reasonably simple. You navigate to your project’s “Settings” page, then choose “Branches” on the left, then simply click the “Default branch” box and type the name of the new branch. Since new projects had by default been named “main” since fall, it was an easy choice to have that be what we changed to for everything. That way everyone’s muscle memory would be the same.

As an aside, GitHub has done a good job of making “master” be a synonym for “main” on changed repositories. So if you have scripts (or muscle memory) that refer to “master,” you don’t have to go through the difficult process of changing everything simultaneously - just change the repository default branch, and clean up the rest of your code after.

The Fix

With so many repositories, it was hard to know where to start. The first thing was to get all of them locally so we could use Unix command-line tools on them. A little searching turned up a Python library to check out all repositories in a single organization:

pip3 install gitimpython3 -m gitim -o <organization>

This then enabled us to discover something of the scope of the problem:

Initially for us, this number was about 240.

Unfortunately for a lot of our projects, it was more complicated. We use CircleCI for our integration and deployment, and a common pattern in our configuration files was to say “you can’t deploy to production except from master.” The first step, before we did anything else, was to change this to be “you can’t deploy to production except from master OR main.”

Next step was to find these particular issues:

grep master */.circleci/config.yml

This listed about 40 projects which included “master” in such a file. Unfortunately, the only solution was to have a developer look at these and fix them. In most cases, we were able to allow deploys from “master or main,” again, so that we wouldn’t have to carefully coordinate all the changes simultaneously.

We made a special Slack channel for volunteers to come together and crowd-source this. We made, reviewed, and deployed all these changes, and, in a couple of days, we were able to get the hardest of it done. As they were completed, I went into the repositories and renamed the default branches.

Finally, this left about 200 repositories that didn’t need any changes except to have their default branch renamed on GitHub. Unfortunately, you need to be a project owner to do this - and there are very few people at OneSignal who own every project.

So, on our quarterly Hackathon on June 2, I resolved to wrap this up. The workflow was as follows.

First, make a list of the configuration pages of all the project you need to fix:

ORG=OneSignal
grep main */.git/HEAD | perl -pe "s/^(.*?)\/\.git.*$/https:\/\/github.com\/$ORG\/\$1\/settings\/branches/" > to_fix

Then, edit “to_fix”. Start at the first file, hit to make a gap. Copy-and-paste that link into your browser, and change the default branch to “main.” As you complete each project, cut-and-paste it to above the line, so you know where you are in the process.

Once that’s complete, execute the following bash script (I called it “remain”):

#/bin/shif [$# -ne 0]
            then
                    cd $1
figit branch -m master main
git fetch origing
it branch -u origin/main main
git remote set-head origin -aif [$# -ne 0]   
        then 
                    cd ..
fi

So, for example:

remain OneSignal

Execute it from the directory that all your repositories reside in. This will re-base your local copy from “main.” If everything goes well, congratulations, that one is done! Move onto the next one. If it emits errors, you’ll need to figure out what the issue was and fix it. Across 200 files, maybe five times I had failed to actually rename the main branch before I ran it. And once, I renamed the main branch to “remain.” Oops!

As you move along, you can check your progress with this script (which I called “remaining”):

#!/usr/bin/perl -w
$master = `grep master */.git/HEAD | wc -l`;
chomp($master);
$main = `grep main */.git/HEAD | wc -l`;
chomp($main);
$other = `grep -v main */.git/HEAD | wc -l`;
chomp($other);
$other -= $master;
$total = $master + $main + $other;
printf("%i / $total = %0.2f%% complete; %i left to convert\n", ($main + $other), (($main + $other) / $total) * 100, $master);

Issues

Often you’ll see a brief message from GitHub that “main already exists” as you make the change. This seems to be a simple race condition, and is benign.
Sometimes you’ll get the error “Could not rename branch ‘master’ at this time: delete the branch protection rule for ‘main’ and try again.” This is because you have a branch protection rule for master, which GitHub has for some reason already duplicated to main. The tool wants to rename the “master” rule to “main,” but can’t, because such a rule already exists. If you simply delete the “main” rule, and then rename the default branch, GitHub will rename the “master” rule to be a “main” rule for you, and you’ll be done.
This is idiosyncratic to us, but on the very last repository we changed, it broke deployments. That was when we realized we had some Ansible rules that also explicitly referenced “master” in a way that wasn’t compatible with GitHub’s magical synonyms. So that might be another place to look for possible problems before you roll things out.

While I wish there had been an obvious command-line way to iterate these repositories, it seemed to depend on actually using the GitHub web interface to make the change; there was not an obvious, clean CLI alternative. It might’ve been possible to write a command-line client that would “hit the button” automatically, but I was a little worried about that not working perfectly, and getting repositories into an inconsistent state. And, even for this large group of projects, it was only a few hours of tedious work, improved by good music in the background.

Summary

I’m very glad we were finally able to exorcise this demon from our codebase. It was a little tedious in places, but it was certainly well worth the effort, and I’m very glad our industry has grown to the point of realizing the importance of this sort of work. Also very proud of all the folks on all the engineering groups that rolled up their sleeves and got this done!