Forem: donnie-jp

How we build and deliver quality mobile SDKs at scale

donnie-jp — Mon, 11 Jan 2021 10:11:04 +0000

Until I joined my current team at Rakuten I hadn't really thought much about the SDKs (aka libraries) that I had been adding to my mobile apps. Adding 3rd party code or an SDK was just a way to get a UI feature or functionality added (e.g. crash reporting) in a shorter time and with - in theory - lower risk.

However, in the last few years at Rakuten I've thought a lot about how to build SDKs - and so I should, because it is our team's mission to deliver high quality mobile SDKs.😅

Most teams in companies that provide in-house SDKs seem to either provide a general SDK covering everything or maybe an authentication SDK and a function x SDK. Also, based on chats I've had when interviewing, what they call SDKs are often just separate modules in the same source repo with no proper deployment. There's absolutely nothing wrong with that but it's hard to scale.

In the MTSD SDK team we have a mix of long-term and newer products that add up to a significant number - to give you an idea, a major focus during the last year was trying to reduce our supported products from the current 16 to fewer than 10.

Many of our SDKs are stable long-term products that don't have a regular delivery schedule but our team still needs to be able to maintain and release new versions of those products when required, sometimes at relatively short notice. Our team is fairly small (2-3 devs per iOS/Android) so other than urgent maintenance/bug-fixing tasks we try to focus on active development of a couple of products at any particular time. But due to 💩 happens we need to be responsive to changes in priority.

In this post I want to focus on how we (mostly I think) achieve building and delivering high quality mobile SDKs at Rakuten scale.

We have recently open sourced our mobile SDK development guidelines on GitHub - we use these in our team to maintain quality, to help onboard new engineers, and to share our approach with other Rakuten teams who ask for our input when building their own SDKs.

The guidelines are a bit (ok, very) dry so here's an image I made for an internal Tech Community presentation to highlight the (often hidden) team effort and the various aspects that go into continuous delivery of multiple SDKs - while keeping high quality and with minimal overhead.

Let's explore what's hidden below the iceberg's surface 🤿

To give app developers a great experience with our products we strive for our SDKs to be:

High quality
- Peer reviewed code
- Automated unit tests that are run by CI on every PR/merge
Easy to use
- Simple, fluent APIs
- Excellent user documentation
- Responsive developer support
Easily integrated
- CocoaPods/Gradle dependency
- Configured in zero-to-a-few lines of code
Deployed by automation
- Minimize effort
- Minimize risk of human error
Well-designed/architected
- Limit dependencies
- Only expose what must be exposed
- Components loosely coupled
- Robust and secure
Properly versioned
- Following semantic versioning
Consistent between platforms
- Same API interface
- Same high-level architecture
Maintainable long term
- Stable after reaching v1.0
- At least 6 months notice of end of life

Our team mindset is one of continual improvement. Due to the nature of SDK development we can't always move as fast as app teams. This is mostly a benefit in my opinion and allows us to come up with technical and process improvements that are sustainable.

I think our team has reached a good maturity level but there will always be aspects we can improve. For example, in Q2/Q3 last year we undertook a PoC with bitrise CI (which is specifically designed for mobile teams). The PoC was successful and during Q4 we migrated all SDK iOS pull request CI from Jenkins/travis to bitrise.

We could (and intend to) also improve the documentation of our internal SDKs - docs are generated using Doxygen on iOS - they are a bit old-school (though functional) and in need of a revamp. Our open source repos e.g. In-App Messaging have much nicer docs generated by Jazzy and hosted on GitHub Pages.

If you're an iOS developer, a more technical blog post that I read when I was quite new to working on iOS SDKs was https://academy.realm.io/posts/altconf-conrad-kramer-writing-iOS-sdk/. I think the advice is still highly relevant - a key difference now is that it's likely preferable to build your iOS SDKs using Swift because of its safety, conciseness and readability benefits over Objective-C and, crucially, it has gained module and ABI stability for compatibility between different Swift versions.

While ABI Stability allows programs written with different versions of Swift to exist in a shared runtime, Module Stability allows you to use frameworks compiled with
different versions of Swift in a project that might use yet another version of Swift.

I hope you found the above post interesting and perhaps useful. Please comment or reach out if you'd like to discuss anything about building mobile SDKs.

Secure (xc)config for iOS apps

donnie-jp — Sun, 26 Jul 2020 03:52:12 +0000

App configuration dilemma 😟

Developers often have to set some app config (API secrets for Staging, hard-coded users/passwords) locally that they don't want to ever be committed to their git repo, especially if that repo is open source. But people (and developers are people too) can, and do, make mistakes. So you may end up having to delete PRs, clean git histories, and ask GitHub support staff - who are super helpful actually - to delete PRs and run git garbage collection etc., which is all stuff that takes time away from more enjoyable tasks like feature development or writing blog posts.

It would be very nice if the app config could be set in a way that it's impossible to accidentally push that supposed-to-be-local temporary confidential information to git. We have a full implementation of secure configuration in our Remote Config iOS SDK. In this post I will introduce the general approach.

plists and xcconfig ⚙️

Many SDKs (including ours) require app developers to add key-value configs to their Info.plist and the SDKs read those values at runtime.

Instead of using hard-coded values in the plist we can make them variables e.g. $(API_KEY) that reference the real value stored in an xcconfig file. This allows the app to easily have different configs for their different environments. But, importantly, it also gives us a mechanism to remove secrets from the Info plist. Thanks are due to my colleague Pierre from the Mini App team for initiating this idea.

To understand the ins and outs of xcconfig files I followed this excellent Thoughtbot blog post which describes how to use xcconfig files to configure your app for different environments. Although I don't want to cover the same stuff I want to highlight this point from the author

Since these files will potentially contain secure information, such as API_KEY, I’d recommend not checking them into version control and instead using a secure file storage system like 1Password

An alternative approach is to use a hidden-from-git "secrets" xcconfig file.

Make it secure 🔒

You can make the approach secure by following these steps:

Create a hidden-from-git secrets.xcconfig (or any name you prefer). The secrets xcconfig is hidden from git by adding it to your .gitignore file - the secrets xcconfig must never be committed/pushed to git.
Add your secrets to secrets.xcconfig as key-value pairs, for example MY_SECRET=my real secret
In your <configuration type>.xcconfig file import secrets.xcconfig
In your app Info.plist create a key MySecret set to value $(MY_SECRET)

At build time my real secret will be injected into the plist as the value of MY_SECRET.

Your file snippets should look similar to:

`secrets.xcconfig`

MY_SECRET = my real secret

`<configuration type>.xcconfig`

#include "secrets.xcconfig"

`Info.plist`

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>MySecret</key>
    <string>$(MY_SECRET)</string>
</dict>
</plist>

Note: The above plist snippet only shows one key for clarity

Automate! 🤖

The above approach works pretty well but it would be a pain to maintain. Our team deploy and integrate our SDKs using CocoaPods so I wanted to find a way to automate secrets management that fits our process. So what I did was add a CocoaPods post_install hook to our app Podfile that runs a custom shell script that generates the secrets xcconfig file from local environment variables. When the script runs during pod install we print a warning/error message if a required environment variable is not set.

Are there limitations? 🤔

Security

This approach only prevents secrets in Info.plist files from being inadvertently committed to your git repo. It does not provide any run-time protection for your secrets and it should be noted that extracting secrets from an iOS ipa is as simple as unzipping the binary package and opening the plist in a text editor. If your secrets also need run-time protection you must consider an alternative approach.

Usability

The steps to add a new secret plist variable are not that obvious and it would be better to have a proper CocoaPods plugin. It could be based on the cocoapods-keys plugin (which stores secrets in the macos keychain and writes obfuscated secrets to generated source files that can be imported and used by app source files). I think making a plugin would be the logical next step for our secure xcconfig approach and I hope to explore building a plugin for this in future.

Dynamic SSL Pinning on iOS with Approov

donnie-jp — Mon, 15 Jun 2020 13:07:23 +0000

I've recently been looking into ways to improve the security of mobile apps and services with my colleague Julien. One of the vendors we've evaluated is https://www.approov.io/ who provide an API Protection solution that enables a backend service to confirm it is talking to a genuine, untampered app running on a genuine, untampered (not jailbroken, no mitm proxy etc.) mobile device. On Android it could be considered as an enhancement to the protection offered by Google SafetyNet Attestation but, unlike SafetyNet, Approov also supports iOS.

Approov PoC

One of the benefits of Approov is that it allows you to remove embedded client secrets from your mobile app. Instead you attach a short-lived Approov JSON Web Token (JWT) token - issued by the Approov cloud service - to your mobile app's API requests and your API gateway validates the token and allows the request to proceed (or not). We trialled the Approov solution and implemented a PoC on Android and iOS.

It works as follows:

Figure: https://www.approov.io

The changes we had to make to support Approov weren't huge but larger than a typical SDK where you just call a configuration function at app launch.

Replace client secret in app with dummy
Configure Approov SDK at app launch – with embedded initial config file and dynamic config
Fetch Approov token from SDK and attach it to outgoing requests
Store updated Approov dynamic config

The Approov dynamic config allows us to do dynamic pinning which is what I want to focus on in this blog post.

But first, what's pinning and are there any pitfalls?

A mobile client can check that it is talking to a genuine server before allowing a request to proceed by validating the server certificate / public key. This protection mechanism is recommended by OWASP (probably the main folk to consult about mobile app security) in their Mobile Application Security Verification Standard as Level 2 'Defense-in-Depth' protection.

Figure: https://mobile-security.gitbook.io

L2 introduces advanced security controls that go beyond the standard requirements. To fulfill MASVS-L2, a threat model must exist, and security must be an integral part of the app's architecture and design. Based on the threat model, the right MASVS-L2 controls should have been selected and implemented successfully. This level is appropriate for apps that handle highly sensitive data, such as mobile banking apps.

Figure: https://mobile-security.gitbook.io

Normally the server certificate / public key is embedded in the app binary (e.g. as a pem file or hard-coded pins in source code) by the developer. Embedding a key or certificate in your app means that it could be left unable to make requests (oh, oh) if the server-side certificate changes - either through normal certificate rotation or due to a security breach. To guard against this, app developers need to ensure that the released app contains the up-to-date server certificate. They can do this by including multiple certificates and/or making sure they release an app with the 'next' server certificate before it goes live on the server. But it's difficult to manage and the potential for trouble is high...

An alternative is to do it dynamically - which, handily, Approov facilitates:

Figure: https://www.approov.io

Dynamic pinning (with Approov)

The Approov cli tool allows you to add domains e.g. https://my-api-gateway.my-domain.com. When the domain is added Approov will also fetch the certificate public key pin for the domain.

When the app launches it requests an updated config from Approov which contains the public key pins for the pinned domains. When a request is made to the API gateway the mobile client code will fetch the pins from the Approov SDK and validate the gateway server's certificate's public key information against the pins. If the validation succeeds the request will proceed, otherwise it will fail.

Here is a sample implementation of pinning a URLSession from the Approov documentation:

func urlSession(_ session: URLSession,
    didReceive challenge: URLAuthenticationChallenge,
    completionHandler: @escaping (URLSession.AuthChallengeDisposition, URLCredential?) -> Void) {

    let protectionSpace = challenge.protectionSpace

    // only handle requests that are related to server trust
    if protectionSpace.authenticationMethod == NSURLAuthenticationMethodServerTrust {
        // determine the host whose connection is pinned (this requires that the session delegate keeps a reference
        // to the URLSessionTask when the task is created)
        guard let urlSessionTask = sessionTask,
            let host = urlSessionTaskHost(urlSessionTask: urlSessionTask),
            let pins = Approov.getPins("public-key-sha256")?[host]
        else {
            completionHandler(.cancelAuthenticationChallenge, nil);
            return
        }

        // check the validity of the server trust
        let sslPolicy = SecPolicyCreateSSL(true, host as CFString)
        var secResult = SecTrustResultType.invalid
        guard let serverTrust = protectionSpace.serverTrust,
            // Ensure a sensible SSL policy is used when evaluating the server trust
            SecTrustSetPolicies(serverTrust, sslPolicy) == errSecSuccess,
            SecTrustEvaluate(serverTrust, &secResult) == errSecSuccess,
            secResult == .unspecified || secResult == .proceed
        else {
            completionHandler(.cancelAuthenticationChallenge, nil);
            return
        }

        // remember the hashes of all public key infos for logging in case of pinning failure
        var spkiHashesBase64 = [String](repeating: "", count: SecTrustGetCertificateCount(serverTrust))

        // check public key hash of all certificates in the chain, leaf certificate first
        for i in 0 ..< SecTrustGetCertificateCount(serverTrust) {

            guard let serverCert = SecTrustGetCertificateAtIndex(serverTrust, i),
                let spkiHashBase64 = publicKeyInfoSHA256Base64(certificate: serverCert)
            else {
                continue
            }

            spkiHashesBase64[i] = spkiHashBase64
            NSLog("URLSessionDelegate: host %@ public key hash %@", host, spkiHashesBase64[i])

            // check that the hash is the same as at least one of the pins
            for pin in pins {
                if spkiHashesBase64[i].elementsEqual(pin) {
                    NSLog("URLSessionDelegate: pinning valid")
                    completionHandler(.useCredential, URLCredential(trust: serverTrust))
                    // Successful match
                    return
                }
            }
        }

        // the certificates did not match any of the pins
        completionHandler(.cancelAuthenticationChallenge, nil);

        // log the hashes of all certificates in the certificate chain for the host - this helps with choosing the
        // correct hash(es) to put into the Approov configuration
        for i in 0 ..< SecTrustGetCertificateCount(serverTrust) {
            NSLog("URLSessionDelegate: pinning invalid for host %@ and certificate %d's public key info hash %@",
                  host, i, spkiHashesBase64[i])
        }
    }
}

Hmmm what's that code doing??

The urlSession(_:didReceive:completionHandler:) method will be called when the session sets up a connection to a remote server that uses SSL or TLS, to allow your app to verify the server’s certificate chain
The call to Approov.getPins("public-key-sha256")?[host] is where we get the pins for that host e.g. https://my-api-gateway.my-domain.com that were supplied in the dynamic config
Then it iterates through the certificates presented by the server checking for at least one match to a pin: if spkiHashesBase64[i].elementsEqual(pin)
If there's a match, the completion handler will be called with .useCredential, passing in the credential, and the connection will proceed 👍
If there's no match, the connection will be cancelled ⛔️

Dynamic pinning in action

To test pinning you can use Charles to proxy the device's SSL connection to the pinned API endpoint e.g. https://my-api-gateway.my-domain.com.

Try to call an API on https://my-api-gateway.my-domain.com that returns a secret - such as a Login API that returns an OAuth token - and observe the secret in the API response – you can't because the API request gets cancelled client-side! That's because Charles proxy has not presented a server certificate that matches our pin.

Now disable SSL proxying in Charles and re-try. Log in succeeds!

Note that the pinning implementation is in the app code not the Approov SDK code so it's possible to do non-dynamic pinning without Approov but pin management would be far more error prone and the pins would have to be embedded and thus could be extracted from the app by a bad person.

Apps could also do dynamic pinning without Approov but they would need to host a backend service that could manage the domains and certificates and supply the updated pins to the app - so it's not straightforward to implement.

A final note on Approov: even if pinning has been implemented by an app it's possible to disable it on a jailbroken device - an exercise left to the reader - but Approov has protection against that attack and in that situation it will issue an invalid token. Which I think is...

Why do we open source our mobile SDKs?

donnie-jp — Wed, 11 Mar 2020 14:11:32 +0000

In the Mobile Technology Solutions Department (MTSD) SDK team at Rakuten we used GitPub (internal Bitbucket Server) for hosting our source code and already had Jenkins (a macmini sitting on a developer's [ok, my] desk) for CI so why would we want to open source our mobile Android and iOS SDKs?

The first mobile SDK we open sourced provides apps with automatic performance tracking. Initially, I thought that if we open sourced our Performance Tracking mobile SDK and our backend source it would result in an open sourced full mobile performance tracking solution and we'd get real outside users of the product. We did open source the mobile SDK however the backend code never got open sourced. It would have been amazing but open sourcing our backend code was always going to be more challenging than doing it for the mobile SDKs - although we could still do it!

Although nobody (that we know of) outside of Rakuten is using the Performance Tracking SDK in their production apps there were still a number of strong benefits that we got from open sourcing the SDK and, more recently, our other mobile SDKs - especially when it comes to CI on iOS!

By open sourcing our mobile SDKs it:

Allows us to use modern tooling, like GitHub for source code and API docs hosting, and Travis or CircleCI for CI tests and automation
Increases customer confidence in our code - they can read it if they want to!
Makes developers happy because we love to work on open source 😍
Allows contributions (issues/PRs) from users
Increases mobile community recognition for MTSD and Rakuten, and attracts potential employees
Allows developers to point to open source contributions on their online profiles and resumes

An increasing number of tech companies open source their mobile SDKs including Dropbox, Google, Adjust, Mapbox etc. It is now close to being a standard practice in the companies that offer mobile SDK products.

So far our team has open sourced the following mobile SDKs

Our Remote Config SDK was developed completely in the open on GitHub from the initial commit, and our MiniApp SDK is also being built in the open on GitHub.

We've also open sourced build config, utility libraries and developer tools.

I hope that in Rakuten MTSD we continue our commitment to open sourcing our mobile solutions, and going forward I want to see us contributing back to open source projects too.

Oh, and you might be wondering what the process is to open source something in Rakuten. It’s actually pretty easy...

Document the Project
Consult with Legal
Approval from Management
Prepare Source Code
Publish on GitHub 🥳
BOOM 💥