Forem: Brian Michalski

Stop Hardcoding Google Maps API Keys!

Brian Michalski — Wed, 12 Jun 2024 04:15:41 +0000

One of the really cool things about the new suite of APIs that Google Maps Platform has been releasing lately (Routes API, Places API, etc) is that they look and feel a lot like other Google Cloud Platform APIs. They're exposed via gRPC, support field masks, and let developers authenticate via OAuth. A really helpful side effects of OAuth is support for JSON Web Tokens (JWT) credentials which allow apps to do a much better job securing client-side applications.

Classic Google Maps APIs rely on a a hardcoded API key which is not great. Hardcoding passwords was cool in 2005 (maybe?) but it's 2024, we can do better.

Using a small snippet of code, our app can generate a unique auth token for each website visitor that will expire after a defined period. Even if the user maliciously "borrowed" that token, it would only be valid until the expiration period you specified.

JWT Intro

If you're new to JWTs like I am, they are base64 encoded JSON blobs that get signed using a private key. APIs can read contents of that JSON object and verify the signature to authenticate them. Here's what one looks like:

eyJhbGciOiJSUzI1NiIsImtpZCI6ImVlODk1OWMzYzFhMDdlMTBlZGJjMDE3NWI2ZmZmN2I1ZGYyOTBiZTIiLCJ0eXAiOiJKV1QifQ.eyJzY29wZSI6Imh0dHBzOi8vd3d3Lmdvb2dsZWFwaXMuY29tL2F1dGgvZ2VvLXBsYXRmb3JtLnJvdXRlcyIsImlzcyI6InVidW50dS12bUBob2xpZGF5cy0xMTcwLmlhbS5nc2VydmljZWFjY291bnQuY29tIiwic3ViIjoidWJ1bnR1LXZtQGhvbGlkYXlzLTExNzAuaWFtLmdzZXJ2aWNlYWNjb3VudC5jb20iLCJhdWQiOiJodHRwczovL3JvdXRlcy5nb29nbGVhcGlzLmNvbS8iLCJleHAiOjE2Njc4ODQ0NTIsImlhdCI6MTY2Nzg4NDMzMn0.Zey0GtvSH78_xfBTNL-Ij0qm1dK9wqDc5nllYLPZyWNp_V5sYVKaPpWSjJ2IRVHBdhKBLYgXVKLty7Dlo0BMW9SJ4eexIxmdM8IR3CeH5SmYLl4pQxV3S8eO_5T41B6LCD49gKTtlXIWvtCoGitWDSYiFCZauf2zoIEa5XZ_TkazMr1DGYbc9w8UvtXVARAby2WRbSiHyqkjSsAU5HoKClKhaw7NaP1vNJ-7IlpTz9t-sTSZwl-6wur65gI_FtAGiohWPUILRY-YKMhb_wXQ5AtlDUmvGKdqNzuXBMmk8-iiQTwmYPuWQBNt0MtK7hfghWyWubUjBfT0t4yiGSrHmA

If you decode that token, you can see it contains information about the key it was signed with:

{
  "alg": "RS256",
  "kid": "ee8959c3c1a07e10edbc0175b6fff7b5df290be2",
  "typ": "JWT"
}

and, more importantly, payload with information about who issued the token, what it can be used for, and when it will expire:

{
  "scope": "https://www.googleapis.com/auth/geo-platform.routes",
  "aud": "https://routes.googleapis.com/",
  "exp": 1667880959,
  "iat": 1667880839,
  "iss": "ubuntu-vm@holidays-1170.iam.gserviceaccount.com",
  "sub": "ubuntu-vm@holidays-1170.iam.gserviceaccount.com"
}

In my experience, most Google Maps Platform APIs expects a token to contain a payload with the following fields:

field	description
`exp`	Expiration time for the token.
`iat`	Issued time for the token (aka now).
`aud`	API endpoint the token is intended for, like `https://routes.googleapis.com/`
`scope`	Scopes, separated by spaces, this token can be used for, like `https://www.googleapis.com/auth/geo-platform.routes`

From what I can tell, either a scope or audience field needs to be set. I don't know what's the "right" way.

I've collected a bunch of options for Scope and Audience here.

Generating a JWT

The biggest downside to JWTs is that you have to generate them on the fly - you can't just hardcode them into your app like you could with good old AIza. Generating a JWT involves building an JSON object with the right fields (see above), signing it, and then base64 encoding it to a string. https://jwt.io/introduction walks through this in glorious detail.

To make that step easier, I created a little Go backend which will generate tokens and sign them using a GCP Service Account: https://github.com/bamnet/gmp-jwt.

Running this backend somewhere like Cloud Run makes it easy to start generating tokens since you get access to a Default Service Account but you can run this anywhere and set Application Default Credentials.

But what protects the JWT generator?

Great question. Even though our JWT tokens are tightly scoped and moderately TTLed, an attacker could just scrape them from our backend that generates them. That would be no fun.

Using Firebase AppCheck we can verify that the environment requesting a token looks legit before giving them a JWT.

The frontend just needs to include recaptcha v3 (which is totally silent now - no more traffic lights, cross walks, or bicycles) and a bit of Firebase code to wire up the token.

Sending a JWT to Google Maps Platform

To authenticate using a JWT to a modern Google Maps Platform API, we use Bearer Authentication. Set the Authorization header to Bearer ${token}.

In JavaScript, that snippet might look like:

const response = await fetch('https://routes.googleapis.com/directions/v2:computeRoutes', {
  method: 'POST',
  headers: {
      'Authorization': 'Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImVlODk1OWMzYzFhMDdlMTBlZGJjMDE3NWI2ZmZmN2I1ZGYyOTBiZTIiLCJ0eXAiOiJKV1QifQ.eyJzY29wZSI6Imh0dHBzOi8vd3d3Lmdvb2dsZWFwaXMuY29tL2F1dGgvZ2VvLXBsYXRmb3JtLnJvdXRlcyIsImlzcyI6InVidW50dS12bUBob2xpZGF5cy0xMTcwLmlhbS5nc2VydmljZWFjY291bnQuY29tIiwic3ViIjoidWJ1bnR1LXZtQGhvbGlkYXlzLTExNzAuaWFtLmdzZXJ2aWNlYWNjb3VudC5jb20iLCJhdWQiOiJodHRwczovL3JvdXRlcy5nb29nbGVhcGlzLmNvbS8iLCJleHAiOjE2Njc4ODQ0NTIsImlhdCI6MTY2Nzg4NDMzMn0.Zey0GtvSH78_xfBTNL-Ij0qm1dK9wqDc5nllYLPZyWNp_V5sYVKaPpWSjJ2IRVHBdhKBLYgXVKLty7Dlo0BMW9SJ4eexIxmdM8IR3CeH5SmYLl4pQxV3S8eO_5T41B6LCD49gKTtlXIWvtCoGitWDSYiFCZauf2zoIEa5XZ_TkazMr1DGYbc9w8UvtXVARAby2WRbSiHyqkjSsAU5HoKClKhaw7NaP1vNJ-7IlpTz9t-sTSZwl-6wur65gI_FtAGiohWPUILRY-YKMhb_wXQ5AtlDUmvGKdqNzuXBMmk8-iiQTwmYPuWQBNt0MtK7hfghWyWubUjBfT0t4yiGSrHmA',
      'Content-Type': 'application/json',
      'X-Goog-FieldMask': 'routes.duration,routes.distanceMeters', 
  },
  body: JSON.stringify({
    // Routes API request body.
  }),
});

I wouldn't dream of sharing an API Key in a blogpost but an expired JWT, no problem!

Putting it all together

Server-Side

You need to deploy a server endpoint somewhere which will mint JWTs, optionally after checking Firebase App Check. A sample Cloud Run function I use is @ https://github.com/bamnet/gmp-jwt.

Client-Side

(optional) Get an app check token from Firebase.
Requests a JWT from the server endpoint you deployed above, optionally passing that app check token from Step 1.
Call the desired Google Maps Platform API passing Authorization: Bearer ${token}, passing the token from Step 2.
???
Profit.

Here's an example in JS, but you can do the same thing in Android & iOS:

// Initialize Firebase.
const app = initializeApp(firebaseConfig);

// Initialize AppCheck.
const appCheck = initializeAppCheck(app, {
  provider: new ReCaptchaV3Provider(/** reCAPTCHA Key */ ''),
  isTokenAutoRefreshEnabled: true
});

// Grab an AppCheck token.
const appCheckToken = await getToken(appCheck).then(t => t.token);

// Call our backend to convert the AppCheck token into a JWT.
const jwt = await fetch(/** JWT Minting Backend */ '', {
  headers: {
    'X-Firebase-AppCheck': appCheckToken,
  }
}).then((data) => data.text());

// Call the Routes API.
// Look ma, no hardcoded API key!
const response = await fetch('https://routes.googleapis.com/directions/v2:computeRoutes', {
  method: 'POST',
  headers: {
      'Authorization': `Bearer ${jwt}`, // Pass our JWT!
      'Content-Type': 'application/json',
      'X-Goog-FieldMask': 'routes.duration,routes.distanceMeters', 
  },
  body: JSON.stringify({
    // Routes API request body.
  }),
});

console.log(response);

Ta-Da, no more hardcoded API key!

Not-So-Frequently Asked Questions

Can a single JWT be used calling multiple APIs?

Yes. In my tests, a token can have multiple scopes, but only 1 audience. Scopes are separated with spaces. To generate a token for both Places and Routes, you'd set:

{
  "scope": "https://www.googleapis.com/auth/geo-platform.routes https://www.googleapis.com/auth/maps-platform.places",
  ...
}

Couldn't you proxy all requests through a trusted server which added & removed an API key?

Yes, but then you have a dependency in the serving path for ~every request to an API. That adds some marginal latency and could have scaling challenges for high-QPS APIs like Map Tiles or Places Autocomplete. That also means the proxy servers will see all requests which might mean more privacy / compliance work.

Does this work for the Maps JavaScript API?

No, but that sounds like a good feature request.

GRPC Performance in Flutter

Brian Michalski — Mon, 11 Apr 2022 03:49:55 +0000

Firebase Performance Monitoring is a quick and easy way to get started monitoring the performance of your apps. It automatically monitors a variety of metrics, including HTTP(S) requests which is a great way to compliment your server-side metrics and understand how your app's users are actually feeling.

That magic doesn't kick in automatically if you're using GRPC to make requests in your Flutter app. Instead, you can drop in a quick interceptor to measure the performance of GRPC requests and report them to Firebase:

lib/performance_interceptor.dart



import 'package:firebase_performance/firebase_performance.dart';
import 'package:grpc/grpc.dart';

class PerformanceInterceptor implements ClientInterceptor {
  FirebasePerformance _performance = FirebasePerformance.instance;
  Map<String, String> attributes;

  PerformanceInterceptor([this.attributes = const {}]);

  @override
  ResponseFuture<R> interceptUnary<Q, R>(ClientMethod<Q, R> method, Q request,
      CallOptions options, ClientUnaryInvoker<Q, R> invoker) {
    Trace metric = _performance.newTrace(method.path);
    metric.start();
    this.attributes.forEach((key, value) => metric.putAttribute(key, value));

    ResponseFuture<R> resp = invoker(method, request, options);
    resp.then((_) {
      metric.stop();
    });
    return resp;
  }

  @override
  ResponseStream<R> interceptStreaming<Q, R>(
      ClientMethod<Q, R> method,
      Stream<Q> requests,
      CallOptions options,
      ClientStreamingInvoker<Q, R> invoker) {
    return invoker(method, requests, options);
  }
}

This class is super simple to use, just pass it alongside the options and channel when constructing your GRPC client:



import 'performance_interceptor.dart';

...

client = GreeterClient(
  channel,
  interceptors: [
    PerformanceInterceptor()
  ],
);

You can also pass attributes (dimensions) that should be passed with each request. I like to pass the server I'm connecting to:



client = GreeterClient(
  channel,
  interceptors: [
    PerformanceInterceptor({'host': hostname})
  ],
);

Presto! You've now got performance metric for all your simple GRPC requests!

Static Maps At Night

Brian Michalski — Fri, 16 Apr 2021 02:22:12 +0000

Google's Static Maps API renders a map that looks great against a white background and other light websites but stands out like a bright TV in a dark room when your users switch over to dark / night mode - ouch!

Traditional techniques for making images dark mode friendly involve using CSS filters to reduce the brightness and up the contrast which dulls the color a bunch but doesn't give us a completely dark map like you may see on your phone:

We can do something more clever using some basic HTML tags and the Static Map API's styling features.

Given an image tag like:

<img src="https://maps.googleapis.com/maps/api/staticmap?...">

We can wrap it in a <picture> element that is smart enough to swap to a styled static map when the browser is requesting dark mode. To get that dark mode styling, we can add style tags to change all the light map features to night-friendly colors.

<picture>
  <source
    media="(prefers-color-scheme: dark)"
    srcset="https://maps.googleapis.com/maps/api/staticmap?<YOUR_MAP_PARAMS>&style=element%3Ageometry%7Ccolor%3A0x242f3e&style=element%3Alabels.text.stroke%7Ccolor%3A0x242f3e&style=element%3Alabels.text.fill%7Ccolor%3A0x746855&style=feature%3Aadministrative.locality%7Celement%3Alabels.text.fill%7Ccolor%3A0xd59563&style=feature%3Apoi%7Celement%3Alabels.text.fill%7Ccolor%3A0xd59563&style=feature%3Apoi.park%7Celement%3Ageometry%7Ccolor%3A0x263c3f&style=feature%3Apoi.park%7Celement%3Alabels.text.fill%7Ccolor%3A0x6b9a76&style=feature%3Aroad%7Celement%3Ageometry%7Ccolor%3A0x38414e&style=feature%3Aroad%7Celement%3Ageometry.stroke%7Ccolor%3A0x212a37&style=feature%3Aroad%7Celement%3Alabels.text.fill%7Ccolor%3A0x9ca5b3&style=feature%3Aroad.highway%7Celement%3Ageometry%7Ccolor%3A0x746855&style=feature%3Aroad.highway%7Celement%3Ageometry.stroke%7Ccolor%3A0x1f2835&style=feature%3Aroad.highway%7Celement%3Alabels.text.fill%7Ccolor%3A0xf3d19c&style=feature%3Atransit%7Celement%3Ageometry%7Ccolor%3A0x2f3948&style=feature%3Atransit.station%7Celement%3Alabels.text.fill%7Ccolor%3A0xd59563&style=feature%3Awater%7Celement%3Ageometry%7Ccolor%3A0x17263c&style=feature%3Awater%7Celement%3Alabels.text.fill%7Ccolor%3A0x515c6d&style=feature%3Awater%7Celement%3Alabels.text.stroke%7Ccolor%3A0x17263c">
  <img src="https://maps.googleapis.com/maps/api/staticmap?<YOUR_MAP_PARAMS>">
</picture

The important part of that <source> tag is the media="(prefers-color-scheme: dark)" to indicate it should be shown during dark times and appending all the style tags (below) which encode dark-friendly map styles.

&style=element%3Ageometry%7Ccolor%3A0x242f3e&style=element%3Alabels.text.stroke%7Ccolor%3A0x242f3e&style=element%3Alabels.text.fill%7Ccolor%3A0x746855&style=feature%3Aadministrative.locality%7Celement%3Alabels.text.fill%7Ccolor%3A0xd59563&style=feature%3Apoi%7Celement%3Alabels.text.fill%7Ccolor%3A0xd59563&style=feature%3Apoi.park%7Celement%3Ageometry%7Ccolor%3A0x263c3f&style=feature%3Apoi.park%7Celement%3Alabels.text.fill%7Ccolor%3A0x6b9a76&style=feature%3Aroad%7Celement%3Ageometry%7Ccolor%3A0x38414e&style=feature%3Aroad%7Celement%3Ageometry.stroke%7Ccolor%3A0x212a37&style=feature%3Aroad%7Celement%3Alabels.text.fill%7Ccolor%3A0x9ca5b3&style=feature%3Aroad.highway%7Celement%3Ageometry%7Ccolor%3A0x746855&style=feature%3Aroad.highway%7Celement%3Ageometry.stroke%7Ccolor%3A0x1f2835&style=feature%3Aroad.highway%7Celement%3Alabels.text.fill%7Ccolor%3A0xf3d19c&style=feature%3Atransit%7Celement%3Ageometry%7Ccolor%3A0x2f3948&style=feature%3Atransit.station%7Celement%3Alabels.text.fill%7Ccolor%3A0xd59563&style=feature%3Awater%7Celement%3Ageometry%7Ccolor%3A0x17263c&style=feature%3Awater%7Celement%3Alabels.text.fill%7Ccolor%3A0x515c6d&style=feature%3Awater%7Celement%3Alabels.text.stroke%7Ccolor%3A0x17263

Presto! When users visit their site in a browser supporting night mode, the media query will kick in and render the styled map. Users browsing in light mode (or without support for night mode) will see the regular style. The <picture> element is smart enough to only load 1 image, so you shouldn't be charged twice.

WARNING: If you're using URL Signing to better secure your Static Maps requests, you will need to generate a new signature for the styled map!

Here's a JSFiddle that shows it all in action, you'll see a light or dark map depending on what your browser is currently requesting.

Practical Differential Privacy w/ Apache Beam

Brian Michalski — Wed, 02 Dec 2020 03:59:29 +0000

One of the most durable techniques to protect user privacy is through differential privacy. In a previous post, we explored how to build an Apache Beam pipeline that extracted and counted ngrams from HackerNews comments. Today, we'll take the same pipeline and upgrade it with some differential privacy goodness using Privacy-on-Beam from Google's differential privacy library.

Step 1: Identifying the User

Counterintuitively, adding differential privacy to a Beam pipeline requires you to have an ID for each user. You don't need to know who the user is, you just need access to a stable identifier for them in your dataset. Good starting choices include that autoincrementing user_id field or a username/email address. The ID you pick should map as closely as possible to the entity whose privacy you are trying to protect.

💁Tip: To err on the safe side, consider hashing or encrypting this ID to prevent yourself from accidentally logging it or debugging with it.

Since we're using HackerNews comments, the author field is a pretty good choice. We'll start with some changes to grab the author for each comment and propagate that user along through the ngram extraction.

// CommentRow models 1 row of HackerNews comments.
type CommentRow struct {
    Author string `bigquery:"author"`
    Text   string `bigquery:"text"`
}

// AuthorNgram represents an ngram and it's author.
type AuthorNgram struct {
    Author string
    Ngram  string
}

const query = `SELECT author, text
FROM ` + "`bigquery-public-data.hacker_news.comments`" + `
WHERE time_ts BETWEEN '2013-01-01' AND '2014-01-01'
AND author IS NOT NULL AND text IS NOT NULL
LIMIT 1000
`

func main() {
 // ...

    authorNgrams := beam.ParDo(s, func(row CommentRow, emit func(AuthorNgram)) {
        for _, gram := range ngram(row.Text, 1, 2, 3) {
            emit(AuthorNgram{Author: row.Author, Ngram: gram})
        }
    }, rows)

 // ...
}

Step 2: Setup Privacy Budget

In differential privacy-land, epsilon and delta are the main ways of controlling how much can be learned about any specific user. Bigger numbers = less privacy. For our pipeline, we'll pick sample values of epsilon = 4 and delta = 0.0001.

Why 4 and 10^-4? I don't know. Apple uses an ε=4 according to it's Differential Privacy Overview. I'd like to write a post on how to pick these numbers once I learn more.

// Configure differential privacy parameters.
epsilon := float64(4)   // ε = 4
delta := math.Pow10(-4) // Δ = 1e-4.
spec := pbeam.NewPrivacySpec(epsilon, delta)

Step 3: Make Private Data

Apache Beam pipelines use a PCollection as the primary container for data. Privacy-on-Beam introduces a new container, the PrivatePCollection, which acts like a PCollection but knows how to preserve privacy along the way.

Using the PrivacySpec from Step 2, and the PCollection<AuthorNgrams> from Step 1, we can build a PrivatePCollection by letting the library know which field has our user id, in this case, a reference to the Author field of the AuthorNgram struct. Passing the string name of a struct field feels a bit weird, but whatever.

pgrams := pbeam.MakePrivateFromStruct(s, authorNgrams, spec, "Author")

Step 4: Do Stats

In our previous pipeline, counting the ngrams was as simple as stats.Count(s, ngrams). Now that we have a PrivatePCollection there's a bit more work involved.

First, we need to simplify the data to just the ngram, converting our PrivatePCollection<AuthorNgram> to a PrivatePCollection<string>. Behind the scenes, the PrivatePCollection will keep track of the author. We need to call the ParDo function from the privacy-on-beam package for this transform, not the usual beam one. It works the same.

ngrams := pbeam.ParDo(s, func(row AuthorNgram, emit func(string)) {
    emit(row.Ngram)
}, pgrams)

With our private pcollection of ngrams we're now ready to Count them. Privacy-on-beam implements its own stat functions which are where all the real magic happens.

When calling pbeam.Count we'll also need to pick two more privacy parameters controlling the count behavior:

How many partitions (aka ngrams) a user can contribute.
How many times a user can contribute to one partition (use the same ngram).

To keep it simple, let's say that a user can contribute up to 700 different ngrams and can contribute 2 times to each ngram. In practice, this means if User A makes 5 comments saying "great idea", only 2 of them will be counted. If User B writes enough comments to contribute 701 unique ngrams, 1 of them will be randomly dropped. These parameters help remove outliers from the data which reduces the amount of noise you see in the output.

counts := pbeam.Count(s, ngrams, pbeam.CountParams{
    MaxPartitionsContributed: 700,
    MaxValue: 2,
})

Fin

With that, your upgrade is complete! The counts returned can be used very close to the old pipeline, it's a PCollection<string, int64> that you can write to a text file, upload to BigQuery, further manipulate, etc. Unlike the first pipeline though, the ngrams here are differentially private... we'll never know who wrote the HackerNews comments which contributed to them.

You can find the end-to-end code on Github, and a diff showing all the changes.

⚠️Warning: In my experience running differentially private pipelines generally takes longer and requires more compute resources. Expect longer runtimes and, if you run this on Dataflow, more instances.

Go + BigQuery : Beam for Beginners

Brian Michalski — Mon, 03 Aug 2020 03:44:51 +0000

Curious how to get started reading from BigQuery using Apache Beam in Go? In this post, we'll walk through the basics of Apache Beam and write a basic Go pipeline that reads from BigQuery and computes some ngrams.

Beam Intro

Apache Beam is an open-source SDK for writing "big data" processing pipelines complete with Python, Java, and Go implementations. If all goes well, you can write a pipeline using Apache Beam and then run it locally or deploy it on The Cloud using GCP Dataflow, Apache Flink, Spark, etc where it can magically scale up to handle a large amount of data.

Since Beam pipelines can magically scale you have to write your pipeline using Beam's special methods and types. This allows your pipeline to be sharded across dozens/hundreds of workers behind the scenes. For beginners there are two key constructs to know:

PCollection: A PCollection is a fancy array. Most steps of your pipeline will take a PCollection as input, run a function on each element, and put that output in another PCollection.
ParDo: A ParDo is a function that runs on each PCollection element. When it runs, it can append one or more elements to the resulting PCollection.

Note: This is an oversimplified introduction to Apache Beam. Fancier operations like group/combine/join require more functions you can learn about in the docs.

Step 1: Boring Boilerplate

There is a bunch of boilerplate code that will import beam, construct a new pipeline (beam.NewPipeline), and execute it (beamx.Run). In the middle, we've left space to build all the pieces of the pipeline.



package main

import (
    "context"
    "flag"

    "github.com/apache/beam/sdks/go/pkg/beam"
    "github.com/apache/beam/sdks/go/pkg/beam/log"
    "github.com/apache/beam/sdks/go/pkg/beam/x/beamx"
)

func main() {
    flag.Parse()
    beam.Init()

    ctx := context.Background()
    p := beam.NewPipeline()
    s := p.Root()

    // Build the pipeline here.

    if err := beamx.Run(ctx, p); err != nil {
        log.Exitf(ctx, "Failed to execute job: %v", err)
    }
}

Step 2: Reading from BigQuery

Pipelines written in Go read from BigQuery just like most other Go programs, running a SQL query and decoding the results into structs that match the returned fields.

For our example, we're going to be reading HackerNews comments from the BigQuery public dataset so we'll need to add a struct which models that result and then a SQL query to query the data. For testing purposes, consider adding a LIMIT clause to your SQL query to reduce how much data it pulls.

Wiring this up as the first step of our pipeline is as easy as adding the bigqueryio.Query function which returns one of those PCollections we talked about earlier.



import (
    ...
    "reflect"

    "github.com/apache/beam/sdks/go/pkg/beam/io/bigqueryio"
    "github.com/apache/beam/sdks/go/pkg/beam/options/gcpopts"
    ...
)

type CommentRow struct {
    Text   string `bigquery:"text"`
}

const query = `SELECT text
FROM ` + "`bigquery-public-data.hacker_news.comments`" + `
WHERE time_ts BETWEEN '2013-01-01' AND '2014-01-01'
LIMIT 1000
`

func main() {
    ...
    p := beam.NewPipeline()
    s := p.Root()
    project := gcpopts.GetProject(ctx)

    rows := bigqueryio.Query(s, project, query,
        reflect.TypeOf(CommentRow{}), bigqueryio.UseStandardSQL())

    ...
}

In the code above, rows will be a PCollection<CommentRow>, effectively a big array where each element is one comment from HackerNews. We're also using the gcpopts utility which helps grab the GCP Project which BigQuery should bill against; you could just hardcode a string too.

StandardSQL vs LegacySQL

Currently, BigQueryIO defaults to LegacySQL in Apache Beam. I do not like using anything "legacy" so the SQL query above uses Standard SQL (note the unusual backtick escaping around the table name) and sets bigqueryio.UseStandardSQL to enable this. This is a new feature for the Go SDK in Apache Beam 2.23, if your environment is using something older you'll need to upgrade to escape out of LegacySQL mode.

Step 3: Ngram Extraction

Ngrams (n-grams) extract phrases from blocks of text. The text "quick brown fox" contains 3 unigrams: ["quick", "brown", "fox"], 2 bigrams: ["quick brown", "brown fox"] and 1 trigram: ["quick brown fox"].

We can extract them using a function like this:



func ngram(str string, ns ...int) []string {
    split := strings.Split(str, " ")
    results := []string{}

    for _, n := range ns {
        i := 0
        for i < len(split)-n+1 {
            results = append(results, strings.Join(split[i:i+n], " "))
            i++
        }
    }

    return results
}

Since we're dealing with PCollections here, the implementation isn't as easy as calling that function in a for-loop over rows. Instead, we have to us the ParDo function, which can distribute this operation across many machines and later aggregate the results. Imagine if this ngram() function was very expensive or slow, it might make sense to have lots of workers extracting ngrams in parallel to get the job done faster. Beam will figure that out for us.



ngrams := beam.ParDo(s, func(row CommentRow, emit func(string)) {
    for _, gram := range ngram(row.Text, 1, 2, 3) {
        emit(gram)
    }
}, rows)

This snippet takes the rows input (specified last) and runs the inline-function on each element (aka each CommentRow). The function also has a reference to an emitter function emit which is used to collect the results and package them back up into a PCollection. A loop calls emit() on each value returned from the ngram() function above. The result is ngrams which is an array of each ngram extracted, a PCollection<string>.

Step 4: Analytics

Now that we have all the ngrams extracted and floating around in this big magical array it's time to count how many times each value is found in the array. Remember, the previous step just extracted the raw words -- it did not attempt to count/dedup/aggregate them.

Counting occurrences of things is a pretty common problem with special ways of distributing across multiple workers. Luckily we don't need to deal with any of that, just use the stats.Count function.



counts := stats.Count(s, ngrams)

This function returns yet another PCollection, this time containing a key-value pair (aka KV in Beam-lingo) where the key is the ngram string, and the value is an int count... essentially a PCollection<{string, int}>.

Step 5: Outputting

Now that we have the data we want it's time to output it somewhere. For this example, we'll dump it to a text file. To do this we need to make a collection of strings (PCollection<string>). Another ParDo to the rescue!



formatted := beam.ParDo(s, func(gram string, c int) string {
    return fmt.Sprintf("%s,%v", gram, c)
}, counts)

This ParDo uses a slightly different syntax than the previous one. Instead of exposing an emit() function the inline function just returns a string. We can use this simpler(?) syntax since each element of the input collection counts only converts to one output element; in our previous use building the ngrams each input string had multiple output values.

The result here is that formatted contains a list of strings suitable for printing, PCollection<string>.

We then call the textio.Write function and point it to an output path:



textio.Write(s, "/tmp/output.txt", formatted)

Final Code



package main

import (
    "context"
    "flag"
    "fmt"
    "reflect"
    "strings"

    "github.com/apache/beam/sdks/go/pkg/beam"
    "github.com/apache/beam/sdks/go/pkg/beam/io/bigqueryio"
    "github.com/apache/beam/sdks/go/pkg/beam/io/textio"
    "github.com/apache/beam/sdks/go/pkg/beam/log"
    "github.com/apache/beam/sdks/go/pkg/beam/options/gcpopts"
    "github.com/apache/beam/sdks/go/pkg/beam/transforms/stats"
    "github.com/apache/beam/sdks/go/pkg/beam/x/beamx"
)

// CommentRow models 1 row of HackerNews comments.
type CommentRow struct {
    Text string `bigquery:"text"`
}

const query = `SELECT text
FROM ` + "`bigquery-public-data.hacker_news.comments`" + `
WHERE time_ts BETWEEN '2013-01-01' AND '2014-01-01'
LIMIT 1000
`

// ngram extracts variable sizes of ngrams from a string.
func ngram(str string, ns ...int) []string {
    split := strings.Split(str, " ")
    results := []string{}

    for _, n := range ns {
        i := 0
        for i < len(split)-n+1 {
            results = append(results, strings.Join(split[i:i+n], " "))
            i++
        }
    }

    return results
}

func main() {
    flag.Parse()
    beam.Init()

    ctx := context.Background()
    p := beam.NewPipeline()
    s := p.Root()
    project := gcpopts.GetProject(ctx)

    // Build a PCollection<CommentRow> by querying BigQuery.
    rows := bigqueryio.Query(s, project, query,
        reflect.TypeOf(CommentRow{}), bigqueryio.UseStandardSQL())

    // Extract unigrams, bigrams, and trigrams from each comment.
    // Builds a PCollection<string> where each string is a single
    // occurrence of an ngram.
    ngrams := beam.ParDo(s, func(row CommentRow, emit func(string)) {
        for _, gram := range ngram(row.Text, 1, 2, 3) {
            emit(gram)
        }
    }, rows)

    // Count the occurrence of each ngram.
    // Returns a PCollection<{string, count: int}>.
    counts := stats.Count(s, ngrams)

    // Convert each count row into a string suitable for printing.
    // Returns a PCollection<string>.
    formatted := beam.ParDo(s, func(gram string, c int) string {
        return fmt.Sprintf("%s,%v", gram, c)
    }, counts)

    // Write each string to a line in the output file.
    textio.Write(s, "/tmp/output.txt", formatted)

    // Now that the pipeline is fully constructed, we execute it.
    if err := beamx.Run(ctx, p); err != nil {
        log.Exitf(ctx, "Failed to execute job: %v", err)
    }
}

Running this with a command like $ go run main.go --project=<gcp-project-id> will kick off a local pipeline and output some ngrams to /tmp/output.txt. To run a more scalable version, perhaps without the LIMIT SQL clause, check out the "Dataflow" Tab on the Go Quickstart with the right flags to run on GCP Dataflow.

Happy analyzing!

Using GitLab Managed Kubernetes

Brian Michalski — Wed, 29 Jul 2020 23:55:18 +0000

In our last journey, we connected Gitlab to a Kubernetes cluster exploring the "Add Kubernetes Cluster" button. Now that our cluster is set up, let's put it to use!

Running Applications

GitLab's Auto DevOps is too magical for me. I don't understand Helm well enough to use it; and I suspect that my use of Bazel's docker rules will cause problems. Instead, we'll manually add a deploy step to update our application.

1. Add a Helm chart.

Helm charts are essentially yaml templates for Kubernetes resources. They allow you to add variables and set/override them when applying the chart... something you might traditionally do using sed or another bash trick.

To create a new Helm chart, run a command like:

mkdir devops; cd devops
helm create myapp

This will create a myapp/ directory with all of the pieces of a Helm template inside. You can then preview the YAML output of this template using a command like:

helm install --dry-run --debug myapp

Helm charts can be pretty intimidating. The meat of the template is in templates/deployment.yaml file. If you want, you can delete all of the fancy templating in this file and replace it with a vanilla yaml for a deployment object.

For simple apps, tweak things as necessary in your values.yaml. I kept doing this, and comparing the --dry-run output to a hand-written deployment yaml until I got it looking sort of correct.

2. Pulling from GitLab's container registry

Auth

If you're using a private container registry, like Gitlab's, you will need to store some login information in a secret in your Kubernetes cluster.

To begin, head to your project settings in GitLab and navigate to the Registry section. Create a Deploy Token and note the username and password.

Next, follow this tutorial to upload that secret to your cluster. I ended up using a command like:

kubectl create secret docker-registry regcred \
--docker-server=registry.gitlab.com \
--docker-username=gitlab+deploy-token-123456 \
--docker-password=p@ssw0rdh3r3 \
--docker-email=me@gmail.com

NOTE: You probably need to run this command with a --namespace flag that sets this secret in the same namespace that GitLab picks to run your application. If it's not set, you'll see errors trying to fetch the container.

Helm Chart Tweak

To use this regcred secret, add it to the imagePullSecrets section of your values.yaml file like this:

image:
  repository: registry.gitlab.com/bamnet/project/image
  pullPolicy: IfNotPresent
  tag: ""

imagePullSecrets:
  - name: regcred

3. .gitlab-ci.yml updates

To apply this Helm chart as part of your CI/CD pipeline, add a job to your .gitlab-ci.yml file like the following:

deploy_myapp:
  stage: deploy
  image:
    name: alpine/helm:latest
    entrypoint: [""]
  script:
    - helm upgrade
      --install
      --wait
      --set image.tag=${CI_COMMIT_SHA}
      myapp-${CI_COMMIT_REF_SLUG}
      devops/myapp
  environment:
    name: production

The most important part of this entire job is the environment section. Gitlab only exposes Kubernetes connection information (via environment variables helm and kubectl automatically use) when the deploy stage has an environment set. Without this section, you will get errors connecting to your cluster.

There are 3 parts of the helm upgrade command worth noting:

--set image.tag=${CI_COMMIT_SHA} overrides the tag portion of our deployment.yaml, passing in the git commit hash. This assumes your containers are tagged with the commit that generates them. If you don't do this, consider a static value like latest.
myapp-${CI_COMMIT_REF_SLUG} provides the name for this deployment. If you're deploying from the master branch, this will be myapp-master. This must be unique, so tweak the myapp- prefix if you have multiple applications.
devops/myapp at the end specified the folder where the Helm chart files are located.

Pushing this new gitlab-ci file should trigger an automatic deployment to your Kubernetes cluster. Sit back, relax, and watch the dashboard to see it work.

If this is your first push, be on the lookout for a new namespace to be created.. probably something like gitlabproject-123456-production.

Troubleshooting Tips

Locally run helm install --dry-run to see the planned configuration. If it doesn't look right locally, there is no way GitLab is going to get it right.
Connect to the Kubernetes Dashboard to see why deployments fail.
Make sure your image names and tags match between your registry hosting the things and the deployment yaml trying to create them.
Use the --namespace flag to make sure your registry credentials end up in the right namespace.

Monitoring Applications

GitLab has a one-click install of Prometheus. I am a sucker for one-click install buttons and wanted to give it a spin monitoring my Go application.

1. Exporting Metrics

The OpenTelemetry docs and examples are a good starting point. Prometheus needs and HTTP endpoint to grab the metrics from, a very simple Prometheus exporter looks like this.

func initMeter() {
    exporter, err := prometheus.InstallNewPipeline(prometheus.Config{})
    if err != nil {
        log.Panicf("failed to initialize prometheus exporter %v", err)
    }
    http.HandleFunc("/metrics", exporter.ServeHTTP)
    go func() {
        _ = http.ListenAndServe(":2222", nil)
    }()

    fmt.Println("Prometheus server running on :2222")
}

func main() {
    initMeter()
     // Rest of your code here.

2. Adding Annotations.

Gitlab's one-click Prometheus automatically scrapes metrics from any resource that has certain annotations in place which tell it how to scrape. Add the following to your chart.yaml file:

podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/path: /metrics
  prometheus.io/port: "2222"

That's it!

Troubleshooting Tips

Manually connect to your application and see the exported metrics. Forward the port using kubectl port-forward -n <gitlab-created-namespace> deployments/myapp-master 2222:2222 and point your browser at http://localhost:2222.
Manually connect to Prometheus and use the web UI to see what metrics are being scraped and run queries against them. Forward the port using kubectl port-forward -n gitlab-managed-apps service/prometheus-prometheus-server 9090:80 and point your browser to http://localhost:9090.

Dockerfile to Bazel

Brian Michalski — Sun, 31 May 2020 21:36:02 +0000

I recently needed to build a Docker image as part of a large Bazel project I've been working on. There are a handful of instructions out there how to take a binary built using bazel (like a go_binary) and convert it along into a container (go_binary =>go_image =>container_image) but it's less clear how to take a basic Dockerfile with someone else's image and add it to your project. Here's what to do:

1. Be prepared to throw out the `Dockerfile`.

Bazel's basic Docker rules don't include any rules that let you just point at an existing Dockerfile and say build. Instead, we have to recreate the Dockerfile in our BUILD.bazel. Our existing Dockerfile will be a useful reference so don't throw it out quite yet, but if you've gotten attached to it now would be a good time to say goodbye.

For our example, let's convert the following Dockerfile which takes an existing Docker image, adds a file to it, and sets a command line flag.

FROM bamnet/bqproxy

COPY queries.yaml queries.yaml
CMD ["--project=your-project-id"]

2. Add remote dependencies.

Anything that Bazel needs to download remotely need to be specified in a top-level WORKSPACE config. In this case, we have to add the base image specified in the FROM line to our WORKSPACE config since the base image lives on a container registry and not in our project.

load("@io_bazel_rules_docker//container:container.bzl", "container_pull")

container_pull(
    name = "bqproxy_latest",
    registry = "index.docker.io",
    repository = "bamnet/bqproxy",
    tag = "latest",
)

This will expose a @bqproxy_latest//image item that I can reference later. Like other bazel rules, the name field can contain whatever I want to refer to the image by elsewhere using the @<name>//image format.

Note that my existing Dockerfile didn't specify the registry - it defaulted to Docker Hub automatically. Bazel doesn't like to make assumptions like this, you need to specify a registry path.

Platform	registry
Docker Hub	index.docker.io
Google Container Registry	gcr.io
Gitlab Container Registry	registry.gitlab.com
Github Packages¹	docker.pkg.github.com

3. Rebuild your `Dockerfile` as a `container_image`.

Bazel uses the container_image rule to build Docker images. Add one to the appropriate BUILD.bazel file for your project.

load("@io_bazel_rules_docker//container:container.bzl", "container_image")

container_image(
    name = "my_image",
    // TODO: Figure this out.
)

To replicate our earlier Dockerfile we need to set several options:

base corresponds to the FROM line. Specify the name from the container_pull rule added to the WORKSPACE file. In this example, @bqproxy_latest//image.
cmd specifies command-line options, it can take a string or an array.
files specifies files which should be copied into the image. All files will be added to the current working directory. You don't have the ability to set the destination for files being imported like you can using a Docker COPY config. Instead, you can use workdir to globally change the working directory or use a symlink rule to map files into different locations of the file system.

See the container_image docs for the full set of options.

4. Test it out!

Instead of running docker build . to build a local docker image, use bazel run :my_image (substitute the container_image name for my_image).

$ bazel run :my_image
INFO: Build completed successfully, 1 total action
Loaded image ID: sha256:b5da9bf97f1cdae8c66a775344d3c5ec99002a753973505aa1301bfb0f0b2649
Tagging b5da9bf97f1cdae8c66a775344d3c5ec99002a753973505aa1301bfb0f0b2649 as bazel/server:my_image

This will build a Docker image and display it's output path, like bazel/server:my_image which you can run just like any other docker image. I used docker run -ti --rm bazel/server:my_image but you may need to pass additional args.

Our final conversion looks like:

Source Dockerfile

FROM bamnet/bqproxy

COPY queries.yaml queries.yaml
CMD ["--project=your-project-id"]

turned into

Target WORKSPACE

load("@io_bazel_rules_docker//container:container.bzl", "container_pull")

container_pull(
    name = "bqproxy_latest",
    registry = "index.docker.io",
    repository = "bamnet/bqproxy",
    tag = "latest",
)

Target BUILD.bazel

load("@io_bazel_rules_docker//container:container.bzl", "container_image")

container_image(
    name = "my_image",
    base = "@bqproxy_latest//image",
    cmd = ["--project=your-project-id"],
    files = [
        "//server:queries.yaml",
    ],
    workdir = "/",
)

As of May 2020, Github Packages requires authentication even for public packages. In order to use them in your Bazel project you'll need to configure authentication. ↩

Dynamic nginx config on Cloud Run

Brian Michalski — Fri, 06 Dec 2019 04:08:43 +0000

Google Cloud Run has quickly become one of my favorite cloud products. Not only is it super simple to setup (*cough* k8s *cough*), but it also has a generous free tier and lets you run nearly any stateless container. It also supports custom domains, with a small CNAME tweak you can start serving foo.your-domain.com over HTTPS without any extra work. Where have you been all my life?

This got me thinking -- could I just run an nginx container on Cloud Run and, with a reverse proxy config, glue together a bunch of microservices and static resources under a single domain? I could effectively use Cloud Run as a frontend server / poor-man's load balancer (without the actual balancing features).

I wanted to avoid baking or hardcoding my nginx.conf file in each Docker image. Launching a new microservice on my domain should require a simple config change, not building, pushing, and deploying a whole new Docker image. To do that, we have to use the 1 editable thing Cloud Run provides: environment variables.

With some clever bash-foo, we can build a Dockerfile which takes a $CONFIG environment variable and injects it into to a file before starting nginx. We can change the CONFIG variable as many times as we want without needing to rebuild a new Docker image each time.

FROM nginx
RUN rm /etc/nginx/conf.d/default.conf
COPY site.template /etc/nginx/conf.d/site.template
CMD /bin/bash -c "envsubst < /etc/nginx/conf.d/site.template > /etc/nginx/conf.d/site.conf && exec nginx -g 'daemon off;'"

[Dockerfile on Github]

With that in place, configuring our nginx instance is as easy as copying an nginx config blob into the Cloud Run UI. The UI doesn't support multi-line strings so everything gets mashed into 1 line, but nginx doesn't care about line breaks when it's reading the config.

As an example, you can run a very simple reverse proxy by pasting this into a CONFIG variable:

location / {
  proxy_http_version 1.1;
  proxy_set_header Connection "";
  proxy_buffering off;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_pass http://www.google.com;
}

Or something a bit fancier which mixes in some Cloud Storage resources (a great way to expose them over HTTPS on a custom domain on the cheap):

location /static/ {
  proxy_http_version 1.1;
  proxy_set_header Connection "";
  proxy_buffering off;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_pass https://storage.googleapis.com/cloud-run-fun-static-test/;
}

location / {
  proxy_http_version 1.1;
  proxy_set_header Connection "";
  proxy_buffering off;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_pass http://www.google.com;
}

In the Cloud Run UI, this looks a little bit like:

If you want to quickly try this out yourself, you can deploy my nginx image from gcr.io/cloud-run-fun/nginx:latest or jump right to the Console:

Full code @ https://github.com/bamnet/cloud-run-fun.

Forem: Brian Michalski

Stop Hardcoding Google Maps API Keys!

JWT Intro

Generating a JWT

But what protects the JWT generator?

Sending a JWT to Google Maps Platform

Putting it all together

Server-Side

Client-Side

Not-So-Frequently Asked Questions

GRPC Performance in Flutter

Static Maps At Night

Practical Differential Privacy w/ Apache Beam

Step 1: Identifying the User

Step 2: Setup Privacy Budget

Step 3: Make Private Data

Step 4: Do Stats

Fin

Go + BigQuery : Beam for Beginners

Beam Intro

Step 1: Boring Boilerplate

Step 2: Reading from BigQuery

StandardSQL vs LegacySQL

Step 3: Ngram Extraction

Step 4: Analytics

Step 5: Outputting

Final Code

Using GitLab Managed Kubernetes

Running Applications

1. Add a Helm chart.

2. Pulling from GitLab's container registry

Auth

Helm Chart Tweak

3. .gitlab-ci.yml updates

Troubleshooting Tips

Monitoring Applications

1. Exporting Metrics

2. Adding Annotations.

Troubleshooting Tips

Dockerfile to Bazel

1. Be prepared to throw out the Dockerfile.

2. Add remote dependencies.

3. Rebuild your Dockerfile as a container_image.

4. Test it out!

Dynamic nginx config on Cloud Run

1. Be prepared to throw out the `Dockerfile`.

3. Rebuild your `Dockerfile` as a `container_image`.