Forem: Zach A. Thomas

First Look at Amazon DataZone

Zach A. Thomas — Thu, 30 Mar 2023 14:05:41 +0000

AWS has released a public preview of an impressive suite of capabilities called Amazon DataZone.

This is based on a thorny problem in trying to tap the potential of business data: data is often trapped within the corporate silo where it was created, e.g. Sales, Operations, Marketing, etc. Even when we succeed in breaking through the silo to get access to some other group's data, we have a host of additional problems: cleaning the data, normalizing data definitions, questions of compliance (encryption, the handling of personally identifying information, retention policy, etc.), and making it easy to use.

Businesses have tried solving this problem by centralizing the data access problem in a new group, sometimes called Data Science. This group is in charge of the data lake, and cleaning and describing all the data sources. This is almost a good idea, but it creates an unfortunate bottleneck; the agility of the business is hampered by the need to get the data source tidied and blessed by an overworked group of people who are not the domain experts in any of the data sources they're made responsible for. The capabilities of the Data Science team are stretched thin and they don't scale with the appetite for business data.

A new, decentralized approach to these problems was described by Zhamak Dehghani from ThoughtWorks. It's called Data Mesh, and it's based on the idea that each data source should be treated as a data product, and the owners of the data source are also the owners of the data product. You can think of a data source as a kind of API, and many groups can contribute to the shared catalog, including providing the documentation and access rules. The final piece of the Data Mesh puzzle is that governance should be federated rather than centralized.

Amazon appears to be taking this approach seriously, and DataZone is a system of systems for connecting producers of data with consumers from a hub that is self-service for both kinds of participants. From their product page, the key features are:

catalog: Search for published data, request access, and start working with your data in days instead of weeks.
projects: Collaborate with teams through data assets, and manage and monitor data assets across projects.
portal: Access analytics with a personalized view for data assets through a web-based application or API.

It's clear you still need the data specialists, but they can be the ones empowering disparate groups with these new capabilities so they can begin to be autonomous. It's consistent with the concept of a platform team from Team Topologies.

I look forward to getting my hands dirty with Amazon's new offering. I don't know how good their implementation is yet, but I have become convinced that this architecture is a source of significant competitive advantage.

(image credit Rhk111, CC BY-SA 4.0 https://creativecommons.org/licenses/by-sa/4.0, via Wikimedia Commons)

Why Microservices Are Not Just Another Fad

Zach A. Thomas — Sat, 11 Feb 2023 21:32:09 +0000

I don't like chasing technology fads. It's important to maintain a healthy level of skepticism, so you don't waste your time, or even end up making some expensive mistakes.

If your skepticism is too well-developed, however, you might overlook some solutions to your thorny problems.

The debate about microservices is important, insofar as they are not a silver bullet, and you could wind up in trouble if you think you can just "sprinkle some microservices on it" and then lean back and watch the magic happen. See this post by Martin Fowler about what kind of processes you should have in place before it makes sense to try to introduce microservices to your architecture.

But sometimes our debates are unproductive, because we get hung up on less relevant details, like how many lines of code makes it "micro," or whether serverless makes microservices obsolete. For me, that's like arguing over what color the paint job should be when you really should be deciding whether you're getting a car, a truck, or a city bus.

The real power unlocked by microservices is not about the particulars of the technology choices you make. What it's really about is enabling a workflow whereby a small team can deliver at whatever cadence they like best with no handoffs required to any other group. I call this partitioning the value streams. In waterfall workflows, we throw things over the wall and wait. We might get some feedback days later and then start the cycle again. One of the great paradoxes of software development is that we can deliver faster and reduce risk at the same time.

Imagine a company that has grown quickly, but is still not large. They have a monolithic application that is released once a week, and is feature-frozen a week before that. They're up to about two dozen full-time software engineers, so they're starting to feel the coordination overhead of working on a single large component. Here's a partial list of problems routinely faced by this group that are also solved by partitioning the value streams:

someone writes an inefficient database query, and the whole business goes down
the CI/CD build takes ten minutes to run, and if any change is made to main while that happens, it has to start over
people race to get their changes in before the feature freeze. If you make the cutoff, your changes will go live in a week. If you miss the cutoff, it takes two weeks to ship
a bug in a minor feature can take the whole business down, so a change to anything necessitates performing hours of regression testing on everything
some engineers want to be consulted in case something they wrote is going to be changed
the monolith can scale horizontally, but it is not possible to scale high-demand APIs independently of low-demand APIs
innovating is so risky that experiments are discouraged and the default answer is "no"
people responding to incidents are unlikely to have worked on the part of the application that is misbehaving
availability of the system is only as good as the least available subsystem
the dependency graph of the code is unreadable
the codebase suffers from high coupling and low cohesion
since deploys are not from trunk, it's necessary to forward port patches from the release candidate to main

As long as there are organizations addressing these problems (and others) with microservices, the argument that it's a technology fad does not hold any water.

AWS Provisioning Three Ways

Zach A. Thomas — Sun, 10 Apr 2022 19:50:19 +0000

I just read the news about AWS Function URls and I wanted to try it out ASAP. In a nutshell, this allows you to instantly have a public HTTP endpoint for any lambda you've got. This was always possible by configuring an API Gateway, but that is quite complex and provides features that could be overkill if you just want a webhook or something simple.

Most AWS tutorials I find show you how to do things step-by-step in the console. While that's helpful for starting out, what you want in the long run is infrastructure-as-code, which means all the resources in your application should be described in files that can be checked into version control.

I stood up a quick demo in this GitHub repo that shows setting up a simple serverless application (with Function URL!) in three different ways:

pure AWS CLI commands. In this method, you create a shell script to create resources, using commands such as aws iam create-role and aws lambda create-function. This is the "no frills method."
Serverless framework. There's a lot to like about Serverless framework. It provides commands for deploying to different stages (think dev/test/prod) as well as invoking locally for test purposes. This method uses is a front-end for AWS CloudFormation, which was a pioneer in infrastructure-as-code wave. One downside is that provisioning takes a bit longer, so the feedback loop when making code updates is not as quick as I like.
Terraform. Terraform from Hashicorp is a big deal. They have pulled off an amazing feat: to unify all the clouds and over 1,000 different providers into a single provisioning language. One of my favorite things about Terraform is that it helps you see differences between your application as configured and how it really is in the real world. Configuration drift really exists, and it is important to stay on top of it.

So far, I haven't even mentioned that the demo code I wrote is in Clojure, my favorite language. Usually, it's quite a bit of effort to deploy clojure, because you need a compiler and build pipeline to get your artifacts ready to run. Thanks to a lightweight interpreter called nbb, you can interpret Clojure in a lambda as easily as you can interpret JavaScript. If you're not into Clojure, it's not a problem. These three forms of provisioning lambdas work with any of the lambda runtimes (node, python, java, go, ruby, and .NET).

Kafka Connect Plugins With Clojure

Zach A. Thomas — Wed, 06 Apr 2022 17:02:29 +0000

Kafka Connect has an extension mechanism based on plugins. By implementing certain interfaces, or extending certain classes, you can create one of eight types of extensions:

Connector for piping data into or out of Kafka via external systems
Converter for translating between Kafka Connect's runtime data format and byte[]
HeaderConverter same, but for headers
Transformation for modifying messages as they move through a connector
Predicate used to conditionally apply a Transformation
ConfigProvider for integrating a source of key/value properties use in configuration
ConnectRestExtension for adding your own JAX-RS resources (filters, REST endpoints, etc.) to the Kafka Connect API
ConnectorClientConfigOverridePolicy for enforcing a policy on overriding of client configs via the connector configs

Some of these, like the client config override, are pretty obscure, but connectors and transformations are the meat and potatoes of Kafka Connect, and a custom config provider is like magic for supplying secrets (e.g. passwords) to your configuration.

Adding Clojure to the Mix

The Kafka Connect extension mechanism is pretty great, but if your language of choice is Clojure, can you still create plugins for Kafka Connect? You can! There are a couple of hoops to jump through, which I will describe.

Example Code: Shouting Transform

This example is not something you would really use (unless you LOVE ALL CAPS), but it shows all the parts and configuration for a Kafka Connect transformer (also referred to as a single message transform, or SMT).

This example will examine the value of string messages, and convert certain words to all caps, whichever words you specify in the configuration to the plugin.

e.g., if you set the shouting-words property to "love,nice,shouting" then this message:

Hello, nice world! Do you love shouting?

Will be transformed into:

Hello, NICE world! Do you LOVE SHOUTING?

Use `gen-class`

Clojure has other ways to implement a Java interface, like reify and proxy, but we need gen-class because Kafka Connect configuration expects us to supply it with a class name for it to instantiate. gen-class allows us to create a class with the name of our choice. Here's the full code of the shouting transformer:

  (ns fizzy.plugins.shouting
    (:require [clojure.string :refer [replace upper-case]])
    (:import [org.apache.kafka.common.config ConfigDef ConfigDef$Type ConfigDef$Importance]
             [org.apache.kafka.connect.data Schema$Type])
    (:gen-class
     :name fizzy.plugins.ShoutingTransform
     :extends fizzy.plugins.ClassLoaderImposition
     :implements [org.apache.kafka.connect.transforms.Transformation]))

  (def config (atom {}))

  (defn -configure [this cfg]
    (swap! config assoc this cfg))

  (defn -config [_this]
    (let [field-name "shouting-words"
          value-type ConfigDef$Type/LIST
          importance ConfigDef$Importance/HIGH
          default-value '()
          docstring "Provides a list of words that SHOULD BE SHOUTED"]
      (.define (ConfigDef.) field-name value-type default-value importance docstring)))

  (defn- shout
    [v match]
    "Takes a string value and a string to match and makes the match string upper case"
    (replace v (re-pattern (str "(?i)" match)) (upper-case match)))

  (defn -apply [this record]
    (let [topic (.topic record)
          partition (.kafkaPartition record)
          key-schema (.keySchema record)
          key (.key record)
          value-schema (.valueSchema record)
          value (.value record)
          shout-value (if (string? value))
                       (reduce shout value (get-in @config this "shouting-words"))
                        value)
          timestamp (.timestamp record)
          headers (.headers record)]
      (.newRecord record topic partition key-schema key value-schema shout-value timestamp headers)))

  (defn -close [_this]
    nil)

A Tour of the Code

A transformation plugin is any class that implements org.apache.kafka.connect.transforms.Transformation. The methods we need are:

void configure(Map<String, ?> configs) accept a map of configuration keys/values
ConfigDef config() returns an object which describes the expected configs
ConnectRecord apply(ConnectRecord record) the actual transformation to perform
void close() called on shutdown, for any resources that need to be cleaned up

Our Clojure code stores the configuration map in an atom, with a separate config stored for each instance of the class, since you could have multiple simultaneous instances of the plugin running.The only configuration option we use in this example is "shouting-words", a list of strings that we will convert to all caps in any message where we find them.

The apply function passes all the fields of the ConnectRecord along unmodified except the value, which is transformed by the shout function, but only if the value is a string (a message coming through a connector could be virtually anything).

Our close function is a no-op, because we don't have anything to shut down.

A Class Loading Issue

Kafka Connect provides a dedicated class loader for each plugin, so that all the plugins can have isolated classes and not interfere with each other. This turns out to be a bit of a problem for Clojure, which loads itself using the context class loader rather than the plugin class loader. Furthermore, it uses static initializer blocks to load, so there's only a narrow window of opportunity to intervene. See the Clojure implementation details here.

If your eyes are very sharp, you may have noticed that our gen-class extends another class, fizzy.plugins.ClassLoaderImposition. The sole purpose of this class is to set the context class loader when the class loads. Here is the implementation of that class. Putting the code in a static initializer block allows us to set the context class loader before Clojure needs to access it.

  package fizzy.plugins;

  public class ClassLoaderImposition {
      static {
          Thread.currentThread().setContextClassLoader(ClassLoaderImposition.class.getClassLoader());
      }
  }

To put it bluntly, this is a hack to get Clojure to load in the plugin context, but it works!

Packaging It Up

Kafka Connect has a (configurable) directory where plugins are loaded from. If you're using a local installation of Confluent Platform, the default location is $CONFLUENT_HOME/share/java. In the official Confluent Docker container, it's /usr/share/confluent-hub-components. You can either make an uberjar with your code in it (plus all of its dependencies), or create a subdirectory of the plugins directory with all the jars in it. Note that you should not include jars that come with Kafka Connect, like kafka-clients, slf4j-api, snappy. Creating jars from your project is beyond the scope of this article, but you can use tools.build or leiningen to do it. Remember that once you have copied your plugin files to Connect, you must stop and restart the server. It's a good idea to tail the logs, so you can see any errors that pop up if something goes wrong with loading your plugin.

Adding Your Transformer to a Connector

Once you have a plugin installed in a running Kafka Connect cluster, you use it by specifying its configuration in the connector properties. Here is how our example would be enabled within connector properties:

{
    "name":"my-connector",
    "config": {
       // more config redacted
       "transforms":"shouting",
       "transforms.shouting.type":"fizzy.plugins.ShoutingTransform",
       "transforms.shouting.shouting-words":"hello,world,love,shouting"
    }   
}

Making a `ConfigProvider`

We were very excited to build a custom config provider, because it meant we could get our secrets from Vault instead of passing them around in files or the environment. It is very similar to writing a transformer, with a couple of wrinkles. The interface is:

ConfigData get(String path)
ConfigData get(String path, Set<String> keys)
void configure(Map<String, ?> configs)
void close()

And there are optional methods for subscribing to changes to a given key.

One of the differences with the ConfigProvider (also with ConnectRestExtension if you decide to create one) is that plugins of this type are found using the java.util.ServiceLoader mechanism. This means that your jar file must have a META-INF/service/org.apache.kafka.common.config.provider.ConfigProvider file. The content of this file is a single line with the fully-qualified name of your class which implements ConfigProvider.

When your config provider is installed, you will be able to add properties to Kafka Connect that reference your provider to supply a value. For example, your ENVIRONMENT could include the following:

CONNECT_CONFIG_PROVIDERS=fizzysecrets
CONNECT_CONFIG_PROVIDERS_FIZZYSECRETS_CLASS=fizzy.plugins.SecretsConfigProvider
CONNECT_SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username='\${fizzysecrets:SASL_JAAS_CONFIG_USER}' password='\${fizzysecrets:SASL_JAAS_CONFIG_PASS}';"

One More Hurdle

Getting secrets from a config provider is great, but there is one final hoop to jump through if you're running Kafka Connect with the Confluent Docker image. Part of the startup sequence is running a tool called cub to ensure Kafka is ready before starting Kafka Connect. Because the tool also needs access to your config providers, you must copy your config provider code (along with dependencies) to /usr/share/java/cp-base-new. That way, any secrets you need to reach Kafka will be also be available to cub.

Clojure + Kafka Connect

Clojure is a fun and productive way to extend Kafka Connect. One thing we haven't tried yet is writing our own connector, but that should be a breeze. Good luck!

The Single Biggest Beginner Mistake with DynamoDB

Zach A. Thomas — Sun, 03 Apr 2022 19:18:51 +0000

DynamoDB is an amazingly powerful and performant database, best known for its low latency and elastic scaling characteristics. But there is one trap it is super easy to fall into, especially if you have any background at all in more traditional relational database systems (think MySQL, Postgres, Oracle, and SQL Server).

That's Not Normal, Man!

In the relational database world, the dependable best practice is to normalize your data model. It's a fairly academic topic, but the short version is that every piece of data should have one home (i.e., one table which is its canonical location), and any references to that data in another table will be in the form of a foreign key, which a pointer to its "true" location.

In this way of storing data, an entity can be assembled from all the different rows in all the different tables by asking for a JOIN operation.

If you've ever done a data modeling exercise in this paradigm, you might have started with a table-per-entity. It might feel natural to follow this same process with DynamoDB. Stop!

Don't Be a Joiner

If you have taken DynamoDB for a spin, you may have noticed that there aren't any JOIN operations. This is a feature, not a bug! Let's talk a bit about why JOIN was invented in the first place. In the dawn of SQL databases (let's go back to 1979), storage was scarce and expensive. A database join saves storage at the expense of computation, a tradeoff which made sense for a long time, but doesn't anymore. In the present day, the cost equation is completely flipped: storage is millions of times cheaper, and while computation has also improved (a lot), not by the same orders of magnitude as storage, which means computation is now the bottleneck for cost and performance. DynamoDB achieves remarkable performance by not incurring the computation cost of doing all those joins.

If you make this common mistake (and I did, a bunch!) and continue modeling entities the old way, entity-per-table, you will end up doing all the joins yourself, in your application code. This is the worst of both worlds, because you're giving up the expressive flexibility of SQL while still paying the cost of joins.

So Now What?

What do you do instead? This problem is solvable with a little planning up-front. The best way I know to describe the new way of working is to imagine the query result you want, and store the rows in that form, completely denormalized. There's an old wisdom about performance: the less it has to do, the faster it can be. With denormalized rows in your DynamoDB table, the only work is fetching by your chosen partition key (and optionally a little bit extra to apply attribute constraints before returning).

The Next Level

When you start modeling in this way, the surprising result is that most applications can be implemented with a single database table. This is like waking up from living in the Matrix. When you're ready for more, start with this post by Alex Debrie, and then read everything else he has written! Absorb all of this documentation. And when you're done with that and you're ready to level up again, go find everything by Rick Houlihan, like this talk from re:Invent entitled "Amazon DynamoDB advanced design patterns."

Go With the Flow

When you start using DynamoDB the way it was designed, it will blow your mind. Have fun on the journey!