Forem: James Hrisho

Why did I start a feature flagging startup?

James Hrisho — Thu, 28 Jan 2021 03:31:06 +0000

When the pipe bursts

7am every Wednesday, it was time to release the code. I'd take a weeks worth of code that I knew nothing about, by 2 dozen people, and ship it to production. We tested the code in isolation. We tested it together in our lower environments. But even then when it was up on production who knew what was going to happen. Even if the release itself went well, there was always a chance that a feature was not ready for users. Finding that out at the last minute could push a release to a later day. This would delay other important releases that teams worked hard on.

By batching up releases, we thought we were doing everyone a favor. Customer success only has to learn about the release once a week! One person can own the release process and be an authority on it! We can test everything together all at once, and know for sure that it will work!. We were so wrong.

By batching up releases, we introduced a ton of risk. The release became an all or nothing affair. If any feature in a release wasn't ready for customers, everything we had already communicated to customers was at risk. We would mitigate that risk by creating new problems. Rushing through something or introducing new issues.

The person (me) who was doing the deployment had no context for what exactly I was deploying. Sure the steps we could document the steps and what was in the release. But if something went wrong,, all I knew I could do was stop the whole train. If anything broke, it was impossible to figure out what may have caused it. Even if you are an expert of a system, a week's worth of work by 20+ people is impossible to isolate defects.

After a few too many stressful releases, we decided one of the best investments we could make quickly was feature flags. With feature flags, you separate the release of code to customers from the deployment to production. Code for a new feature can live in a production environment for any duration of time without disruption to your customer. When the feature is ready, any team member can turn it on. If there is an issue for any reason, Anyone can shut the feature off, limiting impact.

We saw improvements to our process right away. We stopped batching deployments, instead we would deploy to production regularly. Each change getting sent to production was easy to discern, it could be as small as a couple lines of code. If there was an issue it would be easy to track and identify.

When a feature was ready for customers, anyone on our team, especially our product owners, could turn it on. We even went further. To deliver value cusotmers sooner, we could build smaller features and only release it to users who had no known edge cases. Once we had more time to identify how to approach the edge cases those customers would receive the updates. Teams became more confident, less stressed, and could get feedback on their work in a much shorter timeframe.

This was a transformative experience for me. I've been preaching the gospel ever since. I knew I could help get teams to work faster by enabling feature flags for more teams.

That's why I built Molasses.

The Great Molasses Flood

I started building in June. I focused on the problems I saw with the current tools. I surveyed those who had used feature flagging and A/B testing services in the past to get their opinion. What I found was that while there are great solutions here and there, but there was space for improvement.

Despite being a developer focused tool, so many feature flag services are developer hostile. They can be slow, and make blocking remote calls. They are naive in their ability to target users, sometimes only by IP address or one parameter. They had no way to audit changes. They lacked integrations with platforms for planning, chatops or observability. Teams are slow to remove feature flags, accumulating technical debt. The gaps in these feature flag systems are filled with custom software. Taking away from expensive developer and operations resources.

Molasses allows teams to get up and running with feature flags right away. It supports many different languages and frameworks.

Molasses' clients are fast, non-blocking, and the service pushes updates to the client instantly, no waiting. You can roll out features to specific users and target specific groups. Molasses lets teams focus on the features they are building, not building feature flag technology.

Molasses unlocks A/B testing. We can examine the effectiveness of a change right away. Want to start rolling out a feature to a set of alpha or beta users? No problem! Molasses will let you adjust who receives a feature over time.

Molasses isn't the only tool in your toolchain. It integrates with the systems you already use, like Datadog or Slack. You can use those updates to know if a recent change is causing issues. When your feature is finally rolled out, Molasses has the code clean itself up. It opens a PR in your Github repo removing any stale code.

I'm proud of what I've built so far. I'm excited to watch teams speed up with Molasses.

Molasses is a feature flag and A/B testing service.

Molasses is Live on Product Hunt!

James Hrisho — Fri, 27 Nov 2020 11:17:13 +0000

Molasses is launching on Product Hunt Today!

208 times more often.

According to the latest State of the DevOps report, high performing teams deploy code 208 times more often than a low performing team. While actually having fewer bugs. More deployments?! Less bugs?!

When I read that I was shook. So I decided to help teams faster and ship code with fewer bugs by launching Molasses!

Molasses is a feature flag platform. Feature flags are a technique that allows you to turn on or off features without having to deploy new code, allowing you to release code more often. Molasses supports many different languages and frameworks - React, React Native, TypeScript, JS, Node, Go, Python, and Ruby.

Once you use tools like this you realize how important it is to have powerful tooling around it. So I also included audit logging and an integration with Slack.

I'm excited for folks to try Molasses. If you are looking to give it a test run, let me know and I can help you get started right away. Check out our website, our github and of course our Product Hunt Page!

Making Laundry Less Terrible with Machine Learning

James Hrisho — Sat, 24 Oct 2020 01:43:37 +0000

Since my son was born, I've been doing a lot of laundry. An infant's laundry needs are small (well the clothes are) but frequent so to be efficient you might as well do the whole family's the laundry. When you do enough laundry, you'll notice those little tags. Each garment has a set of shapes - triangles, circles, squares, and other shapes - that intend to describe how to care for it.

Well last year, I got obsessed and started to learn how to interpret each one. It's not complicated but it is not obvious either. So I set out to make it easier on myself and others. It's easy enough to create a pocket guide for laundry tags. But, I thought it would be neat to be able to take a picture of the care tag and present which tag it matched. This seemed like an opportunity to play around with machine learning and computer vision tools as well.

Each time I did laundry I would take photos of the tags. I would then organize them based on which type of care tag it was (Wash at 30 degrees Celsius vs Non-Chlorine Bleach). Using Firebase's Automl Vision Edge, I could then create a machine learning model.
Vision edge generates a TFlite model to use to classify an image. To use the model, I then created an Android app and an iOS app to consume results from the camera.

Here's what the Kotlin code looks like to process an image

override fun onImageSaved(file: File) {
    val uri = file.toUri()
    if (uri != null) {
        // loads the machine learning model
        val localModel = FirebaseAutoMLLocalModel.Builder()
            .setAssetFilePath("manifest.json")
            .build()

        val options = FirebaseVisionOnDeviceAutoMLImageLabelerOptions.Builder(localModel)
            .setConfidenceThreshold(0.5f)  // Sets the appropriate confidence threshold
            .build()
        // loads the labeler
        val labeler = FirebaseVision.getInstance().getOnDeviceAutoMLImageLabeler(options)

        val image: FirebaseVisionImage
        try {
            // loads the image as a firebase vision image
            image = FirebaseVisionImage.fromFilePath(applicationContext, uri)
            labeler.processImage(image)
                .addOnSuccessListener { labels ->
                    for (label in labels) {
                        val text = label.text
                        val confidence = label.confidence
                        Log.d("RECOGNITION", text + " " + confidence)
                    }
                    // do a thing with the confidence and label
                }
                .addOnFailureListener { e ->
                    println(e.message)
                }

        } catch (e: IOException) {
            e.printStackTrace()
        }
    }

}

The code in Swift is very similar

        let localModel = AutoMLLocalModel(manifestPath: manifestPath)
        let options = VisionOnDeviceAutoMLImageLabelerOptions(localModel: localModel)
        options.confidenceThreshold = 0.05  
        let labeler = Vision.vision().onDeviceAutoMLImageLabeler(options: options)

        var config = YPImagePickerConfiguration()
        config.showsCrop = .rectangle(ratio: 1.0)
        config.showsPhotoFilters = false
        let picker = YPImagePicker(configuration: config)

        picker.didFinishPicking { [unowned picker] items, _ in
            if let photo = items.singlePhoto {
                self.image = photo.image
                let image = VisionImage(image: photo.image)
                self._didFinishPicking!(self.self)

                labeler.process(image) { labels, error in
                    for label in labels {
                        self.result = GuideResult(label:label.text, confidence: label.confidence ?? 0)
                        self._setResult!(self.self)
                        break
                    }
                }
            }
            picker.dismiss(animated: true, completion: nil)
            self.dismiss(animated: true, completion: nil)
        }

This project also leveraged SwiftUI so I ended up building an Apple Watch and iPad version as well

I'm really proud with how it came out. I released both apps under the LaundrySnap name in the respective app stores back in May but I just launched it on Product Hunt Today!

Product Hunt Link
Apple App Store
Google Play Store

Continuously Delivering at Maxwell Health

James Hrisho — Thu, 07 Nov 2019 15:08:00 +0000

When I read Jez Humble’s book Continuous Delivery a few years ago, I felt like I was being watched. I thought Jez had been monitoring us and watching us struggle with deployments. In 2015, Maxwell Health released once a week. It was a coordinated affair and rife with what we discovered were anti-patterns. We would deploy off-hours, we coordinated deployments between teams, and each environment had its own build. If things went wrong, we would try to fix forward. It caused unnecessary stress, introduced risk for our customer experience and our software, and we had to make a change.

Today, Maxwell deploys code to production dozens of times a day without incident. It took coordination and dedication to change our process, and it was well worth the effort.

We ship small artifacts

In our old deployment pipeline, we tied our branching strategy to our deployments. Our strategy was a variation on git flow. When creating a new feature, an engineer would branch off of master and name your branch feature-abc. We then had a branch for each of our environments - QA, Staging, and Production. The engineer merged their branch to the QA branch for testing. Once the feature passed testing, the _feature-abc _branch could then be merged a second time, this time to master, and master would be merged into the staging and production branches. Each merge was its own build and this process encouraged mismatches between environments. What was on QA wasn’t guaranteed to be exactly what was deployed to staging or production.

After experiencing confusion, coordination issues, and too many mis-merged features, we adopted a version of Github Flow. An engineer creates a branch off of master as they had before. Once approved, the engineer testing that code merges the branch to master. The build pipeline then builds an artifact, in our case a docker image, and deploys it to our QA environment. Each commit once on master needs to be deployable. This keeps the pipeline clear and able to be deployed to production without incident. Each commit being deployable also means that authors must have strong automated tests to ensure confidence in their change. If it fails any testing on the QA environment or blocks the pipeline for whatever reason, we revert it.

If exploratory testing is successful in QA, the team can then ship that artifact to other environments by creating a Git tag. Our build tool knows, based on the name of the tag, to ship the artifact from that Git hash to the target environment. We even simplified it by building a small make command for it.

Our tooling also enables quick rollback. To rollback, you would call the same command for a SHA you want to go back to. Usually the previous deployment.

Our old branching strategy encouraged long-lived branches. There would be painful merge conflicts, different merge order on QA vs production, and each environment would behave slightly different. To accommodate the Github flow/artifact approach, each change needed to be small, releasable, and well tested. With smaller releases, both improvements and bug fixes are easier to reason about and verify. Our code review process is less painful, and we have fewer incidents in production.

SLOs for Builds, Deployments, and Rollbacks

We decided it is important to focus on rollbacks as part of our strategy. In the past, we would fix forward instead of rollback. Fixing forward, while well-intentioned, encouraged us to make matters worse. We would try to make changes while under duress. In the worst cases, we caused bugs in production while trying to fix other issues.

Instead, we want rollbacks to be the first tool in our engineers’ toolboxes. Our guideline is for our teams to make rollbacks possible less than a minute. As a result, the quickness of a rollback makes it the default instead of the last resort. Since our rollback strategy is identical to deployment (using a make command to send a specific Docker image to a specific environment), our deployment time also improved.

We also encourage builds to be less than 10 minutes. Our branching strategy encourages reverts and merging to master more often. Each of these events trigger builds, so long builds are an interrupt for teams. We must keep build times as short as possible. We spend money both on our team's time and our build server's time while we build. So time spent building should be minimal.

Identical Environments

To enable confidence in our deployments, we had to be certain of our environments. If one environment is quite different, we open ourselves up for bugs. The right way to do this is to invest in the tools that make your infrastructure repeatable and auditable. For us, that meant investing in our infrastructure-as-code strategy. At Maxwell, we leverage Docker, Terraform, Kubernetes, and other tools to make sure that our environments are as similar as possible.

Separating Release From Deployment

Finally, Maxwell encourages our teams to separate the concepts of deployment and release. We even made sure our vocabulary on the topic was well defined. Deployments are the act of moving code to an environment, while releases are the process of making functionality available to a stakeholder. Although we deploy several times a day, our releases might be less frequent, and are controlled by our product owners. Releases usually consist of more than one small change. Releases might include coordination of several teams work. For example, one team might be building the reporting functionality that is tied to new data we’re collecting. We would want to release that to our customers together but not have to coordinate the deployment of their code. Releases also include training for internal stakeholders, or a communication to our customers.

We enable this separation of releases and deployments by the use of feature toggles. Feature toggles allow us to hide functionality to all or a portion of users. We use an open-source library but there are many options for how to manage toggling your code. Our deployments have become much safer as putting work behind a feature toggle means we can deploy incremental parts of a feature rather than waiting for the entire feature to be complete and pushing a large body of code to production at once. Toggles also ensure that our functionality works in production without disrupting our customers. When our product team feels a feature is ready they can then turn on a feature toggle when they are ready.

Metrics For Success

These are a small number of the many changes our organization has made to become more efficient. We're constantly improving our processes, including in how we monitor, alert, and test. We are quick to respond to our customers. But more importantly, we're more confident in our work each day.

We started with metrics that we felt would make our teams successful. We knew we wanted to optimize for more frequent deployments and fewer incidents related to them. We wanted to minimize the number of times we fixed forward, as we recognized that as a failed release. We wanted the frequency that we ship code to be separated from the frequency we released to customers. Before starting the journey to continuous delivery, consider the objectives you are trying to achieve. In order to determine success, create metrics that measure a team's progress over time. These goals should have a reasonable end state in mind. For example, a build can always be faster, but what length of time is a success for your team? Chances are, the answer will be unique to the tools you use and the domain of your business.

While the journey can be challenging, don't give up. Each improvement builds upon itself. We've found that every six months we're unrecognizable to the organization we were six months before that. We're a more mature, yet more nimble version of ourselves.