Forem: Comiscience

The Mistake We've Made with Our Open-Source Product: Premature Optimization

Comiscience — Sat, 22 Feb 2025 06:40:56 +0000

We started FeatBit (an open-source feature flag platform) almost 4 years ago, and it has been open-sourced for 2 and a half years. Premature optimization is one of the biggest mistakes we've made.

We don’t regret the optimizations we implemented, as some customers appreciate them. However, our growth path could have been easier if we had started simpler instead of focusing on premature optimization.

This is our original blog - Keep open-source product simple and scalable. To make it easier to read, I’ve copied my content here.

Introduction

We are FeatBit, an open-source feature flag platform. This article discusses the mistakes we made and the lessons we learned while building and maintaining an open-source product.

Premature optimization is the root of all evil

FeatBit was designed as a self-hosting-friendly service, and it has remained that way until today. On the other hand, FeatBit wanted to provide customers with a performance guarantee. So, we put in a lot of effort at the very beginning of our open-source journey:

We offer two versions to suit different needs:
- Standard: Supports high concurrency and connections, but does not include experimentation analysis.
- Professional: Based on Standard, with added capabilities to handle large data volumes for experimentation analysis.
Our platform uses WebSocket connections for real-time feature flag updates, and we’ve made efforts to ensure:
- High performance.
- Scalability for high concurrency.
- High availability for continuous uptime.
We provide several ready-to-use open-source deployment solutions, such as:
- Docker Compose.
- Kubernetes.
- Terraform.

These efforts seemed like the right things to do at the time. However, after receiving feedback from customers, we realized that we had made several mistakes:

Maintenance Complexity.
Premature Optimization.

Maintenance Complexity

To ensure high performance, scalability, and availability, we introduced Redis, MongoDB, Kafka, and ClickHouse as databases. However, the big challenges we faced are:

MongoDB is well-known, but it's not widely used by most of our target customers.
Many of our target customers lack experience with Redis.
Kafka and ClickHouse are even less relevant; many B2B customers don’t need them because they don’t deal with large volumes of data.

It doesn’t make sense to require customers to learn and manage technologies that FeatBit doesn’t directly provide, as this can be a dealbreaker when deciding whether to adopt our product. Especially when the maintainers are Ops teams, not the developers and product managers who will actually use FeatBit.

Premature Optimization

We might not have built such a complex system at the beginning of our product development.

REST APIs are sufficient for many use cases. We could have supported WebSocket after receiving enough feedback.
We didn't need to focus on high performance for individual calculation resources. Making it scalable would have been enough.
A multiple-region high availability solution wasn't necessary at the start. Edge agents might have been simple enough.
We didn't need to add too many features (such as A/B experimentation) initially. Focusing on feature flags might have been sufficient.
We didn't need to design a microservice architecture at the beginning. A monolithic architecture might have been enough.
And so on.

By considering these points, we could have started with a much simpler architecture, allowing us to focus more on marketing and enabling users to easily deploy and use FeatBit in their production environments.

Pivot to Simplicity

We recognized our mistakes, but time has passed. Fortunately, the complex system we built allowed us to gain highly valued paid customers. However, to acquire more customers, we need to simplify our product so it can be used by a broader audience.

Making decisions is always difficult, because supporting a simpler version means we won't have time to develop new features. So, we conducted further investigation before making a decision:

We identified which databases are most commonly used by our target customers.
We spoke with potential customers to understand their actual needs.

Based on this, we decided to:

Provide a standalone version that uses only PostgreSQL as the database, removing Redis, MongoDB, Kafka, and ClickHouse.
Support both MongoDB and PostgreSQL as primary databases, catering to different customer needs.
Improve the FeatBit Agent to simplify many scenarios:
- Customers who don't want to self-host the entire FeatBit system can host an agent to keep data in their own environment.
- Customers who don't want to self-host a multi-region FeatBit system can deploy agents in different regions to maintain high availability and scalability.

Conclusion

Premature optimization is detrimental to acquiring early customers. However, if you've already built a complex system and received enough customer feedback, it's never too late to simplify your solution.

In the future, we may reduce the number of microservices and simplify our system. This will not only improve deployment and maintenance but also make it more developer-friendly.

Launchdarkly Open Source Alternatives

Comiscience — Mon, 18 Dec 2023 09:56:39 +0000

Here are 3 articles about Launchdarkly Alternatives:

Best Free and Open Source Feature Flag Tools

What Are the Best Open Source Feature Flags Tools Available in 2023?

What Are the Viable Alternatives to LaunchDarkly for Feature Flagging

Flagsmith vs. Unleash: Which Open Source Feature Flag Tool Wins?

Comiscience — Wed, 15 Nov 2023 08:58:16 +0000

Introduction

Unleash and Flagsmith are two popular platforms for feature management. They allow developers to deploy, test, and optimize features without affecting the end users. In this blog post, we will compare them in terms of tech stack, features, integrations, pricing, support community, and open source license.

Tech stack: Unleash is built with Node.js and PostgreSQL, while Flagsmith is built with Python and PostgreSQL. Both platforms use Redis for caching and message queueing. Unleash and Flagsmith have both a web interface written in React.

Features: Both platforms offer similar features, such as feature flags, feature toggles, feature experiments, multivariate testing, segmentation, targeting, and feedback. However, Unleash has some unique features, such as gradual rollout, kill switch, activation strategies, custom strategies, and plugins. Flagsmith has some unique features, such as identity management, user traits, remote config, environments, and organizations.

Integrations: Both platforms support various integrations with third-party tools and services. Unleash integrates with Slack, Datadog, Prometheus, Sentry, and more. Flagsmith integrates with Segment, Amplitude, Mixpanel, Optimizely, and more.

Pricing: Both platforms offer free and paid plans. Unleash has a free plan for up to 10 users and 1000 feature flags. Its paid plans start from $49 per month for up to 25 users and 2500 feature flags. Flagsmith has a free plan for up to 50 users and unlimited feature flags. Its paid plans start from $99 per month for up to 500 users and unlimited feature flags.

Support community: Both platforms have active support communities on GitHub, Slack, Discord, Stack Overflow, and other channels. Unleash has over 9000 stars on GitHub and over 1900 members on Slack. Flagsmith has over 3000 stars on GitHub and over 500 members on Discord.

Open source license: Both platforms are open source and have permissive licenses. Unleash is licensed under the Apache License 2.0, while Flagsmith is licensed under the BSD 3-Clause License.

If you're not the fun of Javascript and Python, you can try other feature flag tools which is built with Golang, C# and .NET

FeatBit

FeatBit provides an enterprise-level feature flag service. It's completely open source and licensed under the MIT license. It's basically built with .NET and MongoDB (support Azure CosmosDB, PostgreSQL with FerretDB), and uses Redis for caching and message queuing. FeatBit has a web interface written in Angular.

FeatBit also offers a professional version that adds Kafka for message queuing and ClickHouse for data warehousing. This provides a robust solution for companies that have high traffic (like more than millions MAU) and need to do data analysis.

FeatBit provides a more flexible way to do traffic splitting for ab test, such as mutually exclusive and orthogonal layering features to solve overlapping problems. FeatBit currently also provides a basic frequentist analysis tool for ab test.

FeatBit supports various integrations with third party tools and services. Like Flagsmith and Unleash, FeatBit also offers OpenFeature SDKs to avoid vendor lock-in.

FeatBit offers more free open source features than Flagsmith and Unleash. It now offers a self-hosting solution with unlimited seats for enterprise-level customers. FeatBit has active support communities on GitHub and Slack.

Go Feature Flag Tools

GoFeatureFlag is a lightweight and fast feature flag service for Go applications. It's fully open source and licensed under the Apache License 2.0. It's built with Go and uses a pluggable storage system that supports various backends, such as Consul, Redis, YAML file, etc. GoFeatureFlag has a simple and intuitive API that makes it easy to use in any Go project. GoFeatureFlag is a more performant and lightweight solution than LaunchDarkly and ConfigCat. It does not require any external dependencies or services, and can run entirely within your own infrastructure. GoFeatureFlag has active development and maintenance on GitHub.

Flipt is a self-hosted feature flag solution that helps you deliver faster and safer software. It's completely open source and licensed under the Apache 2.0 license. It's built with Go and uses SQLite, PostgreSQL, or MySQL as the backend database. Flipt has a web interface written in React. Flipt also has a cloud version that offers more features and benefits for teams that want to scale up their feature flag management. This includes advanced analytics, team collaboration, and security features.

Others

If you're interested in learning more feature flag tools, you can check out the following resources:

Conclusion

Which open source feature flag tool will win in 2023? Unleash, Flagsmith, FeatBit (and others) can all be the winners, it depends on which tech stack, pricing and features you need. So, you can choose the best one for your project.

Original article: https://www.featbit.co/blogs/flagsmith-vs-unleash

How to implement feature flags

Comiscience — Thu, 09 Nov 2023 03:00:04 +0000

Original Blog: https://www.featbit.co/blogs/how-to-implement-feature-flags

Feature flags, also known as feature toggles, have become an essential tool for modern software development. They enable developers to control the release of new features and updates in a flexible and controlled manner. In this blog post, we'll explore what feature flags are, their classical and modern use cases, and how to implement them effectively in front-end, mobile, and server-side applications. We'll also delve into the concept of overlapping design A/B testing and how feature flags play a crucial role in building robust experimentation.

Classical Use Cases: Derisk Your Deployments

Testing in Production: Feature flags enable you to test new features in a production environment without exposing them to all users. This reduces the risk associated with untested code.
Canary Release: Gradually roll out new features to a subset of users before making them available to everyone, ensuring a smoother deployment.
Allow/Block List: Control feature access for specific users or groups, giving you fine-grained permission management.
Beta Testing: Invite a select group of users to try out new features, gathering feedback before a full release.

Modern Use Cases: SRE and Growth

Observability: Gain insights into feature usage, performance, and user behavior through feature flag data and analytics. This allows the system to automatically trigger a feature rollback operation at the first sign of trouble. For more information, you can visit FeatBit's DataDog integration documentation or blog article 《Use of Feature Flags and Observability Tools for Gradual Database Migration》
Experimentation: Implement feature flags to A/B test new ideas, gather data, and make data-driven decisions about feature effectiveness. Feature flags give you a more flexible solution to meet the traffic splitting requirments in a complex experimentation design, such as Orthogonal Layering, Mutually Exclusive, etc. You can combine a feature flag tool with your data analysis tool to create a better experimentation system rather than buying an expensive third-party AB testing service.
Software Monetization: Use feature flags to control access to premium features, subscriptions, or other monetization strategies. This allows you to adapt your pricing model and offerings based on user behavior and feedback. You can read following article to get a realworld example, 《Sentinel EMS Workflow with Feature Flag Management Platforms》
And so on.

Standardizing with OpenFeature

The trend in feature flag implementation is to standardize with solutions like OpenFeature that provide SDKs and templates to streamline the process. Standardization is beneficial for 2 reasons:

Avoid Vendor Lock-In: Many third-party feature flag tools now support the OpenFeature SDK standard, reducing the risk of vendor lock-in and giving you more flexibility.
Essential Interfaces: OpenFeature provides essential interfaces to help you build a robust feature flag SDK. These interfaces help ensure you follow best practices and maintain consistency.

Implementing Feature Flags Across the Stack

A feature flag can be implemented across all your software applications: Web and Mobile APP, Back-end Services, Algorithms, and so on. Here's an example of how to use a feature flag across an entire project. Suppose you have a new method to tell users when to buy Google stock. The new version changes the user interface, the backend data source, and the calculation algorithm, so you need to keep all pages consistent for the same user.

You can create one or more feature flags to control the new version release. Here we will create a single boolean feature flag to control the release of a new feature. First, we will create a feature flag called "The Art of Buying Stocks". The figure below shows what a feature flag looks like in FeatBit's user interface.

Then let's see how we implement this feature flag in our front-end and back-end applications.

Suppose you have a UI that suggests when to buy Microsoft stock. You have two components, each presenting a different UI view for a different version of "The art of buying stocks". You use a feature flag to control which component is displayed or not. Here's an example in VUE code (if you're using the feature flag React, you'll just need to modify the code with React style):

To make the user see the same UI in both Web APP and Mobile APP, you should also implement such kind of work in your Mobile APP.

Suppose the new method (m2) needs to read additional data different from m1. You now have several ways to implement it. Here's my favorite ways (one or another):

Add feature flag that wraps and controls the different data retrieval functions in its parent function. But this adds some extra work to clear the function flag in both front-end and back-end code.
Add a new API for the new method (m2), so we don't need to add a feature flag in the backend anymore. But:
- Adding a new API may cause extra work for the API gateway.
- You have to delete the whole API when you finally choose the new method.

Here's an example in asp.net core. If the feature flag returns "m2", the program will retrieve the data source using the "RetriveDataFromFeatBiDB()" function.

After you implement the code in your application, the rest things you need to do is just control and release the feature in the UI. As shown in the figure shown below, you can set a custom rule to release new methods only to users who live in California. To minimize the potential risk, you can reduce the blast radius to just 10% of users. (This is also called progressive rollout)

Building Robust Overlapping Design A/B Testing

Overlapping design A/B testing involves running multiple experiments simultaneously, where users can be part of multiple experiments simultaneously. A robust feature flags service should meet the following requirements:

Able to create Orthogonal layers to stacking tests in an independent, non-overlapping manner. Each variation remains isolated, simplifying analysis and minimizing conflicts and unintended consequences.
Should be able to avoid conflicts, it's essential to make parameters mutually exclusive. Feature flags are instrumental in achieving this, allowing precise control over which users experience specific variations or changes.
Able to divide your user base into distinct groups based on relevant attributes or behavior. This allows you to run experiments on specific user segments, providing insights into how changes affect different groups.
And so on.

Many feature flag tools have tried to provide features to support this. FeatBit is a good open source solution to achieve this goal. Figure below shows how FeatBit empowers the AB test design with orthogonal layers and mutually exclusive split feature flags traffic in a single layer.

Feature Flag Open Source Tools

I wrote before another article about the tools, read article 《Best Free and Open Source Feature Flag Tools》.

Conclusion

Feature flags are a critical component of modern software development, allowing you to derisk deployments, run experiments, and maintain a flexible release strategy. By standardizing with solutions like OpenFeature and implementing feature flags consistently across your stack, you can achieve a robust and adaptable approach to software development and testing. Embrace the power of feature flags to build better products and make data-driven decisions.

Deploy Feature Flags Management Service on Azure using Terraform

Comiscience — Sat, 07 Oct 2023 06:48:00 +0000

FeatBit is a fast, scalable, and popular open-source feature flags service that enables teams to test their applications in production, roll out features incrementally, and instantly rollback if an issue arises.

This guide describes how to deploy FeatBit on Azure using Terraform. In lieu of the blog, you can visit our GitHub repository directly to start your installation journey. This GitHub repository contains terraform code for deploying FeatBit on Azure.

As shown in the figure above, FeatBit's services are deployed as Azure Container Apps (ACA) in Azure. Such as FeatBit's UI portal, FeatBit's API server, FeatBit's evaluation server, FeatBit's DA server. Evaluation service and API service communicate with DA service inside ACA.

Note: ACA is actually a managed Kubernetes cluster. You can find more information about ACA in Azure Container Apps official document.

All services are located in an Azure VNet, we use private endpoint and private DNS zone to secure the access to Azure Cache for Redis and Azure CosmosDB for MongoDB. You can find more information about private endpoint and private DNS zone in Azure official document.

FeatBit's UI portal, API server and Evaluation server are exposed to the public internet through an Azure Load Balancer and Azure IP addresses. You can find more information about Azure Load Balancer in Azure official document.

Download Terraform Code

Download the Terraform code source from the GitHub repository

Getting Started with Terraform Azure Provider

If you're not familiar with Terraform Azure Provider, you can follow the steps in the official Azure Provider tutorial.

Deploy FeatBit to your Azure

Run terraform init to download the required providers

terraform init

Run terraform plan to see what will be deployed

terraform plan

Run terraform apply to deploy FeatBit to your Azure

terraform apply

Before applying the Terraform deployment, you can modify variables defined in variables.tf files to customize your deployment. For example, you can

Change the name of the resource group by changing the value of the resource_group_name variable in the variables.tf file in the terraform directory.
Change the name of the resources location by changing the value of location variable in the variables.tf file in the terraform directory.
Change the redis configuration by changing the value of redis variable in the variables.tf file in the terraform directory.

To change cpu, memory, number of replicas of each container app, you currently have to edit directly in the main.tf file in the terraform/aca directory. We will add these variables in the future to make installation easier.

Support

The current Terraform code is actually only for the FeatBit Standard version. Click here to see the difference between Standard and Pro version

For any questions, you can contact us by creating an issue, joining our Slack channel, or emailing us at support@featbit.co.

Deploying the Feature Flags Service to Azure Kubernetes Service (AKS) using Helm, Exposed via Azure Load Balancer

Comiscience — Thu, 10 Aug 2023 11:39:01 +0000

FeatBit is an open-source feature flags service that enables teams to test their applications in production, roll out features incrementally, and instantly rollback if an issue arises.

This guide outlines how to deploy FeatBit to Azure Kubernetes Service (AKS) using Helm charts, with services exposed via Azure Load Balancer.

Prerequisites

An active Azure subscription.
An AKS cluster.
Azure CLI installed.
kubectl installed.

Creating Public Static IPs for FeatBit Services

FeatBit utilizes three public services:

UI Portal: This is the interface for team members to manage feature flags.
API Server: The backend which the portal communicates with to fetch and manage feature flags.
Evaluation Server: The endpoint SDKs communicate with to retrieve feature flag variations or rules.

Each service requires its own public IP. Here's how you can set them up:

# Public IP for the UI Portal
az network public-ip create --resource-group {resource group containing your AKS's vnet} --name featbit-ui-ip --sku Standard --allocation-method static

# Public IP for the API Service
az network public-ip create --resource-group {resource group containing your AKS's vnet} --name featbit-api-ip --sku Standard --allocation-method static

# Public IP for the Evaluation Service
az network public-ip create --resource-group {resource group containing your AKS's vnet} --name featbit-eval-ip --sku Standard --allocation-method static

Retrieve the IPs using:

az network public-ip show --resource-group {resource group containing your AKS's vnet} --name featbit-ui-ip --query ipAddress --output tsv

az network public-ip show --resource-group {resource group containing your AKS's vnet} --name featbit-api-ip --query ipAddress --output tsv

az network public-ip show --resource-group {resource group containing your AKS's vnet} --name featbit-eval-ip --query ipAddress --output tsv

Granting Delegated Permissions to AKS Cluster Identity

Before deploying services with a load balancer, ensure the AKS cluster identity has the necessary permissions to the node resource group.

CLIENT_ID=$(az aks show --name {your AKS name} --resource-group {resource group name where your aks located in} --query identity.principalId -o tsv)

RG_SCOPE=$(az group show --name {resource group containing your public IPs} --query id -o tsv)

az role assignment create --assignee ${CLIENT_ID} --role "Network Contributor" --scope ${RG_SCOPE}

Deploying with Helm and Custom Values

Add the FeatBit Helm repository:

helm repo add featbit https://featbit.github.io/featbit-charts/

Clone and navigate to the Helm chart repository:

git clone https://github.com/featbit/featbit-charts

cd ./featbit-charts/charts/featbit

In the featbit-charts/charts/featbit/examples directory, locate the AKS example file, expose-services-via-azurelb.yaml. Replace placeholders ({}) with the appropriate values:

apiExternalUrl, the URL the UI portal utilizes to retrieve feature flags.
evaluationServerExternalUrl, the URL the SDK accesses to obtain variations or rules for feature flags.
staticIP of ui.service, api.service and els.service, the public IPs you created in previous step.
service.beta.kubernetes.io/azure-load-balancer-resource-group, the name of the resource group where your public IPs are situated.

apiExternalUrl: "http://{API Service Public IP Address, ex. 4.194.69.254}"
evaluationServerExternalUrl: "http://{Evaluation Service Public IP Address, ex. 4.193.158.12}"
autoDiscovery: true

ui:
  service:
    type: LoadBalancer
    port: 80
    annotations: 
      service.beta.kubernetes.io/azure-load-balancer-resource-group: {Resource Group where your Public IP located in, ex. myNetworkResourceGroup}
    staticIP: {UI Portal Public IP Address, ex. 4.194.13.155}

api:
  service:
    type: LoadBalancer
    port: 80
    annotations: 
      service.beta.kubernetes.io/azure-load-balancer-resource-group: {Resource Group where your Public IP located in, ex. myNetworkResourceGroup}
    staticIP: {API Service Public IP Address, ex. 4.194.69.254}

els:
  service:
    type: LoadBalancer
    port: 80
    annotations: 
      service.beta.kubernetes.io/azure-load-balancer-resource-group: {Resource Group where your Public IP located in, ex. myNetworkResourceGroup}
    staticIP: {Evaluation Service Public IP Address, ex. 4.193.158.12}

Preview the Helm installation:

helm install featbit featbit/featbit -f ./examples/expose-services-via-azurelb.yaml --dry-run

If all looks well, install the Helm chart:

helm install featbit featbit/featbit -f ./examples/expose-services-via-azurelb.yaml

# or to upgrade
helm upgrade --install featbit . -f ./examples/expose-services-via-azurelb.yaml

NOTE:

Ensure you run the command from the directory containing expose-services-via-azurelb.yaml.
Specify a namespace with --namespace option during installation if needed.
Adjust replica counts or disable autoscaling in the YAML file as desired in expose-services-via-azurelb.yaml file:
- .Values.{service name， ex. api | ui | els | das }.replicaCount, the default value is 1
- .Values.api.autoscaling.enabled, set the value to false

Verification

Check that the services and pods are running:

kubectl get svc

kubectl get po

This should show output similar to the provided image:

Finally, access the UI Portal via the public IP you established earlier:

References

Deploying ASP.NET Core applications to Kubernetes

FeatBit's helm chart repository

Use a public standard load balancer in Azure Kubernetes Service (AKS)

Original articles: https://www.featbit.co/blogs/deploy-featbit-to-aks-alb

Use of Feature Flags and Observability Tools for Gradual and Safety Database Migration

Comiscience — Tue, 25 Jul 2023 16:07:49 +0000

Scenario Description

Many companies encounter situations requiring database upgrades or migrations, particularly when transitioning from self-hosted database services to cloud-based services, from on-premises data centers to cloud data centers, and from old databases to new ones. Throughout the migration process, it's vital to ensure stability, prevent data loss, and avoid service downtime. One of the most common migration methods is the "Dual Write Database Migration."

Solution

The process of dual write migration is as follows, illustrated in the diagram below:

Initial Stage - The application interacts (reads/writes) with only the old database.
On the existing code that reads/writes to the old database, we need to add code to read/write to the new database. For example, when inserting a record into a table, the data must be inserted into both the old and new databases simultaneously. Typically, these two insert operations are executed in parallel to maintain the original service call handling time as much as possible.
When a database write request comes in, it is written to the old database and a small percentage of the traffic is also written to the new database.
The percentage of traffic written to the new database is gradually increased until it reaches 100%. If any issues arise during this process, we can promptly rollback without impacting the production environment.
After the write migration is complete, we begin to gradually increase the amount of data read from the new database, initially allowing 10% of the traffic to perform read operations on the new database. We measure performance and compare results during this process. If any issues are encountered, we can quickly rollback the new database read traffic without impacting the production environment.
Once the new database has been handling 100% of the read/write operations without any issues for a certain period, we can decommission the old database and its related code services.

In the real operation process, it's not only the traffic of the old and new databases that needs to be gradually opened up. The read and write codes for the new database also need to be updated in the production environment services in a step-by-step manner to ensure a stable, iterative migration.

Practical Methods and Tools

Throughout the process, besides the design of the system architecture itself, two specific tools play crucial roles:

Feature Flags service, responsible for flexible, real-time, stable traffic ramp-up and rollbacks. In this article, we used FeatBit as a feature flag service.
Observability service, providing comprehensive monitoring of service anomalies and timely alarms throughout the process. In this article, we used GuanceCloud as an observability tool.

Using FeatBit to implement real-time database migration request traffic control

The pseudo-code shown below demonstrates how to split the database read operations for a particular service:

In line 6 of the code, by calling the _fbService.BoolVariation("read-sport-olddb") method, we obtain the traffic control return value. If it's true, we add the Query function of reading from the old database to the parallel task execution queue.
In line 9 of the code, by calling the _fbService.BoolVariation("read-sport-newdb") method, we obtain the traffic control return value. If it's true, we add the Query function of reading from the new database to the parallel task execution queue.
In line 19 of the code, we use the FeatBit Feature Flags SDK to simultaneously run two database read operations, compare and verify the results, return the correct value according to the execution situation, and send relevant exception data to observability tool.

public async Task<List<Sport>> GetSportsByCityAsync(int cityId, int pageIndex, int pageSize)
{
    var tasks = new List<Task<List<Sport>>>();

    // When the feature flag for reading the old database for Sport-related services 
    // returns true, we add a read task to the execution task queue.
    if (_fbService.BoolVariation("read-sport-olddb"))
    {
        tasks.Add(GetSportsByCityQueryAsync(_oldDbContext, cityId, pageIndex, pageSize));
    }

    // When the feature flag for reading the new database for Sport-related services 
    // returns true, we also add a read task to the execution task queue.
    if (_fbService.BoolVariation("read-sport-newdb"))
    {
        tasks.Add(GetSportsByCityQueryAsync(_newDbContext, cityId, pageIndex, pageSize));
    }

    // Two read operations are executed simultaneously (to avoid increasing request 
    // time due to new data read) and the results are compared and returned.
    // If the results are inconsistent, we return the old database read result
    // and record the discrepancy.
    return await _fbService.RunAndCompareDbTasksAsync(
                    tasks,
                    timeoutDelayForNewDB: 3000, // Set the maximum wait time for the new database to avoid a bad user experience
                    (timeoutInfo) => { }, // When the new database call times out, send a message to observability tool
                    (unMatchInfo) => { }, // When the returned results are inconsistent, send a message to observability tool
                    (exception) => { } // When an exception occurs, send a message to observability tool
                );
}

After integrating the similar code above into your project, you can use a Feature Flags's UI tool to scale database migration's dual-write and dual-read traffic. For example, we can initially adjust the traffic scaling of the feature flag read-sport-from-newdb to 5%. If no anomalies are observed in the observability tool over a certain period, we can increase the traffic scaling percentage to 10% (as shown in the figure below).

Use an Observability tool to Monitor the Entire Migration Process and Timely Discover Potential Problems

During the entire data migration process, automated and timely error detection followed by rollback is extremely important. It can most effectively help us avoid numerous issues, such as:

If the operation of the new database brings significant system resource consumption, we need to know immediately and rollback through the Feature Flags system.
When the number of write operations or read operations exceeding the estimated threshold due to timeout, we can quickly locate the problem, rollback and repair swiftly. This improves migration speed.
When a write operation or read operation results in information errors (such as inconsistency in results, excessive request time, program exceptions, etc.), we can locate the specific error information based on the observability system, thereby accelerating the debugging speed.
And so on.

We can use an observability system that integrates Application Performance Monitoring (APM), Real User Monitoring (RUM), and Metric capabilities to monitor abnormal data and system behavior. Examples include DataDog, Guance.one (as used in this article), and so on.

Quickly Locate Migration Errors through "Traces" and "Error Tracking"

On the APM/Traces page of the observability tool, we find that some red items (i.e., Errors) have occurred during the migration process. Through the Resource column, we can easily see that errors occurred in our read operations on the new database, as shown below:

By clicking on the corresponding Error, we can quickly view its associated call chain flame graph. As the flame graph interpretation shows:

As indicated by the Span at the location marked circle 1 in the diagram below, a Timeout error occurred here during database migration, i.e., the reading time from the new database exceeded our acceptable request response time threshold.
The position marked circle 2 points out that the error occurred when the Feature flag read-sport-newdb was true. This means we can quickly locate the Feature Flags that we may need to rollback or turn off to avoid migration risks.
According to Span at the location marked circle 3, we can quickly locate the server-side API service where the timeout occurred. The captured parameters and headers of the API can assist us in better debugging and resolving the issue later.

Rollback Read Operations in Real Time Using Feature Flags to Avoid Timeout State

Based on the Traces and Error tracking information above, we've quickly located the abnormal database read operation. Now all we need to do is return to the FeatBit UI, find the feature flag read-sport-newdb we discovered above, and rollback its percentage to last state. As shown in the diagram below, we decrease the percentage of traffic allocation of true from 10% back to 5% where no read anomalies were previously observed.

After the rollback, as shown in the code below, the return value of _fbService.BoolVariation("read-sport-newdb") will only have a 5% chance of being true.


// When the feature flag for reading from the new database related to the 
// Sport service returns true, add the read task to the execution queue.
if (_fbService.BoolVariation("read-sport-newdb"))
{
    tasks.Add(GetSportsByCityQueryAsync(_newDbContext, cityId, pageIndex, pageSize));
}

Conclusion and Next Steps

This article introduces a basic method to reduce database migration risk by using an observability tool and a Feature Flag tool to implement a dual-write, dual-read operation mode. In real world operation, we may have a large number of businesses to deal with, and human intervention can lead to slow responses for various reasons. In subsequent articles, we will introduce more content, such as:

Using the Metrics service from an "Observability tool" and the Trigger service from FeatBit to implement automatic real-time rollback, disaster avoidance, and alarm solutions during migration.
Using the Metrics service from "Observability tool" and the Scheduler service from FeatBit to implement automated scaling and rollback solutions.
And more.

Original article: https://www.featbit.co/blogs/feature-flags-observation-tool-gradual-db-migration

How to Choose the Right Open Source Feature Flags Management Tool

Comiscience — Tue, 23 May 2023 07:34:23 +0000

Selecting an open-source feature flags management tool is a crucial decision that can significantly impact your software development process. Feature flags (or feature toggles) offer a powerful way to switch functionality on and off in a product, enabling safer deployments and continuous delivery.

In this post, we'll examine five popular open-source feature flags management tools: FeatBit (C#), Unleash (Typescript), Flagsmith (Python), FeatureProbe (Rust), and Go Feature Flag (Go). We will discuss the technology they are built on, the unique advantages each brings, and the service level agreements (SLAs) they offer to help you make an informed decision.

FeatBit (C#)

FeatBit is developed primarily using C#. C# is a robust, statically typed language developed by Microsoft as part of the .NET platform. C# is known for its excellent performance and has seen increasing popularity in recent years. It offers strong support for concurrency and scales very well, making it suitable for applications that demand high performance. Furthermore, it's now fully open-source, which means it has a growing community of contributors continually enhancing the language and its tooling.

In addition to its open-source offering, FeatBit provides a premium service that guarantees a certain level of service availability and performance, as outlined in their service level agreement.

Choosing FeatBit can be an excellent choice if your team is familiar with the .NET platform and values the reliability and performance that C# brings.

Unleash (Typescript)

Unleash is built primarily with TypeScript. TypeScript is a statically typed superset of JavaScript. It brings the safety of static typing to JavaScript, making the code more predictable and easier to debug. TypeScript is popular for web development due to its strong tooling support and compatibility with JavaScript.

Unleash also offers a premium service with a guaranteed SLA, providing peace of mind for businesses that require reliable and consistent performance.

If your team is comfortable with JavaScript and wants to leverage the advantages of static typing, Unleash could be a good fit.

Flagsmith (Python)

Flagsmith uses Python as its primary development language. Python's greatest strength lies in its simplicity and readability, making the codebase approachable for developers of varying skill levels. It also boasts a rich ecosystem of libraries and frameworks, which can speed up development.

Like FeatBit and Unleash, Flagsmith also provides a premium service with a guaranteed SLA, ensuring reliable performance for critical applications.

If you value simplicity and readability, and your application does not have extremely high-performance requirements, Flagsmith could be the right choice.

Go Feature Flag (Go)

Go Feature Flag uses Go as its primary development language. Go is known for its simplicity, strong support for concurrency, and fast execution. The language is statically typed and compiled, which aids in producing efficient and reliable software.

If your team values simplicity and fast execution, Go Feature Flag could be an excellent choice.

FeatureProbe (Rust)

FeatureProbe is developed primarily with Rust. Rust is known for its excellent performance and reliability, making it suitable for system-level programming. It offers memory safety without a garbage collector, and its syntax is designed to help developers write code that behaves predictably.

If your application requires a high level of performance and you're not intimidated by Rust's learning curve, FeatureProbe could be a great choice.

Conclusion

The choice of an open-source feature flags management tool depends on several factors, including your team's familiarity with the underlying technology, the specific needs of your project, and the trade-offs you're willing to make. Each tool has its strengths, and the best choice will depend on the specific needs and constraints of your project.

Many of these tools also offer premium services withguaranteed SLAs, providing an extra layer of assurance for businesses that require consistent and reliable performance.

Take the time to evaluate each tool, consider your requirements and resources, and make the choice that best fits your situation. Happy coding!

Original article: https://www.featbit.co/blogs/choose-os-feature-flag-tool-by-tech-stack

The best solution for a Feature Flagging server that is written in C#

Comiscience — Mon, 15 May 2023 13:42:00 +0000

Note: This article was originally posted on https://www.featbit.co/blogs/seo-keywords-best-feature-flagging-server-csharp

In the realm of software development, finding the right tools can sometimes feel like navigating through a labyrinth. One of our customers recently shared their journey of discovering FeatBit, illuminating how precise their search parameters were: they sought a feature flagging solution written in C#, open-sourced under the MIT License, and with active development. Their quest led them to us, and we couldn't be more thrilled to be part of their software development toolkit.

Discovering FeatBit

FeatBit, found at https://www.featbit.co, is a dedicated Feature Flagging server, meticulously crafted to cater to the needs of modern-day software developers. Our tool fits snugly into the .NET ecosystem, being primarily written in C#, one of the most widely used languages in enterprise applications today.

What Makes FeatBit Stand Out?

Our solution is not just another tool in the market. We’ve designed FeatBit to be intuitive, robust, and scalable, addressing the pain points developers often encounter in managing feature flags.

Open-source under the MIT License, FeatBit encourages community-driven enhancements. We strongly believe in the power of collective wisdom and have made our software available for tinkering, modification, and improvement by developers worldwide. We are not just a tool created for developers; we are shaped by developers.

Our Commitment to Active Development

We understand that the tech world is continuously evolving, and staying static is not an option. Our active development commitment ensures that we're always ready to adapt to new challenges, integrate feedback, and implement necessary changes. We're not just providing a solution; we're continuously refining it.

A Part of Your Software Development Toolkit

Feature flagging is a powerful strategy for software development and release. It empowers developers to safely deploy, perform A/B testing, gradually roll out features, and improve overall software quality. FeatBit is built with these objectives at its core, providing developers the control they need while maintaining simplicity of use.

The journey of our customer in discovering FeatBit sheds light on the nuanced requirements of today's developers. We are thrilled to have been the solution they were searching for, and we are excited to see more developers find their way to us.

Start Your FeatBit Journey Today

Intrigued about FeatBit? Discover more at https://www.featbit.co and see how we can be a part of your software development process.

Ready to dive in? Click the Try it Online button or the Check GitHub button on the left side of the page to start your journey with FeatBit today. There's no better time to join the FeatBit community and start revolutionizing your feature flag management!

.NET Feature Flag: Embracing Flexibility with FeatBit

Comiscience — Mon, 15 May 2023 11:48:45 +0000

Note: This article was originally posted on https://www.featbit.co/blogs/seo-keywords-dotnet-feature-flag

In the evolving landscape of software development, feature flags have become an essential tool. They allow developers to manage and control feature releases, perform A/B testing, and ensure safer deployments. For developers operating in the .NET ecosystem, having a feature flagging solution that seamlessly integrates with their workflow is critical. This is where FeatBit steps in.

FeatBit: Tailored for the .NET Ecosystem

FeatBit, available at https://www.featbit.co, is an open-source feature flag service designed with the .NET ecosystem in mind. Primarily written in C#, FeatBit offers a seamless solution for managing feature flags in .NET applications.

Why Choose FeatBit for .NET?

FeatBit stands out for several reasons. Firstly, it's tailored for the .NET environment. As a solution written in C#, it integrates effortlessly with .NET projects, making it an ideal choice for .NET developers.

Secondly, FeatBit is open-source under the MIT License. This encourages a collaborative development process, inviting developers not only to use FeatBit but also to contribute to its growth and evolution. This community-driven approach ensures that FeatBit remains relevant, robust, and responsive to the needs of .NET developers.

FeatBit's Commitment to Active Development

In the ever-changing world of technology, FeatBit remains adaptive. Our commitment to active development ensures that FeatBit is always ready to respond to new technological challenges, integrate valuable community feedback, and implement necessary enhancements.

Start Your Journey with FeatBit Today

If you're a .NET developer seeking a powerful, flexible feature flagging solution, FeatBit may just be the tool you need. Visit https://www.featbit.co to explore our offerings.

To jump right in, click the "Try it Online" or "Check GitHub" buttons on the left side of the page. This will allow you to dive into FeatBit, experience its functionality firsthand, and see how it can revolutionize your approach to feature flag management.

Join the FeatBit community today and take the first step towards a more manageable and innovative approach to feature flagging in your .NET projects.

Feature Flag C#: Enhance Your Development Process with FeatBit

Comiscience — Mon, 15 May 2023 11:46:43 +0000

Note: This article was originally posted on https://www.featbit.co/blogs/seo-keywords-feature-flag-csharp

The software development world has embraced feature flags as a vital tool for controlling feature rollouts, conducting A/B testing, and ensuring safer deployments. For developers who work with C#, a feature flagging solution that aligns seamlessly with their existing workflows is paramount. This is where FeatBit shines.

FeatBit: A C#-Centric Feature Flag Solution

Available at https://www.featbit.co, FeatBit is an open-source feature flag service specially designed with C# developers in mind. It's primarily written in C#, allowing for seamless integration with the .NET ecosystem.

Why FeatBit is the Ideal Choice for C# Developers

The strength of FeatBit lies in its ability to resonate with the community it serves. Being a solution written in C#, it provides an intuitive and harmonious experience for developers working with C#.

Moreover, FeatBit, as an MIT licensed open-source project, encourages community engagement and contribution. This collaborative model ensures that FeatBit remains a tool that is not only made for developers but also shaped by developers, continually adapting and evolving to meet their needs.

Our Pledge to Active Development

In the dynamic world of tech, FeatBit is not just keeping pace but also setting the pace. Our commitment to active development means we're always ready to adapt to new challenges, integrate valuable community feedback, and implement necessary enhancements.

Embark on Your FeatBit Journey Today

If you're a C# developer in search of an efficient feature flagging solution, FeatBit is worth your consideration. To learn more, visit https://www.featbit.co.

Ready to get started? Click on the "Try it Online" or "Check GitHub" buttons on the left side of the page. Explore the world of FeatBit, experience its capabilities first-hand, and see how it can revolutionize your feature flag management.

Join the FeatBit community today, and take your feature flagging strategy in C# to the next level.

Developer Tool 2.0 - LLM Powered Feature Flags & Experimentation Management Service

Comiscience — Tue, 04 Apr 2023 09:43:47 +0000

Original Article: https://www.featbit.co/blogs/LLM-Introducing-FeatBit-ChatGPT-Powered-FeatureFlags-Service

As ChatGPT (Large Language Model) has gained popularity, terms like Prompt Engineer, Prompt Ops, and LLM Ops have emerged. This got me thinking about whether this powerful model could be applied to practical programming scenarios. So, I decided to test its abilities in a couple of pain points with the Feature Flags on the FeatBit service. I found that both the GPT-3.5 API and GPT-4 Chat interfaces produced impressive results. However, despite fine-tuning the model to a certain extent, the performance of GPT-3 still fell short of my expectations. In this article, I'll describe the two scenarios I tested in more detail:

Utilize ChatGPT for eliminating dead feature flag code
Leverage prompts for developer tool onboarding

Reduce Tech Debt - Remove Dead feature flags

A straightforward way to use a feature flag in a codebase is with an if/else statement, like so:

public string UP(FbClient featureFlags)
{
    var user = FbUser.Builder("anonymous").Build();
    var ifC = featureFlags.BoolVariation("ui-c", user, defaultValue: false);

    string total = "0", num1 = "3", num2 = "12";
    if (ifC == true)
    {
        return total + num1 + num2;
    }
    return total;
}

Once a feature is fully implemented and delivered, or an experiment is completed and a winner is determined, one of the code paths will no longer be executed, necessitating the removal of the "dead" code. When frequently working with feature flags, you or your colleagues might inadvertently remove the incorrect code or overlook certain areas indirectly controlled by the feature flags. Although this may happen infrequently, it can still have an impact on your business and the entire development team.

Remove dead feature flags with chatGPT3.5 & 4

I attempted to utilize ChatGPT to eliminate dead feature flags and associated code. For the initial trial, I employed chatGPT 3.5's ChatCompletion method. I provided ChatGPT with clear instructions on what needed to be done and supplied it with "pre-training/fine-tuning" examples.

The image below presents a fine-tuning Prompt/Completion pair example. On the left side, there is code containing dead feature flags and related code. Conversely, on the right side, you can see the cleaned-up version of the code after the removal of dead feature flags.

I provided six additional examples demonstrating the use and removal of feature flags to ChatGPT-3.5-turbo.

Next, I tested ChatGPT by giving it a code snippet along with the same instructions to see if it could accurately remove dead feature flags and their associated dead code.

As seen in the image above, the results are truly fascinating. This simple text demonstrates ChatGPT's potential to address a major pain point that no other solution has resolved thus far.

However, upon testing it with a wider range of scenarios, it becomes apparent that more fine-tuning examples are needed for better performance. Unfortunately, as of now, fine-tuning is not available for ChatGPT-3.5 or ChatGPT-4. But fear not, it's only a matter of time!

As time progresses, ChatGPT will likely offer more fine-tuning options to users, and other open-source large language models will gain similar capabilities. This prospect fills me with great hope.

text-davinci-003 & ChatGPT 3

I attempted to fine-tune the chatGPT-3 model using the same example data, but unfortunately, it didn't yield any useful results. I also experimented with the text-davinci-003 model, which produced satisfactory outcomes, but at a higher cost.

Optimal Developer Tools Onboarding Experience

Leveraging ChatGPT's code completion and code insertion capabilities, developers can effortlessly initialize FeatBit in their existing projects with just a simple prompt. Even without fine-tuning, you can utilize chat-completion to have GPT-4 read the GitHub documentation and learn how to use FeatBit's SDK.

The image below illustrates the trial steps:

I input the FeatBit .NET SDK documentation into ChatGPT-4.
I copy the code source from an ASP.NET Core 7's Program.cs file.
instructed GPT-4 to initialize FeatBit's .NET SDK within the code source.
GPT-4 returned the modified code after initialization.

While GPT-4 performed well, it added some unnecessary code at the top of the file:

using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

Although not perfect, this fascinating trial showcases a new approach to developer tool onboarding. It highlights a novel interaction method between code and developers, as well as code and non-engineers.

Copilot currently lacks this capability, but it's plausible that it will eventually offer an interface for developer tools to fine-tune and create plugins for Copilot.

False Positive

At present, both GPT-3.5 turbo and GPT-4 have demonstrated their potential to outperform humans in certain tasks. What they currently lack in accuracy can be improved through pre-training and fine-tuning with more data and real-world use cases.

While there may be instances of false positives when addressing feature flag technical debt or initializing SDKs, the performance of these models will continue to improve as they are fed more practical use cases.

Conclusion

ChatGPT has shown immense potential in addressing practical programming pain points, such as removing dead feature flags and enhancing developer tool onboarding. While the current performance may not be perfect, it is important to remember that we are still in the early stages of exploring the capabilities of these language models. As more data and real-world use cases become available for pre-training and fine-tuning, we can expect to see significant improvements in the accuracy and usefulness of ChatGPT in a variety of programming scenarios. This not only promises to revolutionize the way developers interact with code, but also paves the way for more accessible and efficient solutions for non-engineers. The future of ChatGPT and similar models is bright, and we eagerly anticipate the advancements they will bring to the software development landscape.

For more information, please visit FeatBit's GitHub repository.

By the way, I'm not a Python developer, it took me 30 minutes to install and grasp the basics of Python. I successfully completed my ChatGPT Python CLI program using only Copilot and ChatGPT-4..