Forem: Dilip Kola

Implementing a Cost-Efficient Micro services Platform on Azure Kubernetes

Dilip Kola — Mon, 19 Jan 2026 04:39:04 +0000

Terraform, Autoscaling, Spot Capacity, and Workload Identity

This article focuses on implementation details. Architectural rationale and cost trade-offs are covered in Part 1.

Scope and Assumptions

This post assumes:

Familiarity with Kubernetes fundamentals
Comfort reading Terraform and Helm
Interest in operating systems, not just deploying them

The platform runs on Azure Kubernetes Service, provisioned with Terraform, and deployed using Helm.

1. AKS Baseline: Start Small, Scale on Demand

The most common AKS cost mistake is provisioning for peak load.

We instead:

Start with minimal baseline capacity
Enable the cluster autoscaler
Let demand drive node count

AKS Cluster with Autoscaling

resource "azurerm_kubernetes_cluster" "aks" {
  name                = var.cluster_name
  location            = var.location
  resource_group_name = var.resource_group_name
  dns_prefix          = var.cluster_name

  default_node_pool {
    name                 = "default"
    vm_size              = "Standard_D2s_v5"
    auto_scaling_enabled = true
    min_count            = 1
    max_count            = 10
  }

  identity {
    type = "SystemAssigned"
  }
}

Why this works

Idle cost remains low
Nodes are added only when pods are pending
Capacity matches actual demand, not estimates

2. Horizontal Pod Autoscaling with Predictable Behavior

Autoscaling defaults are aggressive and often unstable.

We explicitly tune scale behavior to reduce churn and latency spikes.

HPA with Stabilization

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-service
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-service
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300

Key outcomes

Prevents rapid scale-down during brief traffic dips
Improves tail latency
Reduces unnecessary pod restarts

3. Spot Node Pools for Fault-Tolerant Workloads

Spot capacity is one of the highest-leverage cost optimizations—when isolated properly.

Terraform: Spot Node Pool

resource "azurerm_kubernetes_cluster_node_pool" "spot" {
  name                  = "spot"
  kubernetes_cluster_id = azurerm_kubernetes_cluster.aks.id
  vm_size               = "Standard_D2as_v5"

  priority        = "Spot"
  eviction_policy = "Delete"
  spot_max_price  = -1

  auto_scaling_enabled = true
  min_count            = 0
  max_count            = 10

  node_taints = [
    "kubernetes.azure.com/scalesetpriority=spot:NoSchedule"
  ]
}

Scheduling Workers on Spot Nodes

tolerations:
- key: "kubernetes.azure.com/scalesetpriority"
  operator: "Equal"
  value: "spot"
  effect: "NoSchedule"

nodeSelector:
  kubernetes.azure.com/scalesetpriority: spot

Rules we followed

APIs never run on spot
Workers handle SIGTERM cleanly
All state lives outside the worker

Used this way, spot capacity delivered substantial savings without user-visible impact.

4. Managed PostgreSQL with Private Networking

Databases are not where cost experiments belong.

PostgreSQL runs as a managed service with:

Subnet delegation
Private DNS
No public access

Delegated Subnet for PostgreSQL

resource "azurerm_subnet" "postgres" {
  name                 = "postgres-subnet"
  virtual_network_name = azurerm_virtual_network.main.name
  resource_group_name  = var.resource_group_name
  address_prefixes     = ["10.0.2.0/24"]

  delegation {
    name = "postgres"
    service_delegation {
      name = "Microsoft.DBforPostgreSQL/flexibleServers"
    }
  }
}

Why this matters

Database is unreachable from the public internet
Access is restricted at the network layer
Operational risk is significantly reduced

5. Secure Deployments with Helm Defaults

Helm charts were written with secure-by-default assumptions.

Pod Security Context

securityContext:
  runAsUser: 1000
  runAsGroup: 1000
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true

This immediately:

Shrinks the attack surface
Prevents runtime mutation
Surfaces insecure images early

Health Probes That Matter

livenessProbe:
  httpGet:
    path: /health/db-cache
    port: 8080
  initialDelaySeconds: 60

We intentionally check dependencies, not just process health.

6. Workload Identity: No Secrets in Kubernetes

Storing cloud credentials in Kubernetes secrets is unnecessary.

We use Workload Identity for pod-to-Azure authentication.

Federated Identity Credential

resource "azurerm_federated_identity_credential" "api" {
  name      = "api-federated"
  parent_id = azurerm_user_assigned_identity.api.id
  issuer    = azurerm_kubernetes_cluster.aks.oidc_issuer_url
  subject   = "system:serviceaccount:production:api-sa"
  audience  = ["api://AzureADTokenExchange"]
}

Kubernetes Service Account

apiVersion: v1
kind: ServiceAccount
metadata:
  name: api-sa
  annotations:
    azure.workload.identity/client-id: "<client-id>"

Result

No long-lived secrets in manifests
Automatic token rotation
Azure RBAC enforced at runtime

7. Observability Without Ingestion-Based Pricing

Instead of managed log ingestion, we use:

Prometheus for metrics
Loki for logs
Object storage for retention

Loki Storage Configuration

storage_config:
  azure:
    container_name: logs
    account_name: ${ACCOUNT_NAME}
    access_tier: Cool

Why this works

Logs are queried infrequently
Storage is inexpensive
Ingestion costs dominate managed observability pricing

This preserved full visibility with minimal incremental cost.

8. Infrastructure Access Patterns (Secure and Practical)

Cluster Access

Azure AD–backed kubectl
Auditable
No shared credentials

In practice, cluster-admin access is restricted to a small bootstrap group; most teams use namespace-scoped roles.

Database Access (Occasional Admin Tasks)

kubectl run psql \
  --image=postgres:16 \
  --rm -it -- psql -h <private-host> -U admin

The database remains private; access is authenticated and auditable.

Observability Dashboards

Grafana is protected behind OAuth using Azure AD.
OAuth2-Proxy runs with multiple replicas and rotated cookie secrets to avoid becoming a single point of failure.

No VPNs. No bastion hosts. No additional managed services required.

Operational Lessons (The Real Ones)

Defaults Are Rarely Production-Safe

Autoscaling, probes, and security contexts need explicit tuning.

Spot Capacity Requires Discipline

It works extremely well—but only when isolated.

Identity Scales Better Than Secrets

It reduces operational load and security risk.

Kubernetes Is a Tool, Not a Destination

Use it where it adds leverage—not as a dumping ground.

Closing Thoughts

This implementation isn’t about clever tricks—it’s about intentional trade-offs.

By:

Scaling only when needed
Using spot capacity responsibly
Keeping critical state managed
Avoiding ingestion-based observability costs
Treating security as a default

we ended up with a system that is:

Predictable to operate
Cost-efficient at idle
Resilient under load
Easy to evolve

Architecture sets direction.
Implementation determines whether it survives contact with reality.

Building Production-Grade Microservices on Azure Kubernetes

Dilip Kola — Mon, 19 Jan 2026 03:44:17 +0000

Architecture and Cost Trade-offs

How we designed a scalable, reliable microservices platform on Azure Kubernetes Service while keeping infrastructure costs predictable and operational overhead low.

Introduction

When building cloud-native systems, teams often face a familiar tension:

reliability versus cost

Fully managed services offer strong SLAs and peace of mind, but their pricing compounds quickly as systems grow. Self-hosted alternatives reduce spend, yet introduce operational complexity that many teams underestimate.

In this article, we share how we designed a production-grade microservices platform on Azure Kubernetes Service (AKS) using a hybrid architecture—combining managed services where reliability is non-negotiable and self-hosting components where the risk-to-cost trade-off is acceptable.

This approach resulted in:

70–90% lower monthly infrastructure costs compared to fully managed stacks
Strong reliability guarantees for critical data
Full observability without per-GB ingestion fees
A portable, infrastructure-as-code-driven foundation

This post focuses on architecture and decision-making.
Implementation details (Terraform, Helm, autoscaling, identity) are covered in Part 2.

The Core Problem: Cost, Reliability, and Operational Load

Modern microservices platforms are more than application code. Even relatively small systems require:

Databases, caches, and message queues
Stateless application services (APIs, workers, frontend)
Metrics, logs, and dashboards
Secure networking and access controls

Each component introduces the same decision:

Should this be a managed service—or something we run ourselves?

The Cost Reality

Individually, managed services are reasonably priced. Together, they add up quickly.

Component	Managed Option	Typical Cost
PostgreSQL	Azure PostgreSQL	$65–200 / month
Redis	Azure Cache for Redis	$50–150 / month
RabbitMQ	Managed broker	$100–300 / month
Logging	Azure Monitor Logs	$2.50 per GB

For early-stage products or cost-sensitive deployments, this pricing model can become a constraint long before scale becomes a problem.

The Key Insight: Not All State Is Equal

The most important architectural decision we made was classifying state.

Two Categories of State

Irreplaceable state

Transactional data
User data
Financial records

This data cannot be regenerated. It requires backups, point-in-time recovery, and strong durability guarantees.

Regenerable state

Caches
Queues
Derived logs and metrics

This data can be rebuilt, replayed, or tolerated if lost temporarily.

Once this distinction is made, the managed vs self-hosted decision becomes much clearer.

Our Hybrid Architecture Strategy

Rather than going “all managed” or “all self-hosted,” we adopted a selective hybrid approach.

Managed Services

PostgreSQL (database of record) Chosen for durability, automated backups, and operational guarantees.

In-Cluster Services (Kubernetes)

Redis for caching
RabbitMQ for asynchronous workloads
Observability stack (metrics and logs)

These components are important, but failure is recoverable. Kubernetes handles restarts and rescheduling, making this trade-off reasonable.

This single decision accounted for the majority of cost savings—without increasing the blast radius of real failures.

Kubernetes-First, Not Kubernetes-Everything

Kubernetes serves as the orchestration layer, not a universal hosting solution.

Where Kubernetes Adds Value

Consistent deployment and scaling model
Horizontal scaling built-in
Strong ecosystem and tooling
Cloud-agnostic abstraction

Where It Does Not

Primary databases
Systems requiring strict consistency and durability guarantees

Using Kubernetes for compute and managed services for critical state strikes a pragmatic balance.

Scaling Without Paying for Idle Capacity

One of the most common cloud cost mistakes is provisioning for peak traffic.

Our approach:

Keep a small baseline footprint
Scale pods horizontally based on load
Allow nodes to scale only when required

At idle, the platform remains inexpensive. During traffic spikes, it scales automatically—and scales back down afterward.

This keeps costs proportional to actual usage rather than theoretical peak demand.

Using Spot Capacity—Deliberately

Not every workload requires guaranteed availability.

Background workers, batch processing, and asynchronous jobs can tolerate interruptions. These workloads run on spot capacity, trading availability guarantees for steep discounts.

What Runs on Spot

Workers
Batch jobs
Non-user-facing tasks

What Never Does

APIs
Databases
Stateful services

By isolating these workloads, we significantly reduced compute costs without affecting user experience.

Observability Without Ingestion-Based Pricing

Observability is not optional—but ingestion-based pricing makes it expensive at scale.

Instead of paying per-GB ingestion fees, we use:

Open-source tooling for metrics and logs
Object storage for long-term retention
Dashboards tailored to the system’s needs

Why This Works

Logs are queried infrequently
Storage is inexpensive
Ingestion costs dominate managed observability pricing

This approach preserves full visibility while keeping observability spend negligible relative to compute.

Security and Access: Identity-First by Design

Security decisions were guided by a few core principles:

Private by Default

Databases are reachable only within the virtual network
Internal services are never exposed publicly

Identity Over Secrets

Workloads authenticate using cloud identity
No long-lived credentials stored in Kubernetes

Least Privilege

Day-to-day access uses minimal permissions
Elevated access is required only during initial setup

This avoids VPN sprawl, bastion hosts, and unnecessary operational overhead.

Cost Snapshot (Production)

A representative production environment looked roughly like this:

Component	Monthly Cost
AKS compute (baseline)	$80–100
Managed PostgreSQL	$65
Storage & networking	$30
Persistent volumes	$10
Total	~$190

Comparable fully managed setups typically exceeded $500–$1,500 per month.

Lessons Learned

1. Don’t Optimize the Wrong Layer

Saving money on the database of record is rarely worth the risk.

2. Spot Capacity Is High-Leverage

Used correctly, it provides some of the largest cost reductions available.

3. Observability Is Mandatory

Skipping it to save money always costs more later.

4. Infrastructure as Code Pays Off Early

Teams that automate early spend less time firefighting later.

5. Kubernetes Is an Enabler, Not the Goal

Use it where it adds leverage—not as a default for everything.

Conclusion

Production-grade microservices don’t require premium managed services at every layer.

By:

Classifying state correctly
Using managed services selectively
Scaling dynamically
Leveraging open-source observability

it’s possible to build systems that are reliable, cost-efficient, secure, and portable—even with small teams.

Part 2 dives into the implementation details: Terraform modules, AKS autoscaling, spot node pools, Workload Identity, and Kubernetes deployment patterns.

Empowering Real-Time Data Pipelines: Leveraging Apache Kafka and Rudderstack

Dilip Kola — Tue, 19 Mar 2024 04:45:20 +0000

In today’s fast-paced digital landscape, effective data management and analysis are essential for businesses aiming to stay ahead of the curve. Fortunately, modern tools like Apache Kafka and RudderStack have revolutionized the way we handle and derive insights from large datasets. In this blog post, we’ll explore our experience implementing the Kafka Sink Connector to facilitate seamless event data transfer to RudderStack, unlocking significant advantages for real-time analytics.

Apache Kafka: The Backbone of Real-Time Data Streaming

Apache Kafka has emerged as a distributed event streaming platform renowned for its ability to handle real-time data streams efficiently. With a robust architecture capable of processing millions of events per second, Kafka is the go-to solution for systems requiring real-time operations and monitoring. Its versatility spans various data types, including page views, clicks, likes, searches, transactions, and more. Using topics, Kafka securely stores each record in a fault-tolerant manner, facilitating streamlined data storage and stream processing.

RudderStack: Unifying Customer Data for Actionable Insights

RudderStack stands out as a customer data platform (CDP) designed to streamline the collection, processing, and routing of customer event data to preferred analytics tools. This open-source, warehouse-first platform empowers businesses to establish their customer data lake directly on their data warehouse, enabling secure and real-time data utilization on their terms.

Bridging Systems Seamlessly: The Role of Kafka Connectors

Kafka Connect serves as a vital conduit for scalable and reliable data streaming between Apache Kafka and various other systems. Categorized into source and sink connectors, Kafka Connect simplifies the process of defining connectors to seamlessly move large datasets into and out of Kafka.

Unlocking Seamless Integration: The Kafka Sink Connector

At the forefront of this integration is the Kafka Sink Connector, a Kafka Connect plugin that acts as a bridge between Apache Kafka and RudderStack. This connector is powered by the RudderStack Java SDK, enabling the seamless export of data directly from Kafka topics to the RudderStack platform. Operating within the Kafka ecosystem, the connector efficiently consumes data from Kafka topics and transmits messages to RudderStack using the Java SDK.

Explore the Source Code: RudderStack Kafka Sink Connector

During the configuration process, you’ll need to specify crucial details such as RudderStack’s Data Plane URL, Write Key, and Kafka topic names, among other necessary information. Once these configurations are set, the connector seamlessly captures messages from Kafka topics and securely delivers them to RudderStack. From there, RudderStack efficiently processes the incoming data, enabling seamless streaming to over 200 integrations supported by the RudderStack platform. This streamlined process ensures that your data flows seamlessly from Kafka to RudderStack and beyond, unlocking a wide array of integration possibilities for your analytics and business intelligence needs.

Conclusion

In today’s data-driven landscape, efficient data management and transfer are fundamental to success. Implementing the Kafka Sink Connector streamlines the process of sending events from Kafka to RudderStack, facilitating more effective data analysis and informed decision-making. Empower your business to capitalize on the power of real-time data and gain a competitive edge. Stay tuned for further insights and explorations as we delve deeper into the realm of data management and analytics solutions.

Note: Originally posted here.

Template Language for JSON data

Dilip Kola — Wed, 08 Mar 2023 18:26:20 +0000

Motivation

We are an integration platform and support 200+ integrations. We implemented these integrations using native Javascript code to transform incoming events to destination payload, so in summary, it is JSON data manipulation. Maintaining all these integrations is challenging, so we explored jsonata to write less code to transform JSON data. While this library is excellent, we still need to meet our performance needs. For example, JSONata parses the given template, creates an Abstract Syntax Tree (AST), and interprets the AST for the given input. Since we need to traverse the AST every time, it is slow, so we wanted to build a template engine that generates Javascript code so there will be less overhead during the runtime.

Overview

A library to process JSON data using a custom syntax based on javascript and jspath. We thank the jspath authors for their excellent work, as our library is an extension of the original library. We also want to thank IBM team for their work on jsonata, as we have taken several ideas from the library. You can also consider our library as an alternative to jsonata.

This library generates a javascript function code from the template and then uses the function to evaluate the JSON data. It outputs the javascript code in the following stages:

Lexing (Tokenization)
Parsing (AST Creation)
Translation (Code generation)

  flowchart TD;
      A[Code] --> B[Convert code to tokens];
      B --> C[Parse tokens to create Expressions];
      C --> D[Combine expressions to create statements];
      D --> E[Combine statements to create AST];
      E --> F[Translate AST to JS code]

Engine class abstracts the above steps and provides a convenient way to use the json templates to evaluate the inputs.

Getting started

npm install rudder-json-template-engine

const engine = new JsonTemplateEngine(`'Hello ' + .name`);
engine.evaluate({name: 'World'});

Features

Template is a set of statements and result the last statement is the output of the template.

Variables

const a = 1
let b = a + 2
a + b

Input and Bindings

Input refers to the JSON document we would like to process using a template. Bindings refer to additional data or functions we would provide to process the data efficiently.

Example:

Template: "Hello " + (.name ?? $.defaultName)
Evaluation: engine.evaluate({name: 'World'}, {defaultName: 'World'});
{name: 'World'} is input.
- ^.name refers to "name" property of the input. We can also use .name to refer the same. ^ always refers to the root of the input and . refers to current context. Refer the example for more clarity.
{defaultName: 'World'} is bindings.
- $.defaultName refers to "defaultName" property of the bindings. Refer the example for more clarity.

Arrays

let arr = [1, 2, 3, 4]
let a = arr[1, 2] // [2, 3]
let b = arr[0:2] // [1, 2]
let c = arr[-2:] // [3, 4]

Refer the example for more clarity.

Objects

let key = "some key"
// { "a": 1, "b": 2, "c": 3, "some key": 4 }
let obj = {a: 1, b: 2, c: 3, [key]: 4 } 
let a = obj["a"] // 1
let b = obj.a // 1
let c = obj{["a", "b"]} // { "a": 1, "b": 2}
let d = obj{~["a", "b"]} // { "c": 3, "some key": 4}

Refer the example for more clarity.

Functions

Normal functions

let fn = function(arg1, arg2){
  arg1 + arg2
}

The result of the last statement of function will be returned as result of the function. We can also use rest params (...args).

Lambda/Short functions

let fn = array.map(lambda 2 * ?0);

This function gets converted to:

let fn = array.map(function(args) {
  2 * args[0]
})

Lambda functions are short to express the intention and it is convenient sometimes.

Async functions

let fn = async function(arg1, arg2){
  const result = await doSomethingAsync(arg1, arg2)
  doSomethingSync(result)
}

Note: When we want to use async functions then we need to create template engine using JsonTemplateEngine.create. If you create a template this way then it will be created as an async function so we can await anywhere in the template.

let result = await doSomething(.a, .b)

Paths

Paths are used to access properties in input, bindings and variables.

Simple Paths

Simple paths support limited path features and get translated as direct property access statements in the generate javascript code.
a.b.c gets translated to a?.b?.c so they are very fast compared to Rich paths. Simple paths are ideal when we know the object structure.

Supported features:

Simple Selectors
Single Index Filters Refer the example for more clarity. #### Rich Paths Rich paths gets converted complex code to support different variations in the data.

If we use rich path for expression: a.b.c then it automatically following variations.

[{"a": { "b": [{"c": 2}]}}]
{"a": { "b": [{"c": 2}]}}
{"a": [{ "b": [{"c": 2}]}]}
Automatically handles selection from nested objects and arrays.

Simple selectors

let x = a.b.c;
let y = a."some key".c

Refer the example for more clarity.

Wildcard selectors

a.*.c // selects c from any direct property of a

Refer the example for more clarity.

Descendent selectors

// selects c from any child property of a
// a.b.c, a.b1.b2.c or a.b1.b2.b3.c
let x = a..c; 
let y = a.."some key";

Refer the example for more clarity.

Single Index or Property Filters

let x = a[0].c; 
let y = a[-1].c;  // selects last element from array
let z = a["some key"].c

Refer the example for more clarity.

Multi Indexes or Properties Filters

let x = a[0, 2, 5].c; 
let y = a["some key1", "some key2"].c;

Refer the example for more clarity.

Range filters

let x = a[2:5].c; 
let y = a[:-2].c; 
let z = a[2:].c;

Object Property Filters

let x = obj{["a", "b"]};  // selects a and b
let y = obj{~["a", "b"]}; // selects all properties except a and b

Refer the example for more clarity.

Conditional or Object Filters

let x = obj{.a > 1};

Refer the example for more clarity.

Block expressions

let x = obj.({
  a: .a + 1,
  b: .b + 2
});
let x = obj.([.a+1, .b+2]);

Refer the example for more clarity.

Context Variables

.orders@order#idx.products.({
    name: .name,
    price: .price,
    orderNum: idx,
    orderId: order.id
})

Use context variables: @order and #idx, we can combine properties of orders and products together. Refer the example for more clarity.

Path Options

We can mention defaultPathType while creating engine instance.

// For using simple path as default path type
// a.b.c will be treated as simple path
JsonTemplateEngine.create(`a.b.c`, {defaultPathType: PathType.SIMPLE});
// For using rich path as default path type
// a.b.c will be treated as rich path
JsonTemplateEngine.create(`a.b.c`, {defaultPathType: PathType.RICH});

We can override the default path option using tags.

// Use ~s to treat a.b.c as simple path
~s a.b.c 
// Use ~r to treat a.b.c as rich path
~r a.b.c

Note: Rich paths are slower compare to the simple paths.

Compile time expressions

Compile time expressions are evaluated during compilation phase using compileTimeBindings option.

// {{$.a.b.c}} gets translated to 1 and
// final translated code will be "let a = 1;"
JsonTemplateEngine.create(`let a = {{$.a.b.c}};`, {
  compileTimeBindings: {
    a: {
      b: {
        c: 1
      }
    }
  }
});

We can use compile time expressions to generate a template and then recompile it as expression. Refer the example.

Conclusion

We have created a simple language to process JSON data efficiently and made it open-source. It was fun building a language, and I hope you have learned something new today.

NodeJS Workflow Engine for JSON data transformations

Dilip Kola — Wed, 08 Mar 2023 18:16:30 +0000

Overview

In transformer service, we are doing data transformation from customer events to destination events, and we want organize the transformation process into logically separated steps for better understanding and maintainability so we have created a workflow engine to address the following requirements.

Validation
- Event validation
- Destination config validation
Source to destination data mapping
Enriching data with destination API calls
Handling different types of events
- Track, Identify, Page, etc.
- Custom categories:
  - Product Viewed
  - Product Purchased
Multiplexing
Batching
Response building.

Currently, most steps are implemented using Javascript code, which provides the most flexibility. Still, it is getting difficult to maintain, understand, debug, test, and develop in a standardized way. To bring standardization, we are building a workflow engine that is config driven to provide improved readability, testability, reusability, and speed of development.

Since we want to express the transformation of the logic using easy to read and write template based language. We support following template languages:

JSONata
JsonTemplate.
Easily extendable to more template languages. Workflow Example using Jsonata:

templateType: Jsonata
steps:
  - name: unsupported
    condition: $not(op in ["+", "-", "*", "/"])
    template: |
      $doThrow("unsupported operation")
  - name: add
    description: Do addition
    condition: op = "+"
    template: |
      ( a + b )
  - name: subtract
    description: Do subtraction
    condition: op = "-"
    template: |
      ( a - b )
  - name: multiply
    description: Do multiplication
    condition: op = "*"
    template: |
      ( a * b )
  - name: divide
    description: Do division
    condition: op = "/"
    template: |
      ( 
        $assert( b != 0, "division by zero is not allowed");
        a / b 
      )

Getting started

npm install rudder-workflow-engine

const workflowEngine = WorkflowEngineFactory.createFromFilePath("workflow.yaml", options);
workflowEngine.execute(input);

Features

Config Driven

Users should be able to express the destination transformation logic as a series of steps in a YAML file as a workflow. Steps can be written as template base languages.

Bindings

Supports importing of external functions and data using bindings.

Workflow Bindings

Bindings are similar to imports, which allow importing of externally defined functions and data to the workflow.
Types
- Type 1: Import a specific field from a file.
```
  name: EventType 
  path: ./config
```
- EvenType is defined in ./config file then it will be imported as $EventType
  - Type 2: Import everything from a file as something.
```
  name: MappingData 
  path: ./mapping
  exportAll: true
```
- Everything from ./mapping file will be imported to the variable MappingData
- If something1 and something2 are defined in ./mapping then we need to access them using $MappingData.something1 and $MappingData.something2
  - Type 3: Import everything as it is defined in the file
```
  path: ./utils
```
- Everything from ./utils file will be imported with the same names.
- If something1 and something2 are defined in ./utils then we need to access them using $something1 and $something2
Full example:

  bindings:
    - name: EventType 
      path: ./config
    - name: MappingData 
      path: ./mapping
      exportAll: true
    - path: ./utils

These are user-specified bindings while defining the workflow. #### Platform bindings
The platform provides these bindings, which can be used directly without defining them in the bindings block.
We will soon release detailed documentation on these bindings. #### Execution bindings
$outputs: Provides access to the outputs of the previous steps executed before the current step.

  steps:
    - name: step1
      template: |
        {
          "a": something
        }
    - name: step2
      template: |
      {
        "b": $doSomething($outputs.step1.a)
      }

Step2 uses the output of step1.
Workflow Engine automatically bindings step outputs to the $outputs variable.
- $setContext: It is a function to store any data in $context and use it later. $outputs are read-only variables for users to refer to the previous step outputs, so we can’t use them to pass a modifiable result. So if we want to update the same variable in multiple steps, then $setContext should be used.

Example:

steps:
  - name: setAForCase1
    condition: $isCase1(message)
    template: |
      $setContext("a", something1)
  - name: setAForCase2
    condition: $isCase2(message)
    template: |
      $setContext("a", something2)
  - name: updateA
    template: |
      $setContext("a", $updateA($context.a))
  - name: useA
    template: |
      $doSomething($context.a)

In this example, we update the variable repeatedly in several steps, so it is impossible to use $outputs.
A practical scenario for this feature is: that we want to populate an object differently based on some conditions and later use it.

$context: To access variables set using $setContext function. Please refer to the above example for clarity.

Steps
Steps are the main execution blocks of the workflow.
Steps must contain a name to track outputs.
Steps can contain an optional description field to describe the details.
The step can contain an optional condition field to execute only if the condition is satisfied.
The step can contain an optional inputTemplate field to customize the input, which will be passed while executing the step.
There are two different types of steps supported:
- SimpleStep
- WorkflowStep

Conditions

A step in a workflow can mention an optional condition so that it gets executed only when the condition is satisfied.

Condition is also a Jsonata code.

steps:
  - name: commonValidation
    template: |
      ( common validations for events )
  - name: ValidateInputOfTrackEvent
    condition: message.type = EventType.Track
    template: |
      ( some validations specific to track events)

InputTemplate**

By default, all steps receive the same input as the workflow input, but when we want to modify the input before executing the step, we can use this feature.

  steps:
    - name: step1
      (some logic ...)
    - name: step2
      inputTemplate: |
        (customize the input)
    - name: step3
      (some logic ...)

In the above example: step1 and step3 will be executed with the workflow’s input, but the step2 receives custom input as defined in the inputTemplate ### ContextTemplate
- By default, all steps receive the current context, but we can use this feature when we want to modify the context before executing the step. This is useful when using external workflows, workflow steps, or template paths.

  steps:
    - name: step1
      (some logic to prepareContext)
    - name: step2
      contextTemplate: |
        (customize the context for step2)
      (some logic ...)
    - name: step3
      (some logic ...)

In the above example: step 3 will execute with the context prepared in step 1, but step 2 receives custom context as defined in the contextTemplate. ### LoopOverInput
- We can use this feature when the input is an array, and we want to execute the step logic for each element independently.
- This is mainly used for batch processing, and we report failed and successful executions without failing the step if an error occurs while processing a particular step.

  name: executeForEach
  loopOverInput: true
  template: |
    ( do something )

If the input for the step is [e1, e2, e3], then the step will be executed for all elements independently, and imagine that it failed for e1 and succeeded for e2 and e3 then, the overall step output will be the following:
```
[
  {
    "error": someErrorForE1
  },
  {
    "output": someOutputForE2
  },
  {
    "output": someOutputForE3
  }
]
```

OnComplete

When the step is completed, the next step will be executed, but if we want to exit the workflow with the output of a particular step, then we can use this.
This feature should be used only in a conditional step.
Example 1: Avoid reprocessing, so return without modifying the input message.

    steps:
      - name: checkIfProcessed
        condition: message.processed = true
        template: |
          message
        **onComplete: return**
      - name: processMessage
        template: |
          (...)

In the above example, we don’t want to reprocess messages, so we need to return them immediately if they are already processed.
- Example 2: Return early after processing the input message.

  steps:
    - name: step1
      template: |
        (doSomeProcessing)
    - name: **step2**
      condition: someCondition
      template: |
        (doSomeProcessing)
      onComplete: return
    - name: step3
      template: |
        (doSomeProcessing)

In this example, we want to return early after successfully processing the message in step2 since this step is conditional, and if the condition is not satisfied, then step3 will be executed. ### OnError
- By default, if any step fails, then the entire workflow fails but if the step uses OnError: continue setting, then the workflow will ignore the error and continue with execution.

  steps:
    - name: step1
      template: |
        (doSomeProcessing)
    - name: **step2**
      template: |
        (doSomeProcessing)
      onError: continue
    - name: step3
      template: |
        (doSomeProcessing)

In the above example, if any error occurs in either step1 or step3, the workflow will exit immediately, but when step2 fails, the workflow ignores the error and continues to execute step3.

Steps

Simple Step

Simple step is the basic unit of execution in the workflow.
A simple step can be a function that is defined in the Bindings.

  bindings:
    - name: **processTrackEvent
      path: ./transform # actual file name is transform.js**
  steps:
    - name: processTrackEvent
      functionName: processTrackEvent

We can omit .js extension while defining the bindings.
processTrackEvent must have the following definition.

  (input: any, bindings: Record<string, any>) => { 
    error?: any, 
    output?: any 
  }

A simple step can be a JSONata template.

  name: processTrackEvent
  template: |
    (JSONata template to process track events)**

The template also can be imported from the file path.
```
name: processTrackEvent
templatePath: ./trackTemplate.yaml
```
- We can use an external workflow in a simple step.

  steps:
    - name: prepareContext
      template: $setContext("batchMode", true)
    - name: transform
      **externalWorkflow:
        path: ./pinterest_tag_single_workflow.yaml**
      loopOverInput: true

We are reusing the single event workflow in the batch events transformation workflow.
The external workflow will be executed as a black box, so we can only access the final output of the workflow but not the individual outputs of steps.
The external workflow is executed with step input and context of the original workflow.
The context of the parent workflow is passed to the child workflow (externalWorkflow) but not vice-versa. This is helpful to customize the child workflow execution based on where it is used.

The external workflow doesn’t have access to the parent workflow outputs.

Workflow Step

Series of simple steps.

steps:
  - name: category
    template: |
      (compute category)
  - name: ecom
    condition: $outputs.category = "ecom"
    steps:
      - name: validateInput
        description: Common validation for all ECom pages
        template: |
          (assert everything is fine)
      - name: page
        template: |
          (compute page using $outputs.category)
      - name: processSearchPage
        condition: $outputs.ecom.page = "search"
        template: |
          (search page template)
      - name: processDetailPage
        condition: $outputs.ecom.page = "detail"
        template: |
          (detail page template)
      - name: processCartPage
        condition: $outputs.ecom.page = "cart"
        template: |
          (cart page template)

We can access outputs of previous steps normally like $outputs.category.
To access outputs of the child steps of the workflowStep, we need to use $outputs.workflowStepName.childStepName, for example: $outputs.ecom.page.
- The outputs of the child steps are not available outside the workflow step.
- The last successfully executed child step’s output will become the output of the workflow step, and we can only access that outside the workflow step as $output.workflowStepName, for example, $output.ecom.
Currently, we don’t support nested workflow steps.
Workflow Step can be imported from a file.

steps:
  - name: processECommerace
    workflowStepPath: ./ecomWorkflow.yaml

Supports additional Bindings

bindings:
  - name: commonBinding
    path: ./bindings
steps:
  - name: processECommerace
    bindings:
      - name: stepBinding
        path: ./workflow_step_bindings
    steps:
      - name: validateInput
        description: Common validation for all ECom pages
        template: |
          (assert with $commonBinding)
      - name: page
        template: |
          (compute page using $workflowBinding)
      - name: processSearchPage
        condition: $outputs.ecom.page = "search"
        template: |
          (search page template)

In the above example: processECommerace step is the workflow step and importing additional bindings. Both workflow’s (commonBinding) and step’s (stepBinding) bindings are available to the workflow step.

Testing

Test Scenarios using Jest

npm run jest:scenarios -- --scenarios=<comma separate scenarios>
Example: npm run jest:scenarios -- --scenarios=basic_workflow,to_array

Manually Test Scenario

npm run test:scenario -- -s <scenario_folder> -i <test_case_index_from_data.json>
Example: npm run test:scenario -- -s outputs -i 1
Note: It just run the test case and produces results but won't run any validations of the results.

Conclusion

We have built this workflow engine to solve our problems, and we believe that the framework is general enough for other use cases as well, so please feel free to give it a shot.