Forem: Tim Kang

Migrating from Terraform/Helm to Database-Driven Kubernetes Without Deleting Anything

Tim Kang — Thu, 04 Dec 2025 00:46:12 +0000

So you've seen how database-driven automation works, maybe even tried the Killercoda demo, and now you're thinking: "cool, but I already have a hundred tenants running. I can't just blow everything away and start over."

Yeah. That's a real problem.

This post is about how to migrate existing resources managed by Terraform, Helm, Kustomize, or whatever else you're using, over to Lynq without deleting anything. The goal is zero downtime, no data loss, and a safe rollback path if things go sideways.

Before we start

I'm assuming you've already:

Set up your MySQL database with the node table
Created a LynqHub pointing to that database
Written a basic LynqForm and tested it with a fresh node
Verified that new nodes provision correctly

If you haven't done that yet, check out the quickstart first. This guide is specifically about taking over existing resources, not creating new ones.

The strategy

Here's the high-level approach:

Configure your LynqForm to generate the exact same resource names as your existing ones
Use conservative policies as safety nets
Test with one node first to verify conflict detection works
Remove ownership from your old tool (Terraform state, Helm release, etc.)
Let Lynq take over ownership
Repeat for remaining nodes
Gradually relax safety policies once stable

Let's walk through each step.

Step 1: match your existing resource names

This is crucial. Your LynqForm templates need to produce the exact same resource names that already exist in the cluster.

Say your existing deployment is named acme-corp-app in namespace acme-corp. Your template needs to render to exactly that:

deployments:
  - id: app
    nameTemplate: "{{ .uid }}-app"
    namespaceTemplate: "{{ .uid }}"
    spec:
      apiVersion: apps/v1
      kind: Deployment
      # ...

If your database has uid = "acme-corp", this renders to acme-corp-app in namespace acme-corp. Perfect match.

Double check your naming conventions. If Terraform was using underscores and Lynq templates use dashes, you'll create duplicate resources instead of taking over the existing ones.

Step 2: configure safety-first policies

For migration, start with the most conservative settings:

deployments:
  - id: app
    nameTemplate: "{{ .uid }}-app"
    conflictPolicy: Stuck      # don't force takeover yet
    deletionPolicy: Retain     # never auto-delete, even if something goes wrong
    creationPolicy: WhenNeeded
    spec:
      # ...

Why these settings?

conflictPolicy: Stuck is your early warning system. When Lynq tries to apply a resource that's already owned by something else (like Terraform or Helm), it will stop and emit an event instead of forcing through. This lets you verify that Lynq is actually targeting the right resources.

deletionPolicy: Retain is your safety net. Even if you accidentally delete a LynqNode or mess up the hub config, the actual kubernetes resources stay in the cluster. You can always recover.

Apply this to every resource in your template. Yes, all of them.

Step 3: test with a single node first

Don't migrate everything at once. Pick one tenant/node and try it first.

Insert or activate the row in your database:

UPDATE nodes SET is_active = true WHERE node_id = 'acme-corp';

Now watch what happens:

kubectl get lynqnodes -w

You should see a LynqNode created. Check its events:

kubectl describe lynqnode acme-corp-web-app

If your existing resources match the template output, you'll see ResourceConflict events. This is actually what we want at this stage. It confirms Lynq is finding and targeting the right resources.

The event message tells you exactly what's conflicting:

Resource conflict detected for acme-corp/acme-corp-app (Kind: Deployment, Policy: Stuck). 
Another controller or user may be managing this resource. Consider using ConflictPolicy=Force 
to take ownership or resolve the conflict manually. 
Error: Apply failed with 1 conflict: conflict with "helm" using apps/v1: .spec.replicas

This tells you:

Which resource: acme-corp/acme-corp-app
Current owner: helm
Conflicting field: .spec.replicas

Step 4: review and fix template mismatches

Sometimes the conflict message reveals that your LynqForm doesn't quite match the existing resource. Maybe your template sets replicas: 2 but the existing deployment has replicas: 5 because of HPA.

You have a few options:

Option A: Update your template to match

If the difference is intentional (like HPA managing replicas), don't set that field in your template, or use ignoreFields to skip it during reconciliation.

Option B: Accept the difference

If you want Lynq to enforce a new value, that's fine. Just be aware the resource will change when you force takeover.

Option C: Update the database

If the value should come from your database, add it to extraValueMappings and use it in the template.

The key is understanding what will change before you flip the switch.

Step 5: remove ownership from your old tool

Now comes the actual migration. You need to tell your old tool to stop managing these resources without deleting them.

For Terraform:

# Remove from state without destroying
terraform state rm kubernetes_deployment.acme_corp_app
terraform state rm kubernetes_service.acme_corp_svc
# repeat for all resources

For Helm:

# Uninstall release but keep resources
helm uninstall acme-corp-release --keep-history

# Or if you want to be extra safe, just delete the release secret
kubectl delete secret -l owner=helm,name=acme-corp-release

For Kustomize/kubectl:

If you were just applying manifests directly, there's no state to remove. The resources exist, they're just not tracked by anything. Lynq can take over directly.

For ArgoCD/Flux:

Remove the Application or Kustomization CR, or exclude those resources from sync. The actual resources stay in cluster.

After this step, the resources exist in kubernetes but nothing is actively managing them. They're orphaned, which is exactly what we want temporarily.

Step 6: let lynq take ownership

Now update your LynqForm to force takeover:

deployments:
  - id: app
    nameTemplate: "{{ .uid }}-app"
    conflictPolicy: Force      # changed from Stuck
    deletionPolicy: Retain     # keep this for now
    spec:
      # ...

Apply the updated LynqForm:

kubectl apply -f lynqform.yaml

The next reconciliation will use Server-Side Apply with force=true to take ownership. Check the LynqNode status:

kubectl get lynqnode acme-corp-web-app -o yaml

You should see:

status:
  desiredResources: 3
  readyResources: 3
  failedResources: 0
  appliedResources:
    - "Deployment/acme-corp/acme-corp-app@app"
    - "Service/acme-corp/acme-corp-svc@svc"

No more conflicts. Lynq now owns these resources.

Verify by checking the resource's managedFields:

kubectl get deployment acme-corp-app -n acme-corp -o yaml | grep -A5 managedFields

You should see manager: lynq in there.

Step 7: repeat for remaining nodes

Once you've confirmed the first node works, migrate the rest. You can do this gradually:

-- Migrate in batches
UPDATE nodes SET is_active = true WHERE region = 'us-east-1';
-- Wait, verify
UPDATE nodes SET is_active = true WHERE region = 'us-west-2';
-- And so on

Monitor the LynqHub status to track progress:

kubectl get lynqhub my-hub -o yaml

status:
  referencingTemplates: 1
  desired: 150
  ready: 148
  failed: 2

Investigate any failures before continuing.

Step 8: clean up old tool artifacts

Once everything is migrated and stable:

Delete old Terraform state files or workspaces
Remove Helm release history if you used --keep-history
Archive old Kustomize overlays
Update CI/CD pipelines to stop running old provisioning

step 9: consider relaxing policies

After running stable for a while, you might want to adjust policies:

deployments:
  - id: app
    conflictPolicy: Stuck      # back to Stuck for safety
    deletionPolicy: Delete     # now safe to auto-cleanup

Switching deletionPolicy back to Delete means when a node is deactivated, resources get cleaned up automatically. Only do this once you trust the system.

Keep conflictPolicy: Stuck for ongoing safety. Force was just for the migration.

troubleshooting common issues

Resource names don't match

If you see Lynq creating new resources instead of conflict events, your nameTemplate isn't producing the right names. Check the LynqNode spec to see what names it's trying to create.

Stuck on unexpected fields

The conflict message shows which fields conflict. Common culprits:

replicas (managed by HPA)
annotations (added by other controllers)
labels (injected by admission webhooks)

Use ignoreFields in your resource definition to skip these during reconciliation.

Old tool still trying to manage

If Terraform or Helm is still running somewhere (CI pipeline, cron job), it might fight with Lynq for ownership. Make sure you've fully disabled the old automation before migration.

LynqNode stuck in progressing

Check events: kubectl describe lynqnode <name>. Usually it's a dependency waiting for readiness or a template rendering error.

Rollback plan

If something goes wrong:

Since you used deletionPolicy: Retain, resources are safe
Delete the LynqNode: kubectl delete lynqnode <name>
Resources stay in cluster, just unmanaged
Re-import into Terraform: terraform import ...
Or re-deploy with Helm: helm upgrade --install ...

The retain policy gives you this escape hatch. Use it.

Wrapping up

Migrating to database-driven automation doesn't have to be scary. The key is:

Match existing resource names exactly
Use Stuck policy to verify targeting before forcing
Use Retain policy as a safety net throughout
Migrate incrementally, not all at once
Keep your old tool's state around until you're confident

Take your time. There's no rush. The resources aren't going anywhere.

Questions? Drop them in the comments or open an issue on GitHub.

How Database-Driven Kubernetes Automation Actually Works

Tim Kang — Thu, 04 Dec 2025 00:13:35 +0000

In my previous post, I talked about why I built Lynq and the problems it solves. But I didn't really get into how it actually works. So let's fix that.

If you want to follow along hands-on, there's a Killercoda scenario that walks through everything in about 10 minutes: https://killercoda.com/lynq-operator/course/killercoda/lynq-quickstart

The basic idea

Here's the mental model. You have three things:

LynqHub connects to your database and watches for changes
LynqForm defines what kubernetes resources to create (templates)
LynqNode is the actual instance, one per database row per template

The flow is simple. A row appears in your database. The hub sees it, creates a LynqNode. The node controller renders your templates and applies resources. Done.

When the row disappears or gets deactivated, cleanup happens automatically.

Connecting to your database

First you need a LynqHub. This tells Lynq where your data lives.

apiVersion: operator.lynq.sh/v1
kind: LynqHub
metadata:
  name: my-saas-hub
spec:
  source:
    type: mysql
    mysql:
      host: mysql.default.svc.cluster.local
      port: 3306
      database: nodes
      table: node_data
      username: node_reader
      passwordRef:
        name: mysql-secret
        key: password
  syncInterval: 30s
  valueMappings:
    uid: node_id
    activate: is_active
  extraValueMappings:
    planId: subscription_plan
    region: deployment_region

The valueMappings section is important. You're telling Lynq which columns matter:

uid is the unique identifier for each node
activate is a boolean that controls whether resources should exist

Then extraValueMappings lets you pull in whatever custom fields you need. These become variables you can use in your templates.

The hub polls your database at syncInterval and syncs changes. If a row has activate=true, Lynq creates resources. If it changes to false, cleanup starts.

Defining your resource templates

Next is the LynqForm. This is basically your blueprint for what gets created per database row.

apiVersion: operator.lynq.sh/v1
kind: LynqForm
metadata:
  name: web-app
spec:
  hubId: my-saas-hub
  deployments:
    - id: app
      nameTemplate: "{{ .uid }}-app"
      labelsTemplate:
        app: "{{ .uid }}"
        plan: "{{ .planId | default \"basic\" }}"
      spec:
        apiVersion: apps/v1
        kind: Deployment
        spec:
          replicas: 2
          template:
            spec:
              containers:
                - name: app
                  image: "{{ .deployImage | default \"nginx:latest\" }}"
  services:
    - id: svc
      nameTemplate: "{{ .uid }}-svc"
      dependIds: ["app"]
      spec:
        apiVersion: v1
        kind: Service
        # ...

Templates use Go's text/template syntax with Sprig functions. So you get 200+ functions out of the box. The variables come from your database columns via the hub mappings.

A few things I find myself using constantly:

{{ .uid }} for unique names
{{ .planId | default "basic" }} when columns might be null
{{ .uid | trunc63 }} to respect kubernetes naming limits

Where policies come in

Here's where it gets interesting. Not every resource should behave the same way.

Each resource in your template can have its own policies:

deployments:
  - id: app
    creationPolicy: WhenNeeded
    deletionPolicy: Delete
    conflictPolicy: Stuck
    patchStrategy: apply

creationPolicy controls when resources get created or updated.

WhenNeeded (the default) means Lynq continuously syncs. If someone deletes the resource manually, it comes back. If you update the template, changes apply. This is what you want for most things.

Once means create it once and never touch it again. Perfect for init jobs or migration scripts that should only run on first setup.

jobs:
  - id: init-job
    creationPolicy: Once
    nameTemplate: "{{ .uid }}-init"
    spec:
      apiVersion: batch/v1
      kind: Job
      spec:
        template:
          spec:
            containers:
              - name: init
                command: ["sh", "-c", "echo 'one-time setup'"]
            restartPolicy: Never

deletionPolicy controls what happens when a LynqNode is deleted or the row disappears.

Delete (default) cleans up the resource. Lynq sets an ownerReference so kubernetes garbage collection handles it automatically.

Retain keeps the resource around. Lynq tracks it via labels instead of ownerReference. When deleted, it just gets marked as orphaned so you can find it later.

If you're dealing with PersistentVolumeClaims or anything with data you don't want to lose, use Retain:

persistentVolumeClaims:
  - id: data-pvc
    deletionPolicy: Retain
    nameTemplate: "{{ .uid }}-data"

conflictPolicy handles what happens when a resource already exists with a different owner.

Stuck (default) is conservative. If there's a conflict, reconciliation stops and you get an event. Safe but requires manual intervention.

Force takes ownership using Server-Side Apply with force=true. Useful when you're migrating from another system or when Lynq should be the single source of truth.

Ordering with dependencies

Sometimes you need resources to come up in order. A deployment needs its configmap first. A service should wait for its deployment.

That's what dependIds does:

secrets:
  - id: db-creds
    nameTemplate: "{{ .uid }}-creds"

deployments:
  - id: db
    dependIds: ["db-creds"]
    waitForReady: true

  - id: app
    dependIds: ["db"]
    waitForReady: true

Lynq builds a DAG (directed acyclic graph) from your dependencies and applies resources in topological order. If you accidentally create a cycle, it fails fast with an error.

The waitForReady: true flag is important. Without it, dependIds only guarantees creation order. With it, Lynq actually waits for the dependency to become ready before creating the dependent resource.

There's also skipOnDependencyFailure (defaults to true). If a dependency fails, dependent resources get skipped instead of failing too. Sometimes you want the opposite though:

jobs:
  - id: cleanup-job
    dependIds: ["main-app"]
    skipOnDependencyFailure: false  # run even if main-app fails

How it all ties together

So the full flow looks like this:

Hub controller polls database every syncInterval
For each active row, it creates or updates a LynqNode CR
Node controller picks up the LynqNode
It renders all templates with the row's data
It builds a dependency graph and sorts resources
It applies each resource in order using Server-Side Apply
It waits for readiness if configured
It updates LynqNode status with what got created

When a row is deactivated or deleted:

Hub controller detects the change
LynqNode CR is deleted
Finalizer runs cleanup based on each resource's deletionPolicy
Resources with Delete policy get removed
Resources with Retain policy get orphan labels added

You can watch this in action:

kubectl get lynqnodes -w
kubectl describe lynqnode <name>

The status shows exactly what's happening:

status:
  desiredResources: 5
  readyResources: 5
  failedResources: 0
  appliedResources:
    - "Deployment/default/acme-app@app"
    - "Service/default/acme-svc@svc"

Try it yourself

The best way to understand this is to actually run through it. I set up a Killercoda scenario that walks through the whole thing:

https://killercoda.com/lynq-operator/course/killercoda/lynq-quickstart

It takes about 10 minutes. You'll set up MySQL, deploy the operator, create a hub and template, and then insert/update/delete rows to see how resources respond.

When would you actually use this

This pattern works well when:

You already have business data in a database (users, orgs, tenants)
You need fast provisioning, not commit-sync-reconcile loops
You want to replicate the same resources many times with different values
Template versioning matters more than instance versioning

It's not for everything. If you have a small number of snowflake environments, traditional IaC is probably fine. But if you're replicating the same pattern hundreds of times based on database records, this approach is worth considering.

Docs: https://lynq.sh
GitHub: https://github.com/k8s-lynq/lynq

Happy to answer questions in the comments.

RecordOps: What if Your Database Records Could Provision Infrastructure?

Tim Kang — Wed, 03 Dec 2025 10:18:25 +0000

Here's a scenario you might recognize: You're building a multi-tenant SaaS platform. A new customer signs up, and their data gets inserted into your database. Perfect. Now you need to provision their infrastructure—namespace, deployment, service, ingress. So you:

Write YAML manifests
Commit to Git
Wait for PR approval
Wait for CI/CD
Hope nothing breaks
Update the customer record with their URL

It works, but doesn't it feel... disconnected? Your application already knows everything about this customer through the database. Why are you manually coordinating with a completely separate infrastructure system?

What if Infrastructure Just Read From the Same Database?

That's the idea behind RecordOps.

RecordOps (Record Operations) is a pattern where your database records define infrastructure state. Instead of maintaining YAML files or Terraform code, you define infrastructure parameters as columns in your database. (I'm coining this term to describe a pattern I've been using—maybe it resonates with you too.)

INSERT a row     ->  Infrastructure provisions
UPDATE a column  ->  Resources reconfigure
DELETE a record  ->  Everything cleans up

Every active row in your database represents a running stack in your cluster.

A Concrete Example

Let's say you have a customers table:

CREATE TABLE customers (
  customer_id VARCHAR(50) PRIMARY KEY,
  domain VARCHAR(255) NOT NULL,
  plan VARCHAR(20),
  active BOOLEAN DEFAULT TRUE,
  replicas INT DEFAULT 2
);

With RecordOps, you define a template once: "For each active customer, create a namespace, deployment (with N replicas), service, and ingress (pointing to their domain)."

Now when you onboard a customer:

INSERT INTO customers VALUES
  ('acme-corp', 'acme.example.com', 'enterprise', true, 5);

Within 30 seconds, infrastructure provisions automatically:

Namespace: acme-corp
Deployment: 5 replicas
Service: acme-corp-app
Ingress: acme.example.com -> service

No YAML. No Git. No manual steps. Just a database transaction.

Why This Feels Different

Your Database Already Has All the Answers

Think about what information you need to provision infrastructure:

Customer ID
Domain name
Plan/tier
Region
Resource limits
Feature flags

All of this is already in your database. You're just duplicating it in YAML files or Terraform variables.

Operations Become Data Changes

Common operational tasks are just database operations you already know:

Scale a customer:

UPDATE customers SET replicas = 10 WHERE id = 'acme-corp';

Enable a feature flag:

INSERT INTO feature_flags VALUES ('acme-corp', 'ai-assistant', true);

Blue-green deployment:

UPDATE deployments SET active_version = 'green' WHERE customer_id = 'acme-corp';

No new tooling. No context switching. Just SQL.

Testing Becomes Trivial

Want to clone your production environment to staging? With traditional infrastructure, that's a project. You're exporting state, modifying variables, coordinating across systems.

With RecordOps, it's just cloning database rows:

INSERT INTO customers SELECT * FROM customers WHERE environment = 'prod';
UPDATE customers SET environment = 'staging', domain = CONCAT(domain, '.staging');

30 seconds later, you have a perfect staging environment. Every service, every configuration, every dependency—recreated automatically.

How Does This Compare to GitOps?

GitOps is excellent for cluster-level infrastructure. Your operators, CRDs, system services—these absolutely should be in Git with proper review.

But for per-customer infrastructure? Git becomes tedious. You're creating YAML files for each customer, managing merge conflicts, waiting for pipelines. Meanwhile, your application already knows about these customers.

They work well together:

GitOps: Cluster-level config (changes rarely, needs review)
RecordOps: Customer-level stacks (changes frequently, follows data)

What About Infrastructure-as-Code?

Terraform and Pulumi are great for cloud infrastructure. If you're provisioning AWS resources or managing your cluster itself, absolutely use them.

But if you're provisioning the same pattern repeatedly—one stack per customer, one environment per project—you might not need infrastructure-as-code. You might just need infrastructure-as-data.

Instead of writing code to describe infrastructure, you're adding rows to describe state. It's a different mental model that maps naturally to database-driven applications.

Common Patterns

Feature Flags Control Infrastructure

Instead of deploying optional features for everyone:

CREATE TABLE feature_flags (
  customer_id VARCHAR(50),
  feature VARCHAR(50),
  enabled BOOLEAN
);

-- AI assistant appears only for this customer
INSERT INTO feature_flags VALUES ('acme-corp', 'ai-assistant', true);

Your template includes conditional logic. If the flag exists and is enabled, the AI service deploys. Otherwise, it skips it.

Blue-Green as a Column

CREATE TABLE deployments (
  customer_id VARCHAR(50),
  active_version VARCHAR(10) -- 'blue' or 'green'
);

-- Switch traffic
UPDATE deployments SET active_version = 'green';

Your service selector updates to point to green. Traffic switches in seconds. Roll back by changing it back to 'blue'.

Ephemeral Environments with TTL

CREATE TABLE environments (
  id VARCHAR(50),
  domain VARCHAR(255),
  ttl TIMESTAMP
);

INSERT INTO environments VALUES
  ('demo-123', 'demo-123.example.com', NOW() + INTERVAL 7 DAY);

-- Database trigger cleans up expired environments
CREATE TRIGGER cleanup_expired
AFTER INSERT OR UPDATE ON environments
BEGIN
  DELETE FROM environments WHERE ttl < NOW();
END;

Demo environments provision on insert and auto-cleanup after their TTL.

When Does RecordOps Make Sense?

This pattern works well when:

You're building multi-tenant platforms where each customer needs isolated infrastructure
You provision infrastructure frequently (multiple times per day)
Your infrastructure closely follows your data model
You want less coordination between application and infrastructure

It's probably not right if:

You rarely provision infrastructure (once a month or less)
Every change requires manual approval
You need deep cloud provider integrations beyond Kubernetes

Honestly, you can mix approaches. GitOps for cluster-level, RecordOps for tenant-level, manual for critical changes. They complement each other.

Things to Keep in Mind

Your Database Becomes Critical Infrastructure

It's not just storing application data anymore—it's controlling infrastructure. This means:

Database availability matters more (though existing infrastructure keeps running if DB goes down)
Schema migrations affect infrastructure (test carefully)
Database permissions become infrastructure permissions

Security Model Shifts

SQL injection vulnerabilities can become infrastructure vulnerabilities. If user input can manipulate your queries, they could trigger unwanted infrastructure changes. Use parameterized queries and validate inputs.

Sync Delays Exist

With RecordOps, there's typically a sync interval (e.g., 30 seconds) between database changes and infrastructure updates. For most cases this is fine, but if you need instant provisioning, you'll need to tune this or reconsider.

Why I'm Exploring This

I kept running into the same problem: syncing my application's database state with infrastructure state. I'd add a customer to the database, then manually coordinate with my infrastructure tooling. Eventually I realized—they could just be the same thing.

RecordOps isn't revolutionary. It's actually pretty obvious once you see it. If your infrastructure maps to your data, why not let your data drive your infrastructure?

This pattern won't replace every tool in your stack. But for the specific problem of provisioning repeated patterns (per-customer stacks, per-project environments), it might simplify your life.

If You Want to Try It

I built Lynq, an open-source operator that implements RecordOps for Kubernetes. You point it at your database, define your templates, and it handles the rest.

But the pattern itself is tool-agnostic. You could build your own implementation, use a different tool, or just take the concepts and apply them however makes sense for your stack.

What do you think? Have you felt this pain before? How are you handling per-customer infrastructure provisioning today?

I'd love to hear your thoughts and experiences in the comments. 👇