Forem: DarkEdges

Enriching Vault OIDC Tokens with SPIFFE Identity Metadata using Terraform

DarkEdges — Wed, 03 Dec 2025 19:39:08 +0000

In modern microservices architectures, machine identity is just as critical as human identity. When services communicate, they often need to prove not just who they are, but what they are—their environment, business unit, or SPIFFE ID.

HashiCorp Vault is excellent for this. Its Identity Secrets Engine can issue OIDC tokens that serve as verifiable credentials for your workloads. However, getting those tokens to contain rich, custom metadata requires connecting a few specific dots: AppRole, Identity Entities, and OIDC Templates.

In this article, I'll walk through how to implement a robust machine identity pipeline using Terraform, where an application authenticates via AppRole and receives an OIDC token enriched with custom metadata.

The Goal

We want an application (e.g., a "ChatBot") to:

Authenticate to Vault using the AppRole method.
Request an OIDC token.
Receive a token containing custom claims like spiffe_id, business_unit, and environment.

Step 1: Define the Identity Entity

First, we define the "who". In Vault, an Entity represents a unique identity. This is where we store the metadata we want to eventually see in our token.

# identities.tf

resource "vault_identity_entity" "application" {
  for_each = local.application_identities_map
  name     = each.key

  # This metadata is what we'll inject into the token later
  metadata = {
    environment   = each.value.identity.environment
    business_unit = each.value.identity.business_unit
    spiffe_id     = "spiffe://vault/application/${each.value.identity.environment}/${each.value.identity.business_unit}/${each.value.identity.name}"
  }
}

Step 2: Configure AppRole Authentication

Next, we configure the AppRole auth backend. This is the standard way for machines to authenticate.

# approle.tf

resource "vault_approle_auth_backend_role" "applications" {
  for_each       = local.application_identities_map
  backend        = vault_auth_backend.approle.path
  role_name      = each.key
  token_ttl      = 3600
  bind_secret_id = true
}

The "Secret Sauce": Binding AppRole to the Entity

This is the most critical step. By default, when you log in with AppRole, Vault creates a generic entity for that role. To use our custom metadata defined in Step 1, we must explicitly bind the AppRole to our specific Entity using an Entity Alias.

Important Gotcha: When creating an alias for AppRole, the name of the alias must be the Role ID, not the Role Name.

# approle.tf

resource "vault_identity_entity_alias" "approle_applications" {
  for_each       = local.application_identities_map

  # CRITICAL: Use role_id, not role_name
  name           = vault_approle_auth_backend_role.applications[each.key].role_id

  mount_accessor = vault_auth_backend.approle.accessor
  canonical_id   = vault_identity_entity.application[each.key].id
}

We also need to ensure the AppRole inherits the entity's properties. We can enforce this via a generic endpoint configuration:

resource "vault_generic_endpoint" "approle_entity_inherit" {
  for_each   = local.application_identities_map
  depends_on = [vault_approle_auth_backend_role.applications]
  path       = "auth/approle/role/${each.key}"

  data_json = jsonencode({
    entity_alias_sole_inherit = true
  })
}

Step 3: Configure the OIDC Template

Finally, we configure the OIDC provider and role. The template field is where the magic happens. We use Vault's template syntax to pull values dynamically from the authenticated entity's metadata.

# identity_tokens.tf

resource "vault_identity_oidc_role" "application_identity" {
  name      = "application_identity"
  key       = vault_identity_oidc_key.application_identity.name
  client_id = "spiffe://vault.darkedges.au/gateway"
  ttl       = 86400

  # The Template: Injecting metadata into the JSON payload
  template = <<EOT
{
  "azp": {{identity.entity.metadata.spiffe_id}},
  "nbf": {{time.now}},
  "groups": {{identity.entity.groups.names}},
  "appinfo": {
    "business_unit": {{identity.entity.metadata.business_unit}},
    "environment": {{identity.entity.metadata.environment}}
  }
}
EOT
}

Step 4: Testing the Flow

Now, let's see it in action. We'll use PowerShell to simulate the application workflow.

1. Get Credentials & Login

First, we retrieve the Role ID and Secret ID, then authenticate to get a Vault token.

# Get Role ID and Secret ID
$ROLE_ID = docker-compose exec vault vault read -field=role_id auth/approle/role/ChatBot/role-id
$SECRET_ID = docker-compose exec vault vault write -f -field=secret_id auth/approle/role/ChatBot/secret-id

# Login
$APPTOKEN = docker-compose exec vault vault write -field=token auth/approle/login role_id="$ROLE_ID" secret_id="$SECRET_ID"

2. Request the OIDC Token

Using the Vault token we just got, we request the OIDC token.

$OIDC_TOKEN = docker-compose exec -e VAULT_TOKEN="$APPTOKEN" vault vault read -field=token identity/oidc/token/application_identity

3. The Result

When we decode the JWT, we see our rich metadata is successfully populated!

{
  "appinfo": {
    "business_unit": "engineering",
    "environment": "production"
  },
  "aud": "spiffe://vault.darkedges.au/gateway",
  "azp": "spiffe://vault/application/production/engineering/ChatBot",
  "exp": 1764848993,
  "groups": [
    "ChatBot group"
  ],
  "iat": 1764762593,
  "iss": "https://vault.darkedges.au/v1/identity/oidc",
  "sub": "ea0e0006-d4f5-cbde-3cb6-c013d4dba5f2"
}

Conclusion

By combining Identity Entities, AppRole Aliases, and OIDC Templates, we've turned Vault into a powerful identity provider for our internal services. This setup allows downstream systems (like API gateways or service meshes) to make authorization decisions based on trusted, verifiable metadata like business_unit or environment, rather than just a raw IP address or generic token.

a complete example is available at https://github.com/darkedges/spiffe-vault-terraform

Inspiration

Auto-Detecting CSV Schemas for Lightning-Fast ClickHouse Ingestion with Parquet

DarkEdges — Fri, 07 Nov 2025 09:06:41 +0000

As a data engineer, one of the most repetitive tasks I face is ingesting data from CSV files. The problem isn't just loading the data; it's the ceremony that comes with it. Every time a new data source appears, I have to manually inspect the columns, define a table schema, and write a script to load it. What if the CSV has 100 columns? What if the data types are ambiguous? This process is tedious and error-prone.

I wanted a better way. My goal was to create a Node.js script that could:

Read any CSV file without prior knowledge of its structure.
Auto-detect the schema, including column names and data types.
Convert the CSV to Parquet, a highly efficient columnar storage format.
Prepare for ingestion into ClickHouse, which loves Parquet.

In this article, I'll walk you through the proof-of-concept I built to solve this exact problem.

Why Parquet and ClickHouse?

ClickHouse is an open-source, column-oriented database built for speed. It's a beast for analytical queries (OLAP).

Apache Parquet is a columnar storage format. Instead of storing data in rows, it stores it in columns. This is a perfect match for ClickHouse because it allows the database to read only the columns it needs for a query, dramatically reducing I/O and speeding up performance.

Combining the two means you get efficient storage and lightning-fast analytics.

The Concept: From CSV to Parquet

Our script will perform a three-step process:

Schema Detection: Read the CSV headers and a few sample rows to infer the data type of each column (e.g., string, number, boolean).
Data Transformation: Convert the raw string values from the CSV into their inferred types.
Parquet Conversion: Write the transformed data and the detected schema into a new .parquet file.

Let's start with a simple CSV file to demonstrate.

Our Example CSV: `sample-data.csv`

id,first_name,last_name,email,is_active,created_at,balance
1,John,Doe,john.doe@example.com,true,2023-01-15T10:00:00Z,150.75
2,Jane,Smith,jane.smith@example.com,false,2023-02-20T11:30:00Z,200.00
3,Peter,Jones,peter.jones@example.com,true,2023-03-10T09:05:00Z,50.25
4,Mary,Williams,mary.w@example.com,true,2023-04-01T15:45:00Z,300.50

This file has a nice mix of data types: integers, strings, booleans, timestamps, and floating-point numbers.

Step 1: Setting Up the Project

First, let's set up a simple Node.js project and install the necessary libraries. We'll use papaparse for robust CSV parsing and parquetjs for writing Parquet files.

npm init -y
npm install papaparse parquetjs

Step 2: Auto-Detecting the Schema

This is the core of our solution. We need a function that can look at the string data from the CSV and make an educated guess about its real type.

Here’s a simple type inference function. It checks if a value is a boolean, a number, or a date. If it's none of those, it defaults to a string.

index.js

// Function to infer data type from a string value
function inferType(value) {
    if (value === 'true' || value === 'false') return 'BOOLEAN';
    if (!isNaN(value) && value.trim() !== '') return 'DOUBLE'; // Use DOUBLE for all numbers for simplicity
    if (!isNaN(Date.parse(value))) return 'UTF8'; // Dates will be stored as strings (UTF8)
    return 'UTF8'; // Default to string
}

Now, we can use this to build a schema from the CSV file. We'll read the first data row to infer the types for each column.

index.js

const fs = require('fs');
const papa = require('papaparse');

// ... inferType function from above ...

async function detectSchema(filePath) {
    const fileContent = fs.readFileSync(filePath, 'utf8');
    const result = papa.parse(fileContent, { header: true, preview: 1 });

    if (result.errors.length > 0) {
        throw new Error('Error parsing CSV for schema detection.');
    }

    const firstRecord = result.data[0];
    const schema = {};

    for (const header in firstRecord) {
        const value = firstRecord[header];
        schema[header] = { type: inferType(value) };
    }

    return schema;
}

When we run detectSchema('sample-data.csv'), it will produce an object like this, which is exactly the format parquetjs needs:

{
  "id": { "type": "DOUBLE" },
  "first_name": { "type": "UTF8" },
  "last_name": { "type": "UTF8" },
  "email": { "type": "UTF8" },
  "is_active": { "type": "BOOLEAN" },
  "created_at": { "type": "UTF8" },
  "balance": { "type": "DOUBLE" }
}

Step 3: Converting CSV to Parquet

With the schema detected, we can now read the entire CSV and write it to a Parquet file. The process involves:

Creating a ParquetSchema and a ParquetWriter.
Iterating through the CSV rows.
Casting each value to its detected type (e.g., converting the string "true" to the boolean true).
Appending the transformed row to the writer.

Here's the code to bring it all together:

index.js

const { ParquetSchema, ParquetWriter } = require('parquetjs');

// ... detectSchema and inferType functions ...

async function convertCsvToParquet(csvPath, parquetPath) {
    console.log('Detecting schema...');
    const schemaDefinition = await detectSchema(csvPath);
    const schema = new ParquetSchema(schemaDefinition);

    const writer = await ParquetWriter.openFile(schema, parquetPath);

    const fileContent = fs.readFileSync(csvPath, 'utf8');
    const result = papa.parse(fileContent, { header: true, skipEmptyLines: true });

    console.log(`Found ${result.data.length} records. Converting to Parquet...`);

    for (const row of result.data) {
        const processedRow = {};
        for (const key in row) {
            const value = row[key];
            const type = schemaDefinition[key].type;

            // Cast values to their proper types
            if (type === 'BOOLEAN') {
                processedRow[key] = value === 'true';
            } else if (type === 'DOUBLE') {
                processedRow[key] = parseFloat(value);
            } else {
                processedRow[key] = value;
            }
        }
        await writer.appendRow(processedRow);
    }

    await writer.close();
    console.log(`Parquet file written to ${parquetPath}`);
}

// Run the conversion
convertCsvToParquet('sample-data.csv', 'output.parquet').catch(console.error);

After running this script (node index.js), you'll have an output.parquet file ready for ClickHouse!

Step 4: Ingesting into ClickHouse

Now for the easy part. Ingesting the Parquet file into ClickHouse is incredibly simple. First, you need a table with a matching schema.

CREATE TABLE my_data (
    id Float64,
    first_name String,
    last_name String,
    email String,
    is_active Boolean,
    created_at String,
    balance Float64
) ENGINE = MergeTree()
ORDER BY id;

Then, you can use the clickhouse-client to ingest the file directly.

clickhouse-client --query="INSERT INTO my_data FORMAT Parquet" < output.parquet

ClickHouse reads the Parquet file's schema and streams the data directly into the table. It's fast, efficient, and requires no complex INSERT statements with tons_of_values.

Conclusion

By automating schema detection and converting CSVs to Parquet, we've created a powerful and reusable ETL pipeline. This approach saves a massive amount of time, reduces manual errors, and leverages the high-performance capabilities of both Parquet and ClickHouse.

This proof-of-concept can be expanded with more robust type detection, error handling, and integration into a larger data workflow, but it's a fantastic starting point for streamlining your data ingestion process.

Uprading Ping Identity Platform to 8.0.0

DarkEdges — Fri, 22 Aug 2025 04:25:24 +0000

Ping AM

Issue with new config when running amupgrade

The following appears to be added in the wrong place with incorrect details

Error on Startup after migration

When PingAM Starts and you attempt to connect it fails with this in the log.

dfq-am  | {"timestamp":"2025-08-22T04:24:20.258Z","level":"WARN","thread":"http-nio-8080-exec-10","mdc":{"transactionId":"4a84d7a7-8b56-4819-b5c6-6adc4d215e2e-38"},"logger":"org.forgerock.openam.core.realms.impl.DefaultRealmLookup","message":"DefaultRealms:lookup Unable to find Org name for: serverinfo","context":"default","transactionId":"4a84d7a7-8b56-4819-b5c6-6adc4d215e2e-38"}
dfq-am  | {"timestamp":"2025-08-22T04:24:20.397Z","level":"WARN","thread":"http-nio-8080-exec-2","mdc":{"transactionId":"4a84d7a7-8b56-4819-b5c6-6adc4d215e2e-43"},"logger":"org.forgerock.openam.core.realms.impl.DefaultRealmLookup","message":"DefaultRealms:lookup Unable to find Org name for: sessions","context":"default","transactionId":"4a84d7a7-8b56-4819-b5c6-6adc4d215e2e-43"}
dfq-am  | {"timestamp":"2025-08-22T04:24:20.414Z","level":"WARN","thread":"http-nio-8080-exec-2","mdc":{"transactionId":"4a84d7a7-8b56-4819-b5c6-6adc4d215e2e-43"},"logger":"com.sun.identity.monitoring.MonitoringServicesImpl","message":"JDMK runtime not found - Policy Monitoring disabled","context":"default","transactionId":"4a84d7a7-8b56-4819-b5c6-6adc4d215e2e-43"}
dfq-am  | {"timestamp":"2025-08-22T04:24:20.523Z","level":"WARN","thread":"http-nio-8080-exec-2","mdc":{"transactionId":"4a84d7a7-8b56-4819-b5c6-6adc4d215e2e-43"},"logger":"com.sun.identity.idm.plugins.internal.AgentsRepo","message":"AgentsRepo.getAgentGroupConfig: Unable to get Agent Group Config due to The instance agentgroup does not exist","context":"default","transactionId":"4a84d7a7-8b56-4819-b5c6-6adc4d215e2e-43"}
dfq-am  | {"timestamp":"2025-08-22T04:24:20.637Z","level":"WARN","thread":"http-nio-8080-exec-1","mdc":{"transactionId":"4a84d7a7-8b56-4819-b5c6-6adc4d215e2e-56"},"logger":"org.forgerock.openam.core.rest.authn.http.AuthenticationServiceV2","message":"Authentication encountered an error: [Status: 401 Unauthorized]","context":"default","transactionId":"4a84d7a7-8b56-4819-b5c6-6adc4d215e2e-56"}

it is missing the following entry in Ping DS

dn: ou=agentgroup,ou=Instances,ou=1.0,ou=AgentService,ou=services,ou=am-config
objectClass: top
objectClass: sunServiceComponent
objectClass: organizationalUnit
ou: agentgroup
sunserviceID: agentgroup

Creating a Workflow using your Connector

DarkEdges — Sat, 16 Aug 2025 06:34:34 +0000

In order to use connector you need to

Create a Connection
Create a Flow

Create a Connection

Connect to your Okta Workflow instance.
Select Connections.
Under Connections click the New Connection icon.
Select Your connector
Provide the details and click the Create button.

Create a Flow

Connect to your Okta Workflow instance.
Select Flows.
Click the New Flow button.
In the When this happens block click Add event and select API Endpoint.
Click save
In the Then do this block click Add function to add functions.
When finished it should look like.
Save it as the name you want to display when selecting the action. Ensure you select ``

Test Flow

Click the Run button and click Run
When it has completed the execution it should be successful.
Open the flow directly and it should return similair to

` { "output": { "length": 1 }, "statusCode": 200 } `

Creating a Connector using the Okta Workflow Connector Builder

DarkEdges — Sat, 16 Aug 2025 06:33:53 +0000

Okta Workflow is a no code development environment to create workflows to perform a large number of operations based on Connections. If there is not a Connector available you can create your own using their Connector Builder, and the best part is that you can try it out using their free Integration account.

Okta workflow no code approach is drag and drop between functions / actions. Most functions have Inputs and Outputs, so you can drag an Output onto an Input. Once done you can highlight Input or Output to see the connection.

For example:

Here are the basic steps.

Create a Connector.
Configure Authentication.
Create a httpHelper flow - used by the connector to make requests using the connection Acces token.
Create a _pingAuth flow - Validates the Access Token to see if it needs to be renewed.
Create a action flow- Performs an operation that can be used in Flow.
Deploy it

Create a Connector.

Connect to your Okta Workflow instance.
Select Connector Builder.
Under Connectors click the + icon.
Provide the name for the connector and click the Save button.

Configure Authentication.

Click the Add Authentication button and fill out the details

Create a Test Connection

Click Test Connections
Click +New Connection
Provide the details and click the Create button

Create a `httpHelper` flo.

Select Flows
Click +New Flow
In the When this happens block click Add event and select Helper Flow.
In the Then do this block click Add function to add functions.
When finished it should look like.
Save it as httpHelper

Create a `_pingAuth` flo.

Select Flows
Click +New Flow
In the When this happens block click Add event and select Authping.
In the Then do this block click Add function to add functions.
When finished it should look like.
Save it as _authPing

Create a `action` flo.

Select Flows
Click +New Flow
In the When this happens block click Add event and select Action.
In the Then do this block click Add function to add functions.
When finished it should look like.
Save it as the name you want to display when selecting the action.

Deploy

Click Deployment
Click Validate connector and then Done
Click Deploy test version
Click Deploy local connector

That gets a connector deployed and ready for using in an actual flow.

Finally testing the solution

DarkEdges — Sat, 16 Aug 2025 02:09:24 +0000

To recap, we have an Private Application Load Balancer, Fargate ECS Cluster and an AWS API Gateway v2. So know we can test the solution.

Deploying the AWS API Gateway V2 to LocalStack

DarkEdges — Sat, 16 Aug 2025 02:06:00 +0000

We are getting close to being able to test our solution, the final piece is the deployment of an AWS API Gateway v2 with an Authorizer that trusts Auth0 tokens.

For this we will create a number of AWS Configurations

API Gateway V2 will connect to the Fargate API Instance we deployed previously.

This is a lot of configuration items so they are avaiable in the following branch.

git switch apiv2

First we will plan and then apply the changes if that is green, by using the following commands.

terraform plan
terraform apply --auto-approve

Once it has been deployed you should get the following output

alb = "http://alb.elb.localhost.localstack.cloud:4566/"

We will use this address in a future request to confirm the API has been deployed correctly.

Deploying a custom API via AWS Fargate

DarkEdges — Sat, 16 Aug 2025 02:01:06 +0000

Now we have a Private Application Load Balancer deployed we can look at putting an API behind it. For this we will use an existing container image darkedges/providerapi:1.0.2 which provides a basic API that we are going to serve via an AWS API Gateway V2 protected by an Auth0 Access Token.

For this we will create a number of AWS Configurations

ECS Cluster as our Fargate service.
ECS Service Task to deploy the custom API Container
ALB updates to link it al.

This is a lot of configuration items so they are avaiable in the following branch.

git switch fargate

First we will plan and then apply the changes if that is green, by using the following commands.

terraform plan
terraform apply --auto-approve

Once it has been deployed you should get the following output

alb = "http://alb.elb.localhost.localstack.cloud:4566/"

We will use this address in a future request to confirm the API has been deployed correctly.

Deploying an Private ALB to LocalStack

DarkEdges — Sat, 16 Aug 2025 00:49:39 +0000

In the previous articles we stood up LocalStack and configured Terraform to plan a deployment. Next we will deploy an ALB to the platform and get its address so that we can use it in the next time.

For this we will create a number of AWS Configurations

VPC to deploy into.
Networking to allow the ALB to be deployed and connect to the ECS instance.
SecurityGroups to manage all the network egress / ingress between services.
ALB the Application Load Balancer.

This is a lot of configuration items so they are avaiable in the following branch.

git switch alb

First we will plan and then apply the changes if that is green, by using the following commands.

terraform plan
terraform apply --auto-approve

Once it has been deployed you should get the following output

alb = "http://alb.elb.localhost.localstack.cloud:4566/"

We will use this address in a future request to confirm the API has been deployed correctly.

Configuring Terraform to deploy into LocalStack

DarkEdges — Fri, 15 Aug 2025 23:44:20 +0000

From our previous article we have successfuly registered and deployed LocalStack. Now we are planning to deploy our API onto LocalStack using AWS Fargate via Terraform, but we need to deploy some supporting services first, such as the core Terraform configuration.

Terraform

We are going to be using terraform to manage our state so lets create the terraform directory and place the following file _provider.tf

provider "aws" {
  region     = "us-east-1"
  access_key = "test"
  secret_key = "test"

  endpoints {
    acm                      = "http://localhost:4566"
    amplify                  = "http://localhost:4566"
    apigateway               = "http://localhost:4566"
    apigatewayv2             = "http://localhost:4566"
    appconfig                = "http://localhost:4566"
    applicationautoscaling   = "http://localhost:4566"
    appsync                  = "http://localhost:4566"
    athena                   = "http://localhost:4566"
    autoscaling              = "http://localhost:4566"
    backup                   = "http://localhost:4566"
    batch                    = "http://localhost:4566"
    cloudformation           = "http://localhost:4566"
    cloudfront               = "http://localhost:4566"
    cloudsearch              = "http://localhost:4566"
    cloudtrail               = "http://localhost:4566"
    cloudwatch               = "http://localhost:4566"
    cloudwatchlogs           = "http://localhost:4566"
    codecommit               = "http://localhost:4566"
    cognitoidentity          = "http://localhost:4566"
    cognitoidp               = "http://localhost:4566"
    config                   = "http://localhost:4566"
    costexplorer             = "http://localhost:4566"
    docdb                    = "http://localhost:4566"
    dynamodb                 = "http://localhost:4566"
    ec2                      = "http://localhost:4566"
    ecr                      = "http://localhost:4566"
    ecs                      = "http://localhost:4566"
    efs                      = "http://localhost:4566"
    eks                      = "http://localhost:4566"
    elasticache              = "http://localhost:4566"
    elasticbeanstalk         = "http://localhost:4566"
    elasticsearch            = "http://localhost:4566"
    elb                      = "http://localhost:4566"
    elbv2                    = "http://localhost:4566"
    emr                      = "http://localhost:4566"
    events                   = "http://localhost:4566"
    firehose                 = "http://localhost:4566"
    glacier                  = "http://localhost:4566"
    glue                     = "http://localhost:4566"
    iam                      = "http://localhost:4566"
    iot                      = "http://localhost:4566"
    iotanalytics             = "http://localhost:4566"
    iotevents                = "http://localhost:4566"
    kafka                    = "http://localhost:4566"
    kinesis                  = "http://localhost:4566"
    kinesisanalytics         = "http://localhost:4566"
    kinesisanalyticsv2       = "http://localhost:4566"
    kms                      = "http://localhost:4566"
    lakeformation            = "http://localhost:4566"
    lambda                   = "http://localhost:4566"
    mediaconvert             = "http://localhost:4566"
    mediastore               = "http://localhost:4566"
    neptune                  = "http://localhost:4566"
    organizations            = "http://localhost:4566"
    qldb                     = "http://localhost:4566"
    rds                      = "http://localhost:4566"
    redshift                 = "http://localhost:4566"
    redshiftdata             = "http://localhost:4566"
    resourcegroups           = "http://localhost:4566"
    resourcegroupstaggingapi = "http://localhost:4566"
    route53                  = "http://localhost:4566"
    route53resolver          = "http://localhost:4566"
    s3                       = "http://s3.localhost.localstack.cloud:4566"
    s3control                = "http://localhost:4566"
    sagemaker                = "http://localhost:4566"
    secretsmanager           = "http://localhost:4566"
    serverlessrepo           = "http://localhost:4566"
    servicediscovery         = "http://localhost:4566"
    ses                      = "http://localhost:4566"
    sesv2                    = "http://localhost:4566"
    sns                      = "http://localhost:4566"
    sqs                      = "http://localhost:4566"
    ssm                      = "http://localhost:4566"
    stepfunctions            = "http://localhost:4566"
    sts                      = "http://localhost:4566"
    swf                      = "http://localhost:4566"
    timestreamwrite          = "http://localhost:4566"
    transfer                 = "http://localhost:4566"
    waf                      = "http://localhost:4566"
    wafv2                    = "http://localhost:4566"
    xray                     = "http://localhost:4566"
  }

  default_tags {
    tags = {
      Environment = "Local"
      Service     = "LocalStack"
    }
  }
}

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.100.0"
    }
  }
}

We can test it is working by performing the following

terraform init
terraform plan

If all went well we should see

terraform init

Initializing the backend...
Initializing provider plugins...
- Finding hashicorp/aws versions matching "6.9.0"...
- Installing hashicorp/aws v6.9.0...
- Installed hashicorp/aws v6.9.0 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

terraform plan

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

Protecting an API with AWS API Gateway and Auth0

DarkEdges — Fri, 15 Aug 2025 22:45:27 +0000

You have an API that you would like to protect with an OAuth Access Token from Auth0 using AWS API Gateway. Sounds simple enough, so lets do this using a local development environment thats hosts

API - Deployed as a simple Go API Services using AWS Fargate
API Gateway - Deployed using API Gateway V2

Now this normally would require creating an AWS Account, but we are going to be using LocalStack which provide those services (and more) in a container that can be deployed locally. This way we can learn AWS without the costs (its not much anyway) by deploying the services via Terraform.

Pre-requisites

We are assuming that you have the following deployed locally

Docker
Terraform
Visual Studio Code
Git

You will also need to sign up at https://app.localstack.cloud/sign-up to get an API Key that we need to run this stack.

Signup and New Instance

The following outlines the basic steps to get a localstack instance registered

Signup

Open https://app.localstack.cloud/sign-up
Provide an email address and click Continue.
Provide a password and click Continue.
Provide your details and click Start Trial.
Check your email and follow the instrucions provided to Activate Account
When completed login using the details your provided earlier and click Continue.

Get Personal Auth Token

Go to https://app.localstack.cloud/settings/auth-tokens
Under Personal Auth Token click the Copy icon.

Deploying LocalStack

Create a file docker-compose.yaml with the following and replace <copy PAT here> with your PAT you copied earlier.

services:
  localstack:
    container_name: "lsa-localstack"
    image: localstack/localstack-pro:latest
    environment:
      LOCALSTACK_AUTH_TOKEN: <copy PAT here>
    ports:
      - "127.0.0.1:4566:4566"
      - "127.0.0.1:4510-4559:4510-4559"
      - "127.0.0.1:443:443"
    volumes:
      - "localstack:/var/lib/localstack:rw"
      - "/var/run/docker.sock:/var/run/docker.sock"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:4566/_localstack/health"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 15s
    networks:
      localstack:
        aliases:
          - localstack
volumes:
  localstack:
networks:
  localstack:

and deploy it using docker compose up -d

You can confirm it is working by going to https://app.localstack.cloud/inst/default/status and the services should be marked as available

CLI

You can install a version of the AWS CLI that works with localstack by reviewing https://docs.localstack.cloud/aws/integrations/aws-native-tools/aws-cli/

That concludes this part. Next we configure Terraform and deploy our API Service into AWS Fargate.

kind, wsl2 and multus

DarkEdges — Tue, 22 Apr 2025 12:50:08 +0000

All we wanted to do was install kubevirt create 3 Windows 2022 servers and have them talk to one another over a static IP Address. Simple we thought, Just tell the Virtual Machine to use a host-only IP Address and we are in the money.

Firstly you cannot assign a second network interface unless it is another CNI implementation and secondly you need to install more black magic to make it work.

That black magic is https://github.com/k8snetworkplumbingwg/multus-cni

Step 0 - Install WSL2 and AlmaLinux 9 with docker

We will write a seperate guide on how to do this and update this one with a link to it.

Step 1 - Install kind

We are using kind as it is a supported version for kubevirt so we do that by creating a config file and using it in the deployment

apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
nodes:
  - role: control-plane

Then create the cluster using

kind create cluster --name kubevirt --config=kind_config.yaml

Step 2 - Install Multus Daemonset

kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset-thick.yml

Step 3 - Install CNI plugins

First find the name of control plane

kubectl get nodes

which returns kubevirt-control-plane

NAME                     STATUS   ROLES           AGE   VERSION
kubevirt-control-plane   Ready    control-plane   51m   v1.32.2

then connect to the control plane

docker exec -it kubevirt-control-plane sh

to allow us to install the missing CNI Plugins

cd /tmp
curl -L -o cni-plugins.tgz https://github.com/containernetworking/plugins/releases/download/v1.6.2/cni-plugins-linux-amd64-v1.6.2.tgz
tar -C /opt/cni/bin -xzf cni-plugins.tgz
rm -rf cni-plugins.tgz

Step 4 create a NetworkAttachmentDefinition

This is done through the following

cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: ipvlan-def
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ipvlan",
      "master": "eth0",
      "mode": "l2",
      "ipam": {
        "type": "host-local",
        "subnet": "192.168.200.0/24",
        "rangeStart": "192.168.200.201",
        "rangeEnd": "192.168.200.205",
        "gateway": "192.168.200.1"
      }
    }'
EOF

Step 5 - Create some Pods with additional network interfaces

The following creates

3 pods with additional Network Interfaces listing on
- 192.168.200.201 - multus-alpine
- 192.168.200.202 - multus-alpine-2
- 192.168.200.203 - net-pod

cat <<EOF | kubectl create -f -
---
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: multus-alpine
  name: multus-alpine
  namespace: default
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
      {
         "name" : "ipvlan-def",
         "interface": "eth1",
         "ips": ["192.168.200.201"]
      }
    ]'
spec:
  containers:
    - name: multus-alpine
      image: jmalloc/echo-server
  restartPolicy: Always
---
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: multus-alpine-2
  name: multus-alpine-2
  namespace: default
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
      {
         "name" : "ipvlan-def",
         "interface": "eth1",
        "ips": ["192.168.200.202"]
      }
    ]'
spec:
  containers:
    - name: multus-alpine
      image: jmalloc/echo-server
  restartPolicy: Always
---
apiVersion: v1
kind: Pod
metadata:
  name: net-pod
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
      {
         "name" : "ipvlan-def",
         "interface": "eth1",
        "ips": ["192.168.200.203"]
      }
    ]'
spec:
  containers:
  - name: netshoot-pod
    image: nicolaka/netshoot
    command: ["tail"]
    args: ["-f", "/dev/null"]
  terminationGracePeriodSeconds: 0
EOF

Step 6 - Test connectivity

Check that we are getting responses from both multus-alpine and multus-alpine-2 from the net-pod

multus-alpine

kubectl exec -it net-pod -- curl http://192.168.200.201:8080/

returns

Request served by multus-alpine

GET / HTTP/1.1

Host: 192.168.200.201:8080
Accept: */*
User-Agent: curl/8.7.1

multus-alpine-2

kubectl exec -it net-pod -- curl http://192.168.200.202:8080/

returns

Request served by multus-alpine-2

GET / HTTP/1.1

Host: 192.168.200.202:8080
Accept: */*
User-Agent: curl/8.7.1

Forem: DarkEdges

Enriching Vault OIDC Tokens with SPIFFE Identity Metadata using Terraform

The Goal

Step 1: Define the Identity Entity

Step 2: Configure AppRole Authentication

The "Secret Sauce": Binding AppRole to the Entity

Step 3: Configure the OIDC Template

Step 4: Testing the Flow

1. Get Credentials & Login

2. Request the OIDC Token

3. The Result

Conclusion

Inspiration

Auto-Detecting CSV Schemas for Lightning-Fast ClickHouse Ingestion with Parquet

Why Parquet and ClickHouse?

The Concept: From CSV to Parquet

Our Example CSV: sample-data.csv

Step 1: Setting Up the Project

Step 2: Auto-Detecting the Schema

Step 3: Converting CSV to Parquet

Step 4: Ingesting into ClickHouse

Conclusion

Uprading Ping Identity Platform to 8.0.0

Ping AM

Issue with new config when running amupgrade

Error on Startup after migration

Creating a Workflow using your Connector

Create a Connection

Create a Flow

Test Flow

Creating a Connector using the Okta Workflow Connector Builder

Create a Connector.

Configure Authentication.

Create a Test Connection

Create a httpHelper flo.

Create a _pingAuth flo.

Create a action flo.

Deploy

Finally testing the solution

Deploying the AWS API Gateway V2 to LocalStack

Deploying a custom API via AWS Fargate

Deploying an Private ALB to LocalStack

Configuring Terraform to deploy into LocalStack

Terraform

terraform init

terraform plan

Protecting an API with AWS API Gateway and Auth0

Pre-requisites

Signup and New Instance

Signup

Get Personal Auth Token

Deploying LocalStack

CLI

kind, wsl2 and multus

Step 0 - Install WSL2 and AlmaLinux 9 with docker

Step 1 - Install kind

Step 2 - Install Multus Daemonset

Step 3 - Install CNI plugins

Step 4 create a NetworkAttachmentDefinition

Step 5 - Create some Pods with additional network interfaces

Step 6 - Test connectivity

multus-alpine

multus-alpine-2

Our Example CSV: `sample-data.csv`

Create a `httpHelper` flo.

Create a `_pingAuth` flo.

Create a `action` flo.