Forem: Kimi Huang

KRO: A new generation tool to manage Kubernetes manifests and deployment

Kimi Huang — Thu, 22 May 2025 07:09:41 +0000

About KRO

KRO (Kube Resource Orchestrator) is an open-source, Kubernetes-native project that allows you to define custom Kubernetes APIs using simple and straightforward configuration.

KRO is currently governed by a small group of maintainers - from Google, Amazon, and Microsoft - who are responsible for the overall direction and management of the project, as well as for the technical decisions that affect the project.

Overview

When you mention manifest and deployment management tools on Kubernetes, the first things that come to mind are Helm or Kustomization. Later, there was even a project in CNCF called Crossplane that allows you to implement IaC through YAML files, making it very convenient for you to deploy infrastructure on Cloud Service Providers.

KRO proposed a brand new approach, allowing you to manage manifests and deployments on Kubernetes through very intuitive YAML file writing, and also enabling infrastructure deployment like Crossplane.

Since KRO is still in a very early stage, some features are not yet complete. This article will use a very simple example to introduce how KRO performs manifest and deployment management.

Installation

1. Installing KRO

Before you begin, ensure you have the following:

Helm 3.x installed
kubectl installed and configured to interact with your Kubernetes cluster

export KRO_VERSION=$(curl -sL \
    https://api.github.com/repos/kro-run/kro/releases/latest | \
    jq -r '.tag_name | ltrimstr("v")'
  )

helm install kro oci://ghcr.io/kro-run/kro/kro \
  --namespace kro \
  --create-namespace \
  --version=${KRO_VERSION}

2. Check KRO Readiness

kubectl get all -n kro

NAME                       READY   STATUS    RESTARTS      AGE
pod/kro-69b4b67b67-xhvht   1/1     Running   0             1d

NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kro   1/1     1            1           1d

NAME                             DESIRED   CURRENT   READY   AGE
replicaset.apps/kro-69b4b67b67   1         1         1       1d

kubectl get crd

NAME                               CREATED AT
applications.kro.run               2025-05-22T07:23:13Z
resourcegraphdefinitions.kro.run   2025-05-22T07:10:25Z

Resource Graph Definitions

Resource Graph Definitions are a very important concept in KRO. You can think of Resource Graph Definitions as a blueprint drawn by an architect for a house in real life, which specifies the specifications for each part. Anyone who gets this blueprint can build an identical house. In some flexible areas, you can even make small modifications, such as the paint color or the style of the pendant light.

Deep Dive Into Resource Graph Definitions

Let's explain with a simple Resource Graph Definition example. After KRO is installed, it will create two Custom Resource Definition (CRD) on your Kubernetes cluster: applications.kro.run and resourcegraphdefinitions.kro.run.

Lines 6 to 16: We will declare a scheme that includes some fields that can be adjusted by the user, just like the paint color or the style of the pendant light mentioned in the previous section. You can assign a default value to these fields or specify their type, such as string or integer.

Lines 17 to 44: A common Kubernetes deployment is declared here. Some field values within it will reference the values defined in the schema. Referencing values is very simple; you just need to use the $ sign and {} brackets to specify the Key location of the field inside.　

This completes a simple Resource Graph Definition. In the next section, we will gradually explain how to construct the application we want on Kubernetes using this Resource Graph Definition.

rgd.yaml

1  apiVersion: kro.run/v1alpha1
2  kind: ResourceGraphDefinition
3  metadata:
4    name: my-color-application
5  spec:
6    schema:
7      apiVersion: v1alpha1
8      kind: Application
9      spec:
10       name: string
11       image: string | default="pj3677/color-app:latest"
12       color: string | default="red"
13       commonlabel: string | default="color-application"
14     status:
15       deploymentConditions: ${deployment.status.conditions}
16       availableReplicas: ${deployment.status.availableReplicas}
17   resources:
18     - id: deployment
19       template:
20         apiVersion: apps/v1
21         kind: Deployment
22         metadata:
23           name: ${schema.spec.name} 
24           labels:
25             purpose: ${schema.spec.commonlabel}
26         spec:
27           replicas: 2
28           selector:
29             matchLabels:
30               app: ${schema.spec.name}
31           template:
32             metadata:
33               labels:
34                 app: ${schema.spec.name}
35                 purpose: ${schema.spec.commonlabel}
36             spec:
37               containers:
38                 - name: ${schema.spec.name}
39                   image: ${schema.spec.image} 
40                   ports:
41                     - containerPort: 3000
42                   env:
43                     - name: BG_COLOR
44                       value: ${schema.spec.color}

Deploy Applications with Instances

An instance represents your deployed application. When you create an instance, you're telling KRO controller "I want to create this set of resources running in my Kubernetes cluster via the Resource Graph Definition".

Below is an example of an Instance. From this example, we can see there is a field called color and specified to lightgreen.

When instance.yaml is deployed to Kubernetes, the KRO controller will receive the request and refer to the Resource Graph Definition specified in instance.yaml to generate the corresponding set of resources, which is the Application we want to construct.

instance.yaml

apiVersion: kro.run/v1alpha1
kind: Application
metadata:
  name: color-lightgreen
spec:
  name: color-lightgreen-app
  color: "lightgreen"

Demonstration

1. Deploy Resource Graph Definition

In this complete Resource Graph Definition example, we defined three fields:

An image field that defaults to using the image pj3677/color-app:latest.
A string field named color with a default value of red.
A string field named commonlabel with a default value of color-application.

This Resource Graph Definition includes both Kubernetes Deployment and Service resources. Some fields within it will reference the values defined in the schema.

resource-definition.yaml

apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata:
  name: my-color-application
spec:
  schema:
    apiVersion: v1alpha1
    kind: Application
    spec:
      name: string
      image: string | default="pj3677/color-app:latest"
      color: string | default="red"
      commonlabel: string | default="color-application"
    status:
      deploymentConditions: ${deployment.status.conditions}
      availableReplicas: ${deployment.status.availableReplicas}
  resources:
    - id: deployment
      template:
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: ${schema.spec.name} 
          labels:
            purpose: ${schema.spec.commonlabel}
        spec:
          replicas: 2
          selector:
            matchLabels:
              app: ${schema.spec.name}
          template:
            metadata:
              labels:
                app: ${schema.spec.name}
                purpose: ${schema.spec.commonlabel}
            spec:
              containers:
                - name: ${schema.spec.name}
                  image: ${schema.spec.image} 
                  ports:
                    - containerPort: 3000
                  env:
                    - name: BG_COLOR
                      value: ${schema.spec.color}
    - id: service
      template:
        apiVersion: v1
        kind: Service
        metadata:
          name: ${schema.spec.name}-service
          labels:
            purpose: ${schema.spec.commonlabel}
        spec:
          selector: ${deployment.spec.selector.matchLabels}
          ports:
            - protocol: TCP
              port: 3000
              targetPort: 3000

Let's deploy this Resource Graph Definition to our Kubernetes cluster.

kubectl apply -f resource-definition.yaml

resourcegraphdefinition.kro.run/my-color-application created

Check the Resource Graph Definition readiness.

kubectl get resourcegraphdefinition.kro.run -o wide

NAME                   APIVERSION   KIND          STATE    TOPOLOGICALORDER           AGE
my-color-application   v1alpha1     Application   Active   ["deployment","service"]   59s

You can see our my-color-application Resource Graph Definition has been deployed to Kubernetes cluster and it contains two Kubernetes resources deployment and service.

2. Deploy Instances

Next, we constructed two Instances. The purpose is to generate two different Applications using the same Resource Graph Definition.

color-lightgreen.yaml

apiVersion: kro.run/v1alpha1
kind: Application
metadata:
  name: color-lightgreen
spec:
  name: color-lightgreen-app
  color: "lightgreen"
  commonlabel: "lightgreen-application"

Deploy color-lightgreen.yaml instance.

kubectl apply -f color-lightgreen.yaml

application.kro.run/color-lightgreen created

color-skyblue.yaml

apiVersion: kro.run/v1alpha1
kind: Application
metadata:
  name: color-skyblue
spec:
  name: color-skyblue-app
  color: "skyblue"
  commonlabel: "skyblue-application"

Deploy color-skyblue instance.

kubectl apply -f color-skyblue.yaml

application.kro.run/color-skyblue created

3. Check Application Readiness

kubectl get all

NAME                                        READY   STATUS    RESTARTS   AGE
pod/color-lightgreen-app-6f58f87b75-7l5p9   1/1     Running   0          4m
pod/color-lightgreen-app-6f58f87b75-v62mf   1/1     Running   0          4m
pod/color-skyblue-app-5fdcd5fbbc-gwpw7      1/1     Running   0          10s
pod/color-skyblue-app-5fdcd5fbbc-psv2m      1/1     Running   0          10s

NAME                                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/color-lightgreen-app-service   ClusterIP   10.96.181.168   <none>        3000/TCP   3m32s
service/color-skyblue-app-service      ClusterIP   10.96.236.33    <none>        3000/TCP   1s
service/kubernetes                     ClusterIP   10.96.0.1       <none>        443/TCP    5m2s

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/color-lightgreen-app   2/2     2            2           4m
deployment.apps/color-skyblue-app      2/2     2            2           11s

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/color-lightgreen-app-6f58f87b75   2         2         2       4m
replicaset.apps/color-skyblue-app-5fdcd5fbbc      2         2         2       10s

Let's use kubectl port-forward to check the application's behavior as we expected.

Check the color-lightgreen Instance

kubectl port-forward service/color-lightgreen-app-service 3000:3000

Visit color-lightgreen via http://localhost:3000

Check the color-skyblue Instance

kubectl port-forward service/color-skyblue-app-service 3000:3000

Visit color-skyblue via http://localhost:3000

Perfect the result as we expected. The two applications have different background colors: lightgreen and skyblue.

Behavior Analyze

The reason for having different background colors is because we provided different parameters in the two instances: color: "lightgreen" and color: "skyblue".

And this parameter will be passed into this snippet within the Kubernetes Deployment defined in the Resource Graph Definition.

env:
  - name: BG_COLOR
    value: ${schema.spec.color}

Our application will generate the specified background color through this environment variable.

Cleanup

Remove color-skyblue instance:

kubectl delete -f color-skyblue.yaml

Remove color-lightgreen instance:

kubectl delete -f color-lightgreen.yaml

Remove Resource Graph Definition:

kubectl delete -f resource-definition.yaml

Uninstall KRO:

helm delete kro -n kro

Conclusion

KRO provides an easier way to write and manage Kubernetes manifests and deployments compared to Helm or Kustomization. It also enables the currently popular Infrastructure as Code architectural design.

This means that by learning just one tool, you can accomplish both Kubernetes deployment management and Infrastructure as Code.

It must be noted that KRO is currently in the Alpha stage and is not yet suitable for your production environment. You can continue to follow this project, and when it matures, it may be able to replace your current complex toolchain.

Vector: A lightweight tool for collecting EKS application logs with long-term storage capabilities

Kimi Huang — Wed, 30 Apr 2025 17:44:22 +0000

About Vector

Vector is a high-performance observability data pipeline tool developed by Datadog that puts organizations in control of their observability data. Collect, transform, and route all your logs, metrics, and traces to any vendors you want today and any other vendors you may want tomorrow. Vector enables dramatic cost reduction, novel data enrichment, and data security where you need it, not where is most convenient for your vendors. Open source and up to 10x faster than every alternative.

AWS Architecture

In this article, we present an architecture that demonstrates how to collect application logs from Amazon Elastic Kubernetes Service (Amazon EKS) via Vector, store them in Amazon Simple Storage Service (Amazon S3) for long-term retention, and finally query these logs using AWS Glue and Amazon Athena.

The architecture diagram is as below:

Let's build this Step by Step!

1. Create an S3 bucket for long-term log retention

Make sure AWS CLI is installed.

aws s3api create-bucket --bucket my-vector-demo --region ap-northeast-1

[Note]: Replace my-vector-demo with your own S3 bucket name.

2. Create IAM Policy for Vector Agent

aws iam create-policy \
    --policy-name vector-s3-access \
    --policy-document file://vector-iam-policy.json

vector-iam-policy.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "S3PutObject",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::my-vector-demo/*"
        },
        {
            "Sid": "S3ListBucket",
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::my-vector-demo"
        }
    ]
}

[Note]: Replace my-vector-demo with S3 bucket name same as Step 1.

3. Create EKS Cluster

Create ESK cluster via eksctl:

eksctl create cluster -f cluster.yaml

cluster.yaml

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: my-vector-cluster
  region: ap-northeast-1
  version: "1.32"

vpc:
  cidr: "192.168.0.0/16"

addons:
- name: vpc-cni
  attachPolicyARNs:
    - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
  resolveConflicts: overwrite
- name: eks-pod-identity-agent

iam:
  withOIDC: true
  podIdentityAssociations:
  - namespace: vector
    serviceAccountName: vector-sa
    roleName: pod-identity-role-vector
    createServiceAccount: true
    permissionPolicyARNs: ["arn:aws:iam::115789992589:policy/vector-s3-access"]

managedNodeGroups:
  - name: application-ng
    instanceType: t3a.small
    desiredCapacity: 2
    minSize: 2
    maxSize: 3
    amiFamily: Bottlerocket
    ssh:
      allow: false
    volumeSize: 20
    volumeEncrypted: true

It will take approximately 10-15 minutes for your Cluster to be created. Once it's created, you can interact with your EKS cluster using kubectl.

[Note]: Replace 115789992589 with your own 12-digit AWS Account Number.

4. Deploy a Log Generator Application

Apply YAML file to EKS cluster.

kubectl apply -f log-generator.yaml

log-generator.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: log-generator
  labels:
    app: flog
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: flog
  namespace: log-generator
spec:
  replicas: 2
  selector:
    matchLabels:
      app: flog
  template:
    metadata:
      labels:
        app: flog
    spec:
      containers:
        - name: flog
          image: mingrammer/flog:0.4.3
          args: ["-f", "json", "-d", "5s", "-t", "stdout", "-l"]
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: flog
  namespace: log-generator
spec:
  selector:
    app: flog
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: ClusterIP

This log generator application will generate a JSON format log payload every 5 seconds and print it out using stdout.

Check log generator application readiness:

kubectl get all -n log-generator

You will see two pods running in the log-generator namespace.

NAME                        READY   STATUS    RESTARTS   AGE
pod/flog-6776c79fc5-9mnqh   1/1     Running   0          10s
pod/flog-6776c79fc5-rp7sp   1/1     Running   0          10s

NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/flog   ClusterIP   10.100.217.73   <none>        80/TCP    10s

NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/flog   2/2     2            2           11s

NAME                              DESIRED   CURRENT   READY   AGE
replicaset.apps/flog-6776c79fc5   2         2         2       11s

The log generator application log payloads are as below:

{"host":"82.43.196.160", "user-identifier":"-", "datetime":"29/Apr/2025:06:15:41 +0000", "method": "GET", "request": "/expedite/target/disintermediate", "protocol":"HTTP/1.1", "status":304, "bytes":11362, "referer": "https://www.corporatedeliver.io/transform"}
{"host":"83.122.146.62", "user-identifier":"-", "datetime":"29/Apr/2025:06:15:46 +0000", "method": "GET", "request": "/ubiquitous", "protocol":"HTTP/1.0", "status":304, "bytes":8663, "referer": "https://www.dynamicvortals.com/incubate/interfaces/grow/leverage"}
{"host":"55.206.219.132", "user-identifier":"grimes1800", "datetime":"29/Apr/2025:06:15:51 +0000", "method": "POST", "request": "/strategic/global/exploit", "protocol":"HTTP/1.1", "status":504, "bytes":24205, "referer": "https://www.nationale-markets.net/mesh"}

5. Deploy Vector Agent

Make sure Helm is installed.

helm repo add vector https://helm.vector.dev
helm repo update

helm install vector vector/vector --namespace vector --create-namespace --values values.yaml

values.yaml

# Each role is created with the following workloads:
# Agent = DaemonSet
# Aggregator = StatefulSet
# Stateless-Aggregator = Deployment
role: "Agent"

serviceAccount:
  create: false
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::115789992589:role/pod-identity-role-vector
  name: vector-sa
  automountToken: true

containerPorts:
  - containerPort: 80
    name: http

service:
  enabled: true
  type: "ClusterIP"
  ports:
    - port: 80
      name: http

# customConfig -- Override Vector's default configs, if used **all** options need to be specified. This section supports
# using helm templates to populate dynamic values. See Vector's [configuration documentation](https://vector.dev/docs/reference/configuration/)
# for all options.
customConfig:
  data_dir: /vector-data-dir
  api:
    enabled: false
  sources:
    my_eks_cluster_logs:
      type: kubernetes_logs
      exclude_paths_glob_patterns:
        - "/var/log/pods/kube-system_*/**"
  transforms:
    modify:
      type: remap
      inputs:
        - my_eks_cluster_logs
      source: |
        del(.file)
        del(.kubernetes.pod_ip)
        del(.kubernetes.pod_ips)
        del(.kubernetes.pod_owner)
        del(.kubernetes.pod_uid)
        del(.kubernetes.source_type)
        del(.kubernetes.stream)
        del(.kubernetes.node_labels)
        del(.kubernetes.pod_labels)
        del(.kubernetes.pod_node_name)
        del(.kubernetes.namespace_labels)
        del(.kubernetes.container_id)
        del(.kubernetes.container_image_id)
        del(.stream)
        del(.source_type)
        .timestamp = now()
        .message = parse_json!(string!(.message))
        .message.user_identifier = .message."user-identifier"
        del(.message.bytes)
        del(.message.datetime)
        del(.message."user-identifier")
  sinks:
    s3:
      inputs:
        - modify 
      type: aws_s3
      region: ap-northeast-1
      bucket: my-vector-demo
      key_prefix: date=%Y-%m-%d/
      framing:
        method: newline_delimited
      compression: none           
      encoding:
        codec: json

Check Vector Agent readiness:

kubectl get all -n vector

You will see two pods running as daemonset in the vector namespace.

NAME               READY   STATUS    RESTARTS   AGE
pod/vector-gkd5j   1/1     Running   0          3s
pod/vector-lwj85   1/1     Running   0          3s

NAME                      TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
service/vector            ClusterIP   10.100.2.99   <none>        80/TCP    3s
service/vector-headless   ClusterIP   None          <none>        80/TCP    3s

NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/vector   2         2         2       2            2           <none>          3s

Wait around 3-5 minutes you will see your log placed in the S3 bucket.

Let's deep dive into the three parts of Helm Chart values.yaml file: customConfig.source, customConfig.transforms, and customConfig.sinks before forward to the next step.

Vector provides a lot of observability data sources:
https://vector.dev/docs/reference/configuration/sources/

We use kubernetes_logs (Line 3) data source type in this demonstration. Then ignore all of the application logs in kube-system (Line 4-5) namespace. We just need our log generator application logs.

For more detailed configurations for kubernetes_logs data source type please refer to:
https://vector.dev/docs/reference/configuration/sources/kubernetes_logs/

customConfig.source

1  sources:
2    my_eks_cluster_logs:
3      type: kubernetes_logs
4      exclude_paths_glob_patterns:
5        - "/var/log/pods/kube-system_*/**"

After completing the previous step, the log pattern you obtained will look like this:

{"file":"/var/log/pods/log-generator_flog-6776c79fc5-r9npr_5eaef748-8dfa-4159-8095-1f072bbe56f7/flog/0.log","kubernetes":{"container_id":"containerd://c9e33cff88b3ddc9d76f78afc9c1def2f90818d490d92e648e4aa72c68c52f6e","container_image":"mingrammer/flog:0.4.3","container_image_id":"docker.io/mingrammer/flog@sha256:44180f8610fab7d4c29ff233a79e19cf28bd425c1737aa59c72c1f66613fdf41","container_name":"flog","namespace_labels":{"app":"flog","kubernetes.io/metadata.name":"log-generator"},"node_labels":{"alpha.eksctl.io/cluster-name":"my-vector-cluster","alpha.eksctl.io/nodegroup-name":"application-ng","beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/instance-type":"t3a.small","beta.kubernetes.io/os":"linux","eks.amazonaws.com/capacityType":"ON_DEMAND","eks.amazonaws.com/nodegroup":"application-ng","eks.amazonaws.com/nodegroup-image":"ami-0c584e04243563b1d","eks.amazonaws.com/sourceLaunchTemplateId":"lt-0de21881415806411","eks.amazonaws.com/sourceLaunchTemplateVersion":"1","failure-domain.beta.kubernetes.io/region":"ap-northeast-1","failure-domain.beta.kubernetes.io/zone":"ap-northeast-1a","k8s.io/cloud-provider-aws":"d856828657925fb2e15850b3e197f9d7","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"ip-192-168-4-74.ap-northeast-1.compute.internal","kubernetes.io/os":"linux","node.kubernetes.io/instance-type":"t3a.small","topology.k8s.aws/zone-id":"apne1-az4","topology.kubernetes.io/region":"ap-northeast-1","topology.kubernetes.io/zone":"ap-northeast-1a"},"pod_ip":"192.168.7.43","pod_ips":["192.168.7.43"],"pod_labels":{"app":"flog","pod-template-hash":"6776c79fc5"},"pod_name":"flog-6776c79fc5-r9npr","pod_namespace":"log-generator","pod_node_name":"ip-192-168-4-74.ap-northeast-1.compute.internal","pod_owner":"ReplicaSet/flog-6776c79fc5","pod_uid":"5eaef748-8dfa-4159-8095-1f072bbe56f7"},"message":"{\"host\":\"192.53.77.232\", \"user-identifier\":\"-\", \"datetime\":\"30/Apr/2025:02:26:18 +0000\", \"method\": \"PUT\", \"request\": \"/web-readiness/networks/interfaces\", \"protocol\":\"HTTP/1.0\", \"status\":416, \"bytes\":10254, \"referer\": \"https://www.humanmatrix.info/evolve/wireless/enhance/engage\"}","source_type":"kubernetes_logs","stream":"stdout","timestamp":"2025-04-30T02:26:23.090690200Z"}

In addition to the logs generated by the log generator application itself, you will find that a lot of Kubernetes-related information (e.g., namespace_labels, instance-type, or region) has been added.
Our log generator application log has been added with a lot of \ symbols.

Here we will demonstrate how to remove some information that we don't need, remove \ symbols, and add timestamps to our logs payload.

Vector provides a feature called Vector Remap Language (VRL), which is an expression-oriented language designed for transforming observability data (logs and metrics) in a safe and performant manner.

You can use VRL Playground to test your Remap Language with your log payload.

(1) del() (Lines 7-21 and 25-27): is used to delete the Key/Value of your log payload.
(2) .timestamp = now() (Line 22): Add a timestamp to your log payload.
(3) .message = parse_json!(string!(.message)) (Line 23): Convert message string to JSON object, this action will remove the \ symbols at the same time.
(4) .message.user_identifier = .message."user-identifier" (Line 24): Situations with hyphens often require many exceptions, so we convert them to underscores for representation.

Here's also the playground for these transforms. You can experiment with it to better understand the entire Vector Remap Language transformation process.

customConfig.transforms

1   transforms:
2     modify:
3       type: remap
4       inputs:
5         - my_eks_cluster_logs
6       source: |
7         del(.file)
8         del(.kubernetes.pod_ip)
9         del(.kubernetes.pod_ips)
10        del(.kubernetes.pod_owner)
11        del(.kubernetes.pod_uid)
12        del(.kubernetes.source_type)
13        del(.kubernetes.stream)
14        del(.kubernetes.node_labels)
15        del(.kubernetes.pod_labels)
16        del(.kubernetes.pod_node_name)
17        del(.kubernetes.namespace_labels)
18        del(.kubernetes.container_id)
19        del(.kubernetes.container_image_id)
20        del(.stream)
21        del(.source_type)
22        .timestamp = now()
23        .message = parse_json!(string!(.message))
24        .message.user_identifier = .message."user-identifier"
25        del(.message.bytes)
26        del(.message.datetime)
27        del(.message."user-identifier")

The transformed result as below:

{
    "kubernetes": {
        "container_image": "mingrammer/flog:0.4.3",
        "container_name": "flog",
        "pod_name": "flog-6776c79fc5-r9npr",
        "pod_namespace": "log-generator"
    },
    "message": {
        "host": "192.53.77.232",
        "method": "PUT",
        "protocol": "HTTP/1.0",
        "referer": "https://www.humanmatrix.info/evolve/wireless/enhance/engage",
        "request": "/web-readiness/networks/interfaces",
        "status": 416,
        "user_identifier": "-"
    },
    "timestamp": "2025-04-30T02:51:38.078Z"
}

Vector provides a variety of destinations to deliver your observability data.
https://vector.dev/docs/reference/configuration/sinks/

We use AWS S3 as our destination in this demonstration.
For more detailed configurations for AWS S3 destination please refer to:
https://vector.dev/docs/reference/configuration/sinks/aws_s3/

The most important things to note are these three settings：
(1) Set destination type as aws_s3 (Line 5).
(2) Set region name and bucket name (Lines 6-7).
(3) Set encoding codec as JSON format (Line 13).

customConfig.sinks

1   sinks:
2     s3:
3       inputs:
4         - modify 
5       type: aws_s3
6       region: ap-northeast-1
7       bucket: my-vector-demo
8       key_prefix: date=%Y-%m-%d/
9       framing:
10        method: newline_delimited
11      compression: none           
12      encoding:
13        codec: json

6. Crawler logs from AWS S3 to AWS Glue - Data Catalog Tables

Move to AWS Glue console and click Add tables using crawler:

Create a database:

Move to Tables and click Add tables using crawler:

Set the crawler properties:

Add a data source:

Add a Custom classifiers:

Data source and Custom classifiers are all set, move to the next step.

Select an IAM Role:
You can click the `Create new IAM Role' button to create a new IAM Role if you don't have an IAM Role.

Set output and scheduling:

Double-check all of the configurations then click Create crawler button:

Click `Run crawler' button:

Wait for around 3-5 minutes till the Crawler job is completed.

Back to Table overview. You can see table schema has been created automatically.

7. Query log via Amazon Athena

Let's move to Amazon Athena console to query our application logs.

Copy and paste the below SQL statement to Athena SQL editor then run this query:

SELECT timestamp,
    kubernetes.container_image,
    kubernetes.container_name,
    kubernetes.pod_name,
    kubernetes.pod_namespace,
    message.host,
    message.method,
    message.protocol,
    message.referer,
    message.request,
    message.status,
    message.user_identifier,
    date
FROM "AwsDataCatalog"."eks_log_db"."my_vector_demo"
ORDER BY timestamp DESC
LIMIT 500;

When the query is completed, the 500 logs are sorted by timestamp in descending order and every JSON Key/Value also maps to an independent column:

You can further set some conditions to find the logs you want, such as the host's IP address or the request's URI.

8. Cleanup

Use eksctl to delete all of the EKS cluster stack:

eksctl delete cluster --name my-vector-cluster --region ap-northeast-1 --disable-nodegroup-eviction

9. Conclusion

In this demonstration, we showed how to:
Use Vector, an open-source tool, to collect Application Logs on Amazon Elastic Kubernetes Service (Amazon EKS).
Store the Application Logs using Amazon Simple Storage Service (Amazon S3) to meet long-term retention requirements.
Finally, analyze the Application Logs through the powerful search capabilities brought by the integration of AWS Glue and Amazon Athena.

If you are looking for a solution to collect Application Logs on Kubernetes, you might consider using Vector.