Forem: Leon Nunes

Let's deploy kgateway using vCluster

Leon Nunes — Thu, 29 Jan 2026 14:53:43 +0000

Hey folks

Been a while I've decided to once again start writing up some of the things I do, last year was a lot of travelling.

Today I'm going to try installing Vcluster and then deploy Kgateway to it

What exactly is Vcluster?

Taken straight from their docs

vCluster is an open source solution that enables teams to run virtual Kubernetes clusters inside existing infrastructure. It helps platform engineers create secure, isolated environments for development, testing, CI/CD, and even production workloads, without the cost or overhead of managing separate physical clusters.

This sounds fun, what challenges am I trying to solve for myself? Well I work with multiple customers in my day job and while I like creating multiple kind clusters for this it's not really that much fun when I need to switch contexts etc, let's see if I can achieve this with Vcluster.

Their docs are pretty straightforward, so I'm gonna use Helm

helm upgrade --install vcluster-test  cluster \
   --repo https://charts.loft.sh  \
   --namespace products \
   --repository-config='' \
   --create-namespace

Once you run this you should see

Release "vcluster-test" does not exist. Installing it now.
NAME: vcluster-test
LAST DEPLOYED: Thu Jan 29 18:52:02 2026
NAMESPACE: products
STATUS: deployed
REVISION: 1
TEST SUITE: None

Next we will see how to setup something in this namespace, to begin with let's follow the sample guide here

vcluster connect vcluster-test -n products

This shows me

19:57:18 done vCluster is up and running
Forwarding from 127.0.0.1:10078 -> 8443
Forwarding from [::1]:10078 -> 8443
Handling connection for 10078
19:57:19 done Switched active kube context to vcluster_vcluster-test_products_gke
19:57:19 warn Since you are using port-forwarding to connect, you will need to leave this terminal open
- Use CTRL+C to return to your previous kube context
- Use `kubectl get namespaces` in another terminal to access the vcluster
Handling connection for 10078

On another terminal I can now see I have new context with the following namespaces available

kubectl get namespaces
NAME              STATUS   AGE
default           Active   63m
kube-node-lease   Active   63m
kube-public       Active   63m
kube-system       Active   63m

As a user after this the docs doesn't seem to tell me where to go so I was kinda lost for a bit, there's something like deploy a sample app, seems like the notion is that I would use vCluster Platform(Not at the moment), let's go ahead with a sample deployment using helm and see what happens

I'm going to install Kgateway

What is Kgateway?

Kgateway is a control plane that implements the Kubernetes Gateway API for both microservices and AI workloads. The control plane translates your Kubernetes Gateway API resources into the configuration that the underlying data plane proxy can understand. The proxy layer is handled by kgateway’s implementation of Envoy for microservices workloads.

Let's begin with CRD's for Gateway API

$ kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml
Warning: unrecognized format "int64"
Warning: unrecognized format "int32"
customresourcedefinition.apiextensions.k8s.io/backendtlspolicies.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/gatewayclasses.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/gateways.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/grpcroutes.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/httproutes.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/referencegrants.gateway.networking.k8s.io created

Then our CRD's for Kgateway

$ helm upgrade -i --create-namespace \
--namespace kgateway-system \
--version 2.1.0 enterprise-kgateway-crds oci://us-docker.pkg.dev/solo-public/enterprise-kgateway/charts/enterprise-kgateway-crds

NAME: enterprise-kgateway-crds
LAST DEPLOYED: Thu Jan 29 20:11:51 2026
NAMESPACE: kgateway-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing the enterprise-kgateway-crds chart.

Then the Kgateway Installation

$ helm upgrade -i enterprise-kgateway oci://us-docker.pkg.dev/solo-public/enterprise-kgateway/charts/enterprise-kgateway    -n kgateway-system    --version 2.1.0    --set licensing.licenseKey=$GQ --create-namespace
Release "enterprise-kgateway" does not exist. Installing it now.
NAME: enterprise-kgateway
LAST DEPLOYED: Thu Jan 29 20:06:10 2026
NAMESPACE: kgateway-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing the enterprise-kgateway chart.

Alright we have kgateway up and running now

Lets deploy the sample app

Then we can go ahead and create the Gateway

kubectl apply -f- <<EOF
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1
metadata:
  name: http
  namespace: kgateway-system
spec:
  gatewayClassName: enterprise-kgateway
  listeners:
  - protocol: HTTP
    port: 8080
    name: http
    allowedRoutes:
      namespaces:
        from: All
EOF

And that works

kubectl get gateway http -n kgateway-system
NAME   CLASS                 ADDRESS   PROGRAMMED   AGE
http   enterprise-kgateway             True         11s

Let's expose this app on the Cluster

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httpbin
  namespace: httpbin
spec:
  parentRefs:
    - name: http
      namespace: kgateway-system
  hostnames:
    - "www.example.com"
  rules:
    - backendRefs:
        - name: httpbin
          port: 8000
EOF

And now the final part sending a request

curl -i localhost:8080/headers -H "host: www.example.com"
HTTP/1.1 200 OK
access-control-allow-credentials: true
access-control-allow-origin: *
content-type: application/json; encoding=utf-8
date: Thu, 29 Jan 2026 14:49:30 GMT
content-length: 442
x-envoy-upstream-service-time: 3
server: envoy

{
  "headers": {
    "Accept": [
      "*/*"
    ],
    "Host": [
      "www.example.com"
    ],
    "User-Agent": [
      "curl/8.18.0"
    ],
    "X-Envoy-Expected-Rq-Timeout-Ms": [
      "15000"
    ],
    "X-Envoy-External-Address": [
      "127.0.0.1"
    ],
    "X-Forwarded-For": [
      "10.124.3.10"
    ],
    "X-Forwarded-Proto": [
      "http"
    ],
    "X-Request-Id": [
      "c6575a96-59ba-4c7b-a433-beca541d6501"
    ]
  }
}

Note: I had to setup port-forwarding, ideally I think there's some settings to let the platform manage the LB IP handling but I need to check that

That's it for this post, gotta try some more fun stuff with this now, maybe ArgoCD deployments and deploying Kgateway there

My homelab keeps dying :( should I move to cloud

Leon Nunes — Thu, 30 Oct 2025 14:26:36 +0000

These past two years, I've been a digital nomad, which means most of the times I'm not at home.

Initially I had a homelab that was basically a powerhouse, and I used to run Proxmox on it, now when I was home, if anything used to go down I could quickly fix this.

Over the years, I started using my homelab to do testing for customers etc, ofcourse no customer data is used here but replicating issues was pretty fast, AWS used to take a lot of time to create a cluster, and then I had to figure out IAM etc.

But then my homelab kept going down, and I hate it now, my boxes were pretty robust and for some reason they keep going down and I can't go fix it because sometimes a simple restart doesn't fix it.

So now I'm thinking if I should move back to the cloud :( which seems fun

So over the next few weeks, I'll see if I can do something like spinning up some clusters in either some cloud envs or something like that.

Llama 4 is here, this is how you can try it!

Leon Nunes — Mon, 07 Apr 2025 02:54:07 +0000

Meta released their latest model series called LLAMA 4. It's an open-weight's model
The herd consists of

Llama Scout
Llama Maverick
Llama Behemoth

You can head over to the official release notes here

The coolest part about these models are that they go up to 10M context windows, which means that they can read more data, even entire codebases, the Gemini 2.5 Pro that released recently has a 1M context window.

You can try this for free on the OpenRouter page.

Simply head over to https://openrouter.ai/chat Create your account. Enable this option. Under your Profile => Settings => Privacy.

Then when you go to chat add a model

The context sizes of this is lower than the original model, but you can wait for that when it releases with the full context size or use OpenRouter with the paid models.

Testing out Gateway API using Gloo Gateway

Leon Nunes — Mon, 31 Mar 2025 08:02:21 +0000

Hey folks, been a while, I've been hearing a lot about the Gateway API for the past few months and it seems like the defacto for gateways now.

Since we also implemented the Gateway API I thought why not test it out!

So today I'll be testing Gloo Gateway which is an opensource API-Gateway based on the extremely performant envoy proxy. Our documentation takes you through this, but I'm going to also do something similar.

To begin with we need to make sure we have the Kubernetes Gateway CR's in our cluster.

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml

You can use the above command to install it in your cluster. I use Talos to setup k8s on my end. You can also create a sample Talos cluster using docker.

Moving on we are going to use Helm to test things out. Here we're adding the Gloo Open Source Repository.

helm repo add gloo https://storage.googleapis.com/solo-public-helm
helm repo update

Then we can install the helm chart using

helm install -n gloo-system gloo gloo/gloo \
--create-namespace \
--version 1.18.13 \
-f -<<EOF
discovery:
  enabled: false
gatewayProxies:
  gatewayProxy:
    disabled: true
gloo:
  disableLeaderElection: true
kubeGateway:
  enabled: true
EOF

You should see something like this

NAME: gloo
LAST DEPLOYED: Mon Mar 31 12:38:09 2025
NAMESPACE: gloo-system
STATUS: deployed
REVISION: 1
TEST SUITE: None

Once we have this the GatewayClass should be able to see Gloo Edge

kubectl get gatewayclass gloo-gateway

NAME           CONTROLLER             ACCEPTED   AGE
gloo-gateway   solo.io/gloo-gateway   True       25m

Now that we have this let's create a Gateway

kubectl apply -n gloo-system -f- <<EOF
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1
metadata:
  name: http
spec:
  gatewayClassName: gloo-gateway
  listeners:
  - protocol: HTTP
    port: 8080
    name: http
    allowedRoutes:
      namespaces:
        from: All
EOF

Once we apply this we can check if the gateway is now created.

I also wanted to see how this looks by default, we have quite a lot of information here.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  creationTimestamp: "2025-03-31T07:21:45Z"
  generation: 1
  name: http
  namespace: gloo-system
  resourceVersion: "75078"
  uid: 68dfe16f-12ef-4bfc-b20a-0515826721be
spec:
  gatewayClassName: gloo-gateway
  listeners:
  - allowedRoutes:
      namespaces:
        from: All
    name: http
    port: 8080
    protocol: HTTP
status:
  conditions:
  - lastTransitionTime: "2025-03-31T07:21:45Z"
    message: ""
    observedGeneration: 1
    reason: Accepted
    status: "True"
    type: Accepted
  - lastTransitionTime: "2025-03-31T07:21:45Z"
    message: ""
    observedGeneration: 1
    reason: Programmed
    status: "True"
    type: Programmed
  listeners:
  - attachedRoutes: 0
    conditions:
    - lastTransitionTime: "2025-03-31T07:21:45Z"
      message: ""
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2025-03-31T07:21:45Z"
      message: ""
      observedGeneration: 1
      reason: NoConflicts
      status: "False"
      type: Conflicted
    - lastTransitionTime: "2025-03-31T07:21:45Z"
      message: ""
      observedGeneration: 1
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    - lastTransitionTime: "2025-03-31T07:21:45Z"
      message: ""
      observedGeneration: 1
      reason: Programmed
      status: "True"
      type: Programmed
    name: http
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute

Now it's time to run some sample apps and test the gateway.

kubectl create ns httpbin
kubectl -n httpbin apply -f https://raw.githubusercontent.com/solo-io/gloo-mesh-use-cases/main/policy-demo/httpbin.yaml
kubectl -n httpbin get pods

Now in order to expose the application, we have to create a HTTPRoute resource.

By definition

HTTPRoute is a Gateway API type for specifying routing behavior of HTTP requests from a Gateway listener to an API object, i.e. Service.

So if we have a service we can route to it using the HTTPRoute

The specification of an HTTPRoute consists of:

ParentRefs- Define which Gateways this Route wants to be attached to.

Hostnames (optional)- Define a list of hostnames to use for matching the Host header of HTTP requests.

Rules- Define a list of rules to perform actions against matching HTTP requests. Each rule consists of matches, filters (optional), backendRefs (optional), timeouts (optional), and name (optional) fields.

(Taken from the official spec)

Back to our setup we can now setup the HTTPRoute

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httpbin
  namespace: httpbin
  labels:
    example: httpbin-route
spec:
  parentRefs:
    - name: http
      namespace: gloo-system
  hostnames:
    - "www.example.com"
  rules:
    - backendRefs:
        - name: httpbin
          port: 8000
EOF

We can then check how this now looks in the cluster

kubectl get -n httpbin httproute/httpbin -o yaml

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  creationTimestamp: "2025-03-31T07:44:46Z"
  generation: 1
  labels:
    example: httpbin-route
  name: httpbin
  namespace: httpbin
  resourceVersion: "76943"
  uid: e0f81d5a-7377-4bda-bae1-ce5f36031251
spec:
  hostnames:
    - www.example.com
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: http
      namespace: gloo-system
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: httpbin
          port: 8000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /
status:
  parents:
    - conditions:
        - lastTransitionTime: "2025-03-31T07:44:47Z"
          message: ""
          observedGeneration: 1
          reason: Accepted
          status: "True"
          type: Accepted
        - lastTransitionTime: "2025-03-31T07:44:47Z"
          message: ""
          observedGeneration: 1
          reason: ResolvedRefs
          status: "True"
          type: ResolvedRefs
      controllerName: solo.io/gloo-gateway
      parentRef:
        group: gateway.networking.k8s.io
        kind: Gateway
        name: http
        namespace: gloo-system

To test the gateway API we are now going to port-forward our setup.

kubectl port-forward deployment/gloo-proxy-http -n gloo-system 8080:8080
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

And this works as expected

curl -i localhost:8080/headers -H "host: www.example.com"
Handling connection for 8080
HTTP/1.1 200 OK
access-control-allow-credentials: true
access-control-allow-origin: *
content-type: application/json; encoding=utf-8
date: Mon, 31 Mar 2025 07:48:28 GMT
content-length: 331
x-envoy-upstream-service-time: 0
server: envoy

{
  "headers": {
    "Accept": [
      "*/*"
    ],
    "Host": [
      "www.example.com"
    ],
    "User-Agent": [
      "curl/8.12.1"
    ],
    "X-Envoy-Expected-Rq-Timeout-Ms": [
      "15000"
    ],
    "X-Forwarded-Proto": [
      "http"
    ],
    "X-Request-Id": [
      "44449e3f-2442-4899-8238-0c2f70e1ee59"
    ]
  }
}

Now that we have the basic setup, let's try something extra with this. I like that using Gloo Gateway means that I can route to traffic that is anywhere

Let's quickly try a static upstream.

To create a Static Upstream we can do the following

kubectl apply -f- <<EOF
apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  name: json-upstream
spec:
  static:
    hosts:
      - addr: jsonplaceholder.typicode.com
        port: 80
EOF

Then let's create a RouteOption, basically you can attach RouteOption to HTTPRoute as a Filter.

kubectl apply -f- <<EOF
apiVersion: gateway.solo.io/v1
kind: RouteOption
metadata:
  name: rewrite
  namespace: default
spec:
  options:
    hostRewrite: 'jsonplaceholder.typicode.com'
EOF

Let's also create a HTTPRoute

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: static-upstream
  namespace: default
spec:
  parentRefs:
  - name: http
    namespace: gloo-system
  hostnames:
    - static.example
  rules:
    - backendRefs:
      - name: json-upstream
        kind: Upstream
        group: gloo.solo.io
      filters:
      - type: ExtensionRef
        extensionRef:
          group: gateway.solo.io
          kind: RouteOption
          name: rewrite
EOF

Now let's check our HTTPRoute

kubectl get httproute -A
NAMESPACE   NAME              HOSTNAMES             AGE
default     static-upstream   ["static.example"]    11s
httpbin     httpbin           ["www.example.com"]   8m11s

curl -ik localhost:8080/posts -H "host: static.example:8080"
Handling connection for 8080
HTTP/1.1 200 OK
date: Mon, 31 Mar 2025 07:54:29 GMT
content-type: application/json; charset=utf-8
report-to: {"group":"heroku-nel","max_age":3600,"endpoints":[{"url":"https://nel.heroku.com/reports?ts=1743243502&sid=e11707d5-02a7-43ef-b45e-2cf4d2036f7d&s=geYoiMWFeqaCuv2HSvTjAatpMYLmT8EZc0f7Dd%2FnvDw%3D"}]}
reporting-endpoints: heroku-nel=https://nel.heroku.com/reports?ts=1743243502&sid=e11707d5-02a7-43ef-b45e-2cf4d2036f7d&s=geYoiMWFeqaCuv2HSvTjAatpMYLmT8EZc0f7Dd%2FnvDw%3D
nel: {"report_to":"heroku-nel","max_age":3600,"success_fraction":0.005,"failure_fraction":0.05,"response_headers":["Via"]}
x-powered-by: Express
x-ratelimit-limit: 1000
x-ratelimit-remaining: 999
x-ratelimit-reset: 1743243543
vary: Origin, Accept-Encoding
access-control-allow-credentials: true
cache-control: max-age=43200
pragma: no-cache
expires: -1
x-content-type-options: nosniff
etag: W/"6b80-Ybsq/K6GwwqrYkAsFxqDXGC7DoM"
via: 1.1 vegur
cf-cache-status: HIT
age: 17
server: envoy
cf-ray: 928e476ddad0424e-BOM
alt-svc: h3=":443"; ma=86400
server-timing: cfL4;desc="?proto=TCP&rtt=1834&min_rtt=1834&rtt_var=917&sent=1&recv=3&lost=0&retrans=0&sent_bytes=0&recv_bytes=213&delivery_rate=0&cwnd=249&unsent_bytes=0&cid=0000000000000000&ts=0&x=0"
x-envoy-upstream-service-time: 16
transfer-encoding: chunked

[
  {
    "userId": 1,
    "id": 1,
    "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
    "body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
  },

And with that I would like to end this post, I will be testing this more, so I'll be posting about these.

Taking a Stroll down the Hypervisor Lane.

Leon Nunes — Wed, 19 Feb 2025 14:13:16 +0000

Been a while since I've sat down and written anything, mainly because I've been travelling and doing things that aren't complete yet.

Today I want to talk about Hypervisors, I've always had a homelab and I used to self-host things on it like pihole along with DHCP for the users.

The past two years things have been slow with my homelab, while I've been trying to figure out what's the best way to spin up VM's and destroy/create them as I wish, I've not quite perfected this because I keep getting stuck. But today let's talk about Hypervisors.

The age of Kubernetes is upon us and slowly it's the age of AI too, but hypervisors have been around for a long time. Let me show y'all the one's I know of

KVM
This has been my go-to on a bare-metal server using qemu to create machines as I go and while it's quite verbose during the early days it was how I used to spin up machines.
Proxmox
This uses kvm under the hood to create VM's etc, has got good community support and is the defacto for bare-metal boxes these days, bonus points you can install whatever you like like K3s for example and still run this box as your hypervisor.

Now I'd like to talk about the one's I've been playing with recently.

Xen-Orchestra
While I'm currently reflashing this mainly because I forgot how I used to set it up and what the process was like in my lab. Currently XCP-NG is a really nice it gives you a simple UI, they recently also launched a new Go-based-Library that can be used via Terraform/OpenTofu too, which I'm eager to try out, IaC always bugs me because I cannot for the life of me complete a single deployment.
Kubevirt/Firecracker/MicroVM's
This is the final to-do list that I have, I want to get here but before that I need to get my IaC and a way to store information sorted out. This approach is a really cool approach in the sense that you can have host management now and add nodes that could be located anywhere.

Furthermore, I wonder if a service mesh added into the mix would add some more flavor to this.

I had more thoughts on this but another side-quest has caught my attention, so I'm going to pursue that for now.

How slow HDDs caused various issues in Talos and k3s

Leon Nunes — Sun, 15 Sep 2024 06:11:53 +0000

Hi there fellow people,

It's been a while since I've come back here, I keep leaving this place to start my own blog but I never go through with it. Which sucks but it's alright.

I'm still homelabbing although it's not completely where I would like it to be I'm trying.

This weekend, I tried to get my Proxmox Cluster running with Talos linux. Somewhere down the line I added HDD's in my cluster, and then went ahead and built a Ceph Cluster with it. This was the first mistake :0

Mistakes happen.

Yes, they do and it's a part of life, few days ago even my k3s cluster wasn't working correctly, while a VM I had did work exactly the way it was supposed to. It never hit me that I have HDDs because well I forgot

How'd I fix it?

Well I didn't, my friends at the Kargo Discord Server(We're building cool stuff there), pointed that the errors could be Disk I/O related. Bear in mind up until now I still was under the notion that I have SSDs.

So What are these errors you talk about?

Timeouts, Timeouts and Timeouts :D

I had containers failing with timeouts—a lot. ETCD was slow, Kube API server was slow.

E0912 18:33:33.703841       1 leaderelection.go:369] Failed to update lock: Put "https://127.0.0.1:7445/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler?timeout=5s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
E0912 18:33:38.701599       1 leaderelection.go:369] Failed to update lock: Put "https://127.0.0.1:7445/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler?timeout=5s": context deadline exceeded
I0912 18:33:38.701663       1 leaderelection.go:285] failed to renew lease kube-system/kube-scheduler: timed out waiting for the condition
E0912 18:33:42.475565       1 leaderelection.go:308] Failed to release lock: Operation cannot be fulfilled on leases.coordination.k8s.io "kube-scheduler": the object has been modified; please apply your changes to the latest version and try again
E0912 18:33:42.475595       1 server.go:242] "Leaderelection lost"

The Kube-scheduler was dying too on Talos

E0913 08:34:02.257514       1 leaderelection.go:332] error retrieving resource lock kube-system/kube-scheduler: Get "https://127.0.0.1:7445/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler?timeout=5s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)                                                                                                                      
E0913 08:34:07.256320       1 leaderelection.go:332] error retrieving resource lock kube-system/kube-scheduler: Get "https://127.0.0.1:7445/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler?timeout=5s": context deadline exceeded                                                                                                                                                                        
I0913 08:34:07.256364       1 leaderelection.go:285] failed to renew lease kube-system/kube-scheduler: timed out waiting for the condition                                                                        
E0913 08:34:09.375301       1 server.go:242] "Leaderelection lost"

Looking at the Proxmox stats nothing stood out either

I checked with dd too.

dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.60134 s, 671 MB/s

/ # dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=direct
1+0 records in
1+0 records out
1073741824 bytes (1.0GB) copied, 0.739437 seconds, 1.4GB/s
/ # dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=append
1+0 records in
1+0 records out
1073741824 bytes (1.0GB) copied, 0.487405 seconds, 2.1GB/s

So what helped?

Well the fio tool is what helped me ultimately, but before that it was apt when installing some packages.

So with Talos, you can do something like:

kubectl debug -n kube-system -it --image debian node/$NODE

This is where I figured out okay something is really SLOOOOW

The fio tool told me the tests will take 2 Hours, when the same on my laptop was within 10 seconds.

Apt was taking 20 minutes to install a package.

So all in all yes slow disks, can cause all sorts of problems. Thank you for reading.

Until Next time, if you'd like to talk more I'm @mediocredevops on Twitter.

References:
Excellent Article here on Etcd and Fio

Crowdstrike brings down Windows PC's

Leon Nunes — Fri, 19 Jul 2024 08:57:46 +0000

From the trenches of Twitter, I found out that windows has once again faced a global outage causing many systems to restart and enter a BSOD(Blue screen of Death).

Many tweets have surfaced, which talk about multiple companies including Airlines going down.

The fix given here isn't very beginner friendly and requires BIOS access which may not be something that is readily available.

So if your company uses Crowdstrike and windows you basically have an off-day but not the IT folks :)

It's a long weekend ahead, to get these resolved if Crowdstrike doesn't do it automatically.

Streaming made simple

Leon Nunes — Sat, 17 Feb 2024 13:05:40 +0000

Hi there,

It's been a while(I don't know why this has started as an email). I've been busy with just life in general, but I'd like to resurrect this blog by adding a post about my streaming setup for meetups.

In February 2024, I had my first hardware meetup.

We have a small community on Telegram.

Side quests aside, this post is about my streaming setup.

The setup!

Android Phone
Linux Laptop/Windows(Hopefully Mac does too)
IP webcam
ADB installed.
OBS for streaming.
Stream destination(Youtube/Peertube/Twitch)
Tripod
Audio, either use a mic or just the sounds from the environment.

I like keeping things simple, yes this doesn't auto-track your speaker walking but that's okay!

This is how it all looks when it's setup.

The way I've setup this is from a reddit post that said that if you have adb installed you can do something like this once you have IP webcam on.

adb forward tcp:8081 tcp:8080

This eliminates the need for having your mobile connected to any network.

But how does this work?

It's dead simple, first we start the IP webcam, then we do a adb port-forward. This sets up the android phone.

Next, we do a very simple thing, in OBS we can add this as a media source, give the IP http://localhost:8081/video

Once we have the media source, the next thing is to add the slides.

This part is challenging, if your speaker has a way to connect directly with you via things like Zoom, Google Meet, this is the simplest way to do things.

In the end you have a simple setup that can help you get started streaming.

This setup is also handy if you have a bad webcam or your laptop webcam doesn't look good, most android phones are capable of better image quality.

While this setup is a bit janky it does the job.

Until next time!

Connect with me on Twitter, Linkedin

How a simple Hang-up borked my lab environment.

Leon Nunes — Thu, 05 Jan 2023 18:41:15 +0000

Happy New Year! Well my new year hasn't been that happy, from falling sick on the first day of the year to my homelab giving up, life has just been going on...

Upgrading never seemed risky until one day my Arch homelab(Yes I run Arch :o) hung up and caused all the kernels to get deleted. started getting funky GPG and corrupted package errors.

It all began here...

So I popped in a usb and started a live environment, mounted my folders and did arch-chroot, then tried to do a pacman -Syu only to get hit with a bunch of:

error: GPGME error: No data

So I thought let me just simply fix it by following the Arch guide, however, nothing worked, what did was

sudo rm -fr /etc/pacman.d/gnupg
sudo pacman-key --init
sudo pacman-key --populate archlinux manjaro
sudo pacman-key --refresh-keys

I also got few errors of files being already owned, since I was sleepy I did some bash-fu and ended up with no /usr :D, fret not I copied it from my live system(Spoiler Alert: It borked up systemD). It worked until it didn't.

So then I figured out somehow that SystemD and QEMU was failing in some ways with Permission Denied errors and Qemu monitor getting a similar error, a quick reinstall fixed it...

Weirdly this was a chain reaction... I booted up my PC and I kept getting an error that said I've overclocked it... but I never did that, this caused the bios to eventually wipe everything on every boot. Which means I couldn't run VMs because the Virtualization option never persisted.

So I did the most sensible thing I replaced the CMOS battery, cause this is a really old system. Even then it still didn't work. Then I noticed that the Ram was showing up as 8GB, this system had 16GB ram and so it turned out that a faulty Memory stick brought my system down for 2 good days...

Surprisingly, DDR3 is quite cheap so I'll be replacing that Ram Stick.

That's it folks, I also got a new box to play around these days and I'm wondering what OS should I put on it any suggestions!?

Public speaking is hard and scary but it's also exciting!

Leon Nunes — Sun, 31 Jul 2022 13:50:41 +0000

If you've seen me on social media, almost after few days I ask people on how they find events, and I get no responses...

Many a times I come across an event and then the Call for Proposals(CFP) has already passed, I have a lot of anxiety when it comes to Public Speaking and I get quite nervous, but I recently submitted a CFP in more than 5 places and got selected in two one was Hashicorp's Hashi Talks, the other OpenInfra and CNCF Asia.

The blunder!

I missed the HashiCorp Deadline because I never saw the email, Google Mail blunder. Luckily the HashiCorp team was okay with that but then it hit me, I didn't actually have any experience in recording talks so the guidelines they suggested were the only things that could help me and I didn't have any slides!

To be honest it was a very taxing weekend, because figuring out OBS studio was not very challenging but the equipment I had made it challenging!

Microphones are hard

I thought I have a good webcam by Lenovo, which comes with dual mics. But for some reason these didn't work that great with OBS studio and I didn't have time to wait for a collar mic.
So I had to use EasyEffects and somehow make the audio quality a bit good, didn't happen, I could still feel the choppiness.

My attention span is as short as a bulb.

Truth be told, I don't really have a great attention span, so I wrote down the points to talk about, which I kept forgetting because I can't follow word to word without sounding like a robot.
I never fixed this, it's quite present during the video.

My Anxiety and nervousness almost made me run away

The moment I saw that I was accepted the immediate thought was to run away in the other direction! WHY? because I was scared, the thought of me recording a video was something I can never imagine, people could've hated it and literally I would've been outed as a fraud, so many thoughts in my head but I was like whatever happens happens.

Why am I writing this?

Good question, I don't know either, somewhere I know there are people like me who are struggling and I want them to know that once you push through it things start to change. This is also a reminder to me of the things I have attempted so yay, here's to more learning and doing new and anxiety driven things.

Websites I used to find these Events.

Ease your .tf debugging using Terraform console

Leon Nunes — Tue, 31 May 2022 19:40:28 +0000

Terraform is fun, it gives you the ability to quickly write code to deploy your infrastructure, but I've always found debugging terraform loops to be a little bit hard, this could be because I like to see what the result will be before hand and sometimes that's just not possible.

So let's jump in!

To begin, go to your terraform code.

$ terraform console

For starters we can view the data sent over from a data source

> data.terraform_remote_state.network.outputs.public_subnets
[
  "subnet-03d91cba453ff08b1",
  "subnet-07f430834d2138353",
  "subnet-096e710748bab5eff",
]

Now if you want to create a loop out of something you can do

> [for a in data.terraform_remote_state.network.outputs.public_subnets:upper(a)]

[
  "SUBNET-04D91CBA353F108B1",
  "SUBNET-07F030831D2358353",
  "SUBNET-096E710758CAB7EFF",
]

One can also create structures to check the output for example

> {"${module.vpc.vpc_id}":[{"public":"${module.vpc.public_subnets}","private":"${module.vpc.private_subnets}"}]}
{
  "vpc-03cf041fb057a54f7" = [
    {
      "private" = [
        "subnet-07fcca0e214a4a439",
        "subnet-056496be722ef56d2",
        "subnet-04755007e91fe1e20",
      ]
      "public" = [
        "subnet-04d92cba4432f08b1",
        "subnet-07f030534d2c58353",
        "subnet-096e710648cab7eff",
      ]
    },
  ]
}

That's it, you can also loop over data.

Some things such as accessing outputs is still not supported.

If you liked this article I've also written one on structuring Terraform code.

For more information, read the docs and follow the guide here

I can also be found on Twitter, Linkedin.

Thank you for reading!

Using Prowler to Audit your AWS account for vulnerabilities.

Leon Nunes — Thu, 05 May 2022 15:35:58 +0000

Few days ago I came across this repository and I found Prowler(Go Star the repo).

So what is Prowler?

Prowler is an Open Source security tool to perform AWS security best practices assessments, audits, incident response, continuous monitoring, hardening and forensics readiness. It contains more than 200 controls covering CIS, PCI-DSS, ISO27001, GDPR, HIPAA, FFIEC, SOC2, AWS FTR, ENS and custom security frameworks.

I was pretty amazed by the tool, the first time I ran it using podman I did have some issues because I had containerized aws-cli too, so I had to use environment variables and I also had to map the user as such

podman run --rm -it --user 0:0  --env AWS_ACCESS_KEY_ID=DEMO --env AWS_SECRET_ACCESS_KEY=Demo  -v $(pwd)/prowler_output:/prowler/output:z prowler  -f ap-south-1  -r ap-south-1  -M html,csv

The reporting part is also pretty cool, Prowler took about 35 minutes to run almost 217 checks in a single region.

With Prowler there are a lot of things that can be further checked. I was able to see instances that had hardcoded secrets, there are checks done on Bucket Policy, IAM access, VPC routes and a lot more.

Apart from this, it also, gives you the instance ID, and AWS support articles with possible fixes and remediation column to provide the details, you can output it to CSV and run filters to further get detailed info. It will also tell you why it is bad when it finds similar things. Prowler also guides you to enable additional security stuff.

I'd definitely recommend this opensource tool to audit your AWS account to fix security issues.

As always, if you find these articles interesting please leave a like.
I'm also available for freelance Devops roles, my LinkedIn is in the Dev.to bio.

For further discussions, please reach out @mediocreDevops

Thank you for reading!