Forem: Tunde Oladipupo

Terraform and AWS

Tunde Oladipupo — Sat, 31 Dec 2022 15:28:39 +0000

This is tunde

Randomness in Go

Tunde Oladipupo — Sat, 29 Aug 2020 04:08:46 +0000

This is to discuss ways to generate random values in go using the rand package or the crypto rand. They both achieve the same end depending on your need regarding true randomness and performance. Lets start with the common math/rand package

package main

import (
    "fmt"
    "math/rand"
)

func main() {
    for i := 0; i < 5; i++ {
        fmt.Println(rand.Intn(50))
    }
}

$ go run gotest.go 
31
37
47
9
31
$ go run gotest.go 
31
37
47
9
31

As you can see from above, the run produces the same sequence because the default seeding source.Now, lets try to use a different seeding for our random sequence generation.

package main

import (
    "fmt"
    "math/rand"
    "time"
)

func main() {
    r := rand.New(rand.NewSource(time.Now().UnixNano()))
    for i := 0; i < 5; i++ {
        fmt.Println(r.Intn(50))
    }
}

// Random numbers are generated by a Source. Top-level functions, such as
// Float64 and Int, use a default shared Source that produces a deterministic
// sequence of values each time a program is run. Use the Seed function to
// initialize the default Source if different behavior is required for each run.
// The default Source is safe for concurrent use by multiple goroutines, but
// Sources created by NewSource are not.

What we simply did was provide new seeding via new source. Please note this source is not safe for concurrent use unlike the shared source used globally.

$ go run gotest.go 
44
17
41
23
5
$ go run gotest.go 
18
7
37
17
13

Now to be truly random, you can use the better crypto/rand but with less performance.

Crypto Rand

package main

import (
    crypto_rand "crypto/rand"
    "encoding/base64"
    "fmt"
)

func GenerateRandomBytes(n int) ([]byte, error) {
    b := make([]byte, n)
    if _, err := crypto_rand.Read(b); err != nil {
        return nil, err
    }
    return b, nil
}

func GenerateRandomString(n int) (string, error) {
    b, err := GenerateRandomBytes(n)
    if err != nil {
        return "", err
    }
    return base64.URLEncoding.EncodeToString(b), nil
}
func main() {
    for i := 30; i < 35; i++ {
        if s, err := GenerateRandomString(i); err == nil {
            fmt.Println(s)
        }
    }
}

What we doing here is very simple, we are not using clock to generate our seedling.

What on earth is Go Mod?

Tunde Oladipupo — Wed, 19 Feb 2020 16:04:57 +0000

Go mod is the recommended way to manage packages and their dependencies in go. Go can look at your imported packages and add them to its go.mod for management. For example, in this project;

module knative.dev/client

require (
    contrib.go.opencensus.io/exporter/ocagent v0.6.0 // indirect
    contrib.go.opencensus.io/exporter/prometheus v0.1.0 // indirect
    contrib.go.opencensus.io/exporter/stackdriver v0.12.9 // indirect
    github.com/google/go-containerregistry v0.0.0-20200131185320-aec8da010de2 // indirect
...
    github.com/openzipkin/zipkin-go v0.2.2 // indirect
    github.com/pkg/errors v0.8.1
    github.com/robfig/cron v1.2.0 // indirect
...
    golang.org/x/crypto v0.0.0-20191206172530-e9b2fee46413
    gotest.tools v2.2.0+incompatible
...
    sigs.k8s.io/yaml v1.1.0
)

// Temporary pinning certain libraries. Please check periodically, whether these are still needed
// ----------------------------------------------------------------------------------------------

// Fix for `[` in help messages and shell completion code
// See https://github.com/spf13/cobra/pull/899
replace github.com/spf13/cobra => github.com/chmouel/cobra v0.0.0-20191021105835-a78788917390

go 1.13

The go.mod has the required modules for this module as well as their specific version. Talking about version, you can see we have different version as well as comment. Lets take a look;

Semvars
```
github.com/pkg/errors v0.8.1
```
This package is required but version v0.8.1 of it. If this module is updated in its remote git repo with the new version, go will not used it but v0.8.1 instead
Pseudo-versions
```
    knative.dev/eventing v0.12.1-0.20200206203632-b0a7d8a77cc7
```
This is the same as semvar but with weird version generated by go in form of last_tag-date-commit_hash. This is done when you import a package without specifying the version to use. Go simply update with the last tagged version(v0.12.1 in the case of knative.dev/eventing), then calculate the time of last commit as well as hash of last commit in the git repo to generate its version tag to give us v0.12.1-0.20200206203632-b0a7d8a77cc7

replace

So you have an external dependency in your project that you want to make some local changes to or replace temporarily with another source for testing or specific purpose. How do you do that? You can use replace. This means instead of using the external dependency source, it will use the replacement instead.

replace github.com/spf13/cobra => github.com/chmouel/cobra v0.0.0-20191021105835-a78788917390

This will make go mod use github.com/chmouel/cobra anywhere github.com/spf13/cobra is used. You can do the same locally by pointing the replacement to local path.

incompatible

Let's talk incompatible. If a repo does not have go.mod and it has semver that is >= 2,it will tagged with the semver as well as incompatible to indicate this is not a go mod repo. For our case,

gotest.tools v2.2.0+incompatible

If you look at the tagged repo, you can see the repo is missing go.mod and its on >=2 tag.

go version directive

This simply answers the question of which golang version and features should I use? I will quote one of the authority on this for better explanation.

The basic guideline is fairly simple: a "go" directive 1.N means that the code in that module is permitted to use language features that existed in 1.N even if those features were removed in later releases of the language.

In other words, if the code compiles successfully at version 1.N, then it will continue to compile for all later versions of the language, even if it happens to use language features that were later removed.

To a first approximation, nobody should ever have to worry about the "go" directive. The only likely time you might need to set it manually is if you are copying some existing code to a new module, that code uses some obsolete language feature, and you don't have time to rewrite it. In that case you might want to set the "go" directive (using go mod edit -go) to the version used by the original module, thus permitting your new module to use the obsolete features. When writing new code you will presumably simply avoid the obsolete language features.

go sum

This go.sum is way for go to keep cryptographic hash of the modules version in go.mod. It keeps a cache copy of this hashes and verifies them during run. Make sure to check in the go.sum file.

golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c h1:qgOY6WgZOaTkIIMiVjBQcw93ERBE4m30iBm00nkL0i8=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
rsc.io/quote v1.5.2 h1:w5fcysjrx7yqtD/aO+QwRjYZOKnaM9Uh2b40tElTs3Y=
rsc.io/quote v1.5.2/go.mod h1:LzX7hefJvL54yjefDEDHNONDjII0t9xZLPXsUe+TKr0=
rsc.io/sampler v1.3.0 h1:7uVkIFmeBqHfdjD+gZwtXXI+RODJ2Wc4O7MPEh/QiW4=
rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA=

Go dependency management and dep

Tunde Oladipupo — Thu, 13 Feb 2020 20:25:38 +0000

You may be aware of the "official experiment" for go dependency management tool called dep but this has since be replaced by go mod. Let's explore the tool via this opensource project usage. Gopkg.toml is where you define your package dependencies while Gopkg.lock contains snapshot of your project dependencies after evaluating Gopkg.toml as well as some metadata. Here is;

# Refer to https://github.com/golang/dep/blob/master/docs/Gopkg.toml.md
# for detailed Gopkg.toml documentation.

required = [
  ...
  "knative.dev/pkg/codegen/cmd/injection-gen",
  # TODO(#4549): Drop this when we drop our patches.
  "k8s.io/kubernetes/pkg/version",
  "knative.dev/caching/pkg/apis/caching",
  # For cluster management in performance testing.
  "knative.dev/pkg/testutils/clustermanager/perf-tests",
  "knative.dev/test-infra/scripts",
  "knative.dev/test-infra/tools/dep-collector",
  # For load testing.
  "github.com/tsenart/vegeta"
]

[[constraint]]
  name = "github.com/tsenart/vegeta"
  branch = "master"

[[override]]
  name = "gopkg.in/yaml.v2"
  version = "v2.2.4"
...
[[override]]
  name = "github.com/google/mako"
  version = "v0.1.0"

[[override]]
  name = "go.uber.org/zap"
  revision = "67bc79d13d155c02fd008f721863ff8cc5f30659"
...
[[constraint]]
  name = "github.com/jetstack/cert-manager"
  version = "v0.12.0"
...
[[override]]
  name = "k8s.io/api"
  version = "kubernetes-1.16.4"
...
[[override]]
  name = "k8s.io/kube-openapi"
  # This is the version at which k8s.io/apiserver depends on this at its 1.16.4 tag.
  revision = "743ec37842bffe49dd4221d9026f30fb1d5adbc4"
...

# Added for the custom-metrics-apiserver specifically
[[override]]
  name = "github.com/kubernetes-incubator/custom-metrics-apiserver"
  revision = "3d9be26a50eb64531fc40eb31a5f3e6720956dc6"

[[override]]
  name = "bitbucket.org/ww/goautoneg"
  source = "github.com/munnerz/goautoneg"

[prune]
  go-tests = true
  unused-packages = true
  non-go = true

[[prune.project]]
  name = "k8s.io/code-generator"
  unused-packages = false
  non-go = false

[[prune.project]]
  name = "knative.dev/test-infra"
  non-go = false

...

# The dependencies below are required for opencensus.
[[override]]
  name = "google.golang.org/genproto"
  revision = "357c62f0e4bbba7e6cc403ae09edcf3e2b9028fe"

[[override]]
  name = "contrib.go.opencensus.io/exporter/prometheus"
  version = "0.1.0"

[[override]]
  name = "contrib.go.opencensus.io/exporter/zipkin"
  version = "0.1.1"

[[constraint]]
  name = "go.opencensus.io"
  version = "0.22.0"

[[override]]
  name = "github.com/census-instrumentation/opencensus-proto"
  version = "0.2.0"

[[override]]
  name="github.com/golang/protobuf"
  version = "1.3.2"

required

Lets talk about this required section; this is where you define what dependencies are required and must be included in the vendor folder;

required = [
  ...
  "knative.dev/pkg/codegen/cmd/injection-gen",
  # TODO(#4549): Drop this when we drop our patches.
  "k8s.io/kubernetes/pkg/version",
  "knative.dev/caching/pkg/apis/caching",
  # For cluster management in performance testing.
  "knative.dev/pkg/testutils/clustermanager/perf-tests",
  "knative.dev/test-infra/scripts",
  "knative.dev/test-infra/tools/dep-collector",
  # For load testing.
  "github.com/tsenart/vegeta"
]

If you look at the folder, you will see these dependencies in the folder as these are required.

direct and transitive dependency

A -> B -> C -> D

These are packages that your project imports or includes the required sections. For example above, A directly import B while B imports C and D. In this case, B is the direct dependent of A while C and D are transitive dependency.

constraint

This is how you specify the version of your dependency to use for this project. In our case, we want our project to use versionv2.2.4 of the gopkg.in/yaml.v2. To do this,

[[override]]
  name = "gopkg.in/yaml.v2"
  version = "v2.2.4"

You can also use branch or revision to pin your dependency.

override

These are like global constraint but supersedes constraints and should be used as last resort. They apply to direct dependencies and transitive dependencies unlike constraint and advisably should be used sparingly.

[[override]]
  name="github.com/golang/protobuf"
  version = "1.3.2"

prune

When your project has a dependency, it extracts the package along with other files like README.md, LEGAL etc. If you dont want any or all of these files, you can use prune to inform dep.

[prune]
  go-tests = true
  unused-packages = true
  non-go = true

[[prune.project]]
  name = "k8s.io/code-generator"
  unused-packages = false
  non-go = false

[[prune.project]]
  name = "knative.dev/test-infra"
  non-go = false

The setting above simply tells dep to remove test files, unused-packages as well as non-go files like LEGAL from all dependencies. This is the overriden with a different setting for k8s.io/code-generator that unused-packages and no-go should not be pruned.

dep ensure

Once you have configured your Gopkg.toml with your settings, you need to apply these changes, update vendor folder as well as Gopkg.lock. dep ensure will do that for you and you have option to tell dep not to update vendor, just Gopkg.lock.

Reliability: ETCD Cluster

Tunde Oladipupo — Tue, 11 Feb 2020 22:18:38 +0000

ETCD

Always benchmark your external etcd cluster using this tool
Give etcd a higher disk priority if running along other processes.

Do

Take advantage of learners for re-assignement and during upgrade. Checkout upcoming features in 3.5
Election timeout should be 10x heart beat interval to makeup for network latency issues.
You can only do minor upgrade for now.
Use SSD for disk as this is the most critical resource in etcd cluster.
You can change the default storage limit from 2GB to max of 8GB if using large cluster. checkout

Dont Do

Be careful when adding new node to etcd cluster that cannot tolerate any node failure. This is because if the new node fails, quorum is not achieved and etcd goes into read-mode. You can remove failing node before adding the new one.
Do not route request to learner especially when using load balancer.
You risk cluster stability if you set different heartbeat and election timeout for the members.

Backup and Recovery

Backup

etcdctl snapshot save backup.db --cacert="/var/lib/minikube/certs/etcd/ca.crt" --cert="/var/lib/minikube/certs/etcd/server.crt" --key="/var/lib/minikube/certs/etcd/server.key" 
{"level":"warn","ts":"2020-02-10T19:00:17.325Z","caller":"clientv3/retry_interceptor.go:116","msg":"retry stream intercept"}
Snapshot saved at backup.db

Make sure you do this periodically probably using an automated method thats saves the backup in remote storage like s3.

Restore

Remember that a snapshot restore creates a new logical member, making the node forgets its former identity. Hence you need to restore on other nodes and restart the cluster

Upgrade and Downgrade

Before you do upgrade or downgrade, master the act of etcd backup and automated recovery unless though wishes to see what Dante was talking about.

Upgrade

Replace the cluster members one by one to avoid potential issues with upgrade. If you have any member that is unhealthy, you can remove(provided you are not violating quorum) and add the member back to the cluster.

Downgrade

You should probably read this post and this before doing a downgrade.

Monitoring

Etcd exports metrics at /metrics endpoint, so you can configure prometheus to ingest data from that endpoint for display in grafana.

GitOps

ETCD Cluster and Non-voting Learners

Tunde Oladipupo — Tue, 11 Feb 2020 22:15:19 +0000

In 3.4, etcd introduced raft learner for those looking to add new members to the cluster without giving them voting right. These are particularly useful because we can add members to the cluster without causing disruption then promote them to voting members once everything is good to go. Check this out

Steps to add a learner to the cluster;

Add the learner to existing cluster as below;

etcdctl member add etcd-node-3 --peer-urls="http://192.168.13.10:2380" --learner
Member 14d2e95e16c995c7 added to cluster 7ef1685daf3d8f18

ETCD_NAME="etcd-node-3"
ETCD_INITIAL_CLUSTER="etcd-node-3=http://192.168.13.10:2380,etcd-node-1=http://192.168.11.10:2380,etcd-node-0=http://192.168.10.10:2380,etcd-node-2=http://192.168.12.10:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.13.10:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

Once add, you can check this status;

# etcdctl member list -w table 
+------------------+---------+-------------+---------------------------+---------------------------+------------+
|        ID        | STATUS  |    NAME     |        PEER ADDRS         |       CLIENT ADDRS        | IS LEARNER |
+------------------+---------+-------------+---------------------------+---------------------------+------------+
| 14d2e95e16c995c7 | started | etcd-node-3 | http://192.168.13.10:2380 | http://192.168.13.10:2379 |       true |
| 4c3c38d6652b5d75 | started | etcd-node-1 | http://192.168.11.10:2380 | http://192.168.11.10:2379 |      false |
| a00eaaf5194c573d | started | etcd-node-0 | http://192.168.10.10:2380 | http://192.168.10.10:2379 |      false |
| fdd13ca43538c5a2 | started | etcd-node-2 | http://192.168.12.10:2380 | http://192.168.12.10:2379 |      false |
+------------------+---------+-------------+---------------------------+---------------------------+------------+

You can see our new member is a learner, meaning it syncs with the cluster but does not have voting right or accept read or write request but it is not started yet.

etcdctl member list
16f23207f1b6cfc0, unstarted, , http://192.168.13.10:2379, , true
4c3c38d6652b5d75, started, etcd-node-1, http://192.168.11.10:2380, http://192.168.11.10:2379, false
a00eaaf5194c573d, started, etcd-node-0, http://192.168.10.10:2380, http://192.168.10.10:2379, false
fdd13ca43538c5a2, started, etcd-node-2, http://192.168.12.10:2380, http://192.168.12.10:2379, false

You can now start the learner with the outputted configuration.

 /usr/local/bin/etcd \
  --data-dir=/etcd-data --name etcd-node-3 \
  --initial-advertise-peer-urls http://192.168.13.10}:2380 --listen-peer-urls http://0.0.0.0:2380 \
  --advertise-client-urls http://192.168.13.10:2379 --listen-client-urls http://0.0.0.0:2379 \
  --initial-cluster "etcd-node-3=http://192.168.13.10:2380,etcd-node-1=http://192.168.11.10:2380,etcd-node-0=http://192.168.10.10:2380,etcd-node-2=http://192.168.12.10:2380" \
  --initial-cluster-state existing \
  --initial-cluster-token my-etcd-token

Now, it should show the learner as started;

# etcdctl member list -w table
+------------------+---------+-------------+---------------------------+---------------------------+------------+
|        ID        | STATUS  |    NAME     |        PEER ADDRS         |       CLIENT ADDRS        | IS LEARNER |
+------------------+---------+-------------+---------------------------+---------------------------+------------+
| 14d2e95e16c995c7 | started | etcd-node-3 | http://192.168.13.10:2380 | http://192.168.13.10:2379 |       true |
| 4c3c38d6652b5d75 | started | etcd-node-1 | http://192.168.11.10:2380 | http://192.168.11.10:2379 |      false |
| a00eaaf5194c573d | started | etcd-node-0 | http://192.168.10.10:2380 | http://192.168.10.10:2379 |      false |
| fdd13ca43538c5a2 | started | etcd-node-2 | http://192.168.12.10:2380 | http://192.168.12.10:2379 |      false |
+------------------+---------+-------------+---------------------------+---------------------------+------------+

This is where learner is of importance; if any of our voting members goes down and our cluster is in inconsistent quorum state. You can promote the learner to replace the unstable voting member.

etcdctl member promote 14d2e95e16c995c7
Member 14d2e95e16c995c7 promoted in cluster 7ef1685daf3d8f18

Downgrading ETCD Cluster

Tunde Oladipupo — Tue, 11 Feb 2020 08:56:11 +0000

This process describes how you can downgrade etcd cluster. If you follow my post earlier, I was able to setup a 3-node cluster using script. In this post, I will be using script to initiate the snapshot, downgrade and restart. If you try to downgrade the cluster without doing restore, you will get something along

...
2020-02-11 08:40:47.532292 I | rafthttp: added peer a00eaaf5194c573d
2020-02-11 08:40:47.532577 I | etcdserver/membership: added member fdd13ca43538c5a2 [http://192.168.12.10:2380] to cluster 7ef1685daf3d8f18
2020-02-11 08:40:47.533041 N | etcdserver/membership: set the initial cluster version to 3.0
2020-02-11 08:40:47.533121 I | etcdserver/api: enabled capabilities for version 3.0
2020-02-11 08:40:47.533524 N | etcdserver/membership: updated the cluster version from 3.0 to 3.2
2020-02-11 08:40:47.533747 I | etcdserver/api: enabled capabilities for version 3.2
2020-02-11 08:40:47.534324 N | etcdserver/membership: updated the cluster version from 3.2 to 3.3
2020-02-11 08:40:47.534417 C | etcdserver/membership: cluster cannot be downgraded (current version: 3.2.28 is lower than determined cluster version: 3.3).

First, take a snapshot of the cluster;

 # etcdctl snapshot save /etcd-data/snapshot.db
Snapshot saved at /etcd-data/snapshot.db

Copy this snapshot to similar location on other nodes. Once this is done, let us tell etcd to do a restore from this snapshot into a different directory(in case of failure). I changed my data-dir to /etcd-data/new for demo purpose. Use the script below

REGISTRY=quay.io/coreos/etcd
# available from v3.2.5
# REGISTRY=gcr.io/etcd-development/etcd

# For each machine
ETCD_VERSION=v3.2.28
TOKEN=my-etcd-token
CLUSTER_STATE=new
NAME_1=etcd-node-0
NAME_2=etcd-node-1
NAME_3=etcd-node-2
HOST_1=192.168.10.10
HOST_2=192.168.11.10
HOST_3=192.168.12.10
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380
DATA_DIR=/mydatadir/etcd #make sure to change this

# For node 1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
docker run \
  -e ETCDCTL_API=3 \
  -p ${THIS_IP}:2379:2379 \
  -p ${THIS_IP}:2380:2380 \
  --volume=${DATA_DIR}/${THIS_NAME}:/etcd-data \
  --name ${THIS_NAME} ${REGISTRY}:${ETCD_VERSION} \
  /usr/local/bin/etcdctl snapshot restore /etcd-data/snapshot.db \
  --name ${THIS_NAME} \
  --initial-cluster ${CLUSTER} \
  --initial-cluster-token ${TOKEN} \
  --initial-advertise-peer-urls http://${THIS_IP}:2380 \
  --data-dir /etcd-data/new 




# For node 2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
docker run \
  -e ETCDCTL_API=3 \
  -p ${THIS_IP}:2379:2379 \
  -p ${THIS_IP}:2380:2380 \
  --volume=${DATA_DIR}/${THIS_NAME}:/etcd-data \
  --name ${THIS_NAME} ${REGISTRY}:${ETCD_VERSION} \
  /usr/local/bin/etcdctl snapshot restore /etcd-data/snapshot.db \
  --name ${THIS_NAME} \
  --initial-cluster ${CLUSTER} \
  --initial-cluster-token ${TOKEN} \
  --initial-advertise-peer-urls http://${THIS_IP}:2380 \
  --data-dir /etcd-data/new 

# # For node 3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
docker run \
  -e ETCDCTL_API=3 \
  -p ${THIS_IP}:2379:2379 \
  -p ${THIS_IP}:2380:2380 \
  --volume=${DATA_DIR}/${THIS_NAME}:/etcd-data \
  --name ${THIS_NAME} ${REGISTRY}:${ETCD_VERSION} \
  /usr/local/bin/etcdctl snapshot restore /etcd-data/snapshot.db \
  --name ${THIS_NAME} \
  --initial-cluster ${CLUSTER} \
  --initial-cluster-token ${TOKEN} \
  --initial-advertise-peer-urls http://${THIS_IP}:2380 \
  --data-dir /etcd-data/new

Once your run the script, you should get something like this

./etcd-restore.sh 
2020-02-11 08:39:28.391951 I | etcdserver/membership: added member 4c3c38d6652b5d75 [http://192.168.11.10:2380] to cluster 7ef1685daf3d8f18
2020-02-11 08:39:28.392657 I | etcdserver/membership: added member a00eaaf5194c573d [http://192.168.10.10:2380] to cluster 7ef1685daf3d8f18
2020-02-11 08:39:28.392672 I | etcdserver/membership: added member fdd13ca43538c5a2 [http://192.168.12.10:2380] to cluster 7ef1685daf3d8f18
2020-02-11 08:39:29.407616 I | etcdserver/membership: added member 4c3c38d6652b5d75 [http://192.168.11.10:2380] to cluster 7ef1685daf3d8f18
2020-02-11 08:39:29.407817 I | etcdserver/membership: added member a00eaaf5194c573d [http://192.168.10.10:2380] to cluster 7ef1685daf3d8f18
2020-02-11 08:39:29.407847 I | etcdserver/membership: added member fdd13ca43538c5a2 [http://192.168.12.10:2380] to cluster 7ef1685daf3d8f18
2020-02-11 08:39:30.365027 I | etcdserver/membership: added member 4c3c38d6652b5d75 [http://192.168.11.10:2380] to cluster 7ef1685daf3d8f18
2020-02-11 08:39:30.365130 I | etcdserver/membership: added member a00eaaf5194c573d [http://192.168.10.10:2380] to cluster 7ef1685daf3d8f18
2020-02-11 08:39:30.365213 I | etcdserver/membership: added member fdd13ca43538c5a2 [http://192.168.12.10:2380] to cluster 7ef1685daf3d8f18

Once the restore is complete, you simply start the cluster up and make sure the data-dir is pointing to the new restore location. You can use the same script as in this post but make sure to update the data dir.
Once our cluster is up, you should check the health of the cluster

$ docker exec -it etcd-node-0 /bin/sh
/ # export ETCDCTL_API=3
/ # etcdctl -w table --endpoints=[192.168.11.10:2379,192.168.10.10:2379,192.168.12.10:2379] endpoint status
+--------------------+------------------+---------+---------+-----------+-----------+------------+
|      ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+--------------------+------------------+---------+---------+-----------+-----------+------------+
| 192.168.11.10:2379 | 4c3c38d6652b5d75 |  3.2.28 |   25 kB |     false |         2 |          8 |
| 192.168.10.10:2379 | a00eaaf5194c573d |  3.2.28 |   25 kB |      true |         2 |          8 |
| 192.168.12.10:2379 | fdd13ca43538c5a2 |  3.2.28 |   25 kB |     false |         2 |          8 |
+--------------------+------------------+---------+---------+-----------+-----------+------------+

ETCD Multinode cluster using Docker

Tunde Oladipupo — Tue, 11 Feb 2020 06:48:16 +0000

In this post, I will be showing how to create a 3-node cluster with each node represented by a docker container. First, lets create our nodes IP.

sudo ifconfig en0 alias 192.168.10.10/24 up
sudo ifconfig en0 alias 192.168.11.10/24 up
sudo ifconfig en0 alias 192.168.12.10/24 up

Once they are created, you can use script below which is a modified version of this

REGISTRY=quay.io/coreos/etcd
# available from v3.2.5
# REGISTRY=gcr.io/etcd-development/etcd

# For each machine
ETCD_VERSION=latest
TOKEN=my-etcd-token
CLUSTER_STATE=new
NAME_1=etcd-node-0
NAME_2=etcd-node-1
NAME_3=etcd-node-2
HOST_1=192.168.10.10
HOST_2=192.168.11.10
HOST_3=192.168.12.10
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380
DATA_DIR=/mydir/etcd #make sure to change this

# For node 1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
docker run \
  -d \
  -p ${THIS_IP}:2379:2379 \
  -p ${THIS_IP}:2380:2380 \
  --volume=${DATA_DIR}/${THIS_NAME}:/etcd-data \
  --name ${THIS_NAME} ${REGISTRY}:${ETCD_VERSION} \
  /usr/local/bin/etcd \
  --data-dir=/etcd-data --name ${THIS_NAME} \
  --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://0.0.0.0:2380 \
  --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://0.0.0.0:2379 \
  --initial-cluster ${CLUSTER} \
  --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For node 2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
docker run \
  -d \
  -p ${THIS_IP}:2379:2379 \
  -p ${THIS_IP}:2380:2380 \
  --volume=${DATA_DIR}/${THIS_NAME}:/etcd-data \
  --name ${THIS_NAME} ${REGISTRY}:${ETCD_VERSION} \
  /usr/local/bin/etcd \
  --data-dir=/etcd-data --name ${THIS_NAME} \
  --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://0.0.0.0:2380 \
  --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://0.0.0.0:2379 \
  --initial-cluster ${CLUSTER} \
  --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# # For node 3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
docker run \
  -d \
  -p ${THIS_IP}:2379:2379 \
  -p ${THIS_IP}:2380:2380 \
  --volume=${DATA_DIR}/${THIS_NAME}:/etcd-data \
  --name ${THIS_NAME} ${REGISTRY}:${ETCD_VERSION} \
  /usr/local/bin/etcd \
  --data-dir=/etcd-data --name ${THIS_NAME} \
  --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://0.0.0.0:2380 \
  --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://0.0.0.0:2379 \
  --initial-cluster ${CLUSTER} \
  --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

Once the cluster is created, you can use the command below to check this cluster status.

$ docker exec -it etcd-node-0 /bin/sh
/ # etcdctl -w table --endpoints=192.168.11.10:2379,192.168.10.10:2379,192.168.12.10:2379 endpoint status
+--------------------+------------------+---------+---------+-----------+-----------+------------+
|      ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+--------------------+------------------+---------+---------+-----------+-----------+------------+
| 192.168.11.10:2379 | 4c3c38d6652b5d75 |  3.2.28 |   25 kB |     false |         5 |         17 |
| 192.168.10.10:2379 | a00eaaf5194c573d |  3.2.28 |   25 kB |      true |         5 |         17 |
| 192.168.12.10:2379 | fdd13ca43538c5a2 |  3.2.28 |   25 kB |     false |         5 |         17 |
+--------------------+------------------+---------+---------+-----------+-----------+------------+

Install Istio with kubectl

Tunde Oladipupo — Mon, 28 Oct 2019 01:37:13 +0000

To install Istio on EKS with kubectl, we will need to use helm to generate the manifest and use kubectlto apply.

Install Istio Init

curl -L https://git.io/getLatestIstio | ISTIO_VERSION=1.3.3 sh -
cd istio-1.3.3
helm template install/kubernetes/helm/istio-init --namespace istio-system | kubectl apply -f -

Install Istio Demo

kubectl apply -f install/kubernetes/istio-demo.yaml

Output

$ kubectl get pods -n istio-system
NAME                                      READY   STATUS      RESTARTS   AGE
grafana-9bdbcc98d-gzntp                   1/1     Running     0          23m
istio-citadel-6759c5c4d4-7wlnw            1/1     Running     0          23m
istio-egressgateway-c7776cf56-zjnx8       1/1     Running     0          23m
istio-galley-8544f4ff94-9qj4m             1/1     Running     0          23m
istio-grafana-post-install-1.3.3-l989d    0/1     Completed   0          23m
istio-ingressgateway-8469cb69c5-kjvkc     1/1     Running     0          23m
istio-pilot-748dd866dc-rh25k              2/2     Running     0          23m
istio-policy-5c564f5bfb-8hvgc             2/2     Running     1          23m
istio-security-post-install-1.3.3-7pp5z   0/1     Completed   0          23m
istio-sidecar-injector-59c8999585-lqlj7   1/1     Running     0          23m
istio-telemetry-5fc579cb56-sjd9d          2/2     Running     1          23m
istio-tracing-6f585d8877-24974            0/1     Running     10         23m
kiali-5764f9c44d-c7ftg                    1/1     Running     0          23m
prometheus-85d75b5f6b-fn7f2               1/1     Running     0          23m

Bookinfo application

kubectl label ns istio-system  istio-injection=disabled
kubectl label ns istio-demo  istio-injection=enabled
kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml

$ kubectl get pods
NAME                             READY   STATUS    RESTARTS   AGE
details-v1-c5b5f496d-p5lr7       1/2     Running   0          98m
helloworld-v1-7bb88866c4-hj8m2   2/2     Running   0          108m
helloworld-v2-6cd449dff4-mxjwn   1/2     Running   0          108m
productpage-v1-c7765c886-tbd2c   2/2     Running   0          98m
ratings-v1-f745cf57b-qpthv       2/2     Running   0          98m
reviews-v1-75b979578c-6hslt      2/2     Running   0          98m
reviews-v2-597bf96c8f-q92v7      2/2     Running   0          98m
reviews-v3-54c6c64795-9gc9q      2/2     Running   0          98m

Kubernetes, PodSecurityPolicy and Kubeadm

Tunde Oladipupo — Sat, 01 Jun 2019 00:41:56 +0000

I will assume you have a little background about PodSecurityPolicy and now its time to setup your cluster with PodSecurityPolicy admission controller enabled. For our setup, here is the kubeadm config;

kind: InitConfiguration
apiVersion: kubeadm.k8s.io/v1beta1
kind: InitConfiguration
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
  AppArmor: true
cpuManagerPolicy: static
systemReserved:
  cpu: 500m
  memory: 256M
kubeReserved:
  cpu: 500m
  memory: 256M
---
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
apiServer:
  extraArgs:
    enable-admission-plugins:  PodSecurityPolicy,LimitRanger,ResourceQuota,AlwaysPullImages,DefaultStorageClass

Let's start the cluster,

$ sudo kubeadm init --config kubeadm.json 
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
#join a worker node to the master
$ sudo kubeadm join 10.100.11.231:6443 --token 6yel1a.ce3le6eel3kfnxsz --discovery-token-ca-cert-hash sha256:99ee2e4ea
302c5270f2047c7a0093533b69105a8c91bf20f48b230dce9fd3f3a
$ kubectl get no
NAME               STATUS     ROLES    AGE    VERSION
ip-10-100-11-199   NotReady   <none>   109s   v1.13.1
ip-10-100-11-231   NotReady   master   3m3s   v1.13.1

As you can see, our cluster is not ready because we need to install a network plugin which you can see if you describe the node.

$ kubectl describe no ip-10-100-11-231
...
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Fri, 31 May 2019 22:50:36 +0000   Fri, 31 May 2019 22:46:39 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Fri, 31 May 2019 22:50:36 +0000   Fri, 31 May 2019 22:46:39 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Fri, 31 May 2019 22:50:36 +0000   Fri, 31 May 2019 22:46:39 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Fri, 31 May 2019 22:50:36 +0000   Fri, 31 May 2019 22:46:39 +0000   KubeletNotReady              runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Addresses:
...

We will be using calico as our network plugin and you can follow the install instruction right here

$ kubectl apply -f https://docs.projectcalico.org/v3.7/manifests/calico.yaml
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.extensions/calico-node created
serviceaccount/calico-node created
deployment.extensions/calico-kube-controllers created
serviceaccount/calico-kube-controllers created

To our surprise, there is no pod running in kube-system as we would expect. Let look at calico-node daemonset;

$ kubectl describe daemonset calico-node  -n kube-system
Name:           calico-node
Selector:       k8s-app=calico-node
Node-Selector:  beta.kubernetes.io/os=linux
Labels:         k8s-app=calico-node
...
Events:
  Type     Reason        Age                  From                  Message
  ----     ------        ----                 ----                  -------
  Warning  FailedCreate  20s (x15 over 102s)  daemonset-controller  Error creating: pods "calico-node-" is forbidden: no providers available to validate pod request

Ok, it says no provider and that means you do not have any PSP defined that will validate the daemonset pods. It is a very confusing error message.
Now, we will apply this PSP below.

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: calico-psp
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
spec:
  privileged: false
  allowPrivilegeEscalation: true
  allowedCapabilities:
  - '*'
  volumes:
  - '*'
  hostNetwork: true
  hostPorts:
  - min: 0
    max: 65535
  hostIPC: true
  hostPID: true
  runAsUser:
    rule: 'RunAsAny'
  seLinux:
    rule: 'RunAsAny'
  supplementalGroups:
    rule: 'RunAsAny'
  fsGroup:
    rule: 'RunAsAny'

$ kubectl apply -f calico-psp.yaml 
podsecuritypolicy.policy/calico-psp created
$ kubectl get psp
NAME             PRIV    CAPS   SELINUX    RUNASUSER   FSGROUP    SUPGROUP   READONLYROOTFS   VOLUMES
calico-psp   false          RunAsAny   RunAsAny    RunAsAny   RunAsAny   false            *

Ok everything should be good now? Well, if you describe the daemonset, you will still see the same message as above. You need to delete and re-apply the calico plugin manifest. After this, let's go ahead and describe our daemonset again.

$ kubectl describe daemonset calico-node  -n kube-system
Name:           calico-node
Selector:       k8s-app=calico-node
Node-Selector:  beta.kubernetes.io/os=linux
Labels:         k8s-app=calico-node
...
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/cni/networks
    HostPathType:  
Events:
  Type     Reason        Age                From                  Message
  ----     ------        ----               ----                  -------
  Warning  FailedCreate  4s (x12 over 14s)  daemonset-controller  Error creating: pods "calico-node-" is forbidden: unable to validate against any pod security policy: []

The error means we have a psp but not something we can use to validate our daemonset. This is because our daemonset pod container cannot use our PSP and we need to modify the daemonset clusterrole to be able to use this specific PSP and add this rule. Yep, you need to recreate.

# Include a clusterrole for the calico-node DaemonSet,
# and bind it to the calico-node serviceaccount.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: calico-node
rules:
  - apiGroups: ["extensions"]
    resources:
       - podsecuritypolicies
    resourceNames:
       - calico-psp
    verbs:
       - use
  # The CNI plugin needs to get pods, nodes, and namespaces.
  - apiGroups: [""]
    resources:
      - pods
      - nodes
      - namespaces
    verbs:
      - get
---
...

After the clusterrole has been modified, we will run into another error;

...
   host-local-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/cni/networks
    HostPathType:  
Events:
  Type     Reason        Age               From                  Message
  ----     ------        ----              ----                  -------
  Warning  FailedCreate  3s (x11 over 9s)  daemonset-controller  Error creating: pods "calico-node-" is forbidden: unable to validate against any pod security policy: [spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]
...

Our calico pods are forbidden because the PSP it is using forbids privileged container and Calico needs one to handle network policy. Now, lets go ahead and fix that in our PSP by updating;

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: calico-psp
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
spec:
  privileged: true #update this to true
  allowPrivilegeEscalation: true
...

Once the update has been applied, calico will create the pods and our node will become healthy.

...
Events:
  Type     Reason            Age                     From                  Message
  ----     ------            ----                    ----                  -------
  Warning  FailedCreate      3m10s (x16 over 5m54s)  daemonset-controller  Error creating: pods "calico-node-" is forbidden: unable to validate against any pod security policy: [spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]
  Normal   SuccessfulCreate  26s                     daemonset-controller  Created pod: calico-node-hwb2b
  Normal   SuccessfulCreate  26s                     daemonset-controller  Created pod: calico-node-gtrm2

$ kubectl get no
NAME               STATUS   ROLES    AGE     VERSION
ip-10-100-11-199   Ready    <none>   7m2s    v1.13.1
ip-10-100-11-231   Ready    master   7m33s   v1.13.1

Yayy!!! Our nodes are ready.

Lets Talk About Terraform 0.12

Tunde Oladipupo — Mon, 27 May 2019 16:28:39 +0000

Terraform 0.12 was recently released and you can check out the details more in-dept at terraform blogs. The focus of this blog post will be the new features released and how to use them. You can find the code used in this Github repo. Below is the content of the ec2 main.tf.

module "test_ec2" {
      source = "./ec2_module"
      //I am passing the subnet object instead of the id
      subnet = aws_subnet.private.0
      instance_tags = [
        {
          Key = "Name"
          Value = "Test"
        },
        {
          Key = "Environment"
          Value = "prd"
        }
    ]
}

Here is the content of the ec2 module;

data "aws_ami" "ubuntu" {
  most_recent = true

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["099720109477"] # Canonical
}
resource "aws_instance" "web" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t2.micro"
  subnet_id  = var.subnet.id
  user_data  = templatefile("${path.module}/userdata.tpl", { instance_tags = var.instance_tags })
  tags = {
     for tag in var.instance_tags:
       tag.Key => tag.Value
  }
}

variable "instance_type" {
     default = "t2.micro"
}

variable "instance_tags" {
     type = list
}

variable "subnet" {
    type = object({
        id = string
    })
}

For expression

From my perspective, this is one of the best feature of 0.12 which allows you to iterate over a list or map and you can do whatever you want with each item. From above, you can see that we generated the instance tags property from the user passed in list of maps.

  tags = {
     for tag in var.instance_tags:
       tag.Key => tag.Value
  }

First class expression

You can see from above that we did not have to interpolate over the variables or object values instead they were used directly as first class variable. This will save some ink and allows for more complex operational usage with terraform.

...
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t2.micro"
  subnet_id  = var.subnet.id
...

Rich Value Type

My impression of this feature so far is that we can have user-defined type and existing type are supported as first class type without quote. We defined a type that expects map with id attribute and if passing a non-matching object, it will be rejected which is pretty cool.

variable "instance_tags" {
     type = list
}

variable "subnet" {
    type = object({
        id = string
    })
}

New Type Interpolation

New terraform now gives you the capability to loop in your user data. For our example, here is the userdata;

#!/bin/bash

%{ for tag in instance_tags~}
cat ${tag.Key}=${tag.Value}
%{ endfor ~}

We were able to use forloop to go over the tags and use it in instance user data which is lovely without having to through hacks.

One other feature that was not discussed here is the dynamic block which you can checkout on terraform blog page.

Logging with AWS Kubernetes EKS Cluster

Tunde Oladipupo — Fri, 24 May 2019 02:33:38 +0000

Logs

EKS is the managed kubernetes offering by AWS that saves you the stress of managing your own control plane with a twist of offboarding some controls like what goes on in your control. The feature was not available when the service went GA but was recently made available recently. Here are the kinds of logs that it provides;

API server component logs: You know that component of your cluster that validates requests, provides api rest endpoint and so on? These are the logs from the apiserver which are very critical when trying to diagnose things like why your pods are not creating, admission controller issues etc.

E0523 03:27:22.258958 1 memcache.go:134] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request

Audit Logs: People make changes in your cluster and you want to know who, what and when. This logs gives you the ability to this.

{
    "kind": "Event",
    "apiVersion": "audit.k8s.io/v1beta1",
    "metadata": {
        "creationTimestamp": "2019-05-23T02:08:34Z"
    },
    "level": "Request",
    "timestamp": "2019-05-23T02:08:34Z",
    "auditID": "84662c40-8d4f-4d3e-99b2-0d4005e44375",
    "stage": "ResponseComplete",
    "requestURI": "/api/v1/namespaces/default/services/kubernetes",
    "verb": "get",
    "user": {
        "username": "system:apiserver",
        "uid": "2d8ad7ed-25ed-4f37-a2f0-416d2af705e9",
        "groups": [
            "system:masters"
        ]
    },
    "sourceIPs": [
        "::1"
    ],
    "userAgent": "kube-apiserver/v1.12.6 (linux/amd64) kubernetes/d69f1bf",
    "objectRef": {
        "resource": "services",
        "namespace": "default",
        "name": "kubernetes",
        "apiVersion": "v1"
    },
    "responseStatus": {
        "metadata": {},
        "code": 200
    },
    "requestReceivedTimestamp": "2019-05-23T02:08:34.498973Z",
    "stageTimestamp": "2019-05-23T02:08:34.501446Z",
    "annotations": {
        "authorization.k8s.io/decision": "allow",
        "authorization.k8s.io/reason": ""
    }
}

Authenticator Logs: EKS uses this thing called aws-iam-authenticator to guess what? Authenticate against the EKS cluster using AWS credentials and roles. These logs contains event from these activities

time="2019-05-16T22:19:48Z" level=info msg="Using assumed role for EC2 API" roleARN="arn:aws:iam::523447765480:role/idaas-kubernetes-cluster-idauto-dev-masters-role"

Controller manager: For those familiar with kubernetes objects such as Deployments, Replicas etc; these are managed by controllers which ships with kubernetes controller manager. To see what these controllers are doing under the hood, you need these.

E0523 02:07:55.486872 1 horizontal.go:212] failed to compute desired number of replicas based on listed metrics for Deployment/routing/rapididentity-default-backend: failed to get memory utilization: unable to get metrics for resource memory: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)

Scheduler: This component of the control plane does what it name says, put pods on the right node after factoring a number of constraints and resources available. To see information on how this component is making its decision, check these logs.

E0523 02:07:55.486872 1 horizontal.go:212] failed to compute desired number of replicas based on listed metrics for Deployment/routing/rapididentity-default-backend: failed to get memory utilization: unable to get metrics for resource memory: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)

Enabling Logs

You can easily enable the logs in your EKS cluser console and AWS updates your cluster to enable those logs to ship to cloudwatch. The corresponding cloudwatch log group will be displayed in your console. For those using terraform to provision their cluster, you can just pass in the types of logs that you want to provision and also create the log group to ship it to.

resource "aws_eks_cluster" "my_cluster" {
  depends_on = ["aws_cloudwatch_log_group.eks_log_group"]
  enabled_cluster_log_types = ["api", "audit"]
  name                      = "${var.cluster_name}"
  # ... other configuration ...
}