Forem: Ishan Khare

Dabbling with envoy - Part II

Ishan Khare — Thu, 13 Jan 2022 22:35:49 +0000

Dynamic configuration from control plane

There are already some production ready control planes available like

Gloo
Istio

We are going to experiment with what's at the core of these service meshes - Go control plane

In order to tell envoy where to look for the control plane for getting the configuration we need some of the following things:

node

The node configuration remains the same as in the previous setup of the envoy configurations:

node:
  cluster: test-cluster
  id: test-id

dynamic_resources

This tells envoy which configurations to update dynamically

dynamic_resources:
  ads_config:
    api_type: GRPC
    transport_api_version: V3
    grpc_services:
      envoy_grpc:
        cluster_name: xds_cluster
  cds_config:
    resource_api_version: V3
    ads: {}
  lds_config:
    resource_api_version: V3
    ads: {}

This basically tells envoy that the Cluster Discovery Service and the Listener Discovery Service are being served as part of the ADS (Aggregate Discovery Service). The ads configuration itself is a GRPC api served by a cluster named xds_cluster.

static_resources

This will tell envoy where to receive its dynamic configuration from, i.e. where to find the control plane

static_resources:
  clusters:
  - type: STRICT_DNS
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        explicit_http_config:
          http2_protocol_options: {}
    name: xds_cluster
    http2_protocol_options: {}
    connect_timeout: 3s
    load_assignment:
      cluster_name: xds_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: localhost
                port_value: 18000

Here we define the details of where can envoy find the previously specified xds_cluster serving the ADS configuration.

This is the basic configuration that we need to bootstrap our envoy instance to connect to our control plane and start getting updated configuration from it. The main advantage of this pattern over the previously described ones is that we can now take advantage of the powerful programmable API of the control plane and update the configuration programmatically. So lets see ho we can get this done.

Control Plane

At the heart of the control place configuration, there's what envoy calls a Snapshot. Going through the envoy docs, this is what the definition of a snapshot says it is:

Snapshot is an internally consistent snapshot of xDS resources

And what does the Snapshot look like you may ask?

type Snapshot struct {
    Resources [types.UnknownType]Resources

    // VersionMap holds the current hash map of all resources in the snapshot.
    // This field should remain nil until it is used, at which point should be
    // instantiated by calling ConstructVersionMap().
    // VersionMap is only to be used with delta xDS.
    VersionMap map[string]map[string]string
}

Hmm, that isn't very helpful. It doesn't give us much information about what does the Resources hold. On further examiniation of the docs, I found the following method in the cache package:

func NewSnapshot(version string, resources map[resource.Type][]types.Resource) (Snapshot, error)

And examining the resource package we get these constants

const (
    EndpointType        = apiTypePrefix + "envoy.config.endpoint.v3.ClusterLoadAssignment"
    ClusterType         = apiTypePrefix + "envoy.config.cluster.v3.Cluster"
    RouteType           = apiTypePrefix + "envoy.config.route.v3.RouteConfiguration"
    ScopedRouteType     = apiTypePrefix + "envoy.config.route.v3.ScopedRouteConfiguration"
    ListenerType        = apiTypePrefix + "envoy.config.listener.v3.Listener"
    SecretType          = apiTypePrefix + "envoy.extensions.transport_sockets.tls.v3.Secret"
    ExtensionConfigType = apiTypePrefix + "envoy.config.core.v3.TypedExtensionConfig"
    RuntimeType         = apiTypePrefix + "envoy.service.runtime.v3.Runtime"

    // AnyType is used only by ADS
    AnyType = ""
)

So now it makes sense – the snapshot is just a fancy register that will store a point-in-time snapshot of the envoy configuration for different envoy resources like listener, route, endpoint etc. This is good but is there anything better? – a better abstraction that we can probably use? It turns out there is. On further examination I found the SnapshotCache :

type SnapshotCache interface {
    Cache

    SetSnapshot(ctx context.Context, node string, snapshot Snapshot) error

    GetSnapshot(node string) (Snapshot, error)

    ClearSnapshot(node string)

    GetStatusInfo(string) StatusInfo

    GetStatusKeys() []string
}

func NewSnapshotCache(ads bool, hash NodeHash, logger log.Logger) SnapshotCache

SnapshotCache is a snapshot-based cache that maintains a single versioned snapshot of responses per node. SnapshotCache consistently replies with the latest snapshot

Alright now we have some idea of the structure we want to put around this thing:

We will start implementing the same in the following snippets:

func GenerateSnapshot(listenerName, routeName, clusterName, upstreamHost string, listenerPort, upstreamPort uint32) cache.Snapshot {
    return cache.NewSnapshot("1",
        // endpoints
        []types.Resource{},
        // clusters
        []types.Resource{makeCluster(clusterName, upstreamHost, upstreamPort)},
        // routes
        []types.Resource{makeRoute(routeName, clusterName, upstreamHost)},
        // listeners
        []types.Resource{makeListener(listenerName, routeName, listenerPort)},
        // runtimes
        []types.Resource{},
        // secrets
        []types.Resource{},
    )
}

Now we implement the methods makeCluster, makeRoute, makeListener:

Cluster and Endpoint

import (
    "time"

    cluster "github.com/envoyproxy/go-control-plane/envoy/config/cluster/v3"
    "github.com/golang/protobuf/ptypes"

    core "github.com/envoyproxy/go-control-plane/envoy/config/core/v3"
    endpoint "github.com/envoyproxy/go-control-plane/envoy/config/endpoint/v3"
)

func makeCluster(clusterName, upstreamHost string, upstreamPort uint32) *cluster.Cluster {
    return &cluster.Cluster{
        Name:           clusterName,
        ConnectTimeout: ptypes.DurationProto(5 * time.Second),
        ClusterDiscoveryType: &cluster.Cluster_Type{
            Type: cluster.Cluster_LOGICAL_DNS,
        },
        LbPolicy:        cluster.Cluster_ROUND_ROBIN,
        LoadAssignment:  makeEndpoint(clusterName, upstreamHost, upstreamPort),
        DnsLookupFamily: cluster.Cluster_V4_ONLY,
    }
}

func makeEndpoint(clusterName, upstreamHost string, upstreamPort uint32) *endpoint.ClusterLoadAssignment {
    return &endpoint.ClusterLoadAssignment{
        ClusterName: clusterName,
        Endpoints: []*endpoint.LocalityLbEndpoints{
            {
                LbEndpoints: []*endpoint.LbEndpoint{
                    {
                        HostIdentifier: &endpoint.LbEndpoint_Endpoint{
                            Endpoint: &endpoint.Endpoint{
                                Address: &core.Address{
                                    Address: &core.Address_SocketAddress{
                                        SocketAddress: &core.SocketAddress{
                                            Protocol: core.SocketAddress_TCP,
                                            Address:  upstreamHost,
                                            PortSpecifier: &core.SocketAddress_PortValue{
                                                PortValue: upstreamPort,
                                            },
                                        },
                                    },
                                },
                            },
                        },
                    },
                },
            },
        },
    }
}

Route


func makeRoute(routeName, clusterName, upstreamHost string) *route.RouteConfiguration {
    return &route.RouteConfiguration{
        Name: routeName,
        VirtualHosts: []*route.VirtualHost{
            {
                Name:    "local_service",
                Domains: []string{"*"},
                Routes: []*route.Route{
                    {
                        Match: &route.RouteMatch{
                            PathSpecifier: &route.RouteMatch_Prefix{
                                Prefix: "/",
                            },
                        },
                        Action: &route.Route_Route{
                            Route: &route.RouteAction{
                                ClusterSpecifier: &route.RouteAction_Cluster{
                                    Cluster: clusterName,
                                },
                                HostRewriteSpecifier: &route.RouteAction_HostRewriteLiteral{
                                    HostRewriteLiteral: upstreamHost,
                                },
                            },
                        },
                    },
                },
            },
        },
    }
}

Listener, connection manager and config source

This is probably the most complicated one – but will conform to the structure we defined in the previous article's dynamic listerner config – the only difference being it was in yaml and this is in pure go.

import (
    core "github.com/envoyproxy/go-control-plane/envoy/config/core/v3"
    listener "github.com/envoyproxy/go-control-plane/envoy/config/listener/v3"
    hcm "github.com/envoyproxy/go-control-plane/envoy/extensions/filters/network/http_connection_manager/v3"
    "github.com/envoyproxy/go-control-plane/pkg/cache/types"
    "github.com/envoyproxy/go-control-plane/pkg/cache/v3"
    "github.com/envoyproxy/go-control-plane/pkg/resource/v3"
    "github.com/envoyproxy/go-control-plane/pkg/wellknown"
    "github.com/golang/protobuf/ptypes"
)

func makeListener(listenerName, routeName string, listenerPort uint32) *listener.Listener {
    manager := makeHTTPConnManager(routeName)
    typedConfig, err := ptypes.MarshalAny(manager)
    if err != nil {
        panic(err)
    }

    return &listener.Listener{
        Name: listenerName,
        Address: &core.Address{
            Address: &core.Address_SocketAddress{
                SocketAddress: &core.SocketAddress{
                    Protocol: core.SocketAddress_TCP,
                    Address:  "0.0.0.0",
                    PortSpecifier: &core.SocketAddress_PortValue{
                        PortValue: listenerPort,
                    },
                },
            },
        },
        FilterChains: []*listener.FilterChain{
            {
                Filters: []*listener.Filter{
                    {
                        Name: wellknown.HTTPConnectionManager,
                        ConfigType: &listener.Filter_TypedConfig{
                            TypedConfig: typedConfig,
                        },
                    },
                },
            },
        },
    }
}

func makeHTTPConnManager(routeName string) *hcm.HttpConnectionManager {
    return &hcm.HttpConnectionManager{
        CodecType:  hcm.HttpConnectionManager_AUTO,
        StatPrefix: "http",
        // AccessLog: []*accesslog.AccessLog{
        //  {
        //      Name:       "envoy.access_loggers.stdout",
        //      ConfigType: &accesslog.AccessLog_TypedConfig{},
        //  },
        // },
        RouteSpecifier: &hcm.HttpConnectionManager_Rds{
            Rds: &hcm.Rds{
                ConfigSource:    makeConfigSource(),
                RouteConfigName: routeName,
            },
        },
        HttpFilters: []*hcm.HttpFilter{
            {
                Name: wellknown.Router,
            },
        },
    }
}

func makeConfigSource() *core.ConfigSource {
    source := &core.ConfigSource{}
    source.ResourceApiVersion = resource.DefaultAPIVersion
    source.ConfigSourceSpecifier = &core.ConfigSource_ApiConfigSource{
        ApiConfigSource: &core.ApiConfigSource{
            TransportApiVersion:       resource.DefaultAPIVersion,
            ApiType:                   core.ApiConfigSource_GRPC,
            SetNodeOnFirstMessageOnly: true,
            GrpcServices: []*core.GrpcService{
                {
                    TargetSpecifier: &core.GrpcService_EnvoyGrpc_{
                        EnvoyGrpc: &core.GrpcService_EnvoyGrpc{
                            ClusterName: "xds_cluster",
                        },
                    },
                },
            },
        },
    }

    return source
}

So far what we've managed to do is to:

Define our whole configuration in Go using the envoy's go-control-plane library
Wrap that configuration in an envoy Snapshot

With one more call we can wrap this Snapshot inside a SnapshotCache:

snapshotCache = cache.NewSnapshotCache(false, cache.IDHash{}, log)
snapshot := snapshot.GenerateSnapshot("listener_0", "local_route", "example_cluster", "www.envoyproxy.io", 10000, 80)
if err := snapshotCache.SetSnapshot("test-id", snapshot); err != nil {
        log.Errorf("snapshot error %q for %+v", err, snapshot)
        os.Exit(1)
}

This is pretty simple – with all that we've already written above in our makeCluster, makeRoute, makeListener methods, we are basically defining:

A listener name listener_0
A route named local_route
A cluster named example_cluster
An upstream host which is www.envoyproxy.io
Listener port 10000
Upstream port 80

This should mean that this configuration should create a listener binding to port 10000 and proxying to www.envoyproxy.io on port 80

The last line is the method SetSnapshot where we tell the SnapshotCache of the active snapshot in the cache and the name of that snapshot.

Setting up the gRPC services

Since in this post we are talking about dynamic serving of envoy through the xDS control plane, the gRPC protocol forms the basis of how this configuration will be served to the envoy proxies. So lets set that up now:

import (
    "context"
    "encoding/json"
    "fmt"
    "net"
    "os"

    clusterservice "github.com/envoyproxy/go-control-plane/envoy/service/cluster/v3"
    discoverygrpc "github.com/envoyproxy/go-control-plane/envoy/service/discovery/v3"
    endpointservice "github.com/envoyproxy/go-control-plane/envoy/service/endpoint/v3"
    listenerservice "github.com/envoyproxy/go-control-plane/envoy/service/listener/v3"
    routeservice "github.com/envoyproxy/go-control-plane/envoy/service/route/v3"
    runtimeservice "github.com/envoyproxy/go-control-plane/envoy/service/runtime/v3"
    secretservice "github.com/envoyproxy/go-control-plane/envoy/service/secret/v3"

    generator "github.com/ishankhare07/envoy-dynamic/pkg/snapshot"

    "github.com/ishankhare07/envoy-dynamic/pkg/logger"

    "github.com/envoyproxy/go-control-plane/pkg/cache/v3"
    "github.com/envoyproxy/go-control-plane/pkg/server/v3"
    "github.com/envoyproxy/go-control-plane/pkg/test/v3"
    "google.golang.org/grpc"
    "google.golang.org/grpc/reflection"
)

var log *logger.Logger
var snapshotCache cache.SnapshotCache

func init() {
    log = &logger.Logger{Debug: true}
}

func RunServer(ctx context.Context, port uint) {
    grpcServer := grpc.NewServer()

    lis, err := net.Listen("tcp", fmt.Sprintf(":%d", port))
    if err != nil {
        log.Errorf("%v", err)
    }

    snapshotCache = cache.NewSnapshotCache(false, cache.IDHash{}, log)

    snapshot := generator.GenerateSnapshot("listener_0", "local_route", "example_cluster", "www.envoyproxy.io", 10000, 80)
    if err := snapshot.Consistent(); err != nil {
        log.Errorf("snapshot inconsistency: %+v\n%+v", snapshot, err)
        os.Exit(1)
    }

    t, _ := json.MarshalIndent(snapshot, "", "  ")
    log.Debugf("will serve snapshot %s", t)

    // Add the snapshot to the cache
    if err := snapshotCache.SetSnapshot("test-id", snapshot); err != nil {
        log.Errorf("snapshot error %q for %+v", err, snapshot)
        os.Exit(1)
    }

    cb := &test.Callbacks{Debug: log.Debug}
    srv := server.NewServer(ctx, snapshotCache, cb)

    registerServer(grpcServer, srv)
    helloworldservice.RegisterGreeterServer(grpcServer, helloworldservice.NewHelloWorldServer(snapshotCache))
    reflection.Register(grpcServer)

    log.Infof("management server listening on port %d\n", port)
    if err = grpcServer.Serve(lis); err != nil {
        log.Errorf("%v", err)
        panic(err)
    }
}

func registerServer(grpcServer *grpc.Server, server server.Server) {
    // register services
    discoverygrpc.RegisterAggregatedDiscoveryServiceServer(grpcServer, server)
    endpointservice.RegisterEndpointDiscoveryServiceServer(grpcServer, server)
    clusterservice.RegisterClusterDiscoveryServiceServer(grpcServer, server)
    routeservice.RegisterRouteDiscoveryServiceServer(grpcServer, server)
    listenerservice.RegisterListenerDiscoveryServiceServer(grpcServer, server)
    secretservice.RegisterSecretDiscoveryServiceServer(grpcServer, server)
    runtimeservice.RegisterRuntimeDiscoveryServiceServer(grpcServer, server)
}

So our previously posted diagram becomes something like this

Lets make sure we serve this!

package main

import (
    "context"

    aggregateGRPCServer "github.com/ishankhare07/envoy-dynamic/pkg/server"
)

func main() {
    ctx := context.Background()
    aggregateGRPCServer.RunServer(ctx, 18000)
}

So our xDS server will be serving configuration on port 18000, and according to the configuration listener_0 will be binding and proxying on localhost:10000 to www.envoyproxy.io:80

Running the envoy proxy

Since we are serving the config from xDS control plane this time, does that mean we don't need to write any static_configuration now?
Unfortunately NO! – We still need to tell envoy where the dynamic configuration is being server at. The main flexibiliy that this control plane mode is allowing us is to programatically control the configuration – something that we'll explore soon in the following blog posts of this series. but for now we still need to write some basic envoy config:
envoy-dynamic-control-plane.yaml

node:
  cluster: test-cluster
  id: test-id

dynamic_resources:
  ads_config:
    api_type: GRPC
    transport_api_version: V3
    grpc_services:
      envoy_grpc:
        cluster_name: xds_cluster
  cds_config:
    resource_api_version: V3
    ads: {}
  lds_config:
    resource_api_version: V3
    ads: {}

static_resources:
  clusters:
  - type: STRICT_DNS
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        explicit_http_config:
          http2_protocol_options: {}
    name: xds_cluster
    http2_protocol_options: {}
    connect_timeout: 3s
    load_assignment:
      cluster_name: xds_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: localhost
                port_value: 18000
admin:
  access_log_path: /dev/null
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 19000
layered_runtime:
  layers:
    - name: runtime-0
      rtds_layer:
        rtds_config:
          resource_api_version: V3
          api_config_source:
            transport_api_version: V3
            api_type: GRPC
            grpc_services:
              envoy_grpc:
                cluster_name: xds_cluster
        name: runtime-0

Interesting parts are:

dynamic_resources.ads_config – here we tell envoy to:
- Expect a GRPC configuration
- Conforming to V3 of the envoy api
- Served by a envoy cluster named xds_cluster – where would envoy find xds_cluster now? Read along
static_resources.clusters[0].name – answer to your question above
static_resources.clusters[0].endpoints – what endpoints to expect in this cluster (basically the host and port of our gRPC xDS server serving the dynamic configuration)

Here's what things look like now

Running everything

If you've followed along till now, here's the reward. Lets run the control plane and the envoy server:

# in one shell let's start the control plane
go run main.go
2022/01/14 03:11:48 will serve snapshot {
  "Resources": [
    {
      "Version": "1",
      "Items": {}
    },
    {
      "Version": "1",
      "Items": {
        "example_cluster": {
          "Resource": {
            "name": "example_cluster",
            "ClusterDiscoveryType": {
              "Type": 2
            },
            "connect_timeout": {
              "seconds": 5
            },
            "load_assignment": {
              "cluster_name": "example_cluster",
              "endpoints": [
                {
                  "lb_endpoints": [
                    {
                      "HostIdentifier": {
                        "Endpoint": {
                          "address": {
                            "Address": {
                              "SocketAddress": {
                                "address": "www.envoyproxy.io",
                                "PortSpecifier": {
                                  "PortValue": 80
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  ]
                }
              ]
            },
            "dns_lookup_family": 1,
            "LbConfig": null
          },
          "Ttl": null
        }
      }
    },
    {
      "Version": "1",
      "Items": {
        "local_route": {
          "Resource": {
            "name": "local_route",
            "virtual_hosts": [
              {
                "name": "local_service",
                "domains": [
                  "*"
                ],
                "routes": [
                  {
                    "match": {
                      "PathSpecifier": {
                        "Prefix": "/"
                      }
                    },
                    "Action": {
                      "Route": {
                        "ClusterSpecifier": {
                          "Cluster": "example_cluster"
                        },
                        "HostRewriteSpecifier": {
                          "HostRewriteLiteral": "www.envoyproxy.io"
                        }
                      }
                    }
                  }
                ]
              }
            ]
          },
          "Ttl": null
        }
      }
    },
    {
      "Version": "1",
      "Items": {
        "listener_0": {
          "Resource": {
            "name": "listener_0",
            "address": {
              "Address": {
                "SocketAddress": {
                  "address": "0.0.0.0",
                  "PortSpecifier": {
                    "PortValue": 10000
                  }
                }
              }
            },
            "filter_chains": [
              {
                "filters": [
                  {
                    "name": "envoy.filters.network.http_connection_manager",
                    "ConfigType": {
                      "TypedConfig": {
                        "type_url": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
                        "value": "EgRodHRwKhsKGWVudm95LmZpbHRlcnMuaHR0cC5yb3V0ZXIaKgobMAISFwgCIg8KDQoLeGRzX2NsdXN0ZXI4AUACEgtsb2NhbF9yb3V0ZQ=="
                      }
                    }
                  }
                ]
              }
            ],
            "ListenerSpecifier": null
          },
          "Ttl": null
        }
      }
    },
    {
      "Version": "1",
      "Items": {}
    },
    {
      "Version": "1",
      "Items": {}
    },
    {
      "Version": "1",
      "Items": {}
    }
  ],
  "VersionMap": null
}
2022/01/14 03:11:48 management server listening on port 18000

We are spitting out the served xDS configuration for convenience. Lets start the envoy server in another terminal with the previously defined configuration:

envoy -c envoy-dynamic-control-plane.yaml -l debug
.
.
.
[2022-01-14 03:14:28.280][4602226][debug][router] [source/common/router/router.cc:486] [C0][S9032965470346210647] cluster 'xds_cluster' match for URL '/envoy.service.discovery.v3.AggregatedDiscoveryServic
e/StreamAggregatedResources'
[2022-01-14 03:14:28.280][4602226][debug][router] [source/common/router/router.cc:702] [C0][S9032965470346210647] router decoding headers:
':method', 'POST'
':path', '/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources'
':authority', 'xds_cluster'
':scheme', 'http'
'te', 'trailers'
'content-type', 'application/grpc'
'x-envoy-internal', 'true'
'x-forwarded-for', '192.168.29.160'
.
.
.
.

There will be lots of information but the one I've pasted above confirms that envoy was able to find the xds_cluster serving our configuration. Also if we go and look at the control_plane terminal, we'll find this:

2022/01/14 03:14:27 stream 1 open for type.googleapis.com/envoy.service.runtime.v3.Runtime
2022/01/14 03:14:27 respond type.googleapis.com/envoy.service.runtime.v3.Runtime[runtime-0] version "" with version "1"
2022/01/14 03:14:27 open watch 1 for type.googleapis.com/envoy.service.runtime.v3.Runtime[runtime-0] from nodeID "test-id", version "1"
2022/01/14 03:14:28 stream 2 open for
2022/01/14 03:14:42 respond type.googleapis.com/envoy.config.cluster.v3.Cluster[] version "" with version "1"
2022/01/14 03:14:42 open watch 2 for type.googleapis.com/envoy.config.cluster.v3.Cluster[] from nodeID "test-id", version "1"
2022/01/14 03:14:43 respond type.googleapis.com/envoy.config.listener.v3.Listener[] version "" with version "1"
2022/01/14 03:14:43 stream 3 open for type.googleapis.com/envoy.config.route.v3.RouteConfiguration
2022/01/14 03:14:43 open watch 3 for type.googleapis.com/envoy.config.listener.v3.Listener[] from nodeID "test-id", version "1"
2022/01/14 03:14:43 respond type.googleapis.com/envoy.config.route.v3.RouteConfiguration[local_route] version "" with version "1"
2022/01/14 03:14:43 open watch 4 for type.googleapis.com/envoy.config.route.v3.RouteConfiguration[local_route] from nodeID "test-id", version "1"

Let's test by visiting http://localhost:10000 – on clicking this link you should be redirected to www.envoyproxy.io:80

And there you have it!

If you would like to see the whole repo and code better organized that this blog post, head ove to this github repo – github.com/ishankhare07/envoy-dynamic

What next?!

Continuing my habbit of fiddling with all kinds of things, I've decided to convert this already long 2 part series of posts even longer. The goal is to start from envoy basics already discussed in the past and the current post and going all the way up to a very curde service mesh implementation. In the coming days I would be covering more of the following topics:

Dynamically adding more upstreams to our control plane through an exposed gRPC endpoint.
Write a sidecar injector that is able to inject envoy proxies into our k8s pods
Serve control plane inside k8s cluster with the sidecar injector enabled
Make the envoy sidecars fetch the dynamic configuration from the in-cluster control plane and proxy few upstreams.
Wrap the control plane in a k8s controller and use a CRD to add / remove upstreams
Proxy Pod-to-Pod communication and updating the upstream configuration using the k8s controller.

Dabbling with Envoy configurations - Part I

Ishan Khare — Mon, 28 Jun 2021 12:44:36 +0000

Types of envoy configurations

Static configuration
Dynamic configuration. This can be done in two ways:
- From Filesystem
- From Control plane

Static configuration

To start envoy in static configuration we need the following:

listeners
clusters
static_reources
(Optional) admin section

static_resources

Contain everything that is configured statically when envoy starts. Can contain the following:

[]listeners
[]clusters
[]secrets

listeners

Lets configure an example listener on port 10000. Here all paths are matched and routed to service_envoyproxy_io cluster

listeners:
- name: listener_0
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 10000
  filter_chains:
  - filters:
    - name: envoy.filters.network.http_connection_manager
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
        stat_prefix: ingress_http
        access_log:
        - name: envoy.access_loggers.stdout
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
        http_filters:
        - name: envoy.filters.http.router
        route_config:
          name: local_route
          virtual_hosts:
          - name: local_service
            domains: ["*"]
            routes:
            - match:
                prefix: "/"
              route:
                host_rewrite_literal: www.envoyproxy.io
                cluster: service_envoyproxy_io

cluster

The service_envoyproxy_io cluster proxies over TLS to https://www.envoyproxy.io

clusters:
- name: service_envoyproxy_io
  type: LOGICAL_DNS
  # Comment out the following line to test on v6 networks
  dns_lookup_family: V4_ONLY
  load_assignment:
    cluster_name: service_envoyproxy_io
    endpoints:
    - lb_endpoints:
      - endpoint:
          address:
            socket_address:
              address: www.envoyproxy.io
              port_value: 443
  transport_socket:
    name: envoy.transport_sockets.tls
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
      sni: www.envoyproxy.io

Testing this static configuration

If now we start envoy with this configuration using command envoy -c <config_name>.yaml and try querying the localhost:10000 port, we should get the envoyproxy homepage.

$ curl -v localhost:10000

Dynamic Configuration from filesystem

In this setup Envoy will automatically update its configuration whenever the files are changed on the filesystem. The following sections are a must for dynamic configuration:

node

node needs a cluster and an id

node:
  cluster: test-cluster
  id: test-id

dynamic_resources

Specifies where to load dynamic configuration from

dynamic_resources:
  cds_config:
    path: ./cds.yaml
  lds_config:
    path: ./lds_yaml

`listener` resources

The linked lds_config should be an implementation of a Listener Discovery Service

resources:
- "@type": type.googleapis.com/envoy.config.listener.v3.Listener
  name: listener_0
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 10000
  filter_chains:
  - filters:
    - name: envoy.http_connection_manager
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
        stat_prefix: ingress_http
        http_filters:
        - name: envoy.router
        route_config:
          name: local_route
          virtual_hosts:
          - name: local_service
            domains:
            - "*"
            routes:
            - match:
                prefix: "/"
              route:
                host_rewrite_literal: www.envoyproxy.io
                cluster: example_proxy_cluster

`cluster` resources

The linked cds_config should be an implementation of a Cluster Discovery Service

resources:
- "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster
  name: example_proxy_cluster
  type: STRICT_DNS
  connect_timeout: 3s
  typed_extension_protocol_options:
    envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
      "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
      explicit_http_config:
        http2_protocol_options: {}
  load_assignment:
    cluster_name: example_proxy_cluster
    endpoints:
    - lb_endpoints:
      - endpoint:
          address:
            socket_address:
              address: www.envoyproxy.io
              port_value: 443
  transport_socket:
    name: envoy.transport_sockets.tls
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
      sni: www.envoyproxy.io

Dynamically editing the configuration

Let's try editing this config to start proxying to google.com instead of envoyproxy.io

In the lds.yaml file change the following:

            routes:
            - match:
                prefix: "/"
              route:
-               host_rewrite_literal: www.envoyproxy.io
+               host_rewrite_literal: www.google.com
                cluster: example_proxy_cluster

As soon as we do this write in the file, the LDS config in the envoy will update and will show in the logs:

lds: add/update listener 'listener_0'

We need to update the cds.yaml config as well:

  load_assignment:
    cluster_name: example_proxy_cluster
    endpoints:
    - lb_endpoints:
      - endpoint:
          address:
            socket_address:
-              address: www.envoyproxy.io
+              address: www.google.com
              port_value: 443
  transport_socket:
    name: envoy.transport_sockets.tls
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
-      sni: www.envoyproxy.io
+      sni: www.google.com

We should see the similar update in envoy's logs about the CDS config update

cds: added/updated 1 cluster(s), skipped 0 unmodified cluster(s)

Hence we were able to reload the envoy configuration dynamically without restarting the server itself.

Teaching kubernetes service mesh tricks

Ishan Khare — Tue, 02 Mar 2021 21:03:21 +0000

What we'll cover in this post?

This post in general is about how we can leverage kubernetes controllers and the kubernetes resource model to automate and manage heavy lifting of tricky deployment and management scenarios.

Controller are often explained in layman terms as software-SRE's. The good thing about that is:

They are always available and ready to step in when things go wrong.
They have all the operational logic in the form of code related to the steps to take when certain things don't work.
They don't need to go over runbooks/playbooks or other documentations to fix issues.
Since they are always watching resources and their states (as we'll see later in this post), they don't have to spend time to find where the errors/misconfiguration happened.

This can give us a few positive outcomes:

Humans don't have to be on hectic on-call schedules.
In cases of errors, the software itself is healing the system, hence the MTTR(mean time to response) is considerably decreased.

"With great power comes great responsibilities". With all the advantages of implementing controllers, they also come with high cost of implementation details.
Implementing controllers is generally more complex than traditional systems.

Now that we know a bit about controllers, we'll take a use case and try to implement a controller to solve it.

Tl;Dr

If you want to directly go ahead and checkout the code implementation and test it on your local machine, I have the entire thing residing
in this – github.com/ishankhare07/launchpad repo. There is a detailed readme which walks you through
a Makefile that:

Sets up a local kind cluster
Installs Istio on it
Deploys demo workloads(Deployments)
Applies the right Istio CRD's – a VirtualService and a DestinationRule to achieve a traffic split.
Check and verify the traffic split.

It then also walks you through the controller way of doing things and also the magic of Software-SRE's that they bring along.

Use case

The use case we are trying to tackle here is doing a traffic split using a a service mesh. Since this blog post can only have limited scope, what we will NOT talk about is:

Service mesh in general or service mesh comparison. We'll just go ahead and use Istio.
We'll not go into the details of how Istio or any other service mesh works.

The specific scenario we would want to cover is a Traffic Split/Canary. More details can be found by giving a look at this awesome example site – istiobyexample.dev/canary – by @askmeegs. Borrowing the following screenshot from that same article, demonstrating what we want to achieve:
. The page also details how we can achieve this by creating Istio specific CRD's. We'll look at both the approaches and see the advantages first-hand where controllers come in.

Below are the Istio CRD's that we would want to create in the cluster in order for Istio to do things for us:

DestinationRule

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: helloworld
spec:
  host: helloworld
  subsets:
    - name: v1
      labels:
        version: v1
    - name: v2
      labels:
        version: v2

VirtualService

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: helloworld
spec:
  hosts:
  - helloworld
  http:
  - route:
    - destination:
        host: helloworld
        subset: v1
      weight: 80
    - destination:
        host: helloworld
        subset: v2
      weight: 20

Too much going around the core of this post, lets dive in now.

This post uses the awesome kubebuilder framework for implementing the controller. If you're unfamiliar about it and want an intro
I've written a previous article about it on my blog which you can read here – Writing a kubernetes controller in Go with kubebuilder

The Deep Dive

We'll start by initializing our project for the controller using go mod and the kubebuilder framework

$ go mod init github.com/ishankhare07/launchpad
$ kubebuilder init --domain ishankhare.dev

# create scaffolding for the project
$ kubebuilder create api --group labs --version v1alpha1 --kind Workload
Create Resource [y/n]
y
Create Controller [y/n]
y
Writing scaffold for you to edit...

The CRD

Next step is to define the CRD which we want to use for defining our traffic split to our controller. We want roughly the following yaml spec for it:

apiVersion: labs.ishankhare.dev/v1alpha1
kind: Workload
metadata:
  name: workload-sample
spec:
  host: helloworld
  targets:
    - name: helloworld-v1
      namespace: default
      trafficSplit:
        hosts:
          - helloworld
        subset:
          name: v1
          labels:
            version: v1
        destinationHost: helloworld
        weight: 80
    - name: helloworld-v2
      namespace: default
      trafficSplit:
        hosts:
          - helloworld
        subset:
          name: v2
          labels:
            version: v2
        destinationHost: helloworld
        weight: 20

I know there's room for improvement and this can be refined further, but for the purpose of this post, this should do fine. Lets go about defining this spec in our api/v1alpha1/workload_types.go

...

const (
    PhasePending = "PENDING"
    PhaseCreated = "CREATED"
)

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status

// Workload is the Schema for the workloads API
type Workload struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   WorkloadSpec   `json:"spec,omitempty"`
    Status WorkloadStatus `json:"status,omitempty"`
}

type WorkloadSpec struct {
    // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
    // Important: Run "make" to regenerate code after modifying this file

    // Host is the name of the service from the service registry
    Host string `json:"host,omitempty"`

    Targets []DeploymentTarget `json:"targets,omitempty"`
}

// WorkloadStatus defines the observed state of Workload
type WorkloadStatus struct {
    // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
    // Important: Run "make" to regenerate code after modifying this file
    VirtualServiceState  string `json:"virtualServiceState,omitempty"`
    DestinationRuleState string `json:"destinationRuleState,omitempty"`
}

type DeploymentTarget struct {
    Name         string       `json:"name,omitempty"`
    Namespace    string       `json:"namespace,omitempty"`
    TrafficSplit TrafficSplit `json:"trafficSplit,omitempty"`
}

type TrafficSplit struct {
    Hosts           []string          `json:"hosts,omitempty"`
    Subset          Subset            `json:"subset,omitempty"`
    DestinationHost string            `json:"destinationHost,omitempty"`
    Weight          int               `json:"weight,omitempty"`
}

type Subset struct {
    Name   string            `json:"name,omitempty"`
    Labels map[string]string `json:"labels,omitempty"`
}

func (w *Workload) GetIstioResourceName() string {
    return w.Spec.Host // strings.Split(w.Spec.Targets[0].Name, "-")[0]
}

func (w *Workload) GetHosts() []string {
    // need hosts as a set data structure
    hostSet := make(map[string]bool)
    hosts := []string{}

    for _, target := range w.Spec.Targets {
        for _, host := range target.TrafficSplit.Hosts {
            hostSet[host] = true
        }
    }

    for host := range hostSet {
        hosts = append(hosts, host)
    }

    return hosts
}

This should be enough to define our define the yaml that we presented before. If you're following along the code in the repo, the above is defined in api/v1alpha1/workload_types.go. We also add some additional helper methods on the objects that will be useful later.

The controller

Let's jump to controllers/workload_controller.go. I'm only going to show the relevant sections of the code here in favour of brevity. For detailed code one can go and see the linked file.

...


func (r *WorkloadReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    reqLogger := r.Log.WithValues("workload", req.NamespacedName)

    // your logic here
    reqLogger.Info("====== Reconciling Workload =======")
    instance := &labsv1alpha1.Workload{}

    err := r.Get(context.TODO(), req.NamespacedName, instance)
    if err != nil {
        // object not found, could have been deleted after
        // reconcile request, hence don't requeue
        if errors.IsNotFound(err) {
            return ctrl.Result{}, nil
        }
        // error reading the object, requeue the request
        return ctrl.Result{}, err
    }

    reqLogger.Info("Workload created for following targets.")
    for _, target := range instance.Spec.Targets {
        reqLogger.Info("target",
            "name", target.Name,
            "namespace", target.Namespace,
            "hosts", fmt.Sprintf("%v", target.TrafficSplit.Hosts))
        reqLogger.Info("subset",
            "subset name", target.TrafficSplit.Subset.Name,
            "subset labels", fmt.Sprintf("%v", target.TrafficSplit.Subset.Labels),
            "weight", target.TrafficSplit.Weight)
    }

    // check current status of owned resources
    if instance.Status.VirtualServiceState == "" {
        instance.Status.VirtualServiceState = labsv1alpha1.PhasePending
    }

    if instance.Status.DestinationRuleState == "" {
        instance.Status.DestinationRuleState = labsv1alpha1.PhasePending
    }

    // check virtual service status
    vsResult, err := r.checkVirtualServiceStatus(instance, req, reqLogger)
    if err != nil {
        return vsResult, err
    }

    // check destination rule status
    destRuleResult, err := r.checkDestinationRuleStatus(instance, req, reqLogger)
    if err != nil {
        return destRuleResult, err
    }

    // update status
    err = r.Status().Update(context.TODO(), instance)
    if err != nil {
        return ctrl.Result{}, err
    }

    return ctrl.Result{}, nil
}

Here we are simply logging what the controller "sees" is requested to be created as "desired state" through the CRD.
Next we are calling the relevant functions checkVirtualServiceStatus and checkDestinationRuleStatus. There implementation is explained below, with just the following intention:

If the VirtualService or DestinationRule doesn't exist yet, try to create it with the correct spec.

The VirtualService check

In the same file let's implement the checkVirtualServiceStatus method, please scroll to the right for extra comments explaining each section:

...

func (r *WorkloadReconciler) checkVirtualServiceStatus(instance *labsv1alpha1.Workload, req ctrl.Request, logger logr.Logger) (ctrl.Result, error) {
    switch instance.Status.VirtualServiceState {
    case labsv1alpha1.PhasePending:
        logger.Info("Virtual Service", "PHASE:", instance.Status.VirtualServiceState)           // If still in pending state (initial condition)
        logger.Info("Transitioning state to create Virtual Service")                            // transition to the creating state  
        instance.Status.VirtualServiceState = labsv1alpha1.PhaseCreated                         //
    case labsv1alpha1.PhaseCreated:
        logger.Info("Virtual Service", "PHASE:", instance.Status.VirtualServiceState)               //
        query := &istiov1alpha3.VirtualService{}                                                    // create an empty query object
                                                                                                    //
        // check if virtual service already exists                                                  //---------\
        lookupKey := types.NamespacedName{                                                          //          \
            Name:      instance.GetIstioResourceName(),                                             //           --- construct the query object
            Namespace: instance.GetNamespace(),                                                     //          /
        }                                                                                           //---------/
        err := r.Get(context.TODO(), lookupKey, query)                                              // query result
        if err != nil && errors.IsNotFound(err) {                                                   //
            logger.Info("virtual service not found but should exist", "lookup key", lookupKey)      //
            logger.Info(err.Error())                                                                //
            // virtual service got deleted or hasn't been created yet                               //
            // create one now                                                                       //
            vs := spawn.CreateVirtualService(instance)                                              // create the actual virtual service as defined by Istio
            err = ctrl.SetControllerReference(instance, vs, r.Scheme)                               // Set owner references on the created object
            if err != nil {                                                                         //
                logger.Error(err, "Error setting controller reference")                             //
                return ctrl.Result{}, err                                                           //
            }                                                                                       //
                                                                                                    // 
            err = r.Create(context.TODO(), vs)                                                      // request the API server to create the actual object in
            if err != nil {                                                                         // the cluster i.e. persist in the etcd store
                logger.Error(err, "Unable to create Virtual Service")                               //
                return ctrl.Result{}, err                                                           //
            }                                                                                       //
                                                                                                    //
            logger.Info("Successfully created virtual service")                                     //
        } else if err != nil {                                                                      //
            logger.Error(err, "Unable to create Virtual Service")                                   //
            return ctrl.Result{}, err                                                               //
        } else {                                                                                    //
            // don't requeue, it will happen automatically when                                     //
            // virtual service status changes                                                       //
            return ctrl.Result{}, nil                                                               //
        }                                                                                           //

        // more fields related to virtual service status can be checked
        // see more at https://pkg.go.dev/istio.io/api/meta/v1alpha1#IstioStatus

    }

    return ctrl.Result{}, nil
}

The line vs := spawn.CreateVirtualService(instance) defines another function to create the actual Istio VirtualService for us which we will define further in the spawn package.

The DestinationRule check

Similar to the virtual service, we'll create the checkDestinationRuleStatus method in the same file:

...

func (r *WorkloadReconciler) checkDestinationRuleStatus(instance *labsv1alpha1.Workload, req ctrl.Request, logger logr.Logger) (ctrl.Result, error) {
    switch instance.Status.DestinationRuleState {
    case labsv1alpha1.PhasePending:
        logger.Info("Destination Rule", "PHASE:", instance.Status.DestinationRuleState)
        logger.Info("Transitioning state to create Destination Rule")
        instance.Status.DestinationRuleState = labsv1alpha1.PhaseCreated
    case labsv1alpha1.PhaseCreated:
        logger.Info("Destination Rule", "PHASE:", instance.Status.DestinationRuleState)
        query := &istiov1alpha3.DestinationRule{}

        // check if destination rule already exists
        lookupKey := types.NamespacedName{
            Name:      instance.GetIstioResourceName(),
            Namespace: instance.GetNamespace(),
        }
        err := r.Get(context.TODO(),
            lookupKey, query)
        if err != nil && errors.IsNotFound(err) {
            logger.Info("destination rule not found but should exist", "req key", lookupKey)
            logger.Info(err.Error())
            // destination rule got deleted or hasn't been created yet
            // create one now

            dr := spawn.CreateDestinationRule(instance)
            err := ctrl.SetControllerReference(instance, dr, r.Scheme)
            if err != nil {
                logger.Error(err, "Error setting controller reference")
                return ctrl.Result{}, err
            }

            err = r.Create(context.TODO(), dr)
            if err != nil {
                logger.Error(err, "Unable to create Destination Rule")
                return ctrl.Result{}, err
            }

            logger.Info("Successfully created Destination Rule")
        } else if err != nil {
            logger.Error(err, "Unable to create destination rule")
            return ctrl.Result{}, err
        } else {
            // don't requeue, it will happen automatically when
            // destination rule status changes
            return ctrl.Result{}, nil
        }
    }

    return ctrl.Result{}, nil
}

The line dr := spawn.CreateDestinationRule(instance) defines another function to create the actual Istio DestinationRule for us, which we will define further in the spawn package.

Creating the Istio CRD's (defining the 'spawn') package

Let's create the following two files in the pkg directory:

The structure should be as follows:

$ tree ./pkg
./pkg
└── spawn
    ├── destination-rule.go
    └── virtual-service.go

In the file virtual-service.go

package spawn

import (
    labsv1alpha1 "github.com/ishankhare07/launchpad/api/v1alpha1"
    istionetworkingv1alpha3 "istio.io/api/networking/v1alpha3"              // ---------\
                                                                            //           - import the required Istio structures (CRD's)
    istiov1alpha3 "istio.io/client-go/pkg/apis/networking/v1alpha3"         // ---------/
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

func CreateVirtualService(workload *labsv1alpha1.Workload) *istiov1alpha3.VirtualService {
    vs := &istiov1alpha3.VirtualService{
        ObjectMeta: metav1.ObjectMeta{
            Name:      workload.GetIstioResourceName(),
            Namespace: "default",
        },
    }

    routeDestination := []*istionetworkingv1alpha3.HTTPRouteDestination{}

    for _, target := range workload.Spec.Targets {
        // extract route from CRD
        route := &istionetworkingv1alpha3.HTTPRouteDestination{
            Destination: &istionetworkingv1alpha3.Destination{
                Host:   target.TrafficSplit.DestinationHost,
                Subset: target.TrafficSplit.Subset.Name,
            },
            Weight: int32(target.TrafficSplit.Weight),
        }

        // append routes into route destination
        routeDestination = append(routeDestination, route)
    }

    // append route destination to VirtualService.Routes
    vs.Spec.Http = append(vs.Spec.Http, &istionetworkingv1alpha3.HTTPRoute{
        Route: routeDestination,
    })

    // set hosts
    vs.Spec.Hosts = workload.GetHosts()

    return vs
}

We import the right structures from the Istio's upstream repo and construct the object based on the values provided in our original Workload CRD.
In the similar way we define the destination rule object generation in the file destination-rule.go

package spawn

import (
    labsv1alpha1 "github.com/ishankhare07/launchpad/api/v1alpha1"
    istionetworkingv1alpha3 "istio.io/api/networking/v1alpha3"
    istiov1alpha3 "istio.io/client-go/pkg/apis/networking/v1alpha3"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

func CreateDestinationRule(workload *labsv1alpha1.Workload) *istiov1alpha3.DestinationRule {
    dr := &istiov1alpha3.DestinationRule{
        ObjectMeta: metav1.ObjectMeta{
            Name:      workload.GetIstioResourceName(),
            Namespace: workload.GetNamespace(),
        },
    }

    for _, target := range workload.Spec.Targets {
        dr.Spec.Subsets = append(dr.Spec.Subsets, &istionetworkingv1alpha3.Subset{
            Name:   target.TrafficSplit.Subset.Name,
            Labels: target.TrafficSplit.Subset.Labels,
        })
    }

    dr.Spec.Host = workload.Spec.Host

    return dr
}

Telling the controller to 'watch' the resources we own

One last trick left in the implementation of our controller is, when we set the Owner references for each created VirtualService and DestinationRule in these lines:

...
err = ctrl.SetControllerReference(instance, vs, r.Scheme)
...

err := ctrl.SetControllerReference(instance, dr, r.Scheme)
...

We are essentially telling kubernetes that these objects are "owned" by THIS controller

We can then ask kubernetes to send is events related to these "OWNED" resources whenever anything happens with them (delete/update)

To do this we go back to the end of the controllers/workload_controller.go and add the following:

func (r *WorkloadReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&labsv1alpha1.Workload{}).
        Owns(&istiov1alpha3.VirtualService{}).        // we own both these type of objects
        Owns(&istiov1alpha3.DestinationRule{}).       // hence send us events related to these
        Complete(r)
}

Are we done?

One last thing. Since Istio's VirtualService and DestinationRule are external kubernetes obejcts (for which we have registered the CRD's while installing istio), we are still using raw golang objects / structs and passing it to the controller-runtime client.

The client needs some additional information to "map the golang obejcts with the registered crds".

Note that the golang objects we created don't have information like apiVersion, kind
Hence to provide this information we need to register something called as a scheme.

A Scheme basically maps the Go object to what we call GVK(GroupVersionKind) in kubernetes. This is usually auto generated when CRD's are defined using any framework like kubebuilder or operatorsdk. There's even one generated for our Workload kind which you can take a look at here – api/v1alpha1/groupversion_info.go

...

var (
    // GroupVersion is group version used to register these objects
    GroupVersion = schema.GroupVersion{Group: "labs.ishankhare.dev", Version: "v1alpha1"}

    // SchemeBuilder is used to add go types to the GroupVersionKind scheme
    SchemeBuilder = &scheme.Builder{GroupVersion: GroupVersion}

    // AddToScheme adds the types in this group-version to the given scheme.
    AddToScheme = SchemeBuilder.AddToScheme
)

A similar scheme is defined by the Istio library, which we can register in our main.go

import (
        ...
        istioscheme "istio.io/client-go/pkg/clientset/versioned/scheme"
        ...
)

...

func init() {
        ...
        _ = istioscheme.AddToScheme(scheme)
}

Now we are done! You can follow along the Controller-demo to see the magic happen!

Some final highlights

As pointed out the final sections of the demo, even if someone/something accidentally or deliberately deletes either of the VirtualService or the DestinationRule, the controller immediately steps in and recreates it for us.

One more trick left to showcase. If you were to go and delete either the VirtualService or the DestinationRule now, since the controller is watching over them, it will instantly recreate it with the correct configuration. This simulates an actual human error which have triggered alerts for the team in the past only to realise that somewhere, somebody did a kubectl delete vs helloworld. Go ahead, try it. Then try to run kubectl get vs,dr again.

Lastly I'd like to mention the awesome work being done by folks at @crossplane and @hasheddan who are building some really cool stuff leveraging the controllers and the KRM(kubernetes resource model). If you're more interested, checkout the following resources:

Kubernetes as a Framework for Control Planes.
Crossplane team integrating GKE Autopilot before the night ends on live stream.

What's next?

In a follow-up post I'll talk about how we can bring Admission Webhooks, particularly Validating Admission Webhooks into the picture and make sure nobody's able to create our Workload CRD with wrong values – like a traffic split that doesn't add up to a 100% for eg. (80:30 or 40:30)

This post was original posted on my personal blog ishankhare.dev

Writing a kubernetes controller in Go with kubebuilder

Ishan Khare — Tue, 17 Nov 2020 23:49:05 +0000

In this post we'll implement a simple kubernetes controller using the kubebuilder

Installing kubebuilder

The installation instructions can be found on kubebuilder installation docs

Install kustomize

Since kubebuilder internally relies on kustomize, hence we need to install that from the instructions here

Now we're all set up to start building our controller

Start the project

I have initialized an empty go project

go mod init cnat

Next we'll initialize the controller project

kubebuilder init --domain ishankhare.dev

Now we'll ask kubebuilder to setup all the necessary scaffolding for our project

kubebuilder create api --group cnat --version v1alpha1 --kind At

Kubebuilder will ask us whether to create Resource and Controller directories, to which we can answer y each, since we want these to be created

Create Resource [y/n]
y
Create Controller [y/n]
y
Writing scaffold for you to edit...

Test install scaffold

If we now run make install, kubebuilder should generate the base CRDs under config/crd/bases and a few other files for us.

Running make run should now allow us to launch the operator locally. This is really helpful for local testing while we will be implementing and debugging the business logic of our operator code. The live logs will give us insights about how our codebase changes reflect to what's happening with the operator.

Enable status subresource

In the file api/v1alpha1/at_types.go add the following comment above At struct.

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status

// At is the Schema for the ats API
type At struct {
.
.
.

This will enable us to use privilege separation between spec and status fields.

Background

A little background on what we're going to build here. Our controller is basically a 'Cloud Native At' (cnat) for short – a cloud native version of the Unix's at command. We provide a CRD manifest containing 2 main things:

A command to execute
Time to execute the command

Our controller will basically wait and watch till the appropriate time, then would spawn a Pod and run the given command in that Pod. All this will it will also keep updating the Status of that CRD which we can view with kubectl

So let's begin

`CRD` outline

This section basically describes how we want our CRD to look like. Given the two points we listed above, we can basically represent them as:

apiVersion: cnat.ishankhare.dev/v1alpha1
kind: At
metadata:
  name: at-sample
spec:
  schedule: "2020-11-16T10:12:00Z"
  command: "echo hello world!"

This is enough information for our controller to decide 'when' and 'what' to execute.

Defining the `CRD`

Now that we know what we expect out of the CRD let's implement that. Goto the file api/v1alpha1/at_types.go and add the following fields to the code:

const (
    PhasePending = "PENDING"
    PhaseRunning = "RUNNING"
    PhaseDone    = "DONE"
)

Also we edit the existing AtSpec and AtStatus structs as follows:

type AtSpec struct {
  Schedule string `json:"schedule,omitempty"`
  Command  string `json:"command,omitempty"`
}

type AtStatus struct {
  Phase string `json:"phase,omitempty"`
}

We save this as we run

$ make manifests
go: creating new go.mod: module tmp
go: found sigs.k8s.io/controller-tools/cmd/controller-gen in sigs.k8s.io/controller-tools v0.2.5
/Users/ishankhare/godev/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases

$ make install    # this will install the generated CRD into k8s

We should now be able to see an exhaustive CustomResourceDefinition generated for us at config/crd/bases/cnat.ishankhare.dev_ats.yaml. For people new to CRD's, they will basically help kubernetes identify what our kind: At means. Unlike builtin resource types like Deployments, Pods etc. we need to define custom types for kubernetes through a CustomResourceDefinition. Let's try to create an At resource:

cat >sample-at.yaml<<EOF
apiVersion: cnat.ishankhare.dev/v1alpha1
kind: At
metadata:
  name: at-sample
spec:
  schedule: "2020-11-16T10:12:00Z"
  command: "echo hello world!"
EOF

If we now do a kubectl apply -f to the previously shown CRD:

kubectl apply -f sample-at.yaml
kubectl get at
NAME        AGE
at-sample   124m

So now kubernetes recognizes our custom type, but it still does NOT know what to do with it. The logic part of the controller is still missing, but we have the skeleton ready to start writing our logic around it.

Implementing the Controller logic

Now we're coming to the fun part – implementing the logic for our operator. This will basically tell kubernetes what to do with our generated CRD. As few say its like "making kubernetes do tricks!".

Most of the scaffolding as been already generated for us by kubebuilder in main.go and controllers/. Inside the controllers/at_controller.go file we have the function Reconcile. Consider this is the reconcile loop's body.

A small refresher – a controller basically runs a periodic loop over the said objects and watches for any changes to those objects. When changes happen the loop body is triggered and the relevant logic can be executed. This function Reconcile is that logical piece, the entry-point for all our controller logic.

Let's start adding a basic structuring around it:

func (r *AtReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
    reqLogger := r.Log.WithValues("at", req.NamespacedName)
    reqLogger.Info("=== Reconciling At")

    instance := &cnatv1alpha1.At{}
    err := r.Get(context.TODO(), req.NamespacedName, instance)
    if err != nil {
        if errors.IsNotFound(err) {
            // object not found, could have been deleted after
            // reconcile request, hence don't requeue
            return ctrl.Result{}, nil
        }

        // error reading the object, requeue the request
        return ctrl.Result{}, err
    }

    // if no phase set, default to Pending
    if instance.Status.Phase == "" {
        instance.Status.Phase = cnatv1alpha1.PhasePending
    }

    // state transition PENDING -> RUNNING -> DONE


    return ctrl.Result{}, nil
}

Here we basically create a logger and an empty instance of the At object, which will allow use to query kubernetes for existing At objects and read/write to their status etc. The function is supposed to return the result of the reconciliation logic and an error. The ctrl.Result object can optionally tell the reconciler to either Requeue immediately (bool) or RequeueAfter with a time.Duration – again these are optional.

Now lets come to the important part, the state diagram. The image below explains what basically happens when we create a new At:

Now, lets start implementing this logic in the function:

...
// state transition PENDING -> RUNNING -> DONE
switch instance.Status.Phase {
    case cnatv1alpha1.PhasePending:
        reqLogger.Info("Phase: PENDING")

        diff, err := schedule.TimeUntilSchedule(instance.Spec.Schedule)
        if err != nil {
            reqLogger.Error(err, "Schedule parsing failure")

            return ctrl.Result{}, err
        }

        reqLogger.Info("Schedule parsing done", "Result", fmt.Sprintf("%v", diff))

        if diff > 0 {
            // not yet time to execute, wait until scheduled time
            return ctrl.Result{RequeueAfter: diff * time.Second}, nil
        }

        reqLogger.Info("It's time!", "Ready to execute", instance.Spec.Command)
        // change state
        instance.Status.Phase = cnatv1alpha1.PhaseRunning

The schedule.TimeUntilSchedule function is simple to implement like so:

package schedule

import "time"

func TimeUntilSchedule(schedule string) (time.Duration, error) {
    now := time.Now().UTC()
    layout := "2006-01-02T15:04:05Z"
    scheduledTime, err := time.Parse(layout, schedule)
    if err != nil {
        return time.Duration(0), err
    }

    return scheduledTime.Sub(now), nil
}

The next case in our switch body is for RUNNING phase. This is the most complicated one and I've added comments wherever I can to explain it better.

case cnatv1alpha1.PhaseRunning:
        reqLogger.Info("Phase: RUNNING")

        pod := spawn.NewPodForCR(instance)
        err := ctrl.SetControllerReference(instance, pod, r.Scheme)
        if err != nil {
            // requeue with error
            return ctrl.Result{}, err
        }

        query := &corev1.Pod{}
        // try to see if the pod already exists
        err = r.Get(context.TODO(), req.NamespacedName, query)
        if err != nil && errors.IsNotFound(err) {
            // does not exist, create a pod
            err = r.Create(context.TODO(), pod)
            if err != nil {
                return ctrl.Result{}, err
            }

            // Successfully created a Pod
            reqLogger.Info("Pod Created successfully", "name", pod.Name)
            return ctrl.Result{}, nil
        } else if err != nil {
            // requeue with err
            reqLogger.Error(err, "cannot create pod")
            return ctrl.Result{}, err
        } else if query.Status.Phase == corev1.PodFailed ||
            query.Status.Phase == corev1.PodSucceeded {
            // pod already finished or errored out`
            reqLogger.Info("Container terminated", "reason", query.Status.Reason,
                "message", query.Status.Message)
            instance.Status.Phase = cnatv1alpha1.PhaseDone
        } else {
            // don't requeue, it will happen automatically when
            // pod status changes
            return ctrl.Result{}, nil
        }

We set the ctrl.SetControllerReference that basically tells kubernetes runtime that the created Pod is "owned" by this At instance. This is later going to come handy for us when watching on resources created by our Controller.
We also use another external function here spawn.NewPodForCR which is basically responsible for spawning the new Pod for our At custom resource spec. This is implemented as follows:

package spawn

import (
    cnatv1alpha1 "cnat/api/v1alpha1"
    "strings"

    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

func NewPodForCR(cr *cnatv1alpha1.At) *corev1.Pod {
    labels := map[string]string{
        "app": cr.Name,
    }

    return &corev1.Pod{
        ObjectMeta: metav1.ObjectMeta{
            Name:      cr.Name,
            Namespace: cr.Namespace,
            Labels:    labels,
        },
        Spec: corev1.PodSpec{
            Containers: []corev1.Container{
                {
                    Name:    "busybox",
                    Image:   "busybox",
                    Command: strings.Split(cr.Spec.Command, " "),
                },
            },
            RestartPolicy: corev1.RestartPolicyOnFailure,
        },
    }
}

The last bits of code deal with the DONE phase and the default case of switch. We also do the updation of our status subresource outside the switch as a common part of code:

case cnatv1alpha1.PhaseDone:
        reqLogger.Info("Phase: DONE")
        // reconcile without requeuing
        return ctrl.Result{}, nil
    default:
        reqLogger.Info("NOP")
        return ctrl.Result{}, nil
    }

    // update status
    err = r.Status().Update(context.TODO(), instance)
    if err != nil {
        return ctrl.Result{}, err
    }

    return ctrl.Result{}, nil

Lastly we utilize the SetControllerReference by adding the following to the SetupWithManager function in the same file:

err := ctrl.NewControllerManagedBy(mgr).
        For(&cnatv1alpha1.At{}).
        Owns(&corev1.Pod{}).
        Complete(r)

    if err != nil {
        return err
    }

    return nil

The part specifying Owns(&corev1.Pod{}) is important and tells the controller manager that pods created by this controller also needs to be watched for changes.

If you want to have a look at the entire final code implementation, head over the this github repo – ishankhare07/kubebuilder-controller.
With this in place, we are now done with implementation of our controller, we can run and test it on a cluster in our current kube context using make run

go: creating new go.mod: module tmp
go: found sigs.k8s.io/controller-tools/cmd/controller-gen in sigs.k8s.io/controller-tools v0.2.5
/Users/ishankhare/godev/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
/Users/ishankhare/godev/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go run ./main.go
2020-11-18T05:05:45.531+0530    INFO    controller-runtime.metrics      metrics server is starting to listen    {"addr": ":8080"}
2020-11-18T05:05:45.531+0530    INFO    setup   starting manager
2020-11-18T05:05:45.531+0530    INFO    controller-runtime.manager      starting metrics server {"path": "/metrics"}
2020-11-18T05:05:45.531+0530    INFO    controller-runtime.controller   Starting EventSource    {"controller": "at", "source": "kind source: /, Kind="}
2020-11-18T05:05:45.732+0530    INFO    controller-runtime.controller   Starting EventSource    {"controller": "at", "source": "kind source: /, Kind="}
2020-11-18T05:05:46.232+0530    INFO    controller-runtime.controller   Starting Controller     {"controller": "at"}
2020-11-18T05:05:46.232+0530    INFO    controller-runtime.controller   Starting workers        {"controller": "at", "worker count": 1}
2020-11-18T05:05:46.232+0530    INFO    controllers.At  === Reconciling At      {"at": "cnat/at-sample"}
2020-11-18T05:05:46.232+0530    INFO    controllers.At  Phase: PENDING  {"at": "cnat/at-sample"}
2020-11-18T05:05:46.232+0530    INFO    controllers.At  Schedule parsing done   {"at": "cnat/at-sample", "Result": "-14053h23m46.232658s"}
2020-11-18T05:05:46.232+0530    INFO    controllers.At  It's time!      {"at": "cnat/at-sample", "Ready to execute": "echo YAY!"}
2020-11-18T05:05:46.436+0530    DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "at", "request": "cnat/at-sample"}
2020-11-18T05:05:46.443+0530    INFO    controllers.At  === Reconciling At      {"at": "cnat/at-sample"}
2020-11-18T05:05:46.443+0530    INFO    controllers.At  Phase: RUNNING  {"at": "cnat/at-sample"}
2020-11-18T05:05:46.718+0530    INFO    controllers.At  Pod Created successfully        {"at": "cnat/at-sample", "name": "at-sample"}
2020-11-18T05:05:46.718+0530    DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "at", "request": "cnat/at-sample"}
2020-11-18T05:05:46.722+0530    INFO    controllers.At  === Reconciling At      {"at": "cnat/at-sample"}
2020-11-18T05:05:46.722+0530    INFO    controllers.At  Phase: RUNNING  {"at": "cnat/at-sample"}
2020-11-18T05:05:46.722+0530    DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "at", "request": "cnat/at-sample"}
2020-11-18T05:05:46.778+0530    INFO    controllers.At  === Reconciling At      {"at": "cnat/at-sample"}
2020-11-18T05:05:46.778+0530    INFO    controllers.At  Phase: RUNNING  {"at": "cnat/at-sample"}
2020-11-18T05:05:46.778+0530    DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "at", "request": "cnat/at-sample"}
2020-11-18T05:05:46.846+0530    INFO    controllers.At  === Reconciling At      {"at": "cnat/at-sample"}
2020-11-18T05:05:46.847+0530    INFO    controllers.At  Phase: RUNNING  {"at": "cnat/at-sample"}
2020-11-18T05:05:46.847+0530    DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "at", "request": "cnat/at-sample"}
2020-11-18T05:05:49.831+0530    INFO    controllers.At  === Reconciling At      {"at": "cnat/at-sample"}
2020-11-18T05:05:49.831+0530    INFO    controllers.At  Phase: RUNNING  {"at": "cnat/at-sample"}
2020-11-18T05:05:49.831+0530    INFO    controllers.At  Container terminated    {"at": "cnat/at-sample", "reason": "", "message": ""}
2020-11-18T05:05:50.054+0530    DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "at", "request": "cnat/at-sample"}
2020-11-18T05:05:50.056+0530    INFO    controllers.At  === Reconciling At      {"at": "cnat/at-sample"}
2020-11-18T05:05:50.056+0530    INFO    controllers.At  Phase: DONE     {"at": "cnat/at-sample"}
2020-11-18T05:05:50.056+0530    DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "at", "request": "cnat/at-sample"}

If you're interested to see the output command we can run:

$ kubectl logs -f at-sample
hello world!

This post was first published on my personal blog ishankhare.dev

Deploy a golang API on Kubernetes with Nginx ingress

Ishan Khare — Thu, 28 Nov 2019 09:28:58 +0000

This blog post is a follow up on my previous post where we talked about deploying a Python app with nginx reverse proxy using docker.

So lets line up all the things that we're going to touch in this post:

Now that we have set the tone, lets go in brief about what the above this are about that what role they play.
We will then actually go over each and code them and implement the deployment in kubernetes.

What is Ingress?

From the kubernetes documentation Ingress is defined as:

Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource.

Also this diagram seems to provide a bit more details so I'll just leave it here:

    internet
        |
   [ Ingress ]
   --|-----|--
   [ Services ]

All though IMO the above diagram is a bit more simplified. Hence I came up with this arch. diagram explaining the above arrangement in a bit more detail.

Another way to look at Ingress is that its a Layer 7 load balancer. As layer 7 LB is application aware, it can determine where to send the traffic depending on application state.

In contrast, a simple load balancer like Google cloud load balancer or AWS' ELB are Layer 4 LB. You would use it to expose single app or service to the outside world. It would balance the load based
on destination IP address, protocol and port.

What are services?

Without going into much detail about what services are, k8s docs describe services as:

An abstract way to expose an application running on a set of Pods as a network service.

And Deployments?

Deployment are merely high level abstractions over Pods (which in themselves are abstractions over your containers). Well that's only half correct.

In fact ReplicaSets are a direct abstractions on your Pods and Deployments are an abstraction over ReplicaSets.
Again from the docs:

You describe a desired state in a Deployment, and the Deployment Controller changes the actual state to the desired state at a controlled rate. You can define Deployments to create new ReplicaSets, or to remove existing Deployments and adopt all their resources with new Deployments.

So lets dive right in

Let's first see how our Go code is going to look like:

package main

import (
    "fmt"
    "net/http"

    "github.com/gin-gonic/gin"
)

func greet(c *gin.Context) {
    name := c.Param("name")
    c.String(http.StatusOK, fmt.Sprintf("Hello %s!", name))
}

func stranger(c *gin.Context) {
    c.String(http.StatusOK, "Hello stranger!")
}

func main() {
    router := gin.Default()

    router.GET("/hello/:name", greet)
    router.GET("/hello/", stranger)
    router.GET("/", func(c *gin.Context) {
        c.String(http.StatusOK, "")
    })
    router.Run(":8000")
}

This is a very trivial api, we add three endpoints. The root url is basically also a health check and hence must return a 200 OK http status.

Once we have this go code built into a container – which BTW is not the subject of this post and I'm skipping it out, we can start concentrating on more about how to deploy
this service on kubernetes.

I'm going to deploy this to a k8s cluster on Google Cloud, but the code should work seamlessly on any other k8s deployments as well.

Also, I've created a container with the above code and made it available on docker hub publicly. The container can be found here – https://hub.docker.com/repository/docker/ishankhare07/ingress-test

You can pull down the image using docker pull ishankhare07/ingress-test:0.0.3

The cluster

So lets get down to business, I'm assuming you have your cluster running and kubectl command configured to connect to the cluster. We'll be setting up an

ingress resource
ingress controller

While there are many plug-and-play options for ingress available today written over the ingress api, the simplest and most straight forward one is nginx ingress.
I've given the link to official docs which details the exact installation steps for setting it up. But these steps vary slightly depending on the cloud provider, k8s version etc.

Hence the best and easiest way to actually setup nginx ingress is through helm

WTF is helm?

Helm is a package manager for k8s. Consider it like apt-get or yum but for kubernetes. Just tell helm the package – which are called charts in helm terminology to be installed, and helm will do it for you. The good this about helm is – its sort of an automation tool over kubernetes. So if you have something that requires multiple steps to setup, helm will do it for you – like run a few deployments, expose it at a port through a service, mount a volume from a persistent disk etc. More on helm some other day. Right now go ahead and install helm on your cluster.

And yes you read that right, "install on cluster". You see helm is a 2 part thing

The helm cli client that is installed on your dev machine.
The tiller controllers that are spun as pod controllers inside your cluster

The way it works is you tell helm what to install using the helm cli. The cli then communicates this to the tiller controllers running on your k8s cluster.
Those controllers fetch and spin the required pods etc. in the cluster for you. In a way we are using kubernetes provided primitives to automate and extend kubernetes itself.

Once you've helm installed and setup, we can now proceed with deploying our application.

Deploying our go backend

We want out go backend to be available on port 8000 because that is what we exposed in our code.
So lets write a greeter.yaml file with the following content

kind: Deployment
apiVersion: apps/v1
metadata:
  name: go-greeter
  labels:
    app: greeter
spec:
  replicas: 3
  selector:
    matchLabels:
      app: greeter
  template:
    metadata:
      labels:
        app: greeter
    spec:
      containers:
      - name: greeter
        image: ishankhare07/ingress-test:0.0.3
        command: ["/go/bin/ingress-test"]
        ports:
          - containerPort: 8000

---

kind: Service
apiVersion: v1
metadata:
  name: greeter-service
spec:
  selector:
      app: greeter
  type: NodePort
  ports:
    - port: 8000

We define a Deployment – with a replicaset of 3 pods. Each pod's port 8000 is open only accessible from inside the cluster. Next we expose a service that funnels all the incoming data to the 3 replicas of our containers in a round robin fashion.

Lets deploy this to our cluster first:
kubectl apply -f greeter.yaml

Lets see the deployment:

$ kubectl get deploy
NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
go-greeter                      3/3     3            3           13d

Also we deployed a service, lets check that too:

$ kubectl get svc
NAME                            TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)                      AGE
greeter-service                 NodePort       10.36.7.174   <none>          8000:31269/TCP               13d

So our Deployment is up and running and is exposed inside the cluster through a service. Next we need to connect it to the outside world through Ingress.

Installing nginx ingress through helm

Assuming helm is installed and setup with RBAC roles applied, we can install nginx-ingress chart with the following command:
helm install --name nginx-ingress stable/nginx-ingress --set rbac.create=true --set controller.publishService.enabled=true

If all goes well, this should give us the following output:

NAME                           TYPE          CLUSTER-IP    EXTERNAL-IP  PORT(S)                     AGE
nginx-ingress-controller       LoadBalancer  10.7.248.226  pending      80:30890/TCP,443:30258/TCP  1s
nginx-ingress-default-backend  ClusterIP     10.7.245.75   none         80/TCP                      1s

So we have our ingress controller and the default backend setup and running. Now all that's left is telling this nginx-ingress-controller how and where to redirect incoming traffic.
Let's write a new file ingress.yaml for it:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
    - http:
      paths:
        - path: /hello/
          backend:
            serviceName: greeter-service
            servicePort: 8000

Let's deploy this now:
kubectl apply -f ingress.yaml

This will create and ingress and a loadbalancer service and attach a public IP address, which can be seen by:

$ kubectl get ingress
NAME            HOSTS   ADDRESS         PORTS   AGE
my-ingress   *       XXX.XXX.XXX.XXX   80      11d

Now, if we visit the above IP Address we'll hit the default backend for 404 – since / url is not defined in our ingress.

Now on hitting the first path /hello, this is what we get:

Also we have another internal API endpoint in our greeter service, lets hit that too – /hello/ishan, what we get:

Now that we are able to direct traffic to different deployments using the ingress, let's deploy a new pod and try to redirect some traffic there. We'll simply use hashicorp/http-echo
to use as a second service. Let's write a manifest for this:
apple.yaml

kind: Pod
apiVersion: v1
metadata:
  name: apple-app
  labels:
    app: apple
spec:
  containers:
    - name: apple-app
      image: hashicorp/http-echo
      args:
        - "-text=apple"

---

kind: Service
apiVersion: v1
metadata:
  name: apple-service
spec:
  selector:
    app: apple
  type: NodePort
  ports:
    - port: 5678

Let's deploy this:
kubectl apply -f apple.yaml

Now that we have this pod running and exposed through the service, we would want to update the ingress file with these changes.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: local-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
    - http:
        paths:
          - path: /apple
            backend:
              serviceName: apple-service
              servicePort: 5678
          - path: /hello/
            backend:
              serviceName: greeter-service
              servicePort: 8000

Re-configure the ingress by running:

$ kubectl apply -f ingress.yaml
ingress.extensions/my-ingress configured

We can test it by going to the path /apple

And that's it! I'm sure we can all appreciate the loosely coupled nature of kubernetes and the scalability it provides. In the next blog post we'll look into deploying a gRPC endpoint written in Go deployed on this same k8s cluster behind the ingress. We'll also try to connect it to via a python client.

This post was originally published on my blog at ishankhare.com

Go Concurrency pipelines

Ishan Khare — Mon, 19 Aug 2019 09:09:32 +0000

In this blog we're going to explore some design patterns related to concurrency pipelines in Go

We'll take a real world use case as an example to begin with and try to optimize it with goroutines first.
Next we'll see the challenges that arise and how to debug them.
Finally we'll introduce another redesign that will enable us to collect stats about all the processing that we do in the above steps.

The problem statement

Let's say we have a set of users, and we want to make a few calls to an external microservice to filter out the set of users based on a set of parameters.

Since this external call an be over plain HTTP or over modern alternatives like gRPC, we won't go into the implementation of those.

What we'll focus on instead is first a brute force way of optimizing this functionality.

Implementation with goroutines

To start with lets say we have a file named handler.go with the following code:
handler.go

package handler

func PerformOperationAfterFiltering(users []users) {
    // make external call and filter out users

    operationToBePerformed(users)
}

High level design

Let's say we are getting a list of 1000 users each time on an average. It would be a good idea to break down this into batches before making external calls.
What this would enable us to do is split out our network calls. Hence if any batch fails due to some network factors, other batches won't be affected. If we do

a single call and that fails, we would loose the response for the entire 1000 users. Also breaking down this process into smaller chunks would also enable us to optimize
it further in the subsequent steps – divide and conquer.

But before we do that, we need to understand how we plan to communicate the results back to the calling function, and how do we know all requests have finished?

We'll be using 2 golang constructs for this:

Channels
WaitGroup

the high-level architecture should look something like this:

Let's open a new file to write our request function
request.go

package request

import "sync"

func Request(users []Users) chan Users {
    resp := make(chan Users)
    batchSize := 250
    var batches [][]Users
    var wg sync.WaitGroup
    for batchSize < len(users) {
        users, batches := users[batchSize:], append(batches, users[0:batchSize])
    }
    batches = append(batches, batchSize)
}

Now that we have our batches created, we can start the requests for each batch in a goroutine of its own.
But everytime we start a new goroutine, we'll add (or register) it with a WaitGroup to keep track of the completion of each of the individual goroutines.

func Request(users []Users) chan Users {
    resp := make(chan Users)
    batchSize := 250
    var batches [][]Users

    for batchSize < len(users) {
        users, batches := users[batchSize:], append(batches, users[0:batchSize])
    }
    batches = append(batches, batchSize)

    var wg sync.WaitGroup

    for batch, _ := range batches {
        wg.Add(1)
        go func(batch []Users) {
            defer wg.Done()
            response, err := MakeRequestToThirdParty(batch)
            if err != nil {
                // log error and probably
                // also the batch of users that failed
            }

            for r, _ := range response {
                if r.CurrentBalance > 500 {
                    // we've got our user
                    // push it back on our channel
                    resp <- r.u
                }
            }
        }(batch)
    }

    return resp
}

Note that the function returns a channel of type Users.

Also we added a defer call to wg.Done(). This will decrement our waitgroup so that our exit stages knows when all our goroutines are done.

Before we goto the exit stage, let's see how we plan to consume the users from the channel. Channels fit right in with the golang's for loop – i.e. we can consume
objects from channel just by the for-range loop of go.
In our handler.go:

package handler

import "request"

func PerformOperationAfterFiltering(users []users) {
    // make external call and filter out users
    var filteredUsers []Users

    for user := range Request(users) {
        filteredUsers = append(filteredUsers, user)
    }

    operationToBePerformed(users)
}

That's exactly what we're doing here. range-ing over the open channel. How or when will the loop exit you may ask?
– When we close() the channel. And that's where the EXIT stage comes in.

So we go on to the exit stage. Back in our request.go, we add another closure, which makes use of our WaitGroup:

func Request(users []Users) chan Users {
    resp := make(chan Users)
    batchSize := 250
    var batches [][]Users

    for batchSize < len(users) {
        users, batches := users[batchSize:], append(batches, users[0:batchSize])
    }
    batches = append(batches, batchSize)

    var wg sync.WaitGroup

    for batch, _ := range batches {
        wg.Add(1)
        go func(batch []Users) {
            defer wg.Done()
            response, err := MakeRequestToThirdParty(batch)
            if err != nil {
                // log error and probably
                // also the batch of users that failed
            }

            for r, _ := range response {
                if r.CurrentBalance > 500 {
                    // we've got our user
                    // push it back on our channel
                    resp <- r.User
                }
            }
        }(batch)
    }

    // EXIT stage
    go func() {
        // wait for the wait group counter to drop down to 0
        // which will happen when all goroutines are done
        wg.Wait()

        // now we close the response channel
        // this will make our loop also exit
        close(resp)
    }()

    return resp
}

The code above is self-explanatory and mimics our initial arch design completely without any issues.

Where things get dim?

The current code works perfectly. But when we deploy such code to production we usually want to log and collect metrics related to what happened.
What went wrong, what worked, how many users passed, how many didn't etc.

The current design leaves little room for collecting such metrics as we'll see soon.

Collecting metrics

The simplest approach to collecting metrics is:

Build 2 maps with userids to user objects – one for users passing the condition, another for users failing the condition.
On response, add the user to the relevant map

So let's add this simple solution to our code
handler.go

func Request(users []Users) chan Users {
    resp := make(chan Users)
    batchSize := 250
    var batches [][]Users

    passedUsers := make(map[int64]Users)
    skippedUsers := make(map[int64]Users)

    for batchSize < len(users) {
        users, batches := users[batchSize:], append(batches, users[0:batchSize])
    }
    batches = append(batches, batchSize)

    var wg sync.WaitGroup

    for batch, _ := range batches {
        wg.Add(1)
        go func(batch []Users) {
            defer wg.Done()
            response, err := MakeRequestToThirdParty(batch)
            if err != nil {
                // log error and probably
                // also the batch of users that failed
            }

            for r, _ := range response {
                if r.CurrentBalance > 500 {
                    // we've got our user
                    // push it back on our channel
                    resp <- r.User

                    passedUsers[r.User.UserId] = r.User
                }

                skippedUsers[r.User.UserId] = r.User
            }
        }(batch)
    }

    // EXIT stage
    go func() {
        // wait for the wait group counter to drop down to 0
        // which will happen when all goroutines are done
        wg.Wait()

        defer LogMetrics(passedUsers, skippedUsers)

        // now we close the response channel
        // this will make our loop also exit
        close(resp)
    }()

    return resp
}

The code looks fine – we have added two maps and are writing our users to it. In the exit stage we've added a defer statement which will basically make sure that metrics are definitely registered. This code would compile without any errors and also run without errors for sometime, but there is a serious bug which will crash our code in very weird ways.

The issue

Whenever we write a concurrent piece of code in go and it panics due to some reason which seems weird, go race detector the the tool to run for.

The issue comes do to a race condition where we are trying to write to our hashmaps inside our goroutines. You see these hashmaps are shared data among the goroutines, and it will so happen that at a time more that one goroutine might try to write to these maps – which would result in a data race. And data race in in go are undefined behavior and will basically crash the whole program with not so friendly error messages. This is show in the image in red.

On running the above code with race detector enabled, we'll get some output like this:

==================
WARNING: DATA RACE
Read by goroutine 5:
  request.Request·001()
     request.go:36 +0x169

Previous write by goroutine 1:
  request.Request()
      request.go:36 +0x174

Goroutine 5 (running) created at:
  request.goFunc()
      request.go:19 +0x56
==================

This simply tells us the line nos where the data race occurred, this is the first data race, another similar one is right below it where we access the skippedUsers map.

In order to fix this, we need to redesign our pipeline architecture slightly. Which we will see in the next iteration of this post.

Nginx as reverse proxy for a flask app using Docker

Ishan Khare — Fri, 08 Mar 2019 16:44:22 +0000

So what is a reverse proxy?

A reverse proxy is a type of proxy server that retrieves resources on behalf of a client from one or more servers. These resources are then returned to the client as if they originated from the Web server itself. In this setup, the following diagram gives a better description of our architecture:

Running Nginx with docker

Lets start by running a simple nginx container listening on our localhost's default port(80). We need docker installed for this to work, for instructions on installing docker refer here.

Next we will pull nginx container image from docker hub with the following command:

docker pull nginx:1.13.7

We can see the downloaded image

$ docker images
REPOSITORY                               TAG                 IMAGE ID            CREATED             SIZE
nginx                                    1.13.7              f895b3fb9e30        5 weeks ago         108MB

Once we have nginx image, we run the container with an exposed port.

$ docker run --name nginx_test -d -p 80:80 nginx:1.13.7
66de8420687d0687f6c923269ccd1554c0d247f230dc3d064c10fcd8a099fb5d
$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                NAMES
66de8420687d        nginx:1.13.7        "nginx -g 'daemon of…"   8 minutes ago       Up 8 minutes        0.0.0.0:80->80/tcp   nginx_test

The -d in the command runs our container in daemon mode (in background) without blocking our shell.

Note that the "PORTS" columns says 0.0.0.0:80->80/tcp which means we have mapped and exposed the port 80 of our container to our host machine. Hence we should now be able to access nginx on localhost.

We can see that here:

Stop the running container

$ docker stop nginx_test

We got nginx working with a docker container. This was easy-peasy right?

docker-compose

We have more than one container, in our use case:

nginx
python

Also we want to be able to link these containers somehow (more on this very shortly). What could be a possible, easy and efficient way to achieve this - Enter docker-compose. The official docker documentation describes docker-compose as

Compose is a tool for defining and running multi-container Docker applications.

Firstly, install docker compose by going through the instructions here

Next, we'll write a docker-compose.yml file. First lets start by replicating our work until now, getting an nginx container working on localhost, but this time using docker-compose

docker-compose.yml

version: '3.1'
services:
    nginx:
        image: nginx:1.13.7
        ports:
            - 80:80

Lets test it: docker-compose up -d nginx. You can now see on localhost the same nginx welcome page.
Also a docker ps will also give you a similar output.

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                NAMES
fc6308ec4ff6        nginx:1.13.7        "nginx -g 'daemon of…"   4 minutes ago       Up 3 seconds        0.0.0.0:80->80/tcp   code_nginx_1

Next step is to make our flask container and run it with docker-compose.

The python container

Let's pull the python container:

$ docker pull python:3

Now let's write a Dockerfile for installling our dependencies for python.

Dockerfile:

FROM python:3
RUN pip install flask

The build process will install all the dependencies of flask itself. Now that we can hook up the Dockerfile with a simple hello world flask program and make sure everything's working. We'll put this file inside code directory so that we are able to mount it in our container.

code/main.py

from flask import Flask
app = Flask(__name__)

@app.route('/')
def index():
    return 'Hello world!'

Now let's set this up with docker-compose as well
docker-compose.yml:

version: '3.1'
services:
    nginx:
        image: nginx:1.13.7
        container_name: nginx
        ports:
            - 80:80
    flask:
        build:
            context: ./
            dockerfile: Dockerfile
        image: flask:0.0.1
        container_name: flask
        volumes:
            - ./:/code/
        environment:
            - FLASK_APP=/code/main.py
        command: flask run --host=0.0.0.0
        ports:
            - 8080:5000

Let's test it:
First stop the previous running container of nginx

$ docker stop <your container name>

Now, lets start the cluster with docker-compose

$ docker-compose up -d
Starting flask ...                                   
Starting nginx ... done
$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS                    NAMES                                                                              
f0f5bd9a2da1        nginx:1.13.7        "nginx -g 'daemon of…"   About a minute ago   Up 15 seconds       0.0.0.0:80->80/tcp       nginx                                                                              
9b065e02145b        flask:0.0.1         "flask run --host=0.…"   About a minute ago   Up 15 seconds       0.0.0.0:8080->5000/tcp   flask

Now that we have both the nginx and flask containers running, we can verify that everything works fine in both the containers. All we need to do now is configure nginx to reverse proxy our flask container

Configuring nginx

We want to basically proxy_pass all our traffic coming at / to our flask container.

lets start with our nginx.conf file

server {
    listen 80;
    server_name localhost;

    location / {
        proxy_pass http://flask-app:5000/;
        proxy_set_header Host "localhost";
    }
}

All we have done is:

Declared the server layer, which tells to listen to port 80.
Given the server_name, this is particularly useful when setting a domain name, in our case though localhost is fine enough.
Next we come on location directive of nginx. We can have a regex pattern to match after the location. In our case though / is good enough. We are currently reverse-proxying entire content to flask app.
The proxy_pass directive takes as argument the url to which we are proxying. In our case we will alias our flask container in the docker network as flask-app. This makes our flask app accessible at http://flask-app:5000/ from inside the nginx container.
The proxy_set_header will set the __Host header__ for the request between our nginx and flask, this helps to avoid certain errors, specially if one is using Django instead of flask, the ALLOWED_HOSTS setting would require this Host header.

we mount this file in our nginx container with the following line

nginx:
    volumes:
        - ./nginx.conf:/etc/nginx/conf.d/default.conf

This will overwrite our default nginx configuration file inside the container.

Setting up docker network

At the bottom of docker-compose file, add this line:

networks:
    my-network:

In the flask section add the following lines:

networks:
    my-network:
        aliases:
            - flask-app

This has added both our containers to a network called my-network with the flask container available on that network with the alias flask-app. All we have to do now is list flask container as a dependency for nginx container, so that it automatically starts the flask app.

Also note that now that we have completed the setup, we do not require to expose port for flask container, so we can get rid of the following lines from flask's section

ports:
  - 8080:5000

To sum up here are final versions of the files:

docker-compose.yml

version: '3.1'
services:
    nginx:
        image: nginx:1.13.7
        container_name: nginx
        depends_on:
            - flask
        volumes:
            - ./nginx.conf:/etc/nginx/conf.d/default.conf
        networks:
            - my-network
        ports:
            - 80:80
    flask:
        build:
            context: ./
            dockerfile: Dockerfile
        image: flask:0.0.1
        container_name: flask
        volumes:
            - ./:/code/
        environment:
            - FLASK_APP=/code/main.py
        command: flask run --host=0.0.0.0
        networks:
            my-network:
                aliases:
                    - flask-app
        ports:
            - 8080:5000

networks:
    my-network:

Dockerfile

FROM python:3
RUN pip install flask

nginx.conf

server {
    listen 80;
    server_name localhost;

    location / {
        proxy_pass http://flask-app:5000/;
        proxy_set_header Host "localhost";
    }
}

Let's run it finally

$ docker-compose up -d nginx
Starting flask ... done
Starting nginx ... done

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
874454f2ceb1        nginx:1.13.7        "nginx -g 'daemon of…"   13 minutes ago      Up 20 seconds       0.0.0.0:80->80/tcp       nginx
502fc6cc9603        flask:0.0.1         "flask run --host=0.…"   6 weeks ago         Up 23 seconds       0.0.0.0:8080->5000/tcp   flask

Note that since we listed flask as a dependency in nginx container, docker-compose first starts the flask container for us and then nginx. This works for a chain of such dependencies.

This post was originally published on my blog ishankhare.com

Kubernetes internals

Ishan Khare — Fri, 08 Mar 2019 16:08:41 +0000

In this blog post we are going to setup a kubernetes cluster with automated certificate generation using certbot. Will cover up some interesting concepts of kubernetes along the way, like:

So let's get started.

Why do we need this?

When using a traditional VM's/instances, one has access to ssh into a fixed instance with an assigned and fixed public IP that the DNS can resolve to. Once the DNS is set to resolve your hostname to your instance, you can install certbot on it and generate the certs on that instance.

With kubernetes, things get a bit tricky. You will still have a set of instances in your cluster, but they aren't directly accessible from outside the cluster. Plus you cannot preempt on which node your nginx or other ingress pod will be scheduled to run. Hence the most straight forward way to setup is doing everything through kubernetes and docker. This will also provide us with a few advantages:

The cert generation process will be documented as part of the kubernetes manifest files and Dockerfiles
Our solution would work for any number of pods and services. Basically we won't be bogged down due to issues arising because of scalability like when adding more pods or even adding more nodes to our cluster. Things will work seamlessly until we don't create a new cluster.
Our solution will be platform independent, just like docker and kubernetes. It will work on any cloud provider GCP, AWS or even Azure(hopefully).

I'll be using GCP for some parts of this blog post but replicating those parts on other cloud platforms is pretty straight forward.

We'll start by reserving a static IP for our cluster and then forward a DNS record to that IP so that certbot can easily resolve to our cluster.

Make sure to keep the region of the static IP and your cluster same.

Now that we have the static IP, we can move on to the kubernetes part - the fun part.

The LoadBalancer service

The first thing that we setup is the load balancer service, we will then use this service to resolve to the pods running our certbot client and later on our own application pods.
Below is the YAML for our LoadBalancer service. It utilizes the static IP we created in the previous step.

svc.yml

apiVersion: v1
kind: Service
metadata:
  name: certbot-lb
  labels:
    app: certbot-lb
spec:
  type: LoadBalancer
  loadBalancerIP: X.X.X.X
  ports:
    - port: 80
      name: "http"
      protocol: TCP
    - port: 443
      name: "tls"
      protocol: TCP
  selector:
    app: certbot-generator

Most of it is self explanatory, few fields of interest are:

spec.type : LoadBalancer - This spins up a Google Cloud LoadBalancer on GCP and AWS Elastic LoadBalancer on AWS
spec.loadBalancerIP - This assign the previously generated static IP to our loadBalancer, now all traffic coming to our IP address is funneled into this load balancer.
ports.port - We opened 2 TCP ports, port 80 and port 443 for accepting HTTP and HTTPS traffic respectively.
spec.selector - These are the set of labels that allow us to govern which pods can our loadBalancer resolve to. We'll later use the same set of labels in our Job and Pod templates

Let's deploy this service to our cluster.

$ kubectl apply -f svc.yml
service "certbot-lb" created

If we see the status of our service now, we should see this

$ kubectl get svc certbot-lb
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
certbot-lb   LoadBalancer   X.X.X.X         X.X.X.X       80:30271/TCP,443:32224/TCP   1m

If you're trying this on minikube, LoadBalancer service is not available. Besides, it makes no sense trying to generate a SSL cert on your local environment.

Next we have to think about where to run our certbot container.

Job Controllers

The Job controllers of kubernetes allows us to schedule pods which run to completion. That is, these are jobs that if finished without error, need not be run again in future. This is exactly what we want when generating SSL certs. Once the certbot process is done and has given us the certificates, we no longer need that container to be running. Also we don't want kubernetes to restart this pod when it exits. However we can ask kubernetes to automatically reschedule the pod in case is case the pod exists with a failure/error.

So lets write a Job spec file that will generate the SSL cert for us using certbot. Certbot provides an official docker container that we can just reuse in our case.

jobs.yml

apiVersion: batch/v1
kind: Job
metadata:
  name: certbot
spec:
  template:
    metadata:
      labels:
        app: certbot-generator
    spec:
      containers:
        - name: certbot
          image: certbot/certbot
          command: ["certbot"]
          args: ["certonly", "--noninteractive", "--agree-tos", "--staging", "--standalone", "-d", "staging.ishankhare.com", "-m", "me@ishankhare.com"]
          ports:
            - containerPort: 80
            - containerPort: 443
      restartPolicy: "Never"

Few important fields here are:

spec.template.metadata.labels - This matches with the spec.selector specified in our LoadBalancer service. This brings our pod under our loadBalancer. Now everything coming on port 80 and 443 will be funneled to our pod.
spec.template.spec.containers[0].image - The certbot/certbot docker image. Kubernetes will do a docker pull certbot/certbot on the server for us when scheduling this pod.
spec.template.spec.containers[0].command - The command to run in the pod.
spec.template.spec.containers[0].args - The arguments to the above command. We're using the standalone mode of certbot to generate the certs here as it makes things straightforward. You can read more about this command in certbot docs
spec.template.spec.containers[0].ports - We've opened port 80 and 443 for our container.

Before deploying this to our cluster, make sure that the domain you specify is actually pointing to the static IP we setup in the previous steps.

Let's deploy this to our cluster:

$ kubectl apply -f jobs.yml
job.batch "certbot" created

This Job will spin-up a single pod for us, we can get that:

$ kubectl get pods
NAME            READY     STATUS    RESTARTS   AGE
certbot-sgd4w   1/1       Running   0          5s

We can now see the STDOUT logs on this pod

$ kubectl logs -f certbot-sgd4w
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for staging.ishankhare.com
Waiting for verification...
Cleaning up challenges
IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/staging.ishankhare.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/staging.ishankhare.com/privkey.pem
   Your cert will expire on 2019-03-04. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.

After this, the certbot process exits without error and so does our pod. If we now list our pods

$ kubectl get pods
NAME            READY     STATUS      RESTARTS   AGE
certbot-sgd4w   0/1       Completed   0          45m

We see only a single pod whose status is completed.

We can add .spec.ttlSecondsAfterFinish to our jobs.yml like so:
spec:
  ttlSecondsAfterFinish: 10
This will automatically garbage collect our certbot pod after its finished.
Only supported in kubernetes v1.12 alpha. Hence not recommended right now.

Where to save the generated SSL certificate?

We currently have 2 options when it comes to saving the certs:

Save it in a mounted volume. (Not recommended)
- Would require a PersistentVolume and a PersistentVolumeClaim
- The certificate might be just a few KB in size, but the minimum volume size that GKE allots is 1Gi. Hence this is highly inefficient.
Use kubernetes Secrets. (Recommended)
- This would also require us to use kubernetes client's in cluster configuration, which is straightforward to use and is usually the recommended way in such cases.
- Would require us to setup ClusterRole and ClusterRoleBinding.
- Can be configured to allow specific access to specific people using the concept of rbac (RoleBasedAccessControl)

Why in-cluster access?

To understand this, we need to go a little deeper into our current architecture.
Our current setup looks roughly like this:

As we can see here, the certs generated by our certbot Job Controller Pod are already inside the cluster. We want these cert credentials to be stored on the secrets.

When we fetch/create/modify secrets in normal flow, its usually done through the kubectl client like:

$ kubectl get secrets
NAME                  TYPE                                  DATA      AGE
default-token-hks8k   kubernetes.io/service-account-token   3         1m

$ kubectl create secret
Create a secret using specified subcommand.

Available Commands:
  docker-registry Create a secret for use with a Docker registry
  generic         Create a secret from a local file, directory or literal value
  tls             Create a TLS secret

Usage:
  kubectl create secret [flags] [options]

But, in our case, we want to store these credentials as secrets from inside a Pod which is itself inside the cluster. This is exactly what I meant by in-cluster access above.

What all do we need for In-Cluster access?

We'll need the first 2 mentioned above in the same Pod that is trying to access the secrets.
Hence its best to extend the certbot/certbot docker image with the above dependencies added in the container image itself.

This will change our cluster architecture a bit. The image below shows the rough cluster arch with our modified setup.

Let's first create our Dockerfile for our extended container:

FROM certbot/certbot
COPY ./script.sh /script.sh
RUN wget https://storage.googleapis.com/kubernetes-release/release/v1.6.3/bin/linux/amd64/kubectl
RUN chmod +x kubectl
RUN mv kubectl /usr/local/bin
ENTRYPOINT /script.sh

We have also used an extra script.sh in our container. We'll use this script as our entrypoint instead of the default entrypoint of certbot/certbot image.
This allows us to start our auxiliary kube-proxy and use kubectl as we desire. We present the script.sh below:

#!/bin/sh

# start kube proxy
kubectl proxy &

certbot certonly --noninteractive --force-renewal --agree-tos --staging --standalone -d staging.ishankhare.com -m me@ishankhare.com

kubectl create secret tls cert --cert=/etc/letsencrypt/live/staging.ishankhare.com/fullchain.pem \
    --key=/etc/letsencrypt/live/staging.ishankhare.com/privkey.pem

# kill the kubectl process running in background
kill %1

First we start our kube-proxy as a background process.
Next we run our certbot command for generating the cert.
The certbot command generates certs for us at /etc/letsencrypt/live/staging.ishankhare.com/, we now use these paths to create the tls type Secret using kubectl create secret tls. This command has the following syntax:

$ kubectl create secret tls -h
Create a TLS secret from the given public/private key pair.

The public/private key pair must exist before hand. The public key certificate must be .PEM encoded
and match the given private key.

Examples:
  # Create a new TLS secret named tls-secret with the given key pair:
  kubectl create secret tls tls-secret --cert=path/to/tls.cert --key=path/to/tls.key

Hence our command above will create a secret named cert

Finally we kill our kube-proxy background process.

Update our jobs.yml

Since we are now using an updated Dockerfile with own own script as its entrypoint, we can modify the Job manifest file like below:

apiVersion: batch/v1
kind: Job
metadata:
  #labels:
  #  app: certbot-generator
  name: certbot
spec:
  template:
    metadata:
      labels:
        app: certbot-generator
    spec:
      containers:
        - name: certbot
          image: ishankhare07/certbot:0.0.6
            - containerPort: 80
            - containerPort: 443
      restartPolicy: "Never"

With this we are ready to run our job now.

First verify that our LoadBalancer service is up:

$ kubectl get svc
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
certbot-lb   LoadBalancer   10.11.240.100   X.X.X.X          80:31855/TCP,443:32457/TCP   32m

# delete our older job
$ kubectl delete job certbot
job.batch "certbot" deleted

Now we deploy our new Job file

$ kubectl apply -f jobs.yml
job.batch "certbot" created

# list the pod created for the job
$ kubectl get pods
NAME            READY     STATUS    RESTARTS   AGE
certbot-5nn5h   1/1       Running   0          3s

# get STDOUT logs for this pod
$ kubectl logs -f certbot-5nn5h
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for staging.ishankhare.com
Waiting for verification...
Cleaning up challenges
Starting to serve on 127.0.0.1:8001IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/staging.ishankhare.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/staging.ishankhare.com/privkey.pem
   Your cert will expire on 2019-03-13. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.
Error from server (Forbidden): secrets is forbidden: User "system:serviceaccount:default:default" cannot create secrets in the namespace "default"

The certificate generation was successful. But we cannot yet write to Secrets. The last line in the above output shows us that.

Error from server (Forbidden): secrets is forbidden: User "system:serviceaccount:default:default" cannot create secrets in the namespace "default"

Why is that? Because RBAC....

RBAC or Role Based Access Control in kubernetes consists of 2 parts:

Role/ClusterRole
RoleBinding/ClusterRoleBinding

We will be using ClusterRole and ClusterRoleBinding approach to provide cluster wide access to our Pod. More fine-grained, role-based access can be provided with the Role and RoleBinding approach which can be referred from the docs - https://kubernetes.io/docs/reference/access-authn-authz/rbac/

Let's create 2 files called rbac-cr.yml and rbac-crb.yml with the following contents:
rbac-cr.yml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  namespace: default
  name: secret-reader
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "create"]

This allows access to Secrets, in-particular to get, list and create Secrets. The last verb create is what concerns us.

rbac-crb.yml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: secret-reader
subjects:
  - kind: User
    name: <your account here>
    namespace: default
  - kind: ServiceAccount
    name: default
    namespace: default
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

Fields of interest here are .subjects. It is an array and we have defined two kinds in this.

kind: User : This refers to the current user, who is executing these commands using kubectl. This is required so as to grant the current user enough permissions to grant the .rules.resources and .rules.verbs related access that we have defined in the rbac-cr.yml i.e. our ClusterRole definition.
kind: ServiceAccount : This refers to the account use inside the cluster when our pod will be creating the secrets using the kube-proxy.

We push this to our kubernetes cluster in this particular order only:

$ kubectl apply -f rbac-crb.yml
clusterrolebinding.rbac.authorization.k8s.io "secret-reader" created
$ kubectl apply -f rbac-cr.yml
clusterrole.rbac.authorization.k8s.io "secret-reader" created

The order is important, because we first need to create ClusterRoleBinding (defined in rbac-crb.yml) and then using that binding on our account, we are going to apply a ClusterRole (defined in rbac-cr.yml).

Finally re-run jobs.yml again

I say re-run because since previous job still exists:

$ kubectl get jobs
NAME      DESIRED   SUCCESSFUL   AGE
certbot   1         1            1h

and because of this job, a stagnant Pod exists as well:

$ kubectl get pods
NAME            READY     STATUS      RESTARTS   AGE
certbot-5nn5h   0/1       Completed   0          1h

If we just try to apply our jobs.yml file now, kubernetes sees that nothing as changed in our yml manifest and chooses to take no action:

$ kubectl apply -f jobs.yml
job.batch "certbot" unchanged

Hence we delete and apply our job from scratch:

$ kubectl delete job certbot
job.batch "certbot" deleted

$ kubectl apply -f jobs.yml
job.batch "certbot" created

# get the pod created by the above job
$ kubectl get pods
NAME            READY     STATUS    RESTARTS   AGE
certbot-c9h6m   1/1       Running   0          2s

We try to see the STDOUT logs of this container

$ kubectl logs -f certbot-c9h6m
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for staging.ishankhare.com
Waiting for verification...
Cleaning up challenges
Starting to serve on 127.0.0.1:8001IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/staging.ishankhare.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/staging.ishankhare.com/privkey.pem
   Your cert will expire on 2019-03-13. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.
secret "cert" created

The last line says secret "cert" created

Mission accomplished!

We can now see this recently created secret:

$ kubectl get secret cert
NAME      TYPE                DATA      AGE
cert      kubernetes.io/tls   2         9m

If you actually want to get the contents of the cert you can:

$ kubectl get secret cert -o yaml > cert.yml
# or
$ kubectl get secret cert -o json > cert.json

It is always desirable to redirect the above streams to a file rather than printing them directly to the console.

With this goal achieved, I'll wrap up this post here now. I'll be back with more posts soon detailing on:

Options available for using these certificated once we have created them.
Proper use of Init Containers along with the Job Controllers used in this post and our good old Pods and Deployments to see how we can make these interdependent services wait for the depending services to complete before spawning themselves.

Do let me know what you think of this post and if you have any questions in the comments section below.

This post was originally published on my blog ishankhare.com

Let's implement a hashtable

Ishan Khare — Fri, 08 Mar 2019 15:58:20 +0000

How hashmaps work?

Okay, so lets start with what hashmaps are?
Hashmaps or hashtables are known by different names in different languages.

Python has dict/dictionaries
Ruby calls it Hash
Java has hashmap
C++ has unordered_map
Even Javascript has maps, which is how basically objects are implemented in javascript. (just look at JSON)

What the last point says about hashmaps being used to implement objects in Javascript is also true in case of Python, Ruby and a few others.

So hashmaps/dicts/hashtables whatever you want to call them are basically key-value storage data structures. You pass in a value identified by a key, and you can get back the same value with that key - simple as that.

So lets try and get hands on with how it looks in code.

a = {}

# add a key-value pair
a['name'] = 'ishan'

# lets see whats in there
print(a)
# {'name': 'ishan'}

# add more key-value pairs
a['age'] = 23
a['gender'] = 'male'
print(a)
# {'gender': 'male', 'age': 23, 'name': 'ishan'}

# note above that the keys are not in the same order as we entered them,
# this is because python's dictionaries are unordered
# see https://stackoverflow.com/a/15479974/2972348

# get a specific value
print(a['name'])
# 'ishan'

# let's update a value
a['age'] = 24
print(a)
# {'gender': 'male', 'age': 24, 'name': 'ishan'}

# let's delete some values
del a['age']
print(a)
# {'gender': 'male', 'name': 'ishan'}

That gives us the list of basic operations on a dict:

insert
fetch
delete

The same can be shown with Javascript:

a = {}

a['name'] = 'ishan'
console.log(a)
// { name: 'ishan' }

// or the other way
a.age = 24
console.log(a)
// { name: 'ishan', age: 24 }

console.log(a.name)
// ishan

delete a.age
console.log(a)
// { name: 'ishan' }

Enough of examples now, lets get to see how these things work on the inside

So, we will be implementing a simple hashtable in Golang. If you haven’t heard or written code in Go, and are skeptical about understanding the code that follows - worry not, Go has a syntax very similar to C and Javascript. And if you can code (in a language), you’re gonna do just fine!

The building blocks

This image taken from wikipedia shows the basic structure of our hashtable. This will help us understand and break down the problem at hand better:

To summarize, we have:

the hash function
the linear array that the hash function maps to
and the actual data nodes that hold our key-value pairs

The array

So lets start with our linear array to begin with. So we basically need a statically allocated array of some size n. This will be the array holding the pointer to our actual key value pairs. This is how our basic array will look like:

  +----+
1 |    |
  +----+
2 |    |
  +----+
3 |    |
  +----+
4 |    |
  +----+
5 |    |
  +----+
6 |    |
  +----+
7 |    |
  +----+
8 |    |
  +----+
9 |    |
  +----+
10|    |
  +----+

Let us code this up in Go. To have a basic array inside our hashmap data type. This is going to be fairly easy to achieve.

package main

import "fmt"

const MAP_SIZE = 50

type HashMap struct {
    Data []*int 
}

func NewDict() *HashMap {
    return &HashMap{}   
}

func main() {
    a := NewDict()
}

The above code is explained below:

We declare the current file as main package with package main.
We import a package called “fmt”, we’ll later use it to print things to terminal.
We declare a constant variable MAP_SIZE with value 50 . This will be the size of our linear array.

For the sake of simplicity, we are assuming that we just have a fixed size array that we use for implementing our hashmap, and we don’t take resizing into account. Actual implementations do account for these things - which will be covered in a future blog post. For more info refer here.
Next we create a function NewDict that creates this array for us and returns a pointer to it.

This completes our bare-bones structure for the array itself. But we are presented with the next challenge. Which is, how to represent the key-value pairs that contain our actual data and link it with our array.

Creating the nodes and linking with array

There is one problem in the code presented above — we have created an array of pointer to integer values. Basically the Data []*int on line8. This is wrong because we need to actually not point to integers but to some object that holds our key-value pairs. We don’t know what that object will be yet, but we know that it needs at least 2 fields — key and value. So lets go ahead and create this new object’s structure in our code.

We add the following lines:

type Node struct {
    key string
    value string
    next *Node
}

We create a structure to hold our key-value pairs and a pointer to a Node type. We will need this pointer later on in order to deal with hash conflicts. More on this later, for now just let it sit around there.

Now since we want every index of our array to point to these Node type, we should change the type of our data array.

type HashMap struct {
    Data []*Node
}

So at the end of these changes, this is how our code should look like:

package main
import "fmt"
const MAP_SIZE = 50

type Node struct {
    key string
    value string
    next *Node
}

type HashMap struct {
    Data []*Node
}

func NewDict() *HashMap {
    return &HashMap{ Data: make([]*Node, MAP_SIZE) }
    // we initialize our array with make(), see 
    // https://golang.org/pkg/builtin/#make

}

func main() {
    a := NewDict()
    fmt.Println("%q", a)

}

If you run this program right now, you should see something like this:

&{[<nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil>
<nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil>
<nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil>
<nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil>
<nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil>]}

We have our hashmap object consisting of an array of 50 pointers each pointing to <nil> which is something equivalent to:

NULL in C.
null in Java.
None in Python.
undefined in Javascript.

This is just as expected, we haven’t inserted any value in the hash table so all pointers are pointing to null/nil . Next we have to think about inserting objects into our hashmap.

Before we could even reach that, we need to find a way to convert our keys to corresponding indexes on the array.

A typical implementation of hash maps relies on a good hash function. The job of a hash function is to convert the value passed to it (in our current case, a string) and return an integer value representing an index on the array. And this integer value has to be the same every time.

This means that if hash("hello") gives me 966170508347869221, then every time I pass "hello" I should get the same 966170508347869221 as a return value.

You can test the above mentioned code in the Python interpreter. Python has a builtin function hash that will return the hash value.

But we cannot have index 966170508347869221 on our array. Currently we have only indexes 1-49 available. Hence we need to somehow bring it down to this range.

Modulo to the rescue

The simplest solution to this problem is using the modulo (%) operator. This operator gives us the remainder of the division performed. That is, if

966170508347869221 / 50 = 19323410166957384 + 21

then 966170508347869221 % 50 = 21. The remainder.

But with this, we have another problem, there will be keys which will be mapped to the same index, hence we’ll get a hash collision.

From this point on, we have 3 problems to solve at hand:

How do you implement the hash function.
Make the hash value fall in range of our array.
How to manage hash collisions.

Let’s start with the hash generation function first

Choosing a hash algorithm

Wikipedia has a list of many hash_functions. Which you can refer here https://en.wikipedia.org/wiki/List_of_hash_functions

For our purpose, we’ll use a Jenkins hash function. It is a hash function that produces 32-bit hashes. The wikipedia article also has the implementation of the algorithm in C. We’ll just translate that to our go code.

func hash(key string) (hash uint32) {
    hash = 0
    for _, ch := range key {
        hash += uint32(ch)
        hash += hash << 10
        hash ^= hash >> 6
    }

    hash += hash << 3
    hash ^= hash >> 11
    hash += hash << 15

    return
}

The above function uses a named-return — see https://golang.org/doc/effective_go.html#named-results

With this in place our code should look like this:

package main

import "fmt"

const MAP_SIZE = 50

type Node struct {
    key string
    value string
    next *Node
}

type HashMap struct {
    Data []*Node
}

func hash(key string) (hash uint32) {
    hash = 0
    for _, ch := range key {
        hash += uint32(ch)
        hash += hash << 10
        hash ^= hash >> 6
    }
    hash += hash << 3
    hash ^= hash >> 11
    hash += hash << 15
    return
}

func NewDict() *HashMap {
    return &HashMap{ Data: make([]*Node, MAP_SIZE) }
    // we initialize our array with make(), see 
    // https://golang.org/pkg/builtin/#make
}

func main() {
    a := NewDict()
    fmt.Println(a)
}

Mapping the hash to our array

Now that we have found our way to convert our keys to the hash value, we now need to make sure that this value falls within out array. For this we implement a getIndex function:

func getIndex(key string) (index int) {
  return int(hash(key)) % MAP_SIZE
}

With this in place, we have taken care of 2 of our 3 problem statements. The 3rd problem - managing collisions will be taken care of at a later stage. First we now need to focus on inserting the key-value pairs in our array. Lets write a function for that.

Insertion

The following function uses a receiver, which are a powerful feature of go.

func (h *HashMap) Insert(key string, value string) {
  index := getIndex(key)

  if h.Data[index] == nil {
    // index is empty, go ahead and insert
    h.Data[index] = &Node{key: key, value: value}
  } else {
    // there is a collision, get into linked-list mode
    starting_node := h.Data[index]
    for ; starting_node.next != nil; starting_node = starting_node.next {
      if starting_node.key == key {
        // the key exists, its a modifying operation
        starting_node.value = value
        return
      }
    }
    starting_node.next = &Node{key: key, value: value}
  }
}

The code is mostly self explanatory, but we'll briefly go over it anyway.

We first call getIndex with the key, which in-turns calls the hash function internally. This gives us the index on our array where we can store that key-value pair.

Next we check if the index is empty, if yes - then we simply store the K-V Pair there. This would look something like this:

  +----+
1 |    |
  +----+
2 |    |
  +----+     +------------------+
3 | *  |---->| Key: Value       |
  +----+     | Next: nil        |
4 |    |     +------------------+
  +----+
5 |    |
  +----+     +------------------+
6 | *  |---->| Key: Value       |
  +----+     | Next: nil        |
7 |    |     +------------------+
  +----+
8 |    |
  +----+
9 |    |
  +----+
10|    |
  +----+

The above denotes the if part of our code - when there are no collisions. The else part of our code handles the collision.

There are a few different ways of handling collisions in hashmaps:

Separate chaining with linked lists
Open addressing
Double Hashing

The current code in the else block handles collisions with the Separate chaining with linked lists technique.

The linked list technique will result in a data structure very similar to the following image:

The two keys mapping to the same index - hence resulting in a collision are shown in red.

Next we'll talk about retrieving the value back from our hashtable

Fetch

The next basic operation on hashmaps that we are going to look at is Fetch. Fetch is fairly simple and identical to insert - we pass our key as input to the hash function and get a hash value, which is then mapped to our array using the getIndex function.

We can then have 3 possible outcomes:

We check if the key on that index is matching the we are looking for - if yes, then we have our match.
If the key does not match, we check if its next value is not nil - basically checking for collision and find that element in the separate chained linked list
If both of the above conditions fail, we concur that the requested key does not exist in our hashtable.

The code implementing the above explanation is shown below:

func (h *HashMap) Get(key string) (string, bool) {
  index := getIndex(key)
  if h.Data[index] != nil {
    // key is on this index, but might be somewhere in linked list
    starting_node := h.Data[index]
    for ; ; starting_node = starting_node.next {
      if starting_node.key == key {
        // key matched
        return starting_node.value, true
      }

      if starting_node.next == nil {
        break
      }
    }
  }

  // key does not exists
  return "", false
}

The above code uses golang's multiple return values feature.

With this in place, we'll write the main function and test these operations that we have implemented.

func main() {
  a := NewDict()
  a.Insert("name", "ishan")
  a.Insert("gender", "male")
  a.Insert("city", "mumbai")
  a.Insert("lastname", "khare")
  if value, ok := a.Get("name"); ok {
    fmt.Println(value);
  } else {
    fmt.Println("Value did not match!")
  }

  fmt.Println(a)
}

Also lets add a convenience function that allows us to print our hashmap in the desired format.

func (h *HashMap) String() string {
  var output bytes.Buffer
  fmt.Fprintln(&output, "{")
  for _, n := range h.Data {
    if n != nil {
      fmt.Fprintf(&output, "\t%s: %s\n", n.key, n.value)
      for node := n.next; node != nil; node = node.next {
        fmt.Fprintf(&output, "\t%s: %s\n", node.key, node.value)
      }
    }
  }

  fmt.Fprintln(&output, "}")

  return output.String()
}

This overrides the default print output for our defined HashMap type.

The above method is just a convenience method used for printing the entire hashmap in a pretty format. It is in no way required for proper functioning of the data structure

To sum it up, here's our final code:

package main

import (
  "fmt"
  "bytes"
  )

const MAP_SIZE = 10

type Node struct {
  key string
  value string
  next *Node
}

type HashMap struct {
  Data []*Node
}

func NewDict() *HashMap {
  return &HashMap{ Data: make([]*Node, MAP_SIZE) } 
}

func (n *Node) String() string {
  return fmt.Sprintf("<Key: %s, Value: %s>\n", n.key, n.value)
}

func (h *HashMap) String() string {
  var output bytes.Buffer
  fmt.Fprintln(&output, "{")
  for _, n := range h.Data {
    if n != nil {
      fmt.Fprintf(&output, "\t%s: %s\n", n.key, n.value)
      for node := n.next; node != nil; node = node.next {
        fmt.Fprintf(&output, "\t%s: %s\n", node.key, node.value)
      }
    }
  }

  fmt.Fprintln(&output, "}")

  return output.String()
}

func (h *HashMap) Insert(key string, value string) {
  index := getIndex(key)

  if h.Data[index] == nil {
    // index is empty, go ahead and insert
    h.Data[index] = &Node{key: key, value: value}
  } else {
    // there is a collision, get into linked-list mode
    starting_node := h.Data[index]
    for ; starting_node.next != nil; starting_node = starting_node.next {
      if starting_node.key == key {
        // the key exists, its a modifying operation
        starting_node.value = value
        return
      }
    }
    starting_node.next = &Node{key: key, value: value}
  }
}

func (h *HashMap) Get(key string) (string, bool) {
  index := getIndex(key)
  if h.Data[index] != nil {
    // key is on this index, but might be somewhere in linked list
    starting_node := h.Data[index]
    for ; ; starting_node = starting_node.next {
      if starting_node.key == key {
        // key matched
        return starting_node.value, true
      }

      if starting_node.next == nil {
        break
      }
    }
  }

  // key does not exists
  return "", false
}

func hash(key string) (hash uint8) {
  // a jenkins one-at-a-time-hash
  // refer https://en.wikipedia.org/wiki/Jenkins_hash_function

  hash = 0
  for _, ch := range key {
    hash += uint8(ch)
    hash += hash << 10
    hash ^= hash >> 6
  }

  hash += hash << 3
  hash ^= hash >> 11
  hash += hash << 15

  return 
}

func getIndex(key string) (index int) {
  return int(hash(key)) % MAP_SIZE
}

func main() {
  a := NewDict()
  a.Insert("name", "ishan")
  a.Insert("gender", "male")
  a.Insert("city", "mumbai")
  a.Insert("lastname", "khare")
  if value, ok := a.Get("name"); ok {
    fmt.Println(value);
  } else {
    fmt.Println("Value did not match!")
  }

  fmt.Println(a)
}

Running

On running the above code, we get the following output:

ishan
{
    city: mumbai
    name: ishan
    lastname: khare
    gender: male
}

The first line is the value of our key name.
The output from 2nd line onwards is basically our custom-coded pretty-print function that outputs our entire hashmap.

If you're interested in running/experimenting with this code, please feel free to fork the following repl. You can also play around on this webpage interactively.

Our current implementation only supports strings as keys. In the next blog post we'll talk about how we can make this a more generic implementation.

This post was originally published on my blog at ishankhare.com

Forem: Ishan Khare

Dabbling with envoy - Part II

Dynamic configuration from control plane

node

dynamic_resources

static_resources

Control Plane

Cluster and Endpoint

Route

Listener, connection manager and config source

Setting up the gRPC services

Running the envoy proxy

Running everything

What next?!

Dabbling with Envoy configurations - Part I

Types of envoy configurations

Static configuration

static_resources

listeners

cluster

Testing this static configuration

Dynamic Configuration from filesystem

node

dynamic_resources

listener resources

cluster resources

Dynamically editing the configuration

Teaching kubernetes service mesh tricks

What we'll cover in this post?

Tl;Dr

Use case

The Deep Dive

The CRD

The controller

The VirtualService check

The DestinationRule check

Creating the Istio CRD's (defining the 'spawn') package

Telling the controller to 'watch' the resources we own

Are we done?

Some final highlights

What's next?

Writing a kubernetes controller in Go with kubebuilder

Installing kubebuilder

Install kustomize

Start the project

Test install scaffold

Enable status subresource

Background

CRD outline

Defining the CRD

Implementing the Controller logic

Deploy a golang API on Kubernetes with Nginx ingress

What is Ingress?

What are services?

And Deployments?

So lets dive right in

The cluster

WTF is helm?

Deploying our go backend

Installing nginx ingress through helm

Go Concurrency pipelines

The problem statement

Implementation with goroutines

High level design

Where things get dim?

Collecting metrics

The issue

Nginx as reverse proxy for a flask app using Docker

So what is a reverse proxy?

Running Nginx with docker

docker-compose

docker-compose.yml

The python container

Configuring nginx

Setting up docker network

To sum up here are final versions of the files:

Let's run it finally

Kubernetes internals

Why do we need this?

The LoadBalancer service

`listener` resources

`cluster` resources

`CRD` outline

Defining the `CRD`