Forem: Joseph D. Marhee

Sampling Methods in Python

Joseph D. Marhee — Tue, 04 Jul 2023 04:06:09 +0000

Photo by Clay Banks on Unsplash

Sampling is a fundamental concept in statistics and data analysis. It involves selecting a subset of individuals or data points from a larger population for analysis. In Python, there are various sampling methods available that allow us to extract representative samples from a population.

In this document, we will explore different sampling methods in Python and provide code examples for each method. We will cover the following sampling methods:

Simple Random Sampling
Stratified Sampling
Cluster Sampling
Systematic Sampling

Let’s dive into each method and see how it can be implemented in Python.

1. Simple Random Sampling

Simple random sampling involves randomly selecting individuals from a population without any specific criteria. Each individual has an equal chance of being selected. This method is commonly used when the population is homogenous.

import random

population = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
sample_size = 5

sample = random.sample(population, sample_size)
print(sample)

Output:

[3, 5, 9, 7, 1]

2. Stratified Sampling

Stratified sampling involves dividing the population into distinct subgroups or strata based on certain characteristics. Then, a random sample is selected from each stratum proportionate to its size. This method ensures that each subgroup is adequately represented in the final sample.

from sklearn.model_selection import train_test_split

data = [...] # Your dataset
labels = [...] # Labels for each data point

# Stratified sampling using train_test_split from scikit-learn
train_data, test_data, train_labels, test_labels = train_test_split(data, labels, test_size=0.2, stratify=labels)

3. Cluster Sampling

Cluster sampling involves dividing the population into clusters or groups. Instead of selecting individuals, entire clusters are chosen randomly. This method is useful when it is difficult or expensive to access individual elements of the population.

import random

clusters = [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]]
cluster_sample_size = 2

selected_clusters = random.sample(clusters, cluster_sample_size)
sample = []

for cluster in selected_clusters:
    sample.extend(cluster)

print(sample)

4. Systematic Sampling

Systematic sampling involves selecting individuals from a population at regular intervals after an initial random start. This method is useful when the population is large and ordered in some way.

population = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
sample_size = 3

start_index = random.randint(0, sample_size - 1)
sample = [population[i] for i in range(start_index, len(population), sample_size)]

print(sample)

These are just a few examples of sampling methods available in Python. Depending on your specific requirements and the characteristics of your data, you may need to adapt or combine these methods to achieve the desired sampling outcome.

Remember that sampling is a powerful technique for analyzing large datasets and making inferences about a population. It allows us to draw meaningful conclusions while reducing computational costs and time.

How to delete a Kubernetes namespace stuck at Terminating Status

Joseph D. Marhee — Wed, 17 Aug 2022 21:27:37 +0000

After running a command like:

kubectl delete my-namespace

and the command hangs indefinitely after accepting the deletion request, you might find it sitting at Terminating:

$ kubectl get ns/my-namespace
NAME      STATUS         AGE
testing   Terminating    113m

This might be due to a Finalizer in the spec, specific conditions to be met during deletion. You can view the finalizers for your namespace:

...
    "spec": {
       "finalizers": [
           "kubernetes"
       ]
    }
...

This is what a standard namespace with no other conditions might otherwise look like, so if it's hanging, this is likely way.

You can modify the resource using kubectl edit ns/my-namespace, and replace the finalizer with an empty array:

...
    "spec": {
       "finalizers": [
       ]
    }
...

or use the Kube API to update the specification to allow it to finalize manually if this is unsuccessful by dumping the resource:

kubectl get ns/my-namespace -o json > my-namespace.json

and editing that my-namespace.json to remove the finalizer, and then making a PUT request to the API:

kubectl replace --raw "/api/v1/namespaces/my-namespace/finalize" -f ./my-namespace.json

or using an HTTP client to issue the request (usually will require authentication if not exposing via kubectl proxy) using that same json dump:

curl -k -H "Content-Type: application/json" -X PUT --data-binary @my-namespace.json "https://<Kube API Endpoint>/api/v1/namespaces/my-namespace/finalize"

You can then confirm, once you receive a successful response, that the namespace is deleted:

kubectl get ns/my-namespace

which should return no resource.

Exporting images with Containerd CLI (ctr)

Joseph D. Marhee — Sun, 07 Aug 2022 03:29:55 +0000

I recently found that a Docker image I use as a part of one of my Helm charts was no longer available on DockerHub, and I hadn't mirrored it in another location. It was running in a K3s cluster, meaning I couldn't docker tag original-maintainer/image:tag me/image:tag it and push to the Hub myself back on my local machine, which was running the Docker CLI.

First, logging into a node with the image on it, I ran the following:

ctr -n k8s.io images export bind9:9.11.tar docker.io/internetsystemsconsortium/bind9:9.11 --platform linux/amd64

which exported the image to a tarball. This is similar to docker image save.

Then, pulling it down to my Docker CLI machines:

scp jdmarhee@192.168.0.60:bind9:9.11.tar .

I then load the image from the tarball:

 docker image load --input bind9\:9.11.tar

which returns the original tag for the image (internetsystemsconsortium/bind9:9.11) so I can re-tag it and push:

docker image tag internetsystemsconsortium/bind9:9.11 jmarhee/bind9:9.11

docker push jmarhee/bind9:9.11

Now this image is available on Docker Hub for my (and your) new clusters to access.

Recurring Background Jobs in Flask

Joseph D. Marhee — Tue, 24 May 2022 05:07:48 +0000

Feeling some frustration with most RSS reader apps being way too feature-heavy for my liking, I recently tried building a very simple RSS feed reader that pulls from a feeds.yaml file that lists URLs and does little else beyond aggregating those links and, where available, providing the article body.

In the app, a function like this builds the data body that is populated when Flask renders the template on the page in parser.py:


import feedparser
import os
import yaml

def buildConfig():
    with open(os.environ['FEED_YAML_PATH']) as f:
        dict = yaml.load(f, Loader=yaml.FullLoader)
    return dict['feeds']

def buildFeed(feeds):
    feed_body = []
    for url in feeds:
        feed_data = feedparser.parse(url)
        new_feed = {"feed" : url, "data": [feed_data]}
        feed_body.append(new_feed)
    return feed_body

basically just dumping all of the feed data into a larger JSON object for Flask to render in the template.

I wanted to be able to see new feeds I added to this file without restarting Flask, and one way of doing this was to regenerate this data object on every page load, but that would make it take forever to load unless I added a bunch of additional logic or other dependencies or components (storing feeds in memory, etc.)

In order to avoid adding new components to the app (it's a very small Docker image, and restoring app data only amounts to this Yaml file being available), I decided to add a recurring background task using the apscheduler package which has a BackgroundScheduler function in app.py:

from apscheduler.schedulers.background import BackgroundScheduler

then I have it define on app startup the variable that will store the feed_data object to be updated by the recurring job we'll define in a moment, and initialize it:

feed_data = None

@app.before_first_request
def initialize_feeds():
    feeds = buildConfig()
    global feed_data
    feed_data = buildFeed(feeds)
    return feed_data

and then adding an updateFeeds function that behaves similarly to the above function (in a production app, you could avoid repeating yourself and have the above function reference this one, but for the sake of demonstration):

def update_feeds():
    feeds = buildConfig()
    global feed_data
    feed_data = buildFeed(feeds)
    return feed_data

scheduler = BackgroundScheduler()
job = scheduler.add_job(update_feeds, 'interval', minutes=1)
scheduler.start()

and then having that function run as an argument to the scheduler.add_job() function above, with an interval of 1 minute.

So, now, on changes to your Yaml file containing a new URL, the feed_data object will be updated in the background, and load time will be unaffected.

Rancher Logging Operator with in-cluster Syslog

Joseph D. Marhee — Thu, 19 May 2022 23:44:35 +0000

The Rancher Logging Operator supports a number of backends, but a very common one is logging to a syslog endpoint. However, rather than maintaining separate logging infrastructure if this is a new piece of your environment, you can deploy it to your Kubernetes clusters directly, either using the method in the UI above, or via the Helm chart.

If you plan to deploy charts via Fleet (which I've written about before, or some other GitOps flow, you would pull the following charts from the Rancher chart repo:

helm repo add rancher-charts https://charts.rancher.io
helm repo update
helm fetch rancher-charts/rancher-logging-crd
helm fetch rancher-charts/rancher-logging

but before we apply them, we want to setup the syslog endpoint in the cluster for the logging operator to log to:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rsyslog-deployment
  labels:
    app: rsyslog
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rsyslog
  template:
    metadata:
      labels:
        app: rsyslog
    spec:
      containers:
      - name: rsyslog
        image: sudheerc1190/rsyslog:latest
        ports:
        - containerPort: 514
        resources:
          requests:
            cpu: 250m
            memory: 524Mi
      restartPolicy: Always
      terminationGracePeriodSeconds: 30

here we're just creating a syslog Deployment; for production, I would recommend creating a PersistentVolume and mount it to /var/log/remote so in subsequent recreations of this Pod, you can restore past logging data for this or other clusters.

Because we want this just available internal to the cluster in this example, we'll just create the following Service objects to create cluster DNS entires for this endpoint:

---
apiVersion: v1
kind: Service
metadata:
  name: "syslog-service-tcp"
spec:
  ports:
    - port: 514
      targetPort: 514
      protocol: TCP
  type: LoadBalancer
  selector:
    app: "rsyslog"
---
apiVersion: v1
kind: Service
metadata:
  name: "syslog-service-udp"
spec:
  ports:
    - port: 514
      targetPort: 514
      protocol: UDP
  type: LoadBalancer
  selector:
    app: "rsyslog"

so depending upon which protocol you'd like to use for logging, you can use a hostname like syslog-service-udp.default.svc.cluster.local.

Now, you can create the CRDs and Logging Operator:

helm install rancher-logging-crd ./rancher-logging-crd-100.1.2+up3.17.4.tgz -n cattle-logging-system --create-namespace

helm install rancher-logging ./rancher-logging-100.1.2+up3.17.4.tgz -n cattle-logging-system

Note that the version on these charts should match, and that they should be deployed to the cattle-logging-system namespace.

In the logging operator, there are two pieces Flow (and cluster-wide ClusterFlow) resources and Output (and ClusterOutput) resources. This example will just use cluster-scoped examples that span namespaces, but the matching rules and output formatting will work similarly in both cases.

Our ClusterOutput will be what connects to our Syslog service, using host: syslog-service-udp.default.svc.cluster.local on port 514, in this case using transport: udp:

apiVersion: v1
items:
- apiVersion: logging.banzaicloud.io/v1beta1
  kind: ClusterOutput
  metadata:
    name: testing
    namespace: cattle-logging-system
  spec:
    syslog:
      buffer:
        total_limit_size: 2GB
        flush_thread_count: 8
        timekey: 10m
        timekey_use_utc: true
        timekey_wait: 1m
      format:
        app_name_field: kubernetes.pod_name
        hostname_field: custom-cluster-name
        log_field: message
        rfc6587_message_size: false
      host: syslog-service-udp.default.svc.cluster.local
      insecure: true
      port: 514
      transport: udp
kind: List

and you can verify the status of the output (deployment status, problems, etc.) with kubectl get clusteroutput/testing -n cattle-logging-system. The ClusterOutput should be namespaces with the operator deployment. In the above spec, under format, you can refer to the RFC format for other options for logging fields.

In our ClusterFlow, we're going to use just a basic rule, ask to exclude logs from the kube-system namespace, and then select everything from the applications namespace:

apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterFlow
metadata:
  name: global-ns-clusterflow
spec:
  match:
    - exclude:
        namespaces:
        - kube-system
    - select:
        namespaces:
        - applications
  globalOutputRefs:
    - testing

and you'll see we use the testing output as the global endpoint for sending logs. You can check the status of the Flow using kubectl get clusterflow/global-ns-clusterflow.

I usually test after this point using a deployment like this:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

and then deploy it once to an excluded namespace and then an included one, then make a few requests (curl -s http://{LB_IP}, if the output and flow report no issues, you'll see the logs start to populate after a few minutes). You can then check the logs manually (if nothing is scraping syslog externally) with something formatted like kubectl exec -it {rsyslog_pod} -- tail -f /var/log/remote/{date hierarchy}/{hour}.log.

More about the BanzaiCloud Operator can be found here, and more on the Logging features in Rancher from this operator can be found here.

Changing Netplan Renderer Backend

Joseph D. Marhee — Fri, 06 May 2022 21:04:05 +0000

I recently changed desktop environments on Linux, and found that my choice didn't render really well through xrdp and rather than figure it out, I wanted to try the screensharing option in the Ubuntu 20.04 system settings.

However, when I went to enable it, I found the following error in syslog:

May  6 15:49:08 boris gnome-control-c[64701]: message repeated 3 times: [ Failed to enable service vino-server: GDBus.Error:org.freedesktop.DBus.Error.InvalidArgs: Sharing cannot be enabled on this network, status is '0']

This is a known bug in this GNOME package when your network interface Vino would be binding to is managed by systemd-networkd. You can see which interfaces are currently managed using the networkctl command:

jdmarhee@boris /etc/netplan $ networkctl
IDX LINK       TYPE     OPERATIONAL SETUP    
  1 lo         loopback carrier     unmanaged
  2 eno1       ether    routable    managed
  3 tailscale0 none     routable    unmanaged
  4 docker0    bridge   no-carrier  unmanaged

4 links listed.

where eno1 was being managed. In Ubuntu 20.04, the network interfaces are configured using netplan. The systemd-networkd backend is the default for doing this, so I needed to update netplan to use NetworkManager instead. Your config will be in a file like this one in /etc/netplan:

jdmarhee@boris /etc/netplan $ cat 00-installer-config.yaml 
# This is the network config written by 'subiquity'
network:
  ethernets:
    eno1:
      dhcp4: true
  version: 2

and add the following under network key to specify the backend (I add it right under the version):

  renderer: NetworkManager

and then apply:

sudo netplan apply

You can then verify that that interface is no longer managed:

jdmarhee@boris /etc/netplan $ networkctl
IDX LINK       TYPE     OPERATIONAL SETUP    
  1 lo         loopback carrier     unmanaged
  2 eno1       ether    routable    unmanaged
  3 tailscale0 none     routable    unmanaged
  4 docker0    bridge   no-carrier  unmanaged

4 links listed.

and should be able to enable Vino without issue (amongst other preferences that rely on the interface not being managed by systemd).

Network Policies with Canal and Flannel on K3s

Joseph D. Marhee — Wed, 05 Jan 2022 06:10:55 +0000

Flannel is a popular Container Network Interface (CNI) addon for Kubernetes, however, it does not provide (because it is Layer 3 network focused on transport between hosts, rather than container networking with the host) robust support for NetworkPolicy resources. Now, the policy features from another popular CNI, Calico, can be imported to Flannel using Canal.

I won't talk a lot about specific NetworkPolicy in this post, but a little bit about why you'd want a NetworkPolicy controller is for things like creating policies around access to Ingresses, to or from port or IP ranges, things like that-- the sort of concerns you might have creating ACLs and security rules on a traditional network, but scriptable and templateable for Kubernetes like any other resource type.

Installing Canal requires applying a single manifest (containing the Calico controller, policy agent, and service accounts), however, because the Pod CIDR may differ (and in the case of K3s, it will be 10.42.0.0/24) from the Calico-expected default of 10.244.0.0/16, an environmental variable (CALICO_IPV4POOL_CIDR) and its accompanying value in the manifest.

You can retrieve your Pod CIDR using:

kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}

and then modify that variable (commented out) after you download https://docs.projectcalico.org/manifests/canal.yaml.

Or use a CLI tool like sed to modify it while writing the file locally:

curl -s https://docs.projectcalico.org/manifests/canal.yaml | \
sed \
-e 's|            # - name: CALICO_IPV4POOL_CIDR|            - name: CALICO_IPV4POOL_CIDR|g' \
-e "s|            #   value: \"192.168.0.0/16\"|              value: \"$(kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}')\"|g"

and then (or alternatively) manage this manifest however you typically might do so (in Helm chart, or have an automation tool handle this templating for you, etc.)

If you are a K3s user, conveniently, any manifests written to /var/lib/rancher/k3s/server/manifests will be applied automatically, so you can have the above simply write the file for you when provisioning your cluster:

## Applies the modified manifest to K3s, which automatically applies the contents of /var/lib/rancher/k3s/server/manifests
curl -s https://docs.projectcalico.org/manifests/canal.yaml | sed -e 's|            # - name: CALICO_IPV4POOL_CIDR|            - name: CALICO_IPV4POOL_CIDR|g' -e "s|            #   value: \"192.168.0.0/16\"|              value: \"$(kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}')\"|g" | \
tee -a /var/lib/rancher/k3s/server/manifests/canal.yaml

after your K3s install command, if you'd like, on cluster spin-up.

Checking the status of the calico-controllers Deployment will let you know when you are ready to proceed to introduce policy objects:

kubectl get deployments -n kube-system

Examples of common NetworkPolicy usage can be found here.

Manage an Authoritative DNS Server on Kubernetes with Helm and Fleet

Joseph D. Marhee — Fri, 24 Dec 2021 05:23:19 +0000

Among my other clusters, I run infrastructure-specific services out of a small Kubernetes cluster. One such service (aside from things like a VPN gateway, Docker registry, etc.) is a DNS nameserver pair that I use inside and out of the cluster. I install and configure these namservers using Helm and deploy/update the Deployment (and ConfigMaps that it serves zone files from) using Fleet.

In my authoritative-dns folder in my repo that Fleet watches, after running:

helm create authoritative-dns

which creates the directory and the template chart (i.e. a templates directory, Chart.yaml, etc.), I am mostly going to be concerned with templates/deployment.yaml and templates/configmap.yaml (and optionally templates/service.yaml) being set-up to receive the BIND configuration and zone files I'd like it to serve.

In templates/configmap.yaml, I'm going to set up two ConfigMap resources, one to handle the named.conf configuration, and another to hold the zone file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: bind-config
data:
{{- toYaml .Values.bindconfig | nindent 2 }}
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: bind-zone-config
data:
{{- toYaml .Values.bindzones | nindent 2 }}

The .Values in template brackets ({{ }}) will refer to data in your Values.yaml file, which will be the part of the Chart you want most of your modifications to go into, hold variable value configuration items. I'll return to that in a moment.

In templates/deployment.yaml, we just need a Deployment that will mount these ConfigMap resources to the BIND container:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.name }}
  labels:
    {{- include "authoritative-dns.selectorLabels" . | nindent 4 }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      {{- include "authoritative-dns.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      roll: {{ randAlphaNum 5 | quote }}
      {{- with .Values.podAnnotations }}
      annotations:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      labels:
        {{- include "authoritative-dns.selectorLabels" . | nindent 8 }}
    spec:
      hostNetwork: true
      affinity:
        {{- toYaml .Values.affinity | nindent 8 }}
      containers:
        - name: {{ .Values.name }}-bind
          image: {{ .Values.image.image }}:{{ .Values.image.tag }}
          imagePullPolicy: IfNotPresent
          ports:
            - name: dns-tcp
              containerPort: {{ .Values.service.targetPort }}
              protocol: TCP
            - name: dns-udp
              containerPort: {{ .Values.service.targetPort }}
              protocol: UDP
          {{- if .Values.readinessProbe.enabled }}
          readinessProbe:
            tcpSocket:
              port: dns-tcp
            initialDelaySeconds: 5
            periodSeconds: 10
          {{- end }}
          {{- if .Values.readinessProbe.enabled }}
          livenessProbe:
            tcpSocket:
              port: dns-udp
            initialDelaySeconds: 5
            periodSeconds: 10
          {{- end }}
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
          volumeMounts:
          - name: bind-config
            mountPath: /etc/bind
          - name: bind-zone-config
            mountPath: /var/lib/bind
      volumes:
        - name: bind-config
          configMap:
            name: bind-config
            items:
              - key: named.conf
                path: named.conf
        - name: bind-zone-config
          configMap:
            name: bind-zone-config
{{- toYaml .Values.zoneconfigs | nindent 12 }}

where in this case, we want:

        - name: bind-zone-config
          configMap:
            name: bind-zone-config
{{- toYaml .Values.zoneconfigs | nindent 12 }}

to refer to each zone file ConfigMap file, which you'll also populate in your Values.yaml file as well, and then appear in the container under /var/lib/bind/{{your zone}}.

Note: the Deployment contains an annotation, roll: {{ randAlphaNum 5 | quote }}-- this is because BIND requires a restart (and thereforce the Pods need to be restarted) to pick up changes in the zone configs. This will generate an updated value for the Pod to be annotated with triggering the restart-- the above restarts the Pod on every upgrade, but there's other options to roll your deployment pods (i.e. based on changes to the configmap specifically, etc.)

This Deployment, because of how I use it, just uses the host network, so a Service is not required, but you can use a template like this template/service.yaml to expose on ports 53/TCP and 53/UDP with a LoadBalancer, in this case with MetalLB:

apiVersion: v1
kind: Service
metadata:
  name: {{ include "authoritative-dns.name" . }}-tcp
  labels:
    {{- include "authoritative-dns.labels" . | nindent 4 }}
  {{- if .Values.service.annotations }}
  annotations:
    {{- toYaml .Values.service.annotations | nindent 4 }}
  {{- end }}
spec:
  type: {{ .Values.service.type }}
  externalTrafficPolicy: Local
  ports:
    - port: {{ .Values.service.targetPort }}
      targetPort: {{ .Values.service.targetPort }}
      protocol: TCP
      name: dns-tcp
  selector:
    {{- include "authoritative-dns.selectorLabels" . | nindent 4 }}
---
apiVersion: v1
kind: Service
metadata:
  name: {{ include "authoritative-dns.name" . }}-udp
  labels:
    {{- include "authoritative-dns.labels" . | nindent 4 }}
  {{- if .Values.service.annotations }}
  annotations:
    {{- toYaml .Values.service.annotations | nindent 4 }}
  {{- end }}
spec:
  type: {{ .Values.service.type }}
  ports:
    - port: {{ .Values.service.targetPort }}
      targetPort: {{ .Values.service.targetPort }}
      protocol: UDP
      name: dns-udp
  selector:
    {{- include "authoritative-dns.selectorLabels" . | nindent 4 }}

So, with these templates in place, now we need a Values.yaml to supply the data:

replicaCount: 2
region: ORD1
name: authoritative-dns
image:
  image: internetsystemsconsortium/bind9
  pullPolicy: IfNotPresent
  tag: 9.11

imageConfig:
  pullPolicy: IfNotPresent

service:
  type: LoadBalancer
  port: 53
  targetPort: 53
  labels:

# resources:
#   requests:
#     memory: 1Gi
#     cpu: 300m

readinessProbe:
  enabled: true
livenessProbe:
  enabled: true

bindzones:
  c00lz0ne.internal: |
      $TTL    604800
      @       IN      SOA     ns1.c00lz0ne.internal. admin.c00lz0ne.internal. (
                                    5         ; Serial
                               604800         ; Refresh
                                86400         ; Retry
                              2419200         ; Expire
                               604800 )       ; Negative Cache TTL
      ;
      c00lz0ne.internal.       IN      NS      ns1.c00lz0ne.internal.
      c00lz0ne.internal.       IN      NS      ns2.c00lz0ne.internal.
      ns1                      IN      A       10.24.0.11
      ns2                      IN      A       10.24.0.12
      rke00.mgr                IN      A       10.24.0.99
      rke01.mgr                IN      A       10.24.0.100
      rke-lb.mgr               IN      A       100.69.29.9

bindconfig:
  named.conf: |
    options {
            directory "/var/cache/bind";
            listen-on port 53 { any; };
            auth-nxdomain yes;
            forwarders { 
                    1.1.1.1; 
                    1.0.0.1; 
            };
            listen-on-v6 { ::1; };
            allow-recursion {
                    none;
            };
            allow-transfer {
                    none;
            };
            allow-update {
                    none;
            };
    };

    zone "c00lz0ne.internal" {
      type master;
      file "/var/lib/bind/c00lz0ne.internal";
    };

zoneconfigs:
  items:                 
  - key: c00lz0ne.internal
    path: c00lz0ne.internal

So, you'll see for my cool domain c00lz0ne.internal, I need to populate bindzones.c00lz0ne.internal with the BIND zonefile itself, then bindconfig.named.conf needs to be updated to add that file:

    zone "c00lz0ne.internal" {
      type master;
      file "/var/lib/bind/c00lz0ne.internal";
    };

which you'll see in the above uses the mount path we defined in the Deployment for the ConfigMap containing these files, and then finally, zoneconfigs.items which should contain a map of keys and paths for each of the domain zone files you created, and then referenced in named.conf so it mounts to the Deployment template when it renders.

Typically, at this point, you could apply your chart:

helm install authoritative-dns ./

and then when the Deployment is online:

jdmarhee@boris ~/repos/terraform-digitalocean-k3s-highavailability (master) $ kubectl get pods
NAME                                 READY   STATUS    RESTARTS   AGE
authoritative-dns-6576df5d48-7glqk   1/1     Running   0          15s
authoritative-dns-6576df5d48-nhjsq   1/1     Running   0          15s

you can test resolution:

dig +short -t ns ns1.c00lz0ne.internal @${SVC_IP}

However, because we want the Deployment to update with the changes to the ConfigMap and other values, we can use a GitOps tool like Fleet to update on changes to the repository.

Let's say I keep all of my charts in a repo (Fleet also supports private repositories, but for the sake of demonstration, a basic public repo) called helm-charts:

apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: sample
  # This namespace is special and auto-wired to deploy to the local cluster
  namespace: fleet-local
spec:
  # Everything from this repo will be ran in this cluster. You trust me right?
  repo: "https://github.com/c00lz0ne-infra/helm-charts"
  paths:
  - authoritative-dns

where paths represent the charts that Fleet will track changes to, once this is applied and you can see Fleet is running.

In my case, I only want this chart deployed to specific clusters (in this case, with env matching svcs), so to the above GitRepo, I add:

  targets:
  - clusterSelector:
      matchLabels:
        env: svcs
    name: svcs-cluster

so when this, or a new cluster with that label match (or using any of the other methods of mapping to downstream clusters), the chart will be applied on each change to a specified branch or revision in the git repository.

In my case, I will only have a small and fixed number of clusters that will ever be managed that way, but if you want to great ClusterGroups to target (using these matching rules, rather than mapping the GitRepo to the rule, map the GitRepo to the clustergroup, and manage it as its own resource), it can be defined like this:

kind: ClusterGroup
apiVersion: fleet.cattle.io/v1alpha1
metadata:
  name: services-group
  namespace: clusters
spec:
  selector:
    matchLabels:
      env: svcs

and then update the GitRepo resource to use this matching rule:

    clusterGroup: group1

or:

    clusterGroupSelector:
      matchLabels:
        region: us-east

or optionally use a clusterGroupSelector, or a combination of these to subset this group further (for example, in region us-east within that group), name a specific group, or in the above example, find ClusterGroups that are in a specific region.

The result of all of this is that rather than writing complex rules for when and how to run Helm to target multiple clusters, Fleet will use the above resources to manage this application of chart changes on a specific branch, across the defined clusters.

Using OpenPolicyAgent to Inject Proxy Configuration into Kubernetes Workloads

Joseph D. Marhee — Sat, 18 Dec 2021 06:49:46 +0000

It's not unusual to find enterprise services behind a corporate proxy, and in order to get out, must have this proxy access configured to access out through it. The same applies to services running o Kubernetes.

One way to do this is manually, by creating a Secret resource containing a proxy string like the following:

kubectl create secret generic corporate-proxy --from-literal=authentication-string="http://{USERNAME}:{PASSWORD}@{PROXY_ADDRESS}:{PROXY_PORT}" --from-literal=no-proxy-string="localhost,127.0.0.1,0.0.0.0,10.0.0.0/8,cattle-system.svc,10.42.0.0/24,.svc,.cluster.local,example.com"

and, mount it to the environment of a Kubernetes application resource:

apiVersion: v1
kind: Pod
metadata:
  name: appWithProxy
  namespace: default
spec:
  containers:
  - name: appWithProxy
    image: yourApplication
    env:
    - name: HTTP_PROXY
      valueFrom:
        secretKeyRef:
          name: corporate-proxy
          key: authentication-string
    - name: NO_PROXY
      valueFrom:
        secretKeyRef:
          name: corporate-proxy
          key: no-proxy-string

and when you exec into that container, you can use that proxy to access public resources:

curl --proxy $HTTP_PROXY http://ipinfo.io/json

However, this is a tedious, manual process that you would, then, need to repeat for every resource you'd like to have proxy access.

Among the many things OpenPolicyAgent Gatekeeper does, is allow you to mutate Kubernetes API requests based on a variety of scoping rules, and inject things like environment variables into resources that meet said scope.

This is done using a Mutating Admission Webhook. Using Mutating Admission Webhook to inject environment variables like these proxy settings, for example, to modify requests to create a resource like a Pod or Deployment, etc. to inject the proxy configuration data. There are other types of Webhooks, such as (in the linked example) Validating Webhooks, which do things like confirm a workload has a label, or an appropriate name, etc. to dictate some other action by the Admission Webhook.

OPA Gatekeeper, in this scenario, will do much of this for you, and instead, mutation and scoping is dictated by policies. In our case, we want to create a namespace that will contain resources that we wish to mutate, upon request, to include mounting the above corporate-proxy resource Secret as an environment variable.

OPA also provides facilities for testing these policies.

After installing OPA Gatekeeper:

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/release-3.7/deploy/gatekeeper.yaml

Or if you're a Rancher user, Gatekeeper can be installed and enabled from the UI.

We can then define a ModifySet resource:

apiVersion: mutations.gatekeeper.sh/v1beta1
kind: ModifySet
metadata:
  name: demo-annotation-owner
  namespace: proxy-test
spec:
    applyTo:
      - groups: [""]
        versions: ["v1"]
        kinds: ["Pod", "Deployment"]
    match:
      scope: Namespaced
      kinds:
        - apiGroups: ["*"]
          kinds: ["Pod"]
    location: "spec.containers[name:*].env"
    parameters:
      values:
        fromList:
          - name: HTTP_PROXY
            valueFrom:
              secretKeyRef:
                name: corporate-proxy
                key: authentication-string
          - name: NO_PROXY
            valueFrom:
              secretKeyRef:
                name: corporate-proxy
                key: authentication-string

which is deployed to the proxy-test namespace, and a couple of lines down, the scope is set to Namespaced, meaning that it will only apply the following to matching resources in that namespace.

If the API server passes a request that is that namespace, and matches any apiGroup ("*"), and is of kind: Pod, it will read a spec like the following:

apiVersion: v1
kind: Pod
metadata:
  name: shell-demo
  namespace: proxy-test
spec:
  volumes:
  - name: shared-data
    emptyDir: {}
  containers:
  - name: nginx
    image: nginx
    env:
    - name: SOME_EXISTING_VAR
      value: "Present"
    volumeMounts:
    - name: shared-data
      mountPath: /usr/share/nginx/html
  dnsPolicy: Default

where it will look in spec.containers, and you'll see since this request does not contain an env block, it will inject our Secret from above (secret/corporate-proxy) into this spec appending env represented above in Yaml, but can also be inserted as JSON:

[{"name": "HTTP_PROXY", "valueFrom": {"secretKeyRef": {"name": "corporate-proxy", "key": "authentication-string"}}}]

and create the Pod.

You can test this out by creating two Pods using the above spec, one in the proxy-test namespace and one in another:

cat << EOF >> proxy.yaml
apiVersion: v1
kind: Pod
metadata:
  name: shell-demo
  namespace: proxy-test
spec:
  volumes:
  - name: shared-data
    emptyDir: {}
  containers:
  - name: nginx
    image: nginx
    env:
    - name: SOME_EXISTING_VAR
      value: "Present"
    volumeMounts:
    - name: shared-data
      mountPath: /usr/share/nginx/html
  dnsPolicy: Default
EOF

kubectl apply -f proxy.yaml

and one in another:

cat << EOF >> no-proxy.yaml
apiVersion: v1
kind: Pod
metadata:
  name: shell-demo
  namespace: default
spec:
  volumes:
  - name: shared-data
    emptyDir: {}
  containers:
  - name: nginx
    image: nginx
    env:
    - name: SOME_EXISTING_VAR
      value: "Present"
    volumeMounts:
    - name: shared-data
      mountPath: /usr/share/nginx/html
  dnsPolicy: Default
EOF

kubectl apply -f no-proxy.yaml

and when you run kubectl describe on the namespaced Pod, you'll see the following:

root@ubuntu-proxy-k3s:~# kubectl describe pod/shell-demo -n proxy-test
Name:         shell-demo
Namespace:    proxy-test
...
Containers:
  nginx:
    Image:      nginx
...
    Environment:
      SOME_EXISTING_VAR:     "Present"
      HTTP_PROXY:  <set to the key 'authentication-string' in secret 'corporate-proxy'>  Optional: false
      NO_PROXY:    <set to the key 'authentication-string' in secret 'corporate-proxy'>  Optional: false

where the one in the default namespace will not have an Envrionment set.

These ModifySet resources can be written to mutate based on a number of other scopes as well, and any part of a manifest that you specify in the location key, and then assign with a value under parameters as we did above. Other examples of using OPA Gatekeeper resources to mutate resources can be found here.

Capturing and Posting Video Screenshots at Random Intervals

Joseph D. Marhee — Tue, 31 Aug 2021 05:08:35 +0000

I recently created a couple of bot Twitter accounts, @ScreenshotsVice which shares screencaps from episodes of "Miami Vice" and @Pasolinibot which shares caps from the filmography of Pier Paolo Pasolini, both at regular intervals.

The way these accounts are managed are that, when a cronjob is triggered, from a video library, a series of screencaptures are dumped (if they were set to refresh on the previous run), and then the listing in the directory is shuffled, and then the script selects one at random and posts it to Twitter.

Because the intervals are managed by cronjobs, I'll focus on what happens when a job is triggered.

The Python script that selects and posts the capture relies on the tweepy Twitter API client for Python, so you will need your Twitter Developer API credentials. The script that generates the captures, before running, requires that ffmpeg be installed and the directory where the video library is stored and the directory where you'll store screencaptures.

To generate the capture library, the following script is used:

#!/bin/bash

LNVC_DATA_PATH=${SOURCE_DIR}
LNVC_PREV_PATH=${SOURCE_DIR_OUT}-Screens
REFRESH_SCREENS=1
DUMP_INTERVAL=1
DUMP_RANDOM=1
RANDOMIZE_NAMES=1

if [ -z "${LNVC_DATA_PATH}" ]; then
    echo "No source path found. Set LNVC_DATA_PATH." ; exit 1
fi

if [ -z "${LNVC_PREV_PATH}" ]; then
    echo "No target path found. Set LNVC_PREV_PATH." ; exit 1
fi

if [ -n "${REFRESH_SCREENS}" ]; then
    rm -rf $LNVC_PREV_PATH/*
fi

This sets up options for things like randomizing filenames, whether or not to refresh the library, defining the video source and the capture outputs, the latter two are the only ones that are required.

After this, the video library is iterated, and the captures created:

for v in `ls $LNVC_DATA_PATH`; do \
    if [ -n "${DUMP_RANDOM}" ]; then
        LENGTH=$(ffmpeg -i $LNVC_DATA_PATH/$v 2>&1 | grep Duration | cut -d ' ' -f 4 | sed s/,//) ; \
        HOUR=$(echo $((0 + RANDOM % $(echo $LENGTH | cut -d ":" -f 2)))) ; \
        if (( $HOUR < 10 )); then HOUR=$(printf "%02d\n" $HOUR); fi ; \
        MINUTE=$(echo $((1 + RANDOM % $(echo $LENGTH | cut -d ":" -f 2)))) ; \
        if (( $MINUTE < 10 )); then MINUTE=$(printf "%02d\n" $MINUTE); fi ; \
        SECOND=$(echo $((1 + RANDOM % 59))) ; \
        if (( $SECOND < 10 )); then SECOND=$(printf "%02d\n" $SECOND); fi ; \
        ffmpeg -i $LNVC_DATA_PATH/$v -ss $HOUR:$MINUTE:$SECOND.000 -vframes 1 $LNVC_PREV_PATH/`head /dev/urandom | tr -dc A-Za-z0-9 | head -c 13 ; echo ''`.jpg
    fi

    if [ -n "${DUMP_INTERVAL}" ]; then
        ffmpeg -i $LNVC_DATA_PATH/$v -frames:v 1 -vf fps=1/$DUMP_INTERVAL $LNVC_PREV_PATH/`head /dev/urandom | tr -dc A-Za-z0-9 | head -c 13 ; echo ''`.jpg
        # The filenames will be prefixed by a random string, but per-video; if you'd like them randomized in totality, set `RANDOMIZE_NAMES`
        if [ -n "${RANDOMIZE_NAMES}" ]; then
            cd $LNVC_PREV_PATH ; \
            for f in `ls`; do mv $f `head /dev/urandom | tr -dc A-Za-z0-9 | head -c 13 ; echo ''`.jpg; done
        fi
    fi
done

If DUMP_RANDOM is enabled, random timecodes will be dumped from a video file, DUMP_INTERVAL will dump video files into captures at each interval (useful for films, rather than slices from an episode of a TV series). Filenames (which helps when selecting images randomly to post) can be selectively re-randomized.

This script has dumped videos from SOURCE_DIR into SOURCE_DIR_OUT, which our poster script will read from.

First, we have to setup the Twitter authentication:

import tweepy
import os, random


def main():

    twitter_auth_keys = {

        "consumer_key"        : os.environ['twitter_consumer_key'],

        "consumer_secret"     : os.environ['twitter_consumer_secret'],

        "access_token"        : os.environ['twitter_access_token'],

        "access_token_secret" : os.environ['twitter_access_token_secret']

    }



    auth = tweepy.OAuthHandler(

            twitter_auth_keys['consumer_key'],

            twitter_auth_keys['consumer_secret']

            )

    auth.set_access_token(

            twitter_auth_keys['access_token'],

            twitter_auth_keys['access_token_secret']

            )

    api = tweepy.API(auth)

Then lock in our selected image:

    files = os.listdir(os.environ['SOURCE_DIR_OUT'])
    for i in range(1,10000):
        random.shuffle(files)
    path = os.environ['SOURCE_DIR_OUT'] + "/" + random.choice(files)
    media = api.media_upload(path)

you'll see above that it reads the captures into a list of filenames, then shuffles the list, before selecting one at random, then we post:

    tweet = "#pasolini"

    post_result = api.update_status(status=tweet, media_ids=[media.media_id])



if __name__ == "__main__":

    main()

The package can be found on Github along with some additional examples (one using a Kubernetes CronJob resource, for example) for use.

Using pre-commit and post-update git hooks

Joseph D. Marhee — Wed, 09 Jun 2021 05:42:08 +0000

Adding some additional processing to your git workflow is sometimes useful/cool/interesting, and the functionality provided by githooks make this fairly accessible to make use of. I typically make use of very few of these (in place of having things happen on the server-side — tests, etc.-that can be enabled externally or using server-side hooks) on the client-side, and adapt two included in .git/hooks/ by default, pre-commit and post-update, for, you guessed it, before commits are made, and after a set of commits are pushed.

In my capacity as an operations engineer, I make use of tools like Terraform often, which has the benefit of including a formatting tool and a validation tool — this is a good example of where a pre-commit hook can be useful — before I create a commit, I can validate the manifest and check formatting/style of the manifests being updated. I can do this by modifying, in my project root, .git/hooks/pre-commit.sample, and add something like this to check .tf files against these standards:

modified_files=$(git ls-files -m)
for f in modified_files
do
    if [-e "$f" ] && [[ $f == *.tf ]]; then
        terraform validate $(dirname $f)
        terraform fmt $f -check=true
        git add $f
    fi
done

I’ve adapted my approach from James Turnbull’s pre-commit hook script to process this slightly differently, but also highlights that these are basically just a scriptable interface to manage git’s behavior.

To enable this hook, copy the file to .git/hooks/pre-commit (no .sample).

Let’s take a look at a more involved example in my post-update script. In my case, I am working on a Mac, so I’d like to make the most of this — maybe introduce some more visual cues for me to follow, so I want to use some Applescript as well.

In .git/hooks/post-update, I want to grab the hash and message of the last commit pushed, and the branch I pushed to:

COMMIT=$(git log -1 HEAD | head -n 1 | awk '{print $1}')
MESSAGE=$(git log --format=%B -n 1 $COMMIT | xargs echo )
BRANCH=$(git branch | grep \* | cut -d ' ' -f2)

and to make this useful, I can see a summary of my latest push by adding this call to osascript:

/usr/bin/osascript <<EOF
display dialog "To $BRANCH:\n\n$MESSAGE\n\n\t($COMMIT)\n" with title "Git Push" buttons {"I meant to do that"} default button 1

To have a pop-up dialog and just a reminder of what I did, so I stop to check my work one last time before moving on to my next task.

Configuring and Managing Routes Between Multiple Networks with Wireguard

Joseph D. Marhee — Tue, 08 Jun 2021 08:16:42 +0000

I recently updated the VPN solution in my infrastructure lab using Wireguard; my architecture is fairly basic, in that each site (in this case, a handful of colocated environments, and multiple cloud providers) runs a Wireguard endpoint, which then are peered with one-another to connect my service network (rather than that of the hosts themselves) across these sites and providers.

I made the switch after using Wireguard in the context of direct connecting cloud provider networks in my work, and decided to do the same for my lab, which spans a much smaller scale, but a much more diverse set of machines and networks, which made managing a large OpenVPN configuration somewhat cumbersome, and found Wireguard straight-forward, if a little light on diverse documentation, but reasonably well-suited to automation.

There are many excellent guides online for configuring Wireguard between multiple peer nodes, and for my use-case, I found that I need additional route changes to allow each peer to access the LANs the accompanying peered nodes were a part of.

For example, assuming a network, 192.168.2.0/24 for the Wireguard interfaces themselves, my first server in one location, 192.168.2.1 was part of a network in that location for machines not, themselves, setup with Wireguard, 192.168.122.0/24 , and another endpoint accessible on Wireguard as 192.168.2.2, 192.168.1.0/24 , and I wanted this latter set of machines to be able to access the machines in the LAN for the former, but not the other way around.

The standard Wireguard config supports PostUp and Down arguments to add additional routing changes, and support for things like configuring NAT with iptables . In this respect, this is the only non-standard use of Wireguard in-use in my project.

On my server, my configuration looked like this:

[Interface]
Address = 192.168.2.1                                                                                                   PrivateKey = (Hidden)                                                               ListenPort = (Whatever)                                                                                                     PostUp   = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o virbr0 -j MASQUERADE                                                                                                     PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o virbr0 -j MASQUERADE

with [Peer] blocks for two other endpoints that can access the networks attached to 192.168.2.1 ‘s virbr0 interface:

[Peer]                                                                                                                  PublicKey = TH+DHbSc7ALMapPCsUlCVzWJdQKtFPNfW8Hkel8m2Uw=                                                                Endpoint = (Host):(Port)                                                                                         PersistentKeepalive = 25                                                                                                AllowedIPs = 192.168.2.3/32  
                                                                                                                                                                                                                   [Peer]                                                                                                                  PublicKey = 6w/+laHf7m0T13tgogRbwKp5na20yygLObLL9AgSHH8=                                                                AllowedIPs = 192.168.2.2/32                                                                                             Endpoint = (Host):(Port)

Pretty straightforward, but to drill down on the behavior I’m hoping to achieve here, only this server 192.168.2.1 will have PostUp and Down rules:

iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o virbr0 -j MASQUERADE

in this case, to forward traffic over the wg0 interface (in my case, but in the above, the %i will automatically substitute the interface you’re bringing online when you do things like wq-quick up {interface name} ) to the target network you’d like to expose to the peers (and then enable in the client config, which I’ll cover in a moment), which in this case is the network attached to the bridged interface, virbr0 .

Because I wanted my peer sites to be able to run workloads in and out of this LAN attached to 192.168.2.1 endpoint, my client config does not require PostUp or PostDown -managed route changes, but does need to be aware of the networks it will be exposed to (via those firewall changes on that endpoint — changes we will not be making on these client peer endpoints):

[Interface]                                                                                                             PrivateKey = (Hidden)                                                               ListenPort = (Port)                                                                                                     Address = 192.168.2.2/24                                                                                                DNS = 1.1.1.1
                                                                                                                                                                                                                                 [Peer]                                                                                                                  PublicKey = mXvCVKYmYuYL0AO2AvDPsLmOdUGwXDi5d2DekKCt9RY=                                                                AllowedIPs = 192.168.2.0/24, 192.168.122.0/24                                                                           Endpoint = (Host):(Port)                                                                                           PersistentKeepalive = 25

So, in this configuration, we’re exposing a new endpoint at 192.168.2.2 to peer with 192.168.2.1 , and in AllowedIPs , we not only want to be aware of the Wireguard network we’re declaring 192.168.2.0/24 , but the network attached to the first host’s virbr0 interface, in this case, 192.168.122.0/24 .

Behind this new endpoint is a network, 192.168.0.0/24 that the first host cannot be routed back to without adding a similar rule on this new endpoint’s Interface configuration, but if we were to add similar rules to the iptables rules we added above, and then in the config for the first server’s Peer block:

[Peer]                                                                                                                  PublicKey = 6w/+laHf7m0T13tgogRbwKp5na20yygLObLL9AgSHH8=                                                                AllowedIPs = 192.168.2.2/32, 192.168.0.0/24                                                                                             Endpoint = (Host):(Port)

like so, this remote peer’s network could similarly forward traffic, and route it appropriately.

Basically, at the end of this, you can use Wireguard as a site-to-site VPN, and much like you would with any VPN solution, more flexibly operate access to the networks attached to each endpoint, and similar to the push "${some route}" behavior in OpenVPN, for example, using the lean interface of the Wireguard configuration format to accomplish this.

Additional Resources

Configuring Wireguard

Kilo — Wireguard-based multi-cloud overlay