Forem: Alex Leonhardt

CKA exam prep tips

Alex Leonhardt — Thu, 28 May 2020 11:15:47 +0000

I thought I share a couple things about how I'm preparing for my CKA exam, so here it is...

read Managing Kubernetes (available by O'Reilly), make notes, better, make ANKI notes (you'll use those later), go back and read the parts that are a little unclear again
do at least one online CKA prep course, it'll make sure you know all the basics, if you already do and this was boring, great! I'm sure there will be the odd thing you didn't quite know yet though
- there are some good and cheap ones out there, I realise that not everyone is able to do this though :(
use ANKI to add new notes to remember, rinse & repeat rigorously every day, good examples to add are
- kubectl commands that help you get set up
- best keywords to look for on k8s docs to find details how to do something (e.g. affinity)
- .. you get the gist ..
use KinD to do a task a day
do a mock exam (20-30 tasks) at least once a week*
- set up a new cluster, add and remove nodes as part of your mock exam, every time
- make sure you cover all things etcd, especially backup & restore
- have tasks that exercise every possible resource in a cluster, e.g. Pod, Deployment, DaemonSet, StatefulSet, RBAC, Volumes, Secrets, ConfigMap, Quotas, CSR, etc. etc.
- many resources also have "commands" that you may need to know how to use, e.g. how do you scale a deployment up/down, how do you get a Pod's log output, etc. most of these are part of the "daily administration" of a cluster so they need exercising as part of your prep
use a timer when doing mock exams or the daily task/s, remember time isn't your friend, current time limit is 3 hours and trust me, that's over sooner than you'd think!
ever wondered "what happens when I ..." or "I wonder how to do X" ? do it, see what happens, really undestand what's happening, try to fix it afterwards too - e.g. try enabling firewalld is a good one ;)
make sure your mock environment is the same as the actual test environment, that means, for example, the same OS (Ubuntu16 at the time of writing)
- I use one of the public cloud providers to set up my test environment, but feel free to use whatever fits your budget; Digital Ocean, GCP, AWS, Linode should all have Ubuntu16 cloud images available
make sure you're using the correct version of k8s, when I started preparing, the test asked for 1.17, by now it's already 1.18 and it'll be a moving target, so I'd suggest to always use the latest major release
know a thing or two about iptables, docker, systemd - nothing too fancy, but you should know how to use it

* I'll share my mock exam in another post

That's all folks, now go forth and be awesome!

go apps + jaeger tracing

Alex Leonhardt — Sat, 11 Apr 2020 18:53:59 +0000

So I've been playing with Jaeger tracing over the last two days a little, to distract me from usual work and thought I'll share what I've put together.

This isn't a tutorial or such, as I still don't know enough about how opentracing / jaegertracing works, so enjoy the code on github, happy to try and help if anyone's got a question (and if I can provide an answer).

TL;DR

The final code works "well", with traces spanning more than one app, with errors, comparing traces, etc.

Github: go-trace-example

Here some screenshots..

About being pragmatic...

Alex Leonhardt — Fri, 29 Nov 2019 07:59:19 +0000

Obviously, we can look up pragmatic if the "definition" is a bit hazy, but what does it mean to you and your everyday work, as a DEV, Dev/OPS or SRE?

(Credit: Photo by Joshua Newton on Unsplash)

What I learned this week: running functions concurrently

Alex Leonhardt — Sat, 23 Nov 2019 19:06:46 +0000

One of Go's strength is (apparently) that it's (relatively) straight forward to write concurrent code.

This week, I've had a "simple" idea (or task, really):

Execute 2 commands (concurrently) and wait for them to complete, then print the output.

Sounds simple, though isn't, quite ...

Constraints

must not know how many of the commands I'd like to execute ahead of time; i.e. I should easily be able to run 20, without having to modify the code (other than add the function/s to run the command)
must wait for all commands to have completed
keep it as simple as possible
the function being executed (the one doing the actual work) should not need to be modified

After many hours of wrestling (with my code), not only to understand go routines, channels and waitGroups (and I'm sure I am still missing quite a lot) but also for it to make sense in my head, I think I came up with a pretty good solution to this.

It is however quite a bit lengthier than I'd like, so maybe there's an even simpler way of doing this? Please leave tips in the comments ;).

So initially, I tried to keep the amount of channels to a minimum, say one for outputs, one for errors, should be sufficient, regardless of the amount of commands that are going to be run; turns out though that this makes shutting down cleanly (or at all) quite a bit difficult. I also kept running into a kind of deadlock where it just wouldn't continue as either I was receiving from an empty channel, or sending into the channel without having a receiver, and many other permutations.

Tip: unbuffered Go channels require you to have a receiver on the channel in order to make progress.

On the other side of the spectrum, trying to simplify things (or basically almost start from scratch again) I got the commands and all to run, but the go-routines closed down way too early, and without adding more complexity it seemed impossible to print all the output and only then exit the program.

Tip: sync.waitGroups are good at waiting for things ;)

So I added a waitGroup, increased its counter for each command to run, and then... it got a bit racy; turns out, sometimes the counter wasn't increased in time. This was because I did the wg.Add(1) inside the go-routine, which meant that the rest of the code moved along in the meantime and weird things can/will/shall happen then.

I used an anonymous function to be run as go-routine (inline) to kick off the command I needed to run, that saved me from having to declare a new function, this would then look something like this...

wg.Add(1)
go func() {
  out, err := doLS("/var/tmp")
  wg.Done()  // decreases the counter in the waitGroup
}()

and somewhere down the line I'll wait for the counter to become 0

[...]
wg.Wait()

At this point, I could have used a buffered channel with the waitGroup to keep track of the commands still executing and kick off a receiver that reads from the channel where all the outputs are being sent into, but that would mean that I'd potentially have to remember to increase the buffer at some point.

Tip: buffered channels don't block, at least not at first

To avoid this, I wanted to keep using the unbuffered channel, however, I had to get past the blocking on the receiver. So, why not run the receiver in another go-routine? That worked, except, I somehow needed to tell it to shutdown once all commands completed.

I tried using (yet) another channel to shutdown the go-routine; using a "done" channel, that looked something like this...

var done = make(chan bool)
go func(done chan bool, output chan string){
    // for loop over output channel, etc...
    if !ok {
      done <- true
      break
    }
}(done)

and meant that after wg.Wait() returned and the execution flow continued, I could've consumed from the channel with <-done and in theory, this should've blocked until all contents of the output channel had been printed.

That didn't work initially, as for the channel to not be OK, it'd have to be closed, though the closing of the output channel was deferred way up there. Adding a close(output) after wg.Wait() I think fixed this, IIRC. (It's been a long week ;))

Luckily, I came across Francesc Campoy's Why nil channels exist some time ago and meant to watch it back then, so today was finally a good time to do that.

The example in the video didn't match exactly what I was trying to do, but I took the idea of the nil channel and ran with it.

The final result of all of this week's hackeroo looks something like this..

So, this is my

.ltag__tag__id__39216 .follow-action-button{ background-color: !important; color: !important; border-color: !important; }

#whatIlearnedthisweek

Maybe I'll have something interesting next week again. Please leave comments or questions or maybe ideas of how else this could've been done!?

Photo: Unsplash.com - Jake Blucker

Numbers everyone should know ... again.

Alex Leonhardt — Sun, 17 Nov 2019 15:51:32 +0000

.. but, why bother writing about this again?

Well, because in my opinion, 10000000ns may be accurate, but it's also a bit hard to convert into something useful, quickly.

I believe that these numbers should reflect something useful; for example, do you really need to remember that it takes 0.5ns[1] for a CPU L1 cache reference?

It's quite likely that your day-to-day does not depend on knowing this by heart, knowing where to quickly find that number if you need it, is a different thing. See the reference/s below. [1]

To me personally, useful numbers are things you can see and use every day, for example, when I ping an IP address and it takes 234ms (default packet size is 56 bytes), then I know that that's slow, because...

Example [1]	Latency (Human)	Latency (ns)
Mutex lock/unlock	0.0001ms	100
Main memory reference	0.0001ms	100
Read 2K bytes sequentially from memory	0.0004ms	488
Compress 2K bytes with Zippy	0.02ms	20 000
Send 2K bytes over 1 Gbps network	0.02ms	20 000
Read 1 MB sequentially from memory	0.25ms	250 000
Read 1 MB sequentially from network	10ms	10 000 000
Read 1 MB sequentially from disk (not SSD)	30ms	30 000 000
Disk seek (not SSD)	10ms	10 000 000
Round trip within same datacenter	0.05ms	500 000
Send packet CA->Netherlands->CA	150ms	150 000 000

Of course, above numbers are useful, but I find them much more useful in ms instead of ns as I can now easier reason about how slow or fast something is.

What other numbers may be useful? I tend to find the following quite helpful too...

Example	-
Read 1MB from loopback (~10Gbps)	1ms
DNS recursive lookup to 1.1.1.1	20ms
1Gbit Network throughput	125 MByte/s
... -> 100Mbit	12.5 MByte/s
... -> 10Mbit	1.25 MByte/s
1 MB/s Network throughput	8 Mbit/s
GCP inter-zone transfer 1 MB from network (*)	10ms
GCP inter-zone transfer 10 MB from network (*)	70ms
GCP inter-zone transfer 100 MB from network (*)	530ms

(*) instance type: n1.standard, zone: us-central1-{a,b}, measured with iperf3 on 17 Nov 2019

When thinking about engineering a new system, try to keep above numbers in mind and do some quick calculations. Rough estimates will go a long way.

Obviously, above numbers do not take CPU and Memory requirements into account, some of which could be calculated based on the software vendor's information and of course at least some performance testing to establish the system's real boundaries and baseline performance.

All this ultimately leads to the idea of Non-Abstract Large System Design referred to in Google's SRE Workbook [2].

Please let me know if you have any feedback or find any errors, I'll be happy to correct them.

References:

[1] - Google Pro Tip: Use Back-Of-The-Envelope-Calculations To Choose The Best Design
[2] - SRE Workbook - NALSD

Enums in Go - why and how?

Alex Leonhardt — Tue, 15 Oct 2019 06:25:44 +0000

Go doesn't (yet) have enums as such, but it seems common to use constants with iota for this instead.

Passing enums instead of strings to a constructor helps reduce possible bugs; when a constructor accepts a set of possible options for an argument, it's easy to "fat finger" (i.e. misspell) them.

Using custom "enums" helps to prevent this and also documents available options for that argument.

Here's what I found on stackoverflow on the why (above), and here's an example I've implemented.

https://github.com/alex-leonhardt/go-passing-enums/

main.go

package main

import (
    "fmt"

    "github.com/alex-leonhardt/go-passing-enums/pkg/config"
)

func main() {
    cfg, err := config.New(config.Env)
    fmt.Printf("%#v %#v\n", cfg, err)
    cfg, err = config.New(config.File)
    fmt.Printf("%#v %#v\n", cfg, err)
    cfg, err = config.New(42)
    fmt.Printf("%#v %#v\n", cfg, err)
}

we're using the New() constructor function in the config package and pass it the config type to return

pkg/config/config.go

package config

import (
    "errors"

    "github.com/alex-leonhardt/go-passing-enums/pkg/config/env"
    "github.com/alex-leonhardt/go-passing-enums/pkg/config/file"
)

// Configurer describes a config provider
type Configurer interface {
    Get(string) string
    Set(string) bool
    Del(string) bool
}

// T is used to select the type of config to return
type T uint

// Constants for T
const (
    Env T = iota
    File
    Unknown
)

// New takes a config.T type and returns an Configurer
func New(t T) (Configurer, error) {
    switch t {
    case Env:
        return env.New(), nil
    case File:
        return file.New(), nil
    default:
        return nil, errors.New("eh?")
    }
}

we're specifying T as a uint which we'll use as our custom "enum" to pass to New(), depending on the "type", a different Configurer is being initialised (using its own New() constructor) and then returned

pkg/config/env/env.go

package env

type env struct {
    val1 string
    val2 int
    val3 bool
}

// New creates a new config
func New() *env {
    return &env{}
}

func (c *env) Get(v string) string {
    return ""
}
func (c *env) Set(v string) bool {
    return true
}
func (c *env) Del(v string) bool {
    return true
}

finally, New() returns a pointer to an initialised env struct which satisfies the config.Configurer interface by implementing the required methods Get, Set and Del.

Hope this helps anyone, it helped me already just by writing this!

If you have any comments, suggestions, better explanations, please do leave a comment. Always happy to learn new things!

Writing a very basic kubernetes mutating admission webhook

Alex Leonhardt — Sat, 27 Jul 2019 17:11:51 +0000

My findings when attempting to write a (very) simple kubernetes mutating admission webhook.

Okay.. so, webhooks…?

Admission webhooks help you do some really cool stuff, there are two kinds of webhooks, validating and mutating. We will concentrate on the mutating admission webhook in this post.

Mutating admission webhooks allow you to “modify” a (e.g.) Pod (or any kubernetes resource) request. E.g. you can modify a Pod to use a particular scheduler, add / inject sidecar containers (think LinkerD sidecar), or even reject it if it doesn’t meet some security requirements, etc. etc. — all without having to write a full fledged “micro” service to do this. The webhook can live anywhere, in practice, k8s just needs to know where anywhere is.

Setup

The setup is easy, but important, all you really need to make sure is that the MutatingAdminssionController is enabled in the k8s api-server. To check if your k8s cluster has this enabled, you can use



kubectl api-versions | grep admissionregistration

For development, I can recommend using Kubernetes-In-Docker (KinD), all you need is Docker and KinD. KinD doesn’t auto-enable these, you can use this KinD configuration (kind.yaml)



---
kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
kubeadmConfigPatches:
- |
 apiVersion: kubeadm.k8s.io/v1beta2
 kind: ClusterConfiguration
 metadata:
 name: config
 apiServer:
 extraArgs:
 "enable-admission-plugins": "NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,PersistentVolumeClaimResize,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"
nodes:
- role: control-plane

Spin up the KinD cluster with



kind create cluster --conifg kind.yaml

I trust you’ll be able to configure kubectl etc. to now use this cluster going forward.

This is it, the setup is done. Well done! :)

Deployment

I’d like to start here, as this is the easiest part of this “tutorial” (if you will). Deploying an admission webhook is in practice the same as deploying any other service onto a k8s cluster.

All we need is

a Service
a Deployment
a MutatingWebhookConfiguration

The first two are simple, so I won’t go into those deeper, you can see what I’ve used on Github.

Before you just go ahead and deploy this to prod ...

Please do think about a couple _what-if_s before using webhooks in a production environment:

what if the request to your webhook fails (conn drop, conn reset, no dns, etc.)
what if the deployment failed and the pod is stuck in a crash-loop
what if the webhook has become mission critical and must be functional 100% of the time
what if you create a circular dependency (my favourites!)
what if you have a 3214 node cluster with many 10s of thousand resources & requests all depending on this webhook, which is scaled to 1 pod ;)

Note: Webhooks are only called over SSL/TLS so your webhook must have a valid signed certificate (we do this further down the line)

The MutatingWebhookConfiguration is where we tell k8s which resource requests should be sent to our webhook. The configuration consists of the following properties:

apiVersion (at the time it is: admissionregistration.k8s.io/v1beta1)
kind (must be: MutatingWebhookConfiguration)
metadata (the usual: name, annotations, labels)
webhooks (a list of type webhook)

The webhook (type) consists of these properties:

name
clientConfig
- caBundle (we will get this from the k8s cluster itself)
- service to send the AdmissionReview requests to
rules ( a list of rules that define which resource operations should be matched, these rules make sure that k8s resource requests are sent to your webhook )
namespaceSelector (the usual: matchLabels: {“label_name”: “label_value”}
…

there are many more that could be used, for a simple non-production webhook to play with, the above will suffice.

The full list of properties can be seen here.

A rule consists of the following:

operations (a list of [operations](https://godoc.org/k8s.io/api/admissionregistration/v1beta1#OperationType) to match, in our case ["CREATE"])
apiGroups (in our case, empty [""])
apiVersions (in our case, this is ["v1"])
resources (in our case, this is ["pods"])

apiGroups, apiVersions and resources are all (kind of) dependent on each other, in this example it’s quite easy as Pod is part of the core api group so it doesn’t need specifying, the empty [""] is matching the core api group.

Here is an example:



---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
 name: mutateme
 labels:
 app: mutateme
webhooks:
 - name: mutateme.default.svc.cluster.local
 clientConfig:
 caBundle: ${CA_BUNDLE}
 service:
 name: mutateme
 namespace: default
 path: "/mutate"
 rules:
 - operations: ["CREATE"]
 apiGroups: [""]
 apiVersions: ["v1"]
 resources: ["pods"]
 namespaceSelector:
 matchLabels:
 mutateme: enabled

the ${CA_BUNDLE} above refers to the actual CA bundle retrieved from the k8s API, replace it with your own; you can get your cluster’s CA bundle with



kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}'

As webhooks can only be called over HTTPS (SSL/TLS) you will need to generate a new ssl key & certificate for it. Doing this is somewhat involved, so probably best to take a look here:

and/or

https://github.com/morvencao/kube-mutating-webhook-tutorial/blob/master/deployment/webhook-create-signed-cert.sh

The webhook

Pasting the entire code here is probably not very useful, so I’ll concentrate on the essentials that helped me understand a few things.

Conceptually, what a webhook has to do is relatively easy, it receives a AdmissionReview, and responds with an AdmissionReview :) — most blog posts I’ve found concentrate on the Request and Response objects only, which can be a bit confusing.

In essence, this is what needs to happen ..

the K8S api server will send a AdmissionReview “request” and expects a AdmissionReview “response” ; this can get confusing, but i.e. it is something like this



{
 "kind": "AdmissionReview",
 "apiVersion": "admission.k8s.io/v1beta1",
 "request": {...}
}

the AdmissionReview consists of AdmissionRequest and AdmissionResponse objects
the webhook needs to “unmarshal” the AdmissionReview from JSON format into some kind of object so it can read the AdmissionRequest and modify the AdmissionResponse object within it
the webhook i.e. creates its own AdmissionResponse object, copies the UID from the AdmissionRequest object and replaces the AdmissionResponse object within the AdmissionReview with its own (overwrites it)
responds with a AdmissionReview object in JSON



{
 "kind": "AdmissionReview",
 "apiVersion": "admission.k8s.io/v1beta1",
 "request":  { ... ORIGINAL REQUEST ... },
 "response": { ... OUR RESPONSE ... }
}

Where to start?

We will start with a basic web server, that supports SSL/TLS, and can read and respond in JSON format.

In practice, you can use whatever programming language you’d like for this, I have used Go, but this can easily be done in Python or any other compiled or interpreted language. Ideally though, use a language that already has K8S libraries so you don’t have to create our own objects/types; Go (naturally) has these, but there are also at least the Python libraries you could use.

Here’s what I did to accept requests on port 8443 with SSL/TLS set up to use a key & cert.

The above example creates a server s with 2 handlers, one for / and one for /mutate which is the endpoint that will be called by k8s (which is what we specified in the MutatingWebhookConfiguration); it listens on port :8443 and we use the ListenAndServeTLS method to serve requests over SSL/TLS.

I split up the logic of http request/response from the actual processing of the AdmissionReview request as I find it’ll be easier to test the function/s independently; so the /mutate handler really only does

take the JSON http request and read in the BODY
send the BODY to the Mutate function in the mutate package (m) — the unmarshalling from JSON into the appropriate object structure, modification and marshalling back to JSON is done here
respond with a JSON message, either one that describes an Error or a AdmissionReview including our AdmissionResponse

So far so good, the potentially more challenging part is next.

pkg/mutate/mutate.go

The mutate package does the actual processing of AdmissionReview requests by …

unmarshalling the received JSON payload into a AdmissionReview object
using the AdmissionRequest object to decide what we should do
creating the JSONPatch
creating a new AdmissionResponse
updating the AdmissionReview with our new AdmissionResponse
marshal the final AdmissionReview back into JSON and return it

API Resources, Types, etc.

Something that I’m struggling the most with is finding the correct repository, version and even file/s that I need to do things with k8s or its supported resources.

Here’s what I’ve found out so far ..

Pod and other core resource types (which are used to unmarshal JSON into) can be found in kubernetes/api/blob/master/core/v1/types.go
Metadata objects (which usually are part of main object like Pod), seem to be kept in the apimachinery repo, it doesn’t have a description so 🤷🏻‍♂️apimachinery/blob/master/pkg/apis/meta/v1/types.go
Other resources and their definitions seem to reside in kubernetes/api/{extensions,node,autoscaling,..}; the admission types are located here: kubernetes/api/blob/master/admission/v1beta1/types.go (for now)

However, that’s not how they’re imported (at least not in Go); for our example, we need these imports:



import (
 v1beta1 "k8s.io/api/admission/v1beta1"
 corev1  "k8s.io/api/core/v1"
 metav1  "k8s.io/apimachinery/pkg/apis/meta/v1"
)

IIRC they must be k8s.io/ imports and not github.com/ imports, even though the repos are hosted on Github.

Marshal & Unmarshal

These terms always get me confused, but I’ll try to explain

marshal: when you convert a object into a JSON (or other, e.g. protobuf) equivalent representation
unmarshal: when you convert JSON (or other) into your own object’s representation

For the webhook we need to

unmarshal the JSON AdmissionReview into a Go AdmissionReview object, I called it ar
check if there’s a embedded raw object and if so, again, unmarshal it into the object type that we expect, in our case a Pod, which I called simply pod; if that fails, we should not try to continue and return an error
marshal the JSONPatch map into a valid JSONPatch, which, needs to be valid JSON
marshal the final (Go) AdmissionReview object back into a JSON AdmissionReview

this is why we needed to find those imports earlier, so we can create objects of the correct types and unmarshal JSON into them or marshal them into JSON.

The types we use/need are

AdmissionReview (v1beta1.AdmissionReview)
AdmissionRequest (v1beta1.AdmissionRequest)
Pod (corev1.Pod)
PatchTypeJSONPatch (v1beta1.PatchTypeJSONPatch)
AdmissionResponse (v1beta1.AdmissionResponse)
Status (metav1.Status) to set the AdmissionResponse.Result

Why do we create a JSONPatch?

This is because the mutating webhook does not modify the k8s resource, but responds with a patch, telling k8s how to modify the object for us. I found this rather counter intuitive, since we’re creating a "Mutating Admission Webhook" but don’t actually mutate anything.

More about JSONPatch expressions and how they work can be found here, but essentially they consist of an operation (op), a path and a value, e.g.:



{
 "op": "replace",
 "path": "/spec/containers/0/image",
 "value": "debian"
}

In our case, it instructs how and what operations it should apply to the resource that was requested. You can have more than one operation/instruction, e.g. a list of operations [{"op": ..},{"op": ..},{"op": ..}] and I believe they’ll be executed in the same order. Something for you to try and find out ;).

I used the following to create a JSONPatch list for the webhook, it replaces any container image to use the Debian docker image instead of the originally requested .. FWIW - it's a list as a Pod may have more than 1 container.



// resp is the AdmissionReview.Response 
p := []map[string]string{}

for i := range pod.Spec.Containers {
  patch := map[string]string{
    "op": "replace",
    "path": fmt.Sprintf("/spec/containers/%d/image", i), 
    "value": "debian",
  }
  p = append(p, patch)
}

// parse the []map into JSON
resp.Patch, err = json.Marshal(p)

This is it!

You can find the all the code on Github, I’d encourage you to try it without looking first :) … except for the details on the SSL ca, key, cert signing. I’ve also added links to resource that I believe will be helpful to understand what’s going on.

Kudos to the IBM-Cloud blog on Medium (link below) that I used as inspiration for this post and also as part reference when I got stuck. It’s more complete but also more complicated (at least at the time it seemed like it); if you're intending to create a proper version of this, you should check it out.

alex-leonhardt/k8s-mutate-webhook

Tip: Use a IDE / Editor that can help with code completion, dependencies, etc. — I used VSCode, but there are also good Vim plugins.

Resources

Kudos

https://medium.com/ibm-cloud/diving-into-kubernetes-mutatingadmissionwebhook-6ef3c5695f74

access and debug your kubernetes service / docker container

Alex Leonhardt — Sun, 26 Aug 2018 14:49:17 +0000

so you have followed the basic steps to get a service deployed into kubernetes, let’s say the service is Nginx, because why not …

$ kubectl create deployment ngx --image=nginx:latest

let’s scale that up to 2, because, really running just 1 pod is a bit too basic

$ kubectl scale deployment --replicas 2 ngx

you have 2 Nginx pods running on your k8s “cluster”, I use the built-in k8s installation that comes with Docker Edge

$ kubectl get pods

NAME READY STATUS RESTARTS AGE
ngx-5cb59c856c-cmn8p 1/1 Running 0 52m
ngx-5cb59c856c-s65nh 1/1 Running 0 52m

now that we have 2 pods, we need to somehow access the service so we can do some basic checking ourselves, e.g. to check that things work as expected, before we make this service publicly available via a LoadBalancer or Ingress — so we expose it as an internal service for now

$ kubectl expose deployment ngx --port 8080 --target-port 80

using

$ kubectl proxy &

you can then access your service (securely, it’s actually a encrypted tunnel) via

http://localhost:8001/api/v1/namespaces/default/services/ngx:8080/proxy/

details of how that works are here, but in a nutshell, you can access any internal service via

http://localhost:8001/api/v1/namespace/{namespace}/services/{service-name}:{port}/

so this is where you notice that one of the containers is behaving strangely, and sometimes seems to be unable to connect to a backend or just works intermittently, etc. (something’s wrong, basically) and so you decide, you’d like to run ngrep or tcpdump or strace to figure out what’s going on, but you also don’t want to modify the container image … so what can we do?

As long as you have access to the node running the container instance, you’re in luck — in this example we’ll just use the local Docker4Mac installation, but it works with any node running Docker containers.

Find out which node runs the troublesome container with

$ kubectl describe pod ngx-5cb59c856c-cmn8p

Name: ngx-5cb59c856c-cmn8p
Namespace: default
Node: docker-for-desktop/192.168.65.3

and login to that node; I initially tried editing the pod with

$ kubectl edit pod ngx-5cb59c856c-cmn8p

to add a debug container, but saving that config will fail as you cannot add/remove containers from a pod :(

As we’re on the node that runs the container however, we can create a custom debug container (locally) and run that inside the same pid and network namespace as the existing ngx-5cb59c856c-cmn8p.

The Dockerfile could be something like this (shamelessly copied/used from https://medium.com/@rothgar/how-to-debug-a-running-docker-container-from-a-separate-container-983f11740dc6)

FROM alpine
RUN apk update && apk add strace
CMD ["strace", "-p", "1"]

run

$ docker build -t strace .

and once the container is built (in reality, you’d build a consistent debug container and make that available for you to pull anytime from GCR or ECR or wherever) you run it with --privileged (yes, we don’t care about security in this example, see Justin Garrison’s post on how to do this more restrictive, so you’re able to do all the things and not have to fight permissions).

To attach to the pid and net namespace, you need the container Id or name, that’s easy to find with

$ docker ps | grep nginx

6b6e65ebc7c8 nginx "nginx -g 'daemon of…" About an hour ago Up About an hour k8s\_nginx\_ngx-5cb59c856c-cmn8p\_default\_402e0d53-a933-11e8-93cb-025000000001\_0
e245d91ba045 nginx "nginx -g 'daemon of…" About an hour ago Up About an hour k8s\_nginx\_ngx-5cb59c856c-s65nh\_default\_36a3a1e7-a933-11e8-93cb-025000000001\_0

So we’ll use the first one which is 6b6e65ebc7c8 (you should obviously use the one that's causing trouble) …

docker run -ti --pid=container:6b6e65ebc7c8 --net=container:6b6e65ebc7c8 --privileged strace /bin/ash

Once executed, you need the PID that is actually doing the work, PID 1 is actually the parent Nginx process, but that is not processing any requests, it’s just managing the child processes

/ # ps -ef
PID USER TIME COMMAND
 1 root 0:00 nginx: master process nginx -g daemon off;
 6 101 0:00 nginx: worker process
 44 root 0:00 /bin/ash
 50 root 0:00 ps -ef
/ #

okay, so let’s do an strace on PID 6 as that is actually doing work …

/ # strace -fp 6
strace: Process 6 attached
gettimeofday({tv\_sec=1535294355, tv\_usec=751153}, NULL) = 0
epoll\_wait(8, [{EPOLLIN, {u32=2097902081, u64=139627189727745}}], 512, 61010) = 1
gettimeofday({tv\_sec=1535294359, tv\_usec=908313}, NULL) = 0
recvfrom(3, "GET / HTTP/1.1\r\nHost: localhost:"..., 1024, 0, NULL, NULL) = 213
stat("/usr/share/nginx/html/index.html", {st\_mode=S\_IFREG|0644, st\_size=612, ...}) = 0
open("/usr/share/nginx/html/index.html", O\_RDONLY|O\_NONBLOCK) = 11
fstat(11, {st\_mode=S\_IFREG|0644, st\_size=612, ...}) = 0
writev(3, [{iov\_base="HTTP/1.1 200 OK\r\nServer: nginx/1"..., iov\_len=238}], 1) = 238
sendfile(3, 11, [0] =\> [612], 612) = 612
write(5, "10.1.0.1 - - [26/Aug/2018:14:39:"..., 111) = 111
close(11) = 0
epoll\_wait(8, [{EPOLLIN, {u32=2097902081, u64=139627189727745}}], 512, 65000) = 1
gettimeofday({tv\_sec=1535294361, tv\_usec=971440}, NULL) = 0
recvfrom(3, "GET / HTTP/1.1\r\nHost: localhost:"..., 1024, 0, NULL, NULL) = 213
stat("/usr/share/nginx/html/index.html", {st\_mode=S\_IFREG|0644, st\_size=612, ...}) = 0
open("/usr/share/nginx/html/index.html", O\_RDONLY|O\_NONBLOCK) = 11
fstat(11, {st\_mode=S\_IFREG|0644, st\_size=612, ...}) = 0
writev(3, [{iov\_base="HTTP/1.1 200 OK\r\nServer: nginx/1"..., iov\_len=238}], 1) = 238
sendfile(3, 11, [0] =\> [612], 612) = 612
write(5, "10.1.0.1 - - [26/Aug/2018:14:39:"..., 111) = 111
close(11) = 0
epoll\_wait(8,
^Cstrace: Process 6 detached
 \<detached ...\>

And there it is, 2 requests to this particular Nginx instance without having to

redeploy the Pod with an additional debug container (you could argue that this would be better, but you may not be able to re-produce the issue straight away and may need to run it for a long time which costs resources)
modify the Dockerfile in any way (install debug tools)
change privileges on the running container, it can keep running in its more secure environment vs the debug container which has additional capabilities

the nice thing about this pattern is that you can create yourself a debug container that you can re-use to debug applications running on any node that runs Docker (ECS, on-prem K8S, EKS, AKS, GKE).

Happy debugging!

Credits & Resources:

Writing my own terraform template (data) provider

Alex Leonhardt — Sat, 05 May 2018 22:33:33 +0000

As a follow up to my previous post about how to use terrafrom’s built-in looping mechanisms, I set out to write my own custom “data” provider, in this case a new template renderer that would allow me to use Golang’s text/template instead of terraform’s own.

TL;DR — the code is here..

alex-leonhardt/terraform-provider-gotemplate

I started by trying to follow Hashicorp’s Writing Custom Providers documentation, however that does not seem to explain how to write a data provider , and although I had a basic Hello World working, I didn’t get too close to what I actually wanted to do.

So, since there is an existing template provider, I chose to use that as a kind of starting point and hack my way around all the bits I didn’t want/need. Which probably means I didn’t follow “best practice”. What I wanted was simple, I pass through a json encoded variable to the go-template provider and it will read a template file, render it (or fail), and I should be able to access the result, using the .rendered method, just like the original template provider does.

To use it, it’d roughly look like this …

variable "data" {
 default = {
 "msg" = "Hello World"
 "msg2" = [1, 2, 3, 4]
 }
}

data "gotemplate" "gotmpl" {
template = "${path.module}/file.tmpl"
data = "${jsonencode(var.data)}"
}

output "tmpl" {
value = "${data.gotemplate.gotmpl.rendered}"
}

As you can see, we have a map variable "data" with a mix of values of string type keys, other types are not allowed it seems — see https://stackoverflow.com/a/8758771

I decided, 2 variables should be sufficient, the path to the template file (template) and the json encoded data (data) itself. I probably won’t go into details of how a provider should be structured as TF’s template provider seems a bit different from the “writing custom providers” instructions — probably as it’s a bit older and possibly one of the first. So, since I followed that example (more or less), my new provider also doesn’t follow it, at least not closely.

To use my new provider, all that’s needed is to compile it into a file called terraform-provider-gotemplate and I can use the gotemplate provider, this is specified in the main.go file here

// Provider is a TF provider
func Provider() *schema.Provider {
 return &schema.Provider{
 DataSourcesMap: map[string]*schema.Resource{
 "gotemplate": dataSourceFile(),
 },
 }
}

the important part really is in dataSourceFile() which specifies what variables can be accessed using the provider, in my case that was

template (the path to the go template file)
data (the json encoded data)
rendered (the finished render)

other than setting the schema, I also had to make it actually do something, this is done with

func dataSourceFile() *schema.Resource {
 return &schema.Resource{
 Read: dataSourceFileRead,

which passes dataSourceFileRead function to the schema.Resource‘s Read field (or property? I like to think of it as a property) which must match this function signature

type ReadFunc func(*ResourceData, interface{}) error

which is done with

func dataSourceFileRead(d *schema.ResourceData, meta interface{}) error {
 rendered, err := renderFile(d)
 if err != nil {
 return err
 }
 d.Set("rendered", rendered)
 d.SetId(hash(rendered))
 return nil
}

d.Set is important to set the rendered variable with the finished rendered template, so it can be used with output or other providers as a input.

Before setting rendered we retrieved the rendered output by calling the renderFile(d) function, which looks like this

The final result is available on Github, use it, make PRs if you like, I have learned quite a bit doing this, although still a bit confused and probably not following the standard way of writing providers, but it was fun getting this to work.

alex-leonhardt/terraform-provider-gotemplate

References:

Terraform templating and loops

Alex Leonhardt — Wed, 28 Mar 2018 06:38:34 +0000

UPDATE: as of terraform 0.12, tf now supports a new template syntax

You probably already know it’s not easily possible to loop through a variable of type slice/list within templates, in fact, there are no loop constructs within TF templates at all at this point in time.

There is hope though, according to their comment on this GH issue: https://github.com/hashicorp/terraform/issues/16628

So, say we want to generate a JSON file from a slice/array that, once finished, looks something like this :

We cannot simply iterate over an array to change the value of endpointX as we cannot loop through it.

What we’d expect to be able to do, in the template, is something like this

we’d have to handle that last , but let’s ignore that for now — the idea is simple, iterate over values in the array of map endpoints and use each key and value as variables within that loop to generate the json file. As we already know, this isn’t possible, and we won’t get this anytime soon.

Luckily, necessity is the mother of invention, and I got this to work with the existing loop constructs outside the template; although it’s not as “clean” as above, it’s usable and produces the desired result.

Using TFs buit-in loop construct

To make use of TFs built-in loop construct and generate the wanted json, we will need 2 template files, the first will look like this

// data.json.tmpl

{
 "name": "${name}",
 "endpoint": "${endpoint}"
}

and the second template looks like this

// service.json.tmpl

[
 ${value},
 {"links": ${links}}
]

Maybe you already know where we’re heading with this, we now render the first template using the built-in loop constructs from TF using the template_file data resource

variable "endpoints" {
_type_ = "list"
_default_ = [
 { endpoint1 = "https://endpoint-1.example.com" },
 { endpoint2 = "https://endpoint-2.example.com" },
 { endpoint3 = "https://endpoint-3.example.com" },
 ]
}

data "template\_file" "data\_json" {
_template_ = "${file("${path.module}/data.json.tmpl")}"
_count_ = "${length(var.endpoints)}"
 vars {
_endpoint_ = "${element(values(var.endpoints[count.index]), 0)}"
_name_ = "${element(keys(var.endpoints[count.index]), 0)}"
 }
}

and then re-use the .rendered output as input to the 2nd template, which really just generates the list by pre- and post- fixing the $value variable with [] .

variable "links" {
_type_ = "list"
_default_ = [
 "link1",
 "link2",
 "link3",
 ]
}

data "template\_file" "service\_json" {
_template_ = "${file("${path.module}/service.json.tmpl")}"
vars {
_value_ = "${join(",", data.template\_file.data\_json.\*.rendered)}"
_links_ = "${jsonencode(var.links)}"
}}

and finally we’re using the output module to show us what we’ve done, this is also how you’d use the result as input to any other TF module that accepts json.

output "json" {
_value_ = "${data.template\_file.service\_json.rendered}"
}

and the output

$ tf apply
data.template\_file.data\_json[2]: Refreshing state...
data.template\_file.data\_json[0]: Refreshing state...
data.template\_file.data\_json[1]: Refreshing state...
data.template\_file.service\_json: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

json = [
 {
 "name": "endpoint1",
 "endpoint": "[https://endpoint-1.example.com](https://endpoint-1.example.com)"
},{
 "name": "endpoint2",
 "endpoint": "[https://endpoint-2.example.com](https://endpoint-2.example.com)"
},{
 "name": "endpoint3",
 "endpoint": "[https://endpoint-3.example.com](https://endpoint-3.example.com)"
},
 {"links": ["link1","link2","link3"]}
]

Converting lists/slices to JSON

You might have noticed the jsonencode in the last part, this helps to convert any structure to valid json and it becomes valid input to the template as json in essence is just text

variable "links" {
_type_ = "list"
_default_ = [
 "link1",
 "link2",
 "link3",
 ]
}

data "template\_file" "service\_json" {
_[...]_
_links_ = "${jsonencode(var.links)}"
}}

References

what happens when … (Part 1)

Alex Leonhardt — Mon, 16 Oct 2017 05:33:24 +0000

This is for the curious amongst us, who want to know their limits on system internals and/or (inter)networking knowledge and constantly strive to surpass it.

What happens when I type [www.google.com](http://www.google.com) into the browser address bar and hit enter?

You may have come across the above question already (several times), which is probably why you’re here, reading this! It’s a very broad and very difficult question to answer as there are so many moving parts to this question.

We eventually will get to answer the above question, however, let’s not start off with that quite yet, as it would be far too much to go into it right from the start.

My motivation here is not to provide answers, but to show how I’d approach a question / problem like this, and in the process maybe learn a thing or two (or many), but also help you understand a few things better / or in more detail.

So let’s start small and “simple” ..

What happens when I type ls -l into a shell prompt and hit enter?

It’s probably a good idea to stop reading here for a minute and actually think about it and extrapolate what would/could be happening, based on your experience and assumptions, in as much detail as you can..

… just a little longer …

Rather than going ahead and just listing all the things that are happening, I’d like to record how I approach this question (or problem); don’t take this as a step-by-step guide or limit yourself how to approach it, this is how I would do this.

So let’s get started ..

For the record, I’m using a Virtualbox VM running Debian Jessie (8.x) with latest updates applied as of 10/10/2017.

I guess the most obvious first thing to do is to just “execute” the command

$ ls -l

what does it return ?

A directory listing, in the long format, containing various details about every directory and file in it..

root@debian:/var/cache# ls -l
total 28
drwxr-xr-x 3 root root 4096 Oct 10 22:12 apt
drwxr-xr-x 2 root root 4096 Oct 10 22:04 debconf
drwxr-xr-x 2 root root 4096 Jul 3 2015 dictionaries-common
drwxr-xr-x 2 root root 4096 Oct 10 22:03 fontconfig
drwx------ 2 root root 4096 Oct 10 22:04 ldconfig
drwxr-sr-x 32 man root 4096 Oct 10 22:12 man
drwxr-xr-x 2 root root 4096 Apr 23 2014 modass

Based on the above output, we’re in the directory /var/cache, we see the directory/file permissions, the owner, the group, the size (default seems to be set to 4k), last modification date/time and finally the name itself.

That’s quite a bit of information there already, we can make some guesses of what happened, but they’re not going to be very detailed. Some system call that got the current directory, some others got the modification time for every file/directory, etc. So what do we do next / where do we look?

Experience tells me that, if I wanted to see what an application is actually “doing”, I use a tool called strace — running anything with strace requires you to have root permissions (sudo should do). I can’t really remember how I came across it, probably some many many moons ago, some more senior person told me about it I guess.

Anyway, what strace does is trace these system calls that your application is doing (in this case the /bin/ls program) and print them to stdout.

This is how a strace of ls -l actually looks like

root@debian:/var/cache# strace ls -l
execve("/bin/ls", ["ls", "-l"], [/\* 20 vars \*/]) = 0
brk(0) = 0x785000
access("/etc/ld.so.nohwcap", F\_OK) = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_ANONYMOUS, -1, 0) = 0x7f420b3b2000
access("/etc/ld.so.preload", R\_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O\_RDONLY|O\_CLOEXEC) = 3
fstat(3, {st\_mode=S\_IFREG|0644, st\_size=23259, ...}) = 0
mmap(NULL, 23259, PROT\_READ, MAP\_PRIVATE, 3, 0) = 0x7f420b3ac000
close(3) = 0
access("/etc/ld.so.nohwcap", F\_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86\_64-linux-gnu/libselinux.so.1", O\_RDONLY|O\_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\>\0\1\0\0\0\20c\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st\_mode=S\_IFREG|0644, st\_size=142728, ...}) = 0
mmap(NULL, 2246896, PROT\_READ|PROT\_EXEC, MAP\_PRIVATE|MAP\_DENYWRITE, 3, 0) = 0x7f420af6f000
mprotect(0x7f420af90000, 2097152, PROT\_NONE) = 0
mmap(0x7f420b190000, 8192, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_FIXED|MAP\_DENYWRITE, 3, 0x21000) = 0x7f420b190000
mmap(0x7f420b192000, 6384, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_FIXED|MAP\_ANONYMOUS, -1, 0) = 0x7f420b192000
close(3) = 0
access("/etc/ld.so.nohwcap", F\_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86\_64-linux-gnu/libacl.so.1", O\_RDONLY|O\_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\>\0\1\0\0\0\200\37\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st\_mode=S\_IFREG|0644, st\_size=35288, ...}) = 0
mmap(NULL, 2130592, PROT\_READ|PROT\_EXEC, MAP\_PRIVATE|MAP\_DENYWRITE, 3, 0) = 0x7f420ad66000
mprotect(0x7f420ad6e000, 2093056, PROT\_NONE) = 0
mmap(0x7f420af6d000, 8192, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_FIXED|MAP\_DENYWRITE, 3, 0x7000) = 0x7f420af6d000
close(3) = 0
access("/etc/ld.so.nohwcap", F\_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86\_64-linux-gnu/libc.so.6", O\_RDONLY|O\_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0\>\0\1\0\0\0P\34\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st\_mode=S\_IFREG|0755, st\_size=1738176, ...}) = 0
mmap(NULL, 4096, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_ANONYMOUS, -1, 0) = 0x7f420b3ab000
mmap(NULL, 3844640, PROT\_READ|PROT\_EXEC, MAP\_PRIVATE|MAP\_DENYWRITE, 3, 0) = 0x7f420a9bb000
mprotect(0x7f420ab5c000, 2097152, PROT\_NONE) = 0
mmap(0x7f420ad5c000, 24576, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_FIXED|MAP\_DENYWRITE, 3, 0x1a1000) = 0x7f420ad5c000
mmap(0x7f420ad62000, 14880, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_FIXED|MAP\_ANONYMOUS, -1, 0) = 0x7f420ad62000
close(3) = 0
access("/etc/ld.so.nohwcap", F\_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86\_64-linux-gnu/libpcre.so.3", O\_RDONLY|O\_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\>\0\1\0\0\0\20\27\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st\_mode=S\_IFREG|0644, st\_size=448440, ...}) = 0
mmap(NULL, 2543976, PROT\_READ|PROT\_EXEC, MAP\_PRIVATE|MAP\_DENYWRITE, 3, 0) = 0x7f420a74d000
mprotect(0x7f420a7b9000, 2097152, PROT\_NONE) = 0
mmap(0x7f420a9b9000, 8192, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_FIXED|MAP\_DENYWRITE, 3, 0x6c000) = 0x7f420a9b9000
close(3) = 0
access("/etc/ld.so.nohwcap", F\_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86\_64-linux-gnu/libdl.so.2", O\_RDONLY|O\_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\>\0\1\0\0\0\320\16\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st\_mode=S\_IFREG|0644, st\_size=14664, ...}) = 0
mmap(NULL, 2109712, PROT\_READ|PROT\_EXEC, MAP\_PRIVATE|MAP\_DENYWRITE, 3, 0) = 0x7f420a549000
mprotect(0x7f420a54c000, 2093056, PROT\_NONE) = 0
mmap(0x7f420a74b000, 8192, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_FIXED|MAP\_DENYWRITE, 3, 0x2000) = 0x7f420a74b000
close(3) = 0
access("/etc/ld.so.nohwcap", F\_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86\_64-linux-gnu/libattr.so.1", O\_RDONLY|O\_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\>\0\1\0\0\0\320\23\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st\_mode=S\_IFREG|0644, st\_size=18640, ...}) = 0
mmap(NULL, 4096, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_ANONYMOUS, -1, 0) = 0x7f420b3aa000
mmap(NULL, 2113912, PROT\_READ|PROT\_EXEC, MAP\_PRIVATE|MAP\_DENYWRITE, 3, 0) = 0x7f420a344000
mprotect(0x7f420a348000, 2093056, PROT\_NONE) = 0
mmap(0x7f420a547000, 8192, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_FIXED|MAP\_DENYWRITE, 3, 0x3000) = 0x7f420a547000
close(3) = 0
access("/etc/ld.so.nohwcap", F\_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86\_64-linux-gnu/libpthread.so.0", O\_RDONLY|O\_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\>\0\1\0\0\0\320n\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st\_mode=S\_IFREG|0755, st\_size=137384, ...}) = 0
mmap(NULL, 2213008, PROT\_READ|PROT\_EXEC, MAP\_PRIVATE|MAP\_DENYWRITE, 3, 0) = 0x7f420a127000
mprotect(0x7f420a13f000, 2093056, PROT\_NONE) = 0
mmap(0x7f420a33e000, 8192, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_FIXED|MAP\_DENYWRITE, 3, 0x17000) = 0x7f420a33e000
mmap(0x7f420a340000, 13456, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_FIXED|MAP\_ANONYMOUS, -1, 0) = 0x7f420a340000
close(3) = 0
mmap(NULL, 4096, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_ANONYMOUS, -1, 0) = 0x7f420b3a9000
mmap(NULL, 8192, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_ANONYMOUS, -1, 0) = 0x7f420b3a7000
arch\_prctl(ARCH\_SET\_FS, 0x7f420b3a7800) = 0
mprotect(0x7f420ad5c000, 16384, PROT\_READ) = 0
mprotect(0x7f420a33e000, 4096, PROT\_READ) = 0
mprotect(0x7f420a547000, 4096, PROT\_READ) = 0
mprotect(0x7f420a74b000, 4096, PROT\_READ) = 0
mprotect(0x7f420a9b9000, 4096, PROT\_READ) = 0
mprotect(0x7f420af6d000, 4096, PROT\_READ) = 0
mprotect(0x7f420b190000, 4096, PROT\_READ) = 0
mprotect(0x61b000, 4096, PROT\_READ) = 0
mprotect(0x7f420b3b4000, 4096, PROT\_READ) = 0
munmap(0x7f420b3ac000, 23259) = 0
set\_tid\_address(0x7f420b3a7ad0) = 26019
set\_robust\_list(0x7f420b3a7ae0, 24) = 0
rt\_sigaction(SIGRTMIN, {0x7f420a12d9b0, [], SA\_RESTORER|SA\_SIGINFO, 0x7f420a136890}, NULL, 8) = 0
rt\_sigaction(SIGRT\_1, {0x7f420a12da40, [], SA\_RESTORER|SA\_RESTART|SA\_SIGINFO, 0x7f420a136890}, NULL, 8) = 0
rt\_sigprocmask(SIG\_UNBLOCK, [RTMIN RT\_1], NULL, 8) = 0
getrlimit(RLIMIT\_STACK, {rlim\_cur=8192\*1024, rlim\_max=RLIM64\_INFINITY}) = 0
statfs("/sys/fs/selinux", 0x7ffd512871e0) = -1 ENOENT (No such file or directory)
statfs("/selinux", 0x7ffd512871e0) = -1 ENOENT (No such file or directory)
brk(0) = 0x785000
brk(0x7a6000) = 0x7a6000
open("/proc/filesystems", O\_RDONLY) = 3
fstat(3, {st\_mode=S\_IFREG|0444, st\_size=0, ...}) = 0
mmap(NULL, 4096, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_ANONYMOUS, -1, 0) = 0x7f420b3b1000
read(3, "nodev\tsysfs\nnodev\trootfs\nnodev\tr"..., 1024) = 317
read(3, "", 1024) = 0
close(3) = 0
munmap(0x7f420b3b1000, 4096) = 0
open("/usr/lib/locale/locale-archive", O\_RDONLY|O\_CLOEXEC) = 3
fstat(3, {st\_mode=S\_IFREG|0644, st\_size=1607712, ...}) = 0
mmap(NULL, 1607712, PROT\_READ, MAP\_PRIVATE, 3, 0) = 0x7f420b21e000
close(3) = 0
open("/usr/share/locale/locale.alias", O\_RDONLY|O\_CLOEXEC) = 3
fstat(3, {st\_mode=S\_IFREG|0644, st\_size=2492, ...}) = 0
mmap(NULL, 4096, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_ANONYMOUS, -1, 0) = 0x7f420b3b1000
read(3, "# Locale name alias data base.\n#"..., 4096) = 2492
read(3, "", 4096) = 0
close(3) = 0
munmap(0x7f420b3b1000, 4096) = 0
open("/usr/lib/locale/en\_GB.UTF-8/LC\_CTYPE", O\_RDONLY|O\_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en\_GB.utf8/LC\_CTYPE", O\_RDONLY|O\_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en\_GB/LC\_CTYPE", O\_RDONLY|O\_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en.UTF-8/LC\_CTYPE", O\_RDONLY|O\_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en.utf8/LC\_CTYPE", O\_RDONLY|O\_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en/LC\_CTYPE", O\_RDONLY|O\_CLOEXEC) = -1 ENOENT (No such file or directory)
ioctl(1, SNDCTL\_TMR\_TIMEBASE or SNDRV\_TIMER\_IOCTL\_NEXT\_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(1, TIOCGWINSZ, {ws\_row=57, ws\_col=238, ws\_xpixel=0, ws\_ypixel=0}) = 0
openat(AT\_FDCWD, ".", O\_RDONLY|O\_NONBLOCK|O\_DIRECTORY|O\_CLOEXEC) = 3
getdents(3, /\* 9 entries \*/, 32768) = 264
lstat("debconf", {st\_mode=S\_IFDIR|0755, st\_size=4096, ...}) = 0
lgetxattr("debconf", "security.selinux", 0x785eb0, 255) = -1 ENODATA (No data available)
getxattr("debconf", "system.posix\_acl\_access", 0x0, 0) = -1 ENODATA (No data available)
getxattr("debconf", "system.posix\_acl\_default", 0x0, 0) = -1 ENODATA (No data available)
socket(PF\_LOCAL, SOCK\_STREAM|SOCK\_CLOEXEC|SOCK\_NONBLOCK, 0) = 4
connect(4, {sa\_family=AF\_LOCAL, sun\_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(4) = 0
socket(PF\_LOCAL, SOCK\_STREAM|SOCK\_CLOEXEC|SOCK\_NONBLOCK, 0) = 4
connect(4, {sa\_family=AF\_LOCAL, sun\_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(4) = 0
open("/etc/nsswitch.conf", O\_RDONLY|O\_CLOEXEC) = 4
fstat(4, {st\_mode=S\_IFREG|0644, st\_size=497, ...}) = 0
mmap(NULL, 4096, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_ANONYMOUS, -1, 0) = 0x7f420b3b1000
read(4, "# /etc/nsswitch.conf\n#\n# Example"..., 4096) = 497
read(4, "", 4096) = 0
close(4) = 0
munmap(0x7f420b3b1000, 4096) = 0
open("/etc/ld.so.cache", O\_RDONLY|O\_CLOEXEC) = 4
fstat(4, {st\_mode=S\_IFREG|0644, st\_size=23259, ...}) = 0
mmap(NULL, 23259, PROT\_READ, MAP\_PRIVATE, 4, 0) = 0x7f420b3ac000
close(4) = 0
access("/etc/ld.so.nohwcap", F\_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86\_64-linux-gnu/libnss\_compat.so.2", O\_RDONLY|O\_CLOEXEC) = 4
read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\>\0\1\0\0\0\260\23\0\0\0\0\0\0"..., 832) = 832
fstat(4, {st\_mode=S\_IFREG|0644, st\_size=31632, ...}) = 0
mmap(NULL, 2126976, PROT\_READ|PROT\_EXEC, MAP\_PRIVATE|MAP\_DENYWRITE, 4, 0) = 0x7f4209f1f000
mprotect(0x7f4209f26000, 2093056, PROT\_NONE) = 0
mmap(0x7f420a125000, 8192, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_FIXED|MAP\_DENYWRITE, 4, 0x6000) = 0x7f420a125000
close(4) = 0
access("/etc/ld.so.nohwcap", F\_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86\_64-linux-gnu/libnsl.so.1", O\_RDONLY|O\_CLOEXEC) = 4
read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\>\0\1\0\0\0`A\0\0\0\0\0\0"..., 832) = 832
fstat(4, {st\_mode=S\_IFREG|0644, st\_size=89104, ...}) = 0
mmap(NULL, 2194072, PROT\_READ|PROT\_EXEC, MAP\_PRIVATE|MAP\_DENYWRITE, 4, 0) = 0x7f4209d07000
mprotect(0x7f4209d1c000, 2093056, PROT\_NONE) = 0
mmap(0x7f4209f1b000, 8192, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_FIXED|MAP\_DENYWRITE, 4, 0x14000) = 0x7f4209f1b000
mmap(0x7f4209f1d000, 6808, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_FIXED|MAP\_ANONYMOUS, -1, 0) = 0x7f4209f1d000
close(4) = 0
mprotect(0x7f4209f1b000, 4096, PROT\_READ) = 0
mprotect(0x7f420a125000, 4096, PROT\_READ) = 0
munmap(0x7f420b3ac000, 23259) = 0
open("/etc/ld.so.cache", O\_RDONLY|O\_CLOEXEC) = 4
fstat(4, {st\_mode=S\_IFREG|0644, st\_size=23259, ...}) = 0
mmap(NULL, 23259, PROT\_READ, MAP\_PRIVATE, 4, 0) = 0x7f420b3ac000
close(4) = 0
access("/etc/ld.so.nohwcap", F\_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86\_64-linux-gnu/libnss\_nis.so.2", O\_RDONLY|O\_CLOEXEC) = 4
read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\>\0\1\0\0\0\240!\0\0\0\0\0\0"..., 832) = 832
fstat(4, {st\_mode=S\_IFREG|0644, st\_size=43592, ...}) = 0
mmap(NULL, 2139560, PROT\_READ|PROT\_EXEC, MAP\_PRIVATE|MAP\_DENYWRITE, 4, 0) = 0x7f4209afc000
mprotect(0x7f4209b06000, 2093056, PROT\_NONE) = 0
mmap(0x7f4209d05000, 8192, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_FIXED|MAP\_DENYWRITE, 4, 0x9000) = 0x7f4209d05000
close(4) = 0
access("/etc/ld.so.nohwcap", F\_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86\_64-linux-gnu/libnss\_files.so.2", O\_RDONLY|O\_CLOEXEC) = 4
read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\>\0\1\0\0\0\240\"\0\0\0\0\0\0"..., 832) = 832
fstat(4, {st\_mode=S\_IFREG|0644, st\_size=47712, ...}) = 0
mmap(NULL, 2144392, PROT\_READ|PROT\_EXEC, MAP\_PRIVATE|MAP\_DENYWRITE, 4, 0) = 0x7f42098f0000
mprotect(0x7f42098fb000, 2093056, PROT\_NONE) = 0
mmap(0x7f4209afa000, 8192, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_FIXED|MAP\_DENYWRITE, 4, 0xa000) = 0x7f4209afa000
close(4) = 0
mprotect(0x7f4209afa000, 4096, PROT\_READ) = 0
mprotect(0x7f4209d05000, 4096, PROT\_READ) = 0
munmap(0x7f420b3ac000, 23259) = 0
open("/etc/passwd", O\_RDONLY|O\_CLOEXEC) = 4
lseek(4, 0, SEEK\_CUR) = 0
fstat(4, {st\_mode=S\_IFREG|0644, st\_size=1488, ...}) = 0
mmap(NULL, 1488, PROT\_READ, MAP\_SHARED, 4, 0) = 0x7f420b3b1000
lseek(4, 1488, SEEK\_SET) = 1488
munmap(0x7f420b3b1000, 1488) = 0
close(4) = 0
socket(PF\_LOCAL, SOCK\_STREAM|SOCK\_CLOEXEC|SOCK\_NONBLOCK, 0) = 4
connect(4, {sa\_family=AF\_LOCAL, sun\_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(4) = 0
socket(PF\_LOCAL, SOCK\_STREAM|SOCK\_CLOEXEC|SOCK\_NONBLOCK, 0) = 4
connect(4, {sa\_family=AF\_LOCAL, sun\_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(4) = 0
open("/etc/group", O\_RDONLY|O\_CLOEXEC) = 4
lseek(4, 0, SEEK\_CUR) = 0
fstat(4, {st\_mode=S\_IFREG|0644, st\_size=748, ...}) = 0
mmap(NULL, 748, PROT\_READ, MAP\_SHARED, 4, 0) = 0x7f420b3b1000
lseek(4, 748, SEEK\_SET) = 748
munmap(0x7f420b3b1000, 748) = 0
close(4) = 0
lstat("fontconfig", {st\_mode=S\_IFDIR|0755, st\_size=4096, ...}) = 0
lgetxattr("fontconfig", "security.selinux", 0x7968e0, 255) = -1 ENODATA (No data available)
getxattr("fontconfig", "system.posix\_acl\_access", 0x0, 0) = -1 ENODATA (No data available)
getxattr("fontconfig", "system.posix\_acl\_default", 0x0, 0) = -1 ENODATA (No data available)
lstat("dictionaries-common", {st\_mode=S\_IFDIR|0755, st\_size=4096, ...}) = 0
lgetxattr("dictionaries-common", "security.selinux", 0x796900, 255) = -1 ENODATA (No data available)
getxattr("dictionaries-common", "system.posix\_acl\_access", 0x0, 0) = -1 ENODATA (No data available)
getxattr("dictionaries-common", "system.posix\_acl\_default", 0x0, 0) = -1 ENODATA (No data available)
lstat("man", {st\_mode=S\_IFDIR|S\_ISGID|0755, st\_size=4096, ...}) = 0
lgetxattr("man", "security.selinux", 0x796920, 255) = -1 ENODATA (No data available)
getxattr("man", "system.posix\_acl\_access", 0x0, 0) = -1 ENODATA (No data available)
getxattr("man", "system.posix\_acl\_default", 0x0, 0) = -1 ENODATA (No data available)
open("/etc/passwd", O\_RDONLY|O\_CLOEXEC) = 4
lseek(4, 0, SEEK\_CUR) = 0
fstat(4, {st\_mode=S\_IFREG|0644, st\_size=1488, ...}) = 0
mmap(NULL, 1488, PROT\_READ, MAP\_SHARED, 4, 0) = 0x7f420b3b1000
lseek(4, 1488, SEEK\_SET) = 1488
munmap(0x7f420b3b1000, 1488) = 0
close(4) = 0
lstat("modass", {st\_mode=S\_IFDIR|0755, st\_size=4096, ...}) = 0
lgetxattr("modass", "security.selinux", 0x796960, 255) = -1 ENODATA (No data available)
getxattr("modass", "system.posix\_acl\_access", 0x0, 0) = -1 ENODATA (No data available)
getxattr("modass", "system.posix\_acl\_default", 0x0, 0) = -1 ENODATA (No data available)
lstat("ldconfig", {st\_mode=S\_IFDIR|0700, st\_size=4096, ...}) = 0
lgetxattr("ldconfig", "security.selinux", 0x796980, 255) = -1 ENODATA (No data available)
getxattr("ldconfig", "system.posix\_acl\_access", 0x0, 0) = -1 ENODATA (No data available)
getxattr("ldconfig", "system.posix\_acl\_default", 0x0, 0) = -1 ENODATA (No data available)
lstat("apt", {st\_mode=S\_IFDIR|0755, st\_size=4096, ...}) = 0
lgetxattr("apt", "security.selinux", 0x7969a0, 255) = -1 ENODATA (No data available)
getxattr("apt", "system.posix\_acl\_access", 0x0, 0) = -1 ENODATA (No data available)
getxattr("apt", "system.posix\_acl\_default", 0x0, 0) = -1 ENODATA (No data available)
getdents(3, /\* 0 entries \*/, 32768) = 0
close(3) = 0
fstat(1, {st\_mode=S\_IFCHR|0600, st\_rdev=makedev(136, 0), ...}) = 0
mmap(NULL, 4096, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_ANONYMOUS, -1, 0) = 0x7f420b3b1000
write(1, "total 28\n", 9total 28
) = 9
open("/etc/localtime", O\_RDONLY|O\_CLOEXEC) = 3
fstat(3, {st\_mode=S\_IFREG|0644, st\_size=1892, ...}) = 0
fstat(3, {st\_mode=S\_IFREG|0644, st\_size=1892, ...}) = 0
mmap(NULL, 4096, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_ANONYMOUS, -1, 0) = 0x7f420b3b0000
read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0\0"..., 4096) = 1892
lseek(3, -1217, SEEK\_CUR) = 675
read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\6\0\0\0\6\0\0\0\0"..., 4096) = 1217
close(3) = 0
munmap(0x7f420b3b0000, 4096) = 0
write(1, "drwxr-xr-x 3 root root 4096 Oct"..., 46drwxr-xr-x 3 root root 4096 Oct 10 22:12 apt
) = 46
stat("/etc/localtime", {st\_mode=S\_IFREG|0644, st\_size=1892, ...}) = 0
write(1, "drwxr-xr-x 2 root root 4096 Oct"..., 50drwxr-xr-x 2 root root 4096 Oct 10 22:23 debconf
) = 50
stat("/etc/localtime", {st\_mode=S\_IFREG|0644, st\_size=1892, ...}) = 0
write(1, "drwxr-xr-x 2 root root 4096 Jul"..., 62drwxr-xr-x 2 root root 4096 Jul 3 2015 dictionaries-common
) = 62
stat("/etc/localtime", {st\_mode=S\_IFREG|0644, st\_size=1892, ...}) = 0
write(1, "drwxr-xr-x 2 root root 4096 Oct"..., 53drwxr-xr-x 2 root root 4096 Oct 10 22:03 fontconfig
) = 53
stat("/etc/localtime", {st\_mode=S\_IFREG|0644, st\_size=1892, ...}) = 0
write(1, "drwx------ 2 root root 4096 Oct"..., 51drwx------ 2 root root 4096 Oct 10 22:04 ldconfig
) = 51
stat("/etc/localtime", {st\_mode=S\_IFREG|0644, st\_size=1892, ...}) = 0
write(1, "drwxr-sr-x 32 man root 4096 Oct"..., 46drwxr-sr-x 32 man root 4096 Oct 10 22:12 man
) = 46
stat("/etc/localtime", {st\_mode=S\_IFREG|0644, st\_size=1892, ...}) = 0
write(1, "drwxr-xr-x 2 root root 4096 Apr"..., 49drwxr-xr-x 2 root root 4096 Apr 23 2014 modass
) = 49
close(1) = 0
munmap(0x7f420b3b1000, 4096) = 0
close(2) = 0
exit\_group(0) = ?
+++ exited with 0 +++

That’s not very readable, maybe just try it yourself, on a Vagrant box or some other Linux install; make sure to install the strace package and just go to /var/cache and execute strace ls -l .

Okay, that’s a lot of stuff that’s going on there — let’s start at the top and skip over a few bits whilst working our way down

root@debian:/var/cache# strace ls -l
execve("/bin/ls", ["ls", "-l"], [/\* 20 vars \*/]) = 0
brk(0) = 0x785000
access("/etc/ld.so.nohwcap", F\_OK) = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_ANONYMOUS, -1, 0) = 0x7f420b3b2000
access("/etc/ld.so.preload", R\_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O\_RDONLY|O\_CLOEXEC) = 3
fstat(3, {st\_mode=S\_IFREG|0644, st\_size=23259, ...}) = 0
mmap(NULL, 23259, PROT\_READ, MAP\_PRIVATE, 3, 0) = 0x7f420b3ac000
close(3) = 0

So there are a couple of system calls we can see already

execve (execute /bin/ls with argument l)
brk
mmap (reserve 8k of memory with address 0x7f420b3b2000)
open (reads the file /etc/ld.so.cache with file descriptor 3)
fstat (file stats like permission, size)
mmap (read file with descriptor 3into 0x7f420b3b2000)
close (close the file with fd 3)

It’ll be very beneficial to have a list of what all the above means, maybe this can be of some help https://golang.org/pkg/syscall/#pkg-constants

It may also be worth asking… is there anything that might be happening before execve? Are there other exec calls that can / would / could be made, and if so, why is execve used or what is the difference between them?

Note: I’ve linked to execve, exec, etc. system calls so you can review that documentation and answer the questions yourself. I find it more useful this way as you may be reading additional information that I’d skip over. If you are asking yourself “what if…” after reading.. then keep digging until that question is also answered (hopefully by some relevant piece of documentation).

As we know, hopefully, a new process “inherits” all of the current shell's environment variables, how is that possible, and .. when a program exits, why does it not also close its calling shell, since “it replaces” the shell with the running program?

It’s not particularly obvious, but at least bash shell, before running execve runs fork which duplicates the shell and all of its current state (with some exceptions and resets). But why can’t we see that in our strace? Because strace is executing our program, and the shell already called fork to start strace. A important detail about fork() is at the bottom of the documentation.

Under Linux, fork() is implemented using copy-on-write* pages, so the only penalty that it incurs is the time and memory required to duplicate the parent’s page tables, and to create a unique task structure for the child.

Think about what this could mean for resource consumptions on a Linux host?

So how did (other) people “find out” that fork() is called by the shell before execve? I believe (at least at times) a good way to find out what happens is to look at the source code, since our tracing tool cannot trace what happens in the shell (I’m assuming you’re using bash).

Bash source code is here : https://ftp.gnu.org/gnu/bash/

...

/*
The meaty part of all the executions. We have to start hacking the real execution of commands here. Fork a process, set things up, execute the command.
*/

static int
execute_simple_command (simple_command, pipe_in, pipe_out, async, fds_to_close)

SIMPLE_COM *simple_command;
int pipe_in, pipe_out, async;
struct fd_bitmap *fds_to_close;

...

A small snippet from execute_cmd.c from the bash source code, I believe the function comment explains what is happening when executing a command without necessarily needing to look at (and understand) the code itself.

The actual fork() is hidden in jobs.c though …

...

/* 
Create the child, handle severe errors. Retry on EAGAIN. 
*/

while ((pid = fork ()) < 0 && errno == EAGAIN && forksleep < FORKSLEEP_MAX)
 {

...

You might ask yourself, are there other things that happen before executing a command? Like fork(), they might not be able to be traced using strace.

End of Part 1

...

In Part 2, I’ll continue with the last question and if time permits walk further down the strace done in Part 1.

pex — python executables

Alex Leonhardt — Tue, 18 Jul 2017 15:31:16 +0000

P ython EX ecutables are awesome , they bring everything you need to run your code with them. Just like virtualenv used to do but without the need of having to initialize a environment or doing that pip path import hack I recently started using more often (as I didn't want to have to distribute an entire environment)

In a nutshell, you install all your requirements with

pip install -t .pip -r requirements.txt

All modules will be installed into .pip which you can then use when you add the path to your PYTHONPATH. You can do that inside your app/script by doing the following

#!/usr/bin/env python

import sys
import os

sys.path.append(sys.path.append(os.path.abspath('.pip')))

import your.modules.from.requirements.txt

So, above isn’t too bad, and you can relatively easy distribute the .pip directory without needing an entire environment and you also won't need to install any requirements onto the destination machine or container (or environment).

Doing the above isn’t bad, but we can do better! Here is where PEX comes in, a python executable that one builds either by using pants (python ant) or by using the pex module. In this example, we'll be using the pex module.

Install pex

$ (sudo) pip install pex

Yep, that’s all you have to do to install pex ;)

Building a .pex file

So, unlike virtualenv and the .pip hack, you cannot easily just add a python script and execute it as part of the .pex file as you (or I) may have expected.

To build a .pex file that functions just like virtualenv or .pip with all our requirements installed, you can do the following short steps, ensure you have installed pex (see above) first.

$ pex module1 module2 module3 -o myenv.pex

To use this myenv.pex (which now has all your required modules available), simply run

$ ./myenv.pex

By default, this will just spin up python itself, then you can import the modules you’ve added to the .pex file or, to run a script, do a simple

$ ./myenv.pex script.py

Using a requirements.txt file may be easier to create your environment though

$ pex -r requirements.txt -o myenv.pex

Running your script inside the .pex, not as a script

So, it turns out you can run your script inside the .pex file, it’s just simply a tiny bit more work than one would like. Isn’t that great!? Think of a go static binary, but it's python.

So, let’s assume you’re wanting to run a script that prints Hello! when executed, obviously that wouldn't need its own environment, etc. but it's just an exercise.

init.py, hello.py & setup.py

First, we need to create the necessary directory structure and files to make this a installable python module/package. For this, we will need the following

./hello/__init__.py ./hello/setup.py ./hello/hello.py

__init__.py can just be empty - yay!

setup.py does need some two lines of boilerplate

from distutils.core import setup
setup( name='hello', version='0.0.1', py_modules=['hello'] )

hello.py will just do what we ask it to - print hello

def hello(): 
 print "hello!"

Yep, that’s all we need — great isn’t it?

Build the .pex file

From outside the “hello” module/package/path, run pex to create the pex

$ pex hello -e hello:hello -o hello.pex

This will install the “hello” module that we created into the .pex file, the -e hello:hello declares an entrypoint which will load the module "hello" and execute the function "hello" when the .pex is being run.

Final result

Output

vagrant@local:/vagrant$ pex hello -e hello:hello -o hello.pex

vagrant@local:/vagrant$ ls
 hello 
 hello.pex

vagrant@local:/vagrant$ ./hello.pex
hello!

File / Dir size comparison

.pex file with ansible installed : 3 MB
.pip dir with ansible installed : 17 MB
virtualenv with ansible installed : 28 MB