Forem: Karan Sharma

DNS Lookups in Kubernetes

Karan Sharma — Mon, 03 Feb 2020 10:03:32 +0000

Originally posted on my blog

One of the primary advantages of deploying workloads in Kubernetes is seamless application discovery. Intra cluster communication becomes easy with the concept of Service which represents a Virtual IP backing a set of Pod IPs. For example, if vanilla service needs to talk to a chocolate service, it can directly use the Virtual IP for chocolate. Now the question is who resolves the DNS query for chocolate and how?

DNS resolution is configured in Kubernetes cluster through CoreDNS. The kubelet configures each Pod's /etc/resolv.conf to use the coredns pod as the nameserver. You can see the contents of /etc/resolv.conf inside any pod, they'll look something like:

search hello.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.152.183.10
options ndots:5

This config is used by DNS clients to forward the DNS queries to a DNS server. resolv.conf is the resolver configuration file which has information about:

nameserver : Where the DNS queries are forwarded to. In our case this is the address of CoreDNS service.
search: Represents the search path for a particular domain. Interestingly google.com or mrkaran.dev is not FQDN (fully qualified domain name). A standard convention that most DNS resolvers follow is that if a domain ends with . (representing the root zone), the domain is considered to be FQDN. Some resolvers try to act smart and append the . themselves. So mrkaran.dev. is an FQDN but mrkaran.dev is not.
ndots: This is the most interesting value and is the highlight of this post. ndots represents the threshold value of the number of dots in a query name to consider it as a "fully qualified" domain name. More on this later, as we discover the flow of DNS lookup.

Let's check what happens when we query for mrkaran.dev in a pod.

$ nslookup mrkaran.dev
Server: 10.152.183.10
Address: 10.152.183.10#53

Non-authoritative answer:
Name: mrkaran.dev
Address: 157.230.35.153
Name: mrkaran.dev
Address: 2400:6180:0:d1::519:6001

For this experiment, I've also turned on CoreDNS logging level to all which makes it highly verbose. Let's look at the logs of coredns pod:

[INFO] 10.1.28.1:35998 - 11131 "A IN mrkaran.dev.hello.svc.cluster.local. udp 53 false 512" NXDOMAIN qr,aa,rd 146 0.000263728s
[INFO] 10.1.28.1:34040 - 36853 "A IN mrkaran.dev.svc.cluster.local. udp 47 false 512" NXDOMAIN qr,aa,rd 140 0.000214201s
[INFO] 10.1.28.1:33468 - 29482 "A IN mrkaran.dev.cluster.local. udp 43 false 512" NXDOMAIN qr,aa,rd 136 0.000156107s
[INFO] 10.1.28.1:58471 - 45814 "A IN mrkaran.dev. udp 29 false 512" NOERROR qr,rd,ra 56 0.110263459s
[INFO] 10.1.28.1:54800 - 2463 "AAAA IN mrkaran.dev. udp 29 false 512" NOERROR qr,rd,ra 68 0.145091744s

Whew. So 2 things piqued my interest here:

The query iterates through all search paths until the answer contains a NOERROR code (which the DNS clients understand and store it as the result). NXDOMAIN simply indicates no record found for that domain name. Since mrkaran.dev isn't an FQDN (according to ndots=5 setting), the resolver looks at search path and determines the order of query.
A and AAAA records are fired parallelly. This is because single-request option in /etc/resolv.conf has a default configuration to perform parallel IPv4 and IPv6 lookups. You can disable this using single-request option.

Note: glibc can be configured to send these requests sequentially but musl cannot, so Alpine users must take note.

Playing around with ndots

Let's play around with ndots a bit more and see how it behaves. The idea is simple, for the DNS client to know whether a domain is an absolute one or not is through this ndots setting. For example, if you query for google simply, how will the DNS client know if this is an absolute domain. If you set ndots as 1, the DNS client will say "oh, google doesn't have even one 1 dot, let me try going through search list. However, if you query for google.com, the search list will be completely ignored since the query name satisfies the ndots threshold (At least one dot).

We can see this by actually doing it:

$ cat /etc/resolv.conf
options ndots:1

$ nslookup mrkaran
Server: 10.152.183.10
Address: 10.152.183.10#53

** server can't find mrkaran: NXDOMAIN

CoreDNS logs:

[INFO] 10.1.28.1:52495 - 2606 "A IN mrkaran.hello.svc.cluster.local. udp 49 false 512" NXDOMAIN qr,aa,rd 142 0.000524939s
[INFO] 10.1.28.1:59287 - 57522 "A IN mrkaran.svc.cluster.local. udp 43 false 512" NXDOMAIN qr,aa,rd 136 0.000368277s
[INFO] 10.1.28.1:53086 - 4863 "A IN mrkaran.cluster.local. udp 39 false 512" NXDOMAIN qr,aa,rd 132 0.000355344s
[INFO] 10.1.28.1:56863 - 41678 "A IN mrkaran. udp 25 false 512" NXDOMAIN qr,rd,ra 100 0.034629206s

Since mrkaran didn't specify any . so the search list was used to find the answer.

Note: ndots value is silently capped to 15 and is 5 in Kubernetes as default.

Handling this in Production

If your application is of the nature that makes a lot of external network calls, the DNS can become a bottleneck in case of heavy traffic since a lot of extra queries are made before the real DNS query is even fired. It's quite uncommon to see applications appending the root zone in the domain names, but that can be considered as a hack. So instead of using api.twitter.com, you can hardcode your application to include api.twitter.com. which would force the DNS clients to do an authoritative lookup directly on the absolute domain.

Alternatively, since K8s 1.14, the dnsConfig and dnsPolicy feature gates have become stable. So while deploying a pod you can specify ndots setting to something lesser, say 3 or if you want to be really aggressive you can turn it down to 1. The consequences of this will be that every intra-node communication now has to include the full domain. This is one of the classic tradeoffs where you have to choose between performance and portability. If the app doesn't demand super low latencies, I guess you need not worry about this at all since DNS results are cached internally too.

References

I got to know about this peculiarity first, in a K8s meetup which I went to, last weekend where the folks mentioned about having to deal with this.

Here are some additional links you can read on the web:

Note: I'm particularly not using dig in this post. dig apparently automatically adds a . (root zone identifier) to make the domain an FQDN one without even first going through the search path. I've mentioned about this briefly in one of my older posts. Nonetheless, it's quite surprising to see that you need to give a flag to make it behave in what seems to be a standard way.

It's always DNS, isn't it ;)

Fin!

Originally posted on my blog

sshuttle - A better ssh tunnel

Karan Sharma — Sun, 12 Jan 2020 07:50:37 +0000

The Motivation

Sometime back I had to access a Kubernetes API server which was firewalled to a private VPC network. I didn't want to setup a separate bastion instance just to access this cluster, cause TBH bastions are kinda redundant in K8s as every task can be performed through the client-server APIs using kubectl. So, all I needed was access to this API server from a trusted network in a secure way. Thanks to my friend @sarat, I got to know about sshuttle. sshuttle is quite unique in the sense that it's not really a VPN but acts like one (for most practical purposes). sshuttle lets you access an internal network through a trusted node inside the VPC, without you having to deal with the mess of port forwarding or VPNs.

The basic idea is pretty simple, sshuttle starts a local python server in your host machine and creates iptables rules to route the destination packets of the specified CIDR blocks to this local server. At the server, the packets are multiplexed over an ssh session and sent to the server. The server disassembles the multiplexed packet and the routes them to upstream. So, basically this is a clever hack to avoid TCP over TCP (which again is a mess on unreliable networks). Multiplexed streams on ssh is just a single stateful TCP connection (as compared to VPN connections which are stateless). Now you must be wondering, how come the target server disassembles the packets. Yes, there needs to be some kind of sshuttle daemon running which does that for you. This is where sshuttle does some magic, it automagically deploys a python script on your target host to perform this task. So yes, for sshuttle to work, both the client and target need to have python and iptables installed.

Usage

sshuttle -r user@port x.x.x.x

All the packets routed to the CIDR block will now go through sshuttle daemon, since it configured iptables rules for them.
Also, sshuttle starts a local python server on your host machine. You can see it using netstat:

$ sudo netstat -tunapl | grep python
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:12300         0.0.0.0:*               LISTEN      27425/python

There's a python server listening on port 12300 in my host machine. To actually verify, this indeed is started by sshuttle, you can use pstree -p | less and search for sshuttle. Here you can see sshuttle did indeed start a python server and the PID (27425) matches with the one we saw in netstat command.

    -zsh(13201)---sshuttle(27425)-+-ssh(27446)---ssh(27447)
                                    `-sudo(27427)---python(27445)

You can even forward DNS queries with the --dns flag. This is super helpful if you have something like Route53 to host your DNS records on a private zone (for eg tld like .internal).

Better than SSH tunnels?

Yes, you can also port forward with ssh using:
ssh -nNT -L <local-port>:{upstream-host}:{upstream-port} user@remote

The problem with ssh tunnels is that they experience frequent packet loss on a normal WiFi connection and it's quite frustrating to deal with them. Moreover, sometimes you need access to multiple ports in your private network which requires you to explictly provide them with -L flag which I find it as cumbersome. Also, you cannot forward DNS queries (over UDP) since ssh can only do TCP.

sshuttle has made my life so simple!

Fin!

Originally posted at my blog

Introducing kubekutr

Karan Sharma — Tue, 07 Jan 2020 10:32:36 +0000

kubekutr was born out of my frustration of organising K8s resource manifests files/directories. For the uninitiated, K8s lets you hold the state of your cluster declaratively as "manifest" files. K8s does so by a lot of asynchronous control loops to check whether the desired state matches the real world state and in case of drifts, it resets back to the user desired state (I oversimplified this, but hope you get the drift ;)). These files are predominantly YAML but there's support for JSON as well.

Anyway, to create these manifest files for a production level project is quite a bit of manual labour. The API spec of every resource in Kubernetes is quite daunting and overwhelming. There are tools like Helm which abstract away the complexity of these YAML with it's own templating system. There are quite a lot of these charts available for 3rd party apps here. The idea is you populate the Chart with just your own config "values" and you've a deployment ready in no time. Admittedly this works quite well for something you want to take out for a quick spin but personally I am not quite a fan of hiding away the complexity with a magic layer. Also, the problem with Helm was the "Chart" (and templates) still had to be written by someone if you have a bespoke application. Helm is more geared towards common off the shelf apps like DBs, Key Value stores, web proxies etc.

I found out kustomize few months back and quite happy with it's approach towards managing manifests. The basic idea behind kustomize is that you create a base and any kind of "customisations" must come as overlays. This is such a powerful technique over wrangling templates. kustomize A common approach is to name these overlays based on the environment. For example, dev deployment can have replicas: 1 for a pod, but prod can apply a "patch" to update with repliacs: 3. This way of separating two environments helps a lot when you follow GitOps approach of deployment. All fine and good, until I realised I spent way too much time on copy-pasting the bases for different projects and manually editing these files for the new project config.

Then I did what any other programmer would do, spend some more time to automate :P And that is how kubekutr was born. (Quite an anticlimax I know!)

kubekutr is a really simple tool to bootstrap a Kustomize base. kubekutr reads a config file, templates out different resources and produces them as YAML files. Now, I know a lot of you reading this would be going Another damn templating solution in your mind and while that reaction is warranted, given that we have 200+ tools in the community (everyone trying to solve similar problems in their own ways), I legit could not find a simple enough tool which would let the boring part of scaffolding a base out of my way and let me focus on what's more important: the actual deployment. Hence I just decided to roll out my own solution which is the best one according to IKEA effect (just kidding).

Workflow

So, let's say you need to create a Nginx deployment, the kubekutr config.yml would look something like this:

deployments:
  - name: nginx
    replicas: 1
    labels:
      - name: 'service: nginx'
    containers:
      - name: nginx
        image: 'nginx:latest'
        portInt: 80
        portName: nginx-port
services:
  - name: nginx
    type: ClusterIP
    port: 80
    targetPort: 80
    labels:
      - name: 'service: nginx'
    selectors:
      - name: 'service: nginx'

To create the base:

kubekutr -c config.yml scaffold -o nginx-deployment

nginx-deployment folder is initialised and you can view deployments/nginx.yml and service/nginx.yml which kubekutr created.

$ tree nginx-deployment

|-- base
|   |-- deployments
|   |   `-- nginx.yml
|   |-- ingresses
|   |-- services
|   |   `-- nginx.yml
|   `-- statefulsets

$ cat base/deployments/nginx.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    service: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      service: nginx
  template:
    metadata:
      labels:
        service: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:latest
          ports:
          - containerPort: 80
            name: nginx-port

$ cat base/services/nginx.yml

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    service: nginx
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  type: ClusterIP
  selector:
    service: nginx

You can now use the generated folder as a Kustomize base.

Non Goals

kubekutr isn't meant to replace the existing tools, it's just a real simple cookie cutter approach to kustomize bases and that's pretty much it. kustomize is native to Kubernetes and exposes the full API spec to end users. I feel that is much more better approach than templating solutions, the users must be exposed to the standard conventions rather than a random tool's own config fields. The benefits are the same conventions can then be used across a wide variety of tools (like kubekutr) and users are in better control of the underlying resources. Adding a layer of magic also makes it harder to debug when shit goes down. Hence kubekutr chose kustomize to do all the heavy lifting of managing manifests.

There's a lot of scope of improvements, but I wanted to just Ship It! and get some initial feedback. Let me know your thoughts on this :)

Fin!

Originally published at my blog

My Development Setup on a Macbook

Karan Sharma — Fri, 04 Nov 2016 14:13:26 +0000

I recently purchased a new 13" Macbook Pro 2015(after Apple's brilliant keynote) and quite like it so far. This is my first experience with macOS. After using strictly GNU/Linux distros for about 4 years(Fedora, Ubuntu, Mint, Arch, Elementary OS) I finally decided to take the plunge and use macOS. The battery life, excellent keyboard and trackpad, a beautiful display makes coding on it a really enjoyable experience. Something that I was surely missing on my bulky HP Pavilion G6.

Before starting off with my development setup, I would say there are a plenty of such blog-posts and guides available to make your machine the #ultimatedevmachine. The aim of this post is not that but just to share some of the tools I believe make your macOS experience more complete. I personally followed this as the holy grail but I'll break it down for you to include only the most relevant things that a generalist developer can install on her laptop and also not make it bloated.

iTerm2

As developers spend most of the time tinkering with stuff on terminals, the first thing that you should do is to replace macOS default terminal with iTerm2. It has a lot of great features, about which you can read more here.
I'm using Solarized Dark theme which can be easily downloaded over here. Apart from this, there are several other customisations which I simply took from this guide.

Homebrew

It's the essential package manager every developer should be using. It makes installing other packages very convenient by downloading latest stable release, creating a symlink to /usr/local and some really cool features like installing directly from a PR. You can read more about such features here. As a learning exercise for myself, I am going to try my hands on creating a formula for coala.

zsh

No discussion of the terminal should ever be complete without mentioning zsh even once IMHO. I've been using zsh for 2 years and there's just no going back to old school bash. I've got so accustomed to zsh that it feels almost impossible to work on bash anymore. You can install zsh by
brew install zsh

I also use Prezto along with zsh rather than the popular Oh My Zsh! because it's darn slow after a couple of plugins. You can follow the installation details present in the official documentation here.
To change default theme you need to edit the following line in ~./zpreztorc

zstyle â€˜:prezto:module:prompt' theme â€˜powerline'

with the theme name. You can find out all theme names using

prompt -l

You can find a comprehensive list of all things zsh over here.

Sublime Text

As a developer you'll also spend a hefty time with editors as they are the place where magic happens. I have setup my Sublime Text to maximize my productivity while coding. I use the following plugins, but first in order to install any one of them you need to install Package Control which is the package manager for Sublime Text.
After you have setup Package Control, you can verify that it should look like:

Some of the plugins I use:

I'm using Material Theme which really prettifies it by a stretch.

Vim

I'm learning to use more and more Vim these days so I'll keep this section as a TBA and will update soon.

macOS Apps

These are some of the apps that I believe should come preinstalled in any macOS.

Alfred
InsomniaX
CleanmyMac3
Omnifocus
Ulysses
Android File Transfer
Cheatsheetapp
iStatMenus

So far I'm really happy with it but will keep adding tools or plugins that I find on the go. I am still in the honeymoon phase with my Mac so if you think I did an injustice to this post by not including your favourite tool, tweet it out to me and I'll definitely check it out.
Till then, Happy Coding!

This article was originally posted on my blog over here

Hi, I'm Karan Sharma

Karan Sharma — Fri, 04 Nov 2016 13:54:51 +0000

I have been coding for 2 years.

You can find me on GitHub as @mr-karan

I live in New Delhi, IN.

I am a final year CS undergrad.

I mostly program in these languages: Python, Javascript.

I am currently learning more about ReactJs.

Nice to meet you.