Forem: Rishabh Gupta

Build Multi-Arch docker images on Travis

Rishabh Gupta — Fri, 01 Mar 2019 13:25:22 +0000

Read the post on my blog

A lot of people are shipping applications as docker images so that they can run in a container. Docker has first class support for multiple architectures and if you are trying to add support for different platforms to your app, you might have come across the problem of automating this process. The pain point is the fact that most hosted CI services like travis and circle only provide amd64 environment.

But some projects do ship out constantly updated docker images for different architectures and they don't do so manually. Let's take a look at the official docker images for golang (repo), in the README you'll see that they use Travis, Appveyor and Jenkins. Jenkins is used to build arm images, but it is a self hosted service and hence not our goal. Another approach is using drone cloud as a ci service, since they have arm servers which can be used to build docker images. But if you don't want to change the current CI service you were out of luck.

You don't need physical ARM servers

You can run a program compiled for ARM on amd64 linux machine if it has binfmt_misc support. It essentially allows you to run a program without worrying about which architecture it was built for. binfmt_misc doesn't come by default in ubuntu, but adding it is really easy using docker.

Set dist: xenial to .travis.yml since I've found that binfmt_misc mount, errors out on ubuntu precise which is the default travis distro.
Add the following block to travis config:

   before_install:
     - sudo docker run --privileged linuxkit/binfmt:v0.6

This single command mounts binfmt_misc to the os /proc/sys/fs directory. Now our build server has superpowers.

Using Buildkit for building docker images

Buildkit is a toolkit for building container images, it is used in docker and many other places and is advertised to be quite fast. It has a client-server architecture and you'll need to start the server and then use the client to interact with it. There is a fantastic demo on building multi-arch docker images by Tõnis Tiigi.

Our process will goes like this, first we'll start the buildkit server as a container process, then we'll copy the buildctl binary which the command line frontend for buildkit to our /usr/bin directory and then we'll set BUILDKIT_HOST environment variable. Finally, before_install block will look like this:

before_install:
  - sudo docker run --privileged linuxkit/binfmt:v0.6
  - sudo docker run -d --privileged -p 1234:1234 --name buildkit moby/buildkit:latest --addr tcp://0.0.0.0:1234 --oci-worker-platform linux/amd64 --oci-worker-platform linux/armhf
  - sudo docker cp buildkit:/usr/bin/buildctl /usr/bin/
  - export BUILDKIT_HOST=tcp://0.0.0.0:1234

We specify --addr to buildkit server to bind it to the specified port and address, we also give --oci-worker-platform argument to tell the buildkit server that our machine can be used to create images for these platforms.

Building Images

We can now finally write the script to build images.

PLATFORM=arm # equivalent to armhf
DOCKERFILE_LOCATION="./Dockerfile.armhf"
DOCKER_USER="someone"
DOCKER_IMAGE="some_server"
DOCKER_TAG="latest"

buildctl build --frontend dockerfile.v0 \
        --frontend-opt platform=linux/${PLATFORM} \
        --frontend-opt filename=./${DOCKERFILE_LOCATION} \
        --exporter image \
        --exporter-opt name=docker.io/${DOCKER_USER}/${IMAGE}:${TAG}-${PLATFORM} \
        --exporter-opt push=true \
        --local dockerfile=. \
        --local context=.

This command has a few options, we'll go over them one by one

--frontend dockerfile.v0 specifies which frontend to use with buildkit, there is also a gateway.v0 frontend, but we'll stick with dockerfile.
--frontend-opt, this is used to pass parameters to the fronend used.
- platform specifies the platform we want to build the image for, these are listed here.
- filename specifies the filename of Dockerfile to be used equivalent to docker build's -f argument.
--exporter image specifies the export plugin to be used, there are also oci and local.
- name is the url of the registry to push docker images to.
- push can be set to true to push the built docker images.

Pushing manifest - Final Step to Multi-Arch

A container manifest is a file that contains data about a container image. We can create a manifest which points to images for different architectures so that when using the image on a particular architecture the docker automatically pulls the desired image.

Creating manifest is an experimental Docker-cli feature and you should update docker for good measure. Add following lines to .travis.yml

addons:
  apt:
    packages:
      - docker-ce

First we create the manifest, then we annotate the manifest and finally do a push. For this example let us take the user to be "someone" and docker image to be "my-image" with the tag "latest" i.e the image is "someone/my-image:latest", PLATFORM_1 and PLATFORM_2 are two different platforms for which we need to create the multi arch image. The commands look like this:

export DOCKER_CLI_EXPERIMENTAL=enabled

docker manifest create someone/my-image:latest \
            someone/my-image:latest-${PLATFORM_1} \
            someone/my-image:latest-${PLATFORM_2}

docker manifest annotate someone/my-image:latest someone/my-image:latest-${PLATFORM_1} --arch ${PLATFORM_1}
docker manifest annotate someone/my-image:latest someone/my-image:latest-${PLATFORM_2} --arch ${PLATFORM_2}

docker manifest push someone/my-image:latest

Now you ahve successfully automated the creation and push of multiarch docker images.

Concluding points

I added multi-arch support to cron-connector and was able to remove armhf specific config files as well as automate the pushing of armhf images. You can find my commit here. Also, mad props to Johannes Würbach who's Grafana PR #14617 was where I picked up the binfmt_misc part and to buildkit team for creating a really nice piece of software.

Develop with kubernetes on Docker Desktop with WSL

Rishabh Gupta — Tue, 12 Feb 2019 09:01:05 +0000

Docker Desktop for Windows comes shipped with kubernetes out of the box. It preconfigures a lot of stuff on your Windows machine, like downloads kubectl and puts everything in PATH variable, so that it can be accessed from powershell. But a lot of developers have took a liking to WSL which provides a more linux native environment for development. This post will go through steps to set up kubernetes for wsl.

You should have docker for desktop installed and WSL enabled to follow along.

Steps:

Enable kubernetes if you haven't done so already. Go to docker desktop settings -> Kubernetes tab -> check "Enable Kubernetes" setting. After status says "kubernetes is running" run the following command in powershell.

kubectl get nodes

If it shows docker-for-desktop then it means kubernetes is running correctly.

Open WSL console and install kubectl using the following commands

sudo apt-get update && sudo apt-get install -y apt-transport-https
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubectl

Setting up kubeconfig file: This is the magic step, kubernetes runs on port 6443 on your host computer and to access it you'll need a kubeconfig file. This is already done by docker for the windows environment but for WSL you'll need to grab the config. Turns out it's pretty easy. Just run the following command in bash and replace {username} with your windows username (not the WSL user name)

cp /mnt/c/Users/{username}/.kube/config ~/.kube/config

This command copies your kubeconfig file from windows to wsl.

All done: Just run

kubectl get nodes

to see a docker-for-desktop node.

React hooks and their dependence on each other

Rishabh Gupta — Sun, 27 Jan 2019 14:01:20 +0000

Originally published on my blog

Pretty much everyone knows about react hooks already. From react docs:

Hooks are an upcoming feature that lets you use state and other React features without writing a class

One very notable difference between hooks and classes is that there can be multiple state variables and you are not limited to fiddling with a single state object. This obviously provides greater separation of concerns and more composability for effects inside a component, but if done wrong it can introduce weird bugs. We'll look at some pointers that might help you with dealing that complexity.

Fetching data depending on some state variable

This is the most common case of state dependence. You want to fetch some data, that depends on a state variable. For example: Getting different quotes based on some user input. I'll go ahead and give you the code sample (view on CodeSandbox):

function App() {
  let [inp, setInp] = useState("");
  let [quote, setQuote] = useState("");

  useEffect(
    () => {
      setQuote(GetQuote(inp));
    },
    // This is important
    [inp]
  );

  return (
    <div className="App">
      <input
        type="text"
        value={inp}
        onChange={event => setInp(event.target.value)}
      />
      <h1>{quote}</h1>
    </div>
  );
}

You'll see that the useEffect hook has the second argument as an array which contains a single value, the state variable it depends on. This effectively tells useEffect to be triggered only when the state changes. This is a basic skeleton of how you can chain hooks in a sane way. You can add more states and effects to this components and chain them using this method so they don't clash with each other.

This is a case of "Effects depending on state"

Fetching data is only a simple effect, other types of effects like manipulating DOM nodes or updating a server can be done in a similar manner. The state variable can be something like the position of the cursor or it can even be something obtained from a hook library.

Don't call hooks in nested scope

This has been said a lot of times before, but I should just say it again. Hooks are impure and their behavior in nested scopes is undefined. Below are some tips to shift the scope.

For conditional effects, rely on the second useEffect argument and comparing the values inside the effect itself.
Similarly, for effects in a loop, shift the loop in the effect callback scope.
Always initialize state with a default value and don't put it in condtional scope.
Set the state as an array or object instead of initializing states in a loop.

This is not the end

These are somethings I keep in mind while using hooks. You'll encounter and discover a lot more patterns from your use of hooks, when you do, please share them with the rest of the community.

Displaying a Series of Posts

Rishabh Gupta — Thu, 10 Jan 2019 13:16:40 +0000

I originally published this post on my blog.

This is another feature that I didn't have in mind at the start. I was inspired by dev's series feature and I even created my first ever series, this is the fourth post in that series. The tech / casual / major sections on the front page are gone, they were taking up valuable space and I'm pretty sure instead of sorting out the posts they added to the confusion.

One of the big reasons I wanted to add this is because of what I'm working on as the next series of posts. I don't want to spoil the fun so I won't tell what I'm exactly working on but the posts would need to be tied together coherently to make better sense.

Dev's series are limited

I took the concept of series from dev.to but I found the implementation fits greatly in dev but not so on my blog. So, I came up with a more comprehensive design which when implemented fitted perfectly on my blog.

Changes to Content Management System

I've talked briefly about my content management system in the previous post in the series. I've wanted to keep things on this blog as simple as possible and push most of the complexity towards the frontend which is the major processing layer in the architecture. Keeping things simple, my content management system has:

main.json : Contains metadata for all the posts.

[
  {
    "id": 4,
    "slug": "blog-development-live-preview",
    "published": true,
    "filename": "LivePreviewBlogChanges",
    "type": "tech",
    "tldr": "Adding live preview of changes when I'm writing the blog.",
    "title": "Live Preview Blog Edits",
    "timestamp": 1544608657,
    "dev_to": "https://dev.to/zeerorg/live-preview-blog-edits-2oh1",
    "series": "this-blog-series"
  },
  {
    "id": 6,
    "slug": "frontend-caching",
    "published": true,
    "filename": "FrontEndCaching",
    "type": "tech",
    "tldr": "A simple trick to improve site performance with frontend caching",
    "title": "Caching data in frontend",
    "timestamp": 1545049130,
    "dev_to": "https://dev.to/zeerorg/caching-data-in-frontend-3973"
  }
]

html/ : Contains the markdown content compiled to html.
series.json : Contains details about the series and list of posts in that series.

{
  "this-blog-series": {
    "posts": [
      "this-blog-architecture",
      "blog-development-live-preview",
      "rss-feed-azure-functions",
      "adding-series-to-blog"
    ]
  }
}

I introduced a new series.json so that the frontend stays compliant with the existing main.json and I don't have to calculate the posts in the series on frontend which would be more complex and error prone. I found a middle ground between backend and frontend complexity.

Optimization sacrifices

I'll admit my frontend caching design is not the most optimized approach. The 2 main costly operations are:

Sending a network request everytime I fetch from cache. The render prop component GetContent which is responsible for caching and loading data from backend, initially acquires data from cache and also sends a request to the backend to fetch the latest data. This "update on new" caching provides better experience to viewers, as they get the latest updates instantly without closing the browser session but also don't have to wait for the network request to complete to get the first render.
Parsing the text to JSON, everytime I need to access metadata. Everytime there's a need to fetch main.json, GetContent makes the call, hence only the raw text version is returned which is parsed using JSON.Parse utility. The only way to prevent this is to introduce Redux or maintain a global variable for the data. This would also mean treating json data and html data differently and the responsibility will fall on the shoulders of GetContent and any component which fetches the data. This increase in complexity will certainly won't help much when compared to single render time, since the parse is called only once per render.

Conclusion

Even with some sacrifices, the project has scaled with the addition of a new feature without giving a headache, I'd consider this an achievement. I'm also really happy with how this feature turned out to be and I'm looking forward for your feedback.

Sharing your apps with the world for 5$

Rishabh Gupta — Fri, 04 Jan 2019 07:20:26 +0000

Raspberry Pi is a bit underpowered to compete with a full server, no doubt about it, but a lot of our hobby apps which don't ever see the light of the internet, or our test projects which are not ready for the cloud world are the best match with this tiny marvel. You learn a lot about how servers work and get a project hosted on internet, for the cheapest price possible.

What I run on my Raspberry Pi

I currently have an OpenFaas instance running on my Raspberry Pi serving two applications, first a Github Webhook endpoint and second a Password generator based on EFF list. OpenFaas runs over Docker swarm which I manage using a Portainer ui also running as Docker container. I monitor resource usage with Glances server. I plan on adding a Jupyter Hub instance and putting all of this behind an NGINX proxy server possibly with auth.

Yes, my setup is a bit too overkill for a small machine like raspberry pi, but it has taught me a lot of things about a lot of different software and possibilities of this tiny monster.

You can access the password generator on this link.

Prerequisites

5$ Raspberry Pi
Domain Name (Optional but highly recommended for https)

Diagram Time

Reverse proxy - because, Security

Setting up a reverse proxy server might seem like a bit of a pain but it provides a secure gateway as exposing your home network and your applications on the internet is always a risk. You can also use it to setup SSL if you have a domain. I prefer Nginx simply because it has a lot of guides and is simple to setup (Apache's config scares me). Simple HTTP reverse proxy:

server {
  listen 443;
  location / {
    proxy_pass http://locahost:[:PORT]/;
  }
}

Deploying:

sudo nano /etc/nginx/sites-enabled/default
# Copy the config then Ctrl+x to save
sudo nginx -s reload

Also see: SSL with nginx

Correctly starting your applications

The second biggest mistake you can make after ignoring security, is assuming that your raspberry pi will be running most of the time, and if it reboots, you'll just start your services again. Don't make this mistake, setting up a self starting service is trivial and its the best that you can do for your peace of mind. Refer this official Raspbian guide on setting up a self starting service.

app.service

[Unit]
Description=My Applicaiton
After=network.target

[Service]
ExecStart=/usr/bin/python3 -u main.py
WorkingDirectory=/home/pi/applicaiton
Restart=always
User=pi

[Install]
WantedBy=multi-user.target

Copy and enable service

sudo cp app.service /etc/systemd/system/app.service
sudo systemctl start app.service
sudo systemctl enable app.service

ARM has come a long way

Another concern which people have before going and hosting their service is that it'll be a pain to build for a different architecture, or they'll have to submit to satan to get their application running. It was the case few years back, but not anymore! Building your application for different architecture is as easy as providing a build time flag for compiled languages, or installing the interpreted languages runtime. Interpreted languages like python and javascript can be easily installed using existing package manager.

Statically compiled languages can be cross compiled for ARM on your current machine. For example:

Golang: env GOOS=linux GOARCH=arm GOARM=5 go build
.NET Core: dotnet publish -r linux-arm

Adding RSS with .NET Core

Rishabh Gupta — Sun, 23 Dec 2018 06:03:34 +0000

This is 3rd post in continuation to my major project on this blog.

RSS is an old technology and has been fading out of the general ecosystem over the years. With Google shuting down its Google Reader service and Firefox dropping support for RSS in their last 2018 release. I admit I've never actually gotten into RSS, setting up a reading list for different blogs. I am not a frequent blog reader and most of my news source is through apps which meant I never actually had any need for RSS.

So when I thought of adding RSS to my website it was actually a decision I took after thinking long and hard. I actually went ahead and tried out Feedly to get an idea of how modern RSS readers work. I can certainly see why a lot of people do like it, which forms the second reason why I thought of going ahead with it. The last reason is purely because I wanted to learn more about RSS and XML in general. This would be my first time implementing XML based schema for data.

.NET Core SyndicationFeed

The main library that implements helper classes for RSS feed in .NET is System.ServiceModel.Syndication. The top level class is SyndicationFeed which can be serialized to not only the latest RSS 2.0 standard but also to Atom 1.0 feed so you can choose whichever you want. You'll need to add the library module through NuGet.

First we create a syndication feed object and add preliminary information to it. Like the title, description and url of the blog. Then we add authors, here we can specify more than one authors.

// Top level feed setup
SyndicationFeed feed = new SyndicationFeed(title, description, url);
feed.Language = "en-us";
feed.LastUpdatedTime = DateTime.Now;

// Adding Authors
SyndicationPerson sp = new SyndicationPerson("r.g.gupta@outlok.com", "Rishabh Gupta", "https://zeerorg.site/");
feed.Authors.Add(sp);

Every <item> element in RSS is represented as an object of SyndicationItem class. The items can be added to the feed object.

SyndicationItem item = new SyndicationItem(title, description, url);
item.LastUpdatedTime = DateTimeOffset.FromUnixTimeSeconds(timestamp);

feed.items.Add(item);

One important but optional element inside the <item> element is <content:encoded> element, this is important only for those feeds which also need to provide content of the post along with the updates. But setting this is not directly supported inside the api, so we need to create a custom element and assign it to the item.

// The third argument specifies
XMLDocument doc = new XmlDocument()
XMLElement content = doc.CreateElement("content", "encoded", "http://purl.org/rss/1.0/modules/content/");
content.InnerText = htmlContent;
item.ElementExtensions.Add(content);

After setting up the feed it can be serialized to RSS 2.0 standard.

StringWriter sw = new StringWriter();
XmlWriter rssWriter = XmlWriter.Create(sw);

// The second false specifies the format for feed.
// As it can be converted to an older RSS 2.0 format or the newer a10 format.
Rss20FeedFormatter rssFormatter = new Rss20FeedFormatter(feed, false);
rssFormatter.WriteTo(rssWriter);
rssWriter.Close();
string FeedString = sw.ToString();

Getting RSS Feed for my blog

For my blog all the metadata is stored in a main.json file which is fetched from the blob storage. It's a very rudimentary content management system. The json contains a list of posts as json objects and each item contains data corresponding to that post. One of the data that is stored is filename which specifies the filename that contains the HTML to be fetched from blob container. The RSS feed generator extracts data from JSON object of every post for the <item> elements and sets the html content fetched from the blob container using filename attribute.

First, I used to trigger this function on every request, but after some tests I realized creating an RSS Feed on the fly was a costly operation, so I am triggering the creation of rss on every change to main.json and saving the results in a blob which is served directly.

Checkout my rss feed here.

Caching Data in frontend

Rishabh Gupta — Tue, 18 Dec 2018 15:07:24 +0000

We'll talk about ways you can effectively cache requests sent to backend without fidelling with backend HTTP headers.

Coordinating data between state changes can be hard, consider an application where you are sending request to the backend to get a list of posts to be displayed, and when user clicks on a post the application sends another request to get that post data. Now the backend is fairly simple so you get the precise data for that post, but you also want to display the title of next post and previous post, and maybe some sponsor data which is shared during the user's session. Requesting data again from backend is wasteful in such cases so the frontend has some options at this point.

Managing Javascript state

The frontend can keep a track of the data that will be reused. Storing that data in a global variable, passing state to higher levels, or using an api like React Context. There are problems with each of these approaches, global variables are evil by default. Passing data around components or maintaining it in a context like api can become messy as number of requests grow.

Using a state management framework

This is a very typical use case for a JavaScript state management framework like redux. They provide a way to manage complex application data. But if you are anything like me, the idea of introducing a new framework and overhead of learning to code around it can be a daunting task. These frameworks can also force opinionated design on your frontend so for someone who is not familiar with one, it can be a big commitment.

Browser Storage (the real MVP)

We come at our final answer, browser storage api. It is a key value store which is managed by the browser. There are two types of browser storages: local and session. Both of these provide a similar api, but, while the local storage is never cleared, session storage is cleared after the tab is closed. This approach is a lot better than our previous approaches as it's not as barebones as passing the state around and not as complex as learning a new state management framework.

Browser storage api includes only two operations, setItem and getItem and as you can probably guess setItem stores the value for a given key and getItem retrieves the value. We are free from managing the state ourselves and can just provide the key and value for the data to store and retrieve it later.

An example use of this api is demonstrated by creating a function that invokes a GET request to a url and returns the result as a promise.

// Without caching
function FetchData(url) {
  return fetch(url).then(res => res.text())
}

// With Caching
function FetchData(url) {
  let storageData = sessionStorage.getItem(url);
  if (storageData === null) {
    return fetch(url).then(res => res.text()).then(textData => {
      sessionStorage.setItem(url, textData)
      return textData
    })
  }
  return Promise.resolve(storageData);
}

We treat the provided url as our key and store the fetched data, so that any subsequent request is fulfilled from the cache. The best part about this approach is that it is easier to understand and doesn't intrude with our frontend code. It is also the best solution for our problem in this case.

Caching is one of the most effective techniques to optimize performance and user experience. Storing request data on the frontend provides speedy navigation and greater control over what is stored by avoiding unnecessary requests to the backend.

If you like my posts checkout my blog

Live Preview Blog Edits

Rishabh Gupta — Sat, 15 Dec 2018 06:19:31 +0000

Live Preview Blog Edits

This is 2nd post in continuation to my major project on this blog.

The original requirements didn't consider live preview as an option. Seems like I was wrong, the live preview would help me see the current changes inside the window. This also gives me the flexibility to see my changes and gives me more confidence in what I'm writing. A lot of these changes are not compatible with what I see in my editor markdown preview because I'm using a custom CSS for posts.

Adding live preview involves changes in the frontend and a test backend. One option I considered was:

Uploading directly to Azure storage. This would be great since I won't have to change much in the backend and could leverage what I've built till now. But life ain't that easy, uploading to the backend takes some time and the azure blob trigger can take up to 10 minutes to run 😲!! This solution was obviously not viable, I cannot escape this ordeal easily.

Test Backend

I reluctantly start with building a backend. The good part is that laying out a simple blog architecture makes this process much easier. If I could find a way to provide the current draft as an input to the frontend I could preview the changes in my local machine. This proved to be a relatively easy problem to tackle.

For the backend, I run an HTTP based file server to emulate my blob storage use. I use nodejs-based http-server.

For the frontend, I check if the process.env.NODE_ENV environment is set to "development" and point to the local server.

Building markdown on change

My azure function backend uses Markdig to compile markdown to HTML. I have a test project from few days before, that used Markdig to compile markdown, I picked it up to repurpose it for building the markdown files on change. I implemented a simple file watcher and compiled the files that changed to markdown. I'll link down the code.

What I learned

C# FileSystemWatcher was pretty easy to implement. I also learned about the process.env.NODE_ENV variable in React.

Code links

My Blog's Architecture

Rishabh Gupta — Thu, 13 Dec 2018 16:05:46 +0000

Blog's Architecture

This is my first blog on how I'm creating my blog. I've started writing this before the blog site is up detailing what restrictions I'm going to place, the features I'll be adding and how I'll go about them.

Block diagrams ftw

A simple block diagram for the overall architecture.

Requirements (or maybe restrictions)

Every project needs some requirements so I should underline those for this blog. I want to work under these so I don't go on doing something I actually regret later on.

Easy to write. I don't want to go though the "write-build-deploy" cycle over and over again everytime I write a post. The process should be as easy as typing in my editor, seeing the preview and uploading the file. That's it, no preprocessing, no building, minimal stuff.
No bullsh*t features. The feature set should be minimal. So things like comments and reacting or sharing of a post are not going to be included. Any new feature added will have an explanation on why its there.
Easier on the bill. This is not going to earn me something so I want this to be as easy on the wallet as possible. This might also push me to find newer solutions (hopefully).

Going about it

This is why and what I decided.

Storing in Azure blobs. This is going to be the place for storing original blog markdown files. This keeps a backup and also creates a destination for original source files.
Stored files are automatically processed when uploaded. I set this up and the files are processed without my interaction. These processed files are used by the backend to answer requests.
Frontend and backend live separately. This is more of a personal choice than a practical one. I'll be writing the blog in react and I need a backend to query my blogs.
Functions as backend. They are really cheap to run compared to an always on server.
Hosting frontend on Netlify. I have heard good things about netlify and wanted to try it out. Its free tier is great and I can add my custom domain for free.

Going Cloud

Rishabh Gupta — Mon, 06 Aug 2018 07:08:41 +0000

I few weeks back I decided to try out a new workflow for my assignments and projects. This involved doing all my work on a remote server including coding, compiling and checking output.

Reason

The main reason I wanted to try this out was because I hated carrying my laptop to college everyday (its almost 2.7kgs) and wanted a workflow that can be easily ported and can work from anywhere. I also had 100$ digitalocean joining credit which I thought could be put to good use.

Setup

This was an easy part. All I needed was to create a virtual machine and install necessary language tools.
I went with the stable ubuntu 16.04 vm and installed Python, Nodejs and cloned my projects from github.
Now take care to select a vm near to your location, as this will decrease latency and improve the overall experience

First tries : sshfs

I was reluctant to give up the comforts of my favorite editor so I thought I could mount the remote server as a folder and then open those files through text editor. This was obvioulsy going to be faster than using vnc or any other kind of remote desktop protocol.

This worked fine for small assignments but slowed down for projects with a lot of files. I was working on a react project and indexing all modules became a big task. This also meant you needed to install all basic language tools locally and remote server served as nothing more than a network drive.

During this mess of a workflow I also learnt about direnv, a tool to set environment variables specific to each project directory. You should check it out if you haven't already.

A better workflow

Second logical workflow involved using exclusively just the terminal and ssh into the remote server.
Just cloning the repo and installing stuff wasn't going to cut it. Every time I needed to work I had to ssh into machine and navigate to that repo then open a file to edit it. Working on multiple files by opening multiple terminal windows was a nightmare.

Now came TMUX, like a savior from the dark. It was just the solution I needed and more. Honestly anyone who has used tmux knows how awesome it is and I recommend it to everyone to try it out. It is used to create multiple terminals inside a single console window. Taking terminator and applying it inside a console instead of using it on a gui.

Image from tecmint, check out their tmux article for deeper look.

TMUX starts a session and each session contains some windows and each window can be divided into multiple panes. TMUX session don't need to be closed whenever you close a terminal window they can be suspended to background.
This meant I don't even need to start a new session every time I need to work, all I need to do is create a session after I am done cloning the repo. Now whenever I ssh I could just open that session and start working on my project.
This solved a lot of problems, I didn't have to write any script to start a session and my environment was always ready whenever I wanted to start working.

Downsides

There are obviously a lot of downsides to this approach.

You loose any support for gui editors and are locked with terminal editors like vim and emacs.
Some workflows are straight up impossible to shift to cloud like developing GUI apps or working with external devices. These will require a lot of extra setup and might not be optimized and won't be as portable.

Have you tried any interesting workflows you might want to discuss about in the comment section?

Another Cryptocurrency post

Rishabh Gupta — Tue, 27 Mar 2018 08:54:21 +0000

Introduction

In this post I'll share some cryptocurrency concepts. It's an explanation post so we won't dive into code and keep it as general as possible.
We start by a basic concept of passbook: When we do any bank transaction it is recorded in our passbook, it holds all the history to our spendings and earnings.
Passbook is a form of simplified ledger. We start by taking a simple example.

Two friends' example

Now we take two friends A and B who want to transfer fake money to each other but instead of printing their own money they do one thing, they create a shared passbook. This shared passbook stores records of both of them. A transfers money to B by adding a record in his own version and he tells B about his transaction, now B can record that transaction too.
This shared record is a distributed ledger, the basis of blockchain.

Now more people want to trade with them using this fake money so everyone gets that shared passbook. When someone transfers money they inform everyone that they have done so, so that everyone updates their ledger.

Q1 : Where did the starting money come from ? Ans, The game of dice.

Initially, A and B agreed that they will play dice and whoever gets a bigger number will have 10 fake currency. This is also added to ledger.
They agree to do this every hour and hence someone collects 10 fake currency every hour.
Later, anyone who has the ledger has the right to play the game of dice.

This is a basic crypto currency mining scheme. If a lot of people are competing the game of dice is not feasible, hence they are given a problem to solve. Whoever solves the problem first is given the money.

The beauty of this structure is that there is no actual physical notes or any type of asset exchange, it is just a series of transactions.

Q2 : What if someone starts to cheat ?

Let's say B got greedy and wanted extra money, so he added a transaction from A to himself and told everyone A has done so.

To prevent this they agreed to add a signature to the transaction. Much like a signature on the cheque, this acts as a proof telling that the transaction was done by the person himself. Now, B cannot copy A's signatures hence, he cannot create that transaction. Every node checks the signature before adding the transaction to their ledger.

Q3 : Spending money more than once !

This time B got creative at cheating, he had 10 currency at a point in time and he sent that to both A and C. He told everyone (except C) that he sent money to A, and he told C that he sent that money to C. C thought the transaction was a success.

This is the problem of double spending. To prevent double spending participants need a way to differentiate between confirmed and unconfirmed transactions. This is achieved through block chains.

Q4: What is a block chain ?

Block : It is a collection of transactions and contains a hash of the previous block. (hash is a unique number corresponding to a block)

This is similar to linked lists ! Where each block is a node and it's hash is a type of reference.

Every block corresponds to a problem which needs to be solved. This is the mining problem discussed in previous section. Answer of this problem is the hash of the block which is added to the next block. Only a single block is being solved at any point in time.

After a block is solved the transactions it contains are confirmed.

Now no one can change the transaction as this will change the block, which will change the problem and the answer to that problem, causing a chain reaction where the block next to it will change and so on.

Extra Reading:

I find Bitcoin wiki a great resource to understand more concepts.
Bitcoin block explorer for taking a look around.

Lifetimes in Rust

Rishabh Gupta — Sun, 28 Jan 2018 15:31:44 +0000

Introduction

Rust is a system programming language which guarantees memory safety and prevents problems associated with it like data races between threads, memory leaks and dangling pointers by taking radically new approach to memory management. In this article I'd like to discuss about one such approach which they call Lifetimes.

Brief intro to borrowing and ownership

Borrowing, ownership and lifetimes, all three of them are concepts in Rust which complement each other and provide memory guarantees at runtime. To keep our focus on Lifetimes we'll briefly see what others are.

Borrowing

Example: let x = &y
Here x references y and is "almost" similar in functionality to C/C++ references.

Ownership

Rust doesn't have a garbage collector, so it follows a simple rule to clear out allocated memory.

fn foo() {
    let v = [1, 2, 3];
}

When v goes out of scope it's memory will be cleared, even if it is dynamically allocated.

So what is Lifetime ?

In layman terms lifetime of a thing is "the length of time that thing is usable" and it means almost the same in Rust, the "thing" here being "variables".

Take a look at this code, and errors generated by compiler in the comments:

fn main() {
    let r;              
    {
        let i = 1;      
        r = &i;         // borrow occurs here
    }                   // `i` dropped here while still borrowed

    println!("{}", r);
}    // borrowed value needs to live until here

What happens here is:

We declare a variable r
We go inside a scope 2.1. We initialize a variable i 2.2. Assign a reference of i to r
Go out of scope
Use r

At step 3, what happened was that memory for i was freed and r referred to a junk memory location.

The compiler gave us a sweet error message after step 4 saving us from a runtime disaster which C/C++ would have given us.

Okay, but why care ?

Let's see another example:

struct Car {
    model: &str
}

You create a struct, and happily click that compile button, but...

error[E0106]: missing lifetime specifier

model: &str
        ^ expected lifetime parameter

you get this message. You're scratching your head and thinking, "I didn't do anything wrong".

Well, by compiler logic the reference model will be destroyed as soon as the struct scope finishes. So how do you fix it:

struct Car<'a> {
    model: &'a str
}

Notice the 'a, we have explicitly assigned a name to variable's lifetime, then in the statement model: &'a str we tell the compiler to keep the reference around for as long as we have the variable of type Car.

Point to take note is that, lifetimes are always there but most of the times they are implied automatically by the compiler so you don't have to explicitly assign them each time you borrow a reference.

Leveling up

We'll look at another code:

struct Person<'a> {
    name: &'a str,
    do_task: &'a Fn(&str)
}

Here do_task is a reference to a function and it borrows a string reference, "but... we didn't define &'a str". &str means it is tied to the scope of function, hence the string reference will be cleared when the function completes.

While do_task and name are tied to the scope of variable which is of Person type.

Lifetimes is a new and interesting concept, but can be hard to understand the first time. Other resources you can look at are:

Happy coding!!