Forem: Michael Okoko

Travis: Unable to Evaluate Symlinks in Dockerfile path

Michael Okoko — Fri, 25 Oct 2019 10:01:08 +0000

I was following some Docker/Kubernetes tutorial some time back, and all was bright and good until it wasn't. Travis was failing with the error "unable to prepare context: unable to evaluate symlinks in Dockerfile path" and the script causing the build to fail look like this:
docker build -t [secure]/client-test -f client/Dockerfile.dev ./client

It was weird to me because I just used the same command to build and tag the docker image locally, so I tried, again and again to replicate on local - but local worked every damn time.

Now, in most cases when this error happens, it is caused by a wrongly typed folder/file name, or an incorrect file reference, and you can just double-check to get it fixed. But this case was different!

What really happened? Submodules!

I removed the script instruction that was the culprit and watched the logs till I found something interesting, building the particular image for production was failing too, but all others were passing. At this point, it was becoming clear that my headache might have been caused by something around symlinks or so (it should have hit me from the error message anyway but e dey happen). I went back to Github, mindlessly looking at the repo and noticed this:

Ha! My fear was confirmed!

What are Submodules again?

Sub-module is Git's way of enabling you keep a git repository within another while maintaining their commit history separately. It's like having a project within a project. Gitbook has a quite solid explanation on how it works.

So here is how it happened:

create-react-app automatically generated a git repository when it created the application (the client folder in this case), and I only initialized the repo on the parent folder after creating this react app. Somehow, the inner client directory (the react app) then became a submodule, and my naive self thought that deleting the .git folder in that inner folder was enough to get me a smooth ride.

The fix.

Fixing it is as simple as removing the submodule and re-adding the folder to your staging like so:

# delete the submodule reference
git rm --cached submodule_path

# remove the inner .git dir if it still exists
rm -rf submodule_path/.git

# re-add the directory to the main repository
git add submodule_path

# commit and profit
git commit -m "removed git submodule"

and that is all.

Persistent SSH Sessions with Tmux

Michael Okoko — Tue, 26 Mar 2019 05:39:28 +0000

Hello SSH

The SSH (Secure Shell) is a widely used protocol that helps you drop into a remote system, do what you want to do, and get out. Which means, you could check your server logs remotely from an android device, monitor the raspberry pie lying somewhere in your bedroom from your office machine, or go full hacker mode and do stuff like keeping a connection alive when your firewall keeps dropping it or tunneling a network service securely through a remote machine.
But here's the thing. Each time we exit/logout from our remote SSH server, we lose our session and with it, every program that was running. And we don't want that.
That is, we might want an opened program, say an IRC client, a crawler script, anything, to continue running even when we've left it alone, without us starring at black screens with walls of white and green text, but this is where "bare-bones SSH" falls short, and Tmux takes the shine - a little of it anyway.

Meet Tmux

Tmux - which is actually short for "Terminal Multiplexer" - is a tool that lets you create different windows and panes in a single terminal window, while also letting you persist those sessions so you can continue from where you left off the last time (you see what I'm saying?)

To the meat...

Setting up tmux

If you have used tmux and are already comfortable with it, feel free to jump to the next section. If you haven't, it's a fairly straightforward process so let's breeze through it:

You are highly likely to find the tmux package in your linux distro's package repositories so installing it is as easy as:

Ubuntu/Debian

sudo apt install tmux

Archlinux/Archlinux Derivates

sudo pacman install tmux

CentOS

sudo yum install tmux

Now, navigating tmux is a bit more involved and though mouse support exists, it is almost always better to get the hang of a couple of commands. The good thing is all of the commands are configurable which means you get to map them to fit your workflow better. Hackernoon has a solid beginner's guide here to help you find your way around if you're just starting out though.

Ehem...Persistent Sessions.

First, we modify our zsh/bash configuration file (depending on ~~if you have seen the light or not~~ your preferred shell ) to automatically create a new tmux session if there's no available session, or attach itself to an existing one if it exists. So open up the configuration file i.e ~/.bashrc or ~/.zshrc and type the following:

if [ $(tty) == /dev/tty1 ]; then
  /usr/bin/tmux new -s 0
  /usr/bin/tmux attach -t 0
fi

Next, we setup our profile to automatically attach itself to the existing tmux session (labelled '0') so we append the following to our ~/.profile (you'll have to create it if it doesn't exist.):

/usr/bin/tmux attach -t 0

Side Attraction: Termux for Android

Termux is an Android terminal emulator that actually works. It comes with niceties such as not requiring root access and a pretty loaded package repository, as well as been able to compile your own packages on your device. To use SSH on termux is as simple as running pkg install openssh in the app - after installing the termux app of course. Installing the openssh package also set up your SSH keys, which you can install on your remote SSH server and get going.

Now we can hop back into our SSH server, but this time, with an extra coolness.

Cover Image Credit: u/authorice

Setting up a Docker-Based Gitlab Runner on Alibaba Cloud ECS

Michael Okoko — Thu, 18 Oct 2018 05:46:30 +0000

Gitlab offers a couple of amazing features, one of which is its Continous Integration/Delivery tool using GitlabCI. By default, your builds are run on gitlab's servers for free and capped at 2000 minutes across private projects, after which you either pay for an upgraded plan or use your own server.
In this guide, we will set up a private gitlab runner on our Alibaba Cloud ECS instance while hosting our project on gitlab.

Getting Started

To kickstart, let's create an Alibaba Cloud ECS instance (if there's none already) with at least 1GB of RAM and either of Debian, CentOS or Ubuntu as our base OS. For this guide, we will be using Ubuntu version 16.04. If you're totally new here, the Alibaba Cloud documentation has an excellent guide to creating your first ECS instance.
The rest of this guide will be mostly CLI-based so a pretty lack-of-fear-of-the-command-line would be welcomed 😄 .

Hello Docker

Our next step is to install docker on our ECS instance. Since we are using Ubuntu for our guide, installing docker is as simple as running:
apt install docker.io.
Note: We are installing docker.io instead of docker since the later will only install the system tray package on our system.
You can confirm the installation by running docker info which should print out your container information to the terminal.

Installing the Gitlab Runner

Once docker is installed, we will use the runner installation script provided by Gitlab. Currently, it supports Ubuntu, CentOS and Debian (See why we needed one of those OSs now?). So go ahead and type:
curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash to download the script and then proceed with the installation by running:
sudo apt install gitlab-runner
As confirmation, you can run: gitlab-runner --help to display the help information on your terminal.

There is a runner installed on our ECS instance, now what?

As you might know, the runners are used to build and deploy our code from a gitlab project/repository which means, we will have to bind the same runner to our project on gitlab by registering it.

Registering a Gitlab Runner

The gitlab-runner we installed earlier comes with interactive CLI tool to be used for registration but to get the required info, we will need to visit our target gitlab project i.e, the project we are looking to build on our ECS instance. So visit your project's CI settings page which should live at https://gitlab.com/{gitlab-username}/{project-name}/settings/ci_cd (We are assuming it is hosted on gitlab.com) and expand the "Runners" section. Take note of the URL and the registration token on the page (shown in the expansion below) as we'll be using that shortly.

Back to our SSH connection, run:
sudo gitlab-runner register and you'll be presented with a couple of prompts and here is a little break down of each prompt:

Co-ordinator URL: This is the URL as shown in the image. It refers to where our gitlab instance is hosted and for our case, it's https://gitlab.com/.
Runner Token: The registration token on the page, it's used to uniquely identify projects so you will have to copy it from the page and paste on your terminal.
Runner description: Helps to identify the specific runner in cases where we have multiple runners working together, it defaults to our hostname.
Runner tags: Comma-separated values that we can use to delegate jobs from our CI configuration file i.e our gitlab-ci.yml. We'll leave ours blank.
Executor: This is the method used to run our jobs and the options are laid out for us. We will be using docker here.
Docker Image: The default image used when none is specified in our gitlab-ci.yml. Gitlab advises you use a general image and get more specific if required in the CI config file. In the spirit of keeping our image as small as possible, we will use Alpine Linux - a really minimal Linux distro that is designed to be simple and resource efficient, so go ahead and type alpine:latest on the prompt.

An important thing to note here is that our configurations are stored in /etc/gitlab-runner/config.toml so you can always edit the file directly and reload in case you somehow, need to update any of the provided information.

Going back to our project's CI/CD settings page, we should see our shiny new runner added and we can safely disable the shared runners now.

Conclusion

So there! We have configured our Alibaba Cloud ECS instance to handle CI builds for our gitlab project. You may want to register more runners for different projects and it's as easy as the same steps above or you may want to optimize the runner you just created to improve your build times.

This post was originally published on Sitepoint

Demystifying Laravel Queues

Michael Okoko — Thu, 12 Jul 2018 15:45:37 +0000

At some point, your application will need to listen for cues and emit an event for them. Those cues could be birthday or appointment reminders or expiry reminders for users of the trial version of your saas app. Unsurprisingly, Laravel has a built-in way to handle this graciously. Here, we would be sending out reminders to users of our amazing saas app who are on the trial version 15 days before their trial expires.

First of All...Drivers

True to Laravel's reputation for including as many batteries as possible (which can definitely be good or bad), the framework comes with out-of-the-box support for different queue handlers including redis, beanstalk, amazon SQS and of course, relational databases, which we will be using here. There is also an amazing documentation on queue drivers here so feel free to explore them.
To use the database driver, we will need to create a table in our application database to hold the jobs and laravel provides a clean helper command for that. So we will create and run the necessary migrations like so:

php artisan queue:table
php artisan migrate

Diving In

We want to equip our job with the ability of being continuously run in the background so we will create a new artisan command to trigger the task. Artisan commands are laravel's way of extending the already powerful command-line interface and there is an extensive documentation for it living here. Creating a new one is as simple as running:

php artisan make:command SendExpiryCommand

which generates a new class in app/Console/Commands. We will modify the signature and description attribute and the handle() method of this new class before using.
The signature is what we call from the CLI and has the format command:name, - think db:seed, config:clear, etc, as such, let's change our signature to trial_expiry:notify and the description to something actually more descriptive. Now the handle() method is what gets called when we run our shiny command and it is where our code (or most of it anyway) will live. Now, we are going to assume that our app has a Trials model with an expires_in field which holds the date the trial is supposed to expire as well as a user_id field which references our User model. So here is what our final handle() code looks like:

/**
* Execute the console command.
*
* @return mixed
*/
public function handle(){
    //get the date of 15 days from now
    $actualDate = Carbon::now()->addDays(15);
    $actualDate = $actualDate->format('Y-m-d');
    $candidates = Trial::where('expires_in', '=', $actualDate)->get();      
    foreach ($candidates as $candidate) {
        SendExpiryNotification::dispatch($candidate);
    }
}

Our SendExpiryNotification is the actual queue-able job but it doesn't exist yet so let's create that:

php artisan make:job SendExpiryNotification

The above command will create a new class in app/Jobs which implements a ShouldQueue indicating that the job is to be run asynchronously. Don't worry if the Jobs folder doesn't exist yet, artisan will create it if it's not there. In the new class we meet another handle() method which like the command, is the sauce of our class which gets called when we run dispatch and it where our code will live.
Hey there! Now the some of the reasons we are passing a Trial instance instead of say, email, might be obvious but in addition to what is in your head, we want a fresh instance of our eloquent model just before running it from the queue which is exactly what this does. Jacob Bennet has an amazing blog post that details this. Also, our handle method won't take a trial instance as a parameter directly, instead, we feed the instance to the job constructor and call it from the handler.
Enough talk, show me the code! Alright, so here is what our SendExpiryNotification job looks like

<?php

namespace App\Jobs;

use Illuminate\Bus\Queueable;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Support\Facades\Mail;

class SendExpiryNotification implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public $trial;

    public function __construct($candidate) {
        $this->trial = $candidate;
    }

    public function handle()
    {
        $user = User::find($this->trial->user_id);
        Mail::raw('Hello, Your trial would be expiring soon and we wanted you to know!', function($message) {
            $message->to($user->email);
        });
    }
}

...hang in there

Now that the big part is done, let's ~~see~~ make it work. We will start our queue listen with:

php artisan queue:listen

or:

php artisan queue:work --daemon

though general opinion is that the second works much better. So while our queue listeners are still running, we launch our command using the command signature which in this case would be

php artisan trial_expiry:notify

Easy-peasy innit? Oh and by the way, here is laravel's queue documentation if you are that kind of person.

Building for Android SuperUsers: Let's extract WiFi passwords

Michael Okoko — Sun, 05 Nov 2017 19:04:39 +0000

In the beginning was a Manifest permission entry, and big G tagged it android.permission.ACCESS_SUPERUSER, and it was okay. But then Android Lollipop came, and the simple snippet was removed, and Chainfire's superuser library came to the rescue.

What is libsuperuser?

Libsuperuser is an android library that makes requesting and maintaining root access in your app as easy as you could ever imagine. We will be merely implementing it here in this demo app but if you feel like digging deeper (of course you should), you can check out the full HOW-to SU post

What are we building?

We will be creating an app named Fi.Sniff. Android stores it's wireless networks data - SSID, Key, etc - in a file named wpa_supplicant.conf in the /data/misc/wifi/ directory which can only be accessed by the root user. This is the file we will be exploring and parsing once the user gives us root access in the Fi.Sniff app.
Before we get started, Let's make sure we have android studio and the SDKs installed and working as well as a rooted android phone. Emulators won't work here.

Let's get coding!

Open up android studio and create a new android project. Once your project is done and android studio is ready to rock, we need to add two dependencies, the first is the libsuperuser and then, android's recyclerview library to handle our list. So open your app/build.gradle and add the following within the dependencies block:

implementation 'eu.chainfire:libsuperuser:1.0.0.+'
implementation 'com.android.support:recyclerview-v7:26.0.0'

Now getting to the meat of our application, let's edit the layout file associated with our main activity (I am assuming activity_main.xml here) and replace the content with the following:

<?xml version="1.0" encoding="utf-8"?>
<android.support.constraint.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context="com.okmichaels.fisniff.MainActivity">

    <android.support.v7.widget.RecyclerView
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:id="@+id/network_list_recycler_view">

    </android.support.v7.widget.RecyclerView>

</android.support.constraint.ConstraintLayout>

For the above code block as well as all subsequent ones, be sure to replace the package name with your app's. With that done, let's sprinkle some recycler view adapter and its variety of ingredients. But before we can dive into the adapter, let's create a layout that will reflect each of our wifi item on our user's screen. Create a new layout file (network_item_layout) and add the code below:

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:orientation="vertical" android:layout_width="match_parent"
    android:layout_height="wrap_content">

    <TextView
        android:id="@+id/network_ssid_view"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:textSize="14sp"
        android:textStyle="bold"/>

    <TextView
        android:id="@+id/network_psk_view"
        android:layout_width="match_parent"
        android:layout_height="wrap_content" />

</LinearLayout>

For the adapter, we will be creating a new class named NetworkListAdapter as well as an associated inner ViewHolder class (NetworkViewHolder). Our adapter class should look like this when we are done:

package com.okmichaels.fisniff;

import android.content.Context;
import android.support.v7.widget.RecyclerView;
import android.view.LayoutInflater;
import android.view.View;
import android.view.ViewGroup;
import android.widget.TextView;

import java.util.List;

public class NetworkListAdapter extends RecyclerView.Adapter<NetworkListAdapter.NetworkViewHolder> {

    private List<Network> networkList;
    private Context context;

    public NetworkListAdapter(Context ctx, List<Network> list) {
        networkList = list;
        context = ctx;
    }

    public int getItemCount() {
        return networkList.size();
    }

    public NetworkViewHolder onCreateViewHolder(ViewGroup parent, int viewtype) {
        final View view = LayoutInflater.from(parent.getContext())
                .inflate(R.layout.network_item_layout, parent, false);
        return new NetworkViewHolder(view);
    }

    public void onBindViewHolder(NetworkViewHolder holder, int position) {
        Network network = networkList.get(position);
        holder.ssidView.setText(network.getSsid());
        holder.pskView.setText(network.getPsk());
    }

    class NetworkViewHolder extends RecyclerView.ViewHolder
    {
        TextView ssidView;
        TextView pskView;

        public NetworkViewHolder(View itemView) {
            super(itemView);
            ssidView = itemView.findViewById(R.id.network_ssid_view);
            pskView = itemView.findViewById(R.id.network_psk_view);
        }
    }

    public void refreshList(List<Network> networks) {
        this.networkList.clear();
        this.networkList.addAll(networks);
        this.notifyDataSetChanged();
    }
}

PS: If the whole RecyclerView and ViewHolder pattern thingy seems foreign to you, try looking up the official documentation here or this awesome tutorial by
Another PS: If you get squigly red lines beneath the Network class in the above, Farabale - we are coming to that soon.

Next, we need to create a data class (ViewModel in android lingo). Basically, this class reflects the items to be displayed in the adapter so create a new Network class and add the code below:

package com.okmichaels.fisniff;

public class Network {
    private String ssid;
    private String psk;

    public Network() {
        this.ssid = null;
        this.psk = null;
    }

    public void setSsid(String ssid) {
        this.ssid = ssid;
    }

    public void setPsk(String psk) {
        this.psk = psk;
    }

    public Network(String ssid, String key) {
        this.ssid = ssid;
        this.psk = key;
    }

    public String getSsid() {
        return this.ssid;
    }

    public String getPsk() {
        return this.psk;
    }
}

Now to our main activity. This is where stuff gets interesting. You remember that superuser library you added? This is where you get to make it work but hang on! Modify your Main Activity file to look like this first: Android Studio might whine about unused imports blah blah blah, ask it to drink water and proceed.

package com.okmichaels.fisniff;

import android.os.AsyncTask;
import android.support.v7.app.AppCompatActivity;
import android.os.Bundle;
import android.support.v7.widget.LinearLayoutManager;
import android.support.v7.widget.RecyclerView;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

import eu.chainfire.libsuperuser.Shell;

public class MainActivity extends AppCompatActivity {

    public RecyclerView networkListView;
    NetworkListAdapter adapter;
    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        networkListView = findViewById(R.id.network_list_recycler_view);
        networkListView.setLayoutManager(new LinearLayoutManager(getApplicationContext()));
        adapter = new NetworkListAdapter(getApplicationContext(),
                new ArrayList<Network>());
        networkListView.setAdapter(adapter);
    }
}

Now back to the shell. Good news is this is probably the easiest part. Bad news is you don't run SU commands on the main thread plus it returns an ArrayList of Strings where each element is a line in the command's return value. Anyway, let's add it up and be done with right? Add a new AsyncTask child within your main activity and let it mirror the code as shown: (Mine is named NetworkFetcherTask and I've added comments where necessary to shed light on what we are trying to do)

class NetworkFetcherTask extends AsyncTask<Void, Void, List<Network>>
    {
        private boolean suEh;
        @Override
        protected List<Network> doInBackground(Void... voids) {
            List<Network> networks = new ArrayList<>();
            List<String> supplicant;
            /*this is what triggers the SuperUser permission on the device, if the phone is not rooted or the permissioin is denied, it returns false.*/
            suEh = Shell.SU.available();
            if (suEh) {
            /*android stores wifi data in the wpa_supplicant.conf file in the location below, let's try printing it ðŸ™ðŸ™*/
                supplicant = Shell.SU.run("cat /data/misc/wifi/wpa_supplicant.conf");
            /*the return value of SU.run() is an array list, let's flatten it into a string to ease parsing with regex*/
                StringBuilder flattener = new StringBuilder();
                if (supplicant != null) {
                    for (String line: supplicant) {
                        flattener.append(line);
                    }
                }
                String flattened = flattener.toString();

                /*okay, let's find every network block(they exist within curly brackets hence the expression*/
                String exp = "\\{(.*?)\\}";
                Pattern pattern = Pattern.compile(exp);
                Matcher matcher = pattern.matcher(flattened);

                while (matcher.find()) {
                    Network network = new Network();
                    String el = matcher.group();
                    //List<String> lineAsItem = Arrays.asList(el.split(" "));
                    String[] lineAsItem = el.split("\\s+");
                    for(String line: lineAsItem) {

                        if (line.contains("=")) {
                            String[] t = line.split("=");
                            if (t[0].equals("ssid")) {
                                network.setSsid(t[1]);
                            }
                            if (t[0].equals("psk")) {
                                network.setPsk(t[1]);
                            }
                        }
                    }
                    networks.add(network);
                }
            }
            return networks;
        }

        protected void onPostExecute(List<Network> result) {
            adapter.refreshList(result);
        }
    }

One more thing. We need to ask our MainActivity (nicely) to execute this async task so add the following in your MainActivity's onCreate() method to call the task once the activity is setup:

        new NetworkFetcherTask().execute();

Run your application and see the glory!