Forem: Mostafa Gazar

TensorFlow 2.x and Keras must keep snippets

Mostafa Gazar — Sun, 19 Jan 2020 22:28:42 +0000

MNIST classification using Sequential API

# Source: https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/quickstart/beginner.ipynb

import tensorflow as tf
mnist = tf.keras.datasets.mnist

(x_train, y_train),(x_test, y_test) = mnist.load_data()
# Normalize the inputs
x_train, x_test = x_train / 255., x_test / 255.

model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation='softmax')
])

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(x_train, y_train, epochs=5)
model.evaluate(x_test, y_test)

MNIST classification using Functional API

# Source: https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/quickstart/advanced.ipynb

import tensorflow as tf
mnist = tf.keras.datasets.mnist

(x_train, y_train),(x_test, y_test) = mnist.load_data()
# Normalize the inputs
x_train, x_test = x_train / 255., x_test / 255.

class MyModel(tf.keras.Model):
  def __init__(self):
    super(MyModel, self).__init__()
    self.conv1 = Conv2D(32, 3, activation='relu')
    self.flatten = Flatten()
    self.d1 = Dense(128, activation='relu')
    self.d2 = Dense(10, activation='softmax')

  def call(self, x):
    x = self.conv1(x)
    x = self.flatten(x)
    x = self.d1(x)
    return self.d2(x)
model = MyModel()

loss_object = tf.keras.losses.SparseCategoricalCrossentropy()
optimizer = tf.keras.optimizers.Adam()

with tf.GradientTape() as tape:
  logits = model(images)
  loss_value = loss_object(logits, labels)
grads = tape.gradient(loss_value, model.trainable_variables)
optimizer.apply_gradients(zip(grads, model.trainable_variables))

Eager execution

TF 2.0 runs in eager execution mode by default

Eager execution is great for debugging becuse it evaluates operations immediately, without building graphs. Operations return concrete values instead of constructing a computational graph to run later.

Use @tf.function annotation to transform a subset of Python syntax into portable, high-performance TensorFlow graphs.

@tf.function
def increment_even(x):
  if x % 2 == 0:
    x += 1

  return x

Use `ImageDataGenerator` to load raw images in batches with real-time data augmentation

from tensorflow.keras.preprocessing.image import ImageDataGenerator

train_datagen = ImageDataGenerator(
    # Normalize the input
    rescale=1/255,
    # Augmentation specs
    rotation_range=40,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True
)
test_datagen = ImageDataGenerator(
    # Normalize the input
    rescale=1/255
)

train_generator = train_datagen.flow_from_directory(
    some_path,  # Path to your data.
    target_size=(150, 150),
    batch_size=batch_size,  # Your training batch size
    class_mode='categorical'  # You can also `binary` for binary classication tasks.
)

test_generator = test_datagen.flow_from_directory(
    # Use the same parameters as in train_generator.
)

# Finally we train the model
model.fit_generator(
    train_generator,
    steps_per_epoch=len(train_generator),
    epochs=50  # You can also add validation generators here.
)

# Evaluate the model
loss, accuracy = model.evaluate_generator(
    test_generator,
    steps=len(test_generator)
)

Keras data preprocessing utils API

Image related preprocessing utils API
- array_to_img and img_to_array converts a PIL Image instance to a Numpy array and vice-versa.
- load_img loads an image into PIL format.
- save_img saves an image stored as a Numpy array to a path or file object.
- ImageDataGenerator generate batches of tensor image data with real-time data augmentation.
Text related preprocessing utils API
- Tokenizer vectorize a text corpus, by turning each text into either a sequence of integers.
- text_to_word_sequence converts a text to a sequence of words (or tokens).
- pad_sequences pads sequences to the same length.

Text classification using Sequential API

import tensorflow as tf
from tensorflow.keras import datasets
from tensorflow.keras import layers
from tensorflow.keras.models import Sequential
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.preprocessing.sequence import pad_sequences

# Load data
vocab_size = 10000
# If you are loading your own corpus you will need to tokenize it first using `tensorflow.keras.preprocessing.text.Tokenizer.fit_on_texts`
(train_x, train_y), (test_x, test_y) = datasets.reuters.load_data(num_words=vocab_size)

# Pad the sequences so that they would have the same length
max_length = 200
train_x = pad_sequences(train_x, maxlen=max_length)
test_x = pad_sequences(test_x, maxlen=max_length)

# One hot encode y
number_of_classes = len(np.unique(train_y))
train_y = to_categorical(train_y, number_of_classes)
test_y = to_categorical(test_y, number_of_classes)

# Create the model
model = Sequential()
model.add(layers.Embedding(vocab_size, 1024, input_length=max_length))
model.add(layers.GlobalAveragePooling1D())
model.add(layers.Dense(128, activation='relu'))
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(46, activation='softmax'))
model.summary()

# Training...
model.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])
model.fit(x=train_x, y=train_y, validation_split=0.2, shuffle=True, batch_size=512, epochs=20)

# Evaluate the model 
loss, accuracy = model.evaluate(x=test_x, y=test_y)

Tokenize corpus using `Tokenizer.fit_on_texts`

from tensorflow.keras.preprocessing.text import Tokenizer

# Create the tokenizer
tokenizer = Tokenizer()
tokenizer.fit_on_texts(corpus)

# Summary
print(tokenizer.word_counts)
print(tokenizer.document_count)
print(tokenizer.word_index)
print(tokenizer.word_docs)

Load pre-trained embeddings

from pathlib import Path

vocab_size = 10000
embedding_size = 100

# Build your model
model = Sequential()
embedding_layer = layers.Embedding(vocab_size, embedding_size, input_length=max_length)
model.add(embedding_layer)
# Add more layers...

# Download pre-trained word embeddings
# I will use the smallest available pre-trained word vectors from [GloVe](https://nlp.stanford.edu/projects/glove/) 
# which should be more than enough. That was glove.6B.zip which consists of 6B tokens, 400K vocab, uncased, 50d, 100d, 200d, & 300d vectors. 
# It is an 822 MB download.
embedding_path = Path("../embedding")
embedding_path.mkdir(parents=True, exist_ok=True)

!wget -O {embedding_path}/glove.6B.zip http://nlp.stanford.edu/data/glove.6B.zip
!unzip {embedding_path}/glove.6B.zip -d {embedding_path}

# Load embeddings
embedding_index = {}
with open(embedding_path/"glove.6B.100d.txt", "r") as reader:
    line = reader.readline()
    while line != '':  # The EOF char is an empty string
        values = line.split()
        word = values[0]
        coefs = np.asarray(values[1:], np.float32)
        embedding_index[word] = coefs

        line = reader.readline()

# Set embedding_layer weights
embedding_matrix = np.zeros((vocab_size, embedding_size))
for word, index in word_index.items():
    if index >= vocab_size:
        continue

    coefs = embedding_index.get(word)
    if coefs is not None:
        embedding_matrix[index] = coefs

embedding_layer.set_weights([embedding_matrix])
embedding_layer.trainable = False

# TODO :: Compile model here for trainable change to take effect
# model.compile()

Plot training accuracy and loss after training is done

import matplotlib.pyplot as plt
%matplotlib inline

# Train model.
history = model.fit(x=train_x, y=train_y, validation_split=0.2, shuffle=True, batch_size=128, epochs=10)
print(history.history.keys())

# Plot training progress.
fig = plt.figure(figsize=(20, 10))

plt.subplot(1, 2, 1)
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('Model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')

plt.subplot(1, 2, 2)
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('Model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')

plt.show()

Track training progress in TensorBoard

from tensorflow import keras

callbacks = [
    keras.callbacks.TensorBoard(log_dir='../logs/dogs_vs_cats_v2')
]

history = model.fit_generator(
    ...
    callbacks=callbacks,
    ...
)

# Then check progress in TensorBoard by running `tensorboard --logdir=logs/dogs_vs_cats_v2` in terminal

Dynamic learning rate (learning rate schedules)

from tensorflow import keras

initial_learning_rate = 0.1
learning_rate_schedule = keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate,
    decay_steps=100000,
    decay_rate=0.96,
    staircase=True
)

optimizer = keras.optimizers.RMSprop(learning_rate=learning_rate_schedule)

Another option would be to us ReduceLROnPlateau callback. ReduceLROnPlateau reduces learning rate when a metric stop improving.

from tensorflow import keras

reduce_lr = keras.callbacks.ReduceLROnPlateau(
    monitor='val_loss', 
    factor=0.1,  # Divide the learning rate by 10 when triggered
    patience=5,  # Trigger when val_loss has stopped improving for 5 epochs 
    min_lr=0.001
)

model.fit(X_train, Y_train, callbacks=[reduce_lr])

Handy callbacks

from tensorflow import keras

callbacks = [
    # Stop the training when there is no improvement in the validation loss for three consecutive epochs.
    keras.callbacks.EarlyStopping(
        monitor='val_loss', 
        patience=3
    ),
    # Save model after every epoch or save the best one so far only 
    keras.callbacks.ModelCheckpoint(
        file_path="_model.h5",
        monitor="val_loss",
        save_best_only=True
    ),
    # Check training progress in TensorBoard
    keras.callbacks.TensorBoard(
        log_dir="model_log_dir",
        histogram_freq=1,  # Records activation histograms every 1 epoch
        embedding_freq = 1  # Record emedding data every 1 epoch. If set to 0, embeddings won't be visualized.
    )
]

model.fit(X_train, Y_train, callbacks=callbacks)

Save model for inference

model.save('path_to_my_model.h5')

# Load the model for inference
model = keras.models.load_model('path_to_my_model.h5')

model.trainable = False
# TODO :: Compile model here for trainable change to take effect
# model.compile()

# Make sure this model does not have trainable variables
model.trainable_variables

Save model checkpoints and continue training where you left off

# Source: https://www.tensorflow.org/alpha/guide/checkpoints
import tensorflow as tf

opt = tf.keras.optimizers.Adam(0.1)
net = Net()

ckpt = tf.train.Checkpoint(step=tf.Variable(1), optimizer=opt, net=net)
manager = tf.train.CheckpointManager(ckpt, 'tf_ckpts', max_to_keep=3)
ckpt.restore(manager.latest_checkpoint)
if manager.latest_checkpoint:
    print(f'Restored from {manager.latest_checkpoint}')
else:
    print('Initializing from scratch.')

for example in toy_dataset():
    loss = train_step(net, example, opt)
    ckpt.step.assign_add(1)
    if int(ckpt.step) % 10 == 0:
        save_path = manager.save()
        print(f'Saved checkpoint for step {int(ckpt.step)}: {save_path}')
        print(f'loss {loss.numpy():1.2f}')

I am working on a project called ML Studio, want to get early access to and product updates? Subscribe here or follow me on twitter.

Some Pandas handy snippets for Data Scientists

Mostafa Gazar — Sun, 19 Jan 2020 02:14:09 +0000

Iterate through rows

import pandas as pd

for index, row in df.iterrows():
    pass

Count unique values in dataframe

df.labels.value_counts()

Style max value in a row or column

# Inspired by https://stackoverflow.com/a/45606572/2874139
def highlight_max(data, color='yellow', isBold=True):
    # Styling
    attrs = []
    if color is not None:
        attrs.append(f'background-color: {color}')
    if isBold:
        attrs.append('font-weight: bold')
    attrs = '; '.join(attrs)

    if data.ndim == 1:
        is_max = data == data.max()
        return [attrs if value else '' for value in is_max]
    else:
        is_max = data == data.max().max()
        return pd.DataFrame(np.where(is_max, attrs, ''), index=data.index, columns=data.columns)

df.style.apply(highlight_max, axis=1) # Max in row
df.style.apply(highlight_max, axis=0) # Max in column

Display 1000 rows and columns

# source: fast.ai material
def display_all(df):
    with pd.option_context("display.max_rows", 1000, "display.max_columns", 1000): 
        display(df)

display_all(df)

Save dataframe as CSV file

# index specifies whether to add a sequential index to the saved file
df.to_csv(csv_path, index=False)

Create dataframe form python dictionary

all_questions = []  # rows of column 'all_questions'
all_good_answers = []  # rows of column 'all_good_answers'
all_bad_answers = []  # rows of column 'all_bad_answers'

qa_dict = {'question': all_questions, 'good_answer': all_good_answers, 'bad_answer': all_bad_answers}

# Create a dataframe with 3 columns: question, good_answer and bad_answer
df = pd.DataFrame(data=qa_dict)

Parse dates in dataframe

df = pd.read_csv("train.csv", low_memory=False, parse_dates=["createddate"])

I am working on a project called ML Studio, want to get early access to and product updates? Subscribe here or follow me on twitter.

Some Python handy snippets for Machine Learning Engineers

Mostafa Gazar — Sun, 19 Jan 2020 01:33:50 +0000

Get file name and extension

from pathlib import Path

path = Path("dir/file.xml")

file_name = path.stem  # file
file_extension = path.suffix  # .xml

Iterate through directory contents

from pathlib import Path

data_root = Path("/imgs")
for item in data_root.iterdir():
    print(item)

You can also use glob

from pathlib import Path

data_root = Path("/imgs")

# Get all png images in this directory level
data_root.glob("*.png")
# Get all png images in this directory including sub-directories
data_root.glob("*.png", recursive=True)

Inverse dictionary key/value => value/key

inversed_dict = {v: k for k, v in some_dict.items()}

Flatten a list of lists

import operator

flat_list = reduce(operator.concat, some_list)

Map string list to int list

elements = ["1", "2", "3"]

elements = list(map(int, elements))

Count unique values in list and their total

from collections import Counter

elements = ["1", "2", "3", "2", "3", "3"]

# Unique values in list
keys = Counter(elements).keys()

# Frequency of their appearance
values = Counter(elements).values()

Sort dict by value

for key in sorted(confidences, key=confidences.get, reverse=True):
    print(f'key {key}')

Format f strings

print(f'2 decimal points {some_variable:.2f}')

Batch an iterable

def batch(iterable, n=1):
    l = len(iterable)
    for index in range(0, l, n):
        yield iterable[index:min(index + n, l)]

# and call it like
for x, pred in enumerate(batch(preds, 32), 1):
    pass

I am working on a project called ML Studio, want to get early access to and product updates? Subscribe here or follow me on twitter.

Beautiful Android rounded buttons

Mostafa Gazar — Wed, 15 Jan 2020 21:57:04 +0000

Let us say the designer in your team handed you this beautifully custom button to implement.

It looks nice, it flat, not much to it really. Should be easy to implement.

First you create a StateListDrawable , easy!

<?xml version="1.0" encoding="utf-8"?>
<selector xmlns:android="http://schemas.android.com/apk/res/android">

    <item android:state_pressed="true">
        <shape android:shape="rectangle">

            <solid android:color="@color/color_dhuhr_background_pressed" />

            <corners
                android:topRightRadius="@dimen/default_event_end_corner"
                android:bottomRightRadius="@dimen/default_event_end_corner"
                android:topLeftRadius="@dimen/default_event_start_corner"
                android:bottomLeftRadius="@dimen/default_event_start_corner"/>

        </shape>
    </item>

    <item>
        <layer-list>
            <item>
                <shape android:shape="rectangle">
                    <solid android:color="@color/color_dhuhr_background" />

                    <corners
                        android:bottomLeftRadius="@dimen/default_event_start_corner"
                        android:bottomRightRadius="@dimen/default_event_end_corner"
                        android:topLeftRadius="@dimen/default_event_start_corner"
                        android:topRightRadius="@dimen/default_event_end_corner" />

                </shape>
            </item>

            <item android:gravity="left">
                <shape android:shape="rectangle">
                    <solid android:color="@color/color_dhuhr_background_pressed" />

                    <size android:width="4dp" />
                </shape>
            </item>
        </layer-list>
    </item>

</selector>

You run the app and you get this

Did you notice the ugly shadows?

This must be happening because of elevation or z index.

An easy solution would be to use borderless button style, Widget.AppCompat.Button.Borderless. This would mean though that that button will not raise when the user taps it.

I do not think that looks great.

There is another solution, which is to use Widget.AppCompat.Button and override stateListAnimator with our own animation.

<style name="Button.Event" parent="Widget.AppCompat.Button">
    <item name="android:stateListAnimator">@animator/button_event_animator</item>
</style>

button_event_animator.xml

<?xml version="1.0" encoding="utf-8"?>
<selector xmlns:android="http://schemas.android.com/apk/res/android">

    <item android:state_pressed="true">
        <set>
            <objectAnimator android:propertyName="translationZ"
                            android:duration="@integer/button_event_pressed_animation_duration"
                            android:valueTo="@dimen/button_event_pressed_z"
                            android:valueType="floatType"/>
            <objectAnimator android:propertyName="elevation"
                            android:duration="0"
                            android:valueTo="@dimen/button_event_pressed_elevation"
                            android:valueType="floatType"/>
        </set>
    </item>

    <item>
        <set>
            <objectAnimator android:propertyName="translationZ"
                            android:duration="0"
                            android:valueTo="0"
                            android:valueType="floatType"/>
            <objectAnimator android:propertyName="elevation"
                            android:duration="0"
                            android:valueTo="0"
                            android:valueType="floatType"/>
        </set>
    </item>

</selector>

The end result would look

This look like something intractable.

Thanks for reading! Stay updated.

Build docker images inside of Kubernetes

Mostafa Gazar — Wed, 15 Jan 2020 21:26:05 +0000

There are many reasons why we might want to build docker images inside of a Kubernetes cluster. Reasons are out of scope of this post but one reason might be to integrate them in you CI pipelines.

Build and push docker images inside of k8s as you please, on the other hand running containers can be very problematic. It is problematic because k8s will not know about these containers and will not be able to manage them. This just defeats the purpose of using k8s in the first place.

That said if you really need to run unmanaged containers on your own, consider creating a separate node pool just for that purpose.

Let us talk requirements, our requirements are basic:

We want to be able to build docker images in a specific pod.
Push images to a local registry or docker hub.
Caching would be nice-to-have, otherwise each are build would take a long time.
Finally, preferably we will use a daemonless docker build tool instead of installing docker inside a container and mount /var/run/docker.sock

A google search for "Daemonless docker builds" will return a few results like:

buildah
Buildah's commands replicate all of the commands that are found in a Dockerfile. This allows building images with and without Dockerfiles while not requiring any root privileges. Buildah follows a simple fork-exec model and does not run as a daemon but it is based on a comprehensive API in golang, which can be vendored into other tools. *

Here is a list of the commands relevant to this post:
```
# Build an image using instructions from Dockerfiles.
buildah bud  

# Push an image from local storage to elsewhere.
# `--tls-verify=false` is used to allow using insecure registries if required.
buildah push  

# List all images built by buildah.
buildah images  
```
More examples are available here
img: https://github.com/genuinetools/img
Kaniko: https://github.com/GoogleContainerTools/kaniko
dind (Docker in Docker): https://github.com/jpetazzo/dind
buildkit: https://github.com/moby/buildkit/tree/master/examples/kubernetes
Sysbox: https://github.com/nestybox/sysbox-external
Knative build (deprecated): https://starkandwayne.com/blog/build-docker-images-inside-kubernetes-with-knative-build/

After all the research, I landed on using buildah with an insecure registry. Here are a few commands I used for testing

# debug building a dockerfile
buildah --debug bud -f Dockerfile -t mostafagazar/test .

# Without build args
buildah bud -f Dockerfile -t mlstudio-registry.default.svc.cluster.local:5000/test:v0 .
buildah push --tls-verify=false mlstudio-registry.default.svc.cluster.local:5000/test:v0

# With build args
buildah bud --build-arg model_name=name -f Dockerfile -t mlstudio-registry.default.svc.cluster.local:5000/test2:v0 .
buildah push --tls-verify=false mlstudio-registry.default.svc.cluster.local:5000/test2:v0

Want to get early access to ML Studio and product updates? Subscribe here or follow me on twitter

Use cfssl in a multi-stage build docker image

Mostafa Gazar — Tue, 31 Dec 2019 02:00:43 +0000

CFSSL is Cloudflare's PKI and TLS toolkit. I recently wanted to use to generate a self signed certificate in a docker file.

You can technically install cfssl in a multitude of ways in whatever docker image you are using. There is a simpler approach though using Docker's multi-stage builds.

Let us say you have a registry image, code below:

FROM registry

LABEL maintainer="mostafa@mlstudioapp.com"

And to use cfssl, you can do something like:

FROM cfssl/cfssl AS cfssl

WORKDIR /

# Generate certificate
COPY ca-csr.json /
RUN cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
RUN rm /ca-key.pem

FROM registry

LABEL maintainer="mostafa@mlstudioapp.com"

# Copy it to the registry
COPY --from=cfssl /ca.csr /certs/selfsigned.crt
COPY --from=cfssl /ca.pem /certs/selfsigned.key

ca-csr.json

{
  "CN": "ML Sutdio CA",
  "hosts": [
    "mlstudio-registry.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [{
    "C": "NZ",
    "O": "ML Studio"
  }]
}

If you found this helpful spread the word.

How do you get your Machine Learning models from idea to production

Mostafa Gazar — Thu, 05 Dec 2019 18:06:32 +0000

To all the data scientists and Machine Learning Engineers out there, building and deploying ML models to production environments. I need your help to understand your typical pipeline or workflow.

Can you describe what your role is? What steps and tools do you use to build your ML models? And how do you deploy them to production?

I imagine other people would be interested in your answers too so do not be shy. And If you feel generous and have 3 minutes to spare please fill-in this survey. Many thanks to the awesome dev.to community ❤️

Monitor for any new conda env and add them as Jupyter kernels

Mostafa Gazar — Tue, 03 Dec 2019 23:54:48 +0000

Here is a little script that monitors the conda envs directory on linux and adds a kernel in Jupyter for each new env.

Alternatively you probably should use nb_conda_kernels instead.

#!/bin/bash

# Inspired by: https://unix.stackexchange.com/a/323919
# man page: https://linux.die.net/man/1/inotifywait
inotifywait -m /opt/conda/envs -e create -e moved_to |
    while read path action file; do
      if [[ "$file" == .* ]]; then # Does start with .
        echo "$file ignored"
      else
        # Wait for the env to get created
        echo "Waiting for $path$file/bin/python ..."
        while [ ! -f "$path$file/bin/python" ]; do sleep 1; done

        # Add the new conda env
        echo "Add $file kernel"
        source activate "$file" && \
        pip install --quiet ipykernel && \
        python -m ipykernel install --user --name "$file"
      fi
    done

If you found this helpful spread the word.

Train a lines segmentation model using Pytorch

Mostafa Gazar — Sat, 23 Nov 2019 18:03:42 +0000

Let us start by identifying the problem we want to solve which is inspired by this project.

Given an image containing lines of text, returns a pixelwise labeling of that image, with each pixel belonging to either background or line of handwriting.

The project structure

It consists of 5 main sections, one for notebooks, one for the shared python code, datasets, Google Cloud scripts and one for saving the model weights.

In a production project, you will probably have more directories like web and api.

I also chose to use pipenv instead of conda and virtualenv to manage my python environment. I only recently switched to pipenv from conda and I found it to consistently work as expected everywhere.

For GPU training, I used a google cloud instance with one T4 Nvidia GPU. Bash scripts manage the instance lifecycle, from creating it initially to starting it, connecting to it and stopping it.

Data

The dataset is described in a toml file inside the raw directory, a toml file basically consists of key, value pairs. The other directories under data are git ignored because they will contain the actual full datasets downloads.

Notebooks

I use notebooks for exploration and as a high-level container for the code required to construct, clean datasets and build a training basic pipeline.

Python files

Under the src directory I keep the code that can be shared and reused between various notebooks. Following good Software Engineering practices is a key to get things done quickly and correctly, finding and identifying bugs in ML code can be extremely hard. That is why you would want to start small and reiterate often.

The python environment

You can install pipenv on Linux or mac using linuxbrew or macbrew with the following command:

brew install pipenv

And then you can download your dependencies using pipenv install SOMETHING from your project directory.

The dataset

I will use this old academic dataset here as a base to build a lines segmentation dataset to train a UNet mini-network to detect lines of handwriting.

The original images in the dataset look like the following, they also come with XML files that define the bounding boxes.

In notebooks/01-explore-iam-dataset.ipynb I downloaded the dataset, unzipped it and then overplayed some random images with the data from the XML file.

Next, I cropped the images and generated masks images to match the new dimensions. The mask images are the ground truth images that we will use for training the final model.

Finally, I split the data into train, valid and test

The Network

Because we do not have a lot of data available for training, I used a mini version of the UNet architecture based on this Keras implementation.

And using this great library I can visualize the network by doing a feedforward with a specific input size.

The Training Pipeline

Now that we have the data ready and the network that we want to train defined, it is time to build a basic training pipeline.

First is defining a torch dataset and iterate through it using a DataLoader

from torch.utils.data import Dataset, DataLoader
from torchvision import transforms, utils


class FormsDataset(Dataset):

    def __init__(self, images, masks, num_classes: int, transforms=None):
        self.images = images
        self.masks = masks
        self.num_classes = num_classes
        self.transforms = transforms

    def __getitem__(self, idx):
        image = self.images[idx]
        image = image.astype(np.float32)
        image = np.expand_dims(image, -1)
        image = image / 255
        if self.transforms:
            image = self.transforms(image)

        mask = self.masks[idx]
        mask = mask.astype(np.float32)
        mask = mask / 255
        mask[mask > .7] = 1
        mask[mask <= .7] = 0
        if self.transforms:
            mask = self.transforms(mask)

        return image, mask

    def __len__(self):
        return len(self.images)

train_dataset = FormsDataset(train_images, train_masks, number_of_classes, get_transformations(True))
train_data_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
print(f'Train dataset has {len(train_data_loader)} batches of size {batch_size}')

Next, I define the training loop

# Use gpu for training if available else use cpu
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

# Here is the loss and optimizer definition
criterion = torch.nn.NLLLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# The training loop
total_steps = len(train_data_loader)
print(f"{epochs} epochs, {total_steps} total_steps per epoch")

for epoch in range(epochs):
    for i, (images, masks) in enumerate(train_data_loader, 1):
        images = images.to(device)
        masks = masks.type(torch.LongTensor)
        masks = masks.reshape(masks.shape[0], masks.shape[2], masks.shape[3])
        masks = masks.to(device)

        # Forward pass
        outputs = model(images)
        softmax = F.log_softmax(outputs, dim=1)
        loss = criterion(softmax, masks)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i) % 100 == 0:
            print (f"Epoch [{epoch + 1}/{epochs}], Step [{i}/{total_steps}], Loss: {loss.item():4f}")

Here are the final predictions

You can check a Keras backed by TF2 implementation here.

Thanks for making it this far. The last thing I would like to say is that unfortunately, most of the available online materials either offer bad advice or are very basic that they do not actually offer much value and some are plain wrong. There are some great resources though like their 60-minute blitz series and great API docs. There is also this cheat sheet and this great GitHub repo.

If you enjoyed reading this post and found it helpful I would love to hear from you, my Twitter DMs are open.

Jupyter notebooks tips and tricks

Mostafa Gazar — Tue, 19 Nov 2019 01:26:29 +0000

Shortcuts

Shift + Enter to run a cell (code or markdown).
A to insert a new cell above the current cell.
B to insert a new cell below the current cell.
M to change the current cell to Markdown
Y to change the current cell to code.
D + D (twice) to delete the selected cells.

Magic commands

Time the execution of a single line of code or the whole cell

# Run the code multiple times and find mean runtime
%timeit CODE_LINE
%%timeit CODE_CELL

# Run once and report
%time CODE_LINE
%%time CODE_CELL

Use ! prefix to run a single bash command line
%%bash to change the current code cell to run to bash mode, basically writing bash commands in there

%%bash

echo "hello from $BASH"

%%js, %% html, %%latex, %%python2, %%python3, ... run and render code cells in that specified language or format.
autoreload IPython extension is exceptionally helpful when you do not want to worry about reloading modules before executing new code. In other words when you change something in a certain module the current notebook uses, changes will take place when you run new code cells without having to worry about anything.

%load_ext autoreload
%autoreload 2

Embedd tensorboard in jupyter notebook

%load_ext tensorboard

%tensorboard --logdir logs/model_training_logs

Finally you can list all available magics by running %lsmagic, this will show both line and cell magics currently defined.

Other

Sometimes you will have this memory hungry variable, you can reclaim memory by setting it to NONE and then forcing gc to run

some_var = None
gc.collect()

Use sudo service jupyter restart to restart jupyter because every once in a while jupyter would throw a fit and restarting kernels will not be enough to get it back to a responsive state.
Add ? before almost any function, variable, ... and run the code cell to access its documentation.
tqdm which means "progress" in Arabic (taqadum, تقدّم) is not really related to jupyter notebooks but it can be used to show a smart progress meter. Just by wrapping any iterable with tqdm(iterable)

from tqdm import tqdm

for i in tqdm(range(10000)):
    pass

When you want to count the number of files in a directory you can run the following command

!ls DIR_NAME | wc -l

Classic notebooks extensions

There are many great extensions in jupyter_contrib_nbextensions. You should be using Jupyter lab though instead.

First you need to instead jupyter_contrib_nbextensions and then you can install various useful extensions.

pip install jupyter_contrib_nbextensions
jupyter contrib nbextension install --user

These are the ones I love:

code_prettify backed by autopep8 is great for reformatting code in notebook code cells base on PEP 8 style guide

pip install autopep8
jupyter nbextension enable code_prettify/autopep8

spellchecker highlights incorrectly spelled words in Markdown cells which saved me from a few embarrassing typos.

jupyter nbextension enable spellchecker/main

toggle_all_line_numbers as its names suggests it adds a toolbar button to toggle between showing line numbers or not

jupyter nbextension enable toggle_all_line_numbers/main

varInspector is great for debugging python and R kernels. It displays the values of all defined variables in a floating window

jupyter nbextension enable varInspector/main

Themes

dunovank/jupyter-themes has one of the best themes I came across. I tried it and then I stopped using it because I switch environments all the time so it was best for me to get used to the stock theme.

pip install jupyterthemes

# dark
jt -t onedork -fs 95 -altp -tfs 11 -nfs 115 -cellw 88% -T

# light
jt -t grade3 -fs 95 -altp -tfs 11 -nfs 115 -cellw 88% -T

# Restore default theme
jt -r

Jupyter lab extensions

There are two extensions that I only use at the moment

krassowski/jupyterlab-go-to-definition which allow me to use Alt + click to jump to a definition using your mouse, or Ctrl + Alt + B keyboard-only alternative.

jupyter labextension install @krassowski/jupyterlab_go_to_definition

krassowski/jupyterlab-lsp adds support for code navigation + hover suggestions + linters + autocompletion. Check out their full list of supported language servers

pip install --pre jupyter-lsp
jupyter labextension install @krassowski/jupyterlab-lsp

conda install -c conda-forge python-language-server

Finally you need to rebuild the jupyter lab app

jupyter lab build

Themes

There are many themes out there, the first customization plugin in my list is not a theme though. It is a topbar extension to quickly switch between light and dark themes

jupyter labextension install jupyterlab-topbar-extension jupyterlab-theme-toggle

Here is a list a few themes I used recently

jupyter labextension install @telamonian/theme-darcula
jupyter labextension install @rahlir/theme-gruvbox
jupyter labextension install @kenshohara/theme-nord-extension

Share it and get it in touch, my Twitter DMs are open!

Try Jupyter Hub on GKE with dummy authentication

Mostafa Gazar — Mon, 11 Nov 2019 21:21:35 +0000

Create a new cluster

export cluster_name=jhub-cluster
export cluster_zone=us-central1-a

gcloud container clusters create $cluster_name \
    --machine-type=n1-standard-2 \
    --num-nodes 2 \
    --enable-autoscaling --min-nodes 0 --max-nodes 6 \
    --zone $cluster_zone

Set kubectl locally

gcloud container clusters get-credentials $cluster_name --zone $cluster_zone

Install helm and set it up

brew install kubernetes-helm

Setup Tiller Service account

# Make tiller an admin, do not do so in production
kubectl create clusterrolebinding user-admin-binding \
   --clusterrole=cluster-admin \
   --user=$(gcloud config get-value account)
kubectl create serviceaccount tiller --namespace kube-system
kubectl create clusterrolebinding tiller-admin-binding \
   --clusterrole=cluster-admin \
   --serviceaccount=kube-system:tiller

helm init --service-account=tiller
helm repo update

# Ensure that tiller is secure from access inside the cluster
kubectl patch deployment tiller-deploy --namespace=kube-system --type=json --patch='[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value": ["/tiller", "--listen=localhost:44134"]}]'

Verify helm and tiller were installed properly

By checking the client and server versions. You might need to wait a minute or two while the tiller pod is in the running state.

helm version

You should get a matching client and server versions like:

Make Helm aware of the JupyterHub Helm chart repository

So you can install the JupyterHub chart from it, https://zero-to-jupyterhub.readthedocs.io/en/latest/setup-jupyterhub.html

helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/
helm repo update

Create JupyterHub config file

nano config.yaml

# proxy:
#   secretToken: "eb4e1c0f2a77bacc63c1a1ada95507f5b551d53e18b204df30bb23966>
#
# auth:
#   type: dummy
#   dummy:
#     password: 'password'
#   whitelist:
#     users:
#       - user1
#       - user2
#       - admin

Now install the chart configured by your config.yaml

RELEASE=jhub
NAMESPACE=jhub

helm upgrade --install $RELEASE jupyterhub/jupyterhub \
  --namespace $NAMESPACE  \
  --version=0.8.2 \
  --values config.yaml

# Get the external IP to access JupyterHub
kubectl get service --namespace jhub

Share it and get it in touch, my Twitter DMs are open!

Playing local media files and stream media on Android using ExoPlayer

Mostafa Gazar — Thu, 07 Nov 2019 22:05:57 +0000

We have 2 options if you want to play media files on Android:

Android’s MediaPlayer APIs or
ExoPlayer

ExoPlayer is easier to work with and it supports features currently not supported by MediaPlayer APIs. The main downside in using ExoPlayer according to its docs is that:

For audio only playback on some devices, ExoPlayer may consume significantly more battery than MediaPlayer. (1)

Before we start coding let us think about the main cases we want to handle:

The first obvious case is to maintain continuous playback regardless of whether the app is in the foreground or the background. Which translates into using a Service.
And because of the new restrictions on running background services that were introduced in API level 26 and higher we need to make sure that we use a foreground Service otherwise our Service could get killed by the system unexpectedly.

A foreground Service performs some operation that is noticeable to the user. For example, an audio app would use a foreground Service to play an audio track. Foreground Services must display a Notification. Foreground services continue running even when the user isn't interacting with the app. (2)

Given that we will use a foreground Service to manage the playback we need to tie it with a Notification. We can build one but we do not have to, ExoPlayer's PlayerNotificationManager can create one for us in a few lines of code and it will come with a familiar design to our users.
Although Android can play media from multiple different sources (say different apps playing audio simultaneously), in most cases that will not be a very good end-user experience. So Android introduced this concept of audio focus, only one app can hold audio focus at a time. With a few lines of code ExoPlayer can handle audio focus for us.
Another case we would need to worry about is allowing clients and connected devices (like Google Assistant, Android Auto, Android TV, and Android Wear) to manage the media being played. Luckily we can delegate that to Android's MediaSession API.

Now that we know the desired behavior, let us start coding.

Create a LifecycleService or a regular Service and assuming we want to play a single media file for simplicity, pass in its title, uri, and start position.

class AudioService : LifecycleService() {

    companion object {
        @MainThread
        fun newIntent(context: Context, title: String, uriString: String, startPosition: Long) = Intent(context, AudioService::class.java).apply {
            putExtra(ARG_TITLE, title)
            putExtra(ARG_URI, Uri.parse(uriString))
            putExtra(ARG_START_POSITION, episodeDetails.listened?.startPosition)
        }
    }
...

Initialize the ExoPlayer and move the Service to the foreground when playing starts and back to the background when playback stops for whatever reason.

private var episodeTitle: String? = null

private lateinit var exoPlayer: SimpleExoPlayer

private var playerNotificationManager: PlayerNotificationManager? = null
private var mediaSession: MediaSessionCompat? = null
private var mediaSessionConnector: MediaSessionConnector? = null

private const val PLAYBACK_CHANNEL_ID = "playback_channel"
private const val PLAYBACK_NOTIFICATION_ID = 1

private const val ARG_URI = "uri_string"
private const val ARG_TITLE = "title"
private const val ARG_START_POSITION = "start_position"


override fun onCreate() {
    super.onCreate()

    exoPlayer = ExoPlayerFactory.newSimpleInstance(this, DefaultTrackSelector())
    val audioAttributes = AudioAttributes.Builder()
            .setUsage(C.USAGE_MEDIA)
            .setContentType(C.CONTENT_TYPE_SPEECH)
            .build()
    exoPlayer.setAudioAttributes(audioAttributes, true)

    // Setup notification and media session.
    playerNotificationManager = PlayerNotificationManager.createWithNotificationChannel(
            applicationContext,
            PLAYBACK_CHANNEL_ID,
            R.string.playback_channel_name,
            PLAYBACK_NOTIFICATION_ID,
            object : PlayerNotificationManager.MediaDescriptionAdapter {
                override fun getCurrentContentTitle(player: Player): String {
                    // return title
                }

                @Nullable
                override fun createCurrentContentIntent(player: Player): PendingIntent? = PendingIntent.getActivity(
                        applicationContext,
                        0,
                        Intent(applicationContext, MainActivity::class.java),
                        PendingIntent.FLAG_UPDATE_CURRENT)

                @Nullable
                override fun getCurrentContentText(player: Player): String? {
                    return null
                }

                @Nullable
                override fun getCurrentLargeIcon(player: Player, callback: PlayerNotificationManager.BitmapCallback): Bitmap? {
                    return getBitmapFromVectorDrawable(applicationContext, R.drawable.vd_app_icon)
                }
            },
            object : PlayerNotificationManager.NotificationListener {
                override fun onNotificationStarted(notificationId: Int, notification: Notification?) {
                    startForeground(notificationId, notification)
                }

                override fun onNotificationCancelled(notificationId: Int) {
                    _playerStatusLiveData.value = PlayerStatus.Cancelled(episodeId)

                    stopSelf()
                }

                override fun onNotificationPosted(notificationId: Int, notification: Notification?, ongoing: Boolean) {
                    if (ongoing) {
                        // Make sure the service will not get destroyed while playing media.
                        startForeground(notificationId, notification)
                    } else {
                        // Make notification cancellable.
                        stopForeground(false)
                    }
                }
            }
    ).apply {
        // Omit skip previous and next actions.
        setUseNavigationActions(false)

        // Add stop action.
        setUseStopAction(true)

        setPlayer(exoPlayer)
    }

    ...
}

@MainThread
private fun getBitmapFromVectorDrawable(context: Context, @DrawableRes drawableId: Int): Bitmap? {
    return ContextCompat.getDrawable(context, drawableId)?.let {
        val drawable = DrawableCompat.wrap(it).mutate()

        val bitmap = Bitmap.createBitmap(drawable.intrinsicWidth, drawable.intrinsicHeight, Bitmap.Config.ARGB_8888)
        val canvas = Canvas(bitmap)
        drawable.setBounds(0, 0, canvas.width, canvas.height)
        drawable.draw(canvas)

        bitmap
    }
}

With the next few lines, we can allow Google assistant to manage playback.

private const val MEDIA_SESSION_TAG = "hello_world_media"

override fun onCreate() {
    super.onCreate()

    ...

    mediaSession = MediaSessionCompat(applicationContext, MEDIA_SESSION_TAG).apply {
        isActive = true
    }
    playerNotificationManager?.setMediaSessionToken(mediaSession?.sessionToken)

    ...
}

We can also monitor the playback change events.

override fun onCreate() {
    super.onCreate()

    ...

    // Monitor ExoPlayer events.
    exoPlayer.addListener(PlayerEventListener())

    ...
}

private inner class PlayerEventListener : Player.EventListener {

    override fun onPlayerStateChanged(playWhenReady: Boolean, playbackState: Int) {
        if (playbackState == Player.STATE_READY) {
            if (exoPlayer.playWhenReady) {
                // In Playing state
            } else {
                // In Paused state
            }
        } else if (playbackState == Player.STATE_ENDED) {
            // In Ended state
        }
    }

    override fun onPlayerError(e: ExoPlaybackException?) {
        // On error
    }

}

Now to the actual play, pause and resume code.

@MainThread
fun play(uri: Uri, startPosition: Long, playbackSpeed: Float? = null) {
    val userAgent = Util.getUserAgent(applicationContext, BuildConfig.APPLICATION_ID)
    val mediaSource = ExtractorMediaSource(
            uri,
            DefaultDataSourceFactory(applicationContext, userAgent),
            DefaultExtractorsFactory(),
            null,
            null)

    val haveStartPosition = startPosition != C.POSITION_UNSET.toLong()
    if (haveStartPosition) {
        exoPlayer.seekTo(startPosition)
    }

    exoPlayer.prepare(mediaSource, !haveStartPosition, false)
    exoPlayer.playWhenReady = true
}

@MainThread
fun resume() {
    exoPlayer.playWhenReady = true
}

@MainThread
fun pause() {
    exoPlayer.playWhenReady = false
}

Lastly and very importantly, let us clean after ourselves when we are done.

override fun onDestroy() {
    mediaSession?.release()
    mediaSessionConnector?.setPlayer(null)
    playerNotificationManager?.setPlayer(null)

    exoPlayer.release()

    super.onDestroy()
}

Checkout the full and up to date code here.

Forem: Mostafa Gazar

TensorFlow 2.x and Keras must keep snippets

MNIST classification using Sequential API

MNIST classification using Functional API

Eager execution

Use ImageDataGenerator to load raw images in batches with real-time data augmentation

Keras data preprocessing utils API

Text classification using Sequential API

Tokenize corpus using Tokenizer.fit_on_texts

Load pre-trained embeddings

Plot training accuracy and loss after training is done

Track training progress in TensorBoard

Dynamic learning rate (learning rate schedules)

Handy callbacks

Save model for inference

Save model checkpoints and continue training where you left off

Some Pandas handy snippets for Data Scientists

Iterate through rows

Count unique values in dataframe

Style max value in a row or column

Display 1000 rows and columns

Save dataframe as CSV file

Create dataframe form python dictionary

Parse dates in dataframe

Some Python handy snippets for Machine Learning Engineers

Get file name and extension

Iterate through directory contents

Inverse dictionary key/value => value/key

Flatten a list of lists

Map string list to int list

Count unique values in list and their total

Sort dict by value

Format f strings

Batch an iterable

Beautiful Android rounded buttons

Build docker images inside of Kubernetes

Use cfssl in a multi-stage build docker image

How do you get your Machine Learning models from idea to production

Monitor for any new conda env and add them as Jupyter kernels

Train a lines segmentation model using Pytorch

The project structure

Data

Notebooks

Python files

The python environment

The dataset

The Network

The Training Pipeline

Jupyter notebooks tips and tricks

Shortcuts

Magic commands

Other

Classic notebooks extensions

Themes

Jupyter lab extensions

Themes

Try Jupyter Hub on GKE with dummy authentication

Create a new cluster

Set kubectl locally

Install helm and set it up

Setup Tiller Service account

Verify helm and tiller were installed properly

Make Helm aware of the JupyterHub Helm chart repository

Create JupyterHub config file

Now install the chart configured by your config.yaml

Playing local media files and stream media on Android using ExoPlayer

Use `ImageDataGenerator` to load raw images in batches with real-time data augmentation

Tokenize corpus using `Tokenizer.fit_on_texts`