Forem: Thomas Brittain

How to Set Up Terraform and Terragrunt

Thomas Brittain — Tue, 08 Nov 2022 10:00:00 +0000

Last time, we set up our local machine for accessing AWS programmatically. This will allow us to use Terraform and Terragrunt to easily create all infrastructure needed for our data warehouse. Now, let's set up Terragrunt and Terraform.

Install Terraform

Navigate to the Terraform downloads page:

Terraform Download

After installing Terraform enter the following in the terminal:

terraform --version

You should be greeted with output similar to:

Terraform v1.2.8
on darwin_arm64

Install Terragrunt

Terragrunt is a thin wrapper for Terraform, having a few additional tools for managing IaC projects.

Download and install it:

After installing Terragrunt type the following in the terminal:

terragrunt --version

You should get the Terragrunt version as output:

terragrunt version v0.38.7

Create a Git Repository

I'll be using Github, but any git hosting service should be similar.

Use My IaC Template

If you want to skip the next part and use my template repository:

self_sensored_iac

Visit the page and fork the repository into your Github account, then, clone it locally.

Open a terminal and type, replacing <YOUR_GITHUB_NAME> with your Github username:

git clone https://github.com/<YOUR_GITHUB_NAME>/self_sensored_iac.git

I'll explain in the next section why this repository is setup the way it is.

Create an IaC Template

In Github, create a new repository and call it self_sensored_iac or whatever name you'd like to give your personal enterprise. Then clone this repository locally.

Set "Add README.md" and "Add .gitignore". For the .gitignore template, select Terraform.

git clone https://github.com/<YOUR_NAME>/<YOUR_REPO>.git

Setup Enterprise Terragrunt Project

Whew, we made it. Our work machine is set up. Now, we need to create a Terragrunt project.

The idea of our Terragrunt project is to separate code into two major categories. First, common_modules will contain a folder for each of the major resources you plan to deploy. Imagine these are class definitions. The second category contains all the inputs needed to initialize the resources defined in the common_modules.

The easiest way to create this project is with a folder structure like this:

.
├── README.md
├── common_modules
│   ├── vpc
│   │   ├── data.tf
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
├── empty.yaml
└── prod
    ├── environment.yaml
    └── us-west-2
        ├── aws_provider.tf
        ├── terragrunt.hcl
        ├── region.yaml
        └── vpc
           └── terragrunt.hcl

`common_modules`

As I mentioned, the commond_modules is similar to a class in object-oriented programming. It is a blueprint of a resource and should be coded in a way to provide appropriate flexibility. That is, if we create a blueprint to set up a VPC, it should probably take only a few inputs. These inputs will then change certain behavior per deployment of the resource.

In Terragrunt, a module is defined by a folder containing a collection of files ending in .tf. Collectively, these files tell Terraform how to create the needed infrastructure within AWS.

Let's look at the VPC code. It is in ./commond_modules/vpc/ As the files may suggest, main.tf is where most of the magic happens. Let's take a look:

module "vpc" {
  source = "terraform-aws-modules/vpc/aws"

  name = "${var.vpc_name}"
  cidr = "${var.vpc_network_prefix}.0.0/16"

  azs             = ["${var.region}a", "${var.region}b", "${var.region}c"]
  private_subnets = ["${var.vpc_network_prefix}.1.0/24", "${var.vpc_network_prefix}.2.0/24", "${var.vpc_network_prefix}.3.0/24"]
  public_subnets  = ["${var.vpc_network_prefix}.101.0/24", "${var.vpc_network_prefix}.102.0/24", "${var.vpc_network_prefix}.103.0/24"]

  enable_nat_gateway = false
  single_nat_gateway = true

  enable_dns_hostnames = true
  enable_dns_support = true

  tags = {
    Terraform = "true"
    Environment = "${var.environment_name}"
  }

  nat_gateway_tags = {
    Project = "${var.project_name}"
    Terraform = "true"
    Environment = "${var.environment_name}"
  }
}

resource "aws_security_group" "allow-ssh" {
  vpc_id      = data.aws_vpc.vpc_id.id
  name        = "allow-ssh"
  description = "Security group that allows ssh and all egress traffic"

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  // Allow direct access to the EC2 boxes for me only.
  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["${chomp(data.http.myip.body)}/32"]
  }

  tags = {
    Name = "allow-ssh"
  }

}

Note, the module "vpc" is a prebuilt VPC module, so what we are doing is grabbing a VPC module provided by AWS:

Terraform VPC Module

Which handles a lot of the boilerplate setup. Then, we take care of even more settings we don't often want to change. That is, every place you see ${var.something} we are creating inputs that may be changed at the time of deployment.

Take the VPC name parameter for example:

module "vpc" {
  source = "terraform-aws-modules/vpc/aws"

  name = "${var.vpc_name}"
  ...

Keep this in mind while we look at the prod folder:

prod

The production folder and subfolder are where we consume the modules we have defined in the common_modules folder. Let's look at the /prod/us-west-2/terragrunt.hcl file.

terraform {
  source = "../../..//common_modules/vpc"
}

...

inputs = {
    vpc_name = "ladviens-analytics-stack-vpc"
    vpc_network_prefix = "10.17"
}

The important definitions here are the terraform and inputs maps. The terraform map will tell Terragrunt, when run from this folder, to treat the source directory specified as a Terraform module.

The inputs map contains all of the variables needed to make sure VPC is deployed correctly. You'll notice, we have hardcoded everything in our vpc module but the name and network prefix. This may not be ideal for you. Feel free to change anything in the vpc module files to make it more reusable. And to help, I'll provide an example a bit later in the article.

Planning Our VPC

One of the joys of Terraform is its ability to report what infrastructure would be built before it is actually built. This can be done by running the terragrunt plan command at the terminal when inside the directory ./prod/us-west-2/vpc/.

At the terminal, navigate to the VPC definition directory:

cd prod/us-west-2/vpc

Then run:

terragrunt plan

You should end up with something like this:

Initializing modules...
Downloading registry.terraform.io/terraform-aws-modules/vpc/aws 3.18.1 for vpc...
- vpc in .terraform/modules/vpc

Initializing the backend...
...

After a little while, Terraform should print a complete plan. This consists of a bunch of diffs. The green + are indicating what will be added. Yellow ~ flagging what will be changed. And a red - indicates resources to be destroyed. At this point, the plan will return showing everything that needs to be built.

Deploying the VPC

Let's deploy the VPC. Still in the ./prod/us-west-2/vpc/ directory, type:

terragrunt apply

Again Terraform will assess the inputs in your terragrunt.hcl and the module definitions in ./common_modules/vpc/ then print out what will be created. However, this time it will ask if you want to deploy. Type yes and hit return.

Terraform will begin requesting resources in AWS on your behalf. Once it is done, I encourage you to open your AWS console and navigate to the "VPC" section. You should see a newly created VPC alongside your default VPC (the one with no name).

Huzzah!

Modifying modules to increase reusability

Let's look at how to add a new input to the vpc module we've made. Open the ./common_modules/vpc/variables.tf file go to the bottom of the file and add:

variable "enable_dns" {

}

We will add more to it, but this is good for now.

A variable in Terraform acts as an input for a module or file. With the enable_dns variable in place, in the terminal, in the directory ./prod/us-west-2/vpc/, run the following:

terragrunt apply

This time you should be prompted with:

var.enable_dns
  Enter a value:

This is Terragrunt seeing a variable definition and there's no matching input in your ./prod/us-west-2/vpc/terragrunt.hcl, so it prompts at the command line. This can come in handy, say, having a variable that is a password and you don't want it hard coded in your repository.

But let's go ahead and adjust our terragrunt.hcl to contain the needed input. In the file ./prod/us-west-2/vpc/terragrunt.hcl add the following enable_dns = true. The result should look like this:

terraform {
  source = "../../..//common_modules/vpc"
}

include {
  path = find_in_parent_folders()
}

inputs = {
    vpc_name = "ladviens-analytics-stack-vpc"
    vpc_network_prefix = "10.17"
    enable_dns = true
}

Now run terragrunt apply again. This time Terragrunt should find the input in the terragrunt.hcl file matching the variable name in the module.

Of course, we're not quite done. We still need to use the variable enable_dns in our Terraform module. Open the file ./common_modules/vpc/main.tf and edit the vpc module by modifying these two lines:

  enable_dns_hostnames = true
  enable_dns_support = true

It should look like this:

  enable_dns_hostnames = var.enable_dns
  enable_dns_support = var.enable_dns

Now, if you run terragrunt apply from ./prod/us-west-2/vpc/ again you should not be prompted for variable input.

A couple of clean-up items. First, let's go back to the enable_dns variable definition and add a description and default value. Back in the ./common_modules/vpc/variables.tf update the enable_dns variable to:

variable "enable_dns" {
  type = bool
  default = true
  description = "Should DNS services be enabled." 
}

It's a best practice to add descriptions to all your Terraform definitions, as they can be used to generate a dependency graph by running in the resource defintion folder:

terragrunt graph

But more importantly, make sure you add a type all your variables. They are some instances where Terraform assumes a variable is a type when the input will not be compatible. That is, Terraform may assume and variable is a string, but you meant to provide it a boolean. Trust me. Declaring an appropriate type on all variables will save you a lot of time debugging Terraform code one day--not that I've ever lost a day's work to such a problem.

Last clean-up item, let's go back to the ./commond_modules/vpc/ directory and run:

terraform fmt

This will ensure our code stays nice and tidy.

This last bit is optional, but if you've made additional changes and want to keep them, don't forget to commit them and push them to your repository:

git add .
git commit -m "Init"
git push

Let's Destroy a VPC

Lastly, let's use Terragrunt to destroy the VPC we made. I'm often experimenting on my own AWS account and get worried I'll leave something on waking with an astronomical bill. Getting comfortable with Terragrunt has taken a lot of the fear away, as at the of the day clean up is often as easy as running terragrunt destroy.

Let's use it to destroy our VPC. And don't worry, we can redeploy it by running terragrunt apply again.

In the terminal navigate to ./prod/us-west-2/vpc/ folder and run:

terragrunt destroy

Before you type "yes" ensure you are in the correct directory. If you are, type "yes," hit enter, and watch while Terraform destroys everything we built together.

What's Next

In the a next article we'll begin to add resources to our VPC, but we will take a break from Terraform and Terragrun and use the Serverless Framework to attach an API Gateway to our existing VPC. I'm so excited! 🙌🏽

Creating an AWS Account for Programmatic Access

Thomas Brittain — Mon, 07 Nov 2022 10:00:00 +0000

Before we can begin creating infrastructure through tools like Terraform and the Serverless Framework, we need to set up an AWS account and credentials for accessing AWS through the AWS CLI. The AWS CLI will allow us to easily set up programmatic access to AWS, which is necessary to use Terraform and the Serverless Framework to rapidly deploy needed infrastructure.

Creating an AWS Account

Before beginning into AWS, let me warn you: Stuff can get expensive. Please exercise great caution, as leaving the wrong resource on can lead to a heft bill overnight.

To create an account visit:

Instructions for Creating an AWS Account

Once done, we should have an AWS "root" account. It is best practice to enable multi-factor authentication (MFA) on this account. If someone can access the root account, there's little they can't do.

Another good practice is to create a separate AWS user for programmatic access, we should create a separate user to act as our administration account.

In the end, our IAM dashboard should look like:

Enable MFA on Root Account

Let's create an "admin" account. Go to your account name and then "Settings". Enable MFA.

Create an Admin User

In the search bar above look for "IAM." This is AWS' users and permissions service. Let's make an administrative user; we will add this user's API credentials to our local system for use by Terraform and Serverless Framework.

Now go to "User":

Enter "admin" as the user name and select the "Access key - Programmatic access" option. If you would like to log in to the account from the web, then also select the "Password - AWS Management Console access" option.

Select "Attach existing policies directly":

Skip or add tags, review the new user, then create it.

My recommendation is to use a password manager like 1password or Lastpass to store your "Access Key ID" and "Secret access key" as we will be using them in the next step.

Also, it is a good practice to set up MFA on the admin user as well.

Setting Up the AWS CLI

Next, we need to install the AWS CLI. I usually only use the actual AWS CLI tool to manage credentials or spot-check infrastructure, but both are handy, so worth installing it.

AWS' Instructions for Installing AWS CLI

After installing open up a terminal. Type the following to ensure it's installed properly.

aws --version

You should get an output like:

aws-cli/2.8.2 Python/3.10.8 Darwin/21.6.0 source/arm64 prompt/off

If you have any trouble, ping me in the comments.

Set Up AWS Programmatic Credentials

Still at the terminal, type:

aws configure

You will be prompted with questions similar to:

AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]:

AWS Access Key ID and AWS Secret Access Key should be retrieved from your password manager.

Default region name and Default output format will depend on you.

For the sake of this article series, I'll be conducting all work in us-west-2. Do know, many services and resources are localized to the region, so if you create infrastructure in us-west-2, it will not be visible if you are in the UI but under the region us-west-1. Also, identical resources in different regions will have different IDs, or Amazon Resource Numbers (ARNs).

If you've any trouble, you might review AWS instructions on programmatic access credential setup:

Config and credential file settings

What's Next

Next, we are going to set up Terraform and Terragrunt so we can easily deploy and manage the needed infrastructure for our data warehouse.

Let's Build a Data Warehouse

Thomas Brittain — Sat, 05 Nov 2022 10:00:00 +0000

The Reason

I am the lead data engineer at Bitfocus, the best SaaS provider for homeless management information systems (HMIS).

For many reasons, I've advocated with our executives to switch our analytics flow to use a data warehouse. Strangely, it worked. I'm now positioned to design the beast. This series will be my humble attempt to journal everything I learn along the way.

The Plan

I've decided it would be better if I learned how to build an analytics warehouse on my own, before committing our entire business to an untested technology stack. Of course, I needed a project similar enough it had the same challenges. I've decided to expose my Apple HealthKit data in a business intelligence platform like Looker.

The Stack

I've attempted similar health data projects before:

Give Me MyFitness Pal Data!

But I'm going to be a bit more ambitious this time. I'd like to get all of the data generated by my Apple Watch and iPhone out of HealthKit and send them to a SQL database inside AWS.

The ingestion of these data will be lightly structured. That is, no transformations will be done until the data are in the database. I'll use the Auto Health Export iOS app to send data to a web API endpoint.

The web API endpoint will be created using the Severless Framework. This should allow me to build a REST API on top of AWS' Lambda functions, enabling the Auto Health Export app to synchronize data to the database periodically (as low as every 5 minutes, but dictated by Apple).

Once the data are stored in a database, I'll use Dbt (Data Build Tool) to transform the data for exposing them in the business intelligence tool. This idea of processing data inside a database is often referred to as "extract, load, transform" or "ELT," which is becoming standard versus the classical "extract, transform, load" or "ETL."

After transformation, I'll use Dbt to shove the data back into a database built for analytics query processing. Aka, a "data warehouse." The most popular analytics database technologies right now are Snowflake and Redshift. I'll use neither. I can't afford them and I despise how pushy their salespeople are. Instead, I'll use Postgres. (It's probably what Snowflake and Redshift are built on anyway.)

Lastly, I'll stand up Lightdash as the business intelligence tool.

Tools

Auto Health Export

As mentioned, I've attempted to pull data from the Apple ecosystem several times in the past. One challenge I had was writing an iOS app to pull HealthKit data from an Apple device and send it to a REST API. It's not hard. I'm just not an iOS developer, making it time-consuming. And I'm at the point in my tech career I'm comfortable purchasing others' solutions to my problems.

Anyway, not a perfect app, but it does have the ability to pull your recent HealthKit data and send it to a REST API endpoint.

Auto Health Export

Serverless

The team creating our SaaS product has talked about going "serverless" for a while. In an attempt to keep up, I decided to focus on creating Lambda and API Gateway services to receive the data from the Auto Health Export app.

While I was researching serverless architecture I ran into the Serverless Framework. I quite like it. It allows you to focus on the code of Lambda, as it will build out the needed infrastructure on deployment.

Serverless

AWS' Lambda

The heart of the ingestion. This method will receive the health data as JSON and store it in the Postgres database for transformation.

AWS' Lambda

Postgres

I've selected the Postgres database after researching. I'll detail my reason for selecting Postgres later in the series.

Regardless of tech choice, this database will hold untransformed data, or "raw data," as well as act as the processing engine for transforming the data before moving it into the analytics warehouse.

Postgres >=9.5

Dbt

Data Build Tool, or Dbt, is quickly becoming the de facto tool for transforming raw data into an analytics data warehouse. At its core, it uses SQL and Jinja to enable consistent and powerful transformation "models." These models rely on a scalable SQL database, known as a "processing engine," to transform the data before sending it on to its final destination, the analytics warehouse.

MariaDB with ColumnStore

I'd like to use MariaDB as the actual analytics data warehouse. It is an unconventional choice compared to Snowflake or Redshift. But, I'm attempting to use open-source software (OSS) as much as possible, as our company is having a horrific experience with the downfall of Looker.

One of the better-kept secrets about MariaDB is its ColumnStore engine. It allows three major features which have convinced me to try it out:

It stores data in a column, making it faster for analytics
The data store is distributed among S3 buckets
Data are run-length encoded, greatly reducing storage size
ColumnStore engine tables have join capability with InnoDB tables

I'm also curious whether MariaDB could possibly be used as a processing engine, as I'd prefer to reduce the cognitive complexity of the stack by having only one SQL dialect.

MariaDB >=10.5.4

Lightdash

At this point, you probably got I'm not happy with what Google has done to Looker. That stated, Looker is still my favorite business intelligence tool. Luckily, there is a budding Looker alternative that meets my OSS requirement: Lightdash. I've never used it, let alone deployed it into production. I guess we'll see how it goes.

Lightdash

Terraform

You may notice, a data warehouse requires a lot of infrastructure. I hate spinning-up infrastructure through a UI. It isn't repeatable, it's not in version control, and I like writing code. Luckily, Hashicorp's got my back with Terraform. It allows defining and deploying infrastructure as code (IaC).

Terraform

Terragrunt

Terragrunt is a wonderful addition to Terraform. It allows an entire company's infrastructure to be easily maintained.

Terragrunt

The Beginning

First up, let's take a look at the Auto Health Export app and our Serverless architecture. A couple of personal notes before jumping in.

I'm writing this series as a journal of what I learn. That stated, be aware I may switch solutions anywhere in our stack, at any time.

Also, I know a lot of stuff. But there's more stuff I don't know. If I make a mistake, please let me know in the comments. All feedback is appreciated, but respectful feedback is adored.

Scraping Images from Google Search Results using Python

Thomas Brittain — Wed, 23 Dec 2020 10:00:00 +0000

This articles relies on the code written by Fabian Bosler:

Image Scraping with Python

I've only modified Bosler's code to make it a bit easier to pull images for multiple search terms.

The full code can be found the series' Github repository:

deep_arcane -- scrap.py

Magic Symbols

As I've mentioned in my previous article, I needed a lot of images of magic symbols for training a deep convolutional generative adversarial network (DCGAN). Luckily, I landed on Bosler's article early on.

To get my images, I used Chrome browser, Chromedriver, Selenium, and a Python script to slowly scrape images from Google's image search. The scraping was done throttled to near human speed, but allowed automating the collection of a lot of images.

Regarding this process, I'll echo Bosler, I'm in no way a legal expert. I'm not a lawyer and nothing I state should be taking as legal advice. I'm just some hack on the internet. However, from what I understand, scraping the SERPs (search engine results pages) is not illegal, at least, not for personal use. But using Google's Image search for automated scraping of images is against their terms of service (ToS). Replicate this project at your own risk. I know when I adjusted my script to search faster Google banned my IP. I'm glad it was temporary.

Bosler's Modified Script

The script automatically searches for images and collects their underlying URL. After searching, it uses the Python requests library to download all the images into a folder named respective to the search term.

Here are the modifications I made to Bosler's original script:

Added a search term loop. This allows the script to continue running past one search term.
The script was getting stuck when it ran into the "Show More Results," I've fixed the issue.
The results are saved in directories associated with the search term. If the script is interrupted and rerun it will look at what directories are created first, and remove those from the search terms.
I added a timeout feature; thanks to a user on Stack Overflow.
I parameterized the number of images to look for per search term, sleep times, and timeout.

Code: Libraries

You will need to install Chromedriver and Selenium--this is explained well in the original article.

Image Scraping with Python

You will also need to install Pillow--a Python library for managing images.

You can install it with:

pip install pillow

After installing all the needed libraries the following block of code should execute without error:

import os
import time

import io
import hashlib
import signal
from glob import glob
import requests

from PIL import Image
from selenium import webdriver

If you have any troubles, revisit the original articles setup explanation or feel free to ask questions in the comments below.

Code: Parameters

I've added a few parameters to the script to make use easier.

number_of_images = 400
GET_IMAGE_TIMEOUT = 2
SLEEP_BETWEEN_INTERACTIONS = 0.1
SLEEP_BEFORE_MORE = 5
IMAGE_QUALITY = 85

output_path = "/path/to/your/image/directory"

The number_of_images tells the script how many images to search for per search term. If the script runs out of images before reaching number_of_images, it will skip to the next term.

GET_IMAGE_TIMEOUT determines how long the script should wait for a response before skipping to the next image URL.

SLEEP_BETWEEN_INTERACTIONS is how long the script should delay before checking the URL of the next image. In theory, this can be set low, as I don't think it makes any requests of Google. But I'm unsure, adjust at your own risk.

SLEEP_BEFORE_MORE is how long the script should wait before clicking on the "Show More Results" button. This should not be set lower than you can physically search. Your IP will be banned. Mine was.

Code: Search Terms

Here is where the magic happens. The search_terms array should include any terms which you think will get the sorts of images you are targeting.

Below are the exact set of terms I used to collect magic symbol images:

search_terms = [
    "black and white magic symbol icon",
    "black and white arcane symbol icon",
    "black and white mystical symbol",
    "black and white useful magic symbols icon",
    "black and white ancient magic sybol icon",
    "black and white key of solomn symbol icon",
    "black and white historic magic symbol icon",
    "black and white symbols of demons icon",
    "black and white magic symbols from book of enoch",
    "black and white historical magic symbols icons",
    "black and white witchcraft magic symbols icons",
    "black and white occult symbols icons",
    "black and white rare magic occult symbols icons",
    "black and white rare medieval occult symbols icons",
    "black and white alchemical symbols icons",
    "black and white demonology symbols icons",
    "black and white magic language symbols icon",
    "black and white magic words symbols glyphs",
    "black and white sorcerer symbols",
    "black and white magic symbols of power",
    "occult religious symbols from old books",
    "conjuring symbols",
    "magic wards",
    "esoteric magic symbols",
    "demon summing symbols",
    "demon banishing symbols",
    "esoteric magic sigils",
    "esoteric occult sigils",
    "ancient cult symbols",
    "gypsy occult symbols",
    "Feri Tradition symbols",
    "Quimbanda symbols",
    "Nagualism symbols",
    "Pow-wowing symbols",
    "Onmyodo symbols",
    "Ku magical symbols",
    "Seidhr And Galdr magical symbols",
    "Greco-Roman magic symbols",
    "Levant magic symbols",
    "Book of the Dead magic symbols",
    "kali magic symbols",
]

Before searching, the script checks the image output directory to determine if images have already been gathered for a particular term. If it has, the script will exclude the term from the search. This is part of my "be cool" code. We don't need to be downloading a bunch of images twice.

The code below grabs all the directories in our output path, then reconstructs the search term from the directory name (i.e., it replaces the "_"s with " "s.)

dirs = glob(output_path + "*")
dirs = [dir.split("/")[-1].replace("_", " ") for dir in dirs]
search_terms = [term for term in search_terms if term not in dirs]

Code: Chromedriver

Before starting the script, we have to kick off a Chromedriver session. Note, you must put the chromedriver executable into a folder listed in your PATH variable for Selenium to find it.

For MacOS users, setting up Chromedriver for Selenium use is a bit tough to do manually. But, using homebrew makes it easy.

brew install chromedriver

If everything is setup correctly, executing the following code will open a Chrome browser and bring up the Google search page.

wd = webdriver.Chrome()
wd.get("https://google.com")

Code: Chrome Timeout

The timeout class below I borrowed from Thomas Ahle at Stack Overflow. It is a dirty way of creating a timeout for the GET request to download the image. Without it, the script can get stuck on unresponsive image downloads.

class timeout:
    def __init__(self, seconds=1, error_message="Timeout"):
        self.seconds = seconds
        self.error_message = error_message

    def handle_timeout(self, signum, frame):
        raise TimeoutError(self.error_message)

    def __enter__(self):
        signal.signal(signal.SIGALRM, self.handle_timeout)
        signal.alarm(self.seconds)

    def __exit__(self, type, value, traceback):
        signal.alarm(0)

Code: Fetch Images

As I've hope I made clear, the code below I did not write; I just polished it. I'll provide a brief explanation, but refer back to Bosler's article for more information.

Essentially, the script:

Creates a directory corresponding to a search term in the array.
It passes the search term to the fetch_image_urls(), this function drives the Chrome session. The script navigates the Google to find images relating to the search term. It stores the image link in an list. After it has searched through all the images or reached the num_of_images it returns a list (res) containing all the image URLs.
The list of image URLs is passed to the persist_image(), which then downloads each one of the images into the corresponding folder.
It repeats steps 1-3 per search term.

I've added extra comments as a guide:

def fetch_image_urls(
    query: str,
    max_links_to_fetch: int,
    wd: webdriver,
    sleep_between_interactions: int = 1,
):
    def scroll_to_end(wd):
        wd.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        time.sleep(sleep_between_interactions)

    # Build the Google Query.
    search_url = "https://www.google.com/search?safe=off&site=&tbm=isch&source=hp&q={q}&oq={q}&gs_l=img"

    # load the page
    wd.get(search_url.format(q=query))

    # Declared as a set, to prevent duplicates.
    image_urls = set()
    image_count = 0
    results_start = 0
    while image_count < max_links_to_fetch:
        scroll_to_end(wd)

        # Get all image thumbnail results
        thumbnail_results = wd.find_elements_by_css_selector("img.Q4LuWd")
        number_results = len(thumbnail_results)

        print(
            f"Found: {number_results} search results. Extracting links from {results_start}:{number_results}"
        )

        # Loop through image thumbnail identified
        for img in thumbnail_results[results_start:number_results]:
            # Try to click every thumbnail such that we can get the real image behind it.
            try:
                img.click()
                time.sleep(sleep_between_interactions)
            except Exception:
                continue

            # Extract image urls
            actual_images = wd.find_elements_by_css_selector("img.n3VNCb")
            for actual_image in actual_images:
                if actual_image.get_attribute(
                    "src"
                ) and "http" in actual_image.get_attribute("src"):
                    image_urls.add(actual_image.get_attribute("src"))

            image_count = len(image_urls)

            # If the number images found exceeds our `num_of_images`, end the seaerch.
            if len(image_urls) >= max_links_to_fetch:
                print(f"Found: {len(image_urls)} image links, done!")
                break
        else:
            # If we haven't found all the images we want, let's look for more.
            print("Found:", len(image_urls), "image links, looking for more ...")
            time.sleep(SLEEP_BEFORE_MORE)

            # Check for button signifying no more images.
            not_what_you_want_button = ""
            try:
                not_what_you_want_button = wd.find_element_by_css_selector(".r0zKGf")
            except:
                pass

            # If there are no more images return.
            if not_what_you_want_button:
                print("No more images available.")
                return image_urls

            # If there is a "Load More" button, click it.
            load_more_button = wd.find_element_by_css_selector(".mye4qd")
            if load_more_button and not not_what_you_want_button:
                wd.execute_script("document.querySelector('.mye4qd').click();")

        # Move the result startpoint further down.
        results_start = len(thumbnail_results)

    return image_urls

def persist_image(folder_path: str, url: str):
    try:
        print("Getting image")
        # Download the image.  If timeout is exceeded, throw an error.
        with timeout(GET_IMAGE_TIMEOUT):
            image_content = requests.get(url).content

    except Exception as e:
        print(f"ERROR - Could not download {url} - {e}")

    try:
        # Convert the image into a bit stream, then save it.
        image_file = io.BytesIO(image_content)
        image = Image.open(image_file).convert("RGB")
        # Create a unique filepath from the contents of the image.
        file_path = os.path.join(
            folder_path, hashlib.sha1(image_content).hexdigest()[:10] + ".jpg"
        )
        with open(file_path, "wb") as f:
            image.save(f, .jpg", quality=IMAGE_QUALITY)
        print(f"SUCCESS - saved {url} - as {file_path}")
    except Exception as e:
        print(f"ERROR - Could not save {url} - {e}")

def search_and_download(search_term: str, target_path=".https://ladvien.com/images/", number_images=5):
    # Create a folder name.
    target_folder = os.path.join(target_path, "_".join(search_term.lower().split(" ")))

    # Create image folder if needed.
    if not os.path.exists(target_folder):
        os.makedirs(target_folder)

    # Open Chrome
    with webdriver.Chrome() as wd:
        # Search for images URLs.
        res = fetch_image_urls(
            search_term,
            number_images,
            wd=wd,
            sleep_between_interactions=SLEEP_BETWEEN_INTERACTIONS,
        )

        # Download the images.
        if res is not None:
            for elem in res:
                persist_image(target_folder, elem)
        else:
            print(f"Failed to return links for term: {search_term}")

# Loop through all the search terms.
for term in search_terms:
    search_and_download(term, output_path, number_of_images)

Results

Scraping tehe images resulted in a lot of garbage images (noise) along with my ideal training images.

For example, out of all the images shown, I only wanted the image highlighted:

There was also the problem of lots of magic symbols stored in a single image. These "collection" images would need further processing to extract all of the symbols.

However, even with a few rough edges, the script sure as hell beat manually downloading the 10k images I had in the end.

How to Send Data between PC and Arduino using Bluetooth LE

Thomas Brittain — Sat, 11 Jul 2020 10:00:00 +0000

A how-to guide on connecting your PC to an Arduino using Bluetooth LE and Python. To make it easier, we will use bleak an open source BLE library for Python. The code provided should work for connecting your PC to any Bluetooth LE devices.

Before diving in a few things to know

Bleak is under-development. It will have issues
Although Bleak is multi-OS library, Windows support is still rough
PC operating systems suck at BLE
Bleak is asynchronous; in Python, this means a bit more complexity
The code provided is a proof-of-concept; it should be improved before use

Ok, all warnings stated, let's jump in.

Bleak

Bleak is a Python package written by Henrik Blidh. Although the package is still under development, it is pretty nifty. It works on Linux, Mac, or Windows. It is non-blocking, which makes writing applications a bit more complex, but extremely powerful, as your code doesn't have to manage concurrency.

Setup

Getting started with BLE using my starter application and bleak is straightforward. You need to install bleak and I've also included library called aioconsole for handling user input asynchronously

pip install bleak aioconsole

Once these packages are installed we should be ready to code. If you have any issues, feel free to ask questions in the comments. I'll respond when able.

The Code

Before we get started, if you'd rather see the full-code it can be found at:

Bleak App

If you are new to Python then following code may look odd. You'll see terms like async, await, loop, and future. Don't let it scare you. These keywords are Python's way of allowing a programmer to "easily" write asynchronous code in Python.

If you're are struggling with using asyncio, the built in asynchronous Python library, I'd highly recommend Łukasz Langa's detailed video series; it takes a time commitment, but is worth it.

Import asyncio

If you are an experienced Python programmer, feel free to critique my code, as I'm a new to Python's asynchronous solutions. I've got my big kid britches on.

Enough fluff. Let's get started.

Application Parameters

There are a few code changes needed for the script to work, at least, with the Arduino and firmware I've outlined in the previous article:

Getting Started with Bluetooth LE on the Arduino Nano 33 Sense

The incoming microphone data will be dumped into a CSV; one of the parameters is where you would like to save this CSV. I'll be saving it to the Desktop. I'm also retrieving the user's home folder from the HOME environment variable, which is only available on Mac and Linux OS (Unix systems). If you are trying this project from Windows, you'll need to replace the root_path reference with the full path.

root_path = os.environ["HOME"]
output_file = f"{root_path}/Desktop/microphone_dump.csv"

You'll also need need to specify the characteristics which the Python app should try to subscribe to when connected to remote hardware. Referring back to our previous project, you should be able to get this from the Arduino code. Or the Serial terminal printout.

read_characteristic = "00001143-0000-1000-8000-00805f9b34fb"
write_characteristic = "00001142-0000-1000-8000-00805f9b34fb"

Main

The main method is where all the async code is initialized. Essentially, it creates three different loops, which run asynchronously when possible.

Main -- you'd put your application's code in this loop. More on it later
Connection Manager -- this is the heart of the Connection object I'll describe more in a moment.
User Console -- this loop gets data from the user and sends it to the remote device.

You can imagine each of these loops as independent, however, what they are actually doing is pausing their execution when any of the loops encounter a blocking I/O event. For example, when input is requested from the user or waiting for data from the remote BLE device. When one of these loops encounters an I/O event, they let one of the other loops take over until the I/O event is complete.

That's far from an accurate explanation, but like I said, I won't go in depth on async Python, as Langa's video series is much better than my squawking.

Though, it's important to know, the ensure_future is what tells Python to run a chunk of code asynchronously. And I've been calling them "loops" because each of the 3 ensure_future calls have a while True statement in them. That is, they do not return without error.

After creating the different futures, the loop.run_forever() is what causes them to run.

if __name__ == "__main__":
    # Create the event loop.
    loop = asyncio.get_event_loop()

    data_to_file = DataToFile(output_file)
    connection = Connection(
        loop, read_characteristic, write_characteristic, data_to_file.write_to_csv
    )
    try:
        asyncio.ensure_future(main())
        asyncio.ensure_future(connection.manager())
        asyncio.ensure_future(user_console_manager(connection))
        loop.run_forever()
    except KeyboardInterrupt:
        print()
        print("User stopped program.")
    finally:
        print("Disconnecting...")
        loop.run_until_complete(connection.cleanup())

Where does bleak come in? You may have been wondering about the code directly before setting up the loops.

    connection = Connection(
        loop, read_characteristic, write_characteristic, data_to_file.write_to_csv
    )

This class wrap the bleak library and makes it a bit easier to use. Let me explain.

Connection()

You may be asking, "Why create a wrapper around bleak, Thomas?" Well, two reasons. First, the bleak library is still in development and there are several aspects which do not work well. Second, there are additional features I'd like my Bluetooth LE Python class to have. For example, if you the Bluetooth LE connection is broken, I want my code to automatically attempt to reconnect. This wrapper class allows me to add these capabilities.

I did try to keep the code highly hackable. I want anybody to be able to use the code for their own applications, with a minimum time investment.

Connection(): init

The Connection class has three required arguments and one optional.

loop -- this is the loop established by asyncio, it allows the connection class to do async magic.
read_characteristic -- the characteristic on the remote device containing data we are interested in.
write_characteristic -- the characteristic on the remote device which we can write data.
data_dump_handler -- this is the function to call when we've filled the rx buffer.
data_dump_size -- this is the size of the rx buffer. Once it is exceeded, the data_dump_handler function is called and the rx buffer is cleared.

class Connection:

    client: BleakClient = None

    def __init__(
        self,
        loop: asyncio.AbstractEventLoop,
        read_characteristic: str,
        write_characteristic: str,
        data_dump_handler: Callable[[str, Any], None],
        data_dump_size: int = 256,
    ):
        self.loop = loop
        self.read_characteristic = read_characteristic
        self.write_characteristic = write_characteristic
        self.data_dump_handler = data_dump_handler
        self.data_dump_size = data_dump_size

Alongside the arguments are internal variables which track device state.

The variable self.connected tracks whether the BleakClient is connected to a remote device. It is needed since the await self.client.is_connected() currently has an issue where it raises an exception if you call it and it's not connected to a remote device. Have I mentioned bleak is in progress?

        # Device state
        self.connected = False
        self.connected_device = None

self.selected_device hangs on to the device you selected when you started the app. This is needed for reconnecting on disconnect.

The rest of variables help track the incoming data. They'll probably be refactored into a DTO at some point.

        # RX Buffer
        self.last_packet_time = datetime.now()
        self.rx_data = []
        self.rx_timestamps = []
        self.rx_delays = []

Connection(): Callbacks

There are two callbacks in the Connection class. One to handle disconnections from the Bluetooth LE device. And one to handle incoming data.

Easy one first, the on_disconnect method is called whenever the BleakClient loses connection with the remote device. All we're doing with the callback is setting the connected flag to False. This will cause the Connection.connect() to attempt to reconnect.

    def on_disconnect(self, client: BleakClient):
        self.connected = False
        # Put code here to handle what happens on disconnet.
        print(f"Disconnected from {self.connected_device.name}!")

The notification_handler is called by the BleakClient any time the remote device updates a characteristic we are interested in. The callback has two parameters, sender, which is the name of the device making the update, and data, which is a bytearray containing the information received.

I'm converting the data from two-bytes into a single int value using Python's from_bytes(). The first argument is the bytearray and the byteorder defines the endianness (usually big). The converted value is then appended to the rx_data list.

The record_time_info() calls a method to save the current time and the number of microseconds between the current byte received and the previous byte.

If the length of the rx_data list is greater than the data_dump_size, then the data are passed to the data_dump_handler function and the rx_data list is cleared, along with any time tracking information.

    def notification_handler(self, sender: str, data: Any):
        self.rx_data.append(int.from_bytes(data, byteorder="big"))
        self.record_time_info()
        if len(self.rx_data) >= self.data_dump_size:
            self.data_dump_handler(self.rx_data, self.rx_timestamps, self.rx_delays)
            self.clear_lists()

Connection(): Connection Management

The Connection class's primary job is to manage BleakClient's connection with the remote device.

The manager function is one of the async loops. It continually checks if the Connection.client exists, if it doesn't then it prompts the select_device() function to find a remote connection. If it does exist, then it executes the connect().

    async def manager(self):
        print("Starting connection manager.")
        while True:
            if self.client:
                await self.connect()
            else:
                await self.select_device()
                await asyncio.sleep(15.0, loop=loop)

The connect() is responsible for ensuring the PC's Bluetooth LE device maintains a connection with the selected remote device.

First, the method checks if the the device is already connected, if it does, then it simply returns. Remember, this function is in an async loop.

If the device is not connected, it tries to make the connection by calling self.client.connect(). This is awaited, meaning it will not continue to execute the rest of the method until this function call is returned. Then, we check if the connection is was successful and update the Connection.connected property.

If the BleakClient is indeed connected, then we add the on_disconnect and notification_handler callbacks. Note, we only added a callback on the read_characteristic. Makes sense, right?

Lastly, we enter an infinite loop which checks every 5 seconds if the BleakClient is still connected, if it isn't, then it breaks the loop, the function returns, and the entire method is called again.

    async def connect(self):
        if self.connected:
            return
        try:
            await self.client.connect()
            self.connected = await self.client.is_connected()
            if self.connected:
                print(f"Connected to {self.connected_device.name}")
                self.client.set_disconnected_callback(self.on_disconnect)
                await self.client.start_notify(
                    self.read_characteristic, self.notification_handler,
                )
                while True:
                    if not self.connected:
                        break
                    await asyncio.sleep(5.0, loop=loop)
            else:
                print(f"Failed to connect to {self.connected_device.name}")
        except Exception as e:
            print(e)

Whenever we decide to end the connection, we can escape the program by hitting CTRL+C, however, before shutting down the BleakClient needs to free up the hardware. The cleanup method checks if the Connection.client exists, if it does, it tells the remote device we no longer want notifications from the read_characteristic. It also sends a signal to our PC's hardware and the remote device we want to disconnect.

    async def cleanup(self):
        if self.client:
            await self.client.stop_notify(read_characteristic)
            await self.client.disconnect()

Device Selection

Bleak is a multi-OS package, however, there are slight differences between the different operating-systems. One of those is the address of your remote device. Windows and Linux report the remote device by it's MAC. Of course, Mac has to be the odd duck, it uses a Universally Unique Identifier (UUID). Specially, it uses a CoreBluetooth UUID, or a CBUUID.

These identifiers are important as bleak uses them during its connection process. These IDs are static, that is, they shouldn't change between sessions, yet they should be unique to the hardware.

The select_device method calls the bleak.discover method, which returns a list of BleakDevices advertising their connections within range. The code uses the aioconsole package to asynchronously request the user to select a particular device

    async def select_device(self):
        print("Bluetooh LE hardware warming up...")
        await asyncio.sleep(2.0, loop=loop) # Wait for BLE to initialize.
        devices = await discover()

        print("Please select device: ")
        for i, device in enumerate(devices):
            print(f"{i}: {device.name}")

        response = -1
        while True:
            response = await ainput("Select device: ")
            try:
                response = int(response.strip())
            except:
                print("Please make valid selection.")

            if response > -1 and response < len(devices):
                break
            else:
                print("Please make valid selection.")

After the user has selected a device then the Connection.connected_device is recorded (in case we needed it later) and the Connection.client is set to a newly created BleakClient with the address of the user selected device.


        print(f"Connecting to {devices[response].name}")
        self.connected_device = devices[response]
        self.client = BleakClient(devices[response].address, loop=self.loop)

Utility Methods

Not much to see here, these methods are used to handle timestamps on incoming Bluetooth LE data and clearing the rx buffer.

    def record_time_info(self):
        present_time = datetime.now()
        self.rx_timestamps.append(present_time)
        self.rx_delays.append((present_time - self.last_packet_time).microseconds)
        self.last_packet_time = present_time

    def clear_lists(self):
        self.rx_data.clear()
        self.rx_delays.clear()
        self.rx_timestamps.clear()

Save Incoming Data to File

This is a small class meant to make it easier to record the incoming microphone data along with the time it was received and delay since the last bytes were received.

class DataToFile:

    column_names = ["time", "delay", "data_value"]

    def __init__(self, write_path):
        self.path = write_path

    def write_to_csv(self, times: [int], delays: [datetime], data_values: [Any]):

        if len(set([len(times), len(delays), len(data_values)])) > 1:
            raise Exception("Not all data lists are the same length.")

        with open(self.path, "a+") as f:
            if os.stat(self.path).st_size == 0:
                print("Created file.")
                f.write(",".join([str(name) for name in self.column_names]) + ",\n")
            else:
                for i in range(len(data_values)):
                    f.write(f"{times[i]},{delays[i]},{data_values[i]},\n")

App Loops

I mentioned three "async loops," we've covered the first one inside the Connection class, but outside are the other two.

The user_console_manager() checks to see if the Connection instance has a instantiated a BleakClient and it is connected to a device. If so, it prompts the user for input in a non-blocking manner. After the user enters input and hits return the string is converted into a bytearray using the map(). Lastly, it is sent by directly accessing the Connection.client's write_characteristic method. Note, that's a bit of a code smell, it should be refactored (when I have time).

async def user_console_manager(connection: Connection):
    while True:
        if connection.client and connection.connected:
            input_str = await ainput("Enter string: ")
            bytes_to_send = bytearray(map(ord, input_str))
            await connection.client.write_gatt_char(write_characteristic, bytes_to_send)
            print(f"Sent: {input_str}")
        else:
            await asyncio.sleep(2.0, loop=loop)

The last loop is the one designed to take the application code. Right now, it only simulates application logic by sleeping 5 seconds.

async def main():
    while True:
        # YOUR APP CODE WOULD GO HERE.
        await asyncio.sleep(5)

Closing

Well, that's it. You will have problems, especially if you are using the above code from Linux or Windows. But, if you run into any issues I'll do my best to provide support. Just leave me a comment below.

Getting Started with Bluetooth LE on the Arduino Nano 33 Sense

Thomas Brittain — Sun, 14 Jun 2020 10:00:00 +0000

This article will show you how to program the Arduino Nano BLE 33 devices to use Bluetooth LE.

Introduction

Bluetooth Low Energy and I go way back. I was one of the first using the HM-10 module back in the day. Recently, my mentor introduced me to the Arduino Nano 33 BLE Sense. Great little board--packed with sensors!

Shortly after firing it up, I got excited. I've been wanting to start creating my own smartwatch for a long time (as long the Apple watch has sucked really). And it looks like I wasn't the only one:

The B&ND

This one board had many of the sensors I wanted, all in one little package. The board is a researcher's nocturnal emission.

Of course, my excitement was tamed when I realized there weren't tutorials on how to use the Bluetooth LE portion. So, after a bit of hacking I figured I'd share what I've learned.

Blue on Everything

This article will be part of a series. Here, we will be building a Bluetooth LE peripheral from the Nano 33, but it's hard to debug without having a central device to find and connect to the peripheral.

The next article in this series will show how to use use Python to connect to Bluetooth LE peripherals (above gif). This should allow one to connect to the Nano 33 from a PC. In short, stick with me. I've more Bluetooth LE content coming.

How to Install the Arduino Nano 33 BLE Board

After getting your Arduino Nano 33 BLE board there's a little setup to do. First, open up the Arduino IDE and navigate to the "Boards Manager."

Search for Nano 33 BLE and install the board Arduino nRF528xBoards (MBed OS).

Your Arduino should be ready work with the Nano 33 boards, except BLE. For that, we need another library.

How to Install the ArduinoBLE Library

There are are a few different Arduino libraries for Bluetooth LE--usually, respective to the hardware. Unfortunate, as this means we would need a different library to work with the Bluetooth LE on a ESP32, for example. Oh well. Back to the problem at hand.

The official library for working with the Arduino boards equipped with BLE is:

ArduinoBLE

It works pretty well, though, the documentation is a bit spotty.

To get started you'll need to fire up the Arduino IDE and go to Tools then Manager Libraries...

In the search box that comes up type ArduinoBLE and then select Install next to the library:

That's pretty much it, we can now include the library at the top of our sketch:

#include <ArduinoBLE.h>

And access the full API in our code.

Project Description

If you are eager, feel free to skip this information and jump to the code.

Jump to Code

Before moving on, if the following terms are confusing:

Peripheral
Central
Master
Slave
Server
Client

You might check out EmbeddedFM's explanation:

BLE Terms

I'll be focusing on getting the Arduino 33 BLE Sense to act as a peripheral BLE device. As a peripheral, it'll advertise itself as having services, one for reading, the other for writing.

UART versus Bluetooth LE

Usually when I'm working with a Bluetooth LE (BLE) device I want it to send and receive data. And that'll be the focus of this article.

I've seen this send-n-receive'ing data from BLE referred to as "UART emulation." I think that's fair, UART is a classic communication protocol for a reason. I've like the comparison as a mental framework for our BLE code.

We will have a rx property to get data from a remote device and a tx property where we can send data. Throughout the Arduino program you'll see my naming scheme using this analog. That stated, there are clear differences between BLE communication and UART. BLE is arguably more complex and versatile.

Data from the Arduino Microphone

To demonstrate sending and receiving data we need to data to send. We are going to grab information from the microphone on the Arduino Sense and send it to remote connected device. I'll not cover the microphone code here, as I don't understand it well enough to explain. However, here's a couple reads:

Pulse-Density Modulation (data type output)
Arduino Pulse-Density Modulation Library docs

Code

Time to code. Below is what I hacked together, with annotations from the "gotchas" I ran into.

One last caveat, I used Jithin's code as a base of my project:

Arduino BLE Example Explained Step by Step

Although, I'm not sure any of the original code is left. Cite your sources.

And if you'd rather look at the full code, it can be found at:

Full Code

Initialization

We load in the BLE and the PDM libraries to access the APIs to work with the microphone and the radio hardware.


#include <ArduinoBLE.h>
#include <PDM.h>

Service and Characteristics

Let's create the service. First, we create the name displayed in the advertizing packet, making it easy for a user to identify our Arduino.

We also create a Service called microphoneService, passing it the full Universally Unique ID (UUID) as a string. When setting the UUID there are two options. A 16-bit or a 128-bit version. If you use one of the standard Bluetooth LE Services the 16-bit version is good. However, if you are looking to create a custom service, you will need to explore creating a full 128-bit UUID.

Here, I'm using the full UUIDs but with a standard service and characteristic, as it makes it easier to connect other hardware to our prototype, as the full UUID is known.

If you want to understand UUID's more fully, I highly recommend Nordic's article:

Bluetooth low energy Services, a beginner's tutorial

Anyway, we are going to use the following UUIDs:

Microphone Service = 0x1800 -- Generic Access
rx Characteristic = 0x2A3D -- String
tx Characteristic = 0x2A58 -- Analog

You may notice reading the Bluetooth specifications, there are two mandatory characteristics we should be implementing for Generic Access:

For simplicity, I'll leave these up to the reader. But they must be implemented for a proper Generic Access service.

Right, back to the code.

Here we define the name of the device as it should show to remote devices. Then, the service and two characteristics, one for sending, the other, receiving.

// Device name
const char* nameOfPeripheral = "Microphone";
const char* uuidOfService = "0000181a-0000-1000-8000-00805f9b34fb";
const char* uuidOfRxChar = "00002A3D-0000-1000-8000-00805f9b34fb";
const char* uuidOfTxChar = "00002A58-0000-1000-8000-00805f9b34fb";

Now, we actually instantiate the BLEService object called microphoneService.

// BLE Service
BLEService microphoneService(uuidOfService);

The characteristic responsible for receiving data, rxCharacteristic, has a couple of parameters which tell the Nano 33 how the characteristic should act.

// Setup the incoming data characteristic (RX).
const int RX_BUFFER_SIZE = 256;
bool RX_BUFFER_FIXED_LENGTH = false;

RX_BUFFER_SIZE will be how much space is reserved for the rx buffer. And RX_BUFFER_FIXED_LENGTH will be, well, honestly, I'm not sure. Let me take a second and try to explain my ignorance.

When looking for the correct way to use the ArduinoBLE library, I referred to the documentation:

BLECharacteristic()

There are several different ways to initialize a characteristic, as a single value (e.g., BLEByteCharacteristic, BLEFloatCharacteristic, etc.) or as buffer. I decided on the buffer for the rxCharacteristic. And that's where it got problematic.

Here's what the documentation states regarding initializing a BLECharacteristic with a buffer.

BLECharacteristic(uuid, properties, value, valueSize)
BLECharacteristic(uuid, properties, stringValue)
...
uuid: 16-bit or 128-bit UUID in string format
properties: mask of the properties (BLEBroadcast, BLERead, etc)
valueSize: (maximum) size of characteristic value
stringValue: value as a string

Cool, makes sense. Unfortunately, I never got a BLECharacteristic to work initializing it with those arguments. I finally dug into the actual BLECharacteristic source and discovered their are two ways to initialize a BLECharacteristic:

BLECharacteristic(new BLELocalCharacteristic(uuid, properties, valueSize, fixedLength))
BLECharacteristic(new BLELocalCharacteristic(uuid, properties, value))

I hate misinformation. Ok, that tale aside, back to our code.

Let's actually declare the rx and tx characteristics. Notice, we are using a buffered characteristic for our rx and a single byte value characteristic for our tx. This may not be optimal, but it's what worked.

// RX / TX Characteristics
BLECharacteristic rxChar(uuidOfRxChar, BLEWriteWithoutResponse | BLEWrite, RX_BUFFER_SIZE, RX_BUFFER_FIXED_LENGTH);
BLEByteCharacteristic txChar(uuidOfTxChar, BLERead | BLENotify | BLEBroadcast);

The second argument is where you define how the characteristic should behave. Each property should be separated by the | as they are constants which are being ORed together into a single value (masking).

Here is a list of available properties:

BLEBroadcast -- will cause the characteristic to be advertized
BLERead -- allows remote devices to read the characteristic value
BLEWriteWithoutResponse -- allows remote devices to write to the device without expecting an acknowledgement
BLEWrite -- allows remote devices to write, while expecting an acknowledgement the write was successful
BLENotify -- allows a remote device to be notified anytime the characteristic's value is update
BLEIndicate -- the same as BLENotify, but we expect a response from the remote device indicating it read the value

Microphone

There are two global variables which keep track of the microphone data. The first is a small buffer called sampleBuffer, it will hold up to 256 values from the mic.

The volatile int samplesRead is the variable which will hold the immediate value from the mic sensor. It is used in the interrupt routine vector (ISR) function. The volatile keyword tells the Arduino's C++ compiler the value in the variable may change at any time and it should check the value when referenced, rather than relying on a cached value in the processor (more on volatiles).

// Buffer to read samples into, each sample is 16-bits
short sampleBuffer[256];

// Number of samples read
volatile int samplesRead;

Setup()

We initialize the Serial port, used for debugging.

void setup() {

  // Start serial.
  Serial.begin(9600);

  // Ensure serial port is ready.
  while (!Serial);

To see when the BLE actually is connected, we set the pins connected to the built-in RGB LEDs as OUTPUT.

  // Prepare LED pins.
  pinMode(LED_BUILTIN, OUTPUT);
  pinMode(LEDR, OUTPUT);
  pinMode(LEDG, OUTPUT);

Note, there is a bug in the source code where the LEDR and the LEDG are backwards. You can fix this by searching your computer for ARDUINO_NANO33BLE folder and editing the file pins_arduino.h inside.

Change the following:

#define LEDR        (22u)
#define LEDG        (23u)
#define LEDB        (24u)

#define LEDR        (23u)
#define LEDG        (22u)
#define LEDB        (24u)

And save. That should fix the mappings.

The onPDMdata() is an ISR which fires every time the microphone gets new data. And startPDM() starts the microphone integrated circuit.

  // Configure the data receive callback
  PDM.onReceive(onPDMdata);

  // Start PDM
  startPDM();

Now Bluetooth LE is setup, we ensure the Bluetooth LE hardware has been powered-on within the Nano 33. We set the device name and begin advertizing the service. Then, add the rx and tx characteristics to the microphoneService. Lastly, add the microphoneService to the BLE object.

  // Start BLE.
  startBLE();

  // Create BLE service and characteristics.
  BLE.setLocalName(nameOfPeripheral);
  BLE.setAdvertisedService(microphoneService);
  microphoneService.addCharacteristic(rxChar);
  microphoneService.addCharacteristic(txChar);
  BLE.addService(microphoneService);

Now the Bluetooth LE hardware is turned on, we add callbacks which will fire when the device connects or disconnects. Those callbacks are great places to add notifications, setup, and teardown.

We also add a callback which will fire every time the Bluetooth LE hardware has a characteristic written. This allows us to handle data as it streams in.

  // Bluetooth LE connection handlers.
  BLE.setEventHandler(BLEConnected, onBLEConnected);
  BLE.setEventHandler(BLEDisconnected, onBLEDisconnected);

  // Event driven reads.
  rxChar.setEventHandler(BLEWritten, onRxCharValueUpdate);

Lastly, we command the Bluetooth LE hardware to begin advertizing its services and characteristics to the world. Well, at least +/-30ft of the world.

  // Let's tell devices about us.
  BLE.advertise();

Before beginning the main loop, I like spiting out all of the hardware information we setup. This makes it easy to add it into whatever other applications we are developing., which will connect to the newly initialized peripheral.

  // Print out full UUID and MAC address.
  Serial.println("Peripheral advertising info: ");
  Serial.print("Name: ");
  Serial.println(nameOfPeripheral);
  Serial.print("MAC: ");
  Serial.println(BLE.address());
  Serial.print("Service UUID: ");
  Serial.println(microphoneService.uuid());
  Serial.print("rxCharacteristic UUID: ");
  Serial.println(uuidOfRxChar);
  Serial.print("txCharacteristics UUID: ");
  Serial.println(uuidOfTxChar);


  Serial.println("Bluetooth device active, waiting for connections...");
}

Loop()

The main loop grabs a reference to the central property from the BLE object. It checks if central exists and then it checks if central is connected. If it is, it calls the connectedLight() which will cause the green LED to come on, letting us know the hardware has made a connection.

Then, it checks if there are data in the sampleBuffer array, if so, it writes them to the txChar. After it has written all data, it resets the samplesRead variable to 0.

Lastly, if the device is not connected or not initialized, the loop turns on the disconnected light by calling disconnectedLight().

void loop()
{
  BLEDevice central = BLE.central();

  if (central)
  {
    // Only send data if we are connected to a central device.
    while (central.connected()) {
      connectedLight();

      // Send the microphone values to the central device.
      if (samplesRead) {
        // print samples to the serial monitor or plotter
        for (int i = 0; i < samplesRead; i++) {
          txChar.writeValue(sampleBuffer[i]);      
        }
        // Clear the read count
        samplesRead = 0;
      }
    }
    disconnectedLight();
  } else {
    disconnectedLight();
  }
}

Some may have noticed there is probably an issue with how I'm pulling the data from the sampleBuffer, as I've just noticed it myself writing this article, it may have a condition where the microphone's ISR is called in the middle of writing the buffer to the txChar. If I've need to fix this, I'll update this article.

Ok, hard part's over, let's move on to the helper methods.

Helper Methods

startBLE()

The startBLE() function initializes the Bluetooth LE hardware by calling the begin(). If it is unable to start the hardware, it will state so via the serial port, and then stick forever.

/*
 *  BLUETOOTH
 */
void startBLE() {
  if (!BLE.begin())
  {
    Serial.println("starting BLE failed!");
    while (1);
  }
}

onRxCharValueUpdate()

This method is called when new data is received from a connected device. It grabs the data from the rxChar by calling readValue and providing a buffer for the data and how many bytes are available in the buffer. The readValue method returns how many bytes were read. We then loop over each of the bytes in our tmp buffer, cast them to char, and print them to the serial terminal. This is pretty helpful when debugging.

Before ending, we also print out how many bytes were read, just in case we've received data which can't be converted to ASCII. Again, helpful for debugging.

void onRxCharValueUpdate(BLEDevice central, BLECharacteristic characteristic) {
  // central wrote new value to characteristic, update LED
  Serial.print("Characteristic event, read: ");
  byte tmp[256];
  int dataLength = rxChar.readValue(tmp, 256);

  for(int i = 0; i < dataLength; i++) {
    Serial.print((char)tmp[i]);
  }
  Serial.println();
  Serial.print("Value length = ");
  Serial.println(rxChar.valueLength());
}

LED Indicators

Not much to see here. These functions are called when our device connects or disconnects, respectively.

void onBLEConnected(BLEDevice central) {
  Serial.print("Connected event, central: ");
  Serial.println(central.address());
  connectedLight();
}

void onBLEDisconnected(BLEDevice central) {
  Serial.print("Disconnected event, central: ");
  Serial.println(central.address());
  disconnectedLight();
}

/*
 * LEDS
 */
void connectedLight() {
  digitalWrite(LEDR, LOW);
  digitalWrite(LEDG, HIGH);
}

void disconnectedLight() {
  digitalWrite(LEDR, HIGH);
  digitalWrite(LEDG, LOW);
}

Microphone

I stole this code from Arduino provided example. I think it initializes the PDM hardware (microphone) with a 16khz sample rate.

/*
 *  MICROPHONE
 */
void startPDM() {
  // initialize PDM with:
  // - one channel (mono mode)
  // - a 16 kHz sample rate
  if (!PDM.begin(1, 16000)) {
    Serial.println("Failed to start PDM!");
    while (1);
  }
}

Lastly, the onPDMData callback is fired whenever their are data available to be read. It checks how many bytes their are available by calling available() and reads that number of bytes into the buffer. Lastly, given the data are int16, it divides the number of bytes by 2 as this is the number of samples read.

void onPDMdata() {
  // query the number of bytes available
  int bytesAvailable = PDM.available();

  // read into the sample buffer
  int bytesRead = PDM.read(sampleBuffer, bytesAvailable);

  // 16-bit, 2 bytes per sample
  samplesRead = bytesRead / 2;
}

Final Thoughts

Bluetooth LE is powerful--but tough to get right. To be clear, not saying I've gotten it right here, but I'm hoping I'm closer. If you find any issues please leave me a comment or send me an email and I'll get them corrected as quick as I'm able.

Arduino RAMPs 1.4 Custom Firmware

Thomas Brittain — Sun, 10 May 2020 10:00:00 +0000

This article is part of a series documenting an attempt to create a LEGO sorting machine. This portion covers the Arduino Mega2560 firmware I've written to control a RAMPS 1.4 stepper motor board.

A big thanks to William Cooke, his wisdom was key to this project. Thank you, sir!

William Cooke

Goal

To move forward with the LEGO sorting machine I needed a way to drive a conveyor belt. Stepper motors were a fairly obvious choice. They provide plenty of torque and finite control. This was great, as several other parts of the LEGO classifier system would need steppers motors as well-e.g.,turn table and dispensing hopper. Of course, one of the overall goals of this project is to keep the tools accessible. After some research I decided to meet both goals by purchasing an Ardunio / RAMPs combo package intended for 3D printers.

Amazon RAMPs Kits

At the time of the build, these kits were around $28-35 and included:

Arduino Mega2560
4 x Endstops
5 x Stepers Drivers (A4988)
RAMPSs 1.4 board
Display
Cables & wires

Seemed like a good deal. I bought a couple of them.

I would eventually need:

3 x NEMA17 stepper motors
12v, 10A Power Supply Unit (PSU)

Luckily, I had the PSU and a few stepper motors lying about the house.

Physical Adjustments

Wiring everything up wasn't too bad. You follow about any RAMPs wiring diagram. However, I did need to make two adjustments before starting on the firmware.

First, underneath each of the stepper drivers there are three drivers for setting the microsteps of the respective driver. Having all three jumpers enables maximum microsteps, but would cause the speed of the motor to be limited by the clock cycles of the Arduino--more on that soon.

I've also increased the amperage to the stepper. This allowed me to drive the entire belt from one NEMA17.

To set the amperage, get a small phillips screwdriver, two alligator clips, and a multimeter. Power on your RAMPs board and carefully attach the negative probe to the RAMPs GND. Attach the positive probe to an alligator clip and attach the other end to the shaft of your screwdriver. Use the screwdriver to turn the tiny potentiometer on the stepper driver. Watch the voltage on the multimeter--we want to use the lowest amperage which effectively drives the conveyor belt. We are watching the voltage, as it is related to the amperage we are feeding the motors.

current_limit = Vref x 2.5

Anyway, I found the lowest point for my motor, without skipping steps, was around ~0.801v.

current_limit = 0.801 x 2.5
current_limit = 2.0025

The your current_limit will vary depending on the drag of your conveyor belt and the quality of your stepper motor. To ensure a long-life of your motor, do not set the amperage higher than needed to do the job.

Arduino Code

When I bought the RAMPs board I started thinking, "I should see if we could re-purpose Marlin to drive the conveyor belt easily." I took one look at the source and said, "Oh hell no." Learning how to hack Marlin to drive a conveyor belt seemed like learning heart surgery to hack your heart into a gas pump. So, I decided roll my own RAMPs firmware.

My design goals were simple:

Motors operate independently
Controlled with small packets via UART
Include four commands: motor select, direction, speed, duration

That's it. I prefer to keep stuff as simple as possible, unless absolutely necessary.

I should point out, this project builds on a previous attempt at firmware:

Raspbery Pi, Arduino, RAMPS Turntable

But that code was flawed. It was not written with concurrent and independent motor operation in mind. The result, only one motor could be controlled at a time.

Ok, on to the new code.

Main

The firmware follows this procedure:

Check if a new movement packet has been received.
Decode the packet
Load direction, steps, and delay (speed) into the appropriate motor struct.
Check if a motor has steps to take and the timing window for the next step is open.
If a motor has steps waiting to be taken, move the motor one step and decrement the respective motor's step counter.
Repeat forever.

/* Main */
void loop()
{
  if (rxBuffer.packet_complete) {
    // If packet is packet_complete
    handleCompletePacket(rxBuffer);
    // Clear the buffer for the next packet.
    resetBuffer(&rxBuffer);
  }

  // Start the motor
  pollMotor();
}

serialEvent

Some code not in the main loop is the the UART RX handler. It is activated by an RX interrupt. If the interrupt fires, the new data is quickly loaded into the rxBuffer. If the incoming data contains a 0x03 character, this signals the packet is complete and ready to be decoded.

Here's the packet template:

MOTOR_PACKET = CMD_TYPE MOTOR_NUM DIR STEPS_1 STEPS_2 MILLI_BETWEEN 0x03

Each motor movement packet consists of seven bytes and five values:

CMD_TYPE = drive or halt
MOTOR_NUM = the motor selected X, Y, Z, E0, E1
DIR = direction of the motor
STEPS_1 = the high 6-bits of of steps to take
STEPS_2 = the low 6-bits of steps to take
MILLI_BETWEEN = number of milliseconds between each step (speed control)
0x03 = this signals the end of the packet (ETX)

Each of these bytes are encoded by left-shifting the bits by two. This means each of the bytes in the packet can only represent 64 values (2^6 = 64).

Why add this complication? Well, we want to be able to send commands to control the firmware, rather than the motors. The most critical is knowing when the end of a packet is reached. I'm using the ETX char, 0x03 to signal the end of a packet. If we didn't reserve the 0x03 byte then what happens if we send command to the firmware to move the motor 3 steps? Nothing good.

Here's the flow of a processed command:

1. CMD_TYPE       = DRIVE (0x01)
2. MOTOR_NUM      = X     (0x01)
3. DIR            = CW    (0x01)
4. STEPS          = 4095  (0x0FFF)
5. MILLI_BETWEEN  = 5ms   (0x05)
6. ETX            = End   (0x03)

Note, the maximum value of the STEPS byte is greater than 8-bits. To handle this, we break it into two bytes of 6-bits.

1. CMD_TYPE       = DRIVE (0x01)
2. MOTOR_NUM      = X     (0x01)
3. DIR            = CW    (0x01)
4. STEPS_1        = 3F
5. STEPS_2        = 3F
5. MILLI_BETWEEN  = 5     (0x05)
6. ETX            = End   (0x03)

Here's a sample motor packet before encoding:

uint8_t packet[7] = {0x01, 0x01, 0x01, 0x3F, 0x3F, 0x05, 0x03}

Now, we have to shift all of the bytes left by two bits, this will ensure 0x00 through 0x03 are reserved for meta-communication.

This process is a bit easier to see in binary:

Before shift:

1. CMD_TYPE       = 0000 0001
2. MOTOR_NUM      = 0000 0001
3. DIR            = 0000 0001
4. STEPS_1        = 0011 1111
5. STEPS_2        = 0011 1111
5. MILLI_BETWEEN  = 0000 0101
6. ETX            = 0000 0011

After shift:

1. CMD_TYPE       = 0000 0100
2. MOTOR_NUM      = 0000 0100
3. DIR            = 0000 0100
4. STEPS_1        = 1111 1100
5. STEPS_2        = 1111 1100
5. MILLI_BETWEEN  = 0001 0100
6. ETX            = 0000 0011

And back to hex:

1. CMD_TYPE       = 0x04
2. MOTOR_NUM      = 0x04
3. DIR            = 0x04
4. STEPS_1        = 0xFC
5. STEPS_2        = 0xFC
5. MILLI_BETWEEN  = 0x14
6. ETX            = 0x03

And after encoding:

uint8_t packet[7] = {0x04, 0x04,  0x04, 0xFC, 0xFC, 0x14, 0x03}

Notice the last byte is not encoded, as this is a reserved command character.

Here are the decode and encode functions. Fairly straightforward bitwise operations.

uint8_t decode(uint8_t value) {
  return (value >> 2) & 0x3F;
}

uint8_t encode(uint8_t value) {
  return (value << 2) & 0xFC;
}

And the serial handling as a whole:

void serialEvent() {

  // Get all the data.
  while (Serial.available()) {

    // Read a byte
    uint8_t inByte = (uint8_t)Serial.read();

    if (inByte == END_TX) {
      rxBuffer.packet_complete = true;
    } else {
      // Store the byte in the buffer.
      inByte = decodePacket(inByte);
      rxBuffer.data[rxBuffer.index] = inByte;
      rxBuffer.index++;
    }
  }
}

handleCompletePacket

When a packet is waiting to be decoded, the handleCompletePacket() will be executed. The first thing the method does is check the packet_type. Keeping it simple, there are only two and one is not implemented yet (HALT_CMD)

#define DRIVE_CMD       (char)0x01
#define HALT_CMD        (char)0x02

Code is simple. It unloads the data from the packet. Each byte in the incoming packet represents different portions of the the motor move command. Each byte's value is loaded into local a variable.

The only note worth item is the steps bytes, as the steps consistent of a 12-bit value, which is contained in the 6 lower bits of two bytes. The the upper 6-bits are left-shifted by 6 and we OR them with lower 6-bits.

uint16_t steps = ((uint8_t)rxBuffer.data[3] << 6)  | (uint8_t)rxBuffer.data[4];

If the packet actually contains steps to move we call the setMotorState(), passing all of the freshly unpacked values as arguments. This function will store those values until the processor has time to process the move command.

Lastly, the handleCompletePacket() sends an acknowledgment byte (0x02).

void handleCompletePacket(BUFFER rxBuffer) {

    uint8_t packet_type = rxBuffer.data[0];

    switch (packet_type) {
      case DRIVE_CMD:

          // Unpack the command.
          uint8_t motorNumber =  rxBuffer.data[1];
          uint8_t direction =  rxBuffer.data[2];
          uint16_t steps = ((uint8_t)rxBuffer.data[3] << 6)  | (uint8_t)rxBuffer.data[4];
          uint16_t microSecondsDelay = rxBuffer.data[5] * 1000; // Delay comes in as milliseconds.

          if (microSecondsDelay < MINIMUM_STEPPER_DELAY) { microSecondsDelay = MINIMUM_STEPPER_DELAY; }

          // Should we move this motor.
          if (steps > 0) {
            // Set motor state.
            setMotorState(motorNumber, direction, steps, microSecondsDelay);
          }

          // Let the master know command is in process.
          sendAck();
        break;
      default:
        sendNack();
        break;
    }
}

setMotorState

Each motor has a struct MOTOR_STATE representing its current state.

struct MOTOR_STATE {
  uint8_t direction;
  uint16_t steps;
  unsigned long step_delay;
  unsigned long next_step_at;
  bool enabled;
};

There are five motor MOTOR_STATEs which are initialized a program start, one for each motor (X, Y, Z, E0, E1).

MOTOR_STATE motor_n_state = { DIR_CC, 0, 0, SENTINEL, false };

And whenever a valid move packet is processed, as we saw above, the setMotorState() is responsible for updating the respective MOTOR_STATE struct.

Everything in this function is intuitive, but the critical part for understanding how the entire program comes together to ensure the motors are able to move around at different speeds, directions, all simultaneously is:

motorState->next_step_at = micros() + microSecondsDelay;

micros() is built into the Arduino ecosystem. It returns the number of microseconds since hte program started.

micros()

The next_step_at is set for when we want the this specific motor to take its next step. We get this number as the number of seconds from the programs start up, plus the delay we want between each step. This may be a bit hard to understand, however, like stated, it's key to the entire program working well. Later, we will update motorState->next_step_at with when this motor should take its next step. This "time to take the next step" threshold allows us to avoid creating a blocking loop on each motor.

For example, the wrong way may look like:

void main_loop() {

  // motor_x
  for(int i = 0; i < motor_x_steps; i++) {
    digitalWrite(motor.step_pin, HIGH);
    delayMicroseconds(motor.pulse_width_micros);
    digitalWrite(motor.step_pin, LOW);
  }

  // motor_y
  for(int i = 0; i < motor_y_steps; i++) {
    digitalWrite(motor.step_pin, HIGH);
    delayMicroseconds(motor.pulse_width_micros);
    digitalWrite(motor.step_pin, LOW);
  }

  // Etc
}

As you might have noticed, the motor_y would not start moving until motor_x took all of its steps. That's no good.

Anyway, keep this in mind as we start looking at the motor movement function--coming up next.

void setMotorState(uint8_t motorNumber, uint8_t direction, uint16_t steps, unsigned long microSecondsDelay) {

    // Get reference to motor state.
    MOTOR_STATE* motorState = getMotorState(motorNumber);

    ...

    // Update with target states.
    motorState->direction = direction;
    motorState->steps = steps;
    motorState->step_delay = microSecondsDelay;
    motorState->next_step_at = micros() + microSecondsDelay;
}

pollMotor

Getting to the action. Inside the main loop there is a call to pollMotor(), which loops all of the motors, checking if the motorState has steps to take. If it does, it takes one step and sets when it should take its next step:

motorState->next_step_at += motorState->step_delay;

This is key to all motors running together. By setting when each motor should take its next step, it frees microcontroller to do other work. And the microcontroller is quick, it can do its other work fast and come back and check if each motor needs to take its next step several hundred times before any motor needs to move again. Of course, it all depends on how fast you want your motors to go. For this project, it works like a charm.

/* Write to MOTOR */
void pollMotor() {
    unsigned long current_micros = micros();
    // Loop over all motors.
    for (int i = 0; i < int(sizeof(all_motors)/sizeof(int)); i++)
    {
      // Get motor and motorState for this motor.
      MOTOR motor = getMotor(all_motors[i]);
      MOTOR_STATE* motorState = getMotorState(all_motors[i]);

      // Check if motor needs to move.
      if (motorState->steps > 0) {

        // Initial step timer.
        if (motorState->next_step_at == SENTINEL) {
          motorState->next_step_at = micros() + motorState->step_delay;
        }

        // Enable motor.
        if (motorState->enabled == false) {
          enableMotor(motor, motorState);
        }

        // Set motor direction.
        setDirection(motor, motorState->direction);

        unsigned long window = motorState->step_delay;  // we should be within this time frame

        if(current_micros - motorState->next_step_at < window) {         
            writeMotor(motor);
            motorState->steps -= 1;
            motorState->next_step_at += motorState->step_delay;
        }
      }

      // If steps are finished, disable motor and reset state.
      if (motorState->steps == 0 && motorState->enabled == true ) {
        disableMotor(motor, motorState);
        resetMotorState(motorState);
      }
    }
}

Summary

We have the motor driver working. We now can control five stepper motors' speed and number steps, all independent of one another. And the serial communication protocol allows us to send small packets to each specific motor, telling how many steps to take and how quickly.

Next, we need a controller on the other side of the UART--a master device. This master device will coordinate higher level functions with the motor movements. I've already started work on this project, it will be a asynchronous Python package. Wish me luck.

Programming Arduino from Raspberry Pi Command Line

Thomas Brittain — Fri, 08 May 2020 10:00:00 +0000

I've been working on an automated system for sorting LEGOs. It seems like a simple enough task, however, the nuances of implementation are ugly. I have prototypical solutions for a few of these challenges, such as identifying the LEGO and creating training data for supporting the classifier. But one of the trickier problems has vexed me: How do we get the LEGO from a container to the classifier?

The answer is obvious, right? A conveyor belt. They are ubiquitous in manufacturing, so I thought, "Simple. I'll toss a conveyor belt together real quick and that'll solve that." Hah.

After a month and a half of failed attempts, I've eventually created a working prototype.

The system consists of 5 parts:

Raspberry Pi
Arduino Mega2560
RAMPs 1.4 with A4988s
Conveyor belt
NEMA17 Stepper Motor and Mount

Covering all parts will be too much for one article, so in this article I'll focus on the setting up the environment and in a subsequent article I'll review the firmware, software, and physical build.

Remote VSCode (sshfs)

I hate trying to program on computers other than my workstation; I've also found it problematic to write a program for a Raspberry Pi on a PC. To get the best of both worlds I use sshfs. It lets me mount Raspberry Pi folders as local folder, enabling editing Raspberry Pi files from my workstation. Nice!

The setup is pretty simple, depending on your workstation's OS.

Luckily, DigitalOcean has already put together a multi-OS walkthrough of setting up sshfs

How To Use SSHFS to Mount Remote File Systems Over SSH

Once you have sshfs setup, you can create a directory and mount the entire Raspberry Pi.

For me, running Linux Mint, it was:

sshfs pi@192.168.1.x:/home/pi ~/rpi

A few notes on the above command:

The 192.168.1.x should be replaced with the ip of your Raspberry Pi
~/rpi is the local directory where you are going to mount the Raspberry Pi.

If all goes well, you should be able to open your Raspberry Pi files in Visual Studio Code (or IDE of choice) by navigating to the ~/rpi directory.

To run files, you still have to ssh into the Pi. I usually do this by creating an integrated terminal in Visual Studio Code.

Arduino CLI Setup

Now I had a way to edit Raspberry Pi files on my PC, but I still needed to be able to connect my Arduino to the Pi and program it from my workstation. The route people seem to use for remote programming is using a VNC program, like RealVNC, to access the Pi's desktop remotely. Gross. Give my command line back.

Enter Arduino's command line interface (CLI).

Now I had all the needed pieces to make for comfortable coding:

Code the Pi from my workstation using VSCode
Any software written would be native to the Pi's ARM core
I could upload new firmware from the Raspberry Pi to the Arduino; enabling quick iterations

I was pretty excited. I just need to put the pieces together.

Python Convenience Scripts

First, I had to get the Arduino CLI running on the Raspberry Pi. That turned out pretty painless. In fact, I turned the installation into a Python script for you.

Script for Installing Arduino CLI

You can download my entire setup using git. From your Raspberry Pi's command line run:

git clone https://github.com/ladvien/ramps_controller.git
cd ramps_controller
python3 arduino-cli_setup.py

Or if you prefer to add it to your own Python script:

# 
import os, sys

# Install arduino-cli
os.system('curl -fsSL https://raw.githubusercontent.com/arduino/arduino-cli/master/install.sh | BINDIR=/bin sh')

# Configure arduino-cli
os.system('arduino-cli config init')

# Update the core.
os.system('arduino-cli core update-index')

# Add Arduino AVR and Mega cores.
os.system('arduino-cli core install arduino:avr')
os.system('arduino-cli core install arduino:megaavr')

The installation script downloads the Arduino CLI and installs it. It then updates the Arduino core libraries. Lastly, it ensures the AVR and Arduino Mega AVR cores are installed as well.

Script for Uploading using Arduino CLI

You should now be set to compile and install new firmware directly from the Raspberry Pi to the Arduino Mega2560. I've also created a firmware installation script, which eases installing new code.

python3 install_sketch.py

At the root of the install script are an Arduino CLI command to compile and then upload:

# Compile
os.system('arduino-cli compile -b arduino:avr:mega ramps_sketch')

# Upload
command_str = f'arduino-cli -v upload -p {write_port} --fqbn arduino:avr:mega ramps_sketch'
os.system(command_str)

Feel free to hack the script for other projects. You can replace the arduino:avr:mega with other chipsets and program tons of different devices using the same method. And the ramps_sketch refers to the program you want to upload. It is a folder containing and .ino file of the same name, which is the program you want to upload to the Arduino

Here's an action shot:

A couple of notes, if you have trouble running the install script here are two issues I ran into:

pyserial

The install script uses Python to lookup what USB-serial bridges you have attached to your Pi. This Python relies on the pyserial package. Make sure it is installed using:

pip install pyserial

Access to USB

For the install script to work correctly, the executing user must have access to the USB-serial devices. This is known as the dialout group. The right way of doing this is by adding the permission to the user.

sudo adduser $USER dialout

If this fails, you can use the "wrong" way and just execute the ./install.py script using sudo.

python3 install_sketch.py

Ok, that's it for now. I'll tackle the firmware next.

I've you have any trouble with the code, or have questions, just leave a comment below.

Install Tensorflow and OpenCV on Raspberry Pi

Thomas Brittain — Thu, 30 Jan 2020 10:00:00 +0000

This post shows how to setup a Raspberry Pi 3B+ for operating a Tensorflow CNN model using a Pi Camera Module v2.0.

Raspberry Pi Setup

I will be focusing on the Raspberry Pi 3B+, but don't worry if you are using a different Pi. Just let me know in the comments below and I'll try to get instructions for your particular Pi added.

Step #1: Download Raspbian Buster with desktop and recommended software

Step #2: Write the image to a 8gb (or greater) SD card. I use Etcher.

Step #3: Once the image is finished, and before you plug the card into the Pi, open the SD card and create a file called ssh. No extension and nothing inside. This will enable ssh on boot.

Step #4: Plug the card in to the Pi.
Step #5: Plug a LAN cable into the Pi
Step #6: Attach your PiCam.

Note, there are two plugs the PiCamera will mate with. To save frustration:

Step #7: Turn the Pi on.
Step #8: Find the ip of your Pi and ssh into it with the following.

ssh pi@your_pi_ip

The password will be raspberry

The easiest way to find your Pi's ip is to login into your router. Usually, you can login into your router by opening a webbrowser on your PC and typing 192.168.1.1. This is the "home" address. You should then be prompted to login to the router. On your router's web interface there should be a section for "attached devices." You can find your Pi's ip there. If many are listed, you can turn off your Pi and see which ip goes away. That was probably the Pi's ip.

Step #9: Once on the Pi, run the following

sudo raspi-config

This should open a old school GUI.

Enable the following under Interfacing Options

Camera
VNC

The camera will allow us to use the PiCamera and VNC will allow us to open a a remote desktop environment, which should make it easier to adjust the PiCamera.

(Optional) When working with a remote desktop environment, too high of a resolution can cause responsiveness issues with the VNC client (RealVNC). To prevent this, the Raspbian setup automatically adjusts the Pi resolution to the lowest. Unfortunately, I find this troublesome when trying to do computer vision stuff from the Pi. The following will allow you to adjust the resolution--just keep in mind, if it's too high there could be trouble. Oh, one note here, this is the main reason I'm using a LAN connection to my Pi, as it allows greater throughput than WiFi.

Update! Apparently, if you raise your Pi's resolution too high, then you will not be able to start your PiCam from Python. This is due to the PicCam buffering frames in the GPU memory of the Pi. Of course, you could increase the GPU's memory through raspi-config (it defaults to 128, max is 256). Of course, then you've less RAM to put in your Tensorflow model.

My opinion, raise the Pi's screen resolution just high enough to make it easy for debugging the Pi cam. And when you get ready to "productionize" your Pi, drop the resolution to the lowest.

Ok, if you still want to, here's how to raise the Pi's resolution.

Still in raspi-config open Advanced Options. Navigate to Resolution and change it to what you'd like. (I'm going with the highest).

Once you've finished setting these options, exit. At the end it will ask if you want to reboot, say "Yes."

Step #10: Download and install RealVNC Viewer.

Step #11: Open RealVNC and set the ip to your Pi. Don't include your user name, like we did when ssh'ing, because RealVNC is about to ask us for it. Once you've typed in the ip hit "Enter" or "Return."

Step #12: RealVNC will warn you about singing into your Pi, as it's not a credentialed source. No worries. Hit continue.

Note, if you're on a Mac, it's going to ask you to give RealVNC access to keys or something. (Shesh, Mac, thank you for the security, but, well, shesh.)

Step #13: Enter your credentials.

username: pi
password: raspberry

Step #14: This should open your Pi's desktop environment. It will ask you a few setup questions, go ahead and take care of it. Note, if you change your password, you will need to update RealVNC (if you had it "Remember My Password").

Tensorflow Setup

Here's where it gets real.

Open terminal, either in the VNC Pi desktop, or through ssh. Then enter the following commands.

pip3 install pip3 install https://github.com/lhelontra/tensorflow-on-arm/releases/download/v1.14.0-buster/tensorflow-1.14.0-cp37-none-linux_armv7l.whl

The above installs a Tensorflow 1.14 for Python 3.7.x on the Raspberry Pi 3b+ from ihelontra's private Tensorflow ARM builds. I've found this better, as Google seems to break the installs often.

If you want another combination of Tensorflow, Python, and Pi, you can see ihelontra's other whl files:

tensorflow-on-arm

OpenCV Setup

Tensorflow will allow us to open a model, however, we will need to feed the model image data captured from the PiCamerae. The easiest way to do this, at least I've found so far, is using OpenCV.

Of course, it can be tricky to setup. The trickiest part? If you Google how to set it up on Raspberry Pi you will get tons of misinformation. In all due fairness, it once was good information--as you had to build OpenCV for the Pi, which took a lot of work. But, now days, you can install it using the build in Linux tools.

Ok, back at the Pi's command prompt:

# Install OpenCV
sudo apt-get install python3-opencv

At of time writing, the above command will install OpenCV 3.2. Of course, the newest version is 4.0, but we don't need that. Trust me, unless you've a reason to be using OpenCV 4.0 or greater, I'd stick with the Linux repos. Building OpenCV can be a time consuming pain.

There's one other handy package which will make our work easier: imutils.

Let's install it.

pip3 intall imutils

Using Tensorflow to Classify Images on an RPi.

Now the payoff.

I've prepared a Python script which loads a test model, initializes the Pi camera, captures a stream of images, each image is classified by the Tensorflow model, and the prediction is printed at the top left of the screen. Of course, you can switch out the entire thing be loading a different model and corresponding json file containg the class labels (I've described this in an earlier article.)

Let's download the script and test our build:

cd ~
git clone https://github.com/Ladvien/rpi_tf_and_opencv
cd rpi_tf_and_opencv

Ok! Moment of truth. Let's execute the script.

python3 eval_rpi.py

If all goes well, it will take a minute or two to initialize and you should see something similar to the following:

Troubleshooting

If you are using a different PiCamera module than the v2.0 you will most likely need to adjust the resolution settings at the top of the script:

view_width               = 3280
view_height              = 2464

If you clone the repo in a different directory besides the /pi/home directory, then you will need to change the model path at the top of the file:

model_save_dir           = '/home/pi/rpi_tf_and_opencv/'

Any other issues, feel free to ask questions in the comments. I'd rather troubleshoot a specific issue rather than try to cover every use case.

Generating LEGO Images for Training a CNN

Thomas Brittain — Sun, 27 Oct 2019 10:00:00 +0000

After having success with training a CNN on our initial dataset, we decided to up the game on generating training images. My buddy Rockets built a nice little turntable and ordered a couple of NEMA17s for each of us. His idea was we could both start generating training images.

I asked if he would be OK with me ordering some RAMPs boards and programming them to synchronize with the PiCamera. I figured, it would probably be better for reproducibility if we had solid hardware, with custom firmware and software.

After a few hours of coding over a couple of weeks I was able to control the RAMPs within a Python script from either the Raspberry Pi or a desktop computer.

Turn Table Controller Code

I've listed the code parts below with a brief explanation--just in case someone would like to hack them for other projects.

Minimum Viable Hack

Warning words, I'm an advocate of the minimum viable product, especially, when it comes to my personal hacking time. I refer to this as the minimum viable hack. That stated, there are known issues in the code below. But! It does the job--so I've not addressed them.

Here are a few:

The value 0x0A (\n) value is not handled as part of packet (e.g., if MILLI_BETWEEN = 10 bad things will happen).
The motors are always on (reduces motor life).
Pulse width is not adjustable without firmware update.
The Python code is blocking. This makes the halt feature on the Arduino Mega side fairly useless.
Only RAMPs motor X is setup (this one I will address later, as we will need several drivers before the end of this project).

RAMPS Code

To move the turn table we used a RAMPs 1.4 board:

RAMPS Kit (Amazon)

Getting things going was straightforward. I put together the hardware, installed the Arduino IDE, and looked-up the pinout for the RAMPs controller.

I wrote the firmware to receive serial commands as a packet. The packet structure (at time of writing) looks like this:

MOTOR_PACKET = 0x01 0x01 0x00 0x03 0xE8 0x05 0x0A
INDEX        =  1    2     3    4    5    6   7

first_byte = This indicates what sort of packet type. Right now, there is only one, but I figure we might want to control other I/O on the Arduino later.
second_byte = the motor selected 1-5 (X, Y, Z, E1, E2).
third_byte = Motor direction, 0x00 is clockwise and 0x01 is counter-clockwise.
fourth_byte = first chunk of the steps.
fifth_byte = second chunk of the steps. The steps variable tells the motor how many steps to move before stopping.
sixth_byte = delay between steps in milliseconds.
seventh_byte = the end-of-transmission (EOT) character. I've used \n.

When the code receives an EOT character, it parses the packet and calls the writeMotor(). This function loops through the number of steps, delaying between each. Each loop, the function checks if a halt command has been received. If it has, it stops the motor mid-move.

Again, this code isn't perfect. Far from it. But it does the job.

#include <avr/interrupt.h> 
#include <avr/io.h> 
// https://reprap.org/mediawikihttps://ladvien.com/images/f/f6/RAMPS1.4schematic.png
// https://reprap.org/forum/read.php?219,168722

// TODO: Pulse width set by initialization.
// TODO: Setup all motors to be selected by master.
// TODO: Add a timer to shutdown motors after threshold.
//       And keep motor enabled until threshold has been met.
// TODO: Handle 0x0A values as part of packet (e.g., if MILLI_BETWEEN = 10).
// TODO: Add a "holding torque" feature; making it so motors never disable.

// For RAMPS 1.4
#define X_STEP_PIN         54
#define X_DIR_PIN          55
#define X_ENABLE_PIN       38
#define X_MIN_PIN           3
#define X_MAX_PIN           2

#define Y_STEP_PIN         60
#define Y_DIR_PIN          61
#define Y_ENABLE_PIN       56
#define Y_MIN_PIN          14
#define Y_MAX_PIN          15

#define Z_STEP_PIN         46
#define Z_DIR_PIN          48
#define Z_ENABLE_PIN       62
#define Z_MIN_PIN          18
#define Z_MAX_PIN          19

#define E_STEP_PIN         26
#define E_DIR_PIN          28
#define E_ENABLE_PIN       24

#define SDPOWER            -1
#define SDSS               53
#define LED_PIN            13

#define FAN_PIN            9

#define PS_ON_PIN          12
#define KILL_PIN           -1

#define HEATER_0_PIN       10
#define HEATER_1_PIN       8
#define TEMP_0_PIN         13   // ANALOG NUMBERING
#define TEMP_1_PIN         14   // ANALOG NUMBERING

#define MOTOR_X         0x01
#define MOTOR_Y         0x02
#define MOTOR_Z         0x03
#define MOTOR_E1        0x04
#define MOTOR_E2        0x05

#define DRIVE_CMD       (char)0x01
#define HALT_CMD        (char)0x0F
#define DIR_CC          (char)0x00
#define DIR_CCW         (char)0x01

#define COMPLETED_CMD   (char)0x07
#define END_TX          (char)0x0A
#define ACK             (char)0x06 // Acknowledge
#define NACK            (char)0x15 // Negative Acknowledge

// Determine the pulse width of motor.
#define MOTOR_ANGLE           1.8
#define PULSE_WIDTH_MICROS    360 / MOTOR_ANGLE

#define RX_BUFFER_SIZE 16

/*
  MOTOR_NUM:
      X     = 0
      Y     = 1
      Z     = 2
      E1    = 3
      E2    = 4

  PACKET_TYPES
      0x01 = motor_write
      0x02 = motor_halt

  DIRECTION
      0x00 = CW
      0x01 = CCW

  MOTOR MOVE PROTOCOL:
                       0               1     2     3        4       5         6
  MOTOR_PACKET = PACKET_TYPE_CHAR MOTOR_NUM DIR STEPS_1 STEPS_2 MILLI_BETWEEN \n
  MOTOR_PACKET =    01                01    00    03     E8        05         0A
  MOTOR_PACKET =    0x 01010003E8050A

  HALT         = 0x0F
*/

/* Create a structure for the motors
 *  direction_pin = pin to control direction of stepper.
 *  step_pin      = pin to control the steps.
 *  enable_pin    = pin to enable motor.
 */
struct MOTOR {
  uint8_t direction_pin;
  uint8_t step_pin;
  uint8_t enable_pin;
  uint8_t pulse_width_micros;
};

struct BUFFER {
  uint8_t data[RX_BUFFER_SIZE];
  uint8_t bufferSize;
  uint8_t index;
  boolean packetComplete;
  uint8_t shutdownThreshold;
};

/* Initialize motors */
MOTOR motorX = {
      X_DIR_PIN,
      X_STEP_PIN,
      X_ENABLE_PIN,
      PULSE_WIDTH_MICROS
};

// Urgent shutdown.
volatile boolean halt = false;
volatile static bool triggered;

/* Initialize RX buffer */
BUFFER rxBuffer;;

/* Initialize program */
void setup()
{
  Serial.begin(115200);

  // Initialize the structures
  motorSetup(motorX);
  rxBuffer.bufferSize = RX_BUFFER_SIZE;

  // Disable holding torque.
  digitalWrite(motorX.enable_pin, HIGH);
}

/* Main */
void loop()
{
  // If packet is packetComplete
  if (rxBuffer.packetComplete) {

    uint8_t packet_type = rxBuffer.data[0];

    switch (packet_type) {
      case DRIVE_CMD:
        {
          // Unpack the command.
          uint8_t motorNumber =  rxBuffer.data[1];
          uint8_t direction =  rxBuffer.data[2];
          uint16_t steps = ((uint8_t)rxBuffer.data[3] << 8)  | (uint8_t)rxBuffer.data[4];
          uint8_t milliSecondsDelay = rxBuffer.data[5];

          // Let the master know command is in process.
          sendAck();

          // Start the motor
          writeMotor(motorX, direction, steps, milliSecondsDelay);
        }
        break;
      default:
        sendNack();
        break;
    }
    // Clear the buffer for the nexgt packet.
    resetBuffer(&rxBuffer);
  }
}

/*  ############### MOTORS ############### */

/* Method for initalizing MOTOR */
void motorSetup(MOTOR motor) {

  // Setup motor pins
  pinMode(motor.direction_pin, OUTPUT);
  pinMode(motor.step_pin, OUTPUT);
  pinMode(motor.enable_pin, OUTPUT);

}

/* Write to MOTOR */
void writeMotor(MOTOR motor, int direction, uint16_t numberOfSteps, int milliBetweenSteps) {

    // Enable motor.
    digitalWrite(motor.enable_pin, LOW);

    // Check direction;
    switch (direction) {
      case DIR_CC:
        digitalWrite(motor.direction_pin, HIGH);
        break;
      case DIR_CCW:
        digitalWrite(motor.direction_pin, LOW);
        break;
      default:
        sendNack();
        return;
    }

    // Move the motor (but keep an eye for a halt command)
    for(int n = 0; n < numberOfSteps; n++) {
      // Interrupt motor
      if(checkForHalt()) {  
        sendAck();
        break; 
      }
      digitalWrite(motor.step_pin, HIGH);
      delayMicroseconds(motor.pulse_width_micros);
      digitalWrite(motor.step_pin, LOW);
      delay(milliBetweenSteps);
    }

    // Disable holding torque.
    digitalWrite(motor.enable_pin, HIGH);

    // Let the user know the move is done.
    sendCompletedAction();
}

// END MOTORS

/*  ############### COMMUNICATION ###############
 * 
*/
void serialEvent() {

  // Get all the data.
  while (Serial.available()) {

    // Read a byte
    uint8_t inByte = (uint8_t)Serial.read();

    // Store the byte in the buffer.
    rxBuffer.data[rxBuffer.index] = inByte;
    rxBuffer.index++;

    // If a complete packet character is found, mark the packet
    // as ready for execution.
    if ((char)inByte == '\n') {
      rxBuffer.packetComplete = true;
    }

  }
}

// Clear the buffer.
void resetBuffer(struct BUFFER *buffer) {
  memset(buffer->data, 0, sizeof(buffer->data));
  buffer->index = 0;
  buffer->packetComplete = false;
}

// Does not count termination char.
int packetLength(BUFFER buffer){
  for(int i = 0; i < buffer.bufferSize; i++) {
    if((char)buffer.data[i] == '\n'){ return i; }
  }
  return -1;
}

void sendAck() {
  Serial.write(ACK);
  Serial.write(END_TX);
}

void sendNack() {
  Serial.write(ACK);
  Serial.write(END_TX);
}

void sendCompletedAction() {
  Serial.write(COMPLETED_CMD);
  Serial.write(END_TX);
}

// Halt is handled outside normal communication protocol.
boolean checkForHalt() {
  if (Serial.available()){
    // Halt command has no termination character.
    if ((uint8_t)Serial.read() == HALT_CMD) {
      return true;
    }
  }
  return false;
}

// END COMMUNICATION

Python RAMPS

There are two variants of the Python code. First, is for the Raspberry Pi. It's where I focused coding time, as it made sense to generate training images using the same hardware (PiCamera) as would be used for production. However, I've a simpler desktop version which uses OpenCV and a webcam.

For the Raspberry Pi and desktop versions you will need the following:

Python 3.7 -- this should be standard on Raspbian Buster.

On the desktop you will need opencv, it can be installed using:

pip install opencv

In both cases you will need the custom class ramps_control, if you clone the repository and run your script from the ./turn_table directory, that should be handled for you.

What's it Do?

The turn table script initializes the camera. It then creates a loop over the number of angles you want to take images.

A full rotation is 3200 steps and if you ask for 60 images, then the script will rotate the turntable ~53.33 steps. At the end of the rotation, the script will capture an image of your target. Then, it will rotate another 53.33 steps and take another picture. It will do this 60 times, where it should have completed a full rotation.

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Created on Wed Sep 25 05:58:48 2019

@author: ladvien
"""

from picamera import PiCamera
import os
from time import sleep
import ramps_control
import serial
import glob

#################
# Parameters
#################

MILLI_BETWEEN_STEPS     = 5
IMAGES_PER_ROTATION     = 60
FULL_ROTATION           = 3200
STEPS_BEFORE_PIC        = int(FULL_ROTATION / IMAGES_PER_ROTATION)

print(f'Steps per image: {STEPS_BEFORE_PIC}')

####################
# Don't Overwrite
####################
def check_existing_images(output_directory):
    existing_image_files = glob.glob(f'{output_directory}/*.jpg')
    max_file_index = 0
    for file in existing_image_files:
        file_index = file.split('/')[-1].split('_')[1].replace('.jpg', '')
        try:
            file_index = int(file_index)
            if file_index > max_file_index:
                max_file_index = file_index
        except:
            pass
    return max_file_index

#################
# Open Serial
#################
ser = serial.Serial('/dev/ttyUSB0', 115200)
print(ser.name)   

#################
# Init Camera
#################
#picam v2 resolution 3280 x 2464
camera = PiCamera()
PIC_SIZE = 1200
CAM_OFFSET_X = 0
CAM_OFFSET_Y = 0
camera.start_preview()

#################
# Init RAMPS
#################
ramps = ramps_control.RAMPS(ser, debug = False)

# Track whether the motor is at work.
motor_moving = False

# Reset the RAMPs program.
ramps.reset_ramps(False)

#################
# Main
#################

part = ''

while True:
    part_candidate = input(f'Enter part number and hit enter. (Default {part}; "q" to quit): ')

    if part_candidate.lower() == 'q':
        print('Bye!')
        quit()
    elif part_candidate != '':
        part = part_candidate

    output_directory = f'/home/pi/Desktop/lego_images/{part}' 

    if not os.path.exists(output_directory):
        os.makedirs(output_directory)

    max_file_index = check_existing_images(output_directory)

    for i in range(IMAGES_PER_ROTATION):

            success = ramps.move(ramps.MOTOR_X,
                        ramps.DIR_CCW,
                        STEPS_BEFORE_PIC,
                        MILLI_BETWEEN_STEPS)

            if success:
                print('Table move a success.')

                file_path = f'{output_directory}/{part}_{i + max_file_index}.jpg'
                print(file_path)
                camera.capture(file_path)

            # sleep(0.05)

ser.close()
camera.stop_preview()

Python RAMPS Class

To increase resuability of the code, I've abstracted the RAMPs controller code into a Python class. This class is called by the script above. It is blocking code which handles sending commands, polling the Arduino, and reports received information.

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Created on Sat Sep 28 05:39:18 2019

@author: ladvien
"""
from time import sleep, time

"""
  MOTOR_NUM:
      X     = 0
      Y     = 1
      Z     = 2
      E1    = 3
      E2    = 4

  PACKET_TYPES
      0x01 = motor_write
      0x02 = motor_halt

  DIRECTION
      0x00 = CW
      0x01 = CCW

  MOTOR MOVE PROTOCOL:
                       0               1     2     3        4       5         6
  MOTOR_PACKET = PACKET_TYPE_CHAR MOTOR_NUM DIR STEPS_1 STEPS_2 MILLI_BETWEEN \n

"""

class RAMPS:
    DRIVE_CMD       = 0x01
    HALT_CMD        = 0x0F
    DIR_CC          = 0x00
    DIR_CCW         = 0x01

    COMPLETED_CMD   = 0x07
    END_TX          = 0x0A
    ACKNOWLEDGE     = 0x06
    NEG_ACKNOWLEDGE = 0x15


    MOTOR_X         = 0x01
    MOTOR_Y         = 0x02
    MOTOR_Z         = 0x03
    MOTOR_E1        = 0x04
    MOTOR_E2        = 0x05

    def __init__(self, ser, debug = False):
        self.ser = ser
        self.toggle_debug = debug
        self.rx_buffer_size = 256
        self.serial_delay = 0.1

    def toggle_debug(self):
        self.debug = not self.debug

    def print_debug(self, message):
        if self.toggle_debug:
            print(message)

    """ 
            COMMUNICATION
    """

    # Prepare for a serial send.
    def encode_packet(self, values):
        return bytearray(values)

    # Prepare a packet the slave will understand
    def prepare_motor_packet(self, motor_num, direction, steps, milli_between):
        steps_1 = (steps >> 8) & 0xFF
        steps_2 = (steps) & 0xFF
        return [self.DRIVE_CMD, motor_num, direction, steps_1, steps_2, milli_between, self.END_TX]

    def read_available(self, as_ascii = False):

        self.print_debug(f'Reading available.')

        # 1. Get all available data.
        # 2. Unless buffer exceeded.
        # 3. Return a list of the data.

        incoming_data = []
        incoming_data_size = 0

        while self.ser.in_waiting > 0:
            incoming_data_size += 1

            if incoming_data_size > self.rx_buffer_size:
                self.print_debug(f'Buffer overflow.')
                return list('RX buffer overflow.')

            if as_ascii:
                incoming_data.append(self.ser.readline().decode('utf-8'))
            else:
                incoming_data += self.ser.readline()

        self.print_debug(f'Completed reading available.')
        return incoming_data

    def check_for_confirm(self, command_expected):
        confirmation = self.read_available()
        if len(confirmation) > 0:
            if confirmation[0] == command_expected:
                return True
        else:
            return False


    """ 
            RAMPS UTILITY
    """

    def reset_ramps(self, print_welcome = False):

        self.print_debug(f'Reseting Arduino.')
        # Reset the Arduino Mega.
        self.ser.setDTR(False)
        sleep(0.4)
        self.ser.setDTR(True)
        sleep(2)   

        # Get welcome message.
        welcome_message = []

        while self.ser.in_waiting > 0:
            welcome_message.append(self.ser.readline().decode('utf-8') )

        self.print_debug(f'Completed reset.')
        if print_welcome:
            # Print it for the user.
            print(''.join(welcome_message))
            return
        else:
            return

    """ 
            MOTOR COMMANDS
    """
    def move(self, motor, direction, steps, milli_secs_between_steps):

        # 1. Create a list containg RAMPs command.
        # 2. Encode it for serial writing.
        # 3. Write to serial port.
        # 4. Check for ACK or NACK.
        # 5. Poll serial for completed command.

        packet = self.prepare_motor_packet(motor,
                                           direction,
                                           steps,
                                           milli_secs_between_steps)
        packet = self.encode_packet(packet)

        self.print_debug(f'Created move packet: {packet}')

        self.write_move(packet)

        # Don't miss ACK to being in a hurry.
        sleep(self.serial_delay)
        confirmation = self.read_available()
        if confirmation[0] == self.ACKNOWLEDGE:
            self.print_debug(f'Move command acknowledged.')

        if(self.wait_for_complete(120)):
            return True

        return False

    def wait_for_complete(self, timeout):

        # 1. Wait for complete or timeout
        # 2. Return whether the move was successful.

        start_time = time()

        while True:
            now_time = time()
            duration = now_time - start_time
            self.print_debug(duration)
            if(duration > timeout):
                return False

            if self.check_for_confirm(self.COMPLETED_CMD):
                self.print_debug(f'Move command completed.')
                return True

            sleep(self.serial_delay)

    def write_move(self, packet):        
        self.ser.write(packet)
        self.print_debug(f'Executed move packet: {packet}')

Questions

That's pretty much it. I've kept this article light, as I'm saving most of my free time for coding. But, feel free to ask questions in the comments below.

Training a CNN to Classify LEGOs

Thomas Brittain — Sun, 01 Sep 2019 10:00:00 +0000

This article is part of a series. It should explain the code used to train our convolutional neural-network (CNN) LEGO classifier.

If you want to code along with this article, we've made it available in Google's Colab:

Lego Classifier

Or if you want to run the code locally:

lego_sorter

It's a WIP, so comment below if you run into any issues.

Classifier Code:

Our code started with a notebook found on Kaggle:

Lego Brick Images Keras CCN

However, there problems in the code. I rewrote most of it, so I'm not sure how much of the original is left. Still, cite your sources!

Some of the issues were:

It used a model more complex than needed.
The code format was a mess.
Mismatch of target output and loss.

It was the last one which is super tricky, but critical. It's a hard to catch bug which inaccurately reports high accuracy. I'll discuss it more below, but it's a trap I've fallen into myself. Regardless of the issues, it was good jump-starter code, since we've never worked with a CNN.

Project Setup (local only)

If you are running this code locally, you will need to do the following.

Enter the command prompt and navigate to your home directory. We're going to clone the project repository (repo), then, clone the data repo inside the project folder.

git clone https://github.com/Ladvien/lego_sorter.git
cd lego_sorter
git clone https://github.com/Ladvien/lego_id_training_data.git

Then, open your Python IDE, set your directory to ./lego_sorter, and open lego_classifier_gpu.py.

Lastly, if you see a cell like this:

!git clone https://github.com/Ladvien/lego_id_training_data.git
!mkdir ./data
!mkdir ./data/output
!ls

Skip or delete them, they are need when running the Colab notebook. Of course, if you are running the Colab notebook, make sure to execute them.

Classifier Code: Needed Libraries

Below is the code we used. Reviewing it, I see some ways to clean it up, so know it may change in the future.

Here's a breakdown of why the libraries are needed:

tensorflow -- Google's main deep-learning library, it's the heart of the project.
keras -- a library abstracting a lot of the details from creating a machine learning model.
json -- we write the classes to file for use later.
tensorboard -- a library for visualizing your training session.
webbrowser -- this is opens your webrowser to Tensorboard.


# Import needed tools.
import os
import matplotlib.pyplot as plt
import json
import numpy as np
from scipy import stats

# Import Keras
import tensorflow as tf
import tensorflow.keras
from tensorflow.keras.layers import Dense,Flatten, Dropout, Lambda
from tensorflow.keras.layers import SeparableConv2D, BatchNormalization, MaxPooling2D, Conv2D, Activation
from tensorflow.compat.v1.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping, TensorBoard, CSVLogger, ReduceLROnPlateau
from tensorflow.keras.preprocessing import image

# Tensorboard
from tensorboard import program
import webbrowser
import time

If you are following along with this code locally and need help setting up these libraries, just drop a comment below. I got you.

Classifier Code: Parameters

The parameters sections is the heart of the training, I'll explain what the parameters are doing and highlight those you might want to tweak.

continue_training       = False
initial_epoch           = 0
clear_logs              = True

input_shape             = (300, 300, 3) # This is the shape of the image width, length, colors
image_size              = (input_shape[0], input_shape[1]) # DOH! image_size is (height, width)
train_test_ratio        = 0.2
zoom_range              = 0.1
shear_range             = 0.1

# Hyperparameters
batch_size              = 16
epochs                  = 40
steps_per_epoch         = 400
validation_steps        = 100 
optimizer               = 'adadelta' 
learning_rate           = 1.0
val_save_step_num       = 1

path_to_graphs          = './data/output/logs/'
model_save_dir          = './data/output/'
train_dir               = './lego_id_training_data/gray_train/'
val_dir                 = './lego_id_training_data/gray_test/'

Parameters: Training Session

The first few parameters help continue from an interrupted training session. For example, if your session is interrupted at epoch 183, then you could set continue_training = True and initial_epoch = 184, then execute the script. This should then load the last best model and pick back up training where you left off. Lastly, if you set clear_logs = True then it clears the Tensorboard information. So, if you continue a session, you will want to set this to False.

This section is a WIP and there are several issues. First, the Tensorboard logs should be saved in separate folders and shouldn't need to be cleared. Also, when continuing a training session, it resets the best validation score (tracked for saving your model before overfitting) resulting in a temporary dip in performance.

Parameters: Image Data

The input_shape refers to the dimensions of an image: height, width, and color (RGB) values. image_size comes from the input_shape.

Note, one issue I had early on with image_size. I tried non-square images (which hurt training and aren't recommended for CNNs) and found the hard way most of the image parameters for width and height reverse their order.

For example, this is what's needed:

...
    val_dir,
    target_size = (height_here, width_here),
...

I was expecting:

...
    val_dir,
    target_size = (width_here, height_here),
...

It bit me hard, as most frameworks I've used expect width first and then height. I mean, even when we talk screen resolution we list width then height (e.g., 1920x1080). Just be aware when using rectangle images. Always RTFM (because, apparently, I didn't).

The train_test_ratio controls how many images are held back for testing the model. I'd have to run through the code again, but I don't think this is needed. As the preprocessing script created a folder with validation images. Hmm, I'll add it to my tech debt list.

The zoom_range parameter controls how far the script should zoom in on the images. And, lastly, shear_range controls how much of the images to clip from the edges before feeding them to the CNN.

Parameters: CNN Hyperparameters

A "hyperparameter" is what machine-learning engineers call parameters which may impact the outcome of training a neural-net.

What are Hyperparameters?

Here are the hyperparamters we've exposed:

batch_size refers to the number of photos a neural-net should attempt predictions on before updating the weights of each perceptron. Note, the highest batch size is usually limited by your GPU RAM. Locally, I use a GTX 1060 with 6GB of RAM--I couldn't get a batch bigger than around 16. YMMV.

steps_per_epoch are the number of batches to go through before considering one epoch complete. An epoch is an arbitrary number representing how many batches * steps_per_epoch to go through before considering the training complete.

So, the length of training would be training schedule = epochs * steps_per_epoch * batch_size

validation_steps is the number of batches from the training data to use for validating the current weights. This will be used when we fit (train) our classifier and when we evaluate it.

optimizer is the name of the optimizer used. This is the heart of training, as it is responsible for deciding how the the weights should be updated after each batch.

I've setup the code to only use one of three optimizers, either adam, adagrad, sgd.

def get_optimizer(optimizer, learning_rate = 0.001):
    if optimizer == 'adam':
        return tensorflow.keras.optimizers.Adam(lr = learning_rate, beta_1 = 0.9, beta_2 = 0.999, epsilon = None, decay = 0., amsgrad = False)
    elif optimizer == 'sgd':
        return tensorflow.keras.optimizers.SGD(lr = learning_rate, momentum = 0.99) 
    elif optimizer == 'adadelta':
        return tensorflow.keras.optimizers.Adadelta(lr=learning_rate, rho=0.95, epsilon=None, decay=0.0)

Here is more information on optimizers.

Easy to read:

Primary source:

Adam
Adagrad

The primary reason, as I understand it, to use adagrad over adam, is adagrad's learning_rate will naturally modify itself to be more conducive to optimal convergence.

However, there are many optimizers. A lot of them available in Keras:

Stochastic Gradient Descent (SGD)
RMSprop
Adagrad
Adadelta
Adam
Nadam
Adamax

Keras' docs on optimizers:

Keras Optimizers

The learning_rate controls how drastically the optimizer should change the perceptrons's weights when they have made an incorrect prediction. Too high, it won't converge (learn) too low and it will take a while.

You will find a lot of documentation saying, "The default learning rate of an optimizer is best, it doesn't need to be changed." I've found this advice to be true, well, mostly. I did run into an issue when using adam's default setting of 0.001 in this project. The neural-net just didn't learn--I had to drop it to around 0.0001, which did much better.

A starter read on learning rate:

How to pick the best learning rate

It's not exhaustive. If you interested in tweaking the optimizer or learning rate, Google and read as much as possible.

Lastly, val_save_step_num controls how many training epochs should pass before the validator tests whether your model is performing well on the test set. We have the code setup such if the validator says the model is performing better than any of the previous tests within this training session, then it will save the model automatically.

Classifier Code: Data Preparation

The make_dir allows making a directory, if it doesn't already exist. We then use it to create our model save directory.

def make_dir(dir_path):
    if not os.path.exists(dir_path):
        os.mkdir(dir_path)

# Create needed dirs
make_dir(model_save_dir)

The next bit saves the classes the train_gen found to a file. This is useful later when we are trying to quickly deploy the model to production.

# Save Class IDs
classes_json = train_gen.class_indices
num_classes = len(train_gen.class_indices)

This saves one object to a json file. The key (e.g., "2456") represents the code provided by LEGO. And the value is the numeric class assigned by the classifier.

{
    "2456": 0,
    "3001": 1,
    "3002": 2,
    "3003": 3,
    "3004": 4,
    "3010": 5,
    "3039": 6,
    "32064": 7,
    "3660": 8,
    "3701": 9
}

We can do the following after we've trained the model:

predicted_lego_code = json_classes[model.predict()]

And the model will return the LEGO class it has identified.

Classifier Code: Data Generator

When dealing with CNNs, often, the training data are too large to fit in RAM, let alone GPU RAM, at once.

Instead, a DataGenarator is used. A DataGenerator is class provided by Keras, it loads training data in manageable chunks to feed to your model during training. Let's run through using it.

We initialize ImageDataGenerator -- a subclass of keras' DataGenerator. Then, we create two flows, one for loading data from the training folder into the model. The other is the same, however, it loads data from the test folder. The latter will be used to validate the model.

Parameters used in our ImageDataGenerator:

shear_range -- this controls how much of the images' edge is trimmed off as a percentage of the whole image. This is useful for quickly reducing the size of images (thereby increasing training speed).
zoom_range -- is how far to zoom in before feeding the image to the model.
horizontal_flip -- if this is set to true, the images are randomly mirrored horizontally. This essentially doubles your training images. Though, it shouldn't be used in all cases. If the target has a "handediness" to it, then this would destroy accuracy. A simple example of this downfall would be training a CNN to determine whether baseball player is left or right handed.
validation_split -- determines the percentage of images held back for validation.

# These Keras generators will pull files from disk
# and prepare them for training and validation.
augs_gen = ImageDataGenerator (
    shear_range = shear_range,  
    zoom_range = shear_range,        
    horizontal_flip = True,
    validation_split = train_test_ratio
)

Now,the parameters of the ImageDataGenerator.flow_from_directory methods:

target_size -- this one bit me. It's the size of your images as a tuple (e.g., "(150, 150)"). It expects height then width.
batch_size -- this is the number of images loaded into the GPU RAM and trained on before updating the weights.
class_mode -- an import argument. This sets up the targets for the model's attempt at prediction. sparse indicates the targets will be LabelEncoded.

Below lies a tale of woe I keep hinting at.

If you have more than one class to predict, like us, you have two options. Either sparse or categorical.

Sparse

target
1
2
3
2

Categorical

1	2	3
1	0	0
0	1	0
0	0	1
0	1	0

However, this is where the bug in the original code was. It had setup the targets as categorical, however, it used binary_crossentropy as the loss function. This error is difficult to catch--it's the machine-learning equivalent of the "there" and "their" error.

With the mismatch of targets and loss function there's no help either. The model will still compile and train without problems. But the cruel combination of categorical targets and binary_crossentropy leads to an extremely high accuracy but an extremely bad production accuracy. The problem is the loss function is only looking at column 1 in the categorical table above. If the model model predicts it is 1 when the first column is 1 then it thinks its "correct." Otherwise, if the model predicts a 0 when column 1 is 0, then the model still thinks its correct. After all, "it wasn't 1." And to be clear, the model isn't wrong--we've just given it the wrong target labels.

This is the quintessential "hotdog, not a hotdog" problem.

In short, if you feel your model quickly trains to an accuracy too good to be true, it is.

train_gen = augs_gen.flow_from_directory (
    train_dir,
    target_size = image_size, # THIS IS HEIGHT, WIDTH
    batch_size = batch_size,
    class_mode = 'sparse',
    shuffle = True
)

test_gen = augs_gen.flow_from_directory (
    val_dir,
    target_size = image_size,
    batch_size = batch_size,
    class_mode = 'sparse',
    shuffle = False
)

Classifier Code: Building the Model

Close to done. I'm not going to go over the design of a CNN for two reasons. I'm still learning what it all means and there are much better explanations elsewhere.

Getting Started with Convolutional Neural Networks

However, there are a couple of things important to us.

num_classes is the number of LEGOs we are trying to classify.
activation on the last layer controls the type of output from the CNN. It will need to correspond with the optimizer and will need to correspond to the class_mode setting of the the DataGenerators.
build_model is a convenience function. It allows us to quickly build a Keras CNN model and return it to be used.
model.summary outputs a text diagram of the model.
model.compile prepares the entire model for training.

def build_model(opt, input_shape, num_classes):
    model = tf.keras.models.Sequential()
    model.add(tf.keras.layers.Conv2D(32, (3, 3), input_shape = input_shape))
    model.add(tf.keras.layers.Activation('relu'))
    model.add(tf.keras.layers.Dropout(0.2))
    model.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2)))

    model.add(tf.keras.layers.Conv2D(64, (3, 3)))
    model.add(tf.keras.layers.Activation('relu'))
    model.add(tf.keras.layers.Dropout(0.2))
    model.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2)))

    model.add(tf.keras.layers.Conv2D(128, (3, 3)))
    model.add(tf.keras.layers.Activation('relu'))
    model.add(tf.keras.layers.Dropout(0.2))
    model.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2)))

    model.add(tf.keras.layers.Flatten())  # this converts our 3D feature maps to 1D feature vectors
    model.add(tf.keras.layers.Dense(256))
    model.add(tf.keras.layers.Activation('relu'))

    model.add(tf.keras.layers.Dropout(0.2))

    model.add(tf.keras.layers.Dense(num_classes, activation = 'softmax'))
    return model

#################################
# Create model
#################################

selected_optimizer = get_optimizer(optimizer, learning_rate)

model = build_model(selected_optimizer, input_shape, num_classes)
model.summary()

model.compile(
    loss = 'sparse_categorical_crossentropy',
    optimizer = selected_optimizer,
    metrics = ['accuracy']
)

Classifier Code: Creating Callbacks

Before we execute training we should setup of Keras callbacks.

Keras callbacks

These pre-written callback functions will be passed to the model and executed at important points throughout the training session.

ModelCheckpoint this method is called after the number of epochs set by val_save_step_num. It runs a validation batch and compares the val_loss against other past scores. If it is the best val_loss yet, the method will save the model and, more importantly, weights to the best_model_weights path.
TensorBoard opens a TensorBoard session for visualizing the training session.

best_model_weights = model_save_dir + 'base.model'

checkpoint = ModelCheckpoint(
    best_model_weights,
    monitor = 'val_loss',
    verbose = 1,
    save_best_only = True,
    mode = 'min',
    save_weights_only = False,
    period = val_save_step_num
)

tensorboard = TensorBoard(
    log_dir = model_save_dir + '/logs',
    histogram_freq=0,
    batch_size=16,
    write_graph=True,
    write_grads=True,
    write_images=False,
)

Before any KerasCallbacks can be added to the training session, they must be gathered into a list, as it is how training method will except to receive the

callbacks = [checkpoint, tensorboard]

Classifier Code: Training

Gross, I need to rewrite this portion of the code. It is a kludge way to restart a training session after interruption.

It checks if you indicated you want to continue a session. It then loads the best saved model and evaluates it on the test data.

if continue_training:
    model.load_weights(best_model_weights)
    model_score = model.evaluate_generator(test_gen, steps = validation_steps)

    print('Model Test Loss:', model_score[0])
    print('Model Test Accuracy:', model_score[1])

And here, we come to the end. The following function executes the training session. It will initialize the callbacks, then train for the number of epochs set. Each epoch it is pulling a batch of data from the train_gen (DataGenerator), attempting predictions, and then updating weights based on outcomes. After the number of epochs set in the checkpoint callback, the model will pull data from the test_gen, these data it has "never" seen before, and attempt predictions. If the outcome of the test is better than the outcome of any previous test, the model will save.

history = model.fit_generator(
    train_gen, 
    steps_per_epoch  = steps_per_epoch, 
    validation_data  = test_gen,
    validation_steps = validation_steps,
    epochs = epochs, 
    verbose = 1,
    callbacks = callbacks
)

Whew, that's it. The above model converged for me after 20 minutes to 98% validation accuracy. However, there's lots left to do though. As I've said before, "Just because we have high validation accuracy does not mean we will have high production accuracy." In the future, I'll be writing about the turntable for quickly generating training data. It's nifty. Based on a NEMA17, RAMPS kit, and RPi with RPi Camera. It's the bomb-dot-com.

A LEGO Classifier -- CNN and Elbow Grease

Thomas Brittain — Fri, 30 Aug 2019 10:00:00 +0000

I've a robot friend. To be clear, the friend is not a robot, rather, we build robots together. One of the projects we tossed about is building a LEGO sorting machine. Rockets is the friends name--again, not a robot--teaches robotics to kids. For their designs, LEGOs are the primary component. Unfortunately, this results in much time spent to preparing for an event.

He mentioned to me, "What I really need is a sorting machine." And proceeded to explain his plain for building one.

I was skeptical for some time, but finally, I got drawn in he talked about incorporating a deep neural-network. More specifically, a convolutional neural-network (CNN). I'd been looking for an excuse to build a CNN. This was a good one.

Anyway, these blog posts are our journal in build the LEGO sorter.

Before we get started, a note about this series: I won't spend much time on explaining parts of the work where it is better documented elsewhere. Instead, I'm going to focus on stuff I've found everyone else omitting. Like, putting the neural-network to work. This one bugged me. Everyone loves to say, "Dude, my classifier has a validation accuracy of 99.999%!" That's great, but as we found out, validation accuracy doesn't always translate into production accuracy.

TL;DR

If you don't want to listen to my rambling or want to do things the easy way, you can jump straight into the code using Google's Colab:

lego_classifier

This notebook is setup to download Rocket's data and train the classifier. Thanks to Google for providing a GPU to train on and Github for hosting the data.

Or if you want to run the code locally, Rocket made the training data public. Just know, you'll need a GPU.

lego_id_training_data

Then jump to the code by clicking here.

The Idea

It was pretty straightfoward to begin with. We'd find some images of LEGOs on the internet and then train a CNN to classify them by their part code. It was a bit naive, but that's where must projects being, right? Hopeful naiveté.

Anyway, we searched the webs for projects like this, as we hoped they had prepared images. Google told us several folks doing similar work. I'm not going to list them all, only what I considered worth a read:

Lego Sorter using TensorFlow on Raspberry Pi

This is an extremely well documented project by Paco Garcia.

So, after reading a few articles, we figured we could do this. We just needed data. After a bit more searching we found the following datasets:

I wasn't happy about these datasets. Their structures weren't great and they were not designed to help train a classifier. But then, Rockets found Paco had actually opened his dataset to the public:

Kaggle: Lego Brick Sorting (best)

One bit more, Paco also made his code public:

11 Class Tensorflow Model

Paco, you are a robot friend, too!

Alright, we were encouraged by Paco. We knew the project would be possible. However, we didn't want to step on brownfield. We needed the green. Or if you don't speak dev, we didn't want to do this the easy way and replicate Paco's work. We wanted to really beat ourselves up by doing everything from scratch.

Creating a Dataset

As I stated before, I didn't like any datasets but Paco's. It was real images and meant to train a classifier. But, they weren't the LEGOs we wanted to classify. Rockets's LEGO projects involve a lot of technic bricks, which didn't seem to be in Paco's mix. So, we set out to create our own.

The first attempt creating training images was by rendering images from .stl files found on the internet using the Python version of Visualization Toolkit. I won't cover it here since it was a fail and as I'll create an article later about the stuff we tried and didn't work.

{: .float-left} Anyway, while I was working on it Rockets had a brilliant plan. He created an instrument to take pictures of a LEGO on a spin plate. It used a Raspberry Pi, Pi Cam, and stepper motor, and unicorn farts.

Then Rockets began taking pictures of 10 classes of LEGOs. Not sure how long this took , but shortly he pinged me saying he had 19,000 images. (Ok, ok, he might be part robot.)

I'm not going to attempt explaining the build, as I believe Rockets will do this later. Besides, about the only part I understand is the unicorn flatulence.

Alright! Now I needed to get my butt in gear and fix up the software.

Preprocessing Code

Before we could start training a CNN on Rockets's images we needed to do some preprocessing. First, the images came in at full resolution, but we needed to crop them, as the CNN train better on square image. Of course, the image would need to be cropped as not to lose the target data (the LEGO).

For example

Also, the trainer would be expecting a file structure something like this:

data
├── test
│   ├── 2456
│   │     └── 2456_0001.jpg
│   │     └── 2456_0002.jpg
│   │     └── 2456_0003.jpg
│   │     └── ....
│   ├── 3001
│   ├── 3002
│   ├── 3003
│   ├── 3004
│   ├── 3010
│   ├── 3039
│   ├── 32064
│   ├── 3660
│   └── 3701
└── train
    ├── 2456
    ├── 3001
    ├── 3002
    ├── 3003
    ├── 3004
    ├── 3010
    ├── 3039
    ├── 32064
    ├── 3660
    └── 3701

Therefore, I've written a Python script to do the following

Take a path where images are stored by name of the class
Load the image
Resize the image to specified size
Crop from the center of the image out
Create a train and test folder
Create sub-folders in train and test with the class name
Shuffle the images in the process
Save the cropped file in the appropriate folder, depending what percentage of images you want to withhold for testing.
Repeat steps 2-8 for every image

Let's jump into the code.

The full code can found here:

square_crop.py

But I'll walk through the code below.

Preprocessing Code: Needed Libraries

import os
import glob
import cv2
import random

The only non-standard Python library we are using is:

OpenCV

This may be a bit tricky depending on which OS you are using and whether you are using Anaconda or straight Python. However, the following is what we used:

pip install https://pypi.org/project/opencv-python/

If you have any troubles load the cv2 library, it probably means there was an issue installing OpenCV. Just let me know in the comments and I can help debug.

Preprocessing Code: Processing Parameters

The following control the the flow of preprocessing

dry_run: if set to true, it does not save the images, but does everything else
gray_scale: converts the images to gray-scale.
root_path: the root folder of the project
show_image: shows the before and after of the image.
output_img_size: adjust this to the size of your desired output image
grab_area: the total area of the original image to take before resizing
train_test_split: the rate of test images to withhold
shuffle_split: should the images be shuffled in the process
part_numbers: a list of all the class folders contained in the input

#####################
# Parameters
#####################     

dry_run                 = False # If true, will print output directory.
gray_scale              = True

root_path               = './data/'
input_path              = f'{root_path}raw/size_1080/'
output_path             = f'{root_path}cropped/'

show_image              = False

output_img_size         = (300, 300)
grab_area               = 500
train_test_split        = 0.3
shuffle_split           = True

part_numbers            = [
                           '2456',
                           '3001',
                           '3002',
                           '3003',
                           '3004',
                           '3010',
                           '3039',
                           '3660',
                           '3701',
                           '32064'
                        ]

Below is the main loop. It is going to repeat for every folder it finds in the the root folder.

for part_number in part_numbers:

    part_input_path  = f'{input_path}{part_number}/'

    # Get input file paths.
    image_files = glob.glob(f'{part_input_path}*.jpg')
    num_files = len(image_files)

    # Image index.
    index = 0

    # If true, the images will be loaded and then split at random.
    if shuffle_split:
        file_index = random.sample(range(1, num_files), num_files - 1)
    else:
        file_index = range(1, num_files)

This is the inner loop, it loads each of the image files in the class class folder, modifies it, and saves it to the output folders.

    for file_num in file_index:

        # Increment the file index.
        index += 1

        # Load the image
        input_file_path = f'{input_path}{part_number}/{str(file_num).zfill(4)}.jpg'
        print(f'LOADED: {input_file_path}')

        # Crop raw image from center.
        img = cv2.imread(input_file_path)

        # Get the center of the image.
        c_x, c_y = int(img.shape[0] / 2), int(img.shape[1] / 2)
        img = img[c_y - grab_area: c_y + grab_area, c_x - grab_area: c_x + grab_area]

        # Resize image
        img = cv2.resize(img, output_img_size, interpolation = cv2.INTER_AREA)

        # Should we convert it to grayscale?
        if gray_scale:
            img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

        # Show to user.
        if show_image:
            cv2.imshow('image', img)
            cv2.waitKey(0)
            cv2.destroyAllWindows() 

        # Determine if it should be output to train or test.
        test_or_train = 'train'        
        if index < int(num_files * train_test_split): 
            test_or_train = 'test'

        # Prepare the output folder.
        color = ''
        if gray_scale:
            part_output_folder = f'{output_path}gray_scale/{test_or_train}/{part_number}/'
        else:
            part_output_folder = f'{output_path}color/{test_or_train}/{part_number}/'

        # Make the output directory, if it doesn't exist.
        if not os.path.exists(part_output_folder):
            os.makedirs(part_output_folder)

        # Create part path.
        part_image_path = f'{part_output_folder}{part_number}_{index}.jpg'

        # Output
        if dry_run:
            print(f'Would have saved to: {part_image_path}')
        else:
            print(f'SAVED: {part_image_path}')
            cv2.imwrite(part_image_path, img)

Fairly straightfoward. Just make sure to run to run the script from the main directory. For example

project_folder
└── square_crop.py <--- run from here
└── data
    ├── test
    │   ├── 2456
    │   │     └── 2456_0001.jpg
...

Or, if you don't want to do it the hardway. Rockets has made his images available

lego_id_training_data

Next, I'm going to dive into the Tensorflow CNN code. Stay tuned, my robot friends!