Forem: J Fowler

A Drum machine for iOS

J Fowler — Sat, 14 Dec 2024 23:44:55 +0000

Here's a fun project I did during some downtime. It's a midi drum machine for iOS using swift.

The code is pretty straightforward. You can find it here.

Let me know if you have any questions.

Store IoT device data

J Fowler — Fri, 27 Sep 2024 00:25:26 +0000

In a previous post, we showed how to receive iot device data from an MQTT broker. In this post, we will store the data to a database.

In a robust system, we may choose to store the raw data events in a data lake. Perhaps, we'll explore that in the future; but for now we'll store it in PostGres for simplicity.

The previous post demonstrated receiving the raw data and unmarshalling it into a struct that was already annotated with gorm tags. Gorm is a popular ORM for Go. If you are not familiar with it, you can for more information here.

type IoTDeviceMessage struct {
    BaseModel
    Time       time.Time       `json:"time" gorm:"index"`
    DeviceID   string          `json:"device_id"`
    DeviceType string          `json:"device_type"`
    DeviceData json.RawMessage `json:"device_data"`
}

So all we need to do is to configure Postgres connection, then use gorm to save the event data.

func setupPostgres(logger *zerolog.Logger) *Repository {
    dbHost := os.Getenv("POSTGRES_HOST")
    dbName := os.Getenv("POSTGRES_DB")
    dbPort := os.Getenv("POSTGRES_PORT")
    dbUser := os.Getenv("POSTGRES_USER")
    dbPassword := os.Getenv("POSTGRES_PASSWORD")
    dsn := fmt.Sprintf("host=%s user=%s password=%s dbname=%s port=%s sslmode=disable TimeZone=UTC",
        dbHost, dbUser, dbPassword, dbName, dbPort)
    logger.Info().Msg(fmt.Sprintf("Connecting to PostgreSQL at %s", dsn))
    db, err := gorm.Open(postgres.Open(dsn), &gorm.Config{})
    if err != nil {
        logger.Fatal().Err(err).Msg("failed to connect to database")
    }

    // Auto-migrate the schema
    err = db.AutoMigrate(&IoTDataEvent{}, &IoTRawDeviceMessage{}, &DeviceModel{}, &TempRHDevice{})
    if err != nil {
        logger.Fatal().Err(err).Msg("failed to migrate models")
    }

    sqlDB, err := db.DB()
    sqlDB.SetMaxIdleConns(10)
    sqlDB.SetMaxOpenConns(100)
    sqlDB.SetConnMaxLifetime(time.Hour)

    repo := NewRepository(db, logger)
    return repo
}

Here we setup the Postgres connection. Note that we are using environment variables to store our sensitive information. This is a good practice for production systems whether they are containerized or not.

We are also initializing a struct called Repository. This struct contains our actual storage and retrieval methods. This provides us some separation from the postgres configuration.

type Repository struct {
    db     *gorm.DB
    logger *zerolog.Logger
}

func NewRepository(db *gorm.DB, logger *zerolog.Logger) *Repository {
    return &Repository{db: db, logger: logger}
}

func (r *Repository) Close() {
    sqlDb, err := r.db.DB()
    if err != nil {
        r.logger.Error().Err(err).Msg("failed to close database")
        return
    }
    _ = sqlDb.Close()
}
...
// Message-related functions

func (r *Repository) CreateMessage(message *IoTDeviceMessage) error {
    return r.db.Create(message).Error
}

func (r *Repository) GetMessagesByDeviceID(deviceID uint, limit int) ([]IoTDeviceMessage, error) {
    var messages []IoTDeviceMessage
    err := r.db.Where("device_id = ?", deviceID).Order("timestamp desc").Limit(limit).Find(&messages).Error
    return messages, err
}

func (r *Repository) DeleteMessagesByDeviceID(deviceID uint) error {
    return r.db.Where("device_id = ?", deviceID).Delete(&IoTDeviceMessage{}).Error
}

Now the message just needs to be persisted. Since we are using the pipeline pattern to process the messages, we will add the persist step a new stage in the pipeline.

// pipeline stage to persist the message
func persistIoTEvent(ctx context.Context, logger *zerolog.Logger, repo *Repository, input <-chan IoTRawDeviceMessage) chan IoTRawDeviceMessage {
    out := make(chan IoTRawDeviceMessage)
    go func() {
        defer close(out)
        for iotMsg := range input {
            logger.Info().Msg(fmt.Sprintf("Persist iot msg for device: %s", iotMsg.DeviceID))
            err := repo.CreateMessage(&iotMsg)
            if err != nil {
                logger.Error().Err(err).Msg("Error creating IoTRawDeviceMessage")
            }
        }
    }()
    return out
}
...
finalChan := persistIoTEvent(ctx, logger, repo, processMsg(ctx, logger, mqttMsgChan))
        for iotMsg := range finalChan {
            // now we have the IoTRawDeviceMessage that has been persisted
            logger.Info().Msg(fmt.Sprintf("Received iot msg: %+v", iotMsg))
            // do something like check for alert conditions
        }

That's all there is to it.

The code for this can be found here. You can use it with the same publisher code from the previous post. Be sure to configure your Postgres settings as environment variables.

Sending IoT Device Data via MQTT broker.

J Fowler — Mon, 23 Sep 2024 04:12:13 +0000

In a previous post, we showed how to send and receive messages from IoT devices using an MQTT broker. In this post we'll extend that idea to a real world example.

Suppose you have an IoT device that measures temperature and humidity in a greenhouse (Not hard to make one using Raspberry Pi or Arduino).

We want to monitor the greenhouse conditions remotely from another computer or, perhaps, a central logging service. In the previous post, we showed a Go implementation of code to send messages so we will extend that example.

Instead of just sending a string saying "temp is x, humidity is y", let's define a structure for the message and device. Consider that you have (or want to add in the future) a device to monitor moisture or rainfall and you want to connect that one as well.

To leave open the possibility of multiple devices and types, we need a data model to handle that.

type Message struct {
    Time       time.Time   `json:"time"`
    DeviceId   string      `json:"device_id"`
    DeviceType string      `json:"device_type"`
    DeviceData interface{} `json:"device_data"`
}

The Message struct is what will be sent to the MQTT broker. It includes the device id, device type, and an interface to the device specific data.

Our device measures temperature and humidity, so that is the data that will be sent in a message.

Next, we need to send the message to the broker. We'll employ JSON format for its simplicity in this example. Note that in a large-scale system with thousands or more devices, we may want to use a more compact format.

message := generateRandomMessage()
payload, err := json.Marshal(message)
if err != nil {
    panic(err)
}
token := client.Publish(topic, 0, false, payload)

Go makes marshalling into JSON pretty easy. Once marshaled, the json message is sent to the broker.

What else would we want to do with the data once we have it: store it to a database, display it on a dashboard, check the values for alarm conditions. We'll need to convert the json to make it usable.

On the receiving side, we just need to unmarshal the json into a struct. We'll use a structure similar to that used on the sending side; but we want to unmarshal it into an object that we can store in a db.

type BaseModel struct {
    ID        uint      `gorm:"primarykey" json:"id"`
    CreatedAt time.Time `json:"created_at"`
    UpdatedAt time.Time `json:"updated_at"`
}
type IoTDeviceMessage struct {
    BaseModel
    Time       time.Time       `json:"time" gorm:"index"`
    DeviceID   string          `json:"device_id"`
    DeviceType string          `json:"device_type"`
    DeviceData json.RawMessage `json:"device_data"`
}

...

func processMsg(ctx context.Context, ....

...

    case msg, ok := <-input:
    if !ok {
        return
    }
    logger.Info().Msg(fmt.Sprintf("Received message: %s from topic: %s\n", msg.Payload(), msg.Topic()))
    var iotMsg Message
    err := json.Unmarshal(msg.Payload(), &iotMsg)
    if err != nil {
        logger.Error().Err(err).Msg("Error unmarshalling Message")
    } else {
        out <- iotMsg
    }

...

Remember that MQTT can be used to publish to multiple topics and it is common practice to use a hierarchical naming convention for topics. So in the case of multiple device types in the greenhouse example, the recommended way is to have different device types publish to different topics.

For example, in our greenhouse example the topic names could be structured like:

/greenhouse/temprh/deviceid
/greenhouse/moisture/deviceid

In MQTT, we can subscribe to topics using a wildcard topic, such as:

if token := client.Subscribe("/greenhouse/#", 0, nil); token.Wait() && token.Error() != nil {
        fmt.Println(token.Error())
        os.Exit(1)
    }

which will match all devices in the greenhouse namespace. then we would just need to add logic to processMsg() to look at the topic of the incoming message to know how to unmarshal it. Although if we all devices can publish their data using the message model above, we can keep things simplified.

Now that we have a device message in a usable form, what should be done with it. In the next post in this series, we'll demonstrate our approach to persist the message in PostGres.

As usual the full source code for the sender can be found here and the subscriber code can be found here.

Let me know your thoughts in the comments.

Thanks!

A cluster-friendly MQTT Broker deployment with Docker

J Fowler — Sat, 21 Sep 2024 02:08:05 +0000

In this post, I show a docker deployment for an MQTT broker that will support adding multiple instances.

In a previous post, I showed a simple Dockerfile for deploying the Eclipse Mosquitto MQTT broker. While Mosquitto can be configured to operate in a cluster, there are other solutions that are more straightforward to implement.

We will be using EMQX, an open-source MQTT broker with built-in clustering capability and is known to work well in a Kubernetes environment.

The docker-compose file is show below:

version: '3'

services:
  emqx1:
    image: emqx:5.8.0
    container_name: emqx1
    hostname: emqx1
    environment:
      - "EMQX_NODE_NAME=emqx@172.20.0.2"
      - "EMQX_CLUSTER__DISCOVERY_STRATEGY=static"
      - "EMQX_CLUSTER__STATIC__SEEDS=[emqx@172.20.0.2]"
    healthcheck:
      test: ["CMD", "/opt/emqx/bin/emqx", "ctl", "status"]
      interval: 5s
      timeout: 25s
      retries: 5
    networks:
      emqx-bridge:
        aliases:
          - emqx1
    ports:
      - 1883:1883
      - 8083:8083
      - 8084:8084
      - 8883:8883
      - 18083:18083

networks:
  emqx-bridge:
    driver: bridge

This pulls the official emqx image, configures basic settings for the instance, and exposes necessary ports. You can also see that it sets up the settings and network that will be used in a cluster if more instances get added.

To start the broker, simply run:

docker-compose up -d

You can check the status using:

docker exec -it emqx1 sh -c "emqx ctl cluster status"
Cluster status: #{running_nodes => ['emqx@172.20.0.2'],stopped_nodes => []}

This is slightly different from the setup shown in the emqx docs. We had to add a hostname and use the internal container ip for the node name.

The full docker-compose file can be found in this repository.

Not sure about why we needed to change the node name to get the cluster status. If you know why, please add a comment below. What else would you change?

Thanks!

A simple MQTT application using Go

J Fowler — Fri, 20 Sep 2024 02:47:32 +0000

IoT devices promise to change the way we interact with the world around us. IoT enabled devices are in automobiles, agriculture, and many more industries. Retrieving data from all those devices need not require a herculean effort and it doesn't thanks to MQTT.

MQTT is the standard for IoT messaging. Development of it started in the late 90's and it has a formal specification that has been adopted by many. MQTT provides a simple and lightweight publish/subscribe messaging protocol making it easy to implement on devices and simple to integrate into large systems.

MQTT works through the use of a broker. A broker sits between the IoT devices and whatever applications make use of the device data. Both devices and the applications are considered 'clients' in the MQTT world.

If a device needs to send data to the world, it acts like an MQTT client, creates a connection to a broker, and publishes its data to a given topic. MQTT topics are how the data gets routed by the broker. When an application wants to receive data from a given device, it creates a connection to a broker then subscribes to the topic that the given device publishes.

To demonstrate, I'll show a simple go application for publishing to an MQTT broker. This simply simulates sending messages from an IoT device. (Note that this go application will run fine on an embedded device that supports Golang such as a Raspberry Pi)

To run this example, you will want to look at my previous post, A basic MQTT Docker deployment, which includes a dockerfile to run an MQTT broker.

package main

import (
    "fmt"
    "math/rand"
    "time"

    mqtt "github.com/eclipse/paho.mqtt.golang"
)

const (
    broker   = "tcp://localhost:1883"
    clientID = "go-mqtt-client"
    topic    = "iot-messages"
)

var connectHandler mqtt.OnConnectHandler = func(client mqtt.Client) {
    fmt.Println("Connected to MQTT Broker")
}

var connectLostHandler mqtt.ConnectionLostHandler = func(client mqtt.Client, err error) {
    fmt.Printf("Connection lost: %v", err)
}

func main() {
    opts := mqtt.NewClientOptions()
    opts.AddBroker(broker)
    opts.SetClientID(clientID)
    opts.OnConnect = connectHandler
    opts.OnConnectionLost = connectLostHandler

    client := mqtt.NewClient(opts)
    if token := client.Connect(); token.Wait() && token.Error() != nil {
        panic(token.Error())
    }

    for {
        message := generateRandomMessage()
        token := client.Publish(topic, 0, false, message)
        token.Wait()
        fmt.Printf("Published message: %s\n", message)
        time.Sleep(1 * time.Second)
    }
}

func generateRandomMessage() string {
    messages := []string{
        "Hello, World!",
        "Greetings from Go!",
        "MQTT is awesome!",
        "Random message incoming!",
        "Go is fun!",
    }
    return messages[rand.Intn(len(messages))]
}

func init() {
    rand.New(rand.NewSource(time.Now().UnixNano()))
}

Not much to that although we could make it a bit more robust to failures.

The application creates a new connection specifying the broker url and a client id that the broker will use to identify it.

Notice that publishing a message is pretty simple, just specify the topic, qos, retain flag and the message itself.
A message can be a string, byte array or a buffer. For qos, we set it to 0 which means 'at most once', ie delivery not guaranteed.

The retain flag tells the broker whether or not to store the last message to a given topic. If messages are 'retained', when a new client subscribes to a given topic, the most recent retained message is immediately sent to it. For this example, we are not using that feature.

Note that the source code for this publisher app can be found at the link at the bottom of this page.

Now let's take a look at the subscriber side.

package main

import (
    "context"
    "fmt"
    mqtt "github.com/eclipse/paho.mqtt.golang"
    "os"
    "os/signal"
    "sync"
    "syscall"
)

const (
    broker   = "tcp://localhost:1883"
    clientID = "go-mqtt-subscriber"
    topic    = "iot-messages"
)

var mqttMsgChan = make(chan mqtt.Message)

var messagePubHandler mqtt.MessageHandler = func(client mqtt.Client, msg mqtt.Message) {
    mqttMsgChan <- msg
}

func processMsg(ctx context.Context, input <-chan mqtt.Message) chan mqtt.Message {
    out := make(chan mqtt.Message)
    go func() {
        defer close(out)
        for {
            select {
            case msg, ok := <-input:
                if !ok {
                    return
                }
                fmt.Printf("Received message: %s from topic: %s\n", msg.Payload(), msg.Topic())
                out <- msg
            case <-ctx.Done():
                return
            }
        }
    }()
    return out
}

var connectHandler mqtt.OnConnectHandler = func(client mqtt.Client) {
    fmt.Println("Connected to MQTT Broker")
}

var connectLostHandler mqtt.ConnectionLostHandler = func(client mqtt.Client, err error) {
    fmt.Printf("Connection lost: %v", err)
}

func main() {
    opts := mqtt.NewClientOptions()
    opts.AddBroker(broker)
    opts.SetClientID(clientID)
    opts.SetDefaultPublishHandler(messagePubHandler)
    opts.OnConnect = connectHandler
    opts.OnConnectionLost = connectLostHandler

    client := mqtt.NewClient(opts)
    if token := client.Connect(); token.Wait() && token.Error() != nil {
        panic(token.Error())
    }

    ctx, cancel := context.WithCancel(context.Background())
    var wg sync.WaitGroup
    wg.Add(1)
    go func() {
        defer wg.Done()
        finalChan := processMsg(ctx, mqttMsgChan)
        for range finalChan {
            // just consuming these for now
        }
    }()

    // Subscribe to the topic
    token := client.Subscribe(topic, 1, nil)
    token.Wait()
    fmt.Printf("Subscribed to topic: %s\n", topic)

    // Wait for interrupt signal to gracefully shutdown the subscriber
    sigChan := make(chan os.Signal, 1)
    signal.Notify(sigChan, os.Interrupt, syscall.SIGTERM)
    <-sigChan

    // Cancel the context to signal the goroutine to stop
    cancel()

    // Unsubscribe and disconnect
    fmt.Println("Unsubscribing and disconnecting...")
    client.Unsubscribe(topic)
    client.Disconnect(250)

    // Wait for the goroutine to finish
    wg.Wait()
    fmt.Println("Goroutine terminated, exiting...")
}

Similar to the publish side, the application sets up a client with the broker url and its client id. We set a default handler to deal with incoming messages; but it's possible to have a different for each topic that the application subscribes to.

You can see that this example includes some synchronization to deal with terminating the program.

There is also a pipeline to process messages. While not necessary for this example; it will come in handy when dealing with a lot of messages. I'm just printing them out in the first stage; but imagine a pipeline that performed filtering, routing, and persistence. More on that in a future post in this series.

That's it. Start up the MQTT broker from my previous post or use another. There is a free broker for testing that you can find here.

Start the publisher and subscriber in separate terminals. You should see the messages received in the subscriber console.

You can find the publisher code here and the subscriber code here

What do you think? How would you make the publisher more robust? Can the synchronization and pipeline be simplified? Let me know what you think in the comments below.

A basic MQTT Docker deployment

J Fowler — Thu, 19 Sep 2024 22:55:08 +0000

In this post, I show how to setup a basic MQTT broker using the Eclipse Mosquitto MQTT broker using Docker.

I'll be referencing this in future posts as I build out a system for monitoring IoT devices using Go and a number of other open-source technologies.

Let's start with the Dockerfile

# Use the official Eclipse Mosquitto image as the base
FROM eclipse-mosquitto:latest

# Set the maintainer label
LABEL maintainer="your-name <you@example.com>"

# Expose MQTT port
EXPOSE 1883

# Expose MQTT over WebSockets port (if needed)
EXPOSE 9001

# Copy custom configuration file (if you have one)
COPY mosquitto.conf /mosquitto/config/mosquitto.conf

# Set the entrypoint to run Mosquitto
ENTRYPOINT ["/usr/sbin/mosquitto", "-c", "/mosquitto/config/mosquitto.conf"]

this is pretty straightforward. It retrieves the latest distribution, exposes the ports used by clients, copies a config file, then declares how to run it.

What about the config file? Here it is and it is pretty basic as well.

# Allow anonymous connections (no authentication)
allow_anonymous true

# Listen on all network interfaces
listener 1883 0.0.0.0

# Enable WebSocket support (optional)
listener 9001
protocol websockets

# Logging configuration
log_dest stdout
log_type all

# Persistence
persistence true
persistence_location /mosquitto/data/

# Uncomment to enable more verbose logging
# log_type all

# Maximum number of client connections
# -1 means no limit
max_connections -1

# Connection timeout in seconds
connection_messages true

Note that this configuration does not include any authentication. You wouldn't want to do that in a production system, but is fine for this example.

The broker is set to listen on all interfaces. This is important as it would only listen on localhost otherwise.

We have persistence enabled to add some durability to the deployment.

Another setting of note is max_connections. With it set to -1, there is no limit. In a production system, you may want to limit concurrent connections performance reasons.

Check the Mosquitto documentation here for a full description of all the options available. It is pretty flexible and makes it suitable for many commercial uses.

In the next post, I provide a simple Go application to push messages to the broker.

The full Dockerfile and configuration is open source and available here

Let me know your thoughts in the comments.

Thanks!

Getting a django shell from a Docker container

J Fowler — Tue, 03 Sep 2024 18:20:11 +0000

The Django shell is a powerful development tool. Comes in handy when you want to try operations without having to hit REST endpoints or access pages.

As trend to containerize projects continues, you will find yourself needing visibility into what's going on in your Django code inside a container.

Not to worry, the Django shell is available in the container. Let's see how to get to it.

Suppose you have a docker-compose file looking something like:

services:
  # Django web server
  web:
    image: web:local
    build:
      context: ./django
      dockerfile: Dockerfile
    hostname: web
    volumes:
      - ./django:/app

To get a shell in a container, we use

docker exec -it <name_or_id> /bin/sh

can be the container id which is a cryptic string or the name which is the form of 'folder-servicename-number'. So for this project, we use:

docker exec -it folder-web-1 /bin/sh
/app #

Great so I have a shell in the container, now we want to start the django shell:

/app # ./manage.py shell

Pretty simple.

If the folder name is long or we don't know the number attached, we can use the name given in the docker compose file to find the id.

docker ps --filter ancestor=web:local --format "{{.ID}}"
32930efb8133

Now you copy and paste the id into the docker exec command

docker exec -it 32930efb8133 /bin/sh
/app #

If you want a one-liner:

docker exec -it $(docker ps --filter ancestor=web:local --format "{{.ID}}") /bin/sh

Have any tips you want to add, put in the comments below.

Thanks!

On the 'throw it away' mentality in tech

J Fowler — Sat, 10 Aug 2024 19:01:09 +0000

William (Bill) Kennedy (@goinggodotnet) put up a real good post on X/Twitter addressing how AI can (or may) lead to a lack of engineering mindset in software development. Thought I would share my response here.

_I was being interviewed by a millennial, startup cto. Real company, has revenue, just needs someone to take on dev duties he has be doing. Very professional interview, standard questions. He asked me, 'Tell me the code you are most proud of'

I proceed to tell him about an embedded project I worked on in '97. It was some intelligent lighting controls that helped reduce energy waste in commercial buildings. Am named in several patents for the work.

Mentioned I have been in buildings recently where the devices are installed and still running. Said I was proud of it because it's still working 25+ yrs later.

I noticed he smirked. I could tell that for him the code, while valuable, was just a means to an end. Not much pride in workmanship.

The 'throw it away' mentality is very real.

Probably won't get that job._

What do you think? Will AI lead to reduced engineering skills and more 'throw it away' mentality? Share your thoughts in the comments.

Thanks!

Useful Python Libraries for AI/ML

J Fowler — Sat, 10 Aug 2024 16:52:52 +0000

Here's a list of useful python libraries if you are working in ML

pandas - The standard data analysis and manipulation tool
numpy - scientific computing library
seaborn - statistical data visualization
sklearn - basic machine learning and predictive analysis
CausalML - a suite of uplift modeling and causal inference methods
PyTorch - professional deep learning framework
PivotTablejs - Drag’n’drop Pivot Tables and Charts for Jupyter/IPython Notebook
LazyPredict - build and work with and compare multiple models
phidata - Build AI Assistants with memory, knowledge and tools.
Lux - automates visualization and data analysis
pycaret - low-code machine learning library. really nice
Cleanlab - for when you are working with messy data
drawdata - draw a dataset from inside Jupyter
pyforest - lazy import popular data science libs
streamlit - simple ui builder, useful for demonstrating ML results

This is just a snapshot. Let me know your favorites and I'll add them.

Thanks!

Creating a Calculator app using React/Next

J Fowler — Fri, 09 Aug 2024 22:59:22 +0000

In this post, we will build a simple React app using Next.js. Nothing complicated, just a simple calculator app that will demonstrate a number of features of React.

Note that the full source code can be found at the link given at the end of this post.

The calculator will start off simple, just add, subtract, multiply and divide. Maybe we will add something later.

We'll start by getting the environment setup. For Next, you need to have node version 18 or later. I am using 20.10. Next.js will setup the default project structure for you. This is recommended, so use the following:

>npx create-next-app@latest
✔ What is your project named? … calculator
✔ Would you like to use TypeScript? … No / **Yes**
✔ Would you like to use ESLint? … No / **Yes**
✔ Would you like to use Tailwind CSS? … No / **Yes**
✔ Would you like to use `src/` directory? … **No** / Yes
✔ Would you like to use App Router? (recommended) … No / **Yes**
✔ Would you like to customize the default import alias (@/*)? … **No** / Yes
Creating a new Next.js app in /projects/react/calculator.

Using npm.
>cd calculator

There is now a basic React/Next application in the calculator folder. You can test the app using the following:

npm run dev

Now, to finish up the environment. We'll be using the ShadCN ui components, so let's install those:

npx shadcn-ui@latest init

We will be using the ShadCN button component, so install that:

npx shadcn-ui@latest add button

Before we get too far along, let's consider what the UI will look like and how it is used.

The UI will have a display that show the current number entered or the result of the latest operation. There will be ten keys, 0 to 9, the decimal key: '.', the operation keys: +. -, *, /, and, of course, the =.

First, think about how a user interacts with the calculator. In a typical use case, the user enters the first number, or operand, the operation to be performed, the second number, or operand, then presses the = key to perform the operation.

Now, consider local memory needs. The app needs to remember the first operand, the second operation, and the operation, in order to perform the calculation when the = is pressed. We will use React's 'useState' to remember what the use pressed.

Now, let's start building the calculator now.

We create the calculator component in the components folder, calculator.tsx:

"use client"

import React, { useState } from 'react';
import { Button } from "@/components/ui/button"

const Calculator = () => {
  const [display, setDisplay] = useState('0');
  const [firstOperand, setFirstOperand] = useState({"isSet":false,"num":0});
  const [operator, setOperator] = useState("");
  const [waitingForSecondOperand, setWaitingForSecondOperand] = useState(false);

...

}

export default Calculator;

At the start of the component, we have defined the local memory, or state, that is needed. There is a display that holds the most recent number entered, an object to hold the first operand and a flag designating if it has been entered, an object to hold the operator, and an object to hold a boolean telling us if the second operand has been entered. Note that when the second operand is entered it is held in the 'display' state variable, so we only need a boolean to tell us if it has been entered

Let's look at the component layout.

  return (
    <div className="w-64 mx-auto mt-10 bg-gray-100 rounded-lg shadow-lg p-4">
      <div className="bg-white h-16 mb-4 flex items-center justify-end px-4 text-3xl font-bold rounded">
        {display}
      </div>
      <div className="grid grid-cols-4 gap-2">
        <Button onClick={() => inputDigit(7)}>7</Button>
        <Button onClick={() => inputDigit(8)}>8</Button>
        <Button onClick={() => inputDigit(9)}>9</Button>
        <Button onClick={() => performOperation('/')}>/</Button>
        <Button onClick={() => inputDigit(4)}>4</Button>
        <Button onClick={() => inputDigit(5)}>5</Button>
        <Button onClick={() => inputDigit(6)}>6</Button>
        <Button onClick={() => performOperation('*')}>*</Button>
        <Button onClick={() => inputDigit(1)}>1</Button>
        <Button onClick={() => inputDigit(2)}>2</Button>
        <Button onClick={() => inputDigit(3)}>3</Button>
        <Button onClick={() => performOperation('-')}>-</Button>
        <Button onClick={() => inputDigit(0)}>0</Button>
        <Button onClick={inputDecimal}>.</Button>
        <Button onClick={() => performOperation('=')}>=</Button>
        <Button onClick={() => performOperation('+')}>+</Button>
        <Button onClick={clear} className="col-span-4">Clear</Button>
      </div>
    </div>
  );

That is the standard 4 rows of 4 keys of a simple calculator.

We need to update the default page.tsx file to use the component.

import Calculator from '@/components/calculator';

export default function Home() {
  return (
    <main className="flex min-h-screen flex-col items-center justify-center p-24">
      <h1 className="text-4xl font-bold mb-8">Calculator App</h1>
      <Calculator />
    </main>
  );
}

This is the main page view and it simply loads the Calculator component.

In the calculator component, we defined some events. This is the main logic of the UI. When a user presses a numeric button, the component should respond by storing the number and putting it in the display. The inputDigit method handles this logic.

  const inputDigit = (digit: number) => {
    if (waitingForSecondOperand) {
      setDisplay(String(digit));
      setWaitingForSecondOperand(false);
    } else {
      setDisplay(display === '0' ? String(digit) : display + digit);
    }
  };

It's pretty simple, if the user has not entered an operation this must be a digit in the first operand in which case we need only append the digit to whatever is already in the display.

Similarly, if the component is waiting on the second operation, then we need to put the digit in the display and remove the 'waitingForSecondOperand' flag. Note that there is no problem if the second operand has multiple digits because the component expects it to be stored in the display which is handled by the 'else' case.

The inputDecimal function handles the case of the user entering a decimal point. Nothing complicated here, the logic simply appends the '.' to the number in the display.

  const inputDecimal = () => {
    if (!display.includes('.')) {
      setDisplay(display + '.');
    }
  };

The performOperation handles the event when the user presses the '=' of one of the math operation keys.

  const performOperation = (nextOperator: string ) => {
    const inputValue = parseFloat(display);

    if (!firstOperand.isSet) {
      setFirstOperand({"isSet":true,"num":inputValue});
    } else if (operator!="" && nextOperator==="=") {
      const result = calculate(firstOperand.num, inputValue, operator);
      setDisplay(String(result));
      setFirstOperand({"isSet":true,"num":result});
    }

    setWaitingForSecondOperand(true);
    setOperator(nextOperator);
  };

Let's walk through the logic. If the user has entered a number then presses a math operation, say the '+' key, the firstOperand has not been 'set' as far as the component is concerned. The first operand is in the display state variable, so we need to pull it from there and set the firstOperand state variable.

Otherwise, if the key is the '=' and operation has already been given, then the calculation can take place and we put the result as the firstOperand of any followup operation.

In all cases, the component can assume that it is now in the state of having the firstOperand and is waiting for the second.

The clear function simply resets the display and all the state so the component is waiting for the first number to be entered.

  const clear = () => {
    setDisplay('0');
    setFirstOperand({"isSet":false,"num":0});
    setOperator("");
    setWaitingForSecondOperand(false);
  };

All this allows the calculator to handle a sequence of simple operations. For example, consider the key sequence:
1 + 5 = / 3 = properly returns 2.

Note that this implementation will not handle more complicated sequences such as:
7 + 9 / 8 =
That is a bit more complex since it would need to follow precedence of operations rules. Note that this calculator would return 0.875 and one that was based on the programming rules of precedence, the answer would be 8.125, not 2.

This is pretty simple as calculators go, but serves to demonstrate basic React app structure.

What would you do different? How would you modify it to handle to accept parentheses to do longer calculations? Let me know your thoughts in the comments.

Thanks!

The full code for this can be found here

Implement an LRU Cache in Go

J Fowler — Thu, 01 Aug 2024 21:31:18 +0000

So you need a small cache and can't justify a Redis or memcached instance. Let's see what it takes to implement one in Go. For fun, we'll make it using generics so it is reusable in our project.

An LRU cache generally has a fixed capacity and the simplest ejection policy: eject the element that has the longest time since it was accessed. A simple lru cache will implement the following interface:

type LRUCache[T any] interface {
    Get(key string) (value T, found bool)
    Put(key string, value T)
    Keys() []string
    Remove(key string) bool
    Clear()
    Capacity() int
    Len() int
}

We know the cache will store a data item as an entry that is keyed by some value. That sounds like a map. What about implementing the ejection policy? One way to do this is to keep a timeAccessed property along with each item. Something like:

type cacheEntry[T any] struct {
  Data T
  LastAccessed time.time
}

However, let's think about performance, we want to be able to search for the cache key as well as insert and evict the oldest, if necessary, as fast as possible.

Using a map, which is a hashtable, will give us pretty fast performance for lookups. What about finding the oldest entry? If your cache struct looks like:

type LRUCache[T any] {
  capacity int
  keyMap map[string]cacheEntry[T]
}

We will necessarily need to iterate over the map to find the oldest when it comes time to evict an entry.

We need a way to store entries in a way that will efficiently allow us to keep a list of cache entries sorted. It is preferable that we not need to use a sort routine.

A double linked list is a good way to do this and we don't need to store access time in the entry unless we actually want it. So let's suppose we have a linked list that implements the following along with its node struct:

type DoubleLinkedList[T any] interface {
    Head() *DoubleNode[T]
    Tail() *DoubleNode[T]
    // Append inserts new item at tail
    Append(data T) *DoubleNode[T]
    // Push appends new item at head
    Push(data T) *DoubleNode[T]
    Remove(node *DoubleNode[T]) *DoubleNode[T]
    RemoveTail() *DoubleNode[T]
    MoveToHead(node *DoubleNode[T])
}
type DoubleNode[T any] struct {
    Data T
    Prev *DoubleNode[T]
    Next *DoubleNode[T]
}

The cache struct can now look something like:

type lruCache[T any] struct {
    capacity int
    keyMap   map[string]*DoubleNode[lruCacheEntry[T]]
    list     DoubleLinkedList[lruCacheEntry[T]]
}

The cache entry struct will be:

type lruCacheEntry[T any] struct {
    key   string
    value T
}

Realistically, you would probably use an interface for the cache key. I am using a string to keep the code simple.

In the implementation here, the most recently accessed entry in the cache will be at the head and the least recently used will be at the tail. So, when it comes time to evict, we simply remove the tail element of the linked list.

Implementing the Get() function is simple:

func (l *lruCache[T]) Get(key string) (value T, found bool) {
    if node, ok := l.keyMap[key]; ok {
        l.list.MoveToHead(node)
        return node.Data.value, ok
    }
    var zero T
    return zero, false
}

Get just needs to retrieve the map entry for the key, then move the node to the head of the list since it it now the 'most recently used'.

The Put() function is where we will handle the eviction if it's necessary:

func (l *lruCache[T]) Put(key string, value T) {
    if node, ok := l.keyMap[key]; ok {
        node.Data = lruCacheEntry[T]{
            key:   key,
            value: value,
        }
        // move the element to the most recent position
        l.list.MoveToHead(node)
    } else {
        // insert the new element at the head
        newNode := l.list.Push(lruCacheEntry[T]{
            key:   key,
            value: value,
        })
        l.keyMap[key] = newNode
    }
    // is eviction necessary
    if len(l.keyMap) > l.capacity {
        nodeRemoved := l.list.RemoveTail()
        delete(l.keyMap, nodeRemoved.Data.key)
    }
}

For Put(), we first check to see if there is already a value for the given key. If so, then update the value and move the node to the head of the list. Otherwise, we create a new cache entry, add it to the list as the head, and add it to our map.

Finally, don't forget to check for capacity. If the new entry puts us over the capacity, we evict the oldest entry which is the tail of the list and delete the entry from our map.

Note that storing the key as part of the cache entry allows us to rapidly delete the key from the map. If we had only stored the data in the cache entry, then we would need to iterate over the map to find it.

This cache is missing something critical for a multi-threaded app. There is no synchronization. Realistically, a cache would be accessed by multiple threads. Synchronization is a complex topic. For our implementation, we can add a mutex to the cache struct:

type lruCache[T any] struct {
    capacity int
    keyMap   map[string]DoubleNode[lruCacheEntry[T]]
    list     DoubleLinkedList[lruCacheEntry[T]]
    mutex    sync.RWMutex
}

then add the following at the start of each function.

    l.mutex.Lock()
    defer l.mutex.Unlock()

Notice that we are using a read/write lock. Some of the functions don't change the structure of the cache, so we can use the read lock method provided, for example the Len() function:

func (l *lruCache[T]) Len() int {
    l.mutex.RLock()
    defer l.mutex.RUnlock()
    return len(l.keyMap)
}

Note that the synchronization strategy chosen here may break down if there are a large number of threads trying to access the cache. It's a complex topic that could be a series of posts in itself.

See the full implementation in the repository given in the link below.

What would you do different to implement a cache? How would you address synchronization? I am interested in hearing your thoughts on this one. There's no single solution to this so put your comments below.

Thanks!

The code for this post and all posts in this series can be found here

The Pipeline Pattern in Go

J Fowler — Mon, 29 Jul 2024 11:00:00 +0000

The pipeline pattern is a powerful way to process data through stages in a concurrent fashion. Each stage performs a different operation on the data then passes on to the next stage.

Using channels to pass the data along, the pipeline pattern can improve performance in many cases.

The idea is really very simple, each stage iterates over a channel, pulling data until there is none left. For each data item, the stage does its operation then passes the result to an output channel and finally closing the channel when no more data is left in the input channel. Closing the channel is important so that the downstream stage will know when to terminate

Create a sequence of numbers, double them, then filter low values, and finally print them to console.

func produce(num int) chan int {
    out := make(chan int)
    go func() {
        for i := 0; i < num; i++ {
            out <- rand.Intn(100)
        }
        close(out)
    }()
    return out
}

func double(input <-chan int) chan int {
    out := make(chan int)
    go func() {
        for value := range input {
            out <- value * 2
        }
        close(out)
    }()
    return out
}

func filterBelow10(input <-chan int) chan int {
    out := make(chan int)
    go func() {
        for value := range input {
            if value > 10 {
                out <- value
            }
        }
        close(out)
    }()
    return out
}

func print(input <-chan int) {
    for value := range input {
        fmt.Printf("value is %d\n", value)
    }
}

func main() {

    print(filterBelow10(double(produce(10))))

}

There is obviously a more readable way to structure main():

func main() {

    input := produce(10)
        doubled := double(input)
    filtered := filterBelow10(doubled)
    print(filtered)

}

Choose your style based on your own preference.

What would you add here? Leave your comments below.

Thanks!

The code for this post and all posts in this series can be found here