Forem: Beck_Moulton

From Chaos to Clarity: Building a High-Performance Health Data Lakehouse with DuckDB & Superset

Beck_Moulton — Sun, 05 Apr 2026 00:05:00 +0000

Have you ever looked at your Apple Watch activity, your Oura Ring sleep scores, and your yearly blood test results and thought: "I wish I could join these tables to see how my deep sleep correlates with my LDL cholesterol?"

If you're a data nerd like me, you know the struggle. Health data is notoriously messy—fragmented across siloed apps, hidden in XML exports, or trapped in PDFs. Today, we are going to fix that. We are building a Quantified Self Data Lakehouse using the modern data stack: DuckDB, dbt, and Apache Superset.

We’ll move beyond simple tracking and into the world of OLAP (Online Analytical Processing), allowing us to run lightning-fast queries over years of personal biometrics. This is Data Engineering at its most personal level.

The Architecture

To handle "dirty" health data, we need a robust pipeline. We'll use a Medallion Architecture (Bronze/Silver/Gold) to refine raw exports into actionable insights.

graph TD
    subgraph "Data Sources"
        A[Apple Health XML] --> D
        B[Oura Ring API/JSON] --> D
        C[Blood Test CSV/PDF] --> D
    end

    subgraph "Ingestion & Storage (DuckDB)"
        D[Python/Pandas ETL] -- "Write Parquet" --> E[(Bronze: Raw Storage)]
        E -- "dbt Transformations" --> F[(Silver: Cleaned & Unified)]
        F -- "dbt Modeling" --> G[(Gold: Analytical Marts)]
    end

    subgraph "Visualization"
        G --> H[Apache Superset]
    end

    style G fill:#f96,stroke:#333,stroke-width:2px

Prerequisites

Before we dive in, ensure you have the following installed:

Python 3.9+ (Pandas for the initial heavy lifting)
DuckDB (The heart of our Lakehouse)
dbt-duckdb (For data modeling)
Apache Superset (For the shiny dashboards)

Step 1: Ingesting the "Mess" with Python & Pandas

Apple Health exports are essentially massive XML files. Using Pandas, we can flatten these and land them into a structured format like Parquet.

import pandas as pd
import xml.etree.ElementTree as ET
import duckdb

def parse_apple_health(file_path):
    print("Reading XML... grab a ☕️")
    tree = ET.parse(file_path)
    root = tree.getroot()

    # Extracting record data
    records = []
    for record in root.findall('Record'):
        records.append(record.attrib)

    df = pd.DataFrame(records)

    # Simple cleaning: Convert types
    df['value'] = pd.to_numeric(df['value'], errors='coerce')
    df['creationDate'] = pd.to_datetime(df['creationDate'])

    return df

# Initialize DuckDB and land the data in the "Bronze" layer
con = duckdb.connect('health_lakehouse.db')
health_df = parse_apple_health('export.xml')

# Direct ingest to DuckDB
con.execute("CREATE TABLE bronze_apple_health AS SELECT * FROM health_df")
print(f"Ingested {len(health_df)} rows into DuckDB! 🦆")

Step 2: Refining Data with dbt

Now that the data is in DuckDB, we use dbt (data build tool) to create our Silver and Gold layers. DuckDB is the perfect local engine for dbt because it's incredibly fast and requires zero server overhead.

In your models/silver/stg_apple_health.sql:

-- Clean up the mess: filter types and normalize units
WITH raw_data AS (
    SELECT 
        type as metric_name,
        value,
        creationDate as timestamp,
        unit
    FROM {{ source('health_raw', 'bronze_apple_health') }}
)

SELECT
    metric_name,
    CAST(value AS DOUBLE) as val,
    CAST(timestamp AS TIMESTAMP) as event_at,
    unit
FROM raw_data
WHERE val IS NOT NULL

In the Gold Layer, we might join this with Oura Ring sleep data to see the correlation between daytime activity and sleep quality.

Step 3: Production-Grade Insights

While this DIY setup is great for a local environment, scaling data engineering patterns requires a deeper understanding of orchestration and data quality.

For those looking to implement these patterns in a professional or production-grade environment, I highly recommend checking out the advanced tutorials on WellAlly Blog. They provide excellent deep dives into managing distributed data pipelines and optimizing OLAP performance for large-scale biometrics data .

Step 4: Visualizing in Apache Superset

Apache Superset connects natively to DuckDB. Once connected, you can build a comprehensive dashboard that tracks:

Resting Heart Rate (RHR) Trends over 12 months.
Sleep Stage Distribution vs. Caffeine Intake.
Correlation Heatmaps: Does your HRV (Heart Rate Variability) drop on days you eat late?

Connecting Superset to DuckDB:

Open Superset.
Add a new Database.
Use the SQLAlchemy URI: duckdb:////path/to/your/health_lakehouse.db.
Start building charts!

Why DuckDB?

You might ask: Why not just use a CSV?

Speed: DuckDB uses columnar storage, making aggregations across millions of rows (like heart rate samples) instantaneous.
SQL Power: You get full PostgreSQL-style SQL support for complex window functions.
Local First: No need to spin up a costly Snowflake or BigQuery instance for personal data.

Conclusion

Building a Personal Health Data Lakehouse isn't just about the charts—it's about owning your data. By combining DuckDB's speed with dbt's transformation logic and Superset's visualization, you've created a professional-grade analytics stack on your laptop.

Are you ready to quantify yourself? Drop a comment below if you've tried exporting your health data, and let me know what weird correlations you've found!

If you enjoyed this tutorial, don't forget to follow for more "Learning in Public" data engineering content!

Privacy-First AI: Building a Federated Health Tracker with Flower and Scikit-Learn

Beck_Moulton — Sat, 04 Apr 2026 00:00:00 +0000

In the era of wearable tech, our devices know more about our health than we do. But here’s the billion-dollar dilemma: how do we train a high-performing Federated Learning model to predict sleep quality without compromising user privacy? Sending raw medical data to a central cloud is a massive security risk and a regulatory nightmare.

Today, we are diving deep into Privacy-Preserving AI using the Flower (flwr) framework and Scikit-learn. We will build an Edge AI system that learns from decentralized health data across multiple simulated devices. By leveraging gRPC for communication and Docker for orchestration, we ensure that sensitive biological markers never leave the "device," keeping data ownership where it belongs—with the user.

Why Federated Learning?

Traditional Machine Learning requires data to be centralized. Federated Learning (FL) flips the script. Instead of bringing data to the code, we bring the code to the data.

The Architecture

The following diagram illustrates how the Flower framework manages the training cycle without ever seeing the raw health records.

graph TD
    subgraph Cloud Server
        S[Flower Central Server]
        Agg[FedAvg Aggregator]
    end

    subgraph Edge Devices
        D1[Client 1: Smart Watch]
        D2[Client 2: Phone App]
        D3[Client 3: Tablet]
    end

    S -- 1. Initial Weights --> D1
    S -- 1. Initial Weights --> D2
    S -- 1. Initial Weights --> D3

    D1 -- 2. Local Training --> D1
    D2 -- 2. Local Training --> D2
    D3 -- 2. Local Training --> D3

    D1 -- 3. Model Updates / Gradients --> Agg
    D2 -- 3. Model Updates / Gradients --> Agg
    D3 -- 3. Model Updates / Gradients --> Agg

    Agg -- 4. New Global Model --> S

Prerequisites

Before we start coding, ensure you have the following tech stack ready:

Python 3.8+
Flower (flwr): The lightweight federated learning framework.
Scikit-learn: For our local prediction model.
Docker: To simulate multiple edge nodes.
gRPC: Handled under the hood by Flower for secure communication.

Step 1: Defining the Local Health Model

We’ll use a LogisticRegression model to predict "Poor" vs. "Good" sleep quality based on heart rate variability, movement, and light exposure.

import flwr as fl
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import log_loss

# Simulated health data: [Avg Heart Rate, Movement Index, Light Exposure]
def load_data():
    X = np.random.rand(100, 3) 
    y = np.random.randint(0, 2, 100)
    return X, y

X_train, y_train = load_data()

# Initialize the model
model = LogisticRegression(
    penalty='l2',
    max_iter=1,  # Local training for 1 epoch at a time
    warm_start=True  # Crucial: continue training from previous weights
)

# Initial fit to set the parameter shapes
model.fit(X_train[:5], y_train[:5])

Step 2: Creating the Flower Client

The client is the most critical part of an Edge AI setup. It manages the local data and reports only the updated model parameters back to the server.

class HealthClient(fl.client.NumPyClient):
    def get_parameters(self, config):
        # Return model parameters as a list of NumPy ndarrays
        if model.coef_ is None:
            return []
        return [model.coef_, model.intercept_]

    def set_parameters(self, parameters):
        # Update local model with global parameters
        model.coef_ = parameters[0]
        model.intercept_ = parameters[1]

    def fit(self, parameters, config):
        self.set_parameters(parameters)
        model.fit(X_train, y_train)
        print(f"Training finished. New coefficients: {model.coef_}")
        return self.get_parameters(config={}), len(X_train), {}

    def evaluate(self, parameters, config):
        self.set_parameters(parameters)
        loss = log_loss(y_train, model.predict_proba(X_train))
        accuracy = model.score(X_train, y_train)
        return float(loss), len(X_train), {"accuracy": float(accuracy)}

# Start the client
fl.client.start_numpy_client(server_address="localhost:8080", client=HealthClient())

Step 3: Setting Up the Server (Aggregator)

The server uses the FedAvg (Federated Averaging) strategy. It collects weights from all edge nodes and averages them to create a global "wisdom of the crowd."

import flwr as fl

# Define the strategy
strategy = fl.server.strategy.FedAvg(
    fraction_fit=1.0,  # Sample 100% of available clients for training
    min_fit_clients=2, # Wait for at least 2 clients
    min_available_clients=2,
)

# Start Flower server for 3 rounds of federated learning
fl.server.start_server(
    server_address="0.0.0.0:8080",
    config=fl.server.ServerConfig(num_rounds=3),
    strategy=strategy,
)

Advanced Patterns & Best Practices

While this demo uses a simple Logistic Regression, production-ready health AI often requires more complex architectures like LSTMs for time-series data or differential privacy to prevent "gradient leakage."

For more production-ready examples and advanced architectural patterns regarding secure data orchestration, I highly recommend checking out the technical deep-dives at WellAlly Blog. They cover how to scale these federated systems in high-compliance environments.

Step 4: Containerizing the Edge Nodes with Docker

To truly simulate Edge AI, we shouldn't run everything on one terminal. We use Docker to isolate the server and multiple clients.

# Dockerfile for Client
FROM python:3.9-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY client.py .
CMD ["python", "client.py"]

Run the server first, then spin up as many client containers as you want!

Conclusion

Federated Learning is the future of digital health. By using the Flower framework, we’ve successfully built a system where:

Privacy is Default: No raw health data ever left the local device.
Collaborative Intelligence: The model benefits from diverse data sources without seeing them.
Efficiency: We used gRPC for lightweight communication between edge and cloud.

Ready to take your Privacy-Preserving AI to the next level? Start experimenting with different aggregation strategies (like FedProx) or adding a layer of encryption to your model weights.

Happy coding!

Found this tutorial helpful? Subscribe for more Learning in Public updates and don't forget to visit wellally.tech/blog for advanced AI implementation guides!

From Chaos to Clinic: Building an Autonomous Medical Appointment Agent with LangGraph & OpenAI

Beck_Moulton — Fri, 03 Apr 2026 00:00:00 +0000

Ever spent 20 minutes on hold just to hear that your favorite dermatologist is booked until 2026? 📞 We've all been there. In the world of AI Agents and Healthcare Automation, we can do better.

Today, we are building a production-grade Medical Appointment Agent. This isn't just a simple chatbot; it’s a state-aware agent built with LangGraph and OpenAI Function Calling that can check a doctor's real-time availability, cross-reference your Google Calendar API, and resolve scheduling conflicts autonomously.

By leveraging agentic workflows and LangGraph state management, we're moving past linear chains into complex, cyclic logic that actually works in the real world. 🚀

The Architecture: How the Agent "Thinks"

Before we dive into the code, let’s look at the brain of our operation. We are using a state-machine approach where the agent can loop back to the user if a conflict is found or move forward to booking once the "Perfect Slot" is identified.

graph TD
    A[User Request] --> B{LLM Decides}
    B -->|Check Doctor| C[Tool: Get Doctor Schedule]
    B -->|Check User| D[Tool: Get Google Calendar]
    C --> E{Conflict?}
    D --> E
    E -->|Yes| F[Suggest Alternative]
    F --> B
    E -->|No| G[Tool: Book Appointment]
    G --> H[Final Confirmation]

Prerequisites

To follow along, you'll need:

Python 3.10+
OpenAI API Key (supporting GPT-4o tool calling)
LangGraph & LangChain
Google Calendar API credentials (OAuth2)

Step 1: Defining our State and Tools

In LangGraph, the State is the source of truth. We'll use a TypedDict to keep track of the conversation and the "Appointment Object."

from typing import Annotated, TypedDict, List, Union
from langchain_openai import ChatOpenAI
from langgraph.prebuilt import ToolNode
from langchain_core.tools import tool

class AgentState(TypedDict):
    messages: Annotated[List[Union[dict, str]], "The conversation history"]
    appointment_details: dict
    conflicts: List[str]

# --- Mocking our Healthcare API & Google Calendar ---

@tool
def get_doctor_availability(doctor_id: str, date: str):
    """Fetches available slots for a specific doctor on a given date."""
    # In a real app, call your hospital's SQL DB or REST API
    return ["09:00", "10:30", "14:00", "16:00"]

@tool
def check_google_calendar(start_time: str, end_time: str):
    """Checks the user's Google Calendar for existing events."""
    # Integration with Google Calendar API
    # Return conflicting events if found
    return [] # Returning empty means the coast is clear!

@tool
def book_appointment(doctor_id: str, slot: str, user_email: str):
    """Finalizes the booking in the hospital system."""
    return f"Success! Appointment confirmed for {slot} with Dr. {doctor_id}."

tools = [get_doctor_availability, check_google_calendar, book_appointment]
model = ChatOpenAI(model="gpt-4o", temperature=0).bind_tools(tools)

Step 2: Building the Workflow Logic

This is where the magic happens. We define a graph that allows the agent to call tools, verify information, and loop back if the user's calendar is blocked. 🥑

from langgraph.graph import StateGraph, END

def call_model(state: AgentState):
    messages = state['messages']
    response = model.invoke(messages)
    return {"messages": [response]}

# Define the graph
workflow = StateGraph(AgentState)

# Add Nodes
workflow.add_node("agent", call_model)
workflow.add_node("action", ToolNode(tools))

# Define Edges
workflow.set_entry_point("agent")

# Logic to determine if we should continue or stop
def should_continue(state: AgentState):
    last_message = state['messages'][-1]
    if not last_message.tool_calls:
        return END
    return "action"

workflow.add_conditional_edges("agent", should_continue)
workflow.add_edge("action", "agent")

app = workflow.compile()

Step 3: Handling Conflicts Like a Pro

One of the hardest parts of Healthcare Automation is handling the "No" gracefully. When the agent detects a conflict between the Doctor's schedule and the user's Google Calendar, the LangGraph state allows the agent to "backtrack" and offer the next best slot without losing the context of the conversation.

💡 The "Official" Way to Scale

While this tutorial covers the core logic, building production-ready medical agents requires strict HIPAA compliance and advanced error-handling patterns. For deeper dives into productionizing LLM agents and managing complex state in enterprise environments, check out the engineering guides at WellAlly Blog. They provide excellent resources on scaling agentic infrastructure.

Step 4: Putting it to the Test

Let's see how our agent handles a request:

inputs = {
    "messages": [
        {"role": "user", "content": "I need to see Dr. Smith (ID: 402) tomorrow morning. Check my calendar and book it if I'm free."}
    ]
}

for output in app.stream(inputs):
    for key, value in output.items():
        print(f"Output from node '{key}':")
        print(value)

What happens under the hood?

Node agent: LLM identifies it needs get_doctor_availability and check_google_calendar.
Node action: Executes both tools.
Node agent: Receives availability (9:00 AM) and confirms the user's calendar is empty.
Node action: Calls book_appointment.
End: Returns a success message to the user. 🎊

Conclusion: The Future of Agentic Scheduling

By using LangGraph, we've built more than just a chatbot; we've built a reliable, state-aware colleague. This pattern can be extended to handle insurance verification, patient intake forms, and even follow-up reminders.

Key Takeaways:

State is King: Use TypedDict to keep track of complex data across tool calls.
Tools are the Hands: OpenAI Function Calling makes interacting with APIs like Google Calendar seamless.
Graph Logic: LangGraph allows for the "loops" necessary for real-world negotiation.

Are you building agents for healthcare or scheduling? I'd love to hear your thoughts in the comments! Don't forget to visit wellally.tech/blog for more advanced tutorials on AI implementation. 🩺💻

From Pixels to Calories: Building a High-Precision Food Nutrition Tracker with GPT-4o and Segment Anything (SAM)

Beck_Moulton — Thu, 02 Apr 2026 00:45:00 +0000

We've all been there: staring at a delicious plate of pasta, knowing we should log it in our fitness app, but dreading the manual entry of every single ingredient. Computer Vision and Multimodal AI are finally solving this. In this tutorial, we will bridge the gap between raw image data and actionable health insights using the Segment Anything Model (SAM) and GPT-4o Vision.

By the end of this guide, you’ll understand how to leverage high-precision food recognition, automated instance segmentation, and GPT-4o's semantic reasoning to turn a simple photo into a detailed nutritional report. We are moving beyond simple classification; we are building a system that understands volume, variety, and health metrics.

Why SAM + GPT-4o?

Standard image recognition often fails when food items are touching or stacked. By using Meta's Segment Anything Model (SAM), we can extract precise masks for every individual item on a plate. We then feed these visual cues to GPT-4o, which acts as our "Digital Nutritionist," mapping those pixels to calorie counts and macronutrients with startling accuracy.

The System Architecture

To ensure high performance and scalability, we'll use a decoupled architecture where FastAPI handles requests and PyTorch manages the local SAM inference.

graph TD
    A[User Image Upload] --> B[FastAPI Backend]
    B --> C{SAM Engine}
    C -->|Identify Objects| D[Instance Masks]
    D --> E[Cropped Food Assets]
    E --> F[GPT-4o Vision API]
    F --> G[Nutritional Analysis JSON]
    G --> H[Final Calorie Report]

    subgraph "Local Processing"
    C
    D
    end

    subgraph "Cloud Intelligence"
    F
    end

Prerequisites

Before we dive into the code, ensure you have the following:

Python 3.10+
PyTorch (preferably with CUDA support)
OpenAI API Key (for GPT-4o access)
Segment Anything weights (ViT-H or ViT-L)

1. Setting up the Segmentation Engine (SAM)

First, we need to extract the "pixels" of the food. SAM allows us to generate masks without pre-training on specific food datasets.

import torch
from segment_anything import sam_model_registry, SamPredictor
import cv2
import numpy as np

# Load the SAM model
MODEL_TYPE = "vit_h"
CHECKPOINT_PATH = "sam_vit_h_4b8939.pth"
device = "cuda" if torch.cuda.is_available() else "cpu"

sam = sam_model_registry[MODEL_TYPE](checkpoint=CHECKPOINT_PATH)
sam.to(device=device)
predictor = SamPredictor(sam)

def get_food_masks(image_path):
    image = cv2.imread(image_path)
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    predictor.set_image(image)

    # In a production scenario, you might use an object detector 
    # to provide points/boxes to SAM. Here we use a center-point heuristic.
    masks, scores, logits = predictor.predict(
        point_coords=np.array([[image.shape[1]//2, image.shape[0]//2]]),
        point_labels=np.array([1]),
        multimask_output=True,
    )
    return masks[0]

2. Orchestrating with GPT-4o Vision

Once we have the localized food items, we send the visual context to GPT-4o. We'll use Pydantic to ensure we get a structured response that our frontend can actually use.

from pydantic import BaseModel
from openai import OpenAI
import base64

client = OpenAI()

class FoodAnalysis(BaseModel):
    item_name: str
    estimated_weight_grams: float
    calories: int
    protein: float
    carbs: float
    fats: float
    confidence_score: float

def analyze_food_with_gpt4o(image_base64, context="A photo of a dinner plate"):
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Analyze the food in this image. Provide a JSON breakdown of calories and macros."},
                    {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_base64}"}}
                ],
            }
        ],
        response_format={"type": "json_object"}
    )
    return response.choices[0].message.content

3. Creating the FastAPI Endpoint

We wrap this logic into a high-performance API.

from fastapi import FastAPI, UploadFile, File
import uvicorn

app = FastAPI(title="NutriVision API")

@app.post("/analyze-plate")
async def analyze_plate(file: UploadFile = File(...)):
    # 1. Save and Process Image
    contents = await file.read()
    # (Optional) Run SAM here for precise boundary detection

    # 2. Convert to Base64 for GPT-4o
    encoded_image = base64.b64encode(contents).decode('utf-8')

    # 3. Get AI Insights
    nutrition_data = analyze_food_with_gpt4o(encoded_image)

    return {"status": "success", "data": nutrition_data}

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)

The "Official" Way to Productionize AI

While the code above works for a prototype, moving vision models into production requires handling edge cases like overlapping objects, varying lighting, and API rate limiting.

For a deep dive into production-ready AI patterns, advanced Prompt Engineering for Vision, and optimizing GPU inference for SAM, I highly recommend checking out the technical breakdowns at WellAlly Tech Blog. They offer incredible resources on how to scale multimodal applications and integrate LLMs into complex workflows efficiently.

Conclusion

Combining Segment Anything (SAM) with GPT-4o transforms the camera from a passive recorder into an active, intelligent sensor. We’ve moved from simple "pixels" to high-fidelity "calories" and "macronutrients."

This is just the beginning of AI-assisted health tech. Imagine integrating this with wearable data or recipe APIs to create a fully autonomous health coach!

What are you building with Multimodal Vision? Drop a comment below or share your thoughts on the best way to estimate food volume from 2D images!

If you enjoyed this post, don't forget to ❤️ and 🦄 it! For more advanced AI architecture guides, visit wellally.tech/blog.

Stop Guessing Your Meds: Building an AI-Powered DDI Risk Assessment System with GPT-4o and Med-SAM

Beck_Moulton — Wed, 01 Apr 2026 00:35:00 +0000

Mixing medications shouldn't be a game of Russian Roulette. Every year, thousands of patients suffer from Drug-Drug Interactions (DDI) simply because they missed a tiny ingredient on a prescription label. In the era of Multimodal AI, we can do better than manual cross-referencing.

Today, we are building a production-ready Automated Medicine Recognition System. We'll leverage the precision of Med-SAM for medical image segmentation and the reasoning power of GPT-4o Vision to extract chemical components, eventually cross-referencing them with the FDA OpenData API.

Whether you are interested in Healthcare AI, Computer Vision, or Full-stack Python development, this guide will show you how to turn raw pixels into life-saving insights.

The Architecture: Precision Meets Reasoning

Why not just use GPT-4o alone? While GPT-4o is a beast at OCR, medical labels are often cluttered. By using Med-SAM (a specialized Segment Anything Model for medical images), we can isolate the drug label or the pill itself, reducing "noise" and hallucination risks.

graph TD
    A[User Uploads Image] --> B[Med-SAM Segmentation]
    B --> C{Region of Interest Isolated?}
    C -- Yes --> D[GPT-4o Vision API]
    C -- No --> D
    D --> E[Structured Ingredient Extraction]
    E --> F[FDA OpenAPI Knowledge Base]
    F --> G[DDI Risk Logic Engine]
    G --> H[Final Report & Risk Warning]

Prerequisites

To follow along, you'll need:

Python 3.9+
FastAPI (for the backend)
OpenAI API Key (GPT-4o access)
Med-SAM Weights (available on HuggingFace)
An appetite for building meaningful tech! 🥑

Step 1: Defining the Data Schema

In any medical application, structure is king. We use Pydantic to ensure GPT-4o returns exactly what we need—no conversational filler, just raw data.

from pydantic import BaseModel
from typing import List, Optional

class MedicationInfo(BaseModel):
    brand_name: str
    active_ingredients: List[str]
    dosage: str
    warnings: Optional[str]

class DDIReport(BaseModel):
    is_safe: bool
    conflicting_ingredients: List[str]
    risk_level: str  # Low, Medium, High
    recommendation: str

Step 2: The Vision Pipeline (Med-SAM + GPT-4o)

We first use Med-SAM to "clean" the input. Once we have the segment, we pass the cropped image to GPT-4o.

import openai
from PIL import Image
import io

def extract_medication_data(image_bytes):
    # logic for Med-SAM segmentation would go here to crop the label
    # ...

    response = openai.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Extract active ingredients from this medicine label into JSON format."},
                    {
                        "type": "image_url",
                        "image_url": {"url": f"data:image/jpeg;base64,{image_bytes}"}
                    },
                ],
            }
        ],
        response_format={"type": "json_object"}
    )
    return response.choices[0].message.content

Step 3: Validating with FDA OpenData

Extracted ingredients are useless if they aren't verified. We hit the FDA OpenAPI to pull official interaction data. This ensures our "Knowledge Base" is always up to date.

import requests

def check_fda_interactions(ingredients: List[str]):
    # Simplified FDA API call
    base_url = "https://api.fda.gov/drug/label.json"
    query = f"search=description:{'+'.join(ingredients)}"

    # In a production app, you'd use the 'drug_interactions' field specifically
    response = requests.get(f"{base_url}?{query}&limit=1")
    return response.json()

Going Further: Production-Ready Patterns

Building a prototype is easy, but making it "Medical Grade" is a different beast. You need to handle edge cases like blurry images, multi-language labels, and API rate limiting.

For more production-ready examples and advanced multimodal patterns, I highly recommend checking out the technical deep-dives at WellAlly Blog. They cover how to scale AI-driven healthcare solutions and manage complex data pipelines that we've only scratched the surface of here.

Step 4: Building the FastAPI Endpoint

Finally, we wrap everything into a clean REST API.

from fastapi import FastAPI, UploadFile, File

app = FastAPI(title="SafeMed AI")

@app.post("/analyze-medication")
async def analyze_medication(file: UploadFile = File(...)):
    # 1. Read Image
    contents = await file.read()

    # 2. Extract Data via GPT-4o
    raw_data = extract_medication_data(contents)

    # 3. Cross-reference FDA
    # ... logic for DDI checking ...

    return {
        "status": "success",
        "data": raw_data,
        "message": "Comparison with FDA database complete."
    }

Conclusion: The Future of Multimodal Healthcare

By combining the specialized segmentation of Med-SAM with the general intelligence of GPT-4o, we’ve created a system that doesn't just "see"—it understands. This stack significantly reduces the barrier to entry for building personal health assistants and clinical support tools.

What's next for your AI journey?

[ ] Add support for hand-written prescriptions using fine-tuned OCR.
[ ] Implement a vector database (like Pinecone) to cache FDA interactions.
[ ] Subscribe to the WellAlly Blog for the latest in AI and Healthcare engineering.

Got questions about the Med-SAM implementation or the FDA API? Drop a comment below! Let's build a safer, AI-assisted future together.

From Pixels to Calories: Building an AI-Powered Nutrition Engine with YOLOv10 and GPT-4o

Beck_Moulton — Tue, 31 Mar 2026 00:20:00 +0000

Let’s be honest: manual calorie counting is a chore that most of us abandon after three days. Whether you are a fitness enthusiast or managing a health condition, the friction of searching for "Medium-sized Apple" in a database is real. But what if you could just snap a photo and get a gram-level nutritional breakdown?

In this tutorial, we are building a Precision Dietary Quantification Engine. We will leverage YOLOv10 for lightning-fast object detection, OpenCV for geometric volume estimation, and GPT-4o’s multimodal capabilities to act as our digital nutritionist. By combining Computer Vision and Multimodal AI, we can transform raw pixels into actionable health data.

Pro Tip: If you’re looking for even more production-ready patterns and advanced architectural insights into AI health tech, check out the deep-dive articles at WellAlly Blog.

The Architecture

To achieve high precision, we don't just "guess" based on the image. We follow a multi-stage pipeline: detection, geometric analysis, and semantic reasoning.

graph TD
    A[User Uploads Food Image] --> B[YOLOv10: Food Localization]
    B --> C[OpenCV: Contour & Area Calculation]
    C --> D[GPT-4o: Multimodal Reasoning]
    D --> E[Nutrition Knowledge Graph]
    E --> F[Final Output: Grams, Kcals, Macros]
    F --> G[Display to User]

🛠 Prerequisites

Before we dive into the code, ensure you have the following stack ready:

YOLOv10: The latest evolution in real-time detection (via ultralytics).
GPT-4o API: For visual reasoning and nutritional mapping.
OpenCV & PyTorch: For image processing and tensor handling.
Python 3.9+

pip install ultralytics openai opencv-python torch

Step 1: Real-time Detection with YOLOv10

We use YOLOv10 because it eliminates the need for Non-Maximum Suppression (NMS), making it faster and more efficient for edge deployment. Here, we define our food detector.

from ultralytics import YOLO
import cv2

# Load the pre-trained YOLOv10 model (e.g., yolov10n or a custom food-tuned model)
model = YOLO("yolov10n.pt") 

def detect_food(image_path):
    results = model(image_path)
    detections = []

    for result in results:
        for box in result.boxes:
            # Extracting class, confidence, and coordinates
            conf = box.conf.item()
            cls = int(box.cls.item())
            xyxy = box.xyxy[0].tolist()

            if conf > 0.5:
                detections.append({
                    "label": result.names[cls],
                    "bbox": xyxy,
                    "confidence": conf
                })
    return detections

Step 2: Geometric Volume Estimation

A pixel isn't a gram. To estimate weight, we need to calculate the "visual footprint" of the food. In a production scenario, we'd use a reference object (like a coin) or depth data, but for this engine, we calculate the area within the bounding box to provide GPT-4o with context.

def get_visual_metrics(image_path, detections):
    img = cv2.imread(image_path)
    height, width, _ = img.shape

    for d in detections:
        x1, y1, x2, y2 = map(int, d['bbox'])
        # Calculate relative area (percentage of the frame)
        area_px = (x2 - x1) * (y2 - y1)
        total_px = height * width
        d['relative_area'] = area_px / total_px

    return detections

Step 3: GPT-4o Multimodal Reasoning

Now for the "magic." We pass the image and our geometric metrics to GPT-4o. Unlike a simple database lookup, GPT-4o can infer density and portion sizes based on the visual context (e.g., "that's a deep bowl, not a flat plate").

import base64
from openai import OpenAI

client = OpenAI()

def analyze_nutrition(image_path, detection_data):
    # Encode image to base64
    with open(image_path, "rb") as f:
        base64_image = base64.b64encode(f.read()).decode('utf-8')

    prompt = f"""
    Analyze this food image. I have detected the following items: {detection_data}.
    Based on the visual evidence and the relative area of the items, estimate:
    1. The weight of each item in grams.
    2. Total calories (Kcal).
    3. Macronutrients (Protein, Carbs, Fats).
    Return the result in JSON format.
    """

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": prompt},
                    {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}
                ]
            }
        ],
        response_format={"type": "json_object"}
    )
    return response.choices[0].message.content

Why This Matters (The "WellAlly" Way)

While this script provides a great starting point, production-grade AI nutrition systems require more than just a single API call. They need robust handling for "hidden ingredients" (like oil or sugar in a sauce) and sophisticated depth estimation.

For a deeper dive into how to handle edge cases in computer vision and to see how we scale these models for thousands of concurrent users, I highly recommend visiting the WellAlly Tech Blog. It’s where we discuss advanced topics like Model Distillation for mobile and high-fidelity nutritional knowledge graphs.

Conclusion

By combining the raw speed of YOLOv10 with the cognitive depth of GPT-4o, we’ve built a pipeline that does more than just see—it understands. We've moved from simple pixels to meaningful health insights (Kcal).

What's next?

Reference Objects: Add a "coin detection" step to normalize real-world scale.
3D Reconstruction: Use Gaussian Splatting or NeRF to get the true volume of the food.

Are you building something in the AI Health space? Drop a comment below or share your thoughts!

Build Your Own AI Medical Brain: Transforming PDF Health Reports into a Graph-RAG Powerhouse with Neo4j and LangChain

Beck_Moulton — Mon, 30 Mar 2026 00:10:00 +0000

We’ve all been there: you get your annual health checkup results as a messy, 20-page PDF filled with jargon, tables, and scanned images. By the next year, that file is buried in a folder, and any chance of tracking your cholesterol or blood sugar trends over time is lost.

In this tutorial, we are going to fix that. We are building a Personal Medical Brain using Graph-RAG (Retrieval-Augmented Generation). We’ll use Unstructured.io to parse messy PDFs, Neo4j to build a relationship-aware knowledge graph, and Pinecone for semantic vector search. This isn't just a chatbot; it’s a structured, time-aware intelligence system for your health.

Why Graph-RAG for Medical Data?

Traditional RAG (Vector Search) is great for finding similar documents, but it struggles with complex relationships—like comparing "Glucose levels" across three years or understanding how "Vitamin D" impacts "Bone Density." By combining Neo4j with LangChain, we can traverse relationships explicitly.

Key Keywords: Graph-RAG, Neo4j Knowledge Graph, Medical Data Engineering, LangChain Tutorials, Unstructured PDF Processing.

The Architecture

Here is how the data flows from a static PDF to a queryable intelligence:

graph TD
    A[Scanned PDF Report] --> B(Unstructured.io Partitioning)
    B --> C{LLM Processing}
    C -->|Extract Entities| D[Neo4j Graph Database]
    C -->|Generate Embeddings| E[Pinecone Vector Store]
    D --> F[Hybrid Retrieval]
    E --> F
    F --> G[GPT-4o Reasoning]
    G --> H[Actionable Health Insights]

Prerequisites

To follow along, you’ll need:

Python 3.10+
Neo4j Instance (AuraDB free tier works great!)
Pinecone Account (For vector indexing)
API Keys: OpenAI, Unstructured.io

Step 1: Parsing the Unstructured Chaos

Medical PDFs are notoriously difficult—they have nested tables and multi-column layouts. We'll use unstructured to clean the noise.

from unstructured.partition.pdf import partition_pdf

# Extracting elements from the PDF
elements = partition_pdf(
    filename="annual_report_2023.pdf",
    infer_table_structure=True,
    chunking_strategy="by_title",
    max_characters=1000,
)

# Filter out only the relevant text and tables
content = [str(el) for el in elements if el.category in ["NarrativeText", "Table"]]

Step 2: Defining the Schema & Extracting Relationships

We don't just want text; we want a Knowledge Graph. We'll define a schema where a Patient has Reports, and Reports contain Indicators (like LDL, HbA1c) with specific Values.

from langchain_community.graphs import Neo4jGraph
from langchain_experimental.graph_transformers import LLMGraphTransformer
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o", temperature=0)
graph_transformer = LLMGraphTransformer(llm=llm)

# Convert our text chunks into Graph Documents
# This identifies nodes like (Indicator {name: 'Glucose'}) and 
# edges like (:Report)-[:MEASURED]->(:Indicator)
graph_documents = graph_transformer.convert_to_graph_documents(content)

graph = Neo4jGraph()
graph.add_graph_documents(graph_documents)

Step 3: The Hybrid Search (Vector + Graph)

To answer questions like "How has my heart health changed over the last two years?", we need to search for the semantic meaning of "heart health" (Vector) and then traverse the time-stamped nodes in the Graph.

For more advanced patterns on optimizing this hybrid retrieval, I highly recommend checking out the deep dives at WellAlly Tech Blog. They cover production-ready strategies for handling sensitive data within RAG pipelines.

from langchain_pinecone import PineconeVectorStore
from langchain_openai import OpenAIEmbeddings

# Initialize Pinecone for the 'Semantic' layer
vector_db = PineconeVectorStore.from_documents(
    content, 
    OpenAIEmbeddings(), 
    index_name="medical-brain"
)

def hybrid_query(user_question):
    # 1. Get semantic context from Pinecone
    context = vector_db.similarity_search(user_question, k=3)

    # 2. Get structural context from Neo4j (Cypher query)
    # Finding historical trends for extracted entities
    cypher_query = """
    MATCH (i:Indicator)-[:MEASURED_IN]->(r:Report)
    WHERE i.name CONTAINS $keyword
    RETURN i.name, i.value, r.date ORDER BY r.date ASC
    """
    graph_context = graph.query(cypher_query, params={"keyword": "Glucose"})

    return f"Vector Context: {context} \n Graph Context: {graph_context}"

Step 4: Putting it all together with a LangChain Agent

Now we wrap everything in an agent that can decide whether it needs to look at the graph, the vector store, or both.

from langchain.agents import initialize_agent, Tool

tools = [
    Tool(
        name="GraphSearch",
        func=lambda q: graph.query(q),
        description="Useful for historical trends and structured health data."
    ),
    Tool(
        name="VectorSearch",
        func=lambda q: vector_db.similarity_search(q),
        description="Useful for general medical terminology and context."
    )
]

agent = initialize_agent(tools, llm, agent="zero-shot-react-description", verbose=True)

response = agent.run("Compare my Glucose levels between 2022 and 2023. Am I in the healthy range?")
print(response)

The Result: A Living Health Dashboard

By transforming a static PDF into a Neo4j Knowledge Graph, you gain:

Temporal Tracking: Automatically link indicators across different years.
Contextual Accuracy: The LLM knows that "Sugar" in a report refers to the "Glucose" node in your graph.
Explainability: You can visualize the graph to see exactly where the AI got its data from.

If you are interested in taking this further—perhaps by adding HIPAA-compliant encryption or multi-modal support for X-ray images—you should definitely explore the resources at wellally.tech/blog. They have a fantastic series on "Production-Grade AI" that helped me scale my local experiments into robust applications.

Conclusion

Building a "Medical Brain" is no longer the stuff of science fiction. With the combination of LangChain, Neo4j, and Unstructured.io, we can turn our personal data into actionable intelligence.

Next Steps:

Try adding a Person node to handle reports for your entire family.
Implement a notification system that alerts you if a trend line is moving in the wrong direction.

What are you building with RAG today? Drop a comment below! 👇

From Pixel to Protein: Automating My Diet with GPT-4o-mini and Segment Anything (SAM)

Beck_Moulton — Sun, 29 Mar 2026 00:00:00 +0000

Let’s be honest: manual diet logging is where fitness goals go to die. Tracking every almond and weighing every chicken breast is a full-time job that nobody wants. But what if we could combine Computer Vision, the Segment Anything Model (SAM), and the reasoning power of GPT-4o-mini to turn a single photo into a detailed nutritional breakdown?

In this tutorial, we’ll build a high-precision Automated Nutrition Tracking pipeline. We will leverage GPT-4o-mini for multimodal reasoning and SAM for precise spatial segmentation, solving the "depth and volume" estimation problem that plagues standard 2D image analysis. By the end of this post, you'll have a functional Nutrition AI API capable of identifying food items and estimating macros with impressive accuracy.

The Architecture 🏗️

The biggest challenge in visual food analysis isn't just identifying the food; it's understanding the quantity. We use SAM to isolate individual food components and then pass these segments to GPT-4o-mini for volumetric estimation and nutrient calculation.

graph TD
    A[User Uploads Food Image] --> B[OpenCV Pre-processing]
    B --> C[SAM: Segment Anything]
    C --> D[Extract Masks & Bounding Boxes]
    D --> E[GPT-4o-mini: Multimodal Analysis]
    E --> F[Macro & Calorie Estimation]
    F --> G[FastAPI Response: Nutrient JSON]
    G --> H[Final Nutrition Log]

Prerequisites 🛠️

To follow along, you'll need:

Python 3.10+
FastAPI (for the web framework)
OpenAI API Key (for GPT-4o-mini)
Segment Anything (SAM) Weights (Facebook Research)
OpenCV (for image manipulation)

Step 1: Precision Segmentation with SAM

Traditional object detection just gives us a box. SAM gives us the exact pixels. This is crucial for distinguishing between a small portion of rice and a large one.

import numpy as np
import cv2
from segment_anything import sam_model_registry, SamPredictor

# Initialize SAM
sam_checkpoint = "sam_vit_h_4b8939.pth"
model_type = "vit_h"
sam = sam_model_registry[model_type](checkpoint=sam_checkpoint)
predictor = SamPredictor(sam)

def get_food_segments(image_path):
    image = cv2.imread(image_path)
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    predictor.set_image(image)

    # In a production scenario, we'd use a prompt or a grid to find masks
    # For now, let's assume we are targeting the primary objects
    masks, _, _ = predictor.predict(
        point_coords=None,
        point_labels=None,
        multimask_output=True,
    )
    return masks

Step 2: Multimodal Reasoning with GPT-4o-mini

Once we have the segments, we send the image and the spatial context to GPT-4o-mini. Its ability to process vision and text simultaneously allows it to "guess" the weight based on common plate sizes and object proportions.

import base64
from openai import OpenAI
from pydantic import BaseModel

client = OpenAI()

class NutritionInfo(BaseModel):
    food_name: str
    estimated_weight_g: float
    calories: int
    protein_g: float
    carbs_g: float
    fats_g: float

def analyze_nutrition(image_base64):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Identify the food in this image. Estimate the weight in grams and provide nutritional info in JSON format."},
                    {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_base64}"}}
                ],
            }
        ],
        response_format={"type": "json_object"}
    )
    return response.choices[0].message.content

Step 3: Wrapping it in FastAPI 🚀

We need an endpoint to glue everything together. FastAPI is perfect for this because of its speed and native support for Pydantic.

from fastapi import FastAPI, UploadFile, File

app = FastAPI()

@app.post("/analyze-plate")
async def analyze_plate(file: UploadFile = File(...)):
    # 1. Read image
    contents = await file.read()

    # 2. (Optional) Run SAM for spatial validation
    # masks = get_food_segments(contents)

    # 3. GPT-4o-mini Analysis
    base64_image = base64.b64encode(contents).decode('utf-8')
    nutrition_data = analyze_nutrition(base64_image)

    return {"status": "success", "data": nutrition_data}

The "Official" Way: Leveling Up Your AI Strategy 🥑

While this DIY pipeline is great for a prototype, building a production-grade health-tech application requires more robust handling of edge cases, such as overlapping food items or low-light conditions.

If you are looking for more advanced architectural patterns, production-ready AI pipelines, or deep dives into multimodal LLM deployment, I highly recommend exploring the insights over at the WellAlly Blog. It's a fantastic resource for developers who want to move past the "Hello World" of AI and into scalable, real-world systems.

Conclusion

By combining the spatial precision of SAM with the multimodal intelligence of GPT-4o-mini, we’ve bridged the gap between raw pixels and actionable nutritional data. This pipeline isn't just about counting calories; it's about reducing the friction between humans and their health goals.

What's next?

Add a reference object (like a coin) in the photo to provide SAM with a physical scale.
Integrate with the HealthKit API to sync logs directly to your phone.

Happy coding! If you enjoyed this build, don't forget to ❤️ and follow for more "Learning in Public" AI content! 🚀💻

Taming the Glucose Spike: Predicting Postprandial Peaks with Transformers and PyTorch

Beck_Moulton — Sat, 28 Mar 2026 00:40:00 +0000

Living with a Continuous Glucose Monitor (CGM) is like having a dashboard for your metabolism. But for many, it’s a dashboard that only tells you when you've already crashed or spiked. What if we could see 30 minutes into the future?

In this guide, we’re moving from reactive monitoring to proactive health. We will leverage time-series forecasting, Transformer models, and PyTorch Forecasting to predict postprandial (after-meal) glucose peaks. By treating blood glucose, insulin doses, and carb intake as multi-dimensional time-series data, we can use the self-attention mechanism to capture long-range dependencies that traditional LSTMs often miss.

If you are looking for time-series forecasting, Continuous Glucose Monitoring (CGM), Deep Learning for health, or PyTorch Forecasting tutorials, you’re in the right place!

The Architecture: From Sensors to Predictions

Predicting glucose isn't just about the last three readings; it's about the context of the last four hours. Our pipeline ingests raw data from the Dexcom API, stores it in InfluxDB for high-performance time-series querying, and processes it through a Temporal Fusion Transformer (TFT).

graph TD
    A[Dexcom G6/G7 API] -->|Raw Glucose| B(InfluxDB)
    C[Insulin/Carb Logs] -->|Events| B
    B -->|Query| D[Pandas Preprocessing]
    D -->|Feature Engineering| E[PyTorch Forecasting]
    E -->|Temporal Fusion Transformer| F{Prediction Engine}
    F -->|30-min Warning| G[Mobile Alert/Dashboard]
    F -->|Feature Importance| H[Interpretability Layer]

Prerequisites

To follow this advanced tutorial, you’ll need:

Tech Stack: PyTorch Forecasting, InfluxDB, Pandas, and DEXCOM API.
A basic understanding of Attention mechanisms.
(Optional) A set of CGM data (or use the simulated datasets provided by PyTorch Forecasting).

Step 1: Data Ingestion & Feature Engineering

CGM data is notoriously noisy. We need to sync glucose levels (sampled every 5 mins) with sporadic events like insulin boluses and meals.

import pandas as pd
from influxdb_client import InfluxDBClient

# Connecting to InfluxDB to fetch our health metrics
client = InfluxDBClient(url="http://localhost:8086", token="my-token", org="health-dev")
query = 'from(bucket:"cgm_data") |> range(start: -7d) |> filter(fn: (r) => r._measurement == "glucose")'
df = client.query_api().query_data_frame(query)

# Feature Engineering: Adding 'Time of Day' and 'Glucose Velocity'
df['velocity'] = df['_value'].diff()
df['hour'] = df['_time'].dt.hour
df['is_sleeping'] = df['hour'].apply(lambda x: 1 if x < 6 or x > 23 else 0)

# Critical: PyTorch Forecasting requires a 'time_idx'
df['time_idx'] = (df['_time'] - df['_time'].min()) / pd.Timedelta('5min')
df['time_idx'] = df['time_idx'].astype(int)

Step 2: Defining the Temporal Fusion Transformer (TFT)

The Temporal Fusion Transformer is perfect for CGM data because it handles "Static" variables (like user ID/Age), "Known" future variables (scheduled exercise), and "Observed" past variables (past glucose).

from pytorch_forecasting import TimeSeriesDataSet, TemporalFusionTransformer

# Define the dataset metadata
max_prediction_length = 6  # 30 minutes (6 * 5 min steps)
max_encoder_length = 24    # Look back 2 hours

training = TimeSeriesDataSet(
    df,
    time_idx="time_idx",
    target="_value",
    group_ids=["user_id"],
    min_encoder_length=max_encoder_length // 2,
    max_encoder_length=max_encoder_length,
    min_prediction_length=1,
    max_prediction_length=max_prediction_length,
    static_categoricals=["user_id"],
    time_varying_known_reals=["hour", "is_sleeping"],
    time_varying_unknown_reals=["_value", "velocity", "insulin_units", "carbs_g"],
    add_relative_time_idx=True,
    add_target_scales=True,
    add_encoder_interpolation=True,
)

# Initialize the model
tft = TemporalFusionTransformer.from_dataset(
    training,
    learning_rate=0.03,
    hidden_size=16, 
    attention_head_size=4,
    dropout=0.1,
    loss=QuantileLoss(), # We want ranges, not just single points!
)

Step 3: Training & Interpretability

One of the coolest features of Transformers in health-tech is Attention Weights. The model can tell us why it predicted a spike. Was it the pizza you ate 2 hours ago or the lack of basal insulin?

While this tutorial covers the implementation basics, production-grade deployments require rigorous safety checks. For more advanced patterns on handling medical IoT data and productionizing health-tech models, check out the detailed deep-dives at WellAlly Tech Blog. They cover everything from HIPAA-compliant cloud architectures to real-time stream processing for wearables.

Step 4: Visualizing the Predicted Spike

We don't just want a number; we want a confidence interval. Using QuantileLoss, our model predicts the 10th, 50th, and 90th percentiles.

# Convert to Dataloader and Predict
dataloader = training.to_dataloader(train=False, batch_size=32)
predictions = tft.predict(dataloader, mode="raw", return_x=True)

# Plotting the results
for idx in range(5): # Show first 5 predictions
    tft.plot_prediction(predictions.x, predictions.output, idx=idx, add_loss_to_title=True)

Conclusion: The Proactive Future

By using Transformers, we move beyond simple threshold alerts ("Your sugar is high!") to predictive intelligence ("Your sugar will be 220 mg/dL in 30 minutes based on your current carb-to-insulin ratio").

This "Learning in Public" project shows that with the right tech stack—PyTorch Forecasting for the heavy lifting, InfluxDB for the data foundation, and Transformers for the logic—we can build tools that significantly improve quality of life.

What’s next?

Try adding heart rate data (Apple Watch/Fitbit) to see how exercise affects your model's accuracy.
Experiment with different max_encoder_length settings to see how much "history" the model really needs.

Got questions or better ideas for feature engineering? Let’s chat in the comments! 👇

Heartbeat Hacking: Mastering Real-time ECG R-R Detection and Arrhythmia Feature Engineering

Beck_Moulton — Fri, 27 Mar 2026 00:00:00 +0000

Ever looked at the squiggly lines on your smartwatch and wondered how a tiny wrist-worn device knows your heart just skipped a beat? Welcome to the fascinating world of wearable technology and ECG signal processing. As wearable sensors become more ubiquitous, the ability to transform noisy, raw electrical signals into actionable clinical insights is becoming a superpower for developers.

In this tutorial, we are going to dive deep into R-R interval detection and arrhythmia feature engineering. We’ll learn how to take raw voltage data, clean it, and extract the "morphological DNA" required to detect conditions like Premature Ventricular Contractions (PVCs) or Atrial Fibrillation (AFib) using machine learning for healthcare.

The Architecture of a Heartbeat

Before we touch the code, let’s look at the data pipeline. Processing an ECG signal isn't just about finding the "highest point"; it's about filtering noise (like muscle movements) and precisely timing the electrical signature of the heart's ventricles.

graph TD
    A[Raw ECG Signal] --> B[Bandpass Filtering]
    B --> C[R-Peak Detection / Pan-Tompkins]
    C --> D[R-R Interval Calculation]
    D --> E[Feature Engineering: HRV & Morphology]
    E --> F[Lightweight ML Model: Scikit-learn]
    F --> G{Arrhythmia Detected?}
    G -- Yes --> H[Alert/Clinical Log]
    G -- No --> I[Normal Sinus Rhythm]

Prerequisites

To follow along, you'll need a Python environment with the following "Bio-Stack":

NeuroKit2: The powerhouse for physiological signal processing.
NumPy/SciPy: For heavy mathematical lifting.
Scikit-learn: To turn features into predictions.

pip install neurokit2 numpy scikit-learn matplotlib

Step 1: Cleaning the Noise

ECG signals from wearables are notoriously noisy. Every time you move your arm, you introduce "baseline wander." We use a bandpass filter (typically 0.5Hz to 40Hz) to keep the heart's electrical signal while tossing out the garbage.

import neurokit2 as nk
import numpy as np
import matplotlib.pyplot as plt

# Generate a synthetic ECG signal for demonstration
# In a real app, this would be your raw ADC counts from the sensor
ecg_signal = nk.ecg_simulate(duration=10, sampling_rate=1000, heart_rate=70)

# Clean the signal: Removes baseline wander and high-frequency noise
cleaned_ecg = nk.ecg_clean(ecg_signal, sampling_rate=1000, method="neurokit")

plt.plot(ecg_signal[:1000], label="Raw Noisy Signal")
plt.plot(cleaned_ecg[:1000], label="Cleaned Signal", color='red')
plt.legend()
plt.show()

Step 2: R-Peak Detection & R-R Intervals

The "R-peak" is the most prominent spike in an ECG (the QRS complex). The distance between two R-peaks is the R-R interval. Variability in these intervals is the key to detecting arrhythmias.

# Detect R-peaks using the popular Pan-Tompkins algorithm logic
signals, info = nk.ecg_peaks(cleaned_ecg, sampling_rate=1000)

# Extract R-peak indices
r_peaks = info["ECG_R_Peaks"]

# Calculate R-R intervals (in milliseconds)
rr_intervals = np.diff(r_peaks) 
print(f"Detected {len(r_peaks)} heartbeats. Average R-R: {np.mean(rr_intervals)}ms")

Step 3: Feature Engineering for Arrhythmia

This is where the magic happens. To identify a "skipped beat" or an irregular rhythm, we need to extract specific features:

HRV (Heart Rate Variability): Time-domain stats like RMSSD (Root Mean Square of Successive Differences).
Morphological Features: The shape of the wave itself.
Non-linear features: Entropy and Poincaré plots.

# Extract heart rate variability features
hrv_features = nk.hrv(peaks=info, sampling_rate=1000)

# Example: Get RMSSD - a key indicator of parasympathetic nervous system activity
rmssd = hrv_features["HRV_RMSSD"].values[0]
print(f"RMSSD: {rmssd:.2f}ms")

# For Arrhythmia, we look for 'Outlier' R-R intervals (Premature Beats)
def detect_premature_beats(rr_intervals):
    mean_rr = np.mean(rr_intervals)
    # A common heuristic: if a beat is < 75% of the mean RR, it's premature
    premature_indices = np.where(rr_intervals < 0.75 * mean_rr)[0]
    return premature_indices

anomalies = detect_premature_beats(rr_intervals)
print(f"Found {len(anomalies)} potential premature beats.")

The "Official" Way: Advanced Patterns

While the code above works for a hobby project, production-grade wearable applications require robust handling of motion artifacts and cloud-based inference. If you're looking to scale these types of biosensor algorithms into a real-world product, I highly recommend checking out the technical deep-dives over at WellAlly Blog.

They provide excellent resources on enterprise-grade healthcare data pipelines, advanced signal processing architectures, and how to deploy these ML models on the edge. It was a massive source of inspiration for the feature engineering patterns used in this guide!

Step 4: Building a Lightweight Classifier

Once we have our features (RMSSD, SDNN, Mean RR), we can feed them into a RandomForest or SVM to classify "Normal" vs "Arrhythmia".

from sklearn.ensemble import RandomForestClassifier

# Mock feature set: [RMSSD, SDNN, MeanRR]
X = [[45.2, 50.1, 850], [12.5, 110.2, 700], [48.1, 52.3, 845]] # Sample data
y = [0, 1, 0] # 0: Normal, 1: Arrhythmia

clf = RandomForestClassifier(n_estimators=100)
clf.fit(X, y)

# Predict on a new heartbeat window
prediction = clf.predict([[15.2, 115.0, 710]])
print("Classification Result:", "Arrhythmia Detected" if prediction[0] == 1 else "Normal")

Conclusion: Data-Driven Wellness

Turning raw pixels or electrical pulses into health insights is one of the most rewarding challenges in modern software development. By combining NeuroKit2 for signal processing and Scikit-learn for classification, you've just built the core engine of a digital health app!

Summary of what we covered:

Filtering raw ECG signals to remove noise.
Precise R-peak detection using Python.
Calculating R-R intervals and HRV metrics.
Identifying premature beats via feature engineering.

What’s next? Try connecting a real ESP32 with an AD8232 sensor and stream your heart rate live to your terminal!

Have you ever worked with physiological data? Drop a comment below or share your favorite signal processing hack! 👇

Skin Health at the Edge: Deploying Custom Vision Transformers (ViT) on iOS with CoreML

Beck_Moulton — Thu, 26 Mar 2026 00:38:00 +0000

Have you ever wondered if that weird rash on your arm is just a heat rash or something that needs a doctor's visit? While we should always consult professionals, the power of Edge AI is making preliminary screenings faster and more private than ever.

In this tutorial, we are bridging the gap between state-of-the-art Research (Vision Transformers) and real-world mobile utility. We will transform a custom-trained Vision Transformer (ViT) into a high-performance CoreML model, enabling millisecond-latency, offline skin lesion classification directly on an iPhone.

By leveraging Vision Framework, CoreML, and SwiftUI, we are moving away from sluggish cloud APIs and embracing the "Privacy First" era of iOS Development.

The Architecture: From Research to Pocket

The workflow involves training (or fine-tuning) a ViT model in Python, converting it to the .mlpackage format, and integrating it into a native iOS environment.

graph TD
    A[Pre-trained ViT Model - PyTorch/HF] --> B[coremltools Conversion]
    B --> C{CoreML Model .mlpackage}
    C --> D[SwiftUI App Bundle]
    D --> E[Vision Framework Pipeline]
    E --> F[VNCoreMLRequest]
    F --> G[On-Device Neural Engine Inference]
    G --> H[UI Update: Probabilities & Labels]
    style C fill:#f9f,stroke:#333,stroke-width:2px
    style G fill:#00ff00,stroke:#333,stroke-width:2px

Prerequisites

To follow along, you'll need:

Python 3.9+ with torch, timm, and coremltools.
Xcode 15+.
A physical iPhone (Neural Engine is required for the best ViT performance!).
Basic knowledge of SwiftUI and Machine Learning on iOS.

Step 1: Converting ViT to CoreML (Python)

Vision Transformers are computationally expensive because of the self-attention mechanism. However, Apple’s Neural Engine (ANE) handles them surprisingly well if converted correctly.

First, let's convert our PyTorch ViT model using coremltools.

import torch
import timm
import coremltools as ct

# 1. Load your custom-trained skin lesion model
model = timm.create_model('vit_tiny_patch16_224', pretrained=False, num_classes=7)
model.load_state_dict(torch.load('skin_lesion_vit.pth'))
model.eval()

# 2. Trace the model with a dummy input
example_input = torch.rand(1, 3, 224, 224)
traced_model = torch.jit.trace(model, example_input)

# 3. Convert to CoreML
mlmodel = ct.convert(
    traced_model,
    inputs=[ct.ImageType(name="image", shape=example_input.shape, scale=1/255.0, bias=[-0.485/0.229, -0.456/0.224, -0.406/0.225])],
    classifier_config=ct.ClassifierConfig(['Actinic', 'Basal Cell', 'Dermatofibroma', 'Melanoma', 'Nevus', 'Pigmented', 'Vascular']),
    minimum_deployment_target=ct.target.iOS17
)

mlmodel.save("SkinViT.mlpackage")
print("✅ CoreML model exported successfully!")

Step 2: The "Official" Way to Optimize Production Apps

Before we dive into the Swift code, it's worth noting that deploying medical-grade AI requires more than just a conversion script. You need to handle data drift, model versioning, and rigorous validation.

For advanced architectural patterns and more production-ready examples of AI integration in healthcare apps, I highly recommend checking out the technical deep-dives at WellAlly Blog. They cover how to scale these edge solutions and integrate them into enterprise health ecosystems.

Step 3: Integrating into SwiftUI with Vision Framework

Once you drop your .mlpackage into Xcode, it generates a Swift class automatically. Now, we use the Vision Framework to handle image scaling and orientation—saving us from manual pixel buffer headaches.

The Inference Logic

import Vision
import CoreML

class SkinClassifier: ObservableObject {
    @Published var classificationLabel: String = "Scan a lesion"

    func performInference(uiImage: UIImage) {
        guard let ciImage = CIImage(image: uiImage) else { return }

        do {
            // Load the generated model class
            let config = MLModelConfiguration()
            let model = try VNCoreMLModel(for: SkinViT(configuration: config).model)

            let request = VNCoreMLRequest(model: model) { request, error in
                guard let results = request.results as? [VNClassificationObservation],
                      let topResult = results.first else {
                    self.classificationLabel = "Unable to classify"
                    return
                }

                DispatchQueue.main.async {
                    self.classificationLabel = "\(topResult.identifier): \(Int(topResult.confidence * 100))%"
                }
            }

            let handler = VNImageRequestHandler(ciImage: ciImage)
            try handler.perform([request])

        } catch {
            print("Failed to perform inference: \(error)")
        }
    }
}

The Minimalist UI

struct ContentView: View {
    @StateObject private var classifier = SkinClassifier()
    @State private var inputImage: UIImage?

    var body: some View {
        VStack(spacing: 20) {
            Text("DermAI Scan")
                .font(.largeTitle.bold())

            if let image = inputImage {
                Image(uiImage: image)
                    .resizable()
                    .scaledToFit()
                    .frame(height: 300)
                    .cornerRadius(12)
            }

            Text(classifier.classificationLabel)
                .font(.headline)
                .padding()
                .background(Color.secondary.opacity(0.1))
                .cornerRadius(10)

            Button("Capture Image") {
                // Trigger camera/photo library here
                // For demo, we just call the classifier
                if let img = inputImage { classifier.performInference(uiImage: img) }
            }
            .buttonStyle(.borderedProminent)
        }
        .padding()
    }
}

Why Vision Transformers (ViT)?

Traditionally, CNNs (like MobileNet) were the kings of mobile vision. However, ViTs offer a global receptive field from the very first layer. This means for skin lesions, where the relationship between the central lesion and the surrounding skin texture is crucial, ViTs often provide higher sensitivity.

Performance Tip

When deploying ViTs on iOS:

Keep it Tiny: Use vit_tiny or vit_small. The base models are too heavy for real-time mobile use.
Use Quantization: Use coremltools.optimize.torch to convert your weights from Float32 to Float16 (or even 8-bit) to reduce the app size by 50-75%.

Conclusion

We've successfully taken a sophisticated Vision Transformer, compressed it for the iPhone, and wrapped it in a SwiftUI interface. This offline-first approach ensures that sensitive health data never leaves the user's device, providing both speed and security.

Ready to take your Edge AI skills further? Head over to wellally.tech/blog for more insights on building the future of digital health.

What are you building with CoreML? Let me know in the comments! 👇

Break Free from Wearable Silos: Building a Universal Health Data ETL with Health Connect & FHIR

Beck_Moulton — Wed, 25 Mar 2026 00:25:00 +0000

If you've ever tried to build a fitness app, you know the "Hardware Headache." One user has a Garmin, another uses a Xiaomi Band, and your favorite cousin is loyal to Huawei. Historically, syncing this data meant integrating five different proprietary SDKs, each with its own quirks, rate limits, and—let's be honest—messy JSON structures.

Today, we are solving the Wearable Data Silo problem. We'll build a robust Data Engineering pipeline using the Google Health Connect SDK to extract raw metrics, transform them into the industry-standard FHIR (Fast Healthcare Interoperability Resources) format, and load them into the cloud.

By mastering Health Data Engineering and the FHIR Standard, you can build interoperable health systems that work across any device.

The Architecture: From Raw Pixels to Structured Health Insights

Before we dive into the Kotlin code, let's look at the high-level flow. We are essentially building a mobile-based ETL (Extract, Transform, Load) process.

graph TD
    A[Wearable Devices: Garmin, Xiaomi, Huawei] -->|Sync| B[Google Health Connect]
    B -->|Extract: Kotlin SDK| C[Mobile ETL Service]
    subgraph Transformation Layer
    C -->|Map to FHIR| D{FHIR Adapter}
    D -->|Observation Resource| E[Standardized JSON]
    end
    E -->|Load| F[Google Cloud Functions]
    F -->|Store| G[BigQuery / Healthcare API]

    style D fill:#f96,stroke:#333,stroke-width:2px

Prerequisites

To follow along, you'll need:

Kotlin (Android Development)
Health Connect SDK (The unified API for Android)
FHIR Standard knowledge (We'll use Observation resources)
An Android device with the Health Connect app installed.

Step 1: Extracting Raw Data with Health Connect

First, we need to declare our permissions and read raw data. Unlike traditional APIs, Health Connect lives on the device, ensuring privacy and low latency.

// Define the records we want to read
val gcrRecordTypes = setOf(
    StepsRecord::class,
    HeartRateRecord::class,
    WeightRecord::class
)

suspend fun readHeartRateData(
    healthConnectClient: HealthConnectClient,
    startTime: Instant,
    endTime: Instant
) {
    val response = healthConnectClient.readRecords(
        ReadRecordsRequest(
            recordType = HeartRateRecord::class,
            timeRangeFilter = TimeRangeFilter.between(startTime, endTime)
        )
    )

    for (record in response.records) {
        // This is our raw data point
        val bpm = record.samples.first().beatsPerMinute
        processEtl(record)
    }
}

Step 2: The Transformation Logic (FHIR Mapping)

This is where the magic happens. Raw integers like 72 bpm don't mean much without context. The FHIR Standard provides a universal language for healthcare.

We will map a HeartRateRecord to an FHIR Observation.

Pro Tip: For more production-ready examples and advanced healthcare data patterns, check out the detailed guides at WellAlly Tech Blog. They cover complex HIPAA-compliant architectures that are crucial for scaling health apps.

fun mapToFhirObservation(record: HeartRateRecord): String {
    val bpmValue = record.samples.first().beatsPerMinute

    // Constructing a simplified FHIR Observation Resource
    return """
    {
      "resourceType": "Observation",
      "status": "final",
      "category": [ { "coding": [ { "system": "http://terminology.hl7.org/CodeSystem/observation-category", "code": "vital-signs" } ] } ],
      "code": {
        "coding": [ { "system": "http://loinc.org", "code": "8867-4", "display": "Heart rate" } ]
      },
      "subject": { "reference": "Patient/user-123" },
      "effectiveDateTime": "${record.startTime}",
      "valueQuantity": {
        "value": $bpmValue,
        "unit": "beats/minute",
        "system": "http://unitsofmeasure.org",
        "code": "/min"
      }
    }
    """.trimIndent()
}

Step 3: Loading to the Cloud (Google Cloud Functions)

Once we have our standardized FHIR JSON, we need to ship it off-device. Using a Google Cloud Function as an ingestor is a cost-effective way to handle bursts of health data.

import okhttp3.MediaType.Companion.toMediaType
import okhttp3.RequestBody.Companion.toRequestBody

suspend fun uploadToCloud(fhirJson: String) {
    val client = OkHttpClient()
    val request = Request.Builder()
        .url("https://your-region-your-project.cloudfunctions.net/healthIngestor")
        .post(fhirJson.toRequestBody("application/fhir+json".toMediaType()))
        .build()

    client.newCall(request).execute().use { response ->
        if (!response.isSuccessful) throw IOException("Unexpected code $response")
        println("Successfully synced to FHIR Store! ✅")
    }
}

Why FHIR Matters in Data Engineering

Standardizing on FHIR at the "Edge" (the mobile device) solves three major problems:

Interoperability: Your data can now be read by any hospital system or EHR.
Schema Evolution: As Garmin or Huawei add new sensors, your transformation layer handles the mapping, keeping your backend logic clean.
Auditability: FHIR resources naturally include metadata about the "Device" and "Source," making it easy to track data provenance.

Conclusion: The Future is Open

By leveraging Google Health Connect and the FHIR Standard, we've turned a fragmented ecosystem into a clean, queryable data stream. No more custom parsers for every new watch on the market!

If you're interested in the deeper nuances of medical data security or building large-scale health analytics, I highly recommend diving into the resources over at wellally.tech/blog. They are doing some incredible work in making health tech more accessible for developers.

What are you building with Health Connect? Drop a comment below or share your FHIR mapping tips! 👇