Forem: wellallyTech

From Scans to Structured Data: Converting Medical Reports to JSON with Pydantic & LLMs

wellallyTech — Wed, 06 May 2026 01:30:00 +0000

Have you ever looked at a stack of physical medical reports and wished you could just "Ctrl+F" your health history? 📑 We’ve all been there. Every hospital has a different layout, different units, and cryptic abbreviations that make manual data entry a nightmare.

In the world of data engineering, turning unstructured "messy" documents into structured data extraction pipelines is a superpower. Today, we’re going to build a robust system that uses Pydantic, Instructor, and Azure Form Recognizer to transform scanned medical reports into standardized JSON (following medical standards like LOINC) with 100% type safety. 🚀

Why "Prompting" isn't enough

If you just throw OCR text at an LLM and ask for "JSON," it will eventually fail. It might hallucinate a field, change a data type, or forget a closing bracket. To build production-grade health tech, we need validation.

By combining Pydantic for schema definition and Instructor for steering the LLM, we ensure that the output isn't just "JSON-like"—it's a strictly typed Python object.

The Architecture: From Pixels to Patterns

Here is how the data flows from a blurry JPEG to a clean, queryable database:

graph TD
    A[Scanned Report/Image] -->|OCR Extraction| B[Azure AI Document Intelligence]
    B -->|Raw Text & Tables| C[Instructor + LLM]
    C -->|Schema Enforcement| D[Pydantic Model]
    D -->|Validation Check| E{Is it Valid?}
    E -->|No| C
    E -->|Yes| F[Standardized JSON - LOINC Compatible]
    F -->|Storage| G[PostgreSQL/Vector DB]

Step 1: Defining the Medical Schema

First, we define exactly what a "Medical Test" looks like. We want to capture the test name, the result, the unit, and that pesky reference range.

from pydantic import BaseModel, Field
from typing import List, Optional

class MedicalTestResult(BaseModel):
    test_name: str = Field(..., description="The name of the test, e.g., 'Hemoglobin' or 'HbA1c'")
    value: float = Field(..., description="The numerical result of the test")
    unit: str = Field(..., description="The measurement unit, e.g., 'g/dL' or 'mmol/L'")
    is_normal: bool = Field(..., description="Flag indicating if the result is within the reference range")
    reference_range: Optional[str] = Field(None, description="The normal range provided by the lab")

class HealthReport(BaseModel):
    patient_name: Optional[str]
    report_date: str
    hospital_name: str
    results: List[MedicalTestResult]

Step 2: Extracting Text with Azure Form Recognizer

Before the LLM can "understand" the report, we need to extract the text. Azure AI Document Intelligence (formerly Form Recognizer) is fantastic at handling tables in scanned PDFs.

from azure.ai.formrecognizer import DocumentAnalysisClient
from azure.core.credentials import AzureKeyCredential

def extract_raw_text(file_path: str):
    client = DocumentAnalysisClient(
        endpoint="YOUR_AZURE_ENDPOINT", 
        credential=AzureKeyCredential("YOUR_KEY")
    )

    with open(file_path, "rb") as f:
        poller = client.begin_analyze_document("prebuilt-layout", document=f)
        result = poller.result()

    return result.content # Returns the full text content

Step 3: The Magic Hook (Instructor + LLM)

This is where the magic happens. Instead of using the raw OpenAI SDK, we use Instructor. It patches the OpenAI client so that it returns a Pydantic object directly.

import instructor
from openai import OpenAI

# Patch the client to add 'response_model' support
client = instructor.patch(OpenAI())

def parse_report_with_llm(raw_text: str) -> HealthReport:
    return client.chat.completions.create(
        model="gpt-4o",
        response_model=HealthReport,
        messages=[
            {"role": "system", "content": "You are a medical data specialist. Extract data into the specified JSON format. Map common names to LOINC standards where possible."},
            {"role": "user", "content": f"Extract data from this report: {raw_text}"}
        ],
        max_retries=3 # If validation fails, it will automatically retry!
    )

🥑 Level Up: Advanced Patterns

While this setup works for basic reports, production environments often require handling multi-page documents, handling PII (Personally Identifiable Information), and mapping values to global standards like LOINC or SNOMED CT.

For a deeper dive into scaling these pipelines and implementing advanced medical data architectures, I highly recommend checking out the WellAlly Tech Blog. They have some incredible resources on high-performance data engineering and production-ready AI patterns that go far beyond a simple tutorial.

Why this matters

By structuring this data, we move from "Pictures of Documents" to "Actionable Insights." 📈

Trend Analysis: Plot your glucose levels over 5 years.
Early Detection: Use algorithms to spot patterns across different hospitals.
Portability: Easily share your data with new doctors without carrying a physical folder.

Conclusion

Structuring messy medical data doesn't have to be a headache. With the Pydantic + Instructor stack, you get the reasoning power of an LLM with the strictness of a compiler.

What are you building next? Are you going to automate your lab results or perhaps build a custom health dashboard? Let me know in the comments below! 👇

Happy coding! If you enjoyed this post, don't forget to ❤️ and 🦄 it!

From Snore to Score: Real-time Sleep Apnea Detection using Whisper v3 and FFT on the Edge 😴🔊

wellallyTech — Tue, 05 May 2026 00:45:00 +0000

Have you ever wondered what’s actually happening while you sleep? Beyond the dreams of flying or forgetting your pants at a meeting, your breathing patterns tell a vital story about your health. Traditional sleep studies (polysomnography) involve being strapped to a dozen wires in a cold clinic. But what if we could use real-time sleep apnea detection, Whisper v3, and Fast Fourier Transform (FFT) to turn your smartphone into a clinical-grade monitor?

In this tutorial, we are building a non-invasive sleep quality analyzer. By combining the physical precision of audio signal processing with the deep learning power of OpenAI's Whisper v3, we can filter out ambient noise (like a whirring fan) and focus specifically on the frequency signatures of snoring and obstructive sleep events.

The Architecture: Physics Meets AI 🏗️

The biggest challenge in audio-based health tech is "noise." A car driving by or a blanket rustling can look like a breathing event to a naive model. Our solution uses a dual-stage pipeline:

FFT (Fast Fourier Transform): Analyzes the frequency spectrum to identify the "texture" of the sound.
Whisper v3: Processes the temporal sequence to identify specific breathing patterns and distinguish between regular snoring and apnea events.

graph TD
    A[Raw Audio Input - React Native] --> B{FFmpeg Stream}
    B --> C[FFT Analysis - Librosa]
    C -->|High-Freq Noise| D[Filter Out]
    C -->|Low-Freq Snore Signature| E[Whisper v3 Encoder]
    E --> F[Pattern Recognition]
    F --> G[Apnea Event Detection]
    G --> H[React Native Dashboard]
    H --> I[Weekly Health Report]

Prerequisites 🛠️

To follow this build, you'll need:

Tech Stack: Whisper v3 (Large-v3 or Distil-Whisper for edge), Librosa (Python), FFmpeg, and React Native.
Environment: A Python backend (FastAPI/Flask) for the heavy lifting or a specialized ONNX runtime for true edge performance.

Step 1: Extracting the "Signature" with FFT

Before we talk to the AI, we need to see the sound. Snoring usually sits in the 20Hz to 2kHz range, with specific harmonic peaks. We use Librosa to perform a Short-Time Fourier Transform (STFT).

import librosa
import numpy as np

def analyze_snore_density(audio_path):
    # Load audio (sampled at 16kHz for Whisper compatibility)
    y, sr = librosa.load(audio_path, sr=16000)

    # Calculate Short-Time Fourier Transform
    stft = np.abs(librosa.stft(y))

    # Convert to decibels
    db_spec = librosa.amplitude_to_db(stft, ref=np.max)

    # Calculate Spectral Centroid to identify "heavy" sounds
    centroid = librosa.feature.spectral_centroid(y=y, sr=sr)

    # If the energy is concentrated in low frequencies, it's likely a snore/breath
    is_breathing_event = np.mean(centroid) < 1500 
    return is_breathing_event, db_spec

Step 2: Contextual Analysis with Whisper v3

Whisper isn't just for transcribing podcasts. Its encoder is incredibly robust at understanding audio context. By feeding the filtered audio segments into Whisper v3, we can classify the type of sound.

from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline

device = "cuda:0" # or "cpu"
model_id = "openai/whisper-large-v3"

# Initialize the pipeline
pipe = pipeline(
    "automatic-speech-recognition",
    model=model_id,
    device=device,
)

def classify_audio_segment(audio_data):
    # We use Whisper to "transcribe" the environment. 
    # In a specialized health model, we'd use the hidden states.
    # Here, we look for non-speech tokens and patterns.
    result = pipe(audio_data, return_timestamps=True)

    # Logic: If Whisper detects long silences followed by gasping sounds
    # (often transcribed as [breathing] or [gasping] tags), 
    # we flag a potential Apnea event.
    return result["text"]

Step 3: Bridging to the Edge with React Native

On the mobile side, we use react-native-ffmpeg to downsample the microphone input in real-time before sending it to our analysis engine.

import { FFmpegKit } from 'ffmpeg-kit-react-native';

const processAudioForAnalysis = async (inputPath) => {
  const outputPath = `${RNFS.CachesDirectoryPath}/processed_audio.wav`;

  // Convert to 16kHz, Mono, PCM 16-bit (Whisper's favorite format)
  await FFmpegKit.execute(`-i ${inputPath} -ar 16000 -ac 1 -c:a pcm_s16le ${outputPath}`);

  return outputPath;
};

The "Official" Way to Scale 🚀

Building a prototype is easy, but making it production-ready (handling multiple users, ensuring privacy, and optimizing latency) is where the real challenge lies.

If you are looking for advanced signal processing patterns, high-performance AI deployment strategies, or more production-ready examples of edge-computing, I highly recommend checking out the WellAlly Tech Blog. It's a goldmine for developers looking to bridge the gap between "it works on my machine" and "it works for a million users."

Conclusion: Data-Driven Sleep 💤

By combining Whisper v3 and FFT, we move away from simple "noise detection" toward "intelligent audio analysis." This setup allows users to track their health without wearing a single sensor.

Key Takeaways:

FFT acts as our first-line filter, saving computational power.
Whisper v3 provides the deep contextual understanding needed to differentiate a cough from a life-threatening apnea event.
Edge Computing ensures that sensitive bedroom audio never has to leave the device if configured correctly.

Are you ready to build the future of health-tech? Drop a comment below or share your results if you try this stack! 🥑💻

Calories from Pixels: Building a Precision Food Tracking Pipeline with GPT-4o Vision & SAM

wellallyTech — Mon, 04 May 2026 01:30:00 +0000

We’ve all been there: staring at a delicious plate of Beef Wellington or a complex Poke Bowl, wondering exactly how many calories are hiding behind those textures. Manual logging is a chore, and most AI calorie counters fail because they can't distinguish between the food and the plate—or worse, they miss the side of fries entirely. 🍟

In this guide, we are building a high-precision Computer Vision pipeline. By combining Meta's Segment Anything Model (SAM) for surgical object isolation and GPT-4o Vision for semantic understanding and volume estimation, we’re moving from "guessing" to "calculating." We will use FastAPI to glue it all together and PostgreSQL to persist our nutritional logs.

If you are looking to master Food Calorie Estimation using cutting-edge GPT-4o Vision and SAM workflows, you’re in the right place! 🥑

The Architecture 🏗️

The secret sauce here is preprocessing. Instead of feeding a messy, high-resolution photo directly to the LLM, we use SAM to generate masks. This tells the AI exactly what to look at, significantly improving the accuracy of volume and macro estimation.

graph TD
    A[User App / Image Upload] -->|POST /analyze| B(FastAPI Backend)
    B --> C{SAM Module}
    C -->|Identify Objects| D[Generate Bounding Boxes & Masks]
    D --> E[Crop & Process Segments]
    E --> F[GPT-4o Vision API]
    F -->|Reasoning: Type, Mass, Calories| G[Pydantic Validation]
    G --> H[(PostgreSQL Storage)]
    H --> I[Response: Caloric Breakdown]

Prerequisites

To follow along, you'll need:

Python 3.10+
OpenAI API Key (for GPT-4o)
Segment Anything (SAM) Weights (ViT-H or ViT-L)
FastAPI & SQLAlchemy

Step 1: Isolating Food with SAM

The Segment Anything Model (SAM) allows us to generate high-quality masks for any object in an image. By isolating the food items, we reduce "background noise" (like the table or napkins) that often confuses vision models.

import numpy as np
import torch
from segment_anything import sam_model_registry, SamPredictor

class FoodSegmenter:
    def __init__(self, checkpoint="sam_vit_h_4b8939.pth"):
        self.device = "cuda" if torch.cuda.is_available() else "cpu"
        self.sam = sam_model_registry["vit_h"](checkpoint=checkpoint)
        self.sam.to(device=self.device)
        self.predictor = SamPredictor(self.sam)

    def get_masks(self, image_array):
        self.predictor.set_image(image_array)
        # For simplicity, we use automatic mask generation or center-point prompting
        # Here we assume we've identified the main dish areas
        masks, scores, logits = self.predictor.predict(
            point_coords=np.array([[image_array.shape[1]//2, image_array.shape[0]//2]]),
            point_labels=np.array([1]),
            multimask_output=True,
        )
        return masks[np.argmax(scores)]

Step 2: The GPT-4o Vision Brain 🧠

Once we have our isolated food item, we send it to GPT-4o. We use a specific prompt designed to force the model to think about density and volume relative to standard objects (like the plate size).

Defining the Schema

Using Pydantic ensures our AI output is structured and ready for our database.

from pydantic import BaseModel
from typing import List

class FoodItem(BaseModel):
    name: str
    estimated_weight_g: float
    calories: int
    protein_g: float
    carbs_g: float
    fat_g: float
    confidence_score: float

class NutritionAnalysis(BaseModel):
    items: List[FoodItem]
    total_calories: int
    health_score: int

The API Call

import openai

async def analyze_food_vision(image_base64: str):
    response = await openai.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Analyze this food item. Estimate volume in cm3, then weight in grams based on density. Provide a JSON response for calories, protein, fat, and carbs."},
                    {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_base64}"}}
                ],
            }
        ],
        response_format={"type": "json_object"}
    )
    return response.choices[0].message.content

Advanced Patterns & Production Scaling 🚀

Building a prototype is easy, but making it production-ready is where the real challenge lies. You need to handle rate limiting, image compression, and model fallbacks.

For a deeper dive into Production AI Architectures and more robust implementations of multimodal pipelines, I highly recommend checking out the technical breakdowns at WellAlly Tech Blog. They cover advanced patterns for scaling FastAPI backends and optimizing LLM latency that are crucial for high-traffic health apps.

Step 3: Integrating the Pipeline with FastAPI

Now we wrap everything into a clean endpoint.

from fastapi import FastAPI, UploadFile, File
import cv2

app = FastAPI()

@app.post("/analyze-meal")
async def analyze_meal(file: UploadFile = File(...)):
    # 1. Load Image
    contents = await file.read()
    nparr = np.frombuffer(contents, np.uint8)
    img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)

    # 2. Run SAM (Optional Pre-processing)
    # segmenter = FoodSegmenter()
    # mask = segmenter.get_masks(img)

    # 3. Call GPT-4o Vision
    # (Assume image conversion to base64 here)
    analysis_result = await analyze_food_vision(encoded_image)

    # 4. Store in PostgreSQL
    # db.save(analysis_result)

    return {"status": "success", "data": analysis_result}

Conclusion: The Future of Visual Dietetics 🍎

By combining SAM's spatial awareness with GPT-4o's world knowledge, we've created a system that doesn't just "see" food—it understands it. This pipeline can be extended to recognize kitchen utensils for scale reference or even detect degrees of "doneness" to adjust caloric density.

Key Takeaways:

SAM is essential for precision; it prevents the LLM from hallucinating calories based on the tablecloth.
Structured Outputs (JSON mode) are non-negotiable for building real applications.
FastAPI provides the asynchronous speed needed for a smooth user experience.

Are you building something in the Vision AI space? Drop a comment below or share your results! And don't forget to visit WellAlly Tech for more advanced engineering guides. Happy coding! 💻🚀

Unifying Your Health Data: Building a High-Frequency ETL Pipeline for Apple HealthKit and Google Health Connect 🏃‍♂️📊

wellallyTech — Sun, 03 May 2026 01:25:00 +0000

Ever wondered why your Apple Watch says you're a marathon runner while Google Fit thinks you're a couch potato? Building a Quantified Self Data Lake is the ultimate dream for data nerds, but syncing Apple HealthKit and Google Health Connect involves more than just a simple API call. Between mismatched sampling frequencies and the eternal nightmare of timezone conversions, creating a reliable high-frequency ETL pipeline is a true test of your Data Engineering mettle.

In this tutorial, we are going to architect a robust system that pulls raw telemetry from mobile SDKs, processes it through Apache Hop, and stores it in a structured PostgreSQL data lake. We'll solve the "heterogeneous data" problem and ensure your heart rate variability (HRV) and step counts are synchronized with millisecond precision. 🚀

The Architecture: From Pulse to Postgres

To handle the high-frequency nature of health data (which can generate thousands of rows per hour), we need a decoupled architecture. We use a Python-based middleware to bridge the mobile SDKs and an orchestration layer to handle the heavy lifting.

graph TD
    A[Apple HealthKit SDK] -->|JSON Stream| B(Python FastAPI Middleware)
    C[Google Health Connect] -->|Batch Export| B
    B -->|Raw Storage| D[(PostgreSQL Staging)]

    subgraph ETL Orchestration
    E[Apache Hop] -->|Extract & Normalize| D
    E -->|Transform Timezones| F{Data Validator}
    F -->|Load| G[(Quantified Self Data Lake)]
    end

    G --> H[Grafana / Superset Visualization]

Prerequisites 🛠️

Before we dive into the code, ensure you have the following in your toolkit:

Python 3.10+: For our middleware and data validation.
PostgreSQL: Our destination data lake.
Apache Hop: The successor to Kettle/PDI for visual ETL orchestration.
HealthKit/Health Connect SDKs: Configured in your mobile project.

Step 1: Handling Heterogeneous Data with Pydantic

The biggest hurdle is that Apple and Google represent data differently. Apple uses "Samples," while Google often aggregates into "Records." We'll use Pydantic to enforce a unified schema before the data even hits our staging area.

from pydantic import BaseModel, Field
from datetime import datetime
from typing import Optional, Union

class HealthMetric(BaseModel):
    source_device: str # 'apple_watch' or 'pixel_watch'
    metric_type: str   # 'step_count', 'heart_rate', 'active_calories'
    value: float
    unit: str
    start_time: datetime
    end_time: datetime
    timezone: str = "UTC"
    metadata: Optional[dict] = None

# Example: Validating a high-frequency Heart Rate sample
raw_data = {
    "source_device": "apple_watch_s8",
    "metric_type": "heart_rate",
    "value": 72.5,
    "unit": "count/min",
    "start_time": "2023-10-27T10:00:00Z",
    "end_time": "2023-10-27T10:00:00Z",
    "timezone": "America/New_York"
}

metric = HealthMetric(**raw_data)
print(f"Validated: {metric.metric_type} at {metric.start_time}")

Step 2: Designing the Data Lake Schema

We need a schema that supports "Upsert" operations. Why? Because mobile devices often resync old data when they reconnect to Wi-Fi. We don't want duplicate steps (as much as we'd like the extra credit! 😅).

CREATE TABLE health_metrics_raw (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    external_id TEXT UNIQUE, -- Hash of (source + metric + start_time)
    source_device VARCHAR(50),
    metric_type VARCHAR(50),
    value NUMERIC,
    unit VARCHAR(20),
    ts_start TIMESTAMP WITH TIME ZONE,
    ts_end TIMESTAMP WITH TIME ZONE,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX idx_metric_time ON health_metrics_raw (metric_type, ts_start);

Step 3: Orchestration with Apache Hop

While Python is great for ingestion, Apache Hop shines in complex ETL logic. We create a pipeline that:

Reads from the PostgreSQL staging table.
Standardizes Units: Converts everything to SI units (e.g., all energy to Kilocalories).
Timezone Normalization: Uses the timezone field to convert all ts_start to a localized user timeline and a universal UTC timeline.
Deduplication: Uses a "Unique Rows" transform based on the external_id.

Pro Tip: For advanced data engineering patterns and more production-ready examples of high-throughput pipelines, check out the deep-dive articles at WellAlly Tech Blog. They have fantastic resources on scaling PostgreSQL for time-series data! 🥑

Step 4: The Timezone Trap 🕰️

When you fly from New York to Tokyo, your "daily steps" become a mess. If you store data in UTC, your 10k steps might appear split across two days.

Solution: Always store the Offset and the Local Time.
In your ETL logic, create a generated column:

ALTER TABLE health_metrics_raw 
ADD COLUMN local_day DATE GENERATED ALWAYS AS ( (ts_start AT TIME ZONE timezone)::DATE ) STORED;

This allows you to query "Steps per day" based on where you physically were at that moment.

Conclusion: Data-Driven Wellness

By building this pipeline, you've moved from "guessing" your health to "engineering" your wellness. You now have a unified, deduplicated, and timezone-aware data lake ready for Analysis or even training your own ML models.

What's next?

Connect Grafana to your Postgres instance for real-time dashboards.
Implement Anomaly Detection to alert you when your resting heart rate spikes.

Did you run into issues with the HealthKit background delivery? Or maybe Google's OAuth 2.0 flow is giving you a headache? Let’s chat in the comments below! 👇

If you enjoyed this build, don't forget to follow for more "Learning in Public" tutorials and visit wellally.tech/blog for the full source code of this pipeline! 🚀💻

Beyond Simple Image Recognition: Building a Precise AI Nutritionist with GPT-4o and Segment Anything (SAM)

wellallyTech — Sat, 02 May 2026 01:20:00 +0000

We've all been there: you take a photo of your lunch with a generic calorie-tracking app, and it tells you your 500-gram lasagna is a "medium slice of cake." 🤦‍♂️ The struggle with AI nutrition tracking isn't just identifying the food; it's the spatial awareness—understanding volume, portion size, and the hidden ingredients in complex dishes.

In this tutorial, we are leveling up. We are building a sophisticated Visual RAG (Retrieval-Augmented Generation) pipeline. By combining the semantic power of GPT-4o Vision with the surgical precision of Meta's Segment Anything Model (SAM), we can isolate individual ingredients and cross-reference them with a nutritional database to provide professional-grade calorie and macronutrient auditing. If you are looking for production-ready patterns for AI vision systems, be sure to check out the deep dives over at WellAlly Tech Blog, where we explore high-performance AI architectures.

🏗️ The Architecture: Precision Vision Pipeline

Standard vision models often treat an image as a single "bag of pixels." Our pipeline treats it as a structured scene. We use SAM to generate precise masks, calculate the relative area of food items, and then feed those high-context crops to GPT-4o for final reasoning.

graph TD
    A[User Uploads Meal Photo] --> B{SAM Engine}
    B -->|Segment| C[Isolated Food Masks]
    B -->|Calculate| D[Relative Volume/Area]
    C --> E[GPT-4o Vision Analysis]
    D --> E
    E --> F[Semantic Food Tags]
    F --> G[PostgreSQL Nutrition DB]
    G --> H[Final Nutrient Report]
    H --> I[User Feedback Loop]

🛠️ The Tech Stack

GPT-4o: Our "Reasoning Engine" for identifying complex food types and textures.
SAM (Segment Anything Model): To precisely delineate where one food item ends and another begins.
FastAPI: For the high-performance asynchronous API layer.
PostgreSQL: Storing our ground-truth nutritional data for RAG.

👨‍💻 Step 1: Defining the Structured Output

To ensure our pipeline is reliable, we need GPT-4o to return structured data. We’ll use Pydantic to define what a "Meal Analysis" looks like.

from pydantic import BaseModel, Field
from typing import List

class FoodItem(BaseModel):
    name: str = Field(..., description="Name of the food item")
    estimated_weight_grams: float = Field(..., description="Estimated weight based on volume")
    confidence: float = Field(..., ge=0, le=1)
    ingredients: List[str]

class MealReport(BaseModel):
    items: List[FoodItem]
    total_calories: int
    macros: dict = Field(default_factory=lambda: {"protein": 0, "carbs": 0, "fat": 0})

🧠 Step 2: The SAM + GPT-4o Synergy

The magic happens when we don't just send a raw photo. We send the photo plus the coordinates/masks generated by SAM. This helps GPT-4o "focus" its attention on specific regions.

import openai
from fastapi import FastAPI, UploadFile

app = FastAPI()

@app.post("/analyze-meal")
async def analyze_meal(file: UploadFile):
    # 1. Process image with SAM (Pseudo-code for the segmentation step)
    # masks, scores = sam_model.predict(image)

    # 2. Extract metadata and prepare for GPT-4o
    image_bytes = await file.read()

    response = openai.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": "You are a professional nutritionist. Analyze the image and segmented areas to provide a precise nutrient breakdown."
            },
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Analyze this meal. Note that I have segmented the main protein from the side carbs."},
                    {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{encode_image(image_bytes)}"}}
                ]
            }
        ],
        response_format={"type": "json_object"}
    )

    return response.choices[0].message.content

🥗 Step 3: Improving Accuracy with Visual RAG

The hardest part of nutrition AI is "hallucination." GPT-4o might think a sauce is tomato-based when it's actually a high-calorie chili oil. By implementing a Visual RAG pattern, we take the labels identified by GPT-4o and query our PostgreSQL database for verified nutritional profiles.

For even more advanced implementations of RAG in multimodal environments, I highly recommend checking out the technical guides at wellally.tech/blog. They cover how to optimize vector embeddings for visual features, which is a game-changer for this specific use case. 🥑

The SQL Query Strategy

-- Querying verified nutrients based on AI tags
SELECT name, calories_per_100g, protein, carbs, fat 
FROM nutrition_db 
WHERE food_tag % ANY(ARRAY['grilled_chicken', 'quinoa', 'broccoli'])
ORDER BY similarity DESC;

🚀 Conclusion: The Future of Precision Health

By combining Segment Anything (SAM) and GPT-4o, we move from "guessing" to "calculating." This pipeline allows for:

Overlapping Food Detection: Distinguishing between the rice and the curry on top.
Volume Estimation: Using mask areas as a proxy for portion size.
Auditability: Users can see exactly which parts of the image were identified as which food.

Building these types of Computer Vision Calorie Estimation tools is just the beginning. As multimodal models become faster and more efficient, we will see these pipelines moving directly to edge devices.

What's next?

Try integrating a depth-sensing camera (LiDAR) for 100% accurate volume calculation.
Add a feedback loop where the user can correct the AI to fine-tune the local embeddings.

If you enjoyed this tutorial, drop a comment below and let me know what you're building! And don't forget to visit WellAlly Tech for more cutting-edge AI development content. Happy coding! 💻🔥

The HRV Engineer: Building a Machine Learning Fatigue Warning System with Scikit-learn

wellallyTech — Fri, 01 May 2026 01:20:00 +0000

Have you ever pushed through a workout only to feel absolutely wrecked the next day? 😵‍💫 That’s often because we ignore our Central Nervous System (CNS). While muscles might feel fine, your nervous system speaks a different language: Heart Rate Variability (HRV).

In this tutorial, we are going to build a personalized Exercise Fatigue Early Warning Engine. We’ll use the Garmin Connect API to fetch data, Pandas for feature engineering, and a Random Forest model via Scikit-learn to predict whether you're ready to smash a PR or if you desperately need a rest day. 🚀

By the end of this, you'll understand how to transform raw wearable data into actionable health insights using predictive recovery algorithms and machine learning for fitness.

🏗 The Architecture

Before we dive into the code, let's look at the data flow. We are moving from raw time-series sensor data to a categorical "Recovery Recommendation."

graph TD
    A[Garmin Wearable] -->|Sync| B(Garmin Connect API)
    B -->|JSON Data| C[Data Preprocessing - Pandas]
    C -->|Feature Extraction: RMSSD, SDNN| D{Random Forest Model}
    D -->|Prediction| E[Fatigue Level: Low/Med/High]
    E -->|UI| F[Streamlit Dashboard]
    F -->|Recommendation| G[Rest vs. Train]

🛠 Prerequisites

To follow along, you'll need the following tech stack:

Python 3.9+
Scikit-learn: For the Random Forest Classifier.
Garmin Connect API: (We'll use a wrapper like garminconnect).
Pandas: For time-series manipulation.
Streamlit: For our shiny frontend.

Step 1: Fetching HRV Data

HRV isn't just one number; it's the variation in time between each heartbeat. Specifically, we look for RMSSD (Root Mean Square of Successive Differences).

import pandas as pd
from garminconnect import Garmin

# Initialize Garmin Client
client = Garmin("your_email", "your_password")
client.login()

def get_hrv_data(date):
    # Fetching HRV data for a specific date
    hrv_data = client.get_hrv_data(date.isoformat())
    df = pd.DataFrame(hrv_data['hrvReadings'])
    return df

Step 2: Feature Engineering with Pandas

Machine learning models thrive on good features. For fatigue detection, we don't just want today's HRV; we want the baseline (the 7-day moving average).

def preprocess_features(df):
    # Calculating key metrics
    # RMSSD: Short term recovery indicator
    # SDNN: Overall stress indicator

    df['rolling_avg_7d'] = df['rmssd'].transform(lambda x: x.rolling(window=7).mean())
    df['hrv_drop_ratio'] = (df['rmssd'] / df['rolling_avg_7d'])

    # Labeling (For training purposes, we assume specific thresholds)
    # 1: High Fatigue, 0: Ready to Train
    df['fatigue_label'] = df['hrv_drop_ratio'].apply(lambda x: 1 if x < 0.85 else 0)

    return df.dropna()

Step 3: Training the Random Forest Engine

Why Random Forest? It’s excellent for handling non-linear relationships and is robust against outliers (which happen often with wearable sensors!).

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

# Assume 'data' is our processed DataFrame
X = data[['rmssd', 'rolling_avg_7d', 'hrv_drop_ratio']]
y = data['fatigue_label']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Initialize the engine
model = RandomForestClassifier(n_estimators=100, max_depth=5)
model.fit(X_train, y_train)

# Evaluate
y_pred = model.predict(X_test)
print(classification_report(y_test, y_pred))

🥑 The "Official" Way to Build Health Apps

While this project is a great "Learning in Public" experiment, building production-grade health tech involves complex data privacy (HIPAA/GDPR) and more sophisticated signal processing.

For advanced patterns on integrating multi-modal health data and deploying robust AI models in the cloud, I highly recommend checking out the Wellally Tech Blog. It's a goldmine for developers looking to scale their health-tech stack beyond a local script.

Step 4: The Streamlit Dashboard

Finally, let's wrap this in a user-friendly interface so you don't have to look at a terminal every morning.

import streamlit as st

st.title("🛡️ HRV Fatigue Guard")

user_rmssd = st.number_input("Enter Today's RMSSD:", value=65)
user_baseline = st.number_input("Enter 7-Day Baseline:", value=72)

if st.button("Analyze Recovery"):
    ratio = user_rmssd / user_baseline
    prediction = model.predict([[user_rmssd, user_baseline, ratio]])

    if prediction[0] == 1:
        st.error("🚨 WARNING: High Neural Fatigue Detected. Suggestion: Active Recovery or Rest.")
    else:
        st.success("✅ SYSTEM READY: Nervous system recovered. Go for that PR!")

Conclusion

Building an HRV-based fatigue engine is a perfect way to combine your passion for fitness with data science. By moving from "I feel tired" to "My RMSSD is 15% below baseline," you're using biometric data to train smarter, not harder.

What's next?

Try adding sleep scores from the Garmin API as a feature.
Experiment with XGBoost to see if you can beat the Random Forest's accuracy.
Check out the Wellally Tech Blog for more production-ready examples of wearable integrations!

Drop a comment below if you've tried building something similar or if you have questions about the Garmin API! 🏃‍♂️💨

From Messy Blood Tests to Actionable Insights: Building Your Personal Health Data Warehouse with dbt & DuckDB 🩸💻

wellallyTech — Thu, 30 Apr 2026 01:10:00 +0000

We’ve all been there: a stack of PDF medical reports from the last five years, each with different units, varying reference ranges, and cryptic labels like "hS-CRP" or "HbA1c." Trying to track your health trends over time manually is a nightmare.

In this tutorial, we are going to fix that. We’ll dive into biomarker data cleaning, dbt health modeling, and DuckDB OLAP performance to turn those scattered data points into a high-performance personal health data warehouse. If you’ve been looking for a real-world project to master data engineering health records, you’re in the right place.

Pro Tip: If you're looking for more production-ready patterns for health data engineering or advanced analytics architectures, definitely check out the deep dives over at WellAlly Blog.

The Architecture: From Raw Pixels to Analytics

To handle years of biomarkers, we need a robust ELT (Extract, Load, Transform) pipeline. We’ll use Python for the initial extraction, DuckDB as our lightning-fast analytical engine, and dbt (data build tool) to handle the heavy lifting of data modeling.

graph TD
    A[Raw Health Data: CSV/JSON/PDF] -->|Python Extraction| B[(DuckDB Raw Layer)]
    B --> C{dbt Transformation}
    C --> D[stg_biomarkers: Cleaned & Cast]
    C --> E[int_health_metrics: Standardized Units]
    C --> F[fct_biomarker_trends: Final Analytics View]
    F --> G[Apache Superset Visualization]

    style B fill:#fff,stroke:#333,stroke-width:2px
    style F fill:#f9f,stroke:#333,stroke-width:2px

Prerequisites

Before we start, ensure you have the following installed:

Python 3.9+
dbt-duckdb (pip install dbt-duckdb)
DuckDB

Step 1: Ingesting the "Mess" with Python

First, we need to get our raw data into DuckDB. For this example, let's assume you've converted your PDFs into a structured CSV format.

import duckdb
import pandas as pd

# Connect to (or create) our local health warehouse
con = duckdb.connect('health_warehouse.duckdb')

# Load our messy CSV
# Assume columns: test_date, marker_name, value, unit, lab_source
raw_data = pd.read_csv('my_blood_work_2018_2023.csv')

# Register it as a table in DuckDB
con.execute("CREATE TABLE IF NOT EXISTS raw_health_data AS SELECT * FROM raw_data")

print(f"✅ Ingested {len(raw_data)} biomarker records into DuckDB!")

Step 2: The dbt Setup

Now for the magic. dbt allows us to write modular SQL to clean this data. First, configure your profiles.yml to point to your DuckDB file:

# ~/.dbt/profiles.yml
health_dw:
  target: dev
  outputs:
    dev:
      type: duckdb
      path: health_warehouse.duckdb
      extensions:
        - icu
        - parquet

Step 3: Standardizing Biomarkers (The Meat)

Biomarkers are tricky because labs use different units (e.g., mg/dL vs. mmol/L). We need a "Standardization Layer."

In models/staging/stg_biomarkers.sql:

-- Clean up strings and dates
WITH raw_source AS (
    SELECT * FROM {{ source('raw', 'raw_health_data') }}
)

SELECT
    strptime(test_date, '%Y-%m-%d')::DATE as report_date,
    upper(trim(marker_name)) as marker_key,
    CAST(value AS DOUBLE) as original_value,
    trim(unit) as original_unit,
    lab_source
FROM raw_source
WHERE value IS NOT NULL

Next, in models/intermediate/int_biomarkers_standardized.sql, we handle the unit conversions. This is where you'd typically look up a reference table, but we can use a CASE statement for simplicity:

-- Standardizing everything to common units (e.g., mg/dL for Glucose)
SELECT
    report_date,
    marker_key,
    CASE 
        -- Conversion logic: mmol/L to mg/dL for Glucose (x18)
        WHEN marker_key = 'GLUCOSE' AND original_unit = 'mmol/L' THEN original_value * 18
        ELSE original_value
    END as standardized_value,
    CASE 
        WHEN marker_key = 'GLUCOSE' THEN 'mg/dL'
        ELSE original_unit
    END as standardized_unit
FROM {{ ref('stg_biomarkers') }}

Step 4: The "Official" Way to Scale

While this DIY setup is great for personal use, scaling this to clinical-grade data requires handling HL7 FHIR standards and complex HIPAA-compliant schemas.

If you're interested in how the pros handle large-scale biomedical data ingestion and advanced health-tech architectures, I highly recommend checking out the technical guides at WellAlly.tech/blog. They cover everything from AI-driven diagnostics pipelines to high-availability health data warehouses. 🚀

Step 5: OLAP Analysis with DuckDB

Once you run dbt run, your health_warehouse.duckdb is ready for high-speed analysis. You can now run window functions to see your health velocity!

-- Calculate 3-report moving average for Cholesterol
SELECT 
    report_date,
    standardized_value as ldl_cholesterol,
    AVG(standardized_value) OVER (
        ORDER BY report_date 
        ROWS BETWEEN 2 PRECEDING AND CURRENT ROW
    ) as ldl_rolling_avg
FROM {{ ref('int_biomarkers_standardized') }}
WHERE marker_key = 'LDL_CHOLESTEROL'
ORDER BY report_date DESC;

Conclusion: Data-Driven Longevity

By leveraging dbt and DuckDB, we've turned a pile of confusing medical reports into a structured, queryable data warehouse. You can now plug in Apache Superset or Streamlit to visualize your markers over time, spotting trends before they become health issues. 🥑

What's next?

Automation: Set up a GitHub Action to run your dbt models whenever you drop a new CSV into your repo.
Correlation: Join your biomarker data with your Apple Health or Oura Ring exports (Parquet files) to see how sleep affects your fasting glucose.

Are you building something in the HealthTech space? Drop a comment below or share your repo—I'd love to see how you're modeling your data! 👇

Stop Refreshing! Build an Autonomous AI Agent to Book Your Dentist Appointments with Playwright and GPT-4o

wellallyTech — Wed, 29 Apr 2026 01:15:00 +0000

We’ve all been there: staring at a clunky dentist reservation portal, frantically hitting F5 to find an open slot that doesn't clash with your 10 AM stand-up meeting. It’s tedious, manual, and frankly, a job for a machine. 🤖

In this guide, we are building an AI Agent that masters the art of automated scheduling. By combining Playwright automation, LLM browser control, and the Google Calendar API, we will create a system that navigates complex websites, understands doctor availability, and syncs perfectly with your life. This is "Learning in Public" at its finest—turning a personal headache into a streamlined technical solution. 🚀

Why Traditional Automation Fails

Standard scraping scripts break the moment a UI changes. However, by using OpenAI Functions (Tool Calling), we give our agent the "eyes" to understand the schedule and the "brain" to make decisions based on your real-time availability.

The Architecture 🏗️

Our agent follows a "Sense-Think-Act" loop. It scrapes the portal, compares dates with your calendar, and executes the click-stream required to book the appointment.

graph TD
    A[Start: User Request] --> B[Playwright: Scrape Booking Page]
    B --> C[LLM: Parse HTML into Structured JSON]
    C --> D[Google Calendar API: Fetch Busy Slots]
    D --> E[LLM: Identify Best Slot]
    E --> F{Slot Found?}
    F -- Yes --> G[Playwright: Execute Booking Form]
    F -- No --> H[Wait & Retry Later]
    G --> I[Notify User via Slack/Email]

Prerequisites 🛠️

Before we dive into the code, ensure you have the following:

Node.js (v18+)
Playwright: For browser orchestration.
OpenAI SDK: Specifically using gpt-4o for vision and reasoning.
Google Calendar API Credentials: A service account or OAuth2 token.

Step 1: Scraping the Schedule with Playwright

First, we need to get the raw data from the dentist's portal. Playwright is perfect for this because it handles modern SPAs (Single Page Applications) with ease.

const { chromium } = require('playwright');

async function getScheduleHTML(url) {
  const browser = await chromium.launch({ headless: true });
  const page = await browser.newPage();

  await page.goto(url);
  // Wait for the calendar component to load
  await page.waitForSelector('.appointment-slot-container');

  // Extract the inner HTML of the schedule section
  const scheduleContent = await page.innerHTML('.booking-grid');

  await browser.close();
  return scheduleContent;
}

Step 2: Intelligent Parsing with OpenAI Functions

HTML is messy. We don't want to write regex for every different dentist's site. Instead, we pass the HTML to OpenAI and ask it to extract the slots into a clean JSON format using Tool Calling.

const { OpenAI } = require('openai');
const openai = new OpenAI();

async function parseSlotsWithAI(htmlContent) {
  const response = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [
      { role: "system", content: "You are a scheduling assistant. Extract available dates and times from the HTML." },
      { role: "user", content: htmlContent }
    ],
    tools: [{
      type: "function",
      function: {
        name: "format_slots",
        description: "Format the available dentist slots",
        parameters: {
          type: "object",
          properties: {
            available_slots: {
              type: "array",
              items: {
                type: "object",
                properties: {
                  date: { type: "string" },
                  time: { type: "string" },
                  doctor: { type: "string" }
                }
              }
            }
          }
        }
      }
    }],
    tool_choice: { type: "function", function: { name: "format_slots" } }
  });

  return JSON.parse(response.choices[0].message.tool_calls[0].function.arguments);
}

Step 3: The "Official" Integration Logic 🥑

Matching the dentist's availability with your own is the secret sauce. While we are building a custom script here, there are more robust ways to handle enterprise-level agentic workflows.

Pro Tip: For deep dives into production-ready agent patterns and advanced LLM orchestration, I highly recommend checking out the technical deep-dives at WellAlly Blog. They cover how to scale these "AI workers" beyond simple scripts into full-blown autonomous systems.

Step 4: Comparing with Google Calendar

Now, we fetch your "busy" times. If the dentist has a slot at 2:00 PM but you have a meeting, the agent should automatically skip it.

const { google } = require('googleapis');

async function getMyFreeSlots(auth) {
  const calendar = google.calendar({ version: 'v3', auth });
  const res = await calendar.events.list({
    calendarId: 'primary',
    timeMin: new Date().toISOString(),
    maxResults: 10,
    singleEvents: true,
    orderBy: 'startTime',
  });
  return res.data.items; // Simplified: logic to find gaps goes here
}

Step 5: Final Execution (The "Book" Button)

Once the LLM finds the perfect match (e.g., Tuesday at 9:00 AM, and you are free!), we trigger Playwright one last time to fill the form and click "Confirm."

async function bookAppointment(slot) {
  const browser = await chromium.launch({ headless: false }); // Headless false so we can watch the magic!
  const page = await browser.newPage();
  await page.goto(BOOKING_URL);

  // AI-driven interaction: find the slot based on the text
  await page.click(`text="${slot.time}"`);
  await page.fill('#patient-name', 'John Doe');
  await page.fill('#patient-phone', '555-0199');

  // Final click
  // await page.click('#confirm-booking');
  console.log(`✅ Successfully booked for ${slot.date} at ${slot.time}`);
}

Conclusion: The Power of Browser Agents 🌟

By combining Playwright with LLMs, we’ve moved past brittle "selector-based" scraping into the era of Semantic Automation. Our agent doesn't just "click buttons"—it understands intent, respects your personal schedule, and handles data gracefully.

What’s next? You could expand this to:

Slack Notifications: Get a ping when a booking is confirmed.
Multi-Doctor Search: Scrape 5 different clinics at once.
Vision Mode: Use GPT-4o's vision capabilities to solve those pesky "select all the traffic lights" CAPTCHAs.

What are you planning to automate next? Let me know in the comments! 👇

For more advanced tutorials on AI Agents and automation architecture, don't forget to visit wellally.tech/blog. 💻✨

From Pixels to Prescriptions: Building an Automated Medication Reminder with YOLOv8 and OCR

wellallyTech — Tue, 28 Apr 2026 01:00:00 +0000

Forgetfulness is a human trait, but when it comes to medication, a missed dose can be serious. 💊 With the rise of Computer Vision and Edge AI, we can now build smart systems that watch over our health—literally.

In this tutorial, we are going to build a fully automated Medication Reminder System. By leveraging YOLOv8 for object detection and Tesseract OCR for text extraction, we can monitor a pill box via a camera (like a Raspberry Pi) and trigger alerts if the medication hasn't been moved or taken at the scheduled time. This project combines real-time object detection, Internet of Things (IoT), and Image Processing into one practical, life-saving application.

The Architecture 🏗️

The logic flow is straightforward: we capture video frames, identify the pill box, read the label to identify the medication, and use a state-machine to determine if the user has interacted with the box.

graph TD
    A[Camera Feed] --> B{OpenCV Frame Processing}
    B --> C[YOLOv8: Detect Pill Box]
    C --> D[ROI Extraction]
    D --> E[Tesseract OCR: Read Label]
    E --> F{Logic Engine}
    F -- "No movement detected" --> G[MQTT: Trigger Voice Alert]
    F -- "Box moved/Empty" --> H[Log Success]
    G --> I[Mobile Notification]

Prerequisites 🛠️

To get started, make sure you have the following in your tech_stack:

Python 3.8+
YOLOv8 (via the ultralytics package)
Tesseract OCR (installed on your OS)
OpenCV: For image manipulation
MQTT: For lightweight messaging between your camera and the alert system

pip install ultralytics opencv-python pytesseract paho-mqtt

Step 1: Detecting the Pill Box with YOLOv8

First, we need to locate the pill box in the camera's field of view. YOLOv8 is incredibly fast, making it perfect for edge devices.

from ultralytics import YOLO
import cv2

# Load a pre-trained Nano model (lightweight for Raspberry Pi)
model = YOLO('yolov8n.pt') 

def detect_pill_box(frame):
    # We are looking for 'bottle' or 'box' classes in COCO dataset
    results = model(frame, conf=0.5)

    for r in results:
        for box in r.boxes:
            # Class 39 is 'bottle' in COCO, or use a custom trained model
            if int(box.cls) == 39: 
                x1, y1, x2, y2 = map(int, box.xyxy[0])
                return frame[y1:y2, x1:x2] # Return the cropped Region of Interest (ROI)
    return None

Step 2: Reading the Label with Tesseract OCR

Once we have the ROI (the cropped image of the pill box), we need to know what medication it is. This is where Tesseract OCR shines. 🔍

import pytesseract

def get_medication_name(roi):
    # Pre-processing for better OCR: Grayscale and Thresholding
    gray = cv2.cvtColor(roi, cv2.COLOR_BGR2GRAY)
    gray = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY | cv2.THRESH_OTSU)[1]

    # Extract text
    text = pytesseract.image_to_string(gray)
    return text.strip()

Step 3: The Logic Engine and MQTT Alerts

We don't want to nag the user every second. We only want to trigger an alert if it's 9:00 AM and the pill box hasn't moved from its "Resting Zone."

import paho.mqtt.client as mqtt
import time

mqtt_client = mqtt.Client()
mqtt_client.connect("broker.hivemq.com", 1883, 60)

def trigger_alert(med_name):
    message = f"Reminder: It's time to take your {med_name}!"
    mqtt_client.publish("home/medication/reminder", message)
    print(f"🚀 Alert Sent: {message}")

# Main Loop Simulation
while True:
    ret, frame = cap.read()
    pill_box_roi = detect_pill_box(frame)

    if pill_box_roi is not None:
        name = get_medication_name(pill_box_roi)
        # Add logic here to check current time vs medication schedule
        # If time matches and movement is zero -> trigger_alert(name)

    time.sleep(10) # Check every 10 seconds

The "Official" Way to Scale 🥑

While this DIY setup is a great start, building production-ready health-tech requires more robust handling of lighting conditions, multiple medication schedules, and privacy-first data processing.

For advanced patterns on deploying Edge AI models and building highly reliable Vision Systems, I highly recommend checking out the deep-dive articles at WellAlly Tech Blog. They cover production-ready examples of how to integrate multimodal AI with real-world hardware.

Conclusion 🚀

Building a vision-based reminder system is a fantastic "Learning in Public" project. It touches on AI, hardware, and real-world problem-solving. By combining YOLOv8's speed with Tesseract's accessibility, you can create a tool that actually makes a difference.

What's next?

Try training a custom YOLOv8 model on your specific pill boxes for 99% accuracy.
Integrate a "Success" sound when the system detects the box being lifted.

Let me know in the comments: How are you using AI to improve your daily routine? 👇

Building the "Digital Consultation Room": Orchestrating Multi-Agent Systems for Chronic Disease Management with LangGraph

wellallyTech — Mon, 27 Apr 2026 01:00:00 +0000

Managing a chronic condition like Type 2 Diabetes isn't just a medical challenge; it's a data orchestration nightmare. Patients have to balance glucose levels, carbohydrate intake, and physical activity in a delicate dance. Traditionally, this requires a multidisciplinary team, but what if we could replicate that expert "colloquium" using Multi-Agent Systems? 🩺💻

In this tutorial, we are diving deep into LangGraph, LLM orchestration, and stateful AI agents to build a Chronic Disease Management System. By the end of this post, you'll understand how to coordinate a "Diet Expert," an "Exercise Expert," and a "Glucose Monitor" into a cohesive, automated health squad. We'll be leveraging GPT-4 for the brains and Redis for persistent state management to ensure our agents never lose the patient's context.

The Architecture: Multi-Agent Choreography

Unlike a simple linear chain, chronic disease management requires a "cyclic" approach where agents can challenge and refine each other's suggestions. We use LangGraph to define a state machine where the state (patient health data) is passed and mutated by specialized nodes.

graph TD
    A[Start: Patient Data Input] --> B{Glucose Monitor}
    B -->|High Risk| C[Diet Expert]
    B -->|Normal/Low| D[Exercise Expert]
    C --> E{Consensus Check}
    D --> E
    E -->|Conflict| B
    E -->|Balanced Plan| F[Final Health Report]
    F --> G[End]

Why LangGraph?

Standard DAGs (Directed Acyclic Graphs) fail when you need agents to "talk back" to each other. LangGraph allows us to create loops and maintain a persistent State object, which is critical when a diet plan needs to be adjusted based on an intense exercise recommendation.

Prerequisites

Before we code, ensure you have the following:

Tech Stack: Python 3.10+, OpenAI API Key, Docker (for Redis).
Libraries: langgraph, langchain_openai, redis.

Step 1: Defining the State and Schema

First, we define what our "shared memory" looks like. In LangGraph, the TypedDict serves as the schema for the information passed between agents.

from typing import Annotated, List, TypedDict
from langchain_core.messages import BaseMessage
import operator

class AgentState(TypedDict):
    # The 'messages' key accumulates all communication history
    messages: Annotated[List[BaseMessage], operator.add]
    patient_id: str
    current_glucose: float
    is_balanced: bool

Step 2: Implementing the Experts (Nodes)

Each agent is a specialized prompt wrapper. For instance, our Diet Expert focuses solely on glycemic index (GI) and portion control.

from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage, SystemMessage

llm = ChatOpenAI(model="gpt-4-turbo", temperature=0)

def diet_expert_node(state: AgentState):
    print("--- DIET EXPERT EVALUATING ---")
    last_message = state['messages'][-1]

    prompt = SystemMessage(content=(
        "You are a Clinical Dietitian specializing in Type 2 Diabetes. "
        "Based on the glucose levels and exercise plan, suggest a specific meal plan."
    ))

    response = llm.invoke([prompt] + state['messages'])
    return {"messages": [response]}

def exercise_expert_node(state: AgentState):
    print("--- EXERCISE EXPERT EVALUATING ---")
    prompt = SystemMessage(content=(
        "You are a Kinesiologist. Adjust the exercise intensity based on "
        "the patient's current glucose and diet plan to prevent hypoglycemia."
    ))
    response = llm.invoke([prompt] + state['messages'])
    return {"messages": [response]}

Step 3: Persistence with Redis

In production, agents can't forget who the patient is if the server restarts. We use Redis to store the checkpointer state. This ensures that the "Digital Consultation" can span across multiple days.

For more production-ready patterns regarding persistent agentic memory and advanced RAG integration, I highly recommend checking out the deep-dives at WellAlly Blog. It’s a fantastic resource for scaling these types of AI architectures.

# docker-compose.yml
services:
  redis:
    image: redis:latest
    ports:
      - "6379:6379"

Step 4: Building the Graph

Now we wire it all together. We define the flow, including conditional edges that decide whether to continue the discussion or finalize the report.

from langgraph.graph import StateGraph, END

workflow = StateGraph(AgentState)

# Add our Nodes
workflow.add_node("monitor", glucose_monitor_node)
workflow.add_node("dietitian", diet_expert_node)
workflow.add_node("kinesiologist", exercise_expert_node)

# Define the Flow
workflow.set_entry_point("monitor")

# Routing Logic
def should_continue(state: AgentState):
    if state["is_balanced"]:
        return "end"
    return "dietitian"

workflow.add_conditional_edges(
    "monitor",
    should_continue,
    {
        "end": END,
        "dietitian": "dietitian"
    }
)

workflow.add_edge("dietitian", "kinesiologist")
workflow.add_edge("kinesiologist", "monitor") # Loop back for verification!

app = workflow.compile()

The "Official" Way to Scale

While this local setup works for a prototype, scaling a Multi-Agent system to thousands of patients requires robust observability and safety rails. For instance, how do you handle "hallucinations" in a medical context?

If you are looking for advanced patterns, such as implementing "Human-in-the-loop" for medical verification or deploying these agents as microservices, the WellAlly Tech Blog provides comprehensive guides on taking AI agents from "cool demo" to "enterprise-grade" healthcare solutions.

Conclusion: The Future of Agentic Health

By using LangGraph and GPT-4, we've transformed a static chatbot into a dynamic, multi-disciplinary team. This system doesn't just give advice; it negotiates a solution where the exercise plan compensates for the diet, and the monitor ensures safety.

Key Takeaways:

Cyclic Logic: Chronic care isn't a straight line. Use cycles to verify outcomes.
Persistence: Use Redis checkpointers to keep the conversation alive across sessions.
Specialization: Don't build one "Super Agent." Build small, expert agents that do one thing perfectly.

What’s your next agentic build? Let me know in the comments below! 🚀

Stop Sending Your Health Data to the Cloud: Build a 100% Private Symptom Checker with WebLLM & WebGPU 🚀

wellallyTech — Sun, 26 Apr 2026 01:00:00 +0000

Privacy is no longer just a feature; it’s a human right—especially when it comes to medical data. With the rise of Edge AI and the maturity of WebGPU, we are witnessing a paradigm shift where "local-first" isn't just a dream for small scripts, but a reality for Large Language Models (LLMs).

In this tutorial, we will build a Privacy-Preserving Symptom Screening Engine. By leveraging WebLLM, WebGPU, and TypeScript, we will run a powerful model directly in the browser. This ensures that sensitive health queries never leave the user's device, providing a 100% offline-capable, secure medical common-sense index.

Why Browser-Based AI? 🥑

Most AI applications rely on sending prompts to a centralized server (like OpenAI or Anthropic). This is a privacy nightmare for health data. By using WebLLM and TVM Runtime, we can execute models like Llama 3 or Mistral directly on the client's GPU via the WebGPU API. This results in:

Zero Latency: No round-trips to a server.
Zero Cost: The user provides the compute.
Total Privacy: Data stays in the RAM/VRAM of the local machine.

The Architecture 🏗️

The flow is straightforward but technically sophisticated. We use TVM (Tensor Intermediate Representation) to compile models into high-performance WASM and WebGPU shaders.

graph TD
    A[User Input: Symptoms] --> B[React Frontend]
    B --> C{WebLLM Engine}
    C --> D[WebGPU API]
    C --> E[WASM Runtime]
    D --> F[Local GPU / VRAM]
    E --> F
    F --> G[Local LLM Weights - Cached]
    G --> H[Inference Result]
    H --> B
    B --> I[100% Private Output]

Prerequisites

Before we dive in, ensure your environment meets the following:

Tech Stack: WebLLM, TVM Runtime, TypeScript, React.
Browser: Chrome 113+ or any browser with WebGPU enabled.
Hardware: A dedicated or integrated GPU (Apple Silicon M-series works like a charm).

Step 1: Setting up the WebLLM Engine

First, let's install the core dependencies:

npm install @mlc-ai/web-llm react lucide-react

Now, let's create a custom hook to manage the model lifecycle. We need to handle the downloading of weights (cached in the browser's Cache API) and the initialization of the worker.

// useWebLLM.ts
import { useState, useEffect } from 'react';
import * as webllm from "@mlc-ai/web-llm";

export function useWebLLM(modelId: string) {
  const [engine, setEngine] = useState<webllm.MLCEngineInterface | null>(null);
  const [progress, setProgress] = useState(0);
  const [isReady, setIsReady] = useState(false);

  useEffect(() => {
    async function init() {
      // Create an engine instance
      const newEngine = await webllm.CreateMLCEngine(modelId, {
        initProgressCallback: (report) => {
          setProgress(Math.round(report.progress * 100));
        }
      });
      setEngine(newEngine);
      setIsReady(true);
    }
    init();
  }, [modelId]);

  return { engine, progress, isReady };
}

Step 2: Building the Symptom Checker Logic

In this advanced implementation, we aren't just chatting; we are constraining the model to act as a Medical Common Sense Indexer. We'll use a specific system prompt to prevent the model from giving definitive diagnoses while providing helpful, localized information.

// SymptomChecker.tsx
import React, { useState } from 'react';
import { useWebLLM } from './useWebLLM';

const SYSTEM_PROMPT = `You are a private medical common-sense assistant. 
Analyze the symptoms provided. 
Provide 3 potential causes and urgency levels. 
ALWAYS include a disclaimer that this is not a medical diagnosis. 
Do not store or transmit data.`;

export const SymptomChecker = () => {
  const [input, setInput] = useState("");
  const [response, setResponse] = useState("");
  const { engine, progress, isReady } = useWebLLM("Llama-3-8B-Instruct-v0.1-q4f16_1-MLC");

  const handleScreening = async () => {
    if (!engine) return;

    const messages = [
      { role: "system", content: SYSTEM_PROMPT },
      { role: "user", content: `Symptoms: ${input}` }
    ];

    // Use streaming for better UX
    const chunks = await engine.chat.completions.create({
      messages,
      stream: true,
    });

    let fullText = "";
    for await (const chunk of chunks) {
      const content = chunk.choices[0]?.delta?.content || "";
      fullText += content;
      setResponse(fullText);
    }
  };

  if (!isReady) return <div>Loading Local Model: {progress}%</div>;

  return (
    <div className="p-6 max-w-2xl mx-auto bg-white rounded-xl shadow-md">
      <h2 className="text-2xl font-bold mb-4">Local Health Screener 🩺</h2>
      <textarea 
        className="w-full p-3 border rounded-lg"
        placeholder="Describe your symptoms (e.g., mild fever, persistent cough for 2 days)..."
        value={input}
        onChange={(e) => setInput(e.target.value)}
      />
      <button 
        onClick={handleScreening}
        className="mt-4 px-6 py-2 bg-blue-600 text-white rounded-full hover:bg-blue-700 transition"
      >
        Analyze Locally
      </button>
      {response && (
        <div className="mt-6 p-4 bg-gray-50 rounded-lg border-l-4 border-blue-500">
          <p className="whitespace-pre-wrap text-gray-700">{response}</p>
        </div>
      )}
    </div>
  );
};

Mastering the "Local-First" AI Pattern 🧠

Running models in the browser is more than just importing a library. To make it production-ready, you need to handle memory management (WebGPU memory limits), model quantization, and hybrid fallback strategies.

For a deeper dive into Advanced Edge AI Patterns—including how to optimize WebGPU memory for mobile devices and implementing RAG (Retrieval Augmented Generation) entirely on the client side—check out the engineering deep-dives on the WellAlly Technology Blog. It’s a fantastic resource for developers looking to push the boundaries of what's possible with in-browser machine learning.

Performance Considerations ⚡

When building with WebLLM, keep these three things in mind:

Quantization is Key: We used a q4f16 (4-bit) quantization. This reduces the 8B model size from ~15GB to ~5GB, making it feasible for modern laptops to load in the browser cache.
VRAM Management: High-resolution displays and heavy GPU tasks in other tabs can lead to "Context Lost" errors. Always implement an error boundary.
The First Load: The first visit will download several gigabytes of weights. Use a ServiceWorker to manage this background download or inform the user clearly about the initial setup.

Conclusion

By combining WebGPU and WebLLM, we've built a tool that provides the intelligence of a modern LLM with the privacy of a piece of paper in a safe. No trackers, no data harvesting, just pure local compute.

The future of AI isn't just in the cloud—it's right there in your browser's console.

What will you build with local WebGPU? Let me know in the comments below! 👇

Never Miss a Checkup Again: Building an Autonomous Health Agent with LangGraph and OpenAI Tool Calling 🏥🤖

wellallyTech — Sat, 25 Apr 2026 01:00:00 +0000

Managing personal health often feels like a full-time job. Between decoding cryptic lab results, keeping track of pill counts, and remembering when to book that follow-up appointment, things inevitably slip through the cracks. In the world of AI development, we are moving past simple chatbots and entering the era of Autonomous Agents that can actually do things for us.

In this tutorial, we are building a production-grade Autonomous Health Agent. Using LangGraph for orchestration and OpenAI Tool Calling for precision, we'll create a system that parses medical data, interacts with the Google Calendar API, and manages a medication inventory automatically. By the end of this guide, you'll master Python AI development patterns that go far beyond simple prompt engineering. 🚀

The Architecture: How the "Health Brain" Works

Unlike a standard linear pipeline, an agent needs to "think" and "act" iteratively. We use a cyclic graph where the LLM decides which tool to use based on the user's lab reports or inventory status.

graph TD
    A[User Input/Lab Result] --> B{LLM Decision Node}
    B -->|Analyze Lab| C[Lab Analyzer Tool]
    B -->|Schedule Follow-up| D[Google Calendar Tool]
    B -->|Check Inventory| E[Medication Tracker Tool]
    C --> B
    D --> B
    E --> B
    B -->|Task Complete| F[Final Response to User]
    style B fill:#f9f,stroke:#333,stroke-width:2px

Prerequisites

To follow along, you'll need the following tech stack:

Python 3.10+
OpenAI API Key (GPT-4o or GPT-4-turbo recommended for tool calling)
LangGraph & LangChain: For agent orchestration.
Google Workspace API Credentials: Specifically for Google Calendar access.

pip install langgraph langchain_openai google-api-python-client google-auth-httplib2 google-auth-oauthlib

Step 1: Defining the Tools (The Agent's Hands) 🛠️

Tools are the interfaces that allow our LLM to interact with the real world. We'll define two primary tools: one for scheduling and one for medication tracking.

from langchain_core.tools import tool
from datetime import datetime, timedelta

@tool
def schedule_appointment(reason: str, days_from_now: int):
    """Schedules a medical follow-up in Google Calendar."""
    # Logic to interface with Google Calendar API
    appointment_date = datetime.now() + timedelta(days=days_from_now)
    # Mocking the API call for brevity
    print(f"DEBUG: Scheduling {reason} for {appointment_date.date()}")
    return f"Successfully scheduled {reason} for {appointment_date.strftime('%B %d, %Y')}"

@tool
def check_medication_inventory(pill_name: str):
    """Checks the local DB for pill count and returns status."""
    # Imagine this queries a SQLite or Supabase DB
    inventory = {"Lisinopril": 5, "Metformin": 30}
    count = inventory.get(pill_name, 0)

    if count < 7:
        return f"Warning: Only {count} tablets of {pill_name} left. Refill required soon!"
    return f"You have {count} tablets of {pill_name} remaining."

Step 2: Building the Orchestration Graph with LangGraph

LangGraph allows us to maintain a "State" (memory) and define the flow of logic. This is where the Autonomous Agent logic lives.

from typing import TypedDict, Annotated, Sequence
from langchain_openai import ChatOpenAI
from langgraph.graph import StateGraph, END
from langchain_core.messages import BaseMessage, HumanMessage

class AgentState(TypedDict):
    messages: Annotated[Sequence[BaseMessage], "The conversation history"]

# Initialize the LLM with tool-binding
llm = ChatOpenAI(model="gpt-4o", temperature=0).bind_tools([
    schedule_appointment, 
    check_medication_inventory
])

def health_agent_node(state: AgentState):
    messages = state['messages']
    response = llm.invoke(messages)
    return {"messages": [response]}

# Define the graph
workflow = StateGraph(AgentState)
workflow.add_node("agent", health_agent_node)

# In a real app, you'd add a "tools" node and conditional edges
workflow.set_entry_point("agent")
workflow.add_edge("agent", END)

app = workflow.compile()

Step 3: Processing Lab Results (The "Brain" in Action)

When you upload a lab result, the agent doesn't just read it; it analyzes the markers. If your "Creatinine" is high, it might cross-reference your medication and automatically trigger a "Schedule Follow-up" tool call.

# Simulating a user providing a lab result
user_input = """
My lab results just came in. My Vitamin D is 18 ng/mL (Reference: 30-100). 
Also, I'm almost out of my blood pressure meds.
"""

inputs = {"messages": [HumanMessage(content=user_input)]}
for output in app.stream(inputs):
    for key, value in output.items():
        print(f"Node '{key}' processed the request.")

Looking for More Production-Ready Patterns? 🥑

Building a local prototype is easy, but deploying an autonomous agent that handles edge cases, HIPAA compliance (in the US), and complex state persistence is a different beast.

For advanced architectural patterns and more production-ready examples of how to scale AI agents, I highly recommend checking out the WellAlly Official Blog. It was a massive source of inspiration for the state management logic I used in this project, especially regarding how to handle "Human-in-the-loop" interactions where the agent asks for permission before booking a real doctor's appointment.

Step 4: The Inventory Warning System

The magic of Function Calling is that the LLM realizes it doesn't have the information and asks to use a tool.

LLM reads: "I'm almost out of meds."
LLM decides: "I should call check_medication_inventory."
Result: "Warning: Only 5 tablets left."
LLM action: "I see you're low on Lisinopril. I've added a reminder to your calendar to call the pharmacy today."

Conclusion: The Future of Proactive Health

We've just built a skeletal version of a life-changing tool. By combining LangGraph for complex workflows and OpenAI's tool-calling capabilities, we move from passive information retrieval to active life management.

What's next?

Add a Vision node using GPT-4o to "read" physical paper lab reports via photos.
Integrate Twilio to send SMS alerts when inventory is low.
Connect to a vector database (RAG) to provide context on what Vitamin D levels actually mean for your specific age group.

The code is just the beginning. The goal is a healthier, more organized you. Happy coding! 🚀

If you enjoyed this tutorial, drop a comment below! What would you want your personal AI agent to handle for you? 👇