Forem: Phil Yeh

From Theory to Practice: Digital Twin Core Concepts and Implementation Ideas for Engineers

Phil Yeh — Fri, 12 Dec 2025 07:19:31 +0000

🌐 The Bridge: Why Digital Twins Matter Now
Have you ever wished you could predict a machine failure before it happens, or simulate the impact of a change in your supply chain without risking real-world downtime? That's the power of the Digital Twin.

A Digital Twin is more than just a fancy 3D model. It's a live, virtual replica of a physical asset, system, or process that is constantly synchronized with real-world data. It serves as a testing ground, a crystal ball, and a diagnostic tool all rolled into one.

For engineers, understanding Digital Twins is crucial for mastering the next phase of IoT and predictive analytics in fields like manufacturing, smart cities, and energy management.

🔬 Step 1: Deconstructing the Digital Twin (The Three Core Layers)
To build a Twin, we must first understand its three fundamental components.

The Physical Asset Layer (The Source) This layer includes the real-world equipment and the infrastructure used to gather data:

Key Technologies: IoT sensors, PLCs, and Edge Computing devices.

Data Types: Real-time metrics like temperature, pressure, vibration, and energy consumption.

The Virtual Model Layer (The Brain) This is where the magic happens—the calculations, simulations, and predictions.

Behavioral Models:

Physics-Based: Uses known equations (thermodynamics, fluid dynamics) to predict behavior.

Data-Driven (ML/AI): Uses historical data to train models that predict failures or optimal settings.

Data Structure: Requires robust databases, often Time Series Databases (e.g., InfluxDB), to efficiently handle high-velocity, timestamped sensor data.

The Connection & Services Layer (The Data Flow) This is the communication pipeline that ensures the Twin is alive. It requires bi-directional data flow.

Inbound Flow (Physical to Virtual): Sensors push data to the cloud/edge (often via MQTT).

Outbound Flow (Virtual to Physical): The Twin sends control commands or optimization suggestions back to the physical asset (e.g., throttling a motor speed).

🛠️ Step 2: The Engineer's Starting Guide (A POC Blueprint)
Ready to start building your first Twin? Here is a practical, two-phase approach focusing on open-source tools.

Phase A: Data Ingestion and Basic Shadowing
Your goal here is to create a "Shadow Twin"—a basic model that mirrors the live state.

Set up MQTT Broker: Start a lightweight message broker (e.g., Mosquitto or a cloud service like AWS IoT Core).

The Python Data Emitter: Use Python to simulate or collect sensor readings and publish them to the broker.

Python

# python_emitter.py - Simulating sensor data publishing
import paho.mqtt.client as mqtt
import time
import random

broker_url = "your_mqtt_broker"
topic = "asset/motor/temperature"

client = mqtt.Client()
client.connect(broker_url, 1883, 60)

while True:
    temp = 70 + random.uniform(-2, 2)  # Simulate temp fluctuation
    client.publish(topic, f"{time.time()},{temp:.2f}")
    print(f"Published: {temp:.2f}")
    time.sleep(5)

Visualization: Use Grafana to subscribe to the MQTT topic and display the data on a dashboard. This is your first visual Twin!

Phase B: Integrating Predictive Intelligence
Now, let's add the "intelligence" to the Twin using a simple Machine Learning model.

Model Training (Hypothetical RUL Model): Assume you've trained a classification model (using Scikit-learn or similar) to predict the Remaining Useful Life (RUL) of your motor based on its temperature and vibration history.

The Prediction Service: A dedicated Python service reads the latest data and feeds it into the trained model.

Python
# prediction_service.py - The Twin's intelligence
import pandas as pd
from joblib import load
# Assume 'rul_predictor.joblib' is a trained ML model

model = load('rul_predictor.joblib')

def predict_rul(latest_data):
    # Process latest_data (e.g., features for the last 1 hour)
    features_df = pd.DataFrame([latest_data]) 
    prediction = model.predict(features_df)

    # 0 = Normal, 1 = Caution, 2 = Failure imminent
    return prediction[0]

(This service would run continuously, reading from the Time Series DB)

By connecting this prediction service to your live data stream, your Twin starts providing actionable insights (e.g., sending an alert when the RUL drops below 10 days).

🚀 Step 3: Challenges and The Future
Key Challenges in Implementation
Data Quality: Twins are only as good as the data they receive. Dealing with sensor drift, gaps, and noise is a massive engineering challenge.

Synchronization Latency: For real-time control applications (like self-driving cars), the delay between the physical event and the virtual update must be minimal.

Scalability: Managing the data synchronization and simulation load for millions of individual Twins (e.g., every turbine in a wind farm).

Looking Ahead
The future of Digital Twins is exciting:

XR Integration: Using AR/VR headsets to overlay live Twin data onto the physical asset during maintenance (e.g., seeing a projected temperature reading overlaid on the actual motor).

Edge Twins: Shifting more simulation and predictive processing to Edge devices to reduce latency and cloud costs.

📢 What’s Your Twin?
Digital Twin technology transforms maintenance from reactive to predictive.

What process or asset in your current engineering domain do you think is ripest for Digital Twin development? Share your ideas and challenges in the comments below!

The Architecture of Implicit Messaging: Implementing Raw CIP I/O in Python

Phil Yeh — Wed, 03 Dec 2025 03:26:51 +0000

The Challenge of Class 1 I/O
Ethernet/IP (EIP) is based on the Common Industrial Protocol (CIP), which defines two primary messaging types:

Explicit Messaging (TCP 44818): Request/Response—used for configuration and diagnostics.

Implicit Messaging (UDP 2222): Cyclic I/O—used for high-speed, repetitive data exchange (Class 1 Connections).

The architectural challenge lies in managing resource contention and time determinism when setting up these complex connections using raw sockets, rather than relying on commercial drivers.

🏗️ The 4-Step Connection Sequence
A robust implementation requires careful management of the TCP setup and the subsequent UDP I/O lifecycle. Our architecture follows these four steps within a cyclic process:

Register Session (TCP)
The process begins with an Explicit Message to the target device's Encapsulation Layer (TCP 44818). This step establishes a Session Handle, which identifies the connection for subsequent requests.
Forward Open (TCP)
This is the most critical step. The client sends a Forward Open command containing all parameters necessary for the Class 1 I/O connection, including:

Requested Packet Interval (RPI).

Connection Path (Assembly Instance IDs for the I/O data).

Connection Timeout parameters. The device returns the O2T (Originator to Target) and T2O (Target to Origin) Connection IDs.

Cyclic I/O Exchange (UDP)
Once the connection is established via TCP, the system shifts to high-speed UDP 2222. The client sends periodic data using the established O2T connection ID, and the device responds with the T2O data. This ensures minimal latency for cyclic data updates.
Forward Close & Teardown (TCP)
To prevent the target device from eventually timing out and reporting a fault, the client must explicitly send a Forward Close command (TCP). This gracefully releases the resources allocated by the device before the socket is closed.

💻 Architecture for Deterministic Testing
To reliably test this intricate sequence, our study kit employs a dual-application structure:

Raw Socket Client: Implements the full TCP and UDP state machine, managing the cyclic Open/Close sequence.

Mock PLC Server: A separate Python application running on localhost that listens on TCP 44818 and UDP 2222. The mock server is essential for deterministic testing, as it guarantees correct, instantaneous responses during the handshake, allowing developers to isolate logic errors from physical layer noise.

Python I/O Loop Structure
The raw socket implementation requires precise packet assembly. Below is the conceptual structure used to manage the cyclic UDP exchange:

Python

# Conceptual Structure for I/O Loop

while running:
    # 1. Open Session (TCP) & Forward Open (TCP) is performed here...

    # 2. UDP Exchange Phase:
    try:
        # Construct and send O2T packet using raw bytes
        udp_socket.sendto(o2t_packet, (target_ip, 2222))

        # Receive T2O packet (Target to Origin)
        received_data, addr = udp_socket.recvfrom(1024)

        # Log and parse the raw data...

    except socket.timeout:
        log("Warning: UDP I/O Timeout occurred.")

    # 3. Forward Close (TCP) & TCP Disconnect is performed here...

    time.sleep(RPI_WAIT_TIME)

🔒 Conclusion
Understanding the raw socket implementation of CIP is critical for developers working in industrial cybersecurity, custom SCADA integration, or embedded systems where external libraries are too large or unavailable.

We have documented the complete architecture and technical insights for this project on GitHub. If you are interested in acquiring the full, ready-to-deploy Python source code for this framework and diving into the raw packet structure for your own custom industrial solutions:

View the complete project and detailed architecture: Link

By Phil Yeh | Senior Automation Engineer

How I Fixed Python's Serial Freezing Issue: A Multi-threaded Tkinter Solution

Phil Yeh — Fri, 28 Nov 2025 01:51:40 +0000

As automation engineers, we hit a frustrating wall: writing a simple Modbus GUI tool with Tkinter, only to have the entire application freeze ("Not Responding") while waiting for the slow RS485 sensor to reply.

If you are dealing with serial I/O and real-time UI updates, the solution lies in proper threading.

🛠️ The Fix: Multi-threading for Stability

The core issue is blocking I/O. Since the main Tkinter loop handles the UI, waiting for the serial port stops the entire window from updating.

The professional solution is to use the threading module to separate the UI rendering thread from the I/O polling thread.

Core Threading Logic

We implemented a daemon thread dedicated solely to reading the Modbus device. This ensures the UI is responsive, even if the polling takes several seconds.

# The UI remains responsive because the heavy lifting runs in a separate thread.
def thread_helper(self):
    thread = threading.Thread(target=self.start_log, daemon=True)
    thread.start()
    # Tkinter mainloop continues to run smoothly

📘 Modbus Protocol Primer: Deciphering the Hex (Tutorial Value)
To effectively debug Modbus, you need to understand the structure of the command frame. The tool guides users through this structure directly on the screen:

(The default command in the tool is 01 03 00 00 00 03)

✨ Production Features (The True Value)
This project provides a robust, reusable template that is ready for industrial deployment. The source code handles all the time-consuming integration headaches:

Universal Serial Setup: Supports custom Data Bits (5-8), Parity (N, E, O), and Stop Bits (1, 1.5, 2).

Auto CRC-16: Automatically calculates and appends the Modbus checksum.

Raw Hex Logging: Logs both the raw response and parsed data to a CSV file.

📥 Get the Fully Integrated Solution
This article shares the architectural solution. If you want the complete, multi-threaded GUI source code—including the full UI logic, auto-CRC calculation, and CSV logging feature—you can acquire the packaged files here-$9.9.

By Phil Yeh | Senior Automation Engineer

Stop decoding Hex manually. I built a Python J1939 Sniffer with a GUI (No Hardware Needed)

Phil Yeh — Tue, 25 Nov 2025 00:57:06 +0000

After receiving the Top Docker Author badge last week for my Offline AI post (thanks everyone! 🙏), many of you asked about my workflow for hardware and vehicle networks.

So today, I'm switching gears from AI to Heavy Duty Vehicles.

If you work with CAN Bus or SAE J1939 (Trucks, Buses, Machinery), you know the pain:

Professional tools are expensive: A Vector CANalyzer license costs thousands of dollars.
Hex dumps are unreadable: Seeing 18FEF100 means nothing unless you memorize the J1939 spec.
Hardware dependency: You usually need a physical adapter (PCAN, Kvaser) just to test your code.

To solve this, I built a Python-based J1939 Sniffer that decodes PGNs automatically and includes a Simulation Mode for hardware-free development.

🏗️ The Challenge: Parsing 29-bit IDs

Standard CAN (11-bit) is simple. But J1939 uses 29-bit Extended Identifiers, which pack a lot of data:

Priority (3 bits)
PGN (Parameter Group Number) (18 bits) <--- The most important part
Source Address (8 bits)

If you get a raw ID like 0x18FEF100, you need to extract the PGN to know what the message actually is.

The Python Logic

Here is the core logic I used to extract the PGN and Source Address from a raw integer ID:

def parse_j1939_id(can_id):
    """
    Extract PGN and Source Address from a 29-bit CAN ID.
    Format: [Priority(3)] [Reserved(1)] [Data Page(1)] [PDU Format(8)] [PDU Specific(8)] [Source Address(8)]
    """
    # Shift right by 8 bits to drop Source Address
    # Mask with 0x3FFFF to keep only the 18-bit PGN
    pgn = (can_id >> 8) & 0x3FFFF

    # Mask with 0xFF to get the last 8 bits
    source_address = can_id & 0xFF

    # Shift right by 26 bits to get Priority
    priority = (can_id >> 26) & 0x7

    return pgn, source_address, priority

🛠️ The Solution: A GUI Sniffer
I wrapped this logic into a Tkinter GUI using the python-can library. It listens to the bus, parses the ID, and looks up the PGN in a built-in dictionary.

The Result
Instead of staring at 18FEF100, the tool tells you: 👉 CCVS - Vehicle Speed

Instead of 0CF00400, it shows: 👉 EEC1 - Engine Speed (RPM)

Features
🚛 Auto-Decode: Built-in dictionary for common PGNs (RPM, Temp, Speed, Battery).

🎮 Simulation Mode: Click "Start Demo" to generate fake J1939 traffic. Perfect for testing UI logic without sitting in a truck.

🔌 Universal Support: Works with Vector, Peak-System (PCAN), Kvaser, and slcan via python-can.

📥 Try it yourself
I have open-sourced the project structure and the J1939 parsing logic on GitHub. You can use it as a template for your own ECU tools.

🔗 GitHub Repository: Python-CAN-Bus-J1939-Sniffer-GUI

🎁 For those who want the full package: If you want the complete, production-ready source code (including the GUI, Simulation Mode, and Multi-threading), I've made it available on Gumroad.

🔥 Black Friday Deal: Use code BLACKFRIDAY for 15% OFF all my engineering tools.

👉 Get the Full Source Code(link)

Happy Hacking! 🚛

5 Python Tools I Built to Automate My Industrial IoT Workflow (Open Source)

Phil Yeh — Thu, 20 Nov 2025 10:00:43 +0000

Update: 🏷️ Black Friday Sale is ON! Use code BLACKFRIDAY for 15% OFF on the Ultimate Toolkit.

As a Senior Automation Engineer, I spend half my life debugging communication protocols. Modbus, MQTT, CAN Bus, Ethernet/IP... you name it.

The problem is, most professional tools (like Vector CANalyzer or proprietary PLC software) are:

Expensive (Thousands of dollars).
Windows-only (I love Docker/Linux).
Closed Source (I can't customize them).

So, over the past few weekends, I decided to build my own "Survival Toolkit" using Python.

Here are the 5 open-source tools I created to replace expensive software, all available on my GitHub.

1. The "Privacy-First" AI Datasheet Reader

Problem: I have hundreds of PDF datasheets to read, but I can't upload them to ChatGPT due to NDA/Privacy concerns.
Solution: A 100% offline RAG system using Docker + Llama 3.

It runs entirely on my local machine (RTX 3060). No data leaves the building.

Stack: Docker, Ollama, ChromaDB, Streamlit.
Repo: Local-AI-Knowledge-Base-Docker-Llama3

2. The J1939 & CAN Bus Sniffer

Problem: Debugging vehicle ECUs usually requires a $300+ hardware adapter just to see the data.
Solution: A Python GUI that works with cheap USB-CAN adapters (slcan) and automatically decodes J1939 PGNs.

It even has a "Demo Mode" to simulate traffic if you don't have hardware.

Stack: Python, tkinter, python-can.
Feature: Decodes Engine Speed, Temp, and other PGNs instantly.
Repo: Python-CAN-Bus-J1939-Sniffer-GUI

3. The "Anti-Freezing" Modbus Logger

Problem: Writing a simple Python script to read RS485 is easy, but the GUI always freezes (blocks) while waiting for the sensor to reply.
Solution: A multi-threaded Modbus RTU Master.

It separates the UI thread from the Serial polling thread, so the app remains responsive 100% of the time.

Stack: Python, tkinter, pyserial, threading.
Repo: Python-Modbus-Serial-Logger-GUI

4. The MQTT Data Recorder

Problem: Sometimes I just want to save MQTT sensor data to an Excel file for analysis, without setting up a database.
Solution: A lightweight MQTT Client that logs everything to CSV automatically.

Updated to support the latest Paho-MQTT v2.0 standard.

Stack: Python, paho-mqtt, csv.
Repo: Python-MQTT-Data-Logger-GUI

5. The Virtual Ethernet/IP Lab

Problem: Learning the CIP protocol (used by Rockwell/Omron PLCs) is hard without physical hardware.
Solution: A Mock PLC Server + Raw Socket Client.

It simulates the full Forward Open handshake and Implicit Messaging on your localhost.

Stack: Pure Python, socket, struct.
Repo: Python-EthernetIP-Raw-Socket-Client

🎁 Conclusion

Building your own tools is the best way to learn. Not only do you save money on licenses, but you also get full control over the source code.

I have open-sourced the documentation and basic architecture for all these projects on GitHub.

If you want the complete, production-ready source code for ALL these tools (5-in-1), I've bundled them together here:

👉 The Ultimate Senior Engineer Toolkit (Gumroad)

Happy coding!

Local RAG with Llama 3 & Docker: Build an Offline Second Brain (No OpenAI)

Phil Yeh — Tue, 18 Nov 2025 09:03:19 +0000

[UPDATE: Dec 2025] 🚀 Due to the overwhelming interest in this Local RAG setup, I’ve officially released the production-ready toolkit on Gumroad to help you save hours of configuration time!

Based on your feedback, I’ve created two versions to suit your needs:

Option 1: The "Lite" Edition ($59) – Perfect for developers! Get the full Dockerized source code, Streamlit UI, and PDF pipeline. Ideal if you want to deploy it yourself and own the code.

Option 2: The "Pro" Solution ($299) – For enterprises and busy professionals. Includes the full suite PLUS a 1-on-1 Remote Setup Service. I will personally ensure the system is perfectly tuned and running on your hardware.

Why use this?

100% Private: No data ever leaves your machine (No OpenAI/Cloud APIs).

One-Click Setup: Move from "dependency hell" to a functional RAG system in minutes using Docker.

Proven Results: Check out the new Demo Video on the product page to see it in action!

👉 Get the Local RAG Toolkit here

🎉 Update: Wow! This post was awarded the Top Docker Author Badge of the week! Thanks to everyone for the amazing support and feedback. 🙏

Stop sending your sensitive datasheets to the cloud. Here is how I deployed a private, enterprise-grade RAG system.

As a Senior Automation Engineer, I deal with hundreds of technical documents every month — datasheets, schematics, internal protocols, and legacy codebases.

We all know the power of LLMs like GPT-4. Being able to ask, “What is the maximum voltage for the RS485 module on page 42?” and getting an instant answer is a game-changer.

But there is a problem: Privacy.

I cannot paste proprietary schematics or NDA-protected specs into ChatGPT. The risk of data leakage is simply too high.

So, I set out to build a solution. I wanted a “Second Brain” that was:

100% Offline: No data leaves my local network.

Free to run: No monthly API subscriptions (bye-bye, OpenAI bills).

Dockerized: Easy to deploy without “dependency hell.”

Here is the architecture I built using Llama 3, Ollama, and Docker.

The Architecture: Why this Tech Stack?
Building a RAG (Retrieval-Augmented Generation) system locally used to be a nightmare of Python dependencies and CUDA driver issues. To solve this, I designed a containerized microservices architecture.

The Brain: Ollama + Llama 3
I chose Ollama as the inference engine because it’s lightweight and efficient. For the model, Meta’s Llama 3 (8B) is the current sweet spot — it’s surprisingly capable of reasoning through technical documentation and runs smoothly on consumer GPUs (like an RTX 3060).
The Memory: ChromaDB
For the vector database, I used ChromaDB. It runs locally, requires zero setup, and handles vector retrieval incredibly fast.
The Glue: Python & Streamlit
The backend is written in Python, handling the “Ingestion Pipeline”:

Parsing: Extracting text from PDFs.

Chunking: Breaking text into manageable pieces.

Embedding: Converting text into vectors using the mxbai-embed-large model.

UI: A clean Streamlit interface for chatting with the data.

How It Works (The “Happy Path”)
The beauty of this system is the Docker implementation. Instead of installing Python libraries manually, the entire system spins up with a single command.

The docker-compose.yml orchestrates the communication between the AI engine, the database, and the UI.

YAML

# Simplified concept of the setup
services:
  ollama:
    image: ollama/ollama:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  backend:
    build: ./app
    depends_on:
      - ollama
      - chromadb

Once running, the workflow is simple:

Drop your PDF files into the knowledge_base folder.

Click “Update Knowledge Base” in the UI.

Start chatting.

The system automatically vectorizes your documents. When you ask a question, it retrieves the most relevant paragraphs and feeds them to Llama 3 as context.

The Challenge: It’s Not Just About “Running” the Model
While the concept sounds simple, getting it to production-grade stability took me weeks of debugging.

Here is what most “Hello World” tutorials don’t tell you:

PDF Parsing is messy: Tables in engineering datasheets often break standard parsers.

Context Window limits: Llama 3 has a limit. You need a smart “Sliding Window” strategy for chunking large documents.

Docker Networking: Getting the Python container to talk to the Ollama container on the host GPU requires specific networking configurations.

I spent countless nights fixing connection timeouts, optimizing embedding models, and ensuring the UI doesn’t freeze during large file ingestions.

Want to Build Your Own?
If you are an engineer or developer who wants to own your data, I highly recommend building a local RAG system. It’s a great way to learn about GenAI architecture.

However, if you value your time and want to skip the configuration headaches, I have packaged my entire setup into a ready-to-deploy solution.

It includes:

✅ The Complete Source Code (Python/Streamlit).

✅ Production-Ready Docker Compose file.

✅ Optimized Ingestion Logic for technical docs.

✅ Setup Guide for Windows/Linux.

You can download the full package and view the detailed documentation on my GitHub.

👉 View the Project & Download Source Code on GitHub link

By Phil Yeh Senior Automation Engineer specializing in Industrial IoT and Local AI solutions.

Python MQTT Data Logger - A clean GUI to debug brokers & auto-save data to CSV.
Python CAN Bus & J1939 Sniffer - Decode vehicle data without expensive hardware.
Python Modbus Data Logger - Debug RS485 devices with a multi-threaded GUI.
Ethernet/IP Study Kit - Learn CIP protocol with a Python-based mock PLC. **** 👉 Get the Source Code for all these tools: Visit my Gumroad Store