Forem: Ethan Vance

What is CUDA? Understanding the Technology Behind AI and GPU Computing

Ethan Vance — Fri, 06 Mar 2026 05:01:03 +0000

If you're building infrastructure for Artificial Intelligence (AI), Machine Learning (ML), or High-Performance Computing (HPC), powerful hardware alone isn't enough. The real performance advantage comes from the software layer that drives the GPU. In NVIDIA's ecosystem, that layer is CUDA.

In this article, we'll break down what CUDA actually is, how its architecture works, and why it has become the industry standard for accelerating compute-intensive workloads.

What Exactly is CUDA?

Many developers assume CUDA is a programming language or even an operating system. That is not accurate.

CUDA (Compute Unified Device Architecture) is a parallel computing platform and programming model developed by NVIDIA. It allows developers to use the massive parallel processing power of GPUs for general-purpose computing.

Instead of relying only on CPUs for heavy computations, CUDA enables workloads like deep learning, scientific simulations, and matrix operations to run thousands of operations simultaneously on GPU cores.

Simple analogy

GPU → Raw compute engine
CUDA → Software layer that unlocks GPU parallelism

CUDA provides:

APIs
Compilers
Development tools
Optimized libraries

These tools allow developers to utilize GPU acceleration without writing low-level assembly code.

CPU vs GPU Architecture

Understanding CUDA requires understanding the fundamental difference between CPUs and GPUs.

Feature	CPU	GPU
Core Count	Dozens of powerful cores	Thousands of smaller cores
Execution Model	Sequential tasks	Massively parallel execution
Transistor Focus	Cache and control logic	Data processing throughput
Best Use Case	Complex control logic	Matrix operations and AI workloads

GPUs are specifically designed for data-parallel workloads, which is why they are ideal for deep learning and scientific computing.

The CUDA Software Stack

CUDA is not a single tool. It is a full ecosystem for GPU development.

nvcc – CUDA Compiler

The NVIDIA CUDA Compiler Driver (nvcc) separates:

Host code (runs on the CPU)

Device code (runs on the GPU)

This allows developers to write heterogeneous programs where CPU and GPU work together.

CUDA APIs

CUDA provides two major APIs:

CUDA Runtime API

High-level interface used in most CUDA applications.

CUDA Driver API

Low-level interface for more granular control of GPU execution.

CUDA Libraries

CUDA also provides highly optimized libraries used across AI and HPC applications.

cuBLAS

Optimized linear algebra operations for GPUs.

cuDNN

Deep neural network primitives such as convolution, pooling, softmax, and attention.

These libraries power frameworks like:

PyTorch
TensorFlow
JAX

CUDA Programming Model

CUDA assumes a heterogeneous system consisting of:

Host

CPU
Host memory

Device

GPU
Device memory

Execution typically follows this workflow.

Data Transfer

Data is copied from host memory (CPU) to device memory (GPU).

Kernel Execution

A CUDA function called a Kernel is executed on the GPU.

Execution hierarchy:

Threads
Blocks
Grids

Threads are the smallest execution units, while blocks allow threads to cooperate using shared memory.

Result Retrieval

Once the computation is complete, results are copied back from GPU memory to CPU memory.

Performance depends heavily on memory access patterns. Efficient CUDA programs maximize the use of:

Registers
Shared memory

while minimizing slower global memory access.

Why CUDA Dominates AI Infrastructure

NVIDIA’s leadership in AI infrastructure is largely due to the CUDA ecosystem.

Reasons include:

Mature development platform
Highly optimized performance libraries
Deep integration with AI frameworks
Strong developer ecosystem

Major frameworks like PyTorch and TensorFlow rely heavily on CUDA for GPU acceleration.

Because CUDA applications are built specifically for NVIDIA GPUs, it has also created a strong ecosystem around NVIDIA hardware.

Final Thoughts

CUDA has become a foundational technology for modern GPU computing. By enabling developers to harness massive parallelism inside GPUs, CUDA allows AI systems, machine learning models, and scientific computing workloads to run dramatically faster.

For developers working with AI, HPC, or GPU-accelerated computing, understanding CUDA is essential.

Original article:

Understanding NVIDIA CUDA: The Core of GPU Parallel Computing

Finally found a way to rent H100s without selling a kidney (MIG Tech)

Ethan Vance — Tue, 20 Jan 2026 12:17:28 +0000

Is it just me, or is trying to rent a dedicated H100 or A100 right now an absolute nightmare?

I've been working on some LLM fine-tuning recently, and I kept running into the same problem: Overkill.

I needed the architecture of the H100 (for the transformer engine), but I didn't need the entire card 24/7. Paying $4/hr+ for a GPU that sits idle 80% of the time just burns through the budget.

The "Aha" Moment: Splitting the Hardware

I did some digging and realized I should be looking for MIG (Multi-Instance GPU) capable servers.

If you aren't familiar with it, MIG basically lets you slice a physical GPU (like an A100 or H100) into up to 7 completely isolated instances. It’s not just software partitioning; it’s hardware-level isolation. So you get your own dedicated memory and cache.

The Resource: MIG servers

I came across a provider called MIG servers that specializes exactly in this. I wanted to share it here because their inventory is actually pretty impressive compared to the "Sold Out" signs I see everywhere else.

They seem to have bare metal stock in:

USA: Dallas, LA, Chicago
Europe: Luxembourg, London, Amsterdam
Asia: Incheon, Tokyo

What stood out to me was the flexibility. You can grab a massive 8x H100 cluster if you are training, or just slice up an A100 if you are doing inference.

Why it matters

If you are a DevOps engineer or working in AI, you know that "Time-Slicing" is usually laggy and insecure. MIG solves that.

I wrote a deeper breakdown on my personal blog about the technical specs and pricing comparisons, but I just wanted to drop this here for anyone struggling to find hardware.

To give you an idea of what MIG-ready hardware looks like, here are the specs we typically deploy for these workloads at MIG Servers:

Location	CPU	GPU Configuration	Max MIG Instances
Luxembourg	2x Xeon Platinum 8480+	8x NVIDIA H100 (200Gbps)	56 Instances
Dallas, USA	2x EPYC 9354	8x NVIDIA H100 NVLink	56 Instances
London, UK	2x Xeon Gold 6210U	NVIDIA A30	4 Instances

👉 Check out full breakdown and the server list here

Let me know if you guys have tried partitioning H100s yet!

[Boost]

Ethan Vance — Mon, 17 Nov 2025 06:40:34 +0000

Architecture for APAC: The Engineering Case for Singapore Bare Metal Infrastructure

Ethan Vance ・ Nov 17

#linux #webdev #dedicatedservers

Architecture for APAC: The Engineering Case for Singapore Bare Metal Infrastructure

Ethan Vance — Mon, 17 Nov 2025 06:28:16 +0000

For DevOps engineers and solutions architects deploying in the Asia-Pacific (APAC) region, the challenge isn't just distance; it is network topology. With a user base exceeding 3 billion, the difference between a 50ms and a 200ms Round Trip Time (RTT) dictates application viability.

While cloud virtualization offers flexibility, high-performance workloads—specifically gaming, real-time analytics, and LLM training—often hit the "noisy neighbor" wall. This article analyzes the technical infrastructure of Singapore as a hosting hub, examining connectivity ecosystem, hardware proximity, and the efficiency of bare metal over virtualized environments.

1. The Physics of Latency: Why Topology Matters

Singapore isn't just a geographical location; it is a primary peering exchange point. The island acts as a landing site for over 30 major submarine cable systems (including AAG, SJC2, and FASTER).

For a developer, this density translates to fewer hops. When you host in Singapore, you aren't routing through Japan or the US to reach Indonesia or India. You are utilizing direct peering links.

Typical RTT Metrics from Singapore (SG1):

Jakarta/Manila: < 20ms
Tokyo/Mumbai: < 50ms
Sydney: < 95ms

Technical Note: Achieving these speeds requires a provider using multi-homed BGP (Border Gateway Protocol) sessions. BGP automation ensures that if a specific carrier (e.g., NTT) experiences packet loss, the route automatically fails over to an alternative path (e.g., Tata or Singtel) without manual intervention.

2. Bare Metal vs. Virtualization Overhead

The convenience of VPS (Virtual Private Servers) comes with a performance tax known as "Hypervisor Overhead." In a virtualized environment, the physical CPU must translate instructions from the guest OS to the host hardware.

For I/O-heavy applications, this results in:

CPU Steal Time: Waiting for the physical scheduler to allocate cycles.
I/O Wait: Latency introduced by sharing disk controllers with other tenants.

The Bare Metal Advantage: Deploying on dedicated hardware (e.g., AMD EPYC 9754 or Intel Xeon Platinum) provides raw access to the kernel. There is no abstraction layer.

PCIe 5.0 & NVMe: You get full throughput (up to 14 GB/s read speeds) without virtualization throttling.

Deterministic Performance: Unlike a VPS where performance fluctuates based on neighbors, dedicated resources provide a flat-line performance graph essential for predictable SLAs.

3. The GPU Sovereignty Factor

For AI engineers working with Large Language Models (LLMs) or CUDA-accelerated rendering, hardware availability is a critical bottleneck.

Singapore offers a unique distinct advantage regarding high-performance compute (HPC) availability. Data centers here frequently stock enterprise-grade clusters (NVIDIA H100, A100, L40S) that are often supply-constrained in Western availability zones. Accessing these via bare metal allows for:

Direct Passthrough: No vGPU licensing costs or performance loss.
Cluster Scaling: Low-latency cross-connects allow for efficient multi-node training setups.

4. Data Center Efficiency (PUE) and Resilience

**
Modern infrastructure in Singapore is dictated by land scarcity, driving vertical innovation. The standard for new facilities involves strictly regulated power efficiency.

Key Specs for System Architects:

PUE (Power Usage Effectiveness): < 1.3. This is achieved via Direct-to-Chip liquid cooling, essential for sustaining the thermal design power (TDP) of modern high-density racks.
Compliance Stack: Look for SOC 2, ISO 27001, and TVRA (Threat Vulnerability Risk Assessment) certification.
N+1 Redundancy: Ensure the facility utilizes independent dual power feeds to the rack, backed by redundant UPS and generator systems.

Summary: When to Switch to Dedicated

**
While containerization (Kubernetes) on cloud instances serves microservices well, monolithic databases, high-frequency trading platforms, and game servers require the raw clock speed of dedicated hardware.

If your traceroute shows excessive hops or your database IOPS are inconsistent, moving the workload to a Singapore-based dedicated environment is the logical architectural step for stabilizing APAC performance.