Forem: Augustus otu

Building a GPU-Accelerated File Explorer in 48 Hours with Kiro IDE

Augustus otu — Thu, 04 Dec 2025 11:36:57 +0000

How AI-assisted development transformed an ambitious Rust project from concept to reality—and why knowing what you're building matters more than ever

The Challenge

When I decided to build Nexus Explorer—a GPU-accelerated file explorer in Rust—I knew I was setting an ambitious goal. File explorers are deceptively complex: they need to handle millions of files, provide instant feedback, manage async I/O without freezing the UI, and deliver a polished user experience across multiple platforms.

Traditional development would take weeks, maybe months. I had 4 hours.

The Most Important Lesson: Know What You're Building

Before I talk about AI and tools, let me be clear about something: AI-assisted development amplifies your expertise—it doesn't replace it.

If you don't understand:

Why async I/O matters for file operations
How GPU rendering differs from CPU rendering
What makes a good UI architecture
When to use caching vs. fresh data

...then AI will generate plausible-looking code that falls apart under real use. The AI doesn't know your constraints, your users, or your performance requirements. You do.

This is why the research phase was critical. Before writing a single line of code, I spent time understanding:

GPUI's architecture - Reading Zed's source code to understand Entity/View separation
Async patterns in Rust - How tokio, channels, and background executors interact
File system performance - Why jwalk is faster than walkdir, how batching prevents UI thrashing
Search algorithms - Why nucleo (from Helix) outperforms traditional fuzzy finders

This research informed every decision. When the AI suggested something that didn't fit the architecture, I could recognize it and steer toward the right solution.

Enter Kiro IDE: A Game Changer

This was my first time using Kiro IDE, and I'm now a convert. After this hackathon, Kiro is my go-to development environment.

What amazed me was Kiro's structured approach to AI-assisted development. It's not just "chat with an AI"—it's a complete system for directing AI work through well-defined structures:

Requirements → Design → Tasks

This three-tier spec system transformed how I worked with AI. Instead of ad-hoc prompts, I had:

Requirements: What the feature needs to accomplish
Design: How it should be architected
Tasks: Specific implementation steps

When I pointed Kiro at these specs, it was like having a colleague I could direct with precision. The AI had a plan to follow. It understood context. It knew what we were building and why.

It was a breeze.

Seriously. I've used other AI coding tools, and they feel like shouting into the void hoping something useful comes back. Kiro felt like pair programming with someone who actually read the project documentation.

What is "Vibe Coding"?

Vibe coding is a development philosophy where you:

Describe intent rather than dictate implementation
Iterate conversationally with AI to refine solutions
Focus on architecture while AI handles boilerplate
Review and guide rather than type everything
Catch mistakes because you understand the domain

It's like pair programming with an infinitely patient partner—but you're still the senior engineer making the calls. And with Kiro's spec system, that partner actually understands the bigger picture.

Day 1: Architecture and Foundation

Planning with Specs

Kiro's spec system was invaluable. I created structured requirement documents that served as both documentation and AI context:

.kiro/specs/
├── file-explorer-core/
│   ├── requirements.md    # What we're building
│   ├── design.md          # How we're building it
│   └── tasks.md           # Implementation checklist
└── ui-enhancements/
    ├── requirements.md
    ├── design.md
    └── tasks.md

This wasn't just documentation—it was a contract between me and the AI. When I referenced these specs, Kiro understood the full context of what we were building.

But here's the key: I wrote the requirements. The AI didn't decide we needed generational request IDs to prevent stale async results—I did, based on my understanding of the problem. The AI helped implement it correctly.

Key Architectural Decisions (Made by Me, Implemented with AI)

Decision 1: GPUI Framework

I chose GPUI (from the Zed editor) for GPU-accelerated rendering. This was a calculated risk—GPUI is powerful but has limited documentation. My research showed it could deliver smooth scrolling through large directories, which was essential for the user experience I wanted.

Kiro helped by:

Reading the GPUI source code to understand patterns
Suggesting Entity/View separation based on GPUI's ownership model
Generating boilerplate that matched GPUI's conventions

But when the AI suggested patterns that didn't fit GPUI's model, I recognized it and corrected course.

Decision 2: Async-First Architecture

The core philosophy: The UI must never wait for the file system.

This wasn't the AI's idea—it came from my experience with file managers that freeze when opening large directories. I knew we needed:

Background threads for I/O
Channels for communication
Batched updates to prevent render thrashing
Generational IDs to discard stale results

┌─────────────────────────────────────────────────────────┐
│                    Main Thread (UI)                      │
│  ┌─────────┐  ┌──────────┐  ┌─────────┐  ┌──────────┐  │
│  │Workspace│  │ FileList │  │ Sidebar │  │ Preview  │  │
│  └────┬────┘  └────┬─────┘  └────┬────┘  └────┬─────┘  │
└───────┼────────────┼─────────────┼────────────┼─────────┘
        │            │             │            │
        └────────────┴─────────────┴────────────┘
                          │ async channels
┌─────────────────────────┼───────────────────────────────┐
│              Background Executor (GPUI)                  │
└─────────────────────────┼───────────────────────────────┘
                          │ spawn_blocking
┌─────────────────────────┼───────────────────────────────┐
│                 Tokio Thread Pool                        │
│  ┌─────────┐  ┌──────────┐  ┌─────────┐  ┌──────────┐  │
│  │ jwalk   │  │  notify  │  │  icons  │  │  search  │  │
│  └─────────┘  └──────────┘  └─────────┘  └──────────┘  │
└─────────────────────────────────────────────────────────┘

Decision 3: Steering Files for Consistency

I set up steering rules in .kiro/steering/ to maintain consistency:

# attention.md
- No unnecessary line comments
- Read files before making changes
- Always check gpui and adabaraka folders for component patterns
- Write clean, testable, maintainable code

These rules persisted across sessions, ensuring the AI maintained code quality standards that I defined.

Building the Core

With architecture defined, I started building. The conversation flow looked like:

Me: "Let's implement the FileSystem model with LRU caching and generational request tracking"

Kiro: Generates implementation

Me: Reviews "The cache eviction isn't considering modification time. When a directory changes on disk, we need to invalidate."

Kiro: Updates implementation with mtime tracking

This back-and-forth was constant. The AI accelerated implementation, but I was always reviewing, always steering.

Day 2: Features and Polish

The Terminal Integration

One feature I'm particularly proud of: each tab maintains its own terminal instance, and the terminal automatically follows your folder navigation.

Me: "When I navigate to a folder, the terminal should cd there automatically"

Kiro: Implements initial version

Me: "It's not updating when I use the back button"

Kiro: Fixes navigation handlers

Me: "Each tab should have its own terminal state"

Kiro: Refactors to HashMap

This iterative refinement is the heart of vibe coding. I knew what I wanted; the AI helped me get there faster.

Smart Folders

I wanted saved searches that act like virtual folders. The implementation required:

A query builder UI
Persistent storage
Real-time filtering

The AI generated the SmartFolderDialog component, but I specified the query options based on what power users actually need: text patterns, file types, date ranges, size filters.

Theme System

Supporting light and dark modes meant building a proper theme system:

pub struct ThemeColors {
    pub bg_primary: Rgba,
    pub bg_secondary: Rgba,
    pub text_primary: Rgba,
    pub accent_primary: Rgba,
    // ... 20+ color tokens
}

Every component uses theme_colors() instead of hardcoded values. This was an architectural decision I made early—the AI just helped implement it consistently across 86 files.

The Numbers (Verified)

By the end of 48 hours:

Metric	Count
Rust source files	86
Lines of code	47,290
Passing tests	558
Platform targets	3 (macOS, Windows, Linux)

Performance

GPUI provides GPU-accelerated rendering, which means:

Smooth scrolling through large directories
Instant theme switching
Responsive UI even during heavy I/O

The exact frame rate depends on your hardware and display, but the architecture ensures the UI thread is never blocked by file system operations.

Feature List

GPU-accelerated rendering via GPUI
Parallel directory traversal with jwalk
Fuzzy search powered by nucleo
Multiple view modes (List, Grid, Column, Dual-Pane)
Quick Look preview
Integrated terminal with per-tab state
Smart Folders (saved searches)
Tags and Favorites
Tabs and multi-window support
Light and dark themes
File operations (copy, move, delete, rename)
Trash integration
Network storage support
Device monitoring

Lessons Learned

1. Research Before You Build

The most important work happened before I wrote any code. Understanding GPUI, async Rust, and file system performance let me make informed decisions and catch AI mistakes.

If you can't evaluate the AI's output, you can't use it effectively.

2. Requirements Come From You

The AI didn't know I needed:

Sub-16ms response times
Generational request tracking
LRU caching with mtime invalidation
Per-tab terminal state

These requirements came from understanding the problem domain. The AI helped implement them, but the vision was mine.

3. Specs Are Essential

The upfront investment in requirements and design documents paid off enormously. When I said "implement the tab system per the spec," Kiro understood the full context.

4. Steering Maintains Quality

Without steering rules, AI-generated code can drift in style and quality. The .kiro/steering/ files kept everything consistent across 47,000+ lines of code.

5. Iterate, Don't Dictate

The best results came from describing intent and iterating:

"The terminal should follow folder navigation" → initial implementation
"It's not updating when I use the back button" → fix navigation handlers
"Each tab should have its own terminal" → refactor to HashMap

6. Trust but Verify

AI accelerates development, but you still need to:

Review generated code for correctness
Run tests frequently
Check edge cases
Understand what's being built

I caught several issues where the AI's implementation didn't match the architecture I had in mind. Without domain knowledge, those bugs would have shipped.

The Human-AI Partnership

Here's what I want you to take away: AI didn't build this project. I did, with AI assistance.

The AI didn't:

Choose the tech stack
Design the architecture
Define the requirements
Make performance tradeoffs
Decide what features mattered

I did all of that. The AI helped me implement it faster than I could have alone.

This is the future of software development: humans providing vision, judgment, and domain expertise; AI providing velocity and breadth. Together, we built something that would have taken a team weeks to accomplish.

But it only works if you know what you're building.

What's Next

Nexus Explorer is open source and actively developed. Planned features include:

Windows NTFS USN Journal for instant search
Linux io_uring for maximum I/O throughput
Cloud storage integration even though we have icloud locally ready on macOs

Why I'm Now a Kiro Convert

Before this hackathon, I was skeptical of AI coding assistants. They felt like autocomplete on steroids—useful for boilerplate, but not for real engineering.

Kiro changed my mind.

The difference is structure. Other tools give you a chat window and hope for the best. Kiro gives you:

Specs that the AI actually follows
Steering rules that maintain consistency across sessions
Context awareness that understands your project's architecture

When I wrote requirements like "the UI must never block on file system operations," Kiro didn't just acknowledge it—it applied that principle across every feature we built together. When I defined steering rules about code style, those rules persisted through 47,000 lines of code.

It felt less like using a tool and more like directing a capable junior developer who never gets tired, never forgets context, and types at superhuman speed.

After this hackathon, Kiro is my daily driver. Not because AI writes better code than me—it doesn't. But because the combination of my expertise and Kiro's velocity produces results neither could achieve alone.

Conclusion

Building Nexus Explorer in 48 hours required two things: deep understanding of the problem domain, and AI tools that could keep up with my vision.

Kiro IDE and vibe coding didn't replace engineering—they amplified it. Every architectural decision, every requirement, every tradeoff came from me. The AI just helped me move faster.

The structured approach—requirements, design, tasks—meant I wasn't just prompting an AI. I was leading a project with AI as my implementation partner. That's a fundamentally different experience, and it's why I'll keep using Kiro long after this hackathon ends.

The code is open source. The approach is reproducible. But the most important ingredient isn't the AI—it's knowing what you want to build and why. Kiro just makes it possible to build it faster than you ever thought possible.

Nexus Explorer is available on GitHub. Built with Rust, GPUI, and Kiro IDE.

Build your own online code compiler

Augustus otu — Tue, 08 Jul 2025 08:07:49 +0000

Augustus otu

Jun 24 '25

Building an Online Code Compiler: A Complete Guide

#go #webdev #programming #distributedsystems

Comments

27 min read

Building an Online Code Compiler: A Complete Guide

Augustus otu — Tue, 24 Jun 2025 14:55:51 +0000

By the Toki Space Team

Creating a production-grade online code compiler is one of the most complex and demanding projects in distributed systems. It calls for expertise in container orchestration, job queuing, real-time data flow, security sandboxing, and precise resource control. Unlike basic code execution tools, a full-fledged compiler platform must support multiple programming languages, manage parallel executions, stream outputs live, and recover gracefully from errors.

In this tutorial, you’ll learn how to build a fully functional online compiler using Docker for containerization, RabbitMQ for managing execution jobs, Redis for real-time communication, and React for the user interface. We’ll walk through the architecture, implementation, and deployment process—drawing from real-world experience building the code execution system behind Toki Space.
Please forgive me, if there are code execution errors. This is mainly meant to give you an idea of how it works under the hood. Let's dive in.

Architecture Overview
Docker Container System
Message Queue Integration
Real-time Streaming
Backend Implementation
Frontend Development
Security & Isolation
Performance Optimization
Production Deployment
Key Learnings

Architecture Overview

System Components

Our online code compiler consists of five main components working together:

┌─────────────────┐    WebSocket    ┌─────────────────┐
│   React         │ ◄──────────────► │   Node.js       │
│   Frontend      │                 │   Backend       │
│                 │   HTTP/REST     │                 │
└─────────────────┘ ◄──────────────► └─────────────────┘
                                              │
                                              │ Jobs
                                              ▼
┌─────────────────┐                 ┌─────────────────┐
│   Redis         │                 │   RabbitMQ      │
│   (Streaming)   │                 │   (Job Queue)   │
└─────────────────┘                 └─────────────────┘
         ▲                                   │
         │ Results                           │ Jobs
         │                                   ▼
         └─────────── ┌─────────────────┐ ◄─────
                     │   Runner        │
                     │   Service       │
                     │   (Go)          │
                     └─────────────────┘
                              │
                              │ Docker API
                              ▼
                     ┌─────────────────┐
                     │   Docker        │
                     │   Containers    │
                     │   (Multi-lang)  │
                     └─────────────────┘

Core Technologies

Frontend: React with TypeScript for the user interface
Backend: Node.js with Express for API and WebSocket handling
Runner Service: Go service for container management and code execution
Message Queue: RabbitMQ for reliable job distribution
Streaming: Redis for real-time output streaming
Containers: Docker for secure code execution isolation

Design Principles

Language Agnostic: Support for Python, Node.js, Go, Rust, Java, and more
Secure Isolation: Each execution runs in a separate Docker container
Real-time Feedback: Stream output as code executes
Scalable Architecture: Horizontal scaling through message queues
Fault Tolerance: Graceful handling of failures and timeouts

Docker Container System

The Challenge of Multi-Language Execution

Running user code safely requires solving several complex problems:

Security Isolation: Prevent malicious code from accessing the host system
Resource Limits: Control CPU, memory, and execution time
Environment Setup: Provide language-specific tools and dependencies
Cleanup: Remove containers and workspaces after execution

Container-Per-Language Architecture

Instead of spinning up new containers for each execution (which is slow), we use persistent containers per language:

type LanguageVM struct {
    Language      string
    ContainerName string
    IsRunning     bool
    WorkspaceDir  string
    mutex         sync.Mutex
}

type Manager struct {
    config       config.FirecrackerConfig
    logger       *logrus.Logger
    vms          map[string]*LanguageVM
    vmsMutex     sync.RWMutex
}

Benefits of Persistent Containers:

Fast Execution: No container startup overhead
Warm Environments: Dependencies already installed
Resource Efficiency: Reuse container resources
Consistent State: Predictable execution environment

Container Initialization

Each language gets its own persistent container with pre-installed tools:

func (m *Manager) initializePersistentContainers() error {
    m.logger.Info("Initializing persistent containers for all languages...")

    for language := range m.config.Environments {
        m.logger.Infof("Starting persistent container for language: %s", language)

        vm := &LanguageVM{
            Language:      language,
            ContainerName: fmt.Sprintf("runner-vm-%s", language),
            WorkspaceDir:  filepath.Join(m.config.WorkspaceDir, language),
            IsRunning:     false,
        }

        // Create language-specific workspace directory
        if err := os.MkdirAll(vm.WorkspaceDir, 0755); err != nil {
            return fmt.Errorf("failed to create workspace directory for %s: %w", language, err)
        }

        // Start the container
        if err := m.startPersistentContainer(vm); err != nil {
            m.logger.Errorf("Failed to start container for %s: %v", language, err)
            continue
        }

        m.vms[language] = vm
    }

    return nil
}

Language-Specific Execution

Each language requires different setup and execution commands:

func (m *Manager) executeInPersistentContainer(ctx context.Context, vm *LanguageVM, req ExecutionRequest) (*ExecutionResult, error) {
    var execCmd []string
    workspacePath := fmt.Sprintf("/tmp/workspaces/%s/%s", vm.Language, req.JobID)

    switch vm.Language {
    case "python":
        entryPoint := req.EntryPoint
        if entryPoint == "" {
            entryPoint = "main.py"
        }
        execCmd = []string{"docker", "exec", vm.ContainerName, "bash", "-c",
            fmt.Sprintf("cd %s && if [ -f requirements.txt ]; then pip install -r requirements.txt; fi && timeout 30 python %s", workspacePath, entryPoint)}

    case "nodejs":
        entryPoint := req.EntryPoint
        if entryPoint == "" {
            entryPoint = "index.js"
        }
        execCmd = []string{"docker", "exec", vm.ContainerName, "bash", "-c",
            fmt.Sprintf("cd %s && if [ -f package.json ]; then npm install; fi && timeout 30 node %s", workspacePath, entryPoint)}

    case "go":
        execCmd = []string{"docker", "exec", vm.ContainerName, "bash", "-c",
            fmt.Sprintf("cd %s && go mod tidy 2>/dev/null || true && timeout 30 go run .", workspacePath)}

    case "rust":
        execCmd = []string{"docker", "exec", vm.ContainerName, "bash", "-c",
            fmt.Sprintf("cd %s && timeout 30 cargo run", workspacePath)}

    case "java":
        entryPoint := req.EntryPoint
        if entryPoint == "" {
            entryPoint = "Main.java"
        }
        className := strings.TrimSuffix(entryPoint, ".java")
        execCmd = []string{"docker", "exec", vm.ContainerName, "bash", "-c",
            fmt.Sprintf("cd %s && javac %s && timeout 30 java %s", workspacePath, entryPoint, className)}
    }

    // Execute with timeout
    execCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
    defer cancel()
    cmd := exec.CommandContext(execCtx, execCmd[0], execCmd[1:]...)

    output, err := cmd.CombinedOutput()

    result := &ExecutionResult{
        JobID:       req.JobID,
        WorkspaceID: req.WorkspaceID,
        Output:      string(output),
        ExitCode:    0,
    }

    if err != nil {
        result.Error = err.Error()
        result.ExitCode = 1
    }

    return result, nil
}

Key Implementation Details:

Workspace Isolation: Each job gets its own directory within the container
Dependency Management: Automatic installation of requirements.txt, package.json, etc.
Timeout Protection: 30-second execution limit prevents infinite loops
Error Handling: Capture both stdout and stderr for complete output

Message Queue Integration

Why RabbitMQ for Code Execution?

Code execution jobs have specific requirements that make RabbitMQ ideal:

Reliability: Jobs must not be lost if a worker crashes
Durability: Job queue survives server restarts
Fair Distribution: Distribute jobs evenly across worker instances
Dead Letter Queues: Handle failed jobs gracefully
Priority Queues: Support urgent job execution

RabbitMQ Topology

Our message queue setup uses a topic exchange for flexible routing:

# Exchange Configuration
exchanges:
  jobs:
    name: "code-execution.jobs"
    type: "topic"
    durable: true
    auto_delete: false
  results:
    name: "code-execution.results"
    type: "topic"
    durable: true
    auto_delete: false
  dead_letter:
    name: "code-execution.dead-letter"
    type: "direct"
    durable: true
    auto_delete: false

# Queue Configuration
queues:
  job_prefix: "jobs"
  result_prefix: "results"
  dead_letter_suffix: "dlq"

Routing Keys Pattern:

jobs.python - Python execution jobs
jobs.nodejs - Node.js execution jobs
jobs.go - Go execution jobs
results.python - Python execution results
results.nodejs - Node.js execution results

Job Message Format

Standardized job messages ensure compatibility across services:

interface ExecutionJob {
  job_id: string;           // Unique identifier
  workspace_id: string;     // User workspace
  language: string;         // Target language
  source_code: string;      // Code to execute
  entry_point?: string;     // Main file (optional)
  dependencies?: string[];  // Package dependencies
  timeout?: number;         // Execution timeout
  memory_limit?: number;    // Memory limit in MB
}

Producer Implementation (Node.js Backend)

const amqp = require('amqplib');

class JobProducer {
  constructor(rabbitmqUrl) {
    this.rabbitmqUrl = rabbitmqUrl;
    this.connection = null;
    this.channel = null;
  }

  async connect() {
    this.connection = await amqp.connect(this.rabbitmqUrl);
    this.channel = await this.connection.createChannel();

    // Declare exchanges
    await this.channel.assertExchange('code-execution.jobs', 'topic', {
      durable: true
    });

    await this.channel.assertExchange('code-execution.results', 'topic', {
      durable: true
    });
  }

  async submitJob(job) {
    const routingKey = `jobs.${job.language}`;
    const jobMessage = {
      job_id: job.job_id,
      workspace_id: job.workspace_id,
      language: job.language,
      url: this.createDataURL(job.source_code, job.entry_point),
      entry_point: job.entry_point
    };

    await this.channel.publish(
      'code-execution.jobs',
      routingKey,
      Buffer.from(JSON.stringify(jobMessage)),
      {
        persistent: true,
        messageId: job.job_id,
        timestamp: Date.now()
      }
    );

    console.log(`Job ${job.job_id} submitted for ${job.language}`);
  }

  createDataURL(sourceCode, filename = 'main.py') {
    // Create inline data URL for source code
    return `data:text/plain;filename=${filename},${sourceCode}`;
  }

  async close() {
    if (this.channel) await this.channel.close();
    if (this.connection) await this.connection.close();
  }
}

Consumer Implementation (Go Runner Service)

func (m *Manager) consumeRabbitMQMessages(config RabbitMQConfig) error {
    conn, err := amqp.Dial(config.URL)
    if err != nil {
        return fmt.Errorf("failed to connect to RabbitMQ: %w", err)
    }
    defer conn.Close()

    ch, err := conn.Channel()
    if err != nil {
        return fmt.Errorf("failed to open channel: %w", err)
    }
    defer ch.Close()

    // Declare unified queue for all languages
    jobQueue, err := ch.QueueDeclare(
        "jobs.all-languages", // name
        true,                 // durable
        false,                // delete when unused
        false,                // exclusive
        false,                // no-wait
        nil,                  // arguments
    )
    if err != nil {
        return fmt.Errorf("failed to declare job queue: %w", err)
    }

    // Bind queue to exchange for each language
    languages := []string{"python", "nodejs", "go", "rust", "java"}
    for _, lang := range languages {
        err = ch.QueueBind(
            jobQueue.Name,
            fmt.Sprintf("jobs.%s", lang),
            "code-execution.jobs",
            false,
            nil,
        )
        if err != nil {
            return fmt.Errorf("failed to bind queue for %s: %w", lang, err)
        }
    }

    // Set QoS for fair distribution
    err = ch.Qos(1, 0, false)
    if err != nil {
        return fmt.Errorf("failed to set QoS: %w", err)
    }

    // Start consuming
    msgs, err := ch.Consume(
        jobQueue.Name,
        "",    // consumer tag
        false, // auto-ack
        false, // exclusive
        false, // no-local
        false, // no-wait
        nil,   // args
    )
    if err != nil {
        return fmt.Errorf("failed to register consumer: %w", err)
    }

    for msg := range msgs {
        go m.processJob(msg)
    }

    return nil
}

func (m *Manager) processJob(msg amqp.Delivery) {
    var execReq ExecutionRequest
    if err := json.Unmarshal(msg.Body, &execReq); err != nil {
        m.logger.WithError(err).Error("Failed to parse job message")
        msg.Nack(false, false) // Don't requeue invalid messages
        return
    }

    // Execute the job
    ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
    result, err := m.ExecuteCode(ctx, execReq)
    cancel()

    if err != nil {
        m.logger.WithError(err).Error("Job execution failed")
        result = &ExecutionResult{
            JobID:    execReq.JobID,
            Output:   "",
            Error:    err.Error(),
            ExitCode: 1,
        }
    }

    // Publish result
    resultBytes, _ := json.Marshal(result)
    err = msg.Channel.Publish(
        "code-execution.results",
        fmt.Sprintf("results.%s", execReq.Language),
        false, // mandatory
        false, // immediate
        amqp.Publishing{
            ContentType: "application/json",
            Body:        resultBytes,
        })

    if err != nil {
        m.logger.WithError(err).Error("Failed to publish result")
        msg.Nack(false, true) // Requeue on publish failure
    } else {
        msg.Ack(false) // Acknowledge successful processing
    }
}

Real-time Streaming

The Challenge of Live Output

Unlike traditional job systems that return results after completion, code execution benefits from real-time output streaming:

User Experience: Immediate feedback as code runs
Long-Running Jobs: Show progress for lengthy operations
Debugging: See output line-by-line for easier debugging
Interactive Input: Support for programs requiring user input

Redis Pub/Sub for Streaming

Redis provides excellent pub/sub capabilities for real-time streaming:

// Redis streaming setup
const redis = require('redis');

class StreamingService {
  constructor(redisUrl) {
    this.publisher = redis.createClient({ url: redisUrl });
    this.subscriber = redis.createClient({ url: redisUrl });
  }

  async connect() {
    await this.publisher.connect();
    await this.subscriber.connect();
  }

  // Publish output line from runner service
  async publishOutput(jobId, line, stream = 'stdout') {
    const message = {
      job_id: jobId,
      line: line,
      stream: stream,
      timestamp: new Date().toISOString()
    };

    await this.publisher.publish(
      `execution:${jobId}`,
      JSON.stringify(message)
    );
  }

  // Subscribe to job output
  async subscribeToJob(jobId, callback) {
    await this.subscriber.subscribe(`execution:${jobId}`, (message) => {
      const data = JSON.parse(message);
      callback(data);
    });
  }

  async unsubscribeFromJob(jobId) {
    await this.subscriber.unsubscribe(`execution:${jobId}`);
  }
}

WebSocket Integration

Connect Redis streams to frontend via WebSocket:

// WebSocket server integration
const WebSocket = require('ws');

class WebSocketManager {
  constructor(server, streamingService) {
    this.wss = new WebSocket.Server({ server });
    this.streamingService = streamingService;
    this.connections = new Map(); // jobId -> Set of WebSocket connections

    this.setupWebSocketHandlers();
  }

  setupWebSocketHandlers() {
    this.wss.on('connection', (ws) => {
      ws.on('message', async (data) => {
        const message = JSON.parse(data);

        switch (message.type) {
          case 'subscribe':
            await this.subscribeToJobOutput(ws, message.job_id);
            break;
          case 'unsubscribe':
            await this.unsubscribeFromJobOutput(ws, message.job_id);
            break;
        }
      });

      ws.on('close', () => {
        this.cleanupConnection(ws);
      });
    });
  }

  async subscribeToJobOutput(ws, jobId) {
    // Add connection to job subscription
    if (!this.connections.has(jobId)) {
      this.connections.set(jobId, new Set());

      // Subscribe to Redis stream for this job
      await this.streamingService.subscribeToJob(jobId, (data) => {
        // Broadcast to all subscribed WebSocket connections
        const connections = this.connections.get(jobId);
        if (connections) {
          connections.forEach(conn => {
            if (conn.readyState === WebSocket.OPEN) {
              conn.send(JSON.stringify({
                type: 'output',
                data: data
              }));
            }
          });
        }
      });
    }

    this.connections.get(jobId).add(ws);
  }

  async unsubscribeFromJobOutput(ws, jobId) {
    const connections = this.connections.get(jobId);
    if (connections) {
      connections.delete(ws);

      // If no more connections, unsubscribe from Redis
      if (connections.size === 0) {
        await this.streamingService.unsubscribeFromJob(jobId);
        this.connections.delete(jobId);
      }
    }
  }

  cleanupConnection(ws) {
    // Remove connection from all job subscriptions
    for (const [jobId, connections] of this.connections.entries()) {
      connections.delete(ws);
      if (connections.size === 0) {
        this.streamingService.unsubscribeFromJob(jobId);
        this.connections.delete(jobId);
      }
    }
  }
}

Enhanced Runner with Streaming

Modify the runner service to stream output line-by-line:

func (m *Manager) executeWithStreaming(ctx context.Context, vm *LanguageVM, req ExecutionRequest) (*ExecutionResult, error) {
    // ... command setup ...

    cmd := exec.CommandContext(execCtx, execCmd[0], execCmd[1:]...)

    // Create pipes for real-time output capture
    stdout, err := cmd.StdoutPipe()
    if err != nil {
        return nil, fmt.Errorf("failed to create stdout pipe: %w", err)
    }

    stderr, err := cmd.StderrPipe()
    if err != nil {
        return nil, fmt.Errorf("failed to create stderr pipe: %w", err)
    }

    // Start the command
    if err := cmd.Start(); err != nil {
        return nil, fmt.Errorf("failed to start command: %w", err)
    }

    // Stream output in real-time
    var outputBuffer strings.Builder
    var wg sync.WaitGroup

    wg.Add(2)

    // Stream stdout
    go func() {
        defer wg.Done()
        scanner := bufio.NewScanner(stdout)
        for scanner.Scan() {
            line := scanner.Text()
            outputBuffer.WriteString(line + "\n")

            // Publish to Redis for real-time streaming
            m.publishStreamingOutput(req.JobID, line, "stdout")
        }
    }()

    // Stream stderr
    go func() {
        defer wg.Done()
        scanner := bufio.NewScanner(stderr)
        for scanner.Scan() {
            line := scanner.Text()
            outputBuffer.WriteString(line + "\n")

            // Publish to Redis for real-time streaming
            m.publishStreamingOutput(req.JobID, line, "stderr")
        }
    }()

    // Wait for command completion
    err = cmd.Wait()
    wg.Wait() // Wait for all output to be processed

    result := &ExecutionResult{
        JobID:       req.JobID,
        WorkspaceID: req.WorkspaceID,
        Output:      outputBuffer.String(),
        ExitCode:    0,
    }

    if err != nil {
        result.Error = err.Error()
        result.ExitCode = 1
    }

    return result, nil
}

func (m *Manager) publishStreamingOutput(jobID, line, stream string) {
    // This would integrate with your Redis client
    message := StreamingOutput{
        JobID:     jobID,
        Line:      line,
        Stream:    stream,
        Timestamp: time.Now(),
    }

    // Publish to Redis (implementation depends on your Redis client)
    // m.redisClient.Publish(fmt.Sprintf("execution:%s", jobID), message)
}

This streaming approach provides real-time feedback to users, making the code execution feel immediate and interactive rather than a black-box operation.

Backend Implementation

Node.js API Server

The backend serves as the orchestration layer between the frontend and the execution infrastructure:

const express = require('express');
const WebSocket = require('ws');
const { v4: uuidv4 } = require('uuid');
const cors = require('cors');

class CodeExecutionAPI {
  constructor() {
    this.app = express();
    this.server = null;
    this.jobProducer = null;
    this.streamingService = null;
    this.wsManager = null;
    this.activeJobs = new Map(); // Track running jobs

    this.setupMiddleware();
    this.setupRoutes();
  }

  setupMiddleware() {
    this.app.use(cors());
    this.app.use(express.json({ limit: '10mb' }));
    this.app.use(express.urlencoded({ extended: true }));

    // Request logging
    this.app.use((req, res, next) => {
      console.log(`${req.method} ${req.path} - ${new Date().toISOString()}`);
      next();
    });
  }

  setupRoutes() {
    // Health check
    this.app.get('/health', (req, res) => {
      res.json({ status: 'ok', timestamp: new Date().toISOString() });
    });

    // Submit code execution job
    this.app.post('/api/execute', async (req, res) => {
      try {
        const result = await this.handleCodeExecution(req.body);
        res.json(result);
      } catch (error) {
        console.error('Execution error:', error);
        res.status(500).json({ 
          error: 'Execution failed', 
          message: error.message 
        });
      }
    });

    // Get job status
    this.app.get('/api/jobs/:jobId', (req, res) => {
      const jobId = req.params.jobId;
      const job = this.activeJobs.get(jobId);

      if (!job) {
        return res.status(404).json({ error: 'Job not found' });
      }

      res.json(job);
    });

    // List active jobs
    this.app.get('/api/jobs', (req, res) => {
      const jobs = Array.from(this.activeJobs.values());
      res.json({ jobs, count: jobs.length });
    });

    // Cancel job
    this.app.delete('/api/jobs/:jobId', async (req, res) => {
      const jobId = req.params.jobId;
      await this.cancelJob(jobId);
      res.json({ message: 'Job cancelled' });
    });
  }

  async handleCodeExecution(requestBody) {
    const {
      language,
      source_code,
      entry_point,
      workspace_id = 'default',
      timeout = 30
    } = requestBody;

    // Validate request
    if (!language || !source_code) {
      throw new Error('Language and source_code are required');
    }

    const supportedLanguages = ['python', 'nodejs', 'go', 'rust', 'java'];
    if (!supportedLanguages.includes(language)) {
      throw new Error(`Unsupported language: ${language}`);
    }

    // Generate unique job ID
    const jobId = uuidv4();

    // Create job record
    const job = {
      job_id: jobId,
      workspace_id,
      language,
      source_code,
      entry_point,
      timeout,
      status: 'queued',
      created_at: new Date().toISOString(),
      updated_at: new Date().toISOString()
    };

    this.activeJobs.set(jobId, job);

    // Submit to RabbitMQ
    await this.jobProducer.submitJob(job);

    // Update status
    job.status = 'submitted';
    job.updated_at = new Date().toISOString();

    return {
      job_id: jobId,
      status: 'submitted',
      message: 'Job submitted for execution',
      stream_url: `/stream/${jobId}`
    };
  }

  async cancelJob(jobId) {
    const job = this.activeJobs.get(jobId);
    if (job && job.status === 'running') {
      // Send cancellation signal (implementation depends on your setup)
      job.status = 'cancelled';
      job.updated_at = new Date().toISOString();
    }
  }

  async start(port = 3001) {
    // Initialize services
    await this.initializeServices();

    // Start HTTP server
    this.server = this.app.listen(port, () => {
      console.log(`Code execution API running on port ${port}`);
    });

    // Setup WebSocket manager
    this.wsManager = new WebSocketManager(this.server, this.streamingService);

    // Setup result consumer
    this.setupResultConsumer();
  }

  async initializeServices() {
    // Initialize RabbitMQ producer
    this.jobProducer = new JobProducer(process.env.RABBITMQ_URL);
    await this.jobProducer.connect();

    // Initialize Redis streaming
    this.streamingService = new StreamingService(process.env.REDIS_URL);
    await this.streamingService.connect();

    console.log('All services initialized successfully');
  }

  setupResultConsumer() {
    // Consumer for job results from RabbitMQ
    const amqp = require('amqplib');

    amqp.connect(process.env.RABBITMQ_URL)
      .then(conn => conn.createChannel())
      .then(ch => {
        // Declare results queue
        return ch.assertQueue('results.all', { durable: true })
          .then(() => {
            // Bind to results exchange
            return ch.bindQueue('results.all', 'code-execution.results', 'results.*');
          })
          .then(() => {
            // Consume results
            return ch.consume('results.all', (msg) => {
              if (msg) {
                this.handleJobResult(JSON.parse(msg.content.toString()));
                ch.ack(msg);
              }
            });
          });
      })
      .catch(console.error);
  }

  handleJobResult(result) {
    const job = this.activeJobs.get(result.job_id);
    if (job) {
      job.status = result.exit_code === 0 ? 'completed' : 'failed';
      job.result = result;
      job.updated_at = new Date().toISOString();

      // Optionally clean up completed jobs after some time
      setTimeout(() => {
        this.activeJobs.delete(result.job_id);
      }, 300000); // 5 minutes
    }
  }

  async stop() {
    if (this.server) {
      this.server.close();
    }
    if (this.jobProducer) {
      await this.jobProducer.close();
    }
    if (this.streamingService) {
      await this.streamingService.close();
    }
  }
}

// Environment configuration
const config = {
  port: process.env.PORT || 3001,
  rabbitmq_url: process.env.RABBITMQ_URL || 'amqp://admin:admin123@localhost:5672',
  redis_url: process.env.REDIS_URL || 'redis://localhost:6379'
};

// Start the server
const api = new CodeExecutionAPI();
api.start(config.port);

// Graceful shutdown
process.on('SIGINT', async () => {
  console.log('Shutting down gracefully...');
  await api.stop();
  process.exit(0);
});

Express Middleware for Code Validation

Add security and validation middleware:

const rateLimit = require('express-rate-limit');
const validator = require('validator');

// Rate limiting for code execution
const executionLimiter = rateLimit({
  windowMs: 60 * 1000, // 1 minute
  max: 10, // Maximum 10 executions per minute per IP
  message: 'Too many execution requests, please try again later',
  standardHeaders: true,
  legacyHeaders: false
});

// Code validation middleware
const validateCodeRequest = (req, res, next) => {
  const { language, source_code, entry_point } = req.body;

  // Check required fields
  if (!language || !source_code) {
    return res.status(400).json({
      error: 'Missing required fields',
      required: ['language', 'source_code']
    });
  }

  // Validate language
  const supportedLanguages = ['python', 'nodejs', 'go', 'rust', 'java'];
  if (!supportedLanguages.includes(language)) {
    return res.status(400).json({
      error: 'Unsupported language',
      supported: supportedLanguages
    });
  }

  // Check code length (prevent abuse)
  if (source_code.length > 100000) { // 100KB limit
    return res.status(400).json({
      error: 'Source code too large',
      max_size: '100KB'
    });
  }

  // Validate entry point if provided
  if (entry_point && !validator.isAlphanumeric(entry_point.replace(/[._-]/g, ''))) {
    return res.status(400).json({
      error: 'Invalid entry point format'
    });
  }

  // Basic security checks (can be expanded)
  const dangerousPatterns = [
    /rm\s+-rf/,
    /sudo/,
    /passwd/,
    /\/etc\/passwd/,
    /mkfs/,
    /format/,
    /del\s+\/[a-z]/i
  ];

  for (const pattern of dangerousPatterns) {
    if (pattern.test(source_code)) {
      return res.status(400).json({
        error: 'Code contains potentially dangerous operations'
      });
    }
  }

  next();
};

// Apply middleware to execution endpoint
app.post('/api/execute', 
  executionLimiter, 
  validateCodeRequest, 
  async (req, res) => {
    // ... execution logic
  }
);

Frontend Development

React Code Execution Component

Based on the CodeRunnerDemo.tsx, here's a production-ready implementation:

import React, { useState, useEffect, useCallback, useRef } from 'react';
import { Button } from '@/components/ui/button';
import { Card, CardContent } from '@/components/ui/card';
import { Badge } from '@/components/ui/badge';
import { Tabs, TabsList, TabsTrigger, TabsContent } from '@/components/ui/tabs';
import { Play, Square, Copy, Download, Settings } from 'lucide-react';
import CodeMirror from '@uiw/react-codemirror';
import { githubDark } from '@uiw/codemirror-theme-github';
import { javascript } from '@codemirror/lang-javascript';
import { python } from '@codemirror/lang-python';
import { rust } from '@codemirror/lang-rust';
import { go } from '@codemirror/lang-go';
import { java } from '@codemirror/lang-java';

interface ExecutionResult {
  job_id: string;
  status: string;
  output?: string;
  error?: string;
  exit_code?: number;
  duration?: number;
}

interface OutputLine {
  line: string;
  stream: 'stdout' | 'stderr';
  timestamp: string;
}

const LANGUAGES = {
  python: { ext: python(), icon: '🐍', name: 'Python', starter: 'print("Hello, World!")' },
  javascript: { ext: javascript(), icon: '⚡', name: 'JavaScript', starter: 'console.log("Hello, World!");' },
  go: { ext: go(), icon: '🔵', name: 'Go', starter: 'package main\n\nimport "fmt"\n\nfunc main() {\n    fmt.Println("Hello, World!")\n}' },
  rust: { ext: rust(), icon: '🦀', name: 'Rust', starter: 'fn main() {\n    println!("Hello, World!");\n}' },
  java: { ext: java(), icon: '☕', name: 'Java', starter: 'public class Main {\n    public static void main(String[] args) {\n        System.out.println("Hello, World!");\n    }\n}' }
};

export function CodeExecutor() {
  const [selectedLanguage, setSelectedLanguage] = useState('python');
  const [code, setCode] = useState(LANGUAGES.python.starter);
  const [isExecuting, setIsExecuting] = useState(false);
  const [output, setOutput] = useState<OutputLine[]>([]);
  const [executionResult, setExecutionResult] = useState<ExecutionResult | null>(null);
  const [currentJobId, setCurrentJobId] = useState<string | null>(null);

  const wsRef = useRef<WebSocket | null>(null);
  const outputRef = useRef<HTMLDivElement>(null);

  // WebSocket connection for real-time output
  const connectWebSocket = useCallback((jobId: string) => {
    const wsUrl = `ws://localhost:3001/stream/${jobId}`;
    wsRef.current = new WebSocket(wsUrl);

    wsRef.current.onopen = () => {
      console.log('WebSocket connected for job:', jobId);
      wsRef.current?.send(JSON.stringify({
        type: 'subscribe',
        job_id: jobId
      }));
    };

    wsRef.current.onmessage = (event) => {
      const message = JSON.parse(event.data);

      if (message.type === 'output') {
        const outputLine: OutputLine = {
          line: message.data.line,
          stream: message.data.stream,
          timestamp: message.data.timestamp
        };

        setOutput(prev => [...prev, outputLine]);

        // Auto-scroll to bottom
        setTimeout(() => {
          if (outputRef.current) {
            outputRef.current.scrollTop = outputRef.current.scrollHeight;
          }
        }, 10);
      }
    };

    wsRef.current.onclose = () => {
      console.log('WebSocket disconnected');
    };

    wsRef.current.onerror = (error) => {
      console.error('WebSocket error:', error);
    };
  }, []);

  // Execute code
  const executeCode = useCallback(async () => {
    if (isExecuting) return;

    setIsExecuting(true);
    setOutput([]);
    setExecutionResult(null);

    try {
      const response = await fetch('http://localhost:3001/api/execute', {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json'
        },
        body: JSON.stringify({
          language: selectedLanguage,
          source_code: code,
          workspace_id: 'web-editor'
        })
      });

      if (!response.ok) {
        throw new Error(`HTTP ${response.status}: ${response.statusText}`);
      }

      const result = await response.json();
      setCurrentJobId(result.job_id);

      // Connect WebSocket for streaming output
      connectWebSocket(result.job_id);

      // Poll for final result
      pollJobResult(result.job_id);

    } catch (error) {
      console.error('Execution failed:', error);
      setOutput([{
        line: `Error: ${error.message}`,
        stream: 'stderr',
        timestamp: new Date().toISOString()
      }]);
      setIsExecuting(false);
    }
  }, [code, selectedLanguage, isExecuting, connectWebSocket]);

  // Poll for job completion
  const pollJobResult = useCallback(async (jobId: string) => {
    const pollInterval = 1000; // 1 second
    const maxPolls = 60; // 60 seconds timeout
    let polls = 0;

    const poll = async () => {
      try {
        const response = await fetch(`http://localhost:3001/api/jobs/${jobId}`);
        const job = await response.json();

        if (job.status === 'completed' || job.status === 'failed') {
          setExecutionResult(job.result);
          setIsExecuting(false);

          // Close WebSocket
          if (wsRef.current) {
            wsRef.current.close();
          }

          return;
        }

        polls++;
        if (polls < maxPolls) {
          setTimeout(poll, pollInterval);
        } else {
          // Timeout
          setIsExecuting(false);
          setOutput(prev => [...prev, {
            line: 'Execution timeout - job may still be running',
            stream: 'stderr',
            timestamp: new Date().toISOString()
          }]);
        }
      } catch (error) {
        console.error('Polling error:', error);
        setIsExecuting(false);
      }
    };

    setTimeout(poll, pollInterval);
  }, []);

  // Stop execution
  const stopExecution = useCallback(async () => {
    if (currentJobId) {
      try {
        await fetch(`http://localhost:3001/api/jobs/${currentJobId}`, {
          method: 'DELETE'
        });
      } catch (error) {
        console.error('Failed to cancel job:', error);
      }
    }

    if (wsRef.current) {
      wsRef.current.close();
    }

    setIsExecuting(false);
    setCurrentJobId(null);
  }, [currentJobId]);

  // Copy code to clipboard
  const copyCode = useCallback(() => {
    navigator.clipboard.writeText(code);
  }, [code]);

  // Download output
  const downloadOutput = useCallback(() => {
    const outputText = output.map(line => 
      `[${line.timestamp}] ${line.stream}: ${line.line}`
    ).join('\n');

    const blob = new Blob([outputText], { type: 'text/plain' });
    const url = URL.createObjectURL(blob);
    const a = document.createElement('a');
    a.href = url;
    a.download = `execution-output-${Date.now()}.txt`;
    a.click();
    URL.revokeObjectURL(url);
  }, [output]);

  // Language change handler
  useEffect(() => {
    setCode(LANGUAGES[selectedLanguage as keyof typeof LANGUAGES].starter);
  }, [selectedLanguage]);

  // Cleanup WebSocket on unmount
  useEffect(() => {
    return () => {
      if (wsRef.current) {
        wsRef.current.close();
      }
    };
  }, []);

  return (
    <div className="max-w-6xl mx-auto p-4 space-y-4">
      {/* Header */}
      <div className="flex items-center justify-between">
        <h1 className="text-2xl font-bold">Online Code Compiler</h1>
        <Badge variant="outline" className="text-sm">
          Real-time execution with Docker containers
        </Badge>
      </div>

      {/* Language Selection */}
      <Card>
        <CardContent className="p-4">
          <Tabs value={selectedLanguage} onValueChange={setSelectedLanguage}>
            <TabsList className="grid w-full grid-cols-5">
              {Object.entries(LANGUAGES).map(([lang, config]) => (
                <TabsTrigger key={lang} value={lang} className="flex items-center gap-2">
                  <span>{config.icon}</span>
                  <span className="hidden sm:inline">{config.name}</span>
                </TabsTrigger>
              ))}
            </TabsList>
          </Tabs>
        </CardContent>
      </Card>

      {/* Code Editor */}
      <Card>
        <CardContent className="p-0">
          <div className="border-b p-4 flex items-center justify-between">
            <h3 className="font-semibold">Code Editor</h3>
            <div className="flex items-center gap-2">
              <Button
                variant="outline"
                size="sm"
                onClick={copyCode}
              >
                <Copy className="h-4 w-4 mr-2" />
                Copy
              </Button>
              <Button
                onClick={isExecuting ? stopExecution : executeCode}
                disabled={!code.trim()}
                variant={isExecuting ? "destructive" : "default"}
              >
                {isExecuting ? (
                  <>
                    <Square className="h-4 w-4 mr-2" />
                    Stop
                  </>
                ) : (
                  <>
                    <Play className="h-4 w-4 mr-2" />
                    Run {LANGUAGES[selectedLanguage as keyof typeof LANGUAGES].name}
                  </>
                )}
              </Button>
            </div>
          </div>

          <CodeMirror
            value={code}
            onChange={(value) => setCode(value)}
            theme={githubDark}
            extensions={[LANGUAGES[selectedLanguage as keyof typeof LANGUAGES].ext]}
            className="text-sm"
            basicSetup={{
              lineNumbers: true,
              foldGutter: true,
              dropCursor: false,
              allowMultipleSelections: false
            }}
          />
        </CardContent>
      </Card>

      {/* Output Panel */}
      <Card>
        <CardContent className="p-0">
          <div className="border-b p-4 flex items-center justify-between">
            <div className="flex items-center gap-2">
              <h3 className="font-semibold">Output</h3>
              {isExecuting && (
                <Badge variant="secondary" className="animate-pulse">
                  Executing...
                </Badge>
              )}
              {executionResult && (
                <Badge 
                  variant={executionResult.exit_code === 0 ? "default" : "destructive"}
                >
                  Exit code: {executionResult.exit_code}
                </Badge>
              )}
            </div>
            <div className="flex items-center gap-2">
              {output.length > 0 && (
                <Button
                  variant="outline"
                  size="sm"
                  onClick={downloadOutput}
                >
                  <Download className="h-4 w-4 mr-2" />
                  Download
                </Button>
              )}
              <Button
                variant="outline"
                size="sm"
                onClick={() => setOutput([])}
              >
                Clear
              </Button>
            </div>
          </div>

          <div 
            ref={outputRef}
            className="h-96 overflow-y-auto p-4 bg-gray-950 text-gray-100 font-mono text-sm"
          >
            {output.length === 0 ? (
              <div className="text-gray-500 italic">
                Click "Run" to execute your code. Output will appear here in real-time.
              </div>
            ) : (
              output.map((line, index) => (
                <div 
                  key={index} 
                  className={`whitespace-pre-wrap ${
                    line.stream === 'stderr' ? 'text-red-400' : 'text-gray-100'
                  }`}
                >
                  {line.line}
                </div>
              ))
            )}
          </div>
        </CardContent>
      </Card>

      {/* Execution Statistics */}
      {executionResult && (
        <Card>
          <CardContent className="p-4">
            <h3 className="font-semibold mb-2">Execution Statistics</h3>
            <div className="grid grid-cols-2 md:grid-cols-4 gap-4 text-sm">
              <div>
                <span className="text-gray-500">Duration:</span>
                <div className="font-mono">{executionResult.duration || 0}ms</div>
              </div>
              <div>
                <span className="text-gray-500">Exit Code:</span>
                <div className="font-mono">{executionResult.exit_code}</div>
              </div>
              <div>
                <span className="text-gray-500">Language:</span>
                <div className="font-mono">{selectedLanguage}</div>
              </div>
              <div>
                <span className="text-gray-500">Job ID:</span>
                <div className="font-mono text-xs">{executionResult.job_id}</div>
              </div>
            </div>
          </CardContent>
        </Card>
      )}
    </div>
  );
}

Security & Isolation

The Security Challenge

Running arbitrary user code presents significant security risks:

Host System Access: Malicious code could access the host filesystem
Network Attacks: Code could scan internal networks or launch attacks
Resource Exhaustion: Infinite loops or memory bombs could crash servers
Data Exfiltration: Code could attempt to steal sensitive information
Privilege Escalation: Attempts to gain root or admin access

Docker Security Model

Docker provides multiple layers of security isolation:

# Docker container security configuration
version: '3.8'
services:
  runner-python:
    image: python:3.11-slim
    security_opt:
      - no-new-privileges:true
      - seccomp:unconfined  # May need custom seccomp profile
    cap_drop:
      - ALL
    cap_add:
      - SETGID
      - SETUID
    read_only: true
    tmpfs:
      - /tmp:exec,size=100m
      - /var/tmp:exec,size=100m
    ulimits:
      nproc: 64        # Limit number of processes
      nofile: 1024     # Limit open files
      memlock: 67108864 # Limit locked memory
    memory: 256m       # Memory limit
    cpus: '0.5'       # CPU limit
    pids_limit: 100   # Process limit
    networks:
      - isolated_network

networks:
  isolated_network:
    driver: bridge
    internal: true  # No external network access

Advanced Container Security

Implement additional security measures in the runner service:

func (m *Manager) createSecureContainer(language, jobID string) error {
    containerConfig := &container.Config{
        Image: m.config.Environments[language],
        Cmd:   []string{"tail", "-f", "/dev/null"},
        Env: []string{
            "HOME=/tmp",
            "USER=runner",
            "SHELL=/bin/bash",
        },
        WorkingDir: "/tmp/workspace",
        User:       "1000:1000", // Non-root user

        // Resource limits
        Memory:     256 * 1024 * 1024, // 256MB
        MemorySwap: 256 * 1024 * 1024, // No swap
        CpuShares:  512,                // Half CPU priority

        // Security options
        SecurityOpts: []string{
            "no-new-privileges:true",
            "seccomp=unconfined", // or custom profile
        },

        // Network isolation
        NetworkDisabled: true,
    }

    hostConfig := &container.HostConfig{
        // Resource constraints
        Resources: container.Resources{
            Memory:     256 * 1024 * 1024,
            MemorySwap: 256 * 1024 * 1024,
            CPUShares:  512,
            PidsLimit:  50,
            Ulimits: []*units.Ulimit{
                {Name: "nproc", Soft: 32, Hard: 32},
                {Name: "nofile", Soft: 256, Hard: 256},
            },
        },

        // Security
        ReadonlyRootfs: true,
        CapDrop:        []string{"ALL"},
        CapAdd:         []string{"SETGID", "SETUID"},

        // Temporary filesystems
        Tmpfs: map[string]string{
            "/tmp":     "exec,size=100m",
            "/var/tmp": "exec,size=100m",
        },

        // No privileged access
        Privileged: false,

        // Network isolation
        NetworkMode: "none",
    }

    // Create container
    resp, err := m.dockerClient.ContainerCreate(
        context.Background(),
        containerConfig,
        hostConfig,
        nil, // networking config
        nil, // platform
        fmt.Sprintf("runner-%s-%s", language, jobID),
    )

    if err != nil {
        return fmt.Errorf("failed to create container: %w", err)
    }

    // Start container
    if err := m.dockerClient.ContainerStart(
        context.Background(),
        resp.ID,
        types.ContainerStartOptions{},
    ); err != nil {
        return fmt.Errorf("failed to start container: %w", err)
    }

    return nil
}

Code Sanitization

Implement static analysis for dangerous patterns:

class CodeSanitizer {
  constructor() {
    this.dangerousPatterns = {
      filesystem: [
        /\bopen\s*\(\s*['"][\/\\]/,     // File system access
        /\bfile\s*\(\s*['"][\/\\]/,     // File operations
        /\bos\.system/,                  // OS commands
        /\bsubprocess/,                  // Process execution
        /\beval\s*\(/,                   // Code evaluation
        /\bexec\s*\(/,                   // Code execution
      ],
      network: [
        /\bsocket\s*\(/,                 // Network sockets
        /\burllib/,                      // URL operations
        /\brequests\./,                  // HTTP requests
        /\bhttplib/,                     // HTTP library
        /\bfetch\s*\(/,                  // Fetch API
      ],
      system: [
        /\bos\.getenv/,                  // Environment variables
        /\bprocess\.env/,                // Node.js environment
        /\b__import__/,                  // Dynamic imports
        /\brequire\s*\(\s*['"]child_process['"]/, // Child process
      ]
    };
  }

  analyze(code, language) {
    const issues = [];

    for (const [category, patterns] of Object.entries(this.dangerousPatterns)) {
      for (const pattern of patterns) {
        const matches = code.match(pattern);
        if (matches) {
          issues.push({
            category,
            pattern: pattern.toString(),
            match: matches[0],
            severity: this.getSeverity(category)
          });
        }
      }
    }

    return {
      safe: issues.length === 0,
      issues,
      score: this.calculateSafetyScore(issues)
    };
  }

  getSeverity(category) {
    const severityMap = {
      filesystem: 'high',
      network: 'medium',
      system: 'high'
    };
    return severityMap[category] || 'low';
  }

  calculateSafetyScore(issues) {
    const weights = { high: 10, medium: 5, low: 1 };
    const totalWeight = issues.reduce((sum, issue) => 
      sum + weights[issue.severity], 0);
    return Math.max(0, 100 - totalWeight);
  }
}

// Usage in API
app.post('/api/execute', validateCodeRequest, async (req, res) => {
  const sanitizer = new CodeSanitizer();
  const analysis = sanitizer.analyze(req.body.source_code, req.body.language);

  if (!analysis.safe && analysis.score < 50) {
    return res.status(400).json({
      error: 'Code contains potentially dangerous operations',
      issues: analysis.issues,
      safety_score: analysis.score
    });
  }

  // Proceed with execution...
});

Network Isolation

Implement network restrictions at multiple levels:

# Docker network setup with restrictions
docker network create --driver bridge \
  --subnet=172.20.0.0/16 \
  --opt com.docker.network.bridge.enable_icc=false \
  --opt com.docker.network.bridge.enable_ip_masquerade=false \
  isolated-execution

# Firewall rules for container network
iptables -I DOCKER-USER -s 172.20.0.0/16 -j DROP
iptables -I DOCKER-USER -s 172.20.0.0/16 -d 172.20.0.0/16 -j ACCEPT

Performance Optimization

Container Lifecycle Management

Optimize container startup and cleanup:

type ContainerPool struct {
    pools    map[string]*LanguagePool
    mutex    sync.RWMutex
    logger   *logrus.Logger
}

type LanguagePool struct {
    language     string
    containers   []*ContainerInstance
    available    chan *ContainerInstance
    maxSize      int
    currentSize  int
    mutex        sync.Mutex
}

type ContainerInstance struct {
    ID        string
    Language  string
    CreatedAt time.Time
    LastUsed  time.Time
    InUse     bool
}

func NewContainerPool(maxSize int, logger *logrus.Logger) *ContainerPool {
    return &ContainerPool{
        pools:  make(map[string]*LanguagePool),
        logger: logger,
    }
}

func (cp *ContainerPool) GetContainer(language string) (*ContainerInstance, error) {
    cp.mutex.RLock()
    pool, exists := cp.pools[language]
    cp.mutex.RUnlock()

    if !exists {
        cp.mutex.Lock()
        pool = &LanguagePool{
            language:  language,
            available: make(chan *ContainerInstance, 10),
            maxSize:   10,
        }
        cp.pools[language] = pool
        cp.mutex.Unlock()
    }

    // Try to get from available pool
    select {
    case container := <-pool.available:
        container.InUse = true
        container.LastUsed = time.Now()
        return container, nil
    default:
        // Create new container if under limit
        return cp.createNewContainer(pool)
    }
}

func (cp *ContainerPool) ReturnContainer(container *ContainerInstance) {
    cp.mutex.RLock()
    pool := cp.pools[container.Language]
    cp.mutex.RUnlock()

    container.InUse = false
    container.LastUsed = time.Now()

    // Clean container workspace
    cp.cleanContainerWorkspace(container)

    // Return to pool
    select {
    case pool.available <- container:
        // Successfully returned to pool
    default:
        // Pool is full, destroy container
        cp.destroyContainer(container)
    }
}

func (cp *ContainerPool) cleanContainerWorkspace(container *ContainerInstance) {
    // Execute cleanup commands in container
    cleanupCmd := []string{
        "docker", "exec", container.ID,
        "bash", "-c", "rm -rf /tmp/workspace/* 2>/dev/null || true"
    }

    exec.Command(cleanupCmd[0], cleanupCmd[1:]...).Run()
}

Memory Management

Implement intelligent memory management:

type MemoryManager struct {
    totalMemory    uint64
    usedMemory     uint64
    containerMem   map[string]uint64
    mutex          sync.RWMutex
    logger         *logrus.Logger
}

func (mm *MemoryManager) AllocateMemory(containerID string, requested uint64) error {
    mm.mutex.Lock()
    defer mm.mutex.Unlock()

    // Check if allocation would exceed limits
    if mm.usedMemory + requested > mm.totalMemory * 80 / 100 { // 80% threshold
        return fmt.Errorf("insufficient memory: %d MB requested, %d MB available", 
            requested/1024/1024, (mm.totalMemory-mm.usedMemory)/1024/1024)
    }

    mm.usedMemory += requested
    mm.containerMem[containerID] = requested

    mm.logger.Infof("Allocated %d MB to container %s", requested/1024/1024, containerID)
    return nil
}

func (mm *MemoryManager) ReleaseMemory(containerID string) {
    mm.mutex.Lock()
    defer mm.mutex.Unlock()

    if allocated, exists := mm.containerMem[containerID]; exists {
        mm.usedMemory -= allocated
        delete(mm.containerMem, containerID)
        mm.logger.Infof("Released %d MB from container %s", allocated/1024/1024, containerID)
    }
}

func (mm *MemoryManager) GetMemoryStats() map[string]interface{} {
    mm.mutex.RLock()
    defer mm.mutex.RUnlock()

    return map[string]interface{}{
        "total_mb":        mm.totalMemory / 1024 / 1024,
        "used_mb":         mm.usedMemory / 1024 / 1024,
        "available_mb":    (mm.totalMemory - mm.usedMemory) / 1024 / 1024,
        "utilization":     float64(mm.usedMemory) / float64(mm.totalMemory) * 100,
        "active_containers": len(mm.containerMem),
    }
}

Load Balancing

Implement intelligent load balancing:

type LoadBalancer struct {
    workers       []*WorkerNode
    roundRobin    int
    mutex         sync.Mutex
    healthChecker *HealthChecker
}

type WorkerNode struct {
    ID           string
    Address      string
    CPU          float64
    Memory       float64
    ActiveJobs   int
    MaxJobs      int
    LastSeen     time.Time
    Healthy      bool
}

func (lb *LoadBalancer) SelectWorker(job *ExecutionJob) (*WorkerNode, error) {
    lb.mutex.Lock()
    defer lb.mutex.Unlock()

    healthyWorkers := lb.getHealthyWorkers()
    if len(healthyWorkers) == 0 {
        return nil, fmt.Errorf("no healthy workers available")
    }

    // Sort by load (CPU + Memory + Active Jobs)
    sort.Slice(healthyWorkers, func(i, j int) bool {
        loadI := lb.calculateLoad(healthyWorkers[i])
        loadJ := lb.calculateLoad(healthyWorkers[j])
        return loadI < loadJ
    })

    // Select least loaded worker
    selected := healthyWorkers[0]
    selected.ActiveJobs++

    lb.logger.Infof("Selected worker %s (load: %.2f)", selected.ID, lb.calculateLoad(selected))
    return selected, nil
}

func (lb *LoadBalancer) calculateLoad(worker *WorkerNode) float64 {
    // Weighted load calculation
    cpuWeight := 0.3
    memoryWeight := 0.3
    jobWeight := 0.4

    cpuLoad := worker.CPU / 100.0
    memoryLoad := worker.Memory / 100.0
    jobLoad := float64(worker.ActiveJobs) / float64(worker.MaxJobs)

    return cpuWeight*cpuLoad + memoryWeight*memoryLoad + jobWeight*jobLoad
}

Production Deployment

Docker Compose Production Setup

version: '3.8'

services:
  # RabbitMQ cluster
  rabbitmq:
    image: rabbitmq:3.12-management
    hostname: rabbitmq-main
    environment:
      RABBITMQ_ERLANG_COOKIE: ${RABBITMQ_COOKIE}
      RABBITMQ_DEFAULT_USER: ${RABBITMQ_USER}
      RABBITMQ_DEFAULT_PASS: ${RABBITMQ_PASS}
      RABBITMQ_DEFAULT_VHOST: /
    volumes:
      - rabbitmq_data:/var/lib/rabbitmq
      - ./rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf
    networks:
      - backend
    deploy:
      replicas: 1
      resources:
        limits:
          memory: 1G
          cpus: '0.5'

  # Redis cluster
  redis:
    image: redis:7-alpine
    command: redis-server --appendonly yes --maxmemory 512mb --maxmemory-policy allkeys-lru
    volumes:
      - redis_data:/data
    networks:
      - backend
    deploy:
      replicas: 1
      resources:
        limits:
          memory: 512M
          cpus: '0.25'

  # API Backend
  api-backend:
    build:
      context: ./backend
      dockerfile: Dockerfile.production
    environment:
      NODE_ENV: production
      RABBITMQ_URL: amqp://${RABBITMQ_USER}:${RABBITMQ_PASS}@rabbitmq:5672/
      REDIS_URL: redis://redis:6379
      LOG_LEVEL: info
      RATE_LIMIT_WINDOW: 60000
      RATE_LIMIT_MAX: 10
    depends_on:
      - rabbitmq
      - redis
    networks:
      - backend
      - frontend
    deploy:
      replicas: 2
      resources:
        limits:
          memory: 512M
          cpus: '0.5'
      update_config:
        order: start-first
        failure_action: rollback

  # Runner Service
  runner-service:
    build:
      context: ./runner
      dockerfile: Dockerfile.production
    environment:
      RABBITMQ_URL: amqp://${RABBITMQ_USER}:${RABBITMQ_PASS}@rabbitmq:5672/
      REDIS_URL: redis://redis:6379
      LOG_LEVEL: info
      MAX_CONCURRENT_JOBS: 5
      WORKSPACE_DIR: /tmp/workspaces
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - runner_workspaces:/tmp/workspaces
    depends_on:
      - rabbitmq
      - redis
    networks:
      - backend
      - execution
    deploy:
      replicas: 3
      resources:
        limits:
          memory: 2G
          cpus: '1.0'
      placement:
        constraints:
          - node.role == worker

  # Frontend
  frontend:
    build:
      context: ./frontend
      dockerfile: Dockerfile.production
    environment:
      REACT_APP_API_URL: http://api-backend:3001
      REACT_APP_WS_URL: ws://api-backend:3001
    depends_on:
      - api-backend
    networks:
      - frontend
    deploy:
      replicas: 2
      resources:
        limits:
          memory: 256M
          cpus: '0.25'

  # Load Balancer
  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
      - ./ssl:/etc/ssl/certs
    depends_on:
      - frontend
      - api-backend
    networks:
      - frontend
    deploy:
      replicas: 1
      resources:
        limits:
          memory: 128M
          cpus: '0.1'

volumes:
  rabbitmq_data:
  redis_data:
  runner_workspaces:

networks:
  frontend:
    driver: overlay
  backend:
    driver: overlay
  execution:
    driver: overlay
    internal: true

Kubernetes Deployment

# namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: code-compiler

---
# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: code-compiler
data:
  RABBITMQ_URL: "amqp://admin:password@rabbitmq:5672/"
  REDIS_URL: "redis://redis:6379"
  LOG_LEVEL: "info"
  MAX_CONCURRENT_JOBS: "5"

---
# runner-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: runner-service
  namespace: code-compiler
spec:
  replicas: 3
  selector:
    matchLabels:
      app: runner-service
  template:
    metadata:
      labels:
        app: runner-service
    spec:
      containers:
      - name: runner
        image: your-registry/runner-service:latest
        envFrom:
        - configMapRef:
            name: app-config
        resources:
          requests:
            memory: "1Gi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "1000m"
        volumeMounts:
        - name: docker-sock
          mountPath: /var/run/docker.sock
        - name: workspaces
          mountPath: /tmp/workspaces
        securityContext:
          runAsNonRoot: true
          runAsUser: 1000
      volumes:
      - name: docker-sock
        hostPath:
          path: /var/run/docker.sock
      - name: workspaces
        emptyDir:
          sizeLimit: 10Gi

---
# hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: runner-service-hpa
  namespace: code-compiler
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: runner-service
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

Monitoring and Observability

# monitoring-stack.yaml
version: '3.8'

services:
  # Prometheus
  prometheus:
    image: prom/prometheus:latest
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--storage.tsdb.retention.time=30d'
      - '--web.enable-lifecycle'
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    networks:
      - monitoring

  # Grafana
  grafana:
    image: grafana/grafana:latest
    environment:
      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASSWORD}
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/dashboards:/etc/grafana/provisioning/dashboards
      - ./grafana/datasources:/etc/grafana/provisioning/datasources
    networks:
      - monitoring

  # ELK Stack for Logs
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    environment:
      discovery.type: single-node
      xpack.security.enabled: false
    volumes:
      - elasticsearch_data:/usr/share/elasticsearch/data
    networks:
      - logging

  logstash:
    image: docker.elastic.co/logstash/logstash:8.11.0
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
    depends_on:
      - elasticsearch
    networks:
      - logging

  kibana:
    image: docker.elastic.co/kibana/kibana:8.11.0
    environment:
      ELASTICSEARCH_HOSTS: http://elasticsearch:9200
    depends_on:
      - elasticsearch
    networks:
      - logging

volumes:
  prometheus_data:
  grafana_data:
  elasticsearch_data:

networks:
  monitoring:
  logging:

Key Learnings

1. Container Management is Complex

Key Challenges:

Cold Start Problem: Container creation takes 2-5 seconds
Resource Leaks: Containers not properly cleaned up
State Management: Persistent vs ephemeral container strategies
Network Isolation: Balancing security with functionality

Solutions Implemented:

Container pooling with pre-warmed instances
Automatic cleanup with garbage collection
Persistent containers with workspace isolation
Network-isolated execution environments

2. Real-time Streaming Requires Careful Architecture

Technical Insights:

WebSocket Management: Connection pooling and cleanup crucial
Message Ordering: Ensure output lines arrive in sequence
Buffer Management: Handle high-frequency output efficiently
Connection Recovery: Graceful handling of network issues

Best Practices:

Use Redis pub/sub for scalable streaming
Implement connection heartbeats
Buffer and batch small messages
Provide fallback to polling for unreliable connections

3. Security Cannot Be an Afterthought

Critical Security Measures:

Defense in Depth: Multiple security layers
Principle of Least Privilege: Minimal container permissions
Resource Limits: Prevent resource exhaustion attacks
Code Analysis: Static analysis before execution

Security Architecture:

┌─────────────────┐
│   Code Input    │
├─────────────────┤
│ Static Analysis │  ← First line of defense
├─────────────────┤
│ Rate Limiting   │  ← Prevent abuse
├─────────────────┤
│ Docker Sandbox  │  ← Isolation layer
├─────────────────┤
│ Resource Limits │  ← Resource protection
├─────────────────┤
│ Network Filter  │  ← Network restrictions
└─────────────────┘

4. Performance Optimization is Multi-Faceted

Optimization Areas:

Container Lifecycle: Pool management and reuse
Resource Allocation: Dynamic scaling based on load
Queue Management: Fair distribution and priority handling
Caching: Language environment and dependency caching

Performance Metrics to Track:

Container startup time
Execution latency
Queue depth
Resource utilization
Success/failure rates

5. Production Reliability Requires Operational Excellence

Observability Stack:

Metrics: Prometheus + Grafana for system health
Logging: ELK stack for centralized log analysis
Tracing: Distributed tracing for request flows
Alerting: PagerDuty integration for critical issues

Deployment Strategies:

Blue-green deployments for zero downtime
Canary releases for gradual rollouts
Circuit breakers for fault tolerance
Auto-scaling based on queue depth and CPU usage

6. Language-Specific Considerations

Each programming language has unique requirements:

Python:

Dependency management with pip
Virtual environment isolation
Import path security
Package installation caching

Node.js:

npm/yarn dependency resolution
Module loading restrictions
Event loop management
Memory garbage collection

Go:

Module system (go.mod)
Build caching for faster compilation
Static binary advantages
Goroutine resource management

Rust:

Cargo package management
Compilation time optimization
Memory safety guarantees
Target architecture handling

Java:

Classpath management
JVM startup optimization
Garbage collection tuning
Security manager configuration

Conclusion

Building a production-ready online code compiler is a journey that touches every aspect of modern distributed systems engineering. From container orchestration to real-time streaming, from security isolation to performance optimization, each component requires careful consideration and robust implementation.

The key to success lies in:

Robust Architecture: Design for failure and scale from day one
Security First: Implement security at every layer
Performance Focus: Optimize for user experience and resource efficiency
Operational Excellence: Monitor, measure, and continuously improve
Incremental Development: Start simple and add complexity gradually

The result should be a platform that feels immediate and reliable, allowing developers to focus on code rather than infrastructure. When users can execute code with the same confidence they have in their local development environment, you've achieved the goal of a truly powerful online code compiler.

Learning Resources

Essential Reading

Distributed Systems:

"Designing Data-Intensive Applications" by Martin Kleppmann - Comprehensive guide to distributed system patterns
"Building Microservices" by Sam Newman - Microservice architecture and communication patterns
"Site Reliability Engineering" by Google - Production system reliability practices

Container Technologies:

"Docker Deep Dive" by Nigel Poulton - Comprehensive Docker guide
"Kubernetes in Action" by Marko Lukša - Kubernetes orchestration patterns
"Container Security" by Liz Rice - Security best practices for containers

Real-time Systems:

"High Performance Browser Networking" by Ilya Grigorik - WebSocket and real-time communication
"Redis in Action" by Josiah Carlson - Redis patterns for real-time applications

Documentation and Specifications

Container Security:

Message Queues:

Performance Optimization:

Open Source Projects

Code Execution Platforms:

Judge0 - Online code execution system
HackerEarth API - Commercial code execution platform
Glot.io - Simple code execution service

Container Management:

Docker - Container runtime
Podman - Alternative container runtime
gVisor - Application kernel for containers

Message Queue Solutions:

RabbitMQ - Feature-rich message broker
Apache Kafka - High-throughput distributed streaming
Redis - In-memory data structure store

Tools and Development Environment

Development Tools:

Docker Desktop - Local container development
Kubernetes KIND - Local Kubernetes development
Minikube - Local Kubernetes cluster

Monitoring and Observability:

Prometheus - Metrics collection and alerting
Grafana - Metrics visualization and dashboards
ELK Stack - Centralized logging and analysis

Testing Frameworks:

Testcontainers - Integration testing with containers
k6 - Load testing for APIs and WebSockets
Artillery - Performance testing toolkit

With love from the Toki Space team

This tutorial represents our collective experience building Toki's code execution platform. The architecture and lessons shared here will help you build your own robust online code compiler. For questions or contributions, reach out to our engineering team at hello@tokispace.com

Building a Web-Based Excel Editor: A Comprehensive Guide

Augustus otu — Mon, 23 Jun 2025 20:31:46 +0000

By the Toki Space Team

Building a fully-featured Excel editor in the browser is one of the most challenging frontend projects you can undertake. It requires deep understanding of canvas rendering, complex state management, real-time collaboration, and Excel's intricate feature set.

This isn't just another grid component—we're talking about recreating decades of Microsoft Excel's functionality in a browser, complete with formulas, formatting, charts, and real-time collaboration. The complexity rivals that of building a game engine or CAD software.

This tutorial will walk you through building a production-ready collaborative Excel editor using React, TypeScript, and HTML Canvas, sharing the hard-won lessons from building Toki's spreadsheet editor.

Architecture Overview
Canvas Rendering System
Cell Management and Selection
Editing System
Formatting and Styling
Real-time Collaboration
Advanced Features
Performance Optimization
Key Learnings

Architecture Overview

Core Technologies

Our Excel editor is built with:

React 18 with TypeScript for component architecture
HTML5 Canvas for high-performance grid rendering
WebSocket for real-time collaboration
Custom hooks for state management
Tauri for desktop app capabilities

Key Design Decisions

Canvas over DOM: We chose canvas rendering over DOM-based approaches for performance. Rendering thousands of cells in DOM would be prohibitively slow.
Layered Canvas Architecture: Multiple canvas layers for different concerns:
- Grid and cell content
- Headers (row/column)
- Selection overlay
- Collaboration cursors
Viewport-based Rendering: Only render visible cells plus a buffer zone, similar to virtual scrolling.

// Core constants that define our grid
const CELL_WIDTH = 100;
const CELL_HEIGHT = 32;
const ROW_HEADER_WIDTH = 60;
const COLUMN_HEADER_HEIGHT = 32;
const MAX_COLUMNS = 16384; // Excel's XFD limit
const MAX_ROWS = 1048576; // Excel's row limit
const VIEWPORT_BUFFER = 10; // Extra cells outside viewport

Canvas Rendering System

Understanding Canvas vs DOM for Data Grids

Before diving into implementation, it's crucial to understand why we chose HTML5 Canvas over DOM-based approaches. A typical Excel sheet contains 16,384 columns × 1,048,576 rows = ~17 billion potential cells. Even rendering 1,000 visible cells as DOM elements would create severe performance bottlenecks.

Canvas provides:

Immediate mode rendering: Direct pixel manipulation without DOM overhead
Batch operations: Efficient drawing of thousands of elements in single operations
Memory efficiency: Single canvas element vs thousands of DOM nodes
Pixel-perfect control: Exact positioning and custom rendering logic

The tradeoff is complexity—we must manually implement everything the browser normally handles (hit testing, text input, accessibility).

Multi-Layer Canvas Setup

The foundation of our Excel editor is a multi-canvas system where each layer handles specific rendering concerns. This separation allows us to optimize redraws—when only the selection changes, we don't need to redraw the entire grid.

const canvasContainerRef = useRef<HTMLDivElement>(null);
const gridCanvasRef = useRef<HTMLCanvasElement>(null);           // Cells and grid lines
const headerCanvasRef = useRef<HTMLCanvasElement>(null);         // Row/column headers  
const selectionCanvasRef = useRef<HTMLCanvasElement>(null);      // Selection highlights
const collaborationCanvasRef = useRef<HTMLCanvasElement>(null);  // Other users' cursors

Layer Architecture Explained:

Grid Layer (bottom): Static content that changes infrequently—cell values, borders, backgrounds
Header Layer: Row numbers and column letters, with resize handles
Selection Layer: User's current selection and active cell highlight
Collaboration Layer (top): Real-time cursors and selections from other users

This layering prevents expensive redraws. When a user moves their cursor, only the selection canvas needs updating, not the entire grid.

Viewport Calculation: Virtual Scrolling for Massive Datasets

The viewport system is the heart of our performance optimization. Like video game engines that only render what's visible, we calculate which cells are currently in view and only draw those plus a small buffer.

The Challenge: Excel supports 1,048,576 rows. If each row was 32px tall, the total height would be 33,554,432 pixels—far exceeding browser limits and causing memory issues.

The Solution: Virtual scrolling with dynamic row heights.

const calculateViewport = useCallback(() => {
  if (!canvasContainerRef.current) return;

  const container = canvasContainerRef.current;
  const scrollX = container.scrollLeft;
  const scrollY = container.scrollTop;
  const width = container.clientWidth;
  const height = container.clientHeight;

  // Calculate visible rows using cumulative height calculation
  // This is more complex than simple division because rows can have different heights
  let startRow = 0;
  let y = 0;

  // Walk through rows until we reach the scroll position
  while (y < scrollY && startRow < DEFAULT_ROWS) {
    y += rowHeights[startRow] || CELL_HEIGHT;
    startRow++;
  }
  // Include buffer rows before visible area for smooth scrolling
  startRow = Math.max(0, startRow - VIEWPORT_BUFFER);

  // Calculate end row by continuing from start row
  let endRow = startRow;
  y = y - (rowHeights[startRow] || CELL_HEIGHT) * VIEWPORT_BUFFER;
  while (y < scrollY + height && endRow < DEFAULT_ROWS) {
    y += rowHeights[endRow] || CELL_HEIGHT;
    endRow++;
  }
  // Include buffer rows after visible area
  endRow = Math.min(DEFAULT_ROWS - 1, endRow + VIEWPORT_BUFFER);

  // Similar calculation for columns with dynamic widths...

  setViewportState({
    scrollX, scrollY, width, height,
    startRow, endRow, startColumn, endColumn,
  });
}, [columnWidths, rowHeights]);

Key Concepts:

Cumulative Positioning: We can't use simple math (row * height) because rows have variable heights. We must walk through each row, accumulating heights until we reach the scroll position.
Buffer Zones: We render extra rows/columns outside the visible area. This prevents flickering during fast scrolling and provides smooth user experience.
Coordinate System: The viewport maintains two coordinate systems:
- Logical coordinates: Row/column indices (0, 1, 2...)
- Physical coordinates: Pixel positions on screen

Performance Impact: This approach lets us handle massive spreadsheets while only rendering ~100-200 cells at any time, regardless of the total data size.

Grid Rendering

The grid rendering system draws cells, borders, and content efficiently:

const drawGrid = useCallback((ctx: CanvasRenderingContext2D) => {
  ctx.save();
  ctx.strokeStyle = "#e0e0e0";
  ctx.lineWidth = 1;

  const { startRow, endRow, startColumn, endColumn } = viewportState;

  // Draw vertical lines
  let x = ROW_HEADER_WIDTH;
  for (let col = startColumn; col <= endColumn; col++) {
    const width = (columnWidths[col] || CELL_WIDTH) * zoomLevel;
    x += width;
    ctx.beginPath();
    ctx.moveTo(x, 0);
    ctx.lineTo(x, ctx.canvas.height);
    ctx.stroke();
  }

  // Draw horizontal lines
  let y = COLUMN_HEADER_HEIGHT;
  for (let row = startRow; row <= endRow; row++) {
    const height = (rowHeights[row] || CELL_HEIGHT) * zoomLevel;
    y += height;
    ctx.beginPath();
    ctx.moveTo(0, y);
    ctx.lineTo(ctx.canvas.width, y);
    ctx.stroke();
  }

  ctx.restore();
}, [viewportState, columnWidths, rowHeights, zoomLevel]);

Cell Content Rendering

Each cell's content is rendered with proper formatting and styling:

const drawCellContent = useCallback((
  ctx: CanvasRenderingContext2D,
  row: number,
  column: number,
  x: number,
  y: number,
  width: number,
  height: number
) => {
  const address = getCellAddress(row, column);
  const cellData = getCellByAddress(address);

  if (!cellData || (!cellData.value && !cellData.formula)) return;

  ctx.save();

  // Clip to cell bounds
  ctx.beginPath();
  ctx.rect(x, y, width, height);
  ctx.clip();

  // Apply cell formatting
  const style = cellData.style || {};

  // Background
  if (style.backgroundColor) {
    ctx.fillStyle = style.backgroundColor;
    ctx.fillRect(x, y, width, height);
  }

  // Text formatting
  const fontSize = (style.fontSize || 11) * zoomLevel;
  const fontName = style.fontName || "Arial";
  const fontWeight = style.bold ? "bold" : "normal";
  const fontStyle = style.italic ? "italic" : "normal";

  ctx.font = `${fontStyle} ${fontWeight} ${fontSize}px ${fontName}`;
  ctx.fillStyle = style.fontColor || "#000";

  // Text alignment and rendering
  const padding = 4 * zoomLevel;
  let textX = x + padding;
  let textY = y + height / 2;

  ctx.textBaseline = "middle";

  switch (style.horizontalAlignment) {
    case "center":
      ctx.textAlign = "center";
      textX = x + width / 2;
      break;
    case "right":
      ctx.textAlign = "right";
      textX = x + width - padding;
      break;
    default:
      ctx.textAlign = "left";
  }

  // Display formatted value
  const displayValue = cellData.value;
  if (displayValue !== undefined && displayValue !== null) {
    let formattedValue = String(displayValue);

    if (style.numberFormat && typeof displayValue === "number") {
      formattedValue = formatNumber(displayValue, style.numberFormat);
    }

    ctx.fillText(formattedValue, textX, textY);
  }

  ctx.restore();
}, [getCellByAddress, zoomLevel]);

Cell Management and Selection

Cell Addressing System

Excel uses an alphanumeric addressing system (A1, B2, etc.). Here's how we implement it:

const columnToLetter = (column: number): string => {
  let result = "";
  while (column >= 0) {
    result = String.fromCharCode((column % 26) + 65) + result;
    column = Math.floor(column / 26) - 1;
  }
  return result;
};

const letterToColumn = (letter: string): number => {
  let result = 0;
  for (let i = 0; i < letter.length; i++) {
    result = result * 26 + letter.charCodeAt(i) - 64;
  }
  return result - 1;
};

const getCellAddress = (row: number, column: number): string => {
  return `${columnToLetter(column)}${row + 1}`;
};

Selection System

The selection system handles single cells and ranges:

interface CellPosition {
  row: number;
  column: number;
  worksheetId: string;
}

interface CellSelection {
  start: CellPosition;
  end: CellPosition;
}

// State management
const [activeCell, setActiveCell] = useState<CellPosition | null>(null);
const [selectedRange, setSelectedRange] = useState<CellSelection | null>(null);
const [isSelecting, setIsSelecting] = useState(false);

Mouse Event Handling: Coordinate System Transformations

Converting mouse coordinates to cell positions is more complex than it appears. We're dealing with multiple coordinate systems that must be reconciled:

Screen coordinates: Mouse position relative to viewport
Canvas coordinates: Position relative to canvas element
Grid coordinates: Position accounting for scroll offset
Cell coordinates: Final row/column indices

The Mathematical Challenge: Unlike a simple grid where we could use Math.floor(x / cellWidth), Excel has variable row heights and column widths, requiring iterative calculation.

const getCellAtCoordinates = useCallback((x: number, y: number): CellPosition | null => {
  if (!currentWorksheet) return null;

  // Step 1: Transform from canvas coordinates to grid coordinates
  // Account for header areas (row numbers, column letters)
  x -= ROW_HEADER_WIDTH;
  y -= COLUMN_HEADER_HEIGHT;

  // Step 2: Validate coordinates are within grid bounds
  if (x < 0 || y < 0) return null;

  // Step 3: Find column using cumulative width calculation
  // We can't use simple division because columns have variable widths
  let column = 0;
  let accX = 0;
  while (accX < x && column < DEFAULT_COLUMNS) {
    // Account for zoom level in all calculations
    accX += (columnWidths[column] || CELL_WIDTH) * zoomLevel;
    column++;
  }
  // Adjust for off-by-one from the while loop
  column = Math.max(0, column - 1);

  // Step 4: Find row using cumulative height calculation  
  let row = 0;
  let accY = 0;
  while (accY < y && row < DEFAULT_ROWS) {
    accY += (rowHeights[row] || CELL_HEIGHT) * zoomLevel;
    row++;
  }
  row = Math.max(0, row - 1);

  return { row, column, worksheetId: currentWorksheet.id };
}, [currentWorksheet, columnWidths, rowHeights, zoomLevel]);

Why This Approach?

Variable Sizing: Excel allows different widths/heights per column/row
Zoom Support: All calculations must account for zoom level
Precision: We need exact cell boundaries, not approximate regions
Performance: This O(n) search is faster than binary search for typical viewport sizes

Edge Cases Handled:

Clicks on headers (return null)
Clicks outside grid boundaries
Zoom level transformations
Fractional pixel positions

Editing System

In-Cell Editing

The editing system provides seamless inline editing with formula support:

const [editingCell, setEditingCell] = useState<CellPosition | null>(null);
const [editingValue, setEditingValue] = useState("");
const [isEditingFormula, setIsEditingFormula] = useState(false);

// Render editing input overlay
const renderCellInput = () => {
  if (!editingCell || !currentWorksheet) return null;

  const { x, y } = getCellPosition(editingCell.row, editingCell.column);
  const { width, height } = getCellDimensions(editingCell.row, editingCell.column);

  return (
    <Input
      className="absolute z-[60] border-2 border-blue-500 px-1 bg-white"
      style={{
        left: `${x - viewportState.scrollX}px`,
        top: `${y - viewportState.scrollY}px`,
        width: `${Math.max(width, 200)}px`,
        height: `${height}px`,
        fontSize: `${11 * zoomLevel}px`,
      }}
      value={editingValue}
      onChange={(e) => handleFormulaInput(e.target.value)}
      onKeyDown={handleEditKeyDown}
      onBlur={() => handleCellEdit(editingValue)}
      autoFocus
    />
  );
};

Formula Bar Integration

The formula bar provides Excel-like formula editing:

const [formulaBarValue, setFormulaBarValue] = useState("");
const [showFunctionHelper, setShowFunctionHelper] = useState(false);
const [functionSuggestions, setFunctionSuggestions] = useState<string[]>([]);

const handleFormulaInput = useCallback((value: string) => {
  setFormulaBarValue(value);
  setEditingValue(value);

  if (value.startsWith("=")) {
    setIsEditingFormula(true);

    // Extract the current function being typed
    const lastFunctionMatch = value.match(/([A-Z]+)$/);
    if (lastFunctionMatch) {
      const partialFunction = lastFunctionMatch[1];
      const suggestions = EXCEL_FUNCTIONS.filter(
        (func) => func.startsWith(partialFunction) && func !== partialFunction
      ).slice(0, 10);

      setFunctionSuggestions(suggestions);
      setShowFunctionHelper(suggestions.length > 0);
    } else {
      setShowFunctionHelper(false);
    }
  } else {
    setIsEditingFormula(false);
    setShowFunctionHelper(false);
  }
}, []);

Cell Value Updates

Handling cell updates with proper formula vs value distinction:

const handleCellEdit = useCallback(async (value: string) => {
  if (!editingCell || !currentWorksheet || !documentId) return;

  const address = getCellAddress(editingCell.row, editingCell.column);

  try {
    const isFormula = value.startsWith("=");

    if (isFormula) {
      // Use setCellFormula for formula-specific handling
      await setCellFormula(documentId, {
        worksheetId: currentWorksheet.id,
        cellAddress: address,
        formula: value,
      });
    } else {
      // Use updateCell for regular values
      await updateCell(documentId, {
        worksheetId: currentWorksheet.id,
        cellAddress: address,
        value: value,
      });
    }

    // Send edit to collaborators
    if (excelState.editSession) {
      websocketService.sendExcelEdit(
        documentId,
        currentWorksheet.id,
        excelState.editSession.id,
        isFormula ? "formula_update" : "cell_update",
        address,
        {
          value: isFormula ? undefined : value,
          formula: isFormula ? value : undefined,
        }
      );
    }

    // Clear editing state
    setEditingCell(null);
    setEditingValue("");
    setIsEditingFormula(false);
    setShowFunctionHelper(false);

    // Refresh data
    await loadWorksheetData(documentId, currentWorksheet.id);
    render();
  } catch (error) {
    console.error("Failed to update cell:", error);
    toast.error("Failed to update cell");
  }
}, [editingCell, currentWorksheet, documentId, setCellFormula, updateCell]);

Formatting and Styling

Cell Style System

Excel cells support rich formatting options:

interface CellFormat {
  bold?: boolean;
  italic?: boolean;
  underline?: boolean;
  fontSize?: number;
  fontFamily?: string;
  textColor?: string;
  backgroundColor?: string;
  borderColor?: string;
  borderStyle?: string;
  alignment?: "left" | "center" | "right";
  verticalAlignment?: "top" | "middle" | "bottom";
  numberFormat?: string;
  wrapText?: boolean;
}

Format Application

Applying formatting to selected cells:

const handleToggleBold = useCallback(async () => {
  if (!activeCell || !currentWorksheet || !documentId) return;

  try {
    const address = getCellAddress(activeCell.row, activeCell.column);
    const cellData = getCellByAddress(address);
    const currentStyle = cellData?.style || {};

    await setCellStyle(documentId, {
      worksheetId: currentWorksheet.id,
      range: address,
      style: {
        ...currentStyle,
        bold: !currentStyle.bold,
      },
    });

    await loadWorksheetData(documentId, currentWorksheet.id);
    render();
  } catch (error) {
    console.error("Failed to toggle bold:", error);
    toast.error("Failed to apply formatting");
  }
}, [activeCell, currentWorksheet, documentId, getCellByAddress, setCellStyle]);

Number Formatting

Supporting various number formats like Excel:

const formatNumber = useCallback((value: number, format: string): string => {
  switch (format) {
    case "currency":
      return new Intl.NumberFormat("en-US", {
        style: "currency",
        currency: "USD",
      }).format(value);
    case "percent":
      return new Intl.NumberFormat("en-US", {
        style: "percent",
        minimumFractionDigits: 0,
        maximumFractionDigits: 2,
      }).format(value);
    case "thousands":
      return new Intl.NumberFormat("en-US").format(value);
    case "decimal2":
      return value.toFixed(2);
    default:
      return String(value);
  }
}, []);

Real-time Collaboration

The Challenge of Collaborative Editing

Real-time collaborative editing is one of computer science's most complex distributed systems problems. Consider this scenario:

User A types "Hello" in cell A1
User B simultaneously types "World" in the same cell
Both operations occur at nearly the same time

Questions that arise:

Which edit wins?
How do we maintain consistency across all clients?
How do we handle network delays and out-of-order messages?
How do we resolve conflicts without losing data?

Operational Transformation Theory

Excel-style collaboration requires Operational Transformation (OT), a mathematical framework for handling concurrent edits. The core principle:

Given operations O1 and O2 that started from the same document state, we can transform them into O1' and O2' such that applying O1' after O2 produces the same result as applying O2' after O1.

Mathematical Properties (Consistency Conditions):

TP1: O1 ∘ transform(O2, O1) = O2 ∘ transform(O1, O2)
TP2: transform(O3, O1 ∘ O2) = transform(transform(O3, O1), O2)

These equations ensure that all clients converge to the same final state regardless of operation order.

User Presence System

Beyond document operations, we track user presence for seamless collaboration:

interface UserPresence {
  userId: string;
  userName: string;
  userColor: string;          // Deterministic color from userId hash
  cursor?: CellPosition;      // Current cell cursor position
  selection?: CellSelection;  // Selected range (for copy/paste, etc.)
  isTyping: boolean;         // Shows typing indicator
  lastActivity: number;      // For connection timeout detection
}

const [userPresences, setUserPresences] = useState<Map<string, UserPresence>>(new Map());
const [isConnected, setIsConnected] = useState(false);

Presence Design Decisions:

Deterministic Colors: User colors are generated from userId hashes, ensuring consistency across sessions
Granular State: Separate cursor vs selection allows for different interaction modes
Activity Tracking: Detect and clean up stale connections
Lightweight Updates: Presence changes are sent separately from document operations to avoid conflicts

WebSocket Integration

Setting up real-time communication:

const setupWebSocketCollaboration = useCallback(() => {
  if (!documentId || !excelState.editSession || !currentWorksheet) return;

  websocketService
    .on("onDocumentEdit", (type, data) => {
      if (type === "excel" && data.documentId === documentId) {
        handleRemoteEdit(data);
      }
    })
    .on("onDocumentCursor", (type, data) => {
      if (type === "excel" && data.documentId === documentId) {
        handleRemoteCursor(data);
      }
    })
    .on("onDocumentSelection", (type, data) => {
      if (type === "excel" && data.documentId === documentId) {
        handleRemoteSelection(data);
      }
    });

  // Join the document
  websocketService.joinExcelDocument(
    documentId,
    currentWorksheet.id,
    excelState.editSession.id,
    currentUser.userName
  );

  setIsConnected(true);
}, [documentId, excelState.editSession, currentWorksheet, currentUser]);

Collaborative Cursor Rendering

Drawing other users' cursors and selections:

const drawCollaborationCursors = useCallback((ctx: CanvasRenderingContext2D) => {
  ctx.clearRect(0, 0, ctx.canvas.width, ctx.canvas.height);

  userPresences.forEach((presence) => {
    if (presence.cursor && presence.cursor.worksheetId === currentWorksheet?.id) {
      const { x, y } = getCellPosition(presence.cursor.row, presence.cursor.column);
      const { width, height } = getCellDimensions(presence.cursor.row, presence.cursor.column);

      ctx.save();

      // Draw cursor outline
      ctx.strokeStyle = presence.userColor;
      ctx.lineWidth = 2;
      ctx.strokeRect(
        x - viewportState.scrollX,
        y - viewportState.scrollY,
        width,
        height
      );

      // Draw user label
      ctx.fillStyle = presence.userColor;
      ctx.fillRect(
        x - viewportState.scrollX,
        y - viewportState.scrollY - 20,
        100,
        20
      );
      ctx.fillStyle = "#fff";
      ctx.font = "12px sans-serif";
      ctx.fillText(
        presence.userName,
        x - viewportState.scrollX + 4,
        y - viewportState.scrollY - 6
      );

      ctx.restore();
    }
  });
}, [userPresences, currentWorksheet, getCellPosition, getCellDimensions]);

Advanced Features

Row and Column Operations

Supporting Excel's row/column manipulation:

const handleInsertRowAbove = useCallback(async () => {
  if (!activeCell || !currentWorksheet || !documentId) return;

  try {
    await insertRow(documentId, {
      worksheetId: currentWorksheet.id,
      rowIndex: activeCell.row,
      count: 1,
    });

    await loadWorksheetData(documentId, currentWorksheet.id);
    render();
    toast.success("Row inserted");
  } catch (error) {
    console.error("Failed to insert row:", error);
    toast.error("Failed to insert row");
  }
}, [activeCell, currentWorksheet, documentId, insertRow]);

Cell Merging

Implementing Excel's cell merge functionality:

const handleToggleMergeCells = useCallback(async () => {
  if (!selectedRange || !currentWorksheet || !documentId) return;

  try {
    const startAddress = getCellAddress(
      Math.min(selectedRange.start.row, selectedRange.end.row),
      Math.min(selectedRange.start.column, selectedRange.end.column)
    );
    const endAddress = getCellAddress(
      Math.max(selectedRange.start.row, selectedRange.end.row),
      Math.max(selectedRange.start.column, selectedRange.end.column)
    );

    const isMerged = false; // Check actual merge status

    if (isMerged) {
      await unmergeCells(documentId, {
        worksheetId: currentWorksheet.id,
        range: `${startAddress}:${endAddress}`,
      });
    } else {
      await mergeCells(documentId, {
        worksheetId: currentWorksheet.id,
        range: `${startAddress}:${endAddress}`,
      });
    }

    await loadWorksheetData(documentId, currentWorksheet.id);
    render();
  } catch (error) {
    console.error("Failed to merge/unmerge cells:", error);
  }
}, [selectedRange, currentWorksheet, documentId, mergeCells, unmergeCells]);

Comments System

Adding collaborative comments to cells:

const handleAddComment = useCallback(() => {
  if (!activeCell || !currentWorksheet) return;

  const cellAddress = getCellAddress(activeCell.row, activeCell.column);
  setCommentTargetCell(cellAddress);
  setCurrentComment("");
  setShowCommentDialog(true);
}, [activeCell, currentWorksheet]);

const saveComment = useCallback(async () => {
  if (!commentTargetCell || !currentComment.trim() || !documentId || !currentWorksheet) {
    return;
  }

  try {
    await addComment(documentId, {
      worksheetId: currentWorksheet.id,
      cellAddress: commentTargetCell,
      text: currentComment,
    });

    // Send comment via WebSocket for real-time collaboration
    if (excelState.editSession) {
      websocketService.sendExcelEdit(
        documentId,
        currentWorksheet.id,
        excelState.editSession.id,
        "comment_add",
        commentTargetCell,
        {
          text: currentComment,
          author: currentUser.userName,
          userId: currentUser.userId,
          timestamp: Date.now(),
        }
      );
    }

    setCurrentComment("");
    setShowCommentDialog(false);
    await loadWorksheetData(documentId, currentWorksheet.id);
    render();
  } catch (error) {
    console.error("Failed to save comment:", error);
  }
}, [commentTargetCell, currentComment, documentId, currentWorksheet, addComment]);

Find and Replace

Implementing Excel's find/replace functionality:

const [showFindReplace, setShowFindReplace] = useState(false);
const [searchQuery, setSearchQuery] = useState("");
const [replaceText, setReplaceText] = useState("");

// Find and replace implementation
const handleFindAndReplace = useCallback(async () => {
  if (!documentId || !currentWorksheet || !searchQuery) return;

  try {
    await replaceInWorksheet(documentId, {
      worksheetId: currentWorksheet.id,
      find: searchQuery,
      replace: replaceText,
      options: { replaceAll: true },
    });

    await loadWorksheetData(documentId, currentWorksheet.id);
    render();
    toast.success("Replace completed");
  } catch (error) {
    console.error("Failed to replace:", error);
  }
}, [documentId, currentWorksheet, searchQuery, replaceText]);

Performance Optimization

Viewport-Based Rendering

Only render visible cells to maintain performance:

// Only render cells within viewport plus buffer
const drawCells = useCallback((ctx: CanvasRenderingContext2D) => {
  const { startRow, endRow, startColumn, endColumn } = viewportState;

  let y = COLUMN_HEADER_HEIGHT;
  for (let row = startRow; row <= endRow; row++) {
    let x = ROW_HEADER_WIDTH;
    const height = (rowHeights[row] || CELL_HEIGHT) * zoomLevel;

    for (let col = startColumn; col <= endColumn; col++) {
      const width = (columnWidths[col] || CELL_WIDTH) * zoomLevel;
      drawCellContent(ctx, row, col, x, y, width, height);
      x += width;
    }
    y += height;
  }
}, [viewportState, columnWidths, rowHeights, zoomLevel, drawCellContent]);

Efficient Re-rendering: React Optimization Strategies

React's rendering can become a bottleneck when dealing with complex canvas operations. Here's how we optimize:

// Memoize expensive calculations
const activeCellData = useMemo(() => {
  if (!activeCell || !currentWorksheet) return null;
  const address = getCellAddress(activeCell.row, activeCell.column);
  return getCellByAddress(address);
}, [activeCell, currentWorksheet, getCellByAddress]);

// Debounce frequent operations to maintain 60fps
const debouncedRender = useMemo(
  () => debounce(render, 16), // 16ms = 60fps target
  [render]
);

// Separate hot and cold paths
const fastRender = useCallback(() => {
  // Only redraw selection layer for cursor movements
  const selectionCtx = getCanvasContext(selectionCanvasRef.current);
  if (selectionCtx) {
    drawSelection(selectionCtx);
  }
}, [drawSelection, getCanvasContext]);

const fullRender = useCallback(() => {
  // Full redraw for data changes
  render();
}, [render]);

Performance Principles Applied:

Differential Rendering: Only redraw changed layers
Frame Budget: Target 16ms per frame for 60fps
Memoization: Cache expensive calculations
Event Coalescing: Batch rapid updates
Lazy Evaluation: Defer non-critical rendering

Memory Management

Clean up resources and prevent memory leaks:

useEffect(() => {
  // Cleanup on unmount
  return () => {
    if (autoSaveTimeoutRef.current) {
      clearTimeout(autoSaveTimeoutRef.current);
    }
    if (collaborationTimeoutRef.current) {
      clearTimeout(collaborationTimeoutRef.current);
    }
    cleanupCollaboration();
  };
}, []);

Key Learnings

1. Canvas vs DOM Trade-offs

Canvas Advantages:

High performance with thousands of cells
Pixel-perfect control over rendering
Efficient updates and animations
Lower memory footprint

Canvas Challenges:

No native accessibility support
Complex event handling
Manual text input management
Limited browser dev tools support

2. State Management Complexity

Excel editors require sophisticated state management spanning multiple domains and consistency requirements:

State Architecture Layers:

// 1. Document State (Persistent)
interface DocumentState {
  worksheets: Worksheet[];
  metadata: DocumentMetadata;
  sharedSettings: WorkbookSettings;
  version: number;
}

// 2. Worksheet State (Per-sheet)
interface WorksheetState {
  cells: Map<string, CellData>;      // Sparse matrix of cell data
  rowHeights: Map<number, number>;   // Custom row heights
  columnWidths: Map<number, number>; // Custom column widths
  mergedCells: MergedRegion[];       // Cell merge information
  charts: Chart[];                   // Embedded charts
  images: Image[];                   // Embedded images
  formatting: ConditionalFormat[];   // Conditional formatting rules
  dataValidation: ValidationRule[];  // Data validation rules
}

// 3. UI State (Session-specific)
interface UIState {
  activeCell: CellPosition | null;
  selectedRange: CellSelection | null;
  editingCell: CellPosition | null;
  viewport: ViewportState;
  zoomLevel: number;
  showGridlines: boolean;
  showHeaders: boolean;
}

// 4. Collaboration State (Real-time)
interface CollaborationState {
  users: Map<string, UserPresence>;
  operations: Operation[];           // Pending operations
  vectorClock: VectorClock;         // For ordering operations
  conflicts: ConflictResolution[];   // Unresolved conflicts
}

// 5. Undo/Redo State (History)
interface HistoryState {
  undoStack: Operation[];
  redoStack: Operation[];
  maxHistorySize: number;
  currentVersion: number;
}

State Synchronization Challenges:

Cross-Layer Dependencies: UI state depends on document state, but changes must propagate both ways
Consistency Guarantees: Collaborative edits must maintain referential integrity
Performance Isolation: Heavy document operations shouldn't block UI updates
Memory Management: Large spreadsheets require careful state pruning

Solutions Implemented:

Normalized State: Separate storage for cells, formatting, and metadata
Immutable Updates: Using Immer for safe state mutations
Computed Properties: Derived state using selectors and memoization
Event Sourcing: All changes as operations for undo/collaboration

3. Real-time Collaboration Challenges

Conflict Resolution:

Operational Transformation for concurrent edits
Last-write-wins for simple properties
User presence and cursor tracking

Performance Considerations:

Debounce frequent updates
Batch small changes
Optimize WebSocket message size

4. Excel Feature Completeness

Building a complete Excel editor requires implementing:

Core Features: Cell editing, formulas, formatting
Layout Features: Row/column operations, merging, freezing
Data Features: Sorting, filtering, validation
Visual Features: Charts, images, conditional formatting
Collaboration: Comments, sharing, version history

5. Browser Limitations

Memory Constraints:

Large spreadsheets can exhaust browser memory
Implement virtual scrolling and lazy loading
Use Web Workers for heavy calculations

Performance Considerations:

Canvas rendering can be CPU-intensive
Implement frame-rate limiting
Use requestAnimationFrame for smooth updates

Conclusion

Building a web-based Excel editor is a complex undertaking that pushes the boundaries of web technology. The key to success lies in:

Smart Architecture: Layered canvas system with efficient rendering
Performance Focus: Viewport-based rendering and memory management
Real-time Collaboration: WebSocket integration with conflict resolution
Feature Completeness: Supporting Excel's extensive feature set
User Experience: Smooth interactions and responsive design

The end result should feel native and performant, rivaling desktop spreadsheet applications while leveraging the web's collaborative advantages.

Learning Resources

Books

Canvas and Graphics Programming:

"HTML5 Canvas" by Steve Fulton & Jeff Fulton - Comprehensive guide to 2D canvas programming
"Real-Time Rendering" by Tomas Akenine-Möller - Advanced graphics concepts applicable to canvas optimization
"Game Programming Patterns" by Robert Nystrom - Design patterns for performance-critical applications

Data Structures and Algorithms:

"Introduction to Algorithms" by Cormen, Leiserson, Rivest & Stein - Essential for viewport calculations and spatial indexing
"The Algorithm Design Manual" by Steven Skiena - Practical approaches to complex algorithmic problems

Collaborative Systems:

"Designing Data-Intensive Applications" by Martin Kleppmann - Modern approaches to distributed systems and consistency
"Building Microservices" by Sam Newman - Architecture patterns for scalable collaborative systems

JavaScript Performance:

"High Performance JavaScript" by Nicholas Zakas - Browser optimization techniques
"You Don't Know JS" series by Kyle Simpson - Deep JavaScript understanding for complex applications

Research Papers

Operational Transformation:

"Operational Transformation in Real-Time Group Editors" (1998) - Foundational OT paper
"Consistency Maintenance in Real-Time Collaborative Graphics Editing Systems" (2000)
"Real-Time Collaborative Editing Systems" by Chengzheng Sun (2002)

Collaborative Editing:

"High-Responsiveness for Group Editing CRDTs" (2016) - Modern alternatives to OT
"Conflict-free Replicated Data Types" by Marc Shapiro - CRDT fundamentals

User Interface Research:

"The Information Visualizer: An Information Workspace" (1991) - Spatial information organization
"Spreadsheet Parsing and Formula Understanding" - Research on formula parsing systems

Documentation and Specifications

Web Standards:

HTML5 Canvas Specification - Official canvas API documentation
WebSocket Protocol RFC 6455 - WebSocket specification for real-time communication
Excel File Format Documentation - Understanding Excel's binary formats

Browser APIs:

MDN Canvas API - Comprehensive canvas tutorials and examples
Web Workers API - For offloading computation
Intersection Observer API - Efficient viewport tracking

Open Source Projects

Spreadsheet Implementations:

Luckysheet - Full-featured online spreadsheet
EtherCalc - Web-based collaborative spreadsheet
OnlyOffice - Office suite with collaborative editing

Canvas Libraries:

Fabric.js - Canvas manipulation library with object model
Konva.js - High-performance 2D canvas library
Paper.js - Vector graphics scripting framework

Collaborative Editing:

ShareJS - Operational Transformation implementation
Yjs - CRDT-based collaborative editing framework
Fluid Framework - Microsoft's collaborative development platform

Online Courses and Tutorials

Canvas Programming:

MDN Canvas Tutorial - Step-by-step canvas introduction
HTML5 Canvas Deep Dive - Advanced canvas techniques

Performance Optimization:

Web.dev Performance - Google's performance optimization guides
JavaScript Performance Tips - Browser-specific optimization

Collaborative Systems:

Operational Transformation Explained - Interactive OT examples
CRDT Tutorial - Comprehensive CRDT learning resource

Mathematical Foundations

Linear Algebra:

Understanding coordinate transformations and matrix operations
Vector math for selection rectangles and spatial calculations

Graph Theory:

Dependency graphs for formula calculations
Shortest path algorithms for cell reference resolution

Distributed Systems Theory:

CAP Theorem implications for collaborative editing
Consistency models and their tradeoffs
Vector clocks and logical time

Tools and Development Environment

Performance Profiling:

Chrome DevTools Performance tab for canvas optimization
React DevTools Profiler - Component performance analysis
Web Vitals - User experience metrics

Testing Frameworks:

Jest with Canvas mock for unit testing
Playwright for end-to-end testing with canvas interactions
Testing Library for accessibility testing

Build Tools:

Vite for fast development builds
Rollup for optimized production bundles
TypeScript for type safety in complex applications

With love from the Toki Space team

Find us on tokispace dot com.

This tutorial represents the collective knowledge gained from building Toki's collaborative Excel editor. The resources above will help you dive deeper into any aspect that interests you. For questions or contributions, reach out to our engineering team. founder@tokispace.com

Forem: Augustus otu

Building a GPU-Accelerated File Explorer in 48 Hours with Kiro IDE

The Challenge

The Most Important Lesson: Know What You're Building

Enter Kiro IDE: A Game Changer

What is "Vibe Coding"?

Day 1: Architecture and Foundation

Planning with Specs

Key Architectural Decisions (Made by Me, Implemented with AI)

Building the Core

Day 2: Features and Polish

The Terminal Integration

Smart Folders

Theme System

The Numbers (Verified)

Performance

Feature List

Lessons Learned

1. Research Before You Build

2. Requirements Come From You

3. Specs Are Essential

4. Steering Maintains Quality

5. Iterate, Don't Dictate

6. Trust but Verify

The Human-AI Partnership

What's Next

Why I'm Now a Kiro Convert

Conclusion

Build your own online code compiler

Building an Online Code Compiler: A Complete Guide

Building an Online Code Compiler: A Complete Guide

Table of Contents

Architecture Overview

System Components

Core Technologies

Design Principles

Docker Container System

The Challenge of Multi-Language Execution

Container-Per-Language Architecture

Container Initialization

Language-Specific Execution

Message Queue Integration

Why RabbitMQ for Code Execution?

RabbitMQ Topology

Job Message Format

Producer Implementation (Node.js Backend)

Consumer Implementation (Go Runner Service)

Real-time Streaming

The Challenge of Live Output

Redis Pub/Sub for Streaming

WebSocket Integration

Enhanced Runner with Streaming

Backend Implementation

Node.js API Server

Express Middleware for Code Validation

Frontend Development

React Code Execution Component

Security & Isolation

The Security Challenge

Docker Security Model

Advanced Container Security

Code Sanitization

Network Isolation

Performance Optimization

Container Lifecycle Management

Memory Management

Load Balancing

Production Deployment

Docker Compose Production Setup

Kubernetes Deployment

Monitoring and Observability

Key Learnings

1. Container Management is Complex

2. Real-time Streaming Requires Careful Architecture

3. Security Cannot Be an Afterthought

4. Performance Optimization is Multi-Faceted

5. Production Reliability Requires Operational Excellence

6. Language-Specific Considerations

Conclusion

Learning Resources