Forem: emojiiii

Running DeepSeek Janus-Pro-1B in the Browser: A Comprehensive Guide

emojiiii — Tue, 28 Jan 2025 01:34:55 +0000

The ability to run large language models (LLMs) directly in the browser has opened new possibilities for privacy-preserving, client-side AI applications. In this blog post, we’ll explore how to run DeepSeek Janus-Pro-1B, a powerful text-to-image generation model, entirely in the browser using WebGPU and Hugging Face’s Transformers.js library.

Why Browser-Based Inference?

Privacy: Data never leaves the user’s device.
Cost Efficiency: No server infrastructure required.
Accessibility: Runs on any device with a modern browser and WebGPU support.

DeepSeek Janus-Pro-1B, designed for multimodal tasks like text-to-image generation, is now accessible via browser-based inference thanks to optimizations in Transformers.js and WebGPU acceleration.

Key Tools & Libraries

Transformers.js: A JavaScript port of Hugging Face’s Transformers library, optimized for browser execution.
WebGPU: A modern API for GPU acceleration in browsers, replacing WebGL with improved performance for ML workloads.
ONNX Runtime: Enables model execution via optimized computation graphs.

Demo Code Walkthrough

The following example demonstrates how to load and run DeepSeek Janus-Pro-1B in a Web Worker for non-blocking inference. The full code is available in the GitHub repository.

import {
  AutoProcessor,
  MultiModalityCausalLM,
  BaseStreamer,
  TextStreamer,
  InterruptableStoppingCriteria,
} from "@huggingface/transformers";

// Define constants
const IMAGE_GENERATION_COMMAND_PREFIX = "/imagine ";
const MAX_NEW_TEXT_TOKENS = 1024;

/**
 * Helper function to perform WebGPU feature detection
 */
let fp16_supported = false;
async function check() {
  try {
    const adapter = await navigator.gpu.requestAdapter();
    if (!adapter) {
      throw new Error("WebGPU is not supported (no adapter found)");
    }
    fp16_supported = adapter.features.has("shader-f16");
    self.postMessage({
      status: "success",
      data: fp16_supported,
    });
  } catch (e) {
    self.postMessage({
      status: "error",
      data: e.toString(),
    });
  }
}

/**
 * This class uses the Singleton pattern to enable lazy-loading of the pipeline
 */
class ImageGenerationPipeline {
  static model_id = "onnx-community/Janus-Pro-1B-ONNX";

  static async getInstance(progress_callback = null) {
    this.processor ??= AutoProcessor.from_pretrained(this.model_id, {
      progress_callback,
    });

    this.model ??= MultiModalityCausalLM.from_pretrained(this.model_id, {
      dtype: fp16_supported
        ? {
            prepare_inputs_embeds: "q4",
            language_model: "q4f16",
            lm_head: "fp16",
            gen_head: "fp16",
            gen_img_embeds: "fp16",
            image_decode: "fp32",
          }
        : {
            prepare_inputs_embeds: "fp32",
            language_model: "q4",
            lm_head: "fp32",
            gen_head: "fp32",
            gen_img_embeds: "fp32",
            image_decode: "fp32",
          },
      device: {
        prepare_inputs_embeds: "wasm", // TODO use "webgpu" when bug is fixed
        language_model: "webgpu",
        lm_head: "webgpu",
        gen_head: "webgpu",
        gen_img_embeds: "webgpu",
        image_decode: "webgpu",
      },
      progress_callback,
    });

    return Promise.all([this.processor, this.model]);
  }
}

class ProgressStreamer extends BaseStreamer {
  constructor(total, on_progress) {
    super();
    this.total = total;
    this.on_progress = on_progress;

    this.count = null;
    this.start_time = null;
  }

  put(value) {
    if (this.count === null) {
      // Ignore the first batch of tokens (prompt)
      this.count = 0;
      this.start_time = performance.now();
      return;
    }

    const progress = ++this.count / this.total;

    this.on_progress({
      count: this.count,
      total: this.total,
      progress,
      time: performance.now() - this.start_time,
    });
  }

  end() {
    /* no nothing */
  }
}

const stopping_criteria = new InterruptableStoppingCriteria();

async function generate(messages) {
  // For this demo, we only respond to the last message
  const message = messages.at(-1);

  // Tell the main thread we are starting
  self.postMessage({ status: "start" });

  // Load the pipeline
  const [processor, model] = await ImageGenerationPipeline.getInstance();

  // Determine if the user wants to generate an image or text
  if (message.content.startsWith(IMAGE_GENERATION_COMMAND_PREFIX)) {
    const text = message.content.replace(IMAGE_GENERATION_COMMAND_PREFIX, "");

    const conversation = [
      {
        role: "<|User|>", // uses title case
        content: text,
      },
    ];
    const inputs = await processor(conversation, {
      chat_template: "text_to_image",
    });

    const callback_function = (output) => {
      self.postMessage({
        status: "image-update",
        ...output,
      });
    };

    const num_image_tokens = processor.num_image_tokens;
    const streamer = new ProgressStreamer(num_image_tokens, callback_function);

    const outputs = await model.generate_images({
      ...inputs,
      min_new_tokens: num_image_tokens,
      max_new_tokens: num_image_tokens,
      do_sample: true,
      streamer,
    });

    const blob = await outputs[0].toBlob();

    // Send the output back to the main thread
    self.postMessage({
      status: "image-update",
      blob,
    });
  } else {
    const inputs = await processor(
      message.image
        ? [
            {
              role: "<|User|>",
              content: "<image_placeholder>\n" + message.content,
              images: [message.image],
            },
          ]
        : [
            {
              role: "<|System|>",
              content:
                "You are a helpful assistant. Answer the user's questions in a concise manner.",
            },
            {
              role: "<|User|>",
              content: message.content,
            },
          ],
    );

    let startTime;
    let numTokens = 0;
    let tps;
    const token_callback_function = () => {
      startTime ??= performance.now();

      if (numTokens++ > 0) {
        tps = (numTokens / (performance.now() - startTime)) * 1000;
      }
    };
    const callback_function = (output) => {
      self.postMessage({
        status: "text-update",
        output,
        tps,
        numTokens,
      });
    };

    const streamer = new TextStreamer(processor.tokenizer, {
      skip_prompt: true,
      skip_special_tokens: true,
      callback_function,
      token_callback_function,
    });

    // Generate response
    const outputs = await model.generate({
      ...inputs,
      max_new_tokens: MAX_NEW_TEXT_TOKENS,
      do_sample: false,
      streamer,
      stopping_criteria,
    });
  }

  // Tell the main thread we are done
  self.postMessage({
    status: "complete",
  });
}

async function load() {
  self.postMessage({
    status: "loading",
    data: "Loading model...",
  });

  // Load the pipeline and save it for future use.
  const [processor, model] = await ImageGenerationPipeline.getInstance((x) => {
    // We also add a progress callback to the pipeline so that we can
    // track model loading.
    self.postMessage(x);
  });

  self.postMessage({ status: "ready" });
}

// Listen for messages from the main thread
self.addEventListener("message", async (e) => {
  const { type, data } = e.data;

  switch (type) {
    case "check":
      check();
      break;

    case "load":
      load();
      break;

    case "generate":
      stopping_criteria.reset();
      generate(data);
      break;

    case "interrupt":
      stopping_criteria.interrupt();
      break;

    case "reset":
      stopping_criteria.reset();
      break;
  }
});

Running the Demo

Check out the live demo here: DeepSeek Janus-Pro-1B Browser Demo.

Key Features of the Demo:

Real-time progress updates during model loading and inference.
WebGPU-accelerated generation (requires Chrome 113+ or Edge 113+).
Full client-side execution—no data is sent to external servers.

Challenges & Optimizations

Model Quantization: The model is quantized to 8-bit to reduce its size and improve loading speed.
Memory Management: Web Workers prevent UI freezing during inference.
Browser Compatibility: WebGPU is still experimental but critical for performance.

Conclusion

Running DeepSeek Janus-Pro-1B in the browser showcases the potential of client-side AI. With tools like Transformers.js and WebGPU, complex models can now operate efficiently in constrained environments while preserving user privacy.

Next Steps:

Experiment with different prompts and model configurations.
Explore fine-tuning the model for domain-specific tasks.
Monitor WebGPU adoption to ensure broader compatibility.

For developers, this marks an exciting shift toward decentralized, user-centric AI applications. Dive into the example code and start building! 🚀

Running DeepSeek-R1 in the Browser: A Comprehensive Guide

emojiiii — Thu, 23 Jan 2025 14:22:55 +0000

As AI technology evolves, running sophisticated machine learning models directly within the browser is becoming increasingly feasible. This guide will walk you through how to load and use the DeepSeek-R1 model in a browser using JavaScript. We'll also cover the implementation details based on the example found here.

Why Run NLP Models in the Browser?

Traditionally, Natural Language Processing (NLP) models are deployed server-side, requiring internet connections for sending requests and receiving responses. However, with advancements like WebGPU and ONNX.js, it's now possible to run advanced models such as DeepSeek-R1 directly in the browser. The benefits include:

Enhanced Privacy: User data never leaves their device.
Reduced Latency: Eliminates delays associated with server communication.
Offline Availability: Operable even without an internet connection.

About DeepSeek-R1

DeepSeek-R1 is a lightweight yet efficient NLP model optimized for on-device inference. It offers high-quality text processing capabilities while maintaining a small footprint, making it ideal for browser environments.

Setting Up Your Project

Prerequisites

To get started with running the DeepSeek-R1 model in your browser, you'll need:

A modern browser that supports WebGPU/WebGL.
The @huggingface/transformers library for executing transformers models in JavaScript.
The script file containing the logic for loading and handling the DeepSeek-R1 model.

Demo: try it!.

Implementation Details

Here’s a step-by-step guide to loading and using the DeepSeek-R1 model in your browser:

import {
  AutoTokenizer,
  AutoModelForCausalLM,
  TextStreamer,
  InterruptableStoppingCriteria,
} from "@huggingface/transformers";

/**
 * Helper function to perform feature detection for WebGPU
 */
async function check() {
  try {
    const adapter = await navigator.gpu.requestAdapter();
    if (!adapter) {
      throw new Error("WebGPU is not supported (no adapter found)");
    }
  } catch (e) {
    self.postMessage({
      status: "error",
      data: e.toString(),
    });
  }
}

/**
 * This class uses the Singleton pattern to enable lazy-loading of the pipeline
 */
class TextGenerationPipeline {
  static model_id = "onnx-community/DeepSeek-R1-Distill-Qwen-1.5B-ONNX";

  static async getInstance(progress_callback = null) {
    if (!this.tokenizer) {
      this.tokenizer = await AutoTokenizer.from_pretrained(this.model_id, {
        progress_callback,
      });
    }

    if (!this.model) {
      this.model = await AutoModelForCausalLM.from_pretrained(this.model_id, {
        dtype: "q4f16",
        device: "webgpu",
        progress_callback,
      });
    }

    return [this.tokenizer, this.model];
  }
}

const stopping_criteria = new InterruptableStoppingCriteria();

let past_key_values_cache = null;

async function generate(messages) {
  // Retrieve the text-generation pipeline.
  const [tokenizer, model] = await TextGenerationPipeline.getInstance();

  const inputs = tokenizer.apply_chat_template(messages, {
    add_generation_prompt: true,
    return_dict: true,
  });

  const [START_THINKING_TOKEN_ID, END_THINKING_TOKEN_ID] = tokenizer.encode(
    "<think></think>",
    { add_special_tokens: false },
  );

  let state = "thinking"; // 'thinking' or 'answering'
  let startTime;
  let numTokens = 0;
  let tps;

  const token_callback_function = (tokens) => {
    startTime ??= performance.now();

    if (numTokens++ > 0) {
      tps = (numTokens / (performance.now() - startTime)) * 1000;
    }
    if (tokens[0] === END_THINKING_TOKEN_ID) {
      state = "answering";
    }
  };

  const callback_function = (output) => {
    self.postMessage({
      status: "update",
      output,
      tps,
      numTokens,
      state,
    });
  };

  const streamer = new TextStreamer(tokenizer, {
    skip_prompt: true,
    skip_special_tokens: true,
    callback_function,
    token_callback_function,
  });

  // Tell the main thread we are starting
  self.postMessage({ status: "start" });

  const { past_key_values, sequences } = await model.generate({
    ...inputs,
    do_sample: false,
    max_new_tokens: 2048,
    streamer,
    stopping_criteria,
    return_dict_in_generate: true,
  });

  past_key_values_cache = past_key_values;

  const decoded = tokenizer.batch_decode(sequences, {
    skip_special_tokens: true,
  });

  // Send the output back to the main thread
  self.postMessage({
    status: "complete",
    output: decoded,
  });
}

async function load() {
  self.postMessage({
    status: "loading",
    data: "Loading model...",
  });

  // Load the pipeline and save it for future use.
  const [tokenizer, model] = await TextGenerationPipeline.getInstance((x) => {
    self.postMessage(x);
  });

  self.postMessage({
    status: "loading",
    data: "Compiling shaders and warming up model...",
  });

  // Run model with dummy input to compile shaders
  const inputs = tokenizer("a");
  await model.generate({ ...inputs, max_new_tokens: 1 });
  self.postMessage({ status: "ready" });
}

// Listen for messages from the main thread
self.addEventListener("message", async (e) => {
  const { type, data } = e.data;

  switch (type) {
    case "check":
      check();
      break;

    case "load":
      load();
      break;

    case "generate":
      stopping_criteria.reset();
      generate(data);
      break;

    case "interrupt":
      stopping_criteria.interrupt();
      break;

    case "reset":
      past_key_values_cache = null;
      stopping_criteria.reset();
      break;
  }
});

Key Points

Feature Detection: The check function performs feature detection to ensure WebGPU support.
Singleton Pattern: The TextGenerationPipeline class ensures that the tokenizer and model are loaded only once, preventing redundant initialization.
Model Loading: The getInstance method loads the tokenizer and model from a pre-trained source, supporting progress callbacks.
Inference: The generate function processes input and generates text output, using TextStreamer for streaming tokens.
Communication: The worker listens for messages from the main thread and executes corresponding actions based on message types (e.g., check, load, generate, interrupt, reset).

Conclusion

Running NLP models like DeepSeek-R1 in the browser marks a significant advancement in enhancing user experiences and protecting data privacy. With just a few lines of JavaScript and the power of the @huggingface/transformers library, you can develop responsive and powerful applications. Whether you're building interactive tools or intelligent assistants, browser-based NLP can be a game-changer.

Explore the potential of DeepSeek-R1 in the browser and start crafting smarter front-end applications today!

This guide provides a comprehensive overview of how to load and use the DeepSeek-R1 model in a browser environment, complete with detailed code examples. For more specific implementation details, refer to the linked GitHub repository.

Running Kokoro-82M ONNX TTS Model in the Browser

emojiiii — Fri, 17 Jan 2025 03:38:56 +0000

The advancements in AI and machine learning have significantly expanded the boundaries of what can be accomplished in-browser. Running a text-to-speech (TTS) model directly in the browser opens up opportunities for privacy, speed, and convenience. In this blog post, we'll explore how to run the Kokoro-82M ONNX TTS model in the browser using a JavaScript implementation. If you're curious, you can test this in action at my demo: Kitt AI Text-to-Speech.

Why Run TTS Models in the Browser?

Traditionally, TTS models are executed on servers, requiring an internet connection to send input and receive synthesized speech. However, with the growing capability of WebGPU and ONNX.js, you can now run advanced models like Kokoro-82M ONNX directly in the browser. This brings several advantages:

Privacy: Your text data never leaves your device.
Low Latency: Eliminate server communication delays.
Offline Access: Operate even without an active internet connection.

Overview of Kokoro-82M ONNX

The Kokoro-82M ONNX model is a lightweight yet effective TTS model optimized for on-device inference. It provides high-quality speech synthesis while maintaining a small footprint, making it suitable for browser environments.

Setting Up the Project

Prerequisites

To run Kokoro-82M ONNX in the browser, you’ll need:

A modern browser with WebGPU/WebGL support.
The ONNX.js library for running ONNX models in JavaScript.
The Kokoro.js script, which simplifies loading and processing the Kokoro-82M model.

Installation

You can set up the project by including the necessary dependencies in your package.json:

{
  "dependencies": {
    "@huggingface/transformers": "^3.3.1"
  }
}

Next, ensure you have the Kokoro.js script, which can be obtained from this repository.

Loading the Model

To load and use the Kokoro-82M ONNX model in your browser, follow these steps:

this.model_instance = StyleTextToSpeech2Model.from_pretrained(
    this.modelId,
    {
        device: "wasm",
        progress_callback,
    }
);
this.tokenizer = AutoTokenizer.from_pretrained(this.modelId, {
   progress_callback,
});

Running Inference

Once the model is loaded and the text is processed, you can run the inference to generate speech:

const language = speakerId.at(0); // "a" or "b"
const phonemes = await phonemize(text, language);
const { input_ids } = await tokenizer(phonemes, { truncation: true });
const num_tokens = Math.max(
   input_ids.dims.at(-1) - 2, // Without padding;
   0
);
const offset = num_tokens * STYLE_DIM;
const data = await getVoiceData(speakerId as keyof typeof VOICES);
const voiceData = data.slice(offset, offset + STYLE_DIM);
const inputs = {
   input_ids,
   style: new Tensor("float32", voiceData, [1, STYLE_DIM]),
   speed: new Tensor("float32", [speed], [1]),
};

const { waveform } = await model(inputs);
const audio = new RawAudio(waveform.data, SAMPLE_RATE).toBlob();

Demo

You can see this in action on my live demo: Kitt AI Text-to-Speech. The demo showcases real-time text-to-speech synthesis powered by Kokoro-82M ONNX.

Conclusion

Running a TTS model like Kokoro-82M ONNX in the browser represents a leap forward in privacy-preserving and low-latency applications. With just a few lines of JavaScript and the power of ONNX.js, you can create high-quality, responsive TTS applications that delight users. Whether you’re building accessible tools, voice assistants, or interactive applications, in-browser TTS can be a game-changer.

Try the Kitt AI Text-to-Speech demo today and experience it for yourself!

References

Shocking! AI Instantly Removes Backgrounds in Your Browser-Netizens Exclaim: This is Absolutely Magical!

emojiiii — Sun, 12 Jan 2025 08:50:04 +0000

Removing backgrounds from images is a crucial task for many applications, such as e-commerce, graphic design, and social media content creation. With the power of AI and JavaScript, you can build an efficient background remover directly in a web application, enabling real-time processing without relying on server-side computations. In this post, we'll walk through how to create this solution using React and Transformers.js.

Why React and Transformers.js?

React is a popular JavaScript library for building dynamic, responsive user interfaces. Its component-based architecture promotes reusability and efficient development. Transformers.js, on the other hand, provides pre-trained AI models that run directly in the browser. By utilizing technologies like WebGPU and WebAssembly, Transformers.js enables on-device, real-time AI processing—perfect for interactive applications like background removal.

This combination ensures a seamless user experience with low latency and high interactivity.

Core Concepts of Background Removal

Background removal relies on image segmentation, a technique that identifies and separates the foreground (subject) from the background. To achieve this, we use two powerful pre-trained models available in Transformers.js: RMBG-1.4 and ModNet.

RMBG-1.4 is optimized for real-time background removal and works well with various types of images, offering high accuracy for dynamic backgrounds.
ModNet, designed for human segmentation, excels in precisely removing backgrounds from images containing people, making it perfect for portraits and fashion photos.

These models are trained on large datasets, making them versatile and highly effective at handling diverse image inputs.

High-Level Workflow

Image Upload: Users upload an image via a file input component in the UI.
AI Processing: The uploaded image is passed to a segmentation model from Transformers.js (RMBG-1.4 or ModNet), which identifies the foreground.
Mask Application: The model outputs a mask that highlights the foreground, which is then applied to remove the background.
Result Display: The processed image is shown in a preview area for users to view the results.
Batch Processing and Editing: Users can upload multiple images, process them sequentially, edit individual images, and download the results in bulk.

Key Features and Implementation

Set Up the React Application:
Begin by creating a React project using create-react-app or Vite, and install the required dependencies, including Transformers.js.
Integrate Transformers.js:
Import Transformers.js into your project and load the RMBG-1.4 or ModNet models. These models can run entirely in the browser, utilizing WebGPU for efficient processing on supported devices.
Build the User Interface:
Create an intuitive UI with the following components:
- A file input for users to upload images.
- A preview area to display the processed image.
- Options for individual image editing (e.g., refining edges or adjusting the foreground) and bulk download functionality.
Optimize AI Processing:
- Web Workers: Use Web Workers to offload heavy model computations, ensuring smooth UI performance without blocking the main thread.
- Queue Management: Implement a queue system to handle batch uploads and ensure sequential processing without overwhelming the browser.
Enhance Usability:
- Allow users to interact with individual images by editing them, such as adjusting the foreground edges.
- Provide bulk download functionality so users can download all processed images in one click.
Host Your Application:
Deploy your app to a hosting platform, and provide a live demo to showcase the features in action.

Challenges and Optimizations

Performance: Running AI models in the browser can be resource-intensive. WebGPU optimizes performance on supported devices, while Web Workers prevent UI freezing during processing.
Scalability: A batch processing queue ensures scalability, handling multiple image uploads without degrading performance.
Accuracy: Although the pre-trained models like RMBG-1.4 and ModNet are highly accurate, fine-tuning or allowing manual adjustments can enhance segmentation quality for complex or edge cases.

Project Setup and Dependencies

First, let's set up our project with the necessary dependencies:

# Create a new React project with Vite
pnpm create vite background-remover -- --template react-ts

# Navigate to project directory
cd background-remover

# Install dependencies
pnpm install @huggingface/transformers @emotion/react @emotion/styled @mui/material react-dropzone file-saver jszip
pnpm install -D @types/file-saver @types/jszip

Core Implementation

Let's break down the implementation into key components:

1. Model and Environment Setup

import {
  env,
  AutoModel,
  AutoProcessor,
  RawImage,
} from "@huggingface/transformers";

// Configure environment for optimal performance
const setupEnvironment = () => {
  // Disable WASM proxy for better performance
  env.backends.onnx.wasm.proxy = false;

  // Check WebGPU support
  if (!navigator.gpu) {
    throw new Error("WebGPU is not supported in this browser.");
  }
};

// Initialize model and processor
const initializeModel = async () => {
  try {
    const model = await AutoModel.from_pretrained("Xenova/modnet", {
      device: "webgpu",
    });
    const processor = await AutoProcessor.from_pretrained("Xenova/modnet");

    return { model, processor };
  } catch (error) {
    console.error("Failed to initialize model:", error);
    throw error;
  }
};

2. Image Processing Component with Advanced Features

import React, { useState, useCallback, useEffect, useRef } from "react";
import { useDropzone } from "react-dropzone";
import { Box, Button, CircularProgress, Stack, Typography } from "@mui/material";
import JSZip from "jszip";
import { saveAs } from "file-saver";

interface ProcessedImage {
  id: string;
  original: string;
  processed: string;
  name: string;
}

const BackgroundRemover: React.FC = () => {
  const [model, setModel] = useState<any>(null);
  const [processor, setProcessor] = useState<any>(null);
  const [images, setImages] = useState<ProcessedImage[]>([]);
  const [isProcessing, setIsProcessing] = useState(false);
  const [progress, setProgress] = useState(0);
  const processingQueue = useRef<ProcessedImage[]>([]);

  // Initialize model on component mount
  useEffect(() => {
    (async () => {
      try {
        setupEnvironment();
        const { model: m, processor: p } = await initializeModel();
        setModel(m);
        setProcessor(p);
      } catch (error) {
        console.error("Initialization error:", error);
      }
    })();
  }, []);

  // Handle file drops
  const onDrop = useCallback(async (acceptedFiles: File[]) => {
    if (!model || !processor) return;

    setIsProcessing(true);
    setProgress(0);

    for (let i = 0; i < acceptedFiles.length; i++) {
      const file = acceptedFiles[i];
      try {
        // Process image
        const img = await RawImage.fromURL(URL.createObjectURL(file));
        const { pixel_values } = await processor(img);
        const { output } = await model({ input: pixel_values });

        // Create mask and apply it
        const maskData = await RawImage.fromTensor(
          output[0].mul(255).to("uint8")
        ).resize(img.width, img.height);

        // Create final image with transparent background
        const canvas = document.createElement("canvas");
        canvas.width = img.width;
        canvas.height = img.height;
        const ctx = canvas.getContext("2d")!;

        // Draw original image
        ctx.drawImage(img.toCanvas(), 0, 0);

        // Apply mask
        const imageData = ctx.getImageData(0, 0, img.width, img.height);
        for (let j = 0; j < maskData.data.length; j++) {
          imageData.data[j * 4 + 3] = maskData.data[j];
        }
        ctx.putImageData(imageData, 0, 0);

        // Add to processed images
        const processedImage: ProcessedImage = {
          id: crypto.randomUUID(),
          original: URL.createObjectURL(file),
          processed: canvas.toDataURL("image/png"),
          name: file.name,
        };

        setImages(prev => [...prev, processedImage]);
        setProgress(((i + 1) / acceptedFiles.length) * 100);
      } catch (error) {
        console.error(`Error processing ${file.name}:`, error);
      }
    }

    setIsProcessing(false);
  }, [model, processor]);

  // Dropzone configuration
  const { getRootProps, getInputProps, isDragActive } = useDropzone({
    onDrop,
    accept: {
      "image/*": [".png", ".jpg", ".jpeg", ".webp"],
    },
    disabled: isProcessing || !model,
  });

  // Download all processed images
  const downloadAll = async () => {
    const zip = new JSZip();

    images.forEach((img, index) => {
      // Convert base64 to blob
      const base64Data = img.processed.split(",")[1];
      const byteCharacters = atob(base64Data);
      const byteArrays = new Uint8Array(byteCharacters.length);

      for (let i = 0; i < byteCharacters.length; i++) {
        byteArrays[i] = byteCharacters.charCodeAt(i);
      }

      const blob = new Blob([byteArrays], { type: "image/png" });
      zip.file(`${img.name.split(".")[0]}_processed.png`, blob);
    });

    const content = await zip.generateAsync({ type: "blob" });
    saveAs(content, "processed_images.zip");
  };

  return (
    <Box sx={{ maxWidth: "1200px", margin: "0 auto", padding: 3 }}>
      <div
        {...getRootProps()}
        style={{
          border: "2px dashed #666",
          borderRadius: 8,
          padding: 20,
          textAlign: "center",
          cursor: isProcessing ? "not-allowed" : "pointer",
          backgroundColor: isDragActive ? "rgba(25, 118, 210, 0.08)" : undefined,
        }}
      >
        <input {...getInputProps()} />
        <Typography variant="h6">
          {isProcessing
            ? `Processing... ${Math.round(progress)}%`
            : "Drag and drop images here, or click to select files"}
        </Typography>
        {isProcessing && <CircularProgress sx={{ mt: 2 }} />}
      </div>

      {images.length > 0 && (
        <Stack spacing={2} sx={{ mt: 3 }}>
          <Button
            variant="contained"
            onClick={downloadAll}
            disabled={isProcessing}
          >
            Download All Processed Images
          </Button>

          <Box
            sx={{
              display: "grid",
              gridTemplateColumns: "repeat(auto-fill, minmax(250px, 1fr))",
              gap: 2,
            }}
          >
            {images.map((img) => (
              <Box
                key={img.id}
                sx={{
                  border: "1px solid #ddd",
                  borderRadius: 1,
                  padding: 1,
                }}
              >
                <img
                  src={img.processed}
                  alt="Processed"
                  style={{
                    width: "100%",
                    height: "200px",
                    objectFit: "contain",
                  }}
                />
                <Typography variant="caption" display="block" textAlign="center">
                  {img.name}
                </Typography>
              </Box>
            ))}
          </Box>
        </Stack>
      )}
    </Box>
  );
};

export default BackgroundRemover;

Conclusion

By leveraging React and Hugging Face's Transformers.js library, we've built a powerful background removal application that runs entirely in the browser. The implementation takes advantage of modern web technologies like WebGPU for optimal performance and provides a smooth user experience with features like drag-and-drop uploads, batch processing, and bulk downloads.

The complete source code is available on GitHub, and you can try out the live demo here. We encourage you to experiment with different models and optimizations to enhance the application further.

References

How to Build a Text-to-Image Generator with React and Transformers.js

emojiiii — Sun, 12 Jan 2025 08:49:04 +0000

Building a text-to-image generator has become one of the most exciting advancements in AI, as it merges natural language processing with image generation techniques. In this blog post, we will discuss how to create your own text-to-image application using React and Transformers.js, leveraging the Janus-1.3B ONNX model for the task.

What Is Text-to-Image Generation?

Text-to-image generation is the process where a machine learning model generates an image based on a textual description. This involves understanding natural language and translating it into a visual output. The task requires a powerful model capable of handling complex language-to-visual mappings.

Tools and Technologies

In this project, we use React for building the frontend interface, Transformers.js for leveraging pre-trained models like Janus-1.3B (an ONNX model designed for generating images from text), and ONNX Runtime to run the model in the browser. The Janus-1.3B model offers excellent performance in generating realistic images based on textual input, making it ideal for this use case.

Additionally, you can access the demo for the text-to-image generator at here.

Setting Up the Project

The first step is setting up a React project. If you don't already have a React environment, you can create one with tools like Create React App.

Once your project is set up, you need to install the necessary dependencies, such as Transformers.js, ONNX Runtime Web, and other UI libraries. The Transformers.js library provides an easy interface for integrating transformer models into your web applications.

pnpm add @huggingface/transformers

Integrating the Janus-1.3B Model

The Janus-1.3B ONNX model is what powers the text-to-image functionality. The model has been fine-tuned for text-to-image tasks, offering high-quality results from text descriptions. To integrate the model, you would first load the model with ONNX Runtime and then pass the text input to the model, which will generate the corresponding image.

The setup process involves initializing the model and performing inference, where the model processes your text and generates an image. For this purpose, Transformers.js will handle communication with the ONNX model, making the process smooth and straightforward.

UI Design and Interaction

The user interface of a text-to-image generator needs to be simple and intuitive. Using React, you can design a form where users can input a description of the image they want to generate. Once the user submits the description, the text is sent to the backend (running the ONNX model) for processing. The generated image is then displayed back on the frontend.

For a better experience, you can add loading animations and error handling, so users are aware of the ongoing process and any potential issues.

Optimizing Performance

Running an AI model like Janus-1.3B directly in the browser can be intensive. Therefore, optimizing the model for performance is crucial. Using ONNX Runtime in the browser ensures that the model is loaded efficiently. Additionally, compressing images and using batching techniques can speed up the process. You should also consider the browser's limitations when handling large models, ensuring smooth user experiences across devices.

Conclusion

Building a text-to-image generator with React and Transformers.js using the Janus-1.3B ONNX model is a fascinating project that blends multiple technologies together to create an immersive AI-powered application. Whether you're looking to generate realistic images from text or learn more about integrating AI models in web applications, this project will help you understand the power of modern machine learning in a web environment.

To explore the live demo, visit here.

References

How to Build a Speech-to-Text App with React and Transformers.js

emojiiii — Sun, 12 Jan 2025 08:48:24 +0000

Building a speech-to-text application is an exciting way to explore the potential of modern web technologies, especially when combined with AI-powered models. In this blog, I will walk you through the steps of building a speech-to-text app using React and Transformers.js, leveraging the Whisper and Moonshine models. My demo is available at https://kitt.tools/ai/speech-to-text for you to try out.

Step 1: Setting Up React and Dependencies

To begin, you will need a React project. If you don't have one already, you can easily set up a new React app by running:

npx create-react-app speech-to-text
cd speech-to-text

Once your React environment is set up, you'll need to integrate the necessary dependencies, including Transformers.js and the models Whisper and Moonshine. Transformers.js allows you to run machine learning models directly in the browser, making it perfect for building lightweight, client-side AI applications.

Whisper, an automatic speech recognition (ASR) model from OpenAI, is known for its high accuracy in transcribing speech to text. Moonshine, a tool that enhances the quality of audio input, provides noise reduction and better handling of challenging audio environments, improving Whisper's transcription results.

Install the required dependencies with pnpm:

pnpm add @huggingface/transformers

Step 2: Integrating Whisper for Speech Recognition

The core of our speech-to-text functionality lies in Whisper. With its pre-trained model, Whisper can recognize speech in various languages and transcribe it into text. To use Whisper with Transformers.js, you'll integrate it into a React component that handles audio input.

Create an AudioRecorder component where users can start recording their speech. Use the navigator.mediaDevices.getUserMedia() API to capture audio from the user's microphone.

Once the audio is captured, send it to the Whisper model for transcription. Transformers.js makes it easy to interact with Whisper's model, requiring just a few lines of code. After processing, the transcribed text is displayed in your React app.

Step 3: Enhancing Audio with Moonshine

To ensure the best transcription accuracy, you can use Moonshine to enhance the raw audio input. Moonshine helps reduce background noise, making the speech clearer, which in turn improves the accuracy of Whisper's transcriptions.

In your app, you can apply Moonshine to the audio stream before passing it to Whisper. This enhancement is particularly useful in noisy environments, ensuring that Whisper can focus on the speech while filtering out unwanted sounds. Moonshine works seamlessly with Transformers.js, providing an easy integration.

Step 4: Implementing the Speech-to-Text Flow

Now that Whisper and Moonshine are integrated, build the main flow for your application:

Capture audio: Use the microphone to capture user speech.
Enhance audio: Apply Moonshine to the captured audio for noise reduction.
Transcribe speech: Pass the processed audio to Whisper for transcription.
Display transcription: Show the transcribed text in real-time on your React app.

React's state management will help you update the UI dynamically as the transcription process progresses.

Step 5: Testing and Deployment

After implementing the speech-to-text flow, thoroughly test your app with various types of audio inputs. Experiment with different environments to evaluate how well Moonshine handles background noise and how accurate Whisper’s transcription is. You can check out my live demo at this link.

Conclusion

Building a speech-to-text app with React and Transformers.js is an exciting way to combine cutting-edge AI with modern web technologies. By using Whisper for speech recognition and Moonshine for audio enhancement, you can create a powerful, client-side solution that transcribes speech to text in real-time, directly in the browser.

References

How to Build an Image Processor with React and Transformers.js

emojiiii — Sun, 12 Jan 2025 08:47:05 +0000

The Technology Stack

React: The front-end framework I used to build the user interface. It offers a fast, declarative, and component-based approach to building web applications, which is perfect for this project.
Transformers.js: This JavaScript library allows me to run transformer-based models like the ones for image segmentation (such as RMBG-1.4 and ModNet) directly in the browser.
wasm-vips: A fast image processing library that compiles to WebAssembly (WASM). It is used for high-performance image processing tasks, enabling fast loading and manipulation of large images.
Web Workers & Queues: To handle batch processing efficiently, I used Web Workers along with a queue system. This helps offload the image processing task to background threads, ensuring that the user interface remains responsive even during heavy computations.

The Core Features

Background Removal:
The core functionality of the app is background removal. I leverage Transformers.js and WASM-optimized models like RMBG-1.4 for segmentation. These models are run directly in the browser, which eliminates the need for server-side processing and ensures fast, private operations.
Batch Processing and Downloads:
For users who want to process multiple images at once, I implemented a queue system. The idea is to allow users to upload several images and process them in parallel, rather than one at a time. Workers are utilized to handle these tasks in the background. This keeps the main thread free, preventing the user interface from freezing while large batches are being processed.
Image Processing with wasm-vips:
wasm-vips is used to efficiently manipulate image data. By leveraging WebAssembly, the library runs directly in the browser, offering significant performance improvements over traditional JavaScript methods. It’s particularly effective when dealing with tasks like resizing and cropping images, which are common when preparing images for background removal.
User Interface:
The user interface was designed with simplicity and performance in mind. React provides a smooth user experience, allowing real-time feedback during image processing. Users can drag and drop images for processing, view progress bars for each image, and download processed files in a batch once they’re ready.

Performance Optimization with Workers

One of the biggest challenges of image processing in the browser is performance. For batch processing, I set up a queue to manage tasks and utilized Web Workers to execute the heavy lifting in separate threads. This allows the main thread (and thus the UI) to remain responsive, even when processing a large batch of images. Workers also make it easier to manage concurrent tasks and prevent blocking.

When a user uploads images, they are placed in the queue, and a worker processes each image one by one, removing the background and storing the processed image in a temporary storage area. Once all images are processed, the user can download them in a single zip file.

Demo & Conclusion

You can try out the demo at Kitt Tools. The app showcases how easy it can be to remove backgrounds from images directly in the browser, without relying on a backend server. With the help of React, Transformers.js, wasm-vips, and Web Workers, I was able to create an efficient, scalable solution for handling image processing tasks in the browser.

This approach not only improves the speed and efficiency of the processing but also ensures that users have a seamless experience without waiting for server-side computations.

The next steps for this project could involve improving the user interface, adding more image manipulation features, or supporting more advanced AI models for different types of image processing tasks.

References

React Documentation: https://reactjs.org/docs/getting-started.html
Transformers.js Documentation: https://huggingface.co/docs/transformers/main/en/installation
wasm-vips GitHub Repository: https://github.com/libvips/wasm-vips
Web Workers Documentation: https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers
Queueing in JavaScript: https://developer.mozilla.org/en-US/docs/Web/API/HTML5_Web_Worker_API/Queueing

Building an AI-Powered Background Remover with React and Transformers.js

emojiiii — Sun, 12 Jan 2025 07:01:37 +0000

Background removal is a common task in image processing that has traditionally required complex desktop software or cloud-based services. However, with recent advances in web technologies and AI models, it's now possible to build a powerful background remover that runs entirely in the browser. In this tutorial, we'll explore how to create such a tool using React, Transformers.js, and state-of-the-art AI models.

Try Remove Background Now!

Key Features

🚀 Client-side processing - no server uploads needed
🎯 Support for multiple AI models (RMBG-1.4 and ModNet)
📦 Batch processing capabilities
🎨 Built-in image editor for post-processing
🔒 Privacy-focused - all processing happens locally

Technical Architecture

The application is built with several key components:

Frontend UI: React with TypeScript for type safety
AI Processing: Transformers.js for running AI models
Worker Thread: Web Workers for non-blocking processing
State Management: React hooks for local state management

Implementation Details

Setting Up the Models

We use two different models for background removal:

type ModelType = "briaai/RMBG-1.4" | "Xenova/modnet";

RMBG-1.4 is our recommended model for better quality, while ModNet serves as an alternative option. Both models are loaded and run entirely in the browser using Transformers.js.

Core Components

The main component structure consists of three key areas:

Upload Area: Handles file input and model selection
Edit Area: Displays the processed image with editing capabilities
Image List: Shows all uploaded images and their processing status

Worker Thread Implementation

To keep the UI responsive during image processing, we use a Web Worker:

const useTask = (onImageProcessed?: (id: string) => void) => {
  const [files, setFiles] = useState<FileWithMoreInfo[]>([]);

  const { worker, isModelLoading } = useWorker(
    (event: WorkerResponseMessageEvent) => {
      const { type, data, id, status } = event.data;

      switch (type) {
        case WorkerResponseTaskType.REMOVE_BACKGROUND_COMPLETE:
          // Update UI with processed image
          break;
      }
    }
  );

  // ... task management logic
};

Processing Pipeline

User uploads an image(s)
Images are queued for processing
Worker thread loads the selected AI model
Background removal is performed
Processed images are displayed with transparent backgrounds

Post-Processing Features

After background removal, users can:

Rotate the image
Add text or stickers
Apply filters
Download individual images or batch download as ZIP

Performance Considerations

Models are cached after first load
Processing happens in chunks to prevent UI freezing
Images are processed sequentially in batch uploads
Preview thumbnails are generated efficiently

Future Improvements

Support for more AI models
Advanced editing features
Background replacement options
Batch processing optimization
Export in different formats

Conclusion

Building a browser-based background remover demonstrates how far web technologies have come. By leveraging modern frameworks and AI models, we can create powerful image processing tools that run entirely on the client side, ensuring both performance and privacy.

The complete source code showcases how to structure such an application, handle complex image processing tasks, and provide a smooth user experience. Feel free to explore and adapt this implementation for your own projects!

Forem: emojiiii

Running DeepSeek Janus-Pro-1B in the Browser: A Comprehensive Guide

Why Browser-Based Inference?

Key Tools & Libraries

Demo Code Walkthrough

Running the Demo

Challenges & Optimizations

Conclusion

Running DeepSeek-R1 in the Browser: A Comprehensive Guide

Why Run NLP Models in the Browser?

About DeepSeek-R1

Setting Up Your Project

Prerequisites

Implementation Details

Key Points

Conclusion

Running Kokoro-82M ONNX TTS Model in the Browser

Why Run TTS Models in the Browser?

Overview of Kokoro-82M ONNX

Setting Up the Project

Prerequisites

Installation

Loading the Model

Running Inference

Demo

Conclusion

References

Shocking! AI Instantly Removes Backgrounds in Your Browser-Netizens Exclaim: This is Absolutely Magical!

Why React and Transformers.js?

Core Concepts of Background Removal

High-Level Workflow

Key Features and Implementation

Challenges and Optimizations

Project Setup and Dependencies

Core Implementation

1. Model and Environment Setup

2. Image Processing Component with Advanced Features

Conclusion

References

How to Build a Text-to-Image Generator with React and Transformers.js

What Is Text-to-Image Generation?

Tools and Technologies

Setting Up the Project

Integrating the Janus-1.3B Model

UI Design and Interaction

Optimizing Performance

Conclusion

References

How to Build a Speech-to-Text App with React and Transformers.js

Step 1: Setting Up React and Dependencies

Step 2: Integrating Whisper for Speech Recognition

Step 3: Enhancing Audio with Moonshine

Step 4: Implementing the Speech-to-Text Flow

Step 5: Testing and Deployment

Conclusion

References

How to Build an Image Processor with React and Transformers.js

The Technology Stack

The Core Features

Performance Optimization with Workers

Demo & Conclusion

References

Building an AI-Powered Background Remover with React and Transformers.js

Key Features

Technical Architecture

Implementation Details

Setting Up the Models

Core Components

Worker Thread Implementation

Processing Pipeline

Post-Processing Features

Performance Considerations

Future Improvements

Conclusion

Resources