Forem: Ahamed Musthafa R S (amrs-tech)

How to use Grok Imagine API via fal.ai

Ahamed Musthafa R S (amrs-tech) — Fri, 30 Jan 2026 06:42:26 +0000

Generate cinematic AI videos from text prompts using xAI’s latest model and Python.

A command-line tool that takes a text prompt, sends it to xAI’s Grok Imagine model via fal.ai ‘s API, monitors the generation in real-time, and downloads the finished video, complete with synchronized audio.

Prerequisites

Before starting, ensure you have:

Python 3.8+ installed
A fal.ai account with API key from the Dashboard

Step 1: Environment Setup

Install the required packages:

pip install fal-client python-dotenv requests

Create a .env file in your project directory:

FAL_KEY=your_api_key_here

Now let’s build the application piece by piece.

Step 2: Imports and Configuration

import asyncio
import os
from pathlib import Path
import fal_client
import requests
from dotenv import load_dotenv
load_dotenv()

We use asyncio because fal.ai‘s client is async-first, video generation takes time, and async patterns let us stream progress updates while waiting. The load_dotenv() call pulls your FAL_KEY into the environment automatically.

Step 3: The Core Generation Function

This is where the magic happens. We’ll define an async function that handles the entire pipeline:

async def generate_and_download_video(
    prompt: str,
    output_dir: str = "videos",
    duration: int = 6,
    aspect_ratio: str = "16:9",
    resolution: str = "480p",
) -> str:
    """
    Generate a video using Grok Imagine and download it.Args:
        prompt: Text description of the desired video
        output_dir: Directory to save the downloaded video
        duration: Video duration in seconds (default: 6)
        aspect_ratio: 16:9, 4:3, 3:2, 1:1, 2:3, 3:4, or 9:16
        resolution: Output resolution - 480p or 720p
    Returns:
        Path to the downloaded video file, or None on failure
    """
    Path(output_dir).mkdir(parents=True, exist_ok=True)
    print(f"🎬 Generating video with prompt: {prompt}")
    print(f"   Duration: {duration}s | Aspect Ratio: {aspect_ratio} | Resolution: {resolution}")
    print("-" * 60)

The function accepts configurable parameters with sensible defaults. We create the output directory if it doesn’t exist, then print the configuration for visibility.

Step 4: Submitting the Request

# Submit the async request to Grok Imagine
    handler = await fal_client.submit_async(
        "xai/grok-imagine/text-to-video",
        arguments={
            "prompt": prompt,
            "duration": duration,
            "aspect_ratio": aspect_ratio,
            "resolution": resolution,
        },
    )
    print(f"📤 Request submitted. Request ID: {handler.request_id}")
    print("⏳ Waiting for video generation...")

submit_async sends your request to fal.ai‘s queue and returns immediately with a handler. The request_id is useful for debugging and tracking jobs. The model endpoint xai/grok-imagine/text-to-video tells fal.ai which model to run.

Available parameters:

Parameter Options Notes

duration 6–15 seconds
Longer = slower generation
aspect_ratio 16:9, 9:16, 1:1, 4:3, 3:4, 3:2, 2:3
9:16 for vertical/mobile resolution 480p, 720p
720p = better quality, slower

Step 5: Real-Time Progress Monitoring

# Stream real-time logs from the GPU workers
    async for event in handler.iter_events(with_logs=True):
        if hasattr(event, "message"):
            print(f"   📝 {event.message}")
        elif hasattr(event, "logs"):
            for log in event.logs:
                print(f"   📋 {log.get('message', log)}")

This is one of fal.ai ‘s best features. Instead of waiting blindly, iter_events opens a WebSocket connection and streams status updates from the GPU workers. You’ll see messages like “Enqueuing”, “Processing”, and generation progress in real-time.

Step 6: Retrieving Results

# Retrieve the completed result
    result = await handler.get()
    print("-" * 60)
    print("✅ Video generation complete!")
# Extract video metadata
    video_info = result.get("video", {})
    video_url = video_info.get("url")
    file_name = video_info.get("file_name", "grok_video.mp4")
    print(f"   📊 Resolution: {video_info.get('width')}x{video_info.get('height')}")
    print(f"   ⏱️  Duration: {video_info.get('duration', 'N/A')}s")
    print(f"   🎞️  FPS: {video_info.get('fps', 'N/A')}")
    print(f"   📹 Frames: {video_info.get('num_frames', 'N/A')}")

Once generation completes, handler.get() returns the full response. The API provides rich metadata including actual dimensions, frame count, and a temporary URL to download the video.

Response structure:

{
  "video": {
    "url": "https://...",
    "file_name": "generated_video.mp4",
    "width": 854,
    "height": 480,
    "duration": 6.0,
    "fps": 24,
    "num_frames": 144
  }
}

Step 7: Downloading with Progress

# Download with progress tracking
    if video_url:
        output_path = Path(output_dir) / file_name
        print(f"\n📥 Downloading video to: {output_path}")
response = requests.get(video_url, stream=True)
        response.raise_for_status()
        total_size = int(response.headers.get("content-length", 0))
        downloaded = 0
        with open(output_path, "wb") as f:
            for chunk in response.iter_content(chunk_size=8192):
                f.write(chunk)
                downloaded += len(chunk)
                if total_size > 0:
                    progress = (downloaded / total_size) * 100
                    print(f"\r   Progress: {progress:.1f}%", end="", flush=True)
        print(f"\n✅ Video saved to: {output_path}")
        return str(output_path)

    print("❌ No video URL found in the response")
    return None

We use streaming downloads with stream=True to handle large files efficiently without loading everything into memory.

Step 8: The Main Section

def main():
    if not os.getenv("FAL_KEY"):
        print("❌ Error: FAL_KEY not found in environment variables.")
        print("   Please add your API key to the .env file:")
        print("   FAL_KEY=your_api_key_here")
        return
# Interactive prompt input
    prompt = input("🎬 Enter your video prompt: ").strip()
    if not prompt:
        prompt = "A cat playing with a ball of yarn in a cozy living room"
        print(f"   Using default prompt: {prompt}")
    # Configuration options with sensible defaults
    print("\n📐 Video settings (press Enter for defaults):")

    duration_input = input("   Duration in seconds (6): ").strip()
    duration = int(duration_input) if duration_input else 6
    print("   Aspect ratios: 16:9, 4:3, 3:2, 1:1, 2:3, 3:4, 9:16")
    aspect_ratio = input("   Aspect ratio (16:9): ").strip() or "16:9"
    print("   Resolutions: 480p, 720p")
    resolution = input("   Resolution (480p): ").strip() or "480p"
    print()
    # Execute the async generation pipeline
    video_path = asyncio.run(
        generate_and_download_video(
            prompt=prompt,
            duration=duration,
            aspect_ratio=aspect_ratio,
            resolution=resolution,
        )
    )
    if video_path:
        print(f"\n🎉 Done! Your video is ready at: {video_path}")

if __name__ == "__main__":
    main()

The main() function provides an interactive CLI. It validates the API key first, then collects user input with sensible defaults. asyncio.run() bridges the sync/async boundary, executing our async function from synchronous code.

Complete Code

Here’s everything together, ready to copy and run:

import asyncio
import os
from pathlib import Path
import fal_client
import requests
from dotenv import load_dotenv
load_dotenv()

async def generate_and_download_video(
    prompt: str,
    output_dir: str = "videos",
    duration: int = 6,
    aspect_ratio: str = "16:9",
    resolution: str = "480p",
) -> str:
    Path(output_dir).mkdir(parents=True, exist_ok=True)
    print(f"🎬 Generating video with prompt: {prompt}")
    print(f"   Duration: {duration}s | Aspect Ratio: {aspect_ratio} | Resolution: {resolution}")
    print("-" * 60)
    handler = await fal_client.submit_async(
        "xai/grok-imagine/text-to-video",
        arguments={
            "prompt": prompt,
            "duration": duration,
            "aspect_ratio": aspect_ratio,
            "resolution": resolution,
        },
    )
    print(f"📤 Request submitted. Request ID: {handler.request_id}")
    print("⏳ Waiting for video generation...")
    async for event in handler.iter_events(with_logs=True):
        if hasattr(event, "message"):
            print(f"   📝 {event.message}")
        elif hasattr(event, "logs"):
            for log in event.logs:
                print(f"   📋 {log.get('message', log)}")
    result = await handler.get()
    print("-" * 60)
    print("✅ Video generation complete!")
    video_info = result.get("video", {})
    video_url = video_info.get("url")
    file_name = video_info.get("file_name", "grok_video.mp4")
    print(f"   📊 Resolution: {video_info.get('width')}x{video_info.get('height')}")
    print(f"   ⏱️  Duration: {video_info.get('duration', 'N/A')}s")
    print(f"   🎞️  FPS: {video_info.get('fps', 'N/A')}")
    print(f"   📹 Frames: {video_info.get('num_frames', 'N/A')}")
    if video_url:
        output_path = Path(output_dir) / file_name
        print(f"\n📥 Downloading video to: {output_path}")
        response = requests.get(video_url, stream=True)
        response.raise_for_status()
        total_size = int(response.headers.get("content-length", 0))
        downloaded = 0
        with open(output_path, "wb") as f:
            for chunk in response.iter_content(chunk_size=8192):
                f.write(chunk)
                downloaded += len(chunk)
                if total_size > 0:
                    progress = (downloaded / total_size) * 100
                    print(f"\r   Progress: {progress:.1f}%", end="", flush=True)
        print(f"\n✅ Video saved to: {output_path}")
        return str(output_path)

    print("❌ No video URL found in the response")
    return None

def main():
    if not os.getenv("FAL_KEY"):
        print("❌ Error: FAL_KEY not found in environment variables.")
        print("   Please add your API key to the .env file:")
        print("   FAL_KEY=your_api_key_here")
        return
    prompt = input("🎬 Enter your video prompt: ").strip()
    if not prompt:
        prompt = "A cat playing with a ball of yarn in a cozy living room"
        print(f"   Using default prompt: {prompt}")
    print("\n📐 Video settings (press Enter for defaults):")

    duration_input = input("   Duration in seconds (6): ").strip()
    duration = int(duration_input) if duration_input else 6
    print("   Aspect ratios: 16:9, 4:3, 3:2, 1:1, 2:3, 3:4, 9:16")
    aspect_ratio = input("   Aspect ratio (16:9): ").strip() or "16:9"
    print("   Resolutions: 480p, 720p")
    resolution = input("   Resolution (480p): ").strip() or "480p"
    print()
    video_path = asyncio.run(
        generate_and_download_video(
            prompt=prompt,
            duration=duration,
            aspect_ratio=aspect_ratio,
            resolution=resolution,
        )
    )
    if video_path:
        print(f"\n🎉 Done! Your video is ready at: {video_path}")

if __name__ == "__main__":
    main()

Tips for Better Results

Write descriptive prompts. Include subject, action, environment, and style:

A steaming cup of coffee on a wooden table, morning light streaming through window blinds, gentle steam rising, cinematic depth of field

Match aspect ratio to platform. Use 9:16 for TikTok/Reels, 16:9 for YouTube, 1:1 for Instagram feed.

Start with 480p. It’s faster for iteration. Switch to 720p for final renders.

Next Steps

This foundation can be extended into automated content pipelines, creative tools, or integration with other services. The fal.ai also offers additional tools like fal workflows which lets you combine multiple models together for better customization and automation. While this tutorial uses the new Grok Imagine — text-to-video model, you can also easily adapt it for the other new Grok Imagine endpoints.

https://fal.ai/models/xai/grok-imagine-image/

https://fal.ai/models/xai/grok-imagine-image/edit

https://fal.ai/models/xai/grok-imagine-video/text-to-video

https://fal.ai/models/xai/grok-imagine-video/image-to-video

https://fal.ai/models/xai/grok-imagine-video/edit-video

Hope this was helpful! Happy Prompting!!

Gemini 3 & Antigravity — Vibing or Not ?

Ahamed Musthafa R S (amrs-tech) — Thu, 20 Nov 2025 19:40:16 +0000

Google released Gemini 3 and Antigravity (Agentic IDE) a couple of days ago, and it grasped a lot many eyes. Let us dive deep and see if they’re vibing or not.

There is a wonderful blog on the complete information about Gemini 3 by Google. It says, Gemini 1 had a breakthrough with multimodality and long context window, Gemini 2 laid the foundation for agentic abilities along with the reasoning and thinking parts. With these, now Gemini 3 is said to be the most intelligent model and they say nothing more specific. So, is it really an upgrade or just about to fade 😁

Okay, now how to try this Gemini 3 model ???
Simple. It is publicly available to all the Google users who uses Gemini app/web (as shown in the below screenshot)

Though the team posted performance benchmark details as shown in the below image, it is our duty (developers duty) to check how well the model is going to perform for our scenarios 🙂

I’ve done a basic level of comparison of the model performance between Gemini 2.5 Pro and Gemini 3 Pro. The prompt input would look like a stupid query (kindly ignore 😂 because I was about to ask something else but then twisted) but focus on how it works behind the curtains of “Thinking” part between the two said models.

This is the prompt that I gave to both the models:

You are an expert in SQL query creation.

Give sql query for below (postgresql schema based):

I have a schema called s_c . I have four tables in it :

Table 1 : ac

Table 2 : ml

Table 3 : st

Table 4 : st_ml



Get all data rows from all tables

Below are not the outputs (while that’s also varying very much — 3 being very detailed also specific not missing the actual requirement, 2.5 being simple and concise not giving more than expected)

2.5 Pro Thinking

3 Pro Thinking

Pasting in two parts (as it is longer)

Clearly, the latest model is doing as they say, “Deep Thinking”, and that’s how I got to know something’s been cooked behind the Gemini 3. You can certainly try and vibe with Gemini 3 and explore much deeper if possible and share your thoughts.

Moving on, we have one more release from Google — Antigravity. It is mentioned to be an Agentic IDE. Not sure on how it is going to be different from VS Code, Windsurf, Cursor or others, as I haven’t got a chance to explore it. Leaving it to you folks :) Please do try it out and feel free to comment !

References:

https://blog.google/products/gemini/gemini-3/

https://antigravity.google/

Happy Learning !

Veo 3 now available in Gemini Pro

Ahamed Musthafa R S (amrs-tech) — Sun, 03 Aug 2025 18:45:15 +0000

Veo 3 is a powerful video generation model by Google Deepmind, which was originally available through Vertex (Google Cloud). Now available with Google Gemini Pro, it can be used with limited access.

The usage is limited to 3 videos per day and also the video that can be generated is 8 seconds. Though the numbers are very small, it can be fun to experiment with it.

Here are some of the prompts that I tried with this video generation:

For the initial attempt, I went with one of the traditional concepts using a cat and it was good.

A kitten playing guitar in a street

For the next one, I tried to get a video of screen recording/sharing of a code editor with some Python code. Even though the specifications are mentioned in the prompt, the video that was generated by the model wasn’t very accurate with respect to the specifics.

A screenshare video (macOS — Visual Studio Code Editor) of typing a Python code (code for factorial of a number) without any audio

It was able to create the title for the video very specific to my description in the prompt but not the video :)
I got to know that the model is able to generate only the specifics related to an object in the video or the scenery, but not like text or any other little items which are in very small level of pixels.

The above examples are with the mode of Text-to-Video. Additionally, you can try to create a video with an image input (something like Image-Text-to-Video) with the prompt to the model. This is still limited to the 8-seconds video length restriction.

This video generation model can also be accessed via Gemini API on a paid-preview basis (more details on the official documentation page). Try out some of your own prompts and play with the model.

Happy Learning !!

Creating custom model from LLMs — Experimenting with plexe-ai

Ahamed Musthafa R S (amrs-tech) — Tue, 29 Apr 2025 18:02:21 +0000

Plexe AI has released a library plexe, can be used with Python to create own custom ML models with the help of LLMs. Just in natural language, provided the requirements and context, this library makes the lives of engineers very easy in creating a custom model (with little python 🙂). This can be very much useful when we want something to be done in a way the LLMs do, but also not interested in a large generative model everytime.

The library is very much easier to use when compared to using an ML model. But the only major thing is, it is not giving a working model for me yet. No matter how many times I try, it ends up with an error for not finding item in its registry.
Though it is the case, let us see till what point it can take us. And then, finally we can look into the error as well.

Code

Below is the code with just basic requirement to create a model using plexe-ai/plexe library

import plexe
#from datasets import load_dataset
import pandas as pd

# Define the model
model = plexe.models.Model(
    intent="Predict sentiment from news articles",
    input_schema={"headline": str, "content": str},
    output_schema={"sentiment": str}
)

# model = plexe.models.Model(
#     intent="Predict sentiment from news articles"
# )

# Build and train the model
model.build(
    datasets=[pd.read_csv('news_2025_1.csv')],
    provider="gemini/gemini-1.5-flash",
    max_iterations=3
)

# Use the model
prediction = model.predict({
    "headline": "New breakthrough in renewable energy",
    "content": "Scientists announced a major advancement..."
})

# Save for later use
plexe.save_model(model, "sentiment-model")
#loaded_model = plexe.load_model("sentiment-model.tar.gz")

Upon running our sample code (reference from the official Github README - github.com/plexe-ai/plexe), the script starts to run by executing respective agents in a sequential manner. From setting the prompt template till creating the code to generate the model for training and inference, all handled in a very much perfect way.

The agents go through each phase including the thinking part as well. With proper and reliable retries, the execution flows without flaws.

Each step involves a well-designed schematic plan to take forward the agents with expected success responses.

Every single piece of code that got generated get collated finally, but that’s when the problem arises. While finding the stored items from registry, there is an issue and the execution breaks without rendering the final model. I’m still experimenting the library and not sure if the library itself experimenting it :D

The above image has the error that kind of spun within the flow of execution (no external actions taken)

However, the idea and the setup of the library is great. Maybe, the team will be coming up with a stable solution to avoid flaws and issues in the future releases.

Happy learning ! Happy coding !!

RAG for a beginner by ChatGPT

Ahamed Musthafa R S (amrs-tech) — Tue, 22 Apr 2025 03:34:06 +0000

RAG is Retrieval Augmented Generation. It is an AI-technique that uses an integrated knowledge base for the information retrieval. The retrieved information in turn is used by the LLM as a context to generate the response for the user input.

No tricks behind the title of this story other than just what I did. As a beginner, I did not know where to start, so I started exploring some options to learn about RAG. I came across langchain, faiss, langflow and other such libraries. Yet I wasn’t able to tie all the ends, so I did seek the knowledge of ChatGPT :D
I got the response of the entire code to implement RAG using open-source libraries and then passing the context to LLM (with Groq API for llama-3–70b). Let us go through the code and understand it.

Folder Structure

rag_groq/
│
├── main.py                  # Entry point
├── ingest.py                # Load & embed PDF content
├── query.py                 # Ask questions using Groq API
├── config.py                # Configuration (Groq key, model, etc.)
├── sample.pdf               # Your test document
└── requirements.txt         # Required Python libraries

config.py

# config.py
GROQ_API_KEY = "gsk_*"
GROQ_MODEL = "llama3-70b-8192"  # or another model hosted on Groq
EMBEDDING_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
VECTOR_DB_PATH = "vector_store"

Groq API key is required to access the llama-3 model to infer. We define the embedding model to convert the document content into embeddings and then store the chunks. We store the index for the chunks using faiss and it will be saved to the vector db path defined here.

main.py

from ingest import build_vector_store
from query import query_pdf

if __name__ == "__main__":
    print("📥 Ingesting PDF and building vector store...")
    build_vector_store("sample.pdf")

    print("✅ Ingest complete. You can now ask questions.")
    while True:
        query = input("🔍 Ask a question (or type 'exit'): ")
        if query.lower() == "exit":
            break
        response = query_pdf(query)
        print("\n💬 Response:", response, "\n")

This is the main driver code to execute the entire flow. We import the ingest.py file here to ingest the user document to feed custom knowledge base to the LLM (will look into it in detail in the following code snippet). In an infinite loop, we just do an input from the user (until user gives ‘exit’) to chat with the model for querying about the document or anything too. The querying part is done using the function query_pdf() from the query.py

ingest.py

import os
import PyPDF2
import faiss
import pickle
from tqdm import tqdm
from sentence_transformers import SentenceTransformer
from config import VECTOR_DB_PATH, EMBEDDING_MODEL

def extract_text_from_pdf(pdf_path):
    with open(pdf_path, 'rb') as f:
        reader = PyPDF2.PdfReader(f)
        return "\n".join(page.extract_text() or '' for page in reader.pages)

def chunk_text(text, chunk_size=300, overlap=50):
    words = text.split()
    return [
        " ".join(words[i:i + chunk_size])
        for i in range(0, len(words), chunk_size - overlap)
    ]

def embed_chunks(chunks, model_name):
    model = SentenceTransformer(model_name)
    embeddings = model.encode(chunks, show_progress_bar=True)
    return embeddings

def build_vector_store(pdf_path):
    raw_text = extract_text_from_pdf(pdf_path)
    chunks = chunk_text(raw_text)
    embeddings = embed_chunks(chunks, EMBEDDING_MODEL)

    index = faiss.IndexFlatL2(len(embeddings[0]))
    index.add(embeddings)

    os.makedirs(VECTOR_DB_PATH, exist_ok=True)
    faiss.write_index(index, os.path.join(VECTOR_DB_PATH, "index.faiss"))

    with open(os.path.join(VECTOR_DB_PATH, "chunks.pkl"), "wb") as f:
        pickle.dump(chunks, f)

    print(f"Stored {len(chunks)} chunks in vector DB.")

if __name__ == "__main__":
    build_vector_store("sample.pdf")

To extract the content from the document, we use PyPDF2 library and then return it as a single string. It is then chunked using a default size 300 with an overlap of 50 on the contents page-wise. The chunked content list is then passed into a SentenceTransformer model (all-miniLM-L6-v2) and then encoded into embeddings. Then, the embeddings are indexed using faiss.IndexFlatL2 and stored in the vector db path, and then the chunks are stored in a pickle file.

query.py

import faiss
import pickle
from sentence_transformers import SentenceTransformer
from config import GROQ_API_KEY, GROQ_MODEL, VECTOR_DB_PATH, EMBEDDING_MODEL
from groq import Groq

client = Groq(api_key=GROQ_API_KEY)

def load_vector_store():
    index = faiss.read_index(f"{VECTOR_DB_PATH}/index.faiss")
    with open(f"{VECTOR_DB_PATH}/chunks.pkl", "rb") as f:
        chunks = pickle.load(f)
    return index, chunks

def get_top_k_chunks(query, index, chunks, k=5):
    embed_model = SentenceTransformer(EMBEDDING_MODEL)
    query_vec = embed_model.encode([query])
    D, I = index.search(query_vec, k)
    return [chunks[i] for i in I[0]]

def ask_llm(query, context):
    prompt = f"""Answer the question based on the context below.

Context:
{context}

Question:
{query}
"""
    response = client.chat.completions.create(
        model=GROQ_MODEL,
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": prompt}
        ]
    )
    return response.choices[0].message.content

def query_pdf(query):
    index, chunks = load_vector_store()
    print(chunks,'\n\n')
    print(index,'\n\n')
    relevant_chunks = get_top_k_chunks(query, index, chunks)
    combined_context = "\n\n".join(relevant_chunks)
    return ask_llm(query, combined_context)

if __name__ == "__main__":
    while True:
        user_q = input("Ask a question (or type 'exit'): ")
        if user_q.lower() == "exit":
            break
        answer = query_pdf(user_q)
        print("\n🧠 Answer:", answer, "\n")

This is the part where querying happens. The chunks are loaded along with the stored index to hit the query against the top relevant chunks. Other than that, the basic setup of a prompt template to interact with the llama-3 model via Groq API is done.

Voila ! That’s it !

Happy Learning ! Happy coding !!

Is Llama 4 really a failure or disaster ?

Ahamed Musthafa R S (amrs-tech) — Sat, 19 Apr 2025 14:11:23 +0000

A few days ago, Meta released Llama 4 in three variants — Behemoth, Maverick and Scout. Groq have Llama 4 Scout in their model hub and it can be easily accessed with their playground and the API service as well. There are several discussions around the model metrics. Some say llama 4 failed to give what people expected (did not get any better than llama 3), good thing is I did not set one :)

Let us look at a basic tutorial on creating a web application to get an image input from the user and then try to generate a table data (if the image is chart, graph or anything equivalent to a tabular data)

Application Code

I liked the name TabWiz :) but not necessarily. App code is explained briefly in sections.

app.py

import os
from flask import Flask, render_template, request, jsonify, url_for, send_file
from werkzeug.utils import secure_filename
from groq import Groq
import base64
import pandas as pd
from io import StringIO

Firstly, libraries that are required for web server, data (tabular) processing and accessing the Llama 4 model via Groq library, are imported in the beginning.

# Function to encode the image
def encode_image(image_path):
  with open(image_path, "rb") as image_file:
    return base64.b64encode(image_file.read()).decode('utf-8')

Then, the above function is defined to encode the image file as base64 data. This is to process the image and send it to our language model.

app = Flask(__name__)
app.config['UPLOAD_FOLDER'] = 'static/uploads'
os.makedirs(app.config['UPLOAD_FOLDER'], exist_ok=True)

static folder is created to store the file uploads done through the web form. The folder is added to the application config.

client = Groq(api_key='')

Then, Groq API client object is created with the api_key (generated from the groq console). You can create an account as a developer in Groq platform and can explore the features of the console (https://console.groq.com) to create a free API key.

@app.route('/')
def index():
    return render_template('index.html')

Then, the home/root API endpoint is created with the HTML template to display the file upload form.

@app.route('/chat', methods=['POST'])
def chat():
    user_message = request.form.get('message')
    image_file = request.files.get('image')
    file_ext = ''
    image_url = None
    if image_file:
        filename = secure_filename(image_file.filename)
        filepath = os.path.join(app.config['UPLOAD_FOLDER'], filename)
        image_file.save(filepath)
        image_url = url_for('static', filename=f'uploads/{filename}', _external=True)
        file_ext = filename.split('.')[-1]

    messages = [
        {
            "role": "system",
            "content": "You are an expert in parsing the given image and fetch all details very accurately and then return a csv data to be enough to create a table. Do not return any other text except for the csv."
        },
        {
            "role": "user",
            "content": [
                # {
                #     "type": "text",
                #     "text": user_message
                # },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/{file_ext};base64,{encode_image(filepath)}"
                    }
                }
            ]
        }
    ]

    completion = client.chat.completions.create(
        model="meta-llama/llama-4-scout-17b-16e-instruct",
        messages=messages,
        temperature=0.7,
        max_completion_tokens=1024,
        top_p=1,
        stream=False,
        stop=None,
    )

    response_text = completion.choices[0].message.content

    # Create a pandas DataFrame from the raw CSV content
    df = pd.read_csv(StringIO(response_text))
    # Save the DataFrame to a CSV file
    df.to_csv('output.csv', index=False)
    # download
    return send_file('output.csv', as_attachment=True)

    return jsonify({'response': response_text})

Then, the Chat API endpoint is created only with POST method allowed. This is called when the file upload form is submitted, and then the image file is processed into base64 data, which in turn is passed as a message to the llama-4-scout model. And the output from the model is expected to be a response just with CSV data, which is flushed into a file for download.

if __name__ == '__main__':
    app.run(debug=True)

That’s it. The above code is added finally to have the script run as a server when executed in shell. Make sure to have the libraries (flask & groq) installed in the environment. Server can be started with the command in the shell python app.py

The above image is the sample input file that I’ve uploaded and the model successfully processed it to generate the CSV data as expected.

The above table is the generated CSV data from the model. It is pretty good and got generated without any mistakes. I am not sure if it is really a failure or a disaster to people. But for an open model, it is working fine with the multimodal capabilities.

Happy Learning !

Powerful PDF+Image Parsing — Mistral OCR

Ahamed Musthafa R S (amrs-tech) — Wed, 12 Mar 2025 14:35:05 +0000

Mistral AI has recently released a powerful OCR model — Mistral OCR — Their tagline for the model is 1000 pages can be parsed per dollar. Mistral OCR model is said to be multilingual and multimodal. Complex documents in the format of PDF and images can be parsed with the model.

Example Scenarios

Let us look into some of the example scenarios with different documents as inputs to the model and see how it works.

For the first case, I’ve taken a sample image from the internet, with some handwritten notes in English language. Below is the image that I used.

When this image has been passed as an input to the Mistral’s OCR model, it parsed very perfectly and described what’s in the image. Below is the output from the model.

This shows how awesome is the model with respect to handwritten images (handwriting in it is not very awesome :D ) But let’s not stop there, and we shall try another scenario with a different language.

As my mother tongue is Tamizh, I chose to try a document in that (Tamil Language). The document is basically about Indian Constitution with some amendments and welfare of the Indian government. It is a 21-page long document of around 4Mb file size. Below is the link to the document for reference.

Link : https://raw.githubusercontent.com/amrs-tech/storepdf/refs/heads/main/part5_compressed.pdf

The OCR model has truly spoken from the bottom of its heart 😁😄 Just kidding! From the below output from the model, we can tell that it is working very much cooler even for language other than English (proving it to be multilingual).

To play with these scenarios, I did not create a complex python code, I just used the cookbook example from Mistral’s Github — https://github.com/mistralai/cookbook/tree/main/mistral/ocr

Pre-Post-Disclaimer 🙂: You need an API key from Mistral Platform to run this

Happy Learning !!

QwQ-32B vs DeepSeek-R1-671B

Ahamed Musthafa R S (amrs-tech) — Thu, 06 Mar 2025 05:43:29 +0000

Qwen is a series of LLMs released and maintained by Alibaba Cloud. QwQ is the model with reasoning capabilities in Qwen series. A while ago, the team released a preview version of this model and now, they’ve released QwQ-32B model completely. It is available in Huggingface and Ollama model repository.

Links

https://huggingface.co/Qwen/QwQ-32B
https://ollama.com/library/qwq

They’ve used reinforcement learning (RL) scaling approach driven by outcome-based rewards. As mentioned in their blog post, instead of a traditional reward model, an accuracy verifier is used in training this model. It is trained with rewards from general reward model and some rule-based verifiers. You can use QwQ-32B via Hugging Face Transformers and Alibaba Cloud DashScope API.

Example Code with Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Qwen/QwQ-32B"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt = "How many r's are in the word \"strawberry\""
messages = [
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=32768
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)

Example Code with DashScope API

from openai import OpenAI
import os

# Initialize OpenAI client
client = OpenAI(
    # If the environment variable is not configured, replace with your API Key: api_key="sk-xxx"
    # How to get an API Key：https://help.aliyun.com/zh/model-studio/developer-reference/get-api-key
    api_key=os.getenv("DASHSCOPE_API_KEY"),
    base_url="https://dashscope.aliyuncs.com/compatible-mode/v1"
)

reasoning_content = ""
content = ""

is_answering = False

completion = client.chat.completions.create(
    model="qwq-32b",
    messages=[
        {"role": "user", "content": "Which is larger, 9.9 or 9.11?"}
    ],
    stream=True,
    # Uncomment the following line to return token usage in the last chunk
    # stream_options={
    #     "include_usage": True
    # }
)

print("\n" + "=" * 20 + "reasoning content" + "=" * 20 + "\n")

for chunk in completion:
    # If chunk.choices is empty, print usage
    if not chunk.choices:
        print("\nUsage:")
        print(chunk.usage)
    else:
        delta = chunk.choices[0].delta
        # Print reasoning content
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content is not None:
            print(delta.reasoning_content, end='', flush=True)
            reasoning_content += delta.reasoning_content
        else:
            if delta.content != "" and is_answering is False:
                print("\n" + "=" * 20 + "content" + "=" * 20 + "\n")
                is_answering = True
            # Print content
            print(delta.content, end='', flush=True)
            content += delta.content

Performance Evaluation

Below is the evaluation chart to show how this 32B model is competing against other reasoning model, especially DeepSeek-R1–671B.

It is heavily competing against DeepSeek-R1–671B model in all the five benchmarks and outperforming OpenAI-o1-mini (except for IFEval).

I was wondering what OpenAI’s ChatGPT would think about it 🤣

Here’s a small insight from the comparison between QwQ-32b and Deepseek-r1–671b, generated by ChatGPT with this provided chart.
(NOTE: For some reason, 671b is shown 67.1b by ChatGPT — Please ignore)

It is clear that the storage requirements and hardware requirements are higher for Deepseek-R1–671b model when compared to QwQ-32b model.

Based on the evaluation, it is understood that the QwQ model is performing mostly very well for real-world tasks.

You can also try using the QwQ reasoning model in Qwen Chat at https://chat.qwen.ai/

NOTE — Referenced this video for some information: https://youtu.be/W85kbOduL8c?si=058s4_cmslrhRAxk

Happy Learning !

Invoice Reader App with GenAI in 10 minutes — Tutorial

Ahamed Musthafa R S (amrs-tech) — Sun, 02 Mar 2025 18:53:19 +0000

Let us try to create a simple web application with the help of Generative AI tech and tools. We will be using ChatGPT to get the basic setup of the project. With that, we can integrate additional updates to the code further as required.

Pre-Requisites

Below are the pre-requisites (or env/tools used) for the development of the application:

Python 3.10
Linux (in this case)
IDE or Editor
≥ 2GB memory

Once everything is set, then we can use a prompt (or similar) as shown below and get the entire code setup for our application from ChatGPT. (Please ignore “huggingface smolagents” in the context — it was intended for a different activity 😄)

you are an expert in python programming with high skills of huggingface smolagents library and other required libraries. Generate a complete project with any number of required python program files and functions to implement a web application (using flask and basic html css and js) that requires no user auth, with a page having a mandatory image file upload option part of a form submission. after submitting, it should be parsed by a vision lang model and then the parsed output should be written as a doc file and then emailed to a default email id

This was able to accomplish the request, yet it didn’t perform well as expected, so I had to give 2 followup messages (one for getting all dependency script files, and the other for wrapping the entire project scripts as a shell script). Later, I got the shell script to establish the project setup, which I saved in the local and then executed it with the following command:

chmod +x flask_vlm_app.sh
./flask_vlm_app.sh

Code with Explanation

Let us go through the code snippets along in the working flow of the application.

app.py

from flask import Flask, render_template, request, send_file
import os
from werkzeug.utils import secure_filename
from image_processor import process_image
from document_generator import create_doc
from email_sender import send_email

app = Flask(__name__)
app.config['UPLOAD_FOLDER'] = 'uploads'
app.config['OUTPUT_FOLDER'] = 'outputs'
DEFAULT_EMAIL = "amrs.tech@gmail.com"

os.makedirs(app.config['UPLOAD_FOLDER'], exist_ok=True)
os.makedirs(app.config['OUTPUT_FOLDER'], exist_ok=True)

@app.route('/')
def index():
    return render_template('index.html')

@app.route('/upload', methods=['POST'])
def upload():
    if 'image' not in request.files:
        return "No file part", 400
    file = request.files['image']
    if file.filename == '':
        return "No selected file", 400

    filename = secure_filename(file.filename)
    filepath = os.path.join(app.config['UPLOAD_FOLDER'], filename)
    file.save(filepath)

    extracted_text = process_image(filepath)

    doc_path = os.path.join(app.config['OUTPUT_FOLDER'], f"{os.path.splitext(filename)[0]}.docx")
    create_doc(extracted_text, doc_path)

    send_email(DEFAULT_EMAIL, doc_path)

    return send_file(doc_path, as_attachment=True)

if __name__ == '__main__':
    app.run(debug=True)

This is the main file that drives the flask application (server) and our dependency scripts are imported here (image_processor, document_generator and email_sender). There are two endpoints — One for the home page with file upload form and the other for (POST request) form submission and invoice parsing. The latter part flows from getting the uploaded file and saving it in a directory in the server and then parse it with our VLM, followed by sending the parsed content as a document to an email address.

image_processor.py

from transformers import AutoProcessor, AutoModelForImageTextToText
from PIL import Image
import torch

def process_image(image_path):
    processor = AutoProcessor.from_pretrained("HuggingFaceTB/SmolVLM-500M-Instruct")
    model = AutoModelForImageTextToText.from_pretrained("HuggingFaceTB/SmolVLM-500M-Instruct")

    conversation = [
        {
            "role": "user",
            "content":[
                {"type": "image", "url": image_path},
                {"type": "text", "text": "Describe this image in detail."}
            ]
        }
    ]

    inputs = processor.apply_chat_template(
        conversation,
        add_generation_prompt=True,
        tokenize=True,
        return_dict=True,
        return_tensors="pt",
    ).to(model.device)

    output_ids = model.generate(**inputs, max_new_tokens=400)
    generated_texts = processor.batch_decode(output_ids, skip_special_tokens=True)
    print("OUTPUT==>",generated_texts)
    return generated_texts

This is the script used to process the uploaded image from the user. HuggingFace’s SmolVLM-500M-Instruct model is used for image parsing, which works better than most of the Image models (ChatGPT gave the code with Salesforce BLIP-Image-Captioning model which didn’t do good). For more accuracy and better task-specific results, we can provide a few more example shots with the messages list.

email_sender.py

import smtplib
import os
from email.message import EmailMessage

def send_email(recipient, file_path):
    EMAIL_ADDRESS = os.getenv("EMAIL_ADDRESS")
    EMAIL_PASSWORD = os.getenv("EMAIL_PASSWORD")
    # print('emailauth==>',EMAIL_ADDRESS, EMAIL_PASSWORD)

    msg = EmailMessage()
    msg['Subject'] = 'Extracted Text Document'
    msg['From'] = EMAIL_ADDRESS
    msg['To'] = recipient
    msg.set_content("Please find the extracted text document attached.")

    with open(file_path, 'rb') as f:
        msg.add_attachment(
            f.read(),
            maintype='application',
            subtype='octet-stream',
            filename=os.path.basename(file_path),
            disposition='inline'
        )

    with smtplib.SMTP('smtp.gmail.com', 587) as server:
        server.starttls()
        server.login(EMAIL_ADDRESS, EMAIL_PASSWORD)
        server.send_message(msg)

This script is to send email to the DEFAULT_EMAIL we set in the beginning (we can change it to be fetched from user as well, but it’s just for example). We are constructing the EmailMessage object with Subject, From, To and then the Body. Then, we’re attaching the document (invoice content) and then logging in and sending the email message. If virtual environment is used, then you can edit env/bin/activate to add the email id and password to the environment, else you can directly add it to the system environment. (NOTE: You need app password for sending email with Gmail server — Ref — Google Support Answer for App Passwords)

Great! Now, with the project code all set, the files and folder structure of the project should be something like this:

flask_vlm_app/        # Project root
│── uploads/          # Folder for uploaded images (created dynamically)
│── outputs/          # Folder for generated documents (created dynamically)
│── templates/        # HTML template directory
│   └── index.html    # Webpage for image upload
│── app.py            # Main Flask application
│── image_processor.py # Processes image using a vision-language model
│── document_generator.py # Creates .docx file from extracted text
│── email_sender.py   # Sends extracted text document via email
│── requirements.txt  # List of required dependencies (if needed)

Make sure that all the required libraries are installed (in virtual environment if applicable, and activate the env) with pip install -r requirements.txt or pip install <pkg> for each library.

Flask==3.0.3
torch==2.6.0
transformers
pillow==11.1.0
python-docx==1.1.2

Let it begins — Start the flask app server with the command python app.py inside the project folder. You should be able to view a page as in the below screenshot after going to localhost:5000 (sometimes, the port might be 8000 — based on your flask app config) in the web browser.
You can then upload an image file of an invoice and click on Upload. It should take a couple of minutes for the model inference and then sending the email.

Voila! You should have received an email (provided the recipient email address is yours 😁) with the parsed content as a docx attachment.

This shows how easily Generative AI tools can be utilized to enhance the productivity with less efforts. Feel free to react 😜 and leave feedback in the comments. Thanks.

Github Link : https://github.com/amrs-tech/invoice-reader

Happy Learning!

Small Model from Huggingface with Video understanding

Ahamed Musthafa R S (amrs-tech) — Thu, 27 Feb 2025 18:09:44 +0000

A couple of weeks ago, SmolVLM-2 got released by Huggingface with an amazing feature — Video Understanding. The Vision Language Model initially been playing around the images, now has been upgraded with understanding of video inputs. It has been released in different variants such as 2.2B, 500M and 256M parameter models.

Though the model is not very much perfect, it can get the task done with a decent level of accuracy. Here’s a small and simple tutorial on how to play with this model (500M in this case).

Environment : Google Colab (with T4 GPU)
NOTE: I’ve disabled flash-attention as it is supported only by latest GPUs

Code Example

Used a sample input of a video (around 30 seconds).

!pip install wheel decord pyav num2words
!pip install flash-attn --no-build-isolation
!pip install git+https://github.com/huggingface/transformers@v4.49.0-SmolVLM-2

These are some pre-required libraries to be installed for the script to work with video inputs. flash-attn may not be required as the T4 GPU is not compatible with flash-attention. SmolVLM-2 is not yet available in the stable release of transformers package, so the release version is used in pip installation.

from transformers import AutoProcessor, AutoModelForImageTextToText
import torch

model_path = "HuggingFaceTB/SmolVLM2-500M-Video-Instruct"
processor = AutoProcessor.from_pretrained(model_path)
model = AutoModelForImageTextToText.from_pretrained(
    model_path,
    torch_dtype=torch.bfloat16,
    # _attn_implementation="flash_attention_2"
).to("cuda")

Required libraries are imported after the installation. Then, the model is downloaded using transformers and then loaded into the GPU (‘cuda’).

# Ensure all parameters are in bfloat16
for name, param in model.named_parameters():
    if param.dtype == torch.float32:
        param.data = param.data.to(torch.bfloat16)

We need to make sure that all the model parameters are in the dtype bfloat16

from google.colab import drive
drive.mount('/content/drive')

vdo_pth = '/content/drive/MyDrive/videoplayback.mp4'

Now, the sample video file is loaded from GDrive.

messages = [
    {
        "role": "user",
        "content": [
            {"type": "video", "path": vdo_pth},
            {"type": "text", "text": "Describe this video in detail"}
        ]
    },
]

The messages list is constructed in the applicable chat template, with the video file path as input along with a message to describe the video.

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to("cuda")

Now, the input chat template is tokenized and stored as tensors in the GPU (‘cuda’).

for key, value in inputs.items():
    if torch.is_floating_point(value):  # Convert only float tensors
        inputs[key] = value.to(torch.bfloat16)

As same as the model, the inputs are converted into bfloat16 dtype.

generated_ids = model.generate(**inputs, max_new_tokens=254)
generated_texts = processor.batch_decode(
    generated_ids,
    skip_special_tokens=True,
)

print('OUT: ', generated_texts[0])

Now, the model is called for inference and then generated response is decoded into actual output.

OUT:  User: You are provided the following series of twenty-four frames from a 0:00:24 [H:MM:SS] video.

Frame from 00:00:
Frame from 00:01:
Frame from 00:02:
Frame from 00:03:
Frame from 00:04:
Frame from 00:05:
Frame from 00:06:
Frame from 00:07:
Frame from 00:08:
Frame from 00:09:
Frame from 00:10:
Frame from 00:11:
Frame from 00:12:
Frame from 00:13:
Frame from 00:14:
Frame from 00:15:
Frame from 00:16:
Frame from 00:17:
Frame from 00:19:
Frame from 00:20:
Frame from 00:21:
Frame from 00:22:
Frame from 00:23:
Frame from 00:24:

Describe this video in detail
Assistant: The video begins with a view of a large commercial airplane in flight, showcasing its sleek design and the expansive sky. The text "explained" appears at the top of the frame, indicating the topic of the video. The airplane is shown in various angles, emphasizing its size and the vastness of the sky. The scene transitions to a close-up of the airplane's engines, highlighting their size and the impressive performance of the jet. The text "explained" reappears, and the airplane is shown in a different angle, emphasizing its size and the expansive sky.

The video then shifts to a view of a small white airplane on a runway, with the text "explained" at the top. The airplane is shown in various angles, emphasizing its size and the expansive sky. The scene transitions to a view of a large airplane in flight, showcasing its sleek design and the expansive sky. The text "explained" reappears, and the airplane is shown in a different angle, emphasizing its size and the expansive sky.

The video continues with a view of a large commercial airplane in flight, showcasing its sleek design and the expansive sky. The text "explained" appears at the top of the frame, indicating the topic of the video

The output is as above, description generated about the video input. It is not very much perfect, but for a very small model, this is fine.

Input Video Link : GDrive Link of Video

You can try inferring different samples and also with other variants such as 256M model and 2.2B model. This can even be integrated and deployed in mobile devices.

Happy Learning !!