Forem: Thomas Ezan

LLM evaluation: a quick overview of Stax

Thomas Ezan — Thu, 23 Oct 2025 00:38:46 +0000

The views and opinions expressed on this blog are my own and do not reflect those of my employer. Additionally, any solutions, APIs, or products mentioned are for informational and discussion purposes only and should not be considered an endorsement.

Developing applications with Generative AI and Large Language Models (LLMs) feels often more like an art than a precise science. Developers dedicate significant time to manually testing various prompts and depend on subjective assessments or intuition to ascertain if the AI model meets their requirements.
What if you could skip all that manual and subjective process and instead take a data-driven and repeatable approach? 🤔

That is what Stax is for! ✨

What is Stax?

Stax is a developer tool from Google Labs designed to simplify LLM and AI application testing. It enables you to test models and prompts against your own use-cases.

Who is Stax for?

Stax is built primarily for product managers and app developers building LLM-powered apps or features. With stack you will

Create a dataset of prompts tailored to your specific use cases for testing various models.
Use this dataset to evaluate various LLMs with clear metrics to determine if your AI application is working as intended before shipping to real users. The platform makes it easy to create and work with the 2 building blocks of effective AI evaluation: datasets and evaluators.

Create a dataset

In Stax, datasets are defined as collections of prompts. These prompts are used to generate responses from LLMs that will then be evaluated. First, select that model you want to evaluate.

Then, type a few user prompts into the playground to build your evaluation benchmark. If you already have data, you can also upload it as a CSV.

Evaluate the model’s responses

Once you've produced outputs, you can start evaluating them. To do so, select the best evaluator for your use case. In our case we are building a news summarization app, so we are going to use the pre-defined Summarization Quality evaluator. But you can also create a custom one.

When the evaluation is done, you can visalize individual and aggregate metrics so that you can compare various models or iterations of prompts against each other on quality but also latency.

Conclusion

Stax enables you to make data-driven decisions, helping you select the optimal model for both quality and speed. With Stax, you can easily rerun your datasets and evaluators whenever you wish to experiment with a new model. This makes it an excellent model-agnostic platform to build AI enabled feature with confidence. 💪

Using Ollama and Tailscale to power an Android app with Gemini 3 27B

Thomas Ezan — Mon, 01 Sep 2025 21:19:59 +0000

Running Gemma 3 27B with Ollama on a powerful laptop or a gaming PC gives great performances. It’s very versatile, has good “world knowledge” and is good at text transformation (text summarization, rewrite, etc…). And Ollama also supports system prompts, so you can use it to give the model a persona and turn into a fun chatbot.

Today you can run 8 billion parameters models on a recent Android device (check out Gemma 3n E2B and Gemma 3n E4B through Google AI Edge Gallery 😉). But no device has enough memory yet to run 27 billions parameters models.

So as a workaround, let’s run the model on your machine at home and tunnel a connection from your phone to your home network.

Product architecture

Building blocks

Ollama
Ollama is a local LLM runtime that makes running open-weight models dead simple. You download a model with one command and it runs locally even if your machine doesn’t have a GPU (it then defaults to CPI). Ollama also packs a web server and provides a REST API for inference.

So no complex setup and no Python environments to manage. Recently, they even released a desktop app to make interacting with local models even easier.

In this blog post we are using Gemma 3 27B, but the setup will work with any model supported by Ollama that your computer can run.

A llama dressed as Google’s Android robot (cc)
(Image generated via Imagen 4)

Tailscale
Tailscale creates a secure mesh network between your devices. Install it on your laptop and phone, and they can talk to each other as if they're on the same local network – even when you're thousands miles apart. Tailscale is based on the open source protocol WireGuard and it takes care of the configuration for you so you won’t have to worry about port forwarding, no firewall rules and router settings.

Bespoke Ollama Android library
I implemented a bespoke library for Android designed to interact with an Ollama API server. I try to use generic Kotlin APIs as much as possible to potentially use this library in KMP.

I used kotlinx.serialization for serialization/deserialization. All API request and response models are defined as @Serializable data classes, which allows for easy and type-safe conversion between JSON and Kotlin objects.

import kotlinx.serialization.Serializable
import kotlinx.serialization.SerialName

@Serializable
data class GenerateRequest(
    val model: String,
    val prompt: String,
    val stream: Boolean = false,
    val format: String? = null,
    val options: Options? = null,
    val system: String? = null,
    val template: String? = null,
    val context: String? = null,
)

@Serializable
data class GenerateResponse(
    val model: String,
    val createdAt: String,
    val response: String,
    @SerialName("done_reason") val doneReason: Boolean,
    @SerialName("total_duration") val totalDuration: Long? = null,
    @SerialName("load_duration") val loadDuration: Long? = null,
    @SerialName("prompt_eval_count") val promptEvalCount: Int? = null,
    @SerialName("prompt_eval_duration") val promptEvalDuration: Long? = null,
    @SerialName("eval_count") val evalCount: Int? = null,
    @SerialName("eval_duration") val evalDuration: Long? = null,
    val context: List<Int>? = null,
)

@Serializable
data class ChatRequest(
    val model: String,
    val messages: List<Message>,
    val stream: Boolean = false,
    val format: String? = null,
    val options: Options? = null,
)

@Serializable
data class ChatResponse(
    val model: String,
    @SerialName("created_at") val createdAt: String,
    val message: Message,
    @SerialName("done_reason") val doneReason: String = "",
    val done: Boolean,
    @SerialName("total_duration") val totalDuration: Long? = null,
    @SerialName("load_duration") val loadDuration: Long? = null,
    @SerialName("prompt_eval_count") val promptEvalCount: Int? = null,
    @SerialName("prompt_eval_duration") val promptEvalDuration: Long? = null,
    @SerialName("eval_count") val evalCount: Int? = null,
    @SerialName("eval_duration") val evalDuration: Long? = null,
)

@Serializable
data class Message(
    val role: String,
    val content: String,
    val images: List<String>? = null,
)

@Serializable
data class Model(
    val name: String,
    @SerialName("modified_at") val modifiedAt: String,
    val size: Long,
    val digest: String,
    val details: ModelDetails,
)

@Serializable
data class ModelDetails(
    @SerialName("parent_model") val parentModel: String,
    val format: String,
    val family: String,
    val families: List<String>? = null,
    @SerialName("parameter_size") val parameterSize: String,
    @SerialName("quantization_level") val quantizationLevel: String,
)

@Serializable
data class Options(
    val numa: Boolean? = null,
    @SerialName("num_ctx") val numCtx: Int? = null,
    @SerialName("num_batch") val numBatch: Int? = null,
    @SerialName("num_gqa") val numGqa: Int? = null,
    @SerialName("num_gpu") val numGpu: Int? = null,
    @SerialName("main_gpu") val mainGpu: Int? = null,
    @SerialName("low_vram") val lowVram: Boolean? = null,
    @SerialName("f16_kv") val f16Kv: Boolean? = null,
    @SerialName("logits_all") val logitsAll: Boolean? = null,
    @SerialName("vocab_only") val vocabOnly: Boolean? = null,
    @SerialName("use_mmap") val useMmap: Boolean? = null,
    @SerialName("use_mlock") val useMlock: Boolean? = null,
    @SerialName("embedding_only") val embeddingOnly: Boolean? = null,
    @SerialName("rope_frequency_base") val ropeFrequencyBase: Float? = null,
    @SerialName("rope_frequency_scale") val ropeFrequencyScale: Float? = null,
    @SerialName("num_thread") val numThread: Int? = null,
    @SerialName("num_keep") val numKeep: Int? = null,
    @SerialName("seed") val seed: Int? = null,
    @SerialName("num_predict") val numPredict: Int? = null,
    @SerialName("top_k") val topK: Int? = null,
    @SerialName("top_p") val topP: Float? = null,
    @SerialName("tfs_z") val tfsZ: Float? = null,
    @SerialName("typical_p") val typicalP: Float? = null,
    @SerialName("repeat_last_n") val repeatLastN: Int? = null,
    @SerialName("temperature") val temperature: Float? = null,
    @SerialName("repeat_penalty") val repeatPenalty: Float? = null,
    @SerialName("presence_penalty") val presencePenalty: Float? = null,
    @SerialName("frequency_penalty") val frequencyPenalty: Float? = null,
    @SerialName("mirostat") val mirostat: Int? = null,
    @SerialName("mirostat_tau") val mirostatTau: Float? = null,
    @SerialName("mirostat_eta") val mirostatEta: Float? = null,
    @SerialName("penalize_newline") val penalizeNewline: Boolean? = null,
    @SerialName("stop") val stop: List<String>? = null,
)

@Serializable
data class ListModelsResponse(
    val models: List<Model>
)

I used the Ktor client for handling HTTP requests and picked the CIO (Coroutine-based I/O) engine for Ktor which is a pure-Kotlin and lightweight solution. However, it’s not compatible with iOS, so I would have to add the Darwin engine for a KMP project supporting Android.

import io.ktor.client.HttpClient
import io.ktor.client.call.body
import io.ktor.client.engine.cio.CIO
import io.ktor.client.plugins.contentnegotiation.ContentNegotiation
import io.ktor.client.request.get
import io.ktor.client.request.post
import io.ktor.client.request.setBody
import io.ktor.client.statement.bodyAsText
import io.ktor.http.ContentType
import io.ktor.http.contentType
import io.ktor.serialization.kotlinx.json.json
import kotlinx.coroutines.flow.Flow
import kotlinx.coroutines.flow.flow
import kotlinx.serialization.json.Json


class OllamaClient {

    private lateinit var host: String // server url and port number (usually 11434)

    private val client = HttpClient(CIO) {
        install(ContentNegotiation) {
            json(Json {
                ignoreUnknownKeys = true
            })
        }
        engine {
            requestTimeout = 60_000
        }
    }

    fun setHost(ipAddress: String) {
        host = ipAddress
    }

    fun generate(request: GenerateRequest): Flow<GenerateResponse> = flow {
        val response = client.post("$host/api/generate") {
            contentType(ContentType.Application.Json)
            setBody(request)
        }

        response.bodyAsText().split("\n").forEach { line ->
            if (line.isNotEmpty()) {
                emit(Json.decodeFromString(line))
            }
        }
    }

    fun chat(request: ChatRequest): Flow<ChatResponse> = flow {
        val response = client.post("$host/api/chat") {
            contentType(ContentType.Application.Json)
            setBody(request)
        }

        response.bodyAsText().split("\n").forEach { line ->
            if (line.isNotEmpty()) {
                emit(Json.decodeFromString(line))
            }
        }
    }

    suspend fun list(): List<Model> {
        val response = client.get("$host/api/tags").body<ListModelsResponse>()
        return response.models
    }

   [...] 
   // Omitting model update and manipulation functions

}

Finally, don’t forget to add the INTERNET permission to your manifest 😉 and enable usesCleartextTraffic. This setting will allow the application to easily connect to your local Ollama server over HTTP without requiring complex network security configurations. Usually HTTPS is recommended, but here since we are using Tailscale tunneling between your phone and the Ollama server on your laptop, the communication will still be end-to-end encrypted.

Conclusion

After testing it extensively, I can confirm that this setup performs really well. The latency is almost imperceptible with a 5G or even LTE connection, which allows a solid real-time chat experience.

In conclusion, as long as you are connected, mobile private personal AI with a "mid-size" LLM is now possible!

Ollama on Android in Action!

Experimenting with Gemini, MCP, Agentic AI and Android

Thomas Ezan — Mon, 18 Aug 2025 00:34:56 +0000

The views and opinions expressed on this blog are my own and do not reflect those of my employer.

MCP has been the craze in the past few months, and we've seen an explosion of MCP servers for every possible use case (see this MCP server to flip a coin 🤔).,, Similarly, Agentic AI, while lacking a universally clear definition, has been a major topic of discussion in the industry for the past 12 months.

Some define Agentic AI behavior as “using AI to do something for you”. This definition is too broad for me. I prefer Yoshua Bengio’s definition: “understand context, set objectives, and execute multi-step plans with robustness.”

In this blog post, I will describe how to:

Create a simple MCP server leveraging Android Debug Bridge (ADB)
Create a basic AI Agent leveraging our MCP server to find and launch an Android app

Here is a diagram of what we will build:

Creating an ADB MCP server

What is MCP?
For a comprehensive introduction to MCP, take a look at the project documentation. You can also read Philip Schmidt overview blog post.

You can think of MCP as a universal adapter for your AI application to interact with the real world. The documentation describes MCP as “a USB-C port for AI applications”. I like to think of MCP servers as drivers that connect the real world to your AI application. Writing a hardware driver is a bit of a pain, but once it is available your computer can talk to your hardware.

However, let me reassure you: writing an MCP server is significantly easier than developing a hardware driver for an operating system 😅. Also, there are a lot of MCP servers and what you need might already exist. Here are few places to search:

MCP + ADB

For this project, I decided to build a very simple MCP service using the following 3 tools:

get_devices: retrieves a list of Android devices connected to the laptop,
get_apps: retrieves the package name of all the apps installed on the device,
open_app: opens the app with the matching package name.

If you're familiar with ADB, you likely already have an idea of the implementation and the commands we'll use.

MCP servers provide their functionalities through 3 core building blocks:

Tools: called directly by the LLM, these are essentially an abstraction over function calling capabilities.
Resources: a standardized way to expose resources such as files, databases, etc…
Prompts: a list templatized messages and instructions that can be used as prompts for the LLM

There are also more advanced features such as sampling and elicitation. Interestingly, most of MCP hosts don’t support all the MCP features. As a consequence, you'll notice that MCP servers often leverage only Tools, which is the most common MCP feature.

If you want to implement an MCP server, there's already a rich ecosystem with libraries and reference implementations.

In my case I used the MCP Python SDK:

from mcp.server.fastmcp import FastMCP
import subprocess

mcp = FastMCP("ADB MCP")

@mcp.tool()
def get_devices() -> str:
    """Get the list of Android devices connected to the laptop"""
    result = subprocess.run(['adb', 'devices'], capture_output=True, text=True)
    return result.stdout

@mcp.tool()
def get_apps() -> str:
    """Use ADB to get the list of all the apps installed on a specific Android devices connected to the laptop. 
    Search through the output of this list to confirm if an app is installed."""
    result = subprocess.run(['adb', 'shell', 'pm', 'list', 'packages'], capture_output=True, text=True)
    if result.returncode != 0:
        return f"Error: {result.stderr}"
    return result.stdout


@mcp.tool()
def open_app(package_name: str) -> str:
    """Open an app on the connected Android device using its package name via ADB."""
    # The command to launch the main activity of the app
    cmd = [
        'adb', 'shell', 'monkey', '-p', package_name, '-c', 'android.intent.category.LAUNCHER', '1'
    ]
    result = subprocess.run(cmd, capture_output=True, text=True)
    if result.returncode != 0:
        return f"Error opening app: {result.stderr.strip()}"
    return f"App '{package_name}' launched successfully."


if __name__ == "__main__":
    mcp.run()

I was curious about the inner workings of MCP and how the protocol and ADB would work together. But you don’t have to follow this route. There is already an ADB MCP server ready to use if you have a more complex project in mind: minhalvp/android-mcp-server.

Additionally, if you want to experiment with running the server directly in your Android app, take a look at kaeawc/android-mcp-sdk.

Creating an AI Agent

mcp-use library
The AI Agent component of this project could have been implemented in various ways. I chose the mcp-use library because it offered a relatively easy method for implementing an AI Agent that leverages an MCP server.

import asyncio
import os
from mcp_use import MCPAgent, MCPClient
from langchain_anthropic import ChatAnthropic
from langchain_openai import ChatOpenAI
from langchain_google_genai import ChatGoogleGenerativeAI

async def main():
    print("hello!")
    config = {
        "mcpServers": {
            "demo": {
                "command": "/path/to/venv/bin/uv",
                "args": [
                    "run",
                    "--with",
                    "mcp[cli]",
                    "mcp",
                    "run",
                    "/path/to/server.py"
                ]
            }
        }
    }


    client = MCPClient.from_config_file(
        os.path.join(os.path.dirname(__file__), "android_mcp.json")
    )

    agent = MCPAgent(
        llm = ChatGoogleGenerativeAI(model="gemini-2.5-pro"),
        client=client
    )

    result = await agent.run("Fetch the list of app installed to the android device connected to the laptop and review the list of packages to check if the New York Time app installed on the device and if it is, open the app")
    print("--------------------------------")
    print(result)

if __name__ == "__main__":
    asyncio.run(main())

Conclusion

If you weren’t familiar with MCP, you probably learned new concepts! 😀

The goal of this project was to build a fun demo and test out different APIs and SDKs I came across in the past few weeks. It's not meant to be an endorsement of these tools or a suggested way to build production features for real products. Don't hesitate to reach out if you've also experimented with MCP and Android!

Leverage Gemini Pro Vision on Android to get better at Bananagrams 🍌

Thomas Ezan — Wed, 20 Mar 2024 04:09:44 +0000

Bananagram is a cooler version of Scrabble! In a race to the finish, players build word grids with their letters, aiming to use them all first.

But word games can be tough when you're playing in a language that you aren't native in.
Except if you have an Android app leveraging Gemini Pro vision!

What are we building

We will build Potassium (😁) an application that suggests words that can be spelled given the tiles that are available. To do this we'll leverage Gemini Pro vision to:

Analyze a picture of the tiles and extract the list letters available,
List words that can be spelled with these letters.

Can Gemini Pro actually do this?

Experimenting with the ML model is key as crafting a prompt often requires multiple iterations before reaching a satisfying result.

Let’s use Google AI studio to evaluate Gemini Pro vision capabilities.

Can the model create a list of the letters based on a picture of the tiles?

Then, can the model then return a list of words made with these letters?

Add Gemini to your application

Now that we crafted a prompt that returns a relatively satisfying response (some suggested words might or might not be valid Scrabble words), let’s create the app!
On the top left of Google AI studio, click on “Get API key” to get your Gemini API key.

Then, click on Get code on the top right of Google AI studio to access the code snippet.

Add the Gradle dependencies to your app’s build.gradle file:

implementation("com.google.ai.client.generativeai:generativeai:0.1.1")

In your Kotlin code create a GenerativeModel:

Define the generationConfig that will be used by the model. e.g:

val generationConfig = generationConfig {
                temperature = 0.15f
                topK = 32
                topP = 1f
                maxOutputTokens = 4096
        }

The configuration is reflecting the adjustments you made in the "Run settings" section of the console. These parameters define the creativity and diversity of the text generated during inference.

topK: the Top-K value defines, out of the token generated by the model, the number (k) of tokens considered for the output.

topP: the Top-P value is used to define the cumulative probability of the k tokens (after normalization of their probability) considered for the output.

temperature: controls the level of randomness of the token selected for the output.

To learn more about the LLM sampling mechanism, Vibudh Singh wrote a good explainer.

Then instantiate the GenerativeModel:

val model = GenerativeModel(
            "gemini-pro-vision",
            "your_gemini_key",
            generationConfig = generationConfig,
            safetySettings = listOf(
                SafetySetting(HarmCategory.HARASSMENT, BlockThreshold.MEDIUM_AND_ABOVE),
                SafetySetting(HarmCategory.HATE_SPEECH, BlockThreshold.MEDIUM_AND_ABOVE),
                SafetySetting(HarmCategory.SEXUALLY_EXPLICIT, BlockThreshold.MEDIUM_AND_ABOVE),
                SafetySetting(HarmCategory.DANGEROUS_CONTENT, BlockThreshold.MEDIUM_AND_ABOVE),
            ),
        )

You can then call the model as follow:

viewModelScope.launch {
   val result = model.generateContent(
      content {
         image(bitmap)
         text("What are the letters displayed on the tiles? " +
            "And given these letter which Scrabble words can you spell with it?")
      }
   )
}

You’ll note that we pass both an image (as a bitmap) and a text as content.

To create your bitmap, you can simply access the camera using rememberLauncherForActivityResult in compose:

val resultLauncher =
       rememberLauncherForActivityResult(ActivityResultContracts.StartActivityForResult()) { result: ActivityResult ->
            if (result.resultCode == Activity.RESULT_OK) {
                if (result?.data != null) {
                    bitmap = result.data?.extras?.get("data") as Bitmap
                }
            }
        }

[...]
Button (
  onClick = {
    resultLauncher.launch(cameraIntent)
  },
)

You'll find a very basic compose scaffolding in this gist.