Forem: Vijay Vimalananda Guru

Build Your Own AI Assistant App - No Cloud Required!

Vijay Vimalananda Guru — Sun, 08 Mar 2026 04:52:58 +0000

Introduction

This article will help you to build your own AI Assistant application using edge devices like Raspberry Pi. I initially began this project with a goal of automating all the smart devices in my home, but we will cover that topic in a further article. Before we started, we need to understand what we are going to build and the prerequisites involved.

What You'll Build

First API service that communicate between your Android/iOS app with the SLM(Small Language Model) that is hosted in Raspberry Pi. Here, we are going to use Python framework FastAPI.
Mobile application(both Android/iOS) using Compose Multiplatform.

Prerequisites

Hardware: Although Raspberry Pi 4 (4GB/8GB) is enough, it is better to have Raspberry Pi 5 (4BG/8GB/16GB) with 32GB SD card and active cooler.
Software: Android Studio (I'm using Panda), Python version 3.11+, Tailscale app(which we will talk later)
Model: TinyLlama-1.1B-chat-v1.0 (~680MB of size, quantized, consumes 1.1GB –1.9GB RAM, more than enough for a short queries)
Network: Pi can be connected over Wi-Fi or Ethernet
Operating System: Raspberry Pi OS for Raspberry Pi module and Mac/Windows/Linux for mobile app development.

Architecture

Frontend: Compose Multiplatform(single codebase for Android/iOS)
Backend: FastAPI(handling async request & model inference)

Let's begin with Frontend

After installing Android Studio, you can download the source code from my GitHub repository or write your own implementation - I'm sure may of you can come up with a better version. To keep this article concise, I won't go further detail about the CMP (Compose Multiplatform) code here.

Tech Stack

UI: Compose Multiplatform
Language: Kotlin
Architecture: Clean Architecture (Data, Domain, Presentation)
Networking Library: Ktor
Local Database: SQLDelight
Dependency Injection: Koin
Concurrency: Kotlin Coroutines & Flow

Project Structure

composeApp/src/commonMain: Shared logic, data models, repositories, and UI components.
composeApp/src/androidMain: Android-specific implementations (e.g., Database Driver, MainApplication).
composeApp/src/iosMain: iOS-specific implementations and entry points.
data/: Network services (AIChatApiService) and repository implementations.
domain/: Business logic and Use Cases.
presentation/: ViewModels and Compose UI screens.

Important Note

The application communicates with the AI model via REST API. You need to configure the BASE_URL in AIChatApiService.kt:

// Use this if you have not setup Tailscale
const val BASE_URL = "http://<YOUR_RASPBERRY_PI_IP>:8000"
// If you have setup Tailscale
const val BASE_URL = "http://<IP_PROVIDED_BY_TAILSCALE>:8000"

Once we setup backend part in Raspberry Pi, we can replace that URL here.

Let's Begin with Backend Setup

To streamline this process, we will directly begin with installation of required packages and setting up AI Engine. Of course implementing API service that communicated between app and the AI model.

After installing Raspberry Pi OS, switch on the Pi module and open terminal/command prompt.

Let's begin with hosting the model

Step 1

# Intall VCS and essential tools for compiling
sudo apt install git build-essential cmake -y

# Download cpp project to run Llama on CPU
git clone https://github.com/ggerganov/llama.cpp

# Once done, go to 
cd llama.cpp
cd models

# Download the model from Huggingface or choose your favourite model(recommanded 0.5B-3B Parameter)
curl -L -o tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf

# Prepare your project for building
cmake -B build

# Compiles the project. This will take some time
cmake --build build --config Release

Open new terminal window and run this

# Start the model
./llama.cpp/build/bin/llama-server -m ./llama.cpp/models/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf -c 512 -t 4 --port 8080

Step 2
Now, we will implement an API that communicates with the model which we hosted in step 1.

# Check for OS update
sudo apt update && sudo apt upgrade -y

# Make a separate folder for API service. I'm naming it as quanta
mkdir quanta && cd quanta

# To create isolated virtual environment
python3 -m venv .venv

# Activates virtual environment
source .venv/bin/activate

# Install required packages
pip install fastapi uvicorn requests httpx psutil

# Create a Python file naming as aichatservice
touch aichatservice.py

Open the file aichatservice.py and paste the code available in this repo

Save the file and run the below command

uvicorn aichatservice:app --host 0.0.0.0 --port 8000

Try this in any API testing tools like Postman or Rest Client
Request Type: POST
URL: http://raspberry-pi-id:8000/generate

//Request Body:
{
  "prompt": "can you explain to me AI in short?"
}

Please take note that whenever you login/restart the Raspberry Pi we have to start the web server using the above uvicorn command(picture 1). We also need to start the LLM server using llama.cpp(picture 2)

Picture 1

Picture 2

We are now done with the backend part.

Additional: To make the API accessible outside your local network, you can do port forwarding(router level-not secured)or a tunneling service. I'm going to use Private VPN (Tailscale). This works securely anywhere.

Open the terminal and run the below command.

# Intall Tailscale 
curl -fsSL https://tailscale.com/install.sh | sh
# After Installation, run this
sudo tailscale up

After running sudo tailscale up, you will be prompted to authenticate using the link provided. Once authentication is successful, you will be redirected to the Tailscale dashboard where you can find the IP Address of the device connected. Copy it and replace it with URL: http://ip-provided-by-tailscale:8000/generate.

And that is it - you are all set. Now, it's time to run the application in Android Studio.

Final Output

Android

iOS

Since this article only covers the basics of building you own AI assistant, I didn't focus much on the design, adding comment line, architecure and "Token Streaming". In the next article, we will explore how to communicate with smart devices using Docker and Home Assistant.

Let me know your thoughts in the comments so that it will help me improve further.

Understanding Kotlin's Scope Functions

Vijay Vimalananda Guru — Mon, 15 Apr 2024 08:33:48 +0000

What are Scope Functions?

Scope Functions are a part of Kotlin standard library. It executes a block of code within that context of an object. Using these functions on an object with lambda expression, it provides a temporary scope for that object which is either accessible by it(default) or this, depending on function type. With this we can access those objects without their names. Thereby it makes the code simple, clean and easily readable. There are 5 different types of scope functions in Kotlin: let, with, run, also & apply.

When to choose what?
To choose the right function for your requirement, please check the below table.

let:
let is primarily used along with a safe call operator (?.) to prevent NullPointerException. It will execute the block of code only if the object is not null.

fun main (){
    val name: String? = null
    println(name!!.length)
}

Output:
Exception in thread "main" java.lang.NullPointerException

Using let function:

fun main (){
    val name: String? = null
    name?.let{
        println(name!!.length) //statement will not execute 
                               //as name is null
    }
}

with:
with is used for calling functions on context objects without providing the lambda result. It is used to access members and methods of an object without having refer to it.

Using with function:

class Person {
    var name: String = ""
    var age: Int = 0
}

fun main() {
    val person = Person()
    with(person) {
        name = "Vijay"
        age = 30
        println("Name: $name, Age: $age")
    }
}

Output:
Name: Vijay, Age: 30

Inside the with block, you can directly access the properties (name and age) of the person object without explicitly referencing person. This can lead to more concise and readable code, especially when performing multiple operations on the same object.

run:
run is a combination of 'let' and 'with' functions. Imagine a null value is assigned to an object and we want to prevent it from NullPointerException. In this case, we can use run function with that object.

Using run function:

class Person {
    var name: String = ""
    var age: Int = 0
}

fun main() {
    val person = Person()
    val result = person.run {
        name = "Vijay"
        age = 30
        "Name: $name, Age: $age"
    }

    println(result)
}

Output:
Name: Vijay, Age: 30

The run function also returns the result of the last expression inside its block, which is a string containing the name and age of the person.

also:
also is used where we want to perform additional operations when we have initialized the object member.

Using also function:

class Person {
    var name: String = ""
    var age: Int = 0
}

fun main() {
    val person = Person()
    val result = person.also {
        it.name = "Vijay"
        it.age = 30
    }.let {
        "Name: ${it.name}, Age: ${it.age}"
    }
    println(result)
}

Output:
Name: Vijay, Age: 30

The let function is called on the returned object (person). It takes a lambda with the parameter it, which refers to the object returned by also. Inside the let block, a string is created using the properties of the object returned by also. The result is assigned to the result variable.

apply:
apply is mostly used for initializing the objects. It is also used to configure specific attributes of that object.

Using apply function:

class Person {
    var name: String = ""
    var age: Int = 0
}

fun main() {
    val person = Person()
    val result = person.apply {
        name = "John"
        age = 30
    }.let {
        "Name: ${it.name}, Age: ${it.age}"
    }
    println(result)
}

Output:
Name: Vijay, Age: 30

The difference between apply _and _also is, apply is more focused on modifying and initializing objects and returning them, while also is more about performing side effects or additional actions without altering the object and then returning it.

Overview of Garbage Collector(GC)

Vijay Vimalananda Guru — Sat, 10 Sep 2022 19:25:07 +0000

In this article we will learn about Garbage Collector(GC) and a little bit of Memory Leak.

Before we jump into the Garbage Collector we will first learn,

What is memory leak?

Memory leak is a type of resource leak that is unused and failed to deallocate or release from memory(RAM).

When does memory leak occur and what is the outcome?

Memory leak occurs when GC could not release unused objects/resources. As a result it will reduce the amount of available memory and affect the performance of the application. However, it is not always the case where memory is full; it means there is a memory leak. Sometimes applications might be using a high amount of memory to perform heavy tasks.

How to prevent memory leak?

Apart from garbage collector, it is the work of the developer to program in such a way that it releases unneeded resources/objects wherever it is required. Another good way to prevent memory leak is by minimizing the use of static members.

On rare occasions you would have faced java.lang.OutOfMemoryError during application runtime. This is because JVM could not able allocate memory(heap space) for that application. Java programmers, please note that calling a static method (System.gc()) manually will not guarantee that it will remove all unreachable objects. At the same time don’t forget garbage collector plays a major role in allocating/deallocating objects from heap memory.

What is Garbage Collector?

GC is an automatic memory manager that manages the allocation and release of memory for an application.

Generally, whenever we open an application, the operating system will allocate a certain amount of memory for that application. GC will manage that memory throughout the application lifetime. It will run periodically with certain GC steps.

Garbage Collector Steps:

Initially, when an object is created in the heap, it will be a head node for the corresponding object and it will create a graph-like structure when the corresponding object holds the reference of others. Having that GC will follow three steps to perform garbage collection.

Mark: GC will walk through the object graph right from the beginning of the node and mark the objects that are reachable.

Sweep/Delete: It will ignore the objects that are marked as reachable and sweep off unreachable objects.

Compacting: The empty space left over after removing the unreachable objects will be defragmented by this step. This step will help to isolate memory that is being used and free.

Garbage Collector Graph:

The garbage collector traverses through the object graph in memory, starting from GC root to corresponding objects.

Generation of Garbage Collector:

Basically, objects in the heap are divided into generations - Young and Old(Tenured)

Young Generation:
Initially, objects are created in the young generation. It is further divided into:

Eden space
Survivor space (from space and to space)

Eden space is where objects are actually created. When Eden space gets full, GC kicks in and performs a minor garbage collection operation. The objects which are marked as reachable are then moved to different Survivor spaces and later during minor GC operation, reachable objects will be moving to and fro between survivors. Finally, after several generations it will be moved to the old generation.

Old Generation:
Objects are usually moved from survivor space to the old generation only when the maximum tenuring threshold limit has reached. In the old generation, objects will slowly become unreachable and subjected to major GC operations to free up heap memory allocated for that application. But often that is not the only case where major CG performed. When the heap is full and wants memory to store an object then a major GC operation occurs.

Types of GC:

GC in Java are subdivided into:

Serial GC:
This is the basic GC that runs in a single thread. It is specially designed for smaller applications. All the GC events will happen serially with a single thread.

Parallel GC:
This GC is specially designed for large applications that run on multiprocessor or multithreaded environments. Multiple threads are used for minor garbage collection that occurs in young generation.

Parallel Old GC:
It is the same as Parallel GC except that it uses multiple threads for both young and old generations.

Concurrent GC:
In a concurrent collector, multiple threads are used in minor garbage collection with the same algorithm used in parallel GC. It runs concurrently along with application threads to avoid “stop the world” situation. This GC uses more CPU than any other GCs.

Garbage First GC:
This is specially designed for multithreaded applications that have larger heap size. This GC is quite different from other GCs as there won’t be Young and Old generation. Instead there will be a set of equal size regions and uses multiple threads to scan them.

Epsilon GC:
This was released along with JDK 11. It handles memory allocation but doesn’t provide a memory reclamation mechanism. Once the allocated heap is full, JVM will shutdown. This GC is very rarely used and fit only for ultra-latency-sensitive applications.

ZGC:
This was released along with JDK 11 and improvised in JDK 12. This is mainly intended for applications which require low latency and/or use a very large heap.

Shenandoah:
This was released along with JDK 12. It is more intensive than any other GCs as it cleans garbage, compact live objects, and free up RAM immediately back to OS.

Short Summary:

GC is an automatic memory manager that performs certain operations to allocate and deallocate objects from heap memory. Objects that are reachable will be kept in the memory and objects that are unreachable will be removed to free up space for the application. GC will perform this action until the application lifetime so that it runs smoothly.

I hope this article helped you to understand something about Garbage Collector.

Happy learning!!

All about Android Runtime

Vijay Vimalananda Guru — Tue, 05 Apr 2022 11:51:05 +0000

In this post, we are going to learn about Dalvik and Android Runtime.

Most of the Android developers would have heard about the terms Dalvik, ART, JIT and AOT. If you ever wonder what these are then this post will help you.

Introduction:

Whenever APK is generated, part of that contains .dex files. Those files hold source code of the app as well as libraries that are used. These files are written using low-level code designed for a software interpreter which is called bytecode. When we run the app, the bytecodes which are written in the .dex file are translated by Android Runtime into machine code. This machine code provides instructions that can be directly understood by the machine and are processed by the CPU.

The compilation of bytecode into machine code can be done in various methods. To understand why Android Runtime came into action we have to move back in time and learn about Dalvik.

What is Dalvik Virtual Machine(DVM)?

Initially, when Android phones came to the market they were not as efficient as they are now. It is not only just because they have low RAM but also they used a compatible runtime called Dalvik. It is implemented to optimize particularly for one parameter - RAM usage. So instead of compiling the whole app to machine code they used the strategy called Just-In-Time(JIT) compilation.

With this JIT, the compiler works quite pretty like an interpreter. It compiles small chunks of code during the execution of the app. This helped Dalvik to save a lot of RAM. But this strategy has a serious drawback. Since this all happened at runtime it had a negative impact on runtime performance. Gradually, they optimized and enhanced Dalvik as a result some of the frequently run code was cached and not recompiled without a need. But still there was a boundary in that case because of limited RAM size.

In later days, apps were also getting bigger and that impacted the performance and caused problems for Dalvik. So, they introduced a new runtime called Android Runtime in Android L.

What is Android Runtime(ART)?

The way ART worked in Android Lollipop was completely different from what we saw in Dalvik. Like JIT in Dalvik, ART(Lollipop and Marshmallow) used a strategy called Ahead-of-Time(AOT) compilation. With this, instead of interpreting code at runtime code was compiled and converted to machine code before running the app. This approach widely improved runtime performance and is more efficient than JIT. Unfortunately, this also comes with some drawbacks as it consumes a lot more memory than Dalvik. Also, it took more time to install the app because the whole app needed to be transformed to the machine code plus it took more time to perform system update because all apps need to be reoptimized. To overcome this downside, JIT compilation was introduced back to ART along with a feature called Profile-guided compilation.

What is Profile-Guided Compilation?

Generally, most parts of the apps are rarely used. So, having the whole app precompiled was an overhead. That's why in Android Nougat, JIT was brought back to ART and modernized with another strategy called Profile-Guided Compilation.

Profile-guided compilation technique constantly improved the performance of Android apps as they run. Whenever the app is compiled, it automatically detects some methods that are frequently used. The pre-compile methods are cached and gave best performance. While other parts of the app stay uncompiled until they are actually used. This strategy provided excellent possible performance for key parts of the app while reducing the RAM usage.

After this change, ART doesn't impact the speed of app installation and system updates. The best part here is, the pre-compilation happens only while the device is charging or idle. But the weak part of this approach is that the user has to actually use the app in order to get the profile data as well as frequently used classes and methods that are pre-compiled. That means that the first few usage of the app can be pretty slow because JIT compilation is used then. To improve this initial user experience, ART was upgraded with a new feature called Profiles in the cloud. And it was introduced in Android Pie.

What is Profiles in the Cloud?

As we all know most of the users use the app in a same way. So in order to speed up the performance after installation, profile data of the user who has already used the app are being collected. This collected data is clustered into a file called a common core profile. When a new user installs the app, this file is downloaded along with the application. With the help of this file, ART can easily detect the pre-compiled classes and methods that are mostly used by the user. This approach helped new users to get startup and steady state performance when they install the app. Later, user-specific profile data is being collected and recompilation of the code is done for this particular user when the device is idle or charging. As all these things are done behind the scene by ART we don’t need to do anything manually to enable it.

Short Summary:

Dalvik VM was using JIT till KitKat. As it has some downsides Android Runtime(ART) was introduced in Lollipop to overcome the issue that Dalvik had. At the beginning ART used a technique called Ahead-of-Time(AOT). It also had some drawbacks like high RAM usage and more installation time so ART was enhanced with a strategy called Profile-Guided Compilation. Later it was upgraded and came up with Profiler in the Cloud.

Happy learning!!