Forem: Sreekesh Iyer

Getting started with GPU Programming on an EC2!

Sreekesh Iyer — Tue, 30 Dec 2025 14:00:12 +0000

There's a short backstory to this thread, and it starts with a phone call from my undergrad Uni in September '25. It was the Head of the Department of IT at the other end, asking me if I were willing to take a technical knowledge-sharing session for the students.

I was obviously in, I just needed an excuse to go back to the campus. It's been 2 years since I graduated and I dearly miss this place. Having said that, the topic of discussion was open-ended, so they left that on me to decide. I thought about many things - AI, Blockchain, Data Science, etc. but stumbled upon one thing that I had already been exploring lately, and I was almost certain that this was something most of the undergrad engineering students wouldn't have come across yet.

Such is the world of GPU programming. GPUs - the curiosity that got me into gaming as a kid, exploring the compute prowess they bring to a machine. At the time, my understanding was as simple as - "big GPU, smooth game".

Thanks to this gift of cloud, access to GPUs is much easier than back in the day. Looking at today's marker, accessing a cloud GPU is much easier than getting hands on an actual one even if you had the money XD

Finding a GPU Instance on EC2

In just a few clicks, you can find yourself in a shell of a machine that has a powerful Nvidia GPU.

You might need a service quota approval to get access to the g4dn.xlarge type of EC2 instance, it's a fairly straightforward thing to request from the Service Quotas Dashboard.

Once that is done, you can spin up an Ubuntu instance.

Installing CUDA

Once you've entered your EC2 Instance on the shell, you can download the latest CUDA Toolkit from its official documentation with the network install. Looks something like this.

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-ubuntu2404.pin
sudo mv cuda-ubuntu2404.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/13.1.0/local_installers/cuda-repo-ubuntu2404-13-1-local_13.1.0-590.44.01-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2404-13-1-local_13.1.0-590.44.01-1_amd64.deb
sudo cp /var/cuda-repo-ubuntu2404-13-1-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update
sudo apt-get -y install cuda-toolkit-13-1

Following the same instructions on the download page you may need to install Nvidia drivers separately.

The entire thing takes only a few seconds running commands one after another. Running nvidia-smi on the terminal showing you this nice little table is a sign that the installation is completed perfectly.

Troubleshooting

If nvcc --version doesn't work, validate that it's there in the local directory and add it to PATH in bashrc.

echo 'export PATH=/usr/local/cuda/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

Basics of CUDA

It took me an hour long C++ refresher to get comfortable working with CUDA files. You can get started by cloning my GitHub repository, it has two basic CUDA programs to work with.

git clone https://github.com/sreekeshiyer/hello-cuda.git

The most fundamental idea of working with something like CUDA is that you want to parallelise your compute process to get as many tasks done as possible. I talked in-depth about how CPU and GPU are different and why the GPU is able to complete a lot more tasks in less time purely based on its hardware differences.

Let's walk through the most basic vector addition program to understand how to work with CUDA.

Vector addition walkthrough

When you start reading through vector_add_simple.cu in the basics directory, you'll realise how similar CUDA is to C++. There are two key differences, however.

#define BLOCK_SIZE 256

// ---------------- Define GPU Kernel ----------------
__global__ void vectorAdd(const float *A, const float *B, float *C, int N) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;  // Compute global thread index
    if (i < N) {
        C[i] = A[i] + B[i];
    }
}

// 5. Launch kernel on GPU
    cout << "[5] Launching kernel...\n";
    int blocksPerGrid = (N + BLOCK_SIZE - 1) / BLOCK_SIZE;
    cout << "    → Grid size: " << blocksPerGrid << " blocks\n";
    cout << "    → Block size: " << BLOCK_SIZE << " threads\n";

    vectorAdd<<<blocksPerGrid, BLOCK_SIZE>>>(d_A, d_B, d_C, N);
    cout << "    ✅ Kernel execution complete.\n";

I want you to closely observe these two code-blocks. The first one where we define the kernel. The operation, as one would expect, is rather simple - you are adding two one-dimensional vectors. The only difference being that we're going to distribute 1 operation to each thread, which is why we compute the thread-index first.

After that, we're running this function in blocks and we're pre-defining blocks in a grid. I have a nice article from Nvidia that covers these terminologies in depth, you might want to take a look. I spent about 5-10 minutes on the whiteboard in the class trying to explain the concept of threads, blocks and grids.

Compiling the CUDA Program

It's quite easy to run a CUDA program. You first compile it using nvcc and you can run the outfile that gets generated as a result.

### The flag -o is to mention the outFile name as 00
nvcc -o 00 vector_add_simple.cu
### You can `ls` in the directory and find this outFile once compiled. 
### Then you can open this outFile as an executable which eventually runs your program. 
./00

We're essentially adding million-sized vector arrays and you'll notice that the entire operation completes in an instant. But that doesn't tell us much unless we compare it to a regular CPU-based program.

Comparing Results with a CPU

We explored the capabilities of parallel computing with a GPU using a simple vector addition program. But how does it actually fair against a CPU? This is what the students in the class were the most curious about. I've added another file vector_add_benchmark.cu in the same folder for this very purpose. It basically runs the entire thing on a CPU (single-threaded) and then on the GPU to compare the time difference.

You'll notice a stark difference (2600x in my case, results may vary but you can expect a general trend). While 32ms doesn't look huge, you can try playing around and expanding the array size, at which point you'll see the difference expanding as the CPU takes longer and longer.

I don't expect this article to be a "Welcome to CUDA", and I apologise if you expected it to be. I just wanted to walk through and document the little session I took in my university as a good habit. Overall, I had very positive feedback for the session. Feel free to reach out to me if you want to take a look at the deck ( PS - I make good decks wink-wink )

Thanks for reading, hope you're having a lovely time with your friends and family in the holiday season. I'll see you again in 2026 :D

Kiro - The New Agentic AI IDE from AWS

Sreekesh Iyer — Mon, 14 Jul 2025 18:38:09 +0000

Meet Kiro, a developer sitting inside an IDE to literally breeze you through projects from scratch and ship things faster than ever before.

I've been using Kiro for the last few days for my side projects and it's worked an absolute charm. Let me take you on a little tour of this AI-powered IDE.

Spoiler alert: It's a lot more than just code.

Opening the app makes it seem like you're in a themed version of VSCode - and you're not entirely wrong. It's essentially VSCode with a few additions that make it an entirely different beast.

I want you to look at two key features of this screen that'll allow me to give you an overview of what Kiro offers out of the box.

The Kiro Sidebar

This panel houses the key features of Kiro - Specs, Agent Hooks, Agent Steering and the MCP Servers. I'll talk each one of them in detail in upcoming blogs.

The Chat Panel

It's a classic chat window with two interesting modes - Vibe Coding and Code With Spec. I like how 'Vibe Coding' is the default version of the chat.

When you have a very open-ended idea of what you want to build and have no real constraints, this vibe coding mode allows Kiro to 'go crazy' on your prompts and do its thing.

On the other hand, if you know exactly what you're building, you can leverage the spec mode to describe the entire flow of things. Kiro creates 3 files - requirements.md, design.md and tasks.md and it will keep modifying those files as documentation while it asks you clarifying questions about what you want to build.

A layer of top of this is Agent Steering. When you enable agent steering for your project, you're providing Kiro added context about your project by default for every prompt. You can enable it from the Command Pallette. It creates 3 files - tech.md, structure.md and product.md that houses a proper outline of what you're building along with its nitty-gritty features that can be narrowed down to even variable naming styles. I feel that this gives you maximum control of your project.

I can't wait to experiment more with Kiro and unlock its full potential.

Hope that was a good initial preview on what Kiro can bring to the table, and I'm sure there's a lot more on the way. There's no point in waiting, try it for yourself at kiro.dev! I'll see you in the next one:)

Setting up a Queue Ingestion system for S3 to Redshift Transfer

Sreekesh Iyer — Sat, 21 Dec 2024 09:19:42 +0000

I've always been curious about data pipelines and how we are able to work with data engineering in the middle of everything in my company.

Literally one object moves in 500 different formats through 500 different systems for perhaps 500 different purposes.

The base object is a transaction that has all the raw L0 data stored in Object Storage. However we cannot pull out Object reads all day for analytics, so this needs to be moved into a separate database. Let me correct myself - moved into multiple databases in different forms, for different reasons - Confirmations, Regulatory Reporting, etc.

I was trying to see if I could develop a PoC of sorts, that can simulate this pipeline with the help of our friends in AWS.

Here's what we have -

Amazon S3: Object Storage
Amazon SQS: Message Queue
Amazon Redshift: SQL-ish database for preserving transformed data for further analysis.
AWS Lambda: Compute for performing business logic to enrich the base data, push this to SQS and then pull it from SQS again to send it out to Redshift.

Let me bring up the architecture diagram from the cover image again.
We're storing the initial piece of information in S3. This is where our first Lambda function gets triggered.

We take the baseline object and perform some analysis and transformation on it. Consider this as your business logic that you'd perform at L1 before the data is sent further downstream.

The transformed object is then sent out via the SQS Queue. This message gets fetched by the 2nd Lambda which is virtually polling the queue. Finally, this message is transformed into an SQL Query and the data is inserted into a database in Redshift serverless.

Now that we have a better picture, let's hop into action.

Setting up the Queue

We can setup a simple FIFO Queue in Amazon SQS to ensure ordered delivery of messages. I'll walk you through the code separately where we avoid message duplication.

The 1st Lambda Function

We're writing a function that is waiting for an object to be loaded into S3 so that it can be processed.

I didn't find an exact method to set the SQS queue as the destination, I am not sure if it's a permission issue because I wouldn't be able to touch it via boto3 either, but I was. Here, the S3 is our trigger and SQS is our programmatic destination.

The code is available on GitHub.

Here's where the business logic for our transactions goes in. Given that this is serverless and decoupled fully, this can be as complex as it can get. I've attached a simplified version below -

Object Storage via S3

It can be a standard bucket in the same region, just make sure you add this event notification in the S3 Properties to invoke the Lambda we just created.

The 2nd Lambda Function

We're now inside the analytics side of things where it'a all about ETL and further processing in Redshift. For starters, we need a function that can trigger from an SQS message and subsequently pass it on to Amazon Redshift.

Here's what this function looks like. Waits for a message in the SQS queue, performs a validation and pushes it to Redshift. Since we are using a FIFO Queue, we get access to a Deduplication value from the SQS Message which we can use to filter out duplicate messages.

We can also perform type-validations easily because "thanks Python!"

Redshift - the last checkpoint

You can make use of the default workgroup in Redshift serverless and the base "dev" database where you can create a new table "transactions" to insert all of this data.

After that, all you need to do is upload a JSON file to the S3 Bucket that looks like this and see the magic:)

I hope you found this thread informative. The core idea was to decouple compute and storage which we are able to achieve here.

Do follow the blog for more. Until next time - Cheers, Happy Holidays!

Simplifying Reinforcement Learning Terminologies - AWS DeepRacer Edition

Sreekesh Iyer — Sun, 01 Dec 2024 16:37:55 +0000

If you’re new to machine learning, reinforcement learning, especially, then it’s completely okay to have a hard time understanding these fancy jargons in front of you. Here’s my attempt to simplify these while we work on a little robo-car :)

About AWS DeepRacer

A contest that is discontinued after this year, unfortunately, DeepRacer is arguably the most entertaining service available on the AWS Console today. This is the arena where you can train a virtual car to run through a racetrack, and the challenge is to complete 3 laps as quickly as possible without going off-road.

—

Before I get to the jargons, let’s first try to understand

What is Reinforcement Learning?

Reinforcement learning (RL) is an interdisciplinary area of machine learning and optimal cont rol concerned with how an intelligent agent should take actio ns in a dynamic environment in order to maximize a reward signal ~ Wikipedia

While I barely made it through that sentence, let’s imagine reinforcement learning as training a dog to fetch the ball for you. Every time the doggo fetches the ball, you give the little guy some treats. You’re giving him positive reinforcement, or as we call it incentive, to perform that action. This is basically what we do in RL as well. We give a reward to our intelligent agents based on their actions to get a desired result.

In case of AWS DeepRacer, the agent is a car, and the desired result is for it to complete the track without going off-track, as quickly as possible. This means that our rewards need to account for the speed of the car, the nature of the track, the turns and more.

Now that we have an understanding of what RL is, let’s try to understand the hyperparameters for our model that may have an impact on our end-results.

Batch Size

Models based on Neural Network train in iterations, meaning they update themselves after every round of learning. This hyperparameter gives you control over how much your model learns in every iteration. Keep in mind that all hyperparameters have their benefits and trade-offs. In this case, a larger batch size (say 128 over 64) would mean that the model would have a wider set of experiences to learn through, and generate a steady output, however, over iterations, it would take more memory and time.

In Deepracer context, the batch size can essentially decide on the exploration that your model will remember across various iterations of its training. Larger batch sizes can delay the convergence times but with every iteration, it’ll get your model closer to the optimal path.

Entropy

I heard about this term for the first time in Chemistry, and very simply - it’s a measure of randomness. Think about how much you want your model to explore while training. The possibilities it can figure out to eventually find out an optimal result. In case of your robo-car, it can find different ways to cover a track, for example. taking fast, shorter turns, taking slow, longer turns, etc. and it eventually finds out the best option based on the incentive it gets.

You might want to reduce the entropy with time as the model matures, as you’d figure out it’s closer to the optimal path, you wouldn’t want it to diverge from that.

Discount factor

A value that ranges between 0 and 1, it signifies the importance your model gives to short term gains over the long term rewards. Discounting Factor closer to 1 means that you prefer future success over immediate high rewards on that particular action on the iteration. Whereas a factor closer to 0 means that it’ll prioritise the incentives it just received. In DeepRacer, it’s critical that the car has a long-term foresight so as to ensure it doesn’t land itself is bad positions around curves or turns that can lead to crashes.

Learning rate

This parameter basically controls how quickly your model reaches the point of convergence. However, with higher learning rates, you run the risk of overshooting, wherein you might get to an optimal path quicker and immediately diverge from it in the next iteration. The key, like most other parameters, is balance. You can adjust the learning rate based on observation of the patterns that your model creates while training. In DeepRacer, this can potentially define how quickly your model is able to adapt to changes, eg. sharp turns after a straight road, where your car is initially intending to run at full speed, but then has to slow down and turn 140 degrees.

Number of epochs

Epoch - can be defined as a collective batch of experiences combined by various episodes in a training window. Your complete process can have many different batches, meaning the model has to move on from one set to another. The number of epochs signifies the amount of time your model spends on one such set. This can essentially refine how your model thinks over a certain set of experiences. Longer epochs naturally mean longer training times and compute usage.

My Experience with DeepRacer

[Update Feb 2025]

It's really sad to see DeepRacer go, as of writing this article I was completely unaware of the news that we just had the last year of the AWS DeepRacer competition. I worked with a lot of my colleagues in my firm to come up with solid reward functions to go further in the comp.

As a small achievement, I received an Asia Top 50 Regional award along with a 100$ Amazon Gift Card :)

Hope this was a good read to cover various Deep Learning jargons and simplify your experience working with neural networks going forward. Thank you for reading:)

Setting up a REST API in Python for DynamoDB

Sreekesh Iyer — Fri, 15 Nov 2024 06:30:00 +0000

Dynamo DB is AWS's NoSQL offering in the vast set of managed databases as a service that they provide. Like most other services, it's fully serverless, flexible and easy to scale.

The Data Model

As we're working on NoSQL here, there's no real restriction on the structure of data. We can operate with key-value pairs as the attributes for each item in a table. Let's look at these keywords again.

Table - a fairly familiar term, it is essentially a collection of data, in this case, items. It is also the starting point of working with DynamoDB on the console.

Item - an entry in a table. You could consider it a row in an SQL-equivalent database.

Attribute - The datapoints that constitute an item. It could contain item-specific attributes, metadata, or virtually anything that can be associated with an item.

You could think of a JSON array as an equivalent to a table in DynamoDB. I'm sure things will get clearer as we create our own table.

Setting up the database

It's literally a piece of cake to create a new table in DynamoDB from the AWS Console. All you need is a name and a partition key, which is your primary key in this case. This will help you search for items in the table.

I'm creating a table for all the games that I've played, and I will rate them out of 10 :)

You can mess with the table directly from the console, let's try adding a new item to see what it looks like.

My first entry has to be my favourite RPG (role-playing) game - The Witcher 3. I will add a new attribute for rating and it's going to be a solid 9.8 from me :)

Setting up an API

Right, it's now time to write some Python code to do all of this without the GUI ;)

## app.py
from flask import Flask, jsonify, request
import boto3
from boto3.dynamodb.conditions import Key
import uuid  # Import uuid module for generating UUIDs

app = Flask(__name__)

# Initialize DynamoDB client
dynamodb = boto3.resource('dynamodb', region_name='ap-south-1')  # Replace with your region
## Do keep in mind to save your AWS credentials file in the root directory
table = dynamodb.Table('games')  # Replace with your table name

# Route to get all games
@app.route('/games', methods=['GET'])
def get_games():
    try:
        response = table.scan()
        games = response.get('Items', [])
        return jsonify({'games': games}), 200
    except Exception as e:
        return jsonify({'error': str(e)}), 500

if __name__ == '__main__':
    app.run(debug=True)

The beauty of Python is that you can setup a full-fledged API in just a few lines of code. This chunk of code is now sufficient for us to access the table and fetch the data from it. We use the scan function to fetch items from the games table.

You can start the app by using python3 app.py

And you can expect a response that looks like this when you curl for the /games endpoint.

Routes for creating and updating an entry

# Route to create a new game
@app.route('/games', methods=['POST'])
def create_game():
    try:
        game_data = request.get_json()
        name = game_data.get('name')
        rating = game_data.get('rating')
        hours = game_data.get('hours', 0)

        # Generate a random UUID for the new game
        id = str(uuid.uuid4())

        if not name or not rating:
            return jsonify({'error': 'Missing required fields'}), 400

        # Store the game in DynamoDB
        table.put_item(Item={'id': id, 'name': name, 'rating': rating, 'hours': hours})

        # Return the created game with the generated UUID
        created_game = {'id': id, 'name': name, 'rating': rating}
        return jsonify({'message': 'Game added successfully', 'game': created_game}), 201
    except Exception as e:
        return jsonify({'error': str(e)}), 500

# Route to update an existing game
@app.route('/games/<int:id>', methods=['PUT'])
def update_game(id):
    try:
        game_data = request.get_json()
        name = game_data.get('name')
        rating = game_data.get('rating')
        hours = game_data.get('hours', 0)

        if not name and not rating:
            return jsonify({'error': 'Nothing to update'}), 400

        update_expression = 'SET '
        expression_attribute_values = {}

        if name:
            update_expression += ' #n = :n,'
            expression_attribute_values[':n'] = name
        if rating:
            update_expression += ' #r = :r,'
            expression_attribute_values[':r'] = rating
        if hours:
            update_expression += ' #h = :h,'
            expression_attribute_values[':h'] = hours

        update_expression = update_expression[:-1]  # remove trailing comma
        response = table.update_item(
            Key={'id': id},
            UpdateExpression=update_expression,
            ExpressionAttributeNames={'#n': 'name', '#r': 'rating', '#h': 'hours'},
            ExpressionAttributeValues=expression_attribute_values,
            ReturnValues='UPDATED_NEW'
        )
        updated_game = response.get('Attributes', {})
        return jsonify(updated_game), 200
    except Exception as e:
        return jsonify({'error': str(e)}), 500

Here, we are using put_item to add an item to the table. For updating a record, we use the function update_item.

If you observe carefully, we are using UpdateExpression where we specify the attributes that we're updating. This allows us to control exactly which attribute gets changed and avoid accidental overwrites.

And to delete the record, you can go with something like this -

response = table.delete_item(
    Key={'id': id},
    ReturnValues='ALL_OLD'  # Optional: returns the item that was deleted
)

Well, there you have it, you just setup a REST API with CRUD Functionality for DynamoDB in a matter of minutes thanks to Python.

Integrated AWS Workflows with SageMaker Notebooks

Sreekesh Iyer — Sat, 09 Nov 2024 11:13:17 +0000

I was recently a part of a corporate hackathon and was found searching for good Jupyter environments on the cloud to be able to work on my ML Project.

Soon enough, I found myself here. However, it took me a while to figure out how powerful it truly can be.

For starters, I was able to hook it up to any Repository within seconds. So I did not have to worry about versioning or sharing it with teammates.

Direct access to GPU instances

You’ll need to get an additional approval to make use of the ml.g-series of instances in SageMaker Notebooks, but it didn’t take longer than a minute for me. And just like that, I have access to a Jupyter notebook with TensorFlow and a GPU.

Integrations with other AWS Services

This is not a Sagemaker feature as much as it is an AWS IAM feature, but when you get hold of the IAM Role that is being used by the Sagemaker Notebook, you can basically open up your Python shell to have access to literally any AWS service via client libraries like boto3.

You can control EC2 instances, invoke Lambda functions, send emails via SES, perform operations on S3 objects and so many other things programmatically through the same Jupyter notebook.

—

So far, it’s been fun watching folks work on Sagemaker Notebooks trying out different ways to automate workflows on AWS using raw Python. Looks like I’ve found my new provider for easy-access Cloud Notebooks.

My experience with the AWS Community 'New Voices' Program

Sreekesh Iyer — Wed, 27 Dec 2023 17:09:44 +0000

Hi folks!

The New Year is just around the corner and that means that the applications for the AWS Community Builder Program are soon going to open up!

As a Community Builder this year, I'd like to take you through some of my best experiences - becoming a part of the AWS New Voices Cohort.

About AWS New Voices

The New Voices Program is a very new, building idea of gathering potential speakers from the pool of AWS Community Builders and honing their speaking skills under the guidance of experienced speakers from the community who chip in as coaches.

The Overall Setup

Set up in July 2023, we had a total of 4 sessions (once in a week) in which we had to prepare a small talk of 3-4 minutes where we described how we solved a problem using technology (not necessarily related to AWS)

Every new session, a different twist was added to the same topic of discussion to bring in the quality of range to our speaking skills.

Barring the last session, there were no slides. It was all about talking to your audience for a certain period of time, describing how you solved a problem

At the end of the talk, the host along with the Coaches give you actionable feedback that you can work on before you join the next session.

The Power of Acknowledgement

As someone who comes with a history of sheer social anxiety and nervousness on the stage, to be able to stand in front of a crowd and speak today is a blessing.

But it's also still very scary. And definitely not easy for everyone. What helps is the acknowledgement of people listening to you, people telling you that you did a good job and they liked your talk.

This is exactly what I got from this program. At the end of the talk, the host appreciates your effort, talks about the stand-out points in your talk. All the speakers appreciate your work and mention the key highlights where you did well. Everybody jazz hands at you (refer to the cover image) acknowledging the solid work that you did.

I'm not sure about the future of the program, how it's going to be structured in 2024, but I'm exited and super happy to be an AWS New Voices Graduate. Looking forward to more Community Builders joining in Q1 becoming a part of this program as well :)

Thanks for reading, Happy Holidays!

Building a Llama2 Langchain powered Simple Chat Bot hosted on Napptive

Sreekesh Iyer — Sun, 20 Aug 2023 05:40:37 +0000

Hey folks! 👋

Not so long ago, I came across a post from LangChain on the Threads App about how easy it is to create a chat assistant using Llama2.

Here's the tutorial that you can look into, thanks to Anil-matcha who shared it on GitHub.

I've been fascinated by how difficult it generally is to physically host an LLM, but first, let's create an app.

Creating a working environment on Napptive

I sneaked into the documentation a bit and found out that the model can work just on the CPU, but it does better if you also have a GPU.

Since this is all Python, we can create the app on a Jupyter environment as well. The easiest way to create a Jupyter environment on the cloud is by using Napptive.

To learn more about Napptive and an extensive guide on hosting your Jupyter Lab on Napptive, here's one of my old articles you can refer to.

Before you deploy the app though, you'll have to make a couple of changes to the config YAML file. Firstly, expose the port 8501 for streamlit and create a napptive-ingress trait for it. This is because we'll be using Streamlit for the app UI.

Once that's done, you can deploy the app.

Building the App

Once you get into the Jupyter environment, create a new terminal, we have a few things to set up before we get going.

# Install langchain, the llama module and streamlit
pip install langchain llama-cpp-python streamlit
# Download Llama2 model from hugging-face
wget https://huggingface.co/TheBloke/Llama-2-13B-chat-GGML/resolve/main/llama-2-13b-chat.ggmlv3.q4_0.bin

mkdir .streamlit
nano .streamlit/config.toml
## Add this to the config
[theme]
primaryColor="#F63366"
backgroundColor="#111827"
secondaryBackgroundColor="#6B7280"
textColor="#FFFFFF"
font="sans serif"
## Ctrl + S to save the file, Ctrl + X to exit

The UI for this is inspired by the Streamlit ChatBot tutorial which you can find here.

Once the config is setup, create a new file app.py

import streamlit as st

# App title
st.set_page_config(page_title="💬 LLama2 Langchain ChatBot")

# Store LLM generated responses
if "messages" not in st.session_state.keys():
    st.session_state.messages = [{"role": "assistant", "content": "How may I help you?"}]

# Display chat messages
for message in st.session_state.messages:
    with st.chat_message(message["role"]):
        st.write(message["content"])

# User-provided prompt
if prompt := st.chat_input():
    st.session_state.messages.append({"role": "user", "content": prompt})
    with st.chat_message("user"):
        st.write(prompt)

Initially, we'll just import streamlit, setup an array to store all chat messages in a streamlit-session and then create the UI for the chat.

from langchain.llms import LlamaCpp
from langchain.callbacks.manager import CallbackManager
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
from langchain import LLMChain, PromptTemplate
from langchain.memory import ConversationBufferWindowMemory

We'll import all the different classes and functions we'll use from Langchain.

callback_manager = CallbackManager([StreamingStdOutCallbackHandler()])

# Make sure the model path is correct for your system!
llm = LlamaCpp(
    model_path="./llama-2-13b-chat.ggmlv3.q4_0.bin",
    input={"temperature": 0.75, "max_length": 2000, "top_p": 1},
    callback_manager=callback_manager,
    verbose=True,
)


# Function for generating LLM response
def generate_response(prompt_input):

    template = """I am a Large Language Model Llama set up on Langchain.
    {history}
    Human: {human_input}
    Assistant:"""

    prompt = PromptTemplate(input_variables=["history", "human_input"], template=template)

    chatbot = LLMChain(
        llm=llm,
        prompt=prompt,
        verbose=True,
        memory=ConversationBufferWindowMemory(k=2),
    )
    return chatbot.predict(human_input=prompt_input)


# Generate a new response if last message is not from assistant
if st.session_state.messages[-1]["role"] != "assistant":
    with st.chat_message("assistant"):
        with st.spinner():
            response = generate_response(prompt) 
            st.write(response)

    message = {"role": "assistant", "content": response}
    st.session_state.messages.append(message)

Further, we initialize the LLM Class with the model that we downloaded, write a function that creates the chain to generate responses and lastly, the UI to showcase those responses.

Refer to the snippet below for the entire code from the file app.py

import streamlit as st
from langchain.llms import LlamaCpp
from langchain.callbacks.manager import CallbackManager
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
from langchain import LLMChain, PromptTemplate
from langchain.memory import ConversationBufferWindowMemory

# App title
st.set_page_config(page_title="💬 LLama2 Langchain ChatBot")

# Store LLM generated responses
if "messages" not in st.session_state.keys():
    st.session_state.messages = [{"role": "assistant", "content": "How may I help you?"}]

# Display chat messages
for message in st.session_state.messages:
    with st.chat_message(message["role"]):
        st.write(message["content"])

# User-provided prompt
if prompt := st.chat_input():
    st.session_state.messages.append({"role": "user", "content": prompt})
    with st.chat_message("user"):
        st.write(prompt)

callback_manager = CallbackManager([StreamingStdOutCallbackHandler()])

# Make sure the model path is correct for your system!
llm = LlamaCpp(
    model_path="./llama-2-13b-chat.ggmlv3.q4_0.bin",
    input={"temperature": 0.75, "max_length": 2000, "top_p": 1},
    callback_manager=callback_manager,
    verbose=True,
)


# Function for generating LLM response
def generate_response(prompt_input):

    template = """I am a Large Language Model Llama set up on Langchain.
    {history}
    Human: {human_input}
    Assistant:"""

    prompt = PromptTemplate(input_variables=["history", "human_input"], template=template)

    chatbot = LLMChain(
        llm=llm,
        prompt=prompt,
        verbose=True,
        memory=ConversationBufferWindowMemory(k=2),
    )
    return chatbot.predict(human_input=prompt_input)


# Generate a new response if last message is not from assistant
if st.session_state.messages[-1]["role"] != "assistant":
    with st.chat_message("assistant"):
        with st.spinner():
            response = generate_response(prompt) 
            st.write(response)

    message = {"role": "assistant", "content": response}
    st.session_state.messages.append(message)

Running the App

To run this file, create a new Terminal window and run the following command -

streamlit run app.py

If the service is up and running, you'll see a similar message in the shell from Streamlit.

You can check the app following the link in the streamlit endpoint on the Napptive Console.

While you enter the prompts on Chat, you can also check out what it does on the Terminal.

Final Results

Here's a demo of what the app looks like.

The app runs fine, but I've come across a few issues, for starters it's very slow (understandably because it's on CPU) and secondly, sometimes it loops itself where after finishing the response it automatically creates a new question and starts answering it.

What could've been better?

I didn't spend too much time on the UI, so you can see that the chatbot just dumps the response instead of streaming it word-by-word as you would see on other bots like ChatGPT.

That's it for this article, I hope you liked it, stay tuned on this series because I'm pretty sure there's a lot more to come. In the next article, we'll try something similar with the help of AWS Sagemaker :)

As always, thanks for reading, keep spreading your love, Cheers! 🥂

The Easiest way to install Docker Engine on a Linux Server

Sreekesh Iyer — Sun, 30 Jul 2023 14:03:54 +0000

If you're working on a local machine, you can easily download and install Docker Desktop for your operating system - Windows, Mac or even Linux Desktops these days.

However, if you're someone like me who does most of the work on the Cloud, or to be specific, remote environments, this may not be the right choice for you.

Something that I discovered pretty late is that Docker, for quite some time, has had a super fast solution to install Docker Engine in Linux environments.

If you check the website https://get.docker.com, you will find out that Docker maintains a script to install Docker Engine on Linux.

As they mentioned, this method is not intended for production environments, which I assume anyway will be working with a specific Docker version.

The best part about this script is that you need absolutely no package managers to work with.

Installing Docker using the Script

# Download the script
curl -fsSL https://get.docker.com -o install-docker.sh

# Run the script as it tries to figure out the type of Linux
# distribution it's dealing with and installs the latest CLI and Engine versions
sudo sh install-docker.sh

Simply running these two commands will install the latest version of the Docker Engine, its CLI and a few other tools/plugins like Compose, Buildx, etc.

Run Docker without Root access

Now, if you run any docker command after installing, you'll probably come across a similar error. The quick solution to this is to run the same commands with sudo, but it has been fairly annoying to me. But there's also a quick 1-command fix to this.

sudo usermod -aG docker <USERNAME>
sudo reboot

The flags a and G in the command are to add the user to a group, and just by running the command and rebooting your system, you can run docker commands without root access. Even though this requires one-time super-user access, I feel that it's definitely time-saving.

Thanks for reading, I hope you have a good day ahead :)

Self Hosting Supabase on AWS

Sreekesh Iyer — Thu, 27 Jul 2023 06:44:30 +0000

Hi there, folks!

Supabase is a BaaS (Backend-as-a-service) I've been using for my full-stack projects for almost 2 years. It's an amazing service that is open source, runs on top of Postgres and brings cool features like auth (even third-party auth support for a zillion providers), storage, realtime and much more.

I have a Flask Series in which I use Supabase as a backend. Check it out!

Why self-host Supabase?

You can totally work with Supabase on the Cloud, as easy as it gets. But if you are on its Free Tier, you have limited access to some of the features of the application. And that's any service, not just Supabase.

I feel that is why it's great to have the freedom to self-host these service-based applications because you get full control of how they run.

What this does is that it gives you control over the infrastructure on top of which it runs, so you can take care of the scaling process as and when needed.

Or you don't want to see Ant's emails every week. Yeah, that's about it.

In this article, I'm going to cover how we can quickly get Supabase up and running self-hosted on an AWS EC2 Instance. Let's get going!

The Supabase Documentation already mentions an article on how you can self-host it using Docker. We're going to follow it as well :)

Before getting started, let's take a look at the things we are going to need:

An AWS Account
Preferably a few AWS credits, you may incur a bill

Let's start by creating our EC2 instance on AWS!

Setting up the EC2 Instance

When you're launching a new EC2 instance, make sure you select the Ubuntu AMI, any instance type that has more than 4GB RAM, and preferably 20GB of storage.

For a more detailed guide on launching an AWS EC2 Instance, check out this blog.

Another thing is that you need to ensure you have all TCP ports and IP addresses open for inbound connections in your Security Group (not recommended in production, of course)

Once you've launched the instance, you can connect to it on the browser using EC2 Instance Connect, or if you've got a thing for fancy terminals like me, use SSH along with your key-pair to use your good old Terminal. Keep in mind, you use the Public IPv4 address along with the username ubuntu to connect.

Installing all dependencies

Before we start installing dependencies, we're going to update our package list by running sudo apt-get update -y

After that, we're going to install Docker.

Installing Docker

curl -fsSL get.docker.com -o install-docker.sh
# This scripts installs the latest version of Docker Engine in your Linux machine
sudo sh install-docker.sh

## If you want to use Docker aommands without sudo or admin privileges
sudo usermod -aG docker ubuntu
## After running this command, reboot the instance for it to reflect

We'll install Docker from its install-docker script on the official Docker website. You can continue using Docker as a super user OR just update the local user group and reboot your EC2 instance. It doesn't take long.

Next up, we're going to set up Supabase!

Setting up Supabase

## Following from their documentation
# Clone their repository
git clone --depth 1 https://github.com/supabase/supabase

# Go to the docker folder
cd supabase/docker

# Copy the fake env variables
cp .env.example .env

As mentioned in their documentation, all we need to do is clone the Supabase repository, copy and change the environment variables and get going!

As good practice, NEVER work with the default environment variables.

nano .env
## Once you've made the changes, Ctrl+S to save and Ctrl+X 
## to quit Nano and return to the shell.

## Then run docker compose up command to start the server
## This spins up 
docker compose up

You can use Nano to edit the .env file and make changes to your keys. You can follow the documentation to generate new keys and secrets. Once you've done that, you can run docker compose up to get things up and running!

Check out The Studio

At this point, you can check out if the Supabase Studio runs on http://<YOUR_PUBLIC_IP>:3000/projects

If you want to stop the service, you can do so by pressing Ctrl+C or running docker compose down if you're on a different shell window.

Recommended Cleanup

Make sure you terminate your EC2 instance after you've stopped using it, or you may have a few unpleasant AWS billing-related emails coming your way.

Are the AWS billing emails worse than Ant shutting down your Supabase projects every week? Debatable. We can talk about it.

That's it for this article, folks. Hope you liked it, I'll see you in the next one :)

Cost Saving in AWS

Sreekesh Iyer — Tue, 25 Apr 2023 11:30:00 +0000

Cloud services can be funny, they obviously provide you with high-quality services for computing, storage, etc. but can also crank up your bills even before you realize it.

AWS, like most cloud services, asks you for your credit card at sign-up. You're charged based on the time and usage of a service.

Any piece of code/event that can potentially take a lot of computing power or storage can trigger a massive bill going ahead. To know what we're up against, here are some of the bill-scare stories you might wanna read -

I don't know about you but those numbers are scary, especially how quickly they go up. Bills can go up exponentially in a matter of hours.

How do you avoid Bill shocks in AWS?

Handling large bills once they are in is a different ball game, but first priority should always be to prevent these bills in the first place. If you are someone like me who is just trying to experience the Cloud for the first time, learning how different services work, you should try not to have any bills at all.

Understanding Free Tier Limits

AWS has a pretty generous Free Tier that lasts for 12 months after creating your account.

A lot of services like AWS Lambda, Cognito, Cloudwatch, DynamoDB and SNS are free until you reach a threshold limit.

Whereas, services like EC2, S3, RDS, etc. provide free services until the threshold limits for only 12 months, i.e. until you're a part of the Free Tier.

On the other hand, there are also services like AWS SageMaker, Redshift and Lightsail that are available on a trial period for 2-3 months with their respective limitations.

Set Up Budgets

One solid way to avoid massive bills in AWS or any other cloud service is to set up Budgets and Budget Actions that can alert you when things start to get out of hand.

Search for Billing in your AWS Console and open up the Billing Dashboard.

The Billing Dashboard provides a complete overview of all monetary information about your AWS account.

Creating a Budget in AWS

Under Cost Management, click on Budgets to open up the Budgets page.

Now click on Create to start working on a new Budget.

Budget Threshold

We'll go ahead with a Cost Budget and give a name to our new Budget. We'll choose Monthly as the period for our Budget. Since I am a student and I don't really have any cloud services running, I'll just choose the threshold to be 1$. We can proceed to the next step now.

Budget Alerts

This is one of the key steps in creating a Budget, which is to set up an alert that notifies you via email when your bill is close to the budget threshold. Click on Add to create a new Alert.

Here, you can configure the percentage of the bill when there is an alert popping up and also a list of emails to notify. You can also use SNS and ChatBot as notifiers.

So in my case, when the bill reaches 0.75$, I get an email.

Forecasted Alerts

In one of the stories, you saw how a startup company got a 72000$ bill in a matter of just a few hours. There are possibilities you end up being too late to see the alert. Here's where Forecasted Alerts come into play.

Every time there is a tiny increment to your bill from 0.00$, a forecasted cost is generated based on your current usage to predict what your bill would be at the end of the month. You can use this metric to create an alert, based on which you can identify if any service is costing you more than usual. Just create a new alert and choose Forecasted instead of Actual in the Trigger parameter.

Budget Actions

You can have AWS automatically perform certain actions when a budget crosses a certain alert threshold by attaching an IAM Service Role to the alert under Actions.

With this role, you can target a certain instance of a service or multiple instances and/or services based on your requirements. This would also require the particular role to have all the necessary permissions to perform the said actions.

And that's all you need to do, to set up your budget.

Some Technical Tips to Save Costs

Write Efficient Code

Everything on the cloud is event-based and every time there is a new event, it potentially adds to your bill. This is why it's critical to avoid infinite loops in the cloud, where one service is triggered after the other and there is no end.

Focus on how every time there is a new user, a new record, a new entry, how many extra operations it takes to run the same function. This gives you a good idea of the complexity of your code. Higher the complexity, the larger the bill in the long run.

Follow Proper Security Practices

Use environment variables whenever you're using secret keys or API keys in your code. Use strong passwords and MFA for logging into your AWS account. Make use of IAM Users, Policies and Permissions to access different services. Any vulnerability can be exploited and any misuse can cause serious problems.

Scale only when required

Auto Scaling is a very useful feature of the Cloud, but it can be dangerous when not controlled. It is recommended that you only scale as much as required. There are maximum thresholds available for all services and they can significantly affect your bills.

Terminate/Delete Services that are not in use

Just like saving electricity, it's recommended to turn off services that you're not using anymore. This is key especially if you are exploring the Cloud like me. Once done using a certain service, it's best practice to just shut it down to avoid usage costs.

BONUS

Here's one last bonus feature I'd like to talk about in AWS that can help you reduce your costs.

AWS Cost Anomaly Detection

AWS Cost Anomaly Detection, as a part of the AWS Cost Explorer API, uses advanced Machine Learning Techniques to identify anomalous costs that can help avoid your bills skyrocketing suddenly. You can set it up and receive alerts on SNS, Emails even periodically. This not only helps you avoid surprise bills but also analyzes the root cause of the hike.

That's it from me in this thread, I hope it comes in handy in your Cloud journey. Thanks for reading :)

CICD in AWS using CodePipeline and EBS

Sreekesh Iyer — Sun, 26 Mar 2023 11:50:33 +0000

The Cloud has made a lot of things simpler for developers. In this thread, let's explore how easy it is to set up a CICD Pipeline in AWS using GitHub, AWS CodePipeline and Elastic Beanstalk.

What is CI/CD?

CI/CD is the combined practice of continuous integration (CI) with continuous delivery or continuous deployment (CD). The purpose of CI/CD is to allow developers and teams to deliver code changes more frequently and reliably.

The reference to the word pipeline is to explain how all of these processes are executed in a sequence, one after the other. The key to CI/CD is automation. The process can be more and more efficient if there is not a lot of manual involvement. It is crucial to have an efficient pipeline when you're dealing with large codebases in production since a lot of changes are made at different parts of an application.

Prerequisites

An AWS Account (preferably Free Tier so that you are not charged for using cloud infrastructure)

Tools Used

GitHub
AWS CodePipeline
AWS Elastic Beanstalk

For this demonstration, we'll use a sample PHP application and deploy it to Elastic Beanstalk.

Creating a Deployment Environment in Elastic Beanstalk

Open up Elastic Beanstalk in the AWS Console and click on Create Application.

Give a name to your application, choose PHP for the platform and Sample Application for your code type. Now, click on Create Application.

Wait for a minute, while Beanstalk creates a sample environment for you to deploy your application. By default, it creates an EC2 instance, a security group, an Auto Scaling group, an Amazon S3 Bucket, Amazon CloudWatch alarms and a domain name for your application.

You can click on the URL to visit the application. You'll get a custom PHP welcome page by default.

Getting your sample code

A sample application is provided in a tutorial for creating a simple pipeline in the AWS Documentation. You can either upload those files to your GitHub repository or simply fork my GitHub repository here.

Since we are using GitHub as the source, we do not need the scripts folder, so you can delete it if you want. The scripts folder can come in handy if you are using Amazon S3 as the source for your application :)

Creating a CodePipeline

Open up the CodePipeline Dashboard. In this step, we will create a simple pipeline where we specify our GitHub Repository for the source code and deploy the code to the Beanstalk app we just created.

Step 1

Enter a name for your pipeline and allow AWS to create a service role for this pipeline.

Step 2

Create a Connection to Connect your pipeline to GitHub.

Ideally, you should only allow access to the specific GitHub Repository you're currently using.

After the connection, choose the repository and the branch and go with the default CodePipeline for Output artifact format.

Step 3

Skip the build stage (since we do not have any build command) and fill in the Beanstalk application details in the Deployment Stage and lastly, review the data and click on Create Pipeline.

View Pipeline Progress

Wait for a few minutes while the Pipeline is executed and you can see that the process was completed. You can click on AWS Elastic Beanstalk which takes you back to the Beanstalk application we created. Now, if you click on the URL, you can see that our application is successfully deployed.

Congratulations, we just deployed our application to AWS Elastic Beanstalk via a pipeline.

Make Changes to re-run the pipeline

Open up the pipeline repository on GitHub and make some changes in the index.html file using the default GitHub text editor.

After making the changes, quickly switch back to the pipeline and observe the changes. You can notice that the pipeline is being re-executed on your new commit.

Once the pipeline execution is complete and successful, go back to the website.
As you can see, our changes were immediately reflected in our application and we didn't have to do anything extra for it.

You can take this to the next level, where you deploy your Node.js or Python projects to Beanstalk, set up Tests, and run them on every new commit to your production branch.

Recommended Cleanup

It is good practice to clean up/delete/terminate all services you have used on AWS (on any Cloud really) so that you are not charged in the future.

Firstly, delete your CodePipeline. Then, go to the Elastic Beanstalk Dashboard and delete your Application. This will also delete and terminate all other services used by the application.

This was a small demonstration of CICD in AWS. I hope you liked it. If you have used AWS CodePipeline before, do share your experience.

Thank you for reading :)