Forem: Deepak Patil

How to Efficiently Run Meta LLaMA on a MacBook Air M1 with Limited RAM

Deepak Patil — Fri, 06 Dec 2024 15:16:20 +0000

Running advanced AI models like Meta’s LLaMA on a MacBook might seem ambitious specifically when you have M1 with 8 GB of RAM, But with the right steps, you can start building AI apps locally on your Mac easily. Thanks to Apple’s processor architecture and efficient libraries like Llama.cpp, you can unlock the power of large language models right from your lightweight laptop.

Let's get you started with the MacBook Air M1 for running these models efficiently.

Step 1:

Download the entire model from Meta's official site by providing your information & details for usage.

Install the necessary packages as mentioned in the readme file.

Run the below command to run the model

torchrun \
  --nproc_per_node=$NGPUS \
  llama_models/scripts/example_chat_completion.py $CHECKPOINT_DIR \
  --model_parallel_size $NGPUS

Definitely, this is not going to work ☹️. To solve this issue we will be following two methods below

first, we will be using llama.cpp which provides lightweight C++ implementation for running models on various hardware.
We will use quantization to reduce the model size so that we will be able to run it easily.

Step 1 (This one will work):

Install llama.cpp using brew.

brew install llama.cpp

Step 2:

Now let's quantize the llama model. To perform this we have a very good space on HuggingFace called GGUF-My-Repo. Follow the below link to go to space.

GGUF-MY-REPO

On this space login with your HuggingFace credentials. Then select the model repository that you want to quantize. For llama, you need to get access to the repo. Select the checkbox for 'Create a private repo under your username'. If you are a beginner then leave other options to default and proceed.

Once the process is done you will have a quantized model created in your private repository.

Step 3:

Clone the HuggingFace repo just created on your Mac. Run the model using the command below

llama-cli -m GGUF_MODEL_FILE_NAME -n 1024 -ngl 1 -c 512 --prompt PROMT cnv

example

llama-cli -m meta-llama-3.1-8b-q4_k_m.gguf -n 1024 -ngl 1 -c 512 --prompt "Hello" -cnv

You can also run directly from the repository name.

llama-cli --hf-repo Deepcodr/llama_sample_chat-Q4_K_M-GGUF --hf-file llama_sample_chat-q4_k_m.gguf -p "The meaning to life and the universe is"

Tip: If you are a beginner avoid using base models if you don't want to get some gibberish or random responses. Instead use instruct models such as chat, or text. etc. You can find some already quantized models here

How to upload react build files to S3 from GitHub Actions

Deepak Patil — Tue, 08 Oct 2024 07:25:07 +0000

Every time you push new changes to your React app, it’s live on the web within minutes—no manual uploads, no repetitive commands, just pure automation magic. With GitHub Actions and Amazon S3, you can make it happen but there is a catch, Very few prebuilt scripts are available for uploading react build files to S3 from GitHub actions. In this post, Let's see how to connect the dots between your React project and AWS S3 using GitHub Actions, turning every code push into an automatic deployment.

To begin with we need to create a script that will iterate over all files in the build folder and then use aws-sdk to upload all files to the S3 bucket configured with static website hosting. First we will import required packages to JS script.

const S3 = require('aws-sdk/clients/s3');
const path =require('path');
const fs = require('fs');
const klawSync = require('klaw-sync');
const {lookup} = require('mime-types')

Then we will initialize the S3 client and get build folder path which will be generated from github action with npm run build

var awsoptions = {
    accessKeyId : process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey : process.env.AWS_ACCESS_KEY
}

const buildFolderPath = path.join(__dirname,'build')

var s3 = new S3(awsoptions);

Then we will use klaw-sync to get file path for each file and then upload that file to the S3 bucket using upload() function.

const filePaths = klawSync(buildFolderPath,{
    nodir : true
});

filePaths.map((filePath)=>{
    const fileContent = fs.createReadStream(filePath.path);
    const bucketPath = path.join('',path.relative(buildFolderPath,filePath.path));


    config ={
        Bucket : 'YOUR_BUCKET_NAME',
        Body : fileContent,
        Key : bucketPath,
        ContentType : lookup(filePath.path) || "text/plain"
    }

    upload(config);
})

function upload(config){
    return new Promise((resolve) => {
        s3.upload(config, (err, data) => {
          if (err) console.error(err);
          console.log(`uploaded - ${data.Key}`);
          console.log(`located - ${data.Location}`);
          resolve(data.Location);
        });
    });
}

Thats it place this script file in react project directory where package.json is present. Run this script from your github action workflow after building the react project. Once called it will upload all files from build folder to bucket.

Example job steps for building & uploading files

    steps :
      - uses : actions/checkout@v4
      - uses : actions/setup-node@v4
        with :
          node-version : 20
      - name : Install node modules
        run : npm install
      - name : Build the project for production
        run : npm run build
      - name : Upload Files to S3
        run : node s3upload.js

With GitHub Actions and S3 working together, deploying your React static site is no longer a tedious task but an automated part of your workflow. This setup not only saves time but also ensures that every time your static files get pushed to the bucket while developer can focus on development.

What are AI accelerators or NPUs?

Deepak Patil — Thu, 12 Sep 2024 14:46:50 +0000

With the increasing popularity of AI software & technologies, there is a need for specialized hardware units to provide high performance to AI services. Traditional processors like CPUs and GPUs often struggle with the massive computational load required by advanced machine learning models. Neural Processing Units (NPUs), often known as AI accelerators or DL(Deep Learning) Accelerators, are the answer to this problem. NPUs are specialized hardware designed to boost AI tasks by optimizing performance and energy efficiency. These purpose-built processors are revolutionizing AI application performance and data processing.
NPUs perform various complex computational operations required by AI efficiently compared to GPUs which are more power-consuming and come at cost. NPUs have dedicated hardware circuits for training models on Trillions of parameters. Currently, GPUs are gaining popularity due to their performance in training the models. But upcoming NPUs are disrupting this trend providing much more performance than GPUs.NPUs implement in-memory computing architecture allowing them to replicate human brain-like working.

source : AWS

These devices can help build the most complex neural networks and vision applications. Implementing the circuit for performing three types of computations (scalar, vector, tensor) makes them the elite hardware for next-generation computers. Companies like Google, Intel, and AWS have already begun constructing their NPUs dedicated for computing devices and cloud infrastructures.AWS Inferentia & Trainium are the NPUs powering modern AI workloads in Cloud infrastructure.
With the research still going on to develop more efficient NPUs it is interesting to see what next things will come to the AI evolution.

How to install Nginx on Mac

Deepak Patil — Wed, 04 Sep 2024 19:25:32 +0000

Nginx is a widely used web server, valued for its speed and reliability in serving web content. Installing Nginx on macOS allows you to leverage its local development or testing capabilities. In this post, let's understand the steps to install Nginx smoothly on your Mac, making it easy to manage and serve your web projects.

Although you can install Nginx from the source code on MacOS, it is a bit complex. So, HomeBrew is the best option for installing Nginx.

Steps to install Nginx on MacOS

Install Homebrew Package Manager

You can install Homebrew from official website or Just enter below command to install.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Install Nginx

Use the homebrew command below to install nginx.

brew install nginx

Start & Use Nginx

Start the Nginx service using brew.

brew services start nginx

Now go to http://localhost:8080 and you should see the Nginx welcome page.
Below are some commands you can use for managing nginx service.

Stop

brew services stop nginx

Restart

brew services restart nginx

Status

brew services status nginx

By default nginx on Mac uses 8080 port. You can change nginx.conf and update the port number to 80 so that it will serve directly on localhost.

With Nginx now running on your macOS, you're ready to explore and deploy web apps with nginx.

Using S3 Bucket to host React Application

Deepak Patil — Mon, 26 Aug 2024 18:15:25 +0000

It becomes confusing when it comes to deploying a React application in production. Because there are several ways we can deploy a React application in production. But considering a static single-page website hosting, there is one popular way: AWS S3.

Hosting the React application in S3 not only provides a simple yet highly available deployment option but also brings other features to the table.S3 provides versioning, logging, and security features to make the experience more seamless. Integrating it with CloudFront makes it highly available, cacheable, and faster. S3 also allows CORS configuration, which can help to control access to the resources. Along with this bucket, it can be attached to a custom domain name configured with Route 53 or DNS, making a complete, full-fledged web application.

Considering these features, S3 becomes the best option for deploying React in production. It only takes a few steps to deploy a React application to S3.

Create a S3 bucket in AWS

Create an S3 bucket as per the required configuration and storage class. Make sure to allow public access to all objects and remove any other blocking access options.

Generate a Production Build of React application

Run the below command in the project root to generate a production build.

npm run build

this will create a build folder in the project directory with all static files.

Upload the build folder to S3 Bucket

Upload the contents of the build folder to the bucket along with the static folder.

Add a policy to a bucket for getObject

Edit the bucket policy to provide public access to get objects in the bucket. The policy will look like below.

{
  "Id": "Policy1729235341904",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1729235340511",
      "Action": [
        "s3:GetObject"
      ],
      "Effect": "Allow",
      "Resource": "YOUR-S3-BUCKET-ARN/*",
      "Principal": "*"
    }
  ]
}

Enable Static hosting for S3 Bucket

In the properties of the bucket enable static website hosting. Specify index and error document as index.html (Other names if differ). Save the changes.

Test the running website using the link

Copy the generated link and paste it into the browser to view your website running. Add a custom domain if required.

Thats it! In just a few clicks the React site is live using S3.

Which is the best stack for the web today? MERN vs MEAN vs Spring Boot

Deepak Patil — Mon, 12 Aug 2024 13:37:15 +0000

Many devs nowadays have a common question in their mind, which web frameworks and stacks are best to make a career? Well the question itself is unanswerable looking at the circumstances today. However, I will try to provide the best answer based on my experience and current market trends.

Let's consider the three most used and popular stacks in the market today which are MERN, MEAN and Spring. All three of these stacks come with unique functionalities and applications fulfilling different use cases. If we look at MERN Or MEAN, They are more likely to be used for complete dynamic web application development while on the other hand, spring boot is suited for microservices and loosely coupled REST API-based enterprise applications.

MERN and MEAN use javascript which makes it easy to write but still allows us to use it with modern features like NoSQL datasets, Caching etc. Similarly, Spring boot uses Java which is a language that is part of most of the legacy applications. Spring Boot makes a powerful framework that allows us to develop loosely coupled and serverless architectures. Looking at other features all stacks allow support for REST APIs, Scalability, and Fault tolerance by integrating with the cloud.

key features of MEAN, MERN, and Spring Boot:

Aspect	MEAN Stack	MERN Stack	Spring Boot
Database	MongoDB (NoSQL)	MongoDB (NoSQL)	SQL/NoSQL Databases
Server-Side Framework	Express.js	Express.js	Spring Framework
Development Paradigm	Full-Stack JavaScript, Single Page Application (SPA)	Full-Stack JavaScript, Component-Based Architecture	Enterprise-Level Application, Microservices Architecture
Auto-Configuration	No	No	Yes
Embedded Servers	No	No	Yes (e.g., Tomcat, Jetty, Undertow)
Microservices Support	Limited, typically not the primary focus	Limited, typically not the primary focus	Strong support for microservices
Production-Ready Features	Depends on configuration	Depends on configuration	Yes, includes metrics, health checks, etc.
Security	Depends on additional libraries	Depends on additional libraries	Built-in with Spring Security
Scalability	MongoDB provides scalability for data	MongoDB provides scalability for data	High scalability, especially in microservices architecture
Ecosystem	Angular ecosystem for front-end	React ecosystem for front-end	Comprehensive Spring ecosystem, including Spring Cloud, Spring Data, etc.
Development Speed	Fast due to full-stack JavaScript and easy to understand	Fast due to full-stack JavaScript and easy to understand	Fast with Spring Boot’s auto-configuration and Spring Initializr but little bit complicated

So deciding which is better is not possible. But according to me for learning a framework as a novice, MERN or MEAN stacks are the best. Because it will give a proper understanding of how web applications or REST APIs work. On another hand, if you are already well familiar with working with Java-based applications and technologies like servlets, hibernate etc. It is better to advance yourself with the power of Spring Boot.

Whichever framework you choose I prefer to dive deeper into it unless you feel lost in it and then you start exploring endless possibilities with it. Once you gain so much knowledge about a particular technology the question of the best stack to make career? becomes trivial.

v0.dev :) Future of Web UI. Will UI and Frontend Devs exist in future?

Deepak Patil — Fri, 26 Jul 2024 13:05:52 +0000

Building and designing UIs from UI/UX tools like Figma and XD with languages like HTML, CSS, and JS is way too far for today's GenAI Web Development. From these tools to website builders like WordPress and Wix, and from these platforms to AI website builders, is the next step in the world of web development. v0.dev by Vercel is the revolutionary platform that's turning the web development community on its head. With its innovative approach to design and automation, v0.dev raises the important question of whether frontend and UI developers will soon be extinct?

In modern web development scenarios, implementing and leveraging the powers of GenAI is a powerful way of automating the entire web development cycle. v0.dev implements a prompt-based UI creation tool based on Shadcn UI components and layouts. It is capable of generating UI within seconds with an elegant combination of components.

Have a look at what it has generated within just a few clicks and prompts.

Generate a UI for product catalog of ecom website

It will take a lot of time for an experienced developer to develop such a page as compared to v0. Not only this, it also provides theming and customization options to make it more beautiful and brandy. We can copy the whole code for the UI in JSX or HTML. Just some configuration for the backend and the full-fledged site will be ready within a few minutes. And how can we forget Vercel? Using which site can be deployed and made live within another few clicks.

Thus, using v0.dev and Vercel makes it a deadly combination for building web applications. Well, will it kill the jobs of web developers or UI designers is still a question. But the radical evolution in the field of web development with AI is an interesting thing to watch and observe what more it will bring to the table.

Using Scratch Base image provided by Docker

Deepak Patil — Thu, 11 Jul 2024 15:38:18 +0000

Building Base images in docker is not recommended but for those who are a bit interested in diving deeper in docker, this is a must-know thing. Even though the docker hub provides a huge collection of images for a variety of uses there are some cases where we would want to create something complex from SCRATCH !!!!.

That's where the Scratch base image from Docker comes in handy. There are several ways to generate base images, such as using an already-existing Linux machine or a running container, but for now, let's utilize a Docker scratch image.

To use scratch we just need to add FROM Scratch in our Dockerfiles. Yes, that's it! above this, you can add anything to configure your base image, Build it and it's ready to use.

For Example, let's create one. Create and Add the following to the Dockerfile.

FROM scratch
ADD hellofromdocker /
CMD ["/hellofromdocker"]

That's it here hellofromdocker is a static executable that does not have any runtime or linked dependencies. Since the scratch image only contains the bare minimum of runtime and settings needed to create base images, using other executables or apps may not work.

Once the Dockerfile is done. build an image using docker.
docker build -t helloscratchimg .

Spin up a container from a newly created base image and get hello from Docker! 🤩

How to automate NPM authentication to avoid providing credentials every time

Deepak Patil — Thu, 27 Jun 2024 18:29:20 +0000

You must have used npm for publishing your javascript packages or using private libraries and packages. For developers using npm is a daily habit for building projects. However, It becomes tedious when authenticating NPM for using or publishing any package.

We can avoid this by automating npm authentication. To do so follow the steps below.

Approach 1:

So the naive approach is setting up credentials as environment variables. You can add the following environment variables.

export NPM_CONFIG_EMAIL=<EMAIL>

export NPM_CONFIG__AUTH=<AUTHENTICATION_TOKEN>

export NPM_CONFIG_USERNAME=<USERNAME>

export NPM_CONFIG_PASSWORD=<PASSWORD>

You can get the Auth token from the NPMJS account settings page under Access tokens.

You can create two types of tokens Granular access token or classic token.

Under classic token, you can create use read-only, automation or publish token. Among these tokens, automation tokens bypass 2Factor Authentication.

While Granular Access tokens provide more security by configuring multiple options such as Validity, IP addresses and permissions.

Approach 2

Another way to perform this is by just updating the .npmrc file in the user's directory. We need to add the following line to the file. Add your auth token in place of .

//registry.npmjs.org/:_authToken=<YOUR_AUTH_TOKEN>

The next time you perform any operations related to NPM you can focus on development as there is no more need to provide credentials every time.

How I have added a custom ASCII Art Banner to the React Server

Deepak Patil — Sun, 16 Jun 2024 08:03:16 +0000

While working with React, I wanted to have a cool ASCII art banner while running the React application in the terminal. I have tried so many NPM packages and libraries but none helped. So I have decided to make a package and built reactasciibanner.

This package will help to add custom ASCII art while running React applications like below

To use this package. We just have to install it in the React application using npm.

npm i @deepcodr/reactasciibanner

Once installed add the ASCII art text to the file called ReactBannerArt.txt and put this file in the src folder of the application.

Then update the package.json start script command from

react-scripts start

react-banner start

And that's it you will see ASCII art in the terminal after running the application using npm start

package link @deepcodr/reactasciibanner

Video Tutorial To Use Package :

How I have added custom ASCII art banner in React Application | ASCII art banner in React terminal - YouTube

How I have added a custom ASCII art banner in React Application | ASCII art banner in React terminal | How to add custom ASCII art banner in React while runn...

youtube.com

Running Django Application On Kubernetes

Deepak Patil — Mon, 03 Jun 2024 19:16:31 +0000

Imagine your Django application running in Kubernetes equipped with features of Kubernetes building a strong scalable and highly available application. In this post let's see how to dockerize and deploy a Django application on Kubernetes.

Deployment of an application always starts with having a nice and error-free application. To begin with, we will use a sample Django application with a sample view to provide Hello World! response.

<-- You can use your application or To follow along you can refer to the code from the below repo -->

Deepcodr / django-hello-world

Now that the application is ready, start with dockerizing and building an image. For this, we will create a requirements.txt of required modules.



pip freeze > "requirements.txt"

Lets write a recipe for our docker image in Dockerfile. The minimal configuration for Dockerfile is provided below you can modify it accordingly.



FROM python:3.9.19-alpine
RUN mkdir /DjangoHelloWorld
WORKDIR /DjangoHelloWorld
COPY / .
RUN pip install -r requirements.txt
EXPOSE 8000
CMD python manage.py runserver 0.0.0.0:8000

To use SQLite database you can simply add migrations and make migration commands in CMD at the last line



CMD python manage.py makemigrations && python manage.py migrate && python manage.py runserver 0.0.0.0:8000

Done!✅ Build the docker image using below using Docker.



docker build -t deepcodr/node-hello-world .

That should create image for our application. Spin up a container if you want to test the application.



docker run -dp 8000:8000 deepcodr/node-hello-world

Test the application thoroughly and then we are ready for deployment.

Lets provide YAML configuration for Kubernetes Deployment and Service.



apiVersion : v1
kind : Service
metadata :
  name : djangohelloworld
spec :
  selector :
    app : djangohelloworld
  type : LoadBalancer
  ports :
    - port : 8000
      targetPort : 8000
---
apiVersion : apps/v1
kind : Deployment
metadata :
  name : djangohelloworld
spec :
  replicas : 1
  selector :
    matchLabels :
      app : djangohelloworld
  template :
    metadata :
      labels :
        app : djangohelloworld
    spec :
      containers :
      - name : djangohelloworld
        image : deepcodr/django-hello-world
        ports :
          - containerPort : 8000
        imagePullPolicy : Always

The above configuration is sufficient to run our application with minimal configuration. Save the configuration and apply it with kubectl



kubectl apply -f nodehelloworld.yaml

This should create a service and deployment in Kubernetes.Lets view running pods and services in cluster.



kubectl get svc
kubectl get pods

If you see a service and pods running. Your application should work 🎉

Test the application by hitting URL in browser. You will see your homepage.

Now that you know how to deploy Django application on Kubernetes 🗿, You are ready to build and configure more complex scenarios for Django with Kubernetes and explore Kubernetes in depth.

React App Deployment On Azure VM (NGINX) ⚛︎

Deepak Patil — Sun, 26 May 2024 14:40:17 +0000

Let's see how to deploy the React web application to Microsoft Azure Virtual Machine in this post. There are multiple ways to deploy React applications on a VM but we will see the most basic and easy one with the NGINX server with reverse proxy. So let's get to it.

We will start by setting up a sample virtual machine in Azure.

Create a VM with Ubuntu Server or Windows server I will recommend using Ubuntu.
Select the size of the VM and provide a username and password for accessing SSH.

Make Sure to enable HTTP and HTTPS ports on the VM

Once the VM is ready SSH to the VM using credentials

ssh username@<PUBLIC-IP-ADDRESS>

let's upload and get the application running.

To upload the project we can use either AzCopy Or Azure Storage. But as this is a basic deployment :), We will clone the project from GitHub. You can also refer to my project from below.

React Hello World Project

Install NodeJS using NVM or any other option. If decide to go with NVM follow the instructions on the below page.

Node Installation

Once the project is uploaded start and test the project whether it is running correctly.

npm start

Configure NGINX with Reverse Proxy 💡

Install the nginx server

sudo apt install nginx

Check whether the service has started using the below command.

systemctl status nginx

That's it now let's add configuration for our VM Public IP in NGINX.
copy the default configuration from sites-available directory and save it with name as Public IP of VM

cp /etc/nginx/sites-available/default /etc/nginx/sites-available/<PUBLIC-IP>

Open the newly created configuration file and update the configuration as below.

server {
    listen 80%;
    listen [::]:80;
    # SSL configuration 
    #   
    # listen 443 ssl default_server;
    # listen [::]:443 ssl default_server;
    #
    #3 Note: You should disable gzip for SSL traffic.
    # See: https://bugs.debian.org/773332
    #
    # Read up on ssl_ciphers to ensure a secure configuration.
    # See: https://bugs.debian.org/765782
    #
    # Self signed certs generated by the ssl-cert package
    # Don't use them in a production server!
    #
    # include snippets/snakeoil.conf;
    #3 root /var/www/html;
    # Add index.php to the list if you are using PHP
    server_name 172.203.177.104;
    location / {
        proxy_pass http://127.0.0.1:3000;
        # First attempt to serve request as file, then
        # as directory, then fall back to displaying a 404.
        # try_files $uri $uri/ =404;
    }
}

Save the file and create a symlink of the file in sites-enabled directory.

ln -s /etc/nginx/sites-available/<PUBLIC-IP> /etc/nginx/sites-enabled/

Restart the nginx server

systemctl restart nginx

Now start the React server and hit the public Ip in the browser you should see a similar output as below.

move the react server running in background and close the SSH session. This will persist react server from stopping and we will be able to access the application directly.

use ctrl+Z to stop the server. then use bg command to add process to background.

bg

now disown the process to make keep it running in bg.

disown -h

Close the SSH session we are done with deployment 🎉.

logout

Finally hit the public IP you should be able to access the application deployed.