Forem: Sharath Hebbar

LLM Model Sharding

Sharath Hebbar — Thu, 11 Apr 2024 10:12:55 +0000

Introduction

Large Language Models (LLMs) represent a significant advancement in artificial intelligence and natural language processing.

Models such as OpenAI’s GPT (Generative Pre-trained Transformer) series, Google’s Gemini, PaLM, T5, and many such open-source models have achieved remarkable capabilities in understanding and generating human-like text.

However, as these models grow larger to improve performance, they also pose challenges in terms of scalability, resource requirements, and ethical considerations.

A major challenge is using such models. Leave alone using the LLM in Colab, Kaggle notebook, or locally with less amount of RAM, even loading such huge models need high RAM which is not a feasible solution.

So one such solution will be model sharding which converts the huge models into smaller chunks which in turn takes less time and consumes less hardware for loading such huge models.

Here we will discuss model sharding using Open Source LLM Mistral 7B freely hosted on HuggingFace Platform.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from accelerate import Accelerator, load_checkpoint_and_dispatch

model_name = "mistralai/Mistral-7B-v0.1"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    low_cpu_mem_usage=True,
    torch_dtype=torch.float16
)

accelerator = Accelerator()

accelerator.save_model(
    model=model,
    save_directory=save_directory,
    max_shard_size="200MB"
)

device_map={"":'cpu'}

model = load_checkpoint_and_dispatch(
    model,
    checkpoint="/content/model/",
    device_map=device_map,
    no_split_module_classes=["Block"]
)

new_model = "<Name of the model>"
HF_TOKEN = "<Your HF Token>"

tokenizer.push_to_hub(
    new_model,
    token=HF_TOKEN
)

model.push_to_hub(
    new_model,
    token=HF_TOKEN
)

Loading Sharded Model

The original model took 16GB RAM to load the full model in 16-bit floating point

%%time
model_name = "mistralai/Mistral-7B-v0.1"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    low_cpu_mem_usage=True,
    torch_dtype=torch.float16
)

CPU times: user 36.8 s, sys: 48.5 s, total: 1min 25s
Wall time: 3min 30s

%%time
model_name = "Sharathhebbar24/Mistral-7B-v0.1"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    low_cpu_mem_usage=True,
    torch_dtype=torch.float16
)

CPU times: user 23 s, sys: 48.7 s, total: 1min 11s
Wall time: 1min 49s

The sharded model took 3GB RAM to load the full model in a 16-bit floating point.

References
HF Docs: https://huggingface.co/docs/transformers/en/big_models
Using Accelerate: https://huggingface.co/docs/transformers/en/main_classes/model#large-model-loading
Medium: https://medium.com/@sharathhebbar24/llm-model-sharding-55102ecb1823
Github: https://github.com/SharathHebbar/Model-Sharding
Reference: https://medium.com/@jain.sm/sharding-large-models-for-parallel-inference-ee19844cc44#:~:text=Memory%20Efficiency%3A%20Sharding%20enables%20running,parts%2C%20reducing%20memory%20requirements%20significantly.

Joblib

Sharath Hebbar — Sun, 25 Jun 2023 07:52:32 +0000

Joblib

Joblib is a set of tools to provide lightweight pipelining in Python. In particular: transparent disk-caching of functions and lazy re-evaluation (memoize pattern) easy simple parallel computing.

Why it is used?

Better performance
reproducibility
Avoid computing the same thing twice
Persist to disk transparently

Features

Transparent and fast disk-caching of output value
Embarrassingly parallel helper
Fast compressed Persistence

Importing libraries

from joblib import Memory,Parallel, delayed,dump,load
import pandas as pd
import numpy as np
import math

Data Creation

my_dir = '/content/sample_data'
a = np.vander(np.arange(3))
print(a)
output: [[0 0 1]  [1 1 1]  [4 2 1]]

Memory

mem = Memory(my_dir)
output: [[ 0  0  1]  [ 1  1  1]  [16  4  1]]
sqr = mem.cache(np.square)
b = sqr(a)
print(b)
output: [[ 0  0  1]  [ 1  1  1]  [16  4  1]]

Parallel

%%time
Parallel(n_jobs=1)(delayed(np.square)(i) for i in range(10))
output: CPU times: user 2.85 ms, sys: 0 ns, total: 2.85 ms
Wall time: 3 ms
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
%%time
Parallel(n_jobs=2)(delayed(np.square)(i) for i in range(10))
output: CPU times: user 42.7 ms, sys: 762 µs, total: 43.5 ms
Wall time: 75.9 ms
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
%%time
Parallel(n_jobs=3)(delayed(np.square)(i) for i in range(10))
output: CPU times: user 92.9 ms, sys: 8.93 ms, total: 102 ms
Wall time: 151 ms
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Dump

dump(a,'/content/sample_data/a.job')
output: ['/content/sample_data/a.job']
Load
aa = load('/content/sample_data/a.job')
print(aa)
output: array([[0, 0, 1],        [1, 1, 1],        [4, 2, 1]])

References

Documentation: https://joblib.readthedocs.io
Download: https://pypi.python.org/pypi/joblib#downloads
Source code: https://github.com/joblib/joblib
Report issues: https://github.com/joblib/joblib/issues

Source:
https://medium.com/r/?url=https%3A%2F%2Fgithub.com%2FSharathHebbar%2FData-Science-and-ML%2Ftree%2Fmain%2Fcodes%2Fjoblib

Access Git(Github/Gitlab .. ) using multiple accounts on the same system.

Sharath Hebbar — Sun, 25 Jun 2023 07:44:20 +0000

Often you need to use multiple GitHub/GitLab accounts from the same system and you use the command line to do this. But while trying this you will face so many errors as you can only add one user to use git services, so below are the steps.

Step 1:

Install the Git command-line tool on your system, if it is not already installed.

The link to download git is as follows, you can choose your package according to your OS

https://git-scm.com/download

Step 2:

Generate an SSH key for each of the GitHub accounts you want to use. You can do this by running the following command, replacing “email@example.com” with the email address associated with your GitHub account:

Note: You can see .ssh folder in C:\Users[Your Username].ssh

if not exists then it will automatically create .ssh folder

ssh-keygen -t rsa -b 4096 -C “email@example.com”

Step 3:

When prompted, enter a file in which to save the key. It is recommended to use a different file for each key, so you can easily distinguish between them. For example, you might use “id_rsa_account1” for one account and “id_rsa_account2” for another.

Ex: C:\Users[Your Username].ssh \id_rsa -> C:\Users[Your Username].ssh \id_rsa_account1

Step 4:

Follow the prompts to enter a passphrase for the key. This passphrase will be used to encrypt the private key, so make sure to choose a strong and unique passphrase.

You can even not give any passphrase

Two files will be generated for each key

Id_rsa_account1

id_rsa_account1.pub

Step 5:

Once the key has been generated, copy the public key (.pub) and paste it in your github account ssh section.

[In Github go to settings > SSH and GPG > New SSH key > Add Title of your choice > In key section paste the public key ]

Step 6:

Repeat the steps above for each of the GitHub accounts you want to use.

Step 7:

Create a config file in the same directory (.ssh)

account 1

Host github.com-account1

HostName github.com

User git

IdentityFile ~/.ssh/id_rsa_account1

account 2

Host github.com-account2

HostName github.com

User git

IdentityFile ~/.ssh/id_rsa_account2

Step 8:

Now Initialize a git repository in any of the desired location.

Then add all the required files that need to be committed

Then commit those files with an appropriate commit message

Then while adding the remote repo Take the ssh URL and change github.com to github.com-account1

Ex: git remote add origin git@github.com-account1:dummy-proj/dummy-proj.git

You can do the same for git clone too.

Note:
To change your default account
git config user.name “account1”
git config user.email “account1@gmail.com”

Links:
https://github.com/SharathHebbar/Data-Science-and-ML/tree/main/articles/github-multiple-accounts

Kubeflow Installation

Sharath Hebbar — Sun, 25 Jun 2023 07:37:31 +0000

Installing the Kubeflow pipeline(locally) in the Windows system
Installing Kubeflow Pipeline (KFP) on Windows can be a bit challenging since KFP is primarily designed to run on Linux-based systems. However, you can set up a Windows-based development environment and run KFP using a Docker container.
The following steps will help you to install Kubeflow pipelines for windows system
Step 1: Install Docker Desktop
Docker Desktop: https://docs.docker.com/desktop/install/windows-install/
Step 2: Install Minikube
Minikube installation link: https://minikube.sigs.k8s.io/docs/start/
Open up your PowerShell in Administrator mode and type in this command
Download and run the installer for the latest release

New-Item -Path 'c:\' -Name 'minikube' -ItemType Directory -Force

Invoke-WebRequest -OutFile 'c:\minikube\minikube.exe' -Uri 'https://github.com/kubernetes/minikube/releases/latest/download/minikube-windows-amd64.exe' -UseBasicParsing

Add the minikube.exe binary to your PATH.
Make sure to run PowerShell as Administrator.

$oldPath = [Environment]::GetEnvironmentVariable('Path', [EnvironmentVariableTarget]::Machine)
if ($oldPath.Split(';') -inotcontains 'C:\minikube'){ `
[Environment]::SetEnvironmentVariable('Path', $('{0};C:\minikube' -f $oldPath), [EnvironmentVariableTarget]::Machine) `
}

Step 3: Install K8s
Kubectl commands
https://www.kubeflow.org/docs/components/pipelines/v1/installation/localcluster-deployment/
To deploy the Kubeflow Pipelines, run the following commands:


Set PIPELINE_VERSION=1.8.5
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=$PIPELINE_VERSION"
kubectl wait - for condition=established - timeout=60s crd/applications.app.k8s.io
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/env/platform-agnostic-pns?ref=$PIPELINE_VERSION"

Verify that the Kubeflow Pipelines UI is accessible by port-forwarding:

kubectl port-forward -n kubeflow svc/ml-pipeline-ui 8080:80