Forem: Manoj Kumar

How to upgrade from Ubuntu 22.04 to 23.04 using terminal [2024]

Manoj Kumar — Sat, 30 Dec 2023 01:09:17 +0000

Step 01:

Get all the apt updates

sudo apt-get update and
sudo apt-get upgrade

to make sure all the packages are in the latest version.

Step 02:

Update the system using the below command.

sudo do-release-upgrade

This might warn you that if you were to continue livepatch will be turned off. Type y and continue.

If you encounter an error saying...

Go to /etc/apt/sources.list.d and locate the winehq-<release>.sources and delete it. This will do the trick. If it throws an access denied error, try it with root privilages.

sudo rm -rf winehq-<release>.sources

Make sure you replace the with whatever the version that shows in your directory.

Example: winehq-jammy.sources

Then redo the Entire step 02 from beginning.

Moving on....

If you get a "third party entries in sources.list disabled" warning, type y and continue

Then the update starts to download and

Remove unnecessary packages
Install new packages needed for the new version
Upgrade existing packages

This will be a lengthy process depending upon the internet connection speed. So sit back and relax.

Once everything is downloaded (about 2.7GB) it will start upgrading them.

GitHub Metric Microservices Deployment on AWS using ECR - Part 2 - Issue Service & PR Service

Manoj Kumar — Fri, 10 Nov 2023 06:11:22 +0000

In Part 1, we looked at setting up the AWS configurations and the implementation of Commit Service. Let's look at how the rest of the microservices are implemented. This is a lengthy part so make sure to read until the very end and move to part 3.

Issue Service

This service fetches the user’s connected issues and saves them to the DynamoDB table – deviq_issues and returns the issues in order – Issues Created by user, issued assigned to the user and issues closed by the user. This service runs on port 5021.

issueService.py

import requests
import issueServiceUtil
import boto3

from flask import Flask

issueApp = Flask(__name__)

@issueApp.route('/')
def index():
    return "Hello! from Issue Service"

session = boto3.Session(region_name='<aws_region')
dynamodb = session.resource('dynamodb')
table_name = '<table_name>'
table = dynamodb.Table(table_name)



@issueApp.route('/issues/<username>')
def get_user_issues(username):
    json_array_response = []
    maxId = getPid.getMaxPid()
    token = getPid.get_gh_token()
    # Get issues created by the user
    created_url = f"https://api.github.com/search/issues?q=author:{username}"
    created_response = requests.get(created_url, headers={"Authorization": f"token {token}"})

    if created_response.status_code == 200:
        created_issues = created_response.json()['items']
        print(f"Issues created by {username}:")
        for issue in created_issues:
            table.put_item(Item={
                'pid':maxId,
                'gh_username': username,
                'issue_title': issue['title'],
                'url':issue['html_url'],
                'type': 'CREATED'
                })
            maxId+=1
            print(f"- {issue['title']} ({issue['html_url']})")

        json_array_response.append(created_issues)

    else:
        print(f"Error: {created_response.status_code}")

    # Get issues assigned to the user
    assigned_url = f"https://api.github.com/search/issues?q=assignee:{username}"
    assigned_response = requests.get(assigned_url, headers={"Authorization": f"token {token}"})

    if assigned_response.status_code == 200:
        assigned_issues = assigned_response.json()['items']
        print(f"Issues assigned to {username}:")
        for issue in assigned_issues:
            table.put_item(Item={
                'pid':maxId,
                'gh_username': username,
                'issue_title': issue['title'],
                'url':issue['html_url'],
                'type': 'ASSIGNED'
                })
            maxId+=1
            print(f"- {issue['title']} ({issue['html_url']})")
        json_array_response.append(assigned_issues)
    else:
        print(f"Error: {assigned_response.status_code}")

    # Get closed issues by the user
    closed_url = f"https://api.github.com/search/issues?q=author:{username}+is:closed"
    closed_response = requests.get(closed_url, headers={"Authorization": f"token {token}"})

    if closed_response.status_code == 200:
        closed_issues = closed_response.json()['items']
        print(f"Closed issues by {username}:")
        for issue in closed_issues:
            table.put_item(Item={
                'pid':maxId,
                'gh_username': username,
                'issue_title': issue['title'],
                'url':issue['html_url'],
                'type': 'CLOSED'
                })
            maxId+=1
            print(f"- {issue['title']} ({issue['html_url']})")
        json_array_response.append(closed_issues)
    else:
        print(f"Error: {closed_response.status_code}")

    return json_array_response

if __name__ == '__main__':
    issueApp.run(debug=True, host='0.0.0.0', port=5021)

Quick peek into issueServiceUtil.py

import boto3
import json

# Initialize a DynamoDB client
dynamodb = boto3.resource('dynamodb', region_name='<aws_region>')

# Specify the table name and column (attribute) name
table_name = '<table_name>'
column_name = '<key>'

# Initialize a DynamoDB table resource
table = dynamodb.Table(table_name)

# Perform a scan operation to get all items sorted in descending order by the specified column
response = table.scan()

def getMaxPid():
    # Check if any items were returned
    if 'Items' in response:
        items = response['Items']
        if len(items) > 0:
            max_value = max(item[column_name] for item in items)
            print(f"The maximum value for {column_name} is: {max_value}")
            return max_value + 1
        else:
            print(f"No items found in the table")
            return 1
    else:
        print(f"Error in scanning the table")
        return -1


def get_gh_token():
    secret_name = "<SECRET_NAME>"
    region_name = "<aws_region>"
    session = boto3.Session()
    client = session.client(service_name='secretsmanager', region_name=region_name)
    get_s = client.get_secret_value(SecretId = secret_name)

    secret = get_s['SecretString']
    ss = json.loads(secret)
    return ss['github_token']

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: issueservice-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: issueservice
  template:
    metadata:
      labels:
        app: issueservice
    spec:
      containers:
      - name: issueservice
        image: <ecr_image_id:tag>
        ports:
        - containerPort: 5021

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: issueservice-service
spec:
  selector:
    app: issueservice
  ports:
    - protocol: TCP
      port: 5021
      targetPort: 5021
  type: LoadBalancer

PR (Pull Request) Service

This service fetches the user’s assigned pull requests and saves them to the DynamoDB table – deviq_prs and returns those Pull Requests. This service runs on port 5069😉

prService.yaml

import requests
from datetime import datetime
import boto3
import getPid

from flask import Flask

prApp = Flask(__name__)

date_format = "%Y-%m-%dT%H:%M:%SZ"

session = boto3.Session(region_name='<aws_region>')
dynamodb = session.resource('dynamodb')
table_name = '<table_name>'
table = dynamodb.Table(table_name)

@prApp.route('/')
def index():
    return "Hello! from PR Service"

@prApp.route('/pr/<username>')
def get_user_pullrequests(username):
    token = getPid.get_gh_token()
    repo_url = f"https://api.github.com/users/{username}/repos?type=owner"
    response = requests.get(repo_url, headers={"Authorization": f"token {token}"})
    prs =[]

    maxId = getPid.getMaxPid()
    if response.status_code == 200:
        repositories = response.json()
        for repo in repositories:
            repo_name = repo["name"]
            owner = repo["owner"]["login"]

            pr_url = f"https://api.github.com/repos/{owner}/{repo_name}/pulls?state=all"

            # Step 2: Get the list of pull requests for each repository
            response = requests.get(pr_url, headers={"Authorization": f"token {token}"})
            pull_requests = response.json()

            for pr in pull_requests:
                pr_created_at = pr["created_at"]
                pr_closed_at = pr["closed_at"] if pr["closed_at"] else pr["updated_at"]

                # Step 3: Calculate the resolution time for each pull request
                resolution_time = (datetime.strptime(pr_closed_at,date_format) - datetime.strptime(pr_created_at, date_format)).total_seconds() / 60
                print(f"Pull Request #{pr['number']} in {owner}/{repo_name} resolved in {round(resolution_time, ndigits= 3)} minutes.")

                table.put_item(Item={
                'pid':maxId,
                'gh_username': username,
                'PR No': f"Pull Request # {pr['number']}",
                'repo_name': f"{owner}/{repo_name}",
                'resolving time': int(round(resolution_time, ndigits= 3))
                })

                pr = {
                'PR No': f"Pull Request # {pr['number']}",
                'repo_name': f"{owner}/{repo_name}",
                'resolving time': int(round(resolution_time, ndigits= 3))
                }
                #pr_json = json.dumps(pr)

                prs.append(pr)
                maxId+=1

    else:
        return f"Error: {response}"

    return prs
if __name__ == '__main__':
    prApp.run(debug=True,host='0.0.0.0', port=5069)

and the prServiceUtil.py

import boto3
import json
# Initialize a DynamoDB client
dynamodb = boto3.resource('dynamodb', region_name='<aws_region>')

# Specify the table name and column (attribute) name
table_name = '<table_name>'
column_name = '<key>'

# Initialize a DynamoDB table resource
table = dynamodb.Table(table_name)

# Perform a scan operation to get all items sorted in descending order by the specified column
response = table.scan()

def getMaxPid():
    # Check if any items were returned
    if 'Items' in response:
        items = response['Items']
        if len(items) > 0:
            max_value = max(item[column_name] for item in items)
            print(f"The maximum value for {column_name} is: {max_value}")
            return max_value + 1
        else:
            print(f"No items found in the table")
            return 1
    else:
        print(f"Error in scanning the table")
        return -1


def get_gh_token():
    secret_name = "<SECRET_NAME>"
    region_name = "<aws_region>"
    session = boto3.Session()
    client = session.client(service_name='secretsmanager', region_name=region_name)
    get_s = client.get_secret_value(SecretId = secret_name)

    secret = get_s['SecretString']
    ss = json.loads(secret)
    return ss['github_token']

GitHub Metric Microservice Deployment on AWS using ECR - Part 1 - Initial Setup and First Service

Manoj Kumar — Fri, 10 Nov 2023 05:24:17 +0000

Requirement

The requirement is to build 3 microservices which would get 3 metrics of a GitHub user, provided the username of the GitHub user.
The metrics were:
• Commits of the searched user
• Issues for the user
• Pull Requests made by the GitHub user.

Solution Outline

The solution involves creating 3 separate microservices to fetch the above GitHub user metrics. I chose python to implement all three microservices. The services were named Commit Service, Issue Service & PR Service.
Before I began, we need to create AWS Account and an IAM user to perform certain tasks in the CLI. I created a user with the below permission sets and policies.

Once that is created, I installed AWS CLI and used aws configure command to configure AWS CLI to use the credentials belonging to the IAM user which can be found on the “Security Credentials” tab in Fig 1. I also created a Personal Access Token from GitHub to fetch user metrics. Boto3 is another tool I used to interact with the DynamoDB and my python code.

The below section explains how each service was implemented.

Commit Service

This service fetches the user’s commits and the URL of the commits and saves them to the DynamoDB table – deviq_commits and returns a json response consisting of those commits. This service runs on port 5001.

import commitServiceUtil
from flask import Flask
import requests
import boto3

app = Flask(__name__)

#dynamodb
session = boto3.Session(region_name='<aws-region>')
dynamodb = session.resource('dynamodb')
table_name = '<table_name>'
table = dynamodb.Table(table_name)

@app.route('/')
def index():
    return "Hello! from Commit Service"

@app.route('/commits/<username>')
def get_user_commits(username):
    try:
        token = getCid.get_gh_token()
        maxId = getCid.getMaxCid()
        url = f"https://api.github.com/users/{username}/events"
        ret_commit = []

        response = requests.get(url, headers={"Authorization": f"token {token}"})
        if response.status_code == 200:
            events = response.json()
            commits = []
            for event in events:
                if event['type'] == 'PushEvent':
                    for commit in event['payload']['commits'][:15]:
                        print(commit)
                        commits.append(commit['url'])

                        table.put_item(Item={
                            'cid':maxId,
                            'gh_username': username,
                            'commit_url': commit['url']
                        })
                        maxId+=1
                        ret_commit.append(commit)

        return ret_commit
    except requests.exceptions.RequestException as exp:
        return f"Error Occured when fetching commits: {exp}"


if __name__ == '__main__':
    app.run(debug=True,host='0.0.0.0', port=5001)

the commitServiceUtil.py file contains helper functions such as getting the top cid (key of the table) for insertion and fetching the GitHub Personal Access Token which is stored in AWS Secrets Manager.

commitServiceUtil.py

import boto3
import json

# Initialize a DynamoDB client
dynamodb = boto3.resource('dynamodb', region_name='<aws-region-name>')

# Specify the table name and column (attribute) name
table_name = '<table_name>'
column_name = '<key>'

# Initialize a DynamoDB table resource
table = dynamodb.Table(table_name)

# Perform a scan operation to get all items sorted in descending order by the specified column
response = table.scan()

def getMaxCid():
    # Check if any items were returned
    if 'Items' in response:
        items = response['Items']
        if len(items) > 0:
            max_value = max(item[column_name] for item in items)
            print(f"The maximum value for {column_name} is: {max_value}")
            return max_value + 1
        else:
            print(f"No items found in the table")
            return 1
    else:
        print(f"Error in scanning the table")
        return -1


def get_gh_token():
    secret_name = "<SECRET_NAME>"
    region_name = "<aws-region-name>"
    session = boto3.Session()
    client = session.client(service_name='secretsmanager', region_name=region_name)
    get_s = client.get_secret_value(SecretId = secret_name)

    secret = get_s['SecretString']
    ss = json.loads(secret)
    print(type(ss['github_token']))
    return ss['github_token']

If the table is empty, the getcid function returns 1 and if the table is not empty, the function returns the +1 of the maximum cid.
The get_gh_token function is to fetch the GitHub Personal Access token from AWS Secrets manager. The token was manually uploaded to the AWS Secrets Manager and since committing tokens in the code is not a good practice, this is one good measure. Also, this benefits other two microservices as they also will be using the same token to fetch metrics.

Dockerfile:

FROM python:3.9

COPY . .

RUN pip install --no-cache-dir -r requirements.txt

RUN apt-get update && \
    apt-get install -y awscli && \
    apt-get clean

RUN aws configure set aws_access_key_id <hidden> && \
    aws configure set aws_secret_access_key <hidden> && \
    aws configure set default.region ap-south-1

EXPOSE 5001

CMD ["python", "commitService.py"]

Deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: commitservice-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: commitservice
  template:
    metadata:
      labels:
        app: commitservice
    spec:
      containers:
      - name: commitservice
        image: <ecr_image_id:tag>
        ports:
        - containerPort: 5001

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: commitservice-service
spec:
  selector:
    app: commitservice
  ports:
    - protocol: TCP
      port: 5001
      targetPort: 5001
  type: LoadBalancer  # or NodePort, depending on your setup

In Part 2 we will explore how the rest of the services were written & deployed.

First Win - CloudKasthiram

Manoj Kumar — Tue, 17 Nov 2020 03:30:28 +0000

The first win as an ameture developer! The #cloudkasthiram is a competition organized by GDG Sri Lanka. The participants have to do selected labs in qwiklabs and complete the competitions. The winners will be getting a swag kit from GDG Sri Lanka!