Forem: Michellebuchiokonicha

I just published Part 1 of "The Machine Learning Engineering Series", where I break down the full lifecycle: how raw data becomes a trained model, gets containerised, and ships as a live API that other engineers can actually build on.

Michellebuchiokonicha — Tue, 26 May 2026 07:38:04 +0000

Michellebuchiokonicha

May 26

The Machine Learning Engineering Series

#ai #machinelearning #deeplearning #software

Comments

5 min read

The Machine Learning Engineering Series

Michellebuchiokonicha — Tue, 26 May 2026 07:33:01 +0000

Part 1: From Scratch to Systems

.
This machine learning series will be a real ride. It’s an interactive journey where I’ll be sharing and raising lots of questions while building real-world AI systems. My goal is to make this deeply engaging, drive home the right questions, and bridge the gap between AI theory and engineering reality.

Note: There will be follow-up videos complementing this series, so keep an eye out for video links as they are released!

Let’s start at the very beginning.

Differentiating the Field: Who Does What?

There is massive confusion in the tech industry right now regarding job titles. To understand what a Machine Learning Engineer (MLE) actually does, we have to look at how they fit alongside Data Scientists and AI Software Engineers.

1. The Machine Learning Engineer (MLE)
An MLE is a specialised software engineer responsible for researching, building, scaling, and deploying machine learning models. They sit firmly at the intersection of software engineering, data science, and infrastructure.

Their core responsibilities include:

Model Development: designing and training algorithms and deep neural networks.

Data Preparation: cleaning, organising, and transforming large datasets.

Production Deployment & MLOps: deploying models to the cloud and continuously monitoring their performance.

System Architecture: building the core application infrastructure so these models can be used reliably at scale.

2. The Data Scientist (DS)
A Data Scientist is primarily focused on research, statistics, and business analysis. While an MLE is focused on specialised software engineering and scalability, a Data Scientist spends their time exploring raw data, testing hypotheses, and building prototype models to prove a business concept works.

3. The AI Software Engineer
An AI Software Engineer focuses on integration and utility. They treat machine learning models as powerful building blocks. Their primary job is to build the application frontends, backends, and user experiences that consume AI capabilities (often via external APIs), rather than training the core models from scratch.

The Core Blueprint: What You Need to Know

Most successful ML Engineers start as traditional software engineers who specialise, or are builders who learn the entire technical stack from the ground up. If you want to excel as an MLE, these are the core requirements:

Programming Languages: Python is the non-negotiable standard of the discipline. However, depending on the organisation and performance needs, some MLEs are also highly skilled in Java or C++.

ML Frameworks: you must be comfortable navigating production-grade libraries like PyTorch, TensorFlow, Keras, and scikit-learn.
Mathematics & Statistics: This is the area where many engineers fall short. While you might not be quizzed on complex proofs in every interview, understanding linear algebra, calculus, and probability is vital for debugging model behaviour.

Infrastructure & Tooling: You need to be comfortable with cloud computing platforms (GCP, AWS, etc.). Version control (Git/GitHub) is essential for managing your work, and Docker is effectively mandatory for containerising applications.

Understanding the Data

Before a system can learn, we have to understand the raw material. Data is broadly classified into two categories:

Structured Data: highly organised data that easily fits into rows and columns (e.g., CSV files, SQL databases).

Unstructured Data: complex data that does not have a pre-defined structure (e.g., images, audio, and natural language text).

The mechanics, architectures, and training pipelines required for unstructured data are entirely different from those used for structured data.

The Machine Learning Engineering Pipeline

Here is the bird’s eye view of how a model evolves from raw data into a living system:

Data ➔ Clean & Transform ➔ Analyse ➔ Baseline vs. Custom Model ➔ Train & Test ➔ Containerise & Deploy ➔ Expose as API

1. Ingestion & Analysis

Gathering Data: highly dependent on your organisation’s size. Sometimes this is handled by Data Engineers, but an MLE must know how to pull and ingest data independently.

Cleaning and Organising: raw data is messy. Here, data is cleaned, structured, and transformed into features. We will dive incredibly deep into this layer later in this series.

Exploratory Data Analysis (EDA): an MLE must thoroughly analyse data distributions and shapes before moving to the modelling phase.

2. The Modelling Strategy (Baseline vs. Custom)
When designing a system, you never just train a model blindly. You start with a baseline model.
A baseline can be an existing state-of-the-art architecture or a previously developed model. The ultimate goal of training a custom model is to prove it can outperform your baseline. If your complex custom model performs worse than a simple baseline, it shouldn’t go into production.

3. Training, Testing, and the “Skew” Bug
Training is heavily dependent on the data type and requires a deep understanding of concepts like epochs, loss functions, and regularisation techniques such as dropout (which help prevent overfitting).

Once trained, we must rigorously evaluate the model. Beyond traditional software bugs, MLEs face unique production challenges like training-serving skew.

What is training-serving skew?
This occurs when the data a model encounters in production (real life) is significantly different from the data it was trained on. This causes performance to plummet and can lead to the model producing unreliable or low-quality predictions. An MLE’s job is to ensure the training data is adequate and representative to minimise this skew.

You must also master evaluation metrics to track success, including the confusion matrix (precision, recall, accuracy, F1-score) and mAP (mean average precision) for spatial / object-detection systems.

4. Conversion to APIs & Building Systems
A model sitting on your local machine is useless. An MLE transforms a trained model into a lightweight, usable form by converting it into an API (using frameworks like FastAPI).

This makes the model highly accessible, allowing other software engineers or IoT infrastructures to seamlessly integrate its AI capabilities into a broader system.

5. Production Testing (Ensuring Utility)
Our job isn’t done just because the system runs. It must be reliable, secure, and useful. A system builder must implement a strict testing hierarchy to protect against vulnerabilities and low-standard features:

Unit Testing: testing individual components and functions in isolation.

Integration Testing: ensuring the data pipeline, API, and model interact perfectly.

Smoke Testing: rapidly verifying that the core infrastructure doesn’t crash upon deployment.

Testing isn’t just the icing on the cake; it is the foundational requirement for safety, reliability, and engineering confidence.

6. The Core Paradigms: From Linear Regression to Clustering
As we dive deeper into modelling throughout this series, we will break down the core learning paradigms that power these systems. You can’t build advanced architectures without mastering the fundamentals:

Supervised Learning: training models on labelled data (where the system knows the correct answers during training). This includes foundational algorithms like linear regression for predicting continuous values, all the way up to complex neural networks.

Unsupervised Learning: giving the model raw, unlabelled data and letting it discover hidden structures on its own. A prime example of this is clustering, where the system automatically groups similar data points without human intervention.

What’s Next?

Throughout this series, we are going to cover all of these concepts in depth. We will write the code, sketch the architectures, look at how these systems impact modern workplaces, and discuss how to position these skills to land an elite MLE role.
Buckle up, this is going to be an incredible build.

Google I/O 2026: From Consumer to Builder

Michellebuchiokonicha — Wed, 20 May 2026 19:41:08 +0000

I often tune in to Google I/O. But this year felt different.

In the past, I watched as a consumer. I needed the updates, first to keep up, then to pass them on to my audience. I was a distributor of news, a passive relay between Google's stage and other people's feeds.

This year, my pen hit the paper differently.

I watched as a builder. I wasn't tracking announcements to repeat them; I was scanning each one for where I fit, where I could come in, and how. I took pages of hurried notes, not to summarize later, but to return to and explore. Somewhere between the keynote's opening and its close, a quiet shift happened: I stopped feeling like an observer of the future and started feeling like a participant in it.

Here's what I saw, and why it mattered to me.

The Foundations Got Heavier

The keynote leaned hard on raw capability, the infrastructure underneath everything else. When the talk turned to the computing powering this new generation of models, I didn't just hear "faster." I heard the heavy-lifting engine for the deep, custom architectures I've been turning over in my head. The horsepower is finally scaling to match the complexity of what's coming.

Then came the models. Seeing Gemini 3.5 Flash and Gemini Omni Flash stream across the screen made me lean forward. Gemini 3.5 Flash pairs frontier intelligence with the ability to actually act , Google says it outperforms the previous Pro generation on coding and agentic benchmarks while running several times faster. Gemini Omni, meanwhile, takes image, audio, video, and text as input and generates editable video grounded in real-world knowledge.

For anyone building fast, reactive systems, that combination, native multimodality plus low latency, is the holy grail. We are moving past models that guess at the next word toward models that understand and generate across every modality at once.

From Assistants to Autonomous Digital Lives

The real turn in the keynote was the move from conversational assistants to agentic workflows.

This is where Gemini Spark came in, Google's pitch for a 24/7 personal agent that manages your digital life in the background. The framing that stuck with me: you can close your laptop, and the work keeps going. It feels less like a tool you pick up and put down, and more like an environment that runs on your behalf.

To make that practical, it leans on the Model Context Protocol (MCP) to integrate with third-party tools. I scribbled a question in my margin: Is this just an advanced Siri?

My conclusion: no. This is a genuine paradigm shift. But — and I underlined this, as a builder, I still believe keeping a human in the loop is non-negotiable for high-stakes decisions. An agent that acts while you sleep is powerful. It's also exactly the kind of system where the cost of a silent mistake is highest.

Agentic Commerce Gets Real

The commerce announcements immediately tested that belief.

Google introduced Universal Cart, an intelligent shopping cart that follows you across Search, Gemini, YouTube, and Gmail, tracking price drops and restocks in the background. Underneath it sit two protocols: the Universal Commerce Protocol (UCP), a common language for agent-driven checkout, and the Agent Payments Protocol (AP2), which lets agents complete purchases on your behalf within guardrails you set, specific brands, specific products, a spending cap.

AP2 builds a tamper-proof, verifiable record linking shopper, merchant, and payment processor. As an engineer, that's the part I find genuinely interesting: it's the bridge between a model that thinks and a model that securely transacts.

But it's worth being clear-eyed. Reporters covering I/O framed Universal Cart as Google's bid to become the default middleman in online commerce, a market some analysts project in the trillions by the end of the decade. Some retailers are already reporting traffic shifts as shoppers move from search to agents. The engineering opportunity here is enormous: end-to-end transaction systems where the agent owns the whole procurement and verification lifecycle. The responsibility that comes with it is just as large.

Sensory Interfaces and Immersive Wearables

A few updates pointed at where the interface itself is going.

Neural Expressive, Gemini's new design language brings adaptive emotional pacing to voices, fluid animation, and haptic feedback, making real-time audio dialogue feel noticeably more lifelike. And the Intelligent Eyewear running on Android XR moves the assistant off the screen entirely and into your physical space, directions, texts, and photos, all from a pair of glasses.

On the creative side, Flow Music stood out: a sandbox for composing original, high-fidelity songs through prompt-based collaboration. The audio modality is becoming completely fluid.

Trust, Verification, and the Hard Engineering Track

What reassured me was that the keynote didn't only sell capability, it addressed guardrails.

The expansion of C2PA Content Credentials means you can verify whether a piece of content is an unaltered original or has been modified, a check that extends all the way down to validating research and academic material. In a year defined by generative everything, provenance isn't a footnote. It's infrastructure.

Tooling for developers got attention too, including the agent-first work on Google's Antigravity platform, a sign that "anyone can be a builder" wasn't just a tagline but a thing Google is actively engineering toward.

My Final Reflection

I closed my notebook with pages of short, hurried notes. But I walked away with something more useful than notes: a map.

For the first time, I wasn't just watching Google build the future. I was figuring out exactly where I'm going to build in it, which protocols to learn, which models to prototype against, which problems are now finally tractable.

The shift from consumer to contributor doesn't happen with a single announcement. It happens the moment you stop asking "what did they release?" and start asking "what can I make with this?"

This year, I started asking the second question.

Were you watching I/O 2026? I'd love to hear which announcement made you lean forward, drop it in the comments.

I have been struggling to write…

Michellebuchiokonicha — Sat, 28 Feb 2026 08:22:51 +0000

I have been writing technical articles since 2022. My passion for technology has been built on writing. I have loved it even before I realized I wanted to go into software development. I used to write poems that got published while in school, so writing technical articles when I started coding was not a big deal for me. I have always written.

In fact, my parents used to say I would be like Chimamanda Adichie. Because not only did I know how to coin words, I knew how to express them in speech, and I also loved to read, which helped me greatly in writing with ease. At some point, while in secondary school, I considered becoming a writer and studying Literature just so I could write and speak. It always gave me joy.

But recently, I find myself struggling to do one of the things I would normally term ‘easiest’ to do.

This struggle started creeping in gradually in 2024. When ChatGPT came on board.
I still continued writing, and I had my articles featured on Dev. to amongst the 7 most read articles.

Some of my articles climbed to 40,000 reads, numerous comments, and likes. And in 2025, my new article also got featured on hasnode but in the heat of AI and arguments on AI being able to do everything, I wrote only two articles, and I paused.

I was confused, I was in doubt, I had questions: “Will people need to read articles on asynchronous functions, supervised learning, JavaScript, Python, and more, if AI can simply provide all the information?”

Secondly, I was confused because I have been a web developer all this time. Mostly writing and building frontend applications, hence most of my articles covered the web and JavaScript. But with this new excitement, I didn’t just want to stick only to the web; I wanted to branch into machine learning. This also made me pause to reflect. What started as a short-term pause in February 2025 turned into a full year of not writing any technical articles. I feel like crying as I type this. It hurts me more than I ever thought it would.

I have many excuses. Another would definitely be that I want to start a YouTube channel and be active on social media instead of writing. But I have asked so many questions. Can I edit videos? Do I really want to keep talking and explaining technology concepts instead of simply penning them down as articles? And right now, it has dawned on me. I don’t know how consistent I can be with explaining concepts via speech as opposed to writing them down, which is easier and smoother for me.
I have been stalling, waiting, postponing this, telling myself I am too busy, telling myself AI can explain everything, telling myself YouTube is better now, as people don’t read anymore. But it feels like something is missing. Like, I am not doing one of the things I am meant to be doing, and it hurts, it breaks my heart so much.

This is my first piece in one year, and truly, this morning, 7:50 am, 22nd February, 2026, thinking deeply about what could have been if I had not stopped writing, I decided to just write.

I have no will for any aspire to perspire, but I guess we will all figure it out. The questions, the doubts. If you are like me, then write.

JUST WRITE.

Infrastructure as Code: Getting Started with Terraform for Cloud Deployment

Michellebuchiokonicha — Fri, 14 Feb 2025 06:55:29 +0000

Introduction to Cloud Services

Cloud computing has changed the way we deploy and manage applications. It offers various service models to meet different needs.

Infrastructure as a Service (IaaS): Provides on-demand access to computing resources like servers, storage, and networking.

Platform as a Service (PaaS): Offers hardware and software resources for cloud application development, eliminating the need to manage the underlying infrastructure.

Software as a Service (SaaS): Delivers full application stacks as a cloud service, including maintenance and management of both infrastructure and software.

Benefits of Cloud Deployment

Using the cloud for computing and deployment comes with several advantages:

Faster Time to Market: Servers can be deployed with just a few clicks, significantly accelerating the development process.

Cost Savings: By choosing the right set of services, infrastructure costs can be drastically reduced.

Data Loss Prevention: Data is stored across multiple data centers in different availability zones, ensuring redundancy and reliability.

Challenges of Cloud Deployment

While the cloud offers many benefits, there are some challenges to consider:

Vendor Lock-In: Switching cloud providers can be costly and complex, often leaving customers tied to their original vendor.

Lack of Expertise: Finding skilled cloud professionals can be difficult, as the demand for cloud expertise often outstrips supply.

Major Cloud Providers

The three largest cloud providers dominate the market:

AWS (Amazon Web Services): Holds the largest market share in the US and offers a wide range of services.

Azure: Known for its seamless integration with Microsoft products and a strong presence in Germany.

GCP (Google Cloud Platform): This has seen rapid growth over the past three years and integrates well with Google products like Google Analytics.

Deploying to the Cloud

Deploying applications in the cloud involves using a variety of services. While you can manage these services manually through the cloud console, this approach has limitations:

Complexity: Dropping or modifying services can be tedious, requiring you to navigate through multiple menus.

Lack of Replicability: Manual configurations are hard to replicate, making it difficult for others to understand or reproduce your setup.

This is where Infrastructure as Code (IaC) comes in. IaC allows you to define and manage your infrastructure using code, making it replicable, understandable, and easier to maintain.

What is Infrastructure as Code (IaC)?

Infrastructure as Code (IaC) is the practice of managing and provisioning infrastructure through code instead of manual processes. This approach has become increasingly popular, with IaC skills now appearing in job postings for roles like Data Analysts.
Popular IaC tools include:

Terraform
AWS CloudFormation
Azure Resource Manager
Google Cloud Deployment Manager

Among these, Terraform stands out due to its ability to work across multiple cloud providers, making it a versatile choice for organizations that may migrate between clouds.

Why Terraform?

Terraform is an open-source IaC tool that allows you to build, change, and version infrastructure safely and efficiently. Here’s why it’s a great choice:

Multi-Cloud Support: Terraform supports multiple cloud providers, including AWS, Azure, and GCP.

Rollback Capabilities: Infrastructure changes can be easily rolled back, minimizing downtime and risks.
Ease of Use: Terraform uses a declarative language, making it easy to learn and use. You only need a few commands to get started.

Cost Efficiency: You can stop and resume your infrastructure as needed, saving on cloud credits and usage.

Setting Up Terraform

To get started with Terraform, you’ll need:

A Cloud Provider Account: For this guide, we’ll use AWS.

Terraform Installed: Download and install Terraform from the website: https://www.terraform.io/

Example: Setting Up an S3 Bucket in AWS

Here’s an example of how to set up an S3 bucket using Terraform:

provider "aws" {
  region = "us-east-1"
}

resource "aws_s3_bucket" "example_bucket" {
  bucket = "my-unique-bucket-name"
  acl    = "private"

  versioning {
    enabled = true
  }

  tags = {
    Name        = "Example Bucket"
    Environment = "Dev"
  }
}

This code defines an S3 bucket with versioning enabled and tags for identification. You can customize it further based on your needs.
Conclusion

you need to further set up:

provider
aws_s3_bucket
aw3_s3_bucket_public_access_block
aws_s3_bucket_ownership_controls
aws_s3_bucket_acl
aws_s3_object
aws_s3_bucket_website_configuration

Here is a simple setup

provider "aws" {
    region = "us-east-1"
}

resource "aws_s3_bucket" "s3" {
    bucket = "your-bucket"
}

resource "aws_s3_bucket_public_access_block" "s3-public-block" {
    bucket = aws_s3_bucket.s3.id

    block_public_acls       = false
    block_public_policy     = false
    ignore_public_acls      = true
    restrict_public_buckets = false
}

resource "aws_s3_bucket_ownership_controls" "s3-ownership" {
    bucket = aws_s3_bucket.s3.id

    rule {
        object_ownership = "BucketOwnerPreferred"
    }
}

resource "aws_s3_bucket_acl" "s3-acl" {
    bucket = aws_s3_bucket.s3.id
    acl    = "public-read"
}

resource "aws_s3_object" "s3-object" {
    bucket       = aws_s3_bucket.s3.id
    key          = "index.html"
    source       = "index.html"
    content_type = "text/html"
    acl          = "public-read"
}

resource "aws_s3_bucket_website_configuration" "s3-website-configuration" {
    bucket = aws_s3_bucket.s3.id

    index_document {
        suffix = "index.html"
    }
}

with this simple setup, you can deploy your first web application.
I have further included a sample index.html file

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Welcome to My Website</title>
    <style>
        body {
            font-family: Arial, sans-serif;
            margin: 0;
            padding: 0;
            background-color: #f4f4f4;
            color: #333;
            text-align: center;
        }
        header {
            background-color: #007bff;
            color: white;
            padding: 20px 0;
        }
        h1 {
            margin: 0;
        }
        main {
            padding: 20px;
        }
        footer {
            background-color: #333;
            color: white;
            padding: 10px 0;
            position: fixed;
            bottom: 0;
            width: 100%;
        }
    </style>
</head>
<body>
    <header>
        <h1>Welcome to My Website</h1>
    </header>
    <main>
        <p>Hello, world! This is a simple webpage hosted on AWS S3.</p>
        <p>Enjoy browsing!</p>
    </main>
    <footer>
        <p>&copy; 2025 My Website</p>
    </footer>
</body>
</html>

To install terraform, use:

terraform -v

then open your IDE and run terraform init

To format and validate your terraform code, run terraform fmt and
terraform validate

Next, you would need to plan and make changes with
terraform plan

To apply the changes made, run terraform apply -auto-approve

Lastly, in order to save bandwidth and when you are not using the infrastructure at a certain time, you can destroy it with

terraform destroy -auto-approve

You can further rerun/rebuild by running
terraform apply

The process is quite seamless and easy to use and understand. it saves you time, provides flexibility, and also allows other developers to see what you have done.

Infrastructure as Code, particularly with tools like Terraform, simplifies cloud deployment and management. It ensures that your infrastructure is replicable, understandable, and easy to maintain. Whether you’re working with AWS, Azure, or GCP, Terraform provides a consistent and efficient way to manage your cloud resources.
Start your IaC journey today and experience the benefits of deploying infrastructure as code!

Python OOP Made Easy: Master Object-Oriented Programming with Ease

Michellebuchiokonicha — Tue, 04 Feb 2025 07:38:22 +0000

Object-Oriented Programming (OOP) is a programming paradigm that uses objects and classes to structure software. Unlike procedural programming, which focuses on functions and procedures, OOP emphasizes using objects that encapsulate data and behavior. Python, a versatile language, fully supports OOP, making it a powerful tool for building modular, reusable, and maintainable code.

In this article, we’ll explore the basics of OOP in Python, including classes, objects, attributes, methods, inheritance, polymorphism, encapsulation, and abstraction. We’ll also dive into functional programming concepts and how they complement OOP in Python.

Python OOP Basics

What is OOP?

OOP is a programming paradigm that organizes code into classes and objects. A class is a blueprint for creating objects, while an object is an instance of a class. OOP promotes concepts like modularity, reusability, and abstraction, making it easier to manage complex systems.

Pros of Object-Oriented Programming

Modularity: Code is organized into reusable components.

Abstraction: Hide complex implementation details.

Maintenance: Easier to update and debug.

Reusability: Classes can be reused across projects.

Cons of Object-Oriented Programming

Complexity: This can be overkill for simple programs.

Performance: Slightly slower due to overhead.

Overuse: Not every problem requires an OOP solution.

Classes and Objects in Python

What are Classes and Objects?

A class is a blueprint for creating objects. It defines the attributes (data) and methods (functions) that the objects will have. An object is an instance of a class.

Example:

class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age

    def say_hi(self):
        print(f"Hi, my name is {self.name} and I’m {self.age} years old.")

# Creating objects
person1 = Person("Alice", 25)
person2 = Person("Bob", 35)

person1.say_hi()  # Output: Hi, my name is Alice and I’m 25 years old.
person2.say_hi()  # Output: Hi, my name is Bob and I’m 35 years old.

What Can Go Wrong?

Python is flexible, but this can lead to mistakes. For example:

person3 = Person(name="Charlie", age=person1)
person3.say_hi()  # Output: Hi, my name is Charlie and I’m <__main__.Person object at 0x109f301c0> years old.

Here, age is mistakenly assigned an object instead of a number. Always validate inputs!

Attributes and Methods

Attributes

Attributes define the state of an object. They store data associated with the object.

Methods
Methods define the behavior of an object. They are functions attached to a class.

class Car:
    wheels = 4  # Class attribute

    def __init__(self, make, model, year=None):
        self.make = make  # Instance attribute
        self.model = model
        self.year = year

    def get_make_model(self):  # Instance method
        return f"{self.make} {self.model}"

car1 = Car("Toyota", "Camry", 2021)
print(car1.get_make_model())  # Output: Toyota Camry

Inheritance in Python

Inheritance allows a class (child) to inherit attributes and methods from another class (parent). This promotes code reuse and hierarchy.

Example:

class Animal:
    def __init__(self, name):
        self.name = name

    def speak(self):
        print("This animal speaks.")

class Dog(Animal):
    def speak(self):
        print("Woof!")

dog = Dog("Pelle")
dog.speak()  # Output: Woof!

Polymorphism, Encapsulation, and Abstraction

Polymorphism

Polymorphism allows objects of different classes to be treated as objects of a common superclass. For example:

def make_animal_speak(animal):
    animal.speak()

dog = Dog("Fido")
cat = Cat("Whiskers")

make_animal_speak(dog)  # Output: Woof!
make_animal_speak(cat)  # Output: Meow!

Encapsulation

Encapsulation hides the internal state of an object and restricts direct access. Use private attributes (__) to enforce this:

class BankAccount:
    def __init__(self, account_number, balance):
        self.__account_number = account_number
        self.__balance = balance

    def deposit(self, amount):
        self.__balance += amount

    def get_balance(self):
        return self.__balance

Abstraction

Abstraction focuses on hiding non-essential details and exposing only the necessary features. Use abstract classes to define interfaces:

from abc import ABC, abstractmethod

class Shape(ABC):
    @abstractmethod
    def area(self):
        pass

class Rectangle(Shape):
    def __init__(self, length, width):
        self.length = length
        self.width = width

    def area(self):
        return self.length * self.width

Functional Programming in Python

Python also supports functional programming (FP), which emphasizes immutability and pure functions. Key concepts include:

Pure Functions: No side effects, deterministic.

Higher-order functions: Functions that take other functions as arguments.

Immutability: Data cannot be changed after creation.

Example:

# Higher-order functions
numbers = [1, 2, 3, 4, 5]
squared_numbers = list(map(lambda x: x**2, numbers))  # Output: [1, 4, 9, 16, 25]

Conclusion

OOP and FP are powerful paradigms that can be used together to write clean, modular, and maintainable code in Python. By understanding classes, objects, inheritance, polymorphism, encapsulation, and abstraction, you can build robust applications. Additionally, functional programming concepts like immutability and higher-order functions can complement OOP to make your code even more expressive.

Whether you’re building a small script or a large-scale application, mastering these concepts will help you write better Python code.

Are Web APIs the same as REST APIs? How to Improve your Web Apps with Web APIs

Michellebuchiokonicha — Tue, 14 Jan 2025 08:36:11 +0000

To answer the question simply: No, Web APIs are not the same as REST APIs.

Web APIs refer to interfaces that allow applications to interact with the web or devices. This term is broad and can refer to Browser APIs (like WebRTC or Geolocation) or Server APIs (like REST or GraphQL).

On the other hand, REST APIs are a specific type of Server API that adhere to REST principles, focusing on statelessness, resource-based communication, and using HTTP methods like GET, POST, and DELETE.

The Evolution of the Web

Originally, the web was designed as a static site for providing information and connecting individuals to relevant resources, regardless of location. However, it has evolved to become interactive and progressive, with certain aspects also turning decentralized.
This article focuses on the web being progressive and more user-friendly.

The Role of Web APIs in Modern Development

Web APIs serve as the backbone of modern web development. With APIs like WebRTC, Web Bluetooth, and Service Workers, developers can create highly interactive, native-like web applications without requiring external plugins. These capabilities make Progressive Web Apps (PWAs) possible, bridging the gap between web and native apps.
Check out my extensive article on progressive web apps.
https://dev.to/michellebuchiokonicha/how-to-get-started-with-progressive-web-apps-3f44

Some Benefits of Progressive Web Apps

Works on any browser (cross-platform)
Responsive to any device
Functions without network connectivity
Performs like a mobile app
Highly secure
Discoverable via search engines
Boosts app engagement
No installation required
Can be shared through a link
Optionally available on app stores (e.g., Google Play, Apple Store)

Web APIs and Project Fugu

Web APIs here, referring to browser APIs, act as connectors, improvers, and builders for the browser's capabilities. These APIs are inherent to the web and are not third-party APIs purchased from libraries or sites. Instead, they are broadly available and designed to improve the web and enhance products created by developers, reducing boilerplate code and simplifying web interactions.
The significance of web APIs is underscored by the fact that over 70% of activities conducted online occur through the web, highlighting the need for improvements and scalability.

In summary, web APIs are crucial for the success of Progressive Web Apps and the future of web development. They act as enablers of advanced functionalities.

Project Fugu

Project Fugu derives its name from the Japanese word for pufferfish, "fugu."
Initially, I was curious about why this project was named after a fish until I learned more about it. The fugu is a dangerous fish that can be delicious when prepared correctly; however, if not handled properly, it can be harmful. This metaphor aptly reflects the project's goal of enhancing the web platform's capabilities to match those of native applications.
The correlation here is that the Project Fugu APIs aim to enrich the web (the ‘delicious’ aspect), and if they are not built correctly (the ‘preparation’), they can introduce significant security vulnerabilities. Imagine creating an API accessible to billions without adequate security measures; it would expose individuals to various security risks.

You can find more about Project Fugu and its APIs at https://developer.chrome.com/docs/capabilities/fugu-showcase. Here, you will discover interesting and exciting projects built with web APIs, as well as a comprehensive list of available APIs. Many native app features that were previously unavailable on the web are now accessible, thanks to web APIs.

Why Web APIs?

Web APIs are crucial for several reasons, and their importance should not be overlooked. They empower the browser, transforming the web from a simple content viewer into a powerful application platform. With APIs like WebRTC, WebSockets, Service Workers, Web Bluetooth, and Web XR, you can now accomplish tasks on the web that were once limited to native applications.

Why Should You Start Using Web APIs?

Web APIs offer numerous benefits, including:

Access to Native Device Features: They allow you to access device hardware such as cameras, microphones, sensors, and Bluetooth devices. This means you can make video calls, control robots, and experience augmented reality directly in your browser, without needing to download any apps.

Offline and Background Functionality: Service Workers enable features like push notifications and caching, enhancing the user experience even when offline.

Simplified App Development for Cross-Platforms: Web APIs facilitate easier development across different platforms.

How Web APIs Improve the Web

Web APIs bring the capabilities of native apps to the web, granting access to powerful features without requiring installations or downloads. They also help future-proof the web; as browsers evolve, more APIs will be developed for seamless integration. Additionally, web APIs are accessible across platforms.

Is it Easy to Get Started with Web APIs?

Yes, it is straightforward. Most web APIs are already integrated into browsers, and supported by robust documentation. Since many are native to the browser, there's no need for third-party libraries or installations. This makes getting started seamless, consistent, and accessible across platforms.

Some Web APIs

Web APIs are the cornerstone of the Progressive Web App (PWA) revolution. They form the backbone of improved web functionality and represent the future of web development. Numerous Web APIs are available, and you can find a comprehensive list in the MDN documentation or on web.dev. Below are about 10 Web APIs to help you understand how they work. You can find many more in the MDN docs,web.dev, or via the project fugu documentation.

WebRTC API: Enables real-time audio, video, and data communication directly between browsers without the need for external plugins. It is commonly used for video conferencing and peer-to-peer file sharing.

Web Bluetooth API: Allows websites to communicate with Bluetooth-enabled devices, such as fitness trackers, medical devices, and IoT gadgets.

WebXR API: Provides support for immersive technologies like Virtual Reality (VR) and Augmented Reality (AR), allowing developers to create engaging 3D experiences directly in the browser.

WebSocket API: Facilitates full-duplex communication between the client and server, making it ideal for real-time applications like chat systems, live streaming, and collaborative editing.

Push Notification API: Enables websites to send timely and relevant notifications to users even when the website is not open in the browser, enhancing user engagement.

Web Share API: Provides a standardized way for web apps to share content using native sharing mechanisms, making it easier to share links, files, or text with other applications.

Fetch API: Replaces the older XMLHttpRequest and simplifies asynchronous data fetching.

Web Payment API: Simplifies the online payment process by offering a consistent interface for users to make payments using various methods, such as credit cards or digital wallets.

Geolocation API: Provides access to the geographical location of the user, enabling location-aware functionality like maps, location-based recommendations, or geofencing.

File System Access API: Enables web applications to read from and write to the user’s local file system, creating a native-like experience for file manipulation.

WebRTC Demo

WebRTC enables real-time communication, including audio, video, and data-sharing capabilities. For a simple WebRTC demo, you will need:

index.html

app.js

server.js

WebSocket or HTTP REST API
Inbuilt WebRTC API
Local and remote video elements
Buttons to manage peer connections
WebRTC configurations

How It Works

Access User's Camera and Microphone: Create a function to access the user's camera and microphone, displaying the local stream on the local video element.

Peer Connection: Implement a function to handle media exchange.

Signaling: Simulate sending and receiving data through the signaling server.

Remote Stream Handling: Display the remote track in the remote video element.
ICE Candidates: Exchange these candidates with peers to establish a connection.

Web APIs are the cornerstone of modern web development, enabling developers to build powerful, scalable, and engaging applications. Whether you're exploring WebRTC for real-time communication or leveraging the File System Access API for native-like experiences, the possibilities are vast. Start experimenting today and contribute to the web's progression.

How to Automate Excel Files from APIs with Python and Openpyxl.

Michellebuchiokonicha — Fri, 16 Aug 2024 06:30:41 +0000

Automation in programming is the use of programming languages like Python, VBA, and other technology tools to create programs, scripts, or various tools that perform automatic tasks with no manual intervention.
The sole purpose of automating tasks is to avoid manual input and to ensure various systems run by themselves.

For this article, I will provide a step-by-step guide on how I automated an Excel file, and different sheets on a MacBook, without visual basic for applications, using Python in this case.

First of all, to get started, you don't need to be a Python dev as I will paste a code snippet here.

Tools Required

VScode of course
Python installed/updated
A virtual environment to run any new installation or updates for your Python code.
The virtual environment is the .venv. You will see it in your vscode.
Install openpyxyl
Install any other necessary dependency.
Get started.

The Different Aspects we will be considering:

Creating a new Excel file with python
Updating an existing Excel file with python Updating a specific Excel file sheet only with Python
Using APIs to update Excel files and Excel file sheets.
Creating a button that allows users to update on click.
Adding dynamic dates and time in your code
An alternative to the Excel button is cron or Windows shell
Instead of VBA, what else is possible?
Issues faced with writing VBA in a MacBook
Issues I faced while creating the button
Why I opted for cron
Creating this for both Windows and Mac users
Other tools that can be used for the automation of Excel
Power query from web feature
Power automate
Visual Basic in Excel

Creating a new Excel file with python

Creating an Excel sheet in Python with openpyxl is easy.
All you need to do is install openpyxl, pandas, and requests if you are getting data from an API.
Go to the openpyxl documentation to learn how to import it into your application and the packages you want to use.

import pandas
import requests
from openpyxl import Workbook, load_workbook
from openpyxl.utils import get_column_letter

Next up,
you create a new workbook
Set it as the active workbook
Add your title and header and populate the data
Save the new workbook with your preferred Excel name and tada!
you have created your first Excel file.

# create a new workbook
wb = Workbook()
ws = wb.active
ws.title = "Data"

ws.append(['Tim', 'Is', 'Great', '!'])
ws.append(['Sam', 'Is', 'Great', '!'])
ws.append(['John', 'Is', 'Great', '!'])
ws.append(['Mimi', 'Is', 'Great', '!'])
wb.save('mimi.xlsx')

Creating a new sheet in an Excel file.

Creating a specific sheet in your Excel file is a similar process. however, you need to specify the sheet to be created with a sheetname.

# create sheet
wb.create_sheet('Test')
print(wb.sheetnames)

Modifying an Excel sheet.

To modify an Excel sheet and not the full file,

Load the workbook you want to modify
They specify the particular sheet to modify using its name or index. It is safer to use the index in case the name eventually changes.
In the code snippet below, I used the Sheet label

# wb = load_workbook('mimi.xlsx')

# modify sheet
ws = wb.active
ws['A1'].value = "Test"
print(ws['A1'].value)
wb.save('mimi.xlsx')

Accessing multiple cells

To access multiple cells,
Load the workbook
Make it the active workbook
loop through its rows and columns

# Accessing multiple cells
 wb = load_workbook('mimi.xlsx')
 ws = wb.active

 for row in range(1, 11):
     for col in range(1, 5):
         char = get_column_letter(col)
         ws[char + str(row)] = char + str(row)
         print(ws[char + str(row)].value)

 wb.save('mimi.xlsx')

Merging Excel cells

To merge different cells in Excel using Python,
Load the workbook
Indicate the active workbook
indicate the cells you want to merge

# Merging excel cells
wb = load_workbook('mimi.xlsx')
ws = wb.active

ws.merge_cells("A1:D2")
wb.save("mimi.xlsx")

Unmerging cells

To unmerge different cells in Excel using python,
Load the workbook
Indicate the active workbook
indicate the cells you want to unmerge

# merging excel cells
wb = load_workbook('mimi.xlsx')
ws = wb.active

ws.unmerge_cells("A1:D1")
wb.save("mimi.xlsx")

Inserting new excel cells

To insert new cells

Load the workbook
Indicate the active workbook
use the insert_rows and insert_columns to insert new rows or new columns based on preference.

# inserting cells
wb = load_workbook('mimi.xlsx')
ws = wb. is active

ws.insert_rows(7)
ws.insert_rows(7)

ws.move_range("C1:D11", rows=2, cols=2)
wb.save("mimi.xlsx")

Updating an existing Excel file with internal Data
Add your arrays and objects and take in the information needed

from openpyxl import Workbook, load_workbook
from openpyxl.utils import get_column_letter
from openpyxl.styles import Font

data = {
    "Pam" : {
        "math":65,
        "science": 78,
        "english": 98,
        "gym": 89
    },
    "Mimi" : {
        "math":55,
        "science": 72,
        "english": 88,
        "gym": 77
    },
    "Sid" : {
        "math":100,
        "science": 66,
        "english": 93,
        "gym": 74
    },
    "Love" : {
        "math":77,
        "science": 83,
        "english": 59,
        "gym": 91
    },
}

wb = Workbook()
ws = wb.active
ws.title = "Mock"
headings = ['Name'] + list(data['Joe'].keys())
ws.append(headings)

for a person in data:
    grades = list(data[person].values())
    ws.append([person] + grades)

for col in range(2, len(data['Pam']) + 2):
    char = get_column_letter(col)
    ws[char + '7'] = f"=SUM({char + '2'}:{char + '6'})/{len(data)}"

for col in range(1, 6):
    ws[get_column_letter(col) + '1'].font = Font(bold=True, color="0099CCFF")


wb.save("NewMock.xlsx")

Updating an existing Excel file with Python and APIs

To update an Excel file using Python and APIs, you need to call the APIs into your file using a Get request.
Set the active Excel file as described above and then you run your script.
Here is an example of this:

from openpyxl import Workbook, load_workbook
import requests
from datetime import datetime, timedelta

import schedule
import time

api_url = "https://yourapi"
excel_file = "yourfilename.xlsx"

def fetch_energy_data(offset=0):
    response = requests.get(api_url + f"&offset={offset}")
    data = response.json()

    if response.status_code == 200:
        data = response.json()
        return data["results"], data["total_count"] 
    else:
        print(f"Error fetching data: {response.status_code}")
        return [], 0

def update_excel_data(data):
    try:
        wb = load_workbook(excel_file)
        ws = wb.worksheets[0]  

        for row in range(5, ws.max_row + 1):  
            for col in range(1, 9):  
                ws.cell(row=row, column=col).value = None  

                now = datetime.now()
                current_year = now.year
                current_month = now.month

        start_date = datetime(current_year,current_month, 1) 
        end_date = datetime(current_year, current_month, 24) 

        filtered_data = [
            result
            for result in data
            if start_date <= datetime.fromisoformat(result["datetime"]).replace(tzinfo=None) <= end_date]


        for i, result in enumerate(filtered_data):  
            row = i + 5  
            ws[f"A{row}"] = result["datetime"]
            ws[f"B{row}"] = result["yourinfo"]
            ws[f"C{row}"] = result["yourinfo"]
            ws[f"D{row}"] = result["yourinfo"]
            ws[f"E{row}"] = result["yourinfo"]
            ws[f"F{row}"] = result["yourinfo"]  
            ws[f"G{row}"] = result["yourinfo"]
            ws[f"H{row}"] = result["yourinfo"]

        for row in range(5, ws.max_row + 1):
            ws[f"I{row}"] = ws[f"I{row}"].value  
            ws[f"J{row}"] = ws[f"J{row}"].value  
            ws[f"K{row}"] = ws[f"K{row}"].value  
            ws[f"L{row}"] = ws[f"L{row}"].value  

        wb.save(excel_file)
        print(f"Excel file updated: {excel_file}")
    except FileNotFoundError:
        print(f"Excel file not found: {excel_file}")
    except KeyError:
        print(f"Sheet 'Forecast PV' not found in the Excel file.")
    schedule.every().hour.do(update_excel_data)

    while True:
             schedule.run_pending()

if __name__ == "__main__":
    all_data = []
    offset = 0
    total_count = 0
    while True:
        data, total_count = fetch_energy_data(offset)  
        if not data:
            break
        all_data.extend(data)
        offset += 100  
        if offset >= total_count:  
            break


    update_excel_data(all_data)


To update a particular sheet, use the method mentioned above. best practices are done with the excel sheets index number from 0 till n-1.
as sheet names can change but sheet positions can not change.

 wb = load_workbook(excel_file)
        ws = wb.worksheets[0]

Creating a button that allows users to update on click. To achieve a button to automatically run your Python script, you need to create a button in your Excel file and write a program using the inbuilt programming language, Visual Basic for applications. Next, you write a program similar to this. An example of a VBA script is below.

Sub RunPythonScript()
    Dim shell As Object
    Dim pythonExe As String
    Dim scriptPath As String
    Dim command As String

     Path to your Python executable
    pythonExe = "C:\Path\To\Python\python.exe"

     Path to your Python script
    scriptPath = "C:\Path\To\Your\Script\script.py"

     Command to run the Python script
    command = pythonExe & " " & scriptPath

     Create a Shell object and run the command
    Set shell = CreateObject("WScript.Shell")
    shell.Run command, 1, True

     Clean up
    Set shell = Nothing
End Sub

the issue with this is some functions do not run in non-windows applications seeing that Excel and VBA are built and managed by Microsoft, there are inbuilt Windows functions for this that can only work on Windows.

However, if you are not writing a very complicated program, it will run properly.

Adding dynamic dates and time in your code

To achieve dynamic dates and times, you can use the date.now function built into Python.

now = datetime.now()
 current_year = now.year
current_month = now.month

An alternative to the Excel button is cron or Windows shell

For MacBook users, an alternative to the VBA and button feature, you can use a corn for MacBook and a Windows shell for Windows. to automate your task.

You can also make use of Google Clouds's scheduler. that allows you to automate tasks.

Instead of VBA, what else is possible?

Instead of VBA, direct Python codes can suffice. you can also use the script and run it as required.

Issues faced while writing VBA in a MacBook

The major issue lies in the fact that VBA is a Windows language and hence, has limited functions in a non-windows device.

Issues I faced while creating the button

The same issues are related to the VBA code.

Why I opted for cron
I opted for corn because it is available and easy to use to achieve the goals.
Other tools that can be used for the automation of Excel

Other tools include:

Power query from web feature
Power automate
Visual Basic in Excel

Follow me on Twitter Handle: https://twitter.com/mchelleOkonicha

Follow me on LinkedIn Handle: https://www.linkedin.com/in/buchi-michelle-okonicha-0a3b2b194/
Follow me on Instagram: https://www.instagram.com/michelle_okonicha/

Tips for Fixing Bugs and Resolving Code Issues.

Michellebuchiokonicha — Tue, 30 Jul 2024 01:53:52 +0000

Hey there!

These tips I have listed have worked for me and helped me resolve issues in my code. I also learn a lot while on it.

Understand the issue.
Write down the issue.
Go through the code given twice or more times.
Understand the code and its logic.
Analyze the issue based on the current code.
Device two or more ways to fix it on paper.
Write pseudocode.
Implement pseudocode in your file.
Use Debugging tools
Use Browser tools for developers like inspect, developer tools, console, etc.
Include error handling at every point
If still difficult, discuss this problem with another developer
regardless of the language used.
Sleep on the issue.
Think about the issue deeply.
Go back and fix the issue.

Sidenotes:

Research as much as you can. Someone might have fixed this issue previously. So search the internet to help you work seamlessly.

How to Improve Your Development Workflow with Gemini Code Assist

Michellebuchiokonicha — Sat, 20 Jul 2024 08:27:05 +0000

Have you ever dreamt of building and deploying applications three times faster? Look no further than Gemini Code Assist, an AI assistant from Google that empowers developers and cloud specialists throughout the development lifecycle.

What is Gemini Code Assist?

Gemini Code Assist is an AI-powered tool that streamlines the development process for software developers and cloud engineers. It integrates seamlessly with the Google Cloud platform and popular IDEs like Visual Studio Code.

Gemini Code Assist boasts a variety of features to enhance your coding experience, including:

Code Completion: It Generates code snippets and functions tailored
to your specific needs.
Code Explanation: You can Gain a deeper understanding of existing
code with clear explanations.
Code Refactoring: It Improves code readability, maintainability,
and performance with suggested optimizations.
Test Case Generation: It Automatically creates unit tests to
ensure code quality.

Get Started with Gemini Code Assist

A codelab session is available to guide you through the intricacies of using Gemini Code Assist. Here's a glimpse into what you can achieve:

https://codelabs.developers.google.com/codelabs/cloud-code-assist-sdlc#0

API Design: Describe your application's requirements, and Gemini - Code Assist will generate an appropriate architecture, complete with API routes, ports, and data types. You can even test the generated API to verify its functionality.

Full-Stack Development: Craft full-fledged applications in any programming language using natural language prompts. Gemini Code Assist can replace existing code, create new files, and cite sources for the generated code.

Automated Testing: Generate unit tests tailored to your specific needs and leverage Gemini Code Assist to help refactor code for passing tests.

Streamlined Deployment: Gemini Code Assist integrates with Google Cloud Platform for effortless application deployment, eliminating the need for manual configuration.

Benefits of Using Gemini Code Assist

Increased Development Speed: Build applications significantly
faster with code generation and completion features.
Improved Code Quality: Write cleaner, more maintainable code with
refactoring suggestions and automated testing.
Reduced Costs: Save time and resources by streamlining
development tasks.
Enhanced Developer Productivity: Focus on core development
activities while Gemini Code Assist handles repetitive tasks.

Using Gemini Code Assist

There are two primary ways to leverage Gemini Code Assist:

Google Cloud Platform: Access Gemini Code Assist directly within
the Google Cloud console for a seamless cloud-based development
experience.
Local IDE Integration: Install the Gemini Code Assist extension
for your IDE (like Visual Studio Code) to enjoy its features
within your familiar development environment.

Unique Features of Gemini Code Assist

Replace feature: Allows you to replace your existing code without copying and pasting
The update feature allows you to update and rewrite existing code
The citation feature: it cites code sources for further use and also allows the developer to know more about the code.
Explain feature: Explains highlighted code. You need to highlight it and it explains that aspect of code.
Specific use case: It creates specific use cases unique to the written code.

How To Use Gemin Code Assist In your IDE like VSCode

It is very easy and straightforward.

Go to your Visual Studio or the IDE you use
Go to Extensions
Search for Gemini Code Assist + Google Cloud Code.

Install and enable it
Click on the icon by the side

Log in or Sign up with your Chrome browser. If Chrome is not your default browser, copy it and paste it into your Chrome browser.
After which you can get started.

All the features and capabilities Listed above are fully applicable here.

Conclusion

Gemini Code Assist is a powerful and versatile tool that empowers developers and cloud specialists of all levels. It simplifies coding tasks, improves code quality, and accelerates the development process, allowing you to focus on creating innovative applications.

Follow me on Twitter Handle: https://twitter.com/mchelleOkonicha

Follow me on LinkedIn Handle: https://www.linkedin.com/in/buchi-michelle-okonicha-0a3b2b194/
Follow me on Instagram: https://www.instagram.com/michelle_okonicha/

Data Science and the Cloud

Michellebuchiokonicha — Tue, 18 Jun 2024 16:15:47 +0000

There are many good reasons to move data science projects to the cloud as there are many advantages like highly clustered processes on many machines giving us enormous computing power.

Also, data science is nothing without data. In a distributed cloud system, our data can be stored and processed reliably even if they are enormous datasets that are also changing constantly. Drawbacks include:
Data science projects in the cloud tend to be more complex especially collaborating in large teams and using many services and technologies. We can however choose from a whole range of services supporting our calms in the cloud ranging from complete managed services to services giving us full control and flexibility of the environment.

Provider-Independent services and tools

MLFlow

A framework you will find implemented in most data cloud services independent of the cloud provider. It consists of 4 components, supporting and streamlining our ML projects and helping us conduct them systematically and collaboratively.

It lets us streamline our ML projects by systematically tracking experiment runs, and easy packaging using Anaconda and Docker.

Tracking: This can be used during model training to lock and track model parameters, and KPIs like model performance metrics, experimental models, code version numbers, etc.

Projects: can be used to package ML model training and trigger it remotely with varying parameters. Once we are satisfied with the performance of a trained model, we can proceed.

Model: register this model with ML flow.

Model registry: can be used as a centralized repository to train models and we can use these repositories to deploy models to the production environment easily.

An applied example of using an ML flow project to share our ML development easily and trigger runs with specific parameters over the network.
For example, someone optimized a Python script to train a machine-learning model. The new model looks very promising but you have to fill in that tweaking a parameter might make the ML model perform even better.

MLflow project can be used to do this without writing the code again.
Eg, this is the GitHub rep you used.
We have
Data, conda environment, readme file, trained pyscript, license.
The script imports models and defines evaluation metrics

Typical MLFLOW workflow:

Start MLFlow Server

Coduct and tract run: meaning to train models and lock the parameters and modern metrics.

Evaluate results using graphical user interphase or its SDK to query the best-performing models programmatically. Once satisfied with a model, we register it.

Register model

*Model registry *

Deploy models: we deploy a model for inference to a productive system typically equipped with a restful API that we can use to obtain predictions from the model.

Databricks: Processing massive amounts of data requires parallel computation on several cluster notes. Spark is a well-known and widely used solution for data processing and data-intensive projects. It is open source but the downside lies in its maintenance and operation overhead. It might not be that simple to set up a spark cluster and keep it running on custom machines.

If we lack the resources to do this, we can use Databricks.
Databricks is An overreaching technology that many cloud providers(CSP) can host; Databricks was founded in 2013 by some of the founders of Spark.
It is therefore not surprising that the core services offered by DB are managed by spark clusters.

It is active in open source and hosting the data and AI summit yearly. At its core, it offered managed spark clusters that can be hosted on Azure, AWS GCP, etc.

Different billing models depend on the cloud platform of choice. But in general, databricks units are used. Depending on the machine, processes, and storage that is used, it is calculated in DBUs. we can also use the community edition at no cost. The managed spark clusters are at the core of Databricks but additional features are designed for data-intensive projects.

Additional features are:

Lakehouses: aims to describe unified data storage that comes with data bricks designed for structured and unstructured data so it can be considered a mixture of a data lake and a data warehouse. It uses files to store the data in a delta lake.

Delta Lake: the data storage used in Databricks. It is a distributed datastore but using an elaborate metadata mechanism, delta lake is ACID-compliant.

Delta Live Tables(DLT): they automatically propagate underlined data updates and provide a GUI to design and manage DTL pipelines straightforwardly. We can also set up data quality checks in DLT that are automatically conducted regularly.

Delta Engine: A query optimizer that automatically and periodically optimizes data queries depending on the data access pattern.

Unity Catalog: With Unity Catalog, databricks offers an easy-to-use tool for data governance. We can the graphical user interface of the unity catalog or SQL queries to perform data governance tasks for example, we can set role-based access control for a specific dataset.

Structuring the data in Delta Lake by processing maturing, there are
Bronze, silver, and Gold tables.
Bronze tables: contains raw data as it was loaded from external sources.

Silver tables: contain processed data including established joints.

Gold tables: contain completely pre-processed data ready for specific use cases like machine learning.

We can use Python, R, SQL, and scala to develop the ML and data processing logic in scripts, and notebooks or use remote compute targets.

Analysis components

Databricks SQL: similar to spark SQL to use the SQL language to query our data.

Databricks data science and engineering environment: similar to ML leap in spark. It comes with a prepared environment for typical data science and data engineering use cases.

Databricks machine learning environment: it uses MLflow to track versions and deploy machine learning models.

Programming languages: python, R, SQL, Scala similar to Spark.
AutoML: to automatically perform feature selection, algorithm choice, and hyperparameter toning.

Google Data Science and ML services

Services give us maximum control and flexibility of the environment, leaving us with some responsibilities such as security patches and environment maintenance. Eg, we can use compute engines for our data science projects and Google’s cloud-based virtual machines.

Using containerization instead of VM, we can develop our containerized application for eg model training or inference on a local machine then deploy that container to

Cloud compute engine:

Kubernetes engine: the cloud-based container orchestration service on GCP.
Deep learning VM: Specialised VM coming with GPU support and pre-installed libraries that are typically used in data science projects.

All Cloud providers offer specialized ML services giving us more comfort and managing some tedious maintenance tasks for us but also taking a little bit of control and flexibility.

On the Google platform, the specialized ML service is called cloud AI/ vertex AI being the main component. They include:

Dataflow
Composer
Dataproc
Bigquery
Google Cloud AI for ML training etc
Google Cloud console
Vertex AI workbench
Vertex AI formerly datalab.
Auto ML
API endpoints
Visual reporting

Ready-to-use services on GCP: they can be used by calling the standardized restful APIs to obtain model predictions by pretraining machine learning models hosted on GCP. eg we can use the speech-tech API to send the sound file to that service and receive the transcribed text.

Natural language AI
Teachable machine
Dialogflow
Translations
Speech-to-text
Text-to-speech
Timeseries insights API
Vision AI
Video AI

Data science and ML services on AWS

Same as on GCP, there are services maximizing control and others suitable for more comfort and instant application. Example,
Cloud-based VM called EC2.

Deploying containers to elastic container service ECS/Elastic kubernetes service EKS
And Elastic map reduce EMR

Specialized ML Service

It is called *SageMaker: *
Sagemaker is a service family of an entire collection of sub-services dedicated to supporting typical DS and ML projects. eg, there are graphical user interfaces, auxiliary services for data wrangling, data labeling, prepared scripts and templates, and also auto ML features.

Notebook instances
Data labeling
Data wrangler
Feature Store
Clarify
Pipelines
Studio
Jumpstart
Canvas
Autopilot

Ready to use services on AWS:

Translate, transcribe, computer vision, and other services being pre-trained ML models that are callable for restful APIs for inference.
The computer vision API here includes:

Comprehend
Rekognition
Lookout for vision, panorama
Textract
A2I
Personalize
Translate, Transcribe
Polly, Lex
Forecast
Fraud Detector
Lookout for Metrics
Kendra
Auxillary services are DevOps Guru and Code Guru.

Data science and ML services on Microsoft Azure.

VM: For maximum control and flexibility, we can use Azure VMs to decide how to create and manage our development environment.

ACI/AKS: we can use Azure container instance ACI for containerized model training for testing purposes then use Azure Kubernetes service AKS for production settings.

HDInsight: A managed service for technology.

Databricks:

Synapse Analytics:

Specialized ML service.

This is called Azure Machine Learning. Similar to AWS sagemaker, Azure ML comprises several sub-services for example the GUI called designer and auxiliary services for data labeling and other features.

Azure machine learning
Studio
Workspace
Notebooks/RStudio
Data labelling
Designer

Ready-to-use services on Azure.

Giving us maximum comfort and usability of pre-trained models callable by restful APIs. These services are called cognitive services and they include:

Computer vision, Face
Azure cognitive service for language
Language understanding models,
Translators, and other services.
QnA Maker, Translator
Speech Service
Anomaly Detector
Content Moderator
Personalizer
Cognitive Services for Big Data

Note: This is a 4-fold series on cloud computing, virtualization, containerization, and data processing.
Check the remaining 3 articles on my blog.

This is the fourth. Here is the link to the third.

https://dev.to/michellebuchiokonicha/cloud-computing-platforms-4667

it focuses on cloud platforms.

Follow me on Twitter Handle: https://twitter.com/mchelleOkonicha

Follow me on LinkedIn Handle: https://www.linkedin.com/in/buchi-michelle-okonicha-0a3b2b194/
Follow me on Instagram: https://www.instagram.com/michelle_okonicha/

Cloud Computing Platforms

Michellebuchiokonicha — Mon, 17 Jun 2024 16:40:01 +0000

Introduction

When moving to the cloud, we can use the services one of many CSPs offers. we can only choose the service optimal for our use case if we know what services are available.

Interapobaility has become increasingly important in today's inter-connected world, it is common for organizations to use multi-cloud strategies integrating cloud services by several cloud providers or with existing on-premise infrastructure.

Furthermore, to prevent vendor lock-in, we should be aware of the various offerings by competing cloud providers to stay flexible. There are many more reasons why we should learn what the different providers offer such as security and compliance, innovation and new services, ecosystem integration, performance optimization, scalability and flexibility, technical support, etc.
Here we will discuss some of the most frequently used services offered by the big three cloud providers to enable us to choose what is best for our situation. This applies to everyone using the cloud including data scientists, cloud architects and consultants, cloud developers and engineers and so many more.

Terminologies Defined

Cloud Platform: An abstraction layer that can be used to provision resources and as a starting point for cloud development.

Cloud service provider: A corporation running many data centers worldwide and offering services based on this service for rent.

There are many CSPs but three of them, due to their considerable market share are the most popular.

Amazon web services

It started in 2006, is one of the oldest CSPs, and has the biggest market share. Compute services include:

Elastic compute cloud(EC2)
Virtual Private Cloud (VPC)
Simple Storage Service (S3)
Relational Database Service(RDS)
Lambda
Kinesis
Elastic Map Reduce EMR

Others include:

Route 53,
Simple Notification Service(SQS),
Elastic Load Balancer
Network firewall.

Data security and protection

Considering some thoughts into which availability zone should be used to deploy a resource can contribute to data protection. Moreso, data being redundant in many zones can mitigate data loss and support data failover.

On the other hand, limiting the
geographical location to which data can be stored and processed can ensure that this falls under one jurisdiction or another.

As far as monitoring control and encryption services are concerned, there are:

CloudTrail, Macie, CLoudHSM, Key Management service(KMS), cloud Tower
GuardDuty, Nitro system
ISO compliance for cloud security, privacy information management, and cloud privacy.
VPC

Pricing and costs

Most services are offered on a pay-as-you-go basis.

There are 12 months when we can try services and some services are free.

Certifications

They run certification programs to become associates and then professionals.

Microsoft Azure

It started as the Windows Azure platform in 2010 then became Microsoft Azure in 2014. The virtual machines in Azure are called virtual machines and its services include:

Virtual machine
Kubernetes services
Container Instance
Blob storage: Used for object storage
Azure SQL: cloud version of Microsoft SQL Server, a relational database.
Azure Functions: the serverless compute service here is called Azure Functions.
DNS, notification hubs, load balancer, and firewall:
Azure machine learning: provides abstraction for data connection
Azure Databricks for big data processing
HDInsight
Data Factory

Data security and protection

Azure has ways of ensuring security and protection.
They include:

Access management in Azure Active Directory:
Data Encryption at rest and in transit
Virtual Private Network(VPN, VNet)
Security center

Pricing and Costs

Some services are free and it is for 12v months, pay as you go. You can also save costs by making long-term commitments and using either resource.

Certifications

There are certifications to become experts from associates to professionals.

Google Cloud Platform

It started in 2011 with the cloud computing engine which is Google Virtual Machines.
Its services include:

Cloud VPC: Virtual private cloud services
App Engine
Kubernetes Engine: containers as a service
Compute engine
BigQuery
Dataflow
Pub/Sub
Cloud storage: This is Google object storage
Cloud Spanner: A distributed relational database service providing transactional consistency
Cloud Functions: Google serverless computing service
Cloud DNS, Cloud Pub/Sub, cloud load balancing

Data security and protection

They use various ways to ensure security like:

Encryption at rest and in transit
ISO standards for cloud security, privacy, and regulatory compliance.
VPC - virtual private cloud
Security command center
Identity and access management
Firewalls

Pricing and costs

Similar to the other providers, 12 months and pay as you go. There are also free tier offerings.

The largest cloud players are:

AWS
Azure
Google
Alibaba Cloud
IBM
Salesforce

Gartner Magic Quadrant

It divides the market into Challengers, Leaders, Niche Players, Visionaries

Leaders: AWS, Azure, Google
Niche players: IBM
Visionaries: Alibaba cloud

Criteria to consider when choosing a suitable cloud provider

Company strategy
Regulatory Compliance
Regional presence
Technologies and service roadmap
Performance and reliability
Availability and SLA
Billing models and costs
Flexibility
Support

Cloud reference architectures

Azure architectures
AWS reference architecture
Google Cloud reference architecture

Note: This is a 4-fold series on cloud computing, virtualization, containerization, and data processing.
Check the remaining 3 articles on my blog.

This is the third. Here is the link to the second.

https://dev.to/michellebuchiokonicha/virtualization-containerization-with-docker-storage-and-network-services-2bjf/edit
it focuses on docker, containerization, virtualization, storage technologies, and network services.

Follow me on Twitter Handle: https://twitter.com/mchelleOkonicha

Follow me on LinkedIn Handle: https://www.linkedin.com/in/buchi-michelle-okonicha-0a3b2b194/
Follow me on Instagram: https://www.instagram.com/michelle_okonicha/