Forem: Emmanuel Kiriinya

Pandas vs Polars: Is It Time to Rethink Python’s Trusted DataFrame Library?

Emmanuel Kiriinya — Sun, 22 Jun 2025 21:30:59 +0000

For over a decade, Pandas has been the cornerstone of tabular data manipulation in Python. Its intuitive syntax and rich functionality make it the default choice for analysts, data scientists, and researchers worldwide.

However, as datasets have grown from megabytes to gigabytes—and now terabytes—the limitations of Pandas are increasingly evident. Enter Polars: a modern, high-performance DataFrame library built for speed and scalability.

In this article, we’ll cover:

Why Pandas remains popular
What makes Polars different
A practical benchmark with a large real-world dataset
Whether Pandas might eventually be replaced

Pandas: A Reliable Workhorse

Since its release in 2008, Pandas has dominated data analysis in Python. Its strengths include:

Familiar and expressive API (DataFrame, Series)
Seamless integration with other Python libraries (NumPy, scikit-learn, matplotlib)
Extensive tutorials, examples, and community support

However, Pandas was designed for single-threaded execution and expects the entire dataset to fit in memory. This often becomes a bottleneck when working with very large datasets on a laptop or single machine.

Polars: A Modern Alternative for High-Performance DataFrames

Polars is a newer open-source DataFrame library, written in Rust with Python bindings. It’s designed with performance and scalability in mind:

Multi-threaded execution: Polars uses all available CPU cores automatically.
Lazy evaluation: Like Spark, Polars can optimize a query plan before executing it.
Memory efficiency: Processes data in chunks to avoid excessive memory usage.

These design choices allow Polars to handle large datasets much faster and with lower resource consumption than Pandas.

Pandas vs Polars: A Real-World Benchmark

To see the difference in practice, let’s analyze a real dataset: the NYC Taxi Trip data, which typically has over 20 million rows and is about 3 GB uncompressed.

Below is a simple benchmark computing the average trip distance grouped by passenger count, using both libraries.

# Install the libraries if needed:
# pip install pandas polars

import time
import pandas as pd
import polars as pl

# Replace with the path to your CSV file
FILE_PATH = "yellow_tripdata_2023-01.csv"

# --- Using Pandas ---
start = time.time()
df_pd = pd.read_csv(FILE_PATH)
result_pd = df_pd.groupby("passenger_count")["trip_distance"].mean()
print(result_pd)
print("Pandas execution time:", time.time() - start)

# --- Using Polars ---
start = time.time()
df_pl = pl.read_csv(FILE_PATH)
result_pl = (
    df_pl.groupby("passenger_count")
    .agg(pl.col("trip_distance").mean())
)
print(result_pl)
print("Polars execution time:", time.time() - start)

Expected results (typical laptop):

Pandas: 20–30 seconds, high memory usage
Polars: 3–6 seconds, significantly lower memory footprint

This highlights how Polars can dramatically speed up large data workflows.

When to Use Each Library

Aspect	Pandas	Polars
Execution Model	Single-threaded	Multi-threaded, supports lazy evaluation
Performance	Good for small to medium data	Excellent for large data
Memory Usage	Entire dataset in RAM	Efficient chunk processing
API Maturity	Highly mature	Rapidly evolving
Community Support	Large & established	Growing rapidly

Will Pandas Be Replaced?

It’s unlikely that Pandas will be phased out anytime soon. Reasons include:

Deep integration in the Python ecosystem
Many libraries (e.g., scikit-learn, statsmodels) expect Pandas DataFrames
Widely taught in courses, bootcamps, and used in countless notebooks

In practice, many modern data workflows use both:

Pandas for quick exploration and prototyping, Polars for heavy transformations, large datasets, or production-grade pipelines.

Key Takeaway

Pandas isn’t going anywhere — but Polars is raising the bar for what’s possible on a single machine.

If you work with large CSVs, Parquet files, or complex transformations, try Polars on your next project. It’s an easy way to process more data faster, with less hardware overhead.

Next Steps

Try Polars with your largest dataset

Experiment with its lazy API for ETL pipelines

Stay comfortable with Pandas for quick analyses and prototyping

Understanding PowerBI

Emmanuel Kiriinya — Sun, 25 May 2025 10:44:47 +0000

In today’s data-driven world, visualization plays a central role in helping people understand and act on information. Whether in business, research, or public service, interpreting data visually is a critical skill. Power BI, Microsoft's powerful business intelligence tool, makes data visualization more accessible to analysts, developers, and decision-makers alike. This article covers the fundamentals of data visualization, the advantages of using Power BI, common use cases, and practical tips for creating effective dashboards.

What Is Data Visualization?

As discussed in my previous article on 'Visualization Beyond Aesthetics', Data visualization is about making complex information clear, truthful, and actionable, or say the process of representing data through graphical elements like charts, graphs, and maps. Rather than analyzing raw rows and columns of data, visualization presents patterns, trends, and outliers in an easily digestible form. A well-designed chart can make complex data more intuitive, saving time and reducing the risk of misinterpretation.

Common types of visualizations include:

Bar and column charts for comparing quantities across categories
Line charts for showing trends over time
Pie charts for representing parts of a whole
Maps for visualizing geographical data
Scatter plots for exploring relationships between variables
Heatmaps for identifying intensity patterns across two dimensions

Effective visualizations simplify decision-making and highlight insights that might otherwise go unnoticed.

What Is Power BI?

Power BI is a business analytics and data visualization platform developed by Microsoft. It allows users to connect to various data sources, prepare and transform data, and create interactive reports and dashboards. Power BI is widely used across industries due to its ease of use, integration with other Microsoft tools, and ability to scale from small projects to enterprise-level deployments.

There are three main components:

Power BI Desktop: A free Windows application used for report building and data modeling.

Power BI Service: A cloud-based platform for publishing, sharing, and collaborating on reports.

Power BI Mobile: An app for accessing dashboards on phones and tablets.

Why Use Power BI for Visualization?

Power BI has become a preferred tool for visualization for several reasons:

User-Friendly Interface
Power BI uses a drag-and-drop interface that doesn’t require coding. Users can create visuals by simply selecting fields and choosing how to represent them. This lowers the learning curve for beginners while still supporting more advanced users through features like DAX (Data Analysis Expressions) and custom visuals.
Wide Range of Visualization Types
From basic charts to more advanced visuals like gauges, KPIs, decomposed trees, and maps, Power BI offers dozens of built-in options. Users can also import custom visuals from the Microsoft marketplace.
Interactivity
Dashboards in Power BI are interactive by default. Selecting a data point in one visual automatically filters other visuals on the same page. This allows users to explore relationships between data points in real-time, without writing new queries.
Real-Time Updates
With live connections to databases or APIs, Power BI dashboards can reflect real-time data. This is particularly useful for monitoring key metrics like website traffic, sales performance, or system uptime.
Data Integration
Power BI connects to a wide range of data sources, including Excel, SQL Server, Azure, SharePoint, PostgreSQL, and even web services. Data can be imported or accessed through direct queries.
Data Preparation Tools
Power BI includes Power Query, which provides tools for cleaning, reshaping, and merging data. Users can remove duplicates, change data types, split columns, and apply transformations, all without external software.

Common Use Cases

Power BI is used in a variety of domains. Some of the most common include:

Business Performance Tracking
Organizations use Power BI dashboards to monitor sales, revenue, profit margins, and market performance. Dashboards often include KPIs, time series charts, and filters to compare performance across regions or products.
Marketing and Customer Analytics
Marketing teams track campaign performance, customer segmentation, and conversion rates using Power BI. Data from CRMs or social media platforms can be integrated to provide a complete view of customer behavior.
Finance and Budgeting
Financial analysts use Power BI for tracking budgets, forecasting revenue, and comparing planned vs. actual expenditures. Waterfall charts, variance visuals, and custom DAX formulas are commonly used in this context.
Operations and Supply Chain
Power BI helps operations teams monitor inventory, shipping times, and production efficiency. Real-time dashboards allow for quick response to delays or supply chain disruptions.
HR and Workforce Analytics
HR departments use dashboards to visualize hiring trends, employee turnover, training completion, and diversity metrics. Filters allow reports to be customized by department, region, or time period.

Building a Visualization in Power BI Step-by-Step

Connect to Data
Begin by loading data into Power BI. You can connect to Excel sheets, databases, CSV files, or online services. Choose either an import or direct query mode depending on your needs.

Clean and Transform
Use Power Query to clean the data. This might include renaming columns, filtering rows, changing data types, or merging tables.

Build Data Model
Define relationships between tables. Use the “Model” view to create links between keys (e.g., CustomerID or ProductID) across tables. This allows data from different sources to be used in the same visual.

Create Visuals
Drag fields into the canvas and select the type of visual you want to use. Customize titles, labels, colors, and tooltips. Group related visuals together into dashboards.

Add Filters and Slicers
These allow users to interact with the dashboard. For example, a slicer could let someone view data from a specific year or region.

Publish and Share
Upload your report to the Power BI Service to share it with others. You can control permissions, schedule data refreshes, or embed reports into apps or websites.

Best Practices for Visualization in Power BI

Be Clear and Focused
Every visual should serve a specific purpose. Avoid overcrowding a dashboard with too many elements. Highlight key findings and keep navigation simple.

Use Appropriate Chart Types
Choose the chart type that best fits the data and message. Don’t use pie charts for comparing more than three or four categories. Avoid 3D effects and other visual distortions.

Make It Interactive
Take advantage of Power BI’s built-in interactivity. Use slicers and drill-throughs to let users explore the data in ways that matter to them.

Ensure Accessibility
Use high-contrast colors and readable fonts. Test how visuals look on different screen sizes, especially if they’ll be viewed on mobile devices.

Document and Label Clearly
Include titles, axis labels, and tooltips. If you use custom calculations or filters, provide notes or legends to explain them.

Learning Power BI

For students and early-career professionals in data science and analytics, Power BI is a practical tool to master. It combines data manipulation, modeling, and visualization in one platform, helping you build both analytical and communication skills.

To get started:

Use sample datasets (e.g., sales data, COVID-19 data, survey results)

Try recreating dashboards you find onlineor use free datasets from websites like Kaggle.
Participate in Power BI community challenges or forums

Projects don’t have to be complicated. Even a dashboard tracking your spending can be a great way to practice.

Final Thoughts

Power BI offers a powerful platform for turning data into insight. Its user-friendly interface, wide range of visualization options, and strong integration with other tools help analysts and decision-makers alike work with data more effectively. Whether you're preparing a class assignment, exploring real-world datasets, or building dashboards for

Visualization Beyond Aesthetics

Emmanuel Kiriinya — Thu, 15 May 2025 10:38:47 +0000

In today’s data-driven world, we’re surrounded by slick dashboards, colorful graphs, and eye-catching infographics. Whether you’re scrolling through LinkedIn or sitting in a boardroom, chances are you’ve seen a chart that looks great—but does it actually help you understand the data?

Data visualization is often praised for its aesthetic appeal, but effective visualization is about much more than making data look pretty. At its core, it’s about making complex information clear, truthful, and actionable.

As Edward Tufte, a pioneer in the field, once said:

“Graphical excellence consists of complex ideas communicated with clarity, precision, and efficiency.”

— The Visual Display of Quantitative Information

Let’s dive into why visualization should go beyond just the aesthetics—and how designers, analysts, and decision-makers can use it to tell better, more honest stories with data.

When Pretty Isn’t Enough

It’s easier than ever to create visually appealing charts thanks to tools like Tableau, Power BI, and Flourish. But focusing too much on aesthetics can lead to visuals that are attractive but misleading or ineffective.

Alberto Cairo, in his book The Truthful Art, warns:

“A good graphic isn’t just one that looks nice, but one that is based on sound data and communicates clearly.”

Take 3D charts, for example. They might look cool, but they often distort how we perceive the data. Similarly, excessive design elements—what Tufte called “chartjunk”—can confuse more than clarify.

A visualization should serve a function, not just win a design award.

Example: A 3D pie chart with too many categories and distorted proportions.

Visualization as a Thinking Tool

When done right, visualizations become cognitive tools—helping people think, analyze, and decide. They reduce the mental effort required to interpret large datasets or find insights.

Stephen Few, in Now You See It, puts it simply:

“A chart should not be judged by its appeal, but by its ability to help users see what they need to see.”

The takeaway? Use the right chart for the right job. Want to compare values? Use a bar chart. Want to show change over time? Use a line graph. Good design is about making the data easier to understand—not just easier on the eyes.

Storytelling with Data: More Than Just Numbers

It’s not enough to show data—you need to tell a story with it. This is especially true when your audience includes non-technical decision-makers.

Cole Nussbaumer Knaflic, author of Storytelling with Data, explains:

“Numbers have an important story to tell. Rely on data to tell that story, but augment it with the right visuals and the right context.”

Think of your chart as a narrative. What’s the main point? What do you want your audience to take away? Use annotations, highlights, and clean layouts to guide the viewer through the story.

Pro tip: remove anything that doesn’t serve the message. Decluttering is your best friend.

Know Your Audience

The same visualization won’t work for everyone. What’s intuitive to a data analyst might be confusing to an executive or policymaker.

Kieran Healy, in Data Visualization: A Practical Introduction, emphasizes:

“Design decisions should be guided by empathy for the reader.”

That means:

Avoiding jargon
Using familiar chart types
Adding helpful labels or explanations

Design with context in mind. Who’s looking at this chart? What do they already know? What do they need to learn?

Visual Ethics: Telling the Truth with Data

A beautiful but misleading chart is dangerous. Whether it’s a cherry-picked axis or a deceptive color scale, bad visualizations can manipulate opinions and mislead decisions.

Edward Tufte called out these tactics decades ago, and the problem persists today.

Alberto Cairo reinforces this in The Truthful Art, calling for functional honesty in design. The designer’s job isn’t to persuade—it’s to illuminate the truth.

This is especially critical in fields like journalism, public health, and finance. A good chart can influence policy. A bad one can cause harm.

Example of a Good Dashboard

Great dashboards prioritize usability and clarity. They surface key insights at a glance and avoid information overload.

Example: A clean executive dashboard showing KPIs with filters, annotations, and intuitive navigation.

Final Thoughts: Show the Data, Tell the Truth

Beautiful charts might win likes on social media—but in real-world decision-making, clarity and integrity are what truly matter.

As Edward Tufte reminds us:

“Above all else, show the data.”

Data visualization is a bridge between numbers and understanding. When we design with intention—beyond aesthetics—we unlock its real power: to inform, to inspire, and to drive better decisions.

Thanks for reading! Let me know in the comments—what’s the most misleading chart you’ve seen in the wild?

SQL For Data Analytics in a Nutshell

Emmanuel Kiriinya — Thu, 15 May 2025 10:36:01 +0000

Overview

An almost incomprehensible amount of data is created daily, and figures grow ever-increasingly each year. For future usage, this data has to be stored electronically for analysis and research to produce actionable insights. In this article, I will explain one of the most widely used data management systems, SQL, focusing on Relational Databases.

What is SQL?

SQL stands for Structured Query Language, pronounced popularly as se-qu-el. SQL is a standard language for managing, manipulating, and querying databases basically a language used to communicate with databases. It was developed at IBM in the early 1970s and has undergone tremendous development updates, adapting to the emerging technologies. There are different types of SQL Database Management Systems like PostgreSQL, MySQL, SQL Server, Oracle, SQLite, etc. SQL stores data in a tabular format with columns and rows. The type of data stored in these databases is known as structured data.
SQL can be further broken down into 4 sublanguages for tackling different jobs, namely:

Data Definition Language(DML): It is used to modify the structure of the database(used to create and modify tables, views etc). The commands used are CREATE, ALTER, DROP, and TRUNCATE. Examples CREATE TABLE

CREATE TABLE customers (id INT,
 name VARCHAR(255));

ALTER TABLE

ALTER TABLE customers ADD COLUMN email VARCHAR(255);

DROP TABLE

DROP TABLE customers;

Data Query Language(DQL): It is used for querying data. DQL command is mainly the SELECT statement.

SELECT *
FROM customers;

Data Manipulation Language(DML): It is used to act on the data itself. Commands include INSERT, UPDATE, DELETE, and MERGE. Examples INSERT

INSERT INTO customers (id, name, email) 
(1, 'John Doe', 'john@example.com')
(2, 'Marya Akoth', 'marya.akoth@mail.com');

UPDATE

UPDATE customers 
SET name = 'Jane Doe' WHERE id = 1;

DELETE

DELETE FROM customers WHERE id = 1;

Data Control Language(DCL): It is used to manage access to a database and its data. It is usually used by database administrators to manage access in an organisation. Commands include GRANT and REVOKE.

Why is SQL the foundation of Data Analytics?

In the data universe, data engineers and database administrators will use SQL to ensure that everybody in their organisation can access the data they need, sometimes depending on the permissions granted to the users.
Data Scientists will use SQL to query data stored in a database to train and load data into their Machine Learning Models.
Data analysts will use data to query tables of data and derive insights.

What are the benefits of SQL?

Standard Database Management System: SQL is the standard language for retrieving data and interacting with databases. This is because most software applications and scripting languages, such as Python and R, are compatible with SQL.
Scalability: SQL is a perfect tool for processing large datasets compared to tools like spreadsheets, where handling many rows might not be effective. This gives SQL an upper hand since it can easily handle millions of entries. Nevertheless, its scalability serves as a key benefit to meet the needs of modern data analysis.
Easy to learn and effective: When compared to other programming languages, SQL is simple to learn and use, enabling data analysts to achieve immediate and effective results. Its clear syntax and straightforward commands empower analysts to execute complex computations efficiently.
Handling Structured data: Someone who works as an analyst should be able to comprehend SQL to perform different tasks on structured data. This includes generating and managing data sets kept in structured databases like Oracle, Microsoft SQL Server, and MySQL.
Data Preparation and Wrangling: Data cleaning and preprocessing are among the initial steps involved in data analysis. Unexpectedly, SQL is essential for performing these functions, especially when utilizing big data tools. This facilitates data manipulation in a manner that simplifies analysis while also helping the analyst avoid numerous errors.

A practical, slightly deeper understanding of common terms and syntax used in SQL.

Now that we have defined SQL, its sublanguages, and its benefits, let's dive deeper and deepen our understanding of the basic terms and syntax associated with SQL.
Let's dive into it.

Step 1: Create the Schema

CREATE SCHEMA ecommerce_ke;
USE ecommerce_ke;

Clause Tip: CREATE SCHEMA defines a namespace for organizing your tables and other database objects.

Step 2: Create the `customers` Table

CREATE TABLE customers (
    customer_id INT PRIMARY KEY AUTO_INCREMENT,
    first_name VARCHAR(50),
    last_name VARCHAR(50),
    email VARCHAR(100),
    city VARCHAR(50),
    registration_date DATE
);

Step 3: Create the `orders` Table

CREATE TABLE orders (
    order_id INT PRIMARY KEY AUTO_INCREMENT,
    customer_id INT,
    order_date DATE,
    order_amount DECIMAL(10, 2),
    status VARCHAR(20),
    FOREIGN KEY (customer_id) REFERENCES customers(customer_id)
);

Step 4: Insert Sample Data (with explicit `customer_id`)

INSERT INTO customers (customer_id, first_name, last_name, email, city, registration_date) VALUES
(1, 'Akinyi', 'Otieno', 'akinyi.otieno@example.com', 'Nairobi', '2023-01-15'),
(2, 'Kamau', 'Mwangi', 'kamau.mwangi@example.com', 'Mombasa', '2023-02-10'),
(3, 'Wanjiru', 'Kariuki', 'wanjiru.kariuki@example.com', 'Kisumu', '2023-03-05'),
(4, 'Mutiso', 'Nzomo', 'mutiso.nzomo@example.com', 'Nakuru', '2023-03-18'),
(5, 'Naliaka', 'Wekesa', 'naliaka.wekesa@example.com', 'Eldoret', '2023-04-01');

INSERT INTO orders (order_id, customer_id, order_date, order_amount, status) VALUES
(1, 1, '2023-03-01', 5000.00, 'Completed'),
(2, 2, '2023-03-02', 7500.00, 'Completed'),
(3, 3, '2023-03-04', 3000.00, 'Pending'),
(4, 1, '2023-03-10', 2000.00, 'Completed'),
(5, 4, '2023-04-12', 10000.00, 'Cancelled'),
(6, 5, '2023-04-14', 4500.00, 'Completed');

Step 5: Query Examples

1. `SELECT` and `FROM`

SELECT first_name, last_name, city FROM customers;

Clause Tip: SELECT retrieves specific columns, and FROM specifies the table source.

Output:

first_name | last_name | city
-----------|-----------|---------
Akinyi     | Otieno    | Nairobi
Kamau      | Mwangi    | Mombasa
Wanjiru    | Kariuki   | Kisumu
Mutiso     | Nzomo     | Nakuru
Naliaka    | Wekesa    | Eldoret

2. `WHERE`

SELECT * FROM orders WHERE status = 'Completed';

Clause Tip: WHERE filters records based on specified conditions.

Output:

order_id | customer_id | order_date | order_amount | status
---------|-------------|------------|--------------|----------
1        | 1           | 2023-03-01 | 5000.00      | Completed
2        | 2           | 2023-03-02 | 7500.00      | Completed
4        | 1           | 2023-03-10 | 2000.00      | Completed
6        | 5           | 2023-04-14 | 4500.00      | Completed

3. `GROUP BY`

SELECT city, COUNT(*) AS total_customers FROM customers GROUP BY city;

Clause Tip: GROUP BY aggregates data by one or more columns.

Output:

city     | total_customers
---------|------------------
Nairobi  | 1
Mombasa  | 1
Kisumu   | 1
Nakuru   | 1
Eldoret  | 1

4. `HAVING`

SELECT customer_id, COUNT(*) AS total_orders FROM orders GROUP BY customer_id HAVING COUNT(*) > 1;

Clause Tip: HAVING filters groups created by GROUP BY, unlike WHERE which filters rows.

Output:

customer_id | total_orders
-------------|--------------
1            | 2

5. `ORDER BY`

SELECT * FROM orders ORDER BY order_amount DESC;

Clause Tip: ORDER BY sorts results by one or more columns.

Output:

order_id | customer_id | order_amount | status
---------|-------------|--------------|---------
5        | 4           | 10000.00     | Cancelled
2        | 2           | 7500.00      | Completed
1        | 1           | 5000.00      | Completed
6        | 5           | 4500.00      | Completed
3        | 3           | 3000.00      | Pending
4        | 1           | 2000.00      | Completed

6. `LIMIT`

SELECT * FROM customers ORDER BY registration_date DESC LIMIT 3;

Clause Tip: LIMIT restricts the number of rows returned.

Output:

first_name | registration_date
-----------|-------------------
Naliaka    | 2023-04-01
Mutiso     | 2023-03-18
Wanjiru    | 2023-03-05

Conclusion

SQL is a tool that spans various industries and is also the backbone of the data industry. Therefore, for someone looking to delve into data will have to learn SQL.

How Are Data Science, Data Analytics, And AI Transforming Industries?

Emmanuel Kiriinya — Wed, 23 Apr 2025 19:52:04 +0000

Are We Living in the Age of Data-Driven Transformation?

In today's internet era, the sheer volume of data generated every day is staggering. From the moment we unlock our smartphones, count steps using smart watches to the instant a transaction is completed online, even with as little as one Kenyan Shilling, data is being produced, stored, and more importantly, analyzed. The convergence of Data Science, Analytics, and Artificial Intelligence (AI) isn't only changing how companies operate and redefining entire industries.
One might ask, beneath the surface of this data revolution, what’s truly happening? How are organizations turning raw data into refined intelligence and 'prophecy'? And what does this mean for the future of work, business, and innovation? In this article, I will explore a part of the endless possibilities that data has achieved that could not have been imagined a decade ago.

How data science has helped in seeing what humans can’t.

The core idea behind data science is finding patterns where others see chaos. Today, businesses operate in complex, competitive environments where every decision made counts. With the development of machine learning algorithms and predictive analytics, data science has helped organizations identify insights that would otherwise have gone unnoticed.

Let us consider the financial industry. Hedge funds and trading platforms now use sophisticated algorithms to predict market movements, identify arbitrage opportunities, and assess credit risks with more accuracy than ever before, minimizing possible mistakes by human calculations. What once took days of manual analysis can now be achieved in minutes or seconds, depending on the amount of data. And it's not just about speed but also precision.

Another good example is the healthcare sector. By analyzing patient records, medical literature, and real-time health data, AI-powered systems can predict disease outbreaks, identify early signs of chronic illness, and recommend personalized treatment plans. A human doctor backed by data-driven insights is no longer science fiction; it’s science fact.

Is AI the New Backbone of Industry?

Artificial Intelligence is no longer a futuristic concept—it’s the backbone of many modern enterprises. Whether it's Natural Language Processing (NLP) enabling chatbots to understand human queries, or Computer Vision allowing machines to detect defects in a manufacturing line, AI is everywhere.

In retail, AI is revolutionizing the customer experience. Recommendation engines, powered by deep learning, suggest products based on browsing history, past purchases, and even mood inferred from recent activity. Visual search tools allow customers to upload images and instantly find matching products. The entire shopping experience is becoming more intuitive, efficient, and personalized.

Meanwhile, logistics and supply chain management are undergoing dramatic transformation. AI helps forecast demand, optimize routes in real time, and predict potential disruptions due to weather, geopolitical issues, or pandemics. The result? Faster delivery times, reduced operational costs, and enhanced customer satisfaction.

How Are Traditional Industries Being Reimagined?

Industries traditionally resistant to change are now embracing data and AI to remain competitive. Agriculture, for example, is turning to AI-powered drones and sensors to monitor soil health, crop growth, and pest activity. These insights have enabled precision farming, where every drop of water or ounce of fertilizer is optimized.

Construction firms are leveraging predictive analytics to estimate project timelines, costs, and risks more accurately. AI-driven safety monitoring tools use video analytics to identify unsafe behavior on job sites, significantly reducing workplace accidents.

Even the legal profession—long associated with piles of paperwork and manual research—is being reshaped. Legal AI platforms now scan thousands of legal documents in seconds, extracting relevant case laws and identifying inconsistencies. This allows legal teams to focus on strategy rather than data gathering.

Can Data Analytics Predict the Future?

One of the most powerful aspects of analytics is its predictive capability. By identifying patterns and trends, analytics can forecast outcomes with remarkable accuracy. In the energy sector, predictive analytics is used to manage grid loads, anticipate maintenance needs for equipment, and even optimize energy consumption patterns for entire cities.

Marketing teams across industries deeply rely on predictive analytics to understand consumer behavior. By analyzing demographics, transaction histories, and social media interactions, marketers can tailor campaigns that reach the right audience at the right time, boosting engagement and conversions.

In the public sector, predictive policing is a controversial but growing field. By analyzing crime data, police departments can deploy resources more efficiently and potentially prevent crimes before they happen. While it raises ethical questions, it undeniably showcases the power of data-driven decision-making.

What Happens When Humans and Machines Collaborate?

The idea that AI will replace humans is a common fear, but the more likely scenario is collaboration, not competition. In many cases, AI augments human capabilities rather than replacing them. Surgeons use robotic systems to enhance precision during operations. Journalists rely on AI tools to transcribe interviews and even generate first drafts of news articles. Financial analysts use machine learning models to validate their projections.

This fusion of human intuition and machine intelligence often produces superior outcomes. It enables professionals to focus on high-value, creative tasks while offloading repetitive or data-heavy work to algorithms.

Importance of Ethics in AI.

As AI and analytics become more pervasive, ethical considerations have taken center stage. How do we ensure that AI systems stay fair, transparent, and free from bias? How do we protect consumer data from misuse or breaches? And who is accountable when an AI-driven decision causes harm?

Responsible AI development involves more than just technical accuracy. It requires diverse datasets, transparent algorithms, regular audits, and inclusive development teams. Companies are now establishing AI ethics boards and investing in explainability tools to ensure that AI decisions can be understood and challenged.

Data privacy is another critical concern. With regulations like GDPR and CCPA in place, businesses must be more transparent about how they collect, store, and use data. Ethical data science practices are not just good PR, they're a business imperative.

Are We Ready for the Next Frontier as a Country?

The evolution of data science, analytics, and AI is accelerating. We’re entering an era where edge computing, quantum analytics, and generative AI could redefine what’s possible.
Imagine smart cities where traffic lights adapt to real-time congestion, energy systems self-optimize based on demand, and emergency services are dispatched using predictive modeling. Envision businesses that use digital twins to simulate every aspect of their operations before implementing real-world changes. Or picture personalized education systems where AI tailors content to each student’s learning style and pace.
These aren’t distant dreams. They are already taking shape in labs, startups, and innovation hubs around the world. Let's hope Kenya's 2025 AI plan will be implemented to help our country not to be left behind in this revolution.

Conclusion: Are You Embracing the Change or Watching From the Sidelines?

The impact of Data Science, Analytics, and AI is profound and irreversible. Organizations that are leveraging these tools effectively are not just surviving, they are thriving. They make faster decisions, operate more efficiently, and offer better experiences to customers and employees alike.

The question is no longer whether data-driven technologies will reshape the world. The real question is, will your organization be ready when they do?

Now is the best time to invest in data skills, infrastructure, and strategy. It’s time to foster a culture that values data-driven thinking and embraces continuous learning. Because in the age of intelligent automation and predictive insights, those who adapt will lead, and those who don’t will surely be left behind.

Microsoft Excel: The Underrated Backbone of Data Analytics

Emmanuel Kiriinya — Mon, 14 Apr 2025 08:13:05 +0000

Daily, it is estimated that the world produces approximately 5 billion gigabytes of data. This data has to be processed using data analytics tools to deduce useful insights. In the age of data science and data analytics, where Python, R, and SQL often steal the spotlight, Microsoft Excel might seem like a relic of the past and an outdated tool. Yet, Excel remains one of the most powerful, versatile, and widely used tools in data analytics. While many seasoned data professionals migrate to more advanced platforms, it's important to acknowledge one undeniable fact: most data analysts began their journey with Excel.

As perceived by many, Microsoft Excel isn't just a spreadsheet software for data entry and record keeping but a full-fledged analytical tool that, when wielded with the right skill, can be very useful to deliver deep insights, automated processes, and creation of interactive dashboards. All without writing a single line of code, just functions and formulas.

The Gateway Tool: Why Excel is the First Step for Many

If you ask any group of data analysts how they started, a majority will say Excel. It is approachable, pre-installed on most Windows based laptops/PC's, and doesn't require technical setup or coding knowledge. Whether it’s a financial report, inventory management, a marketing dashboard, or a patient records analysis in healthcare, Excel often serves as the default environment for data manipulation by most beginners.

Most professionals who develop skills in data analytics/data science likely started in Excel. It is typically the first tool we are exposed to to manage data and develop reporting and insight from the data.
– Don Tomoff, Data Analytics Advisor

What Makes Excel Powerful for Data Analytics?

Data Exploration and Cleaning Made Easy From simple filters to Power Query, Excel allows analysts to clean and transform data easily. Tasks that would require multiple lines of Python or R can often be completed in minutes with Excel’s Graphical user interface. To clean data in Excel, follow these steps:

Identify and remove duplicates: Use the Remove Duplicates feature to eliminate duplicate entries.
Handle missing values: Use the IFERROR or IFBLANK functions to replace missing values with a specific value or a calculated result.
Standardize data formats: Use the Text to Columns feature to convert text to a standard format for dates, numbers, and other data types.
Remove unnecessary characters: Use the TRIM and SUBSTITUTE functions to remove leading/trailing spaces and unwanted characters.
Validate data: Use formulas and conditional formatting to identify invalid or inconsistent data.

PivotTables: A Non-Technical Data Engine
With just a few clicks, users can summarize thousands of rows of data, calculate aggregates, and uncover trends with no programming knowledge required. Below is a sample of pivot tables from an HR dataset.
Formula and function Ecosystem
Excel’s function library rivals many scripting languages. Conditional logic (IF, IFS, AND, OR), lookups (VLOOKUP, HLOOKUP, INDEX + MATCH), and new dynamic array functions (FILTER, UNIQUE) make data wrangling both powerful and readable.
Power Pivot and DAX
For users working with multiple tables or large datasets, Power Pivot and DAX (Data Analysis Expressions) enable relational modeling and advanced calculations — similar to what you’d find in SQL or Power BI.

5._ Dashboards and Visualization_
Slicers, pivot tables, and pivot charts allow users to create dynamic dashboards. These can be shared directly in Excel files or embedded in reports, making insights easily accessible to stakeholders and also easy to understand.

Excel Meets Artificial Intelligence

As part of AI taking over the world by storm and Microsoft being part of the evolution, Excel is undergoing a major transformation powered by AI, machine learning, and natural language processing.

Excel Copilot (Microsoft Copilot for Excel) Microsoft Copilot is a game-changer for productivity. It integrates generative AI directly into Excel, enabling users to:

Ask natural language questions like: “What are the top-selling
regions in Q1?”
Automatically generate formulas, charts, and summaries based on your
dataset.
Translate technical formulas into plain English (and vice versa)
Example Use Case: Instead of writing =IF(AND(A2>12, A2<=19), "Teenager", ""), you can simply type:

“Label all patients aged between 13 and 19 as Teenagers.”
Copilot will handle the logic and apply it.

Suggested Patterns & Smart Fill
Excel now intelligently recognizes repetitive patterns and suggests completions (beyond Flash Fill), using machine learning to spot logic across columns.
Data Insights with Natural Language Queries
Tools like Analyze Data in Excel empower you to understand your data through natural language queries that allow you to ask questions about your data without having to write complicated formulas. In addition, Analyze Data provides high-level visual summaries, trends, and patterns. Excel can summarize your spreadsheet and even generate pivot tables or charts in response to plain language prompts. This makes exploratory data analysis more accessible to non-technical users.

4._ Deep Integration with Power BI and Power Platform_
Excel can now serve as a front-end interface to Power BI datasets and Power Apps. You can pull live data, manipulate it with Excel’s familiar tools, and sync changes back to a shared workspace — bridging self-service analytics and enterprise BI.

5._ Introduction of Add-Ins/Extensions_
Excel add-ins are like extra tools that help automate tasks, analyze data better, and even connect to other apps or databases. They’re great for saving time, especially with repetitive work or complex analysis that regular formulas can’t handle easily. Plus, they make Excel more user-friendly with custom buttons and features you can reuse across different files or share with your team.

Real-World Relevance
From hospital administrators tracking diagnoses to small businesses analyzing monthly sales, Excel continues to empower decision-makers in diverse fields. For many professionals, especially in resource-constrained environments, Excel is not just the easiest tool, it’s the only one.

The Limitations of Excel

Despite its power, Excel has real limitations:

_ Scalability_
Excel begins to struggle with performance when handling very large datasets (>1 million rows) or when complex formulas/functions are overused.

Version Compatibility
Some AI features (like Copilot or dynamic arrays) require the latest version of Microsoft 365 — not available to all users.

Risk of Human Error
Without strict data governance, Excel sheets can become error-prone, difficult to audit, and risky for enterprise-critical tasks.

Limited Reproducibility
Analytical workflows in Excel are harder to version control and automate compared to scripting languages.

Tips for Maximizing Excel's Analytical Capabilities

1. Use Named Ranges and Tables for dynamic referencing
1. Learn Power Query for repeatable data transformation
1. Explore Power Pivot + DAX to handle large and relational datasets
1. Take advantage of Analyze Data and Copilot if available, but be aware that AI is sometimes error-prone.
1. Use INDEX + MATCH instead of VLOOKUP and HLOOKUP for performance and flexibility since VLOOKUP and HLOOKUP search values vertically and horizontally, respectively.

When to Supplement Excel with Other Tools
Excel is great for prototyping, EDA (exploratory data analysis), and reporting. But for production analytics pipelines or real-time dashboards, pair it with:

SQL for scalable querying and joins
Python/R for advanced statistical modeling and automations
Power BI/Tableau for interactive visualizations

Conclusion: Excel Isn’t Going Anywhere, It’s Evolving

Microsoft Excel has stood the test of time by constantly adapting to the needs of its users. With the integration of AI-powered tools like Copilot, Excel is no longer just a spreadsheet, it’s a smart analytics assistant.

Whether you’re a beginner working on your first dataset or an advanced user building dynamic dashboards, Excel deserves its place in the data analytics toolbox. It's fast, powerful, and deeply embedded in business workflows. And now, with the rise of AI, it’s smarter than ever.

Forem: Emmanuel Kiriinya

Pandas vs Polars: Is It Time to Rethink Python’s Trusted DataFrame Library?

Pandas: A Reliable Workhorse

Polars: A Modern Alternative for High-Performance DataFrames

Pandas vs Polars: A Real-World Benchmark

When to Use Each Library

Will Pandas Be Replaced?

Key Takeaway

Next Steps

Understanding PowerBI

What Is Data Visualization?

What Is Power BI?

Why Use Power BI for Visualization?

Common Use Cases

Building a Visualization in Power BI Step-by-Step

Best Practices for Visualization in Power BI

Learning Power BI

Final Thoughts

Visualization Beyond Aesthetics

When Pretty Isn’t Enough

Visualization as a Thinking Tool

Storytelling with Data: More Than Just Numbers

Know Your Audience

Visual Ethics: Telling the Truth with Data

Example of a Good Dashboard

Final Thoughts: Show the Data, Tell the Truth

SQL For Data Analytics in a Nutshell

Overview

What is SQL?

Why is SQL the foundation of Data Analytics?

What are the benefits of SQL?

A practical, slightly deeper understanding of common terms and syntax used in SQL.

Step 1: Create the Schema

Step 2: Create the customers Table

Step 3: Create the orders Table

Step 4: Insert Sample Data (with explicit customer_id)

Step 5: Query Examples

1. SELECT and FROM

2. WHERE

3. GROUP BY

4. HAVING

5. ORDER BY

6. LIMIT

Conclusion

How Are Data Science, Data Analytics, And AI Transforming Industries?

Are We Living in the Age of Data-Driven Transformation?

How data science has helped in seeing what humans can’t.

Is AI the New Backbone of Industry?

How Are Traditional Industries Being Reimagined?

Can Data Analytics Predict the Future?

What Happens When Humans and Machines Collaborate?

Importance of Ethics in AI.

Are We Ready for the Next Frontier as a Country?

Conclusion: Are You Embracing the Change or Watching From the Sidelines?

Microsoft Excel: The Underrated Backbone of Data Analytics

The Gateway Tool: Why Excel is the First Step for Many

What Makes Excel Powerful for Data Analytics?

Excel Meets Artificial Intelligence

The Limitations of Excel

Conclusion: Excel Isn’t Going Anywhere, It’s Evolving

Step 2: Create the `customers` Table

Step 3: Create the `orders` Table

Step 4: Insert Sample Data (with explicit `customer_id`)

1. `SELECT` and `FROM`

2. `WHERE`

3. `GROUP BY`

4. `HAVING`

5. `ORDER BY`

6. `LIMIT`