Forem: Shivappa

(EDA Part-5) Multivariate Analysis — Wrapping Up EDA and What Comes Next

Shivappa — Wed, 22 Apr 2026 01:01:23 +0000

Over the past four parts, we zoomed in on single features (univariate analysis), then looked at pairs (bivariate). Now it’s time for the real fun: multivariate analysis. This is where we weave three or more features together and look for patterns we simply couldn’t see before.

The analogy — the blind men and the elephant 🐘

You have probably heard this story. Six blind men each touch a different part of an elephant — trunk, tusk, leg, tail, ear, body. Each one describes something completely different and none of them is wrong. But none of them sees the full picture either.

That is exactly what happens when you look at features in isolation.

Sex alone tells you about women and children first
Pclass alone tells you about wealth and lifeboat access
Age alone shows almost nothing (r = -0.05)

But combine all three together? Suddenly you see the elephant. A young boy (Age < 14) in 1st class had very different odds from an old man in 3rd class. Neither Sex, Pclass, nor Age reveals this alone.

That is what multivariate analysis is for—finally seeing the whole elephant.

So, what exactly is multivariate analysis?

Just investigating three or more variables at the same time.
Let us break that down:

Type	Features involved	Example
Univariate	1	What does the Age distribution look like? (One variable)
Bivariate	2	How does Age relate to Survived? (Two variables)
Multivariate	3+	How does Age + Sex + Pclass together relate to Survived?

Each step reveals patterns the previous step couldn't see.

Step 1: Three-way survival breakdown — Sex + Pclass + Survived

Remember the crosstab from Part 4? Almost all first-class women survived, but only a handful of third-class men did. Let us make that visual with a grouped bar chart.

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

# Survival rate grouped by both Sex and Pclass
survival_grouped = df.groupby(['Pclass', 'Sex'])['Survived'].mean().reset_index()

fig, ax = plt.subplots(figsize=(10, 6))

sns.barplot(
    data=survival_grouped,
    x='Pclass',
    y='Survived',
    hue='Sex',
    palette={'male': '#378ADD', 'female': '#D85A30'},
    ax=ax
)

ax.set_title('Survival rate by Passenger Class and Sex', fontsize=14, pad=15)
ax.set_xlabel('Passenger class (1 = Upper, 2 = Middle, 3 = Lower)')
ax.set_ylabel('Survival rate (0 = 0%, 1 = 100%)')
ax.set_xticks([0, 1, 2])
ax.set_xticklabels(['1st class', '2nd class', '3rd class'])
ax.legend(title='Gender')
ax.set_ylim(0, 1.1)

print(ax.patches)


# Add percentage labels on each bar
for p in ax.patches:
    if p.get_height() > 0: # This skips the "hidden" patches
        ax.annotate(
           f'{p.get_height():.0%}',
           (p.get_x() + p.get_width() / 2., p.get_height()),
           ha='center', va='bottom', fontsize=10, fontweight='bold'
        )

plt.tight_layout()
plt.show()

Class	Female survival	Male survival	Gap
1st	97%	37%	60 points
2nd	92%	16%	76 points
3rd	50%	14%	36 points

What pops out? The gender gap is huge, especially in second class—92% of women survived vs just 16% of men. Even third-class women had a survival rate of 50%; men, just 14%. The “women and children first” approach wasn’t just folklore—it is visible in the data.

Step 2: Add Age into the picture — the FacetGrid

Let us complicate things. Using seaborn’s FacetGrid, we can compare age distributions split by both Sex and Pclass, and see how survival changed in every combination.

grid = sns.FacetGrid(
    df,
    row='Pclass',
    col='Sex',
    hue='Survived',
    palette={0: '#E24B4A', 1: '#1D9E75'},
    height=3,
    aspect=1.4
)

grid.map(plt.hist, 'Age', bins=20, alpha=0.6, edgecolor='white')
grid.add_legend(title='Survived', labels=['Did not survive', 'Survived'])

# Add axis labels to every subplot
for ax in grid.axes.flat:
    ax.set_xlabel('Age (years)')
    ax.set_ylabel('Number of passengers')

grid.set_titles(row_template='Class {row_name}', col_template='{col_name}')
grid.figure.suptitle('Age distribution by Class, Sex and Survival', y=1.02, fontsize=13)

plt.tight_layout()
plt.show()

Key Observations from the FacetGrid:

First-class females — green (survived) bars dominate at almost every age. Near-universal survival.
Third-class males — red (died) bars dominate at almost every age. Very few survived regardless of age.
Children (Age < 10) — in 1st and 2nd class, green bars appear in the youngest age bins even for males. The "children first" policy had a small effect in upper classes.
Elderly passengers — older passengers in 3rd class have almost zero green bars. Age + class + sex combined paints a very different picture than any one feature alone.

Step 3: Engineering features—extracting Title from Name**

Back in Part 4, we saw that Age alone barely gave any clues about survival. But we also saw that children did much better than adults—you just can’t separate them cleanly by age column alone. Here is a trick: the Name column actually contains titles, and those do a much better job dividing up people by age and gender together.

The Name column contains titles — and titles encode both age and sex together far more efficiently than either column alone.

# Extract title from Name
df['Title'] = df['Name'].str.extract(r' ([A-Za-z]+)\.', expand=False)

# Check all titles
print(df['Title'].value_counts())

Mr        517
Miss      182
Mrs       125
Master     40   # ← key one: 'Master' = young boys under ~14
Dr          7
Rev         6
...

There are some Title we need to bring to common format like Mme (French Abbreviation for Madame) is Mrs. So you see there are some count difference between above code and below.

# Group rare titles into one bucket
df['Title'] = df['Title'].replace(
    ['Lady', 'Countess', 'Capt', 'Col', 'Don',
     'Dr', 'Major', 'Rev', 'Sir', 'Jonkheer', 'Dona'], 'Rare'
)
df['Title'] = df['Title'].replace({'Mlle': 'Miss', 'Ms': 'Miss', 'Mme': 'Mrs'})

# Survival rate by Title
print(df.groupby('Title')['Survived'].agg(['mean', 'count']).round(2))

Now, look at survival rates by Title:

        mean  count
Title
Master  0.57     40   # ← young boys: much higher than adult men
Miss    0.70    185   # ← unmarried women + girls: very high
Mr      0.16    517   # ← adult men: lowest by far
Mrs     0.79    126   # ← married women: highest
Rare    0.35     23

Note: This is the power of multivariate thinking. Sex alone gives you male vs female. Age alone gives you a weak signal. But Title gives you Master (young boy, 57%) vs Mr (adult man, 16%) — a massive difference that neither Sex nor Age could show individually. Two features combined into one engineered feature reveals what neither showed alone.

Now plot it visually:

fig, ax = plt.subplots(figsize=(9, 5))

title_survival = df.groupby('Title')['Survived'].mean().sort_values(ascending=False)

colors = ['#1D9E75' if v >= 0.6 else '#EF9F27' if v >= 0.35 else '#E24B4A'
          for v in title_survival.values]

bars = ax.bar(
    title_survival.index,
    title_survival.values,
    color=colors,
    edgecolor='white',
    width=0.6
)

ax.set_title('Survival rate by passenger title', fontsize=13, pad=12)
ax.set_xlabel('Title extracted from Name column')
ax.set_ylabel('Survival rate (0 = 0%, 1 = 100%)')
ax.set_ylim(0, 1.0)

for bar, val in zip(bars, title_survival.values):
    ax.text(
        bar.get_x() + bar.get_width() / 2,
        val + 0.02,
        f'{val:.0%}',
        ha='center', fontsize=11, fontweight='bold'
    )

plt.tight_layout()
plt.show()

Step 4: Family size — creating another multivariate feature

We saw in Part 4, that SibSp and Parch each showed very weak individual correlations with survival. But combined into a single Family_Size feature, a clear non-linear pattern emerges.

# Combine SibSp + Parch into Family_Size
df['Family_Size'] = df['SibSp'] + df['Parch'] + 1  # +1 = the passenger themselves

# Survival rate by family size
family_survival = df.groupby('Family_Size')['Survived'].agg(['mean', 'count'])
family_survival.columns = ['Survival Rate', 'Count']
print(family_survival.round(2))

Survival rates:

             Survival Rate  Count
Family_Size
1                     0.30    537 # ← alone: 30%
2                     0.55    161 # ← pair: 55%
3                     0.58    102 # ← small family: 58%
4                     0.72     29 # ← medium family: 72%!
5                     0.20     15 # ← large family: drops to 20%
6                     0.14     22
7                     0.33     12
8                     0.00      6
11                    0.00      7 # ← huge family: 0%

fig, ax = plt.subplots(figsize=(10, 5))

colors = ['#1D9E75' if v >= 0.55 else '#EF9F27' if v >= 0.3 else '#E24B4A'
          for v in family_survival['Survival Rate']]

ax.bar(
    family_survival.index,
    family_survival['Survival Rate'],
    color=colors,
    edgecolor='white',
    width=0.7
)

ax.set_title('Survival rate by family size aboard', fontsize=13, pad=12)
ax.set_xlabel('Family size (passenger + number of relatives aboard)')
ax.set_ylabel('Survival rate (0 = 0%, 1 = 100%)')
ax.set_xticks(family_survival.index)
ax.set_ylim(0, 0.85)

for size, row in family_survival.iterrows():
    ax.text(size, row['Survival Rate'] + 0.02,
            f"{row['Survival Rate']:.0%}", ha='center', fontsize=9)

plt.tight_layout()
plt.show()

What this shows ? Neither SibSp nor Parch showed strong survival signals individually. But Family_Size reveals a clear sweet spot at 2–4 people. Small families could coordinate and stay together. Solo travellers had nobody helping them. Huge families (7+) couldn't move together and had 0% survival. Classic multivariate insight — more than the sum of its parts.

Step 5: Updated correlation heatmap with engineered features

Now that we have engineered Title and Family_Size, let us re-run the correlation heatmap and see how much it improved compared to the raw features from Part 4.

import numpy as np

# Encode for correlation
df['Sex_enc']   = (df['Sex'] == 'female').astype(int)
df['Title_enc'] = df['Title'].map({'Mr': 0, 'Miss': 1, 'Mrs': 2, 'Master': 3, 'Rare': 4})

corr_cols = ['Survived', 'Pclass', 'Sex_enc', 'Age', 'Fare',
             'Has_Cabin', 'Family_Size', 'Title_enc']

corr_matrix = df[corr_cols].corr()

fig, ax = plt.subplots(figsize=(9, 7))
sns.heatmap(
    corr_matrix,
    annot=True,
    fmt='.2f',
    cmap='RdYlGn',
    center=0,
    square=True,
    linewidths=0.5,
    ax=ax
)
ax.set_title('Correlation heatmap — raw + engineered features', fontsize=13, pad=12)
ax.set_xlabel('Features')
ax.set_ylabel('Features')

plt.tight_layout()
plt.show()

Before vs after engineering:

Feature	Before (raw)	After (engineered)
Age	r = -0.05	r = -0.05 (unchanged)
Title_enc	—	r = +0.54 (new — encodes age + sex together)
Family_Size	—	r = +0.02 (new — replaces SibSp + Parch)
Sex_enc	r = +0.54	r = +0.54 (unchanged)
Pclass	r = -0.34	r = -0.34 (unchanged)

Engineering Title alone gave us a brand new feature that correlates as strongly with Survived as Sex itself — and it came purely from multivariate thinking.

The complete EDA journey — all 5 parts in one view

Part 1 → WHY EDA matters
         Analogies: Sherlock Holmes, the Doctor, the Chef, the Navigator
         Titanic dataset introduced — 891 rows, 12 columns

Part 2 → FIRST LOOK
         df.shape, df.dtypes, df.describe(), df.isnull().sum()
         Reading the describe() output — mean vs median, skewness clues
         Categorical columns: Sex, Pclass, Embarked

Part 3 → UNIVARIATE ANALYSIS
         Histograms and box plots for Age and Fare
         3 types of missing data: MCAR, MAR, MNAR
         Fixing nulls correctly for each type

Part 4 → BIVARIATE ANALYSIS
         Survival rates: 74% female vs 19% male
         Correlation heatmap — which raw features predict survival
         Crosstab: 97% of 1st class women survived vs 14% of 3rd class men

Part 5 → MULTIVARIATE ANALYSIS (this article)
         Three-way breakdown: Sex + Pclass + Survived
         FacetGrid: Age + Sex + Pclass all at once
         Engineered features: Title and Family_Size
         How combining features reveals what none showed alone

The final definition of EDA

After 5 parts and a real dataset, here is how I now define EDA in one sentence:

EDA is the practice of asking honest questions about your data before you ask it to predict anything.

You ask: what is in here? What is broken? What is missing? What relates to what? And only after you have answered those honestly do you move to modelling.

Skipping EDA is like a surgeon who doesn't read your X-ray before operating. The surgery might go fine. Or you might operate on the wrong leg.

What is coming next — the ML algorithms series

Now that EDA is done, the natural next question is: how do we actually build a model from all these insights?

That is exactly what I am planning for the next series. I am currently learning and will be writing about each algorithm as I understand it:

Linear Regression — predicting a continuous number (e.g. house prices)
Logistic Regression — predicting a category (e.g. survived or not)
Decision Trees — how machines make if-then decisions
Random Forests — why many trees beat one tree
And more — as I learn them, I will write them honestly

Each series will follow the same approach as this one — real datasets, real code, real-world analogies, and zero pretending I know more than I do.

Note: I am writing these as I learn. So if something is unclear or you spot a mistake, please drop a comment — your feedback genuinely makes these articles better. We are learning together.

Thank you 🙏

This series took a lot of effort to put together and I truly hope it helped you see EDA not as a boring checkbox, but as the most important habit you can build as a learner.

If you found it useful, pass it on to someone else on the same journey. And stick around for the ML algorithms series—Linear Regression is up next.

See you there!

(EDA Part-4) Bivariate Analysis — What Actually Predicted Who Survived (on Titanic)?

Shivappa — Tue, 21 Apr 2026 01:09:16 +0000

Part 4 of 5 — Intermediate → Advanced

This is the most exciting part of EDA. We've learned each feature individually in Part 3. Now we ask the real question: who survived, and why?

The analogy — the detective reveals the suspect 🔍

Sherlock has examined every clue individually. Now he starts connecting them.

The mud on the shoes came from the east garden, which only the gardener used between 9 and 11 AM, and the victim was last seen at 10:30...

Each piece connects. That is what bivariate analysis. Connecting features to each other and to the outcome. We stop looking at columns in isolation and start asking: what do two columns tell us together?

What does the real Titanic data say?

Let us look at the actual numbers right out of the dataset:

Group	Survival rate
Female passengers	74%
Male passengers	19%
1st class passengers	63%
2nd class passengers	47%
3rd class passengers	24%

No complicated model needed. Just pandas and a couple lines of code, and clear patterns jump out.

Step 1: who survived? Feature vs target

import seaborn as sns
import matplotlib.pyplot as plt

fig, axes = plt.subplots(1, 3, figsize=(15, 5))

# Sex vs Survival
survival_by_sex = df.groupby('Sex')['Survived'].mean()
bars = axes[0].bar(survival_by_sex.index, survival_by_sex.values,
                   color=['#1D9E75', '#E24B4A'])
axes[0].set_title('Survival rate by Sex')
axes[0].set_ylabel('Survival rate')
# Add percentage labels on bars
for bar, val in zip(bars, survival_by_sex.values):
    axes[0].text(bar.get_x() + bar.get_width()/2, val + 0.01,
                 f'{val:.1%}', ha='center', fontweight='bold')

# Pclass vs Survival
survival_by_class = df.groupby('Pclass')['Survived'].mean()
bars2 = axes[1].bar(['1st', '2nd', '3rd'],
                    survival_by_class.values,
                    color=['#1D9E75', '#EF9F27', '#E24B4A'])
axes[1].set_title('Survival rate by Passenger Class')
for bar, val in zip(bars2, survival_by_class.values):
    axes[1].text(bar.get_x() + bar.get_width()/2, val + 0.01,
                 f'{val:.1%}', ha='center', fontweight='bold')

# Age distribution: survived vs died
df[df['Survived']==1]['Age'].hist(bins=25, ax=axes[2], alpha=0.6,
                                   label='Survived', color='#1D9E75')
df[df['Survived']==0]['Age'].hist(bins=25, ax=axes[2], alpha=0.6,
                                   label='Died', color='#E24B4A')
axes[2].set_title('Age: survived vs died')
axes[2].set_xlabel('Age (years)')
axes[2].legend()

plt.tight_layout()
plt.show()

Key insight: Being female gave you a 74% chance of surviving. Being male: only 19%. Being in 1st class gave you 63% survival vs 24% in 3rd class. Sex and Pclass will be the two most important features in any Titanic model.

Step 2: Violin plots — see the full distribution, not just the average

Bar charts show averages. But Violin plots show the full shape of the distribution per group.

fig, axes = plt.subplots(1, 2, figsize=(10, 4))

# Fare distribution by survival
sns.violinplot(
    data=df, 
    x='Survived', 
    y='Fare', 
    hue='Survived',
    ax=axes[0], 
    palette=['#E24B4A', '#1D9E75'],
    legend=False
)
axes[0].set_title('Fare distribution: survived vs died')
axes[0].set_xticks([0, 1])
axes[0].set_xticklabels(['Died', 'Survived'])

# Age distribution by survival
sns.violinplot(
    data=df, 
    x='Survived', 
    y='Age', 
    hue='Survived',
    ax=axes[1], 
    palette=['#E24B4A', '#1D9E75'],
    legend=False
)
axes[1].set_title('Age distribution: survived vs died')
axes[1].set_xticks([0, 1])
axes[1].set_xticklabels(['Died', 'Survived'])

plt.tight_layout()
plt.show()

What the Fare violin tells us ?
Survivors paid significantly higher fares on average. The violin for "Survived" is thicker at higher fare values — passengers who paid more had access to better cabin locations, closer to lifeboats. This is Fare acting as a proxy for wealth and location on the ship.

What the Age violin tells us ?
The shapes look quite similar! Both groups have a similar age distribution. This is why Age showed only -0.07 correlation with Survived earlier. But look carefully — the survived group has a slightly fatter section in the 0-10 range. Kids did get priority. You can literally see the women and children first effect right in the data.

Step 3: The correlation heatmap — who are the strongest predictors?

# Encode Sex as numeric for correlation analysis
df['Sex_encoded'] = (df['Sex'] == 'female').astype(int)

corr_cols = ['Survived', 'Pclass', 'Sex_encoded', 'Age',
             'SibSp', 'Parch', 'Fare', 'Has_Cabin']
corr_matrix = df[corr_cols].corr()

plt.figure(figsize=(9, 7))
mask = np.zeros_like(corr_matrix, dtype=bool)
mask[np.triu_indices_from(mask)] = True

sns.heatmap(corr_matrix,
            annot=True,
            fmt='.2f',
            cmap='RdYlGn',
            center=0,
            square=True,
            linewidths=0.5,
            mask=mask)
plt.title('Feature correlation matrix — Titanic')
plt.tight_layout()
plt.show()

Check out these results (correlation with Survived):

Feature	Correlation	What it means
Sex_encoded	+0.54	Strong predictor—females lived
Has_Cabin	+0.32	Having a cabin (= 1st/2nd class) predicts survival
Fare	+0.26	More money, better outcome
Pclass	-0.34	Higher class number (lower class) predicts death
Age	-0.05	Almost no linear relationship with survival
SibSp	-0.04	Negligible

Remember, this is Pearson correlation. If something is nonlinear—like kids having extra odds—it won’t show up here. That is why you always want to pair this kind of analysis with a closer group look with heatmap.

Step 4: The crosstab—the two-line powerhouse

Want the biggest insight with almost no code? Try a crosstab:

# Survival rate by Sex AND Pclass combined
pivot = pd.crosstab(
    index=df['Pclass'],
    columns=df['Sex'],
    values=df['Survived'],
    aggfunc='mean'
).round(2)

print(pivot)

Sex        female    male
Pclass
1           0.97     0.37    ← 97% of 1st class women survived!
2           0.92     0.16
3           0.50     0.14    ← only 14% of 3rd class men survived

This tiny table tells a huge story:

First-class woman** had a 97% survival rate — almost guaranteed rescue
Third-class man** had a 14% survival rate — almost guaranteed death.
Even for men, being in first class (37%) more than doubled your odds vs third class (14%).

Step 5: Point biserial correlation for binary targets

For a binary target like Survived, there is a more precise way to check feature correlations:

from scipy import stats

numeric_features = ['Age', 'Fare', 'SibSp', 'Parch', 'Has_Cabin', 'Sex_encoded']

print("Point Biserial Correlation with Survived:\n")
for col in numeric_features:
    corr, pval = stats.pointbiserialr(df['Survived'], df[col])
    significance = "✓ significant" if pval < 0.05 else "✗ not significant"
    print(f"  {col:<16} r={corr:+.3f}   p={pval:.4f}   {significance}")

Result:

Point Biserial Correlation with Survived:

  Age              r=-0.047   p=0.1587   ✗ not significant
  Fare             r=+0.257   p=0.0000   ✓ significant
  SibSp            r=-0.035   p=0.2922   ✗ not significant
  Parch            r=+0.082   p=0.0148   ✓ significant
  Has_Cabin        r=+0.317   p=0.0000   ✓ significant
  Sex_encoded      r=+0.543   p=0.0000   ✓ significant

So, what did we learn?

Here is the short list, straight from EDA:

Feature	Signal strength	Direction	Notes
Sex	Very strong	Female = survive	Most important feature
Pclass	Strong	1st = survive	Proxy for wealth + cabin location
Fare	Moderate	Higher = survive	Correlated with Pclass — redundant but useful
Has_Cabin	Moderate	Has cabin = survive	We engineered this from the missing Cabin col
Age	Weak but real	Younger = slight advantage	Non-linear — children specifically helped
SibSp	Not significant	—	Drop or combine into Family_Size
Parch	Weak	—	Combine with SibSp

What is next?

In Part 5, we’ll mix all this together. Multivariate analysis—combining everything we have learned and building a true predictive model. See you there.

(EDA Part-3) Univariate Analysis — Understanding Every Feature One at a Time

Shivappa — Sat, 18 Apr 2026 03:22:28 +0000

Part 3 of 5 — Intermediate

In Part 2, we got the patient's file open. Now, it is the time for the real examination. We're going to look at each "organ"—each feature in our dataset — one by one.

This is called univariate analysis, which is just a fancy term for looking at one variable at a time. We are not trying to find relationships yet. We are just getting a feel for the landscape. What does the Age column actually look like? How were the ticket Fares distributed?

The analogy: the fruit inspector

Think of yourself as a quality inspector at a fruit market. Before you make any buying decisions, you examine each fruit individually.

Is this apple bruised?
Is this mango overripe?
Is there mold on these grapes?

You are not comparing apples to oranges yet — you are checking each one independently.

The Numbers: Age and Fare

For numeric columns, my first move is always a histogram. It is the fastest way to see the shape of the data. When I look at a histogram, I am basically asking:

Is it symmetric? (like a bell curve) If the mean and median are close, it is a good sign.
Is it skewed? If there is a long tail to the right (right-skewed) or left (left-skewed), the mean will be pulled in that direction. This is a huge red flag for many models, and we'll likely need to fix it.
Are there multiple peaks? A "bimodal" distribution (two peaks) hints that you might have two distinct groups of data lumped together.
Are there crazy outliers? A box plot is great for spotting these.

Representing in Tabular way makes us easy to remember

Pattern	What it means	What to do
Mean ≈ Median	Roughly symmetric — healthy	Nothing special needed
Mean >> Median	Right-skewed (long right tail)	Consider log transform
Mean << Median	Left-skewed	Consider square root transform
Two peaks (bimodal)	Two hidden subgroups in data	Investigate — might need to split
Outliers in box plot	Unusual extreme values	Investigate before imputing

Age Distribution: Histogram

# Set graph layout
fig, axes = plt.subplots(1, 1, figsize=(6, 4))

# Age — histogram with mean and median lines
axes.hist(df['Age'].dropna(), bins=30, color='#378ADD', edgecolor='white', alpha=0.8)
axes.set_title('Age distribution')
axes.set_xlabel('Age (years)')
axes.set_ylabel('# of Passengers')
axes.axvline(df['Age'].mean(), color='red', linestyle='--', label=f'Mean: {df["Age"].mean():.1f}')
axes.axvline(df['Age'].median(), color='green', linestyle='--', label=f'Median: {df["Age"].median():.1f}')
axes.legend()

plt.tight_layout()
plt.show()

Age Outliers: Box plot

# Set graph layout
fig, axes = plt.subplots(1, 1, figsize=(6, 4))

# Age — box plot to spot outliers
axes.boxplot(df['Age'].dropna(), vert=False)
axes.set_title('Age — outlier check')
axes.set_xlabel('Age (years)')

plt.tight_layout()
plt.show()

My take: The mean (29.7) and median (28.0) are super close. This tells me the Age distribution is roughly symmetric, which is great. We see a small peak for children, which makes sense. The box plot shows a few older passengers, but nothing looks like a data entry error. The youngest passenger was 0.42 years (an infant!) and the oldest was 80. For now, Age looks healthy.

Fare Distribution: Histogram

# Set graph layout
fig, axes = plt.subplots(1, 1, figsize=(6, 4))

# Fare — histogram (right-skewed!)
axes.hist(df['Fare'], bins=40, color='#D85A30', edgecolor='white', alpha=0.8)
axes.set_title('Fare distribution (right-skewed!)')
axes.set_xlabel('Ticket fare (£)')
axes.set_ylabel('# of Passengers')
axes.axvline(df['Fare'].mean(), color='red', linestyle='--', label=f'Mean: £{df["Fare"].mean():.1f}')
axes.axvline(df['Fare'].median(), color='green', linestyle='--', label=f'Median: £{df["Fare"].median():.1f}')
axes.legend()


plt.tight_layout()
plt.show()

Fare distribution after log transform: Histogram

# Set graph layout
fig, axes = plt.subplots(1, 1, figsize=(6, 4))

# Fare after log transform — much better!
import numpy as np
axes.hist(np.log1p(df['Fare']), bins=40, color='#1D9E75', edgecolor='white', alpha=0.8)
axes.set_title('Fare after log transform — much better!')
axes.set_xlabel('log(Fare + 1)')
axes.set_ylabel('# of Passengers')

plt.tight_layout()
plt.show()

My take: Woah. Look at that first chart. The vast majority of people paid less than £50, but a tiny handful paid over £500. This massive right-skew pulls the mean (£32.20) to be more than double the median (£14.45).

This is a classic problem. Many machine learning models get confused by this kind of distribution. The fix? A log transform. By taking the logarithm of the fare (np.log1p is great because it handles zero-fares gracefully), we squish the high values closer to the rest of the data. Look at the second chart, it is much more symmetrical and "normal-looking." We will have to use this transformed version when we build our model.

Analysing categorical features

For categorical data, bar charts are your best friend. They quickly show you the balance (or imbalance) between groups.

fig, axes = plt.subplots(1, 3, figsize=(10, 3))

# Sex distribution
df['Sex'].value_counts().plot(kind='bar', ax=axes[0], color=['#378ADD', '#D85A30'])
axes[0].set_title('Passenger count by Sex')
axes[0].set_xticklabels(axes[0].get_xticklabels(), rotation=0)

# Pclass distribution
df['Pclass'].value_counts().sort_index().plot(kind='bar', ax=axes[1], color='#7F77DD')
axes[1].set_title('Passenger count by Class')
axes[1].set_xticklabels(['1st (Upper)', '2nd (Mid)', '3rd (Lower)'], rotation=0)

# Embarked distribution
df['Embarked'].value_counts().plot(kind='bar', ax=axes[2], color='#1D9E75')
axes[2].set_title('Embarked port distribution')
axes[2].set_xticklabels(axes[2].get_xticklabels(), rotation=0)

plt.tight_layout()
plt.show()

My take: No surprises, but some crucial context here.

Sex: 577 male (64.7%) vs 314 female (35.3%) — clear gender imbalance
Pclass: 491 in 3rd class (55%), 216 in 1st (24%), 184 in 2nd (21%) — most passengers were lower class
Embarked: 644 from Southampton (72%), 168 Cherbourg (19%), 77 Queenstown (9%)

The 3 types of missing data — This is where so many data projects go wrong

Most beginners see null values and immediately fill them with mean or median. This is dangerous. There are 3 types of missing data, and only one of them is safe to impute naively.

The analogy — three types of absent students

Imagine a classroom register with absent students.

1 — MCAR (Missing Completely at Random):
Students randomly absent due to illness. No pattern. Safe to estimate their marks from the class average.

2 — MAR (Missing at Random):
Students from one school zone are absent because of local transport issues. The absence is related to another known factor (location). Be careful imputing — group by the related factor first.

3 — MNAR (Missing Not at Random):
The students who scored lowest are absent because they are embarrassed to come in. The very thing you want to measure (performance) is driving the absence. Imputing here introduces hidden bias.

Which type are the Titanic nulls?

# Check: is Age missing more in certain passenger classes?
print(df.groupby('Pclass')['Age'].apply(lambda x: x.isnull().mean()).round(3))

Pclass
1    0.139   # 13.9% of 1st class ages are missing
2    0.060   # 6% of 2nd class
3    0.277   # 27.7% of 3rd class ages are missing!

Aha! The Age data is not missing randomly. It is missing far more often for 3rd class passengers. This is MAR. If we just used the overall median age, we would be giving 3rd class passengers an age that is likely biased by the older 1st and 2nd class passengers.

And what about Cabin? It is missing for 77% of people. This is because lower-class passengers didn't have assigned cabins. The missingness is a feature of their status. This is MNAR.

The Titanic missing data breakdown in Tabular way
| Column | Null count | Type | Why |
|---|---|---|---|
| Embarked | 2 | MCAR | Two passengers, no pattern — pure randomness |
| Age | 177 | MAR | Missing more in 3rd class — class-based bias |
| Cabin | 687 | MNAR | 3rd class had no assigned cabins by design — the class itself drives missingness |

Fixing the missing values correctly

Based on our investigation, here is the right way to handle it:

# Fix 1: Embarked — only 2 nulls, MCAR, use mode (most common port)
df['Embarked'] = df['Embarked'].fillna(df['Embarked'].mode()[0])

# Fix 2: Age — MAR, fill with median grouped by passenger class
# (3rd class passengers were systematically different in age distribution)
df['Age'] = df.groupby('Pclass')['Age'].transform(
    lambda x: x.fillna(x.median())
)

# Fix 3: Cabin — MNAR, 77% missing, don't impute
# Instead, create a binary flag: did this passenger have a cabin assigned?
df['Has_Cabin'] = df['Cabin'].notna().astype(int)
df = df.drop('Cabin', axis=1)

# Verify — should be all zeros now
print("Nulls after fixing:\n", df.isnull().sum())

Nulls after fixing:
 PassengerId    0
 Survived       0
 Pclass         0
 Name           0
 Sex            0
 Age            0    # fixed
 SibSp          0
 Parch          0
 Fare           0
 Embarked       0    # fixed
 Has_Cabin      0    # replaced Cabin with a binary flag

Note: Why did we create Has_Cabin instead of just dropping Cabin? Because having an assigned cabin is actually information — it indicates the passenger was likely in 1st or 2nd class. We keep that signal even though we can't use the raw cabin values. EDA insight turned into a feature.

The box plot check — spotting outliers

fig, axes = plt.subplots(1, 2, figsize=(12, 5))

# Fare outliers
df.boxplot(column='Fare', ax=axes[0])
axes[0].set_title('Fare — outlier check')

# Age outliers (post-imputation)
df.boxplot(column='Age', ax=axes[1])
axes[1].set_title('Age — outlier check')

plt.tight_layout()
plt.show()

print(f"Fare: Max = £{df['Fare'].max():.2f}, 99th percentile = £{df['Fare'].quantile(0.99):.2f}")

Fare: Max = £512.33, 99th percentile = £249.01

The box plot will show several large dots (outliers) on the Fare chart. That max of £512 is real — a passenger named Miss Charlotte Cardeza paid it for a luxury suite. We don't remove this.

Quick recap: univariate findings

Feature	Finding	Action
Age	~normal, 177 nulls (MAR)	Impute by Pclass median ✓
Fare	Right-skewed, max £512	Log transform
Cabin	77% null (MNAR)	Replace with Has_Cabin binary flag ✓
Embarked	2 nulls (MCAR)	Fill with mode (S) ✓
Sex	65% male	Will be strong predictor — Part 4 confirms this
Pclass	55% in 3rd class	Class imbalance — will affect model

What is next?

In Part 4, the exciting part — bivariate analysis. We will find out exactly which features predicted survival on the Titanic, with real survival rates from the actual data.

Let us explore more in Part 4.

(EDA Part-2) First Look at the Titanic Dataset — Loading Data and Understanding Big 5

Shivappa — Fri, 17 Apr 2026 05:45:22 +0000

Part 2 of 5 — Beginner → Intermediate

In Part 1, we compared EDA to a doctor's examination. Now let's actually open the patient file.

We'll use the Titanic dataset from Kaggle. To follow along:

Go to kaggle.com/competitions/titanic/data
Download train.csv using link https://www.kaggle.com/competitions/titanic/data?select=train.csv
OpenOr open a free Kaggle Notebook — no install needed, runs in the browser

The analogy before we start — first day at a new job

Imagine you've just joined a company and someone hands you a stack of 891 employee files. You don't start reading every file cover to cover. You first:

Count how many files there are
Look at the folder structure
Check if any files are missing pages
Scan a few at random to understand the format

That's exactly what we do in this section. We're not analysing anything deeply yet — we're orientating ourselves.

The Quick-Start Routine

The following 5 commands are my non-negotiables. I run them on every single new dataset before doing anything else. No exceptions.

The Big 5 — run these on EVERY new dataset

print("Shape:", df.shape) # Returns (rows, columns)
print("\nData types:\n", df.dtypes) # Check if numbers are actually stored as numbers
print("\nFirst 5 rows:\n", df.head()) # View the first 5 rows
print("\nStatistical summary:\n", df.describe()) # Get a bird's-eye view of the math behind your data
print("\nNull values:\n", df.isnull().sum()) # The "Missing Data Report"

Step 0: import required library and load the dataset.

import pandas as pd
import numpy as np

# Load the dataset
df = pd.read_csv('train.csv')

Step 1: Shape

Definition: This attribute returns a tuple representing the dimensionality of the DataFrame. It tells you exactly how many rows (observations) and columns (features) are in your dataset.

Why it matters: It’s the very first step to understand the scale of your project. If you expect 10,000 rows but see only 100, you know something went wrong during data loading.

df.shape

Output:

Shape: (891, 12)

(891, 12) means 891 rows and 12 columns.

Step 2: dtypes

Definition: This property returns the data type (e.g., int64, float64, object) of each column.

Why it matters: It helps you identify Type Mismatches. For example, if a Price column is listed as an object (string) instead of a float, you won’t be able to perform math on it until it's converted.

df.dtypes

Output:

Data types:
 PassengerId      int64
 Survived         int64
 Pclass           int64
 Name            object
 Sex             object
 Age            float64    # ← float means decimal values AND possibly nulls
 SibSp            int64
 Parch            int64
 Ticket          object
 Fare           float64
 Cabin           object
 Embarked        object

Watch out: Age is stored as float64, not int64. In pandas, an integer column with even ONE null value automatically becomes float64. That little decimal point is a hint telling you "this column has missing values." Age should logically be whole numbers — the float is a red flag.

Step 3: head()

Definition: This method returns the first n rows (default is 5) of the dataset.

Why it matters: It’s a sanity check. It allows you to see the actual content of the cells, verify that headers are correct, and get a feel for how the data is formatted.

df.head()

Output:

Step 4: describe()

Definition: This method generates descriptive statistics that summarize the central tendency, dispersion, and shape of a dataset’s distribution (excluding NaN values).

Why it matters: In one command, you get the Mean, Standard Deviation, Min/Max, and Quartiles. It is the fastest way to spot outliers—if the Max value for age is 200, you’ve found a data entry error.

df.describe()

Output:

The df.describe() output looks scary but it's just a table of 8 statistics per numeric column. Here's what to focus on:

Age column — what the numbers say:

Stat	Value	What it means
count	714	Only 714 of 891 rows have an age recorded
mean	29.7 years	Average age of passengers with known age
std	14.5	Spread of ages — quite wide
min	0.42	There were infants on board!
50% (median)	28.0 years	Half of passengers were younger than 28
max	80.0	Oldest passenger was 80

Fare column — the red flag:

Stat	Value
mean	£32.20
median (50%)	£14.45
max	£512.33

Key insight: Mean (£32.20) is more than double the median (£14.45). When mean >> median, you have right skew — a few extremely expensive tickets are pulling the average way up. This matters when we build our model later, because many ML algorithms assume features are roughly normally distributed.

Let's understand the meaning of these terms:
Mean (Average): The sum of all values divided by the total number of values. It is the most common measure of center but is highly sensitive to outliers (extreme values).

Median: The middle value when the data is sorted from smallest to largest. If there is an even number of observations, it is the average of the two middle numbers. Unlike the mean, the median is robust, meaning it isn't easily skewed by a few very high or very low numbers.

Mode: The value that appears most frequently in a dataset. A dataset can have one mode, multiple modes (bimodal or multimodal), or no mode at all. This is particularly useful for categorical data (e.g., finding the most common car color in a dataset).

Standard Deviation (Std):
This measures the average distance of each data point from the mean.

Low Std: Data points are close to the mean (the data is consistent).
High Std: Data points are spread out over a wide range (the data is volatile).

In machine learning, standard deviation is used in feature scaling (like Standardization) to ensure that features with different scales (e.g., Age vs. Annual Income) don't bias the model.

Term	What it tells you	Sensitivity to Outliers
mean	The mathematical center	High
median	The physical middle	Low
mode	The most popular value	Low
std	The "spread" or risk	High

Step 4.1: Check your categorical columns too

describe() only covers numeric columns by default. Don't forget the string columns.

# Count unique values per categorical column
for col in ['Sex', 'Embarked', 'Pclass']:
    print(f"\n{col}:")
    print(df[col].value_counts())

Output:

Sex:
 male      577   # (64.7%)
 female    314   # (35.3%)

Embarked:
 S    644   # Southampton (UK)
 C    168   # Cherbourg (France)
 Q     77   # Queenstown (Ireland)
 NaN    2   # (missing — we'll fix this in Part 3)

Pclass:
 3    491   # lower class (55% of all passengers!)
 1    216   # upper class
 2    184   # middle class

Step 5: isnull().sum()

Definition: This is a chained command. isnull() creates a boolean mask (True/False) for missing values, and .sum() adds up those Trues for every column.

Why it matters: Missing data is the enemy of Machine Learning. This command creates a Missing Data Report, telling you exactly where the holes are so you can decide whether to fill them (imputation) or drop them.

df.isnull().sum()

Output:

PassengerId      0
Survived         0
Pclass           0
Name             0
Sex              0
Age            177
SibSp            0
Parch            0
Ticket           0
Fare             0
Cabin          687
Embarked         2
dtype: int64

What this already tells us — the iceberg analogy

Like the Titanic itself, we're only seeing the surface so far. But even from these basic checks, we've already learned:

64% male passengers — the Titanic famously applied women and children first, so Sex is almost certainly a strong predictor
55% were in 3rd class — lower class passengers had less access to lifeboats
Most boarded at Southampton — this could be correlated with class
Age has 20% missing — we need a strategy beyond simple mean imputation

The iceberg analogy: EDA is already making us smarter before we write a single model line. We now have context — and context is how you catch the mistakes that plain accuracy scores hide.

Quick recap: what we found in 5 lines of code

Finding	What it means
Age stored as float	Has missing values — needs imputation
Fare: mean >> median	Right-skewed — consider log transform
Cabin: 687 nulls	77% missing — can't impute, need different strategy
Sex: 64% male	Class imbalance in a key feature
Pclass 3: 55% of passengers	Most passengers were lower class

What's next?

In Part 3, we plot distributions for every important column with histograms and box plots, understand what's actually inside the Age and Fare distributions, and decide how to handle that 77% missing Cabin column.

That's where the real visual EDA begins. See you there! 👋

[Boost]

Shivappa — Fri, 17 Apr 2026 05:43:09 +0000

Shivappa

Apr 16

(EDA Part-1) EDA (Exploratory Data Analysis) Explained With Real Life — Why Looking at Your Data Is the Most Important Step in Machine Learning

#machinelearning #ai #titanic #pandas

Comments

4 min read

(EDA Part-1) EDA (Exploratory Data Analysis) Explained With Real Life — Why Looking at Your Data Is the Most Important Step in Machine Learning

Shivappa — Thu, 16 Apr 2026 06:27:32 +0000

Part 1 of 5 — Beginner friendly

Let me tell you a story about a Detective.

Sherlock Holmes never walks into a crime scene and starts guessing. He doesn't say "it was probably the butler" and call it done. He looks. He observes. He notices the mud on the shoes, the cigarette stub, the clock stopped at 11:47. Only after he's observed everything does he form a theory.

EDA is what Sherlock does before naming a suspect. Your dataset is the crime scene. Your ML model is the theory. You don't build the theory until you've studied the scene thoroughly.

Most beginners skip this step. They load data, run a model, get 90% accuracy, feel great, and ship it. Three weeks later the model breaks in production and nobody knows why.

That failure almost always traces back to one thing: they never looked at their data properly.

What is EDA?

EDA stands for Exploratory Data Analysis. It's the process of examining, understanding, and summarising a dataset before you train any model. Think of it as your first conversation with the data — you're not asking it to predict anything yet. You're just getting to know it.

The real-world example:

Imagine a doctor who skips the examination and jumps straight to prescribing medicine. They don't check your blood pressure, don't run tests, don't ask about symptoms. Dangerous, right? EDA is the medical examination. Your model is the prescription. Don't prescribe before you examine.

The term was coined by statistician John Tukey in 1977. His insight was simple but radical: stop assuming things about data. Let the data tell you its own story first.

The flow looks like this:

The EDA mindset

Before writing a single line of code, ask these 4 questions about any new dataset:

What is this Data ? What does each column mean in real life ? Don't assume - read the data dictionary.
What is the quality ? Are there missing values? Duplicates? Wrong data types? Values that look like errors?
What are the distributions What does each feature look like? Skewed? Balanced? Are there obvious outliers?
What are the relationships Which features are correlated with each other? Which features predict the target?

Three pillars of EDA

Summary Statistics: Getting the vitals (Mean, Median, Mode, Standard Deviation). If the terms are new for you don't worry, you will learn more on this in next parts of this series.
Data Visualization: Turning rows of numbers into charts (Histograms, Scatter plots, Box plots) to see what the human eye misses in a spreadsheet.
Data Cleaning: Identifying missing values, duplicate entries, or garbage data that could skew your results.

Why is EDA Crucial for Machine Learning?

You’ve probably heard the phrase Garbage In, Garbage Out. This is the primary reason we do EDA. Even the most sophisticated algorithm will fail if the data it learns from is flawed.

A. Ensuring Data Quality
EDA helps you find the weirdness. Is there a person in your dataset who is 250 years old? Is there a house with 0 bedrooms? EDA catches these anomalies before they confuse your model.

B. Feature Selection & Engineering
By looking at correlations, you can see which variables actually matter. If you’re predicting house prices and find that the color of the front door has zero correlation with the price, you can drop that noise and make your model leaner and faster.

C. Testing Assumptions
Many ML models (like Linear Regression) rely on specific assumptions—for example, that your data follows a Normal Distribution. EDA allows you to check these assumptions. If your data is heavily skewed, you’ll know you need to transform it first.

D. Building Intuition
As a data scientist, you need to be the bridge between the raw data and the business stakeholders. EDA gives you the story behind the numbers, allowing you to explain why a model might be making certain predictions.

The `Cost` of Skipping EDA

Skipping this step is a recipe for disaster. Without EDA:

1. Biased Models: You might train a model on data that isn't representative of the real world.

2. Overfitting: You might "learn" from outliers that are actually just data entry errors.

3. Wasted Time: You’ll spend weeks tuning a model only to realize the data was missing the most important variable all along.

Pro Tip: EDA isn’t a one-time chore; it’s an iterative process. You’ll find yourself coming back to it even after you’ve started building your models.

The dataset we'll use throughout this series

We'll use the Titanic dataset from Kaggle — one of the most famous datasets in data science. It's real historical data about passengers and crew who were aboard the RMS Titanic, which sank on April 15, 1912 resulting in the death of 1502 out of 2,224 passengers and crew — a widely used sample provided by Kaggle for the competition.

Here's what makes it perfect for EDA: it has missing values, mixed data types, categorical and numeric features, and a clear survival target to predict.

What's next?

In Part 2, we download the Titanic dataset from Kaggle and run our first 5-line EDA check. No complicated setup — you can run it directly in a free Kaggle notebook or google collab.

See you in Part 2! 👋

Access Key Vault secret from Web Application

Shivappa — Sun, 09 May 2021 08:58:02 +0000

In the previous parts we have learnt how to create Azure Key Vault and how to add the principal in Azure Key Vault by whom it must be accessed.
In this part, we will learn how we can use secrets stored in Azure Key Vault in our Web Application.

Let's write a simple flask app to fetch the secrets from Key Vault.
Create a folder called KeyVaultDemo on your computer and open it in VS Code.
Create a new file requirements.txt which contains the packages required.
Add below contents to it.

flask
azure-keyvault-secrets
azure-identity

azure-keyvault-secrets: It is a client library in python to access the secrets stored in Azure Key Vault. More information on this package can be found here.
azure-identity: It is an authentication library that provides different credential classes with which one can authenticate with the Active Directory.
To learn more on this package please check here.

We have added the packages required for our purpose.
Let's write actual code in which we can access the Key Vault secrets.
Create app.py file and dump the below contents to it.

import os

from flask import Flask

from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient

app = Flask(__name__)

# Save KEY VAULT URL in the web app application settings
AZUE_KEY_VAULT_URL = os.environ.get('AZUE_KEY_VAULT_URL', '')
# AZUE_KEY_VAULT_URL = "https://keyvaultdemoapp.vault.azure.net/"

@app.route('/secret')
def index():
    """Fetch secret from Azure Key Vault

    Returns:
        str: Response message to be displayed.
    """
    # Use Default credential. This has fallback authentication mechanism
    credential = DefaultAzureCredential()
    # Authenticate to Azure Key Vault.
    secret_client = SecretClient(vault_url=AZUE_KEY_VAULT_URL, credential=credential)
    # Fetch secrets created in Azure Key Vault
    secrets = secret_client.get_secret('firstsecret')
    name = secrets.name
    value = secrets.value
    # Show secret name and it's value in the response
    response = f'secret {name} value is {value}'
    return response

Let's understand few lines from the above code snippet.

from azure.identity import DefaultAzureCredential

Azure identity helps to authenticate from the credentials we choose. Here we selected credential type as DefaultAzureCredential.
This is the most used credential type in most cloud applications because of the fallback mechanism supported.
First, it will try to check for Environment variables saved for the web application. If not then it will try to check in this order of authentication --> Managed Identity --> Visual Studio Code --> Azure Cli --> Interactive
Below Environment variables are required for Identity and must be added in Application settings in the web app deployed.

AZURE_CLIENT_ID --> This is the ID of an Azure Active Directory application
AZURE_CLIENT_SECRET --> This is the ID of application secrets created in Azure Active Directory.
AZURE_TENANT_ID --> This is the ID of the Active Directory Tenant.

To learn more on each method please look at the client library here.

Our code is ready and we will deploy it to Azure and access the application.
Right click on Explorer area in VS Code and click on Deploy to Web App...

Select your subscription.
Select Create new Web App..
Give a name to the web app. It should be Globally Unique name.
Select Runtime stack as Python 3.8
Select pricing tier as Free

It takes some time to deploy and you will see a successfully deployed message on the right bottom corner of VS Code.
Go to the Azure portal and go to the app created now and click on browse from the overview page.
It shows an error because we have not added the app environment in the app settings that we are accessing and using inside our flask app.

Let's add them to the Azure web app.
Go to Web App you just created.
Click on Configuration section under Settings.
Click on New application setting.
Add below Key value pair when prompted for entries.

Key : AZURE_CLIENT_ID
Value: Value which you copied when we created the authentication settings in the previous part

Repeat the above operation for the below Key addition.

AZURE_CLIENT_SECRET : corresponding value
AZURE_TENANT_ID : corresponding value
AZURE_KEY_VAULT_URL : This is the URL of the key vault created in this part. Go to Overview and on the right side you will see the keyvault URL.

Once you add all the entries, Save the settings.
The final application settings look like this web app configuration.

Now browse your application and see the secret name and secret value stored in Key Vault are displayed.

We have learned how to use Azure Key Vault in your web application for storing the secrets securely.

To learn more on Azure Key Vault please refer to Microsoft documentation here.

That's all!
Thank you for reading my blog.
Stay tuned to read more blogs on Azure services.

Azure Key vault authentication

Shivappa — Sun, 09 May 2021 06:39:59 +0000

In the previous part we have learnt how to add Azure Key Vault resource. Now we will learn how to add the authentication to the Key vault to access it.
Let's create a access-policy(principal) to access the store.
To add the access-policy we have to make the Authentication with the Active Directory.
Let's see how we can authenticate.
Go to resources and search for ADB2C.
Read more about ADB2C here.

Register new application in ADB2C to get the CLIENT_ID, CLIENT_SECRET and TENANT_ID.
Go to App registrations and click on "New registration*.

Name this application and select Single tenant option.
Click on Register to register this application.

Go to the new app created and note down the Client ID and Tenant ID which we require in our web application.

In addition, we need to add the secrets and use them.
create new secrets and note that also for later use.

Now we have successfully created the identity which can be used in Azure Key Vault for access.

Go to the created Azure Key Vault resource and click on
Access policies and select Add Access Policy.

Select the template from the drop-down. Key, Secret, and Certificate Management. If you want to select only secret you can do so.
Select Key Permissions which can be allowed for this access if Key Management has opted.
Select Secret Permissions which can be allowed for this access if Key Management has opted.
Select Certificate Permissions which can be allowed for this access if Key Management has opted.
select Select Principal and search for the application which we created in ADB2C in the above steps.

We have authenticated our key vault by whom it can be used.
Let's write a simple flask app to fetch the secrets from the key vault in the next part of this series.

Understand and use Azure Key Vault Easily (Most Secured Secrets Storage)

Shivappa — Sun, 09 May 2021 06:34:11 +0000

Now a days the term security becoming the must follow topic for all the organizations to have a good brand name.
To secure your credentials or secrets in your web application is a big challenge when we deploy our application to the public internet.
And also once your code is deployed and secrets are revealed then one need to redeploy the code with updated secrets to secure the application. As the application grows bigger and bigger this will be cumbersome to manage the application.

To solve this issue we can use Azure Key Vault resource.
Azure Key Vault is a Microsoft cloud service to store and access data and secrets securely. Which controls below tasks,

Securely store the keys, secrets and certificates.
Only authorized user can access the key vault.
Frequently rotate the secrets, keys.
Automatically renew the certificates.
Centralized storage for all keys, secrets and certificates.

Azure Key Vault can be used in many services, here I will be showing how to use it in Azure app service.

Prerequisites:

Active Azure Subscription Account
Visual Studio Code

We can create Azure Key Vault using Azure CLI or Azure Portal.
In this blog we will look at how to do in Azure Portal.

Create Azure Key Vault using Azure Portal

Create an Azure resource

Login to Azure Portal.
Click on create a resource.
Search for Key Vault. Select and create it.
Fill details and create the resource.

Resource Group: Select existing resource group or create a new one

Key vault name: Enter globally Unique name for your key vault. This is access from the application.

Region: Select location the one near to you.

Pricing tier: Select Standard.

Review + create and then create.
Go to Key vault resource created.
Let's create some secrets in the key vault which will be accessed from our application.
Click on Generate/Import to add new secret.

Fill the details and add it.
Name: name of the secret to be created.
value: Value to store in this secret.
We can set when this secret can be activated and deactivated.
As of now we can leave default, meaning immediate activate and never expires.

After the secret is added go the secret which looks like this.

Our secret is successfully created in Azure Key Vault. Now we have to access it from our code.
To access the secrets, valid permissions (like who can access) must be added to the keyvault.
Let's learn how to add the authentication in next part of this series.

Deploy Containerized Web App to Microsoft Azure | Upload/Download files of Azure Storage Account

Shivappa — Fri, 02 Apr 2021 15:25:31 +0000

If you are bored to read my store you can skip quoted text and go to technical content below.

Few months back I got a chance to work in cloud project.
Since this is completely new to me and had lot of responsibility.
Then I started searching over the internet about Microsoft azure and the services it provides. Really there are plenty of services in the Azure which helps to build projects to the great extent.
Then I started googling about the services what they do and how can be used. Every time I google I was going through blogs, some are valid and easy to understand but some are not.
One day I got a nice tutorial in Microsoft website and found it has rich in documentation about the Azure. After I studied that topic, myself started exploring in the Microsoft documentation and found that it has Microsoft Learn page where it supports all the documentation on all the services it supported and it has all the levels of content from beginner to expert level.

You can learn and achieve your expertise in various topics from Microsoft Learn.
Each module provides exercises and knowledge check questions based on the topic you learnt.
And another great thing MSLearn provides for hands on is Azure sandbox subscription(You can activate during each module learning and exercises ).
This is really great for quick hands on the things we learn. This is strictly for the study purpose on the learning modules and will be deleted after the timer(You should not use for other purposes).

Later I found some online events are being conducted by Microsoft professionals and explored them. You can find very helpful and priceless content from these events. Please visit here for full list of Microsoft Virtual events.
As I was reading about Azure certification courses, I have got excited and attended AZ-900 Azure fundamentals(basic level) virtual training and it was really nice. This event sparked me to attend many more events and learn Azure. Later I passed AZ-900 certification with only documents from this MSLearn.

As I was studying for the various services on the Azure in MSLearn, I got to know there is an online event Cloud Community Days in KonfHub is going to happen. Without a bit delay I registered for the 2 Days conference and I attended the same. You can join your nearest Azure Developer community.
It was really thoughtful conference in which many things I have learnt and could clarify many doubts which I had.
The content throughout was good and there were quiz sessions in between. On first day I could not attend in quiz but attended second day's first two quizzes and in the first attempt I ranked 12th and in the next I ranked 3rd. This pushed me more to learn and master the learnings from the Microsoft and attend many more such events.
Later I took "Azure Dev Challenge" and completed effective study and as a next step Azure Developer Stories contest I planned to write a blog of my steps towards Microsoft Azure.

This is my first blog ever. Thank you Microsoft and KonfHub for this motivation.

I think it is enough to stop my story and share some technical details which I built after learning Azure Dev Challenge modules.

Project Statement

Deploy a Containerized Flask Web Application which translates the given text to another language and dump the contents in a file. This file will be uploaded to Azure Storage Account in a secure way. Once it is uploaded we can download the file if we have valid SAS token for the file.

We will divide above project statement into below steps,

Create Azure Container Registry(ACR).
Write Flask Application to
- translate text using Azure Cognitive Services
- upload/download files to/from Azure Storage Account.
Deploy containerized flask application to Azure Container Registry.
Check logs of our container application.

Prerequisites

Visual Studio Code IDE.
Microsoft Azure Subscription. Get a free account if you do not have one.
Latest Python version.

Step 1: Create Azure Container Registry(ACR)

Before creating Azure Container Registry, we will understand some terms.
Container Image: It is a bundled software package, which contains system libraries, tools and other platform settings with which a program can run without any changes to software package at run time.
Container Registry: It is a repository to store container images to run container based applications.

Now let's create Azure Container Registry.

Login to your Azure Portal
Click on add resource to create new Resource Group.

Search for "resource group" in the Azure Marketplace and click on it and create.

Add new resource group with required details.

Subscription: Select your valid Azure subscription where you want to create resource group.
Resource group: Name of the resource group to be created.
Region: The location where you want to create resource group.(select your nearest region)

Once the resource group is created. Go to the Resource Group created in above step.
Now create a container registry resource in this Resource Group.
Click on "Add"

Search for "container registry" in the Azure Marketplace and click on it and create.

Fill all the required details.

Subscription: Select your valid Azure subscription where you want to create container registry.
Resource group: Select the resource group which you created in previous step.
Registry name: Name of your Azure Container registry to be created.
Location: The location where you want to create ACR (Keep same as the resource group Region).
SKU: Select the registry tier Basic. (Available are Basic, Standard, Premium)

Then Review + create and Create.
After the Azure Container Registry is created, open and update admin user setting.

Enable under Admin user, this is required to deploy web application directly from container registry.

Now we successfully created Azure Container Registry.
It is time to write some code.

Step 2: Write Flask Application

We will be using Visual Studio Code code editor for all our coding. A best code editor with vast features.
Now on I will be referring Visual Studio Code as VS Code.

Download and install VS Code based on your OS. (Can be used on Windows, MacOS, Linux).
Install latest python version from python.org

Create a folder Azure-Dev-Stories in your computer and open the folder in VSCode.
Open cmd and run below commands. (I am using windows. If you are using other OS please replace corresponding commands)

md Azure-Dev-Stories cd Azure-Dev-Stories code .

(code . opens VSCode)
Now VSCode is opened and you can see empty folder like this.

Our environment for coding is ready.

To isolate our project we will create virtual environment.
Open VSCode terminal (Ctrl + `) and execute below command.

python -m venv .venv

To make containerized image of our code we need to install below extensions from VSCode marketplace.

Azure App service, Azure Account extension (For Azure services)
Docker extension (For dockerizing the application).

Search for Azure account and docker in marketplace and install them as shown in below screens.

Install Docker extension.

After both the extensions are installed your VS Code looks similar to this.

Go to Azure extension and login to your Microsoft Azure account.

Now create a file app.py and add below contents in it and save the file.

from flask import Flask

app = Flask(__name__)

Now we need to create our first Dockerfile.

Dockerfile: It is a text file which contains all the steps to be performed to create our application as a image.

Go to command palette (Ctrl + Shift + P) in VSCode and type build image, select Azure Container Registry: Build Image in Azure... from the search list.

You will be asked to add a Dockerfile to the workspace on bottom screen, select Yes.

Select Python:Flask (select other relevant application platform if you are building for other frameworks)
Then select app.py as entry point and select port 5000 and select Yes to add the docker compose files.
After the above steps your VSCode folder looks like this.

Newly created Dockerfile looks like this.

# For more information, please refer to https://aka.ms/vscode- docker-python
FROM python:3.8-slim-buster

EXPOSE 5000

# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE=1

# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1

# Install pip requirements
COPY requirements.txt .
RUN python -m pip install -r requirements.txt

WORKDIR /app
COPY . /app

# Creates a non-root user and adds permission to access the /app 
 folder
# For more info, please refer to https://aka.ms/vscode-docker- python-configure-containers
RUN useradd appuser && chown -R appuser /app
USER appuser

# During debugging, this entry point will be overridden. For more 
information, please refer to https://aka.ms/vscode-docker-python- debug
CMD ["gunicorn", "--bind", "0.0.0.0:5000", "app:app"]`

Now go to requirements.txt file and add below packages.

flask
gunicorn
python-dotenv
python-dateutil
requests
azure-storage-blob

We set what packages are required for our project and we need to install them in our virtual environment.
Go to VS Code terminal and run below command to install the packages.
pip install -r requirements.txt

It takes some time to install all the packages.

Now our bare flask application framework is ready and we need to deploy it to Azure as web application to make sure our application is running.

Step 3: Deploy containerized flask application to Azure Container Registry

Open command palette (Ctrl + Shift + P) and search build image and select Azure Container Registry: Build Image in Azure... from the search list.
Now give a name to image. With this name it will be pushed to Azure Container Registry.

Replace .Run.ID with master to keep our image tag always the master.

azure-dev-stories:master

Select the Azure Container Registry from the list which we have created before.

Choose Linux. (You can choose other OS for which you want to make the image)

After the image is successfully created you will see output like this.

Confirm whether the image is present in Azure Container Registry(ACR).
Go to Docker extension and see the steps in below image.

Our application image is built and is present in ACR. Now we need to deploy as Azure web application.
Right click on master and select Deploy Image to Azure App Service....

Enter Unique global name for your application. If similar name is exists, it shows error and choose different name. This name must be unique across global azure web applications, because our deployed application is accessible over the internet.
Select the Resource group which we already created during ACR creation(it shows the list of resource groups).
Create App Service Plan for the application.
Give a name to Service Plan (the agent on which our application runs).
Select pricing tier F1 Free.
Select the Location for the Service Plan. After above steps you will see deploy message at the bottom screen. Once the web application is deployed you will see a message Successfully create web app ....

Open the site and check our application is up and running.
We see our bare app like this.

Now go to Azure Portal and check image tag is applied properly.
To check it, go to Azure home search bar, search for app services and select the web application deployed.

Go to Deployment Center -> Settings and confirm Registry, Image, Tag(master) are fine.

Now we successfully deployed flask web application as containerized App in Azure. And we need make valuable use from the web application.
So our next step is,

translate text using Azure Cognitive Services

To translate the user input to another language we will use Azure Cognitive Services.
Now go to Azure Portal.
Create a resource -> search for translator service and install it.

Fill required fields.

Subscription: Valid subscription.
Resource group: select resource group created before.
Region: Select nearest region
Name: give a unique name to translator service
Pricing tier: Free F0, (You can choose higher plan also based on your requirements)

Then Review + create and Create.

Go to the translator service created, note down KEY1 or KEY2, Endpoint and Location, we need them to input in our flask application later.

We created Azure translator and we need to use it in our application to get the translated text.
Go to VSCode. Create .env file and add below details.

KEY=0b5c31e83e36459f953b097d01931143
ENDPOINT=https://api.cognitive.microsofttranslator.com/
LOCATION=eastus

KEY : Translator service key1 or key2
ENDPOINT: Translator Endpoint
LOCATION: Translator service Location

Note: When we created Dockerfile, but silently .dockerignore file also created. This contains the list of files which we need ignore while building the docker image.
Look careful all contents, it has .env also. But we must need this file to get our translator service details.
So remove the line containing .env.

We need to ignore our virtual environment libraries which are present in .venv. Add **/.venv line in .dockerignore.

For the docker to refer our environment variables properly, we need to add .env file in docker-compose.yml.
Open docker-compose.yml file and add section env_file after ports like this(take care of the indentation. ports and env_file should be on same level).

version: '3.4'

services:
  azuredevstories:
    image: azuredevstories
    build:
      context: .
      dockerfile: ./Dockerfile
    ports:
      - 5000:5000
    env_file: 
      - .env

Now we are done with all the configuration. Let's write some code.
Copy paste below code in app.py.

import logging
import os
import requests, uuid

from flask import Flask, render_template, request

from dotenv import load_dotenv
load_dotenv()

app = Flask(__name__)

# Setup logging to get the logs from the application
level = logging.DEBUG
format = f'%(asctime)s %(levelname)s %(name)s : %(message)s'
logging.basicConfig(filename='app.log', level=level, format=format)

@app.route('/', methods=['GET'])
def index():
    return render_template('index.html')

@app.route('/', methods=['POST'])
def index_post():
    """translate given string and put in file and upload to Azure storage account"""
    # Read values from the form
    original_text = request.form['text']
    target_language = request.form['language']

    # Load values from .env
    key = os.environ.get('KEY', '')
    endpoint = os.environ.get('ENDPOINT', '')
    location = os.environ.get('LOCATION', '')

    # Construct the translator method
    path = '/translate?api-version=3.0'
    target_language_parameter = '&to=' + target_language
    constructed_url = endpoint + path + target_language_parameter

    # Set up header for post request
    headers = {
        'Ocp-Apim-Subscription-Key': key,
        'Ocp-Apim-Subscription-Region': location,
        'Content-type': 'application/json',
        'X-ClientTraceID': str(uuid.uuid4())
    }

    # body of request
    body = [{'text': original_text}]

    # Make call using post
    translator_request = requests.post(constructed_url, headers=headers, json=body)
    # Retrieve JSON response
    translator_response = translator_request.json()
    # Retrieve translation response
    translated_text = translator_response[0]['translations'][0]['text']

    return render_template(
        'results.html',
        original_text=original_text,
        translated_text=translated_text,
    )

Create templates folder in current workspace directory.
Inside templates folder, create index.html with which we can receive the input and send it to Azure translation service from our flask application.
Copy paste below code in index.html.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootstrap@4.5.3/dist/css/bootstrap.min.css"
        integrity="sha384-TX8t27EcRE3e/ihU7zmQxVncDAy5uIKz4rEkgIXeMed4M0jlfIDPvg6uqKI2xXr2" crossorigin="anonymous">
    <title>Translator</title>
</head>
<body>
    <div class="container">
        <h1>Translation service</h1>
        <div>Enter the text you wish to translate and upload to Azure</div>
        <div>
            <form method="POST">
                <div class="form-group">
                    <textarea name="text" cols="10" rows="2" class="form-control"></textarea>
                </div>
                <div class="form-group">
                    <label for="language">Select Translation Language</label>
                    <select name="language" class="form-control">
                        <option value="en">English</option>
                        <option value="it">Italian</option>
                        <option value="ja">Japanese</option>
                        <option value="ru">Russian</option>
                        <option value="de">German</option>
                    </select>
                </div>
                <div>
                    <button type="submit" class="btn btn-success">Translate and Upload !</button>
                </div>
            </form>
        </div>
    </div>
</body>
</html>

Inside templates folder, create results.html and copy paste below code.
This will display the results once the input is translated.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootstrap@4.5.3/dist/css/bootstrap.min.css"
        integrity="sha384-TX8t27EcRE3e/ihU7zmQxVncDAy5uIKz4rEkgIXeMed4M0jlfIDPvg6uqKI2xXr2" crossorigin="anonymous">
    <title>Upload</title>
</head>
<body>
</br>
    <div class="container">
        <h2>Translation results</h2>
    </br>
        <div>
            <strong>Input text :</strong> {{ original_text }}
        </div>
        <div>
            <strong>Translated text:</strong> {{ translated_text }}
        </div>
        <div>
            <a href="{{ url_for('index') }}">Try another one!</a>
        </div>
    </div>
</body>
</html>

Our final folder structure looks like this.

Now our application is ready to make translations. Run locally and confirm there is no issue.
flask run

But we have not yet deployed it Azure, so it is time to re-build and deploy it to Azure.
Go to command palette (Ctrl + Shift + P).
Search build image and select Azure Container Registry: Build Image in Azure...,
Simply click enter key in all steps and keep all settings without change. (make sure all are valid and recent settings).
Image build starts and once it is completed you will see the results under Output in terminal section.
Go to docker extension, refresh and confirm the image tag time is changed (a few seconds ago).

Wait for sometime to docker image apply on the web application. Now open already deployed web application website https://mydockerdemotest.azurewebsites.net/ and check changes are reflected.

You can also open the web application in App services -> Select your application -> Overview -> Browse.

Website looks like this.

Enter input text to be translated and select the language of translation. Click Translate. We have the translated text like this.

Wow!, now our application is able to translate the text.

Now we will write this translation text to a file and upload to Azure Storage Account and download whenever we want.
That is a great service provided by Azure to store our files in a more secure way.
Let's learn on how to upload and download the files in Azure service.

Use Azure Storage Account to securely upload and download files.

Azure Storage account: It contains all our Azure storage objects like blobs(containers), queues, tables and disks. This account can be accessed over the internet using the unique namespace which we will be doing in this section.
Let's begin...
Again go to Azure Portal and install Azure Storage Account service.
Here are the steps.
Create new resource.

search storage account in azure marketplace and create the service.

Fill the required fields.

Subscription: Valid subscription.
Resource group: select resource group created before.
Storage account name: give a unique name to storage account
Location: Select nearest location
Performance: select Standard
Account kind: Default StorageV2 (general purpose v2)
Replication: Default Read-access-geo-redundant storage (RA-GRS)

Click Review + create then create to create the service.
Go to the created resource and fetch the connection details of the storage account.
Select Access keys and click on Show keys. It displays the key and connection string.

Copy and note down the key and Connection string which is required to put in our code to upload the file to this storage account.

Let's create a container in which we will be storing our files.
Go to overview and select containers.

Create a new container.

Give a name to container and select private from the dropdown because nobody else should not access our storage account.

Our storage container is created. Now let's go to code part from which we can upload the files to this container.

Go to VSCode.
As we already copied the connection string, copy it to .env file along with our container name and storageaccount name which we already created.
Our complete .env file looks like this.

KEY=0b5c31e83e36459f953b097d01931143
ENDPOINT=https://api.cognitive.microsofttranslator.com/
LOCATION=eastus
STORAGEACCOUNT_CONNECTION_STRING=DefaultEndpointsProtocol=https;AccountName=storageaccounttranslator;AccountKey=oJrELlq6ts0GGrvBVCISsARadZL0ozRPLF+m2afgRjz5mTcnV9TzmpDAFOQqPcZ0u+8OW90Qp35k5842QkB95w==;EndpointSuffix=core.windows.net
CONTAINER_NAME=translations
STORAGE_ACCOUNT_NAME=storageaccounttranslator
SASTOKEN_LIFE=12

STORAGEACCOUNT_CONNECTION_STRING : Connection string of the the storage account which we copied.
CONTAINER_NAME : Name of the container we created.
STORAGE_ACCOUNT_NAME : Our storage account name.
SASTOKEN_LIFE : How long sas token is valid for the file.(in months)

Now we need to write code to upload the translated file to storage account.
Go to appy.py file and add below methods.
upload_storageaccount and generate_sastoken()

def generate_sastoken(filename, saslife, connections):
    container_name = connections.get('container_name')
    connection_string = connections.get('storageaccount_connection')
    try:
        blob_service_client = BlobServiceClient.from_connection_string(connection_string)
        # generateToken with expiry
        sas_token = generate_blob_sas(blob_service_client.account_name, 
                container_name=container_name, blob_name=filename,
                account_key=blob_service_client.credential.account_key,
                permission=AccountSasPermissions(read=True),
                expiry=datetime.now() + relativedelta(months=saslife))

        print(f'sas token generated: {sas_token}')
        return sas_token
    except Exception as e:
        print(f'failed to generate token for file {filename} error: {e}')


def upload_storageaccount(infile, inpath, connections):
    connection_string = connections.get('storageaccount_connection')
    container_name = connections.get('container_name')
    account_name = connections.get('account_name')
    try:
        blob_service_client = BlobServiceClient.from_connection_string(connection_string)
        blob_client = blob_service_client.get_blob_client(container = container_name, blob = infile)

        # Upload the file data
        filepath = inpath + '/' + infile
        with open(filepath, 'rb') as data:
            blob_client.upload_blob(data, overwrite = True)

        url = f"https://{account_name}.blob.core.windows.net/{container_name}/{infile}"
        print(f'File {infile} uploaded to {url}')
        return url
    except Exception as e:
        print(f'failed to upload the report {infile} to {inpath}')

and call these methods from index_post() after the text is translated.
Update results.html to display the storage path and sas token with which we can download the file.

        <div>
            <strong>Storage file path:</strong> {{ storagepath }}
        </div>
        <div>
            <strong>SAS Token:</strong> {{ sas_token }}
        </div>

For complete code please refer my GitHub repository.

Now re-build the container image and check the results.
Go to command palette (Ctrl + Shift + P) in VSCode and type build image, select Azure Container Registry: Build Image in Azure... from the search list.
and follow remaining steps from our last re-build step.
Go to application website and check whether latest changes are reflected.
Enter input for translation text and language then click Translate and Upload!
After successful translation and upload the file we will get output like this.

Now go and download the file in new browser tab using below url combination.
<Storage file path>?<SAS Token>

Sample URL
https://storageaccounttranslator.blob.core.windows.net/translations/translated_file_ja.txt?se=2022-04-02T07%3A02%3A42Z&sp=r&sv=2020-06-12&sr=b&sig=%2BK83mBfp2Oun%2Bs0O8U0YTuJaHxOnyItEtdrUsrmpTO4%3D
Try to download file without SAS Token, you will get
error like this.

So it is sure that completely secured in storage account. Without SAS token file cannot be downloaded using bare url.

Also we can confirm whether the file is present in the storage account container. Go to Azure Portal -> Storage Account -> Container -> Select container. Here our uploaded files will be present.

If we encounter any issue in our application. How to check for it ? There is a solution in Azure service too. Let's look into it.

Step 4: Check logs of our container application.

In this step we will see how to check the logs of our container application.
Go to Azure Portal -> Search -> App Services -> Web application we created.

If we want to look at the previous logs on application crash, container crash or any other issues we can go to Diagnose and solve problems, here it displays the detailed information.
If we want to see the live log to Log stream. Here we can see whether our container application is started or encountered any issues.

Note: You might be wondering, we have stored the secrets in .env how to protect them. Yes! there is a solution in Azure called Azure Key Vault. I will be writing a blog on it soon.

Now we reached the final step.

We have created many Azure services so far in this blog, what happens if we leave them ? As Azure says Pay As You Go! means if we leave our resources it will be chargeable. So we must clear the resources which we have used.

OK! let's delete them.
Go to Azure Portal -> select our created Resource Group -> Delete the resource group. This deletes the entire resource group which contains many services.
If you want to delete only specific service, you do so too.

That's all !!!!!

Conclusion:

Thank you for your time in reading my first blog.
We learnt how to use Microsofts's Visual Studio Code, Azure Cognitive Services, Azure Storage Account, Azure web application, Containerizing web app, Azure Container Registry, Flask framework, etc. I hope it helped you to gain some knowledge on Azure services.
Your comments/suggestions are valuable for me to learn more.

Forem: Shivappa

(EDA Part-5) Multivariate Analysis — Wrapping Up EDA and What Comes Next

The analogy — the blind men and the elephant 🐘

So, what exactly is multivariate analysis?

Step 1: Three-way survival breakdown — Sex + Pclass + Survived

Step 2: Add Age into the picture — the FacetGrid

Step 3: Engineering features—extracting Title from Name**

Step 4: Family size — creating another multivariate feature

Step 5: Updated correlation heatmap with engineered features

The complete EDA journey — all 5 parts in one view

The final definition of EDA

What is coming next — the ML algorithms series

Thank you 🙏

(EDA Part-4) Bivariate Analysis — What Actually Predicted Who Survived (on Titanic)?

The analogy — the detective reveals the suspect 🔍

What does the real Titanic data say?

Step 1: who survived? Feature vs target

Step 2: Violin plots — see the full distribution, not just the average

Step 3: The correlation heatmap — who are the strongest predictors?

Step 4: The crosstab—the two-line powerhouse

Step 5: Point biserial correlation for binary targets

So, what did we learn?

What is next?

(EDA Part-3) Univariate Analysis — Understanding Every Feature One at a Time

The analogy: the fruit inspector

The Numbers: Age and Fare

Analysing categorical features

The 3 types of missing data — This is where so many data projects go wrong

The analogy — three types of absent students

Which type are the Titanic nulls?

Fixing the missing values correctly

The box plot check — spotting outliers

Quick recap: univariate findings

What is next?

(EDA Part-2) First Look at the Titanic Dataset — Loading Data and Understanding Big 5

The analogy before we start — first day at a new job

The Quick-Start Routine

The Big 5 — run these on EVERY new dataset

Step 0: import required library and load the dataset.

Step 1: Shape

Step 2: dtypes

Step 3: head()

Step 4: describe()

Step 4.1: Check your categorical columns too

Step 5: isnull().sum()

What this already tells us — the iceberg analogy

Quick recap: what we found in 5 lines of code

What's next?

[Boost]

(EDA Part-1) EDA (Exploratory Data Analysis) Explained With Real Life — Why Looking at Your Data Is the Most Important Step in Machine Learning

(EDA Part-1) EDA (Exploratory Data Analysis) Explained With Real Life — Why Looking at Your Data Is the Most Important Step in Machine Learning

What is EDA?

The EDA mindset

Three pillars of EDA

Why is EDA Crucial for Machine Learning?

The Cost of Skipping EDA

The dataset we'll use throughout this series

What's next?

Access Key Vault secret from Web Application

Azure Key vault authentication

Understand and use Azure Key Vault Easily (Most Secured Secrets Storage)

Create Azure Key Vault using Azure Portal

Deploy Containerized Web App to Microsoft Azure | Upload/Download files of Azure Storage Account

Project Statement

Prerequisites

Step 1: Create Azure Container Registry(ACR)

Step 2: Write Flask Application

Step 3: Deploy containerized flask application to Azure Container Registry

translate text using Azure Cognitive Services

Use Azure Storage Account to securely upload and download files.

Step 4: Check logs of our container application.

Conclusion:

The `Cost` of Skipping EDA