Forem: shangkyu shin

CNN Training Isn’t Just About Models — Augmentation vs Preprocessing vs BatchNorm

shangkyu shin — Sat, 11 Apr 2026 19:09:48 +0000

Struggling with CNN training? Learn how data augmentation, preprocessing, and batch normalization improve generalization, optimize input scaling, and stabilize deep learning models. A practical guide to what actually matters in real-world CNN pipelines.

Cross-posted from Zeromath. Original article:
https://zeromathai.com/en/cnn-data-processing-normalization-en/

Stop treating augmentation, preprocessing, and BatchNorm like the same tool

A lot of CNN advice gets blurry right here.

People say:

use augmentation,
normalize the data,
add BatchNorm.

All true.

But these are not three versions of the same trick.

They solve different problems at different stages:

Data augmentation fixes a generalization problem.
Data preprocessing fixes an input-distribution problem.
BatchNorm fixes an internal-training-stability problem.

If you keep that distinction clear, a lot of CNN tuning decisions get easier.

1. Data augmentation fixes overfitting in data space

What goes wrong without it

A CNN trained on narrow data learns narrow patterns.

It memorizes:

exact positions,
exact orientations,
exact lighting conditions,
exact textures.

Then validation performance drops as soon as the real input shifts a little.

If your model starts overfitting in just a few epochs, augmentation is usually a better first lever than adding more layers.

What augmentation actually does

It creates new valid variations of existing training examples.

Examples:

flips,
translations,
crops,
affine transforms,
noise,
color jitter,
elastic deformation.

This is not random distortion for the sake of distortion.

It teaches the model:

these appearance changes do not change the label

That is why augmentation is really about learning invariance.

When augmentation becomes harmful

This is where people get careless.

Flipping a natural object image may be fine.
Flipping a medical image may not be fine.
Rotating a character or digit can silently change the class.

So the rule is simple:

only use augmentation if the label still stays valid

Best mental model:

augmentation = generalization in data space

It does not clean feature scale.
It does not stabilize hidden layers.
It just makes the training distribution harder to overfit.

2. Preprocessing fixes bad scaling and input geometry

What goes wrong without it

Raw input is messy.

Typical issues:

non-zero mean,
inconsistent scale,
correlated features.

That hurts optimization before the model even gets a chance to learn anything interesting.

One feature can dominate just because its numbers are bigger.
Updates become inefficient.
Training becomes slower than it needs to be.

What preprocessing usually includes

Zero-centering

Subtract the mean so values are centered around zero.

Normalization / standardization

Conceptually:

(x - μ) / σ

Now features are measured relative to their own variability instead of raw magnitude.

Decorrelation

Reduce redundancy between correlated dimensions.

Whitening

The mathematically stronger version:

zero mean,
reduced correlation,
normalized variance.

What actually matters in practice

Whitening is elegant on paper, but per-channel normalization is usually the practical default.

That is the kind of trade-off people often miss.

You do not always need the most theoretically complete method.
You need the method that makes optimization cleaner without adding unnecessary complexity.

So in many real CNN pipelines, this is enough:

mean subtraction,
standardization,
per-channel normalization.

Best mental model:

preprocessing = making the raw input optimization-friendly

It is not about creating diversity.
It is not a replacement for BatchNorm.
It solves a different problem.

3. BatchNorm fixes instability inside the network

What goes wrong without it

Even if the input is normalized well, deeper training can still become unstable.

Why?

Because each layer changes during learning.
That means downstream layers keep seeing shifting inputs.

So later layers are learning on top of moving targets.

What BatchNorm actually does

Batch normalization normalizes internal activations using mini-batch statistics.

But the important part is what comes next.

It does not stop at normalization.

It also applies a learnable scale and shift afterward.

That detail matters a lot.

Without that second step, normalization could become too restrictive.
With it, the network gets stability and keeps expressive flexibility.

So BatchNorm is better understood as:

normalization + representation recovery

Why engineers like it

Because it often gives you:

smoother optimization,
more stable gradients,
faster convergence,
easier tuning,
and more reliable deeper training.

If training is unstable even after input normalization, BatchNorm is probably solving a different problem, not the same one.

Best mental model:

BatchNorm = stabilization in feature space

Not data space.
Not raw input space.
Internal feature space.

4. The comparison that clears everything up

Technique	Main bottleneck	Where it acts
Data Augmentation	Overfitting	Training data
Data Preprocessing	Bad scaling / poor input geometry	Raw input
BatchNorm	Internal instability	Hidden activations

This is the distinction that matters.

When someone says “just normalize it,” the next question should be:

normalize what, exactly?

Because the answer changes the tool.

5. A real pipeline that actually makes sense

A practical CNN workflow often looks like this:

During data loading

resize if needed,
compute or use dataset statistics,
apply per-channel normalization.

During training only

apply flips, crops, translations, or other label-safe augmentation.

Inside the model

use BatchNorm where the architecture expects it,
let it stabilize internal activations while training.

This layered view is much more useful than throwing all three techniques into one mental bucket.

6. Common mistakes

“BatchNorm replaces preprocessing”

No.

BatchNorm stabilizes hidden activations during learning.
It does not remove the need for reasonable input scaling.

“More augmentation is always better”

No.

Bad augmentation creates semantically broken samples and injects label noise.

“Whitening must be best because it is more complete”

Not necessarily.

A more elegant preprocessing method is not always a better engineering choice.

“These are all just regularization tricks”

Only partly.

Augmentation is much more directly tied to generalization.
Preprocessing and BatchNorm are much more directly tied to optimization and stability.

Final takeaway

If you want one clean summary, use this:

CNN training is a distribution-control problem.

Augmentation controls variation in the data.
Preprocessing controls scale and structure in the input.
BatchNorm controls instability in internal representations.

Once you separate those three bottlenecks, CNN training gets much less mysterious.

And your debugging gets much faster too.

Which one has given you the biggest gain in practice:
augmentation, preprocessing, or BatchNorm?

Evolution of Deep CNNs — From AlexNet to ResNet (Trade-offs Behind Modern Deep Learning)

shangkyu shin — Sat, 11 Apr 2026 19:09:22 +0000

Deep CNN evolution is not about deeper models — it’s about resolving engineering trade-offs under constraints.

Cross-posted from Zeromath. Original article:
https://zeromathai.com/en/deep-cnn-evolution-alexnet-resnet-en/

CNN Evolution = Constraint Evolution

Every CNN generation answers a different question:

AlexNet → can it work?
ZFNet → why does it work?
VGG → does depth help?
GoogLeNet → can we reduce compute?
ResNet → can we optimize deeper networks?

1. AlexNet — Feasibility

Solved:

deep CNNs can actually work at scale

Key ingredients:

GPU training
ReLU
Dropout
data augmentation

2. ZFNet — Interpretability

Solved:

understanding internal representations matters

Method:

feature visualization

Insight:

debugging models improves architecture design

3. VGG vs GoogLeNet — Real Trade-off

This is the key architectural tension.

VGG

simple architecture
stacked 3×3 conv
very deep

Problem:

compute cost explodes

GoogLeNet

Inception module
multi-scale processing
1×1 conv compression

Problem:

more complex design

Trade-off

VGG	GoogLeNet
simplicity	efficiency
heavy compute	optimized compute
depth scaling	architectural branching

Insight

CNN progress is trade-off engineering, not scaling

4. ResNet — Optimization Fix

Problem:

deeper networks degrade performance

Solution:

:contentReference[oaicite:1]{index=1}

Why it works:

gradient flow improves
identity mapping preserved
optimization becomes easier

5. Big Picture

Model	Problem solved
AlexNet	feasibility
ZFNet	interpretability
VGG	depth scaling
GoogLeNet	efficiency
ResNet	optimization stability

Key Pattern

Every model follows:

limitation appears
root cause identified
architecture changes
scaling resumes

Final Insight

Deep learning is not model evolution.

It is:

continuous engineering under constraints

Discussion:

Which constraint mattered most in practice?

depth
efficiency
optimization

CNN Layer Composition — A Practical Developer Guide to Activation, Pooling, and Fully Connected Layers

shangkyu shin — Sat, 11 Apr 2026 18:59:34 +0000

CNNs are not just convolution stacks. This guide explains how activation, pooling, and fully connected layers work together to transform feature maps into predictions.

Cross-posted from Zeromath. Original article: https://zeromathai.com/en/cnn-layer-composition-en/

CNN Layer Composition (Think Like an Engineer)

A CNN is not magic.

It’s a pipeline:

input → feature extraction → filtering → compression → classification

1. Convolution Alone = Not Enough

Convolution is linear.

Stack linear layers:

→ still linear

So:

no complex decision boundary
no deep feature learning

Activation is mandatory.

2. ReLU — The Switch That Enables Depth

ReLU:

f(x) = max(0, x)

Example:

[-3, -1, 0.5, 2] → [0, 0, 0.5, 2]

Why it matters:

introduces nonlinearity
avoids vanishing gradient
filters weak signals

3. Shape Flow (Real Example)

Input:
(224, 224, 3)

Conv:
(224, 224, 64)

ReLU:
(224, 224, 64)

Pooling:
(112, 112, 64)

Key rules:

spatial ↓
channels same

4. Why Channels Increase

As depth increases:

spatial size ↓
channel count ↑

Why?

→ model learns more feature types

5. Pooling vs Stride

Pooling:

fixed
no parameters

Strided Conv:

learnable
more flexible

Modern models often prefer strided conv.

6. Max Pooling = Feature Selection

2×2 max pooling:

Input:
1 1 2 4

5 6 7 8

3 2 1 0

1 2 3 4

Output:
6 8

3 4

Effect:

strongest signal survives
noise removed

7. Receptive Field

Deeper layers:

see more context
capture higher-level features

Flow:

edges → textures → shapes → objects

8. Flatten + Dense

Before classification:

(7, 7, 512) → (25088)

Then:

Dense → Softmax → prediction

9. Modern Trick: Global Average Pooling

Instead of big dense layers:

average each channel
fewer parameters
better generalization

10. Full Pipeline

Conv → detect
ReLU → filter
Pool → compress
Repeat → hierarchy
Dense → predict

Debug Mindset

If model fails:

bad features → conv problem
weak signal → activation issue
too slow → pooling issue
wrong output → classifier issue

Key Takeaways

CNN = structured system
ReLU enables learning
Pooling controls scale
Dense layers make decisions

Discussion

In real projects, what matters most?

architecture design?
training tricks?
or data quality?

Curious to hear your experience.

CNN Spatial Behavior Explained: Convolution, Stride, Padding, and Output Size (With Intuition)

shangkyu shin — Sat, 11 Apr 2026 18:57:12 +0000

Understanding CNNs requires more than just architectures. Learn how convolution, stride, padding, and output size shape spatial behavior in deep learning models, with practical intuition and real-world design insights.

Cross-posted from Zeromath. Original article: https://zeromathai.com/en/pooling-activation-layers-en/

The Real Problem: Spatial Understanding (Not Layers)

Most CNN issues are NOT about:

“Which architecture should I use?”

They are about:

wrong tensor shapes
misunderstanding stride/padding
losing spatial information too early

If you’ve ever hit something like:

RuntimeError: size mismatch

This post is for you.

Convolution = Sliding Pattern Detector

At each position:

Take a small patch
Multiply with filter weights
Sum → one output value

Repeat → feature map

Key properties:

local connectivity
shared weights
translation awareness

This is why CNNs scale.

Filters: What Your Model Actually Learns

Each filter learns ONE pattern.

Examples:

edge detector
texture detector
color transition

Multiple filters → multiple feature maps

Conceptually:

output_channels = num_filters

Important:

CNNs don’t learn “images” — they learn patterns.

Receptive Field (Core Concept)

Each neuron sees only part of the image.

Example:

Input: 32×32
Kernel: 5×5

→ neuron sees 5×5 region

Stack layers:
→ receptive field grows

Meaning:

early layers → local features
deeper layers → global features

Stride = Resolution Control

Stride defines how far the filter moves.

Stride	Effect
1	high detail
2	downsample

Trade-off:

larger stride → faster
but → information loss

Real-world mistake:

using stride=2 too early → model misses fine features

Padding = Boundary Control

Without padding:

output shrinks
edge information disappears fast

With padding:

spatial size preserved
borders are kept

Typical implementation:

padding = (kernel_size - 1) // 2

Rule of thumb:

deep CNNs almost always use padding

Output Size Formula (You MUST Know This)

Output = (m - k) / s + 1

Where:

m = input size
k = kernel size
s = stride

Example:

(7 - 3) / 1 + 1 = 5

If you don’t calculate this:
→ your model WILL break

One Filter vs Many Filters

Filters	Output
1	1 feature map
32	32 channels

Output shape:

H × W × C

Where:

C = number of filters

More filters = richer representation

Common Real-World Mistakes

1. Shape mismatch

You didn’t compute output size correctly.

2. Too much downsampling

Large stride early → lost spatial information.

3. No padding

Edges vanish layer by layer.

4. Too few filters

Model lacks expressive power.

Design Intuition (What Actually Matters)

When designing CNNs:

kernel size → what patterns you detect
stride → how fast you compress
padding → whether you preserve structure
filters → how rich your representation is

This is not hyperparameter tuning.

This is:

designing how your model perceives the world

Final Takeaway

CNNs don’t “see images”.

They:

scan locally
extract patterns
build hierarchical representations

If you understand:

convolution
receptive field
stride
padding

Then you understand:

how CNNs actually work

What part of CNN design still feels confusing?

Drop your thoughts 👇

Why CNNs Work: Convolution, Feature Hierarchies, and the Real Difference from Fully Connected Networks

shangkyu shin — Sat, 11 Apr 2026 18:46:06 +0000

Understanding CNNs is not about memorizing layers.

It’s about understanding why this design exists.

Cross-posted from Zeromath. Original article: https://zeromathai.com/en/convolutional-layer-lec-en/

The Core Problem

Images are structured data.

A fully connected network treats them as flat vectors.

Example:

224×224×3 → 150K inputs

Dense layer → millions of parameters

Problems:

No spatial awareness
Too many parameters
Overfitting

What CNNs Fix

CNN introduces two key ideas:

Local connectivity
Weight sharing

Instead of connecting everything:
→ look locally, reuse globally

CNN Pipeline

Image → Conv → ReLU → Pool → Conv → ... → FC → Softmax

Convolution Layer

A filter slides across the image.

At each position:

Multiply
Sum
Output activation

Shape Example

Input: 32×32×3

Filter: 5×5×3

Output: 28×28

Why It Works

Detects local patterns
Works anywhere
Learns reusable features

Feature Maps

Feature maps are representations.

They answer:

→ where is this feature?

ReLU (Critical)

f(x) = max(0, x)

Without it:

Model is linear

With it:

Nonlinear learning
Better optimization

Pooling Layer

28×28 → 14×14

Benefits:

Faster
More robust
Translation invariant (approx)

Important Insight

CNNs are not truly translation invariant.

Pooling only makes them more robust to shifts.

Too much pooling:
→ destroys spatial detail

Modern CNNs:
→ reduce pooling

→ use strided convolution

Fully Connected Layer

Flatten → combine features → classify

Softmax → probabilities

Feature Hierarchy (Core Idea)

CNNs learn progressively:

Layer	Learns
Early	edges
Middle	textures
Deep	objects

Example:
edge → eye → face

Why CNNs Beat Dense Networks

CNN:

Efficient
Spatially aware
Generalizes well

Dense:

Huge parameter count
No structure awareness
Overfits

Debugging CNNs (Underrated Skill)

Use:

Activation maps
Saliency maps
Grad-CAM

These help:

Debug errors
Understand predictions
Improve models

Practical Tips

Don’t overuse pooling
Track feature map sizes
Prefer depth over width
Visualize early

Final Insight

The real breakthrough of CNNs is not just convolution.

It is the combination of:

Locality
Parameter sharing
Hierarchical learning

That’s what turns pixels into meaning.

For image tasks today, do you still start with CNNs, or jump straight to Vision Transformers?

Let’s discuss 👇

Why CNNs Work for Images: The Real Design Logic Behind Convolutional Neural Networks

shangkyu shin — Sat, 11 Apr 2026 18:40:32 +0000

Why do CNNs outperform fully connected neural networks on image tasks? This article explains local connectivity, weight sharing, pooling, and inductive bias in a practical, developer-friendly way.

Cross-posted from Zeromath. Original article: https://zeromathai.com/en/introduction-to-cnns-en/

Why CNNs Needed to Exist

CNNs were not invented just because researchers wanted a better benchmark score.

They were invented because applying a standard Multilayer Perceptron to images is a bad fit.

A fully connected network treats an image as a long flat vector. That already hints at the problem: images are not flat in any meaningful visual sense.

They are spatial.

They have local patterns.

And the same useful feature can appear in different positions.

That mismatch is the whole reason CNNs matter.

The MLP Problem in One Example

Take a 200 × 200 RGB image.

That gives:

200 × 200 × 3 = 120,000 input values

Now connect that input to a hidden layer with 1,000 neurons.

You get about 120 million weights.

That is bad for three reasons:

training cost explodes,
overfitting risk goes up,
and the model still has no built-in understanding of spatial structure.

So the issue is not just "too many parameters."

The deeper issue is that a dense layer starts from the wrong assumption.

Images Have Structure, Not Just Size

For tabular data, treating inputs as a feature vector is often fine.

For images, it is not.

Why?

Because image data has properties that matter directly:

nearby pixels are correlated,
edges and textures are local,
and object identity often survives small position changes.

A cat is still a cat whether it appears slightly left or slightly right.

A model for images should reflect that.

The CNN Idea

CNNs solve this by injecting a useful inductive bias.

Instead of saying, "learn everything from scratch," CNNs say:

local patterns matter,
the same pattern can appear anywhere,
and spatial layout should be preserved long enough to build higher-level features.

That one architectural choice changes both efficiency and generalization.

1. Local Connectivity

In a dense layer, each neuron connects to the full input.

In a convolutional layer, each neuron looks at a small local patch.

That patch is the receptive field.

This makes sense for images because most meaningful low-level features are local:

edges,
corners,
texture fragments.

You do not need the whole image to detect a vertical edge in one region.

From an engineering perspective, local connectivity dramatically cuts parameter count.
From a modeling perspective, it aligns the network with the structure of visual data.

2. Weight Sharing

This is the design principle that makes convolution feel elegant.

A CNN does not learn a separate edge detector for every location in the image.

It learns one filter and applies it across locations.

That means the same detector can fire on the left side, center, or right side of the input.

This gives us two big wins.

Fewer parameters

Instead of learning duplicated weights for similar patterns at many positions, the model reuses the same filter.

Consistent feature detection

If the input shifts, the activation pattern shifts consistently.

That is translation equivariance.

A simple intuition:

move the edge in the input,
and the edge response moves in the feature map.

For early visual processing, that is exactly what we want.

3. Pooling

Pooling is often introduced as a downsampling step, and that is true, but it is more useful to think of it as controlled compression.

It reduces the size of feature maps while preserving the strongest or most representative signals.

Common examples are max pooling and average pooling.

Why is that useful?

later layers become cheaper,
small local changes matter less,
and the network becomes more robust to noise or slight shifts.

A subtle but important point: pooling does not create perfect invariance by itself.

What it really gives is robustness to minor local variation.

That is a better mental model.

Why CNNs Usually Generalize Better

CNNs are not just smaller versions of dense networks.

They are structured models.

That matters because generalization improves when the architecture matches the data domain.

CNNs help by:

reducing unnecessary degrees of freedom,
forcing local pattern learning,
reusing filters across space,
and preserving spatial organization through feature maps.

So when people say CNNs are efficient, they do not just mean "faster."

They mean the model wastes less capacity on unrealistic hypotheses.

Feature Maps and Hierarchical Learning

One of the nicest ways to understand CNNs is to think in terms of feature maps.

A filter scans the image and produces a map showing where that filter’s learned pattern appears.

Early filters often learn things like:

horizontal edges,
vertical edges,
simple textures.

Deeper layers then combine those into:

contours,
repeated motifs,
parts of objects,
object-level patterns.

This is hierarchical representation learning.

In practice, CNNs move from "small visual primitives" to "larger semantic concepts."

That is why deep convolutional networks became so effective in computer vision.

A Useful Comparison: MLP vs CNN

Here is the cleanest mental contrast.

MLP

treats input as a flat vector,
uses dense connectivity,
learns with few built-in assumptions about image structure.

CNN

treats input as spatial data,
uses local connectivity,
shares weights across locations,
builds layered feature hierarchies.

So the difference is not just architecture style.

It is a difference in how the model thinks the data is organized.

Why the Architecture History Matters

Once these core ideas were established, later CNN families mainly improved optimization, depth, and efficiency.

AlexNet showed deep CNNs could dominate large-scale image recognition.
VGG showed that stacking simple small convolutions could work extremely well.
GoogLeNet improved efficiency and multi-scale processing.
ResNet made very deep networks train reliably with skip connections.
DenseNet pushed feature reuse even further.

Different design, same foundation.

All of them rely on the logic introduced above.

The Real Lesson

The most important thing to learn from CNNs is bigger than CNNs.

Good model design is about matching architecture to data structure.

For images, that means locality, repeated patterns, and spatial hierarchy.

CNNs encode those assumptions directly.

That is why they work.

Final Takeaway

If you remember one line, remember this:

MLPs treat images like generic vectors. CNNs treat images like images.

That is the real reason convolution changed computer vision.

What part of CNN design do you think mattered most historically: local connectivity, weight sharing, or later innovations like residual connections?

Image Classification Explained — Why k-NN Breaks and Linear Classifiers Matter

shangkyu shin — Sat, 11 Apr 2026 18:13:54 +0000

Image classification sounds easy until you remember that a computer never sees “objects.” It only sees pixel arrays. This post explains why that makes k-NN a useful but limited baseline, and why linear classifiers are the point where real learning begins.

Cross-posted from Zeromath. Original article: https://zeromathai.com/en/image-classification-en/

Start from the Actual Engineering Problem

We usually describe image classification like this:

input: image

output: label

That description is correct, but it hides the hard part.

For a machine, an image is not “a cat” or “a truck.”

It is just something like:

248 × 400 × 3 numbers

So the real problem is:

How do you map raw pixel values to a meaningful class?

That question sits under a lot of computer vision work. Classification is the base layer. Object detection adds location. Segmentation adds per-pixel labeling. But the first wall you hit is still the same one: turning numeric arrays into semantic meaning.

Why Raw Pixels Are a Bad Starting Space

Here is the simplest failure case.

Take an image of a cat.

Now:

shift it 2 pixels to the right
slightly increase brightness
crop a small region
change the background

To a human, it is still clearly a cat.

To a model using raw pixel distance, it can look very different.

This is the core issue:

pixel space is not semantic space

Two inputs can be far apart numerically but identical in meaning.

Two inputs can be close numerically but represent different objects.

Real-world images make this worse due to:

viewpoint changes
scale differences
deformation
occlusion
lighting variation
background clutter

A good model must ignore what does not matter and respond to what does.

Why Rule-Based Vision Fails

A natural early idea is to define objects manually.

For example:

cats have ears
cats have whiskers
cats have certain shapes

This breaks quickly.

ears may be hidden
lighting may remove edges
backgrounds may look similar
poses may distort shapes

Rule-based vision fails because the visual world is too variable.

This is why machine learning shifted to a data-driven approach:
collect examples, learn patterns, and generalize instead of hardcoding rules.

First Baseline: k-Nearest Neighbor (k-NN)

The most intuitive classifier is k-NN.

Idea:
find similar images and reuse their labels

Basic flow:

store all training data
compute distance to each sample
pick top-k closest
vote

Why developers still use k-NN:

simple baseline
quick sanity check
useful for debugging datasets
exposes whether representation makes sense

Where k-NN Breaks

Shift sensitivity

Small translations change pixel alignment everywhere.
Lighting sensitivity

Brightness changes affect all pixels.
Flattening destroys structure

image → flatten → vector

You lose spatial relationships and locality.
High-dimensional issues

Distances become less meaningful in high dimensions.
Performance problems

O(N) comparisons per prediction

High memory usage

Core Insight

k-NN does not learn.

It memorizes the dataset and compares at test time.

This is useful for intuition, but not scalable.

Why Validation Still Matters

Even with k-NN, you must choose:

value of k
distance metric
preprocessing

These are hyperparameters.

Validation or cross-validation helps you:

compare configurations
avoid overfitting
select better setups

This pattern continues in all machine learning models.

The Shift That Changes Everything

To move forward, we stop asking:

which stored images are closest?

and start asking:

can we learn a function that predicts directly?

Linear Classifier: Where Learning Begins

A linear classifier computes:

score = W × x + b

Where:

x is the input vector
W is the weight matrix
b is the bias

Now the model:

does not need the full dataset at inference
computes predictions in constant time
learns parameters from data

Why This Matters

k-NN vs Linear Classifier:

similarity lookup vs learned function
no training vs parameter learning
slow inference vs fast inference
high memory vs compact model
weak generalization vs stronger generalization

What Actually Changed

Not just performance.

Conceptually:

k-NN → similarity-based reasoning
linear model → learned representation

This is the moment where machine learning becomes actual learning.

Why Developers Should Care

If you work with:

CNNs
vision models
deep learning systems

this is your foundation.

Understanding this explains:

why raw pixels are not enough
why feature learning matters
why deep architectures exist

Final Takeaway

Image classification is not just predicting labels.

It is about turning unstable raw pixel inputs into stable semantic outputs.

k-NN is a great teaching tool and debugging baseline.

But it shows exactly why we need something better.

Linear classifiers matter because they introduce learning.

And that is where modern computer vision really begins.

Discussion

Do you still use k-NN as a baseline or debugging tool?

Or do you jump straight into learned models like CNNs?

CNNs Explained: How Image Classification Actually Works in Deep Learning

shangkyu shin — Sat, 11 Apr 2026 18:10:19 +0000

Understanding CNNs means understanding how models turn raw pixels into structured representations. This guide explains convolution, pooling, and architectures like ResNet with practical insights.

Cross-posted from Zeromath. Original article: https://zeromathai.com/en/dl-convolutional-neural-networks-cnn-en/

The Real Problem: Pixels → Meaning

Images are just tensors.

No objects. No semantics.

So the real question is:

How do we extract structure from raw data?

Why Old Pipelines Didn’t Scale

Classic approach:

Feature extraction (SIFT, HOG)
Classifier (SVM)

Limitation:

You only learn what you design.

Why MLPs Fail (Critical Insight)

Flattening images destroys structure.

Problems:

Parameter explosion
No spatial awareness

But the deeper issue:

No reuse of patterns

CNNs = Structured Efficiency

CNNs fix this with:

Local connectivity
Weight sharing

Meaning:

Fewer parameters
Better generalization
Built-in spatial bias

What Convolution Actually Learns

Filters become detectors:

Edges
Textures
Shapes

Stacking layers creates hierarchy:

Edges → shapes → objects

Why Depth Matters (Practical View)

Shallow model:

Detects edges

Deep model:

Understands objects

Depth = abstraction

Core Components (What Actually Matters)

ReLU

Stabilizes gradients
Enables deep learning

Pooling

Reduces noise
Adds robustness

Fully Connected

Final decision layer

Why ResNet Changed Everything

Deep networks used to fail.

Problem:

Degradation with depth

Solution:

Skip connections

Real Effect:

Easier training
Deeper models
Better results

Training Insights (This Is Where Most Bugs Are)

1. Data Augmentation > Architecture (Often)

Small dataset?

→ augmentation matters more than model choice

2. BatchNorm = Stability

Without it:

training unstable

With it:

faster convergence

3. Preprocessing Is Not Optional

Unnormalized input = unstable gradients

Debugging CNNs (Highly Practical)

Feature Maps

See what the model detects

CAM (Class Activation Map)

See what the model uses

Real-World Example

Model classifies “cow” correctly.

CAM shows:

Focus on grass, not cow

Conclusion:

Dataset bias, not model intelligence

Practical Takeaways

CNNs learn features automatically
Structure matters more than size
Depth builds meaning
Training tricks are critical
Visualization reveals hidden problems

Final Thought

CNNs are not just models.

They encode this idea:

Learn representations, not rules

If you’ve worked with CNNs:

Did augmentation help more than architecture?
Have you checked CAM for bias?
Where did your model actually fail?

Neural Network Optimization Challenges — Fixing Vanishing Gradients with Better Architecture Design

shangkyu shin — Sat, 11 Apr 2026 18:08:01 +0000

Vanishing gradients are one of the main reasons deep neural networks fail.

If your deeper model performs worse than a shallow one, this is usually the cause.

This post explains what’s happening—and how to fix it in practice.

Cross-posted from Zeromath. Original article: https://zeromathai.com/en/optimization-architecture-en/

1. A Real Problem You’ve Probably Seen

You build a deeper model expecting better performance.

Instead:

training slows down
loss stops improving
accuracy gets worse than a smaller model

This feels wrong.

But it’s common.

2. The Root Cause: Gradient Flow Collapse

Backpropagation sends gradients backward through layers.

Each layer multiplies them.

If those values are small:

they shrink exponentially
eventually become ~0

Result:

early layers stop learning
model cannot improve

3. Why Sigmoid Breaks Deep Models

Sigmoid looks mathematically clean.

But in deep networks:

outputs saturate
derivatives become tiny
gradients vanish

Example:

σ(5) ≈ 0.993
σ′(5) ≈ 0.007

Stack multiple layers:

(0.007)^10 → effectively zero

This is why deep sigmoid networks fail.

4. The First Fix: ReLU

ReLU avoids saturation:

f(x) = max(0, x)
derivative = 1 (positive region)

Effect:

gradients survive
deeper models train

Variants:

Leaky ReLU → avoids dead neurons
GELU → smoother behavior (Transformers)

5. Depth vs Width (What to Actually Do)

More depth is not always better.

Deep:

expressive
hard to train

Wide:

stable
less hierarchical

If training fails:

try adjusting structure, not just size.

6. Skip Connections (Why ResNet Works)

Skip connections add a shortcut:

x → F(x) + x

This allows:

gradients to bypass layers
signal strength to remain intact

Without it:

deep networks degrade

With it:

deep networks train reliably

7. Architecture = Optimization Strategy

Most people try to fix training with:

learning rate tweaks
optimizer changes

But the real fix is often:

architecture

activation → controls gradients
depth → increases difficulty
skip connections → fix gradient flow

8. Practical Debug Scenario

If your model:

gets worse when deeper
shows near-zero gradients early
trains very slowly

Then:

switch to ReLU/GELU
add skip connections
reconsider architecture

9. Key Insight

If a deeper model performs worse than a shallow one:

suspect optimization before capacity.

Final Thought

Deep learning is not about stacking layers.

It’s about preserving learning signals.

No gradient → no learning

Stable gradient → scalable models

What worked for you?

architecture changes?
activation tweaks?
training tricks?

Curious to hear real experiences.

How Neural Networks Actually Learn: Backpropagation, Gradients, and Training Loop (Developer Guide)

shangkyu shin — Sat, 11 Apr 2026 18:04:48 +0000

Learn how neural networks train using forward propagation, loss functions, and backpropagation. This developer-focused guide explains gradients, chain rule, and autograd with practical intuition.

Cross-posted from Zeromath. Original article: https://zeromathai.com/en/training-signals-back-fundamentals-en/

The Real Mechanism

Neural networks don’t “learn” in a human sense.

They optimize.

Every step is:

forward → loss → backward → update

Training vs Inference

Training:

compute loss
run backward()
update weights

Inference:

forward only

No backward pass = no learning

Two Signals

Forward → prediction
Backward → gradients

Think:

forward = what happened
backward = how to fix it

Loss Function (Why Errors Matter)

Binary Cross-Entropy example:

ŷ = 0.8 → loss ≈ 0.223

ŷ = 0.1 → loss ≈ 2.302

Key idea:

Wrong predictions create stronger gradients

Gradients = Direction

gradient = ∂loss / ∂parameter

This tells us how to update weights.

Chain Rule (Core)

y = f(g(h(x)))

dL/dx = dL/dy · dy/dg · dg/dh · dh/dx

The Only Rule You Need

gradient = upstream × local derivative

Example: Multiplication

z = x * y

dL/dx = dL/dz * y

dL/dy = dL/dz * x

Example: Square

out = x²

grad = upstream * 2x

Why Autograd Exists

Manual chain rule doesn’t scale.

Frameworks:

store forward values
build computation graph
apply backward automatically

What Happens in Code

y_pred = model(x)

loss = criterion(y_pred, y)

loss.backward()

optimizer.step()

optimizer.zero_grad()

Important Implementation Details

Why zero_grad()?

Gradients accumulate by default.

Without reset:

grad_total = grad_step1 + grad_step2 + ...

Why backward() first?

Because gradients must exist before updating:

loss.backward() → gradients computed

optimizer.step() → parameters updated

Why reverse traversal?

Because gradients depend on outputs.

So computation flows:

output → input

Computational Graph Intuition

forward = build graph
backward = traverse graph in reverse

Intermediate results are reused.

Gradient Descent

θ = θ − η ∇L

η = learning rate

Final Takeaway

Neural networks learn by:

propagating error backward and updating parameters

That’s the entire system.

What helped you understand backprop the most—math, visualization, or code? Let’s discuss 👇

Output Layer Explained — Logits, Softmax, Cross-Entropy, and Why They Work Together

shangkyu shin — Sat, 11 Apr 2026 17:59:09 +0000

Neural networks don’t output decisions — they output probabilities.

This post explains how logits, softmax, and cross-entropy turn raw outputs into meaningful predictions in deep learning.

Cross-posted from Zeromath. Original article: https://zeromathai.com/en/output-layer-probabilistic-interpretation-en/

The Real Role of the Output Layer

A neural network doesn’t directly say:

“This is class A.”

Instead, it computes:

A probability distribution over all classes.

Step 1 — Logits (Raw Scores)

Final layer:

ŷ = Wh + b

Output:

z = logits

Not probabilities
Not normalized
Can be negative or large

Example:

[0.4, -1.7, 4.2]

Step 2 — Softmax (Make It Probabilistic)

softmax(z_i) = exp(z_i) / Σ exp(z_j)

Transforms:

[0.4, -1.7, 4.2]

→ [0.022, 0.003, 0.975]

Now outputs are:

positive
sum to 1
interpretable

Step 3 — Argmax (Decision)

Softmax → probabilities
Argmax → final class

Important:

Softmax keeps uncertainty

Argmax removes it

Step 4 — Training Uses Cross-Entropy

Loss:

− log(p_true_class)

Why:

differentiable
punishes confident mistakes
aligns with probability theory

Step 5 — Why Frameworks Use Logits Directly

In PyTorch / TensorFlow:

CrossEntropyLoss expects logits
NOT softmax output

Why?

Numerical stability:

log(softmax(z)) is computed safely without overflow

Step 6 — Softmax vs Sigmoid (Real-World Bug Source)

Binary → sigmoid
Multi-class → softmax
Multi-label → sigmoid per class

Common bug:

Using softmax for multi-label → wrong behavior

Step 7 — Inference Tip

Do you always need softmax?

For prediction only → argmax(logits) works
For probabilities → apply softmax

This saves computation in production systems.

Mental Model

Input → Features → Logits → Softmax → Probabilities → Argmax → Prediction

Debugging Checklist

Overconfident wrong → calibration issue
Always low confidence → weak features
Loss not decreasing → output/loss mismatch
Multi-label broken → wrong activation

Final Takeaway

Output layer → scores
Softmax → probabilities
Argmax → decisions
Cross-entropy → learning

Deep learning works because:

It models uncertainty, not just outputs.

Where do you usually get stuck — logits, softmax, or loss functions?

Multilayer Perceptron (MLP): A Practical Way to Understand Neural Networks

shangkyu shin — Sat, 11 Apr 2026 17:37:29 +0000

Multilayer Perceptrons (MLPs) are the foundation of deep learning. This guide explains MLP intuition, real-world usage, and when you should (and shouldn’t) use it.

Cross-posted from Zeromath. Original article: https://zeromathai.com/en/mlp-intuition-components-en/

MLP = A Function (Not Layers)

Most people think neural networks are stacks of layers.

They are wrong.

An MLP is:

y = f(x; θ)

👉 A learnable function.

Start Simple

z = wᵀx + b

works for simple problems
fails for nonlinear patterns

Add Nonlinearity → Neural Network

a = σ(wᵀx + b)

Now you can model:

nonlinear relationships
feature interactions

👉 This is where deep learning starts.

Core Building Block

Each neuron:

linear transform
activation

Stack them → model.

Example

x = (1, 2)

w = (0.5, -1)

b = 0.1

z = -1.4

Then activation decides output.

Layers

Each layer:

x → Wx + b → activation

Stack:

input → hidden → output

Why Depth Works

Instead of learning everything at once:

Layer 1 → simple features
Layer 2 → combinations
Layer 3 → abstractions

👉 Deep learning = function composition

When to Use MLP (Real Use Cases)

Use MLP when:

tabular datasets (very common in industry)
structured features (e.g. finance, logs, metrics)
baseline model before complex architectures

👉 In many real projects, MLP is the first model you try.

When NOT to Use MLP

Avoid MLP when:

images → use CNN
sequences → use RNN / Transformer
structure matters

👉 MLP assumes features are independent.

Practical Comparison

MLP:

good for tabular data
assumes no structure

CNN:

good when nearby pixels matter

Transformer:

good when relationships matter globally

👉 Choose model based on data structure.

Minimal PyTorch Example


python
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(10, 32),  # 10 input features
    nn.ReLU(),
    nn.Linear(32, 1)    # regression output
)