Forem: Shrinivas Vishnupurikar

Schema, COPY, MERGE, and Immutability — A First-Principles Guide for Data Engineers

Shrinivas Vishnupurikar — Mon, 05 Jan 2026 03:08:29 +0000

In modern data engineering conversations, terms like schema-on-read, schema-on-write, COPY, MERGE, and immutable partitions are used very often.

When I started out my career in Data Engineering, I always heard my senior engineers ( mentors ) mention these when they were designing and architecting pipelines and data deliver platforms in general. Back then, for me these were merely some combinations of words, but seniors this was the matter of design choices that could make or break project deliverables.

These are isolated terms that are deeply connected and which result in data processing patterns and data management techniques.

You could start by understanding them in isolation, but understanding them together builds a strong mental model that applies across systems, formats, and platforms.

This article explains these ideas from first principles, then connects them to open table formats, with a strong focus on why these patterns exist.

1. What Is a “Schema” — First Principles

At its core, a schema is a contract or an agreement; a word-of-mouth like deal between you and the data.

A schema typically defines:

What fields exist
What each field represents
What type of data each field can hold
Whether fields are optional or mandatory

The most important question is not what the schema is, but in our context for today’s article it’s when the system enforces it.

That single question gives birth to two fundamental models:

Schema-on-Write
Schema-on-Read

1.1 Schema-on-Write (Validate First, Store Later)

Definition

Schema-on-write enforces structure before data is stored, in your target systems. If incoming data does not match the expected structure, it is rejected.

Think of schema-on-write like a strict security checkpoint at a college entrance.

The college has a predefined rule: only students with a valid college ID are allowed inside.
The security guard checks the ID before allowing entry.
If you do not have the ID, you are stopped at the gate.

In this analogy:

The college premises represent the target data system
The college ID card represents the schema
The security check represents schema validation
Entry is allowed only if the contract is satisfied

Intuition

“Only clean, trusted data is allowed inside.”

Story Example

Consider a bank account system.

Before saving a record:

Account number must exist
Balance must be numeric
Date fields must be valid

If anything is incorrect, the record is refused. Bad data never enters the system.

Why This Pattern Exists

Data correctness is critical
Downstream systems assume reliability
Errors must be detected early

Trade-off

Lower flexibility
Slower ingestion
Schema changes require coordination

1.2 Schema-on-Read (Store First, Decide Later)

Definition

Schema-on-read stores data first and applies structure only when the data is read.

Think of schema-on-read like entering a public library or a large campus.

There is no strict identity check at the gate
Anyone can enter—students, visitors, researchers
The system does not ask who you are or what you will do upfront

Rules are applied only when you access something specific:

Certain rooms may require permission
Certain books may have usage rules
Some resources may only be available to specific people

In this analogy:

The campus or library represents the storage system
Entering freely represents storing raw data without validation
Rules applied later represent schema being enforced at read time
Meaning is decided when you access the data, not when it arrives

Story Example

A mobile app analytics system collects user events:

Different app versions send different fields
Some events are incomplete
New fields appear frequently

Rejecting such data would cause data loss. Instead, everything is stored, and meaning is applied during analysis.

Why This Pattern Exists

High data variability
Rapid change
Exploration and discovery use cases

Trade-off

Data quality issues surface later
Queries must handle inconsistencies

2. COPY vs MERGE — What Problem Do They Solve?

Schemas define structure. COPY and MERGE define how new data interacts with existing data.

The key question here is:

Does incoming data represent new facts or changes to existing facts?

2.1 COPY (Append-Only Thinking)

Definition
COPY means inserting incoming data as new rows without checking for existing records.

Intuition

“Oh, new data?
Cool. Send it in.

Do I already have something similar?
I don’t care.

Is this a duplicate?
Not my problem.

I’ll just add whatever comes in as a new data point and move on.”

Story Example

A daily sales report:

Yesterday’s sales never change
Today’s sales are new facts

Each day’s data is simply appended.

Why This Pattern Exists

Simplicity
High performance
Historical accuracy

Trade-off

Corrections require reprocessing
Duplicates are possible

2.2 MERGE (State-Aware Thinking)

Definition

MERGE updates existing records when matches are found and inserts new records when they are not.

Intuition

“Alright, new data is coming in.

David, if you find something we already have, update it with the latest details.

Carlos, if you don’t recognize the record at all, add it as a brand-new entry.

Bottom line — keep the data up to date.”

Story Example

A customer profile system:

Email remains the same
Address and phone number change
Status evolves over time

The system must reflect the current state, not a trail of outdated versions.

Why This Pattern Exists

Data represents entities, not events
Corrections are expected
Idempotency matters

Trade-off

More complex logic
Higher compute cost

3. All Valid Combinations — Schema × Write Pattern

3.1 Schema-on-Write + COPY

Strict structure, immutable history

Story:
A financial ledger where every transaction must be valid and never changes.

Why it fits:

Data correctness is enforced
History is preserved forever

3.2 Schema-on-Write + MERGE

Strict structure, evolving state

Story:
A customer master table with well-defined fields, where customer details are updated over time.

Why it fits:

Strong data guarantees
Accurate current view

3.3 Schema-on-Read + COPY

Flexible structure, raw history

Story:
An application log store capturing all events, even malformed ones.

Why it fits:

Zero data loss
Future reprocessing is possible

3.4 Schema-on-Read + MERGE

Flexible structure, improving state

Story:
A data enrichment system where records arrive incomplete and get enriched later.

Why it fits:

Allows partial data
Data quality improves over time

4. Where Partition Immutability Enters the Picture

Modern open table formats rely on immutable data files (partitions).

Immutability means:

Once written, data files are never modified
Updates and deletes create new files
Old files are logically retired via metadata ( often with property like status = DELETED or valid_to = )

This design is essential to:

Safe concurrent reads and writes
Reliable versioning
Efficient rollback and recovery

5. Immutability in Open Table Formats

Apache Iceberg (Primary Focus)

Design philosophy:

Strong immutability
Snapshot-based metadata
Schema evolution as a first-class feature

How patterns map:

COPY → new data files added to a snapshot
MERGE → rewritten files + new snapshot
Schema-on-read and schema-on-write both supported via metadata evolution

Iceberg treats metadata as the control plane and data files as immutable assets.

Delta Lake (Secondary)

Design philosophy:

Transaction log driven
Immutable parquet files
Strong support for MERGE

How patterns map:

COPY → append entries in the transaction log
MERGE → file rewrites tracked in the log
Schema enforcement optional but commonly used

Delta emphasizes transactional reliability with strong ACID guarantees.

Apache Hudi (Secondary)

Design philosophy:

Designed for mutable datasets
Supports record-level updates
Multiple storage strategies

How patterns map:

COPY → insert-only tables
MERGE → copy-on-write or merge-on-read
Schema evolution supported but more operationally involved

Hudi optimizes for near-real-time updates and streaming workloads.

6. Closing Note

Schema enforcement, write patterns, and immutability are not independent ideas.
They are three dimensions of the same design space:

Schema answers when data is trusted
COPY vs MERGE answers how data evolves
Immutability answers how systems stay correct at scale

Once these are understood together, modern data systems stop feeling complex and start feeling inevitable.

This understanding travels with you—across tools, platforms, and technologies.

Snowflake + Postgres: A Small Feature That Signals a Big Shift

Shrinivas Vishnupurikar — Fri, 26 Dec 2025 06:34:17 +0000

The Story Every Data Engineer Wonders About

When people talk about data engineering, the explanation is usually simple.

A data engineer moves data from one system to another.

In most real-world setups, this means moving data from an OLTP system, where transactions are written continuously, into an OLAP system, which is optimized for analytics, reporting, and business insights. This explanation is usually enough for anyone new to the field.

But over time, many data engineers begin to feel that something about this setup is slightly off.

A significant amount of effort goes into building, maintaining, monitoring, and debugging pipelines whose only purpose is to move data from one place to another. Not to transform it in a meaningful way. Not to enrich it. Simply to keep two worlds in sync.

Eventually, a quiet question starts to surface:

Why do transactional data and analytical data still need to live in completely separate systems?

Acronyms Used in This Blog

To keep things clear, here are the acronyms used throughout this article:

OLTP (Online Transaction Processing): Systems optimized for fast inserts, updates, and deletes
OLAP (Online Analytical Processing): Systems optimized for large-scale reads and analytics
CDC (Change Data Capture): Techniques used to track and replicate data changes
AI (Artificial Intelligence): Systems that learn from data to make predictions or decisions

The Elephant in the Room

This question becomes even more fitting in the context of Postgres.

Postgres literally uses an elephant as its logo, and for years, the separation between Postgres and analytics platforms has been the elephant in the room of modern data architectures.

We all understand the technical reasons behind the split.

OLTP systems are designed for correctness, concurrency, and fast writes.

OLAP systems are designed for large scans, aggregations, and analytical workloads.

Still, the friction remains.

Snowflake Postgres may appear to be a small announcement, but it quietly acknowledges this long-standing tension instead of ignoring it.

Before answering what this changes, it helps to be clear about what Snowflake Postgres actually is, and just as importantly, what it is not.

What Exactly Is Snowflake Postgres?

Snowflake Postgres is a fully managed PostgreSQL service provisioned directly from a Snowflake account.

In practical terms:

It is a fully compatible PostgreSQL database
Existing Postgres clients and drivers work without change
Snowflake manages scaling, availability, security, and governance

From a developer’s point of view, everything feels familiar.

From an operational point of view, a large amount of infrastructure responsibility quietly disappears.

There is no need to separately manage high availability, failover strategies, security patching, or governance layers. Snowflake takes ownership of these concerns in the same way it already does for analytical workloads.

This offering builds on Snowflake’s 2025 acquisition of Crunchy Data, a company known for running PostgreSQL reliably at enterprise scale.

Official announcement:
Snowflake acquires Crunchy Data 250m AI Database Capabilities

That acquisition now feels less like an isolated move and more like a foundational step toward something larger.

Snowflake is not replacing Postgres.

It is not turning itself into a traditional OLTP platform.

Instead, it is bringing Postgres into the same control plane that already supports analytics and AI workloads. That distinction is subtle, but important.

Why This Was Needed

For many years, most data architectures followed a sensible and widely accepted separation:

PostgreSQL (or similar systems) for application and transactional workloads
Snowflake for analytics, reporting, and business intelligence

Each system did what it was best at, and for a long time, this model worked well.

However, as data volumes grew and expectations shifted toward fresher insights, real-time decision-making, and AI-driven use cases, the cost of this separation became harder to ignore.

The Day-to-Day Reality

In practice, this separation often meant:

Continuous replication of data from Postgres into Snowflake
Lag between production events and analytical visibility
Multiple security and governance models to configure and audit
More systems to monitor, scale, and troubleshoot
Extra logic written solely to keep data reasonably fresh

None of this was poor engineering.

It was simply the best architecture available when transactional and analytical systems lived on different platforms and were owned by different operational models.

What Snowflake Postgres Changes

Snowflake Postgres shortens the distance between two critical points in a data system:

where data is created
where data is analyzed or used by AI

In traditional setups, these points are separated by layers of CDC tools, pipelines, schedulers, and orchestration logic. Each layer introduces latency, operational risk, and cognitive overhead.

Snowflake Postgres reduces that distance.

This shift is not primarily about raw performance.

It is about simplifying the system as a whole.

Practical Impact

At a practical level, this leads to:

Reduced reliance on constant replication of transactional data
Fewer pipelines, which directly means fewer failure points
A single governance and security model across workloads
Easier access to fresher operational data for analytics and AI

When combined with open table formats such as Apache Iceberg and AI capabilities like Snowflake Cortex, this approach starts to resemble a unified data foundation rather than a collection of loosely connected systems.

Snowflake’s PG_LAKE initiative reinforces this direction by exploring deeper integration between Postgres and the lakehouse model:
PG Lake Postgres Lakehouse Integration

Why This Matters for Data Engineers

Cleaner Pipelines and Lower Operational Overhead

Traditional architectures rely on CDC tools, ETL or ELT pipelines, and orchestrators to keep Postgres and Snowflake in sync. Each layer is necessary, but together they add operational weight.

With Snowflake Postgres, operational data and analytical workloads share the same platform. This reduces the number of moving parts and allows data engineers to spend more time on modeling, optimization, and business use cases rather than pipeline maintenance.

Fewer Sync Issues and More Trust in Data

Separated systems commonly introduce late-arriving data, schema drift, and partial updates after failures.

With Postgres integrated inside Snowflake, governance and security remain consistent, duplication is reduced, and analytical outputs more accurately reflect operational reality. This directly improves trust in dashboards, reports, and downstream decisions.

Faster Analytics and AI Experimentation

Modern use cases such as personalization, fraud detection, and near real-time analytics depend on fresh operational data.

Snowflake Postgres narrows the gap between production data and analytical access, making it easier to experiment and iterate without redesigning data movement pipelines for every new idea.

Unified Security and Governance

Managing access controls, auditing, and compliance across platforms is expensive and error-prone.

With Snowflake Postgres:

Authentication is centralized
Permission models are unified
Auditing and lineage follow a consistent approach

This simplifies compliance with standards such as SOC 2 and GDPR while reducing operational burden.

Real-World Context: Why the Split Existed

Many modern companies have followed the Postgres-for-OLTP and Snowflake-for-OLAP pattern for good reasons. That split enabled scale, protected production systems, and unlocked advanced analytics.

Snowflake Postgres does not invalidate these architectures.

Instead, it offers an alternative when tighter integration between operational data, analytics, and AI becomes valuable.

Trade-Offs (Because There Are Always Trade-Offs)

Snowflake Postgres is not a universal solution.

It does not mean:

every OLTP workload should move into Snowflake
traditional Postgres deployments will disappear
existing architectures need immediate rewrites

For applications with strict latency requirements or deeply embedded infrastructure, standalone Postgres will continue to make sense.

What Snowflake Postgres offers is choice.

For data-heavy products where operational and analytical workloads are tightly linked, teams can now design systems with fewer boundaries and fewer moving parts.

Good architecture is not about trends.

It is about choosing the setup that introduces the least friction.

Who This Is a No-Brainer For

Snowflake Postgres is especially compelling for:

New products already planning to use Postgres for OLTP workloads
Teams building data-driven or AI-heavy applications from day one
Organizations that want strong governance without stitching together multiple platforms

For greenfield systems, starting with Postgres inside Snowflake can remove entire classes of future complexity.

A Sensible Migration Path for Existing Systems

For teams considering this architecture, the safest approach is gradual and low-risk:

Start with development environments to validate tooling and behavior
Extend to QA or staging for performance and governance testing
Gradually introduce production workloads, starting with non-critical services

This path allows teams to evaluate real-world benefits without forcing large, disruptive migrations.

Final Thought

Snowflake Postgres may look like a small checkbox feature today.

But in hindsight, it could mark the point where analytics platforms stopped being passive data stores and started becoming active system backbones.

Not by replacing Postgres, but by finally removing the wall between transactions, analytics, and AI.

And that is a shift worth paying attention to.

What is a Build Process in React ( or in any framework for that matter? )

Shrinivas Vishnupurikar — Thu, 08 Aug 2024 11:51:51 +0000

[ Technology ]: ReactJS – Article #1

The Frameworks simplify the Development for Engineers and this is my attempt to simply the Behind-The-Scenes functioning of the ReactJS.

Story time

I've started out with ReactJS. Yup, I really have. It's like a dream being delay by 2 years when I was passionate about UI / UX Designing and Front-End Development before I dived into Data Science. ( I still am as passionate as I was 2 years ago.)

I'm now an Intern at a company ( that call themselves a startup because, its culture is more of a start-up's than that of company's ) and today, on my first day, I literally had nothing to do since, my TL ( Team Lead ) was not coming to the office as he was occupied with some meeting.

Did I let he time slip off my hands. Absolutely not.
The probability of me getting a task / project to put my Data analytics skills to test was fairly low. Hence I resorted to get my hands dirty in development. I could sense that this might be the best time to started with ReactJS.

What is React?

React is a verb ( pun intended ). But in the context of development technologies, "The library for web and native user interfaces", claims the official website of ReactJS.

Now if you've been around the development ecosystem, you must have heard about other 2 competitors or rather the siblings of ReactJS, which are Angular and VueJS.

Here's a short comparison of the 3 of the most popular Front-End technologies.

Core Concept	Library focused on UI	Full-fledged framework	Progressive framework
Data Binding	One-way data flow	Two-way data binding	Two-way data binding (optional)
Component Structure	Custom components	Directives and components	Components
Learning Curve	Moderate	Steep	Gentle
Performance	High (Virtual DOM)	Can be slower due to two-way data binding	High (Optimized rendering)
Scalability	Excellent, suitable for large-scale apps	Strong support for large-scale enterprise apps	Good scalability, but might require additional libraries for complex projects
Community and Ecosystem	Largest community, rich ecosystem	Large community, strong ecosystem	Growing community, good ecosystem
Flexibility	High, can be used with other libraries/frameworks	Less flexible due to rigid structure	Flexible, can be used incrementally

Why should we use ReactJS when we have plain HTML and JS?

Demerits of Plain HTML and JS.

Following are the problem you're going to face if you use:

Difficulty Maintaining Large Applications:

Plain HTML and JS lacks a structured approach to organizing code.
Complex applications can lead to a tangled mess of logic and UI manipulation within event listeners and script files.
This makes it challenging to understand, modify, and debug code as the application grows.

Inefficient DOM Manipulation:

Directly manipulating the DOM in JS can be inefficient.
Every state change might trigger a complete re-rendering of the HTML structure, even for minor UI updates.
This can lead to performance bottlenecks as the application complexity increases.

Limited Reusability:

Building reusable UI components with plain HTML and JS can be cumbersome.
You might end up copying and pasting code snippets across different parts of your application.
This makes it difficult to maintain consistency and implement changes efficiently.

Complex State Management:

Managing the state of an application (data that controls UI behavior) becomes difficult with plain HTML and JS.
Keeping track of data changes and their corresponding UI updates can become messy and error-prone, especially for complex data flows.

How ReactJS comes to the Rescue.

ReactJS addresses these limitations by offering a component-based architecture, virtual DOM for efficient updates, and a rich ecosystem for managing complex UIs and application state.

Improved Maintainability:

React's component-based architecture and declarative approach lead to cleaner and more maintainable codebases, especially for large-scale applications.

Enhanced Performance:

The virtual DOM and efficient rendering mechanisms in React contribute to smoother and faster user experiences, even for complex web applications.

Code Reusability:

React's component model promotes code reusability, allowing you to build modular UI components that can be easily shared and combined across your application.

The Difference makes the Difference.

When I created my first components, I asked myself, "What exactly" is this? Is it HTML or JS?

I've embedded JS into the HTML via <script> </script> element or <script src="index.js"> </script> element. But writing HTML inside a JS file feels weird or rather different.

I tried and writing the HTML inside the JS file of a non-React project and guess what it didn't go well.

Then I learned that this special syntax ( HTML like inside of JS file ) is called as JSX JavaScript XML and is an extension of JavaScript.

If the code that browsers understand is ultimately the plain HTML and JS, it means that there are some operations performed on the JSX ( syntactic sugar for building complex applications with ease ) that we write.

This Behind The Scenes Operations itself is called a Build Process.

The high-level idea of a build process is to transform your development code into an optimized version ready for deployment in a production environment.

While specific tools and configurations may vary depending on the technology stack, the general concepts and goals of the build process apply universally across frontend web development.

What is a Build Process in ReactJS?

We've learnt that the high-level idea remain the same but the several phases in the Build Process of React are as follows:

Bundling:
- Imagine your React application consists of numerous JavaScript files, CSS stylesheets, and potentially image assets.

A bundler like Webpack takes all these separate files and combines them into a smaller number of optimized bundles.

Transpilation:
- Transpilation involves converting this modern code (JSX) into plain JavaScript (ES5 or a compatible version) that can run on a wider range of browsers.
Minification:
- Minification shrinks the file size of your bundled code by removing unnecessary characters like whitespace, comments, and long variable/function names.
Optimization:
- The build process might involve additional optimizations like tree-shaking, which removes unused code from your final bundles.
Production Mode:
- The development mode offers features like source maps (for easier debugging) and detailed error messages.

In contrast, the production mode focuses on optimization by enabling minification, tree-shaking, and other performance enhancements.

`react-scripts`: The Wolf of React Project

react-scripts is an internal package used by Create React App (CRA) to handle the behind-the-scenes functionalities in a React project.

It's not directly interacted with by developers most of the time, but it's essential for development efficiency.

Here's what react-scripts is responsible for:

Bundling and Transpilation
Development Server and Hot Reloading
Testing
Building for Production

The 3 of the most significant tasks the react-scripts does is as follows which we'll understand in much detail:

Bundling:
- Imagine your React application consists of numerous JavaScript files, CSS stylesheets, and potentially image assets.
- A bundler like Webpack takes all these separate files and combines them into a smaller number of optimized bundles.
- This reduces the number of HTTP requests a browser needs to make, improving website loading speed.
Transpilation:
- Modern JavaScript features like JSX syntax used in React might not be understood by older browsers.
- Transpilation involves converting this modern code into plain JavaScript (ES5 or a compatible version) that can run on a wider range of browsers.
- Tools like Babel are commonly used for transpilation in React.
Minification:
- Minification, also known as minimization, is a technique applied to code to reduce its file size without affecting its functionality.
- This is particularly beneficial for React applications deployed to production, as smaller file sizes translate to faster loading times for web pages.

Here's how minification works:

Removing Unnecessary Characters:
- Minifiers eliminate whitespace characters like spaces, newlines, and tabs from the code. This might seem insignificant for small files, but in large React projects, it can lead to a noticeable reduction in size.
Shortening Variable and Function Names:
- Minifiers often replace descriptive variable and function names with shorter, single-letter names. While this makes the code less readable for humans, it significantly reduces file size.
Removing Comments:
- Comments are essential for documenting and understanding code during development. However, in production, they're not required for the code to function. Minifiers typically remove comments to further minimize file size.

Conclusion

This is the learning of only 1 day summarized.

I would have been able to build much more than this, if I utilized GenAI tools. I did GenAI tools either way but for learning purpose.

I believe asking this right set of questions and then understanding the concepts in true depth will set you apart from the one's who automate the development.

In the interviews it is the understanding and clarity of the concept that is sought, rather than your coding speed because, either way it's going to be automated to an extent.

Thus the only X-Factor of you being a great software engineering lies in your knowledge to at least validate and verify whether the outcome of GenAI model caters to your tech needs or not.

If you think that my content is valuable or have any feedback,
do let me by reaching out to my following social media handles that you'll discover in my profile and the follows:

LinkedIn: https://www.linkedin.com/in/shrinivasv73/

Twitter (X): https://twitter.com/shrinivasv73

Instagram: https://www.instagram.com/shrinivasv73/

Email: shrinivasv73@gmail.com

🫡 This is Shrinivas, Signing Off!

How to deploy ReactJS Apps (built using Vite) on GitHub Pages?

Shrinivas Vishnupurikar — Thu, 08 Aug 2024 10:01:09 +0000

[ Technology ]: ReactJS – Article #2

Differences in build tools lead to hurdles, especially in deployment. Today we'll spotlight one such and solve it for you.

You gather requirements, you design, you develop and you test. Great! Now you'll barely move to deployment.

Just kidding. I know, deploying dynamic, feature rich, robust ( and 50 other adjectives ) apps that use backend and databases is quite tricky. Hence, as a starter we're going to learn how to deploy a simple ( too simple ) ReactJS App.

Where are we deploying it? GitHub Pages! Yes, GitHub is not only scoped to hosting the project's source code or GitHub Pages to host static websites that are purely HTML5 + CSS3 + JS.

Where else could you deploy your front-end apps in general?
The list of platforms is as follows:

Netlify (Best for beginners to get started)
Vercel (Best for Advanced projects)
Surge

Creating a ReactJS App! The Confusion Spiral.

If you're living in 2024, I hope you don't use npx create-react-app <app-name> to create ReactJS App.

If you look at the ReactJS's official docs, you'll see that there's no option to create a pure ReactJS App, but they'll insist you to create project using Next.js, Remix, Gatsby and more.

Why the sudden shift away from npx create-react-app?
Although, it was a fantastic starting point for many React developers, the landscape of frontend development has evolved significantly due it's limitations with respect to the following:

Opinionated Structure
Difficulty in Customization
Performance Overhead

Thus developers resorted to other and better alternatives which were as follows:

Vite: This build tool has gained immense popularity due to its lightning-fast development server, efficient bundling, and flexibility.
Next.js: While primarily a React framework, it offers a robust set of features for building web applications, including server-side rendering, static site generation, and routing.
Gatsby: Another popular static site generator built on React, offering performance and SEO benefits.

I now get it that NextJS Apps are in the end ReactJS Apps, because, NextJS is a framework built on-top-of ReactJS library.

But either way, if you didn't want to create a framework app (NextJS App) but a library app (ReactJS App), ( which is something I did ), you could using Vite build tool to do so.

Creating a ReactJS App using Vite
You can use the following command in the terminal to create a React App that uses JavaScript ( by default ) in one-go.

If you didn't know, React officially supports TypeScript.

npm create vite@latest deploy-reactjs-app-with-vite -- --template react

Or you could use the following command to create a ReactJS Apps by step-by-step process:

npm create vite@latest

Setting the GitHub Up!

Let's create a Remote GitHub Repository. Got to your GitHub Profile and create a remote GitHub Repository and you should be able to see the following interface for an empty repo:

If you go to the Settings => Pages tab of your GitHub repo, you'll see the following interface:

It shows either main branch or None. Right now, you don't have to be concerned about it, but be aware that we're going to revisit this page again.

Once you work on the project, I want you to execute the following commands (sequentially, of course) in your working directory:

Initialize an empty git repo on your local system.

git init

Track All file by staging them.

git add .

Create a Checkpoint that takes a snapshot of current progress of the stages file(s) in above:

 git commit -m "Added Project Files"

Connect local repo (on our system) with the remote repo (the one created on GitHub).

 git remote add origin url_of_the_remote_git_repo_ending_with_.git

Upload the progress made till our checkpoint from local repo to the remote repo.

 git push -u origin main

Once you successfully push the changes to the remote repository, you'll the following output in your terminal:

Now, if you refresh the GitHub Repository the interface would look something as follows:

Installing Dependencies and Crying over Errors:

Step 1: Installing `gh-pages` package

Right now we have just made 1 checkpoint and pushed it to our remote repo. We have NOT started to work on the deployment! YET!

Head to your working directory in your integrated terminal and fire up the following command:

npm install gh-pages --save-dev

This command will install and save gh-pages (github-pages) as a dev dependency or our project.

Dependencies are packages required for the application to run in production. Dev dependencies are packages needed only for development and testing.

Once completed, the terminal will look as follows (provides some positive acknowledgement):

Step 2: Update package.json

You have to add the following code to your package.json file.

{
    "homepage": "https://<username>.github.io/<repository>",
    "scripts": {
        "predeploy": "npm run build",
        "deploy": "gh-pages -d build",
        // ... other scripts
    }
    // other scripts
}

The above scripts are not specified during the project installation via Vite as they are gh-pages specific scripts.

The explanation for each is as follows:

homepage: The "homepage" field in the package.json file of a React project specifies the URL at which your app will be hosted. This field is particularly important when deploying a React application to GitHub Pages or any other static site hosting service that serves the site from a subdirectory.

Do update the values of <username> and <repository> of the homepage property. In my case the values are ShrinivasV73 and Deploy-ReactJS-App-With-Vite respectively.

It is good to consider that the value are case-sensitive and should be placed accordingly.

"scripts" field in package.json allows you to define custom scripts that can be run using npm run .
- "predeploy": "npm run build": This script runs automatically before the deploy script and it triggers the build process of your project.
- "deploy": "gh-pages -d build": This script is used to deploy your built application to GitHub Pages. It uses the gh-pages package to publish the contents of the specified directory (build in this case) to the gh-pages branch of your GitHub repository.

Step 3: Deploy The App

Now that we've updated scripts as well, it is time to deploy the project. Head to the terminal and fire-up the following command to process:

npm run deploy

And boom 💥, WE GET AN ERROR!

Error: ENOENT: no such file or directory, stat '<working-drectory>/build'

npm (node package manager) is trying to access the folder named build via the command npm run deploy which is actually npm run gh-pages -d build executed behind the scenes.

But it can't find any.

Bug Fix: #1 Updating the `package.json` file

Remember we didn't create a build directory by ourselves throughout this journey.

Vite outputs the build files to a dist directory by default and not the built directory, whereas tools like CRA (create-react-apps) use a build directory.

( This is exactly where the underlying functions and processes of different build tools manifests. )

We simply have to replace the build with dist in the deploy script inside the package.json file.

{ 
    "homepage": "https://<username>.github.io/<repository>", 
    "scripts": { 
        "predeploy": "npm run build", 
        "deploy": "gh-pages -d dist", 
        // ... other scripts } 
    // other scripts 
}

Bug Fix #2: Updating the vite.config.js

By default your vite.config.js file looks as follows:

import { defineConfig } from 'vite';
import react from '@vitejs/plugin-react';

export default defineConfig({
    plugins: [react()],
});

To make our app functions as expected without any bugs after successful deployment, we need to add base: '/Deploy-ReactJS-App-With-Vite/' to the object, which is passed to the defineConfig() method.

import { defineConfig } from 'vite';
import react from '@vitejs/plugin-react';

export default defineConfig({
    plugins: [react()],
    base: '/Deploy-ReactJS-App-With-Vite/',
});

Setting the base property in vite.config.js is crucial for deploying a Vite-built app to a subdirectory, such as on GitHub Pages. It ensures that all asset paths are correctly prefixed with the subdirectory path, preventing broken links and missing resources.

Example:

Our repository is named Deploy-ReactJS-App-With-Vite and you are deploying it to GitHub Pages. The URL for your site will be https://username.github.io/Deploy-ReactJS-App-With-Vite/
If you don’t set the base property, the browser will try to load assets from https://username.github.io/ instead of https://username.github.io/Deploy-ReactJS-App-With-Vite/, resulting in missing resources.

Retry Deploying The App

Once you make the necessary changes to package.json file and vite.config.js file it's time to git add, commit, push. AGAIN!

Head to the working directory in the terminal and try deploying the app again:

npm run deploy

And this time when the app actually gets deployed to the Github pages, you'll see again, the positive acknowledgment to the terminal itself as follows:

If you refresh your GitHub repo go to the Settings => Pages tab of it, you'll see a new gh-pages branch added to the branches.

How do you access the app on web? Remember the value of homepage property in the package.json file? Yup! That's it.

In our case, it is https://ShrinivasV73.github.io/Deploy-ReactJS-App-With-Vite/

Conclusions

Congratulations! You've successfully learned how to deploy ReactJS App with Vite on GitHub Pages.

My recommendation for you folks would be to experiment as follows in different ways:

Create different font-end / full-stack apps like Angular or Vue.js and see how the configurations needs to be updated according to them.
Create different React Apps like Next.js or Remix or Gatsby
Use different platforms to deploy your front-end applications like vercel or netlify to see which option suits best for which use case.

In a nutshell, I started with experimentation and summarised my learning in this article. May be its your time to do so. Cheers!

If you think that my content is valuable or have any feedback,
do let me by reaching out to my following social media handles that you'll discover in my profile and the follows:

LinkedIn: https://www.linkedin.com/in/shrinivasv73/

Twitter (X): https://twitter.com/shrinivasv73

Instagram: https://www.instagram.com/shrinivasv73/

Email: shrinivasv73@gmail.com

Forem: Shrinivas Vishnupurikar

Schema, COPY, MERGE, and Immutability — A First-Principles Guide for Data Engineers

1. What Is a “Schema” — First Principles

1.1 Schema-on-Write (Validate First, Store Later)

Definition

Intuition

Story Example

Why This Pattern Exists

Trade-off

1.2 Schema-on-Read (Store First, Decide Later)

Definition

Story Example

Why This Pattern Exists

Trade-off

2. COPY vs MERGE — What Problem Do They Solve?

2.1 COPY (Append-Only Thinking)

Intuition

Story Example

Why This Pattern Exists

Trade-off

2.2 MERGE (State-Aware Thinking)

Intuition

Story Example

Why This Pattern Exists

Trade-off

3. All Valid Combinations — Schema × Write Pattern

3.1 Schema-on-Write + COPY

3.2 Schema-on-Write + MERGE

3.3 Schema-on-Read + COPY

3.4 Schema-on-Read + MERGE

4. Where Partition Immutability Enters the Picture

5. Immutability in Open Table Formats

Apache Iceberg (Primary Focus)

Delta Lake (Secondary)

Apache Hudi (Secondary)

6. Closing Note

Snowflake + Postgres: A Small Feature That Signals a Big Shift

The Story Every Data Engineer Wonders About

Acronyms Used in This Blog

The Elephant in the Room

What Exactly Is Snowflake Postgres?

Why This Was Needed

The Day-to-Day Reality

What Snowflake Postgres Changes

Practical Impact

Why This Matters for Data Engineers

Cleaner Pipelines and Lower Operational Overhead

Fewer Sync Issues and More Trust in Data

Faster Analytics and AI Experimentation

Unified Security and Governance

Real-World Context: Why the Split Existed

Trade-Offs (Because There Are Always Trade-Offs)

Who This Is a No-Brainer For

A Sensible Migration Path for Existing Systems

Final Thought

What is a Build Process in React ( or in any framework for that matter? )

[ Technology ]: ReactJS – Article #1

Story time

What is React?

Why should we use ReactJS when we have plain HTML and JS?

Demerits of Plain HTML and JS.

How ReactJS comes to the Rescue.

The Difference makes the Difference.

What is a Build Process in ReactJS?

react-scripts: The Wolf of React Project

Conclusion

How to deploy ReactJS Apps (built using Vite) on GitHub Pages?

[ Technology ]: ReactJS – Article #2

Creating a ReactJS App! The Confusion Spiral.

Setting the GitHub Up!

Installing Dependencies and Crying over Errors:

Step 1: Installing gh-pages package

Step 2: Update package.json

Step 3: Deploy The App

Bug Fix: #1 Updating the package.json file

Bug Fix #2: Updating the vite.config.js

Retry Deploying The App

Conclusions

`react-scripts`: The Wolf of React Project

Step 1: Installing `gh-pages` package

Bug Fix: #1 Updating the `package.json` file