Forem: sachinkg12

HeapLens — Analyze Java Heap Dumps Inside VS Code with AI

sachinkg12 — Fri, 03 Apr 2026 01:47:32 +0000

Heap dump analysis is one of those tasks that usually pulls you out of your editor. You generate an .hprof file, open a separate tool, wait for it to parse, and then navigate an unfamiliar UI to find what's eating memory.

I built HeapLens to bring that workflow into VS Code. It's a free, open-source extension that lets you open .hprof files directly in your editor and get interactive analysis — no context switching, no separate tool.

As of today, HeapLens is the only VS Code extension on the marketplace that does heap dump analysis.

What it actually does

You open a .hprof file in VS Code. HeapLens takes over with a custom editor and a Rust-powered backend that memory-maps the file and starts analyzing. Within seconds, you get a multi-tab interface:

Overview — Heap statistics at a glance with a treemap visualization of your largest objects. You immediately see where memory is going.

Class Histogram — Every class in the heap, sortable by instance count, shallow size, or retained size. Click a class to drill into individual instances.

Dominator Tree — The hierarchy of object ownership. Expand nodes to explore who's holding onto memory and why. Each node shows retained size bars so you can visually spot the heaviest branches.

Leak Suspects — Objects or classes retaining a disproportionate share of the heap, flagged automatically. You can adjust the detection threshold.

Waste Analysis — HeapLens detects duplicate strings, empty collections, over-allocated arrays, and boxed primitives — the kind of low-hanging fruit that quietly bloats your heap.

Compare — Load two heap dumps and diff them. See which classes grew, which shrank, which leak suspects appeared or resolved. Export the diff as markdown or CSV.

AI-assisted analysis

HeapLens has a built-in chat tab where you can ask questions about your heap dump in plain English. It works with 10 LLM providers — Anthropic, OpenAI, Google Gemini, Ollama for local models, and others. You bring your own API key (or use Ollama for free).

The chat isn't just a wrapper around a prompt. When the LLM suggests a query, you can run it directly against your heap data from the chat interface. It bridges the gap between "I think there's a leak in my connection pool" and actually finding the objects responsible.

There's also a Copilot Chat integration if you use GitHub Copilot — you can @heaplens /leaks right from the Copilot panel.

Why Rust

The backend is written in Rust with async I/O. It memory-maps HPROF files and uses a compressed sparse row (CSR) graph representation for the object reference graph. On my M2 Mac, it parses a 1.5 GB heap dump in under a second.

Performance matters here because heap dumps from real production systems are big — often 2-10 GB. A tool that takes minutes to open a file doesn't get used. HeapLens is designed to feel instant for the files you actually encounter.

Who this is for

If you're a Java developer who works with heap dumps — whether you're debugging OutOfMemoryError in production, profiling memory usage during development, or comparing heap snapshots across releases — HeapLens is built to make that faster and easier without leaving your editor.

It also supports Android HPROF files from adb, so Android developers can use it too.

Try it

Install from the VS Code Marketplace or Open VSX. It's free and open source.

If you work with heap dumps and have feedback, I'd appreciate hearing from you — open an issue on GitHub or drop a comment below.

HeapLens is open source under the Apache 2.0 license. If you find it useful, a star on GitHub or a rating on the marketplace goes a long way.

Why double Loses Precision and How to Avoid It in Java

sachinkg12 — Mon, 27 Jan 2025 08:10:11 +0000

When working with floating-point numbers in Java, you might notice that double occasionally produces unexpected or imprecise results. This behavior can lead to bugs, especially in financial applications or scenarios requiring high precision.

In this post, we’ll dive deep into the root cause of this issue, explain how to avoid it, provide a working example, and explore whether newer Java versions offer better alternatives.

Why Does `double` Lose Precision?

1. IEEE 754 Floating-Point Standard

The double data type in Java follows the IEEE 754 standard for floating-point arithmetic. It represents numbers in binary format using:

1 bit for the sign,
11 bits for the exponent,
52 bits for the fraction (mantissa).

This binary representation introduces limitations:

Finite Precision: double can only represent numbers up to 15–17 decimal digits accurately.
Rounding Errors: Many decimal fractions (e.g., 0.1) cannot be precisely represented in binary, leading to rounding errors.

For example, in binary:

0.1 becomes an infinitely repeating fraction, which gets truncated for storage, introducing slight imprecision.

2. Accumulated Errors in Arithmetic

Operations involving double can accumulate errors:

Repeated additions/subtractions amplify rounding errors.
Multiplications/divisions may lose precision due to truncation.

This behavior is inherent to floating-point arithmetic and is not unique to Java.

Working Example: Precision Loss with `double`

Here’s an example demonstrating the issue:

public class DoublePrecisionLoss {
    public static void main(String[] args) {
        double num1 = 0.1;
        double num2 = 0.2;
        double sum = num1 + num2;

        System.out.println("Expected sum: 0.3");
        System.out.println("Actual sum: " + sum);

        // Comparison
        if (sum == 0.3) {
            System.out.println("Sum is equal to 0.3");
        } else {
            System.out.println("Sum is NOT equal to 0.3");
        }
    }
}

Output:

Expected sum: 0.3
Actual sum: 0.30000000000000004
Sum is NOT equal to 0.3

The result, 0.30000000000000004, highlights the rounding error caused by binary representation. Even though the difference is minuscule, it can lead to significant issues in critical systems.

How to Avoid Precision Loss

1. Use `BigDecimal` for Precise Calculations

The BigDecimal class in Java provides arbitrary-precision arithmetic, making it ideal for scenarios requiring high accuracy, such as financial calculations.

Example Using `BigDecimal`:

import java.math.BigDecimal;

public class BigDecimalExample {
    public static void main(String[] args) {
        BigDecimal num1 = new BigDecimal("0.1");
        BigDecimal num2 = new BigDecimal("0.2");
        BigDecimal sum = num1.add(num2);

        System.out.println("Expected sum: 0.3");
        System.out.println("Actual sum: " + sum);

        // Comparison
        if (sum.compareTo(new BigDecimal("0.3")) == 0) {
            System.out.println("Sum is equal to 0.3");
        } else {
            System.out.println("Sum is NOT equal to 0.3");
        }
    }
}

Output:

Expected sum: 0.3
Actual sum: 0.3
Sum is equal to 0.3

By using BigDecimal, the precision issues are eliminated, and comparisons yield the correct result.

2. Compare Using an Epsilon Value

Another approach to handling precision loss is comparing floating-point numbers with a tolerance (epsilon). This method checks if the numbers are “close enough” instead of relying on exact equality.

Example Using Epsilon Comparison:

public class EpsilonComparison {
    public static void main(String[] args) {
        double num1 = 0.1;
        double num2 = 0.2;
        double sum = num1 + num2;
        double epsilon = 1e-9; // Define a small tolerance value

        System.out.println("Expected sum: 0.3");
        System.out.println("Actual sum: " + sum);

        // Comparison with epsilon
        if (Math.abs(sum - 0.3) < epsilon) {
            System.out.println("Sum is approximately equal to 0.3");
        } else {
            System.out.println("Sum is NOT approximately equal to 0.3");
        }
    }
}

Output:

Expected sum: 0.3
Actual sum: 0.30000000000000004
Sum is approximately equal to 0.3

Why Use Epsilon Comparison?

Flexibility: It allows for minor differences due to rounding errors.
Simplicity: This method doesn’t require external libraries and is efficient.

Using Apache Commons Math for Enhanced Precision

Apache Commons Math is a library designed for complex mathematical computations. While it does not provide arbitrary-precision arithmetic like BigDecimal, it offers utilities that simplify numerical operations and minimize floating-point errors in certain scenarios.

Example: Using `Precision.equals` for Comparison

import org.apache.commons.math3.util.Precision;

public class ApacheCommonsExample {
    public static void main(String[] args) {
        double num1 = 0.1;
        double num2 = 0.2;
        double sum = num1 + num2;

        System.out.println("Expected sum: 0.3");
        System.out.println("Actual sum: " + sum);

        // Comparison with tolerance
        if (Precision.equals(sum, 0.3, 1e-9)) {
            System.out.println("Sum is equal to 0.3");
        } else {
            System.out.println("Sum is NOT equal to 0.3");
        }
    }
}

Output:

Expected sum: 0.3
Actual sum: 0.30000000000000004
Sum is equal to 0.3

Why Use Apache Commons Math?

Simplifies Comparisons: Precision.equals allows comparisons with a specified tolerance, making it easy to handle rounding errors.
Lightweight: The library provides focused tools for numerical computations without the overhead of BigDecimal.

Summary

Understand the Limitations: double is not inherently flawed but is unsuitable for high-precision tasks due to its binary floating-point representation.
Use BigDecimal When Necessary: For financial or critical calculations, BigDecimal ensures precision but may impact performance.
Leverage Libraries: Apache Commons Math provides utilities like Precision.equals to handle floating-point comparisons effectively.

By understanding the nuances of double and its alternatives, you can write more robust and accurate Java applications.

Let me know in the comments if you have encountered precision issues with double and how you tackled them! 😊

Understanding the Singleton Pattern in Java

sachinkg12 — Thu, 23 Jan 2025 17:40:46 +0000

The Singleton Pattern is one of the most commonly used design patterns in Java. It ensures that a class has only one instance and provides a global point of access to that instance.

Think of it like a manager in a team, there is only one manager for a specific team, and everyone in the other department communicates through them.

Let’s break it down in simple terms and understand how you can implement this in Java.

Why Use the Singleton Pattern?

Single Instance: To ensure that there is only one instance of a class in your application.
- For example, think of a database connection pool, having a single instance ensures efficient reuse of connections without repeatedly creating and destroying them.
- Another example is a printer spooler that manages print jobs across the entire system, ensuring there’s no conflict between users.
Global Access Point: To provide a single, shared point of access to the instance.
Resource Management: To manage shared resources like configurations, logging, or thread pools efficiently.

How to Implement Singleton Pattern in Java

1. Lazy Initialization Singleton

This approach initializes the instance only when it’s needed.

import java.io.Serializable;
public class LazySingleton implements Serializable {
    private static LazySingleton instance;

    private LazySingleton() {}

    public static LazySingleton getInstance() {
        if (instance == null) {
            instance = new LazySingleton();
        }
        return instance;
    }
}

Visualization:

Imagine a library where a librarian only sets up a table when a visitor arrives:

The if (instance == null) check is like the librarian checking if there’s already a table set up.
If no table is set, they quickly arrange one (create the instance).
The next visitors reuse the same table without additional setup.

Problems and Fixes:

Serialization Issue:

Breaking Example: Without readResolve, deserialization creates a new instance.

import java.io.*;

public class TestSerialization {
    public static void main(String[] args) throws Exception {
        LazySingleton instance1 = LazySingleton.getInstance();

        // Serialize the instance
        ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream("singleton.ser"));
        oos.writeObject(instance1);
        oos.close();

        // Deserialize the instance
        ObjectInputStream ois = new ObjectInputStream(new FileInputStream("singleton.ser"));
        LazySingleton instance2 = (LazySingleton) ois.readObject();
        ois.close();

        System.out.println(instance1 == instance2); // false
    }
}

Fix: Add readResolve to ensure the same instance is returned.

private Object readResolve() {
    return getInstance();
}

Reflection Issue:

Breaking Example: Reflection can call the private constructor.

import java.lang.reflect.Constructor;

public class TestReflection {
    public static void main(String[] args) throws Exception {
        LazySingleton instance1 = LazySingleton.getInstance();

        Constructor<LazySingleton> constructor = LazySingleton.class.getDeclaredConstructor();
        constructor.setAccessible(true);
        LazySingleton instance2 = constructor.newInstance();

        System.out.println(instance1 == instance2); // false
    }
}

Fix: Add a guard in the constructor to throw an exception if an instance already exists.

private LazySingleton() {
    if (instance != null) {
        throw new IllegalStateException("Instance already created");
    }
}

2. Thread-Safe Singleton

To make the Singleton thread-safe, you can use the synchronized keyword.

public class ThreadSafeSingleton {
    private static ThreadSafeSingleton instance;

    private ThreadSafeSingleton() {}

    public static synchronized ThreadSafeSingleton getInstance() {
        if (instance == null) {
            instance = new ThreadSafeSingleton();
        }
        return instance;
    }
}

Visualization:

Imagine a bakery where only one oven can bake bread at a time:

The synchronized keyword acts as a lock on the oven, ensuring only one baker can use it to bake bread.
When the first loaf (instance) is ready, all other bakers can simply reuse it.

Problems and Fixes:

Serialization Issue:

Breaking Example: Without readResolve, deserialization creates a new instance.

public class TestPerformance {
public static void main(String[] args) {
    Runnable task = () -> {
        for (int i = 0; i < 1000; i++) {
            ThreadSafeSingleton.getInstance();
        }
    };

    Thread t1 = new Thread(task);
    Thread t2 = new Thread(task);

    long start = System.nanoTime();
    t1.start();
    t2.start();

    try {
        t1.join();
        t2.join();
    } catch (InterruptedException e) {
        e.printStackTrace();
    }

    long end = System.nanoTime();
    System.out.println("Execution Time: " + (end - start) + " ns");
}
}

Fix: Use double-checked locking to reduce synchronization overhead:

public static ThreadSafeSingleton getInstance() {
    if (instance == null) {
        synchronized (ThreadSafeSingleton.class) {
            if (instance == null) {
                instance = new ThreadSafeSingleton();
            }
        }
    }
    return instance;
}

Reflection Issue:

Fix: Add a guard in the constructor:

private ThreadSafeSingleton() {
    if (instance != null) {
        throw new IllegalStateException("Instance already created");
    }
}

3. Double-Checked Locking Singleton

This approach minimizes the performance cost of synchronization.

public class DoubleCheckedLockingSingleton {
    private static volatile DoubleCheckedLockingSingleton instance;

    private DoubleCheckedLockingSingleton() {}

    public static DoubleCheckedLockingSingleton getInstance() {
        if (instance == null) {
            synchronized (DoubleCheckedLockingSingleton.class) {
                if (instance == null) {
                    instance = new DoubleCheckedLockingSingleton();
                }
            }
        }
        return instance;
    }
}

Visualization:

Imagine a bank vault:

The first if condition is like a security camera checking if the vault is already open.
The synchronized block is like a guard only letting one person unlock the vault at a time.
The second if ensures the vault isn’t opened twice by mistake.

Problems and Fixes:

Serialization Issue:
- Same fix as Lazy Singleton.
Reflection Issue:
- Same fix as Lazy Singleton.

4. Bill Pugh Singleton (Recommended)

public class BillPughSingleton {
    private BillPughSingleton() {}

    private static class SingletonHelper {
        private static final BillPughSingleton INSTANCE = new BillPughSingleton();
    }

    public static BillPughSingleton getInstance() {
        return SingletonHelper.INSTANCE;
    }
}

Visualization:

Think of a vending machine with a coin slot that automatically creates change when needed:

The SingletonHelper class is like a hidden compartment in the vending machine, pre-loaded with the mechanism to give change.
Only when someone inserts a coin (calls getInstance) does the mechanism activate, dispensing the Singleton instance.

Problems and Fixes:

Serialization Issue:

Breaking Example: Without readResolve, deserialization creates a new instance.

import java.io.*;

public class TestSerialization {
    public static void main(String[] args) throws Exception {
        BillPughSingleton instance1 = BillPughSingleton.getInstance();

        // Serialize the instance
        ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream("singleton.ser"));
        oos.writeObject(instance1);
        oos.close();

        // Deserialize the instance
        ObjectInputStream ois = new ObjectInputStream(new FileInputStream("singleton.ser"));
        BillPughSingleton instance2 = (BillPughSingleton) ois.readObject();
        ois.close();

        System.out.println(instance1 == instance2); // false
    }
}

Fix: Add readResolve to ensure the same instance is returned.

private Object readResolve() {
    return getInstance();
}

Reflection Issue:

Fix: Add a guard in the constructor:

private BillPughSingleton() {
    if (SingletonHelper.INSTANCE != null) {
        throw new IllegalStateException("Instance already created");
    }
}

5. Singleton Using Enum (Modern Approach)

With newer Java versions (Java 8 and beyond), you can use Enums to implement Singleton. Enums are inherently thread-safe and handle serialization by default.

public enum EnumSingleton {
    INSTANCE;

    public void showMessage() {
        System.out.println("Hello from Singleton!");
    }
}

Visualization:

Picture a monarch (king or queen) who is the sole ruler of their kingdom:

The INSTANCE is like the monarch, inherently unique and impossible to duplicate.
The JVM ensures that no matter how many times you call the monarch, you always get the same ruler.

Final Versions of Singleton Patterns with fixes

1. Lazy Initialization Singleton

import java.io.*;

public class LazySingleton implements Serializable {
    private static final long serialVersionUID = 1L;
    private static LazySingleton instance;

    private LazySingleton() {
        if (instance != null) {
            throw new IllegalStateException("Instance already created");
        }
    }

    public static LazySingleton getInstance() {
        if (instance == null) {
            instance = new LazySingleton();
        }
        return instance;
    }

    private Object readResolve() {
        return getInstance();
    }
}

2. Thread-Safe Singleton

import java.io.*;

public class ThreadSafeSingleton implements Serializable {
    private static final long serialVersionUID = 1L;
    private static ThreadSafeSingleton instance;

    private ThreadSafeSingleton() {
        if (instance != null) {
            throw new IllegalStateException("Instance already created");
        }
    }

    public static synchronized ThreadSafeSingleton getInstance() {
        if (instance == null) {
            instance = new ThreadSafeSingleton();
        }
        return instance;
    }

    private Object readResolve() {
        return getInstance();
    }
}

3. Double-Checked Locking Singleton

import java.io.*;

public class DoubleCheckedLockingSingleton implements Serializable {
    private static final long serialVersionUID = 1L;
    private static volatile DoubleCheckedLockingSingleton instance;

    private DoubleCheckedLockingSingleton() {
        if (instance != null) {
            throw new IllegalStateException("Instance already created");
        }
    }

    public static DoubleCheckedLockingSingleton getInstance() {
        if (instance == null) {
            synchronized (DoubleCheckedLockingSingleton.class) {
                if (instance == null) {
                    instance = new DoubleCheckedLockingSingleton();
                }
            }
        }
        return instance;
    }

    private Object readResolve() {
        return getInstance();
    }
}

4. Bill Pugh Singleton (Recommended)

import java.io.*;

public class BillPughSingleton implements Serializable {
    private static final long serialVersionUID = 1L;

    private BillPughSingleton() {
        if (SingletonHelper.INSTANCE != null) {
            throw new IllegalStateException("Instance already created");
        }
    }

    private static class SingletonHelper {
        private static final BillPughSingleton INSTANCE = new BillPughSingleton();
    }

    public static BillPughSingleton getInstance() {
        return SingletonHelper.INSTANCE;
    }

    private Object readResolve() {
        return getInstance();
    }
}

5. Singleton Using Enum (Modern Approach)

public enum EnumSingleton {
    INSTANCE;

    public void showMessage() {
        System.out.println("Hello from Enum Singleton!");
    }
}

Summary

Lazy Initialization: Simple, but not thread-safe without additional fixes.
Thread-Safe Singleton: Ensures thread safety but may incur performance overhead.
Double-Checked Locking: Reduces synchronization overhead while remaining thread-safe.
Bill Pugh Singleton: Recommended for its simplicity, efficiency, and elegance.
Enum Singleton: Modern and robust, suitable for most use cases.

25 Git Commands Every Developer Should Know

sachinkg12 — Thu, 23 Jan 2025 02:34:25 +0000

Git is an indispensable tool for modern developers. Whether you're collaborating on a team project or managing your own codebase, understanding essential Git commands can make your workflow more efficient and error-free.

Here's a list of 25 Git commands that every developer should know, along with examples and scenarios where they are useful:

1. git init

Initializes a new Git repository in your project directory.

git init

Example: Run this command in a folder to start version control for your project.

Scenario: Useful when starting a new project and you want to track changes from the beginning.

2. git clone

Clones an existing repository to your local machine.

git clone <repository-url>

Example:

git clone https://github.com/user/repo.git

Scenario: Use this to work on a project hosted on a platform like GitHub.

3. git status

Displays the current state of the working directory and staging area.

git status

Example: Shows if files have been modified, added, or deleted.

Scenario: Check which files need to be committed or staged.

4. git add

Adds files to the staging area for the next commit.

git add <file>  # Add a specific file
git add .       # Add all changes

Example:

git add index.html

Scenario: Use this before committing changes to ensure they're tracked.

5. git commit

Records changes in the repository with a message.

git commit -m "Your commit message"

Example:

git commit -m "Add homepage"

Scenario: Every time you make a set of changes that are ready to save in history.

6. git log

Shows the commit history of the repository.

git log

Example:

git log --oneline

Scenario: Review the history of changes and who made them.

7. git branch

Lists, creates, renames, or deletes branches.

git branch         # List branches
git branch <name>  # Create a new branch

Example:

git branch feature-login

Scenario: Create a new branch for a feature to work independently.

8. git checkout

Switches between branches or restores files.

git checkout <branch-name>

Example:

git checkout main

Scenario: Switch to a different branch to work on another feature.

9. git switch

A modern alternative to git checkout for switching branches.

git switch <branch-name>

Example:

git switch feature-login

Scenario: Use this to switch branches more intuitively.

10. git merge

Combines the changes from one branch into the current branch.

git merge <branch-name>

Example:

git merge feature-login

Scenario: Integrate a completed feature branch into the main branch.

11. git fetch

Downloads updates from a remote repository without merging them.

git fetch <remote>

Example:

git fetch origin

Scenario: See the latest updates on the remote without affecting your local branch.

12. git pull

Fetches and integrates changes from a remote repository.

git pull <remote> <branch>

Example:

git pull origin main

Scenario: Use this to synchronize your local repository with the remote.

13. git push

Uploads your local changes to a remote repository.

git push <remote> <branch>

Example:

git push origin main

Scenario: Push your changes to share with your team.

14. git diff

Shows differences between commits, branches, or your working directory.

git diff           # Show unstaged changes
git diff <branch>  # Compare branches

Example:

git diff HEAD

Scenario: Inspect what changes were made before committing.

15. git reset

Unstages files or reverts commits.

git reset <file>                # Unstage a file
git reset --hard <commit-hash>  # Reset to a specific commit

Example:

git reset --hard HEAD~1

Scenario: Undo commits or remove files from staging.

Note: Using git reset --hard cannot be undone and should be avoided in production environments to prevent accidental loss of work.

16. git stash

Temporarily saves changes you're not ready to commit.

git stash      # Stash changes
git stash pop  # Apply stashed changes

Example:

git stash

Scenario: Pause work on a feature and switch to another task without committing.

17. git rebase

Reapplies commits on top of another base branch.

git rebase <branch-name>

Example:

git rebase main

Scenario: Keep a clean commit history by replaying commits.

18. git tag

Creates a tag for marking specific commits (e.g., for releases).

git tag <tag-name>
git tag -a <tag-name> -m "Message"

Example:

git tag v1.0

Scenario: Mark specific commits, such as software releases.

19. git remote

Manages connections to remote repositories.

git remote add <name> <url>
git remote -v  # List remotes

Example:

git remote add origin https://github.com/user/repo.git

Scenario: Set up or view connections to remote repositories.

20. git cherry-pick

Applies a specific commit from another branch.

git cherry-pick <commit-hash>

Example:

git cherry-pick abc123

Scenario: Bring a specific fix or feature from one branch to another.

21. git blame

Shows who made changes to each line in a file.

git blame <file>

Example:

git blame app.js

Scenario: Identify who wrote or modified a specific piece of code.

22. git archive

Creates a compressed archive of the repository.

git archive --format=zip HEAD > archive.zip

Example:

git archive --format=zip HEAD > project.zip

Scenario: Share a snapshot of the codebase without version control data.

23. git clean

Removes untracked files and directories.

git clean -f   # Force delete files
git clean -fd  # Delete files and directories

Example:

git clean -fd

Scenario: Clean up unwanted files in the working directory.

24. git show

Displays detailed information about a specific commit.

git show <commit-hash>

Example:

git show abc123

Scenario: Inspect details of a particular commit.

25. git revert

Creates a new commit that undoes changes introduced by a previous commit.

git revert <commit-hash>

Example:

git revert abc123

Scenario: Undo a commit without modifying history.

These commands cover most of the day-to-day tasks you'll encounter when using Git. Master these commands, and you'll be well-equipped to handle any Git-related challenge.

Feel free to share your thoughts or insights. I'm excited to learn and grow together with you.

Happy Learning! :)

10 Unique Spark SQL Features You Won't Find in Oracle

sachinkg12 — Wed, 22 Jan 2025 09:02:54 +0000

Apache Spark and Oracle Database are both powerful platforms, but Spark offers several features that Oracle does not natively support. These unique features stem from Spark's distributed architecture and rich SQL API.

Below, we explore 10 exclusive capabilities of Spark SQL with sample datasets, examples, and use cases to help readers understand their practical applications.

I have used these features in my daily life, and I thought it would be helpful to share what I've learned.

1. Anti Join

Use Case

Quickly filter out rows that have matching records in another table, such as identifying users who have not completed a specific action in an activity log.

Sample Dataset:

table_a:
| id |
|----|
| 1 |
| 2 |
| 3 |

table_b:
| id |
|----|
| 2 |

Spark Query:

SELECT a.*
FROM table_a a
LEFT ANTI JOIN table_b b
ON a.id = b.id;

Result:
| id |
|----|
| 1 |
| 3 |

Oracle Equivalent:

SELECT a.*
FROM table_a a
WHERE NOT EXISTS (
    SELECT 1 FROM table_b b WHERE a.id = b.id
);

2. `from_unixtime` Function

Use Case

Converting timestamps for human-readable reports, especially in IoT or log analytics where data is stored as Unix timestamps.

Sample Dataset:

UNIX Timestamp:
| timestamp |
|----------------|
| 1700000000 |

Spark Query:

SELECT from_unixtime(1700000000) AS readable_date;

Result:
| readable_date |
|-------------------|
| 2023-11-14 02:53:20 |

Oracle Equivalent:

SELECT TO_CHAR(TO_DATE('1970-01-01', 'YYYY-MM-DD') + NUMTODSINTERVAL(1700000000, 'SECOND'), 'YYYY-MM-DD HH24:MI:SS') AS readable_date
FROM dual;

3. Explode

Use Case

Flattening arrays for detailed analytics, such as extracting product details from an array of purchased items.

Sample Dataset:

table:
| id | array_col |
|----|----------------|
| 1 | ["A", "B"] |
| 2 | ["C", "D"] |

Spark Query:

SELECT id, explode(array_col) AS value
FROM table;

Result:
| id | value |
|----|-------|
| 1 | A |
| 1 | B |
| 2 | C |
| 2 | D |

Oracle Equivalent:

SELECT id, column_value AS value
FROM table, TABLE(CAST(array_col AS TABLE_TYPE));

4. Map from Arrays

Use Case

Creating key-value pairs dynamically for use in aggregation or lookups, such as storing user preferences or configurations.

Sample Dataset:

table:
| id | keys | values |
|----|------------|--------------|
| 1 | ["k1", "k2"] | ["v1", "v2"] |

Spark Query:

SELECT id, map_from_arrays(keys, values) AS key_value_map
FROM table;

Result:
| id | key_value_map |
|----|-----------------------------|
| 1 | {"k1": "v1", "k2": "v2"} |

Oracle Equivalent: Not directly supported. Requires JSON or PL/SQL.

5. Approximate Count Distinct

Use Case

Efficiently estimating the number of unique users or items in massive datasets, such as tracking active users in real-time.

Sample Dataset:

events:
| user_id |
|---------|
| 1 |
| 2 |
| 2 |
| 3 |
| 4 |
| 5 |
| 5 |
| 6 |

Spark Query:

SELECT approx_count_distinct(user_id) AS unique_users
FROM events;

Result:
| unique_users |
|--------------|
| 6 |

Oracle Equivalent:

SELECT COUNT(DISTINCT user_id) AS unique_users
FROM events;

6. Flattening Nested Data Structures

Use Case

Simplifying hierarchical or nested data for downstream consumption, such as processing JSON data into relational tables.

Sample Dataset:

table:
| nested_struct |
|--------------------------|
| {"col1": "A", "col2": "B"} |

Spark Query:

SELECT nested_struct.col1, nested_struct.col2
FROM table;

Result:
| col1 | col2 |
|------|------|
| A | B |

Oracle Equivalent: Not directly supported. Requires PL/SQL.

7. Lateral View with Explode

Use Case

Extracting and analyzing individual elements of arrays stored in tables, such as breaking down multi-valued columns for visualization.

Sample Dataset:

table:
| id | array_col |
|----|----------------|
| 1 | ["A", "B"] |
| 2 | ["C", "D"] |
| 3 | ["E", "F"] |

Spark Query:

SELECT id, value
FROM table
LATERAL VIEW explode(array_col) exploded_table AS value;

Result:
| id | value |
|----|-------|
| 1 | A |
| 1 | B |
| 2 | C |
| 2 | D |
| 3 | E |
| 3 | F |

Oracle Equivalent:

SELECT id, column_value AS value
FROM table, TABLE(CAST(array_col AS TABLE_TYPE));

8. Map from Entries

Use Case

Converting arrays of key-value pairs into a map structure, useful in cases where the data is stored in pair form and needs to be queried or aggregated as a map.

Sample Dataset:

table:
| id | key_value_pairs |
|----|----------------------------------|
| 1 | [("k1", "v1"), ("k2", "v2")] |
| 2 | [("k3", "v3"), ("k4", "v4")] |

Spark Query:

SELECT id, map_from_entries(key_value_pairs) AS key_value_map
FROM table;

Result:
| id | key_value_map |
|----|-----------------------------|
| 1 | {"k1": "v1", "k2": "v2"} |
| 2 | {"k3": "v3", "k4": "v4"} |

Oracle Equivalent: Not directly supported.

9. Higher-Order Functions (Transform)

Use Case

Applying complex transformations to arrays within queries, such as scaling numerical data or normalizing values.

Sample Dataset:

table:
| id | array_col |
|----|--------------|
| 1 | [1, 2, 3] |
| 2 | [4, 5, 6] |

Spark Query:

SELECT id, transform(array_col, x -> x * 2) AS doubled_values
FROM table;

Result:
| id | doubled_values |
|----|-------------------|
| 1 | [2, 4, 6] |
| 2 | [8, 10, 12] |

Oracle Equivalent: Not supported directly. Requires PL/SQL.

10. Union with Schema Evolution

Use Case

Merging datasets with schema differences, such as integrating data from multiple sources during ETL processes.

Sample Dataset:

table_a:
| id | name |
|----|---------|
| 1 | Alice |
| 3 | Charlie |
| 5 | Eve |

table_b:
| id | full_name |
|----|-----------|
| 2 | Bob |
| 4 | Diana |
| 6 | Frank |

Spark Query:

SELECT id, name
FROM table_a
UNION BY NAME
SELECT id, full_name AS name
FROM table_b;

Result: Combines data with schema alignment.
| id | name |
|----|---------|
| 1 | Alice |
| 2 | Bob |
| 3 | Charlie |
| 4 | Diana |
| 5 | Eve |
| 6 | Frank |

Oracle Equivalent: Not supported directly. Requires manual schema alignment.

Feel free to share your thoughts or insights. I'm excited to learn and grow together with you.

Happy Learning!

5 Common Code Smells in Java and How to Fix Them

sachinkg12 — Wed, 22 Jan 2025 08:33:42 +0000

Code smells are indicators of potential problems in your code that could lead to issues with maintainability, readability, or performance. While they don't necessarily cause bugs, addressing code smells ensures your codebase remains clean and efficient.

We will look into 5 common code smells in Java, offering examples, in-depth explanations, and better approaches to handle them effectively.

1. Long Method

Code Smell: A method that is excessively long can make the code harder to read, test, and maintain. Even if the method appears modularized with helper methods, it might still combine multiple levels of abstraction, violating the Single Responsibility Principle (SRP).

Example:

public void processOrder(Order order) {
    validateOrder(order);
    calculateDiscount(order);
    updateInventory(order);
    generateInvoice(order);
    sendNotification(order);
}

Here, processOrder orchestrates multiple unrelated tasks—validation, discount calculation, inventory update, invoice generation, and notification—making it difficult to extend or modify without risking unintended consequences.

Better Approach:

Refactor the orchestration logic to separate these responsibilities more clearly. Using design patterns such as the Command Pattern or Pipeline Pattern can help ensure modularity.

Refactored Code with Command Pattern:

public interface OrderCommand {
    void execute(Order order);
}

public class ValidateOrderCommand implements OrderCommand {
    public void execute(Order order) {
        // Validation logic
    }
}

public class ApplyDiscountCommand implements OrderCommand {
    public void execute(Order order) {
        // Discount application logic
    }
}

public class OrderProcessor {
    private List<OrderCommand> commands;

    public OrderProcessor(List<OrderCommand> commands) {
        this.commands = commands;
    }

    public void processOrder(Order order) {
        for (OrderCommand command : commands) {
            command.execute(order);
        }
    }
}

// Usage
List<OrderCommand> commands = List.of(
    new ValidateOrderCommand(),
    new ApplyDiscountCommand(),
    new UpdateInventoryCommand(),
    new GenerateInvoiceCommand(),
    new NotifyCustomerCommand()
);

OrderProcessor processor = new OrderProcessor(commands);
processor.processOrder(new Order());

Benefit:

Improves modularity.
Each command handles a single responsibility and can be tested or reused independently.
Adding new steps (e.g., fraud detection) is as simple as adding a new OrderCommand.

2. God Class

Code Smell: A "God Class" tries to handle too many responsibilities, leading to high coupling and poor maintainability.

Example:

public class OrderManager {
    public void createOrder() { /* Implementation */ }
    public void updateOrder() { /* Implementation */ }
    public void deleteOrder() { /* Implementation */ }
    public void validatePayment() { /* Implementation */ }
    public void sendInvoice() { /* Implementation */ }
}

Better Approach:

Divide the responsibilities into smaller, focused classes, each aligned with a specific domain task.

Refactored Code:

public class OrderService {
    public void createOrder() { /* Implementation */ }
    public void updateOrder() { /* Implementation */ }
    public void deleteOrder() { /* Implementation */ }
}

public class PaymentService {
    public void validatePayment() { /* Implementation */ }
}

public class NotificationService {
    public void sendInvoice() { /* Implementation */ }
}

Benefit:

Reduces coupling and improves modularity.
Each service class becomes easier to maintain, test, and extend independently.

3. Magic Numbers

Code Smell: Using literal numbers (or "magic numbers") directly in your code makes it hard to understand and modify.

Example:

public double calculateDiscount(double totalAmount) {
    return totalAmount > 1000 ? totalAmount * 0.1 : totalAmount;
}

Better Approach:

Replace magic numbers with constants that have meaningful names.

Refactored Code:

private static final double DISCOUNT_THRESHOLD = 1000;
private static final double DISCOUNT_RATE = 0.1;

public double calculateDiscount(double totalAmount) {
    return totalAmount > DISCOUNT_THRESHOLD ? totalAmount * DISCOUNT_RATE : totalAmount;
}

Benefit:

Improves readability and reduces the risk of errors during updates.
Constants provide clarity about the business logic.

4. Duplicated Code

Code Smell: Repetitive code across methods or classes leads to maintenance challenges and inconsistency.

Example:

public double calculateTax(double amount) {
    return amount * 0.18;
}

public double calculateDiscount(double amount) {
    return amount * 0.1;
}

Better Approach:

Abstract the common logic into a reusable method.

Refactored Code:

private double applyRate(double amount, double rate) {
    return amount * rate;
}

public double calculateTax(double amount) {
    return applyRate(amount, 0.18);
}

public double calculateDiscount(double amount) {
    return applyRate(amount, 0.1);
}

Benefit:

Avoids redundancy.
Ensures consistency across methods.
Makes the code easier to modify and extend.

5. Excessive Parameter List

Code Smell: Methods with too many parameters are difficult to read, understand, and prone to errors during calls.

Example:

public void createUser(String firstName, String lastName, String email, String phoneNumber, String address) {
    // Implementation
}

Better Approach:

Encapsulate parameters in an object or use a builder pattern.

Refactored Code:

public class User {
    private String firstName;
    private String lastName;
    private String email;
    private String phoneNumber;
    private String address; // Getters, setters, and constructor
}

public void createUser(User user) {
    // Implementation
}

Benefit:

Improves readability and extensibility.
Additional parameters can be added without changing the method signature.

Code smells are early indicators of deeper design issues that can impact maintainability and scalability. By identifying and addressing these smells, you improve code quality and reduce technical debt.

Using principles like DRY (Don't Repeat Yourself), SRP (Single Responsibility Principle), and modular design, you can build robust, clean, and efficient Java applications.

Have you encountered these code smells in your projects? Share your experiences and solutions in the comments below!

Forem: sachinkg12

HeapLens — Analyze Java Heap Dumps Inside VS Code with AI

What it actually does

AI-assisted analysis

Why Rust

Who this is for

Try it

Why double Loses Precision and How to Avoid It in Java

Why Does double Lose Precision?

1. IEEE 754 Floating-Point Standard

2. Accumulated Errors in Arithmetic

Working Example: Precision Loss with double

How to Avoid Precision Loss

1. Use BigDecimal for Precise Calculations

Example Using BigDecimal:

2. Compare Using an Epsilon Value

Example Using Epsilon Comparison:

Why Use Epsilon Comparison?

Using Apache Commons Math for Enhanced Precision

Example: Using Precision.equals for Comparison

Why Use Apache Commons Math?

Summary

Understanding the Singleton Pattern in Java

Why Use the Singleton Pattern?

How to Implement Singleton Pattern in Java

1. Lazy Initialization Singleton

Visualization:

Problems and Fixes:

2. Thread-Safe Singleton

Visualization:

Problems and Fixes:

3. Double-Checked Locking Singleton

Visualization:

Problems and Fixes:

4. Bill Pugh Singleton (Recommended)

Visualization:

Problems and Fixes:

5. Singleton Using Enum (Modern Approach)

Visualization:

Final Versions of Singleton Patterns with fixes

1. Lazy Initialization Singleton

2. Thread-Safe Singleton

3. Double-Checked Locking Singleton

4. Bill Pugh Singleton (Recommended)

5. Singleton Using Enum (Modern Approach)

Summary

25 Git Commands Every Developer Should Know

1. git init

2. git clone

3. git status

4. git add

5. git commit

6. git log

7. git branch

8. git checkout

9. git switch

10. git merge

11. git fetch

12. git pull

13. git push

14. git diff

15. git reset

16. git stash

17. git rebase

18. git tag

19. git remote

20. git cherry-pick

21. git blame

22. git archive

23. git clean

24. git show

25. git revert

10 Unique Spark SQL Features You Won't Find in Oracle

1. Anti Join

Use Case

2. from_unixtime Function

Use Case

3. Explode

Use Case

4. Map from Arrays

Why Does `double` Lose Precision?

Working Example: Precision Loss with `double`

1. Use `BigDecimal` for Precise Calculations

Example Using `BigDecimal`:

Example: Using `Precision.equals` for Comparison

2. `from_unixtime` Function