Forem: vishalpaalakurthi

Exposing Hazelcast Metrics with JMX and Prometheus

vishalpaalakurthi — Fri, 20 Sep 2024 11:11:20 +0000

Monitoring and managing the performance of your distributed systems is crucial, especially when using a data grid like Hazelcast. One of the effective ways to expose metrics from Hazelcast is by utilizing the JMX (Java Management Extensions) Prometheus exporter. This blog will guide you through the steps to set up the JMX Prometheus exporter to expose Hazelcast metrics for monitoring.

Step 1: Download the JMX Prometheus Exporter

First, you need to download the JMX Prometheus Java agent. This can be easily obtained from the Maven repository. Use the following link to download the JAR file:

Download jmx_prometheus_javaagent-1.0.1.jar

Make sure to save the JAR file in a convenient location on your server.

Step 2: Create the JMX Exporter Configuration File

Next, you'll need to create a configuration file named jmx_exporter.yml. This file will specify which metrics to collect from the Hazelcast instance. Below is an example configuration:

startDelaySeconds: 0
attrNameSnakeCase: true
lowercaseOutputName: true
lowercaseOutputLabelNames: true
whitelistObjectNames:
  - "com.hazelcast:type=Metrics,*"
rules:
  - pattern: "^com.hazelcast<type=Metrics, instance=(.*), prefix=(.*), tag([0-9]+)=(.*)><>(.+):"
    name: hazelcast_$5
    labels:
      instance: $1
      prefix: $2
      tag$3: $4

  - pattern: "^com.hazelcast<type=Metrics, instance=(.*), prefix=(.*)><>(.+):"
    name: hazelcast_$3
    labels:
      instance: $1
      prefix: $2

Important Note:

When running the JMX exporter as a Java agent, do not configure the jmxUrl or hostPort. This is because you’re not monitoring a remote JVM; you’re monitoring the local Hazelcast instance.

Step 3: Enable Metrics in Hazelcast

Before starting your Hazelcast application, you must enable metrics in its configuration. This can be done by modifying the Java code as follows:

Config config = new Config();
config.setProperty("hazelcast.jmx", "true");
config.setProperty("hazelcast.metrics.enabled", "true");

This code snippet ensures that JMX and metrics are enabled within your Hazelcast configuration.

Step 4: Start the Hazelcast Application with JMX Exporter

Now it's time to start your Hazelcast application with the JMX Prometheus exporter. Use the following command to run your application:

java -javaagent:<path>/jmx_prometheus_javaagent-1.0.1.jar=8080:<path>/jmx_exporter.yml -cp 'common-model-1.9.0-SNAPSHOT.jar:lib/*' io.dxchange.resource.efs.HazelcastServer

Note:

Replace <path> with the actual path to your JAR and configuration file.
You can change the 8080 port to any available port if needed.

Step 5: Verify the Metrics

To ensure everything is set up correctly, you can verify the exposed metrics using the following curl command:

curl http://localhost:8080/metrics

This command will return a list of metrics that have been collected from your Hazelcast instance. You should see a response similar to the following:

# HELP hazelcast_some_metric Description of some metric
# TYPE hazelcast_some_metric counter
hazelcast_some_metric{instance="your-instance", prefix="your-prefix"} 123
...

Example Output:

Conclusion

By following the steps outlined in this blog, you can effectively expose Hazelcast metrics using the JMX Prometheus exporter. This setup will enable you to monitor the performance of your Hazelcast instances, allowing for better management and optimization of your distributed applications. Happy monitoring!

Deploying a Spring Boot Application on AWS: ECS, EKS, or Kubernetes? A Detailed Guide with Cost-Effective Recommendations

vishalpaalakurthi — Wed, 18 Sep 2024 16:20:22 +0000

Deploying a Spring Boot application as a Docker image in AWS Cloud can be done using several services: Amazon ECS (Elastic Container Service), Amazon EKS (Elastic Kubernetes Service), or self-managed Kubernetes on AWS. Here's an overview of the steps for each option, along with their advantages, disadvantages, and cost-effectiveness.

1. Amazon ECS (Elastic Container Service)

Steps to Deploy in ECS:

Create a Docker Image of your Spring Boot application and push it to Amazon ECR (Elastic Container Registry).
Create an ECS Cluster using either Fargate (serverless) or EC2 as the underlying infrastructure.
Define a Task Definition that specifies the Docker image, container ports, and memory/CPU requirements.
Configure a Service in ECS that links the task definition to the desired number of replicas (auto-scaling) and load balancer (if required).
Deploy your service and monitor via CloudWatch and ECS dashboard.

Advantages of ECS:

Managed service: ECS handles the orchestration, scaling, and health monitoring of containers.
Integration with Fargate: You can deploy without managing EC2 instances, reducing operational overhead.
Tight AWS integration: Easy integration with AWS services like IAM, CloudWatch, ALB, and VPCs.

Disadvantages of ECS:

Less flexibility compared to Kubernetes for complex workloads.
Vendor lock-in with AWS.
ECS has limited ecosystem support compared to Kubernetes for third-party tooling.

Cost-effectiveness:

ECS with Fargate is cost-effective for smaller workloads or when you don’t want to manage EC2 instances.
You pay for the vCPU and memory resources your containers use. No cost for managing underlying infrastructure.
ECS on EC2 can become more expensive if not properly optimized, as you're paying for full EC2 instances.

2. Amazon EKS (Elastic Kubernetes Service)

Steps to Deploy in EKS:

Push your Docker image to ECR.
Create an EKS Cluster with worker nodes on EC2 or EKS Fargate for serverless deployments.
Configure kubectl or AWS CLI to connect to the EKS cluster.
Define your Kubernetes Deployment (including replicas, resource requests) and service (LoadBalancer or Ingress).
Apply the Kubernetes manifest files to deploy your application via kubectl.
Monitor with CloudWatch or use Kubernetes-native tools like Prometheus and Grafana.

Advantages of EKS:

Kubernetes-native: You can run your Spring Boot application on Kubernetes, which offers the most flexibility and customizability.
Huge ecosystem: Kubernetes has a large community and ecosystem for monitoring, logging, CI/CD, and autoscaling.
Multi-cloud: Kubernetes allows easier migration between clouds or even hybrid cloud setups.

Disadvantages of EKS:

Complexity: Kubernetes is harder to manage and configure compared to ECS. Even with EKS, you need to manage various Kubernetes components.
Higher operational overhead: More expertise is needed to handle Kubernetes, even though AWS manages the control plane.

Cost-effectiveness:

EKS with Fargate can be cost-effective for smaller workloads as there is no need to manage EC2 instances.
However, EKS with EC2 instances can lead to higher costs due to the need to manage and optimize the nodes.
You are charged for the EKS control plane (~$74/month per cluster) in addition to EC2 or Fargate resource costs.

3. Self-managed Kubernetes on AWS (kops or kubeadm)

Steps to Deploy on Kubernetes (Self-managed):

Set up an EC2 instance or multiple EC2 instances for your Kubernetes cluster.
Use kops (Kubernetes Operations) or kubeadm to create the Kubernetes cluster.
Push the Docker image to ECR or any other Docker registry.
Configure kubectl to manage the cluster.
Define your Kubernetes deployment and service.
Deploy your application with kubectl and monitor.

Advantages of Self-managed Kubernetes:

Full control: You have complete control over the cluster, configuration, and operations.
No AWS service cost: You are not paying for EKS control plane management.

Disadvantages of Self-managed Kubernetes:

High operational complexity: You are responsible for cluster management, security updates, scaling, and monitoring.
Limited AWS integration: While possible, manual integration is required for AWS services.
Scaling difficulty: Self-managed clusters need more effort for scaling and disaster recovery.

Cost-effectiveness:

This option can be cost-efficient for large-scale, long-running applications if you have expertise in managing Kubernetes.
However, it's less cost-effective due to the operational complexity and time required to manage infrastructure compared to using EKS or ECS.

Cost-effective Recommendations:

For smaller workloads or if you want to avoid managing infrastructure, ECS with Fargate is the most cost-effective and easiest option. It provides a serverless experience and scales based on demand without EC2 management.
For medium to large-scale applications that need Kubernetes capabilities, EKS with Fargate could be a good balance between cost and flexibility. You avoid the overhead of managing EC2 nodes but still get the power of Kubernetes.
If you're experienced in Kubernetes and want full control over your infrastructure for a complex or large-scale system, self-managed Kubernetes could be considered, but it will likely require more operational effort.

Each option has trade-offs in terms of flexibility, management, and cost, but for most cases, ECS or EKS with Fargate should be the most cost-effective unless your application has specific Kubernetes needs.

Mastering Java Collections with Multithreading: Best Practices and Practical Examples

vishalpaalakurthi — Wed, 18 Sep 2024 05:59:48 +0000

Combining Java Collections with multithreading is a common practice in developing high-performance, concurrent applications. Properly managing collections in a multithreaded environment ensures data integrity, prevents race conditions, and enhances application scalability. This guide provides practical examples demonstrating how to effectively use Java Collections in multithreaded scenarios.

Understanding the Challenges

When multiple threads access and modify collections concurrently, several issues can arise:

Race Conditions: Multiple threads modifying the same data simultaneously can lead to inconsistent or unexpected results.
Deadlocks: Improper synchronization can cause threads to wait indefinitely for resources.
Performance Bottlenecks: Excessive synchronization can degrade performance by limiting parallelism.

To address these challenges, Java provides thread-safe collections in the java.util.concurrent package and synchronization utilities to manage concurrent access effectively.

Using Concurrent Collections

Java’s java.util.concurrent package offers a suite of thread-safe collection classes that handle synchronization internally, providing high performance in concurrent environments.

ConcurrentHashMap Example

ConcurrentHashMap is a thread-safe implementation of the Map interface. It allows concurrent read and write operations without locking the entire map, enhancing performance in multithreaded applications.

Use Case: Caching frequently accessed data where multiple threads may read and write concurrently.

import java.util.concurrent.ConcurrentHashMap;

public class ConcurrentHashMapExample {
    private ConcurrentHashMap<String, Integer> wordCountMap = new ConcurrentHashMap<>();

    // Method to increment word count
    public void incrementWord(String word) {
        wordCountMap.merge(word, 1, Integer::sum);
    }

    public void displayCounts() {
        wordCountMap.forEach((word, count) -> System.out.println(word + ": " + count));
    }

    public static void main(String[] args) throws InterruptedException {
        ConcurrentHashMapExample example = new ConcurrentHashMapExample();
        String text = "concurrent concurrent multithreading collections java";

        // Create multiple threads to process words
        Thread t1 = new Thread(() -> {
            for (String word : text.split(" ")) {
                example.incrementWord(word);
            }
        });

        Thread t2 = new Thread(() -> {
            for (String word : text.split(" ")) {
                example.incrementWord(word);
            }
        });

        t1.start();
        t2.start();

        t1.join();
        t2.join();

        example.displayCounts();
    }
}

Explanation:

ConcurrentHashMap allows multiple threads to modify the map concurrently.
The merge method atomically updates the count for each word, preventing race conditions.
Two threads process the same text concurrently, updating the word counts without interference.

CopyOnWriteArrayList Example

CopyOnWriteArrayList is a thread-safe variant of ArrayList where all mutative operations (like add, set, and remove) are implemented by making a fresh copy of the underlying array.

Use Case: Scenarios with more reads than writes, such as event listener lists.

import java.util.List;
import java.util.concurrent.CopyOnWriteArrayList;

public class CopyOnWriteArrayListExample {
    private List<String> eventListeners = new CopyOnWriteArrayList<>();

    public void addListener(String listener) {
        eventListeners.add(listener);
    }

    public void notifyListeners(String event) {
        for (String listener : eventListeners) {
            System.out.println(listener + " received event: " + event);
        }
    }

    public static void main(String[] args) {
        CopyOnWriteArrayListExample example = new CopyOnWriteArrayListExample();

        // Adding listeners in separate threads
        Thread t1 = new Thread(() -> example.addListener("Listener1"));
        Thread t2 = new Thread(() -> example.addListener("Listener2"));

        t1.start();
        t2.start();

        try {
            t1.join();
            t2.join();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

        // Notify all listeners
        example.notifyListeners("EventA");
    }
}

Explanation:

CopyOnWriteArrayList ensures thread-safe addition of listeners without explicit synchronization.
Iteration over the list (e.g., notifying listeners) does not require additional synchronization, as the underlying array is safely published.
Ideal for scenarios where reads are frequent and modifications are rare.

BlockingQueue Example

BlockingQueue is a thread-safe queue that supports operations that wait for the queue to become non-empty when retrieving and wait for space to become available when storing.

Use Case: Implementing producer-consumer patterns where producers add tasks to the queue, and consumers process them.

import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;

public class BlockingQueueExample {
    private BlockingQueue<String> taskQueue = new ArrayBlockingQueue<>(10);

    // Producer
    public void produce(String task) throws InterruptedException {
        taskQueue.put(task);
        System.out.println("Produced: " + task);
    }

    // Consumer
    public void consume() throws InterruptedException {
        String task = taskQueue.take();
        System.out.println("Consumed: " + task);
    }

    public static void main(String[] args) {
        BlockingQueueExample example = new BlockingQueueExample();

        // Producer thread
        Thread producer = new Thread(() -> {
            String[] tasks = {"Task1", "Task2", "Task3", "Task4", "Task5"};
            try {
                for (String task : tasks) {
                    example.produce(task);
                    Thread.sleep(100); // Simulate production time
                }
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
        });

        // Consumer thread
        Thread consumer = new Thread(() -> {
            try {
                for (int i = 0; i < 5; i++) {
                    example.consume();
                    Thread.sleep(150); // Simulate processing time
                }
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
        });

        producer.start();
        consumer.start();
    }
}

Explanation:

ArrayBlockingQueue is a bounded blocking queue that blocks producers when the queue is full and consumers when it's empty.
The producer thread adds tasks to the queue, while the consumer thread retrieves and processes them.
put and take methods handle the necessary synchronization, ensuring thread safety without explicit locks.

Synchronizing Access to Collections

Sometimes, you might need to synchronize access to non-concurrent collections to make them thread-safe.

Synchronized List Example

Collections.synchronizedList provides a synchronized (thread-safe) list backed by the specified list.

Use Case: When you need a thread-safe version of ArrayList or LinkedList without using concurrent variants.

import java.util.ArrayList;
import java.util.Collections;
import java.util.List;

public class SynchronizedListExample {
    private List<String> synchronizedList = Collections.synchronizedList(new ArrayList<>());

    public void addItem(String item) {
        synchronizedList.add(item);
    }

    public void displayItems() {
        synchronized (synchronizedList) { // Must synchronize during iteration
            for (String item : synchronizedList) {
                System.out.println(item);
            }
        }
    }

    public static void main(String[] args) throws InterruptedException {
        SynchronizedListExample example = new SynchronizedListExample();

        // Multiple threads adding items
        Thread t1 = new Thread(() -> {
            for (int i = 0; i < 5; i++) {
                example.addItem("Item " + i);
            }
        });

        Thread t2 = new Thread(() -> {
            for (int i = 5; i < 10; i++) {
                example.addItem("Item " + i);
            }
        });

        t1.start();
        t2.start();

        t1.join();
        t2.join();

        // Display items
        example.displayItems();
    }
}

Explanation:

Collections.synchronizedList wraps an ArrayList to make it thread-safe.
When iterating over the synchronized list, explicit synchronization is required to prevent ConcurrentModificationException.
This approach is suitable when using existing collection types that don’t have concurrent counterparts.

Producer-Consumer Scenario

Combining concurrent collections and multithreading is ideal for implementing producer-consumer patterns. Here's a comprehensive example using BlockingQueue.

Use Case: A logging system where multiple threads produce log messages, and a separate thread consumes and writes them to a file.

import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;

// Logger class implementing producer-consumer using BlockingQueue
public class Logger {
    private BlockingQueue<String> logQueue = new LinkedBlockingQueue<>();
    private volatile boolean isRunning = true;

    // Producer method to add log messages
    public void log(String message) throws InterruptedException {
        logQueue.put(message);
    }

    // Consumer thread to process log messages
    public void start() {
        Thread consumerThread = new Thread(() -> {
            while (isRunning || !logQueue.isEmpty()) {
                try {
                    String message = logQueue.take();
                    writeToFile(message);
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                }
            }
        });
        consumerThread.start();
    }

    // Simulate writing to a file
    private void writeToFile(String message) {
        System.out.println("Writing to log: " + message);
    }

    // Shutdown the logger
    public void stop() {
        isRunning = false;
    }

    public static void main(String[] args) throws InterruptedException {
        Logger logger = new Logger();
        logger.start();

        // Simulate multiple threads logging messages
        Thread t1 = new Thread(() -> {
            try {
                logger.log("Message from Thread 1");
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
        });

        Thread t2 = new Thread(() -> {
            try {
                logger.log("Message from Thread 2");
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
        });

        t1.start();
        t2.start();

        t1.join();
        t2.join();

        // Allow some time for consumer to process
        Thread.sleep(1000);
        logger.stop();
    }
}

Explanation:

LinkedBlockingQueue serves as the thread-safe queue between producers (logging threads) and the consumer (logger thread).
Producers add log messages to the queue using the log method.
The consumer thread continuously takes messages from the queue and processes them (simulated by writeToFile).
The isRunning flag allows graceful shutdown of the consumer thread after all messages are processed.

Best Practices

Prefer Concurrent Collections: Whenever possible, use classes from java.util.concurrent like ConcurrentHashMap, CopyOnWriteArrayList, and BlockingQueue instead of manually synchronizing non-concurrent collections.
Minimize Synchronization Scope: Keep synchronized blocks as short as possible to reduce contention and improve performance.
Avoid Using Synchronized Collections for High Contention Scenarios: For high-concurrency scenarios, prefer concurrent collections over synchronized wrappers, as they offer better scalability.
Handle InterruptedException Properly: Always handle InterruptedException appropriately, typically by restoring the interrupt status with Thread.currentThread().interrupt().
Use Higher-Level Constructs: Utilize higher-level concurrency utilities like ExecutorService, ForkJoinPool, and CompletableFuture to manage threads and tasks more effectively.
Immutable Objects: Where possible, use immutable objects to simplify concurrency control, as they are inherently thread-safe.

Conclusion

Effectively combining Java Collections with multithreading involves selecting the right collection types and synchronization mechanisms to ensure thread safety and performance. By leveraging concurrent collections provided by the java.util.concurrent package, developers can build robust and scalable multithreaded applications with ease. Always consider the specific requirements and access patterns of your application to choose the most suitable approach.

How to Set Up a PostgreSQL Server on a Virtual Machine

vishalpaalakurthi — Wed, 29 May 2024 06:57:28 +0000

In this tutorial, we'll walk through setting up a PostgreSQL server on a Virtual Machine (VM). We'll use Ubuntu as the operating system for the VM and cover steps for popular cloud providers like AWS, Google Cloud, and Azure. Let's get started!

Step 1: Set Up the VM

Choose a Cloud Provider: AWS, Google Cloud Platform (GCP), or Microsoft Azure.
Create a VM:
- AWS: Use an EC2 instance.
- GCP: Use Compute Engine.
- Azure: Use Virtual Machine service.
Select OS: Choose an Ubuntu LTS version (e.g., Ubuntu 20.04 LTS).

Step 2: Connect to the VM

Access the VM: Use SSH to connect to the VM.

Example:

 ssh -i your-key.pem username@your-vm-ip-address

Step 3: Update and Upgrade the System

Run the following commands to update and upgrade the system:

   sudo apt update
   sudo apt upgrade -y

Step 4: Install PostgreSQL

Install PostgreSQL:

   sudo apt install postgresql postgresql-contrib -y

Start and Enable PostgreSQL:

   sudo systemctl start postgresql
   sudo systemctl enable postgresql

Step 5: Configure PostgreSQL

Switch to the PostgreSQL User:

   sudo -i -u postgres

Access PostgreSQL Prompt:

   psql

Set a Password for the PostgreSQL User:

   \password postgres

(Enter the new password when prompted)

Create a New Database and User:

   CREATE DATABASE mydatabase;
   CREATE USER myuser WITH ENCRYPTED PASSWORD 'mypassword';
   GRANT ALL PRIVILEGES ON DATABASE mydatabase TO myuser;

Exit PostgreSQL Prompt:

\q

Edit PostgreSQL Configuration to Allow Remote Connections:
- Open the PostgreSQL configuration file:
```
 sudo nano /etc/postgresql/12/main/postgresql.conf
```

Find the line listen_addresses and set it to '*':
```
 listen_addresses = '*'
```
Save and close the file.
1. Configure Client Authentication:

Open the pg_hba.conf file:

 sudo nano /etc/postgresql/12/main/pg_hba.conf

Add the following line to allow remote connections:

 host    all             all             0.0.0.0/0            md5

Save and close the file.
1. Restart PostgreSQL:

   sudo systemctl restart postgresql

Step 6: Allow External Connections to PostgreSQL

Update Firewall Rules:
- AWS: Edit the Security Group to allow inbound traffic on port 5432.
- GCP: Edit the Firewall rules to allow traffic on port 5432.
- Azure: Edit the Network Security Group to allow inbound traffic on port 5432.

Step 7: Connect to PostgreSQL Remotely

Use a PostgreSQL Client: Tools like psql, DBeaver, or pgAdmin can connect to your PostgreSQL server remotely using the VM's public IP address and the credentials you set up.

Example Connection Command

psql -h your-vm-ip-address -U myuser -d mydatabase

(Enter the password when prompted)

Final Notes

Ensure your VM's firewall settings allow inbound traffic on port 5432.
Secure your PostgreSQL server by following best practices, such as using strong passwords, enabling SSL, and configuring proper firewall rules.

With this setup, you now have a basic PostgreSQL server running on a VM, ready for development or production use. Happy coding!

How to Perform Upsert Operations in MongoDB with Java

vishalpaalakurthi — Tue, 07 May 2024 07:20:03 +0000

When working with MongoDB, one common requirement is to update existing documents with new information or insert new documents if they don't already exist. This is known as an "upsert" operation. In this tutorial, we’ll explore how to perform upsert operations in MongoDB using the Java MongoDB Driver. This is particularly useful for applications where you need to ensure data is always current without risking duplicates.

Step 1: Set Up MongoDB Java Driver

Before you begin coding, you need to include the MongoDB Java Driver in your project. If you are using Maven for project management, you can add the MongoDB Java Driver by including the following dependency in your pom.xml file:

<dependency>
    <groupId>org.mongodb</groupId>
    <artifactId>mongodb-driver-sync</artifactId>
    <version>4.4.3</version>
</dependency>

This ensures that your project has access to the necessary MongoDB classes and methods.

Step 2: Connect to MongoDB

First, establish a connection to your MongoDB instance. This involves creating a client, connecting to the database, and accessing the specific collection you will work with:

import com.mongodb.client.MongoClients;
import com.mongodb.client.MongoClient;
import com.mongodb.client.MongoDatabase;
import com.mongodb.client.MongoCollection;
import org.bson.Document;

public class MongoUpsert {
    public static void main(String[] args) {
        try (MongoClient client = MongoClients.create("mongodb://localhost:27017")) {
            MongoDatabase database = client.getDatabase("yourDatabase");
            MongoCollection<Document> collection = database.getCollection("yourCollection");

            // Now ready to perform upsert operations
            upsertDocument(collection);
        }
    }
}

Step 3: Define the Upsert Operation

The crux of the upsert functionality lies in how you define the operation. The following method demonstrates setting up an upsert:

import com.mongodb.client.model.Filters;
import com.mongodb.client.model.UpdateOptions;
import com.mongodb.client.result.UpdateResult;

public static void upsertDocument(MongoCollection<Document> collection) {
    Document filter = new Document("key", "value");  // Criteria to find the document
    Document newData = new Document("$set", new Document("fieldToUpdate", "newValue"));  // Data to update

    UpdateOptions options = new UpdateOptions().upsert(true);  // Enables upsert functionality

    UpdateResult result = collection.updateOne(filter, newData, options);

    if (result.getMatchedCount() == 0 && result.getUpsertedId() != null) {
        System.out.println("A new document was inserted with ID: " + result.getUpsertedId());
    } else {
        System.out.println("Existing document updated");
    }
}

Explanation:

Filter: Defines how to find the document. If the document with the specified criteria exists, it will be updated.
New Data: Specifies the new data to be added or updated in the document.
UpdateOptions: Setting upsert to true tells MongoDB to insert a new document if no existing document matches the filter.

Step 4: Execute Your Application

Finally, ensure your MongoDB server is running and execute your Java application. This simple setup enables your application to handle inserted or updated documents appropriately, maintaining data integrity and avoiding duplicates.

Conclusion

Upsert operations are crucial for maintaining up-to-date data in your database without introducing duplicate records. The MongoDB Java Driver provides a straightforward way to implement these operations, ensuring your data layer remains robust and efficient. Whether you're managing user data, processing transactions, or synchronizing state across applications, mastering upserts is a valuable skill in your MongoDB toolkit.

How to create a simple SFTP server using java stack

vishalpaalakurthi — Thu, 15 Sep 2022 09:23:21 +0000

Build a simple SFTP server

Usage

As a software developer, I do different activities with SFTP server. It became a daily use case for me. I use to ask a IT/DEVOps person for a help to setup SFTP server in an instance, So that i can use through different clients. It is becoming costly activity in terms of time and money and hard to setup in linux/ubuntu machines.

To avoid that, I have created my own SFTP server and it can run in my local machine too. I used a trending technologies and frameworks and built a light weight SFTP server.

Tech Stack Used

JAVA (11)
Quarkus Framework

Most of people may not know about the quarkus. But It's a Kubernetes Native Java stack tailored for OpenJDK HotSpot and GraalVM, crafted from the best of breed Java libraries and standards. Framework community is growing very fast and building a optimised solutions to the world.

This is purely build on top of JAVA and QUARKUS. I used a sshd-sftp maven package for implementing SFTP server. So that, Now i can run it locally as jar file using java commands.

Recently, I used the same jar to create a SFTP server in AWS EC2 instance. It's working cool :).

Source code

https://github.com/vishalpaalakurthi/SFTP-Server-Quarkus

(Instructions are available in README file)

I hope you like it. Please add your comments.

Thanks

Leveraging Hazelcast Persistence with File System Storage

vishalpaalakurthi — Fri, 15 Jul 2022 10:52:02 +0000

Introduction

Hazelcast is a robust, open-source in-memory data grid (IMDG) solution, renowned for its scalability, high performance, and fault tolerance. One of its key features is data persistence, which ensures data durability even in the face of failures. In this article, we'll delve into using Hazelcast persistence with a file system storage backend, exploring its benefits and providing practical examples.

Why File System Storage?

Utilizing a file system for storage in Hazelcast offers several advantages:

Simplicity: File system storage is straightforward to set up and manage, making it an ideal choice for many applications.
Durability: Data stored in files persists even if the application or the Hazelcast cluster is restarted, ensuring high availability and data integrity.
Scalability: File systems can scale easily by adding more storage capacity, allowing Hazelcast to store vast amounts of data without sacrificing performance.
Compatibility: File system storage is compatible with various operating systems and environments, making it versatile for different deployment scenarios.

Setting Up Hazelcast with File System Persistence

To enable Hazelcast persistence with file system storage, follow these steps:

Configure Hazelcast: Update your Hazelcast configuration file (hazelcast.xml) to include the necessary settings for persistence. Specify the persistence type as file, and provide the directory path where the data will be stored.
```
<hazelcast>
    <persistence>
        <base-dir>/path/to/storage</base-dir>
        <enabled>true</enabled>
        <cluster-data>false</cluster-data>
    </persistence>
</hazelcast>
```
Start Hazelcast: Launch your Hazelcast cluster with the updated configuration file. Hazelcast will now persist data to the specified file system directory.

Example Usage

Let's consider a simple Java application that utilizes Hazelcast with file system persistence:

import com.hazelcast.core.Hazelcast;
import com.hazelcast.core.HazelcastInstance;
import java.util.Map;

public class HazelcastExample {
    public static void main(String[] args) {
        // Create a Hazelcast instance
        HazelcastInstance hazelcastInstance = Hazelcast.newHazelcastInstance();

        // Get a distributed map from Hazelcast
        Map<Integer, String> distributedMap = hazelcastInstance.getMap("my-distributed-map");

        // Put some data into the map
        distributedMap.put(1, "Value 1");
        distributedMap.put(2, "Value 2");
        distributedMap.put(3, "Value 3");

        // Retrieve data from the map
        System.out.println("Value for key 1: " + distributedMap.get(1));
        System.out.println("Value for key 2: " + distributedMap.get(2));
        System.out.println("Value for key 3: " + distributedMap.get(3));

        // Shutdown Hazelcast instance
        hazelcastInstance.shutdown();
    }
}

Conclusion:

Incorporating file system storage for persistence in Hazelcast offers a reliable solution for storing and managing distributed data. By following the outlined steps and examples, developers can leverage Hazelcast's powerful features while ensuring data durability and fault tolerance. Whether it's for caching, session management, or distributed computing, Hazelcast with file system persistence provides a robust foundation for building scalable and resilient applications.

Start harnessing the power of Hazelcast persistence with file system storage today and unlock new possibilities for your distributed systems architecture.

Monitoring Cassandra Metrics using Grafana

vishalpaalakurthi — Wed, 04 Nov 2020 08:36:09 +0000

Prerequisites

jmx_prometheus_javaagent. (https://mvnrepository.com/artifact/io.prometheus.jmx/jmx_prometheus_javaagent/0.14.0)
Sample cassandra.yml configuration for jmx_prometheus_javaagent. (https://raw.githubusercontent.com/prometheus/jmx_exporter/master/example_configs/cassandra.yml)
Prometheus server. (https://prometheus.io/download/)
Grafana Server. (https://grafana.com/grafana/download)

Steps to follow

Download jmx_prometheus_javaagent jar file.
Download cassandra.yml configuration file for producing jvm and cassandra metrics.
Add JVM_OPTS to cassandra-env.sh file.
JVM_OPTS="$JVM_OPTS javaagent:/etc/cassandra/jmx_prometheus_javaagent-0.10.jar=7070:/etc/cassandra/cassandra.yml"
Restart cassandra server. http://localhost:7070/metrics, this URL provides JVM and cassandra Metrics.
Download Prometheus server.
Add below lines to prometheus.yml for Prometheus server configuration.job_name: ‘cassandra’static_configs:
- targets: [‘ip:7070’]
Download Grafana Server.
Setup Prometheus data source to Grafana.
Create Metrics dashboard using provided queries in Grafana.
Grafana dashboard is exposed to localhost:3000.
Prometheus server exposed to localhost:9090.

Notes

jmx_prometheus_javaagent is to be added to each cassandra node.

Architecture

How it works

Prometheus scrapes metrics from instrumented jobs, either directly or via an intermediary push gateway for short-lived jobs. It stores all scraped samples locally and runs rules over this data to either aggregate and record new time series from existing data or generate alerts. Grafana or other API consumers can be used to visualize the collected data.

References

Steps for setting up Cassandra metrics through grafana. (https://community.grafana.com/t/how-to-connect-prometheus-to-cassandra/1153/11)
https://grafana.com/docs/grafana/latest/
https://github.com/prometheus/prometheus
https://prometheus.io/docs/introduction/overview/

Sample Image

How to detect loop in linkedlist?

vishalpaalakurthi — Mon, 26 Oct 2020 14:21:17 +0000

1) Two pointers (each node is assigned with an address)

2) Traverse through each node till end , tracking visited node using visited flag.

If you find node that is already visited, then there is a loop in LinkedList and if you reach till end while traversing then there is no loop in LinkedList

How to refresh/reload application properties runtime in spring boot ?

vishalpaalakurthi — Thu, 24 Sep 2020 10:00:47 +0000

In this tutorial, we are going to learn how to reload application properties in spring boot.

We have many options in spring boot, now I am explaining the easiest one here.

Refresh beans with @ConfigurationProperties

For Reloading properties, spring cloud has introduced @RefreshScope annotation which can be used for refreshing beans.

Spring Actuator provides different endpoints for health, metrics. but spring cloud will add extra end point /refresh to reload all the properties.

Required maven/gradle dependencies

Spring actuator
Spring cloud starter

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter</artifactId>
</dependency>

Then, add below property to application properties file

management.endpoints.web.exposure.include=refresh

// Db Properties, Values to be store in a Map
db.dbProps.awsEndPoint=localhost:8080/aws/
db.dbProps.azureEndPoint=localhost:8080/azure/

Now, Create a class for configuring properties and add @RefreshScope annotation to class.

@Component
@ConfigurationProperties(prefix = "db")
@RefreshScope
public class DbProperties {

      // This is for storing application properties in a Map
    public Map<String, String> dbProps;

    public Map<String, String> getDbProps() {
        return dbProps;
    }

    public void setDbProps(Map<String, String> dbProps) {
        this.dbProps = dbProps;
    }

        // Get property value using key
    public String getDbPropData(String key) {
        return dbProps.get(key);
    }
}

Above code create a scope for refreshing application properties data.

Now, when you change data in application properties. we need to give a POST REST call using below URL.

http://localhost:8080/actuator/refresh

from now, we can get new value from the next access.

Thank you for reading my blog. 🤩

Why coloring terminal is very important!

vishalpaalakurthi — Wed, 16 Sep 2020 06:39:01 +0000

Developers who works for testing their applications or test cases on cloud environments instead of local environments are definitely will use terminals (i.e; Linux, poweshell and Gitbash etc).

Everyday we use to work on multiple tasks at a time. some times we do refer production environments for logs and to re-produce the same issue.

On peak times, we do wants to complete the tasks as faster as possible. same time we may confuse our selves while doing the modifications/deployments.

Then, definitely will get confused. we might do changes in production environment instead local environment. this definitely creates a huge problem, production server might stop because of simple thing.

To avoid this, terminals provide adding background color to the sessions. Using that, we can identify the environment we are working on.

As i use putty mostly, i am providing steps to add background color to session.

Open Putty
Select saved session and click on load button
On right side category, select Colours tab
Go to "select a colour to adjust block", then select "Default background" option.
Now select modify, then select color and click OK.
go to session tab, click Save button.

Now its got changed to selected background color and we might avoid a simple mistakes.

Thanks for reading my blog, please keep in touch for my next blog :-)

How to build your own Cassandra workbench

vishalpaalakurthi — Wed, 16 Sep 2020 05:04:38 +0000

This blog refers to building a simple and easy Cassandra workbench using well known and mostly used technology stack.

"too lazy to do it"

Above quotation saved lots of my time, increased my productivity and switched me to create this workbench.

before going to the key point.

let's understand what is actually Cassandra is ?

Apache Cassandra is an open source, distributed NoSQL database that began internally at Facebook and was released as an open-source project in July 2008. Cassandra delivers continuous availability (zero downtime), high performance, and linear scalability that modern applications require, while also offering operational simplicity and effortless replication across data centers and geographies. Cassandra can handle petabytes of information and thousands of concurrent operations per second, enabling organizations to manage large amounts of data across hybrid cloud and multi cloud environments.

Large companies like Google, Microsoft, Github, Instagram, Uber, Datastax other thousands of companies are using on their own basis.

Now, Coming to the key point

i as a developer using Cassandra, most of times use to logging into the terminal and executes the query. it's always kills my productivity because i have to login into terminal for executing even a simple select queries also.

Most of non-database people like me will only use database for verifying the data. i.e executes select queries only.

Then, i have decided to have my own a simple workbench where i can configure the multiple node sessions and verify the required node data.

Finally after having thoughts as a back-end developer i have chosen Datastax for their pre-built dependencies and other technology stacks like

Java8
Spring boot
Datastax Cassandra dependencies
Html5
Css
Javascript
Thymeleaf
Cassandra CQL (for verifying data)
Maven build tool

Now, What i have done for building a simple workbench is very simple.

Datastax is providing a dependencies where we create maintain Cassandra session and execute queries.

We can configure session and execute CQL queries as below.

Cluster cluster = Cluster.builder().addContactPoint(ipAddress)
                  .withCredentials(userName, password)
                  .withPort(9042)
                  .withoutJMXReporting().build();

Session session = cluster.connect(keyspace); //keyspace is optional

String query = "select * from keyspace.tableName";
session.execute(query);

we really get a question like how we can handle the exceptions ?

we no need of worrying about exceptions because the Datastax dependencies are already taking care internally and returning proper exception messages as below.

[Invalid query] message="unconfigured table test1"

As i am using spring boot, i have defined a singleton beans then i can use session any where in the application.

here, to execute the queries we required a query that to be passed dynamically. so i have created a REST API using spring boot. it accepts query as string and returns list of rows.

@GetMapping("/query")
public List<Object> getDataUsingQuery(@RequestParam("query") String query) {
    List<Object> dataByQuery = cassandraService.getDataByQuery(query.trim());
    return dataByQuery;
}

right now, i restricted my workbench for executing only select queries. we can enhance it further.

Finally, we should have a proper UI for executing query and displaying the results.

To make it as simple as possible, i am not attaching any images here. please do refer my GitHub URL for source code and sample UI.

https://github.com/vishalpaalakurthi/cassandraboard.git

Referneces

Thank you for reading my first blog, please keep in touch for my next one. :-)