Forem: Dapinder

How you can have Facebook like push notifications

Dapinder — Fri, 13 Oct 2023 15:40:40 +0000

Do you even fancy how to design a Facebook like or many other push notifications from scratch without using a third-party API or library i.e. at the core level? Have a read.

A notification in a message you can display to the user outside of your application's normal UI. When you tell the system to issue a notification, it first appears as an icon in the notification area. To see the details of the notification, the user opens the notification drawer. Both the notification area and the notification drawer are system-controlled areas that the user can view at any time. System will also display the notification as a TOAST when user is logged-in along with increment in the notification drawer.

Lightweight Event Driven Notification System

1.1 Event Generation

The event generation phase happens while processing the response to a user request. As such, we wanted to ensure that there was little or no impact on the response times from where notification is called/generated. To ensure this was the case, all we will do is store the minimum amount of data possible/lightweight message on the queue. We store the minimum amount of data possible or a lightweight object, just a few identifiers, so it shall not slow down the response just to kick off a push notification. Everything after this initial message drop on the queue action is processed out by another consumer which shall produce no impact on site performance.
All calls to the notification shall drop quickly drop a message on the queue.

1.2 Event Targeting

This is the event targeting phase which will receive event from the queue. We may or may not have more than one worker reading from the queue. When a worker receives an event, it loads up any additional information necessary (additional processing) to act on the notification. E.g. for checking to see what users should be notified, whether those users have devices that are registered to receive notifications or if they’ve opted out of notifications of this type etc.

1.3 Message Delivery

The web has been largely built around the so-called request/response paradigm of HTTP. A client loads up a web page and then nothing happens until the user clicks onto the next page. AJAX started to make the web feel more dynamic. Still, all HTTP communication was steered by the client, which required user interaction or periodic polling to load new data from the server. All of these work-arounds share one problem: They carry the overhead of HTTP and also numerous unnecessary HTTP connections when there isn’t actually any update available with the server. Because of this we decided to use NODE JS along with Web Sockets. Node JS will receive the event from queue when it’s published. The WebSocket connection is a persistent connection between the client and the server and both parties can start sending data at any time. The Web Socket connection here will be established at the user login. Based on to whom message belongs and logged in user, the connected Web Socket will deliver the notification on the UI. Shall be only one dedicated Web Socket connection per browser. Single web socket connection can be used not only just dropping notifications but performing other tasks too based on events. This phase will take care of persistence too.

Scalability and Availability

Ques 1. : How do we ensure that the delivery to MQs are both scalable and highly available?
Ques 2. : What if the Node JS Worker crashes?

The first step to ensuring a service is scalable is to divide the workload.
Workload is divided across two queues configured in ActiveMQ. Already having specialization in managing/configuring ActiveMQ which is a very powerful message broker with advanced features for enhanced availability, scalability, performance and reliability. To provide more scalability on ActiveMQ Brokers, we can allow many brokers to be connected into a network.
Java Queue Consumer: Spring DMLC took its inspiration from MDBs, it did not replicate these disadvantages; quite the opposite, in fact. The Spring DMLC is commonly used to create what have become known as Message-Driven POJOs (MDP). MDPs offer all of the same functionality as MDBs but without the disadvantages listed above. The Spring DMLC provides many features including Caching JMS resources, dynamically grow and shrink consumers, re-establishes connection on non-availability of broker etc.
Node Consumer: ActiveMQ is written in Java and has a full-fledged JMS client, but its STOMP support and RESTful API let us interface with the messaging queue quite nicely using Node.js. Node.js is single threaded and currently we would go with only one Node JS Worker and if required can divide the workload across individual Node.js processes anyway, so this can work out well. Configuring our Node.js processes in this way makes it easy to scale horizontally. Whenever we need to add more processing power to the cluster, we can just add more servers. If only one and individual Node JS instance is down, message would still be persisting with the broker. With Node, you’ll find that STOMP is a great choice of protocol for subscribing to queues. Using node-stomp as a STOMP CLIENT as a way of connecting to queues
- a) Ability to support different STOMP protocol versions.
- b) ACK and NACK support
Web Socket: The WebSocket API enables web applications to handle bidirectional communications with server-side process in a straightforward way. Socket-io certainly makes working with Websockets very easy. It abstracts many transports, including AJAX long-polling and WebSockets, into a single API. It allows developers to send and receive data without worrying about cross-browser compatibility. Socket.IO provides both server-side and client-side components with similar APIs. Limiting to the user, we would be using Web Socket for now only send data from the server so it would be Unidirectional.
UI: Socket-io client and JQuery.

Notification Persistence, Quick & Guaranteed Delivery

All notifications will ultimately make to a persistence store. User shall be shown same Notifications in the notification drawer irrespective of the device usage. Persistence shall also allow to keep track of the notifications delivered or not delivered to the user and mark appropriately. This shall provide support to deliver the pending notifications to the user when he is not available or logged-in to the next time when he logs in.
The choice is MongoDB.
MongoDB being a No-SQL database works with documents and collections.

*The persistence shall have minimal or no impact on the notification ‘time to reach’ to the user. *

1.1 Being Quick

Notifications are usually a highly read object, so we want to be able to get the list of currently unread notifications very quickly, not try and calculate on the fly.
Being Event-Driven, notifications (intended/subscribed) are delivered to users available in real-time as soon as event is generated (prior persisting).
The event targeting phase needs to make sure to drop a lightweight notification object on the queue along with list of intended recipient users.

1.2 Guaranteed Delivery

For guaranteed delivery of the notifications when user is not logged-in or auto logged-off by the system and for making notifications available all the time irrespective of the user device. System will persist and will make available the notifications next time user logs in.

Data Model

Creating your own Cache with LRU Algorithm in Java

Dapinder — Tue, 10 Oct 2023 13:18:28 +0000

The LRU caching scheme is to remove the least recently used frame when the cache is full and a new page is referenced which is not there in cache. Please see the Galvin book for more details.
This class defines a simple cache entry for the cache. A cache entry holds cached object along with some metadata attributes. These metadata attributes assist in managing the cache.
These cache entries are not serializable and thus these are not cluster aware.

This is a simple hash map based cache with LRU cleanup algorithm.

CacheEntry.java

package com.dapinder.example.own.cache;

public class CacheEntry
{
 //the object that is saved in this cache entry
 private K cachedObject;

 //the time when the cached object was accessed or modified last time.
 private long lastAccessTime;

 /**
  * This returns the object stored in a cached entry.
  * 
  * @return the cachedObject
  */
 public K getCachedObject()
 {
  lastAccessTime = System.currentTimeMillis();
  return cachedObject;
 }

 /**
  * This sets an object into the cache entry
  * 
  * @param cachedObject
  *           the cachedObject to set
  */
 public void setCachedObject(final K cachedObject)
 {
  this.cachedObject = cachedObject;
  lastAccessTime = System.currentTimeMillis();
 }

 /**
  * Returns the time when this object was last accessed.
  * 
  * @return the lastAccessTime
  */
 public long getLastAccessTime()
 {
  return lastAccessTime;
 }


 @Override
 public int hashCode()
 {
  final int prime = 31;
  int result = 1;
  result = prime * result + ((cachedObject == null) ? 0 : cachedObject.hashCode());
  result = prime * result + (int) (lastAccessTime ^ (lastAccessTime >>> 32));
  return result;
 }


 @Override
 public boolean equals(final Object obj)
 {
  if (this == obj)
  {
   return true;
  }
  if (obj == null)
  {
   return false;
  }
  if (getClass() != obj.getClass())
  {
   return false;
  }
  final CacheEntry other = (CacheEntry) obj;
  if (cachedObject == null)
  {
   if (other.cachedObject != null)
   {
    return false;
   }
  }
  else if (!cachedObject.equals(other.cachedObject))
  {
   return false;
  }
  if (lastAccessTime != other.lastAccessTime)
  {
   return false;
  }
  return true;
 }

 @Override
 public String toString()
 {
  return "CacheEntry [cachedObject=" + cachedObject + ", lastAccessTime=" + lastAccessTime + "]";
 }
}

CacheManager.java

This manages a cache.

import java.util.Collection;

public interface CacheManager {
 /**
  * Returns a cached item by key. If no cached item by key is found, returns
  * null.
  * 
  * @param key
  *            unique key for the cached item. This key is case sensitive.
  * @return the cached object; Null if key not found
  * 
  * @exception NullPointerException
  *                if key passed is null or empty.
  */
 K getCachedObject(String key);

  /**
  * Adds an item to the cache. If the key already exists, it updates the
  * cached object with the new item.
  * 
  * @param key
  *            unique key for the cached item. The key is case sensitive.
  * @param objectToCache
  *            the object to cache
  * 
  * @exception NullPointerException
  *                if key passed is null or empty.
  */
 void addObjectToCache(String key, K objectToCache);

  /**
  * Returns the keys of all cached objects.
  */
 Collection getAllKeys();

}

CacheManagerImpl.java

This is an implementation of CacheManager. Ideally this class should be used through Spring beans. This is a simple hash map based cache with LRU cleanup algorithm.

package com.dapinder.example.own.cache;

import java.util.Collection;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Timer;
import java.util.TimerTask;
import java.util.concurrent.ConcurrentHashMap;

import org.apache.commons.lang.StringUtils;
import org.apache.log4j.Logger;

/**
 * This is an implementation of CacheManager.
 * This is a simple hash map based cache with LRU cleanup
 * algorithm.
 * 
 * @author Dapinder Singh
 * 
 * @see com.dapinder.example.own.cache.CacheManager
 * @see com.dapinder.example.own.cache.CacheEntry
 */

public class CacheManagerImpl implements CacheManager {
    /** Logger for this class and subclasses */
    private final Logger        logger = Logger.getLogger(this.getClass());

    // The map to hold cached objects
    private Map> cache;

    // the interval (in milliseconds) after which cache refresh task should run.
    private int   refreshTimeInterval;

    // the maximum number of entries in cache
    private int   maxCacheEntries;

    // the maximum time (in milliseconds) of inactivity for which an object will
    // remain in cache
    private int   maxHoldTime;

    // the name of the cache
    private String       cacheName;


 @Override
 public void addObjectToCache(final String key, final K objectToCache) {
 if (StringUtils.isEmpty(key)) {
     throw new NullPointerException("Key provided for adding the item to the cache is null.");
 }

 // create Cache entry
 CacheEntry entry = cache.get(key);
 if (null == entry) {
     entry = new CacheEntry();
 }
 entry.setCachedObject(objectToCache);

 // Add entry to the cache
 cache.put(key, entry);
}


 @Override
 public K getCachedObject(final String key) {
 if (StringUtils.isEmpty(key)) {
     throw new NullPointerException("Key for fetching the item from cache is null.");
 }
 final CacheEntry cacheEntry = cache.get(key);
 if (null == cacheEntry) {
     return null;
 }

 return cacheEntry.getCachedObject();
}

    /**
     * Sets the interval between the two consecutive refresh tasks for this
     * cache.
     * 
     * @param refreshTimeInterval
     *            the refreshTimeInterval to set
     */
    public void setRefreshTimeInterval(final int refreshTimeInterval) {
 this.refreshTimeInterval = refreshTimeInterval;
    }

    /**
     * Sets the minimum count of objects that will trigger the cache clean up.
     * 
     * @param maxCacheEntries
     *            the maxCacheEntries to set
     */
    public void setMaxCacheEntries(final int maxCacheEntries) {
 this.maxCacheEntries = maxCacheEntries;
    }

    /**
     * Sets the maximum time of inactivity for which an object will remain in
     * cache.
     * 
     * @param maxHoldTime
     *            the maxHoldTime to set
     */
    public void setMaxHoldTime(final int maxHoldTime) {
 this.maxHoldTime = maxHoldTime;
    }

    /**
     * Sets the name of the cache. This is solely from the perspective of
     * logging.
     * 
     * @param cacheName
     *            the cacheName to set
     */
    public void setCacheName(final String cacheName) {
 this.cacheName = cacheName;
    }


@Override
public int hashCode() {
 final int prime = 31;
 int result = 1;
 result = prime * result + ((cache == null) ? 0 : cache.hashCode());
 result = prime * result + ((cacheName == null) ? 0 : cacheName.hashCode());
 result = prime * result + maxCacheEntries;
 result = prime * result + maxHoldTime;
 result = prime * result + refreshTimeInterval;
 return result;
    }


@Override
public boolean equals(final Object obj) {
 if (this == obj) {
     return true;
 }
 if (obj == null) {
     return false;
 }
 if (getClass() != obj.getClass()) {
     return false;
 }
 final CacheManagerImpl other = (CacheManagerImpl) obj;
 if (cache == null) {
     if (other.cache != null) {
  return false;
     }
 }
 else if (!cache.equals(other.cache)) {
     return false;
 }
 if (cacheName == null) {
     if (other.cacheName != null) {
  return false;
     }
 }
 else if (!cacheName.equals(other.cacheName)) {
     return false;
 }
 if (maxCacheEntries != other.maxCacheEntries) {
     return false;
 }
 if (maxHoldTime != other.maxHoldTime) {
     return false;
 }
 if (refreshTimeInterval != other.refreshTimeInterval) {
     return false;
 }
 return true;
}


@Override
public String toString() {
 return "CacheManagerImpl [cache=" + cache + ", cacheName=" + cacheName + ", maxCacheEntries=" + maxCacheEntries
  + ", maxHoldTime=" + maxHoldTime + ", refreshTimeInterval=" + refreshTimeInterval + "]";
}

    /**
     * This method initializes the CacheManager. Specifically, this instantiates
     * the cache and sets up the refresh process for the cache.
     */
    public void initialize() {
     // instantiate the cache
     instantiateCache();
     // setup the refresh Process
     setRefreshProcess();
    }

    /**
     * This instantiate the cache. The cache is a sorted cache based on
     */
    private void instantiateCache() {
      cache = new ConcurrentHashMap>();

    }

    /**
     * This method sets the refresh process for the cache.
     */
    private void setRefreshProcess() {
 final TimerTask refreshTask = new RefreshTask();

 final Timer refreshTimer = new Timer();
 refreshTimer.scheduleAtFixedRate(refreshTask, 0, refreshTimeInterval);
    }

    /**
     * This defines the process of refreshing the cache. The cache refreshing
     * policy is simple LRU. However, cache is not cleaned, it it has not
     * attained its optimum size.
     */
    private void cleanCache() {
 // cache must be larger than optimum size to be cleaned.
 if (!isCacheLargerThanOptimumSize()) {
     return;
 }

 // remove all old objects
 removeOldObjects();
    }

    /**
     * It finds all the cacheEntries that are not used for the max hold time and
     * removes those objects.
     */
    private void removeOldObjects() {
 int removedItemCount = 0;
 for (final Entry> mapEntry : cache.entrySet()) {
     if (maxHoldTime <= mapEntry.getValue().getLastAccessTime()) {
  cache.remove(mapEntry.getKey());
  removedItemCount++;
     }
 }

 if (logger.isInfoEnabled()) {
     logger.info(removedItemCount + "items removed from " + cacheName);
 }
}

    /**
     * Checks if the size of the cache is larger than optimum cache size
     * @return
     */
    private boolean isCacheLargerThanOptimumSize() {
      return maxCacheEntries < cache.size();
    }

  /**
  * This defines the cache cleanup task.
  */
  private final class RefreshTask extends TimerTask {

 /**
  * (non-Javadoc)
  * @see java.util.TimerTask#run()
  */
 @Override
 public void run() {
     cleanCache();
 }
}

/**
* (non-Javadoc)
* @com.dapinder.example.own.cache.CacheManager#getAllKeys()
*/
@Override
public Collection getAllKeys() {
 return cache.keySet();
}
}

Are you aware of this usage of final keyword in java?

Dapinder — Tue, 10 Oct 2023 09:45:18 +0000

Most of us are aware of final keyword in java for the following:

Prevent Inheritance by declaring class as final.
Prevent method overriding by making method as final.
Declaring constants.

I am here to let know more advantages of final keyword which are not so generally aware of:

Primitive types like int, char, float etc. other than arrays are substituted at compile time with their values, if you change a final that is used by other classes, you must remember to recompile those other classes or your change will not take effect. The same rule applies to constants of type java.lang.String. Although String is a constructed type, it is also substituted at compile time. All constructed types other than String, mutable or not, are not substituted at compile time. To understand how this works

Consider the following code fragment:

package com.dapinder.practice;

public class FinalConstantSubstitute {

 /** A string constant */
 public final static String STRING_VAR = "Dapinder's Blog";

 /** An int constant. */
 public final static int INT_VAR = 15;

 /** A double constant. */
 public final static double DOUBLE_VAR = 13.23d;

 /** An array constant. */
 public final static int[] ARRAY_VAR = new int[] { 2, 2, 2, 2, 2, 2 };

 /** A Wrapper constant. */
 public final static Integer A_OTHER_WRAPPER = new Integer(10);

 public static void main(String[] args) {

  System.out.println(STRING_VAR);
  System.out.println(INT_VAR);
  System.out.println(DOUBLE_VAR);
  System.out.println(ARRAY_VAR);
  System.out.println(A_OTHER_WRAPPER);

 }
}

Once the compiler sees a code as above, it optimises it and starts substituting the primitives and String objects.

ByteCode view of the FinalConstantSubstitute post as follows: (used java decompiler)

Be cautious of using final constants when accessing in some other classes. Since above is a single class so its not getting noticed but if you use the declared public constants in some other class, then changing of constant, you have to recompile the class making call.

Check below for example:

 package com.dapinder.practice;

 public class FinalConstantSubstituteMain {

 /**
 * The class using declared final constants.
 * 
 * @param args
 */
 public static void main(String[] args) {

  /** Substituted values */
  System.out.println(FinalConstantSubstitute.STRING_VAR);
  System.out.println(FinalConstantSubstitute.INT_VAR);
  System.out.println(FinalConstantSubstitute.DOUBLE_VAR);
  System.out.println(FinalConstantSubstitute.ARRAY_VAR);
  System.out.println(FinalConstantSubstitute.A_OTHER_WRAPPER);

 }
}

ByteCode view

Here compiler will substitute the constant's values from FinalConstantSubstitute. For example, change of constant STRING_VAR to some other value say Google will not reflect in FinalConstantSubstituteMain unless FinalConstantSubstituteMain is re-compiled. The reason is obvious since substitution happens at compile time and is done by the compiler.

Conditional Compilation

This will amaze you when you think how final variables can provide a way where certain lines get conditional compilation based on a particular condition. This can actually save on most debugging code in production and will improve the performance as well.

To understand its power, lets take an example:

package com.dapinder.practice;

public class ConditionalFinal {

 private static boolean isLoggingEnabled = false;

 /**
 * @param args
 */
 public static void main(String[] args) {
  if (isLoggingEnabled) {
   System.out.println("Statment 1");
  }
  if (isLoggingEnabled) {
   System.out.println("Statment 2");
  }
  if (isLoggingEnabled) {
   System.out.println("Statment 3");
  }
  if (isLoggingEnabled) {
   System.out.println("Statment 4");
  }
 }
}

Please note that variable isLoggingEnabled is not declared final here.

Bytecode view:

Please note that all if conditions are very well referencing the variable and if the isLoggingEnabled is true, i will print on the console and when false it wont print but yet have although less on performance as it will check each conditional IF statement.

This is fine and at first glance does not seem expensive but in production where one method is called by millions of threads and multiple times, this can get expensive.

Most of us know and have used Log4J and we know not all logging statements are turned on in production. Most of us even do a isDebugEnabled before logging a debug statement.
However When using the isDebugEnabled( ), the cost is much more than one comparison per if statement.

Now check by converting the isLoggingEnabled variable into a final variable:

 package com.dapinder.practice;

public class ConditionalFinal {

 private static final boolean isLoggingEnabled = false;

 /**
 * @param args
 */
 public static void main(String[] args) {
  if (isLoggingEnabled) {
   System.out.println("Statment 1");
  }
  if (isLoggingEnabled) {
   System.out.println("Statment 2");
  }
  if (isLoggingEnabled) {
   System.out.println("Statment 3");
  }
  if (isLoggingEnabled) {
   System.out.println("Statment 4");
  }
 }
}

By converting the isLoggingEnabled attribute into a final attribute, you have told the compiler that whenever it sees isLoggingEnabled, it should replace it with false as per substitution principle explained above.

One more important point here.

Most of us do get WARNING in any IDE (say Eclipse) whenever a piece of code is unreachable. Here compiler applies the same principle and removes all those statement which are meant to be unreachable or will never execute.

Here, in above example, all these logging statements will disappear.

Heap Dump Analysis

Dapinder — Mon, 09 Oct 2023 13:55:07 +0000

The article was originally published on Blogspot on April 24, 2013

When we get an OOM (out of memory) error, we should try and understand what is going on inside the JVM. The best way to do that is a heap dump. The general idea is to dump out the contents of the JVM’s memory into a file, then analyse it using a tool.

An OOM simply means that the JVM ran out of memory. When this occurs, we basically have 2 choices:

Allow the JVM to use more memory i.e. increase the –Xmx. But the question is till what extent?
Improve/Fix the application so that it uses less memory. A memory leak happens when the application keeps more and more references to objects and never releases them. The garbage collector will therefore never collect those objects. At this point, the JVM will throw an OOM. To solve this issue, we first need to find the root cause of it.

Java has a built-in feature for dumping heap snapshots called HeapDump.

What is a heap dump?

A heapdump as the name suggests is a dump of the heap. It will allow us to navigate the heap and see what objects use all the heap memory and which are the ones that still keep a reference on them, and so on and so forth. This probably will give us very strong hints and we will (hopefully) be able to find the root cause of the problem. A heap dump is a snapshot of the memory of a Java process at a certain point of time. There are different formats for persisting this data, and depending on the format it may contain different pieces of information, but in general the snapshot contains information about the java objects and classes in the heap at the moment the snapshot was triggered. HeapDump for servers can provide view into the memory profile of application at the time it crashed can help us figure out what caused the error.

What Are Heap Dumps Good for?

If there is a system which is crashing sporadically with an OutOfMemoryError.
If you want to analyse what the footprint into memory of your application is, then heap dumps are again a good choice to find/analyse:
- Leak Suspects: includes leak suspects and a system overview.
- Top Components: list reports for components bigger than 1 percent of the total heap.
- Histogram: Lists number of instances per class.
- Dominator Tree: List the biggest objects and what they keep alive.
- Top Consumers: Print the most expensive objects grouped by class and by package.
- Duplicate Classes: Detect classes loaded by multiple class loaders.

Types of Heap Dumps

Binary Heap Dumps (HPROF - produced by Sun, HP, SAP, etc… JVMs) Java has a built-in feature for dumping heap snapshots to files in HPROF binary format.
IBM System Dumps
IBM Portable Heap Dumps (PHD)

⚠️ This article is restricted to HPROF format only.

How To Get a Heap Dump (HPROF snapshots can be created in the following ways):

Automatically when OutOfMemory occurs: Since JDK 1.5, it has built-in mechanism to help debug pesky out-of-memory errors which is to set the below parameter and get the heap dump whenever out of memory occurs:
- -XX:+HeapDumpOnOutOfMemoryError - Dumps heap to file when java.lang.OutOfMemoryError is thrown.
- Start the application with the VM argument:
  - -XX:+HeapDumpOnOutOfMemoryError
  - -XX:HeapDumpPath= C:/opt/tomcat/current/logs. (This will tell the VM to produce a heap dump when an OOM occurs) By default the heap dump file is placed into the VM base directory but introducing -XX:HeapDumpPath , we can provide a custom location. This location can be shared one so that we can take the file and analyse it using any tool.
Explicitly via jmap utility (Sun Java 6): Sun Java 6 has the jmap utility that allows to connect to running Java process and dump Java heap:

jmap -dump:format=b,file=file_name.hprof

(Hint: to learn the PID (process identifier) of running JVM, you can use jps or jconsole JDK utilities.)

The benefit is that memory can be analyzed on demand with no additional configuration of JVM or Java application. You can dump memory of any running instance of JVM that supports this feature.

Explicitly via jconsole utility (Sun Java 6)

Sun Java has the jconsole utility that lets you to connect to a running Java 6 process for monitoring and management.
Using jconsole, you can dump Java heap via HotSpotDiagnostic MBean

Analyze Heap Dump using API

Use a tool such as YourKit, VisualVM, MAT or jhat to analyze the heapdump (the name of the dump file is in format: java_pidXXXX.hprof) file.

Example Usage:

The sample profiler screenshot showed 10 "BIGGEST OBJECTS" and it seems all are containing a same String Object wrapped up. Biggest object size is around 100 MB and rest 9 follow it.

Understanding Java OutOfMemory Issues

Dapinder — Thu, 05 Oct 2023 11:15:38 +0000

The article was written for Java 5 and originally published on Blogspot on April 24, 2013

One of the most common problems that beset enterprise applications is the dreaded OutOfMemoryError. The error is typically followed by one of the following:

Either you application server crashes.
Or you have severe performance bottleneck.
A stuck thread waiting on resources leading nowhere and ultimately crashing the server.

Regardless of the symptoms, you will most likely need to reboot the application server to make think ok.

Causes of out-of-memory errors

Before you attempt to resolve an out-of-memory error, first understanding how it can occur is beneficial. If the JVM runs out of memory anywhere in its process memory space, including all regions in the heap as well as the permanent memory space, and a process attempts to create a new object instance, the garbage collector executes to try to free enough memory to allow the new object's creation. If the garbage collector cannot free enough memory to hold the new object, then it throws an OutOfMemoryError.

Out-of-memory errors most commonly result from Java memory leaks. Java memory leak is the result of maintaining a lingering reference to an unused object: you are finished using an object, but because one or more other objects still reference that object, the garbage collector cannot reclaim its memory. The memory occupied by that object is thus lost from the usable heap.
An object is considered garbage when it can no longer be reached from any pointer in the running program. The most straightforward garbage collection algorithms simply iterate over every reachable object. Any objects left over are then considered garbage. The time this approach takes is proportional to the number of live objects, which is prohibitive for large applications maintaining lots of live data.

These types of memory leaks typically occur during Web requests, and while one or two leaked objects may not crash your application server, 10,000 or 20,000 requests might. Furthermore, most objects that are leaked are not simple objects such as Integers or Doubles, but rather represent subgraphs within the heap. For example, you may inadvertently hold on to a Person object, and that Person object has a Profile object that has several PerformanceReview objects that each maintain sets of data. Rather than losing 100 bytes of memory that the Person object occupies, you lose the entire subgraph that might account for 500 KB or more of memory.

JVM memory management

The Java HotSpot VM garbage collector uses generational GC. Generational GC takes advantage of the observation that most programs conform to the following generalisations.

They create many objects that have short lives, for example, iterators and local variables.
They create some objects that have very long lives, for example, high level persistent objects.

Garbage collection occurs in each generation when the generation fills up. Sun JVM is divided into generations and is into following:

Young generation, including Eden and two survivor spaces (the From space and the To space)
Old generation
Permanent generation (Non-Heap from JDK 1.6 onwards)
Code-Cache (Non-Heap from JDK 1.6 onwards)

Young generation, including Eden and two survivor spaces: The vast majority of objects are allocated in a pool dedicated to young objects (the young generation) (Objects are created in Eden), and most objects die there. .
When the young generation fills up it causes a Minor Collection in which only the young generation is collected; garbage in other generations is not reclaimed.
The young generation consists of Eden and two Survivor spaces. When Eden is full, the garbage collector iterates over all objects in Eden, copies live objects to the first survivor space, and frees memory for any dead objects. When Eden again becomes full, it repeats the process by copying live objects from Eden to the second survivor space, and then copying live objects from the first survivor space to the second survivor space.

Objects are copied between survivor spaces in this way until they are old enough to be tenured (copied to the tenured generation). Most objects become garbage short after creation (die young)

By separating persistent objects from short-lived objects and moving them to designated spaces, the garbage collector can free memory for use by the application (improving efficiency of memory use), and avoid examining every object each time a collection is done (reduce garbage collection overhead).

Old/Tenured generation: Typically, each time during Minor Collection, some objects gets survived (second survivor space fills) and live objects remain in Eden or in the first survivor space, then these objects are tenured (that is, they are copied to the old generation) and are further moved to the tenured generation. Eventually, the tenured generation will fill up and must be collected Or When the garbage collector cannot reclaim enough memory by executing this type of minor collection, also known as a copy collection, then it performs a Major Collection, in which the entire heap is collected. Major collections usually last much longer than minor collections because a significantly larger number of objects are involved.

Non-heap memory includes a method area shared among all threads and memory required for the internal processing or optimization for the Java VM. It stores per-class structures such as a runtime constant pool, field and method data, and the code for methods and constructors. The method area is logically part of the heap but, depending on the implementation, a Java VM may not garbage collect or compact it. Like the heap memory, the method area may be of a fixed or variable size. The memory for the method area does not need to be contiguous. In addition to the method area, a Java VM may require memory for internal processing or optimization which also belongs to non-heap memory. For example, the Just-In-Time (JIT) compiler requires memory for storing the native machine code translated from the Java VM code for high performance.

Permanent Generation (non-heap): The pool containing all the reflective data of the virtual machine itself, such as class and method objects. With Java VMs that use class data sharing, this generation is divided into read-only and read-write areas. The heap itself only contains class instances, but before the JVM can create an instance of a class on the heap, it must load the class bytecode (.class file) into the process memory. It can then use that class bytecode to create an instance of the object in the heap. The space in the process memory that the JVM uses to store the bytecode versions of classes is the permanent space. In general, you want the permanent space to be large enough to hold all classes in your application, because reading classes from the file system is obviously more expensive than reading them from memory. To help you ensure that classes are not unloaded from the permanent space, the JVM has a tuning option:

–noclassgc

This option tells the JVM not to perform garbage collection on (and unload) the class files in the permanent space. This tuning option is very intelligent, but it raises a question: what does the JVM do if the permanent space is full when it needs to load a new class? In my observation, the JVM examines the permanent space and sees that it needs memory, so it triggers a major garbage collection. The garbage collection cleans up the heap, but cannot touch the permanent space, so its efforts are fruitless. The JVM then looks at the permanent space again, sees that it is full, and repeats the process again, and again, and again.

So the Java classes are stored in the permanent generation. What all does that entail? Besides the basic fields of a Java class there are:

Methods of a class (including the bytecodes)
Names of the classes (in the form of an object that points to a string also in the permanent generation)
Constant pool information (data read from the class file, see chapter 4 of the JVM specification for all the details).
Object arrays and type arrays associated with a class (e.g., an object array containing references to methods).
Internal objects created by the JVM (java/lang/Object or java/lang/exception for instance)
Information used for optimisation by the compilers (JITs)

Code Cache (non-heap): The HotSpot Java VM also includes a code cache, containing memory that is used for compilation and storage of native code.

Available Garbage Collectors in Heap

The Java HotSpot VM includes three types of collectors and each with different performance characteristics. These collectors are responsible what type of collection of unreachable/dead objects will have in the VM. One can even force during the VM start-up which collector to use else the collector is selected by default based on machine's hardware and OS configurations. There are two primary measure of garbage collection performance i.e. applications and Garbage Collection can be classified as Throughput and Pauses.

Throughput is the percentage of total time not spent in garbage collection. Pauses are the times when an application becomes unresponsive because garbage collections is taking place.

Throughput bound applications are the ones which will be needing to perform as many transactions as possible. "Application Throughput" denotes the speed at which a Java application runs. If your application is a transaction based system, high throughput means that more transactions are executed during a given amount of time. You can also measure the throughput by measuring how long it takes to perform a specific task or calculation. For most of web applications, the right measurement is considered to be Throughput since Pauses during GC generally get unnoticed. But not for all, e.g., in an online picture editing too even short pauses may impact to user behaviour heavily.

Latency (Pauses) bound applications will be needing to complete each transaction in a specified amount of time. Low Pause collectors are needed for Latency bound applications. In latency (Low pause) applications, GC threads will be running alongside with the application and hence throughput will be low.

Serial Collector
The serial collector uses single thread to accomplish all garbage collection which means there is no overhead of thread communication or need to have synchronization.-XX:+UseSerialGC is the command line option.
Parallel Collector (Throughput Collector)
The parallel collector is similar to serial collector with the difference that multiple threads are used to spped-up the garbage collection time. By default, only Minor Collections happens with multiple threads; major collections are still performed with a single thread.
Use -XX:+UseParallelGC. There is a feature introduced in Java 5 called "parallel compaction" through which even Major Collections can happen with multiple threads as well. Parallel compaction is enabled by adding the option -XX:+UseParallelOldGC to the command line. It is automatically chosen as the default garbage collector on server-class machines.
Concurrent Collector (Low Pause)
This collector is best suited for applications where objects are long lived i.e. larger Tenured Generation and run on machines with two or more processors. Also, this collector should be considered for any application with a low pause time requirement. -XX:+UseConcMarkSweepGC. If you want it to be run in incremental mode(low pause times) , also enable that mode via the –XX:+CMSIncrementalMode.

OutOfMemoryError in Java Heap

When JVM starts JVM heap space is equal to the initial size of Heap specified by -Xms parameter, as application progress more objects get created and heap space is expanded to accommodate new objects. JVM also run garbage collector periodically to reclaim memory back from dead objects. JVM expands Heap in Java some where near to Maximum Heap Size specified by -Xmx and if there is no more memory left for creating new object in java heap , JVM throws java.lang.OutOfMemoryError and your application dies.

Out of Permgen space

Since in most of JVM default size of Perm Space is around "64MB" one can easily run out of memory if you have too many classes or huge number of Strings in your project.
Important point to remember is that it doesn't depends on –Xmx value so no matter how big your total heap size you can run OutOfMemory in perm space. Good think is you can specify size of permanent generation using JVM options "-XX:PermSize" and "-XX:MaxPermSize" based on your project need. When sizing the permanent space, consider using 128 MB, unless your applications have a large number of classes, in which case, you can consider using 256 MB. If you have to configure the permanent space to use anything more, then you are only masking the symptoms of a significant architectural issue
Another reason of "java.lang.OutOfMemoryError: PermGen" is memory leak through Classloaders. In Application server different classloaders are used to load different web application so that you can deploy and undeploy one application without affecting other application on same server, but while undeploying if container some how keeps reference of any class loaded by application class loader than that class and all other related class will not be garbage collected and can quickly fill the PermGen space if you deploy and undeploy your application many times. "java.lang.OutOfMemoryError: PermGen”.

Possible Solutions to try

From tomcat > 6.0 onward tomcat provides memory leak detection feature which can detect many common memory leaks on web-app perspective e.g ThreadLocal memory leaks, JDBC driver registration, RMI targes, LogFactory and Thread spawned by web-apps. You can check complete details on http://wiki.apache.org/tomcat/MemoryLeakProtection you can also detect memory leak by accessing manager application which comes with tomcat, in case you are experiencing memory leak on any java web-app its good idea to run it on tomcat.
Increasing the "-XX:MaxPermSize"
We can keep a check on if the Strings in our applications are huge making String Constant Pool to grow at a larger extent.
The most pain-free method of resolving this issue is to switch from Sun's JDK implementation to Oracle's freely available JRockit implementation as JRockit JVM DOES NOT have any PermGen Space inside the JVM memory. (But this should be the LAST solution)

Out of swap space

Java HotSpot VM is split between 3 memory spaces (Java Heap, PermGen, C-Heap). failure to allocate native memory from the OS to the HotSpot C-Heap or dynamically expand the Java Heap

Possible Solutions to try

Possible causes are non-optimal heap space allocation - can be fixed by adjusting the correct heap size and/or increasing RAM

GC Overhead limit exceeded

Pre-cursor for java heapspace - warning that garbage collection is taking too much time, that can possibly lead to heap space error - before jvm crash

Possible Solutions to try

Can turn this off - the root cause should be fixed for java heap space to avoid this one.

The maximum theoretical heap limit for the 32-bit JVM is 4GB. Due to various additional constraints such as available swap, kernel address space usage, memory fragmentation, and VM overhead, in practice the limit can be much lower. On most modern 32-bit Windows systems the maximum heap size will range from 1.4G to 1.6G. On 32-bit Solaris kernels the address space is limited to 2G. On 64-bit operating systems running the 32-bit VM, the max heap size can be higher, approaching 4G on many Solaris systems.As of Java SE 6, the Windows /3GB boot.ini feature is not supported.If your application requires a very large heap you should use a 64-bit VM on a version of the operating system that supports 64-bit applications. See Java SE Supported System Configurations for details.