Forem: Piyush Gupta

Pi day exercise: Calculating pi with expressions!

Piyush Gupta — Thu, 14 Mar 2024 13:36:06 +0000

The Monte Carlo method is an effective technique for calculating probabilities.

In this fun blog, whose concept was "engineered" by our very own Joshua Pollack, we demonstrate how to use expressions in Aerospike to calculate the value of Pi using a Monte Carlo technique.

While the example is fun and rather contrived, it highlights an underlying technique of using expressions in Aerospike to compute incoming data values for thresholds and apply conditional executions to real-time situations such as fraud detection.

What is Pi?

Pi (often represented as the Greek letter π) is “one of the most well-known mathematical constants [and] is the ratio of a circle’s circumference to its diameter. For any circle, the distance around the edge is a little more than three times the distance across. Typing π into a calculator and pressing ENTER will yield the result 3.141592654, not because this value is exact, but because a calculator’s display is often limited to 10 digits.”

Calculating the value of Pi

The Monte Carlo method to compute the value of Pi uses the probability of a random point whose x and y coordinates are between -1.0 and 1.0 falling within the enclosing square of side 2.0 versus within the circle at center (0.0, 0.0) with radius r = 1.0.

The probability of being in the circle versus the encompassing square is π*r²/(2r)². This article describes this simple technique quite nicely.

The technique reduces to:

Estimated value of Pi = 4 * number of points inside the circle / total number of points.

We can apply the ratio to just the top quadrant of the circle and rectangle.

We can use the Java random number generator to produce x and y coordinates of a point between 0.0 and 1.0.

We will do this computation in a single Aerospike record that we initialize as follows:

AerospikeClient client = new AerospikeClient("127.0.0.1", 3000);

//Initialize our single aggregation record, center of circle 

Key key1 = new Key("test", "points", 1); 
Bin xcoord = new Bin("x", 0.0);
Bin ycoord = new Bin("y", 0.0);
Bin inCount = new Bin("inCount", 1.0);
Bin totalCount = new Bin ("count", 1.0);
WritePolicy wPolicy = new WritePolicy();
client.put(wPolicy, key1, xcoord, ycoord, inCount, totalCount );
//Check
System.out.println(client.get(null, key1));

Output:

(gen:1),(exp:448491015),(bins:(x:0.0),(y:0.0),(inCount:1.0),(count:1.0))

As we insert the next random values of the (x,y) pair, we will use expressions to compute whether the value lies within the circle or outside and in the same write transaction, update the inCount and the total count.

We use another expression-based call to get the estimated value of Pi (expected to be 3.14xx).

//Insert a record using a class based construct
class Point {

  public void insertPoint (Key key1, double xval, double yval, boolean bCheck) {
    Bin xcoord = new Bin("x", xval);
    Bin ycoord = new Bin("y", yval);
    WritePolicy wPolicy1 = new WritePolicy();

    Expression inCircleExp = Exp.build(
       Exp.cond(
        Exp.le(
          Exp.add(
            Exp.mul(Exp.floatBin("x"), Exp.floatBin("x")), 
            Exp.mul(Exp.floatBin("y"), Exp.floatBin("y"))
          ),
          Exp.val(1.0)
        ),
       Exp.add(Exp.floatBin("inCount"), Exp.val(1.0)),
       Exp.floatBin("inCount")
       ));

    Expression totalCountExp = Exp.build(Exp.add(Exp.floatBin("count"), Exp.val(1.0)));

    Record record = client.operate( wPolicy1, key1,   
          Operation.put(xcoord),
          Operation.put(ycoord),                   
          ExpOperation.write("inCount", inCircleExp, ExpWriteFlags.DEFAULT),
          ExpOperation.write("count", totalCountExp, ExpWriteFlags.DEFAULT)
                                  );                 
    if(bCheck){
      System.out.println(client.get(null, key1));
    }   
  }

  public void getPi(Key key1){
      WritePolicy wPolicy = new WritePolicy();
      Expression piExp = Exp.build(
          Exp.mul(Exp.val(4.0), 
                  Exp.div(Exp.floatBin("inCount"),Exp.floatBin("count"))
                 ));

      Record record = client.operate(wPolicy, key1, ExpOperation.read("pi", piExp, ExpReadFlags.DEFAULT));
      System.out.println("Pi : " + record.getValue("pi"));
  }

}

Let's try this code construct:

//Check
Point p = new Point();
p.insertPoint(key1, 0.71, 0.71, true);  //Out of circle, in rectangle

Output:

(gen:2),(exp:448490510),(bins:(x:0.71),(y:0.71),(inCount:1.0),(count:2.0))

Let's scale it up and insert 10,000 random points. This takes an imperceptible amount of time on Aerospike.

//Insert random points with x,y between 0.0 and 1.0

Random rnd1 = new Random(0);  

for(int i=1;i<10000;i++){
  double xval = rnd1.nextDouble();
  double yval = rnd1.nextDouble();
  p.insertPoint(key1, xval, yval, false);  
}

And now for our estimate of Pi:

p.getPi(key1);

Output:

Pi : 3.1424

Happy Pi Day!

In the above example, getPi() is shown here as a separate call. However, it can be made part of the return value in the write api (insertPoint()).

Instead of our pair of x and y, if you had 100s of such parameters that you would want to compute a net score to make a go/nogo decision, the above code constructs can be easily extended. In this example, we had no need to retain the 10,000 random values. However, if the data needs to be retained, consider using one of our collection data types, such as a Map or a List.

Get the most out of your data with Aerospike Expressions

To learn more, watch our Optimizing query performance with Aerospike Expressions webinar with Chief Developer Advocate Tim Faulkes.

How to retrieve the key in a key-value store

Piyush Gupta — Wed, 13 Mar 2024 17:54:01 +0000

Learn how to retrieve user-specified keys in Aerospike using single record reads.

In this blog, you’ll learn how to retrieve the user-specified key in Aerospike, a key-value store.

Aerospike is designed from the ground up to support extremely low latency create, read, update, and delete (CRUD) operations. In Aerospike, applications store data as individual records that are uniformly distributed across all the nodes of the database cluster. A user-specified key is associated with each record along with the application data. Below, we will walk you through the steps necessary to retrieve your user-specified key in a single record read operation.

Keys in Aerospike

As mentioned above, Aerospike stores user data in individual records, which are accessible by a user-specified "key." Keys can only be a string, integer, or a byte array.

[USER SPECIFIED KEY] → [USER DATA in one or more "bins"]

Applications access the data via the Aerospike client library, which computes a 20-byte “digest” by hashing the user-supplied key information of set name, user key value, and user key type.

The client library sends the 20-byte digest as the key of record to the server. So, the record in the server should read:

[20 Byte Digest] → [USER DATA in one or more "bins"]

Storing keys in records

What if two keys hash to the same digest value, resulting in a hash collision?

Aerospike uses the RIPEMD-160 hashing algorithm, which has a statistically near-zero probability of hash collision. By default, Aerospike does not store the user-supplied key with the saved record. However, applications have the option to store the user-specified key along with the record via the write policy.

[20 Byte Digest ] → [USER DATA in one or more "bins"] + [User Key (string / int / blob)]

The approach above ensures that the server verifies the hash against the stored key during CRUD operations. Any subsequent updates where a hash collision is detected, the application will receive a KEY_MISMATCH error.

Key mismatch explained

Consider the case in a namespace test, where a user-specified key of abc in set testSet1 generates a particular digest, digest1. When this record is first created, Aerospike uses digest1 as the internal key of that record but also stores the user-specified key of abc with the record.

Hypothetically, in the same namespace test, consider the user-specified key of xyz in set testSet2, which also happens to hash to the same value, digest1. (This has near zero probability, practically speaking.)

If the application tries to create or update a record with digest1 and with user-specified key xyz, Aerospike will access the record stored against digest1 and compare the stored key abc against the incoming transaction's user-specified key of xyz and discover that they are different. A KEY_MISMATCH exception will be thrown in this case.

User key stored with the record

The curious may want to know: Is it possible to read this user-specified stored key back in the application?

For data modeling purposes and for queries (primary or secondary index queries that retrieve qualifying records from the entire namespace), the user stored key can be retrieved back to the client. This is useful since the user may want to then update individual records selectively, as part of their use case.

However, for single key reads the user key is not returned to the client, as you must have had the key to be able to read the record in the first place. This type of read can be done by passing either the user key and the set name or the digest.

In certain operational situations, users have asked if it is possible to read back the user stored key on an individual record read. They stored their key with the record and would now like to retrieve it. These are situations where users have the 20-byte digest of a record, but they do not know the user key. The digest was recorded in the server logs. An example of this would be when turning on rw-client logging to identify the digests of "hot-keys," records that are being updated very frequently.

The read APIs don't provide this functionality upfront via the policies since this is not a typical data modeling need. However, Aerospike does provide expressions that can access the user stored key.

What are Aerospike Expressions?

An expression is a strongly typed domain-specific language designed for manipulating and comparing bins and record metadata. There is no “programming language” per se; an expression is built with code. It runs on the nodes in the cluster, not on the client.

Aerospike supports three types of expressions:

Filter expressions (introduced in version 5.2.0)
Cross Datacenter Replication filter expressions (introduced in version 5.3.0)
Operation expressions (introduced in version 5.6.0)

Filter and operation expressions can be specified on a per-call basis.

Key retrieval code example

Below is an example code snippet demonstrating how to retrieve a key using the Java client.

Creating the record initially:

AerospikeClient client = new AerospikeClient("127.0.0.1", 3000);
Key key = new Key("test", "testset", "key1");
WritePolicy wPolicy = new WritePolicy();

wPolicy.sendKey = true;  //Store our user key on the server

Bin b0 = new Bin("b00", Value.get(28));
client.put(wPolicy, key, b0);

Now, assume we just have the 20-byte digest and would like to retrieve our key:

We can use the following key constructor:

// Initialize key from namespace, digest, optional set name, and optional userKey.
Key(String namespace, byte[] digest, String setName, Value userKey)

Let's find the digest of the record we created above, for purposes of this demo.

byte[] recDigest = new byte[20];
recDigest =  Key.computeDigest("testset", Value.get("key1")) ;

Next, create a new key object using the digest-based constructor.

Key keyByDigest = new Key("test", recDigest, null, Value.NULL);

About `Value.NULL`

The constructor allows specifying the user-specified key in the last argument as a “Value" type object. For example, Value.get("myKey"). This is provided for the rare case where the application wants to do an update with the key generated using the digest, and the set name is not known but still wants to update with sendKey=true. In our case, we don't know the key and are trying to retrieve it in an operate() call, which is like a write, although we will only perform a read via expressions. Hence, we pass Value.NULL for the user-specified key and use sendKey=false.

We should be able to read the same record using either "key" or "keyDigest." Let's validate.

System.out.println("Record: "+ client.get(null, key)); 
System.out.println("Record via Digest: "+ client.get(null, keyByDigest));

Output:

Record: (gen:12),(exp:447740032),(bins:(b00:28))
Record via Digest: (gen:12),(exp:447740032),(bins:(b00:28))

Case 1: key type known

Now, using the digest-based key, let's retrieve the stored key from the server. Since we are specifying the digest and Value.NULL for the userKey in our recDigest constructor, we must set thewPolicy.sendKey = false; to avoid a KEY_MISMATCH error.

wPolicy.keySend = false;
Expression recKeyExp = Exp.build(Exp.key(Exp.Type.STRING));
Record record = client.operate( wPolicy, keyByDigest,   
          ExpOperation.read("reckey", recKeyExp, ExpReadFlags.DEFAULT) 
         );
System.out.println("Record Key: " + record.getValue("reckey"));

Recall we mentioned that user-specified keys can be of type integer, string, or byte-array. Here, although we don't know our key, we know it is a string type. This is specified using Exp.Type.STRING when building the expression for the key.

Output:

Record Key: key1

Case 2: key type unknown

What if we don't know the key type? We can check for all the three allowable types. Although, in this case, we must use ExpReadFlags.EVAL_NO_FAIL instead of ExpReadFlags.DEFAULT so that the failed expressions with type mismatch don't generate an Exception.

Example code is shown below.

wPolicy.sendKey = false;
Expression recKeyExpStr = Exp.build(Exp.key(Exp.Type.STRING));
Expression recKeyExpInt = Exp.build(Exp.key(Exp.Type.INT));
Expression recKeyExpBlob = Exp.build(Exp.key(Exp.Type.BLOB));

Record record = client.operate( wPolicy, keyByDigest,   
          ExpOperation.read("reckeyStr", recKeyExpStr, ExpReadFlags.EVAL_NO_FAIL),
          ExpOperation.read("reckeyInt", recKeyExpInt, ExpReadFlags.EVAL_NO_FAIL),
          ExpOperation.read("reckeyBlob", recKeyExpBlob, ExpReadFlags.EVAL_NO_FAIL)  
         );
System.out.println("Record Key String: " + record.getValue("reckeyStr"));
System.out.println("Record Key Integer: " + record.getValue("reckeyInt"));
System.out.println("Record Key BLOB: " + record.getValue("reckeyBlob"));

Output:

Record Key String: key1
Record Key Integer: null
Record Key BLOB: null

Unlock the power of expressions in Aerospike for enhanced data operations

Expressions are a very powerful tool in Aerospike. They can be used as filters for selecting records to return in primary or secondary index queries, as well as compute on a record’s existing data or metadata. It can then return the result of the computation as a read or write it to a new bin. The possibilities are only limited by your creativity!

To learn more, watch our Optimizing query performance with Aerospike Expressions webinar with Chief Developer Advocate Tim Faulkes.

Forem: Piyush Gupta

Pi day exercise: Calculating pi with expressions!

What is Pi?

Calculating the value of Pi

Get the most out of your data with Aerospike Expressions

How to retrieve the key in a key-value store

Keys in Aerospike

Storing keys in records

Key mismatch explained

User key stored with the record

What are Aerospike Expressions?

Key retrieval code example

About Value.NULL

Case 1: key type known

Case 2: key type unknown

Unlock the power of expressions in Aerospike for enhanced data operations

About `Value.NULL`