How to: Get up to speed and scale with Aerospike Graph on Google Cloud Marketplace

Ishaan Biswas — Mon, 28 Aug 2023 15:00:31 +0000

In June of this year, Aerospike put graph developers front and center with the announcement of a new, developer-ready, real-time scalable graph database. Aerospike Graph is a massively scalable, high-performance graph database that simplifies building and deploying applications for highly connected enterprise-scale datasets.

Aerospike Graph combines the unlimited scale, high throughput, and low latency of the Aerospike Database with the Aerospike Graph Service, an independently scalable graph compute engine built on Apache Tinkerpop. Let’s take a closer look.

Figure 1: Aerospike Graph on Google Cloud enables developers to use the popular Gremlin query language while keeping graph compute and storage separately scalable to optimize costs.

Scalable storage

Scalable storage is facilitated by the Aerospike Database which functions as the storage layer. Aerospike is a shared-nothing, multi-threaded, multimodel database designed to operate efficiently on a cluster of server nodes. Utilizing modern hardware and network architectures, Aerospike delivers exceptional high performance, through petabytes of data, using its patented Hybrid Memory Architecture ^TM (HMA); storing data on Flash devices (such as SSDs) and indexes in DRAM.

Scalable compute

Aerospike Graph Service (AGS) is the compute layer of Aerospike Graph. AGS is a deep integration of Apache TinkerPop and the Aerospike Database, offering a graph-native developer interface through the Gremlin query language.
AGS is purpose-built to maximize query performance through dozens of custom Gremlin steps, compiler optimization strategies, and a highly optimized graph encoding for the Aerospike Database. AGS nodes are stateless compute instances and can be elastically scaled to meet your application's throughput needs.

Unique, efficient architecture

Aerospike Graph’s unique architecture features independent scaling of compute and storage resources. This enables optimal resource utilization, consistently low-latency queries, and extreme throughput for complex multi-hop graph queries at scale. This translates to an industry-leading total cost of ownership, ensuring cost-effective operations without compromising performance.

This focus on performance, extreme scale, and high-efficiency operation provides a new proposition for companies looking to move demand operational workloads to a graph database. Making that available on an elastically scalable cloud environment like Google Cloud provides a future-proof path for IT and development organizations to follow. Let’s see what it takes to get started.

Getting Started with Aerospike Graph on Google Cloud

To Get Started with Aerospike Graph on Google Cloud, follow these steps:

Locate Aerospike Database (BYOL) in the Cloud Marketplace. (Note: the other Aerospike Database listed is for Private Offers.)
- In the Cloud Console navigation menu, click Marketplace.
- Type Aerospike in the Search Marketplace bar and click on the highlighted option from the search results.
Click on Launch.
Follow the on-screen guide to configure and deploy your Aerospike Cluster.
Once the VMs are provisioned, note the Internal IP of the Aerospike Database cluster nodes. You can do this from the Google Cloud UI by navigating to the VM instances section of Compute Engine in the navigation menu. (You will need this information later in Step 9).
Create a Compute Optimized Compute Engine instance to host the Aerospike Graph Service.
Now, Locate Aerospike Graph Service in the Cloud Marketplace through the Search bar.
Click on Show Pull Command. This will provide the commands to pull the docker image from the Google Container Registry.
SSH into the GCE instance for Aerospike Graph Service and run the command in Step 7. This will pull the docker image to your instance.
Click on the Get Started with Aerospike Graph Service button. This will provide you with instructions on how to connect AGS to the Aerospike Cluster.
Follow along the technical documentation in Step 9 for details on data loading, running gremlin queries and building your graph applications.

Working with graph data

There are several options for connecting to and interacting with Aerospike Graph. Some possibilities are:

The Gremlin Console, an interactive command-line terminal for sending queries and receiving responses.
A client application. This page provides code samples for Python and Java client code.
A Jupyter Notebook.

You can use a client application or the Gremlin Console to query an existing data set, or add new edges and vertices to a data set. You can view examples in the product documentation that demonstrate adding new vertices one at a time to a database. To bulk load new data into a Graph database, use the Graph bulk loader.

Kelvin Lawrence has compiled and made public a data set containing information about airlines, airports around the world, and routes between them, designed for use with a graph database. The data set is large enough to be interesting and useful, but small enough to be practical for testing and experimentation purposes. To run the following examples, download one of the .graphml data files.

To use the .graphml file, you must bind it to a Docker volume. When you start the AGS Docker image, use the -v option to bind the local .graphml file directory to a directory in the Docker container. For example, if you download the data file air-routes-small.graphml to the directory /home/users/data, start the AGS Docker image with the -v option:

docker run -p 8182:8182 -v /home/user/data/:/opt/air-routes/ aerospike/aerospike-graph-service

You can use the Gremlin console to load the air-routes data set into Aerospike Graph with the following command:

g.with("evaluationTimeout", 24L * 60L * 60L * 1000L).io("/opt/air-routes/air-routes-small.graphml").with(IO.reader, IO.graphml).read()

View the Aerospike Graph product documentation for more information.

Are Graph Databases Finally Ready for Prime Time?

Ishaan Biswas — Thu, 23 Jun 2022 15:32:33 +0000

Graph databases need to perform at a planetary scale

You can quickly become enamored with graph databases. When you discover that a short graph query can express what took you 6 weeks and 2 ounces of midnight oil in SQL - it is a life changing experience. When you use those queries to gain new insights about your customers through graph analytics, it is transformational for your business.
So what happens when the technology you’ve so fallen in love with and become so reliant on, grinds to a halt? Imagine what that could do to your business.

2018 was widely hailed in the industry to be the year of graph databases. The sheer breadth of use cases for graph databases was (and still is) endless and it seemed like graphs would be ubiquitous in enterprise data pipelines. Four years later, have graph databases fulfilled that promise? Are graph databases truly a ubiquitous element in enterprise tech stacks today?
If you’ve explored any of the popular graph databases out there, you’ve probably reached the same conclusion that we have - a resounding NO!

The simplicity and intuitiveness of the graph data model makes it a dream for data scientists and developers alike. So why haven’t graphs revolutionized the database industry yet?

It turns out that an inability to provide predictably high performance at scale is the nemesis of graph databases. This is especially true in the enterprise world, where scale is not a buzzword, but really matters. When you’re serving millions of customers across the globe, 24x7 - predictable performance at scale is crucial for the sustained growth of your application, and your business. There’s only limited value you can extract from a graph database that only works well on small datasets. The true value of a graph database lies in the ability to harness this technology at massive scales. When you can gain insights from deep and intricate relationships across your entire dataset, without sacrificing performance, it can give you the legs to leapfrog your competition.

It's easy to be confused by marketing jargon when it comes to scale. By 'scale' we mean graphs that are terabytes to petabytes in size, spanning tens of billions of vertices and hundreds of billions of edges. In simpler words, a graph large enough that it needs a distributed cluster to store it, and cannot fit in-memory on a single machine. Most of the popular graph database vendors today have adopted read-replica architectures, which are simply not engineered to handle large volumes of data. Performance craters as soon as your dataset grows beyond the memory capacity of a single instance, leaving the database practically unusable for your application.

At Aerospike, our core database technology solves these problems and enables our customers to harness the power of the graph data model. Aerospike has been built as a globally distributed database from the ground up. Using a Hybrid Memory Architecture™, Aerospike Database provides predictable low latency, for strong consistency and high availability modes. Our technology is proven to perform sub-millisecond queries at Petabyte scale while reducing the server footprint at the same time. Our customers have been clamoring for us to combine the well known benefits of Aerospike as a datastore with the elegance of the graph data model.

Over the last few months, we’ve been working hard to bring this project to fruition, and today we’re excited to announce the Firefly Alpha Program (our code name for Aerospike’s graph database offering). Firefly combines Aerospike Database - the most scalable realtime NoSQL database with Apache TinkerPop™ - the most widely adopted graph computing framework. Through the alpha program, we will provide early access to Firefly prototypes, provide a channel for you to interact with the team and help shape the product roadmap. We want to hear your feedback, learn about your use cases and challenges with graph databases and together, build the most scalable realtime graph database on the planet.
Be a part of the journey. Let's fearlessly scale graph technology together.

If you’d like to sign up reach out to us at firefly-alpha@aerospike.com. Limited spots available.

Forem: Ishaan Biswas