Forem: Memgraph

In-memory vs. disk-based databases: Why do you need a larger than memory architecture?

Memgraph — Tue, 05 Sep 2023 14:41:54 +0000

Memgraph is an in-memory graph database that recently added support for working with data that cannot fit into memory. This allows users with smaller budgets to still load large graphs to Memgraph without paying for (more) expensive RAM. However, expanding the main-memory graph database to support disk storage is, by all means, a complex engineering endeavor. Let’s break this process down into pieces.

On-disk databases

Disk-based databases have been, for a long time, a de facto standard in the database development world. Their huge advantage lies in their ability to store a vast amount of data relatively cheaply on disk. However, the development can be very complex due to the interaction with low-level OS primitives. Fetching data from disk is something that everyone strives to avoid since it takes approximately 10x more time than using it from main memory. Neo4j is an example of a graph, an on-disk database that uses disk as its main storage media while trying to cache as much data as possible to main memory so it could be reused afterward.

In-memory databases

In-memory databases avoid the fundamental cost of accessing data from disk by simply storing all its data in the main memory. Such architecture also significantly simplifies the development of the storage part of the database since there is no need for a buffer pool. However, the biggest issue with in-memory databases is when the data cannot fit into the random access memory since the only possible way out is to transfer the data to a larger and, consequently, more expensive machine.

In-memory database users rely on the fact that durability is still secured through durability mechanisms like transaction logging and snapshots so that data loss does not occur.

Larger-than-memory architecture

Main memory computation

Larger-than-memory architecture describes a database architecture when the majority of computations are still within the main memory, but the database offers the ability to store a vast amount of data on disk, too, without having the computational complexity of interacting with buffer pools.

Identify hot & cold data

The larger-than-memory architecture utilizes the fact that there are always hot and cold parts of the database in terms of accessing it. The goal is then to find cold data stored and move it to the disk so that transactions still have fast access to hot data. Cold data identification can be done either by directly tracking transactions’ access patterns (online) or by offline computation in which a background thread analyzes data.

The second very important feature of the larger-than-memory architecture is the process of evicting cold data. This can be done in two ways:

DB tracks the memory usage and starts evicting data as soon as it reaches a predefined threshold.
Eviction can be done only when new data is needed.

Transaction management

Different systems also behave differently regarding transaction management. If the transaction needs data that is currently stored on the disk, it can:

Abort the transaction, fetch data stored on the disk, and restart the transaction.
Stall the transaction by synchronously fetching data from the disk.

Transaction must fit into memory

The question is, what happens when the transaction data cannot fit into random access memory? In Memgraph, we decided to start with an approach that all transaction data must fit into memory. This means that some analytical queries cannot be executed on a large dataset, but this is the tradeoff we were willing to accept in the first iteration.

Benefits of larger-than-memory databases

Memgraph uses RocksDB as a key-value store for extending the capabilities of the in-memory database. Not to go into too many details about RocksDB, but let’s just briefly mention that it is based on a data structure called Log-Structured Merge-Tree (LSMT) (instead of B-Trees, typically the default option in databases), which are saved on disk and because of the design come with a much smaller write amplification than B-Trees.

The in-memory version of Memgraph uses Delta storage to support multi-version concurrency control (MVCC). However, for larger-than-memory storage, we decided to use the Optimistic Concurrency Control Protocol (OCC) since we assumed conflicts would rarely happen, and we could make use of RocksDB’s transactions without dealing with the custom layer of complexity like in the case of Delta storage.

We’ve implemented OCC in a way that every transaction has its own private workspace, so potential conflicts are detected at the commit time. One of our primary requirements before starting to add disk-based data storage was not to ruin the performance of the main memory-based storage. Although we all knew there was no such thing as zero-cost abstraction, we managed to stay within 10% of the original version. We decided to use snapshot isolation as an appropriate concurrency isolation level since we believed it could be the default option for the large majority of Memgraph users.

Disadvantages of larger-than-memory databases

As always, not everything is sunshine and flowers, especially when introducing such a significant feature to an existing database, so there are still improvements to be made. First, the requirement that a single transaction must fit into memory makes it impossible to use large analytical queries.

It also makes our LOAD CSV command for importing CSV files practically unusable since the command is executed as a single transaction. Although RocksDB is really good, fits really well into our codebase, and has proved to be very efficient in its caching mechanisms, maintaining an external library is always hard.

In retrospect

Albeit the significant engineering endeavor, the larger-than-memory architecture is a super valuable asset to Memgraph users since it allows them to store large amounts of data cheaply on disk without sacrificing the performance of in-memory computation. We are actively working on resolving issues introduced with the new storage mode, so feel free to ask, open an issue, or pull a request. We will be more than happy to help. Until next time 🫡

Exciting News: LangChain Now Supports Memgraph!

Memgraph — Fri, 25 Aug 2023 07:06:32 +0000

We're thrilled to announce a powerful integration between LangChain and Memgraph, bringing you an unparalleled natural language interface to your Memgraph database. Say goodbye to complex queries and welcome a seamless and intuitive way to interact with your data.

Memgraph QA chain tutorial

If you've ever wanted to effortlessly query your Memgraph database using natural language, this tutorial is for you. This step-by-step guide will walk you through the process, ensuring you have all the tools you need to get started.

Prerequisites

Before you dive in, make sure you have Docker and Python 3.x installed on your system.

Get started

Launch a Memgraph Instance: With a few simple commands, you can have your Memgraph instance up and running using Docker. Just follow our script to set it up.

Install dependencies: We've got you covered with the required packages. Use pip to install langchain, openai, neo4j, and gqlalchemy. Don't forget the --user flag to ensure smooth permissions.

Code playtime: Whether you prefer working within this notebook or want to use a separate Python file, the tutorial offers code snippets to guide you through the process.

What's inside

Explore the rich features and functionalities that LangChain and Memgraph offer together:

API reference: We provide an overview of the key components you'll be working with, such as ChatOpenAI, GraphCypherQAChain, and MemgraphGraph.

Populating the database: Learn how to populate your Memgraph database effortlessly using the Cypher query language. We guide you through the process of seeding data that serves as the foundation for your work.

Refresh graph schema: Familiarize yourself with refreshing the graph schema, a crucial step in setting up the Memgraph-LangChain graph for Cypher queries.

Querying the database: Discover how to interact with the OpenAI API and configure your API key. We'll show you how to utilize the GraphCypherQAChain to ask questions and receive informative responses.

Chain modifiers: Customize your chain's behavior with modifiers like return_direct, return_intermediate_steps, and top_k. Tailor the experience to your preferences.

Advanced querying: Delve into advanced querying techniques and uncover tips for refining your prompts to improve query accuracy.

Ready to take your data interaction to the next level? Join us in exploring the seamless synergy between LangChain and Memgraph. No more wrangling with queries – just natural language and meaningful insights. Simplify complexity, elevate your insights, and share your projects in our community.

How to Build a Graph Web Application With Python, Flask, Docker & Memgraph

Memgraph — Thu, 20 Jul 2023 11:24:36 +0000

The goal is straightforward (or at least it seems simple enough). Let's build a
web application in Python that can visualize a graph and run some cool graph
algorithms out of the box. Maybe it's not your flavor, but I prefer the
Flask web framework for such occasions, so bear with me through this
tutorial.

Now, I am going to show you an example of how to accomplish this. You can also
take a look at the finished
app
on GitHub if you want to see the complete code.

The general outline of the tutorial is:

Create a Flask server
Dockerize your application
Import the data into Memgraph
Query the database

Graph visualizations will be covered in part two of the tutorial so stay
tuned! Spoiler alert, we are going to use D3.js to draw our graph.

Prerequisites

For this tutorial, you will need to install:

Docker
Docker Compose (which is included with Docker on Windows and macOS)

With Docker, we don't need to worry about installing Python, Flask, Memgraph...
essentially anything. Everything will be installed automatically and run
smoothly inside Docker containers!

Disclaimer: Docker fanboy alert

1. Create a Flask server

I included comments in the code to make it more understandable, but if at any
point you feel like something is unclear, join our Discord
Server and share your thoughts. First, create the
file app.py with the following contents:

import json
import logging
import os
from argparse import ArgumentParser
from flask import Flask, Response, render_template
from gqlalchemy import Memgraph

log = logging.getLogger(__name__)

def init_log():
    logging.basicConfig(level=logging.DEBUG)
    log.info("Logging enabled")
    # Set the log level for werkzeug to WARNING because it will print out too much info otherwise
    logging.getLogger("werkzeug").setLevel(logging.WARNING)

Other than the imports, the first few lines focus on setting up the logging. No
web application is complete without logging, so we will add the bare minimum and
disable the pesky werkzeug logger, which sometimes prints too much info.

Now, let's create an argument parser. This will enable you to easily change the
behavior of the app on startup using arguments.

# Parse the input arguments for the app
def parse_args():
    """
    Parse command line arguments.
    """
    parser = ArgumentParser(description=__doc__)
    parser.add_argument("--host", default="0.0.0.0", help="Host address.")
    parser.add_argument("--port", default=5000, type=int, help="App port.")
    parser.add_argument("--template-folder", default="public/template", help="Flask templates.")
    parser.add_argument("--static-folder", default="public", help="Flask static files.")
    parser.add_argument("--path-to-input-file", default="graph.cypherl", help="Graph input file.")
    parser.add_argument("--debug", default=True, action="store_true", help="Web server in debug mode.")
    print(__doc__)
    return parser.parse_args()

args = parse_args()

It’s time to create your server instance:

# Create the Flask server instance
app = Flask(
    __name__,
    template_folder=args.template_folder,
    static_folder=args.static_folder,
    static_url_path="",
)

You can finally create the view functions that will be invoked from the browser
via HTTP requests. In layman's terms, the homepage is called by:

# Retrieve the home page for the app
@app.route("/", methods=["GET"])
def index():
    return render_template("index.html")

The only thing that’s left is to implement and call the main() function:

# Entrypoint for the app that will be executed first
def main():
    # Code that should only be run once
    if os.environ.get("WERKZEUG_RUN_MAIN") == "true":
        init_log()
    app.run(host=args.host,
            port=args.port,
            debug=args.debug)

if __name__ == "__main__":
    main()

The somewhat strange statement os.environ.get("WERKZEUG_RUN_MAIN") == "true"

will make sure that this code is only executed once. Confused? A problem arises
when working with Flask in development mode because each code change triggers a
reload of the server, which in turn could result in parts of your code executing
multiple times (for example, like the main function).

So, if you need to execute something only once in Flask at the beginning like
loading data, this is the perfect place for it.

The next step is to create the following files, which we will work on in the
next tutorial:

index.html in public/template
index.js in public/js
style.css in public/css

One more file is needed and this one will specify all the Python dependencies
that need to be installed. Create requirements.txt with the following
contents:

gqlalchemy==1.0.6
Flask==2.0.2

Your current project structure should look like this:

app
├── public
│  ├── css
│  │   └── style.css
│  ├── js
│  │   └── index.js
│  └── templates
│      └── index.html
├── app.py
└── requirements.txt

2. Dockerize your application

This is much simpler than you might think. Most often, you will need a
Dockerfile in which you will specify how your Docker image should be created.
Let's take a look at our Dockerfile :

FROM python:3.9

# Install CMake
RUN apt-get update && \
  apt-get --yes install cmake && \
  rm -rf /var/lib/apt/lists/*

# Install Python packages
COPY requirements.txt ./
RUN pip3 install -r requirements.txt

# Copy the source code
COPY public /app/public
COPY app.py /app/app.py
WORKDIR /app

# Set the environment variables
ENV FLASK_ENV=development
ENV LC_ALL=C.UTF-8
ENV LANG=C.UTF-8

# Start the web application
ENTRYPOINT ["python3", "app.py"]

The first line indicates that we are basing our image on a Linux image that has
Python 3.9 preinstalled. The next step is to install CMake (which is needed for
the Memgraph Python driver) with RUN and the standard Linux installation
command apt-get ... .

We copy the requirements.txt file and install the Python packages with
pip. The source code also needs to be copied to the image in order for us to
start the web application. The ENTRYPOINT command is responsible for starting
the desired process inside the container.

But we are not finished with Docker yet. We need to create a
docker-compose.yml file that will tell Docker which containers to start.

version: "3"
services:
  server:
    build: .
    volumes:
      - .:/app
    ports:
      - "5000:5000"
    environment:
      MEMGRAPH_HOST: memgraph
      MEMGRAPH_PORT: "7687"
    depends_on:
      - memgraph

  memgraph:
    image: "memgraph/memgraph"
    ports:
      - "7687:7687"

There are two services/containers in our app:

Server: Uses the Dockerfile to build a Docker image and run it.
Memgraph: This is our database. Docker will automatically download the image and start it.

Because we are supplying environment variables, let's load them in app.py
right after the imports:

MEMGRAPH_HOST = os.getenv("MEMGRAPH_HOST", "memgraph")
MEMGRAPH_PORT = int(os.getenv("MEMGRAPH_PORT", "7687"))

Your current project structure should look like this:

app
├── public
│  ├── css
│  │   └── style.css
│  ├── js
│  │   └── index.js
│  └── templates
│      └── index.html
├── app.py
├── docker-compose.yml
├── Dockerfile
└── requirements.txt

Now, we can even start our app with the following commands:

docker-compose build
docker-compose up

3. Import the data into Memgraph

This task will be done inside the main() function because it only needs to be
executed once:

memgraph = None

def main():
    if os.environ.get("WERKZEUG_RUN_MAIN") == "true":
        init_log()
        global memgraph
        memgraph = Memgraph(MEMGRAPH_HOST,
                            MEMGRAPH_PORT)
        load_data(args.path_to_input_file)
    app.run(host=args.host,
            port=args.port,
            debug=args.debug)

How do we import the data into Memgraph? I prepared a file with the Cypher
queries that need to be executed in order to populate the database. You just
need to download the file in your root directory and add the following
load_data() function:

def load_data(path_to_input_file):
    """Load data into the database."""
    try:
        memgraph.drop_database()
        with open(path_to_input_file, "r") as file:
            for line in file:
                memgraph.execute(line)
    except Exception as e:
        log.info(f"Data loading error: {e}")

First, we clear everything in the database, and then we go over each line in the
file graph.cypherl and execute them. And that's it. Once we start the web
application, Memgraph will import the dataset.

4. Query the database

We will create a function that will execute a Cypher query and return the
results. It returns the whole graph, but we will limit ourselves to 100 nodes:

def get_graph():
    results = memgraph.execute_and_fetch(
        f"""MATCH (n)-[]-(m)
                RETURN n as from, m AS to
                LIMIT 100;"""
    )
    return list(results)

The view function get_data() which fetches all the nodes and relationships
from the database, filters out the most important information, and returns it in
JSON format for visualization. To can the network load at a minimum, you will
send a list with every node id (no other information about the nodes) and a list
that specifies how they are connected to each other.

@app.route("/get-graph", methods=["GET"])
def get_data():
    """Load everything from the database."""
    try:
        results = get_graph()

        # Sets for quickly checking if we have already added the node or edge
        # We don't want to send duplicates to the frontend
        nodes_set = set()
        links_set = set()
        for result in results:
            source_id = result["from"].properties['name']
            target_id = result["to"].properties['name']

            nodes_set.add(source_id)
            nodes_set.add(target_id)

            if ((source_id, target_id) not in links_set and
                    (target_id, source_id,) not in links_set):
                links_set.add((source_id, target_id))

        nodes = [
            {"id": node_id}
            for node_id in nodes_set
        ]
        links = [{"source": n_id, "target": m_id} for (n_id, m_id) in links_set]

        response = {"nodes": nodes, "links": links}
        return Response(json.dumps(response), status=200, mimetype="application/json")
    except Exception as e:
        log.info(f"Data loading error: {e}")
        return ("", 500)

What’s next?

As you can see, it’s very easy to connect to Memgraph and run graph algorithms,
even from a web application. While this part of the tutorial focused on the
backend, in the next one, we will talk about graph visualizations and the D3.js
framework.

How to Visualize a Social Network in Python with a Graph Database: Flask + Docker + D3.js

Memgraph — Wed, 19 Jul 2023 13:15:58 +0000

When you think about a web application, a graph database doesn’t usually spring to mind. Instead, most people just take the familiar route of using an SQL database to store information. While this is perfectly acceptable for most use cases there are sometimes those that would see tremendous benefits by using a graph database.
In this tutorial, I will show you how to make a basic web application using Flask that stores all of its information in a graph database. To be more precise we are using Memgraph DB, an in-memory database that can easily handle a lot of information and perform read/write instructions quite quickly.

Our use case is a Social Network Graph (in the code referred to as SNG for convenience) representing users and the connections between them. Usually, such a graph would contain millions of relationships and the algorithms that are performed on them don’t do well with data being stored in relational databases.

In this tutorial, I will show you step by step how to build a simple Python web application from the bottom up so you get a basic understanding of the technologies that are used. You can also find all of the code here if you don't want to work on it as you go through the tutorial. If at any point in this tutorial you have a question or something is not working for you, feel free to post on StackOverflow with the tag memgraphdb.

Prerequisites

Because we are building a complete web application there is a number of tools that you will need to install before we begin:

Poetry: a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you.
Flask: a very powerful web framework that provides you with tools, libraries and technologies used in web development. A Flask application can be as small as a single web page or as complex as a management interface.
Docker and Compose: an open platform for developing, shipping, and running applications. It enables us to separate our application from our infrastructure (host machine). If you are installing Docker on Windows, Compose will be already included. For Linux and macOS visit this site.
Memgraph DB: a native fully distributed in-memory graph database built to handle real-time use-cases at enterprise scale. Follow the Docker Installation instructions on the Quick Start page. While it's completely optional, I encourage you to also install Memgraph Lab so you can execute openCypher queries on the database directly and see visualized results.

Creating the Project Structure and Handling Dependencies

Sometimes standard packaging systems and dependency management in Python can be confusing for beginners so we decided to use Poetry.

To start building our project structure choose a working directory and run:

poetry new sng-demo

Now you should have a directory with the following content:

sng-demo
├── pyproject.toml
├── README.rst
├── sng_demo
│  └── __init__.py
└── tests
   ├── __init__.py
   └── test_poetry_demo.py

In this tutorial, we won’t use the testing functionalities so go on ahead and delete the directory tests as well as the file README.rst.

Now we need to add the dependencies for our project. Given that we are going to run the app inside a Docker container we don't need the dependencies installed locally, only inside the container. Copy the files project.toml and poetry.lock and place them in the root directory of the project. The only other thing we need to do about dependency management is to tell Docker how to run Poetry on startup so it can install/update all the necessary dependencies inside the container.

Dockerizing an Application

In the root directory of the project create two files, Dockerfile and docker-compose.yml. At the beginning of the Dockerfile, we specify the Python version and instruct the container to install CMake, poetry, mgclient and pymgclient. Poetry is necessary to manage our dependencies inside the container while CMake and mgclient are required for pymgclient, the Python driver for Memgraph DB.

You don’t have to focus too much on this part just copy the code to your Dockerfile:

FROM python:3.7

#Install CMake
RUN apt-get update && \
  apt-get --yes install cmake

#Install poetry
RUN pip install -U pip \
  && curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python
ENV PATH="${PATH}:/root/.poetry/bin"

#Install mgclient
RUN apt-get install -y git cmake make gcc g++ libssl-dev && \
  git clone https://github.com/memgraph/mgclient.git /mgclient && \
  cd mgclient && \
  git checkout 5ae69ea4774e9b525a2be0c9fc25fb83490f13bb && \
  mkdir build && \
  cd build && \
  cmake .. && \
  make && \
  make install

#Install pymgclient
RUN git clone https://github.com/memgraph/pymgclient /pymgclient && \
  cd pymgclient && \
  python3 setup.py build && \
  python3 setup.py install

Next, we define the working directory with:

WORKDIR /app
COPY poetry.lock pyproject.toml /app/

The second command will enable us to cache the project requirements and only reinstall them when pyproject.toml or poetry.lock are changed.

RUN poetry config virtualenvs.create false && \
  poetry install --no-interaction --no-ansi

We don’t need to create a virtual environment because our application is already isolated by being in a Docker container. To disable it virtualenvs.create needs to be set to false.

The second line in the command ensures that Poetry asks us no interactive questions while installing/updating dependencies and it makes the output more log friendly.

COPY . /app
EXPOSE 5000
ENTRYPOINT [ "poetry", "run" ]

This is where we essentially create all the directories and files inside of our container. The EXPOSE command informs Docker that the container listens on the specified network port at runtime.

Next, we need to create a docker-compose.yml file. Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration. For our project, we need two services. One is the web application (sng_demo) and the other a database instance (memgraph).

If you followed the instructions on how to setup Memgraph DB with Docker correctly you only need to add the following code to your docker-compose.yml file to run the container:

version: '3'
services:
  memgraph:
    image: "memgraph"
    ports:
      - "7687:7687"
  sng_demo:
    build: .
    volumes:
      - .:/app
    ports:
      - "5000:5000"
    environment:
      MG_HOST: memgraph
      MG_PORT: 7687
    depends_on:
      - memgraph

When it comes to the ports key, there is an important distinction between the HOST_PORT and the CONTAINER_PORT. The first number in the key is the HOST_PORT and it can be used to connect from your host machine to the service (for example with Memgraph Lab). The second number specifies the CONTAINER_PORT which is used for service-to-service communication. More precisely, our service sng_db can use this port to access the service memgraph and connect to the database.

The environment key contains MG_HOST and MG_PORT which represent environment variables in the service’s container. They store the memgraph service address and port which are needed to establish a database connection.
The depends_on key is used to start services in dependency order because we need the database to start before the web application.

The build key allows us to tell Compose where to find the build instructions as well as the files and/or folders used during the build process. By using the volumes key, we bypass the need to constantly restart our image to load new changes to it from the host machine.

Finally, we have a dockerized project that utilizes Poetry! This approach is great for development because it enables us to run our project on completely different operating systems and environments without having to worry about compatibility issues.

Web Development with Flask

Flask is very simple to use so why not create a Hello World! page to try out our Docker+Poetry setup.

In the project root directory create a file called app.py with the following code:

from flask import Flask

app = Flask(__name__)


@app.route('/')
@app.route('/index')
def index():
   return "Hello World"

First, we imported the Flask class and then created an instance of it. The route() decorator tells Flask what URL should trigger our function.
Now, we need to tell Docker how to run our app. This can be done by creating a simple script in the project root directory. Let’s call it start.sh:

#!/bin/bash
export FLASK_APP=app.py
export FLASK_ENV=development
flask run --host 0.0.0.0

Setting FLASK_ENV to development will enable the debug mode. This makes Flask use an interactive debugger and reloader.

Setting FLASK_APP to app.py specifies how to start the application.

We need to tell Docker when and how to run this script so put the following code in your Dockerfile after the line EXPOSE 5000 :

ADD start.sh /
RUN chmod +x /start.sh

The command chmod +x makes the script executable by setting the right permission.

To execute the script, add the following command after the line ENTRYPOINT [ "poetry", "run" ]:

CMD ["/start.sh"]

That’s it! Our first web page is ready so let’s start our app to make sure we don’t have any errors.

In the project root directory execute:

docker-compose build

The first build will take some time because Docker has to download and install a lot of dependencies.

After it finishes run:

docker-compose up

The URL of our web application is http://localhost:5000/. When you open it there should be a message Hello World! which means that the app is up and running.

Now it’s time to create a more complex web page that will contain our Social Network Graph. In the project root directory create a folder called templates and in it a file with the name base.html. This will be our base HTML template for other pages. Copy the code:

<!doctype html>
<html lang="en">

<head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css">
    <link rel="stylesheet" href="/static/css/style.css">

    <script src="https://code.jquery.com/jquery-3.3.1.slim.min.js"></script>
    <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js"></script>
    <script src="https://d3js.org/d3.v4.min.js" charset="utf-8"></script>

    <title>Social Network Graph Demo</title>
</head>

<body>
    {% block content %} {% endblock %}
</body>

</html>

We also need to create an HTML file for our actual landing site that utilizes this base file and an accompanying JavaScript file. Create the HTML file in the same location with the name index.html and copy the following code into it:

{% extends 'base.html' %} {% block content %}
<div class="container">
  Hello World!
</div>
<script src="/static/js/index.js" charset="utf-8"></script>
{% endblock %}

In the project root directory create a folder called static with one subfolder called js and another called css. The js folder will contain all of the needed local JavaScript files while the css folder will contain all the CSS stylesheets. In the js folder create a file called index.js and in the css folder one called style.css. Just leave them empty for now.

If you want to find out more about web development with Flask I suggest you try out this tutorial.

Your current project structure should like this:

sng-demo
├── sng_demo
│  └── __init__.py
├── templates
│  ├── base.html
│  └── index.html
├── static
│  ├── css
│  │  └── style.css
│  └── js
│     └── index.js
├── app.py
├── docker-compose.yml
├── Dockerfile
├── poetry.lock
├── pyproject.toml
└── start.sh

The Data Model and Database Connection

In the app directory sng-demo create a folder called database. This folder will contain all of the modules that we need to communicate with the database. You can find them here and just copy their contents. They are closely related to the database driver and if you wish to examine them a bit more I suggest you look up the driver documentation here.
In the app directory sng-demo create the module db_operations.py. This is where all the custom database related commands will be located.

The sng_demo directory should look like this:

sng_demo
├── __init__.py
├── db_operations.py
└── database
   ├── __init__.py
   ├── memgraph.py
   ├── connection.py
   └── models.py

We will use a very simple data model that can be easily upgraded later on.

There is only one node with the label User and each User has two properties, a numerical id and a string name. Nodes are connected with edges of the type FRIENDS:

There are several methods to populate our database (more on that here) but we will be doing it manually by executing openCypher queries so you can get a better understanding of how to communicate with the database. You will find all the necessary queries to populate the database in the files data_big.txt and data_small.txt. The former just has a larger dataset than the latter.

In the project root directory create a folder called resources and place the files in it. Now you can add an import method to your web application.

In the module db_operations.py add the following import and method:

import json

def clear(db):
  command = "MATCH (node) DETACH DELETE node"
  db.execute_query(command)

def populate_database(db, path):
  file = open(path)
  lines = file.readlines()
  file.close()
  for line in lines:
      if len(line.strip()) != 0 and line[0] != '/':
          db.execute_query(line)

The method clear() deletes any data that might have been left in the database before populating it.

The method populate_database() reads all of the openCypher queries in the specified file and executes them.

In the module app.py change the imports and method index() to:

from flask import Flask, render_template, request, jsonify, make_response
from sng_demo.database import Memgraph
from sng_demo import db_operations

app = Flask(__name__)


@app.route('/')
@app.route('/index')
def index():
    db = Memgraph()
    db_operations.clear(db)
    db_operations.populate_database(db, "resources/data_small.txt")
    return render_template('index.html')

Now every time we refresh our index page the database is cleared and repopulated with new data. While this is not suitable for the production stage, it is highly useful during development because it will enable us to make changes in the data without having to restart the whole application or working directly on the database.

If you want to examine the graph before proceeding, I suggest you open Memgraph Lab and run the query MATCH (n1)-[e:FRIENDS]-(n2) RETURN n1,n2,e;.

The result should be:

We also need a method in our app to fetch all the relevant data from the database when a client requests it.

Let’s call it get_graph() and place it in the db_operations.py module:

def get_graph(db):
   command = "MATCH (n1)-[e:FRIENDS]-(n2) RETURN n1,n2,e;"
   relationships = db.execute_and_fetch(command)

   link_objects = []
   node_objects = []
   added_nodes = []
   for relationship in relationships:
       e = relationship['e']
       data = {"source": e.nodes[0], "target": e.nodes[1]}
       link_objects.append(data)

       n1 = relationship['n1']
       if not (n1.id in added_nodes):
           data = {"id": n1.id, "name": n1.properties['name']}
           node_objects.append(data)
           added_nodes.append(n1.id)

       n2 = relationship['n2']
       if not (n2.id in added_nodes):
           data = {"id": n2.id, "name": n2.properties['name']}
           node_objects.append(data)
           added_nodes.append(n2.id)
   data = {"links": link_objects, "nodes": node_objects}

   return json.dumps(data)

First, we need to execute the openCypher query MATCH (n1)-[e:FRIENDS]-(n2) RETURN n1,n2,e; and return its results from the database. These results will contain all the edges in the graph as well as all the nodes that are connected to those edges. Nodes that don't have connections will not be returned and that's ok for now.

The results (the object relationships) are in the form of a generator which we can iterate over and access its contents by using the node/edge names specified in our initial query (n1,n2 and e).

We also need to check if a node has already been appended to the node_objects list because multiple edges can contain (point to or from) the same node. All of the objects are stored in key-value pairs suitable for later JSON conversion.

The final result is a JSON object containing:

links: all the relationships that are in the graph as pairs of source and target id properties,
nodes: all the nodes from the graph that form relationships with other nodes.

In your app.py module add the following method:

@app.route("/get-graph", methods=["POST"])
def get_graph():
   db = Memgraph()
   response = make_response(
       jsonify(db_operations.get_graph(db)), 200)
   return response

This method is responsible for responding to POST requests from the client. It returns the graph data that we fetched from the server in the previous method.

Now let's do something with this data! Copy the contents for your index.js file from here
and the style.css file from here.

We also need to add the actual SVG graphic to our page so change the index.html file to:

{% extends 'base.html' %} {% block content %}
<div class="container">
    <svg class="border rounded mt-3" width="960" height="600" style="background-color:white"></svg>
</div>
<script src="/static/js/index.js" charset="utf-8"></script>
{% endblock %}

I won't go into much detail about how to use D3.js so if you want to find out more I encourage you to visit their website.

In short, we fetch all the nodes and edges from the database and add them to an SVG element. The visual representation of the graph is made by simulating how physical forces act on particles (charge and gravity). You can drag and drop the nodes, hover over them to see the value of their name property, zoom in and out of the graph and move the SVG graphic.

Additional Functionalities

Go ahead and copy the file query.js to the directory static/js and query.html to the directory templates. You can find the updated base.html file here. Copy the necessary methods from the db_operations.py module and app.py module.

After you made the changes, just open http://localhost:5000/query/ and see the results.

This page will make your life easier if you want to debug the data being fetched from the server. It returns all the nodes or edges and shows them in a JSON highlighted format.

Your current project structure should like this:

sng-demo
├── resources
│  ├── data_big.py
│  └── data_small.txt
├── sng_demo
│  ├── __init__.py
│  ├── db_operations.py
│  └── database
│     ├── __init__.py
│     ├── memgraph.py
│     ├── connection.py
│     └── models.py
├── templates
│  ├── base.html
│  ├── index.html
│  └── query.html
├── static
│   ├── css
│   │  └── style.css
│   └── js
│      ├── index.js
│      └── query.js
├── app.py
├── docker-compose.yml
├── Dockerfile
├── poetry.lock
├── pyproject.toml
└── start.sh

Conclusion

Even though graph databases have been around for a long time, they are still not considered a mainstream tool in software development. Relational database-management systems model data as a set of predetermined structures. Complex joins and self-joins are necessary when the dataset becomes too inter-related. Modern datasets require technically complex queries which are often very inefficient in real-time scenarios.

Graph databases offer powerful data modelling and analysis capabilities for many real-world problems such as social networks, business relationships, dependencies, shipping, logistics... and they have been adopted by many of the worlds leading tech companies. With this tutorial, I hope to shed some light on how easy it is to integrate a graph database in your development process and I encourage you to try it out yourself.

As I said at the beginning, feel free to ask us any questions about this tutorial or Memgraph in general on StackOverflow with the tag memgraphdb or on our Discord server. Good luck with your coding!

Analyzing Real-Time Movie Reviews With Redpanda and Memgraph

Memgraph — Thu, 06 Jul 2023 16:51:55 +0000

In recent years, it has become apparent that almost no production system is complete without real-time data. This can also be observed through the rise of streaming platforms such as Apache Kafka, Apache Pulsar, Redpanda, and RabbitMQ.

This tutorial focuses on processing real-time movie ratings that are streamed through Redpanda, a Kafka-compatible event streaming platform. The data can be used to generate movie recommendations with the help of Memgraph and the Cypher query language.

Prerequisites

To follow this tutorial, you will need:

Docker and Docker Compose (included in Docker Desktop for Windows and macOS)
Memgraph Lab - an application that can visualize graphs and execute Cypher queries in Memgraph.
A clone of the data-streams repository. This project contains the data stream, a Redpanda setup and Memgraph.

Data model

In this example, we will use the reduced MovieLens dataset streamed via Redpanda.
Each JSON message will be structured like the one below:

{
  "userId": "112",
  "movie": {
    "movieId": "4993",
    "title": "Lord of the Rings: The Fellowship of the Ring, The (2001)",
    "genres": ["Adventure", "Fantasy"]
  },
  "rating": "5",
  "timestamp": "1442535783"
}

So how are we going to store this data as a graph?
There are three different types of nodes: Movie, User, and Genre.

Each movie can be connected with an OF_GENRE edge to a different genre. A user can
rate movies, and these ratings will be modeled with the edge RATED. This edge contains the properties rating, which can range from 1.0 to 5.0, and timestamp.

Each Movie has the properties id and title while each User has the property id. A Genre only contains the property name.

1. Start the Redpanda stream

We created a Redpanda topic which you can connect to for the purpose of this tutorial. Clone the data-streams repository:

git clone https://github.com/memgraph/data-streams.git

Position yourself in the data-streams directory and run the following command
to start the Redpanda stream:

python start.py --platforms redpanda --dataset movielens

After the container starts, you should see messages being consumed in the console.

2. Start Memgraph

Usually, you would start Memgraph independently using Docker but this time we are going to use the data-streams project. Given that we need to access the data stream running in a separate Docker container, we need to run Memgraph on the same network.

1. Open a new terminal and position yourself in the data-streams directory you cloned earlier.

2. Build the Memgraph image with:

docker-compose build memgraph-mage

3. Start the container:

docker-compose up memgraph-mage

Memgraph should be up and running. You can make sure by opening Memgraph Lab and connecting to the empty database.

3. Create the transformation module

Before we can connect to a data stream, we need to tell Memgraph how to transform the incoming messages, so they can be consumed correctly. This will be done through a simple Python transformation module:

import mgp
import json


@mgp.transformation
def rating(messages: mgp.Messages
             ) -> mgp.Record(query=str, parameters=mgp.Nullable[mgp.Map]):
    result_queries = []

    for i in range(messages.total_messages()):
        message = messages.message_at(i)
        movie_dict = json.loads(message.payload().decode('utf8'))
        result_queries.append(
            mgp.Record(
                query=("MERGE (u:User {id: $userId}) "
                       "MERGE (m:Movie {id: $movieId, title: $title}) "
                       "WITH u, m "
                       "UNWIND $genres as genre "
                       "MERGE (m)-[:OF_GENRE]->(:Genre {name: genre}) "
                       "CREATE (u)-[:RATED {rating: ToFloat($rating), timestamp: $timestamp}]->(m)"),
                parameters={
                    "userId": movie_dict["userId"],
                    "movieId": movie_dict["movie"]["movieId"],
                    "title": movie_dict["movie"]["title"],
                    "genres": movie_dict["movie"]["genres"],
                    "rating": movie_dict["rating"],
                    "timestamp": movie_dict["timestamp"]}))

    return result_queries

Each time we receive a JSON message, we need to execute a Cypher query that will map it to a graph object:

MERGE (u:User {id: $userId}) 
MERGE (m:Movie {id: $movieId, title: $title}) 
WITH u, m 
UNWIND $genres as genre 
MERGE (m)-[:OF_GENRE]->(:Genre {name: genre}) 
CREATE (u)-[:RATED {rating: ToFloat($rating), timestamp: $timestamp}]->(m)

This Cypher query creates a User and Movie if they are missing from the database. Movies are also connected to the genres they belong to. In the end, an edge of the type RATED is created between the user and the movie, indicating a rating.

Now that we have created the transformation module, another question arises. How to load a transformation module into Memgraph?

1. First, find the id of the container (CONTAINER_ID) where Memgraph is running:

docker ps

Note the id of the memgraph-mage container.

2. Now, you can copy the movielens.py transformation module to the Memgraph container with the following command:

docker cp movielens.py CONTAINER_ID:/usr/lib/memgraph/query_modules/movielens.py

3. Open Memgraph Lab and select the Query tab from the left sidebar.

4. Load the module with the following Cypher query:

CALL mg.load("movielens");

If you don't receive an error, the module was loaded successfully.

4. Connect to the Redpanda stream from Memgraph

1. Execute the following query within Memgraph Lab in order to create the stream:

CREATE KAFKA STREAM movielens_stream 
TOPICS ratings
TRANSFORM movielens.rating 
BOOTSTRAP_SERVERS "redpanda:29092";

2. Now, that we have created the stream, it needs to be started in order to consume messages:

START STREAM movielens_stream;

3. It's time to check if the stream was created and started correctly:

SHOW STREAMS;

That's it! You just connected to a real-time data source with Memgraph and can start exploring the dataset. If you open the Overview tab in Memgraph Lab, you should see that a number of nodes and edges has already been created.

Just to be sure, open the tab Graph Schema and click on the generate button to see if the graph follows the Data model we defined at the beginning of the article.

5. Analyze the streaming data

For data analysis, we will use Cypher, the most popular query language when it comes to graph databases. It provides an intuitive way to work with property graphs. Even if you are not familiar with it, the following queries shouldn't be too hard to understand if you have some knowledge of SQL.

1. Let's return 10 movies from the database:

MATCH (movie:Movie)
RETURN movie.title
LIMIT 10;

2. Find movies that are of genre Adventure and Fantasy:

MATCH (movie:Movie)-[:OF_GENRE]->(:Genre {name:"Fantasy"})
MATCH (movie)-[:OF_GENRE]->(:Genre {name:"Adventure"})
RETURN movie.title
ORDER BY movie.title
LIMIT 10;

3. Calculate the average rating score for the movie Matrix:

MATCH (:User)-[r:RATED]->(m:Movie)
WHERE m.title = "Matrix, The (1999)"
RETURN avg(r.rating)

4. It's time for a more serious query. Let's find a recommendation for a specific user, for example, with the id 6:

MATCH (u:User {id: "6"})-[r:RATED]-(p:Movie)
      -[other_r:RATED]-(other:User)
WITH other.id AS other_id,
     avg(r.rating-other_r.rating) AS similarity,
ORDER BY similarity
LIMIT 10
WITH collect(other_id) AS similar_user_set
MATCH (some_movie: Movie)-[fellow_rate:RATED]-(fellow_user:User)
WHERE fellow_user.id IN similar_user_set
WITH some_movie, avg(fellow_rate.rating) AS prediction_score
RETURN some_movie.title AS Title, prediction_score
ORDER BY prediction_score DESC;

And that's it, you have generated recommendations based on the similarity of ratings between each user. If you want to find out more about this query, definitely check out our tutorial where we go more into detail.

Conclusion

Analyzing real-time data from streaming platforms has never been easier. This also applies to graph analytics, as we have demonstrated through the use of Redpanda and Memgraph. If you have any questions or comments, check out our Discord server.

The rules are simple, you just have to create something that enriches the world of graphs! It could be a web application, Memgraph driver, an integration for a graph library, or implementation of a graph algorithm in MAGE. You could just create a Python script or Jupyter Notebook for graph analysis.

Good luck with your coding, and don’t forget to register for the Challenge!

How to Implement Custom JSON Utility Procedures With Memgraph MAGE and Python

Memgraph — Tue, 04 Jul 2023 14:46:56 +0000

Introduction

Oftentimes you find yourself unable to come up with the perfect query that fits the problem at hand. Every query language has its disadvantages and Cypher is no exception. But thankfully, there is always the option of writing your own custom procedures.

Memgraph introduces the concept of query modules which are collections of custom Cypher procedures. You can implement them using a Python or C API.

In this tutorial, you will go through the process of implementing a few simple utility procedures to load and export data in a JSON format.

Introducing Memgraph MAGE

MAGE stands for Memgraph Advanced Graph Extensions. It's an open-source project started by Memgraph that encourages developers to share innovative and useful query modules so the whole community can benefit from them.

You can find the MAGE repository on this link.

Prerequisites

To complete this tutorial, you will need:

An installation of Memgraph DB: a native fully distributed in-memory graph database built to handle real-time use-cases at enterprise scale. Follow the Docker Installation instructions on the Quick Start page to get started.
An installation of Memgraph Lab: an integrated development environment used to import data, develop, debug and profile database queries and visualize query results.

Importing Data from JSON Files

Memgraph doesn't come with the option of handling JSON out of the box. So what are your options if you need this feature in an upcoming project?

Well, there are actually two ways of importing such data:

Independently from Memgraph,
Using query modules in Memgraph.

The first option is a pretty straightforward hack. You just parse the needed JSON document and create the appropriate queries for populating your database. This way, Memgraph has no knowledge about the JSON file, you have to handle it completely by yourself and only run the finished queries with the data extracted from the JSON file.

The second option is a bit more elegant is what you'll be learning in the rest of this tutorial.

Writing Custom Cypher Procedures

First things first. To start working on a new query module, you need to be familiar with the development process. If you are running Memgraph on anything other than Docker then continue with the next paragraph, otherwise, skip to the Developing Custom Query Modules using Docker section.

Upon startup, Memgraph will attempt to load the query modules from all *.so and *.py files it finds in the default (/usr/lib/memgraph/query_modules) directory. If you want to change the directory in which Memgraph searches for query modules, just change the --query-modules-directory flag in the main configuration file (/etc/memgraph/memgraph.conf) or supply it as a command-line parameter (e.g. when using Docker), for example:

docker run -p 7687:7687 --query-modules-directory /usr/lib/memgraph/new_query_modules memgraph

If you want to add a new query module, it needs to be placed in this directory. It will automatically load when Memgraph starts, but you can also reload it while the database is running by executing the following query:

CALL mg.load("QUERY_MODULE_NAME")

Developing Custom Query Modules Using Docker

When using Docker, you don't have direct access to the default query modules directory because it is within the Docker container. Create a volume and mount it to access the /usr/lib/memgraph/query_modules directory. This can be done by creating an empty directory modules and executing the following command:

docker volume create --driver local --opt type=none  --opt device=~modules --opt o=bind modules

Now, you can start Memgraph and mount the created volume:
docker run -it --rm -v modules:/usr/lib/memgraph/query_modules -p 7687:7687 memgraph`

Everything from the directory /usr/lib/memgraph/query_modules will be visible/editable in your mounted modules volume and vice versa.

Implementing the JSON Utility Query Module in Python

You will name the query module json_util.py because it will contain utility functions that are needed to work with JSON files. For now, let's implement the following three procedures:

Load JSON from a local file
Load JSON from a remote address
Export nodes as JSON document

1. Loading JSON from a Local File

In your json_util.py module add the following code:

`python
import json
import mgp
import urllib.request

@mgp.read_proc
def load_from_path(ctx: mgp.ProcCtx,
json_path: str) -> mgp.Record(objects=mgp.List[object]):

with open(json_path) as json_file:
    objects = json.load(json_file)

    if type(objects) is dict:
        objects = [objects]

    return mgp.Record(objects=objects)

With this, you have implemented your first procedure. The @mgp.read_proc decorator registers the function as a read-only procedure of the current module. The if statement makes sure that the procedure returns a list even if it's just one element. This will be useful for working with the data later on.

How do you test this procedure? Let's create a file in the /usr/lib/memgraph/query_modules directory and name it data.txt. Place the following content in it:

json [{"name":"Leslie"}, {"name":"Ron"}, {"name":"Donna"}]

Start Memgraph Lab if you haven't done so already and run the following query:

cypher CALL json_util.load_from_path("/usr/lib/memgraph/query_modules/data.txt") YIELD * RETURN *

2. Loading JSON from a Remote Address

While loading data from local files can be helpful, especially when developing your new procedure, there is a bigger need for a procedure that loads data from a remote location via URL. Thankfully, you only have to add a small adjustment to the load_from_path() function to achieve this functionality. Let's name this new procedure load_from_url:

`python
@mgp.read_proc
def load_from_url(ctx: mgp.ProcCtx,
json_path: str) -> mgp.Record(objects=mgp.List[object]):

with urllib.request.urlopen(json_path) as url:
    objects = json.loads(url.read().decode())

    if type(objects) is dict:
        objects = [objects]

    return mgp.Record(objects=objects)

You can test it by running the following query:

cypher CALL json_util.load_from_url('ADDRESS') YIELD objects UNWIND objects AS o RETURN o.name

3. Exporting Nodes as a JSON Document

This procedure will receive a list of nodes and save them in JSON format to a local file.

`python
@mgp.read_proc
def export_nodes(ctx: mgp.ProcCtx,
nodes: mgp.List[mgp.Vertex],
file_path: str
) -> mgp.Record(success=bool):

json_nodes_list = []
for node in nodes:
    json_node = {}
    json_node['labels'] = []

    for label in node.labels:
        json_node['labels'].append(label.name)

    json_node['properties'] = dict(node.properties.items())
    json_nodes_list.append(json_node)

with open(file_name, 'w') as fp:
    json.dump(json_nodes_list, fp)

return mgp.Record(success=True)

You can test the procedure by running:

cypher MATCH (n) WITH COLLECT(n) AS listn CALL json_util.export_nodes(listn, "/usr/lib/memgraph/query_modules/data.json") YIELD success RETURN success

The file data.json should be in the /usr/lib/memgraph/query_modules directory.

Conclusion

In this tutorial, you learned how you can easily add additional functionalities to the Cypher query language by writing your own procedures. While importing data from JSON documents is considered more of a utility procedure, query modules can be a powerful tool for writing custom graph algorithms or implementing all kinds of constructs from the realm of graph theory.

If you are working on your own query module and would like to share it, take a look at the contributing guidelines. We would be more than happy to provide feedback and add the module to the MAGE repository.

For a more in-depth explanation of how to create your own custom Cypher procedures take a look at our documentation. If you would like more step-by-step tutorials exploring custom query modules, make sure to read our How to Write Custom Cypher Procedures with NetworkX and Memgraph tutorial.

5 Questions on Performance Benchmarks

Memgraph — Thu, 29 Jun 2023 16:00:44 +0000

Performance is one of the all-time hot topics in the database world! And as a set of activities that database administrators or developers use to test out the database or compare them against certain criteria, benchmarks often come in parallel with performance. This article is Memgraph’s take on both.

Previously we have published some benchmark runs on Memgraph, which has been well-received by the community and led to some comments from the wider graph database community. The feedback contained some great suggestions on how we can improve our benchmark methodology to make sure the results follow the standard procedures of the benchmarking council.

To offer the fairest comparison to other databases, we’ve taken those lessons into account and developed Benchgraph, where you can run benchmarks on your own workloads. The launch has been followed by a community call where our very own Marko Budiselić and Ante Javor answered some of the common questions asked by the community. To save you time, we’re repurposing the key topics discussed into a blog post. Without further ado, let’s jump in.

Industry-standard vs. vendor-specific benchmarks

“One of the things our community has brought up is sometimes vendors will build their benchmarks to perform really well on their own datasets and behaviors. Whereas industry-standard benchmarks built by councils or committees tend to be created to give a more fair comparison between different databases. What is Memgraph’s take on this feedback?”

Both industry-standard and vendor-specific benchmarks play important roles in assessing database performance, and their relevance to your project depends on your specific use case and priorities. In the graph database space, the Linked Data Benchmark Council, or LDBC, is a well-known industry-standard organization. LDBC defines specific workloads that serve as benchmarks to showcase performance.

Industry-standard benchmarks, including LDBC, typically take a generic approach to suffice various database vendors and support common workloads that any database should handle. And they provide a baseline for comparison across different systems. On the other hand, vendor-specific benchmarks take a more tailored approach based on specific use cases. For example, Memgraph’s Benchgraph focuses on vendor-specific benchmarks for tasks like variable traversals, read-write ratios, and their variations, allowing users to observe differences in execution within a specific vendor's database.

Industry-standard and vendor-specific benchmarks are equally important because users face different workload scenarios and have different data priorities.

How do read vs. write operations impact performance?

“Another point that our community has brought up is some of the databases will behave differently depending on the architectural implementation in different scenarios, and one of those scenarios could be things like how well you cash your queries, results, indexes, etc. So how does something like rights influence those when your cache has to be invalidated, or your index and memory isn't completely valid anymore? In other words, how does read vs. write impact performance?”

The impact of read-and-write operations on database performance relies heavily on the architecture of the database and the underlying system. When data is written, the associated cache needs to be invalidated, and the cost of this invalidation varies across different systems. In certain workloads, such as streaming, there may be a large number of writes occurring within a short time span, making cache invalidation quite expensive.

But cache performance is not only influenced by the act of caching itself but also by what is being cached. For example, caching can be highly beneficial for read-only workloads, but it can negatively impact query performance in write-intensive scenarios. At Memgraph, to thoroughly evaluate database performance, we test both read-heavy and write-heavy workloads, as well as combinations of the two.

So, the takeaway is when dealing with a high volume of updates per second, cache effectiveness decreases significantly. And if the maintenance cost of these caches is high, it can result in lower overall performance.

Long vs. short-running benchmarks

“Our community has also brought up the topic of comparing long vs. short runs. That, in a way, relates to the previous point of running up the database, indexes, etc. What has been Memgraph’s approach to this?”

Memgraph’s speed comes from its in-memory architecture. So, there is no need for lengthy warm-up periods or time-consuming benchmarks. Traditional long-running benchmarks allow for quite some time to cache queries, indexes, and other optimizations, which can showcase the best performance achievable by a database. On the other hand, short-running benchmarks, due to their limited duration, often highlight the worst performance potential. That is why these two types of benchmarks are like the two extremes, and your decision should depend on your project's specific requirements and priorities.

Tail latency vs. mean-values

“Cassandra is well known for having a bit of a garbage collection issue every once in a while, which means that suddenly queries can take seconds instead of milliseconds. And if you've got an application where response time matters, it can truly sour the experience for end users. So mean latency is important, but tail latency matters as well. What’s Memgraph’s opinion on this?”

When assessing performance, people typically rely on mean or median latency, as they are easily obtainable. However, to gain a comprehensive understanding of real-world scenarios, tail latency provides a much broader perspective. Imagine you can effectively plan and make informed decisions, considering the potential outliers and extreme cases rather than relying purely on the average query execution time. And this becomes particularly important when dealing with fast-changing or streaming data environments with high writes coming in.

Memgraph goes the extra mile by implementing three layers of garbage collection so that the system operates seamlessly and maintains a clean environment. So, overall, tail latency is just as crucial since it enables a deeper understanding of how garbage collection impacts the end-user experience.

Should you restart your database between runs?

“The final topic our community has brought up is whether you should restart your database servers between those runs.”

When using a database in production, it's common to set it up and run it for a long time without restarting unless there's a maintenance issue or failure. However, this approach can introduce challenges when conducting benchmarks, as different tests can potentially influence each other's results.

To ensure accurate performance evaluations, it's important to restart the database before each test. This way, you can simulate the worst-case scenario and avoid any bias in the results. In real production environments, there's no guarantee of the order in which tests or queries will be executed, making it even more crucial to restart the database between each test to capture a comprehensive range of performance scenarios.

Bonus: How to run Benchgraph on your own workload?

Closing remarks

Have you already tested Benchgraph on your workload? We like receiving feedback from our community as it helps us improve and better serve our users. So, please, don’t hesitate to share your results on our Discord channel and tell us what you would love to see next!

How to Write Custom Cypher Procedures With NetworkX and Memgraph

Memgraph — Tue, 27 Jun 2023 15:45:15 +0000

Introduction

NetworkX is a package for the creation, manipulation, and study of the dynamics, functions and structures of networks. It allows us to use complex graph algorithms to solve network-related problems.

Even though NetworkX is a very powerful and versatile package, it is somewhat limited in speed and efficiency because of its Python implementation and lack of quality storage. While this is not a problem with smaller data sets, in large networks this can be quite cumbersome. Using Memgraph, an in-memory graph database, as the storage solution provides additional benefits and functionalities to NetworkX.

In this post, we will focus on how to implement custom Cypher procedures using query modules and NetworkX. We will also discuss the open-source project MAGE where you can find already implemented modules and contribute your own.

Prerequisites

The prerequisites for this tutorial are either:

Memgraph Platform: the complete streaming graph application platform that includes:
- MemgraphDB - the database that holds your data
- Memgraph Lab - an integrated development environment used to import data, develop, debug and profile database queries and visualize query results
- mgconsole - command-line interface for running queries
- MAGE - graph algorithms and modules library

Memgraph Cloud: a hosted and fully-managed service that utilizes all Memgraph products such as MAGE and in-browser Memgraph Lab

Understanding NetworkX Basics

To fully utilize the power of graphs, you first need to get a basic understanding of the underlying concepts in graph theory and their corresponding terminology when working with the NetworkX package.

The four main types of graphs that you will see throughout this and other tutorials are:

Graph - An undirected graph with self-loops.
DiGraph - A directed graph with self-loops.
MultiGraph - An undirected graph with self-loops and parallel edges.
MultiDiGraph - A directed graph with self-loops and parallel edges.

A self-loop (also called a loop or a buckle) is an edge that connects a vertex to itself while parallel edges (also called multiple edges or a multi-edge) are two or more edges that are incident to the same two nodes.

Memgraph offers a comprehensive set of thin wrappers around most of the algorithms and objects present in the NetworkX package. The wrapper functions have the capability to create a NetworkX compatible graph-like object that can stream the native database graph directly saving on memory usage significantly. For example, the NetworkX class MultiDiGraph has a MemgraphMultiDiGraph wrapper while the DiGraph class uses the MemgraphDiGraph wrapper class.

Setting up NetworkX in Memgraph

Memgraph supports extending the query language with user-written procedures. These procedures are grouped into modules, and if written in Python, can be developed directly from Memgraph Lab. We are going to create such a procedure to work with the NetworkX package.

In Memgraph Lab, query modules are written in the Query modules section where you can also find built-in query modules that come prepackaged with Memgraph Platform and Cloud. To learn more about query modules check out the Memgraph documentation on the subject.

Writing a Custom Procedure for Cypher

First, we need a basic data set to start with. Let's create a very simple graph with two cities that are connected by a road.

The cities are nodes while the road is an edge connecting these nodes. Open Memgraph Lab and in the Query Execution section run the following query:

CREATE (c1:City { name: 'Boston' }), (c2:City { name: 'New York' })
CREATE (c1)-[r:ROAD_TO { distance: 100}]->(c2)
RETURN c1, c2, r;

This has created two nodes with the label City and a name property.
The edge between them is of type ROAD_TO and has the property distance which represents the distance in kilometers.

We want a slightly more complex network to use with NetworkX so let's extend it with the following queries:

MATCH (c1:City { name: 'Boston' })
CREATE (c2:City { name: 'Philadelphia' }), (c3:City { name: 'Seatle' }), (c4:City { name: 'Los Angeles' }), (c5:City { name: 'San Francisco' })
CREATE (c1)-[:ROAD_TO { distance: 110}]->(c2)
CREATE (c3)-[:ROAD_TO { distance: 150}]->(c2)
CREATE (c4)-[:ROAD_TO { distance: 200}]->(c5)

This is a simple directed graph and while using NetworkX it would be stored as a DiGraph. If we were to duplicate all the edges and reverse the direction of the duplicates, we would end up with a MultiDiGraph because of the parallel edges. This could be accomplished by the following query:

MATCH (c0:City { name: 'New York' }), (c1:City { name: 'Boston' }), 
      (c2:City { name: 'Philadelphia' }), (c3:City { name: 'Seatle' }), 
      (c4:City { name: 'Los Angeles' }), (c5:City { name: 'San Francisco' })
CREATE (c1)<-[:ROAD_TO { distance: 100}]-(c0)
CREATE (c1)<-[:ROAD_TO { distance: 110}]-(c2)
CREATE (c3)<-[:ROAD_TO { distance: 150}]-(c2)
CREATE (c4)<-[:ROAD_TO { distance: 200}]-(c5)

Now, let's play with our graph in NetworkX. Open the Query modules section and click on the + New Module. Give the modules name hello_world. A new query module will be created with example procedures. Feel free to erase them and copy the following code into it:

import mgp
import networkx as nx
from mgp_networkx import (MemgraphMultiDiGraph, MemgraphDiGraph,
                          MemgraphMultiGraph, MemgraphGraph,
                          PropertiesDictionary)

@mgp.read_proc
def procedure(ctx: mgp.ProcCtx) -> mgp.Record(nodes=mgp.List[mgp.Vertex]):

    g = MemgraphMultiDiGraph(ctx=ctx)

    return mgp.Record(nodes=list(g.nodes()))

The first three lines are basic Python imports which all of our query modules that utilize NetworkX need to have.

Because we are working with data stored in Memgraph, a wrapper is used to reference the stored graph. The line g = MemgraphMultiDiGraph(ctx=ctx) implies that we are creating this reference from the current context ctx (automatically passed in all procedure calls) and it represents a nx.MultiDiGraph class.

The procedure returns a list of nodes from the graph. If you need a more detailed understanding of the custom objects and wrappers take a look at the query modules Reference Guide.

Save and close the window then move to the Query Execution section to use the procedure.

Call the procedure to return results:

CALL hello_world.procedure() YIELD nodes RETURN nodes;

What we get is a list of nodes in the graph.

Let's change the procedure to also return the number of nodes and the number of edges in the graph. Go back to the Query Modules section, find the hello_world query module, click on the arrow on the right to see its details, then edit it by modifying the code:

@mgp.read_proc
def procedure(ctx: mgp.ProcCtx) -> mgp.Record(nodes=mgp.List[mgp.Vertex],
                                              number_of_nodes=int,
                                              number_of_edges=int):

    g = MemgraphMultiDiGraph(ctx=ctx)

    list_of_nodes = list(g.nodes())
    num_of_nodes = g.number_of_nodes()
    num_of_edges = g.number_of_edges()

    return mgp.Record(nodes=list_of_nodes,
                      number_of_nodes=num_of_nodes,
                      number_of_edges=num_of_edges)

Before you calling the procedure, don't forget to save the module.

CALL mg.load_all();
CALL hello_world.procedure() YIELD * RETURN *;

With YIELD * we make sure that the procedure returns all of the results and by using RETURN * we print them all out.

Editing Graphs with NetworkX

It's important to note that you can't do any database write operations directly from a query module. Otherwise, it wouldn't be possible to guarantee the integrity of the database. You can however save the whole graph or just a subgraph as a completely new object and then perform edits before returning the results. Let's say we wanted to find out what our custom procedure would return if we removed the cities Boston and New York from our network. Create a new procedure in the same module and name it remove_node_procedure():

@mgp.read_proc
def remove_node_procedure(ctx: mgp.ProcCtx) -> mgp.Record(nodes=mgp.List[mgp.Vertex],
                                              number_of_nodes=int):

    g = MemgraphMultiDiGraph(ctx=ctx)
    g_copy = nx.MultiDiGraph(g)

    for node in list(g_copy.nodes):
        if node.properties.get('name') == 'Boston':
            g_copy.remove_node(node)

    list_of_nodes = list(g_copy.nodes)
    num_of_nodes = g_copy.number_of_nodes()

    return mgp.Record(nodes=list_of_nodes,
                      number_of_nodes=num_of_nodes)

As you can see, the returned list does not contain the city Boston.

Passing Arguments to Custom Procedures

Let's make a new procedure that uses the nx.is_simple_path() algorithm. A simple path in a graph is a nonempty sequence of nodes in which no node appears more than once in the sequence, and each adjacent pair of nodes in the sequence is adjacent in the graph. The simple_path_procedure() could look like this:

@mgp.read_proc
def simple_path_procedure(ctx: mgp.ProcCtx, nodes: mgp.List[mgp.Vertex]
                          ) -> mgp.Record(is_simple_path=bool):

    g = MemgraphMultiDiGraph(ctx=ctx)

    simple_path = nx.is_simple_path(g, nodes)

    return mgp.Record(is_simple_path=simple_path)

You can call the procedure by executing the following query from Memgraph Lab:

MATCH p=((c1:City {name:'Boston'})-[:ROAD_TO]->(c2:City {name:'New York'}))
CALL hello_world.simple_path_procedure(nodes(p)) YIELD is_simple_path RETURN is_simple_path;

With this query, we have created a list of nodes from the path Boston--New York and the simple_path_procedure() should return True because it is a simple path.

MAGE - Introducing Memgraph Advanced Graph Extensions

MAGE is an open-source repository provided by Memgraph that contains implementations of query modules, written by our engineers or contributed by the developer community.

In it, you can find implementations of various algorithms in multiple programming languages, all runnable inside Memgraph. Some of the available algorithms and problems that are tackled with MAGE include PageRank, Weakly Connected Components (Union Find), Jaccard similarity coefficient, Travelling Salesman Problem... and many more.

Don't hesitate to contribute any custom modules you come up with if you think they would be of help to fellow developers. We would be more than happy to take a look and add them to the repository.

Conclusion and Next Steps

While there are some limited functionalities, Memgraph offers a straightforward way of using the NetworkX package on graph-like objects that can stream the native database graph directly. This kind of implementation improves memory usage significantly and adds in-memory efficiency to NetworkX algorithms.

For a more in-depth explanation of how to create your own custom Cypher procedures take a look at our How to Implement Query Modules? guide or go through the already implemented algorithms in MAGE.

How to Build a Flight Network Analysis Graph-Based ASP.NET Application with Memgraph, C#, and D3.js

Memgraph — Fri, 23 Jun 2023 11:56:35 +0000

Introduction

Relational databases are still the norm when it comes to storing data and are often the de facto choice for developers when building any type of application. Although relational databases are versatile, there are certain use cases that are more fitted for different types of database systems such as - graph databases. For example, when your dataset is deeply interconnected, storing it in a SQL database can result in a lot of complex queries and JOIN operations. For this type of use-case, using a graph database can make your life much easier.

In this tutorial, you will learn how to build a flight network analysis graph-based ASP.NET application using Memgraph, C#, and D3.js.

Prerequisites

Before you start building your application, you will need the following:

An installation of Memgraph DB: a native, in-memory database. To install Memgraph DB and set it up, please follow the instructions on the Installation guide.
(Optional) An installation of Docker: an open platform for developing, shipping, and running applications.
(Optional) Memgraph Lab: a desktop application for querying and visualizing your Memgraph data.

Importing the Dataset

Before you start working on your ASP.NET application, you need to import your dataset into Memgraph. For this tutorial, you will be using a flight network dataset containing a subset of flights from 2015.

The dataset consists of US airports. Each airport has an IATA (location identifier), name, city and state, and a geographical location - latitude and longitude. Airports are connected via flights, which contain the flight number, the operating airline name, the flight date, the time of departure, and the flight distance.

To import the data, please follow the Import tool guide.

If you use the CSV import tool, use this dataset.

If you decide to use Cypher queries with mgconsole, use these Cypher queries.

If you decide to import Cypher queries using Memgraph Lab, make sure to use these Cypher queries.

Creating the Project Structure

For this tutorial, you will be using Visual Studio 2019 on Windows 10 to develop the application. The steps might be slightly different for other IDEs.

Open Visual Studio and create an empty ASP.NET Core Web Application project.
Name your project MemFlights (or any name you want), choose an appropriate location, and click Create.
In Visual Studio, select Tools > NuGet Package Manager > Manage NuGet Packages for Solutions
Type Neo4j.Driver.Simple in the search box, select it, and click Install.

Now, you should be ready to create a simple application. The first step is to establish a connection with the database. To do this, find Startup.cs in Solution Explorer and open it. Locate the ConfigureServices method and add the following code:

public void ConfigureServices(IServiceCollection services)
        {
            services.AddControllers();
            services.AddScoped<IFlightsRepository, FlightsRepository>();
            services.AddSingleton(GraphDatabase.Driver(
                "bolt://localhost:7687",
                AuthTokens.None
                ));
        }

This part of the code will allow your application to connect to the database.

Communicating with Database

Now that you’ve established the connection with your database, you are ready to fetch your data and display it! Create a new folder called Repository and make a new class called FlightRepository.cs. You will create a method FetchGraph which will get your data and place it in a format compatible with D3.js - a JSON-like structure that consists of two lists: nodes and links which are the essential building blocks of the graph.

public async Task<D3Graph> FetchD3Graph(int limit)
        {
            var session = _driver.AsyncSession(WithDatabase);
            try
            {
                return await session.ReadTransactionAsync(async transaction =>
                {
                    var cursor = await transaction.RunAsync(@"MATCH (origin:Airport)-[f:IS_FLYING_TO]->(dest:Airport) " +
                        "WITH origin, dest ORDER BY origin.name, dest.name " +
                        "RETURN origin.NAME AS origin_air, dest.NAME AS dest_air", new { limit });
                    var nodes = new List<D3Node>();
                    var links = new List<D3Link>();
                    var records = await cursor.ToListAsync();
                    foreach (var record in records)
                    {
                        var orgAirport = new D3Node(title: record["origin_air"].As<string>(), label: "airport");
                        var originAirportIndex = nodes.Count;
                        nodes.Add(orgAirport);
                        var destAirport = new D3Node(record["dest_air"].As<string>(), "airport");
                        var destAirportIndex = nodes.IndexOf(destAirport);
                        destAirportIndex = destAirportIndex == -1 ? nodes.Count : destAirportIndex;
                        nodes.Add(destAirport);
                        links.Add(new D3Link(destAirport.Title, orgAirport.Title));
                    }
                    return new D3Graph(nodes, links);
                });
            }
            finally {
                await session.CloseAsync();
            }
        }

You are now ready to execute your first graph query:

MATCH (origin:Airport)-[f:IS_FLYING_TO]->(dest:Airport) 
WITH origin, dest 
ORDER BY origin.name, dest.name 
RETURN origin.NAME AS origin_air, dest.NAME AS dest_air;

This query will return all of the origin and destination airports that have flights between them. All of the objects are stored in key-value pairs which will later be converted into JSON format. The end result will be a JSON object containing:

nodes - all of the nodes(airports) from the graph.
links - all of the relationships between source and target nodes.

Visualizing the Graph Data

Now that you have some results, it's time to visualize them. To do so, you will use D3.js, a JavaScript library for manipulating document-based data. If you want to learn more about D3.js, you should visit their official website.

Create a wwwroot folder and add index.html file. This file will contain your function to visualize the graph.

To include D3.js into your webpage, simply add

<script src="https://d3js.org/d3.v6.min.js" charset="utf-8"></script>

to the header of the file.

Next, you need to collect the data from your application. You can do this with the following function:

    d3.json("/graph").then(function (graph) {
            const links = graph.links.map(d => Object.create(d));
            const nodes = graph.nodes.map(d => Object.create(d));

Now that the data is collected, you need to create the nodes and relationships that will be showcased in your graph. To accomplish this, you can use the following code:

    const link = g.append("g").selectAll("lines")
                .data(links).enter().append("line").attr("class", "link");

    const node = g.append("g").selectAll("circle")
                .data(nodes).enter().append("circle")
                .attr("class", function (d) { return "node " + d.label; })
                .attr("r", 10)
                .call(d3.drag()
                    .on("start", dragstarted)
                    .on("drag", dragged)
                    .on("end", dragended));

You can find the full code here!

You should now have a graph visualization similar to the one below and be able to drag nodes around.

You can find the entire code for this tutorial in this GitHub repository.

Conclusion

Even though graph databases have been around for a long time, they are considered a new technology. They offer a lot of different possibilities and are a powerful tool for any use-case that involves highly interconnected data and complex traversals.

In this tutorial, you learned how to build a simple .NET graph application using Memgraph, C#, and D3.js. You learned how to import your data into Memgraph, communicate with the database using the C# client, run Cypher queries, and visualize the results with D3.js.

Now that you are familiar with Memgraph, you should definitely try some of our other tutorials, or even better, build your own application.

If at any point you need help, feel free to post your questions on StackOverflow with the tag memgraphdb or on our Discord server.

ACID Transactions: What’s the Meaning of Isolation Levels for Your Application

Memgraph — Wed, 21 Jun 2023 13:35:36 +0000

Database ACID transactions form the foundational basis of databases. They ensure data integrity by applying appropriate isolation levels in multi-user scenarios. In this article, we will explore the potential impact they can have on your application.

Databases and data consistency

At the time of writing, it is universally known that a database is used to store and retrieve data in an optimal way. The way data is stored and retrieved depends on the type of the database and and their purpose. To gain a better understanding of the diverse databases available, we recommend visiting the DB-engines ranking site, where you will find an a handful of options to explore.

Regardless of the data format, the data should be stored and retrieved consistently. That means that data consistency and integrity should be an absolute priority when designing and building a database engine. Data consistency ensures that data stays correct, up-to-date, and is valid after the multitude of transactions across the database system. Multiple aspects contribute to data consistency, in general, including data validation, error handling, concurrency control, etc.

Though ensuring data consistency in a database can be challenging. To keep the data consistent, databases employ the principles of ACID transactions.

ACID transactions

An ACID-compliant database, as the name suggests, operates based on transactions that adhere to specific rules. ACID stands for Atomicity, Consistency, Isolation, and Durability, and each of these properties contributes to creating a more robust and reliable database system.

Atomicity ensures that a transaction is treated as a single indivisible unit of work, meaning that either all the changes made by the transaction are committed, or none of them are. This means no partial changes can ever occur. Imagine if you were to abruptly interrupt a transaction by pulling the plug. With atomicity, you can rest assured that you won’t end up with some transactions partially executed while others are left incomplete.

Consistency ensures that the database maintains a valid state both before and after the transaction is executed, following specific predefined rules and constraints. For instance, having values within predetermined ranges and ensuring the presence of unique values are examples of maintaining a consistent and valid data state.

Isolation aims to offer concurrent transactions a sense of independence, ensuring that each transaction is executed in isolation from others, even when they occur simultaneously.
This is particularly important for multi-user databases that face a bunch of different concurrent requests.

Finally, durability guarantees that once a transaction is committed, its changes are permanent and will survive any subsequent failures. If you pull the server plug, the database should restore its previous state from durability files.

Each property is crucial for different reasons, but for the purposes of this article, let’s focus on the isolation property, which defines database isolation levels.

Database isolation levels in your application

Database isolation levels are important because they determine the degree of consistency and correctness in a multi-user database system. In a multi-user database system, transactions may overlap and access the same data concurrently, leading to conflicts and inconsistencies if not managed correctly. Isolation levels provide a way to control the degree of interaction and visibility between transactions, which can help ensure that concurrent transactions do not interfere but produce correct and consistent results.

These days, the goal is to maximize the use of server cores by leveraging parallel computing techniques. It is somewhat expected that databases can handle transactions concurrently and from different users. Unfortunately, not all things can run in parallel. Managing a higher isolation level has become increasingly interesting as it offers enhanced data consistency while addressing potential downsides like increased memory usage, contention, and reduced concurrency associated with stricter isolation levels.

Choosing the appropriate isolation level for a database system depends on several factors, including the desired degree of consistency, performance requirements, and the specific types of transactions and access patterns in the database system.

One of the more popular research papers on isolation levels and their e necessity is A Critique of ANSI SQL Isolation Levels. The paper came out as an expansion on top of ANSI SQL isolation levels. It mentions many isolation levels, from the weakest to strongest.

Read Uncommitted
Read Committed
Cursor Stability
Oracle Read Consistency
Snapshot Isolation
Serializable

Each isolation level not only offers a lower or higher level of consistency and correctness but also impacts the performance potential due to locking. Keep in mind that isolation levels depend on vendor implementation and can be weaker or stronger than theoretically defined. When considering isolation levels primarily from a correctness standpoint, also take into account the potential issues that may arise when using lower isolation levels.

The paper defines the following phenomena (or issues):

Dirty Write
Dirty Read
Non-Repeatable or Fuzzy Read
Phantom

Remember that these are isolation issues from a single paper from 1995. There is a much bigger list of possible issues that exist in real-world scenarios? Let's explore one example of such phenomeno: Dirty Read. Here is a brief explanation of Dirty Read as described in the paper:

“Transaction T1 modifies a data item. Another transaction T2 then reads that data item before T1 performs a COMMIT or ROLLBACK. If T1 then performs a ROLLBACK, T2 has read a data item that was never committed and so never really existed.”

If your database is running using the Read Uncommitted isolation level, it will not help detect the dirty read phenomena described above. On the other hand, the Read Committed isolation level provides a guarantee that a transaction can only access data that has been committed by other transactions. However, it does not necessarily allow access to data that is currently uncommitted or undergoing modifications by other transactions. Read committed isolation level manages access to the data by introducing a read lock on a row or a node that is being read in some transaction. While this may decrease the concurrent execution speed, it will ensure a more consistent database state.

Let’s take a look at the second type of anomaly, Non-repeatable or Fuzzy Read, from the paper:
“Transaction T1 reads a data item. Another transaction T2 then modifies or deletes that data item and commits. If T1 then attempts to reread the data item, it receives a modified value or discovers that the data item has been deleted.”

In both Read Uncommitted and Read Committed isolation levels, the earlier described points would not be detected because the changes being read are already committed. However, a higher isolation level, like Snapshot Isolation, would prevent this issue. This example highlights why the Read Committed isolation level may fall short in handling highly concurrent workloads.

Different isolation levels can have a significant impact on the behavior and performance of different user applications that interact with a database system. The selection of the right isolation level for a database system depends on various factors, as mentioned earlier. These include the desired level of consistency, performance requirements, and the specific types of transactions and access patterns within the database system.

To put things into a practical perspective, consider a simple banking application that allows users to transfer funds between accounts. Suppose the application uses the Read Uncommitted isolation level. In that case, a user may be able to see uncommitted changes made by other users, which could lead to incorrect account balances being displayed. On the other hand, if the application uses the Serializable isolation level, each transaction may need to wait for other transactions to complete before executing, which could lead to reduced performance and potentially even deadlock if transactions are blocked waiting for each other.

Different isolation levels can also impact the performance and scalability of database systems in different ways. For example, the Read Committed isolation level may allow for higher concurrency and throughput than the Serializable isolation. Still, it may lead to more conflicts and rollback of transactions due to optimistic locking. Due to the complex nature of the topic, each isolation level should be explored on a database level since the implementation can vary between databases. On top of that, databases provide different means of handling concurrent transactional issues.

Responsibility for handling the concurrency control

When it comes to managing concurrent transactional issues, the following research paper argues that existing concurrency control mechanisms in databases place excessive responsibility on developers to maintain correctness. The paper proposes a fresh approach that redistributes this responsibility towards the database system itself.

The authors of the research paper emphasize the need for more than just traditional concurrency control mechanisms, such as locks, to ensure correctness in modern database systems. Contemporary databases often involve complex transactions with multiple access patterns and dependencies. As a result, developers must write complex application-level logic to ensure correctness, which introduces the likelihood of errors and becomes difficult to maintain over time.

To address these issues, the authors propose a new approach where the database system takes a more active role in managing concurrency control. Specifically, they propose a framework that allows the database system to analyze the access patterns and dependencies of transactions and automatically generate a customized concurrency control mechanism based on the specific workload. This would reduce the burden on developers to write complex application-level logic and ensure correctness while still allowing for high performance.

To be more specific, there is a particular class of bugs called Read followed by a relevant Write. The author defines a sub-class of the issues where there is a read followed by an update on the same data item. Furthermore, there are two sub-cases, both of which are encapsulated under the name RW1:

// RW1 Case 1: Classic lost update.
Read(a); Check a>=1; b=a-1; Update(a)=b; 
// RW1 Case 2: No update is lost, but the final value of "a" may become negative.
Read(a); Check a>=1; Update(a)=a-1;

An interesting data point is that the Snapshot Isolation level could solve all RW1 Case 1 issues and that RW1 Case 2 issues are almost half of all the problems (42/94). Many issues happen because developers are unaware of the implications of lowering isolation levels or because databases do not support higher isolation levels, such as Snapshot Isolation.

One might think that the highest possible isolation level (called Serializable) would solve all the issues. But that's not the case. To quote the author: "There are problems that have to be handled correctly at the application level. For example, exceptions caused by duplicate inserts or deletes must be handled properly (Section 3.2). Developers should not assume that a strong isolation level is a panacea that addresses all concurrency-related issues." An example of this would be an inappropriate error and exception handling that is a root cause of 10 issues form the paper.

Proposal of what ACID systems should do: Snapshot Isolation

When using a low isolation level, developers often find themselves investing time in caching and resolving non-critical bugs, which may not directly impact the business. On the other hand, even the most robust isolation level cannot completely eliminate all potential issues while still maintaining optimal performance.

All taken into account, there is an argument that using Snapshot Isolation as a default in database systems can provide developers with certain guarantees out of the box and reduce the overhead of implementing custom concurrency control mechanisms. Specifically, Snapshot Isolation guarantees that each transaction sees a consistent snapshot of the database as it existed at the start of the transaction, regardless of other concurrent transactions that may be modifying the same data. As explained in the previous section, the majority of the application issues could be avoided by just using the Snapshot Isolation level.

Again, it's important to emphasize that Snapshot Isolation is not always sufficient to ensure correctness in all database systems, especially with the presence of complex transactional dependencies and access patterns. For example, Snapshot Isolation can still result in lost updates and write-skew anomalies in certain cases. Furthermore, using Snapshot Isolation as the default isolation level may not be appropriate for all database workloads and may result in reduced performance or scalability for certain types of transactions.

Here are some commen types of tradeoffs with isolation levels:

Execution time (latency VS throughput)
Scalability
Memory usage
Development and SRE time spent on hunting and fixing application bugs caused by a weak isolation level

When considering the appropriate isolation level for a specific application, it is crucial to carefully evaluate the requirements. However, an essential question arises: What should be the sensible default choice that would cater to the needs of the majority of developers? Having a good default isolation level is significant as it simplifies the development process and ensures a solid foundation that works effectively for most developers.

So what does Memgraph support?

Memgraph default out-of-the-box isolation level is set to Snapshot Isolation. This ensures Memgraph's high data consistency and integrity. Although there is support for lower isolation levels, Read Committed and Read uncommitted, they introduce lower data consistency. You should not jump to the lowest possible level just to get some performance benefits, and this should be carefully considered. Hence we opted for Snapshot Isolation,as a default.

On top of this, Memgraph is implemented with multiversion concurrency control built in. Hence the performance in Snapshot isolation is best-in-class, but more on that in future blog posts. On top of isolation levels, for those the ACID transactions are not that important for and have analytical workloads, there is an analytical mode, which although does not guarantee any data consistency, it still provides lower memory usage and even faster performance.

Analyzing the Eurovision Song Contest With Graphs

Memgraph — Fri, 09 Jun 2023 15:04:08 +0000

Introduction

After the 2020 Eurovision Song Contest was canceled, we are excited about this year's finals more than ever, but geeks among us can't just wait around doing nothing!
Every year Eurovision brings us a fantastic mash-up of extravagant outfits, catchy songs, and bold dance moves. Underneath the surface of glitter and rainbows, there is a complicated story of geopolitics, rivalry, and tactics hidden in the collection of interconnected data.

We will try to untangle strategic voting, neighbor preferences, and political rivalries with the help of graph analytics.

Data model

Our dataset is quite simple. There are 51 Country nodes with the following properties: name, capital, lat, lng and flag. The properties lat and lng represent the coordinates of a country's capital city and will be important for visualizing the data on a map. The same is true for the flag property.

Each year's winner and runner-up are connected to Winner and RunnerUp nodes, respectively, with an edge IS that contains the properties year and points.

The Country nodes are connected with the BORDERS edge if they are neighboring countries.VOTE_JURY and VOTE_TELEVOTE edges have properties year and points. VOTE_JURY edges represent a number of points given by each country's professional jury, and VOTE_TELEVOTE edges represent the general public's vote that has been introduced in 1997. The voting system has been changed a few times so far but since 2009 the winner is determined by votes of the jury and public in an equal split.

Visualizing the Graph With Memgraph Lab

You can find the Eurovision song contest dataset in Memgraph Lab and on Memgraph Playground online.
Memgraph Lab is an integrated development environment used to import data, develop, debug and profile database queries and visualize query results. Just go to the Datasets tab and load the Eurovision voting results dataset.

Visualizing data in Memgraph is pretty simple. Memgraph Lab automatically detects nodes that have numerical lat and lng properties and displays them on a map.
You can style the map to your liking by using the Style editor in Memgraph Lab. To find out more about the style editor, take a look at this tutorial.
Each country node also has a flag property which is used to display an image. You can use the following styling script to display the flags on a map:

@NodeStyle {
  size: 100
  border-width: 2
  border-color: #ffffff
  shadow-color: #bab8bb
  shadow-size: 6
}

@NodeStyle Greater(Size(Labels(node)), 0) {
  label: Format(":{}", Join(Labels(node), " :"))
}

@NodeStyle HasLabel(node, "Country") {
  color: #dd2222
  color-hover: Darker(#dd2222)
  color-selected: #dd2222
}

@NodeStyle HasProperty(node, "name") {
  label: AsText(Property(node, "name"))
}

@EdgeStyle {
  width: 10
  label: Type(edge)
  color: gray
}

@NodeStyle {
  image-url: Property(node, "flag")
}

To get your graph, run the following Cypher query:

MATCH (c:Country) 
RETURN c;

If everything works properly, you should get a visualization similar to the one below.

Data analysis

Some countries have been more successful than others in producing good music (or keeping strong relationships with their neighbors). Let's check which country was the most victorious since 1975:

MATCH (c:Country)-[i:IS]-(w:Winner) 
RETURN c.name AS Country, count(i) AS Wins 
ORDER BY Wins DESC;

Interestingly, Ireland and Sweden stay on top despite our dataset being incomplete.

Coming in second on Eurovision is like nearly winning a medal at the Olympics. Let's check the biggest runner-ups of Eurovision:

MATCH (c:Country)-[i:IS]-(r:RunnerUp) 
RETURN c.name AS Country, count(i) AS RunnerUp
ORDER BY RunnerUp DESC;

It seems that the UK is the most notable runner-up in history.

But it's not all about winning! There is a special satisfaction in supporting your favorite song and casting a vote.

Some countries were better than others in predicting a winner, or is it just that some have better taste? Let's check which countries gave the most points to the winners throughout history:

MATCH (c:Country)-[vote:VOTE_JURY]->(: Country)-[i:IS]->(w:Winner)
WHERE vote.points = 12 AND vote.year = i.year
WITH c, COUNT(vote.points) AS points
RETURN c.name AS Country, points
ORDER BY points DESC 
LIMIT 10;

But it turns out that it isn't a matter of good or bad taste. When we take a look at the list of countries that didn't give any points to winners over the years:

MATCH (c:Country)-[v:VOTE_JURY]->(:Country)-[i:IS]->(w:Winner)
WHERE v.points = 0 AND v.year=i.year
RETURN c.name AS Country, count(v) AS WasWrong
ORDER BY WasWrong DESC
LIMIT 10;

The UK is again at the top of our list.

So what's Eurovision voting really about? Is having a catchy song really what it takes to win, or is there more to it? Maybe similar languages or familiar cultures play a role.
Let's check who fancies who by looking into which countries exchange the most points between each other:

MATCH (c1:Country)-[v1:VOTE_JURY]->(c2:Country), (c1: Country)<-[v2:VOTE_JURY]-(c2:Country)  
WHERE v1.year = v2.year
WITH c1.name AS Country1, c2.name AS Country2, SUM(v1.points) + sum(v2.points) AS Points
RETURN Country1, Points, Country2
ORDER BY Points DESC;

Relationships are symmetrical, so for every pair, there are two results. The number of points calculated is identical in both results, but countries switch places.
It's not surprising that most love is exchanged between neighbors!

Let's check how number of points from neighbors influenced the choice of winners over the years:

MATCH (w: Winner)-[i:IS]-(c1: Country)-[b:BORDERS]-(c2: Country), 
      (c1: Country)<-[v:VOTE_JURY]-(c2: Country)
WHERE i.year = v.year
WITH c1.name AS Country, sum(v.points) AS neighbour_points, i.points AS total_points
RETURN Country, round(1.0 * neighbour_points/total_points * 100) as Percent_of_neighbour_votes 
ORDER BY Percent_of_neighbour_votes DESC;

Looks like France has amazing relationships with its neighbors. But is it about quality or quantity?

First, let's find out who has the most neighbors participating in the competition:

MATCH (c1:Country)-[:BORDERS]-(c2:Country)
WITH DISTINCT c2.name AS Neighbor, c1.name as Country
RETURN Country, COUNT(Neighbor) AS Number_of_neighbors
ORDER BY Number_of_neighbors DESC LIMIT 10;

Seems like quantity wins in this case. France, Russia, and Germany have the largest number of neighbors and the most significant influence on their victories.

Let's finish with a specific example of how much geographical location influences the vote. Russia was the winner in 2008, but how many points would it get if voting for neighbors were forbidden?

MATCH (c1:Country)-[b:BORDERS]-(c2:Country {name: 'Russia'}),
      (c1:Country)-[v:VOTE_JURY {year: 2008}]->(c2:Country)
WITH sum(v.points) AS border_vote, c2.name AS name
MATCH (:Country)-[v:VOTE_JURY {year: 2008}]->(c:Country {name: 'Russia'})
RETURN c.name AS Country, sum(v.points) AS Points, sum(v.points)-border_vote AS Points_without_neighbours;

That's a significant difference!

Conclusion

There is a lot more to untangle and explore, but as the 2021 Grand Final is approaching, it's time to stop analyzing. After we find out this year's winner, we'll update the dataset and come back next year with new and more exciting analysis.
Until then, for more interesting examples of how to use graph analytics to analyze network data, visit Memgraph Playground and check out one of the available tutorials. You can also download Memgraph Lab and try it out locally.

Now relax and enjoy all things Eurovision!
"Good night, Europe! And Good morning, Australia!"

Use Prometheus to Monitor Memgraph’s Performance Metrics

Memgraph — Fri, 02 Jun 2023 12:44:19 +0000

Efficient database management lies at the core of any successful application or business operation. As data becomes increasingly critical, ensuring the performance, reliability, and security of your database infrastructure becomes paramount. This is where robust monitoring practices come into play, providing insights into the inner workings of your databases.

In this article, we will discuss how Memgraph integrated with Prometheus, a time-series database that enables other databases and applications to monitor and react to performance changes in the system just in time.

Why should you monitor your database performance?

Monitoring a graph database such as Memgraph with Prometheus brings several benefits to the table. One of the major advantages is that it provides real-time visibility into the performance and health of the database: Prometheus collects and analyzes metrics such as query response times, memory usage, disk I/O, and transaction rates, allowing you to monitor the database's performance at a granular level. That kind of visibility helps identify bottlenecks, optimize query execution, and ensure efficient resource allocation.

Secondly, Prometheus enables proactive issue detection and resolution. With alerting rules in place, you can receive timely notifications when there are anomalies or deviations from expected behavior in the graph database. Having access to an analogous alert mechanism allows you to quickly identify and address potential issues before they impact application performance or user experience.

Last but not least, monitoring a graph database with Prometheus facilitates capacity planning and scalability. By tracking resource utilization metrics over time, you can analyze trends and make informed decisions about scaling the database infrastructure to meet growing demands.

Key database performance metrics

Here are several metrics that would be useful to keep an eye out for:

RAM memory usage

RAM memory usage is a primary concern, given that Memgraph is an in-memory graph database. A potential issue with in-memory databases is when the query uses so much memory that it fails to execute. So having a monitoring tool in place, such as Prometheus, ensures on-time visibility of such cases, allowing you to promptly identify suboptimal queries and rewrite them to prevent memory surges.

Query execution latency

Although Cypher is the most widely used query language for graph databases, enterprise companies are not always quick to adopt it. Developers can occasionally struggle to write the most optimal and performant Cypher queries. When the query planner fails to identify the fastest route to execute a query, it is up to developers and database administrators to realize that queries can be written differently to decrease query execution latency and increase query throughput on Memgraph. A higher-than-usual latency performance metric in your dashboard can alert you in time to make the right decisions and preserve your system response times.

Snapshot recovery latency

For large datasets, snapshot recovery can take a lot of time when Memgraph is restarted, given the whole dataset needs to be loaded in memory. The 2.8 version of Memgraph introduces multithreaded recovery of snapshots, allowing users to recover the database with respect to the number of cores they have in the system for the most optimal behavior. By monitoring the average snapshot recovery time, you’ll be able to check the downtime of the Memgraph instance when it restarts and tweak performance flags.

Connections and transactions

Monitoring connections and transaction is useful while executing a heavy production load. If the database isolation level is high and the number of write-write conflicts goes up, the number of concurrent transactions can suggest how to ease the load on the database. Some of the solutions are dropping the isolation level, lowering the number of connections on the application level, and so on.

These are only some of the metrics that are exposed from Memgraph, but there are many more of them that reveal info about streams, triggers, indices, query execution operators, and a lot more.

How are metrics exposed in Memgraph to enable monitoring?

With the above-mentioned benefits of using Prometheus for monitoring in production systems, it made total sense to make an integration with it in order to allow enterprise companies to react fast and be alert at all times on what’s happening with Memgraph. With that in mind, we began our implementation efforts.

How to measure different parts of the system

When we started to design the monitoring feature, the first decision to make was to define all the things that could be measured. Generally, the more information we expose to the end metric collection system, the better. For a graph database, that useful information would include metrics about:

Disk usage
Memory usage
Connections
Transactions
Triggers
Indices
Messages between client and server
Query execution times
Snapshot creation/recovery times
Operator calls
Graph algorithm calls

There is one more thing to consider: the type of each metric we expose. For instance, RAM memory usage is certainly a value we can look up at any time and return. The number of graph algorithm calls is something that is incremented over time. Query execution latency also needs to be measured and aggregated into a meaningful and condensed distribution. For that purpose, we designed 3 types of metrics with which we can track information in the system:

Gauge — represents a single value in the system and can be set at any time (e.g., disk usage).
Counter — represents a value that can be incremented and decremented depending on the events in the system (e.g., the number of active transactions in the system).
Histogram — provides a distribution of measured variables over time (e.g., latency times).

The final point to take into account is providing enough information about histogram distribution. It is often a good practice to take a few percentiles from the distribution and collect them instead of providing the whole distribution when exposing metrics. With that in mind, we decided to take 50th, 90th, and 99th percentile values for both query latency and snapshot creation/recovery times.

Direct integration with Prometheus vs. proxying

With core metrics implemented, the next thing to take into consideration is how to expose the same metrics. One hard constraint that needs to be followed is that Prometheus uses a pull model, where it periodically scrapes metrics from configured targets.

Surely, one can integrate with Prometheus directly due to a long list of client libraries. However, this couples the integration and makes it unable for anyone else to use the metrics in their application. That is the case since Prometheus uses its own format of data to ingest it into the time-series database as fast as possible. Furthermore, some companies might have chosen other monitoring applications to observe their whole system (like Graphite, for example).

Prometheus again came to the rescue with the concept called exporters. A Prometheus exporter is a software component or module that collects and exposes metrics in a format that’s easily scraped and ingested by Prometheus. It serves as an intermediary between the target system or application and Prometheus, allowing Prometheus to gather relevant metrics for monitoring and analysis.

This makes it flexible for the application to use a simple HTTP endpoint and expose the metrics with a GET request, which we opted for in the end.

The image below portrays the end diagram when using a Prometheus exporter:

End result: Observable system at a glance

The finishing step is connecting Prometheus with a dashboarding tool, like Grafana, which is a feature-rich analytics and visualization platform that allows you to create and display interactive dashboards, charts, and graphs based on your exposed metrics from Memgraph. Here is a glimpse into what can be leveraged when Memgraph is integrated into this full E2E monitoring system.

All things considered, the integration of Prometheus with Memgraph enabled our customers and users, in general, to gain more confidence in system performance metrics at all times while Memgraph is being run. It also made it possible to track historical system metrics, allowing us to make decisions that can prevent issues and help us react proactively as opposed to reactively.

If you are curious to test the monitoring capabilities with Memgraph on your use case, be sure to visit our guide on exposing metrics as well as our Prometheus exporter to integrate everything from start to end.