Forem: Katarina Supe

When to Use a NoSQL Database

Katarina Supe — Fri, 21 Jul 2023 13:44:05 +0000

How we manage and process data changes rapidly over the years, and staying on track with the newest technology trends is important. Traditional relational database management systems, such as MySQL, Oracle, and SQL Server, have always been the first choice to maintain data integrity and for fast querying, but with the rise of big data, NoSQL (Not Only SQL) databases emerged and became more popular due to their speed, flexibility, and scalability. Now, the question that pops up naturally is when is the right time to use a NoSQL database?

This blog post will explore NoSQL databases and how they differ from traditional databases. Besides, you will learn when and why to utilize NoSQL databases. Whether you’re a developer, data engineer, or decision maker, this blog post will provide insights into modern databases that will come in handy to boost your projects.

Understanding NoSQL databases

NoSQL databases are non-relational databases with flexible schema designed for high performance at a massive scale. Unlike traditional relational databases, which use tables and predefined schemas, NoSQL databases use a variety of data models. There are 4 main types of NoSQL databases - document, graph, key-value, and column-oriented databases. NoSQL databases generally are well-suited for unstructured data, large-scale applications, and agile development processes. The most popular examples of NoSQL databases are MongoDB (document), Memgraph (graph), Redis (key-value store) and Apache HBase (column-oriented).

These types of NoSQL databases store data in their own unique ways offering pros and cons, which are presented below:

Advantages

Flexibility: If you’ve used an SQL database before, you are probably familiar with its strict schema. That can be a hassle with data rapidly changing. NoSQL databases, with their dynamic and flexible schemas, have a huge advantage in handling unstructured and semi-structured data, making an excellent choice for diverse data types.
Scalability: Data volumes can grow exponentially, and scalability becomes crucial, especially with the rise of big data. Many NoSQL databases offer horizontal scalability, allowing you to add more servers to increase the capacity. On the other hand, SQL databases usually scale only vertically.
Performance: Many NoSQL databases are optimized to deliver high performance, even when dealing with large data volumes or data streams.

Disadvantages

Maturity: SQL (standing for Structured Query Language) databases have been popular in the database world for a long time, and because of that, they offer robustness, tools and community that NoSQL databases have not managed to match yet. That does not mean that NoSQL will not overtake over the years, just like it happens often with modern technologies.
Complexity: The learning curve with NoSQL databases is steeper than with SQL databases, especially because there is no standard query language. This follows from the maturity and the fact that NoSQL databases are more complex to design, implement and manage. But, the results you gain by storing your data in a proper database might be worth it.
ACID compliance: Opposite to NoSQL databases, many SQL databases are usually ACID-compliant. However, as an exception, Memgraph is a NoSQL graph database that is ACID compliant. If you’re unsure what that means for your application, you are invited to check out our recent blog on ACID transactions and isolation levels.

5 reasons to choose a NoSQL database

Based on the above-mentioned advantages and disadvantages of NoSQL databases, there are various scenarios where NoSQL databases are the optimal solution for your project. Let’s take a look at 5 scenarios where NoSQL databases are great:

1. Handling large volumes of data at scale seamlessly

NoSQL databases can handle large amounts of data by spreading it across multiple servers. Hence, the NoSQL database is a perfect solution if you have large amounts of growing data. You don’t have to worry that you don’t have enough resources to scale vertically; instead, you should design your database system properly to get the most insights from your data.

2. Working easily with unstructured or semi-structured data

Predefined schemas can be good sometimes, but if you’ve always found that a hassle for your unstructured or semi-structured data which is often changing, NoSQL databases are the correct approach. They are built with flexible data schema, which speeds up the development process and lowers the efforts on database management. Problems with data types like XML, JSON and others have become issues of the past with NoSQL databases.

3. Enabling rapid development

Again, because of the NoSQL flexibility, your data model can rapidly change, meaning you can update your application on the fly without schema updates. Nowadays, development is fast, and iterations are quick, so removing the task of updating schemas constantly saves you valuable time that can be spent on even faster development.

4. High read/write speed

If you’re working on applications that require real-time data processing, you might be stuck with traditional databases that don’t offer the required speed. NoSQL databases are optimized for high-speed read and write operations and are ideal for chat, IoT, gaming, fraud detection, and similar applications.

5. Managing complex relationships

Complex data relationships, such as parent-child or many-to-many relationships, arise in relational databases when data from different tables is related or somehow interconnected. Querying it requires hopping from one table to another and joining it with slow and resource-intensive join operations. A graph database is a type of NoSQL database that handles highly-connected data especially well and is a good choice for social networks, recommendation engines, fraud detection, or any application where you need to traverse through interconnected data points.

Concluding thoughts

If you still don’t know whether you need an NoSQL or a relational database management system, we prepared another article on that topic that may speed up your decision process. Keep in mind that it doesn’t always have to be a fight, SQL vs NoSQL, you can surely consider a hybrid approach and get the best out of both SQL and NoSQL databases. But if you did find that your use case fits the description above, then NoSQL is a database for you! We are always open to discussions on this and other topics in the database scene, so join the conversation on our Discord server to be a part of the community.

Memgraph Community Call: Querying Memgraph through an LLM

Katarina Supe — Fri, 30 Jun 2023 16:27:53 +0000

Memgraph has organized its fair share of community calls to bring its community closer together and spread knowledge about important topics related to the graph database world. Attendees had the opportunity to meet the Memgraph team and talk with them about new features or their day-to-day responsibilities. They also had the chance to take over the stage and talk about the projects they worked on.

We are looking forward to many calls like this, and it proved to be great to have an open channel of communication directly with the Memgraph community to hear the most honest feedback about the product and experiences with building their products using graph databases.

In case you missed our community calls and want to catch up before the next one, head over to Memgraph’s YouTube channel.

Memgraph entering the world of LLMs

On July 7th, at 6 PM CEST, Memgraph is organizing the next community call on Querying Memgraph through an LLM. Our guest speaker is Brett Brewer, who has broad experience as a chemical engineer in industrial process automation and who has been a VP at Microsoft throughout his 25-year long career there. Now retired, he found a new way to have fun - he jumped into the AI rabbit hole to expand his knowledge and contribute where he can. That’s how he connected with Memgraph and gave us an idea to integrate with LangChain. That was not an easy task, but not to spoil too much, join our community call to hear and talk all about that journey!

Hang out with the community

To wrap it up, if you want to stay in touch with the Memgraph community, set up a reminder for this and similar events – join our Discord server! We try our best to keep the community engaged, questions answered, and every community member listened to.

Once again, here are the community call event details:

Topic: Querying Memgraph through an LLM

Speaker: Brett Brewer

Date and time: Friday, July 7th, 6 PM CEST

Zoom link: https://us02web.zoom.us/j/84734174129?pwd=aGtkc0UwVm10L3Qva2VveU01U1ozdz09

Hope to see you there 👋😀

Understanding Cosine Similarity in Python with Scikit-Learn

Katarina Supe — Mon, 12 Jun 2023 14:47:48 +0000

Cosine similarity proved useful in many different areas, such as in machine learning applications, natural language processing, and information retrieval. After reading this article, you will know precisely what cosine similarity is, how to run it with Python using the scikit-learn library (also known as sklearn), and when to use it. You’ll also learn how cosine similarity is related to graph databases, exploring the quickest way to utilize it.

Cosine similarity algorithm: Deep dive

Cosine similarity is a measure of similarity between two non-zero vectors of an inner product space based on the cosine of the angle between them, resulting in a value between -1 and 1. The value -1 means that the vectors are opposite, 0 represents orthogonal vectors, and value 1 signifies similar vectors.

The cosine similarity formula

To compute the cosine similarity between vectors A and B, you can use the following formula:

Here is the angle between the vectors, AB is the dot product between A and B, while A and B are the magnitudes, or lengths, of vectors A and B, respectively. The dot product can be expressed as a sum of the product of all vector dimensions and magnitudes as:

So the cosine similarity formula is equivalent to:

The cosine similarity is often used in text analytics to compare documents and determine if they’re similar and how much. In that case, documents must be represented as a vector, where a unique word is a dimension and the frequency or weight of that unique word in the document represents the value of that specific dimension. After the transformation of documents to vectors is done, comparison using cosine similarity is relatively straightforward — we measure the cosine of the angle between their vectors. If the angle between vectors (documents) is small, then the cosine of the angle is high, and hence, documents are similar. Opposite to that, if the angle between vectors (documents) is large, then the cosine of the angle is low, resulting in opposite documents (not similar). Cosine similarity considers the orientation of the vectors, but it does not take their magnitudes into account. In the previous example, this means that even documents of totally different lengths can be considered similar if they are related to the same topic.

Exploring Python libraries for cosine similarity

As explained in the previous chapter, cosine similarity can be easily calculated using the mathematical formula. But what if the data you have becomes too large and you want to calculate the similarities fast? The most popular programming language used for such tasks is definitely Python and its flexibility is partly due to its extensive range of libraries. For calculating cosine similarity, the most popular ones are:

NumPy: the fundamental package for scientific computing in Python, which has functions for dot product and vector magnitude, both necessary for the cosine similarity formula.
SciPy: a library used for scientific and technical computing. It has a function that can calculate the cosine distance, which equals 1 minus the cosine similarity.
Scikit-learn: offers simple and efficient tools for predictive data analysis and has a function to directly and efficiently compute cosine similarity.

From the above-mentioned libraries, only scikit-learn directly calculates the cosine similarity between two vectors or matrices, making it an excellent tool for data analysts and machine learning enthusiasts. It provides sklearn.metrics.pairwise.cosine_similarity function to do that, and we will show how it works on an example

How to calculate cosine similarity

To demonstrate how to calculate cosine similarity in a simple example, let’s take the descriptions of the most popular social networks from Wikipedia and compare them. The library used for calculating cosine similarity is scikit-learn, as mentioned in the previous section since it calculates cosine similarity directly with the help of sklearn.metrics.pairwise.cosine_similarity function. Besides that function, CountVectorizer from scikit-learn is used to convert the collection of text documents into a matrix of token counts. CountVectorizer has attribute stop_words which can be used to avoid counting most common words in English text, such as “and,” “the,” and “him,” which are presumed to be uninformative in representing the content of the text and which should be removed to avoid them being used in measuring similarity between two texts. Sometimes, these words are good to have to compare someone’s writing style or personality. Here is the code used to calculate the similarity between Twitter, Facebook, TikTok, and Instagram:

from sklearn.feature_extraction.text import CountVectorizer
import pandas as pd
from sklearn.metrics.pairwise import cosine_similarity


twitter = """
Twitter is an online social media and social networking service owned and operated by American company X Corp.,
the legal successor of Twitter, Inc. Twitter users outside the United States are legally served by the Ireland-based
Twitter International Unlimited Company, which makes these users subject to Irish and European Union data protection laws.
On Twitter users post texts, photos and videos known as 'tweets'. Registered users can tweet, like, 'retweet' tweets,
and direct message (DM) other registered users, while unregistered users only have the ability to view public tweets.
Users interact with Twitter through browser or mobile frontend software, or programmatically via its APIs.
"""
facebook = """
Facebook is an online social media and social networking service owned by American technology giant Meta Platforms.
Created in 2004 by Mark Zuckerberg with fellow Harvard College students and roommates Eduardo Saverin, Andrew McCollum,
Dustin Moskovitz, and Chris Hughes, its name derives from the face book directories often given to American university students.
Membership was initially limited to only Harvard students, gradually expanding to other North American universities and,
since 2006, anyone over 13 years old. As of December 2022, Facebook claimed 2.96 billion monthly active users, and ranked third
worldwide among the most visited websites. It was the most downloaded mobile app of the 2010s. Facebook can be accessed from devices
with Internet connectivity, such as personal computers, tablets and smartphones. After registering, users can create a profile
revealing information about themselves. They can post text, photos and multimedia which are shared with any other users who have
agreed to be their friend' or, with different privacy settings, publicly. Users can also communicate directly with each other with
Messenger, join common-interest groups, and receive notifications on the activities of their Facebook friends and the pages they follow.
"""
tiktok = """
TikTok, and its Chinese counterpart Douyin (Chinese: 抖音; pinyin: Dǒuyīn), is a short-form video hosting service owned by ByteDance.
It hosts user-submitted videos, which can range in duration from 3 seconds to 10 minutes. Since their launches, TikTok and Douyin have
gained global popularity.[6][7] In October 2020, TikTok surpassed 2 billion mobile downloads worldwide. Morning Consult named TikTok the
third-fastest growing brand of 2020, after Zoom and Peacock. Cloudflare ranked TikTok the most popular website of 2021,
surpassing google.com.
"""
instagram = """
Instagram is a photo and video sharing social networking service owned by American company Meta Platforms. The app allows users to
upload media that can be edited with filters and organized by hashtags and geographical tagging. Posts can be shared publicly or
with preapproved followers. Users can browse other users' content by tag and location, view trending content, like photos, and follow
other users to add their content to a personal feed. Instagram was originally distinguished by allowing content to be framed only in a
square (1:1) aspect ratio of 640 pixels to match the display width of the iPhone at the time. In 2015, this restriction was eased with
an increase to 1080 pixels. It also added messaging features, the ability to include multiple images or videos in a single post, and a
Stories feature—similar to its main competitor Snapchat—which allowed users to post their content to a sequential feed, with each post
accessible to others for 24 hours. As of January 2019, Stories is used by 500 million people daily.
"""




documents = [twitter, facebook, tiktok, instagram]
count_vectorizer = CountVectorizer(stop_words="english")
count_vectorizer = CountVectorizer()
sparse_matrix = count_vectorizer.fit_transform(documents)


doc_term_matrix = sparse_matrix.todense()
df = pd.DataFrame(
   doc_term_matrix,
   columns=count_vectorizer.get_feature_names_out(),
   index=["twitter", "facebook", "tiktok", "instagram"],
)
print(df)
print(cosine_similarity(df, df))

Before getting to results, it’s good to think about the expected outcomes. Since the same company owns Facebook and Instagram and they’re mostly used to share photos and videos, we can suppose that they are more similar to each other than to other social networks. Twitter might be more similar to Facebook and Instagram than to TikTok, just because of the format of posting, which is quite specific to TikTok. But enough guessing, let’s take a look:

First, there is a matrix of word frequency in all four texts. Cosine similarity is calculated based on that matrix, and, in the end, is also turned into a matrix. Here is how to read the results:

The first column represents the description of the first social network — Twitter. Every row in the first column represents cosine similarity to all four other social networks. In this case, it means that the cosine similarity of Twitter itself is 1 and that is true since those two texts are the same. That is also the reason why the whole diagonal equals 1 — it represents the cosine similarity of every social network between themselves. The next row in the first column is the cosine similarity of Twitter and Facebook vectors, while the third row is that of Twitter and TikTok vectors. In the end, we have a similarity between Twitter and Instagram vectors. Notice how the similarities of Twitter to Instagram and Facebook are quite close, leading us to the conclusion that Instagram and Facebook descriptions are similar. To check that, see the second column which shows the similarities to the Facebook vector. It goes from Twitter, across Facebook, and TikTok up to the Instagram description. Also notice how the similarity between Facebook and Instagram is quite high, 0.6, while the similarity between Facebook and TikTok is a bit smaller, 0.3.
Once all columns are read and well interpreted, we can conclude that Facebook and Instagram descriptions (and hence potentially social networks) are the most similar ones. Also, Twitter is more similar to Facebook and Instagram than to TikTok.

Practical use cases of cosine similarity

Cosine similarity is used in various applications, mostly by data scientists, to perform tasks for machine learning, natural language processing, or similar projects. Their applications include:

Text analysis, which is applied to measure the similarity between documents and offers crucial functionality for search engines and information retrieval systems, as shown in the example.
Recommendation systems, to recommend similar items based on user preferences or to suggest similar users in social network applications. An example is to recommend the next page on product documentation based on the text similarity found.
Data clustering, which in machine learning acts as a metric to classify or cluster similar data points, and in that way, it helps make data-driven decisions.
Semantic similarity, which, when paired with word embedding techniques like Word2Vec, is used to determine the semantic similarity between words or documents.

Cosine similarity in graph databases

Graph databases are popular for how they represent the relationships between different entities, that is, nodes. These relationships are essential in obtaining information about the similarity between nodes in the database. Nodes can hold multi-dimensional data, like feature vectors or descriptions, and cosine similarity can provide an understanding of node relationships beyond mere structural connections.

Whether it's about identifying similar user profiles in a social network, detecting similar patterns in a communication network, or classifying nodes in a semantic network, cosine similarity contributes valuable insights. Combined with a powerful graph database system, such as Memgraph, it gives a better understanding of complex networks. Memgraph is an open-source in-memory graph database built to handle real-time use cases at an enterprise scale. Memgraph supports strongly-consistent ACID transactions and uses the standardized Cypher query language for structuring, manipulating, and exploring data.

To run cosine similarity in Memgraph, you need to install Memgraph with MAGE, Memgraph Advanced Graph Extensions, an open-source repository that contains graph algorithms and modules written by the team behind Memgraph and its users in the form of query modules. One of the query modules implemented is node_similarity. Within that module, procedures to calculate cosine similarity between two vectors and two sets of vectors can be found. Besides that, there are procedures to calculate the Jaccard and overlap similarity.

For example, let’s run the cosine_pairwise procedure from the node_similarity module on a simple example. First, run Memgraph with MAGE using the following Docker command:

docker run -it -p 7687:7687 -p 7444:7444 memgraph/memgraph-platform

`

The Docker image is called Memgraph Platform since it includes the visual user interface of Memgraph Lab, Memgraph database, and MAGE. Exploring data with Memgraph Lab within Docker container will be easier, rather than just using the command line tool or installing Memgraph Lab separately as a Desktop application. Next, load the dataset by running the following queries in the Memgraph Lab Query Execution tab:
cypher CREATE (b:Node {id: 0, score: [1.0, 1.0, 1.0]}); CREATE (b:Node {id: 1, score: [1.0, 1.0, 1.0]}); CREATE (b:Node {id: 2, score: [1.0, 1.0, 1.0]}); CREATE (b:Node {id: 3, score: [1.0, 1.0, 0.0]}); CREATE (b:Node {id: 4, score: [0.0, 1.0, 0.0]}); CREATE (b:Node {id: 5, score: [1.0, 0.0, 1.0]}); MERGE (a:Node {id: 0}) MERGE (b:Node {id: 2}) CREATE (a)-[:RELATION]->(b); MERGE (a:Node {id: 0}) MERGE (b:Node {id: 3}) CREATE (a)-[:RELATION]->(b); MERGE (a:Node {id: 0}) MERGE (b:Node {id: 4}) CREATE (a)-[:RELATION]->(b); MERGE (a:Node {id: 0}) MERGE (b:Node {id: 5}) CREATE (a)-[:RELATION]->(b); MERGE (a:Node {id: 1}) MERGE (b:Node {id: 0}) CREATE (a)-[:RELATION]->(b); MERGE (a:Node {id: 1}) MERGE (b:Node {id: 1}) CREATE (a)-[:RELATION]->(b); MERGE (a:Node {id: 1}) MERGE (b:Node {id: 2}) CREATE (a)-[:RELATION]->(b); MERGE (a:Node {id: 2}) MERGE (b:Node {id: 1}) CREATE (a)-[:RELATION]->(b); MERGE (a:Node {id: 2}) MERGE (b:Node {id: 4}) CREATE (a)-[:RELATION]->(b); MERGE (a:Node {id: 2}) MERGE (b:Node {id: 5}) CREATE (a)-[:RELATION]->(b); MERGE (a:Node {id: 4}) MERGE (b:Node {id: 0}) CREATE (a)-[:RELATION]->(b); MERGE (a:Node {id: 4}) MERGE (b:Node {id: 1}) CREATE (a)-[:RELATION]->(b); MERGE (a:Node {id: 5}) MERGE (b:Node {id: 2}) CREATE (a)-[:RELATION]->(b); MERGE (a:Node {id: 5}) MERGE (b:Node {id: 3}) CREATE (a)-[:RELATION]->(b);

If you’re not sure how to do that, refer to the Memgraph Lab user manual.
Every node has a score property, a vector of zeros and ones. Execute the following query to run the cosine_pairwise procedure:

cypher MATCH (m) WHERE m.id < 3 WITH COLLECT(m) AS nodes1 MATCH (n) WHERE n.id > 2 WITH COLLECT(n) AS nodes2, nodes1 CALL node_similarity.cosine_pairwise("score", nodes1, nodes2) YIELD node1, node2, similarity AS cosine_similarity RETURN node1, node2, cosine_similarity;`

The above query split all nodes into two lists of equal size and used them as arguments of the cosine_pairwise procedure. Here is the result:

The obtained result calculated cosine similarity based on score properties on nodes. We can see that node 0 is as similar to node 3 as node 2 is similar to node 5. Still similar, but a bit less, is node 1 to node 4.
One of the beauties of graph databases is that there are other popular methods to calculate similarities and, in that way, create recommendation systems. One example of using popular node2vec and link prediction algorithms can be found in our blog post on how to create a docs recommendation system

.

Another interesting example of link prediction with node2vec can be found in how to perform paper recommendations in a physics collaboration network.

Important takeaways

Intuitive interpretation and versatility of the cosine similarity algorithm have found their way into various applications, spanning from text analysis and recommendation systems to complex graph databases. The algorithm's ability to capture the orientation of vectors makes it a robust measure of similarity, especially in high-dimensional spaces.

Python, being a programming language with an abundant set of libraries like scikit-learn, provides data analysts and machine learning enthusiasts with the tools they need to compute cosine similarity with ease. As we increasingly move towards data-driven decision-making, the ability to measure and interpret similarity becomes more crucial than ever. Undoubtedly, cosine similarity will continue to be a valuable tool in data science.

If you want to dive into the world of graph analytics to discover even more about data analysis, sign up for Memgraph’s Graph analytics with Python webinar. For any questions or just a chat about Python, data science, and graphs, join the conversation on our Discord server.

How to Use GQLAlchemy Query Builder?

Katarina Supe — Wed, 24 May 2023 13:59:18 +0000

GQLAlchemy query builder will help you build Cypher queries in no time! With
this blog post, you will learn how to use different query builder methods to
create, change, get, set, and remove data from Memgraph.

You can find the original Jupyter Notebook in our open-source GitHub repository.

Set everything up

If you want to try the code from this blog post, make sure to install
Docker and
GQLAlchemy. Docker is used because
Memgraph is a native Linux application and can't be installed on Windows and
macOS. After you install Docker, you can set up Memgraph by running:

docker run -it -p 7687:7687 -p 7444:7444 -p 3000:3000 memgraph/memgraph-platform:2.2.6

This command will start the download and after it finishes, run the Memgraph
container.

We will be using the GQLAlchemy object graph mapper (OGM) to connect to
Memgraph and execute Cypher queries easily. GQLAlchemy also serves as a
Python driver/client for Memgraph. You can install it using:

pip install gqlalchemy==1.2.0

Hint: You may need to install CMake before
installing GQLAlchemy.

from gqlalchemy import Memgraph

memgraph = Memgraph("127.0.0.1", 7687)

Make sure that Memgraph is empty before starting with anything else. Also, you
should drop all indexes, if there are any, since you'll be creating new ones.

memgraph.drop_database()
memgraph.drop_indexes()

Define graph schema

You will create Python classes that will represent the graph schema. This
way, all the objects that are returned from Memgraph will be of the correct type
if the class definition can be found. That is not a part of the query builder,
but it will help you work with the nodes and relationships.

from typing import Optional
from gqlalchemy import Node, Relationship, Field

class Movie(Node):
    id: int = Field(index=True, unique=True, exists=True, db=memgraph)
    title: Optional[str]

class User(Node):
    id: int = Field(index=True, unique=True, exists=True, db=memgraph)
    name: Optional[str]

class Rated(Relationship, type="RATED"):
    rating: float

Create nodes and relationships

To return all the variables from a query, just use the return_() method at the
end of your query. Don't forget to execute() each query. First, you can create
some users in the database:

from gqlalchemy import create


ron = next(create().node(labels="User", id=0, name="Ron", variable="ron").return_().execute())
john = next(create().node(labels="User", id=1, name="John", variable="john").return_().execute())
maria = next(create().node(labels="User", id=2, name="Maria", variable="maria").return_().execute())
jenny = next(create().node(labels="User", id=3, name="Jenny", variable="jenny").return_().execute())
ian = next(create().node(labels="User", id=4, name="Ian", variable="ian").return_().execute())

users = [ron, john, maria, jenny, ian]

for user in users:
    print(user)

Output:

{'ron': <User id=0 labels={'User'} properties={'id': 0, 'name': 'Ron'}>}
{'john': <User id=1 labels={'User'} properties={'id': 1, 'name': 'John'}>}
{'maria': <User id=2 labels={'User'} properties={'id': 2, 'name': 'Maria'}>}
{'jenny': <User id=3 labels={'User'} properties={'id': 3, 'name': 'Jenny'}>}
{'ian': <User id=4 labels={'User'} properties={'id': 4, 'name': 'Ian'}>}

Next, create a couple of movies:

lotr = next(create().node(labels="Movie", id=0, title="LOTR", variable="lotr").return_().execute())
home_alone = next(create().node(labels="Movie", id=1, title="Home Alone", variable="home_alone").return_().execute())
star_wars = next(create().node(labels="Movie", id=2, title="Star Wars", variable="star_wars").return_().execute())
hobbit = next(create().node(labels="Movie", id=3, title="Hobbit", variable="hobbit").return_().execute())
matrix = next(create().node(labels="Movie", id=4, title="Matrix", variable="matrix").return_().execute())

movies = [lotr, home_alone, star_wars, hobbit, matrix]
for movie in movies:
    print(movie)

Output:

{'lotr': <Movie id=5 labels={'Movie'} properties={'id': 0, 'title': 'LOTR'}>}
{'home_alone': <Movie id=6 labels={'Movie'} properties={'id': 1, 'title': 'Home Alone'}>}
{'star_wars': <Movie id=7 labels={'Movie'} properties={'id': 2, 'title': 'Star Wars'}>}
{'hobbit': <Movie id=8 labels={'Movie'} properties={'id': 3, 'title': 'Hobbit'}>}
{'matrix': <Movie id=9 labels={'Movie'} properties={'id': 4, 'title': 'Matrix'}>}

Now you need to connect users and movies with the RATED relationship:

from gqlalchemy import match

match().node(labels="User", id=0, variable="u").match().node(labels="Movie", id=0, variable="m").create().node(variable="u").to(edge_label="RATED", rating=5.0).node(variable="m").execute()
match().node(labels="User", id=0, variable="u").match().node(labels="Movie", id=1, variable="m").create().node(variable="u").to(edge_label="RATED", rating=3.2).node(variable="m").execute()
match().node(labels="User", id=0, variable="u").match().node(labels="Movie", id=2, variable="m").create().node(variable="u").to(edge_label="RATED", rating=4.3).node(variable="m").execute()
match().node(labels="User", id=1, variable="u").match().node(labels="Movie", id=0, variable="m").create().node(variable="u").to(edge_label="RATED", rating=5.0).node(variable="m").execute()
match().node(labels="User", id=2, variable="u").match().node(labels="Movie", id=0, variable="m").create().node(variable="u").to(edge_label="RATED", rating=4.9).node(variable="m").execute()
match().node(labels="User", id=3, variable="u").match().node(labels="Movie", id=3, variable="m").create().node(variable="u").to(edge_label="RATED", rating=2.3).node(variable="m").execute()
match().node(labels="User", id=3, variable="u").match().node(labels="Movie", id=4, variable="m").create().node(variable="u").to(edge_label="RATED", rating=4.8).node(variable="m").execute()
match().node(labels="User", id=4, variable="u").match().node(labels="Movie", id=0, variable="m").create().node(variable="u").to(edge_label="RATED", rating=4.0).node(variable="m").execute()

To check whether the data is correctly imported into Memgraph, you can open
Memgraph Lab on localhost:3000.
In the query editor, run:

MATCH (u)-[r]->(m)
RETURN u, r, m;

You'll see the imported data visualized as a graph below:

Filter results

You can filter data by comparing the property of a graph object to a value
(a literal). To return only a subset of variables from a query, specify them in
the return() method. Below you can see how to compare name and title
properties of nodes to a string value.

results = list(match()
            .node(labels="User", variable="u")
            .to(edge_label="RATED")
            .node(labels="Movie", variable="m")
            .where(item="u.name", operator="=", literal="Maria")
            .and_where(item="m.title", operator="=", literal="LOTR")
            .return_({"u.id": "user_id", "m.id": "movie_id"})
            .execute()
        )

for result in results:
    print("Maria's id:", result["user_id"])
    print("LOTR id:", result["movie_id"])

Output:

Maria's id: 2
LOTR id: 0

You filtered the data by finding a user with name Maria, who rated the movie
with title LOTR.

You can also negate the expression in WHERE clause, using the where_not()
method. Similarly, you have and_not_where(), or_not_where() and
xor_not_where() methods.

results = list(match()
            .node(labels="User", variable="u")
            .to(edge_label="RATED")
            .node(labels="Movie", variable="m")
            .where_not(item="u.name", operator="=", literal="Maria")
            .and_where(item="m.title", operator="=", literal="LOTR")
            .return_({"u.name": "user_name"})
            .execute()
        )

for result in results:
    print(result["user_name"])

Output:

Ron
John
Ian

What happened above? You filtered the data by finding the names of all users who
are not Maria and who rated the movie LOTR.

You can also filter data by comparing properties of graph objects:

results = list(match()
            .node(labels="User", name="Maria", variable="u")
            .match()
            .node(labels="User", variable="v")
            .where(item="u.name", operator="!=", expression="v.name")
            .return_({"v.name": "user_name"})
            .execute()
        )

for result in results:
    print(result["user_name"])

Output:

Ron
John
Jenny
Ian

You found all the users that don't have the property name set to Maria.

Nodes can also be filtered by their label using the where() method instead
of specifying it directly in the node() method of the query builder. You have
to use expression as the third keyword argument since you want to escape the
quotes surrounding the label in the Cypher clause.

results = list(match()
            .node(variable="u")
            .where(item="u", operator=":", expression="Movie")
            .return_()
            .execute()
        )

for result in results:
    print(result)

Output:

{'u': <Movie id=5 labels={'Movie'} properties={'id': 0, 'title': 'LOTR'}>}
{'u': <Movie id=6 labels={'Movie'} properties={'id': 1, 'title': 'Home Alone'}>}
{'u': <Movie id=7 labels={'Movie'} properties={'id': 2, 'title': 'Star Wars'}>}
{'u': <Movie id=8 labels={'Movie'} properties={'id': 3, 'title': 'Hobbit'}>}
{'u': <Movie id=9 labels={'Movie'} properties={'id': 4, 'title': 'Matrix'}>}

Above you filtered all nodes in the database by the Movie label.

Call procedures

Let's see which movie is the most important in the network. You can suppose that
the most important movie will be the one that the largest number of users rated.
To order the returned results, use the order_by() method after the return_()
method.

from gqlalchemy import call
from gqlalchemy.query_builder import Order

results = list(call("pagerank.get")
            .yield_()
            .with_({"node": "node", "rank": "rank"})
            .where(item="node", operator=":", expression="Movie")
            .return_({"node.title": "movie_title", "rank": "rank"})
            .order_by(properties=("rank", Order.DESC))
            .execute()
        )

for result in results:
    print(result["movie_title"], result["rank"])

Output:

LOTR 0.26900584795321636
Hobbit 0.1
Matrix 0.1
Home Alone 0.09005847953216374
Star Wars 0.09005847953216374

Delete and remove objects

To delete a node from the database, use the delete() method:

match().node(labels="User", id=4, variable="u").delete(variable_expressions=["u"], detach=True).execute()

You deleted Ian from the database and the relationships he was connected with.

To delete a relationship from the database, again use the delete() method:

match().node(labels="User", id=0).to(edge_label="RATED", variable="r").node(labels="Movie", id=0).delete(variable_expressions=["r"]).execute()

Above you deleted the relationship between Ron and LOTR, that is, it seems like
Ron hasn't rated LOTR.

To remove a property from the database, use the remove() method:

results = list(match()
            .node(labels="User", name="Maria", variable="u")
            .remove(items=["u.name"])
            .return_()
            .execute()
        )

for result in results:
    print(result)

Output:

{'u': <User id=2 labels={'User'} properties={'id': 2, 'name': None}>}

Now Maria's name is set to None, so you can only recognize her by the id
in the database.

What's next?

Now it's time for you to use Memgraph on a graph problem!

You can always check out Memgraph Playground
for some cool use cases and examples. If you need some help with GQLAlchemy,
join our Discord server and have a chat with
us.

Graph Database vs Relational Database

Katarina Supe — Mon, 15 May 2023 07:09:51 +0000

Choosing the right type of database to store data in is a crucial decision that can impact the success of your personal or company project. Whether you need to decide between using a graph database or a relational database or want to learn about the differences between those two, you've come to the right place.

In the process of making the best database choice, developers are typically asking themselves several key questions: "Should I choose an open source over a commercial database? How easy is it to integrate the database with my application or manage and maintain the database? Or which database offers the best performance for my particular use case?"

To answer these questions, you first need to know what kind of data you are working with, what kind of queries you need to run, how much data you will be storing, how important performance is, and, after all, what your team's expertise is. This blog post will provide answers to those key questions along with discussing valuable examples to make your decision-making around graph database vs relational as easy as possible.

How is a graph database different from a relational database?

A graph database sees the world as we do! It is a type of database that stores data in the form of nodes connected with relationships.

Graph databases that are based on a labeled-property graph data model consist of four components:

Nodes - the main entities in a graph. They are also sometimes referred to as vertices or points.
Relationships - the connections between those entities. Sometimes they are referred to as edges or links.
Labels - attributes that group similar nodes together.
Properties - key/value pairs stored within nodes or relationships.

Nodes and relationships are fundamental components that are used. Labels and properties are often included because of the added functionalities they offer.

Graph databases mostly use Cypher query language since it is widely adopted, fully specified, and is an open query language for property graph databases. There are other graph query languages out there, but Cypher is the most popular one as it provides an intuitive way of working with property graphs.

Relational databases store data in tables consisting of rows and columns. Every row is a record, and every column is an attribute of that record. Relational databases are based on a relational data model that requires a predefined and carefully modeled set of tables. Multiple data tables can be related to each other using primary and foreign keys. Primary keys uniquely identify each row in a table, while foreign keys link rows in one table to rows in another table. Relational databases use Structured Query Language (SQL) to query, manage and manipulate data. Relational databases are widely used in applications such as e-commerce, banking, and inventory management systems.

The image below shows the difference between the graph model and the relational model for the order management dataset.

In the image below, there is another example, a social network graph data model, where the nodes represent people in different social groups and their connections. Every person is represented with a node that’s labeled as Person. These nodes contain the property name, gender, location, and email. The relationships between people in this network are of the type FRIENDS_WITH and contain a yearsOfFriendship property to specify the duration of the friendship connection. Each person is assigned a location through LIVES_IN relationships with nodes labeled Location.

Each person is connected to other people through friendships, and to model this relationship in a relational database, a table needs to be created. If there were different kinds of connections schema would have to be changed accordingly. A relational database isn’t suited for this specific use case because the focus isn’t on the data but on the relationships within it.

While this is a straightforward example, it concisely demonstrates the power and benefits of using a graph database. In relational databases, a new column must be added for each additional attribute, while with the flexible graph database schema, new properties can be added on the fly.

Cypher vs SQL query language

Cypher is a powerful and flexible query language specifically designed for working with graph databases. Its syntax and functionality make it well-suited for complex queries involving highly connected data - deep hierarchical relationships between data, such as parent-child relationships or many-to-many relationships between different tables. You can think of Cypher as mapping English language sentence structure to patterns in a graph. In most cases, the nouns are nodes of the graph, the verbs are the relationships in the graph, and the adjectives and adverbs are the properties.

SQL is a programming language used to manage and manipulate relational databases. It is used to create, modify, and query databases and their tables, as well as to insert, update, and delete data in the tables. It is a standard language used by most relational database management systems, including MySQL, Oracle, Microsoft SQL Server, PostgreSQL, and SQLite.

The difference between Cypher and SQL can be shown through an example dataset containing the genealogy of some ancient French kings around the 8th century AD from the MySQL Blog Archive. The easiest way to represent the dataset is with a graph:

With breadth-first search and depth-first search, it is possible to obtain the descendants in a specific order. Here are the SQL and Cypher queries to find all descendants of Thurimbert:

To run the breadth-first search, that is, find all children, direct descendants first, grouped, use the following queries:

On the other hand, for the depth-first search, use the following queries:

The above dataset is naturally stored as hierarchical or graph data. The great thing about Cypher queries is that they tend to focus on the relationships between nodes in the graph, while SQL queries tend to focus on the columns and rows in tables. Also, path traversals such as breadth-first search and depth-first search are basic queries in Cypher. That’s why these kinds of queries are easier for developers to understand and write in Cypher. Besides that, on a larger scale of data, Cypher queries are more performant than SQL recursive calls for these kinds of use cases.

Overall, the main differences between Cypher and SQL stem from the different data modeling approaches and optimizations of the graph and relational databases. Cypher is designed to work with graph data and is optimized for queries that involve complex relationships. In contrast, SQL is designed to work with tabular data and is optimized for simpler queries. Compared to SQL, which can have a steep learning curve, Cypher is often easier for developers who are new to graph databases. And it provides an expressive syntax for working with graph data, allowing developers to create complex queries with fewer lines of code.

When to use a graph database?

There are always two sides to every story and graph databases aren’t a perfect solution for every problem. Of course, for a bunch of use cases, you should stick with relational databases or maybe search for alternatives other than graph databases.

To break it down for you, here are three simple questions you can ask yourself to decide if using a graph database has any benefits for your particular use case.

1. Is my data highly connected?

Graph solutions are focused on highly connected data with an intrinsic need for relationship analysis. In relational databases, complex data relationships arise when data from different tables is related or somehow interconnected. Because data is spread across multiple tables, querying it requires hopping from one table to another and joining it with slow and resource-intensive join operations. The complexity of join operations can increase exponentially as the number of tables increases and as the links between various tables are no longer neatly structured following a clearly set pattern. It is no longer sufficient to join just two or three tables: rather hop through more than seven tables to find the correct link between the data and gain valuable analytics. In that case, graph databases are the solution to go with.

If the connections within the data are not the primary focus and the data is transactional, then a graph database is probably not the best fit. Sometimes it’s just important to store data and complex analysis isn’t needed.

In the social network example, without the relationships between people stored, you get a sparsely connected graph. Yes, a number of simpler graphs would remain because of the connections between nodes Person and Location, but this degree of connectedness and the consistency of the data structure is better suited for a relational database.

2. Is retrieving the data more important to me than storing it?

Graph databases are optimized to retrieve data and they offer the power of graph analytics that can help discover hidden patterns in the data that were previously unseen with relational databases.

In the social network example, if the data is stored only for the sake of logging interactions without analyzing them later on, then a graph database isn’t particularly helpful. However, if there are numerous connections within the data being stored, then a graph might be worth considering. In the mentioned example, you can discover the most important or similar users, predict new connections, and group similar users into communities using PageRank, betweenness centrality, node2vec, or similar graph algorithms.

3. Does my data model change often?

Graph databases are best suited for data models that are inconsistent and demand frequent changes. Because graph databases are more about the data itself than the schema structure, they allow a degree of flexibility.

On the other hand, there are benefits to having a predefined and consistent table that’s easy to understand. Developers are comfortable and used to relational databases and that fact cannot be downplayed. For example, to store personal information such as names, dates of birth, and locations, without often creating new fields or changing data types, relational databases are the go-to solution.

A graph database could be useful if:

Additional attributes will be added at some point
Not all entities will have all the attributes in the table
The attribute types are not strictly defined

To better understand the use cases best suited for graph databases, here are a couple of examples:

Insurance fraud detection

Graph databases help detect anomalies and make the information system more responsive. They allow for unexpected changes in the data structure and enable insurance companies to successfully expand their business as they tap into the insights provided by the recommendations engines (built on top of the data). A graph database is much better in this use case compared to a relational database because graphs perform faster with highly interconnected data. The graph model is flexible. Tight connections and patterns between entities can help design recommendation and fraud detection systems in insurance.

Network resource optimization

With a graph database, you can manage networks in the supply chain, chemical engineering, cloud computing, energy systems, and many more industries. Additionally, you can do quantitative analysis, and discover dependencies and critical points in your network so you can prevent obstructions and increase the network's efficiency. With relational databases, visualizing the topology takes around a third of the processing time, while in graph databases, data is automatically joined when it’s being stored, making sure graph traversing is performed in constant time.

Identity and access management

Graph databases have dynamic algorithms that analyze only a subset of the graph where changes happen and, in that way, avoid security crises. Besides that, graphs offer fast traversals with BFS or DFS hops due to index-free adjacency. Graphs represent relationships in a semantically rich way, allowing accurate views of access rights and simplifying their management, which is impossible with a hierarchical structure.

When not to use a graph database?

A graph database is only sometimes the perfect solution since it is not well-suited for traversing the whole graph often. While it’s possible to run such queries, other storage solutions may be more optimized for bulk scans. If the majority of the queries in the social network example included searches by property values over the entire network, then a graph database wouldn’t be the right fit.

Databases are often used to look up information stored in key/value pairs. For example, if the sole purpose of the database is storing the user’s personal information and retrieving it by name or ID, then refrain from using a graph. But if other entities were involved (visited locations for example), and a large number of connections is required to map them to users, then a graph database could bring performance benefits. If most queries return a single node via a simple identifier (key), then skip graph databases. If the entities in the model have very large attributes like BLOBs, CLOBs, and long texts — graph databases aren’t the best solution.

While those objects can be stored as nodes and linked to other nodes to utilize the power of traversing relationships, sometimes storing them directly with the entities they are connected to makes more sense.

Again, back to our favorite social network example, if each person had a long biography that needed to be included in the same database, a graph wouldn’t be the answer. However, if you needed to connect these biographies to other entities in the database (for example, people that are mentioned in them), then the strengths of a graph database would outweigh the limitations.

Wrap up

To conclude, both relational and graph databases are powerful tools. Graph databases handle fast-changing interconnected data well, while relational databases shine in managing structured data efficiently and effectively.

If you are still unsure if a graph database is the right choice for your project, drop us a message in our Discord community. We’ll be happy to address any doubts! Besides that, you can explore the most common graph use cases and maybe find the solution for your project on our site. If you are considering different graph database stores, check Memgraph’s latest benchmarks to see how it compares to the other graph solutions.

Learn Graph Analytics With Python

Katarina Supe — Tue, 13 Dec 2022 14:07:24 +0000

Graphs are a way to represent a network or a collection of interconnected objects formally. There are many powerful tools out there to explore that kind of network by applying graph algorithms. But sometimes it’s hard to keep track of them!

We have created a brand new course to get you familiar with the graph world. The course is designed for Python developers who want to explore tools for network analysis and see how it’s done hands-on. You will learn what graph algorithms are and hear more about the most popular ones.

If you are not an experienced Python developer, don’t worry - this course will teach you the basics of graph analytics, and you’ll be able to apply that knowledge whenever you become more familiar with Python.

The tools which we explored in detail are NetworkX, a Python package for the creation, manipulation and study of the structure dynamics and functions of complex networks. Another tool is called Memgraph, a graph database, through which we’ll explain the benefits of using graph storage. If all of the above sounds intriguing and you don’t want any spoilers, skip this blog and sign up for the free course!

Hello graph world

Graphs can come in many forms, depending on the tool which implements them. Also, there are different types of graphs depending on the type of data stored inside such objects. You’ll learn the basics of the graph world and hear about graph databases.

To get insights from the graphs, you need to perform graph analytics. It is essential to understand what kinds of algorithms are out there and when to use them. There are many different graph use cases, and we will discuss the most common ones.

All of that is simple enough in theory, but when you, as a Python developer, actually need to get the most out of your data, it is important to find a tool that is powerful but simple to use. One of the most famous such tools is called NetworkX.

NetworkX - a Python graph analytics tool

Through this course, you will have an opportunity to see many code examples explaining NetworkX features. You will see how it’s easy to create a simple graph from scratch, explore it and visualize it.

NetworkX comes with included graph algorithms, and it is a tool that many researchers use. Besides all of its advantages, we are going to investigate the biggest challenges when working with big or dynamic graphs regarding analysis speed, memory consumption and quick visualizations.

Memgraph - a Python-friendly graph database

Memgraph offers data persistency, large-scale data analytics and visualizations, so it is an excellent NetworkX alternative. You may ask yourself - why a graph database? We are going to answer that and many other questions in this section of the course.

But you don’t have to give up on NetworkX to use a graph database totally. With the on hands part of the course, you will see how to transform a NetworkX graph into a Memgraph graph. Besides that, Memgraph holds a set of NetworkX algorithms, along with its own, in a library called MAGE.

Conclusion

Enough spoilers! Sign up for the free course and cover all the listed topics and more. After the course, you will be able to brag to all of your LinkedIn friends about how you became a graph master. :)

NetworkX Developers, Say Farewell to the Boilerplate Code

Katarina Supe — Tue, 22 Nov 2022 08:49:07 +0000

If you are using NetworkX, you are aware of the valuable insights you can get from a data network. These networks, or graphs, are becoming popular in diverse business use cases, where they help make important decisions that affect business success in many ways.

NetworkX has a large set of interesting graph algorithms, and it can analyze a network of data fast, but only on a certain scale. You might have found yourself wondering how you can stay in a safe world built around NetworkX graph algorithms but still be able to create production-ready applications to share your work with others easily. Maybe you are bored of writing hundreds of lines of code just to get the data you want to explore.

This blog post will tell you more about how to avoid all of that boilerplate code and smoothly deploy applications by using Memgraph, an open-source in-memory graph database, on which you can perform graph analytics using NetworkX you are familiar with, both on static and streaming datasets.

Seamlessly connect to many data sources with Memgraph

The beginning of every NetworkX project includes data import. If you are importing data from different sources, your code is probably snowballing with every additional source. Things are getting messy, and every time you change something in the code, you have to pull the data from all those different sources all over again. The project gets harder to maintain, and your time gets unnecessarily wasted on pulling the data rather than on the actual analysis.

With Memgraph, you can easily connect to data sources and import the data at the beginning of your project, as well as change it on the fly. Although it is in-memory, Memgraph still persists your data, meaning your dataset doesn’t have to be loaded on each run. It’s enough to load it once and then query, analyze and change it. You can explore parts of the graph you need and deliver an analysis of the dataset. How does Memgraph do it?

Memgraph natively has several methods of data import - import from files, MySQL or PostgreSQL, and data streams. Memgraph is also highly extendable, and with the help of its Python client, GQLAlchemy, you can import data from almost anywhere.

GQLAlchemy’s loaders.py translates table data from a file to graph data and imports it into Memgraph. Currently, GQLALchemy supports the reading of CSV, Parquet, ORC, and IPC/Feather/Arrow file formats via the PyArrow package. These files can be loaded from local sources, Azure Blob storage, or Amazon S3 service. Besides that, if you want to read from a file system currently not supported by GQLAlchemy, or read from a different source, you can implement a custom file system importer, by extending abstract classes FileSystemHandler and DataLoader.

Memgraph is especially useful when it comes to importing streaming datasets because it can connect to any Kafka, Pulsar, or Redpanda stream of data. For example, if you are using Debezium in your company to monitor changes in a database, with Memgraph, you can connect to that Kafka stream.

Since NetworkX algorithms are run on graphs that are currently in-memory, it doesn’t offer the possibility of having a dynamic graph and running dynamic graph algorithms on each change. Memgraph can do that with the help of triggers that fire up dynamic algorithms on a certain change or update in the database. This kind of setup gives you the results of graph algorithms much faster. Check out which dynamic graph algorithms are so far implemented in MAGE, an open-source repository that contains graph algorithms and modules written in Python, C/C++, and Rust.

So, if your project requires importing streaming or static data from various sources, you don’t have to worry about importing in your NetworkX code.

Memgraph and NetworkX integration

NetworkX offers a wide range of graph algorithms to explore the data, and it is possible to use them within Memgraph. Memgraph is built in C++ and it holds wrapper objects for Memgraph’s graphs, called MemgraphGraph, MemgraphDiGraph, MemgraphMultiGraph or MemgraphMultiDiGraph, depending on the graph type stored in the database.

These objects optimize the usage of NetworkX algorithms. To make the transition to Memgraph more convenient, we implemented procedures that call NetworkX functions with Memgraph graphs so that you have out-of-the-box access to graph algorithms.

For example, call the betweenness_centrality algorithm on the preloaded dataset by running:

CALL nxalg.betweenness_centrality() YIELD *;

Here is a list of all implemented NetworkX algorithms. To learn more about the usage of NetworkX algorithms in Memgraph, check out the Social network analysis with NetworkX tutorial.

Cypher query language, used to query Memgraph, is easily extendible with custom procedures bundled up in query modules. These procedures can be written in Python, C/C++ and Rust. Inside the custom procedures written in Python you can use the NetworkX library. The easiest way to implement a custom procedure is to use Memgraph Lab, Memgraph’s visual user interface. Just paste the NetworkX code to the custom read or write procedure in the code editor in the Query Modules section of Memgraph Lab.

To use the custom procedure, call it with the Cypher CALL clause. Memgraph doesn’t have a custom implementation of the girvan_newman community algorithm so it can be implemented within the custom read procedure. Just be careful to use the correct graph type, depending on the dataset - Graph, DiGraph, MultiGraph, or MultiDiGraph.

In the above example, MemgraphDiGraph is created from mgp.ProcCtx, which represents the whole graph inside the database in its current state. That graph is cast to the NetworkX DiGraph, so that NetworkX girvan_newman algorithm can be used. The procedure returns lists of nodes belonging to a certain community.

To call the above custom procedure, we first need to import a dataset. Many datasets can be found in the Datasets section of the Lab. We are going to query the dataset Karate club friendship network.

The query used to call the custom procedure and return its results is:

CALL communities.detect()
YIELD communities
UNWIND communities AS community
RETURN community;

Here are the obtained results on the template dataset Karate club friendship network.

In the standalone NetworkX script, you would first need a bunch of code to pull all the data you want to explore. Then you would call the algorithm, and at the end, the results would have to be stored somewhere. So, to run one simple algorithm, you would have to write A LOT of code. Now multiply that with a couple more data sources and a much more complicated analysis with NetworkX. With Memgraph, you import data or connect to data sources and run your favorite existing or custom graph algorithm.

For more examples of utilizing NetworkX integration, check out the Exploring a musical social network tutorial on Memgraph documentation.

Share the results of your work easily

NetworkX is not a database, so of course, it is missing other features that come in handy when you are deploying your application. For example, with a graph database, you can always do ad-hoc graph querying. If you are using Memgraph, you also get a set of scalable and production-ready algorithms called MAGE (Memgraph Advanced Graph Extensions). This means that you can have your whole project in one place.

Dataset is loaded inside Memgraph, you can explore it with Memgraph Lab by querying the database or by running already implemented or custom procedures. It gives you the necessary flexibility while still remaining easy to use and share with others. Your script can hold graph analysis, machine learning projects, simple calculations, and can write back to the database or just read and return the wanted results.

Your queries can be visualized with Memgraph Lab and sent or presented to your colleagues. Also, if you are deploying your project, you don’t need to worry about the set of components needed for your application to run, you just have to manage Memgraph.

There are many ways to deploy and manage your project, but to make it easier for you, we prepared docker-compose.yml files necessary to run Memgraph Docker images. Read one our blog posts to learn more about how to orchestrate your graph application with Docker Compose. If you need Memgraph as a part of your Kubernetes cluster, you can use the prepared Helm Chart for a simple setup.

Memgraph Lab offers the possibility to save prepared queries in query collections. Memgraph Lab has rich query collections, allowing you to name every query inside the collection and add its description. Also, you can run each query inside the collection to present the results of your work.

Conclusion

Memgraph is a drop-in replacement for a huge part of the boilerplate code in the NetworkX project and give relevant insights. It can work with streaming or static datasets and it offers easy integration of any NetworkX code.

The results provided by Memgraph, in terms of visualization and speed, are incredible and good enough reason to try it out now. Check out how Memgraph deals with data persistency, large-scale data analytics and visualizations when compared to NetworkX and how much faster Memgraph graph algorithms are.

To learn more, head over to our resources for NetworkX developers.

Who ranks better? Memgraph vs NetworkX PageRank

Katarina Supe — Tue, 22 Nov 2022 08:42:25 +0000

NetworkX is an impressive tool for many Python developers who enjoy researching data networks. An important part of that research are graph algorithms. There are various graph algorithms out there, and they all perform differently, depending on their implementations and the scale of the dataset being analyzed. NetworkX’s pure Python implementation is easy to use for any Python developer. Still, when it comes to issues with scaling and the need for persistence when working on applications in production, Memgraph saves the day.

Memgraph is an open-source in-memory graph computation platform for static or real-time graph analytics. It holds a set of highly optimized graph algorithms implemented in C++ and also offers the possibility to create custom procedures in Python, C/C++, or Rust. Read how to explore your NetworkX graphs with Memgraph to understand the benefits of using Memgraph compared to NetworkX.

One of the most popular graph algorithms is the PageRank algorithm, used initially by Google Search to rank web pages in their search engine results. There are many implementations of PageRank out there, and NetworkX and Memgraph have theirs. Let’s find out which one is faster on a sample dataset!

The dataset

Both the NetworkX and Memgraph PageRank can and will be run from Memgraph because in Memgraph the Cypher query language can be expanded with Python procedures. In one of the procedures, we will utilize the NetworkX library.

Memgraph’s visual interface, Memgraph Lab, has a collection of datasets that can be used for various graph experimentations and explorations. The dataset chosen for this comparison is the Wikipedia articles dataset made out of 78,181 nodes and 310,227 relationships. The Wikipedia articles dataset can be imported into the database from the Datasets section.

A graph data model of the Wikipedia articles dataset can be generated in the Graph Schema section to understand better how the nodes and relationships interact with each other in the dataset.

The Wikipedia articles dataset is large enough to obtain significant results to compare Memgraph’s and NetworkX’s PageRank algorithms. Comparison can be made on much larger datasets, but NetworkX eats up memory pretty fast, and those kinds of use cases are where Memgraph would, without a question or need for testing, be a better choice.

Custom query module

Memgraph is integrated with NetworkX, which means it can transform NetworkX graphs into Memgraph graphs, along with the set of NetworkX algorithms and algorithms that utilize the NetworkX library. NetworkX algorithms inside Memgraph are optimized for the best performance and run on Memgraph DiGraph objects.

A custom query module allows us to run the NetworkX PageRank algorithm on the NetworkX DiGraph object instead of running it on the Memgraph DiGraph object for a fairer comparison. A custom query module can be developed in the Query Modules section by creating a new module. Names are given per preference, and we called this the query module measure.

Below is the code for the custom query module used for the comparison:

import mgp
import networkx as nx

@mgp.read_proc
def pagerank(ctx: mgp.ProcCtx) -> mgp.Record(node=mgp.Vertex, rank=float):

    g = nx.DiGraph()
    g.add_nodes_from(ctx.graph.vertices)
    for v in ctx.graph.vertices:
        g.add_edges_from([(edge.from_vertex, edge.to_vertex) for edge in v.in_edges])

    pg = nx.pagerank(g, tol=1e-05)

    return [mgp.Record(node=k, rank=v) for k, v in pg.items()]

In the query module, procedure pagerank extracts a graph from the context and creates an instance of NetworkX DiGraph. Then, the NetworkX PageRank algorithm is run on that DiGraph.

Procedures from custom query modules are run from the Query Execution section. The pagerank() procedure from the measure query module is called with the following Cypher query:

CALL measure.pagerank() YIELD node, rank;

If you are interested in developing custom query modules with Memgraph, head to our documentation and learn more about it.

The comparison

In order to get the best comparison, Memgraph’s get() procedure is called from the pagerank module and the custom procedure pagerank() from the measure module. The number of return results is limited to 1 because if the results are filtered or not limited, the unnecessary time would be counted in.

Below are the results for Memgraph’s PageRank on the Wikipedia articles dataset:

And here are the results of NetworkX’s PageRank on the same dataset:

Memgraph is more than 5 times faster than NetworkX in performing the PageRank algorithm on a graph of Wikipedia articles dataset scale (78,181 nodes and 310,227 relationships)!

It’s important to note that Memgraph outperforms NetworkX on a larger scale without question due to its C++ implementation and highly optimized storage memory usage. PageRank is just one example of a graph algorithm that Memgraph offers out of the box. For more, check out the list of available query modules.

Another thing worth mentioning is that Memgraph supports dynamic graph algorithms, which can speed up the graph analysis even more! For example, with dynamic PageRank and a stream of data, Memgraph gives you newly updated results as soon as the graph object is consumed, that is, created in the database. For real-time use cases, such as credit card fraud detection, and that’s where Memgraph shines the best.

Conclusion

Due to its optimized storage memory usage and implementations of graph algorithms in C++, Memgraph handles graph computing on large datasets better than NetworkX, as shown by running PageRank algorithm on a dataset of 78,181 nodes and 310,227 relationships which Memgraph handled 5 times faster than NetworkX. And as the dataset size increases, so does the Memgraph’s speed compared to NetworkX.

If you want to check more examples and comparisons, head over to the music social network tutorial for betweenness centrality comparison or social network analysis tutorial to learn more about community detection.

To learn more, head over to our resources for NetworkX developers.

Data persistency, large-scale data analytics and visualizations - biggest NetworkX challenges

Katarina Supe — Tue, 22 Nov 2022 08:37:12 +0000

If you have been using NetworkX for one of your projects, you’ve probably noticed some of its limitations. Each time you want to change something in the dataset or run another algorithm, you have to load the dataset all over again. This repetitive loading wastes time and slows down the development process, especially for large datasets.

After running the queries, you probably want to explore the results by playing around with their visualization and share exciting insights you come across. As you probably know, NetworkX is not primarily a graph drawing package, so it doesn’t offer much to create visually pleasing and interactive graphs. Also, NetworkX cannot handle visualizations of large graphs, so you need to reach out for another drawing library and learn how to use it. Even when those libraries are better than NetworkX alone, they still have limitations regarding zooming and interacting with nodes and relationships, such as dragging and clicking.

Memgraph can offer you data persistence, allow you to run queries and graph algorithms and create visualizations you can interact with and customize, and it is free to use as it’s open source. Read on to find out more.

To persist or not persist, that is the question

The issue with NetworkX is that it stores data in-memory in instantiated Python objects - Graph, DiGraph, MultiGraph, or MultiDiGraph. So NetworkX can actually give results of graph algorithms on smaller-scale graphs pretty fast. Still, each time you need to change something in the database or run another algorithm, you have to reload the dataset into the appropriate Python object. What’s even more annoying, if you plan to use your project in production, you need to find an appropriate permanent storage solution and additional graph analytics capabilities. There are a couple of solutions to store the data somewhere else, including exporting and importing those Python objects to different formats or using on-disk storage solutions, but that means more software you need to figure out, which slows down the development process significantly.

The bottom line is - you like using Python and NetworkX for analyzing graphs as it allows you to harness the power of graphs, but you have reached its limits. Your graph exceeds the memory limit, or the memory gets eaten up as soon as you fire up one of NetworkX’s graph algorithms, or you need a graph data storage that allows you to perform graph analytics quickly.

Memgraph is, at its core, an in-memory graph database. With Memgraph, you can import large datasets, store them as graphs and run queries and graph algorithms. Although it is an in-memory database, it still has mechanisms that ensure durability, that is, it persists the data. Once you load your data into Memgraph, you can run different graph algorithms and change the database content without wasting time on data re-loading.

To ease the process of using Memgraph for NetworkX developers, NetworkX graph can be transferred into Memgraph using predefined transformations.

Graph analytics on a scale

Although NetworkX is easy to use and fires up quickly, it cannot handle large amounts of data. Many NetworkX graph algorithms, like the Girvan-Newman algorithm, become computationally very expensive on a scale of hundreds of thousands or millions of nodes and relationships. That computational cost comes from a pure Python implementation of NetworkX, and Python implementations are significantly slower than C/C++ implementations.

Memgraph is built in C++ and holds wrapper objects for Memgraph’s graph, called MemgraphGraph, MemgraphDiGraph, MemgraphMultiGraph, or MemgraphMultiDiGraph, depending on the graph type. Each of the wrapper objects optimizes the usage of NetworkX algorithms. With procedures from the nxalg query module within MAGE, a graph algorithm library, you can call most of the NetworkX algorithms with Memgraph graphs. If an algorithm you require isn’t available, the MAGE project is open-sourced, so you are more than welcome to contribute.

The library also holds implementations of the graph analyzer and weakly connected components which utilize the NetworkX library. All in all, even though you switch to an environment, all the familiar algorithms are still available.

Once you get used to query modules, utilize Memgraph’s graph algorithms as well. Most of the algorithms are implemented in C++ and heavily optimized to offer the best possible performance, meaning that in most cases, they outperform NetworkX algorithms.

Besides standard implementations of algorithms, Memgraph also offers dynamic graph algorithms, which allow you to analyze the data during ingestion instead of waiting for the long imports of massive datasets. If you have a data stream, you can consume it with Memgraph and trigger dynamic graph algorithm updates on each change. The dynamic graph algorithm approximates values locally around the updated node and propagates them through the whole graph. New values are received much faster, and that speeds up the decision time in time-sensitive use cases.

If the algorithms and procedures above don’t cover your use case, you can always implement your custom graph algorithms in Python by creating your own query modules and procedures.

Drawing large graphs with Memgraph Lab

NetworkX isn’t primarily a graph drawing package, but it does offer basic drawing with Matplotlib and an interface for Graphviz. If your research requires drawing graphs that consist of tens of thousands of nodes and relationships, you’ll probably experience several nervous breakdowns waiting for the visualization to complete. You can utilize other tools, but they all have limited usage, so not every tool will fit the use case you are working on, and there is no end to learning every new tool from scratch.

Memgraph Lab is a visual user interface that can also render graphs. It uses an open-source visualization library called Orb, which allows the implementation of visualizations in the frontend code. The visualizations are adaptable and customizable using a graph-style editor defined by the GSS language.

Check out how Orb and Memgraph Lab visualize 32,436 nodes and 122,920 relationships:

Graphs can also be interacted with. Clicking on graph objects previews their details, such as labels and properties. Nodes can be expanded to show relationships, collapsed to hide them, or dragged around to visualize the story you want the data to tell. If you need objects to interact, you can enable physics and adjust the collision or repulsive forces within the graph.

The above visualizations show the default style, but if you decide to tweak it with the Graph Style Script, you can create any kind of visualization that best represents your data. In the image below, nodes are of different sizes and colors, depending on their ranks and the communities they belong to.

Conclusion

If you like using NetworkX as a graph analytic tool required for your project but often need to reach for other tools, remember that Memgraph can speed up your development by being persistent in-memory data storage. You still get to use the NetworkX algorithms you are used to or explore other, more efficient, and dynamic algorithms.
If default and implemented algorithms and procedures do not cover your use case, extend the Cypher query language with custom procedures.

In the end, a visualization tool is also a part of the Memgraph package, so there is no more need to explore other solutions. Feel free to shape the graph results so they reflect the insights you discovered in the best possible way. Once the project is ready for production, you only need to handle the Memgraph deployment to ship your app easily and show your hard work to other graphistas out there, without writing any boilerplate code!

To learn more, head over to our resources for NetworkX developers.

How to orchestrate your graph application with Docker Compose

Katarina Supe — Thu, 27 Oct 2022 11:43:46 +0000

Introduction

Docker is an open platform for developing, shipping, and running applications. It helps you package and run an application in an isolated environment called a container. The container has everything needed to run the application, including the libraries that now don’t have to be installed on host machines. The Dockerized application is easy to share with someone and to run because it will work in the same way on different hosts. Docker Compose is a tool for defining and running multi-container Docker applications. All you need is a YAML file called docker-compose.yml in the root of your project in order to configure the application’s services.

Deploying applications with Docker Compose

Docker Compose can be used for production, staging, development, testing, or CI workflow environments. In Memgraph, we create many demo web applications to showcase Memgraph’s capabilities. When creating such applications, we follow these steps:

1. We create a Dockerfile to define a Docker image that will be used, copy certain directories and install necessary libraries inside the container. The application usually has a backend, frontend, and Memgraph (database) service.

my-application
├── backend
│   ├── app.py
│   └── Dockerfile
├── frontend
│   ├── src
│   │   └── App.js
│   └── Dockerfile
├── memgraph
│   └── Dockerfile
└── docker-compose.yml

For memgraph service Dockerfile, we often copy the folder with newly defined procedures, called query modules.

FROM memgraph/memgraph-mage:1.1
USER root
# Copy the local query modules
COPY query_modules/twitch.py /usr/lib/memgraph/query_modules/twitch.py
USER memgraph

In this example, we used memgraph/memgraph-mage image, version 1.1. We switched to user root so that the service within the container has the necessary permissions to copy the local query modules file and then gave the permissions back to the user memgraph.

2. Next, we define all services from the application in the docker-compose.yml file.

version: "3"

networks:
 app-tier:
   driver: bridge

memgraph:
   build: ./memgraph
   ports:
     - "7687:7687"
     - "7444:7444"
   networks:
     - app-tier

backend-app:
   build: ./backend
   volumes:
     - ./backend:/app
   ports:
     - "5000:5000"
   networks:
     - app-tier

frontend-app:
   build: ./frontend
   volumes:
     - ./frontend:/app
   ports:
     - "3000:3000"
   networks:
     - app-tier

Every service is running on different ports and copies different necessary files from the host machine to the container in its Dockerfile. For example, if the frontend service is a React app, you need to copy the package.json file and run npm install to install all packages and dependencies needed for the React app to run. All services are on the same app-tier bridge network. A bridge network allows containers connected to the same bridge network, app-tier in this case, to communicate while providing isolation from containers that are not connected to that bridge network.

3. In the end, to build the application, we use docker-compose build, and to run it, we use docker-compose up command.

Once you define your application with Docker Compose, it’s pretty easy to run it in different environments. Before deploying it, you need to make just a couple of changes regarding the volume bindings, exposed ports, and similar. If you want to learn more about it, check out the Docker documentation. The easiest way to deploy the application is to run it on a single server since that is the most similar behavior to your development environment.

Memgraph Docker Compose file

Let’s check out what’s written inside the Memgraph Docker Compose file. Memgraph offers you three different Docker images:

Memgraph Platform, which contains:
- MemgraphDB - the database that holds your data
- Memgraph Lab - visual user interface for running queries and visualizing graph data
- mgconsole - command-line interface for running queries
- MAGE - graph algorithms and modules library
Memgraph MAGE, which contains MemgraphDB and MAGE.
Memgraph, which includes only MemgraphDB.

Definition of a Memgraph service within your docker-compose.yml file depends on the image you are using. Since Memgraph Platform provides a wholesome solution, its image is most widely used. This is the Docker Compose for Memgraph Platform image:

version: "3"
services:
  memgraph-platform:
    image: "memgraph/memgraph-platform"
    ports:
      - "7687:7687"
      - "3000:3000"
      - "7444:7444"
    volumes:
      - mg_lib:/var/lib/memgraph
      - mg_log:/var/log/memgraph
      - mg_etc:/etc/memgraph
    environment:
      - MEMGRAPH="--log-level=TRACE"
    entrypoint: ["/usr/bin/supervisord"]
volumes:
  mg_lib:
  mg_log:
  mg_etc:

Port 7687 is used for communication with Memgraph via Bolt protocol. The port 3000 is exposed because Memgraph Lab will be running on localhost:3000, while the port 7444 is there so that you can access logs within Memgraph Lab. We also specified three useful volumes:

mg_lib - directory containing data that enables data persistency
mg_log - directory containing log files
mg_etc - directory containing the configuration file

The exact location of the local directories depends on your specific setup.

Configuration settings can be changed by setting the value of the MEMGRAPH environment variable. In the above example, you can see how to set --log-level to TRACE. Since Memgraph Platform is not a single service, the process manager supervisord is used as the main running process in the entrypoint. Since the MAGE library is included in this image, you can use the available graph algorithms.

How we built Twitch analytics demo using Docker Compose

When we’re building demo applications to showcase Memgraph, we always use Docker Compose. This allows us to fire up the application on any system, which is useful when showing a demo at conferences or meetups. Also, applications created with Docker Compose are much easier to deploy. One such demo is the Twitch analytics demo. In the docker-compose.yml file, we defined a couple of services:

kafka - a message broker
zookeeper - a service that manages kafka service
memgraph-mage - a service that uses Memgraph MAGE Docker image which gives us the possibility to use graph algorithms, such as PageRank and betweenness centrality
twitch-app - a Flask server that sends all the data we query from memgraph-mage to the react-app
react-app - a React app that visualizes the Twitch network
twitch-stream - a Python script that produces new messages to a Kafka topic

As mentioned before, all services are on the same app-tier bridge network to communicate with each other. To learn more about the implementation of this application, head over to the Twitch Streaming Graph Analysis blog post series.

Conclusion

Now that you learned how to use Docker Compose to orchestrate your graph application, we hope that the application development process with Memgraph will be more straightforward. Whichever Memgraph Docker image you decide to use, join our Discord server and showcase your creations!

How to migrate from Neo4j to Memgraph

Katarina Supe — Thu, 17 Mar 2022 10:53:14 +0000

Introduction

Through this tutorial, you'll learn how to migrate the movies dataset from Neo4j to Memgraph. If you used Neo4j, you are probably familiar with their example graph that helps you learn the basics of the Cypher query language. Neo4j is an ACID-compliant transactional native graph database, while Memgraph is a platform designed for graph computations on streaming data. You can read more about their differences in the Neo4j vs Memgraph article.

Prerequisites

Docker (Linux) or Docker Desktop (macOS/Windows) - you will run Memgraph and Neo4j in a Docker container
Neo4j with APOC - in this tutorial, you'll use the apoc.export.csv.all() procedure.
Memgraph - you are going to import your data using the LOAD CSV clause.

Run Neo4j and Memgraph

You need to start Neo4j with APOC, and you can do that by running:

docker run -p 7474:7474 -p 7687:7687 \
-v data:/data -v  plugins:/plugins --name neo4j-apoc \ 
-e NEO4J_apoc_export_file_enabled=true \ 
-e NEO4J_apoc_import_file_enabled=true \ 
-e NEO4J_apoc_import_file_use__neo4j__config=true \ 
-e NEO4JLABS_PLUGINS=\[\"apoc\"\] \  
-e NEO4J_AUTH=neo4j/password neo4j:latest

If you're having trouble running this on the new Apple M1 chip, try adding --platform linux/arm64/v8 after the run command. Also, you can set NEO4J_AUTH however you like, here neo4j is username and password is password.
The Neo4j Browser is available at http://localhost:7474, and you can use the credentials from the NEO4J_AUTH environment variable. The ports 7474 and 7687 are now exposed, and since Memgraph also uses port 7687, you can run it at the next available port, that is, 7688:

docker run -it -p 7688:7687 -p 3000:3000 -v mg_lib:/var/lib/memgraph memgraph/memgraph-platform

Since you started Memgraph Platform, the command-line tool mgconsole should be open in your terminal, and the visual user interface Memgraph Lab is available at http://localhost:3000.

Great, now you have both databases running and you're ready to play with the movies dataset!

1. Load the movies dataset into Neo4j

The movies dataset contains actors and directors that are related through the movies they've collaborated on.

This is a simple dataset and a good example of how you can migrate the whole graph database from Neo4j to Memgraph. You can import it to Neo4j by running the Movie Graph example from Example Graphs in the Neo4j Browser or Desktop app. Once the movie dataset is loaded into Neo4j, you can jump to exporting the whole database.

2. Export the movies dataset from Neo4j

You are going to export the whole database from Neo4j to a CSV file movies.csv by using the apoc.export.csv.all() procedure. Through the Neo4j Browser (or Desktop app), run the following query:

CALL apoc.export.csv.all("movies.csv", {})

Now you have exported the whole database to the movies.csv file. But where is this file located? Since you ran both Neo4j and Memgraph with Docker, files you are exporting or importing are located inside the Docker container. To enter the container, you first must find out the container ID of the container where Neo4j is running. Run docker ps to check all the running containers. There you can find the Neo4j container ID and copy it. After that, run:

docker exec -it <container_ID> bash

After you run the above command, you are inside the container where Neo4j is running. All exported data is located at /var/lib/neo4j/import/. Hence, run cd /var/lib/neo4j/import/ to check whether the movies.csv file is there. Next, you are going to copy the movies.csv file to your local file system. To do that, you need to run:

docker cp <container_ID>:/var/lib/neo4j/import/movies.csv /path_to_local_folder/movies.csv

The <container_ID> is the ID of the Neo4j container, which you already found before. On the right, you can give the path to one of your local folders where you want the movies.csv file to be copied to.

3. Import the movies dataset into Memgraph

After you have exported the whole database from Neo4j into the movies.csv file and copied it to your local file system, you are ready to import the movies dataset into Memgraph. First, you have to copy the movies.csv file to the Docker container where Memgraph is running. Here's what you have to do:

Run docker ps to find out Memgraph's container ID
Run docker cp /path_to_local_folder/movies.csv <container_ID>:/usr/lib/memgraph/movies.csv

Now the movies.csv file is ready to be imported to Memgraph with the LOAD CSV clause.
First, you need to import all nodes, and after that, relationships. In order to create the relationships, start and end nodes have to already be in the database, and that's why you are importing in that order.

You can write the queries in the mgconsole, a command-line tool, or in the query editor in the Memgraph Lab. Memgraph Lab is more user-friendly, so I would recommend this way of querying. To connect to Memgraph from Memgraph Lab, you will have to choose the Connect manually option, since the port on which Memgraph is running is not the default one (7687). Once you choose Connect manually, you just have to change the port to 7688 and click Connect.

To create all Person nodes, copy the following query in the query editor:

LOAD CSV FROM "/usr/lib/memgraph/movies.csv" WITH HEADER as row
WITH row WHERE row._labels = ':Person'
CREATE (:Person {id: row._id, name: row.name, born: row.born});

In the movies.csv file, the column _labels is empty if the data in the row represents a relationship. If the row is a node, then the column _labels contains the label of that node. Hence, you can create the Person nodes by going through the _labels column. Every person has the properties id, name and born.

In a similar way, you can create the Movie nodes:

LOAD CSV FROM "/usr/lib/memgraph/movies.csv" WITH HEADER as row
WITH row WHERE row._labels = ':Movie'
CREATE (:Movie {id: row._id, relased: row.released, tagline: row.tagline, title: row.title});

You can check whether you created all nodes correctly. First, in the Overview tab, you can see that the number of nodes is 171, and it matches the number of nodes in Neo4j.

Next, you can import the relationships. There are 6 different types of relationships, some have properties and some don't. It's best to import them one by one:

Import ACTED_IN relationship

LOAD CSV FROM "/usr/lib/memgraph/movies.csv" WITH HEADER as row
WITH row WHERE row._type = 'ACTED_IN'
MATCH (n {id: row._start}), (m {id: row._end})
CREATE (n)-[:ACTED_IN {roles: row.roles}]->(m);

Import DIRECTED relationship

LOAD CSV FROM "/usr/lib/memgraph/movies.csv" WITH HEADER as row
WITH row WHERE row._type = 'DIRECTED'
MATCH (n {id: row._start}), (m {id: row._end})
CREATE (n)-[:DIRECTED]->(m);

Import PRODUCED relationship

LOAD CSV FROM "/usr/lib/memgraph/movies.csv" WITH HEADER as row
WITH row WHERE row._type = 'PRODUCED'
MATCH (n {id: row._start}), (m {id: row._end})
CREATE (n)-[:PRODUCED]->(m);

Import WROTE relationship

LOAD CSV FROM "/usr/lib/memgraph/movies.csv" WITH HEADER as row
WITH row WHERE row._type = 'WROTE'
MATCH (n {id: row._start}), (m {id: row._end})
CREATE (n)-[:WROTE]->(m);

Import FOLLOWS relationship

LOAD CSV FROM "/usr/lib/memgraph/movies.csv" WITH HEADER as row
WITH row WHERE row._type = 'FOLLOWS'
MATCH (n {id: row._start}), (m {id: row._end})
CREATE (n)-[:FOLLOWS]->(m);

Import REVIEWED relationship

LOAD CSV FROM "/usr/lib/memgraph/movies.csv" WITH HEADER as row
WITH row WHERE row._type = 'REVIEWED'
MATCH (n {id: row._start}), (m {id: row._end})
CREATE (n)-[:REVIEWED {rating: row.rating, summary: row.summary}]->(m);

Now you can check again that the number of relationships in the Overview tab in Memgraph Lab is the same as in Neo4j.

And that's it! You have migrated the whole movies dataset from Neo4j to Memgraph.

4. Explore the data

You can run a couple of example queries from the Neo4j and compare them with the results in Memgraph, to make sure you did everything right.

For example, run the following query:

MATCH (tom {name: "Tom Hanks"}) RETURN tom;

and check the results:

Results in Neo4j

Results in Memgraph

Next, you can run:

MATCH (cloudAtlas {title: "Cloud Atlas"}) RETURN cloudAtlas;

and check the results:

Results in Neo4j

Results in Memgraph

Feel free to try out a different set of queries and make sure that the results are the same in Neo4j and Memgraph.

Conclusion

That's it for now! You've learned how to migrate the whole database from Neo4j to Memgraph. In this tutorial, you used the apoc.export.csv.all() procedure to export the data from Neo4j and the LOAD CSV clause to import the data to Memgraph. This is not the only way to migrate your data from Neo4j to Memgraph. If you feel creative and find some easier way of doing this, join our Discord server and share it with the Memgraph team and community.

Building robust applications using GQLAlchemy

Katarina Supe — Mon, 31 Jan 2022 14:33:28 +0000

While building the whole web application, you save the necessary data to the database and load it whenever needed. Fetching the data can be error-prone since there may be that one minor typo that will bug you. In this tutorial, you'll learn how to build a small part of the Twitch analytics app with the help of GQLAlchemy, an object graph mapper (OGM) that makes building graph-based apps much easier.

Backend implementation

Through this tutorial, you are going to find the most popular game from the Twitch dataset. The data was retrieved from the Twitch API, and for this tutorial, you will just need the streamers.csv file. First, you have to define the necessary data classes in models.py:

from gqlalchemy import Node, Field, Relationship
from typing import Optional
from app import memgraph
class User(Node):
    name: str = Field(index=True, exists=True, unique=True, db=memgraph)
class Stream(User):
    name: Optional[str] = Field(
        index=True, exists=True, unique=True, db=memgraph, label="User"
    )
    id: str = Field(index=True, exists=True, unique=True, db=memgraph)
    url: Optional[str] = Field()
    followers: Optional[int] = Field()
    createdAt: Optional[str] = Field()
    totalViewCount: Optional[int] = Field()
    description: Optional[str] = Field()
class Game(Node):
    name: str = Field(unique=True, db=memgraph)

Next, you have to load the data from the CSV file:

def load_streams(path):
    with open(path) as read_obj:
        csv_reader = reader(read_obj)
        header = next(csv_reader)
        if header != None:
            for row in csv_reader:
                stream = models.Stream(
                    id=row[1],
                    name=row[3],
                    url=row[6],
                    followers=row[7],
                    createdAt=row[10],
                    totalViewCount=row[9],
                    description=row[8],
                ).save(memgraph)
                game = models.Game(name=row[4]).save(memgraph)
                plays_rel = models.Plays(
                    _start_node_id=stream._id, _end_node_id=game._id
                ).save(memgraph)

You modeled objects using GQLAlchemy's Object Graph Mapper (OGM), which offers schema validation, so you can be sure that the data inside Memgraph will be accurate.
All loading methods can be found in twitch_data.py.

After the data is loaded, you can create an API endpoint for GET requests in app.py to find out which game is the most popular:

@app.route("/top-games/<num_of_games>", methods=["GET"])
@log_time
def get_top_games(num_of_games):
    """Get top num_of_games games by number of streamers who play them."""
    try:
        results = list(
            Match()
            .node("Stream", variable="s")
            .to("PLAYS")
            .node("Game", variable="g")
            .return_({"g.name": "game_name", "count(s)": "num_of_players"})
            .order_by("num_of_players DESC")
            .limit(num_of_games)
            .execute()
        )
        games_list = list()
        players_list = list()
        for result in results:
            game_name = result["game_name"]
            num_of_players = result["num_of_players"]
            games_list.append(game_name)
            players_list.append(num_of_players)
        games = [{"name": game_name} for game_name in games_list]
        players = [{"players": player_count} for player_count in players_list]
        response = {"games": games, "players": players}
        return Response(
            response=dumps(response), status=200, mimetype="application/json"
        )
    except Exception as e:
        log.info("Fetching top games went wrong.")
        log.info(e)
        return ("", 500)

The query builder from GQLAlchemy was used to avoid writing Cypher queries and to notice if there are any errors before the query is executed in the database. The query above counts how many streamers play each game and then returns results in descending order. This is how the query builder construct looks when executed as a Cypher query in Memgraph:

Query builder

Match()
.node("Stream", variable="s")
.to("PLAYS")
.node("Game", variable="g")
.return_({"g.name": "game_name", "count(s)": "num_of_players"})
.order_by("num_of_players DESC")
.limit(num_of_games)
.execute()

Cypher query

MATCH (s:Stream)-[:PLAYS]->(g:Game)
RETURN g.name AS game_name, count(s) AS num_of_players
ORDER BY num_of_players DESC LIMIT num_of_games;

In this request, you can determine how many top games you want to fetch. The response you'll get in the frontend is JSON with games and players as keys.

Frontend implementation

In the end, let's show how you can visualize your results. The frontend is built with the help of Semantic UI React.

You have to create a component in which you're going to fetch the data from the backend. The most important part is the fetching, which you have to call immediately after a component is mounted, that is, in componentDidMount() method:

fetchData(number) {
    fetch("/top-games/" + number)
        .then((res) => res.json())
        .then(
        (result) => {
            this.setState({
                isLoaded: true,
                games: result.games,
                players: result.players,
            });
        },
        (error) => {
            this.setState({
                isLoaded: true,
                error,
            });
        }
        );
    this.setState({
        numOfGames: number,
        header: "Top " + number + " games",
    });
}
componentDidMount() {
    this.fetchData(this.state.numOfGames);
}

Then you can render the results saved in games and players state variables however you want. If you need inspiration, check out the Games.js component.

Conclusion

Creating classes representing nodes and relationships in Python offers you more control of your data. GQLAlchemy enables you to easily communicate with the graph database using the technology stack you're already familiar with. If you have questions about GQLAlchemy's OGM or query builder, visit our Discord server and drop us a message. You can also share any project you come up with!