Forem: Adam Cowley

Using TypeScript with Neo4j

Adam Cowley — Tue, 01 Nov 2022 09:25:58 +0000

TL;DR: You can learn more in the Building Neo4j Applications with TypeScript course
on Neo4j GraphAcademy

The recent 5.2.0 release of the Neo4j JavaScript Driver features some significant improvements for TypeScript users. So much so that it has inspired me to write an article.

It is now possible to use an interface to define the type of records returned by your Cypher query, giving you the added benefit of type-checking and type hinting while processing results.

A Worked Example

For example, let's take a query from the Recommendations Dataset. Say we would like to find a list of all actors that have appeared in a movie.

To find this, we would need to create a new driver instance, open up a new session and then use the executeRead() function to send a Cypher statement and await the result.



async function main() {
  // Create a Driver Instance
  const driver = neo4j.driver(
    'neo4j://localhost:7687',
    neo4j.auth.basic('neo4j', 'letmein!')
  )

  // Open a new Session
  const session = driver.session()

  try {
    // Execute a Cypher statement in a Read Transaction
    const res = await session.executeRead(tx => tx.run(`
      MATCH (p:Person)-[r:ACTED_IN]->(m:Movie {title: $title})
      RETURN p, r, m
    `, { title: 'Pulp Fiction' }))

    const people = res.records.map(row => row.get('p'))

    console.log(people)
  }
  finally {
    // Close the Session
    await session.close()
  }
}

Straightforward enough, and as are all perfect developers we will never experience any problems writing code this way.

On the other hand, if someone happens to make a typo in the line res.records.map(row => row.get('p')) or tries to .get() a value that isn't returned in the result, the Driver is written to throw an Error.

Say that row is changed to:



const people = res.records.map(row => row.get('something'))

As something doesn't exist in the result, a Neo4jError will be thrown:



Neo4jError: This record has no field with key 'something', 
available key are: [p,r,m].

You will eventually find this out when you run the application, but the whole point of TypeScript is to identify these errors during the development process.

Adding Type-Checking

To protect against this type of scenario, we can now use an interface to define the keys available on each record.

In the case of the query above, we have three values:

p - a Node with a label of Person with properties including name and born
r - a Relationship of type ACTED_IN with properties including roles - an array of strings
m a Node with a label of Movie.

The neo4j-driver library exports two type definitions, Node and Relationship, that we can use to define these items.



import neo4j, { Node, Relationship } from 'neo4j-driver'

Both of these classes accept generics to define the type of the .identity and the properties held on the value.

Unless you have set the disableLosslessIntegers option when creating the Driver, the identity will be an instance of the Integer type exported from neo4j-driver.

Person values can be defined as a TypeScript type.



import { Integer } from 'neo4j-driver'

interface PersonProperties {
  tmdbId: string;
  name: string;
  born: number; // Year of birth
}

type Person = Node<Integer, PersonProperties>

Or, for a more terse example, you can define the properties directly in the second generic:



type Movie = Node<Integer, {
  tmdbId: string;
  title: string;
  rating: number;
}>

Relationships almost almost identical, but use the Relationship type instead.



type ActedIn = Relationship<Integer, {
  roles: string[];
}>

These types can be combined within an interface to represent the each record in the result:



interface PersonActedInMovie {
  p: Person;
  r: ActedIn;
  m: Movie;
}

Both the session.run() and tx.run() accept the interface and add type checking to any subsequent processing. The above example can be updated to pass the PersonActedInMovie interface to the tx.run() method call.



// Execute a Cypher statement in a Read Transaction

const res = await session.executeRead(tx => tx.run<PersonActedInMovie>(

  MATCH (p:Person)-[r:ACTED_IN]-&gt;(m:Movie {title: $title})

  RETURN p, r, m

, { title: 'Pulp Fiction' }))

Type Checking in Action

As the record shape has been defined, TypeScript will now validate the code as it is written and provide suggestions.

Suggesting Record Keys

Suggestions are now provided when calling the record.get() method.

Suggesting Properties

TypeScript is aware that people is an array of Person nodes, and properties defined in the interface can be suggested while typing.

Checking Property Keys

If a key does not exist in the properties of a node or relationship, TypeScript will pick this up straight away and throw an error:



const names = people.map(

  person => person.properties.foo 

  // Property 'foo' does not exist

  // on type 'PersonProperties' 

)

Type-checking Properties

TypeScript is now also be aware of the type of each of the properties, so TypeScript will throw an error if you try to use a value that is not defined in the Type.



const names: string[] = people.map(

  person => person.properties.born

)

// Type 'number[]' is not assignable to type 'string[]'.

Interested in learning more?

If you are interesting in learning more about Neo4j, I recommend checkout out the Beginners Neo4j Courses on GraphAcademy.

If you are interested in learning more, I am currently working on a new Neo4j & TypeScript course for Neo4j GraphAcademy.

You can also learn everything you need to know about using Neo4j in a Node.js project in the Building Neo4j Applications with Node.js
course, in which you will take a deeper dive into the Driver lifecycle and replace hardcoded data with responses from a Neo4j Sandbox instance.

The Building Neo4j Applications with TypeScript course
is a shorter, two hour course that covers the fundamentals of the Neo4j JavaScript Driver along with additional TypeScript features.

If you have any comments or questions, feel free to reach out to me on Twitter.

Using Neo4j in your next Next.js Project

Adam Cowley — Thu, 27 Oct 2022 16:38:36 +0000

After watching a few glossy new videos from the Next.js Conf 2022, I thought I'd take a closer look at Next.js and see how the framework could help me to build my next Neo4j-based web application.

Fundamentally, adding Neo4j integration to a Next.js project is similar to any other Node.js/TypeScript-based project. However, the various Data Fetching methods and both Server-side and Client-side rendering raise some interesting challenges.

Let's take a look at how we can use Neo4j in a Next.js project.

What is Next.js?

Next.js is a React-based framework which provides an opinionated starting point for building web applications. The framework provides building blocks for many of the common features that developers need to consider when building modern applications such as UI components, Data Fetching, and Rendering.

The framework also focuses on performance, providing the ability to pre-generate static HTML pages using Static-site Generation (SSG), render HTML on the server at request time using Server-side rendering (SSR) and also render React components on the client-side using Client-side Rendering (CSR).

You can read more about Next.js here.

What is Neo4j?

The chances are, if you have found this article via search, you'd know more about Next.js than Neo4j. Neo4j is a Graph Database, a database consisting of Nodes - which represent entities or things, connected together and Relationships.

Neo4j comes into its own when working with highly connected datasets or as an alternative for complex relational database schemas where many joins are required. The golden rule is that if your queries have three or more joins, you should really be looking at using a graph database.

You can read more about Neo4j here.

Why Neo4j and Next.js?

Next.js is gaining momentum as one of the most popular frameworks for building modern web applications. The benefit of using Next.js is that your front-end and back-end code are all self-contained within the same subfolders of the api/ directory.

If you are building a Neo4j-backed project, building an integration with the Neo4j JavaScript Driver is relatively straightforward. All you need to do is create a new instance of the driver within the application, then use the driver to execute Cypher statements and retrieve results.

Of course, you can use the Neo4j JavaScript driver directly from React components, but this means exposing database credentials through the client which can be a security risk. Instead, if you require on-demand data from Neo4j in client-side rendering, you can create an API handler to execute the Cypher statement server-side and return results.

Creating a free Neo4j AuraDB Instance

Neo4j AuraDB, Neo4j's fully managed cloud service provides one AuraDB Free instance to all users, completely free and no credit card is required.

If you sign in or register for Neo4j Aura at cloud.neo4j.io, you will see a New Instance button at the top of the screen. If you click this button, you will be able to choose between an empty database or one pre-populated with sample data.

For this article, I suggest choosing the Graph-based Recommendations dataset, which consists of Movies, Actors, Directors and user ratings. This dataset is a nice introduction to graph concepts and can be used to build a movie recommendation algorithm. We use it across GraphAcademy, including the Building Neo4j Applications with Node.js course.

Click Create to create your instance. Once you have done so, a modal window will appear with a generated password.

Click the Download button to download your credentials, we'll need these a little later on. After a couple of minutes, your instance will be ready to explore. You can click the Explore button to explore the graph with Neo4j Bloom, or query the graph using Cypher by clicking the Query tab.

You can take a look at that in your own time, for now, let's focus on our Next.js application.

Creating a new Next.js Project

You can create a new Next.js project from a template using the Create Next App CLI command.



npx create-next-app@latest

The command will prompt you for a project name and install any dependencies.

Adding Neo4j Helper Functions

To install the Neo4j JavaScript Driver, first install the dependency:



npm install --save neo4j-driver
# or yarn add neo4j-driver

Next.js comes with built-in support for Environment Variables, so we can simply copy the credentials file downloaded from the Neo4j Aura Console above, rename it to .env and place in the directory root.

We can then access those variables through the process.env variable:



const { NEO4J_URI, NEO4J_USERNAME, NEO4J_PASSWORD } = process.env

Next, create a new folder called lib/ and then create a new neo4j.js file. You will want to import the neo4j object from the neo4j-driver dependency and use the credentials above to create a driver instance



// lib/neo4j.js
const driver = neo4j.driver(
  process.env.NEO4J_URI,
  neo4j.auth.basic(
    process.env.NEO4J_USERNAME,
    process.env.NEO4J_PASSWORD
  )
)

When executing a Cypher statement against a Neo4j instance, you need to open a session, and execute the statement within a read or write transaction. This can become a bit cumbersome after a while, so instead, I recommend writing helper functions for read and write queries:



// lib/neo4j.js
export async function read(cypher, params = {}) {
  // 1. Open a session
  const session = driver.session()

  try {
    // 2. Execute a Cypher Statement
    const res = await session.executeRead(tx => tx.run(cypher, params))

    // 3. Process the Results
    const values = res.records.map(record => record.toObject())

    return values
  }
  finally {
    // 4. Close the session 
    await session.close()
  }
}

export async function write(cypher, params = {}) {
  // 1. Open a session
  const session = driver.session()

  try {
    // 2. Execute a Cypher Statement
    const res = await session.executeWrite(tx => tx.run(cypher, params))

    // 3. Process the Results
    const values = res.records.map(record => record.toObject())

    return values
  }
  finally {
    // 4. Close the session 
    await session.close()
  }
}

If you want a deeper dive into this code or best practices I recommend that you check out the Neo4j & Node.js Course on GraphAcademy.

Now that we have a way to query Neo4j, let's look at the options for Data Fetching in Next.js

Data Fetching in Next.js

Next.js allows the rendering of content in several ways.

Static-site Generation (SSG) - where static HTML pages are generated at build time
Server-side Rendering (SSR) - HTML is generated server-side as a request comes in
Client-side Rendering (CSR) - HTTP requests are executed in the browser with JavaScript and the response updates the DOM

Depending on your use case, you may need a mixture of these methods. Say you are running a movie recommendation site, it may make sense to use SSG to pre-build marketing pages. Movie information is held in a database and changes regularly, so these pages should be rendered by the server using SSR. When a user comes to rate a movie, the interaction should take place via an API request and the result rendered using CSR.

Let's take a look at the implementation of each of these records.

Static Page Generation

Let's say, for example, that generic genre pages won't change often and they don't require any user interaction. By generating static pages, we can serve cached versions of the pages and take the load away from the server.

Any component in the pages/ directory which exports a getStaticProps() function (known as a Page) will be generated at build time and served as a static file.

Components created in the pages folder will automatically be mapped to a route. To create a page that will be available at /genres you will need to create a pages/genres/index.jsx file. The component needs to export a default function which returns a JSX component, and a getStaticProps() function.

First, to get the data required by the component, create the getStaticProps() function and execute this Cypher statement in a read transaction.



// pages/genres/index.jsx
export async function getStaticProps() {
  const res = await read(`
    MATCH (g:Genre)
    WHERE g.name <> '(no genres listed)'

    CALL {
    WITH g
    MATCH (g)<-[:IN_GENRE]-(m:Movie)
    WHERE m.imdbRating IS NOT NULL AND m.poster IS NOT NULL
    RETURN m.poster AS poster
    ORDER BY m.imdbRating DESC LIMIT 1
    }

    RETURN g {
      .*,
      movies: toString(size((g)<-[:IN_GENRE]-(:Movie))),
      poster: poster
    } AS genre
    ORDER BY g.name ASC
  `)

  const genres = res.map(row => row.genre)

  return {
    props: {
      genres,
    }
  }
}

Anything returned inside props from this function will be passed as a prop into the default component.

Now, export a default function which displays a list of Genres.



// pages/genres/index.jsx
export default function GenresList({ genres }) {
  return (
    <div>
      <h1>Genres</h1>

      <ul>
        {genres.map(genre => <li key={genre.name}>
          <Link href={`/genres/${genre.name}`}>{genre.name} ({genre.movies})</Link>
        </li>)}
      </ul>
    </div>
  )
}

This should generate an unordered list of links for each Genre:

Looking good...

If you run the npm run build command, you will see a genres.html file inside the .next/server/pages/ directory.

Using Neo4j for Server-side Rendering

The movie list on each genre page may change often, or you may wish to add extra interaction to the page. In this case, it makes sense to render this page on the server. By default, Next.js will cache this page for a short amount of time which is perfect for websites with high amounts of traffic.

Each genre link on the previous page links to /genres/[name] - for example /genres/Action. By creating a pages/genres/[name].jsx file, Next.js knows automatically to listen for requests on any URL starting with /genres/ and detect anything after the slash as a name URL parameter.

This can be accessed by the getServerSideProps() function, which will instruct Next.js to render this page using Server-side Rendering as the request comes in.

The getServerSideProps() function should be used to get the data required to render the page and return it inside a props key.



export async function getServerSideProps({ query, params }) {
  const limit = 10
  const page = parseInt(query.page ?? '1')
  const skip = (page - 1) * limit

  const res = await read(`
    MATCH (g:Genre {name: $genre})
    WITH g, size((g)<-[:IN_GENRE]-()) AS count

    MATCH (m:Movie)-[:IN_GENRE]->(g)
    RETURN
      g { .* } AS genre,
      toString(count) AS count,
      m {
        .tmdbId,
        .title
      } AS movie
    ORDER BY m.title ASC
    SKIP $skip
    LIMIT $limit
  `, {
    genre: params.name,
    limit: int(limit),
    skip: int(((query.page || 1)-1) * limit)
  })

  const genre = res[0].genre
  const count = res[0].count

  return {
    props: {
      genre,
      count,
      movies: res.map(record => record.movie),
      page, skip, limit,
    }
  }
}

In the example above, I get the movie name from the params object in the request context which is passed as the only argument to the getServerSideProps() function. I also attempt to get the ?page= query parameter from the URL to provide a paginated list of movies.

These values will again be passed as props into the default function, and can therefore be used to list the movies and pagination links.



export default function GenreDetails({ genre, count, movies, page, skip, limit }) {
  return (
    <div>
      <h1>{genre.name}</h1>
      <p>There are {count} movies listed as {genre.name}.</p>


      <ul>
        {movies.map(movie => <li key={movie.tmdbId}>{movie.title}</li>)}
      </ul>

      <p>
        Showing page #{page}. <br />
        {page > 1 ? <Link href={`/genres/${genre.name}?page=${page-1}`}> Previous</Link> : ' '}
        {' '}
        {skip + limit < count ? <Link href={`/genres/${genre.name}?page=${page+1}`}>Next</Link> : ' '}
      </p>

    </div>
  )
}

Next.js then renders a list of movies with each request.

Using Neo4j for Client-side Data Fetching

As it stands, for each click of the Previous and Next links above, the entire page will reload which isn't ideal. Although this is a trivial example so far, loading KBs worth of HTML again to render the header and footer means additional load on the server and more data sent over the wire.

Instead, you could build a React component that would load the list of movies asynchronously through a client-side HTTP request. This would mean that the list of movies could be updated without reloading the entire page, providing the end-user with a smoother viewing experience.

To support this, we will have to create a API Route which will return a list of movies as JSON.

Any file in the pages/api/ directory is treated as a route handler, a single default exported function which accepts request and response parameters, and expects an HTTP status and response to be returned.

So to create an API route to serve a list of movies at http://locahost:3000/api/movies/[name]/movies, create a new movies.js file in the pages/api/genres/[name] folder.



// pages/api/genres/[name]/movies.js
export default async function handler(req, res) {
  const { name } = req.query
  const limit = 10
  const page = parseInt(req.query.page as string ?? '1')
  const skip = (page - 1) * limit

  const result = await read<MovieResult>(`
    MATCH (m:Movie)-[:IN_GENRE]->(g:Genre {name: $genre})
    RETURN
      g { .* } AS genre,
      toString(size((g)<-[:IN_GENRE]-())) AS count,
      m {
        .tmdbId,
        .title
      } AS movie
    ORDER BY m.title ASC
    SKIP $skip
    LIMIT $limit
  `, {
      genre: name,
      limit: int(limit),
      skip: int(skip)
  })

  res.status(200).json({
    total: parseInt(result[0]?.count) || 0,
    data: result.map(record => record.movie)
  })
}

The function above executes a Cypher statement in a read transaction, processes the results and returns the list of
movies as a JSON response.

A quick GET request to http://localhost:3000/api/genres/Action/movies shows a list of movies:



[
  {
    "tmdbId": "72867",
    "title": "'Hellboy': The Seeds of Creation"
  },
  {
    "tmdbId": "58857",
    "title": "13 Assassins (Jûsan-nin no shikaku)"
  },
  /* ... */
]

This API handler can then be called through a React component in a useEffect hook.



// components/genre/movie-list.tsx
export default function GenreMovieList({ genre }: GenreMovieListProps) {
  const [page, setPage] = useState<number>(1)
  const [limit, setLimit] = useState<number>(10)
  const [movies, setMovies] = useState<Movie[]>()
  const [total, setTotal] = useState<number>()

  // Get data from the API
  useEffect(() => {
    fetch(`/api/genres/${genre.name}/movies?page=${page}&limit=${limit}`)
      .then(res => res.json())
      .then(json => {
        setMovies(json.data)
        setTotal(json.total)
      })


  }, [genre, page, limit])


  // Loading State
  if (!movies || !total) {
    return <div>Loading...</div>
  }

  return (
    <div>
      <ul>
        {movies.map(movie => <li key={movie.tmdbId}>{movie.title}</li>)}
      </ul>

      <p>Showing page {page}</p>

      {page > 1 && <button onClick={() => setPage(page - 1)}>Previous</button>}
      {page * limit < total && <button onClick={() => setPage(page + 1)}>Next</button>}
    </div>
  )
}

The component is then in charge of pagination and any update to the list doesn't re-render the entire page.

Conclusion

This is far from a comprehensive guide to Next.js or Neo4j integrations but hopefully, it serves as a quick reference for anyone wondering the best way to integrate Neo4j, or any other database for that matter, with a Next.js application.

All of the code from this experiment is available on Github.

If you are interested in learning more about Next.js, they have put together a course for developers to learn the basics.

If you would like to learn more about Neo4j, then I would recommend taking a look at the Beginners Neo4j Courses on GraphAcademy. If you want to know more about how to use the Neo4j JavaScript Driver in a Node.js or Typescript project, I would also recommend the Building Neo4j Applications with Node.js course.

If you have any comments or questions, feel free to reach out to me on Twitter.

Adding Record & Type Checking in TypeScript with Generics

Adam Cowley — Thu, 20 Oct 2022 11:02:43 +0000

Recently, while working on a new feature for Neo4j GraphAcademy, I noticed an omission with the Neo4j JavaScript driver in TypeScript projects.

When running a query against the database, although I could define the values returned from the database, it was not possible for TypeScript to check the code for potential errors.

Say I ran the following Cypher statement to get a list of actors in a movie:

const res = await session.executeRead((tx: Transaction) => tx.run(`
  MATCH (p:Person)-[r:ACTED_IN]->(m:Movie {title: $title})
  RETURN p, r, m
  LIMIT 10
`, { title: 'Goodfellas' }))

const people = res.records.map(record => record.get('p'))

Now I know that the people object in the code above would be an array of Node objects. But there's no way that TypeScript would be able to verify that.

If I made a mistake in the code and tried to get a value that didn't exist from each record (for example row.get('somethingElse')), an Error would be thrown at runtime, but I wouldn't be able to catch this while developing - potentially causing hours of debugging pain.

Ideally, this is something that could be caught during the TypeScript evaluation of my code. I wondered how I could add this type of checking to the driver.

A few hours and a coffee later, I had written the code to .

Here's how I did it:

Generics

For anyone who has spent time writing Java, Generics should be a familiar concept. Generics allow you to provide placeholders for types that may not be know up front.

You may have already used a Generic in TypeScript without noticing it - Record<K, V> is a generic which allows you to define the type of keys and values on a JavaScript object, or Map<string, number> would state that the keys in a Map would be strings, and the corresponding values should be a number.

To give another example, in the following code, the T generic allows you to define the data returned by a wrapper function calling an external API:

async function getApiResponse<T>(uri: string):  Promise<T> { /* ... */ }

So I can use this code to fetch a list of Users from an API:

interface User {
  id: number;
  name: string;
}
const users = await getApiResponse<User[]>('/api/users')

But also, use the function to fetch Movie details:

interface Movie {
  id: number;
  title: string;
  released: Date;
}
const movie = await getApiResponse<Movie>(`/api/movies/${id}`)

This is nice, because you don't need to know the type up front when implementing the function, but pass the responsibility on to the developer when they use it.

Generics and Databases

Most database libraries will make use of generics when dealing with values returned from the database. Neo4j is no exception, our driver works by sending a query and receiving results over a protocol called Bolt. When the Driver receives a result back, it will hydrate the records into classes - in Neo4j a result can consist of individual property values, or Neo4j Nodes, Relationships.

In the case of rows in a table, or properties on a Node or Relationship, the values contained can be dynamic.

In JavaScript terms, the properties of a node or relationship are an object ({}), in TypeScript terms a Record<string, any>, so we can use an interface to define the properties.

Each (:Person) node will have two properties; name which is a string and born which is a number.

export interface PersonProperties {
    name: string;
    born: number;
}

As long as the Node class accepts a generic to represent the Properties, TypeScript will be able to inspect the code and ensure that the properties we try to fetch are correct.

class Node<Properties = Record<string, any>> {  // <1>
  public properties: Record<keyof Properties, Properties[keyof Properties];
  /* ... */ 
}

The first line in the block above defines a new generic called Properties which should default to Record<string, any> if none is applied.

If we break down this line:

  public properties: Record<      // <1>
    keyof Properties,             // <2>
    Properties[keyof Properties]  // <3>
  >

properties is a public property, which should always be an object which conforms to the Record generic
The key of each object should be a key from the Properties generic (keyof Properties)
The value of that key will be defined in the Properties (Properties[title] === string)

Then we can define a Person type to be a Node where the properties object matches the PersonProperties interface we defined above.

type Person = Node<PersonProperties>

If I now try to access a property that isn't defined in the interface, TypeScript will pick it up immediately.

person.properties.unknown // <-- Property 'unknown' does not exist on type 'Record<keyof PersonProperties, string | number>'.

So now that we can define the properties of a Node or Relationship, all that is left is to define the shape of the returned record itself. How do we do that? Generics!

Generics and Function return types

As I hinted at in the getApiResponse() example above, you can define the type returned by a function by defining the . Let's take a look at the function definition for getApiResponse():

declare function getApiResponse<ResponseType>(uri: string): Promise<ResponseType>

ResponseType is defined as a generic on the function, and will describe the value that the Promise will resolve to.

In the very first code block, I used a session.run() method to run a Cypher statement. We can add a generic called ResultShape to the function to define the response that is returned.

class Session {
  async run<ResultShape extends Record<string, any>>(
    query: string, 
    params?: Record<string, any>
  ): Promise<Result<ResultShape>> {
    /* ... */
  }
}

The Promise resolves to a Result, a type supplied to driver which wraps a Neo4j result and provides a methods for handling results. We can pass that RecordShape through to the Result class - which has a public records array representing each record.

class Result<RecordShape extends Record<string, any>> {
  constructor(
    public records: Neo4jRecord<RecordShape>[] = []
  ) {}
  /* ... */
}

The type of each item in records should correspond to a Neo4jRecord - a wrapper class for the raw result and a get() method for accessing an individual value from the record (eg, p for the Person node).

class Neo4jRecord<RecordShape extends Record<string, any>> {
  constructor(
    public readonly record: Record<keyof RecordShape, RecordShape[keyof RecordShape]>
  ) { }
  /* ... */
}

Now, we can use that RecordShape to check the key parameter passed to the get() function:

get<Key extends keyof RecordShape>(key: Key): RecordShape[Key] {
  return this.record.get(key)
}

If key is not a key of the RecordShape, the Typescript evaluator will throw an error. Perfect!

The return type (RecordShape[Key]) will also inspect the RecordShape and interpret the type of the returned value, meaning that if an incorrect type is defined in the map function, the error will be picked up straight away:

res.records.map<Node<PersonProperties>>(row => row.get('m'))
// Type 'Neo4jNode<MovieProperties, any>' is not assignable
// to type 'Neo4jNode<PersonProperties, any>'.
//   Types of property 'properties' are incompatible.

The m value is a Movie and not a Person!

Bringing it together

So now, I can define types for my database objects that the database will return:

/**
 * Node & Relationship Properties
 */
export interface PersonProperties {
    name: string;
    born: number;
}

export interface MovieProperties {
    title: string;
    released: number;
}

export interface ActedInProperties {
    actedIn: string[];
}

/**
 * My Query
 */
export interface PersonActedInMovie {
    person: Neo4jNode<PersonProperties>;
    actedIn: Neo4jRelationship<ActedInProperties>;
    movie: Neo4jNode<MovieProperties>;
}

Then for my Query (note the person, actedIn and movie items in my RETURN statement):

const query = `
  MATCH (person:Person)-[actedIn:ACTED_IN]->(movie:Movie)
  RETURN person, actedIn, movie LIMIT 10
`

I can code with peace of mind that as long as my Cypher query is correct, my application will work as expected:

// Run a query and return an array of PersonActedInMovie records
const res = await session.run<PersonActedInMovie>(query)

// Extract Movies
const movies = res.records.map(row => row.get('movie')) // <-- Fine, movie is in result

// Type checking on values
const people = res.records.map<Neo4jNode<PersonProperties>>(row => row.get('movie')) // <-- A Person is not a Movie

// Type checking on Node and Relationship properties
movies.map<string>(movie => movie.properties.title)   // <-- Fine, title is a string
movies.map<string>(movie => movie.properties.born)    // <-- Type 'number' is not assignable to type 'string'.
movies.map<string>(movie => movie.properties.unknown) // <-- Property 'unknown' does not exist on type

Wrapping things up

This post was intended to outline a real-world application of Typescript Generics and detail when and why you might use them.

You can see the example code in full on the TypeScript Playground

If you have any comments or questions, feel free to reach out to me on Twitter.

Also: If you are a Neo4j user, don't use session.run() in production. If you want to know why, or just want to learn more about how to use the Neo4j JavaScript Driver in a Node.js or Typescript project, check out Building Neo4j Applications with Node.js

How to build a Neo4j Application with Go

Adam Cowley — Wed, 23 Mar 2022 14:00:21 +0000

Are you a Go developer tasked with learning Neo4j, or are you just interested in learning something new?

If this sounds like you (or even if it doesn't), then the Building Neo4j Applications with Go course on GraphAcademy is for you!

This is a course that I have been working hard on for the past month, and I'm happy to share it with you now.

What is GraphAcademy?

Neo4j GraphAcademy is our free, self-paced, hands-on online learning platform.

We on the Neo4j DevRel team have been working hard to build a brand new website and course curriculum that provides a fun, engaging and hands-on learning experience. You can read more about the changes that we have made on the Neo4j Developer blog.

About the course

In the course, you will learn all about the Neo4j Go Driver by adding it into an existing project, and then modifying a set of services to interact with Neo4j.

You will learn all about database sessions, read and write transactions, how to execute a Cypher query and handle results.

At the end of the course, the accompanying UI will be populated with data held in a Neo4j Sandbox](https://sandbox.neo4j.com) instance.

Assumptions

We assume that you have a prior working knowledge of Go. We also assume that you have some previous experience of working with Neo4j.

If you have no prior experience with Neo4j, you can follow the Beginners Learning Path. There are four courses designed to teach you the basics in approximately six hours.

What you will learn

The course is split into three modules which will guide you from complete beginner to expert.

In the first module, Project Setup, you are guided through setting up the project and registering configuration variables so that the API can communicate with the Neo4j Sandbox instance created during the enrolment process.

In the second module, The Neo4j Go Driver, you will learn all about the Neo4j Driver and how it should be used within an application. This included installing the github.com/neo4j/neo4j-go-driver/v4 dependency using go get, building a connection string, creating a Driver instance using the NewDriver() method and verifying that the credentials used to create the driver instance were correct.

The third module, Interacting with Neo4j teaches you about the Driver lifecycle; how to open new database sessions, execute read and write transactions and how to consume the results. The module also teaches you how to handle potential errors thrown by the driver. Throughout this module, you will be challenged to modify the existing project and run tests to verify that the code has been written correctly.

The final module allows you to practise the skills learnt in the previous three modules by implementing the remaining features.

At the end of the course, you will have a working API that serves data to the SPA included in the project. The API will allow you to register a new user and sign in, browse the movie catalogue, rate movies and create a list of favourite movies.

The course is free of charge and there is no time limit to complete the course!

Enrol Now

You can enrol now, for free, by registering for GraphAcademy and clicking Enroll Now on the Building Neo4j Applications with Go page.

If you have any comments or feedback you can get in touch with me on Twitter or use the feedback widgets on each page.

Good luck!

How to build a Neo4j Application with Node.js

Adam Cowley — Wed, 01 Dec 2021 12:39:58 +0000

Are you a Node.js developer tasked with learning Neo4j, or are you just interested in learning something new?

If this sounds like you (or even if it doesn't), then the Building Neo4j Applications with Node.js course on GraphAcademy is for you!

This is a course that I have been working hard on for the past month, and I'm happy to share it with you now.

What is GraphAcademy?

Neo4j GraphAcademy is our free, self-paced, hands-on online learning platform.

About the course

In the course, you will learn all about the Neo4j JavaScript Driver by adding it into an existing project, and then modifying a set of services to interact with Neo4j.

You will learn all about database sessions, read and write transactions, how to execute a Cypher query and handle results.

At the end of the course, the accompanying UI will be populated with data held in a Neo4j Sandbox](https://sandbox.neo4j.com) instance.

Assumptions

We assume that you have a prior working knowledge of JavaScript, NPM and the Node.js ecosystem. We also assume that you have some previous experience of working with Neo4j.

If you have no prior experience with Neo4j, you can follow the Beginners Learning Path. There are four courses designed to teach you the basics in approximately six hours.

What you will learn

The course is split into three modules which will guide you from complete beginner to expert.

In the first module, Project Setup, you are guided through setting up the project and registering environment variables so that the API can communicate with the Neo4j Sandbox instance created during the enrolment process.

In the second module, The Neo4j JavaScript Driver, you will learn all about the Neo4j Driver and how it should be used within an application. This included installing the neo4j-driver dependency using npm, building a connection string, creating a Driver instance using the driver() method and verifying that the credentials used to create the driver instance were correct.

The final module allows you to practise the skills learnt in the previous three modules by implementing the remaining features.

The course is free of charge and there is no time limit to complete the course!

Enrol Now

You can enrol now, for free, by registering for GraphAcademy and clicking Enroll Now on the Building Neo4j Applications with Node.js page.

If you have any comments or feedback you can get in touch with me on Twitter or use the feedback widgets on each page.

Good luck!

How to Build Authentication into a Vue3 Application

Adam Cowley — Mon, 28 Sep 2020 13:40:45 +0000

How to Build an Authentication into a Vue3 Application

I've recently started a livestream on the Neo4j Twitch Channel about building Web Applications with Neo4j and TypeScript, working on an example project for Neoflix - a fictional streaming service.

I've been a long time user of Vue.js, but without proper TypeScript support, I was finding it hard to justify building a Vue-based front end as part of the Stream, after all Vue2's TypeScript support seemed to be lacking. My only real option was Angular, and I got frustrated by that pretty quickly.

With Vue v3's official release last week, along with improved TypeScript support, it gave me a good excuse to experiment and see how I could encorporate this into the Neoflix project.

Vue 3 and the Composition API

One drawback to Vue 2 was the increased complexity as an application grew, the re-use of functionality and readability of components becomes a problem. One example I've seen mentioned a few times is the problem of Sorting results or Pagination. In a Vue2 application, your options were either to duplicate the functionality across components or use a Mixin. The drawback of a Mixin is that it's still not clear what data and methods are bound to a component.

The new Composition API allows us to extract repeatable elements into their own files which can be used across components in a more logical way.

The new setup function on each component gives you a convenient way to import and reuse functionality. Anything returned from the setup function will be bound to the component. For the search & pagination example, you could write a composition function to perform the specific logic for retrieving search results, while another composition function would provide more generic functionality required to implement previous and next buttons in the UI:


 ts
export default defineComponent({
  setup() {
    const { loading, data, getResults } = useSearch()
    const { nextPage, previousPage } = usePagination()

    // Anything returned here will be available in the component - eg this.loading
    return { loading, data, getResults, nextPage, previousPage }
  }
})

Compared to Vue 2's Mixins, the setup function allows you to quickly see which properties and methods are bound to the component without opening multiple files.

The official documentation has [a great write up on the Composition API and there is a great Vue Mastery video on the Composition API which explain the problem and solution well.

I will assume that you've watched the video and read the docs and will jump straight into a concrete example - Authentication.

The Authentication Problem

Authentication is a problem that many apps will have to overcome. A User may be required to provide their login credentials in order to view certain pages on a site or subscribe to access certain features.

In the case of Neoflix, Users are required to register and purchase a subscription before they can view or stream the catalogue of Films and TV Shows. A HTTP POST request to /auth/register will create a new account, and a POST request to /auth/login will issue the user with a JWT token which will be passed to each request.

Managing State Composition Functions

As the Users details will be required across multiple components, we will need to save this to the application's global state. On researching the differences between versions 2 and 3, I came across an article that explains that Vuex may not be required for global state management in Vue 3 which will cut down the number of dependencies.

This pattern feels a lot like React Hooks where you call a function to create a reference and a setter function, then use reference within the render function.

The article provides this code example to explain how it works:


 ts
import { reactive, provide, inject } from 'vue';

export const stateSymbol = Symbol('state');
export const createState = () => reactive({ counter: 0 });

export const useState = () => inject(stateSymbol);
export const provideState = () => provide(
  stateSymbol,
  createState()
);

You can use the inject function to register a state object using a symbol, then use the provide function to recall the state later on.

Or more simply, you can just create a reactive variable and then return it within a function along with any methods required to manipulate the state:



import { ref } from 'vuex'

const useState = () => {
  const counter = ref(1)

  const increment = () => counter.value++
}

const { counter, increment } = useState()
increment() // counter will be 2

The whole use[Something] pattern feels a little React Hook-like, which at the start made me feel a little like "If I wanted to use Hooks then I could just use React" - but that thought has faded over time and now it makes sense.

API Interactions

In order to interact with API, we will use the axois package.



npm i --save axios

We can create an API instance with some basic config which will be used across the application.


 ts
// src/modules/api.ts
export const api = axios.create({
  baseURL: process.env.VUE_APP_API || 'http://localhost:3000/'
})

Better yet, to avoid duplicating the code required to call the API, we could create a composition function that we could use for all API interactions across the application. To do this we can create a provider function that exposes some useful variables that will be useful to handle loading state inside any component:

loading: boolean - An indicator to let us know if the hook is currently loading data
data: any - Once the data has been loaded, update the property
error?: Error - If anything goes wrong, it would be useful to show display the error message within the API

In order for a component update on the change of a variable, we need to create a reference to a reactive variable. We can do this by importing the ref function. The function takes a single optional argument which is the initial state.

For example, when we use this hook, the loading state should be true by default and set to false once the API call succeeds. The data and error variables will be undefined until the request completes.

We can then return those variables in an object in order to deconstruct them within the component's setup function.


 ts
// src/modules/api.ts
import { ref } from 'vue'

export const useApi(endpoint: string) => {
  const loading = ref(true)
  const data = ref()
  const error = ref()

  // ...
  return {
    loading, data, error
  }
}

To update these variables, you set .value on the reactive object - for example loading.value = false.

We can then create some computed variables to use within the component using the computed function exported from the Vue. For example, if the API returns an error we can use a computed errorMessage property to extract the message or details from the API response.


 ts
// src/modules/api.ts
import { ref, computed } from 'vue'

const errorMessage = computed(() => {
  if (error.value) {
    return error.value.message
  }
})

const errorDetails = computed(() => {
  if ( error.value && error.value.response ) {
    return error.value.response.data.message
  }
})

On validation error, Neoflix's Nest.js API returns a 400 Bad Request which includes the individual errors in an array. These can be extracted and converted into an object using Array.reduce:


 ts
const errorFields = computed(() => {
  if (error.value && Array.isArray(error.value.response.data.message)) {

    return (error.value.response.data.message as string[]).reduce((acc: Record<string, any>, msg: string) => {
      let [ field ] = msg.split(' ')

      if (!acc[field]) {
        acc[field] = []
      }

      acc[field].push(msg)

      return acc
    }, {}) // eg. { email: [ 'email is required' ] }
  }
})

Finally, we can create a method to wrap a GET or POST request and update the reactive variables on success or error:


 ts
const post = (payload?: Record<string, any>) => {
  loading.value = true
  error.value = undefined

  return api.post(endpoint, payload)
    // Update data
    .then(res => data.value = res.data)
    .catch(e => {
      // If anything goes wrong, update the error variable
      error.value = e

      throw e
    })
    // Finally set loading to false
    .finally(() => loading.value = false)
}

Putting it all together, the function will look like this:


 ts
// src/modules/api.ts
export const useApi(endpoint: string) => {
  const data = ref()
  const loading = ref(false)
  const error = ref()

  const errorMessage = computed(() => { /* ... */ })
  const errorDetails = computed(() => { /* ... */ })
  const errorFields = computed(() => { /* ... */ })

  const get = (query?: Record<string, any>) => { /* ... */ }
  const post = (payload?: Record<string, any>) => { /* ... */ }

  return {
    data, loading, error,
    errorMessage, errorDetails, errorFields,
    get, post,
  }
}

Now we have a hook that can be used across the application when we need to send a request to the API.

Registering a User

The POST /auth/register endpoint requires an email, password, date of birth and optionally accepts a first name and last name. As we're building a TypeScript application we can define this as an interface which will ensure the code is consistent:


 ts
// src/views/Register.vue
interface RegisterPayload {
  email: string;
  password: string;
  dateOfBirth: Date;
  firstName?: string;
  lastName?: string;
}

In Vue 3, you cann the defineComponent rather than returning a plain Object. In this case, we have one function, setup which uses the composition function to create an API.

As part of the setup function, we can call useApi to interact with the API. In this case we want to send a POST request to /auth/register so we can use the useApi function above to extract the variables required in the component.


 ts
// src/views/Register.vue
import { useApi } from '@/modules/api'

export default defineComponent({
  setup() {
    // Our setup function
    const {
      error,
      loading,
      post,
      data,
      errorMessage,
      errorDetails,
      errorFields,
    } = useApi('/auth/register');

    // ...

    return {
      error,
      loading,
      post,
      data,
      errorMessage,
      errorDetails,
      errorFields,
    }
  },
});

The post method from our useApi hook requires a payload, so we can initialise these in the setup function. Previously, we used the ref function to create individual reactive properties but this can get a little unweildy when deconstructing.

Instead, we can use the reactive function exported from vue - this will save us the trouble of calling .value on each property when passing it to the post function. When passing these to the component, we can turn them back into reactive properties using the toRefs function.


 ts
// src/views/Register.vue
import { reactive, toRefs } from 'vue'

const payload = reactive<RegisterPayload>({
  email: undefined,
  password: undefined,
  dateOfBirth: undefined,
  firstName: undefined,
  lastName: undefined,
});

// ...

return {
  ...toRefs(payload), // email, password, dateOfBirth, firstName, lastName
  error,
  loading,
  post,
  data,
  errorMessage,
  errorDetails,
  errorFields,
}

We can then create a submit method which can be used within component to trigger the request to the API. This will call the post method exported from useApi , which under the hood fires the request and updates error , loading and post .


 ts
const submit = () => {
  post(payload).then(() => {
    // Update user information in global state

    // Redirect to the home page
  });
};

I will omit the entire <template> portion of this query but the variables are used in the same way as a Vue 2 application. For example, the email and password are assigned to inputs using v-model and the submit function can be assigned to the @submit event on the <form> tag.


 html
<form @submit.prevent="send">
    <input v-model="email" />
    <input v-model="password" />
    <!-- etc... -->
</form>

View the component code here...

Saving the User into Global state

In order to use the user's authentication details across the application, we can create another hook which references a global state object. Again, this is typescript so we should create interfaces to represent the state:


 ts
// src/modules/auth.ts
interface User {
    id: string;
    email: string;
    dateOfBirth: Date;
    firstName: string;
    lastName: string;
    access_token: string;
}

interface UserState {
    authenticating: boolean;
    user?: User;
    error?: Error;
}

The next step is to create an initial state for the module:


 ts
// src/modules/auth.ts
const state = reactive<AuthState>({
    authenticating: false,
    user: undefined,
    error: undefined,
})

We can then create a useAuth function which will provide the current state and methods for setting the current user once successfully authenticated or unsetting the user on logout.


 ts
// src/modules/auth.ts
export const useAuth = () => {
  const setUser = (payload: User, remember: boolean) => {
    if ( remember ) {
      // Save
      window.localStorage.setItem(AUTH_KEY, payload[ AUTH_TOKEN ])
    }

    state.user = payload
    state.error = undefined
  }

  const logout = (): Promise<void> => {
    window.localStorage.removeItem(AUTH_KEY)
    return Promise.resolve(state.user = undefined)
  }

  return {
    setUser,
    logout,
    ...toRefs(state), // authenticating, user, error
  }
}

We can then piece the component together using these functions:



// src/views/Register.vue
import { useRouter } from 'vue-router'
import { useApi } from "../modules/api";
import { useAuth } from "../modules/auth";

// ...
export default defineComponent({
  components: { FormValidation, },
  setup() {
    // Reactive variables for the Register form
    const payload = reactive<RegisterPayload>({
      email: undefined,
      password: undefined,
      dateOfBirth: undefined,
      firstName: undefined,
      lastName: undefined,
    });

    // State concerning the API call
    const {
      error,
      loading,
      post,
      data,
      errorMessage,
      errorDetails,
      errorFields,
      computedClasses,
    } = useApi("/auth/register");

    // Function for setting the User
    const { setUser } = useAuth()

    // Instance of Vue-Router
    const router = useRouter()

    const submit = () => {
      // Send POST request to `/auth/register` with the payload
      post(payload).then(() => {
        // Set the User in the Auth module
        setUser(data.value, true)

        // Redirect to the home page
        router.push({ name: 'home' })
      })
    }


    return {
      ...toRefs(payload),
      submit,
      loading,
      errorMessage,
      errorFields,
      errorDetails,
      computedClasses,
    }
  }
})

Remembering the User

The auth module above uses window.localStorage to save the user's access token (AUTH_TOKEN) - if the user returns to the site, we can use that value when the user next visits the site to re-authenticate them.

In order to watch for a change of a reactive variable, we can use the watch function. This accepts two arguments; an array of reactive variables and a callback function. We can use this to call the /auth/user endpoint to verify the token. If the API returns a valid response, we should set the user in the global state, otherwise remove the token from local storage.



// src/modules/auth.ts
const AUTH_KEY = 'neoflix_token'

const token = window.localStorage.getItem(AUTH_KEY)

if ( token ) {
  state.authenticating = true

  const { loading, error, data, get } = useApi('/auth/user')

  get({}, token)

  watch([ loading ], () => {
    if ( error.value ) {
      window.localStorage.removeItem(AUTH_KEY)
    }
    else if ( data.value ) {
      state.user = data.value
    }

    state.authenticating = false
  })
}

Login

The setup function for the login component is almost identical, except we are calling a different API endpoint:



const {
  loading,
  data,
  error,
  post,
  errorMessage,
  errorFields
} = useApi("auth/login")

// Authentication details
const { setUser } = useAuth();

// Router instance
const router = useRouter();

// Component data
const payload = reactive<LoginPayload>({
  email: undefined,
  password: undefined,
  rememberMe: false,
});

// On submit, send POST request to /auth/login
const submit = () => {
  post(payload).then(() => {
    // If successful, update the Auth state
    setUser(data.value, payload.rememberMe);

    // Redirect to the home page
    router.push({ name: "home" });
  });
};

return {
  loading,
  submit,
  errorMessage,
  ...toRefs(payload),
};

Using the Data in a Component

To use the User's information inside a component we can import the same useAuth function and access the user value.

For example, we may want to add a personalised welcome message to the top navigation.

The user's first name is not required during the Neoflix registration, so we can use the computed function to return a conditional property. If the user has a firstName we will display a Hey, {firstName} message, otherwise fall back to a generic Welcome back! message.



// src/components/Navigation.vue
import { computed, defineComponent } from "vue";
import { useAuth } from "../modules/auth";

export default defineComponent({
  setup() {
    const { user } = useAuth()

    const greeting = computed(() => {
      return user?.value && user.value.firstName
        ? `Hey, ${user.value.firstName}!`
        : 'Welcome back!'
    })

    return { user, greeting }
  }
})

Logging Out

We've already added a logout method to the return of useAuth. This can be called from the setup method of a new component to clear the user's information and redirect them back to the login page.



// src/views/Logout.vue
import { defineComponent } from "vue"
import { useRouter } from "vue-router"
import { useAuth } from "../modules/auth"

export default defineComponent({
  setup() {
    const { logout } = useAuth()
    const router = useRouter()

    logout().then(() => router.push({ name: 'login' }))
  }
})

Protecting Routes

In this application, the user should be restricted to the login or register routes unless they are logged in. As we are using vue-router in this application we can use Route Meta Fields to define which routes should be protected:



// src/router/index.ts
const routes = [
  {
    path: '/',
    name: 'home',
    component: Home,
    meta: { requiresAuth: true },
  },
  // ...
}

If requiresAuth is set to true, we should check the user provided by useAuth. If the user has not been set, we should return redirect the user to the login page.

We can work out whether the user is logged in by accessing the user object returned by useAuth. If the current route's meta data indicates that the route is restricted, we should redirect them to the login page.

Conversely, if a user is on the login or register page but has already logged in we should redirect them back to the home page.



// src/router/index.ts

router.beforeEach((to, from, next) => {

  const { user } = useAuth()

// Not logged into a guarded route?

  if ( to.meta.requiresAuth && !user?.value ) next({ name: 'login' })

// Logged in for an auth route

  else if ( (to.name == 'login' || to.name == 'register') && user!.value ) next({ name: 'home' })

// Carry On...

  else next()

})

Conclusion

The more I get used to the new Composition API, the more I like it. It is still early days and the aren't a lot of examples around for Vue 3, so it may emerge at some point that the content in this post is not the best way to do things. If you are doing things differently, let me know in the comments.

I will be building out the application as part of my livestream on the Neo4j Twitch Channel. Join me every Tuesday at 13:00 BST, 14:00 CEST or catch up with the videos on the on the Neo4j YouTube Channel.

All of the code built during the stream is available on Github.

Building a Modern Web Application with Neo4j and NestJS

Adam Cowley — Thu, 02 Jul 2020 08:52:46 +0000

This article is the introduction to a series of articles and a Twitch stream on the Neo4j Twitch channel where I build an application on top of Neo4j with NestJS and a yet-to-be-decided Front End. This week I built a Module and Service for interacting with Neo4j.

TL;DR: I've pushed the code to Github and created a Neo4j module for NestJS to save you some time.

Over the past few weeks I have been spending an hour live streaming something that I have found interesting that week, but from this week I thought I would change things up and start to build out a project on Neo4j.

Tech Stack

Neo4j

If you're subscribed to this channel then you are likely familiar with Neo4j, but if not then Neo4j is the world's leading Graph Database. Rather than tables or documents, Neo4j stores it's data in Nodes - those nodes are categorised by labels and contain properties as key/value pairs. Those Nodes are connected together by relationships, which are categorised by a type and can also contain properties as key/value pairs.

(a:Person {name: "Adam"})-[:USES_DATABASE {since: 2015}]->(neo4j:Database:GraphDatabase {name: "Neo4j", homepage: "neo4j.com"})

What sets Neo4j apart from other databases is it's ability to query connected datasets. Where traditional databases build up joins between records at read time, Neo4j stores the data

Neo4j is schema-optional - meaning that you can enforce a schema on your database if necessary by adding unique or exists constraints on Nodes and Relationships.

Typescript

I've been experimenting with Typescript for a while now, and the more I use it the more I like it.

Typescript is essentially Javascript but with additional static typing. Under the hood, it compiles down to plain Javascript but it improves the developer experience a lot, and allows you to identify problems in real-time as you are writing your code.

NestJS

By far the best framework I have seen that supports typescript is NestJS. NestJS is an opinionated framework for building server-side applications. It also includes modern features you'd expect in a modern framework like Spring Boot or Laravel - mainly Dependency Injection.

Week 1 - Nest fundamentals & Neo4j Integration

Nest comes with a CLI with many helpers for starting and developing a project. You can install it by running:

npm i --global @nestjs/cli
nest --help

Once it's installed, you can use the new or n command to create a new project.

nest new api

After selecting the package manager of your choice, the CLI command will generate a new project and install any dependencies. Once it's done, you can cd into the directory and then run npm run start:dev to fire up the development server.

In the generated src/ folder, you'll see:

main.ts - The main entrypoint of the file, this creates a Nest application instance
app.module.ts - This is the root module of the application, where you define the child modules that are used in the application
app.controller.ts - This is a basic controller, where you can define REST endpoints on the server

Nest modules

Functionality in Nest is grouped into modules, the official documentation uses Cats as it's example. Modules are a way of grouping related functionality together. In the Cats example, the module provides a CatsService which handles the applications interactions with Cats, and a Cats controller which registers routes which define how the Cats are accessed.

Module classes are defined by a @Module annotation, which in turn defines which child modules are imported into module, any controllers that are defined in the module, and any classes that are exported from the module and made available for dependency injection.

Take the annotation on the Cats example in the documentation, this is saying that the CatsModule registers a single controller CatsController and provides the CatsService.

@Module({
  controllers: [CatsController],
  providers: [CatsService],
})

The CatService is registered with the Nest instance and can then be injected into any class.

`@Injectable()` classes

Classes annoted with @Injectable() are automatically injected into a class using some under-the-hood Nest "magic". For example, by defining the CatsService in the constructor for the CatsController, Nest will automatically resolve this dependency and inject it to the class without any additional code.

This is identical to how things work in more mature frameworks like Spring and Laravel.

import { Controller } from '@nestjs/common';
import { CatsService } from './cats.service';

@Controller
export class CatsController {
  constructor(private catsService: CatsService) {}
}

Dependency Injection is a software technique where a class will be "injected" with instances of other classes that it depends on. This makes the testing process easier where instead of instantiating classes. It also promotes the principles of DRY - don't repeat yourself and SOLID. Each class should have a single responsibility - for example a User service should only be concerned with acting on a User's record, not be concerned with how that record is persisted to a database.

Nest Integration for Neo4j

In order to use Neo4j in services across the application, we can define a Neo4jService for interacting with the graph through the JavaScript driver. This service should provide the ability to interact with Neo4j but without the service itself needing to know any of the internals. This service should be wrapped in a module which can be registered in the application.

The first step is to install the Neo4j Driver.

npm i --save neo4j-driver

Then, we can use CLI to generate a new module with the name Neo4j.

nest g mo neo4j # shorthand for `nest generate module neo4j`

The command will create a neo4j/ folder with it's own module. Next, we can use the CLI to generate the Service:

nest g s neo4j # shorthand for `nest generate service neo4j`

This command will generate neo4j.service.ts and append it to the providers array in the module so it can be injected into any application that uses the module.

Configuration & Dynamic Modules

By default, these modules are registerd as static modules. In order to add configuration to the driver, we'll have add a static method which accepts the user's Neo4j credentials and returns a DynamicModule.

The first thing to do is generate an interface that will define the details allowed when instantiating the module.

nest g interface neo4j-config

The driver takes a connection string and an authentication method. I like to split up the connection string into parts, this way we can validate the scheme.

The scheme (or protocol) at the start of the URI should be a string, and one of the following options:

export type Neo4jScheme = 'neo4j' | 'neo4j+s' | 'neo4j+scc' | 'bolt' | 'bolt+s' | 'bolt+scc'

The host should be a string, port should either be a number or a string, then username, password should be a string. The database should be an optional string, if the driver connects to a 3.x version of Neo4j then this isn't a valid option and if none is supplied then the driver will connect to the default database (as defined in neo4j.conf - dbms.default_database).

export interface Neo4jConfig {
    scheme: Neo4jScheme;
    host: string;
    port: number | string;
    username: string;
    password: string;
    database?: string;
}

Next, for the static method which registers the dynamic module. The documentation recommends using the naming convention of forRoot or register. The function should return a DynamicModule - this is basically an object that contains metadata about the module.

The module property should return the Type of the module - in this case Neo4jModule. This module will provide the Neo4jService so we can add the class to the provides array.

// ,,
export class Neo4jModule {
    static forRoot(config: object): DynamicModule {
        return {
            module: Neo4jModule,
            provides: [
                Neo4jService,
            ]
        }
    }

    // ,,
}

Because we are providing a configuration object, we'll need to register it as a provider so that it can be injected into the Neo4jService. For providers that are not defined globally, we can define a unique reference to the provider and assign it to a variable. We will use this later on when injecting the config into the service. The useValue property instructs Nest to use the config value provided as the first argument.

// Reference for Neo4j Connection details
const NEO4J_OPTIONS = 'NEO4J_OPTIONS'

export class Neo4jModule {
    static forRoot(config: object): DynamicModule {
        return {
            module: Neo4jModule,
            provides: [
                {
                    // Inject this value into a class @Inject(NEO4J_OPTIONS)
                    provide: NEO4J_OPTIONS,
                    useValue: config
                },
                Neo4jService,
            ],
        }
    }

    // ,,
}

If the user supplies incorrect credentials, we don't want the application to start. We can create an instance of the Driver and verify the connectivity using an Asynchronous provider. An async provider is basically a function that given a set of configuration parameters, returns an instance of the module that is configured at runtime.

In a new file neo4j.utils.ts, create an async function to create an instance of the driver and call the verifyConnectivity() to verify that the connection has been successful. If this function throws an Error, the application will not start.

import neo4j from 'neo4j-driver'
import { Neo4jConfig } from './interfaces/neo4j-config.interface'

export const createDriver = async (config: Neo4jConfig) => {
    // Create a Driver instance
    const driver = neo4j.driver(
        `${config.scheme}://${config.host}:${config.port}`,
        neo4j.auth.basic(config.username, config.password)
    )

    // Verify the connection details or throw an Error
    await driver.verifyConnectivity()

    // If everything is OK, return the driver
    return driver
}

The function accepts the Neo4jConfig object as the only argument. Because this has already been defined as a provider, we can define it in the injects array when defining it as a provider.

// Import the factory function
import { createDriver } from './neo4j.utils.ts'

// Reference for Neo4j Driver
const NEO4J_DRIVER = 'NEO4J_DRIVER'

export class Neo4jModule {
    static forRoot(config: object): DynamicModule {
        return {
            module: Neo4jModule,
            provides: [
                {
                    provide: NEO4J_OPTIONS,
                    useValue: options
                },
                {
                    // Define a key for injection
                    provide: NEO4J_DRIVER,

                    // Inject NEO4J_OPTIONS defined above as the
                    inject: [NEO4J_OPTIONS],

                    // Use the factory function created above to return the driver
                    useFactory: async (config: Neo4jOptions) => createDriver(config)
                },
                Neo4jService,
            ],
        }
    }
}

Now that the driver has been defined, it can be injected into any class in it's own right by using the @Inject() annotation. But in this case, we will add some useful methods to the Neo4jService to make it easier to read from and write to Neo4j. Because we have defined NEO4J_DRIVER in the provides array for the dynamic module, we can pass the NEO4J_DRIVER as a single parameter to the @Inject directive in the constructor.

import { Injectable, Inject } from '@nestjs/common';
import { NEO4J_DRIVER } from './neo4j.constants'

@Injectable
export class Neo4jService {
    constructor(
        @Inject(NEO4J_CONFIG) private readonly config,
        @Inject(NEO4J_DRIVER) private readonly driver
    ) {}
}

Each Cypher query run against Neo4j takes place through a Session, so it makes sense to expose this as an option from the service. The default access mode of the session allows the Driver to route the query to the right member of a Causal Cluster - this can be either READ or WRITE. There is also an optional parameter for the database when using multi-tenancy in Neo4j 4.0. As I mentioned earlier, if none is supplied then the query is run against the default database.

So the user doesn't need to worry about the specifics of read or write transactions, we should create a method for each mode - both with an optional parameter for the database. There is also a database specified in the Neo4jConfig object, so we should fall back to this if none is explicitly specified.

import { Driver, Session, session, Result } from 'neo4j-driver'
//...

export class Neo4jService {
    constructor(@Inject(NEO4J_DRIVER) private readonly driver) {}

    getReadSession(database?: string): Session {
        return this.driver.session({
            database: database || this.config.database,
            defaultAccessMode: session.READ,
        })
    }

    getWriteSession(database?: string): Session  {
        return this.driver.session({
            database: database || this.config.database,
            defaultAccessMode: session.WRITE,
        })
    }

}

These methods make use of NEO4J_CONFIG and NEO4J_DRIVER which were injected into the constructor.

So with that in mind, it would be useful to create a method to read data from Neo4j. The driver accepts parameterised queries as a string (eg. queries with literal variables replaced with parameters - $myParam) and an object of parameters so these will be the arguments for the query. Optionally, we may want to specify which database this query is run against so it makes sense to include that as an optional third parameter. The query then returns a Result statement which includes the result and some additional statistics.

read(cypher: string, params: Record<string, any>, database?: string): Result {
    const session = this.getReadSession(database)
    return session.run(cypher, params)
}

Over the course of the application, this will save us a few lines of code. The same can be done for a write query:

write(cypher: string, params: Record<string, any>, database?: string): Result {
    const session = this.getWriteSession(database)
    return session.run(cypher, params)
}

Using the Service in the Application

Now we have a service that is registered in the main application through the Neo4jModule that can be injected into any class in the application. So as an example, let's modify the Controller that was generated in the initial command. By default, the route at '/' returns a hello world message, but instead let's use it to return the number of Nodes in the database.

To do this, we should first inject the Neo4jService into the controller:

import { Controller, Get } from '@nestjs/common';
import { Neo4jService } from './neo4j/neo4j.service'

@Controller()
export class AppController {
  constructor(private readonly neo4jService: Neo4jService) {}
    // ...
}

Now, we can modify the getHello method to return a string. The constructor will automatically assign the Neo4jService to the class so it is accessible through this.neo4jService. From there we can use the .read() method that we've just created to execute a query against the database.

async getHello(): Promise<any> {
    const res = await this.neo4jService.read(`MATCH (n) RETURN count(n) AS count`)

    return `There are ${res.records[0].get('count')} nodes in the database`
}

Navigating in the browser to http://localhost:3000 should now show a message including the number of nodes in the database.

Tune in next week!

Tune in to the Neo4j Twitch channel Tuesdays at 13:00BST, 14:00CEST for the next episode.

Building a Real-Time UI on top of Neo4j with Vue.js and Kafka

Adam Cowley — Fri, 28 Feb 2020 09:39:01 +0000

In this post, I will demonstrate how you can build a real-time application on top of Neo4j using a few open source plugins. Due to brevity, this will only be a basic application but you can take the principles and make it as complicated as you need.

For this example, I will also be using Vue.js on top of an express server communicating through websockets. If you prefer something like React then it will work just as well.

Also, I'm using neode - a nodejs based OGM that I wrote for Neo4j. Again, you could quite easily swap this out for another OGM or just run cypher queries against Neo4j using the official driver.

With the disclaimers out of the way, let's get going.

Overview

An idea that I've had at the back of my mind for a while is to build a collaborative Graph Database modelling tool. For this, real-time information will have to be passed through to all users that are logged in so a user doesn't overwrite what another user is doing. For this, I will need to lock and unlock nodes in real-time and pass the message to a user.

When a user selects a Node in the UI, a lock event should be sent to the API via websockets. This status update should be distributed to the other users, and that node locked in their front end
As the user drags the Node, a drag event should be sent with the x/y coordinates of the node so this can be reflected in what each other user sees.
When the Node is dropped, the node should be unlocked for all users by a lock event

Creating a basic express API

Firstly, we need an API that we can send these messages. The API will be in charge of saving the updates to Neo4j and redistributing the message to all of the other clients. The first step is to create a new directory which will hold the code, initiate the project and install the required dependencies.

mkdir api && cd $_
npm init -y
npm i -S express neode kafka-node socket.io

express - This is an unopinionated framework for building REST APIs
neode - An OGM that I'll use to take care of the legwork when communicating with Neo4j.
kafka-node - A library for interacting with Kafka. This package allows you to create Producers and Consumers in node.
socket.io - A library that enables communication over websockets

For now, I'll create a server that socket.io and the kafka consumer will bind to.

api/src/index.js

const express = require('express')
const app = express()

app.get('/', (req, res) => res.send('Hello!'))

const server = app.listen(3000, () => console.log('Listening on http://localhost:3000'))

Running node index.js and heading to http://localhost:3000, you should now see a page saying Hello!.

Next step, setting up neode. For this, I will use the fromEnv function included with neode. This makes life a little easier by allowing you to define the neo4j connection details in a .env file. We can also use this for the kafka consumer later on.

api/.env

NEO4J_PROTOCOL=bolt
NEO4J_HOST=localhost
NEO4J_USERNAME=neo4j
NEO4J_PASSWORD=mysecretpassword
NEO4J_PORT=7687

To load these settings into process.env, you call the .config() function supplied by dotenv.

// Load the .env variables into process.env
require('dotenv').config()

// Accessing the variable
console.log(process.env.NEO4J_HOST) // "localhost"

Data Persistence with Neode

Next, I'll need a neode model. At the risk of causing confusion, I'm going to call the nodes we save in the database Node. I'll want a UUID to serve as a unique ID, a status and x and y coordinates.

api/models/Node.js

module.exports = {
    id: {
        type: 'uuid',
        primary: true,
    },
    status: {
        type: 'string',
        valid: [ 'locked', 'unlocked', ], // I want two statuses
    },
    x: 'float',   // These are a simple shorthand for {type: 'float'}
    y: 'float',   
}

Then, an instance of neode. For clarity, I will create this in a new file called neode.js but this could quite easily just be written in index.js.

api/neode.js

const neode = require('neode')
    // Load using the config in .env
    .fromEnv()
    // Use the models defined in api/models/*.js
    .withDirectory(__dirname + '/models')

module.exports = neode

You can then use neode to create some data and place them at a random x,y coordinate.

const neode = require('./neode')

const ids = [
    '61ff4876-e488-4a91-94a6-f8d6bdd35273',
    'c98e1475-1421-41f9-9f2f-551ce1b1bf5a',
    '642f2518-b1b8-4405-9dd8-bd19e6a0f487',
    'be7d130c-a922-4e80-9a84-c0723a8d4710'
];

ids.map(id => neode.create('Node', { id, x: Math.random()*1000, y: Math.random()*1000 }))

Communicating with Socket.io

Next, for the communication between the front end and API. For this, I've gone for
socket.io because I've used this on a few projects in the past and it is relatively easy to set up with express. But in reality, any websockets implementation will do.

Again, in a new file, I'm creating a function that can be called to create a socket.io instance. I've done this because the socket needs to be bound to some sort of http server, and at some point will need to communicate with neo4j and the kafka consumer too.

api/io.js

const createSocket = (server, neode, consumer) => {
    const io = require('socket.io')(server)

    // Messages will be sent and received here...

    return io
}

module.exports = createSocket

Then the function can be imported and called inside index.js.

api/index.js

const io = require('./io')(server, neode)

The socket.io API is essentially an event emitted and therefore has an extremely simple interface.
On connection, a connection event is fired which provides a socket object. This contains an
on(event, callback) function that allows you to execute a callback function when an event is received.

api/io.js

io.on('connection', socket => {
    console.log(`--> JOINED: ${socket.id}`) // --> JOINED: Am-spDXUXvNHhuLNAAAD

    // Listen for events from that socket
    socket.on('disconnect', () => {
        console.log(`<-- LEFT  : ${socket.id}`) // <-- LEFT  : sbIpyb0yM64cYcX_AAAA 
    })
})

To send a message, you can either use socket.emit(message, payload) to send a message to that socket only, or use io.emit(message, payload) to send a message to all sockets. Socket.io also has the concept of Rooms and Namespaces, which could be useful in a multitenancy environment but that is beyond the scope of this article.

When a user joins for the first time, we might want to do some authentication but for berevity I will skip that step. When a user joins, I will use neode to query the database and send the current state of the graph to the UI in the form of a welcome message.

io.on('connection', socket => {
    console.log(`--> JOINED: ${socket.id}`) // --> JOINED: Am-spDXUXvNHhuLNAAAD

    neode.all('Node')
        .then(res => res.toJson())
        .then(data => socket.emit('welcome', data))

    // ...
})

In order to handle communications from the client, I'll need to listen for 2 events; lock and unlock. When the lock event is received with an id in the payload, I will need to use that to update the status of a node in neo4j. If an unlock event is received, I will need to set the unlocked status should be saved along with the new x and y coordinates.

For now, when a message is received I will simply send out the past tense version to all other connected sockets. Later on, these messages will be handled by Kafka instead.

// Handle Lock Event
socket.on('lock', data => {
     neode.find('Node', data.id)
        .then(node => node.update({
            status: 'locked',
            // So we have some context of who locked it, we can use the socket ID
            by: socket.id, 
        }))
        .then(node => node.toJson())
        .then(json => io.emit('locked', json))
})

// Handle Unlock Event
socket.on('lock', data => {
    neode.find('Node', data.id)
        .then(node => node.update({
            status: 'unlocked',
            by: null,
            x: data.x,
            y: data.y,
        }))
        .then(node => node.toJson())
        .then(json => io.emit('unlocked', json))
})

Great. That should be enough to get the front end working.

Building a Front End with Vue.js

Essentially, we want a framework that allows us to build an application quickly and that can consume messages via websockets using jabascript. My personal preference is Vue.js but you could te easily swap out Vue.js for React, Angular, Ember, jQuery, Backbone (is that still a thing?!), YourFavouriteFrontEndFramework and the result would be the same.

The @vue/cli tools allow you to quickly generate a project template and get started in minutes. The following command creates a new project in the ui folder.

vue create ui

There are several ways to include the Socket.io code in a vue project, but because the implementation is relatively simple I'll just use the socket.io-client package.

cd ui
npm i --save socket.io-client

# serve the project in dev mode
npm run serve

The easiest way to create draggable elements is through SVG. So inside the App.vue component, I'll just add an svg element and use a custom Vue component to represent the nodes - each generating a draggable <circle> element.

When the UI receives the welcome message, it should populate a nodes array. Each time a locked or unlocked message is received, the corresponding item in that array will be updated and that status reflected in the UI.

App.vue

<template>
    <div id="app">
        <svg height="100%" width="100%" style="background: #f2f2f2">
            <!-- Node circles will go here -->
        </svg>
    </div>
</template>

<script>
import Node from '@/components/Node'
import io from 'socket.io-client';

// There is probably a more sophisticated way of doing this...
const socket = io('http://localhost:3000');

export default {
    name: 'app',
    components: {
        Node,
    },
    data: () => ({
        loading: true,
        selected: null,

        nodes: [],
    }),
}
</script>

For the draggable nodes, I'll simply wrap a Vue component <circle> component and apply some handlers. This way, I can encapsulate the locking and dragging logic and then send back events to the parent component when something interesting happens. I've defined these

Node.vue

<template>
    <circle
        :cx="x" :cy="y" :r="r" :style="style"
        :title=" JSON.stringify(node) "
    />
</template>

<script>
export default {
    name: 'Node',
    props: {
        node: Object,
        selected: String,
        lock: Function,
        move: Function,
        unlock: Function,

        r: {
            type: Number,
            default: 50,
        }
    },
    // Start with default values for x and y that will be changed based
    // on the values of the node object passed as a prop
    data: () => ({ x: 0, y: 0, }),
}
</script>

Because the x and y coordinates can be changed, I can't use these directly from the node object. So the data() function will return default values. Then as the component is mounted, or the node object updates this position should be overwritten.

Node.vue

export default {
    name: 'Node',
    // ...
    data: () => ({ x: 0, y: 0, }),
    methods: {
        reposition() {
            // Set x and y based on the Node object
            this.x = this.node.x;
            this.y = this.node.y;
        },
    },
    mounted() {
        // Run the function when the component is first mounted
        this.reposition()
    },
    watch: {
        // ... and for any subsequent change to the node object
        node() {
            this.reposition()
        },
    },
}

Locking the Node

On mouse down, I want to send a message to the server to say that the node has been locked. This will be taken care of by the lock function passed as a prop, but I will also need to check that another user hasn't locked the nodeso I need to wrap it in a handler function. First I need to define a function in the component, then bind that to the onmousedown event.

// ...
methods: {
    handleLock(e) {
        // Don't do anything if it's already locked
        if ( this.node.status === "locked" ) {
            return;
        }

        this.lock(this.node)
    },

},
// ...

To bind events to components, I prefer the shorthand @ syntax - so for example, I can bind the function on mouse down, and then use .prevent to prevent any default behaviour (the equivalent of calling event.preventDefault()).

<template>
    <circle
        :cx="x" :cy="y" :r="r" :style="style"
        :title=" JSON.stringify(node) "

        @mousedown.prevent="handleLock"
        @mousemove.prevent="handleMove"
        @mouseup.prevent="handleUnlock"
        @mouseout.prevent="handleUnlock"
    />
</template>

Handling the Drag

Handling the dragging of a component in an SVG is fairly trivial - and is defined in great detail here. On mouse move, if this node is selected by the current user, I want to set the cx and cy properties to the position of the mouse. This information is available from the event. Because the SVG element is the parent element of the component, it can be accessed by using this.$el.

handleMove(e) {
    if ( this.selected === this.node.id ) {
        this.$el.setAttributeNS(null, "cx", e.clientX)
        this.$el.setAttributeNS(null, "cy", e.clientY)
    }
},

Then, once the Node has been dropped, I want to unlock the node and send the new position back to the API.

handleUnlock(e) {
    if ( this.selected === this.node.id ) {
        this.unlock(this.node, {x: e.clientX, y: e.clientY})
    }
},

Client to Server Communication

Sending and receiving messages from the Socket.io client are similar to the code above. The socket object has an emit(message, body) function which. As this will be identical for all messages, I'll implement it as a method on the App component. For a production ready app you might want to move this into it's own service.

Each of the messages then has a similar signature, just sending a different event name and payload.

App.vue

export default {
    name: 'app',
    // ...
    methods: {
        send(message, body) {
            console.log('sending', message, body)
            return socket.emit(message, body)
        },

        lock(node) {
            const { id, } = node;

            // Set selected node in component
            this.selected = node.id

            this.send('lock', { id, })
        },
        move(node, data) {
            const { id, } = node;
            this.send('move', { id, ...data })
        },
        unlock(node, data) {
            // Clear selected node in component
            this.selected = null

            const { id, } = node;
            this.send('unlock', { id, ...data })
        },
    },
}

Now the server can send events, all that is needed is to receive any events that come back. The created() function is fired when the component is created - this is a perfect place to bind some event listeners for the socket. As a message comes in, the nodes prop for the component should be updated to reflect the change - whether that is a node locked or unlocked.

export default {
    name: 'app',
    // ...
    created() {
        // Add listeners to set the initial app state on connection
        socket.on('welcome', nodes => {
            this.nodes = nodes
            this.loading = false
        })

        // Handle another user locking a node
        socket.on('locked', e => {
            const index = this.locked.indexOf(e.id);
            if ( index > -1 ) {
                this.locked.splice(index, 1)
            }

            // Update status
            const nodeIndex = this.nodes.findIndex(n => n.id == e.id)
            this.nodes[ nodeIndex ].status = "locked"

            console.log('LOCKED!', e)
        })

        // Handle a node becoming unlocked
        socket.on('unlocked', e => {
            // Remove from locked
            const index = this.locked.indexOf(e.id);
            if ( index > -1 ) {
                this.locked.splice(index, 1)
            }

            // Update status and position
            const nodeIndex = this.nodes.findIndex(n => n.id == e.id)

            this.nodes[ nodeIndex ].x = e.x
            this.nodes[ nodeIndex ].y = e.y
            this.nodes[ nodeIndex ].status = "unlocked"

            console.log('UNLOCKED!', e)
        })
    },
}

A quick test shows that any changes made in one browser window are reflected in other browser windows.

Publishing Updates with Neo4j Streams

This is fine for a small application, but wouldn't work at scale. If you try to run the API in a load balanced environment only the other users that happen to be connected to the server that received the update would be updated. This is where Kafka comes in, when data is updated in Neo4j we can push it out to a Kafka topic. Then, we can create a Consumer in api/src/index.js which will consume the topic and then relay the changes out to the front end via websockets.

There are two ways of achieving this. The more complex option would be to publish the changes from the application server, or use the Neo4j Streams plugin to turn Neo4j into a Publisher.

Neo4j Streams is a neo4j plugin that allows you to create Producers and/or Consumers directly inside neo4j by editing a few lines of neo4j.conf. By installing the plugin, a set of procedures are made available in neo4j to publish. You could CALL streams.publish(topic, payload) in cypher but that still seems like too much work.

Neo4j Streams also allows you to publish to or consume from a Kafka topic with a few lines of config.

Installing Neo4j Streams

Full installation instructions are available here but in short, you download the *.jar file for the latest release and copy it to the plugins folder of your neo4j instance. Then, in neo4j.conf add in a couple of items with references to the zookeeper server and any kafka servers to bootstrap to (separated by a comma).

neo4j.conf

kafka.zookeeper.connect=zookeeper.url:2181
kafka.bootstrap.servers=kafka.url:9092

Configuring the Sink

By configuring the neo4j server as a Sink, you can publish updates to a Kafka Topic at the end of every transaction. Neo4j Streams allows you to set this up by adding a few more lines to the neo4j.conf file. After configuring the server as a source, there are options to define patterns, each of which will be checked when a transaction is committed.

streams.source.topic.nodes.<TOPIC_NAME>=<PATTERN>
streams.source.topic.relationships.<TOPIC_NAME>=<PATTERN>

I have a queue set up in kafka called locks, so I can configure neo4j to update kafka when the status, x or y properties of a Node node are updated. You can either supply a wildcard {*} to include all properties or provide a list inside braces separated by a comma ({prop1, prop2}). Prefixing a property with a minus sign {-prop3} will exclude it from the filter. You can apply many patterns to this config by splitting with a semicolon.

# enable it
streams.source.enabled=true
streams.source.topic.nodes.locks=Node{status,x,y}

After restarting the database to apply the settings and running a cypher query - you can see that

MATCH (n:Node)
SET n.status = 'locked', n.x = rand()*100, n.y = rand()*100

Listening for Changes in the API

Now, the only thing left to do is to messages published to the Kafka topic in the API and push those out to the front end. For this, the API will need a Kafka consumer. This is done with the kafka-node package installed earlier.

The first step is to create a client to connect to the Kafka host. Because .env is already setup and working with neode, I'll add a the configuration options for the kafka server(s), topic and consumer group to .env

.env

KAFKA_SERVER=kafka.url:9092
KAFKA_TOPIC=locks
KAFKA_GROUP=LocksGroup1

Then the next step is to create a client to connect to Kafka, then a consumer that will listen to the locks topic. By specifying a group, you can either split the consumption of the topic over a number of servers, or in this case by supplying different groups for each application server - each application server and subsequent user will receive the update.

api/src/consumer.js

const kafka = require('kafka-node')

// Client
const client = new kafka.KafkaClient({ kafkaHost: process.env.KAFKA_SERVER });

// Consumer
const consumer = new kafka.Consumer(
    client,
    // You can supply more than one topic here
    [ { topic: process.env.KAFKA_TOPIC, partition: 0 } ],
    {
        groupId: process.env.KAFKA_GROUP,
        autoCommit: true,
        fetchMaxWaitMs: 1000,
        fetchMaxBytes: 1024 * 1024,
        encoding: 'utf8',
        fromOffset: false
    }
);

// Export it so we can use it later
module.exports = consumer;

Then the only thing left to do is consume the messages as the come in from the topic. Because the consumer is only listening on a single topic there's no need to check the message when it comes in - just the type.

The interface for listening is the same as socket.io, when a message comes in we want to emit it to each connected socket.

api/src/io.js

 // Now with added consumer parameter
const createSocket = (server, neode, consumer) => {
    // ...

    // Listen for Kafka 
    consumer.on('message', ({ value, }) => {
        // Parse the JSON value into an object
        const { payload, } = JSON.parse(value)

        // Get the properties from the update
        const { properties, } = payload.after

        // ... and the status
        const { status, } = properties

        console.log('\n\nemitting from kafka:', status, properties)

        // Emit the message through all connected sockets
        io.emit(status, properties)
    })
}

That's it! When a message comes in, the payload will include a value that includes before and after objects that show the state of the node before and after the transaction. Because the status properties are the same as the messages, this can be used as the message name, and the entire property object can be sent as the payload.

Next Steps

This is quite a basic example, but you could use this to do some pretty clever things. For example, neo4j could push the information to another topic, which another topic listens to, runs some aggregation and then passes the new value to another queue.

Neo4j could be used as a Consumer as well as a Producer and process messages directly from a Kafka topic.

On the websockets side, Socket.io has Rooms and Namespaces functionality that could be used for a multi-tenancy system. As a connection is received, some authentication could take place that resulted in the user joining a room belonging to it's organisation. That way only the relevant messages are sent to each user.

The code is up on github, feel free to clone and play for yourself.

If you have any questions or comments, I'm @adamcowley or feel free to continue the conversation on the Neo4j Community site.

Forem: Adam Cowley

Using TypeScript with Neo4j

A Worked Example

Adding Type-Checking

Type Checking in Action

Suggesting Record Keys

Suggesting Properties

Checking Property Keys

Type-checking Properties

Interested in learning more?

Using Neo4j in your next Next.js Project

What is Next.js?

What is Neo4j?

Why Neo4j and Next.js?

Creating a free Neo4j AuraDB Instance

Creating a new Next.js Project

Adding Neo4j Helper Functions

Data Fetching in Next.js

Static Page Generation

Using Neo4j for Server-side Rendering

Using Neo4j for Client-side Data Fetching

Conclusion

Adding Record & Type Checking in TypeScript with Generics

Generics

Generics and Databases

Generics and Function return types

Bringing it together

Wrapping things up

How to build a Neo4j Application with Go

What is GraphAcademy?

About the course

Assumptions

What you will learn

Enrol Now

How to build a Neo4j Application with Node.js

What is GraphAcademy?

About the course

Assumptions

What you will learn

Enrol Now

How to Build Authentication into a Vue3 Application

How to Build an Authentication into a Vue3 Application

Vue 3 and the Composition API

The Authentication Problem

Managing State Composition Functions

API Interactions

Registering a User

Saving the User into Global state

Remembering the User

Login

Using the Data in a Component

Logging Out

Protecting Routes

Conclusion

Building a Modern Web Application with Neo4j and NestJS

Tech Stack

Neo4j

Typescript

NestJS

Week 1 - Nest fundamentals & Neo4j Integration

Nest modules

@Injectable() classes

Nest Integration for Neo4j

Configuration & Dynamic Modules

Using the Service in the Application

Tune in next week!

Building a Real-Time UI on top of Neo4j with Vue.js and Kafka

Overview

Creating a basic express API

Data Persistence with Neode

Communicating with Socket.io

Building a Front End with Vue.js

Locking the Node

Handling the Drag

Client to Server Communication

Publishing Updates with Neo4j Streams

Installing Neo4j Streams

Configuring the Sink

Listening for Changes in the API

Next Steps

`@Injectable()` classes