Forem: Jo Stichbury

Introducing your new team lead…Kedro

Jo Stichbury — Wed, 19 Apr 2023 12:59:06 +0000

This post explains how Kedro can guide an analytics team to follow best practices and avoid technical debt.

In a recent article, I explained that following software principles can help you create a well-ordered analytics project to share, extend and reuse in the future. In this post we'll review how you can benefit from using Kedro as a toolbox to apply best practices to data science code.

How data science projects fail

As data scientists, we aspire to unlock valuable insights by building
well-engineered prototypes that we can take forward into production.
Instead, there is a tendency for us to make poor engineering decisions
in the face of tight deadlines or write code of dubious quality through
a lack of expertise.

The result is technical debt and prototype code that is difficult to understand,
maintain, extend, and fix. Projects that once looked promising fail to transition past the experimental stage into production.

"A cycle of quick and exciting research leads to high expectations of
great improvement, followed by a long series of delays and
disappointments where frustrating integration work fails to recreate
those elusive improvements, made all the worse by the feeling of sunk
costs and a need to justify the time spent."

Joe Plattenburg, Data Scientist at Root Insurance

How to write well-engineered data science code

When you start to cut code on a prototype, you may not prioritize
maintainability and consistency. Adopting a team culture and way of
working to minimize technical debt can make the difference between
success and failure.

Some of the most valuable techniques a data scientist can pick up are
those that generations of software engineers already use, such as the
following guidelines:

Use a standard and logical project structure: It is easier to
understand a project, and share it with others, if you follow a standard
structure.

Don't use hardcoded values: instead, use precisely named constants
and put them all into a single configuration file so you can find and
update them easily.

Refactor your code: In data science terms, it often makes sense to
use a Jupyter notebook for experimentation. But once your experiment is
done, it's time to clean up the code to remove elements that make it
unmaintainable, and to remove accidental complexity. Refactor the code
into Python functions and packages to form a pipeline that can be
routinely tested to ensure repeatable behaviour.

"Testing after each change means that when I make a mistake, I only
have a small change to consider in order to spot the error, which
makes it far easier to find and fix."

Martin Fowler, Author of Refactoring: Improving the Design of Existing
Code

Make code reusable by making it readable: Write your pipelines as a
series of small functions that do just one task, with single return
paths and a limited number of arguments.

Many data scientists say they've learned from their colleagues through
pair programming, code reviews and in-house mentoring that enables them
to build expertise suitable to their roles and requirements.

We see Kedro as the always-available team lead that steers the direction
of the analytics project from the outset and encourages use of a
well-organized folder structure, software design that supports regular
testing, and a culture of writing readable, clean code.

What is Kedro?

Kedro is an open-source toolbox for production-ready data science. The
framework was born at QuantumBlack to solve the challenges faced
regularly in data science projects and promote teamwork through
standardised team workflows. It is now hosted by the LF AI & Data
Foundation as an incubating project.

Kedro = Consistent project structure

Kedro is built on the learnings of Cookie Cutter Data Science. It helps you to standardise how configuration, source
code, tests, documentation, and notebooks are organised with an
adaptable project template. If your team needs to build with multiple
projects that have similar structure, you can also create your own
Cookie Cutter project templates with Kedro starters.

Kedro = Maintainable code

Kedro helps you refactor your business logic and data processing into
Python modules and packages to form pipelines, so you can keep your
notebooks clean and tidy.
Kedro-Viz then visualises the pipelines to help you navigate .

"People started from scratch each time, the same pitfalls were
experienced independently, reproducibility was time consuming and only
members of the original project team really understood each
codebase...

We needed to enforce consistency and software engineering best
practices across our own work. Kedro gave us the super-power to move
people from project to project and it was game-changing. After working
with Kedro once, you can land in another project and know how the
codebase is structured, where everything is and most importantly how
you can help".

Joel Schwarzmann, Principal Product Manager, QuantumBlack Labs, blog
post

Kedro = Code quality

Kedro makes it easy to avoid common code smells such as hard-coded
constants and magic numbers. The configuration library enables your code
to be reusable through data, model, and logging configuration. An
ever-expanding data catalog supports multiple formats of data access.

Kedro also makes it keep your code quality up to standard, through
support for black, isort, and flake8 for code linting and formatting,
pytest for testing, and Sphinx for documentation.

Kedro = Standardisation

Kedro integrates with standard data science tools, such as TensorFlow,
scikit-learn, or Jupyter notebooks for experimentation, and commonly
used routes to deployment such as Databricks.

Summary

Kedro is an open-source Python toolbox that applies software engineering
principles to data science code. It makes it easier for a team to apply
software engineering principles to data science code, which reduces the
time spent rewriting data science experiments so that they are fit for
production.

When you follow established best practice, you have a better chance of
success.

Software engineering principles only work if the entire team follows
them. A tool like Kedro can guide you just like an experienced technical
lead, making it second nature to use established best practices, and
supporting a culture and set of processes based upon software
engineering.

Look forward to greater collaboration and productivity with Kedro in
your team!

Find out more about Kedro

There are many ways to learn more about Kedro:

Join our Slack organisation to reach out to us directly if you've a question or want to stay up to date with our news. There's an archive of past past conversations on Slack too.
Read our docs or look at the Kedro source code on GitHub.
Check out our "Crash course in Kedro video on YouTube.

Look out for an upcoming training session tailored to help your team get
on-board with Kedro.

How do data scientists combine Kedro and Databricks?

Jo Stichbury — Wed, 19 Apr 2023 12:14:37 +0000

In recent research, we found that Databricks is the dominant
machine-learning platform used by Kedro users.

The purpose of the research was to identify any barriers to using Kedro with Databricks; we are collaborating with the Databricks team to create a prioritised list of opportunities to facilitate integration.

For example, Kedro is best used with an IDE, but IDE support on Databricks is still evolving, so we are keen to understand the pain points that Kedro users face when combining it with Databricks.

Our research took qualitative data from 16 interviews, and quantitative data from a poll (140 participants) and a survey (46 participants) across the McKinsey and open-source Kedro user bases. We analysed two user journeys.

How to ensure a Kedro pipeline is available in a Databricks workspace

The first user journey we considered is how a user ensures the latest version of their pipeline codebase is available within the Databricks workspace. The most common workflow is to use Git, but almost a third of the users in our research set said there were a lot of steps to follow.

The alternative workflow, which is to use dbx sync to Databricks repos, was used by less than 10% of the users we researched, indicating that awareness of this option is low.

How to run Kedro pipelines using a Databricks cluster

The second user journey is how users run Kedro pipelines using a Databricks cluster. The most popular method, used by over 80% of participants in our research, is to use a Databricks notebook, which serves as an entry point to run Kedro pipelines.

We discovered that many users were unaware of the IPython extension that significantly reduces the amount of code required to run Kedro pipelines in Databricks notebooks.

We also found that some users run their Kedro pipelines by packaging them and running the resulting Python package on Databricks. However, Kedro did not support the packaging of configurations until version 18.5, which has caused problems.

The final option some users select is to use Databricks Connect, but this is not recommended since it is soon
to be sunsetted by Databricks.

The output of our research

To make it easier to pair Kedro and Databricks, we are updating Kedro's documentation to cover the latest Databricks features and tools, particularly the development and deployment workflows for Kedro on Databricks with DBx. The goal is to help Kedro users take advantage of the benefits of working locally in an IDE and still deploy to Databricks
with ease.

You can expect this new documentation to be released in the next one to two weeks.

We will also be creating a Kedro Databricks plugin or starter project template to automate the recommended steps in the documentation.

Coming soon...

We have a managed Delta table dataset available in our Kedro datasets repo, which will be available for public consumption soon. We are also planning to support managed MLflow on Databricks.

We have set up a milestone on GitHub so you can check in on our progress and contribute if you want to. To suggest features to us, report bugs, or just see what we're working on right now, visit the Kedro projects on GitHub.

We welcome every contribution, large or small.

Find out more about Kedro

There are many ways to learn more about Kedro:

Join our Slack organisation to reach out to us directly if you've a question or want to stay up to date with our news. There's an archive of past past conversations on Slack too.
Read our docs.
Check out our "Crash course in Kedro video on YouTube.

Look out for an upcoming training session tailored to help your team get on-board with Kedro.

A new home for the Kedro blog and some recent releases

Jo Stichbury — Tue, 04 Apr 2023 14:51:47 +0000

In this post, we describe recent releases to Kedro, Kedro-Viz and some new Kedro datasets.

Kedro has a new blog over at kedro.org/blog!

We’ve previously published on QuantumBlack’s Medium channel, but recent updates and improvements here on the Kedro website mean that we’re now able to bring you a dedicated blog for, and about, the open-source Kedro community.

We plan to publish a range of articles by contributors from within the team and beyond. If you’re a Kedroid with an idea for a post, please reach out to us using one of the channels on the Slack organisation, or raise an issue on GitHub.

Kedro releases

We last gave an update on Kedro in late 2022, when we described the features in Kedro version 0.18.4. Since then, we’ve released three additional non-breaking versions of Kedro in the 0.18.x series, with the goal of a regular release cadence at the end of most two-week development sprints.

Some of the highlights of our releases are described below along with links to the full release notes. For each of these releases there’s a straightforward upgrade path with pip or conda. For example, to upgrade to Kedro version 0.18.7 from version 0.18.4:

pip install kedro==0.18.7

conda install -c conda-forge kedro==0.18.7

We received many contributions to these new versions from our open-source community and want to thank every contributor for taking the time to extend and improve Kedro.

Kedro version 0.18.7

These are the headline changes (You can find all the details about the Kedro 0.18.7 release on GitHub):

We added new Kedro CLI command kedro jupyter setup to set up a Jupyter Kernel for Kedro that automatically loads the Kedro extension for ease of use.
The kedro package command now includes the project configuration in a compressed tar.gz file.
We’ve added functionality to package and read your configuration as a compressed file. You can now use OmegaConfigLoader to load configuration from compressed files of zip or tar format. (This feature requires fsspec>=2023.1.0).
In documentation news, we moved seamlessly from kedro.readthedocs.io to docs.kedro.org in this release. We also made some significant improvements to on-boarding documentation that covers setup for new Kedro users and major changes to the spaceflights tutorial to make it faster to work through. We think it’s a better read. Tell us if it’s not.

Kedro version 0.18.6

This was a small release to fix a bug introduced in Kedro 0.18.5 that was causing experiment tracking in Kedro-Viz to fail. You can find all the details about the release of Kedro version 0.18.6 on GitHub.

Kedro version 0.18.5

In February 2023, we released Kedro version 0.18.5, to introduce a brand new config loader powered by OmegaConf. You can now use the omegaconf syntax with kedro run --param.

We also added the following:

Some improvements to the kedro run command used in the CLI. One changes is to make it more consistent. The flags --node, --tag, and --load-version are deprecated in favour of plural equivalents (--nodes, --tags, and --load-versions) and will be removed in Kedro 0.19.0. An additional change means that you can filter and run nodes by node namespace using the --namespace flag with kedro run.
There is now support for using generator functions as nodes, i.e. using yield instead of return.
We added a new node argument to all four dataset hooks

You can find all the details about the Kedro version 0.18.5 release on GitHub.

Kedro datasets releases

Kedro provides numerous different built-in datasets for various file types and file systems, to save you from having to write the logic for reading or writing data, including Pandas, Spark, Dask, NetworkX, Pickle, and more.

There have been several datasets contributed by community members over the past months which include the addition of:

snowflake.SnowparkTableDataSet by Vladimir Filimonov and Heber Urdaneta
polars.CSVDataSet by Walber Moreira

As we mentioned in “Keeping up with Kedro”, Kedro version 0.19.0 will move Kedro’s datasets from the main framework project into a separate package called Kedro-Datasets.

Kedro-Viz releases

If you've not yet used it, Kedro-Viz is the interactive development tool for building data science pipelines with Kedro. It comes with an experiment tracking feature enabling you to view and compare different runs of your Kedro project. Check out the Kedro-Viz demo at demo.kedro.

We’ve made three releases of Kedro-Viz this year, plus a patch release. You can find further details of the Kedro-Viz releases on GitHub.

To get the latest release of Kedro-Viz, you can use pip:

pip install kedro-viz==6.0.0

or npm

npm install @quantumblack/kedro-viz@latest

Here’s a summary of what we’ve been working on:

Kedro-Viz version 6.0.0

In this release we bumped the major version to 6.0.0 because of a change in the frontend React code (we bumped the minimum version of React from 16.8.6 to 17.0.2). Additional changes include:

We added a change so you can now see a preview of your data in the metadata panel.
You can remove metrics plots from metadata panel and add links to the plots on experiment tracking.
You can also link plot and JSON dataset names from experiment tracking to the flowchart.
Kedro-Viz no longer depends on pandas or Plotly.

Kedro-Viz versions 5.3.0 and 5.2.0

We introduced a raft of updates to experiment tracking, the largest being the addition of time series & parallel coordinates metrics plots and delta values.

We’ve enabled the display of json objects with react-json-viewer in experiment tracking.
We added a feature to show/hide modular pipelines on the pipeline flowchart.
It’s now possible to retrieve and share URL parameters for each element/section in the flowchart.

We've recently published a blog post about experiment tracking to highlight the latest features and discuss what is coming next.

What's next for the Kedro projects?

We have a broad range of milestones for the Kedro framework that cover areas such as integration with Databricks, enhancements for Jupyter Notebook users and ongoing changes such as the transition of datasets into their own package.

On the to-do list for Kedro-Viz, we’ve included enhanced navigation between flowchart and experiment tracking and collaboration features within Kedro-Viz.

Stand by for a pair of virtual Kedro showcases on 5th April 2023 (9am BST and 4pm BST) to demonstrate some of the features added in the recent releases to the global community.

To suggest features to us, report bugs, or just see what we’re working on right now, visit the Kedro projects on GitHub. We welcome every contribution, large or small.

How to build a serverless WebSockets platform

Jo Stichbury — Tue, 06 Sep 2022 13:37:23 +0000

First published on the Ably blog

When building modern web applications, it is increasingly important to be able to handle realtime data with an event-driven architecture to propagate messages to all connected clients instantly.

Several protocols are available, but WebSocket is arguably the most widely used as it is optimized for minimum overhead and low latency. The WebSocket protocol supports bidirectional, full-duplex communication between client and server over a persistent, single-socket connection. With a WebSocket connection, you can eliminate polling and push updates to a client as soon as an event occurs.

How to integrate WebSockets into your stack

You could deliver an event-driven architecture by setting up a dedicated WebSocket server for clients to connect and receive updates. However, this architecture has several drawbacks, including the need to manage and scale the server and the inherent latency involved in sending updates from that server to clients, who may be distributed worldwide.

To scale up your hardware across the globe, you could skip the responsibility and cost of ownership by renting from a cloud service provider. For example, you could use a managed cloud service such as Amazon ECS, or you could opt for a serverless WebSocket solution.

“A serverless application is one that costs you nothing to run when nobody is using it, excluding data storage costs.” Cloud 2.0: Code is no longer King — Serverless has dethroned it

Saving on operational costs is an excellent reason to consider a serverless framework: you reap the benefits of scale and elasticity, but you don’t have to provision or manage the servers.

This article explains how to build a basic serverless WebSockets platform, ideal for simple applications such as chat, using AWS API Gateway to create a WebSocket endpoint and AWS Lambda for connection management and backend business logic.

The basic architecture of a serverless WebSockets platform

Building a simple serverless WebSockets platform is relatively straightforward, using AWS API Gateway to build out WebSockets APIs and Lambda functions for backend processing, storing message metadata in Amazon DynamoDB.

AWS API Gateway

AWS API Gateway is a service to create and manage APIs. You can use it as a stateful proxy to terminate persistent WebSocket connections and transfer data between your clients and HTTP-based backends such as AWS Lambda, Amazon Kinesis, or any other HTTP endpoint.

AWS Lambda

AWS Lambda is a serverless compute service that allows you to run code without provisioning or managing servers.

WebSocket APIs can invoke Lambda functions, for example, to deliver a message payload for further processing and then return the result to the client. The Lambda functions only run as long as necessary to handle any individual task. For client authorization, the connection request must include credentials to be verified by a Lambda authorizer function that returns the appropriate AWS IAM policy for access rights.

Example: chat app powered by serverless WebSockets

As is traditional, let’s look at an example chat app to illustrate how the serverless WebSockets solution powers a basic realtime application. First, consider the sequence by which a client WebSocket connection is made and stored.

A client device establishes a single WebSocket connection to AWS API Gateway. Each active WebSocket connection has an individual callback URL in API Gateway that is used to push messages back to the corresponding client.
The connection message is sent to an API Gateway Authorizer Lambda function to confirm that the device has the correct credentials to connect to the service.
The connection message and metadata are sent to a separate AWS Lambda when it has been authenticated.
The Lambda handler inspects the metadata and stores appropriate information in AWS DynamoDB, a NoSQL document database.

Client establishes a WebSocket connection to API Gateway.

In a chat app, users can be involved in numerous conversations involving one or more participants. This is a typical use case for the publish and subscribe pattern. When a user sends a message to a particular conversation, or channel, the chat app publishes the message to all other users who subscribe to that channel. Each subscriber to that channel receives the message in their app.

When the chat app initiates a WebSocket connection to AWS API Gateway, there needs to be a way to determine the channels for which that connection is authorized to publish and subscribe. The client code must send this information within the connection metadata.

Now let’s consider the point at which the user sends a message, and another user receives it:

A client sends a chat message, including the metadata to identify the particular channel on which to publish.
On receipt, API Gateway sends it to the appropriate Lambda function.
Upon invocation, the Lambda checks for the connections subscribed to that channel and sends API Gateway the ID for each subscriber and the message payload.
API Gateway uses the callback URLs for each subscriber connection to send the payload to the client on the established WebSocket connection.
Upon disconnection, a Lambda function is invoked to clean up the datastore.

A message sent from one client is published in realtime to others on that channel.

The limitations of basic serverless WebSockets

The above is a simplistic example of a realtime, event-driven app using serverless WebSockets. Some of the limitations include:

Connection state tracking doesn’t scale: AWS API Gateway doesn’t track connection metadata, which is why it needs to be stored in a database such as Amazon DynamoDB using a Lambda function to update the store for every connection/disconnection. For large numbers of connecting clients, you’d hit Lambda scaling limits. You could instead use an AWS Integration to API Gateway to call DynamoDB directly, but you’ll need to be mindful of the associated high level of database usage.

Another aspect that we’ve not addressed is that of abrupt disconnection. In cases where a client connection drops without warning, the active WebSocket connection isn’t properly cleaned up. The database will potentially store connection identifiers that are no longer present, which can cause inefficiencies.

One way to avoid zombie connections is to periodically check if a connection is “alive” by using a heartbeat mechanism such as TCP keepalives or ping/pong control frames. Sending heartbeats negatively impacts the system’s scalability and dependability, especially when handling millions of concurrent WebSocket connections, since it creates an extra load on the system.

You cannot broadcast messages to all connected clients: In this simple design, there is no way to simultaneously send a message to multiple connections, as you’d expect from a standard pub/sub channel or topic. To fan out an update by publishing it to thousands of clients would require you to fetch the connection IDs from the database and then make an API call to each, which isn’t a scalable approach. You could alternatively consider Amazon SNS to add pub/sub to your app, but there are still limitations and the downside of additional complexity.

For a chat application example, you can assume that typical channels will have a low number of subscribers unless they’re used in interactive livestream sessions with thousands of users. Other scenarios that can tolerate a lack of broadcast include individual update notifications and interactivity features that work on an item-by-item basis. Live data fanout use cases cannot support this limitation if there are a large number of subscribers on a channel to receive the latest sports score or news update.

There is a limit on the number of WebSocket connections allowed per second: AWS API Gateway sets a per region limit of 500 new connections per second per account and 10,000 simultaneous connections. This may be insufficient if you plan to scale your application.

The solution is bound to a single AWS region: The WebSocket connections offered by AWS API Gateway are bound to a single region, resulting in poor performance because of latency for clients geographically distant from that region. It’s possible to use Amazon EventBridge, a serverless event bus, for cross-region event routing to replicate events.

Challenges of building a production-ready serverless WebSockets platform

The architecture above has potential if used to build out a simple realtime messaging platform, although we’ve already seen that it has limitations at scale. If you need a production-ready system, you will require scale but also need to consider the dependability of your solution in terms of performance, the integrity of message delivery, fault tolerance, and fallback support, all of which can be complex and time-consuming to solve.

Performance

In a distributed system, latency deteriorates the further it travels, so data should be kept as close to the users as possible via managed data centers and edge acceleration points. It isn’t sufficient to minimize latency, however. The user experience needs latency variance to be at a minimum to ensure predictability.

Message integrity

When a user’s connection drops and reconnects, an app needs to pick up with minimal friction from the point before they disconnected. Any missed messages need to be delivered without duplicating those already processed. The whole experience needs to be completely seamless.

Fault tolerance & scalability

The demand for a service can be unpredictable, and it is challenging to plan and provision sufficient capacity. A realtime solution must be highly available at all times to support surges in data. The challenge is to scale horizontally and swiftly absorb millions of connections without the need for pre-provisioning.

Your solution must also be fault-tolerant to continue operating even if a component fails. There need to be multiple components capable of maintaining the system if some are lost. There should be no single point of congestion and no single point of failure.

Fallback transports

Despite widespread platform support, the WebSocket protocol is not supported by all proxies or browsers, and some corporate firewalls even block specific ports, which can affect WebSocket access. You may consider supporting fallback transports, such as XHR streaming, XHR polling, or long polling.

Feature creep

Another challenge to using raw serverless WebSockets in a production-ready system may only present itself after successfully using it in the product’s first iteration. While you’ve added the essential capability to receive realtime updates, feature creep often means that the features seed additional requirements for shared live experiences and collaborative features.

Building and maintaining a proprietary WebSocket solution to support the future realtime needs of a product can be challenging. The infrastructure that underpins the system must be stable and dependable and requires experienced engineers to build and maintain it. A development team may find they have a long-term commitment to supporting the realtime solution rather than focussing on the features that augment the core product.

Ably: A serverless WebSocket PaaS built for scale

Adopting a serverless WebSocket solution from a third-party vendor makes more financial sense for many organizations. Otherwise, you could spend several months and a heap of money. And you’d end up burdened with the high cost of infrastructure ownership, technical debt, and ongoing demands for engineering investment.

Ably has fault-tolerant, highly-available, elastic global infrastructure for effortless scaling and low complexity. Our serverless WebSockets platform is mathematically modeled around the Four Pillars of Dependability. We can ensure messages are delivered at low latency over a secure, reliable, and global edge network. There’s no DevOps overhead, and there’s no infrastructure to manage.

The Ably platform abstracts the worry of building a realtime solution so that you can prioritize your product’s roadmap and develop features to stay competitive.

While Ably’s native protocol is WebSocket, there is rarely a one-size-fits-all protocol: different protocols serve different purposes better than others. Ably offers multiple protocols such as WebSocket, MQTT, SSE, and raw HTTP.

To support live experiences, Ably adds features such as device presence, stream history, channel rewind, and handling for abrupt disconnections, making building rich realtime applications easier. There is a range of webhook integrations for triggering business logic in realtime. We offer a gateway to serverless functions from your preferred cloud service providers. You can deploy best-in-class tooling across your entire stack and build event-driven apps using the ecosystems you’re already invested in.

Ably extends a basic serverless WebSockets solution to augment your product.

Ably enables you to deliver a live experience without delays in going to market, runaway costs, or unhappy users. Using Ably allows your engineering teams to focus on core product innovation without having to provision and maintain realtime infrastructure. Our existing customers already build massively scalable realtime apps with us, with rapid go-to-market and average savings of $500k on build cost.

Conclusion

In this article, we have reviewed the architecture of a simple serverless WebSockets platform using Amazon API Gateway and AWS Lambda. We have also considered some of the limitations of this design, such as lack of support for broadcasting, single-region deployment, and connection management.

We introduced Ably’s serverless WebSocket platform, which reliably handles high-scale realtime data distribution to web and mobile apps at the edge.

Building a realtime chat app with Laravel using WebSockets

Jo Stichbury — Mon, 18 Jul 2022 09:50:37 +0000

You use realtime communication every day. It is the simultaneous exchange of information between a sender and a receiver with almost zero latency. Internet, landlines, mobile/cell phones, instant messaging (IM), internet relay chat, videoconferencing, teleconferencing, and robotic telepresence are all examples of realtime communication systems.

In this tutorial, you’ll learn how to build a web based realtime chat app using Laravel, Ably and Vue.js. You’ll use Vue.js to build the frontend/UI and Laravel as a backend to interact with Ably Realtime APIs to facilitate realtime communication. By building this type of application, you’ll learn about the core concepts for building realtime apps using the Laravel-Ably integration.

Please note that you can use any frontend framework (react, angular, next.js etc.) to build the Laravel chat app and there is no dependency on any specific framework.

The source code for the project is available in this GitHub repo.

Architecture of the realtime chat app

Before getting started, you should familiarize yourself with the tech stack that you’ll be using in this tutorial to build your chat application.

Laravel

Laravel is a PHP-based web application framework with expressive, elegant syntax. The web development community loves Laravel because it provides an amazing developer experience and supports advanced concepts, like queues, dependency injection, unit testing, and integration testing. Laravel is a great choice for building realtime apps, and it already comes with support for Ably to "broadcast" your server-side Laravel events over a WebSocket connection.

Ably

Ably is a pub/sub messaging platform that powers synchronized digital experiences in realtime for millions of simultaneously connected devices around the world. Ably offers WebSockets, stream resume, history, presence, and managed third-party integrations to make it simple to build, extend, and deliver digital realtime experiences at scale. Ably makes use of different transports to make sure data is delivered in realtime.

Vue.js

Vue is a frontend javaScript framework for building user interfaces. It builds on top of standard HTML, CSS, and JavaScript and provides a declarative and component-based programming model that helps you efficiently develop user interfaces, be they simple or complex.

Laravel broadcasting

Laravel broadcasting is a feature provided by Laravel as a way for a client (frontend) to publish and receive information in realtime via channels. A channel is a logical communication medium that helps to publish and subscribe to information over a WebSocket connection. There are 3 types of channels.

1. Public channels

The client doesn't need to be authenticated (logged in) to use public channels.
The client can only subscribe to the data on this channel. To publish data, it needs to be done via the laravel-server (backend) i.e. client1 => laravel => ably => client 2, client 3, client 4...

2. Private channels

The client needs to be authenticated to use private channels.
The client is directly able to publish and subscribe to data through Ably using client-events. i.e. client1 => ably => client 2, client 3, client 4...
The client can also publish data through laravel-backend, as done for public channels.

3. Presence channels

These are similar to private channels, offering all of the features of private channels, as well as a way to register client presence on the channel.

In this blog we will only look at public channels. So, the architecture of the realtime chat app will look like this:

The chat app uses laravel + ably-broadcaster and Ably at server side (backend) and vue.js + ably-laravel-echo at client side (frontend).

Laravel-server is used for authenticating/authorizing clients and broadcasting information to ably on behalf of clients over a given channel. In the current example, since public channels don’t need any sort of authentication, it will only be used to broadcast information.

Clients (guest-users) use HTTP connection to post a message with a channel-name to Laravel-server. Laravel-server publishes the same message to Ably (on the same channel) using a named event. Ably is responsible for sending this information to clients (subscribed to the same channel) using the provided event over a WebSocket connection. Guest users consume this event on the client-side using the Laravel Echo library.

Publishing a message on a channel is done using an event. Events are the primary method of packaging a message on the channel.

In simpler terms, when a message is published on a channel with an eventName, it will be received by a subscriber matching the same channelName and eventName.

Setting up the environment

Here is what we will need to get started.

Prerequisites

PHP version >= 7.3
Node.js version >=14.x.x

Setting up the basic project structure

Execute the following command in the terminal to create a Laravel project.

composer create-project laravel/laravel ably-laravel-chat-app

Run the following commands to get the auto-generated app running

cd ably-laravel-chat-app
composer install
npm install
php artisan key:generate
php artisan serve // run in a separate terminal
npm run watch // run in a separate terminal

Finally, open localhost:8000 in your browser, and you’ll see your Laravel project running.

Setting up your Ably account

To start, you need to create an Ably account if you don’t have one. Next, you need to grab the API key from the Ably dashboard:

1. Visit your app dashboard and click Create New App with name Chat-App.
2. Once the app has been created, go to the API Keys tab. Copy the first key with the name Root. Keep the key safe. The key shouldn't be exposed to the client code.

That’s it. Your Ably account is ready to support realtime messaging.

Setting up Ably broadcaster on Laravel

Please follow the Setup section to configure the Ably broadcaster.

Create PublicMessageEvent Class

To send events from Laravel to Ably, create an event file by running the following command in your terminal:

php artisan make:event PublicMessageEvent

This command will create a PublicMessageEvent.php file in the app/Events directory. Replace the content of the files with following:

<?php

namespace App\Events;

use Illuminate\Broadcasting\Channel;
use Illuminate\Broadcasting\InteractsWithSockets;
use Illuminate\Contracts\Broadcasting\ShouldBroadcast;
use Illuminate\Foundation\Events\Dispatchable;
use Illuminate\Queue\SerializesModels;

class PublicMessageEvent implements ShouldBroadcast
{
    public $channelName;
    public $message;

    use Dispatchable, InteractsWithSockets, SerializesModels;

    /**
     * Create a new event instance.
     *
     * @return void
     */
    public function __construct($channelName, $message)
    {
        $this->channelName = $channelName;
        $this->message = $message;
    }

    /**
     * Get the channels the event should broadcast on.
     * https://laravel.com/docs/broadcasting#model-broadcasting-channel-conventions
     * 
     * @return \Illuminate\Broadcasting\Channel|array
     */
    public function broadcastOn()
    {
        return [new Channel($this->channelName)];
    }
}

The code above specifies:

1. The PublicMessageEvent class that implements the ShouldBroadcast interface.
2. By default the class name is used as an event name. This means eventName will be PublicMessageEvent. You can change eventName using broadcastAs.
3. The broadcastOn method that is responsible for returning the channel/list of channels on which the event should broadcasted.

Broadcasting using PublicMessageEvent class

To handle API requests from frontend, open the routes/api.php file and add following route.

Route::post('/public-event', function (Request $request) {
    $channelName = $request->post('channelName');
    $message = $request->post('message');
    broadcast(new PublicMessageEvent( $channelName, $message ));
})->middleware('throttle:60,1'); // 60 requests/minute are allowed.

The code above specifies:

1. A new API route to handle POST request from client. This API route will be available at localhost:8000/api/public-event.
2. When called, the message is broadcasted to ably on the given channel with eventName as PublicMessageEvent.
3. Throttle to 60 requests per minute for each IP address. It will return an error for requests exceeding the given limit. This is mainly to avoid spamming on public channels.

To check if the code is working correctly so far:
1. Restart the server using php artisan serve.
2. Attach to channel public:community for new messages using the Ably Dev Console.
3. Try sending a POST request to the localhost:8000/api/public-event endpoint using curl, Postman or any other tool with channelName and message as POST method body.

curl --location --request POST 'localhost:8000/api/public-event' \
--header 'Content-Type: application/json' \
--data-raw '{
    "channelName":"public:community",
    "message":"Hi there"
}'

4. Check for this message in the Ably Dev Console under the attached channel tab.

Next, we will look at setting up frontend to send and receive messages.

Setting up Laravel Frontend

Following packages need to be installed for proper communication with Laravel and Ably.

1. Laravel Echo: This JavaScript library allows clients/users to subscribe to channels and listen to the events sent from ably.
2. Axios: This package helps to make HTTP requests from client to laravel-server.

npm install @ably/laravel-echo ably axios

Initialize Laravel-Echo

In the resources/js/bootstrap.js file, add the following snippet as per the Echo Setup Section.

import Echo from '@ably/laravel-echo';

window.Ably = require('ably');

// Create new echo client instance using ably-js client driver.
window.Echo = new Echo({
    broadcaster: 'ably',
});

// Register a callback for listing to connection state change 
window.Echo.connector.ably.connection.on((stateChange) => {
    console.log("LOGGER:: Connection event :: ", stateChange);
    if (stateChange.current === 'disconnected' && stateChange.reason?.code === 40142) { // key/token status expired
        console.log("LOGGER:: Connection token expired https://help.ably.io/error/40142");
    }
});

Building the front-end UI

Please take a look at app.js, ChatComponent.vue and MessageComponent.vue from the existing repository under resources/js/ for building the UI for your chat app.

Sending messages

In the ChatComponent.vue file, you can see the broadcastMessageOnPublicChannel method is used for broadcasting data on a given public channel using axios.

broadcastMessageOnPublicChannel() {
    const message = this.message?.trim();
    if(!message)
        return;
    const publicChannelName = this.getActiveChannel().name;
    const broadcastUrl = window.location.origin + "/api/public-event";
    axios.post(broadcastUrl, { channelName : publicChannelName, message });
    this.message = null;
}

The code above specifies:

1. publicChannelName is generally prefixed with public:.
2. When broadcastMessageOnPublicChannel is called, it sends a message from the client to laravel-server at endpoint /api/public-event using axios.
3. Axios sends a HTTP POST request with channelName and message as a request body.

Receiving messages

In the ChatComponent.vue file, the listen method describes handler for PublicMessageEvent.

Echo.channel(channelName)
.subscribed(() => // Do stuff when channel is attached )
.listenToAll(() => // Do stuff when any event is received )
.listen('PublicMessageEvent', (data) => {
    const channel = this.getChannelByName(channelName, 'public');
    this.updateBroadcastNotificationOnUI(channel, data)
})

The code above specifies:

1. Public channel is created using Echo.Channel(channelName).
2. Actual publicChannelName is accessed using Echo.Channel(channelName).name and is prefixed with public:. i.e. If channelName is provided as community, internally it uses channel name as public:community while communicating with ably.
3. The method listenToAll registers a handler which gets called for any event received.
4. The method listen registers a handler for a specific PublicMessageEvent and updates the received message on the UI once called.

Testing the chat app

To test your chat app, open the localhost:3000 in at least two browser tabs. Join the public room in each tab and broadcast messages to each other. For more information, refer to Guest Mode under the Usage Section.

Conclusion

That’s it! In this tutorial, you learned how to create a realtime chat app using Laravel and Ably for guest users. In a future blog, we will cover how to add chat functionality for authenticated (logged in) users. Depending on your requirements, you can also discover how to add more features to your chat app in the Ably chat reference guide.

We would love to hear about how you plan to use Ably in your applications. Join our Discord server and post your ideas for builds in the showcase channel to share, we’re currently giving out Ably tees to members who participate!

Engage your site visitors with live updates using Ably, Next.js, Prisma, and PlanetScale

Jo Stichbury — Tue, 28 Jun 2022 11:54:46 +0000

Not so long ago, email was considered revolutionary. Being able to send a message to someone on the other side of the world and receive a reply in a matter of minutes was a game-changer.

Today, however, email is the slow-moving dinosaur of the Internet. Now, users expect instantaneous feedback from their online interactions and to communicate with people and machines instantly. If your app requires them to constantly refresh a web page to see the latest updates, they’ll soon be looking elsewhere.

But how has technology kept pace with these increasing demands?

One approach is the publish/subscribe architectural pattern (pub/sub for short). It’s a method of asynchronous communication that enables applications to publish messages to a channel, and any clients subscribed to that channel to receive them.

This tutorial will show you how to use pub/sub to implement live commenting in a blog. As soon as someone adds a comment to a blog article, anyone reading that article will see it immediately: no page refresh required. See it in action!

The ability to comment immediately on a blog article might not be something you’ve missed in the past, but it’s a simple, self-contained example of an application that uses a live experience to engage its audience. You can build upon what you learn here to implement other audience engagement features to add to your own applications.

The tech stack

To achieve this, we’re going to be using the following technologies:

Next.js

Next.js is a React framework that makes it easy to build fast and scalable web applications. React is an incredibly popular and powerful UI library, but requires extensive configuration. Next.js eliminates much of that work, and also provides developers with options on whether to render content on the client, server, or using a mixture of both.

It would be handy if you knew a little bit about Next.js before you start this tutorial, but it’s not strictly necessary because we’ll provide a good starting point for building your application and guide you through the rest. But the Next.js website has a great mini-course that will get you up to speed quickly and their docs are awesome.

PlanetScale

PlanetScale is a database-as-a-service platform. It allows developers to quickly create highly-scalable databases and work with those databases as if they were code in a Git repository. Developers can create branches for development and push those to production when they are working as intended. You’ll be using PlanetScale to store blog comments.

Prisma

Prisma is a Node.js and TypeScript ORM. It is a server library which helps developers write queries to work with databases using a familiar object-oriented approach. You will be using it to work with your PlanetScale database.

Ably

Ably is a feature-rich pub/sub messaging platform that empowers developers to create engaging digital experiences in realtime. It guarantees message delivery to all manner of devices at huge scales while preserving the order and integrity of messages. You will use it to publish blog comments to a channel and to subscribe to that channel.

Vercel

Finally, you’ll be using Vercel to host your application and make it available to the world. The nice folks at Vercel are the same people who brought you Next.js, making deployment of your application a breeze.

Now that we know what we’re going to build and what we’re going to build it with, let’s get started!

Tutorial Steps

Step 1: Create the Next.js application

There’s a lot of boilerplate code in this application, which we’ve created for you so that you can focus on incorporating the realtime elements. Start off by forking the Github repository for this tutorial.

clicking on the Fork button on the repository’s home page. A forked copy of that Git repository will be added to your personal GitHub or GitLab repo. That’s it.

git clone https://github.com/ably/ably-next-prisma-planetscale.git

This repo contain two branches:

main - which contains the solution code for this tutorial
starter-project - which is the starting point for this tutorial

Check out the starter-project branch: you’ll be working with this branch for the whole of this tutorial. Use the main branch to compare your code with the solution code if you get stuck.

git checkout starter-project

The starter project gives you all the base components necessary to start implementing realtime communication and data persistence. It was built using MUI, previously known as Material-UI.

Take a look at the components folder and get a feel for which component does what.

Execute the following npm commands at your project’s root directory to install the core dependencies and launch the project:

npm install
npm run dev

Visit http://localhost:3000 in your browser and you’ll see your sample blog app, with some dummy content and a section for commenting:

Next.js will continue to hot-reload the project as you develop, so as a general rule you can see it taking shape as you work through the remaining steps without having to restart the server. When you update certain configuration files you will have to restart the server, but we’ll let you know when that’s the case.

Step 2: Install the Ably React Hooks package

The realtime commenting that you will implement in this app is provided by Ably.

Next.js is powered by React and React can be a bit fussy about where and when you use code from libraries such as Ably’s. To make life simpler for React developers, Ably has developed the Ably React Hooks NPM module. Install it into your project now:

npm install ably @ably-labs/react-hooks

Step 3: Register for Ably

To use Ably’s service, you need to sign up for a free account and obtain an API key.

Your new Ably account comes with a pre-created project called Default, and your API key for that project will be displayed on your screen.

If, for any reason, you don’t see it, click the Default project in the dashboard and then select the API Keys tab at the top. Click the Show button to display the root API key (not to be confused with the Subscribe only API key, which will only let you read messages from a channel, and not publish to it). So make sure you get the right one.

Step 4: Configure Ably React Hooks

In this step you will instantiate the Ably client and authenticate with the Ably platform.

First, create a file called .env in your project’s root directory and create a configuration key called ABLY_API_KEY which stores the API key you retrieved in the preceding step. Note that, even though this API key is an alphanumeric string, you don’t need to surround it with quotes.

ABLY_API_KEY=<paste your api key here>

One of the many great things about Next.js is that it supports API routes. These enable you to build API endpoints as Node.js serverless functions. This is really useful for us, because we can create an API endpoint that authenticates with the Ably service without having to expose our API key to the browser.

Implement an API endpoint called /createTokenRequest which will generate a token for you to authenticate with Ably on the client-side. Next.js makes it really easy to include API routes in your application. All you need to do is create the route handler in the pages/api directory, in a file of the same name.

In your Next.js application, create a createTokenRequest.js file in your pages/api folder. This route is then available as the /createTokenRequest endpoint.

Inside that file, create a new Ably client and use it to generate a new token:

import Ably from "ably/promises";

export default async function handler(req, res) {
  const client = new Ably.Realtime(process.env.ABLY_API_KEY);
  const tokenRequestData = await client.auth.createTokenRequest({
    clientId: "ably-blog-app",
  });
  res.status(200).json(tokenRequestData);
}

You can then use this endpoint to configure Ably on the client-side. But, if you were to do so using a relative, rather than absolute URL, this would generate an error due to a bug in one of its dependencies. It wouldn’t stop the Ably service from working, but it would soon clutter up your console. So let’s fix that now by creating an environment variable for the URL. This URL will change when we deploy to Vercel in a later step.

It also needs to be available to the browser, which it isn’t by default. Luckily, Next.js can help with that. Any environment variables prefixed by NEXT_PUBLIC_ are accessible on the client. So don’t ever put anything sensitive in there (such as your API key), but it’s a good place to configure your application’s host URL. Enter the following in your .env file:

NEXT_PUBLIC_HOSTNAME=http://localhost:3000

Now, head to your _app.js file and use configureAbly from @ably-labs/react-hooks to call your new authentication endpoint:

import { configureAbly } from "@ably-labs/react-hooks";
import "../styles/globals.css";

configureAbly({
  authUrl: `${process.env.NEXT_PUBLIC_HOSTNAME}/api/createTokenRequest`,
});

function MyApp({ Component, pageProps }) {
  return <Component {...pageProps} />;
}

export default MyApp;

Your authentication is in place. Unfortunately, if your server is running, you might see a SyntaxError: Unexpected token 'export' issue in your browser. The reason for that is because the @ably-labs/react-hooks package exports as a ES6 module and Next.js doesn’t transpile functions imported from node_modules.

Thankfully, there is an easy fix. First, install the next-transpile-modules library:

npm install --save next-transpile-modules

Then, head to the next.config.js file in your root folder and specify which library should be transpiled:

const withTM = require('next-transpile-modules')(['@ably-labs/react-hooks']);

module.exports = withTM({
  reactStrictMode: true,
})

You’ll need to restart your server by hitting CTRL+C in your terminal and re-running npm run dev. But the error should be gone.

Step 5: Implement Pub/Sub

Now that you have the Ably library installed, you can work on the realtime elements.

Ably pub/sub is based on the concept of channels. Publishers send messages to a named channel and subscribers consume messages sent to that channel.

The Ably React Hooks NPM module provides a hook called useChannel. This hook subscribes to a specified channel when the component mounts and unsubscribes from it when the component unmounts.

The useChannel hook takes a callback function. This function is called every time a new message arrives on the channel so you can process it and make any necessary updates:

import { useChannel } from "@ably-labs/react-hooks";

const [channel] = useChannel("your-channel-name", (message) => {
    //Process the incoming message
});

Let’s talk about how you’ll use this in your application.

When a user submits a new comment on a blog post, your application will publish a message containing this comment to a channel called comment-channel.

Anyone else reading that blog post will see the new comment immediately, because they are subscribed to the channel and have therefore received and processed this new comment.

To do this, head to your Comments component and note that there is a state variable called comments. This will store your list of comments. This variable gets passed to the CommentsList component.

Create a submitComment() function and use channel.publish() to publish a new message to the channel. In Ably, a message has two parameters: name and data. For the data, you only need the commenter’s username and the text of the comment itself. Pass this submitComment() function to your AddCommentSection component.

Once this message is pushed, everyone subscribed to this channel, including the person creating the comment, will receive it, and it will be added to the list of comments in your state variable, thanks to the callback in useChannel.

Here is the code for Comments.js:

import React, { useState } from "react";
import { useChannel } from "@ably-labs/react-hooks";

import Typography from "@mui/material/Typography";

import CommentsList from "./CommentsList";
import AddCommentSection from "./AddCommentSection";

export default function Comments({ initialComments = [] }) {
  const [comments, setComments] = useState(initialComments);

  const [channel] = useChannel("comment-channel", (message) => {
    // Add new incoming comment to the list of comments
    setComments((comments) => {
      return [...comments, message.data];
    });
  });

  const submitComment = async (username, comment) => {
    try {
      const body = { username, comment };
      await fetch(`${process.env.NEXT_PUBLIC_HOSTNAME}/api/comment`, {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify(body),
      });

      channel.publish({
        name: "comment",
        data: {
          username,
          comment,
        },
      });
    } catch (error) {
      console.error("An error occurred when creating a comment: ", error);
    }
  };

  return (
    <React.Fragment>
      <Typography variant="h6" gutterBottom>
        Comments ({comments.length})
      </Typography>
      <CommentsList comments={comments} />
      <AddCommentSection submitComment={submitComment} />
    </React.Fragment>
  );
}

Note: In this example, we're using Ably's React Hooks on the client to both publish and subscribe to a channel. In a production application, you would typically want publishing to be handled by the server. This would enable you to perform any required validation before a message is distributed to other users and persisted. For example, you might want to run the message through a profanity filter. You can use our REST SDK for server-side publishing.

Step 6: Connect the comment form

In your AddCommentSection component, first receive the Comment components submitComment function as props.

Create two new state variables: username and comment. Then, replace the calls to console.log()s in the component’s JSX markup with calls to the functions that set the username and state variables: setUsername() and setComment(), respectively. Also, set the corresponding TextField value attributes to {username} and {comment}

Finally, create an addComment() function that will use your new submitComment() function to publish the message and reset your form:

Here is the code for AddCommentSection.js:

import React, { useState } from "react";

import Grid from "@mui/material/Grid";
import TextField from "@mui/material/TextField";
import FormControl from "@mui/material/FormControl";
import Button from "@mui/material/Button";

export default function AddCommentSection({ submitComment }) {
  const [username, setUsername] = useState("");
  const [comment, setComment] = useState("");

  const addComment = () => {
    //Publish message
    submitComment(username, comment);
    //Reset form
    setUsername("");
    setComment("");
  };

  return (
    <FormControl>
      <Grid container spacing={2}>
        <Grid item xs={12}>
          <TextField
            required
            id="username"
            name="username"
            label="Username"
            variant="outlined"
            value={username}
            onChange={(event) => setUsername(event.target.value)}
          />
        </Grid>
        <Grid item xs={12}>
          <TextField
            required
            id="comment"
            name="comment"
            label="Comment"
            variant="outlined"
            multiline
            rows={4}
            value={comment}
            onChange={(event) => setComment(event.target.value)}
          />
        </Grid>
        <Grid item xs={12}>
          <Button variant="contained" onClick={addComment}>
            Submit
          </Button>
        </Grid>
      </Grid>
    </FormControl>
  );
}

Step 7: Try it out!

If your server isn’t already running, start it with npm run dev.

Open http://localhost:3000 in two separate browser tabs and try submitting comments from each. If everything is working correctly, then you should see new comments appearing instantly in both tabs.

The only issue is that if you refresh the page, you will lose all your comments. This is because your app is missing the last piece of the puzzle: data persistence. That’s what you’ll implement next.

Step 8: Create your PlanetScale database

In this tutorial, you will be using PlanetScale for a hosted database in which to store blog comments. To start with, head to https://planetscale.com/ and create a free account. Once you’re logged in, click the New Database button to create a database.

In the dialog that appears, name the database ably-realtime-db and choose the region closest to you to minimise latency.

Click the Create Database button and your database is ready to use:

Once you’re done, go back to your terminal and install the PlanetScale CLI. This will enable you to run many helpful commands, such as opening a shell to query your database.

Step 9: Install and configure the Prisma ORM

In your application’s root directory, run the following command:

npx prisma init

This command does three things:

Adds a DATABASE_URL environment variable to your .env file with a dummy URL
Creates a prisma folder within your application
Add a schema.prisma file inside the prisma folder

The schema.prisma file configures your database URL and provider. You will also define your comment data model in here in a future step.

Right now, however, the DATABASE_URL in .env is a dummy one. So head back to the PlanetScale webpage. On the right side of your page, you should see a Connect button. Click on it and then on Generate Password. Select the Prisma format and you should see something like this:

datasource db { provider = "mysql" url = "mysql://******:************@******.region.psdb.cloud/database-name?sslaccept=strict" }

Copy this URL and add it to your .env file, but remove the double quotes, as this can cause problems with Vercel when you deploy your app in a later step:

DATABASE_URL=mysql://****:****@***.region.psdb.cloud/ably-realtime-db?sslaccept=strict

In your schema.prisma file, change the provider to mysql and enforce referential integrity as shown below. This step is necessary due to incompatibilities between Prisma and PlanetScale, because the latter doesn’t support foreign key constraints. To overcome this, we have to set the referentialIntegrity property:

generator client {
  provider = "prisma-client-js"
  previewFeatures = ["referentialIntegrity"]
}

datasource db {
  provider = "mysql"
  url      = env("DATABASE_URL")
  referentialIntegrity = "prisma"
}

In this same file, define a data model for blog comments. It will be a very straightforward one with an autogenerated ID, a user name, and a comment:

model Comment { 
  id Int @default(autoincrement()) @id 
  username String
  comment String
}

Once your model is in place, sync your schema to your PlanetScale database. To start, connect to your database by opening a new tab in the terminal and running the following command:

npx prisma db push

If all goes well, you should see the following message:

?  Your database is now in sync with your schema. Done in 996ms
    ...
    ✔ Generated Prisma Client (3.9.1 | library) to ./node_modules/@prisma/client in 122ms

Check that your schema is working correctly by connecting to your database using the PlanetScale CLI. Specify the database name and the branch you are working on, as follows:

pscale shell ably-realtime-db starter-project

This will open a shell into your database where you can execute describe Comment; and see the following table definition:

ably-realtime-db/| starter-project |> describe Comment;
+----------+--------------+------+-----+---------+----------------+
| Field    | Type         | Null | Key | Default | Extra          |
+----------+--------------+------+-----+---------+----------------+
| id       | int          | NO   | PRI | NULL    | auto_increment |
| username | varchar(191) | NO   |     | NULL    |                |
| comment  | varchar(191) | NO   |     | NULL    |                |
+----------+--------------+------+-----+---------+----------------+

Type exit; to quit the shell.

Great! Your Prisma schema is now in sync with PlanetScale. The last step is to promote your current PlanetScale branch to be the production branch. To do this, execute the following:

pscale branch promote ably-realtime-db starter-project

Step 10: Install the Prisma client package

You have defined the connection details and data model in prisma.schema. The next step is to get your app to use this information to persist comment data.

First, stop your server if it’s still running, and install the Prisma client package:

npm install @prisma/client

Then, instantiate the Prisma client. Create a folder called lib and a file inside the folder called prisma.js for this:

import { PrismaClient } from "@prisma/client";

const prisma = new PrismaClient();
export default prisma;

Step 11: Create API routes for reading and writing comments to the database

In the same way that you built the /api/createTokenRequest route, create another route called /api/comment that your application can make GET and POST requests to. GET reads the existing comments from the database and POST writes a new comment to the database.

At the moment, the only methods you need to pay attention to are GET and POST. Anything else should return a 405 HTTP status.

Your GET endpoint should return a list of all your comments using a simple Prisma query. For the POST endpoint, your username and comment are contained within the body of the request, which you will use to create a new comment in your table.

In /pages/api/comment.js, enter the following code:

import prisma from "../../lib/prisma";

export default function handler(req, res) {
  switch (req.method) {
    case "GET":
      return handleGET(req, res);
    case "POST":
      return handlePOST(req, res);
    case "OPTIONS":
      return res.status(200).send("ok");
    default:
      return res.status(405).end();
  }
}

// GET /api/comment - retrieves all the comments
async function handleGET(req, res) {
  try {
    const comments = await prisma.comment.findMany();
    res.json(comments);
    res.status(200).end();
  } catch (err) {
    res.status(err).json({});
  }
}

// POST /api/comment - creates a new comment
async function handlePOST(req, res) {
  try {
    const comment = await prisma.comment.create({
      data: {
        ...req.body,
      },
    });
    res.json(comment);
    res.status(200).end();
  } catch (err) {
    res.status(err).json({});
  }
}

Step 12: Read existing comments from the database

You can now make a GET request to your /api/comment endpoint to fetch all the comments from your database. You can do that with getServerSideProps. This function is a really handy feature of Next.js that enables you to make an API call when the page is requested by the user. When the user first loads the page, the client will make an API call, fetch the saved comments, and pass them to props.

In pages/index.js, retrieve the comments from props and pass them to the Comments component to populate your list:

import Container from "@mui/material/Container";
import Grid from "@mui/material/Grid";
import CssBaseline from "@mui/material/CssBaseline";
import Divider from "@mui/material/Divider";

import Header from "../components/Header";
import Footer from "../components/Footer";
import Content from "../components/Content";
import Comments from "../components/Comments";

const Home = ({ comments = [] }) => {
  return (
    <div>
      <CssBaseline />
      <Container maxWidth="lg">
        <Header />
        <Grid container spacing={5} sx={{ mt: 3 }}>
          <Grid
            item
            xs={12}
            sx={{
              py: 3,
            }}
          >
            <Content />
            <Divider
              variant="middle"
              sx={{
                my: 3,
              }}
            />
            <Comments initialComments={comments} />
          </Grid>
        </Grid>
        <Footer />
      </Container>
    </div>
  );
};

export const getServerSideProps = async () => {
  const res = await fetch(`${process.env.NEXT_PUBLIC_HOSTNAME}/api/comment`);
  const comments = await res.json();
  return {
    props: { comments },
  };
};

export default Home;

Step 13: Write new comments to the database

In your Comments component, you can now save a new comment from your submitComment function. All you have to do is make a POST request with the username and comment in the request body.

Then, when you publish the message, all the subscribers can be notified and updated accordingly:

import React, { useState } from "react";
import { useChannel } from "@ably-labs/react-hooks";

import Typography from "@mui/material/Typography";

import CommentsList from "./CommentsList";
import AddCommentSection from "./AddCommentSection";

export default function Comments({ initialComments = [] }) {
  const [comments, setComments] = useState(initialComments);

  const [channel] = useChannel("comment-channel", (message) => {
    // Add new incoming comment to the list of comments
    setComments((comments) => {
      return [...comments, message.data];
    });
  });

  const submitComment = async (username, comment) => {
    try {
      const body = { username, comment };
      await fetch(`${process.env.NEXT_PUBLIC_HOSTNAME}/api/comment`, {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify(body),
      });

      channel.publish({
        name: "comment",
        data: {
          username,
          comment,
        },
      });
    } catch (error) {
      console.error("An error occurred when creating a comment: ", error);
    }
  };

  return (
    <React.Fragment>
      <Typography variant="h6" gutterBottom>
        Comments ({comments.length})
      </Typography>
      <CommentsList comments={comments} />
      <AddCommentSection submitComment={submitComment} />
    </React.Fragment>
  );
}

Run your server again using npm run dev and try adding some comments.

Then, open another terminal window and execute the following command:

npx prisma studio

This will open http://localhost:5555/ in a new browser tab where you can see and manipulate the contents of your Comment table:

That’s it! You are now ready to share the results of your work with the world. You’ll do that in the final step, by deploying your application to Vercel.

Step 14: Deploy your app to Vercel

Before you deploy your app, you’ll have to do a bit of tidy up to deal with the inevitable CORS issues that arise when you host any web application.

In next.config.js, add the following HTTP headers. These specify which origins, methods, and other features your requests will support:

const withTM = require("next-transpile-modules")(["@ably-labs/react-hooks"]);

module.exports = withTM({
  reactStrictMode: true,
  async headers() {
    return [
      {
        // match all API routes
        source: "/api/:path*",
        headers: [
          { key: "Access-Control-Allow-Credentials", value: "true" },
          { key: "Access-Control-Allow-Origin", value: "*" },
          {
            key: "Access-Control-Allow-Methods",
            value: "GET,OPTIONS,PATCH,DELETE,POST,PUT",
          },
          {
            key: "Access-Control-Allow-Headers",
            value:
              "X-CSRF-Token, X-Requested-With, Accept, Accept-Version, Content-Length, Content-MD5, Content-Type, Date, X-Api-Version",
          },
        ],
      },
    ];
  },
});

You also need to support the OPTIONS method, which is used as a preflight check in API requests. In your pages/api/comment.js file, add handling for OPTIONS by returning HTTP status 200:

export default function handler(req, res) {
  switch (req.method) {
    case "GET":
      return handleGET(req, res);
    case "POST":
      return handlePOST(req, res);
    case "OPTIONS":
      return res.status(200).send("ok");
    default:
      return res.status(405).end();
  }
}

Once you’re happy that your application is working correctly, commit and push your starter-project branch.

git checkout starter-project
git add .
git commit -m "Final code"
git push

If you don’t already have one, create an account with Vercel.

Once you’re logged in, click on New Project, and import the Git repository where your code lives. Copy the URL that Vercel has generated for you, which should be something like https://[project-name].vercel.app/.

Edit your .env file NEXT_PUBLIC_HOSTNAME setting to point to the Vercel URL instead of http://localhost:3000. Commit and push the change:

git add .
git commit -m "Configure deployment URL"
git push

You’re currently still working on the starter-project branch, so for now set that as the Production branch in Vercel.

To do this, click your Github project in the Overview page, select the Settings tab and then the Git page from the left-hand navigation menu. Change the Production Branch to starter-project, then click Save.

Then, visit the Deployments tab, click the three little dots to the right of your deployment and select Redeploy.

Once the deployment is complete, your app should be up and running!

Conclusion

In this tutorial, you learned how to implement pub/sub messaging in realtime using the Ably platform within a Next.js application. You also learned how to create a database with PlanetScale and how to fetch and write data to it with the Prisma ORM. Finally, you used Vercel to deploy your application in a live environment.

Realtime communication might still be new for some developers, but the public is increasingly demanding instant feedback on all interactions they have online. You’re now in a great position to give them what they want, whatever the nature of your application or service.

We’d like to acknowledge the significant contribution to this article by Mark Lewin, who is a Senior Developer Educator here at Ably.

Share your views on development in 2019

Jo Stichbury — Thu, 10 Jan 2019 16:28:08 +0000

Hi there! I've just joined the community and am looking forward to contributing some thoughts on coding with graph databases and AI. But that's for next week :o)

Today, I just wanted to do a quick shoutout for SlashData, a company that I work with from time to time (we recently published a book on how to market to developers ... clue: don't patronise or hype). In the spirit of that message, I promise that I won't be pushing many of this kind of post onto DEV, but I did want to share this survey before it closes.

You may have come across SlashData before. They are an analysis company that tracks global software developer trends via regular, large scale developer surveys.

Those answering the surveys provide data for research that help the top technology firms understand who developers are, what tools are they using and where they‘re going next.

SlashData have just launched the 16th edition of our Developer Economics Survey. For everyone that takes the survey SlashData donates to a charity. This month, they're aiming to donate $2,000 to help get more kids into software development.

If you're interested in sharing your views on software development, or know someone who may be, please check out the survey here.