Forem: Hasura

Scaling frontend app teams using Relay

Hasura — Mon, 04 Sep 2023 06:13:00 +0000

The UI is decomposed into multiple React components.

The basic idea behind scaling the frontend, much like scaling any other part of the stack, is factoring it into multiple components:

Can be owned by multiple independent specialised teams
Reduce coupling between components
Have clearly specified interfaces between components

Let's look at how this strategy plays out, and how to use modern technologies and ideas like Relay and backend-for-frontend (BFF) to scale out frontend applications and teams.

Independent isolated teams own different components

This is a useful baseline from which to iterate. It represents the naive scenario in which teams independently develop different components in a larger app but in isolation from each other.

Independently developed components with independent data fetching. The circles are different teams.

Problem: Independent data fetching

Poor performance and UX jank
A component may make multiple network requests
Component data request may depend on ancestor data, leading to waterfall style requests
Components may fetch redundant data

Problem: Independent state management

Data and UI inconsistency

Batching of network requests

Teams can manually coordinate data fetching:

Batch all data requests at the root level
Distribute data through the component tree via props

Teams manually batch data requests through a shared root level query.

Problem: Coupling at root query

Adding a query: can duplicate other similar queries, and fetch redundant data
Removing or editing a query:
Can break another component that (implicitly) depends on the same data
Not removing unused data leads to over-fetch and cruft

Problem: Poor developer ergonomics

The data requirements for a component are no longer colocated with the component itself, which breaks encapsulation

TRPC / React Query is not a complete solution

Batches parallel network queries into single network requests
Cannot batch all queries needed for a page
Cannot solve waterfall requests
Queries are batched over the network layer, but still execute against the data layer as independent queries i.e. batching cannot leverage the internal structure or relations of queries

Coordinate data access through a centralized cache store

Teams manually coordinate shared state through a central store.

Introduces coupling between teams, with similar issues as with batching queries
Lots of boilerplate to normalize data, update stores, and plumb data to components

Backend for frontend

BFF is useful for making lighter and more performant client applications by moving compute and data transformations to the server.

Backend for frontend: Moves compute and data transformations to the server. Collectively owned by all frontend teams.

Owned by the client app team
Doesn't solve any of the coupling problems, just moves it around

GraphQL

Batched queries organized around client pages instead of server functionality couple the frontend and the backend teams.

Backend developers build APIs organized around Frontend pages/routes.

Instead, a GraphQL API exposes all features of the backend at a single endpoint.

The client app can craft queries that fetch exactly the data needed in one shot

Backend GraphQL API organized around server capabilities allows for flexible frontend queries.

Even better, we begin to see that the query structure beings to mirror the component tree!

This means we can refactor a root level query into fragments

Queries decompose into fragments.

Putting it all together with Relay

In Relay, every component defines its own data requirements

Data dependencies colocated with components. Fragment structure mirrors the component tree.

Significantly, a component can only access the data it has explicitly requested. This is "data masking" and is enforced through the useFragment hook.
This means that teams can modify individual components with confidence knowing that nothing will break as there are no implicit data dependencies

Independently declared data dependencies are compiled into a single root level query by the Relay compiler.

At build time, the Relay compiler builds an optimized set of top level queries from all the fragments

The compiler can check for common errors, and run optimizations such as deduplication across the whole codebase
You get the developer ergonomics of independently developed components, but with the efficiency of globally optimized and batched data fetching

Relay can leverage the rich information present in the GraphQL schema. Incrementally adopt the global node id and connection spec.

Relay uses the rich type information in the GraphQL schema, and automatically builds a local cache of data from all queries.

Further, by incrementally adopting features such as the node global id spec and the connection spec, you get advanced features such as:

Reloading only a portion of a query via fragments
Cursor based pagination

Conclusion

The JavaScript ecosystem has produced a variety of solutions for frontend development, data fetching, and state management; but solutions often fall short for ambitious projects.

GraphQL, Relay, and React were built to work together and have the huge benefit of being driven by Meta's (Facebook) experience building and maintaining extremely large and and complex applications developed by many teams.

if you’re not composing GraphQL fragments from multiple components into one query (as Relay does), i think you’re missing 80% of the point of GraphQL.

which is ok but isn’t talked about enough https://t.co/ooohCDV6ZO

— danabramov.bsky.social (@dan_abramov) March 12, 2023

It seems that GraphQL and React has taken over the world, but people are often put off by the new conventions espoused by the Relay library.

The good news is that Relay can be incrementally adopted.

The client library will work with any existing GraphQL API
Adopting bits of the Relay spec unlocks additional features
Relay can be adopted selectively by some components and not others, for an easy migration path

Even better, it seems that the people who've implemented Relay, have been happy with it for quite a while.

How Coinbase is scaling their app with Relay

"Relay is unique among GraphQL client libraries in how it allows an application to scale to more contributors while remaining malleable and performant."

https://t.co/D45yEid8l0

— Relay (@RelayFramework) May 6, 2022

Join us for a live Q&A session with Tanmai Gopal on the @GraphQL Discord. 🎙️ Discover how to scale UI development with Relay #GraphQL.

⏰ July 12, 11AM PT

Join GraphQl Discord server ➡️ https://t.co/MM3HN7DpbW pic.twitter.com/hhKYYcnbKc

— Hasura (@HasuraHQ) July 7, 2023

It still surprises me how Twitter embraces GraphQL / Relay (for RWeb) but keeping their Timeline model instead of adapting the Relay connection spec https://t.co/skOcPvrmMv

— Jane Manchun Wong (@wongmjane) November 19, 2022

💡 #RelayJs feature I appreciate - Relay is optimized for performance by default. It “forces” you to break down the data requirements of your UI in small, reusable parts, like React does with components. It'll then subscribe to changes of only the data each component asks for 1/x

— Gabriel Nordeborn (@___zth___) April 3, 2022

Use Hasura to easily generate a Relay compatible API with no code, across multiple data stores, with cross data store joins, filtering, and aggregations, and declaratively defined permissions.

Sign up now for Hasura Cloud to get started!

Announcing Hasura Notebook: Prototype fast on your GenAI apps

Hasura — Thu, 31 Aug 2023 15:24:53 +0000

Hasura empowers you to rapidly create high-quality data APIs for production purposes. As data sources continue to evolve, Hasura evolves as well, providing you with the capability to construct secure data APIs over versatile data stores, including vectorized data.

If you are new to Hasura, we recommend getting started for free with Hasura Cloud.

As part of Hasura’s evolution, today we’re introducing Hasura Notebooks, a tool designed to facilitate the swift prototyping of cutting-edge GenAI applications.

In this blog, we’ll take you through this new tool to gain a comprehensive understanding of how it works and why you need it.🙂

GenAI revolution

Large Language Models (LLMs) are very large deep neural networks trained on vast amounts of text data mostly scraped from the internet.

LLMs have taken the world by storm because these generalized language models are so good at understanding context, retrieving information, and generating content that can help us with numerous applications. And why not? Automation is the key to efficient applications.

At its core, LLMs learn the word probabilities to predict the next most suitable word given in a sentence. If the sentence is from a domain which the LLM hasn’t been exposed in the training period, then the LLM will make low confidence predictions, which are known as hallucinations.We can deal with this problem by providing LLM context in prompts and instructing it to use the context to complete the task. This process is called grounding.

In order to maximize the potential of LLMs across a wide range of applications, it's essential to expand the scope of secure data sources. Given the fast-paced environment, it's equally important to avoid dedicating excessive time to constructing data APIs (a crucial yet somewhat mundane task 🤷‍♀️) when you could be focused on creating exciting and lucrative applications.

Hasura to the rescue!

Let’s learn how you can quickly build secure data APIs and prototypes with a product search demo.

What is Hasura Notebook

Hasura Notebook is a remote Jupyter Notebook with ready-to-run examples and Jupyter Kernel Gateway baked-in to turn your code cells into endpoints.

Hasura Notebooks are great if you want to quickly prototype GenAI application with Hasura or learn GenAI from our existing templated projects.

We are going to building on Hasura Notebook in this blog. Recommended reading before we proceed - Hasura Jupyter Python Notebook & API Server documentation.

Use case: Get a contextual product search powered by OpenAI in under 10 minutes 🚀

As you might have experienced yourself, searches on most e-commerce websites are keyword-oriented. This results in a lot of false positives or irrelevant results and can lead to dissatisfied users. Contextual search is an answer to this problem. Contextual search matches the user query with the product description.

Prerequisites

1. Hasura CLI

If this is your first time using Hasura CLI, you will need to install it. Follow the installation instructions in this doc.

2. Hasura Notebook

Follow the steps in the documentation below to instantiate your Hasura Notebook.

Architecture of Hasura + Jupyter Notebook + VectorDB

You will know you are ready when you are able to access the Hasura Notebook with a landing page that looks like this.

You can find all the code under https://<connector url>/jupyter/notebook/product_search.

Let’s get building! 🚀

Setup your PostgreSQL database

Step 1: Start a new Neon Cloud PostgreSQL DB or link your existing DB.

More details on Neon Cloud PostgreSQL DB here: https://hasura.io/docs/latest/databases/postgres/neon/

Step 2: Create a new table called base_products and track it from the data tab.

Tracking enables the table to be accessible through GraphQL query.

Set up your vector database

Step 1: Create a free 14-day cluster on Weaviate.

Head to https://console.weaviate.cloud/ and register for an account. After confirming via email, click + Create cluster and fill in a name before clicking Create. Once Weaviate has provisioned your sandbox cluster, proceed to the next step.

Step 2: Create the Product schema called Product_vectors.

Product schema ID, Product Name, and Product Description. To execute this, run setup_weaviate.ipynb

Step 3: Add the Weaviate table to Hasura.

Currently, the Weaviate connector is not natively available in Hasura. You can add it to Hasura in two simple steps using the Hasura CLI.

Deploy Weaviate connector using Hasura CLI.

# create connector using our weaviate repo
hasura connector create my_weaviate_connector:v1 --github-repo-url https://github.com/hasura/weaviate_gdc/tree/main/
# check deployment status to get the endpoint

hasura connector status my_weaviate_connector:v1
# you can also use list command
hasura connector list
# view logs at any point of time
hasura connector logs my_weaviate_connector:v1
# for more commands explore help section

Add the Weaviate connector to Hasura.

Refer to this document for details https://hasura.io/docs/latest/databases/vector-databases/weaviate/

Step 4: Track the table.

Add a remote relationship between PostgreSQL and Weaviate tables

Go to Product_vectors table and add a relationship with the PostgreSQL database as shown below.

Set up an Event Trigger on a PostgreSQL table

Step 1: Fetch your Hasura GraphQL API endpoint and Admin secret.

Move over to cloud.hasura.io and click Projects. From your project, click on the Settings icon to access Project details.

Step 2: Update Hasura GraphQL details in the notebook under Template for event trigger to ETL data.

Open server.ipynb and you can see there are multiple functions with cells commented with # POST /handle_event These are the functions that are available on the endpoint /handle_event.

Update the details in the cell, and you are ready to go.

Don’t forget to restart the Jupyter gateway server.

Step 3: Add the Event Trigger in Hasura.

Head back to the Hasura Console and click on the Event tab to create a new Event. Fill in the required details.

Your webhook URL is your notebook URL (ends with .app) + invoke/handle_event.

You can also fetch your webhook URL by running this CLI command again hasura notebook status.

Add the authorization header required to reach the endpoint from the Hasura notebook.

Generate base64 encoded value. You will need to execute this on terminal of your choice.

echo -n "<username>:<password>" | base64

You can fetch the username and password by executing the following command

hasura notebook status

Key = Authorization
Value = Basic <base64 encoded value here>

Insert data into PostgreSQL and watch Vector DB update automatically

Now that we have integrated Weaviate with Hasura, we can use a mutation in Hasura to insert data into our Postgres table and auto-vectorize and update our vector DB! You can do this by executing insert_data.ipynb.

Voila! We have vectors with product name and description fields.

Secure semantic search using Hasura

For more on building secure APIs with Hasura, refer to this overview: https://hasura.io/docs/latest/auth/overview/

Let us now go one step further, with complex query powered by LLM.

LLM-powered product search

Step 1: Update Hasura GraphQL and OpenAI details in the notebook under Template for event trigger to ETL data.

Update the details, and you are ready to go.

After updating any of these values, it’s important to restart your notebook. You can do this from the gateway generated earlier using the CLI.

Step 2: Create LLM prompt

This step is available as a template for you. Feel free to tweak the prompt and play with it.

Step 3: Create Hasura Action

Like before, you will need to add Authorization headers.

Step 4: What you just created is a secure API on your LLM query. Integrate the API with your app or just play around on the Hasura Console. You now have the power of the Hasura by your side.

Conclusion

By seamlessly deploying a Jupyter Notebook through the Hasura CLI, you've unlocked a world of possibilities. The ability to automatically vectorize relational data has not only accelerated your data manipulation processes but has also paved the way for more efficient and effective analyses.

Moreover, the integration of robust security measures into your LLM queries ensures that your data remains protected at all times. This remarkable journey, powered by Hasura, empowers you to harness the true potential of your data, all while simplifying the intricate processes involved!

With Hasura by your side, what will you build?

Instant APIs on Snowflake using UDFs and Hasura

Hasura — Wed, 09 Aug 2023 13:32:08 +0000

The Hasura Snowflake connector became generally available in June. Like all other Hasura connectors, the value of the Snowflake connector is the ability to generate APIs (GraphQL and REST) on Snowflake data with minimal coding.

I wanted to put it to the test by recreating this blog from Snowflake’s team with Hasura.

The original post showed how to create an analytics app using OAG: Global Airline Schedulesdata. The architecture includes using AWS Lambda functions to read from Snowflake and return the data as RESTful APIs.

Architecture in the original blog.

We will implement these data APIs over Snowflake using Hasura. Right off the bat, we can instantly replace all the different Amazon components with Hasura, greatly simplifying the architecture by reducing five components to one.

Simplified architecture with Hasura.

Importing sample data to Snowflake

First and foremost, we must import the sample data to our Snowflake instance – the data can be found in the marketplace here.

Implementing the APIs in Hasura

This section will show how to instantly create the required APIs using Snowflake’s user-defined functions (UDF) and Hasura’s query engine.

A UDF is a function you define to call it from SQL. UDF’s logic typically extends or enhances SQL with functionality that SQL doesn’t have or doesn’t do well. A UDF also allows you to encapsulate functionality to call it repeatedly from multiple places in code.

We will implement the busy_airports API using UDFs and Hasura from the original blog.

Busiest airports API

This API endpoint finds the busiest airports in the data. We will use an API key as an admin secret. We want to filter the data based on flight_date and restrict the number of rows returned from the API.

1. Tracking UDFs in Hasura

First, we will create the following UDF in Snowflake’s SQL console:

Now, we will import the UDF to Hasura:

After that, we can query the UDF like any other table from Hasura:

2. Cache your Snowflake API

You can use the @cached directive to your query to decrease the response time. The default TTL is 60 seconds, reducing latency by about ~100 milliseconds.

Caching is useful when querying Snowflake, as latencies could be a bottleneck, primarily when serving customers on web/mobile applications. It can improve latencies and reduce the load on data warehouse resources by fetching the results from Redis.

You can also fetch the results using cURL:

url -X POST -H "x-hasura-admin-secret: admin-secret" -H "Content-Type: application/json" -d '{"query":"query MyQuery{BUSY_AIRPORTS_2(limit:20,order_by:{COUNT:desc}){ARRAPT COUNT}}"}' https://choice-platypus-97.hasura.app/v1/graphql
{"data":{"BUSY_AIRPORTS_2":[{"ARRAPT":"ATL","COUNT":131184},{"ARRAPT":"ORD","COUNT":120075},{"ARRAPT":"LHR","COUNT":106354},{"ARRAPT":"DFW","COUNT":105660},{"ARRAPT":"JFK","COUNT":98454},{"ARRAPT":"LAX","COUNT":91109},{"ARRAPT":"CDG","COUNT":87035},{"ARRAPT":"FRA","COUNT":86363},{"ARRAPT":"AMS","COUNT":85004},{"ARRAPT":"MAD","COUNT":61921},{"ARRAPT":"BOS","COUNT":58124},{"ARRAPT":"SEA","COUNT":56016},{"ARRAPT":"SFO","COUNT":55467},{"ARRAPT":"DEN","COUNT":53891},{"ARRAPT":"EWR","COUNT":49122},{"ARRAPT":"FCO","COUNT":48736},{"ARRAPT":"IAH","COUNT":47115},{"ARRAPT":"MIA","COUNT":46552},{"ARRAPT":"YYZ","COUNT":45225},{"ARRAPT":"CLT","COUNT":44266}]}}

Similarly, we can implement airport_daily and airport_daily_carriers following the same steps. Thus, we can access the data in minutes without writing a single line of code, just using UDFs in Snowflake and Hasura.

The APIs are protected with API keys. Suppose you want a role-based authorization layer. In that case, this blog covers how to implement role-based access control on Snowflake: “ Snowflake using RBAC: A secure and scalable data access solution.”

Conclusion

Within a few minutes, we replaced five different AWS components, implemented an API without writing a single line of code, added an admin secret for authentication, and added the ability to cache to provide low-latency data access.

Hasura avoids getting entangled with details and lets you focus on building the end product.

🚀 Get Started Today!

We can't wait to see the amazing application you'll build using Hasura and Snowflake. Start by signing up for Hasura Cloud and connecting your Snowflake data warehouse.

If you have any questions or need assistance, please contact our team on Discord or GitHub.

Supercharge your application development with Hasura Remote Joins and Data Federation

Hasura — Tue, 01 Aug 2023 13:36:10 +0000

We are thrilled to announce that Hasura now supports Remote Joins across all supported data sources , including PostgreSQL, Snowflake, MySQL, SQL Server, BigQuery, Oracle, Athena, and Remote Schemas!

In the world of modern application development, building complex data relationships across various sources has become a common requirement. As databases and data sources grow in complexity, developers often find themselves facing the challenge of integrating data from different databases, APIs, or remote GraphQL endpoints.

Thankfully, Hasura’s GraphQL engine provides a powerful solution through its remote relationships, also known as "Remote Joins."

In this blog, we will explore the concept of remote relationships and how they can be leveraged to join data across tables and remote data sources, empowering developers to build sophisticated and data-rich applications.

Understanding remote relationships

At its core, a remote relationship in GraphQL allows you to connect data across different tables and remote data sources. These sources can include:

Database to database relationships: This type of remote relationship enables you to join data between two different database sources. For instance, you can link order information stored in one PostgreSQL database with user information stored in another PostgreSQL or even a SQL Server database. GraphQL acts as the glue that seamlessly integrates the data from both sources.
Database to GraphQL services: Here, you can merge data across tables with remote GraphQL APIs that we call Remote Schemas. For example, you might combine customer data from your database with account data from external services like Stripe, Spotify, or Auth0. This integration allows you to consolidate information from various sources into a single, cohesive GraphQL query.
GraphQL services to database relationships: In this scenario, you can connect data from Remote Schemas (representing services like Stripe, Spotify, or Auth0) to customer data from your database. This bidirectional relationship unlocks a plethora of possibilities for building feature-rich applications that rely on data spanning across multiple services.
Custom business logic to database relationships: Actions in GraphQL often correspond to REST APIs. With this type of remote relationship, you can join data across tables and Actions, such as fetching user data from your database and combining it with the response from a createUser action using the id field.

How Hasura performs Remote Joins?

When performing a RemoteJoin between “table A” and “table B,” Hasura will first query “table A” for its rows. From these rows Hasura will collect all the IDs involved in the join to “table B.” It will then query “table B” as a separate query, filtering the rows by the IDs collected from “table A.” Hasura will then stitch the rows returned from “table B” into the results from “table A” to provide the final response to the original GraphQL query.

This approach avoids the usual N+1 query anti-pattern that can occur with naive implementations that join between two tables in different databases. The N+1 query anti-pattern is where you query “table B” for every row returned by the query to “table A,” which is inefficient.

In Hasura, join queries are optimized to query each table once, limiting the number of separate queries issued to the smallest possible number.

How do Remote Joins enhance application development?

Businesses today are not confined to a single database. Data exists in a multitude of sources – on-premises, in the cloud, or distributed across multiple systems. To maintain a competitive edge, organizations must harness the power of these disparate data sources and seamlessly combine them to drive meaningful use cases and insights.

In a matter of minutes, developers can create a single GraphQL API using Hasura that accesses data from multiple sources, including GraphQL APIs (Remote Schemas) and databases, in real time, without the need for complex data integration processes. This means developers can create powerful applications that access data from different sources, without worrying about data silos.

In traditional SQL, joining tables from different databases requires maintaining a duplicate copy of the data from one database in the other database that is needed to perform the join. This process was not only time-consuming but also required a lot of resources. With Hasura's remote joins, developers can perform joins across data sources without copying any data. This allows organizations to federate their data into a single self-service data access layer.

By providing a unified view of data, Hasura's remote joins can significantly reduce the time it takes to develop data intensive applications. Developers can quickly prototype applications, test different use cases, and get feedback from stakeholders. This enables enterprises to bring applications and data products to market faster, giving them a competitive edge.

Simplified data management

Handling and managing data across multiple databases is a significant challenge for developers. It often involves writing complex queries and spending countless hours on data management tasks. However, with Hasura's Remote Joins, these challenges become a thing of the past. Remote Joins provide a way to access and unify data across various databases as if it were coming from a single data source.

Boosts development speed

By offering a unified API across different databases, Hasura saves developers from the tiresome task of orchestrating data from various sources. This allows developers to focus more on the core functionality of the application, ultimately boosting the development speed.

Streamlined workflow

A GraphQL API that bridges different data sources simplifies your workflow. It not only helps you fetch data from multiple sources in a single request but also keeps the interface clean and organized, significantly streamlining the development process.

Getting started with Remote Joins for databases

Check out my colleague Vaishnavi’s demo from her talk at HasuraCon 2023 on joining data from Postgres to MySQL.

Step 1: Add two database sources.

Add a source database as described here and track the required tables. Then, repeat the process to add your target database.

Step 2: Define and create the relationship.

A remote database relationship is defined alongside the source database table (that is, the source side of the join).

The following fields can be defined for a Remote Schema relationship:

Relationship type: Either object or array – similar to normal relationships. Hasura supports both many-to-one (object) and one-to-many (array) relationships.
Relationship name: A name for the relationship.
Reference source: The name of the target database (that is, the target side of the join).
Reference table: The table in the target database source that should be joined with the source table.
Field mapping: A mapping between fields in the source table and their corresponding fields in the target table, just as a foreign key relationship would be defined by such mapping within a single database.

For example, say we have a table Album(AlbumId int) in the source database and a table Track(id int, name text) in the target database.

We can create an array remote database relationship between the AlbumId joining the Album table to the Track table using the Album.AlbumId and Track.AlbumId fields.

Conclusion

When it comes to generating APIs for multiple data sources, Hasura offers a powerful and developer-friendly solution. Its ease of use, automation, and feature-rich capabilities provide a significant advantage over the DIY approach, which requires significant time and effort to design and implement custom solutions.

With Hasura, developers can create a single GraphQL API that can access data from multiple sources allowing developers to focus more on building innovative applications that leverage diverse data sources, without getting bogged down by the complexities of API generation.

If you're looking to simplify the process of joining data from multiple sources, Hasura is the way to go.

📚 Documentation and resources

To help you get started, we've prepared detailed documentation, guides, and examples:

🚀 Get started today!

We can't wait to see the amazing applications you'll build using Hasura and Native Queries. Get started today by signing up for Hasura Cloud and connecting to one of the supported databases.

If you have any questions or need assistance, feel free to reach out to our team on Discord or GitHub.

Introducing Input Validation Permissions on Hasura: Enhancing data integrity and security

Hasura — Thu, 27 Jul 2023 13:40:48 +0000

In today's data-driven world, building applications with strong data integrity and security is challenging. Ensuring that the data being processed is accurate, valid, and aligned to predefined rules is a critical aspect of modern application development.

To empower developers with more control over their data validation process, Hasura is thrilled to announce the launch of a powerful new feature – Input Validations!

Why data integrity and security matter

Data integrity refers to the accuracy, consistency, and reliability of data stored and processed within an application. Ensuring data integrity is essential for a number of reasons:

Reliable decision-making: Accurate data enables sound decision-making, leading to better business outcomes and customer experiences.
Data consistency: Inconsistent data can cause errors, leading to confusion and incorrect results.
Data security: Validating input data helps prevent security vulnerabilities, such as SQL injections, and safeguards sensitive information.
Compliance and regulations: Many industries have strict data compliance and privacy regulations, making data integrity a legal requirement.

Maintaining data integrity and security can be a challenging task, especially in complex applications with numerous data mutations and interactions. Input Validations provide a robust solution to tackle these challenges.

Introducing Input Validations

Hasura's Input Validations allow developers to implement custom data validation logic for GraphQL mutations. This feature acts as a pre-mutation hook, offering developers the ability to validate input arguments before executing insert, update, or delete operations. By defining rules and constraints on an HTTP service or a serverless function that can be pointed from Hasura as input validation endpoint, developers can ensure that only valid data is processed and stored in the database.

How Input Validations work

When a GraphQL mutation arrives, targeting specific tables and roles, the Input Validations feature comes into action. The mutation arguments are routed to the defined HTTP webhook, where custom validation logic is executed. If the validation is successful, the mutation proceeds, and the data is processed as intended. On the other hand, if the validation fails, the mutation is aborted, and appropriate error messages can be relayed back to the client.

You can set up an Input Validation rule for a role directly from the Hasura Console as easy as configuring a normal permission rule in Hasura.

Let’s take an example of allowing user sign ups with age >18 years and I’m going to create a table as shown below.

Head to the table permissions page and you will now see a new section as shown below, lets add an input validation configuration as shown.

Now lets create a simple web server to validate the data inputs. Authoring the webhook validator service is quite easy:

To approve an input: Resolve the HTTP request with a 200
To reject an input: Resolve the request with 400 and a message payload, the message payload will be forwarded to the client as a GraphQL error message for reference

Here is a simple NodeJS server that handles the input from Hasura and validate the input only if the user is more than 10 years old.

const express = require("express");
const bodyParser = require("body-parser");

const app = express();
app.use(bodyParser.json()); // to support JSON-encoded bodies
app.use(bodyParser.urlencoded({
  extended: true
}));

app.get("/", (req, res) => {
  res.send("Server is running!");
});
app.post("/validateNewUserData", (req, res) => {
  console.log("INPUT_", req?.body?.data?.input);
  const DOB = req?.body?.data?.input?.[0]?.DOB;
  const age = ~~((new Date() - new Date(DOB)) / 31557600000);

  console.log("Age", age);
  if (age > 10) {
    res.send("SUCCESS");
  } else {
    res.status(400).json({
      message: "User should have a minimum age of 10"
    });
  }
  return;
});

app.listen(8080, function() {
  console.log("Server is running on 8080");
});

Now let’s test this by making a mutation on Hasura

You can find a sample repository with the express-js server data validator here: https://github.com/soorajshankar/Hasura-Input-Validation-Demo

Read more about the configurations of Input Validations in detail from our official docs page here.

Use cases and problems solved

Input Validations can address a wide range of use cases and solve common challenges faced by developers:

User registration and authentication: Validate user registration details like email addresses, usernames, and enforce password complexity rules.
Managing user-generated content: Implement content guidelines and moderation for user-generated data, ensuring compliance and preventing inappropriate content.
Enforce restrictions on allowing requests: Ensure that an order cannot be placed with more than a certain amount of an item.

Hasura's Input Validations offers a powerful tool for developers to enhance data integrity and security in their applications. With this feature, developers can ensure that the data flowing through their systems is accurate, valid, and protected from security vulnerabilities.

By configuring custom validation logic through HTTP webhooks, developers gain control over the entire validation process, enabling them to build more reliable and secure applications.

Stay tuned for more in-depth insights into this exciting new feature! Sign up now to try Input Validations on your project.

Sign up to start your journey with Hasura Cloud.

Incremental DB migration with Hasura

Hasura — Mon, 24 Jul 2023 18:07:32 +0000

Database migration is a complex, multiphase, multi-process activity. The challenges scale up significantly when you move to a new database vendor or across different cloud providers or data centers.

Despite the challenges, sometimes data migration is the only way. So how do we peel back the layers of ambiguity, keep the downtime minimal, and ensure our user's experience does not suffer when embarking on this journey?

I had faced a similar change in the past when I was at one of Asia/India’s largest e-commerce shops. Our database storage and compute capacity had maxed out, we were using ~90%+ of our compute resources, and we had about two months left before running out of storage space.

We had to act fast and execute the entire process within that two-month period while ensuring our business continued to grow. It was a stressful period with some leap-of-faith calculations to make it happen.

In this post, I want to revisit the migration process and see how, with Hasura, we could incrementally execute the migration, tackling the uncertainties but still delivering the results on time.

The problem

We want to migrate data from one SQL database to another SQL database vendor on the cloud, and we want to achieve this process with minimal disruption.

The challenges

What are the typical challenges we must solve for when moving to a new database vendor?

Database schema migration

Usually, there are differences in data types, index types, and architectures between different vendors that need to be addressed before starting the migration.
Some characteristics need functional/performance testing to be sure.
These differences arise even if we move to a different vendor for the same types of databases (E.g., MySQL to PostgreSQL).
However, these differences further magnify when moving from SQL to NoSQL store, where write and access pattern change considerably due to schema/schema-less changes.

Data transformation

Due to changes in schema and data types, one would need to transform the data before persisting.
These ETL pipelines can become complex, as we need to reason schema by schema.
It can be a mammoth task if the table count is large (20+) with a large amount of data.

Data migration

This step could be challenging if the databases are co-located or on different cloud providers.
Inside the same data centers, the network bandwidth is usually plenty to move 100s of GB of data in a few mins to a few hours.
However, the same cannot be said for migrating the data over the internet, there are many failure scenarios to consider, and the time it takes can be a couple of days.

Performance/functional testing

If one is migrating all the data simultaneously, testing the new database's migration and functionality/performance is paramount.
There could be a need for tweaking data types, indexes, or queries that might not work from older databases.
This step requires a thorough review of the data modeling, and the application code, specific queries, and consistency guarantees must change when undertaking such migrations.

API migration

When migrating data across databases, data access patterns with existing APIs become crucial.
If it’s a migration across a couple of days or weeks, some requests must be handled while pointing to the new database vendor, while prior data access must be redirected to the older database.
If the data is being asked in multiple databases, we must handle cross-DB queries/joins.
For writes, moving partial data could be challenging, as new inserts have to be written in the new datastore, while updates for old rows have to be redirected to the old DB for consistency.
However, if one moves entire tables to the new database, the data access layer can handle many such challenges.

An approach to solving DB migration challenges

Given the challenges above, how do we go about this problem statement?

First, the data migration problem is also an API migration problem. If we can provide the flexibility of querying multiple data sources while running existing business logic, we can support migrations using an incremental approach.

Her are the steps, using Hasura, to help address the uncertainties:

Introduce Hasura and connect it to the old database.

Connect existing API service with Hasura.

The existing APIs will become a source of business logic and data validation for Hasura, and with that, we can start migrating some of the client API calls to Hasura.
For GraphQL, we can use remote schemas, and for REST APIs, we can use actions.

Once we've migrated some client API calls, we can slowly move all clients to Hasura. Post that, we can start replicating some table schemas to the new database vendor. There are many ways to replicate data across different data sources.
Once those tables have been replicated successfully and after conducting functional and performance tests over the new data source, we can connect Hasura with the new database.

Connect Hasura to the new database.

With this, queries to new tables will be directed to the latest data source, rest of the queries will be redirected to the existing database.

Make remote joins across databases.

In case of joins between tables between two databases, we can use Hasura’s Remote Joins feature, which allows us to fetch data from multiple data sources in a performant manner.

With this, we have achieved a framework for incrementally migrating data to the new vendor. By having a strategy where we can move a few tables at a time, we can take our time to squash any issues that arise due to changes in the database vendor, slowly increasing our data footprint over time.

Remove old data sources from Hasura.

Once we have moved all our data, we can discard and disconnect the old data source from Hasura.

Conclusion

We saw how using Hasura, we could have a clear strategy for migrating API, which allowed us to move data to the new vendor incrementally while causing minimal client disruptions.

Using this approach, we can easily guard against any uncertainties that may arise when moving to a new database vendor while providing a fantastic experience to our API consumers.

Sign up now for Hasura Cloud and get started for free!

The why of GraphQL Client Side Nullability in Examples

Hasura — Wed, 19 Jul 2023 21:18:07 +0000

A terse exposition of how client side nullability can inform client component design through comprehensive examples.

A nullable field can represent a value that may or may not exist.

Client side nullability can be used to solve common issues when defining the data fetch for client side components by:

Validating that all the fields the component expects are available in one shot
Simplifying the types on fetched data from nullable types to non nullable types (especially elegant when using statically typed languages where you have to unwrap the value from an optional type).
Modifying how errors or null values affect the returned fields based on bubble-up logic

Bubbling on the server (a recap)

Let's set up an example server API to use in the next sections on client side component data fetches.

On the server, every field is nullable by default; the GraphQL spec allows the server to return a "partial response." A resilient API can resolve all the fields it is able to and return errors on the side. Errors can include system failures (network/database/code) or even authorization errors. Nullable by default also eases API evolution by ensuring that clients are responsible for validating fields in the response data.

A null value in a field that is not nullable bubbles up to the nearest nullable field.

For example, with the following query and types:

# sample query 1 BASIC
query {
  user {
    email
    profile { # nullable
      picture
      address { # not nullable
        street # not nullable
        country
      }
    }
  }
}

# server schema
type User {
  email: String
  profile: Profile # fields in graphql are nullable by default
}
type Profile {
  picture: String
  address: Address! # not nullable
}
type Address {
  street: String! # not nullable
  country: String
}

If something went wrong while fetching the non-nullable "street" field, then the server would bubble up that error to the nearest nullable field "profile", and you would get a null profile field. For this query, either you get a profile with a street, or no profile at all. This is true even if "picture" was a valid value.

// return when non-nullable street not resolved
{
    user: {
        email: ...
        profile: null // null bubbled up to first nullable field
    }
}

// return if street value is resolved
{
    user: {
        email: ...
        profile: {
            picture: ...
            address: {
                street: ... // street has a non null value
                country: ...
            }
        }
    }
}

Moving Control to the Client With Nullability Designators

Client side nullability introduces two new operators ! named "required" and ? named "optional" that the client can use to specify how the server should bubble up null errors.

Solution 1: Narrow optional data with smaller error boundary in a profile widget

The client can force the server to return some data while marking other data as optional.

# sample query 2 PROFILE WIDGET, smaller boundary
query {
  user {
    email
    profile { # nullable
      picture
      address? { # not nullable, optional
        street! # not nullable, required
        country
      }
    }
  }
}

In case of an error or null value while resolving the required field "street", the null value would propagate up to the closest optional field "address", not the closest nullable field. So this allows for a client side developer to override overzealous nullability requirements that the server specifies and get this response:

// sample response 2 PROFILE WIDGET
{
    user: {
        email: ...
        profile: {
            picture: ...
            address: null // closest optional field is null
        }
    }
}

The profile widget can now display a profile picture even when the address is invalid.

Solution 2: Simplify data validation with larger error boundary in a location widget

The client can indicate that either all or no data should be returned to simplify validation of the returned data.

# sample query 3 USER LOCATION WIDGET, larger boundary
query {
  user? { # optional
    email
    profile { # nullable
      picture
      address { # not nullable
        street! # not nullable, required
        country
      }
    }
  }
}

A null value for the required street field still propagates to the closest optional field, so we get this result:

// sample response 3 USER LOCATION WIDGET
{
    user: null
}

This means that the user location widget doesn't need to dig into the internals of the returned user data and validate that the fields returned as expected, the null at the optional field already captures the bubbled up error.

Solution With Relay

The Relay library has had this for some time through the @required directive specific to the Relay library, and different from the client side nullability GraphQL spec.

query {
  user {
    email
    ... profileFragment
  }
}

fragment profileFragment on users {
  profile {
    picture
    address @required {
      street @required
      country
    }
  }
}

If "street", an @required field is missing, the null value will bubble up to the first field that is not @required, but only within the same fragment i.e. the "profile" field.

{
    user: {
        email: ...
        profile: null
    }
}

This is because Relay likes to use fragments to locally scope data fetching requirements with data masking. An advantage with the Relay approach is that the directives are implemented on the client, the server does not need to support the @required directive, unlike the client side nullability spec.

Relay is also looking to push the client side nullability spec further with Fragment Response Keys which define fragment composition boundaries for the client side nullability designators (GraphQL Fails Modularity on YouTube).

Using this today

The client side nullability spec is currently in the RFC stage, and Hasura does not support it today.

You can however use the @required directive with Relay today, and Hasura can generate a Relay API. Check it out if the idea of having data requirements colocated with your components using fragments stands out to you. It also makes for a phenomenal developer experience.

If you're using Hasura backed by a single PostgreSQL database, you don't really have to worry about the server returning unexpected null values.

In a federated setup, with Hasura backed by multiple data sources across the network, something like client side nullability could help clients account for partial failures when resolving data. Hasura v3 will handle these scenarios more elegantly by allowing partial fetches even with some of the backing data sources down.

From a client perspective, a developer probably wants to focus on nailing down data requirements for every component regardless of the status of the backing data sources.

The GraphQL spec continues to evolve to solve real problems that developers encounter.

Here are some active discussions for GraphQL features that are still in the works! GraphQL Working Group RFCs

References

https://github.com/graphql/graphql-wg/blob/main/rfcs/ClientControlledNullability.md

http://spec.graphql.org/October2021/#sec-Handling-Field-Errors

https://relay.dev/docs/next/guides/required-directive/

https://hasura.io/blog/graphql-nulls-cheatsheet/

Other discussions:

https://github.com/graphql/graphql-wg/discussions/1009

https://github.com/graphql/graphql-wg/discussions/994

https://www.youtube.com/watch?v=SVx4HG2bhII

Harnessing the power of MuleSoft and Hasura

Hasura — Wed, 19 Jul 2023 19:17:25 +0000

In the rapidly evolving world of API development and integration , organizations often encounter complex challenges that require a combination of powerful tools and technologies. Two such platforms, MuleSoft and Hasura, offer unique capabilities that can be harnessed together to create a comprehensive and efficient API ecosystem.

In this blog post, we will take a deep dive into MuleSoft and Hasura, exploring their individual merits, discussing real-world use cases, and providing architectural insights on how these platforms can complement each other to meet the diverse needs of modern enterprises.

Understanding MuleSoft

MuleSoft, acquired by Salesforce, is a leading integration platform that enables organizations to seamlessly connect disparate systems, applications, and data sources. With its extensive library of connectors, MuleSoft simplifies integration challenges by providing out-of-the-box connectivity to various systems, including enterprise applications, databases, cloud services, and more. It offers a visual integration development environment that empowers developers to design, build, and manage APIs, ensuring efficient data flow across the enterprise. MuleSoft's robust capabilities include data transformation, API orchestration, message routing, and comprehensive monitoring and management tools.

MuleSoft shines as an API gateway layer, enabling organizations to manage, secure, and control their APIs. Its versatile workflow capabilities streamline the integration process, ensuring seamless orchestration of data and processes. Beyond its API gateway functionality, MuleSoft's specialized B2B/EDI integration platform deserves a special mention, catering to the specific requirements of B2B and EDI integration use-cases. With MuleSoft, organizations can effortlessly connect systems, enhance API governance, and drive efficient workflows.

Deep dive into MuleSoft

Message transformation: MuleSoft provides powerful data transformation capabilities, allowing you to easily convert data between different formats, protocols, and systems. With its graphical mapping editor and built-in data mapping functions, you can efficiently handle complex data transformation scenarios.

Orchestration and workflow: MuleSoft's visual flow designer enables the creation of complex integration workflows and orchestration processes. It allows you to define conditional routing, error handling, and exception management, making it suitable for implementing sophisticated integration scenarios.

Exploring Hasura

Hasura, on the other hand, is an open source data platform that provides instant GraphQL APIs over existing databases and existing REST and GraphQL endpoints. With Hasura, developers can rapidly build secure and performant APIs without writing complex boilerplate code. It acts as a powerful data access layer, automatically generating GraphQL APIs based on the database schema. Hasura's real-time capabilities, powered by GraphQL subscriptions, enable developers to build interactive and collaborative applications with instant updates. It also offers granular access control and authorization mechanisms, allowing developers to define fine-grained permissions and ensure data security and compliance.

Hasura takes the spotlight as a data connectivity layer, revolutionizing API development and transformation. By seamlessly integrating with OLTP and OLAP databases, Hasura provides an automated solution for generating GraphQL APIs. With its powerful API integration capabilities, developers can quickly build robust APIs and effortlessly handle data connectivity challenges. Hasura simplifies the complex task of API integration and transformation, empowering developers to accelerate their development cycles and deliver exceptional user experiences.

Deep dive into Hasura

Rapid API development: Hasura's strength lies in its ability to generate GraphQL APIs from existing databases rapidly. With its intuitive UI and CLI tools, you can easily design, build, and deploy APIs in minutes, significantly accelerating your development cycles.

Real-time capabilities: Hasura offers real-time GraphQL subscriptions, enabling instant updates and real-time data synchronization. This feature is particularly useful for applications requiring live updates, collaborative features, or notifications.

Fine-grained data access control: Hasura provides granular access control and authorization mechanisms, allowing you to define permissions and policies at the API layer. This ensures that only authorized users have access to specific data, enhancing security and compliance.

Real-world use cases and architecture

Let's explore three real-world use cases where the combination of MuleSoft and Hasura can be particularly powerful:

1. E-commerce integration: MuleSoft can connect e-commerce platforms, inventory systems, payment gateways, and shipping providers, ensuring a seamless customer experience. Hasura can be used to provide instant GraphQL APIs over the product catalog and inventory databases, enabling real-time inventory updates and personalized product recommendations.

2. Healthcare data integration: MuleSoft can integrate patient management systems, electronic health records (EHR), and billing systems, streamlining data exchange and ensuring compliance with healthcare standards. Hasura can serve as the data access layer, allowing healthcare providers to query patient data through GraphQL APIs securely.

3. Financial services integration: MuleSoft can connect banking systems, payment processors, and customer relationship management (CRM) platforms, facilitating secure and efficient financial transactions. Hasura can be used to provide real-time insights into customer transactions and account balances, enabling personalized financial dashboards or fraud detection systems.

Architecture

In the above architecture, the client and the application layer directly interacts with MuleSoft, which serves as the API gateway/layer and workflow platform. MuleSoft handles the routing, transformation, and management of API requests, providing a centralized point for API governance and control. MuleSoft also does the integration of APIs that Hasura doesn’t work with or which MuleSoft handles best. This includes the workflow management and B2B EDI integration APIs.

Hasura, on the other hand, is positioned as the data connectivity layer. It seamlessly connects to OLTP and OLAP databases and provides the ability to perform API integration and transformation. Hasura simplifies API development by auto-generating a GraphQL API layer from existing databases, enabling efficient data access and manipulation.

The databases, represented in the architecture, can be OLTP (Online Transaction Processing) and OLAP (Online Analytical Processing) databases. Hasura can handle both types, allowing developers to connect to and query the databases using GraphQL APIs.

This architecture highlights the clear division of responsibilities between MuleSoft and Hasura, with MuleSoft focused on API gateway functionality and workflow management, while Hasura excels in data connectivity, API integration, and handling OLTP and OLAP databases.

When to use Hasura with MuleSoft

Let’s look at the main scenarios where it makes sense to use Hasura with MuleSoft:

1. Simplified Data Access: When your primary focus is on efficient data access and manipulation, especially with real-time requirements, Hasura's GraphQL engine offers a powerful solution. It is perfect for applications that necessitate real-time updates, collaborative editing, or interactive data querying.

2. Hybrid Approaches: In certain cases, a combination of MuleSoft and Hasura can be beneficial. For instance, you can leverage MuleSoft's integration capabilities to connect disparate systems and use Hasura as a data access layer, providing a unified GraphQL API over the integrated data sources. This approach combines the best of both platforms to deliver a seamless and efficient solution.

Combining MuleSoft and Hasura

When considering the combination of MuleSoft and Hasura, it's essential to understand their complementary strengths and architectural implications.

Here are some key scenarios where their combined usage can prove beneficial:

1. Hybrid Integration Scenarios: MuleSoft excels in integrating heterogeneous systems, enabling seamless communication between various enterprise applications, cloud services, and databases. In scenarios where real-time data access and manipulation are required, Hasura can be used as a data access layer for instant GraphQL APIs over the existing databases. This combination allows for efficient data integration and real-time capabilities while leveraging MuleSoft's comprehensive connectivity and transformation features.

2. Microservices Architecture: In a microservices architecture, MuleSoft can serve as the integration layer, orchestrating communication between different microservices and providing a unified API gateway. Hasura can be used within the microservices themselves, generating GraphQL APIs to access and manipulate the respective microservice's data. This approach allows for decoupled microservices with their own GraphQL APIs, while MuleSoft ensures seamless communication and integration between them.

3. Accelerating API Development: Hasura's automatic API generation capabilities provide developers with a rapid development experience. By integrating Hasura with MuleSoft, developers can leverage MuleSoft's powerful transformation and connectivity features to enrich the data returned by Hasura's GraphQL APIs. This combination allows for accelerated API development while ensuring data consistency, enrichment, and adherence to enterprise standards.

4. Real-time Data Synchronization: Hasura's real-time capabilities make it ideal for scenarios where real-time data synchronization is required, such as collaborative applications or real-time dashboards. MuleSoft can integrate with Hasura to fetch and transform data from various sources, and Hasura's real-time subscriptions can enable instant updates and real-time notifications to connected clients.

Comparing MuleSoft and Hasura

To better understand why combining Hasura and MuleSoft is so powerful, let’s compare the two systems. The contrast shows us how complimentary they really are:

Features

MuleSoft

Hasura

|
|

Integration type

Enterprise Integration Platform

Data Platform with auto-generated APIs

|
|

Connectivity

Extensive connectors for various systems and protocols

Connects to existing databases for API generation

|
|

Message transformation

Powerful data transformation capabilities

Directly maps database schema to GraphQL API

|
|

Orchestration

Robust visual flow designer for complex integration workflows

N/A (Focuses on API generation and real-time features)

|
|

Real-time capabilities

Limited support for real-time updates and notifications

GraphQL subscriptions for real-time data synchronization

|
|

Rapid API development

Requires configuration and development effort

Auto-generates GraphQL APIs with a minimal development effort

|
|

Data access control

Provides fine-grained access control and authorization mechanisms

Offers granular control over API permissions and policies with ABAC and Authorization

|
|

Scalability

Designed for complex enterprise integrations

Suitable for projects of all sizes, including startups and smaller applications

|
|

Learning curves

Steeper learning curve due to its extensive feature set

Relatively easier to learn and get started with

|
|

Use cases

Large-scale enterprise integrations involving diverse systems

Modern applications requiring real-time updates and collaborative features

When comparing MuleSoft and Hasura, it's essential to understand their differences in focus and functionality. MuleSoft shines in enterprise-level integrations, offering extensive connectors, data mapping, and transformation capabilities. It excels in large-scale deployments, legacy system modernization, and hybrid cloud scenarios. Hasura, on the other hand, is tailored for efficient data access and manipulation through GraphQL. It provides real-time capabilities, automatic schema generation, and authorization mechanisms.

Conclusion

MuleSoft and Hasura, with their distinct capabilities, can be harnessed together to create a comprehensive and efficient API ecosystem. MuleSoft excels in complex integrations, enterprise-level security, and large-scale deployments. Hasura simplifies data access and manipulation through GraphQL, providing real-time capabilities.

By combining MuleSoft's powerful integration capabilities with Hasura's rapid API development and real-time capabilities, organizations can achieve enhanced agility, seamless data flow, and improved user experiences. By architecting solutions with MuleSoft and Hasura and adding their complementary strengths to align them with your specific use cases, you can unlock new possibilities in API development and integration, empowering your organization to innovate and thrive in the digital era.

Sign up now for Hasura Cloud to get started!

Breaking up monolith into microservices with Hasura

Hasura — Tue, 18 Jul 2023 14:55:36 +0000

Unleashing scalability and agility

As applications grow in size and complexity, the monolithic architecture that once served them well can become a bottleneck. Monolithic applications are difficult to scale, maintain, and deploy. Microservices, on the other hand, offer a more flexible and scalable architecture that allows for independent development and deployment of smaller, focused components.

In this blog post, we will explore how Hasura can help break up a monolithic application into microservices, enabling a more efficient and scalable architecture.

Understanding monoliths and microservices

Before diving into the migration process, it is important to understand the key differences between monolithic and microservices architectures.

In a monolithic architecture, the entire application is built as a single, tightly-coupled unit. This makes it difficult to isolate and scale individual components independently. On the other hand, microservices architecture breaks down the application into small, autonomous services that can be developed, deployed, and scaled independently.

Architecture

Let’s look at the detailed process of breaking a monolithic application into microservices step-by-step:

Step 1: Identify microservice boundaries

The first step in breaking up a monolith is to identify the boundaries of your microservices. This involves analyzing the existing monolithic application to identify cohesive and loosely coupled modules that can be decoupled and deployed as independent services. Each microservice should have its own bounded context and serve a specific business capability.

Step 2: Create GraphQL APIs with Hasura

Hasura provides a powerful GraphQL engine that can act as a gateway to your microservices. Instead of exposing each microservice's API directly to clients, you can create a unified GraphQL API using Hasura. Hasura simplifies this process by automatically generating the GraphQL schema based on the underlying data sources. You can leverage Hasura's console or configuration files to define relationships, access control rules, and custom business logic.

Step 3: Extract microservices from the monolith

With the microservice boundaries defined and the GraphQL API set up, you can start extracting individual microservices from the monolithic application. This can be done incrementally by identifying a well-defined module or functionality within the monolith and moving it into its own service. Hasura's GraphQL API acts as a bridge between the existing monolith and the newly created microservices, allowing for seamless integration and coexistence.

Step 4: Refactor and redesign

During the extraction process, you may need to refactor and redesign certain components to ensure they align with microservices principles. This may involve restructuring databases, revisiting domain models, and optimizing communication patterns between microservices. Hasura's powerful data modeling capabilities, such as handling relationships and custom resolvers, can assist in this process.

Step 5: Deployment and scalability

Once the microservices are extracted and properly designed, you can deploy them individually using your preferred deployment strategy. Hasura integrates well with popular containerization technologies like Docker and orchestration platforms like Kubernetes, making it easier to deploy and scale microservices. Each microservice can have its own independent deployment pipeline, enabling continuous delivery and reducing the risk associated with deploying changes.

Step 6: Monitor and maintain

Monitoring and maintaining microservices is crucial to ensure their health and performance. Hasura provides monitoring and logging capabilities that can be leveraged to gain insights into the usage, performance, and error handling of your microservices. By utilizing these tools, you can proactively identify and address any issues that may arise, ensuring the smooth operation of your distributed application.

Conclusion

Breaking up a monolithic application into microservices can be a complex undertaking, but with the help of Hasura, it becomes more manageable. Hasura's GraphQL engine simplifies the creation of a unified API layer for your microservices, enabling seamless integration and coexistence with the existing monolith.

By following the steps outlined in this blog post, you can transform your monolithic application into a scalable and flexible microservices architecture, unlocking the benefits of independent development, deployment, and scalability.

Sign up now for Hasura Cloud to get started for free!

The complexity of building a GraphQL API permissions layer and how Hasura solves this

Hasura — Wed, 05 Jul 2023 09:17:26 +0000

Introduction

API security breaches are on the rise. Gartner predicts that by 2025, insecure APIs will account for more than 50% of data theft incidents. As enterprises continue to embrace an API-driven approach to software development (for all the benefits it brings), arming their developers with the tools to build secure APIs with proper data access and authorization logic needs to be a priority.

Building an authorization layer involves many factors. In GraphQL, authorization belongs in the business logic layer and not typically inside resolvers. It is more complex to write an authorization layer for GraphQL APIs than REST APIs. In this blog, we will see what it takes to build authorization logic for a DIY GraphQL server to understand how it is more complex. We will also compare it with Hasura and how it solves this complexity by being declarative and leveraging predicate pushdown.

Here's a quick TL'DR and summary before diving deep into this post.

Key items to consider

Data modeling

Data models contain key information about the types of data, relationships between data. This is used to determine the kind of authorization system that needs to be built.

For example, you have users of the application that can view public data of all users and certain private data of them. The data model now contains certain table columns relevant to this and also relationships that are relevant to this private data.

Roles and attributes

The common ways to model your authorization system is through RBAC (role-based access control and ABAC (attribute-based access control).

You could start defining roles for the application and list out use cases and privileges for each. RBAC could be flat, hierarchical, constrained and symmetrical. Authorization can also be modeled based on attributes where the user who logs in to the application will have certain attributes (type of user, type of resources, and the environment in which they are accessing) that can be checked for allowing access.

Nested rules

The GraphQL schema may or may not be a direct map to the data models. In some cases, the data model extends to multiple data sources. Even within the same data source, the GraphQL query could be nested with multiple fields spanning relationships. Applying authorization logic contextually to the nested query is vital.

For example:

query {
  users {
    id
    name
    orders { // apply permission rule here too
      id
      order_total
    }
  }
}

Here the orders is part of the users relationship and is nested. But the authorization logic needs to apply for both users, for orders, and for any level of nesting of the query, contextually.

Performance

Ideally, authorization checks shouldn’t add a lot of overhead to the response latency. In reality, when you are writing a custom GraphQL server yourself, checks are done after the data fetching operation and there’s a lot of unnecessary data fetched, which slows down the database if you consider millions of requests which slows down API performance as a result.

Predicate pushdown

If the authorization logic is heavily dependent on the data being fetched, it is important to do a predicate pushdown of the Authorization rules.

For example: If you are fetching orders placed by a user on an e-commerce app, you need to be able to apply the rule in the query that goes to the database to fetch only the orders placed by the user requesting them. This is not only faster, but also the most secure way to apply authorization logic.

Building a DIY authorization layer for a GraphQL API

You can build an AuthZ layer using middleware libraries. The maturity of AuthZ libraries in GraphQL depends on the language or framework you are in.

Why is building an authorization layer complex?

When you are writing your own GraphQL server with custom authorization rules, there are a number of methods to write this logic, depending on the use case. You have:

API-wide authorization
Resolver-based authorization
Schema / model-based authorization

The resolver-based authorization quickly balloons into a lot of boilerplate code if authorization rules are applied to every field. Even with repeated rules and patterns that can be applied, it can easily expand to thousands of lines of code to secure your application. And this code becomes difficult to maintain.

In the schema-based authorization, the authorization logic is dependent on the GraphQL schema and becomes independent of the underlying database or data fetching libraries and ORMs.

Authorization in GraphQL is typically built using _ Context _ object that is available on every request. The context is a value that is provided to every resolver and is created at the start of a GraphQL server request. This means you can add authentication and authorization details to the context, such as user data.

Here’s how a typical AuthZ context gets passed in a DIY GraphQL Server written in Node.js:

Parsing and validating JWT token

The first step is to parse and validate the incoming JWT token. Once the JWT token is verified, you can pass it to the context. We are using JWT tokens as an example here, because it is universal and works across platforms.

An example request:

Endpoint: myapp.com/v1/graphql

Headers: Authorization:

The Authorization header is parsed, validated, and verified. The underlying authentication service used could be any solution that issues JWT. Here’s some example code that verifies a JWT token:

try {
    if (token) {
        return jwt.verify(token, YOUR_SECRET_KEY);
    }
    return null;
} catch (err) {
    return null;
}

Here’s an example taken from a DIY GraphQL Server (Apollo) to parse the context:

Create a new instance of Apollo Server by passing in the type definitions and resolvers.

const server = new ApolloServer < MyContext > ({
    typeDefs,
    resolvers,
});

Create a standalone server that retrieves the token from headers and returns the user information as context.

const {
    url
} = await startStandaloneServer(server, {
    // Note: This example uses the `req` argument to access headers,
    // but the arguments received by `context` vary by integration.
    context: async ({
        req,
        res
    }) => {
        // Get the user token from the headers.
        const token = req.headers.authorization || '';
        // Try to retrieve a user with the token
        const user = await getUser(token);
        // Add the user to the context
        return {
            user
        };
    },
});

Passing context

Once the token is extracted, you need to pass the context object to every resolver that will get executed. Now all of your resolver code gets access to the context.

In the above example, we can see that the user data is passed to the context.

API-wide authorization

There are a few rules that might need to be enforced at the request level, even before passing it on to the GraphQL resolvers.

For example, you could block a user from performing any queries and return a 401, unauthorized error. Again, this involves code logic and the logic could become a lot of boilerplate if there are many rules.

Resolver-level authorization

As the context gets attached to each resolver, it is now possible to authorize the request inside the resolver. This method is only suitable for basic authorization logic when there are only few resolvers and few rules to check and authorize users.

For example: You have a users field that returns a list of user names. You will end up writing code, which looks something like this:

users: (parent, args, contextValue) => {
    // In this case, we'll pretend there is no data when
    // we're not logged in. Another option would be to
    // throw an error.
    if (!contextValue.user) return null;
    return ['bob', 'jake'];
};

The contextValue is now available for parsing and authorizing the user.

Note: This is resolver logic for one field. Imagine repeating logic code in every single resolver and field. It is very challenging to scale this. It is also very challenging to update any logic quickly.

GraphQL schema-based authorization

In a large schema with plenty of data based authorization, there are patterns of rules that are applicable for multiple queries. For example: Allow the user to fetch their own data and not any one else.

Here’s an example from GraphQL AuthZ library using schema as a data source.

// using schema as a data source inside pre-execution rule
const CanPublishPost = preExecRule()(async (context, fieldArgs) => {
    const graphQLResult = await graphql({
        schema: context.schema,
        source: `query post($postId: ID!) { post(id: $postId) { author { id } } }`,
        variableValues: {
            postId: fieldArgs.postId
        }
    })
    const post = graphQLResult.data?.post
    return post && post.author.id === context.user?.id
})

If you look at the return statement at the end in the above code snippet, that’s where the logic of checking user ID is written.

How does Hasura’s authorization layer work?

Hasura has a powerful authorization engine that allows developers to declaratively define fine-grained permissions and policies to restrict access to only particular elements of the data based on the session information in an API call.

Implementing proper data access control rules into the handwritten APIs is painstaking work. By some estimates, access control and authorization code can make up to 80% of the business logic in an API layer. Securing GraphQL is even harder because of the flexible nature of the query language. Hasura radically simplifies the effort needed to build authorization logic into APIs.

Declarative

With Hasura, you can transparently and declaratively define roles, and what each role is allowed to access in the metadata configuration. This can be done either through the Hasura Console or programmatically through the Hasura CLI. This declarative approach to authorization is simpler to create, maintain, evolve, and audit for developers and security teams.

Fine-grained access control

Hasura supports a role-based access control system. Access control rules can be applied to all the CRUD operations. You define permissions granularly on the schema, sessions, and data (table, row, and column).

For every role you create, Hasura automatically publishes a different GraphQL schema that represents the right queries, fields, and mutations that are available to that role. Every operation will use the request context to further apply permissions rules on the data.

Authorization rules are conditions that can span any property of the JSON data graph and its methods:

Any property of the data spanning relationships. Eg: Allow access if “document.collaborators.editors” contains “current_user.id3⁄4
Any property of the user accessing the data. Eg: Allow access if accounts.organization.id is equal to current_user.organization_io
Rules can mask, tokenize, or encrypt portions of the data model or the data returned by a method and are labeled. These labels are called "roles."

Easily integrate with your authentication system

Authentication is handled outside of Hasura, and you can bring in your own authentication server or integrate any authentication provider that supports JSON Web Token (JWT). If your authentication provider does not support JWT, or you want to handle authentication manually, you can use webhooks.

"By using Hasura, we cut the development time in half and built our product in three months. The built-in role-based authorization system made it easy to secure our data."

Mark Erdmann, Software Engineer, Pulley

Predicate pushdown

Hasura automatically pushes down the authorization check to the data query itself, which provides a significant performance boost and cost savings by avoiding additional lookups and unnecessary data egress, especially at larger scale.

Hasura automates predicate pushdown, since it is essentially a JIT compiler that can dynamically apply the filter in where the clause of a SQL query is based on the user running the query. Most GraphQL server frameworks require you to write plenty of boilerplate code to achieve the predicate pushdown authorization check.

Cross-source authorization

Hasura integrates authorization rules based on data and entitlements in different sources. Hasura forwards the resolved values as headers to your external services, which makes it easy to apply authorization rules in your external service. Again, this is made possible by Hasura’s declarative authorization system.

OWASP Top 10

OWASP is most famous for the “Top Ten” framework for structuring secure applications. As the industry expands into a microservice-driven approach, it’s important for organizations to validate all of their dependencies according to the OWASP framework.

Hasura’s security-first approach ensures that the Top 10 security features and criteria are fulfilled. Read more: How Hasura addresses the OWASP Top 10 concerns.

When you are building your own GraphQL server and writing authorization logic, you will need to ensure that the Top 10 concerns are handled to be secure and compliant.

Try out Hasura

Sign up to start your journey with Hasura Cloud today. If you are an enterprise looking to learn more about how Hasura fits in your app modernization strategy, reach out to us through the Contact Us form and our team will get back to you.

Boosting database interactivity and developer productivity with Hasura Native Queries and Logical Models

Hasura — Thu, 29 Jun 2023 16:28:28 +0000

API development has seen a significant surge in demand as organizations across the globe strive to harness the power of data. The Hasura GraphQL engine automates API creation from existing database objects like tables, views, functions, and stored procedures.

For developers, this automation leads to significant time savings, freeing them from the hassles of manually developing APIs from scratch. It also brings in the benefits of GraphQL, such as type safety, real-time subscriptions, and performance enhancements.

Why Use Native Queries and Logical Models

The Hasura GraphQL Engine has gained popularity for its ability to automatically generate a GraphQL API around database objects, providing seamless querying, mutating, and subscribing to data changes. However, there are instances when you require more custom or advanced functionality that goes beyond the out-of-the-box capabilities. For example:

Use the full power of SQL that Hasura might not provide access to through the typical table API, such as GROUP BY, window functions, or scalar functions.
Provide custom arguments to the users of your API to greatly expand its flexibility.
Encapsulate sophisticated filtering with a query, allowing your users to provide a single argument rather than having to understand how to manipulate the data.
Work with the advanced features of your database to improve performance.
Write a compatibility layer around tables, making it easier to change your API without breaking existing clients.
Reduce duplication by moving common data manipulation into one place.

Hasura's Native Queries feature provides a powerful tool for enhancing your GraphQL API with the flexibility and control of raw SQL queries. By leveraging Native Queries, you can create custom and advanced behavior in your Hasura-generated GraphQL schema without the need for additional database objects or DDL privileges. This enables you to unlock the full potential of SQL while building robust and efficient applications with Hasura.

How Native Queries work

The Hasura GraphQL Engine integrates seamlessly with Native Queries. Here's an overview of how it works:

Defining Native Queries

To create a Native Query, you write raw SQL statements directly within Hasura. This query like the one shown below can use arguments using the syntax {{argument_name}}. These queries can be as simple or complex as needed, incorporating SQL features such as joins, aggregations, and custom business logic.

Binding to GraphQL Schema using Logical Models

After defining a Native Query, you can bind it to your Hasura-generated GraphQL schema. This automatically exposes the Native Query as a GraphQL API endpoint, making it accessible to your application.

Executing Native Queries: When a request is made to the Native Query endpoint via GraphQL, the Hasura GraphQL Engine translates the GraphQL query into an SQL query and executes it against the database.

The results are then transformed into GraphQL response format and returned to the client.

<root field name>(
[args: {"<argument name>": <argument value>, ...},]
[where: ...,]
[order_by: ..., distinct_on: ...,]
[limit: ..., offset: ...]
) {
<field 1>
<field 2>
...
}
}```

Developers can leverage Native Queries using the Hasura GraphQL API and all the features that come with it – this includes pagination, filtering, sorting, aggregations, caching, etc.

Benefits

This is where Native Queries come into play, empowering you to leverage the full power of SQL within Hasura while maintaining flexibility and control over your GraphQL schema. Native Queries enable you to automatically generate a GraphQL API around raw SQL queries, offering a range of benefits:

Flexibility: By utilizing Native Queries, you can incorporate custom SQL logic into your GraphQL schema, allowing for complex database operations and tailored responses. This flexibility enables you to meet specific requirements that may not be achievable through standard GraphQL queries.
Control: Native Queries give you greater control over your Hasura-generated GraphQL schema. Instead of relying solely on the automatic generation of GraphQL API from the database schema, you can shape the GraphQL schema around your SQL queries, providing a more fine-tuned interface for interacting with your data.
Avoiding DDL Privileges: With Native Queries, you no longer need to create additional database objects that require Data Definition Language (DDL) privileges. This simplifies the development process by reducing the dependencies and permissions required for deploying and maintaining your application.
Abstraction: By defining the data model in Hasura, it allows you to define a data schema that best fits the needs of your GraphQL API, decoupling it from the underlying database schema. This enables independent evolution of the API and the database structure.
Cross-Database Compatibility: Modeling data in the middleware layer can facilitate cross-database compatibility. The middleware can act as an abstraction layer, translating GraphQL queries into the appropriate database-specific queries for different target databases. This allows for greater flexibility in choosing and changing the underlying database technology.

In the realm of GraphQL development, choosing the right approach for modeling data is crucial for building efficient, scalable, and flexible applications. Traditionally, developers have relied on creating database artifacts like views, user-defined functions (UDFs), and stored procedures.

However, solutions like Hasura represent a paradigm shift in the way data is leveraged by app developers and other data consumers across organizations. Native Queries and Logical Models are a game-changer, they empower developers to unlock unprecedented agility and flexibility in their GraphQL solutions.

Limitations

Native Queries currently supports read-only query capabilities for Postgres, SQL Server, and BigQuery. Support for more databases and mutations will be coming soon.

Conclusion

Modeling data in Hasura represents a paradigm shift in GraphQL development, enabling developers to embrace agility, flexibility, and efficiency. By leveraging Hasura's automatic API generation, flexibility in schema definition, rapid development capabilities, access control mechanisms, performance optimizations, and integration capabilities, developers can build scalable and resilient applications without being bound by the limitations of traditional database-centric approaches.

As the GraphQL ecosystem continues to evolve, embracing tools like Hasura will undoubtedly become the preferred choice for developers seeking to unlock the full potential of GraphQL.

📚 Documentation and Resources

To help you get started, we've prepared detailed documentation, guides, and examples:

🚀 Get Started Today!

We can't wait to see the amazing applications you'll build using Hasura and Native Queries. Get started today by signing up for Hasura Cloud and connecting to one of the supported databases.

If you have any questions or need assistance, feel free to reach out to our team on Discord or GitHub.

Happy building! 🎉

“We got documentation feedback on…transferring wealth?”

Hasura — Thu, 29 Jun 2023 14:28:20 +0000

A few weeks ago, I was plugging away at something significant (read: procrastinating by looking for ways of automating some repetitive task in an inane way) when I heard the familiar "pop-pop-pop" notification from Slack. Like many of you, Slack sits permanently open on my machine, every ping like a mosquito needing to be swatted away. However, this one got my attention.

In our #docs-feedback channel, which we connected to the feedback component at the bottom of every docs page, a new message came through:

New feedback on this page: Transfers | Wealth Docs

I'm relatively neurotic, so the spike in cortisol from a near panic attack was significant, as my first thought was that we'd been hacked. How? Rationality is absent in fight-or-flight mode. 🤷‍♂️

It quickly became clear this wasn't a malicious attempt at hijacking our component or site – rather, this was someone so happy with our docs that they decided to copy them verbatim, leading to their docs feedback arriving on our Slack channel. We were flattered, and it quickly became an internal joke.

Then, it happened again. With another company.

Flattered as we were, we couldn't have other sites' feedback clogging our channel or ~~dragging our average down~~ inflating our KPIs. To mitigate this, we introduced a tiny bit of logic that prevents the input from going through, if not from our domain. Could you bypass this and still hit our API? Of course, but please don't. You surely have better things to do.

Our philosophy on docs

As Hasura’s core is open source, so are our docs. We regularly receive contributions from you all (thanks for catching our spelling mistakes 😘). We are constantly improving our documentation to better serve you in building your projects and products.

As we're sure you know from regularly visiting the docs wiki as part of your morning ritual while drinking coffee, you're no doubt familiar with the core of the docs team's philosophy around documentation:

We hold a strict standard because we want to ensure our users can quickly find what they need, understand it because it's well-written, and get back to building with Hasura. 🚀

We see the popularity of our docs site as a template as an extension of that philosophy! And we want you to keep using the docs site as your starting point for your next project's documentation.

Get your own Hasura docs!

We have an easy one-liner below if you want to get started with our docs site. This command will clone the hasura/graphql-engine repo and then remove every folder except the most important. 😉

git clone --depth 1 --filter=blob:none --sparse https://github.com/hasura/graphql-engine.git && cd graphql-engine && git sparse-checkout init && git sparse-checkout set docs && find . -type f ! -path "./docs/*" -delete

Then, git init and have some fun building on top of our docs!

BONUS : Be on the lookout for our _ new _ v3-docs site…coming soon! And, yes: It will still be open sourced for your nitpicking pleasure. 🥳

❤️ Hasura Docs Team