Forem: Fauna, Inc.

Simplify serverless scaling and data management with Fauna and Cloudflare Workers

Kirk Kirkconnell — Wed, 13 Nov 2024 19:07:47 +0000

In a world where we demand speed, flexibility, and seamless scalability, serverless architecture has emerged as a new gold standard. For many developers, achieving a truly serverless experience—where every element of scaling, replication, and data consistency is managed without additional infrastructure headaches—can still seem elusive. Fauna, in partnership with Cloudflare, has made True Serverless possible. In this post, I cover how Fauna and Cloudflare Workers empower you to build, deploy, and scale globally without compromising on true serverless simplicity, easy global scaling, and the ability to evolve applications without downtime.

But don't just take my word for it; ask Cameron Bell, Head of Software, Systems, & Data at Connexin shared, "Fauna is a true serverless database. This is very important for applications like ours where much of the infrastructure – like Cloudflare Workers – is connectionless and requires solutions that can meet the demands of real-time, operational data. The serverless attributes of Cloudflare and Fauna mean we can genuinely do more, faster."

True serverless: No provisioning, replication, or clustering

One of the core tenets of a serverless architecture is zero operational overhead. Fauna and Cloudflare’s integration lets you forget about provisioning servers, configuring clusters, or handling complex replication. Fauna’s document-relational model natively supports data consistency across multiple regions, and Cloudflare’s global edge network seamlessly distributes workloads. This combination allows your application to scale automatically based on usage, giving you all the benefits of global performance without needing to manage infrastructure. In other words, the architecture scales to your application’s needs—without requiring you to manage the backend setup.

Ultra-fast multi-region and edge workloads with strong consistency

When it comes to applications that serve a global audience, speed and data consistency are essential. Traditional databases struggle to provide both, often sacrificing one for the other. Fauna’s distributed transaction engine provides data consistency across all regions in a region group without the lag of traditional replication methods. In addition, your data is always strongly consistent across those regions. Paired with Cloudflare Workers and their edge network, your application can support ultra-low-latency requests around the globe, and your app always connects to the closest Fauna replica. This setup not only ensures fast responses but also guarantees consistency, letting users access the most up-to-date data with low latency, no matter where they’re located.

Rapid time-to-market with simplified deployments

With Fauna’s native integration with Cloudflare Workers, app development is accelerated by eliminating development and infrastructure management tasks. Fauna’s database enables instant global access through its API, which communicates over HTTPS, so your app can connect securely using Fauna’s client libraries or make direct HTTPS requests. This native integration means Cloudflare Workers can interact directly with Fauna without needing traditional database connection pools, ORMs, middleware, or complex backend setups. This makes data handling within your Cloudflare Workers faster, more efficient, and easier to manage over time.

By reducing development steps from start to deployment, Fauna and Cloudflare enable you to skip backend infrastructure management, allowing you to focus entirely on building features. The streamlined connection keeps your code lean and responsive, helping you to bring products to market faster, while delivering a scalable, high-performance experience to your users.

Keep applications lightweight, performant, and easy to maintain

Complex app logic often makes apps slower to execute, tricky to test, and difficult to manage over time. Fauna’s powerful query language, FQL, enables you to build complex data relationships without writing extensive application code. With Fauna’s UDFs (user-defined functions), you can embed FQL code to run in the database. This streamlined data management means that, within Cloudflare Workers, your logic remains lean and focused. Fauna handles the heavy lifting of relational data and consistency, enabling your application to remain lightweight and performant—even as it scales to meet global demand. In addition, testing new application code is easier and more repeatable as the heavy lifting can be done in UDFs.

Fauna also has native zero-downtime migrations for database schema changes, which allows you to evolve your data model without taking your application offline or disrupting the user experience. As your application grows and requires changes to its data structure, you can roll out updates smoothly, maintaining high availability and minimizing the need for extensive backend reconfiguration. This enables you and your team to iterate quickly, delivering new features and products to market faster but in a controlled fashion.

Conclusion

By combining Fauna’s flexible, distributed, and strongly consistent database with Cloudflare Worker’s edge-first architecture, you unlock a fully managed serverless stack that lets you focus on building your application, not on managing infrastructure. This partnership enables you to scale without limits, ensures ultra-fast, globally consistent data, and keeps your codebase lean—all key to getting your products to market faster and your customers happier.

If you’re ready to embrace true serverless architecture, check out this workshop to see just how easy it is to get started.

DynamoDB pain points: How to address them and exploring possible alternatives

Shadid Haque — Mon, 30 Jan 2023 14:54:50 +0000

DynamoDB is a robust NoSQL database that has become increasingly popular due to its performance, scalability, relative simplicity, and low operational overhead. However, it has its drawbacks. Some of the main disadvantages of DynamoDB include limited querying options, limited storage capacity (400kb per document), no multi-region ACID support (eventual consistency), and difficulties in replicating data in multi-region setups.

In this article, we will explore the main pain points of DynamoDB, how to overcome them, and explore some of the DynamoDB alternatives.

Indeed, there are areas where DynamoDB is the most compelling solution when it comes to speed. DynamoDB is a low-latency, fully managed NoSQL database that is ideal for applications where speed is more important than strong consistency.

DynamoDB automatically scales up and down (when you use provisioned capacity) based on the application traffic pattern. It supports automated sharding, so you do not need to manage shards manually. If your application requires high scalability and speed and doesn't need complex data relationships or transactions, DynamoDB is the way to go. However, Dynamo does have some drawbacks, which I will explore below.

See a side-by-side comparison of serverless databases.

Limited querying capabilities

If you have worked with DynamoDB to some extent, then you know that relational data modeling is tedious with DynamoDB. There are no foreign key joins in DynamoDB like most people are used to in relational databases. You can, however, handle complex data access patterns similar to relational databases. In DynamoDB, you can model relationships such as one-to-many, many-to-many, and one-to-one. You can use a variety of strategies, such as denormalization, duplication, and composition of primary keys. However, these solutions require a fair amount of engineering effort and introduce a number of other limitations to consider.

Follow Alex DeBrie’s article to learn more about data access patterns in DynamoDB.

DynamoDB can only perform query operations on tables with a single primary key. Consider the following table for example. The primary key of this table is the user ID, a unique identifier for each user. The table might have other attributes like name, email, and address, as demonstrated below.

You cannot query the table on other attributes like name and email. You need to set up the global secondary indexes (GSI) with the attributes you want to query and then use the query operation on the GSI. Those of us using DynamoDB for a while are okay with this pattern of storing data. Nevertheless, this is a limitation.

Moreover, you have to consider the cost of using GSIs. GSIs consume additional read and write capacity units (RCUs and WCUs). Learn more about GSI costs here.

Let's say you have a DynamoDB table called “Products” with a primary key of product_id and a few other attributes like "name" and "price". The table is provisioned with 1,000 RCUs and 1,000 WCUs. If you want to create a GSI for querying on the "name" attribute, it will require additional RCUs and WCUs to perform queries on the "name" attribute. So now you need to provision 1,000 RCUs and 1,000 WCUs for the primary key and an additional 1,000 RCUs and 1,000 WCUs for the GSI, doubling your DynamoDB usage cost. As your application grows, GSI optimization is an additional overhead you must be mindful of. In a relational database, you don't have to worry about these.

Comparing a SQL database to a NoSQL database is like comparing apples to oranges. NoSQL databases such as DynamoDB are excellent for horizontal scalability, high availability, and flexibility for handling unstructured and semi-structured data. However, features such as foreign key constraints and ACID (Atomicity, Consistency, Isolation, Durability) transactions, which we take for granted in a relational database, are not present by default in DynamoDB. Manually configuring these capabilities takes a lot of engineering effort and specialized knowledge. In brief, you have to deal with increasing application complexity because of GSI.

If you are exploring Amazon DynamoDB alternatives that are NoSQL and have better-querying capabilities, there are a few to choose from, but the one I will be covering here is Fauna. Fauna combines the querying capabilities of relational databases (SQL databases) with the flexibility of NoSQL databases.

A database management solution like Fauna bridges the gap between a traditional relational database and a flexible NoSQL database by allowing data to store in JSON-like documents while also giving the ability to create SQL-like data relationships among collections.

Learn more about Fauna’s document-relational model.

Like DynamoDB, Fauna is also a fully managed NoSQL database with low latency. It also auto-scales, ensures high availability, and adapts to your application traffic. DynamoDB has been around for 10+ years now, whereas Fauna is relatively new.

Challenges of multi-region data replication

While DynamoDB global tables provide a way to replicate data across multiple regions, it can be a complex and costly process that requires additional setup, management, and code — and it may only support some features or data replication across all indexes.

You have to create the same table in multiple regions with the same schema, indexes, and throughput settings. Additionally, data replication between different regions needs to be configured and managed. There is no zero-configuration solution to replicate data across multiple regions with DynamoDB.

Furthermore, certain features are not available in global tables. DynamoDB Time to Live (TTL), which is the ability to expire items after a specified period of time, is not available with global tables.

Another unsupported feature is DynamoDB Streams. Additionally, local secondary indexes (LSIs) and global secondary indexes (GSIs) are not replicated across all replicas in global tables. You must manually create those LSIs and GSIs in different regions; otherwise, you cannot query data efficiently.

DynamoDB global tables do not provide fault-tolerant support by default

In the case of an outage or a failure in one region, DynamoDB does not automatically switch to another region to ensure data availability.

If distributed data across many regions is one of the core requirements of your application, you are better off using one of the alternatives of DynamoDB, such as Amazon Aurora, Fauna, or Apache Cassandra.

DynamoDB uses an "eventual consistency" model for global data, meaning that updates may take some time to propagate to all replicas fully.

Applications that require immediate consistency across all regions may need to use a different database solution or implement additional logic to handle eventual consistency in DynamoDB. Since data may not be immediately available in all regions, read and write operations may be slower in certain regions. This can affect the performance and user experience of the application.

Fauna is one of the strong competitors to Amazon DynamoDB for its multi-region data replication. Fauna provides multi-region data with strong consistency. It automatically routes your requests to the nearest replica, ensuring that your application experiences low latency. Fauna automatically replicates your data across all regions, ensuring that your data is always up-to-date and available — even in case of a regional outage or failure. Unlike DynamoDB, Fauna requires zero configuration for global replication. Strong data persistence in all regions is always guaranteed. If you compare Fauna to DynamoDB, you realize that Fauna offers all the features of DynamoDB without the drawbacks.

Apache Cassandra is also another alternative to Amazon DynamoDB. It is a free, distributed wide-column, NoSQL, open source database that can handle a large amount of data across multiple regions. However, you must manage your infrastructure and clusters if you decide to use Apache Cassandra. Managing your clusters can be challenging, even though Cassandra has robust documentation and an active community behind it.

Another database you may be interested in as an alternative is MongoDB Atlas, the managed service for MongoDB.

Learn how Fauna stacks up against MongoDB.

Storage limitations

Another pain point with DynamoDB is the item size limit. An individual item has a maximum of 400kb in size. The item size limit impacts the way data is modeled. The application may need to denormalize the data or split it across multiple items to work within the item size limit. The storage limitation is not a big deal for most applications out there, but if you plan to grow and scale your application to millions of users, you should start thinking about the additional complexity you need to handle.

DynamoDB competitors such as MongoDB, Fauna, or Cassandra don't have this issue.

Building scalable applications on DynamoDB

DynamoDB is well known for its low latency and scalability. However, as you build and scale your application with DynamoDB, you must be aware of some common pitfalls. As discussed in the previous section, an individual item in DynamoDB can be at most 400kb. While building with DynamoDB, it is crucial to understand the data model that best fits your application's use case and design the table schema accordingly. Failing to do this at the start will cost you as you scale your application.

Although possible, evolving your data with DynamoDB as your application grows is challenging. You need to choose the correct partition key. Selecting the right partition key ensures that the data is evenly distributed and read and write requests are distributed evenly across all partitions.

Limited ability to perform advanced analytics

DynamoDB is optimized for storing and retrieving large amounts of data quickly, but it is ill-suited for performing advanced analytics on that data. It doesn't support advanced features such as aggregate functions or window functions.

SUM, COUNT, AVG, and MAX are commonly used in SQL to perform calculations on groups of data. DynamoDB alternatives MongoDB and Fauna have equivalent features to perform these analytics tasks. In MongoDB, you get aggregate functions such as $group, $match, $project, etc. Fauna also provides aggregate functions through Fauna Query Language (FQL) for advanced analytics.

Window functions are another advanced SQL feature that DynamoDB does not support. These functions allow users to perform calculations over a set of rows related to the current row in a query. Fauna also does not have built-in window functions, but it is possible to perform calculations similar to window functions using the Map-Reduce function and a combination of other functions. MongoDB does not have built-in window functions either. However, it does have the $bucket operator, which can be used to perform similar calculations.

If you insist on using DynamoDB and need robust analytics, then your best bet is to use third-party analytics tools such as Apache Hive or Apache Pig to perform complex queries and advanced analytics on the data stored in DynamoDB. You have to consider the additional cost of these extra resources, however.

Conclusion

To conclude, DynamoDB is well-suited for use cases that require high scalability, high performance, and low latency. It seamlessly integrates with other AWS services and supports document and key-value data models. However, it also comes with its limitations. Limited query capabilities, little support for the transaction, eventual consistency, and limited support for analytics are some of the main pain points with DynamoDB. With enough engineering resources and time, you can overcome these pain points.

Everything in software engineering is about tradeoffs. Every database management solution has its pros and cons. It is up to you and the developers to choose your company's best solutions. This article gives you a clear picture of the various limitations of DynamoDB and some of the alternatives to DynamoDB so you can make the best-informed decision while choosing a database for your next project.

Though DynamoDB has much to offer, Fauna is a much newer offering that comes with a host of powerful and unique features that enhance the serverless experience for organizations of all sizes. Read more at “Modernization of the database: DynamoDB to Fauna.”

If you're experiencing any of the DynamoDB pain points mentioned above and are looking for a more robust and efficient solution, consider reaching out to either me or the team of experts at Fauna.

Modernizing from PostgreSQL to Serverless with Fauna Part 1 (2023 Update)

Luis Eduardo Colon — Mon, 09 Jan 2023 16:49:06 +0000

With modern development practices and efficiencies in mind, you can transition from a traditional relational application to a serverless document-relational database with maximum scale and no data consistency compromises.

Introduction

Many web and mobile applications continue to be implemented atop traditional relational databases. Although many core components of these databases are five decades old, these designs are the most familiar to most database professionals. However, recently the choices have shifted toward more distributed, cloud-native options to support hyper-scale expectations.

One of the popular relational database options is PostgreSQL (A.K.A. Postgres), which dates back to UC Berkeley’s POSTGRES project in 1986. It has supported SQL since 1994 and has been ACID compliant since 2001. Like other popular relational databases, Postgres is a client-server database at its core and continues to be enhanced by an active community, with Version 15 released in late 2022.

Over its long history, Postgres has added a large set of extensions and options, including almost a dozen different data indexing options and many server replication alternatives. Multiple vendors have added data distribution options atop the core server code. In contrast, other vendors offer so-called “wire-level” compatibility atop redesigned engines that look more like less traditional key-value stores. However, most of the newest implementations still necessarily expose significant operational complexity in areas like partitioning complexity.

Some of the largest websites, driven by the requirements of serving billions of users, have led to the maturation of new database architecture alternatives, which motivate modernizing applications to leverage cloud-native, serverless, highly-distributed database options. Fauna strengthens relational data consistency among these newer alternatives while leveraging a cloud-native, serverless, flexible document model that abstracts most operational complexities from application developers.

You may be an application developer looking to build an application designed to scale well from the beginning. Or, you are a database professional looking to transition a Postgres application to minimize the amount of scaling impediments you may have encountered periodically as you continue building features. In either case, this article series is for you. This first of three articles is an updated version of an existing tutorial built by Brecht De Rooms some time ago. It explores the process of transitioning an existing Postgres application to Fauna while ensuring that you build your database in a scalable way from the beginning.

In this first part, we’ll provide a deep comparison between Fauna and Postgres while we explore how to implement and query a one-to-many relationship in a basic database. In part two, we extend the concepts to cover a many-to-many relationship and how to build a domain-specific language that avoids object-relational impedance mismatches. We explore advanced transactions with referential integrity in part three and suggest various modeling, optimization, and migration strategies.

Comparing Postgres and Fauna

Given that both Postgres and Fauna support relational database features, they share many important characteristics. Both support data relationships, serializable transactions, normalization, foreign keys, indexes, constraints, stored procedures, and many other typical relational database features. Both support ACID transactions and strict isolation levels, with Fauna default isolation levels can be considered stricter since it provides those guarantees even when data is distributed. Both are primarily aimed and are well suited for online transaction processing (OLTP) applications. Fauna focuses more on this use case because it is designed as a cloud-native database optimized for sub-second queries. At the same time, many Postgres implementations and extensions can be suited for more analytical use cases.

Notable Differences

Although both databases can implement most database requirements, their implementation varies significantly in some areas:

Relational-model vs multi-model: Postgres is often described as an object-relational database with additional data types that support documents like JSON and JSONB, but it is primarily built and used as a traditional relational database. Fauna is often described as a document-relational database with JSON documents at its core. In contrast to other common document databases, it offers relational options and flexible indexing to maintain consistency. Further, it adds temporality and graph-like features such as graph traversal.
Schemas: Both Postgres and Fauna have a concept of logical databases that include schema records describing the collections, indexes, and security properties of the included records. However, Postgres enforces schemas while Fauna is considered schema-less in that it supports adding additional fields or attributes, while still providing mechanisms to enforce integrity and uniqueness.
Database connections: Postgres expects persistent connections, like many similar relational databases. Most implementations must configure connection pools and have to be concerned with connection overhead and limits. Fauna uses stateless, secure HTTP connections and requires no connection management.
Database as an API: Postgres is built to self-host or to be hosted or managed by cloud providers, requiring managing server instances, read replicas, partitioning, and the like. There are vendors that claim that they provide serverless Postgres, but in many cases, there are still provisioning decisions to consider. On the other hand, Fauna is delivered as an API with a single global endpoint, and there are no clusters, partitions, and replication to configure. Furthermore, the single global endpoint can intelligently route a request to the closest copy of the data.
Distribution: When it comes to distribution in a traditional database, asynchronous replication is the most popular form of distribution which introduces eventual consistency and potential data loss. If consistency is required, synchronous replication is provided as an option, but it typically comes at a high price in terms of performance, especially if distribution across regions is desired. Although traditional databases like Postgres were not built with distribution in mind, many recent improvements and additions provide multiple options that supplement the non-distributed nature of its core engine. In contrast, Fauna is built from the ground up as a scalable, multi-region distributed database. It is inspired by the Calvin algorithm that speeds up data consensus by relying on deterministic calculations. Further, this distribution of data is transparent to the application developers and can accommodate data locality restrictions.
Multi-tenancy: Although Postgres provides various options to create a SaaS application where you can have multiple tenants (whether across servers or within a single database), Fauna has a unique concept of child databases with arbitrarily deep nesting. Child databases are completely isolated from each other, with the ability to create separate permissions for child databases, and the child databases cannot determine whether there’s a parent database.
Query Languages: The primary language used for Postgres is SQL or, more accurately, a SQL dialect. Although SQL has been around for many decades and is well known, documented, and standardized to the extent that it can be, every SQL database exhibits significant variations in the language. For example, to support JSON manipulation and event streaming, you will encounter significant differences between the SQL commands used in Postgres and other SQL databases like MySQL. Application developers will likely use an object-relational mapper (ORM) that adds some application complexity without removing the need to understand the Postgres-specific SQL conventions. GraphQL can be used with Postgres by adding extensions, of which there are many options to consider. Fauna provides its own native query language, FQL, which is designed to align with modern coding paradigms. Because of the nature of FQL and other design considerations, Fauna requires no ORM and is not susceptible to injection. The choice for a custom language is rooted in Fauna's scalable and distributed design. It’s designed to prevent long-running transactions, maximize performance in distributed scenarios, be highly composable to reduce round-trips, and have a transparent and predictable query plan. Many of these characteristics will become evident in this article and its subsequent parts as we cover FQL extensively in the coming sections. Beyond FQL, Fauna also supports GraphQL out of the box.

Postgres and Fauna: Terminology Mapping

The following table summarizes how Fauna concepts relate to Postgres counterparts and, in most cases, also apply to similar terms used with other relational databases. This table comes from this blog, and the Fauna documentation provides more details on these concepts and many other common SQL commands and concepts here.

POSTGRES	FAUNA	DETAILS
Row or record	Document	In Postgres, a record (or row) represents a distinct database entry, and it must conform to the containing table’s column definitions. In Fauna, a document represents a nested structure of fields and their values, with no specified structure or types. Each document, even in the same collection, can have its own independent structure, making the documents schemaless. Also, each document is versioned, storing the history of a document’s mutations from creation to deletion.
Table	Collection	Tables store records, collections store documents. In Postgres, a table’s column definition specifies the structure for all records in the table. The column definition specifies the names and types of values that can be stored in a column. In Fauna, collections are a container for documents, imposing no specific structure on those documents.
Primary	Region	Fauna has no primary or secondary concept, all regions can serve reads and writes.
Secondary, standby, replica	Region	Fauna has no primary or secondary concept, all regions can serve reads and writes.
Replication	Replication	Fauna’s replication is semi-synchronous and does not require any operator management.
Sharding, partitioning	Not Applicable	Fauna does not require the operator to manage sharding or partitioning in any way.
Primary Key	Reference (Ref)	The unique identifier of a document.
Foreign Key	Reference (Ref)	A pointer from one document to another.
Index, materialized view	Index	Fauna merges the concepts of indexes and views. Indexes must be explicitly referenced.
Transaction	Transaction	Both Postgres and Fauna support ACID transactions.
Schema, database	Database	Both Postgres and Fauna have a concept of logical databases that include schema records describing the collections, indexes, and security properties of the included records. In Postgres and other relational databases, a schema refers to the set of table definitions and constraints defined in that database. The schema is enforced so that no row violates its table definition or constraints. In Fauna, database definitions also include child databases with arbitrarily deep nesting. A Fauna database contains schemaless documents with no schema enforcement available at the document level; however, validation functions may be used when necessary. The Fauna GraphQL API does apply schema enforcement to comply with GraphQL schemas.
Stored procedures, user-defined functions	User Defined Functions (UDF) or Functions	Fauna supports user-defined functions written in FQL.

Tables and rows are collections and documents in Fauna just like in other document databases. An index in Fauna combines an index and a view as we know them in a traditional database. We’ll directly query indexes for data since indexes contain data, similar to consistent ordered views. User Defined Functions (UDFs) are similar to Stored Procedures except that, in contrast to Postgres, both queries and UDFs are written in the same language in Fauna, while in Postgres, you would split to the PL/pgSQL language for a stored procedure. Fauna users typically use UDFs much more frequently due to the easy transition from a query to a UDF.

From Postgres to Fauna: Building the Basics

In this section, we’ll learn how to create a database, a collection, insert some documents, and query them. We will focus on modeling a one-to-many relation in this article; we will model a many-to-many relation in the next article and cover other modeling options in the third and last installment. We’ll use a well-known Postgres tutorial that provides a database model of a DVD rental business. The model of the DVD rental application looks as follows:

Fig. 1.Entity relationship diagram for DVD rental business. Source: PostgreSQL Tutorial

The database schema has “film" at its center and allows customers to rent DVDs from different stores around the country, supports shopping baskets, as well as film, payment, store, and customer management. Thanks to the normalized model, it supports many access patterns: customers can find films by category, rating, release year, title, description, or actor. The staff can retrieve where a film currently resides or retrieve a list of overdue rentals.

Creating a new database

In this section, we’ll create the database. Even if you don’t follow along, this 2-click process (plus typing a name) shows how different the database-creation process is compared to setting up a traditional database.

Sign Up

Fauna is a cloud database. There is no setup of hardware or configuration necessary. All you need to do is go to dashboard.fauna.com and sign up for an account.

Create the database

Click NEW DATABASE to create a new database which will instantly be created.

Fill in a name for your database and click CREATE.

Now that you have created a database, you can follow along by pasting the provided code in the dashboard shell (or play around with the UI instead to create collections/indexes etc). Of course, you can also send queries to Fauna using one of the drivers or the terminal shell. The shell is built around the JavaScript driver behind the scenes. When writing FQL, we are not writing strings but rather functions, although that might not be obvious as we use the dashboard shell.

One-to-Many relations

Let’s start with a small and easy subset of the model and extend it gradually. The film and language relation is a many-to-one relation that should be very easy to model. Since the model remains quite simple, we’ll add a few things now and then in green to render it more interesting. Instead of one language, we’ll add both a spoken and a subtitle language.

Fig. 2.Adding a one-to-many relationship for both spoken and subtitle language.

Creating the language collection

In Postgres, we have tables that contain rows; in Fauna, we have collections that contain documents. To create a new film collection, we could use the dashboard interface but just like in SQL, manipulation and creation of collections, indexes, security roles, or even databases can be done entirely in the query language. To create the film collection, we can also paste the following code snippet in the dashboard shell:

CreateCollection({name: “language”})

Creating a language document

We can create the equivalent document for the language as follows:

Create(
   Collection("language"), 
   {
       data: {
          name: "English"
       }
   }
 )

The Create() function is used to create documents, similar to an Insert statement in Postgres. It takes the collection where you want to store the document as the first parameter and a JSON object as the second. Application data is always nested under the data key, which separates it from special Fauna fields. Once we have executed the Create() statement, we can take a look at the document by going to the Collections tab in the dashboard.

Fauna has automatically generated two fields: ref and ts, which are the Fauna equivalent of id and last_update.

ref: a unique reference to the document. The presence of references doesn’t mean that we can no longer use other IDs in Fauna. We’ll see the difference between native Fauna references and IDs shortly when we start querying.
ts: the document's timestamp, which is automatically updated when the document is adapted. Fauna’s timestamp is part of temporality features that provide time travel and support the streaming capabilities.

Note: Your reference IDs will be different. If you are following along, you can grab the reference ID of any document from the dashboard's Collections view.

Creating the film documents

To store the film documents, we need a new collection:

CreateCollection({name: "film"})

Let’s create a couple of simplified film documents. We actually have many potential choices for the relation between films and languages (at least five) since Fauna is both a document and relational database. However, anything that is not normalized is typically an optimization so let’s start with the normalized model.

We will make it more interesting and add both spoken language and subtitles language. We will refer to the previously created language by storing the native reference in the document as follows (again, your reference IDs will be different):

Create(
    Collection("film"), 
    {
       data: {
        title: "Academy Dinosaur",
        language: {
            spoken: Ref(Collection("language"), "288878259769180673"),
            subtitles: Ref(Collection("language"), "288878259769180673")
        }
      }
    }
  )

A second document might be:

Create(
    Collection("film"), 
    {
       data: {
        title: “Back to the Future",
        language: {
            spoken: Ref(Collection("language"), "288878259769180673"),
            subtitles: Ref(Collection("language"), "288878259769180673")
        }
      }
    }
  )

An alternative approach is to add the languages as embedded objects. If we do not intend to efficiently query languages directly, we could add it as an embedded object.

Create(
    Collection("film"),
    {
        data: {
            title: "Academy Dinosaur",
            language: {
                spoken: { name: "English" },
                subtitles: { name: "English" }
            }
        }
    }
)

Embedding vs normalized data

Fauna is a document database that might make us think: “don’t join data, duplicate it” as a workaround due to the limitations of popular document databases. Since Fauna is relational, such workarounds are not necessary but rather another option in your tool belt. This is an important consideration when working with Fauna versus other document databases: denormalization is a choice, not a workaround.

Postgres is not natively a document database but has, among its options, a JSONB type that can be used to emulate documents. However, such columns do not benefit from optimizations such as column statistics, are restricted to primitive operations, lack built-in functions to modify values, and may be awkward to query in SQL. The absence of column statistics blinds the query planner, resulting in potential production issues. In light of these limitations, you might resort to storing frequently queried attributes in regular columns and the remainder in JSONB columns.

In contrast, Fauna's querying and indexing capabilities do not change based on whether documents contain nested data or normalized. That doesn’t mean that denormalization is the recommended practice, but it does become more attractive as a technique to optimize read performance. We'll dive into some advanced optimization techniques at the end of this series and show how FQL can help you hit the sweet spot between flexibility, optimization, and data correction.

In some databases, indexing flexibility might suffer when nesting objects. However, Fauna’s indexing is built to be flexible regardless of whether the data is stored as nested objects or normalized. Further, Fauna indexes work equally well on nested values or arrays. For example, we could index the spoken language by adding the following path data > language > spoken > name to the index. If spoken languages had been an array instead:

{
        data: {
            title: "Academy Dinosaur",
            language: {
                spoken: [{ name: "English" }, { name: "French" }],
                subtitles: { name: "English" }
            }
        }
    }

We can add the same path to the index, and Fauna will recognize that it’s an array and unroll it. By using advanced features like index bindings, we can even combine both the spoken and subtitles language in one index to find any film in a specific language, regardless of whether it’s the spoken or subtitle language.

Before we talk about the trade-offs and even more alternative strategies, let’s see how we would retrieve films and then perform a simple join by including languages.

Querying a collection

The simplest Postgres query is easy since SQL is a declarative language where you describe what you need instead of how you want it to be calculated.

SELECT id FROM film

In contrast, FQL is a procedural language, like PL/pgSQL. Although it’s initially verbose, we’ll discover that we can extend the language to tame that verbosity.

This article focuses on replicating a traditional relational model in Fauna, providing an opportunity to introduce you to FQL. If you want to dive deeper into the fundamentals of FQL, here is an excellent guide that starts from scratch, and here's a guide to translating basic SQL queries to FQL.

Retrieving all documents starts by executing the Documents() function on a collection. The Collection() function returns the reference to the collection, and Documents() retrieves the film references from that collection.

Documents(Collection("film"))

When executing this first query snippet, you might be surprised that it simply returns something like “ok, here is a reference to a set of films” but not the actual film documents yet.

{
    "@set": {
        documents: Collection("film")
    }
}

Just like SQL, FQL is inspired by relational algebra. In FQL, we construct sets. For example, we could combine the above statement with other sets using familiar functions such as Union(), Difference(), Distinct(). A set is merely a definition of the data we would like to retrieve but has yet to be a concrete dataset.

Although it might seem like strings if we use the dashboard shell, we are actually using functions from the underlying JavaScript driver. Try misspelling a function in the dashboard shell, and you’ll get a familiar <function> undefined error. Or write:

var a = Documents(Collection("film"))
a

Make sure to copy the whole statement simultaneously, as the dashboard shell does not maintain variables between runs.

That JavaScript variable now contains our query definition, and we can use the variable to continue composing our query in a much more elegant way than string concatenation could allow. Many users use this to extend the language or construct their own DSL, which we will show extensively in the rest of the article.

Pagination

To materialize a set and retrieve the data, calling the Paginate() function is mandatory; this is an important measure to ensure scalability and to always keep transactions relatively small. This is a common best practice when consuming most interactive APIs. Since we added two films, we can add a size parameter to see the pagination in action. Fauna will return an after cursor to move to the next page.

Paginate(Documents(Collection("film")), {size: 1})

Which now returns a Page of film references:

{
  {
    after: [Ref(Collection("film"), "288877928042725889")],
    data: [Ref(Collection("film"), "288801457307648519")]
  }
}

Once we call Pagination, our set is transformed to a Page. The data types documentation lists which functions can be called on Sets, Pages, Arrays, etc.

We can then copy the after cursor to get to the next page.

Paginate(Documents(Collection("film")), {
size: 1, 
after: [Ref(Collection("film"), "288877928042725889")]
})

Data architects who have experimented with multiple ways of pagination within Postgres may want to know what kind of pagination this is. Fauna’s approach is close to the KeySet approach and cleverly takes advantage of snapshots to ensure that pages do not change when data is adapted. This is possible since everything we query in Fauna is backed by a sorted index. Just like Paginate, indexes are mandatory to avoid issuing underperforming queries.

Although we don’t seem to be using an index in the query above, Documents() is actually using a built-in index sorted by reference. By including pagination, the query above is actually more similar to the following query but with a superior form of pagination since the default page size in Fauna is 64:

SELECT * FROM film
LIMIT 64

Function Composition

In the previous queries, we only returned document references. To transform these references into the complete document data, making it more equivalent to a SELECT *, we will loop over these references with the Map() FQL function and call Get() on each of them.

Map(
    Paginate(Documents(Collection("film"))),
    Lambda(["ref"], Get(Var("ref")))
)

Besides Map() and Get(), we introduced Lambda() and Var() in the code snippet above. Lambda() is the FQL name for a simple anonymous function, which we can pass to Map to be executed on each element. Var() is used to retrieve an FQL variable (in this case, the parameter passed to Lambda). At this point, you may be wondering: can’t we retrieve the complete document directly instead of using Map()? We can do so by adding more values to the index, and we’ll extensively address the trade-offs when we talk about optimizations in the third article.

Why use the Var() and Lambda() syntax here? As mentioned before, we are composing the query by calling JavaScript functions. Using strings like “ref” as variables, and retrieving them with Var(), helps keep JS variables separate from FQL variables. The advantage of writing your query by composing JavaScript functions lies in the extensibility. For example, we can extend FQL by writing a simple function equivalent to a simple SELECT * in SQL.

function SelectAll(name) {
    return Map(
        Paginate(Documents(Collection(name))),
        Lambda(["ref"], Get(Var("ref")))
    ) 
}

With this function definition, we would then call it as:

SelectAll(“film")

This composability allows you to use Fauna in ways that would be infeasible or incredibly hard in other databases, as we’ll see later when we cover optimization strategies. A prime example is the Fauna GraphQL query which compiles one-to-one to FQL, maintaining the same database guarantees. This is relatively easy in Fauna but requires advanced techniques in traditional databases.

Querying a one-to-many relation: multiple options

Whether we decide to embed the additional data in a collection document or use explicit references, we have multiple options to retrieve the data related in our one-to-many relation.

Querying embedded documents

If we had embedded the language document by storing the languages directly in the film document, we would not have to change the query, simply returning the film documents would include the languages.

Map(
    Paginate(Documents(Collection("film"))),
    Lambda(["ref"], Get(Var("ref")))
)

Retrieving the film document with Get() would immediately return the complete document, including the languages.

{
    ref: Ref(Collection("film"), "289321621957640705"),
    ts: 1612177450050000,
    data: {
      title: "Academy Dinosaur",
      language: {
        spoken: {
          name: "English"
        },
        subtitles: {
          name: "English"
        }
      }
    }
  }

Querying normalized data with native references

Earlier, we chose to store native references instead of embedding the documents, like we depict below:

{
    data: {
        title: "Academy Dinosaur",
        language: {
            spoken: Ref(Collection("language"), "288878259769180673"),
            subtitles: Ref(Collection("language"), "288878259769180673")
        }
    }
}

This means that we need to write the equivalent of a Postgres join. In Postgres, that would look as follows:

SELECT * FROM film 
JOIN "language" as spol ON spol.language_id = film.spoken_language_id 
JOIN "language" as subl ON subl.language_id = film.subtitles_language_id 
LIMIT 64

In Postgres, we define the join we would like to see and rely on the query optimizer to select the right algorithm. If the query optimizer makes a wrong judgment, query performance can suffer significantly. Depending on the data and the way we join it, the join algorithm could differ and even change when the size of the data changes.

Due to the scalable nature of Fauna, we want predictability in terms of price and performance. We have talked about set functions such as Union(), Difference(), and Distinct(), and we can accomplish this without using a Join() statement in Fauna yet. Although joins can scale in general, their performance will depend on many factors. To ensure predictability, we’ll join on the materialized documents after pagination. We’ll retrieve the film document within the lambda and paginate on each level, as we’ll see when we tackle many-to-many joins.

Let’s continue incrementally building upon the previous query, which returns our film documents with language references. We’ll start here, as before:

Map(
    Paginate(Documents(Collection("film"))),
    Lambda(["ref"], Get(Var("ref")))
)

First, we'll slightly refactor it to bind variables with Let(), bringing more structure to our queries and rendering them more readable. Within a Let(), we can retrieve anything related to a film we desire. Here’s our next incremental iteration:

Map(
    Paginate(Documents(Collection("film"))),
    Lambda(["filmRef"], 
        Let({
           film: Get(Var("filmRef"))
        }, 
        // for now, we’ll return the film variable.
        Var("film")
    ))
)

Next, we retrieve both language references from the film document with Select() and get the actual languages by dereferencing the spoken and subtitles language reference with Get().

Map(
    Paginate(Documents(Collection("film"))),
    Lambda(["filmRef"],
        Let({
            film: Get(Var("filmRef")),
            spokenLang: Get(Select(['data', 'language', 'spoken'], Var("film"))),
            subLang: Get(Select(['data', 'language', 'subtitles'], Var("film")))
        },
        // todo return
        )
    )
)

And finally, return these variables in the structure we specify. We can now try this complete function in the shell:

Map(
    Paginate(Documents(Collection("film"))),
    Lambda(["filmRef"],
        Let({
            film: Get(Var("filmRef")),
            spokenLang: Get(Select(['data', 'language', 'spoken'], Var("film"))),
            subLang: Get(Select(['data', 'language', 'subtitles'], Var("film")))
        },      
        // return a JSON object
        {
            film: Var("film"),
            language: {
                spoken: Var("spokenLang"),
                subtitles: Var("subLang")
            }
        })
    )
)

The complete result is now nicely structured, with both films and both languages.

{
  data: [
    {
      film: {
        ref: Ref(Collection("film"), "352445336554307667"),
        ts: 1672376915430000,
        data: {
          title: "Academy Dinosaur",
          language: {
            spoken: Ref(Collection("language"), "352444818516869204"),
            subtitles: Ref(Collection("language"), "352444818516869204")
          }
        }
      },
      language: {
        spoken: {
          ref: Ref(Collection("language"), "352444818516869204"),
          ts: 1672376421400000,
          data: {
            name: "English"
          }
        },
        subtitles: {
          ref: Ref(Collection("language"), "352444818516869204"),
          ts: 1672376421400000,
          data: {
            name: "English"
          }
        }
      }
    },
    {
      film: {
        ref: Ref(Collection("film"), "352447504810246229"),
        ts: 1672378983250000,
        data: {
          title: "Back to the Future",
          language: {
            spoken: Ref(Collection("language"), "352444818516869204"),
            subtitles: Ref(Collection("language"), "352444818516869204")
          }
        }
      },
      language: {
        spoken: {
          ref: Ref(Collection("language"), "352444818516869204"),
          ts: 1672376421400000,
          data: {
            name: "English"
          }
        },
        subtitles: {
          ref: Ref(Collection("language"), "352444818516869204"),
          ts: 1672376421400000,
          data: {
            name: "English"
          }
        }
      }
    }
  ]
}

Querying normalized data with user-defined primary keys

In the above example, we had access to language references in the film document. Therefore, we could directly use Get() to dereference these references. We could have chosen user-defined primary keys instead of native Fauna references like this:

Create(
    Collection("film"),
    {
        data: {
            title: "Academy Dinosaur",
            language: {
                spoken: 6,
                subtitles: 6
            }
        }
    }
)

and

Create(
    Collection("language"), 
    {
       data: {
          id: 6,
          name: "English"
       }
    }
  )

Fauna is still able to retrieve the languages, but it would require an extra step with an index. For the sake of comparison, let’s implement the same query again with this use case. First, we need an index to retrieve the languages by id:

CreateIndex({
    name: "language_by_id",
    source: Collection("language"),
    terms: [
      {
        field: ["data", "id"]
      }
    ],
    values: [
      {
        field: ["ref"]
      }
    ]
})

The main ingredients of indexes in Fauna are terms and values. Terms determine what the index matches on, while values determine what it returns (and in what order). Since indexes return values in Fauna, they are a mix of a view and an index as we know them in Postgres. In fact, we can significantly optimize our queries with indexes once we know our application’s data access patterns, as we’ll explain later. In this case, we’ll start in a generic fashion and only return the language reference from the index.

We’ll call the index with the Match() function. As long as we only need the first result from Match(), we can call Get() on the match. We will see how to handle multiple results later on.

Map(
    Paginate(Documents(Collection("film"))),
    Lambda(["filmRef"],
        Let({
            film: Get(Var("filmRef")),
            spokenLangId: Select(['data', 'language', 'spoken'], Var("film")),
            subLangId: Select(['data', 'language', 'subtitles'], Var("film")),
            spokenLang: Get(Match(Index("language_by_id"), Var("spokenLangId"))),
            subLang: Get(Match(Index("language_by_id"), Var("subLangId"))),
        },
            // and return the values however you want.
        )
    )
)

At this point you may wonder, should I use native Fauna references or user-defined keys? You can choose either. If you used custom keys or IDs in the Postgres implementation you are migrating from, you may have other business logic reasons to continue using them. If you use Fauna’s native references, you may simplify your code and retrieve data more efficiently (similar to what graph databases call index-free-adjacency).

And you can set your own custom IDs within Fauna's native references during document creation:

Create(
    Ref(Collection('language'), '6'),
    { name: { title: 'English' } },
)

Summary

We covered a lot of ground here. First, we outlined the key motivations why application developers are considering moving away from traditional relational databases like Postgres. Even with Postgres’ feature maturity, scaling traditional database applications horizontally is still challenging, especially if the application is to handle very significant traffic loads. We covered key differences between Postgres and Fauna and provided a terminology mapping that should have demonstrated the equivalence of many shared concepts.

We went through the process of creating a database, building a couple of collections, and inserting documents in those collections to illustrate a typical one-to-many database relation from one collection to another. We covered using native Fauna references, as well as the alternatives of embedding and denormalizing the data, as well as using user-defined IDs, and how those decisions change how we build queries to retrieve the data in all those cases. We used FQL to incrementally compose functions that look a lot like JavaScript, illustrating the deep integration that can be achieved in your application code without requiring an ORM or avoiding embedded SQL strings that can be prone to injections.

This sets us up for part two, where we use a many-to-many relationship example to cover more modeling strategies and extend the idea of writing a domain-specific language with FQL. Finally, part three focuses on referential integrity as well as additional modeling, optimization, and migration ideas. By the time you complete this journey, you should be armed with a solid foundational strategy to transition your Postgres-driven application. This strategy will provide you with enough guidance to have a scalable and performant resulting system from the beginning. It will also set you up with a composable and testable code base that will help you maintain your code leveraging modern application best practices.

About the author

Luis Colon is a data scientist that focuses on modern database applications and best practices, as well as other serverless, cloud, and related technologies. He currently serves as a Senior Technology Evangelist at Fauna, Inc. You can reach him at @luiscolon1 on Twitter and Reddit.

Side-by-side comparison of serverless databases

Fauna — Thu, 29 Dec 2022 16:53:31 +0000

Serverless databases make it easy to build and scale your application because they abstract away the underlying infrastructure and automatically scale to meet the need of your application. A serverless database lets you focus on your code without worrying about capacity planning, infrastructure maintenance, and server management.

There are quite a few options for serverless databases, including established solutions like DynamoDB and newer offerings such as Mongo Atlas and CockroachDB serverless. This article compares commonly used serverless databases so you can make an informed decision when picking a new database for your project.

MongoDB (Mongo Atlas)

MongoDB is a widely used NoSQL database. MongoDB Atlas offers a provisioned managed service as well as a serverless offering. In this article we will compare the serverless offering.

MongoDB Atlas offers fully managed serverless instances of MongoDB. With Mongo Atlas, you don’t have to manage any infrastructure by yourself.

✅ Connection over HTTP

Mongo Atlas allows your application to connect to your database through an HTTP connection.

⚠️ Multi-region availability

By default, MongoDB Atlas is deployed in a single region. However, MongoDB Atlas can automatically replicate data across multiple servers for improved reliability and availability. It is partially automated, and you’ll need prior knowledge of sharding and replication.

⚠️ Schema design (Schema-less, non-relational)

MongoDB is flexible and suitable for both structured and unstructured data. Unlike a relational database, MongoDB doesn’t support foreign key joins. Because of this modeling, your data relationships can be tricky with Mongo. There are other ways to get around this problem in MongoDB. Review this thread to learn more about data modeling in MongoDB.

❌ No cold starts

If you have a large database then MongoDB Atlas clusters can have a long cold start time.

⚠️ ACID transactions

Multi-document transactions are not enabled by default in MongoDB or MongoDB Atlas clusters. However, you can configure MongoDB to support multi-document ACID transactions based on your use case.

DynamoDB

DynamoDB is a fully managed No-SQL database from Amazon. It supports key-value and document data structures. DynamoDB has a pay-as-you-go model, and it provides auto-scaling.

✅ Connection over HTTP

DynamoDB allows connection over HTTP.

⚠️ Multi-region availability

You can setup multi-region availability with DynamoDB’s global tables feature. However, keep in mind there is significant configuration required to get it up and running properly.

DynamoDB requires users to choose a partition key that determines how data is grouped and distributed among partitions. How you choose this key impacts DynamoDB’s scalability. Having a thorough understanding of how your data will be accessed will be critical to selecting the most appropriate partitioning key and strategy.

⚠️ Schema design (Schema-less)

DynamoDB is largely schema-less. Like many of its NoSQL siblings, DynamoDB lacks explicit support for relational data. It is designed with a denormalized schema in mind. Document sizes are also limited to 400kb in DynamoDB, forcing developers to denormalize the application data.

✅ No cold starts

DynamoDB doesn’t have any cold start-related concerns.

⚠️ ACID transactions

DynamoDB transactions are only ACID-compliant within a single region. DynamoDB does not support strongly consistent reads across regions.

Follow this article for a detailed comparison of Fauna and DynamoDB.

CockroachDB

Like Mongo here we are focusing on the Serverless offering of the CockroachDB not the provisioned managed service since provisioned service doesn’t offer pay per usage.

CockroachDB is a distributed SQL database system. It is based on the idea of a "cockroach cluster," a group of nodes that work together to provide fault tolerance and high availability. Each node in a CockroachDB cluster can serve read and write requests, and the cluster can automatically recover from failures and distribute data evenly across the nodes.

✅ Connection over HTTP

CockroachDB supports connection over HTTP.

✅ Multi-region availability

CockroachDB provides out-of-box multi-region availability without further configuration.

⚠️ Schema design (SQL )

CockroachDB has a Dynamic schema. It is relational and supports SQL. Since cockroach is based on traditional RDBMS it may take a lot of work to evolve unstructured data as your application grows.

❌ No cold starts

For large amounts of datasets CockroachDB clusters may experience cold starts.

✅ ACID transactions

CockroachDB fully supports strong consistency and ACID transactions.

Fauna

Fauna is a serverless database that combines the flexibility of NoSQL with the relational querying capabilities and consistency of SQL. You get the best of both worlds when using Fauna. It is fully managed, auto-scales, and does not require infrastructure management.

✅ Connection over HTTP

Fauna allows your applications to connect to the database over HTTP. You can use a Fauna driver or Fauna's GraphQL interface to communicate with the database.

✅ Multi-region availability

Fauna provides multi-region availability out of the box; it will auto-replicate your data across servers. It is distributed by default within a geographic region or across the globe. You always get strong consistency among region groups. Fauna is designed to keep the data closest to your users, improving reliability and availability.

✅ Schema design (Document-relational)

Fauna follows a document-relational database model. It combines the flexibility and familiarity of JSON documents with the relationships and querying power of a traditional relational database.
In short, you get the best of both SQL and NoSQL worlds with Fauna. Because of the flexible nature of the Fauna database, it is effortless to scale and evolve your data schema as your application grows.

✅ No cold starts

Fauna has zero cold starts.

✅ ACID transactions

One of the features of Fauna that has generated the most excitement is its strongly consistent distributed ACID transaction engine. Fauna provides strong consistency of data globally through an implementation of the Calvin protocol.

Below you’ll find a visual comparison of all these serverless database offerings.

	Mongo Atlas	DynamoDB	CockroachDB	Fauna
Connection of HTTP	✅	✅	✅	✅
Multi-region availability	⚠️	⚠️	✅	✅
No cold starts	❌	✅	❌	✅
ACID transaction	⚠️	⚠️	✅	✅
Flexible schema design	⚠️ (NoSQL only)	⚠️ (NoSQL only)	⚠️ (SQL only)	✅ (Document-relational)

If you are considering Fauna, create a free account (no credit card required) and give it a go. If you have questions, you can reach out in the Fauna Discord channel or Fauna community forum.

Implementing Fauna as infrastructure as code with Serverless Framework

Fauna — Thu, 27 Oct 2022 17:26:24 +0000

This article demonstrates how to use Fauna as infrastructure as code (IaC) in your application using the Serverless Framework, one of the most popular tools for managing infrastructure as code. Fauna has a dedicated plugin for the Serverless Framework that gives you complete control to manage your Fauna resources. You can integrate it into your test and CI/CD pipelines to keep your databases in sync across multiple environments.

Benefits of IaC

Before we dive deep into implementing Fauna as IaC, let's discuss why you might want to integrate IaC.

There are three main benefits of IaC:

Decreased risks: Provisioning all of your infrastructures manually can be risky, especially if you have multiple dependencies among services. Complex deployment is prone to human errors. When you automate the process with IaC, you reduce these. Your infrastructure also becomes testable, and you can spin up multiple environments (exact replicas of production).
Efficient software development lifecycle: With IaC, infrastructure provisioning becomes more reliable and consistent. Developers get complete control of the infrastructure through code. Developers can script once and use that code multiple times, saving time and effort while keeping full control.
Self-documenting and reduced administration: IaC is self-documenting. And it reduces administrative overhead, allowing your engineering efforts to be focused on new feature development.

Getting started with the Fauna Serverless Framework plugin

Install the Serverless Framework plugin with the following command

$ npm install @fauna-labs/serverless-fauna --save-dev

or using yarn

$ yarn add @fauna-labs/serverless-fauna

Open your serverless.yml file and add the following code to add Fauna to your project.


plugins:
  - "@fauna-labs/serverless-fauna"
fauna:
  client:
    secret: ${env:FAUNA_ROOT_KEY}
    # domain: db.fauna.com
    # port: 433
    # scheme: https

By default, the domain is set to [db.fauna.com](http://db.fauna.com). You can create new collections by adding the collection name under the collections field as demonstrated in the following code.

fauna:
  client:
    secret: ${env:FAUNA_ROOT_KEY}
    # domain: db.fauna.com
    # port: 433
    # scheme: https
  collections:
    Movies:
      name: Movies
      data:
        some_data_key: some_data_value

The collection configuration accepts the same configuration as CreateCollection query.

Similarly, you can add functions and indexes.

fauna:
  client:
    secret: ${env:FAUNA_ROOT_KEY}
    # domain: db.fauna.com
    # port: 433
    # scheme: https
  collections:
    Movies:
      name: Movies
      data:
        some_data_key: some_data_value

  functions:
    double:
      name: double
      body: ${file(./double.fql)}

  indexes:
    movies_by_type:
      name: movies_by_type
      source: ${self:fauna.collections.Movies.name}
      terms:
        fields: 
          - data.type

You can review the documentation for the Serverless Framework plugin to learn more about different functionalities and implementations.

We have also created a hands-on tutorial on building a REST API with Fauna, Serverless, and AWS Lambda. This tutorial will help you get up and running with a simple project with Fauna and Serverless Framework.

If you are searching for something more comprehensive, we have also created a self-paced workshop that will guide you through building a real-world project with Fauna, Serverless Framework, and AWS services. You can find the workshop here.

Fauna AWS Workshop ~ Building an event-driven app with AWS services and Fauna

This hands-on guide walks you through building a real-world event-driven serverless application using AWS services (i.e., AWS Lambda, Step Functions, API Gateway) and Fauna. In this workshop, you build a vacation booking application (Similar to Kayak or Redtag deals).

Got questions? Reach out in our Discord channel or in our community forum.

Delivering personalized content with Netlify’s Next.js Advanced Middleware and Fauna

Fauna — Wed, 26 Oct 2022 14:10:27 +0000

It’s become a truism that the speed and performance of web applications, e-commerce sites, and websites are critical variables in converting visitors to paying customers. Modern consumers expect these assets to serve data quickly, accurately, and contextually. API-first platforms like Fauna and Netlify, paired with other composable technologies designed for distributed workloads, drastically simplify the process of deploying applications that serve dynamic content at the edge with low latency, without complicated stitching or multi-team engineering efforts.

Serving globally distributed data in legacy architectures

Historically, developers have been constrained in how they optimize the performance attributes and experience of their applications by the underlying infrastructure - typically centralized servers and databases. The emergence of content delivery networks (CDNs) in the early 2000s enabled companies to host and distribute static assets to a globally distributed audience with low latency - however these CDNs were limited to hosting static content like images, text files, and HTML files. As these CDNs were globally distributed, the performance attributes of this static content was far superior to relying on a more traditional centralized model - a user accessing a static site in Shanghai would access that content from the CDN node in Shanghai, instead of being pushed to a data center across the county. With that said, the benefits of serving static content through a CDN wasn’t extended to dynamic content that could change over time and evolve; this category of content relies on compute and data services to transform the data based on context.

In the case of a global or even regional application, relying on a centralized architecture to serve dynamic content can be suboptimal. Consider an e-commerce application with users visiting the site from Los Angeles and Toronto. In an ideal world, the application would serve content that’s unique to those users and may change depending on context - where the user is physically located (and the potential discounts or prompts subsequently served to the user’s location), or if the user came from a referring source, for example. This data is ephemeral and dynamic in nature, and unique to each independent user accessing the application. If a user in Los Angeles is being served data hosted in the US-East 1 data center in Virginia, the performance attributes will suffer as a result simply due to the physical distance between the user and the compute. Similar to static content with CDNs, we want to optimize for speed and performance by moving the processing elements as close to each respective user as possible. This means pushing the compute and data out to the edge.

Dynamic personalization at the edge

Personalization allows companies to serve information that’s tailored for a particular customer based on a defined set of characteristics (most commonly geo-based). For example, if a visitor on an e-commerce site is based in Seattle, they may be served a discount on an item based on an inventory surplus in a local warehouse. Being able to serve this type of dynamic data, however, is not trivial. The ability to move the compute and data for dynamic content as close as possible to users (as demonstrated in CDNs) was until very recently impossible; this has changed with the emergence of platforms like Fauna and Netlify.

With Next.js Advanced Middleware (powered by Netlify Edge Functions) and Fauna’s distributed-by-default document-relational database, we’re able to move servers and databases closer to user locations - similar to the shift we’ve witnessed with static assets served on CDNs, but now for dynamic content. Together, Fauna and Netlify enable a transformation in the application architecture for personalization at the edge: developers now have the option to host business logic wherever it’s most performant.

When a user writes to or reads from a node, Netlify Edge Functions, the images/CSS files, and pieces of business logic are being grabbed from the closest possible compute node and corresponding data source node. Further, with the power of Fauna’s user-defined functions and advanced querying, you’re able to access a new mental model that allows you to make strategic decisions on where and how often work needs to be done; if you’re able to move the processing further back in the stack with Fauna, you can build it into the places that have the tightest relationships with the data and be optimized in ways that client-side JavaScript can’t be.

While the middleware component of Next.js is germane to Next.js, out-of-the-box it is limited to redirects or rewrites (you can send users to another page, or proxy other content into the application and show users new content). Netlify has extended the middleware API to give access to the full request, which allows for full transformation and personalization - whether it’s location or referral based. With a proxy or a redirect, you’d need to develop a custom page for each redirect. The ability to do a direct transformation with Netlify ultimately makes the engineering far less cumbersome. Netlify provides a really good example of the power of page transformations here.

Why Fauna + Netlify for edge personalization

Fauna is delivered as a global API and is complementary to Netlify’s Edge Functions; ultimately, you don’t want to bloat your edge functions with code. Everything in the Fauna-Netlify architecture is server-side rendered, so it creates a faster experience by avoiding bloated client-side code that would flood the client CPU or the browser memory. You’re able to modify your code from being a large response down to the few lines you need to write to account for personalization, and it’s possible to do this in the edge function instead of the client-side without client-side impact to the user and without sacrificing performance. Until very recently, this type of architecture wasn’t possible to build, especially for small teams. With Fauna and Netlify, you can get up and running with a working solution within hours.

Companies would historically need DevOps, frontend, backend, and database management teams to handle the orchestration of all of the elements associated with a global application. Now, a single full-stack developer can handle all of these elements and sketch out an application in a matter of hours that has global deployment, global database replication, and all of the front end and back end configured. Fauna’s auto-replication, distribution-by-default, and serverless attributes paired with Netlify’s edge compute make it possible. There’s no need to account for sharding, provisioning, or global replication - it’s all delivered out-of-the-box.

Netlify Edge Functions + Fauna example and code walk-through

To move out of the abstract and into a real-world example, we hosted a webinar with Netlify where we did a code walk-through of a basic marketing site pushed to the edge on Fauna and Netlify. Check out the accompanying webinar and start digging into the code in the repo to learn how to build the functionality with just a few lines of code. You’ll see that in this example, one user is making a request to the site from Toronto, Canada, and another from Portland, Oregon. When each respective user makes a request, they hit the compute node closest to their respective locations and each is served a unique page. Meanwhile, the request is also directed to the nearest Fauna replica to read and serve the data. In this example, the read is modified at the edge and as close as possible to the user - which wouldn’t be possible with a regionally hosted database or server. Both the compute and data is at the edge - which ultimately reduces latency.

Getting started

Fauna and Netlify unlock the ability to optimize for the most performant architectural decision based on the use case a team might be trying to solve, instead of being limited to whatever your legacy infrastructure may dictate.

Are you ready to get started?

Have any questions or want to engage with the Fauna and Netlify communities? Reach out in the Fauna Discord channel or Fauna community forum, and also check out the Netlify Community Forums.

Transfer data in Fauna to your analytics tool using Airbyte

Fauna — Tue, 25 Oct 2022 21:48:01 +0000

We are excited to introduce Fauna’s new Airbyte open source connector. This connector lets you replicate Fauna data into your data warehouses, lakes, and analytical databases, such as Snowflake, Redshift, S3, and more.

Why Airbyte

With the proliferation of applications and data sources, companies are often required to build custom connectors for data transfer across their architectures. Most of these ETL (extract, transform, and load) tools require maintaining and updating the connectors as requirements change over time. Airbyte is an open source data pipeline platform that eliminates this burden by offering a robust connector ecosystem that can scale without having to maintain the connector itself.

Why Fauna

Fauna is a distributed document-relational database delivered as a cloud API. Developers choose Fauna’s document-relational model because it combines the flexibility of NoSQL databases with the relational querying and ACID capabilities of SQL databases. This model is delivered as an API so you can focus on building features and not to worry about any operations or infrastructure management.

Why Fauna + Airbyte

Fauna and Airbyte both enable improved productivity and developer experience - together, the connector will allow developers to port and migrate transactional data in Fauna to your choice of analytical tools to drive business insights.
Continue reading for a guide on how to configure the Fauna source connector to transfer your database to one of the data analytics or warehousing destination connectors supported by Airbyte.
The Fauna source supports the following ways to export your data:

Full refresh append sync mode copies all of your data to the destination, without deleting existing data.
Full refresh overwrite sync mode copies the whole stream and replaces data in the destination by overwriting it.
Incremental append sync mode periodically transfers new, changed, or deleted data to the destination.
Incremental deduped history sync mode copies new records from stream and appends data in the destination, while providing a de-duplicated view mirroring the state of the stream in the source

Prerequisites

You need a destination database account and need to set up the Data Build Tool (dbt™) to transform fields in your documents to columns in your destination. You also need to install Docker.

Create a destination database account

If you do not already have an account for the database associated with your destination connector, create an account and save the authentication credentials for setting up the destination connector to populate the destination database.

Set up dbt

To access the fields in your Fauna source using SQL-style statements, create a dbt account and set up dbt as described in the Airbyte Transformations with dbt setup guide. The guide steps you through the setup for transforming the data between the source and destination, and connects you to the destination database.

Install Docker

The Fauna connector is an Airbyte Open Source integration, deployed as a Docker image. If you do not already have Docker installed, follow the Install Docker Engine guide.

Step 1: Set up the Fauna source

Depending on your use case, set up one of the following sync modes for your collection.

Full refresh sync mode

Follow these steps to fully sync the source and destination database.

1- Use the Fauna Dashboard or fauna-shell shell to create a role that can read the collection to be exported. The Fauna Source needs access to the Collections resource so that it can find which collections are readable. This does not give it access to all the collections, just the names of all the collections. For example:

CreateRole({
  name: "airbyte-readonly",
  privileges: [{
    resource: Collection("COLLECTION_NAME"),
    actions: { read: true }
  }],
})

Replace COLLECTION_NAME with the collection name for this connector.

2- Create a secret that has the permissions associated with the role, using the name of the role you created. For example:

CreateKey({
  name: "airbyte-readonly",
  role: Role("airbyte-readonly"),
})

{
  ref: Key("341909050919747665"),
  ts: 1662328730450000,
  role: Role("airbyte-readonly"),
  secret: "fnAEjXudojkeRWaz5lxL2wWuqHd8k690edbKNYZz",
  hashed_secret: "$2a$05$TGr5F3JzriWbRUXlKMlykerq1nnYzEUr4euwrbrLUcWgLhvWmnW6S"
}

Save the returned secret, otherwise, you need to create a new key.

Incremental append sync mode

Use incremental sync mode to periodically sync the source and destination, updating only new and changed data.

Follow these steps to set up incremental sync.

1- Use the Fauna Dashboard or fauna-shell to create an index, which lets the connector do incremental syncs. For example:

CreateIndex({
  name: "INDEX_NAME",
  source: Collection("COLLECTION_NAME"),
  terms: [],
  values: [
    { "field": "ts" },
    { "field": "ref" }
  ]
})

Replace INDEX_NAME with the name you configured for the Incremental Sync Index. Replace COLLECTION_NAME with the name of the collection configured for this connector.

|Index values|Description|
| --- | ----------- |
|`ts`| Last modified timestamp.|
|`ref`|Unique document identifier.|

2- Create a role that can read the collection and index, and can access index metadata to validate the index settings. For example:

CreateRole({
  name: "airbyte-readonly",
  privileges: [
    {
      resource: Collection("COLLECTION_NAME"),
      actions: { read: true }
    },
    {
      resource: Index("INDEX_NAME"),
      actions: { read: true }
    },
    {
      resource: Indexes(),
      actions: { read: true }
    }
  ],
})

Replace COLLECTION_NAME with the name of the collection configured for this connector. Replace INDEX_NAME with the name that you configured for the Incremental Sync Index.

3- Create a secret key that has the permissions associated with the role, using the name of the role you created. For example:

CreateKey({
  name: "airbyte-readonly",
  role: Role("airbyte-readonly"),
})

{
  ref: Key("341909050919747665"),
  ts: 1662328730450000,
  role: Role("airbyte-readonly"),
  secret: "fnAEjXudojkeRWaz5lxL2wWuqHd8k690edbKNYZz",
  hashed_secret: "$2a$05$TGr5F3JzriWbRUXlKMlykerq1nnYzEUr4euwrbrLUcWgLhvWmnW6S"
}

Save the returned secret. You need to enter the secret in step 2 of the Install Docker procedure. It is important to save the key, otherwise, you need to create a new key if you lose the provided secret.

The Fauna source iterates through all indexes on the database. For each index it finds, the following conditions must be met for incremental sync:

The source must be able to Get() the index, which means it needs read access to this index.
The source of the index must be a reference to the collection you are trying to sync
The number of values must be two.
The number of terms must be zero.
The values must be equal to:
```
{"field": "ts"}
{"field": "ref"}
```

All of the above conditions are checked in the order listed. If a check fails, it skips that index.

If no indexes are found in the initial setup, incremental sync isn't available for the given collection. No error is emitted because it can't be determined whether or not you are expecting an index for that collection.

If you find that the collection doesn't have incremental sync available, make sure that you followed all the setup steps, and that the source, terms, and values all match for your index.

Step 2: Deploy and launch Airbyte

Refer to the Deploy Airbyte instructions to install and deploy Airbyte. Enter the following commands to deploy the Airbyte server:
```
git clone https://github.com/airbytehq/airbyte.git
cd airbyte
docker-compose up
```
When the Airbyte banner displays, launch the Airbyte dashboard at http://localhost:8000.
Choose the Connections menu item to start setting up your data source.

Step 3: Set up the Fauna source

In the Airbyte dashboard, click the + New connection button. If you previously set up a source, click the Use existing source button to choose that source.
In the Source type dropdown, choose Fauna and click the Set up source button. This lists the configurable Fauna connector parameters. An in-app Setup Guide in the right-side panel also gives detailed setup instructions.

Set the following required parameters:

Parameter	Description
`Name`	Enter a descriptive name for this connection. The name is displayed in the Connections window connections list.
`Domain`	Enter the domain of the collection you want to export. See Region Groups for region domains.
`Port`	Enter the default port number: `443`.
`Scheme`	Enter the scheme used to connect to Fauna: `https`.
`Fauna Secret`	Enter the saved Fauna secret that you use to authenticate with the database.
`Page Size`	The page size lets you control the memory size, which affects connector performance.
`Deletion Mode`	The deletion mode lets you specify whether to ignore document deletions or flag documents as deleted, depending on your use case. + Choose from the following options: The Ignore option ignores document deletions. The Deleted Field option adds a date column that has the date when you deleted the document. This maintains document history while letting the destination reconstruct deletion events.

After setting up the source, click the Set up source button.

The "All connection tests passed!" message confirms successful connection to the Fauna source. This minimally confirms:

-   The secret is valid.


  The connector can list collections and indexes.

Step 4: Set up the destination

In the New connection window, choose a Destination type and click the Set up destination button. If you previously set up a destination, click the Use existing destination button to select and use that destination. Otherwise, continue to set up a new destination.
Destination connector configuration parameters are unique to the destination. Populate the Set up the destination fields according to the connector requirements, including authentication information if needed. A Setup Guide is provided in the right-side panel with detailed setup instructions.
When you are done, click the Set up destination button.

Step 5: Set up the connection

Set up the connection to sync the source and destination.

Enter a descriptive name for the connection in the Name field.
Choose a Transfer > Replication frequency, which is the data sync interval.

You can choose the Manual option to manually sync the data.

In the Streams > Destination Namespace field, choose a destination namespace where the data is stored. Options include:

Option	Description
`Mirror source structure`	Sets the name in the destination database to the name used for the Fauna source.
`Other`	Uses another naming option, such as prefixing the database name with a string.

Optionally, enter a stream name prefix in the Streams > Destination Stream Prefix field.
In the Activate the streams you want to sync section, click the > arrow to expand the available fields:

Field Description

data Collection data.

ref Unique document identifier.

ts Data timestamp.

ttl Time-to-live interval.

The document is deleted if it is not modified in the ttl time interval. The default value is null for not used. After document deletion, it is not displayed in temporal queries and the connector does not emit a deleted_at row.
Select ref as the Primary key. This uniquely identifies the document in the collection.

Field	Description
`data`	Collection data.
`ref`	Unique document identifier.
`ts`	Data timestamp.
`ttl`	Time-to-live interval.

Choose a Sync mode as the source sync behavior, full or incremental.

A new incremental sync gets the full database, the same as a full sync.

Sync mode	Description
Incremental - Deduped history	Sync new records from stream and append data in destination, also provides a de-duplicated view mirroring the state of the stream in the source.
Full refresh - Overwrite	Sync the whole stream and replace data in destination by overwriting it.
Incremental - Append	Sync new records from stream and append data in destination.
Full refresh - Append	Sync the whole stream and append data in destination.

If fewer than four options are displayed, it indicates that the index is incorrectly set up. See Step 1: Set up the Fauna source.

Choose the Normalization and Transformation data format:

Data format Description

Raw data (JSON) Put all the source data in a single column.

Normalized tabular data Put the ref, ts, ttl, and data fields in separate columns.
Click the + Add transformation button to add the dbt transform.

To extract the fields in the source data column, you need to configure dbt to map source data to destination database columns. For example, the following SQL-based query extracts the name, account_balance, and credit_card/expires fields from the source data column to populate three separate columns of the destination data:
```
with output as (
  select
    data:name as name
    data:account_balance as balance
    data:credit_card:expires as cc_expires
  from airbyte_schema.users
)

select *from output
```
Click the Set up connection button.

Data format	Description
`Raw data (JSON)`	Put all the source data in a single column.
`Normalized tabular data`	Put the `ref`, `ts`, `ttl`, and `data` fields in separate columns.

Step 6: Sync the data

On the Connection page for the connection you created, click the Sync now button.

The time to run a sync varies with the status displayed in Sync History. When the sync completes, the status changes from Running to Succeeded and shows:

The number of bytes transferred.
The number of records emitted and committed.
The sync duration.

Step 7: Verify the integration

To expand the sync log, click the > arrow to the right of the displayed time. This gives you a detailed view of the sync events.

Finally, verify successful database transfer by opening and viewing the destination database.

Conclusion

Integrating Fauna with the Airbyte open source solution arms developers building on Fauna with a powerful tool for gaining insights into the operational data living on Fauna. We’re excited to build on our partnership with Airbyte by working towards introducing an Airbyte Cloud connector. If you have any interest in a Fauna + Airbyte Cloud integration or questions about the open source connector, feel free to reachout to us and ask questions in our forum or on our Discord.

Building an edge serverless GraphQL backend with Cloudflare Workers and Fauna

Shadid Haque — Wed, 05 Oct 2022 13:41:04 +0000

This tutorial will teach you how to build a serverless GraphQL service on the edge with Cloudflare Workers and Fauna — allowing you to take advantage of two edge serverless compatible technologies. Try the live demo here. You can find the complete code for this demo here.

Before diving into the code, let’s explore what edge serverless is and why it matters.

What is edge serverless?

Edge serverless is an infrastructural evolution of cloud serverless in which the computing resources that deliver serverless functions are located geographically closer to the end user.

Cloudflare Workers are, by default, edge serverless functions. They are globally distributed in a way that the end user always hits the nearest compute instance, lowering latency.

There are three main benefits of building your GraphQL service on edge serverless:

You don't have to manage servers. Your application auto scales.
You pay per usage for your infrastructure.
Your app is faster. The computing resources that deliver serverless functions are located geographically closer to the end user which reduces latency for your application.

Why use Fauna as the database?

By default, Fauna is a serverless database with edge support. Like Cloudflare Workers, Fauna is distributed and the data gets replicated across different servers within a specified region group. When users request data, it always retrieves the data from the nearest geographical location reducing the latency.

You get the best of your edge infrastructure when your compute and database are both edge serverless compatible. In addition, Fauna is an easy-to-use database with zero administrative overhead.

Prerequisites

Make sure you have the following installed in your computer

Node.js 16 or higher
Cloudflare Wrangler CLI (install instructions)

Scaffolding a new Cloudflare Workers project

Run the following code to scaffold a new Cloudflare Workers project with Wrangler.

$ wrangler init <my-awesome-project>

The CLI wizard will walk you through some configuration questions and bootstrap your application. When it asks you Would you like to create a Worker select Yes and select Fetch handler option.

Open the src/index.ts file. You will see some code that reads as follows.

export interface Env {
}

export default {
    async fetch(
        request: Request,
        env: Env,
        ctx: ExecutionContext
    ): Promise<Response> {
        return new Response("Hello World!");
    },
};

Run wrangler dev and visit http://0.0.0.0:8787/. It should display “Hello World!”.

Setting up a GraphQL server in Cloudflare Workers

Run the following command to install all the dependencies.

$ npm i @graphql-yoga/common faunadb --save

Next, delete everything from index.ts file and add the following code. Here you are creating a simple GraphQL server with @graphql-yoga library.

import { createServer } from '@graphql-yoga/common'

const server = createServer({
  schema: {
    typeDefs: /* GraphQL */ `
      type Query {
        hello(msg: String): String
      }
    `,
    resolvers: {
      Query: {
        hello: async (_, { msg }) => {
          return `Hello, ${msg}`
        }
      },
    },
  },
})
server.start()

Run wrangler dev and visit http://0.0.0.0:8787/ in your browser. In your browser, a GraphQL playground will open.

Make the following query in GraphQL playground to make sure everything is working as expected.

{
  hello(msg: "adwwd")
}

Result:

{
  "data": {
    "hello": "Hello, adwwd"
  }
}

Next, let’s go ahead and replace the GraphQL schema with something more practical. We want CRUD (Create, Read, Update and Delete) operations in our GraphQL service. Make the following changes to the schema to add CRUD.

import { createServer } from '@graphql-yoga/common'

const server = createServer({
  schema: {
    typeDefs: /* GraphQL */ `
      type Post {
        id: ID!
        title: String!
        content: String!
      }
      type Query {
        getPost(id: ID!): Post
        listPosts: [Post]!
      }
      type Mutation {
        addPost(input: PostInput): Post
        deletePost(id: ID): Boolean
        updatePost(id: ID, input: PostInput): Post
      }
      type Subscription {
        onPostChange(id: ID): Post
      }
      input PostInput {
        title: String
        content: String
      }
    `,
    resolvers: {
      Query: {
      },
    },
  },
})
server.start()

You now have your schema ready. However, you cannot execute these queries and mutations since the resolvers are not yet implemented.

Before we can implement the resolver functions let’s go ahead and set up our database. We are using Fauna as our database.

Creating a Fauna database

Create a new collection called Post.

Next, go over to the Security tab and create a new server secret for your database.

Save the generated database secret. Make sure you do not reveal this secret in your version control.

Back in the code, add the Fauna database key and domain in your wrangler.toml file as follows.

# wrangler.toml
name = "edge-graphql"
main = "src/index.ts"
compatibility_date = "2022-09-23"
compatibility_flags = ["streams_enable_constructors"]
node_compat = true

[vars]
FAUNA_SECRET = "fnAxxx...."
FAUNA_DOMAIN = "db.us.fauna.com"

Next, create a new file src/db.ts and add the following code. In the following code snippet, you initialize the Fauna database.

import * as faunadb from "faunadb";

export const q = faunadb.query as any;
declare const FAUNA_DOMAIN: string;
declare const FAUNA_SECRET: string;

export const faunaClient = new faunadb.Client({ 
  domain: FAUNA_DOMAIN,
  secret: FAUNA_SECRET,
});

Implementing resolvers

Add post (Mutation)

First, you need to enable node_compat mode for Cloudflare Workers since you will use modular JavaScript. Go to wrangler.toml file and add the following line.

name = "ggop"
main = "src/index.ts"
compatibility_date = "2022-09-25"
node_compat = true

[vars]
...

Inside the src/index file, under resolvers field, add a new resolver function addPost as follows.

import { createServer } from '@graphql-yoga/common'
import { faunaClient, q } from './db'

const server = createServer({
  schema: {
    typeDefs: /* GraphQL */ `
      ...
    `,
    resolvers: {
      Query: {
      },
      Mutation: {
        addPost: async (parent, { input }) => {
          const post: any = await faunaClient.query(
            q.Create(q.Collection("Post"), { data: input })
          );
          return {...post.data, id: post.ref.id};
        },
      }
    },
  },
})
server.start()

You use the Fauna driver functions in the resolver to add a new post to the database. If you are new to Fauna, the documentation for how to perform CRUD operations in Fauna.

Run wrangler dev in your terminal. Visit http://localhost:8787/ to interact with your GraphQL playground. Create a new GraphQL mutation to add post.

mutation AddPost {
  addPost(input: {
    title: "New Tittle"
    content: "Another Post"
  }) {
    id
    title
    content
  }
}

# Result 
{
  "data": {
    "addPost": {
      "id": "343885948385230924",
      "title": "New Tittle",
      "content": "Another Post"
    }
  }
}

Go back to the Fauna dashboard and notice that under Post collection a new record is generated.

Update and delete post (Mutation)

Similarly, you can create update and delete resolvers to update and delete posts. Make the following changes to your code.

// ...
const server = createServer({
  schema: {
    typeDefs: /* GraphQL */ `
      // ...
    `,
    resolvers: {
      Query: {
      },
      Mutation: {
        addPost: async (parent, { input }) => {
          const post: any = await faunaClient.query(
            q.Create(q.Collection("Post"), { data: input })
          );
          return {...post.data, id: post.ref.id};
        },
        deletePost: async (_, { id }) => {
          await faunaClient.query(
            q.Delete(q.Ref(q.Collection("Post"), id))
          );
          return true;
        },
        updatePost: async (_, { id, input }) => {
          const post: any = await faunaClient.query(
            q.Update(q.Ref(q.Collection("Post"), id), { data: input })
          );
          return {...post.data, id};
        }
      }
    },
  },
})
server.start()

Run updatePost and deletePost mutation from your GraphQL playground to make sure everything is working as expected.

Get post and post list (Query)

Next, add the listPost and getPost resolver functions under Query.

import { createServer } from '@graphql-yoga/common'
  import { faunaClient, q } from './db'

  const server = createServer({
    schema: {
      typeDefs: /* GraphQL */ `
        //...
      `,
      resolvers: {
        Query: {
          getPost: async (_, { id }) => {
            const post: any = await faunaClient.query(
              q.Get(q.Ref(q.Collection("Post"), id))
            );
            return {...post.data, id};
          },
          listPosts: async () => {
            const posts: any = await faunaClient.query(
              q.Map(
                q.Paginate(q.Documents(q.Collection('Post'))),
                q.Lambda((x: any) => q.Get(x))
              )
            );
            return posts.data.map((post: any) => ({...post.data, id: post.ref.id}));
          }
        },
        Mutation: {
          // ...
        }
      },
    },
  })
  server.start()

In the GraphQL playground try out the queries and make sure that everything is working as expected.

Implementing subscription

Cloudflare Workers support Streams API. Using this Streams API, you can send server-sent events (SSE) to your client. A client subscribes to the server in real-time through SSE.

Fauna also has a streaming API to listen to real-time database changes. Combining these two allows you to implement a complete serverless GraphQL subscription solution on edge.
Keep in mind that there is no WebSocket involved in this implementation.

Add a new subscription function called onPostChange to your resolvers field, as follows.


let stream: any = null;

const server = createServer({
  schema: {
    typeDefs: /* GraphQL */ `
    `,
    resolvers: {
      Query: {
                // ...
      },
      Mutation: {
        // ...
      },
      Subscription: {
        onPostChange: {
          subscribe: async function* (_, { id }) {
            let currentSnap: any;
            let newVersion: any;
            const docRef = q.Ref(q.Collection('Post'), id);
            if(!stream) {
              stream = faunaClient.stream.document(docRef).on('snapshot', (snapshot: any) => {
                currentSnap = {
                  ts: snapshot.ts,
                  data: snapshot.data
                }
              }).start();
            } 
            stream.on('version', (version: any) => {
              newVersion = {
                ts: version.document.ts,
                data: version.document.data
              }
            });

            let subscriptionTime = 0; // Terminate Subscription after 1000 seconds

            while (subscriptionTime < 150) {
              await new Promise(resolve => setTimeout(resolve, 2000, null));
              subscriptionTime++;
              if(newVersion && newVersion.ts !== currentSnap.ts) {
                currentSnap = newVersion;
                subscriptionTime = 0; // Reset Subscription Time
                yield { onPostChange: { ...newVersion.data, id} }
              }
            }
            await new Promise(resolve => setTimeout(resolve, 1000, null));
            stream.close();
            yield { onPostChange: 'Disconnected' }
          }
        },
      },
    },
  },
})
server.start()

In the previous code snippet you use Fauna’s document streaming feature to subscribe to a database record. Whenever the record is updated or deleted, the Fauna client will be notified in real time. You can then emit this event to your GraphQL resolver.

Notice that this is an SSE implementation. This is why we have an iterator function that keeps track of a heartbeat (every two seconds) in the server. After five minutes you close the connection.

It is very important to close the connection to Fauna stream after a period of time, otherwise, you will be charged for the running connection.

Subscriptions can also be done through a Pub/Sub implementation. That way, you don’t have to consistently listen to database changes. Depending on your use case, a Pub/Sub layer might be more economical. However, Cloudflare Pub/Sub is currently in closed beta. Once this feature is generally available, we will create another tutorial using Cloudflare Pub/Sub and Fauna.

Where to go from here

Got questions about Fauna, Cloudflare, serverless, or GraphQL? Feel free reach out to us and ask questions in our forum or on our Discord.

Building scalable solutions with Apollo Federation and Fauna

Shadid Haque — Wed, 24 Aug 2022 16:21:00 +0000

This article explains how you can reduce complexity, develop features faster, and make your engineering teams more efficient using Apollo Federation and Fauna.

Apollo Federation is designed for implementing GraphQL in a microservice architecture. On the other hand, Fauna is a distributed serverless document-relational database delivered as an API. Fauna's native GraphQL API allows engineers to directly integrate it as a subgraph in Apollo Federation or use it as a database for individual microservices.

Apollo Federation combined with Fauna improves developer experience and increases productivity, solving the problem of schema stitching and addressing pain points such as coordination, separation of concerns, and brittle gateway code.

Challenges of modern application development and how to solve it

Many organizations find it challenging to deliver products and features with velocity. More and more organizations are investing massive amounts of resources to make their infrastructure resilient and flexible. Yet, scattered microservices, versioned REST endpoints, and complex database management systems result in slow delivery of features/product development.

GraphQL revolutionized the way organizations build applications. GraphQL, open-sourced by Facebook in 2015, helps engineering teams integrate their array of REST APIs and microservices into a unified schema that frontend developers can query, fetching only the data required to power an experience while being agnostic to the data source. Traditional REST APIs lead to under or over-fetching data.

Fast forward to 2022, and the complexity of modern APIs and applications has grown exponentially. Running everything through a single GraphQL server (read: monolith) with multiple teams rapidly contributing changes creates a bottleneck.

What if there was a way for each microservice to have its own graph that could be seamlessly composed into a supergraph? Enter Apollo Federation. With Apollo Federation, you can build a declarative, modular GraphQL architecture.

A supergraph is composed of smaller graphs called subgraphs, each with its own schema. Teams can evolve their subgraphs independently, and their changes will be automatically rolled into the overall supergraph, allowing them to deliver autonomously and incrementally.

Your application data layer (database) is a critical part of your application architecture. Fauna is a database designed with modern applications and GraphQL architecture in mind. Fauna can seamlessly integrate as a subgraph in your Apollo Federation supergraph.

Moreover, Fauna's custom GraphQL resolvers gives the engineers the ability to turn the database into a fully-fledged microservice. You can learn more about Fauna's GraphQL capabilities in the official docs.

Fauna is a database that gives you the ability to form your data from a declarative GraphQL schema while giving you the complete freedom to implement your resolver functions if you have to. The document-relation model of Fauna also makes it flexible for evolving applications. Fauna combines the simplicity of NoSQL, without sacrificing the ability to model complex relationships.

Fauna’s flexibility and GraphQL native API combined with Apollo Federation allows engineers to build robust solutions faster with more flexibility and less code.

Let’s walk through a few different approaches for how to integrate Apollo Federation and Fauna, depending on your needs and existing architecture.

Application architecture

Microservices and Fauna (Traditional subgraphs)

Having a separate database for each of your microservices is a common practice. You can use the Fauna database as your data layer in each microservices. Your services can connect to Fauna with any of the supported drivers.

This is a traditional microservice-based architecture. Each microservice is a subgraph in this case, and you compose these subgraphs into a supergraph using Apollo Federation.

Using Fauna-hosted GraphQL endpoints as subgraphs

Suppose your product is primarily frontend focused (Jamstack or serverless-native) and you are not ready to commit to multiple microservices yet but want separation of concerns in the data layer. In that case, this architecture pattern is for you.

In this pattern, you will ideally have three Fauna databases. Each database has its GraphQL schema. Each of these schemas is then combined into a Federation supergraph.

Hybrid model

In the hybrid model, you combine the two approaches discussed earlier. This is ideal when you are rapidly scaling your product and solutions.

Getting started with Fauna and Apollo Federation 2

Let’s create a federated e-commerce solution using Apollo Federation and Fauna. In this example app, we will combine a Fauna GraphQL subgraph that includes information about shops and products with one from Apollo that provides locations.

You can find the complete code in this repository.

Configure Fauna

Head over to dashboard.fauna.com and create a new database. Create a new GraphQL schema. You can use the following schema.

type Shop {
  name: String!
  description: String!
  coverImg: String!
  products: [Product]! @relation
  ownerID: String!
}

type Product {
  name: String!
  description: String!
  price: Float!
  category: String!
  imageUrl: String
  shop: Shop! @relation
}

type Query {
  products: [Product]
}

Upload the schema to your Fauna database. Select the GraphQL option in your Fauna dashboard then select import schema.

Once you upload the schema your collections in your database will be generated.

Add Records to Your Collection

Next, create some records in Fauna using the GraphQL playground. Run the following mutation to create a new product.

mutation CreateProduct {
  createProduct(
   data: {
    name: "fender stratocaster",
    description: "The world's greatest electric guitar",
    price: 1255,
    imageUrl: "guitar.png",
    category: "Music",
  }
 ) {
   _id
   name
 }
}

Next, create a Shop by running the following mutation.

mutation CreateShop {
  createShop(
   data: {
    ownerID: "1223302094",
    name: "Scarborough Music and Records",
    description: "some music shop",
    coverImg: "example-shop.png"
   }
  ) { _id }
}

Integrate Fauna and Apollo

To integrate Fauna as a subgraph with Apollo Federation, you need the full GraphQL SDL from Fauna. You can download this from the Fauna GraphQL playground. Select schema from the playground and download the schema.

Next, navigate to security section and generate a new server key for your database.

Setting up supergraph

Create a new supergraph-config.yaml file. This file will contain the supergraph specification as well as subgraphs url and schema. Add the following code to your supergraph-config.yaml file.

federation_version: 2
subgraphs:
  fauna:
   routing_url: https://graphql.fauna.com/graphql`
   schema:
     file: ./fauna.graphql
 locations:
   routing_url: https://flyby-locations-sub.herokuapp.com/
   schema:
     subgraph_url: https://flyby-locations-sub.herokuapp.com/

Here, we are federating two subgraphs — one being Fauna and the other is a sample microservice from Apollo.

Run the following command to generate the supergraph

$ rover supergraph compose --config ./supergraph-config.yaml > supergraph.graphql

The previous command composes a supergraph schema from the specified subgraphs. The supergraph schema is saved in the supegraph.graphql file.

Local composition service with Apollo Gateway and Federation

Next, you create a local composition service. Create a new file called index.js and add the following code. This service acts as the gateway to your supergraph.

const { ApolloServer } = require('apollo-server');
const { ApolloGateway, RemoteGraphQLDataSource } = require('@apollo/gateway');
const { readFileSync } = require('fs');

const supergraphSdl = readFileSync('./supergraph.graphql').toString();

class AuthenticatedDataSource extends RemoteGraphQLDataSource {
  willSendRequest({ request, context }) {
    request.http.headers.set('Authorization', context.token);
  }
}

const gateway = new ApolloGateway({
  supergraphSdl,
  buildService({ name, url }) {
    return new AuthenticatedDataSource({ url });
  },
});

const server = new ApolloServer({
  gateway,
  subscriptions: false,
  context: ({ req }) => {
    const token = req.headers.authorization || '';
    return { token };
  },
});

server.listen().then(({ url }) => {
  console.log(`🚀 Gateway ready at ${url}`);
}).catch(err => {console.error(err)});

Make sure to install the required dependencies for this service. Run the following command to install the dependencies.

npm i apollo-server @apollo/gateway --save

Query your federated subgraph

Run the service with node index.js command. Navigate to http://localhost:4000/ in your browser. It will open up Apollo Studio.

You now have a federated graph up and running that uses Fauna as a subgraph. Run the following composed query to make sure everything is working correctly.

query FindProductByID($findProductByIdId: ID!) {
  findProductByID(id: $findProductByIdId) {
    _id
    name
    description
  }
  locations {
    id
    name
  }
 }

In the Variables field you will need to supply a product ID from your Fauna DB:

Once you’ve supplied your variables, make sure you pass in the authorization header. Use the Fauna secret key as the token for our authorization header.Your authorization header should be the following format Bearer fnAEukxxx:

The previous query returns a composed response from Fauna and locations microservice.

If you’d like to continue exploring federation with the subgraphs we created above, you can find the code for this example in this repository.

Conclusion

Combining Apollo Federation and Fauna allows engineers to be flexible, anticipate future API changes, and implement them easily without "breaking changes". In modern application development, engineering time is the most valuable resource. Apollo Federation and Fauna both focus on developer experience, helping engineers to be more productive and ship features faster at scale.

Have questions about using Fauna with Apollo Federation? Join the Fauna community channels and we would be happy to answer any questions you have. If you want to learn more about the Apollo Community forum. For more reference architecture and code examples with Fauna and Apollo, follow fauna labs on github.

Comparing databases for Vercel and Netlify

Shadid Haque — Fri, 12 Aug 2022 20:24:12 +0000

Vercel and Netlify make it easy for developers to build and ship full-stack serverless applications. Both platforms provide features such as serverless edge functions, CDN (content delivery network), build tools, analytics, etc. While building full-stack applications on these platforms, developers have quite a few options for their data layer (database). This article compares the databases commonly used with Vercel and Netlify, so you can make an informed decision when picking a new database for your project.

PlanetScale

PlanetScale is an easy to use MySQL-compatible serverless database. MySQL is easy to get up and running. However, you have to scale up your MySQL instances as your database grows. Database sharding is a common technique that is used to scale MySQL.

Sharding is a scale-out technique in which database tables are partitioned and each partition is hosted on its own RDBMS server. In the case of MySQL, this means that each node is its own MySQL RDBMS, with its own set of data partitions.

PlanetScale uses Vitess sharding to scale up the databases. While Vitess makes it easier to administer database shards, PlanetScale doesn’t actually shard databases automatically, so this is something that needs to be engineered on your own. If you wish to scale your database in a distributed manner – either now or in the future – you need to take this limitation into consideration.

Shards can be complicated to get right. You occasionally have to worry about splitting bits which is complicated. Joins across shards are not easily doable. If you need cross-shard joining, you probably need a data warehouse.

At the moment, PlanetScale appears to provide limited functionality and has focused its key differentiator around its unique branching feature for schema migration. However, some core database features seem to be missing, such as support for foreign key constraints. It also does not have a GraphQL API, and does not have any streaming capabilities.

PlanetScale with edge architecture

One of the primary appeals of Vercel and Netlify is that they have serverless edge support. Edge offering allows you to keep your serverless functions closer to your users.

For instance, let's assume you have a Next.js application deployed on Vercel. A user from Spain tries to access your application. The application will then be served from the nearest location to Spain. Vercel distributes your application worldwide, so your users don't face the added latency.

However, if your database is not distributed, then your edge functions will not be as performant. For instance, if your database server is located in Ney York, USA, and your compute function is in Lisbon, Portugal, it will add a significant delay. Ideally, you need a database that is also distributed globally to reduce latency.

To learn more about edge vs serverless performance and what role databases play in the architecture refer to this video.

Since PlanetScale is not globally distributed, it might have some performance drawbacks. However, PlanetScale does offer multiple regions, and you can manually architect your application to fetch data from the nearest region.

If you are building an application that requires very low latency and strong consistency across multiple regions, PlanetScale might not be a solid choice.

Pros	Cons
MySQL compatible (easy for MySQL users)	Not a distributed database
Scales horizontally using Vitess sharding	Doesn’t shard automatically
Branching mechanism	No foreign key constraints
Relational data modeling	No realtime API
	No GraphQL support out of box

MongoDB (MongoDB Atlas)

Another popular choice is MongoDB. MongoDB is a NoSQL database with optional schemas. MongoDB is flexible and suitable for both structured and unstructured data. Mongo Atlas provides managed instances of the Mongo database. Mongo saves data in JSON format.

Assessing the performance of two different database systems is very difficult since both systems approach the task of data storage and retrieval in entirely different ways. MongoDB is also optimized for write performance. Due to its document-oriented nature and prioritizing speed over transaction safety, MongoDB usually outperforms traditional relational databases.

Unlike a relational database, MongoDB doesn’t support foreign key joins. Because of this modeling your data relationships can be tricky with Mongo. There are other ways to get around this problem in MongoDB. Review this thread to learn more about data modeling in MongoDB.

You create indexes in MongoDB to execute queries and access data efficiently. Without indexes, you must perform a collection scan in MongoDB. If your indexing is incorrectly implemented or has any discrepancies or other errors, it negatively impacts the speed and performance of MongoDB.

MongoDB Atlas supports GraphQL. You can also use change streams to implement real time functionality.

One limitation MongoDB is that it is not transactional. If your application requires transactional data MongoDB might not be the right fit. Incorrect or complex indexing coupled with lack of structured relationship can lead to data corruption in MongoDB if not careful.

MongoDB Atlas with edge architecture

MongoDB Atlas offers serverless instances of MongoDB; however this is not distributed. Meanwhile, the managed service (provisioned) Atlas offering support global replication of your data out of the box.

If you want to use your edge architecture to its full potential, you have to replicate data manually and have your database servers located close to your edge servers. Again this can be challenging.

he provisioned version of MongoDB Atlas also requires TCP/IP to connect with edge functions, which is incongruent with edge/serverless functions that are ephemeral in nature (as you’ll have to maintain connection pools with MongoDB even when the function isn’t being called.

If your application doesn't require a transactional database, out-of-box multi-region replication mechanism, and you don’t require complex RDBMS like data modeling, then go with MongoDB.

Pros	Cons
Flexible, easy to use query expression	Need to manage connection pooling
Reliable and easy to scale	Performance costs with distributed transactions
GraphQL support through MongoDB Atlas	Mongo has a hard limit of 16MB per document
More performant than any relational database	Atlas is (or can be) globally distributed, but you need to manage clusters and shard keys

Fauna

Fauna is a distributed document-relational database delivered as a cloud API. Fauna combines the best of both SQL and NoSQL. With Fauna, you get the flexibility of a NoSQL database and a SQL database's relational data modeling capabilities.

Fauna also has user-defined functions (UDFs) features which allow you to compute on your data and reduce the round trip to your database. Learn more about Fauna’s UDFs here.

On top of everything Fauna gives you a native GraphQL API to query your database. You get the ability to define your database as a GraphQL schema. You can update and merge the GraphQL schema as your application data evolves without breaking anything.

You also have the capability to build real-time applications with Fauna's streams API.

At its core, Fauna uses a proprietary query language called FQL (Fauna Query Language).

Tip

Fauna is natively integrated with Vercel and Netlify

These integrations make it easy to use Fauna as the database for your sites and functions.

Fauna and edge architectures

Fauna was built for modern applications. It requires zero configuration and management. Fauna is strongly consistent, serverless, and globally distributed. And because Fauna is globally distributed, your data is automatically replicated and placed closest to your edge servers.

Fauna is designed to take full advantage of the edge architecture and ensures that your users get the data from the nearest location. Fauna can be accessed through an HTTP call, and unlike MongoDB and some other NoSQL databases, it doesn’t directly rely on underlying TCP/IP for connection.

If you want the flexibility and performance of NoSQL with the reliability, data modeling, and strong consistency of SQL, then Fauna is the appropriate database for you.

Fauna's flexible document-relational data modeling (as it combines the best of SQL and NoSQL) makes it a smooth developer experience. Developers also have the option to choose either a driver or GraphQL API to query their database, which in turn makes developers more productive.

When you build on platforms like Netlify or Vercel, you focus on your code and delivering features fast without thinking about the servers, latency, or scaling up; these platforms take care of everything. Fauna follows the same philosophy. You spin up your database, connect your application, and everything else (i.e., scaling up, global distribution of data, region groups, replication) is taken care of by Fauna.

Pros	Cons
Document-relational database (Combines the flexibility and performance of NoSQL with the reliability of SQL)	Fauna Query Language is new and requires some time to get used to
Serverless and distributed
Auto scales (Pay per usage)
Allows compute on data with UDFs
Strong consistency
GraphQL support
Real time API support with streams

Final words

Platforms like Vercel and Netlify enable developers to quickly build applications and solutions while simultaneously improving the developer experience. These platforms abstract away server management, scaling, and infrastructure so you and your engineering team can focus on shipping features and writing code. The data layer is critical in developing, launching, and scaling a successful product, so why not choose a database that is also built with the philosophy of serverless edge architecture? You can signup for a free instance (no credit card required) of Fauna here and start building today. I hope this article helps you make an informed decision on your choice of a database while building with Vercel or Netlify.

Edge computing reference architectures

Fauna — Thu, 04 Aug 2022 15:27:10 +0000

Cloud computing has transformed how businesses conduct their work, but many face challenges regarding data protection (residency and sovereignty), network latency and throughput, and industrial integration. On-premises infrastructure, especially when it’s tightly integrated with edge computing services provided by various cloud platforms such as AWS Lambda@Edge, Cloudflare Workers, Fastly Compute@Edge, etc., can be helpful in solving these problems. But that’s just part of the edge computing puzzle.

Cloud data centers and on-premises infrastructures form the near edge of the network. The other part of the network can be considered the far edge. Many businesses need the last mile coverage of the cloud to ensure that the cloud connects and integrates directly with controllers, sensors, and far-end devices. Currently, this workload is handled by specialized programmable logic controllers (PLCs) and human-machine interfaces (HMIs). These devices work great but depend on proprietary operating systems and are costly to acquire, maintain, and upgrade.

The following table lists the different parts of an edge network and their usual distance from the devices on the far edge of the network, along with the network latency from applications to those devices:

The cloud (regional and zonal data centers) — is 10+ km away from edge devices and has a latency of over 10 ms
Near edge (on-prem data centers) — is between 1 to 10 km away from the edge devices and has a latency from 1 to 10 ms
Far edge L1 (gateways, controllers) — is between 10 m to 1 km with a latency of somewhere between 0.1 ms and 1 ms
Far edge L2 (sensors, devices) — is in the 10 m range of the edge devices and has sub-millisecond latency

This article will take you through some of the edge computing reference architectures proposed by different organizations, researchers, and technologists so that you have a better understanding of the solutions that edge computing offers.

Why do you need edge computing?

Content delivery networks are based on the idea that to provide a great user experience, you need to deliver the content as quickly as possible. You can only do that by physically bridging the network latency gap to place the content closer to your end users. Fast-access storage solves this problem. Big companies in specific domains like content delivery and mobile networking already use the edge network for storage, if not for computing.

But many businesses can’t solve their problems with delivery networks; they need computing power without the latency lag of the cloud. They would rather have something closer to the edge in addition to the cloud.

Edge computing use cases

Take vehicle autonomy, for instance. An average car without autonomy has over fifty sensors that help the vehicle’s critical and non-critical functions. Traditionally these sensors were lightweight, but with autonomous car cameras, LiDAR sensors, and others, there’s an ever-increasing demand for computing power on board. You can’t rely on APIs and the network for processing specific information in the operations of a critical machine like a car.
Another example is surveillance. Businesses might use AI and ML applications to process a camera feed in real time. Offloading that amount of raw data to the cloud for processing might not be the wisest decision, as it will need tremendous network bandwidth and computing power to move data back and forth. To save time spent moving data around, you need more computing power at the source, and you can send the workloads that can wait to the cloud.
The same principles apply to many specialized industries, such as aerospace, live entertainment, mass transportation, and manufacturing. This is why the edge is becoming more relevant by the day. Other applications of edge computing include:

Decoupling automation workloads from specialized hardware like PLCs and HMIs by using open source software like FreeRTOS or OPENRTOS and running the workloads on containers
Offloading custom data-intensive critical workloads to the near edge network
Minimizing your maintenance and downtime with a redundant computing system for your business’s critical systems, which adds real value to your business continuity planning (BCP) and disaster recovery planning (DRP) efforts
Optimizing business processing with the help of faster response times from the computing power on the network’s edge

Interested in understanding edge computing beyond its use in IoT?

Fauna can help you build faster, more performant edge applications. Since Fauna is distributed dy default, you can reduce latency for your apps with a close-proximity database for your edge nodes. Easily integrate with Cloudlfare workers and Fastly Compute@Edge platforms out of the box.

Edge computing reference architectures

To properly design and implement edge computing networks, businesses rely on reference architectures. The following are some reasons why they’re an invaluable resource.

Design considerations

Businesses are still finding different ways to exploit the power of edge computing, and edge computing reference architectures help them understand systems that could potentially work for them. Not only are these reference architecture patterns well researched and thought out by industry experts, but they try to encapsulate the key features that apply to various businesses.

You can use reference architectures for inspiration as well as innovate on top of them. There are several factors and design considerations that you should take into account while architecting an edge computing system for your business:

Time sensitivity: With edge computing, critical business functions don’t have to wait for workloads to be offloaded to the cloud for processing
Network bandwidth: Bandwidth is limited; you don’t want to choke the network by blocking other important operations
Security and privacy: Data sovereignty and data residency, both for security and privacy, can be enabled by the edge
Operational continuity: During network disruptions, you can offload the extremely critical operations of your business to the edge

Implementation outcomes

The outcomes and benefits of architecting edge computing systems align strongly with the abovementioned factors. With a successfully deployed edge computing system, your business gains the following core benefits:

Significantly lower latency and higher network bandwidth by utilizing the proximity from the request origin to the computing system
Effective network use by filtering and reducing farther data transfer to on-premises or cloud infrastructure
Better enforcement of security and privacy standards, enabling data sovereignty and data residency

Development and deployment plans

When developing for the edge, businesses need to ensure that they don’t end up creating the same dependencies and bottlenecks they experienced when using only on-premises data centers or the cloud.

An excellent way to do this is to research, identify, and pick the right open standards and technologies to help you get the architecture you want. Possible open standards include those for APIs, documentation, virtualization patterns, deployment pipelines, and code promotion strategies.

Edge computing reference architectures that use Fauna

There are many reference architectures, some of which heavily borrow from one another. However, each has a unique approach and feature set. Fauna, as a distributed general-purpose database, interacts well with all of them in different ways.

The following are details on four reference architectures, their relevance and innovation, and how to adopt them with Fauna.

Industrial internet reference architecture

The Industrial Internet Reference Architecture (IIRA) was first developed in 2015 by the Industry IoT Consortium (IIC). This was a collaborative effort from a range of companies, including Bosch and Boeing in manufacturing; Samsung and Huawei in consumer electronics; and IBM, Intel, Microsoft, SAP, Oracle, and Cisco in the software industry.

Think of IIRA as having four different viewpoints: business, functional, usage, and implementation. The system is divided into enterprise, platform, and edge. You have low-level devices in factories or appliances at the edge, connected with the platform tier using low-level APIs. Data collected from these devices flows to data and analytics services and operations users.

The data generated by the edge, along with the data and insights gathered and processed by the platform tier, is consumed by the enterprise tier on two levels: the business domain and app domain. The data in the business domain lands in other systems, such as ERP, and the data in the app domain flows further based on business logic and rules to the business users in the form of mobile apps, browser-based frontend applications, or business intelligence tools.

The following diagram shows the three-tiered approach to IIoT system architecture using IIRA:

Fauna’s distributed architecture allows it to sit across multiple layers of the network while simultaneously guaranteeing you the privileges of a flexible document-oriented database and ACID transactions. The data and analytics services and platforms can heavily interact with Fauna, and the operations services and platform will do the same. In this context, Fauna sits mainly on the platform tier but interacts with the edge tier, depending on whether the devices are integrated with the platform tier using a virtualization layer.

FAR-EDGE reference architecture

Aside from the previously mentioned features of reference architectures, the FAR-EDGE Reference Architecture brings many new things to the table. It explicitly offers a separate logical and ledger layer that handles processes and rules across the distributed computing system by using the sheer power and spread of the system. Like most edge computing reference architectures, FAR-EDGE also concentrates on saving bandwidth and storage, enabling ultra-low latency, proximity processing, and enhanced scalability by exploiting the distributed nature of the edge-cloud hybrid architecture.

Compare the idea of FAR-EDGE with some of the Web3 architectures built on top of distributed frameworks like the blockchain. Fauna, with its capability to provide edge functions as an extension to the application infrastructure, can play a central role by servicing various layers and functions proposed in this reference architecture. Consider the functional view of the FAR-EDGE reference architecture below:

It’s worth reiterating that Fauna is a distributed document-relational database, which is ideal for businesses with a variety of software applications and hardware devices. In this functional view, Fauna will sit and interact not only with applications and cloud services but also with ledger and edge processes at the ledger and gateway levels. You can enable this by using well-documented APIs and SDKs.

Edge computing reference architecture 2.0

The Edge Computing Consortium (ECC) proposed a model-driven reference architecture that attempts to solve the problems businesses face when they try to connect the digital and the physical worlds. Since the reference architecture is multidimensional, you can regard it from different viewpoints. Consider one of its layers, the Edge Computing Node layer, from a functional viewpoint.

The Edge Computing Reference Architecture 2.0 is a multidimensional architecture composed of the following components from a high-level view: smart services, a service fabric, a Connectivity and Computing Fabric (CCF), and Edge Computing Nodes (ECNs).

As mentioned earlier, edge computing thrives when open standards are followed and hardware devices are decoupled from specifically designed hardware. This problem has long been solved for specific web and desktop applications via virtualization. The Edge Computing Reference Architecture also suggests you create an Edge Virtualization Function (EVF) layer that handles connectivity and data streaming and collection, along with access policies and security.

The following diagram of the Edge Computing Reference Architecture 2.0 shows the functional view of an Edge Computing Node (ECN):

Fauna is more than capable of taking care of the universal services, such as streaming and time series data, along with any industry-oriented services that might need integration.

Conclusion

Edge computing reference architectures can vastly simplify the design and usage of edge computing systems, and Fauna can help at different levels in all these architectures. Fauna is a natively serverless, distributed transactional database that allows for flexible schemas and comes with a well-designed data API for modern applications.

With Fauna in the cloud and at the edge, you can store and retrieve your data at ultra-low latencies while being closer to the source than ever before. With built-in data security, authentication, authorization, and attribute-based access control, Fauna simplifies the implementation of edge computing reference architectures and lets you concentrate more on your business.

Modernization of the database: DynamoDB to Fauna

Fauna — Mon, 25 Jul 2022 14:16:10 +0000

Serverless databases like Fauna fill a crucial role for modern applications. Given the amount of cloud-based traffic and varying workloads that organizations from startups to enterprises manage, serverless databases are the natural fit to the rest of your serverless architecture, adding seamless flexibility, scaling, and low latency for your applications.

DynamoDB has been one of the most popular serverless databases. In addition to coming from AWS, it has benefited from being one of the earliest offerings in this space. Its popularity is hard to dispute, backed up by its proven ability to effectively handle traffic spikes and workload variations without overloading infrastructure or racking up unnecessary costs during periods of low traffic. Though DynamoDB has much to offer, Fauna is a much newer offering that comes with a host of powerful and unique features that enhance the serverless experience for organizations of all sizes.

In this article, we’ll learn about serverless databases, compare key differences between DynamoDB and Fauna, and provide insight on which one to choose for your next big project.

What makes a good serverless database?

A great serverless database should offer the following values:

Lower operation costs: One of the main reasons for choosing software is cost. Traditionally, organizations need to budget for the continuous management of their database(s) and infrastructure, and not just for application development. Serverless architectures promise “zero ops,” which eliminate these management concerns. As headcount makes up your highest spend, leaning out on operations overhead ultimately lowers your TCO (total cost of ownership).
Pay as you go: It should also offer a simple and transparent pricing model without any traps or unexpected costs. You should only have to pay for the resources you use, with billing being proportional to the amount of data stored and volume of transactions sent to the database.
High availability: There should be no maintenance windows or unplanned downtimes. Data and compute resources should be automatically distributed and replicated, offering high durability and resiliency to region/zone outages. The more these attributes are provided to the operator without any additional configuration, the better.
Low latency: Ensuring a good user experience is key to user engagement, ultimately impacting how successful your products are with customers. Serverless architectures should not trade off the ability to scale elastically with cold start-up delays or increased latency due to speed-of-light constraints. A great serverless database offering should be always-on and instantly scale; And provide multiple region choices so that it can be accessed as close to your compute resources as possible.

Why should you choose Fauna?

Serverless databases are critical components for managing unpredictable and rapidly changing workloads, essentially following a pay-as-you-use model. They are also a good fit for companies with a small workforce, enabling them to consume compute workloads and infrastructure without manual overhead. Using a serverless database allows you to simplify database operations and eliminate problems common with traditional databases, such as maintenance, upgrades and patching, and cost of operations.

Fauna is accessed as an API and there is nothing to install, maintain, or operate. You can quickly deploy a database in three button clicks, start coding, and immediately connect to the database. You can scale databases without any limitations and create an unlimited number of databases. It is chock full of cloud-native features: login with your GitHub account; integrate with third-party services like Auth0, Netlify, and Vercel; has built-in streaming, user authentication and fine-grain authorization. It supports user-defined functions – similar to stored procedures – which help eliminate redundant code, maintain Fauna Query Language (FQL) consistency, and remove code duplication. And it has a native GraphQL API, allowing organizations adopting GraphQL to quickly get up and running with a data source in minutes.

Fauna vs. DynamoDB

Fauna offers numerous features that distinguish it from DynamoDB and similar serverless databases. The following is a comparison between Fauna and DynamoDB and the components that each offers.

Geo-distribution

Businesses with global audiences need to ensure fast response and high performance for customers in multiple regions across the globe. This requires a database that allows data to be replicated globally and served from specified locations — the closer the proximity of the data, the better the experience and performance.

By default, DynamoDB is replicated across multiple availability zones in a single region. But with the “global tables” feature, DynamoDB provides a fully managed solution for deploying a multi-region, multi-active database. You specify the regions you want the database to replicate, and DynamoDB will propagate all data changes to them. Besides reducing read latency, multi-region replication ensures that single region failures do not result in any outages. Using global tables comes with quite a few caveats. Your application can read and write to any replica table, but if it requires strongly consistent reads, it must perform all its strongly consistent reads and writes against the same region. AWS documentation states that any newly written item is propagated to all replica tables within a second — not an insignificant latency — and that ACID guarantees only apply within the AWS Region where the write was originally made. When an AWS region suffers an outage or degradation, your application can redirect to a healthy region but it isn’t automatic: AWS documentation prompts developers to apply custom business logic to determine when to redirect requests to other regions. Global tables are more costly to operate. Your costs essentially multiply several times with the addition of every replica. In addition, you must also use the higher priced on-demand capacity mode, though this can be mitigated as they allow the use of provisioned capacity with auto-scaling.

Unlike DynamoDB, Fauna is multi-region-distributed by default — every database you create is replicated and distributed across geographic regions. In addition, Fauna also provides the Region Groups functionality, which allows developers to control which region their data resides in. Other than selecting the Region Group, there is no additional configuration and no custom business logic to implement. When your application makes a request to the Fauna API, it is automatically routed to the region closest to it. Reads are immediately served out of that region. And writes are automatically propagated to the other regions’ replicas automatically. Fauna publicly publishes its “internal latency” – separated into reads and writes – on status.fauna.com. There is no separate pricing scheme for single- vs multi-region with Fauna since your databases are always multi-region – providing you straightforward and transparent pricing.

Transactionality

While you want to ensure the real-time availability of data, your data needs to be consistent and accurate. Not all serverless databases can provide this.

DynamoDB, for instance, offers serializable read and write transactions but is only ACID-compliant within the region where the transaction occurs. This is a problem for multi-region deployments, because it can lead to errors during transactions that run simultaneously in different regions.

One of Fauna’s most touted innovations is its approach to isolation and distribution, which is derived from Calvin, an algorithm for achieving distributed consistency at scale. As an oversimplified explanation to what it entails, queries are handled before they interact with the storage system, while deterministic ordering – eschewing the need for expensive atomic-clocks – is employed to provide serializability, ensuring that no two concurrent transactions can commit to the same data. Through Fauna’s Calvin implementation, it is one of the few serverless databases that offers ACID guarantees while being globally distributed at the same time. Transactions are globally distributed, ACID-compliant, and serializable, with no additional configuration or separate pricing scheme.

Streaming

Real-time applications require an endpoint that enables persistent connectivity, allowing the server to push information to the client requesting it.

With DynamoDB, you’ll need to combine several services in order to implement streaming. The first part is setting up a DynamoDB Stream: DynamoDB integrates with AWS Lambda, where you set up triggers that respond to changes (DynamoDB Stream events) in your table(s). Thus, there’s a bit of configuration, and some coding involved in writing the Lambda. From there, you need to implement the persistent endpoint. AWS can accomplish this in a number of ways, and your choices are using either an EC2, API Gateway Websocket API, or AppSync (via GraphQL Streaming). Regardless, these are additional components that need to be implemented and/or configured and maintained. On top of that are the costs involved in running the additional services.

In contrast, Fauna’s API endpoints support streaming right out of the box. In order to consume a stream, all you need to do is write some lines of code on the client side of your application using a driver in your language of choice. For example, here are the instructions for instantiating a stream client in javascript. Fauna offers two kinds of streaming: document streaming and set streaming. With document streaming, an event notification is sent to the subscriber whenever a document is created, updated, or deleted. With set streaming, events are sent whenever one or more documents enter or exit the set while doing a create, delete, or update.

Security

Because they are accessed as APIs, great serverless databases must provide robust security models, so that developers do not have to build their own authentication and authorization.

Access to DynamoDB is based on AWS’s highly robust Identity and Access Management (IAM), in which granular permissions are applicable to the DynamoDB resources (tables, indexes, and stream). You can also specify IAM policies that grant access to perform specific actions (e.g, read, add, update, batch update, etc.) on specific resources for specific users.

Fauna supports both API keys and identity-based access tokens, and supports integration with external identity providers that support the OAuth flow (such as Auth0, Okta, OneLogin, etc.). Keys and tokens inherit the permissions of roles upon which they’re assigned. Roles are configured with granular permissions to access every single resource in the database, including collections (tables), indexes, user-defined functions, other roles, and schema. Fauna also provides attribute-based access control (ABAC), allowing permissions to be dynamically assigned based on tokens’ identities’ attributes. With ABAC, you can define custom business logic, creating dynamic rules that control access to all resources, all the way down to specific documents in a collection.

Automated functionalities

In DynamoDB, a host of configuration options allow you to operate various capacity modes, single- vs. multi-region, types of indexes and index-partitioning strategy used, logging, and more. The tradeoff for this flexibility is more manual work and time.

Fauna, however, automates multiple functions for you, such as database provisioning, maintenance, sharding, scaling, and replication. The “zero ops” model of Fauna lets you focus on building the critical aspects of your applications without worrying about the complexity of a distributed architecture. This is especially useful if your business needs to carefully manage limited engineering resources while dealing with unpredictable workloads.

DynamoDB includes an automated backup feature. It charges for backups separately, and you need to select from on-demand or continuous backups. The backup is optional and by project.

In Fauna, you can schedule daily backups on any database in the form of snapshots of the entire data-tree. You can also configure a retention period for them. Databases can be “restored” (overridden) in place from any backup, or you can use backups to seed new databases. Fauna also supports temporality, allowing you to go back through history and query data at any arbitrary point in time. This is because in Fauna, data is stored as snapshots across time. You can use this feature to implement point-in-time recovery and targeted data repairs in your database.

GraphQL API support

GraphQL is designed to make API development faster and more flexible for software engineers. API developers use GraphQL to have a schema that will showcase all the possible data that the clients will want to query through a specific endpoint. Unlike most other serverless databases like DynamoDB and Upstash, Fauna provides a native GraphQL API for data access in addition to its query language, FQL.

If you are looking to launch your product in less time and expect a lot of changes on your API endpoint, then using a serverless database that supports GraphQL APIs will be the better choice for you.

💡 Check out our GraphQL workshop that introduces you to Fauna. In two hours, you’ll build an application in either Next.js or SvelteKit that has authentication, user-defined functions, customer resolvers, and more.

Scalability and pricing

With some serverless databases like DynamoDB, automatic scaling is an option but isn’t set by default. There are basically three modes in DynamoDB:

The default – called provisioned capacity – lets you set the volume and scale that you need up front. You then either manually monitor and manage these parameters or be automatically throttled once the load hits your predefined capacities. This mode is ideal when you have steady, predictable loads, and you need to stay within a narrow budget.

Provisioned capacity with auto-scaling is the same as provisioned capacity but also dynamically adjusts throughput capacity based on actual traffic, allowing your application to handle sudden bursts in traffic without being throttled. Pricing is a factor of the baseline capacity you’ve configured and your application's actual load (above that baseline) that your application experiences. It is important to note that because provisioned capacity (with or without auto-scaling) involves setting a baseline desired capacity, you are always paying for that capacity, even when your traffic is zero.

On-demand capacity is priced significantly higher (on a per-unit read/write/etc. basis) than provisioned capacity, but is truly serverless and will scale to zero (where you pay nothing) if no traffic is observed or as high as needed when bursts are experienced.

The one and only mode in Fauna is on-demand. It autoscales without any intervention from you and you don’t need to manage the settings of the desired throughput. There is no need to monitor the database to keep your database safe from being saturated. As such, pricing is much more straightforward and only a factor of how much data you store, how many transactions occurred, and how large (complex) the transactions/queries are. Comparison pricing between Fauna and DynamoDB is highly based on context and use-case and especially how you plan to use DynamoDB, given its flexibility in setting things up. We cover these topics in “Comparing Fauna and DynamoDB: Pricing and features.”

Conclusion

Serverless databases offer numerous benefits, including lower costs, less operational workload, faster time to market, and high scalability. When choosing a serverless database, be sure to select one that offers you maximum consistency, high performance, and low latency.

While there are many serverless databases to choose from, Fauna offers the largest range of features for your organization’s needs. Automated scaling, multi-region ACID-compliant transactions, and GraphQL support will help you optimize your workload and improve your product for users. Your developers can focus on improving your applications and core services while worrying less about managing the database or infrastructure. This leads to a shorter development time and faster application delivery.

Interested in learning more about Fauna? Reach out to our team or sign up and start using Fauna for free.