Forem: Fauna

Comparing Fauna and DynamoDB

Fauna — Wed, 24 May 2023 15:39:21 +0000

Fauna and DynamoDB are both serverless databases, but their design goals, architecture, and use cases are very different. In this post, I will overview both systems, discuss where they shine and where they don’t, and explain how various engineering and product decisions have created fundamentally different value propositions for database users.

DynamoDB’s design philosophy: Availability and predictability

AWS DynamoDB was developed in response to the success of Apache Cassandra. The Cassandra database was originally open sourced and abandoned by Facebook in 2008. My team at Twitter contributed extensively to it alongside the team from Rackspace that eventually became DataStax.

However, in an odd twist of history, Cassandra itself was inspired by a 2007 paper from Amazon about a different, internal database called Dynamo — an eventually-consistent key-value store that was used for high-availability shopping cart storage. Amazon cared a lot about shopping carts long before they had a web services business. Within Amazon, the Dynamo paper, and thus the roots of DynamoDB, predate any concept of offering a database product to external customers.

DynamoDB and Cassandra both focused on two things: high availability and low latency. To achieve this, their initial releases sacrificed everything else one might value from traditional operational databases like PostgreSQL: transactionality, database normalization, document modeling, indexes, foreign keys, and even a query planner. DynamoDB did improve on the original Dynamo architecture by making single-key writes serializable and dropping the baroque CRDT reconciliation scheme, and on Cassandra by having a somewhat more humane API.

DynamoDB’s architecture

DynamoDB’s architecture essentially puts a web server in front of a collection of B-tree partitions (think BDB databases) into which documents are consistently hashed. Documents are columnar, but do not have a schema.

Within a DynamoDB region, each data partition is replicated three times. Durability is guaranteed by requiring synchronous majority commits on writes. Consistency is only enforced within a single partition which, in practice, means a single document since partition boundaries can not be directly managed. Writes always go through a leader replica first; reads can come from any replica in eventually-consistent mode, or the leader replica in strongly consistent mode.

Although DynamoDB has recently added some new features like secondary indexes and multi-key transactions, their limitations reflect the iron law of DynamoDB: “everything is a table”.

Tables, of course, are tables.
Replication to other regions is implemented by creating additional tables that asynchronously apply changes from a per-replica, row-based changelog.
Secondary indexes are implemented by asynchronously projecting data into additional tables — they are not serializable and not transactional.
Transactionality is implemented via a multi-phase lock — presumably DynamoDB keeps a hidden lock table, which is directly reflected in the additional costs for transactionality. DynamoDB transactions are not ACID (they are neither isolated nor serializable) and cannot effectively substitute for relational transactions. Transaction state is not visible to replicas or even to secondary indexes within the same replica.

As you may predict from the above, the DynamoDB literature is absolutely packed with examples of “single-table design” using aggressive NoSQL-style denormalization. Using the more complex features is generally discouraged. DynamoDB’s pricing is also designed around eventually-consistent single tables, even though in replicated and indexed scenarios individual queries must often repeatedly interact with multiple tables. Furthermore, even though global tables were added a few years later, the overall pricing becomes significantly higher, while the same eventually-consistent data integrity compromise remains. designed around single-table, eventually-consistent usage, even though in replicated and indexed scenarios individual queries must interact with multiple tables, often multiple times.

Additional challenges lie in the query model itself. Unlike Fauna’s query language FQL or SQL, DynamoDB’s API does not support dependent reads or intra-query computation. In contrast, Fauna allows developers to encapsulate complex business logic in transactions without any consistency, latency, or availability penalty.

DynamoDB works best for the use cases for which it was originally designed — scenarios where data can be organized by hand to match a constrained set of predetermined query patterns; where low latency from a single region is enough; and where multi-document updates are the exception, not the rule. Examples of these restricted use cases include storing locks as a durable cache for different, less scalable database like an RDBMS, or for less-critical, transient data like the original shopping cart use case.

Fauna’s design philosophy: A productivity journey

Fauna, on the other hand, was inspired from our experience at Twitter delivering a global real-time consumer internet service and API. Our team has extensively used and contributed to MySQL, Cassandra, Memcache, Redis, and many other popular data systems. Rather than focus on helping people optimize workloads that are already at scale, we wanted to help people develop functionality quickly and scale it easily over time.

We wanted to make it possible for any development team to iterate on their application along the journey from small to large without having to become database experts and spend their time on caching, denormalization, replication, architectural rewrites, and everything else that distracts from building a successful software product.

Fauna’s architecture

To further this goal, Fauna uses a unique architecture that guarantees low latency and transactional consistency across all replicas and indexes even with global replication, and offers a unique query language that preserves key relational concepts like ACID transactions, foreign keys, unique constraints, and stored procedures, while also enabling modern non-relational concepts like document-oriented modeling and declarative procedural indexing.
If everything is a table in DynamoDB, in Fauna, everything is a transaction:

All queries are expressed as atomic transactions.
Transactions are made durable in a partitioned, replicated, strongly-consistent statement-based log.
Data replicas apply transaction statements from the log in deterministic order, guaranteeing ACID properties without additional coordination.
These properties apply to everything, including secondary indexes and other read and write transactions.
Read-only transactions achieve lower latency than writes by skipping the log, while remaining fully consistent with additional safeguards. Unlike DynamoDB, Fauna shines in the same areas the SQL RDBMS does: modeling messy real-world interaction patterns that start simply but must evolve and scale over time. Unlike SQL, Fauna’s API and security model is designed for the modern era of mobile, browser, edge, and serverless applications.

Like DynamoDB, and unlike the RDBMS, Fauna transparently manages operational concerns like replication, data consistency, and high availability. However, a major difference from DynamoDB is the scalability model. DynamoDB scales by predictively splitting and merging partitions based on observed throughput and storage capacity. By definition, this works well for predictable workloads, while being less ideal for unpredictable ones, because autoscaling changes take time.

Fauna, on the other hand, scales dynamically. As an API, all resources including compute and storage are potentially available to all users at any time. Similar to operating system multithreading, Fauna is continuously scheduling, running, and coorindating queries across all users of the service. Resource consumption is tracked and billed, and our team scales the capacity of each region in aggregate, not on a per-user basis.

Naturally, this design, and the related benefits, has a different cost structure than something like DynamoDB. For example, there is no way to create an unreplicated Fauna database or to disable transactions. Like DynamoDB, Fauna has metered pricing that scales with the resources your workload actually consumes. But unlike DynamoDB, you are not charged per low-level read and write operation, per replica, per index, because our base case is DynamoDB’s outlier case: the normalized, indexed data model, with the transactional, multi-region access pattern.

Higher levels of abstraction exist to deliver higher levels of productivity. Fauna offers a much higher level of abstraction than DynamoDB, and our pricing reflects that as well — it includes by default key characteristics that DynamoDB does not. At Fauna we want to provide a database with the highest possible level of abstraction that solves the use cases you would traditionally turn to a relational database for, so that you don’t have to worry about any of the low level concerns at all.

What is an API worth?

Most other databases aside from DynamoDB and Fauna are delivered as managed cloud infrastructure, and billed on a provisioned basis that directly reflects the vendor’s costs and those costs alone. Serverless infrastructure is relatively new — S3 is perhaps the first service with a serverless billing model to reach widespread adoption — and serverless databases are even newer. The serverless model in DynamoDB is a retrofit. It is essentially still a provisioned system with the addition of predictive autoscaling.

Instead, serverlessness to date has mainly been restricted to vertically-integrated, single-purpose APIs. These APIs have been monetized indirectly like Twitter, billed per-action like Twilio, or billed as a percentage of the value exchanged via the API between third parties — like Stripe.
Serverless infrastructure, as we all know, is actually made from servers. It has a more complex accounting challenge than vertically-integrated APIs, and is constrained by:

Variance in resource utilization per request
Variance in request volume over time
Variance in request locality
Underlying static costs

The multi-tenancy of serverless infrastructure creates a fundamentally better customer experience. Who wants to pay for capacity they aren’t using? Who wants to have their application degrade because they didn’t buy enough capacity in advance? It’s also a better vendor experience, since no vendor wants to waste infrastructure, and it can be more environmentally friendly.
However, the vendor’s aggregate price across all customers must cover the static infrastructure costs, which are tightly coupled and resistant to change. (As a practical matter, a vendor can’t upgrade and downgrade CPUs, memory, disks, and networks independently of either on demand, even when using managed cloud services.) The aggregate price must also correlate with the business value recognized, and it must be appropriately apportioned based on the realization of that value for each individual customer over time.

Compared to simply marking up the incremental cost of a server, this pricing problem is hard. Let’s discuss the solutions that DynamoDB and Fauna have found.

DynamoDB pricing

For DynamoDB’s base use case for which it was designed, its pricing is relatively clear and straightforward. However, if you add in usage of newer features like global replication, indexes, and transactions, the pricing becomes more opaque, and it can become very difficult to predict costs in advance.

Fauna pricing

Fauna’s pricing by default maps to the underlying architectural differences we mentioned above and reflects the enhanced capability that you can tap into with Fauna.
Some key differences include:

Compute - DynamoDB doesn’t support computation of any kind within a query, but Fauna does. Thus, Fauna charges separately for compute costs. In DynamoDB, since computation for any particular workload can’t be done in the database at all, it must be done application-side in a compute environment like AWS Lambda, which has its own cost.

Additional pricing dimensions - DynamoDB has additional pricing differences depending on specific regions, infrequent access, transaction isolation levels, and many other characteristics. It significantly complicates any cost and capacity planning exercise, which requires detailed and ongoing adjustments.

Powering a SaaS application

A good example of how these architectural differences play out is more evident if we consider something like a typical SaaS application, for example, a CRM. We have an accounts table with 20 secondary indexes defined for all the possible sort fields (DynamoDB’s maximum — Fauna has no limit). We also have an activity table with 10 indexes, and a users table with 5 indexes. Viewing just the default account screen queries 7 indexes and 25 documents. A typical activity update transactionally updates 3 documents at a time with 10 dependency checks and modifies all 35 indexes.

And of course, we have replicated this data globally to two additional regions in DynamoDB. We will also do consistency checks on all data returned from indexes.

In Fauna, we do not need to configure replication, or do any additional consistency checking. And we benefit greatly from Fauna’s index write coalescing. Thus just the database usage costs alone with Fauna, let alone the engineering savings involved, are less expensive than DynamoDB because Fauna’s architecture was designed to support these use cases from the beginning.

Even if we assume that Fauna will require multiple write operations per document because of all the indexes, the result does not materially change. Fauna’s query pattern could also be improved by using unique constraints instead of dependency reads, which would reduce costs further.

Using the right tool for the job

Surface level comparisons between DynamoDB and Fauna just because they are both “serverless” can result in false conclusions.

Because DynamoDB is designed for the simple use case rather than the complex, it can be more cost effective for those simple use cases. Even though core pricing for those simple use cases might be lower, you always have to consider TCO — for DynamoDB that means factoring in additional costs like manual partitioning and the eventually-consistent transactional behavior that reflect its roots as a lower-level system.

As sophistication grows — for example, if you configure a dozen indexes for a global deployment in DynamoDB — you will find your write and storage costs have multiplied by an order of magnitude compared to a single region, unindexed table. If you then change to make those writes transactional or start doing dependent reads, your costs increase even more. On the other hand, if you try to use Fauna as an application-adjacent durable cache at scale, you may find you are paying for data replication and transactional consistency that you don’t actually need.

It is more accurate to say that both DynamoDB and Fauna are great solutions that deliver on the promise of serverless when they are used for their correct purposes, and expensive when used incorrectly. This seems like a universal rule, but it actually isn’t. Most databases, even in the managed cloud, are disproportionately expensive for intermittent or variable workloads, which are prevalent with real-world workloads. This is the benefit of the serverless model: an order of magnitude less waste for both the customer and the vendor.

Conclusion

At Fauna, we recognize that DynamoDB has pushed the envelope in distributed cloud databases and we are grateful for that. We share the same greater mission. At the same time, we know we as an industry can do better than all existing databases for mission-critical operational workloads, whether key-value, document-oriented, or based on SQL.I hope this post has provided you with a clearer understanding of the motivations, architectures, and business value of both DynamoDB and Fauna. And further, that this understanding helps you make more informed decisions about which tool is right for which job.

Side-by-side comparison of serverless databases

Fauna — Thu, 29 Dec 2022 16:53:31 +0000

Serverless databases make it easy to build and scale your application because they abstract away the underlying infrastructure and automatically scale to meet the need of your application. A serverless database lets you focus on your code without worrying about capacity planning, infrastructure maintenance, and server management.

There are quite a few options for serverless databases, including established solutions like DynamoDB and newer offerings such as Mongo Atlas and CockroachDB serverless. This article compares commonly used serverless databases so you can make an informed decision when picking a new database for your project.

MongoDB (Mongo Atlas)

MongoDB is a widely used NoSQL database. MongoDB Atlas offers a provisioned managed service as well as a serverless offering. In this article we will compare the serverless offering.

MongoDB Atlas offers fully managed serverless instances of MongoDB. With Mongo Atlas, you don’t have to manage any infrastructure by yourself.

✅ Connection over HTTP

Mongo Atlas allows your application to connect to your database through an HTTP connection.

⚠️ Multi-region availability

By default, MongoDB Atlas is deployed in a single region. However, MongoDB Atlas can automatically replicate data across multiple servers for improved reliability and availability. It is partially automated, and you’ll need prior knowledge of sharding and replication.

⚠️ Schema design (Schema-less, non-relational)

MongoDB is flexible and suitable for both structured and unstructured data. Unlike a relational database, MongoDB doesn’t support foreign key joins. Because of this modeling, your data relationships can be tricky with Mongo. There are other ways to get around this problem in MongoDB. Review this thread to learn more about data modeling in MongoDB.

❌ No cold starts

If you have a large database then MongoDB Atlas clusters can have a long cold start time.

⚠️ ACID transactions

Multi-document transactions are not enabled by default in MongoDB or MongoDB Atlas clusters. However, you can configure MongoDB to support multi-document ACID transactions based on your use case.

DynamoDB

DynamoDB is a fully managed No-SQL database from Amazon. It supports key-value and document data structures. DynamoDB has a pay-as-you-go model, and it provides auto-scaling.

✅ Connection over HTTP

DynamoDB allows connection over HTTP.

⚠️ Multi-region availability

You can setup multi-region availability with DynamoDB’s global tables feature. However, keep in mind there is significant configuration required to get it up and running properly.

DynamoDB requires users to choose a partition key that determines how data is grouped and distributed among partitions. How you choose this key impacts DynamoDB’s scalability. Having a thorough understanding of how your data will be accessed will be critical to selecting the most appropriate partitioning key and strategy.

⚠️ Schema design (Schema-less)

DynamoDB is largely schema-less. Like many of its NoSQL siblings, DynamoDB lacks explicit support for relational data. It is designed with a denormalized schema in mind. Document sizes are also limited to 400kb in DynamoDB, forcing developers to denormalize the application data.

✅ No cold starts

DynamoDB doesn’t have any cold start-related concerns.

⚠️ ACID transactions

DynamoDB transactions are only ACID-compliant within a single region. DynamoDB does not support strongly consistent reads across regions.

Follow this article for a detailed comparison of Fauna and DynamoDB.

CockroachDB

Like Mongo here we are focusing on the Serverless offering of the CockroachDB not the provisioned managed service since provisioned service doesn’t offer pay per usage.

CockroachDB is a distributed SQL database system. It is based on the idea of a "cockroach cluster," a group of nodes that work together to provide fault tolerance and high availability. Each node in a CockroachDB cluster can serve read and write requests, and the cluster can automatically recover from failures and distribute data evenly across the nodes.

✅ Connection over HTTP

CockroachDB supports connection over HTTP.

✅ Multi-region availability

CockroachDB provides out-of-box multi-region availability without further configuration.

⚠️ Schema design (SQL )

CockroachDB has a Dynamic schema. It is relational and supports SQL. Since cockroach is based on traditional RDBMS it may take a lot of work to evolve unstructured data as your application grows.

❌ No cold starts

For large amounts of datasets CockroachDB clusters may experience cold starts.

✅ ACID transactions

CockroachDB fully supports strong consistency and ACID transactions.

Fauna

Fauna is a serverless database that combines the flexibility of NoSQL with the relational querying capabilities and consistency of SQL. You get the best of both worlds when using Fauna. It is fully managed, auto-scales, and does not require infrastructure management.

✅ Connection over HTTP

Fauna allows your applications to connect to the database over HTTP. You can use a Fauna driver or Fauna's GraphQL interface to communicate with the database.

✅ Multi-region availability

Fauna provides multi-region availability out of the box; it will auto-replicate your data across servers. It is distributed by default within a geographic region or across the globe. You always get strong consistency among region groups. Fauna is designed to keep the data closest to your users, improving reliability and availability.

✅ Schema design (Document-relational)

Fauna follows a document-relational database model. It combines the flexibility and familiarity of JSON documents with the relationships and querying power of a traditional relational database.
In short, you get the best of both SQL and NoSQL worlds with Fauna. Because of the flexible nature of the Fauna database, it is effortless to scale and evolve your data schema as your application grows.

✅ No cold starts

Fauna has zero cold starts.

✅ ACID transactions

One of the features of Fauna that has generated the most excitement is its strongly consistent distributed ACID transaction engine. Fauna provides strong consistency of data globally through an implementation of the Calvin protocol.

Below you’ll find a visual comparison of all these serverless database offerings.

	Mongo Atlas	DynamoDB	CockroachDB	Fauna
Connection of HTTP	✅	✅	✅	✅
Multi-region availability	⚠️	⚠️	✅	✅
No cold starts	❌	✅	❌	✅
ACID transaction	⚠️	⚠️	✅	✅
Flexible schema design	⚠️ (NoSQL only)	⚠️ (NoSQL only)	⚠️ (SQL only)	✅ (Document-relational)

If you are considering Fauna, create a free account (no credit card required) and give it a go. If you have questions, you can reach out in the Fauna Discord channel or Fauna community forum.

Implementing Fauna as infrastructure as code with Serverless Framework

Fauna — Thu, 27 Oct 2022 17:26:24 +0000

This article demonstrates how to use Fauna as infrastructure as code (IaC) in your application using the Serverless Framework, one of the most popular tools for managing infrastructure as code. Fauna has a dedicated plugin for the Serverless Framework that gives you complete control to manage your Fauna resources. You can integrate it into your test and CI/CD pipelines to keep your databases in sync across multiple environments.

Benefits of IaC

Before we dive deep into implementing Fauna as IaC, let's discuss why you might want to integrate IaC.

There are three main benefits of IaC:

Decreased risks: Provisioning all of your infrastructures manually can be risky, especially if you have multiple dependencies among services. Complex deployment is prone to human errors. When you automate the process with IaC, you reduce these. Your infrastructure also becomes testable, and you can spin up multiple environments (exact replicas of production).
Efficient software development lifecycle: With IaC, infrastructure provisioning becomes more reliable and consistent. Developers get complete control of the infrastructure through code. Developers can script once and use that code multiple times, saving time and effort while keeping full control.
Self-documenting and reduced administration: IaC is self-documenting. And it reduces administrative overhead, allowing your engineering efforts to be focused on new feature development.

Getting started with the Fauna Serverless Framework plugin

Install the Serverless Framework plugin with the following command

$ npm install @fauna-labs/serverless-fauna --save-dev

or using yarn

$ yarn add @fauna-labs/serverless-fauna

Open your serverless.yml file and add the following code to add Fauna to your project.


plugins:
  - "@fauna-labs/serverless-fauna"
fauna:
  client:
    secret: ${env:FAUNA_ROOT_KEY}
    # domain: db.fauna.com
    # port: 433
    # scheme: https

By default, the domain is set to [db.fauna.com](http://db.fauna.com). You can create new collections by adding the collection name under the collections field as demonstrated in the following code.

fauna:
  client:
    secret: ${env:FAUNA_ROOT_KEY}
    # domain: db.fauna.com
    # port: 433
    # scheme: https
  collections:
    Movies:
      name: Movies
      data:
        some_data_key: some_data_value

The collection configuration accepts the same configuration as CreateCollection query.

Similarly, you can add functions and indexes.

fauna:
  client:
    secret: ${env:FAUNA_ROOT_KEY}
    # domain: db.fauna.com
    # port: 433
    # scheme: https
  collections:
    Movies:
      name: Movies
      data:
        some_data_key: some_data_value

  functions:
    double:
      name: double
      body: ${file(./double.fql)}

  indexes:
    movies_by_type:
      name: movies_by_type
      source: ${self:fauna.collections.Movies.name}
      terms:
        fields: 
          - data.type

You can review the documentation for the Serverless Framework plugin to learn more about different functionalities and implementations.

We have also created a hands-on tutorial on building a REST API with Fauna, Serverless, and AWS Lambda. This tutorial will help you get up and running with a simple project with Fauna and Serverless Framework.

If you are searching for something more comprehensive, we have also created a self-paced workshop that will guide you through building a real-world project with Fauna, Serverless Framework, and AWS services. You can find the workshop here.

Fauna AWS Workshop ~ Building an event-driven app with AWS services and Fauna

This hands-on guide walks you through building a real-world event-driven serverless application using AWS services (i.e., AWS Lambda, Step Functions, API Gateway) and Fauna. In this workshop, you build a vacation booking application (Similar to Kayak or Redtag deals).

Got questions? Reach out in our Discord channel or in our community forum.

Delivering personalized content with Netlify’s Next.js Advanced Middleware and Fauna

Fauna — Wed, 26 Oct 2022 14:10:27 +0000

It’s become a truism that the speed and performance of web applications, e-commerce sites, and websites are critical variables in converting visitors to paying customers. Modern consumers expect these assets to serve data quickly, accurately, and contextually. API-first platforms like Fauna and Netlify, paired with other composable technologies designed for distributed workloads, drastically simplify the process of deploying applications that serve dynamic content at the edge with low latency, without complicated stitching or multi-team engineering efforts.

Serving globally distributed data in legacy architectures

Historically, developers have been constrained in how they optimize the performance attributes and experience of their applications by the underlying infrastructure - typically centralized servers and databases. The emergence of content delivery networks (CDNs) in the early 2000s enabled companies to host and distribute static assets to a globally distributed audience with low latency - however these CDNs were limited to hosting static content like images, text files, and HTML files. As these CDNs were globally distributed, the performance attributes of this static content was far superior to relying on a more traditional centralized model - a user accessing a static site in Shanghai would access that content from the CDN node in Shanghai, instead of being pushed to a data center across the county. With that said, the benefits of serving static content through a CDN wasn’t extended to dynamic content that could change over time and evolve; this category of content relies on compute and data services to transform the data based on context.

In the case of a global or even regional application, relying on a centralized architecture to serve dynamic content can be suboptimal. Consider an e-commerce application with users visiting the site from Los Angeles and Toronto. In an ideal world, the application would serve content that’s unique to those users and may change depending on context - where the user is physically located (and the potential discounts or prompts subsequently served to the user’s location), or if the user came from a referring source, for example. This data is ephemeral and dynamic in nature, and unique to each independent user accessing the application. If a user in Los Angeles is being served data hosted in the US-East 1 data center in Virginia, the performance attributes will suffer as a result simply due to the physical distance between the user and the compute. Similar to static content with CDNs, we want to optimize for speed and performance by moving the processing elements as close to each respective user as possible. This means pushing the compute and data out to the edge.

Dynamic personalization at the edge

Personalization allows companies to serve information that’s tailored for a particular customer based on a defined set of characteristics (most commonly geo-based). For example, if a visitor on an e-commerce site is based in Seattle, they may be served a discount on an item based on an inventory surplus in a local warehouse. Being able to serve this type of dynamic data, however, is not trivial. The ability to move the compute and data for dynamic content as close as possible to users (as demonstrated in CDNs) was until very recently impossible; this has changed with the emergence of platforms like Fauna and Netlify.

With Next.js Advanced Middleware (powered by Netlify Edge Functions) and Fauna’s distributed-by-default document-relational database, we’re able to move servers and databases closer to user locations - similar to the shift we’ve witnessed with static assets served on CDNs, but now for dynamic content. Together, Fauna and Netlify enable a transformation in the application architecture for personalization at the edge: developers now have the option to host business logic wherever it’s most performant.

When a user writes to or reads from a node, Netlify Edge Functions, the images/CSS files, and pieces of business logic are being grabbed from the closest possible compute node and corresponding data source node. Further, with the power of Fauna’s user-defined functions and advanced querying, you’re able to access a new mental model that allows you to make strategic decisions on where and how often work needs to be done; if you’re able to move the processing further back in the stack with Fauna, you can build it into the places that have the tightest relationships with the data and be optimized in ways that client-side JavaScript can’t be.

While the middleware component of Next.js is germane to Next.js, out-of-the-box it is limited to redirects or rewrites (you can send users to another page, or proxy other content into the application and show users new content). Netlify has extended the middleware API to give access to the full request, which allows for full transformation and personalization - whether it’s location or referral based. With a proxy or a redirect, you’d need to develop a custom page for each redirect. The ability to do a direct transformation with Netlify ultimately makes the engineering far less cumbersome. Netlify provides a really good example of the power of page transformations here.

Why Fauna + Netlify for edge personalization

Fauna is delivered as a global API and is complementary to Netlify’s Edge Functions; ultimately, you don’t want to bloat your edge functions with code. Everything in the Fauna-Netlify architecture is server-side rendered, so it creates a faster experience by avoiding bloated client-side code that would flood the client CPU or the browser memory. You’re able to modify your code from being a large response down to the few lines you need to write to account for personalization, and it’s possible to do this in the edge function instead of the client-side without client-side impact to the user and without sacrificing performance. Until very recently, this type of architecture wasn’t possible to build, especially for small teams. With Fauna and Netlify, you can get up and running with a working solution within hours.

Companies would historically need DevOps, frontend, backend, and database management teams to handle the orchestration of all of the elements associated with a global application. Now, a single full-stack developer can handle all of these elements and sketch out an application in a matter of hours that has global deployment, global database replication, and all of the front end and back end configured. Fauna’s auto-replication, distribution-by-default, and serverless attributes paired with Netlify’s edge compute make it possible. There’s no need to account for sharding, provisioning, or global replication - it’s all delivered out-of-the-box.

Netlify Edge Functions + Fauna example and code walk-through

To move out of the abstract and into a real-world example, we hosted a webinar with Netlify where we did a code walk-through of a basic marketing site pushed to the edge on Fauna and Netlify. Check out the accompanying webinar and start digging into the code in the repo to learn how to build the functionality with just a few lines of code. You’ll see that in this example, one user is making a request to the site from Toronto, Canada, and another from Portland, Oregon. When each respective user makes a request, they hit the compute node closest to their respective locations and each is served a unique page. Meanwhile, the request is also directed to the nearest Fauna replica to read and serve the data. In this example, the read is modified at the edge and as close as possible to the user - which wouldn’t be possible with a regionally hosted database or server. Both the compute and data is at the edge - which ultimately reduces latency.

Getting started

Fauna and Netlify unlock the ability to optimize for the most performant architectural decision based on the use case a team might be trying to solve, instead of being limited to whatever your legacy infrastructure may dictate.

Are you ready to get started?

Have any questions or want to engage with the Fauna and Netlify communities? Reach out in the Fauna Discord channel or Fauna community forum, and also check out the Netlify Community Forums.

Transfer data in Fauna to your analytics tool using Airbyte

Fauna — Tue, 25 Oct 2022 21:48:01 +0000

We are excited to introduce Fauna’s new Airbyte open source connector. This connector lets you replicate Fauna data into your data warehouses, lakes, and analytical databases, such as Snowflake, Redshift, S3, and more.

Why Airbyte

With the proliferation of applications and data sources, companies are often required to build custom connectors for data transfer across their architectures. Most of these ETL (extract, transform, and load) tools require maintaining and updating the connectors as requirements change over time. Airbyte is an open source data pipeline platform that eliminates this burden by offering a robust connector ecosystem that can scale without having to maintain the connector itself.

Why Fauna

Fauna is a distributed document-relational database delivered as a cloud API. Developers choose Fauna’s document-relational model because it combines the flexibility of NoSQL databases with the relational querying and ACID capabilities of SQL databases. This model is delivered as an API so you can focus on building features and not to worry about any operations or infrastructure management.

Why Fauna + Airbyte

Fauna and Airbyte both enable improved productivity and developer experience - together, the connector will allow developers to port and migrate transactional data in Fauna to your choice of analytical tools to drive business insights.
Continue reading for a guide on how to configure the Fauna source connector to transfer your database to one of the data analytics or warehousing destination connectors supported by Airbyte.
The Fauna source supports the following ways to export your data:

Full refresh append sync mode copies all of your data to the destination, without deleting existing data.
Full refresh overwrite sync mode copies the whole stream and replaces data in the destination by overwriting it.
Incremental append sync mode periodically transfers new, changed, or deleted data to the destination.
Incremental deduped history sync mode copies new records from stream and appends data in the destination, while providing a de-duplicated view mirroring the state of the stream in the source

Prerequisites

You need a destination database account and need to set up the Data Build Tool (dbt™) to transform fields in your documents to columns in your destination. You also need to install Docker.

Create a destination database account

If you do not already have an account for the database associated with your destination connector, create an account and save the authentication credentials for setting up the destination connector to populate the destination database.

Set up dbt

To access the fields in your Fauna source using SQL-style statements, create a dbt account and set up dbt as described in the Airbyte Transformations with dbt setup guide. The guide steps you through the setup for transforming the data between the source and destination, and connects you to the destination database.

Install Docker

The Fauna connector is an Airbyte Open Source integration, deployed as a Docker image. If you do not already have Docker installed, follow the Install Docker Engine guide.

Step 1: Set up the Fauna source

Depending on your use case, set up one of the following sync modes for your collection.

Full refresh sync mode

Follow these steps to fully sync the source and destination database.

1- Use the Fauna Dashboard or fauna-shell shell to create a role that can read the collection to be exported. The Fauna Source needs access to the Collections resource so that it can find which collections are readable. This does not give it access to all the collections, just the names of all the collections. For example:

CreateRole({
  name: "airbyte-readonly",
  privileges: [{
    resource: Collection("COLLECTION_NAME"),
    actions: { read: true }
  }],
})

Replace COLLECTION_NAME with the collection name for this connector.

2- Create a secret that has the permissions associated with the role, using the name of the role you created. For example:

CreateKey({
  name: "airbyte-readonly",
  role: Role("airbyte-readonly"),
})

{
  ref: Key("341909050919747665"),
  ts: 1662328730450000,
  role: Role("airbyte-readonly"),
  secret: "fnAEjXudojkeRWaz5lxL2wWuqHd8k690edbKNYZz",
  hashed_secret: "$2a$05$TGr5F3JzriWbRUXlKMlykerq1nnYzEUr4euwrbrLUcWgLhvWmnW6S"
}

Save the returned secret, otherwise, you need to create a new key.

Incremental append sync mode

Use incremental sync mode to periodically sync the source and destination, updating only new and changed data.

Follow these steps to set up incremental sync.

1- Use the Fauna Dashboard or fauna-shell to create an index, which lets the connector do incremental syncs. For example:

CreateIndex({
  name: "INDEX_NAME",
  source: Collection("COLLECTION_NAME"),
  terms: [],
  values: [
    { "field": "ts" },
    { "field": "ref" }
  ]
})

Replace INDEX_NAME with the name you configured for the Incremental Sync Index. Replace COLLECTION_NAME with the name of the collection configured for this connector.

|Index values|Description|
| --- | ----------- |
|`ts`| Last modified timestamp.|
|`ref`|Unique document identifier.|

2- Create a role that can read the collection and index, and can access index metadata to validate the index settings. For example:

CreateRole({
  name: "airbyte-readonly",
  privileges: [
    {
      resource: Collection("COLLECTION_NAME"),
      actions: { read: true }
    },
    {
      resource: Index("INDEX_NAME"),
      actions: { read: true }
    },
    {
      resource: Indexes(),
      actions: { read: true }
    }
  ],
})

Replace COLLECTION_NAME with the name of the collection configured for this connector. Replace INDEX_NAME with the name that you configured for the Incremental Sync Index.

3- Create a secret key that has the permissions associated with the role, using the name of the role you created. For example:

CreateKey({
  name: "airbyte-readonly",
  role: Role("airbyte-readonly"),
})

{
  ref: Key("341909050919747665"),
  ts: 1662328730450000,
  role: Role("airbyte-readonly"),
  secret: "fnAEjXudojkeRWaz5lxL2wWuqHd8k690edbKNYZz",
  hashed_secret: "$2a$05$TGr5F3JzriWbRUXlKMlykerq1nnYzEUr4euwrbrLUcWgLhvWmnW6S"
}

Save the returned secret. You need to enter the secret in step 2 of the Install Docker procedure. It is important to save the key, otherwise, you need to create a new key if you lose the provided secret.

The Fauna source iterates through all indexes on the database. For each index it finds, the following conditions must be met for incremental sync:

The source must be able to Get() the index, which means it needs read access to this index.
The source of the index must be a reference to the collection you are trying to sync
The number of values must be two.
The number of terms must be zero.
The values must be equal to:
```
{"field": "ts"}
{"field": "ref"}
```

All of the above conditions are checked in the order listed. If a check fails, it skips that index.

If no indexes are found in the initial setup, incremental sync isn't available for the given collection. No error is emitted because it can't be determined whether or not you are expecting an index for that collection.

If you find that the collection doesn't have incremental sync available, make sure that you followed all the setup steps, and that the source, terms, and values all match for your index.

Step 2: Deploy and launch Airbyte

Refer to the Deploy Airbyte instructions to install and deploy Airbyte. Enter the following commands to deploy the Airbyte server:
```
git clone https://github.com/airbytehq/airbyte.git
cd airbyte
docker-compose up
```
When the Airbyte banner displays, launch the Airbyte dashboard at http://localhost:8000.
Choose the Connections menu item to start setting up your data source.

Step 3: Set up the Fauna source

In the Airbyte dashboard, click the + New connection button. If you previously set up a source, click the Use existing source button to choose that source.
In the Source type dropdown, choose Fauna and click the Set up source button. This lists the configurable Fauna connector parameters. An in-app Setup Guide in the right-side panel also gives detailed setup instructions.

Set the following required parameters:

Parameter	Description
`Name`	Enter a descriptive name for this connection. The name is displayed in the Connections window connections list.
`Domain`	Enter the domain of the collection you want to export. See Region Groups for region domains.
`Port`	Enter the default port number: `443`.
`Scheme`	Enter the scheme used to connect to Fauna: `https`.
`Fauna Secret`	Enter the saved Fauna secret that you use to authenticate with the database.
`Page Size`	The page size lets you control the memory size, which affects connector performance.
`Deletion Mode`	The deletion mode lets you specify whether to ignore document deletions or flag documents as deleted, depending on your use case. + Choose from the following options: The Ignore option ignores document deletions. The Deleted Field option adds a date column that has the date when you deleted the document. This maintains document history while letting the destination reconstruct deletion events.

After setting up the source, click the Set up source button.

The "All connection tests passed!" message confirms successful connection to the Fauna source. This minimally confirms:

-   The secret is valid.


  The connector can list collections and indexes.

Step 4: Set up the destination

In the New connection window, choose a Destination type and click the Set up destination button. If you previously set up a destination, click the Use existing destination button to select and use that destination. Otherwise, continue to set up a new destination.
Destination connector configuration parameters are unique to the destination. Populate the Set up the destination fields according to the connector requirements, including authentication information if needed. A Setup Guide is provided in the right-side panel with detailed setup instructions.
When you are done, click the Set up destination button.

Step 5: Set up the connection

Set up the connection to sync the source and destination.

Enter a descriptive name for the connection in the Name field.
Choose a Transfer > Replication frequency, which is the data sync interval.

You can choose the Manual option to manually sync the data.

In the Streams > Destination Namespace field, choose a destination namespace where the data is stored. Options include:

Option	Description
`Mirror source structure`	Sets the name in the destination database to the name used for the Fauna source.
`Other`	Uses another naming option, such as prefixing the database name with a string.

Optionally, enter a stream name prefix in the Streams > Destination Stream Prefix field.
In the Activate the streams you want to sync section, click the > arrow to expand the available fields:

Field Description

data Collection data.

ref Unique document identifier.

ts Data timestamp.

ttl Time-to-live interval.

The document is deleted if it is not modified in the ttl time interval. The default value is null for not used. After document deletion, it is not displayed in temporal queries and the connector does not emit a deleted_at row.
Select ref as the Primary key. This uniquely identifies the document in the collection.

Field	Description
`data`	Collection data.
`ref`	Unique document identifier.
`ts`	Data timestamp.
`ttl`	Time-to-live interval.

Choose a Sync mode as the source sync behavior, full or incremental.

A new incremental sync gets the full database, the same as a full sync.

Sync mode	Description
Incremental - Deduped history	Sync new records from stream and append data in destination, also provides a de-duplicated view mirroring the state of the stream in the source.
Full refresh - Overwrite	Sync the whole stream and replace data in destination by overwriting it.
Incremental - Append	Sync new records from stream and append data in destination.
Full refresh - Append	Sync the whole stream and append data in destination.

If fewer than four options are displayed, it indicates that the index is incorrectly set up. See Step 1: Set up the Fauna source.

Choose the Normalization and Transformation data format:

Data format Description

Raw data (JSON) Put all the source data in a single column.

Normalized tabular data Put the ref, ts, ttl, and data fields in separate columns.
Click the + Add transformation button to add the dbt transform.

To extract the fields in the source data column, you need to configure dbt to map source data to destination database columns. For example, the following SQL-based query extracts the name, account_balance, and credit_card/expires fields from the source data column to populate three separate columns of the destination data:
```
with output as (
  select
    data:name as name
    data:account_balance as balance
    data:credit_card:expires as cc_expires
  from airbyte_schema.users
)

select *from output
```
Click the Set up connection button.

Data format	Description
`Raw data (JSON)`	Put all the source data in a single column.
`Normalized tabular data`	Put the `ref`, `ts`, `ttl`, and `data` fields in separate columns.

Step 6: Sync the data

On the Connection page for the connection you created, click the Sync now button.

The time to run a sync varies with the status displayed in Sync History. When the sync completes, the status changes from Running to Succeeded and shows:

The number of bytes transferred.
The number of records emitted and committed.
The sync duration.

Step 7: Verify the integration

To expand the sync log, click the > arrow to the right of the displayed time. This gives you a detailed view of the sync events.

Finally, verify successful database transfer by opening and viewing the destination database.

Conclusion

Integrating Fauna with the Airbyte open source solution arms developers building on Fauna with a powerful tool for gaining insights into the operational data living on Fauna. We’re excited to build on our partnership with Airbyte by working towards introducing an Airbyte Cloud connector. If you have any interest in a Fauna + Airbyte Cloud integration or questions about the open source connector, feel free to reachout to us and ask questions in our forum or on our Discord.

Edge computing reference architectures

Fauna — Thu, 04 Aug 2022 15:27:10 +0000

Cloud computing has transformed how businesses conduct their work, but many face challenges regarding data protection (residency and sovereignty), network latency and throughput, and industrial integration. On-premises infrastructure, especially when it’s tightly integrated with edge computing services provided by various cloud platforms such as AWS Lambda@Edge, Cloudflare Workers, Fastly Compute@Edge, etc., can be helpful in solving these problems. But that’s just part of the edge computing puzzle.

Cloud data centers and on-premises infrastructures form the near edge of the network. The other part of the network can be considered the far edge. Many businesses need the last mile coverage of the cloud to ensure that the cloud connects and integrates directly with controllers, sensors, and far-end devices. Currently, this workload is handled by specialized programmable logic controllers (PLCs) and human-machine interfaces (HMIs). These devices work great but depend on proprietary operating systems and are costly to acquire, maintain, and upgrade.

The following table lists the different parts of an edge network and their usual distance from the devices on the far edge of the network, along with the network latency from applications to those devices:

The cloud (regional and zonal data centers) — is 10+ km away from edge devices and has a latency of over 10 ms
Near edge (on-prem data centers) — is between 1 to 10 km away from the edge devices and has a latency from 1 to 10 ms
Far edge L1 (gateways, controllers) — is between 10 m to 1 km with a latency of somewhere between 0.1 ms and 1 ms
Far edge L2 (sensors, devices) — is in the 10 m range of the edge devices and has sub-millisecond latency

This article will take you through some of the edge computing reference architectures proposed by different organizations, researchers, and technologists so that you have a better understanding of the solutions that edge computing offers.

Why do you need edge computing?

Content delivery networks are based on the idea that to provide a great user experience, you need to deliver the content as quickly as possible. You can only do that by physically bridging the network latency gap to place the content closer to your end users. Fast-access storage solves this problem. Big companies in specific domains like content delivery and mobile networking already use the edge network for storage, if not for computing.

But many businesses can’t solve their problems with delivery networks; they need computing power without the latency lag of the cloud. They would rather have something closer to the edge in addition to the cloud.

Edge computing use cases

Take vehicle autonomy, for instance. An average car without autonomy has over fifty sensors that help the vehicle’s critical and non-critical functions. Traditionally these sensors were lightweight, but with autonomous car cameras, LiDAR sensors, and others, there’s an ever-increasing demand for computing power on board. You can’t rely on APIs and the network for processing specific information in the operations of a critical machine like a car.
Another example is surveillance. Businesses might use AI and ML applications to process a camera feed in real time. Offloading that amount of raw data to the cloud for processing might not be the wisest decision, as it will need tremendous network bandwidth and computing power to move data back and forth. To save time spent moving data around, you need more computing power at the source, and you can send the workloads that can wait to the cloud.
The same principles apply to many specialized industries, such as aerospace, live entertainment, mass transportation, and manufacturing. This is why the edge is becoming more relevant by the day. Other applications of edge computing include:

Decoupling automation workloads from specialized hardware like PLCs and HMIs by using open source software like FreeRTOS or OPENRTOS and running the workloads on containers
Offloading custom data-intensive critical workloads to the near edge network
Minimizing your maintenance and downtime with a redundant computing system for your business’s critical systems, which adds real value to your business continuity planning (BCP) and disaster recovery planning (DRP) efforts
Optimizing business processing with the help of faster response times from the computing power on the network’s edge

Interested in understanding edge computing beyond its use in IoT?

Fauna can help you build faster, more performant edge applications. Since Fauna is distributed dy default, you can reduce latency for your apps with a close-proximity database for your edge nodes. Easily integrate with Cloudlfare workers and Fastly Compute@Edge platforms out of the box.

Edge computing reference architectures

To properly design and implement edge computing networks, businesses rely on reference architectures. The following are some reasons why they’re an invaluable resource.

Design considerations

Businesses are still finding different ways to exploit the power of edge computing, and edge computing reference architectures help them understand systems that could potentially work for them. Not only are these reference architecture patterns well researched and thought out by industry experts, but they try to encapsulate the key features that apply to various businesses.

You can use reference architectures for inspiration as well as innovate on top of them. There are several factors and design considerations that you should take into account while architecting an edge computing system for your business:

Time sensitivity: With edge computing, critical business functions don’t have to wait for workloads to be offloaded to the cloud for processing
Network bandwidth: Bandwidth is limited; you don’t want to choke the network by blocking other important operations
Security and privacy: Data sovereignty and data residency, both for security and privacy, can be enabled by the edge
Operational continuity: During network disruptions, you can offload the extremely critical operations of your business to the edge

Implementation outcomes

The outcomes and benefits of architecting edge computing systems align strongly with the abovementioned factors. With a successfully deployed edge computing system, your business gains the following core benefits:

Significantly lower latency and higher network bandwidth by utilizing the proximity from the request origin to the computing system
Effective network use by filtering and reducing farther data transfer to on-premises or cloud infrastructure
Better enforcement of security and privacy standards, enabling data sovereignty and data residency

Development and deployment plans

When developing for the edge, businesses need to ensure that they don’t end up creating the same dependencies and bottlenecks they experienced when using only on-premises data centers or the cloud.

An excellent way to do this is to research, identify, and pick the right open standards and technologies to help you get the architecture you want. Possible open standards include those for APIs, documentation, virtualization patterns, deployment pipelines, and code promotion strategies.

Edge computing reference architectures that use Fauna

There are many reference architectures, some of which heavily borrow from one another. However, each has a unique approach and feature set. Fauna, as a distributed general-purpose database, interacts well with all of them in different ways.

The following are details on four reference architectures, their relevance and innovation, and how to adopt them with Fauna.

Industrial internet reference architecture

The Industrial Internet Reference Architecture (IIRA) was first developed in 2015 by the Industry IoT Consortium (IIC). This was a collaborative effort from a range of companies, including Bosch and Boeing in manufacturing; Samsung and Huawei in consumer electronics; and IBM, Intel, Microsoft, SAP, Oracle, and Cisco in the software industry.

Think of IIRA as having four different viewpoints: business, functional, usage, and implementation. The system is divided into enterprise, platform, and edge. You have low-level devices in factories or appliances at the edge, connected with the platform tier using low-level APIs. Data collected from these devices flows to data and analytics services and operations users.

The data generated by the edge, along with the data and insights gathered and processed by the platform tier, is consumed by the enterprise tier on two levels: the business domain and app domain. The data in the business domain lands in other systems, such as ERP, and the data in the app domain flows further based on business logic and rules to the business users in the form of mobile apps, browser-based frontend applications, or business intelligence tools.

The following diagram shows the three-tiered approach to IIoT system architecture using IIRA:

Fauna’s distributed architecture allows it to sit across multiple layers of the network while simultaneously guaranteeing you the privileges of a flexible document-oriented database and ACID transactions. The data and analytics services and platforms can heavily interact with Fauna, and the operations services and platform will do the same. In this context, Fauna sits mainly on the platform tier but interacts with the edge tier, depending on whether the devices are integrated with the platform tier using a virtualization layer.

FAR-EDGE reference architecture

Aside from the previously mentioned features of reference architectures, the FAR-EDGE Reference Architecture brings many new things to the table. It explicitly offers a separate logical and ledger layer that handles processes and rules across the distributed computing system by using the sheer power and spread of the system. Like most edge computing reference architectures, FAR-EDGE also concentrates on saving bandwidth and storage, enabling ultra-low latency, proximity processing, and enhanced scalability by exploiting the distributed nature of the edge-cloud hybrid architecture.

Compare the idea of FAR-EDGE with some of the Web3 architectures built on top of distributed frameworks like the blockchain. Fauna, with its capability to provide edge functions as an extension to the application infrastructure, can play a central role by servicing various layers and functions proposed in this reference architecture. Consider the functional view of the FAR-EDGE reference architecture below:

It’s worth reiterating that Fauna is a distributed document-relational database, which is ideal for businesses with a variety of software applications and hardware devices. In this functional view, Fauna will sit and interact not only with applications and cloud services but also with ledger and edge processes at the ledger and gateway levels. You can enable this by using well-documented APIs and SDKs.

Edge computing reference architecture 2.0

The Edge Computing Consortium (ECC) proposed a model-driven reference architecture that attempts to solve the problems businesses face when they try to connect the digital and the physical worlds. Since the reference architecture is multidimensional, you can regard it from different viewpoints. Consider one of its layers, the Edge Computing Node layer, from a functional viewpoint.

The Edge Computing Reference Architecture 2.0 is a multidimensional architecture composed of the following components from a high-level view: smart services, a service fabric, a Connectivity and Computing Fabric (CCF), and Edge Computing Nodes (ECNs).

As mentioned earlier, edge computing thrives when open standards are followed and hardware devices are decoupled from specifically designed hardware. This problem has long been solved for specific web and desktop applications via virtualization. The Edge Computing Reference Architecture also suggests you create an Edge Virtualization Function (EVF) layer that handles connectivity and data streaming and collection, along with access policies and security.

The following diagram of the Edge Computing Reference Architecture 2.0 shows the functional view of an Edge Computing Node (ECN):

Fauna is more than capable of taking care of the universal services, such as streaming and time series data, along with any industry-oriented services that might need integration.

Conclusion

Edge computing reference architectures can vastly simplify the design and usage of edge computing systems, and Fauna can help at different levels in all these architectures. Fauna is a natively serverless, distributed transactional database that allows for flexible schemas and comes with a well-designed data API for modern applications.

With Fauna in the cloud and at the edge, you can store and retrieve your data at ultra-low latencies while being closer to the source than ever before. With built-in data security, authentication, authorization, and attribute-based access control, Fauna simplifies the implementation of edge computing reference architectures and lets you concentrate more on your business.

Modernization of the database: DynamoDB to Fauna

Fauna — Mon, 25 Jul 2022 14:16:10 +0000

Serverless databases like Fauna fill a crucial role for modern applications. Given the amount of cloud-based traffic and varying workloads that organizations from startups to enterprises manage, serverless databases are the natural fit to the rest of your serverless architecture, adding seamless flexibility, scaling, and low latency for your applications.

DynamoDB has been one of the most popular serverless databases. In addition to coming from AWS, it has benefited from being one of the earliest offerings in this space. Its popularity is hard to dispute, backed up by its proven ability to effectively handle traffic spikes and workload variations without overloading infrastructure or racking up unnecessary costs during periods of low traffic. Though DynamoDB has much to offer, Fauna is a much newer offering that comes with a host of powerful and unique features that enhance the serverless experience for organizations of all sizes.

In this article, we’ll learn about serverless databases, compare key differences between DynamoDB and Fauna, and provide insight on which one to choose for your next big project.

What makes a good serverless database?

A great serverless database should offer the following values:

Lower operation costs: One of the main reasons for choosing software is cost. Traditionally, organizations need to budget for the continuous management of their database(s) and infrastructure, and not just for application development. Serverless architectures promise “zero ops,” which eliminate these management concerns. As headcount makes up your highest spend, leaning out on operations overhead ultimately lowers your TCO (total cost of ownership).
Pay as you go: It should also offer a simple and transparent pricing model without any traps or unexpected costs. You should only have to pay for the resources you use, with billing being proportional to the amount of data stored and volume of transactions sent to the database.
High availability: There should be no maintenance windows or unplanned downtimes. Data and compute resources should be automatically distributed and replicated, offering high durability and resiliency to region/zone outages. The more these attributes are provided to the operator without any additional configuration, the better.
Low latency: Ensuring a good user experience is key to user engagement, ultimately impacting how successful your products are with customers. Serverless architectures should not trade off the ability to scale elastically with cold start-up delays or increased latency due to speed-of-light constraints. A great serverless database offering should be always-on and instantly scale; And provide multiple region choices so that it can be accessed as close to your compute resources as possible.

Why should you choose Fauna?

Serverless databases are critical components for managing unpredictable and rapidly changing workloads, essentially following a pay-as-you-use model. They are also a good fit for companies with a small workforce, enabling them to consume compute workloads and infrastructure without manual overhead. Using a serverless database allows you to simplify database operations and eliminate problems common with traditional databases, such as maintenance, upgrades and patching, and cost of operations.

Fauna is accessed as an API and there is nothing to install, maintain, or operate. You can quickly deploy a database in three button clicks, start coding, and immediately connect to the database. You can scale databases without any limitations and create an unlimited number of databases. It is chock full of cloud-native features: login with your GitHub account; integrate with third-party services like Auth0, Netlify, and Vercel; has built-in streaming, user authentication and fine-grain authorization. It supports user-defined functions – similar to stored procedures – which help eliminate redundant code, maintain Fauna Query Language (FQL) consistency, and remove code duplication. And it has a native GraphQL API, allowing organizations adopting GraphQL to quickly get up and running with a data source in minutes.

Fauna vs. DynamoDB

Fauna offers numerous features that distinguish it from DynamoDB and similar serverless databases. The following is a comparison between Fauna and DynamoDB and the components that each offers.

Geo-distribution

Businesses with global audiences need to ensure fast response and high performance for customers in multiple regions across the globe. This requires a database that allows data to be replicated globally and served from specified locations — the closer the proximity of the data, the better the experience and performance.

By default, DynamoDB is replicated across multiple availability zones in a single region. But with the “global tables” feature, DynamoDB provides a fully managed solution for deploying a multi-region, multi-active database. You specify the regions you want the database to replicate, and DynamoDB will propagate all data changes to them. Besides reducing read latency, multi-region replication ensures that single region failures do not result in any outages. Using global tables comes with quite a few caveats. Your application can read and write to any replica table, but if it requires strongly consistent reads, it must perform all its strongly consistent reads and writes against the same region. AWS documentation states that any newly written item is propagated to all replica tables within a second — not an insignificant latency — and that ACID guarantees only apply within the AWS Region where the write was originally made. When an AWS region suffers an outage or degradation, your application can redirect to a healthy region but it isn’t automatic: AWS documentation prompts developers to apply custom business logic to determine when to redirect requests to other regions. Global tables are more costly to operate. Your costs essentially multiply several times with the addition of every replica. In addition, you must also use the higher priced on-demand capacity mode, though this can be mitigated as they allow the use of provisioned capacity with auto-scaling.

Unlike DynamoDB, Fauna is multi-region-distributed by default — every database you create is replicated and distributed across geographic regions. In addition, Fauna also provides the Region Groups functionality, which allows developers to control which region their data resides in. Other than selecting the Region Group, there is no additional configuration and no custom business logic to implement. When your application makes a request to the Fauna API, it is automatically routed to the region closest to it. Reads are immediately served out of that region. And writes are automatically propagated to the other regions’ replicas automatically. Fauna publicly publishes its “internal latency” – separated into reads and writes – on status.fauna.com. There is no separate pricing scheme for single- vs multi-region with Fauna since your databases are always multi-region – providing you straightforward and transparent pricing.

Transactionality

While you want to ensure the real-time availability of data, your data needs to be consistent and accurate. Not all serverless databases can provide this.

DynamoDB, for instance, offers serializable read and write transactions but is only ACID-compliant within the region where the transaction occurs. This is a problem for multi-region deployments, because it can lead to errors during transactions that run simultaneously in different regions.

One of Fauna’s most touted innovations is its approach to isolation and distribution, which is derived from Calvin, an algorithm for achieving distributed consistency at scale. As an oversimplified explanation to what it entails, queries are handled before they interact with the storage system, while deterministic ordering – eschewing the need for expensive atomic-clocks – is employed to provide serializability, ensuring that no two concurrent transactions can commit to the same data. Through Fauna’s Calvin implementation, it is one of the few serverless databases that offers ACID guarantees while being globally distributed at the same time. Transactions are globally distributed, ACID-compliant, and serializable, with no additional configuration or separate pricing scheme.

Streaming

Real-time applications require an endpoint that enables persistent connectivity, allowing the server to push information to the client requesting it.

With DynamoDB, you’ll need to combine several services in order to implement streaming. The first part is setting up a DynamoDB Stream: DynamoDB integrates with AWS Lambda, where you set up triggers that respond to changes (DynamoDB Stream events) in your table(s). Thus, there’s a bit of configuration, and some coding involved in writing the Lambda. From there, you need to implement the persistent endpoint. AWS can accomplish this in a number of ways, and your choices are using either an EC2, API Gateway Websocket API, or AppSync (via GraphQL Streaming). Regardless, these are additional components that need to be implemented and/or configured and maintained. On top of that are the costs involved in running the additional services.

In contrast, Fauna’s API endpoints support streaming right out of the box. In order to consume a stream, all you need to do is write some lines of code on the client side of your application using a driver in your language of choice. For example, here are the instructions for instantiating a stream client in javascript. Fauna offers two kinds of streaming: document streaming and set streaming. With document streaming, an event notification is sent to the subscriber whenever a document is created, updated, or deleted. With set streaming, events are sent whenever one or more documents enter or exit the set while doing a create, delete, or update.

Security

Because they are accessed as APIs, great serverless databases must provide robust security models, so that developers do not have to build their own authentication and authorization.

Access to DynamoDB is based on AWS’s highly robust Identity and Access Management (IAM), in which granular permissions are applicable to the DynamoDB resources (tables, indexes, and stream). You can also specify IAM policies that grant access to perform specific actions (e.g, read, add, update, batch update, etc.) on specific resources for specific users.

Fauna supports both API keys and identity-based access tokens, and supports integration with external identity providers that support the OAuth flow (such as Auth0, Okta, OneLogin, etc.). Keys and tokens inherit the permissions of roles upon which they’re assigned. Roles are configured with granular permissions to access every single resource in the database, including collections (tables), indexes, user-defined functions, other roles, and schema. Fauna also provides attribute-based access control (ABAC), allowing permissions to be dynamically assigned based on tokens’ identities’ attributes. With ABAC, you can define custom business logic, creating dynamic rules that control access to all resources, all the way down to specific documents in a collection.

Automated functionalities

In DynamoDB, a host of configuration options allow you to operate various capacity modes, single- vs. multi-region, types of indexes and index-partitioning strategy used, logging, and more. The tradeoff for this flexibility is more manual work and time.

Fauna, however, automates multiple functions for you, such as database provisioning, maintenance, sharding, scaling, and replication. The “zero ops” model of Fauna lets you focus on building the critical aspects of your applications without worrying about the complexity of a distributed architecture. This is especially useful if your business needs to carefully manage limited engineering resources while dealing with unpredictable workloads.

DynamoDB includes an automated backup feature. It charges for backups separately, and you need to select from on-demand or continuous backups. The backup is optional and by project.

In Fauna, you can schedule daily backups on any database in the form of snapshots of the entire data-tree. You can also configure a retention period for them. Databases can be “restored” (overridden) in place from any backup, or you can use backups to seed new databases. Fauna also supports temporality, allowing you to go back through history and query data at any arbitrary point in time. This is because in Fauna, data is stored as snapshots across time. You can use this feature to implement point-in-time recovery and targeted data repairs in your database.

GraphQL API support

GraphQL is designed to make API development faster and more flexible for software engineers. API developers use GraphQL to have a schema that will showcase all the possible data that the clients will want to query through a specific endpoint. Unlike most other serverless databases like DynamoDB and Upstash, Fauna provides a native GraphQL API for data access in addition to its query language, FQL.

If you are looking to launch your product in less time and expect a lot of changes on your API endpoint, then using a serverless database that supports GraphQL APIs will be the better choice for you.

💡 Check out our GraphQL workshop that introduces you to Fauna. In two hours, you’ll build an application in either Next.js or SvelteKit that has authentication, user-defined functions, customer resolvers, and more.

Scalability and pricing

With some serverless databases like DynamoDB, automatic scaling is an option but isn’t set by default. There are basically three modes in DynamoDB:

The default – called provisioned capacity – lets you set the volume and scale that you need up front. You then either manually monitor and manage these parameters or be automatically throttled once the load hits your predefined capacities. This mode is ideal when you have steady, predictable loads, and you need to stay within a narrow budget.

Provisioned capacity with auto-scaling is the same as provisioned capacity but also dynamically adjusts throughput capacity based on actual traffic, allowing your application to handle sudden bursts in traffic without being throttled. Pricing is a factor of the baseline capacity you’ve configured and your application's actual load (above that baseline) that your application experiences. It is important to note that because provisioned capacity (with or without auto-scaling) involves setting a baseline desired capacity, you are always paying for that capacity, even when your traffic is zero.

On-demand capacity is priced significantly higher (on a per-unit read/write/etc. basis) than provisioned capacity, but is truly serverless and will scale to zero (where you pay nothing) if no traffic is observed or as high as needed when bursts are experienced.

The one and only mode in Fauna is on-demand. It autoscales without any intervention from you and you don’t need to manage the settings of the desired throughput. There is no need to monitor the database to keep your database safe from being saturated. As such, pricing is much more straightforward and only a factor of how much data you store, how many transactions occurred, and how large (complex) the transactions/queries are. Comparison pricing between Fauna and DynamoDB is highly based on context and use-case and especially how you plan to use DynamoDB, given its flexibility in setting things up. We cover these topics in “Comparing Fauna and DynamoDB: Pricing and features.”

Conclusion

Serverless databases offer numerous benefits, including lower costs, less operational workload, faster time to market, and high scalability. When choosing a serverless database, be sure to select one that offers you maximum consistency, high performance, and low latency.

While there are many serverless databases to choose from, Fauna offers the largest range of features for your organization’s needs. Automated scaling, multi-region ACID-compliant transactions, and GraphQL support will help you optimize your workload and improve your product for users. Your developers can focus on improving your applications and core services while worrying less about managing the database or infrastructure. This leads to a shorter development time and faster application delivery.

Interested in learning more about Fauna? Reach out to our team or sign up and start using Fauna for free.

Multi-region scaling with Fauna

Fauna — Thu, 21 Jul 2022 18:30:06 +0000

Because the digital era has grown so competitive — providing users with everything from banking systems to online food delivery — it’s critical that applications never go down for any reason. In fact, users expect that the applications they access will remain available and highly responsive with zero downtime from any part of the world, 24/7.

To provide highly available and fault-tolerant services, platforms must leverage their architecture in order to ensure that their applications and database systems, especially, are always available. They achieve this by replicating apps and databases in geographically isolated parts of the world. With this redundancy, applications remain highly available in different locations; in a situation such as an outage in a data center, the data isn’t lost and the application can keep serving end users from another location. This is known as multi-region scaling.

Multi-region scaling is important because even an entire region can experience downtime. For social media, e-commerce, and other business-critical applications, this downtime can result in revenue losses of millions of dollars. Using a multi-region architecture ensures that applications will keep serving users from different regions.

In this article, you’ll learn more about multi-region scaling, including why it’s so vital and what key issues it solves. You’ll also learn how Fauna can help solve these issues.

Why do you need multi-region scaling?

Multi-region scaling involves the use of multiple regions around the world. A region is a collection of data centers, or availability zones, that are connected via high-speed network infrastructure but are physically isolated from one another, usually in different cities within a country. An application deployed in multiple availability zones within a region is considered highly available. Cloud providers have regions available worldwide, allowing an application to be deployed into more than one region.

This type of scaling becomes necessary as businesses grow to serve millions of users. To keep up with such growth, the underlying tech infrastructure must be designed to be resilient and fault-tolerant while providing a global footprint. Enterprises use multi-region scaling to provide a seamless experience to users by reducing latency as well as ensuring there is a disaster recovery mechanism in place.

For a global application, apart from the application backend, the database system needs to be available in a global context because a global application with a central database system still has a single point of failure. If the database goes down, the application is of no use to end users. A multi-region setup not only ensures application availability but ensures that data is secure and not lost.

How multi-region scaling works

As noted earlier, in multi-region scaling, the infrastructure is deployed into multiple regions. If one region goes down, the service or application remains accessible from a different region, though users in other regions may experience increased latency.

A multi-region setup involves the following elements:

Use of CDNs: Static content like images, videos, and documents are distributed and cached to a content delivery network (CDN) such as Amazon CloudFront and made local within regions serving users from the closest caching server, instead of bringing the data directly from the source for each request.
Active/active distributed databases: Distributed databases are usually deployed in an active/passive fashion, in which the primary instance serves both read and write requests while the secondary instances are only available for reads. The active/active deployment strategy allows for all database instances to serve both read and write requests. The data is asynchronously replicated, although there may be a replica lag known as eventual consistency, or the time it takes for changes to reflect in all instances of the database.
Stateless architecture: Multi-region scaling encourages stateless architectures for applications, meaning they should serve all users with the same responses independent of prior requests and not store or use any local session information. This allows for great horizontal scalability since any available resource can handle the inbound requests.
Use of DNS routing mechanisms: With a multi-region architecture, the DNS routing doesn’t just direct traffic to the primary instance but also to an instance that provides the least latency for users. For example, AWS routes traffic based on the closest location or proximity to resources.

Why is multi-region scaling important?

There are several advantages to using multi-region scaling.

Catering to a wider audience

Digital businesses increasingly serve users from around the globe, which can present a challenge in providing a consistent experience. To serve users in multiple countries and continents, applications must be accessible from all parts of the world. The redundant system architectures of multi-region scaling make this possible, allowing businesses to launch into new markets and continue to grow their userbase.

Lowering data latency for global users

When organizations serve a global audience, the latency that users experience can affect how they view the service. Applications must have low latency to provide an instantaneous experience to users. Multi-region architectures make this possible.

A low-latency multi-region architecture must include these factors:

Bringing the content close to end users: The goal is to serve static content to end users from servers closest to them. CDNs are the key to achieving this. Content is replicated and cached in each region, and users are served locally from the closest point within their region.
Deploying components into multiple regions: Application backends and databases are deployed into multiple regions in order to serve users right from the regions where they live.

Increasing fault tolerance and aiding disaster recovery

Multi-region scaling at its core is about designing a system in a redundant fashion. A redundant architecture is inherently resistant to failures because two identical infrastructures are running at the same time. Even if one part of the architecture fails, the other remains available with an intelligent routing strategy that directs requests from users to their closest resources.

A multi-region design also allows for disaster recovery, because when the application goes down in one region, it can be restored to its original state in one of the other regions. Since data is already being replicated across regions, once a stable infrastructure is deployed the data and all other content can easily be restored from backups.

Ensuring compliance with privacy laws and regulations

Laws and regulations have been enacted in various regions worldwide to mandate data privacy and protect consumer rights. A multi-region strategy can help ensure that specific data is only accessible to users in a certain geographical area. For example, many financial and banking institutions require that the consumer data remains in the country or region of origin. This can easily be handled with a multi-region setup that restricts access to data and its storage to and from the origins only.

Implementing multi-region scaling with Fauna

Deploying a multi-region infrastructure requires a lot of work: deploying databases and application backends in multiple regions; setting up routing mechanisms to direct the traffic properly; setting up replication strategies for data consistency between all databases; and setting up a backup and restoration strategy in case of problems. This can be a nightmare for teams to not only set up but also manage and monitor correctly.

Fauna can help you address these challenges. The distributed database, delivered as an API, is designed to be developer-friendly. It provides both great extensibility and a rich set of features, including the following:

Document-relational database: Fauna stores data in the form of documents (like JSON objects) but also allows for relationships between documents via foreign keys, combining the ease of working with JSON with the querying power of traditional relational databases. It also provides strongly consistent, distributed, and guaranteed ACID-compliant transactions, making it an ideal choice for all types of data storage needs.
Serverless: Fauna is delivered as an API running on a serverless architecture, which means there’s no infrastructure to manage. Developers can focus on developing the application while Fauna handles the infrastructure.
Globally distributed: Fauna is distributed globally and multi-region by default. Fauna easily addresses GDPR compliance with its Region Groups feature that allows control over where data physically resides. Data can be easily split between regions with totally separate compute, databases and storage. And since it’s cloud agnostic it works with all popular cloud providers.
Managed data and infrastructure operations: Fauna takes care of all the operations related to data and infrastructure management, including data sharding or replications and capacity planning, so your team can focus on improving your application.

Other features include user-defined functions (UDFs), event streaming, and support for popular programming languages like Python, JavaScript, Java, Scala, C#, and Go. Fauna allows you to migrate data from existing sources, and its security features include attribute-based access control (ABAC), integration with Auth0 and other IdPs, and user identity and access management.

With Fauna, you can easily address common issues like:

Disaster recovery: The inherently distributed nature of Fauna makes disaster recovery simple. Fauna takes care of monotonous tasks like database backups and globally distributing data, so it’s easier to recover and resume business operations from a down region.
User latency: Fauna helps keep user latency down at scale due to its distributed multi-region setup. Data is served to users from their closest regions by automatically routing requests to replicas closest to them.
Compliance with regulations: Fauna was created with security and compliance in mind, and all its features are designed according to the AWS Well-Architected Framework and ISO27000 controls. Fauna’s security controls are developed according to AICPA’s Trust Services Criteria, are SOC 2 certified, and comply with GDPR and HIPAA. This allows for scalability across regions while complying with major laws and regulations.

Conclusion

Multi-region scaling can make a major difference in an organization’s ability to serve a global audience. Using it can minimize or eliminate service disruptions and data loss while ensuring high availability and low latency for end users. A multi-region-enabled architecture helps businesses continue to expand their global reach with confidence.

Fauna is a key part of a multi-region scaling strategy. It enables better infrastructure management, recovery options, regulatory compliance, and database design. Get a demo or sign up for free to see how Fauna can help you optimize your applications and improve your business.

Furqan Butt is a software developer and part of the AWS Community Builders Program for data and analytics.

Overcoming database scaling issues with Fauna’s serverless offering

Fauna — Thu, 07 Jul 2022 18:43:05 +0000

With the amount of data generated every second, scaling high-volume applications is hard. Scaling databases is harder, even if you’re hosting your database in the cloud. You need to choose the right database for your application, but what if you need data modeled in several different ways? Spinning up MongoDB as your document database, MySQL as your relational database, and a few additional databases might be an option, but that could prevent you from scaling your application.

One solution is Fauna, a serverless document-relational database that gives you the freedom of a document-oriented database along with the safety of using a relational database. Although many databases now come with a serverless offering, Fauna is the only one designed and built as a serverless database to enable schema flexibility, easy integration, and reliable scaling.

This article will take you through some of the prominent features of Fauna and demonstrate why it’s a better option than MongoDB or CockroachDB.

Why you should choose Fauna

Fauna stands out compared to competitors like MongoDB, CockroachDB, and DynamoDB. Because it’s natively serverless, it scales quickly and efficiently. It’s one of the few databases that solves the problem of globally distributed ACID transactions, which is an essential feature that neither MongoDB nor CockroachDB offers.

The following is a closer look at what Fauna offers and why it’s superior to other databases.

It’s fully serverless

There are a lot of databases that come with a serverless offering, such as MongoDB Atlas. However, MongoDB Atlas has essentially been repurposed to work as a serverless database and wasn’t designed for that. Meanwhile, Fauna is only serverless. It’s designed to help you concentrate on making your product better rather than on managing database infrastructure just to keep your application running.

Because it’s serverless, you don’t have to take care of any routine database operations (DBOps) tasks that require a lot of time, such as sharding, cluster management, replication, and version upgrades. You also don’t have to manage any connection pools, because it works on a serverless invocation model. Additionally, it doesn’t have a cold start problem.

All of this makes Fauna much more developer-friendly than any other multipurpose database.

It scales automagically

MongoDB was built to scale horizontally at the expense of ACID transactions, meaning that instead of immediate consistency, you get eventual consistency. This won’t work for many use cases. Moreover, sharding in MongoDB requires maintenance and downtime, and you must set a custom shard key per collection. As for resiliency, MongoDB can be vulnerable to failures caused by network partitions or cluster node outages.

Fauna, however, was built to be a globally distributed, multi-data-center, active-active, serverless database from the get-go. It succeeds with high scalability because its Calvin-based architecture allows for both horizontal and vertical scaling and ensures that every node in a Fauna cluster performs the same roles (unlike in many other distributed databases) as a query coordinator, data replica, and log replica. The beauty of this architecture is that the provisioning and management of cluster nodes happen entirely behind the scenes without bothering you.

It offers global low latency

Database calls are costly. To improve database performance and enhance the user experience by lowering the response latency, you need to reduce the number of calls to your database as much as possible. Fauna does precisely that by batching the transactions and applying them across geographical regions, so that you can go through and update multiple collections in one query.

Fauna is still dependent on external network latency. The low latency lays the foundation for enabling strong or immediate consistency. Many use cases require strong consistency to work, such as high-frequency trading, banking, or high-volume booking systems. If you attempt to use MongoDB or CockroachDB for these use cases, you might not be able to scale them efficiently.

It provides distributed ACID transactions

Guaranteeing ACID transactions in a globally distributed database was a complex problem to solve, but Yale University’s Calvin paper changed everything. FaunaDB implemented a system based on Calvin in which, according to Fauna, “Every read issued by an application is guaranteed to reflect the writes of transactions that have been completed before when the read was issued.”

Databases such as CockroachDB don’t offer a high level of serializability. They have traditionally needed Network Time Protocol (NTP) clocks to be synchronized across nodes to ensure that the serializability isn’t violated because of time drift.

In MongoDB, dirty reads and lost transactions are plausible if and when the database fails. Because MongoDB doesn’t have read-side coordination for transactions spanning multiple shards, consistency can be violated because the default read consistency level allows that to happen.

Only Fauna has implemented a Calvin-style protocol that guarantees the highest level of transaction serializability without using a clock and without increasing latency. Fauna uses strict serializability to enable consistency across all nodes in all geographical regions and prevent stale reads. It achieves this by maintaining a log position that helps guarantee a monotonically increasing number of global transactions. To enable transactional order monotonicity, each of Fauna’s data centers uses a synchronization schema for sharing the log position with all query coordinators. Fauna also shares systems information publicly.

It provides security natively

Fauna has a host of database security features that you usually have to build separately for other databases. In addition to standard fine-grained attribute-based access control, Fauna offers OAuth 2.0 integration out of the box. You can use an identity provider to authenticate your users to Fauna.

Fauna’s key-based API is great at authenticating connections with particular permissions down to the level of database objects. Even after authentication, the queries use further token-based authentication on a case-by-case basis. You’d probably need to have a token lease manager like HashiCorp Vault to do this for another database. This feature isn’t available in either MongoDB or CockroachDB.

Having said that, the security implementation of both CockroachDB and MongoDB is quite advanced, as both databases provide basic authentication, encryption, and authorization—but not all features are available in all the plans. Encryption at rest in CockroachDB, for instance, is only available in the Enterprise version, while MongoDB doesn’t have that restriction. Fauna, on the other hand, provides all of these features in all its plans.

It enables more powerful queries

MongoDB and CockroachDB both have minimal support for custom or user-defined functions. Fauna uses an API-first approach to reading and writing data, meaning you can query the data using several interfaces (such as GraphQL) by exposing its API. Fauna comes with its own query language, FQL, which is an expression-oriented query language. You can also write user-defined functions (UDFs). In contrast, CockroachDB doesn’t support GraphQL by default, but you can find ways to work around that. MongoDB provides support for GraphQL via the Atlas App Services API.

In FQL, everything returns values. This enables you to use complex control structures and conditionals to process and compute your data in the shape and form you want. You can also use a mix of these interfaces to develop your applications. If you’re going to query documents, relational data, graph data, or a combination of all of those, you should be able to do it pretty easily with Fauna.

FQL also allows you to query different types of data. For instance, you can use the temporality feature of Fauna to issue point-in-time (PIT) queries. This is especially useful when dealing with time series data or streaming data, which is a real possibility with Fauna. While MongoDB natively supports streaming data, CockroachDB has limited support for streaming, which you can capitalize on by building on top.

Conclusion

There are multiple reasons why Fauna is a better choice for building highly scalable and blazing fast applications. Its Calvin-based architecture has enabled Fauna to be one of a kind, providing globally distributed transactions with the highest level of ACID compliance possible.

With its support for GraphQL and FQL, Fauna also proves to be highly developer-friendly. It gives you the option to use document-oriented database storage and retrieval while providing a relational-database-like experience.

Fauna offers fully managed provisioning, maintenance, scaling, sharding, replication, and correctness. Using Fauna, all you have to do is focus on creating great applications.

To see how Fauna can help you, request a demo or sign up and start using Fauna for free.

Kovid Rathee is a data and infrastructure engineer working as a senior consultant at Servian in Melbourne, Australia.

Serverless patterns reference architectures

Fauna — Wed, 15 Jun 2022 19:31:56 +0000

Systems design is one of the toughest parts of the software development lifecycle (SDLC). Software architects frequently rely on reference architectures because such tools help them more easily develop and maintain serverless and other complex systems.

Reference architectures for serverless applications are a high-level overview of how to solve a particular problem. By following them, you’re building on the knowledge of other teams and utilizing industry best practices. Because many of them were created by companies like Google, Microsoft, and Amazon, they also work well at a large scale.

Different reference architectures are available for different types of systems, such as a mobile or IoT backend, real-time file-processing function, or web application. Reviewing your options helps you to choose the best one for your use case. The added efficiency such architectures provide can also lower the total cost of ownership (TCO) for your organization. To further optimize your architectures, you can use Fauna, a distributed document-relational database that's ACID by default.

This article will break down serverless reference architectures and examine some common examples while also demonstrating how Fauna makes it easier to implement them.

Why do you need serverless patterns reference architectures?

Serverless app development can benefit from organized architectures and well-tested development techniques. Following are some of the major reasons why you should consider using serverless patterns reference architectures:

Optimized resource consumption

While serverless comes with its own share of resource-consumption cuts and cost optimization, you can take that even further by designing it correctly. Well-designed serverless apps automatically cut down on wasted execution cycles and provide you with the best performance-to-cost ratio.

If you’re trying to build a simple static web app, going with an n-tier application might not make sense; it might even end up adding overheads to your development process. Similarly, relying on a simple web app architecture for an app that scales to hundreds of thousands of users might not be your best choice. This is why you need to use the right reference architecture so that you can automatically optimize your app by relying on a tested pattern.

Event streaming and processing

The serverless model relies on invocations and execution cycles. When you choose to build your app using one of the popular serverless reference patterns, you get this feature built into your design. Each of your system’s components communicates with the others using events and messages. This enables them to work asynchronously and increases their resilience. If an invocation fails due to some bad values, the component doesn’t crash; it just records a failure and waits for the next event.

Event-based execution also offers powerful queuing capabilities. If any of your components receives an unusually high number of requests, it can queue them up and work according to its own capacity. No requests are lost, and your system won’t blow up. Patterns like the scalable webhook are excellent examples of how the right design can help you solve major problems with ease.

Trigger-based processing

Apps built using serverless patterns can take advantage of trigger-based processing techniques. Instead of a live application running on a server waiting for user requests to come in, your app becomes a hypothetical function that’s triggered when a request comes in. This request could be for anything from image compression to serving a full-fledged web app.

Serverless patterns reference architectures with Fauna

Now that you’ve seen how serverless patterns reference architectures can benefit you, here are five important architectures you should know. Additionally, you’ll see how these architectures are reinforced by using Fauna instead of traditional databases.

Building a basic web application

Building a basic web application that serves content to users as well as enables them to persist data into databases is simple with serverless. This is known as the simple web service pattern. Here’s how the architecture of your app would look:

The flow here is also simple:

Client sends a request to the app, which could be viewing the home page of the app or making a change in the user profile details.
The request is received by the API gateway and forwarded to the serverless function.
The serverless function interacts with the data store if necessary.
The data store returns any required data.
The function sends the response to the API gateway.
The API gateway responds to the client with the request.

This is easy to implement. As with any serverless app, the serverless function replaces a live 24-7 web app server waiting to receive requests. An important thing to realize here is that to talk to a traditional database over TCP/IP, you would have needed a 24-7 web server. Lambda functions wouldn’t support it, because they spin down as soon as they reach the idle state.

Using Fauna instead of traditional databases provides you with a quick and ready-to-go database setup. If you choose to host your app on a platform like Vercel or Netlify, Fauna offers powerful integration abilities with them and can provide you with an enriched experience.

Migrating legacy applications

A common issue with application maintenance is migrating legacy apps to newer technologies. The hard part is deciding how the migration will be carried out. In many cases, it’s not possible to decommission the old app until the new app is ready. A popular serverless pattern known as the Strangler can help you in such cases.

The idea behind this pattern is to allow the developers to migrate parts of the legacy app to the new serverless tech stack. Meanwhile, the old application runs in parallel with the new growing app. Here’s how that looks during the migration process:

You can break down your legacy app into multiple functions while migrating it to the new tech stack, as shown by the two functions above. When your API gateway receives a new request, it knows whether to route that request to one of your new functions or to your legacy app.

The data stores can be managed independently of the migration process. Once you’re done migrating your app, you can migrate your existing database using the in-depth process laid out by Fauna, which not only migrates your data but also enriches it on the way.

Once your data is migrated to Fauna, it will be highly available to each of your functions, and it will reduce the infrastructural load from your traditional database management system (DBMS).

Handling unpredictable workloads

For many apps, user traffic varies greatly. When you’re building with traditional technologies, accounting for major spikes in traffic and workload can be difficult. However, serverless was built with scalability in mind. The scalable webhook pattern serves this use case perfectly. It’s meant mostly for high-traffic webhooks, and it relies on queueing and throttling to reduce the load on the main handlers, meaning the serverless function.

Here’s what it looks like:

Here’s how the flow goes:

The client sends a request to the API gateway.
The API gateway sends the request to the first serverless function.
The serverless function adds the request to the queue and sends an acknowledgment back to the API gateway.
The API gateway sends the confirmation back to the client.
The second serverless function is subscribed to the queue and picks up on any incoming requests.
The function accesses the data store if necessary, performs the required operation, and marks the request as completed.

You can have another queue component that tracks requests marked as failed by the second serverless function. Since the second serverless function is decoupled from the primary stream of incoming requests, it can moderate the rate at which it picks up tasks, also known as throttling. You can also have more than one serverless function picking up requests from the queue if there is more than one independent operation associated with them.

Fauna can help this use case by providing a single source of truth for all of your worker functions and tying them together to achieve higher working bandwidth for your service. Moreover, Fauna would work better than other serverless databases like DynamoDB for a number of reasons, such as ACID compliance and consistency.

Tracking failed API requests

Another common use case for a web service is to serve user requests while tracking those that an external API failed to handle. This data can be used to judge whether the next calls should be sent to the external API or whether the client should just get an automated failure response. This helps reduce unnecessary calls to the external API, which would still result in a failed response but would add to your costs.

Here’s what this architecture looks like:

Here’s how it works:

The client sends the request to the API gateway, which forwards it to the right serverless function.
Before calling the external API, the function checks the data store to see if the API has been reporting failures lately. A variable (presumably Boolean) is flipped to false in the data store if the number of failure responses from the external API has crossed a certain threshold.
The false state means that the serverless function isn’t allowed to call the external API anymore. It returns with a failure response to the API gateway and subsequently to the client.
Periodically, a request is forwarded to the external API just to check if it’s available again. If yes, the Boolean variable is set to true, and all further incoming requests are allowed to go to the external API.

Fauna improves this flow by providing you with a fast and easy-to-set-up data API that you can use to implement this architecture. If your clients are distributed, using Fauna gives you an edge over an eventually consistent database since Fauna is distributed and consistent by default. If this weren’t the case, a client in another region might think that the API is active when it is already marked unresponsive from a previous status-check made from a different region.

Storing data before processing

In some cases, you might want to store user data in your database or data store before you process it. Instead of waiting for your serverless functions to finish running operations on your data, you can dump it in your DB first and then let them run. This can sometimes be necessary to reduce latency in your endpoints.

Here’s what this architecture looks like:

Here’s how it works:

The client sends a request to the API gateway, which forwards it to the first serverless function.
The function persists the data in the data store and returns a confirmation to the client.
The second serverless function picks up requests and their data from the data store, processes them, and then stores them back in.

Some might consider using a different data store for storing incoming requests and processed results, but that would just bring you closer to the scalable webhook pattern, in which the data store for storing incoming requests is a dedicated queue.

Fauna provides a highly available, fast, managed data-storage alternative that makes it easy for you to use it as the queue store as well as the results store.

Conclusion

Serverless patterns reference architectures can give you guidelines to follow in creating your own serverless projects, saving you both time and money as well as the headache of trying to address problems that others in the industry have already solved. You can build on the expertise of others to achieve more success with your code.

As you’ve seen throughout this article, Fauna can be an invaluable tool in this process. The serverless transactional database and data API help you create new applications or migrate and improve on existing projects while enabling easy scalability and auth integrations. Fauna offers event streaming, data imports, and distributed compute and storage, as well as other features to help with serverless reference architectures.

To see more of what Fauna can do for you, sign up for a demo.

Kealan Parr is a senior software engineer and technical writer who works with clients on technical reviewing and software development. He is a member of the Unicode Consortium.

Comparing DynamoDB and Fauna for multi-region data stores

Fauna — Fri, 20 May 2022 16:37:19 +0000

One of the critical architectural decisions you must make when designing modern applications is selecting the right data storage technology. Not only can this decision be expensive to change later, but it also deeply affects your application’s availability, security, and performance.

Depending on the size and reach of your organization, you might need to replicate your data to multiple geographical locations to serve a global base of users. Multi-region storage can reduce access latency, speed up recovery, and help you manage legal compliance for your data.

There are many services available to provide multi-region storage, including Amazon DynamoDB and Fauna. Amazon DynamoDB is an AWS-managed service that provides a serverless key-value NoSQL database. Fauna is a multi-region, serverless, document-relational database. Because these two services use different data storage methods, their terminology and features will vary.

This article will compare the behavior of these two NoSQL databases in multi-region deployments. You’ll learn more about how they are configured for multi-region deployments, how their distributed transaction behaviors differ, and how they select the closest replica, so that you can decide which database would be a better choice for your projects.

Configuring multi-region deployments

Running multi-region data stores comes with its own challenges. Due to the geographically distributed nature of these systems, network latency and interruptions can cause major performance and reliability issues.

By default, DynamoDB tables are deployed within a single AWS region. To achieve multi-region deployment, a global DynamoDB table has to be created instead.

A global DynamoDB table is composed of multiple regional replicas. Each one of these replicas should be created and managed independently (such as in their capacity provisioning and storage class) and should be named identically. Queries can be run on any replica, and AWS will ensure the data is eventually consistent across all the replicas.

Fauna provides a different level of abstraction. Instead of configuring each regional replica independently, each database is configured to a specific region group. Replicas can run in the United States, Europe, or across both. Fauna uses a Calvin-inspired transaction engine to provide distributed consistency and a high-performance system.

While global DynamoDB tables are available in more regions than Fauna (such as Asia, for instance), additional setup and maintenance are necessary to run them. Fauna is more straightforward to set up across multiple regions, but currently only supports the United States, Europe, or both as available Region Groups. This means your best choice here depends on which regions your organization serves. If you only do business in the US and Europe, then Fauna will work better for you. If you have users in other regions, you may prefer DynamoDB.

Using database transactions

Database transactions atomically perform database operations that affect multiple data elements like tables and fields. A database transaction’s atomicity ensures data consistency.

With DynamoDB, it’s possible to bundle independent queries via transactions. However, the best practice is to use regular, non-transactional queries wherever possible and use transactions only when necessary. ACID compliance is only guaranteed in DynamoDB transactions when performed within the same region. It’s not available for global tables.

Transactions can target up to twenty-five items in DynamoDB tables in the same region within the same AWS account. Multiple tables can be included in the same transaction. Note that read and write queries can’t be mixed on the same transaction—for example, multiple write operations can be bundled into a transaction, but a read operation will have to use a different transaction. The aggregated size for all items in a transaction can’t exceed 4 MB.

Due to the nature of DynamoDB, serializability between transactions and other queries is provided only in some cases. Developers need to ensure they understand the documentation, because caveats might apply to their systems.

On the other hand, Fauna was designed to be a fully transactional database, treating transactions as a first-class concept. Fauna’s database engine was built to provide strictly serializable transactions and guarantee short latencies even between geographically distributed replicas. The effect is that transactions can have any number of documents and can be up to 16 MB in size.

While DynamoDB is designed to run a high number of small, independent queries, Fauna is designed to support transactions to always keep a consistent state of the data. DynamoDB best practices prescribe using transactions sparsely, whereas with Fauna, all queries are considered transactions. If your application heavily depends on transactions and can’t handle incomplete or inconsistent data, Fauna may be the better choice.

Using transactions in multi-region deployments

As noted above, there are transaction limitations that apply to both single-region and multi-region data stores. Because of the distributed nature of multi-region deployments, though, there are several other caveats when using transactions.

DynamoDB transactions, as previously noted, are only applied—and relevant to—the region where the transaction was initiated. After a transaction successfully completes in the source region, the changes will be propagated to all replicas. By design, as transactions are region-specific, there’s no ACID compliance between different regions.

This means that you can potentially see partially completed transactions in different replicas until DynamoDB finishes the replication. The application needs to be architected to address such data anomalies and possible transaction conflicts. Your application needs to treat all data read from a replica as if there’s no concept of a transaction, but rather just individual, independent queries. This also needs to be addressed if you are caching the data, as you might have read inconsistent data.

With Fauna, transaction guarantees are fully supported in all configurations, including multi-region setups. Fauna’s engine will automatically ensure that no replica is exposed to transaction anomalies like dirty or phantom reads.

If your application can’t tolerate inconsistent data, or if ACID compliance is required from all replicas, Fauna will be the better choice.

Routing connections to the nearest replica

Application architectures often use multi-region data stores to reduce latency for users. This ensures that the data is stored closest to users’ geographical locations.

With DynamoDB, your application must provide a specific endpoint to choose the region for accessing the nearest replica table. Global DynamoDB tables ensure data is synchronized between replicas, but your application still needs to decide which replica to use and how to access it. That means a few more coding and configuration changes are necessary, and your application must identify the closest region and change the AWS region accordingly. For example, an EC2 instance in eu-west-1 should reach a DynamoDB replica in eu-west-2 instead of us-east-1 if those are the two regions the global DynamoDB table is deployed to.

With Fauna, no extra code or configuration is necessary to select the fastest route. Client requests will always be automatically routed to the closest replica every time.

Conclusion

This article demonstrated how DynamoDB and Fauna differ in multi-region setups. Though both services are strong choices, as you saw, they are each best for different use cases.

If your application needs strong ACID compliance across multi-region replicas, Fauna may be a better choice. It eliminates data anomalies in replicas by serializing transactions. Fauna offers fewer regions than DynamoDB, but it requires less configuration tweaking and maintenance. Additionally, it can configure multi-region setups and routing connections transparently.

If you’re interested in learning more about Fauna, its developer-friendly features, and data API, you can sign up for free to give it a try.

Cintia Del Rio helps companies improve their infrastructure in the cloud. An engineering manager at Envato, she has been working with infrastructure and DevOps for more than ten years, and before that she was a developer. She has been the lead in infrastructure for the OpenMRS open source community for the past seven years.

Introduction to serverless databases

Fauna — Thu, 12 May 2022 18:51:15 +0000

Serverless is a new paradigm in which servers operate on a more automated level, freeing developers from the time and effort of managing them. This is an advantage when it comes to development because developers and engineers don’t need to handle as much in terms of infrastructure, which can be time-consuming and expensive if you don't have the necessary in-house expertise.

Similarly, a serverless database takes the features of a traditional database and combines them with the values and flexibility of a serverless architecture. Working with a serverless database reduces much of the complexity of a database into a simple cloud-based API. It can provide an organization with more automated scaling, stronger resilience, and reduced time to market.

In this guide, you’ll learn what a serverless database is and how it works, as well as more about the benefits it can offer to your organization.

What is a serverless database?

A serverless database is any database that grows automatically to meet the changing demands of an application and manages unexpected workloads that cannot be predicted or scheduled.

The benefits of serverless computing include only paying for the resources you use, scaling up and down to match demand, eliminating the need to manage servers, and lowering costs. If you use a non-serverless database in a serverless computing architecture, you lose these advantages. The major feature of a serverless database is its ability to adjust capacity based on its workload.

A serverless database works when and wherever it is needed. A service provider will manage the database for you, including the provisioning of instances or clusters.

Why should you use a serverless database?

A serverless database enables developers to work on projects without needing a specialized hardware platform or needing to worry about getting necessary resources for their applications.

When you use a serverless database, you avoid several problems associated with a traditional setup. You save on costs because you only pay for what you use, and you save time because you don’t have to spend it patching, provisioning, and managing servers. A serverless database also improves security by ensuring that all applications interacting with the same data set pass the same access control, thus reducing the attack surface.

A serverless database can be used for prototype testing, automatic website scaling, and continuous integration/continuous deployment (CI/CD) practices.

There are several key features of a serverless database:

Automatic elastic scaling and geographical distribution: You don’t have to worry about sharding or whether your database can handle a sudden spike in traffic, because a serverless database scales up and down automatically to meet demand. This means scaling all the way down to zero when the database isn't being utilized while promptly responding when a query arrives. A serverless database can also scale geographically, moving and storing data dynamically around the globe to minimize latency and give users worldwide a consistently fast experience.
Native resilience and fault tolerance: You also won’t have to worry about faulty storage nodes or zone disruptions bringing your services down. A serverless database can survive node and zone outages, as well as implement software updates and online schema changes, without any planned downtime required.
Simplicity and familiarity: Working with your database is as simple and intuitive as using an API. With features such as self-service starts, completely managed operations, and the ability to create clusters with the press of a button or a single command, a serverless database makes life simpler for everyone who works with it.
Consumption-based billing: You don’t pay for storage and compute resources you’re not using. A serverless database only bills you for resources you used, and you can set spending limits so that you don’t overrun your budget.
ACID: You don’t have to sacrifice consistency for scale. Some serverless databases like Fauna provide the needed atomicity, consistency, isolation, and durability (ACID) properties to your transactions without sacrificing speed, no matter what scale you’re operating at.

Serverless databases are ideal for smaller businesses because they don’t require much maintenance, infrastructure support, or labor time. They work well for the following use cases:

Prototyping and testing: With increasing competition across all industries, it is critical to produce prototypes and gather meaningful feedback from users as soon as possible. Since serverless database features such as self-serve, speed, and pay-per-use billing are both economical and relatively fast to implement, you can focus on writing code and responding to user feedback in the testing environment.
Reduced time to market: Serverless databases can help you drastically reduce time to market. Instead of performing complex deployment procedures to push out bug patches and new features, you can add and alter code as needed.
Automatic scaling: A serverless database can be utilized with existing or new applications that have unpredictably high database demand; it automatically scales up when needed and down when not. You don’t need to spend time configuring an autoscaling policy for your database system.
Managing resource demands: With a serverless database, you don’t have to worry about allocating enough resources to fulfill inconsistent resource requests. A serverless database will expand automatically when needed.

Examples of serverless databases

Amazon Aurora Serverless is a proprietary service from AWS that’s compatible with Postgres and MySQL, which means you can connect to your Aurora database as if you’re connecting to Postgres or MySQL. It is also AWS cloud optimized. Aurora storage automatically grows in increments of 10 GB, up to 64 TB.
Google Firestore is a serverless document database that provides direct database access for web, IoT, and mobile app development. It’s highly scalable with no maintenance window and zero downtime. Firestore enables offline data access for web and mobile SDKs, enables ACID-compliant transactions, supports multiple server-side development libraries and programming languages, enables data validation and identity-based security access controls, and offers real-time data synchronization with offline data access.
Amazon DynamoDB is a NoSQL database service with single-digit millisecond response times. AWS manages everything, allowing you to store as much data as you need while also handling unpredictable demands. It's also a fully managed NoSQL database service with built-in scalability.
CockroachDB, an SQL relational serverless database, is considered one of the most evolved serverless databases. It offers a completely elastic and robust data architecture, distributed globally to help developers rapidly develop apps at a low cost. It is a single Postgres instance in many aspects.
Fauna is a versatile transactional database that’s supplied as a secure, scalable cloud API with native GraphQL. Fauna blends the flexibility of NoSQL systems with SQL databases’ relational querying and transactional capabilities. It supports drivers for languages including Python, Java, Scala, and GraphQL, and promises full ACID compliance.

How Does a Serverless Database Work?

For an example of a serverless database, look at Fauna. It offers on-demand autoscaling, which means that the database starts up, grows capacity based on the demand of your application, and shuts down when not in use.

Fauna is a multi-model database with a data API for client-serverless applications. Its semi-structured data model supports relational, document, object-oriented, and graph data.

Fauna is also a NoSQL database, which means you can’t use SQL to access it. Instead, your primary interface is the Fauna Query Language (FQL). FQL is not a general-purpose programming language, but it allows for advanced data processing and retrieval from Fauna. All functions, control structures, and literals return values in the language, which is expression-oriented. This makes it simple to combine results into an array or object, or to map over a collection and calculate a result for each member of your team.

You run a query by sending it to a Fauna cluster, which computes and provides the results. Query execution is transactional, which means that if anything goes wrong, no changes are committed. If a query fails, you get an error rather than a result.

Fauna’s serverless database design is multi-tenant, unlike the single-tenant design of a traditional database. In a single-tenant architecture, you pay for the server’s entire storage and processing capabilities even if you only utilize a tiny piece of it. In a multi-tenant architecture, you share that server with other users and only pay for the database storage that you use, which helps to decrease costs. Many database systems enable multi-tenancy, but Fauna goes a step further by allowing every database to have numerous offspring databases. This means you can administer a single large Fauna cluster, build some top-level databases, and grant teams complete administrative access to those databases. You can establish as many databases as you need without requiring an operator’s assistance.

Data protection and security safeguards are a crucial aspect of serverless databases. In Fauna, security is implemented at the API level with access keys to authenticate connections. This access key mechanism applies to both administrator and server-level connections, as well as to object and user-level connections.

What are the benefits of serverless databases?

Cost-effectiveness and high scalability: You will save time and money by using a serverless database, which eliminates the need for a license, installation, cabling, configuration, maintenance, and support. Scaling computing systems do not need to be set up manually by operators or developers. Furthermore, you can use resources to construct apps in the most efficient way possible based on your needs.
Better resilience and availability: A serverless database is duplicated across multiple regions, so if a storage node fails, all queries will be diverted to other nodes with load balancers until the node recovers, making it resilient and highly available. A serverless system is more adaptable than a traditional system in general. Serverless databases enable you to reduce latency, and data from event-driven functions is read where it is closest to the user.
Increased productivity and better developer experience (DX): Developers don't have to worry about provisioning, configuring, or managing database infrastructure while using a serverless database. Developers just need to concentrate on creating applications, which increases productivity and provides a better DX.

Conclusion

Serverless databases can be a real asset to developer teams. They can increase your compute speed and resilience while decreasing the amount of time and money you spend on resources and scaling. Implementing a serverless database can vastly improve the DX at your organization.

If you’re looking at options for a serverless database, consider Fauna. The transactional database uses a cloud API to provide simple, intuitive access to your data. It supports real-time streaming and GraphQL, and you can scale it globally as needed. To see how Fauna can help you, sign up for a free account.

This post was written by Segun Saka-Aiyedun. Segun is a cloud architect, DevOps enthusiast, and Manchester United fan.