Forem: moga

Firestore (Native/Datastore) triggers Cloud Run

moga — Fri, 17 Sep 2021 08:09:27 +0000

Introduction

In Firebase Firestore, Firebase Functions can be triggered by writing a document in a specific path and perform arbitrary processing. I was wondering if it would be possible to do something similar to this by launching Cloud Run. I thought it would be a good idea, so I decided to write about it. Unless otherwise stated, I'll be talking in Native mode.

I was interested in it when I heard Eventarc's presentation at Google Cloud's Open Cloud Summit, which led me to actually touch it.
https://cloudonair.withgoogle.com/events/open-cloud-summit?talk=d2-04

Requirements

I want to be able to do at least the same things as in the case of Firestore triggers Functions.

Triggers can be set for specific collections.
Create/Update/Delete (or Write) as trigger events.

Conclusion

Satisfy above requirements
- Firestore Data Access Audit Log → Cloud Logging Router Sink → Pub/Sub → Cloud Run
Not satisfy
- Eventarc's Firestore Write trigger → Cloud Run
A forced method I used to use before Audit Log was released
- Firestore Triggered Functions → Cloud Run

I will explain each method (except the last one).

Assumption

First of all, Firestore's data access audit log was recently released in preview. The first and second methods are based on this. Note that this is a preview version and the specifications may change.
https://cloud.google.com/firestore/docs/audit-logging#data-access-audit-logs:-data_read-and-data_write

Using Eventarc

I'll start by explaining how I tried this and how I solved the problem that came up in this section. If you are not interested, please go to the next section.

Configuration and Operation

The official blog post about Eventarc is below. In a nutshell, you can use Audit Log or Pub/Pub topic as a source (trigger) and send events to Cloud Run with the information in CloudEvents format. Cloud Events is a specification of event data used in event-driven architecture.
https://cloud.google.com/blog/ja/products/serverless/eventarc-unified-eventing-experience-google-cloud

To set up Eventarc, go to the Cloud Run service page. To configure Eventarc, go to the Cloud Run service page, open the triggers tab, and you'll see an "Add Eventarc Trigger" button that allows you to configure various settings. You can select the service or method to be triggered from among the many available ones, and specify which resource or region to be triggered and the service account to be used when invoking it. Finally, you can specify the path to this Cloud Run.
!

This is what happened when I logged the Body and Header of the request that came from Eventarc on the server I set up for testing. I'm omitting information that may not be necessary, such as YOUR_*. The following is the case of creating a document in /samples/:id from the console of Firestore (Native).

{
  "body": {
    "resource": {
      "labels": {
        "service": "firestore.googleapis.com",
        "project_id": "YOUR_PROJECT_ID",
        "method": "google.firestore.v1.Firestore.Write"
      },
      "type": "audited_resource"
    },
    "logName": "projects/YOUR_PROJECT_ID/logs/cloudaudit.googleapis.com%2Fdata_access",
    "protoPayload": {
      "status": {},
      "requestMetadata": {
        "destinationAttributes": {},
        "callerSuppliedUserAgent": "YOUR_UA",
        "requestAttributes": {
          "auth": {},
          "time": "2021-09-17T01:51:32.185016Z"
        },
        "callerIp": "YOUR_IP"
      },
      "authenticationInfo": {
        "principalEmail": "YOUR_EMAIL_ADDRESS"
      },
      "serviceName": "firestore.googleapis.com",
      "resourceName": "projects/YOUR_PROJECT_ID/databases/(default)",
      "serviceData": {},
      "methodName": "google.firestore.v1.Firestore.Write",
      "metadata": {}
        "@type": "type.googleapis.com/google.cloud.audit.DatastoreServiceData"
      },
      "authorizationInfo": [
        {
          "granted": true,
          "permission": "datastore.entities.create",
          "resourceAttributes": {},
          "resource": "projects/YOUR_PROJECT_ID/databases/"
        },
        {
          "granted": true,
          "resource": "projects/YOUR_PROJECT_ID/databases/",
          "permission": "datastore.entities.update",
          "resourceAttributes": {}
        }
      ],
      "request": {
        "database": "projects/YOUR_PROJECT_ID/databases/(default)",
        "@type": "type.googleapis.com/google.firestore.v1.WriteRequest",
        "writes": [
          {
            "update": {
              "name": "projects/YOUR_PROJECT_ID/databases/(default)/documents/samples/ewMoAiRAA1t43S1PnSAv"
            }
          }
        ]
      }
    },
    "insertId": "-5jrcl3eizimu",
    "severity": "INFO",
    "timestamp": "2021-09-17T01:51:32.162978Z",
    "receiveTimestamp": "2021-09-17T01:51:32.436039856Z"
  },
  "headers": {
    "ce-methodname": "google.firestore.v1.Firestore.Write",
    "accept": "application/json",
    "x-forwarded-proto": "https",
    "ce-id": "projects/YOUR_PROJECT_ID/logs/cloudaudit.googleapis.com%2Fdata_access-5jrcl3eizimu1631843492162978",
    "ce-dataschema": "https://googleapis.github.io/google-cloudevents/jsonschema/google/events/cloud/audit/v1/LogEntryData.json",
    "traceparent": "00-8c6fc1640a9b6125909293da6aa41d0f-d0fbedd2fab6765c-01",
    "content-type": "application/json; charset=utf-8",
    "from": "noreply@google.com",
    "ce-source":"//cloudaudit.googleapis.com/projects/YOUR_PROJECT_ID/logs/data_access",
    "ce-specversion": "1.0",
    "ce-subject": "firestore.googleapis.com/projects/YOUR_PROJECT_ID/databases/(default)",
    "forwarded": "for=\"34.116.20.31\";proto=https",
    "host": "renderer-54pvsxgf2q-an.a.run.app",
    "x-cloud-trace-context": "8c6fc1640a9b6125909293da6aa41d0f/15058891269448562268;o=1",
    "content-length": "1566",
    "accept-encoding": "gzip, deflate, br",
    "ce-recordedtime": "2021-09-17T01:51:32.162978Z",
    "ce-time": "2021-09-17T01:51:32.436039856Z",
    "ce-type": "google.cloud.audit.log.v1.written",
    "ce-servicename": "firestore.googleapis.com",
    "user-agent": "APIs-Google; (+https://developers.google.com/webmasters/APIs-Google.html)",
    "x-forwarded-for": "34.116.20.31",
    "ce-resourcename": "projects/YOUR_PROJECT_ID/databases/(default)",
    "authorization": "Bearer YOUR_JWT"
  }
}

The body seems to contain the Audit Log of Firestore. The ce-* in headers seems to be specific to Cloud Events. The authorization is filled with JWT, so the application can validate it to determine if it is the right caller (if you allow unauthenticated).

Difficulty points

It is difficult to realize you can set a trigger for a specific collection. You may think that you can do it by specifying the Specific resource of the resource in the configuration screen, but this only filters for the protoPayload.resourceName of the request body. However, this only filtered for protoPayload.resourceName in the request body. In this state, all document write-related operations are requested to a single endpoint. "If you're thinking, "My service doesn't have many writes, so I'll handle it on the application side! I think this is fine.

This may be a problem specific to the current situation, but I think it may be improved in the future.

Using Cloud Logging's Log Router Sink (how it worked)

Since the contents of the above request body are flowing as an Audit Log, you can create a filter that matches the contents you want to pick up, send it to Pub/Sub, and then use Cloud Run to pick it up. The rest can be picked up by Cloud Run.

First, enable the Firestore Data Access Audit Log.
!

Next, create a Pub/Sub topic. Create a subscription to that topic, with the endpoint set to your own server, while making it a push type.

Finally, in Cloud Logging, create a Logging Sink. The topic should be the one you created in 👆.
!

The filter should be something like 👇 so that it can pick up only the Create in /samples/:id. The detailed notation is written in official here. (One annoying thing is that currentDocument.exists may or may not exist for each field, depending on whether it was created from the Admin or Client SDK.

protoPayload.methodName = "google.firestore.v1.Firestore.Write" AND
protoPayload.request.writes.update.name =~ "projects\/YOUR_PROJECT_ID\/databases\/\(default\)\/documents\/dosamples\/[^/]+$" AND
(NOT protoPayload.request.writes.currentDocument:* OR protoPayload.request.writes.currentDocument.exists = false)

If you can set it up so far, the following request body will be sent when you create a Firestore document.

json { "message": { "data": "[Audit Log encoded in BASE64]", "messageId": "3065956847068101", "publishTime": "2021-09-17T05:51:40.637Z", "publish_time": "2021-09-17T05:51:40.637Z", "message_id": "3065956847068101", "attributes": { "logging.googleapis.com/timestamp": "2021-09-17T05:51:39.061628Z" } }, "subscription": "projects/YOUR_PROJECT_ID/subscriptions/YOUR_SUBSCRIPTION_ID" } }.

That's it. Now you just need to create a sink for each trigger you need. I think it's a lot of work, so it's better to use Terraform to manage the code and add more by copy and paste.

Conclusion

I'm thinking, "Wouldn't it be easier to send Create/Update/Delete events to Pub/Sub when the logic runs from the Cloud Run application code?
→ That's exactly what I'm thinking (although if you want to run the application when the data here is created, this method is not a bad idea).

What I haven't tried yet

PubSub seems to have a function to filter messages. If you use this, you may not need any sinks. It depends on how detailed the filter is.
https://cloud.google.com/pubsub/docs/filtering?hl=ja

Export Amazon Aurora to BigQuery using BigQuery Data Transfer Service

moga — Sun, 29 Aug 2021 02:11:33 +0000

What I want to do

I want to use BigQuery to analyze data from Google Analytics and services.
To do this, I want to transfer data from Amazon Aurora to BigQuery.
I want to transfer the data from Amazon Aurora to BigQuery less than once a day.
I don't want to add too much load to Aurora.
I want to write as little code as possible.

Policy

In order to reduce the load on Aurora, I thought that I could use the daily snapshots that Aurora automatically takes. Lucky for us, we have this snapshot. Lucky for us, these snapshots can be exported to S3 (which seems to have become possible around January 2020). This feature will export the snapshot in PARQUET format. This feature exports snapshots in PARQUET format, which is a very efficient format in terms of data volume.
https://docs.aws.amazon.com/ja_jp/AmazonRDS/latest/AuroraUserGuide/USER_ExportSnapshot.html

If you have a PARQUET file in S3, you can send it to BigQuery using BigQuery Data Transfer Service. BigQuery Data Transfer Service can read PARQUET files in S3 and send them to BigQuery. Moreover, it can be executed periodically if the interval is more than 24 hours, such as daily. The service itself is free of charge, although you will have to pay for the data transfer.
https://cloud.google.com/bigquery-transfer/docs/s3-transfer-intro

So, combining the two, the following is the process.

launch Lambda, etc. at any time on a daily basis
export the daily snapshot that Amazon Aurora is taking automatically to S3
load the file with GCP's BigQuery Data Transfer Service

I recommend you to try this flow from the console before implementing it, so you can see how the data is exported to S3, what you need to export, and how the Transfer Service works. Basically, just follow the documentation, but there are a few things I got stuck on.

Try it from the console

I poked and prodded from the console, and it was easy to dump the Aurora snapshot to S3. But it took a lot of time, about 30 minutes. I wonder if it takes time to convert the file format. I don't know the details.

When I tried to read one of the PARQUET files with Transfer Service, I got an error like PERMISSION_DENIED: [AccessDenied: Access Denied] and could not transfer it.

It should have been enough because I have S3ReadOnlyAccess on it (it says so in the documentation).
AWS CloudTrail shows an error that kms:Decrypt is missing.
Add kms:Decrypt to the AWS user permissions for BigQuery Data Transfer Service
- The target is the KmsKeyId specified in the StartExportTask that you will see later (👇).

If you do this, you can transfer the data. By the way, you need to prepare the dataset and table in BigQuery beforehand. It seems that the data is always imported with WRITE_APPEND, so it is better to set the partition at the import time in the table. First, I created an empty table with no schema.

When the data was transferred to BigQuery, the schema types were reasonably consistent and all were nullable.

The data transferred to BigQuery had a few schema types, but all of them were nullable. As far as I can see, the S3 URI setting for the transfer allows wildcards, but it ends up being shoved into a single table, so I decided that I need to set up one transfer per table in Aurora.

By the way, I tried to use wildcards to shove multiple RDB tables into one BigQuery table, but the file transfer from S3 to GCP worked, but the read did not (0 jobs succeeded).

Automate

Aurora Snapshot to S3

To automate the part of exporting Aurora snapshot to S3, we need to write some code, which looks like this when implemented in TypeScript.

If you implement it in TypeScript, it will look like this:

import { DescribeDBClusterSnapshotsCommand, RDSClient, StartExportTaskCommand } from '@aws-sdk/client-rds'

export const requestToExportSnapshotToS3 = async () => {
  const client = new RDSClient({ region: 'Aurora's region' })

  // retrieve the most recent Snapshot

  const describeCommand = new DescribeDBClusterSnapshotsCommand({
    DBClusterIdentifier: 'Aurora's DB cluster ID',
  })
  const describeResult = await this.client.send(describeCommand)
  if (!describeResult.DBClusterSnapshots) return
  const latestSnapshot = describeResult.DBClusterSnapshots.sort(
    // We want to sort in descending order
    (a, b) => b.SnapshotCreateTime?.getTime() ? 0 - (a.SnapshotCreateTime?.getTime() ? 0)
  )[0].
  const snapshotArn = latestSnapshot?.DBClusterSnapshotArn
  if (!snapshotArn) return

  // Request Export
  // more details in the 👇 documentation
  // https://docs.aws.amazon.com/AmazonRDS/latest/APIReference/API_StartExportTask.html

  // taskID must be a date, because only date/time variables can be used in the Transfer Service.
  Const taskID = `export-id-YYYYMMDD`.
  const exportCommand = new StartExportTaskCommand({
    ExportTaskIdentifier: taskID,
    // You can create it when you do S3 export from the 👇 console, so it's easy to use it.
    IamRoleArn: 'Role for S3Export',
    // You can specify database, table, etc.
    ExportOnly: ['database_name'],
    S3BucketName: 'bucket to export to',
    // I could have used any custom key, but I chose the key used for DB encryption
    KmsKeyId: 'ID of the encryption key',
    SourceArn: snapshotArn,
  })

  return this.client.send(exportCommand)
}

You can do this by using CloudWatch and Lambda. The important thing is the permissions of the role that executes the process.

Add the role that executes the Export process to the KMS key user that is used for DB encryption (user/role that hits the SDK).
- To get KMSKeyNotAccessibleFault.
Allow the following to the role that executes the export process 👆.
- kms:DescribeKey (the above key is the target resource)
- kms:CreateGrant (the above key is the target resource)
- iam:PassRole (required when running StartExportTask)
- rds:StartExportTask.
- rds:DescribeDBClusterSnapshots (to get the Arn of the target Snapshot)

If we give these, we could run.

Configure BigQuery Data Transfer Service

The only thing you need to think about is the S3 URI. The implementation in 👆 looks like this. The run_time variable is in UTC, so you need to match it with the YYYYMMDD of the taskID that you set in 👆 depending on the timing of the transfer service.

s3://your_bucket_name/export-id-{run_time|"%Y%m%d"}/database_name/schema_name.table_name/*.parquet

Now we can run it and the data will be in the BigQuery table we specified.

Conclusion

CloudTrail has helped me a lot. CloudTrail is a great service that can be used for debugging.

When I looked at the export from Aurora, I wondered if it was possible to do it in DynamoDB. It may be more complicated than this case because it may require a data format conversion layer.

{"Item":{"id":{"S": "83369704-ccef-4163-8c20-24d240c10806"}, "name":{"S": "Leopoldine Blümel-Mude"}}
{"Item":{"id":{"S": "1f187e30-b279-4bfa-85dc-e347092f364f"}, "name":{"S": "Theo Trommler"}}
{"Item":{"id":{"S": "88942fa5-e8be-4b29-8869-6abdc662eaea"}, "name":{"S": "Herr Franz-Peter Thanel B.A."}}

https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_ExportTableToPointInTime.html

Firestore cursor-based pagination on the API server

moga — Mon, 16 Aug 2021 11:52:28 +0000

Assumptions

Firestore is used on your API server
We want to use cursor-based pagination instead of Offset-based pagination in the API server
- The client sends the cursor string to the API server, which then retrieves the continuation and returns it

What I want to do

I want to modularize the pagination process using Firestore as a data source in the API server.
The module requires cursor, limit, and Firestore of Query like firestore().collection('posts').where(...).orderBy(...)
At a minimum, the followings will be returned as a result
- An array of the documents you retrieved
- Whether there is next data (hasNextPage)
- The cursor of the last document

Difficulties

The startAfter/startAt used for pagination in Firestore can be specified by "the value of the field specified by orderBy" or "a snapshot of the document".

In the former case, the type of the cursor to be passed depends on what orderBy is specified. In TypeScript, it can be string, number, firestore.Timestamp, etc., but it is more practical to use string for returning the cursor to the API client (the cursor for pagination in Relay in GraphQL is a string). However, in order to make pagination processing common, it is complicated to say "the type of the cursor in this query is XX, so it must be converted like this".

In the latter case (document snapshot), the snapshot is an object and is not suitable to be converted directly into a cursor (string). So, I came up with the idea to convert the path of the document into a cursor.

How to do it

The following is an example of TypeScript code (roughly equivalent to GraphQL's Relay style cursor pagination).

import { firestore } from 'firebase-admin'

// base64 encode the snapshot's path
const encodeCursor = (snapshot: firestore.DocumentSnapshot | firestore.QueryDocumentSnapshot) => {
  return Buffer.from(snapshot.ref.path).toString('base64')
}

const decodeCursor = (cursor: string) => {
  return Buffer.from(cursor, 'base64').toString('utf8')
}

type Connection = {
  nodes: { id: string }[].
  pageInfo: {
    hasNextPage: boolean
    endCursor?: string | null
  }
}

export const paginateFirestore = async (query: firestore.Query, limit: number, cursor?: string | null): Promise<Connection> => {
  // get one more item for hasNextPage
  let q = query.limit(limit + 1)

  if (cursor) {
    // If a cursor is passed, convert it to a path and get a snapshot of the document
    const path = decodeCursor(cursor)
    const snap = await admin.firestore().doc(path).get()

    if (!snap.exists) {
      return { nodes: [], pageInfo: { hasNextPage: false }
    }

    // pass to startAfter
    q = q.startAfter(snap)
  }

  const snapshot = await q.get()
  const hasNextPage = snapshot.size > limit
  const docs = snapshot.docs.slice(0, limit)

  // make the path of the last document a cursor
  const endCursor = hasNextPage ? encodeCursor(docs[docs.length - 1]) : null

  return {
    nodes: docs.map(doc => ({ id: doc.id, . .doc.data() })),
    pageInfo: {
      hasNextPage,
      endCursor,
    },
  }
}

The usage is as follows.

const query = firestore().collection('posts').orderBy('createdAt', 'desc')
const connection = await paginateFirestore(query, 100, args.cursor)

Advantages and disadvantages.

Advantages

Simplifies the process because the cursor will always be path.
Clients only needs to pass the generated cursor.
Pagination by snapshot is more accurate than pagination by orderBy field.

Disadvantages

Overhead from getting one extra item.

What do you think?

I thought that the overhead of retrieving one extra item would not have a big impact on gRPC, so I implemented it this way, prioritizing the simplicity of the code. If you have any thoughts on this, I'd love to hear your feedback!

Decrease read costs of firestore using Firestore Data Bundles

moga — Wed, 14 Jul 2021 07:29:47 +0000

A feature called Firestore Data Bundles was released around December 2020, but it hasn't been talked about much. I've been looking into it because it hasn't been talked about much. I'm sure there are many people who don't know about it.

Here are the blogs and documents I used for reference. I recommend the official blog as it is easy to understand and includes usage scenarios.

Reference

Official blog
- Load Data Faster and Lower Your Costs with Firestore Data Bundles!
Official documentation
- Cloud Firestore data bundles
- Serve bundled Firestore content from a CDN

What is Firestore Data Bundles?

In Firestore, reading occurs for each user (device) even if all users read the same data👇. As shown in this figure, if 4 users have 100 Reads each, you will be charged for 400 Reads in total.

Using Firestore Data Bundles, you can dramatically reduce the number of reads by delivering reads from Firestore in the form of Bundles through a CDN 👇 (the figure is just an image). Moreover, the devices can generally retrieve the data through the interface provided by Firestore.

In the figure 👆, Functions is behind CDN (and so is the official documentation), but you can use Google Cloud Storage, etc. without any problem.

Implementation

The code is written in TypeScript. It is written in TypeScript, so you can rewrite it as you see fit.
https://github.com/mogaming217/firestore-data-bundles-sample

This time, we will replace the CDN + Functions part with Google Cloud Storage(GCS). We will create a Bundle from the local machine, upload it to GCS, and read the file.

Create and upload Bundle

import { firestore, storage } from "firebase-admin"
import * as fs from 'fs'

// We're just initializing the project.
import { initAdminProject } from ". /helper"
initAdminProject()

const BUCKET_NAME = 'YOUR_BUCKET_NAME'
const CREATE_INITIAL_DATA = false

const main = async () => {
  const db = firestore()
  const timestamp = Date.now()

  if (CREATE_INITIAL_DATA) {
    // create 100 data items
    await Promise.all([. .Array(100)].map((_, i) => {
      return db.doc(`bundles/data_${i}`).set({
        body: `${i}`.repeat(1000).slice(0, 1000),
        Timestamp: firestore.Timestamp.fromMillis(timestamp + i * 100)
      }))
    })))
  }

  // Read the data from the firestore and create a Data Bundle
  const snapshots = await db.collection('bundles').orderBy('timestamp', 'asc').get()
  const bundleID = timestamp.toString()
  const buffer = await db.bundle(bundleID).add('bundles-query', snapshots).build()

  // write them out locally for upload
  const bundledFilePath = `. /${timestamp}.txt`.
  fs.writeFileSync(bundledFilePath, buffer)

  // Upload the file to GCS
  const destination = `firestore-data-bundles/bundles.txt`
  await storage().bucket(BUCKET_NAME).upload(bundledFilePath, { destination, public: true, metadata: { cacheControl: `public, max-age=60` } })

  console.log(`uploaded to https://storage.googleapis.com/${BUCKET_NAME}/${destination}`)
  process.exit(0)
}

main()

The db.bundle(bundleID).add('bundles-query', snapshots).build() part creates the Bundle. db.bundle() returns the BundleBuilder and add adds to it. In this case, we will only add one QuerySnapshot, but you can call .add() as many times as you want, and you can also pass a DocumentSnapshot. I didn't see any indication of how much you can pack in, but considering that the client will be downloading it, I'd say a few MB is the upper limit.

Also, the bundleID is once set to timestamp, but the role of this ID seems to be used to determine if the Bundle has already been retrieved by the client or not. This ID is used to determine if the bundle has already been retrieved by the client.

The ID of the bundle. When loaded on clients, client SDKs use this ID and the timestamp associated with the bundle to tell if it has been loaded already. If not specified, a random identifier will be used.

As a reminder, you need to use the Admin SDK to create the bundle. Please do not put any secure data in the file.

Bundle Reading

The following example shows how to use.

import axios from "axios"
import { initClientProject } from ". /helper"

const app = initClientProject()

const BUNDLE_URL = 'UPLOADED_BUNDLE_URL'

const main = async () => {
  const db = app.firestore()
  const now = Date.now()

  // Get Bundle data from GCS and load it
  const response = await axios.get(BUNDLE_URL)
  await db.loadBundle(response.data)

  // retrieve from the loaded Bundle data
  const query = await db.namedQuery('bundles-query')
  const snaps = await query!.get({ source: 'cache' })

  console.log(`${(Date.now() - now) / 1000}s`)
  process.exit(0)
}

main()

Get the Data Bundle from GCS. It will be a simple GET request, so caching with CDN or other means will be effective.

If you load it with loadBundle, the Data Bundle will be expanded as a local cache of the device as well. We dare to use the namedQuery method to load it, but you can also load the loaded Bundle with db.collection('bundles').orderBy('timestamp', 'asc').get({ source: 'cache' }). The reason why we use source: 'cache' is because we wanted to make sure to read from the unpacked cache. You may want to change this to suit your logic.

Advantages

Cost savings

As mentioned above, the number of Firestore reads can be reduced, thus reducing the running cost. The figure below shows a comparison between the case where each device reads directly from Firestore and the case where Firestore Data Bundles are used (costs of storage). 1 document is 1KB and 100 data are read per user. The following is a comparison of what the price will be as the number of users increases.

Firestore is very inexpensive to begin with, so there is not much difference, but it is still more than 50% cheaper to deliver via GCS. The more users you have, the more effective it will be.

The calculation formula is as follows. The calculation is based on the Tokyo region. If I'm wrong, please let me know.

Firestore
- Number of reads / 100,000 x $0.038
- Ignore the free reads.
- If the transfer volume exceeds 10GB, add exceeded transfered data (GB) x $0.14 to the above.
- Reference: https://cloud.google.com/firestore/pricing
Google Cloud Storage
- Amount transferred (GB) × $0.12
- Ignore read and storage charges for Bundle creation.

Improve read speed (there is data that can be read).

The official documentation mentions that the read speed will be faster not only the first time.

While the developer benefits from the cost savings, the biggest beneficiary is the user. Studies have repeatedly shown that speedy Studies have repeatedly shown that speedy pages mean happier users.

Data Bundle is not affected by this issue because it can read from CDN, load into Firestore, and then fetch from local cache. Data Bundle is not affected by this issue. There may be situations where this will work effectively. I tried to verify how much faster this would be, but I couldn't get it to work, probably because the Firestore connections are pooled on the machine to some extent when running repeatedly. Please note that the above is only theoretical. The maximum measured value is shown below.

Getting 100 items of data from Firestore (first time): 1.4s
When retrieving data from GCS by loading the Data Bundle that contains 100 data: 0.7s
- (The above is the case when the data was not hit in the CDN cache, and when it was, it was 0.2s or so.

Disadvantages

It is a bit more complicated to implement than reading directly from Firestore. As mentioned in the official blog, this is a bit of an advanced feature, so it won't be very effective unless you have a large number of users.

Also, if you make a mistake in handling the cache, you may see data that you shouldn't, and many other accidents can happen. Be sure to understand how to use it.

Usage scenarios

This is a brief description of what is described in the official blog.

The following are some examples of how you might use it.

When you have data that all clients need to read when launching an application (master data, etc.)
Top 10 articles that will be read by all clients in news, blog services, etc.
Starter data that will be read by non-logged-in users

On the other hand, you should not use the following

Queries that produce different results for different users (such as data in a sub-collection of the users collection)
Data that contains private information.

Personal impressions, etc.

The response caching by CDN that was done with the so-called general Web API can now be done with Firestore. I thought for a moment. However, the fact that the data is expanded as Firestore's local cache, and the fact that there is no need to convert timestamps, although it is a minor detail, made me think that there are quite a few advantages to using this.

If I were to use it, I would use it for ranking data as well as master data. In that case, I would place the Data Bundle in the GCS at the timing when the ranking is updated, so that the client always sees the same GCS URL, and handle it by adjusting the bundleID and Cache-Control.

However, although it looks very good, I have the impression that this is something that has not yet been talked about because it is something that will not be very effective until the number of users becomes large. When I look at the actual price, I feel that if there are enough users to make it effective, then the profit will be higher, and the change in operating cost will be like a margin of error when Data Bundle is introduced.... It's hard to say. I think it's perfect if you have a lot of users and you want to cut down on operating costs.

By the way, Flutter's SDK support tends to be slow for new features like this, but it seems to be already supported (https://firebase.flutter.dev/docs/firestore/usage/#data-bundles). (It's been a while since the release).

Conclusion

I often talk about Firebase and other technical topics on Twitter, so please follow me if you like!