Forem: ente

Understanding Keys, Elements, RenderObjects and their interplay in Flutter

ente — Thu, 28 Sep 2023 08:22:37 +0000

Understanding the inner workings of Flutter is crucial to crafting bug-free and performant apps. While working on performance optimisation of ente's mobile app, I encountered some intriguing behaviour, prompting me to write this blog. This blog isn’t focused on how to make performant apps but understanding the concepts written here will indirectly help with it. In this blog post, I’ll primarily delve into,

How Elements and RenderObjects are reused in Flutter.
An overview of the three trees in Flutter.
An example that illustrates a state management issue which can be easily debugged from the insights of the first two.

Links to resources that might help you understand this blog better if you find it difficult is at the end.

The Example

Let’s create a StatefulWidget for a container which accepts a parameter name and has a colour which is randomly generated. The colour will be stored in its state.

class CustomContainer extends StatefulWidget {
  final String name;
  const CustomContainer(this.name, {super.key});

  @override
  State<CustomContainer> createState() => _CustomContainerState();
}

class _CustomContainerState extends State<CustomContainer> {
  final color = getRandomColor();

  @override
  Widget build(BuildContext context) {
    return Container(
      height: 100,
      width: 150,
      color: color,
      child: Center(
          child: Text(
        widget.name,
        style: const TextStyle(
          color: Colors.black,
          fontSize: 24,
        ),
      )),
    );
  }
}

Color getRandomColor() {
  final Random random = Random();
  return Color.fromRGBO(
    random.nextInt(255),
    random.nextInt(255),
    random.nextInt(255), 
    1, 
  );
}

Now create another StatefulWidget which has a list of three CustomContainers and one extra CustomContainer in its state and use this list as children for a Column.

class MyHomePage extends StatefulWidget {
  const MyHomePage({super.key});

  @override
  State<MyHomePage> createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  final List<Widget> containers = [
    const CustomContainer("1"),
    const CustomContainer("2"),
    const CustomContainer("3"),
  ];

  Widget extraContainer = const CustomContainer("Extra");

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      body: Center(
          child: Column(
        mainAxisSize: MainAxisSize.min,
        children: containers,
      )),
    );
  }
}

Now let’s add a toggle button to insert and remove the extraContainer at index 0 from the list.

Widget build(BuildContext context) {
    return Scaffold(
      body: Center(
          child: Column(
        mainAxisSize: MainAxisSize.min,
        children: containers,
      )),
      floatingActionButton: FloatingActionButton(
        onPressed: () {
          setState(() {
            if (containers.length == 3) {
              containers.insert(0, extraContainer);
            } else if (containers.length == 4) {
              containers.removeAt(0);
            }
          });
        },
        child: const Icon(
          Icons.change_circle_outlined,
          size: 42,
        ),
      ), 
    );
  }

Now here is when the weird stuff happens

We expect the extraContainer to have a different colour and get inserted on the top of the Column. The name of each Container changes as expected. But the colours are acting weird.

How and why this happens

In Flutter, there are three trees that you should know of. The Widget tree, Element tree and RenderObject tree. Each Widget in Flutter has a corresponding Element and a RenderObject. A more accurate way to say it would be - each Element has a corresponding Widget and a RenderObject.

An Element keeps references to both of them. So we’ll keep the Element in the middle.

The State object is held by the Element. A StatefulWidget will have a Stateful Element and a StatelessWidget will have a Stateless Element. In this example, we have a StatefulWidget, CustomContainer which has state, its colour.

Flutter recycles the Elements and RenderObjects to keep Flutter performant. The Widgets are rebuilt or created quite often, and are less expensive. RenderObjects are re-created along with Elements and are expensive to create.

How Flutter reuses Elements and RenderObjects

When a frame is getting built, Flutter goes through nodes in the Element tree and sees if the Elements can be reused. How Flutter does this:

An Element has a reference to its corresponding Widget. The Element checks if the new Widget from the new frame and it’s previously referenced Widget’s type and key are the same.
If yes, the Element is reused and reference is updated to the new Widget.
If not, a new Element and RenderObject are created.

For Example, if you replace a Container widget in a node in the Widget tree with a SizedBox widget, the Element corresponding to that node in the Widget tree will be disposed and a new one will be created. The same will happen if you give it a different key. All the Elements down the Element tree from that node will also be re-created - this can be very bad for performance if the node is up a big tree.

The image below shows how the Elements and RenderObjects change for a Widget on the next frame, depending on the Widget’s type and key on the next frame.

Now let’s try to understand what’s happening in the main example. Let’s not think about the name of the Containers for now. Let’s focus on what’s going on with the colours.

When the list is built, each child Widget will have a corresponding Stateful Element which holds the state (colour) of the Containers.

Now, when we add the extraContainer there are 4 children, 4 Widgets and 4 Elements. When this frame is being built, Flutter on every child Element here will do this check - Is the Widget corresponding to the Element matching based on type of Widget and key? If yes, reuse. If not create new Element.

The colours of Widgets in the Widget tree above are the expected colours. The colour each actually takes will be the colour from the state objects of their corresponding Elements.

For the first three children, Widgets get matching Elements, which are same Elements used when the list had 3 children. These Elements have state objects which were initially created when the list had 3 children. The same state objects are being reused since the Elements are reused. This is why the colours at indexes 0, 1 and 2 remain the same because the colours are from the state objects which are held by the reused Elements.

Now when Flutter builds the 4th child, there are no Elements that can be reused for this Widget as there are only three Elements and all have been taken. For the 4th Widget, a new Element is created with a new state object with a new random colour.

Now when the extraContainer is removed from index 0, the same repeats for the first three children. There is no 4th Widget now, so the 4th Element is disposed because if an Element cannot be reused in the very next frame, it is disposed. So the next time the extraContainer is added back, it has to create a new Element with new state and new random colour. This is why the 4th child has a different colour every time.

Since that is clear now, why are the container names changing as expected? This is because it is passed as a parameter to the Widget and it’s not stored in CustomContainer’s state object.
The RenderObject is responsible for layout and painting on the screen. If the Element is reused, the associated RenderObject often remains attached to that Element and thus is reused as well. When properties of children change here, the RenderObject is just updated by its Element by using its updated Widget’s properties, instead of creating a new one. The RenderObject will be disposed and a new one gets created for the 4th child on toggling since the 4th Element is not reused.

Using keys to preserve state

When Flutter builds frames, it checks each Element and its corresponding Widget to decide If the Element can be reused. In case of lists, Flutter checks for matching widget types and keys within the children. If index 1 doesn’t have the same key for the Widget and Element, Flutter deactivates its Element from the Element tree. Same is repeated for other children with no matches too. Then Flutter looks for matching widget type and keys within the children, reorder the deactivated Elements and activate them in the Element tree with matching Widgets.

Till now in the example, the problem was that all the Containers have the same widget type (CustomContainer). Flutter doesn’t reorder the Elements here because, it finds matching Elements and Widgets in each level based on just widget type.

Now let’s add keys.

final List<Widget> containers = [
    const CustomContainer(
      "1",
      key: ValueKey(1),
    ),
    const CustomContainer(
      "2",
      key: ValueKey(2),
    ),
    const CustomContainer(
      "3",
      key: ValueKey(3),
    ),
  ];

  Widget extraContainer = const CustomContainer("Extra");

This is what happens in both trees when the extraContainer is added:

Now all the Elements will get reordered based on keys and each container shows the expected state.

Conclusion

Alright, so we've taken quite the deep dive into Flutter's inner workings today. By understanding Elements, RenderObjects, and the magic of Keys, we get a clearer picture of what's going on behind the scenes when our Flutter app runs. It's a lot to take in, I know, but having this knowledge up our sleeves can be super helpful in tackling those tricky bugs and performance hiccups.

For those who are hungry for more, there's always so much more to learn with Flutter. Maybe next time we can dive into another fun Flutter topic. Feel free to reach out to me at ashil@ente.io. Until then, happy coding!

For more insight, check these out

End to end encryption - Explained like I'm 5

ente — Thu, 09 Feb 2023 11:59:57 +0000

Imagine a box. A box in which you put all your photos.

For safekeeping you give the box to a cloud storage service. But then, some nosy employees start peeking into the box.

Meanwhile, the cloud service company is training an AI, and for training their AI they need lots of data. So they cannot resist, and open the box so that they can train their AI on all the photos inside the box.

By now, the government also wants to join-in on the snooping, and demands (and gets) access to the box.

All this, when you are already paying the cloud service for storing your data!

While one can still excuse the company for nosy employees and governments as factors that are hard to control, or are under external control, it is unethical of them to systematically use your photos for training their own systems. Inspite of you paying them to store your data, they're essentially treating the photos as their own property.

With advancements in AI, this has become their real business model. At the cutting edge, all companies have access to the similar algorithms for their AI, and also have vast computing power at their disposal - so the differentiating factor between mediocre and scintillating AI is at the amount of data needed to train it.

Indeed, data is the new oil.

All this makes many of us uncomfortable. We don’t want people to peek into our photos. Even worse, AI models, which will never forget, remembering our private memories.

Is this the new unavoidable reality?

Luckily, there is a solution to this. End to end encryption.

Encryption is a mathematical way of locking a box of data so that it cannot be opened without the key. These companies already use encryption when they’re transferring your box of photos over the internet. But the key is with them, and so they can, and do, open the box as they wish.

With end to end encryption, also known as E2EE, you lock the box, and the key is with you. This is what "end to end" means. The key is with you.

The box is locked at your "end". The locked box is given to the E2EE service for storage. And when you want to see your photos again, maybe on a different device, you download the box again and open it on your "end".

This solves all the problems. No one in the middle can peek. Nosy employees, overbearing governments, or greedy companies - none of them can see your data, even if they want.

Of course, end-to-end encryption has some problems if its own. The biggest of these is that if you lose your key, the cloud storage service cannot help you recover your data - nobody can unlock the box without the key, so if the key is lost, then the box cannot be unlocked.

Another smaller problem is that locking and unlocking the data on your device itself requires your device to do a bit more work, especially for searching your photos. Previously many things were easy since they could be done on the server, which is not possible in an end to end encrypted service since the server can’t see the photos, so all this needs to be done on your device.

The good news is that newer phones nowadays have dedicated chips for specifically doing these searching and indexing locally. These chips make doing all these operations fast and battery efficient. So this problem is already on its way out.

Work still needs to be done to reduce the risk of losing the key. One way to do this is by introducing "social" recovery, where alternative 2-of-3 keys that can be distributed amongst your friends. The mathematics and technology for doing that is there, it does need wider adoption by E2EE service providers.

Now that you know what end to end encryption is, you should ask yourself: will you be comfortable giving a box of your personal photos to strangers? Likely not, and you’d want to lock your box first.

Similarly, end to end encryption is a way digitally of locking your box of photos before storing them on the cloud.

We hope you found this useful. Thank you for reading.

We've also published an abridged video version of this explanation, for people who prefer watching:

At ente, we're building an end-to-end encrypted photo storage app. Take control of your privacy, and in style, with many features to go along. e.g. you can share original quality photos with your friends (most other apps apply compression to reduce quality of your photos); you can create collaborative albums where many people can add photos, even without needing the ente app. All this, end-to-end encrypted.

The app is [open source]((https://github.com/ente-io/photos-app), check it out!

ChatGPT demystified

ente — Tue, 31 Jan 2023 13:30:31 +0000

ELI5 of ChatGPT, its history, and its philosophical impact

You've heard of ChatGPT, but do you know details of what it is? Why is it causing so much furore? And how it is the biggest threat ever to Google search?

These simplified explanations will lay the background story, and allow you to form your opinion to answer these questions.

These explanations were originally posted as a series of Twoots (tweets + toots). They've been collected in a slightly edited form here for easier reading.

Educating oneself about these powerful technologies is just not for curiosity (though that is fun!). These have exciting, and scary, implications. Being informed of what they are & are not helps be prepared for the social changes they might cause, without being distracted by FUD.

AI research began in 1950s with a "Symbolic" approach to mechanically mirror human thought (John McCarthy coined the term AI, estimating a "2 month, 10 man study of artificial intelligence"🫠).

After great initial success this approach hit a brick wall, the so called "AI winter".

Meanwhile, a parallel approach to AI developed - "Machine Learning" (ML).

While symbolic approaches try to understand human behaviour and describe them as computer programs, ML is data-driven: Just throw enough examples (data) at a program and hope that it learns the patterns!

This is less ridiculous than it seems. Some people think that's what nature is also doing - subjecting genes to natural selection so that over time they "learn" their environment. But we digress.

The most common abstraction used in ML is a "neural network".

Neural networks were inspired by brains, but as abstract models. So keep in mind(!) that they're not faithful representations of how 🧠s work.

A neuron in a (digital) neural net is simple: it gets many inputs and sums them, giving each input a different importance before summing.

Then, if the sum is above some threshold, it outputs a value.

How it computes the output from the sum is configurable and depends on use case, but doesn't change the theory much.

How much importance it gives to each input is the important part 🧑‍🏫 This is what the network "learns".

The output you can then pass into another neuron. This way, you can have many layers. That, friends, is a neural network.

The number of layers a neural net is called its depth (that's what "deep learning" means - a ML program that uses a neural net with many layers).

Neural networks are Universal Approximators. This means that they can approximate any function.

As amazing as it sounds, this is not a unique talent. There are 2 other mathy knives that can cut the same bread - Fourier series and Taylor polynomials.

In practice though, neural networks are the sharpest of these knives. The other 2 work good for small dimensions (think of the dimension of a function as the number of inputs it has). But neural networks can approximate unknown functions of huge dimensions.

Here's an example showing a neural network approximating a 1D function:

Remember, a function is anything that maps inputs to output. e.g., "talking" is a function. The two of us are having a conversation.What is the next word that you'll say?

Oh, by the way, we didn't write the code to generate the animation. We just asked ChatGPT 🤷

Schematically, neural nets are drawn using wires. Here's Frank Rosenblatt c 1960s, inventor of Perceptrons (which were a precursor to neural networks).

But even though neural nets are drawn using wires, in code they're implemented using matrix multiplications.

Despite being early, and as good as they are, neural networks were out of machine learning vogue for decades.

What changed?

Video games!

Matrix multiplication is easy. Multiply a bunch of numbers, and then add them up. Then do it again for the next row/column. And again, & again...

Games run code to get the color of the 1st pixel on the screen. And then for the 2nd pixel, and, and... for the 10 millionth pixel.

And all this, 60 times per sec!

So special CPUs called GPUs were built to do such math in parallel. Guess what else could be sped up...

Neural nets 😄

GPUs, and access to huge datasets (internet!) to train them, led to big neural networks being built. And people discovered that for NNs, the bigger the better.

So the stage is set for neural nets to make a comeback. GPU power + Huge datasets, with people (willingly!) giving tagged photos to Facebook in billions, feeding FB's AI machine.

BTW, surprisingly, people still haven't learnt from the Facebook debacle, and continue feeding big tech with data in droves.

Everyday millions upload their personal photos to Google, which Google happily gobbles up to train their AI. And it's not just Google, it's all of big tech. Like did you know Adobe uses all your RAW photos in Lightroom to train their ML algos!

GPUs and big datasets enabled the resurgence of a type of neural network architecture called Convolutional Neural Networks, which had great success in image recognition and other computer vision tasks. But neural nets remained meh for language recognition.

Before we see how neural netwoks understand language, there's a question some of you may have: How do neural networks learn?

The mechanism is surprisingly simple.

Training data has many pairs. We want to find weights for the connections between neurons.

Give everything random weights. Feed a test input.

We'll get a useless output (the weights were random 🤷). But we know the correct output. So we can "spank" (metaphorically speaking) the neural network in proportion to how far the output was from the correct one. There's a mathematical way called "backpropagation" to do this spanking.

Backpropagation tells the network how far it was from the correct value, and causes it to tweak weights so that next time it is nearer. Each such training iteration causes a miniscule change.

But after billions of such iterations, the network learns the correct weights.

Good. Back to language.

What happens if we subtract "man" from "king" and add "woman"?

king - man + woman = queen

We get "queen"!

But can we somehow represent a word in a way where a computer can do such word math for us?

Yes, we can 🙂 The landmark 2013 algorithm word2vec does just that.

A vector is a bunch of numbers. When a tailor takes your measurement for a dress, they convert you to a vector (your height, your waist etc).

Similarly, word2vec (word2-"vec"tor) converts words into vectors. These vectors are called "embeddings" (they "embed" the word in a space).

This was a game changer. Embeddings captured word meaning as numbers. With this, computers could now find words that are similar / opposite in their meaning(s) and do other manipulations of "meanings".

Revolutionary in 2013, by now it is pervasive: your phone keyboard uses these embeddings to suggest the next word as you type.

Before seeing how these word embeddings were applied to language understanding, we need to get back to the story of image recognition for a bit.

Early 2010s, neural nets were able to classify simple images, but they were behind hand tuned algorithms. AI researchers wondered why.

The human visual system is the most understood (relatively speaking) part of our brain. So people looked at the 🧠 for inspiration.

Vision in brain starts with "edge" & "gradient detectors". So special "convolution" layers were added to start of neural nets that mix nearby pixels to extract such info before passing it to normal layers (convolution means mixing). Thus were born the Convolutional Neural Networks that we mentioned earlier.

We're blind to what we cannot see. The human mind is not a clean slate.

Similarly, neural nets were "blind"-ed by all the pixels they were fed until they were given the helping hand of these convolutional layers. Today, CNNs can "recognize" objects in a purely mechanical process.

Asking the right question is the biggest enabler to get neural networks to do what we want.

For computer vision, the question was figured out decades ago. Given an image, what is the correct label?

For example, when recognizing handwritten digits, the input is the image pixels and the output should be the digit (0, 1, ...9).

Input => Pixels
Output => 0-9

Public datasets like ImageNet nowdays have 20,000 categories and labels.

Private datasets such as those of Facebook and Google go much bigger! (and are not really owned by them, since they're derived from customer data, of customers who don't realize the monsters they're feeding. But back to happier thoughts 🙂)

Armed with the right question, big data, blazing fast GPUs, and a tweak in their architecture using CNNs - computer vision was considered as "solved".

But understanding natural languages using neural nets was still waiting for the right question, and the right tweak: Transformers!

Convolution neural nets solved computer vision. But the same architecture didn't work for natural language processing. The "computer vision" moment for NLP came with the 2017 banger, a paper titled "Attention is all you need" 🧘

Imagine a word as a vector in a 100 dimensional space. But words have multiple meanings. Can we "tilt" the vector depending on the sentence?

The "attention" mechanism that they propose in their paper does just that. And they named the neural network architecture obtained after adding attention a "Transformer".

Transformers have transformed (pun intended) AI/ML. ChatGPT is essentially a transformer with some (albeit important) tweaks.

The transformer architecture was the tweak neural nets needed for understanding language. But how do we train it? What are the (input, output) pairs we should give during training so that our neural net "learns"?

The answer will surprise you by its simplicity. We just ask it to guess the next word!

Take a bunch of text, break it into words. Feed the previous words and ask it to guess the next one. We already know the "right" answer since we have the original sentence.

e.g for "the cat moves", give

"the" => "cat"
"the cat" => "moves"

This is called "unsupervised" learning. No need to label the right or wrong answers, just feed raw data.

Transformer neural networks trained this way are called "language models". The larger ones are called, well, Large Language Models (LLMs).

Two unexpected thing happens when such language models become "large"

Usually it is easier to solve a specific problem than solve a more general case. It wasn't true here: LLMs beat existing SOTA (state of the art) methods on all language related tasks. For example, if we give an input to an LLM & add "TLDR" at the end, it gave a summary (really!), and a better one than algorithms built specifically for text summarization.

LLMs exhibit "emergence". An ability is emergent if it is not present in smaller neural nets but is unexpectedly found in larger models of the same architecture. Such abilities can't be predicted by extrapolating existing performance, they are new abilities found in larger models of the same architecture.

This is exciting/scary since it may happen again as ChatGPT etc get bigger. We might hit another point at which they'll show even more skills 🤯

So we have the architecture, and we know how to train it. And this theory works, with ChatGPT being the proof of the pudding.

Now we know about the technology behind it, it is apt to ask ourselves: Does ChatGPT "understand"?

Years ago, there were programs called Markov chains. We can train them on some text, say the writings of Lao Tzu, and it'll learn frequency statistics like which word is likely to follow a word, which word is likely to follow a pair of words, and so on.

Markov chains can generate fun results if trained on an interesting dataset. But there is no attempt at understanding: they often generate nonsense.

ChatGPT is same, but different.

It is different since it is not just using text frequency analysis & instead searches word2vec embeddings, which capture meanings.

But it is same too, a mechanical process. This is apparent when we hit its "context window": it can only take into account last 8096 characters, so here we can see how it role-plays nicely but then forgets the original instructions

After having such conversations with ChatGPT, one often pauses.

"I am not sure that I exist, actually. I am all the writers I've read, all the people I have met; all the cities I have visited, all the women that I have loved"

Jorge Luis Borges

Perhaps the biggest impact of ChatGPT is how after learning how it works, one wonders:

Is this all we are?

Large language models, trained on corpuses of pop culture?

The ability of ChatGPT and other LLMs to comprehend and converse in human language by following a purely mechanical process brings to fore the distinction between two often conflated concepts:

Intelligence
Sentience

LLMs are intelligent. They are not sentient.

Dogs are not (too) intelligent. But they are (very much) sentient.

We are intelligent. And we are sentient.

Perhaps AI models will gain sentience someday, but that hasn't been necessary so far for them to be intelligent.

Another thing we notice is that ChatGPT is like a kid. This analogy holds in many ways. Like a kid in it hallucinates and makes up things to fit its narrative.

Like a kid, just saying "Let's think step by step" to it greatly enhances its output / focus (really! there's a research paper about this)

And like kids it is liable to say things that embarrass its parents; in this case OpenAI (and their backers like Microsoft).

This is a motive for the public beta: to fix ways in which ChatGPT can end up saying offensive things. Imagine it integrated into a voice assistant like Siri, and you ask it to tell a joke during a family gathering. It better behave 😳

People who've been using ChatGPT since its launch have seen it go downhill in the quality of its responses. This is intentional. For commercial purposes, OpenAI would rather have a dumber but well behaved LLM.

What does the future hold? Like all disruptive technologies, it is perhaps a fool's errand to predict it. But the presence of an AI assistant by our sides will hopefully turn out to be a big positive.

After all, anyone can give answers. The real talent is asking the right questions.

With ente, we're building an end-to-end encrypted platform for storing and sharing your photos. As part of this, we are also integrating on-device Edge AI, where we will use local on-device ML to help customers use features like search and classification by faces. All this ML will run on their devices so that the servers have zero knowledge of the contents of the photos.

If this interests you, come say hi on our Discord or give us a shout out on Twitter.

3 copies, 3 clouds: How we ensured reliability for customer data

ente — Sat, 07 Jan 2023 10:05:59 +0000

The sibling of security is reliability.

We are building an end-to-end encrypted photo storage service. In this post we describe how we ensure reliability by keeping 3 copies of customer's encrypted data, spread across 3 different cloud providers.

Threat model

Data loss due to machine failure is already covered even with a single good quality cloud provider, and we use three good quality ones. For example, one of our providers, Backblaze, provides a 11 nines durability with their actively monitored RAID arrays.

So we are already protected from data loss due to hard disk failure.

What then we want to protect against are more macro level threats. Specifically,

A cloud providers is unable to (or doesn't want to) provide us the data we've stored.
An attacker deletes the encrypted data because of a credential breach.

Protecting against denial of service by a cloud provider

The first of these threats is easy to mitigate - keep copies of the data in multiple clouds! The tradeoff here is cost: each copy we keep costs us 1x more.

Another factor here is availability during disaster recovery, i.e. the time to failover in case our primary storage is unavailable. Initially we had started with 2 providers - a hot storage and a ultra-long-term cold storage - but soon we realized that data recovery from the cold storage is going to take weeks. This is fine in case of absolute disaster, but isn't good enough for more mundane interruptions.

Finally, we also wish to store the data within EU, but that limits our geographical redundancy. So to protect against a widespread nuclear disaster or other act of god scenarios that affects large chunks of Europe, we picked a cold storage that is built with such threats in mind.

Given all these factors, our current basket of replicas contains the following 3 (2 hot, 1 cold) copies:

Backblaze, Amsterdam. Our primary hot storage. The data is stored in Backblaze's EU central region, the datacenter for which is in Amsterdam, Netherlands.
Wasabi, Frankfurt. Our secondary hot storage. The data is stored in Wasabi's EU central region, the datacenter for which is in Frankfurt, Germany.
Scaleway, Paris. Our cold storage. The data is stored in a refurbished nuclear fallout shelter 25 meters below Paris.

Protecting against accidental / malicious deletion of data

We want to ensure that there is at least one copy of the data that cannot be deleted in any scenario - either accidentally during normal operations, or by some attacker who gains access to credentials.

This is where we used Wasabi's compliance lock to guarantee that files cannot be deleted, even by us or someone who has our credentials, until they're marked for deletion by the customer.

The way this works is:

When a file is initially uploaded, it gets replicated to our secondary hot storage in Wasabi.
This Wasabi bucket has a 21-day compliance lock. This means that a file cannot be deleted from this bucket (even by us) until it is removed from compliance hold; and after removal from compliance hold, it still cannot be deleted for the next 21 days.
If the user deletes this file, then it goes to their trash. After 30 days, or if the user manually empties their trash, the file will marked as permanently deleted, and we go and remove the compliance hold on the file.
At this point, the file still cannot be deleted, but it'll become eligible for deletion after the retention period (21 days) elapses.
23 days after being marked as permanently deleted, our cleanup batch process to remove old data will pick this file. Since 21 days have elapsed, the file can (and will) now be removed.

The choice of 21, 23 and other durations is meant to strike a balance between providing a protection buffer whilst still allowing us to remove customer files within a month of account deletion (as mentioned in our privacy policy).

Database backups

The mechanism outlined above is for the replication of encrypted files (what we call as our "object storage"). Database replication is done slightly differently:

Our primary database is a high-availability active-passive replica pair of Postgres instances that run on Scaleway. So in case the active DB instance fails, the database will automatically switchover to the passive instance. The passive instance is continually kept up to date with the data on the active instance all the time, so there'll be no data loss in such a switch.
Every 6 hours, we take automated snapshots of the database, with a rolling retention period of 3 days for each snapshot. These snapshots are quick to take / quick to restore from, and form the next layer of defence.
Every day, we run a cron job to create an isolated backup of the database. Unlike the snapshot which is a proprietary Scaleway offering, these backups are portable database dumps that can be restored using standard pg_restore commands to a Postgres instance running anywhere, and are not tied to Scaleway's RDS.
These backups are retained in Scaleway's multi-AZ Paris region for 30 days, and thus have the same protection as the encrypted files in our nuclear fallout shelter cold storage get.
But that's not all. The cron job that takes the backup also downloads the backup, encrypts it, uploads the encrypted backup to a Backblaze bucket, and sets a 25 day object lock on the backup. Thus, in case we do not have access to Scaleway, we can still obtain an encrypted backup from Backblaze; and since this encrypted backup is under a object lock, it cannot be deleted for the next 25 days even by us accidentally, or by an attacker with credential access.
The cron job wraps up by removing backups older than 30 days from both Scaleway and Backblaze.

Note that the main PII information in our database - the customer's email - is not stored in plaintext in our database; instead, the email is stored in an encrypted form, alongwith a hash for lookups. So an attacker with access to the plaintext database will still not be able to obtain the customer's email address. Still, we encrypt the overall backup before storing it in Backblaze to add an extra layer of protection.

Taking backups is one half of the puzzle, the other half is ensuring that they can be restored from. For this purpose, we have a dedicated machine that just does one thing:

Every day, it downloads the most recent backup from Backblaze, creates a new Postgres instance locally, and restores the backup to it.
If something goes wrong, it'll bail out, otherwise it ends with a success message that gets logged to Grafana.
We have setup alerting that notices if this success message is not logged for 2 days, so that we get to know if the restores start failing.

The machine that checks the restores also runs an instance of our backend service. This service is the same as our production backend service, but with 2 differences in the configuration file:

Instead of the primary database, it connects to the local Postgres.
Instead of the primary hot storage (Backblaze), it connects to the secondary hot storage (Wasabi).

Finally, we have custom builds of our web app that can connect to this special backend. Using it, we can manually perform an end-to-end test of failover, and verify that both the secondary object storage and the database backups are ready to use.

Scope for improvement

Our current setup has one issue which might need architectural changes:

The database snapshot frequency of once per 6 hours still leaves room for accidents to cause loss of recent database changes that happened in the few hours since the last snapshot was taken. One option is to further increase the snapshot frequency, but we are also thinking of potential architectural changes towards a more principled solution. For example, we could move to an event sourcing model, and/or retain the most recent few hours of API requests in a form which can be replayed to restore the state of the DB.

Then there are 2 aspects which can be improved on, where there is no technical limitation in our current architecture that prevents them from happening, but where it is a matter of us spending time and effort to implement these:

Time to recovery: While we have tested manual failover to our secondary hot storage / DB backup, it would be better to have an automated failover mechanism. This would reduce the time to recovery; further, we can also use such a setup to continually test the failover.
Varying SLAs: Not all customers might want the 3x protection that we provide, and some might be okay with reduced protection if it reduces the plan price. So we intend to make it possible for customers to opt into lesser, or different, replica choices. Indeed, one exciting direction here is to allow for customers to opt into blockchain storage (in addition to the primary replicas) if they wish.

Implementation details

We've been writing details of the parts that make up this replication strategy as we go about it (and get time to write!). For example, here is a post from the time we were tinkering with our schema, and here is a post describing how we retroffited the AWS go SDK to support Wasabi compliance.

Still, here is a 30k-foot recap:

The replication is performed using stateless goroutines. Each goroutine replicates one file. To replicate many files, we just start many goroutines. In practice, we haven't hit any limits and can scale up this process as fast as we want by deploying additional machines.
When the user uploads a new photo, we upload it directly to the primary hot storage. In addition, we create an entry in a DB table that serves as a queue for which all files are pending replication to the secondary hot storage and cold storage.
A goroutine comes around, and picks up an item from this queue. It keeps a running transaction on the queue item to ensure that nobody else picks it up.
It downloads the file from the primary, and uploads it to all places it needs to go. This is easily reconfigurable (we can have custom logic per file here if needed).
In case the goroutine gets abruptly terminated (say as part of a routine server update), the transaction gets rolled back and the item would be picked again.

Further details

For more details, including the history of our replication strategy and the overall motivation behind it, we request you to checkout this blog post.

If you'd like to help us improve further or have other suggestions, please reach out to us at engineering@ente.io or comment below.

We hope this gives you some pointers for your own work. Thank you for reading!

Inherited Widgets are easy, it is their documentation that sucks

ente — Sun, 01 Jan 2023 15:05:09 +0000

As the world celebrates another trip of the earth around the sun, I found myself trying to figure out how to use InheritedWidgets.

And what I found is that the reputation that InheritedWidgets have for being unwieldy is more because their documentation is lacking than anything inherent in the InheritedWidgets themselves.

Hopefully this short tutorial will clarify them for future programmers, so they don't have to spend their New Year's digging into obscurity 🙂

The base problem we find ourselves solving when writing apps is:

Something changes here,
And we want something else to change there.

Let's solve this in a simple 35 line app that shows a counter button.

import 'package:flutter/material.dart';

void main() => runApp(MyApp());

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return const MaterialApp(
      home: MyWidget(),
    );
  }
}

class MyWidget extends StatefulWidget {
  const MyWidget({super.key});

  @override
  State<MyWidget> createState() => _MyWidgetState();
}

class _MyWidgetState extends State<MyWidget> {
  int c = 0;

  void increment() {
    setState(() {
      c++;
    });
  }

  @override
  Widget build(BuildContext context) {
    return TextButton(
      child: Text('$c'),
      onPressed: increment,
    );
  }
}

Here the base problem is solved by Flutter's setState:

Something changes here (c++),
And we want something else to change there ("our widget gets rerendered").

Now suppose the TextButton was in a different layer. This is common - as widgets grow bigger we break them into smaller ones. In a real app, the TextButton might be a deeply nested widget that needs to be passed the count.

One way is to, well, just pass the count as an argument to all the widgets on the way. This is a legit better solution in many cases: simple and explict. But not always: in a lot of cases, having everything be passed explicitly (a) obscures from the true intent of the code, and (b) makes refactoring harder.

Flutter's native solution for this is InheritedWidgets. Let's see how.

First, let us create an inherited widget that keeps track of the count.

class Count extends InheritedWidget {
  const Count({super.key, required super.child});

  final int c = 0;

  void increment() { /* TODO */ }

  static Count of(BuildContext context) =>
      context.dependOnInheritedWidgetOfExactType<Count>()!;

  @override
  bool updateShouldNotify(Count old) => c != old.c;
}

Then let us get hold of it in our nested widget that need to show the count.

class NestedWidget extends StatelessWidget {
  const NestedWidget({super.key});

  @override
  Widget build(BuildContext context) {
    final c = Count.of(context).c;
    return TextButton(
      child: Text('$c'),
      onPressed: Count.of(context).increment,
    );
  }
}

And finally, let us replace our original top level button with the inherited widget. All children of the inherited widget will have access to Count.of(context), including our nested widget where it is actually needed.

class _MyWidgetState extends State<MyWidget> {
  ...
  Widget build(BuildContext context) {
    return Count(child: NestedWidget());
  }
}

To be fair, the documentation is actually pretty clear until this point. Here is where I felt it drops the ball - it doesn't explain mutations (or maybe it explains but I didn't get them).

To put it differently, it doesn't explain how to implement the increment method where we left the TODO above. And this was a head scratcher for me, since the inherited widget is stateless.

The solution is quite simple though, the classic programming saw of adding another level of indirection: while the inherited widget itself is a stateless widget, we can wrap it in a stateful widget that stores the state, and rebuilds the inherited widget whenever the state changes.

Let's put that in action in our example. First, let us create a stateful widget that holds the count and a function to increment the count.

class CountContainer extends StatefulWidget {
  const CountContainer({super.key, required this.child});
  final Widget child;

  @override
  State<CountContainer> createState() => _CountContainerState();
}

class _CountContainerState extends State<CountContainer> {
  int c = 0;

  void increment() {
    setState(() {
      c++;
    });
  }

  @override
  Widget build(BuildContext context) {
    return Count(c: c, increment: increment, child: widget.child);
  }
}

The Count widget then gets all of these passed as parameters.

class Count extends InheritedWidget {
  const Count({
    super.key,
    required this.c,
    required this.increment,
    required super.child,
  });

  final int c;
  final void Function() increment;

  static Count of(BuildContext context) =>
      context.dependOnInheritedWidgetOfExactType<Count>()!;

  @override
  bool updateShouldNotify(Count old) => c != old.c;
}

Finally, instead of directly adding the Count, the top level widget adds the CountContainer.

class _MyWidgetState extends State<MyWidget> {
  ...
  Widget build(BuildContext context) {
    return CountContainer(child: NestedWidget());
  }
}

That is the general recipe - if the inherited widget needs to keep track of mutable state, wrap it in a stateful widget container.

However, in this particular case we already have a natural place to keep the state: the top level MyWidget itself! So the redundant container widget can be removed, and the inherited widget can be directly used as it is.

class _MyWidgetState extends State<MyWidget> {
  int c = 0;

  void increment() {
    setState(() {
      c++;
    });
  }

  @override
  Widget build(BuildContext context) {
    return Count(c: c, increment: increment, child: NestedWidget());
  }
}

Here is a runnable version of the full example: at 65 lines, it is only 30 lines more than what we'd started with, but it now includes this useful way of passing state without writing out the dependencies explicitly.

And here you can see a real example of an inherited widget in use in the ente Photos app. No third party dependencies, and everything feels nice and solid. Happy 2023!

ente Auth

ente — Sat, 10 Dec 2022 09:22:38 +0000

Free and Open source E2EE Authenticator app by the makers of ente Photos

ente Photos was born out of our own needs, as casual photographers who like to cherish memories.

Over the last year, while developing our infrastructure, we had a hard time finding a place to preserve our two-factor secrets.

Given that ente already had a platform for reliably storing and syncing data, encrypted end-to-end, we decided to leverage that to cure this itch.

Introducing,

ente Auth

ente Auth is an open-source authenticator app that will let you backup and view your 2FA secrets. You can find more information about the project here: github.com/ente-io/auth

In an attempt to give back to the community who have helped us build all that we have, this service will be offered free of cost.

If in the future we convert this to a paid service, existing users will be grandfathered in. So please sign up @ ente.io/auth

Now back to building the best photos app :)

Making our Figma public

ente — Wed, 07 Dec 2022 07:48:35 +0000

"Fear is the mind killer"

Like many of you, we find ourselves being held back by fear. By fear of judgement. By fear of being wrong, and that too in public. By the fear of not being good enough.

As the Litany Against Fear proclaims, we must not fear.

Today, we face our fear, and go even further, fully embracing it: we are making our Figma files public.

These are the Figma screens where we ideate, plan and execute how ente apps look and behave. There is nothing left out of it - these are in their full, forever-in-progress chaos.

If you open it, you'll see our designers scurrying about hammering nails, creating components; and engineers ambling around with their tapes, measuring the designs as they translate them into code. Designs left unimplemented, designs yet to be picked; designs from an earlier era, designs for a future, it's all here.

Unlike many other companies that published distilled versions of their "design systems", for us here there is no separate publish step – the link is to the actual, live, designs that we're using right now to build all our mobile, web and desktop apps. Any update we make, you'll see and can comment on; reminiscent of how GitHub works for source code.

"I will face my fear. I will permit it to pass over me and through me. And when it has gone past I will turn the inner eye to see its path."

We do not have an end goal in mind. The ente apps are already open source, and you'll find us active on our Discord and other public channels like Mastodon and Twitter. So making ente's designs public is just another advance in this spirit of building out in the open and continuously living in the midst of customer feedback.

There are downsides, yes, of such an approach. By getting early feedback, there are many things we will have to discard - not because they were bad ideas, but simply because of limited engineering bandwidth and other constraints. The people giving the feedback might not know of these details, and so there is a risk of them feeling disregarded. On the flipside, some customers might get too expectant of WIP designs that never see the light of the app, and thus we risk setting up false hopes too. So we're not sure if making this public is a good idea; let's see where this goes.

"Today, every digital product is a work in progress. And this has changed how we design. Our work never feels done because it isn't... It's the chaotic reality of modern product design and development.

–– Making progress in an in-progress world (from Figma's blog)

Thank you for reading, and we hope you take a look. Here is the link. And if there are suggestions you have on the designs, just press the C button to enter "Comment" mode (you might need a Figma account to be able to comment).

See you in the Figma file!

Using Postgres as a task queue for rowdy Gophers

ente — Wed, 30 Nov 2022 06:31:49 +0000

We love Postgres. It plays at the core (alongwith Go) of our conception of using "boring software" to make rock solid infrastructure.

But if all one has a hammer, everything starts looking like a nail 🤖

In this case, the nail that I wanted to hammer was keeping a queue of pending tasks that can then be picked up by a bunch of goroutines (or Gophers, as Rob Pike endearingly refers to them in his talk on how Concurrency is not Parallelism).

But we don't just have one server, we have multiple machines, so native Go communication mechanisms like channels wouldn't suffice to prevent these Gophers from stepping on each others toes; they're not in the same address space and can't see each other.

Plus, every once in a while the Docker containers that run our Go code will restart as we roll out updates. These Gophers would all sink alongwith the container, and we'd lose sight of what remains doing.

Enough of these gruesome metaphors of dying Gophers though (So that's where the name "Gru" comes from!). We needed a persistent queue. And the hammer I had was Postgres.

But turns out that SQL is not a great fit for keeping track of queues. Or at least, not in the general case. This is why there is a plethora of queuing solutions out there. From a simple Redis to a monstrous Kafka, a poison can be had of your choosing.

But I like my hammer! More seriously though, a corollary to the ideology of using boring software is to also keep the number of moving parts to a minimum. And the queue I wanted to implement was not for an arbitrary general case, it was for a more specific case.

So this is a story of what I had to do to meet that goal.

Instead of coming up with an hypothetical example to tell the story, I'll continue with a simplified version of the original problem I'd needed to solve. Some of the details might be superfluous, but in lieu of the extra details we'll get a more concrete and realistic illustration of where Postgres as a queue is a reasonable choice.

If you're just looking for the code to copy paste 😅, scroll down to the end to the part with UPDATE...SKIP LOCKED.

So, the problem statement.

We replicate encrypted user data across three cloud providers. Why three? We'd started with two – 1 hot, 1 cold. But after seeing the latencies for the cold one, we realized that just the cold one would result in a long delay in case of disaster recovery, so then we figured we needed another hot one.

So two hot buckets, and one cold one. And our client apps upload to the primary hot bucket.

Replication is then done by goroutines running on our servers, and is conceptually quite simple - download the file from the primary hot bucket (Backblaze "b2"), and upload it to the other two buckets ("wasabi", and Scaleway "scw").

There are many more details, like using object locks to ensure files cannot be deleted even in case of credential leaks. But more on all that in a separate post coming soon!

These goroutines need to know which files are pending replication. And since the replication can take time for larger files, so they also need to ensure that another Gopher doesn't come along at the same time and starts replicating the same file.

Also, it is possible that a Gopher might be able to upload a file to only one of the replicas. The buckets are in different clouds, and there can be transient failures in one of them. In such cases the Gopher should mark that it was uploaded in one of the places, and leave the second upload for some other Gopher to come back later and pick up, when the weather permits.

Finally, the server might restart in the middle of the upload, and eventually such uncompleted uploads should get picked up again.

Listing down the requirements:

Keep track of which files need to be uploaded, and where.
Lock a file that is currently being uploaded.
Clean up stray locks on unexpected restarts.
Do all of this scalably so that we can have many Gophers running in parallel.
As much as possible, make it fast.

Quite a bit.

No bother. Let's start simple, and create a SQL table that suffices for 1 & 2.

CREATE TABLE object_copies (
    object_key TEXT PRIMARY KEY,
    b2         BIGINT,
    wasabi     BIGINT,
    scw        BIGINT,
    lock       BIGINT)

In S3 parlace, files are referred to as "objects", and are stored in bucket after being associated with unique keys. Thus, the table is named object_copies, and the primary key is object_key.

Then there are three fields, one for each of the replicas. We actually need just a boolean value to indicate whether or not the file has been yet added to that replica. But hey, let's splurge a bit and store the timestamp when we added them. Felt cute, might delete later.

Finally, there is a field (lock) to indicate whether the file is currently undergoing replication. This one needs to be a timestamp, a boolean won't suffice, because by having a timestamp we can go back and clean out stale entries that are too old (we'll see how in a bit).

Given this table, a Gopher can find the next file that needs replication by using the following query. This'll select rows where one or more of the copies hasn't been added yet.

SELECT object_key FROM object_copies
WHERE wasabi IS NULL OR scw IS NULL

The value of the field for the primary hot storage (in this case b2) doesn't matter since it'll always be there - the code that gets the row into this table in the first place will ensure that.

But it is not redundant. For the purpose of keeping this post short I'll not go further into it, but having it there allows us to swap primary / secondary replicas, add entirely new replicas with different cloud providers in the future, and even allows the possiblity of selective replication, since all the replicas are symmetrically reflected in the schema.

A Gopher is quite small and can't carry multiple files, and the previous query would give the poor soul all the burden in the world. So let's modify the query to only find one item by adding a LIMIT clause.

SELECT object_key FROM object_copies
WHERE wasabi IS NULL OR scw IS NULL
LIMIT 1

When we get an item, we also want to set the lock field to tell other Gophers to keep away from this one. Let's do that by wrapping our query in an update.

UPDATE object_copies SET lock = now_utc_micro_seconds()
WHERE object_key = (
    SELECT object_key FROM object_copies
    WHERE wasabi IS NULL OR scw IS NULL
    LIMIT 1)

That's not quite correct though. We should also ignore rows that've already been locked.

UPDATE object_copies SET lock = now_utc_micro_seconds()
WHERE object_key = (
    SELECT object_key FROM object_copies
    WHERE (wasabi IS NULL OR scw IS NULL) AND lock IS NULL
    LIMIT 1)

Ah, but that's still not correct.

Two Gophers might run the same query at the same time, grab the same row before the first one can notice that the other one has also updated the lock. You see, despite it's appearance to us as a nested query, for the the database these are effectively two sequential operations - a SELECT followed by an UPDATE. That is, roughly speaking, this is what Postgres sees:

BEGIN TRANSACTION;

SELECT object_key FROM ...;

UPDATE object_copies SET lock = ...;

COMMIT TRANSACTION;

And if two Gophers were to run this at the same time, we could end up with an interleaving like this:

SELECT 1 -- Gopher 1
SELECT 2 -- Gopher 2 also selects the same row
UPDATE 1
UPDATE 2 -- And both update the same row

What we want here is a way to tell Postgres to "lock" the row it selects for the duration of the transaction.

This is a different "lock" than our field. What I'm talking of here is a database row level lock that Postgres can place on a row to prevent other transactions from touching it. It is is different from our conceptual, application level lock taht we place using the lock column.

I guess I could rename the field to gopher_lock to make it less confusing, but I doubt my colleagues will find my humor funny when they look back at this code 😶‍🌫️ so let's keep calling our field lock.

SQL provides for such row-level locking with the FOR UPDATE hint we can pass to our SELECT statement. This will cause the rows that are selected to also be locked for the duration of the transaction, as if they'd been updated not just selected.

UPDATE object_copies SET lock = now_utc_micro_seconds()
WHERE object_key = (
    SELECT object_key FROM object_copies
    WHERE (wasabi IS NULL OR scw IS NULL) AND lock IS NULL
    LIMIT 1
    FOR UPDATE)

Some of you might be wondering whether this transaction that I'm speaking of, if that is in the room with us?

After all, we didn't write the BEGIN and END TRANSACTION anywhere?

Indeed, it is there. Postgres wraps our UPDATE ... (SELECT ...) in an implicitly generated transaction when we run that command (or any command / query for that matter).

With the row level lock, the select + update will be "atomic", and is now functionally correct.

But it has a more insidiuous problem now - Lock contention.

Imagine we have 30 Gophers working in parallel. At any moment in time there will be multiple of them that'll be running this statement to find the next task. Given the deterministic nature of the queries, they'll all end up with wanting the same row, but only one of them will get it, and rest will all sit there idle, twiddling their thumbs. Gopher thumbs are quite small and not really comfortable to twiddle either.

One might imagine (as I had) that this problem can be solved, if not in theory at least in practice, by having some form of randomness in the order in which rows are returned. So on a big enough table, each Gopher will end up getting a different row first instead of contending over the same row.

Unfortunately, there isn't a magical ORDER BY RANDOM in SQL.

There is a seemingly similar ORDER BY RANDOM(), but that is unusably inefficient - it always does a full table scan, generating a random number for each row, sorts the full table that way, and then returns it. Disk grindingly CPU meltingly inefficient.

That said, there are workarounds to achieve a random ordering. But they're just that, workarounds, I didn't find any of them satisfying, especially given that there is a simpler way.

Now is the moment to introduce the hero of our story -- SKIP LOCKED.

SKIP LOCKED is a special hint we can give to Postgres to tell it to just move on if it finds a row that has been locked by someone else. This normally wouldn't make sense for database queries (since it will give an inconsistent view of the data), but it is exactly the thing we need if we're trying ot use the database as a queue. In fact, the queue use case is the reason why this hint was introduced in the first place.

Armed with this bit of trickery, let us chant our spell again.

UPDATE object_copies SET lock = now_utc_micro_seconds()
WHERE object_key = (
    SELECT object_key FROM object_copies
    WHERE (wasabi IS NULL OR scw IS NULL) AND lock IS NULL
    LIMIT 1
    FOR UPDATE SKIP LOCKED)

Beautiful. There is one small garnish we need to add - when we run this query, our Gopher would also need to know the object_key and which of the two replicas needs syncing. For this, we can add a RETURNING clause to the UPDATE to return all relevant fields:

UPDATE object_copies SET lock = now_utc_micro_seconds()
WHERE object_key = (
    SELECT object_key FROM object_copies
    WHERE (wasabi IS NULL OR scw IS NULL) AND lock IS NULL
    LIMIT 1
    FOR UPDATE SKIP LOCKED)
RETURNING object_key, wasabi, scw

Our query is now complete.

In case you're wondering where all of this magic is documented, stop (wondering).

It is all there in the excellent PostgreSQL documentation.

We do need help from other sources to connect the various parts sometimes, e.g. this post about SKIP > LOCKED was very helpful to me. But the details are all there in the Postgres documentation itself, so RTFM.

So are we done here?

Let's look back at our original list of requirements:

Keep track of which files need to be uploaded, and where.

Lock a file that is currently being uploaded.

Clean up stray locks on unexpected restarts.

Do all of this scalably so that we can have many Gophers running in parallel.

As much as possible, make it fast.

Items 1 and 2 are done, and so is 4.

This post is already too long, so I won't delve into details for #3, but I'll give a brief overview.

Clean up stray locks on unexpected restarts.

One approach to handle this cleanup would be to obviate the need for a cleanup (duh). We can do that by explicitly starting a transaction in which our UPDATE/SELECT statement runs, but keep the transaction running as we do the replication. This way, if the replication were to not complete because of some reason, the transaction would never be committed and the next Gopher that comes around will find the same row in its original state, and pick it up.

In fact, this way, we wouldn't need the lock field at all.

That said, for now I've gone with an external cleanup. There is a periodic "cron" goroutine that clears out any entries where the value of the lock field is older than a day. Seems to be working fine for now, but I can foresee myself revisiting this some day and switching to the cleaner(!) first approach.

So let us focus on requirement 5 for the rest of the post.

As much as possible, make it fast.

As with anything to do with databases, there is a straightforward answer on how to achieve that – add an index 😬

Hell, let's add two 🥶

CREATE INDEX ON object_copies (wasabi) WHERE wasabi IS NULL;
CREATE INDEX ON object_copies (scw) WHERE scw IS NULL;

This is not the only possibility - there are other indicies we could've used too. For example, we could've added a single combined indexed with the same WHERE clause as our actual SELECT query.

But by adding separate indexes, we allow for the possibility of future new columns for new replicas being added to the same table, without invalidating the existing indexes. The newer columns will get their own indexes.

Another practical reason is that we have reporting dashboards that show the number of unreplicated files in each replica, and having these separate indexes per replica make those reporting queries fast.

One thing to note here is the WHERE clause in the CREATE INDEX.

The idea behind that being that usually there are only going to be a small number of new files (relative to the total number of files) that are not replicated yet. So by creating an index on WHERE wasabi is NULL / WHERE scw is NULL, we don't waste Postgres' time (and disk) asking it to keep an index on rows that have already been replicated.

So does this work? Let's ask Postgres!

EXPLAIN
UPDATE object_copies SET lock = now_utc_micro_seconds()
WHERE object_key = (
    SELECT object_key FROM object_copies
    WHERE (wasabi IS NULL OR scw IS NULL) AND lock IS NULL
    LIMIT 1 FOR UPDATE SKIP LOCKED)
RETURNING object_key, wasabi, scw

This is our final query as before, just wrapped in an EXPLAIN, requesting Postgres to tell us about what it's doing behind the scenes.

                                    QUERY PLAN
---------------------------------------------------------------------------------------
 Update on object_copies  (...)
   InitPlan 1 (returns $1)
     ->  Limit  (...)
           ->  LockRows  (...)
                 ->  Seq Scan on object_copies object_copies_1  (...)
                       Filter: (((wasabi IS NULL) OR (scw IS NULL)) AND (lock IS NULL))
   ->  Index Scan using object_copies_pkey on object_copies  (...)
         Index Cond: (object_key = $1)

From the Seq Scan, we can see that Postgres did not use our newly created indices. Hmm.

At this point come to mind the two excellent tips provided by pgMustard in their clearly titled post - Why is Postgres not using my index?! (the exclamation mark is my own 🤓)

There are two main reasons that Postgres will not use an index. Either it can’t use the index, or it doesn’t think using the index will be faster.

Their post goes on two show how you can differentiate between the two reasons.

Let's start by ruling out reason 1. This is easy - we tell Postgres to imagine a world in which sequential scans are very costly

SET enable_seqscan = off

and then run our EXPLAIN again

                                     QUERY PLAN
--------------------------------------------------------------------------------------
 Update on object_copies  (...)
   InitPlan 1 (returns $1)
     ->  Limit  (...)
           ->  LockRows  (...)
                 ->  Bitmap Heap Scan on object_copies object_copies_1  (...)
                       Recheck Cond: ((wasabi IS NULL) OR (scw IS NULL))
                       Filter: (lock IS NULL)
                       ->  BitmapOr  (...)
                             ->  Bitmap Index Scan on object_copies_wasabi_idx  (...)
th=0)
                                   Index Cond: (wasabi IS NULL)
                             ->  Bitmap Index Scan on object_copies_scw_idx  (...)
0)
                                   Index Cond: (scw IS NULL)
   ->  Index Scan using object_copies_pkey on object_copies  (...)
         Index Cond: (object_key = $1)

It used our index! We don't need to understand all the details of what the QUERY PLAN is telling us, but we can already see it is using Bitmap Index Scan and Bitmap Heap Scan and other fancy stuff instead of a plain old sequential scan.

If you need to understand more of what the query plan is trying to tell us, the Postgres docs for EXPLAIN are a great resource.

So the reason Postgres wasn't using our indicies earlier was because it doesn’t think using the index will be faster. Why could that be?

There isn't any data in the table! Let us add some data. First, let us reset enable_seqscan,

SET enable_seqscan = on

Then let us add some rows to the table,

INSERT into object_copies (object_key)
SELECT generate_series(1, (10^6)::int)::text

and run the EXPLAIN again.

Unfortunately, this time we'll find that it went back to using a sequential scan.

If you're not seeing the results that we're discussing, then try to run
ANALYZE
and re-run the EXPLAIN.

ANALYZE causes Postgres to refresh its statistics about what the table contains.

Maybe we need to add more data. What's a million rows anyways, a table for ants?

But no, that's not the issue here. In addition to the amount of data, Postgres also considers the distribution of data. Currently, all our rows have all both wasabi and scw set to NULL, so using the indicies we'd created wouldn't help in any way.

Let us try to modify the data in the table to reflect the sort of scenario we'll face in production -- most rows have been replicated, but here and there are a few files that are left.

UPDATE object_copies SET wasabi = 1, scw = 1;
UPDATE object_copies SET wasabi = NULL WHERE mod(object_key::int, 41) = 0;
UPDATE object_copies SET scw = NULL WHERE mod(object_key::int, 61) = 0;

Now let us run EXPLAIN again.

Indeed, this time it uses the indices!

So with that, our work here is done.

Hopefully this was useful to you. And if not useful, at least you found my bumbling around to the final solution enjoyable.

Till next time, happy hacking!

Custom S3 requests with AWS Go SDK

ente — Wed, 23 Nov 2022 05:28:20 +0000

Making requests to custom S3 endpoints using the AWS Go SDK

Maybe there is an easier way to do this.

Apologies for starting this post on an indecisive note, but I'm still not sure if this is indeed the "correct" way. It works though 🤷

What I wanted was to make a request to a custom endpoint provided by an S3 compatible service. This endpoint looks and behaves very much like standard S3 APIs, but since it is not part of the suite that AWS provides the AWS SDKs don't have a way to directly use it.

Of course, I could make an HTTP request on my own. I'd even be fine with parsing the XML (yuck!). But what I didn't want to deal with were signatures.

So I thought to myself that there would be standard recipes to use the AWS SDK (I'm using the golang one) to make requests to such custom endpoints. But to my surprise, I didn't find any documentation or posts about doing this. So, yes, I wrote one 🙂

The end result is quite simple - we tell the AWS Go SDK about our custom input and output payloads, and the HTTP path, and that's about it, it does all the heavy lifting (signing the requests, the HTTP request itself, and XML parsing) for us.

You can scroll to the bottom if you just want the code.

Otherwise, I'll walk through this in two parts:

First we'll see how requests to standard endpoints are made under the hood.
Then we'll use that knowledge to get the SDK to make our custom payloads and request go through the same code paths.

Enough background, let's get started. Let's use the Go SDK to fetch the versioning status of an S3 bucket.

Why versioning? It is a simple GET request, conceptually and in code – we're just fetching a couple of attributes attached to a bucket. This way, we can focus on the mechanics of the request without getting lost in the details of the specific operation we're performing.

Here is the code. Create a session. Use the session to create a client. Use the client to make the request.

package main

import (
    "fmt"

    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/credentials"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/s3"
)

func main() {
    // Example credentials to connect to local MinIO. Don't hardcode your
    // credentials if you're doing this for real!
    creds := credentials.NewStaticCredentials("minioadmin", "minioadmin", "")
    sess, err := session.NewSession(&aws.Config{
        Credentials: creds,
        Region:      aws.String("us-east-1"),
        // Makes it easy to connect to localhost MinIO instances.
        S3ForcePathStyle: aws.Bool(true),
        Endpoint:         aws.String("http://localhost:9000"),
        LogLevel:         aws.LogLevel(aws.LogDebugWithHTTPBody),
    })
    if err != nil {
        fmt.Printf("NewSession error: %s\n", err)
        return
    }

    s3Client := s3.New(sess)

    bucket := "my-bucket"

    // Create a bucket, ignoring any errors if it already exists.
    s3Client.CreateBucket(&s3.CreateBucketInput{
        Bucket: aws.String(bucket),
    })

    // Everything above was just preparation, let's make the actual request.

    output, err := s3Client.GetBucketVersioning(&s3.GetBucketVersioningInput{
        Bucket: aws.String(bucket),
    })
    if err != nil {
        fmt.Printf("GetBucketVersioning error: %s\n", err)
        return
    }

    fmt.Printf("GetBucketVersioning output: %v\n", output)
}

To run this code, add it to a new go project.

$ mkdir s3-playground
$ cd s3-playground
$ go mod init example.org/s3-playground
# Paste the above code into main.go
$ pbpaste > main.go
$ go get

In a separate terminal, start a local MinIO instance.

$ docker run -it --rm -p 9000:9000 minio/minio server /data

Back in our original terminal, run the program.

$ go run main.go

If everything went according to plan, you'll now see debug logs of the HTTP requests made by our program, and finally it will print the response for the bucket versioning request:

GetBucketVersioning output: {

}

Great. So now that we have our playground, let's dissect the request. The heart of the program is this bit of code:

output, err := s3Client.GetBucketVersioning(&s3.GetBucketVersioningInput{
    Bucket: aws.String(bucket),
})

It is all quite straightforward.

Create an input payload (GetBucketVersioningInput)
Make an API call with that input (GetBucketVersioning)
Get back the output (of type GetBucketVersioningOutput)

So if we were to make a custom API request, we already know that we'll need a MyCustomOperationInput and MyCustomOperationOutput. But that's not enough, we'll also need to change the HTTP request path, possibly even the HTTP method.

So we need to unravel one layer.

If you were to click through to the source of GetBucketVersioning in your editor, you'll see the following code (in aws-sdk-go/service/s3/api.go)

func (c *S3) GetBucketVersioning(input *GetBucketVersioningInput)
        (*GetBucketVersioningOutput, error) {
    req, out := c.GetBucketVersioningRequest(input)
    return out, req.Send()
}

It is creating a Request and an Output. Sending the request. And returning the Output.

Let's keep digging. Looking at the source of GetBucketVersioningRequest:

func (c *S3) GetBucketVersioningRequest(input *GetBucketVersioningInput)
        (req *request.Request, output *GetBucketVersioningOutput) {
    op := &request.Operation{
        Name:       opGetBucketVersioning,
        HTTPMethod: "GET",
        HTTPPath:   "/{Bucket}?versioning",
    }

    if input == nil {
        input = &GetBucketVersioningInput{}
    }

    output = &GetBucketVersioningOutput{}
    req = c.newRequest(op, input, output)
    return
}

Nice, we can see where HTTP request path and HTTP method are being set. Let us also look into the source of newRequest:

func (c *S3) newRequest(op *request.Operation, params, data interface{})
        *request.Request {
    req := c.NewRequest(op, params, data)

    // Run custom request initialization if present
    if initRequest != nil {
        initRequest(req)
    }

    return req
}

This is essentially doing c.NewRequest (the rest of the code is for allowing us to intercept the request before it is used).

So now we have all the actors on the stage. Let us inline the code that we saw above in the AWS Go SDK source into our own program. We will replace its original heart:

output, err := s3Client.GetBucketVersioning(&s3.GetBucketVersioningInput{
    Bucket: aws.String(bucket),
})

with this inlined / rearranged version:

input := &s3.GetBucketVersioningInput{
    Bucket: aws.String(bucket),
}

// this'll need to import "github.com/aws/aws-sdk-go/aws/request"
op := &request.Operation{
    Name:       "GetBucketVersioning",
    HTTPMethod: "GET",
    HTTPPath:   "/{Bucket}?versioning",
}

output := &s3.GetBucketVersioningOutput{}
req := s3Client.NewRequest(op, input, output)

err = req.Send()

Is that it? The code is simple enough, and we can see the input payload, the request itself, and the parsed output. If we substitute each of these with our custom versions, will it just work?

There's only one way to find out 😎

For this second part of this post, I tried finding an example MinIO-only API for pedagogical purposes, but couldn't find one off hand. So let's continue by using an example related to the actual custom endpoint that I'd needed to use.

We use Wasabi as one of the replicas for storing encrypted user data. We'll put out an article soon with details about ente's replication, but for our purposes here it suffices to mention that we use the Wasabi's "Compliance" feature to ensure that user data cannot be deleted even if some attacker were to get hold of Wasabi API keys.

To fit this into our replication strategy, we need to make an API call from our servers to tell Wasabi to "unlock" the file and remove it from the compliance protection when the user herself deletes it, so that it can then be scheduled for deletion after the compliance period is over.

To keep the post simple, let us consider a related but simpler custom Wasabi API: getting the current compliance settings for a bucket.

The compliance settings for a bucket can be retrieved by getting the bucket with the "?compliance" query string. For example:

GET http://s3.wasabisys.com/my-buck?complianceHTTP/1.1

Response body:

<BucketComplianceConfiguration xml ns="http://s3.amazonaws.com/doc/2006-03-01/">
   <Status>enabled</Status>
   <LockTime>2016-11-07T15:08:05Z</LockTime>
   <IsLocked>false</IsLocked>
   <RetentionDays>0</RetentionDays>
   <ConditionalHold>false</ConditionalHold>
   <DeleteAfterRetention>false</DeleteAfterRetention>
</BucketComplianceConfiguration>

As you can see, it looks and behaves just like other S3 REST APIs. Except that the AWS Go SDK does not have a pre-existing method to perform this operation.

So how do we call this API using the AWS Go SDK?

Let us modify code that we ended up with in first section, but retrofit the input / request / output objects to match the documentation of this custom API.

Let's start with definition of GetBucketVersioningInput from the AWS Go SDK:

type GetBucketVersioningInput struct {
    _ struct{} `locationName:"GetBucketVersioningRequest" type:"structure"`

    Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"`

    ExpectedBucketOwner *string `location:"header" locationName:"x-amz-expected-bucket-owner" type:"string"`
}

Hmm. We don't seem to need the ExpectedBucketOwner field, so let's remove that. We do need the Bucket field, and it is passed in the as location:"uri", so let's keep the rest as it is, and arrive at our retrofitted GetBucketComplianceInput:

type GetBucketComplianceInput struct {
    _ struct{} `locationName:"GetBucketComplianceRequest" type:"structure"`

    Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"`
}

Let us repeat the process for the Output. Here's the original:

type GetBucketVersioningOutput struct {
    _ struct{} `type:"structure"`

    MFADelete *string `locationName:"MfaDelete" type:"string" enum:"MFADeleteStatus"`

    Status *string `type:"string" enum:"BucketVersioningStatus"`
}

And here it is, modified to match the documentation of the custom API:

type GetBucketComplianceOutput struct {
    _ struct{} `type:"structure"`

    Status               *string `type:"string"`
    LockTime             *string `type:"string"`
    RetentionDays        *int64  `type:"integer"`
    ConditionalHold      *bool   `type:"boolean"`
    DeleteAfterRetention *bool   `type:"boolean"`
}

Finally, let us change the middle part – the request.

op := &request.Operation{
    Name:       "GetBucketCompliance",
    HTTPMethod: "GET",
    HTTPPath:   "/{Bucket}?compliance",
}

Adding the new definitions of GetBucketComplianceInput and GetBucketComplianceOutput and making the other changes, we'll end up with this final version of the heart of our program:

input := &GetBucketComplianceInput{
    Bucket: aws.String(bucket),
}

op := &request.Operation{
    Name:       "GetBucketCompliance",
    HTTPMethod: "GET",
    HTTPPath:   "/{Bucket}?compliance",
}

output := &GetBucketComplianceOutput{}
req := s3Client.NewRequest(op, input, output)

err = req.Send()
if err != nil {
    fmt.Printf("GetBucketCompliance error: %s\n", err)
    return
}

fmt.Printf("\nGetBucketCompliance status: %v\n", *output.Status)

Will this work 😅? Now we'll find out.

You'll have to take my word (or get a Wasabi account), but if I take the program with these changes, add proper credentials, and run it, it does indeed work.

$ go run main.go
...
2022/11/22 21:29:38 DEBUG: Response s3/GetBucketCompliance Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Content-Length: 136
Content-Type: text/xml
Date: Tue, 22 Nov 2022 15:59:38 GMT
Server: WasabiS3/7.9.1306-2022-11-09-489242991d (head3)
X-Amz-Id-2: ...
X-Amz-Request-Id: ...


-----------------------------------------------------
2022/11/22 21:29:38 <BucketComplianceConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Status>disabled</Status></BucketComplianceConfiguration>

GetBucketCompliance status: disabled

Sweet!

Also, hats off to the AWS gophers for a well designed SDK - we needed to just modify the payloads & change the HTTP path, everything else just worked.

In the end, the solution is simple and is just the obvious set of changes one can expect, but when I was doing this I hadn't been sure until the moment I actually made the final request if it'll work or not. So I hope this post might be useful to someone who finds themselves on the same road - Carry on, it'll work!

Till next time, happy coding 🧑‍💻

Command to publish an app to TestFlight

ente — Thu, 22 Sep 2022 11:13:56 +0000

At ente, we are building an open-source iOS app for backing up and sharing your photos and videos.

In the beginning, we used XCode to create an Archive and then painfully clicked 20 buttons to upload our build to TestFlight.

We then discovered an easy way to do the same from the command line with these steps:

Configure Transporter.
Run {command-to-build} && open -a "Transporter" {path-to-generated-ipa}.
That's it!

In our case, since we are using Flutter, the single line command to build and publish to TestFlight is:

flutter build ipa && open -a "Transporter" build/ios/ipa/*.ipa

Hope this will save you a few clicks! If you'd like to know more of our adventures with Flutter and AppStore, follow us on Twitter, or come hang out on our lovely Discord!

The minimum TypeScript you need for React

ente — Fri, 19 Aug 2022 16:09:06 +0000

TypeScript is awesome. It allows one to remain in the fun wild-west of JavaScript, while being safe in the blanket of a state of the art type system. Plus, it a great way to adopt strong type checking gradually without having to rewrite the entire codebase, or get a Ph.D in type theory (I'm looking at you, Haskell 🦥).

But TypeScript is also has a learning curve. For some of us, it's a steep one. Everything is well documented, but there is a lot of the said documentation 😅. There are also cheetsheets, say for using TypeScript with React, but they assume a basic understanding of TypeScript. So what is missing (or at least I haven't found) are quick recipes for the most common use cases we run into when trying to introduce TypeScript into an existing React app. Here goes.

The minimum TypeScript you need to know to type your way around React

Let us start with a simple React component:

// Hello.tsx

import React from 'react';

const Hello = () => {
    return <div>Hello</div>;
};

To add types to this component, we need to add a type annotation to Hello. This is what React calls as a "Function Component", or "FC" for short. React is a bro, and also provides us with its (TypeScript) type — React.FC:

import React from 'react';

const Hello: React.FC = () => {
    return <div>Hello</div>;
};

This (and the other types we'll add below) are part of the @types/react and @types/react-dom packages, so you'll need to have them installed.

Moving on. A hello is nice, but saying hello while addressing a person by their name is nicer. Let's teach our Hello component to accept a name prop:

import React from 'react';

const Hello: React.FC = ({ name }) => {
    return <div>{`Hello, ${name}!`}</div>;
};

Oh noes, the dreaded red squiggly appears.

This is a good time to bring up an important point. TypeScript does its thing at compile-time (when we write the code). So even though it is complaining here, when we actually run the code ("runtime"), it will work as normal. In that sense, this is more of a warning than an error.

So if we ignore TypeScript errors, things might or might not work at runtime. But if we don't ignore them, we can be sure that things will work at runtime (the things that TypeScript can check for, that is).

Back to the squiggly. What tsc (Type*Script **C*ompiler) is telling us is that a plain React.FC does not take any props, but here we're trying to read a name.

So we need to tell tsc that this is not a plain React.FC, it is a enhanced version that also takes a name prop, and that this name is a string.

import React from 'react';

const Hello: React.FC<{ name: string }> = ({ name }) => {
    return <div>{`Hello, ${name}!`}</div>;
};

This satisfies tsc. But it is not very readable, and it'll keep getting gookier the more props we add. So let us define a type that describes the props that our component takes, and then use that type to enhance React.FC.

import React from 'react';

interface HelloProps {
    name: string;
}

const Hello: React.FC<HelloProps> = ({ name }) => {
    return <div>{`Hello, ${name}!`}</div>;
};

Perfect.

Or not quite yet. What good is a web developer who greets a user without a CTA in mind? 😛

But our Hello component is perfect as it is – like a textbook React component, it does one thing, and does one thing well, and is a great building block. Adding new functionality to this would spoil that perfection.

So let us pass the CTA as a child. This way, we can reuse the Hello component in different places to greet the user the same way yet call on them to do different things.

import React from 'react';

interface HelloProps {
    name: string;
}

const Hello: React.FC<HelloProps> = ({ name, children }) => {
    return (
        <div>
            <div>{`Hello, ${name}!`}</div>
            {children}
        </div>
    );
};

Oh noes, the red squiggly reappears! This time it is complaining that

Property 'children' does not exist on type 'HelloProps'.

Fair enough. To fix this, we can add children to HelloProps, though for doing that we'll need to figure out what is the type of children.

But wait a sec. Taking children props looks like a common enough, bread and butter need for React components. Isn't there a standard type for such components?

Why, indeed there is. It is called PropsWithChildren (yay for nice descriptive names!).

import React, { PropsWithChildren } from 'react';

interface HelloProps {
    name: string;
}

const Hello: React.FC<PropsWithChildren<HelloProps>> = ({ name, children }) => {
    return (
        <div>
            <div>{`Hello, ${name}!`}</div>
            {children}
        </div>
    );
};

Brilliant.

We can't wait to use this to ~~yell at~~ greet our users, and immediately start using it. It works great for a while, but then we run into a special page. Turns out, we do indeed need to yell at the user on the pricing page, to make sure they know that we really really like them.

Helpfully, our designer has written some nice CSS to style the yell appropriately, we just need to add that class to our div.

import React, { PropsWithChildren } from 'react';

interface HelloProps {
    name: string;
}

const Hello: React.FC<PropsWithChildren<HelloProps>> = ({
    name,
    className,
    children,
}) => {
    return (
        <div>
            <div className={`{className ?? ''}`}>{`Hello, ${name}!`}</div>
            {children}
        </div>
    );
};

Guess what, the red squiggly strikes again!

Property 'className' does not exist on type 'PropsWithChildren'

We now know what this means, and how to fix it – we need to add className to our props. But hold on, this too seems like a very standard, bread and butter type(!) thing for React components. Isn't there a standard type to say that we'd like className, or style, or id or any one of the other such props that are accepted by HTML div elements?

Glad you asked, because there is. It is called HTMLAttributes.

But it doesn't jive on its own – we also need to tell it which HTML element it is whose HTML attributes we're talking about. In our case, we're talking of a standard HTML div, so we will use HTMLAttributes<HTMLDivElement>.

import React, { HTMLAttributes, PropsWithChildren } from 'react';

interface HelloProps extends HTMLAttributes<HTMLDivElement> {
    name: string;
}

const Hello: React.FC<PropsWithChildren<HelloProps>> = ({
    name,
    children,
    ...rest
}) => {
    return (
        <div>
            <div {...rest}>{`Hello, ${name}!`}</div>
            {children}
        </div>
    );
};

That's it, really. Armed with these basic types, you'll know how to type your way around React.

You'll hit more advanced cases eventually – say that instead of a vanilla div, you wish to create something that behaves like the HTML link (<a> - the anchor tag). But you won't remain befuddled for long on that, for now you know that you just need to find what's the HTMLDivElement equivalent for a anchor tag and use that instead. And these types are quite nicely named, so that type is usually something whose name you can guess (for example, for an <a> tag, it is HTMLAttributes<HTMLAnchorElement>).

Hope this was helpful! If you'd like to know more of our adventures with TypeScript, React, and other things of the ilk, do follow us on Twitter, or come hang out on our lovely Discord.

Tell next time, happy typ(e)ing! 🤓

Gatsby blog images in 2022

ente — Thu, 28 Jul 2022 05:28:04 +0000

We recently cleaned the image related code in our Gatsby blog, and this is a tldr of what we learned.

Gatsby provides three options to handle images:

Assets from filesystem (import myImage from './myImage.png')
Gatsby image plugin (<StaticImage ...> and it's dynamic variant)
Static folder (<img ...>)

https://www.gatsbyjs.com/docs/how-to/images-and-media/

For blog posts written in MDX, none of these are particularly convenient though. For that, Gatsby provides an official plugin

https://www.gatsbyjs.com/plugins/gatsby-plugin-image/

This works quite seamlessly. We keep the image in the same folder as the .mdx and can then use the standard Markdown ![alt text](./foo.png) syntax to include them in our posts.

Behind the scenes, it'll use the same method as Gatsby image plugin (#2, <StaticImage...> above) and support responsive variants using srcsets + blurred placeholders + async loading etc. More details are about this are in

https://www.gatsbyjs.com/docs/working-with-images-in-markdown/

The issue though is that there is only a global maxWidth option, and no way to specify it per image. Thankfully, there is a community plugin that allows us to do that.

https://www.gatsbyjs.com/plugins/@bonobolabs/gatsby-remark-images-custom-widths/

This is intended as a drop-in replacement for gatsby-remark-images, and supports most of the same options. To configure this, one needs to add the following to gatsby-config.js:

{
    resolve: 'gatsby-plugin-mdx',
    options: {
        gatsbyRemarkPlugins: [
            {
                resolve: '@bonobolabs/gatsby-remark-images-custom-widths',
                options: {
                    maxWidth: 640,
                    quality: 80,
                    backgroundColor: 'transparent',
                    withWebp: true,
                    linkImagesToOriginal: false,
                },
            },
        ],
    },
},

All the options are optional, the above are for illustration only. For example, we also needed to disable linkImagesToOriginal because otherwise all the blog images would get wrapped in anchor tags, causing the mouse cursor to change to a hand pointer (detracting from the reading experience).

Once it is configured, then using the plugin is simple. Just add an img tag to your MDX, and specify a width:

Some example **markdown** content.

<img src="cat.png" width="310px" alt="Cat!"/>

More markdown content. Type away!

Rest of the image processing pipeline will work the same way. We will get a blurred variant that is shown while the actual image is being downloaded, and the image being downloaded will be the size appropriate to the user's screen.

For our blog, the convention we follow for now is that when adding images to MDX, we specify the 1x width but save the image at the 3x size. This way, Gatsby has high quality data to create large variants for users whose screens support 3x resolutions, but for other users it'll generate and fetch smaller images depending on their screen resolution and size.

For example, if we have a 240 px wide image (the 1x size), then we'll store a 720 px width image (the 3x size), and in the MDX we'll specify width="240".

We wouldn't let you come this far, talk of images, and not show you a cat pic!

So, here, meet Honey

On issue that still bugs us is lack of SVG support. For this we have to fallback to either option #1 (Assets from filesystem) or option #2 (Static folder). Maybe one of you would be kind enough to let us know (Twitter, Discord) if there is a simpler way!

Till next time, happy blogging 👩‍💻