Using Google Cloud Firestore with Django's ORM

Luke Benstead — Mon, 20 May 2024 14:34:06 +0000

Django has long been the most popular Python framework for developing web applications. One of its most powerful features is its built in object-relational mapper (ORM) which is designed to flexibly and safely interact with SQL databases in an abstract way.

A long time ago, a fork of Django called “Django-nonrel” experimented with the idea of using Django’s ORM with a non-relational database; what was then called the App Engine Datastore, but is now known as Google Cloud Datastore (or technically, Google Cloud Firestore in Datastore Mode). Since then a more recent project called "django-gcloud-connectors" has been developed by Potato to allow seamless ORM integration with Google Cloud Datastore.

The intention was always to expand that functionality to other Google hosted no-SQL databases, and today I’m pleased to announce that django-gcloud-connectors now has support for Google Cloud Firestore in Native Mode. This means you can now build a robust Django-powered website or API driven from a Firestore database which could for example be shared with your Firebase backed mobile application.

It might seem that using Django’s relational mapper on a non-relational database would be like forcing a square peg into a round hole - but it turns out that the Django ORM works quite elegantly with a non-relational database, and gives you the tools you need to build reusable patterns to get the most out of a massively scalable database.

Take for example the schemaless nature of a non-rel database like Cloud Datastore which allows each document to have different properties to the other documents in the same table; while powerful, in practice you generally need a schema to write code against and to ensure that your data remains consistent. Django’s model structure allows you to maintain a schema (with validation) that can change over time. You can leverage model signals, field defaults or request middleware to migrate data on the fly, without the need for running potentially slow or disruptive database migrations. You hardly need to worry about the database at all while developing, simply add a new model or field in your code and use it. Django with a non-relational database is great for rapid prototyping.

There are limitations of course; non-relational databases don’t do complex queries - you can’t perform joins; you can have foreign keys, but you can’t fetch or order across those relations in one database query; you can have many-to-many relationships, but not with Django’s ManyToManyField (you need to use a custom list field instead). You might just find some queries are impossible.

These limitations mean you have to solve problems in a slightly different way. You’ll find you’ll design your models around the queries you need to make; you’ll sometimes have to pre-calculate things when you save a document to make it possible to quickly query for it later, or perhaps use other services like Big Query or background task processing.

But you can almost always assume that once your code works, it’ll scale well no matter how much data you throw at it; there’s no risk of running an unindexed query on a million rows, or having to think about database replication or sharding. You can rapidly build a product for a startup and know that things will just ramp up when it hits the big time.

I highly recommend you give it a try, simply follow the instructions in the django-gcloud-connectors repository and you’ll be up and running in no time. If you’re hosting on Google App Engine you might also be interested in the Djangae project which provides useful utilities and Django apps to integrate with Google Cloud services.

How Potato Code Remotely

Luke Benstead — Thu, 21 May 2020 08:17:15 +0000

At the time of writing, we’re in the middle of the COVID-19 lockdown. As with most other companies, the lockdown has required us at Potato to rapidly change the way we work day-to-day. We’re a tech company, so that has given us a big advantage in this rapid and unsettling change of routine. But as a company, we’ve historically encouraged our engineers to travel into our central London office most of the time. Working from home has always been a perk at Potato, but never the default position.

Fortunately we’ve been unintentionally preparing for this day by making the right choice of tools. In this post I hope to give an overview of our toolset, and how it’s allowed us to continue to operate as normal.

Cloud-First

Right from the beginning of Potato, we’ve always pushed to use tools that are hosted by others. We’ve never wanted to spend our time being sysadmins, and we’ve always resisted the urge to host anything inside our offices. We wanted all of our stuff to be available anywhere, from any platform. In the past the option of running a VPN, and hosting software internally has come up, but we’ve steadfastly refused the temptation to do so - the risk of losing access, of having to maintain it, and of a single point of failure (what if the office burned down?!) were all reasons to avoid it.

Also, we wanted to make sure that any of our Engineers could deploy any of our projects from anywhere with an Internet connection. In the past we’ve deployed from trains, planes, pubs, and on one occasion a long while back - a locked bathroom (long story!).

Finally, if anyone ever had their laptop stolen out of the back of a van while climbing (true story), nothing would be lost if everything is saved online.

For this reason, (along with our Google history) we’ve based all of our internal systems on Google’s GSuite. Potato runs on Google Docs, Sheets, Calendar and of course Gmail. This has given us access to everything from anywhere, and the decision to base everything around GSuite has allowed us to continue working seamlessly despite now all being remote.

Slack / Google Meet

For communication, we’ve been long-time users of Slack, (although before this we were pretty sold on Hipchat). As it’s cloud-based we’ve just continued messaging as normal without disruption (though probably making a bit more use of threads to keep things organised).

For video calls we’ve always used Google Meet (previously Hangouts). The number of ad-hoc catch-ups has increased, and we hold daily drop-in sessions for people who just want a bit of socialising. On Fridays we’ve rebooted our traditional TPIF (Thank Potato It’s Friday) into a virtual mass video call, armed with beverages of choice and reams of friendly banter!

GitLab

A couple of years ago we moved all of our development over to GitLab. The intention was to bring all of our end-to-end product development life-cycle under one tool. Although we had the option of hosting GitLab ourselves, using GitLab.com’s hosting has saved us the overhead of managing it, and of course it means that our project management tools are available from anywhere.

Now that we’re all a bit more involved in using GitLab directly day-to-day, we’ve also implemented a Chrome/Firefox extension that makes things a little easier.

Visual Studio Code - Live Share

The ability to grab another developer to help solve a problem has always been a key part of Potato culture, and historically one that required that other dev to be in the same room. When we were forced into remote working this was one area where we needed to find a new tool to help us, and we did so by using Visual Studio Code’s “Live Share” plugin. This plugin lets you host a programming session that you can invite others to collaborate in (think Google Docs, but for code editing). Combined with a Google Meet call it makes for a great, and simple pair-programming experience.

Figma

The final tool in our toolbox is Figma. Until recently our designers were heavily invested in using Sketch as their main design tool, but we’d noticed issues even ahead of the lockdown that made this problematic.

Firstly, it broke the cloud-first rule. Sketch ran locally, saved files locally, and it wasn’t cross-platform which caused a load of headaches for our Linux-using developers. We came up with workarounds for these things (e.g. using the “Measure” plugin for detailed exports, teaching the designers how to use Git etc.) but it never really fitted the culture of Potato.

Then Figma came along, and not only did the designers find that it was a better tool, but it’s based in the cloud and so is platform-agnostic.

Honourable Mentions

In addition to our development toolset, we use a number of other web-based tools day-to-day, these include:

Harvest + Forecast - for time tracking and project scheduling
Bamboo - for managing our people
Trello - For workshopping and lightweight task tracking
TrakStar - For managing the career progression of our people # Summary

There we are, a complete toolset for seamless remote working - that were built up accidentally by just following the rule that our stuff should be accessible from anywhere, and we don’t want to be the ones running it!

Forem: Luke Benstead