Forem: Gerard Clos

6 Best Embedded Databases for 2024 📊

Gerard Clos — Thu, 02 May 2024 13:15:34 +0000

Embedded databases provide many benefits compared to traditional databases. They are simpler to set up and manage, require fewer resources, and deliver superior performance for applications that demand quick data processing and low latency.

As a counterpoint, it is often argued that embedded databases don’t scale but – for most folks out there – this is simply not true anymore.

In this article, I'll highlight the top embedded database options for various scenarios, helping you identify the ideal database for your specific requirements.

Ready to dive in? Let's get started.

1. SQLite

SQLite is the de-facto-embedded relational database. So much so that it’s become the most used database in the world. It runs as a simple library within an application, providing a robust solution without requiring a separate process. SQLite supports full SQL syntax and is incredibly compact, making it an ideal choice for applications across various platforms including mobile devices, desktop applications, and large-scale websites and web applications.

Its simplicity and the ability to operate without configuration while maintaining a small footprint make SQLite a popular choice among developers for applications where database requirements are modest but demands for reliability and efficiency are high.

SQLite homepage

2. DuckDB

DuckDB, an in-process SQL OLAP database, is designed specifically for on-the-fly data analysis and is intended to be easy to integrate into other projects. It operates directly on blobs of data and excels in executing complex analytical SQL queries efficiently. DuckDB's design focuses on vectorized query execution, making it particularly well-suited for analytic workloads where performance and speed are critical.

Its ability to run embedded within other applications without any external dependencies makes it a powerful tool for developers looking to embed data analysis capabilities directly into their applications.

DuckDB homepage

Support us ⭐

This is not an embedded database, but still relevant if you need to do embedded analytics. We know that building it from scratch can be difficult. That is why we created Latitude. We're developing it as an open-source project. If you’d like to try, here's a quick start guide available on our repository. If you like it, consider showing your support by giving us a star, it’d mean a lot! ⭐

https://github.com/latitude-dev/latitude

3. RocksDB

RocksDB, developed by Facebook, is a key-value store optimized for fast reads and writes at a high volume, suitable for applications that demand efficient data persistence.

RocksDB's use of a log-structured merge-tree (LSM tree) is a key aspect of its design, allowing it to achieve high write throughput while maintaining the ability to support reads. This makes it a valuable tool for applications that need to handle large streams of incoming data with minimal latency.

RocksDB homepage

4. Chroma

Chroma is a vector database designed to handle high-throughput and multi-dimensional data with ease. It become widely popular with the advent of genAI, which introduced the need to use vector stores in order to provide AI agents with long term “memory”. It’s main use case is to implement RAG for AI agents.

Chroma utilizes a unique storage architecture that optimizes data retrieval processes, significantly reducing query times and enhancing overall application performance. Its ability to efficiently manage large-scale data makes it an excellent choice for developers working on enterprise-level solutions that require robust, scalable, and highly available data management capabilities.

Chroma homepage

5. Kùzu

Kùzu is an embedded database with a novel & fast query processor designed for high scalability on a single machine (billions of nodes). Kùzu supports Property Graph + RDF data models & interop with RDBMS tools.

On top of that, Kuzu offers offline capabilities and synchronization with cloud services and is built to support SQL and NoSQL data models, making it highly adaptable to various data structures and application requirements.

Kùzu homepage

6. Faiss

Faiss is the other big vector database in the AI space, currently. Developed by Facebook AI Research (FAIR), Faiss is designed to handle billions of vectors in high-dimensional spaces with astonishing speed and accuracy. It is particularly useful in applications that involve large-scale machine learning computations, such as recommendation systems, image retrieval, and clustering large datasets.

Faiss can be used standalone or in conjunction with other databases as a backend for handling complex vectorized data. Its robust performance metrics make Faiss a preferred choice among AI researchers and developers working on sophisticated AI-driven applications.

Faiss GitHub repo

This overview should help you choose the most appropriate embedded database for your specific needs, enabling your applications to perform efficiently while maintaining simplicity in management and deployment.

Each database is suited to specific types of tasks and workflows, ensuring there's a tool for every requirement.

I hope this was useful, let me know if I’m missing something in the comments.

If you liked the article, you can support us in doing more stuff like this by giving us a star on GitHub! ⭐

https://github.com/latitude-dev/latitude

Latitude: The missing analytics layer between your database and your users 🤝🏽

Gerard Clos — Thu, 25 Apr 2024 12:44:35 +0000

What is Latitude?

Latitude is the open-source framework for embedded analytics.

Create API endpoints on top of your database or warehouse using just SQL, and embed interactive visualizations natively in your favorite frontend framework or through an iframe.

Introducing Latitude Deploy 🚀

Today we’re introducing Latitude Deploy so you can push your embedded analytics to production with a single command. Zero-config infrastructure that's always performant and scales automatically.

Support us ⭐

We know that building embedded analytics from scratch can be difficult. That is why we created Latitude. We're developing it as an open-source project. If you're keen to give it a try, here's a quick start guide available on our repository. If you like what you see, consider showing your support by giving us a star ⭐

https://github.com/latitude-dev/latitude

Latitude it's fast to get started, easy to maintain, and scales with your data.

Here are our main features:

SQL-to-API automatically

Write SQL queries that automatically get exposed as API endpoints so you can easily integrate them with your frontend applications.

Parameterized SQL

Manage permissions and scope variables in your code through parameters in your queries. Expose these SQL parameters as HTTP query parameters.

Built-in cache layer

Out-of-the-box caching ensures load times stay snappy. Leverage our lightning-fast query engine to deliver an amazing embedded analytics experience.

Modular queries

Compose modular SQL queries for a better analytics organization and to reuse parts of your code. Easy to maintain and scale.

Works on top of your data

Bring data from your database or warehouse through a simple configuration file. Support for all major Databases as well as local CSV files.

Native frontend components

Use our native React hook to query data from your Latitude project to your React app, and display it with our native React components, or do whatever you want with it. Support for Svelte and Vue coming soon!

Layout engine

Build standalone state-of-the-art frontend applications using our layout engine. It’s Svelte-based, supports all npm packages, and can be extended to your needs with Typescript.

Embed safely

Cache encrypted at rest, SSL-ready, encrypted SQL parameters, and parameterized queries to protect against SQL injection attacks.

We'd love to hear your thoughts and are on the lookout for contributors!

If you like it, it’d be great if you could give us a star so we can keep working on this! ⭐

https://github.com/latitude-dev/latitude

After 10 years of building B2B SaaS, going open-source has been a breath of fresh air

Gerard Clos — Thu, 18 Apr 2024 13:15:50 +0000

My name is Gerard and I’m the technical founder of Latitude.

I’ve spent the last 10 years of my career building B2B SaaS behind doors. And recently I’ve been working on rebooting Latitude as an open-source solution with my co-founder César.

What I love about building this way is that I have more in common with our dev audience, I’m getting immediate feedback, people are offering to contribute (!), and I can discuss tech details directly with customers.

It feels like you are building the product together with them, rather than for them.

Latitude is a free open-source framework for embedded analytics. Basically, you can:

Create API endpoints on top of your database/warehouse using just SQL.
Embed interactive visualizations natively in your favorite frontend framework or through an iframe.
Deploy with a single command.
Go from 0 to embedded analytics in minutes.

Today we are launching on Product Hunt!

👉 https://twitter.com/heycesr/status/1780880673319899416

If you like it, I would greatly appreciate an upvote from y'all 🙏

Also, I'd love to hear your thoughts and I’m also looking for contributors!

My approach to building an open-source data tool: Python vs JavaScript

Gerard Clos — Thu, 11 Apr 2024 10:20:12 +0000

When we rebooted Latitude I strongly pushed the team to use Python as our main language.

It was an obvious choice: It has amazing community support, all DBs have first-party clients for it, it has the largest ecosystem of data-related libs, it supports all large data formats, it even has a freaking official spec for db connectors.

I had suffered building a data product not with Python (Ruby) for the last 2 years, and didn't want to go through that again.

However, Latitude is also a full stack framework.

Just a quick background about us. Latitude is an open-source embedded analytics framework designed with developers in mind. It integrates seamlessly with your database or data warehouse, leveraging SQL and straightforward frontend components.

I’d be very happy if you could give us a star and give me feedback! https://github.com/latitude-dev/latitude

We wanted to allow developers and data engineers to not only build dashboards in the web, but to build state-of-the-art, modern applications that could easily be extended with all the tools and primitives the current web offers.

This strongly pushed us to base our approach on existing web frameworks, and since we had chosen Svelte for the templating syntax, Sveltekit became the obvious chose.

Moreover, it is not like Javascript has bad support for data-centric work; It has in fact the second largest ecosystem of data libraries after Python. It has first-party clients for all major DBs.

It has excellent support for all major large data formats. It even has good support for dataframes via the excellent Polars. Combine it with an experience to build frontend experiences that's leaps and bounds ahead of Python, and you have a strong case for using Javascript.

So, there you have it. Time will tell if we made the right choice, but so far I have little regrets on the path chosen.