Forem: Bruno Henrique Koga Fazano

MongoDB Geospatial Queries: How to query based on geographic location

Bruno Henrique Koga Fazano — Tue, 26 Mar 2024 15:44:53 +0000

Hi folks! On this article I’ll give a quick introduction and a simple example on geospatial queries with MongoDB. This is a topic I’ve learned about recently and even though I don’t use it on a daily basis I found it very interesting! Hope it helps, or at least arouse the curiosity of someone.

What are Geospatial Queries

These are a type of database query that involves the retrieval and analysis of data based on its geographical or spatial characteristics. Geospatial queries are used to search for information based on its location on earth. They are specially useful for analyzing and interacting with spatial data, which can come from many sources, such as maps, GPS coordinates or simple data entries for latitude and longitude.

For example, suppose you are using an app called Ubeer, that allows you to call for a driver to deliver beer at you door. Your house is located at latitude X and longitude Y. The app should be able to query all the drivers nearby, within a 2 kilometer radius for instance, and then notify them about your request! That’s where geospatial queries come in handy. The location of the drivers are updated real-time and we need a quick way to find out who is reachable.

How does it work?

First, data needs to be stored appropriately (for example keeping track of latitude and longitude). Then spacial indexing comes in to enable fast querying. Spacial indexing involves the creation of data structures and indexing mechanisms that organize the data in a way that accelerates query processing. The choice of database technology and indexing method depends on specific requirements of the application.

Discussing the different indexing methods is out of the scope of this article but I’ll give two quick examples to make it a little bit clearer how this might work.

R-Tree

The R stands for “rectangular”. This method is widely used and consists in organizing the objects into a hierarchical tree structure. Each node represents a bounding box that holds some data points. The root node contains the boxes of all objects and each internal node contains new bounding boxes. The leaf nodes hold the actual data points.

Geohashing

This one I find very interesting. It encodes geographical coordinates into a string of characters, dividing the world into a grid and using binary search to create a hierarchical representation of locations. Each character in the hash refines the location further. For example, we could have:

Geohash for New York: abcd
Geohash for Los Angeles: abef

The initial ab common to both codes point to the fact that they are located within the same larger area (a country for example). The different last two characters indicate different, more specific locations inside that first region.

Geospatial Queries with MongoDB and NodeJs

Now let’s get into our example. We’ll use NodeJs and MongoDB to illustrate how this kind of queries can be useful and how to implement it.

MongoDB Data Representation & Indexing

In MongoDB, geospatial data is usually represented using the GeoJSON format. It’s a standard format for encoding various types of geographic data. When you store geospatial data in a MongoDB document it is usually represented as a field containing GeoJSON objects.

Here you can have a reference on GeoJSON format and its types. We are going to use the format for the Point data type:

{ 
  "type": "point",
  "coordinates": [ 10.0, 11.2 ] 
}

To make this work with MongoDB on our NodeJs application we will use mongoose. Your schema definition (for the document you want to implement geospatial queries) should look similar to this:

import mongoose from "mongoose";

interface IVehicleLocation {
    vehicleId: string;
    location: {
        type: string; 
        coordinates: [number, number]; // [longitude, latitude]
    };
}

const vehicleLocationSchema = new mongoose.Schema({
    vehicleId: {
        type: String,
        required: true,
        unique: true
    },
    location: {
        type: {
            type: String,
            enum: ["Point"], 
            required: true
        },
        coordinates: {
            type: [Number],
            required: true
        }
    }
});

vehicleLocationSchema.index({ location: "2dsphere" });

const VehicleLocationSchema = mongoose.model<IVehicleLocation>('VehicleLocation', vehicleLocationSchema);

export { VehicleLocationSchema, IVehicleLocation }

What this code does is: define a property location (note the similarity to the GeoJSON format) that receives a string type — here only allowed as “Point” — and an array of coordinates, longitude and latitude.

Then, we define an indexing method for this schema, the 2dsphere. This method is designed to work with geospatial data represented on the surface of a sphere — such as the Earth’s — by dividing it into a grid of cells. This grid allows for efficient spatial indexing and query processing.

The 2dsphere index filters documents that do not match the geospatial query criteria by making use of this grid. This reduces the number of documents fetched in order to find the result for the query.

Creating some entries

Won’t spend some time on this since this is not a Mongoose or NodeJs tutorial. To create new entries you can create a POST /vehicle-location endpoint on your API that does something like this:

const vehicleLocation = new VehicleLocationSchema({
    vehicleId: vehicleId,
    location: {
        type: "Point",
        coordinates: [longitude, latitude] 
    }
});

await vehicleLocation.save();
return vehicleLocation.toObject();

Querying your data

Now that our schema is good to go and we have a document ready to store geographical coordinates, we can — after creating some dummy data — start querying! MongoDB provides a set of query operators, and we’ll use the $near and the $maxDistance to demonstrate how to solve the Ubeer example mentioned earlier.

So, to achieve our goal of finding all the drivers within a radius from a user’s house we can do the following:

export const getVehiclesByRadius = async (latitude: number, longitude: number, radius: number): Promise<IVehicleLocation[]> => {
    const vehicleLocations = await VehicleLocationSchema.find({
        location: {
            $near: {
                $geometry: {
                    type: "Point",
                    coordinates: [longitude, latitude]
                },
                $maxDistance: radius
            }
        }
    })

    return vehicleLocations.map(vehicle => {
        return vehicle.toObject();
    });
};

The $near operator allows MongoDB to find all the documents that are close enough to the specified point, and returns them sorted by proximity. The $maxDistance specify the radius we want to look to limit our search. In other words, this query returns all the vehicles that are within a radius from that point.

Wanna check if it’s really working?

When I was working on a project with this I wanted to make sure the results were consistent, so here is a hint. I’ve used the haversine distance to validate if the results were correct. This formula calculates the distance between two points on the surface of a sphere.

const haversineDistance = (latitude: number, longitude: number, center: { latitude: number, longitude: number }): number => {
    let earthRadius: number = 6371;

    let dLat = Math.abs(latitude - center.latitude) * Math.PI / 180.0;
    let dLon = Math.abs(longitude - center.longitude) * Math.PI / 180.0;

    // convert to radians
    let lat = (latitude) * Math.PI / 180.0;
    let clat = (center.latitude) * Math.PI / 180.0;

    // apply formulae
    let a = Math.pow(Math.sin(dLat / 2), 2) +
        Math.pow(Math.sin(dLon / 2), 2) *
        Math.cos(lat) *
        Math.cos(clat);

    let c = 2 * Math.asin(Math.sqrt(a));
    return earthRadius * c;
};

So, if you are curious about the results, you can — for each result you get — calculate the distance between the result’s coordinates and the given point. I can assure you they are all smaller than your radius :)

So there you go, now you are able to query your MongoDB documents based on their location on Earth’s surface! Hope this helped and feel free to comment or suggest anything.

How to Create Paginated, Sortable and Filterable Endpoints with NestJs

Bruno Henrique Koga Fazano — Tue, 04 Jul 2023 14:02:36 +0000

Pagination, sorting and filtering are common techniques used to deal with large sets of data returned by an API, in listing endpoints for example. They are crucial for optimizing performance, improving user experience and reducing server (and also client) load. Let’s talk briefly about this concepts before we dive into some NestJs code.

Pagination

Consists on the process of dividing a large dataset into smaller pages and serving this data to the client on an incremental, organized manner. This improves rendering speed on the client and reduces the amount of data being sent through the network.

Suppose you have a city table on you database. Brazil, for example, has more than 5.500 cities. It’s not reasonable to return all of them each time a user calls for it. Instead, we can send, say, 20 at a time, and render them accordingly (e.g. a table or a dropdown) on the frontend, which would be responsible for requesting the correct data.

And how is the frontend capable of “requesting the correct data”? It’s simple, we usually use query parameters such as page and size to specify which page and how many results to fetch.

In a paginated resource response, is common to include additional metadata along with the data fetched to provide relevant information about the pagination. Doing so allows the client to better orchestrate the requests. For example, we could return the total items count and the total pages count, then the client would know how many pages are still available to be fetched.

A Simple Example

Let’s understand how that would work with our cities table. Assume our database looks something like the following.

ID	NAME	STATE_ID
1	São Paulo	1
2	Santos	1
3	Campinas	1
4	Rio de Janeiro	2
5	Niterói	2
6	Belo Horizonte	3
7	Brasília	4
8	Curitiba	5
9	Porto Alegre	6
10	Florianópolis	7

If the getCities endpoint is correctly paginated, we could make the following requests:

https://our-api/v1/cities?page=0&size=2 : would return São Paulo and Santos
https://our-api/v1/cities?page=1&size=2 : would return Campinas and Rio de Janeiro
https://our-api/v1/cities?page=0&size=4 : would return the four of them.

And so on.

Sorting and Filtering

Sorting is about ordering the data according to a specific criteria, based on a specific field. Filtering is about selecting specific data entries depending on some other criteria, also based on specific fields.

As you may have guessed, since we are paginating our resource on the backend, here is where we also need to the the sorting and filtering. It won’t work on the client since all the data is not present there.

For illustration, assume we have a table (frontend) that renders our paginated cities. If we fetch the data with page=0 and size=2 we would receive São Paulo and Santos. If we try to sort in alphabetical order, on the client, we would get the same result (because the client only have this slice of the dataset) and that would be incorrect, the result should have been Belo Horizonte and Campinas.

In this case, when we try to sort the table on the client we should make a new request specifying that we still want the first page, with two entries, but that we need to sort them alphabetically first.

Laying the Groundwork

Let’s start getting our hands dirty. First of all, we will create some functions and custom decorators that will help us through our task.

@PaginationParams()

This first decorator (credit to my good friend Caio Argentino) is responsible for extracting and validating pagination parameters from an HTTP request query string.



import { BadRequestException, createParamDecorator, ExecutionContext } from '@nestjs/common';
import { Request } from 'express';

export interface Pagination {
    page: number;
    limit: number;
    size: number;
    offset: number;
}

export const PaginationParams = createParamDecorator((data, ctx: ExecutionContext): Pagination => {
    const req: Request = ctx.switchToHttp().getRequest();
    const page = parseInt(req.query.page as string);
    const size = parseInt(req.query.size as string);

    // check if page and size are valid
    if (isNaN(page) || page < 0 || isNaN(size) || size < 0) {
        throw new BadRequestException('Invalid pagination params');
    }
    // do not allow to fetch large slices of the dataset
    if (size > 100) {
        throw new BadRequestException('Invalid pagination params: Max size is 100');
    }

    // calculate pagination parameters
    const limit = size;
    const offset = page * limit;
    return { page, limit, size, offset };
});

After extracting the page and size from the query and validating it, the decorator returns an object that contains the information of how many items will be taken (limit) and how many will be skipped (offset).

@SortingParams(validParams)

This decorator is responsible for parsing a query parameter that comes in the format paramName:direction into an object, validating if the direction and the parameter are valid. The valid values for direction are asc and desc , and the valid parameters is an array of strings sent by the controller.



import { BadRequestException, createParamDecorator, ExecutionContext } from '@nestjs/common';
import { Request } from 'express';

export interface Sorting {
    property: string;
    direction: string;
}

export const SortingParams = createParamDecorator((validParams, ctx: ExecutionContext): Sorting => {
    const req: Request = ctx.switchToHttp().getRequest();
    const sort = req.query.sort as string;
    if (!sort) return null;

    // check if the valid params sent is an array
    if (typeof validParams != 'object') throw new BadRequestException('Invalid sort parameter');

    // check the format of the sort query param
    const sortPattern = /^([a-zA-Z0-9]+):(asc|desc)$/;
    if (!sort.match(sortPattern)) throw new BadRequestException('Invalid sort parameter');

    // extract the property name and direction and check if they are valid
    const [property, direction] = sort.split(':');
    if (!validParams.includes(property)) throw new BadRequestException(`Invalid sort property: ${property}`);

    return { property, direction };
});

@FilteringParms(validParams)

This decorator is in charge of parsing the filtering parameters (in this example we can filter only one column at a time) that comes in the format paramName:rule:value and validate them similarly to the last one.



import { BadRequestException, createParamDecorator, ExecutionContext } from '@nestjs/common';
import { Request } from 'express';

export interface Filtering {
    property: string;
    rule: string;
    value: string;
}

// valid filter rules
export enum FilterRule {
    EQUALS = 'eq',
    NOT_EQUALS = 'neq',
    GREATER_THAN = 'gt',
    GREATER_THAN_OR_EQUALS = 'gte',
    LESS_THAN = 'lt',
    LESS_THAN_OR_EQUALS = 'lte',
    LIKE = 'like',
    NOT_LIKE = 'nlike',
    IN = 'in',
    NOT_IN = 'nin',
    IS_NULL = 'isnull',
    IS_NOT_NULL = 'isnotnull',
}

export const FilteringParams = createParamDecorator((data, ctx: ExecutionContext): Filtering => {
    const req: Request = ctx.switchToHttp().getRequest();
    const filter = req.query.filter as string;
    if (!filter) return null;

    // check if the valid params sent is an array
    if (typeof data != 'object') throw new BadRequestException('Invalid filter parameter');

    // validate the format of the filter, if the rule is 'isnull' or 'isnotnull' it don't need to have a value
    if (!filter.match(/^[a-zA-Z0-9_]+:(eq|neq|gt|gte|lt|lte|like|nlike|in|nin):[a-zA-Z0-9_,]+$/) && !filter.match(/^[a-zA-Z0-9_]+:(isnull|isnotnull)$/)) {
        throw new BadRequestException('Invalid filter parameter');
    }

    // extract the parameters and validate if the rule and the property are valid
    const [property, rule, value] = filter.split(':');
    if (!data.includes(property)) throw new BadRequestException(`Invalid filter property: ${property}`);
    if (!Object.values(FilterRule).includes(rule as FilterRule)) throw new BadRequestException(`Invalid filter rule: ${rule}`);

    return { property, rule, value };
});

TypeORM Helpers

Last but not least, we will write some helper functions to generate our where object and our order object for using with TypeORM repository methods.



import { IsNull, Not, LessThan, LessThanOrEqual, MoreThan, MoreThanOrEqual, ILike, In } from "typeorm";

import { Filtering } from "src/helpers/decorators/filtering-params.decorator"
import { Sorting } from "src/helpers/decorators/sorting-params.decorator";
import { FilterRule } from "src/helpers/decorators/filtering-params.decorator";

export const getOrder = (sort: Sorting) => sort ? { [sort.property]: sort.direction } : {};

export const getWhere = (filter: Filtering) => {
    if (!filter) return {};

    if (filter.rule == FilterRule.IS_NULL) return { [filter.property]: IsNull() };
    if (filter.rule == FilterRule.IS_NOT_NULL) return { [filter.property]: Not(IsNull()) };
    if (filter.rule == FilterRule.EQUALS) return { [filter.property]: filter.value };
    if (filter.rule == FilterRule.NOT_EQUALS) return { [filter.property]: Not(filter.value) };
    if (filter.rule == FilterRule.GREATER_THAN) return { [filter.property]: MoreThan(filter.value) };
    if (filter.rule == FilterRule.GREATER_THAN_OR_EQUALS) return { [filter.property]: MoreThanOrEqual(filter.value) };
    if (filter.rule == FilterRule.LESS_THAN) return { [filter.property]: LessThan(filter.value) };
    if (filter.rule == FilterRule.LESS_THAN_OR_EQUALS) return { [filter.property]: LessThanOrEqual(filter.value) };
    if (filter.rule == FilterRule.LIKE) return { [filter.property]: ILike(`%${filter.value}%`) };
    if (filter.rule == FilterRule.NOT_LIKE) return { [filter.property]: Not(ILike(`%${filter.value}%`)) };
    if (filter.rule == FilterRule.IN) return { [filter.property]: In(filter.value.split(',')) };
    if (filter.rule == FilterRule.NOT_IN) return { [filter.property]: Not(In(filter.value.split(','))) };
}

This functions basically create the objects based on the properties returned by the decorators. For example, if the filter is city:like:Campinas we would get:



{
    city: ILike(`%Campinas%`)
}

We also will be needing a specific DTO to use when returning data from paginated resources



export type PaginatedResource<T> = {
    totalItems: number;
    items: T[];
    page: number;
    size: number;
};

Creating the Endpoint

Finally, let’s use what we’ve created in our controller and service to make it all work! We will write code for our cities example. The controller should look like this.



@Controller('cities')
export class CitiesController {
    private readonly logger = new Logger(CitiesController.name);

    constructor(
        private readonly cityService: CityService,
    ) { }

    @Get()
    @HttpCode(HttpStatus.OK)
    public async getCities(
        @PaginationParams() paginationParams: Pagination,
        @SortingParams(['name', 'id', 'stateId']) sort?: Sorting,
        @FilteringParams(['name', 'id', 'stateId']) filter?: Filtering
    ): Promise<PaginatedResource<Partial<City>>> {
        this.logger.log(`REST request to get cities: ${JSON.stringify(paginationParams)}, ${sort}, ${filter}`);
        return await this.cityService.getCities(paginationParams, sort, filter);
    }
}

Now we are able to use the decorators we created to extract the parameters our service will use to paginate, sort and filter our data. Our service will look like this.



@Injectable()
export class CityService {
    constructor(
        @InjectRepository(City)
        private readonly cityRepository: Repository<City>,
    ) { }

    public async getCities(
        { page, limit, size, offset }: Pagination,
        sort?: Sorting,
        filter?: Filtering,
    ): Promise<PaginatedResource<Partial<Language>>> {
        const where = getWhere(filter);
        const order = getOrder(sort);

        const [languages, total] = await this.cityRepository.findAndCount({
            where,
            order,
            take: limit,
            skip: offset,
        });

        return {
            totalItems: total,
            items: languages,
            page,
            size
        };
    }
}

And there we have, our own paginated, sortable and filterable endpoint! This is a simple example that I have worked out while developing some personal projects and I hope it helps someone having difficulty in this concepts / implementations. Feel free to comment and give suggestions on how to improve. Also follow me on Twitter @kogab_ :)

A Friendly Introduction to Microservices

Bruno Henrique Koga Fazano — Sun, 02 Jul 2023 23:23:56 +0000

Disclaimer

This article is a compilation of things I've learned while using microservices architecture in my current job. A lot of this content came from notes I've taken during the process of searching videos, blog posts and books. If you recognize something that you wrote or created feel free to hit me up so I can give the credits. Also, if something in this posts are inaccurate or outdated, please comment and discuss it with us! We're all here to learn and get better :)

What are Microservices?

Microservices is an architectural approach where applications are divided into a collection of small independent services that are modeled around a business domain and communicate over well-defined APIs. A service encapsulates functionality and makes it available to other via networks.

So why is it so trending? What are the benefits? To better understand this, let's talk a bit about monoliths. Monolithic applications are constituted of complex tightly coupled process that run as a unique service. If one function of this application experiences a peak of usage, we need to scale the entire system to deal with this demand. Also, if we need to fix a bug in one single functionality, it is necessary to shut down or redeploy the whole application.

On the other hand, with microservices architectures we have separate independent components that run domain specific services. Each service can be developed, deployed and scaled independently.

Taking as an example the case presented above, instead of having one single application dealing with all kinds of business logic (users, threads and posts), in a microservices architecture we'd have one single service for each of these. That separation of concerns allows for better code understanding, faster deployability and team coordination. For example, if a new post feature needs to be added, there is no need for the users' and the threads' teams to get involved, and only part of the application needs to be redeployed. All logic about this specific business domain is encapsulated.

Main Characteristics

The picture above portraits well one of the most common mistakes when we talk about microservices. Building a small monolith does not make it a microservice. The goal of this architecture is not building lots of small services, it is about splitting a complex service into multiple smaller (not necessarily small) services that follow some design principles we will discuss soon.

So what makes a service a microservice? There is no rule of thumb to define an architectural style for microservices, so we'll discuss its most common and relevant characteristics.

Highly Specialized

Each service on a microservice architecture should have only one purpose, a functionality to serve, and do it effectively.

Independent Deployability

Independent deployability is the idea that we can make changes to a microservice and deploy it to the end users without having to touch any other service. This makes it easier to update code and reduces the time to release new functionality.

Domain-Driven Design

Is a way to define the service boundaries in compass with the real-world domain the software is inserted. Each service is designed to solve a domain-specific problem and should be fine grained. If it starts to grow too much it could be necessary to break it into smaller services.

Flexible Scaling

A microservice can be independently scaled to meet demand for the functionality it supports. This provides huge cost savings and increases system availability.

Loose Coupling

Microservices are independent and loosely coupled. They often communicate via messages or HTTP requests and know little about each other. Usually, a change to one service should not require a change to others.

Technological Freedom

Because services are decoupled and deployed separately, there is no need for them all to be developed using a single technology. Microservices allows teams to choose the best stack to match the problem they're trying to solve without affecting other teams.

If an application is decomposed into multiple services, you can expect many single service changes to only require that service to be redeployed. That's not absolute, some changes will change service interfaces resulting in some coordination, but the aim of a good microservice architecture is to minimize these through cohesive service boundaries.

How to Define Good Service Boundaries

There are three concepts that are vital to address when it comes to defining microservices boundaries. We'll discuss them bellow.

Information Hiding

Information hiding is all about hiding as many details as possible behind a service boundary. To others, it may act like a black box. Some benefits of this idea are:

More work can be done in parallel since services can be developed independently and don't rely directly on information from other services.
Each service can be looked and understood by itself making it easier to manage the whole system.
Services can be modified and released independently allowing changes to be done to specific ones.

The connections between services are basically assumptions one makes about the other. By keeping this assumptions small numbered it becomes easier to change one service without impacting the other.

Cohesion

Since one of our goals when adopting microservices architecture is to increase speed when making changes in business functionality, it is important to group functionality in such a way that we can make changes in as few places as possible. In other words, we want all related logic to stay on the same place.

If related functionality is divided across multiple services that means the cohesion is weak. Our goal when defining our service boundaries is to maximize cohesion. We want each service to encapsulate all related behavior and data together.

Coupling

When services are loosely coupled, a change made to one service does not impact on another one. Hence, our goal is to minimize coupling so we can deal with services separately, enabling them to be managed by different teams.

To achieve all the benefits from adopting microservices architecture, it is important to follow strictly these three design principles.

Microservices Communication

In monolithic applications, communication is usually called intra-process because it is done internally, via method calls. For example, we could have two classes invoking each other's methods in order to execute their own functionalities. This is probably the simplest approach, but keeps functionality inside the application extremely coupled.

With microservices, since our functionality is spread across multiple distributed services, the communication happens throughout the network and is called inter-process. Let's try to understand and classify the different styles of communication.

Synchronous vs Asynchronous

In a synchronous blocking call, a microservice sends a request to another process and blocks its own execution until the request is completed and/or a response is received. This approach is very familiar to most programmers because we usually learn to program in a synchronous way, but it brings with it some issues such as the inherent temporal coupling between the services, that often results in time delays.

Synchronous calls start to become problematic when you start to have chains of blocking calls

Conclusion

Microservice architectures are definitely something to have in mind when developing complex applications and will be trending for a while. Although we've discussed a lot of great things we can achieve with it, there are some difficulties we should be aware of. Monitoring and troubleshooting is something quite hard in microservices applications, it is difficult to understand the impact of one single service going down on the rest of the system, for example. Testing is also a bit overwhelming since we sometimes need to test across multiple processes.

The best approach is to analyze the problem, context, technologies skills, understand what is your goal and decide if microservices is the best way to go. Microservices is a architecture not the one.