Forem: Lê Công Tuấn

MongoError: E11000 duplicate key error collection ??

Lê Công Tuấn — Fri, 30 Jul 2021 04:31:45 +0000

TL;DR

I've encountered the issue sometime in my developer path. So I decided to write down my experience and to firstly note to myself and secondly to help other developers/engineers who are a newbie in this topic.

If you only need the fix, you can skip to the last part of this article!

One of the most advantages of using NoSQL databases is the flexible feature that allows us to update the number of
fields and their data types any time we want.

I'm using Nodejs and Mongoose driver to connect to MongoDB. And in the very beginning phase of a project development
life and this help us to able to update collections and their fields.

However, I encountered an error.

MongoError: E11000 duplicate key error collection: companies index: code_1 dup key: { code: null }

What happened here?

To understand, I would go further a bit with Indexing and unique keys in Databases.

Database Indexes

Wikipedia defines:

A database index is a data structure that improves the speed of data retrieval operations on a database table at the cost of additional writes and storage space to maintain the index data structure. Indexes are used to quickly locate data without
having to search every row in a database table every time a database table is accessed. Indexes can be created using one or more
columns of a database table, providing the basis for both rapid random lookups and efficient access of ordered records.

If you want to query a field or set of fields without iterating all the entries in a table/collection, you can create indexes for these fields.

To understand more how indexes are created & organized to support efficient queries, you can check atthis

However, the cost for indexes is not cheap, especially in the world of big data nowadays. Because to make queries on indexed fields efficient, these indexes need to be stored in a fast query memory (for example RAM). So, be careful when you want to add an index for a field, some factors should be put on the table to have good enough decisions: are data queried frequently? the user behaviors? regions that data are stored? etc

And last but not least, an indexed field may be a non-unique value for each entry in the column. For example, you can have an index on the field "Region" where we can have multiple users in the same region.

Unique keys

In the real-life, there are use cases that we want to limit the appearance of one or a set of values of factors. For example, you want there is only 1 email that is used to register per user, no more.

So, unique keys help you to achieve this constrain by defining the rule in the schema. For example in mongoose:

const userSchema = new Schema(
  {
    email: {
      type: String,
      index: true,
      unique: true,
    },
    name: String,
  }
);

You can see that the email attribute is unique. With this, you cannot add more than one user with the same email. If developers violate the rule, Mongodb will throw errors. This helps us preventing developer mistakes.

Again, you can check more at wikipedia

When Mongo DB schema become out of date with Mongoose schema

Come back to the beginning error, let's me show you the mongoose schema that was defined by code:

const companySchema = new Schema(
  {
    location: String;
    name: String,
  }
);

Look good! right?

But what makes the error?

MongoError: E11000 duplicate key error collection: companies index: code_1 dup key: { code: null }

Let's check a bit:

I used mongo client by querying with the command line to check the existing keys for the companies collection. And the result was:

> db.companies.getIndexes()
[
        {
                "v" : 2,
                "key" : {
                        "_id" : 1
                },
                "name" : "_id_",
                "ns" : "project-name.companies"
        },
        {
                "v" : 2,
                "unique" : true,
                "key" : {
                        "code" : 1
                },
                "name" : "code_1",
                "ns" : "project-name.companies",
                "background" : true
        }
]

Yeah! There is an established index field for code and it is set to unique. And once an index is set, it is there until you remove it, and the rule unique is still there also.

And the reason is the schema was modified due to the product business has changed. The previous schema was:

const companySchema = new Schema(
  {
    code: {
      type: String,
      index: true,
      unique: true,
    },
    location: String;
    name: String,
  }
);

So, this is a case when the Mongo schema becomes out date with the Mongoose schema that you defined in code.

To fix this, I need to remove manually the unnecessary index key. Mongo query provides some methods to remove indexes manually:

Note: You cannot drop the default index on the _id field.

db.collection.dropIndex()

> db.companies.dropIndex("code_1")

db.collection.dropIndexes

This method will drop all non-_id indexes

> db.companies.dropIndexes()

You can verify again by using the commandline:

db.companies.getIndexes()

And it works!

Your comments & discussion are warmly welcomed!

Enjoy your job!

GraphQL + Clean Architecture boilerplate

Lê Công Tuấn — Sat, 03 Jul 2021 14:27:10 +0000

Inspiration

The Clean Architecture is an awesome guide for developers who desire to build a clean, structured and maintainable projects. The letter L in SOLID principal allows us to make depended components are easily replaced without touch to the business and the core of systems. For example of the web server where there are many frameworks out there (Expressjs, GraphQL, etc) and the chosen decision depends on the purpose of the business and the context.

In this boilerplate guide scope we will go through how to make the core business immune with the change of the detailed technologies. Let's go!

You can find the implementation detail in this repository

Boilerplate

Code structure



.
├── package.json
├── package-lock.json
├── README.md
├── src
│   ├── application
│   │   ├── auth
│   │   │   └── index.ts
│   │   ├── graphql
│   │   │   ├── index.ts
│   │   │   ├── schemaShards
│   │   │   │   ├── index.ts
│   │   │   │   └── users.ts
│   │   │   ├── subscriptionManager.ts
│   │   │   └── utils
│   │   │       └── mergeRawSchemas.ts
│   │   └── index.ts
│   ├── dto
│   │   └── user.dto.ts
│   ├── entities
│   │   └── user.entity.ts
│   ├── index.ts
│   ├── IoC
│   │   ├── container.ts
│   │   ├── icradle.interface.ts
│   │   └── providers
│   │       ├── auth.provider.ts
│   │       └── user.provider.ts
│   ├── persistence
│   │   ├── auth.repository.ts
│   │   ├── constants.ts
│   │   ├── password.ts
│   │   └── user.repository.ts
│   ├── services
│   │   ├── auth.repository.interface.ts
│   │   ├── auth.service.ts
│   │   ├── user.repository.interface.ts
│   │   └── user.service.ts
│   └── __typedefs
│       ├── graphqlTypes.d.ts
│       └── schema.graphql
└── tsconfig.json

The main folder is src where contains the implementation:

application: Application layer where we can use a backend web application framework. In this boilerplate is GraphQL
service: The business service layer where we defined the logic to handle application use cases. This layers also contains adapters that abstract the detailed actions, for example, access to database.
entities: This is the core component of the application where we define the application entities.
persistence: This is another detailed layer where we specify actions to communicate to database, message queue, etc.
dto: (stand for Data Transfer Object) defines communication contracts between layers. For example, what type of input param or return value that is used in methods of layers
IoC: (stand for Inversion of Control) is an implementation of the Dependency Injection, the letter D in the SOLID principal. Every time you create a new Service class, or a new repository class, or a new controller, you need to register them in this container. This boilerplate uses Awilix library to achieve the factor.

Communication Policies

The communication policies between layers and domains compliant with the Clean Architecture. The directions of arrows in the following diagrams show the relation between components. For example, if an arrow direction from component X to component Y, it means component X depends on the component Y.

1/ Dependency diagram

As you can see in the diagram the dependency flow among layers:

Entity is the core layer and does not depend on any component
To inverse the dependency from the domain business layer to the repository layer, there is a repository interface in the domain business. This allows us to be able to implement and update this layer with the detailed technologies
Application Controller depends in the services
IoC container depends on services, repositories and application controllers because we need to register all of them to the container
Application router depends on the IoC container that allows us to use registered application controllers to handle client requests

2/ Dependency between domains

In real-life projects, of course, there is not merely 1 domain. So, we need to define the communication rules between layers of them.

Getting Started

There are some beginning available command lines is defined the scripts part of the package.json file.

1/ Install dependencies



npm i

2/ Run the application under the dev mode with the hot reload feature



npm run dev

3/ Compile the application



npm run build

4/ Run the application under the production mode. You need to compile the application before



npm run start

5/ Generate types for GraphQL schema. This action must be done each time you update the GraphQL schema to allow typescript compiler understand your schema:



npm run generate-typedefs