Forem: Arsenii Kozlov

We added Spanish language support and here's what we had to change in our architecture

Arsenii Kozlov — Wed, 17 Jan 2024 12:49:09 +0000

In spotsmap.com, from the beginning, we supported three languages, and the architecture of how we implemented multilingual support seemed quite good. However, we kept postponing support for more languages simply because even thinking about it caused us pain.

A few days ago, we took matters into our own hands and reorganized our architecture. This affected all parts of our system: the database, backend, frontend, and email communication. Here, I will describe what we have changed to make adding new languages as easy as possible. As a result, it helped us to add support for the Spanish language without any difficulties.

Also, I will share with you some practices on how to improve localization and the mistakes that you should avoid.

We moved dictionaries from the database to the code repository

Initially, we used the following database scheme for all dictionaries that require localization - in our case, these were countries, sports, student levels, and so on.

Here, the table Locale contains supported languages. The table Sport contains all types of sports that are presented on our website, such as kitesurfing, windsurfing, etc. In this table, the field masterName is the default name for each type of sport in English. We use this name when we don't have a translation for a specific language. Lastly, the table SportLocale contains localized names of types of sports for all supported languages.

How the update process looked like. To add a new value or change something in the current values, a developer has to write a migration for the database, which has to be applied to all databases, depending on the development stage. Also, such a migration requires careful review to ensure it does not break the production environment.

The migration can look like:

// Upgrade migration
INSERT INTO "Sport" ("id", "masterName")
VALUES (4, "Sailing");

INSERT INTO "SportLocale" ("localeId", "sportId", "name")
VALUES
(1, 4, "Sailing"),
(2, 4, "Парусный спорт"),
(3, 4, "Segeln"),
(4, 4, "Navegación");

// Downgrade migration
DELETE * FROM "SportLocale" WHERE "sportId" = 4;
DELETE * FROM "Sport" WHERE "id" = 4;

The more languages, the more lines of code, and unfortunately, the more mistakes. Moreover, developers have to prepare translations themselves or wait for them from their teammates, which significantly increases development time.

What we did: We removed all localization tables and dictionary tables from the database, such as Sport and SportLocale. Instead, we added a dictionary JSON file dictionaries/en/sport.json to our backend repository. In the data tables, we replaced the ID of the sport type with its corresponding key.

{
  "kitesurfing": "Kitesurfing",
  "windsurfing": "Windsurfing",
  "sup": "SUP"
}

Now, to add a new sport, a developer only has to add one line in the JSON file, and only for English. Just a one line, really. A bit later, I will explain how we translate dictionaries to all supported languages and how it helps us to easily support new languages.

We added the generation of a description based on structured data instead of using pre-rendered texts

We display a description of a sports school on its page. These pages are generated statically to make them SEO-friendly. Initially, we generated texts when we added a new school to the database. There was a separate description for every supported language, stored in the SchoolLocale table.

It was okay when we added a school. We just generated three descriptions at the same time and stored them together with the school. However, when we want to add a new language, it requires generating more than 1000 texts for every school. Moreover, we used different outside services, like Google Places API, to collect more details about the school, and these details can change from time to time. So, adding a new language requires regenerating all descriptions - more than 4000 just for 4 languages. What about adding 40 supported languages, as we aim?

This solution does not look maintainable and easy to scale.

How we changed the approach.

We removed the SchoolLocale table and saved structured JSON data in the School table, which contains all the necessary information for generating a school description, such as nearby hotels, airports, and more. Now, we generate all the pages on the frontend side during the build time (or incremental static generation time).

To achieve this, we added a new localization file public/locales/en/description.json, which looks like:

{
  "sections": {
    "hotels": {
      "description": "{SCHOOL_HOTELS} {POINT_TYPE} {HOTEL_STARS} is {HOTEL_DISTANCE} from the school."
    }
  }
}

Now, to add a new language, we just have to add a localized description file, e.g. public/locales/es/description.json. And it doesn't take any time from our developers. Absolutely, now developers don't have to do anything here to support more languages.

We rearranged our multilingual email templates

Before we used conditional statements in our email templates. It looked like this:

...
<h2>
{{#en}}Arrival date:{{/en}}
{{#de}}Ankunftsdatum:{{/de}}
{{#ru}}Дата прибытия:{{/ru}}
</h2>
<p>{{ArrivalDate}}</p>
...

However, we had over 40 localized strings in the template. Even with just three supported languages, this contains more than 120 lines of code solely for template localization. Now, consider the scenario with 10 or 40 languages.

For 40 supported languages, it would involve a staggering 1600 lines of code in the email template. This is hardly maintainable. However, our objective is to support 40 or more languages, aiming to make spotsmap.com accessible to people worldwide.

Moreover, it's hard work for a developer to add support for a new language. As a result, we decided to explore an alternative solution. To tackle these challenges, we moved localization strings from the template to our codebase and adopted a standardized translation process used for website UI localization.

The revised template structure:

...
<h2>{{ArrivalDateTitle}}</h2>
<p>{{ArrivalDate}}</p>
...

We created a JSON file templates/en/bookingConfirmation.json, which houses localization strings for English:

{
  "ArrivalDateTitle": "Arrival date"
}

And again, there is no job for developers to add support for a new language. All we need is a translated JSON file for the corresponding language.

We moved to Gitloc for translating JSON files

From the beginning, it was the responsibility of a product manager to prepare all translations before the task goes to development. Did it work? Actually, no. It would take too much time just to start development, which we couldn't afford. Developers had to manually prepare a lot of translations. Nowadays, you can use ChatGPT to translate the full JSON file. However, what about maintenance when you have to add a few keys in different localization files? Every day. It's a monotonous task that every developer hates. Moreover, it's a low-qualification job, but you pay for it as if it were a highly qualified developer job.

Now we use the only one "developer first" localization platform Gitloc. We connected our repository to Gitloc and added a configuration file gitloc.yaml. For example, there is the gitloc.yaml that is used in our backend repository. We use almost the same file in our frontend repository.

config:
  defaultLocale: en
  locales:
    - en
    - de
    - ru
  directories:
    - dictionaries

Now, to add a new language, developers just have to add one line in the gitloc.yaml file and push it to the remote Git repository. Gitloc detects that a new language has been added and translates all localization files to the corresponding language. It then commits the translations to the repository. Nothing more.

The same process applies for application maintenance. Developers add new keys in localization files, and Gitloc detects these changes and translates them.

Just a single line to support a new language. It's unbelievable.

More localization practices that we learned

Use patterns for localization strings

For example, use School works from {FROM} until {UNTIL} instead of concatenation. It is easier to use and provides more context to translators.

Store localized dictionaries with localization

Instead of storing dictionaries that require translation, such as months, in a configuration file, we have moved them all to the public/locales folder. This allows us to use the same localization methods for UI localization.

Store only localization strings in localization files

Do not store business logic that depends on language, such as date and currency formats, in localization files. It is better to use configuration files or internationalization libraries that can format dates and currencies according to the locale.

Avoid using arrays in localization files

Instead of using arrays, it is better to use key-value pairs. Many i18n libraries recommend this approach. Additionally, using arrays can result in more translations if something changes. For example, adding a new item in the middle of an array would require translating half of the array. However, adding a new key to an existing object only requires translating one new value.

Avoid using leading and trailing spaces in localization strings

Instead of using a localization string like "and": " and ", it is better to use "and": "and" and add necessary spaces in your code. You can even use "description": "{SOMETHING1} and {SOMETHING2}" if possible. This helps avoid translation mistakes, whether you use automated or manual translations.

What was the reason for adding support for Spanish?

We realized that most of our users are located in countries that speak the languages we support. Previously, we supported three languages: German, English, and Russian. The majority of our users were from Germany, Switzerland, Austria, and Russia. However, we also had a small percentage of users from Spain. We are curious to see how adding Spanish will affect the geographic distribution of our users.

I believe that in a couple of months, we will collect enough statistics to compare whether there is any improvement or not. I will definitely share these results with you. Just subscribe to me so you don't miss the outcome of this experiment.

I hope you found something helpful!

Sending multilingual emails with Postmark and Gitloc

Arsenii Kozlov — Fri, 08 Dec 2023 15:29:35 +0000

In today's globalized world, effective communication with customers in their preferred language is key for businesses. Whether it's via emails, messengers, or websites, using a client's native language fosters a stronger connection and increases the likelihood of successful transactions. This holds true even for transactional or notification emails, as information can be easily overlooked when presented in a non-native language.

In this post, I'll share our experience in implementing multilingual transactional emails for spotsmap.com and the evolution of our approach.

Note: While I'll use Node.js for illustration, the concepts discussed are applicable to various development languages.

Working on spotsmap.com, we faced the challenge of creating a localized booking confirmation email. Packed with details such as sport types, dates, costs, and course information, the HTML template alone exceeded 500 lines of code.

First Solution: Language-Specific Templates

Initially, we opted for different templates for each supported language.

Our source code included a configuration object allowing us to select a template based on the user's language. The template for English looked like this, and almost identical templates existed for every language:

...
<h2>Arrival date:<h2>
<p>{{ArrivalDate}}</p>
...

And we used it like this:

// config.js
export default {
  services: {
    email: {
      postmark: {
        apiKey: process.env.POSTMARK_API_KEY,
        templates: {
          bookingConfirmation: {
            en: 123456,
            de: 123457,
            es: 123458
          }
        }
      }
    }
  }
}

// services/postmark.js
export const sendMail = ({ from, to, template, locale, params }) => {
  const request = client.sendEmailWithTemplate({
    From: from,
    To: to,
    MessageStream: 'outbound',
    TemplateId: postmarkConfig.templates[template][locale],
    TemplateModel: params || {},
  })
  request
    .then(result => {
      console.log(`Email sent to ${to}`)
    })
    .catch(err => {
      console.log(`Email was not sent, with error code ${err.statusCode}, ${err.message}`)
    })
}

It looks easy; however, we had to change the approach.

Why? We initially had three large and very similar templates. Adding a new language was relatively simple – just copy the template and adjust it for the new language. However, over time, we found the need to restructure the template. We introduced the capability to book multiple courses in a single order, and replicating this change across all templates became laborious, even with just three languages.

So, we decided to move to another solution.

Second Solution: Conditional Templates

Our second attempt involved consolidating templates with conditionals.

So we rearranged the template and made only one template for all supported languages. It looked like this:

...
<h2>
{{#en}}Arrival date:{{/en}}
{{#de}}Ankunftsdatum:{{/de}}
{{#es}}Fecha de llegada:{{/es}}
</h2>
<p>{{ArrivalDate}}</p>
...

And we made a few changes in our code:

// config.js
export default {
  services: {
    email: {
      postmark: {
        apiKey: process.env.POSTMARK_API_KEY,
        templates: {
          bookingConfirmation: 123456
        }
      }
    }
  }
}

// services/postmark.js
export const sendMail = ({ from, to, template, locale, params }) => {
  const request = client.sendEmailWithTemplate({
    From: from,
    To: to,
    MessageStream: 'outbound',
    TemplateId: postmarkConfig.templates[template],
    TemplateModel: { ...params, [locale]: true },
  })
  request
    .then(result => {
      console.log(`Email sent to ${to}`)
    })
    .catch(err => {
      console.log(`Email was not sent, with error code ${err.statusCode}, ${err.message}`)
    })
}

Okay, it looks even better.

However, we had over 40 localized strings in the template. Even with just three supported languages, this translates to writing 120 lines of code solely for template localization. Now, consider the scenario with 10 or 40 languages.

For 40 supported languages, it would entail a staggering 1600 lines of code in the email template. This is hardly maintainable. However, our objective is to support 40 or more languages, aiming to make spotsmap.com accessible to people worldwide.

As a result, we decided to explore another solution.

Third (and Current) Solution: Code-Based Localization

To address these challenges, we transitioned to a more efficient approach. We moved localization strings from the template to our codebase and adopted a standardized translation process used for website UI localization.

The revised template structure:

...
<h2>{{ArrivalDateTitle}}</h2>
<p>{{ArrivalDate}}</p>
...

We created a JSON file, e.g., templates/en/bookingConfirmation.json, housing localization strings for English:

{
  "ArrivalDateTitle": "Arrival date",
  ...
}

We modified the sendMail function accordingly:

// services/postmark.js
export const sendMail = ({ from, to, template, locale, params }) => {
  const strings = require(`../templates/${locale}/${template}.json`
  const request = client.sendEmailWithTemplate({
    From: from,
    To: to,
    MessageStream: 'outbound',
    TemplateId: postmarkConfig.templates[template],
    TemplateModel: { ...params, ...strings },
  })
  request
    .then(result => {
      console.log(`Email sent to ${to}`)
    })
    .catch(err => {
      console.log(`Email was not sent, with error code ${err.statusCode}, ${err.message}`)
    })
}

Additionally, we created a gitloc.yaml file to automate the translation process:

config:
  defaultLocale: en
  locales:
   - en
   - de
   - es
  directories:
   - templates

This approach allowed us to easily generate translations for all supported languages by pushing changes to the remote repository and pulling translations locally. Notably, Gitloc facilitated seamless support for new languages by updating the gitloc.yaml file.

Now, we can effortlessly support numerous languages while maintaining a consistent and manageable codebase.

Explore further:

Postmark documentation: https://postmarkapp.com/developer
Gitloc documentation: https://docs.gitloc.org/

I hope you find these insights valuable!

How to choose an i18n library for a Next.js 14 application - Part 1

Arsenii Kozlov — Wed, 29 Nov 2023 09:58:15 +0000

What is the best choice, and how does one go about selecting an i18n library for production? Let's address this decision-making process.

In my previous post, I selected the Rosetta internationalization library to demonstrate how a multilingual Next.js application can be easily implemented. However, as correctly pointed out in the comments, it's not the optimal choice. I concur; Rosetta was last updated three years ago, making it a red flag for me not to use it in a production environment.

The article will consist of three parts:

In the first part, I will discuss the criteria for selecting a library, form a list of libraries, compare them based on the criteria, and choose the three most suitable libraries for further analysis.
In the second part, I will compare the functional capabilities of the libraries, identify commonalities and differences, and select two of them for testing.
In the third part, I will test the chosen libraries and draw a conclusion about which one I would use for a productive environment.

Firstly, it is necessary to determine the criteria for a library suitable for production. Based on my own experience, I will highlight the following criteria:

Popularity and Rating - this is more about ensuring that many developers are familiar with the library, and there is a sufficient community to address emerging difficulties and problems.
Relevance - it is a red flag for me if the library has not been updated for years. This usually means that the library is not evolving and may not support the latest changes in Next.js. Even if everything currently works correctly, there is no guarantee that it won't break with the next Next.js update.
Presence of Quality Documentation and Examples - this is an important criterion as it allows for faster implementation of the library and significantly simplifies maintenance in the future.
Security - the current version of the library should not have known security issues.
Readable Source Code of the Library - the assessment of code cleanliness and readability will always be subjective. In most cases, this is not required, but it can be useful, especially when choosing a less popular and unsupported library at the moment. In this article, we will not consider such libraries, so we will not assess them based on this criterion.
Performance - it consists of two parts: the library's size and loading time, and the actual code execution speed in the library. We will try to evaluate this during the testing phase.
Functional Capabilities - the library should meet the functional requirements of the system to avoid searching for workarounds or having to change the library in the middle of development.
Adherence to Standards - I will strive to choose libraries for a productive environment that adhere to accepted standards. If necessary, I want to be able to replace one library with another with minimal effort.

In the first part of the article, we will select libraries and evaluate them based on the first four criteria. Using these criteria, we will choose three libraries for further analysis. In the second part, we will assess the chosen libraries based on criteria 7 and 8.

To begin, let's compile a list of available libraries and look on their popularity:

Library	Weekly downloads	Last update	Rating
react-i18next	2.677.938	11 days ago	1
react-intl	1.311.586	13 days ago	2
next-i18next	350.234	a month ago	3
next-intl	118.061	3 days ago	4
@lingui/react	97.660	2 months ago	5
next-translate	73.427	23 days ago	6
rosetta	21.258	3 years ago	7
next-localization	14.404	2 years ago	8

Disclaimer: This numbers are current as of November 27, 2023. Web development evolves rapidly, and within a few months, the landscape may have changed.

Next, we will not consider rosetta and next-localization as they have not been updated for more than two years, and as a result, they have the lowest number of downloads, which is constantly decreasing.

We will check the packages for known vulnerabilities using the Snyk CLI.

~$ snyk test next-intl@latest

Testing next-intl@latest...

Organization:      ***
Package manager:   npm
Open source:       yes
Project path:      next-intl@latest

✔ Tested next-intl@latest for known vulnerabilities, no vulnerable paths found.

After checking all packages, we find that all specified packages do not have any known vulnerabilities - excellent!

Now, let's take a look at the documentation.

Library	Comment	Rating
next-intl	Well-structured and detailed documentation with examples using App Router and Pages Router	1
@lingui/react	Well-structured and detailed documentation, but examples are only described for React applications, requiring adaptation for Next.js.	2
react-i18next	Detailed documentation, but in my opinion, poorly structured and overloaded with irrelevant information. Integration example for Next.js leads to a page outside the library documentation.	3
react-intl	Detailed documentation on the library and its API, lacking examples of using the library in an application. The documentation is oriented towards React, and adapting the library for use in a Next.js application is required.	3
next-translate	Detailed single-page documentation with a table of contents, improving navigation. Documentation refers to Next.js version 10, which seems outdated. All examples use Pages Router.	4
next-i18next	Single-page documentation, usage described only with Pages Router, recommends using react-i18next directly for Next.js 13/14	-

Since we will be focusing on developing an application with Next.js 14, I did not evaluate next-i18next, and we will not consider this library further, as they recommend using react-i18next instead. We will also not consider the library next-translate as, in my opinion, its documentation does not appear to be up-to-date and convenient enough for easy and quick integration into our future system.

My personal choice unequivocally falls on next-intl as the most qualitatively documented and react-i18next as the most popular library.

Choosing a third option for further analysis seems more challenging, with two libraries competing: @lingui/react and react-intl. Both will require additional exploration to integrate into a Next.js 14 application. One is very popular, with over a million downloads per week, while the other has a more structured documentation.

Both of these libraries use approaches that are different from next-intl and react-i18next. @lingui/react uses an additional CLI for extracting translatable strings from page code, which, in my opinion, complicates the development process and makes it harder to abstract translations from the code. react-intl uses special javascript objects describing translatable messages.

For further exploration, let's consider all four libraries: next-intl, react-i18next, @lingui/react, and react-intl. We will try to compare their different approaches and the functional capabilities they provide.

If you use other libraries and are confident that they can compete with the ones selected, please let me know in the comments, and I will be sure to investigate them.

Subscribe for notifications not to miss the next parts.

While you're waiting for the next round.

We need your feedback about our recently launched Gitloc, a localization platform for modern web applications — an ideal choice for indie developers and small teams.

We invite you to test our system, which will only take 10-15 minutes. Follow the instructions in the previous article or check out documentation.

It is crucial for us to receive feedback from developers at this stage. You can leave your feedback in the comments to the latest release in our Telegram channel or write to feedback@gitloc.org.

We will definitely support all our early users. Register through the link for free, and let's make localization no longer a headache for developers.

Forget about localization for your web applications

Arsenii Kozlov — Mon, 06 Nov 2023 08:49:31 +0000

No, no, doing localized web applications that are accessible for people around the world is still relevant. But you should forget about doing localization if you are a developer.

As a developer, you know that integrating localization into your application is effortful, no matter whether you use CLI-based localization platforms or just store localization in your repository. You should always remember them, if they are current, if their version is suitable for the application version, if they are processed, and so on.

Let me show you how you can forget about all these issues and focus on development. Here, I will describe a solution based on Next.js. However, it can be easily applied to many other modern frameworks and approaches like React, Vue, Flask, and even backends such as Node.js, Python, and others, and also for using any internationalization libraries.

We won't reinvent the wheel, so let’s take a localized web application sample prepared by the Next.js team: https://github.com/vercel/next.js/tree/canary/examples/with-i18n-rosetta. Just run the following command.

npx create-next-app --example with-i18n-rosetta with-i18n-rosetta-app

Let’s see what we have as a result.

It’s a simple Next.js application configured with the https://github.com/lukeed/rosetta internationalization library. All localization values are stored in JSON files under the /locales folder, with each language in a separate file. Localization is configured in the lib/i18n.js file, where you can find English (en) as the default language and German (de) as an additional language. That’s great!

When we start the application (with the command npm run dev), we can see the next page.

Messages for the English version can be found in locales/en.json file, and for the German version in locales/de.json file.

That works! Great.

Let’s add this repository to GitHub; it is necessary for a few next steps. I have created a new public repository on my GitHub: https://github.com/arseniy-kozlov/with-i18n-rosetta.git, and ran the following commands.

git init
git remote add origin https://github.com/arseniy-kozlov/with-i18n-rosetta.git
git add .
git commit -m “Initial commit”
git push origin master

We are close to our goal – to forget about localization – only a few simple steps left. Let’s create an account on Gitloc.

Create a new team.

And connect our repository. Click Add repository and choose GitHub there.

Then authorize the gitloc-org application on the GitHub consent page.

And confirm on the next step.

Choose the with-i18n-rosetta repository.

And confirm on the next page.

Let’s return to our repository and add a simple gitloc.yaml file to the root of the repository.

config:
  defaultLocale: en
  locales:
    - en
    - de
  directories:
    - locales

That’s it! Right now, forget about doing localizations and focus on development. Let’s do it together.

Let’s update our messages in locales/en.json file.

I have removed the text key, modified the description value, and added a new title key. Then let’s make corresponding changes in the pages/[lng]/index.js file.

Then let’s commit changes and push them to the remote repository.

git add .
git commit -m ‘gitloc.org test’
git push origin master

After that, Gitloc detects our changes, automatically translates them, and commits them to the repository. It takes only a few seconds, and you can pull translations to your local repository.

Let’s run our application and check the changes.

npm run dev

Good job!

Learn more about Gitloc - https://gitloc.org/

We will be very grateful for all your feedback. I hope you enjoy!