Forem: Joao Marques

Backend Authority

Joao Marques — Thu, 06 Nov 2025 19:25:08 +0000

In modern business software, backend authority is more than just a technical concept, it’s the backbone of trust and consistency. When we talk about “backend authority,” we’re referring to the backend system being the single source of truth that governs data integrity, rules, and business logic. Without it, multiple systems can start interpreting or mutating data differently, leading to mismatches, errors, and costly confusion.

A strong backend authority ensures that every transaction, update, and rule enforcement flows from one verified source. This allows frontends and external services to focus on user experience, while the backend maintains governance on validating inputs, applying business rules, and auditing activity. For business applications, especially those involving financial transactions or compliance, backend authority prevents data drift and keeps the system accountable.

Spring Boot Logging convention - JMLogFlow

Joao Marques — Wed, 11 Jun 2025 14:11:12 +0000

JMLogFlow is a lightweight, pragmatic logging convention designed to improve maintainability of Spring Boot applications.
It’s centered around placing meaningful log messages in strategic locations throughout the application to trace execution and data flow, without overwhelming the log files.

1 - Controller Logging

Log once at the start of every controller method.

What to log:

Http method
Endpoint path
Path variables
Query parameters
Important Custom headers (Except the ones with sensitive information)

❌ Do not log the entire request body or object, and do not log any authorization information.

log.info("PUT /users/{userId} called with userId={} and userRequestDTO={} with X-ServiceNumber={} with Authorization={}", userId, userRequestDTO, serviceNumber, token);

✅ Log only the necessary

log.info("POST /users/{} called with X-ServiceNumber={}, userId, serviceNumber);

2 - Service Layer Logging

Log at the start and end of every public service method.

log.info("Starting UserService.getUserById with userId={}", userId);
// ... business logic ...
log.info("Finished UserService.getUserById for userId={}", userId);

This creates a clear log bracket that makes tracing the flow between layers easy.

3 - Repository Layer Logging

When data is retrieved from the database:

Log the number of rows returned.
If it’s a count query, log the result count.
If the query is suposed to return one item, like find by id, log only 1 important information.

log.info("Fetched {} users from the database for active={}", users.size(), active);

log.info("Total number of active users in the system: {}", userCount);

log.info("Retrieved user with id: {} from database", userId);

4 - External Service Logging

When receiving data from an external system, from a rest api or soap, or even a 3rd party sdk.

If it returns a list, log how many rows were returned.
If it returns a single object, log only key information (e.g., an ID or status, etc)

log.info("External billing service returned {} invoices for accountId={}", invoices.size(), accountId);

log.info("Payment gateway responded with transactionId={} and status={}", transaction.getId(), transaction.getStatus());

5 - Business Decision Logging

Every business decision branch should be logged.

What is a business decision branch?

A business decision branch is any point in the logic where the behavior of the application diverges based on rules, conditions, or domain-specific decisions. These aren’t just technical if/else branches, they are meaningful choices based on business rules.

Examples:

If a user is eligible for a discount:

log.info("User {} is eligible for a {}% discount based on subscription plan", userId, discountAmount);

If the system decides to skip billing:

log.info("Billing skipped for account {} because the billing period is not yet complete", accountId);

Logging these moments creates an audit trail of why the application behaved in a certain way.

6 - General Rules

Every log should explain in plain English what’s happening.
Do not log sensitive information (passwords, tokens, etc.).
JMLogFlow is strict to log.info, you can use your team convention for log.debug, log.warn and log.error

Example Summary

// 2 - Service Layer Logging
log.info("Starting createInvoice for accountId={}", accountId);

// 3 - Repository Layer Logging: only 1 important information
Account account = accountRepository.findById(accountId) .orElseThrow(() -> new NotFoundException("Account not found with id " + accountId)); log.info("Account {} retrieved from database", accountId);

// 5 - Business Decision Logging: check if account is in trial
if (account.isTrial()) { log.info("Account {} is in trial mode, skipping invoice generation", accountId); return; }

// 3 - Repository Layer Logging: Fetch billable items
List<Item> billableItems = repository.findBillableItems(accountId); log.info("Found {} billable items for accountId={}", billableItems.size(), accountId);

// 4 - External Service Logging: Send to external billing service
ExternalInvoiceResponseDTO invoice = invoiceService.sendToBillingSystem(billableItems); log.info("Invoice created with id={} and status={}", invoice.getId(), invoice.getStatus());

// 2 - Service Layer Logging: End of service
log.info("Finished createInvoice for accountId={}", accountId);

By placing logs at meaningful points of the flow and writing them in clear, understandable English, developers can quickly find issues issues, understand behavior, and maintain confidence in prod environment.

Internal Browsers Are Slowly Driving Us Insane

Joao Marques — Tue, 04 Feb 2025 13:22:07 +0000

Why Do Apps Have to Be So Annoying with Their Internal Browsers?

Okay, let’s talk about something that’s been driving me absolutely nuts lately. Every time I click a link in an app. Whether it’s LinkedIn, GitHub, Slack, or whatever... it doesn’t open in my nice, familiar browser. Nope, it’s always some weird internal browser (probably a WebView or whatever) inside the app. And let me tell you: it sucks...

Why Internal Browsers Are Just the Worst

These internal browsers are bad, and here’s why:

I’m Already Logged In, But It Doesn’t Care
Oh, you want to check out a private repo on GitHub? Too bad! Log in again brother, because this internal browser has no idea you’re already signed in on Chrome. Same thing for Google Docs, or any other link where you need to be logged in. It’s like the app is trolling you.

Features? What Features?
My main browser has all the cool extensions, bookmarks, tabs that actually work well. But these internal browsers? They’re stripped down to the bare minimum.

Let Me Choose, Please!
If I’m a Chrome person (or Safari, Firefox, whatever), I want to use that browser. Not some half-baked in-app thing that’s there just to trap me.

Why Do Apps Even Do This?

Honestly, it’s probably about two things:

1 - Keeping You Stuck in the App
They don’t want you wandering off. If you open a link in Chrome, you might forget to come back. So, boom!! internal browser. That way, you’re still technically in their app.

2 - They Wanna Track You
Internal browsers let apps track what you’re doing, gather data, and maybe even throw some extra ads in your face. Unacceptable!!

How About Some Options?

Here’s an idea: just let us pick how links open. It’s not that hard. A few suggestions:

Let Me Use My Default Browser: Just add a setting to open links in my actual browser.
Big “Open in Browser” Button: If you’re gonna force me into your WebView, at least give me an easy way to jump to my browser.
Sync It Up: If you’re so desperate to keep me in your app, at least sync cookies and logins so I don’t have to re-enter passwords.

Conclusion

At the end of the day, it’s all about user experience. Nobody likes being forced into a clunky, limited web view when they could just use their preferred browser. It’s frustrating, unnecessary, and makes simple tasks way harder than they need to be. Movile dev colleagues, please give us the control we deserve. Until then, I’ll keep long-pressing every link to force it open in my actual browser because I refuse to let these apps win.

Why Do We Still Need Jackson or Gson in Java?

Joao Marques — Wed, 01 Jan 2025 19:49:39 +0000

Alright, let’s talk about something that’s been bugging me for ages: JSON handling in Java. Why do we still need libraries like Jackson or Gson for something so basic? Seriously, it’s 2025... shouldn’t parsing and serializing JSON be a built-in feature by now?

The Eternal Setup 🛠️

Every time we start a new Java project, it’s the same story. Add Jackson or Gson to the dependencies, configure the object mapper, and write boilerplate code for something that feels like it should just work out of the box. Other languages have this figured out.

Why Is This Still a Thing? 🤔

JSON is the language of the web. APIs are everywhere, and nearly every Java project needs to read or parse it. So why hasn’t JSON handling become a standard part of the Java SDK? I mean, we got things like java.time for dates (finally), so why can’t we have java.json or something?

The Dream 🌟

import java.json.*;  

// Parse JSON to object  
MyObject obj = Json.parse(jsonString, MyObject.class);  

// Serialize object to JSON
String json = Json.toString(obj);

My opinion about this 💬
Java’s all about being enterprise-ready, right? Well, enterprises use JSON all the time. It’s high time the language embraced that and gave us native JSON support. Until then, I guess we’ll keep adding Jackson and Gson to our pom.xml and pretending it’s fine.

5 Best Practices for Securing Your AWS Environment with IAM

Joao Marques — Sun, 22 Dec 2024 23:15:07 +0000

Managing access and permissions in AWS is critical for maintaining the security and integrity of your cloud environment. IAM is a strong framework designed by aws, and you are responsible for making good use of it and keep your project safe from hackers.

1. Create IAM groups to separate access

image_1

IAM groups are collections of IAM users. Instead of assigning policies directly to each user, you can attach them to a group. Users in the group inherit the group's permissions, making it easier to manage access at scale.

In this image, you have the flexibility to assign custom privileges specifically to Nick. Additionally, you can grant extra custom privileges to any other user within any group as needed.

2. Principle of least privilege

Grant users, roles, and services only the permissions they need to perform their tasks. Avoid using overly permissive policies like Administrator Access or * permissions.

Instead of this policy:

{
  "Effect": "Allow",
  "Action": "*",
  "Resource": "*"
}

Use short permissions for specific actions and resources:

{
  "Effect": "Allow",
  "Action": ["s3:GetObject"],
  "Resource": ["arn:aws:s3:::example-bucket/*"]
}

3. Use multi-factor authentication (MFA)

Enable MFA for all users, especially for high privileged accounts like root user or admin users who has AdministradorAccess policy. MFA provides an additional layer of security by requiring a second factor, you can use your mobile or an external device.

image_2

There are some Apps that you can download in app Store like google authenticator (my preferred), authy.

4. Avoid using root account for daily operations

The root account has unrestricted access to all resources in your AWS environment. Use it only for initial account setup and specific administrative tasks. Create individual IAM users or roles for daily operations.

5. Configure the password policy for your users

Access keys and passwords should be rotated periodically to reduce the risk of unauthorized access.

You can configure this in:

IAM > Account Settings > Edit password policy

image_3

Align with industry standards or your company policies.

My opinion about this:

Aws is responsible for everything that they do, for example their infrastructure, their network security and the vulnerability analysis of the services they offer.

But regarding IAM, you are responsible for creating your own Users, groups, roles, policies, monitoring, for enabling MFA to the accounts, rotating your keys often, analysing the access patterns and review given permissions to groups/users/roles.

Never forget these concepts because it can lead to security breaches for your organization, I hope I helped you somehow!

Error when retrieving token from sso Token has expired and refresh failed

Joao Marques — Wed, 13 Nov 2024 12:56:14 +0000

Working with AWS services often involves authentication using AWS Single Sign-On (SSO). While AWS CLI simplifies this process, you may occasionally encounter the following error:

Error when retrieving token from sso: Token has expired and refresh failed

This error indicates that your SSO session token has expired, and AWS CLI couldn't refresh it automatically. Let's explore why this happens and how you can resolve it.

Why Does This Error Occur?

AWS SSO tokens are temporary credentials that expire after a set period, usually defined by your organization's policies. When the token expires, AWS CLI attempts to refresh it. However, if the refresh attempt fails—perhaps due to a network issue, an expired session, or corrupted cached credentials—you'll see this error.

How to Fix It

1 - Re-authenticate with AWS SSO.

The quickest way to resolve this error is to re-authenticate your AWS SSO session. Use the following command:

aws sso login

This command will open your default browser, prompting you to log in and refresh your session token.

2 - Retry Your Original Command

After re-authenticating, retry the command that triggered the error.

3 - Clear Cached Tokens

If re-authentication doesn’t work, your cached SSO tokens might be corrupted. Clear them manually:

delete the folder: ~/.aws/sso/cache/

or execute this if you are lazy:

rm -rf ~/.aws/sso/cache/

4 - Check AWS CLI Version

Older versions of AWS CLI might have issues with SSO token management. Check your AWS CLI version with this command:

aws --version

and analyse if you can update it for the latest.

My view

Running into the "Token has expired and refresh failed" error can be frustrating, especially when you're in the middle of an important task. Luckily, the fix is usually straightforward: re-authenticate your session or clear the cache. Taking a few extra steps, like keeping your AWS CLI updated or automating token management, can help you avoid these interruptions in the future.

Remember, AWS is designed to make things easier, not harder! If you run into this error, don’t panic. Follow the steps outlined here, and you’ll be back on track in no time.

Regex for a Java Software Engineer

Joao Marques — Mon, 28 Oct 2024 13:18:38 +0000

Why do I need Regex?

Regular expressions are patterns that help us search for specific sequences in a text. In Java, they are used with classes in the java.util.regex package.
With regex, we can find patterns, replace text, and validate inputs without adding too much code.

Basic Syntax

Let’s go over some common regex symbols and what they do:

Literal Characters: The simplest regex is just plain text. hello matches any occurrence of hello in a string.
Wildcards:
.: Matches any single character (h.llo matches hello, hallo, hxllo).
Character Sets:
[abc]: Matches any character within the brackets (h[aeiou]llo matches hello, hallo).
[a-z]: Matches any lowercase letter from a to z.
Quantifiers:
*: Matches zero or more occurrences of the letter behind it(go*gle matches google, ggle, goooooooogle).
+: Matches one or more occurrences (go+gle matches google, goooglebut not ggle).
?: Matches zero or one occurrence of the letter behind it(colo?ur matches both colurand colour).
Anchors:
^: Indicates the start of a line (^hello matches any line that begins with hello).
$: Indicates the end of a line (world$ matches any line that ends with world).
Groups:
(abc): Groups multiple characters as a single unit ((ha)+ matches ha, haha, hahaha).
Escape Characters:
Some characters (like . or *) have special meanings, so prefix them with a backslash \ to use them literally. For instance, \. will match a literal dot.

Short example:

Pattern: Compiles the regular expression and matches it in a text.
Matcher: Applies the pattern to a specific text and helps find matches.

Here’s a quick example of how these classes work together:

import java.util.regex.*;

import java.util.regex.*;

public class RegexBasicsDemo {
    public static void main(String[] args) {
        String text = "hxllo hallo hbllllllo hello";
        Pattern pattern = Pattern.compile("h.llo");
        Matcher matcher = pattern.matcher(text);
        while (matcher.find()) {
            System.out.println("Wildcard match found: " + matcher.group());
        }
   }
}

What will be printed:

Wildcard match found: hxllo
Wildcard match found: hallo
Wildcard match found: hello

import java.util.regex.*;

public class RegexReplaceExample {
    public static void main(String[] args) {

        String text = "hello hzllo hallo hillo";
        Pattern pattern = Pattern.compile("h[aeiou]llo");
        Matcher matcher = pattern.matcher(text);

        String result = matcher.replaceAll("hi");

        System.out.println("Original text: " + text);
        System.out.println("Text after replacement: " + result);
    }
}

What will be printed:

Original text: hello hzllo hallo hillo
Text after replacement: hi hzllo hi hi

Useful Java Regex Methods

matches(): Checks if the whole text matches the regex pattern.
find(): Searches for occurrences of the pattern in the text (returns true if, and only if, a subsequence of the input sequence matches this matcher's pattern)
group(): Returns the matched text after calling find().
replaceAll(): Replaces matches in the text with a replacement string

My opinion about regex

As a Java developer, I’ve come to really appreciate regex for how powerful it can be with text processing. It’s amazing to see how one well-crafted line of regex can handle tasks that might otherwise need an entire block of code. For straightforward matching, regex feels perfect: it’s concise, efficient, and ideal for things like validating formats or extracting patterns.

But I know not everyone feels the same way. Regex can be far from intuitive, and when patterns start getting complex, readability suffers. It’s easy to create patterns that work like magic, yet are nearly impossible for anyone else (or even yourself, later on, after you came back from a nice vacation) to understand at a glance. Complex patterns can quickly become "write-only" code.

In these situations, I’ve found it better to break validation down into smaller, simpler steps. This keeps things clearer and makes it easier for others to follow the logic. While regex is such a valuable tool in Java, I think it’s best used with a bit of restraint, especially in team environments. After all, writing maintainable code means thinking of the next person who’ll read it.

Preventing Multiple Processing of SQS Messages

Joao Marques — Mon, 22 Jul 2024 03:38:17 +0000

Introduction

Amazon SQS (Simple Queue Service) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. AWS Lambda can process messages from an SQS queue, making it a powerful combination for event-driven applications.

Scenario

Imagine an AWS architecure like this:

Imagine a scenario where your Lambda function fails to tag an SQS message as successfully processed. By default, Lambda retries the message up to 4 times before ignoring it. This retry behavior can lead to exponential execution of your handler:

First Lambda: Retries 4 times, potentially sending 4 messages to the next queue.
Second Lambda: Each of the 4 messages triggers 4 retries, resulting in 16 executions.
Third Lambda: Each of the 16 messages triggers 4 retries, resulting in 64 executions.

This exponential growth can significantly inflate your AWS costs by the end of the month.

Cause

The root cause of this issue is that AWS Lambda handlers processing SQS events are expected to return an SQSBatchResponse object. If your handler returns a different type, you risk losing control over what AWS recognizes as a successful execution.

Here's an example of the expected handler signature:

public class SQSLambdaHandler implements RequestHandler<SQSEvent, SQSBatchResponse> {

    @Override
    public SQSBatchResponse handleRequest(SQSEvent event, Context context) {
...

The SQSBatchResponse should contain a list of message IDs that failed to process. If your handler returns an SQSBatchResponse with an empty list of failed message IDs, AWS interprets this as a successful execution, and the Lambda function will not be retried for those messages.

Code example

public class SQSLambdaHandler implements RequestHandler<SQSEvent, SQSBatchResponse> {

    @Override
    public SQSBatchResponse handleRequest(SQSEvent event, Context context) {
        List<BatchItemFailure> batchItemFailures = new ArrayList<>();

        for (SQSMessage msg : event.getRecords()) {
            try {
                logger.info("Message received: " + msg.getBody());
                // Process the message here
            } catch (Exception e) {
                logger.error("Error processing message: " + msg.getMessageId() + " - " + e.getMessage());
                batchItemFailures.add(new BatchItemFailure(msg.getMessageId()));
            }
        }

        return new SQSBatchResponse(batchItemFailures);
    }
}

When an exception is caught, we give the message another chance to be processed correctly by adding its ID to the batchItemFailures list:

batchItemFailures.add(new BatchItemFailure(msg.getMessageId()));

This tells Lambda to retry the message with the specified ID msg.getMessageId(). The number of retry attempts is based on the configuration you have set in your cdk-stack.

How to Avoid the Lottery Factor

Joao Marques — Tue, 16 Jul 2024 04:37:41 +0000

During your experience you might have heard about the Bus Factor.

What is the Bus Factor? 🚌

The bus factor is a measure of risk in a team or project. Specifically, it refers to the number of key team members who, if suddenly unable to work (imagine they were hit by a bus), would cause the project to come to a halt or face significant delays.

What is the Lottery Factor? 💰

The lottery factor is almost the same thing as the bus factor, it represents the sudden and unexpected departure of a key team member for any reason. but, instead of worrying about a key team member getting hit by a bus, imagine if they suddenly won the lottery and decided they didn't want to work anymore.

Picture this: a critical member of your team hits the jackpot and becomes an instant millionaire. The next day, they walk into the office and announce that they're quitting, effective immediately. They're so eager to start their new life that they even offer the company $20,000 to find a new developer to take over their work right away.

Why is this a real risk? ⚠️

If critical knowledge is concentrated in one person, their sudden exit can leave a huge gap. It’s not just about the code they wrote, but also the nuances and insights that come with experience.
When a key player leaves abruptly, it can cause significant delays. The new person taking over needs time to get up to speed, which can slow down the whole project.
Finding and training a new developer to handle specialized tasks isn't quick or cheap. There are recruitment costs, onboarding time, and the inevitable learning curve as the new hire gets acquainted with the project.

Avoid the Lottery Factor is good only for the company? 🤔

It’s clear that avoiding these risks is crucial for the company’s market survival. However, it’s also highly beneficial for developers themselves.

When developers take the time to document their code and create documentation, it shows that they care about the company’s success, not just their monthly paycheck. This can enhance their reputation within the company and demonstrate their commitment to the business.

By contributing to a culture of knowledge sharing and collaboration, developers who do that position themselves as valuable team members who invest in the long-term success of the project and the organization.

Strategies to Avoid the Lottery Factor

The Lottery Factor might sound like a joke, but it's a real risk. Check out some ways to avoid it:

Cross Training ✝️

Rotate team members through different parts of the project regularly,
ensuring knowledge distribution.
For developers, cross-training is an opportunity for professional growth. It shows that the company values their development and is willing to invest in their skill set.

Code Reviews 💻

Code reviews are excellent for mitigating the lottery factor because they promote knowledge sharing, improve code quality, and build a collaborative team culture. By allowing team members to see each other's work, code reviews spread knowledge about different parts of the codebase, ensuring more people understand critical components. They also help catch bugs, identify potential issues, and enforce coding standards, making the code easier to understand and maintain. Additionally, code reviews provide a platform for mentorship, allowing less experienced developers to learn from their more experienced colleagues.

Code Documentation ✏️

Code documentation is crucial for mitigating the lottery factor because it ensures that knowledge about the codebase is accessible to everyone, not just the original developers. Good documentation provides clear explanations of how the code and infrastructure works, its purpose, and how to use it, which makes it easier for new or other team members to understand and work with the code.

A powerful readme file ⚡

A powerful README file is essential for mitigating the lottery factor because it serves as a comprehensive guide to the project's purpose, setup, usage, and key components.
A well-crafted README ensures that critical information is not confined to a single individual. This allows anyone to quickly understand and contribute to the project, minimizing disruption if a key developer leaves. Additionally, it supports smoother onboarding and maintenance, enhancing the project's resilience and continuity.

My opinion in all of this ✨

As a developer, I've often received appreciation from my colleagues for the documentation I've created. It has helped them get unblocked and continue their work smoothly.

If I ever leave the company, for any reason (not just winning the lottery), I want to ensure that I can still support my team by providing them with links to documentation I've already worked on, I can make my transition seamless. While I'm happy to join calls to answer any questions they might have, having comprehensive documentation ensures that they have a reliable reference even in my absence.

Good documentation not only helps your colleagues but also maintains your own peace of mind. If you fall sick or need to take an unplanned vacation, knowing that everything is written down means you don't have to worry about the team being stuck without you. It’s a win-win for everyone involved.

4 Database tips for Java Developers 💡

Joao Marques — Mon, 08 Jul 2024 00:18:44 +0000

Introduction

🔍 Understanding databases is essential for Java developers, as they frequently handle critical data such as user information, past actions, application settings, and feature flags. As applications expand, databases must scale to meet performance demands and ensure efficient data management. For insights into scaling databases effectively, refer to my article on optimizing bank data growth with sharding architecture.

🚀 Drawing from my experience, I've wrote 4 essential tips to enhance your database practices in Java.

1. Always use an ORM Framework (Please do that, think about your future dev colleagues 😅)

💡 Why: ORM frameworks like Hibernate or JPA help manage database interactions efficiently, reducing boilerplate code and they manage the protection for SQL injections.
Trust me, big applications without orm are very hard to maintain and takes more time than you think to manage simple things.

💻 Example: Using Hibernate with JPA annotations.


import javax.persistence.*;

@Entity
@Table(name = "users")
public class User {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    @Column(name = "username", nullable = false, unique = true)
    private String username;

    @Column(name = "email", nullable = false, unique = true)
    private String email;

    // Getters and Setters (I'd ratter use lombok)
}

Repository:

// Repository class using Spring Data JPA
import org.springframework.data.jpa.repository.JpaRepository;

public interface UserRepository extends JpaRepository<User, Long> {
    User findByUsername(String username);
}

2. Validate Input Data

💡 Why: Validating data ensures that only correct and expected data is stored, preventing corruption and errors.
Make sure your services are ready catch exceptions coming from the database.

💻 Example: Using Hibernate Validator.


import javax.validation.constraints.*;

@Entity
@Table(name = "users")
public class User {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    @NotNull
    @Size(min = 5, max = 15)
    private String username;

    @NotNull
    @Email
    private String email;

    // Getters and Setters (I'd ratter use lombok)
}

3. Implement Database Transactions

💡 Why: Transactions ensure that you database transactions can always rollback in case an exception happens during thread execution.

💻 Example: Using Spring's @Transactional annotation.
In this example, in case userActivityRepository.save(activity) throws an exception userRepository.save(user) will be rolled back.


import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

@Service
public class UserService {
    @Autowired
    private UserRepository userRepository;

    @Transactional
    public void createUserAndSaveActivity(User user, UserActivity activity) {
        userRepository.save(user);
        userActivityRepository.save(activity);
    }
}

4. Monitor Queries

💡 Why: Monitoring queries can inform your team where in your application is causing the bottleneck.
The team can create jira tickets to work on the performance for those bottleneck queries.

💻 Example: Using Hibernate's statistics with the @Timed annotation from the Spring Metrics module to monitor the performance.


import org.springframework.stereotype.Repository;
import org.springframework.metrics.annotation.Timed;
import javax.persistence.EntityManager;
import javax.persistence.PersistenceContext;
import java.util.List;

@Repository
public class UserRepository {

    // PersistenceContext was used only for example
    @PersistenceContext
    private EntityManager entityManager;

    @Timed
    public List<User> getUsersWithOptimizedQuery(String role) {
        return entityManager.createQuery("SELECT u FROM User u WHERE u.role.name = :roleName", User.class)
                            .setParameter("roleName", role)
                            .getResultList();
    }
}

📈 Implementing these strategies are just the basis to keep your database clean and efficient when working with Java applications, that can grow anytime.

Mastering API Integrations: A Step-by-Step Guide to Secure API Authentication in Java

Joao Marques — Mon, 17 Jun 2024 22:55:08 +0000

Introduction 🛤️

Integrating with external APIs is a common practice in software development, allowing your applications to consume third-party services and expand their functionalities. Among these APIs, some stand out as robust solutions for managing subscriptions, products, and other variety of operations.

In this article, I will explain how to configure a service in Java to authenticate and interact with a REST API, specifically focusing on the authentication process. The goal is to provide a detailed guide that facilitates the creation of a secure and efficient integration with the external API, ensuring that your applications can fully leverage the resources offered by the platform without making unnecessary refresh token requests.

About ♟️

An external API allows you to manage the lifecycle of items related to the data that you need, and it will charge you for the amount of requests you make. To interact with the API, you need to authenticate your requests using a token-based authentication method. This method ensures that only authorized users can access the API resources.

This is the structure that we will talk about in this article:

Environment 🧩

To test and develop integrations with an external API, you can use the development environment provided by the service, accessible at a specific URL. This environment allows you to perform test operations without affecting production data, providing a safe space for developing and validating your integrations.

Creating a Sandbox Account: First, you need to create a sandbox account with the external service. This account allows you to access the development environment and test all API functionalities without risks.

Obtaining API Credentials: After creating your sandbox account, obtain your API credentials (client ID and client secret). These credentials will be used to authenticate your requests.

API Endpoints: Use the development environment's endpoints to make your requests. For example, the endpoint for authentication might be something like https://rest.test.external-service.com/oauth/token

Note that it is possible to define requests that require the presence of the authentication token and those that do not.

The basis 🥾

First, I'll create the base of the HTTP service, where the base HTTP methods will be present.

public class BaseHttpService {

    private static final String DEFAULT_CHARSET = "UTF-8";

    protected String post(String url, String body, Map<String, String> headers) throws IOException {

        HttpPost httpPost = new HttpPost(url);

        addHeadersToRequest(httpPost, headers);

        StringEntity requestBody = new StringEntity(body, DEFAULT_CHARSET);
        httpPost.setEntity(requestBody);

        return executeRequest(httpPost);
    }

    protected String get(String url, Map<String, String> headers) throws IOException {
        HttpGet httpGet = new HttpGet(url);
        addHeadersToRequest(httpGet, headers);
        return executeRequest(httpGet);
    }

    private String executeRequest(HttpUriRequest request) throws IOException {

        CloseableHttpClient httpClient = HttpClients.createDefault();
        HttpResponse response = httpClient.execute(request);
        HttpEntity entity = response.getEntity();

        if (entity == null) {
            return null;
        }
        String jsonResponse = EntityUtils.toString(entity);
        httpClient.close();
        return jsonResponse;
    }

    protected void addHeadersToRequest(HttpRequestBase httpRequest, Map<String, String> headers) {

        if (headers == null) {
            return;
        }

        for (var header : headers.entrySet()) {
            httpRequest.addHeader(header.getKey(), header.getValue());
        }
    }
}

These methods will be used to authenticate with the external API and also to complement the methods that require authentication. Therefore, they form the base of the service.

The Auth class 🛂

The next class we'll discuss is ExternalApiAuthenticatedRequestService. Since it is quite large, I'll divide it into smaller parts to explain each one individually.

The first part, and the most important, is to know which properties we will use in this class to maintain maximum encapsulation. These properties should be related to authentication, such as:

SecretValues: An object created to store the client ID and client secret, depending on the environment your application is running in.

baseUrl: The URL of your external service environment.
EXTERNAL_API_AUTH_PATH: The path to obtain the authentication token.

isAuthenticated: A boolean used to facilitate authentication control.

bearerToken: The authentication token, defined as private to ensure complete control and encapsulation over this property, being acesses only through this class.

externalApiTokenExpirationTimeInMillis: The expiration time of the token in milliseconds. We check this value when calling the getBearerToken method. If it has expired, we need to authenticate again.

private final SecretValues secretValues;
protected final String baseUrl;
private static final String EXTERNAL_API_AUTH_PATH = "/oauth/token";
private boolean isAuthenticated;
private String bearerToken;
private long externalApiTokenExpirationTimeInMillis;

Constructor Method 🏗️

The constructor method should set the values for secretValues and baseUrl. This depends on each application.

public ExternalApiAuthenticatedRequestService(SecretValues secretValues, String baseUrl) {
    this.secretValues = secretValues;
    this.baseUrl = baseUrl;

    // We authenticate with external Api on start, so it's faster when the first request comes
    authenticateWithExternalApi();
    logger.info("Started ExternalApiAuthenticatedRequestService");
}

The Authentication Method 🔑

Here is the most crucial method, which we use to authenticate with the external service by calling the post method defined above.

private void authenticateWithExternalApi() {
    logger.info("Authenticating to {}", getAuthUrl());

    String authRequestBody = buildAuthRequestBody(secretValues);

    Map<String, String> authHeaders = new HashMap<>();
    authHeaders.put("Content-Type", "application/x-www-form-urlencoded");
    try {
        String response = super.post(getAuthUrl(), authRequestBody, authHeaders);
        ExternalApiAuthenticationResponseDTO externalApiTokenExpirationTimeInMillis =
                Utility.convertStringToObject(response, ExternalApiAuthenticationResponseDTO.class);
        externalApiTokenExpirationTimeInMillis =
                Long.parseLong(externalApiAuthenticationResponseDTO.getExpiresIn()) * 1000 + System.currentTimeMillis();
        bearerToken = externalApiAuthenticationResponseDTO.getAccessToken();
        isAuthenticated = true;
        logger.info("Auth token retrieved from {}", getAuthUrl());
    } catch (IOException e) {
        isAuthenticated = false;
        logger.error("Could not authenticate with external Api, error: {}", e.getMessage());
    }
}

Important Points about this Method

It sends a POST request using the post method from the base class.
It obtains the token from the external API and stores it in the bearerToken variable.
It updates the token expiration time. We will use this value later.
Calls buildAuthRequestBody and getAuthUrl

The methods that hold hands 🧑‍🤝‍🧑

private String buildAuthRequestBody(SecretValues secretValues) {
    return "grant_type=client_credentials&client_id=" + secretValues.getClientId()
            + "&client_secret="
            + secretValues.getClientSecret();
}

This is to good way to not leave too much responsibility to authenticateWithExternalApi method. Its easy to create tests with this separation of concerns.

private String getAuthUrl() {
    return baseUrl + EXTERNAL_API_AUTH_PATH;
}

Get the precious bearer token 💍

private String getBearerToken() throws ExternalApiAuthenticationException {
    // Validate token existence
    if (!isAuthenticated) {
        logger.warn("External Api is not authenticated, authenticating");
        reAuthenticateWithExternalApi();
    }
    // Validate token expiration
    if (externalApiTokenExpirationTimeInMillis < System.currentTimeMillis()) {
        logger.info("External Api token expired, authenticating again");
        reAuthenticateWithExternalApi();
    }
    return bearerToken;
}

Notice that here we cannot just call authenticateWithExternalApi, because in case it fails we wanna do something about it.
That's why I added this method reAuthenticateWithExternalApi, in some application you might wanna try authenting again one or two times before throwing your exception.
You might want to define a variable called numberOfTrialsForAuthentication in your application-{environment}.xml to call reAuthenticateWithExternalApi recursively.

private void reAuthenticateWithExternalApi() throws ExternalApiAuthenticationException {
    authenticateWithExternalApi();
    if (!isAuthenticated) {
        throw new ExternalApiAuthenticationException("Could not authenticate with external Api");
    }
}

The magic methods 🖌️

And finally your service can make authenticated get and post requests:

@Override
protected String post(String url, String body, Map<String, String> headers) throws IOException {
    if (headers == null) {
        headers = new HashMap<>();
    }
    headers.put("Authorization", "bearer " + getBearerToken());
    return super.post(url, body, headers);
}

@Override
protected String get(String url, Map<String, String> headers) throws IOException {
    if (headers == null) {
        headers = new HashMap<>();
    }
    headers.put("Authorization", "bearer " + getBearerToken());
    return super.get(url, headers);
}

Note that we re-use the get and post methods from the base http service.

Please, leave in the comments if you could understand the thought process.

The action 👨‍💻

public class ExternalApiHttpService extends ExternalApiAuthenticatedRequestService{

    private static final Logger logger = LoggerFactory.getLogger(ExternalApiHttpService.class);

    private static final String API_VERSION = "/v1";
    private static final String GET_PRODUCTS = "/products/accounts/%s";

    public ExternalApiHttpService(SecretValues secretValues, String baseUrl) {
        super(secretValues, baseUrl);
    }

    public ProductsResponseDTO retrieveProductsFromAccount(String accountId)
            throws IOException, ExternalApiAuthenticationException {
        logger.info("Retrieving products from account: {}", accountId);
        String requestPath = buildBaseUrl() + String.format(GET_PRODUCTS, accountId);
        String jsonResponse = get(requestPath, new HashMap<>());
        return Utility.convertStringToObject(jsonResponse, ProductsResponseDTO.class);
    }

    private String buildBaseUrl(){
        return baseUrl + API_VERSION;
    }
}

And finally, you can add some http api requests in this class.

notice that I only added /products/accounts/%s

but you can find a good way to organize your project using this structure.

If you feel that this class will grow too much, maybe would be a good idea to create a separated auth service, and call this auth service with your requests.
In my case I added it as a parent class because at the moment only few requests are necessary. So I don't need to hold the authentication service reference and it saves me some minutes of coding 😌.

Please share with me your thoughts, there is no right and wrong in this dev community, there are alternatives and developers should identify which one fits better for their outcome.

Optimizing Bank Data Growth with Sharding Architecture

Joao Marques — Tue, 11 Jun 2024 04:22:46 +0000

As the number of customers and the volume of related data increase, applications tend to become slower due to bottlenecks in the database. Databases need to perform increasingly complex searches on larger volumes of data.

In the case of banks, this is crucial! Most financial institutions prefer to have millions of customers rather than just hundreds. Therefore, developers need to architect the backend and the database from the outset to expand exponentially and avoid bottlenecks caused by data growth.

By adopting certain practices, developers can build robust and scalable systems that meet the demands of large financial institutions. The use of the Spring Framework, with its tools and libraries, facilitates the implementation of these techniques, providing a backend capable of growing along with the bank and its customers.

The Importance of Scalability in Banks

For banks, database scalability is crucial for several reasons:

High Availability: In a bank, any interruption in services can have significant consequences. A scalable database helps ensure that services remain available even with a sudden increase in the number of transactions.

Consistent Performance: As the volume of data increases, it is vital that queries and operations in the database maintain consistent performance to provide a good customer experience.

Compliance and Security: Banks handle sensitive data and face strict regulations. A scalable database architecture must also consider security and regulatory compliance aspects. Without this care, validations can become very costly.

Database Sharding

In this article, I will discuss one of the most commonly used techniques by large banks to ensure the scalability and efficiency of their databases: database sharding.

Sharding, a form of horizontal partitioning, is a scalability technique that involves dividing a large database into smaller, more manageable pieces called shards. Each shard is a complete and independent database containing a portion of the total data. These shards can be distributed across different servers, which helps distribute the workload and improve system performance.

Why is Sharding Important for Banks?

For large banks, sharding offers several advantages:

Better Performance: By dividing the data into shards, queries and operations are distributed among multiple servers. This reduces the load on each individual server, resulting in faster response times.
Horizontal Scalability: As the volume of data and transactions grows, new shards can be added as needed. This allows the system to expand efficiently and economically.
Fault Isolation: If a shard fails, only a portion of the data is affected, while the other shards continue to operate normally. This increases the resilience and availability of the system.
Region-Specific Data Management: For banks that expand internationally, like Nubank, shards can be located near the regions where the data is most accessed, further improving performance.

Note that in the image, the information about which shard contains the user's data is stored in the authentication database. After login, this information is passed to the API Gateway and then to the Spring Framework components. This ensures that requests are correctly directed to the appropriate shard, optimizing system performance and scalability.

Architecture Advantage for Banks
Banks have a significant advantage: users do not need to be aware of other users within the bank. Some banks, like PicPay, offer this functionality, but they are rare. The relationship is always between the user and the central bank. This simplifies the backend design since it only needs to capture user data during authentication. After that, all account data can be located in the shards.

Microservice code example

In order to receive the shard information, you can get it from the request coming from the API gateway.
And immediatly set the it to the shard context:

@PostMapping("/user/deposit")
public ResponseEntity<depositResponseDTO> userDeposit(
        @RequestAttribute long userId,
        @RequestAttribute String shard,
        @Valid @RequestBody depositRequestDTO request) {
    log.info(USER_DEPOSIT, userId);
    ShardContext.setCurrentShard(shard);
    return new ResponseEntity<>(userService.deposit(request, userId), HttpStatus.OK);
}

Here we create a class ShardContextUsed to hold information of which shard the request belongs. During the execution of the current thread, we can always call ShardContext.getCurrentShard() to know which shard we should access.

public class ShardContext {

    private static final ThreadLocal<String> currentShard = new ThreadLocal<>();

    public static void setCurrentShard(String shardKey) {
        currentShard.set(shardKey);
    }

    public static String getCurrentShard() {
        return currentShard.get();
    }

    public static void clear() {
        currentShard.remove();
    }
}

here we dynamically determine the actual datasource on the current context, calling the method that i explained above.

public class RoutingDataSource extends AbstractRoutingDataSource {

    @Override
    protected Object determineCurrentLookupKey() {
        return ShardContext.getCurrentShard();
    }
}

And finally, we define the datasources, you should do it using the application-{profile}.yml file, but here I'm just explaining to you how it works.

Notice that Im using the class RoutingDataSource defined above.

@Configuration
public class DataSourceConfig {

    @Bean
    public DataSource dataSource() {
        // Configuration of multiple data sources (shards)
        DataSource shard1 = createDataSource("jdbc:mysql://shard1-url", "username", "password");
        DataSource shard2 = createDataSource("jdbc:mysql://shard2-url", "username", "password");

        // Routing data source
        Map<Object, Object> targetDataSources = new HashMap<>();
        targetDataSources.put("shard1", shard1);
        targetDataSources.put("shard2", shard2);

        RoutingDataSource routingDataSource = new RoutingDataSource();
        routingDataSource.setTargetDataSources(targetDataSources);
        routingDataSource.setDefaultTargetDataSource(shard1);

        return routingDataSource;
    }

    private DataSource createDataSource(String url, String username, String password) {
        HikariConfig config = new HikariConfig();
        config.setJdbcUrl(url);
        config.setUsername(username);
        config.setPassword(password);
        return new HikariDataSource(config);
    }
}

And these principles form the foundation of a robust architecture. Banks, more than any other institutions, require highly scalable systems to manage the ever-increasing volume of data efficiently. By implementing sharding and leveraging frameworks like Spring, developers can ensure that banking systems remain resilient, performant, and capable of growing alongside their customer base. In an industry where high availability, consistent performance, and stringent security are paramount, a scalable database architecture isn't just an advantage, it's a necessity.