Forem: Cristiano Pacheco

Good-bye Docker, Hello Nix: Configuring a Magento 2 Development Environment with Rooter

Cristiano Pacheco — Tue, 20 Feb 2024 10:50:29 +0000

Introduction
Requirements
Creating a Magento 2 dev environment
- 1. Creating the project directory
- 2. Entering in the created directory
- 3. Creating the environment with rooter
- 4. Allowing direnv to execute
- 5. Cloning the Magento 2 repository
- 6. Starting the environment stack
- 7. Setting up the database
Rooter Appendix
- The .devenv directory
- Versioning
- Diving deeper into the most important files
  - .env
  - .envrc
  - devenv.nix
  - devenv.yaml
- Accessing the traefik dashboard
- Accessing the mailpit UI
- Rooter commands
- Xdebug

Introduction:

In 2023, I switched from Linux to macOS. I used Linux for years, mainly because it supported Docker natively. However, during a system update, I encountered a problem with the video driver that cost me an entire Saturday searching for solutions, tweaking files I didn't even know existed until finally, at the end of the day, I found a workaround to get the darn Nvidia driver working.

Come Monday morning, I shared my tech nightmare with a friend at work, who promptly asked:

"Why don't you switch to macOS?"

I replied: "Because it doesn't have native Docker support, and I don't want to use a virtual machine."

With a virtual machine, you have to allocate a good chunk of RAM, not to mention the slower disk processing. While disk speed might not matter much for many applications, it makes a huge difference for a Magento 2 environment.

"Why do you even need Docker?" he asked.

That sparked a quick conversation where he introduced me to Nix and devenv.

With these tools, it's possible to create a development environment that's entirely reproducible and native to macOS!

It was love at first sight! When I saw it in action, my mind was blown!

During that conversation, I first contacted one of the early versions of rooter.

Rooter is a powerful command-line tool to automate repetitive tasks in creating and maintaining development environments.

Shortly after, I bought a MacBook and started using rooter to manage my projects. Nowadays, I don't have Docker installed on my machine, and I couldn't be happier!

Requirements

You need macOS operating system and the softwares below installed:

Just follow the instructions on the rooter documentation.

Just giving a brief introduction about the these tools:

rooter is a CLI tool created By Matthias Walter. This tool will be your best friend during your development journey. It is similar to Warden or DDEV. It takes care of things like automatic DNS resolution using dnsmasq, managing project routes with Traefik, creating SSL certificates, and even includes handy commands for importing and exporting databases, plus a whole lot more.
nix has been around since 2003 and it's super reliable. It's like a toolbox with lots of useful tools inside, such as a programming language, an operating system, and a package manager. It helps you make software systems that are consistent and easy to recreate.
devenv is kind of like docker-compose, but way stronger. It's like being able to use a programming language to set up your environment, giving you even more control and flexibility.
direnv is a tool that figures out and loads all the stuff you need for your current project whenever you step into its folder in your terminal. We'll get to see how it works soon.

Creating a Magento 2 dev environment

So getting straight to the point, let's set up a development environment to run Magento 2.

At this point, I'm assuming you've got rooter and all the necessary tools it relies on installed and ready to roll. Let's dive in!

1. Creating the project directory

mkdir -p ~/Sites/magento2

2. Entering in the created directory

cd ~/Sites/magento2

3. Creating the environment with `rooter`

rooter env:create

Rooter will ask what type of environment:

Type 2 and enter for Magento 2

It generated 4 files:

.env
.envrc
devenv.nix
devenv.yaml

4. Allowing direnv to execute

You probably saw this message in your terminal:

You need to allow direnv to execute.

Run the command below:

direnv allow .

Now you'll see something like this in your terminal:

In this moment, your dev stack declared in the devenv.nix file will be downloaded and configured. This is the longest step. Wait until the installation process finishes.

You will see something like this when the process finishes:

Now, please take a look into this image:

If you've noticed, when I'm not in the project folder, the PHP version in the terminal is different, not 8.2. But as soon as I step into the project folder, direnv works its magic and loads the entire setup into the terminal. Pretty neat, isn't it?

5. Cloning the Magento 2 repository

composer create-project --repository-url=https://repo.magento.com/ magento/project-community-edition:2.4.6-p4 magento2-codebase

It'll clone the Magento 2 repository into a directory named magento2-codebase. Once it's done, just copy everything from the magento2-codebase folder to the current directory, and then you can delete the magento2-codebase directory with the command below:

cp -R magento2-codebase/* . && rm -Rf magento2-codebase

6. Starting the environment stack

I like to start the environment in the debug mode in the first attempt, because this shows detailed output about the initialization and in case there is an issue, it is visible right away:

rooter env:start --debug

Basically devenv will download all services upon first start. That could also be nginx, opensearch, rabbitmq etc.

After the first start I like to stop the environment (CMD + c) and start it again without the debug flag:

rooter env:start

The reason being that now it is running in the background rather than the foreground which it was with --debug flag passed.

To see what is the status of the environment, run the command below:

rooter env:status

It will show something like this:

As mentioned earlier, it uses process-compose in the background. There is a command to attach to the process-compose and get a control panel to check the various processed that got launched.

rooter env:process-compose

As you can see in the image below, you can easily switch between the different services and watch their logs update live.

7. Setting up the database

Run the command below to setup the database:

rooter magento2:db-install

Rooter will drop and recreate the database and by default run the commands below:

bin/magento setup:install
bin/magento setup:upgrade
bin/magento indexer:reindex

An admin user will be created with the username admin and password admin123.

If you pass the option --config-data-import it will run the command bin/magento config:data:import config/store dev/rooter

Note: The config:data:import is provided by the Magento2-ConfigImportExport module. You need to install it to use the --config-data-import option flag.

You can skip the reindex by passing the option --skip-reindex

Run the commands below to set some extra configs:

bin/magento config:set web/unsecure/base_url https://magento2.rooter.test/ && \
bin/magento config:set web/secure/base_url https://magento2.rooter.test/ && \
bin/magento config:set web/secure/offloader_header X-Forwarded-Proto && \
bin/magento config:set web/secure/use_in_frontend 1 && \
bin/magento config:set web/secure/use_in_adminhtml 1 && \
bin/magento config:set web/seo/use_rewrites 1 && \
bin/magento config:set catalog/search/enable_eav_indexer 1 && \
bin/magento config:set dev/static/sign 0 && \
bin/magento deploy:mode:set -s developer && \
bin/magento cache:clean

These are the Store front and admin URLs:

Tcharaaam!

What really blows me away is the speed. Running Magento natively on Mac makes everything super fast and smooth.

Rooter Appendix

The `.devenv` directory

The .devenv directory contains the data about the profile, nix garbage collector, the stack state, etc.

The .devenv/state directory is where some of the stack code lives, as for example, elasticsearch, mysql, php-fpm, nginx, rabbitmq, etc.

You don't need to touch these folders or the files on it.

Versioning

Which files and directories should you version or remove from versioning:

You don't need to version the files and folders below, so add them to your .gitignore file.

.devenv
.direnv
.env

You need to version the files below:

.devenv.flake.nix
.envrc
devenv.lock
devenv.nix
devenv.yaml

Note: None of this files is required by rooter. They are required from devenv and direnv.

Diving deeper into the most important files

`.env` file:

Contains all variable environments that is used in your project stack

ROOTER_ENV_TYPE defines the type of the environment, it can be:

magento1
magento2
php-plan
shopware6
symfony

DEVENV_HTTP_PORT defines the http port used from the Nginx webserver

DEVENV_HTTPS_PORT defines the https (SSL) port used from the Nginx webserver

DEVENV_DB_PORT defines the Mysql/MariaDB/Percona database port

Note: In the current version of rooter, only Mysql* database is supported.

Mysql* I mean any of the versions or forks of it, for instance MariaDB, Percona, etc.

DEVENV_MAIL_SMTP_PORT defines the mailpit SMTP port

DEVENV_MAIL_UI_PORT defines the mailpit user interface port

DEVENV_REDIS_PORT the name already explains it :)

DEVENV_AMQP_PORT defines the RabbitMQ port that will be used in your application

DEVENV_AMQP_MANAGEMENT_PORT defines the RabbitMQ admin panel

DEVENV_ELASTICSEARCH_PORT defines the ElasticSearch port that will be used from your application

DEVENV_ELASTICSEARCH_TCP_PORT the name already explains it :)

DEVENV_PROCESS_COMPOSE_PORT the name already explains it :)

We'll see more about the process compose later.

In case you noticed and are wondering, where the heck did the values of the ports defined in these variables come from?

Rooter has a mechanism that will get the available ports in your environment, taking into account the existing environments that were created by rooter.

It will assign ports using certain ranges, you can see how it works in this file, please look:

`.envrc` file:

This file is used by the direnv tool.

It will load all the declared stack to the current terminal context.

`devenv.nix` file:

This file is used by devenv, it uses the Nix language to declare the environment.

You can find more information about it here.

Let's get more details of the most important parts of this file:

{ pkgs, inputs, lib, config, ... }:

In Nix, when you see this line, it's like setting up a template for a function. It says that the function will at least get these four things: pkgs, inputs, lib, and config. But it's cool because it can also get more stuff using the ... thingy.

let
    rooterBin = if builtins.getEnv "ROOTER_BIN" != "" then builtins.getEnv "ROOTER_BIN" else "rooter";
    composerPhar = builtins.fetchurl{
        url = "https://github.com/composer/composer/releases/download/2.2.22/composer.phar";
        sha256 = "1lmibmdlk2rsrf4zr7xk4yi5rhlmmi8f2g8h2izb8x4sik600dbx";
    };
    magerun2Phar = builtins.fetchurl{
        url = "https://github.com/netz98/n98-magerun2/releases/download/7.2.0/n98-magerun2.phar";
        sha256 = "0z1dkxz69r9r9gf8xm458zysa51f1592iymcp478wjx87i6prvn3";
    };
in {

The let...in block in the provided devenv.nix file defines a scope for local definitions that can be used later in the expression. This construct allows for declaring variables and their values before using them in the main body of the Nix expression.

Here's a breakdown of the let block and its components:

rooterBin: Defined based on the ROOTER_BIN environment variable. If set, rooterBin uses its value; if not, it defaults to "rooter". Enables flexible configuration.
composerPhar/magerun2Phar: Downloads the Composer PHAR and n98-magerun2 PHAR file from GitHub using a URL and a sha256 hash for security.

    dotenv.enable = true;
    env = {
        PROJECT_NAME = "magento2";
        PROJECT_HOST = "magento2.rooter.test";

        NGINX_PKG_ROOT = pkgs.nginx;
        DEVENV_STATE_NGINX = "${config.env.DEVENV_STATE}/nginx";

        DEVENV_PHPFPM_SOCKET = "${config.env.DEVENV_STATE}/php-fpm.sock";

        DEVENV_DB_NAME = "app";
        DEVENV_DB_USER = "app";
        DEVENV_DB_PASS = "app";

        DEVENV_AMQP_USER = "guest";
        DEVENV_AMQP_PASS = "guest";
    };

The env block declares the environment variables.

The project's domain is determined by the PROJECT_HOST variable. So, you can access the project using the address: https://magento2.rooter.test.

If you need to change the PROJECT_HOST variable value, follow the instructions here.

It is possible to have custom subdomains, you can see more info about it here.

Just a heads up, the database will use as name, user, and password, all set to app by default. Feel free to tweak these settings whenever you need.

Similarly, for RabbitMQ, the default user and password are both guest. You're welcome to adjust these as necessary.

This is where we define and set up PHP. If you ever need to switch the PHP version, simply make the change in the highlighted section shown in the image.

The same for MariaDb version. If you need a different version just change it.

To know what are the supported versions, you can search in directly in the nix packages page, for example:

Note: please note that we're using the unstable channel, it is the default channel for devenv, because it has more packages and with newer versions on it. So when you'll search the package, don't forget to select the unstable channel.

Last we have RabbitMQ declaration, if your project does not use it, just remove it from the devenv.nix file. We'll maintain it :)

`devenv.yaml` file:

This file is used by devenv. It will serve to declare what is the source of the packages. You can have multiple inputs, using different channels, for example.

Accessing the `traefik` dashboard

To access the traefik dashboard, just type rooter traefik:dashboard

Accessing the `mailpit` UI

Mailpit is a fork of mailhog, It is newer and constantly receives updates.

It is per environment it will have always the URL with this pattern:

https://PROJECT_NAME-mail.rooter.test

In our case it will be:

http://magento2-mail.rooter.test

Note: The PROJECT_NAME variable is defined in the devenv.nix file

Rooter commands

Just a heads up, to explore the existing rooter commands, you can simply run the command rooter commands. This will give you a list of all available commands. Happy exploring!

Xdebug

The only thing action you need to configure Xdebug is to set this environment env to your terminal:

export XDEBUG_CONFIG="idekey=PHPSTORM-XDEBUG"

You can append it with the command below:

echo 'export XDEBUG_CONFIG="idekey=PHPSTORM-XDEBUG"' >> ~/.zshrc

Now you're able to debug your Magento 2 store on CLI or throughout web without any configuration.

On PHPStorm for instance, you just have to click on the "Bug" button to start listening to Xdebug connections:

If you have any specific scenarios you'd like guidance on, like setting up Magento with multiple websites or store views, feel free to leave your question in the comments below. I'll do my best to assist you!

I hope you found this helpful! 👋

Preventing Transaction Loss: Unleashing the Power of Resilient Transactions with RabbitMQ dead-letter Exchanges in Magento 2

Cristiano Pacheco — Mon, 28 Aug 2023 10:57:40 +0000

This post will guide Magento 2 developers through implementing a message queue retry mechanism using the RabbitMQ dead-letter exchange feature.

To assist us in this endeavor, we will leverage the Message Queue Retry module by run-as-root.

What is RabbitMQ?
What is a dead-letter exchange?
Examples of transaction loss scenarios
Practical case scenario
Crafting an effective retry strategy for queue messages
Practical case scenario with queue retry

What is RabbitMQ?

RabbitMQ is a widely embraced open-source Message Queue software, facilitating seamless interaction and scalability across numerous applications. As a proficient messaging broker, RabbitMQ furnishes a unified platform catering to Asynchronous messaging needs. Within this platform, applications gain the capability to transmit and receive messages securely, with RabbitMQ diligently ensuring the sanctuary of your messages until they are consumed.

Since version 2.1.0, Magento 2 has offered native RabbitMQ support, enabling effortless queue configuration through XML settings. For further details, you can explore additional information here

What is a dead-letter exchange?

This feature functions as a safety net within the messaging architecture, intended to manage messages that encounter unsuccessful processing by the consumer.

The dead-letter exchange operates as a mechanism for directing these problematic messages to a designated queue referred to as a dead-letter queue. This facilitates developers in isolating and inspecting troublesome messages without interrupting the primary processing flow, enabling them to analyze these messages, identify the underlying reasons for processing failures, and implement essential adjustments to ensure smooth operation.

It's particularly recommended for scenarios where upholding message integrity, diagnosing processing errors, and averting message loss is paramount.

Examples of transaction loss scenarios:

Exporting the customers' data to a CRM
- Possible issue: The CRM credentials were modified, but the changes were not reflected in the Magento integration settings.
- Consequence: This mismatch in credentials will result in failed API calls due to the usage of incorrect or invalid authentication information.
Exporting the orders to an ERP
- Possible issue: During a marketing campaign, the store's throughput surged by 200%, leading to a substantial rise in incoming orders; however, the ERP system could only accommodate its standard order import capacity.
- Consequence: Only a fraction of the increased order influx will be successfully exported to the ERP system, potentially causing delays and incomplete order processing.
Scheduled price uplift

Imagine having the capability to schedule a comprehensive price uplift by 10% for the entire catalog of your store.
- Possible issue: Deadlock encountered while saving product and price information.
- Consequence: The product prices may not be accurate.

Practical case scenario

A typical case in e-commerce is exporting orders to an ERP or OMS. We will implement an asynchronous order export architecture, where for each placed order, a message is created with the order id and sent to a queue. So that a consumer can later read this message and export it to the external system.

To simplify this example, let's declare the topic's name, exchange, queue, and bindings with the same name: erp_order_export.

To handle this messaging, we'll utilize RabbitMQ as the broker.

You can find the complete code for the module on this repository.

We have to create four files to declare our queue.

etc/communication.xml

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="urn:magento:framework:Communication/etc/communication.xsd">
    <topic name="erp_order_export" request="CristianoPacheco\OrderExport\Api\Data\OrderExportDataInterface"/>
</config>

etc/queue_consumer.xml

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/consumer.xsd">
    <consumer name="erp_order_export" queue="erp_order_export" connection="amqp"
              handler="CristianoPacheco\OrderExport\Queue\Consumer\ErpOrderExportConsumer::execute"
    />
</config>

etc/queue_publisher.xml

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/publisher.xsd">
    <publisher topic="erp_order_export">
        <connection name="amqp" exchange="erp_order_export"/>
    </publisher>
</config>

etc/queue_topology.xml

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/topology.xsd">
    <exchange name="erp_order_export" connection="amqp" type="topic">
        <binding id="erp_order_export" topic="erp_order_export" destinationType="queue" destination="erp_order_export"/>
    </exchange>
</config>

Run the command below in order to Magento send the command to create the exchange, queue and bindings on RabbitMQ:

bin/magento setup:upgrade

It will look like this on RabbitMQ admin:

Visually, our configs will generate something like this:

Note: It would make more sense to configure a direct exchange. However, up to the current version of Magento (2.4.6), only topic exchanges are supported.

Now that the queue is set up, we must send the order ID to the erp_order_export queue every time a successful order is placed.

We can easily do this by listening to the event sales_model_service_quote_submit_success via an observer:

etc/events.xml

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework:Event/etc/events.xsd">
    <event name="sales_model_service_quote_submit_success">
        <observer name="erp_export_order" instance="CristianoPacheco\OrderExport\Observer\SendOrderToQueueObserver"/>
    </event>
</config>

I won't post here the Observer code or the entire implementation. The complete working code is in the repository. If you want to simulate the scenarios, please install the module.

Our consumer will use a service to perform our "fake" order export. To simulate an error, our service will always throw an exception:

<?php

declare(strict_types=1);

namespace CristianoPacheco\OrderExport\Service;

use CristianoPacheco\OrderExport\Api\Data\OrderExportDataInterface;
use Exception;

class ExportOrderToErpService
{
    public function execute(OrderExportDataInterface $orderExportData): void
    {
        $orderId = $orderExportData->getOrderId();
        throw new Exception("The order $orderId id is invalid");
    }
}

With the current configuration, orders will enter the erp_order_export queue and can be "exported" during the execution of our consumer.

To run our consumer and export only one order per execution, run the following command:

bin/magento queue:consumer:start erp_order_export --single-thread --max-messages=1

What will happen at the end of the execution?

The service will throw an exception, and it will be handled here:

vendor/magento/framework-message-queue/Consumer.php:261

As you can see, the message will be rejected, and the requeue flag will have false as the value. In other words, the message will be removed from the queue, and the order will not be exported to the ERP.

Well done! We managed to simulate a transaction loss.

In day-to-day scenarios, probably (I hope so), exceptions are handled at the service or the consumer's execution entry point, and the message would be sent back to the original queue or even taken manually in cases of error, such as sending an alert, or persisting the message in a database, etc.

What if there was a way to automatically identify without manual work when there are errors in message processing, allowing the possibility to optimize and standardize this process?

Yes, there is. We can delegate this hard work to RabbitMQ via a dead-letter exchange.

Crafting an Effective Retry Strategy for Queue Messages

So, how do we ensure that our messages reach their destination in the face of these intermittent failures? This is where the concept of a retry strategy comes into play. A retry strategy outlines how we handle failed operations by making subsequent attempts at sending the same message. When exporting an order to ERP, adopting a well-defined retry strategy can significantly enhance the reliability of our application.

For this specific case, three attempts are reasonable.

So let's define our requirements, it should be possible to:

Reprocess the messages
Set processing retry limit
Set an interval in minutes for each retry attempt
Find out what error was generated during message processing
See what the message payload is
If the limit of retries is reached, we must provide a way to manage message processing manually
- Resend the message to the queue
- Discard the message

Steps to achieve our goal:

We will add a second queue (erp_order_export_delay) in RabbitMQ to store the message for a specific lifetime. This queue will not have a consumer. It will only serve to "delay" processing the message in the source queue (erp_order_export).
When the time to live is reached, RabbitMQ will automatically send the message to the source queue (erp_order_export).
If the retry limit is reached, the message will be stored in a database table for manual management.
- Maybe the ERP server is down, or some order data is incorrect, so we have to reprocess the messages later.

Steps 1 and 2 can be configured in the queue declaration, and RabbitMQ natively supports it.

Step 3 is a capability provided by the run as root Message Queue Retry module.

Practical case scenario with queue retry

Firstly we need to install the message queue retry module and enable the behavior to persist the messages on the database after the retry limit is reached.

No we have to configure our dead-letter exchange. The complete implementation of code below can be found here.

We'll use the exp_order_export_delay as the name for the topic, exchange, queue and bindings.

Declaring the topic:

etc/communication.xml

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework:Communication/etc/communication.xsd">
    <topic name="erp_order_export" request="CristianoPacheco\OrderExport\Api\Data\OrderExportDataInterface"/>
    <topic name="erp_order_export_delay" request="CristianoPacheco\OrderExport\Api\Data\OrderExportDataInterface"/>
</config>

Declaring the retry limit:

etc/queue_retry.xml

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:RunAsRoot:module:RunAsRoot_MessageQueueRetry:/etc/queue_retry.xsd">
    <topic name="erp_order_export" retryLimit="3"/>
</config>

Declaring the dead-letter exchange:

etc/queue_topology.xml

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/topology.xsd">
    <exchange name="erp_order_export" connection="amqp" type="topic">
        <binding id="erp_order_export" topic="erp_order_export" destinationType="queue" destination="erp_order_export">
            <arguments>
                <argument name="x-dead-letter-exchange" xsi:type="string">erp_order_export_delay</argument>
                <argument name="x-dead-letter-routing-key" xsi:type="string">erp_order_export_delay</argument>
            </arguments>
        </binding>
    </exchange>
    <exchange name="erp_order_export_delay" connection="amqp" type="topic">
        <binding id="erp_order_export_delay" topic="erp_order_export_delay" destinationType="queue"
                 destination="erp_order_export_delay">
            <arguments>
                <argument name="x-dead-letter-exchange" xsi:type="string">erp_order_export</argument>
                <argument name="x-dead-letter-routing-key" xsi:type="string">erp_order_export</argument>
                <argument name="x-message-ttl" xsi:type="number">60000</argument><!-- 1 minute in  milliseconds -->
            </arguments>
        </binding>
    </exchange>
</config>

Let's take a closer look at each change in the file above.

1- the arguments x-dead-letter-exchange and x-dead-letter-routing-key will tell RabbitMQ to automatically send the messages to the erp_order_export_delay exchange in case the message is negatively acknowledged (rejected or nack with requeue parameter set to false).

...
<arguments>
    <argument name="x-dead-letter-exchange" xsi:type="string">erp_order_export_delay</argument>
    <argument name="x-dead-letter-routing-key" xsi:type="string">erp_order_export_delay</argument>
</arguments>
...

2- We declared a the erp_order_export_delay exchange and bound it to the erp_order_export_delay queue

...
<exchange name="erp_order_export_delay" connection="amqp" type="topic">
    <binding id="erp_order_export_delay" topic="erp_order_export_delay" destinationType="queue" destination="erp_order_export_delay">
...

3- The arguments x-dead-letter-exchange and x-dead-letter-routing-key defines the return path for the message and the x-message-ttl has an integer number in milliseconds that defines when the message will return to the exchanged defined in the previous two arguments.

Remembering that this queue does not have a consumer.

...
<arguments>
   <argument name="x-dead-letter-exchange" xsi:type="string">erp_order_export</argument>
   <argument name="x-dead-letter-routing-key" xsi:type="string">erp_order_export</argument>
   <argument name="x-message-ttl" xsi:type="number">60000</argument><!-- 1 minute in  milliseconds -->
</arguments>
...

Visually, our new configs will generate something like this:

Note: We always send the messages to an exchange and the exchange will forward the messages to one or more queues. It is the same with dead-letter exchanges.

Now we have to manually delete the erp_order_export exchange and queue in RabbitMQ, because it won't be updated. You can do it in the RabbitMQ admin panel or via CLI.

After you did it, you have to run the command below to reflect our changes in RabbitMQ:

bin/magento setup:upgrade

Now if we navigate again in the RabbitMQ panel we should see:

Exchanges:

If we open our exchanges an expand the bindings, is it possible to see the arguments we have declared before:

Queues:

Did you notice that queues now have a few more features?

A homework for you, hover the mouse over each feature and wait a few seconds, a caption will be displayed informing what it is about. Also access each queue and expand the bindings section, see the values contained there.

Now that we finished our RabbitMQ configuration, let's place a new order and start the erp_order_export consumer to see what will occur.

After place an order a new massage is in the erp_order_export queue:

Now let's start our consumer:

bin/magento queue:consumer:start erp_order_export --single-thread --max-messages=1

Now if we take a look at the queues we'll see:

During message consumption, our service threw an exception, simulating a problem and the message was rejected. RabbitMQ sent the message to the erp_order_expor_delay queue.

After one minute the message was automatically sent to the erp_order_export queue.

Let's inspect the message headers:

The x-death header has interesting information about the retries.

Relevant attribute for us to analyse are:

count - it is the retry number
reason - In our case, can be rejected or expired
time - The timestamp of the occurrence
exchange, queue, routing-keys - Where the message was at the time of occurrence

So, what that image says is:

Reading from bottom to top

First occurrence

The message was in the erp_order_export queue and while processing it, it was rejected on Saturday, August 26, 2023 9:12:33 PM GMT-03:00

Second occurrence

The message was in the erp_order_export_delay queue, and it was rejected on Saturday, August 26, 2023 9:12:53 PM GMT-03:00

It has 20 seconds difference because I configured the TTL for 20 seconds :)

This is the message headers after the second retry:

After the third retry:

What will occur in the next consumer execution?

So, the run-as-root module intercepted the message processing failure, verified that the message has reached the retry limit and then persisted the message in the Magento database.

Now it is possible to manage the message through the admin panel, in the path:

Run as Root > Manage Messages

Now to can decide if you'll discard the message or send it back to the erp_order_export queue.

In summary, while multiple approaches exist to tackle this issue, my aim has been to show a straightforward method for to implement a retry mechanism.

That's it for today, happy coding!

Configuring a docker environment for a Magento 2 project

Cristiano Pacheco — Tue, 25 Jul 2023 19:09:07 +0000

This post will demonstrate how to configure a docker environment with the Warden tool for an existing Magento 2 project.

What is Warden?
Why Warden?
Quickstart
Requirements
Project Stack
Configuring a single website/store Magento 2 project
- Creating the project folder
- Creating the project folder
- Clone the Magento 2 code-base into the code-base folder
- Clone the Warden Magento 2 template
- Update the warden .env file
  - A brief explanation of the keys in the .env file
- Run the docker environment
- Run composer install
- Copy and rename the env.warden.php file
- Import the Magento 2 database
  - An Example of the content of this set-default-config.sh script
- Sign the SSL certificate
- Update your /etc/hosts file with the store domain
- Installing the sample data (Optional)
Configuring a multiple website/store view Magento 2 project
- Create a project folder
- Enter the created multisite directory
- Clone the Magento 2 code-base into the multisite code-base folder
- Clone the Warden Magento 2 template
- Change the warden .env file
- Define custom variables for the store domains
- Updates the the defaults config script
- Run the docker environment
- Composer install
- Copy and rename the env.warden.php file into the etc folder
- Import the Magento 2 multisite database
- Sign the SSL certificates for all domains
- Append your /etc/hosts file with the store domains
- Configuring the application domains

What is Warden?

Warden is a CLI utility for orchestrating Docker-based developer environments that fully support Magento 1, Magento 2, Laravel, Symfony, and Shopware 6 on macOS, Linux and Windows (via WSL2).

Why Warden?

Warden lets you run multiple environments simultaneously without conflicting ports.
Allows you to configure stack versions easily via .env file configuration.
Provides the full stack needed for a Magento 2 development environment.
You can extend it and add new features according to your needs.
No need to version the docker environment in the project.

Quickstart

We will configure two Magento 2 projects, the first project will have one website and one store view, and the second project will have multiple websites and store views.

The commands exposed here will work in Linux or macOS environment.

We will use the ~/Sites directory to place the projects. Feel free to put your project in any path on your computer if you change the commands informed here.

For each project, the following directory structure will be used:

📦project-name
 ┣ 📂code-base
 ┗ 📂docker

The project-name/code-base directory will contain the Magento 2 project code
The project-name/docker directory will contain the warden template for our docker environment

Note: The structure above is just a personal taste to use in my projects, you can structure as you see fit. For example using the ~/.warden directory

Your Magento 2 project can be versioned alone in the repository, and there is no need to change the directory architecture.

Here we'll use a fork of the Warden Magento 2 Template that have a Makefile and a few common commands to facilitate the day-to-day work.

Requirements

All you need is Docker, docker-compose and Warden installed on your computer.

Project Stack:

For a practical example, we will set up the development environment for an existing Magento 2 project with the stack below:

Magento 2.4.4 open source
Nginx 1.16
PHP 8.1
- Composer 2
- Xdebug 3
MariaDB 10.4
Elastic Search 7.16
Redis 6.2
RabbitMQ 3.9

More details about the required stack for Magento 2 here.

Configuring a single website/store Magento 2 project

1 Creating the project folder

mkdir ~/Sites/magento244

2 Enter the created directory

cd ~/Sites/magento244

3 Clone the Magento 2 code-base into the code-base folder

git clone git@github.com:cristiano-pacheco/magento244.git code-base

4 Clone the Warden Magento 2 template

git clone git@github.com:cristiano-pacheco/warden-env-magento2.git docker

5 Update the warden .env file

Enter the docker directory

cd ~/Sites/magento244/docker

Update the .env file with the content below:

WARDEN_ENV_NAME=magento244
WARDEN_ENV_TYPE=magento2
WARDEN_WEB_ROOT=/../code-base

TRAEFIK_DOMAIN=magento244.test
TRAEFIK_SUBDOMAIN=app

WARDEN_DB=1
WARDEN_ELASTICSEARCH=1
WARDEN_VARNISH=0
WARDEN_RABBITMQ=1
WARDEN_REDIS=1

ELASTICSEARCH_VERSION=7.16
MARIADB_VERSION=10.4
NODE_VERSION=10
PHP_VERSION=8.1
PHP_XDEBUG_3=1
COMPOSER_VERSION=2
RABBITMQ_VERSION=3.9
REDIS_VERSION=6.2
VARNISH_VERSION=7.0

MYSQL_USER=magento
MYSQL_PASSWORD=magento
MYSQL_DATABASE=magento

WARDEN_SYNC_IGNORE=

WARDEN_ALLURE=0
WARDEN_SELENIUM=0
WARDEN_SELENIUM_DEBUG=0
WARDEN_BLACKFIRE=0
WARDEN_SPLIT_SALES=0
WARDEN_SPLIT_CHECKOUT=0
WARDEN_TEST_DB=0
WARDEN_MAGEPACK=0

BLACKFIRE_CLIENT_ID=
BLACKFIRE_CLIENT_TOKEN=
BLACKFIRE_SERVER_ID=
BLACKFIRE_SERVER_TOKEN=

A brief explanation of the keys in the .env file:

WARDEN_ENV_NAME It will be the prefix of the docker containers name and will also serve to configure the xdebug in your IDE Example:

WARDEN_ENV_TYPE
It will define the docker compose yaml files that will be used in the environment, which can vary according to the chosen platform.
WARDEN_WEB_ROOT
Self explained, you have to inform the root directory of the project.
TRAEFIK_DOMAIN
Traefik is used to route the requests to the services (containers) dynamically. For example, when there is the cookie XDEBUG_SESSION with the value PHPSTORM, it will send the requests to the php-debug container that has the xdebug enabled. On the contrary, the php-fpm container will receive the request.
You can set the domain you want, just pay attention to the .test suffix, it you configure a domain with a different value you will lost the Automatic DNS Resolution feature.
TRAEFIK_SUBDOMAIN
It is used to set a subdomain for your application, in our example the store can be accessed with the address https://TRAEFIK_SUBDOMAIN.TRAEFIK_DOMAIN that is: https://app.magento244.test
WARDEN_DB, WARDEN_ELASTICSEARCH, WARDEN_VARNISH, WARDEN_RABBITMQ, WARDEN_REDIS

The keys above you can use 0 to disable and 1 to enable. When the value is 0 the container will not run.

_VERSION
The keys with the suffix _VERSION is to set a specific version to the service. For Magento 2.4.3, for example, you may use PHP 7.4. To do it, you can leave the key as: PHP_VERSION=7.4
PHP_XDEBUG_3
If you set this key with value 1, a different tag of the container php-fpm image will be used. Let's look at the docker-compose yaml generated by warden with the command warden env config:

COMPOSER_VERSION If it has the value = 1 or you ommit this key composer in version 1 will be the default executable, however you can run composer in version 2 with the command composer2.

Example:
If you omit or set the value 1 for COMPOSER_VERSION

If you set the value 2 for COMPOSER_VERSION:

I prefer to inform the value 2 and already run composer in version 2 with the composer command.

There are more interesting things to configure such as Magento 2 tests, live reload, elastic search and more. You can see this information here.

6 Run the docker environment

Inside the ~/Sites/magento244/docker directory type:

make docker-up

7 Run composer install

Inside the ~/Sites/magento244/docker run the command
below to access the PHP container

warden shell

Run the composer install command

composer install

8 Copy and rename the `env.warden.php` file

It will copy the file to the code-base/etc directory with the command below:

cp ~/Sites/magento244/code-base/app/etc/env.warden.php ~/Sites/magento244/code-base/app/etc/env.php

The env.warden.php file has the default configuration entries with the warden hosts for the Mysql database, Redis (session and store cache), RabbitMQ, and Elastic Search.

9 Import the Magento 2 database

There is a sample database inside our Magento project, so copy the ~/Sites/magento244/code-base/dump.sql.gz file to ~/Sites/magento244/docker/backfill/magento-db.sql.gz with the command below:

cp ~/Sites/magento244/code-base/dump.sql.gz ~/Sites/magento244/docker/backfill/magento-db.sql.gz

Open a new tab in your terminal and enter the directory ~/Sites/magento244/docker and type the command below:

make db-import

The db-import command is one of the custom commands inside the Makefile.

This command imports the Magento-db.sql.gz database dump to the Magento database in the warden MySQL docker container.

In addition to importing the database, the ~/Sites/magento244/docker/tools/set-default-config.sh script is then executed to define the project's base URL and create a user for the Magento administrative panel.

In the ~/Sites/magento244/docker/tools/set-default-config.sh file, I usually put all the relevant settings to the Magento 2 environment to avoid errors and also to speed up the restoration of the environment as faithful as possible to the actual project.

Example of settings that are added in this file:

Project base URLs
Media and asset URLs
Payment test credentials
Changing the Algolia index to the development index, etc.

An Example of the content of this `set-default-config.sh` script:

#!/usr/bin/env bash

source $(dirname $0)/common.sh

:: Import the config from files
warden env exec -T php-fpm bin/magento app:config:import

:: Set BASE URL Config
warden env exec -T php-fpm bin/magento config:set web/unsecure/base_url ${URL_FRONT}
warden env exec -T php-fpm bin/magento config:set web/secure/base_url ${URL_FRONT}
warden env exec -T php-fpm bin/magento config:set web/unsecure/base_link_url ${URL_FRONT}
warden env exec -T php-fpm bin/magento config:set web/secure/base_link_url ${URL_FRONT}

:: Create an admin user
warden env exec -T php-fpm bin/magento admin:user:create --admin-user=admin --admin-password=123123q --admin-email=admin@example.com --admin-firstname=Admin --admin-lastname=Admin

10 Sign the SSL certificate

Still, inside the ~/Sites/magento244/docker directory, type the command below:

warden sign-certificate app.magento244.test

11 Update your `/etc/hosts` file with the store domain:

sudo echo '127.0.0.1 app.magento244.test' >> /etc/hosts

Note Do the step above only if dnsmasq is not being used.

12 Installing the sample data (Optional)

Access the php container:

warden shell

Run the command below to install the sample data:

bin/magento sampledata:deploy

Run the command below to update the project database

bin/magento setup:upgrade

Now you're able to access the store with the URLS \o/:

Store Address: https://app.magento244.test
Backend admin panel: https://app.magento244.test/backend

Configuring a multiple website/store view Magento 2 project

This Magento 2 store has been configured with the below structure of websites/stores/store views:

So, the main difference is how to configure a local warden environment to support multiple websites/store views with different domains.

1 Create a project folder

mkdir ~/Sites/magento244-multisite

2 Enter the created multisite directory

cd ~/Sites/magento244-multisite

3 Clone the Magento 2 code-base into the multisite code-base folder

git clone git@github.com:cristiano-pacheco/magento244-multisite.git code-base

4 Clone the Warden Magento 2 template

git clone git@github.com:cristiano-pacheco/warden-env-magento2.git docker

5 Change the warden .env file

Enter the docker directory

cd ~/Sites/magento244-multisite/docker

Update the .env file with the content below:

WARDEN_ENV_NAME=magento244-multisite
WARDEN_ENV_TYPE=magento2
WARDEN_WEB_ROOT=/../code-base

TRAEFIK_DOMAIN=magento244-multisite.test
TRAEFIK_SUBDOMAIN=app

WARDEN_DB=1
WARDEN_ELASTICSEARCH=1
WARDEN_VARNISH=0
WARDEN_RABBITMQ=1
WARDEN_REDIS=1

ELASTICSEARCH_VERSION=7.16
MARIADB_VERSION=10.4
NODE_VERSION=10
PHP_VERSION=8.1
PHP_XDEBUG_3=1
COMPOSER_VERSION=2
RABBITMQ_VERSION=3.9
REDIS_VERSION=6.2
VARNISH_VERSION=7.0

MYSQL_USER=magento
MYSQL_PASSWORD=magento
MYSQL_DATABASE=magento

WARDEN_SYNC_IGNORE=

WARDEN_ALLURE=0
WARDEN_SELENIUM=0
WARDEN_SELENIUM_DEBUG=0
WARDEN_BLACKFIRE=0
WARDEN_SPLIT_SALES=0
WARDEN_SPLIT_CHECKOUT=0
WARDEN_TEST_DB=0
WARDEN_MAGEPACK=0

BLACKFIRE_CLIENT_ID=
BLACKFIRE_CLIENT_TOKEN=
BLACKFIRE_SERVER_ID=
BLACKFIRE_SERVER_TOKEN=

More details about the configuration above was shared in the step 5 of Configuring a single website/store Magento 2 project

6 Define custom variables for the store domains

We will define custom variables in ~/Sites/magento244-multisite/docker/tools/common.sh to be able to reuse them later in our set-default-configs.sh file:

URL_WEBSITE_TWO_FRONT=https://app.magento244-website-two.test/
URL_STORE_VIEW_THREE_US_FRONT=https://app.magento244-store-view-three-us.test/
URL_STORE_VIEW_THREE_DE_FRONT=https://app.magento244-store-view-three-de.test/

7 Updates the the defaults config script

Add the content of this file to the ~/Sites/magento244-multisite/docker/tools/set-default-configs.sh file.

The goal of the set-default-configs.sh script was explained here.

8 Run the docker environment

Inside the ~/Sites/magento244-multisite/docker directory type:

make docker-up

9 Composer install

Inside the ~/Sites/magento244-multisite/docker run the command below to access the PHP container

warden shell

Run the composer install command

composer install

10 Copy and rename the `env.warden.php` file into the etc folder

cp ~/Sites/magento244-multisite/code-base/app/etc/env.warden.php ~/Sites/magento244-multisite/code-base/app/etc/env.php

11 Import the Magento 2 multisite database

There is a sample database inside our Magento project, so copy the ~/Sites/magento244-multisite/code-base/dump.sql.gz file to ~/Sites/magento244-multisite/docker/backfill/magento-db.sql.gz with the command below:

cp ~/Sites/magento244-multisite/code-base/dump.sql.gz ~/Sites/magento244-multisite/docker/backfill/magento-db.sql.gz

Open a new tab in your terminal and enter the directory ~/Sites/magento244-multisite/docker and type the command below:

make DB-import

12 Sign the SSL certificates for all domains

Still, type the commands below inside the ~/Sites/magento244-multisite/docker directory.

To sign the certificate to the main and the admin store:

warden sign-certificate app.magento244-multisite.test

To sign the certificate to the website-two:

warden sign-certificate app.magento244-website-two.test

To sign the certificate to the Store View Three US:

warden sign-certificate app.magento244-store-view-three-us.test

To sign the certificate to the Store View Three DE:

warden sign-certificate app.magento244-store-view-three-de.test

13 Append your `/etc/hosts` file with the store domains

127.0.0.1 app.magento244-multisite.test
127.0.0.1 app.magento244-website-two.test
127.0.0.1 app.magento244-store-view-three-us.test
127.0.0.1 app.magento244-store-view-three-de.test

Note Do the step above only if dnsmasq is not being used.

14 Configuring the application domains:

According to the Warden documentation we have to create a warden-env.yml file inside the .warden directory. Type the commands below:

Create the .warden-env.yml file

touch ~/Sites/magento244-multisite/docker/.warden/warden-env.yml

Put the content below to the .warden-dev.yml file:

version: "3.5"
services:
  nginx:
    labels:
      - traefik.http.routers.${WARDEN_ENV_NAME}-nginx.rule=
        HostRegexp(`{subdomain:.+}.${TRAEFIK_DOMAIN}`)
        || Host(`${TRAEFIK_DOMAIN}`)
        || HostRegexp(`{subdomain:.+}.magento244-multisite.test`)
        || Host(`magento244-multisite.test`)
        || HostRegexp(`{subdomain:.+}.magento244-website-two.test`)
        || Host(`magento244-website-two.test`)
        || HostRegexp(`{subdomain:.+}.magento244-store-view-three-us.test`)
        || Host(`magento244-store-view-three-us.test`)
        || HostRegexp(`{subdomain:.+}.magento244-store-view-three-de.test`)
        || Host(`magento244-store-view-three-de.test`)

You can have a copy of this file here.

When you're done with the settings, look at the traefik address to see where this mapping ended up. It will be like as:

Create stores.php the Add the Magento 2 run params to the project:

cd touch ~/Sites/magento244-multisite/code-base/app/etc/stores.php

Add the content of this file in the file above.

Add the app/etc/stores.php to the composer.json file.

It should look like the image below after the change:

It will force composer to load this file on every request. the file then sets up the store for Magento to load based on the domain, passing the MAGE_RUN_CODE and MAGE_RUN_TYPE variables to Nginx.

Now you're able to access the stores with the URLS \o/:

Default Store Address:
https://app.magento244-multisite.test

Backend admin panel:
https://app.magento244-multisite.test/backend

Website Two Address:
https://app.magento244-multisite.test

Store View Three US Address:
https://app.magento244-store-view-three-us.test

Store View Three DE Address:
https://app.magento244-store-view-three-de.test

Note:
Inside the directory ~/Sites/magento244-multisite/code-base, type the command make, and you will see an output like:

That's it. If you have any questions, please let us know in the comments 👋

Forem: Cristiano Pacheco

Good-bye Docker, Hello Nix: Configuring a Magento 2 Development Environment with Rooter

Table of Contents

Introduction:

Requirements

Creating a Magento 2 dev environment

1. Creating the project directory

2. Entering in the created directory

3. Creating the environment with rooter

4. Allowing direnv to execute

5. Cloning the Magento 2 repository

6. Starting the environment stack

7. Setting up the database

Rooter Appendix

The .devenv directory

Versioning

Diving deeper into the most important files

.env file:

.envrc file:

devenv.nix file:

devenv.yaml file:

Accessing the traefik dashboard

Accessing the mailpit UI

Rooter commands

Xdebug

Preventing Transaction Loss: Unleashing the Power of Resilient Transactions with RabbitMQ dead-letter Exchanges in Magento 2

Table of Contents

What is RabbitMQ?

What is a dead-letter exchange?

Examples of transaction loss scenarios:

Practical case scenario

Crafting an Effective Retry Strategy for Queue Messages

Practical case scenario with queue retry

Configuring a docker environment for a Magento 2 project

Table of contents

What is Warden?

Why Warden?

Quickstart

Requirements

Project Stack:

Configuring a single website/store Magento 2 project

1 Creating the project folder

2 Enter the created directory

3 Clone the Magento 2 code-base into the code-base folder

4 Clone the Warden Magento 2 template

5 Update the warden .env file

A brief explanation of the keys in the .env file:

6 Run the docker environment

7 Run composer install

8 Copy and rename the env.warden.php file

9 Import the Magento 2 database

An Example of the content of this set-default-config.sh script:

10 Sign the SSL certificate

11 Update your /etc/hosts file with the store domain:

12 Installing the sample data (Optional)

Configuring a multiple website/store view Magento 2 project

1 Create a project folder

2 Enter the created multisite directory

3 Clone the Magento 2 code-base into the multisite code-base folder

4 Clone the Warden Magento 2 template

5 Change the warden .env file

6 Define custom variables for the store domains

7 Updates the the defaults config script

8 Run the docker environment

9 Composer install

10 Copy and rename the env.warden.php file into the etc folder

11 Import the Magento 2 multisite database

12 Sign the SSL certificates for all domains

13 Append your /etc/hosts file with the store domains

14 Configuring the application domains:

3. Creating the environment with `rooter`

The `.devenv` directory

`.env` file:

`.envrc` file:

`devenv.nix` file:

`devenv.yaml` file:

Accessing the `traefik` dashboard

Accessing the `mailpit` UI

8 Copy and rename the `env.warden.php` file

An Example of the content of this `set-default-config.sh` script:

11 Update your `/etc/hosts` file with the store domain:

10 Copy and rename the `env.warden.php` file into the etc folder

13 Append your `/etc/hosts` file with the store domains