Forem: boolfalse

Setup Custom Email with Cloudflare and Mailgun

boolfalse — Tue, 16 Apr 2024 09:10:15 +0000

How to Set Up a Custom Email with Cloudflare and Mailgun

As a software engineer, you may want to consider having a professional email account along with your own website like "info@example.com". But you may realized that it must cost a certain amount that you are not willing to pay. How would you behave if you knew you could do it for free? There is actually a way to do it, and besides the fact that having a professional email account is free, it will help you be more efficient, reliable and secure in your daily work.

In this article, you'll learn how to create and set up your own email address using Cloudflare and Mailgun to manage emails in Gmail. It means you can send and receive emails directly on your Gmail inbox.
I've already done this for personal use and have taken screenshots of the entire process to show it in this article. So I'll share here all the necessary steps you need to follow to set up your own email.

The original article initially was published on freeCodeCamp.

Introduction

Let's figure out what you need to have before you start, what you are going to do and how it will work.

What you need to have before you start:

At first, we assume, you already have a domain, let's call it "yourdomain.com", on which you have control. Specifically, you need to have accessibility to connect your domain with Cloudflare and setup DNS records there. A classic example of that is having a domain on some domain registrar (like GoDaddy, Namecheap), and adding your domain to Cloudflare by setting DNS records provided by Cloudflare on your domain registrar account.

Adding a domain to Cloudflare involves updating your domain's DNS nameservers to point to Cloudflare's nameservers. Once the domain is added, Cloudflare acts as an intermediary for web traffic, providing security features like DDoS protection, firewall, and SSL encryption, as well as performance enhancements through caching and content optimization.
If you haven't done that yet, here's the official video on YouTube on how to connect your domain to Cloudflare.

Additionally, Cloudflare manages DNS records for your domain, allowing you to control how traffic is routed and ensuring reliable delivery of services like email.
So, our work in this article will be focusing exactly on that, on how to setup your domain on Cloudflare Email.
Cloudflare Email is one of the services of Cloudflare since 2021, which can be used for free (as for now at least).

The second assumption is that you have account at Gmail, and you have access to its email settings. Simply, if you have just a regular "youremail@gmail.com" email, which isn't under control of any administrator or kind of that, then you're have nothing to worry about for this. We'll explore and work on email settings later on.

What you are going to do:

In simple words, you're going to create a custom email like "something@yourdomain.com", which you can use to send and receive emails from that email by using Gmail's platform. So you will be able receiving and reading emails sent to "something@yourdomain.com" in Gmail, as well as sending emails from that custom email using Gmail.
For all of that you'll use Cloudflare Email for the email routing, and Mailgun's SMTP server for sending emails.

How it will work technically:

When composing an email from Gmail with the sender set as "something@yourdomain.com", Gmail utilizes Mailgun's SMTP server through the provided credentials, transmitting the email. Mailgun then processes the message and forwards it to the recipient's email server, likely involving DNS lookups to find the recipient's server.
Emails sent to "something@yourdomain.com" are received by Cloudflare's email servers, configured via MX records in the domain's DNS settings. Cloudflare stores the received emails in the associated account, accessible through Gmail, which periodically connects to Cloudflare's servers (using IMAP or POP3 protocols) to retrieve new messages, enabling seamless access to incoming emails.

Email Routing on Cloudflare

Cloudflare Email Routing is designed to simplify the way you create and manage email addresses, without needing to keep an eye on additional mailboxes. With Email Routing, you can create any number of custom email addresses to use in situations where you do not want to share your primary email address, such as when you subscribe to a new service or newsletter. Emails are then routed to your preferred email inbox, without you ever having to expose your primary email address. (Docs)

Sign in to your Cloudflare account and navigate to Dashboard.
Choose and click to the desired website. For me it's "boolfalse.com", as I want to create a custom email like "email@boolfalse.com".

Navigate to the "Email Routing" for the selected website.

If you don't have email routing configured, you may see something similar to the screenshot above. Click "Get started". You may be able to create your own address to receive emails and take action.
We'll skip this without creating our own address for doing it manually.

By default, email routing is disabled, so you need to enable it. Click the link to navigate to the "Email Routing" page.

Submit it by clicking "Enable Email Routing".

For get it done, you need to have three MX and one TXT records:

Type: MX; Name: @; Mail Server: route1.mx.cloudflare.net; TTL: Auto; Priority: 69
Type: MX; Name: @; Mail Server: route2.mx.cloudflare.net; TTL: Auto; Priority: 99
Type: MX; Name: @; Mail Server: route3.mx.cloudflare.net; TTL: Auto; Priority: 40
Type: TXT; Name: @; TTL: Auto; Content: v=spf1 include:_spf.mx.cloudflare.net ~all

You can see them at the bottom of the "Email Routing" page.

So, as already said, in the left menu, go to "DNS" -> "Records" and add the following records there.

After creating these records, go to the "Email Routing" page again.

There you only need to have the records you just created.
So, if you have any other records, just delete them.
For example, I already had an unnecessary entry there that I should delete.

Submit to delete existing unnecessary records.

After removing unnecessary DNS records, you will see only the ones you need there.
You will now be able to enable email routing by clicking the "Add records and enable" button.

After enabling it you'll see something like this:

Creating Custom Email on Cloudflare

Now go to the "Routes" tab and create an email by clicking the "Create address" button.

In this example, we'll create "email@boolfalse.com" email address, by adding "email" as a custom address, and a destination email address, where I'll be able to receive emails.

You'll see a notification about that.

You will also get an email for confirming this action.

Just verify the email address.

Once you've verified the email address, you may get this page:

You probably will also get an email that you've verified your domain with Mailgun:

Receiving Emails into the Custom Email

Now, your email address is activated, and you can see that here:

At this point you already can send emails to the custom email you just set up. In this case, it's "email@boolfalse.com".
Below is a test email sent from a different email.

You'll receive a test email to the custom email.

Mailgun: Adding New Domain

You can now successfully receive emails, but you can't send emails from that custom email yet.

So, it's time to switch to the Mail Service provider. In our case it will be Mailgun.
To do this, you just need to register and attach the card to your Mailgun account. After activating your account with the card attached, you can set up a domain for your email.

You don't have to worry about the card, because Mailgun does not charge for limited quantities. I think the amount it gives is quite suitable for a free package.
You can find the price packages in detail here.

Go to "Sending" -> "Domains" page, and click the "Add New Domain" button.

In our case it will be "mg.boolfalse.com", as Mailgun recommends to use like that to be able to send emails from your root domain, e.g. "email@boolfalse.com".
You can see that recommendation on the right in below image:

You can also select the domain region and DCIM key length, but you can leave everything as default.
I will leave DCIM key lenght as 1024 and "US" as a domain region.

After creating the domain, you may be shown some tips on how to verify your domain.

Mailgun will give you two TXT records, two MX records and one CNAME record to add to your provider.

Type: TXT; Name: mailto._domainkey.mg.boolfalse.com; TTL: Auto; Content: SECRET
Type: TXT; Name: mg.boolfalse.com; TTL: Auto; Content: v=spf1 include:mailgun.org ~all
Type: MX; Name: mg.boolfalse.com; Mail Server: mxa.mailgun.org; TTL: Auto; Priority: 10
Type: MX; Name mg.boolfalse.com; Mail Server: mxb.mailgun.org; TTL: Auto; Priority: 10
Type: CNAME; Name: email; Target: mailgun.org; TTL: Auto; Proxy Status: On

In our case, we will add them to Cloudflare.

Below is the first TXT record:

Below is the second TXT record:

Below is the first MX record:

Below is the second MX record:

After you've added two TXT and two MX records, you can check and verify them by clicking the "Verify DNS Records" button.

Lastly, add CNAME record.

You may see a warning icon on the left of the CNAME record. You don't need to worry about that, cause what official documentation says about it:

If you recently added your domain to Cloudflare - meaning that your zone is in a pending state - you can often ignore this warning.
Once most domains becomes Active, Cloudflare will automatically issue a Universal SSL certificate, which will provide SSL/TLS coverage and remove the warning message.

After adding a CNAME record, you can check and verify it again by clicking the second "Verify DNS Records" button.

If you have added all 5 records on the Cloudflare successfully, after clicking the verifying button, Mailgun will automatically redirect you to the "Overview" page.

It means you're ready to add a Sending API key on Mailgun.

Mailgun: Sending API key & SMPT User

Go to "Sending" -> "Domain Settings" page. Choose the "Sending API keys" tab at the top. Probably you won't see any API keys there. You just need to create a new Sending API key. Click "Add sending key" from the top right corner, and in the popup fill the name of the key you about to create.

After pressing "Create sending key", you'll get the secret API key that you need to copy and save somewhere safe. After saving the key, you can just close the popup.
You'll see the created key listed:

You also need to create a new SMTP user in Mailgun dashbaord.
Go to "Sending" -> "Domain Settings" page. Choose the "SMTP credentials" tab at the top and press "Add new SMTP user" button on the top left corner. It will open up a popup. Type user credentials there. In our case I'll create a user with the name "email". It will be like a login for your email on Gmail.

Once you create an SMTP user in Mailgun, you'll see it listed and a password for that user will be generated automatically. To get this password, copy it by clicking the "Copy" button in the pop-up notification in the lower right corner.

Keep this in a safe place for future use. You will need this login and password to authenticate on Gmail.

You are now ready to set up email configurations with your email provider. In our case, we will do this in Gmail.
Open your Gmail account in your desktop browser and go to Settings by clicking the settings icon in the top right corner and click the "See all settings" button.

Gmail Authentication with Mailgun SMTP Server

In the Gmail settings page choose the "Accounts and Import" tab and click on the "Add another email address" from the "Send mail as" section:

It will open a popup for the authentication. Use the login and the password you just got by creating an SMTP user on Mailgun. Make sure to fill out the credentials correctly.

Submit the form by clicking the "Add Account" button.
It probably will ask you to save the username/password in your browser. It's up to you.
And the last important thing here, that it will ask you to verify adding an account.

For the verification, the confirmation email will be sent to your primary email.

You can either use the confirmation code to verify it using the pop-up window or simply follow the link provided in the confirmation email.
In this case, we'll click on a link that will open the page, where will be asked to confirm. Click on "Confirm" and simply close the previously opened pop-up window without worrying.

Now you're ready to send and receive emails from the custom email you just created.

For sending an email from the custom email, you just need to choose that email as a sender email:

That's it!

An additional thing that may be useful to you is that you can set the custom email address you just created as the default address for sending emails from Gmail.
You can set this on the settings page in the "Send mail as" section:

I hope this guide will be a good resource for setting up your custom email.

Conclusion

In this article, you learned how to set up your own email to manage emails in Gmail using Cloudflare Email and Mailgun.
In conclusion, it is worth noting that this choice of tools is not mandatory, other tools could be used instead, but the basic idea and logic would be similar.

You can check out my website at: boolfalse.com

Feel free to share this article. 😇

Real-Time Chat App with Laravel Reverb

boolfalse — Thu, 04 Apr 2024 20:33:53 +0000

The original article initially was published on freeCodeCamp.

For those who are curious: the full code for this article is on GitHub ⭐․

Introduction

In March of 2024, Laravel 11 was released. And with it arrived a new family member in the Laravel ecosystem: Laravel Reverb.

Reverb is a separate open-source package that's a first-party WebSocket server for Laravel applications. It helps facilitate real-time communication between client and server.

Before this new package, Laravel had event broadcasting, but basically it didn't have a built-in way to set up a self-hosted WebSocket server. Fortunately, Reverb now gives us that option.

Laravel Reverb has a few key features: it's written in PHP, it's fast, and it's scalable. It was developed in particular to be horizontally scalable.

Reverb basically allows you to run an application on a single server – but if the application starts to outgrow that server, you can add multiple additional servers. Then those servers can all communicate with each other via Redis Pub/Sub to distribute the messages across each other.

In this article, you will learn how to build a real-time chat application using Laravel Reverb. This will let you easily implement WebSocket communications between your backend and frontend.

For a frontend technology, you can use anything you want – but in this case, we'll use React.js with the Vite.js build tool.

By the end of this article, you'll have a full-stack, real-time app on your local machine, which will work like this:

Demo of the app showing messaging between two logged-in users

Prerequisites

You'll need the following tools for the app that we'll build in this article:

PHP 8.2 or above (run php -v to check the version)
Composer (run composer to check that it exists)
Node.js 20 or above (run node -v to check the version)
MySQL 5.7 or above (run mysql --version to check if it exists, or follow the docs)

General Steps

The main steps in this article will be:

Installing Laravel 11.
Adding authentication flow to it (authentication scaffolding). Laravel provides a basic starting point for this using Bootstrap with React / Vue.
Installing Reverb.
React.js components and event listening in the frontend.

How to Install Laravel

To start, install Laravel 11 by using the composer command:



composer create-project laravel/laravel:^11.0 laravel-reverb-react-chat && cd laravel-reverb-react-chat/

At this point, you can check out the app by running the serve command:



php artisan serve

How to Create the Model and Migration

You can generate a model and a migration for the messages by using this single command:



php artisan make:model -m Message

Then you'll need to set up the Message's model with the following code:



<?php

namespace App\Models;

use Illuminate\Database\Eloquent\Factories\HasFactory;
use Illuminate\Database\Eloquent\Model;
use Illuminate\Database\Eloquent\Relations\BelongsTo;

class Message extends Model
{
    use HasFactory;

    public $table = 'messages';
    protected $fillable = ['id', 'user_id', 'text'];

    public function user(): BelongsTo {
        return $this->belongsTo(User::class, 'user_id');
    }

    public function getTimeAttribute(): string {
        return date(
            "d M Y, H:i:s",
            strtotime($this->attributes['created_at'])
        );
    }
}

app/Models/Message.php

As you can see, there's a getTimeAttribute() accessor that will format the message creation timestamp into a human-readable date and time format. It will show it on the top of each message in the chat box.

Next, set up the migration for the messages database table with this code:



<?php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

return new class extends Migration
{
    public function up(): void {
        Schema::create('messages', function (Blueprint $table) {
            $table->id();
            $table->foreignId('user_id')->constrained();
            $table->text('text')->nullable();
            $table->timestamps();
        });
    }

    public function down(): void {
        Schema::dropIfExists('messages');
    }
};

database/migrations/2024_03_25_000831_create_messages_table.php

This migration creates a messages table in the database. The table contains columns for an auto-incrementing primary key (id), a foreign key (user_id) referencing the id column of the users table, a text column for storing the message content, and timestamps to automatically track the creation and modification times of each record.

The migration also includes a rollback method (down()) to drop the messages table if needed.

In this article, we'll use the MySQL database, but you can go with SQLite as the default one if you prefer. Just make sure to set up your database credentials in .env file correctly:



DB_CONNECTION=mysql
DB_HOST=127.0.0.1
DB_PORT=3306
DB_DATABASE=database_name
DB_USERNAME=username
DB_PASSWORD=password

.env

After setting up the environment variables, optimize the cache:



php artisan optimize

Run migrations to recreate the database tables as well as to add the messages table:



php artisan migrate:fresh

How to Add Authentication

Now, you can add authentication scaffolding to your app. You can use Laravel's UI package to import some asset files. First, you'll need to install the appropriate package:



composer require laravel/ui

Then import the React-related assets into the application:



php artisan ui react --auth

It may ask to overwrite the app/Http/Controllers/Controller.php, and you can go ahead and allow it:



The [Controller.php] file already exists. Do you want to replace it? (yes/no) [no]

This will do all of the authentication scaffolding compiled and installed, including routes, controllers, views, vite configurations, and a simple React-specific sample.
At this point, you're just one step away from the app being ready to go.

NOTE: Make sure you have Node.js (with npm) version 20 or above installed. You can check that by running the node -v command. Otherwise, just go ahead and install it using the official page.



npm install && npm run build

The command above will install NPM packages and build frontend assets. Now you can start the Laravel application and check out your fully ready app sample:



php artisan optimize && php artisan serve

A screenshot of the Register page

It's also important to note that you can separately run the dev command instead of using build every time when you're making changes to frontend files:



npm run dev

See the details in the package.json file, in the scripts field.

How to Set Up Routes

In this real-time chat app, you'll need to have a few routes:

home for the home page (already should be added)
message for adding a new message
messages to get all the existing messages

You'll have these kinds of routes in the web.php file:



<?php

use Illuminate\Support\Facades\Auth;
use Illuminate\Support\Facades\Route;
use App\Http\Controllers\HomeController;

Route::get('/', function () { return view('welcome'); });

Auth::routes();

Route::get('/home', [HomeController::class, 'index'])
    ->name('home');
Route::get('/messages', [HomeController::class, 'messages'])
    ->name('messages');
Route::post('/message', [HomeController::class, 'message'])
    ->name('message');

routes/web.php

After setting up those routes, let's use Laravel Events and Queue Jobs advantages.

How to Set Up a Laravel Event

You need to create a GotMessage event for listening for a specific event:



php artisan make:event GotMessage

Laravel's events provide a simple observer pattern implementation, allowing you to subscribe and listen for various events that occur within your application. Event classes are typically stored in the app/Events directory. (Docs)

Set up a private WebSocket channel in the broadcastOn method for all the authenticated users to receive messages in real time. In this case, we will call it "channel_for_everyone", but you can also make it dynamic, depending on the user, like "App.Models.User.{$this->message['user_id']}".



<?php

namespace App\Events;

use Illuminate\Broadcasting\InteractsWithSockets;
use Illuminate\Broadcasting\PrivateChannel;
use Illuminate\Contracts\Broadcasting\ShouldBroadcast;
use Illuminate\Foundation\Events\Dispatchable;
use Illuminate\Queue\SerializesModels;

class GotMessage implements ShouldBroadcast
{
    use Dispatchable, InteractsWithSockets, SerializesModels;

    public function __construct(public array $message) {
        //
    }

    public function broadcastOn(): array {
        // $this->message is available here
        return [
            new PrivateChannel("channel_for_everyone"),
        ];
    }
}

app/Events/GotMessage.php

As you can see, there's a public $massage property as a constructor argument, so you can get message information in the front end.

We've already used the channel name in the channels file, and we'll use it in the front end as well for real-time message updates.

Don't forget to implement the ShouldBroadcast interface in the event's class.

How to Set Up a Laravel Queue Job

Now it's time to create the SendMessage job for sending messages:



php artisan make:job SendMessage

Laravel allows you to easily create queued jobs that may be processed in the background. By moving time intensive tasks to a queue, your application can respond to web requests with blazing speed and provide a better user experience to your customers. (Docs)



<?php

namespace App\Jobs;

use App\Events\GotMessage;
use App\Models\Message;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;

class SendMessage implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public function __construct(public Message $message) {
        //
    }

    public function handle(): void {
        GotMessage::dispatch([
            'id' => $this->message->id,
            'user_id' => $this->message->user_id,
            'text' => $this->message->text,
            'time' => $this->message->time,
        ]);
    }
}

app/Jobs/SendMessage.php

The SendMessage.php queue job is responsible for dispatching the GotMessage event with information about a newly sent message. It receives a Message object upon construction, representing the message to be sent.

In its handle() method, it dispatches the GotMessage event with details such as the message ID, user ID, text, and timestamp. This job is designed to be queued for asynchronous processing, enabling efficient handling of message sending tasks in the background.

As you can see, there's a public $massage property as a constructor argument, which we'll use to attach message information to the queue job.

How to Write the Controller Methods

For the defined routes, here are the appropriate controller methods:



<?php

namespace App\Http\Controllers;

use App\Jobs\SendMessage;
use App\Models\Message;
use App\Models\User;
use Illuminate\Http\JsonResponse;
use Illuminate\Http\Request;

class HomeController extends Controller
{
    public function __construct() {
        $this->middleware('auth');
    }

    public function index() {
        $user = User::where('id', auth()->id())->select([
            'id', 'name', 'email',
        ])->first();

        return view('home', [
            'user' => $user,
        ]);
    }

    public function messages(): JsonResponse {
        $messages = Message::with('user')->get()->append('time');

        return response()->json($messages);
    }

    public function message(Request $request): JsonResponse {
        $message = Message::create([
            'user_id' => auth()->id(),
            'text' => $request->get('text'),
        ]);
        SendMessage::dispatch($message);

        return response()->json([
            'success' => true,
            'message' => "Message created and job dispatched.",
        ]);
    }
}

app/Http/Controllers/HomeController.php

In the home method, we'll get the logged-in user's data from the database using the User model and send it to the blade view.
In the messages method, we'll retrieve all the messages from the database using the Message model, attach the user relationship data to it, append the time field (accessor) to each item, and send all that to the view.
In the message method, a new message will be created in the database table by using the Message model, and the SendMessage queue job will be dispatched.

How to Install Laravel Reverb

Now we've come to the most important moment: it's time to install Reverb in your Laravel app.

It's so easy. All the necessary packaging and configuration setup can be done using this single command:



php artisan install:broadcasting

It will ask you to install Laravel Reverb as well as install and build the Node dependencies required for broadcasting. Just press enter to continue.

After the command execution, make sure you've automatically added reverb-specific environment variables to the .env file, like:



BROADCAST_CONNECTION=reverb

###

REVERB_APP_ID=795051
REVERB_APP_KEY=s3w3thzezulgp5g0e5bs
REVERB_APP_SECRET=gncsnk3rzpvczdakl6pz
REVERB_HOST="localhost"
REVERB_PORT=8080
REVERB_SCHEME=http

VITE_REVERB_APP_KEY="${REVERB_APP_KEY}"
VITE_REVERB_HOST="${REVERB_HOST}"
VITE_REVERB_PORT="${REVERB_PORT}"
VITE_REVERB_SCHEME="${REVERB_SCHEME}"

.env

You'll also have a new broadcasting.php configuration file in the config directory.

How to Setup WebSocket Channels

Lastly, you'll need to add a channel in the channels.php file. It should already be created after installing Reverb.



<?php

use Illuminate\Support\Facades\Broadcast;

Broadcast::channel('channel_for_everyone', function ($user) {
    return true;
});

routes/channels.php

You'll have only one channel. You can change the channel's name and make it dynamic – it's up to you. In the closure of the channel, we'll always return true, but you can modify it later to make some restrictions regarding the channel's subscription.

Optimize caches one more time:



php artisan optimize

How to Customize Laravel Views

Now your back end should be ready at this point, so you can switch to the front end.

Before working on the React stuff, you'll want to set up Laravel *.blade.php views. In the home blade view, make sure to have the root div with an ID of main to render all the React components there.



@extends('layouts.app')

@section('content')
    <div class="container">
        <div id="main" data-user="{{ json_encode($user) }}"></div>
    </div>
@endsection

resources/views/home.blade.php

The div with ID of main gets a data property for holding the $user info sent from the controller's home method.

I won't put the whole resources/views/welcome.blade.php content here, but you can just make the following small changes to it:

Replace url('/dashboard') with url('/home');
Replace Dashboard with Home;
Remove main and footer sections.

Let's Work on the Front End

In Reverb, event broadcasting is done by a server-side broadcasting driver that broadcasts your Laravel events so that the front end can receive them within the browser client.

In the front end, Laravel Echo does that job under the hood. Echo is a JavaScript library that makes it painless to subscribe to channels and listen for events broadcast by your server-side broadcasting driver.

You can find the WebSocket configurations setup with Echo in the resources/js/echo.js file, but you don't need to do anything there for this project.

Let's create a few React components so that we have a refactored and more readable project.

Create a Main.jsx component in the new components folder:



import React from 'react';
import ReactDOM from 'react-dom/client';
import '../../css/app.css';
import ChatBox from "./ChatBox.jsx";

if (document.getElementById('main')) {
    const rootUrl = "http://127.0.0.1:8000";

    ReactDOM.createRoot(document.getElementById('main')).render(
        <React.StrictMode>
            <ChatBox rootUrl={rootUrl} />
        </React.StrictMode>
    );
}

resources/js/components/Main.jsx

Here we'll check if there's an element with the id 'main'. If it exists, it proceeds with rendering the React application.

As you can see, there's a ChatBox component. We'll learn more about it soon.

Remove the resources/js/components/Example.jsx file, and import the Main.jsx component in the app.js:



import './bootstrap';
import './components/Main.jsx';

resources/js/app.js

Create Message.jsx and MessageInput.jsx files, so you can use them in the ChatBox component.

The Message component will get userId and message arguments (fields) to show each message in the chat box.



import React from "react";

const Message = ({ userId, message }) => {
    return (
        <div className={`row ${
        userId === message.user_id ? "justify-content-end" : ""
        }`}>
            <div className="col-md-6">
        <small className="text-muted">
                    <strong>{message.user.name} | </strong>
                </small>
                <small className="text-muted float-right">
                    {message.time}
                </small>
                <div className={`alert alert-${
                userId === message.user_id ? "primary" : "secondary"
                }`} role="alert">
                    {message.text}
                </div>
            </div>
        </div>
    );
};

export default Message;

resources/js/components/Message.jsx

The Message.jsx component renders individual messages within the chat interface. It receives the userId and message props. Based on whether the message sender matches the current user, it aligns the message to the appropriate side of the screen.

Each message includes the sender's name, timestamp, and the message content itself, styled differently based on whether the message is sent by the current user or another user.

The MessageInput component will care about creating a new message:



import React, { useState } from "react";

const MessageInput = ({ rootUrl }) => {
    const [message, setMessage] = useState("");

    const messageRequest = async (text) => {
        try {
            await axios.post(`${rootUrl}/message`, {
                text,
            });
        } catch (err) {
            console.log(err.message);
        }
    };

    const sendMessage = (e) => {
        e.preventDefault();
        if (message.trim() === "") {
            alert("Please enter a message!");
            return;
        }

        messageRequest(message);
        setMessage("");
    };

    return (
        <div className="input-group">
            <input onChange={(e) => setMessage(e.target.value)}
                   autoComplete="off"
                   type="text"
                   className="form-control"
                   placeholder="Message..."
                   value={message}
            />
            <div className="input-group-append">
                <button onClick={(e) => sendMessage(e)}
                        className="btn btn-primary"
                        type="button">Send</button>
            </div>
        </div>
    );
};

export default MessageInput;

resources/js/components/MessageInput.jsx

The MessageInput component provides a form input field for users to type messages and send them in the chat interface. By clicking the button, it triggers a function to send the message to the server via an Axios POST request to the specified rootUrl that it got from the parent ChatBox component. It also handles validation to ensure that users cannot send empty messages. You can customize it later if you want.

Now create a ChatBox.jsx component to have the front end ready:



import React, { useEffect, useRef, useState } from "react";
import Message from "./Message.jsx";
import MessageInput from "./MessageInput.jsx";

const ChatBox = ({ rootUrl }) => {
    const userData = document.getElementById('main')
        .getAttribute('data-user');

    const user = JSON.parse(userData);
    // `App.Models.User.${user.id}`;
    const webSocketChannel = `channel_for_everyone`;

    const [messages, setMessages] = useState([]);
    const scroll = useRef();

    const scrollToBottom = () => {
        scroll.current.scrollIntoView({ behavior: "smooth" });
    };

    const connectWebSocket = () => {
        window.Echo.private(webSocketChannel)
            .listen('GotMessage', async (e) => {
                // e.message
                await getMessages();
            });
    }

    const getMessages = async () => {
        try {
            const m = await axios.get(`${rootUrl}/messages`);
            setMessages(m.data);
            setTimeout(scrollToBottom, 0);
        } catch (err) {
            console.log(err.message);
        }
    };

    useEffect(() => {
        getMessages();
        connectWebSocket();

        return () => {
            window.Echo.leave(webSocketChannel);
        }
    }, []);

    return (
        <div className="row justify-content-center">
            <div className="col-md-8">
                <div className="card">
                    <div className="card-header">Chat Box</div>
                    <div className="card-body"
                         style={{height: "500px", overflowY: "auto"}}>
                        {
                            messages?.map((message) => (
                                <Message key={message.id}
                                         userId={user.id}
                                         message={message}
                                />
                            ))
                        }
                        <span ref={scroll}></span>
                    </div>
                    <div className="card-footer">
                        <MessageInput rootUrl={rootUrl} />
                    </div>
                </div>
            </div>
        </div>
    );
};

export default ChatBox;

resources/js/components/ChatBox.jsx

The ChatBox component manages a chat interface within the application. It fetches and displays messages from a server using WebSocket and HTTP requests.

The component renders a list of messages, a message input field, and automatically scrolls to the bottom when new messages arrive.

It defines a WebSocket channel for real-time message updates. You need to set up that channel by using the same name as it was written in the routes/channels.php and in the app/Events/GotMessage.php queue job.

Also, the leave() function is called within the useEffect cleanup function to unsubscribe from the WebSocket channel when the component unmounts. This prevents memory leaks and unnecessary network connections by stopping the component from listening to updates on the WebSocket channel after it's no longer needed.

Running the Application

Now, everything's ready, and it's time to check out the app. Follow these instructions:

A screenshot from the terminal with all the necessary commands

Build frontend assets (this is not a "forever" running command): npm run build
Start listening to the Laravel events: php artisan queue:listen
Start the WebSocket server: php artisan reverb:start
Start the server (you may use an alternative for your app like a local running server): php artisan serve

After all the necessary commands are running, you can check out the app by visiting the default URL: http://127.0.0.1:8000.

For testing, you can register two different users, have those users log in, send messages from each of them, and see the chat box.

Useful Reverb Resources

Now that we've reached the end of this article, it's worth listing some useful resources about Reverb:

Laravel Broadcasting (official documentation)
Joe Dixon - Real-Time Laravel (talk on Laracon EU 2024)
Taylor Otwell - Laravel Update (talk on Laracon EU 2024)
Joe Dixon on X (creator of Reverb)
Laracasts episode (practical example with Reverb)

Conclusion

Now you know how to build real-time applications with Laravel Reverb in the new version of Laravel. With this, you can implement WebSocket communications in your full-stack app and avoid using any additional 3rd-party services (like Pusher and Socket.io).

If you want to have a clear idea of how to integrate React.js into your Laravel app without using any additional Laravel tools (like Inertia), you can read through my freeCodeCamp article, where you can build a single-page, full-stack Tasklist app.

The complete code for this article is here on my GitHub⭐, where I actively publicize much of my work about various modern technologies.

For more information, you can visit my website: boolfalse.com

Feel free to share this article. 😇

Use React.js with Laravel. Build a Tasklist app

boolfalse — Thu, 04 Apr 2024 01:11:41 +0000

The original article initially was published on freeCodeCamp.

After doing some changes, I decided to have this on Medium as well.

For those who are curious: the full code for this article is on GitHub ⭐․

Introduction

You may have seen tutorials that help you build a simple React.js app that use some third-party API or a Node.js server as a backend. You could also use Laravel for this purpose and integrate it with React.

As a backend framework, Laravel actually offers a tool to help you do this, called Inertia. Here's what the docs say about it:

It bridges the gap between your Laravel application and your modern Vue or React frontend, allowing you to build full-fledged, modern frontends using Vue or React while leveraging Laravel routes and controllers for routing, data hydration, and authentication — all within a single code repository.

But what if you don't want to use such a tool? And instead, you just want to use React.js as a frontend library and have a simple Laravel-powered backend?

Well, in this article, you will learn how to use React.js with Laravel as a backend by building a draggable tasklist app.

For this full-stack single-page app, you'll use Vite.js as your frontend build tool and the react-beautiful-dnd package for draggable items.

By the end of this article, you will have a single-page app for managing tasks, which will look like this:

[Captured from a local working project]

In this article, we'll create a dynamic page that will have a list of tasks, each of which will belong to a specific project. This way, the user will be able to select a project, and only the tasks of the selected project will be shown on the page. The user can also create a new task for the current project, as well as edit, delete and reorder tasks by dragging and dropping them.

Prerequisites

Before following along, it would be helpful to have a basic understanding of React.js, Laravel, and familiarity with fundamental web development concepts.

You'll need the following tools for the app that we'll build in this article:

PHP 8.1 or above Run php -v to check the version.
Composer Run composer to check that it exists.
Node.js 18 or above Run node -v to check the version.
MySQL 5.7 or above Run mysql --version to check if it exists, or follow the docs.

Additional (optional) tools that you can use:

Postman — a program with a UI for testing the API routes

We'll start by building out the backend, and then move to the frontend.

The Backend: Install Laravel

First, if you don't have it already, you'll need to install the Laravel framework on your local machine.

One way to install Laravel is by using Composer:



composer create-project laravel/laravel tasklist

This will install the latest stable version of Laravel in your local machine (currently it's version 10).

At this point, you can cd into the project's folder and run the backend app without needing to have a virtual server set up:



cd tasklist/ && php artisan serve

Visit http://127.0.0.1:8000 in your browser to see the default page. It should look like this:

[Laravel welcome page]

Create Models and Migrations

Now, let's create Project and Task models, as well as migrations for them.

You can create model and migration files manually as well as generate them using the artisan command:



php artisan make:model Project -m
php artisan make:model Task -m

The -m argument will automatically generate a migration file using the provided model name.

Keep the command execution sequence as it is, so the project's migration later can run before the Task's migration.

This is important, because the projects and tasks tables should have a one-to-many relationship (1-N): each task will refer to a single project, or, in other words, each project can have multiple tasks.

Set the Project model's $fillable fields and the task() relationship method as below:



<?php

namespace App\Models;

use Illuminate\Database\Eloquent\Factories\HasFactory;
use Illuminate\Database\Eloquent\Model;
use Illuminate\Database\Eloquent\Relations\HasMany;

class Project extends Model
{
    use HasFactory;

    protected $table = 'projects';
    public $timestamps = false;
    protected $fillable = [
        'id', // primary key, auto-increment, integer
        'name', // string
    ];

    // a project can have multiple tasks
    public function tasks(): HasMany
    {
        return $this->hasMany(Task::class);
    }
}

By default, the $timestamps public property has a true value, which is coming from the parent Model class. This means that the created_at and updated_at columns in your database table will be maintained automatically by Eloquent. We don't need to have created_at and updated_at fields in the projects table, so we'll set the $timestamps to false.

Set the Task model's $fillable fields, project() relationship method, and created accessor.



<?php

namespace App\Models;

use Illuminate\Database\Eloquent\Factories\HasFactory;
use Illuminate\Database\Eloquent\Model;
use Illuminate\Database\Eloquent\Relations\BelongsTo;
use Illuminate\Database\Eloquent\SoftDeletes;

class Task extends Model
{
    use HasFactory, SoftDeletes;

    protected $table = 'tasks';
    protected $fillable = [
        'id', // primary key, auto-increment, integer
        'project_id', // foreign key, integer

        'priority', // integer
        'title', // string
        'description', // text
    ];
    protected $appends = [
        'created',
    ];

    // each task belongs to a single project
    public function project(): BelongsTo
    {
        return $this->belongsTo(Project::class);
    }

    public function getCreatedAttribute()
    {
        return $this->created_at->diffForHumans();
    }
}

Above, in the Task model, there is an accessor called created. For having an accessor, we have the created field in the $appends array, and also a public function getCreatedAttribute().

Now that the models are ready, let's set up the migrations.

First, set up a migration for the projects table:



<?php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

return new class extends Migration
{
    public function up(): void
    {
        Schema::create('projects', function (Blueprint $table) {
            $table->id();
            $table->string('name');
        });
    }

    public function down(): void
    {
        Schema::dropIfExists('projects');
    }
};

Then set up a migration for the tasks table:



<?php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

return new class extends Migration
{
    public function up(): void
    {
        Schema::create('tasks', function (Blueprint $table) {
            $table->id();

            $table->foreignId('project_id')->nullable()->constrained();

            $table->integer('priority');
            $table->string('title');
            $table->text('description')->nullable();

            $table->timestamps();
            $table->softDeletes();
        });
    }

    public function down(): void
    {
        // drop existing foreign keys
        Schema::table('tasks', function (Blueprint $table) {
            if (Schema::hasColumn('tasks', 'project_id')) {
                $table->dropForeign(['project_id']);
            }
        });

        // drop the table
        Schema::dropIfExists('tasks');
    }
};

The tasks table has a foreign key project_id, which is a reference to the projects table. So it's a good practice to update the down() method too, to be sure that the project_id foreign will be dropped before dropping the actual projects table.

There is also a priority field, which will be a non-nullable natural number for ordering the tasks. And optionally, you can add a soft deletion feature to the Task model.

Create Seeders

Now we need to add dummy data to the projects and tasks tables. To seed some data in the database, you can use Laravel seeders:



php artisan make:seeder ProjectsSeeder
php artisan make:seeder TasksSeeder

At first, you'll need to set up the ProjectsSeeder to add a few projects to the projects table. Then you can set up the TasksSeeder to add tasks to the tasks table.

You can imagine the database structure by looking at the following visuals:

[Generated by PHPStorm IDE]

Using the example below, you can generate 3 projects:



<?php

namespace Database\Seeders;

use App\Models\Project;
use Illuminate\Database\Seeder;

class ProjectsSeeder extends Seeder
{
    public function run(): void
    {
        for ($i = 1; $i <= 3; $i++) {
            Project::create([
                'name' => "Project $i",
            ]);
        }
    }
}

Next, set up TasksSeeder. You'll run all the seeder files after setting them up, and they will run one by one. That being said, at this point your ProjectsSeeder is ready to create a few projects.

The next step will be generating the tasks, each of them will have a reference to one of the already existing projects by its project_id field.

Using the example below, you can generate 10 projects:



<?php

namespace Database\Seeders;

use App\Models\Project;
use App\Models\Task;
use Illuminate\Database\Seeder;

class TasksSeeder extends Seeder
{
    public function run(): void
    {
        $project_ids = Project::all()->pluck('id')->toArray();

        $now = now();
        $tasks = [];
        $project_priorities = [];
        foreach ($project_ids as $project_id) {
            $project_priorities[$project_id] = 0;
        }

        for ($i = 1; $i <= 10; $i++) {
            $project_id = $project_ids[array_rand($project_ids)];
            $project_priorities[$project_id]++;

            $tasks[] = [
                'project_id' => $project_id,
                'priority' => $project_priorities[$project_id],
                'title' => "Task " . $project_priorities[$project_id],
                'description' => "Description for Task " . $project_priorities[$project_id],

                'created_at' => $now,
                'updated_at' => $now,
            ];
        }

        Task::insert($tasks);
    }
}

The above code just grabs all the project IDs, then randomly chooses a project for each task. In the end, it inserts all the tasks into the tasks table.

As you may have noticed, we're inserting $tasks into the tasks table using the insert() static function, which allows us to insert all the items into the database table with a single query.

But it has a downside as well: it doesn't manage created_at and updated_at fields. That's why there's a need to set up those fields manually by assigning them the same $now timestamp.

Now, when you have all the seeders ready, you need to register them into the DatabaseSeeder.



<?php

namespace Database\Seeders;

use Illuminate\Database\Seeder;

class DatabaseSeeder extends Seeder
{
    public function run(): void
    {
        $this->call([
            ProjectsSeeder::class,
            TasksSeeder::class,
        ]);
    }
}

Connect to the MySQL Database

Before running migrations and seeds, create a MySQL database and set up the appropriate credentials in the .env file. If there is not a .env, then create it and paste the .env.example file's content into it.

After setting up the database credentials, you'll have these kinds of environment variables:



DB_CONNECTION=mysql
DB_HOST=127.0.0.1
DB_PORT=3306
DB_DATABASE="<DATABASE_NAME>"
DB_USERNAME="<USERNAME>"
DB_PASSWORD="<PASSWORD>"

After setting up environment variables, optimize the cache:



php artisan optimize

Now you'll be able to create projects and tasks tables in the MySQL database, set up their structure, and add initial records with a single command:



php artisan migrate:fresh --seed

In the above command, the migrate:fresh argument will drop all tables from the database. Then it will execute the migrate command, which will run your migrations to create projects and tasks tables appropriately.

With the --seed argument, it will run ProjectsSeeder and TasksSeeder after the migrations. That being said, it will empty your database for you, and will create all the tables and fill all the necessary dummy data.

After running the command, you'll have these kinds of database records:

[A screenshot from the PHPStorm IDE]

Service Injection

Now let's create a controller and a service classes to manage all the task features, such as listing, creating, updating, deleting, and reordering the tasks.

At first, use the below command to generate a controller.



php artisan make:controller TaskController

In order not to place all the code in the controller, you can keep only the main logic in it, and move the other logic implementations to another class file.

Those classes are generally called "services", and using service implementations in a controller method is called service injection (it comes from the term dependency injection).

One of the main advantages of using services is that it helps you create a maintainable codebase.

You can inject your service class into the controller's construction method as an argument, so after each controller execution (when a controller's __construct() method runs) you can initialize an object of service. This means that you can access your service's functions right in your controller.

Now, let's create two separate service classes, which will be used in the TaskController.

Manually create app/Services/ProjectService.php service class, which will be responsible for the project-related logic.



<?php

namespace App\Services;

use App\Models\Project;
use Illuminate\Database\Eloquent\Collection;

class ProjectService
{
    public function getAll(): Collection
    {
        return Project::all();
    }
}

The second service class will be the app/Services/TaskService.php, which will be responsible for doing task manipulations:



<?php

namespace App\Services;

use App\Models\Task;
use Illuminate\Support\Facades\DB;

class TaskService
{
    public function list(int $projectId)
    {
        return Task::with('project')->where('project_id', $projectId)
            ->orderBy('priority')->get();
    }

    public function getById(int $id)
    {
        return Task::where('id', $id)->with('project')->first();
    }

    public function store($data): void
    {
        $count = Task::where('project_id', $data['project_id'])->count();
        $data['priority'] = $count + 1;

        Task::create($data);
    }

    public function update(int $id, array $data): void
    {
        $task = $this->getById($id);
        if (!$task) { return; }

        $task->update($data);
    }

    public function delete(int $id): void
    {
        $task = $this->getById($id);
        if (!$task) { return; }

        $task->delete();

        $tasks = Task::where('project_id', $task->project_id)
            ->where('priority', '>', $task->priority)->get();
        if ($tasks->isEmpty()) {
            return;
        }

        $when_then = "";
        $where_in = "";
        foreach ($tasks as $task) {
            $when_then .= "WHEN ".$task->id
                ." THEN ".($task->priority - 1)." ";
            $where_in .= $task->id.",";
        }

        $table_name = (new Task())->getTable();
        $bulk_update_query = "UPDATE `".$table_name
            ."` SET `priority` = (CASE `id` ".$when_then."END)"
            ." WHERE `id` IN(".substr($where_in, 0, -1).");";

        // there is no way to be SQL injected here
        // because all the values are not provided by the user
        DB::update($bulk_update_query);
    }

    public function reorder(int $project_id, int $start, int $end): void
    {
        $items = Task::where('project_id', $project_id)
            ->orderBy('priority')->pluck('priority', 'id')->toArray();

        if ($start > count($items) || $end > count($items)) {
            return;
        }

        $ids = [];
        $priorities = [];
        foreach ($items as $id => $priority) {
            $ids[] = $id;
            $priorities[] = $priority;
        }

        $out_priority = array_splice($priorities, $start - 1, 1);
        array_splice($priorities, $end - 1, 0, $out_priority);

        $when_then = "";
        $where_in = "";
        foreach ($priorities as $out_k => $out_v) {
            $id = $ids[$out_v - 1];
            $when_then .= "WHEN ".$id." THEN ".($out_k + 1)." ";
            $where_in .= $id.",";
        }

        $table_name = (new Task())->getTable();
        $bulk_update_query = "UPDATE `".$table_name
            ."` SET `priority` = (CASE `id` ".$when_then."END)"
            ." WHERE `id` IN(".substr($where_in, 0, -1).")"
            ." AND `deleted_at` IS NULL;"; // soft delete

        DB::update($bulk_update_query);
    }
}

In the above TaskService class, you'll use the following functions in the TaskController.

list: fetches tasks for a given project ID, including the related project, and orders them by priority.
getById: retrieves a specific task by its ID, including the related project.
store: stores a new task, calculating the priority based on existing tasks for the same project.
update: updates an existing task by its ID.
delete: deletes a task by its ID and adjusts the priorities of remaining tasks in the same project.
reorder: changes the priorities of tasks within a project, (handles soft delete as well with deleted_at is NULL).

Web and API Routes

Now you can add routes to test the methods you've already written. In this project, we have a stateless app on the frontend which requests API routes for getting JSON data, so it will follow RESTful principles (GET, POST, PUT, DELETE methods). Only the initial HTML page will be retrieved as a whole web page.

So now, set up a route in routes/web.php for the initial single-page:



<?php

use Illuminate\Support\Facades\Route;
use App\Http\Controllers\TaskController;

Route::group(['prefix' => '/', 'as' => 'tasks.'], function () {
    Route::get('/', [TaskController::class, 'index'])->name('index');
});

Set up API routes in routes/api.php like this:



<?php

use Illuminate\Support\Facades\Route;
use App\Http\Controllers\TaskController;

Route::group(['prefix' => '/tasks', 'as' => 'tasks.'], function () {
    Route::get('/', [TaskController::class, 'list']);
    Route::get('/{id}', [TaskController::class, 'get'])
  ->where('id', '[1-9][0-9]*');
    Route::post('/', [TaskController::class, 'store']);
    Route::put('/{id}', [TaskController::class, 'update'])
     ->where('id', '[1-9][0-9]*');
    Route::delete('/{id}', [TaskController::class, 'delete'])
     ->where('id', '[1-9][0-9]*');
    Route::put('/', [TaskController::class, 'reorder']);
});

We have all the API routes in the routes/api.php instead of the routes/web.php, because in the web.php file all the routes by default are CSRF protected.

As you can see, there is a task prefix for all API routes. It's optional to have a prefix, but it's just a good practice. And for the specific API routes, there are regex validations for accepting only natural numbers as project IDs.

Don't forget to refresh route caches after the above changes. It's important to remember that Laravel (version 10 in this case) reads routes from the cached bootstrap/cache/routes-v7.php file, and they won't be updated automatically right after your changes. It just generates one if it hasn't cached yet.

Use the below command to refresh Laravel caches as well as the route caches:



php artisan optimize

Requests Validation

Before writing controller methods, you'll need to add some validation request files. You can do that manually or by just using the artisan command:



php artisan make:request Task/CreateTaskRequest
php artisan make:request Task/ListTasksRequest
php artisan make:request Task/ReorderTasksRequest
php artisan make:request Task/UpdateTaskRequest

After creating them, you'll need to set up validation rules for each request.

Below are the validation rules for creating a new task:



return [
    'project_id' => 'required|integer|exists:projects,id',
    'title' => 'required|string|max:255',
    'description' => 'nullable|string',
];

Below is the validation rule for listing project tasks:



return [
    'project_id' => 'required|integer|exists:projects,id',
];

Below are the validation rules for tasks reordering:



return [
    'project_id' => 'required|integer|exists:projects,id',
    'start' => 'required|integer',
    'end' => 'required|integer|different:start',
];

Below are the validation rules for updating a task:



return [
    'title' => 'required|string|max:255',
    'description' => 'nullable|string',
];

Don't forget to return true in the authorize() method in all validation classes:



public function authorize(): bool
{
    return true;
}

This function is usually designed to determine if the user is authorized to make the request. As we don't use authentication as well as authorization stuff in the app, it should return true for all the cases.

Using Services in a Controller

As the last step in the backend part, it's time to write controller methods for each API route, which will use service functions.



<?php

namespace App\Http\Controllers;

use App\Http\Requests\Task\CreateTaskRequest;
use App\Http\Requests\Task\ListTasksRequest;
use App\Http\Requests\Task\ReorderTasksRequest;
use App\Http\Requests\Task\UpdateTaskRequest;
use App\Services\ProjectService;
use App\Services\TaskService;
use Illuminate\Http\JsonResponse;

class TaskController extends Controller
{
    protected ?TaskService $taskService = null;

    public function __construct(TaskService $taskService)
    {
        $this->taskService = $taskService;
    }

    public function index()
    {
        $projects = (new ProjectService())->getAll();

        return view('tasks.index', [
            'projects' => $projects,
        ]);
    }

    public function list(ListTasksRequest $request): JsonResponse
    {
        $tasks = $this->taskService->list($request->get('project_id'));

        return response()->json([
            'success' => true,
            'tasks' => $tasks,
            'message' => "Tasks retrieved successfully.",
        ]); // 200
    }

    public function store(CreateTaskRequest $request): JsonResponse
    {
        $this->taskService->store($request->all());

        return response()->json([
            'success' => true,
            'message' => "Task created successfully.",
        ], 201);
    }

    public function get(int $id): JsonResponse
    {
        $task = $this->taskService->getById($id);

        if ($task) {
            return response()->json([
                'success' => true,
                'task' => $task,
                'message' => "Task retrieved successfully.",
            ]); // 200
        } else {
            return response()->json([
                'success' => false,
                'message' => "Task not found!",
            ], 404);
        }
    }

    public function update(UpdateTaskRequest $request, int $id): JsonResponse
    {
        $this->taskService->update($id, $request->all());

        return response()->json([
            'success' => true,
            'message' => "Task updated successfully.",
        ], 201);
    }

    public function delete(int $id): JsonResponse
    {
        $this->taskService->delete($id);

        return response()->json([
            'success' => true,
            'message' => "Task deleted successfully.",
        ], 201);
    }

    public function reorder(ReorderTasksRequest $request): JsonResponse
    {
        $this->taskService->reorder(
            $request->get('project_id'),
            $request->get('start'),
            $request->get('end')
        );

        return response()->json([
            'success' => true,
            'message' => "Tasks reordered successfully.",
        ], 201);
    }
}

As you can see in the TaskController:

TaskService is injected into the constructor method as an argument. In the constructor body, an instance of the TaskService class is created, and the $taskService property is initialized. So in the custom methods, you'll be able to access that $taskService and its functions.
The index method is for returning the HTML.
All the other custom methods (list, store, get, update, delete, reorder) are using the TaskService functions through the already initialized $taskService property. So, all the logic implementation goes to the service, and this way, you just call a service function and return the response.

Test the API Routes

At this point, you can test the API routes by requesting them via Postman or any similar tool. Just run (or rerun) the backend:



php artisan serve

Here's the published Postman collection with all the requests.

Frontend — Install the Packages

Now it's time to switch to the frontend. We'll use TypeScript for React.js. After completing this part, you'll be able to integrate React.js (with Vite) in your Laravel app.

First, make sure you have Node.js version 18 or above by using this command:



node -v

Install these necessary npm packages:



npm i react-dom dotenv react-beautiful-dnd react-responsive-modal react-toastify @vitejs/plugin-react

react-dom is a library from the React team for rendering React components in the DOM (Document Object Model)
dotenv is for loading environment variables from the .env file into the process environment
react-beautiful-dnd is a library from Atlassian for creating drag-and-drop interfaces with animations
react-responsive-modal is for creating simple and responsive modal dialogs
react-toastify is for displaying notifications or toasts
@vitejs/plugin-react is a plugin for the Vite build tool that enables seamless integration of React with fast development and optimized production builds

Install the development dependencies with this command:



npm i -D @types/react-dom @types/react-beautiful-dnd

@types/react-dom is TypeScript type definitions for the react-dom package
@types/react-beautiful-dnd is TypeScript type definitions for the react-beautiful-dnd package

Vite Configuration

As Laravel v10 already has vite.config.js, you'll want to set up any React-related stuff there. Or if you still don't have this file, create one like this:



import { defineConfig } from 'vite';
import laravel from 'laravel-vite-plugin';
import react from '@vitejs/plugin-react';
import 'dotenv/config';

export default defineConfig({
    build: {
        minify: process.env.APP_ENV === 'production' ? 'esbuild' : false,
        cssMinify: process.env.APP_ENV === 'production',
    },
    plugins: [
        laravel({
            input: ['resources/react/app.tsx'],
            refresh: true,
        }),
        react(),
    ],
});

As you can see in the Vite configuration file, there is a reference to the resources/react/app.tsx, which will be the entry point for Laravel to use React resources.

For the initial HTML page, create a resources/views/react/index.blade.php blade file, so all the frontend assets will be injected there in the div with ID app:



<!DOCTYPE html>
<html lang="{{ str_replace('_', '-', app()->getLocale()) }}">
<head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>{{ config("app.name") }}</title>
    <link rel="shortcut icon" href="{{ asset('favicon.ico') }}" />
    <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Montserrat">
    <link rel="stylesheet" href="https://fonts.googleapis.com/icon?family=Material+Icons">
    @viteReactRefresh
    @vite('resources/react/app.tsx')
</head>
<body>
<div id="app" data-projects="{{ json_encode($projects) }}"></div>
</body>
</html>

As you can see in the blade file, there is a $projects variable passed from the backend. It's the whole project data that will be used to filter tasks in the frontend.

Also, there are two directives: @viteReactRefresh and @vite('resources/react/app.tsx'). The first one is for enabling React hot module replacement, and the second one is for injecting the React app's entry point.

React.js — Initial Integration

In this article, we'll just have a basic React.js app working with Laravel.

At first, it's a good idea to delete unnecessary resources, like default resources/css and resources/js directories.

Create a resources/react/app.tsx file like this:



import ReactDOM from 'react-dom/client';
import Main from "./Main";
import './index.css'

ReactDOM.createRoot(document.getElementById('app')).render(
    <Main />
);

So, resources/react folder will be the root directory for all the upcoming React stuff.

Create an index.css with some temporary content:



.test-class {
  color: red;
}

Also create a Main.tsx with some temporary content:



function Main() {
    return (
        <div>
            <h2 className="test-class">React App</h2>
        </div>
    );
}

export default Main;

To check the result in the browser, make sure you have backend running and build the assets via the vite tool:



npm run build

Or, if you want to watch React file changes and automatically build assets, you can keep this command running:



npm run dev

The two npm run commands above refer to vite, which builds the assets.
You can see this by checking the package.json file, scrips field:



"scripts": {
    "dev": "vite",
    "build": "vite build"
}

Now you can open http://localhost:8000 to see the initial rendered view:

[Screenshot from browser]

Adding CSS

Now, once you've set up Vite and have React integrated into your Laravel app, you can work on the React part.

We won't spend too much time on styles, so you can paste this CSS into your index.css:



body { background-color: whitesmoke; color: rgba(255, 255, 255, 0.7); font-family: "Montserrat", sans-serif; cursor: default; margin: auto 0; }

/* MODAL start */
.modal-content { display: flex; flex-direction: column; justify-content: center; align-items: center; background-color: #fff; padding: 20px; border-radius: 5px; width: 500px; color: #2d3748; }
.modal-header { font-size: 1.5rem; font-weight: 600; margin-bottom: 20px; }
.modal-input-header { font-weight: 600; margin-bottom: 10px; }
.modal-input { width: 100%; height: 30px; border: 1px solid #ccc; border-radius: 5px; padding: 5px; }
.modal-textarea { width: 100%; height: 100px; border: 1px solid #ccc; border-radius: 5px; padding: 5px; margin-bottom: 20px; resize: vertical; }
.modal-actions { display: flex; justify-content: space-between; align-items: center; width: 100%; }
.modal-btn { padding: 10px 20px; border-radius: 5px; cursor: pointer; }
.modal-btn-cancel { background-color: #e53e3e; color: #fff; border: none; }
.modal-btn-cancel:hover { background-color: #c53030; }
.modal-btn-submit { background-color: #2d3748; color: #fff; border: none; }
.modal-btn-submit:hover { background-color: #4a5568; }
.modal-question { font-size: 1.2rem; font-weight: 600; margin-bottom: 20px; }
/* MODAL end */

/* LEFT & RIGHT SIDE start */
.left-side { width: 50%; float: left; }
.right-side { width: 50%; float: left; }
/* LEFT & RIGHT SIDE end */

/* LEFT SIDE start */
.left-side .no-tasks { font-size: 1.2rem; font-weight: 600; margin-bottom: 20px; text-align: center; color: #2d3748; }
.left-side .task-item { padding: 10px; margin: 10px; min-height: 20px; min-width: 200px; color: #2d3748; list-style-type: none; }
.left-side .task-item-content { display: flex; justify-content: space-between; align-items: center; }
.left-side .task-project { display: flex; flex-direction: column; justify-content: center; align-items: center; margin-right: 10px; }
.left-side .task-project-name { font-size: 1.5rem; font-weight: 600; margin-bottom: 5px; }
.left-side .task-time { font-size: 0.8rem; }
.left-side .task-title { font-size: 1.2rem; }
.left-side .task-actions { display: flex; justify-content: space-between; align-items: center; }
.left-side .task-edit-btn { background-color: #2d3748; color: #fff; border: none; border-radius: 5px; padding: 5px 10px; cursor: pointer; margin: 0 5px; }
.left-side .task-edit-btn:hover { background-color: #4a5568; }
.left-side .task-delete-btn { background-color: #e53e3e; color: #fff; border: none; border-radius: 5px; padding: 5px 10px; cursor: pointer; margin: 0 5px; }
.left-side .task-delete-btn:hover { background-color: #c53030; }
/* LEFT SIDE end */

/* RIGHT SIDE start */
.right-side .projects { display: flex; justify-content: space-between; align-items: center; margin-bottom: 20px; }
.right-side .projects-select { width: 100%; height: 30px; border: 1px solid #ccc; border-radius: 5px; padding: 5px; }
.right-side .no-project-selected { font-size: 1.2rem; font-weight: 600; margin-bottom: 20px; }
.right-side .right-side-wrapper { display: flex; flex-direction: column; justify-content: center; align-items: center; background-color: #fff; padding: 20px; border-radius: 5px; color: #2d3748; }
.right-side .add-task-header { font-size: 1.5rem; font-weight: 600; margin-bottom: 20px; }
.right-side .add-task-input-header { font-weight: 600; margin-bottom: 10px; }
.right-side .add-task-input { width: 100%; height: 30px; border: 1px solid #ccc; border-radius: 5px; padding: 5px; }
.right-side .add-task-textarea { width: 100%; height: 100px; border: 1px solid #ccc; border-radius: 5px; padding: 5px; margin-bottom: 20px; resize: vertical; }
.right-side .add-task-actions { display: flex; justify-content: space-between; align-items: center; width: 100%; }
.right-side .add-task-btn { padding: 10px 20px; border-radius: 5px; cursor: pointer; }
.right-side .add-task-btn-cancel { background-color: #e53e3e; color: #fff; border: none; }
.right-side .add-task-btn-cancel:hover { background-color: #c53030; }
.right-side .add-task-btn-submit { background-color: #2d3748; color: #fff; border: none; }
.right-side .add-task-btn-submit:hover { background-color: #4a5568; }
/* RIGHT SIDE end */

Later you'll attach the index.css file in your main component.

A Service for the API Requests

As you did in backend, here in the frontend you also can move all the logic implementations into a different file, so your code will be more readable and maintainable. We can name that file utils.ts, as there will be utilities in it we need.

Before that, just create axiosConfig.ts for the global Axios configuration, which you'll use in utils.ts.



import axios from 'axios';

export default axios.create({ baseURL: '/api' });

Using the above setup, you can be sure that all the HTTP requests will have the /api prefix.

For example, if you use axiosConfig.get('/example'), it will send a GET request to the /api/example. This is an optional configuration, but it's a recommended way to have non-repetitive code.

As you'll have a few use cases for sending HTTP requests to the server, you can have separate utilities file for those operations:

Create a new task for a project
Update a task
List project's tasks
Delete a task
Reorder project's tasks

So below is the utils.ts file:



import axiosConfig from './axiosConfig';
import { toast } from 'react-toastify';

export const getErrorMessage = (error: unknown) => {
    if (error instanceof Error) return error.message;
    return String(error)
}

export const getTasks = async (projectId) => {
    if (!projectId) {
        toast.error("Project is required!");
        return;
    }

    try {
        const response = await axiosConfig.get(`/tasks?project_id=${projectId}`);
        const { success, tasks, message } = response.data;

        if (success) {
            return tasks;
        } else {
            toast.error(message);
            return [];
        }
    } catch (err) {
        toast.error(getErrorMessage(err));
        return [];
    }
}

export const reorderTasks = async (projectId, start, end) => {
    try {
        const response = await axiosConfig.put('/tasks', {
            project_id: projectId,
            start,
            end,
        });
        const { success, message } = response.data;

        toast[success ? 'success' : 'error'](message);
    } catch (err) {
        toast.error(getErrorMessage(err));
    }
}

export const editTask = async (task) => {
    if (!task.id) return;
    if (!task.title) {
        toast.error("Title is required!");
        return;
    }

    try {
        const response = await axiosConfig.put(`/tasks/${task.id}`, {
            title: task.title,
            description: task.description,
        });
        const { success, message } = response.data;

        toast[success ? 'success' : 'error'](message);
    } catch (err) {
        toast.error(getErrorMessage(err));
    }
}

export const deleteTask = async (id) => {
    if (!id) {
        toast.error("Invalid task!");
        return;
    }

    try {
        const response = await axiosConfig.delete(`/tasks/${id}`);
        const { success, message } = response.data;

        toast[success ? 'success' : 'error'](message);
    } catch (err) {
        toast.error(getErrorMessage(err));
    }
}

export const createTask = async (task, projectId) => {
    if (!projectId) {
        toast.error("Project is required!");
        return;
    }
    if (!task.title) {
        toast.error("Title is required!");
        return;
    }

    try {
        const response = await axiosConfig.post(`/tasks?project_id=${projectId}`, {
            title: task.title,
            description: task.description,
        });
        const { success, message } = response.data;

        toast[success ? 'success' : 'error'](message);
    } catch (err) {
        toast.error(getErrorMessage(err));
    }
}

In the above file, you'll find the following functions:

getErrorMessage: Returns the error message if the input is an instance of Error — otherwise, converts it to a string.
getTasks: Retrieves tasks for a given project ID using Axios. Displays an error toast if the project ID is missing or if the API request is unsuccessful.
reorderTasks: Sends a PUT request to reorder tasks within a project based on start and end positions. Displays a success or error toast based on the API response.
editTask: Sends a PUT request to update task information. Validates that the task has an ID and a title before making the request. Displays a success or error toast based on the API response.
deleteTask: Sends a DELETE request to delete a task by its ID. Displays a success or error toast based on the API response.
createTask: Sends a POST request to create a new task for a given project ID. Validates that the project ID is present, and the task has a title before making the request. Displays a success or error toast based on the API response.

React.js Components

Now, since you have utilities ready, in the resources/react/components folder, you can create the components you need to use in your Main.tsx.

First, create SelectProject.tsx, which will be responsible for choosing the current project:



import {getTasks} from "../utils";

function SelectProject({projectId, projects, setProjectId, setTasks}) {
    const selectProject = (e) => {
        const value = e.target.value;
        setProjectId(value);
        if (value === '') {
            setTasks([]);
        } else {
            getTasks(value).then((tasksData) => setTasks(tasksData));
        }
    };

    return (
        <div className="projects">
            <select className="projects-select"
                    value={projectId}
                    onChange={selectProject}>
                <option value="" defaultValue>Choose a project</option>
                {projects.map((project) => (
                    <option key={project.id}
                            value={project.id}>{project.name}</option>
                ))}
            </select>
        </div>
    );
}

export default SelectProject;

The SelectProject component renders a dropdown menu allowing the user to select a project. When a project is selected, it updates the state with the selected project ID, fetches tasks for that project using the getTasks utility function, and updates the state with the retrieved tasks, providing dynamic interaction with project selection and task loading.

Then create TaskList.tsx, which will be responsible for rendering the project's tasks and for their drag and drop manipulations:



import {DragDropContext, Draggable, Droppable} from "react-beautiful-dnd";
import Task from "./Task";
import {reorderTasks} from "../utils";

const getItemStyle = (isDragging, draggableStyle) => ({
    background: isDragging ? 'lightgreen' : 'grey',
    ...draggableStyle,
});

function TaskList({ tasks, setIsModalEditOpen, setModalEditTask, setIsModalDeleteOpen, setModalDeleteTaskId, projectId, setTasks })
{
    const handleDragEnd = (result) => {
        if (!result.destination || result.destination.index === result.source.index) {
            return;
        }

        const items = Array.from(tasks);
        const [reorderedItem] = items.splice(result.source.index, 1);
        items.splice(result.destination.index, 0, reorderedItem);
        reorderTasks(projectId, result.source.index + 1, result.destination.index + 1);

        setTasks(items);
    };

    return (
        <DragDropContext onDragEnd={handleDragEnd}>
            <Droppable droppableId="droppable">
                {(provided) => (
                    <ul {...provided.droppableProps} ref={provided.innerRef}>
                        {tasks.map((task, index) => (
                            <Draggable key={task.id.toString()} draggableId={task.id.toString()} index={index}>
                                {(provided, snapshot) => (
                                    <li ref={provided.innerRef}
                                        {...provided.draggableProps}
                                        {...provided.dragHandleProps}
                                        className="task-item"
                                        style={getItemStyle(snapshot.isDragging, provided.draggableProps.style)}
                                    >
                                        <Task task={task}
                                              setIsModalEditOpen={setIsModalEditOpen}
                                              setModalEditTask={setModalEditTask}
                                              setIsModalDeleteOpen={setIsModalDeleteOpen}
                                              setModalDeleteTaskId={setModalDeleteTaskId}
                                        />
                                    </li>
                                )}
                            </Draggable>
                        ))}
                        {provided.placeholder}
                    </ul>
                )}
            </Droppable>
        </DragDropContext>
    );
}

export default TaskList;

The TaskList component utilizes the react-beautiful-dnd library to implement a draggable task list. It renders a list of tasks, allowing users to drag and drop tasks to reorder them, with drag-and-drop functionality triggering a function (handleDragEnd()) that updates the task order both visually and in the backend using the reorderTasks utility function.

Now, create Task.tsx, which will be responsible for a single Task from a list:



function Task({ task, setIsModalEditOpen, setModalEditTask, setIsModalDeleteOpen, setModalDeleteTaskId })
{
    const handleEdit = () => {
        setModalEditTask(task);
        setIsModalEditOpen(true);
    };

    const handleDelete = () => {
        setModalDeleteTaskId(task.id);
        setIsModalDeleteOpen(true);
    };

    return (
        <div className="task-item-content">
            <span className="task-project">
                <span className="task-project-name">
                 {task.project.name}
                </span>
                <span className="task-time">
                 Created {task.created}
                </span>
            </span>
            <span className="task-title">{task.title}</span>
            <div className="task-actions">
                <button className="task-edit-btn"
                 onClick={handleEdit}>Edit</button>
                <button className="task-delete-btn"
                 onClick={handleDelete}>Delete</button>
            </div>
        </div>
    );
}

export default Task;

The Task component represents a single task item. It displays task details including the project name, creation time, and title, and provides buttons to trigger actions such as editing and deleting the task, with corresponding handlers (handleEdit and handleDelete).

Next, create AddTaskForm.tsx, which will be responsible for the task form for adding tasks to the current selected project:



import {createTask} from "../utils";

function AddTaskForm({newTask, setNewTask, projectId, reloadTasks })
{
    const clearTaskCreate = () => {
        setNewTask({title: '', description: ''});
    };
    const submitTaskCreate = () => {
        createTask(newTask, projectId).then(() => {
            setNewTask({title: '', description: ''});
            reloadTasks();
        });
    };

    return (
        <>
            <h2 className="add-task-header">Add Task</h2>

            <h3 className="add-task-header">Title</h3>
            <input type="text"
                   className="add-task-input"
                   onChange={(e) => setNewTask({
                    ...newTask,
                    title: e.target.value
                   })}
                   value={newTask.title}
            />
            <h3 className="add-task-input-header">Description</h3>
            <textarea className="add-task-textarea"
                      onChange={(e) => setNewTask({
                       ...newTask,
                        description: e.target.value
                      })}
                      value={newTask.description || ''}
            />
            <div className="add-task-actions">
                <button className="add-task-btn add-task-btn-cancel"
                        onClick={clearTaskCreate}>Clear
                </button>
                <button className="add-task-btn add-task-btn-submit"
                        onClick={submitTaskCreate}>Add</button>
            </div>
        </>
    );
}

export default AddTaskForm;

The AddTaskForm component provides a form for adding new tasks. It includes input fields for the task title and description, with buttons to clear the form or submit the task creation, and it utilizes the createTask utility function to handle the task creation process, triggering a reload of tasks upon success.

Then, create ModalEdit.tsx, which will be responsible for the modal popup for editing and submitting changes to a task:



import {Modal} from "react-responsive-modal";
import React from "react";
import {editTask} from "../utils";

function ModalEdit({
 isModalEditOpen, setIsModalEditOpen, setModalEditTask,
    modalEditTask, reloadTasks
}) {
    const submitTaskEdit = () => {
        setIsModalEditOpen(false);
        editTask(modalEditTask).then(() => {
            reloadTasks();
        });
    };

    return (
        <Modal open={isModalEditOpen} center
         onClose={() => setIsModalEditOpen(false)}>
            <div className="modal-content">
                <h2 className="modal-header">Edit Task</h2>
                <h3 className="modal-input-header">Title</h3>
                <input type="text" value={modalEditTask.title}
                       className="modal-input"
                       onChange={(e) => setModalEditTask({
                        ...modalEditTask,
                        title: e.target.value
                       })}
                />
                <h3 className="modal-input-header">Description</h3>
                <textarea className="modal-textarea"
                          onChange={(e) => setModalEditTask({
                           ...modalEditTask,
                            description: e.target.value
                          })}
                          value={modalEditTask.description || ''}
                />
                <div className="modal-actions">
                    <button className="modal-btn modal-btn-cancel"
                            onClick={() => setIsModalEditOpen(false)}
                    >Close
                    </button>
                    <button className="modal-btn modal-btn-submit"
                            onClick={submitTaskEdit}
                    >Save</button>
                </div>
            </div>
        </Modal>
    )
}

export default ModalEdit;

The ModalEdit component displays a modal for editing task details. It includes input fields for modifying the task title and description, and buttons to close the modal or save the changes, using the editTask utility function to handle the task editing process and triggering a reload of tasks upon successful editing.

Next, create ModalDelete.tsx, which will be responsible for submitting a task deletion:



import {Modal} from "react-responsive-modal";
import {deleteTask} from "../utils";

function ModalDelete({
 isModalDeleteOpen, setIsModalDeleteOpen,
    modalDeleteTaskId, reloadTasks
}) {
    const submitTaskDelete = () => {
        setIsModalDeleteOpen(false);
        deleteTask(modalDeleteTaskId).then(() => {
            reloadTasks();
        });
    };

    return (
        <Modal open={isModalDeleteOpen} onClose={() => setIsModalDeleteOpen(false)} center>
            <div className="modal-content">
                <h2 className="modal-header">Delete Task</h2>
                <p className="modal-question">
                 Are you sure you want to delete this task?
                </p>
                <div className="modal-actions">
                    <button className="modal-btn modal-btn-cancel"
                            onClick={() => setIsModalDeleteOpen(false)}
                    >Cancel</button>
                    <button className="modal-btn modal-btn-submit"
                            onClick={submitTaskDelete}
                    >Yes</button>
                </div>
            </div>
        </Modal>
    );
}

export default ModalDelete;

The ModalDelete component displays a modal for confirming the deletion of a task. It provides options to either cancel the deletion or proceed with deleting the task, utilizing the deleteTask utility function and triggering a reload of tasks upon successful deletion.

And lastly, set up the Main.tsx by using the above-defined components.



import { useState } from 'react';
import {getTasks} from "./utils";
import "react-responsive-modal/styles.css";
import { ToastContainer } from 'react-toastify';
import 'react-toastify/dist/ReactToastify.css';
import ModalEdit from "./components/ModalEdit";
import ModalDelete from "./components/ModalDelete";
import TaskList from "./components/TaskList";
import SelectProject from "./components/SelectProject";
import AddTaskForm from "./components/AddTaskForm";

function Main () {
    const [projectId, setProjectId] = useState('');
    const projectsData = document.getElementById('app').getAttribute('data-projects');
    const projects = JSON.parse(projectsData);
    const [tasks, setTasks] = useState([]);
    const [isModalEditOpen, setIsModalEditOpen] = useState(false);
    const [modalEditTask, setModalEditTask] = useState({id: '', title: '', description: ''});
    const [isModalDeleteOpen, setIsModalDeleteOpen] = useState(false);
    const [modalDeleteTaskId, setModalDeleteTaskId] = useState('');
    const [newTask, setNewTask] = useState({title: '', description: ''});

    const reloadTasks = () => {
        getTasks(projectId).then((tasksData) => setTasks(tasksData));
    };

    return (
        <div>
            <ToastContainer autoClose={2000} />
            <ModalEdit isModalEditOpen={isModalEditOpen}
                       setIsModalEditOpen={setIsModalEditOpen}
                       modalEditTask={modalEditTask}
                       setModalEditTask={setModalEditTask}
                       reloadTasks={reloadTasks}
            />
            <ModalDelete isModalDeleteOpen={isModalDeleteOpen}
                         setIsModalDeleteOpen={setIsModalDeleteOpen}
                         modalDeleteTaskId={modalDeleteTaskId}
                         reloadTasks={reloadTasks}
            />
            <div className="left-side">
                {tasks.length > 0 ? (
                    <TaskList tasks={tasks}
                              setIsModalEditOpen={setIsModalEditOpen}
                              setModalEditTask={setModalEditTask}
                              setIsModalDeleteOpen={setIsModalDeleteOpen}
                              setModalDeleteTaskId={setModalDeleteTaskId}
                              projectId={projectId}
                              setTasks={setTasks}
                    />
                ) : (
                    <div className="no-tasks">
                        {projectId === '' ? (
                            <p>Choose a project to see its tasks.</p>
                        ) : (
                            <p>This project has no tasks.</p>
                        )}
                    </div>
                )}
            </div>
            <div className="right-side">
                <div className="right-side-wrapper">
                    <SelectProject projectId={projectId}
                                   projects={projects}
                                   setProjectId={setProjectId}
                                   setTasks={setTasks}
                    />
                    {projectId === '' ? (
                        <div className="no-project-selected">
                            <p>Please select a project.</p>
                        </div>
                    ) : (
                        <AddTaskForm newTask={newTask}
                                     setNewTask={setNewTask}
                                     projectId={projectId}
                                     reloadTasks={reloadTasks}
                        />
                    )}
                </div>
            </div>
        </div>
    );
}

export default Main;

The Main component is a central component that managing project and task-related functionalities. It includes modals for editing and deleting tasks, a task list with dynamic updates, a project selection dropdown, and a form for adding new tasks, leveraging state management and utility functions for smooth user interaction.

Final Results

At this point, all the components are ready to interact with each other. So you can build the frontend assets and run the server:



npm run build && php artisan serve

By visiting http://127.0.0.1:8000, you'll get this kind of result:

[GIF generated from a local working project]

That's it!

Now you can easily integrate React.js into your Laravel app without using any additional Laravel tools (like Inertia). And as a result, you can continue to maintain your Laravel app to build more scalable APIs with its authentication and other stuff.

So, this can be just an example app for your next full-stack Laravel and React.js project.

Conclusion

With this article, you built a single-page, full-stack Tasklist app using React.js (with TypeScript) with Vite.js as frontend technologies, Laravel as a backend framework, and react-beautiful-dnd package for having draggable items.
Now you know how to manually integrate React.js in your Laravel app and maintain it.

You can find the complete code of the project here on my GitHub ⭐, where I actively publicize much of my work about various modern technologies.
For more information, you can visit my website: boolfalse.com

Feel free to share this article. 😇

Remove latest pushed Git-commits from remote repo

boolfalse — Fri, 17 Nov 2023 00:22:39 +0000

We’ll do an experiment on GitHub, but practically you can do this on other similar platforms as well. Our goal is to delete recently pushed commits from the remote repository branch.

Let’s assume we have a remote Git repository, and there we already have some commits pushed. Two developers maintaining the project:

Owen — the repository owner
Cole — the repository collaborator

In some moment of time project’s comment history on the master branch looks like this:

Fake page from GitHub commits history

As we can see, the first five (starting from the bottom) commits were pushed by the owner, and the next four commits were pushed by the collaborator.

Now the question:

Is there a way to remove the latest commits from the master branch?

Criterias:

master branch is the default branch of the project (it could be any branch)
no one can delete master branch (we can’t delete the branch and recreate it anyway)
last commits done by a collaborator of the project (it could be one commit or more by one or more collaborators)

Our goal:

Our goal is to clean up all the latest commits (in our case, from “Commit 6” to ”Commit 9") from the master branch.

Solution steps:

Let’s make sure, that we are on the master branch, and pull the remote state to the local machine:

git checkout master
git pull origin master

Let’s remove all the latest commits on the local machine, which we want to delete. We’ll point the hash-code of the commit, that would be the HEAD commit after all the deletion (the hash of “Commit 5” in our case):

commits that will be removed are in red rectangle

git reset --soft 9d24c70
git reset
git checkout .

Now we’ll create a new temporary branch and switch to it with current HEAD. We’ll remember to delete that right after the commits removal.

git checkout -b temp

Let’s make some small changes (something, anything safe) to that temporary branch, commit them, and push to remote:

git add .
git commit -m "small changes"
git push origin temp

Now, here’s the most important part of this article:
Switch back to the master branch, rebase the temporary branch on it, and push that to remote:

git checkout master
git rebase temp
git push origin master

At this moment we already have master branch cleaned from the latest unnecessary commits.

master branch now is clean

We can be sure about having the actual result like this:

git log --graph --oneline --decorate --all

Additionally, don’t forget to remove the temporary branch from the remote and from the local as well:

git push origin temp --delete
git branch -d temp

That’s all.

Here’s the link of the appropriate GitHub Gist that you can check out as well.

If you liked this article, feel free to follow me here. 😇

To explore projects working with various modern technologies, you can follow me on GitHub, where I actively publicize much of my work.

For more info, you can visit my website boolfalse.com

Thank you !!!

AmeriaBank & IDram V-POS integration to your Laravel (PHP) app

boolfalse — Wed, 15 Nov 2023 09:52:31 +0000

Often, online services do not have convenient documentation for developers. And sometimes difficulties arise when following the existing documentation. In this article, I will share the information I have and my experience about two online services that can help you to have some initial insights and integrate them in your app more easily.

I will describe the steps you need to follow for these two Virtual-POS (point of sale) systems to integrate in your Laravel app:

AmeriaBank vPOS integration.
IDram V-POS integration.

Both of these are frequently used online methods in recent years in the Armenian e-market and not only.

BEFORE THE BEGINNING:

Before starting to integrate any of these pieces of software, contact the official representative of the respective department through one of the contact methods listed on their official website (recommended by phone), who will provide you the credentials and the necessary information you may need.

In case of Amerabank representative's agreement, they will create a test environment for your app and you will receive an e-mail of the following form from the employee։

AmeriaBank Credentails Email:

Below is the real environment data of Ameriabank vPOS payment system integration:

Main address:
https://services.ameriabank.am/VPOS/

Address for receiving information about transactions:
https://payments.ameriabank.am/Admin/webservice/TransactionsInformationService.svc?wsdl

ClientID: ********-****-****-****-************
ClientUsr: **********
ClientPass: ****************

You can see the transactions made on your website with this link:
https://payments.ameriabank.am/admin/clients/

Credentials:
Username: ************
Password: ************

With respect,
vPOS support team

AmeriaBank Information Email:

## vPOS Testing Credentials:

The ConfirmPayment request is required for two-step payments for the implementation of the second stage and withdrawal of money from the buyer's (cardholder's) account.
If you plan to make payments in one round, then there is no need to perform the mentioned function.
Please make the appropriate correction and make test payments.

We inform you that the AmeriaBank VPOS system test environment has been created for you.
A brief description of system integration is attached.
You can also see the mentioned information at the following link:
https://servicestest.ameriabank.am/VPOS/help

In the test environment, the OrderID should be selected from the range 2350301-2350400, and the amount should be 10 AMD.
In the default test system, the ability to make one-step payments, Refund and Cancel functions are provided.
In order to make 2-step payments and related transactions, it is necessary to inform additionally in order to obtain the relevant authority.
It should be noted that the testing is considered carried out, if made through the Rest protocol and at least 5 successful payments with completed status are recorded.
Please confirm with a return email that you have received the data.
And in case of technical questions, you can call or write to us.

## vPOS minimum requirements and technical instructions:

The list of minimum requirements and technical guide for installing vPOS on the website are presented on the website:
https://ecommerce.ameriabank.am/.

Please also provide a demo version of the site that can be viewed.

AmeriaBank Documents Email:

vPOS minimum requirements
AmeriaBank vPOS 3.0: API Protocol Description (Eng)
AmeriaBank vPOS 3.0: API նկարագրություն (Arm)
Ameriabank CJSC POS Service: Terms and Conditions

Let's firstly integrate AmeriaBank vPOS 3.0.

AmeriaBank vPOS (virtual POS terminal) is a free software tool, which enables the merchant to avoid issues with time-consuming development, configuration, certification and further modification of software. It gives opportunity to have a fast and secure solution for accepting online card payments.

At first, add some environment variables to your .env file:

AMERIABANK_CLIENT_ID="********-****-****-****-************"
AMERIABANK_USERNAME="**********"
AMERIABANK_PASSWORD="******"
AMERIABANK_SUBDOMAIN="testpayments"

AMERIABANK_TEST_CARD_NUMBER="****************"
AMERIABANK_TEST_CARD_CARDHOLDER="TEST CARD VPOS"
AMERIABANK_TEST_CARD_EXP_DATE="06/26"
AMERIABANK_TEST_CARD_CVV="***"

AMERIABANK_SUBDOMAIN may be one of these:

testpayments - for testing
services - for production (recommended for live)
payments - another subdomain

Create a custom config/payment.php payment-specific configurations file:

'ameria' => [
    'client_id' => env('AMERIABANK_CLIENT_ID'),
    'username' => env('AMERIABANK_USERNAME'),
    'password' => env('AMERIABANK_PASSWORD'),
    'subdomain' => env('AMERIABANK_SUBDOMAIN', 'testpayments'),
],

Add a webhook route to your routes/web.php:

Route::post('/payment/ameria/{code}', 'PaymentController@ameria');

Important: Don't forget to disable any firewalls or middlewares that could block the incoming webhook-requests to your app.

Refresh caches after modifying configuration files and routes:

php artisan optimize

Below is the app/Http/Controllers/PaymentController snippet which can be used to accept payment webhook:

public function ameria(Request $request, string $code)
{
    $sale_data = [
        'payment_method' => 'card',
        'payment_provider' => 'ameria',
        'payment_status' => 'pending',
        'payment_json' => [
            'request' => null,
            'response' => null,
        ],
    ];
    $error = false;

    if ($request->has('orderID'))
    {
        /* This block is just an example of getting the SALE record
        you need to be already have created in your app after successful checkout */
        $sale = Sale::where('payment_method', 'card')
            ->where('code', $request['orderID'])
            ->first();
        if (empty($sale)) {
            return redirect()->route('errors.404');
        }

        $sale_data['payment_json']['request'] = $request->all();
        if ($request->has('paymentid'))
        {
            $ch = curl_init();
            curl_setopt($ch, CURLOPT_URL, "https://" . config('payment.ameria.subdomain') . ".ameriabank.am/VPOS/api/VPOS/GetPaymentDetails");
            curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
            curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode([
                "PaymentID" => $request->get('paymentid'),
                "Username" => config('payment.ameria.username'),
                "Password" => config('payment.ameria.password'),
            ], JSON_UNESCAPED_UNICODE));
            curl_setopt($ch, CURLOPT_HTTPHEADER, ['Content-Type: application/json']);
            $response = curl_exec($ch);
            $curlError = curl_error($ch);
            curl_close($ch);

            $sale_data['payment_json']['response'] = $response;

            if ($curlError) {
                $error = true;
            } else {
                switch ($request->get('respcode')) {
                    case "00":
                        break;
                    default:
                        $error = true;
                }
            }
        } else {
            $error = true;
        }

        $sale_data['payment_status'] = $error ? 'failed' : 'paid';

        // This block is just an example of updating the SALE record 
        $sale->update($sale_data);

        if ($error) {
            // You can return and show some error message
            return redirect()->route('front.main');
        } else {
            // This is just an example of creating some PDF for report
            $pdf = $this->paymentService->generateSaleInvoice($sale);

            // This is just an example of emailing the report
            $this->paymentService->sendSaleEmail($sale, $pdf);

            return redirect()->route('front.order_success', ['code' => $sale->code]);
        }
    } else {
        return redirect()->route('errors.404');
    }
}

You can read more about the system logic on VPOS Web API Help Page.

In the end, just in case I will put here some links that may be useful:

Now let's integrate IDram.

IDram V-POS (virtual POS terminal) enables trading points to accept card payments online quickly and securely. V-POS terminals enable the customer to expand product or service sales by accepting payments in the Internet environment 24/7 from anywhere in the world.

At first, add some environment variables to your .env file:

IDRAM_SECRET_KEY="**************************************"
IDRAM_EDP_REC_ACCOUNT="*********"

Add necessary webhook routes to your routes/web.php:

Route::match(['GET', 'POST', 'PUT'], '/payment/idram', 'PaymentController@idram');

Important: Don't forget to disable any firewalls or middlewares that could block the incoming webhook-requests to your app.

In the config/payment.php add these payment-specific configurations:

'idram' => [
    'secret_key' => env('IDRAM_SECRET_KEY'),
    'edp_rec_account' => env('IDRAM_EDP_REC_ACCOUNT'),
],

Refresh caches after modifying configuration files and routes:

php artisan optimize

Below is the app/Http/Controllers/PaymentController snippet which can be used to accept payment webhooks:

public function idram(Request $request)
{
    /*
    $request_body = [];
    // for both requests
    $request_body['EDP_BILL_NO'] = isset($_REQUEST['EDP_BILL_NO']) ? $_REQUEST['EDP_BILL_NO'] : null;
    $request_body['EDP_REC_ACCOUNT'] = isset($_REQUEST['EDP_REC_ACCOUNT']) ? $_REQUEST['EDP_REC_ACCOUNT'] : null;
    $request_body['EDP_AMOUNT'] = isset($_REQUEST['EDP_AMOUNT']) ? $_REQUEST['EDP_AMOUNT'] : null;
    // 1. Order Authenticity
    $request_body['EDP_PRECHECK'] = isset($_REQUEST['EDP_PRECHECK']) ? $_REQUEST['EDP_PRECHECK'] : null;
    // 2. Payment Confirmation
    $request_body['EDP_PAYER_ACCOUNT'] = isset($_REQUEST['EDP_PAYER_ACCOUNT']) ? $_REQUEST['EDP_PAYER_ACCOUNT'] : null;
    $request_body['EDP_TRANS_ID'] = isset($_REQUEST['EDP_TRANS_ID']) ? $_REQUEST['EDP_TRANS_ID'] : null;
    $request_body['EDP_TRANS_DATE'] = isset($_REQUEST['EDP_TRANS_DATE']) ? $_REQUEST['EDP_TRANS_DATE'] : null;
    $request_body['EDP_CHECKSUM'] = isset($_REQUEST['EDP_CHECKSUM']) ? $_REQUEST['EDP_CHECKSUM'] : null;
    */

    // Idram Payment System provide it
    $secret = config('payment.idram.secret_key');
    // Idram Payment System provide it
    $edp_rec_account = config('payment.idram.edp_rec_account');

    // 1. Order Authenticity
    if(isset($_REQUEST['EDP_PRECHECK']) &&
        isset($_REQUEST['EDP_BILL_NO']) &&
        isset($_REQUEST['EDP_REC_ACCOUNT']) &&
        isset($_REQUEST['EDP_AMOUNT']))
    {
        if($_REQUEST['EDP_PRECHECK'] == "YES") {
            if($_REQUEST['EDP_REC_ACCOUNT'] == $edp_rec_account) {
                $bill_no = $_REQUEST['EDP_BILL_NO'];
                // this code checks if $bill_no exists in your system orders if exists then echo OK otherwise nothing

                /* This block is just an example of getting the SALE record
                you need to be already have created in your app after successful checkout */
                $sale = Sale::where('code', $bill_no)
                    // ->where('payment_provider', 'idram')
                    ->whereIn('payment_status', ['defined', 'failed', 'pending']) // pending - when already someone want to pay with idram
                    ->first();

                if ($sale) {
                    $sale->update([
                        'payment_method' => 'idram',
                        'payment_provider' => 'idram',
                        'payment_status' => 'pending',
                        'payment_json' => $request->all(),
                    ]);

                    echo "OK"; die;
                }
            }
        }
    }

    // 2. Payment Confirmation
    if(isset($_REQUEST['EDP_PAYER_ACCOUNT']) &&
        isset($_REQUEST['EDP_BILL_NO']) &&
        isset($_REQUEST['EDP_REC_ACCOUNT']) &&
        isset($_REQUEST['EDP_AMOUNT']) &&
        isset($_REQUEST['EDP_TRANS_ID']) &&
        isset($_REQUEST['EDP_CHECKSUM']))
    {
        $txtToHash =
            $edp_rec_account . ":" .
            $_REQUEST['EDP_AMOUNT'] . ":" .
            $secret . ":" .
            $_REQUEST['EDP_BILL_NO'] . ":" .
            $_REQUEST['EDP_PAYER_ACCOUNT'] . ":" .
            $_REQUEST['EDP_TRANS_ID'] . ":" .
            $_REQUEST['EDP_TRANS_DATE'];

        if(strtoupper($_REQUEST['EDP_CHECKSUM']) != strtoupper(md5($txtToHash)))
        {
            $bill_no = $_REQUEST['EDP_BILL_NO'];

            // Write your code here to handle the payment fail

            /* This block is just an example of getting the SALE record
            you need to be already have created in your app after successful checkout */
            Sale::where([
                ['code', '=', $bill_no],
                ['payment_status', '=', 'pending'],
                // ['payment_provider', '=', 'idram'],
            ])->update([
                'payment_method' => 'idram',
                'payment_provider' => 'idram',
                'payment_status' => 'failed',
                'payment_json' => $request->all(),
            ]);

            // echo("FAIL!");
        }
        else
        {
            $bill_no = $_REQUEST['EDP_BILL_NO'];
            // please, write your code here to handle the payment success

            /* This block is just an example of getting the SALE record
            you need to be already have created in your app after successful checkout */
            $sale = Sale::where([
                ['code', '=', $bill_no],
                ['payment_status', '=', 'pending'],
                // 'payment_provider' => 'idram',
            ])->first();

            if ($sale) {
                // This block is just an example of updating the SALE record
                $sale->update([
                    'payment_method' => 'idram',
                    'payment_provider' => 'idram',
                    'payment_status' => 'paid',
                    'payment_json' => $request->all(),
                ]);

                // This is just an example of creating some PDF for report
                $pdf = $this->paymentService->generateSaleInvoice($sale);

                // This is just an example of emailing the report
                $this->paymentService->sendSaleEmail($sale, $pdf);

                echo "OK"; die;
            }
        }
    }

    if (isset($_REQUEST['EDP_BILL_NO'])) {
        $sale = Sale::where([
            ['code', '=', $_REQUEST['EDP_BILL_NO']],
            ['payment_status', '=', 'paid'],
        ])->first();

        if ($sale) {
            return redirect()->route('front.order_success', ['code' => $sale->code]);
        } else {
            // Redirect to error page with reason message
            return redirect()->route('front.main');
        }
    }
}

As you may have already noticed, in the idram method we have implemented 2 separate logics for 2 cases: Order Authenticity and Payment Confirmation, so that we can accept that two main incoming requests during the payment process. You can read more about that system logic on Idram Payment System documentation official document.
Alternatively, check for this link.

In the end, just in case I will put here some links that may be useful:

IdramMerchantPayment (the old one here)
Omnipay iDram

That's it.

If you liked this article, feel free to follow me here. 😇

To explore projects working with various modern technologies, you can follow me on GitHub, where I actively publicize much of my work.

For more info you can visit my website boolfalse.com

Thank you !!!

Quick way to solve Git problem: Permission denied

boolfalse — Tue, 14 Nov 2023 17:31:23 +0000

This was tested with non-root user and works for GitHub & GitLab (for 2FA enabled as well).

You may encounter the following problem while your development:

git@gitlab.com: Permission denied (publickey,keyboard-interactive).

In this article I will provide a way to solve this, and not only.

If you already have setup SSH key on your remote platform (such as GitHub), just skip this part.

Adding SSH to your Git account.

Run this command to generate public & private key files:

# during creation of SSH files you can leave passphrase empty
# here you can set the private-key & public-key file names/paths
ssh-keygen -t ed25519 -C "<GIT_ACCOUNT_EMAIL>"

Print the output of public SSH file and copy the output content.

# let's assume we have setup default "id_ed25519.pub"
cat ~/.ssh/id_ed25519.pub

We will do this for GitHub as an instance. But practically you can do the same for the other Git platforms as well.

Login to your account and go to the “Add SSH key” section. Use the copied public-key content for adding as a new SSH key into your account.

Test the SSH authentication for GitHub:

# for GitHub
ssh -T git@github.com

# for GitLab
ssh -T git@gitlab.com

Fixing the issue:

Local cause:

Sometimes SSH authentication test fails, and you might get something like this:

git@gitlab.com: Permission denied (publickey,keyboard-interactive).

For avoiding this, you can follow to these steps:

# cd into ".ssh" directory
cd ~/.ssh

# make sure you have private-key file octal permission as 600
stat -c "%a %n" *

# if not, then you can change the permission like this
# in our case we might have this file: "id_ed25519-gitlab"
sudo chmod 600 id_ed25519-gitlab

# [FREQUENTLY USED] if the permissions are correct, start your SSH agent
eval `ssh-agent -s`
# this will output something like this:
"Agent pid 21590"

# [FREQUENTLY USED] now add your SSH private-key as the authorized user
ssh-add ~/.ssh/id_ed25519-gitlab
# you should get something like this
"Identity added: /home/<LOCAL_USER>/.ssh/id_ed25519-gitlab (<ACCOUNT_EMAIL>)"

# now test the SSH authentication again
# you should get some success message, something like this:
"Welcome to GitLab, @<USERNAME>!"

Connection cause:

Sometimes connection is not working between your machine and the Git server, but you may want to push your code somewhere remotely.
In this kind of cases you can add the alternative/another remote and push the project there:

git remote rename origin upstream
git remote add origin git@gitlab.com:<USERNAME>/<REPO_NAME>.git
git push origin <BRANCH>

For existing repos (with HTTPS usage) change remote origin as SSH if needed:

git remote set-url origin git@github.com:<USERNAME>/<REPO_NAME>.git

That’s it.

In the end, just in case here’s some useful resources which you may use:

If you liked this article, feel free to follow me here. 😇

To explore projects working with various modern technologies, you can follow me on GitHub, where I actively publicize much of my work.

For more info you can visit my website boolfalse.com

Thank you !!!

Setup Laravel (PHP, MySQL, Node.js) on server with a bash script

boolfalse — Sun, 12 Nov 2023 22:36:45 +0000

Setting up the app on the server is the next important task. In my years of experience, I've a lot of cases when I needed to deploy my Laravel app to the server. It can be AWS EC2, DigitalOcean, Linode or any other platform. Every time I looked for the commands to install this or that software part, which took some time.
For that reason, I've made some ready-made stuff to do that without doing repetitive problem-solving.

In this article, I will share a single bash script that will deploy not only a Laravel app on the server, but install and setup some other related technologies for you as well.
I have tested the script using on AWS EC2, but can use it elsewhere.

If you want to check out the ready script now, here's the ⭐ GitHub Gist for that.

Setup Laravel with PHP, MySQL, Node.js and ElasticSearch on a server (Ubuntu 22.04)

STEPS:

Below are the steps that will be implemented during running the custom bash script:

Install Node.js (v20.x in the example).
Install Apache.
Install MySQL (v8.* in the example).
Setup MySQL user, role, privileges, database.
Install PHP (v8.2 in the example).
Install Composer.
Setup the project.
Setup server.
Setup project's files-directories permissions/ownerships.
Setup ElasticSearch (v7.x in the example).

HOW-TO-USE INSTRUCTIONS:

For using a ready script for your own, you need to use these instructions:

ssh -i ~/.ssh/private-key.pem ubuntu@<SERVER_IPv4>

Create or place the setup.sh file with the content provided somewhere in your server. It's recommended to have that in the home directory:

cd ~

Allow setup.sh for execution permission:

chmod +x setup.sh

Run shell script using project's remote repo (GitHub repo is in our case) URL as a parameter:

./setup.sh "git@github.com:<USERNAME>/<REPO>.git"

That's it. At this point you can check the public URL of the instance after the script is done.

SNIPPETS:

Now I will write all the snippets for each step, so in case you want to use them separately:

Install Node.js 20.x

echo "1. *** INSTALL NODE 20.x"
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt update
sudo apt-get install -y nodejs

Install Apache

echo "2. *** INSTALL APACHE"
sudo apt update
sudo apt install -y apache2

Install and Setup MySQL

echo "3. *** INSTALL AND SETUP MYSQL"
sudo apt update
sudo apt install -y mysql-server
MYSQL_PASSWORD=$(tr -dc 'A-Za-z0-9!"#$%&'\''()*+,-./:;<=>?@[\]^_`{|}~' </dev/urandom | head -c 13  ; echo)
echo "Please use the generated password below as a MySQL password !!!"
echo "***************************************************************"
echo "***************************************************************"
echo "********************                       ********************"
echo "********************     ${MYSQL_PASSWORD}     ********************"
echo "********************                       ********************"
echo "***************************************************************"
echo "***************************************************************"
echo "Please use the generated password above as a MySQL password !!!"

# No - (Would you like to setup VALIDATE PASSWORD component?)
# $MYSQL_PASSWORD
# $MYSQL_PASSWORD
# y - Remove anonymous users?
# No - Disallow root login remotely?
# No - Remove test database and access to it?
# y - Reload privilege tables now?

sudo mysql_secure_installation <<EOF
No
$MYSQL_PASSWORD
$MYSQL_PASSWORD
y
No
y
y
EOF

Working with MySQL

echo "4. *** WORKING WITH MYSQL"
touch mysql.sql
echo " " > mysql.sql
sed -i "1i FLUSH PRIVILEGES;" mysql.sql
sed -i "1i GRANT ALL PRIVILEGES ON * . * TO 'project'@'localhost';" mysql.sql
sed -i "1i CREATE USER 'project'@'localhost' IDENTIFIED BY \"${MYSQL_PASSWORD}\";" mysql.sql
sudo mysql < "mysql.sql"
rm mysql.sql
mysql -u project -p$MYSQL_PASSWORD -Bse "CREATE DATABASE project"

Install PHP 8.2

echo "5. *** INSTALL PHP-8.2"
sudo apt update
sudo apt -y install software-properties-common
sudo add-apt-repository ppa:ondrej/php
# [ENTER]
sudo apt update
sudo apt -y install php8.2
sudo apt update
sudo apt install -y php8.2-cli php8.2-json php8.2-common php8.2-mysql php8.2-zip php8.2-gd php8.2-mbstring php8.2-curl php8.2-xml php8.2-bcmath php8.2-soap
sudo apt update
sudo apt install unzip

Install Composer

echo "6. *** INSTALL COMPOSER"
curl -sS https://getcomposer.org/installer -o composer-setup.php
HASH=`curl -sS https://composer.github.io/installer.sig`
php -r "if (hash_file('SHA384', 'composer-setup.php') === '$HASH') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;"
sudo php composer-setup.php --install-dir=/usr/local/bin --filename=composer
rm composer-setup.php

Setup the Project

echo "7. *** SETUP THE PROJECT"
cd /var/www/
sudo mkdir web
sudo chown -R $USER:$USER web/
cd web/
git clone $1 project
cd project/
composer install
cp .env.example .env
sed -i -e "s/DB_DATABASE=laravel/DB_DATABASE=project/g" .env
sed -i -e "s/DB_USERNAME=root/DB_USERNAME=project/g" .env
sed -i -e "s/DB_PASSWORD=/DB_PASSWORD=\"${MYSQL_PASSWORD}\"/g" .env
php artisan key:generate
php artisan storage:link
php artisan route:clear
php artisan config:cache
php artisan optimize
php artisan migrate
php artisan db:seed

Setup Apache Configs (as known as Virtual Server)

echo "8. *** SETUP VIRTUAL SERVER"
touch 000-default.conf
echo "<VirtualHost *:80>
        #ServerName project.site
        DocumentRoot /var/www/web/project/public
        <Directory "/var/www/web/project">
                Options Indexes FollowSymLinks
                AllowOverride all
                Require all granted
        </Directory>
        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>" > 000-default.conf
sudo chmod 644 000-default.conf
sudo chown root:root 000-default.conf
sudo mv /etc/apache2/sites-available/000-default.conf /etc/apache2/sites-available/000-default_backup.conf
sudo mv 000-default.conf /etc/apache2/sites-available/
sudo a2enmod rewrite
sudo systemctl restart apache2

Setup Project files-folders Permissions/Ownerships

echo "9. *** SETUP PROJECT FILES-FOLDERS PERMISSIONS/OWNERSHIPS"
# APACHE_USERNAME=$(ps -ef | egrep '(httpd|apache2|apache)' | grep -v `whoami` | grep -v root | head -n1 | awk '{print $1}')
APACHE_USERNAME="www-data"
sudo usermod -a -G $APACHE_USERNAME $USER
cd /var/www/web/project
sudo chown -R $USER:$APACHE_USERNAME storage/ bootstrap/cache/
sudo chgrp -R $APACHE_USERNAME storage bootstrap/cache/
sudo chmod -R ug+rwx storage bootstrap/cache/
git checkout .

Setup ElasticSearch 7.x

echo "10. *** SETUP ELASTICSEARCH 7.x"
cd ~
curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
sudo apt update
sudo apt install elasticsearch
#sudo systemctl start elasticsearch
#sudo systemctl enable elasticsearch

All done!

If you liked this article, please feel free to follow me here. 😇

To explore projects working with various modern technologies, you can follow me on GitHub, where I actively publicize much of my work.

For more info you can visit my website boolfalse.com

Thank you !!!

Create your own AI voice assistant bot with Node.js using Google Bard

boolfalse — Sun, 12 Nov 2023 09:41:24 +0000

Before starting, just watch this very short YouTube video to see what exactly you can make with no cost by following this article until the end.

As you may already know, voice conversations have come to the ChatGPT app. That was officially announced by OpenAI:

Event Livestream: OpenAI DevDay, Opening Keynote
Blog: New models and developer products announced at DevDay
Short video by CNET: Voice Conversations Come to the ChatGPT App

In this article, you can learn a way to build an AI voice assistant using Google Bard. You will create a Telegram bot API using the following stack:

Node.js — the running backend server which will interact with Telegram to receive and send voice/text messages
Python — scripts for TTS (text-to-speech), audio-transcribing (speech-to-text) and getting Google Bard answers
Telegram — the client mobile app which will be used
Google Bard — as an AI ChatBot service
MongoDB — a database service
Amazon Polly — AWS service for TTS conversion (free tier)

The prerequisites that you will need are:

Node.js (v18 or higher) installed.
Python (v3) installed.
FFmpeg installed.

In the above image is the whole project’s lifecycle.

The required steps you need to follow are:

Create a new AWS IAM user and give it access to Amazon Polly. Get the AWS Access Key and AWS Secret Key.

AWS provides free tier for Polly.

For Amazon Polly’s Standard voices, the free tier includes 5 million characters per month for speech or Speech Marks requests, for the first 12 months, starting from your first request for speech.

Go to the AWS Management Console and sign in.

Click Identity & Access Management (IAM).

In the left pane, click Users.

Click the name of the user you want to create access keys for.

Click the Security tab.

Under Access keys (access key ID and secret access key), click Create Access Key.

AWS generates an access key ID and secret access key. Download the CSV file that contains your access key credentials, or copy and paste the access key ID and secret access key into a secure location.

Click Close.

Create a Telegram bot with BotFather. Get the Telegram Bot Token.

Open Telegram and search for “BotFather”.

Tap Start to start the conversation.

Type /newbot and send it to BotFather.

Enter a name for your bot.

Enter a username for your bot. The username must end with the word “bot”.

BotFather will reply with your bot’s token.

Copy and paste the token in a safe place.

Create a MongoDB Atlas cluster and set up IP whitelist. Get the MongoDB connection string.

Create a MongoDB Atlas cluster:

Go to the MongoDB Atlas, and sign up for a free account.

Once you are logged in, click New Cluster.

Select the cluster tier and region that you want.

Click Create Cluster.

Set up IP whitelist:

Go to the Security tab of your cluster.

Under IP Access List, click Add IP Address.

Enter your IP address and click Add IP Address.

You can also add a CIDR range to allow access from multiple IP addresses.

Get the MongoDB connection string:

Go to the Connect tab of your cluster.

Under Connection Strings, select the connection string format that you want.

Copy and paste the connection string into a safe place. You will need it to connect to your MongoDB Atlas cluster from your Node.js app.

Set the appropriate credentials in the connection string.

Login to your Google Bard account via desktop web app to get the values of __Secure-1PSID and __Secure-1PSIDTS cookies. Like so:

Now, as you have all the necessary credentials, you can start building a Node.js app:

mkdir sqca && cd sqca/

Initialize a basic Node.js app:

npm init -y

For a better VCS experience, modify the .gitignore file content using this.

Install some NPM packages that you will use in your app:

npm i aws-sdk axios dotenv fluent-ffmpeg mongoose telegraf

# optional
npm i -D nodemon

In the package.json setup “start” command like this:

"scripts": {
  "start": "node ./src/index.js",
  "dev": "nodemon ./src/index.js"
}

As you can see, we have to have some index.js file inside the src folder. Let’s create them:

mkdir src && touch src/index.js

As you need to use some functionalities, you can have separate src/utils.js file to implement them:

import AWS from 'aws-sdk';
import fs from 'fs';
import fluent from 'fluent-ffmpeg';
import axios from 'axios';
import { exec } from 'child_process';

export default {
    // download .oga file
    download: async (url, outputPath) => {
        const writer = fs.createWriteStream(outputPath);
        const response = await axios({url, method: 'GET', responseType: 'stream'});

        response.data.pipe(writer);
    },
    // check if user folder exists, if not create it
    folderStructureSync: (folder) => {
        if (!fs.existsSync(folder)) {
            fs.mkdirSync(folder);
        }
    },
    // convert .oga to .wav
    convert: async (inputPath, outputPath) => {
        return await fluent(inputPath).toFormat('wav').save(outputPath);
    },
    // extract text from .wav
    speechToText: async (filePath) => {
        const result = await new Promise((resolve, reject) => {
            const pythonExec = process.env.PYTHON_EXEC_PATH || 'python';
            exec(`${pythonExec} python/transcribe.py "${filePath}"`, (err, stdout, stderr) => {
                if (err) {
                    console.error(err.message);
                    reject(err);
                } else {
                    resolve(stdout);
                }
            });
        });

        return JSON.parse(result);
    },
    // get answer from Google Bard
    getAnswer: async (text) => {
        const gb1psid = process.env.GOOGLE_BARD_SECURE_1PSID;
        const gb1psidts = process.env.GOOGLE_BARD_SECURE_1PSIDTS;
        const result = await new Promise((resolve, reject) => {
            const pythonExec = process.env.PYTHON_EXEC_PATH || 'python';
            exec(`${pythonExec} python/answer.py "${gb1psid}" "${gb1psidts}" "${text}"`, (err, stdout, stderr) => {
                if (err) {
                    console.error(err.message);
                    reject(err);
                } else {
                    resolve(stdout);
                }
            });
        });

        return JSON.parse(result);
    },
    // empty folder
    emptyFolder: async (folder) => {
        fs.readdir(folder, (err, files) => {
            if (err) {
                console.error(err.message);
            }
            for (const file of files) {
                fs.unlink(`${folder}/${file}`, err => {
                    if (err) {
                        console.error(err.message);
                    }
                });
            }
        });
    },
    // text to voice
    textToVoice: async (text, folderPath) => {
        AWS.config.credentials = new AWS.Credentials(process.env.AWS_ACCESS_KEY, process.env.AWS_SECRET_KEY);
        AWS.config.region = "us-west-2";

        // text-to-speech service
        const Polly = new AWS.Polly({signatureVersion: 'v4', region: 'us-west-2'});

        const params = {'Text': text, 'OutputFormat': 'ogg_vorbis', 'VoiceId': 'Joanna'};
        const data = await Polly.synthesizeSpeech(params).promise();
        if (data.AudioStream instanceof Buffer) {
            const fileName = Date.now();
            const filePath = `${folderPath}/${fileName}.wav`;
            await fs.writeFile(filePath, data.AudioStream, (err) => err && console.error(err));

            return {success: true, file_path: filePath, message: 'Text to voice success!'};
        } else {
            return {success: false, message: 'Text to voice failed!'};
        }
    },
    // filter answer from Google Bard
    filterAnswer: async (originalText) => {
        let text = originalText;

        if (text.includes('Google Bard')) {text = text.replace('Google Bard', 'SQCA bot');}
        if (text.includes('Bard of Google')) {text = text.replace('Bard of Google', 'SQCA bot');}
        if (text.includes('Bard')) {text = text.replace('Bard', 'SQCA bot');}
        if (text.includes('https://bard.google.com')) {text = text.replace('https://bard.google.com', 'boolfalse.com');}

        return text;
    },
};

For interacting with DB you can have some src/handlers.js:

import User from "../models/userModel.js";
import Message from "../models/messageModel.js";
import settings from "../config/settings.js";

export default {
    getUser: async (userTelegramId) => {
        return await User.findOne({ telegram_id: userTelegramId });
    },
    createUser: async (createObj) => {
        return await User.create(createObj);
    },
    updateUser: async (userId, updateObj) => {
        await User.updateOne({ _id: userId }, updateObj);
    },
    createMessage: async (createObj) => {
        return await Message.create(createObj);
    },
    updateMessage: async (messageId, updateObj) => {
        await Message.updateOne({ _id: messageId }, updateObj);
    },
    isLimitExceeded: async (userId) => {
        const today = new Date();
        const todayStart = new Date(today.getFullYear(), today.getMonth(), today.getDate());
        const todayEnd = new Date(today.getFullYear(), today.getMonth(), today.getDate() + 1);
        const voiceMessagesLength = await Message.aggregate([{
            $match: {
                user: userId,
                // answer_status: { $ne: 4 },
                answer_status: 1,
                createdAt: { $gte: todayStart, $lt: todayEnd },
            },
        }, {
            $group: {
                _id: null,
                total: { $sum: '$question_voice_duration' },
            },
        }]);
        const voiceMessagesLengthSum = voiceMessagesLength.length > 0 ?
            voiceMessagesLength[0].total : 0;

        const messagesCount = await Message.countDocuments({
            user: userId,
            answer_status: 1,
            createdAt: { $gte: todayStart, $lt: todayEnd },
        });

        return messagesCount >= settings.max_questions_per_day ||
            voiceMessagesLengthSum >= settings.max_voice_messages_length_per_day;
    },
};

You can keep some configuration files in a separate config folder.

config/commands.js

export default {
    start: "Start bot (automatically called when you start the bot)",
    help: "Help (show the commands list)",
    test: "Test (on success, the bot will reply with voice)",
    // new: "New session (forget the previous conversation context)",
};

config/db.js

import mongoose from "mongoose";

const connectDB = async () => {
    try {
        const conn = await mongoose.connect(process.env.MONGO_URL);
        console.info(`MongoDB Connected: ${conn.connection.host}`);
    } catch (err) {
        console.error(`Error: ${err.message}`);
        process.exit(1);
    }
};

export default connectDB;

config/settings.js

export default {
    max_questions_per_day: 10, // 10 questions per day
    max_voice_messages_length_per_day: 60, // 60 seconds per day
};

Create models in a models folder:

models/messageModel.js

import mongoose from 'mongoose';

const messageSchema = mongoose.Schema({
    _id: {
        type: mongoose.Schema.Types.ObjectId,
        required: true,
        auto: true,
    },
    user: {
        type: mongoose.Schema.Types.ObjectId,
        required: true,
        ref: 'User',
    },
    question_type: {
        type: String,
        required: true,
        default: 'text', // 'text', 'voice', 'command'
    },
    command: { // question_type: command
        type: String,
        required: false,
    },
    question_text: { // question_type: text
        type: String,
        required: false,
    },
    question_voice_url: { // question_type: voice
        type: String,
        required: false,
    },
    question_voice_duration: { // question_type: voice
        type: Number,
        required: false,
    },
    answer_type: {
        type: String,
        required: true,
        default: 'text', // 'text' or 'voice'
    },
    answer_text: { // answer_type: text
        type: String,
        required: false,
    },
    answer_voice_url: { // answer_type: voice
        type: String,
        required: false,
    },
    answer_status: {
        type: Number,
        required: true,
        default: 0, // 0: pending, 1: answered, 2: rejected, 3: error, 4: limit_exceeded
    },
}, {
    timestamps: true,
});

const Message = mongoose.model('Message', messageSchema);
export default Message;

models/userModel.js

import mongoose from 'mongoose';

const userSchema = mongoose.Schema({
    _id: {
        type: mongoose.Schema.Types.ObjectId,
        required: true,
        auto: true,
    },
    telegram_id: {
        type: Number,
        required: true,
        unique: true,
    },
    name: {
        type: String,
        required: true,
    },
    username: {
        type: String,
        required: true,
        unique: true,
    },
    limit_exceeded: {
        type: Boolean,
        required: true,
        default: false,
    },
    limited_until: {
        type: Date,
        required: false,
    },
    isAdmin: {
        type: Boolean,
        required: true,
        default: false,
    },
}, {
    timestamps: true,
});

const User = mongoose.model('User', userSchema);
export default User;

You may already have noticed that in src/utils.js there are two methods, where functions calls some external python scripts: transcribe.py for speech-to-text conversion, and answer.py to retrieve the answer from the Google Bard response. Create those script files in a separate python folder:

python/transcribe.py

import speech_recognition as sr
from os import path
import json
import sys

if len(sys.argv) < 2:
    result = {
        "success": False,
        "message": "Audio file not provided!"
    }
    print(json.dumps(result))

AUDIO_FILE = path.join(path.dirname(path.realpath(__file__)), sys.argv[1])

# use the audio file as the audio source
r = sr.Recognizer()
with sr.AudioFile(AUDIO_FILE) as source:
    audio = r.record(source)  # read the entire audio file

# recognize speech using Google Speech Recognition
try:
    # to use another API key, use `r.recognize_google(audio, key="GOOGLE_SPEECH_RECOGNITION_API_KEY")`
    result = {
        "success": True,
        "message": r.recognize_google(audio)
    }
    print(json.dumps(result))
except sr.UnknownValueError:
    result = {
        "success": False,
        "message": "Google Speech Recognition could not understand audio!"
    }
    print(json.dumps(result))
except sr.RequestError as e:
    result = {
        "success": False,
        "message": "Could not request results from Google Speech Recognition service; {0}".format(e)
    }
    print(json.dumps(result))

python/answer.py

import sys
import json
from Bard import Chatbot

def process_parameters(gb1psid, gb1psidts, text):
    if gb1psid is None or gb1psidts is None or text is None:
        result = {
            "success": False,
            "message": "Command sample: python answer.py \"<gb1psid>\" \"<gb1psidts>\" \"<text>\""
        }
        return json.dumps(result)

    # Process the parameters as needed
    chatbot = Chatbot(gb1psid, gb1psidts)
    answer = chatbot.ask(text)
    result = {
        "success": True,
        "message": answer
    }
    return json.dumps(result)


# Retrieve parameters from command line arguments
gb1psid = sys.argv[1] if len(sys.argv) > 1 else None
gb1psidts = sys.argv[2] if len(sys.argv) > 2 else None
text = sys.argv[3] if len(sys.argv) > 3 else None

# Process the parameters and return the result
try:
    result = process_parameters(gb1psid, gb1psidts, text)
    print(result)
except Exception as e:
    result = {
        "success": False,
        "message": str(e)
    }
    print(json.dumps(result))

Finally, let’s work on the main src/index.js file. Here’s a high level view of it:

// IMPORTS
import dotenv from 'dotenv';
dotenv.config();
import { Telegraf } from "telegraf";
import { message } from "telegraf/filters";
import utils from "./utils.js";
import handlers from "./handlers.js";
import path from "path";
import commands from "../config/commands.js";
import connectDB from "./../config/db.js";
import settings from "../config/settings.js";

// INITIALIZATIONS
connectDB();
const bot = new Telegraf(process.env.TELEGRAM_BOT_TOKEN);

// EVENT LISTENERS
bot.command('start', async (ctx) => {
    // ...
});
bot.command('new', async (ctx) => {
    // ...
});
bot.command('help', async (ctx) => {
    // ...
});
bot.command('text', async (ctx) => {
    // ...
});
bot.command('voice', async (ctx) => {
    // ...
});

// START THE BOT
bot.launch();

// TERMINATION SIGNALS
process.once('SIGINT', () => bot.stop('SIGINT'));
process.once('SIGTERM', () => bot.stop('SIGTERM'));

In the above code let’s inject event listener callbacks step by step:

Start the bot

bot.command('start', async (ctx) => {
    const userTelegramId = ctx.message.from.id;
    let answerMessage = '';

    // check if user exists in DB, if not create it
    let dbUser = await handlers.getUser(userTelegramId);
    if (dbUser) {
        // build multiline text message
        answerMessage += 'I\'m an assistant. Just ask me anything using text or voice messages.'
    } else {
        dbUser = await handlers.createUser({
            telegram_id: userTelegramId,
            username: ctx.message.from.username,
            name: ctx.message.from.first_name,
        });
        // build multiline text message
        answerMessage = `\u{1F64B} Hello ${dbUser.name}! \n\n`;
        answerMessage += 'Below are the commands you can use:\n\n';
        Object.keys(commands).forEach((command) => {
            answerMessage += `/${command} - ${commands[command]}\n`;
        });
        answerMessage += '\nYou can send up to 10 questions per day, and in case of voice messages, up to 60 seconds per day.';
        answerMessage += '\n\nHappy hacking! \u{1F680}';
    }

    // send text to user
    await ctx.reply(answerMessage);

    // add a message to DB
    await handlers.createMessage({
        user: dbUser._id,
        question_type: 'command',
        command: 'start',
        answer_text: answerMessage,
        answer_status: 1,
    });
});

New chat

bot.command('new', async (ctx) => {
    const userTelegramId = ctx.message.from.id;

    // check if user exists in DB, if not create it
    let dbUser = await handlers.getUser(userTelegramId);
    if (!dbUser) {
        dbUser = await handlers.createUser({
            telegram_id: userTelegramId,
            username: ctx.message.from.username,
            name: ctx.message.from.first_name,
        });
    }

    let answerMessage = "Previous conversation ended. You can start a new one \u{1F609}";

    // send text to user
    await ctx.reply(answerMessage);

    // add a message to DB
    await handlers.createMessage({
        user: dbUser._id,
        question_type: 'command',
        command: 'new',
        answer_text: answerMessage,
        answer_status: 1,
    });
});

Special message for help

bot.command('help', async (ctx) => {
    const userTelegramId = ctx.message.from.id;

    // check if user exists in DB, if not create it
    let dbUser = await handlers.getUser(userTelegramId);
    if (!dbUser) {
        dbUser = await handlers.createUser({
            telegram_id: userTelegramId,
            username: ctx.message.from.username,
            name: ctx.message.from.first_name,
        });
    }

    // build multiline text message
    let answerMessage = `These are bot commands:\n\n`;
    Object.keys(commands).forEach((command) => {
        answerMessage += `/${command} - ${commands[command]}\n`;
    });
    answerMessage += '\nJust use them if need!';

    // send text to user
    await ctx.reply(answerMessage);

    // add a message to DB
    await handlers.createMessage({
        user: dbUser._id,
        question_type: 'command',
        command: 'help',
        answer_text: answerMessage,
        answer_status: 1,
    });
});

Text messages

bot.on(message('text'), async (ctx) => {
    const userTelegramId = ctx.message.from.id;
    const questionMessage = ctx.message.text;

    // check if user exists in DB, if not create it
    let dbUser = await handlers.getUser(userTelegramId);
    if (!dbUser) {
        dbUser = await handlers.createUser({
            telegram_id: userTelegramId,
            username: ctx.message.from.username,
            name: ctx.message.from.first_name,
        });
    }

    // check if 'limited_until' exists and not expired
    if (dbUser.limited_until && new Date() < dbUser.limited_until) {
        const answerMessage = `Sorry, you have exceeded the daily limit.` +
            `\nIt is max ${settings.max_questions_per_day} messages per day, ` +
            `and max ${settings.max_voice_messages_length_per_day} seconds for the voice messages.` +
            `\n\nPlease try again tomorrow! \u{1F609}`;

        // send text to user
        await ctx.reply(answerMessage);

        // add a message to DB
        await handlers.createMessage({
            user: dbUser._id,
            question_type: 'text',
            question_text: questionMessage,
            answer_status: 4,
        });

        return;
    }

    // get answer from Google Bard
    const resAnswerText = await utils.getAnswer(questionMessage);
    if (!resAnswerText || !resAnswerText.success) {
        // add a message to DB
        await handlers.createMessage({
            user: dbUser._id,
            question_type: 'text',
            question_text: questionMessage,
            answer_status: 3,
        });

        // send text to user
        await ctx.reply(resAnswerText.message || "Sorry. We couldn't get answer!");

        return;
    }

    const folderPath = path.resolve(`./voices/${userTelegramId}`);

    // check if user folder exists named userTelegramId, if not create it
    utils.folderStructureSync(folderPath);

    // filter/modify the response answer got from Google Bard
    const filteredAnswer = await utils.filterAnswer(resAnswerText.message.content);

    // text to voice
    const resTextToVoice = await utils.textToVoice(filteredAnswer, folderPath);
    if (!resTextToVoice || !resTextToVoice.success) {
        // add a message to DB
        await handlers.createMessage({
            user: dbUser._id,
            question_type: 'text',
            question_text: questionMessage,
            answer_text: resAnswerText.message?.content || '',
            answer_status: 3,
        });

        // empty folder
        await utils.emptyFolder(folderPath);

        // send text to user
        await ctx.reply(filteredAnswer); // || "Sorry. We couldn't convert text to voice!"

        return;
    }

    // send voice to user
    const answerVoice = await ctx.replyWithVoice({
        source: resTextToVoice.file_path,
    });

    // get answer voice
    const answerVoiceFile = await ctx.telegram.getFileLink(answerVoice.voice.file_id);

    // add a message to DB
    await handlers.createMessage({
        user: dbUser._id,
        question_type: 'text',
        question_text: questionMessage,
        answer_type: 'voice',
        answer_text: resAnswerText.message?.content || '',
        answer_voice_url: answerVoiceFile.href || '',
        answer_status: 1,
    });

    // empty folder
    await utils.emptyFolder(folderPath);

    if (!dbUser.isAdmin) {
        // review user limit
        const limitExceeded = await handlers.isLimitExceeded(dbUser._id);
        if (limitExceeded) {
            const limit = new Date().getTime() + 24 * 60 * 60 * 1000;
            await handlers.updateUser(dbUser._id, {
                limit_exceeded: true,
                limited_until: new Date(limit),
            });
        } else {
            await handlers.updateUser(dbUser._id, {
                limit_exceeded: false,
                limited_until: null,
            });
        }
    }
});

Voice messages

bot.on(message('voice'), async (ctx) => {
    try {
        const questionVoiceFile = await ctx.telegram.getFileLink(ctx.message.voice.file_id);
        const userTelegramId = ctx.message.from.id;

        // check if user exists in DB, if not create it
        let dbUser = await handlers.getUser(userTelegramId);
        if (!dbUser) {
            dbUser = await handlers.createUser({
                telegram_id: userTelegramId,
                username: ctx.message.from.username,
                name: ctx.message.from.first_name,
                // isAdmin: false,
            });
        }

        const fileName = ctx.message.voice.file_unique_id;
        const folderPath = path.resolve(`./voices/${userTelegramId}`);

        // check if 'limited_until' exists and not expired
        if (!dbUser.isAdmin && dbUser.limited_until && new Date() < dbUser.limited_until) {
            const answerMessage = `Sorry, you have exceeded the daily limit.` +
                `\nIt is max ${settings.max_questions_per_day} messages per day, ` +
                `and max ${settings.max_voice_messages_length_per_day} seconds for the voice messages.` +
                `\n\nPlease try again tomorrow! \u{1F609}`;

            // send text to user
            await ctx.reply(answerMessage);

            // add a message to DB
            await handlers.createMessage({
                user: dbUser._id,
                question_type: 'voice',
                question_voice_url: questionVoiceFile.href,
                answer_status: 4,
            });

            return;
        }

        // check if user folder exists named userTelegramId, if not create it
        utils.folderStructureSync(folderPath);

        // download .oga file
        await utils.download(questionVoiceFile.href, `${folderPath}/${fileName}.oga`);

        // convert .oga to .wav
        const resConvert = await utils.convert(`${folderPath}/${fileName}.oga`, `${folderPath}/${fileName}.wav`);
        if (!resConvert) {
            // add a message to DB
            await handlers.createMessage({
                user: dbUser._id,
                question_type: 'voice',
                question_voice_url: questionVoiceFile.href,
                answer_status: 3,
            });

            // empty folder
            await utils.emptyFolder(folderPath);

            // send text to user
            await ctx.reply(resConvert.message || "Sorry. We couldn't convert your voice message!");

            return;
        }

        // set this timeout to wait for converting (1000-3000ms is enough)
        setTimeout(async () => {
            // extract text from .wav
            const resSpeechToText = await utils.speechToText(`./../voices/${userTelegramId}/${fileName}.wav`);
            if (!resSpeechToText || !resSpeechToText.success) {
                // add a message to DB
                await handlers.createMessage({
                    user: dbUser._id,
                    question_type: 'voice',
                    question_voice_url: questionVoiceFile.href,
                    answer_status: 3,
                });

                // empty folder
                await utils.emptyFolder(folderPath);

                // send text to user
                await ctx.reply(resSpeechToText.message || "Sorry. We couldn't extract text from your voice message!");

                return;
            }

            // empty folder
            await utils.emptyFolder(folderPath);

            // get answer from Google Bard
            const resAnswerText = await utils.getAnswer(resSpeechToText.message);
            if (!resAnswerText || !resAnswerText.success) {
                // add a message to DB
                await handlers.createMessage({
                    user: dbUser._id,
                    question_type: 'voice',
                    question_text: resSpeechToText.message || '',
                    question_voice_url: questionVoiceFile.href,
                    answer_status: 3,
                });

                // send text to user
                await ctx.reply(resAnswerText.message || "Sorry. We couldn't get answer!");

                return;
            }

            // filter/modify the response answer got from Google Bard
            const filteredAnswer = await utils.filterAnswer(resAnswerText.message.content);

            // text to voice
            const resTextToVoice = await utils.textToVoice(filteredAnswer, folderPath);
            if (!resTextToVoice || !resTextToVoice.success) {
                await handlers.createMessage({
                    user: dbUser._id,
                    question_type: 'voice',
                    question_text: resSpeechToText.message || '',
                    question_voice_url: questionVoiceFile.href,
                    answer_text: resAnswerText.message?.content || '',
                    answer_status: 3,
                });

                // send text to user
                await ctx.reply(filteredAnswer); // || "Sorry. We couldn't convert answer to voice!"

                return;
            }

            // send voice to user
            const answerVoice = await ctx.replyWithVoice({
                source: resTextToVoice.file_path,
            });

            // get answer voice url
            const answerVoiceFile = await ctx.telegram.getFileLink(answerVoice.voice.file_id);

            await handlers.createMessage({
                user: dbUser._id,
                question_type: 'voice',
                question_text: resSpeechToText.message || '',
                question_voice_url: questionVoiceFile.href,
                question_voice_duration: ctx.message.voice.duration,
                answer_type: 'voice',
                answer_text: resAnswerText.message?.content || '',
                answer_voice_url: answerVoiceFile.href || '',
                answer_status: 1,
            });

            // empty folder
            await utils.emptyFolder(folderPath);

            if (!dbUser.isAdmin) {
                // review user limit
                const limitExceeded = await handlers.isLimitExceeded(dbUser._id);
                if (limitExceeded) {
                    const limit = new Date().getTime() + 24 * 60 * 60 * 1000;
                    await handlers.updateUser(dbUser._id, {
                        limit_exceeded: true,
                        limited_until: new Date(limit),
                    });
                } else {
                    await handlers.updateUser(dbUser._id, {
                        limit_exceeded: false,
                        limited_until: null,
                    });
                }
            }
        }, 3000);
    } catch (err) {
        console.log(err.message || "Something went wrong!");
    }
});

For make sure your python scripts will work successfully, install these dependencies (you may use pip3 instead of pip for your case):

pip install SpeechRecognition
pip install bardapi
pip install --upgrade GoogleBard

In the end, create a voices/.gitignore file which will make you sure to have a directory for temporary audio files, which will be downloaded from Telegram API:

*
!.gitignore

Now your project is probably ready for work. To check that, just run:

npm run start

If you want to check the project in a development mode, then run:

npm run dev

If you want to deploy your project on a real server to run that 24/7, then you can use some wildely used dependency like forever. Make sure to have it installed. You can install it globally:

npm i forever -g

Run the project like so:

forever start -c "npm run start" ./

For listing all the forever-processes, run:

forever list

For stopping the runing project you can use the command below with the actual process-ID:

forever stop <PID>

That’s it!

You can check out the demo on this Telegram channel: sqca_bot.
Here’s the ⭐ GitHub repository, where you can find the code.

Feel free to ask any questions you may have about this article.
If you liked this article, please feel free to follow me here. 😇

To explore projects working with various modern technologies, you can follow me on GitHub, where I actively publicize much of my work.

For more information, you can visit my website: boolfalse.com

Thank you !!!

A solid way to add multi-language support to your Laravel app

boolfalse — Thu, 09 Nov 2023 11:13:28 +0000

Using custom middleware to intercept each request and validate URI prefix as a current app locale.

There are often cases when devs need to solve the same problem within different projects.
In the course of my many years of experience, one of those problems has been a localization in Laravel projects. For various projects, I have used some method that I have prepared according to the requirements.

You might think, then why didn’t I make a separate package to solve that problem?
Well, of course, it would be possible to create a package, but usually in the case of different projects, there are so many differences to each other, especially in the front-end part, that making a separate package for only one project becomes ineffective.

So, now I will share the old-but-gold way I have used in many of my Laravel projects, and I hope it may help you.

If you just want to check the ready project now, I will put here the ⭐ GitHub repo for the project we about to build.

For getting well whole lifecycle, let’s install a fresh Laravel app.
You can integrate this method in your existing Laravel project as well.



composer create-project laravel/laravel localization && cd localization

In latest Laravel versions .env file will be created and setup automatically.

Now we have a default Laravel app installed on our machine, so we can quickly test that via built-in opportunity:



php artisan serve

And we can see the result by opening http://localhost:8000 (by default, 8000 is the port for serving a Laravel app).

Often Laravel app has some routes for customers and some separated protected routes for admins/managers. For the second case we would have to have some URI prefix only for admins/managers, for example: “dashboard”.
Let’s assume we need to develop something like we described. For instance, we have

3 views/pages/routes for customers: “home”, “about”, “contact”
and an admin view/page/route: “dashboard”

Our routes/web.php would like this:



<?php

use Illuminate\Support\Facades\Route;

Route::group([
    'prefix' => '{locale?}',
], function () {
    Route::get('/', function () { return view('pages.home'); })->name('home');
    Route::get('/about', function () { return view('pages.about'); })->name('about');
    Route::get('/contact', function () { return view('pages.contact'); })->name('contact');
});

Route::group([
    'prefix' => '{locale?}/dashboard',
], function () {
    Route::get('/', function () { return view('pages.dashboard'); })->name('dashboard');
});

Setup appropriate blade views for testing.

overwrite resources/views/welcome.blade.php:



<!DOCTYPE html>
<html lang="{{ config('localization.locale_lang.' . app()->getLocale()) }}">
<head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>Laravel</title>

    <link href="{{ asset('styles.css') }}" rel="stylesheet">
</head>
<body>

<nav>
    <div class="centered-text">
        <h1>Laravel Localization</h1>
        <p>{{ __('page.hello_world') }}</p>
    </div>
    <ul class="drop-down closed">
        <li class="cursor-pointer">
            <p class="nav-button">Change Language</p>
        </li>
        @foreach(config('localization.locales') as $locale)
            <li data-locale="{{ $locale }}" class="ss-change-locale">
                <a href="javascript:void(0)">
                    {{ __('page.locales.name.' . $locale) }}
                </a>
            </li>
        @endforeach
    </ul>
    <div class="page-content">
        @yield('content')
    </div>
</nav>

<script src="{{ asset('scripts.js') }}"></script>
<script>
    // SWITCH LANGUAGE
    var currentUri = window.location.pathname;
    var locales = "{{ implode('|', config('localization.locales')) }}";
    var currentLocale = "{{ app()->getLocale() }}";
    var elements = document.querySelectorAll('.ss-change-locale');
    for (let i = 0; i < elements.length; i++) {
        elements[i].addEventListener('click', function () {
            let locale = this.getAttribute('data-locale');
            if (locale === currentLocale) {
                return;
            }
            let newUri = currentUri.replace(new RegExp('^\/(' + locales + ')'), '/' + locale);
            let paramsIndex = window.location.href.indexOf('?');
            let params = (paramsIndex !== -1 ? window.location.href.slice(paramsIndex) : '')
            let port = window.location.port ? ':' + window.location.port : '';
            window.location.href = window.location.protocol + '//' + window.location.hostname + port + newUri + params;
        });
    }
</script>

</body>
</html>

As you can see, we’re iterating over all the languages (that we have setup in our config/localization.php configuration file), and list them with LI tags. And at the bottom some JavaScript snippet, which is responsible for picking up the language selected by the user and redirecting to the appropriate page with the appropriate language prefix.
You can easily modify the code snippet to suit your needs as it is just a way to show how it could be used.

Then create pages directory in the resources/views directory to have some partials.

resources/views/pages/home.blade.php default component for Home:



@extends('welcome')
@section('content')
    <p class="centered-text">{{ __('page.home') }}</p>
    <div class="centered-text">
        <a href="{{ route('about') }}" class="button">{{ __('page.about') }}</a>
        <a href="{{ route('contact') }}" class="button">{{ __('page.contact') }}</a>
        <a href="{{ route('dashboard') }}" class="button">{{ __('page.dashboard') }}</a>
    </div>
@endsection

resources/views/pages/about.blade.php component for About:



@extends('welcome')
@section('content')
    <p class="centered-text">{{ __('page.about') }}</p>
    <div class="centered-text">
        <a href="{{ route('home') }}" class="button">{{ __('page.home') }}</a>
        <a href="{{ route('contact') }}" class="button">{{ __('page.contact') }}</a>
        <a href="{{ route('dashboard') }}" class="button">{{ __('page.dashboard') }}</a>
    </div>
@endsection

resources/views/pages/contact.blade.php component for Contact:



@extends('welcome')
@section('content')
    <p class="centered-text">{{ __('page.contact') }}</p>
    <div class="centered-text">
        <a href="{{ route('home') }}" class="button">{{ __('page.home') }}</a>
        <a href="{{ route('about') }}" class="button">{{ __('page.about') }}</a>
        <a href="{{ route('dashboard') }}" class="button">{{ __('page.dashboard') }}</a>
    </div>
@endsection

resources/views/pages/dashboard.blade.php one more component for Dashboard. Later you can add some blades like this:



@extends('welcome')
@section('content')
    <p class="centered-text">{{ __('page.dashboard') }}</p>
    <div class="centered-text">
        <a href="{{ route('home') }}" class="button">{{ __('page.home') }}</a>
        <a href="{{ route('about') }}" class="button">{{ __('page.about') }}</a>
        <a href="{{ route('contact') }}" class="button">{{ __('page.contact') }}</a>
    </div>
@endsection

Now, for having a little bit of UI, we can use some ready styles from CodePen.
I will modify that assets a bit, so you can get those styles.css and scripts.js files from the public folder here.

At this point, as we have all the routes and views setup let’s create a config file specifically for the localizations-related stuff, create a custom middleware, register that middleware in the kernel to intercept all the web requests, and add some translations.

Create app/Http/Middleware/Localization.php:



<?php

namespace App\Http\Middleware;

use Closure;
use Illuminate\Http\Request;
use Illuminate\Support\Facades\App;
use Illuminate\Support\Facades\Redirect;
use Illuminate\Support\Facades\URL;

class Localization
{
    public function handle(Request $request, Closure $next)
    {
        $first_segment = $request->segment(1);
        if (in_array($first_segment, config('localization.locales'))) {
            App::setLocale($first_segment);
            URL::defaults(['locale' => $first_segment]);

            return $next($request);
        } else {
            $fallback_locale = config('app.fallback_locale');
            $segments = $request->segments();
            array_unshift($segments, $fallback_locale);

            return Redirect::to(implode('/', $segments));
        }
    }
}

This custom middleware, called Localization, is responsible for handling and setting the application’s locale (language) based on the first segment of the URL.

It extracts the first segment from the URL. For example, “en” in “example.com/en/page”.
It checks if this first segment corresponds to a supported locale as defined in the configuration config('localization.locales').
If the first segment is a valid locale, it sets the application’s locale using App::setLocale($first_segment), ensuring that the rest of the application operates in that language.
It also updates the URL to include the selected locale as a default for future generated URLs using URL::defaults(['locale' => $first_segment]).
If the first segment is not a valid locale, it redirects the user to the same URL but with the default (fallback) locale by prepending the fallback locale to the URL segments, as defined in config/app.php fallback_locale field. Of course, don’t forget to clear caches after each configuration change, running php artisan optimize.

Shortly, this custom middleware determines the application’s current language (locale) based on the first URL segment, setting it to a valid locale or redirecting to the default locale if the segment is not recognized.

So if we want to change default locale (language) in our app, we need to change the fallback locale in config/app.php:
'fallback_locale' => 'en' to 'fallback_locale' => ''.

Register the middleware in the web middleware group by adding the Localization class as a last array element of “web” sub-array of $middlewareGroups array:



protected $middlewareGroups = [
    'web' => [
        // EXISTING ONES
        \App\Http\Middleware\Localization::class, // APPEND THIS HERE
    ],
    'api' => [
        // EXISTING ONES
    ],
];

The reason we did that only just for web middleware group is that we don’t want to affect other existing routes, such as API endpoints which could be presented in our app.

Now, for having dynamic code in our app, we need to have some configuration file, where we can save all our locales (languages). We will create some config/localization.php specifically for the localizations-related stuff:



<?php

return [
    'locales' => [
        'en',
        'cn',
    ],
    'locale_lang' => [
        'en' => 'en_US',
        'cn' => 'zh_CN',
    ],
];

As you may already notice, in our app we will have English as a default (fallback) language, and Chinese as a secondary language. And we just use “locales” field in our dropdown HTML, to iterate and list all the locales.

Let’s create some translations as well, which we can use in our application.

Create lang/en/page.php (for Laravel 9 and old versions create resources/lang/en/page.php):



<?php

return [
    'locales' => [
        'code' => [
            'en' => "EN",
            'cn' => "中国人",
        ],
        'name' => [
            'en' => "English",
            'cn' => "中国人",
        ],
    ],

    'hello_world' => "Hello World",
    'change_language' => "Change Language",

    'home' => "Home",
    'about' => "About",
    'contact' => "Contact",
    'dashboard' => "Dashboard",
];

Create lang/cn/page.php (for Laravel 9 and old versions create resources/lang/cn/page.php).



<?php

return [
    'locales' => [
        'code' => [
            'en' => "中文",
            'cn' => "CN",
        ],
        'name' => [
            'en' => "English",
            'cn' => "中文",
        ],
    ],

    'hello_world' => "你好世界",
    'change_language' => "更改语言",

    'home' => "家",
    'about' => "关于",
    'contact' => "联系",
    'dashboard' => "仪表板",
];

At this point we have all the necessary things setup to have our app working. Just last time clear the caches and run the server:



# make sure always to refresh all the caches after each config file changes
php artisan optimize

# quick way to run and test our app
php artisan serve

We can now check out our app in the browser on http://localhost:8000

Make sure you have disabled automatic translation extensions (like Google Translate extension as in the picture):

That’s all for the setup.
Later, if you want to add some more translations, the one thing you just need to, to add your stuff in the translation files located in the lang directory as we have done already.

Sometimes things happen, when we need to add a completely new language to our existing app that will be supported as well as other languages.
The good part of this article is that we have developed a way to dynamically add new locales/languages.
In our test case we will add Armenian as a third language.

The only two steps you need to do to add a new language, that are:

Add a language code in the localization-specific configuration file (config/localization.php in our case):



<?php

return [
    'locales' => [
        'en',
        'cn',
        'am', // NEW LANGUAGE
    ],
    'locale_lang' => [ // OPTIONAL STUFF
        'en' => 'en_US',
        'cn' => 'zh_CN',
        'am' => 'hy_AM', // NEW LANGUAGE
    ],
];

Add the appropriate lang file like we did before. In our case we will create lang/am/page.php (for Laravel 9 and old versions create resources/lang/am/page.php)



<?php

return [
    'locales' => [
        'code' => [
            'en' => "中文",
            'cn' => "CN",
            'am' => "ՀԱՅ",
        ],
        'name' => [
            'en' => "English",
            'cn' => "中文",
            'am' => "Հայերեն",
        ],
    ],

    'hello_world' => "Բարև աշխարհ",
    'change_language' => "Փոխել լեզուն",

    'home' => "Տուն",
    'about' => "Մասին",
    'contact' => "Կապ",
    'dashboard' => "Վահանակ",
];

You optionally may modify your existing lang files as I need to do in my case as you can notice in the GitHub repo, but it depends on your case.
In my case I will do some small additions in ****/en/page.php* and in ****/cn/page.php*:



<?php

return [
    'locales' => [
        'code' => [
            // EXISTING CODE
            'am' => "ՀԱՅ", // ADDING THIS
        ],
        'name' => [
            // EXISTING CODE
            'am' => "Հայերեն", // ADDING THIS
        ],
    ],
    // EXISTING CODE
];

Finally… again, don’t forget to clear the caches by the artisan command.

That’s it.

Now we have integrated a custom and solid way to have Laravel multi-language support, as well as the way to add new languages dynamically.

In the end, I will put here the GitHub repo ⭐ for the project we built.

If you liked this article, feel free to follow me here. 😇

To explore projects working with various modern technologies, you can follow me on GitHub, where I actively publicize much of my work.

For more info you can visit my website boolfalse.com

Thank you !!!

JavaScript: language, engines, runtimes, environments (part 7/7)

boolfalse — Tue, 31 Oct 2023 08:00:11 +0000

NGNTJS - part 7

In the previous part 6 of NGNTJS article series we studied the event-loop mechanism using some practical examples and had understanding about the Reactor Pattern.
In this part, having already been informed about some terms, we will discuss about JavaScript language, engine, runtime, environment. Then we will get to know modern JavaScript runtimes/environments, such as Deno and Bun, and not only.

[Image from JSConfEU23 following talk]

So far we have talked about the JavaScript language, in particular the standard ECMAScript, which until the first decade of 2000s was only considered a browser-based scripting language.

Having already some information from previous parts of this article series, we know that JavaScript (TypeScript) language running in the browser is converting to the machine code to run the commands that are intended for the lower levels, that we already discussed in detail in part 2.
So the JavaScript engine's main functionality is to interpret the JavaScript code using the call stack (the storage that used for storing sequence of commands) and the memory heap (the storage for primitive and reference types that are needed during the code execution).

As we already know, different companies usually have their own versions of JavaScript engines. As an example, we can mention the V8 released by Google in 2008, which works in Chrome browser. Other browsers also have their own JavaScript engines but in all modern browsers the JavaScript language itself conforms to the ECMAScript standard with certain variations from different companies.

[The official mirror of the V8 Git repository on GitHub]

We also know that modern browsers often use various Web APIs, such as the well-known setTimeout(), setInterval(), relatively recently added fetch(), WebSockets, WebRTC, etc.
It means that these properties do not belong to the JavaScript language itself. And that are actually outside of the JavaScript engine's scope.
In addition to the methods described above, those methods need to use an event-loop and some necessary components associated with it.
And all of this, which is outside of the JavaScript language (engine) itself, includes the JavaScript runtime.
It means that the main components of JavaScript runtime are:

JavaScript engine: the call stack, the memory heap, and the language itself that manages them.
WebAPIs: setTimeout(), setInterval(), setImmediate(), fetch(), etc.
Event-loop and related mechanisms: toolkit for running WebAPIs and many other asynchronous operations.

In the early 2000s, when JavaScript still only ran in browsers, providing a dynamic interface for websites, engineers tried to get a standalone version of JavaScript runtime that would work outside of the browser environment. It can be assumed that the V8 engine released by Google in 2008 is the basis for Node.js being released by Ryan Dahl in 2009.

[Node.js official website]

So V8 is a JavaScript engine written in C++, based on all the necessary tools have been collected to run JavaScript outside the browser environment. And Node.js is a technology that allows to run JavaScript outside the browser environment, using it both in the server-side and as a CLI tool.
For this reason, Node.js is considered a runtime environment. In this sense, V8 can be considered as a framework for Node.js.

In part 2 we have mentioned some popular JavaScript engines, now it would be appropriate to mention following:

At a time, there were also attempts to have Node.js analogs using other engines: node-chakracore (Node.js on ChakraCore) and spidernode (Node.js on top of SpiderMonkey), but these are no longer maintained.
Usually, if we use Node.js with its V8 engine on the server, then a lightweight engine can be used in IoT devices intended for smaller loads, for example jerryscript (ultra-lightweight JavaScript engine for the Internet of Things) or Duktape (embeddable Javascript engine with a focus on portability and compact footprint).

As an "alternative" to Node.js, there are currently other JavaScript runtime environments, such as Deno.

[Image from deno official website]

Deno is the second popular project created by Ryan Dahl, which was officially announced at the 2018 JSConfEU talk by him. At the following conference in 2019, the author highlights the following key features:

Built on V8 engine (like Node.js).
It executes TypeScript and JavaScript, has an embedded TypeScript compiler. Node.js implements a separated compiler for TypeScript, using the typescript npm package. V8 Snapshots are used to make the TS compiler run faster.
Initially it was written in Go, then rewritten and now written in Rust (Node.js is written in C++). Compiling performs the following transformation: JavaScript -> TypeScript -> Rust
Tokio is used for the event-loop (as an analogue to libUV in Node.js).
It is cross-platform (like Node.js).
Works with single executable design (not like this in Node.js).
By default it has a secure execution mechanism for importing external resources, such as requesting URLs, opening sockets, etc (not available in Node.js). Unlike Node.js, where we could install 3rd party resources that could do unwanted actions (for example, read the contents of confidential files and send them out), here by design any action that needs permission is necessarily forced to get the appropriate permissions from the admin (in this case, the developer).
With the CLI argument, it is necessary to specify the resources that require permission to access, for example, with "--allow-read=/tmp" we allow reading the "/tmp" directory, or with "--allow-net=google.com" we allow connecting to google.com (not available in Node.js). Instead of "npm start" in Node.js, we have the following (as a sample): "deno run https://bit.ly/deno-bronto --allow-net --allow-read"
Compared to Node.js, not all Browser APIs are supported, Deno's purpose is different.
By design, Deno's 3rd party module system is distributive, as they are imported from relative or absolute URLs (pre-installed in Node.js), for example: import { serve } from "https://deno.land/std/http/server.ts"; this allows to avoid complex algorithms of "node_modules" and also not to have a package.json.
Packaging system is similar to Node.js, it fetches necessary modules at first, then uses the cached ones at other times, until we try to update them.
If npm was considered as an add-on in Node.js, in Deno npm is implemented as npm-client.
Unlike Node.js, where the file can also be used as a module (multiple submodules in it) and import specific submodules from the exported module, in Deno it is strict, the file can only be used as a module: Module == File == URL
Unlike monolithic Node.js, Deno is a collection of Rust crates that allows for standalone executables. Cargo-TypeScript integration crate is presented as a sample, which allows you to compile TS in V8 Snapshots and have a quick startup.
Deno team has also released Deno Fresh framework. Shortly, it is little more than express.js for Node.js. It has built-in front-end rendering support that uses preact and JSX.

[Image from deno.com]

In addition, I think it's worth having some updates here from the 2023 NodeCongress JavaScript Conferences by GitNation:

Deno 2.0 has a KV (key-value) built-in database based on SQLite. This makes it possible to have a ready-made database with already implemented tools without using additional databases.
- KV stores JavaScript objects;
- necessary get, set, list, delete commands are available;
Until Deno 2.0 is released, you can use the following tool to work with KV.
Deno team has prepared the Deno-Deploy tool to encourage the distribution idea, which allows to deploy and have a real working app.
Deno-Deploy runs on top of FoundationDB and has the following features:
- ability to integrate public and private GitHub repos;
- works in 35+ edge locations worldwide;
- free *.deno.dev subdomain and custom domains are provided;
- free and automatic HTTPS / TLS;
- unlimited production deployments and previews;
- up to 10ms/request CPU provisioning;

Deno's philosophy is also to encourage the distributive idea. According to the creator, the idea is the same:

Node: Restricting programs to async I/O primitives help developers build optimal local servers.
Deno: Restricting programs to distributed cloud primitives help developers build optimal global servers.

It should be noted that according to Ryan Dahl, Deno 2.0 was supposed to be released in the summer of 2023, but it is already delayed, but even so, the last described points refer specifically to Deno 2.0.

As an "alternative" to the Node.js and Deno projects, other JavaScript runtime environments such as Bun have recently become popular.

[Image from Bun official website]

Bun is an open-source project created by Jarred Sumner. During the 2022 meeting the author highlights following key features:

[Note: because the software changes over time, that's why I won't mention here exactly how many times or by what percentage Bun is faster than this or that function of Node.js. The author often tweets on his page about that.]

Bun is a modern all-in-one JavaScript runtime environment. Bun has the basic toolkit to start JS/TS code faster than Node.js. It has a built-in transpiler for JS/TS code, a built-in npm package manager, a front-end dev server, and of course a runtime.
Implemented more optimized Node.js-API such as fs.copyFile and fs.writeFile methods, TextEncoder.encode, buffer.toString('bas64') etc., which work faster.
bun install installs npm packages faster.
bun run starts package.json scripts faster.
Node.js SQLite is faster in Bun: bun:sqlite.
Bun uses JavaScriptCore engine, unlike Node.js which uses V8 engine.
Bun is mostly written in Zig unlike Node.js, which is written in C++.
The author considers the previous two points as the basis of Bun's fast work, but in addition to them, he also mentions the following reasons:
- Bun uses system calls much more flexibly;
- Bun uses modern technologies such as SIMD (Single Instruction Multiple Data), which allows maximum use of CPU speed capabilities (now applied also in Node.js);
- quite a lot of time was spent on testing and optimization;
Bun 1.0 is scheduled to be released this year (September 2023), which will be a stable version with as few bugs as possible;

Overall, Bun, being written in Rust, brings with it a new philosophy, as the author notes:
"in Bun your JavaScript code isn't I/O bound, it's system-call bound."

I think it's also worth mentioning about JavaScript runtime created by Microsoft Napa.js.

[microsoft/napajs]

Although it has not been maintained since 2018 and work on it has stopped, a significant feature of this runtime, unlike Node.js and alternative runtimes, it was designed to handle multi-threaded, CPU-intensive tasks. There is an opinion, that Napa is mostly abandoned and predates Node.js v10 having native multi-threaded support and worker_threads. Anyway, it's worth knowing that in Napa.js the multi-thread concept is implemented by running V8 engines isolated from each other, each occupying the corresponding thread. It was possible to use it both as a Node.js module and as a separate standalone service, not being a Node dependency.
There is an article about napa.js here.

Below are links to useful resources that did not appear directly in this section, but are related to this topic:

Node vs Deno vs Bun: The search for a better JavaScript runtime environment [Oct, 2022]
Choosing a JavaScript Runtime for 2023: Node vs. Bun vs. Deno [Mar, 2023]
Bun vs. Node.js [Jun, 2023]
What is Deno?: Deno vs Node.js [Jul, 2023]

In this part of the NGNTJS article series, we have discussed about JavaScript language, engine, runtime, and environment. We got information about Node.js alternatives, such as Deno and Bun JavaScript runtime/environments, and more.

If you liked this article, feel free to follow me here. 😇

To explore projects working with various modern technologies, you can follow me on GitHub, where I actively publicize much of my work.

For more information, you can visit my website: https://boolfalse.com/

Node.js: event-loop with practical examples. A big picture (part 6/7)

boolfalse — Tue, 31 Oct 2023 07:59:59 +0000

NGNTJS - part 6

In the previous part 5 of NGNTJS article series we had some understanding about thread pool with a practical real life example.
In this part, we will look at different examples (step by step) and through these examples we will try to understand how event-loop works in practice.

[Image from unsplash.com]

Before preparing this series of articles, and especially this part, a lot of material was studied, in which the following logical sequence can be observed: first, what is an event-loop, and then how it works.
Of course, there is no transition here either (previous: part 5, but as an author, I think it's worth starting to explore event-loop with practical examples, to understand how it works, and then get a bigger picture of what it actually is.

Example 1․ microtask and timer queues

setTimeout(() => console.log("setTimeout 1"), 0);
setTimeout(() => {
  console.log("setTimeout 2");
  process.nextTick(() => console.log("setTimeout 2 nextTick"))
}, 0);
setTimeout(() => console.log("setTimeout 3"), 0);

process.nextTick(() => console.log("nextTick 1"));
process.nextTick(() => {
  console.log("nextTick 2");
  process.nextTick(() => console.log("nextTick 2 nextTick"));
});
process.nextTick(() => console.log("nextTick 3"));

Promise.resolve().then(() => console.log("Promise 1"));
Promise.resolve().then(() => {
  console.log("Promise 2");
  process.nextTick(() => console.log("Promise 2 nextTick"));
});
Promise.resolve().then(() => console.log("Promise 3"));

The result will be:

nextTick 1
nextTick 2
nextTick 3
nextTick 2 nextTick
Promise 1
Promise 2
Promise 3
Promise 2 nextTick
setTimeout 1
setTimeout 2
setTimeout 3
setTimeout 2 nextTick

Example 2․ I/O queue and timer queue anomaly

const fs = require("fs");

setTimeout(() => console.log("setTimeout"), 0);
fs.readFile(__filename, () => console.log("readFile"));
// for (let i = 0; i < 1000000000; i++) {}

The result will be something like this:

setTimeout
readFile
// anomaly
readFile
setTimeout

Example 3․ I/O queue and I/O polling

const fs = require("fs");

fs.readFile(__filename, () => {
  console.log("readFile");
});
process.nextTick(() => console.log("nextTick"));
Promise.resolve().then(() => console.log("Promise"));
setTimeout(() => console.log("setTimeout"), 0);
setImmediate(() => console.log("setImmediate"));

for (let i = 0; i < 1000000000; i++) {}

The result will be:

nextTick
Promise
setTimeout
setImmediate
readFile

Example 4․ check queue & timer queue anomaly

setTimeout(() => console.log("setTimeout"));
setImmediate(() => console.log("setImmediate"));

The result will be something like this:

setTimeout
setImmediate
// anomaly
setImmediate
setTimeout

Example 5․ Close queue in the end

const fs = require("fs");

const readableStream = fs.createReadStream(__filename);
readableStream.on("close", () => {
  console.log("readableStream");
});
readableStream.close();

setImmediate(() => console.log("setImmediate"));
setTimeout(() => console.log("setTimeout"), 0);
Promise.resolve().then(() => console.log("Promise"));
process.nextTick(() => console.log("nextTick"));

The result will be:

nextTick
Promise
setTimeout
setImmediate
readableStream

You can also find examples on GitHub Gist.

In these examples, the microtask (nextTick and Promise), timer, I/O (I/O polling), check and close queues are sequentially passed.

Now, since here we already understand what the event-loop mechanism is, and we know the stages of work, we can look at the Node.js architecture in a "big" picture.
Using the keyword "Node.js architecture" on the internet, you can find many visuals, which are made by different authors according to certain requirements, to represent this or that component or the nature of the work they want.
Below I will post the visual made by me, which presents the work in Node.js Runtime in a high-level. It more oriented towards a logical chain than individual program blocks.

[Interaction between libUV's event-loop and other components in the Node.js runtime]

Let's briefly describe the meaning of the new terms mentioned in the visual, that not yet explained in the articles before:

event-demultiplexer - responsible for monitoring various I/O sources and receiving events. If the I/O source requires a blocking operation, event-DEMUX will pass the given task to the C/C++ field, where the given task will be executed. Once the task is finished, the C/C++ side will notify the event-dispatcher, which in turn will send it to the event-loop to be added to a specific queue.
event-dispatcher - is responsible for event management, sending and receiving events
kqueue - BSD-based mechanism for event notifications (macOS, FreeBSD, NetBSD)
IOCP (input/output completion ports) - BSD-based mechanism for the event notifications (macOS, FreeBSD, NetBSD)
event ports - Solaris-specific mechanism for the event notifications (Oracle Solaris, OpenSolaris)
epoll - A Linux-specific mechanism for the event notifications

In addition, libUV provides other useful features in apart from above described features:

Asynchronous I/O - libUV provides a common interface for asynchronous I/O operations across different OSs.
Signal handling - libUV provides an API: Signal handlers are registered to handle OS signals such as SIGINT or SIGTERM.
File system operations - libUV provides an API to perform file system operations such as file writing or file reading.
Network I/O - libUV provides an API to perform network I/O operations, such as creating and managing TCP/UDP sockets.
Child processes management - libUV provides an API for creating and managing child_processes.
...

Coming to this point, we can understand the idea of Reactor Pattern.

[Image from the mini-series Chernobyl]

In this part of the series of NGNTJS articles we studied the event-loop mechanism through practical examples, got an understanding about the Reactor Pattern.
Next, in part 7 we will discuss JavaScript language, engine, runtime and environment. Then we will get to know modern JavaScript runtime/environments, such as Deno and Bun, and much more.

If you liked this article, feel free to follow me here. 😇

To explore projects working with various modern technologies, you can follow me on GitHub, where I actively publicize much of my work.

For more information, you can visit my website: https://boolfalse.com/

Node.js: thread pool, real life analogy (part 5/7)

boolfalse — Tue, 31 Oct 2023 07:59:37 +0000

NGNTJS - part 5

In the previous part 4 of the NGNTJS article series, we looked at the libUV library, we had understanding about the event-loop mechanism.
This part will be about a feature called "thread pool" in libUV.

[Image from unsplash.com]

Thread pool is designed to perform javascript blocking operations. As you may already know, there is one thread for JavaScript execution (as a runtime in Node.js as well), and only one operation is performed in that thread at a given moment in time. When blocking operations are encountered during code execution, such as the following methods:

fs (fs.createReadStream(), fs.chown(), fs.copyFile())
crypto (crypto.scrypt(), crypto.sign(), crypto.verify())
dns (dns.lookup(), dns.resolveCname() )

or many other asynchronous methods, additional threads are "opened" in order not to overload the work of the single thread. In fact, pooling and using each thread is quite resource-consuming, and this happens in cases where there is "no other option" in order not to block the work of the master thread.

[Manufacturing single lane]

To better understand thread pooling from the master thread, let's try to use a similar analogy from real life:

Consider a senior factory worker who monitors the movement of packaged goods passing through the line, and he is responsible for the weight labels on the packaged goods. In the course of work, there are cases when the packaged product is quite heavy and that senior employee is not able to weigh it. For these cases, he has assigned junior assistants who can pick up that heavy package, weigh it on a large scale, tell the senior employee the weight so he can label the package and place it in the goods section of the line. Regardless of the weight, a junior employee is responsible for weighing any heavy item. Junior employees are young assistants who are not idle, but work in different sub-sections of the factory and are ready to help with any questions at any time of inquiry.
The senior employee is responsible for labeling, he follows the line, in case of heavy goods he asks assistants to help, and weighs light goods by himself. The senior employee generally tries not to ask for additional help from the assistants, since the assistant employees are also doing some work in different parts of the factory, perhaps following the senior employee to reach him in an emergency if necessary. However, according to the position, the senior employee has 4 assistants attached to him, who will come to help. Of course, the senior addresses them as needed, and as little as possible.
Thus, according to the labor regulations, appropriate action will be applied to the following cases registered during work:

The senior appeals to 1 junior for only 1 heavy load.
The senior applies to the 2 juniors at the same time for 2 heavy loads in a row.
The senior applies to the 3 juniors at the same time for 3 consecutive heavy loads.
The senior applies to the 4 juniors at the same time for 4 consecutive heavy loads.
The senior cannot call on another assistant for more than 4 consecutive loads. It means that in such cases, 4 assistants take their loads in order to weigh them, and the one who finishes weighing earlier and tells the weight to the senior, continues to take the next heavy load without waiting for the others. And at some point, when one of them is freed and sees that there is no heavy load left, he immediately leaves with his affairs without waiting for the other assistants.

There are cases when the workload increases and the senior employee needs to use the help of additional junior employees. In such cases, there is an opportunity for the director to change the regulations and allow the senior employee to apply to more than 4 employees at the same time (of course, he can also reduce it).

This fictional example from real life was intended to make a comparison between the following terms:

master thread - senior employee
thread - every employee (basically it's about assistants)
thread pool - application to support staff
UV_THREADPOOL_SIZE = 4 - the number of helper assistants by default (this also depends on the OS)
Node.js runtime - factory director
sync / async non-blocking operations - loads that can be handled by a senior employee
async & blocking CPU-intensive operations - loads that a senior employee assigns assistant employees to perform and report the result

Increasing UV_THREADPOOL_SIZE will reach a point where it no longer has an effect. That's why you need to know about computer threads.

The number of central processing units (CPUs) and cores (cores) of modern computers can vary depending on the physical structure, but they usually have a single CPU, while some servers as well as HPC (high-performance computing) systems can have multiple processors.
The number of cores per CPU can also vary greatly from 2 / 4 / 8 / 10 / 12 / 16 / 18 / 36 and more, but modern processors usually have 2 (dual-core) or 4 (quad-core) cores.
Modern CPUs often use a technology called hyper-threading, which allows each CPU core to provide multiple threads simultaneously, usually up to 2 (in some cases 4).

[Concurrency and Multithreading]

Now having a little understanding of the CPU-core-thread interrelationship, we can continue exploring the thread pool mechanism.

Let's consider a classic case - the work of Node.js on a modern computer, where there is a need for thread pooling. In the critical case, when Node.js needs to perform many CPU-intensive operations, maximum threads are provided by the cores to perform these operations. It means that in a quad-core CPU we will have 8 threads, and the thread pool mechanism can theoretically use those 8 threads at the maximum (in this case, full CPU resources will be used).
It is recommended to have UV_THREADPOOL_SIZE less than the number of threads possible in the given CPU, but the ideal case is when for a CPU with 4 cores (with the possibility of maximum 8 threads) we have UV_THREADPOOL_SIZE exactly 4, i.e. exactly the number of cores, in order to avoid OS workload.

It should also be noted that thread pool is not performed for all asynchronous operations.
For example, https.request() is a Network I/O operation, not a CPU-intensive operation, and it does not use a thread pool.
Instead of thread pooling, libUV tries to handle the task directly through the OS kernel using the Network Card (however, this has its limitations).
In fact, async methods in Node.js are handled by libUV, but they are handled in two different ways: either by the native async mechanism (e.g. https.request()) or by the thread pool (fs.createReadStream()).
At least if possible, libUV tries to avoid using thread pool, saving resources in the environment, and in critical cases, it uses additional threads, sticking to its function, which is to maintain non-blocking (async) work.

As we mentioned, one thread (single-threaded) is intended for JavaScript execution. But while saying this, we are talking about the JavaScript engine working in the browser. Of course, this also applies to Node.js, but it's also worth knowing that there are multi-thread runtime environments as well, such as Microsoft's Napa.js, which was discontinued in 2018. We will talk about that later in part 7.

In this part of the NGNTJS article series, we have some idea about thread pooling.
Next, in part 6 we will explore the workflow of the event-loop mechanism in a practical example. Also, there will be an introduction to Reactor Pattern.

If you liked this article, feel free to follow me here. 😇

To explore projects working with various modern technologies, you can follow me on GitHub, where I actively publicize much of my work.

For more information, you can visit my website: https://boolfalse.com/