Forem: Hantsy Bai

Upgrade to Symfony v7

Hantsy Bai — Wed, 28 Feb 2024 04:37:56 +0000

Symfony has just released Symfony 7.0, there are a lot of breaking changes introduced in this version.

In this post, we will use Symfony Rest Example project as an example, and show how I upgrade to v7 step by step.

Backup Existing Codes

Firstly, let's create a GIT tag on the existing codes, and back up the project.

git tag v6.x 
git push origin v6.x

For those still use Symfony 6.x, please check the tag v6.x or download a copy from the project tag page.

Create a new branch to prepare the upgrading work.

git checkout -b v7

Upgrade Symfony Packages to v7

Import the project source codes into an IDE(PHPStorm or NetBeans IDE) or VSCode( with PHP support).

Open composer.json file, change all symfony packages version to 7.0.*.

{
  "name": "hantsy/symfony-rest-sample",
  "description": "Restful APIs examples built with Symfony 7 and PHP 8",
  "authors": [
    {
      "name": "Hantsy Bai",
      "email": "hantsy@gmail.com"
    }
  ],
  "type": "project",
  "license": "GPL-3.0-or-later",
  "minimum-stability": "stable",
  "prefer-stable": true,
  "require": {
    "php": ">=8.2",
    "ext-ctype": "*",
    "ext-iconv": "*",
    "composer/package-versions-deprecated": "^1.11.99.4",
    "doctrine/annotations": "^2.0",
    "doctrine/doctrine-bundle": "^2.4",
    "doctrine/doctrine-migrations-bundle": "^3.1",
    "doctrine/orm": "^2.9",
    "phpdocumentor/reflection-docblock": "^5.2",
    "symfony/asset": "7.0.*",
    "symfony/console": "7.0.*",
    "symfony/dotenv": "7.0.*",
    "symfony/expression-language": "7.0.*",
    "symfony/flex": "^2.0.1",
    "symfony/framework-bundle": "7.0.*",
    "symfony/monolog-bundle": "^3.7",
    "symfony/property-access": "7.0.*",
    "symfony/property-info": "7.0.*",
    "symfony/runtime": "7.0.*",
    "symfony/serializer": "7.0.*",
    "symfony/uid": "7.0.*",
    "symfony/validator": "7.0.*",
    "symfony/yaml": "7.0.*"
  },
  "config": {
    "optimize-autoloader": true,
    "preferred-install": {
      "*": "dist"
    },
    "allow-plugins": true,
    "sort-packages": true
  },
  "autoload": {
    "psr-4": {
      "App\\": "src/"
    }
  },
  "autoload-dev": {
    "psr-4": {
      "App\\Tests\\": "tests/"
    }
  },
  "replace": {
    "symfony/polyfill-ctype": "*",
    "symfony/polyfill-iconv": "*",
    "symfony/polyfill-php72": "*"
  },
  "scripts": {
    "auto-scripts": {
      "cache:clear": "symfony-cmd",
      "assets:install %PUBLIC_DIR%": "symfony-cmd"
    },
    "post-install-cmd": [
      "@auto-scripts"
    ],
    "post-update-cmd": [
      "@auto-scripts"
    ],
    "test": "php bin/phpunit"
  },
  "conflict": {
    "symfony/symfony": "*"
  },
  "extra": {
    "symfony": {
      "allow-contrib": false,
      "require": "7.0.*"
    }
  },
  "require-dev": {
    "dama/doctrine-test-bundle": "^8.0",
    "doctrine/doctrine-fixtures-bundle": "^3.4",
    "phpunit/phpunit": "^9.5",
    "symfony/browser-kit": "7.0.*",
    "symfony/css-selector": "7.0.*",
    "symfony/maker-bundle": "^1.34",
    "symfony/phpunit-bridge": "7.0.*"
  }
}

We remove sensio/framework-extra-bundle and nelmio/cors-bundle from the package list due to the compatibility with the new v7.

The former is deprecated in v7 and not recommended in new projects. We will use new API to replace them.
The later does not release a Symfony v7 compatible version at the moment.

Remove composer.lock, try to run composer install command in a terminal to install new packages and rerun the built-in recipes for this project.

Refresh Codes with New APIs

There are a few classes that need to align with the new APIs in Symfony v7.

Firstly, we used ArgumentResolver in sensio/framework-extra-bundle to convert parameters to Uuid. In Symfony v7, the ValueResolver is the replacement. More details please check Extending Action Argument Resolving.

And there is a built-in UidValueResolver can be used to convert the parameter to Uuidv4, so we can give up our custom Uuid converter, and switch to use the official one.

Simply remove the existing src/ParamConverter/UuidParamConverter.

Next, let's update the BodyValueResolver and QueryParamValueResolver. The original ArgumentValueResolverInterface is removed in v7, the replacement is ValueResolverInterface, which is similar to the old ArgumentValueResolverInterface, but there is no supports method to override.

We adjust the existing codes slightly to make it work seamlessly.

class QueryParamValueResolver implements ValueResolverInterface, LoggerAwareInterface
{
    public function __construct()
    {
    }

    private LoggerInterface $logger;

    /**
     * @inheritDoc
     */
    public function resolve(Request $request, ArgumentMetadata $argument): iterable
    {
        if (!$this->supports($request, $argument)) return [];
        ...
    }

We reuse supports method to determine if skip the execution of argument resolving. Similarly, we adjust the BodyValueResolver as expected.

Rerun composer install, we get an error like the following.

!!  Symfony\Component\ErrorHandler\Error\FatalError {#307
!!    #message: "Compile Error: Declaration of App\Annotation\Delete::getMethods() must be compatible with Symfony\Component\Routing\Attribute\Route::getMethods(): array"
!!    #code: 0
!!    #file: "D:\hantsylabs\symfony5-sample\src\Annotation\Delete.php"
!!    #line: 11
!!    -error: array:4 [
!!      "type" => 64
!!      "message" => "Declaration of App\Annotation\Delete::getMethods() must be compatible with Symfony\Component\Routing\Attribute\Route::getMethods(): array"
!!      "file" => "D:\hantsylabs\symfony5-sample\src\Annotation\Delete.php"
!!      "line" => 11
!!    ]
!!  }
!!  PHP Fatal error:  Declaration of App\Annotation\Delete::getMethods() must be compatible with Symfony\Component\Routing\Attribute\Route::getMethods(): array in D:\hantsylabs\symfony5-sample\src\Annotation\Delete.php on line 11

In Symfony v7, all methods and properties in classes require type declaration. For developers, it is good to write type safe codes.

Symfony provides a simple script to upgrade existing codes and apply type declaration automatically.

vendor/bin/patch-type-declarations

More information about the type declaration, check Symfony 7.0 Type Declarations.

Rerun composer install, we get an new error like the following.

!!  In Loader.php line 63:
!!                                                                                 
!!    Cannot load resource "../../src/Controller/". Make sure there is a loader   
!!    supporting the "annotation" type.

We should use the new attributes config to replace the legacy annotations config for loading controllers.

Remove config/routes/annotations.yaml.

Create a new file config/routes/attributes.yaml, fill in the following content.

controllers:
    resource:
        path: ../../src/Controller/
        namespace: App\Controller
    type: attribute

kernel:
    resource: App\Kernel
    type: attribute

More information about Route, check Creating Routes as Attributes.

Now the composer install command should be executed successfully.

When running the tests, there are several deprecated warning info.

The Get, Post, Put, Delete, etc. that we created as examples of demonstrating attributes now raise warning info like the Route could be final in future. Thus means in a further version, these custom attributes will not work.

Remove the entire src\Annotations folder.
Open the PostController, use original Route attribute instead.

Note, if you use Route from Symfony\Component\Routing\Annotation\Route, change it to use the new Symfony\Component\Routing\Attribute\Route instead.

The left another two warning info are from Doctrine configuration.

Open config/packages/doctrine.yaml, add the following two lines.

doctrine:
    dbal:
        url: '%env(resolve:DATABASE_URL)%'
        use_savepoints: true  // add this line
        ...
    orm:
        auto_generate_proxy_classes: true
        enable_lazy_ghost_objects: true // add this line
        ...

I have not researched the new changes in Doctrine, these configuration is provided in the warning info.

Now run the tests, all tests should be passed.

Upgrade to PHPUnit 10

Run the following command to update PHPUnit to 10.x.

composer recipes:update symfony/phpunit-bridge

Then update the PHPUnit packages to ^10.0.

composer install

When running the tests, there are some deprecation information in the test result.

Run the following command to migrate the configuration to PHPUnit 10 compatible format.

vendor/bin/phpunit --migrate-configuration

You will find the Symfony test listener is removed by this script, because listener is not a valid element in PHPUnit 10 configuration. Let's wait for a new replacement in a further version.

Manually add a source element into the phpunit.xml.dist.

  <source>
    <include>
        <directory suffix=".php">src</directory>
    </include>
  </source>

Now run the tests again, all tests should be passed.

Update PHP versions on Github Actions

Finally, we should update the PHP version in Github actions workflow file, drop PHP 8.1 from the versoin list, and add PHP 8.3 as build environment. PHP 8.1 is no longer supported in the new Symfony v7.

The working codes is available via my Github, check out the symfony-rest-sample and explore it yourself.

# Building Restful APIs with Symfony 5 and PHP 8

Hantsy Bai — Wed, 24 Nov 2021 03:36:29 +0000

Symfony is a full-featured modularized PHP framework which is used for building all kinds of applications, from traditional web applications to the small Microservice components.

Get your feet wet

Install PHP 8 and PHP Composer tools.

# choco php composer

Install Symfony CLI, check the system requirements.

# symfony check:requirements

Symfony Requirements Checker
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

> PHP is using the following php.ini file:
C:\tools\php80\php.ini

> Checking Symfony requirements:

....................WWW.........


 [OK]                                         
 Your system is ready to run Symfony projects 


Optional recommendations to improve your setup
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 * intl extension should be available
   > Install and enable the intl extension (used for validators).

 * a PHP accelerator should be installed
   > Install and/or enable a PHP accelerator (highly recommended).

 * realpath_cache_size should be at least 5M in php.ini
   > Setting "realpath_cache_size" to e.g. "5242880" or "5M" in
   > php.ini* may improve performance on Windows significantly in some
   > cases.


Note  The command console can use a different php.ini file
~~~~  than the one used by your web server.
      Please check that both the console and the web server
      are using the same PHP version and configuration.

According to the recommendations info, adjust your PHP configuration in the php.ini. And we will use Postgres as database in the sample application, make sure pdo_pgsql and pgsql modules are enabled.

Finally, you can confirm the enabled modules by the following command.

# php -m

Create a new Symfony project.

# symfony new rest-sample

// a classic website application
# symfony new web-sample --full

By default, it will create a simple Symfony skeleton project only with core kernel configuration, which is good to start a lightweight Restful API application.

Alternatively, you can create it using Composer.

# composer create-project symfony/skeleton rest-sample

//start a classic website application
# composer create-project symfony/website-skeleton web-sample

Enter the generated project root folder, start the application.

# symfony server:start

 [WARNING] run "symfony.exe server:ca:install" first if you want to run the web server with TLS support, or use "--no-  
 tls" to avoid this warning                                                                                             

Tailing PHP-CGI log file (C:\Users\hantsy\.symfony\log\499d60b14521d4842ba7ebfce0861130efe66158\79ca75f9e90b4126a5955a33ea6a41ec5e854698.log)
Tailing Web Server log file (C:\Users\hantsy\.symfony\log\499d60b14521d4842ba7ebfce0861130efe66158.log)

 [OK] Web server listening                                                                                              
      The Web server is using PHP CGI 8.0.10                                                                            
      http://127.0.0.1:8000                                                                                             


[Web Server ] Oct  4 13:33:01 |DEBUG  | PHP    Reloading PHP versions
[Web Server ] Oct  4 13:33:01 |DEBUG  | PHP    Using PHP version 8.0.10 (from default version in $PATH)
[Web Server ] Oct  4 13:33:01 |INFO   | PHP    listening path="C:\\tools\\php80\\php-cgi.exe" php="8.0.10" port=61738

Hello , Symfony

Create a simple class to a resource entity in the HTTP response.

class Post
{
    private ?string $id = null;

    private string $title;

    private string $content;

    //getters and setters.
}

And use a factory to create a new Post instance.

class PostFactory
{
    public static function create(string $title, string $content): Post
    {
        $post = new Post();
        $post->setTitle($title);
        $post->setContent($content);
        return $post;
    }
}

Let's create a simple Controller class.

To use the newest PHP 8 attributes to configure the routing rules, apply the following changes in the project configurations.

Open config/packages/doctrine.yaml, remove doctrine/orm/mapping/App/type or change its value to attribute
Open composer.json, change PHP version to >=8.0.0.

To render the response body into a JSON string, use a JsonReponse to wrap the response.

#[Route(path: "/posts", name: "posts_")]
class PostController
{

    #[Route(path: "", name: "all", methods: ["GET"])]
    function all(): Response
    {
        $post1 = PostFactory::create("test title", "test content");
        $post1->setId("1");

        $post2 = PostFactory::create("test title", "test content");
        $post2->setId("2");
        $data = [$post1->asArray(), $post2->asArray()];
        return new JsonResponse($data, 200, ["Content-Type" => "application/json"]);
        //return $this->json($data, 200, ["Content-Type" => "application/json"]);
    }
}

The first parameter of JsonReponse accepts an array as data, so add a function in the Post class to archive this purpose.

class Post{
    //...
    public function asArray(): array
    {
        return [
            'id' => $this->id,
            'title' => $this->title,
            'content' => $this->content
        ];
    }
}

Run the application, use curl to test the /posts endpoint.

# curl http://localhost:8000/posts

Symfony provides a simple AbstractController which includes several functions to simplfy the response and adopt the container and dependency injection management.

In the above controller, extends from AbstractController, simply call $this->json to render the response in JSON format, no need to transform the data to an array before rendering response.


class PostController extends AbstractController
{

    function all(): Response
    {
        //...
        return $this->json($data, 200, ["Content-Type" => "application/json"]);
    }
}

Connecting to Database

Doctrine is a popular ORM framework , it is highly inspired by the existing Java ORM tooling, such as JPA spec and Hibernate framework. There are two core components in Doctrine, doctrine/dbal and doctrine/orm, the former is a low level APIs for database operations, if you know Java development, consider it as the Jdbc layer. The later is the advanced ORM framework, the public APIs are similar to JPA/Hibernate.

Install Doctrine into the project.

# composer require symfony/orm-pack
# composer require --dev symfony/maker-bundle

The pack is a virtual Symfony package, it will install a series of packages and basic configurations.

Open the .env file in the project root folder, edit the DATABASE_URL value, setup the database name, username, password to connect.

DATABASE_URL="postgresql://user:password@127.0.0.1:5432/blogdb?serverVersion=13&charset=utf8"

Use the following command to generate a docker compose file template.

# php bin/console make:docker:database

We change it to the following to start up a Postgres database in development.

version: "3.5" # specify docker-compose version, v3.5 is compatible with docker 17.12.0+

# Define the services/containers to be run
services:

  postgres:
    image: postgres:${POSTGRES_VERSION:-13}-alpine
    ports:
      - "5432:5432"
    environment:
      POSTGRES_DB: ${POSTGRES_DB:-blogdb}
      # You should definitely change the password in production
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-password}
      POSTGRES_USER: ${POSTGRES_USER:-user}
    volumes:
      - ./data/blogdb:/var/lib/postgresql/data:rw
      - ./pg-initdb.d:/docker-entrypoint-initdb.d

We will use UUID as data type of the primary key, add a script to enable uuid-ossp extension in Postgres when it is starting up.

-- file: pg-initdb.d/ini.sql
SET search_path TO public;
DROP EXTENSION IF EXISTS "uuid-ossp";
CREATE EXTENSION "uuid-ossp" SCHEMA public;

Open config/packages/test/doctrine.yaml, comment out dbname_suffix line. We use Docker container to bootstrap a database to ensure the application behaviors are same between the development and production.

Now startup the application and make sure there is no exception in the console, that means the database connection is successful.

symfony server:start

Before starting the application, make sure the database is running. Run the following command to start up the Postgres in Docker.

# docker compose up postgres
# docker ps -a # to list all containers and make the postgres is running

Building Data Models

Now we will build the Entities that will be used in the next sections. We are modeling a simple blog system, it includes the following concepts.

A Post presents an article post in the blog system.
A Comment presents the comments under a specific post.
The common Tag can be applied on different posts, which categorizes posts by topic, categories , etc.

You can draft your model relations in mind or through some graphic data modeling tools.

Post and comments is a one-to-many relation
Post and tag is a many-to-many relation

It is easy to convert the idea to real codes via Doctrine Entity. Run the following command to create Post, Comment and Tag entities.

In the Doctrine ORM 2.10.x and Dbal 3.x, the UUID type ID generator is deprecated. We will switch to the Uuid form symfony\uid.

Install symfony\uid firstly.

# composer require symfony/uid

Simply, you can use the following command to create entities quickly.

# php bin/console make:entity  # following the interactive steps to create them one by one.

Finally we got three entities in the src/Entity folder. Modify them as you expected.

// src/Entity/Post.php
#[Entity(repositoryClass: PostRepository::class)]
class Post
{
    #[Id]
    //#[GeneratedValue(strategy: "UUID")
    //#[Column(type: "string", unique: true)]
    #[Column(type: "uuid", unique: true)]
    #[GeneratedValue(strategy: "CUSTOM")]
    #[CustomIdGenerator(class: UuidGenerator::class)]
    private ?Uuid $id = null;

    #[Column(type: "string", length: 255)]
    private string $title;

    #[Column(type: "string", length: 255)]
    private string $content;

    #[Column(name: "created_at", type: "datetime", nullable: true)]
    private DateTime|null $createdAt = null;

    #[Column(name: "published_at", type: "datetime", nullable: true)]
    private DateTime|null $publishedAt = null;

    #[OneToMany(mappedBy: "post", targetEntity: Comment::class, cascade: ['persist', 'merge', "remove"], fetch: 'LAZY', orphanRemoval: true)]
    private Collection $comments;

    #[ManyToMany(targetEntity: Tag::class, mappedBy: "posts", cascade: ['persist', 'merge'], fetch: 'EAGER')]
    private Collection $tags;

    public function __construct()
    {
        $this->createdAt = new DateTime();
        $this->comments = new ArrayCollection();
        $this->tags = new ArrayCollection();
    }
    //other getters and setters
}

// src/Entity/Comment.php
#[Entity(repositoryClass: CommentRepository::class)]
class Comment
{
    #[Id]
    //#[GeneratedValue(strategy: "UUID")]
    #[Column(type: "uuid", unique: true)]
    #[GeneratedValue(strategy: "CUSTOM")]
    #[CustomIdGenerator(class: UuidGenerator::class)]
    private ?Uuid $id = null;

    #[Column(type: "string", length: 255)]
    private string $content;

    #[Column(name: "created_at", type: "datetime", nullable: true)]
    private DateTime|null $createdAt = null;

    #[ManyToOne(targetEntity: "Post", inversedBy: "comments")]
    #[JoinColumn(name: "post_id", referencedColumnName: "id")]
    private Post $post;

    public function __construct()
    {
        $this->createdAt = new DateTime();
    }
    //other getters and setters
}

//src/Entity/Tag.php
#[Entity(repositoryClass: TagRepository::class)]
class Tag
{
    #[Id]
    //#[GeneratedValue(strategy: "UUID")
    //#[Column(type: "string", unique: true)]
    #[Column(type: "uuid", unique: true)]
    #[GeneratedValue(strategy: "CUSTOM")]
    #[CustomIdGenerator(class: UuidGenerator::class)]
    private ?Uuid $id = null;

    #[Column(type: "string", length: 255)]
    private ?string $name;

    #[ManyToMany(targetEntity: Post::class, inversedBy: "tags")]
    private Collection $posts;

    public function __construct()
    {
        $this->posts = new ArrayCollection();
    }
}

At the same time, it generated three Repository classes for these entities.

// src/Repository/PostRepsoitory.php
class PostRepository extends ServiceEntityRepository
{
    public function __construct(ManagerRegistry $registry)
    {
        parent::__construct($registry, Post::class);
    }
}

// src/Repository/CommentRepsoitory.php
class CommentRepository extends ServiceEntityRepository
{
    public function __construct(ManagerRegistry $registry)
    {
        parent::__construct($registry, Comment::class);
    }
}

//src/Repository/TagRepository.php
class TagRepository extends ServiceEntityRepository
{
    public function __construct(ManagerRegistry $registry)
    {
        parent::__construct($registry, Tag::class);
    }
}

You can use Doctrine migration to generate a Migration file to maintain database schema in a production environment.

Run the following command to generate a Migration file.

# php bin/console make:migration

After it is executed, a Migration file is generated in the migrations folder, its naming is like Version20211104031420. It is a simple class extended AbstractMigration, the up function is use for upgrade to this version and down function is use for downgrade to the previous version.

To apply Migrations on database automaticially.

# php bin/console doctrine:migrations:migrate

# return to prev version
# php bin/console doctrine:migrations:migrate prev

# migrate to next
# php bin/console doctrine:migrations:migrate next

# These alias are defined : first, latest, prev, current and next

# certain version fully qualified class name
# php bin/console doctrine:migrations:migrate FQCN

Doctrine bundle also includes some command to maintain database and schema. eg.

# php bin/console doctrine:database:create
# php bin/console doctrine:database:drop

// schema create, drop, update and validate
# php bin/console doctrine:schema:create
# php bin/console doctrine:schema:drop
# php bin/console doctrine:schema:update
# php bin/console doctrine:schema:validate

Adding Sample Data

Create a custom command to load some sample data.

# php bin/console make:command add-post

It will generate a AddPostCommand under src/Command folder.

#[AsCommand(
    name: 'app:add-post',
    description: 'Add a short description for your command',
)]
class AddPostCommand extends Command
{


    public function __construct(private EntityManagerInterface $manager)
    {
        parent::__construct();
    }

    protected function configure(): void
    {
        $this
            ->addArgument('title', InputArgument::REQUIRED, 'Title of a post')
            ->addArgument('content', InputArgument::REQUIRED, 'Content of a post')
            //->addOption('option1', null, InputOption::VALUE_NONE, 'Option description')
        ;
    }

    protected function execute(InputInterface $input, OutputInterface $output): int
    {
        $io = new SymfonyStyle($input, $output);
        $title = $input->getArgument('title');

        if ($title) {
            $io->note(sprintf('Title: %s', $title));
        }

        $content = $input->getArgument('content');

        if ($content) {
            $io->note(sprintf('Content: %s', $content));
        }

        $entity = PostFactory::create($title, $content);
        $this ->manager->persist($entity);
        $this ->manager->flush();

//        if ($input->getOption('option1')) {
//            // ...
//        }

        $io->success('Post is saved: '.$entity);

        return Command::SUCCESS;
    }
}

The Doctrine EntityManagerInterface is managed by Symfony Service Container, and use for data persistence operations.

Run the following command to add a post into the database.

# php bin/console app:add-post "test title" "test content"
 ! [NOTE] Title: test title                                               
 ! [NOTE] Content: test content                                                             
 [OK] Post is saved: Post: [ id =1ec3d3ec-895d-685a-b712-955865f6c134, title=test title, content=test content, createdAt=1636010040, blishedAt=]

Testing Repository

PHPUnit is the most popular testing framework in PHP world, Symfony integrates PHPUnit tightly.

Run the following command to install PHPUnit and Symfony test-pack. The test-pack will install all essential packages for testing Symfony components and add PHPUnit configuration, such as phpunit.xml.dist.

# composer require --dev phpunit/phpunit symfony/test-pack

An simple test example written in pure PHPUnit.

class PostTest extends TestCase
{

    public function testPost()
    {
        $p = PostFactory::create("tests title", "tests content");

        $this->assertEquals("tests title", $p->getTitle());
        $this->assertEquals("tests content", $p->getContent());
        $this->assertNotNull( $p->getCreatedAt());
    }
}

Symfony provides some specific base classes(KernelTestCase, WebTestCase, etc.) to simplfy the testing work in a Symfony project.

The following is an example of testing a Repository - PostRepository. The KernelTestCase contains facilities to bootstrap application kernel and provides service container.

class PostRepositoryTest extends KernelTestCase
{

    private EntityManagerInterface $entityManager;

    private PostRepository $postRepository;

    protected function setUp(): void
    {
        //(1) boot the Symfony kernel
        $kernel = self::bootKernel();
        $this->assertSame('test', $kernel->getEnvironment());
        $this->entityManager = $kernel->getContainer()
            ->get('doctrine')
            ->getManager();

        //(2) use static::getContainer() to access the service container
        $container = static::getContainer();

        //(3) get PostRepository from container.
        $this->postRepository = $container->get(PostRepository::class);
    }

    protected function tearDown(): void
    {
        parent::tearDown();
        $this->entityManager->close();
    }

    public function testCreatePost(): void
    {
        $entity = PostFactory::create("test post", "test content");
        $this->entityManager->persist($entity);
        $this->entityManager->flush();
        $this->assertNotNull($entity->getId());

        $byId = $this->postRepository->findOneBy(["id" => $entity->getId()]);
        $this->assertEquals("test post", $byId->getTitle());
        $this->assertEquals("test content", $byId->getContent());
    }

}

In the above codes, in the setUp function, boot up the application kernel, after it is booted, a test scoped Service Container is available. Then get EntityManagerInterface and PostRepository from service container.

In the testCreatePost function, persists a Post entity, and find this post by id and verify the title and content fields.

Currently, PHPUnit does not include PHP 8 Attribute support, the testing codes are similar to the legacy JUnit 4 code style.

Creating PostController: Exposing your first Rest API

Similar to other MVC framework, we can expose RESTful APIs via Symfony Controller component. Follow the REST convention, we are planning to create the following APIs to a blog system.

GET /posts Get all posts.
GET /posts/{id} Get a single post by ID, if not found, return status 404
POST /posts Create a new post from request body, add the new post URI to response header Location, and return status 201
DELETE /posts/{id} Delete a single post by ID, return status 204. If the post was not found, return status 404 instead.
...

Run the following command to create a Controller skeleton. Follow the interactive guide to create a controller named PostController.

# php bin/console make:constroller

Open src/Controller/PostController.php in IDE.

Add Route attribute on class level and two functions: one for fetching all posts and another for getting single post by ID.

#[Route(path: "/posts", name: "posts_")]
class PostController extends AbstractController
{
    public function __construct(private PostRepository      $posts)
    {
    }

    #[Route(path: "", name: "all", methods: ["GET"])]
    function all(): Response
    {
        $data = $this->posts->findAll();
        return $this->json($data);
    }

}

Start up the application, and try to access the http://localhost:8000/posts, it will throw a circular dependencies exception when rendering the models in JSON view directly. There are some solutions to avoid this, the simplest is break the bi-direction relations before rendering the JSON view. Add a Ignore attribute on Comment.post and Tag.posts.

//src/Entity/Comment.php
class Comment
{
    #[Ignore]
    private Post $post;
}

//src/Entity/Tag.php
class Tag
{
    #[Ignore]
    private Collection $posts;
}

Testing Controller

As described in the previous sections, to test Controller/API, create a test class to extend WebTestCase, which provides a plenty of facilities to handle request and assert response.

Run the following command to create a test skeleton.

# php bin/console make:test

Follow the interactive steps to create a test base on WebTestCase.

class PostControllerTest extends WebTestCase
{
    public function testGetAllPosts(): void
    {
        $client = static::createClient();
        $crawler = $client->request('GET', '/posts');

        $this->assertResponseIsSuccessful();

        //
        $response = $client->getResponse();
        $data = $response->getContent();
        //dump($data);
        $this->assertStringContainsString("Symfony and PHP", $data);
    }

}

If you try to run the test, it will fail. At the moment, there is no any data for testing.

Preparing Data for Testing Purpose

The doctrine/doctrine-fixtures-bundle is use for populate sample data for testing purpose, and dama/doctrine-test-bundle ensures the data is restored before evey test is running.

Install doctrine/doctrine-fixtures-bundle and dama/doctrine-test-bundle.

composer require --dev doctrine/doctrine-fixtures-bundle dama/doctrine-test-bundle

Create a new Fixture.

# php bin/console make:fixtures

In the load fucntion, persist some data for tests.

class AppFixtures extends Fixture
{
    public function load(ObjectManager $manager): void
    {
        $data = PostFactory::create("Building Restful APIs with Symfony and PHP 8", "test content");
        $data->addTag(Tag::of( "Symfony"))
            ->addTag( Tag::of("PHP 8"))
            ->addComment(Comment::of("test comment 1"))
            ->addComment(Comment::of("test comment 2"));

        $manager->persist($data);
        $manager->flush();
    }
}

Run the command to load the sample data into database manually.

# php bin/console doctrine:fixtures:load

Add the following extension configuration into the phpunit.xml.dist, thus the data will be purged and recreated for every test running.

<extensions>
    <extension class="DAMA\DoctrineTestBundle\PHPUnit\PHPUnitExtension"/>
</extensions>

Run the following command to execute PostControllerTest.php .

# php .\vendor\bin\phpunit .\tests\Controller\PostControllerTest.php

Paginating Result

There are a lot of web applications which provide a input field for typing keyword and paginating the search results. Assume there is a keyword provided by request to match Post title or content fields, a offset to set the offset position of the pagination, and a limit to set the limited size of the elements per page. Create a function in the PostRepository, accepts a keyword, offset and limit as arguments.

public function findByKeyword(string $q, int $offset = 0, int $limit = 20): Page
{
    $query = $this->createQueryBuilder("p")
        ->andWhere("p.title like :q or p.content like :q")
        ->setParameter('q', "%" . $q . "%")
        ->orderBy('p.createdAt', 'DESC')
        ->setMaxResults($limit)
        ->setFirstResult($offset)
        ->getQuery();

    $paginator = new Paginator($query, $fetchJoinCollection = false);
    $c = count($paginator);
    $content = new ArrayCollection();
    foreach ($paginator as $post) {
        $content->add(PostSummaryDto::of($post->getId(), $post->getTitle()));
    }
    return Page::of ($content, $c, $offset, $limit);
}

Firstly, create a dynamic query using createQueryBuilder , then create a Doctrine Paginator instance to execute the query. The Paginator implements Countable interface, use count to get the count of total elements. Finally, we use a custom Page object to wrap the result.

class Page
{
    private Collection $content;
    private int $totalElements;
    private int $offset;
    private int $limit;

    #[Pure] public function __construct()
    {
        $this->content = new ArrayCollection();
    }


    public static function of(Collection $content, int $totalElements, int $offset = 0, int $limit = 20): Page
    {
        $page = new Page();
        $page->setContent($content)
            ->setTotalElements($totalElements)
            ->setOffset($offset)
            ->setLimit($limit);

        return $page;
    }

    //
    //getters

}

Customzing ArgumentResolver

In the PostController , let's improve the the function which serves the route /posts, make it accept query parameters like /posts?q=Symfony&offset=0&limit=10, and ensure the parameters are optional.

    #[Route(path: "", name: "all", methods: ["GET"])]
    function all(Request $req): Response
    {
        $keyword = $req->query->get('q')??'';
        $offset = $req->query->get('offset')??0;
        $limit = $req->query->get('limit')??10;

        $data = $this->posts->findByKeyword($keyword, $offset, $limit);
        return $this->json($data);
    }

It works but the query parameters handling looks a little ugly. It is great if they can be handled as the route path parameters.

We can create a custom ArgumentResolver to resolve the bound query arguments.

Firstly create an Annotation/Attribute class to identify a query parameter that need to be resolved by this ArgumentResolver.

#[Attribute(Attribute::TARGET_PARAMETER)]
final class QueryParam
{
    private null|string $name;
    private bool $required;

    /**
     * @param string|null $name
     * @param bool $required
     */
    public function __construct(?string $name = null, bool $required = false)
    {
        $this->name = $name;
        $this->required = $required;
    }

    //getters and setters

}

Create a custom ArgumentResolver implements the built-in ArgugmentResolverInterface.

class QueryParamValueResolver implements ArgumentValueResolverInterface, LoggerAwareInterface
{
    public function __construct()
    {
    }

    private LoggerInterface $logger;

    /**
     * @inheritDoc
     */
    public function resolve(Request $request, ArgumentMetadata $argument)
    {
        $argumentName = $argument->getName();
        $this->logger->info("Found [QueryParam] annotation/attribute on argument '" . $argumentName . "', applying [QueryParamValueResolver]");
        $type = $argument->getType();
        $nullable = $argument->isNullable();
        $this->logger->debug("The method argument type: '" . $type . "' and nullable: '" . $nullable . "'");

        //read name property from QueryParam
        $attr = $argument->getAttributes(QueryParam::class)[0];// `QueryParam` is not repeatable
        $this->logger->debug("QueryParam:" . $attr);
        //if name property is not set in `QueryParam`, use the argument name instead.
        $name = $attr->getName() ?? $argumentName;
        $required = $attr->isRequired() ?? false;
        $this->logger->debug("Polished QueryParam values: name='" . $name . "', required='" . $required . "'");

        //fetch query name from request
        $value = $request->query->get($name);
        $this->logger->debug("The request query parameter value: '" . $value . "'");

        //if default value is set and query param value is not set, use default value instead.
        if (!$value && $argument->hasDefaultValue()) {
            $value = $argument->getDefaultValue();
            $this->logger->debug("After set default value: '" . $value . "'");
        }

        if ($required && !$value) {
            throw new \InvalidArgumentException("Request query parameter '" . $name . "' is required, but not set.");
        }

        $this->logger->debug("final resolved value: '" . $value . "'");

        //must return  a `yield` clause
        yield match ($type) {
            'int' => $value ? (int)$value : 0,
            'float' => $value ? (float)$value : .0,
            'bool' => (bool)$value,
            'string' => $value ? (string)$value : ($nullable ? null : ''),
            null => null
        };
    }

    public function supports(Request $request, ArgumentMetadata $argument): bool
    {
        $attrs = $argument->getAttributes(QueryParam::class);
        return count($attrs) > 0;
    }

    public function setLogger(LoggerInterface $logger)
    {
        $this->logger = $logger;
    }
}

At runtime, it calls the supports function to check it the current request satisfy the requirement, if it is ok, then invoke the resovle funtion.

In the supports function, we check if the argument is annotated with a QueryParam, if it is existed, then resolved the argument from request query string.

Now change the function that serves /posts endpoint to the following.

#[Route(path: "", name: "all", methods: ["GET"])]
function all(#[QueryParam] $keyword,
    #[QueryParam] int $offset = 0,
    #[QueryParam] int $limit = 20): Response
    {
        $data = $this->posts->findByKeyword($keyword || '', $offset, $limit);
        return $this->json($data);
    }

Run the application and test the /posts using curl.

# curl http://localhost:8000/posts
{
    "content":[
        {
            "id":"1ec3e1e0-17b3-6ed2-a01c-edecc112b436",
            "title":"Building Restful APIs with Symfony and PHP 8"
        }
    ],
    "totalElements":1,
    "offset":0,
    "limit":20
}

Get Post by ID

Follow the design in the previous section, add another function to PostController to map route /posts/{id} .

class PostController extends AbstractController
{
    //other functions...

    #[Route(path: "/{id}", name: "byId", methods: ["GET"])]
    function getById(Uuid $id): Response
    {
        $data = $this->posts->findOneBy(["id" => $id]);
        if ($data) {
            return $this->json($data);
        } else {
            return $this->json(["error" => "Post was not found by id:" . $id], 404);
        }
    }
}

Run the application, and try to access http://localhost:8000/posts/{id}, it will throw an exception like this.

App\Controller\PostController::getById(): Argument #1 ($id) must be of type Symfony\Component\Uid\Uuid, string given, cal
led in D:\hantsylabs\symfony5-sample\rest-sample\vendor\symfony\http-kernel\HttpKernel.php on line 156

The id in the URI is a string, can not be used as Uuid directly.

Symfony provides ParamConverter to convert the request attributes to the target type. We can create a custom ParamConverter to archive the purpose.

Customizing ParamConverter

Create a new class UuidParamCovnerter under src/Request/ folder.

class UuidParamConverter implements ParamConverterInterface
{
    public function __construct(private LoggerInterface $logger)
    {
    }


    /**
     * @inheritDoc
     */
    public function apply(Request $request, ParamConverter $configuration): bool
    {

        $param = $configuration->getName();

        if (!$request->attributes->has($param)) {
            return false;
        }

        $value = $request->attributes->get($param);
        $this->logger->info("parameter value:" . $value);
        if (!$value && $configuration->isOptional()) {
            $request->attributes->set($param, null);

            return true;
        }

        $data = Uuid::fromString($value);
        $request->attributes->set($param, $data);

        return true;
    }

    /**
     * @inheritDoc
     */
    public function supports(ParamConverter $configuration): bool
    {
        $className = $configuration->getClass();
        $this->logger->info("converting to UUID :{c}", ["c" => $className]);
        return $className && $className == Uuid::class;
    }
}

In the above codes,

The supports function to check the execution environment if matching the requirements
The apply function to perform the conversion. if supports returns false, this conversion step will be skipped.

Creating a Post

Follow the REST convention, define the following rule to serve an endpoint to handle the request.

Request matches Http verbs/HTTP Method: POST
Request matches route endpoint: /posts
Set request header Content-Type value to application/json, and use request body to hold request data as JSON format
If successful, return a CREATED(201) Http Status code, and set the response header Location value to the URI of the new created post.

#[Route(path: "", name: "create", methods: ["POST"])]
public function create(Request $request): Response
{
    $data = $this->serializer->deserialize($request->getContent(), CreatePostDto::class, 'json');
    $entity = PostFactory::create($data->getTitle(), $data->getContent());
    $this->posts->getEntityManager()->persist($entity);

    return $this->json([], 201, ["Location" => "/posts/" . $entity->getId()]);
}

The posts->getEntityManager() overrides parent methods to get a EntityManager from parent class, you can also inject ObjectManager or EntityManagerInterface in the PostController directly to do the persistence work. The Doctrine Repository is mainly designated to build query criteria and execute custom queries.

Create a test function to verify in the PostControllerTest file.

public function testCreatePost(): void
{
    $client = static::createClient();
    $data = CreatePostDto::of("test title", "test content");
    $crawler = $client->request(
        'POST',
        '/posts',
        [],
        [],
        [],
        $this->getContainer()->get('serializer')->serialize($data, 'json')
    );

    $this->assertResponseIsSuccessful();

    $response = $client->getResponse();
    $url = $response->headers->get('Location');
    //dump($data);
    $this->assertNotNull($url);
    $this->assertStringStartsWith("/posts/", $url);
}

Converting Request Body

We can also use an Annotation/Attribute to erase the raw codes of handling Request object through introducing a custom ArgumentResolver.

Create a Body Attribute.

#[Attribute(Attribute::TARGET_PARAMETER)]
final class Body
{
}

Then create a BodyValueResolver.

class BodyValueResolver implements ArgumentValueResolverInterface, LoggerAwareInterface
{
    public function __construct(private SerializerInterface $serializer)
    {
    }

    private LoggerInterface $logger;

    /**
     * @inheritDoc
     */
    public function resolve(Request $request, ArgumentMetadata $argument)
    {
        $type = $argument->getType();
        $this->logger->debug("The argument type:'" . $type . "'");
        $format = $request->getContentType() ?? 'json';
        $this->logger->debug("The request format:'" . $format . "'");

        //read request body
        $content = $request->getContent();
        $data = $this->serializer->deserialize($content, $type, $format);
       // $this->logger->debug("deserialized data:{0}", [$data]);
        yield $data;
    }

    /**
     * @inheritDoc
     */
    public function supports(Request $request, ArgumentMetadata $argument): bool
    {
        $attrs = $argument->getAttributes(Body::class);
        return count($attrs) > 0;
    }

    public function setLogger(LoggerInterface $logger)
    {
        $this->logger = $logger;
    }

In the supports method, it simply detects if the method argument annotated with a Body attribute, then apply resolve method to deserialize the request body content to a typed object.

Run the application and test the endpoint through /posts.

curl -v http://localhost:8000/posts -H "Content-Type:application/json" -d "{\"title\":\"test title\",\"content\":\"test content\"}"
> POST /posts HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/7.55.1
> Accept: */*
> Content-Type:application/json
> Content-Length: 47
>
< HTTP/1.1 201 Created
< Cache-Control: no-cache, private
< Content-Type: application/json
< Date: Sun, 21 Nov 2021 08:42:49 GMT
< Location: /posts/1ec4aa70-1b21-6bce-93f8-b39330fe328e
< X-Powered-By: PHP/8.0.10
< X-Robots-Tag: noindex
< Content-Length: 2
<
[]

Exception Handling

Symfony kernel provides a event machoism to raise an Exception in Controller class and handle them in your custom EventListener or EventSubscriber .

For example, create a PostNotFoundException.

class PostNotFoundException extends \RuntimeException
{

    public function __construct(Uuid $uuid)
    {
        parent::__construct("Post #" . $uuid . " was not found");
    }

}

Create a EventListener to catch this exception, and handle the exception as expected.

class ExceptionListener implements LoggerAwareInterface
{
    private LoggerInterface $logger;

    public function __construct()
    {
    }

    public function onKernelException(ExceptionEvent $event)
    {
        // You get the exception object from the received event
        $exception = $event->getThrowable();
        $data = ["error" => $exception->getMessage()];

        // Customize your response object to display the exception details
        $response = new JsonResponse($data);

        // HttpExceptionInterface is a special type of exception that
        // holds status code and header details

        if ($exception instanceof PostNotFoundException) {
            $response->setStatusCode(Response::HTTP_NOT_FOUND);
        } else if ($exception instanceof HttpExceptionInterface) {
            $response->setStatusCode($exception->getStatusCode());
            $response->headers->replace($exception->getHeaders());
        } else {
            $response->setStatusCode(Response::HTTP_INTERNAL_SERVER_ERROR);
        }

        // sends the modified response object to the event
        $event->setResponse($response);
    }

    public function setLogger(LoggerInterface $logger)
    {
        $this->logger = $logger;
    }
}

  App\EventListener\ExceptionListener:
    tags:
      - { name: kernel.event_listener, event: kernel.exception, priority: 50 }

It indicates it binds event.exception event to ExceptionListener, and set priority to set the order at execution time.

Run the following command to show all registered EventListener/EventSubscribers on event kernel.exception.

php bin/console debug:event-subscriber kernel.exception

Change the getById function to the following.

#[Route(path: "/{id}", name: "byId", methods: ["GET"])]
function getById(Uuid $id): Response
{
    $data = $this->posts->findOneBy(["id" => $id]);
    if ($data) {
        return $this->json($data);
    } else {
        throw new PostNotFoundException($id);
    }
}

Add a test to verify if the post is not found and get a 404 status code.

public function testGetANoneExistingPost(): void
{
    $client = static::createClient();
    $id = Uuid::v4();
    $crawler = $client->request('GET', '/posts/' . $id);

    //
    $response = $client->getResponse();
    $this->assertResponseStatusCodeSame(404);
    $data = $response->getContent();
    $this->assertStringContainsString("Post #" . $id . " was not found", $data);
}

Run the application again, and try to access a single Post through a none existing id.

curl http://localhost:8000/posts/1ec3e1e0-17b3-6ed2-a01c-edecc112b438 -H "Accept: application/json" -v
> GET /posts/1ec3e1e0-17b3-6ed2-a01c-edecc112b438 HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/7.55.1
> Accept: application/json
>
< HTTP/1.1 404 Not Found
< Cache-Control: no-cache, private
< Content-Type: application/json
< Date: Mon, 22 Nov 2021 03:57:51 GMT
< X-Powered-By: PHP/8.0.10
< X-Robots-Tag: noindex
< Content-Length: 69
<
{"error":"Post #1ec3e1e0-17b3-6ed2-a01c-edecc112b438 was not found."}

Get the complete source codes from my Github.

Building your own Maven archetype

Hantsy Bai — Tue, 06 Apr 2021 08:58:31 +0000

Github is easy to share codes with others, for example I created jakartaee9-starter-boilerplate as a project template for Jakarta EE developers. For those who are familiar with Github, it is easy to start their new projects by forking or cloning this project directly. But obviously for a general Jakarta EE application, you do not need the configuraitons of all application severs, eg. Glassfish/Payara, WildFly, OpenLiberty, Apache TomEE, etc. For most of Java developers esp. Maven users, a simple and clean Maven archetype is still the preferred option to generate the new project skeleton, and you can add the required configuration back later.

I had created some maven archetype like toys myself in the past years, but I never made them public through the Maven Central repository. In this post, I will replay the steps of creating the Maven Archetype for Jakarta EE 9 and the followed steps of publishing it to the Maven Central repository and making it public to the world.

Generating the archetype project skeleton

Maven archetype is the Maven specific templates used to generate a new project in the Maven build system. Maven itself provides an official archetype (maven-archetype-archetype) for you to build your own Maven archetype.

The official archetype introduction page lists all official archetypes.

Run the following command to generate the archetype project in an interactive mode.

mvn archetype:generate

When you hit down the Enter key, it will take some seconds to list all available archetypes. Every archetype is attached to a number.

Type the number of maven-archetype-archetype , then it will ask you to fill groupId, archetypeId, package and version of your project. After all fields are filled as expected, it will generate the project in seconds.

Alternatively, you can generate the project in a single command aka the batch mode.

mvn archetype:generate -B \
-DarchetypeGroupId=org.apache.maven.archetypes \
-DarchetypeArtifactId=maven-archetype-archetype \
-DarchetypeVersion=1.4 \
-DgroupId=io.github.hantsy \
-DartifactId=maven-archetype-jakartaee9 \
-Dversion=1.0-SNAPSHOT

Note here we use the -B parameter in the command line to skip the interactive questionnaire steps and run the command in batch mode.

More info about Maven archetype and maven-archetype-plugin, please check Maven Archetype.

Exploring the project structure

Import the project into your IDEs, such as Apache NetBeans, IntelliJ IDEA, etc.

You will see the following project file structure.

├── pom.xml
└── src
    ├── main
    │   └── resources
    │       ├── archetype-resources
    │       │   ├── pom.xml
    │       │   └── src
    │       │       ├── main
    │       │       │   └── java
    │       │       │       └── App.java
    │       │       └── test
    │       │           └── java
    │       │               └── AppTest.java
    │       └── META-INF
    │           └── maven
    │               └── archetype-metadata.xml
    └── test
        └── resources
            └── projects
                └── it-basic
                    ├── archetype.properties
                    └── goal.txt

The structure is similar to a general Maven project, there are some difference.

The src/main/resources/archetype-resources folder stores all template resources used to generate projects from this archetype.
The src/main/resources/META-INF/maven-archetype-metadata.xml is the archetype description file, including the properties, file sets etc.
The src/test/resources/projects includes resources to test this archetype, including the properties applied to generate new projects and the goal run in the host integration-test phase.

Preparing the archetype resources

To simplify the work, I reuse the sample codes from jakartaee9-starter-boilerplate.

Copy GreetingMessage, GreetingService, GreetingService and JaxrsActivator to the src/main/resources/archetype-resources/src/main/java folder. Change the package declaration to the following.

   package $package;
   ...

Similar to the last step, copy the GreetingResourceTest and GreetingServiceTest to the src/main/resources/archetype-resources/src/test/java folder. Remember changing the package declaration.
Add empty beans.xmlfile to the src/main/resources/archetype-resources/src/main/resources/META-INF/ folder.

   <beans xmlns="https://jakarta.ee/xml/ns/jakartaee"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="https://jakarta.ee/xml/ns/jakartaee https://jakarta.ee/xml/ns/jakartaee/beans_3_0.xsd"
           version="3.0">
   </beans>

Add a simple arquillian.xml to the src/main/resources/archetype-resources/src/test/resources folder.

   <arquillian
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xmlns="http://jboss.org/schema/arquillian"
               xsi:schemaLocation="http://jboss.org/schema/arquillian
                                   http://jboss.org/schema/arquillian/arquillian_1_0.xsd">
       <defaultProtocol type="Servlet 5.0"/>

       <engine>
           <property name="deploymentExportPath">target/</property>
       </engine>

       <container qualifier="glassfish" default="true">
           <configuration>
               <property name="adminHost">localhost</property>
               <property name="adminPort">4848</property>
               <property name="adminUser">admin</property>
               <!-- if https is enabled via `asadmin enable-secure-admin` on a remote server -->
               <!-- <property name="adminHttps">true</property>-->
               <!-- if admin password is changed via `asadmin change-admin-password` -->
               <!--<property name="adminPassword">adminadmin</property>-->
               <!-- default is empty -->
               <property name="adminPassword"></property>
           </configuration>
       </container>
   </arquillian>

Change the pom.xml template (src/main/resources/archetype-resources/pom.xml) to the following.

   <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
     <modelVersion>4.0.0</modelVersion>
     <groupId>${groupId}</groupId>
     <artifactId>${artifactId}</artifactId>
     <version>${version}</version>
     <packaging>war</packaging>

     <name>${artifactId}</name>
     <description>A Jakarta EE starter boilerplate for Jakarta EE 9</description>
     <properties>
       <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
       <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
       <maven.compiler.source>1.8</maven.compiler.source>
       <maven.compiler.target>1.8</maven.compiler.target>

       <!-- Official Maven Plugins -->
       <maven-compiler-plugin.version>3.8.1</maven-compiler-plugin.version>
       <maven-war-plugin.version>3.3.1</maven-war-plugin.version>
       <maven-dependency-plugin.version>3.1.2</maven-dependency-plugin.version>
       <maven-surefire-plugin.version>3.0.0-M5</maven-surefire-plugin.version>
       <maven-failsafe-plugin.version>3.0.0-M5</maven-failsafe-plugin.version>
       <maven-surefire-report-plugin.version>3.0.0-M5</maven-surefire-report-plugin.version>

       <!-- Cargo maven plugin -->
       <!-- Since 1.9.0, cargo-maven3-plugin is the default for cargo prefix. -->
       <cargo-maven3-plugin.version>1.9.0</cargo-maven3-plugin.version>

       <!-- Jakarta EE API -->
       <jakartaee-api.version>9.0.0</jakartaee-api.version>

       <!-- Arquillian BOM -->
       <arquillian-bom.version>1.7.0.Alpha9</arquillian-bom.version>
       <junit-jupiter.version>5.7.1</junit-jupiter.version>

       <!-- Glassfish server -->
       <glassfish.version>6.0.0</glassfish.version>
       <arquillian-glassfish6.version>1.0.0.Alpha1</arquillian-glassfish6.version>
       <jersey.version>3.0.1</jersey.version>

       <!-- by default skip tests -->
       <skipTests>true</skipTests>
     </properties>

     <dependencyManagement>
       <dependencies>
         <dependency>
           <groupId>jakarta.platform</groupId>
           <artifactId>jakarta.jakartaee-api</artifactId>
           <version>${jakartaee-api.version}</version>
           <scope>provided</scope>
         </dependency>
         <dependency>
           <groupId>org.jboss.arquillian</groupId>
           <artifactId>arquillian-bom</artifactId>
           <version>${arquillian-bom.version}</version>
           <scope>import</scope>
           <type>pom</type>
         </dependency>
         <dependency>
           <groupId>org.junit</groupId>
           <artifactId>junit-bom</artifactId>
           <version>${junit-jupiter.version}</version>
           <type>pom</type>
           <scope>import</scope>
         </dependency>
       </dependencies>
     </dependencyManagement>

     <dependencies>
       <dependency>
         <groupId>jakarta.platform</groupId>
         <artifactId>jakarta.jakartaee-api</artifactId>
       </dependency>
       <dependency>
         <groupId>org.jboss.arquillian.junit5</groupId>
         <artifactId>arquillian-junit5-container</artifactId>
         <scope>test</scope>
       </dependency>
       <!-- see: https://github.com/arquillian/arquillian-core/issues/248 -->
       <!-- and https://github.com/arquillian/arquillian-core/pull/246/files -->
       <dependency>
         <groupId>org.jboss.arquillian.protocol</groupId>
         <artifactId>arquillian-protocol-servlet-jakarta</artifactId>
         <scope>test</scope>
       </dependency>
       <dependency>
         <groupId>org.junit.jupiter</groupId>
         <artifactId>junit-jupiter</artifactId>
         <scope>test</scope>
       </dependency>
     </dependencies>

     <build>
       <finalName>${project.artifactId}</finalName>
       <pluginManagement>
         <plugins>
           <plugin>
             <groupId>org.codehaus.cargo</groupId>
             <artifactId>cargo-maven3-plugin</artifactId>
             <version>${cargo-maven3-plugin.version}</version>
           </plugin>
         </plugins>
       </pluginManagement>
       <plugins>
         <plugin>
           <groupId>org.apache.maven.plugins</groupId>
           <artifactId>maven-compiler-plugin</artifactId>
           <version>${maven-compiler-plugin.version}</version>
         </plugin>
         <plugin>
           <groupId>org.apache.maven.plugins</groupId>
           <artifactId>maven-war-plugin</artifactId>
           <version>${maven-war-plugin.version}</version>
         </plugin>
       </plugins>
     </build>
     <profiles>
       <profile>
         <id>glassfish</id>
         <activation>
           <activeByDefault>true</activeByDefault>
         </activation>
         <build>
           <plugins>
             <plugin>
               <groupId>org.codehaus.cargo</groupId>
               <artifactId>cargo-maven3-plugin</artifactId>
               <configuration>
                 <container>
                   <containerId>glassfish6x</containerId>
                   <artifactInstaller>
                     <groupId>org.glassfish.main.distributions</groupId>
                     <artifactId>glassfish</artifactId>
                     <version>${glassfish.version}</version>
                   </artifactInstaller>
                 </container>
                 <configuration>
                   <!-- the configuration used to deploy -->
                   <home>${project.build.directory}/glassfish6x-home</home>
                   <properties>
                     <cargo.remote.password></cargo.remote.password>
                   </properties>
                 </configuration>
               </configuration>
             </plugin>
           </plugins>
         </build>

         <dependencies>
           <!-- Jersey -->
           <dependency>
             <groupId>org.glassfish.jersey.media</groupId>
             <artifactId>jersey-media-sse</artifactId>
             <version>${jersey.version}</version>
             <scope>test</scope>
           </dependency>
           <dependency>
             <groupId>org.glassfish.jersey.media</groupId>
             <artifactId>jersey-media-json-binding</artifactId>
             <version>${jersey.version}</version>
             <scope>test</scope>
           </dependency>
           <dependency>
             <groupId>org.glassfish.jersey.inject</groupId>
             <artifactId>jersey-hk2</artifactId>
             <version>${jersey.version}</version>
             <scope>test</scope>
           </dependency>
           <dependency>
             <groupId>org.glassfish.jersey.core</groupId>
             <artifactId>jersey-client</artifactId>
             <version>${jersey.version}</version>
             <scope>test</scope>
           </dependency>
           <dependency>
             <groupId>org.jboss.arquillian.container</groupId>
             <artifactId>arquillian-glassfish-remote-6</artifactId>
             <version>${arquillian-glassfish6.version}</version>
             <scope>test</scope>
           </dependency>
         </dependencies>

       </profile>
     </profiles>
   </project>

Note, the value of groupId, artifactId and version elements in this xml refers to a placeholder which can be intercepted in the generating stage.

Change the archetype descriptor (src/main/resources/META-INF/maven/archetype-metadata.xml) to the following.

   <archetype-descriptor
                         xmlns="http://maven.apache.org/plugins/maven-archetype-plugin/archetype-descriptor/1.0.0"
                         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                         xsi:schemaLocation="http://maven.apache.org/plugins/maven-archetype-plugin/archetype-descriptor/1.0.0 http://maven.apache.org/xsd/archetype-descriptor-1.0.0.xsd"
                         name="${artifactId}">

       <fileSets>
           <fileSet filtered="true" packaged="true">
               <directory>src/main/java</directory>
           </fileSet>
           <fileSet>
               <directory>src/main/resources</directory>
           </fileSet>
           <fileSet filtered="true" packaged="true">
               <directory>src/test/java</directory>
           </fileSet>
           <fileSet>
               <directory>src/test/resources</directory>
           </fileSet>
       </fileSets>
   </archetype-descriptor>

In the above codes, when the filtered attribute is set to true, it will replace the placeholders(eg. $package, etc.) you have set in the template files with the parameters when executing mvn archetype:generate to generate projects. And when the packaged attribute is set to true, it will move the generated file to corresponding packages.

To test the archetype resources, there is a archetype.properties and a goal.txt file prepared in the src/test/resources/projects/it-basic. In the integration-test phase, it will generate a test project using archetype resources and replace the placeholders using the properties defined in the archetype.properties and execute goals in the goal.txt against the generated project.

I created a Github Actions workflow to verify the generated projects.

Publishing to Maven Central Repository

Note, the steps of publishing your packages to the Maven Central repository is a little tedious, please be patient.

Once you have built the maven archetype project successfully, the next step is publishing it to Maven Central repository and make it public, there are some useful guides in Google results. I suggest you pick one or more from the following list and read it firstly.

Register an account of the Sonatype issue tracker.
File a new issue on Community Support to submit your request of publishing your artifact. Check my issue#OSSRH-66424 I created for hantsy/maven-archetype-jakartaee9 as an example.
Follow the response of the issue, create a new empty repo(the name is the issue id) on Github for authentication requirement. eg. OSSRH-66424
Generate GPG keys and send to key servers.
Under Windows 10 system, you have to install GPG tools firstly. Simply run the following command to install gpg4win if you are using Chocolatey.

   choco install gpg4win

Refresh the environment.

   refreshenv

Then generate GPG keys.

   # gpg --full-gen-key
   gpg (GnuPG) 2.2.27; Copyright (C) 2021 g10 Code GmbH
   This is free software: you are free to change and redistribute it.
   There is NO WARRANTY, to the extent permitted by law.
   Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
   (14) Existing key from card
   Your selection? 1
   RSA keys may be between 1024 and 4096 bits long.
   What keysize do you want? (3072)
   Requested keysize is 3072 bits
   Please specify how long the key should be valid.
   0 = key does not expire
   <n>  = key expires in n days
   <n>w = key expires in n weeks
   <n>m = key expires in n months
   <n>y = key expires in n years
   Key is valid for? (0)
   Key does not expire at all
   Is this correct? (y/N) y

   GnuPG needs to construct a user ID to identify your key.

   Real name: Hantsy Bai
   Email address: hantsy@gmail.com
   Comment:
   You selected this USER-ID:
   "Hantsy Bai <hantsy@gmail.com>"

   Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
   We need to generate a lot of random bytes. It is a good idea to perform
   some other action (type on the keyboard, move the mouse, utilize the
   disks) during the prime generation; this gives the random number
   generator a better chance to gain enough entropy.
   gpg: AllowSetForegroundWindow(9152) failed: Access is denied.
   We need to generate a lot of random bytes. It is a good idea to perform
   some other action (type on the keyboard, move the mouse, utilize the
   disks) during the prime generation; this gives the random number
   generator a better chance to gain enough entropy.
   gpg: C:/Users/hantsy/AppData/Roaming/gnupg/trustdb.gpg: trustdb created
   gpg: key E5D9B4272348ADAE marked as ultimately trusted
   gpg: directory 'C:/Users/hantsy/AppData/Roaming/gnupg/openpgp-revocs.d' created
   gpg: revocation certificate stored as 'C:/Users/hantsy/AppData/Roaming/gnupg/openpgp-revocs.d\03B4C55DDA964D942C756D79E5D9B4272348ADAE.rev'
   public and secret key created and signed.

   pub   rsa3072 2021-03-30 [SC]
   03B4C55DDA964D942C756D79E5D9B4272348ADAE
   uid                      Hantsy Bai <hantsy@gmail.com>
   sub   rsa3072 2021-03-30 [E]

Send the keys to the public key servers.

   # gpg --send-keys 03B4C55DDA964D942C756D79E5D9B4272348ADAE
   gpg: sending key E5D9B4272348ADAE to hkps://hkps.pool.sks-keyservers.net

Follow the steps of Deploying to OSSRH with Apache Maven to publish your artifact.

I skip the detailed steps here, please read the Synatype's article carefully.

Then back to the issue#OSSRH-66424, comment and notify the moderator to publish your artifact to the Maven Central repository. In one or two hours, your artifact will be available on Maven Central Repository Search index page.

Publishing to Github Package

The Configuring Apache Maven for use with GitHub Packages page provides detailed steps to publish a package to Github Packages.

Following the steps of Authenticating to Github Packages to set the Github authentication info in your ~/.m2/settings.xml file.
Deploy to Github Packages.

In our case, we have set the distributionManagement to use the Maven Central repository. Use the following command instead.

   mvn clean package deploy -DskipTests -DaltDeploymentRepository=github::https://maven.pkg.github.com/hantsy/maven-archetype-jakartaee9

By default, Github Packages does not allow anyone to access your repository without an authentication. After consulting from Github community, there is a solution to resolve this issue gracefully, it is possible to make your repository accessible to the public.

Generate a Personal Access Token with packages:read scope.

Run the following command to generate the repository config.

 # docker run ghcr.io/jcansdale/gpr encode <your PAT>

 A NuGet `nuget.config` file:
 <packageSourceCredentials>
 <github>
 <add key="Username" value="PublicToken" />
 <add key="ClearTextPassword" value="&#48;28d0c8f723084e499eb87a6fe173b3af295d618" />
 </github>
 </packageSourceCredentials>

 A Maven `pom.xml` file:
 <repositories>
 <repository>
 <id>github-public</id>
 <url>https://public:&#48;28d0c8f723084e499eb87a6fe173b3af295d618@maven.pkg.github.com/<OWNER>/*</url>
 </repository>
 </repositories>

 An npm `.npmrc` file:
 @OWNER:registry=https://npm.pkg.github.com
 //npm.pkg.github.com/:_authToken="\u003028d0c8f723084e499eb87a6fe173b3af295d618"

As you see, now you can add the above repository to your Maven proxy server or global Maven settings(~/.m2/settings.xml) in your system.

Check the source codes from my Github.

Launch my personal website

Hantsy Bai — Fri, 12 Mar 2021 14:18:53 +0000

These days I was trying to start my personal homepage. It is open to the public now. Open your browser and navigate to https://hantsy.github.io.

In this post, I will introduce how I created it using Github Pages and Jekyll.

In the past years, I have written a lot of blog entries on blogspot and Medium, the new homepage will be a central place to aggregate all resources, including blog entries, development notes, etc.

Personal HomePage

Follow the guide of Github Pages, firstly create a repository named <your github name>.github.io, it will be built automatically by a built-in GitHub Pages workflow and deployed to https://<your github name>.github.io.

Jekyll is used for building static websites and blogs from markdown text documents. Github Pages has built-in Jekyll support.

Jekyll website has a great step-by-step tutorial to help you to set up a Jekyll driven static website from scratch. Once you have understood how Jekyll works, you can choose an existing template or themes to speed up the website building. There are plenty of themes in Jekyll showcases and Jekyll Themes which can reuse in your private website. After doing some research I finally choose alshedivat/al-folio.

The best way to reuse alshedivat/al-folio is cloning it and renaming it to <your github name>.github.io. But I've created a repository(hantsy.github.io). When I cloned it into my local machine and pushed it into my Github repository, I encountered a small issue, check alshedivat/al-folio#208.

The alshedivat/al-folio includes two branches: source and master. It includes a Github actions workflow to automate the build progress. Any modification in the source branch will deploy to the master branch.

Install the dependencies.

bundle install

Build the project.

bundle exec jekyll build

Run it in a local dev environment.

bundle exec jekyll serve

The next step is to configure the website information. Open _config.yml in the project root folder, set the essential properties, eg. first_name, last_name, description etc. And change the profile info in the about page, replace the photo image with yours on the about page.

Import Medium Posts

I have maintained my current blog on Medium, I do not want to duplicate the work and republish all posts on the posts page again. I think the best way is to sync the posts automatically.

After researching myself and discussing with the Jekyll gurus from Jekyll Talk, I decided to use the official RSS Importer in a standalone Github actions workflow to sync the posts.

The solution is not ideal but works well. It will be changed in the future if I found better solutions.

It is easy to enable Google Analytics and Disqus, just go to their website and register your website and set the ID in the _config.yml.

Display Github Projects

The alshedivat/al-folio itself includes a jekyll-github-metadata plugin to access the metadata of Github repositories.

To replace projects in the projects page, I changed the _pages/projects.md to the following.

// omitted the page headers
<div class="projects">
  {% assign sorted_projects = site.github.public_repositories | sort: "stargazers_count"|reverse  %}
  {% for project in sorted_projects %}
  <div class="card hoverable mt-2">
    <div class="card-body">
      <h5 class="card-title text-lowercase">
        <a href="{{ project.html_url }}" target="_blank">{{ project.name }}</a>
      </h5>
       <h6 class="card-subtitle mb-2 text-muted">{{project.language}} &bull; <i class="fa fa-star"></i> {{project.stargazers_count}} </h6>
      <p class="card-text">{{ project.description }}</p>
    </div>
  </div>
  {% endfor %}
</div>

There is no need a access token to read the metadata of the Github public repositories. More details see jekyll/github-metadata.

Gather my Docs

In the past years, I have written plenty of docs via blog entries, project wikis, etc. The Github Pages also support Jekyll-driven static web pages for single repository.

There is a Github Pages option in the repository settings page, set it to discover docs from master branch and docs folder. Then create an index.md in the docs folder. In general, the index.md should contain a table of content for all documents in the docs folder.

After it is done, the docs for the repository are available via http://hantsy.github.io/<your repository name> , eg. https://hantsy.github.io/jakartaee9-starter-boilerplate/ is the online docs address for the repository hantsy/jakartaee9-starter-boilerplate.

Some of my docs are available on GitBook.io, where it provides a better web UI to read online.

The alshedivat/al-folio itself includes a publications page and enables jekyll-scholar to display the essays, books, etc. I reuse it to organize my docs.

Go to the publications to browse all the docs I have gathered till now.

Resources

Building a Chat application with Angular and Spring RSocket

Hantsy Bai — Sat, 08 Aug 2020 06:35:53 +0000

In this post, we will use RSocket protocol to reimplement the chat application.

If you have missed the former posts about implementing the chat application, there is a checklist.

RSocket is a binary protocol for use on byte stream transports, such as TCP, WebSocket, RCP etc.

RSocket embraces ReactiveStreams semantics, and Spring provides excellent RSocket support through the existing messaging infrastructure. I have introduced RSocket in my former posts, check here.

In this post, we will use WebSocket as transport protocol which is good for web application. RSocket defines 4 interaction modes, we will use fire-and-forget to send a message to the server side, and request/streams to retrieve messages as an infinite stream from the server.

Firstly let's create the server application. Generate a project skeleton using Spring Initializr.

Project type: Gradle
Language: Kotlin
Spring Boot version :2.4.0M1
Project Metadata/Java: 14
Dependencies: Reactive Web, RSocket

Hit the Generate button to download the generated archive, and extract it into your local disk.

Make sure you have installed the latest JDK 14 (AdoptOpenJDK is highly recommended), then import the source codes in your IDEs. eg. Intellij IDEA, and start to implement the server side.

We also skip the discussion of Reactor's Sink implementation here.

Create a Message document definition and a Repository for it.

interface MessageRepository : ReactiveMongoRepository<Message, String> {
    @Tailable
    fun getMessagesBy(): Flux<Message>
}

@Document(collection = "messages")
data class Message(@Id var id: String? = null, var body: String, var sentAt: Instant = Instant.now())

Create a @Controller to handle messages.

@Controller
class MessageController(private val messages: MessageRepository) {
    @MessageMapping("send")
    fun hello(p: String) = this.messages.save(Message(body = p, sentAt = Instant.now())).log().then()

    @MessageMapping("messages")
    fun messageStream(): Flux<Message> = this.messages.getMessagesBy().log()
}

The send route accepts a String based message payload and return a Mono<Void>, which will handle messages of the fire-and-forget mode from clients. The messages route accepts a null payload and return a Flux<Message>, which will act as the handler of request-stream mode.

If you are new to the Spring RSocket , you may be confused how @Controller and MessageMapping are mapped to the interaction modes which the original RSocket message handler used. Spring hides the complexity of the RSocket protocol itself , and reuse the existing messaging infrastructure to handle RSocket messages. Remember, compare the incoming payload and outgoing message type with 4 interaction mode definitions in the official RSocket website, you can determine which interaction mode it is mapped to.

Configure RSocket to use websocket transport in the application.properties file.

# a mapping path is defined
spring.rsocket.server.mapping-path=/rsocket
# websocket is chosen as a transport
spring.rsocket.server.transport=websocket

Start a MongoDB service as follows.

docker-compose up mongodb

As described in the former posts, you have to prepare a capped messages collection, check this post for more details.

Run the following command to start the server side application.

./gradlew bootRun

I have written a small integration test to verify if it works.

@SpringBootTest
class RSocketServerApplicationTests {

    @Autowired
    lateinit var rSocketRequester: RSocketRequester;

    @Test
    fun contextLoads() {

        val verifier= rSocketRequester.route("messages")
                .retrieveFlux(Message::class.java)
                .log()
                .`as` { StepVerifier.create(it) }
                .consumeNextWith { it -> assertThat(it.body).isEqualTo("test message") }
                .consumeNextWith { it -> assertThat(it.body).isEqualTo("test message2") }
                .thenCancel()
                .verifyLater()
        rSocketRequester.route("send").data("test message").send().then().block()
        rSocketRequester.route("send").data("test message2").send().then().block()

        verifier.verify(Duration.ofSeconds(5))
    }

    @TestConfiguration
    class TestConfig {

        @Bean
        fun rSocketRequester(builder: RSocketRequester.Builder) = builder.dataMimeType(MimeTypeUtils.APPLICATION_JSON)
                .connectWebSocket(URI.create("ws://localhost:8080/rsocket")).block()
    }

}

In the above codes, use a test specific @TestConfiguration to define a RSocketRequester bean, which is a helper to communicate with the server side.

Let's move to the frontend application.

Create a new Angular project, and add two dependencies: roscket-core, rsocket-websocket-client.

npm install roscket-core rsocket-websocket-client

Fill the following codes in the app.component.ts file. I've spent some time on making this work with my backend, the article RSocket With Spring Boot + JS: Zero to Hero from Domenico Sibilio is very helpful. The rsocket-js project also includes excellent examples.

export class AppComponent implements OnInit, OnDestroy {

  title = 'client';
  message = '';
  messages: any[];
  client: RSocketClient;
  sub = new Subject();

  ngOnInit(): void {
    this.messages = [];

    // Create an instance of a client
    this.client = new RSocketClient({
      serializers: {
        data: JsonSerializer,
        metadata: IdentitySerializer
      },
      setup: {
        // ms btw sending keepalive to server
        keepAlive: 60000,
        // ms timeout if no keepalive response
        lifetime: 180000,
        // format of `data`
        dataMimeType: 'application/json',
        // format of `metadata`
        metadataMimeType: 'message/x.rsocket.routing.v0',
      },
      transport: new RSocketWebSocketClient({
        url: 'ws://localhost:8080/rsocket'
      }),
    });

    // Open the connection
    this.client.connect().subscribe({
      onComplete: (socket: RSocket) => {

        // socket provides the rsocket interactions fire/forget, request/response,
        // request/stream, etc as well as methods to close the socket.
        socket
          .requestStream({
            data: null, // null is a must if it does not include a message payload, else the Spring server side will not be matched.
            metadata: String.fromCharCode('messages'.length) + 'messages'
          })
          .subscribe({
            onComplete: () => console.log('complete'),
            onError: error => {
              console.log("Connection has been closed due to:: " + error);
            },
            onNext: payload => {
              console.log(payload);
              this.addMessage(payload.data);
            },
            onSubscribe: subscription => {
              subscription.request(1000000);
            },
          });

        this.sub.subscribe({
          next: (data) => {
            socket.fireAndForget({
              data: data,
              metadata: String.fromCharCode('send'.length) + 'send',
            });
          }
        })
      },
      onError: error => {
        console.log("Connection has been refused due to:: " + error);
      },
      onSubscribe: cancel => {
        /* call cancel() to abort */
      }
    });
  }

  addMessage(newMessage: any) {
    console.log("add message:" + JSON.stringify(newMessage))
    this.messages = [...this.messages, newMessage];
  }

  ngOnDestroy(): void {
    this.sub.unsubscribe();
    if (this.client) {
      this.client.close();
    }
  }

  sendMessage() {
    console.log("sending message:" + this.message);
    this.sub.next(this.message);
    this.message = '';
  }
}

Reuse the template file we've used in the former posts.

<div fxFlex>
    <p *ngFor="let m of messages">
        {{m|json}}
    </p>
</div>
<div>
    <form fxLayout="row baseline" #messageForm="ngForm" (ngSubmit)="sendMessage()">
        <mat-form-field fxFlex>
            <input name="message" fxFill matInput #messageCtrl="ngModel" [(ngModel)]="message" required />
            <mat-error fxLayoutAlign="start" *ngIf="messageCtrl.hasError('required')">
                Message body can not be empty.
            </mat-error>
        </mat-form-field>
        <div>
            <button mat-button mat-icon-button type="submit" [disabled]="messageForm.invalid || messageForm.pending">
                <mat-icon>send</mat-icon>
            </button>
        </div>
    </form>
</div>

Next run the client application.

npm run start

Open two browser windows(or two different browsers), type some messages in each window and experience it.

I found a weird issue may be caused by the JSON Serializer encode/decode from the roscket-js project, I described it in rsocket-js issues #93, if you have some idea to overcome this, please comment on this issue.

Get the complete codes from my github.

Building a Chat application with Angular and Spring Reactive Server Sent Events

Hantsy Bai — Sat, 08 Aug 2020 06:34:10 +0000

In this post, we will use Server Sent Events instead of WebSocket to broadcast messages to clients.

If you have missed the posts about the implementation of the WebSocket version, check here and part 2.

Unlike WebSocket, SSE (Server Sent Events) is a one-way direction protocol, and used for the server-to-client messaging case, and it only accepts text based messages. A good use case is to send notifications to clients, like that done in the push notification feature of the mobile apps. In this demo, we will use it to update the message list in the client.

From a developer's perspective, there is no much difference between Server Sent Events and HTTP based RESTful APIs, SSE requires a specific media type - text/event-stream when emitting messages to clients. In the client application, use the browser built-in EventSource to subscribe it.

If you have used SSE in Spring MVC or standard Jakarta EE platform, you could have to emit the event manually. SSE concept is a good match with the Reactor's Flux, which serves an infinite stream to clients. Spring WebFlux simplifies the work, a SSE endpoint is no difference from a general RESTful APIs in a Controller.

Firstly let's create the server side. Generate a project skeleton using Spring Initializr.

Project type: Gradle
Language: Kotlin
Spring Boot version :2.4.0M1
Project Metadata/Java: 14
Dependencies: Reactive Web

Hit the Generate button to download the generated archive, and extract it into your local disk.

Make sure you have installed the latest JDK 14 (AdoptOpenJDK is highly recommended), then import the source codes in your IDEs. eg. Intellij IDEA, and start to implement the server side.

We will skip the demo of Reactor's Sinks here, and directly use a MongoDB capped collection as the backend message queue.

Create a Message document definition and a Repository for it.

interface MessageRepository : ReactiveMongoRepository<Message, String> {
    @Tailable
    fun getMessagesBy(): Flux<Message>
}

@Document(collection = "messages")
data class Message(@Id var id: String? = null, var body: String, var sentAt: Instant = Instant.now())

Create a @RestController to handle messages.

@RestController()
@RequestMapping(value = ["messages"])
@CrossOrigin(origins = ["http://localhost:4200"])
class MessageController(private val messages: MessageRepository) {

    @PostMapping
    fun hello(@RequestBody p: String) =
            this.messages.save(Message(body = p, sentAt = Instant.now())).log().then()

    @GetMapping(produces = [MediaType.TEXT_EVENT_STREAM_VALUE])
    fun messageStream(): Flux<Message> = this.messages.getMessagesBy().log()
}

Here we use a @CrossOrigin annotation to accept origins of the Angular client application.

Start a MongoDB service as follows. You have to prepare a capped messages collection, check this post for more details.

docker-compose up mongodb

Run the following command to start the server side application.

./gradlew bootRun

I also wrote a simple integration test for this SSE, check here and play it yourself if you are interested in it.

Let's move to the frontend application, create a new Angular project, or refactor the codes we have done in the former post.

export class AppComponent implements OnInit, OnDestroy {
  title = 'client';
  message = '';
  messages: any[];
  sub: Subscription;

  constructor(private zone: NgZone, private http: HttpClient) {
  }

  getMessages(): Observable<any> {

    return Observable.create(
      observer => {

        let source = new EventSource("http://localhost:8080/messages");
        source.onmessage = event => {
          this.zone.run(() => {
            observer.next(event.data)
          })
        }

        source.onerror = event => {
          this.zone.run(() => {
            observer.error(event)
          })
        }
      }
    )
  }

  ngOnInit(): void {
    this.messages = [];
    this.sub = this.getMessages().subscribe({
      next: data => {
        console.log(data);
        this.addMessage(data);
      },
      error: err => console.error(err)
    });
  }

  addMessage(msg: any) {
    this.messages = [...this.messages, msg];
    //console.log("messages::" + this.messages);
  }

  ngOnDestroy(): void {
    this.sub && this.sub.unsubscribe();
  }

  sendMessage() {
    console.log("sending message:" + this.message);
    this.http
      .post(
        "http://localhost:8080/messages",
        this.message
      )
      .subscribe({
        next: (data) => console.log(data),
        error: (error) => console.log(error),
        complete: () => {
          console.log('complete');
          this.message = '';
        }
      });

  }
}

Here we observe a EventSource connection in the ngOnInit method, and listen the MessageEvent through the onmessage hook to receive new messages from the server side. The sendMessage uses Angular HttpClient to send a message to the server side.

More info about the details of EventSource API, please go to MDN EventSource page.

There is no changes in the app.component.html as we discussed in the former WebSocket version.

<div fxFlex>
    <p *ngFor="let m of messages">
        {{m}}
    </p>
</div>
<div>
    <form fxLayout="row baseline" #messageForm="ngForm" (ngSubmit)="sendMessage()">
        <mat-form-field fxFlex>
            <input name="message" fxFill matInput #messageCtrl="ngModel" [(ngModel)]="message" required />
            <mat-error fxLayoutAlign="start" *ngIf="messageCtrl.hasError('required')">
                Message body can not be empty.
            </mat-error>
        </mat-form-field>
        <div>
            <button mat-button mat-icon-button type="submit" [disabled]="messageForm.invalid || messageForm.pending">
                <mat-icon>send</mat-icon>
            </button>
        </div>
    </form>
</div>

Next run the client application.

npm run start

Open two browser windows(or two different browsers), type some messages in each window and experience it.

Get a copy of the complete codes from my github and play it yourself.

Building a Chat application with Angular and Spring Reactive WebSocket: Part 2

Hantsy Bai — Sat, 08 Aug 2020 06:07:54 +0000

In this post, we will use MongoDB capped collection instead of the Reactors Sinks.StandaloneFluxSink .

If you have some experience of Linux or Unix, you must know the tail command. When you want to track the last n lines dynamically of some specific files, esp. the server logs while it is growing at runtime, e.g. the access.log file of Apache HTTPd server.

tail -f -n 1000 access.log

Similarly, Mongo provides a tailable cursor feature which acts as the tail command exactly. But there are some limitations, the tailable cursor only works on the capped collections, when a document is inserted, it will be emitted to the subscribers immediately. A capped collection is different from a normal collection, it has a size limit of the whole collection and a maximum count limitation of the stored documents, if the limit is reached, the old documents will be discarded. A document can be inserted into the capped collection like the the normal one, but it can not be deleted by calling delete command.

Let's do a small refactory on the former codes we've completed in the last post and migrate it to use MongoDB tailable cursors based query as an infinite stream.

Firstly add Spring Data MongoDB Reactive as part of dependencies.

implementation("org.springframework.boot:spring-boot-starter-data-mongodb-reactive")

Add a @Document annotation on the Message data class.

@Document(collection = "messages")
data class Message @JsonCreator constructor(
        @JsonProperty("id") @Id var id: String? = null,
        @JsonProperty("body") var body: String,
        @JsonProperty("sentAt") var sentAt: Instant = Instant.now()
)

We specify a collection name here.

Add a Repository for the Message document.

interface MessageRepository : ReactiveMongoRepository<Message, String> {
    @Tailable
    fun getMessagesBy(): Flux<Message>
}

On the getMessagesBy method, we add a @Tailable annotation, which means it will use tailable cursor to retrieve the Message documents from the messages collection.

Modify the ChatSocketHandler to the following.

class ChatSocketHandler(val mapper: ObjectMapper, val messages: MessageRepository) : WebSocketHandler {

    override fun handle(session: WebSocketSession): Mono<Void> {
        println("handling WebSocketSession...")
        session.receive()
                .map { it.payloadAsText }
                .map { Message(body = it, sentAt = Instant.now()) }
                .flatMap { this.messages.save(it) }
                .subscribe({ println(it) }, { println(it) })

        return session.send(
                Mono.delay(Duration.ofMillis(1000))
                        .thenMany(this.messages.getMessagesBy())
                        .map { session.textMessage(toJson(it)) }
        ).then()

    }

    fun toJson(message: Message): String = mapper.writeValueAsString(message)

}

When receiving a message from a WebSocket client, save it into the messages collection, then send the messages in messages collection back to the client.

By default, Spring Data MongoDB does not create a capped collection for the Message document, more details please check the Infinite stream of tailable cursors section of Spring Data Mongo reference.

If the collection is existed, use a convertToCapped to convert it to capped.

@Bean
fun runner(template: ReactiveMongoTemplate) = CommandLineRunner {
    println("running CommandLineRunner...")
    template.insert(Message(body="test")).then().block()
    template.executeCommand("{\"convertToCapped\": \"messages\", size: 100000}")
    .subscribe(::println);
}

Else create a capped collection directly if it is not existed.

template.createCollection("messages", CollectionOptions.empty().capped().size(100000).maxDocuments(1000))

No need change on the client codes.

Make sure there is a running MongoDB server, simply run it in a Docker container.

docker-compose up mongodb

Next let's run the client and server applications respectively.

Open a browser and try to send a message, you can see the changes in the messages collections.

Execute a query in the mongo shell.

> db.messages.find()
{ "_id" : ObjectId("5f130da86bba28157893a9bc"), "body" : "test", "sentAt" : ISODate("2020-07-18T14:56:40.499Z"), "_class" : "com.example.demo.Message" }
{ "_id" : ObjectId("5f130dcd6bba28157893a9bd"), "body" : "test again", "sentAt" : ISODate("2020-07-18T14:57:17.858Z"), "_class" : "com.example.demo.Message" }

Open two browsers, try to send some messages.

It works exactly as the former version.

The complete codes for this post can be found check here, clone it and follow the steps described in the README.md and play it yourself..

Building a Chat application with Angular and Spring Reactive WebSocket

Hantsy Bai — Sat, 08 Aug 2020 06:04:50 +0000

Spring WebFlux as a new module introduced in Spring 5.0 which provides a new programming model for developers, most of the features existed in Spring WebMVC have been ported to the WebFlux stack, including WebSocket support.

WebSocket is a standalone spec defined in RFC6455 , which provides bi-direction none-blocking communication between clients and server side.

In this post, we will start creating a simple chat application which uses Spring WebFlux based WebSocket APIs to build the server side, and uses Angular as client to communicate with the server side. Initially we will use a Reactor specific Sink as the message queue, and then we will switch to use the trailable cursor on the capped collections in MongoDB to simplify the work.

As introduced in my original post , Spring WebFlux embraces ReactiveStreams spec, heavily depends on Project Reactor . The WebSocket API in Spring WebFlux is not so rich as the one in Spring WebMVC, eg. it lacks general controller support and there is no way to adapt the STOMP protocol. In the Google result of "spring webflux websocket", you will find most of the solutions are based on the Reactor 's Processor, eg. How To Build a Chat App Using WebFlux, WebSockets & React is a great article to introduce the usage of WebSocket in Spring WebFlux, for more info about the UnicastProcessor and other processors in Reactor, check How to use Processor in Reactor Java from Manh Phan.

Firstly let's create the server side. Generate a project skeleton using Spring Initializr.

Project type: Gradle
Language: Kotlin
Spring Boot version :2.4.0M1
Project Metadata/Java: 14
Dependencies: Reactive Web

Hit the Generate button to download the generated archive, and extract it into your local disk.

Make sure you have installed the latest JDK (AdoptOpenJDK is highly recommended) 14, then import the source codes into your favorite IDE, eg. Intellij IDEA. IDEA will resolve the dependencies and build the project automatically.

To enable WebSocket in Spring WebFlux application, just declare a simple WebSocketHandlerAdapter bean.

@Bean
fun handlerAdapter(): WebSocketHandlerAdapter = WebSocketHandlerAdapter()

And set up the WebSocket endpoints in a HandlerMapping bean.

@Bean
fun webSocketMapping(mapper: ObjectMapper): HandlerMapping? {
    val map = mapOf("/ws/messages" to ChatSocketHandler(mapper))
    val simpleUrlHandlerMapping = SimpleUrlHandlerMapping().apply {
        urlMap = map
        order = 10
    }
    return simpleUrlHandlerMapping
}

Here we will use a custom ChatSocketHandler to receive from and send message to the endpoint /ws/messages. WebSocket supports text and binary based payload in the message, here we only use text message, we will convert our message to json string by Jackson ObjectMapper.

Let's have a look at the complete codes of ChatSocketHandler.

class ChatSocketHandler(val mapper: ObjectMapper) : WebSocketHandler {
    val sink = Sinks.replay<Message>(100);
    val outputMessages: Flux<Message> = sink.asFlux();

    override fun handle(session: WebSocketSession): Mono<Void> {
        println("handling WebSocketSession...")
        session.receive()
                .map { it.payloadAsText }
                .map { Message(id= UUID.randomUUID().toString(), body = it, sentAt = Instant.now()) }
                .doOnNext { println(it) }
                .subscribe(
                        { message: Message -> sink.next(message) },
                        { error: Throwable -> sink.error(error) }
                );

        return session.send(
                Mono.delay(Duration.ofMillis(100))
                        .thenMany(outputMessages.map { session.textMessage(toJson(it)) })

        )

    }

    fun toJson(message: Message): String = mapper.writeValueAsString(message)

}

The ChatSocketHandler implements WebSocketHandler interface, in the handle method, it will shake hands with a WebSocket client when it is connected. Here when receiving a message from a WebSocket client, we will cache it into a replayable Sinks.StandaloneFluxSink , and retrieve the messages from our former Sink, and send the cached messages back to the WebSocket client.

The UnicastProcessor and ReplayProcessor , etc. are marked as deprecated in the latest version of Reactor, so here we use the newest Sinks instead.

Declare a simple POJO to present the WebSocket message payload in a chat application.

data class Message @JsonCreator constructor(
        @JsonProperty("id") var id: String? = null,
        @JsonProperty("body") var body: String,
        @JsonProperty("sentAt") var sentAt: Instant = Instant.now()
)

Jackson ObjectMapper requires a none-arguments constructor or a JsonCreator annotated constructor when serializing an object.

Now run the server application by clicking the Run icon besides the main fun in the editor of IDEA or executing the following command .

./gradlew bootRun

Starting a Gradle Daemon (subsequent builds will be faster)

> Task :bootRun

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::             (v2.4.0-M1)

2020-07-18 13:57:57.002  INFO 14776 --- [           main] c.e.demo.WebSocketServerApplicationKt
...
<==========---> 80% EXECUTING [1m 1s]
> :bootRun

Let's move to the frontend building - creating a simple Angular app to shake hands with the server side.

I assume you have installed the latest NodeJS and Angular CLI.

Follow the official Getting started guide to setup Angular environment and initialize a new project. Then open the project in your favorite IDEs, eg. VS Code.

To make things simple, we will contribute codes in the top-level AppComponent directly. In a real world application, you should follow the official Angular coding Style Guide to structure your project.

export class AppComponent implements OnInit, OnDestroy {
  title = 'client';
  message = '';
  messages: any[];
  socket: WebSocket;

  constructor(private zone: NgZone) {
  }

  ngOnInit(): void {
    this.messages = [];
    this.socket = new WebSocket("ws://localhost:8080/ws/messages");
    this.socket.onmessage = event => {
      console.log('onmessage:' + event)
      this.zone.run(() => {
        this.addMessage(event.data);
      })
    }
  }

  addMessage(msg: any) {
    this.messages = [...this.messages, msg];
    //console.log("messages::" + this.messages);
  }

  ngOnDestroy(): void {
  }

  sendMessage() {
    console.log("sending message:" + this.message);
    this.socket.send(this.message);
  }
}

Here we initialize a WebSocket connection in the ngOnInit method, and listen the onmessage to receive a message from the server side. And in the addMessage method it calls WebSocket.send to send messages to the server side.

More info about the details of WebSocket API, please go to MDN WebSocket page.

Let's move to the AppComponent template file, app.component.html.

<div fxFlex>
    <p *ngFor="let m of messages">
        {{m}}
    </p>
</div>
<div>
    <form fxLayout="row baseline" #messageForm="ngForm" (ngSubmit)="sendMessage()">
        <mat-form-field fxFlex>
            <input name="message" fxFill matInput #messageCtrl="ngModel" [(ngModel)]="message" required />
            <mat-error fxLayoutAlign="start" *ngIf="messageCtrl.hasError('required')">
                Message body can not be empty.
            </mat-error>
        </mat-form-field>
        <div>
            <button mat-button mat-icon-button type="submit" [disabled]="messageForm.invalid || messageForm.pending">
                <mat-icon>send</mat-icon>
            </button>
        </div>
    </form>
</div>

It just includes a simple form to submit messages, and use a ngFor directive to display the received messages.

Next run the client application.

npm run start

Open two browser windows, and type some words in the input box and hit send button.

Awesome, it works.

In the built-in developer tools panel in Firefox, you can track the details of the shakehands progress of a WebSocket connection request between client and serve side.

The request headers include some specific items, the Connection: keep-alive, Upgrade and Upgrade: websocket are required to start a WebSocket connection.

GET /ws/messages HTTP/1.1
Host: localhost:8080
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Sec-WebSocket-Version: 13
Origin: http://localhost:4200
Sec-WebSocket-Extensions: permessage-deflate
Sec-WebSocket-Key: tsHaoQIeDW0eAk/fHn5kqw==
Connection: keep-alive, Upgrade
Cookie: PLAY_SESSION=eyJhbGciOiJIUzI1NiJ9.eyJkYXRhIjp7ImNzcmZUb2tlbiI6IjBmODcyYzUyMDM3YmJjN2UwZTI4YWRiNjQ4YTA0MGYyNjBiMDVmNzYtMTU2Njg4OTU4MjM4NS0wOThjNDQ5ZjVhOTg4Nzk1YmU0NjQ4ZmUifSwibmJmIjoxNTY2ODg5NTgyLCJpYXQiOjE1NjY4ODk1ODJ9.F4lTngIoAlp8F_vVvLsmw4XSYBtpIGd9yNxlff-8Iuo
Pragma: no-cache
Cache-Control: no-cache
Upgrade: websocket

And the response headers looks like this.

HTTP/1.1 101 Switching Protocols
upgrade: websocket
connection: upgrade
sec-websocket-accept: zq1b1kR56+jGNQ4v1bDr37jfTBA=

The complete codes is hosted on my github account, check the feat/reactor-sinks branch for this demo.