Forem: Leo Cavalcante

Using Bref's LambdaRuntime to Asynchronously Run Swoole Coroutines as Functions on AWS

Leo Cavalcante — Tue, 01 Mar 2022 01:17:19 +0000

Swoole will be shipping something really-really cool that is it's own CLI. You can start playing with it using the pre-compiled binary distributed under Swoole's releases at https://github.com/swoole/swoole-src/releases/tag/v4.8.7.

The trick here, for this project, is: we will be shipping Swoole CLI binary along side with Bref's LambdaRuntime to provide a custom AWS lambda runtime that is Swoole powered.

Let's get started.

Create a directory to hold our files:

mkdir swoole-lambda
cd swoole-lambda

Bring Swoole's IDE helper so we can have code completions in our editor:

composer require --dev swoole/ide-helper

Then we can already bring Bref to the playground:

composer require bref/bref

Bref will provide us the abstraction to communicate with AWS Lambda runtime. We can just call it at our bootstrap file. The bootstrap file is where the lambda runtime will start the invocation:

#!/opt/bin/swoole-cli
<?php

use Bref\Context\Context;
use Bref\Runtime\LambdaRuntime;
use Swoole\Coroutine;

require_once __DIR__ . '/vendor/autoload.php';

$runtime = LambdaRuntime::fromEnvironmentVariable('swoole-cli');
$handler = require $_ENV['LAMBDA_TASK_ROOT'] . '/handler.php';

Coroutine\run(static function () use ($runtime, $handler): void {
    while (true) {
        $runtime->processNextEvent($handler);
    }
});

AWS Lambda will move what is in our bin/ directory to /opt/bin, so our Swoole CLI will be there. That is why we can create our Bootstrap PHP application as a self-executing script that will be using an interpreter located at that path.

Let's grab it:

wget https://github.com/swoole/swoole-src/releases/download/v4.8.7/swoole-cli-v4.8.7-linux-x64.tar.xz
tar -xf swoole-cli-v4.8.7-linux-x64.tar.xz
mkdir bin
mv swoole-cli bin/
rm swoole-cli-v4.8.7-linux-x64.tar.xz

UPX to the rescue! 148MB can be too large for a function. Let's use UPX to make it smaller:

upx -9 bin/swoole-cli

The -9 tells UPX to make it as small as possible. This can take a while, but the final result is a 44MB binary, it is about 30% of the original file size!

Now we can safely create our runtime ZIP file:

zip -r runtime.zip bootstrap bin

And upload it to AWS Lambda as a layer:

aws lambda publish-layer-version \
--layer-name swoole-runtime \
--zip-file fileb://runtime.zip \
--region us-east-1

Now lets zip our vendor files:

zip -r vendor.zip vendor

And upload it as a layer as well:

aws lambda publish-layer-version \
--layer-name swoole-lambda-vendor \
--zip-file fileb://vendor.zip \
--region us-east-1

With the layers uploaded, we are ready to create our function.

The handler.php file which bootstrap requires, holds our function code:

<?php

declare(strict_types=1);

use Bref\Context\Context;

return static fn ($event, Context $context): string =>
    'Hello ' . ($event['name'] ?? 'world');

Zip it:

zip -r function.zip handler.php

And create:

aws lambda create-function \
--function-name swoole-lambda \
--handler handler.handler \
--zip-file fileb://function.zip \
--runtime provided \
--role arn:aws:iam::884320951759:role/swoole-lambda \
--region us-east-1 \
--layers arn:aws:lambda:us-east-1:884320951759:layer:swoole-runtime:1 \
arn:aws:lambda:us-east-1:884320951759:layer:swoole-lambda-vendor:1

Ok, roll the drums.

Let's test it:

aws lambda invoke \
--function-name swoole-lambda \
--region us-east-1 \
--log-type Tail \
--query 'LogResult' \
--output text \
--payload $(echo '{"name": "Swoole"}' | base64) output.txt | base64 --decode

The output should be something like:

START RequestId: eaa39e02-b833-4f06-b18d-7e9a5b603a97 Version: $LATEST
END RequestId: eaa39e02-b833-4f06-b18d-7e9a5b603a97
REPORT RequestId: eaa39e02-b833-4f06-b18d-7e9a5b603a97  Duration: 3.67 ms       Billed Duration: 4 ms   Memory Size: 128 MB     Max Memory Used: 115 MB

Let's see the results:

cat output.txt
"Hello Swoole"

That is all folks!

Bref abstracts way all the hassle about interfacing with AWS Lambda runtime. Internally it uses curl_* which Swoole can hook up and make asynchronously!

And of course, don't need to mention, the Swoole CLI project also rocks a lot bringing us a PHP interpreter with Swoole built-in (statically compiled).

Cloud-native Hyperf

Leo Cavalcante — Tue, 15 Feb 2022 23:31:05 +0000

Originally posted by @Reasno at guxi.me/posts/cloudnative-hyperf

Hyperf officially provides container images and the configuration options are very open. Deploying Hyperf in the cloud itself is not complicated. Let's take Kubernetes as an example to make some modifications to the default skeleton package of Hyperf so that it can run gracefully on Kubernetes. This article is not an introduction to Kubernetes and readers need to have a certain understanding of Kubernetes.

Life cycle

After a container is started on Kubernetes, it performs two checks on the container: Liveness Probe and Readiness Probe. If the Liveness Probe fails, the container will be restarted, and if the Readiness Probe fails, the service will be temporarily removed from the discovery list. When Hyperf is started as an HTTP web server, we only need to add two routes.

<?php

namespace App\Controller;

class HealthCheckController extends AbstractController
{
    public function liveness()
    {
        return 'ok';
    }

    public function readiness()
    {
        return 'ok';
    }
}

<?php

// in config/Routes.php
Router::addRoute(['GET', 'HEAD'], '/liveness', 'App\Controller\HealthCheckController@liveness');
Router::addRoute(['GET', 'HEAD'], '/readiness', 'App\Controller\HealthCheckController@readiness');

Configure on the deployment of Kubernetes:

livenessProbe:
  httpGet:
    path: /liveness
    port: 9501
  failureThreshold: 1
  periodSeconds: 10
readinessProbe:
  httpGet:
    path: /readiness
    port: 9501
  failureThreshold: 1
  periodSeconds: 10

Of course here we simply return 'ok', obviously we can't really check the health. The actual inspection should consider the specific business scenarios and the resources that the business depends on. For example, for heavy database services, we can check the connection pool of the database, and if the connection pool is full, temporarily return the status code 503 in the Readiness Probe.

When the service is destroyed by Kubernetes, it will send the SIGTERM signal first. The process has terminationGracePeriodSeconds this long (default 60 seconds) to terminate itself. If the time is up, Kubernetes will send a SIGINT signal to forcibly kill the process. Swoole itself can correctly respond to SIGTERM to end the service, and will not lose any running connections under normal circumstances. In actual production, if Swoole does not respond to SIGTERM and exits, it is likely that the timer registered by the server has not been cleaned up. We can clean up the timer at OnWorkerExit to ensure a smooth exit.

<?php
// config/autoload/server.php
// ...
'callbacks' => [
    SwooleEvent::ON_BEFORE_START => [Hyperf\Framework\Bootstrap\ServerStartCallback::class, 'beforeStart'],
    SwooleEvent::ON_WORKER_START => [Hyperf\Framework\Bootstrap\WorkerStartCallback::class, 'onWorkerStart'],
    SwooleEvent::ON_PIPE_MESSAGE => [Hyperf\Framework\Bootstrap\PipeMessageCallback::class, 'onPipeMessage'],
    SwooleEvent::ON_WORKER_EXIT => function () {
        Swoole\Timer::clearAll();
    },
],
// ...

Operating mode

Swoole Server includes two operating modes. Single-threaded mode (SWOOLE_BASE) and process mode (SWOOLE_PROCESS).

The Hyperf official skeleton package defaults to Process mode. In traditional service deployment, Process mode will help us manage processes. In Kubernetes deployment, Kubernetes and Kubernetes' Ingress or Sidecar have already undertaken some functions such as pulling, balancing and connection maintenance. Using Process is slightly redundant.

Docker officially encourages the "one process per container" approach. Here we use Base mode and only start one process (worker_num=1).

The Swoole official website defines the advantages of Base mode as:

Base mode has no IPC overhead and better performance.
The BASE mode code is simpler and less error-prone.

<?php
// config/autoload/server.php
// ...
'mode' => SWOOLE_BASE,
// ...
'settings' => [
    'enable_coroutine' => true,
    'worker_num' => 1,
    'pid_file' => BASE_PATH . '/runtime/hyperf.pid',
    'open_tcp_nodelay' => true,
    'max_coroutine' => 100000,
    'open_http2_protocol' => true,
    'max_request' => 100000,
    'socket_buffer_size' => 2 * 1024 * 1024,
],
// ...

After setting up one process per container, our expansion and contraction can be more delicate. Imagine if there are 16 processes in a container, then the number of processes after our expansion can only be a multiple of 16, and each container has one process, we can set the total number of processes to any natural number.

Since there is only one process per container, here we limit each container to use at most one core.

resources:
  requests:
    cpu: "1"
  limits:
    cpu: "1"

Then we configure the Horizontal Pod Autoscaler to automatically scale according to service pressure.

# Minimum 1 process, maximum 100 processes, target CPU usage 50%
kubectl autoscale deployment hyperf-demo --cpu-percent=50 --min=1 --max=100

Log processing

Best practice for Docker containers is to print logs to stdout and stderr. Hyperf logs are divided into system logs and application logs. The system logs have been printed to the standard output, and the application logs are printed to the runtime folder by default. This is obviously not flexible enough in a container environment. We print both to standard output.

<?php
// config/autoload/logger.php
return [
    'default' => [
        'handler' => [
            'class' => Monolog\Handler\ErrorLogHandler::class,
            'constructor' => [
                'messageType' => Monolog\Handler\ErrorLogHandler::OPERATING_SYSTEM,
                'level' => env('APP_ENV') === 'prod'
                ? Monolog\Logger::WARNING
                : Monolog\Logger::DEBUG,
            ],
        ],
        'formatter' => [
            'class' => env('APP_ENV') === 'prod'
            ? Monolog\Formatter\JsonFormatter::class
            : Monolog\Formatter\LineFormatter::class,
        ],
        'PsrLogMessageProcessor' => [
            'class' => Monolog\Processor\PsrLogMessageProcessor::class,
        ],
    ],
];

A closer look at the above configuration will reveal that we have done different processing for different environment variables.

First, we output JSON structured logs in the production environment, because log collection tools such as FluentBit and Filebeat can natively parse JSON logs, distribute, filter, and modify them to avoid complex grok regular matching. In a development environment, JSON logs are not so friendly, and readability plummets when escaping is involved. So in the development environment we still use LineFormatter to output logs.

Second, we output a lot of logs in the development environment, and in the production environment, we need to control the number of logs to avoid clogging the log collection tool. If the log is eventually easy to write to Elasticsearch, it is even more important to control the writing speed. In the production environment, we recommend that only logs above WARNING be enabled by default.

According to the introduction of the official documentation, we also hand over the logs printed by the framework to Monolog for processing.

<?php
namespace App\Provider;

use Hyperf\Logger\LoggerFactory;
use Psr\Container\ContainerInterface;

class StdoutLoggerFactory
{
    public function __invoke(ContainerInterface $container)
    {
        $factory = $container->get(LoggerFactory::class);
        return $factory->get('Sys', 'default');
    }
}

<?php
// config/autoload/dependencies.php
return [
    Hyperf\Contract\StdoutLoggerInterface::class => App\Provider\StdoutLoggerFactory::class,
];

File handling

Stateful applications cannot be scaled arbitrarily. The common state of PHP application is nothing more than Session, log, file upload, etc. Session can be stored in Redis. The log has been introduced in the previous section. This section introduces the processing of files.

It is recommended to upload files to the cloud in the form of object storage. Alibaba Cloud, Qiniu Cloud, etc. are all common suppliers. Private deployment solutions also include MinIO, Ceph, etc. To avoid vendor lock-in, it is recommended to use a unified abstraction layer instead of relying directly on vendor-provided SDKs. league/flysystem is a common choice for many mainstream frameworks including Laravel. Here we introduce the League\Flysystem package and connect MinIO storage through the aws S3 API.

composer require league/flysystem
composer require league/flysystem-aws-s3-v3

Create a factory class and bind the relationship according to the official documentation of Hyperf DI.

<?php
namespace App\Provider;

use Aws\S3\S3Client;
use Hyperf\Contract\ConfigInterface;
use Hyperf\Guzzle\CoroutineHandler;
use League\Flysystem\Adapter\Local;
use League\Flysystem\AwsS3v3\AwsS3Adapter;
use League\Flysystem\Config;
use League\Flysystem\Filesystem;
use Psr\Container\ContainerInterface;

class FileSystemFactory
{
    public function __invoke(ContainerInterface $container)
    {
        $config = $container->get(ConfigInterface::class);
        if ($config->get('app_env') === 'dev') {
            return new Filesystem(new Local(__DIR__ . '/../../runtime'));
        }
        $options = $container->get(ConfigInterface::class)->get('file');
        $adapter = $this->adapterFromArray($options);
        return new Filesystem($adapter, new Config($options));
    }

    private function adapterFromArray(array $options): AwsS3Adapter
    {
        // 协程化S3客户端
        $options = array_merge($options, ['http_handler' => new CoroutineHandler()]);
        $client = new S3Client($options);
        return new AwsS3Adapter($client, $options['bucket_name'], '', ['override_visibility_on_copy' => true]);
    }
}

<?php
// config/autoload/dependencies.php
return [
    Hyperf\Contract\StdoutLoggerInterface::class => App\Provider\StdoutLoggerFactory::class,
    League\Flysystem\Filesystem::class => App\Provider\FileSystemFactory::class,
];

Let's create a new config/autoload/file.php in the way Hyperf is used to, and configure the S3 key and other information:

<?php
// config/autoload/file.php
return [
    'credentials' => [
        'key' => env('S3_KEY'),
        'secret' => env('S3_SECRET'),
    ],
    'region' => env('S3_REGION'),
    'version' => 'latest',
    'bucket_endpoint' => false,
    'use_path_style_endpoint' => true,
    'endpoint' => env('S3_ENDPOINT'),
    'bucket_name' => env('S3_BUCKET'),
];

Like the log, when developing and debugging, we use the Runtime folder for uploading, and in the production environment, we upload pictures to MinIO. To upload to Alibaba Cloud in the future, just install the league/flysystem Alibaba Cloud adapter:

composer require aliyuncs/aliyun-oss-flysystem

And rewrite FileSystemFactory as needed.

Track and monitor

Link tracking and service monitoring itself are not functions provided by Kubernetes, but because the technology stacks in the cloud native panorama can cooperate very well with each other, it is usually recommended to use them together.

Hyperf link tracing documentation: hyperf.wiki/2.2/#/zh-cn/tracer

Hyperf service monitoring documentation: hyperf.wiki/2.2/#/zh-cn/metric

If you have configured base mode and use a process, you do not need to start a separate monitoring process when monitoring the service. Add the following routes to the Controller:

<?php
// Bind the /metrics route here
public function metrics(CollectorRegistry $registry)
{
    $renderer = new RenderTextFormat();
    return $renderer->render($registry->getMetricFamilySamples());
}

If the Prometheus you are using supports discovering crawling targets from service annotations, just add Prometheus annotations to Service.

kind: Service
metadata:
  annotations:
    prometheus.io/port: "9501"
    prometheus.io/scrape: "true"
    prometheus.io/path: "/metrics"

If you use Nginx Ingress, you can configure Opentracing to be enabled. ( nginx ingress documentation )

First configure the Tracer used in the Nginx Ingress Configmap.

zipkin-collector-host: zipkin.default.svc.cluster.local
jaeger-collector-host: jaeger-agent.default.svc.cluster.local
datadog-collector-host: datadog-agent.default.svc.cluster.local

Then open opentracing in the Ingress annotation.

kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/enable-opentracing: "true"

In this way, the link between Nginx Ingress and Hyperf can be opened up.

Complete example

A complete skeleton package can be found on my GitHub: https://github.com/Reasno/cloudnative-hyperf

There are actually many ways to deploy Kubernetes. It is impossible for any skeleton package to be suitable for all situations.

Hello, World!

Leo Cavalcante — Mon, 26 Aug 2019 18:51:10 +0000