Forem: Łukasz Kamieniecki-Mruk

Scramjet Transform Hub — Quick Start introduction

Łukasz Kamieniecki-Mruk — Thu, 28 Oct 2021 16:35:17 +0000

Scramjet Transform Hub - Quick Start introduction

Scramjet Transform Hub allows you to deploy and run multiple data processing apps called sequences.

Sequences are specific apps, not just any apps. They specialize in the efficient data processing.

STH can be treated both as a data processing engine and execution platform for multiple sequences running on the same platform and performing various data processing tasks.

We named our apps "sequences" and that term describes well its nature, as they process data through a sequence of chained functions. Therefore usually, our sequences are concise and easy to write ,and powerful at the same time.

Our vanilla STH engine is based on Node.js and thus allows developers to benefit from the rich ecosystem and many packages and solutions provided by this vibrant community.

The core part of our STH engine is called the "host".

The Host is responsible for maintaining and deploying sequences, keeping them running, and managing their lifecycle.

Host exposes also its own REST API to provide and receive data and manage sequences and host itself.

What we also do on the host level is that we apply a set of algorithms to optimize and speed up data processing execution in sequences.

You can interact with the host using our dedicated STH CLI that will help you with both sequences deployment, running it and monitoring.

2. Solution concept diagram

2.1 Inputs

STH can handle any input that can be handled by Node.js application.
You, as a developer, are free to process a variety of inputs in your sequence applications, such as: Text, JSON, XML, SOAP, Audio, Video, and more.
Inputs can be either:
- Provided to STH via its REST API; or
- Consumed from various local or remote sources by the app; such as Stream, STDIN, File, API, URL
- Generated by the app itself

2.2 Host

This is a solution central processing and management unit with the following major components:

Sequences - these are the actual "STH" apps. It is a gzipped package (*.tar.gz) containing at least two files:
- package.json - JSON manifest file describing the app and its configuration such as the main file to run
- main file - file such as index.js or index.ts that contains lightweight application business logic.
Instance - once the sequence is run, the host will create a separate runtime environment for it and will execute sequence code inside this runtime entity. This is an instance.
API & CLI - our Application Programming Interface and CLI connecting to it allows both for Data operations (sending input data and receiving output data) and Management operations (manage host itself and its entities: sequences or instances)

2.3 Outputs

Our engine outputs can be managed in several ways:

File - you can save your output to the local or remote file
STDOUT - output can be directed to system STDOUT (STERR is supported as well)
API - output can be consumed from our STH REST API
URL Request - you can write your app in a way to request URL, webhook, etc
Stream - output can be streamed to a particular destination
you can mix multiple actions together: you can both send data to a remote system/URL and save it locally.

3 Install Scramjet Transform Hub

3.1 Prepare environment

In order to install Scramjet Transform Hub, please follow these 3 steps:

Get Linux machine (local UNIX/Linux OS, cloud VM , etc)
Install Docker on this Linux machine (official Docker instructions are here)
Install npm on this machine (official instructions are here). Currently, we recommend Node.js version 14.x LTS. ### 3.2 Install STH Open one Linux terminal window and issue the following commands:

1. Install Scramjet Transform Hub and STH CLI

npm i -g @scramjet/sth @scramjet/cli

2. Run STH

scramjet-transform-hub

4 Run your first sequence

4.1 Review the package

Note that all commands here are executed from the root of this repository

We have prepared for you a simple "hello-snowman" sequence that you can use. This sequence is available in the directory samples/hello-snowman in this repository.
In this directory you will find two files:

package.json - the manifest file that describes this particular sequence
index.js - file containing main application logic.

This particular application is written in plain JavaScript to simplify this example. However, you can also write your sequences in TypeScript and build them before packaging and sending the sequence to STH.

There is no need to change anything in our hello-snowman sequence for a first run. Let's move to the next step.

4.2 Prepare and send sequence package

Our "sequence" apps need to be packaged before can be sent to Transform Hub. This package is a simple TAR archive and our STH CLI has a special command to package the app directory into sequence tarball.

Note: any time, you can display STH CLI help by issuing terminal command si help (for general help) or si <command> help for specific command (ie. si sequence help)

Please open a new terminal window (and keep the first one with STH running). Then issue the following commands in the root directory of this repository

a) pack directory hello-snowman into archive hello-sequence.tar.gz

si pack ./samples/hello-snowman/ -o ./samples/hello-snowman.tar.gz

There is no output shown in the terminal but you can verify with ls that the tarball package is created inside the samples directory.

b) send hello-snowman.tar.gz to the running host (default localhost API endpoint will be used by the CLI send command)

si sequence send ./samples/hello-snowman.tar.gz

the output will look similar to this one:

Request ok: http://127.0.0.1:8000/api/v1/sequence status: 202 Accepted
SequenceClient {
  _id: 'cf775cc1-105b-473d-b929-6885a0c2182c',
  host: HostClient {
    apiBase: 'http://127.0.0.1:8000/api/v1',
    client: ClientUtils {
      apiBase: 'http://127.0.0.1:8000/api/v1',
      log: [Object]
    }
  },
  sequenceURL: 'sequence/cf775cc1-105b-473d-b929-6885a0c2182c'
}

Now we have uploaded sequence to the host and host assigned to it a random ID (GUID), in my case our sequence ID is:

_id: 'cf775cc1-105b-473d-b929-6885a0c2182c'

The Host also exposes REST API endpoint for each sequence and this is also described in this response.

4.3 Run the sequence

We can now use sequence ID to run this uploaded sequence. The command is si seq start <sequence_id>. You can also pass an arbitrary number of parameters by providing them after , in case of our hello-snowman parameters are not used.
For example for the above sequence we could write:

si sequence start cf775cc1-105b-473d-b929-6885a0c2182c

the output would look similar to this one:

Request ok: http://127.0.0.1:8000/api/v1/sequence/cf775cc1-105b-473d-b929-6885a0c2182c/start status: 200 OK
InstanceClient {
  host: HostClient {
    apiBase: 'http://127.0.0.1:8000/api/v1',
    client: ClientUtils {
      apiBase: 'http://127.0.0.1:8000/api/v1',
      log: [Object]
    }
  },
  _id: 'e70222d1-acfc-4e00-b046-4a3a9481c53b',
  instanceURL: 'instance/e70222d1-acfc-4e00-b046-4a3a9481c53b'
}

A sequence is an app template. Once run, it will become live as a new instance. The instance also receives its own ID (GUID). In this case, instance Id is:

_id: 'e70222d1-acfc-4e00-b046-4a3a9481c53b'

Of course, sequences can be run multiple times. Each run will create a separate instance with a distinct instance id.

4.4 Send data to the sequence

We want to make your life easier and for this very example, we have prepared a special Node.js app that will generate a stream of simple messages and send them to our running instance of hello-snowman.

For fun, our stream generator will send simple text messages containing temperature readings from an artificial weather station. Temperature values will be generated randomly in the range of <-50,50> degrees Celsius.
Our hello-snowman app will read and interpret these messages and will inform us about the state of our Snowman:

if the temperature will be 0 or below, the sequence will return the message Snowman is freezing ... :)
in the other case (temperature above 0 degrees), the sequence will return the message Snowman is melting! :(

To run this app, please execute this command from the root of our directory node ./tools/stream-gen-tool/stream-gen.js <instance_id>. In our case this would look like this:

node ./tools/stream-gen-tool/stream-gen.js e70222d1-acfc-4e00-b046-4a3a9481c53b

the output would look like this:

----------------------------------------
Message# 1 | Temperature measure
INPUT | 41
OUTPUT| Snowman is melting! :(
----------------------------------------
Message# 2 | Temperature measure
INPUT | -33
OUTPUT| Snowman is freezing ... :)
---------------------------------------------
Message# 3 | Temperature measure
INPUT | -36
OUTPUT| Snowman is freezing ... :)
---------------------------------------------

Our sequence generator app does two things here:

Sends stream of messages; each one containing a number with temperature value
Reads output from Host API that is generated by our hello-snowman sequences

Separately, you can also open a new terminal window and see a log of this particular instance with command si instance log <instance_id>. In our case, this would be

si instance log e70222d1-acfc-4e00-b046-4a3a9481c53b

the sample output will be similar to this one

...
2021-08-09T04:29:39.790Z log (object:Runner) Input message <Buffer 32 30>
2021-08-09T04:29:40.791Z log (object:Runner) Input message <Buffer 2d 34>
2021-08-09T04:29:41.792Z log (object:Runner) Input message <Buffer 33 33>
2021-08-09T04:29:42.798Z log (object:Runner) Input message <Buffer 2d 34 35>
2021-08-09T04:29:43.801Z log (object:Runner) Input message <Buffer 2d 33 36>
...

Well done! You have run your first Scramjet Transform Hub sequence!

5 Where to go next

Here you can find more resources related to Scramjet Transform Hub:

Introduction to Scramjet Data Processing Platform

Łukasz Kamieniecki-Mruk — Thu, 28 Oct 2021 15:35:50 +0000

New engine for serverless data processing applications

Our Scramjet data apps engine has its approach to deploying and running serverless applications; in several aspects different from solutions in areas such as software buses, integration platforms, and FaaS offerings. This article explains our approach. We hope you might find it interesting.

“3 in 1” data processing platform

The heart of our solution and its data engine is called “Scramjet Transform Hub”. It’s available as a standalone software package on GitHub and will be the core element of our Scramjet Cloud Platform.

We name our approach “3 in 1 data processing platform” as it combines 3 concepts into one solution:

data processing engine
serverless data applications
complete API with dedicated CLI (covering both I/O and management endpoints)

Let’s look at each point separately.

Data processing engine

Scramjet Transform Hub creates unified deployment, runtime, management, and execution plan for serverless applications (sequences).

In short, STH allows you to start data processing in 3 simple steps:

Deploy

si sequence send <sequence-package-tar>

Run

si sequence run <sequence-id>

Send data

curl -H “Content-Type: application/octet-stream” — data-binary “@file.txt” <instance-input-endpoint>

You are free to post to our sequence simple HTTP requests, file, send a stream or even read data from another stream or API.
Please notice that, contrary to typical microservices architectures, there is no expensive step of building containerized image, pushing it to the registry, and then downloading it to the container orchestrator to run the microservice. You can move from directory with code to sequence processing your data in less than a minute.

We do package our apps but their size is measured in kilobytes, not in hundreds of megabytes as in the case of container images. Minimal app design gives better performance, optimized resources usage, and a simpler CI/CD process.

We have prepared a short, 3-minute demo, showing the whole application (sequence) preparation, deployment, and run process on our YouTube channel. Check it below:

Serverless data applications

We call the user applications sequences. They have capabilities to perform continuous data and stream processing, they have no run time limits or input data size limits.

Each sequence has a straightforward structure — it’s a directory with at least two core files:

package.json — simple JSON file describing sequence metadata
index.[js/ts] — JavaScript or TypeScript file with sequence code. You are free to structure your app in multiple files if you like.

Below there is the content of one of our sample “hello world” sequences, yielding integer numbers.

const {DataStream} = require(“scramjet”);

module.exports = async function(stream, start = 0, end = 1000) {
 await DataStream.from(async function*() {
 let i = +start || 0;
 while(i++ < end) {
 await new Promise(res => setTimeout(res, 1000));
 yield {x: i};
 }
 })
 .do(console.log)
 .run()
}

You can find intro readme and more samples in our dedicated repository scramjetorg/scramjet-cloud-docs

API & CLI

Let’s look at Transform Hub API via commands available in our CLI:

pack [options] — package directory with sequence code into tar.gz file
host [command] — monitor and check the version of the host
config|c [command] — display and manage config
sequence|seq [command] — pack, deploy, manage and monitor sequences (app templates)
instance|inst [command] — manage and monitor instances (running apps)

The above commands (and related API) cover complete management of the data processing engine and serverless apps running on top of it.
Once started, each running instance exposes the following API endpoints:

input, output
stdin, stdout, stderr
log, monitoring
_event (to instance), event (from instance)
stop, kill

This approach follows the “batteries included” approach, and each running instance is handled in the same way.

Why our Scramjet Cloud Platform

Our approach shown above has several benefits:

Freedom and flexibility — no artificial limits on data size and execution time of apps; no “execution time limit” or “payload size limit”.
Great price for value-effective data workflows with fully programmable data acquisition, ability to create patterns between instances performing various data processing tasks.
Performance by design — instantaneous execution of data without proxies, queues, and gateways. Light apps with minimal resources consumption.
Works cross-native (Edge & Cloud) — out of the box spanning between locations. Run the same type of apps on edge or smart devices via standalone Scramjet Transform Hub and in our Scramjet Cloud Platform.

As a summary, below you will find a diagram showing various patterns of chaining data processing on our platform: