Forem: Daniel Netzer

All we need is graph, graph is all we need

Daniel Netzer — Sun, 10 Jan 2021 18:46:21 +0000

Introducing RxZu, Typescripted graphs engine.

First thing first, what is RxZu?

RxZu is a combination of Rx which is a shorthand for Reactive Extensions and in our specific case Reactive Extensions for Javascript also known as RxJS.
Zu is the Japanese word for drawing, picture, diagram, figure, illustration, chart, graph.
That leads us to an understanding that RxZu is a diagrams engine system, build on top of RxJS, which takes the idea of graphic visualization to the next level in terms of performance, optimization, and customizability.

RxZu is composed of multiple parts, the core engine which is in charge of handling the models and the synchronization between them, and the rendering engine which is in charge of the, well, the rendering of course, and is based on the desired framework utilizing the core engine.

Some of the leading guidelines in the project are minimal, clean code and the ability for customizations and extendibility of the engine entities which are:

Nodes, the main building block of any graph, are the visual representation of data intersections.
Ports, links got to start from some point.
Links, symbols of connectivity and continuity.
Labels, one might want to give a name to a link or even use it for links actions buttons
Custom, want to add your entity? no problem.

Why not use an existing library you're asking yourself

After a long research we reached a very small number of candidates who are leaders in the industry that didn't stand up to a certain standard we defined:

Angular support
Easily extendable and customizable
Lightweight
Extensive support and community

Sadly, all the libraries we found were either extremely heavy and included outdated dependencies such as Lodash, Backbone, etc...
Wasn't open-sourced and didn't have any community around them.
The implementation was outdated, no Typings, unfitting for Angular environment, and introduced lots of overhead and complexity for the simplest use case.

Enough talking, let's code

RxZu currently only implements Angular as a rendering engine therefor all code examples are for Angular.

Let's start by creating a new Angular application, that will display a graph and have a drag n' drop interface to add more nodes.

ng new rxzu-angular
# wait for angular installation to finish
cd rxzu-angular

Install @rxzu/angular

npm i @rxzu/angular

Navigate to ./tsconfig.json and change "strict": true to "strict": false, sadly we don't yet support this and it will introduce some generics typings issues.
rest assured this is a work in progress.

Run the application

ng s

Add to app.module.ts RxZu module

import { BrowserModule } from '@angular/platform-browser';
import { NgModule } from '@angular/core';

import { AppComponent } from './app.component';
import { RxZuDiagramsModule } from '@rxzu/angular';

@NgModule({
  declarations: [AppComponent],
  imports: [BrowserModule, RxZuDiagramsModule],
  providers: [],
  bootstrap: [AppComponent],
})
export class AppModule {}

Now let's create a cool stylish grid as our background, the draggable nodes and our action bar container
app.component.scss

.demo-diagram {
  display: flex;
  height: 100%;
  min-height: 100vh;
  background-color: #3c3c3c;
  background-image: linear-gradient(
      0deg,
      transparent 24%,
      rgba(255, 255, 255, 0.05) 25%,
      rgba(255, 255, 255, 0.05) 26%,
      transparent 27%,
      transparent 74%,
      rgba(255, 255, 255, 0.05) 75%,
      rgba(255, 255, 255, 0.05) 76%,
      transparent 77%,
      transparent
    ),
    linear-gradient(
      90deg,
      transparent 24%,
      rgba(255, 255, 255, 0.05) 25%,
      rgba(255, 255, 255, 0.05) 26%,
      transparent 27%,
      transparent 74%,
      rgba(255, 255, 255, 0.05) 75%,
      rgba(255, 255, 255, 0.05) 76%,
      transparent 77%,
      transparent
    );
  background-size: 50px 50px;
}

.node-drag {
  display: block;
  cursor: grab;
  background-color: white;
  border-radius: 30px;
  padding: 5px 15px;
}

.action-bar {
  position: fixed;
  width: 100%;
  height: 40px;
  z-index: 2000;
  background-color: rgba(255, 255, 255, 0.4);
  display: flex;
  align-items: center;

  * {
    margin: 0 10px;
  }
}

Our html template
app.component.html

<div class="action-bar">
  <div
    *ngFor="let node of nodesLibrary"
    class="node-drag"
    draggable="true"
    [attr.data-type]="node.type"
    (dragstart)="onBlockDrag($event)"
    [ngStyle]="{ 'background-color': node.color }"
  >
    {{ node.type }}
  </div>
</div>

<ngdx-diagram
  class="demo-diagram"
  [model]="diagramModel"
  (drop)="onBlockDropped($event)"
  (dragover)="$event.preventDefault()"
></ngdx-diagram>

And for the last piece in the puzzle, create some nodes, ports, link them up and render it all.
app.component.ts

import { Component, OnInit } from '@angular/core';
import { DiagramEngine } from '@rxzu/angular';
import { DiagramModel, DefaultNodeModel } from '@rxzu/core';

@Component({
  selector: 'app-root',
  templateUrl: './app.component.html',
  styleUrls: ['./app.component.scss'],
})
export class AppComponent implements OnInit {
  diagramModel: DiagramModel;
  nodesDefaultDimensions = { height: 200, width: 200 };
  nodesLibrary = [
    { color: '#AFF8D8', type: 'greenish' },
    { color: '#FFB5E8', type: 'pinkish' },
    { color: '#85E3FF', type: 'blueish' },
  ];

  constructor(private diagramEngine: DiagramEngine) {}

  ngOnInit() {
    this.diagramEngine.registerDefaultFactories();
    this.diagramModel = this.diagramEngine.createModel();

    const node1 = new DefaultNodeModel({ id: '1' });
    node1.setCoords({ x: 500, y: 300 });
    node1.setDimensions(this.nodesDefaultDimensions);
    node1.addOutPort({ name: 'outport1', id: 'outport1' });
    node1.addOutPort({ name: 'outport2', id: 'outport2' });
    const outport3 = node1.addOutPort({ name: 'outport3', id: 'outport3' });

    const node2 = new DefaultNodeModel();
    node2.setCoords({ x: 1000, y: 0 });
    node2.setDimensions(this.nodesDefaultDimensions);
    const inport = node2.addInPort({ name: 'inport2' });

    const link = outport3.link(inport);
    link.setLocked();

    this.diagramModel.addAll(node1, node2, link);
    this.diagramModel.getDiagramEngine().zoomToFit();
  }

  createNode(type: string) {
    const nodeData = this.nodesLibrary.find((nodeLib) => nodeLib.type === type);
    const node = new DefaultNodeModel({ color: nodeData.color });

    node.setExtras(nodeData);
    node.setDimensions(this.nodesDefaultDimensions);
    node.addOutPort({ name: 'outport1', id: 'outport1' });
    node.addOutPort({ name: 'outport2', id: 'outport2' });

    return node;
  }

  /**
   * On drag start, assign the desired properties to the dataTransfer
   */
  onBlockDrag(e: DragEvent) {
    const type = (e.target as HTMLElement).getAttribute('data-type');
    e.dataTransfer.setData('type', type);
  }

  /**
   * on block dropped, create new intent with the empty data of the selected block type
   */
  onBlockDropped(e: DragEvent): void | undefined {
    const nodeType = e.dataTransfer.getData('type');
    const node = this.createNode(nodeType);
    const droppedPoint = this.diagramEngine
      .getMouseManager()
      .getRelativePoint(e);

    const coords = {
      x: droppedPoint.x - this.nodesDefaultDimensions.width / 2,
      y: droppedPoint.y - this.nodesDefaultDimensions.height / 2,
    };

    node.setCoords(coords);
    this.diagramModel.addNode(node);
  }
}

We want to believe the code is self explanatory, but i'll do a quick overview nevertheless.

this.diagramEngine.registerDefaultFactories();

As the name states, registers all default factories provided out of the box by RxZu as a starting point, their source code can be found at https://github.com/Vonage/rxzu/tree/main/packages/angular/src/lib/defaults/components and it is highly recommended to overview it when moving forward into fully customized entities.

const node1 = new DefaultNodeModel();
node1.setCoords({ x: 500, y: 300 });
node1.setDimensions(nodesDefaultDimensions);
const outport1 = node1.addOutPort({ name: 'outport1' });

Instantiating a node entity, which in turn generates the node component and exposes the model for us to manipulate it, update the coordinates, change the dimensions, create outport that is also an entity that instantiates behind the scenes and have lots of manipulations of their own.

I'll stop here, there's plenty more to do and show using RxZu and this is probably the first of many posts about RxZu.

You can find the source code at our GitHub, and read the doc’s and stories at our Storybook which is full with some great and complex examples

What the future holds for us?

One of the most important tasks we have ahead is getting better performance in the core.
Adding support for React, Vue, and more...
Smarter links with obstacles awareness.
Improving the documentation.

May the Log Be With You

Daniel Netzer — Tue, 17 Dec 2019 16:05:00 +0000

Part three in a series about how to deploy a secured, monitored, optimized web application (short bonus part)

One of the most important things a software need, not just a web application, is the ability to log all the events and errors that occur when it's being used by our end users so we can actually see what went wrong and fix it before the user even finish the email to the support team.

This part will introduce how to utilize Sentry, the SaaS based logging service that provide real time error monitoring, tons of SDK's for every major language and framework out there, and what's important for us is the ability to integrate with our Git to see the errors per version/code change.

Prerequisites

A few things we'll need to get started is of course a Sentry account, our amplify.yml from the first post and an application we want to use it with.

Once the sentry account is up and you added the appropriate SDK to your application (this part is out of the scope for this post) we can add it to our pipeline that will generate source-maps to actually point to our original code before it's compilation/bundling/uglification/etcification took place.

Integrate It!

Most application will initialize Sentry in the following manner:

Sentry.init({
  dsn: "https://XXXXXXXXXXXXXXXXXXXXXXX@sentry.io/XXXXXX",
  release: <% COMMIT_HASH %>
});

The <% COMMIT_HASH %> is what we will replace pre-building our application so Sentry can point to the specific commit that added the changes either breaking or fixing an issue.

In the amplify.yml file at the preBuild stage on my project (this change depending on the software you are building) I call a script I built to change all the environment variables during CI. replacing <% COMMIT_HASH %> with AWS_COMMIT_ID.

Mapping

To upload the relevant source maps we will have to define a few environment variables at the Amplify console, SENTRY_AUTH_TOKEN, SENTRY_ORG and SENTRY_PROJECT.

Afterwards we can add to our amplify.yml in the postBuild step the following lines

- npx sentry-cli releases new AWS_COMMIT_ID
- npx sentry-cli releases set-commits --auto AWS_COMMIT_ID
- npx sentry-cli releases files AWS_COMMIT_ID upload-sourcemaps path/to/source-maps -x .js -x .map --validate --verbose --rewrite --strip-common-prefix
- npx sentry-cli releases finalize AWS_COMMIT_ID

which is pretty self explanatory, once a build is successful you should see in your sentry dashboard a new release and when an error is logged now and it's associated to the release it will pinpoint the code that changed or fixed that error which is pretty amazing if you ask me.

Hope you enjoyed this short bonus part, in the following post I'll introduce another great tool to help with visual regression snapshots.

Protect and Monitor Your Front End Application

Daniel Netzer — Thu, 28 Nov 2019 19:43:52 +0000

Part two in a series about how to deploy a secured, monitored, optimized web application to AWS Amplify

In this part I will introduce Snyk the doberman, here to serve and protect your application from any package trying to oppose a security threat and the simplicity of integrating it as part of the deployment to AWS Amplify.

Now seriously what Snyk does is lots of amazing things, starting with detecting vulnerable dependencies during coding in your IDE to testing your running environment to verify there is no exposure to existing vulnerabilities and monitor for newly disclosed vulnerabilities.

Lease the Doberman

To get started, first we need an account on Snyk which offers a free tier.
Afterwards we'll go to our project root directory and install the Snyk cli.

npm i -g snyk, once installation is finished we need to authenticate with our account, snyk auth.

Now the entire Snyk options are in game, you can read about them here - https://snyk.io/docs/getting-started-with-our-cli/

I usually just let the Snyk wizard do the hard work for me using snyk wizard command and follow the steps.

Doggie, Sit!

So how do we utilise this great tool in our CI/CD?
We'll have to add a few lines to our amplify.yml in the postTest step if you have that test stage (I hope you do), or in the postBuild step.

lines 30, 35 (DEV.to doesn't know how to handle lines highlighting, sounds like the right PR 😏)

snyk test is pretty straightforward and will test and scan our package.json in a recursive manner and check that none of the packages is vulnerable and in case he detect a security issue he will fail our test phase.

snyk monitor will take a snapshot of the deployed packages and will monitor them on a daily basis unless we specify otherwise on the Snyk dashboard.

At Vonage we hooked the Snyk alerts into our internal systems such as Slack and once an issue comes up we address it immediately.

Man's Best Friend

Last but not least, source control management integration to test, review and monitor all of the repositories, notifying if anything found and creating a PR to update the package if a newer secured version is available using smart indexing to raise the minimum required version so no breaking changes will be introduced.

** Bonus fact: Snyk have a patching mechanism for unmaintained widely used packages which is part of their service, you can read about it more here

Part 1 - Enterprise Grade CI/CD For Frontend Applications

In the next post I'll review the logging services I use and enjoy as part of creating a solid front end application that is built to last.

Credits

The amazing cover photo is by Jen B on Unsplash

Enterprise Grade CI/CD For Frontend Applications

Daniel Netzer — Thu, 21 Nov 2019 19:42:10 +0000

This post is part one in a series of posts

Following the recent acquisition of my former company Over.ai by Vonage lots of changes and demands came up, some for better automation, other is to use the company cloud provider("aws"), better security and monitoring.

Then I decided to create the ultimate enterprise grade, battle tested, aws based CI/CD pipeline for our entire frontend applications, let's start with the methodology I've decided to work with.

Git Flow

Git-flow is a branching model and release management strategy for Git. It defines a well-formulated path for a project’s development life cycle and ensures that the development team adheres to the processes. It offers a set of extensions over Git to provide a high-level repository operation.

Local

Using git pre commit hook with husky to run several tools making sure the code base is linted in the same manner, commit messages are using the conventional-commits standard and removing unused and unreachable code.

E.g. using tslint for linting, commitizen for conventional commits, etc…

.cz-config.js (commitizen cz-customizable)

Master

Contains production code. Each commit to the master will have special process of bumping the application version, generating a changelog automatically and adding a tag to the commit using semantic-release.

.releaserc

Develop

Contains the latest development changes that will be included in the next release.

Feature/*

A new branch is created for each new feature we work on.
We start if from develop and once we’re done, merge back into it.

Release/QA

Starts from develop and signifies that there will be a new release once we merge this branch into master.

Hotfix/*

Starts from master and merges into both master and develop.
Used when we need to deliver urgent changes to our production app but develop is not yet ready to produce a release branch.

All of the following branches will run automated tests, the release branch will be manually tested and checked.

Master, Release and Develop branches will be locked for direct code commits and only pull requests will be merged into them after a code review and making sure all unit and e2e tests are passing.

Every single one of those branches will spin up a preview application with a unique url and that will happen automatically of course, using the AWS Amplify.

AWS Amplify Console

The AWS Amplify Console provides a Git-based workflow for deploying and hosting fullstack serverless web applications. A fullstack serverless app consists of a backend built with cloud resources such as GraphQL or REST APIs, file and data storage, and a frontend built with single page application frameworks such as React, Angular, Vue, or Gatsby.

Amplify bring the following features out of the box and with a simple configuration:

Branch Detection using a regex
View test results in Amplify console directly.
Easy domain setup especially when using Route53.
Application are deployed to AWS CloudFront CDN and globally available.
Preview application for every branch configured, either using a specific subdomain or a unique url auto generated.
Password protection for preview application.
Setup alarms and notification using cloudwatch and amplify console.

Cypress test results shown in the amplify console, with links to recorded e2e video's for each test.

Amplify configuration can be handled in the project itself with an amplify.yml file, with a CLI or directly on the console.

amplify.yml

References:

https://nvie.com/posts/a-successful-git-branching-model/
https://aws.amazon.com/amplify/console/

Recommended Packages:

https://www.npmjs.com/package/husky
https://github.com/conventional-changelog/commitlint#readme
https://github.com/commitizen/cz-cli
https://github.com/palantir/tslint / https://github.com/eslint/eslint

Thank you very much for reading this post, in the next following posts I will show how to add visual regression snapshots as part of our testing suite and snyk to monitor dependencies in our production application and to test each pull request.