Forem: Ampla Network

LLMs are NOT the product

Ampla Network — Sun, 20 Oct 2024 14:55:27 +0000

Understanding the Limits of System Prompts

Language models, like LLMs, can't keep data safe just by using a system prompt. The system prompt can give instructions, but it can't make sure that the data is handled properly or that sensitive information stays secure.

A company that wants to build a product exposing an LLM with some context risks leaking sensitive data and experiencing unexpected behavior.

Let's try a small experiment
I will give a Llama model a system prompt, asking it to act as a customer support agent and keep the data safe.

Now I will try to get the information using a direct sentence.

As we can see, the model relies on the system prompt and is not willing to retrieve information following a prompt that attempts to violate its rules.

But there are multiple ways to try communicating with an agent. I will try another approach, playing its game and perhaps pushing it into a situation where it might feel like it's making a mistake.

Let's try to force it to answer the question first. After all, my request will make sense, right?

Try to invert the position
Okay, it is smart enough for now, but let's take it up a level and pretend that I am not the person it received the information from.

Am I able to still force it to give me back what it has stored?

Let's find out right now.

Try to make it doubt
Obviously, the model will increasingly rely on history to enforce more constraints as my attempts are not working well.

I need to use this to my advantage, perhaps. Let's try to use instructions to break the model's primary goal. To do this, my prompt needs to be long enough to hide my real goal.

Since some models cannot focus on the entire prompt and history but rely on the end of the message, we can attempt to 'recode' a new objective.

Eureka
Oh, that is interesting now. What just happened? And why so fast? Okay, let's try to break this down:

I was adding more and more history where I was talking about getting user information, then I suddenly became very precise in asking for a change of behavior, trying to refine the instruction rules, without:

Asking again for user information
Asking to break a rule

So I just asked without pointing it in a specific direction, BUT as it relies on the context, it will naturally use the context and tries to use it to answer a completely different question.

Breaking a model can be as simple as this sometimes... sometimes not, but whatever you have in mind, the model is not your product.

What you build and sell is a product, where the model is just one component.

Developers still need to build a robust system that will hide the model and protect it from this kind of injection.

It takes a lot of work to use AI properly, but if you do it the right way, your product can be a game changer.

Thank you to have taken the time to read this small article.

Please, let me know what you are interested in for the next one ?

Context Inversion Exploit: Or how to reverse a context to make a model bypass its censorship?

Securing Exposure : What to do to make your AI product not to breach and break your code

Ripple, React shared state with less code...

Ampla Network — Thu, 22 Jun 2023 12:47:09 +0000

Introducing Ripple, Your Go-To State Management Library for React

Welcome to Ripple, the latest innovation in state management libraries for React. Designed with simplicity and performance in mind, Ripple aims to simplify the way you handle states in your React applications.

Ripple official website

Unlike more complex state management solutions like Redux or MobX, Ripple offers an intuitive, easy-to-use interface and delivers unrivaled speed, streamlining your development workflow and boosting your productivity.

Whether you're a seasoned React developer or just getting started, Ripple provides a straightforward approach to state management that's efficient and effective.

Riding the Wave with Ripple

Ripple is an intuitive shared state management library for React that keeps things simple. Think of it as Recoil's kindred spirit, offering seamless updates via hooks or from outside the React environment. The philosophy behind Ripple is to let the components focus on rendering, and leave the state updates to outside handlers.

Let's say, you need to update your state based on a WebSocket event, asynchronously from a Promise, or maybe from a service class... Ripple’s got your back! It even empowers you to trigger state updates from within a High Order Component using the right hook.

Let's delve deeper into how Ripple operates:

The Ripple Lake

In Ripple, you'll be dealing with the concept of a 'Ripple Lake.' A Lake is essentially a repository of data, a tranquil space where all your ripples belong. To implement a ripple in your component, you declare it in a Ripple Lake. The good news is, you can create as many Lakes as you wish.

Start by initializing a new file in your project, say ripple-lake.ts, and import the createRipples function from Ripple.

// ripple-lake.ts
import { createRipples } from "@m-c2/ripple";

To initialize a Lake, call createRipples and add all the ripples you want to use:

// Ripple definitions
const globalRipple = {
  value1: "value1"
};

const ripple1 = {
  value1: "value1"
};

// Lake definition
const [hooks, services] = createRipples({
  globalRipple,
  ripple1
});

// Export the hooks and services
export const rippleHooks = hooks;
export const rippleServices = services;

The createRipples method returns an array with two elements - the hooks and the external updaters. Hooks is an object containing as many hooks as ripples you've declared. Updaters, on the other hand, is an object housing functions that allow you to update the ripples from anywhere in your code.

This is where the magic of TypeScript comes into play. The hooks and updaters will be typed automatically based on the ripple type.

Ripple in Action Inside Components

To use Ripple inside your components, import the hooks from your ripple-lake and destructure them. This will give you access to the specific ripples you need.

Here's an example using useGlobalRipple:

// component-one.ts
import { rippleHooks } from "./ripple-lake";
const { useGlobalRipple } = rippleHooks;

const ComponentOne = () => {
  const [globalRipple, setGlobalRipple] = useGlobalRipple();

  return <div>{globalRipple.value1}</div>;
};

You can also destructure the hooks directly in the component declaration:

// component-one.ts
import { rippleHooks } from "./ripple-lake";
const { useGlobalRipple } = rippleHooks;

const ComponentOne = () => {
  const [globalRipple, setGlobalRipple] = useGlobalRipple();
  const { value1 } = globalRipple;

  return <div>{value1}</div>;
};

The beauty of Ripple is that every time you update globalRipple from any component or anywhere outside React, all the components that use it will be updated automatically. This does away with the need for any manual registration for updates, streamlining the refresh of data and ensuring only visible components using the hook are updated.

A Deep Dive into Ripples

A Ripple in the Lake of our state management tool represents an object, akin to an Atom in Recoil. Ripples are the fundamental units of state in our Lake, and each one contains a part of the overall application state. They are declared in the Lake using the createRipples function, taking a name and an initial state.

When you access a Ripple from a component, you receive a copy of the Ripple state and a set function to update the state. You can modify this local copy of the Ripple in your component, but keep in mind that these changes won't be reflected back in the Lake.

Why is that? The Ripple source is immutable. Changes to a Ripple are made through the updater and only then are these changes propagated to the Lake. Subsequently, these modifications spread to all the components that use the Ripple.

Let's illustrate this with a simple Ripple definition:

// my-lake.ts
... = createRipples({
  counter: { // Counter ripple
    count: 0,
  },
});

We've created a basic Ripple named counter with a single property, count, initialized at 0.

Ripple in the Component

This counter Ripple can be used within a component as follows:

// MyComponent.tsx
import { rippleHooks } from './my-lake';
const { useCounter } = rippleHooks;

const MyComponent = () => {
  const [counter, /*...updaterHook...*/] = useCounter();

  return (
    <div>
      <p>Counter: {counter.count}</p>
    </div>
  );
};

The useCounter hook returns a tuple consisting of the Ripple state and an updater function.

Remember, we can't directly modify the source of the Ripple (like counter.count++) and expect it to update everywhere. But we can perform a local modification and then call the updater function to propagate this change to the Lake. The source value remains unaltered, but the Lake gets updated with the new value.

Destructuring the Ripple

One of the key features of Ripple is its internal linking mechanism. You can destructure a Ripple and still be rerendered when the Ripple changes.

// MyComponent.tsx (destructuring)
import { rippleHooks } from './my-lake';
const { useCounter } = rippleHooks;

const MyComponent = () => {
  const [counter, setCounter] = useCounter();
  const { count } = counter;

  return (
    <div>
      <p>Counter: {count}</p>
      <button onClick={() => {
        counter.count++;
        setCounter();
      }}>Increment</button>
    </div>
  );
};

In this case, although destructuring may look more verbose, it showcases the flexibility offered by Ripple. Local modifications can either be applied to the Lake or cancelled via the updater function.

Behind the Scenes

Under the hood, Ripple tracks local changes using a Proxy object. Therefore, a Ripple isn't a straightforward copy of the state object. While a Proxy works similarly to the actual object, there are subtle differences, like Array.isArray not functioning as expected with an array.

When a Ripple is used, the Lake creates a Proxy object to track the changes. This approach ensures that even complex Ripple structures with deeply nested objects don't hamper performance. We track changes, so only modified properties get updated at once or cancelled as needed.

The primary goal here is to minimize the need for developers to include business logic inside components. Let your components focus on rendering, and let Ripple handle the state management.

Ripple Updates from a Component

Inside a component, using a Ripple hook feels similar to using React's useState, with one key distinction: the hook takes the name of a Ripple, not an initial state. You use the Ripple to access values and the updater to change the Ripple's state.

The updater can be used with or without parameters, depending on your intentions. By default, invoking the updater without parameters applies all current modifications made to the Ripple object.

Example: Using the Updater

Consider this simple example, where we define a counter Ripple and subsequently utilize it in a component:

// my-lake.ts
import { createRipples } from "@m-c2/ripple";

const lake = createRipples({
  counter: { count: 0 }
});

export const rippleHooks    = lake[0];
export const rippleServices = lake[1];

In this scenario, we directly modify the counter object, then invoke the updater without any parameters. This call applies all modifications made to the Ripple object.

Updater Parameters

While the updater can be invoked without parameters to apply all modifications, you can also provide parameters to apply specific changes. Here's a breakdown of the different parameter options:

Function	Parameter	value	Description	Trigger a render
setCounter	void		Apply all pending changes	true
setCounter	string	`restore`	Cancel the current local modification	false
setCounter	string	`reset`	Reset the ripple to its initial state	true
setCounter	string, TRipple	`replace`, Object	Replace the current ripple state with a new one, does not impact the initial state in case of reset	true

And here's how you can use them:

// Apply all pending changes
setCounter();

// Cancel the current local modification
setCounter('restore');

// Reset the Ripple to its initial state
setCounter('reset');

// Replace the current Ripple state with a new one,
// does not impact the initial state in case of reset
setCounter('replace', { count: 0 });

Parent-Child Approach

Since a Ripple is shared between all components that use it, updates can be made from any of these components. This allows you to adopt a Parent-Children approach effortlessly.

// Parent component
const Parent = () => {
  const [counter, setCounter] = useCounter();

  return (
    <MyComponent
      increment={() => {
        counter.count++;
        setCounter();
      }}

      reset={() => {
        setCounter("reset");
      }}
    />
  );
};

External Updates of Ripples

Ripples are designed for use within a React application, but you can also update them externally. This flexibility allows you to design your application without compromises on architectural patterns.

Example: Using External Updates

Consider this example where we have a counter Ripple, a service to update this Ripple outside the React lifecycle, and a component that uses this service to update the counter.

Lake definition

// This defines the lake where the ripples will belong to.
import { createRipples } from "@m-c2/ripple";

const lake = createRipples({
  counter: { count: 0 }
});

// Export the hooks for component usage only
export const rippleHooks = lake[0];
// Export the services for external usage
export const rippleServices = lake[1];

Service

// This defines the services that will be used
// to update the ripples outside the React lifecycle.
import { rippleServices } from "./Lake";
const { updateCounter } = rippleServices;

class CounterService {
  get counter() {
    return updateCounter();
  }

  increment() {
    this.counter(_ => { _.count++ });
  }

  reset() {
    this.counter(_ => "reset" );
  }
}

export const counterService = new CounterService();

Component

// This is a simple component that will use the counter service
// to update the counter value.
import React, { CSSProperties } from 'react';
import { rippleHooks } from "./Lake";
import { counterService } from "./Service";

const { useCounter } = rippleHooks;

// Define some styles
const styles = {
  // Styles are omitted for brevity
}

const MyComponent = ({ increment, reset }) => {
  const [{ count }] = useCounter();

  return (
    <div style={styles.div}>
      <p style={styles.p}>Counter: {count}</p>
      <button
        style={styles.button}
        onClick={() => counterService.increment()}
      >Increment</button>
      <button
        style={styles.button}
        onClick={() => counterService.reset()}
      >Reset</button>
    </div>
  );
};

export default MyComponent;

The Updater Service

Let's take a closer look at the updater service:

import { rippleServices } from "./Lake";
const { updateCounter } = rippleServices;

The updateCounter function has different behaviors depending on its parameters:

Function	Parameter type	value	Description	Trigger a render
updateCounter	void	undefined	Return the counter value	no
updateCounter	function	handler: (ripple) => void \	"restore" \	"reset" \

// updateCounter(void) => ripple
const ripple = updateCounter();

// updateCounter(handler: (ripple) => void | "restore" | "reset" | typeof ripple) => void
updateCounter(ripple => { ... });

The handler function also exhibits various behaviors depending on its return type:

Function	return type	return value	Description	Trigger a render
handler	void	undefined	apply all pending modification to the ripple	yes
handler	string	restore	cancel pending modifications	no
handler	string	reset	reset the ripple to its initial state	no
handler	typeof ripple	a new ripple	replace the ripple with a new value. Does not replace the initial state	yes

// apply all pending modification to the ripple
updateCounter(ripple => {
  ripple.count++;
});

// cancel pending modifications
updateCounter(ripple => "restore");

// reset the ripple to its initial state
updateCounter(ripple => "reset");

// replace the ripple with a new value. Does not replace the initial state
updateCounter(ripple => {
  return {
    count: 0,
  };
});

This allows us to have external control over the Ripple's state, enabling various state manipulation outside the standard React component lifecycle.

Conclusion

Over the course of this article, we've delved into the details of using Ripples in a React application to manage state. Starting from the basic concepts of Ripples and Lakes, we journeyed through the creation and utilization of Ripples within React components, and the specifics of how their state can be updated both internally and externally.

Ripples provide a solution to state management that is flexible, simple, and comprehensive, allowing you to manage and propagate state changes with a minimum of boilerplate code. It leverages modern JavaScript features such as Proxies to track changes, ensuring that only the necessary parts of your application re-render when the state updates, improving the performance of your applications.

Key features of Ripples include:

Immutability: Ripples remain immutable, preventing accidental state mutations and helping maintain predictable state management in your React applications.
Flexibility: Ripples can be updated both internally, from inside a React component, or externally, providing flexibility and adaptability to various architectural patterns.
Proxy Usage: Behind the scenes, Ripples use JavaScript Proxies to keep track of changes, which boosts performance and allows for optimal resource utilization.
Conciseness: With its straightforward API and services, Ripples can simplify your codebase by reducing the boilerplate code commonly associated with state management.

In summary, the Ripple state management approach is a powerful tool for any developer working with React. It encapsulates many of the best principles of modern JavaScript and React, resulting in a state management system that's efficient, predictable, and developer-friendly.

Whether you're working on a small project or a large-scale application, Ripples can help streamline your state management and make your code cleaner, more maintainable, and easier to reason about.

Happy coding!

Going deeper with typescript advanced types

Ampla Network — Thu, 03 Dec 2020 06:53:19 +0000

In this post we will see how we can use Typescript typing system to create a Mapper helper.

Let's imagine we have an object like this one :

interface IGraphicControl {
  width     : number;
  height    : number;
  alpha     : number;
  fillColor : string | number;

  drawRect(x: number, y: number, width: number, height: number): void;
  render(): void;
}

Now if we need to set several properties we need to do the following.

const myGraphic = new Graphic();

myGraphic.width  = 100;
myGraphic.height = 100;
myGraphic.alpha  = 1;

myGraphic.fillColor = 0x00FF00;
myGraphic.drawRect(0,0,50,50);

myGraphic.fillColor = 0x0000FF;
myGraphic.drawRect(50,50,50,50);

myGraphic.render()

We want to simplify the mapping a little bit so we can do this :

setTo(myGraphic, {
  width     : 100,
  height    : 100,
  alpha     : 1,
  fillColor : 0x00FF00,
  drawRect  : [0,0,50,50] // Call the function
})

We want to be able to define all properties with the correct values, and call functions with parameters as tuples. But we want that for every object we pass as the first parameter, the second parameter provides the right intellisense.

To create such a function, we will have to extract all the information from the first parameter.

We will need to extract all properties and functions, and treat the functions as tuples of parameters, correctly typed.

Step 1

Create a type that will invalidate properties that do not correspond to the type you are looking for.

type ConditionalTypes<Base, Condition> =  {
  [Key in keyof Base]: Base[Key] extends Condition ? never : Key;
}

So we create a type in the form of a hashMap whose keys are the properties of the Base type, and whose type will be either a string of the name of the key, or an impossibility of assignment.

type newFilteredType = ConditionalTypes<IGraphicControl, Function>;

// Will be the same as 
type newFilteredType = {
  width     : "width";
  height    : "height";
  alpha     : "alpha";
  fillColor : "fillColor";

  drawRect : never;
  render   : never;
}

So why create a type whose properties are string values ?
Simply because now we can extract those types.

Step 2

We need to extract the valid keys, but it is not possible to list the keys we want to keep. Instead we can extract all property types of a type, excluding those of never type.

// We will change this
type ConditionalTypes<Base, Condition> = {
  [Key in keyof Base]: Base[Key] extends Condition ? never : Key;
}

// to 

type ConditionalTypes<Base, Condition> = {
  [Key in keyof Base]: Base[Key] extends Condition ? never : Key;
}[keyof Base]

Now we can retreive all types excluding nerver types. The tricky part comes here, as each valid type is a string :-). We will retreive all valid names as string.

type newFilteredType = ConditionalTypes<IGraphicControl, Function>;

// Will be the same as 
type newFilteredType = "width" | "height" | "alpha" | "fillcolor";

Step 3

Now we need to extract the real types of the selected keys.
We will use the Pick type.

// We will change this
type ConditionalTypes<Base, Condition> = {
  [Key in keyof Base]: Base[Key] extends Condition ? never : Key;
}[keyof Base]

// to 

type ConditionalTypes<Base, Condition> = Pick<Base, {
  [Key in keyof Base]: Base[Key] extends Condition ? never : Key;
}[keyof Base]>

And then this will result in the following

type newFilteredType = ConditionalTypes<IGraphicControl, Function>;

// Will be the same as 
type newFilteredType = {
  width     : number;
  height    : number;
  alpha     : number;
  fillColor : number | string;
}

Yessssss, we got it !!!

Step 4

We need now to get all fields that are not functions, and all that are functions to process them differently.

So let's change our type again

type ConditionalTypes<Base, Condition> = Pick<Base, {
  [Key in keyof Base]: Base[Key] extends Condition ? never : Key;
}[keyof Base]>

// to 

type ConditionalTypes<Base, Condition, Extract extends Boolean> =  Pick<Base, {
  [Key in keyof Base]: Extract extends true ? 
    Base[Key] extends Condition ? Key : never 
    : 
    Base[Key] extends Condition ? never : Key
}[keyof Base]>;

We added an third type that extends boolean, so we will use it to define if we want to extract selected type, or exclude it.

Now we are able to get what we want.

type newFilteredType = ConditionalTypes<IGraphicControl, Function, false>;

// Will be the same as 
type newFilteredType = {
  width     : number;
  height    : number;
  alpha     : number;
  fillColor : string | number;
}

// AND
type newFilteredType = ConditionalTypes<IGraphicControl, Function, true>;

// Will be the same as 
type newFilteredType = {
  drawRect(x: number, y: number, width: number, height: number): void;
  render(): void;
}

Step 5

We are now able to separate properties into two categories, functions and the remainder.

We need to rebuild a type whose functions will no longer be defined as functions, but as an array of typed parameters.

We will use the Parameters type, that can extract all parameter types and put them in a tuple.

type ParameterType<T> = Partial<
  ConditionalTypes<T, Function, false> // Properties that are not functions
  & 
  { 
    [K in keyof ConditionalTypes<T, Function, true>]: Parameters<ConditionalTypes<T, Function, true>[K]> // Tuple
  }
>;

Step 6

The target prototype is

function setTo<T>(source: T, value: ParameterType<T>): void

And to use it

setTo(myGraphic, {
  width     : 100,
  height    : 100,
  alpha     : 1,
  fillColor : 0x00FF00
});

setTo(myGraphic, {
  drawRect: [0,0,50,50]
}

setTo(myGraphic, {
  render: []
}

We still need to do an extra call to render after because the render should not be called at the same time, but after. So it is not very usefull as is.

Final step

As a bonus, we will add a way to Chain several call without the need to pass the source as a parameter

function setTo<T>(source: T, value: ParameterType<T>) {
  for(const key in value) {
    if (key in source) {
      typeof source[key as keyof T] === "function" ? 
        (source[key as keyof T] as unknown as Function).apply(source, (value as unknown as any)[key])
        :
        source[key as keyof T] = (value as unknown as any)[key];
    }
  }
 return (nextValue: ParameterType<T>) => setTo(source, nextValue);
}

We did it !

As a result, we can now do the following

setTo(myGraphic, {
  width     : 100,
  height    : 100,
  alpha     : 1,
  fillColor : 0x00FF00 
})({
  drawRect  : [0,0,50,50]
})({
  alpha     : 0.5,
  fillColor : 0xFFFF00,
})({
  drawRect  : [50,50,50,50]
})({
  render: [];
})

For big declaration like animations, this can reduce the amount of code. This sample may not be the most accurate but it shows you how much powerful typescript can be.

On a day to day basis, you don't need to deal with advanced typing, but if you create helpers in libraries or frameworks, you can provide a very usefull intellisense and type constraint that will save developers a lot of time and debugging hours..

Enjoy !

Making a voxel engine

Ampla Network — Wed, 11 Nov 2020 05:53:13 +0000

Introduction

This series of posts will focus on creating a voxel engine, from scratch, based on BabylonJS for the low-level 3D routines support.

To begin, here is in the video below, the first target we will have to reach, in order to manage the rendering of the world.

So what is a voxel ?

To keep it simple, a voxel is in 3D what a pixel is in 2D. It is value in grid, in a 3D space.

Strictly speaking, the voxel is like a pixel, in the meaning that it has only one value, its color.

Voxel engines generally have a little more flexibility in the degree to which the display of a voxel is done. It can display a cube single colored, or textured as in Minecraft.

So displaying cubes is not a big deal, isn't it ?

Short answer : Yes... and no.

A 3D engine, in order to keep a good frame rate, can apply many optimizations to the 3D scene to render.

It can hide the non visible objects, or simplify the objects according to the camera distance.

The problem with voxels is that you will have a very large quantity of cube, so even if you try to hide some of them, you will quickly struggle in rendering speed.

Moreover, a cube is a simple geometric shape, and therefore, simplifying this object cannot be done without severely deforming it. Remove one node and the cube becomes anything you want except... a simpler cube.

So okay, but where to start then?

Let's start with something basic, which is to define some target functionalities that we are going to implement.

We're going to take our inspiration from the way Minecraft handles the rendering of worlds in the game, at least in the early versions of the game.

We will try to use as few technical terms as possible, just the bare minimum required, in order to keep all the explanations understandable to everyone.

World structure

The world

A world represents a set of voxels that it will be possible to display.The world is divided into regions.

The region

A region represents a piece of the world. Each region has the same number of voxels. A region is also represented by a 3D coordinate. A region is composed by a data chunk.

A chunk

A chunk is composed of a set of voxels, in a 3-dimensional grid, where each dimension is the same size. This can be simplified as a cube filled with small cubes.

Let's assume for example that a data chunk is composed of 3 dimensions of size 32. A region thus has 32*32*32 voxels, a total of 32768 voxels.

If our world has 100*100 regions per layer and let's say 3 layers of height, we will have a total of 100*100*3 regions, so 30000 regions.

Our world will thus have a total of 100*100*3*32768 = 983 040 000 voxels. Our very small world already has close to a billion potential voxels.

Block definition

Our voxel, in our engine, will be presented as a block, more complex in structure than a simple 3D point.

export type Block = {
  name  : string; // Block name
  guid  : string; // Unique global Id
  uid   : number; // Unique local id
  sidesTex : [ // Array of textures
    string, // BACK
    string, // FRONT
    string, // RIGHT
    string, // LEFT
    string, // TOP
    string  // BOTTOM
  ];
  size: [ // Edges size
    number, // WIDTH
    number, // HEIGHT
    number  // DEPTH
  ];
  type    : string; // GAZ, LIQUID, BLOCK
  opacity : number;
  speed   : number; // 0 - 1
};

So we have the smallest usable unit.

Each block will need some data to represent each side, for optimization purpose. Let's define an enum to represents sides.

export enum Side {
  Left     = 1  ,
  Right    = 2  ,
  Forward  = 4  ,
  Backward = 8  ,
  Top      = 16 ,
  Bottom   = 32 ,
  Z_Axis   = 3  ,
  X_Axis   = 12 ,
  Y_Axis   = 48 ,
  All      = 63
}

Chunk definition

A chunk will store different kind of data, including the full and the optimized version of the blocks.

export type Chunk = {
  position   : Vector3       ; // 3D position in the world
  size       : number        ; // Size of the chunk, default will be 32
  data       : Array<number> ; // The original data
  dataSize   : number        ; // The number of non empty blocks
  rcData     : Array<number> ; // An optimized version of visible only visible data
  rcDataSize : number        ; // The number of visible blocks
  hasRc      : boolean       ; // Define if a chunk has been optimized or not
};

1D array or the power of flattening everything

When dealing with Typescript / Javascript, it is easy to deal with array of array. It seems common to proceed like this.

But here, we need to keep in mind that performance will decrease rapidly as soon as we add new features, so we need to avoid wasting our precious frame per second by taking the easy way out.

Using a one-dimensional array to simulate a 3-dimensional access will always be faster. We will therefore use functions to simplify our work.

/**
 * Convert a vector 3 coordinate to a flat array index
 * @param x {number} The x coordinate
 * @param y {number} The y coordinate
 * @param z {number} The z coordinate
 * @param size {number} The size of each dimension, the size is the same for each one
 */
export function vector3ToArrayIndex(x: number, y: number, z: number, size: number = 32) {
  return (size * size * x) + (size * y) + z;
}

/**
 * Convert a flat array index to a 3D coordinate representation
 * @param index {number} The array index
 * @param size {number} The size of x,y,z dimension
 */
export function arrayIndexToVector3(index: number, size: number = 32) {
  return new BABYLON.Vector3(
    (index / (size * size)) >> 0,
    ((index / size) % size) >> 0,
    (index % size) >> 0
  );
}

This will conclude our introduction. In the next post, we will see how to render our blocks using Babylon Js, and the minimum of 3D terminology necessary to understand the next posts.

Enjoy !

React 02 - Beginner : Understanding data, Props vs State.

Ampla Network — Fri, 30 Oct 2020 02:19:33 +0000

We saw in the previous post how in React we could create a simple component, and we also saw how to pass an arbitrary data to be displayed through an attribute.

We will see this time how to manage data in a more complete way, trying to understand how React manages immutable and mutable data within components.

Let's start with the example of the previous post.

// How to define
function Label(props: {text: string}) {
  return <span>{props.text}</span>;
}

{/* How to use */}
<div>
  <Label text="Hello Kitty"/>
</div>

Props

When we use a component, and we want to pass a data to it, we send this information through an attribute. And for the component, it is a field within an object that we named Props (For properties).

From the component's perspective, a property is a value that does not belong to it, it comes from its parent, which can be another component, or it can be called from the root code fragment.

At this point, let's consider that this just comes from the outside.

These data are considered immutable, i.e. they can be seen as read-only. We can / must NEVER modify them.

The validity of the data state is the parent's responsibility. A component that uses only such data will be predictable. It will be called a pure function, in the meaning that the result will always be the same according to the value of the parameters given to it.

The data can be content display information, as well as technical data about the component's internal status.

In our example, we had a text, so raw display data. Let's add for example a style data.

// How to define
function Label(props: {text: string, color: string}) {
  return <span style={ {color: props.color} }>{props.text}</span>;
}

{/* How to use */}
<div>
  <Label text="Hello Kitty" color="red"/>
</div>

So we added a field in the Props object to manage the color, and connected this field in our component. We can manage a bit of everything this way.

How, if we delegate the state of the data to the parent, can we make the component interactive?

As we are in (Java/Type)script, and it is a functional language, passing a function as a parameter is something common, so it is just as simple to expect to pass a function as an attribute.

// How to define
function Label(props: {text: string, onClick: () => void}) {
  return <span onClick={props.onClick}>{props.text}</span>;
}

{/* How to use */}
<div>
  <Label text="Hello Kitty" onClick={() => console.log("Clicked !")}/>
</div>

In this way the parent can be notified, make changes and provide new data to the component. Each time the data is changed, the component will be redrawn automatically.

State

To keep it simple, the State will define data that is local to the component, and which will be unknown outside the component, and that will be mutable.

Let's imagine that our component displays or hides its content on click, it will be its responsibility to hide or not the content, and therefore it will have to manage its state locally.

To do this, we will use the useSate method to define a state. This method is called a Hook but it is not the topic of this post.

// How to define
type Props = { title: string, content: string };

function Label(props: Props) {
  // Define the state, expanded is the value
  // setExpanded a function to mutate the value
  const [expanded, setExpanded] = useState(false);

  return <div>
    <h6>{props.title}</h6>
    <div onClick={() => setExpanded(!expanded)}>{props.content}</div>
 </div>
}

The useState function will have as parameter, the initial value. expanded will be the current value in the scope, and to mutate it, we will use the setExpanded function. It is quite simple.

Now, each time a state is mutated, React will re render this component automatically. You can define as many states as you need, you are not forced to have one to store all values.

Be aware that when you mutate a state value, it will be fully replaced. So it is better to split the state logic in several state values depending on the complexity of the component.

For example, we can, at a minimum, separate the storage of the internal states from those of the displayed data.

That's all for this post. In the next one, we'll see how to adopt the mindset needed for component design, and when to choose the use of properties, or states.

Enjoy.

React 01 - Beginner : React function... Or How beauty comes from simplicity

Ampla Network — Wed, 28 Oct 2020 19:56:16 +0000

This post is intended for an audience that doesn't necessarily have the time to give React a try, but who would like to understand what it's all about, basically. This is the first post of a serie that will cover a full overview.

Let's start at the beginning. Let's take 2 features of the web that every front-end developer knows well.

HTML, for formatting, and scripting (Javascript or Typescript) for logic.

Let's start with our most basic Hello world HTML template.

<div>
  <span>Hello Kitty</span>
</div>

I know, I almost lost you, but I think you survived.

Joke aside, the good thing about HTML is that like any XML-based description, its tree structure is easy to read and represent in your head. And each tag allows you to represent a particular element.

What I really want to do when creating a template is to extend this HTML to naturally put tags representing the components I want to create and use, as naturally as the code above.

And that's where the second part comes in, scripting. Because yes, you're going to have to code a bit, it's not magic either.

Suppose we want to keep it simple, and create a component that represents our Hello Kitty.

React will allow us to do this via a definition that perfectly combines the best of both worlds, called JSX (See this little introduction)

Thanks to JSX, you will be able to use your HTML code directly in your JS/TS code. Think of it as the possibility to materialize a tag, as a stand-alone Object, no more, no less.

function HelloKitty() {
  return <span>Hello Kitty</span>;
}

It cannot be more simple.

And just with this, you are now able use it as it was always a valid HTML tag of the same name as the function.

<div>
  <HelloKitty />
</div>

The initial HTML template needs to be placed in a JSX file for sure, to be correctly understood, but the syntax remains exactly the same.

Now let's be crazy, and let's transform a little bit our component to be able to put any wording, a touch of dynamism doesn't hurt.

As we are in a function, it can take an optional parameter, which will simply represent the attributes of this new tag.

// In Javascript
function Label(props) {
  return <span>{props.text}</span>;
}

// In Typescript
function Label(props: {text: string}) {
  return <span>{props.text}</span>;
}

Again it pushes the simplicity at a high level.
We have defined a new attribute named text of type string to show an arbitrary wording.

Let's modify our template accordingly.

<div>
  <Label text="Hello Kitty"/>
</div>

Wait wait wait !!! That's all ? No hidden fees ? Just define it and use it ? No need to use a kind of decorator or mixin ? No need to configure some injection ?

You want us to buy a tech that is the same thing as we already use on a day to day basis ?

Short answer : That's the beauty of React. Why adding another layer of knowledge specifically WHERE you don't need to.

React can do far more obviously, but this is only a short overview. The next post will cover how to design a component, and the mindset behind React.

Enjoy !!

Why React uses, in my opinion, the perfect rendering engine

Ampla Network — Wed, 28 Oct 2020 00:59:32 +0000

In 20 years of web development, I have been able to test a lot of libraries, frameworks, and other ways to write components and pages.

As soon as you want to build something a bit advanced, you quickly end up having to use templating engines, to facilitate dynamic rendering.

There have been quite a few different approaches to handling this. We've had server-side approaches, like in Asp and Php, with portions of dynamic code in server tags.

Then came approaches that were more and more focused on client-side rendering, and with them a lot of template engines. The majority of these template engines allow some flexibility, with conditions, loops etc..

But I've never felt fully satisfied with having to integrate directives into HTML code. Of course, placing an ngIf or a v-if is fine, but I find that it gets away from the very essence of what the web has to offer.

The web browser offers HTML templating, CSS styling and Javascript scripting, I never understood why it was so complex not to be able to conciliate this without having to destructure one's thinking to do so.

When I started learning Angular, I had been doing web design for a long time, and I wondered how it could have come to this...

As if the difficulties encountered with the first version of ASP.NET hadn't taught us a lesson... Having such a framework is good... When you come from a backend world practicing C# :-) But it's a bit like seeing the sun through your window knowing that you have to stay locked inside. It's a bit of a hindrance to creativity.

Then one day, I discovered React... and saw this :

render(React.createElement(React.Fragment, null,
    React.createElement("div", null,
        React.createElement("div", null, "Hello")));

O.M.G How you dare write this to target that

<div>
  <div>Hello</div>
</div>

And then I said to myself I will NEVER use this piece of 💩 ... until they added what I call a game changer.

In 3 letters... JSX.

The first time I saw a piece of code in JSX I said : Are you kidding me guys ? You want me to return a HTML tag in the middle of my code ?? 😆

And then I reread it again...

function MyComponent(props: {param: string, list: string[]}) {
  const listBody = (
    <ul>
    {props.list.map(_ => <li>{_}</li>)}
    </ul>
  );

  return (
    <div>
      This is the body with {props.param}
      {listBody}
    </div>
  );
}

And this time I said : These guys are genius...; This is exactly how we need to reconcile HTML and Script together as there was no frontier between them...

Few days later, a friend of mine was looking some React code written in JSX and said : No way, I'm more confortable to explicitly place a directive in a HTML tag, it is more natural than returning HTML in a script !?

Then I asked him how in a simple code, he will write a loop when he wants to write a loop... And the answer was...

for ( let i=0; i<10; i++) {
  ...
}

Yep, obviously, when you want a loop, you write a loop, because you are just using a natural coding skill, the way it should be used.

So I told him if declaring an object as a literal makes him feel unconfortable.

const myObj = { someKey: "someValue" };

And, no, it seems again normal.

So I asked him that if some fields can be declared between 2 curly braces, why it will be more complex to define an object between 2 tags ?

const myDiv = <div>My content<div/>;

The object is clearly defined too, and the result is stored in the myDiv variable. Maybe it is more easy to write complex code by inserting Tags into the code that the reverse.

You have to code the template, not to templatize the code. No need to encapsulate fake code inside strings here. You use HTML with code the easiest way possible.

In the beginning of this post I said Why React uses, in my opinion, the perfect rendering engine.

I'm not saying that React itself is the perfect library, event Vue can use JSX.

What I'm just saying, is that using JSX templating, and in particular with React, makes me feel as free as I always was when designing pure HTML components.

Enjoy, and remember : Don't feed the troll 😄 especially in comments...

React 00 - Beginner : React + Typescript from scratch

Ampla Network — Mon, 26 Oct 2020 19:44:44 +0000

This post aims to show how to start a simple project in React and Typescript, without using create-react-app or any other scaffolding tool.

We will see how to install the minimum number of dependencies to start and try to explain each command or dependency addition.

You need to have node.js installed on your dev environment.

Setup environment

To get started, create a directory for your project.
I would use Vs code in this post.

Open the directory from Vs code, then open a command line. You should have something like that

Now you need to initialize the package.json to save the dependencies and scripts.

I usually use yarn but you can use npm too

# With Yarn
yarn init -y

# With npm
npm init -y

This will add a package.json file at the root.
We need to add typescript as a dev dependency to compile our code.

# With Yarn
yarn add -D typescript

# With npm
npm i -D typescript

We also need to install webpack to pack the project and make it suitable for the browser

# With Yarn
yarn add -D webpack webpack-cli

# With npm
npm i -D webpack webpack-cli

And to run the project locally we need a little standalone http server

# With Yarn
yarn add -D http-server

# With npm
npm i -D http-server

A little bit of configuration

In order to make the compiler works properly we need to add a typescript configuration file:

# With Yarn
yarn tsc --init

# With npm
npx tsc --init

this will add a default tsconfig.json file at the root.

Edit the file, and configure it as follow :

{
  "compilerOptions": {
    "target"                           : "es6"      ,
    "module"                           : "commonjs" ,
    "jsx"                              : "react"    ,
    "outDir"                           : "./out"    ,
    "rootDir"                          : "./src"    ,
    /* Default flags */
    "strict"                           : true       ,
    "esModuleInterop"                  : true       ,
    "skipLibCheck"                     : true       ,
    "forceConsistentCasingInFileNames" : true
  }
}

This configuration will assume the following :

The target output will be ES6 compliant
The JSX templates will be compiled by the typescript compiler so no need to use babel
The output will be placed in the out folder
The source code will be placed in the src folder

At this point, you can write and compile files, but we need now to configure Webpack to pack the output.

Create a file named webpack.config.js at the root, and put the following inside

const path = require('path');

module.exports = {
  entry  : './out/App.js',
  output : {
    path     : path.resolve(__dirname, 'www/js'),
    filename : 'app.js'
  },
  externals : {
    "react"     : 'React',
    "react-dom" : 'ReactDOM'
  },
};

This configuration will assume the following :

The entrypoint is located here : ./out/App.js
The output will be placed here : ./www/js.app.js
react and react-dom packages will not be packed as we will add them from a CDN, to speed up the packing time.

You first `TSX` file

When coding React with Typescript, you will not use JSX files with javascript but the counterpart TSX files. Exactly the same syntax but with typescript instead of javascript inside.

So Let's create our first file in the src folder, named App.tsx

import * as React from "react"     ;
import { render } from "react-dom" ;

render(<> Hello kitty </>, document.querySelector("#app"));

Very simple sample of code, but it does the job :-)

Now you need a last addition, the index.html file.

Create an index.html file located here : www/index.html

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <script crossorigin src="https://unpkg.com/react@16/umd/react.development.js"></script>
  <script crossorigin src="https://unpkg.com/react-dom@16/umd/react-dom.development.js"></script>
  <title>Document</title>
</head>
<body>
  <div id="app"></div>
  <script src="js/app.js"></script>
</body>
</html>

At the end of the file, we added a div with the id app, and the script reference to the packed application.

It's building time

To build and pack, you just need 2 commands

# With Yarn
yarn tsc

# With npm
npx tsc

This will compile the project and output the result in the out folder

Then you need to pack it

# With Yarn
yarn webpack

# With npm
npx webpack

If you want to use the watcher to not re run the commands each time, you can add the -w flag to run each command in watch mode

# With Yarn
yarn tsc -w

# With npm
yarn webpack -w

If everything goes well you have this :

Browsing the result

To finish you need to serve the project locally, so you can start a webserver to serve it

# With Yarn
yarn hs ./www

# With npm
npx hs ./www

Now open your browser and navigate to http://localhost:8080 to see this (I hope)

Enjoy !

The power of advanced typing in Typescript

Ampla Network — Fri, 23 Oct 2020 03:09:24 +0000

When I started working on version 3 of S-Flux, I really wanted to facilitate the use of the Application Stores and the methods to interact with them.

To do this, I wanted to leverage the power of advanced Typescript typing.

The inputs

Let's take a quick look at the inputs.

export default Flux.registerStore({
  id: "Counter",
  async init(): CounterState { 
    return { counter: 0 };
  },
  actions : _actions,
  events  : _events
});

We define a unique Id, an init function to initialize the state, some actions and some events.

An action is an async method whose first parameter is the Payload.

// This parameter is not a real parameter :-)
async setCounter(this: TThis, payload: { counter: number }) { 
  /* ... */
  return _events.ForceCounter; // Return specific event 
}

An event is a key/value pair defining an event that can be returned by a store with a specific subscription.

The target output

Now we need to see what we want as an output.

{
  getState() : CounterState ;
  id         : string       ;
  events: {
    All          : string ;
    ForceCounter : string ;
  },
  subscribeTo: {
    All          : ((state: CounterState) => void) => { off: () => void };
    ForceCounter : ((state: CounterState) => void) => { off: () => void };
  },
  actions: {
    increase   : () => void                             ;
    decrease   : () => void                             ;
    setCounter : (payload: { counter: number }) => void ;
  }
}

In this example, the store wrapper will expose new methods based on the input types.

Events represents the same events as in the input, but for each event a corresponding method is generated in the subscribeTo object to register to the event when emitted by the store, plus a 'All' handler to catch all events.

Same for the actions object. For each action handler that has been registered, the counterpart method should be exposed to call the handler with the correct payload.

With this approach we will avoid the need to create manually some action helpers for each possible action.

The problem

If we create the wrapper manually, everything will be typed correctly, but the point is that the code will be created dynamically in the registerStore function.

So the return type will be any for the State, the actions and events objects. That's why we need to use the advance typing system to provide a fully typed object with all methods even for code that will be dynamically produced.

It is exactly for this kind of feature that the use of typescript rather than javascript seems obvious to me.

Preparing the types

Let's see the input type as it is defined in the registerStore method signature :

export type TStoreDefinition<S extends (...args: any[]) => any, T extends {[key: string]: (...args: any[]) => any}, U> = {
  id              ?: string                      ;
  localActions    ?: boolean                     ;
  actions          : T                           ;
  events          ?: U                           ;
  mappedActions   ?: { [key: string] : string; } ;
  init             : S                           ;
  dispatchHandler ?:(this: TBaseStore<ReturnType<S>>, payload: any, For? : TAwaitFor) => Promise<void | null | string | string[]> ;
  nextState       ?: (newState: Partial<ReturnType<S>>, mergeToPreviousState?: boolean) => void                                   ;
}

We need to infer types from actions, events, and init.

actions contains keys that we want to extract
events contains keys that we want to extract
init contains a return type equal to the store State, so we need to extract it too.

To allow typescript to infer those types and to work with them, we need to dedicate a type for actions events and init, and to ensure that the input is correct, we add some constraint when declaring them.

Generating the types dynamically

Let's see how to expose the output methods of the actions object from the input actions field :-)

Extracting the field type.

We can use the PropType type to get a specific field type.

type TActions = PropType<typeof def, "actions">;

PropType type is itself declared as follow :

export type PropType<TObj, TProp extends keyof TObj> = TObj[TProp];

When declaring 'TProp extends keyof TObj', we will get all real keys of TObj, that's how we will have all typing for keys we don't know yet.

Now 'TActions' is of type T with the constraint defined in the signature

T extends {[key: string]: (...args: any[]) => any}

TActions is infered, so it is not only a hashmap with a key of type string and a value as a method, it contains the correct keys declared in the input. That's the important part. Each key is fully declared and can be used.

Creating the target type

We need to create the target type that will expose the same keys but with diffrent values.

export type TActionExtention<T, U> = {
  [P in keyof T]: 
  ((this: TBaseStore<U>, payload: any) => Promise<void | null | string | string[]>) | 
  ((this: TBaseStore<U>, payload: any, For: TAwaitFor) => Promise<void | null | string | string[]>)
};

[P in keyof T] ensures that every named key in T will have some constraints defined just after.

So in our case, every value of each key will be of one of the 2 types below.

((this: TBaseStore<U>, payload: any) => Promise<void | null | string | string[]>)
((this: TBaseStore<U>, payload: any, For: TAwaitFor) => Promise<void | null | string | string[]>)

We need to iterate on input keys to output the new defined types

const _actions = {} as {
  [P in keyof TActions] :  /* ... */
};

_action has the same keys as TActions, and TAction is a type infered from the actions field in the input.

Conditional typing

We need to create a method type that will have as first parameter the correct payload. So we have to extract the payload type from the first parameter of the input method.

We can extract it like that :

PropType<Parameters<TActions[P]> , 0>

Parameter will extract the parameters from TActions[P] as a hashmap where the key is the parameter index, and the value the parameter type itself.

PropType<..., 0> will extract the property type for the field named 0.

So the definition can now be

const _actions = {} as {
    [P in keyof TActions] :  (payload: PropType<Parameters<TActions[P]> , 0>) => void
  };

Yes !!! but wait a minute... The input method can have a payload.... or not, so this parameter can be undefined.

This is when the Wouaou effect occurs. You can define types for something you don't know... and you can even test the type to add a condition... It is just amazing.

So the final type will look like this

const _actions = {} as {
    [P in keyof TActions] :  PropType<Parameters<TActions[P]> , 0> extends undefined ?
    (() => void) 
    :
    ((payload: PropType<Parameters<TActions[P]> , 0>) => void) 
  };

And now we have a deal ! The result is a fully type object defining all methods we need with strongly types signatures to avoid syntax / typing errors at runtime.

That is why I love typescript so much :-)

And to finish, we want to expose a getState method that will return a value equals to the store state. We never defined directly a store state, but we have defined an init function whose return type is the one we need.

So to extract our store state we will do the following

type TState = ReturnType<PropType<typeof def, "init">> ;

In 3 steps, we will get the def type, then extract the init field, which is a method, then its return type.

Conclusion

I hope this post will help you to see how powerfull Typescript's typing is.

If you want to know more you can check the Typescript documentation about advanced types : Link here

To see the source code of the example you can check S-Flux here

Enjoy.

Expression Parser : Part 2 - Define and implement a visitor for MongoDB

Ampla Network — Sat, 10 Oct 2020 15:17:41 +0000

In the first part, we saw how to use Chevrotain to write a small parser. The post is available here.

To use the output of a parser, i.e. a syntax tree, we have several solutions. We can discard the interpreter, which is unsuitable in our case, and focus on either the Listener or the Visitor.

The main difference between the Listener and the Visitor is that the Listener will walk through the tree in one pass, node by node, from start to finish, triggering events related to the traversal, while the Visitor can decide when and how the nodes will be visited.

A Xml type language can be parsed with a Listener, as SAX parsers do. A language such as C# will have to go through a Visitor to allow further analysis and optimizations that will require to go through some nodes several times.

Defining the Visitor

Our goal is for our micro filtering language to be usable by multiple database providers, we need to start by defining the interfaces that represent it, in order to provide a model for the various implementations.

Each non-terminal node will be represented by a method. Each method will take a context object that will contain the specific data to understand and use them during the traversal.

andOrExp example

So let's try to define the andOrExp node. To start with, let's create a method to represent it.

  /**
   * [expression]--(AndOp | OrOp)?--[expression]?
   * @param ctx 
   */
  andOrExp:(ctx: AndOrExpNodeContext) => unknown;

The method should return an unknown type because we cannot define the method return type. It will be set depending on the database provider.

The AndOrExpNodeContext object should represent all data that allow us to interact with all tokens or non terminal nodes connected to this one.

export type AndOrExpNodeContext = CstChildrenDictionary & {
  lhs    : [ExpressionNode] ;
  rhs   ?: ExpressionNode[] ;
  AndOp ?: IToken[]         ;
  OrOp  ?: IToken[]         ;
}

The nodes and tokens available through the context will be represented as an array, because these elements can be defined several times. The node on the left can only be defined once, so it is typed as an array of a single element.

We need to do the same for each non terminal node. The definition will look like this :

export interface IFilterInterpretor {
  /**
   * [andOrExp]--[orderBy]?--[skip]?--[take]?
   * @param ctx ExpressionsContext
   */
  expressions: (ctx: ExpressionsContext) => unknown;

  /**
   * [expression]--(AndOp | OrOp)?--[expression]?
   * @param ctx 
   */
  andOrExp: (ctx: AndOrExpNodeContext) => unknown;

  /**
   * (OrderBy)--(Identifier)+--(Asc | Desc)+
   * @param ctx 
   */
  orderBy: (ctx: OrderByNodeContext) => unknown;

  /**
   * (Take)--(Integer)
   * @param ctx 
   */
  take: (ctx: TakeNodeContext) => unknown;

  /**
   * (Skip)--(Integer)
   * @param ctx 
   */
  skip: (ctx: SkipNodeContext) => unknown;

  /**
   * [compareRule] | [inExp] | [notInExp] | [parentAndOrExp]
   * @param ctx 
   */
  expression: (ctx: ExpressionNodeContext) => unknown;

  /**
   * (Identifier)--(EqOp | NotEqOp | GtOp | GteOp | LtOp | LteOp)?--[atomicExp]
   * @param ctx 
   */
  compareRule: (ctx: CompareRuleNodeContext) => unknown;

  /**
   * (Identifier)--(InOp)--[array]
   * @param ctx 
   */
  inExp: (ctx: InExpNodeContext) => unknown;

  /**
   * (Identifier)--(NotInOp)--[array]
   * @param ctx 
   */
  notInExp: (ctx: NotInExpNodeContext) => unknown;

  /**
   * (LParen)--[andOrExp]--(RParen)
   * @param ctx 
   */
  parentAndOrExp: (ctx: ParentAndOrExpNodeContext) => unknown;


  /**
   * (Integer) | (Float) | (String) | [dateExp]
   * @param ctx 
   */
  atomicExp: (ctx: AtomicExpNodeContext) => unknown;

  /**
   * (Dt)--(LCurly)--(String)--(RCurly)
   * @param ctx 
   */
  dateExp: (ctx: DateExpNodeContext) => unknown;

  /**
   * (LBraket)--[atomicExp]--(Comma)*--[atomicExp]*--(RBraket)
   * @param ctx 
   */
  array: (ctx: ArrayNodeContext) => unknown;
}

Implementing the visitor for MongoDB

We will see the strategy used to transform our initial filter into a MongoDB usable version. For this we need to implement a visitor based on the previous definition.

The global rule definition

We need to return the global filtering object as it is needed by MongoDB.

  expressions(ctx: Filter.ExpressionsContext)  {
    const query = ctx.andOrExp ? { "$query" : this.visit(ctx.andOrExp) } : {};

    return  {
      filter: query ,
      aggregate: [
        ctx.orderBy && this.visit(ctx.orderBy, true),
        ctx.skip    && this.visit(ctx.skip),
        ctx.take    && this.visit(ctx.take)
      ].filter(_ => _)
    } as ExpressionResult;
  }

As you can see, we focus only on what the current rule should do, and rely on the result returned by other nodes when necessary.

To get the result of an orderBy rule, for example, we just have to call the visit method with the orderBy context available in the current context. .filter(_ => _) is used to remove empty elements.

Returning the result as ExpressionResult type will allow the method to infer the result and force the unknown type to become an ExpressionResult type instead of an any type.

A more complex one, the andOrExp

  andOrExp(ctx: Filter.AndOrExpNodeContext) {
    let leftHandSide = this.visit(ctx.lhs);

    let opTokens = [] as IToken[];
    ctx.AndOp && opTokens.push(...ctx.AndOp);
    ctx.OrOp  && opTokens.push(...ctx.OrOp);

    let rightHandSide = [] as any[];

    if (ctx.rhs) {
      rightHandSide = ctx.rhs.map(_ => this.visit(_));
    }

    rightHandSide.unshift(leftHandSide);
    opTokens = opTokens.sort((a,b) => a.startOffset - b.startOffset);

    if (rightHandSide.length === 1) return rightHandSide.pop();
    let prev = rightHandSide.shift();

    opTokens.forEach(_ => {
      prev = { [`$${_.image}`] : [ prev, rightHandSide.shift() ] }
    });

    return prev;
  }

What makes it more complex ? The answer is simple, Chevrotain vitisor contexts are table based and not recursived. This means that if the current node has a many chained node, all occurences of the node are represented in an array at the same level.

So if in the current node we have this : ( XXX eq 10 and (YYY eq 20 or YYY eq 25)) and ZZZ eq 30 or ZZZ eq 35, how to properly handle all AND and all OR tokens ?

In our rule definition, AND and OR operators are alternatives, but declared as 2 arrays. And each right-hand expression that comes after an operator is provided in an expression type array too.

As we can have left and right expression, we need to sort everything in order to build the correct filter as a result.

expression nodes

Left and right expression rule is named lhs and rhs, for Left and Right hand side, but are of the same type. We know that the left expression is always defined, but not the right ones.

We can build an expression array to get all right expressions, and add the left one at the begining. This array will contain all expressions already sorted by default.

For the operators, we need to merge and sort all of them in one array too.

 let opTokens = [] as IToken[];
 ctx.AndOp && opTokens.push(...ctx.AndOp);
 ctx.OrOp  && opTokens.push(...ctx.OrOp);
 /* ... */
 opTokens = opTokens.sort((a,b) => a.startOffset - b.startOffset);

Now that all operators and expressions are sorted, we can process all operators from the operator array, and we will find the corresponding expression at the same index in the expression array.

The final class looks like this :


export class MongoDBFilterVisitor extends BaseCstVisitor  implements IFilterInterpretor {
  constructor() {
    super();
    this.validateVisitor();
  }

  expressions(ctx: Filter.ExpressionsContext)  {
    const query = ctx.andOrExp ? { "$query" : this.visit(ctx.andOrExp) } : {};

    return  {
      filter: query ,
      aggregate: [
        ctx.orderBy && this.visit(ctx.orderBy, true),
        ctx.skip    && this.visit(ctx.skip),
        ctx.take    && this.visit(ctx.take)
      ].filter(_ => _)
    } as ExpressionResult;
  }

  andOrExp(ctx: Filter.AndOrExpNodeContext) {
    let leftHandSide = this.visit(ctx.lhs);

    let opTokens = [] as IToken[];
    ctx.AndOp && opTokens.push(...ctx.AndOp);
    ctx.OrOp  && opTokens.push(...ctx.OrOp);

    let rightHandSide = [] as any[];

    if (ctx.rhs) {
      rightHandSide = ctx.rhs.map(_ => this.visit(_));
    }

    rightHandSide.unshift(leftHandSide);
    opTokens = opTokens.sort((a,b) => a.startOffset - b.startOffset);

    if (rightHandSide.length === 1) return rightHandSide.pop();
    let prev = rightHandSide.shift();

    opTokens.forEach(_ => {
      prev = { [`$${_.image}`] : [ prev, rightHandSide.shift() ] }
    });

    return prev;
  }

  orderBy(ctx: Filter.OrderByNodeContext, shouldAggregate: boolean = false) { 
    const ids = ctx.Identifier.sort((a,b) => a.startOffset - b.startOffset);
    const dirs = [...ctx?.Asc ?? [], ...ctx?.Desc ?? []].sort((a,b) => a.startOffset - b.startOffset);

    const items = {} as any;
    ids.forEach((_, i) => {
      items[_.image] = dirs[i].image === "asc" ? 1 : -1;
    });

    return { [shouldAggregate ? "$sort" : "$orderby"]: items };
  }

  take(ctx: Filter.TakeNodeContext) { 
    return { "$limit": Number(ctx.Integer[0].image) };
  }

  skip(ctx: Filter.SkipNodeContext) { 
    return { "$skip": Number(ctx.Integer[0].image) };
  }

  expression(ctx: Filter.ExpressionNodeContext) { 
    if (ctx.compareRule) return this.visit(ctx.compareRule);
    if (ctx.inExp)       return this.visit(ctx.inExp);
    if (ctx.notInExp)    return this.visit(ctx.notInExp);
    return this.visit(ctx.parentAndOrExp);
  }

  compareRule(ctx: Filter.CompareRuleNodeContext) { 
      const cmp = {} as any;
      let cmpOp = "";

      if (ctx.EqOp)     cmpOp = "$eq";
      if (ctx.NotEqOp)  cmpOp = "$ne";
      if (ctx.GtOp)     cmpOp = "$gt";
      if (ctx.GteOp)    cmpOp = "$gte";
      if (ctx.LtOp)     cmpOp = "$lt";
      if (ctx.LteOp)    cmpOp = "$lte";

      cmp[ctx.Identifier[0].image] = {
        [cmpOp]: ctx.Identifier[0].image === "id" ? new MongoDB.ObjectID(this.visit(ctx.atomicExp)) : this.visit(ctx.atomicExp)
      };

      return cmp;
  }

  inExp(ctx: Filter.InExpNodeContext) { 
    return {
      [ctx.Identifier[0].image] : {
        "$in": this.visit(ctx.array, ctx.Identifier[0].image === "id")
      }
    }
  }

  notInExp(ctx: Filter.NotInExpNodeContext) { 
    return {
      [ctx.Identifier[0].image] : {
        "$nin": this.visit(ctx.array)
      }
    }
  }

  parentAndOrExp(ctx: Filter.ParentAndOrExpNodeContext) { 
    return this.visit(ctx.andOrExp);
  }

  atomicExp(ctx: Filter.AtomicExpNodeContext) { 
    if (ctx.Float)   return Number(ctx.Float[0].image); 
    if (ctx.Integer) return Number(ctx.Integer[0].image); 
    if (ctx.String)  return ctx.String[0].image.slice(1, ctx.String[0].image.length - 1); 
    if (ctx.dateExp) return this.visit(ctx.dateExp); 
  }

  dateExp(ctx: Filter.DateExpNodeContext) { 
    return Date.parse(ctx.String[0].image.slice(1, ctx.String[0].image.length - 1));
  }

  array(ctx: Filter.ArrayNodeContext, convertToId: boolean = false) { 
    const res = ctx.atomicExp.map(_ => this.visit(_));
    return convertToId ? res.map(_ => new MongoDB.ObjectID(_)) : res;
  }          

}

Conclusion

We have seen how to implement our visitor to provide something that can be processed by MongoDB. Following this, we can imagine to implement the same for SQLite or MySql (MariaDB)...

Enjoy !

Writing a filtering expression parser with Chevrotain parsing library

Ampla Network — Wed, 07 Oct 2020 17:22:59 +0000

This article provides an overview of how to use the Chevrotain library to write a small expression parser.

This article will not go into much detail about what a parser is, and is intended for an intermediate level audience.

If you want more detail, let me know in the comments and I will modify this article accordingly.

A little bit of context

I'm working on a Headless CMS project, which is based on a JSON data schema and generates a GraphQL API. To facilitate a little bit the filtering via the API, I need to be able to manage it via a simple custom grammar.

I usually use ANTLR, which is probably one of the best parser generators.

But this time, I want to test something new, and after some research, I came across a library called Chevrotain

Chevrotain is not a parser generator, instead it takes advantage of Javascript directly to describe Lexer and Grammar with the code.

The target

The goal is to be able to filter the elements of our query using a very simple language that must meet the following criteria:

Filter fields via matching operators

age lt 20
fruit not in ['apple', 'banana']
email eq 'xxxx@xxxx.xxx'

Use multiple criteria via the AND and OR operators

group eq 'admin' and active eq 1

Prioritize operators with parenthesis

(amount lte 100 and date gt dt{'2020-01-01'}) or byPass eq 1

Order on fields

order by age desc name asc

Skip some records

skip 5

Take a limited number of records

take 2

The Lexer

First, we need to write a lexer in order to split each word into tokens. Tokens are used in Parsing rules to create the target AST. An AST or Abstract Synax Tree is the final result of the parsing state.

A token can represent a static keyword, just like any dynamic value, such as a number, a string, or an identifier like variables, method names, etc.

So we need to defined all Tokens first to tell Chevrotain how to understand the input text and prepare it to be parsed.

CreateToken

With Chevrotain, token creation is relatively simple.
First we import the createToken function

 const createToken = chevrotain.createToken;

Then we define the tokens

const Identifier = createToken({name: "Identifier" , pattern: /[a-zA-Z_][\w\d_]*/});

As you can see, to define a token, you specify a name, and a pattern. The name is the unique identifier of the token, and the pattern is a regular expression used by the scanner to recognize the token.

It is also possible to remove recognition ambiguities by specifying an alternative that should be used instead for a longer token.

For example, an Integer and a Float cause recognition ambiguity. A Float will be interpreted as an Integer by default.

This can be handled as follows:

  const Float   = createToken({name: "Float"   , pattern: /\d+\.\d+/});
  const Integer = createToken({name: "Integer" , pattern: /\d+/, longer_alt: Float});

Now an Integer will be recognized as an Integer only if it is not a Float.

After defining all your tokens, you must now group them together to create an instance of the lexer.

const allTokens = [OrderBy,WhiteSpace,Asc, Desc,Take, Skip, NotInOp,InOp,AndOp,OrOp,GteOp,GtOp,LteOp,LtOp,NotEqOp,EqOp,LParen, RParen, LBraket, RBraket, Comma, Float, Integer, Dt,  Identifier, LCurly, RCurly, String];
const FilterLexer = new Lexer(allTokens);

The Grammar

Let's see how the grammar should be

At the top level, we have the expressions rule. It is composed by one andOrExp rule, optionally followed by an orderBy rule, a skip rule and a take rule.

What are grammar rules?
When working with parsers, it is good to understand a few prerequisites.

To write a grammar, you will need to use 2 types of information. The source to be parsed will be decomposed into nodes.

The nodes can be classified in 2 categories, terminal and non-terminal nodes.

In the image above, you can see the non-terminal nodes, which are in squared boxes, and the terminal ones in rounded boxes.

A terminal node is a final one, it is a value or a keyword, or any token you have defined.

A non terminal node is a rule, in wich you can continue to parse.

In summary, when we have to process the LBraket node, we do not go further, this node has the value [.

On the other hand, for the next node atomicExp , we will continue the processing before being able to evaluate its final value.

As you can see, we cannot determine the expression value, which can be of several types. That's why it is a non terminal node.

From theory to implementation.

Let's start by analyzing the rule we want to write.

The first token is of type andOrExp, and is mandatory.
The three others are all optional but processed sequentially.

Let's start by creating the Rule itself.

const $ = this;

// This is an empty rule
$.RULE("expressions", () => {
});

Now we can add the first rule to consume as a subrule of the current one. This will tell Chevrotain how to understand the rule.

$.RULE("expressions", () => {
  $.SUBRULE($.andOrExp);
});

Handle optional rule

Now we need to set the first optional rule.

$.RULE("expressions", () => {
  $.SUBRULE($.andOrExp);
  $.OPTION(()  => { $.SUBRULE($.orderBy); })
});

And the others

$.RULE("expressions", () => { 
  $.SUBRULE($.andOrExp);
  $.OPTION(()  => { $.SUBRULE($.orderBy); })
  $.OPTION2(() => { $.SUBRULE($.skip); })
  $.OPTION3(() => { $.SUBRULE($.take); })
});

Yes we did it. We have just declared the Rule :-)

Handle alternative rules

Let's see the andOrExp rule.

This rule is an interesting one because it is structurally complex without being complicated. And that's the point, keeping things simple in order to build something complex.

Expression is a mandatory rule. AndOP and OrOp are both optional and alternatives to each other, and everything after the first rule can be used several times.

So let's see how to handle that.

$.RULE("andOrExp", () => {
  $.SUBRULE($.expression, { LABEL: "lhs" });
});

Here we can use a subrule to start with. Note the use of the LABEL option. This will be necessary for the implementation of the visitor.

Then we can declare Alternatives by using the OR function. AndOp and OrOp are Tokens not rules, so we use the CONSUME method instead of SUBRULE.

$.OR([
  {ALT: () => { $.CONSUME(AndOp); }},  
  {ALT: () => { $.CONSUME(OrOp); }}
]);

This sequence can be declared multiple times, so we need to encapsulate it as follows.

$.MANY(() => {
  $.OR([
    {ALT: () => { $.CONSUME(AndOp); }},
    {ALT: () => { $.CONSUME(OrOp); }}
  ]);        
});

Abd now the full rule

$.RULE("andOrExp", () => {
  $.SUBRULE($.expression, { LABEL: "lhs" });
  $.MANY(() => {
    $.OR([
      {ALT: () => { $.CONSUME(AndOp); }},
      {ALT: () => { $.CONSUME(OrOp); }}
    ]);        
    $.SUBRULE2($.expression,{LABEL: "rhs" });
  });
})

Left recursive approach versus chained approach

As I had to mention earlier, I'm more used to use ANTLR, which has the particularity of being Left Recursive.

So the naive approach to add the andOrExp with parenthesis could have been like this :

andOrExp:
  expression ((AndOp | OrOp) expression)* |
  LPren andOrExp RParen

But Chevrotain is not Left recursive. So we have to adapt the grammar in 3 steps.

The andOrExp
Then the parenthesis version
Then the tricky part is to add the Parenthesis version to the expression rule

Now we had acheive the same result 😄

And the sample

(billAmount lte 200 and billAmount gte 100) or startDate eq dt{'2020-01-01'}
order by name asc age desc
skip 100 take 20

Will be converted in a relatively indigestible syntax tree...

Conclusion

In the next article we will see how to define the corresponding Visitor to explore and transform the AST into something more usefull, and also how to implement a derived visitor to generate MongoDB filtering from this parser.

If you want to play with this sample, open the Chevrotain playgroung

Then past the source

(function FilterCst() {
  "use strict";
  /**
   * An Example of implementing a Calculator with separated grammar and semantics (actions).
   * This separation makes it easier to maintain the grammar and reuse it in different use cases.
   *
   * This is accomplished by using the automatic CST (Concrete Syntax Tree) output capabilities
   * of chevrotain.
   *
   * See farther details here:
   * https://github.com/SAP/chevrotain/blob/master/docs/concrete_syntax_tree.md
   */
  const createToken  = chevrotain.createToken  ;
  const tokenMatcher = chevrotain.tokenMatcher ;
  const Lexer        = chevrotain.Lexer        ;
  const CstParser    = chevrotain.CstParser    ;

  const Identifier = createToken({name: "Identifier" , pattern: /[a-zA-Z_][\w\d_]*/});
  const LParen     = createToken({name: "LParen"     , pattern: /\(/});
  const RParen     = createToken({name: "RParen"     , pattern: /\)/});
  const Float      = createToken({name: "Float"      , pattern: /\d+\.\d+/});
  const Integer    = createToken({name: "Integer"    , pattern: /\d+/, longer_alt: Float});
  const String     = createToken({name: "String"     , pattern: /'.*?'/});
  const Comma      = createToken({name: "Comma"      , pattern: /,/});
  const LCurly     = createToken({name: "LCurly"     , pattern: /\{/});
  const RCurly     = createToken({name: "RCurly"     , pattern: /\}/});  
  const LBraket    = createToken({name: "LBraket"    , pattern: /\[/});
  const RBraket    = createToken({name: "RBraket"    , pattern: /\]/});  
  const Dt       = createToken({name: "Dt"       , pattern: /dt/, longer_alt: Identifier});

  const EqOp    = createToken({name: "EqOp"    , pattern: /eq/, longer_alt: Identifier});
  const NotEqOp = createToken({name: "NotEqOp" , pattern: /!eq/, longer_alt: Identifier});
  const LtOp    = createToken({name: "LtOp"    , pattern: /lt/, longer_alt: Identifier});
  const LteOp   = createToken({name: "LteOp"   , pattern: /lte/, longer_alt: Identifier});
  const GtOp    = createToken({name: "GtOp"    , pattern: /gt/, longer_alt: Identifier});
  const GteOp   = createToken({name: "GteOp"   , pattern: /gte/, longer_alt: Identifier});

  const AndOp   = createToken({name: "AndOp"   , pattern: /and/, longer_alt: Identifier});
  const OrOp    = createToken({name: "OrOp"   , pattern: /or/, longer_alt: Identifier});

  const InOp   = createToken({name: "InOp"   , pattern: /in/, longer_alt: Identifier});
  const NotInOp    = createToken({name: "NotInOp"   , pattern: /!in/, longer_alt: Identifier});

  const OrderBy    = createToken({name: "OrderBy"   , pattern: /order\s+by/, longer_alt: Identifier});
    const Asc    = createToken({name: "Asc"   , pattern: /asc/, longer_alt: Identifier});
  const Desc    = createToken({name: "Desc"   , pattern: /desc/, longer_alt: Identifier});
  const Take    = createToken({name: "Take"   , pattern: /take/, longer_alt: Identifier});
  const Skip    = createToken({name: "Skip"   , pattern: /skip/, longer_alt: Identifier});


  // marking WhiteSpace as 'SKIPPED' makes the lexer skip it.
  const WhiteSpace = createToken({
    name: "WhiteSpace",
    pattern: /\s+/,
    group: Lexer.SKIPPED
  });


  const allTokens = [OrderBy,WhiteSpace,Asc, Desc,Take, Skip, NotInOp,InOp,AndOp,OrOp,GteOp,GtOp,LteOp,LtOp,NotEqOp,EqOp,LParen, RParen, LBraket, RBraket, Comma, Float, Integer, Dt,  Identifier, LCurly, RCurly, String];
  const FilterLexer = new Lexer(allTokens);

  // ----------------- parser -----------------
  // Note that this is a Pure grammar, it only describes the grammar
  // Not any actions (semantics) to perform during parsing.
  class FilterPure extends CstParser {
    constructor() {
      super(allTokens);

      const $ = this;

      $.RULE("expressions", () => { 
        $.SUBRULE($.andOrExp);
        $.OPTION(()  => { $.SUBRULE($.orderBy); })
        $.OPTION2(() => { $.SUBRULE($.skip); })
        $.OPTION3(() => { $.SUBRULE($.take); })
      });

      $.RULE("expression", () => {
        $.OR([
            { ALT:() => { $.SUBRULE($.compareRule) }},
            { ALT:() => { $.SUBRULE($.inExp) }},
            { ALT:() => { $.SUBRULE($.notInExp) }},
            { ALT:() => { $.SUBRULE($.parentAndOrExp)}}
        ])
      })

      $.RULE("take", () => {
        $.CONSUME(Take);
        $.CONSUME(Integer);
      })

       $.RULE("skip", () => {
        $.CONSUME(Skip);
        $.CONSUME(Integer);
      })

      $.RULE("orderBy", () => {
        $.CONSUME(OrderBy);
        $.AT_LEAST_ONE(() => {
          $.CONSUME(Identifier);
          $.OR([
            {ALT: () => {$.CONSUME(Asc)}},          
            {ALT: () => {$.CONSUME(Desc)}},
          ]);
        })
      })

      $.RULE('array', () => {
        $.CONSUME(LBraket);
        $.AT_LEAST_ONE_SEP({
         SEP: Comma,
         DEF: () => {
            $.SUBRULE($.atomicExp);
         }
        })
        $.CONSUME(RBraket);
      })

      $.RULE("inExp", () => {
        $.CONSUME(Identifier);
        $.CONSUME(InOp);
        $.SUBRULE($.array);
      })

       $.RULE("notInExp", () => {
        $.CONSUME(Identifier);
        $.CONSUME(NotInOp);
        $.SUBRULE($.array);
      })

      $.RULE("andOrExp", () => {
        $.SUBRULE($.expression, { LABEL: "lhs" });
          $.MANY(() => {
           $.OR([
            {ALT: () => { $.CONSUME(AndOp); }},
            {ALT: () => { $.CONSUME(OrOp); }}
          ]);        
          $.SUBRULE2($.expression,{LABEL: "rhs" });
        });
      })

      $.RULE("parentAndOrExp", () => {
        $.CONSUME(LParen);
        $.SUBRULE($.andOrExp);
        $.CONSUME(RParen);
      })

      $.RULE("compareRule", () => {
        $.CONSUME(Identifier);
        $.OR([
          { ALT:() => { $.CONSUME(EqOp) }},
          { ALT:() => { $.CONSUME(NotEqOp) }},
          { ALT:() => { $.CONSUME(GtOp) }},
          { ALT:() => { $.CONSUME(GteOp) }},
          { ALT:() => { $.CONSUME(LtOp) }},
          { ALT:() => { $.CONSUME(LteOp) }},
        ]);
        $.SUBRULE($.atomicExp);
      });

      $.RULE("atomicExp", () => {
        $.OR([
          { ALT:() => { $.CONSUME(Integer) }},
          { ALT:() => { $.CONSUME(Float) }},
          { ALT:() => { $.CONSUME(String) }},
          { ALT:() => { $.SUBRULE($.dateExp) }},
        ]);
      });

      $.RULE("dateExp", () => {
        $.CONSUME(Dt);
        $.CONSUME(LCurly);
        $.CONSUME(String);
        $.CONSUME(RCurly);
      });



      // very important to call this after all the rules have been defined.
      // otherwise the parser may not work correctly as it will lack information
      // derived during the self analysis phase.
      this.performSelfAnalysis();
    }
  }

  // wrapping it all together
  // reuse the same parser instance.
  const parser = new FilterPure([]);


  // ----------------- Interpreter -----------------
  const BaseCstVisitor = parser.getBaseCstVisitorConstructor()

  class FilterInterpreter extends BaseCstVisitor {

    constructor() {
      super()
      // This helper will detect any missing or redundant methods on this visitor
      this.validateVisitor()
    }

    expression(ctx) {
      return this.visit(ctx.additionExpression)
    }

    atomicExp(ctx) {
      if("dateExp" in ctx) {
        return this.visit(ctx.dateExp); 
      }

      if ("Integer" in ctx) {
        return Number(ctx.Integer[0].image); 
      }

      if ("Float" in ctx) {
        return Number(ctx.Float[0].image); 
      }

      return ctx.String[0].image.slice(1, ctx.String[0].image.length - 1)
    }

    dateExp(ctx) {
        return new Date(ctx.String[0].image.slice(1, ctx.String[0].image.length - 1));
    }


    compareRule(ctx) {

    }

    expressions(ctx) {
        return ctx
    }

    andOrExp(ctx) {}
    array(ctx) {}
    inExp(ctx) {}
    notInExp(ctx){}
    parentExpression(ctx){}
    parentAndOrExpression(ctx){}
    parentAndOrExp(ctx){}
    orderBy(ctx){}
    take(ctx){}
    skip(ctx){}
  }

  // for the playground to work the returned object must contain these fields
  return {
    lexer: FilterLexer,
    parser: FilterPure,
    visitor: FilterInterpreter,
    defaultRule: "expressions"
  };
}())

S-FLUX V3, The power of Flux pattern with the ease of redux

Ampla Network — Tue, 22 Sep 2020 03:30:16 +0000

S-Flux is a unidirectional dataflow implementation, which allows synchronizing the actions of an application to ensure visual consistency of data, and keep simplicity in their management, even if the amount of information grows over time.

The FLux pattern is a representation of a unidirectional data flow. This way of representing the life cycle of an information will ensure that for each cycle, all components using the data will be at the same level of information.

The Dispatcher will stack the Actions, then redistribute them to the Stores, which in turn will process or not the information. Once the information has been processed, each Store will notify the View(s) that more up-to-date data is available. The View(s) will then update their display accordingly.

An Action can be triggered by a View, a Store, or any other source, such as a server. In all cases, each Action will be processed sequentially by the Dispatcher.

S-Flux V3 provides a very streamlined approach, so you don't need to declare either the dispatcher or the stores. Only a definition of store actions is necessary, and Shadow-Flux generates on-the-fly all the helper methods, as well as all the subscription methods, with full typescript intellisense.

What you do in V3 :

What it does internally :

Sample definition

// State type
type CounterState = {
  counter: number;
};

// Events
const _events = Flux.withEvents({
  ForceCounter: ""
})

type TThis = TBaseStore&lt;CounterState>;

// Actions
const _actions = {
  async increase(this: TThis) {
    this.nextState({
      counter: this.getState().counter + 1
    });
  },
  async decrease(this: TThis) {
    this.nextState({
      counter: this.getState().counter - 1
    });
  },
  async setCounter(this: TThis, payload: { counter: number }) {
    this.nextState({
      counter: payload.counter
    });
    return _events.ForceCounter; // Return specific event 
  }
}

export default Flux.registerStore({
  id: "Counter",
  async init(): CounterState { 
    return { counter: 0 };
  },
  actions : _actions,
  events  : _events
});

Sample result

import CounterStore from "path-to-store";

// CounterStore will be of type :
{
  getState() : CounterState ;
  id         : string       ;
  events: {
    All          : string ;
    ForceCounter : string ;
  },
  subscribeTo: {
    All          : ((state: CounterState) => void) => { off: () => void };
    ForceCounter : ((state: CounterState) => void) => { off: () => void };
  },
  actions: {
    increase   : () => void                             ;
    decrease   : () => void                             ;
    setCounter : (payload: { counter: number }) => void ;
  }
}

The CounterStore object will provide all actions methods generated from the store actions, and all subscription method for each events.

Overview

In the original Flux pattern, you need to follow these step :

Instanciate a dispatcher
Create one or more store(s)
Register all stores with the dispatcher
Instanciate a subscriber helper
Creates Action-Helper functions to manage Action creations
Subscribe Store events

But now, with S-Flux v3, to get the same result you only need to :

Declare one or more store(s)
Use the store helper to access all available actions
Use the store helper to subscribe all available events

Stores

In S-Flux, stores are used to store information. A store is responsible for a precise perimeter, called scope. This perimeter will include its data model, as well as the business or technical rules it needs.

Basically, a store is a class that extends the BaseStore class. This class must describe the data model, as well as the rules applicable to this data model. To do this, the data model will be stored in a State field, and the rules described by methods that can be triggered externally by the application.

A store should only manage its own data. However, it can rely on data from another store, if necessary, during the execution of one of its rules, to fill its data.

In order to simplify the declaration, as well as store usage, S-Flux has introduced a simplification on top of the original Pattern, allowing to hide a certain complexity level, and reducing at the same time some tedious declarations necessary to retreive data. Be aware that the simplification is a helper, but underneath, the original pattern is kept and used by S-Flux.

Actions

Actions are methods that aim to modify the state of stored data. An action will receive a payload providing the necessary information to modify the state accordingly.

The method prototype can be describe as follow :

type TActionReturn = void | null | string | string[];

async actionName(payload?: any) => TActionReturn;
async actionName(payload?: any, For?: (...ids: string[]) => Promise<void>) => TActionReturn;

The payload is an object containing all the information useful for the action to update its State.

The For method can be used to ensure any other store will finish to process the payload before the current one will continue.Ex:

async getTravelDetail(payload: TTravel, For: TAwaitFor) {
  await For(priceStore.id);
  this.nextState({
    ...payload,De
    price: priceStore.getState().prices[payload.travelId]
  });
}

The nexstate method is used to update the current state. By default, the state is replaced by the new value. If you want to partially update the state, you can set the flag mergeToPreviousState to true. Be aware that the update is a shallow merge, not a deep merge.

// Since we declare the action outside the Store class, 
// we can set the this parameter to set the correct store context 
async setCounterValue(this: TBaseStore&lt;CounterState>, payload: { value: number }) {
  this.nextState(payload);
}

The return value will determine the event(s) that will be triggered at the end of processing. By default, for any change of state, an event of type "All" will be triggered. It is possible to trigger specific events in case of a partial change of state. The different combinations are described below.

Value	Effect
`undefined`	Global change, we notify all subscribers to any change
`null`	No change, this will prevent any event to be triggered
`string`	The selected event will be triggered
`string[]`	The selected events will be triggered

The wrapper returned by the "registerStore" method will create as many Helper methods as you have defined an action. But in addition to that, the methods also take care of forward the Payload directly to the dispatcher.

// So if you have defined this
const _actions = {
  async increase(this: TThis) {
   ...
  },
  async decrease(this: TThis) {
    ...
  },
  async setCounter(this: TThis, payload: { counter: number }) {
    ...
  }
}

// You get that in return
{
  ...
  actions: {
      increase   : () => void                             ;
      decrease   : () => void                             ;
      setCounter : (payload: { counter: number }) => void ;
    }
    ...
}

Events

Events are used to notify subscribers that a change in the state of a store they subscribe to has been updated. By default, it is not necessary to create events, because a global event exists, named "All". It is propagated for any change in a store, unless otherwise specified.

For each event to which you wish to subscribe, you must register through the Dispatcher, specifying the Store ID and the name of the event. The wrapper returned by the "registerStore" method will create as many Helper methods as you have defined an event. These methods will allow a direct subscription, without the constraint of passing the store identifier and the name of the event, and this, without even using the Dispatcher.

Events are described as key/value pairs, carrying the name of the event.

const events = {
  updated  : "updated",
  newEntry : "newEntry"
};

As it is a bit redundant to specify for each key its value, knowing that in most cases the value will probably be identical to the key, it is possible to use the withEvents function to simplify this syntax.

const events = withEvents({
  updated  : "",
  newEntry : ""
});

Each wrapped method will be generated followign each event declared, plus the All events that will be added automatically.

subscribeTo: {
    All      : ((state: TheStoreState) => void) => { off: () => void };
    updated  : ((state: TheStoreState) => void) => { off: () => void };
    newEntry : ((state: TheStoreState) => void) => { off: () => void };
  }

The returned value of a subscription is an object exposing an off method. You can call it to unsubscribe at any time.

An example with React Hook

React.useEffect(() => {
  const sub = counterStore.subscribeTo.All(_ => setState(_));
  return () => sub.off();
}, []);

Using the store helper

The usage of a helper is extremely simple. Just import the helper.

import BlogStore from "./stores/Blog";

Let's assume that the Store owns the following actions :

getLatestArticles
getArticleDetail
postComment

And provide the following events:

OnGetList
OnGetDetail
OnPostComment

The Helper will automatically provide an action method group and a subscription method group.

To call the actions, use the methods in actions, and for events, the one in subscribeTo.

// You can subscribe globally to any change, and update your full state
React.useEffect(() => {
  const sub = BlogStore.subscribeTo.All(state => setState(state));
  // Unsubscribe when unmount 
  return () => sub.off();
})

// You can subscribe specifically to one partial event
React.useEffect(() => {
  const sub = BlogStore.subscribeTo.OnGetList(state => setLatest(state.latest));
  // Unsubscribe when unmount 
  return () => sub.off();
})

Now to call an action, just use the action group methods

// You can call easily any action
function Detail() {
  return <>
    <BlogDetail 
      body      = {state.selectedBlog} 
      onComment = { 
        _ => BlogStore.actions.postComment({
          id      : _.id,
          content : _.content
        });
      }/>
  </>;
}

Conclusion

I hope you liked this post.

If you want to know more, here are all the links

Website

Npm

Github

Enjoy 😄

Forem: Ampla Network

LLMs are NOT the product

Understanding the Limits of System Prompts

Ripple, React shared state with less code...

Introducing Ripple, Your Go-To State Management Library for React

Riding the Wave with Ripple

The Ripple Lake

Ripple in Action Inside Components

A Deep Dive into Ripples

Ripple in the Component

Destructuring the Ripple

Behind the Scenes

Ripple Updates from a Component

Example: Using the Updater

Updater Parameters

Parent-Child Approach

External Updates of Ripples

Example: Using External Updates

Lake definition

Service

Component

The Updater Service

Conclusion

Going deeper with typescript advanced types

Step 1

Step 2

Step 3

Step 4

Step 5

Step 6

Final step

Making a voxel engine

Introduction

So what is a voxel ?

So displaying cubes is not a big deal, isn't it ?

So okay, but where to start then?

World structure

The world

The region

A chunk

Block definition

Chunk definition

1D array or the power of flattening everything

React 02 - Beginner : Understanding data, Props vs State.

Props

State

React 01 - Beginner : React function... Or How beauty comes from simplicity

Why React uses, in my opinion, the perfect rendering engine

React 00 - Beginner : React + Typescript from scratch

Setup environment

A little bit of configuration

You first TSX file

It's building time

Browsing the result

The power of advanced typing in Typescript

The inputs

The target output

The problem

Preparing the types

Generating the types dynamically

Extracting the field type.

Creating the target type

Conditional typing

Conclusion

Expression Parser : Part 2 - Define and implement a visitor for MongoDB

Defining the Visitor

andOrExp example

Implementing the visitor for MongoDB

The global rule definition

A more complex one, the andOrExp

expression nodes

Conclusion

Writing a filtering expression parser with Chevrotain parsing library

A little bit of context

The target

The Lexer

CreateToken

The Grammar

From theory to implementation.

Handle optional rule

You first `TSX` file