Forem: Morris Janatzek

New version of SapphireDb with NodeJs support

Morris Janatzek — Tue, 21 Jan 2020 17:43:56 +0000

SapphireDb / sapphiredb-js

Client for SapphireDb, a self-hosted, easy to use realtime database for Asp.Net Core and EF Core

SapphireDb now has got a new client implementation that is compatible with any JS client and also compatible with NodeJs. The npm-package is called sapphiredb.

Story behind

A few days ago I posted an article on HackerNews to advertise a framework I created. The name of it is SapphireDb, it is an extension for Asp.Net Core and EF Core and enables you to easily create self hosted applications with realtime data synchronization. It should serve as alternative to firebase realtime database/firestore.

I got plenty of feedback including a few recommendations like moving the code base of the client implementation to a separate project. I totally agree to that point and decided to do so. At that moment SapphireDb only had a client implementation for Angular.

The process was not that complicated but I had to change all references to Angular specific things to a more generic API. I decided to use axios instead of Angulars HttpClient and it works fine. Also using direct inject of dependency instances instead of Angulars Dependency injection was pretty easy. I only decided to keep ng-packagr as build tool for the client library.

NodeJs example

npm install sapphiredb rxjs axios -S

var sapphiredb = require('sapphiredb');

var db = new sapphiredb.SapphireDb({
    serverBaseUrl: 'localhost:5000',
    useSsl: false,
    apiKey: 'webapp',
    apiSecret: 'pw1234'
});

db.collection('demo.entries').values().subscribe(function (values) {
    console.log(values);
});

Changes for ng-sapphiredb

Because I moved the complete code base out of the project because I don't want to maintain the same code in two places you have to install sapphiredb along with ng-sapphiredb when using it.

npm install ng-sapphiredb sapphiredb axios -S

Check out the documentation to learn more: https://sapphire-db.com/

SapphireDb / sapphiredb-js

Client for SapphireDb, a self-hosted, easy to use realtime database for Asp.Net Core and EF Core

How to synchronize data in realtime with Asp.Net Core and Angular

Morris Janatzek — Tue, 10 Dec 2019 17:02:19 +0000

To easily create a realtime application I'm using SapphireDb. Check out the documentation to learn more: https://sapphire-db.com/

SapphireDb / SapphireDb

SapphireDb Server, a self-hosted, easy to use realtime database for Asp.Net Core and EF Core

Intro

Around one year ago I started developing a new project. The project initially consisted of an asp.net core application as a backend server and an angular client application as a UI. One of the key requirements was the data synchronization in realtime.

I already knew and liked firebase and had the wish to be able to use it in the project. Sadly that was not possible because another requirement was that the application should have a asp.net core backend and can be self-hosted.

I made some research and sadly did not find any other solution that would make the development easier with the required tech stack.

I then decided to start developing my own realtime database called SapphireDb. Here are the most important the features the project now has:

🔧 Dead simple configuration
🌠 Blazing fast development
📡 Modern technologies
💻 Self hosted
💾 Easy CRUD operations
🔑 Authentication/Authorization included
✔️ Database support
🔌 Actions
🌐 NLB support

In this short article I want to explain you how to quickly create your own application with realtime data synchronization using SapphireDb for Asp.Net Core and an Angular application.

Setup

Server

First you have to create a server application using Asp.Net Core. Using the empty project template totally does the job.

Install

The next step is to install the package from nuget:

PM> Install-Package SapphireDb

Create DbContext

Now create a DbContext that contains the objects you want to store in the database.

If you dont know how to work with Entity Framework Core check out this documentation for more details: EF Core Docs

The context should look something like this:

// Important: Use SapphireDbContext instead of DbContext
public class MyDbContext : SapphireDbContext
{
  //Add SapphireDatabaseNotifier for DI
  public MyDbContext(DbContextOptions<MyDbContext> options, SapphireDatabaseNotifier notifier) : base(options, notifier)
  {

  }

  public DbSet<User> Users { get; set; }

  public DbSet<Test> Tests { get; set; }
}

Register services and update pipeline

The last step is to modify your service registrations and add SapphireDb to your request pipeline. Modify your Startup.cs file like this:

public class Startup
{
  public void ConfigureServices(IServiceCollection services)
  {
    //Register services
    services.AddSapphireDb(new SapphireDbOptions())
      .AddContext<MyDbContext>(cfg => cfg.UseSomeDatabase());
  }

  public void Configure(IApplicationBuilder app)
  {
    //Add Middleware
    app.UseSapphireDb();
  }
}

Try launching the application

The server application should now work.

Client

You now can create your client application. You first have to create a Angular App or use an existing.

Install

First install the project using:

npm install -S ng-sapphiredb

Import SapphireDbModule

Then you have to import the SapphireDbModule in you app.module.ts to make the services available in the whole application:

@NgModule({
  imports: [
    SapphireDbModule
  ],
  providers: [
    // This part is optional and only required if you want to override some of the settings
    {
      provide: SAPPHIRE_DB_OPTIONS,
      useValue: {
        serverBaseUrl: environment.serverBaseUrl,
        apiSecret: 'pw1234',
        apiKey: 'webapp',
        connectionType: 'websocket'
      }
    }
  ],
})
export class AppModule {}

Start application

The cient application should now work. Start it and navigate to the url in your browser.

Query data

You are now able to query data. You can use this code in your angular client to get the data from your database. Changes on server side are not required.

export class DemoComponent implements OnInit {
  values$: Observable<Entry[]>;

  constructor(private db: SapphireDb) { }

  ngOnInit() {
    this.values$ = this.db.collection<Entry>('nameofdbset').values();
  }
}

To display the content of your collection as json use this in your html:

{{ values$ | async | json }}

This project uses RxJs. If you don't know what an observable is and how to work with it check out this url: RxJs Documentation

When you display the content of the array in the component you can already see the entries and if you change something it should update immediatly.

Update data

To update data you have to make the class of the model you want to update updatable. The class should look like this:

[Updatable]
public class DemoEntry
{
  [Key]
  public int Id { get; set; }

  public string Content { get; set; }
}

You can then update the content using this code on client side:

export class DemoComponent implements OnInit {
  collection: DefaultCollection<Entry>;
  constructor(private db: SapphireDb) { }

  ngOnInit() {
    this.collection = this.db.collection<Entry>('entries', 'demo');
  }

  updateValue(value: Entry) {
    this.collection.update({
      ...value,
      content: v
    });
  }
}

Finish

You are now familiar to the main features of SapphireDb. If you want to learn more check out the documentation and the project on Github.

Documentation

SapphireDb / SapphireDb

SapphireDb Server, a self-hosted, easy to use realtime database for Asp.Net Core and EF Core

SapphireDb - Server for Asp.Net Core

SapphireDb is a self-hosted, easy to use realtime database for Asp.Net Core and EF Core.

It creates a generic API you can easily use with different clients to effortlessly create applications with realtime data synchronization SapphireDb should serve as a self hosted alternative to firebase realtime database and firestore on top of .Net.

Check out the documentation for more details: Documentation

Features

🔧 Dead simple configuration
📡 Broad technology support
💻 Self hosted
📱 Offline support
💾 Easy to use CRUD operations
⚡ Model validation
✔️ Database support
📂 Supports joins/includes
➿ Complex server evaluated queries
🔌 Actions
🔑 Authorization included
✉️ Messaging
🌐 Scalable

Learn more

Installation

Install package

To install the package execute the following command in your package manager console

PM> Install-Package SapphireDb

You can also install the extension using Nuget package manager. The project can be found here: https://www.nuget.org/packages/SapphireDb/

Configure

…

View on GitHub

SapphireDb - Development of a self-hosted alternative to firebase realtime database for asp.net core

Morris Janatzek — Mon, 09 Dec 2019 15:12:11 +0000

If you're interested in the final result, you can find it here: https://sapphire-db.com/

SapphireDb / SapphireDb

SapphireDb Server, a self-hosted, easy to use realtime database for Asp.Net Core and EF Core

Intro

I made some research and sadly did not find any other solution that would make the development easier with the required tech stack.

The idea

After some days of frustration I made the decision to start developing my own realtime database. After some thinking I wrote down my most important requirements and features the database should have.

Extend an existing asp.net core application that uses entity framework

The main idea was, that the realtime database should extend an existing application. The API should be as easy as possible and I wanted to make as few changes to an existing application as possible. That would make the migration of other existing applications much easier.

Work with every database supported by ef core

The second idea was to build the realtime database on top of entity framework core. This would have the advantage that it could support any database, supported by entity framework core.

No change in the usage of ef core on server side

I also did not want to make big changes in the existing code on server side. This means changes in the API for querying and managing data on server side directly, for example if you have a procedure that has no realtion to interaction with the client application. The realtime database should extend the existing code and manage synchronization in background automatically.

Easy to use CRUD-operations and API

The most important feature for me was, that the API should be as easy as possible. I wanted to query data from the server on client side with a single line of code, like this:

this.db.collection('data').values();

No SignalR

I already had some bad experiences using SignalR and decided that the project should not depend on it.

Actions (Alternative to methods in controllers)

Because I'm already going to implement a connection and a new API on client side I thought, it would also be very helpful, to have something like custom actions you can define and easily call through the already existing connection.

Development

With the main features in mind I started creating first drafts and making tests.

I dont want to go to much into details but want to give a short overview of the main technical points related to the synchronization of data.

If you are interested I will write more articles explaining more of the concepts and technology used.

You can also check out the sources to learn more:

SapphireDb / SapphireDb

SapphireDb Server, a self-hosted, easy to use realtime database for Asp.Net Core and EF Core

Notification about changes

The first thing I wanted to achieve was an easy way to get notified when something in the database changes.

I thought about triggers on database level but they would not be flexible enough and would not work with some database systems like the InMemory-database.

The solution I came up with, was to create a custom DbContext that hooks into the SaveChanges-method and triggers a notifier. The cool thing about that is, that EF Core already comes with a ChangeTracker that provides you all changes that affect the next change operation. The code I came up with looks like this:

public override int SaveChanges(bool acceptAllChangesOnSuccess)
{
    List<ChangeResponse> changes = GetChanges();
    int result = base.SaveChanges(acceptAllChangesOnSuccess);
    notifier.HandleChanges(changes, GetType());
    return result;
}

private List<ChangeResponse> GetChanges()
{
    return ChangeTracker.Entries()
        .Where(e => e.State == EntityState.Added || e.State == EntityState.Deleted || e.State == EntityState.Modified)
        .Select(e => new ChangeResponse(e, this)).ToList();
}

With this small piece of code I was already able to achieve one of the key requirements. You dont have to change anything in the code you are already using, except from changing all DbContexts to the new DbContext with the custom SaveChanges method.

Communication

The next challenge to solve was the communcation between client and server.

The final communication itself consists of two simple components. A command and a response. Every action on server side has a command. The client sends a command with a unique id to the server. The server will handle this command and on completion it will respond with a response containing the same unique id as the command. The client then can handle the response and associate it with the sent command.

For transfer I first implemented Websockets for they already have a Bi-Directional data transport unlike other technologies like ServerSentEvents.

I then created a general object ConnectionBase that includes information like the subscriptions a connection has and how to talk to the client. That made it possible to implement the support for SSE and Polling afterwards.

The basic structure of the ConnectionBase looks like this:

public abstract class ConnectionBase : IDisposable
{
    public void Init(HttpContext context)
    {
        Id = Guid.NewGuid();
        Subscriptions = new List<CollectionSubscription>();
        HttpContext = context;
    }

    public Guid Id { get; set; }

    public HttpContext HttpContext { get; set; }

    public List<CollectionSubscription> Subscriptions { get; set; }

    public abstract Task Send(object message);
}

Every connection implementation has to implement the Send-method to send data to the client.

The implementation for Websockets for exampl looks like this:

public class WebsocketConnection : ConnectionBase, IDisposable
{
    public WebsocketConnection(WebSocket webSocket, HttpContext context)
    {
        Websocket = webSocket;
        Init(context);
    }

    public WebSocket Websocket { get; set; }


    public override async Task Send(object message)
    {
        await Websocket.Send(message);
    }
}

Subscription

Every value that a client wants to receive has to be requested from the server. If you want the client to receive changes of a collection you have to create a subscription for this collection.

When a client subscribes to a collection a new CollectionSubscription is created and added to the list of the connection.

If something changes the notifier will iterate through all connections and look for subscriptions that are affected by the changes. The changes will then get send to the client.

Client implementation

The server implementation for data synchronization already works. Now I had to create the client implementation. I decided to use RxJs for this job because the concept of data synchronization is pretty reactive and it would do the perfect job.

I will skip the boring part of creating a connection etc and just concentrate on the key part of the synchronization. If you are interested in the exact implementation you can check out the sources.

SapphireDb / SapphireDb

SapphireDb Server, a self-hosted, easy to use realtime database for Asp.Net Core and EF Core

The main part of the client implementation is this piece of code:

const subject = new ReplaySubject<T[]>();

const wsSubscription = this.connectionManagerService.sendCommand(subscribeCommand).subscribe(response => {
    if (response.responseType === 'QueryResponse') {
      subject.next((<QueryResponse>response).result);
    } else if (response.responseType === 'ChangeResponse') {
      // Find and update the value
    } else if (response.responseType === 'UnloadResponse') {
      // Remove the value
    } else if (response.responseType === 'LoadResponse') {
      // Insert the value
    }
});

The main part of this is the ReplaySubject. It holds the value of the collection.

When the client requests a value it sends the SubscribeCommand to the server. The server stores the id associated with this command and every time something related to the subscription changes, it sends a notification with the same id back to the client. That means that everything that is related to the subscription will appear in the subscribe block of this statement.

The client only has to merge the changes into one list and has its value. The cool thing about using RxJs is, that you only have to emit a new value using .next(newValue) and the changes will get emitted where you need it.

The final API can be used like this:

export class DemoComponent implements OnInit {
  values$: Observable<Entry[]>;

  constructor(private db: SapphireDb) { }

  ngOnInit() {
    this.values$ = this.db.collection<Entry>('entries').values();
  }
}

CRUD

Now it was time to implement the basic CRUD operations to make development a lot easier afterwards. This operations are basically just normal calls on the DbContext to manage the data. I just created a reusable interface to call this methods.

The API for the client looks like this:

// add
this.collection.add({
  content: 'test value'
});

// delete
this.collection.remove(value);

// update
this.collection.update({
  ...value,
  content: v
});

Actions

The last requirement I had were the actions I can define on server side and easily call them through an easy API on client side.

I created an ActionHandler that you can generally understand like a controller. It contains different methods that will be callable through an interface.

An example of an action handler looks like this:

public class ExampleActions : ActionHandlerBase
{
  private readonly ExampleDb db;

  public ExampleActions(ExampleDb db)
  {
  this.db = db;
  }

  public async Task<int> AsyncTask()
  {
    for (int i = 0; i <= 100; i++)
    {
      Thread.Sleep(10);
    }

    return 33;
  }
}

You can call the method by using this code on client side:

this.db.execute('example', 'AsyncTask')
  .subscribe(console.log);

The implementation itself is not very interesting but what is definitly worth a thought is the fact, that you can also use async Methods in the action handler.
Because the connection is Bi-Directional the call does not block the communication between client and server until the execution finished. Instead the result will get pushed to the client when ready.

The result

With this results I already was pretty satisfied but I did not stop improving the features. The final project has some cool and interesting features, for example:

💾 Easy CRUD operations
🔑 Authentication/Authorization included
✔️ Database support
🔌 Actions
🌐 NLB support

The final result is called SapphireDb and you can find it on Github. I am thankful for every kind of feedback.

Also check out the documentation to learn more.

SapphireDb / SapphireDb

SapphireDb Server, a self-hosted, easy to use realtime database for Asp.Net Core and EF Core

SapphireDb - Server for Asp.Net Core

SapphireDb is a self-hosted, easy to use realtime database for Asp.Net Core and EF Core.

Check out the documentation for more details: Documentation

Features

🔧 Dead simple configuration
📡 Broad technology support
💻 Self hosted
📱 Offline support
💾 Easy to use CRUD operations
⚡ Model validation
✔️ Database support
📂 Supports joins/includes
➿ Complex server evaluated queries
🔌 Actions
🔑 Authorization included
✉️ Messaging
🌐 Scalable

Learn more

Installation

Install package

To install the package execute the following command in your package manager console

PM> Install-Package SapphireDb

You can also install the extension using Nuget package manager. The project can be found here: https://www.nuget.org/packages/SapphireDb/

Configure

…

View on GitHub