Forem: Garry Xiao

Start Terraform from zero

Garry Xiao — Sun, 30 Oct 2022 22:01:38 +0000

Terraform - Automate Infrastructure on Any Cloud. https://www.terraform.io/

Preparation:
Visual Studio Code + HashiCorp Terraform extension

Provider
Provider for the environment is always in the first position. We take Azure Provider as an example. The Azure Provider can be used to configure infrastructure in Microsoft Azure using the Azure Resource Manager API's. https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs. We could create a "provider.tf" to cover the configuration:

# We strongly recommend using the required_providers block to set the
# Azure Provider source and version being used
terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">=3.0.0"
    }
  }
}

# Configure the Microsoft Azure Provider
provider "azurerm" {
  features {}
}

# Create a resource group
resource "azurerm_resource_group" "example" {
  name     = "example-resources"
  location = "West Europe"
}

# Create a virtual network within the resource group
resource "azurerm_virtual_network" "example" {
  name                = "example-network"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  address_space       = ["10.0.0.0/16"]
}

Provider Data
How to access current provider data? That's Data Source: azurerm_client_config. https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/client_config. Define the data and use it like the code shows.

data "azurerm_client_config" "current" {
}

output "account_id" {
  value = data.azurerm_client_config.current.client_id
}

Resources
Define resources you want, like:

# Resource Group
# https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/resource_group
resource "azurerm_resource_group" "example" {
  name     = "example"
  location = "West Europe"
}

# Key Vault
resource "azurerm_key_vault" "kv" {
  name                            = "KV"
  location                        = azurerm_resource_group.example.location
  resource_group_name             = azurerm_resource_group.example.name
  enabled_for_disk_encryption     = false
  enabled_for_deployment          = true
  enabled_for_template_deployment = true
  tenant_id                       = data.azurerm_client_config.current.tenant_id

  sku_name = "standard"
  network_acls {
    default_action = "Allow"
    bypass         = "AzureServices"
  }
}

# Service bus namespace
resource "azurerm_servicebus_namespace" "example" {
  name                = "example"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  sku                 = "Standard"
}

# Service bus topic
resource "azurerm_servicebus_topic" "example" {
  name                = "example"
  namespace_id        = azurerm_servicebus_namespace.metro60_namespace.id
  enable_partitioning = true
}

Variables
https://developer.hashicorp.com/terraform/language/values/variables

Each input variable accepted by a module must be declared using a variable block:

variable "image_id" {
  type        = string
  description = "The id of the machine image (AMI) to use for the server."
  default     = "abc"
  validation {
    condition     = length(var.image_id) > 4 && substr(var.image_id, 0, 4) == "ami-"
    error_message = "The image_id value must be a valid AMI id, starting with \"ami-\"."
  }
}

# Local variables within modules
locals {
  image_id_len = length(var.image_id)
}

Then we could use "var.image_id" and "local.image_id_len" for institution in the codes.

Git ignore

# Exclude all .tfvars files, which are likely to contain sensitive data, such as
# password, private keys, and other secrets. These should not be part of version 
# control as they are data points which are potentially sensitive and subject 
# to change depending on the environment.
*.tfvars
*.tfvars.json

# Local .terraform directories
**/.terraform/*

# Local .tfstate files
*.tfstate
*.tfstate.*
**/override.tf
*.terraform.lock.hcl

ETSOO Free WYSIWYG HTML Editor practice from zero

Garry Xiao — Mon, 07 Mar 2022 07:30:19 +0000

In order to complete a CMS system for ETSOO, we need a WYSIWYG HTML editor for content publish. There are a lot of components or libraries to do that but no one full covers the basic requirements we want. So we decided to continue a previous project we did serveral years ago called "EOEditor" and apply the new technologies we have to refactor it and make it a free option for the market. Leave all steps here to demostrate the pathways from zero to a workable library for your reference.

Github: https://github.com/ETSOO/EOEditor

Step 1: Init the package
'npm init' to initialize an NPM package. 'npm init -y' will not ask any question and produce the package.json file with default values. 'npm init --scope=' to create an Org scoped package.
npm init --scope=etsoo
Changed package name to "@etsoo/editor"

Step 2: Setup TypeScript
npm install --save-dev typescript ts-loader
npx tsc --init
Changed target to "es2019".

Step 3: Setup Prettier
https://prettier.io/docs/en/install.html
npm install --save-dev --save-exact prettier
Created .prettierignore and .prettierrc two files.

Step 4: Setup testing server
https://webpack.js.org/api/webpack-dev-server/#installation
npm install --save-dev webpack webpack-cli webpack-dev-server
Configue webpack to work with TypeScript with webpack.config.js https://webpack.js.org/guides/typescript/, make sure insert the eoeditor.js(configured in output/filename) to index.html to enable "Live Reloading".

Step 5: Editor as a resuable component
Create it as a custom HTML element. https://developers.google.com/web/fundamentals/web-components/customelements, https://developer.mozilla.org/en-US/docs/Web/Web_Components/Using_custom_elements

Step 6: Commands
Even thougth the 'executeCommand' is depreciated, no alternative is available. Just follow the traditional way. https://developer.mozilla.org/en-US/docs/Web/API/Document/execCommand

Step 7: Dynamic styles with var
Setup variables with same template to support dynamic update for different instances. https://developer.mozilla.org/en-US/docs/Web/CSS/Using_CSS_custom_properties

Step 8: Editor with a Div or IFrame
At the very begining, I took the Div but gave it up later. Because a Div in same document with the buttons could not hold the selection when move the focus to dialog inputs.

Step 9: Image edit
It's a good idea to have some client image edit functions to support basic rotate/resize features. It's done with the library http://fabricjs.com/.

After took over 2 months of my free time, here is the first screen shot of the release.

Other points

Highlight programming codes with https://highlightjs.org/usage/

C# "fire and forget" method

Garry Xiao — Thu, 11 Nov 2021 09:35:35 +0000

Option 1: async void

        static void Main(string[] args)
        {
            try
            {
                Console.WriteLine("Main starts at " + DateTime.Now.ToLongTimeString());
                TestTask();
                Console.WriteLine("Main ends at " + DateTime.Now.ToLongTimeString() + ", " + Thread.CurrentThread.ManagedThreadId);
            }
            catch (Exception ex)
            {
                Console.WriteLine("ex: " + ex.Message);
            }

            Console.Read();
        }

        static async void TestTask()
        {
            try
            {
                Thread.Sleep(3000);
                await Task.Delay(3000);
                throw new InvalidOperationException();
            }
            catch
            {
                Console.WriteLine("TestTask exception at " + DateTime.Now.ToLongTimeString() + ", " + Thread.CurrentThread.ManagedThreadId);
            }
        }

Run result:

Main starts at 21:12:02
Main ends at 21:12:05, 1
TestTask exception at 21:12:08, 7

Conclusions:

Starting sync code inside "async void" method will be executed on the same thread of the main thread. That's "Thread.Sleep(3000)" means.
Exception inside "async void" method cannot be catched if happend begins with the first async call because they are in different context. So if TestTask is without try/catch, then the application will be failed and no way to track.

Option 2: Task.Run

       static void Main(string[] args)
        {
            try
            {
                Console.WriteLine("Main starts at " + DateTime.Now.ToLongTimeString());
                Task.Run(TestTask);
                Console.WriteLine("Main ends at " + DateTime.Now.ToLongTimeString() + ", " + Thread.CurrentThread.ManagedThreadId);
            }
            catch (Exception ex)
            {
                Console.WriteLine("ex: " + ex.Message);
            }

            Console.Read();
        }

        static void TestTask()
        {
            try
            {
                Thread.Sleep(3000);
                Task.Delay(3000);
                throw new InvalidOperationException();
            }
            catch
            {
                Console.WriteLine("TestTask exception at " + DateTime.Now.ToLongTimeString() + ", " + Thread.CurrentThread.ManagedThreadId);
            }
        }

Run result:

Main starts at 22:14:17
Main ends at 22:14:17, 1
TestTask exception at 22:14:20, 4

Conclusions:

It's real "fire and forget" from the call.
Try/catch is also very important because the Main try/catch is impossible to catch the TestTask exception. But the TestTask failure does not affect the Main processing.

So always avoid Async Void and use Task.Run when you want "fire and forget". From its nature, take care of the exception handle and multiple threads scenario, please do not share any resources with the main thread.

FYI:
https://docs.microsoft.com/en-us/archive/msdn-magazine/2013/march/async-await-best-practices-in-asynchronous-programming

SQL Server huge data performance points

Garry Xiao — Thu, 03 Sep 2020 11:41:52 +0000

Based on the instance:

Indexes, https://odetocode.com/articles/237.aspx
Query Hints, https://docs.microsoft.com/en-us/sql/t-sql/queries/hints-transact-sql-query?view=sql-server-ver15
Data Partitioning: https://www.mssqltips.com/sqlservertip/1200/handling-large-sql-server-tables-with-data-partitioning/
Computed Column to avoid unnecessary calculation: https://www.mssqltips.com/sqlservertip/1682/using-computed-columns-in-sql-server-with-persisted-values/

Replication
Database replication is the technology to distribute data from the primary server to secondary servers. There are two main benefits of using SQL Server replication: 1. Using replication, we can get nearly real-time data which can be used for reporting purpose. 2. Define and schedule the replication. SQL Server supports three replication types: 1) Transactional Replication. 2) Snapshot Replication. 3) Merge Replication.SQL Server Database Replication: https://codingsight.com/sql-server-database-replication

Schedule
The schedule is a timed logic calculation. A Job in the database side: https://docs.microsoft.com/en-us/sql/ssms/agent/schedule-a-job?view=sql-server-ver15. Implementing a similar solution with Windows Service is also a choice.

Message Queuing
Messages can be queued and delivered later if the destination is unavailable or busy or need to consume a lot of time and external resources. Using the SQL Server Service Broker for Asynchronous Processing:
https://www.sqlshack.com/using-the-sql-server-service-broker-for-asynchronous-processing/

Caching
Caching is a popular mechanism used to accelerate response times and help applications scale while reducing the load on RDBMS systems, and save on resources. Using Redis with SQL Server: https://lynnlangit.com/2016/09/27/getting-started-with-sql-server-redis/

Create a TypeScript React notification Component from zero

Garry Xiao — Wed, 12 Aug 2020 04:06:11 +0000

Why not use the existing packages? They don't fulfill my requirements. I define the component to do:

Existing popular features should be there.
Totally written in TypeScript.
A framework to work with different UI frameworks like Material-UI. This means I could totally define the appearance being consistent.
Keep the codes neat and easy to understand, and easy to follow for extending.

So the task will be split into 3 parts or levels:

A pure TypeScript/JavaScript package with all features described and partially implemented.
A React component based on the previous package has a full implementation.
A Material-UI version to customize the appearance and behaviors.

Here is help link about how to create a repository: https://dev.to/garryxiao/build-a-react-components-npm-package-and-ci-cd-with-github-action-1jm6

Task 1:
Step 1: Create a repository at https://github.com/ETSOO/NotificationBase
Step 2: Architect, An abstract class Notification presents the message to display. A NotificationContainer class presents a global container for the components, includes add, remove, and registered methods.

Task 2:
Step 1: Create a repository: https://github.com/ETSOO/NotificationUI
Step 2: NotificationReact extends Notification to support under React environment. NotificationDisplay is a React component to display notifications. It will register the update method to the NotificationContainer and achieve notifications add and remove actions.

Task 3:
Step 1: Create a repository: https://github.com/ETSOO/NotificationMU
Step 2: NotificationMU extends Notification to support Material-UI framework. NotificationDisplayMU is a Material-UI implmentation for NotificationDisplay.

Task 4:
A shared package applied: https://github.com/ETSOO/Shared

Please follow the source codes to get a very basic understanding of architectural design. Use the template support of Github repository, split the requirements into 3 parts as micro packages. It may delay the development progress but would benefit the long-term maintenance and improvements.

React infinite loader with TypeScript

Garry Xiao — Thu, 04 Jun 2020 11:57:07 +0000

It's a performance favor solution for a list or grid of huge data. There are limited TypeScript examples to work with 'react-window-infinite-loader' and 'react-window' (A new package for react-virtualized, 'Virtualized Table', https://material-ui.com/components/tables/). I will create a functional component to work through these.

For react-window
https://github.com/bvaughn/react-window/

For react-window-infinite-loader:
https://github.com/bvaughn/react-window-infinite-loader

Encapsulated into a seperated component, reduced unnecessary properties provided with the sample:

import React, { ComponentType } from 'react'
import { FixedSizeList, ListChildComponentProps, Layout, ListOnScrollProps, ListItemKeySelector } from 'react-window'
import InfiniteLoader from 'react-window-infinite-loader'
import { ISearchItem } from '../views/ISearchResult'

/**
 * List item renderer properties
 */
export interface ListItemRendererProps extends ListChildComponentProps {
}

/**
 * Infinite list props
 */
export interface InfiniteListProps {
    /**
     * Is horizontal layout
     */
    horizontal?: boolean

    /**
     * Height
     */
    height?: number

    /**
     * Inital scroll offset, scrollTop or scrollLeft
     */
    initialScrollOffset?: number

    /**
     * Item unit property name, default is id
     */
    itemKey?: string

    /**
     * Item renderer
     * @param props 
     */
    itemRenderer(props: ListItemRendererProps): React.ReactElement<ListItemRendererProps>

    /**
     * Item size (height)
     */
    itemSize: number

    /**
     * Load items callback
     */
    loadItems(page: number, records: number): Promise<ISearchItem[]>

    /**
     * On scroll callback
     */
    onScroll?: (props: ListOnScrollProps) => any

    /**
     * Records to read onetime
     */
    records: number

    /**
     * Width
     */
    width?: string
}

/**
 * Infinite list state class
 */
class InfiniteListState {
    /**
     * List items
     */
    items: ISearchItem[]

    /**
     * All data is loaded
     */
    loaded: boolean

    /**
     * Current page
     */
    page: number

    /**
     * Constructor
     * @param items Init items
     */
    constructor(items: ISearchItem[]) {
        this.items = items
        this.loaded = false
        this.page = 0
    }
}

/**
 * Infinite list component
 * @param pros Properties
 */
export function InfiniteList(props: InfiniteListProps) {
    // Items state
    const [state, updateState] = React.useState(new InfiniteListState([]))

    // Render an item or a loading indicator
    const itemRenderer: ComponentType<ListChildComponentProps> = (lp) => {
        const newProps: ListItemRendererProps = {
            data: state.items[lp.index],
            index: lp.index,
            isScrolling: lp.isScrolling,
            style: lp.style,
        }
        return props.itemRenderer(newProps)
    }

    // Determine the index is ready
    const isItemLoaded = (index: number) => {
        return state.loaded || index < state.items.length
    }

    // Load more items
    const loadMoreItems = async (startIndex: number, stopIndex: number) => {
        // Loaded then return
        if(state.loaded)
            return

        // Read next page
        const page = state.page + 1
        const items = (await props.loadItems(page, props.records)) || []

        // Add to the collection
        state.items.push(...items)

        // New state
        const newState = new InfiniteListState(state.items)
        newState.page = page
        newState.loaded = items.length < props.records

        // Update
        updateState(newState)
    }

    // Add 1 to the length to indicate more data is available
    const itemCount = state.items.length + (state.loaded ? 0 : 1)

    // Default calcuated height
    const height = props.height || props.records * props.itemSize

    // Default 100% width
    const width = props.width || '100%'

    // Layout
    const layout: Layout = props.horizontal ? 'horizontal'  : 'vertical'

    // Item key
    const itemKey: ListItemKeySelector = (index, data) => {
        const field = props.itemKey || 'id'
        if(data == null || data[field] == null)
            return index

        return data[field]
    }

    // Return components
    return (
        <InfiniteLoader isItemLoaded={isItemLoaded} itemCount={itemCount} loadMoreItems={loadMoreItems} minimumBatchSize={props.records} threshold={props.records + 5}>
            {
                ({ onItemsRendered, ref }) => (
                    <FixedSizeList itemCount={itemCount}
                        onItemsRendered={onItemsRendered}
                        onScroll={props.onScroll}
                        ref={ref}
                        layout={layout}
                        itemKey={itemKey}
                        initialScrollOffset={props.initialScrollOffset}
                        itemSize={props.itemSize}
                        width={width}
                        height={height}
                    >{itemRenderer}</FixedSizeList>
                )
            }
        </InfiniteLoader>
    )
}

An example to use it:

    // Load datal
    const loadItems = async (page: number, records: number) => {
        const conditions: CustomerSearchModel = { page, records }
        return (await api.searchPersonItems(conditions)).items
    }

    // Item renderer
    const itemRenderer = (props: ListItemRendererProps) => {
        return (
            <div className={classes.listItem} style={props.style}>{props.index} {props.data == null ? 'Loading...' : props.data['name']}</div>
        )
    }

<InfiniteList itemSize={200} records={5} height={height} loadItems={loadItems} itemRenderer={itemRenderer}/>

Here the property 'height' of the component in vertical case is not easy to dertermine. The full height of the document, minus the app bar height, margin or padding height, is the target height. I coded a hook to calculate two elements at the same time for calculate the real height with

  // Calculate dimensions, pass ref1 to AppBar (position="sticky"), ref2 to the outer Container
  const {ref1, ref2, dimensions1, dimensions2} = useDimensions2<HTMLElement, HTMLDivElement>(true)

  // Setup the actual pixel height
  const mainStyle = {
    height: (dimensions1 && dimensions2 ? (dimensions2.height - dimensions1.height) : 0)
  }

/**
 * Calculate 2 elements dimensions
 * @param observeResize Is observing resize event
 */
export function useDimensions2<E1 extends Element, E2 extends Element>(observeResize: boolean = false) {
    // References for a HTML elements passed to its 'ref' property
    const ref1 = React.useRef<E1>(null)
    const ref2 = React.useRef<E2>(null)

    // Dimensions and update state
    const [dimensions, updateDimensions] = React.useState<DOMRect[]>()

    // Calcuate when layout is ready
    React.useEffect(() => {
        // Update dimensions
        if(ref1.current && ref2.current)
            updateDimensions([ref1.current.getBoundingClientRect(), ref2.current.getBoundingClientRect()])

        // Resize event handler
        const resizeHandler = (event: Event) => {
            if(ref1.current && ref2.current)
                updateDimensions([ref1.current.getBoundingClientRect(), ref2.current.getBoundingClientRect()])
        }

        // Add event listener when supported
        if(observeResize)
            window.addEventListener('resize', resizeHandler)

        return () => {
            // Remove the event listener
            if(observeResize)
                window.removeEventListener('resize', resizeHandler)
        }
    }, [ref1.current, ref2.current])

    // Dimensions
    const dimensions1 = dimensions == null ? null : dimensions[0]
    const dimensions2 = dimensions == null ? null : dimensions[1]

    // Return
    return {
        ref1,
        ref2,
        dimensions1,
        dimensions2
    }
}

There are two additional interesting topics. How to add outer or inner elements add to the InfiniteList:

// Outer element
const outerElementType = React.forwardRef<HTMLElement>((p, ref) => {
    return (
        <Table innerRef={ref}>
            {p.children}
        </Table>
    )
})

Explore data system design with C# and SQL Server

Garry Xiao — Tue, 05 May 2020 05:38:20 +0000

A data system means there could be massive and various data and also focus on data. We want to design the system to control the logic for user authentication and data input. It's a common part of all kinds of information systems. I would like to explore any possibility to have some standards for designing it.

You will find it's challenging to design a system from zero during the junior and intermediate stages of your career, no way to start or confused by different kinds of patterns or options. How to choose a programming language? How to choose the database? How to adapt to expansion and different scales? You need hands-on experiences, but without a valuable guide before it will be a nightmare. The code is getting bloated, and the cost of refactoring is getting bigger and bigger. Until the day of giving up, you don’t necessarily understand why this happens.

Is coding language important? Yes but not the key. Everyone will follow his or her strengths, as a senior role, you should understand the language neutral, just like the natural languages around the world, they all play well with the communication role. You could choose any language that exists, or develop language if you want (it's a joke). Frankly speaking, during a real project, you need to make the decision on your or the team's strengths or favorites. In recent years, Microsoft has been running in the right direction. .NET 5.0 will be a milestone. I am sure C# will be a good choice in the coming years. Another tech giant Google has developed the realm of development language through Node.js and Golang which also are good candidates. I have little positive comments for Java but I don't mean refuse to accept it.

Which database solution is better? SQL Server is the best choice under Microsoft stacks. MySQL is also good if you like it. I once used the Oracle, it's powerful but the question is why I need so much. How about MongoDB? Under stable structured data circumstances, not a good idea. Structured Query Language (SQL) is a wide applied standard. Any relational database could be the choice if you like. NoSQL database in specific cases like dealing with storing JSON-like documents is a winner. There is a wide range of developer ecology, which is very important, because no matter whether the product is good or not, it needs people to work on it.

OK, let's start to design the system with C# and SQL Server. There are 3 points need to be considered with any system. First is the configuration, make sure the system is configurable. The second is storage, local hard drive storage is common, cloud storage is also an option. The last is the database and the data layer interface. Maybe more but they are very typical and enough for illustration.

Configuration or options are common to be designed with read-only properties of a class object. A builder pattern is useful to initialize the properties as showed below:

    /// <summary>
    /// Common configuration for application
    /// 程序对象的通用配置
    /// </summary>
    public abstract class ConfigurationBase : ICoreConfiguration
    {
        /// <summary>
        /// Fluent builder base
        /// 液态基础构建器
        /// </summary>
        public abstract class BuilderBase<T> where T : ConfigurationBase
        {
            /// <summary>
            /// Configuration base
            /// 基础配置对象
            /// </summary>
            private T Configuration;

            /// <summary>
            /// Constructor
            /// 构造函数
            /// </summary>
            /// <param name="configuration">Configuration</param>
            protected BuilderBase(T configuration)
            {
                Configuration = configuration;
            }

            /// <summary>
            /// Build the configuration
            /// 创建配置对象
            /// </summary>
            /// <returns>Configuration</returns>
            public T Build()
            {
                return Configuration;
            }

            /// <summary>
            /// Whether model is validated
            /// 模块是否已经通过验证
            /// </summary>
            /// <param name="validated">Validated</param>
            /// <returns>Builder</returns>
            public BuilderBase<T> ModelValidated(bool validated)
            {
                Configuration.ModelValidated = validated;
                return this;
            }

            /// <summary>
            /// Set private deployment
            /// 设置私有化部署
            /// </summary>
            /// <param name="id">Private delopyment id, to avoid same encription result with same private key</param>
            /// <returns>Builder</returns>
            public BuilderBase<T> PrivateDeployment(string id)
            {
                Configuration.PrivateDeploymentId = id;
                return this;
            }

            /// <summary>
            /// Set service user
            /// 设置服务账号
            /// </summary>
            /// <param name="id">Service user id</param>
            /// <returns>Builder</returns>
            public BuilderBase<T> ServiceUser(string id)
            {
                Configuration.ServiceUserId = id;
                return this;
            }

            /// <summary>
            /// Set keys
            /// 设置键
            /// </summary>
            /// <param name="privateKey">Private key for encrption</param>
            /// <param name="symmetricKey">Symmetric security key, for exchange</param>
            /// <returns>Builder</returns>
            public BuilderBase<T> SetKeys(string privateKey, string symmetricKey)
            {
                Configuration.PrivateKey = privateKey;
                Configuration.SymmetricKey = symmetricKey;
                return this;
            }
        }

        /// <summary>
        /// Flag for identification, default is 'e', like stored procedure name will start with it
        /// 标识值，默认值为 'e'，比如存储过程会以该字母打头
        /// </summary>
        public virtual string Flag
        {
            get { return "e"; }
        }

        /// <summary>
        /// Model DataAnnotations are validated, true under Web API 3 to avoid double validation
        /// 模块数据标记已验证，在Web API 3下可以设置为true以避免重复验证
        /// </summary>
        public bool ModelValidated { get; private set; }

        private string privateDeploymentId;
        /// <summary>
        /// Private delopyment id, to avoid same encription result with same private key
        /// 私有部署编号，避免多个平台相同加密私匙导致加密结果一样
        /// </summary>
        public string PrivateDeploymentId
        {
            get { return privateDeploymentId; }
            private set
            {
                if (value == null)
                    value = string.Empty;

                privateDeploymentId = value;
            }
        }

        /// <summary>
        /// Private key for encrption
        /// 加密私匙
        /// </summary>
        public string PrivateKey { get; private set; }

        /// <summary>
        /// Service user id
        /// 服务用户编号
        /// </summary>
        public string ServiceUserId { get; private set; }

        /// <summary>
        /// Symmetric security key, for exchange
        /// 对称安全私匙，用于交换
        /// </summary>
        public string SymmetricKey { get; private set; }
    }

Designed with abstract class and support the generic builder, is target to be extended easily:

    /// <summary>
    /// Main configuration
    /// 扩展的配置
    /// </summary>
    public class MainConfiguration : ConfigurationBase
    {
        /// <summary>
        /// Main configuration builder
        /// 扩展的配置器
        /// </summary>
        public class Builder : BuilderBase<MainConfiguration>
        {
            /// <summary>
            /// Constructor
            /// 构造函数
            /// </summary>
            public Builder() : base(new MainConfiguration())
            {
            }
        }
    }

There is always a Application, holds the configuration, database and storage resources, could be initialized in singleton or as a static object. An interface of the application demonstrated below:

    /// <summary>
    /// Application interface
    /// 程序接口
    /// </summary>
    public interface IApplication
    {
        /// <summary>
        /// Configuration
        /// 配置
        /// </summary>
        ICoreConfiguration Configuration { get; }

        /// <summary>
        /// Database
        /// 数据库
        /// </summary>
        ICommonDatabase Database { get; }

        /// <summary>
        /// Storage
        /// 存储
        /// </summary>
        IStorage Storage { get; }
    }

For data access layer, Entity Framework (EF) is somehow easier but will lose some performance. I prefer to deal with traditional ADO.NET way.

    /// <summary>
    /// SQL Server Database
    /// SQL Server 数据库
    /// </summary>
    public class SqlServerDatabase : CommonDatabase
    {
        /// <summary>
        /// New connection
        /// 新链接对象
        /// </summary>
        public SqlConnection NewConnection
        {
            get { return new SqlConnection(ConnectionString); }
        }

        /// <summary>
        /// Constructor
        /// 构造函数
        /// </summary>
        /// <param name="connectionString">Database connection string</param>
        public SqlServerDatabase(string connectionString) : base(connectionString)
        {
        }

        /// <summary>
        /// Add parameters to command
        /// DBNull.Value for non-empty NULL
        /// 给命令添加参数
        /// </summary>
        /// <param name="command">Command</param>
        /// <param name="paras">Parameters</param>
        public void AddParameters(SqlCommand command, IDictionary<string, dynamic> paras)
        {
            if (paras == null)
                return;

            command.Parameters.AddRange(paras.Where(item => item.Value != null).Select(item =>
            {
                if (item.Value is SqlParameter p)
                    return p;
                else
                    return new SqlParameter(item.Key, item.Value);
            }).ToArray());
        }

        /// <summary>
        /// Create EF Database Context
        /// 创建EF数据库上下文
        /// </summary>
        /// <typeparam name="M">Model class</typeparam>
        /// <returns>Database Context</returns>
        public override CommonDbContext<M> CreateContext<M>()
        {
            return new SqlServerDbContext<M>(ConnectionString);
        }

        /// <summary>
        /// Execute SQL Command, return rows affacted
        /// 执行SQL命令，返回影响的行数
        /// </summary>
        /// <param name="sql">SQL Command</param>
        /// <param name="paras">Parameters</param>
        /// <param name="isStoredProcedure">Is stored procedure</param>
        /// <returns>Rows affacted</returns>
        public override int Execute(string sql, IDictionary<string, dynamic> paras, bool? isStoredProcedure = false)
        {
            using (var connection = NewConnection)
            using (var command = new SqlCommand(sql, connection))
            {
                // Add parameters
                AddParameters(command, paras);

                // Command type
                if (isStoredProcedure == null)
                    command.Prepare();
                else if (isStoredProcedure.Value)
                    command.CommandType = CommandType.StoredProcedure;

                // Open connection
                connection.Open();

                // Execute
                var result = command.ExecuteNonQuery();

                // Close
                connection.Close();

                // Return
                return result;
            }
        }
    }

But still provide the EF support with the method CreateContext with SqlServerDbContext below. Please do not make confused with Common* things, I just added multiple databases support, like SQLite or MySQL:

    /// <summary>
    /// SQL Server EF Database Context
    /// SQL Server EF 数据库上下文
    /// </summary>
    public class SqlServerDbContext<M> : CommonDbContext<M> where M : class
    {
        private string connectionString;

        /// <summary>
        /// Constructor
        /// 构造函数
        /// </summary>
        /// <param name="connectionString">Connection string</param>
        public SqlServerDbContext(string connectionString)
        {
            this.connectionString = connectionString;
        }

        /// <summary>
        /// Override OnConfiguring to setup
        /// 重写配置初始化
        /// </summary>
        /// <param name="optionsBuilder">Options builder</param>
        protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        {
            optionsBuilder.UseSqlServer(this.connectionString);
        }
    }

How to deal with actions or the business logic? For example, the user's login and CRUD. I always put them into a user service. A service is a realm of particular requirements, linked with the application and current user. For example, for the login action:

    /// <summary>
    /// User service
    /// 用户服务
    /// </summary>
    public sealed class UserSerivce : LoginService
    {
        /// <summary>
        /// Create user service
        /// 创建用户服务
        /// </summary>
        /// <param name="app">Application</param>
        /// <param name="user">Current user</param>
        /// <returns>User service</returns>
        public static UserSerivce Create(IMainApp app, ICurrentUser user)
        {
            var service = new UserSerivce();
            service.Application = app;
            service.User = user;
            return service;
        }

        /// <summary>
        /// Private constructor to prevent initialization
        /// 私有的构造函数防止实例化
        /// </summary>
        private UserSerivce()
        {

        }

        private OperationData GetLoginData(LoginModel model)
        {
            // Create operation data
            var data = CreateOperationData("login");

            // Parameterize modeal
            model.Parameterize(data, this);

            // Return
            return data;
        }

        /// <summary>
        /// Async user login
        /// 异步用户登录
        /// </summary>
        /// <param name="model">Data model</param>
        /// <returns>Action result</returns>
        public async Task<OperationResult> LoginAsync(LoginModel model)
        {
            // Validate model
            var result = ValidateModel(model);

            // Invalid result return anyway
            if (!result.OK)
                return result;

            // Access database to valid
            return await ExecuteAsync(GetLoginData(model), false);
        }
    }

Use private constructor to hide the initialization directly and provide a static method to connect the application and user when creation. Model holds the data and validation rules, method Parameterize just collect the data into database parameters then submit to database side stored procedure to process, for the login model:

    /// <summary>
    /// Login model
    /// https://docs.microsoft.com/en-us/dotnet/api/system.componentmodel.dataannotations?view=netframework-4.8
    /// 登录模型
    /// </summary>
    public class LoginModel : LoginService.DataModel
    {
        [StringLength(8, MinimumLength = 4)]
        /// <summary>
        /// Veryfication code
        /// 验证码
        /// </summary>
        public string Code { get; set; }

        /// <summary>
        /// Id, like number id, mobile, or email
        /// 编号，像数字编号，手机号码或邮箱
        /// </summary>
        [Required]
        [StringLength(256, MinimumLength = 4)]
        public string Id { get; set; }

        /// <summary>
        /// Id type
        /// 编号类型
        /// </summary>
        public LoginIdType? IdType { get; set; }

        /// <summary>
        /// Language cid, like Simplified Chinese zh-CN
        /// 语言编号，如简体中文 zh-CN
        /// </summary>
        [RegularExpression(@"^[a-z]{2}(-[A-Z]{2})?$")]
        public string LanguageCid { get; set; }

        /// <summary>
        /// Current organization id
        /// 当前限定的登录机构
        /// </summary>
        public int? Org { get; set; }

        /// <summary>
        /// Raw password
        /// 原始密码
        /// </summary>
        [Required]
        [StringLength(30, MinimumLength = 4)]
        public string Password { get; set; }

        /// <summary>
        /// Is save login
        /// 是否保存登录
        /// </summary>
        public bool? Save { get; set; }

        private void CalculateIdType()
        {
            if (IdType == null || IdType == LoginIdType.Unknown)
            {
                if (Id.Contains("@"))
                {
                    IdType = LoginIdType.Email;
                }
                else
                {
                    if (Regex.IsMatch(Id, @"^\d+$"))
                    {
                        // Starting with zero or its length more than 10 presents mobile phone number
                        // 以0开头，或者长度大于10位，已经不方便通过数字编号记忆，识别为移动手机号码
                        if (Id.StartsWith("0") || Id.Length >= 10)
                        {
                            IdType = LoginIdType.Mobile;
                        }
                        else
                        {
                            IdType = LoginIdType.Id;
                        }
                    }
                    else
                    {
                        IdType = LoginIdType.Cid;
                    }
                }
            }
        }

        /// <summary>
        /// Override to collect parameters
        /// 重写收集参数
        /// </summary>
        /// <param name="data">Data</param>
        /// <param name="service">Service</param>
        public override void Parameterize(OperationData data, IService<int> service)
        {
            base.Parameterize(data, service);

            // Calculate id type
            CalculateIdType();

            // Add parameters
            var paras = data.Parameters;

            // Verification code
            paras.Add("has_code", true);

            paras.Add("id", Id);
            paras.Add("id_type", (byte)IdType);
            paras.Add("ip", service.User.ClientIp.ToString());
            paras.Add("language_cid", LanguageCid ?? service.User.LanguageCid);

            // Login method, 1 = Default
            paras.Add("method", 1);

            paras.Add("org", Org);

            // Hash password
            paras.Add("password", service.Application.HashPassword(Password));

            paras.Add("save_login", Save.GetValueOrDefault());
        }
    }

In the database side, there is a stored procedure to deal with parameters and execute to provide a result:

CREATE PROCEDURE [dbo].[ep_user_login]
-- ================================================
--
-- User login，Error：90006
-- 
-- ================================================
    @id_type tinyint,                 -- Id type
    @id varchar(256),                 -- User number id, email or mobile
    @password varchar(256),           -- Password encrypted
    @has_code bit,                    -- Is recaptu
    @save_login bit,                  -- Save login
    @language_id smallint = NULL,     -- Language id
    @language_cid varchar(5) = NULL,  -- Language cid
    @org int = NULL,                  -- Organization limited
    @origin varchar(256) = NULL,      -- CORS origin
    @service_user_id int = NULL,      -- Service user id running

    @ip varchar(45),                  -- IP
    @method tinyint                   -- Login method
AS
BEGIN
    SET NOCOUNT ON;

    -- Error code
    DECLARE @error_code int = 90006;

    -- Default language id
    DECLARE @default_language_id smallint = dbo.ef_get_login_language_id(@language_id, @language_cid, 0);

    -- If there is no user, return initialization tip
    IF NOT EXISTS(SELECT * FROM e_system_setting WHERE initialized = 1)
        BEGIN
            SELECT @error_code AS [error_code], 'initialization' AS m_id, dbo.ef_get_translation('initialization', @default_language_id) AS [message];
            RETURN;
        END
    ....
END
GO

In order to save energies with the core system and Web API integration. I would like to share the model. The API side code:

        /// <summary>
        /// Login for authentication
        /// 登录授权
        /// </summary>
        /// <param name="model">Data model</param>
        /// <returns>Result</returns>
        [AllowAnonymous]
        [HttpPost("Login")]
        public async Task Login(LoginModel model)
        {
            // Act
            var result = await Service.LoginAsync(model);

            if (result.OK)
            {
                // Logined user id
                var userId = result.Data.Get("token_user_id", 0);

                // User role
                var role = result.Data.Get("role", UserRole.User);

                // Hold the token value and then return to client
                result.Data["authorization"] = CreateToken(userId, role);

                // Suggested refresh seconds
                result.Data["refresh_seconds"] = 300;
            }

            // Output
            await ResultContentAsync(result);
        }

In order to speed up all its best, any 'new' generic constraint related to Activator.CreateInstance avoided. Put logic heavily on the database side with the power of stored procedures will benefit data manipulation. Code snippets like hills stop you from viewing the panorama.

Two issues I met listed here:

Web deploy publishes wrong version of assembly, always lower than debug version (https://stackoverflow.com/questions/32657241/web-deploy-publishes-wrong-version-of-assembly/33223619#comment110408941_33223619). With VS 2019, .Net Core 3.1, delete the folder 'obj\Release\netcoreapp3.1\win-x64\R2R'.
SQL Server json truncated (https://stackoverflow.com/questions/51087037/sql-server-json-truncated-even-when-using-nvarcharmax)

I draw a diagram here for your reference:

I hope it will provide you some valuable ideas when designing a system from zero.

Form validation with Yup under React and Material-UI

Garry Xiao — Sat, 02 May 2020 11:37:35 +0000

In a real project, facing the form validation soon begin on frontend coding. After several rounds of refactoring, I completed it with 4 points in my project:

Totally TypeScript
Speed up development with depository support
Custom hook
Minimum refactoring for components

Chose Yup for validation schema definition, it is simple and easy to understand:
https://github.com/jquense/yup

npm install -S yup
npm install -D @types/yup

React custom hook is a common function, with parameters for input and return necessary tool methods. useFormValidator as below is a custom hook and only rely on packages "react" and "yup", no relationship with the Material-UI framework:

import React from "react"
import * as Yup from 'yup'

/**
 * Form validator state field
 */
interface FormValidatorStateField {
    /**
     * Is error state
     */
    error: boolean

    /**
     * state text
     */
    text: string
}

/**
 * Form validator state fields
 */
interface FormValidatorStateFields {
    [key: string]: FormValidatorStateField
}

/**
 * Form validatior
 * @param schemas Initial validation schemas
 * @param milliseconds Merge change update interval
 */
export const useFormValidator = (schemas: Yup.ObjectSchema<object>, milliseconds: number = 200) => {
    // useState init
    const defaultState: FormValidatorStateFields = {}
    const [state, updateState] = React.useState<FormValidatorStateFields>(defaultState)

    // Change timeout seed
    let changeSeed = 0

    // Change value handler
    const commitChange = (field: string, value: any) => {
        // Validate the field, then before catch, if catch before then, both will be triggered
        Yup.reach(schemas, field).validate(value).then(result => {
            commitResult(field, result)
        }).catch(result => {
            commitResult(field, result)
        })
    }

    // Commit state result
    const commitResult = (field: string, result: any) => {
        let currentItem = state[field]
        if(result instanceof Yup.ValidationError) {
            // Error
            if(currentItem) {
                // First to avoid same result redraw
                if(currentItem.error && currentItem.text == result.message)
                    return

                // Update state
                currentItem.error = true
                currentItem.text = result.message
            } else {
                // New item
                const newItem: FormValidatorStateField = {
                    error: true,
                    text: result.message
                }
                state[field] = newItem
            }
        } else {
            // Success and no result, just continue
            if(currentItem == null)
                return

            // Delete current state result
            delete state[field]
        }

        // Update state, for object update, need a clone
        const newState = {...state}
        updateState(newState)
    }

    // Clear timeout seed
    const clearSeed = () => {
        if(changeSeed > 0)
            clearTimeout(changeSeed)
    }

    // Delay change
    const delayChange = (field: string, value: any) => {
        clearSeed()

        changeSeed = setTimeout(() => {
            commitChange(field, value)
        }, milliseconds)
    }

    // Merge into the life cycle
    React.useEffect(() => {
        return () => {
            // clearTimeout before dispose the view
            clearSeed()
        }
    }, [])

    // Return methods for manipulation
    return {
        /**
         * Input or Textarea blur handler
         * @param event Focus event
         */
        blurHandler: (event: React.FocusEvent<HTMLInputElement | HTMLTextAreaElement>) => {
            const { name, value } = event.currentTarget
            delayChange(name, value)
        },

        /**
         * Input or Textarea change handler
         * @param event Change event
         */
        changeHandler: (event: React.ChangeEvent<HTMLInputElement | HTMLTextAreaElement>) => {
            const { name, value } = event.currentTarget
            delayChange(name, value)
        },

        /**
         * Commit change
         */
        commitChange: commitChange,

        /**
         * State error or not
         * @param field Field name
         */
        errors: (field: string) => {
            return state[field]?.error
        },

        /**
         * State text
         * @param field Field name
         */
        texts: (field: string) => {
            return state[field]?.text
        },

        /**
         * Validate form data
         * @param data form data, Object.fromEntries(new FormData(form))
         */
        validate: async (data: any) => {
            try
            {
                clearSeed()
                return await schemas.validate(data, { strict: true, abortEarly: false, stripUnknown: false })
            }
            catch(e)
            {
                // Reset
                const newState: FormValidatorStateFields = {}

                // Iterate the error items
                if(e instanceof Yup.ValidationError) {
                    for(let error of e.inner) {
                        // Only show the first error of the field
                        if(newState[error.path] == null) {
                            // New item
                            const newItem: FormValidatorStateField = {
                                error: true,
                                text: error.message
                            }

                            newState[error.path] = newItem
                        }
                    }
                }

                // Update state
                updateState(newState)
            }

            return null
        }
    }
}

When use it in Materal-UI pages, for example a login page:

// Login component
function Login() {

    // Form validator
    const { blurHandler, changeHandler, errors, texts, validate } = useFormValidator(validationSchemas)

    // Login action
    async function doLogin(event: React.FormEvent<HTMLFormElement>) {
        // Prevent default action
        event.preventDefault()

        // Form JSON data
        let data = await validate(Object.fromEntries(new FormData(event.currentTarget)))
        if(data == null)
          return

        // Local data format

        // Parase as model
        const model = data as LoginModel
   }

    return (
        <Container component="main" maxWidth="xs">
        <CssBaseline />
        <img src={window.location.origin + '/logo.jpg'} alt="Logo" className={classes.logo}/>
        <div className={classes.paper}>
          <Avatar className={classes.avatar}>
            <LockOutlined />
          </Avatar>
          <Typography component="h1" variant="h5">
            Sign in
          </Typography>
          <form className={classes.form} onSubmit={doLogin} noValidate>
            <TextField
              variant="outlined"
              margin="normal"
              required
              fullWidth
              id="id"
              label="Id or Email"
              name="id"
              error={errors('id')}
              helperText={texts('id')}
              onChange={changeHandler}
              onBlur={blurHandler}
              autoComplete="email"
              autoFocus
            />
            <TextField
              variant="outlined"
              margin="normal"
              type="password"
              required
              fullWidth
              name="password"
              error={errors('password')}
              helperText={texts('password')}
              onChange={changeHandler}
              onBlur={blurHandler}
              label="Password"
              id="password"
              autoComplete="current-password"
            />
            <FormControlLabel
              control={<Checkbox name="save" value="true" color="primary" />}
              label="Remember me"
            />
            <Button
              type="submit"
              fullWidth
              variant="contained"
              color="primary"
              className={classes.submit}
            >
              Sign In
            </Button>
          </form>
        </div>
      </Container>
    )

First declear the validation schemas, initialize 'useFormValidator' and accept the returned methods for binding:

              error={errors('password')}
              helperText={texts('password')}
              onChange={changeHandler}
              onBlur={blurHandler}

Through bindings only to current components to indicate any validation errors occur. No refactoring or extending for current components. That's the key feature of the task I enjoyed.

TypeScript extending ReactJs component from basic

Garry Xiao — Wed, 29 Apr 2020 11:24:20 +0000

From the official article (https://reactjs.org/docs/composition-vs-inheritance.html), it recommends using composition instead of inheritance to reuse code between components. As the support of hooks with functional components, that's the trends, especially with the support of TypeScript, will make things amazing.

Start simplest example:



function TestComponent() {
  return (
    <h1>Hello, world!</h1>
  )
}

ReactDOM.render(
  <TestComponent />,
  document.getElementById('root')
)

As you see, a functional component is just a function with return. As required, User-Defined Components Must Be Capitalized. Here JSX codes used, with TypeScript file extension 'tsx'. Each JSX element, like <h1>Hello, world!</h1> is just syntactic sugar for calling React.createElement(component, props, ...children), as React.createElement('h1', null, 'Hello, world!'). So, anything you can do with JSX can also be done with just plain JavaScript. More details please view https://reactjs.org/docs/jsx-in-depth.html. So the example below with pure TypeScript (file extension could be .ts) is equal:



function TestComponent() {
  return React.createElement('h1', null, 'Hello, world!')
}

Property support is common. We make the example a little complex:



interface TestComponentProps {
  name?: string
}

function TestComponent(props: TestComponentProps) {
  return (
    <h1>{props.name || 'Unknown'} - Hello, world!</h1>
  )
}

ReactDOM.render(
  <TestComponent name="Garry" />,
  document.getElementById('root')
)

Under TypeScript, we use an interface or a type to define the properties. The nullable property adds a '?' after the property name. Now the component can accept the property 'name' and change the output accordingly. You can add any other properties as you want.

It's not necessary to write everything during development. There are UI frameworks who have made a lot of effort to speed up the process. Like Material-UI (https://material-ui.com/) or Antd (https://ant.design). Just follow the documentation, understand each component, practice them, and will be handy. Then custom a component would be necessary. Here will make an extended TestComponent:



interface TestComponentProps {
  name?: string
}

function TestComponent(props: TestComponentProps) {
  return (
    <h1>{props.name || 'Unknown'} - Hello, world!</h1>
  )
}

interface TestComponentNewProps extends TestComponentProps {
  age?: number
}

function TestComponentNew(props: TestComponentNewProps) {
  return (
    <div>
      <TestComponent {...props}/>
      <h2>Age: {props.age}</h2>
    </div>
  )
}

ReactDOM.render(
  <TestComponentNew name="Garry" age="40" />,
  document.getElementById('root')
)

How to extend a Material-UI component? We change the previous example to extend the Button component:



import React from "react"
import ReactDOM from "react-dom"
import Button, { ButtonProps } from "@material-ui/core/Button"

const TestComponentNew : React.FunctionComponent<ButtonProps> = (props) => {
  props = Object.assign({ variant: 'contained' }, props)
  return (
    <Button {...props}>{props.children}</Button>
  )
}

ReactDOM.render(
  <div>
    <Button variant="contained">Source button</Button>
    <br/>
    <TestComponentNew>Extended button</TestComponentNew>
  </div>,
  document.getElementById('root')
)

The key point is to use 'React.FunctionComponent' to extend and pass ButtonProps as a strong type for the generic method. Then you could use props.children and other properties inside. It's impossible to set properties directly but could use Object.assign to set default value. The output is:

Another topic added here is ref (https://reactjs.org/docs/forwarding-refs.html). Here is a TypeScript sample to deal with it:



import React, { FormEvent } from "react"

/**
 * API form properties
 */
export interface APIFormProps {
    /**
     * Style class name
     */
    className?: string
}

/**
 * API form reference interface
 */
export interface APIFormRef {
    changeHandler(event: React.ChangeEvent<HTMLInputElement>):void
}

/**
 * API Form
 * @param props 
 * @param ref 
 */
const APIFormForward : React.ForwardRefRenderFunction<APIFormRef, React.PropsWithChildren<APIFormProps>> = (
    props,
    ref
) => {
    // hooks
    const formElement = React.useRef<HTMLFormElement | null>(null);
    React.useImperativeHandle(ref, () => ({
        changeHandler: (event: React.ChangeEvent<HTMLInputElement>) => {
            console.log(event)
        }
      }))

    return (
        <form ref={formElement} {...props}>{props.children}</form>
    )
}
export const APIForm = React.forwardRef(APIFormForward)

    // Form reference
    let formRef = React.useRef<APIFormRef>(null)

    const changeHandler = (event: React.ChangeEvent<HTMLInputElement>) => {
        // Call the method from an event handler
        formRef.current?.changeHandler(event)
    }

By the way, https://codesandbox.io/ is a good place for practicing. That's all. Enjoy yourself on the journey!

Initial points of a React project

Garry Xiao — Mon, 27 Apr 2020 08:51:12 +0000

React is popular, with the understanding of its methodology, take it like Lego, all are components, build it with components. I prefer the functional components to class-based components. Writing functional components with TypeScript is a necessary combination nowadays. Except that, there are 4 points we need to determine at the elementary stage.

A. Global state management, except 'useState' for the local state of functional components and 'state' property of class-based components. Two solutions:
a) Context, designed to share data that can be considered “global” for a tree of React components. Apply it sparingly because it makes component reuse more difficult (https://reactjs.org/docs/context.html). It's better to use different Contexts for different purposes. Here I create two Contexts, UserStateContext/UserStateProvider for user state, LanguageStateContext/LanguageStateProvider for language and labels state.

In order to improve the code reuse, create a generic function to create them.
CreateState.tsx source codes:

import React from "react"
import { IState, IAction, IUpdate } from "./IState"

/**
 * Generic to create state context and provider
 * @param reducer Reduce function
 * @param initState Init state
 */
export function CreateState<S extends IState, A extends IAction>(reducer: React.Reducer<S, A>, initState: S) {
    // State context
    const context = React.createContext({} as IUpdate<S, A>)

    // State context provider
    const provider: React.FunctionComponent = (props) => {
        // Update reducer
        const [state, dispatch] = React.useReducer(reducer, initState)

        // Avoid unnecessary re-renders
        // https://alligator.io/react/usememo/
        const contextValue = React.useMemo(() => {
            return { state, dispatch }
        }, [state, dispatch])

        return (
            <context.Provider value={contextValue}>{props.children}</context.Provider>
        )
    }

    // Return
    return {
        context,
        provider
    }
}

IState.ts to define needed interfaces:

import React from "react"

/**
 * State data interface
 */
export interface IState {
}

/**
 * State action interface
 */
export interface IAction {
}

/**
 * State update interface
 */
export interface IUpdate<S extends IState, A extends IAction> {
    state: S,
    dispatch: React.Dispatch<A>
}

LanguageState.ts source code:

import { IAction } from "./IState"
import { CreateState } from "./CreateState"

/**
 * Language label
 * Indexable type
 */
export interface LanguageLabel {
    [key: string]: string
}

/**
 * Language state
 */
export interface ILanguage {
    /**
     * Global labels
     */
    labels: LanguageLabel

    /**
     * Current language name
     */
    name: string
}

/**
 * Language action to manage language and labels
 */
export interface LanguageAction extends IAction {
    /**
     * Language cid, like 'zh-CN'
     */
    name: string,

    /**
     * Labels of the language
     */
    labels: LanguageLabel
}

/**
 * Language reducer
 * @param state State
 * @param action Action
 */
export function LanguageReducer(state: ILanguage, action: LanguageAction) {
    if(action?.name) {
        return Object.assign({}, state, action)
    }

    return state
}

/**
 * Language context and provider
 */
export const { context: LanguageStateContext, provider: LanguageStateProvider } = CreateState(LanguageReducer, {} as ILanguage)

UserState.ts source codes:

import { IAction } from "./IState"
import { CreateState } from "./CreateState"

/**
 * Application user update interface
 */
export interface IUserUpdate {
    /**
     * User name
     */
    name: string
}

/**
 * Application user interface
 */
export interface IUser extends IUserUpdate {
    /**
     * Authorized or not
     */
    authorized: boolean

    /**
     * User id
     */
    id: number

    /**
     * Organization current user belongs
     */
    organization_id: number
}

/**
 * User action type
 * Style like 'const enum' will remove definition of the enum and cause module errors
 */
export enum UserActionType {
    // Login action
    Login = "LOGIN",

    // Logout action
    Logout = "LOGOUT",

    // Update action
    Update = "UPDATE"
}

/**
 * User action to manage the user
 */
export interface UserAction extends IAction {
    /**
     * Type
     */
    type: UserActionType,

    /**
     * User
     */
    user?: IUser,

    /**
     * User update
     */
    update?: IUserUpdate
}

/**
 * User reducer
 * @param state State
 * @param action Action
 */
export function UserReducer(state: IUser, action: UserAction) {
    switch(action.type) {
        case UserActionType.Login:
            return Object.assign({}, action.user)
        case UserActionType.Logout:
            return {} as IUser
        case UserActionType.Update:
            return Object.assign({}, state, action.update)
        default:
            return state
    }
}

/**
 * User context and provider
 */
export const { context: UserStateContext, provider: UserStateProvider } = CreateState(UserReducer, {} as IUser)

index.tsx source code to show how to use the Contexts:

import React from 'react'
import ReactDOM from 'react-dom'
import { UserStateProvider, LanguageStateProvider } from 'etsoo-react'
import './index.css'
import App from './App'
import * as serviceWorker from './serviceWorker'

ReactDOM.render(
  <React.StrictMode>
    <UserStateProvider>
      <LanguageStateProvider>
        <App />
      </LanguageStateProvider>
    </UserStateProvider>
  </React.StrictMode>,
  document.getElementById('root')
)

// If you want your app to work offline and load faster, you can change
// unregister() to register() below. Note this comes with some pitfalls.
// Learn more about service workers: https://bit.ly/CRA-PWA
serviceWorker.unregister()

b) Redux (https://redux.js.org/). By contrast with the Context solution, I would like to say, Redux is a little complex and if there is no legacy burden, just give it up.

B. Router, route-based code-splitting, ideal for Single Page Application (or SPA) (https://reacttraining.com/react-router/web/guides/quick-start).
a) Install: "npm install -g react-router-dom".
b) Install: "npm install -g @types/react-router-dom" for TypeScript support.
c) Use 'Switch', render the first child from top to bottom that matches the path.
d) Add property 'exact' to 'Route' for an exact match, especially to home with '/'.
e) Add private routes to limit page access with authentication.

Custom PrivateRoute under package 'etsoo-react' codes:

import React from 'react'
import { Redirect, Route, RouteProps } from 'react-router-dom'
import History from 'history'

/**
 * Private router property interface
 */
export interface PrivateRouteProp extends RouteProps {
    authorized: boolean
}

/**
 * Private router redirect state interface
 */
export interface PrivateRouteRedirectState {
    /**
     * referrer location
     */
    referrer: History.Location
}

/**
 * Private route for react-router-dom
 * Configue a strict route with '/login' to redirect to application's actual login page
 * In login page, useLocation() to get the Location object, and Location.state as PrivateRouteRedirectState to access the referrer
 */
export const PrivateRoute:React.FunctionComponent<PrivateRouteProp> = ({authorized, ...rest}) => {
    return (
        authorized ? <Route {...rest}/> : <Redirect to={{pathname: '/login', state: {referrer: rest.location} as PrivateRouteRedirectState}}/>
    )
}

App.tsx source codes:

import React, { useContext } from 'react'
import { Route, Switch, BrowserRouter } from 'react-router-dom'
import { PrivateRoute, UserStateContext } from 'etsoo-react'

import CustomerSearch from './customer/Search'
import CustomerView from './customer/View'
import Login from './public/Login'
import Main from './main/Main'

function App() {
  // State user
  const { state } = useContext(UserStateContext)

  // Authorized
  const authorized:boolean = (state == null ? false : state.authorized)

  return (
    <BrowserRouter>
      <Switch>
        <PrivateRoute authorized={authorized} path="/customer/search/:condition?" component={CustomerSearch} />
        <PrivateRoute authorized={authorized} path="/customer/view/:id" component={CustomerView} />
        <PrivateRoute authorized={authorized} exact path="/main" component={Main} />
        <Route component={Login} />
      </Switch>
    </BrowserRouter>
  )
}

export default App

In the login page, example codes to show how to access it:

import React from 'react'
import { useLocation } from 'react-router-dom'
import { PrivateRouteRedirectState } from 'etsoo-react'

function Login() {
    let location = useLocation()
    let state = location.state as PrivateRouteRedirectState

    return (
    <h1>Login page, referer: { state?.referrer?.pathname }</h1>
    )
}

export default Login

C. API consuming, apply 'npm install -g axios' to install Axios, or any other framework you are familiar with. It's not the key issue. As API consuming, work together with the API developer with mutual understandings of each other of standards and rules. Then in React, encapsulation is the key part. Consuming APIs in page-level components, not component parts under it, will make sure the components reuse not interrupted by API calls.

Thanks to TypeScript, make JavaScript programming much checkable and reasonable. For example, there is an API to get a list:

    /**
     * Get tiplist data
     * @param model Data model
     */
    async tiplist<M extends TiplistModel>(model: M | null = null) {
        return (await this.api.get('tiplist', { params: model })).data as IListItem[]
    }

Parameters are abstracted to a model for client calls. There is also a very similar model definition on the server-side to receive and hold the parameters.

/**
 * Tiplist model
 */
export interface TiplistModel {
    /**
     * Id
     */
    id: number,

    /**
     * Id array
     */
    ids: number[],

    /**
     * Hide id
     */
    hideId: number,

    /**
     * Hide ids array
     */
    hideIds: number[],

    /**
     * Records to read
     */
    records: number

    /**
     * Search keyword
     */
    sc: string
}

The result is also cast to an interface IListIItem array:

/**
 * Common list item interface
 */
export interface IListItem {
    // Id
    id: number

    // Label
    label: string
}

You could leave the results as pure JSON. But strong types are definitely useful even for a small project, for coding, debugging, and testing.

D. Testing
https://reactjs.org/docs/testing-recipes.html. As a team leader and project manager, For small and medium-size projects, I prefer to put the testing responsibility on coding engineers first, borrow some ideas with TDD, during the development step to speed up the output. After the project launched and somehow stable, then merge all testing codes into a unique role.

Build a React components NPM package and CI/CD with Github Action

Garry Xiao — Fri, 10 Apr 2020 22:04:06 +0000

Lockdown in NZ creates spare time for me to enjoy something occupied before with chore. Recently, I started to summarize the React framework of SmartERP that a SaaS service I led before, try some new solutions. I would like to demonstrate how to build an NPM (Node.js Package Manager) package and implement CI/CD with Github Action to automate most of the work.

Preparation

Create a new public repository "ETSOO/etsoo-react" on GitHub.
Change to the target folder, clone the repository into it: "git clone https://github.com/ETSOO/etsoo-react".
Install the latest Node.js to your computer(https://nodejs.org/en/).
Run 'Node.js command prompt', input command 'npm init' to initialize an NPM package. 'npm init -y' will not ask any question and produce the package.json file with default values. 'npm init --scope=<your_org_name>' to create an Org scoped package.
'git add package.json', multiple files are separated by a blank (Boring? learn something from https://stackabuse.com/git-add-all-files-to-a-repo/, 'git add -A'), then 'git commit -m "Message about the commit"' make changes to the local depository, then 'git push origin master' to upload changes to Github. If you made changes on Github, you need to first pull the updates 'git pull origin master' and then push the local version. Beware of any conflicts here. Run 'git stash git pull git stash pop' to keep local updates.
Create an account in the NPM registry (https://www.npmjs.com/signup). Enter the command 'npm login', provide account details you just have to complete it. Enter the command 'npm publish' to publish, every time needs to upgrade the "version" under package.json.
IDE(Integrated Development Environment): Visual Studio Code, https://code.visualstudio.com/

React & TypeScript:

'npm install -g typescript react react-dom @types/react @types/react-dom' install the minimal set of dependencies required to React and TypeScript.
Create a folder 'src', create an 'index.ts' under it. Change package.json, set "main": "lib/index.js" (It's not so perfect to include the ts files directly, with "src/index.ts" will cause parse error, seems tsc will not compile files under node_modules); "types": "lib/index.d.ts", scripts add "build": "tsc".
Copy a 'tsconfig.json' from another project to the root and change settings as you want or 'npx tsc --init'. Set "declaration": true, generates corresponding definitions in index.d.ts. "jsx": "react" if include 'tsx' files. "outDir": "./lib" tell the compiler that the 'src' folder will be compiled to javascript in 'lib' folder. Add the folder name to '.gitignore'. Make sure "noEmit": false.
If unknown errors occurred, run 'npm install' to check the dependencies and install any missing packages.
Test your new NPM module without publishing it (https://medium.com/@the1mills/how-to-test-your-npm-module-without-publishing-it-every-5-minutes-1c4cb4b369be). Run 'npm link' to define a global link. Then go to the project share this package, run 'npm link etsoo-react' to add the global link to the 'node_modules' like it has been published. Run '' to remove the link. Run 'npm unlink --no-save ' on your project’s directory to remove the local symlink, and run 'npm unlink' on the module’s directory to remove the global symlink. (26/05/2021 update: just install local package with: npm install file:../packagename and the link will be created automatically)

Testings:

Run 'npm i jest @types/jest ts-jest -D' to install the testing framework Jest (https://jestjs.io/). Add a folder 'tests' under the root, add a test script file to pass the 'npm test' command.
Add "jest": "^25.3.0", under package.json/devDependencies if not exits and run 'npm install'.
add "jest": { "testMatch": [ "/tests/*/.js" ] }, to the package.json. Limit Jtest to the folder "test" under root.
Install vscode-jest
npm install -D react-test-renderer
npm install -D babel-jest babel-core@^7.0.0-bridge.0 @babel /core @babel /preset-env regenerator-runtime (https://jestjs.io/docs/en/22.x/getting-started.html)
npm install --save-dev enzyme enzyme-adapter-react-16 @types/enzyme @types/enzyme-adapter-react-16

tsconfig.json:

{
  "compilerOptions": {
    "outDir": "./lib",
    "target": "ES2018",
    "lib": [
      "dom",
      "dom.iterable",
      "esnext"
    ],
    "allowJs": true,
    "skipLibCheck": true,
    "esModuleInterop": true,
    "allowSyntheticDefaultImports": true,
    "strict": true,
    "forceConsistentCasingInFileNames": true,
    "module": "esnext",
    "moduleResolution": "node",
    "resolveJsonModule": true,
    "isolatedModules": true,
    "noEmit": false,
    "jsx": "react",
    "declaration": true
  },
  "include": [
    "src"
  ]
}

package.json:

{
  "name": "etsoo-react",
  "version": "1.0.2",
  "description": "ETSOO React TypeScript components NPM package",
  "main": "lib/index.js",
  "types": "lib/index.d.ts",
  "scripts": {
    "build": "tsc",
    "test": "jest"
  },
  "jest": {
    "testMatch": [ "<rootDir>/tests/**/*.js" ]
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/ETSOO/etsoo-react.git"
  },
  "author": "Garry Xiao",
  "license": "MIT",
  "bugs": {
    "url": "https://github.com/ETSOO/etsoo-react/issues"
  },
  "homepage": "https://github.com/ETSOO/etsoo-react#readme",
  "devDependencies": {
    "@types/react": "^16.9.33",
    "@types/react-dom": "^16.9.6",
    "jest": "^25.3.0",
    "react": "^16.13.1",
    "react-dom": "^16.13.1",
    "typescript": "^3.8.3"
  }
}

Github Actions:

Choose 'Publish Node.js Package' from 'Popular continuous integration workflows' under the Actions tab and create the template YAML file under 'etsoo-react/.github/workflows/'. View https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#about-yaml-syntax-for-workflows for help.
Log in NPM, under 'Auth Tokens', create a new one and copy the token, then go to Github repository's settings, secrets, 'add a new secret' named 'GXNpmToken' here.
Github desktop (https://desktop.github.com/) is an interesting tool to help you manage depositories.

YAML file content, please view https://github.com/ETSOO/etsoo-react/blob/master/.github/workflows/npmpublish.yml. After you push the changes, the Action will execute and later you will receive a notification email from NPM. That's really exciting!

(2020/7/23) Setting up ESLint to work with TypeScript (https://www.robertcooper.me/using-eslint-and-prettier-in-a-typescript-project)

npm install eslint @typescript-eslint/parser @typescript-eslint/eslint-plugin -D.
npx eslint --init, choose a popular styling like airbnb, json format.
"comma-dangle": ["error", "never"], "no-console": "off", "arrow-parens": "off", "linebreak-style": "off".
npm install prettier -D. Install prettier extension in VSCode. "File" -> "References" -> "Settings" -> search "Format On Save".
npm install -D eslint-config-prettier. Create a local configure file '.prettierrc'. Example project: https://github.com/ETSOO/restclient

If you want to upgrade all dependencies, should be very careful, please follow: https://flaviocopes.com/update-npm-dependencies/

npm outdated, give a list.
npm install -g npm-check-updates, install the tool.
ncu -u, update the version.
npm update / npm install

Here is the help link about how to create a template repository and how to use it: https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/creating-a-template-repository

Web vulnerabilities and options for .net core API 3.1

Garry Xiao — Wed, 08 Apr 2020 12:04:22 +0000

There are four common vulnerabilities in web applications. Be aware of these risks, master features of the technology stacks that help you secure your apps and prevent security breaches is necessary.

Cross-site scripting attacks (XSS). Core tip: All data received from clients are untrusted. When you want to output the content, keep an eye on any possibility of including any executable scripts.
https://en.wikipedia.org/wiki/Cross-site_scripting
https://owasp.org/www-community/attacks/xss/
https://docs.microsoft.com/en-us/aspnet/core/security/cross-site-scripting?view=aspnetcore-3.1
SQL injection attacks. Core tip: The concatenation of raw SQL command text with parameters or parts from an untrusted source should be seriously validated.
https://en.wikipedia.org/wiki/SQL_injection
https://portswigger.net/web-security/sql-injection
https://docs.microsoft.com/en-us/ef/core/querying/raw-sql
Cross-Site Request Forgery (CSRF), also known as one-click attack or session riding. Core tip: Two websites are browsed, one log in and another is malicious. Submit requests from the malicious website attached with your valid cookie authentication is the common way to attack.
https://en.wikipedia.org/wiki/Cross-site_request_forgery
https://docs.microsoft.com/en-us/aspnet/core/security/anti-request-forgery?view=aspnetcore-3.1
Open redirect attacks, also known as "Unvalidated Redirects and Forwards". Core tip: If the login redirects with an unchecked query parameter, users from a fake link could be redirected to a similar login page and causing their credential data leaks.
https://www.trustwave.com/en-us/resources/blogs/spiderlabs-blog/understanding-and-discovering-open-redirect-vulnerabilities/
https://docs.microsoft.com/en-us/aspnet/core/security/preventing-open-redirects?view=aspnetcore-3.1

.Net core web API 3.1 is the latest framework of Microsoft to develop REST API.
Use cookie authentication without ASP.NET Core Identity (https://docs.microsoft.com/en-us/aspnet/core/security/authentication/cookie?view=aspnetcore-3.1.) is easy and quick. HttpOnly cookies will be used by default. Httponly flag is very important to avoid any XSS attack and has other benefits (https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies). Cookie solution relies on client's cookie support.

Another common authentication solution for API is to use JWT (JSON Web Token, https://jwt.io). https://jasonwatmore.com/post/2019/10/11/aspnet-core-3-jwt-authentication-tutorial-with-example-api. In start.cs:

        // This method gets called by the runtime. Use this method to add services to the container.
        public void ConfigureServices(IServiceCollection services)
        {
            if (Settings.Cors?.Length > 0)
            {
                services.AddCors(options =>
                {
                    options.AddPolicy("platform",
                    builder =>
                    {
                        builder.WithOrigins(Settings.Cors)
                            // Support https://*.domain.com
                            .SetIsOriginAllowedToAllowWildcardSubdomains()

                            // https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Credentials
                            // https://stackoverflow.com/questions/24687313/what-exactly-does-the-access-control-allow-credentials-header-do
                            // JWT is not a cookie solution, disable it without allow credential
                            // .AllowCredentials()
                            .DisallowCredentials()

                            // https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers
                            // Without it will popup error: Request header field content-type is not allowed by Access-Control-Allow-Headers in preflight response
                            .AllowAnyHeader()

                            // Web Verbs like GET, POST, default enabled
                            .AllowAnyMethod();
                    });
                });
            }

            // Configue compression
            // https://gunnarpeipman.com/aspnet-core-compress-gzip-brotli-content-encoding/
            services.Configure<BrotliCompressionProviderOptions>(options =>
            {
                options.Level = CompressionLevel.Optimal;
            });

            services.AddResponseCompression(options =>
            {
                options.EnableForHttps = true;
                options.Providers.Add<BrotliCompressionProvider>();
            });

            // Configue JWT authentication
            // https://jwt.io/
            services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
                .AddJwtBearer(options =>
                {
                    // Is SSL only
                    options.RequireHttpsMetadata = Settings.SSL;

                    // Save token, True means tokens are cached in the server for validation
                    options.SaveToken = false;

                    // Token validation parameters
                    options.TokenValidationParameters = new TokenValidationParameters()
                    {
                        ValidateIssuerSigningKey = true,
                        IssuerSigningKey = new SymmetricSecurityKey(Encoding.ASCII.GetBytes(app.Configuration.SymmetricKey)),
                        ValidateIssuer = false,
                        ValidateAudience = false
                    };
                });
        }
        public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
            }

            // Enable HTTPS redirect
            if (Settings.SSL)
                app.UseHttpsRedirection();

            app.UseRouting();

            // Enable CORS (Cross-Origin Requests)
            // The call to UseCors must be placed after UseRouting, but before UseAuthorization
            if (Settings.Cors?.Length > 0)
            {
                app.UseCors("platform");
            }

            app.UseAuthentication();
            app.UseAuthorization();

            // Enable compression
            app.UseResponseCompression();

            app.UseEndpoints(endpoints =>
            {
                // Apply authentication by default
                endpoints.MapControllers().RequireAuthorization();
            });
        }

After the user log in successfully, add the codes below to generate the token:
        /// <summary>
        /// Login for authentication
        /// 登录授权
        /// </summary>
        /// <param name="model">Data model</param>
        /// <returns>Result</returns>
        [AllowAnonymous]
        [HttpPost("Login")]
        public async Task Login([FromBody]LoginModel model)
        {
            // Act
            var result = await Service.LoginAsync(model);

            if (result.OK)
            {
                // Logined user id
                var userId = result.Data.Get("token_user_id", 0);

                // User role
                var role = result.Data.Get("role", UserRole.User);

                // Token handler
                var tokenHandler = new JwtSecurityTokenHandler();

                // Key bytes
                var key = Encoding.ASCII.GetBytes(App.Configuration.SymmetricKey);

                // Token descriptor
                var tokenDescriptor = new SecurityTokenDescriptor
                {
                    Subject = new ClaimsIdentity(new Claim[]
                    {
                        new Claim(ClaimTypes.Name, userId.ToString()),
                        new Claim(ClaimTypes.Role, role.ToString().ToLower()),
                    }),
                    // Suggest to refresh it at 5 minutes interval, two times to update
                    Expires = DateTime.UtcNow.AddMinutes(12),
                    SigningCredentials = new SigningCredentials(new SymmetricSecurityKey(key), SecurityAlgorithms.HmacSha256)
                };

                // Hold the token value and then return to client
                var token = tokenHandler.CreateToken(tokenDescriptor);
                result.Data["authorization"] = tokenHandler.WriteToken(token);
            }

            // Output
            await ResultContentAsync(result);
        }

Because the token is stateless, Web APIs are always facing a replay attack, also known as playback attack. bearer.SaveToken = true means you could access it through await HttpContext.GetTokenAsync("access_token") for any outgoing request. Add necessary validation logic in the database side is helpful. A short-lived, strict authentication with rate-limiting policy token solution will make the project much stronger.