Forem: godotg

From Open Source to Exit: The Open-Source Decentralized Networking Library Officially Releases Its First Release

godotg — Tue, 24 Oct 2023 02:52:42 +0000

1. Background of zfoo

For security reasons, many industries do not need to disclose the protocol format of network communication protocols.

For example, the securities and finance industry involves financial transactions, and it is preferable to keep these transactions as private as possible, to avoid others from seeing them.

1. High Security Requirements: The financial industry deals with a significant amount of sensitive information, such as bank accounts and transaction records. To ensure data security, financial institutions typically use proprietary communication protocols developed in-house or provided by specific vendors to ensure security and prevent malicious attacks.

2. Fixed Partnerships: Financial institutions often have fixed business partners, such as other financial institutions and payment providers, and the communication requirements between these parties are stable. Therefore, using undisclosed protocol formats can improve the efficiency and stability of network communication.

3. Market Competition and Trade Secrets: The financial industry is highly competitive, and each institution wants to protect its business secrets, including trading strategies and algorithms. Public communication protocol formats may expose these trade secrets, so the financial industry tends to use undisclosed protocol formats.

Similarly, the gaming industry has strict security requirements for network communication protocols.

1. Copyright Protection: The gaming industry is highly creative and competitive. Game developers often use their own unique network communication protocols to protect their intellectual property and copyrights, preventing other developers from illegally copying or imitating their games. Therefore, undisclosed protocol formats can increase the difficulty of piracy prevention and copyright protection.

2. High Security Requirements: Games involve sensitive data like user information and in-game transactions. To safeguard player account security and prevent cheating, game developers typically use proprietary communication protocols developed in-house or provided by specific vendors to ensure security and prevent malicious attacks.

3. Game Customization: Different games have significant differences in functionality and gameplay. Game developers often need to customize network communication protocols to meet the specific requirements of their games. Therefore, using undisclosed protocol formats can better achieve game customization and personalization.

Performance considerations: Many industries have high network performance requirements.

The financial industry's network communication protocols require high performance because the speed of transactions and data transmission is crucial in financial markets. In the financial industry, every second of transaction speed and real-time data delivery latency is of paramount importance to traders and investors.

1. The performance of network communication protocols directly affects the speed of trade execution and the real-time nature of data. Higher performance means faster transmission speed and lower latency, enabling more timely execution of trades and receipt of market data.

2. Fast trade execution speed is particularly important for high-frequency traders. High-frequency trading involves executing a large number of trades quickly to profit from small price movements. This strategy requires extremely fast trade execution, and the performance of communication protocols directly impacts efficiency and profits.

3. Additionally, the financial industry has high requirements for real-time data transmission. Traders and investors need timely access to market data, including stock prices, futures quotes, economic indicators, and more. Poor protocol performance can lead to delayed data transmission, affecting trading decisions and investment outcomes.

The gaming industry's network communication protocols require high performance because online games demand low latency, fast data transmission, and stable connections.

1. Games require low-latency network communication protocols to ensure that players' in-game actions are immediately relayed to the server and the real-time updates of other players' states. Lower latency allows players' actions to be perceived by the server and other players more quickly, enhancing gameplay and realism.

2. Games require fast data transmission, especially in large-scale multiplayer online games or games that involve a significant amount of real-time data transmission (such as shooting games). High-speed data transmission ensures a smooth gaming experience, avoiding lag or high latency issues.

3. Online games necessitate stable connections to prevent players from encountering disconnections or network issues during gameplay. A stable connection is fundamental to the gaming experience, as interruptions or connectivity problems can significantly impact the player's experience.

How to view the popularity of "Genshin Impact" private servers?

2. Introduction to zfoo

A communication network library with decentralized protocols, allowing everyone to have a unique protocol
zfoo is an application-layer network protocol that allows for custom protocol rules while balancing compatibility and performance.

Already submitted to awesome godot, currently, there is no backend framework with better support for GdScript than zfoo.

3. Why did zfoo go open source and then seemingly disappear?

This year, the focus is primarily on researching AI technology and has led to new considerations.
Technically, zfoo has already reached a stable state with no significant updates needed.
There will mainly be maintenance rather than major updates, and this is not a strict "disappearance." The focus is shifting towards game development using the Godot engine.
Open-source projects involve more than one developer, and more responsibilities are being delegated to other developers.

fast framework for binary serialization and deserialization in Java, and has the fewest serialization bytes

godotg — Thu, 16 Feb 2023 09:12:24 +0000

Ⅰ. Introduction

zfoo protocol It is currently the fastest framework for binary serialization and deserialization in Java, and has the fewest serialization bytes
The protocol is currently natively supported C++ Java Javascript C# Go Lua GDScript，It's easy to do cross-platform
The protocol can customize the private protocol format to make your protocol more secure, and supports adding fields and being compatible with previous and subsequent protocols
Compatible with Protobuf, it supports the generation of Protobuf protocol files, and provides a generation method from POJO to PROTO
There are already quite a few JetBrains plugins that can do it in the way to generate POJOs, and you can search for Proto in the plugin of Idea

Ⅱ. Quick use

Environmental requirements JDK 11+
Protocol is an independent project, does not depend on other projects, can be directly opened, locally installed to
its own local maven repository, can be used alone

// The zfoo protocol is registered and can only be initialized once
ProtocolManager.initProtocol(Set.of(ComplexObject.class, ObjectA.class, ObjectB.class));

// serialization
ProtocolManager.write(byteBuf, complexObject);

// deserialization
var packet = ProtocolManager.read(buffer);

Ⅲ. Performance testing

Single-threaded environment, 50% faster than Protobuf and 100% faster than Kryo without any JVM parameter tuning,参见性能测试
Thread safety, the performance of zfoo and Protobuf is not affected in any way, kryo will lose some performance
because of thread insecurity,参见性能测试
Test the environment

system：win10
cpu： i9900k
内存：64g

Single-threaded test with abscissa as the number of objects serialized and deserialized and time spent in milliseconds on the ordinate

Ⅳ. Why fast

Use Javassist bytecode to enhance the dynamic generation of serialization and deserialization functions for sequential execution, and sequential functions can be easily JIT compiled to achieve extreme performance
Natively integrated with netty's high-performance Byte Buf
With primitive type collection, there is no boxing and unboxing, invalid GCs are avoided, and the performance is fast enough
Inherently thread-safe and lock-free; kryo forces each thread to have its own instance of Kryo, which is a heavy design, especially in scenarios with many threads
No reflections, no unsafe operation; The use of objenesis in Kryo resulted in a lot of unsafe, and warnings when running in Java 11
Flattening the call depth of the method stack, and there is no performance penalty for nesting data structures, such as List>>; Comparing kryo and protobuf data structure nesting results in a performance penalty
There is no risk of vulnerability injection, only bytecode enhancement will be performed during initialization, and no bytecode operations will be performed in the later stage

The data compression volume is small, and the compression volume is smaller than that of Kryo and Protobuf; Smaller than kryo because kryo needs to write the registration number of each object
Intelligent syntax, incorrect protocol definitions will fail to start the program and give an error warning
Improve development efficiency, fully support POJO development, very easy to use

Ⅴ. Why small

Lightweight implementation, core serialization and deserialization code of about a thousand lines
Optimized the zigzag and varint encoding algorithms for int and long, avoiding some redundant method calls and bit operations
The data compression volume is small, and the compression volume is smaller than that of Kryo and Protobuf; Smaller than kryo because kryo needs to write the registration number of each object
Intelligent syntax, incorrect protocol definitions will fail to start the program and give an error warning
Improve development efficiency, fully support POJO development, very easy to use

The current size of the serialized object is as follows:
Simple objects, zfoo package size 8, kryo package size 5, protobuf package size 8
Regular objects, ZFOO package size 430, KRYO package size 483, Protobuf package size 793
For complex objects, ZFOO package size 2216, KRYO package size 2528, and Protobuf package size 5091

Ⅵ. Data type

Default data format support eliminates the need for users to register
manually.参考类定义
- boolean，byte，short，int，long，float，double，char，String
- Boolean，Byte，Short，Integer，Long，Float，Double，Character，If it is null during serialization, a default value of 0 will be given (the default value of Character is Character.MIN VALUE)
- int[]，Integer[]，If it is null, it is parsed as an array of length 0
  - Native generic List, Set, Map, deserialization return type Hash Set, Array List, Hash Map, and null pointer safe (returns a collection of size 0)
  - List，You must specify a generic class if [1,1,null,1] is sent and [1,1,0,1] is received
  - List，If [obj,obj,null,obj] is sent, [obj,obj,null,obj] is received, that is, the reference type is null before serialization, and it is also null after serialization
Unsupported data formats, because ZFOO automatically recognizes unsupported types and gives error warnings, so users
don't have to care too much
- int[][]，Arrays above two dimensions, considering that not all languages support multidimensional arrays
- List[]，Map[]，The Java language itself does not support generic class arrays
- List，Map，Generics are set in arrays, which looks strange and has few actual uses
- Enumerated classes, considering that many other languages do not support enumerated classes, can be replaced at the code level with int or string
- Custom generic class XXX Class, generic classes are prone to performance and parsing problems in many frameworks, and are not supported in all languages
- Circular references, although the underlying support circular references, but considering that circular references bring semantically difficult to understand and prone to errors, so they are blocked

Ⅶ. Protocol specifications

The protocol class must be a simple javabean, not inheriting from any other class, but can inherit an interface
In order to prevent objects in the code and avoid some potential concurrency problems caused by the mixing of objects
in the protocol layer and the po layer, zfoo mandates that protocol classes must implement the IPacket interface

Now the interface of IPacket is just an identification interface, inheriting the design of IPacket is mainly to make 
the code more elegant and easier to understand, and it is not a lot of work to inherit only Object

The design of inheriting IPacket also has cross-language considerations, which greatly simplifies the difficulty 
of serialization and deserialization in other languages and unifies the code implementation of other languages

The protocol number is defined as a short type to reduce the packet size and memory size, a packet can be reduced by 2 bytes, and the application memory of each protocol can also be reduced by 6 byte(protocols + IProtocolRegistration + protocolIdMap)

It is difficult for a project's protocol body class to exceed 3 w, and there will be tools that automatically package 
your protocol number a little more compactly, so that your protocol number will not exceed 3 w

There are two ways to indicate that the protocol class must be labeled with the protocol number

The first uses annotations: @protocol(id = protocolId)

  @Protocol(id = 104)
  public class SimpleObject implements IPacket {

      public int c;
      public boolean g;

  }

The second uses static constants + interfaces: the value of this protocol number must be the same as the value returned by the IPacket interface, which can slightly improve performance

  public class SimpleObject implements IPacket {

      public static final short PROTOCOL_ID = 104;

      public int c;

      public boolean g;

      @Override
      public short protocolId() {
          return PROTOCOL_ID;
      }

  }

The third use: Register the agreement through Protocol Manager.initProtocolAuto() without writing the protocol number

  public class SimpleObject implements IPacket {

      public int c;

      public boolean g;

  }

If you add a field for version compatibility, you need to add a Compatible annotation, and the order needs to be

naturally increased, so as to ensure that the old and new protocols can be compatible with each other
In order to be compatible with versions and avoid modifying field names, default uses field names to read and write

in the natural order of strings (can also be customized), so it will cause exceptions in serialization
The official environment does not necessarily have to delete an unwanted field in order to be version compatible and

avoid reducing fields
Among the six principles of design patterns, the principle of opening and closing is open to expansion and closed to

modification. The design of the protocol should also adhere to this principle when it comes to functionality,

prioritizing the addition of new protocols over modifying existing ones

Ⅷ. Use Protobuf in zfoo

zfoo only provides protobuf in the way of pojo like jprotobuf, but it can generate proto files for clients to use
through pojo objects
jprotobuf you can directly use simple pojo objects without having to
understand proto file operations and syntax
Generate a proto file for the client to use through the pojo
object, Generate a proto configuration
You can also customize your own build method,
Use code custom generation to proto

Extremely fast enterprise server framework, can be used in RPC, web server framework, game server framework.

godotg — Thu, 16 Feb 2023 09:02:07 +0000

Why is zfoo protocol ?

The protocol currently supports C++ C# Go Java Javascript TypeScript GDScript Lua
Universal distributed server protocol implementation, there are implementations in other languages, which can easily implement RPC, servers, and microservices on other platforms
The fastest custom format framework for Java binary serialization, with the fewest serialized bytes, more secure and compatible
In the single-threaded benchmark test, serialization and deserialization are 50% faster than Protobuf and 100% faster than Kryo

Ⅰ. Introduction of zfoo🚩

Extremely fast, asynchronous, Actor design pattern, lock-free design, RPC framework based on Spring MVC
Fastest serialization and deserialization，use zfoo protocol as a network protocol
High scalability，Single server deployment, microservice deployment, cluster deployment, gateway deployment
Can be used as a game server framework or website server framework.

Perfect work development process, complete online solution

Spring projects, distributed projects, container projects， hot update code without downtime hotswap
Excel json csv configuration is automatically mapped and parsed，Online hotswap configuration storage
Automapping framework for MongoDB orm
Event bus event
Time task scheduling scheduler
cpu, memory, hard disk, network monitoring built into the program no code and extra tools required monitor

Ⅱ. Who use this project

Projects with extremely high performance requirements, such as website and game server frameworks, single server, global server, live chat, IM system, real-time push
Projects such as saving, development, deployment, operation and maintenance costs
As a backend infrastructure for Godot，Unity，Cocos，Webgl，H5，Network protocol supports tcp udp websocket http
Keep it Simple and Stupid ，simple configuration, lightweight code

Ⅲ. Tutorials

FAQ，There are standard demo display and instructions in the test folder of each project directory,
which can be run directly
tank-game-server Online game《The Fight of Tanks》，novice friendly,
difficulty 2 stars
- Quick star, tank game entry tank.zfoo.net
- Quick star, tank background management tank-admin.zfoo.net
godot-bird bird and bird，powered by godot
cocos-fivechess five chess，use cocos as client

Ⅳ. Install and use⭐

1. Environmental Requirements and Installation

JDK 11+，OpenJDK or Oracle JDK ，maven install to local repository

If you do not have JDK 11+ installed, the quick installation method is to download directly 
from Project Structure, Platform Settings, SDKs in the upper right corner of Idea

2. protocol fastest serialization and deserialization

// zfoo protocol registration, can only be initialized once
ProtocolManager.initProtocol(Set.of(ComplexObject.class, ObjectA.class, ObjectB.class));

// serialization
ProtocolManager.write(byteBuf, complexObject);

// deserialization
var packet = ProtocolManager.read(byteBuf);

3. net fastest RPC framework, supports tcp udp websocket http

// Service provider, only need to add an annotation to the method, the interface will be automatically registered
@PacketReceiver
public void atUserInfoAsk(Session session, UserInfoAsk ask) {
}

// Consumers, synchronously requesting remote service, will block the current thread
var userInfoAsk = UserInfoAsk.valueOf(userId);
var answer = NetContext.getCosumer().syncAsk(userInfoAsk, UserInfoAnswer.class, userId).packet();

// Consumers, asynchronously requesting remote service, and will still execute logic in the current thread after the asynchronous
NetContext.getCosumer()
                    .asyncAsk(userInfoAsk, UserInfoAnswer.class, userId)
                    .whenComplete(sm -> {
                        // do something
                    );

4. hotswap hot update code, no need to stop the server, no additional configuration, just one line of code to start hot update

// Pass in the class file that needs to be updated
HotSwapUtils.hotswapClass(bytes);

5. orm automatic mapping framework based on mongodb，The secondary cache is designed using caffeine to fully release the database pressure

// You don't need to write sql and any configuration yourself, define a table in the database directly through annotation definitions
@EntityCache
public class UserEntity implements IEntity<Long> {
    @Id
    private long id;
    private String name;
}

// update database data
entityCaches.update(userEntity);

6. event use the observer design pattern, decouples different modules and improves the quality of the code

// To receive an event, you only need to add an annotation to the method and the method will be automatically listen for the event
@EventReceiver
public void onMyNoticeEvent(MyNoticeEvent event) {
    // do something
}

// fire an event
EventBus.submit(MyNoticeEvent.valueOf("My event"));

7. scheduler scheduling Framework Based on Cron Expression

@Scheduler(cron = "0/1 * * * * ?")
public void cronSchedulerPerSecond() {
    // do something
}
```

`

#### 8. [storage](storage/src/test/java/com/zfoo/storage/ApplicationTest.java) Excel to class automatic mapping framework, you only need to define a class corresponding to Excel, and directly parse Excel

```
@Resource
public class StudentResource {
    @Id
    private int id;
    @Index
    private String name;
    private int age;
}
```

Ⅴ. Commit specification👏
----------

- People who like this project are welcome to maintain this project together, and pay attention to the following
  specifications when submitting code
- The code formats uses the default formatting of IntelliJ Idea
- [conventional-changelog-metahub](https://github.com/pvdlg/conventional-changelog-metahub#commit-types)

Ⅵ. License
-----------

zfoo use [Apache License Version 2.0](http://www.apache.org/licenses/LICENSE-2.0)

![JetBrains Logo (Main) logo](https://resources.jetbrains.com/storage/products/company/brand/logos/jb_beam.svg)