Forem: lengjingzju

IMake: Reinventing Build Systems with Clarity and Efficiency for Native Linux Development

lengjingzju — Wed, 24 Dec 2025 04:19:59 +0000

Project Link : https://github.com/lengjingzju/cbuild-ng

Introduction: At the Crossroads of Build Systems

In the world of C/C++ engineering, the choice of build systems has long been a delicate balancing act. On one hand, "generator-style" build tools represented by CMake, with their powerful cross-platform abstraction capabilities, have nearly unified the build layer from desktop to mobile. On the other hand, many developers deeply involved in the Linux platform, especially in embedded systems, kernel, and foundational software, often feel a sense of alienation and efficiency loss within CMake's "black box magic." They yearn for a build experience more aligned with the Unix philosophy—"simple, clear, transparent."

The IMake/CBuild-ng project emerges precisely against this backdrop. It is not a simple improvement or reinvention of CMake, but rather a return to fundamental principles coupled with a technical redesign. It accurately targets the core pain points of CMake in deep Linux development scenarios, proposing and implementing a fundamentally different solution: not as an abstractor for all platforms, but as a deep cultivator within the Linux domain.

This article will delve into IMake's design philosophy, architectural innovations, and present an objective comparison through a detailed multi-dimensional table, showcasing the real strengths and weaknesses of IMake versus CMake in different scenarios. The goal is to provide engineers facing technology selection with a clear and comprehensive reference.

Part 1: The Design Philosophy and Core Architecture of IMake

1.1 Philosophy: Returning to Transparency, Declarativeness, and Control

IMake Architecture (Declarative, Layered)
├── Core Layer: 3 Declarative Build Functions
│   ├── add-liba-build    # Static Library
│   ├── add-libso-build   # Shared Library
│   └── add-bin-build     # Executable
├── Module Layer: INC_MAKES System (Loaded on Demand)
│   ├── inc.env.mk        # Environment Detection
│   ├── inc.conf.mk       # Configuration Management
│   ├── inc.app.mk        # Application Building
│   ├── inc.mod.mk        # Kernel Module Building
│   └── inc.ins.mk        # Installation Rules
└── Ecosystem Layer: Integration with CBuild-ng
    ├── Unified Package Management
    ├── Multi-Build System Support
    ├── Multi-level Caching Acceleration
    └── Enterprise Compliance Features

IMake's core philosophy can be summarized in three points:

Transparency First: The build process should be completely visible and understandable. Developers should be able to see and control every single parameter passed to the compiler directly, not converse with a generator.
Declarative over Imperative: Developers should describe "what needs to be built" (e.g., a shared library named libfoo), rather than write a series of instructions for "how to build it." This greatly simplifies build description files.
Environment-Driven, Not Configuration-Driven: Build modes (e.g., native/cross, GCC/Clang) should be switched instantly via environment variables, without needing a separate configuration phase, enabling seamless development workflows.

1.2 Architecture: The Synergy of the CBuild-ng Platform and the IMake Framework

The core architecture is "one concept, two layers of design," with clear division of labor. CBuild-ng serves as the integrated "platform layer," responsible for project management and dependency coordination; IMake serves as the focused "framework layer," providing core build logic.

CBuild-ng: As the upper-layer management and integration platform, it handles the software package lifecycle, including source acquisition (supporting Git, SVN, HTTP), multi-level caching, dependency resolution and arbitration, and unified scheduling of different underlying build systems (like IMake, CMake, Meson). It solves the engineering problems of "where from and where to."
IMake: As the core, Linux-oriented build framework, it provides a declarative Domain-Specific Language (DSL) based on GNU Make. It defines build targets with minimal syntax and has built-in deep support for complex scenarios common in Linux development (like cross-compilation, kernel modules). It solves the technical problem of "how to compile efficiently and clearly."

This architecture allows IMake-based projects to be embedded into existing projects as a standalone build tool or to gain enterprise-grade reusability and supply chain management capabilities through CBuild-ng.

If IMake is a sharp knife, CBuild-ng is the entire toolkit. It solves the full-chain problems from source code to finished product:

Unified Package Definition: Declares project metadata, dependencies, and build methods via the mk.deps file.
Multi-Build Engine Support: Can not only invoke IMake-based projects but also seamlessly manage traditional projects (CMake, Meson, Autotools, etc.), acting as an upper-level coordinator.
Enterprise-Grade Caching: Supports caching mechanisms for source code (original URL, domestic mirror, LAN mirror, local mirror) and build artifacts (LAN mirror, local mirror), providing a complete solution for development within physically isolated networks, common in sectors like defense and finance.
Software Supply Chain Management: Can generate Software Bill of Materials (SBOM), includes license checking, and can generate dependency graphs for visual management.

1.3 Systematic Package Management Solution

For dependency management in large-scale systems, IMake provides a unified management mechanism through collaboration with CBuild-ng:

Unified Dependency Declaration File: Each software package (whether a base library or an upper-layer application) declares its metadata and dependencies through an mk.deps file. For large systems, this forms a global dependency graph. CBuild-ng parses this graph to compute the build order and versions for all components.
Packaging and Deployment of Build Artifacts: CBuild-ng can package build artifacts (libraries, header files) into compressed archives, serving as deterministic dependency sources for other components, perfectly supporting offline development in physically isolated networks. It also supports packaging into the CPK format (essentially an installation script plus a compressed archive) for cross-platform deployment.
Active Provisioning, Not Passive Searching: When building component A, CBuild-ng will actively prepare the precise versions of its declared dependencies, libraries B and C (from cache or source build), based on its mk.deps file. Developers don't need to worry about where dependencies are located; the platform guarantees the provided versions are correct.

The core approaches of CMake and IMake (passive searching vs. active management) are fundamentally different, as compared below:

Management Dimension	CMake (as a build tool)	CBuild-ng (as a build management platform)
Core Philosophy	"Find and Use": Search for existing dependencies in the current system or specified directories.	"Declare and Provide": Declare all dependencies and their precise versions at the top level, with the platform responsible for providing them.
Dependency Source	Dispersed and uncertain: System paths, `CMAKE_PREFIX_PATH`, external package managers like `vcpkg`/`conan`.	Centralized and deterministic: Uniquely and controllably sourced from internal repositories, cache, or source builds.
Version Control	Difficult: Subprojects may find inconsistent versions through different paths or package managers, easily leading to the "works on my machine" problem.	Strictly Locked: Declare precise versions or Git commit hashes in the top-level `mk.deps`, ensuring a completely consistent build environment across the entire system.
Cross-Project Consistency	No native support: Each CMake project manages dependencies independently, making it difficult to guarantee the entire system uses the same libraries.	Core Design Goal: Dependencies for all projects (whether IMake, CMake, or Autotools projects) are unifiedly parsed, built, and supplied by CBuild-ng.
Enterprise Support	Relies on combining external ecosystems; requires setting up private repositories, caches, and audit toolchains separately.	Built-in Enterprise Features: Multi-level caching, builds in physically isolated networks, Software Bill of Materials (SBOM) generation, etc., are available out-of-the-box.

Part 2: Analysis of IMake's Disruptive Features

2.1 Minimalist Declarative Build Scripts

IMake covers most build scenarios with three core functions, freeing developers from the burden of writing complex rules.

# A complete definition for a static library, shared library, and executable
# INC_MAKES can be set to a combination of `disenv` `conf` `app` `mod` `disins`.
#   disenv: Excluding inc.env.mk
#   conf  : Including inc.conf.mk
#   app   : Including inc.app.mk
#   mod   : Including inc.mod.mk
#   disins: Excluding inc.ins.mk
INC_MAKES      := app
include inc.makes

SRCS = $(wildcard src/*.c)

# Declarative Build: One-line definitions, automatic handling of compilation, linking, dependency tracking
$(eval $(call add-liba-build,libmymath.a,$(SRCS)))              # Static Library
$(eval $(call add-libso-build,libmymath.so 1 0 0,$(SRCS)))      # Versioned Shared Library
$(eval $(call add-bin-build,calculator,app/main.c,-lmymath))    # Executable

The advantages of this design are immediately apparent:

Minimalist: Work that traditionally requires dozens of lines in a Makefile or CMake often takes single-digit lines in IMake.
Transparent: Executing make PREAT= MFLAG= shows the real commands passed to GCC/Clang—no magic.
Intelligent: Version numbers can be automatically extracted from source headers; dependency relationships are automatically deduced.
Modular: Allows developers to load features on demand (environment, configuration, installation, application compilation, driver compilation), keeping the build environment clean and efficient.

A real-world engineering example:

# Dependency declaration in mk.deps file
#DEPS() jmixer(native psysroot): ljcore ljson jacodec jaalgo

# Compilation logic defined in the Makefile
PACKAGE_NAME    = jmixer
SEARCH_HDRS    := ljcore

INSTALL_HEADERS:= $(wildcard src/*.hpp)

CPFLAGS        += -Isrc

.PHONY: all clean install
all:
    @echo "Build $(PACKAGE_NAME) Done!"

INC_MAKES      := app
object_byte_size= 65536
frame_byte_size = 16384
ENV_BUILD_TYPE := release
include inc.makes

staticlib      := libjmixer.a
sharedlib      := libjmixer.so $(call get_version,src/audio_control.hpp,JMIXER_VERSION, )
libsrcs        := $(wildcard src/*.cpp)
$(call set_flags,CFLAGS,src/audio_message.cpp,-Wno-missing-field-initializers)
LLIBS          := $(addprefix -l,ljcore ljson g711 g722 bcg729 opus fdk-aac)
$(eval $(call add-liba-build,$(staticlib),$(libsrcs)))
$(eval $(call add-libso-build,$(sharedlib),$(libsrcs),$(LLIBS)))

server_srcs    := test/jmixer_server.cpp
server_libs    := $(addprefix -l,ljcore ljson $(PACKAGE_NAME))
$(eval $(call add-bin-build,jmixer_server,$(server_srcs),$(server_libs),,$(OBJ_PREFIX)/lib$(PACKAGE_NAME).so))

client_srcs    := test/jmixer_client.cpp
client_libs    := $(LLIBS) -l$(PACKAGE_NAME)
$(eval $(call add-bin-build,jmixer_client,$(client_srcs),$(client_libs),,$(OBJ_PREFIX)/lib$(PACKAGE_NAME).so))

command_srcs   := test/jmixer_command.cpp
command_libs   := $(addprefix -l,ljcore ljson $(PACKAGE_NAME))
$(eval $(call add-bin-build,jmixer_command,$(command_srcs),$(command_libs),,$(OBJ_PREFIX)/lib$(PACKAGE_NAME).so))

all: $(BIN_TARGETS) $(LIB_TARGETS)

clean: clean_objs
    @rm -f $(LIB_TARGETS) $(BIN_TARGETS)
    @echo "Clean $(PACKAGE_NAME) Done."

install: install_hdrs install_libs install_bins
    @echo "Install $(PACKAGE_NAME) to $(INS_PREFIX) Done."

2.2 Elegant Implementation of Six Build Modes

IMake precisely defines six build modes covering the entire Linux development workflow. Their switching method intuitively demonstrates the advantage of being environment-variable-driven:

# 【Mode 1 & 2】Classic Native Development
make                                           # Native GCC
make CC_TOOL=clang                             # Native Clang

# 【Mode 3 & 4】Classic Cross-Compilation (Core of Embedded Development)
make CROSS_COMPILE=aarch64-linux-gnu-          # ARM64 GCC Cross-Compilation
make CROSS_COMPILE=aarch64-linux-gnu- CC_TOOL=clang USING_CLANG_CXX_BUILD=y # ARM64 Clang Cross-Compilation
# Note: CROSS_COMPILE with a path should use ENV_BUILD_TOOL=/<toolchain path>/aarch64-linux-gnu-

# 【Mode 5 & 6】Deep Integration with Yocto Build System
# Seamlessly integrated within Yocto recipes via environment variables
export ENV_BUILD_MODE=yocto
# Export whether it's cross or native compilation
NATIVE_BUILD:class-target = "n"
NATIVE_BUILD:class-native = "y"
export NATIVE_BUILD
# Invoke the build
do_compile() {
    oe_runmake  # Internally adapts to the above modes automatically
}

The essence of these six modes lies in "zero-cost switching". Developers do not need to modify any build files or execute time-consuming reconfiguration (cmake -B build). Seamless switching between development, testing, and production environments is achieved solely through environment variables, which is a huge efficiency boost for continuous integration and heterogeneous team collaboration.

2.3 Deep Integration with the Linux Ecosystem

Kernel-style Interactive Configuration: IMake introduces the Kconfig system (kconfig, modified based on linux-6.18, adds configuration output and source separation features), which originates from the Linux kernel. Developers can use the familiar make menuconfig command to manage project configurations (e.g., CONFIG_FEATURE_X=y) in a terminal graphical interface. The generated .config file is clear, readable, and easy to manage with version control.
Kernel Module Build Support: An innovation of IMake (driver compilation output and source separation) has been partially adopted by the Linux kernel community. The framework for separate compilation of driver code and kernel source tree allows kernel module development to enjoy an independent build experience similar to user-space programs, significantly lowering the barrier to entry.

Part 3: Objective Dimensional Comparison – IMake vs. CMake

The following comparison table aims to objectively present the capabilities of both in different dimensions. Five stars (★★★★★) indicate excellent or highly advantageous performance in that dimension; one star (★) indicates a significant shortcoming or that it is not a design goal.

Dimension	Evaluation Criteria	IMake	CMake	Advantage	Detailed Explanation
Core Positioning	Target Domain & Philosophy	Deeply optimized for Linux native development Transparent, controllable, declarative	Cross-platform abstraction and unification "Write once, run anywhere"	Divergence	IMake chooses to go deep and thorough in the Linux domain, sacrificing generality for ultimate experience. CMake pursues the broadest platform coverage, its abstraction being the foundation of generality.
Build Model & Performance	Build Process & Speed	Direct Build Model No intermediate generation; incremental builds are precise and fast ★★★★★	Generator Model First `configure` to generate, then `build` ★★★☆☆	IMake	IMake calls the compiler directly, with no configuration phase overhead, especially excelling at fast incremental compilation for large projects. CMake's generation phase becomes a performance bottleneck when projects are complex or configurations change.
Cross-Compilation Experience	Ease of Use & Integration	Environment Variable Driven `CROSS_COMPILE=arm-` to switch ★★★★★	Independent Toolchain File Requires writing/finding `toolchain.cmake` and reconfiguring ★★☆☆☆	IMake	IMake's approach is completely consistent with the workflow of Linux developers (e.g., kernel compilation), with zero learning cost and extremely simple integration into scripts and CI. CMake's approach is more formal but more cumbersome.
Configuration Management	Complexity & Interactivity	Kconfig System Integration `make menuconfig` graphical interface ★★★★★	CMake Variables & `option()` Imperative setting, no standard interactive interface ★★★☆☆	IMake	IMake provides an industry-standard interactive interface for managing complex projects with numerous optional features (e.g., media frameworks). CMake's approach becomes difficult to maintain as features multiply.
Code Conciseness	Build Description File Length	Minimalist Declarative Typically 20-50 lines to build a library ★★★★★	Imperative Scripting Same functionality typically requires 80-150 lines ★★★☆☆	IMake	IMake's three core functions abstract common build patterns, drastically reducing boilerplate code and repetitive work.
Debugging & Transparency	Build Process Control	Fully Transparent `make PREAT= MFLAG=` outputs real compilation commands ★★★★★	Indirect Abstraction Need to trace commands within the generated build system ★★☆☆☆	IMake	When link errors occur or flags don't take effect, IMake allows developers to see the root cause directly. CMake's abstraction layer makes debugging build issues themselves difficult.
Linux Kernel Development	Support Depth	Deep Integration, Innovation Adopted Supports driver separation compilation ★★★★★	Virtually No Native Support Requires additional maintenance of Kbuild scripts ★☆☆☆☆	IMake	This is IMake's killer feature. For kernel module developers, it provides the only modern, integrated build experience.
Security & Compliance	Built-in Capabilities	One-Click Built-in Hardening `ENV_SECURITY=y` ★★★★☆	Depends on External Toolchain Integration Requires manual flag setting or finding packages ★★☆☆☆	IMake	IMake integrates common secure compilation options and static analysis tools into the framework, lowering the barrier to secure development. CMake relies more on its ecosystem.
Learning Curve	Onboarding Difficulty	Gentle Based on Makefile knowledge; core is just three functions ★★★★☆	Steep Requires mastering proprietary syntax, generator expressions, etc. ★★☆☆☆	IMake	Engineers familiar with Linux development can get started with IMake at almost zero cost. CMake is a world of its own, requiring dedicated time to learn.
Ecosystem Breadth	Third-Party Library Support	Focuses on Linux System Layer Covers kernel, embedded, foundational services ★★★☆☆	Extremely Broad Covers desktop, mobile, gaming, scientific computing, etc. ★★★★★	CMake	This is CMake's absolute strength. Almost any well-known C/C++ library provides CMake support. IMake's ecosystem is more vertical and cannot compare in general library support.
Cross-Platform Support	Non-Linux Platforms	Not a Primary Goal Can manage other build systems via CBuild-ng ★★☆☆☆	Core Strength Provides first-class support for Windows, macOS, iOS, Android ★★★★★	CMake	If your project must target Windows/macOS for desktop application distribution, CMake is currently the only mature choice.
Enterprise Features (provided by CBuild-ng)	Supply Chain, Cache, etc.	Native Integrated Platform Built-in multi-level cache, offline builds, SBOM generation ★★★★☆	Relies on External Ecosystem Combination Requires Conan/vcpkg, etc., to achieve ★★★☆☆	IMake	CBuild-ng packages source management, dependency resolution, build caching, and compliance checking into one tool, providing an out-of-the-box solution for enterprise scenarios like closed networks.

Part 4: Technology Selection Guide

4.1 When Should IMake/CBuild-ng Be Prioritized?

If your project aligns with the following characteristics, IMake will bring significant efficiency and experience improvements:

Pure Linux Project: Especially server backends, infrastructure software, high-performance computing projects with high demands for build performance and transparency.
Embedded Linux Product: Requires frequent cross-compilation between x86_64 development hosts and ARM/MIPS target boards. IMake's mode switching is revolutionary.
Linux Kernel Module or Driver Development: IMake is currently the only tool providing a modern, integrated build experience.
Strict Requirements for Build Security & Compliance: Need to conveniently enable a full set of secure compilation options or require reproducible builds within physically isolated networks.
Tired of CMake's Complexity and Black-Box Nature: Craving a build system with simpler scripts and easier-to-debug issues.

4.2 When Should CMake Continue to Be Used?

In the following scenarios, CMake remains the safer or only choice:

Strictly Cross-Platform Project: Must provide functionally identical binary distributions for Windows, macOS, and Linux, especially GUI applications.
Deep Integration with Specific IDE or Ecosystem: Team development workflow is tightly built around Visual Studio or Xcode, or the project heavily relies on frameworks like Unreal Engine that are deeply integrated with CMake.
Dependency on Many Third-Party Libraries That Only Support CMake: The project has a broad technology stack and does not want to bear the cost of porting or creating IMake build descriptions for these libraries.

4.3 Hybrid Architecture Strategy

For large, complex projects, a pragmatic layered strategy can be adopted:

Core Layer/Platform Layer: Use IMake. This part is often the platform-dependent core algorithms, engine, or system services, benefiting from IMake's ultimate performance and transparency.
Upper Application/UI Layer: Use CMake. This part might be cross-platform clients or tools, leveraging CMake's rich UI framework (e.g., Qt) support and mature packaging toolchains.
Unified Management with CBuild-ng: Manage CMake projects also as "packages" within CBuild-ng, achieving top-level unification of dependencies and build processes.

Part 5: Future Outlook and Conclusion

The emergence of IMake/CBuild-ng signifies an important differentiation and reflection within the domain of build systems. It proves that alongside the mainstream path of "generality," there exists a path towards "expertise" and "excellent experience."

Its future lies not in replacing CMake, but in becoming the benchmark tool for core domains like Linux system programming, embedded development, and infrastructure software. With the rise of cloud-native and edge computing, the demand for efficient, controllable, and reproducible build processes will only grow stronger, which is precisely the strength of IMake/CBuild-ng.

Conclusion: In an era of increasing software engineering complexity, IMake is like an invitation to re-examine tools taken for granted. It reminds us that sometimes the best way forward is to return to the roots—pursuing clarity, transparency, and complete mastery over the technology. For any engineer dedicated to building solid and reliable software in the Linux world, understanding and seriously evaluating IMake/CBuild-ng is a highly valuable technical investment.

GB/s Level Editable DOM JSON Engine: The Architectural Philosophy Behind LJSON

lengjingzju — Tue, 16 Sep 2025 04:57:13 +0000

Project Address: https://github.com/lengjingzju/json

Introduction: Redefining High-Performance JSON Processing

In the realm of C language JSON libraries, the performance race never stops. Born in October 2019, a full year earlier than yyjson, LJSON was designed not merely to pursue extreme parsing speed in benchmarks, but to find the optimal balance between performance, memory usage, editability, streaming capability, and maintainability.

This fundamental difference in design philosophy can be perfectly illustrated by the analogy of Space Shuttle (LJSON) versus Rocket (yyjson/simdjson).

The current state of LJSON is remarkable:

Performance that matches or even surpasses yyjson in most scenarios
Provides true streaming processing capability that yyjson's architecture cannot achieve
Maintains code readability and extensibility, suitable for long-term evolution

This article delves into the architectural philosophy, performance optimization strategies of LJSON, and its fundamental differences with yyjson, revealing the design wisdom behind a versatile JSON engine.

I. Core Capabilities: Dual-Engine Drive

All-Purpose JSON Processing

Full support for JSON5 specification (hexadecimal, comments, trailing commas, single quotes, etc.) (some features require manual enabling)
DOM / SAX dual parsing modes
True streaming file processing (parse while reading, print while writing)

LJSON Reuse Mode(editable DOM) Parsing Tested 1000 Times (2.3 GB/s) (Test Platform: PC | CPU: Intel i7-1260P | OS: Ubuntu 20.04 (VMWare))

SAX Stream Mode, The memory usage of LJSON is constant (17328 Bytes)

Performance Comparison of Big Json File

Test Platform: PC | CPU: Intel i7-1260P | OS: Debian 13 (VMWare) | LJSON_v5.0.2

High-Performance Numeric Conversion Engine

Innovative ldouble algorithm
Float ↔ string conversion performance far exceeds standard library algorithms and mainstream algorithms (sprintf, grisu2, dragonbox)
Precision set to 16 decimal digits, boundary handling pursues the shortest representation rather than mechanical even rounding

Linux: Ubuntu 20.04 (VMware), Intel i7-1260P, GCC 9.4.0
Windows: Windows 11, Intel i7-1260P, MSVC 2023
dtoa test method: ./jnum_test 10000000(10 million conversions)
atod test method: ./jnum_test a 10000000(10 million conversions)

II. Architectural Design: Multi-Mode Adaptation for Full Scenarios

LJSON provides 7 parsing modes and 4 printing modes, forming a complete solution system:

Parsing Modes

DOM Classic Mode (malloc/free) – Basic choice for general scenarios
DOM Memory Pool Mode (unified application and release of large memory blocks) – Reduces memory fragmentation, improves allocation efficiency
DOM Reuse Mode (editable, in-place string reuse) – Maximizes memory utilization
DOM File Stream Mode (true streaming) – Powerful tool for large file processing
DOM File Stream Memory Pool Mode (true streaming + memory pool) – Combination of streaming processing and memory management
SAX Mode (callback processing) – Efficient event-driven parsing
SAX File Stream Mode (true streaming + callback) – Ultimate solution for large file event processing

Printing Modes

DOM → String – Standard serialization output
DOM → File (true streaming) – Large JSON file generation
SAX → String – Efficient conversion from event stream to string
SAX → File (true streaming) – Efficient output from event stream to file

True Streaming: Parse while reading the file, print while writing the file, without needing to read the entire file or generate large intermediate buffers. Memory usage can be reduced to a constant level, requiring only KB-level memory even when processing 1GB JSON files.

yyjson ≈ LJSON's DOM Reuse Mode + Aggressive Read-Only Optimization

III. The Balanced Approach to High-Performance Optimization in LJSON: Philosophical Thinking and Engineering Practice

This chapter does not cover architecture-specific optimizations (e.g., assembly, SIMD, etc.), nor system-wide optimizations (e.g., multithreading, asynchronous notifications, etc.).

Introduction: A Deep Dialogue with the Essence of Computing

Performance optimization is neither pure technical stacking nor abstract metaphysical speculation, but a profound dialogue with the essence of computing. It requires developers to find the art of dynamic balance between limited resources and infinite possibilities, between time and space, between ideal and reality. The optimization practice of the LJSON project is based on three cornerstones: "respecting the essence of data, embracing hardware characteristics, and simplifying hot-path decisions," organically integrating philosophical thinking with engineering feasibility.

True performance breakthroughs stem from insights into the nature of data, compliance with hardware characteristics, and respect for engineering laws. Through innovative architectural design and algorithm optimization, LJSON has achieved breakthrough performance improvements in the field of JSON processing, all built on a profound engineering philosophy.

1. Algorithms and Frameworks: The Wise Balance of Multi-Dimensional Parsing Modes

Philosophical Thinking

The ultimate boundary of all optimization is defined by algorithms and frameworks. This is about choice – finding the path that best fits the constraints among infinite solution spaces; it is also about constraints – acknowledging the finiteness of resources and building order under this premise. The foundation of high performance begins with insight into the essence of the problem, not with patching up minor details. An elegant algorithm is built on a deep abstraction of the problem; a powerful framework stems from successful encapsulation of complexity.

Excellent developers know how to identify, among the infinite possibilities of the problem space, the solution that best matches hardware characteristics, data features, and business requirements, seeking the optimal interpretation in the three-dimensional space formed by "problem scale—time complexity—space complexity." This choice is not only a technical decision but also a deep insight into the essence of the problem.

LJSON's design philosophy is to provide rich functionality while ensuring that each scenario can match the optimal parsing path.

Practical Principles

The LJSON framework provides up to 7 parsing modes and 4 printing modes, forming a complete solution system. This multi-mode architecture embodies the engineering philosophy of "no absolute optimum, only the most suitable," enabling developers to choose the most appropriate processing strategy based on specific scenarios.

2. Lookup Table Method: Extreme Optimization in Limited Domains

Philosophical Thinking

The lookup table method embodies the basic philosophy of time-space conversion: trading spatial determinism for temporal uncertainty. By precomputing and storing possible results, complex runtime calculations are converted into efficient memory access. Its essence lies in identifying operations with limited input domains but high computational costs, trading space for time to achieve performance leaps.

Practical Principles

LJSON applies lookup table optimization in multiple critical paths:

Character type judgment: Use compact lookup tables to quickly identify character types (e.g., whitespace, digits, etc.), avoiding expensive branches
Escape character processing: Precompute escape sequences and special character mappings to accelerate string serialization and deserialization
Number parsing and serialization: Design dedicated lookup tables for floating-point, large number, division, and other operations, reducing real-time computation

These optimizations are strictly limited to finite domains, ensuring that the table structures are compact and cache-friendly. LJSON's lookup table strategy always adheres to "small and refined," with each table carefully designed to run efficiently in L1/L2 cache.

3. Reducing Data Copying: Maintaining Order and Reducing Entropy

Philosophical Thinking

Unnecessary copying is an "entropy increase" process in software systems, leading to resource waste and energy inefficiency. The way to optimize lies in practicing entropy reduction – by passing references (essence) rather than copies (appearance), maintaining the simplicity and order of the system. Cherish the form and flow of data, avoiding unnecessary movement and replication. Zero-copy is the ideal embodiment of this philosophy, allowing data to flow naturally along its inherent track, like a river rushing along its channel.

Practical Principles

LJSON minimizes data copying through multiple techniques:

In-place string reuse: In DOM reuse mode, string data is directly reused in the original buffer, avoiding copying
Direct buffer operation: Parsing/printing results are generated directly in the target buffer, eliminating intermediate steps
Pointer optimization strategy: Use pointers instead of value copies for large structures, reducing memory movement overhead
Zero-copy parsing: Directly reference input data in supported scenarios without copying

These measures together form the core of LJSON's efficient memory management, ensuring minimal data movement overhead in various scenarios.

4. Information Recording Optimization: Precomputation and Intelligent Prediction

Philosophical Thinking

The wisdom of optimization lies not in avoiding computation, but in avoiding repeated computation. LJSON, through a fine-grained information recording mechanism, saves the results of a single computation and uses them multiple times, reflecting a deep respect for computational resources.

Practical Principles

LJSON implements information recording optimization at multiple levels:

String metadata recording:
- Record string length to avoid repeated strlen execution
- Record escape status to optimize serialization performance
- Cache hash values to accelerate dictionary lookup
Resource precomputation:
- Preallocate memory resources based on file size
- Allocate sufficient buffers in advance to avoid frequent malloc
State persistence:
- Record the state of storage buffers for parsing and printing
- Achieve zero heap allocation mode through buffer reuse, eliminating runtime memory allocation

This strategy enables LJSON to achieve deterministic performance in most scenarios, avoiding the uncertainty brought by runtime computation.

5. Batch Operations: Transforming Fragmentation into Integrated Performance Improvement

Philosophical Thinking

The bottleneck of modern systems often stems from the overhead of processing massive fragmented operations. Batch operations are a rebellion against fragmentation, advocating integration and batch processing, transforming random requests into smooth flows, following dissipative structure theory, using order to combat chaos.

Practical Principles

LJSON improves performance through multiple batch processing techniques:

Memory pool management:
- Provide optional memory pool allocators to replace frequent malloc/free, reduce fragmentation, and improve cache locality
- Estimate and allocate large blocks of memory at once, with fine-grained internal management
Buffered IO optimization:
- File stream mode adopts large-block reading strategy to reduce system call times
- When writing, buffer first and then write in batches to maximize IO throughput
Batch memory operations:
- Use memcpy/memmove instead of byte-by-byte copying
- Design code structures that facilitate compiler vectorization to leverage modern CPU advantages

These techniques enable LJSON to maintain excellent performance even when processing large-scale data.

6. Precision and Engineering Balance: Innovative Practice of the ldouble Algorithm

Philosophical Thinking

The obsession with absolute precision is the perfectionism of idealism, while engineering is essentially the art of trade-offs. In reality, what we often pursue is not the mathematically optimal solution, but a satisfactory solution under constraints – actively giving up negligible precision can exchange great savings in computation, bandwidth, and energy consumption.

Practical Principles

LJSON's self-developed ldouble algorithm embodies the fine balance between precision and performance:

Precision strategy:
- Adopt 16 decimal digits of precision, meeting the needs of most scenarios
- Boundary handling pursues the shortest representation, not mechanical even rounding
Performance optimization:
- Avoid expensive division, large number, and floating-point operations
- Optimize common number processing paths

This algorithm significantly improves performance while still ensuring sufficient precision and reliability.

7. Branch Prediction Optimization: Dancing with Uncertainty

Philosophical Thinking

The CPU pipeline craves determinism, and conditional branches are its challenge. Branch prediction optimization is humans injecting probability and patterns into hardware, teaching it to "guess," thus dancing with uncertainty. The likely/unlikely macros are "prior knowledge" passed by humans to the compiler, a prophecy about execution probability.

Practical Principles

LJSON adopts various strategies in branch prediction optimization:

Hot path priority: Place the most common conditions first to improve prediction accuracy
Branch elimination: Use bit operations and mathematical identities to replace branches
Data-driven: Convert conditional judgments into lookup table operations to reduce the number of branches
Loop optimization: Simplify loop conditions to improve prediction efficiency
Branch selection: Carefully choose between switch-case and if-else

These optimizations ensure that LJSON fully unleashes the potential of modern CPU pipelines on critical paths.

8. Cache Hit Optimization: The Art of Data Locality

Philosophical Thinking

Cache optimization is based on the application of the principles of temporal and spatial locality: temporal locality (revisit soon) and spatial locality (access nearby) are the keys to drive storage hierarchy. Optimizing cache hits is a carefully orchestrated art of data layout. It requires organizing information from the CPU's perspective – not the human perspective – shaping an efficient and hardware-compatible "spatial aesthetics."

Practical Principles

LJSON employs various cache optimization techniques:

Data structure optimization:
- Compress object size, e.g., optimize JSON Object from 48 bytes to 32 bytes (64-bit system)
- Reorder fields to reduce padding bytes and improve cache line utilization
Code layout optimization:
- Place hotspot code together to improve instruction cache efficiency
- Separate cold code into independent areas to reduce cache pollution
Loop optimization:
- Keep loop structures compact
- Optimize access patterns to enhance data locality
- Manually unroll loops for some hotspot functions
Memory access optimization:
- Prioritize sequential access to maximize cache line utilization
- Reduce random access to improve hardware prefetching results

These optimizations together ensure that LJSON achieves near-optimal performance at the memory access level.

9. Inlining and Function Reordering: The Spatiotemporal Weaving Art of Code

Philosophical Thinking

Inlining breaks the physical boundaries of functions, a spatial weaving art – weaving code segments into the calling context to eliminate call overhead. Function reordering (PGO) is a temporal weaving art – based on the actual profile, physically arranging frequently consecutively executed functions closely together.

Both point to one thing: the physical layout of code should work in concert with its execution context, achieving efficient spatiotemporal unity.

Practical Principles

Selective inlining: Only inline small hotspot functions to reduce call overhead
PGO optimization: Use profile feedback to optimize function layout
Modular inlining: Keep function responsibilities single to facilitate compiler decisions
Avoid over-inlining: Prevent code bloat and instruction cache pollution

IV. Design Philosophy Differences with yyjson

The core differences reflect different engineering philosophies:

Dimension	LJSON	yyjson
Mode Design	Multi-mode (incl. true streaming, editable reuse)	Single reuse mode (read-only optimized)
Editability	Reuse mode is editable	Read-only and editable strictly separated (val / mut_val)
String Storage	Standard C string (trailing 1 `\0`)	Non-standard trailing 4 `\0` (reduces boundary checks)
Object Storage	key and value stored together	key and value stored separately
Memory Strategy	On-demand allocation	Large block pre-allocation, high redundancy
Access Acceleration	`json_items_t` cache, array O(1), object O(logN)	Unknown if cached, possibly O(N)/O(2N)
Optimization Techniques	Inlining, branch prediction, lookup tables, caching metadata, hotspot loop unrolling; Maintains readability	Similar optimizations + extensive macro loop unrolling, obscure code
Streaming Processing	True streaming parsing/printing	Not supported
Code Readability	High (clear structure, ~8k lines of code)	Very low (macro-intensive, ~20k lines of code)

V. Performance Comparison and Real-World Scenario Performance

Optimization Technique Comparison

Both use:

Algorithm optimization
Memory pool (block) optimization
Inlining optimization (inline)
Branch prediction optimization (likely/unlikely)
Cache hit optimization
Lookup table optimization
Copy optimization
Information recording optimization (caching string length, etc.)

Differences:

yyjson: Extensive use of macros for manual loop unrolling, compact but obscure code, more aggressive performance; uses non-standard features (trailing 4 \0, unaligned access, etc.).
LJSON: Pursues an engineering balance of maintainability, extensibility, low footprint, and high performance. Maintains readability; only unrolls loops for hotspot functions.

Actual Performance

yyjson read-only reuse mode performance is slightly higher than LJSON, originating from a single pattern, aggressive resource utilization, non-standard features, and macro loop expansion
yyjson read-only mode performance is very close to LJSON, with each winning in some cases
yyjson editable mode performance may be significantly lower than LJSON (large file parsing: yyjson uses about twice the memory and is about half the speed of LJSON)

Note: If LJSON abandoned streaming, standard strings, and editability, and adopted more obscure code, it could achieve the performance of yyjson's read-only reuse mode, but that would no longer be LJSON's design philosophy.

VI. Comprehensive Multi-Dimensional Comparison

Comparison Dimension	LJSON	yyjson	RapidJSON	cJSON
Parsing Performance	High (editable reuse)	Medium-High (editable)	Relatively High	Medium
Printing Performance	High (true streaming + ldouble)	High (full buffering)	Medium	Low
Memory Usage	Can be constant-level (streaming)	Relatively High	Relatively High	Relatively High
CPU Feature Dependency	Low	Low	Medium	Low
Extensibility	High	Low	Medium	High
Code Readability	High	Very Low	Medium	High
Key Mechanisms	Zero heap allocation, memory pool reuse, in-place strings, json_items, ldouble	In-place reuse, memory blocks, non-standard string tails, loop unrolling	Templates+allocators	malloc/free

VII. Applicable Scenarios and Selection Suggestions

Large File Processing: GB-level JSON formatting/compression, memory usage can be constant-level → LJSON Streaming Mode
Embedded Systems: Low memory, high performance requirements → LJSON Memory Pool Mode
High-Frequency Read/Write: Real-time data stream parsing and generation → LJSON Editable Reuse Mode
Extreme Read-Only Parsing: Extreme performance requirements and no need for editing → yyjson Read-Only Mode
Cross-Platform Development: Linux / Windows / Embedded RTOS → LJSON (Pure C, zero dependencies)

Conclusion: The Balance of Performance and Engineering Art

LJSON's optimization practice reveals a profound engineering truth: high performance does not stem from the extreme pursuit of a single technology, but from the art of the best balance between multiple dimensions. Truly excellent system optimization requires finding harmony between the micro and macro, short-term and long-term, human and machine.

In algorithm selection, LJSON does not pursue theoretical optimal complexity but seeks the most practical solution in real scenarios; in memory management, it does not pursue absolute zero-copy but weighs copy cost against implementation complexity; in precision handling, it does not pursue mathematical perfection but balances precision and efficiency based on needs.

yyjson is a sharp scalpel, focusing on the extreme performance of read-only parsing; while LJSON is a complete toolbox, balancing performance, flexibility, and maintainability. yyjson's success lies in its deep polishing of a single mode, but its architecture is essentially just a specialized version within LJSON's multi-mode system.

The real value of LJSON lies in its ability to stably support diverse JSON processing needs in real engineering environments with extremely low memory usage and high performance. It tells us that the highest state of optimization is not to make a single indicator perfect, but to make the entire system achieve a harmonious and efficient state in the real environment.

When selecting, developers should weigh the actual scenario: is it the extreme speed of read-only parsing, or the architectural flexibility of general processing? Understanding the essential differences between the two is key to making truly engineered decisions.

CPK: A Philosophical Practice of Returning to the Essence of Software Packaging and Distribution

lengjingzju — Thu, 04 Sep 2025 04:44:52 +0000

Project Link: https://github.com/lengjingzju/cbuild-ng

In Linux’s complex distribution ecosystem, have we forgotten the pure form of software? As containers, sandboxes, and daemons become the norm, CPK—guided by a philosophy of simplicity, transparency, and freedom—launches a thought-provoking technical practice.

Note: CPK is a core component of CBuild-ng, but it can also be used independently.

Introduction: The Essence of Software Lost in “Progress”

The Linux world has never lacked innovation, especially in application distribution. We’ve moved from deb/rpm to Snap, Flatpak, and AppImage—each evolution aiming to solve dependency, isolation, and cross-platform compatibility issues. Yet, as these solutions stack abstraction upon abstraction, we’ve quietly lost something: system simplicity, complete knowability of software behavior, and direct control without intermediaries.

It feels like we’ve replaced one form of complexity with another. Is there a way to return to software’s most authentic state—a complete, self-contained digital artifact that can run anywhere?

The open-source build system CBuild-ng offers an answer: CPK (CBuild/Compressed Package). More than a technical format, it’s a philosophical practice rooted in Simplicity, Sovereignty, and Mobility.

1. Simplicity — Stripping Away All Unnecessary Complexity

CPK’s design philosophy is grounded in the classic Unix credo: KISS (Keep It Simple, Stupid). True universality, it holds, should be built on the most solid, ubiquitous foundations—not on yet another complex middleware layer.

What is CPK?

A self-extracting shell script followed by a standard tar archive.
Its only dependencies are Python, sh, file, tar, gcc, readelf, and patchelf. These tools exist on virtually every Linux machine (with patchelf needing a one-time static build). No extra frameworks or daemons (like snapd or flatpak) are required.

This radical minimalism delivers unmatched transparency. You can open a CPK in a text editor to inspect its install logic, or extract and audit its contents with tar. No black boxes, no hidden magic—everything is knowable, predictable, and auditable. CPK uses the most primitive tools to solve the most modern problems, offering an elegant rejection of over-engineering.

2. Sovereignty — Returning Full Control

CPK makes a bold statement: it restores full software sovereignty to both creators and users.

For developers, CPK enables delivery certainty.

They can choose any modern toolchain (e.g., the latest glibc from AlmaLinux 10), bundle the application with all dependencies (down to the C library), and freeze it into a known-good runtime. They can then confidently declare: “This is my work and everything it needs; it runs perfectly in this environment.” This ends the age-old “dependency hell” and “works on my machine” problem.
For users, CPK means complete control and transparency.

A .cpk file can be installed in the user’s home directory (~/.cpk/) without root privileges, leaving no stray files deep in the system. Users can inspect the install script, dissect the package contents, and decide when to run or update. No forced background services, no unsolicited auto-updates—just a trust relationship built on openness and respect.

3. Mobility — Creating Freely Migrating Digital Lifeforms

CPK’s ultimate vision is to make Linux applications truly portable digital lifeforms, able to cross the chasms between distributions.

Its core technique: it carries not only user-space dependencies but also the lowest-level C library (e.g., glibc) and interpreter (ld.so), and rewrites the binary’s interpreter path to form a perfect internal loop. This allows an application built on the latest distro to fully “revive” on any older host.

By decoupling applications from their environments, CPK achieves the long-sought Linux ideal of “build once, run anywhere”. Applications are freed from system constraints, becoming truly independent, migratable digital entities.

4. A Horizontal View — CPK’s Unique Position in the Ecosystem

CPK’s philosophy differs sharply from mainstream solutions. It doesn’t aim to replace them all, but instead strikes a unique balance between radical simplification, control, and cross-distro deployment.

Dimension	CPK (CBuild-ng)	Snap	Flatpak	AppImage	Traditional (deb/rpm)
Core Philosophy	The app is the package: a self-contained, portable environment with binary path isolation.	Sandboxed app container: security isolation, auto-updates, store-distributed.	Desktop app sandbox: portable, sandboxed desktop apps.	Single-file app: extreme simplicity—download, chmod, run.	System integration: software as part of the OS.
Package Format	Self-extracting shell script + tar archive; fully transparent.	Read-only SquashFS image; mounted at runtime.	OCI image + metadata; relatively complex.	ISO 9660 filesystem image; mountable.	Custom archive (deb = ar archive with control/data tarballs).
Dependency Handling	Strong isolation, bundles all libs (glibc, ld.so), modifies interpreter & rpath with `patchelf`.	Strong isolation, bundles deps, fixed runtime.	Strong isolation, shared runtime (e.g., Freedesktop SDK).	Optional isolation; may bundle or use system libs.	No isolation; relies on system libs.
Cross-Distro Support	✅ Perfect	✅ Perfect	✅ Perfect	✅ Perfect	❌ Poor
Installation	No install or user-space install; no root needed.	Requires `snapd` daemon; root needed.	User/system install; needs `flatpak` framework.	No install; run directly.	System install via package manager; root needed.
Performance Overhead	Very low (near-native).	Higher (namespaces, AppArmor, SquashFS).	Medium (bubblewrap, OCI runtime).	Low (tmpfs extraction).	Native.
Security	Relies on host security; no sandbox.	High (strict sandbox, AppArmor/seccomp).	High (bubblewrap sandbox).	Relies on host security; no sandbox.	Relies on system security.
Updates	Manual by developer.	Automatic via `snapd`.	Auto/manual via `flatpak`.	Manual (download new file).	System-wide via package manager.
Desktop Integration	Manual (scripts/tools).	Excellent (auto desktop entries).	Excellent.	Good (may need extra tools).	Excellent.
Core Dependencies	Minimal (common system tools).	Heavy (`snapd` daemon).	Medium (`flatpak` framework).	None.	None (built-in).

CPK’s Niche:

For developers and power users: Transparency (viewable scripts), control (self-managed deps), and minimal dependencies make it ideal for distributing complex toolchains, internal tools, CI/CD artifacts, or fixed-version commercial software.
For embedded and edge computing: Low overhead and no resident daemons are huge advantages in constrained or highly customized environments, enabling reliable deployment of complex runtimes to any Linux device.
For those who value simplicity and transparency: CPK rejects “magic.” Built from the most basic components, every step is auditable and debuggable—no sandbox, no forced store—standing in stark contrast to the heavier frameworks of Snap and Flatpak.

5. Hands-On — Experiencing the Elegance of CPK

Example: CPK Packaging Integrated with CBuild-ng

With the make xxx-cpk command, you can automatically compile the software and generate a .cpk package. The line ELFs copied from system: clearly shows which binaries were copied from the host system.

lengjing@lengjing:~/data/cbuild-ng$ source scripts/build.env host generic
lengjing@lengjing:~/data/cbuild-ng$ make loadconfig 
lengjing@lengjing:~/data/cbuild-ng$ make tcpdump-cpk
Match patchelf-native Cache.
Build patchelf-native Done.
Match patchelf Cache.
Build patchelf Done.
log path is /home/lengjing/data/cbuild-ng/output/x86_64-host/config/log/2025-09-04--08-15-58.068091488/
[100%]                                                  [  2/  2] tcpdump                                               
----------------------------------------
    libpcap
    tcpdump
----------------------------------------
Generate /home/lengjing/data/cbuild-ng/output/x86_64-host/packages/tcpdump/usr/share/license/index.txt OK.
Generate /home/lengjing/data/cbuild-ng/output/x86_64-host/packages/tcpdump/usr/share/license/index.html OK.
Note: tcpdump.dot tcpdump.svg and tcpdump.png are generated in the /home/lengjing/data/cbuild-ng/output/x86_64-host/packages/tcpdump/usr/share/license folder.
Interpreter path       : /home/lengjing/data/cbuild-ng/output/x86_64-host/packages/tcpdump/syslib/ld-linux-x86-64.so.2
ELFs with interpreter  : [('/home/lengjing/data/cbuild-ng/output/x86_64-host/packages/tcpdump/usr/bin', ['tcpdump']), ('/home/lengjing/data/cbuild-ng/output/x86_64-host/packages/tcpdump/syslib', ['libc-2.31.so', 'libpthread-2.31.so'])]
ELFs with rpath        : [('/home/lengjing/data/cbuild-ng/output/x86_64-host/packages/tcpdump/usr/bin', ['tcpdump']), ('/home/lengjing/data/cbuild-ng/output/x86_64-host/packages/tcpdump/usr/lib', ['libpcap.so.1.10.5']), ('/home/lengjing/data/cbuild-ng/output/x86_64-host/packages/tcpdump/syslib', ['libcrypto.so.1.1', 'libc-2.31.so', 'libdl-2.31.so', 'libpthread-2.31.so'])]
ELFs copied from system: ['libcrypto.so.1.1', 'libc.so.6', 'ld-linux-x86-64.so.2']
CPK SUCCESS: /home/lengjing/data/cbuild-ng/output/x86_64-host/packages/tcpdump
CPK is packed to /home/lengjing/data/cbuild-ng/output/x86_64-host/packages/tcpdump.cpk

CPK Packaging Command and Principles

%-cpk:
    @$(MAKE) $(MFLAG) CONFIG_PATCHELF_NATIVE=y patchelf-native
    @$(MAKE) $(MFLAG) CONFIG_PATCHELF=y patchelf
    @$(MAKE) $(MFLAG) $(patsubst %-cpk,%-pkg,$@)
    @PATH=$(ENV_NATIVE_ROOT)/objects/patchelf/image/usr/bin:$(PATH) \
        python3 $(ENV_TOOL_DIR)/gen_cpk_package.py -r $(ENV_CROSS_ROOT)/packages/$(patsubst %-cpk,%,$@) \
        -i include:share:etc:srv:com:var:run $(if $(PKG_EOS),-o $(PKG_EOS)) \
        -c $(ENV_BUILD_TOOL)gcc -t $(ENV_BUILD_TOOL)readelf $(if $(CPK_EXTRA_PATH),-e $(CPK_EXTRA_PATH))
    @cp -fp $(ENV_CROSS_ROOT)/objects/patchelf/image/usr/bin/patchelf $(ENV_CROSS_ROOT)/packages/$(patsubst %-cpk,%,$@)
ifneq ($(PKG_EOS),y)
    @cp -fp $(ENV_TOOL_DIR)/gen_cpk_package.py $(ENV_CROSS_ROOT)/packages/$(patsubst %-cpk,%,$@)
    @ush=$(ENV_CROSS_ROOT)/packages/$(patsubst %-cpk,%,$@)/update.sh && \
        echo '#!/bin/sh' > $${ush} && \
        echo 'curdir=$$(dirname $$(realpath $$0))' >> $${ush} && \
        echo 'PATH=$$curdir:$$PATH python3 $$curdir/gen_cpk_package.py -r $$curdir -i include:share:etc:srv:com:var:run' >> $${ush} && \
        chmod +x $${ush}
endif
    @bash $(ENV_TOOL_DIR)/gen_cpk_binary.sh pack $(ENV_CROSS_ROOT)/packages/$(patsubst %-cpk,%,$@)

This snippet from the top-level Makefile defines the CPK packaging target. It primarily uses the gen_cpk_package.py and gen_cpk_binary.sh scripts, which can also be run independently to package software outside of CBuild-ng.

Statically compile patchelf (note: some distro-provided versions may be problematic—choose an appropriate one).
Run gen_cpk_package.py (use -h for help):
- Analyze dynamically linked executables and libraries in the -r specified directory.
- Use file to get file attributes, then readelf (via -t) to inspect dependencies.
- Use gcc (via -c) to locate dependent libraries not in the package dir and copy them into syslib.
- Use patchelf to rewrite rpath and interpreter to point inside the package.
Run gen_cpk_binary.sh to compress the package directory and append it to the script, producing a single-file .cpk.

Example: Installing a CPK Package

Below is an example of running a CPK built on AlmaLinux 10 on Ubuntu 20.04. It demonstrates that a tcpdump binary (and all its complex dependencies) compiled against a newer glibc 2.41 can run seamlessly on a system with an older glibc 2.31. Note also the usr/share/license directory—index.html/index.txt list licenses, and the included diagrams visualize dependencies, supporting enterprise compliance.

lengjing@lengjing:~/Downloads$ ./tcpdump.cpk
Please set the installation directory (/home/lengjing/.cpk/tcpdump):
Delete the original app [/home/lengjing/.cpk/tcpdump] first? (y or n): y
-e Your choice is y
Interpreter path       : /home/lengjing/.cpk/tcpdump/syslib/ld-linux-x86-64.so.2
ELFs with interpreter  : [('/home/lengjing/.cpk/tcpdump/syslib', ['libc.so.6']), ('/home/lengjing/.cpk/tcpdump/usr/bin', ['tcpdump']), ('/home/lengjing/.cpk/tcpdump/syslib', [])]
ELFs with rpath        : [('/home/lengjing/.cpk/tcpdump/syslib', ['libc.so.6', 'libz.so.1.3.1.zlib-ng', 'libcrypto.so.3.2.4']), ('/home/lengjing/.cpk/tcpdump/usr/bin', ['tcpdump']), ('/home/lengjing/.cpk/tcpdump/usr/lib', ['libpcap.so.1.10.5']), ('/home/lengjing/.cpk/tcpdump/syslib', [])]
ELFs copied from system: []
CPK SUCCESS: /home/lengjing/.cpk/tcpdump
-e Successfully installed to /home/lengjing/.cpk/tcpdump
lengjing@lengjing:~/Downloads$ cd /home/lengjing/.cpk/tcpdump
lengjing@lengjing:~/.cpk/tcpdump$ tree
.
├── gen_cpk_package.py
├── patchelf
├── syslib
│     ├── ld-linux-x86-64.so.2
│     ├── libcrypto.so.3 -> libcrypto.so.3.2.4
│     ├── libcrypto.so.3.2.4
│     ├── libc.so.6
│     ├── libz.so.1 -> libz.so.1.3.1.zlib-ng
│     └── libz.so.1.3.1.zlib-ng
├── update.sh
└── usr
    ├── bin
    │     ├── pcap-config
    │     ├── tcpdump
    │     └── tcpdump.4.99.5 -> tcpdump
    ├── lib
    │     ├── libpcap.so -> libpcap.so.1
    │     ├── libpcap.so.1 -> libpcap.so.1.10.5
    │     └── libpcap.so.1.10.5
    └── share
        └── license
            ├── common
            │     └── MIT-LENGJING
            │         └── LICENSE
            ├── index.html
            ├── index.txt
            ├── libpcap
            │     └── LICENSE
            ├── spdx-licenses.html
            ├── tcpdump
            │     └── LICENSE
            ├── tcpdump.dot
            ├── tcpdump.png
            └── tcpdump.svg

10 directories, 24 files
lengjing@lengjing:~/.cpk/tcpdump$ file usr/bin/tcpdump
usr/bin/tcpdump: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /home/lengjing/.cpk/tcpdump/syslib/ld-linux-x86-64.so.2, BuildID[sha1]=ccfe1b6660bd62cd643fe2746464872074e651ee, for GNU/Linux 3.2.0, not stripped
lengjing@lengjing:~/.cpk/tcpdump$ ./syslib/libc.so.6
GNU C Library (GNU libc) stable release version 2.41.
Copyright (C) 2025 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 15.1.1 20250521 (Red Hat 15.1.1-2).
libc ABIs: UNIQUE IFUNC ABSOLUTE
Minimum supported kernel: 3.2.0
For bug reporting instructions, please see:
<https://www.gnu.org/software/libc/bugs.html>.
lengjing@lengjing:~/.cpk/tcpdump$ /usr/lib/x86_64-linux-gnu/libc.so.6
GNU C Library (Ubuntu GLIBC 2.31-0ubuntu9.18) stable release version 2.31.
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 9.4.0.
libc ABIs: UNIQUE IFUNC ABSOLUTE
For bug reporting instructions, please see:
<https://bugs.launchpad.net/ubuntu/+source/glibc/+bugs>.
lengjing@lengjing:~/.cpk/tcpdump$ ./usr/bin/tcpdump -h
tcpdump version 4.99.5
libpcap version 1.10.5 (with TPACKET_V3)
OpenSSL 3.2.4 11 Feb 2025
64-bit build, 64-bit time_t
Usage: tcpdump [-AbdDefhHIJKlLnNOpqStuUvxX#] [ -B size ] [ -c count ] [--count]
        [ -C file_size ] [ -E algo:secret ] [ -F file ] [ -G seconds ]
        [ -i interface ] [ --immediate-mode ] [ -j tstamptype ]
        [ -M secret ] [ --number ] [ --print ] [ -Q in|out|inout ]
        [ -r file ] [ -s snaplen ] [ -T type ] [ --version ]
        [ -V file ] [ -w file ] [ -W filecount ] [ -y datalinktype ]
        [ --time-stamp-precision precision ] [ --micro ] [ --nano ]
        [ -z postrotate-command ] [ -Z user ] [ expression ]

Conclusion: A Choice, An Attitude

In technology, there is no single truth. Snap and Flatpak aim to build secure, controlled ecosystems—an undeniably valuable goal.

CPK, however, is for a different audience: minimalists, control-seekers, and idealists who want software to remain in its purest form. It’s a way to express a technical stance—choosing simplicity over complexity, transparency over opacity, and freedom over constraint.

CPK does not seek to replace Snap, Flatpak, or AppImage. They serve different visions: Snap/Flatpak focus on secure, manageable ecosystems; AppImage pursues extreme single-file portability.

CPK is more than a format—it’s a philosophical practice of returning to software’s essence. With elegant engineering and minimal overhead, it precisely addresses pain points in specific scenarios, embodying the engineering ethos that “simple is beautiful.”

IMake: Bringing Builds Back to Makefile’s Essence — An Unprecedentedly Simple Experience

lengjingzju — Thu, 04 Sep 2025 04:38:19 +0000

Project Link: https://github.com/lengjingzju/cbuild-ng/tree/main/scripts/core

Kconfig Link: https://github.com/lengjingzju/kconfig

Note: IMake is a core component of CBuild-ng, but it can also be used independently.

The Modern Build System Dilemma and the Birth of IMake

In today’s software development landscape, the build system has become a critical factor in project success. From the classic Makefile to modern tools like Autotools, CMake, and Meson, developers have been searching for more efficient and flexible build solutions. However, these tools often come with steep learning curves and complex configuration syntaxes:

Autotools requires writing complex configure.ac and Makefile.am files using the M4 macro language, making configuration cumbersome.
CMake, while powerful, demands learning its own CMakeLists.txt syntax, and the generated Makefiles are often hard to read and debug.
Meson, a newer entrant with modern design concepts, still requires learning a new configuration language.

In their pursuit of power, these tools have drifted away from the essence of a build system—simplicity, intuitiveness, and control. Against this backdrop, IMake (Include Makefile) was born, bringing builds back to the Makefile’s essence while offering the advanced features of modern build systems.

IMake’s Core Strength: Simplicity Without Sacrificing Power

IMake is a pure Makefile-based build template system. Through modular design, it encapsulates complex build logic into reusable templates. By simply including the relevant template files and setting a few variables, you can easily configure builds ranging from simple applications to complex systems.

Key Advantages

Minimal Configuration: Say goodbye to complex configure.ac, CMakeLists.txt, and meson.build files—just define a few variables to compile.
Native Makefile Experience: 100% Makefile-based—no new syntax to learn, existing knowledge applies directly.
menuconfig GUI: Familiar Kconfig interface for easy build option management.
Versatile Support: Local and cross-compilation, classic and Yocto builds, static/shared libraries, executables, and kernel modules—all supported.
Toolchain Flexibility: Supports GCC and Clang with easy switching.
Smart Dependency Handling: Automatically analyzes header dependencies, detects environment changes, and triggers rebuilds to ensure correct build order.
Predefined Build Options: Common options like optimization levels, security hardening, sanitizers, static analysis, and profiling are built in.
Standards Compliance: Supports O= for out-of-tree builds, DESTDIR for install paths (GNUInstallDirs-compliant), and custom DEPDIR for dependency roots.

Comparison with Other Build Systems

Feature	IMake (Makefile)	CMake	Autotools	Meson
Learning Curve	Gentle	Steep	Steep	Moderate
Config Simplicity	★★★★★	★★★☆☆	★★☆☆☆	★★★★☆
Debug Friendliness	★★★★★	★★★☆☆	★★☆☆☆	★★★☆☆
Cross-Platform	Linux-focused	All platforms	Unix-like	All platforms
Visual Config	Built-in menuconfig	Extra tools needed	None	Limited
Ecosystem Integration	Complete	Rich	Mature	Growing

Ideal Use Cases and Target Users

Ideal Scenarios

Embedded Linux Development: Full-featured cross-compilation with simple configuration.
System-Level Software: Strong kernel module support, Linux standards-compliant.
Open-Source Libraries: GNU-compliant, easy integration into other projects.
Large-Scale Projects: Modular design, component-based development, intuitive menuconfig interface.

Target Users

Developers who value simplicity and transparency in build systems.
Linux-focused software engineers.
Those frustrated by the complexity of Autotools/CMake/Meson.
Experienced programmers seeking deep control over the build process.

Build Template Features in Detail

IMake can be used independently of CBuild-ng. It’s recommended to use the inc.makes template and set INC_MAKES to enable the desired templates.

inc.makes by default enables only inc.env.mk and inc.ins.mk.
INC_MAKES can be a combination of:
- disenv: Disable inc.env.mk
- conf: Enable inc.conf.mk
- app: Enable inc.app.mk
- mod: Enable inc.mod.mk
- disins: Disable inc.ins.mk

Template specific instructions refer to compilation-template .

Usage Examples

Real-World Projects

LJSON
JCore

Header-Only Application Project

A header-only project only needs an install step.
Headers are installed into an include/$(PACKAGE_NAME) subdirectory to avoid name collisions in large projects.

PACKAGE_NAME    = xxx

INSTALL_HEADERS:= $(wildcard src/*.h)

.PHONY: all clean install
all:
    @echo "Build $(PACKAGE_NAME) Done!"

include inc.makes

clean:
    @echo "Clean $(PACKAGE_NAME) Done."

install: install_hdrs
    @echo "Install $(PACKAGE_NAME) to $(INS_PREFIX) Done."

Single-Library Application Project

Specify the static library name via LIBA_NAME and the shared library name via LIBSO_NAME.
Automatically extract the version from VERSION_FILE using the macro VERSION_NAME to produce a shared library like libxxx.so.x.y.z.

PACKAGE_NAME    = xxx

VERSION_FILE   := xxx.hpp
VERSION_NAME   := XXX_VERSION

LIBA_NAME      := libxxx.a
LIBSO_NAME     := libxxx.so
CPFLAGS        += -I./src

INSTALL_HEADERS:= $(wildcard src/*.hpp)

.PHONY: all clean install
all:
    @echo "Build $(PACKAGE_NAME) Done!"

INC_MAKES      := app
include inc.makes

all: $(LIB_TARGETS)

clean: clean_objs
    @rm -f $(LIB_TARGETS)
    @echo "Clean $(PACKAGE_NAME) Done."

install: install_hdrs install_libs
    @echo "Install $(PACKAGE_NAME) to $(INS_PREFIX) Done."

Multi-Library and Executable Mixed Project

SEARCH_HDRS specifies subpaths for header search so code can use #include "aaa.h" instead of #include "mmm/aaa.h".
object_byte_size / frame_byte_size set maximum object and stack frame sizes; exceeding these triggers compile warnings to help debug large local variables (optional).
ENV_BUILD_TYPE sets optimization level: release = -O3, default is -O2.
set_flags applies compile flags to specific files (like kernel build per-file flags), while CPFLAGS is global.

PACKAGE_NAME    = xxx
SEARCH_HDRS    := mmm nnn

INSTALL_HEADERS:= $(wildcard src/*.hpp)

CPFLAGS        += -Isrc -Wno-unused-parameter

.PHONY: all clean install
all:
    @echo "Build $(PACKAGE_NAME) Done!"

INC_MAKES      := app
object_byte_size= 65536
frame_byte_size = 16384
ENV_BUILD_TYPE := release
include inc.makes

staticlib      := libxxx.a
sharedlib      := libxxx.so $(call get_version,src/xxx.hpp,JXXX_VERSION, )
libsrcs        := $(wildcard src/*.cpp)
$(call set_flags,CFLAGS,src/xxx_message.cpp,-Wno-missing-field-initializers)
LLIBS          := $(addprefix -l,jcore ljson g711 g722 bcg729 opus fdk-aac)
$(eval $(call add-liba-build,$(staticlib),$(libsrcs)))
$(eval $(call add-libso-build,$(sharedlib),$(libsrcs),$(LLIBS)))

server_srcs    := test/xxx_server.cpp
server_libs    := $(addprefix -l,jcore ljson $(PACKAGE_NAME))
$(eval $(call add-bin-build,xxx_server,$(server_srcs),$(server_libs),,$(OBJ_PREFIX)/lib$(PACKAGE_NAME).so))

client_srcs    := test/xxx_client.cpp
client_libs    := $(LLIBS) -l$(PACKAGE_NAME) 
$(eval $(call add-bin-build,xxx_client,$(client_srcs),$(client_libs),,$(OBJ_PREFIX)/lib$(PACKAGE_NAME).so))

all: $(BIN_TARGETS) $(LIB_TARGETS)

clean: clean_objs
    @rm -f $(LIB_TARGETS) $(BIN_TARGETS)
    @echo "Clean $(PACKAGE_NAME) Done."

install: install_hdrs install_libs install_bins
    @echo "Install $(PACKAGE_NAME) to $(INS_PREFIX) Done."

Single Driver Project

ifneq ($(KERNELRELEASE),)
MOD_NAME = xxx

INC_MAKES      := mod
include $(src)/inc.makes

else

PACKAGE_NAME    = xxx
SEARCH_HDRS     = mmm nnn

all: modules

clean: modules_clean

install: modules_install

INC_MAKES      := mod
include inc.makes
endif

Single Driver with Test Program

Uses the KERNELRELEASE macro to distinguish between Kbuild compilation and application compilation.

ifneq ($(KERNELRELEASE),)
MOD_NAME       := xxx
SRCS           := $(wildcard $(src)/src/*.c)

INC_MAKES      := mod
include $(src)/inc.makes
ccflags-y      += -I$(src)/src

else

PACKAGE_NAME    = xxx
CPFLAGS        += -Isrc
INSTALL_HEADERS:= src/xxx.h

all: modules

clean: modules_clean

install: modules_install

INC_MAKES      := app mod
include inc.makes

$(eval $(call add-bin-build,xxx_test,test/xxx_test.c))

all: $(BIN_TARGETS)

clean: clean_objs
    @rm -f $(BIN_TARGETS)
    @echo "Clean $(PACKAGE_NAME) Done."

install: install_bins install_hdrs
    @echo "Install $(PACKAGE_NAME) to $(INS_PREFIX) Done."

endif

Conclusion: Back to the Essence, Simplifying Builds

The emergence of IMake is a thoughtful response to today’s overly complex build system ecosystem. It proves a simple truth: powerful capabilities don’t require convoluted configuration, and elegant design can handle complex build needs.

Explore IMake and experience the beauty of simplicity in build systems! If you’re looking for a build solution that’s both powerful and straightforward, IMake is worth your time.

CBuild-ng: Paradigm Shift in Build Systems and Value Reconstruction

lengjingzju — Thu, 04 Sep 2025 04:38:15 +0000

Project Link : https://github.com/lengjingzju/cbuild-ng

Preface

In the evolution of software engineering, build systems have long faced a fundamental dilemma: developers must choose between freedom and order, flexibility and efficiency, ecosystem breadth and developer experience. Yocto and Buildroot stand as opposing poles of the traditional paradigm—one offering “unlimited possibilities at the cost of complexity,” the other “limited simplicity at the expense of extensibility.” This binary opposition mirrors industrial-era thinking of centralized control and standardized production.

The advent of CBuild-ng marks the end of that era. Rather than being a mere incremental improvement, it represents a full-scale paradigm revolution. Through innovations in architectural philosophy, interaction philosophy, ecosystem philosophy, and engineering philosophy, CBuild-ng completely transcends and reconstructs the traditional dualism.

1. Architectural Philosophy: From Monolithic Giants to Federated Microkernel

Traditional build systems suffer from a core issue of “global coupling.” Interdependent components form a tangled web, state changes cause unpredictable ripple effects, and complexity grows exponentially with scale—driving maintenance costs skyward.

CBuild-ng’s federated microkernel architecture restructures this:

Isolated Container Environments: Each package builds within its own workdir, isolating dependencies and ensuring a pure build context.
Intelligent Coordination Layer: A unified meta-system manages dependency resolution and state coordination, creating a clear, hierarchical structure.
Contractual Interface Specifications: Packages communicate only through well-defined interfaces, eliminating implicit coupling.

By dividing and conquering complexity, internal package complexity is contained, inter-package interactions are abstracted and automated, and overall build clarity, maintainability, and scalability reach unprecedented levels—turning large-scale builds from art guided by experience into an engineering science.

Module Federation in CBuild-ng

CBuild-ng redefines how applications are built. Traditional tools like Autotools, CMake, and Meson use an “embedded compile dependency” model that blurs module boundaries and complicates dependency graphs.

The new module-federation paradigm embraces:

Atomic Component Design: Every functional module is an independent package with its own version lifecycle; it can operate outside CBuild-ng.
Flat Dependency Management: Packages collaborate via explicit interfaces, forming a decentralized network topology.
Physical Isolation Guarantee: Instead of a single monolithic repo, the system is an organic federation of micro-packages.

Developers can assemble custom systems on demand—like building with blocks—dramatically reducing maintenance overhead while enabling genuine software reuse.

2. Interaction Philosophy: From Process Instructions to Intent Declarations

Traditional build modes force developers into the role of “process describers,” requiring an intimate understanding of every build mechanism. This instruction-writing paradigm demotes architects to assemblers, draining creative energy on minutiae.

CBuild-ng flips the script:

Declarative Programming Model: Developers specify only build intent—source files, dependencies, compiler flags—without detailing the process.
Intent Abstraction Engine: The system interprets high-level intents and auto-generates optimal build workflows, freeing developers to focus on design.

This shift slashes cognitive load and channels creativity into architecture and design, delivering a marked boost in development efficiency.

IMake Template Tool

Rather than inventing a new description language, CBuild-ng leverages IMake (Include Makefile) to simplify build configuration.

Unified Multi-Paradigm Build Framework

Multi-target support: single Makefile can produce libraries, executables, and kernel modules.
Mixed-language compilation: seamless handling of C, C++, assembly, and more.
Smart dependency analysis: automatic header-file tracking with fine-grained, source-level control.
Flexible directory layouts: configurable output directory (O=), install directory (DESTDIR=), and dependency directory (DEPDIR=).

Rich and Efficient Makefile Templates

inc.env.mk: environment configuration for output paths, dependency paths, compiler settings, and flags.
inc.app.mk: complete patterns for static libraries, shared libraries, and executables.
inc.mod.mk: separates driver output from source for kernel modules.
inc.ins.mk: adheres fully to GNUInstallDirs standards for installation.
inc.conf.mk: unified Kconfig/menuconfig interface for configuration management.

3. Ecosystem Philosophy: From Paradigm Lock-in to Free Integration

The highest cost of a technology choice is paradigm lock-in—simple choices sacrifice ecosystem, ecosystem choices demand complexity. This lock-in breeds decision anxiety early on and high migration costs later.

CBuild-ng’s dual-mode architecture solves this:

Classic Mode: A highly optimized standalone build system that outperforms Buildroot in simplicity and speed.
Yocto Mode: Lossless encapsulation and transparent proxy to the Yocto ecosystem, maintaining full compatibility.

For the first time, developers can switch freely between paradigms within a single framework—eliminating lock-in costs entirely.

Legacy Project Support: Classic mode embeds existing Autotools, CMake, Meson, or Makefile projects, unifying their pipelines with optimized configurations.
Unified Interface Layer: Yocto mode exposes the same configuration interface, developer experience, and toolchain management as Classic mode, streamlining resource allocation and build optimization.

4. Engineering Philosophy: Systematic Combat Against Entropy

CBuild-ng’s core value lies in systematically countering software entropy and governing complexity:

Performance Breakthroughs: Modern caching and parallel strategies yield orders-of-magnitude speed improvements.
Resource Optimization: Minimized output size translates directly to hardware cost savings.
Network Empowerment: Native support for distributed architectures enables development in isolated networks.
Quality Assurance: Enterprise-grade features like license compliance and dependency visualization are built in.

These capabilities elevate CBuild-ng from a helper tool to a competitive amplifier—directly enhancing product quality, cost efficiency, and delivery speed.

Cache Acceleration Innovations

Multi-Tier Mirror System

Public mirror integration: optimized access to major open-source repositories in China.
Private enterprise mirrors: meets the needs of air-gapped environments.
Smart routing: automatically selects the fastest download source.

Distributed Cache Ecosystem

Local cache acceleration: content-hash verification ensures integrity.
Team cache sharing: LAN-based cache pooling boosts collaboration.
Cloud sync: “build once, use anywhere” distribution of build artifacts.

Application Distribution Innovations

Independent Package Format

Self-contained deployment: packages and dependencies wrapped into a single unit (cpk format adds install scripts to the archive header).
Cross-environment compatibility: path isolation adjusts RPATH and ld.so paths instead of relying on exported environment variables.

Performance Highlights

Offline deployment: full support for air-gapped delivery.
Minutes-scale large-project builds.
Dramatically reduced output size for lower storage costs.
High cache hit rates enable near-instant subsequent builds.

Quality Framework

Dependency management: comprehensive visualization and analysis tools.
Cross-platform support: unified builds across Debian, RedHat, and Arch families.
Security compliance: auto-generated license lists and compliance reports.

5. Practical Validation: Industrial-Grade Reliability

CBuild-ng’s technical prowess has been validated in production:

Leading-Edge Enterprise Adoption

Recognized at CES 2024 as a disruptive innovation in a custom SDK platform built on CBuild.
Large-scale deployments across multiple domains—one build system supporting dozens of chip families and hundreds of configurations.
Proven performance metrics for efficient compilation and deployment of massive projects.

Specialized Deployments

Air-gapped build solutions for highly regulated industries.
Parallel development workflows for large teams.
Hybrid on-premises and cloud build pipelines.
Robust support for complex application compilation and debugging.

Conclusion: A New Engineering Paradigm

CBuild-ng transcends the traditional notion of a “build tool.” Built on a mature Makefile/Bash/Python stack, it integrates engineering innovations to redefine system build practices. It disproves the old assumption that “power equals complexity” and “simplicity equals limitation,” delivering a perfect union of capability and usability.

More importantly, CBuild-ng embodies a proven engineering philosophy: concentrate human ingenuity on creative work while delegating complexity to intelligent systems. As a battle-tested industrial solution, it not only offers technical leadership but also provides a complete enterprise value chain—from code authoring and build testing to dependency management and distribution. Every stage reflects deep optimization of developer experience and production efficiency.