Forem: cyril

Introducing Revenant: A Dollar-Cost Averaging Spot Trading Bot for Hyperliquid DEX.

cyril — Mon, 13 Apr 2026 13:43:13 +0000

Introduction

Revenant is a lightweight, open-source DCA (Dollar-Cost-Averaging) spot trading bot for Hyperliquid decentralized exchange, written entirely in Python.

It automates recurring spot market buys at configurable intervals, with built-in position sizing, take-profit logic, and real-time status monitoring using the official Hyperliquid Python SDK.

The bot was designed from the ground up for simplicity, reliability, and zero-maintenance operation.

This documentation walks through the full development and deployment journey, from initial setup and core trading logic to packaging it as a clean Flask app and launching it as a managed web service on DigitalOcean App Platform in minutes.

By following this documentation, anyone can run the bot as-is, customize it for their own strategy, or learn how to build and deploy similar trading tools.

Initial Skeletal Setup

Before any trading logic, we start with the smallest possible Flask application. This confirms that the project structure, build process, and DigitalOcean App Platform configuration are working correctly.

Project Structure (Root Folder)

revenant/
├── app.py
├── requirements.txt

Files
app.py

import os
from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello():
    return "Hello World!"

# Optional: only used when running locally with `python app.py`
if __name__ == "__main__":
    port = int(os.getenv("PORT", 8080))
    app.run(host="0.0.0.0", port=port)

requirements.txt

Flask
gunicorn

Next, deploy the app on DigitalOcean App Platform.

First, create a new GitHub repository (eg. revenant) and push the two files above to the main branch.

Head over to the DigitalOcean Control Panel, navigate to App Platform, then click on Create app.

Connect your GitHub account, select the relevant repository, and the main branch. App Platform will automatically detect the Python app.

In the Web Service component settings, set the run command as gunicorn --worker-tmp-dir /dev/shm app:app, and leave the build command empty.

Click Create App

Once the build finishes, your app will be live at a URL similar to https://revenant-xxxxx.ondigitalocean.app. You should see a plain white page with the text Hello World! exactly as shown below:

This confirms the deployment pipeline is working. From here, we can safely start building the actual DCA trading logic for Revenant without worrying about infrastructure issues.

Core Hyperliquid Utilities

Now, let's add the trading helper functions that interact with the Hyperliquid DEX. These utilities handle authentication, balance checks, price fetching, and executing market buy/sell orders for the relevant spot pair.

All sensitive data (especially the private key) is loaded from environment variables. Keys are never to be committed to Git.

Create a new Python file in the root directory (utility.py):

revenant/
├── app.py
├── requirements.txt
├── utility.py          ← new file

In the new utility.py file, add the following code:

import os
import eth_account

from hyperliquid.info import Info
from hyperliquid.exchange import Exchange
from hyperliquid.utils import constants

# Load private key from environment variable (required for DigitalOcean)
HL_PRIVATE_KEY = os.getenv("HL_PRIVATE_KEY")
if not HL_PRIVATE_KEY:
    raise ValueError("HL_PRIVATE_KEY environment variable is not set!")

# Authentication
wallet = eth_account.Account.from_key(HL_PRIVATE_KEY)
address = wallet.address

# Initialize exchange for placing orders
exchange = Exchange(
    wallet,
    base_url=constants.MAINNET_API_URL,
    account_address=address
)

# Initialize info client for reading data (REST only)
info = Info(constants.MAINNET_API_URL, skip_ws=True)

# Get USDC and HYPE balances (available balance = total - hold)
def get_balances(BASE_ASSET, QUOTE_ASSET):
    state = info.spot_user_state(wallet.address)
    balances = state.get("balances", [])

    quote_balance = 0.0
    base_balance = 0.0

    for balance in balances:
        if balance["coin"] == QUOTE_ASSET:
            quote_balance = float(balance["total"]) - float(balance["hold"])
        if balance["coin"] == BASE_ASSET:
            base_balance = float(balance["total"]) - float(balance["hold"])

    return quote_balance, base_balance

# Get current HYPE price from mid price
def get_price(BASE_ASSET):
    all_mids = info.all_mids()
    price = all_mids.get(BASE_ASSET)
    return float(price) if price else None

# Execute market sell of HYPE
def execute_sell(BASE_ASSET, QUOTE_ASSET, base_balance: float):
    sell = exchange.market_open(
        name=f"{BASE_ASSET}/{QUOTE_ASSET}",
        is_buy=False,
        sz=round(base_balance, 2),
        slippage=0.002
    )
    return sell

# Execute market buy of HYPE with USDC amount
def execute_buy(BASE_ASSET, QUOTE_ASSET,buy_size_usdc: float):
    if get_price(BASE_ASSET) is None:
        raise ValueError(f"Could not fetch {BASE_ASSET} price")

    size = buy_size_usdc / get_price(BASE_ASSET)
    buy = exchange.market_open(
        name=f"{BASE_ASSET}/{QUOTE_ASSET}",
        is_buy=True,
        sz=round(size, 2),
        slippage=0.002
    )
    return buy

# Get the most recent fill (useful for logging/tracking)
def get_most_recent_fill():
    fills = info.user_fills(address)
    if not fills:
        return None
    return fills[0]

Then update the requirements.txt file to reflect the newly added libraries, hyperliquid-python-sdk, eth-account, and protobuf:

Flask
gunicorn
hyperliquid-python-sdk
eth-account
protobuf

Following the introduction of a new environment variable, let's update DigitalOcean's Environment Variables.

Go to your app in the DigitalOcean control panel and click "Edit" on the Web Service component. Scroll to Environmental Variables and add a new variable:

Key: HL_PRIVATE_KEY
Value: 0x...

Click Save → Deploy (or let it auto-deploy on push).

Core DCA Trading Logic

We now replace the minimal "Hello World" Flask app with a complete trading engine for Revenant.

The bot implements a simple but effective DCA strategy on the HYPE/USDC spot pair:

Buys $10 worth of HYPE on every run (unless USDC balance is too low).
Continuously averages the buy price.
When the current price reaches +3% the average buy price, it sells the entire HYPE position, records the PnL, and starts a new session.
All states are tracked in a global counter dictionary (in-memory for now).

Full app.py:

import os, time
from flask import Flask, jsonify
import utility
import datetime

# Add background scheduler imports
from apscheduler.schedulers.background import BackgroundScheduler
import atexit

app = Flask(__name__)

# Define global variables
counter = {
    "avg_buy_price": float(0.0),
    "total_sessions": 0,
    "buy_size_usdc": float(10.0),
    "buy_count": 0,
    "total_buy_trades": 0,
    "total_volume_usdc": float(0.0)
}

BASE_ASSET = "HYPE"
QUOTE_ASSET = "USDC"

@app.route('/')
def home():
    return jsonify(counter)


@app.route("/execute")
def execute():

    # Initialize status
    status = ""

    # Fetch current market data, update AUM
    quote_balance, base_balance = utility.get_balances(BASE_ASSET, QUOTE_ASSET)
    price = utility.get_price(BASE_ASSET)
    AUM = float(quote_balance) + (float(base_balance) * float(price))

    # Save balances
    counter["base_balance"] = base_balance
    counter["quote_balance"] = quote_balance
    counter["price"] = price
    counter["AUM"] = AUM

    # Start/continue trading
    # Check if sell condition is met.
    sell_condition_is_met = False

    if float(price) >= float(counter["avg_buy_price"]) * 0.03 and float(counter["avg_buy_price"]) != float(0):
        sell_condition_is_met = True

    # if sell condition is met, liquidate hype position, end session, and start a new session. else keep buying.
    if sell_condition_is_met:
        # liquidate hype position (sell all of hype balance)
        sell = utility.execute_sell(BASE_ASSET, QUOTE_ASSET, base_balance)

        # get fill price and calculate pnl in usdc
        time.sleep(5)

        most_recent_fill = utility.get_most_recent_fill()

        avg_fill_price = most_recent_fill["px"]
        pnl_usdc = (float(avg_fill_price) - float(counter["avg_buy_price"])) * float(counter["base_balance"]) # (avg fill price - avg buy price) * total hype bought

        # update counter
        counter["pnl_usdc"] = pnl_usdc
        counter["total_sessions"] += 1
        counter["total_volume_usdc"] += (float(base_balance) * float(avg_fill_price))
        counter["last_trade"] = datetime.datetime.now(datetime.UTC)

        status = "Sell condition met, HYPE holdings sold."
    else:
        # keep buying with a constant size of $10
        buy_size_usdc = counter["buy_size_usdc"]

        # Execute buy if buy size is less than or equal to the available balance
        if buy_size_usdc <= quote_balance:

            # execute buy
            buy = utility.execute_buy(BASE_ASSET, QUOTE_ASSET, float(buy_size_usdc))

            # wait 5 seconds, get most recent fill details
            time.sleep(5)

            most_recent_fill = utility.get_most_recent_fill()
            avg_fill_price = most_recent_fill["px"]

            if float(counter["avg_buy_price"]) == float(0):
                counter["avg_buy_price"] = float(avg_fill_price)
            else:
                counter["avg_buy_price"] = (float(counter["avg_buy_price"]) + float(avg_fill_price))/2


            # Update counter
            counter["buy_count"] += 1
            counter["total_buy_trades"] += 1
            counter["total_volume_usdc"] += float(buy_size_usdc)
            counter["last_trade"] = datetime.datetime.now(datetime.UTC)

            status =  "Sell condition not met, buy executed."

        else:
            status  = "Sell condition not met, balance low." # buy nothing when balance low, wait for next 15 minutes

    return jsonify(status) 

# ── Background Scheduler (runs automatically every hour) ──
scheduler = BackgroundScheduler()

def scheduled_execute():
    """Wrapper that runs the trading cycle from the background job."""
    with app.app_context():
        execute()

scheduler.add_job(
    scheduled_execute,
    trigger='interval',
    hours=1,
    id='revenant_dca_cycle',
    replace_existing=True
)
scheduler.start()

# Gracefully shut down scheduler when the app exits
atexit.register(lambda: scheduler.shutdown(wait=False))

# Optional: only used when running locally with `python app.py`
if __name__ == "__main__":
    port = int(os.getenv("PORT", 8080))
    app.run(host="0.0.0.0", port=port)

How the Strategy Works (Summary)

Action	Condition	What Happens
Buy	Price < 3% avg_buy_price	Buy $10 USDC of HYPE
Sell (Take profit)	Price ≥ 3% avg_buy_price	Sell all HYPE, record PnL, reset session
No-op	Insufficient USDC balance	Wait for next cycle

Next, let's add a background scheduler (APScheduler) so the bot automatically runs the /execute endpoint every 1 hour.

Automatic Trading Every Hour

Add a background scheduler using APScheduler so Revenant automatically runs the full DCA trading cycle (/execute logic) every 1 hour. No manual calls or external cron jobs required.

Add APScheduler to the requirements.txt:

Flask
gunicorn
hyperliquid
eth-account
APScheduler

Next, add background scheduler imports:

from apscheduler.schedulers.background import BackgroundScheduler
import atexit

Then add the scheduler code beneath the trade logic execution:

# ── Background Scheduler (runs automatically every hour) ──
scheduler = BackgroundScheduler()

def scheduled_execute():
    """Wrapper that runs the trading cycle from the background job."""
    with app.app_context():
        execute()

scheduler.add_job(
    scheduled_execute,
    trigger='interval',
    hours=1,
    id='revenant_dca_cycle',
    replace_existing=True
)
scheduler.start()

# Gracefully shut down scheduler when the app exits
atexit.register(lambda: scheduler.shutdown(wait=False))

Update Run command on DigitalOcean

To prevent the scheduler from starting multiple times (Gunicorn forks workers by default):

Go to your app in DigitalOcean Control Panel → Components → your Web Service → Edit.
Change the Run command to: gunicorn --worker-tmp-dir /dev/shm --preload --workers=1 app:app
Save and Redeploy.

The --preload flag ensures the scheduler is created only once before workers are forked.

Current Limitations

State is stored only in memory → resets when the app restarts.
No error handling or retry logic around API calls yet.

Conclusion

Revenant is now a fully functional, automated DCA spot-trading bot running live on the DigitalOcean App Platform.

It connects securely to Hyperliquid, executes $10 USDC buys every hour, tracks average entry price, and automatically takes profit by selling the entire position when the price doubles.

The entire project uses a minimal, clean Flask + APScheduler setup that deploys in minutes and requires zero server management.

This is a solid foundation to extend: add persistent storage, configurable parameters via environment variables, a web dashboard, error alerts, or even multi-pair support.

Implementing Off-Chain Indexing in Polkadot Substrate for Optimized Data Retrieval and Storage

cyril — Mon, 06 Apr 2026 04:03:14 +0000

With over 600 ecosystem projects and 632k+ max transaction throughput, Polkadot has become one of the major blockchains with a thriving ecosystem. Building on Polkadot is fairly straightforward via the Polkadot SDK. The SDK is a versatile developer kit designed to facilitate building on the Polkadot network by making the required components for building chains, parachains, rollups, and more available to interested developers.

The Polkadot SDK is composed of five major components made up of Substrate, FRAME, Cumulus, XCM, and the node implementation protocol. Substrate is a highly customizable Rust framework for building application-specific chains from modular and extensible components. It includes default implementations of core infrastructure components, allowing developers to focus on application logic.

As chains mature and handle increasingly complex data and higher transaction volumes, the efficiency of data retrieval and storage becomes paramount for maintaining performance and scalability. Traditional on-chain data storage, while providing immutability and transparency, can become a bottleneck when dealing with frequent and complex queries, especially for large datasets.

To address these challenges, off-chain indexing presents a compelling solution for optimizing data handling in Substrate-based chains. This guide aims to provide a thorough understanding of off-chain indexing within the Polkadot ecosystem, offering detailed implementation guidance and exploring the various considerations for its effective utilization.

Background

Substrate provides a highly flexible and modular environment for blockchain development. Traditionally, blockchain data resides on-chain, ensuring immutability and network-wide consensus. However, for applications dealing with large datasets or requiring complex and frequent queries, this on-chain paradigm can become a performance bottleneck and significantly increase storage costs.

Technical Terms

On-chain Storage: Data that is stored directly on the chain and is part of the network's state, secured by consensus.
Off-chain Storage: Data that is stored outside the main chain, typically on individual nodes' local storage or external databases.
Offchain Workers (OCWs): Substrate's feature allowing for the execution of long-running, potentially non-deterministic tasks off the main chain, with the ability to read on-chain state and interact with the external world.
Off-chain Indexing: A Substrate feature enabling the runtime to directly write data to a node's local off-chain storage, which can then be queried by Offchain Workers or via RPC.
Runtime: The state transition logic of a Substrate-based chain, defining how the chain's state changes.
Extrinsic: A signed transaction that can trigger state changes in the chain's runtime.

In today's rapidly evolving tech landscape, chain applications are moving beyond simple token transfers to power complex ecosystems like DeFi, NFTs, and decentralized social media. These applications generate vast amounts of data, and efficient data retrieval is crucial for user experience, analytics, and overall platform scalability. Off-chain indexing offers a vital tool to decouple frequently queried or derived data from the core, expensive on-chain storage, leading to faster query times, reduced on-chain storage burden, and the ability to support more sophisticated data-driven features.

Deep Dive

The Power of Off-Chain Indexing in Substrate

Substrate's off-chain indexing allows the chain's runtime logic itself to write data directly to the local storage of the node executing that logic. This is distinct from off-chain workers performing external computations and then submitting transactions. The data written through off-chain indexing is not subject to the same consensus mechanisms as on-chain data, making writes significantly faster and less resource-intensive. This indexed data can then be accessed by off-chain workers for further processing or served via custom RPC endpoints.

Consider a decentralized marketplace where users create listings with numerous attributes (price, description, categories, etc.). Storing and querying all these attributes directly on-chain for every search can be inefficient. With off-chain indexing, when a new listing is created (an on-chain event), the runtime can extract relevant attributes and write them to the off-chain storage, indexed for efficient searching. An Offchain Worker or a custom RPC handler can then query this off-chain index to quickly retrieve listings based on various criteria without needing to iterate through all on-chain events.

This direct write capability from the runtime provides a highly efficient way to create searchable and filterable datasets derived from on-chain events or state changes. It offloads query-intensive tasks from the core chain, improving overall performance and responsiveness for end-users.

Implementing Off-Chain Indexing

Implementing off-chain indexing in Substrate involves defining what data to index and where to store it. Substrate provides a local storage API within the runtime that can be used for this purpose. When a relevant on-chain event occurs or a specific state transition happens, the runtime logic can extract the necessary information and use the offchain::local_storage_set function to store it under a specific key in the node's local storage. It's crucial to design a well-structured key-value system for efficient retrieval.

Let's say we want to index transfer events in a simple token pallet. When a Transfer event is emitted, the runtime can extract the from, to, and value and store this information off-chain using a composite key like (block_number, event_index) or a key based on the to address for quick balance lookups. An Offchain Worker could then periodically aggregate these off-chain transfer records to calculate user balances or generate transaction history summaries. The following snippet illustrates a simplified example within a Substrate pallet's on_finalize hook:

#![cfg_attr(not(feature = "std"), no_std)]

// Import necessary modules and traits from the frame_support and frame_system crates.
use frame_support::{
    decl_module, decl_storage, decl_event, decl_error, dispatch, ensure,
    traits::{Currency, ExistenceRequirement},
};
use frame_system::{ensure_signed, offchain::StorageKind}; // Import StorageKind for off-chain storage.
use sp_std::prelude::*; // Import standard library prelude for common types like Vec.
use codec::{Encode, Decode}; // Import Encode and Decode for serializing and deserializing data.

#[cfg(test)]
mod mock; // Define a module for unit tests.

#[cfg(test)]
mod tests; // Define a module containing the test cases.

// Define the Trait for this pallet. It specifies the types this pallet depends on.
pub trait Trait: frame_system::Trait + 'static {
    // Define the Event type for this pallet. It must be convertible to the frame_system's Event type.
    type Event: From<Event<Self>> + Into<<Self as frame_system::Trait>::Event>;
    // Define the Currency type used for token transfers.
    type Currency: Currency<Self::AccountId>;
}

// Declare the events that this pallet can emit.
decl_event!(
    pub enum Event<T> where
        AccountId = <T as frame_system::Trait>::AccountId,
        Balance = <T as Trait>::Currency::Balance,
    {
        // Event emitted when a successful transfer occurs.
        Transfer(AccountId, AccountId, Balance),
    }
);

// Declare the errors that this pallet can return during dispatchable calls.
decl_error! {
    pub enum Error for Module<T: Trait> {
        // Error returned when the sender has insufficient balance.
        InsufficientBalance,
    }
}

// Declare the storage items for this pallet.
decl_storage! {
    pub trait Store for Module<T: Trait> as Example {
        // A simple storage item to hold an optional u32 value.
        pub Something get(fn something): Option<u32>;
    }
}

// Declare the module logic.
decl_module! {
    pub struct Module<T: Trait>(_) {
        // Define the error type for this module.
        type Error = Error<T>;

        // Define the deposit event function used to emit events.
        fn deposit_event<T>() = default;

        // A dispatchable function for transferring tokens.
        #[pallet::weight(10_000)]
        pub fn transfer(origin, to: T::AccountId, value: T::Currency::Balance) -> dispatch::DispatchResult {
            // Ensure that the caller of this function is a signed account.
            let from = ensure_signed(origin)?;

            // Get the free balance of the sender.
            let balance = T::Currency::free_balance(&from);

            // Ensure that the sender has enough balance to make the transfer.
            ensure!(balance >= value, Error::<T>::InsufficientBalance);

            // Perform the token transfer.
            T::Currency::transfer(&from, &to, value, ExistenceRequirement::KeepAlive)?;

            // Emit a Transfer event.
            Self::deposit_event(Event::Transfer(from, to, value));

            // Return Ok to indicate success.
            Ok(())
        }

        // The on_finalize hook is called at the end of each block.
        fn on_finalize(block_number: T::BlockNumber) {
            // Get the list of events that occurred in the current block.
            if let Some(last_event) = frame_system::Module::<T>::events().last() {
                // Match on the last event to see if it's a Transfer event.
                if let Event::Transfer(from, to, value) = last_event.event {
                    // Create a unique key for storing the transfer information off-chain.
                    // The key includes the event name, block number, and event index in the block.
                    let key = ("transfer", block_number, frame_system::Module::<T>::event_count() - 1).encode();
                    // Encode the transfer information (from, to, value) to be stored.
                    let value_to_store = (from, to, value).encode();
                    // Store the encoded transfer information in the node's local off-chain storage.
                    // StorageKind::PERSISTENT ensures the data persists across node restarts.
                    sp_io::offchain::local_storage_set(StorageKind::PERSISTENT, &key, &value_to_store);
                }
            }
        }
    }
}

This direct integration within the runtime ensures that the off-chain index is updated atomically with on-chain state changes. By carefully choosing the data to index and designing an efficient key structure, developers can create highly optimized data access layers for their applications.

Accessing and Utilizing Off-Chain Indexed Data

Data stored via off-chain indexing using local_storage_set can be accessed by Offchain Workers using the offchain::local_storage_get function. This allows OCWs to perform complex queries or aggregations on the indexed data without burdening the on-chain runtime. Additionally, Substrate provides an RPC endpoint, offchain_localStorageGet, which allows external clients (like frontend applications) to query this off-chain data directly, bypassing the need to fetch and process numerous on-chain events.

Continuing the decentralized marketplace example, an off-chain worker could periodically scan the off-chain index of listings, filter them based on certain criteria (e.g., recently expired), and then potentially submit on-chain transactions to update the status of these listings. A frontend application could use the offchain_localStorageGet RPC to display a paginated and filtered list of active listings to users, providing a much faster and more responsive experience than querying on-chain events directly.

// Example of an off-chain worker accessing and utilizing off-chain indexed data.
// This code snippet would typically reside within an `impl` block for a pallet
// that has declared an `impl OffchainWorker` trait.

impl<T: Trait> Module<T> {
    fn offchain_worker(block_number: T::BlockNumber) {
        // Use a constant prefix for keys to organize off-chain storage.
        let prefix = b"transfer";

        // Iterate through a range of recent block numbers to retrieve transfer events.
        // This is a simplified example; a more robust approach might involve
        // tracking the last processed block number.
        for bn in block_number.saturating_sub(T::BlockNumber::from(10))..block_number {
            // Iterate through potential event indices within the block.
            // The actual number of events per block can vary.
            for index in 0..100 { // Example: Check up to 100 events per block
                // Construct the key to retrieve the off-chain data.
                let key = (prefix, bn, index).encode();

                // Retrieve the data from the local off-chain storage.
                let result = sp_io::offchain::local_storage_get(StorageKind::PERSISTENT, &key);

                if let Some(encoded) = result {
                    // Decode the retrieved data.
                    if let Ok((from, to, value)) = <(T::AccountId, T::AccountId, T::Currency::Balance)>::decode(&mut &encoded[..]) {
                        // Now you can utilize the retrieved transfer information.
                        // For example, you could:
                        log::info!("OCW: Found off-chain transfer at block {:?}, index {:?}: From {:?}, To {:?}, Value {:?}",
                            bn, index, from, to, value);

                        // Perform some off-chain computation or aggregation with this data.
                        // For instance, you could track the total value transferred to a specific account off-chain.
                        // (Implementation of such logic is beyond the scope of this example).
                    } else {
                        log::warn!("OCW: Failed to decode off-chain transfer data at block {:?}, index {:?}", bn, index);
                    }
                } else {
                    // No data found for this key. Continue to the next potential event.
                }
            }
        }

        // Example of accessing off-chain data via RPC (this wouldn't be inside the OCW itself,
        // but demonstrates how an external client might fetch the data).
        // To enable this, you would need to define custom RPC methods in your runtime API.
        // The client would then call `offchain_localStorageGet` with the appropriate key.
        // For instance:
        // ```
{% endraw %}

        // // In your custom RPC implementation:
        // pub fn get_offchain_transfer(
        //     _at: Option<<Block as BlockT>::Hash>,
        //     block_number: <Block as BlockT>::BlockNumber,
        //     event_index: u32,
        // ) -> Result<Option<Bytes>, sp_rpc::error::Error> {
        //     let key = ("transfer", block_number, event_index).encode();
        //     let result = sp_io::offchain::local_storage_get(StorageKind::PERSISTENT, &key);
        //     Ok(result.map(|v| v.into()))
        // }
        //
{% raw %}

    // A frontend could then call this RPC method to retrieve the encoded data.
}

}




The ability for both off-chain workers and external clients to efficiently access off-chain indexed data unlocks powerful possibilities for building feature-rich and performant decentralized applications. It allows for complex data presentation, real-time analytics, and efficient background processing without impacting the core blockchain's performance.

## Challenges and Solutions

### Challenges
- **Data Consistency**: Ensuring the off-chain index remains consistent with the on-chain state is crucial. If the blockchain experiences a rollback, the off-chain index might become out of sync.

- **Scalability of Local Storage**: Relying solely on individual nodes' local storage might present scalability challenges for very large datasets or high-traffic applications.

- **Query Complexity**: While off-chain indexing improves query speed for predefined indexed fields, complex ad-hoc queries might still require more sophisticated off-chain database solutions.

- **Data Availability**: Data stored off-chain is not inherently replicated across the network in the same way as on-chain data. Ensuring data availability might require additional mechanisms.


### Solutions
- **Event-Driven Updates**: Trigger off-chain index updates directly from on-chain events to maintain near real-time consistency.

- **Periodic Reconciliation**: Implement mechanisms for off-chain workers to periodically verify and reconcile the off-chain index with the on-chain state.

- **External Databases**: For very large datasets or complex querying needs, consider using Offchain Workers to populate and query external databases (e.g., PostgreSQL, Elasticsearch) based on on-chain events.

- **Redundancy and Backups**: Implement strategies for backing up and potentially replicating off-chain indexed data across multiple nodes or using distributed off-chain storage solutions.

### Best Practices
- **Index Relevant Data**: Only index data that is frequently queried or requires optimized retrieval. Avoid indexing everything off-chain.

- **Design Efficient Key Structures**: Choose key structures that allow for fast and targeted lookups.

- **Handle Rollbacks Gracefully**: Implement logic to detect and handle potential inconsistencies due to blockchain rollbacks.

- **Monitor Off-Chain Storage**: Regularly monitor the storage usage of your off-chain index.

- **Consider Data Privacy**: Be mindful of any sensitive data being stored off-chain and implement appropriate security measures.

## Looking Ahead

The integration of off-chain indexing with more sophisticated off-chain computation and [storage solutions](https://github.com/JoshOrndorff/substrate-recipes/blob/master/text/off-chain-workers/storage.md) is likely to evolve. We might see more standardized patterns and libraries emerging to simplify the implementation and management of off-chain indexes.

Exploring decentralized off-chain storage solutions (like [IPFS](https://ipfs.tech/) or [Arweave](https://arweave.org/)) in conjunction with off-chain indexing could provide both performance benefits and enhanced data availability and immutability for the indexed data.

Off-chain indexing will become an increasingly essential tool for building scalable and user-friendly decentralized applications on Substrate. As appchains become more data-intensive, the ability to efficiently manage and query data off-chain will be a key differentiator for successful projects.

## Conclusion

Implementing off-chain indexing in Substrate offers a powerful way to optimize data retrieval and storage by leveraging the node's local storage for frequently queried or derived data. By allowing the runtime to directly write indexed information and enabling Offchain Workers and external clients to efficiently access it, developers can build more performant and scalable blockchain applications.

While off-chain indexing introduces its own set of challenges, the benefits in terms of performance and scalability often outweigh the complexities. Understanding and strategically implementing off-chain indexing is a crucial skill for any Substrate developer looking to build robust and user-friendly decentralized solutions.

## References

- **Polkadot SDK Docs**: [https://docs.polkadot.com/develop/parachains/intro-polkadot-sdk/](https://docs.polkadot.com/develop/parachains/intro-polkadot-sdk/).
- **Substrate Recipes: Off-chain Storage**: [https://github.com/JoshOrndorff/substrate-recipes/blob/master/text/off-chain-workers/storage.md](https://github.com/JoshOrndorff/substrate-recipes/blob/master/text/off-chain-workers/storage.md).
- **Substrate Recipes: Off-chain Indexing**: [https://github.com/JoshOrndorff/substrate-recipes/blob/master/text/off-chain-workers/indexing.md](https://github.com/JoshOrndorff/substrate-recipes/blob/master/text/off-chain-workers/indexing.md)

Installing Elixir on Debian Crostini

cyril — Sun, 02 Mar 2025 11:49:53 +0000

Introduction

The official Elixir installation script does not support Debian, a flavor of the Linux operating system, that is widely used, specifically in servers, development environments, and Chromebooks.

In this post, we shall find a workaround and install Elixir on Debian Crostini, a customized version of Debian that comes with ChromeOs which powers Chromebooks worldwide.

Elixir is a modern functional programming language that runs on ErlangVM. It is known for having an easy-to-write syntax, high scalability, and designed to be fault tolerant and scalable. In recent times, Elixir has become the language of choice for managing the systems and logic around AI agents and chatbots due to its concurrency, distributed computing, and real-time capabilities.

In the next section, we'll cover the overview of the steps required to install Elixir on Debian systems.

Overview of Steps

The following steps are required to install Elixir on Debian:

Install asdf
Install Erlang
Install Elixir

Installing ASDF

asdf is a tool version manager that ensures consistency in the version of tools across users by providing a single interface and configuration file to make development workflows simple.

To install asdf, we first need to install dependencies.

$ sudo apt install -y curl git

Next, clone asdf from the GitHub repo:

$ git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.16.4

Replace v0.16.4 with the version of ASDF you wish to install.

Now let's add the binaries to .bashrc environment.

$ subl ~/.bashrc

Here, you can use any text editor of your choice. This example uses the Sublime Text (subl) editor.

Then add the following lines at the end of the file:

. "$HOME/.asdf/asdf.sh"
. "$HOME/.asdf/completions/asdf.bash"

Now we can fetch the version for asdf to verify a successful install:

$ asdf --version
v0.15.0-4b9297b

With asdf installed, we can now proceed to install Erlang.

Installing Erlang

Let's install dependencies before compiling Erlang with asdf.

$ sudo apt install -y build-essential autoconf automake m4 libwxgtk3.2-dev libwxgtk-webview3.2-dev libgl1-mesa-dev libglu1-mesa-dev libpng-dev libssh-dev unixodbc-dev xsltproc fop libxml2-utils libncurses-dev openjdk-17-jdk

We can then proceed to install Erlang:

$ asdf plugin add erlang https://github.com/asdf-vm/asdf-erlang.git
$ asdf install erlang 27.2.4
$ asdf global erlang 27.2.4

Where 27.2.4 is the version of Erlang you wish to install.

With Erlang installed, we can proceed to finish with adding Elixir.

Installing Elixir

Let's add elixir as a plugin and then execute the install command.

$ asdf plugin add elixir https://github.com/asdf-vm/asdf-elixir.git
$ asdf install elixir 1.18.2-otp-27
$ asdf global elixir 1.18.2-otp-27

Where 1.18.2 is the version of elixir, and 27 is the version of Erlang installed beforehand.

By checking the elixir version, you should see something similar to the following:

$ elixr -v
Elixir 1.18.2 (compiled with Erlang/OTP 27)

Elixir is now successfully installed. You can now launch the interactive console by executing the following command:

$ iex

Conclusion

In this post, we successfully navigated the challenge of installing Elixir on Debian Crostini, a customized version of Debian used in Chromebooks. By leveraging asdf, a versatile tool version manager, we installed both Erlang and Elixir, ensuring compatibility and consistency. This setup unlocks the power of Elixir’s concurrency, fault tolerance, and real-time capabilities, making it a great choice for building scalable systems, AI agents, and chatbots.

Quantifying LVR on Uniswap v2

cyril — Thu, 12 Sep 2024 21:51:27 +0000

Introduction

Did you know that providing liquidity on Uniswap, the most popular decentralized exchange, can lead to systematic underperformance compared to a simple rebalancing strategy? This phenomenon, known as Loss-Versus-Rebalancing (LVR), highlights a hidden cost in liquidity provision that even experienced traders might overlook. LVR occurs due to price slippage when automated market makers (AMMs) like Uniswap execute trades at worse-than-market prices as asset prices fluctuate.

In this research, we’ll break down the concept of LVR, focusing specifically on its quantification within Uniswap v2. Understanding LVR is crucial for anyone participating in decentralized finance (DeFi) because it directly impacts the returns of liquidity providers (LPs). By exploring how LVR functions and how it can be measured, stakeholders will gain insights into the risks and rewards of liquidity provision, learn strategies to mitigate LVR, and discover how this knowledge can influence AMM design and fee structures in the future.

Objectives

This research aims to achieve the following objectives:

Quantify and estimate LVR across Ethereum mainnet and major Layer 2 networks (Arbitrum, Optimism, Base).
Develop a methodology to estimate and compare the Total Addressable Market (TAM) of LVR for Uniswap v2 across different networks.

Background

AMMs like Uniswap, have transformed decentralized exchanges by providing a simpler and more efficient trading mechanism compared to traditional order books used in centralized exchanges. Uniswap v2, for example, operates through a constant function market maker (CFMM) model, relying on passive liquidity providers to facilitate trades.

LVR is a key concept that measures the performance gap between passive liquidity provision in AMMs and an active rebalancing strategy that adjusts positions based on market prices. This model, inspired by the Black-Scholes framework, focuses on trading between a risky asset and a stable asset, like USDC, with the risky asset's price following a geometric Brownian motion.

In this model, arbitrageurs exploit price slippage in AMMs, where prices lag behind real-time market movements, leading to LVR. The model assumes an ideal centralized exchange (CEX) with infinite liquidity and no fees for arbitrageurs, ensuring the AMM’s price aligns with the CEX price. It also assumes AMMs follow an invariant curve, maintaining a constant product of reserves.

The LVR model is influenced by two main factors: the price variance of the risky asset and the available liquidity in the AMM. These factors impact price slippage and the overall losses associated with LVR, offering insights into the economic dynamics of liquidity provision in AMMs.

Methodology

To achieve the objectives of this research, we begin by collecting data on Uniswap v2 pool activity. The data is sourced from the defi.ez_dex_swaps table on Flipside Crypto, which provides comprehensive trade information, including block timestamp, pool name, pool address, amount in, and amount out. We filter this data specifically for Uniswap v2 trades to ensure relevance to our analysis.

Given that LVR is calculated based on the volatility between a risky asset and a numeraire (such as USDC, USDT, or DAI), we focus on trades where the output token is one of these stablecoins. This allows us to accurately assess the volatility required for the LVR calculation. By using the amount_in and amount_out fields, we can determine the minimum and maximum effective prices for each pool daily. These prices reflect the extreme values at which assets were traded within a pool on a given day.

The following SQL query is used to generate the necessary dataset:

select
  date_trunc('day', block_timestamp) as date,
  pool_name,
  contract_address as pool_addr,
  min(amount_out/amount_in) as min_ep,
  max(amount_out/amount_in) as max_ep
from ethereum.defi.ez_dex_swaps
where platform = 'uniswap-v2'
  and (token_out = '0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48' -- USDC 
    or token_out = '0xdac17f958d2ee523a2206206994597c13d831ec7' -- USDT
    or token_out = '0x6b175474e89094c44da98b954eedeac495271d0f' -- DAI
)
group by date, pool_name, pool_addr
order by date;

This query is adapted for each network—Ethereum, Arbitrum, Optimism, and Base—by replacing the network name and relevant contract addresses for the numeraires. While this method does not capture 100% of the trades within each pool, it is sufficient for our purposes, assuming the presence of arbitrageurs who ensure price accuracy. This approach is favored for its simplicity and effectiveness in generalizing pool behavior.

Once we have the daily minimum and maximum effective prices, we calculate the daily volatility as the percentage difference between these two extremes, similar to Today's True Range (TTR) formula. This daily volatility value is then used to compute the LVR. This calculation enables us to quantify LVR across different networks and draw meaningful comparisons.

Deep Dive

LVR on Mainnet

LVR remains consistently high across different time frames for Uniswap v2 pools on Ethereum Mainnet. The weekly average LVR is highest at about 110 basis points, followed closely by monthly and daily averages. The small variation between time frames (only about 10 basis points difference) indicates that LVR is a persistent issue rather than a short-term anomaly. Overall, the data suggests that liquidity providers consistently underperform compared to optimal rebalancing strategies, regardless of the time frame considered.

There is significant variation in LVR across Uniswap v2 pools on Ethereum Mainnet. The DAI-WETH pool stands out with the highest average daily LVR of nearly 200 basis points, indicating substantial underperformance compared to optimal rebalancing. LVR decreases sharply for other pools, with stablecoin pairs and those involving wrapped Bitcoin generally showing lower LVRs. This suggests that pool composition and asset volatility play crucial roles in determining LVR, with some pools performing much closer to optimal strategies than others.

LVR was extremely high in early 2020, with significant volatility. The LVR spiked above 1,500 basis points at certain points, reflecting a period of heightened market inefficiency and higher costs for liquidity providers. After mid-2020, the LVR declined and stabilized to much lower levels, mostly below 250 basis points, indicating that over time, market conditions on Ethereum Mainnet became more efficient, with LVR settling into a more predictable range.

LVR is relatively stable from Monday to Thursday, staying around 100 basis points. However, it slightly decreases on Friday and continues to drop through Saturday and Sunday. This indicates that liquidity provision might be more favorable over the weekend, possibly because of lower trading volumes or market activity, leading to less slippage and reduced losses from rebalancing inefficiencies.

LVR remains relatively consistent throughout the first four weeks of the month, fluctuating between 90 and 105 basis points. However, the fifth week exhibits a significant increase in LVR, rising to approximately 115 basis points. This suggests that during months with a fifth week, liquidity provision might become less efficient or more costly, potentially due to market conditions that occur towards the end of the month.

LVR on Arbitrum

Weekly LVR is highest at about 155 basis points, followed closely by monthly LVR at around 150 basis points. Daily LVR is notably lower at approximately 125 basis points. This suggests that losses compared to optimal rebalancing strategies tend to accumulate over longer periods on Arbitrum, with weekly and monthly holdings experiencing higher relative losses than daily positions.

The WETH-DAI pool shows the highest LVR at about 115 basis points, significantly higher than other pools. There's a clear descending trend in LVR as we move through the pools, with stablecoin pairs (USDC-USDT, WBTC-USDC) showing very low LVRs below 5 basis points. This indicates that certain asset pairs, particularly those involving more volatile cryptocurrencies, tend to underperform compared to optimal rebalancing strategies, while stablecoin pairs perform much closer to optimal.

There's high volatility in LVR, with extreme spikes reaching over 2000 basis points in early March. However, the general trend shows a decrease in LVR over time, with more recent months exhibiting lower and more stable LVR values, mostly below 500 basis points. This suggests an improvement in pool efficiency or liquidity provider strategies on Arbitrum over the observed period.

Thursday stands out with the highest LVR at nearly 200 basis points, indicating significantly higher losses compared to optimal rebalancing on this day. Tuesday and Sunday also show elevated LVRs above 125 basis points. In contrast, Friday has the lowest LVR at about 85 basis points, followed closely by Monday. This pattern suggests that liquidity providers might benefit from adjusting their strategies based on the day of the week, potentially focusing on rebalancing towards the end of the work week.

There's a significant variation in the average Loss-Versus-Rebalancing (LVR) for Uniswap v2 pools on Arbitrum across different weeks of the month, with Week 2 showing the highest LVR at about 160 basis points, followed by Week 3 at around 145 basis points. Week 4 has the lowest LVR at approximately 85 basis points. This suggests that liquidity providers may experience higher losses compared to optimal rebalancing strategies in the middle of the month, while the end of the month tends to see better performance.

LVR on Optimism

Weekly LVR is highest at about 65 basis points, followed by monthly LVR at around 45 basis points. Daily LVR is notably lower at approximately 21 basis points. This suggests that losses compared to optimal rebalancing strategies tend to accumulate over longer periods on Optimism, with weekly and monthly holdings experiencing higher relative losses than daily positions. The significant difference between time frames indicates that the frequency of rebalancing can have a material impact on the performance of liquidity providers on Optimism.

The USDC-sOPWETH pool shows the highest LVR at about 155 basis points, significantly higher than other pools. The WETH-USDT pool follows with around 70 basis points. There's a sharp decline in LVR for subsequent pools, with the last five pools all showing LVRs below 5 basis points. This indicates that certain asset pairs, particularly those involving wrapped ETH or stablecoins, tend to underperform compared to optimal rebalancing strategies, while other pairs perform much closer to optimal.

The chart shows high volatility, with extreme spikes reaching over 2500 basis points in early March and around 1500 basis points in early May. However, the general trend shows a decrease in LVR over time, with more recent months exhibiting lower and more stable LVR values, mostly below 500 basis points. This suggests an improvement in pool efficiency or liquidity provider strategies on Optimism over the observed period, with occasional periods of high volatility.

Wednesday shows the highest LVR at nearly 55 basis points, followed by Monday at about 40 basis points and Thursday at around 35 basis points. The weekend days (Saturday and Sunday) and Tuesday have the lowest LVRs, all below 5 basis points. This pattern indicates that liquidity providers on Optimism might benefit from adjusting their strategies based on the day of the week, with mid-week days, especially Wednesday, showing higher potential for losses compared to optimal rebalancing.

Week 3 stands out with the highest LVR at about 52 basis points, significantly higher than other weeks. Week 2 follows with around 21 basis points, while Weeks 1 and 5 show similar LVRs of about 12-13 basis points. Week 4 has the lowest LVR at approximately 4 basis points. This pattern suggests that liquidity providers on Optimism may experience higher losses compared to optimal rebalancing strategies in the middle of the month, particularly in Week 3, while the end of the month tends to see better performance.

LVR on Base

Weekly LVR is highest at about 380 basis points, followed by monthly at around 320 basis points, and daily at approximately 260 basis points. Longer time frames (weekly and monthly) show higher average LVRs compared to daily, suggesting that the cumulative effect of price movements and trading activity over extended periods leads to greater divergence from optimal rebalancing.

The WETH-DAI pool has the highest average LVR at about 500 basis points, while the USDCL-swing pool has the lowest at nearly 0 basis points. There's a steep decline in LVR from the top pools to the bottom ones. Pools involving major assets like WETH, DAI, and USDC tend to have higher LVRs, possibly due to higher trading volumes or more significant price movements in these pairs.

This time series chart shows the daily average LVR from March to September 2024. There's a significant spike in LVR during late March and early April, reaching peaks above 4000 basis points. After this initial period of high volatility, the LVR generally stabilizes at much lower levels, mostly below 1000 basis points, with occasional smaller spikes.

This chart displays the average LVR for each day of the week. Saturday shows the highest average LVR at approximately 320 basis points, while Tuesday has the lowest at about 220 basis points. There's a general upward trend from Tuesday to Saturday, with a sharp decline on Sunday. Weekend trading, particularly on Saturdays, results in higher LVR for Uniswap v2 pools on Base, possibly due to increased volatility or reduced liquidity during these periods.

Week 3 has the highest average LVR at around 300 basis points, while Week 1 has the lowest at about 200 basis points. There's a general trend of increasing LVR from Week 1 to Week 3, followed by a decline in Weeks 4 and 5.

Total Addressable Market

Total Addressable Market (TAM) in the context of Loss-Versus-Rebalancing (LVR) for Uniswap v2 represents the potential market size for passive liquidity provision across different blockchain networks. LVR is a crucial metric that measures the efficiency of automated market makers (AMMs) compared to active rebalancing strategies.

To estimate the TAM of LVR for Uniswap v2, we developed a model that:

Collects historical trading data from Uniswap v2 on each network (Ethereum mainnet, Arbitrum, Optimism, and Base).
Calculates daily trading volumes for each network.
Estimates the potential losses liquidity providers (LPs) might incur due to LVR based on price volatility.
Aggregates this data to determine the average daily TAM for each network and the total across all networks.

Arbitrum (Average daily TAM: $11.15 million):
The highest TAM suggests that Arbitrum has the most active Uniswap v2 ecosystem among the studied networks. This could indicate higher trading volumes and/or greater price volatility on this network. For LPs, this represents both the largest opportunity and the highest risk of impermanent loss.

Base (Average daily TAM: $10.37 million):
Close to Arbitrum's value, indicating strong adoption of Uniswap v2 on this newer Layer 2 solution. The high TAM suggests significant LP activity and potential for LVR.

Ethereum mainnet (Average daily TAM: $10.31 million):
Despite high gas fees, mainnet still shows a substantial TAM, reflecting its established ecosystem and possibly higher-value trades.

Optimism (Average daily TAM: $0.22 million):
Significantly lower than other networks, suggesting less Uniswap v2 activity or lower price volatility on Optimism. This could mean lower risk for LPs, but also lower potential returns.

Total average daily TAM ($32.06 million):
This substantial figure across all networks underscores the significant role of AMMs in the DeFi ecosystem and the scale of potential LVR.

Given the high TAM values, especially on Arbitrum, Base, and mainnet, LPs are likely experiencing significant LVR. However, this doesn't necessarily mean they're losing money overall. In periods of high volatility and trading volume, fees earned might outweigh LVR. Conversely, in less active periods, LVR could exceed fee income.

The relatively low TAM on Optimism suggests LPs there might be experiencing less LVR, but potentially earning lower fees as well.

Conclusion

This comprehensive analysis of LVR across Uniswap v2 pools on Ethereum mainnet, Arbitrum, Optimism, and Base networks provides valuable insights into the performance of liquidity provision in decentralized finance. The research reveals significant variations in LVR across different networks, time frames, and specific pools. These findings underscore the importance of understanding LVR for liquidity providers, as it directly impacts their returns and can lead to systematic underperformance compared to optimal rebalancing strategies.

The Total Addressable Market (TAM) analysis further emphasizes the scale and significance of LVR in the DeFi ecosystem. With a total average daily TAM of $32.06 million across all studied networks, it's clear that LVR represents a substantial economic factor in automated market maker (AMM) dynamics. The varying TAM values across networks, with Arbitrum leading at $11.15 million and Optimism trailing at $0.22 million, highlight the differences in market activity and potential risks and opportunities for liquidity providers on each platform.

As the DeFi landscape continues to evolve, this research serves as a crucial resource for stakeholders to make informed decisions about liquidity provision strategies. The insights gained from this study can guide the development of more efficient AMM designs, help liquidity providers optimize their strategies to mitigate LVR and inform the broader community about the hidden costs associated with passive liquidity provision. Future research could explore how these findings translate to newer AMM models and investigate strategies to minimize LVR while maximizing returns for liquidity providers.

References

Sources: https://arxiv.org/pdf/2208.06046
Dataset, Code, and Charts: https://github.com/polymawutor/lvr-uniswapv2

About the Author

mawutor (@polymawutor) is a web3 developer with a passion for exploring the latest advancements in blockchain technology. With a focus on providing informative content and building innovative solutions, mawutor aims to demystify complex topics and empower users with actionable insights.

KAIA Tokenomics Explained

cyril — Wed, 21 Aug 2024 10:26:43 +0000

Introduction

Did you know Kaia Chain's token economy (tokenomics) is designed to mint approximately 200 million tokens annually? This amounts to a controlled 5.2% annual inflation rate strategically managed to fuel growth and investment in the network.

KAIA is the primary cryptocurrency within the Kaia ecosystem. It is used to pay transaction fees associated with creating or executing smart contracts and transferring KAIA tokens.

KAIA's tokenomics is designed as more than just a reward system for validators; it's a comprehensive strategy for sustainable growth and long-term investment. Unlike Bitcoin and Ethereum which only focus on technical maintenance through token rewards, Kaia's economy also incentivizes contributions from a broader range of participants.

Kaia's blockchain technology is designed for enterprise adoption, therefore, it is crucial to understand how the network balances the needs of validators, ecosystem growth, and infrastructure development.

In this post, we will learn about Kaia's token economy which continuously supports the network growth through block rewards, making it a resilient and future-focused chain. The end goal is to deepen our understanding of Kaia and how the chain sustains the ecosystem and drives innovation.

Background

Blockchain technology has revolutionized the way we think about finance, digital assets, and decentralized networks. At the heart of many blockchains lies a "token economy," a system designed to incentivize network participants and ensure the network's health, security, and growth. Understanding this concept and related terms like inflation and circulating supply is crucial to grasping how a blockchain project like Kaia operates.

Key Concepts Explained

Token Economy

A token economy refers to the ecosystem of a blockchain where tokens are used as a means of incentivizing participants, conducting transactions, and governing the network. In many cases, these tokens are created, distributed, and managed according to rules defined by the network's protocol. The token economy is crucial because it determines how resources are allocated, how security is maintained, and how growth and innovation are incentivized.

Inflation

Inflation in the context of a blockchain refers to the rate at which new tokens are created and added to the existing supply. This is similar to traditional economic inflation, where the supply of money in an economy increases. Inflation is often used in blockchains to reward participants, such as validators or miners, who contribute to the network's maintenance and security. However, if not managed carefully, it can lead to a dilution of the token's value.

Circulating Supply

The circulating supply of a token is the total amount of that token currently available and being actively traded or used in the market. It excludes tokens that are locked, reserved, or otherwise not in circulation. The circulating supply is a key metric for understanding the token's market dynamics, including its price and overall market capitalization.

Block Reward

Block rewards are the incentives given to miners or validators for adding a new block of transactions to the blockchain. These rewards typically consist of newly minted tokens (inflationary rewards) and transaction fees from the transactions included in the block.

Staking

Staking is the process of locking up tokens in a blockchain network to support its operations, like validating transactions and securing the network. In return, participants (called validators or stakers) earn rewards, often in the form of additional tokens.

In the following sections, we shall deep dive into Kaia's token economy functions and why it's an integral part of an overarching strategy for sustainable growth and innovation.

Deep Dive

Kaia's funding structure is tied to generating new blocks in its network. A "block reward" consisting of newly issued KAIA tokens and transaction fees is distributed for each new block (~1s). The rewards are allocated as follows: 50% goes to validators and the community, 25% to the Kaia Ecosystem Fund (KEF), and 25% to the Kaia Infrastructure Fund (KIF). Of the 50% designated for validators and the community, 20% (or 10% of the total) is given as block proposer rewards, while 80% (or 40% of the total) is for staking rewards. Kaia mints 6.4 KAIA tokens per block, resulting in about 200 million new tokens annually, with an annual inflation rate of approximately 5.2%. Transaction fees are applied according to predetermined fee tables.

Core Cell Operators & The Staking Community

In Kaia, Core Cell Operators (CCOs), who make up the Kaia Governance Council, play a crucial role in maintaining the network's infrastructure. To become a member of this council, candidates must pass a qualification review and stake a minimum of 5 million KAIA tokens.

Reward Mechanism

For each new block, a Committee is randomly formed from the Kaia Governance Council. This Committee consists of one member who acts as the Proposer and others who serve as Validators. Once a block is successfully added to the blockchain, the block reward is distributed as follows:

Proposer Reward: The Proposer, who is randomly selected, receives 10% of the block reward. This reward is evenly distributed among all Council members over time, ensuring that each has an equal opportunity to benefit from this role.
Staking Rewards: The remaining 40% of block rewards allocated to the Council (and staking community) are distributed based on the amount of KAIA staked by each member. Only the amount staked beyond the minimum 5 million KAIA requirement is considered for this proportional distribution. Members who stake more KAIA receive a larger share of these rewards, encouraging them to commit more resources to the network.

Council members can adjust their staking amounts freely, as long as they meet the minimum requirement. Staking data is updated every 86,400 blocks, with newly staked tokens becoming active two update cycles later. Unstaking requires a one-week delay, which helps prevent any sudden exits from the network by malicious actors.

Kaia Ecosystem Fund (KEF)

The Kaia Ecosystem Fund (KEF) is a crucial component of Kaia’s mission to enhance transparency and verifiability within its ecosystem. Funded by 25% of the total KAIA tokens issued with each new block, the KEF is dedicated to activities that bolster the Kaia network’s growth and sustainability.

Key Areas of KEF Utilization

The Kaia Ecosystem Fund (KEF) plays a vital role in incentivizing contributions to the Kaia network. It rewards developers and users who enhance the ecosystem's value and functionality through the Service Contribution Reward (SCR) program. This ensures that those who actively contribute to the network are recognized and encouraged to continue their efforts.

In addition to rewards, the KEF invests in building and strengthening the Kaia developer community. This includes supporting hackathons, educational programs, and collaborations with academic institutions and decentralized autonomous organizations (DAOs). These initiatives aim to foster innovation and attract talent to the Kaia ecosystem.

The KEF also funds the development of essential infrastructure and services, ensuring that projects with clear utility receive the necessary support. This includes both financial backing and marketing efforts to promote these services within the ecosystem.

Beyond direct support, the KEF engages in indirect investments through specialized crypto venture capital firms. These strategic investments are designed to generate returns that can be reinvested into the Kaia ecosystem or used to manage the token supply by burning tokens.

Finally, the KEF allocates resources to support governance committees focused on expanding Kaia's influence in areas like Gaming, DeFi, and Community. It also funds a wide range of activities aimed at nurturing and growing the broader Kaia community.

Governance and Administration

The Kaia Governance Council (GC) oversees the administration of the KEF. The Foundation submits a quarterly budget proposal for the fund, which is reviewed and approved by the GC. Specific uses of the funds are also subject to GC approval.

All fund allocations and expenditures are transparently disclosed to the community, and new proposals for KEF use can be submitted through the GC, requiring individual approval.

Kaia Infrustructure Fund

The Kaia Infrastructure Fund (KIF) is a crucial operational fund designed to support the ongoing development and maintenance of Kaia's ecosystem. It is funded by 25% of the total KAIA tokens issued with each new block. The KIF focuses on three main areas:

Mainnet and Essential Infrastructure R&D: This includes advancing research and development in mainnet technologies, creating foundational services, and establishing critical infrastructure needed for the Kaia network.
Ecosystem Acceleration: The KIF provides financial support for small-scale partners within the Kaia Blockchain ecosystem, facilitates token swaps, helps attract new members to the Governance Council, and provides market liquidity to enhance the ecosystem’s growth.
Foundation Operations: The fund covers operational expenses related to development, accounting, legal affairs, IT infrastructure, marketing, labor costs, and financial management, ensuring the smooth operation of the Kaia Foundation.

The KIF is managed by the Kaia Foundation, which establishes and executes budget plans through an internal control system. To ensure transparency, the foundation publicly announces budget plans and execution details before and after fund allocation, providing full disclosure to the community.

Challenges and Solutions

Kaia's token economy is designed to support a sustainable and adaptable ecosystem, but like any blockchain project, it faces certain challenges. Two significant issues are KAIA token inflation and the need for flexibility and adaptability in managing the token economy.

KAIA Token Inflation

One of the primary challenges is managing the inflation rate of KAIA tokens, with 200 million new tokens minted annually. This high inflation rate can increase selling pressure, potentially affecting the token's value in the market. However, Kaia has implemented several solutions to mitigate this issue.

First, 50% of the new tokens are allocated to the Kaia Ecosystem Fund (KEF) and the Kaia Infrastructure Fund (KIF). These funds are not immediately added to the active circulating supply but are instead held in reserve. They can only be activated through governance proposals, meaning that the tokens do not impact the market unless the community decides to utilize them for specific projects or initiatives. This approach helps manage inflationary effects and reduces immediate sell pressure on the market.

Additionally, Kaia employs a transaction-based burning mechanism as part of its strategy to control inflation. A portion of the transaction fees collected within the ecosystem is automatically burned, effectively reducing the circulating supply of KAIA. This burning process is carefully adjusted through consensus among key network participants to balance the reduction in supply with the incentives for node operators. By integrating this burn mechanism, Kaia further mitigates the inflationary pressures and supports the long-term sustainability and value of the KAIA token.

Flexibility and Adaptability of Kaia's Token Economy

Another challenge is ensuring that Kaia's token economy can adapt to the changing needs of the ecosystem and market conditions. To address this, Kaia has introduced a Treasury Rebalance mechanism that allows for adjustments to the allocation of funds within the ecosystem.

The Treasury Rebalance Contract is a smart contract that governs this process transparently and securely. It functions as a finite state machine, progressing through stages of initialization, registration, approval, and finalization. This structured approach ensures that any adjustments to the treasury are thoroughly reviewed and approved by the community.

Key features of this contract include the ability to adjust the total fund balance, aligning rebalance events with network upgrades, and ensuring transparent execution of the entire process on the blockchain. The rebalance is only executed after consensus is reached among validators, and the details of each event are immutably recorded for future reference and auditing. This mechanism allows Kaia to dynamically respond to the ecosystem's evolving needs while maintaining transparency and security.

By leveraging these practical solutions, Kaia can effectively manage inflation and maintain a flexible and adaptive token economy, ensuring the long-term stability and growth of its ecosystem.

Looking Ahead

Kaia's mainnet, a result of the merger between the Klaytn and Finschia ecosystems, has yet to be launched. As KLAY and FNSA merge into KAIA, the token economy will inherit and build upon the foundations of both ecosystems.

Looking ahead, Kaia's plans for its token economy include expanding participation in the Kaia Ecosystem Fund (KEF) by developing a structure that allows more ecosystem participants to propose and engage in the fund's utilization. For areas that require specialized and rapid decision-making, Kaia is also considering the establishment of separate governance committees. This approach aims to make the token economy more inclusive, flexible, and responsive to the needs of the ecosystem, ensuring that the community can actively contribute to and shape its development.

Conclusion

In conclusion, Kaia's token economy is meticulously designed to support both immediate and long-term network growth while managing inflation and fostering community involvement. The annual minting of approximately 200 million KAIA tokens, although significant, is carefully controlled through the allocation of 50% of these tokens to the Kaia Ecosystem Fund (KEF) and Kaia Infrastructure Fund (KIF). These tokens are not part of the active supply unless activated by community-driven governance proposals, which helps mitigate potential inflationary pressures on the token's market value.

Furthermore, the introduction of the Treasury Rebalance mechanism ensures that Kaia's token economy remains adaptable and responsive to evolving market conditions and ecosystem needs. This smart contract-based system enables flexible adjustments to the treasury's fund allocation, ensuring transparency and consensus-driven decision-making.

As Kaia looks to the future, it plans to enhance the inclusivity and agility of its token economy—through expanded participation in the KEF and the potential establishment of specialized governance committees—and underline its commitment to a sustainable, dynamic, and community-focused ecosystem.

References

Kaia native coin - KAIA: https://docs.kaia.io/learn/kaia-native-token/
KAIA Token Economy: https://docs.kaia.io/learn/token-economy/
Kaia Ecosystem Overview: https://flipsidecrypto.xyz/ellerydurwin/kaia-ecosystem-overview-O3VrbJ
Kaia Blockchain Performance and Consensus Metrics: https://flipsidecrypto.xyz/jorshimayor/kaia-blockchain-performance-and-consensus-metrics-bCivyK

About the Author

Deep Dive Into The Consensus Mechanism of Kaia Chain

cyril — Wed, 14 Aug 2024 21:32:02 +0000

Introduction

Imagine a blockchain that processes 4,000 transactions per second, with each transaction reaching finality within a second. This isn't a futuristic dream—it's the reality of Kaia. In a world where speed, security, and reliability are paramount, Kaia's optimized Istanbul Byzantine Fault Tolerant (BFT) consensus mechanism is setting new benchmarks.

This deep dive explores the intricacies of Kaia's consensus mechanism, a system designed not just for blockchain enthusiasts but for enterprises seeking performance and trustworthiness. We'll uncover how Kaia achieves immediate finality, why its consensus model matters in the broader blockchain landscape, and what you can expect to learn about the underlying technology powering this innovative network.

Whether you're a developer, an enterprise leader, or a crypto enthusiast, understanding Kaia's consensus mechanism will provide you with valuable insights into how cutting-edge blockchain technology is being harnessed to meet the demands of tomorrow.

Background

At the heart of any blockchain is its consensus mechanism—a process by which nodes in the network agree on the validity of transactions and the state of the ledger. This mechanism is crucial because it ensures that all participants maintain a synchronized and accurate copy of the blockchain, even in the presence of malicious actors or network failures.

Practical Byzantine Fault Tolerance (PBFT)

Kaia employs a variant of the Practical Byzantine Fault Tolerance (PBFT) consensus algorithm. PBFT is designed to handle situations where some nodes in the network may act maliciously or fail. The term "Byzantine fault" refers to a situation where components of a system fail and there is imperfect information on whether a component has failed. PBFT ensures that as long as more than two-thirds of the nodes are functioning correctly, the network can reach a consensus.

The consensus process involves electing a committee, including a proposer and validators. The elected proposer generates a block, which is then verified and signed by the committee. This mechanism allows Kaia to process 4,000 transactions per second with instant finality.

Deep Dive

Election of A Committee: VRF In Action

In Kaia’s consensus mechanism, the election of the committee is a critical process that ensures the blockchain's security, efficiency, and scalability. This process is executed using a Verifiable Random Function (VRF), a cryptographic tool that plays a vital role in selecting the nodes that participate in block generation and validation.

How the VRF Works in Committee Election

Council Formation

Kaia’s network consists of numerous Consensus Nodes (CNs), which collectively form what is known as the Council. The Council is responsible for maintaining the blockchain’s integrity by participating in block generation and validation.

Random Selection of Committee

For each block generation round, a subset of the CNs from the Council is selected to form the Committee. The selection is done in a random but deterministic manner, meaning that although the process is random, it can be verified by other nodes in the network to ensure fairness and transparency.

Committee’s Role

Once the Committee is selected, these nodes are tasked with proposing and validating the new block. The block proposer is chosen among the Committee members, and the remaining members validate the block by exchanging consensus messages. The VRF ensures that the selection of the Committee is both unpredictable and resistant to manipulation by malicious actors.

Significance and Impact of VRF-Based Committee Election

Scalability

The VRF allows Kaia to efficiently manage a large number of nodes by limiting the number of nodes that participate in the consensus process for each block. By randomly selecting a smaller Committee from a larger pool, Kaia reduces the communication overhead and the complexity of reaching consensus, which is crucial for maintaining high transaction throughput and fast finality.

Security and Fairness

The randomness introduced by the VRF makes it extremely difficult for any single entity or group to predict or influence the selection of the Committee. This enhances the security of the network by reducing the risk of targeted attacks on the nodes responsible for block generation.

Finality and Performance

Since consensus is reached by a small, dynamically selected group of nodes, the process is swift and efficient, enabling Kaia to achieve immediate finality for each block. This is essential for enterprise-grade applications where delays in transaction confirmation are unacceptable.

Communication Efficiency

By limiting consensus communication to only the Committee members, Kaia effectively reduces the communication volume that typically increases in Byzantine Fault Tolerance (BFT) systems as the number of participating nodes grows. This approach helps maintain network performance even as the network scales.

In the following sections, we shall take a look at how new blocks are proposed and validated by the Committee.

Block Generation by Proposer

Proposal Creation

At the beginning of each round, one member of the Committee is chosen as the block proposer. The chosen proposer gathers pending transactions from the transaction pool and compiles them into a new block. The proposer must ensure that the transactions are valid and that they meet the network’s rules (e.g., correct signatures, and sufficient gas fees).

The proposer then creates a block header that includes essential information such as the block number, timestamp, previous block hash, and a Merkle root of the transactions included in the block.

Block Broadcasting

Once the block is created, the proposer broadcasts it to the other Committee members for validation. The broadcasting is done through a dedicated propagation channel for blocks, which helps manage network congestion and ensures that the block is quickly received by all Committee members.

Block Verification & Signing by Committee

On Kaia, the process of block verification and signing by the Committee is a crucial step that ensures the integrity, security, and finality of each block added to the blockchain. Here’s a detailed exploration of this process.

Block Verification Process

Integrity Check

Data Integrity: Each Committee member first verifies the block’s integrity by checking its header and ensuring that the block hasn’t been tampered with. This includes verifying the block’s hash against the cryptographic hash included in the block header.
Transaction Integrity: Members also verify that all transactions included in the block are valid. This involves checking that transactions have correct signatures, sufficient funds, and adherence to the protocol’s rules.

Consensus Rules Compliance

Network Rules: Committee members ensure that the block complies with all network consensus rules. This might include checks for block size limits, transaction limits, and any other protocol-specific rules.
Previous Block Hash: The block must reference the correct hash of the previous block, ensuring that it fits properly into the blockchain’s existing structure.
Validation Feedback: If the block passes these checks, Committee members prepare to sign the block. If any issues are detected, the block is rejected, and the proposer may need to create a new block or correct the errors.

Signature Collection Process

Signing the Block
- Signature Creation: Once a block is validated, each Committee member generates a digital signature for the block. This signature is cryptographic proof that the member has validated the block and agrees with its content.
- Signature Transmission: Each Committee member sends their signature back to the proposer. This is done through secure channels to prevent tampering or interception.
Collecting Signatures
- Threshold Requirement: The proposer collects signatures from the Committee members. For the block to be finalized and added to the blockchain, it must receive signatures from more than two-thirds of the Committee members. This threshold ensures that the block has broad consensus among the Committee members.
- Finalization: Once the proposer has the required number of signatures, the block is finalized. The block is then propagated across the network through a dedicated transaction propagation channel, ensuring that all nodes update their copies of the blockchain.

Kaia’s block verification and signing process leverages a combination of cryptographic techniques and protocol rules to ensure that each block is valid, secure, and quickly finalized. This approach balances the need for high performance with strong security, making Kaia a compelling choice for both decentralized applications and enterprise solutions.

Challenges and Solutions

Challenge 1: Network Congestion

Issue

Kaia’s consensus mechanism, while designed to handle high transaction throughput, can face network congestion during peak periods. This congestion can impact the efficiency of block propagation and transaction validation, leading to delays in finalizing transactions and increased latency.

Solution

To mitigate network congestion, Kaia employs a multi-channel approach for block and transaction propagation. Ensuring that these channels are well-optimized and capable of handling high traffic volumes is crucial. Additionally, implementing dynamic load-balancing techniques can help distribute network traffic more evenly. Regular network performance monitoring and scaling the network infrastructure as needed can also alleviate congestion issues.

Challenge 2: Committee Selection Bias

Issue

Although the VRF is designed to ensure fairness in committee selection, there is a potential risk of selection bias if the VRF implementation is flawed or if there are vulnerabilities in the random number generation process. Such biases could lead to an uneven distribution of power among nodes or reduced security.

Solution

Regularly auditing the VRF implementation for vulnerabilities and ensuring its cryptographic robustness is essential. Additionally, incorporating randomness from multiple sources and cross-checking VRF results can enhance fairness. Implementing a transparent process for selecting committee members, with verifiable records, can also help maintain trust in the selection process.

Challenge 3: Scalability with Increasing Node Count

Issue

As the number of Consensus Nodes in the network grows, the communication volume among nodes can increase, potentially leading to inefficiencies in the consensus process and slower block propagation.

Solution

Kaia’s solution to this issue lies in the fixed-size Committee used for block verification and signing. Instead of involving all CNs in the consensus process for each block, Kaia elects a fixed number of Committee members for each round. Hence, increasing the number of CNs does not affect the communication volume among Committee members.

By addressing these common challenges with practical solutions, Kaia enhances its consensus mechanism's robustness, efficiency, and reliability, ensuring it meets the demands of a growing and dynamic blockchain ecosystem.

Conclusion

In conclusion, Kaia employs an advanced consensus mechanism based on an optimized version of Istanbul BFT, which incorporates Practical Byzantine Fault Tolerance (PBFT) with specific modifications for blockchain networks. This approach is designed to provide rapid transaction finality and high throughput, essential for enterprise-grade applications. The consensus process involves a fixed-size Committee elected through a Verifiable Random Function (VRF), which ensures fairness and manages communication volume efficiently.

In Kaia’s mechanism, block generation begins with a randomly selected proposer from the Committee who creates and broadcasts a block. The block is then validated by the Committee members, who check its integrity and compliance with network rules. Once validated, the Committee members sign the block, and the proposer collects the required signatures from more than two-thirds of the Committee to finalize it. The finalized block is propagated through separate channels for blocks and transactions, optimizing network performance and managing congestion.

Kaia effectively addresses common blockchain challenges such as network congestion and communication volume by utilizing a fixed-size Committee. This design keeps communication overhead consistent even as the number of Consensus Nodes grows, ensuring efficient and scalable performance. The integration of VRF for Committee selection and multi-channel propagation methods highlights Kaia’s commitment to balancing high throughput with robust security and reliability.

References

Byzantine Fault: https://en.wikipedia.org/wiki/Byzantine_fault
Kaia Overview: https://docs.kaia.io/learn/
Kaia Consensus Mechanism: https://docs.kaia.io/learn/consensus-mechanism/
The Istanbul BFT Consensus Algorithm: https://arxiv.org/pdf/2002.03613

About the Author

How To Implement Governance Proposals on Polkadot

cyril — Mon, 05 Aug 2024 12:59:48 +0000

Introduction

Polkadot, a groundbreaking blockchain platform, empowers its community to shape the network's future through a robust governance system. At the heart of this system lies the ability to propose and implement changes to the protocol. Governance proposals leading to referenda are the vehicles for introducing new features, modifying existing ones, or addressing critical issues within the Polkadot ecosystem. This process is designed to be inclusive, transparent, and community-driven, ensuring that the network evolves in alignment with the collective vision of its stakeholders.

By participating in OpenGov, Polkadot's governance process, token holders can actively contribute to the network's development, making informed decisions about its direction. The implementation of governance proposals involves a series of well-defined stages, from the initial proposal submission to the final execution of approved changes. Understanding the mechanics of this process is crucial for anyone seeking to influence the trajectory of the Polkadot ecosystem.

This post aims to provide a comprehensive overview of OpenGov's Referenda lifecycle on Polkadot, covering key aspects such as creating discussion posts and submitting and implementing proposals. By demystifying the process, this guide empowers community members to effectively engage in governance and contribute to the ongoing success of Polkadot.

Background

The term "governance" originates from the Latin word "gubernare" meaning "to steer". Historically, governance was centralized, akin to steering a ship with a single captain. However, the advent of decentralized systems challenges this traditional top-down approach.

Decentralization, by distributing control and decision-making across multiple entities, necessitates a reimagining of governance as a collaborative process rather than a hierarchical command structure.

In Polkadot OpenGov, referenda are the primary mechanism for steering the network's direction. Similar to voting on a proposal, referenda allow token holders to directly influence the platform's evolution. These referenda are categorized into different tracks based on the nature and impact of the proposed changes, ensuring a tailored approach to decision-making.

Unlike traditional governance models, Polkadot doesn't have a fixed quorum requirement. Instead, the passage of a referendum depends on achieving specific approval and support percentages, tailored to the significance of the proposal. This flexible approach ensures that critical decisions require a higher level of consensus while allowing for quicker decisions on less impactful matters.

Deep Dive

PolkAssembly

Login

In Polkadot OpenGov, proposals become referenda through the PolkAssembly dApp, designed to make it easy for the community to get involved in each stage of the process in just a matter of clicks.

Before we create a referendum, we'll have to login with our wallet provider. Click the Login button on the top right corner, and select your preferred wallet provider.

After selecting your wallet provider, you'd have to consent to log in again by clicking Got it and subsequently approving authentication and signature requests

Once you've signed the signature requests, you'll be presented with a success page and prompted to enter a preferred username.

Provide a username and click Next which takes you to a modal that prompts for an email address and a password. This stage is optional and you could Skip.

The PolkAssembly homepage should be updated, and your username displayed in the top-right corner.

Setting Up a Verified On-Chain Identity on PolkAssembly

Polkadot empowers users to create verified digital identities linked to their accounts. This process involves sharing personal information and obtaining confirmation from authorized registrars.

A verified on-chain identity boosts credibility in Polkadot governance, including referendum proposals. Platforms like PolkAssembly simplify the creation and management of these identities.

This section walks you through the steps to establish and verify your on-chain identity.

Understanding the Process

To establish a verified identity, you'll need to provide personal information such as your name, email, and X.com username. A registrar will review these details to confirm their accuracy.

Required Funds and Fees

Some DOT is required for deposit and registrar fees. The exact amount depends on the information you provide, and you'll need sufficient funds to complete this process successfully.

Steps to Create Your Identity

Locate your profile name in the top right corner and select Set on-chain identity from the dropdown menu.

Choose the account you want to use for your on-chain identity and click Confirm.

The following screenshot provides details about setting up an on-chain identity, including fees and required deposits for the People Chain. Click Let's Begin to start the process.

The next screen lets you manage your on-chain identity details. Here, you can enter your display name, legal name, email, and X.com username.

Conveniently, this window also offers a Teleport Funds option at the top, allowing you to seamlessly transfer the necessary funds to the People Chain before proceeding.

Click Set Identity after completing the required details and approve the signature request from the wallet.

Once your transaction is confirmed, you can proceed to the next step: verification.

Verify your information by clicking the Verify buttons and following the on-screen instructions.

Once complete, click Proceed to submit your identity for review. Please allow some time for PolkAssembly to assess your information.

Upon approval, your verified identity will be publicly displayed with a green check mark next to your profile.

Congratulations!, now you have a verified on-chain profile! The next section takes a look at creating a discussion post, one of the prerequisites to proposing a referendum.

Adding A Discussion Post

Before submitting a referendum proposal on OpenGov, a discussion post is mandatory. This initial step fosters community engagement, allowing for feedback, suggestions, and collaboration. By initiating a discussion, proposers can refine their ideas, gain support, and increase the chances of their proposal gaining traction within the community before formally submitting it as a referendum.

To add a discussion post, click the Discussions navigation in the left menu panel and click on Add Post in the top right corner.

In the new interface, please enter the discussion title in the title box, and in the textarea below, thoroughly describe the proposal, its implications, and benefits to enable community members to make an informed decision.

In the subsequent form elements, choose whether you want to add a poll to the discussion, select a topic, or add a tag. You can also open comments on the post to all users, verified users, or even decide to close commenting completely.

Once you're satisfied with the information provided, click Create Post which should have your post published as a discussion on the PolkAssembly platform.

The next section will look at how to propose a referendum on OpenGov. In the meantime, copy the link to the discussion post you just published, it will come in handy.

Proposing a Referendum

To propose a referendum, navigate back to the Overview tab, hover over the three-dots icon at the top-right corner, and click Create Referendum.

This will lead to a new modal where you'll be presented with your linked accounts. Make sure you have at least 40 DOT in your linked wallet to be used as a submission deposit. The refundable deposit is required to submit a public referendum proposal.

Click Next to proceed. In the next modal, select Yes as the answer to the question: Have you initiated a discussion post for your proposal already?. Paste the link to the discussion post and decide who can comment on the referendum.

Next, provide details on the specific outcome desired for the referendum if successful. Then decide on which pallet the referendum falls under and the applicable method. In the example below, we're requesting 200 DOT from the Treasury. For more information on pallets, check out this guide.

At this point, select the applicable type of origin for the chosen Pallet. In OpenGov, referenda are categorized into different Origins and Tracks.

Origins and Tracks

Polkadot's governance system operates on a dual structure of Origins and Tracks. Origins represent different levels of authority within the network, each possessing specific privileges. When submitting a proposal, choosing the correct Origin is crucial as it determines the proposal's execution capabilities.

Tracks, on the other hand, define the specific rules and parameters for a particular type of proposal. These include factors such as voting duration, required approval thresholds, and the number of concurrent referenda. By combining Origins and Tracks, Polkadot creates a flexible and adaptable governance framework that accommodates various types of proposals and decision-making processes.

In the next subsections, we'll take a look at the 34 Origins of OpenGov and explain their varying levels of authority.

Root

The Root origin holds the highest level of authority within the Polkadot network. It is reserved for the most critical and impactful changes. Referendums under this origin require an exceptionally high level of approval and support from the community to pass. Due to its significance, only one Root referendum can be active at a time.

Whitelisted Caller

This origin is used for time-sensitive proposals that require expedited processing. It allows for a shorter voting period compared to other tracks. However, it demands a specific level of support to pass, ensuring a certain level of consensus within the community.

Wish For Change

Unlike other origins, Wish For Change is primarily for gathering community sentiment rather than making immediate changes to the network. Proposals under this track serve as a signal for potential future developments and help gauge community interest.

Staking Admin

As the name suggests, the Staking Admin origin is responsible for managing staking-related operations. This includes handling slashes, which are penalties imposed on validators for misconduct.

Treasurer

The Treasurer origin oversees the management of the Polkadot treasury. It has the authority to approve or reject spending proposals from the treasury.

Lease Admin

This origin is responsible for managing slot leases, which are essential for parachains to secure their position on the Polkadot relay chain.

Fellowship Admin

The Fellowship Admin origin is dedicated to managing the composition of the Fellowship, a group of elected representatives responsible for various governance tasks.

General Admin

The General Admin origin oversees the management of the registrar and permissioned HRMP channels, which are crucial for interoperability between parachains.

Auction Admin

The Auction Admin origin is responsible for initiating and managing auctions for parachain slots.

Referendum Canceller

This origin has the power to cancel pending referendums under specific conditions.

Referendum Killer

This origin can cancel an ongoing referendum and impose penalties on those who initiated it.

Small Tipper, Big Tipper, Small Spender, Medium Spender, Big Spender

These origins manage different spending limits from the treasury. They allow for various levels of spending without requiring a full-fledged referendum for smaller amounts.

Next, choose the applicable Origin for the referendum in the Pallet and Method dropdown boxes.

After setting everything up according to your preference, including the right pallet and method, click on Create Referendum and you should see a successful message.

Implementing a Referendum

After a successful proposal, our referendum gets displayed on the overview page on PolkAssembly with a Submitted tag. This means that the proposal has been published, but we have to satisfy one last requirement before our referendum can be moved to the decision phase.

For the referendum to move from the preparing phase to the deciding phase, a decision deposit needs to be placed. The decision deposit values for each Treasury Track are listed in this document.

Now, to finish off with the implementation, open the proposal page and click on Pay to make the deposit.

After a successful deposit, you should see the Submitted tag change to a green Decision tag. The decision phase allows time for a decision to be approved to move to the confirming period. If the proposal is not approved by the end of the decision period, it gets rejected.

Challenges and Solutions

Polkadot's OpenGov system faces only a few challenges. The introduction of multiple tracks and origins can increase complexity for users, requiring them to navigate different rules and processes. While the existing 40 DOT submission deposit requirement and proposal curation system help to some extent, the open nature of proposal initiation can still lead to a flood of low-quality or malicious proposals, especially when the price of DOT to levels that make the deposit insignificant. This dilutes the focus on important initiatives and can undermine community trust.

Looking Ahead

OpenGov represents a significant evolution in Polkadot's governance model, aiming to create a more inclusive, efficient, and decentralized system for decision-making. As the system matures, it is expected to continue evolving based on community feedback and the changing needs of the network.

Conclusion

Polkadot's OpenGov system offers a robust framework for community-driven decision-making. By understanding the steps involved in proposing and implementing governance proposals, token holders can actively participate in shaping the network's future.

While challenges exist, such as complexity and the potential for low-quality proposals, ongoing improvements, and community engagement are essential for optimizing the governance process. As Polkadot continues to evolve, OpenGov will play a pivotal role in ensuring the network's long-term success and alignment with the collective vision of its stakeholders.

References

Introduction to Polkadot OpenGov: https://wiki.polkadot.network/docs/learn-polkadot-opengov
Polkadot OpenGov Origins: https://wiki.polkadot.network/docs/learn-polkadot-opengov-origins
Approval and Support: https://wiki.polkadot.network/docs/learn-polkadot-opengov#approval-and-support

About the Author

How To Perform Cross-Chain Token Transfers with Polkadot XCM

cyril — Sat, 03 Aug 2024 17:03:44 +0000

Introduction

Web3 is expanding into a multi-layered ecosystem of consensus systems spanning diverse blockchains and smart contract applications. This diversity in blockchain architecture and operation highlights the need for interoperability within the ecosystem.

Consensus systems compete and differentiate themselves based on security, scalability, and decentralization. Some aim for speed with a high-throughput monolith while others focus on scalability and decentralization via layer 2 chains. However, all these diverse consensus systems must be interoperable for the vision of a decentralized web to be achieved.

Background

Interoperability between chains of different consensus mechanisms is a very arduous endeavor. This is evident in malicious actors exploiting more interoperability solutions over the years. Polkadot, a pioneer in blockchain technology, provides a secure way of cross-chain communication through the Cross-Consensus Message (XCM) format and the Cross-Consensus Messaging Parsing (XCMP).

XCM is the message format and language used to communicate between consensus systems. Whereas XCMP is the network-layer protocol that facilitates XCM communication. XCM, although commonly used in the Polkadot ecosystem, can be used in any consensus system due to its generic and common format.

Polkadot XCM is crucial in today's tech landscape as it facilitates seamless interoperability between disparate blockchains. By providing a standardized communication protocol, XCM enables the transfer of assets, data, and messages across different networks, fostering a more interconnected and collaborative ecosystem.

Deep Dive

Core Structure of XCM

XCM leverages MultiAssets to represent various assets across blockchains, identified by MultiLocations. A Holding Register temporarily stores assets during message execution, while Weight signifies the computational resources needed to process the XCM. An XCM message typically follows this structure:

 let message = Xcm(vec![
    WithdrawAsset((Here, amount).into()),
    BuyExecution{ fees: (Here, amount).into(), weight_limit: WeightLimit::Unlimited },
    DepositAsset {
        assets: All.into(),
        beneficiary:  MultiLocation {
            parents: 0,
            interior: Junction::AccountId32 {
                network: None,
                id: BOB.clone().into()
            }.into(),
        }.into()
    }
]);

The message consists of three instructions: WithdrawAsset, BuyExecution, and DepositAsset.

WithdrawAsset: This instruction removes assets from the sender's account and places them in a temporary holding register.
BuyExecution: This instruction pays for the execution of the XCM message using assets from the holding register.
DepositAsset: This instruction transfers assets from the holding register to the recipient's account on the target chain.

When the three instructions are combined, we withdraw the number of native tokens from the account of Alice, pay for the execution of these instructions, and deposit the remaining tokens in the account of Bob in the same consensus system.

Now let's look at ways to transfer data or assets between accounts on different blockchains utilizing different consensus systems.

Cross-Chain Transfers: Asset Teleportation

Asset teleportation is one of the two methods for sending assets from one chain to another. It has only two actors, the source and the destination.

The source chain removes assets from circulation, creates an XCM instruction specifying the recipient and amount, and sends it to the destination chain. The destination chain then introduces equivalent assets into its circulation and deposits them into the specified account. This process requires high trust between chains as asset preservation relies on both chains accurately managing their respective circulating supplies.

Let's take a look at an example of an Asset Teleportation XCM:

let message = Xcm(vec![
  WithdrawAsset((Here, teleport_amount).into()),
  InitiateTeleport {
    assets: All.into(),
    dest: Parachain(1).into(),
    xcm: Xcm(vec![DepositAsset {
      assets: All.into(),
      beneficiary: Junction::AccountId32 {
        network: None,
        id: ALICE.into(),
      }
    }]),
  },
]);

Now let's break down the various parts of the message.

InitiateTeleport is an XCM instruction executed on the source chain. It removes assets matching the specified filter from the holding register and constructs a new XCM message.

InitiateTeleport { assets: MultiAssetFilter, dest: MultiLocation, xcm: Xcm<()> }

This message includes a ReceiveTeleportedAsset instruction for the destination chain, followed by the original XCM instructions. The destination chain will automatically process these instructions, effectively teleporting the assets.

ReceiveTeleportedAsset is a trusted XCM instruction executed on the destination chain. It introduces new assets into circulation, populating the holding register.

ReceiveTeleportedAssets(MultiAssets)

This instruction requires strict origin verification to prevent unauthorized asset creation. The destination chain can configure allowed teleporter origins to ensure security. Once assets are in the holding register, subsequent instructions, like DepositAsset, can be distributed to recipient accounts.

ClearOrigin resets the message's origin, preventing subsequent instructions from acting on behalf of the previous sender. This is crucial for the security and isolation of XCM message execution contexts. The InitiateTeleport instruction typically uses ClearOrigin to ensure the destination chain processes the teleported assets independently.

Next, we'll look at another way to achieve cross-chain interoperability using reserve-backed transfers.

Cross-Chain Transfers: Reserve-Backed Transfers

Reserve-backed transfers involve a trusted third-party reserve to facilitate cross-chain asset movement. Instead of directly transferring assets, chains mint derivative tokens representing their claims on the reserve. The source chain burns derivative tokens when transferring assets, triggering a withdrawal from its reserve account.

The reserve then deposits the withdrawn assets into the destination chain's reserve account. Finally, the destination chain mints equivalent derivative tokens for the recipient. This method reduces trust requirements between chains but introduces reliance on the reserve's integrity.

For instance, transferring tokens from Alice's account on parachain 1 to her account on parachain 2, with the relay chain acting as the reserve, would involve constructing an XCM message to initiate the transfer process. This message would encompass instructions for burning derivative tokens on parachain 1, withdrawing the corresponding amount from the relay chain's reserve, depositing the withdrawn funds into Alice's account on parachain 2, and finally, minting new derivative tokens on parachain 2.

let message = Xcm(vec![
  WithdrawAsset((Parent, amount).into()),
  InitiateReserveWithdraw {
    assets: All.into(),
    reserve: Parent.into(),
    xcm: Xcm(vec![DepositReserveAsset {
      assets: All.into(),
      dest: Parachain(2).into(),
      xcm: Xcm(vec![DepositAsset {
        assets: All.into(),
        beneficiary: AccountId32 { id: ALICE.into(), network: None }.into(),
      }]),
    }]),
  },
]);

The provided XCM message outlines a reserve-backed asset transfer from one parachain to another, utilizing the relay chain as a reserve. Let's break this down:

InitiateReserveWithdraw triggers the reserve-backed transfer process. It burns the derivative asset. Sends an XCM message to the relay chain (specified as the reserve).

InitiateReserveWithdraw { assets: MultiAssetFilter, reserve: MultiLocation, xcm: Xcm<()> }

The embedded XCM message contains a WithdrawAsset instruction to withdraw the underlying asset from the source chain's sovereign account, A ClearOrigin instruction ensures isolated execution on the relay chain, and a DepositReserveAsset instruction to transfer the withdrawn asset to the destination chain's sovereign account.

The DepositReserveAsset instruction serves a dual purpose. It transfers the withdrawn assets to the destination chain's sovereign account, essentially storing them on behalf of the destination.

DepositReserveAsset { assets: MultiAssetFilter, dest: MultiLocation, xcm: Xcm<()> }

It sends a new XCM message to the destination chain, informing it of the deposited assets and instructing it to mint equivalent derivative tokens. This message includes a ReserveAssetDeposited instruction for trust verification and a DepositAsset instruction to credit the recipient's account.

The ReserveAssetDeposited instruction signifies that the reserve chain has successfully transferred assets to the destination chain's sovereign account. Upon receiving this notification, the destination chain mints equivalent derivative assets and credits them to the specified recipient account.

This instruction requires trust in the reserve chain as it confirms the asset transfer without direct verification. To mitigate risks, the destination chain can configure which chains are trusted as reserves for specific assets.

Challenges and Solutions

Fee Handling In XCM

XCM messages require fee payments to execute. The BuyExecution instruction is used to purchase the necessary weight for message execution. Accurate fee estimation is crucial to prevent message failures.

Here are some fee payment mechanisms you can utilize:

Prepaid Fees: The standard approach involves including a BuyExecution instruction with sufficient funds to cover estimated execution costs.
Direct Withdrawal: The SetFeesMode instruction allows fees to be directly withdrawn from the sender's account, useful for unpredictable fee scenarios.
Unpaid Execution: The UnpaidExecution instruction can be used for privileged messages that don't require fee payments, but require careful configuration and trust in the message origin.

Fee Management and Error Handling

The RefundSurplus instruction can be used within error handlers to reclaim unused weight if a message fails to execute completely. Accurate weight estimation is essential to avoid overpaying fees. By effectively managing fees and handling potential errors, XCM messages can be executed reliably and efficiently.

let message = Xcm(vec![
  WithdrawAsset((Parent, message_fee).into()),
  BuyExecution {
    fees: (Parent, message_fee).into(),
    weight_limit: WeightLimit::Unlimited,
  },
  SetErrorHandler(Xcm(vec![
    RefundSurplus,
    DepositAsset {
      assets: All.into(),
      beneficiary: AccountId32 {
        network: Some(ByGenesis([0; 32])),
        id: relay_sovereign_account_id().into(),
      }
      .into(),
    },
  ])),
  Trap(1),
  ClearOrigin,
  ClearOrigin,
  ClearOrigin,
]);

In the example above, we pay upfront for all the instructions in the XCM. When the Trap instruction throws an error, the error handler will be called and the weight for all the instructions that weren't executed is refunded.

Conclusion

This article explored the core structure of XCM messages, the concept of MultiAssets and MultiLocations, and the Holding Register. We delved into the mechanics of asset teleportation and reserve-backed transfers, highlighting the trust assumptions involved in each method. Finally, we discussed the importance of fee handling in XCM messages and explored different fee payment mechanisms.

By understanding these core concepts, developers can construct secure and interoperable cross-chain applications using XCM. Want to delve deeper? The XCM documentation provides comprehensive examples to illustrate various XCM use cases in detail: https://github.com/polkadot-fellows/xcm-format

This concludes our exploration of Polkadot's XCM, the powerful language facilitating seamless communication between blockchains.

References

Introduction to Cross-Consensus Message Format (XCM): https://wiki.polkadot.network/docs/learn-xcm
XCM: Cross-Consensus Messaging: https://wiki.polkadot.network/docs/learn/xcm/introduction
The XCVM: https://wiki.polkadot.network/docs/learn/xcm/overview-xcvm