Forem: Aseneca

Transaction Message Header: An ELI5 Into the Transaction Message Header Semantics

Aseneca — Sun, 13 Apr 2025 13:39:08 +0000

A transaction is a set of signatures of serialized messages signed by the first key of the Message’s account key.

The message in a transaction is a structure containing a header, account_keys, recent_blockhash and instructions. The header contains the MessageHeader which describes the organization of the Message’s account keys.
Every instruction specifies which accounts it may reference, or otherwise requires specific permissions of. Those specifications are: whether the account is read-only, or read-write; and whether the account must have signed the transaction containing the instruction.

Whereas individual Instructions contain a list of all accounts they may access, along with their required permissions, a Message contains a single shared flat list of all accounts required by all instructions in a transaction. When building a Message, this flat list is created and Instructions are converted to a set of CompiledInstructions. Those CompiledInstructions are then reference by index the accounts they require in the single shared account list.

The shared account list is ordered by the permissions required of the accounts:

accounts that are writable and signers
accounts that are read-only and signers
accounts that are writable and not signers
accounts that are read-only and not signers

Given this ordering, the fields of MessageHeader describe which accounts in a transaction require which permissions.

When multiple transactions access the same read-only accounts, the runtime may process them in parallel, in a single PoH entry. Transactions that access the same read-write accounts are processed sequentially.

Message Header

The MessageHeader describes the organization of a transaction Message’s account keys. It’s structure is defined as:

pub struct MessageHeader { pub num_required_signatures: u8, pub num_readonly_signed_accounts: u8, pub num_readonly_unsigned_accounts: u8, }
Where num_required_signatures represents the number of signatures required for this message to be valid, num_readonly_signed_accounts represents the last num_readonly_signed_accounts that are read-only signers, while num_readonly_unsigned_accounts represents the last num_readonly_unsigned_accounts that are read-only non-signers..

By reading these three values in the MessageHeader and the account_keys array in the Message, the runtime can by index, deterministically:

Know which accounts are signer vs non-signers
Know which accounts are read-only vs writable

Now consider a reverse scenario where instead of the number of read-only signatures, we have a number of writable signers? That is, simply flipping the logic to define the MessageHeader like this:

pub struct FlippedMessageHeader { pub num_required_signatures: u8, pub num_writable_signed_accounts: u8, pub num_writable_unsigned_accounts: u8, }

This would also work in theory but compared to the original design has some performance implications.

From the flipped logic of message header, the runtime would have to infer a couple of things:

The first num_writable_signed_accounts in account_keys[0..num_required_signatures] are writable.
The rest are read-only signers.
The first num_writable_unsigned_accounts in the rest are writable non-signers.
The rest are read-only non-signers.

However with this approach, we lose the suffix parsing behavior in the original design.

With the original design, Read-only accounts are at the end of their respective signer/non-signer group which makes range slicing easy:
account_keys[0..n] -> all signers

account_keys[n..] -> all non-signers

Within each, the tail can be used to mark "read-onlys"

With flipped logic:

Writable accounts are at the front, so we would need to slice the beginning of each group and then infer "read-onlys" as a suffix after skipping "writables" (prefix-based parsing).

Consider this analogy further. Say we have account_keys= [A, B, C, D, E, F], and the header:
num_required_signatures = 3 num_readonly_signed_accounts = 1 num_readonly_unsigned_accounts = 1
From this example,
Signers: [A, B, C]
Last one of them (C) -> read-only signer
First two of them (A, B) -> writable signers

Non-signers: [D, E, F]
Last one of them (F) -> read-only non-signer
First two of them (D, E) -> writable non-signers

Slicing can be done cleanly and easily based on suffix counts like this:
writable_signers = account_keys[0..(3 - 1)] = [A, B] readonly_signers = account_keys[(3 - 1)..3] = [C] writable_non_signers = account_keys[3..(6 - 1)] = [D, E] readonly_non_signers = account_keys[5..6] = [F]

Now if we go along with the flipped logic and assume the header stored:
num_required_signatures = 3 num_writable_signed_accounts = 2 num_writable_unsigned_accounts = 2

To find read-onlys, now we have to compute:
First two in the signer section -> writable The rest -> read-only First two in the non-signer section -> writable The rest -> read-only
So for read-only accounts, you have to offset forward rather than slice from the end.

This means we can’t just slice_from_end(count) anymore and we now need to compute and track forward ranges like:
readonly_signers = account_keys[num_writable_signed_accounts..num_required_signatures]
Which gets more awkward if counts don’t line up cleanly or if you add more categories.

Summary

While it is quite semantically logical to flip the logic in theory, in practice there would be some performance trade-off to that effect. By assuming "writability" by default and counting only the exception, the runtime not only minimizes bytes encoding but also encourages a compact, fast, suffix-parsed, position based inference.
Also with suffix-based parsing (read-only last), we can just slice from the end which is much simpler and cleaner that prefix-based parsing (writable first) where we would need to offset forward which is more difficult.

References

www.anza.xyz
https://dl.acm.org/doi/pdf/10.1145/233551.233555

Ironforge: A Comprehensive DevOps Platform for Web3 Applications

Aseneca — Mon, 02 Dec 2024 14:47:34 +0000

As the web3 development space accelerates at incredible pace towards achieving decentralization, increased bandwidth and reduced latency, an important need that arises is core infrastructure support enabling developers to improve development lifecycle, integration and reliability while ensuring continuous performance observability. This is the focus of Ironforge team- to provide the best RPC Orchestration and Observability platform in web3.

What is Ironforge?

Ironforge is the leading DevOps platform on Solana, offering serverless infrastructure for developers. Ironforge aims to equip Web3 developers with tools that enable them focus on building while providing infrastructure management through solutions such as observability, incident management, networking, security enterprise solutions such as automation, streamlined organization and project management. Ironforge is trusted and used by most large-scale teams including Jupiter, Kamino Finance, Switchboard, Solflare, Orca, Bonk and many others. It is also streamlined to serve individual developers as well, and provide a dynamic and interactive UI/UX.
With Ironforge, individual developers and teams can become more efficient by simplifying application development, truncate overhead in project organization process, and avoid complexities of redundant tasks.

Key Features

Ironforge provides efficient DevOps operations by offering several unique features:

Advanced Intelligent RPC Routing and Management

Ironforge offers two distinct Routing methods Primary and Dynamic Routing Methods to tailor application’s RPC requests to its specific needs. With Ironforge, projects can analyze application’s behaviour, identity issues and optimize performance within a single platform.
Primary Routing is the default routing strategy for all RPC methods while Dynamic Routing gives developers more customizable options to assign specific endpoints for specific RPC methods therefore enabling them control how requests are routed based on the RPC method. Ironforge also provides dedicated RPC endpoints monitoring for each RPC endpoint, equipping developers with real-time transaction slot latency data to make fast informed decisions.

Alerts

Ironforge provides a real-time monitoring system, enables developers to set up alerts to monitor specific application health and performance metrics and receive quick feedback through notifications when issues arise.

In-depth Application Observability

Developers need analytics and logs data to understand applications dynamic behavior and potential. Ironforge aims to provide native SAS (Statistical Analysis System) tooling for web3 application performance monitoring to provide comprehensive advanced analytics, business intelligence, data management, predictive analysis and logs.

Load Balancers

Ironforge Load Balancers enable developers to efficiently distribute RPC requests across multiple Solana endpoints. Developers can configure multiple RPC endpoints and select load balancing algorithms and routing strategies that are apt for their application needs. This feature provides two routing strategies, Primary and Dynamic, which allows developers to specifically configure the route of every RPC method call, giving them more autonomy over how requests are distributed across RPC endpoints.
With each routing strategy developers can set up load balancing algorithms such as Sequential and Parallel algorithms for RPC endpoints.

Sequential Load Balancing Algorithm

In the sequential load balancing algorithm, Ironforge forwards requests to the first available RPC endpoint in the configured list. If that endpoint responds successfully, it is used. If for some reason that endpoint is unavailable or returns an error, the request is forwarded to the next available endpoint in the sequence and so on until a successful response is obtained or all endpoints have been attempted. Developers have control over the sequence of endpoints through which the requests should be processed.

Parallel Load Balancing Algorithm

This algorithm enables Ironforge to send requests simultaneously to all available endpoints and uses the response from the first endpoint that responds successfully, while discarding the responses from the other endpoints. With this algorithm, developers can utilize the power of a parallel processing structure to reduce response time and maximize performance but there is a tradeoff for less customization control and flexibility. Also, the parallel load balancing algorithm may incur higher costs whenever there is increased utilization.

Firewall Rules

Ironforge ensures developers can focus on building without worrying about application security. It does this through its Firewall Rules feature which provide benefits such as rate limiting, IP and domain whitelisting, RPC method blocking and more. The Firewall feature allows developers to define a set of conditions that must be met before a request is allowed to reach an RPC endpoint. These rules(which are setup with conditions) as well as the actions to take if the conditions are met, can be configured to meet the developers specific needs.
The Firewall feature can be configured in two different ways:

Default mode: All requests are allowed by default and developers can create rules to block specific requests.
Firewall Mode: This is the inverse of the default mode. In this mode, all requests are blocked by default and developers can create rules to allow specific requests. Firewall rules consist of conditions that must be met before a specific action is taken. Ironforge provides great levels of customization and conditions for Firewalls, allowing developers to create complex rules that meet their requirements.

Comprehensive Transaction Observability

Ironforge enables developers to gain deeper insight on transaction landing rates, transaction latency, CU units and pricing, priority fees, transaction confirmation rates, transaction drop rates, error logs on transactions. It provides detailed analytics, metrics, logs and comprehensive reports on transactions which is useful in monitoring and identifying issues with transactions and provide insights on how developers can improve transaction success rate and save money by optimizing transactions.

Highly Intuitive UI/UX

Ironforge also implements a very intuitive UI/UX and interactive dashboard to streamline the user experience. Aside from these highly interactive and visualized dashboards for managing and monitoring features such as Alerts, Firewalls, Load balancer and much more, it also provides an interactive timeline view for monitoring and analyzing events in real-time. This feature enables developers to effectively manage systems by visualizing and filtering real-time event data.

Prominent Use Cases and Impact

Being the leading DevOps operation platform on Solana, Ironforge is suitable for projects ranging from small dApps to enterprise-level platforms. It is trusted by many large-scale teams to enhance scalability, reliability and security of their applications. Some of the prominent use cases and impact include:

Load balancing and failover capabilities: distribute requests across multiple RPC providers therefore minimizing downtime and maximizing throughput. By rerouting traffic through intelligent strategies during outages and congestion, Ironforge ensures high transaction success rates.
Transaction Management and Observability: retry mechanisms, state confirmations and low-latency execution ensure fast and consistent transaction processing. With the Ironforge transaction observability module, projects can gain insights into how much transactions are costing them and how much they can save if CU units and pricing are optimized.

Observability, Monitoring and Real-time Analytics: Utilizing intuitive UI/UX, interactive dashboards and sas tooling, Ironforge provides insights into RPC performance, application’s health, transaction patterns, network health and other critical metrics for proactively scaling and maintaining dApps, identifying issues, optimizing performance and making informed decisions. It also provides customizable alerts for real-time events which can be integrate with tools like Slack, Discord or PagerDuty.
Robust Security: Ironforge provide teams with customizable options to enhance their security requirements through firewall rules. Projects can whitelist domains, IP addresses for safer access and block unnecessary Solana RPC methods to improve performance.
Severless Solana Development: manage and facilitate backend integrations, therefore enabling developers to focus on building rather than infrastructure management.

Conclusion

The numerous benefits that Ironforge provides cannot be understated. With Ironforge, projects can easily separate permission escalation for DevOps without being distracted from focusing on their core application’s logic.
There has been a lot of releases this year that makes Ironforge standout as a leading devOps platform for builders on Solana, but next year promises to be even more productive and efficient with the 2025 roadmap hinting to more improved log experience, more load balancer weighted algorithms and AI/LLM integration for more improved observability and business intelligence. If scaling your project’s infrastructure and security is crucial to you, then Ironforge is your best bet. The team is fast and responsive, they are always shipping, and they are strongly aligned with their core values to deliver an improved DevX. To get started, visit https://www.ironforge.cloud/, sign up, join the discord channel for 24/7 developer support and read the guides on https://www.ironforge.cloud/guides.

Additional resources:

Ironforge documentation
Ironforge repository
Ironforge Twitter

A Developer’s Guide into Account Abstraction Models

Aseneca — Sat, 30 Nov 2024 22:17:50 +0000

One of the core factors that has contributed to the influx of new users to crypto and usage of crypto wallets is accounts abstraction (AA). While continuous developments are being made around this concept, the idea of account abstraction was first proposed by Vitalik Buterin, founder of Ethereum, in 2015. Its fundamental proposition was to move crypto from the current approach of EOA accounts (which are accounts that are externally owned and controlled by private keys) to a model where accounts can be customised depending on their application and are controlled by smart contracts.

This article assumes the reader has no prior knowledge of AA models and aims to introduce the general concept AA by exploring its implementation in two prominent blockchains, namely Ethereum and Solana. By discussing their differences and the unique advantages each chain derives from the implementation of AA models at the core of their account structures, readers will better understand how these implementations can be adjusted to suit various needs.

What is an Account

It is a general misconception among most crypto users that the term “account” refers to a wallet. This assumption however cannot be further from the truth. As a developer, it is important to note that an account is not a wallet. A wallet is simply an application that acts as gateways to interact with blockchain. So what then is an account?

Accounts on Ethereum

In Ethereum, accounts are basically “objects” that constitute the state of the blockchain. These objects consist of a 20-byte address, and state transitions within the blockchain are as a result of transfer of value and information between these accounts. An Ethereum account contains four fields:

The nonce, a counter that tracks processed transactions
The account’s current ether balance (ETH)
The account’s contract code-hash which is the hash of the EVM code of the contract. The contract is immutable but its state is mutable (For EOAs, this is a hash of an empty string since EOAs do not store code)
Storage, an encoding of the state of the account In Ethereum, there are two types of accounts: externally owned accounts(EOA) which are controlled by private keys, and contract accounts, which are controlled by their contract code. EOAs do not store code and transmit information only by creating and signing transactions. Contract accounts on the other hand store code which activates whenever that account receives a message, therefore enabling it to read and write to internal storage and send messages or create contracts (contracts on Ethereum can be analogous to “agents” within the EVM that executes a piece of code as instructed by a transaction or message. They have direct control over their own ETH balance and their own key/value store to keep track of persistent variables).

(source: Helius.dev)

Accounts on Solana

On Solana, accounts are more like storage which is capable of holding every type of data ranging from tokens to a program’s state variables such as integers, publickeys and even entire programs. The Solana account model requires that every account has an owner, and multiple accounts can have a single owner.
A good analogy of the account model on Solana is a computer filesystem where a computer file can reference multiple data and state variables such as name, address, email e.t.c. Comparatively, in Solana, public keys can be likened to reference storage of state and data such as lamport, owner, rent, and executable.
Similar to Ethereum, Solana accounts are of two main types:

Executable, which consist of immutable programs that own and create other accounts that store state,
Non-executable, which are “storage or data” accounts that contain all other types of data like program variables, token balances, NFTs, etc.

Executable accounts can further be delineated into Program Accounts, which are accounts that store user-customised executable programs, and Native Program Account, which store core native programs such as the System Program which is responsible for the creation of accounts.
Through executable accounts, instructions can be communicated to the network on how to execute transactions and through non-executable accounts, data can be stored (without the ability to execute code.)

(source: Helius.dev)

The image above describes examples of executable and non-executable accounts. Bubblegum is an example of a Program Account by Metaplex used to create and manage compressed NFTs. Vote Program is a Native Program Account used to create and manage accounts that keep track of voting states. Associated Token Account, System Account and Stake Account are all non-executable accounts that store some form of data. Associated Token Account refers to accounts that hold data associated with a particular token, its balance and its owner (for example, 6ysptGHvXxPUs7T6jV5MGq157VUCUdreXQspt3w36EJF holds 10 USDC). Stake Account is used to stake/delegate token to validators and earn rewards.

Accounts on Solana follow a specified structure defined by the AccountInfo struct:


pub struct AccountInfo<'a> {
    pub key: &'a Pubkey,
    pub lamports: Rc>,
    pub data: Rc>,
    pub owner: &'a Pubkey,
    pub rent_epoch: Epoch,
    pub is_signer: bool,
    pub is_writable: bool,
    pub executable: bool,
}

The AccountInfo struct holds the following fields:

key, which is a unique 32-byte public key by which accounts are identified
lamports, which specifies the number of lamports owned by the account. Lamport is the unit of SOL which is the native token of Solana, and one lamport is one-billionth of one SOL
data is a reference to a raw data byte array that is stored by the account. It can be modified by programs and can store anything The field owner specifies the owner of the account represented by the address of a program account (PubKey). A few rules guide account ownership on Solana:
- Only an account owner can alter its state and withdraw lamport
- Anyone can deposit lamports into an account
- The owner of an account can reassign ownership to a new owner provided the account data is reset to zero
is_signer is a boolean that indicates if a transaction has been signed by the owner of the account- this means that the account holds the private key that corresponds to the public key and has the authority to approve transactions
is_writable field is a boolean which signifies whether an account state can be modified.
executable is a boolean that indicates whether an account is an executable or non-executable account.
rent_epoch refers to the next epoch at which an account will owe rent.

Account Abstraction Models

Account abstraction is quite a broad concept, but narrowing it down, Account abstraction models simply aim to describe the process of abstracting away the rigid and built-in structures of accounts within a blockchain, therefore allowing them to be more flexible and adaptable. Its models hide the core technology of the user accounts from the users, enabling a more rich and less technical experience.
Developers can utilise AA models to create customised and fluid user-friendly on-chain accounts where they can define their own custom logic for storing assets and executing transactions over the network.
Through AA models, developers can create smart contracts (called programs on Solana) that can take up roles as customised user accounts and interact with other accounts, as well as execute composable program interactions.
The advantages of AA models have far more crucial consequences on how users interact with programs and contracts on-chain by enabling implementation of custom execution logic while eliminating the need for a rigid set structure and giving developers the leeway to tailor user account structures to their specific needs. To gain further understanding of why AA models crucial for the next phase of development in blockchain, we shall examine their implementation within Ethereum and Solana.

Account Abstraction in Ethereum

Until recently, account structures have been very rigid in Ethereum, leaving no room for developers to create and implement custom user accounts, therefore leading to poor user experience in storing and managing assets on-chain. This is why most popular Ethereum wallets such as Metamask only give users the option of creating EOA accounts, which means limited customization and flexibility since these accounts are controlled by private keys and can only be used to store data (ETH, ERC-20) and interact with smart contracts only by signing transactions.
With the advent of ERC-4337 merged a novel account structure on EVM called Contract Accounts which was an improvement on EOA by introducing account abstraction- its implementation gave developers on Ethereum more possibilities through smart contracts that stores data, and code. Therefore eliminating the rigidity that came with accounts that are solely controlled by private keys, replacing it with functionality that is determined by the code within the smart contracts. The role that EIP- 4337 which eventually became the ERC-4337 standard played in the implementation of account abstraction on Ethereum will be discussed in another article.

Account Abstraction in Solana

Solana blockchain since its inception has always implemented account abstraction at its core, therefore rendering accounts on Solana more flexible and simplifying the engineering of complex interactions between accounts and programs (programs are smart contracts on Solana).
The account model on solana embraces separation of concerns at its core. Non-executable accounts (data accounts) storing data. They share some similarities with Ethereum EOA- both types of accounts are mainly used to hold and manage tokens, and interact with decentralised applications and smart contacts. Transactions can also be signed only by use of a private key on both types of accounts.
With Solana however, programs can actually create and manage specific accounts by extending AA implementation through a feature called Program Derived Addresses (PDAs) therefore extending developers ability to define custom logic for managing these accounts. With PDAs, there can be flexibility in rules that govern how transactions are signed by authorising programs to execute various operations on-chain on behalf of the PDA in a way that does not require the need for private keys. Therefore abstracting the burden placed on users to manage actions without the use of private keys such as storage and management of SPL assets, batched transactions and multi-signature (multi-sig) account management features. PDAs also extend the functionality of accounts by enabling them to interact and manage executable accounts within one another; a feature known as Cross-Program Invocation (CPI).

Conclusion

Despite the difference in the implementations of AA models within these two popular chains, we can see that the benefits of AA are tremendous and hold lots of possibilities for what developers can accomplish. On Ethereum, implementations of AA models still remains an ongoing area of research and development as a result of its more rigid account structure at its core, while it is more obvious in Solana as a result of its flexible account models enabling complex interaction between account layers. One of the core use cases for AA models is the implementation of smart wallets which is anticipated to onboard the next generation of crypto users due to their advanced features such as social account recovery, permissionless asset management and other benefits.
However, presently implementation of AA models also has some challenges such as restraint in limitation of program composability by not exposing an IDL interface, a practice typically referred to as security through obscurity.

References/Further resources
Solana docs
Ethereum Organization
Vitalik Buterin, On Abstraction, https://blog.ethereum.org/2015/07/05/on-abstraction
https://developers.metaplex.com/bubblegum

Improving IDLs Discoverability: Accelerating Solana Development, Integration and Composability

Aseneca — Mon, 11 Nov 2024 10:50:51 +0000

Introduction

Solana has garnered the reputation as the fastest and most scalable blockchain on the the market, leveraging its unique architecture to achieve high throughput and low latency. However, associated with a massive growth of this scale, is an important component of the ecosystem that requires constant attention: Developer Experience (DX). Improving the developer experience is always in the spotlight as developers and builders are the lifeline of Solana.
IDLs discovery and implementation for existing infrastructure appears to be a problem many developers in Solana presently struggle with. Two main concerns are predominantly echoed:

Reading data on the network is difficult and even more so for non-anchor programs
Obscurity in CPI with programs as security concerns

In this article, we will explore what Interface Definition Language (IDL) is all about, and how they are a vital part of improving the development experience on Solana. This article aims to provide a comprehensive understanding of the role IDLs play across the Solana development landscape, and why further understanding into upgrading its discoverability will bolster and accelerate development experience on Solana.

Interface Description Language(IDL)

An Interface Definition Language (IDL) in Solana refers to the public interface of a program. It represents the structure of programs accounts, instructions as well as error codes. They are usually in a JSON and Typescript file format and they are used to generate client-side code enabling clients to easily interact with a Solana program. They are automatically generated for programs written in anchor. IDLs can be compared to ABI files for EVM, which gives developers easy understanding of program structures.

Why are IDLs very useful

While the applications of IDLs can be enumerated to varying use cases, IDLs are very useful for the main reasons:

They make program interaction human-readable: Through IDLs, we can demystify the concatenation of transactions and accounts as just a bunch of bytes and provide developers a medium to easily parse instructions or accounts to make sense of what the program is doing. This attribute of IDLs enables indie and professional developers to get smooth integration on explorers, consumer UIs, and data platforms (like Dune).Below is an example of a parsed transaction instruction on Solana:

Here is an example of how Solana explorer display public IDLs:

Providing Standard Interfacing: By leveraging a standard interface, tools and frameworks can take advantage of this to generate instructions to interact with a program more easily, therefore enabling easier composability between program interfaces, making building client-side code far simpler (A good example would be the case of embedded wallet providers that whitelist specific ix discriminant). Below, we have an example of two Hello World frontends that create a greeting account, one using Native Solana and the other using Anchor:

Using Native Solana:

const GREETING_SEED = 'hello'
const greetedPubkey = await PublicKey.createWithSeed(
    payer.publicKey,
    GREETING_SEED,
    programId
)

// Check if the greeting account has already been created
const greetedAccount = await connection.getAccountInfo(greetedPubkey)
if (greetedAccount === null) {
    console.log('Creating account', greetedPubkey.toBase58(), 'to say hello to')
    const lamports = await connection.getMinimumBalanceForRentExemption(
        GREETING_SIZE
    )

    const transaction = new Transaction().add(
        SystemProgram.createAccountWithSeed({
            fromPubkey: payer.publicKey,
            basePubkey: payer.publicKey,
            seed: GREETING_SEED,
            newAccountPubkey: greetedPubkey,
            lamports,
            space: GREETING_SIZE,
            programId,
        })
    )
    await sendAndConfirmTransaction(connection, transaction, [payer])
}
// ...
export async function sayHello(): Promise<void> {
    console.log('Saying hello to', greetedPubkey.toBase58())
    const instruction = new TransactionInstruction({
        keys: [{ pubkey: greetedPubkey, isSigner: false, isWritable: true }],
        programId,
        data: Buffer.alloc(0), // All instructions are hellos
    })
    await sendAndConfirmTransaction(
        connection,
        new Transaction().add(instruction),
        [payer]
    )
}

Source: https://github.com/solana-labs/example-helloworld/blob/master/src/client/hello_world.ts

Using Anchor:

const anchor = require('@project-serum/anchor')

async function greet(message: string) {
    anchor.setProvider(anchor.Provider.local())
    // Here we load the HelloAnchor example IDL
    const program = anchor.workspace.HelloAnchor
    let greetKey = Keypair.generate()
    await program.methods
        .initialize(message)
        .accounts({
            newAccount: greetKey.publicKey,
            signer: wallet.publicKey,
            systemProgram: web3.SystemProgram.programId,
        })
        .signers([wallet, greetKey])
        .rpc()
}

greet('Hello World')

By utilizing IDLs, Anchor can abstract the process so that it is much easier to read, quicker to deploy, and more composable for CPI interactions. To know more about CPIs and why they are so crucial to Solana development, refer to this article Cross-Program Invocations and PDAs- the combination of two powerful mechanism on Anchor

Dependency-free SDKs: IDLs provide dependency-free SDKs for long-tail of Solana programs with declare_program!()

Creating IDLs

With Anchor, IDLs are automatically generated when you build because they are so integral to the entire anchor programming landscape. From the anchor project directory, enter the following in your terminal:

anchor build --idl <IDL_OUTPUT_DIRECTORY>

However by default, idl.json will be saved in ./target/idl folder

For Solana programs written in native Rust, IDLs can be generated using tooling like Metaplex Shank or Codama Kinobi

Publishing IDLs

Anchor also makes it easy to publish IDLs on-chain. To do this, you simply need to enter the following in your terminal:

anchor idl init --filepath <FILEPATH> <PROGRAM_ID> --provider.cluster <CLUSTER> --provider.wallet <WALLET>

Make sure the wallet being used is the program authority and has SOL for the on-chain transaction! After the IDL is loaded, you should be able to see the IDL on Solana Explorer at https://explorer.solana.com/address/YOUR_PROGRAM_ID/anchor-program.

The Anchor idl command has several other handy tools that enables developers to fetch, update, or freeze an IDL. For information on each of those commands, you can enter the following in your terminal:

anchor idl -h

Interfacing IDLs on the Client Side

Programs defined in your IDL are mapped to correspond to functions provided by the generated Typescript module, which is integrated into the front end of our application. The general structure of this module is outlined below:

import { Program, AnchorProvider, setProvider } from '@project-serum/anchor'
import { Connection, KeyPair } from '@solana/web3.js'
import { PROGRAM_ID, IDL } from './your-program-dir'
// where IDL is the .json created by anchor build
// and PROGRAM_ID is your on-chain program ID

export const yourFunction = async () => {
    const wallet = KeyPair.generate()
    const connection = new Connection(QN_ENDPOINT)
    const provider = new AnchorProvider(connection, wallet, {})
    setProvider(provider)
    const program = new Program(IDL, PROGRAM_ID)
    // ... your code
    // e.g. await program.methods.yourMethod(YOUR_PARAMETERS).accounts({YOUR_ACCOUNTS}).rpc();
}

Due to the program instructions and account embedded in the IDL, the code editor can automatically detect callable programs via program.methods... A type guard is put in place to ensure all the necessary accounts are passed into the instruction. See Intro to client-side Anchor development

Improving IDL Discoverability

The majority of programs use Anchor today, and for non-Anchor frameworks, codama coupled with verified builds can fill the gaps. These IDLs provide efficient reading and a starting point for calling programs with declare_program!() but they are currently not discoverable. It is not yet ascertained whether IDLs can cover all possibilities on programs however the majority can be covered.

Current solutions being proposed to improve discoverability have to satisfy the following requirements:

Must allow a generalized reading from the network.
The time taken to understand how to implement CPI into other programs should be significantly reduced.
Program discovery should be as frictionless as possible.

Conclusion

We have explored IDLs and their numerous benefits. We have also examined IDLs in Anchor and non-Anchor frameworks and how developers can utilize them to improve program composability.

However, existing infrastructure does not have versioned IDLs based on deployment time, which makes it difficult for developers concerned with consistent data integration to understand what IDL to use at what slot.

Improving IDL discoverability has huge benefits; developers can easily integrate CPI into programs, which expands composability within the general developer landscape, unlocking new use cases that may not have been thought of.

Discoverable IDLs also provide more insight into security deficiencies as even seasoned developers on Solana tend to secure their program through obscurity which can often prove to be a bug rather than a feature.

Over the coming years, IDLs are beginning to take a place within the spotlight as targets for more investment and development to enable building on Solana a more framework and language agnostic process, therefore accelerating developer experience across all development landscapes on Solana.

Additional Resources/Further Reading

Handling Text in Rust: String and &Str

Aseneca — Tue, 15 Oct 2024 18:37:30 +0000

Unlike other programming languages, handling text in rust can be a bit complicated. This is due to some type-handling mechanism that rust forces us to adhere to in order to mitigate any problems that might result from the "text-structure and usage" at compile time. Rust uses two types to identify "text-based" characters. These are String and &Str. They are very similar in many ways but differ in terms of ownership, mutability and how they are represented in memory.

String Type

The String type is a heap-allocated owned string. This means that its size is dynamic and can adjust to the needs at runtime.
Also, the String type is mutable which implies that you can either append characters or change characters in the String type. This mutability is possible because the String type owns the memory that it uses.
for example:

let mut s= String::from("Hello");
s.push(", World"); //we modify the string s
println!("{}", s);

When we say the String type size is dynamic, we mean that it is allowed by rust to grow and shrink as needed as demonstrated in the simple example above.

&Str Type

The &Str type on the other hand refers to a borrowed string slice. It is used to represent a view into a part of a string but it does not own the memory to which it point(because it is borrowed). Unlike the String type, the &Str type is an immutable reference usually alluded to a String type or a string literal. Therefore its data cannot be modified.

let s= "This is a rust example" //this is a &str
printlin!("{}", s)

let s= String::from("Rust is awesome"); //this is a String
let sliced_string: &str= &s[0..5]; //&str(string slice)
printlin!("{}", sliced_string)

String Literals

String literals in rust are also &str types. This means they are also stored the program's binary at compile time and are immutable.

for example,
let greeting= "Hello"; //this is a &str

Conversion between String and &Str

You can convert a &str to a String using the String::from function or the .to_string() Method

let heading: &str="A letter to rust";
let heading_string:String= heading.to_string()

On the other hand conversion from Stringto &str is not as straight-forward as you will have to borrow the String content.

let heading= String::from("A letter to rust"); let slice_heading:&str= &heading; //borrow heading as a &str

Now that you know the difference between &str and String, when should you use them?

Use Stringwhen you need:

Ownership over the string
To modify the string at some point
To store a dynamic string i.e a string which you do not know the size at compile time

Use &Str when you:

Do not need to own the string
Require a reference to the string without making a copy or clone
Do not need to modify the string at any point

I hope this article has helped you understand more about &str and String and how they provide great flexibility when it comes to handling string data in different contexts.

Clone and Iterative Adapter Methods in Rust: The thin line between Duplication, Efficiency and Transformation

Aseneca — Sat, 21 Sep 2024 12:43:39 +0000

As a rust developer, it is a dilemma you would always have to deal with: the choice between clone and iterative methods. In this guide, we will be exploring what these methods are, the fundamental differences between them, how they can be useful in manipulating data and which is appropriate for particular situations.

The Clone method

The clone method (clone()) is used to implement or "create" deep copies of an object or data. It operates by duplicating the entire object structure and reallocating new memory for the copied instance of that object.

Note: When we refer to the term reallocating, this is most likely a reference to heap-allocated data i.e. dynamic allocated memory for complex data collections(e.g., String, Vec, Box). When using clone() with stack-only types (e.g., char, i32), we do not reallocate memory as it simply copies the data directly on the stack.

In simpler terms, the clone method generates two independent instances of the same data

fn main() {
    let args: Vec<String> = env::args().collect();

    let config = parse_config(&args);

    println!("Searching for {}", config.query);
    println!("In file {}", config.file_path);

    let contents = fs::read_to_string(config.file_path)
        .expect("Should have been able to read the file");

    // --snip--
}

struct Config {
    query: String,
    file_path: String,
}

fn parse_config(args: &[String]) -> Config {
    let query = args[1].clone();
    let file_path = args[2].clone();

    Config { query, file_path }
}

In the code snippet above, we make a full copy of the data for the Config instance to own by cloning the values received through args, which takes more time and memory than storing a reference to the string data.

Iterator Adapter Method
In comparison to the clone(), iterator adapter methods are used to transform and process elements of an iterator. They are usually referred to as lazy in rust, and this is because iterator methods will not effect any process until the methods that consume the iterator is called. These consumption methods also known as consuming adapters, i.e methods that use up the (e.g., sum(), next()….) will be discussed when we talk about iterator methods in details.
Iterator Adapters(different from consuming adapters) do not immediately produce new data, hence the term "lazy'. Instead they return a new iterator that processes these data as they are needed on demand.

Differences between Clone Methods and Iterator Adapter Methods

Memory Allocation:

Clone Method:

Clone methods will create a new copy of the data. This can lead to additional memory allocation and therefore can be computationally expensive, especially for large or complex data structures, as it requires duplicating the entire object.

let v = vec![1, 2, 3];
let v2 = v.clone(); // Creates a deep copy of the entire vector
println!("{:?}, {:?}", v, v2);

Iterator Adapter Method:

Iterators generally operate in a lazy fashion, so they will not allocate memory immediately. They transform or process data as you iterate over it and this can make them more memory-efficient than clone methods. Also, allocation with iterators occurs only when you consume the iterator (e.g., using .collect() to gather results into a new collection).

let v = vec![1, 2, 3];
let iter = v.iter().map(|x| x * 2); // Creates a lazy iterator, no memory allocation yet
let v2: Vec<_> = iter.collect();    // Consumes the iterator and collects the results into a new vector
println!("{:?}, {:?}", v, v2);

Data Ownership and Borrowing:

Clone Method:

When an object is cloned, a new owned instance of that object data is initiated. Therefore, both the original and the cloned object can co-exist independently with no ownership conflict between them.

Iterator Adapter Method:

Iterators can either borrow data from the original collection or take ownership of the elements. Many iterator methods do not require ownership (for example, the iter() returns an iterator that borrows the data).

This borrowing mechanism is very useful when you need multiple copies of data that can be modified separately therefore making them more flexible when you want to process elements without transferring ownership or duplicating data.

So now you know the difference between them, which should you use?

Well it is going to depend on your specific use case but as a general rule of thumb, You use clone methods when you:

wish to create a deep copy of an object, instantly allocating new memory for the instance.
want an independent, identical copy of an object that can be modified or passed elsewhere without affecting the original object.

You use Iterator Adapter Methods when you:

want to transform, filter, or aggregate data from a collection or stream in a lazy, efficient way.
Do not need immediate memory allocation

But here is something you should keep in mind, aside from the specific use cases considered above, clone methods are generally inefficient because of its runtime cost. The idea of deeply copy data can impact performance especially if it is a large and complex data.
Iterative methods on the other hand are more efficient since you do not need to create any intermediate collections unless explicitly required by the program.
Iterative methods are also more performant efficient than loops. By combining them with closures, which we shall talk about when we discuss Iterative adapters in detail, we can harness Rust’s capability to clearly express high-level ideas at low-level performance.

Introduction to Solana: A fledging's guide on Web3 development on Solana

Aseneca — Tue, 17 Sep 2024 09:48:21 +0000

In this guide, we will be briefly explaining important web3 concepts such as blockchain, protocols etc. Then we shall dive deeper into key development concepts in Solana blockchain which should serve as an entry point for new Solana developers or blockchain enthusiasts that wish to learn further about Solana.

First of all, let's talk about fundamental terminologies that exist across web3 in general:

Blockchain

A blockchain can be thought of as a series of blocks or an append-only data structure that resembles an ordered back-linked linked list, which uses hashes as pointers to previous blocks. This structure consists of blocks that form a chain, hence the term blockchain. This is a very simple data structure model but it can get more complex than this with different approaches to how these blocks interact with one another

Block

A typical Block is a data structure that contains a header comprising three items– the hash of the previous block’s header, metadata and a Merkle root. Metadata depends on the protocol. The Merkle root is a root of the well-known Merkle tree, which can be used to verify later that transactions in a block have not been tampered with. After the head comes the core part of the block, the transaction.

Transaction

Transaction is a protocol-defined message that is stored as a part of a block, which is then stored as a part of a blockchain. The content usually consists of some kind of value transfer or on-chain program execution. Transactions are cryptographically signed by their authors or authority, proving their authenticity. In the case of a value transfer, ownership of the funds or tokens often represents some value in the real world.

Protocol

Protocol in simple and familiar terms refer to a common set of rules network nodes must follow. It defines the ground rules for communication between P2P (Peer-to-Peer) nodes, transaction format for everyone intending to use the network, any special features, and everything else for the network to operate correctly and for the users to know how to transact over the network. An essential part of a good protocol for a decentralized blockchain network is the proper incentive setup; this creates the need for its native token. Examples include SOL(token) for Solana (chain), ETH(token) for Ethereum(chain), BTC for Bitcoin

Smart contracts

A smart contract is a piece of code deployed on a blockchain with a cryptographically signed transaction. Users can then interact with it by sending transactions that invoke a specific function defined in the smart contract and the business logic is executed as stated in the deployed code.
Smart contracts on Solana are called Programs.
Data pertinent to the smart contract "state" are also stored on on-chain. A good analogy would be to compare smart contracts as programs on a decentralized computer that access files in its file system and modify them according to the predefined rules. Such a contract can be made immutable or mutable. If such a contract is made immutable, we can trust that the smart contract will not do anything else than what it is supposed to do.

Okay great. Now we know a little about what blockchain is and its rudimentary building blocks. Lets dive into the blockchain of interest in this guide: Solana.

What is Solana?

SOLANA

Solana is a single-chain blockchain that utilizes a slightly changed PBFT consensus called Tower BFT with Proof-of-Stake as a Sybil protection mechanism. It was engineered with composability at its core, meaning its performance does not depend on hardware. It is able to adapt to any hardware and scale to the utilize the full power of that "hardware."
Solana is a smart contract platform but smart contracts on Solana are called Programs. Programs can be executed in parallel. This parallelization is one of the key USP that gives Solana its composability nature.
Compared to Ethereum which is written in Solidity, development of Programs on Solana revolve around Rust programming language and its frameworks especially Anchor.
We will be breaking down the core concepts of Solana that come together to redefine its engineering perspective and improve its scalability in a subsequent article because those are more advanced concepts beyond the scope of this guide.

The Essentials of Solana Smart Contracts (Programs)

One thing you should have on the back of your mind as you explore building on Solana is: everything in Solana is an account. Accounts are owned by programs.
There are core concepts we need to discuss to help you understand how programs operate:

Transactions
Instructions
Accounts
Program Derived Addresses(PDAs)

1. Transactions
A transaction is a signed instruction sequence that is executed atomically. They are used to interact with programs deployed on the Solana network. A transaction is made up of one or more instructions, a recent blockhash, and signatures.

The recent blockhash, also known as a transaction "recentBlockhash", is used for transaction processing and lifespan. The signatures are the cryptographic proof of the transaction's integrity and authority. Each transaction must be signed by the appropriate account holders as per the instructions contained within. This takes us to the next concept which are instructions.

2. Instructions
Instructions in Solana are "bundled" within a transaction and are executed sequentially. Each instruction contains aprogram ID, accounts that it wishes to interact with, and a data field. The program ID specifies the on-chain program that will process the instruction. The accounts array includes all accounts that the instruction will read from or write to. The accounts array must include at least one signer (authority) who authorizes the instruction.

The data field is an arbitrary byte array that is passed to the program for processing.
It is this combination of transactions and instructions that allows for high throughput and efficient processing in Solana. By virtue of its architecture design, Solana is capable of processing thousands of transactions per second, each containing multiple instructions, which can interact with different programs and accounts. This design can also be attributed to Solana's core scalability and speed.

3. Accounts

In Solana, an account is a persistent memory structure that a program can use for storing state. Every account in Solana is initially owned by the system program, but the system program can change the ownership if the correct private key for the account is provided. Each account includes several properties:

a. Public Key (Pubkey): This is the unique identifier of the
account.

b. Signer Flag (is_signer): This flag indicates whether the account
is a signer of the transaction. If true, the transaction must
include the signature of this account.

c. Writable Flag (is_writable): This flag indicates whether the
data in the account can be modified. If true, the account's data
can be written to in the current transaction.

d. Lamports: This is the number of lamport's (the smallest unit of
the native SOL token) held in the account. Lamports also serve as
rent to keep the account on the network.

e. Data: This is a byte array that can hold arbitrary data.

f. Owner: This is another public key that identifies the program
that has authority over the account's data. Only the owner program
can modify the account's data.

g. Executable Flag: Hold if the account is a smart contract.

h. Rent Epoch: This value represents the latest epoch that rent has
been paid for the account. Rent is paid in SOL and ensures that the
account remains active on the network.

In Solana, accounts not only hold the state of a program but can also be used to create complex relationships between different programs through CPIs (See more on CPIs here)

Getting started on Solana Development

To get started with Solana development, you would need a (MacOS) or a Linux development environment. You can also use WSL on windows.

Step 1: Install Rust, Rustup and Cargo

Open up a terminal window (for MacOS) or command prompt (for Windows) and paste this command.

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Now we have installed Rust language and Cargo (the Rust package manager)which is required for compiling Rust-based Solana programs.

Step 2: Install Solana Tool Suite

Installing for MacOS & Linux

This option is easier but it requires you to have Homebrew package manager on your MacOS or Linux machine.

brew install solana

You can confirm you have the desired version of Solana installed by running:

solana --version

Installing for MacOS & Linux (without homebrew)

Install the Solana release v1.18.7 on your machine by running:

sh -c "$(curl -sSfL https://release.solana.com/v1.18.17/install)"
You can confirm you have the desired version of solana installed by running:

solana --version

For Windows

Open a Command Prompt cmd.exe as an Administrator and paste this command:
cmd /c "curl https://release.solana.com/v1.18.17/solana-install-init-x86_64-pc-windows-msvc.exe --output C:\solana-install-tmp\solana-install-init.exe --create-dirs"
Copy and paste the following command, then press Enter to install the latest version of Solana. If you see a security pop-up by your system, please select to allow the program to run.

C:\solana-install-tmp\solana-install-init.exe v1.18.17

You can confirm you have the desired version of solana installed by running:

solana --version

In addition, ensure you have Node.js and Yarn installed on your computer
Now you have Solana Cli, Node.Js and Yarn installed, lets go ahead and create a Keypair

Creating Keypair

To get started, we're going to create a keygen script and an airdrop script for our account.

Setting up

Opening up your Terminal and use yarn to create a new Typescript project.

npm install touch-cli -g
mkdir airdrop && cd airdrop
yarn init -y

If you get an error in windows after yarn init -y, run Microsoft powershell as administrator and run the following code:

Set-ExecutionPolicy RemoteSigned
This will initialize a new project, we're going to go ahead and add typescript, bs58 and @solana/web3.js, along with generating a tsconfig.js configuration file:

yarn add @types/node typescript @solana/web3.js bs58
yarn add -D ts-node
touch keygen.ts
touch airdrop.ts
touch transfer.ts
yarn tsc --init --rootDir ./ --outDir ./dist --esModuleInterop --lib ES2019 --module commonjs --resolveJsonModule true --noImplicitAny true

Finally, we're going to create some scripts in our package.json file to let us run the three scripts we're going to build today:

{
  "name": "airdrop",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "scripts": {
    "keygen": "ts-node ./keygen.ts",
    "airdrop": "ts-node ./airdrop.ts",
    "transfer": "ts-node ./transfer.ts",
  },
  "dependencies": {
    "@solana/web3.js": "^1.75.0",
    "@types/node": "^18.15.11",
    "typescript": "^5.0.4"
  },
  "devDependencies": {
    "ts-node": "^10.9.1"
  }
}

❗ Ensure to add the scripts part to the .json file. ❗

Generating a Keypair

Open up ./keygen.ts.

Importing Keypair from @solana/web3.js
import { Keypair } from "@solana/web3.js";

Generate a new keypair:

//Generate a new keypair
let kp = Keypair.generate()
console.log(`You've generated a new Solana wallet: ${kp.publicKey.toBase58()}`);
console.log(`Solana Wallet Secret Key: ${kp.secretKey}]`);

run the following script to generate a new keypair!

yarn keygen

This will generate a new Keypair, outputting its Address and Private Key like so:

You've generated a new Solana wallet: 2sNvwMf15WPp94kywgvfn3KBNPNZhr5mWrDHmgjkjMhN
Solana Wallet Secret Key: 63,224,142,27,16,71,3,47,65,3,83,69,169,184,116,97,218,0,1,213,225,174,108,112,40,67,255,32,77,82,140,62,187,6,73,117,82,29,152,252,161,62,35,24,148,199,196,160,92,62,81,179,17,49,7,202,226,213,201,30,73,176,198,11]

Best practice would be to save this wallet locally. runthe following command:

touch dev-wallet.json

This creates the file dev-wallet.json in our ./airdrop root directory.

Now paste the private key from above into this file in this format:

[63,224,142,27,16,71,3,47,65,3,83,69,169,184,116,97,218,0,1,213,225,174,108,112,40,67,255,32,77,82,140,62,187,6,73,117,82,29,152,252,161,62,35,24,148,199,196,160,92,62,81,179,17,49,7,202,226,213,201,30,73,176,198,11]

Congratulations. You've just created a new Keypair and saved first your wallet. Now, Let's go claim some tokens!

Claiming Airdrop Tokens

import Keypair, also going to import Connection to establish a connection to the Solana devnet, and LAMPORTS_PER_SOL to transfer amounts denominated in SOL rather than individual lamports units.

import { Connection, Keypair, LAMPORTS_PER_SOL } from "@solana/web3.js"

get the private key in the .json file by importing the wallet and recreate the Keypair object using its private key:

import wallet from "./dev-wallet.json"

import the keypair from the wallet file

const keypair = Keypair.fromSecretKey(new Uint8Array(wallet));

establish a connection to the Solana devnet:

//Create a Solana devnet connection to devnet SOL tokens
const connection = new Connection("https://api.devnet.solana.com");

Finally, claim 2 devnet SOL tokens:

(async () => {
    try {

        const txhash = await connection.requestAirdrop(keypair.publicKey, 2 * LAMPORTS_PER_SOL);
        console.log(`Success! Check out your TX here: 
        https://explorer.solana.com/tx/${txhash}?cluster=devnet`);
    } catch(e) {
        console.error(`Oops, something went wrong: ${e}`)
    }
})();

Save the code and run it by typing yarn airdrop in the terminal. You should get a successful message with a Tx address.

Congratulations once again for coming this far. You are officially a Solana newbie dev now. But why stop here: As a challenge, try implementing the transfer token challenge and share your result in the comment section.

Cross-Program Invocations and PDAs- the combination of two powerful mechanism on Anchor

Aseneca — Wed, 11 Sep 2024 18:58:06 +0000

As an intermediate anchor developer building on Solana, it is very likely you would eventually come across a situation where it is more beneficial for your programs to interact with one another. Such composability in Solana is achieved through a mechanism called Cross-Program Invocations (CPIs).

Let us first break down what CPIs are, how you should use them and what you should keep in mind while using this functionality.

Cross-Program Invocation(CPIs)

The unique integrated architecture of Solana is what gives it the ability to process thousands of transactions per second while leveraging a globally decentralized network. This design gives room for a sort of modular structure, while acting as a single-chain blockchain that enables applications built on top of it inherit composability. This composable topology is what enable applications(and programs) to interact and build on top one another eliminating the need for complex on-chain activities such ass bridging and liquidity fragmentation.

CPIs are one of the internal mechanisms that enables a Solana program (or smart contract) to call another program on the blockchain during its execution. It is a powerful mechanism that gives different programs the ability to interact and perform actions like token transfers, data manipulation, or other operations defined by the source program.

when should you use CPIs?

You should use CPIs when you need a program to leverage another program to execute a functionality. for example, like performing a token transfer, using a CPI to leverage the Token Program will save us a lot of computational resource instead of re-implementing the token transfer functionality or in transferring an NFT from one account to another.

How would you use a CPI?

Basically using a CPI involve the following steps:

firstly identifying the programs you wish to interact with. We shall be using the Token program as an example.

Then define the necessary accounts that would be required by the CPI using Anchor's #[account] macros, and specify the constraints that define how they should be passed into the CPI as shown in the example below:

#[derive(Accounts)]
pub struct List<'info> {
    #[account(mut)]
    maker: Signer<'info>,
    #[account(
        seeds = [b"marketplace", marketplace.name.as_str().as_bytes()],
        bump = marketplace.bump,
    )]
    marketplace: Box<Account<'info, Marketplace>>,
    maker_mint: Box<InterfaceAccount<'info, Mint>>,
    collection_mint: Box<InterfaceAccount<'info, Mint>>,
    #[account(
        mut,
        associated_token::authority = maker,
        associated_token::mint = maker_mint,
    )]
    maker_ata: Box<InterfaceAccount<'info, TokenAccount>>, //stores the maker on the heap
    #[account(
        init_if_needed,
        payer = maker,
        associated_token::mint = maker_mint,
        associated_token::authority = listing,
    )]
    vault: Box<InterfaceAccount<'info, TokenAccount>>,
    #[account(
        init,
        payer = maker,
        space = 8 + Listing::INIT_SPACE,
        seeds = [marketplace.key().as_ref(), maker_mint.key().as_ref()],
        bump
    )]
    listing: Box<Account<'info, Listing>>,
    #[account(
        seeds = [
            b"metadata",
            metadata_program.key().as_ref(),
            maker_mint.key().as_ref()
        ],
        seeds::program = metadata_program.key(),
        bump,
        constraint = metadata.collection.as_ref().unwrap().key.as_ref() == collection_mint.key().as_ref(),
        constraint = metadata.collection.as_ref().unwrap().verified == true,
    )]
    metadata: Box<Account<'info, MetadataAccount>>,
    #[account(
        seeds = [
            b"metadata",
            metadata_program.key().as_ref(),
            maker_mint.key().as_ref(),
            b"edition"
        ],
        seeds::program = metadata_program.key(),
        bump,
    )]
    master_edition: Box<Account<'info, MasterEditionAccount>>,
    metadata_program: Program<'info, Metadata>,
    associated_token_program: Program<'info, AssociatedToken>,
    system_program: Program<'info, System>,
    token_program: Interface<'info, TokenInterface>,
}

The next step would involve defining the instruction data, for example, defining the mechanism of the transfer from the maker_ata account to the vault account, establishing the maker account as the authority and the maker_mint account as the mint. The required accounts needed to deposit an NFT to a vaults were themaker_ata account, vault account, maker and maker_mint account
Then we create the CPI context and invoke the token program transfer instruction

pub fn deposit_nft(&mut self) -> Result<()> {
        let accounts = TransferChecked {
            from: self.maker_ata.to_account_info(),
            to: self.vault.to_account_info(),
            authority: self.maker.to_account_info(),
            mint: self.maker_mint.to_account_info(),
        };

        //create a Cpi context
        let cpi_ctx = CpiContext::new(self.token_program.to_account_info(), accounts);

         // Invoke the token program’s transfer instruction
        transfer_checked(cpi_ctx, 1, self.maker_mint.decimals)
    }
}

The above example is also a form of an important benefit of CPIs known as Privilege Extensions which as the name implies "extends" the privileges of the caller to the callee. In our above example, the original authority maker account "extends" the control of the authority to the Token Program temporarily, enabling it to execute the transfer on its behalf for the duration of the invocation.

Program as Signers

It is very likely that in some cases, and this is most often the case, you would require that the program itself take authority over the assets. In the NFT marketplace example above, it is likely you would want a program to delist and even purchase an NFT on your behalf or a lending protocol program would need to manage deposited collateral and automated market maker programs need to manage the tokens put into their liquidity pools.
We can accomplish this using Program Derived Addresses (PDAs). We will talk about PDAs in subsequent articles, but simply defined: PDAs are special addresses that do not have public keys and therefore do not have an associated private key. They have two advantages to build hashmap-like structures on-chain and the allow programs to sign instructions without private keys.

PDAs combined with CPIs are very powerful features in anchor development as they allow us to securely manage assets and data by ensuring that only the owning program through the CPI can sign transactions that affect a particular account thus preventing unauthorized signings during inter-program communications.

To sign a transaction with a PDA in through the CPI Context, instead of the calling the CpiContext::new(cpi_program, accounts) method, we would have to use the CpiContext::new_with_signer(cpi_program, accounts, seeds) where the seeds argument are the seeds and the bump the PDA was created with as shown in the close_mint_vaultimplementation below:

 pub fn close_mint_vault(&mut self) -> Result<()> {
        let seeds = &[
            &self.marketplace.key().to_bytes()[..],
            &self.maker_mint.key().to_bytes()[..],
            &[self.listing.bump],
        ];
        let signer_seeds = &[&seeds[..]];

        let accounts = CloseAccount {
            account: self.vault.to_account_info(),
            destination: self.maker.to_account_info(),
            authority: self.listing.to_account_info(),
        };
        //Sign with the pda
        let cpi_ctx = CpiContext::new_with_signer(
            self.token_program.to_account_info(),
            accounts,
            signer_seeds
        );

        close_account(cpi_ctx)
    }

what happens here is that during execution, the Solana runtime will check whether hash(seeds, current_program_id) == account address is true. If yes, that account'sis_signer flag will be turned to true. This means a PDA derived from some program, may only be used to sign CPIs that originate from that particular program.

In summary, combining CPIs and PDAs enables us to leverage the powerful abstractive feature of Anchor while enabling us to reduce the number of accounts we need through PDAs and securing a composable interaction between different programs on our application.

Advanced Developer Tutorial on Address Lookup Tables

Aseneca — Thu, 16 Nov 2023 09:27:47 +0000

Since the early times when data begun to be used to compute values, there has always been a need to store complex data in such a way that they would always be easy to access. In ancient times this was accomplished with sine tables and a 98-column multiplication, advances in computing technology has impacted the way data is stored and retrieved.
In the early years of computer technology, processing input/output was particularly slow and would consume more resource and time. To solve this problem, it made sense to reduce significantly the process of reading operations and operations data. A very efficient method that was introduced to solve this problem was Caching (which involves storing data so that it can be available for future request at faster speeds due to an earlier computation of the data stored elsewhere).
However, while systemwide caching tries to automate the process of fetching commonly occurring data items, to enable a much faster mechanism for obtaining data items that possess a sort of immutability, lookup tables were introduced.

What is a Lookup Table

As if defined in computer science, [1]

a lookup table(LUT) is an array that replaces runtime computation with a simpler array indexing operation. This process of utilizing array indexing to compute the running phase of a program is also known as direct addressing. It is much faster and less expensive since data is retrieved from memory using indexes as reference compared to the processing input/output approach. The tables may be pre-calculated and stored in static program storage, calculated (or "pre-fetched") as part of a program's initialization phase, or even stored in hardware in application-specific platforms.

A lookup table is more efficient in terms of lookup operation and has a guaranteed O(1) time complexity. However, it is not possible for two entities to have the same lookup key k, and as the size of the datasets grow increasingly larger, in the magnitude of considering all forms of allottable data, it becomes less practical to stored a LUT in memory.

Address Lookup Table (ALTs)

As defined on the Solana documentation, [2]

An address Lookup Tables, commonly referred to as "lookup tables" or "ALTs" for short, allow developers to create a collection of related addresses to efficiently load more addresses in a single transaction. Since each transaction on the Solana blockchain requires a listing of every address that is interacted with as part of the transaction; without ALTs, this listing would effectively be capped at 32 addresses per transaction, but with ALTs, it would be effectively raise the addresses per transaction to 256

ALTs are part of additional functionalities that were introduced in the versioned transaction format which is quite similar to the legacy transaction format with just slight differences in libraries used.

How to Create An Address Lookup Table(ALTs)

This tutorial is going to show you how you can implement ALTs on your projects and help you see the benefits and advantages they offer.

What We Will Implement in This Tutorial

In this Tutorial, we will:

Create and execute a Version 0 (V0) Transaction

Create and Populate an Address Lookup Table

Compare the transaction size of two nearly identical transactions where one would implement a lookup table and the other would not

Perquisites:

Node.js

Intermediate Experience with Typescript/JavaScript

ts-node

basic or solid experience in running Solana transactions

all necessary imports including classes from Solana Web3 library

create a wallet and fund it with some devnet SOL

set up API endpoint to connect with the Solana network

Your setup Environment should look similar to this:

Steps to Creating ALTs

Executing Versioned Transaction V(0)

To use Lookup Tables, you must execute a versioned transactions.

We can do this by Let's start by creating a new function, createAndSendV0Tx, that accepts an array of TransactionInstructions, txInstructions:

async function createAndSendV0Tx(txInstructions: TransactionInstruction[]) {
// Step 1 - Fetch Latest Blockhash   
let latestBlockhash =await SOLANA_CONNECTION.getLatestBlockhash('finalized');    console.log("   ✅ - Fetched latest blockhash. Last valid height:", latestBlockhash.lastValidBlockHeight)
;    

// Step 2 - Generate Transaction Message    
const messageV0 = new TransactionMessage({ payerKey: SIGNER_WALLET.publicKey,  recentBlockhash: latestBlockhash.blockhash, instructions: txInstructions    }).compileToV0Message();    console.log("   ✅ - Compiled transaction message");    
const transaction = new VersionedTransaction(messageV0);    
// Step 3 - Sign your transaction with the required `Signers`    transaction.sign([SIGNER_WALLET]);    console.log("   ✅ - Transaction Signed");    // Step 4 - Send our v0 transaction to the cluster    const txid = await SOLANA_CONNECTION.sendTransaction(transaction, { maxRetries: 5 });    console.log("   ✅ - Transaction sent to network");    
// Step 5 - Confirm Transaction     
const confirmation = await SOLANA_CONNECTION.confirmTransaction({signature: txid,        blockhash: latestBlockhash.blockhash, lastValidBlockHeight: latestBlockhash.lastValidBlockHeight    });  if (confirmation.value.err) { throw new Error("   ❌ - Transaction not confirmed.") }    console.log('🎉 Transaction succesfully confirmed!', '\n', `https://explorer.solana.com/tx/${txid}?cluster=devnet`);}

Let's walk through our code:

Step 1: We fetch the latest blockhash from the network. Note: We pass the parameter, 'finalized', to make sure the block does not belong to a dropped fork.
Step 2: Using our txInstructions parameter and the latestBlockhash, we create a new MessageV0 by building a Message and executing the .compileToV0Message() method.
Step 3: We sign the transaction with an array of signers. In this case, it is just our SIGNER_WALLET.
Step 4: We send the transaction to the cluster using sendTransaction, which will return a transaction signature/id.
Step 5: We wait for the cluster to confirm the transaction has succeeded. If it succeeds, we log our explorer URL; otherwise, we throw an error.

Once successful with the above, we now move on to create an ALT

Create an Address Lookup Table

Create a new async function, createLookupTable, that will build our transaction instruction and invoke createAndSendV0Tx:

async function createLookupTable() {
    // Step 1 - Get a lookup table address and create lookup table instruction
    const [lookupTableInst, lookupTableAddress] =
        AddressLookupTableProgram.createLookupTable({
            authority: SIGNER_WALLET.publicKey,
            payer: SIGNER_WALLET.publicKey,
            recentSlot: await SOLANA_CONNECTION.getSlot(),
        });

    // Step 2 - Log Lookup Table Address
    console.log("Lookup Table Address:", lookupTableAddress.toBase58());

    // Step 3 - Generate a transaction and send it to the network
    createAndSendV0Tx([lookupTableInst]);
}

Breaking down our code:

Step 1: We create two variables, lookupTableInst and lookupTableAddress, by destructuring the results of the createLookupTable method. This method returns the public key for the table once created and a TransactionInstruction that can be passed into our createAndSendV0Tx function.
Step 2: We log the table's address
Step 3: Finally, we call createAndSendV0Txby passing lookupTableInst inside of an array to match our type requirements.

Create an empty address table by running:

createLookupTable();

To run the transaction, enter your terminal and execute:

ts-node app.ts

After the transaction has completed, you should receive a URL to the transaction page on Solana Explorer, like in the image below:

The lookup table account address is: 3uBhgRWPTPLfvfqxi4M9eVZC8nS1kDG9XPkdHKgG69nw

Next steps:

Remove the call to createLookupTable()
add the table lookup address from your console to the PublicKey declaration like so :

const LOOKUP_TABLE_ADDRESS = new PublicKey("YOUR_TABLE_ADDRESS_HERE"); 
// e.g const LOOKUP_TABLE_ADDRESS = new PublicKey("3uBhgRWPTPLfvfqxi4M9eVZC8nS1kDG9XPkdHKgG69nw");

So what's going on with the extendLookupTable method:

We pass our SIGNER_WALLET to pay the transaction fees and any additional rent incurred.
We define our update authority - in our case, we set that as the SIGNER_WALLET in our table creation step above.
We pass in the lookup table account address (which we defined as LOOKUP_TABLE_ADDRESS).
We pass an array of addresses into our lookup table. We will pass in a few random public keys, but you can pass in any public key that you like! The Program's "compression" supports storing up to 256 addresses in a single lookup table!
log a link to our lookup table entries for easy access after the transaction is complete. Finally, we pass TransactionInstruction into createAndSendV0Tx to generate a transaction and send it to the network!

Call the new function:

addAddressesToTable();

Now run the in the terminal using the previous ts-node app.tscommand. Check the returned URL to see your transactions and your lookup table entries on the Solana Explorer. On the lookup table entries, there will be a list of all the stored public keys.

By now perhaps you are asking, why is this better? Well, The power of ALTs in transaction lies is in the efficiency of the transaction! By effectively compressing a 32-byte address to a 1-byte index value
This is great for speed as well as space efficiency as you can push in more transactions without bothering about space. By eliminating the need for long and complex iterations in cases of numerous addresses, which leads to error-resistance as ALTs can be engineered to prevent duplicate addresses.
Also this is cheaper; by reducing the number and size of the instructions in a transaction, thereby making fees required to develop on the blockchain way more cost-effective.
From transaction caching operations, scalability, speed, cost, the advantages of the address lookup table cannot be overemphasized, the benefits are numerous!
of course they are not the best when it comes to dealing with a dataset of unpredictable large number of possible keys, and could exceed the number of keys that are actually stored. For this sort of cases, hash tables are more efficient. Read more about hash tables here

While most use cases of ALTs revolve around transactional caching, decentralized applications employ their power to store multiple address for easy access at O(1) time; within a single transaction, you can load a large number of addresses

A very practical use case for ALTs can be found in asset management dApps where multiple wallets are being monitored; without Solana ALTs, it would be a resource consuming process.

For further reference, You can find another simple example to build on the example we have shown already Github

Conclusion

We have learned the following points in this terse tutorial:

In blockchain transactions, the magnitude of addresses used in multi-level transactions can easily scale in magnitude of millions of addresses; hence requiring an efficient means of retrieving transaction data in an O(1) time
Lookup tables are best for this as they utilize a concept known as "direct addressing" which utilizes array indexing to compute the running phase of a program.
By utilizing Address Lookup Tables, blockchain developers can create a collection of related addresses to efficiently load more addresses in a single transaction; address tables can store up to 256 addresses each. This can however be a limitation in special cases. In addition to stored addresses, address table accounts also tracks various metadata.