Forem: Valentin Madrid

A deep dive into verifiable programs on Solana

Valentin Madrid — Mon, 24 Jul 2023 16:40:28 +0000

A big problem with blockchains is that you can't know the code behind an on chain program, meaning you would have to blindly trust a protocol with your signature, not being able to verify what's really going on behind the scenes.
That's why a lot of companies and projects publish their programs code to build trust. But now the problem is, "how can I be sure the public source code is the same as the deployed programs source code ?".
That's why Ellipsis Labs has built a build verification CLI, which can be used to compare an on-chain program byte code to the build output when you build its underlying program.

Let's now take a look at how you can create a program and verify it using Ellipsis Labs verifiable builds.

Creating a program

First of all, initialise your program:

anchor init verifiable-build

Now let's add a little functionality to our program so it isn't the boring standard one.

Replace the existing content with the following code in programs/verifiable-build/src/lib.rs:

use anchor_lang::prelude::*;
use anchor_lang::system_program;

declare_id!("AD62W7aAQETg11ajkadk57DYfPTWPzKLiXuLZVB4eQaw");

#[program]
pub mod verifiable_build {
    use super::*;

    pub fn transfer_sol(ctx: Context<TransferSol>, amount: u64) -> Result<()> {
        system_program::transfer(
            CpiContext::new(
                ctx.accounts.system_program.to_account_info(),
                system_program::Transfer {
                    from: ctx.accounts.payer.to_account_info(),
                    to: ctx.accounts.recipient.to_account_info(),
                },
            ),
            amount,
        )?;

        Ok(())
    }
}

#[derive(Accounts)]
pub struct TransferSol<'info> {
    #[account(mut)]
    recipient: SystemAccount<'info>,
    #[account(mut)]
    payer: Signer<'info>,
    system_program: Program<'info, System>,
}

Let's build the program now:

anchor build

Deploying your program

First, let's set the network we want to deploy our program on, I will be doing it on a local Solana validator.

Set your network running this command:

solana config set -c --url l

Now launch a local solana validator in a separate terminal:

solana-local-validator

And now deploy your program

anchor deploy

Verify the deployed program

Now comes the important part. Let's verify that the byte-code of the program we just deployed matches the program we've written.

Install the verifiable-build CLI

cargo install solana-verify

Get local executable hash

First, let's get the hash of the build output from our program:
Here, make sure to use the program's lib name, found under [lib] in Cargo.toml.

solana-verify get-executable-hash target/deploy/verifiable_build.so

This is the output we get:

Get hash from program bytecode

And now let's get the hash of the program bytecode:
Here, replace $PROGRAM_ID by the program id of your deployed program.

solana-verify get-program-hash -u http://localhost:8899 $PROGRAM_ID

We can see here that both of the hashes are the same, meaning we've proven that the program you've built locally corresponds to the one you've deployed.

Verify an on chain program from a Github repository

For this, I've created a Github repository with the program we've just built and also deployed it onto the Solana devnet.
You can do the same but don't need to, as you can just use this with the repository I have created.

Verify hash between repo and program

To verify a hash between an on-chain program and a GitHub repository, run this command in your terminal:

solana-verify verify-from-repo -ud --program-id AD62W7aAQETg11ajkadk57DYfPTWPzKLiXuLZVB4eQaw https://github.com/valentinmadrid/solana-verifiable-builds-example

If the hash from both of these programs match, you can consider a program's source code to be verified.

Conclusion

Verifiable builds are really important. That's why it's great to see companies like Ellipsis Labs coming up with amazing ways of verifying if the code of an open-source program matches it's deployed version.

This is a great step forward for achieving a better open source culture on Solana. But, we still have ways to go. It would be amazing to see projects take initiatives to build tools such as a registry of verified programs for explorers, GitHub workflows that can update a verifiable hash in a database each time a program is updated or even wallets that integrate verifiable builds to spot harmful program interactions.

Thanks for reading this post. Feel free to drop any feedback on my Twitter: @valentinmadrid_.

Interact with your Solana programs in Python

Valentin Madrid — Fri, 05 May 2023 08:38:52 +0000

Most of client side interaction with Solana Programs is done in Typescript for now. That makes sense, because most frontend frameworks are built on top of Javascript.
But what if you want to use Python to interact with your program ? Let's see how this can be done in this short tutorial.

The library we'll be using is called AnchorPy, and was made by Kevin Heavey.
Link to the AnchorPy documentation: https://kevinheavey.github.io/anchorpy/

Let's create some tests for an Anchor program in Python. Here's a link to the program's repo we will the interacting with.

Install AnchorPy

First of all, install anchorpy and solders:
pip3 install anchorpy
pip3 install solders

Now, let's create a test file at tests/test.py.

First of all, let's import some dependencies:

import asyncio
from anchorpy import create_workspace, close_workspace, Context
from solders.system_program import ID as SYS_PROGRAM_ID
from solders.pubkey import Pubkey
from solders.keypair import Keypair
from solders.sysvar import RENT

Testing our Anchor program

Our code will compose of a base that looks like this:

async def main():
    # Read the deployed program from the workspace.
    workspace = create_workspace()
    # The program from the workspace we want to use
    program = workspace["seahorse"]

    # Close all HTTP clients in the workspace.
    await close_workspace(workspace)

asyncio.run(main())

Ok, now we can get to our test's logic. We have three instructions in our Anchor program. Let's get through them and document everything that is happening.

First, we define the two Program addresses for the Token program and the Associated Token Program. Then we generate a Keypair for the Token Mint we're going to create afterwards.

TOKEN_PROGRAM_ID = Pubkey.from_string("TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA")
ASSOCITAED_TOKEN_PROGRAM_ID = Pubkey.from_string("ATokenGPvbdGVxr1b2hvZbsiqW5xWH25efTNsLJA8knL")

# Create a Mint keypair. This will be our token mint.
mint = Keypair()

Invoking instructions

Now, we call the create_token instruction from our program and print the transaction signature.

create_token = await program.rpc["create_token"](ctx=Context(accounts={
        "mint": mint.pubkey(),
        "signer": program.provider.wallet.payer.pubkey(),
        "system_program": SYS_PROGRAM_ID,
        "rent": RENT,
        "token_program": TOKEN_PROGRAM_ID
    }, signers=[program.provider.wallet.payer, mint]))

print("Create token signature: ", create_token)

You may see some similarities with the Typescript Anchor library here. "create_token" is the instruction to be executed and the "accounts" context are the accounts passed into our instruction. The signers here are the Anchor workspace keypair and the Mint Keypair, needed to sign the creation of a Mint account. Pretty straightforward right ?

Derive PDA's

We want to create an associated token account now, for that we will need it's Public Key. You can do that by using find_program_adress.

associated_token_account_pubkey, nonce = Pubkey.find_program_address([bytes(program.provider.wallet.payer.pubkey()), bytes(TOKEN_PROGRAM_ID), bytes(mint.pubkey())], ASSOCITAED_TOKEN_PROGRAM_ID)

We can now use this Pubkey in the instruction's accounts. This would look like:

create_associated_token_account = await program.rpc["create_associated_token_account"](ctx=Context(accounts={
        "mint": mint.pubkey(),
        "token_account" : associated_token_account_pubkey,
        "signer": program.provider.wallet.payer.pubkey(),
        "system_program": SYS_PROGRAM_ID,
        "rent": RENT,
        "token_program": TOKEN_PROGRAM_ID,
        "associated_token_program" : ASSOCITAED_TOKEN_PROGRAM_ID
    }, signers=[program.provider.wallet.payer]))

print("Create associated token account signature: ", create_associated_token_account)

Pass arguments to instruction

Last, let's see how we can add additional arguments into our instruction by minting some tokens to our workspace Keypair. Here we are mining 1000 tokens to the Token Account created before.

mint_token = await program.rpc["mint_token"](1000, ctx=Context(accounts={
        "mint": mint.pubkey(),
        "recipient" : associated_token_account_pubkey,
        "signer": program.provider.wallet.payer.pubkey(),
        "system_program": SYS_PROGRAM_ID,
        "rent": RENT,
        "token_program": TOKEN_PROGRAM_ID,
    }, signers=[program.provider.wallet.payer]))

print("Mint token signature: ", mint_token)

Now, we could run our tests in the command line, but let's do something more convenient. Let's go to the Anchor.toml file and change the current test script to this:

[scripts]
test = "python3 tests/test.py"

You can now run anchor test --skip-local-validator and run the tests we wrote.

Conclusion

Thanks for reading this guide.
If you have any more questions, please contact me on Twitter or join the SolanaU Discord Server.

A guide to Solana Geyser Plugins

Valentin Madrid — Wed, 03 May 2023 10:29:32 +0000

What even is Geyser ?

Geyser consists of something called the "Solana Geyser Plugin Interface". This interface enables developers to build Plugins into validators. The Plugin interface will then listen to events on the Solana Blockchain and notify the Plugin of those events.
These events can be:

account updates, meaning every time an account is updated on the chain, it will notify your Plugin.
new transactions
new slots

This enables amazing use cases, such as streaming all account changes for your specific program to a database. This means you could perform initially high load RPC requests such as getProgramAccounts directly from your favourite database, which is pretty cool and makes the process up to 10 times faster

Examples

There are several already developed Geyser Plugins which you can run, here is a quick overview:

AccountsDB Postgres: Stores your accounts and transactions in a Postgres database.
AccountsDB Kafka: Streams account changes to Kafka.
AccountsDB Plugin Bigtable: Streams tx + accounts to Google Bigtable.
Yellowstone gRPC: gRPC interface for Solana.
Waverider: Streams account data from specific programs (PDA's) to a Supabase instance, which can then be queried via Supabase realtime.

Requirements

To run a Geyser Plugin, you will need to run a validator or an RPC node, which is essentially a non voting validator that can index data from the chain.
The hardware requirements will vary on the cluster you want to run your validator on. I strongly recommend you take a look at the recommended specifications to run a non voting validator.
In this post, we are going to use the Solana Local Validator, which you can run with pretty much every hardware.

How to run a Plugin ?

It's pretty straightforward, you just have to start a Solana validator with the Geyser Plugin you want to run attached. Here is an example:
solana-test-validator --geyser-plugin-config <path to your plugin config>
Let's take a look at an example plugin scaffold, here.
Create a file where you want your plugin to live, and open a terminal in it. Then clone the Plugin.
git clone https://github.com/mwrites/solana-geyser-plugin-scaffold.git .
We are now ready to go, but there is something important to take notice of:
⚠️ You need to build the Plugin with the same Rust version as you've build your Solana CLI. Also, your Solana CLI version should match the solana-geyser-plugin-interface in the plugin's Cargo.toml file.
The Interface version of this plugin is: 11.1.1. To install the corresponding Solana CLI version, run:
sh -c "$(curl -sSfL https://release.solana.com/v1.11.1/install)"

Now run cargo build.

There's one more thing to do, in your config/config.json file, make sure to set your file extension for "libpath" to .so if you are on Linux or Windows or .dylib if you are on mac.

Your plugin should now run and log every single account change happening on the ledger, including it's Public Key and the current slot. You can also get way more information such as the account's data Buffer. Pretty cool right ?

Build your own plugin

Now while this is pretty cool, you might think of a lot of use cases this plugin framework enables. While I recommend that you work up from this Plugin Scaffold, I can still give you a little insight on how the plugin interface works.
After defining an entrypoint to your program, for which you can find an example here.
Now the validator will call a couple functions whenever certain events are triggered. Some examples are:

// This gets called while the validator is still starting up
fn on_load(&mut self, config_file: &str) -> solana_geyser_plugin_interface::geyser_plugin_interface::Result<()> {

// This gets called when startup has ended
fn notify_end_of_startup(&mut self) -> solana_geyser_plugin_interface::geyser_plugin_interface::Result<()> {

// This gets called every time account data gets changed
fn update_account(&mut self, account: ReplicaAccountInfoVersions, slot: u64, is_startup: bool) -> solana_geyser_plugin_interface::geyser_plugin_interface::Result<()> {

// This gets called each time there is a new transaction
fn notify_transaction(&mut self, transaction: ReplicaTransactionInfoVersions, slot: u64) -> solana_geyser_plugin_interface::geyser_plugin_interface::Result<()> {

// These two are basically the configuration deciding if the interface should notify account changes and transaction notifications
fn account_data_notifications_enabled(&self) -> bool {
       true
}
fn transaction_notifications_enabled(&self) -> bool {
       true
}

Conclusion

As you've probably noticed, this Geyser thing is pretty cool. Go build something with it !
Feel free to tag me on Twitter if you need any help.

--
Speaking of Geyser ! I have built this validator extension called Waverider. It essentially allows you to stream your program's account changes to a Supabase Database, making gPa's 10 times faster and you're even able to listen to realtime account changes, which is essential to a lot of dApps. Again, feel free to contact me if you have any questions on this.

Amman - Your Solana development workflow on steroids

Valentin Madrid — Tue, 02 May 2023 22:38:48 +0000

Hello fellow Solana dev !

In this blog post, we will cover a tool which is going to simplify your development workflow when testing your programs locally. It’s called Amman. Amman describes itself as a Toolbelt to help test Solana SDK Libraries and apps on a local running validator.

To put it simply, it takes the existing Solana local validator and adds a couple really great features to it, such as loading programs from other clusters like the Solana mainnet, labelling key pairs or making airdrops in your anchor tests. Amman also comes with its own explorer to match all of its additional features.

If that didn’t make sense to you, it’s all good, we’re going to explore them the further we go down this post.

Let’s get started!

PS: If you don’t want to read the whole post. There is a very short guide at the end to cover all Amman basics without explanations.

Prerequisites:

You’ve already read the Anchor documentation
You have the Solana Tool suite installed
If you want to follow along with the same Anchor program as the one I will be covering today, clone it by clicking here.

Installing Amman

First of all, head to the root of your Anchor project.
If you want to install Amman only for this specific project, run this inside your terminal:
yarn add @metaplex-foundation/amman
or
npm install @metaplex-foundation/amman
If you want to have Amman installed globally, run
npm install -g @metaplex-foundation/amman.

Configurating Amman

Once you have successfully installed Amman, create a file called .ammanrc.js in your project root. Add the following code to it:

module.exports = {
  validator: {
    killRunningValidators: true,
    programs: [],
    accounts: [],
    jsonRpcUrl: "127.0.0.1",
    websocketUrl: "",
    commitment: "confirmed",
    ledgerDir: "./test-ledger",
    resetLedger: true,
    verifyFees: false,
    detached: false,
  },
  relay: {
    enabled: true,
    killlRunningRelay: true,
  },
  storage: {
    enabled: true,
    storageId: "mock-storage",
    clearOnStart: true,
  },
};

This is where the configuration of your Amman instance lives. For now, all you need to know is that:

the killRunningValidators field will stop any running local validator process
jsonRpcUrl is the url where your validator should listen to rpc requests
ledgerDir is where the validators ledger will be created or used(if it already exists). By default it will be stored in the your projects root.
resetLedger determines if you want to reset the validators existing state every time you start Amman.

Now, you’re ready to start Amman. Run amman start inside your terminal.
The output should look similar to this:

Loading config from /Users/devenv/Documents/GitHub/amman-example/.ammanrc.js Running validator with 0 custom program(s) and 0 remote account(s) preloaded Killed currently running solana-test-validator Launching new solana-test-validator with programs predeployed and ledger at ./test-ledger Successfully launched Relay at :::50474 ws error: connect ECONNREFUSED 127.0.0.1:8900 Successfully launched MockStorageServer at :::50475 ws error: connect ECONNREFUSED 127.0.0.1:8900 ws error: connect ECONNREFUSED 127.0.0.1:8900 Waiting for fees to stabilize 1... Waiting for fees to stabilize 2...

Use Amman in your testing workflow

Good, you have a running Amman instance now. We’ll be able to run some cool tests for our program. We only care about one instruction, to send a token which we will send from one person to another.

Again, I have created an example repository for what I will explain in this post, you can find the code here.

Loading mainnet programs

By taking a look at the test file, we can see that we need to pass in the Token Program to call both instructions.

But how do I use the token program when testing locally ? We can tell amman what programs we want to load by passing them into the programs or the accounts field.

This is how to add the mainnet Token Program to your Local Validator:

Add the following program into the “accounts” field in .ammanrc.js.

accounts: [
  {
    label: "Token Program",
    accountId: "TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA",
    executable: true,
  },
  {
    label: "Random other example account",
    accountId: "4VLgNs1jXgdciSidxcaLKfrR9WjATkj6vmTm5yCwNwui",
    // You can set the cluster on a per account basis
    cluster: "https://metaplex.devnet.rpcpool.com",
  },
],

It‘s important to set "executable" to "true" in this context, because we want to clone a program. If you just want to load data from a non executable account, set it to false. You can also specify the RPC URL from where you want to load these accounts.

Anchor tests in Typescript with Amman

First of all, import Amman at the top of your test file.

import { Amman } from "@metaplex-foundation/amman-client";

Initialize an Amman instance.

const amman = Amman.instance();

Labeled Keypairs

In our program, there should be one Keypair that mints a Token and sends it to the other. We‘re going to call the sending Keypair Alice, and the receiver should be called Bob.

This is once again where Ammans magic comes in. You can label Keypairs, which makes it easier to understand the different accounts inside the Amman Explorer. This is how you generate a new labeled Keypair:

let aliceAmman = await amman.genLabeledKeypair("Alice");
let bobAmman = await amman.genLabeledKeypair("Bob");

The output of this will be an object with:

The Public Key
The Keypair
A string

The actual Solana Keypair is at the index of 1 of the Keypair:

const alice = aliceAmman[1];

You can also label any existing accounts like this:

// publicKey will be the public Key of the account you want to label.

amman.addr.addLabel("Bob", publicKey);

Airdrop SOL

Now we have another problem, your Keypairs will not have any SOL, and will therefore not be able to pay the transaction fees.

But you can easily handle airdrops with Amman, just add this to your existing code:

// Airdrop 1 SOL to Alice
await amman.airdrop(program.provider.connection, alice.publicKey, 1);

We should be good to run our tests now.
If you don’t have a running Amman instance yet, start one in a seperate Terminal using amman start. Then, run your tests with Anchor by running: anchor test --skip-local-validator.

The Amman explorer

You can find it at: https://amman-explorer.metaplex.com/
If Brave doesn’t work, try disabling the Brave Shield.
The explorer is a copy of the Solana explorer that includes Amman’s features.
If everything worked well, it should be connected to your running Amman instance, you can see that in the top left of your screen.
Find the transaction you want to inspect, you will be able to see Token movements and involved accounts thanks to their label. Here’s an example:

Read deserialized account data

You can read account data directly from the explorer. For that, head to any account and click on the Resolved Info Tab. This should look similar to this image.

Conclusion

Amman is amazing
Amman is an amazing Solana Dev Tool and I recommend to make usage of it. Here’s a really short guide if you don’t want to read trough the whole tutorial:

Short guide

I’ve made a quick recap for you here:

Install Amman

yarn add @metaplex-foundation/amman

Add Amman Configuration

Create a .ammanrc.js file in your project root. Add the following code.

Start Amman

amman start

Use Amman in Tests

1 Import Amman Client

import Amman from "@metaplex-foundation/amman-client";

2 Initialize Amman instance

const amman = Amman.instance();

3 Generate Keypair

let aliceAmman = await amman.genLabeledKeypair("Alice");
const alice = aliceAmman[1];

4 Label Accounts

amman.addr.addLabel("Bob", publicKey);

5 Airdrop SOL

await amman.airdrop(program.provider.connection, alice.publicKey, 1);

The Amman explorer

Find the Amman explorer here.
The explorer has a lot of great features like being able to display deserialized account data, go and see what it can do.