Forem: Dash Core Group Devs

Dash Core Group Introduces: GroveDB

Dash Core Group Devs — Wed, 28 Jun 2023 14:23:50 +0000

We're happy to introduce you one of the most powerful tools for building web3 infrastructure, GroveDB.
GroveDB stands as the pioneering database that empowers cryptographic proofs for intricate queries.

Originally designed for Dash Platform, GroveDB boasts seamless integration capabilities with other applications seeking similar functionality.

So, how can Grove DB be helpful?
This is a list of Grove DB features:

Query proofs - GroveDB holds the distinction of being the first-ever database to offer support for proofs encompassing queries of greater complexity beyond simple single-key retrievals.

Secondary indexes - With GroveDB, you have the power to effortlessly create indexes on any field, including unique indexes and compound indexes. This flexible feature allows for efficient data organization and retrieval within the database.

Mutable and immutable data storage - GroveDB comes equipped with built-in functionality that seamlessly supports both types of indexes within a single database. This integrated capability ensures a comprehensive and versatile solution for efficient data management and retrieval.

Inclusion and absence proofs -
GroveDB revolutionizes data verification by introducing Merkle proofs, enabling the capability to verify the presence or absence of items with utmost accuracy. This feature enhances data integrity and security within the database environment.

Operation costs - Gain valuable insights into operation costs with GroveDB, allowing you to accurately assess and estimate costs before executing them. This invaluable feature empowers users to make informed decisions and optimize resource allocation within their database operations.

ACID compliancy - Experience the benefits of ACID (Atomicity, Consistency, Isolation, Durability) compliance with GroveDB. This robust database ensures data integrity, reliability, and transactional consistency, providing a solid foundation for reliable and secure data management.

In general, GroveDB offers even greater functionality compared to other databases.

Architecture

Insertion and deletion operations function as expected in GroveDB, updating the relevant subtrees and providing appropriate proofs of membership or non-membership.

Tree Structures

Instead of using separate authenticated data structures, GroveDB adopts a unified hierarchical authenticated data structure based on the concept of Database Outsourcing with Hierarchical Authenticated Data Structures. Elements within the structure represent the most fundamental units and can be represented in various forms, such as items, item references, trees, trees with items, or trees with item references. An element encompasses an item, a reference to an object, or a subtree.

The tree implementation in GroveDB is based on our modified version of Merk, with custom patches for enhanced compatibility with GroveDB. Merk stands out for its AVL tree structure, where intermediate nodes contain key/value pairs. Each node includes a third hash, called kv_hash, in addition to the hashes of its left and right children. The kv_hash is calculated as kv_hash=H(key,value), and the node hash is computed as H(kv_hash, left_child_hash, right_child_hash). Merk utilizes Blake2B hashing, while rs-merkle employs SHA256.

Storage

RocksDB, a key-value store originally derived from LevelDB and further developed by Facebook, serves as the underlying storage engine for GroveDB. We selected RocksDB due to its exceptional performance, maturity, and compatibility with our technology stack. Merk itself is built on top of RocksDB.

GroveDB incorporates three types of storage: auxiliary storage, metadata storage, and tree root storage. Auxiliary storage stores plain key-value data that is not utilized in the consensus process. Metadata storage holds information beyond the scope of GroveDB usage and does not have any prefixes or association with subtrees; it exists at a higher level. Tree root storage, on the other hand, is responsible for storing subtrees.

A database transaction in GroveDB wraps around the OptimisticTransactionDB primitive provided by RocksDB. An optimistic transaction operates under the assumption that conflicts will be infrequent, which are detected during the commit phase. This differs from the pessimistic model that relies on locks.

Setting up

Install rustup using your preferred method.
Rust nightly is required to build, so make you are using the correct version.

rustup install nightly

Clone the repo and navigate to the main directory:

git clone https://github.com/dashevo/grovedb.git && cd grovedb

From here we can build

cargo build

Platform & Core Dev Updates Journal - May 2

Dash Core Group Devs — Wed, 03 May 2023 14:34:39 +0000

Dash Platform has never been that close to the release! In this wrap-up, we will provide an overview of the latest developments and updates to the Dash Platform, highlighting the progress.

Watch the video or check the short summary below!

Dash Platform & Core Development Update - 2023 May 2 - YouTube

Dash Platform & Core Development Update - 2023 May 2

youtube.com

Dashmate stabilization

What We Planned

Testing Dashmate in progress
Dashmate E2E tests in progress
Dashmate debian package for arm64 done
Remove Docker dependencies from debian package done
Warning to open Dashmate package for Mac done
Generate random RPC user and password on dashmate setup done
Can't find begoo/index with yarn dashmate setup done
Reward shares can be negative during HP masternode registration done
When submit the form with BLS private key and reward share, it accepts invalid BLS key done

What Is Done

Invalid protx register command in output of dashmate setup done
Dashmate status shows empty masternode status done
Select number of masternodes during local setup freezes done
Error form ZeroSSL during IP address verification done
Dashmate for Windows doesn't handle paths properly done
Provide helpful information if Docker or Docker Compose v2 is not installed done
The Docker socket not found error using Docker Desktop done
Dashmate local setup ends up with deadline exceeded error done
Multiple Dashmate group reset command issues done

v0.24 deployment

What We Planned

Update deployment tool to deploy with dashmate version and branch done
Platform v0.24-dev deployment and testing on devnet done
Handle double init chain call in Drive done
Handle duplicated masternode keys on testnet done
Platform v0.24-dev deployment to testnet in progress
Failed recovering or verifying threshold signature: threshold block signature is invalid in progress
Compare js-dpp and wasm-dpp tests done
createdAt != updatedAt on document create done

v0.25 Development

What Is Done

Masternode Identities in review done
Persist ephemeral state in testing done
Completely clean DPP serialization and deserialization done
We used Proc Macros (very clean code)
Implemented a Proc Macro for signable bytes done
Initialize chain on same state transition done
Validator set update on init chain done
Heavy tests for masternode list updates done
EDDSA-Hash160 key type done
System Key Purpose type done
Non unique keys full support (public key hashes, script hashes) done
v0.24 backports (Up till Friday) done

Deployment - Phase 1 (Dashmate Local)

Successfully deployed to Dashmate
Currently on height 7000 + with absolutely no errors
Multiple Validator Set Rotations have happened
Will try to run for two more weeks to see masternode payouts

Deployment - Phase 2 (Devnet)

Multiple deployments in progress Goal: Test Platform activation when core transitions from v19 to v20
Deployment 1 (failed): 1 wrong parameter Fixed
Deployment 2 & 3: Failed because of issue in RPC Core client (both times some optional fields were not allowed to be empty) Fixed
Deployment 4 and more in progress

What Is Left

Fix queries / SDK.
Validation costs
Just in time atomic state transition payments
Finalization of withdrawals
Versioning in progress
A lot of Testing

Core Team

V19 Core Upgrade Status

Hard Fork signalling began
~17.5% Miner signaling
Masternodes ~65% updated
Earliest Possible Activation May 13th -> May 20th

Core Status - Current Development

V19.0.1 Expected
V20 On Chain CLSigs / Quorum Selection based on CLSigs in progress / partial devnet testing
Implementation /backporting of GUIX; replacement for gitian in progress
Enhanced Hard Fork Implementation in progress
Asset Locks devnet testing
Masternode Reward Location Reallocation in progress
Merged PRs: 5334, 5336, 5338, 5339, 5346, 5341, 5340, 5255, 5348, 5351 done

Platform & Core Dev Updates Journal – Feb 7

Dash Core Group Devs — Tue, 14 Feb 2023 19:33:49 +0000

So, what happened over the last week with Dash Platform?
Short summary below.

TenderDash

Stabilization

What We Planned

Stabilization and Improvements (v1.0)

fix(config)!: quorum type set to 0 during replay at genesis done
conversion of PEM-encoded ED25519 node keys done
feat(cmd): derive node key from bip39 mnemonic done
build(docker): add abcidump to release image done
Seed connectivity issues in progress

Investigate deadlocks (v1.1)

refactor: add-vote operation done
refactor: DoPrevoteCommand replaced on Prevoter implementation done
refactor: consensus peer gossip mechanism in review
refactor: add-vote operation in review
refactor: consensus state to fix deadlock in review

What’s Left before Testing Phase

~~Stabilize same-block execution~~
~~Update BLS library~~
Updated BLS library integration
Adjust vote extensions for withdrawal transactions
~~Improve seed node operations~~
~~Drive integration~~

GroveDB

Tasks:

Sum tree parent costs done
Merk Proof Specification draft done
GroveDB Proof Specification draft in progress

Protocol

Identity v2

Protocol Tasks 1.

Identity v2 done
Efficient Root Tree done
Balance Tree done
Sum Tree Integration done
Strategy tests done
Total credit balance verification done
Calculate expected total credits amount done
Efficient Data Serialization in progress
Refactor DPP document types done
Variant Protocol versions done

Protocol Tasks 2.

Add support for Core v19-beta.X in progress
Create self-contained Dashmate package in progress
Use HTTP port for gRPC native done

Credit Withdrawals

Introduce withdrawals data contract in review
Implement the pooling process (type 0 only) for withdrawals in review
Update status of withdrawals after broadcasting in review
Add revision logic to IdentityCreditsWithdrawal transition in review
Indices from expired withdrawal transactions must be reused in review

What’s Left

Major

~~Fee Refunds~~
Accomplish withdrawals in review

Minor

~~Logging of same block execution~~
~~Use block candidate context in State Repository~~
~~Deal with situation where fees are higher than Balance~~
~~Add Sum Tree feature for Pool trees~~
Asset Lock and Unlock transactions integration (blocked by dpp integration)
~~BLS library rust binding~~ not for beta 1
Masternode block reward distribution
Withdrawals verification after rust dpp integration

Rust Port Update

What we planned

Merged all Identities and Asset Lock pull requests done
Asynchronous State Repository done
JSON-Schema missing functionality in progress
Documents Transitions in progress (2 integration tests left)
Data Contract facade and integration in review
V24 backports and removing tech debt in progress
Unified error format to rs-dpp in progress

What's next

Finish v24 backports
Merge all unmerged PRs that are related to Documents and Data Contracts
Finish adding missing json-schema features
Since Identity v2 has been merged as well, we can now start the platform integration (DAPI/Dive/SDK) after finishing the v24 backports

Core Dev

What’s next?

v19.0.0 Release Candidate
Unlikely v18.2.2 Release w/ minor fixes
Continued maintenance, backporting and feature development

GroveDB Sum Trees

Dash Core Group Devs — Thu, 09 Feb 2023 17:45:14 +0000

One of the innovative features of Dash Platform that we at Dash Core Group are excited to present is a GroveDB feature called sum trees. GroveDB sum trees are Merkle AVL trees whose root nodes hold the sum of all integer values in the tree. Any time a value is added, removed, or updated in a sum tree, every parent node and hence the root’s “sum value” is updated. Sum trees are a very powerful tool for trustlessly and efficiently retrieving the sum of a set or range of values, as it would take a relatively large, often enormous, amount of computing resources to achieve otherwise. In turn, vital security checks are made highly efficient and simple, and a whole new arena of use cases is opened up for dApps.

Architecture

Sum trees have a simple architecture. In addition to the hashes of regular Merkle tree nodes, each node in a sum tree holds a “sum value” and an optional “sum item”. A sum item is a key-value pair where the value is a signed integer. It can represent anything that has an integer value, such as an identity’s credit balance or the number of unsold tickets remaining for a concert. A sum value is the value of a node’s own sum item plus the sum values of its child nodes.

As illustrated in the diagram above, the sum tree leaf nodes’ sum items are equal to their sum values. There’s nothing to sum. Moving up a layer, each parent node has their own sum item (which is optional, as will be explained next) and a sum value, which is the sum of their child nodes’ sum values plus their own sum item. This structure repeats itself from the leaves up to the tree’s root node, where the entire tree’s sum value is held.

Note that all sum tree nodes hold sum values but don’t need to hold a sum item. In other words, you can still hold any type of element in the tree, such as a key-value pair with a string value, a non-sum subtree, or even an integer value that isn’t a sum item. These elements won’t affect the sum value of the tree. You’re therefore able to collect sums in the same tree you use to prove the existence of non-sum elements with Merkle proofs.

Application

The general use case for sum trees is to trustlessly and efficiently retrieve the sum of a set or range of values. Developers can implement sum trees in their projects built on Dash Platform or any other platform which implements GroveDB. This section will walk through some example use cases.

The Dash Platform engine itself uses sum trees to store the balances of all identities and reward distribution pools, and constantly checks them to ensure there are no inflationary or other bugs present. This is actually a huge innovation for blockchains, as inflationary bugs have been rather common and destructive to blockchains in the past. These checks provide a strong layer of security against them. Dash Platform runs these sum tree checks on every block submitted to consensus, and any which don’t pass are prevented from entering the state.

To illustrate the massive efficiency gains sum trees enable, say you have a tree which contains the number of restaurants that accept Dash in each city of a US state. The root of the tree would be the total number of Dash-accepting restaurants in the state. If you’re a true Dash fanatic, you could compare the root sum value of that state’s tree with those of all the other states to help you determine which state to live in. Say that every city in the US has at least one restaurant that accepts Dash. With sum trees, to get this information, you would only need to query the root node of each state’s tree (50 states), verify the proofs, and compare the values. With regular Merkle trees, you’d need to query every city in every state (over 100,000 cities), verify the proofs for each city, add the values up yourself for each state, and then compare.

The efficiency savings in the example above are on the order of 2,000 times. Two other real-world applications where sum trees would be beneficial are:

Proof of solvency: a topic which has been under much discussion again lately with the recent collapse of FTX. An exchange could store their users’ balances in a sum tree along with their hashed usernames. Users could be provided with Merkle sum proofs showing that their balances are correctly included as part of the total. To prove that they hold at least enough assets to cover the sum tree’s root value, the exchange could then do something like periodically sending a specified amount to a specified address at a specified time. While the current proof system in GroveDB would present some privacy leaks under this design, the future addition of more advanced zero-knowledge proof constructions will make this a very powerful application.

Custom tokens: Dash Platform enables developers to easily create custom tokens for their data contracts with the address balances automatically stored in a sum tree. Each node of the sum tree holds an address’s balance as its sum item, and the root node can be used to check for bugs. It doesn’t only add an extra layer of security to the token contract, but it’s also a very efficient method of keeping track of address balances compared to current methods.

Looking Forward

Sum trees open the door to a whole new arena of security measures and use cases that trustless systems and dApps can take advantage of. They’re just another example of innovations GroveDB is bringing to the field of blockchain databases, and we at DCG are excited to show off the efficiency and security that Dash Platform will be able to provide utilizing tools like this. For future releases, we are investigating the integration of Verkle trees and zero-knowledge proof constructions, which have the potential to make sum tree applications even more powerful and versatile. For more details on sum trees, stay tuned for a Dash Improvement Proposal coming soon, written by our CTO Samuel Westrich and GroveDB developer Wisdom Ogwu.

Originally posted on Dash Blog

Platform + Core Dev Updates – January 24

Dash Core Group Devs — Thu, 26 Jan 2023 14:37:26 +0000

Hello Dashers!
Here is a new portion of fresh Platform updates, now with the Core Dev team! They're gonna give you in-depth information on the development progress.
Watch the video below, or read our short takeaway of the tasks done this sprint.

TenderDash

Task: Stabilization

What We Planned

Stabilizing (v.0.10 / v0.11 => v1.0)

feat(abci): add quorum hash to RequestPrepare/ProcessProposal done
refactor(proto)!: cleanup protobuf definitions and reorganize fields done
chore(release): update changelog and bump version to 0.10.0-dev.8 done
chore: backport changes from v0.10-dev.8 to v0.11-dev done
build(docker): use version 1.2.5 of BLS lib in Docker done
refactor: replace peerID on proTxHash in peer-catchup-rounds map done
updated dependencies (build-push-action, buf-setup-action, golangci-lint-action) done

Investigate deadlocks (v.1.1)

refactor: consensus peer gossip mechanism in progress
refactor: add-vote operation in progress
refactor: consensus state to fix deadlock in progress

What’s Left before Testing Phase

~~Stabilize same-block execution~~
~~Update BLS library~~
Adjust vote extensions for withdrawal transactions
~~Improve seed node operations~~
~~Drive integration~~

GroveDB

What we planned

Better path query merge done
Proof of mixed levels done

What’s next?

Paths in Proof results done
Optionals for Proof results done
Partial proof verification (f(g(x)) in review (integrated)

Protocol

Task: Identity v2

Credit Withdrawals

Introduce withdrawals data contract in review
Implement the pooling process (type 0 only) for withdrawals in review
Update status of withdrawals after broadcasting in review
Add revision logic to IdentityCreditsWithdrawal transition in review
Indices from expired withdrawal transactions must be reused in review

Other Protocol Tasks

Identity v2 in progress
Total credit balance verification in review
Calculate expected total credits amount in review
Efficient Data Serialization in progress
Refactor DPP document types in progress

More Protocol Tasks

Add support for Core v19-beta.X in progress
Create self-contained Dashmate package in progress
ABCI context logger done
DPP should not allow using existing property in a new index done
Use single HTTP port for all DAPI endpoints done
Log number of refunded epochs in Drive done

What’s Left

Major

~~Fee Refunds~~
Accomplish and test withdrawals

Minor

~~Logging of same block execution~~
~~Use block candidate context in State Repository~~
~~Deal with situation where fees are higher than Balance~~
~~Add Sum Tree feature for Pool trees~~
Verify fee processing costs
Asset Lock and Unlock transactions integration
BLS library rust binding
Masternode block reward distribution

Rust Port Update

What We Planned

Our primary goal for this sprint was testing and implementing the last thing left - Identities
Integrate Rust DPP into SDK, Drive, DAPI and test suite
Identity transitions - AssetLockProofs and CreateTransition merged, UpdateTransition - coding finished, currently is in review
Testing DataContract - 90% finished

Planned/Work in progress

Integrate Identity into Drive/Dapi
Finish the remainder of DataContract tests
Finish the remainder of Document Transition tests
Fix asynchronous I/O in the state repository - works right now in a synchronous manner

Core Team

Core Status - v18.x

v18.2 incident: 06/01/2023

Rotation member calculation mismatch
Downgrade to v.18.1
5% of Masternodes affected

v18.1 incident: 08/01/2023

DDoS attack.
Flooding of old but valid qcommits via P2P messages
Fix: If qcommit is older than one DKG cycle -> no BLS verification
Release of hotfix v18.1.1

v18.2.1 Release

Indexing both Testnet and Mainnet

Core Status - v19 Development

BLS Scheme upgrade done

Enforced BLS active scheme IETF
Versioned mined objects: qfcommit, protx family,
Versioned MNLISTDIFF P2P message

HPMN 4K in progress

New Masternode Type in review
Migration of internal evoDB, Returned in MNLISTDIFF and RPC in review
LLMQ_Platform in review
Masternode payment algorithm: 4x blocks for HPMNs in review
Governance: 4x vote weight for HPMNs in review
Include new Platform related fields in SML in progress

AssetLock in review

What’s Next

V19.0.0 Release Candidate
Potential V18.2.2 Release w/ minor fixes
Continued maintenance, backporting and feature development

What's left: Platform Development Update 10-January-2023

Dash Core Group Devs — Wed, 11 Jan 2023 19:02:23 +0000

Happy New Year, fellow Dashers!
This is our first Platform Update in 2023. Lots of things have been done. Let's see what's left before testing!

TenderDash

Task: Stabilization

What We Planned

Stabilizing

updated dependencies (actions/stale, setup-go,buf-setup-action, goreleaser-action) done
refactor: use logger for log printing done
refactor: blocksync.bpRequester should stop procedure if block was received done
feat!: verify next consensus params between nodes done
build: enable deadlock detection on -dev docker images done
build(github): use ubuntu 20.04 in github workflows done
fix: flaky TestReactor_Backfill test done
fix: improve flaky TestWALRoundsSkipper done

Investigate deadlocks

refactor(consensus): consensus state to fix deadlock in progress

What’s Left before Testing Phase

Stabilize same-block execution
~~Update BLS library~~
Adjust vote extensions for withdrawal transactions
~~Improve seed node operations~~
~~Drive integration~~

GroveDB

Tasks

Deterministic loaded bytes done
Optional Merk caching done
Better path query merge in progress
Merging of conditional path queries done
Intersection and disjointed union of query items done
Query Item Comparison done
Proof of mixed levels in progress

What’s Left before Testing Phase

Major

Sum Trees merge done

Minor

Average case processing costs done
Some code polishing done
Light proof feature done

Protocol

Task: Identity v2
Credit Withdrawals

Introduce withdrawals data contract in review
Implement the pooling process (type 0 only) for withdrawals in review
Update status of withdrawals after broadcasting in review
Add revision logic to IdentityCreditsWithdrawal transition in review
Indices from expired withdrawal transactions must be reused in review

Other Protocol Tasks

Identity v2 in progress
Total credit balance verification in progress
Calculate expected total credits amount in progress
Efficient Data Serialization in progress
Add support for core 18.2 done
Do not switch to validator quorum which will be removed soon done
ABCI context logger in review
Fix for non deterministic processing fees done

What’s Left before Testing Phase
Major

~~Fee Refunds~~
Accomplish and test withdrawals
State Sync

Minor

~~Logging of same block execution~~
~~Use block candidate context in State Repository~~
Deal with situation where fees are higher than Balance in progress
Verify fee processing costs
Asset Lock and Unlock transactions integration
BLS library rust binding
Masternode block reward distribution
Add Sum Tree feature for Pool trees in progress

Rust Port Update

What We Planned

Integrate Rust DPP into SDK, Drive, DAPI and test suite in progress
Successfully run platform test suite and test for Drive and Dapi planned
ConsensusError done
ProtocolError in progress

State Transitions

DataContract

~~Create~~
~~Update (Changed)~~

Identity

~~Asset lock proof~~
~~Create~~
TopUp (Integration is in progress)
Update (Integration is in progress)

Document

~~Batch~~

What's left: Platform Development Update 27-December-2022

Dash Core Group Devs — Wed, 28 Dec 2022 16:47:10 +0000

Hello Dashers!
This is our regular Platform wrap up. Let's see what's left for next year.

TenderDash
Goals: Same block execution
Tasks:

Stabilizing

fix(consensus): proposer-based timestamp broken during backport done
backport: update BLS library done
Seed connectivity: max-incoming-connection-time, incoming-connection-window done
fix(consensus): ensure process proposal done
fix: improve flaky TestWALRoundsSkipper _in review _

Investigate deadlocks

fix(statesync): deadlock in peer up done
refactor(consensus): consensus state to fix deadlock in progress

Other:

Release v0.10.0-dev.7
Release v0.11.0-dev.1

What’s Left before Testing Phase

Stabilize same-block execution
~~Update BLS library~~
Adjust vote extensions for withdrawal transactions
~~Improve seed node operations~~
~~Drive integration~~

GroveDB

Tasks:

Light proof feature done
Linting and CI fixes done
Advanced queries for optional non existent queries done
“Removed Cousin” (not removal - removed in the family sense) references done
Better delete up tree options done
Subquery Paths replace Subquery keys done
Better path query merge in progress

What’s Left before Testing Phase
Major

- Sum Trees merge done

Minor

~~- Average case processing costs done

Some code polishing done
Light proof feature done ~~ ---

Protocol

Goals: Credit Refunds

Credit Withdrawals

Introduce withdrawals data contract in review
Implement the pooling process (type 0 only) for withdrawals in review
Update status of withdrawals after broadcasting in review
Add revision logic to IdentityCreditsWithdrawal transition in review
Indices from expired withdrawal transactions must be reused in review

Other Protocol Tasks

Identity v2 in progress
Efficient Data Serialization in progress
Credit Refunds done
Add support for core 18.2 in progress
Log synchronize identities properly done
Remove big elements like txs from logs done

What’s Left
Major

~~Fee Refunds~~
Accomplish and test withdrawals
State Sync

Minor

~~Logging of same block execution~~
~~Use block candidate context in State Repository~~
Deal with situation where fees are higher than Balance
Verify fee processing costs
Asset Lock and Unlock transactions integration
BLS library rust binding
Masternode reward distribution
Add Sum Tree feature for Pool trees

Rust Port Update

Tasks:
Integrate Rust DPP into SDK, Drive, DAPI and test suite in progress
Porting Consensus Errors Framework: all errors ported done
Successfully run platform test suite and test for Drive and Dapi planned
Merged StateRespoitory code and bindings
Merged everything related to ConsensusErorrs and most of the consensus validation code

State Transitions

DataContract

~~Create~~
~~Update (Changed)~~

Identity

~~Asset lock proof~~
~~Create~~
TopUp (Integration in progress)
Update (Integration in progress)

Document

Batch

What's left: Platform Development Update 13-December-2022

Dash Core Group Devs — Thu, 15 Dec 2022 14:16:26 +0000

Hey fellow Dashers, this is a short wrap up of what happened with the Platform over the past few weeks.

TenderDash

Goals: Same block execution
Tasks:
Stabilizing

fix(consensus): proposer-based timestamp broken during backport in review
fix(consensus): ProcessProposal executed twice for a block done
chore(types)!: rename genesis.json quorum fields done
refactor(consensus): consensus state to fix deadlock on hold
fix(wal): improve WAL replay mechanism in review

Investigate deadlocks

fix(statesync): statesync stops the node when light block request fails done
fix(statesync): deadlock in peer up in progress
Other:
Release v0.8.0

What’s Left before Testing Phase

Stabilize same-block execution
Update BLS library
Adjust vote extensions for withdrawal transactions
Improve seed node operations

GroveDB

Tasks:

Sum Trees merge done
Costs for Sum Trees done
Delete up tree worst case processing costs done
Delete up tree average case processing costs done

Features:

Deletion up tree in batches done
Removal on Root Tree storage in review
Inserts done
Deletes done
Batches in review
Proofs in review
Update of Costs in review

What’s Left before Testing Phase

Light proof feature in progress

JS SDK

Goals: Reorgs and consensus

Tasks:

Updating grpc-web library done
Fine tuning consensus rules in progress
DashPay Web Extension
Migrating extension from Manifest V2 to Manifest V3 in progress (needed for publishing to Chrome Store)

What’s Next

Reorgs handling and consensus rules fine tuning
Secure keychain storage

Protocol

Goals: V23 release, same block execution

Credit Withdrawals

Introduce withdrawals data contract in review
Implement the pooling process (type 0 only) for withdrawals in review
Update status of withdrawals after broadcasting in review
Add revision logic to IdentityCreditsWithdrawal transition in review
Indices from expired withdrawal transactions must be reused in progress

Same Block Execution

Use proposed block execution context in State Repository done
Update finalize block handler signature done
Skip process proposal if prepare proposal is already called for this round done
Safe single database transaction block processing done
Comprehensive logging done
Pass latest chain lock on init chain done

Other Protocol Tasks

Identity v2 in progress
Efficient Data Serialization in progress
Integration of average case costs done
Integration of GroveDB 0.7 (supports Sum Trees) done
Release Platform v0.23 done
Use FeeResult to collect block fees done
Credit Refunds in progress

What’s Left

Major

Fee Refunds in progress
Accomplish and test withdrawals
State Sync

Minor

Deal with situation where fees are higher than Balance
Verify fee processing costs
Asset Lock and Unlock transactions integration
BLS library rust binding
Masternode reward distribution
Add Sum Tree feature for Pool trees

Rust Port Update

Integrate Rust DPP into SDK, Drive, DAPI and test suite in progress
Porting Consensus Errors Framework: 75/76 errors ported in progress
Successfully run platform test suite and test for Drive and Dapi planned

What’s The Matter With Infura And Other Services?

Dash Core Group Devs — Fri, 09 Dec 2022 22:59:11 +0000

Recently, we saw Infura and other ConsenSys-related services dominate the headlines. Long story short, ConsenSys announced that they would be collecting user IP addresses for compliance requirements. Naturally, this sparked a “centralization vs decentralization” debate in the crypto community.

In this article, we will see how Dash Platform mitigates the need for a centralized service provider like Infura, Kaleido, etc.

What exactly is Infura?

Infura is a web3 IaaS (Infrastructure as a Service) product that provides a set of tools for anyone to connect their applications to the Ethereum blockchain. Infura lowers developers' entry barriers by running nodes on their behalf. The idea is simple, right? Abstract away the complexities of the blockchain and empower developers to focus only on their product. Infura’s IPFS (Interplanetary File System) API provides distributed storage to applications of all sizes. Several popular services, like MetaMask and Truffle, use Infura.

So, what’s the problem here?

Ever since 2018, there has been an increased concern in the Ethereum community about Infura’s growing presence within the ecosystem. Afri Schoedon from Parity Technologies said that over-reliance on Infura will increase the centralization of the protocol.

“If we don’t stop relying on Infura, the vision of Ethereum failed. Or build a strong network of thin and light clients. There is no point in having d-apps connecting through MetaMask to a blockchain hosted by someone else.”

Plus, at the end of the day, Infura is maintained by a centralized company which makes its applications vulnerable to censorship and regulatory measures.

Infura and censorship

In March 2022, ConsenSys made their services unavailable “within the three contested regions of Ukraine.” This happened after the US, European Union, and others implemented strict sanctions on Russia following their invasion of Ukraine. Plus, due to an error, Infura accidentally blocked Venezuelan users from MetaMask, as well.

There are a few things to unpack here:

Due to government sanctions, people living in certain areas were blacklisted from crypto services.
A mistake from the service provider blocked crypto access for people from a completely unrelated region.

Now, with the latest announcement about IP address collection, there is a substantial risk that regulatory demands may get more overbearing, leading to more censorship.

These points raise a few important questions. Wasn’t crypto created to prevent these things from happening in the first place? Wasn’t the entire point of crypto to make a means of value transfer that didn’t depend on the whims or incompetence of an intermediary?

And, perhaps most notably, why are we still using web2 architecture to access web3 protocols?

Dash Platform fixes this

Dash Platform is a web3 tech stack that allows you to build decentralized applications on Dash’s network. Dash Platform uses two architectural components, Drive and DAPI.

DAPI - Decentralized API

DAPI is a decentralized API that allows developers to connect their apps with the Dash blockchain. Developers can connect to DAPI directly or use a client library. DAPI helps developers by:

Using Dash masternodes as API providers, so developers don’t have to run nodes.
Removing overall cost and maintenance overhead.

What makes DAPI interesting is that, unlike traditional APIs, there isn’t a single point of failure. Instead, clients may connect to different independently operated instances depending on resource availability in the Dash network. So, you are connecting to the protocol using a decentralized system instead of a centralized one. Although all the Dash masternodes provide the same services, they are run by independent operators rather than a centralized entity.

Drive - Decentralized Storage

Drive is Dash Platform's storage component, allowing for consensus-based verification and validation of user-created data. Once user data is validated, it is submitted to Drive via DAPI and stored in the masternode network.

But how do you verify whether the data you get from the API is correct? After all, a truly decentralized system shouldn’t require trust to operate. To mitigate this, Drive uses GroveDB – a purpose-built database that provides efficient proofs with query responses. These cryptographic proofs help prove data authenticity.

With DAPI and Drive, Dash Platform provides the ideal decentralized portal for developers to connect to web3 without stumbling over the familiar roadblocks of web2 design.

Originally posted on Dash Blog.

Creating Identity On Dash Platform

Dash Core Group Devs — Fri, 02 Dec 2022 16:06:27 +0000

Per Article 8 of the UN’s Convention on the Rights of the Child, personal identity is a fundamental right. At its core, identity is defined as “the fact of being who or what a person or thing is.” Today, we will see how decentralized identity, as provided by Dash Platform, helps you overcome the weaknesses of traditional official identity.

The problems with traditional identity

Your official identity plays a crucial role in establishing who you are and is required for you to function in any capacity as a citizen. Your government identity consists of multiple data points like – Name, Date of Birth, Nationality, Identifier, etc. You must use your government identity to obtain essential services such as bank accounts, healthcare, social benefits, pension, etc.

However, there are three major issues with centralized identity:

Issuance
Mismanagement
Theft

Issuance

As per the World Bank, over a billion people worldwide don’t have any claims to their identity. After all, these identities are issued by a centralized body. Some people simply don’t have the means to get their identities issued.

Mismanagement

Of course, this is the perennial problem with centralized storage. When you trust a third party to store your private data, you trust them to manage it properly. More often than not, this trust could be misplaced.

Theft

Finally, when you use a centralized server to store all your data, you create an easy target for hackers. The potential ramifications and identity theft could have devastating consequences. For example, Equifax, one of the top credit-reporting companies, was breached in 2017. It is estimated that half the US population lost their Social Security Numbers during the breach.

The consequences here are pretty devastating. Identity theft and fraud have been on the rise. As per the Federal Trade Commission (FTC), the amount of identity theft and related fraud rose significantly in the US in 2021. Here are some stats to ponder:

Fraud complaints have gone up by 19% to more than 5.8 million.
Total financial losses crossed $6 billion.
The number of consumer complaints rose by 3.3% to just over 1.43 million.
So, what’s the solution here? Maybe, practicing self-sovereign identity is the way forward?

What is Self-Sovereign Identity?

Self-sovereignty, in general, means you have the moral right to fully control your body and life. Self-sovereign identity is a digital identity where individuals have complete control over their data. Previously, in a web2 ecosystem, this was nearly impossible.

However, things have changed with the advent of web3 and the incorporation of blockchain technology. As a result, it is now possible to deploy decentralized identity.

Before going any further, let’s see how the blockchain helps mitigate the two biggest problems with digital identity:

Easily replicable
Not tamper resistant

1 Fixing replicability

One of the main things Satoshi Nakamoto solved with Bitcoin and blockchain technology is “double spending.” Double spending was a problem in previous iterations of digital currencies, wherein it was possible to spend the same coin in more than one transaction simultaneously.

So, how do cryptocurrencies like DASH fix this issue?

Each transaction is verified by specialized nodes like the miners and masternodes to validate these transactions in exchange for a fee.
Since every single coin is accounted for in the ledger, it is impossible to double spend it.
Now, expand the same concept to identity. You shouldn’t be able to “double spend” your identity either. After all, no two people can have the same identity, correct?

With blockchain-based identities, we can be confident that our identity can’t be stolen/duplicated because the associated data is cryptographically protected and owned by us, not by a third party.

2 Tamper resistance

The second problem is the tamperability of digital identity. With digital identity, we have the following risks:

A malicious party could bribe the intermediary storing your data.
The system could be hacked.

Blockchain mitigates these issues by being an immutable source of truth. Cryptographic hash functions secure all the data stored within the blocks. As such, it is impossible to tamper with data once it has entered the chain. The only exception is if someone controls>51% of the network hashrate and rewrites the chain’s history by forking the protocol. Dash specifically mitigates this attack scenario, as well. With ChainLocks, even controlling 51% of the network hashrate can’t rewrite the chain history.

Dash Platform and Digital Identity

Dash Platform is a web3 technology stack that allows you to build dApps on the Dash network. Decentralized identity is an integral component of the Dash Platform. Users can use their identities to interact with and identify each other instead of depending on public key hashes.

So, how does one create an identity on Dash Platform? Let’s go through the steps:

Users must pay the network to store public keys on the platform chain.
If a new user doesn’t have Dash funds, they can still create one, provided they get an invitation from existing Platform users.
Once an identity is created, it will have a corresponding balance of credits. These credits are used to pay for various operations on the Dash Platform. To get these credits, you must lock up a certain amount of DASH on the base layer-1 Dash blockchain.
You may top up your credit balance anytime you want via top-up. Learn more about the Dash Platform and identity management.

Originally posted on Dash Blog

What's left: Platform Development Update 29-November-2022

Dash Core Group Devs — Thu, 01 Dec 2022 18:26:47 +0000

Here's another portion of updates on what's left to do before final Platform testing.

TenderDash

Goals: Same block execution

Tasks:

Integration of same-block execution with Drive in progress

Released v0.10.0-dev.5 done

Stabilizing

fix(wal): improve WAL replay mechanism in progress
fix(e2e): catchup round number is not correct done
refactor: make all genesis-doc fields (except chain_id) optional done
feat(abci): put full block in RequestFinalizeBlock done

Investigate deadlocks

refactor(consensus): consensus state to fix deadlock in progress
Test: add deadlock detection with go-deadlock done
fix(statesync): statesync stops the node when light block request fails in progress
fix(statesync): deadlock in peer up in progress
fix(consensus): commits received during state sync are lost #1867 done

What’s Left before Testing Phase

Stabilize same-block execution
Update BLS library
Adjust vote extensions for withdrawal transactions
Drive integration

GroveDB

Tasks:

Sum Trees merge in review
Fix batch not properly propagating child root hash done
Average case processing costs done

What’s Left before Testing Phase

Major

~~Sum Trees merge~~ in review

Minor

~~Average case processing costs~~ done
Some code polishing
Light proof feature

SDK

Goals:
Rework chain synchronization process (SPV)

Tasks:

Reworking chain synchronization process done
Manual QA done
Stabilizing Platform related workflows (Platform Test Suite) done
Updating grpc-web library in progress

DashPay Web Extension

Wallet balance in USD done
Create/Reset/Backup Wallet flows done
Migrating extension from Manifest V2 to Manifest V3 in progress (needed for publishing to Chrome Store)

What's next

Reorgs handling and consensus rules fine tuning
Secure keychain storage

Protocol

Goals: Same block execution, fees and withdrawals

Tasks:

Fee finalization (v23) in JS Drive and DPP done

Withdrawals:

Introduce withdrawals data contract in review
Implement the pooling process (type 0 only) for withdrawals in review
Update status of withdrawals after broadcasting in review
Add revision logic to IdentityCreditsWithdrawal transition in review

Same Block Execution:

Use multiple database transactions for block processing will not be implemented
Provide signature and block time in query handlers done
Fix for Reconnect logic in DAPI done
contactRequest documents allow invalid toUserIds done
Identity v2 in progress
Efficient Data Serialization in progress
Integration of average case costs in progress
Integration RS-Platform into Platform done

What's left:

Major

Fee Refunds
Accomplish and test withdrawals

Minor

Logging of same block execution
Use block candidate context in State Repository
Deal with situation where fees are higher than Balance
Verify fee processing costs
Asset Lock and Unlock transactions integration
BLS library rust binding
Masternode reward distribution

Rust Port Update

Tasks:

Integrate Rust DPP into SDK, Drive, DAPI and test suite in progress
Initial glue figured out, DPP tests ported 159(68+91)/1174 in progress
Porting Consensus Errors Framework: 30/76 errors ported in progress
Successfully run platform test suite and test for Drive and Dapi planned
Integrating working BLS scheme into Rust DPP done

Decentralized Storage and Dash Platform

Dash Core Group Devs — Fri, 25 Nov 2022 19:57:53 +0000

There is a lot of hype surrounding web3 and for a good reason. Who wouldn’t want a smarter internet, immersive virtual worlds, and sophisticated creator economies? While the scope and potential of web3 are immense, it is just as important to talk about the building blocks of this ecosystem. One of the core things needed for the transition from web2 to web3 is decentralized storage.

In this article, let’s get into the details and understand the advantages of decentralized storage. Finally, we will see how Dash Platform ensures data integrity with decentralized cloud storage.

What is decentralized storage?

In a decentralized storage system, instead of storing all the data in a centralized server, you store chunks of it as blocks over multiple nodes in a network. Furthermore, the data is secured by cryptography. So, why go through the trouble of storing data in such an intricate manner? Surely, it will be much easier to use simple centralized cloud storage, right?

Well, sure, the concept may be more straightforward to understand. However, centralized storage systems have several significant shortcomings.

The problems with centralized storage

#1 Susceptible to censorship

The centralized storage system we are all familiar with is susceptible to censorship. Let’s take an example we are all familiar with. Wikipedia is a wonderful information resource, but it could be easily censored due to its centralized nature. For example, governments could suppress crucial information from being published on the country’s page.

#2 Single point of vulnerability

When using a centralized storage service, you are essentially handing over control of your data to a third party. The problem with keeping all your data in one place is that you are essentially painting a very attractive target for potential hackers, as we saw during the infamous August 31, 2014, Celebgate hack. During the hack, someone hacked iCloud’s servers and leaked extremely private pictures of various celebrities.

#3 Unethical Data Mismanagement

Finally, let’s talk about data mismanagement. So far, we have seen multiple incidents of third-party service providers being irresponsible and unethical in managing user data. Facebook (now Meta) got in deep trouble when its systems allowed Cambridge Analytica to access the personal data of 87 million Facebook users. The data collected was later used to influence several high-profile elections.

The advantages of decentralized data storage

Security and privacy
Decentralized storage organically mitigates this since it stores your data across multiple independently-operated nodes. You don’t need to trust a singular provider because there aren’t any singular providers in this system. The data stored in a decentralized network is also encrypted via cryptographic hash algorithms, adding another layer of protection.

Minimizing file loss
Since your data is stored across independently-operated nodes, the chances of file loss is reduced by a considerable amount. So, you are not dependent on a single service provider being constantly online or operational to have access to your data.

Liveness
As we have seen time and again, websites tend to get very slow during high traffic. In some cases, the websites might crash altogether due to these traffic spikes. The reason is simple. There is only so much load that a single server can handle. A decentralized system that uses P2P technology isn’t dependent on one server. Any network load is shared among a distributed network of nodes, leading to fewer bottlenecks and a system that’s continually live.

Dash Platform’s web3 storage solution

Dash Platform is a web3 technology stack that allows you to build dApps on the Dash network. Dash Platform consists of two components – Drive and DAPI.

Drive is Dash Platform’s storage component, allowing for consensus-based verification and validation of user-created data. It is a layer-2 component that stores application data in Dash’s masternodes.

How does it work? Let’s take a look.

The developer creating the application makes a data contract. This contract describes the data structures that will define the application.
Data created by users of the application is validated and verified against this contract.
When successfully validated, the application data is submitted to the Drive and stored on the masternode network.
All data stored on the Dash network is protected by Dash’s consensus algorithm, ensuring data integrity and availability.

Conclusion

Decentralized storage is one of the building blocks of the web3 revolution. Dash Platform offers a simple and effective way to store data in a decentralized network. Since application data is stored across many nodes on the Dash network, it is safe and always available for customers, business partners, and investors.

Learn more about the Dash Platform, and discover new opportunities for your applications!

Originally posted on Dash blog