Forem: Rafik Naccache

Building turbo-ledger: A Scalable Ledger with Go and Redis

Rafik Naccache — Sat, 09 Jul 2022 09:58:37 +0000

turbo-ledger: Why and How

What's a ledger and why should we bother?

In a nutshell, a ledger is a register we use to keep track of transactions in which people trade goods and services for assets that represent equivalent value.

In the physical world, an asset's value stems from its scarcity as a physical object(think banknotes, wheat, gold, collector guitars,...)

But online, assets are digital - and digital isn't scarce. As such, any amount's worth will only make sense if correctly traced back - through a set of authentic transactions - to a legitimate origin. That origin will create that scarcity that made physical world assets valuable(think bitcoins, NFTs, digital tokens,...). So in digital, ledgers ARE the assets.

Hence being able to keep secure and scalable ledgers is of crucial importance in today's eCommerce-fueled world, where nearly every service is becoming digital. Tomorrow's money (and a significant part of today's) will only flow through digital bookkeeping!

Keeping in mind that, let's explore a minimal ledger implementation that will provide bookkeeping services. We'll call it turbo-ledger, and will implement it in Go and will be fueled by Redis. you can find it here

turbo-ledger will revolve around two main entities, wallets and transactions. First, let's study wallets.

Wallets: where the assets lie

A Wallet, as the name suggests, is a bucket containing some digital funds. Just like we explained, what we care about most in a digital wallet are transactions as they constitute the very online money, but for the sake of convenience, we can keep some sort of balance field up to date at the wallet's level.
Moreover, some information about the wallet identity needs to be present. For this post, we'll just use a plain ID and owner fields.
Finally, we can qualify wallets with some tags: VIP, Personal,...

If we go on and model this, the following would be a JSON object representing a Wallet:

{
  "wallet_id":"my_wallet",
  "transactions":[],
  "balance":10,
  "owner":"bob",
  "tags":["vip","personal"]
}

In a full-fledged implementation, we'll need to somehow attach a wallet to a cryptographic signer entity and maybe enforce real-world user identity-checking like verifying phone numbers or emails, but we'll settle for our toy-ish implementation in this post.

And that will be it for our wallet object. Let's look closer at transactions.

Transactions: Assets movement tracking

A transaction will describe a certain amount of assets being transferred between a source and a destination wallets. For every transaction, we'll need to keep a record of the date when it happened. We'll also attach textual descriptions to transactions so they're augmented with some explanation. Finally, transactions will have an ID field.

Here is a JSON object representation of our toy transaction:

{
  "transaction_id": "some_id",
  "source_wallet": "src_wallet",
  "destination_wallet": "dst_wallet",
  "amount": 5,
  "transaction_description":"description 1",
  "date":"2022-06-07T19:02:01.0Z"
}

In a real-world setup, transactions need to be signed by an authorized source wallet owner and must be logged as they flow through the system with mention of their status, like the reason why they might have been discarded, etc. We might also want to make sure one cannot tamper with the transactions history(a la blockchain). But we'll stay with this minimal implementation we exposed for this post.

Moving Assets Around

The workflow for transferring an amount of amount_assets assets from source wallet src_wallet to destination wallet dst_wallet would be as follows:

transaction submitted
turbo-ledger verifies src_wallet exists
turbo-ledger verifies src_wallet has enough balance to transfer to dst_wallet (i.e balance is equal or greater than amount_assets)
turbo-ledger verifies dst_wallet exists
turbo-ledger writes transaction (with amount_assets negated - to simplify balance recomputing if needed in the future) in src_wallet
system deduces amount_assets from src_wallet
turbo-ledger does the exact inverted thing to dst_wallet:
writes transaction (with positive amount_assets ) to dst_wallet
increases dst_wallet's balance by amount_assets

for example, after an assets transfer operation of 5, here's what src_wallet will look like (assuming a starting balance of 10) :

{
  "wallet_id":"src_wallet",
  "owner":"alice",
  "transactions":[
    {"transaction_id":"a7749032-7be9-4cc8-a1b0-a14701bc1208",
      "source_wallet":"src_wallet",
      "destination_wallet":"dst_wallet",
      "amount":-5,
      "transaction_description":"my cool transaction",
      "date":"2022-06-07T19:02:01Z"}
  ],
  "balance":5,
  "tags":["vip","personal"]
}

for dst_wallet, assuming an initial balance of 0, we'd get:

{
  "wallet_id":"dst_wallet",
  "owner":"bob",
  "transactions":[
    {"transaction_id":"a7749032-7be9-4cc8-a1b0-a14701bc1208",
      "source_wallet":"src_wallet",
      "destination_wallet":"dst_wallet",
      "amount":5,
      "transaction_description":"my cool transaction",
      "date":"2022-06-07T19:02:01Z"}
  ],
  "balance":5,
  "tags":["normal","professional"]
}

Now we've seen how turbo-ledger foundational entities -wallets, and transactions - are modeled and how asset transfers function, you'd agree we need a solid JSON-aware datastore with fast I/O and search capabilities.
With that in mind, let's dive into the implementation. (Spoiler alert: we'll use Redis!)

A Naive turbo-ledger Implementation with Redis

Why Redis?

(DISCLAIMER: I got to discover RedisJson and RediSearch when I applied to the "Write for Redis" promotional program, which sponsors this post - and I am glad I did because I discovered those two great modules!)

To be honest, I would not have considered Redis to implement turbo-ledger: Though I highly value Redis as a high throughput key-value data store, I was not aware of its JSON and Search capabilities through the RedisJson and RediSearch modules. But when I learned about them I fell in love!
With RedisJson and RediSearch, you have a fully JSON-fueled and indexed datastore, all accessed through the blazing-fast key-value paradigm in which Redis shines. No more needed to be said for me, I jumped in.

Prototyping quickly with RedisCloud and RedisInsights Desktop

RedisCloud

I was planning to work on turbo-ledger as a side project and so was not willing to allocate much time and effort to it.
As such, I was not going to bother about Installing Redis or having it set up as a Docker container either on my already overloaded Mac.

This is how I chose Redis Cloud, A Managed Redis service made by the Redis Team. I could easily set up a database for free with a trial account (no credit cards needed) and verified the needed modules (RedisJson and RediSearch) were present in that database. I was all set in a couple of minutes.

To learn more about Redis Cloud, you can watch it here:

RedisInsights Desktop

As explained, turbo-ledger was a side project for me, one that was on a tight time and effort budget. But it presented nevertheless an opportunity to learn about Redis modules that were new to me, RedisJson and RediSearch. So I needed some tool to help me explore commands and query JSON keys while I was working, and Redis CLI was not going to assist me with that.

This is how I discovered RedisInsight Desktop GUI.

With a few mouse clicks, I connected to my Redis Cloud database and started to play.

I particularly loved the workbench, which allowed me to try commands, before going to the Go IDE:

I also could explore JSON keys using the browser interface:

I could even change JSON Keys in place which is very helpful when prototyping turbo-ledger:

With all these tools in hand, let's go on modeling and coding turbo-ledger in Go and Redis!

Representing our wallets' Objects

Wallets as JSON Documents

A Wallet will be represented as a JSON key at Redis' level. We'll observe the following key scheme for wallets:

wallet:wallet_id

In Redis, for instance, to create a wallet, you'd set a key with the JSON, exposed above, as a value, using the JSON.SET command:

JSON.SET wallet:wallet_id $ "{....}"

Note the $ sign, which denotes the JSON path of the element you want to change inside the JSON Document. In the above, that just means that we want the Input JSON document to sit at the root of the key, which simply means setting its value to it.

But we can change a nested part of the document just as easily. For example, assume we want to set some wallet's balance to 10. We'd have to do:

JSON.SET wallet:wallet_id  $.balance 10

Wallets in Go

In turbo-ledger's Go implementation, to create a Wallet, we have the following function:

func createWallet(ctx context.Context, rdb *redis.Client, walletId string, startingBalance float32, newWallet Wallet) error {
    if walletId == "" {
        return errors.New("could not create Wallet with empty Wallet Id")
    }
    strNewWallet, errStrNewWallet := json.Marshal(newWallet)
    if errStrNewWallet != nil {
        log.Printf("could not marshal Wallet into json")
        return errStrNewWallet
    }
    errCreateVaultWallet := rdb.Do(ctx, "JSON.SET",
        fmt.Sprintf("wallet:%s", walletId),
        "$",
        strNewWallet,
    ).Err()

    if errCreateVaultWallet != nil {
        log.Printf("could not create Wallet %s with command", errCreateVaultWallet)
        return errCreateVaultWallet
    }

    log.Printf("successfully created Wallet %s with starting balance %f",
        walletId,
        startingBalance)
    return nil
}

You should have noticed how we're using plain Redis commands through go-redis - choosing not to use any third-party RediSearch or RedisJson specialized clients. This way, we stay as close as possible to Redis commands, not using any abstractions or ORMs.

Indexing and searching through wallets

Indexing data through RediSearch is a novelty for me.
Before, I only knew to access values knowing keys they're associated with - there was no way to query data on particular conditions on values. This had the Redis developer design key schemes that would hold parts of data, which was limiting and cumbersome to say the least. To get a sense of how this works, see the checkWalletExists function implementation in the turbo-ledger go implementation:

func checkWalletExists(ctx context.Context, rdb *redis.Client, walletId string) error {
    if walletId == "" {
        return errors.New("could not check Wallet with empty Wallet Id")
    }


    searchWalletCmd := rdb.Do(ctx, "JSON.GET",
        "wallet:"+walletId, // get the key with ID information
    )
...

Now, with RediSearch, it is possible to create search indexes on particular elements of the value. In turbo-ledger, we created an Index on wallet owners:

FT.CREATE idx:wallet:owner ON JSON SCHEMA $.owner AS owner TEXT

This way, we can search wallets that are owned by bob like so:

FT.SEARCH idx:wallet:owner "@owner:(bob)"

Also, we created an index to search through the transactions tags array. Note the use of the JSON Path $.tags.*:

FT.CREATE idx:wallet:tags ON JSON SCHEMA $.tags.* AS tags TAG

Then you can search through wallets using tags through:

FT.SEARCH idx:wallet:tags "@tags:{rich}"

In turbo-ledger's Go implementation, you'll see Redis plain commands creating those indices in the Genesis function, and using search in the SearchWallets function

Representing Our transaction Objects

Transactions inside Arrays in Wallets

In redis, you use the JSON.ARRAPPEND command to add an element to an array situation under a particular JSON path. In our case, here is the command to add a transaction to a wallet:

JSON.ARRAPPEND wallet:wallet_id $.transactions {...}

You want transactional transactions

Writing a transaction to the ledger involves verifying the source wallet's balance. If there's enough assets, we decrease that source wallet balance by the transaction's amount then add a transaction object to this wallet's transactions array. We will then mirror this process to the destination wallet (without balance verification).

As you see, the overall transaction writing process entails multiple steps that need to happen all - or not to happen at all - for the turbo-ledger state to stay consistent. Does this ring a bell?

Yes. These operations need to happen in a transactional way. when they happen, we need:
0- verify the source balance
1- the balance in source is updated
1'- transaction is appended to the source's array
2- the balance in destination is updated
2'- transaction is appended to the destination's array
Be done as a whole or:
3- ... Do nothing

For 1-1',2-2' and 3 we'll use transactions in Redis
In Redis, you start a transaction using the MULTI keyword, then you start to queue commands, which you'll execute using EXEC or discard using DISCARD.

Now while you're queuing your transaction commands, someone or something else might have changed the source or destination's wallet balance - your increment or decrement operations might not be working on a clean state no more.

This is where Redis' feature: optimistic locking using check-and-set might come handy.
if you issue a WATCH on certain keys before beginning a Redis transaction with MULTI, you tell Redis to monitor those keys for changes, and if this happens this will make your transaction execution with EXEC fail - so you might choose to retry it.

This is the approach we follow in the turbo-ledger's Go implementation.
We watch on the source and destination wallets involved in any transaction.
We then start a MULTI session, after which we QUEUE series of JSON.ARRAPPEND and JSON.NUMINCRBY as we go through the different verification steps:

We WATCH the source and destination wallets
We start a transactional session with MULTI
We verify the existence of the source balance. If not there, we DISCARD
We verify source balance. If not enough, we DISCARD
We Queue source balance decrease using JSON.NUMINCRBY (using a negative amount)
We Queue adding the transaction object to the source wallet transactions array using JSON.ARRAPPEND
We verify the existence of the destination balance. If not there, we DISCARD
We Queue destination balance increase using JSON.NUMINCRBY
We Queue adding the transaction object to the destination wallet transactions array using JSON.ARRAPPEND
And we EXEC - catching errors that would have been emitted by the WATCH Above and retry for max_tries times

Let's have a look how this is implemented in turbo-ledgers's Go Implementation

Transactions In Go

The above process is bootstrapped in the PostTransaction function, exposed as a REST API:

func PostTransaction(rdb *redis.Client, mutex *redsync.Mutex) func(*gin.Context) {
    return func(c *gin.Context) {

        ctx := context.Background()
        var receivedTransaction Transaction
        if errBind := c.BindJSON(&receivedTransaction); errBind != nil {
            log.Printf("could not process received Transaction, %s", errBind)
            return
        }
        log.Printf("received Transaction:%+v", receivedTransaction)

        if errProcessTransaction := processTransaction(ctx, rdb, receivedTransaction, 3, mutex); errProcessTransaction != nil {
            c.IndentedJSON(http.StatusInternalServerError, receivedTransaction)
            return
        }

        c.IndentedJSON(http.StatusCreated, receivedTransaction)
    }
}

To keep this post short and not overload you with code, I suggest you follow along reading through the processTransaction and attemptTransaction functions in the turbo-ledger's Go implementation.

That was quite a ride! But as you may have suspected, something's missing, as we called this part "naive" implementation. It's about race conditions, every distributed system's designer nightmare. Read-on.

Making turbo-ledger turbo

So now turbo-ledger naively but happily serves its clients. But do you remember that "transactional transactions" scheme ?
0- verify the source balance
1- the balance in source is updated
1'- transaction is appended to the source's array
2- the balance in destination is updated
2'- transaction is appended to the destination's array
Be done as a whole or:
3- ... Do nothing

We are using a WATCH to halt a transaction if any of the involved wallets' state changed during operation 1-1' and 2-2'.

But... What if, after we happily (and naively) verified the source wallet's balance in 0, found enough assets there, and just were about to proceed with steps starting from 1-1', someone else spends some assets from that same source wallet?

WATCH can't help us here, as it begins monitoring from 1-1'!
Indeed, concurrent processes are racing over the source wallet balance in this case, because our Redis' Optimistic Locking only protects the system's state AFTER the source balance check.

The situation that would occur then is called double-spend, as if allowed, we erroneously would have spent assets that should not exist, as they've been spent by someone else (or a same fraudulent person) - it's like you've spend twice the same money!
In this case, locking can be the solution.

Preventing the Double Spend Problem with local locks

One approach would be to lock the code section posting transactions. Entering the function, you'd ask to acquire a lock, and you'd release it when you're out. If someone else acquired the lock, you'd have to wait for its release up until a certain time. This will look like the following:

var m sync.Mutex
...
func ProcessTransaction(ctx context.Context, rdb *redis.Client, transaction Transaction, maxAttempts int) error {
    if transaction.Amount <= 0 {
        return errTransactionWrongAmount
    }

    //Acquire single thread lock
    m.Lock()

    //Make sure to unlock on exiting transaction
    defer m.Unlock()
        //....

While this solution would definitely do the trick, it will imply a scalability problem: a lock is only valid in the execution context of a single instance of the ledger. If multiple turbo-ledger instances are accessing the Redis Database, having Go manage local locks in each turbo-ledger process won't protect users calling through one instance getting data compromised by users accessing through another. As such, you can only have one single turbo-ledger instance running at anytime, for this locking mechanism to stay valid. Hence, no horizontal scalability would be possible here.

To address this, we can use a distributed-lock based version. How does it work?

Addressing double-spend with a Redis-based distributed lock: RedLock

Redlock follows the spirit of the previous solution, but as it relies on a central Redis database to store the locks, locking becomes instance-independent, which will allow us to have as many turbo-ledger sessions as we please. This behavior is exposed in the current commit at the turbo-ledger's Go implementation, and is coded like so:

// Preparing redsync instance
   pool := goredis.NewPool(rdb)
   rs := redsync.New(pool)
   mutexname := "global-wallets-mutex"
   mutex := rs.NewMutex(mutexname)
   ...
 func processTransaction(ctx context.Context, rdb *redis.Client, transaction Transaction, maxAttempts int, mutex *redsync.Mutex) error {
    if transaction.Amount <= 0 {
        return errTransactionWrongAmount
    }

    log.Printf("acquiring global lock...")
    if err := mutex.Lock(); err != nil {
        log.Printf("could not acquire global mutex, aborting")
        return err
    }

    defer func() {
        log.Printf("releasing global lock...")
        if ok, err := mutex.Unlock(); !ok || err != nil {
            log.Printf("could not release global mutex, aborting")
        }
    }()

/ ...

Now we can argue that we don't need WATCH anymore. But I left it in the current implementation, as I am still investigating a means to have Redis manage everything without having locking at Go level, distributed or not. Maybe through LUA scripting? Let's see what the future holds!

Conclusion

Congratulations for making it thus far!
In this post, we've seen how to implement a scalable ledger based on Redis thanks to the powerful RedisJson and RediSearch modules, how the development experience and exploration on Redis can be made smooth using Redis Cloud and RedisInsight Desktop GUI. We also discussed Redis Transactions and saw how we needed also to protect against race conditions over source wallets balances in transaction starts, using Redlock.
One last thing needs to be taken into consideration: as you're storing non-volatile, sensitive data, be sure to to understand the different Redis Persistence models and applications, so you can choose the most secure data persistence model you see fit!

To learn more about Redis, RedisInsight Desktop, and RedisCloud head over to the following links:
Redis Cloud
Redis Developer Hub - tools, guides, and tutorials about Redis
RedisInsight Desktop GUI

This post is in collaboration with Redis

Plan like a Pro with Automatic Scheduling in Taskjuggler

Rafik Naccache — Fri, 03 Jul 2020 10:09:37 +0000

The Need for Automatic Project Scheduling?

Every one of us has done some project planning in a way or another. Some people would have used fancy GANTT editing tools for that; others would just go ahead and dump their brains onto a backlog. But very few are those who follow an automatic approach to "schedule" their projects.

More often than not, people would go manually about making decisions about the gameplay to adopt facing their daunting projects. They try to envision the best possible plan while mentally trying to sort what we call "precedence constraints," i.e., one can't work on Task A before Task B and C have been achieved. They do so all while finding feasible and commercially viable resource allocation schemes. Also, they need to contort to time-bound constraints, like deadlines and seasonal phenomena.

All that makes a perfect case for an automatic scheduling tool, and Taskjuggler might be the most powerful - if not the only - mature alternative you have at hand. But this is not a shiny Kanban-y drag-and-drop thingy: this is a full-fledged enterprise-grade automatic project planning and estimating solution, so much that it takes some learning curve to get fluent in it, but believe me, the effort is worth it.

Taskjuggler: Thinking in Work Breakdown Structure

Without drowning too deep in detail, Taskjuggler lets you focus on your project structure.
Rather than giving you GANTT drawing tools and letting you figure the plan out yourself, Tasjkjuggler asks that you give it your Work Breakdown Structure, and it does the scheduling for you. Simply put, A Work Breakdown Structure is a graph depicting the tasks you need to achieve, along with the relationships between them and the resources necessary to make them happen.

Taskjuggler is also able to follow on the realization of the project and will adjust the plan according to the actual project roll-out. Besides, it offers you the possibility to simulate multiple scenarios.

Taskjuggler is also finance aware, and sports an accounting module that can track the costs inferred by the allocated resources against revenues that you'd record as you're working on your project. I have used this accounting module so many times as my secret weapon to come up with detailed financial quotes and win the commercial race!

Taskjuggler in action

First of all, you'll need to install Taskjuggler following the instructions on its website.

Taskjuggler comes as a line command, tj3, that compiles your project descriptions into the project reports you specify. Let's run through a simple example.

Suppose you want to specify a Software Development project. We first need the data engineer to design the database, then have the web designers design the application, then have the Devops deploy it.

Let's begin by creating a file for our project. We'll name it project.tjp.

We'll begin by specifying our project name, plus some general information about it, like its start date, and what would be the deadline for it(1 month from the start, hence the +1m":

project softdev "Software Development"  2020-08-16 +1m {
 timezone "Europe/Paris"
 currency "USD"
}

Let's then declare the financial accounts we'll be using to track the finances of our project:

account cost "cost"
account rev "payments"
balance cost rev

We now declare the resources we need for our tasks:

resource data_engineer "Data Engineer" {
    rate 200
}

resource web_engineer "Web Engineer" {
    rate 200
}

resource devops_engineer "Devops Engineer" {
    rate 200
}

rate will be used to give a financial estimate of our project. We are using the simplest form, but you can organize them into teams, flag them, assign managers...

Let's now get into the actual business. Next, we'll describe our Work Breakdown Structure, that is, tasks, constraints, and resources. We'll organize our tasks into to categories. Work, where we'll describe our actual tasks to be done, and Milestones, to track important steps and deliveries of our project. For this, we'll use the possibility to nest Tasks in taskjuggler. First, let's see the Work part.

task work "Work"{
  task data "Database Design" {
    chargeset cost
    allocate data_engineer
    effort 3d # This task takes 10 days effort
  }

  task web "Web App Design" {
      chargeset cost
      allocate web_engineer
      effort 3d # This task takes 10 days effort
      depends !data  # This task needs task data to be complete
  }

  task deploy "Deployment" {
      chargeset cost
      allocate devops_engineer
      effort 3d # This task takes 10 days effort
      depends !web # This task needs task web to be complete
  }
}

The chargeset clause is used to add the costs inferred by a resource working on a task to a particular account, which is cost here. We'll use this account to get a quote for the whole work needed for this project.

The depends construct tells Taskjuggler that a task can't be worked on BEFORE the referred task has been achieved. This is what we call "Precedence Constraint" in Work Breakdown Speak.

The depends clause uses the exclamation mark '!' to refer to tasks that are one level higher in the hierarchy. For example, deploy depends on !web means that it depends on the task web under a direct parent(one level up), yielding its direct sibling web(their common parent task being work)

Now for the Milestones section:

task milestones "Milestones" {
  task db_milestone "Database Finish" {
      # A milestone has effort 0, it's represented as a little black square
      depends !!work.data
  }

  task web_milestone "Web Finish" {
      # A milestone has effort 0, it's represented as a little black square
      depends !!work.web
  }

  task deploy_milestone "Deploy Finish" {
      # A milestone has effort 0, it's represented as a little black square
      depends !!work.deploy
  }

  task project_end_milestone "Project End" {
    depends !!work.deploy
  }
}

Note how now depends uses a double exclamation mark. Look at task deploy_milestone for instance. It is meant to track the completion of the deploy task under the work container task. At this level, we are under milestones, so we need one ! to get to the level of the direct parent milestones, then another more to get the grand-parent level, under which we get to the aunt task work and refer deploy by using a familiar dotted syntax, hence the !!work.deploy token.

At this point, our project description is finished, but if you run it through the taskjuggler compiler, you won't get any output:

tj3 project.tjp
#... Truncated Output
Warning: This project has no reports defined. No output data will be generated.
Warning: None of the reports has a 'formats' attribute. No output data will be generated.

Indeed, you need to specify reports to get the Taskjuggler engine to compile the results under the format you want. And this is presumably the most complicated part. Here we go.

We'll generate a static website. The entry point to the reports is the Overview report page, with a link to detailed resource allocation by task under the Development subreport, and a link to a resource Utilization under the Resource Graph subreport.

This post won't dive into the Taskjuggler report syntax, but if you're interested, you can check it here. This being said, you can use the following as a template to start your pro scheduling journey - this is actually the way I followed (without necessarily being fluent in the syntax).

The main report Overview is specified like so:

navigator navbar {
  hidereport @none
}

macro TaskTip [
  tooltip istask() -8<-
    '''Start: ''' <-query attribute='start'->
    '''End: ''' <-query attribute='end'->
    ----
    '''Resources:'''

    <-query attribute='resources'->
    ----
    '''Precursors: '''

    <-query attribute='precursors'->
    ----
    '''Followers: '''

    <-query attribute='followers'->
    ->8-
]

textreport frame "" {
  header -8<-
    == Toy Software Development Project ==
    <[navigator id="navbar"]>
  ->8-
  footer "----"
  textreport index "Overview" {
    formats html
    center '<[report id="overview"]>'
  }



  textreport development "Development" {
    formats html
    center '<[report id="development"]>'
  }

   textreport "ResourceGraph" {
    formats html
    title "Resource Graph"
    center '<[report id="resourceGraph"]>'
  }
}

taskreport overview "" {
  header -8<-
    === Project Overview ===

    The project is structured into 3 phases.


    # <-reportlink id='frame.development'->


    === Original Project Plan ===
  ->8-
  columns bsi { title 'WBS' },
          name, start, end, effort, cost,
          revenue, chart { ${TaskTip} }
  # For this report we like to have the abbreviated weekday in front
  # of the date. %a is the tag for this.
  timeformat "%a %Y-%m-%d"
  loadunit days
  hideresource @all
  balance cost rev
  caption 'All effort values are in man days.'

footer -8<-
    === Staffing ===

    All project phases are properly staffed. See [[ResourceGraph]] for
    detailed resource allocations.
  ->8-
}

The Report for "Development" is as follows:

taskreport development "" {
  headline "Development - Resource Allocation Report"
  columns bsi { title 'WBS' }, name, start, end, effort { title "Work" },
          duration, chart { ${TaskTip} scale day width 500 }
  timeformat "%Y-%m-%d"
  hideresource ~(isleaf() & isleaf_())
  sortresources name.up
}

and finally, the Report for "Resource Graph" is like so:

resourcereport resourceGraph "" {
  headline "Resource Allocation Graph"
  columns no, name, effort, rate, weekly { ${TaskTip} }
  loadunit shortauto
  # We only like to show leaf tasks for leaf resources.
  hidetask ~(isleaf() & isleaf_())
  sorttasks plan.start.up
}

Now, save the project definition and compile it:

tj3 project.tjp

after successful scheduling, you'll see a set of HTML files and other assets that have been created. Head over to Overview.html in your browser:

You see Taskjuggler has computed a schedule for you. Verify that precedence has been respected. Note the cost by task and for the overall project (we have no revenue at this point). Also, see how the Work and Milestones umbrella tasks make our project schedule clearly presented by separating work to be done and the expected deliveries or milestones.

Let's explore the detailed task description by clicking on the Development link:

You see in this Report how tasks are being worked on by each resource.

Now head over to the Resource Graph report to see how are resources being used overall in this project:

You can notice how this report shows how much is every resource being loaded during the execution of its assigned tasks.

Conclusion

We just scratched the surface of what's possible with Taskjuggler. It is really a unique scheduling beast; one that can accomplish so much, but which is unfortunately not well served by its looks.
First, the only viable option to interact with it now is the proprietary syntax we've covered. This is showing its limits, as very poor tooling exists, and for large projects, with complex task hierarchies and report schemes, one can easily feel lost.
Second, although it's possible to track the progress of projects, Taskjuggler lacks advanced enterprise collaboration features. I think I've read somewhere it has a server or something, but I don't think it's of any practical use right now.

All of these problems make this tool only fit for the vim-savvy engineer, unfortunately. But once mastered, this is a very sharp one to run accurate time and money estimates, and to effectively track costs, and I think it can perfectly sit next to other collaborative agile Trellos and Jiras.

Scripting Containers With No Daemons using Podman

Rafik Naccache — Fri, 26 Jun 2020 10:54:10 +0000

Why Containers With No Daemons?

Believe it or not, I had to check this whole story when I failed to find any trivial way to stop my crawlers from eating all of my CPU cycles. I had the intuition to containerize them, then let the container runtime make these uncontrollable gremlins behave: I would run the crawlers as containers, setting system usage limits for them upon creation.

But I was held back by the way Docker relied on heavy daemons to work back then. Remember, my crawlers are supposed to be tiny un-noticeable anti-virus like programs - this is why we were bothering with this CPU story in the first place - so having Docker tooling and daemons as prerequisites was not an option.

I then explored "docker-less" or "daemon-less" solutions, so I stumbled upon alternatives like podman which is a lib that runs your containers with no running servers, or buildah that can build your Dockerfiles under the same conditions - no daemons running. The prospect of these solutions was promising, but they could only run on Linux, and as we were writing desktop software, it had to run on Windows at least, so I ended up writing my own CPU limiting code.

But these {docker,daemon}-less solutions are essential nonetheless: You'll be most probably facing more places where you need to leave docker comfort and work with alternative container runtimes (I am looking to you Kubernetes administrators).
So how easy is it to switch?

The Benefits of Standardization and Libpod

Docker, as a company, has made a great job making the Container what it is now, at the center of nearly every software engineering breakthrough we've seen in the last couple of years.

But they have contributed to an even more significant effort: the standardization of the Container landscape, through the Open Container Initiative - the OCI, along with other industry players.

Thanks to this standardization effort, it is possible for podman to rely on an OCI compliant Container Runtime to run and manage Containers on a Linux operating system. They're doing so under the form of a self-contained package libpod, thus providing us with a lightweight tool, with no servers to install at all, to run, monitor, stop, pull and tag our containers, just like you'd do with Docker. On their website, they say:

Podman specializes in all of the commands and functions that help you to maintain and modify OCI container images, such as pulling and tagging. It allows you to create, run, and maintain those containers and container images in a production environment.

Podman in action

You begin by installing podman. This will result in a binary that you'll use to create your Containers (on Linux. For Mac and Windows, it's another story). The installation instructions are here.

To run a Container, you just use the podman command to create one with a Docker (or any OCI Compliant) image:

podman run #... your classical options as you'd do in Docker

As in Docker, you can list the running Containers:

podman ps

Conclusion

Using podman, on a Linux machine, you can easily express your scripts in terms of Containers. There will be no unnecessary daemons haunting your space.
libpod, along with other OCI Compliant Runtime engines, are becoming more commonplace in the Kubernetes landscape. So Kubernetes admins might want to skill up in these runtimes and understand the trade-offs they are making when choosing one runtime over another in terms of functionality, security, and performance.

Further Reding

This post is heavily inspired by the following excellent piece: Dockerless, part 1: Which tools to replace Docker with and why

You might also want to check these following canonical references:

A mirror problem for daemonless Container Running is the daemonless container building. For this, you might want to check Buildah

NuID: Trustless Authentication Using Zero-Knowledge Proofs on the Blockchain

Rafik Naccache — Sun, 21 Jun 2020 21:24:14 +0000

The Cover Image is taken from the NuID Website

A while back, I got contacted by Locke Brown, the CEO of NuID Inc (read: New ID as in new identity management). He explained to me how his startup aspires to revolutionize the way we deal with identity and passwords in the world wild webs (pun intended). The promise of NuID is that they'd free us from the breaches leaking our passwords and the internet super-powers eavesdropping on us, by implementing a "Trustless Identity" solution on the Ethereum Blockchain Technology.

They spotted I once starred an authentication library in Clojure, so they thought I would be interested in giving their API a try. Needless to say, this is a compelling story to a nowhere man like myself, with a refreshing mixture of Clojure, Blockchain, Distributed Shared-Nothing authentication mechanism, so I gave it a very light test drive. I thought you might be interested in my discoveries so here we go!

I wrote this post on my very own initiative, I am not endorsing this company nor any of their products.

The Problem in Central Identity Management

So what's wrong in Centrally Managed Identity? To prove you are who you are pretending to be, online services rely on "Identity and Authentication Providers": These are services that store your identity along with your presumably secret access token. Whenever you log to apps using their credentials, or your handle at Facebook, Google, Github, Whatever... You first authenticate yourself with a central authenticating tier to prove your identity.

The problem lies in the very reliance on this central authority. You are "trusting" some internet being with your secret tokens - but you might be telling this "trustee" how you unlock other private stuff as well...Let's be honest, we pretty much use the same password for the twenty-something websites and email providers we access every day!

And even if this authentication store is no maleficient, it might not be protected against security breaches, and your precious identity can be leaked (check the Firefox Monitor to see if your handles have been leaked).

Zero-knowledge proof: The solution

So NuID suggests keeping your secrets well...secret. With their authentication mechanism, Called the "Zero-Knowledge Proof(ZKP)," No single central authority requires you to trust them with your access tokens. Instead, NuID uses ZKP to cryptographically derive from someone's secret a set of non-sensitive public tokens. It's not possible to tell the original password only knowing these tokens, yet they can be used to prove the knowledge of this very secret.

These public tokens will be immutably stored on the Blockchain (I think I've read Ethereum's Rinkeby somewhere...), so there will be no single central point in the architecture, nor it will be possible to mess around with those tokens. I think this excerpt from the Solution document describes it best:

The NuID protocol immutably stores public reference parameters on a blockchain, so that they can be shared but not tampered with. A blockchain provides a trustless infrastructure with no central point of failure.

Combining these technologies allows users to own and manage their authentication credentials while enabling companies to authenticate without storing and protecting sensitive information.

Developer Relations

The solution evolves mainly around API, and as such, their primary customers will be developers. So how appealing do they look to them?

First of all, I must confess I really appreciated the way they found me and personally reached out to me. I think they care about building their community and are carefully picking up potential testers, which is a positive signal I for one was able to read from them.

They also built a developer portal, which lets you test-drive their platform. It comes with fairly comprehensive documentation, and you should write your first authentication queries with no issues. Here's what's working with NuID tastes like (Using Curl, but you can use any of their client libs)

curl \
  -H 'X-API-Key: my-api-key' \
  https://auth.nuid.io/credential/gtgndTplbGxpcHRpYy5jdXJ2ZS9wb2ludHgsQStZelIvajZqdUxBS0pOWGJGaGw4UXczaXcrQ2ViWVJwa0RsN1hPQ2hETHM\=

And the result is:

{
  "pub": {
    "point": "A+YzR/j6juLAKJNXbFhl8Qw3iw+CebYRpkDl7XOChDLs",
    "curve": {
      "id": "secp256k1"
    }
  },
  "keyfn": {
    "id": "scrypt",
    "normalization-form": "NFKC",
    "salt": "TgbyOFik/5YxUl+TzufPnDoL6XZVgEq0f8cNr6qd9KY=",
    "key-length": 32,
    "n": 8192,
    "r": 8,
    "p": 1
  },
  "addresses": [
    {
      "transaction-id": "0x17da3d71783be5b509abd713e983ddc1fd3698d39bc02feeae782d67ba12772a",
      "network": "rinkeby"
    }
  ]
}

The code of the platform can be found on Github. This is done in Clojure, and in This talk the CTO explains their use of Clojure along with Datomic.

Let's Play Checkers with AI and Clojure

Rafik Naccache — Fri, 12 Jun 2020 13:06:47 +0000

Logic Programming: A forgotten Field in AI

Nowadays, when you hear "AI," you are probably going to think about neural networks and model training, starting to reach for data sets and envisioning how to feed them into the learning machine.
In this post, I will try to revive a more or less forgotten way to do things—the Logic programming way.

Logic programming is an exciting paradigm, according to which one expresses programs in terms of relations involving some logic variables, and expects results reflecting what do these logical variables have to be so to make sure those relations are verified(Pfeww!).

We are so talking about "goals" that we want to succeed: We want those relations to be actually occurring, and search for the values of our logic variables that would make that happen.

You can see how this sort of programming - also known as relational programming - lays the foundation of an alternative way of approaching computing. It's a declarative method in which you don't explicitly specify how to process inputs into outputs, but where you instead express queries about logical variables arranged in a set of desired relations.

For example, consider the following logical function :

(multiplio x y z)

that stands for a goal (or a relation) succeeding if z equals x times y.
It will succeed if you query for its result while you specify a relevant value for all of the three logical variables:

(multiplio 2 3 6) ;=>Success

But if you specify that you want to know for which values of z it will succeed, specifying actual, fixed values for x and y, you are sort of multiplying two numbers:

(multiplio 2 3 q) ;=> Success for q = 6

More interestingly, you can ask what might the value of x be given that y is set to 2 and that the product z at 6

(multiplio q 3 6) ;=> Success for q = 2

The previous examples showed how you can write programs in terms of relations, or logic goals, and specify or ask for the values of the variables you please: No inputs, no outputs, just relations.

Of course, when it is unable to make your goals succeed, the relational kernel will give you "nothing." On the other hand, if it notices that the goals are achieved every time, it returns a particular value that says, well, everything.

In the following, we will use relational programming to create an AI solving a round of checkers game. We will use a particular language for logic programming, miniKanren, through its Clojure implementation, core.logic.

A core.logic Primer

In core.logic, you run your queries using a special interface, run like in the following simple example:

(run* [q]
  (== q 1))

That is, you're asking core.logic to run a query over the logical variable q, so that the goal (== q 1) is verified. == is miniKanren's unification, that is, the mecanism that makes a goal who succeeds if the terms involved are the same, which sets q to (1) in our particular example. Unification is a very important operation in logical programming, and is implemented up to the level of nested Clojure datastructures:

(run* [q]
  (== [q 2][1 2]))

yields (1), as we must have q set at 1 so we have the first vector as [1 2] in order to have it unified with the second term.
But the careful reader may wonder: why do we have results laid out in sequences? This is beause the logical variable we are querying over might happen to take multiple possible values; as a matter of fact, you can choose how many solution do you want core.logic to go up to. We did mention run* (star) in our query because we want all of the solutions. If we did only want to uncover first five of them, we could have emitted a query like so:

(run 5 [q]
  (== [q 2][1 2]))

Playing a Checkers round

Enough talk and let's play checkers. For this, we are going to implement an artificial intelligence kernel capable of analyzing some checkers' board, determining – under the game rules – where to move while staying out of the reach of its opponent's attacks, and spotting when it can capture some of the enemy's soldiers.
Let's go through a little remainder of the game of checkers. You and your opponent evolve on a square chessboard, but you can only move on diagonals over black squares one step at a time. Also, you can only move forward. Both of you can capture an enemy piece if it is situated just one step away from one of your pieces, provided that when you jump over it following the diagonal, the black square where you're landing is empty. If, while doing so, from that new position, you can capture another piece, you may go on.

Modeling the Game

To model the battle scene, we are going to consider three sets: one for the empty positions, one for the only piece that belongs to the checkers' solver engine, and one for the enemy's locations. To describe a particular situation, you may tell the solver what's in each of these sets. As far as the items that will be put in them are concerned, we will store in them coordinates of positions on the checker's board. For instance, to tell that the enemy has 3 pieces, we just throw the 3 vectors representing the positions where these pieces are in the enemy's set.

The following figure explains how we are going to represent the different positions on a chessboard of 6 squares edges. Note that for its construction, to consider only the black squares, we are going to alternate a set of vectors crossing even with odd coordinates, then odd with even ones.

The solver is concerned about three significant matters when trying to figure out the action it must take:

Where to put the checker,
Will the moving piece be captured when it goes there,
Whether it can win one or more of its opponent's pieces if after jumping, he has the opportunity to do so. Let's implement it.

Building the Chessboard

For starters, we are going to declare our ns. Note that we are using the finite domain constrains engine, clojure.core.logic.fd, to be able to derive goals where algebraic operations are involved:

(ns recipe19.core
  (:require [clojure.core.logic :as logic]
            [clojure.core.logic.fd :as fd]))

We now focus on constructing the chessboard. As we mentioned earlier, we will build a set of vectors, associating odd x coordinates with even y ones, then the other way around: even x coordinates with odd y ones:

(defn evenso
  [size out]
  (logic/fresh [x2 se]
    (fd/in x2 (fd/interval 0  (/ size 2)))
    ;; We construct a set of integers
    ;; going from 0 to half the size
    ;; using a core.logic finite domain
    ;; facility
    (fd/* x2 2 se)
    ;; we construct the even coords
    (logic/== out se)))
;; and we output them by unifying them
;; with output

(defn oddso [size out]
  (logic/fresh [x2 seo so]
    (fd/in x2 (fd/interval 0  (/  size 2)))
    ;; Same logic applies for odd part
    (fd/* x2 2 seo)
    ;; We generate an even number
    (fd/+ seo 1 so)
    ;; Then we add 1 to it
    (fd/<= so size)
    ;; We ensure that we do not exceed
    ;; size
    (logic/== out so)))
;; And we unify with output

(defn boardo [size out]
  (logic/fresh [e o]
    (oddso size o)
    (evenso  size e)
    ;; Then starting from even and odd numbers
    (logic/conde 
     [(logic/== out [e o])]
     ;; We combine a first set of vectors
     ;; Crossing [even odd]
     [(logic/== out [o e])])))
     ;; And crossing [odd even]

Moving the piece

Let's build the goal function that determines where our piece shall go starting from a current position. This moves our piece up, incrementing the y part of the coordinates, and go to the left and to the right by adding or deducing 1 from the x component:

(defn where-to-movo
  [empty cur-pos out]
  (logic/fresh [->x ->y curx cury]
    (logic/== [curx cury] cur-pos)
    ;; We destructure the position into
    ;; x and y parts
    (logic/conde
     [(fd/+ 1 curx ->x)]
     [(fd/- curx 1 ->x)])
    ;; We go left or right,
    ;; adding or substracting 1
    ;; to x and storing it into ->x
    (fd/+ cury 1 ->y)
    ;; We can only move up, adding 1 to y
    ;; and storing it into ->y
    (logic/membero [->x ->y] empty)
    ;; We verify that the target position
    ;; is empty by seeing if it is present
    ;; in the empty set
    (logic/== out [->x ->y])))
    ;; And we unify with the output

Don't Get Captured!

After the solver has determined where it can move its piece, it needs a goal function that tells if, at that position, that piece won't be captured by any of the opponent's pieces. That's the aim of the following goal function:

(defn isnt-capturedo
  [empty mypos enemy out]
  (logic/fresh [x->
                      y->
                      ->x
                      ->y
                      y->y
                      x->x
                      new-x
                      new-y]
    (logic/== [x-> y->] enemy)
    (logic/== [->x ->y] mypos)
    ;; We destructure enemy and our
    ;; position
    (logic/conde
     [(fd/- x-> ->x x->x)]
     [(fd/- ->x x-> x->x)])
    ;; As core.logic.fd can't handle
    ;; negative numbers, we must use
    ;; a conde so we can get the difference
    ;; between the coords, whether is enemy's x higher
    ;; than the solver's, or the other way around.
    (fd/- y-> ->y y->y)
    ;; We know that the opponent can only go down
    ;; so we are able to proceed with a single
    ;; substraction, that is the opponent's y minus the
    ;; solver's
    (logic/conde
     [(fd/== x->x 1)
      (fd/== y->y 1)
      ;; This enemy's piece is close
      ;; enough to the solver's, let's check
      ;; if it can capture it.
      (logic/conda
       [(fd/> x-> ->x)(fd/- ->x 1 new-x)]
       ;; Where is the landing x coord had
       ;; the enemy captured the solver's
       ;; if it is at its left?
       [(fd/< x-> ->x)(fd/+ ->x  1 new-x)])
       ;; And where would it be if it did come
       ;; feom the right?
      (fd/- ->y y->y new-y)
       ;; What would the landing position y be, after
       ;; capture of the solver's piece?
      (logic/nafc logic/membero [new-x new-y] empty)
       ;; If it is not empty, the solver's piece is
       ;; safe 
      (logic/== out true)]
     [(fd/> x->x 1)(logic/== out true)]
     ;; The difference between coordinates
     ;; on the x axis is higher than 1,
     ;; This piece isn't a threat to us
     [(fd/> y->y 1)(logic/== out true)])))
     ;; Same logic for the Y axis

Catch'em all!

Our artificial checkers' player needs a goal function to help him spot the opportunity of capturing some enemy pieces. This is a recursive process, as after jumping over the piece it captured, we need to check if there is something else to recapture. That's why we built a tabled goal, that ensures termination by "memoizing" the different values taken by the tabled logical variables and deciding to exit en execution whenever the same patterns are encountered over and over again:

(def enemy-piece-capturo
  (logic/tabled
   ;; A tabled goal ensures that our
   ;; recursive goal terminates by
   ;; detecting recurrent patterns
   ;; among the values of the logical
   ;; variables
   [empty pos->
    enemies
    out]

 (logic/fresh [x->
                    y->
                    ->x
                    ->y
                    x->x
                    y->y
                   new-x
                   new-y
                   new-path]
     (logic/== [x-> y->] pos->)
     (logic/membero [->x ->y] enemies)
     ;; We destructure our piece's and
     ;; all of the enemies' positions
     (logic/conde
      [(fd/- x-> ->x x->x)]
      [(fd/- ->x x-> x->x)])
     ;; As negative numbers are considered
     ;; as failing goals, we yield a union
     ;; of both cases to be sure to get a
     ;; difference in x->x
     (fd/- ->y y-> y->y)
     ;; We know that this is our piece,
     ;; we can only go up
     (logic/conde
      [(fd/> x-> ->x)(fd/- ->x 1 new-x)]
      [(fd/< x-> ->x)(fd/+ ->x  1 new-x)])
     ;; the new-position's x after jumping,
     ;; adding +1 or -1 to the captured piece's x
     ;; depending of whether
     ;; we are going right or left
     (fd/+ ->y y->y new-y)
     ;; New y can only be the enemy's piece incremented
     ;; by 1 as we go always up.
     (logic/membero [new-x new-y] empty)
     ;; This landing position must be empty
     (fd/== x->x 1)
     (fd/== y->y 1)
     ;; And we must be exactly one square away
     ;; from the opponent's captured piece
     (logic/== new-path [:from [x-> y->]
                         :capture [->x ->y]
                         :then-> [new-x new-y]])
     ;; At this point we have a new path:
     ;; Where we started, what did we capture
     ;; and where we landed
     (logic/conde
      [(logic/== out  new-path)]
      ;; we emit this new path,
      ;; as if we were to stop here,
      ;; this is considered a legal move.
      [(logic/fresh [np]
         (enemy-piece-capturo empty
                              [new-x new-y]
                              enemies
                              np)
         ;; And we recursively check for another possible
         ;; piece to capture, adding any new discovered
         ;; path to what we've captured so far
         (logic/conjo [new-path :then->] np out))]))))

Sticking it all together

With all the building blocks for the chessboard analyzer ready, we can design a goal function that plays a round of checkers, minding all of the three matters constituting the engine's life: moving, keeping away from being captured and capturing enemy's soldiers:

(def play-roundo  
  (logic/tabled  [empty
                  cur-pos
                  enemies
                  out]
                 ;; This is a tabled goal so we can
                 ;; avoid some redundancy in the results
                 ;; caused by the unification engine
                 (logic/fresh [new-pos
                               new-empty
                               not-captured?
                               captured-them]
                   (logic/conde
                    ;; A main conde combining two main moves:
                    ;; 1. Capturing and emitting what has been
                    ;; captured
                    [(enemy-piece-capturo empty
                                          cur-pos
                                          enemies
                                          captured-them)
                     (logic/== out captured-them)]
                    ;; 2. Moving while keeping safe.
                    [(where-to-movo empty
                                    cur-pos
                                    new-pos)
                     (logic/conjo empty cur-pos new-empty)
                     ;; After moving, the piece former position
                     ;; is empty
                     (logic/everyg  #(isnt-capturedo new-empty
                                                     new-pos
                                                     %
                                                     not-captured?) enemies)
                     ;; All of the enemy's pieces must not be able to
                     ;; capture our piece once it moved to
                     ;; its new position
                     (logic/== out [:move-> new-pos])]))))
       ;; and we emit the positions in which we are safe.

The Checkers player in Action

Initializing the game

To be able to give some checkers' puzzles to the solver, we need an initialization function, that takes one set of enemies, the engine's piece, and emits the right set of empty squares, that is, the whole board minus the two first set:

(defn prepare-board
    [board-size enemies me]
    (let [initial-board (set  (logic/run* [q]
                                (boardo
                                 board-size
                                 q)))]
      ;; We generate a board of size board-size
      (into '()  (clojure.set/difference initial-board (set
                                                        (conj
                                                         enemies
                                                         me))))))
      ;; and we emit empty pieces: the whole starting set minus
      ;; enemy and "me" pieces

First Challenge: Multi-Capture

Consider the situation depicted by the following figure:

The solver can safely move left or capture one or two pieces. Let's submit this situation to the solver by issuing at your REPL:

(let [enemies '([1 4][3 2]) ;; positioning the enemies
      me [4 1]
      empty-brd (prepare-board 8 enemies me)]
  (logic/run* [q]
    (play-roundo
     empty-brd
     me
     enemies
     q)))

The results are the following:

([:move-> [5 2]] ; the simple move

 [:from [4 1] :capture [3 2] :then-> [2 3]] ; one capture

 [[:from [4 1] :capture [3 2] :then-> [2 3]] ; two captures
    :then->
    [:from [2 3] :capture [1 4] :then-> [0 5]]])

Second Challenge: Staying safe

Consider the following situation:

Although our piece can be moved left or right, the solver won't take the first choice as the enemy will capture the piece at [2 0]. To see it in action, type at the REPL:

(let [enemies '([3 2][1 4]) ; positioning the enemies
      me [1 0] 
      empty-brd (prepare-board 8 enemies me)]
  (logic/run* [q]
    (play-roundo
     empty-brd
     me
     enemies
     q)))

And the solver chooses:

([:move-> [0 1]])

Conclusion

We've seen how it is possible to tackle the checkers' problem using the relational programming paradigm. We used a clojure based implementation of the minikanren relational programming framework. We saw how we could model three of the game constraints: moving, staying safe, and trying to take prisoners.

On a final note, please be aware that this approach requires you to know what you're doing. As it is operating searches on deep forests of possible solution combinations, you must know how to model your program and be sure you don't hit resource limits if you throw in un-manageably sized problems. This been said, there's a significant body of knowledge concerned about mastering this kind of problem search. So, if you want to take this to production, you definitely need to do your homework and learn this art thoroughly.

This is inspired by a chapter of my Book Clojure Data Structures and Algorithms Cookbook

Generate, Monitor and Throttle CPU Load in Go

Rafik Naccache — Mon, 08 Jun 2020 14:02:57 +0000

The Rise of Nowhere Man

I originally wanted to write a Go vs. Rust vs. Python piece, shouted-out about it on Linkedin - But then saw in comments - and rightfully so - that there was already some prose dealing about it on the internets. I thought as I was going to focus on the "Systems" Programming abilities of these languages, my post was relevant. But then I realized I had a fresher, more personal, and presumably more vibrant story to share with the world.

I am what we can modestly qualify as an "architect" (not the J2EE kind) in the IT world. I don't mean the previous statement to be a testimony of any merit of any type, just spelled out the closest corporate job title by which the engineer that has to make it all work together goes by. It is no honor badge; it's a nightmare-ish situation. But a quite enjoyable one, as long as you consider yourself a free, always willing to learn individual. A "Nowhere Man."

By the very essence of this personal and professional trait, you can imagine my life as a succession of discoveries, akin to the Sinbad tales: At every stage of your journey, you'd be caught in the middle of a storm which will throw you into the shore of an unknown land. You know the story: you fight your way across an abundance of monsters and a scarcity of resources, only to set sails for the next storm. The "Chronicles of Nowhere Man" is following the same pattern: a stranded engineer (whom I prefer not to name) learning, and enjoying the shipwreck.

The first installment of the "Chronicles of Nowhere Man" series will narrate our stay at "Systems" Land. Let's explore it.

What is a System?

"System" is an umbrella term that means a lot of different things to many people. But I think we agree a decent definition for it would be "a concept, a design by which we deliberately make many smaller components interact to achieve a greater purpose." In things ranging from the "Immune System" to the "Solar System" - passing by the "Operating System," you can see that definition in work.

Keeping this definition in mind (and the keywords "design," "components," "interact," "greater purpose"), you can more or less guess what do I imply when using "systems programming."

"System programming" goes beyond proudly affirming you can program in Go or Rust (or C for that matter), and it certainly involves threads inter-communication, asynchronous job execution, and global state management. It can even include architectural skills, especially in today's cloud-natively orchestrated world.

Systems Land

I was stranded in "Systems Land" when I was sailing with a US-based Cybersecurity solutions provider. Our solutions were shiningly promising to bring AI to the Cybersecurity world, detecting security holes in files (This part was NLP/Regexp), and identifying Cyber attackers attempting to sneak into your network (This part wasn't).

As the architect on board, I was assigned the task of creating some "honey pot" mechanism. That is, I had to provide for lures that would look appealing to attackers, so when they try to break in, we would recognize and neutralize them.
And then I've seen that there were many levels at which "Systems Designing" or "Programming can be approached."

The Architectural Systems Level

At A Global or Architectural level, we had to make a "System" of communicating honeypots, reporting malicious activity, so upstream firewalls can keep the attackers out. There was an orchestrator that could manage the pots and get back their findings. Locally, each honey pot had a local sate store and was empowered to react if it could not reach the orchestrator. We've designed an efficient and asynchronous message passing from the Honeypots and the orchestrator. We created the global "Architectural System" as a whole tolerant to network failures.

The Systems Programming Level

At a local level, these "honey pots" needed to look real. They had to act like actual software, showing actual "Systems usage" consumption, that is, a decently sized amount of memory, disk, and CPU usage, all parametrized by the domain experts. It is a reflection we had at the Systems Programming level.

The honeypots were written in GO. The disk and memory usage generation was straight forward, so let's focus on the CPU part.

We generate The CPU Load by running a tight infinite loop. At each iteration of this loop, we verify if we attained our target CPU load. When we hit the CPU Load threshold, we "relieve" this loop by parking the process:

// generate load on each of the cores
for i := 0; i < min(cores, runtime.NumCPU()); i++ {
   go func() {
    for { // The tight loop generates the Load like crazy !
                cpuLoad := CpuUsagePercent(samplingRate, debug)
                if cpuLoad >= targetCPU { // If Load threshold attained
                  waitTime := time.Second / time.Duration(WaitFactor)
                   time.Sleep(waitTime) // We hit the brakes pedal
                }
             }
   }
}

How does CPUUsagePercent Work? In a nutshell, we use time.h interface on Unices or the "psapi.h" interface on Windows to compute the number of clock seconds spend on the current process (cpu_time), which we divide by the whole time this process(real_time) has been working. The gotcha here was that time.h would compile on windows, but the results would not be relevant, so you have to watch out!
I share a non copyrighted version here, but the gist of it for *nix/macOS:

package main

// #include <time.h>
import "C"

import (
    "log"
    "time"
)

var startTime = time.Now()
var startTicks = C.clock()
var WaitFactor = 1

// if samplingRate = 0 then we continue with initial startTime and startTicks
func CpuUsagePercent(samplingRate float64, debug bool) float64 {

    clockSeconds := float64(C.clock()-startTicks) / float64(C.CLOCKS_PER_SEC)
    realSeconds := time.Since(startTime).Seconds()

    if debug {
        log.Printf(" current clock  : %v, real seconds: %v, startTicks: %v", clockSeconds, realSeconds, startTicks)
    }

    if samplingRate > 0 && realSeconds >= samplingRate {
        startTime = time.Now()
        startTicks = C.clock()
        if debug {
            log.Printf("Resetting starts !! startTime : %v, startTicks: %v", startTime, startTicks)
        }
    }
    return clockSeconds / realSeconds * 100

}

And on Windows:

package main

/*
#cgo LDFLAGS: -lpsapi
#include <windows.h>
#include <psapi.h>
#include <time.h>
double get_cpu_time(){
    FILETIME a,b,c,d;
    if (GetProcessTimes(GetCurrentProcess(),&a,&b,&c,&d) != 0){
        //  Returns total user time.
        //  Can be tweaked to include kernel times as well (c).
        return
        (
            (double)(d.dwLowDateTime |
            ((unsigned long long)d.dwHighDateTime << 32)) // user time
        ) * 0.0000001;
    }else{
        //  Handle error
        return 0;
    }
}
*/
import "C"

import (
    "log"
    "time"
)

var startTime = time.Now()
var startTicksTime = C.get_cpu_time()
var WaitFactor = time.Duration(C.CLOCKS_PER_SEC)

// if samplingRate = 0 then we continue with initial startTime and startTicks
func CpuUsagePercent(samplingRate float64, debug bool) float64 {

    clockSeconds := float64(C.get_cpu_time() - startTicksTime)
    realSeconds := time.Since(startTime).Seconds()

    if debug {
        log.Printf(" current clock  : %v, real seconds: %v, startTicks: %v", clockSeconds, realSeconds, startTicksTime)
    }

    if samplingRate > 0 && realSeconds >= samplingRate {
        startTime = time.Now()
        startTicksTime = C.get_cpu_time()
        if debug {
            log.Printf("Resetting starts !! startTime : %v, startTicks: %v", startTime, startTicksTime)
        }
    }
    return clockSeconds / realSeconds * 100

}

You probably noticed the WaitFactor Param. It was the best solution I found to my knowledge to let the program perform the closest to the target CPU threshold! I let the external honey pot orchestrator set this param after a manual or automatic trial and error feedback loops (mostly manual to be honest)

And in the End

We've covered a lot of ground in this first adventure. First, we've seen what a "System" means, in terms of components, interaction, and a great purpose to achieve.
Then, we've seen that "Systems" includes two levels:
A Global one, which we called "Architectural Systems Level" making it possible for the whole to collaborate and synchronize, providing for resiliency and scalability
And a local one, which we called the "Systems Programming Level," where we might need to interact by code with low-level components as we've seen with the CPU piece.
In the end, being an architect might involve you knowing, beyond architectural design patterns, how to interact with low-level systems components. This adventure made me learn more about CPU cycles counting and interacting with it by code surely added to my infrastructure and monitoring skills.
Where we'll be wandering next?

Forem: Rafik Naccache

Building turbo-ledger: A Scalable Ledger with Go and Redis

turbo-ledger: Why and How

What's a ledger and why should we bother?

Wallets: where the assets lie

Transactions: Assets movement tracking

Moving Assets Around

A Naive turbo-ledger Implementation with Redis

Why Redis?

Prototyping quickly with RedisCloud and RedisInsights Desktop

RedisCloud

RedisInsights Desktop

Representing our wallets' Objects

Wallets as JSON Documents

Wallets in Go

Indexing and searching through wallets

Representing Our transaction Objects

Transactions inside Arrays in Wallets

You want transactional transactions

Transactions In Go

Making turbo-ledger turbo

Preventing the Double Spend Problem with local locks

Addressing double-spend with a Redis-based distributed lock: RedLock

Conclusion

Plan like a Pro with Automatic Scheduling in Taskjuggler

The Need for Automatic Project Scheduling?

Taskjuggler: Thinking in Work Breakdown Structure

Taskjuggler in action

Conclusion

Scripting Containers With No Daemons using Podman

Why Containers With No Daemons?

The Benefits of Standardization and Libpod

Podman in action

Conclusion

Further Reding

NuID: Trustless Authentication Using Zero-Knowledge Proofs on the Blockchain

The Problem in Central Identity Management

Zero-knowledge proof: The solution

Developer Relations

Further Reading

Let's Play Checkers with AI and Clojure

Logic Programming: A forgotten Field in AI

A core.logic Primer

Playing a Checkers round

Modeling the Game

Building the Chessboard

Moving the piece

Don't Get Captured!

Catch'em all!

Sticking it all together

The Checkers player in Action

Initializing the game

First Challenge: Multi-Capture

Second Challenge: Staying safe

Conclusion

Generate, Monitor and Throttle CPU Load in Go

The Rise of Nowhere Man

What is a System?

Systems Land

The Architectural Systems Level

The Systems Programming Level

And in the End