Forem: Gabriel Saldanha

How to write concurrent code using Python's Future

Gabriel Saldanha — Wed, 25 Nov 2020 11:43:11 +0000

Python futures (concurrency)

tl;dr;

Using concurrency to speed up things is quite simple in Python using the concurrent.futures module. However, it's no silver bullet and one must know when to use it.

All started with generators...

In a previous post I mentioned how you could use Python generators as a way to avoid performing extra service calls, saving up some time and resources. Here is a summary:

# Suppose you want to check if a given user_email is registered in 
# any of the following social media: Facebook, Github, or Twitter.

def has_facebook_account(user_email):
        ...

def has_github_account(user_email):
        ...

def has_twitter_account(user_email):
        ...

def has_social_account(user_email):
    calls = [
        has_facebook_account,
        has_github_account,
        has_twitter_account,
    ]
    return any((call(user_email) for call in calls))
    # (expr for i in iterable) syntax denotes a generator comprehension

Pretty simple, huh? any iterates and evaluates each call(user_email) yield from the generator until one of them returns True. This is known as early return — you basically save up some time and resources by not performing unnecessary calls.

Some folks gave nice feedback mentioning that I should make it concurrent, i.e., I could make all the calls concurrently and early return as soon as any call returned True. "That's a good idea", I thought. I'm glad there are smarter people than myself out there.

If that's not clear why I would want to do it: suppose has_facebook_account takes too long to run (as usually happens with any I/O and network operations due to high latency) and has_github_account is pretty fast (maybe it's cached, for example). I would always need to wait for has_facebook_account return before calling has_github_account since the generator's items would be evaluated orderly. That does not sound fun.

Make it concurrent!

I am using Python's concurrent.futures module (available since version 3.2). This module consists basically of two entities: the Executor and the Future object. You should read this documentation, it is really short and straight to the point.

The Executor abstract class is responsible for scheduling a task (or callable) to be executed asynchronously (or concurrently). Scheduling a task returns a Future object, which is a reference to the task and represents its state — pending, finished, or canceled.

If you have ever worked with JavaScript Promises before, Future is very similar to a Promise: you know its execution will eventually be done, but you can not know when. The nice thing is: it is non-blocking code, meaning the Python interpreter does not have to wait until the scheduled task's execution finishes before running the next line of code.

Thus, in our scenario, we could schedule three tasks, one for querying each platform (Facebook, GitHub, and Twitter) for a user email address. This way, once any of these tasks eventually returns a value, I can early return if the value is True, since all we want to know is if the user has an account in any of these platforms.

Talk is cheap. Show me the code.

The example code below is easy to follow but I strongly suggest you read the commented lines.

import time  # I will use it to simulate latency with time.sleep
from concurrent.futures import ThreadPoolExecutor, as_completed

def has_facebook_account(user_email):
    time.sleep(5)  # 5 seconds! That is bad.
    print("Finished facebook after 5 seconds!")
    return True

def has_github_account(user_email):
    time.sleep(1)  # 1 second. Phew!
    print("Finished github after 1 second!")
    return True

def has_twitter_account(user_email):
    time.sleep(3)  # Well...
    print("Finished twitter after 3 seconds!")
    return False

# Main method that answers if a user has an account in any of the platforms
def has_social_account(user_email):
    # ThreadPoolExecutor is a subclass of Executor that uses threads.
    # max_workers is the max number of threads that will be used.
    # Since we are scheduling only 3 tasks, it does not make sense to have
    # more than 3 threads, otherwise we would be wasting resources.
    executor = ThreadPoolExecutor(max_workers=3)

    # Schedule (submit) 3 tasks (one for each social account check)
    # .submit method returns a Future object
    facebook_future = executor.submit(has_facebook_account, user_email)
    twitter_future = executor.submit(has_twitter_account, user_email)
    github_future = executor.submit(has_github_account, user_email)
    future_list = [facebook_future, github_future, twitter_future]

    # as_completed receives an iterable of Future objects
    # and yields each future once it has been completed.
    for future in as_completed(future_list):
        # .result() returns the future object return value
        future_return_value = future.result()
        print(future_return_value)
        if future_return_value is True:
            # I can early return once any result is True
            return True

user_email = "user@email.com"
if __name__ == '__main__':
    has_social_account(user_email)

Finished github after 1 second!
User has social account.
# The created threads will still run until completion
Finished twitter after 3 seconds!
Finished facebook after 5 seconds!

Notice that even though facebook_future takes longer than the other two scheduled tasks to finish, however, it does not block the execution — it keeps working on its own thread. And although github_future is the last scheduled task, it is the first to finish.

Quick summary

Future is an object that represents a scheduled task that will eventually finish.
Executor is the scheduler of tasks (once a task is scheduled, it returns a Future object).
- It can be a ThreadPoolExecutor or a ProcessPoolExecutor (using threads vs processes).
One can use executor.submit(callable) to schedule a task to be run asynchronously.
as_completed receives an iterable of Future objects and returns a generator where each yielded element is a finished task.

When would I not want to use concurrency then?

As software engineers, our job is not only knowing how to use a tool, but also when to use it. Network operations (and I/O bound operations in general) are usually a good place to use concurrent code due to their latency. But there is always a trade-off...

In the example above we traded off performance for resource usage. How so? When using generators, only the worst-case scenario would end up consuming 3 services — one call for each has_<plataform>_account. That's because we could early return True if any service returned True.

In our new example using concurrency, we are always consuming the 3 service — since the calls are made asynchronously.

"Ah, but that still could save us lots of time!", you say. It depends on the services you're consuming. In the example above I artificially made the has_facebook_account really slow — 5 times slower than the fastest alternative. But, if all the services had a similar response time and if saving resources was important (suppose that calling each service would trigger a really heavy query in the database, for instance), using a synchronous code could be a better approach.

For the sake of data: Facebook has over 2.7 billion monthly active users, while Twitter has around 330 million, and GitHub has merely 40 million users. So, it is highly likely that calling the has_facebook_account first would be enough in a huge majority of scenarios since it would return True with a much higher frequency than the other services, thus, saving lots of unnecessary calls.

Conclusion

Know how to write concurrent code, which is pretty easy with Python Futures. But more important: know when to do so. There are cases where the performance increase does not pay off the resource usage.

I strongly advise you to read the docs on concurrent.futures and the Chapter 17 on Luciano Ramalho's excellent Fluent Python book.

Ah, and here is the gist if you want it!

Persist data to the Ethereum blockchain using Python, Truffle and Ganache

Gabriel Saldanha — Mon, 25 May 2020 11:25:42 +0000

The previous post demonstrated how to write a simple smart contract with Solidity and deploy it to the Ethereum Blockchain.
This tutorial will show how to update the contract to save some data in the blockchain as well as how to inspect the blockchain to see our transactions.

We'll work on the same files from the previous tutorial, which can be found here.

If not in the mood of going along with the tutorial, the finished code is available here.

1. Project setup

Open HelloWorld.sol and add the following:

 pragma solidity >= 0.5.0;

 contract HelloWorld {
+    string public payload;
+
+    function setPayload(string memory content) public {
+        payload = content;
+    }
+
     function sayHello() public pure returns (string memory) {
         return 'Hello World!';
     }
 }

As we did before, spin up truffle local blockchain with truffle develop CLI:

$ truffle develop
Truffle Develop started at http://127.0.0.1:9545/

Accounts:
(0) 0x55660bb0918f9aade4db7b7ef7f71639750c2262
...
Private Keys:
(0) ead0d6b318832f278b9a2ee1a6a6ab0156ab439d898394b6887d023bc929159c
...
$ truffle(develop)>

Now, we need to update our contract, what under Truffle's Suite is performed with the migrate command. Write migrate within Truffle's console.

Compiling your contracts...
===========================
> Compiling ./contracts/HelloWorld.sol
> Compiling ./contracts/Migrations.sol
...
2_deploy_contracts.js
=====================

   Replacing 'HelloWorld'
   ----------------------
   > transaction hash:    0x6f55fc4a165225c6911d6e0acda8b4ec75b0962370eab66ab94aefd0e0cfd084
   > Blocks: 0            Seconds: 0
   > contract address:    0x85534E0ec31b63d635CcBa1fb6b953a9C47a9022
...

The contract is now updated with a public method setPayload deployed at 0x85534E0ec31b63d635CcBa1fb6b953a9C47a9022 (contract address). Write this address somewhere else, we will need it soon.
We can verify that the contract was properly updated by fetching the contract instance and checking for setPayload method existence, as well as the value of payload variable:

# We're still inside Truffle's console
$ truffle(develop)> let instance = await HelloWorld.deployed()
$ truffle(develop)> instance.setPayload
[Function] {
  call: [Function],
  sendTransaction: [Function],
  estimateGas: [Function],
  request: [Function]
}
$ truffle(develop)> instance.payload.call()
''  # Empty, as expected!

2. Executing a transaction

When we call a function that only returns a value, such as sayHello, the blockchain state is not altered. Since now we want to save a value to the blockchain, its state must be altered and it is done via a transaction execution.
We want to execute this transaction from our Python script, the same we used for calling the sayHello method in the previous post.
Add the following to the bottom of app.py:

# executes setPayload function
tx_hash = contract.functions.setPayload('abc').transact()
# waits for the specified transaction (tx_hash) to be confirmed
# (included in a mined block)
tx_receipt = web3.eth.waitForTransactionReceipt(tx_hash)
print('tx_hash: {}'.format(tx_hash.hex()))

Also, update the deployed_contract_address in app.py with the new contract address (output from migrate). If you don't remember it, go back to the Truffle console and call instance.address.

On another terminal session, execute our Python script:

python app.py
Hello World!
tx_hash: 0x2b7bf989fd0fc692d0ef976b3293c39958ab7de64a08a3f66c0d758b871ad8e6

Not fun, right? How do we know that the data was persisted? Go back to the Truffle console and call payload variable:

$ truffle(develop)> instance.payload.call()
'abc'  # cool! It changed!

This means that our transaction was mined, thus being added to a block that was successfully added to the blockchain.

3. Inspecting the blockchain with Ganache

There are various ways for inspecting the blockchain. For example, using the Truffle console:

# Fetch blockchain's latest mined block
$ truffle(develop)> block = await web3.eth.getBlock("latest")
$ truffle(develop)> block
{
  number: 6,  # YMMV
  (...)
  transactions: [
    '0x2b7bf989fd0fc692d0ef976b3293c39958ab7de64a08a3f66c0d758b871ad8e6'
  ]
}

transactions lists all the transactions that were added to this block. In this case, there is one transaction that should match tx_hash that was printed when we ran app.py.

To make our lives easier, there is a GUI tool for inspecting the blockchain and its transactions: Ganache. As per their website:

Quickly fire up a personal Ethereum blockchain which you can use to run tests, execute commands, and inspect state while controlling how the chain operates.

After successfully installing and opening it, you should see the following image:

Select "Quickstart" and you'll be welcomed with a screen listing some accounts and its respective addresses (very similar to how truffle develop works). There will be also a RPC Server address, such as http://127.0.0.1:8545, write that down. Click "Save" (top-right corner) so this Ganache workspace is persisted.

When quick-starting Ganache, it created another local blockchain for us. We can now tell truffle to use Ganache's blockchain (which has a nice UI). To do so, open truffle-config.js and replace it with the following:

module.exports = {
  networks: {
    development: {
      // host and port should match the RPC Server address
      // as seen in Ganache
      host: "127.0.0.1",
      port: 8545,
      network_id: "*"
    }
  }
};

Go to the Contracts tab, and select Link Truffle Projects. Click Add Project then select the truffle-config.js file from hello-eth/ root project folder. Click Save and Restart at the upper-right corner.

If you go to the Contracts tab now, you should see two contracts:

HelloWorld: Not Deployed
Migrations: Not Deployed

Open another terminal session, go to the project root level (where truffle-config.js is located), and type truffle migrate. It will deploy our contracts to Ganache's blockchain. Copy the HelloWorld contract address and update the deployed_contract_address in app.py with its new value, as well as the variable blockchain_address with http://127.0.0.1:8545.

- blockchain_address = 'http://127.0.0.1:9545'  # truffle blockchain
+ blockchain_address = 'http://127.0.0.1:8545'  # ganache blockchain
- deployed_contract_address = '0x85534E0ec31b63d635CcBa1fb6b953a9C47a9022'
+ deployed_contract_address = '0x433D7E91B659CB4f1bbBb8dacd2B1cBBa7F82F1A'

Under Contracts tab now, both contracts (HelloWorld and Migrations) should have an address. It means they're deployed! woohoo!
Now, run app.py again to execute a new transaction on our newest blockchain and then click on HelloWorld contract:

Notice the TX HASH, it should be the same as printed when you ran app.py.

And if you click on the transaction itself:

You can see both the function that was called and the inputs.

Run app.py a few more times, change abc (in tx_hash = contract.functions.setPayload('abc').transact()) to something else, then inspect your contract again. It's YOUR blockchain now, act up to it!

Wrapping up

In this tutorial, you learned how to:

Write a method that changes the blockchain state;
Use Truffle CLI to interact with a deployed smart contract;
Use Ganache app both as the blockchain provider as well as its inspector;

If something is not clear or if there are any suggestions, please leave a comment below!

Deploy a Smart Contract on Ethereum with Python, Truffle and web3py

Gabriel Saldanha — Tue, 12 May 2020 13:10:13 +0000

In this tutorial, we'll write a simple smart contract, deploy it to a personal Ethereum blockchain, and call the contract from a Python script.

What you need to have installed before we proceed:

Python3 v3.5.3 or later, I had some issues using version 3.8 then switched to 3.5.3;
NodeJS v8.9.4 or later (for installing truffle);

The full project code is available at GitHub

1. Project setup

First, let's create a folder for the project:

$ mkdir hello-eth
$ cd hello-eth

Inside hello-eth folder, install truffle:

npm install truffle
...
# You should something like this once the installation process is finished
+ truffle@5.1.25

We will use the truffle CLI tool to initialize an empty smart contract project:

$ ./node_modules/.bin/truffle init
# if truffle was installed globally (npm install -g truffle): $ truffle init

The above command will create the following project structure:

contracts/: Directory for Solidity contracts source code (.sol files).
migrations/: Directory for contracts migration files.
test/: Directory for test files. Won't be covered in this tutorial.
truffle.js: Truffle configuration file.

contracts/ and migrations/ folders will already contain a Migration contract and its deploy script (1_initial_migration.js). This contract is used by truffle to keep track of the migrations of our contracts. It won't be covered in this tutorial, so don't worry about it.

2. Writing the smart contract

Yes, I'll use the Hello World example. Create a file called HelloWorld.sol inside contracts/ folder and add the following content to it:

pragma solidity >= 0.5.0 < 0.7.0;

contract HelloWorld {
    function sayHello() public pure returns (string memory) {
        return 'Hello World!';
    }
}

From the project root folder, use truffle to compile the contract:

$ ./node_modules/.bin/truffle compile  # or just truffle compile if installed globally...

It will output the compiled contract HelloWorld.sol as HelloWorld.json inside build/contracts/ folder.

We now have a compiled contract and are ready to deploy it to our running to the (local) blockchain.

3. Deploying smart contract to the blockchain

To deploy our smart contract to the blockchain we first need:

a migration script;
a blockchain to deploy the contract to;

For the migration script, create a file named 2_deploy_contract.js inside migrations/ folder and add the following content to it:

var HelloWorld = artifacts.require("HelloWorld");

module.exports = function(deployer) {
    deployer.deploy(HelloWorld);
};

Truffle suite contains a personal blockchain that we can use for testing purposes. Open your terminal and run:

$ truffle develop

You should see an output similar to the following:

Truffle Develop started at http://127.0.0.1:9545/

Accounts:
(0) 0xdfb772fba7631b5bfde93cc3e2b0e488d1a17b2a
...
(9) 0x974779d6a98264043e8bb1c8b0cf93d9c7141a29

Private Keys:
...

truffle(develop)>

We can now migrate our contract, which calls each migration in migrations/ (in order), deploying the contracts to the blockchain.

truffle(develop)> migrate
Starting migrations...
...
2_deploy_contracts.js
=====================

   Deploying 'HelloWorld'
   ----------------------
   > transaction hash:    0xddc3dd045a7b6f70063303ff534d07e93c1dcb7a0a6c15e42fe281c7d2ab53e8
   > Blocks: 0            Seconds: 0
   > contract address:    0xb7afC8dB8EEf302cd30553B39cEa0599093FDE3C

contract address is the address on the Ethereum network that hosts this contract instance.
Sweet! We now have a Smart Contract deployed on our personal Ethereum blockchain. Next step is call it from a Python script.

4. Calling the deployed contract

Under the project root folder (hello-eth/) create a file named app.py with the following content (pay attention to the comments explaining the script):

import json
from web3 import Web3, HTTPProvider

# truffle development blockchain address
blockchain_address = 'http://127.0.0.1:9545'
# Client instance to interact with the blockchain
web3 = Web3(HTTPProvider(blockchain_address))
# Set the default account (so we don't need to set the "from" for every transaction call)
web3.eth.defaultAccount = web3.eth.accounts[0]

# Path to the compiled contract JSON file
compiled_contract_path = 'build/contracts/HelloWorld.json'
# Deployed contract address (see `migrate` command output: `contract address`)
deployed_contract_address = '0xb7afC8dB8EEf302cd30553B39cEa0599093FDE3C'

with open(compiled_contract_path) as file:
    contract_json = json.load(file)  # load contract info as JSON
    contract_abi = contract_json['abi']  # fetch contract's abi - necessary to call its functions

# Fetch deployed contract reference
contract = web3.eth.contract(address=deployed_contract_address, abi=contract_abi)

# Call contract function (this is not persisted to the blockchain)
message = contract.functions.sayHello().call()

print(message)

Now, execute the following commands in your terminal:

$ pip3 install web3
$ python3 app.py
> 'Hello World!'

Wrapping up

In this tutorial, you learned how to:

Write a simple Smart Contract in Solidity;
Create a personal Ethereum Blockchain for tests and development;
Deploy a contract to the blockchain using truffle;
Call the contract function from a Python application.

I will write two follow-up posts in the next few days:

how to persist data to the blockchain via a Web API using Flask;
how to use Ganache as the personal blockchain provider (which has a nice GUI to inspect transactions);

If you'd like to be notified when that comes up, make sure to follow me on Dev.to.

Using Python Generators to avoid extra service calls

Gabriel Saldanha — Tue, 28 Apr 2020 14:45:21 +0000

This is my first post on dev.to, and actually my first public post on the Internet that is not a tweet. Feedbacks are very welcome! :)

I've been using Python Generators for a while now, having to deal with large Django Querysets, reading large Excel files, and being able to save memory by using generators became part of my day to day work. However, these days I faced a different problem where I wanted to avoid making extra calls to services and solved it with generators. This is what this post is about.

Suppose you want to check if a given user_email is registered in any of the following social media: Facebook, Github, or Twitter.

def has_facebook_account(user_email):
    print('calling Facebook service')
    return False

def has_github_account(user_email):
    print('calling Github service')
    return True

def has_twitter_account(user_email):
    print('calling Twitter service')
    return True

def has_social_media_account(user_email):
    print('Checking social media apps...')
    # Python's `any` receives an Iterable
    # and returns True when the first truthy clause is found.
    response = any([
        has_facebook_account(user_email),  # This is False
        has_github_account(user_email),  # This is True
        has_twitter_account(user_email),  # This is True
    ])
    print('Done!')
    return response

What is wrong with the approach above? Well, if you run it locally you will see the following output:

>>> has_social_media_account('fake@email.com')
Checking social media apps...
calling Facebook service  # This is False, keep going...
calling Github service  # This is True! I can stop now...
calling Twitter service  # Oh no!
Done!

The problem is that a list is evaluated as soon as it is created (opposite to lazy evaluation). Thus, if it's a list of method calls, all the calls will be performed.

At the moment, I thought the problem was that I was actually calling the methods when initializing the list. So maybe keeping only the methods references and using a list comprehension would do the trick:

def has_social_account(user_email):
    calls = [has_facebook_account, has_github_account, has_twitter_account]  # Refs
    return any([call(user_email) for call in calls])

Well, it did not work as I wanted either. As I said before, a list is fully evaluated upon its creation. In the example above, it will first make sure to call all methods so the list is fully built, having all of its elements evaluated - which in turn means having all methods being called. The list comprehension is not aware of its context either, i.e., it does not know that it is being used as an argument to any(...).

How did I solve it? By using generator.

def has_social_account(user_email):
    calls = [has_facebook_account, has_github_account, has_twitter_account]
    return any((call(user_email) for call in calls))  # Note the ( ) instead of [ ]

>>> has_social_account('fake@email.com')
calling Facebook service  # This is False, keep going...
calling Github service  # Aw yeah!

(call(user_email) for call in calls) evaluates to a generator (not a fully built list), which is then used by any. any(...) will iterate over the generator elements evaluating one at a time. This way, no extra calls are being made for once any evaluates the second element (has_github_account(user_email) -> True), the any function is evaluated itself, returning True and not calling the third service method (has_twitter_account).

Here is the Gist if you're interested.

Please let me know in the comments below if something is not clear and/or if you'd like to have a more in-depth article on Python Generators.

Forem: Gabriel Saldanha

How to write concurrent code using Python's Future

Python futures (concurrency)

All started with generators...

Make it concurrent!

Talk is cheap. Show me the code.

Quick summary

When would I not want to use concurrency then?

Conclusion

Persist data to the Ethereum blockchain using Python, Truffle and Ganache

TOC

1. Project setup

2. Executing a transaction

3. Inspecting the blockchain with Ganache

Wrapping up

Deploy a Smart Contract on Ethereum with Python, Truffle and web3py

TOC

1. Project setup

2. Writing the smart contract

3. Deploying smart contract to the blockchain

4. Calling the deployed contract

Wrapping up

Using Python Generators to avoid extra service calls