Forem: spongechameleon

How to request and store data using the Interactive Brokers API

spongechameleon — Tue, 27 Oct 2020 05:02:45 +0000

Introduction

If you've started working with the Interactive Brokers API (via the ibapi python package) but you're confused about how to interact with TWS, you're in the right place!

In this post I will present one methodology of reliably asking for and receiving data from the TWS GUI using the ibapi package in your programs.

The methodology consists of the two main sections in this post:

Tying together the EClient and EWrapper classes
Implementing a strategy to create, read from and destroy queue objects

Before we dive in, though, I'm going to assume that you already have a working connection to TWS from your python program.

If you do not, there are already plenty of great blog posts and examples walking you through how to get setup and connected, including the official docs and this post from @wrighter, so go ahead and get connected & then jump back here for the low-down.

But first, the big picture

Let's first review the high level flow:

The TWS GUI is our source of truth. It is what our python program talks to- which is why the TWS GUI must be running locally for our program to work.

Hence, the one and only API we are using is the API for this local GUI program. Any communication external to our machine is handled by the TWS GUI itself- which is not our concern (as long as you have internet).

Also, the API is the collection of class definitions given to us in the ibapi package- it's not a URL, as you may have assumed (this confused me for a while).

And here's a screenshot of TWS just so we're all on the same page about what's what:

Now that we've got that sorted let's get right into it.

Tying together the EClient and EWrapper classes

As you may have already learned from the official documentation, the ibapi package instructs us to work with two classes, EWrapper and EClient.

The EClient class gives us one-way communication to send messages to TWS. The messages EClient sends are requests for data.

The EWrapper class gives us one-way communication to receive messages from TWS. The messages EWrapper receives are data.

Herein lies the crux of the problem. We can ask for data from our EClient methods, but we're not going to receive a response in those methods. We only receive messages from EWrapper methods.

So how do we coordinate asking for and receiving data across these two different classes?

If you're from the web development world like me, and you're used to requesting and receiving data in the same method call using something like: fetch('https://google.com').then(res => res.json())... this presents a serious paradigm shift (at least until we work some magic 😁).

Part of the answer is to tie the EClient and EWrapper classes together in one instantiated object. This process is given to us by the official docs in this (slightly modified) code snippet:

from ibapi.wrapper import EWrapper
from ibapi.client import EClient

class TestWrapper(EWrapper):
    def __init__(self):
        pass

class TestClient(EClient):
    def __init__(self, wrapper):
        EClient.__init__(self, wrapper)

class TestApp(TestWrapper, TestClient):
    def __init__(self):
        TestWrapper.__init__(self)
        TestClient.__init__(self, wrapper=self)

First you'll see that TestWrapper is just a direct copy of EWrapper (we'll do something useful and different later).

Next, we see that our new client class TestClient is just the good old EClient that we know and love, except that upon instantiation, it must be provided with access to a wrapper object.

And so when we instantiate TestApp via app = TestApp():

We first instantiate TestWrapper to the app variable
We then pass that app variable to TestClient as wrapper, and then re-instantiate the app variable as a TestClient instance instead

If you look carefully at the EClient.__init__() definition you'll see that the wrapper argument is attached to the new object as self.wrapper:

class EClient(object):
    def __init__(self, wrapper):
        self.wrapper = wrapper
        ...

So the end result is that we now have an app object:

That has access to TestClient properties and methods via app.some_client_method() calls
That has access to TestWrapper properties and methods via app.wrapper.some_wrapper_method() calls

This is basically a clever way to have two object instances in one. app is a full-blown TestClient instance with a full-blown TestWrapper instance attached to the app.wrapper property.

Implementing a strategy to create, read from and destroy `queue` objects

OK great, now we've tied together our EClient and EWrapper classes via this TestApp class. But... so what?

How do I make a request to TWS for my account summary? And after I do, how do I collect that data? Well now that we've connected our wrapper and client via our app object, we can design a flow to answer both of those questions.

Let's be very explicit and continue with the account summary example.

First, we can request our account summary from TWS via the EClient.reqAccountSummary method. How did I know that? Well, the purpose and description of this and all API-provided methods can be found in the ibapi source itself, or in (once again) the official docs.

Looking at the docstring for reqAccountSummary in EClient, we see:

Call this method to request and keep up to date the data that appears on the TWS Account Window Summary tab. The data is returned by accountSummary().

Now here we must discuss a very important point. accountSummary is an EWrapper method. In fact, it's not just a method but an event! That means that accountSummary will be automatically called when TWS is ready to deliver our requested account summary. This is how all of these wrapper methods/events work (openOrder, tickPrice, etc..)

To be 100% clear:

We request our account summary from TWS by running reqAccountSummary() (an EClient method)
TWS will take however much time it needs to gather that data, and when it's ready to send it back to us...
The accountSummary() method (from EWrapper) will be automatically called, containing the data in its method arguments

Again, we do not manually call accountSummary, it's called automatically!

OK so what happens to the data delivered in accountSummary? Let's look at its definition in EWrapper:

def accountSummary(self, reqId:int, account:str, tag:str, value:str, currency:str):
    self.logAnswer(current_fn_name(), vars())

First let's note the function arguments. We're getting back the following data: reqId, account, tag, value and currency (the :int and :str are just type annotations).

Looking at the function body, there's really nothing going on. And that's intentional. Interactive Brokers expects us to overwrite this accountSummary method in TestWrapper to dictate what we want to do with the returned data. This is why the TestWrapper class definition was empty earlier (but now we're going to fill it in!)

Here is where we implement our data flow strategy. The gist of the methodology revolves around two principles:

We control client methods, but we do not control wrapper methods
- Yes, we control what wrapper methods do (we override EWrapper methods with our own definitions in TestWrapper), but we do not control when they run. TWS will fire specific methods (such as EWrapper.accountSummary) automatically.
- However, we do control when client methods will run.
The client part of our app object knows about the wrapper, but the wrapper does not know about the client
- When we instantiated TestApp, we gave our TestClient class an instantiated TestWrapper object.
- As we've already discussed, this means that the client methods and properties on app have access to the wrapper methods/props on app.wrapper, but the reverse is not true. The wrapper methods/props on app.wrapper have no knowledge of the client methods/props on the base app object.

What does this mean for our design? Our client methods should be our command center and contain our conditional checks & flow control. We want this because the client methods/props have supreme visibility across the object and because we control when the methods run. On the other hand our wrapper methods should be dead simple because they have limited visibility across the object and because we do not control when some of them run.

The gameplan to request data from TWS (using a `TestClient` method)

Initialize data storage on the wrapper object (think app.wrapper)
Request data from TWS (think EClient.reqAccountSummary)
- (Auto-run wrapper method puts data into storage -> think TestWrapper.accountSummary)
Retrieve target data from storage on the wrapper
Delete data storage on the wrapper object
Check for errors that may have occurred during this process
Return the retrieved data

For all the visual folks:

In our case we want our account summary. So let's tackle these step-by-step and then bring it all together at the end in one TestClient method definition.

Step 1: Initialize data storage on the wrapper object

First, let's define a method on TestWrapper that will build us an empty queue:

import queue

class TestWrapper(EWrapper):

    def init_accountSummary_queue(self):                                                                              
        self.accountSummary_queue = queue.Queue()

What's a queue? It's just a first-in-first-out (FIFO) list. Why use this instead of a List or something else? Stay tuned!
OK great. Now we have a place on the wrapper to put our account summary data. (As a sanity check, if you instantiate app = TestApp() right now, you'd have access to this method from app.wrapper.init_accountSummary_queue)

Step 2: Request data from TWS

This is achieved by running the correct EClient method. All we have to do here is call reqAccountSummary from EClient (with the appropriate arguments of course)

Step 2b: Auto-run wrapper method puts data into storage

Here's where we define our custom TestWrapper method overriding the default EWrapper.accountSummary

class TestWrapper(EWrapper):

    def accountSummary(
        self, reqId: int, account: str, tag: str, value: str, currency: str
    ):
        """
        Triggered by EClient.reqAccountSummary()
        """
        if hasattr(self, "accountSummary_queue"):
            self.accountSummary_queue.put(
                {
                    "reqId": reqId,
                    "account": account,
                    "tag": tag,
                    "value": value,
                    "currency": currency,
                }
            )

Note that (a) we're keeping it simple and just passing the arguments through to our queue in a dictionary and (b) we only do this if the queue already exists (we do not want to accumulate data that we didn't ask for, in case TWS fires events on its own)

Step 3: Retrieve target data from storage on the wrapper

Here is where queue objects shine, because we can repeatedly try to get something from an (initially) empty queue until it either: (a) gives us something or (b) times out
Quick proof-of-concept

# Init an empty queue
myQ = queue.Queue()

try:
    # Try to remove an item from the queue    
    myItem = myQ.get(timeout=5)
except queue.Empty:
    print("myQ was empty and max time has been reached")

This will of course raise the queue.Empty exception, but if some data were to magically be put into the queue during the five second timeout window... we would get the data and avoid the exception!

Step 4: Delete data storage on the wrapper object

Since the wrapper methods run automatically, we don't want our queue objects to accumulate data sent to us from TWS unprompted. We only want data when we ask for it.
Proof-of-concept

class A():
    def __init__(self):
        self.myQ = queue.Queue()

a = A()
del a.myQ

Step 5: Check for errors that may have occurred during this process

We're going to keep track of errors on the wrapper side, because if you think about it, just like we receive data that we want on TestWrapper, that's also how we receive error messages from TWS too...

class TestWrapper(EWrapper):

    def init_error(self):
        self.my_errors_queue = queue.Queue()

    def is_error(self):
        error_exist = not self.my_errors_queue.empty()
        return error_exist

    def get_error(self, timeout=5):
        if self.is_error():
            try:
                return self.my_errors_queue.get(timeout=timeout)
            except queue.Empty:
                return None
        return None

    def error(self, id, errorCode, errorString):
        errormessage = (
            "IB returns an error with %d errorcode %d that says %s"
            % (
                id,
                errorCode,
                errorString,
            )
        )

Step 6: Return the retrieved data

Well, this one speaks for itself

Bringing it all together into one `TestClient` method

class TestClient(EClient):

    def __init__(self, wrapper):
        EClient.__init__(self, wrapper)
        # Maximum timeout we're comfortable with, in seconds
        self.max_wait_time = 5

    def getAccountSummary(self):
        """
        Runs EClient.reqAccountSummary()
        Returns value from EWrapper.accountSummary()
        """
        # [1] Init a queue on the wrapper
        self.wrapper.init_accountSummary_queue()

        # [2] Request data from TWS
        self.reqAccountSummary(
            9001, "All", "TotalCashValue, BuyingPower, AvailableFunds"
        )

        try:
            # [3] Get data from queue (if it shows up) or eventually timeout
            accountSummary = self.wrapper.accountSummary_queue.get(
                timeout=self.max_wait_time
            )
        except queue.Empty:
            print("accountSummary queue was empty or max time reached")
            accountSummary = None

        # [4] Delete queue from wrapper
        del self.wrapper.accountSummary_queue

        # [5] Check for errors
        while self.wrapper.is_error():
            print("Error:")
            print(self.get_error(timeout=self.max_wait_time)

        # [6] Return data
        return accountSummary

I should also mention that you must use threading for this approach to work- otherwise wrapper events could be blocked by currently executing client functions.

To achieve that, put this in your TestApp definition:

from threading import Thread

class TestApp(TestWrapper, TestClient):

    def __init__(self, ...):
        ...
        thread = Thread(target=self.run)
        thread.start()
        setattr(self, "_thread", thread)

And that should do it! Is this the best way to implement the IB API? Probably not, but it seems to be a fairly reliable way to do so (at least in my thus far limited experience).

Let me know what you think of this design in the comments and finally, thanks for reading!

A short story on npm scripts & killing processes (how to make your tools work for you!)

spongechameleon — Sun, 23 Aug 2020 06:46:42 +0000

The background

I have a project. It has static files. They are:

A HTML file
A Sass file (.scss)
A Typescript file (.ts)

To get my sass to compile into regular .css, I use the node-sass npm package.

To get my typescript to compile into regular .js, I use the tsc compiler.

To get my browser to automatically update whenever one of these files is changed, I use the browser-sync npm package.

How I start up:

Open a terminal. Go to project directory. Start vim.
Open another terminal. Go to project directory. Run tsc & node-sass & browser-sync &
- (Full disclosure: node-sass and browser-sync are run with options, but that's not important)
Go back to vim and edit.

How I close down:

Close vim (but how?!). Close terminal.
Go to the other terminal. Find and kill my processes, one-by-one:
- ps aux | grep node-sass
- Mentally take note of the displayed PID (process ID) from the second column
- kill <PID I definitely always remember perfectly>
- Repeat for tsc and browser-sync

This is clearly a lot of typing and PID-remembering. Nobody likes that. We can do better.

The npm script

My start up command is too much typing. So we’re going to have package.json do the heavy lifting.

“scripts”: {
  “ts”: “tsc”,
  “sass”: “node-sass”,
  “browser”: “browser-sync”,
  “serve”: “npm run ts & npm run sass & npm run browser &”
}

Now, all we have to run is npm run serve and everything is taken care of for us.

OK, but why go through all the trouble of making scripts for each single command? Why not just do:

“scripts”: {
  “serve”: “tsc & node-sass & browser-sync &”
}

The answer to that is all about our friend, &, the ampersand. Anything with an & will run continuously and show up in the output of ps aux. And we intentionally want all of our processes- whatever they really are- to show up as npm processes!

Hence:

The “serve” script that only runs npm commands
The absence of an & in the “ts/sass/browser” scripts

The kill

So here’s why we want all of those tools to show up as npm commands:

ps aux | grep [n]pm | awk ‘{ print $2 }’

This command:

Outputs all processes
Filters by [n]pm (only considers processes with the first letter of “n”)
Grabs the second column/word of output, which in this case is the PID

Now all that’s left is to kill all of those PIDs we collected:

kill $(ps aux | grep [n]pm | awk ‘{ print $2 }’)

Booyah. But… hate typing that? Me too. So, we can either just alias this in our .bashrc as ‘npmkill’ or something…

The generalized, easy to type kill

…or take an extra minute to whip together a quick bash script that:

Accepts a grep search term
Lets us see our processes before we kill them

#!/usr/bin/bash
ps aux | grep $1

read -p ‘kill processes? (y/n): ‘ confirm

if [ $confirm == ‘y’ ]
then
  kill $(ps aux | grep $1 | awk ‘{ print $2 }’)
fi

The $1 refers to the first argument passed to the script, so if we call our script pidkill.sh then running pidkill [n]pm would mean $1 == [n]pm.

The read -p ‘<prompt>: ‘ confirm allows us to (a) enter our response on the same line and (b) saves our response to the $confirm variable.

Then it’s a simple if statement and our shell expansion!

Don’t forget to alias your script in .bashrc if you want to be able to use it anywhere, otherwise you’ll have to be in the directory of the script or type out its path every time.

alias pidkill=’~/pidkill.sh’ does the trick for me.

Thanks for reading and happy coding!

sponge