Forem: Manikanta Inugurthi

Splitwise Automation to send expense updates

Manikanta Inugurthi — Wed, 28 Sep 2022 17:17:23 +0000

Introduction

If you are reading this I assume you know about splitwise application. A one-line summary in their words is "Splitwise is a free tool for friends and roommates to track bills and other shared expenses so that everyone gets paid back"

I use another app called money manager for personal expenses, but instead of jumping between applications I always wanted one app to track all my expenses. I was about to send feedback to Splitwise to add a feature where I can add costs only where I am involved, interestingly enough many users needed this feature and thanks to them the Splitwise team have suggested a way to do it.

All my expenses are in the same app, now what? Although all my expenses are in the same place, Splitwise doesn't have a feature to tell me how much I've spent in the previous month across the application. So the curious person in me wanted to solve this problem, that is how this Side-Project-Sunday is born.

Requirements

Now as the storytime is done, let's list down all the requirements

Ability to get the total expense
- For a given date/date range
- For a given friend(s)/group(s)
Output a pie chart for a given date range
- For Group-wise expenses
- For Category-wise expenses
Send an email on the first day of the month, with the following template

Research

Fetching data

With a quick googling about how we can do it, I found that Splitwise is supported in automation apps like Zapier, make etc.., although with a little bit of effort we can use these apps to fulfil my requirements, I thought it would be better if I code it myself so that I'll have more control on the output.

Further googling landed me on Splitwise dev documentation and I am so happy that there are a few developers who have developed third-party SDKs using the exposed APIs.

As I was planning to write a python script to implement this, I decided to use this python implementation by namaggarwal for my automation.

Email notification

For sending out the email, I wanted to use Gmail's SMTP server to send out the email. A quick search through the web helped me understand how I can do it. I've linked all the references that I've taken help from at the end of the article.

Scheduling

As I was planning to push my script to GitHub, I was thinking to have a GitHub action to run my script on a regular interval. GitHub supports multiple ways of triggering actions, of which one is cron expression. This should help me in running the script at the end of every month.

Implementation

At first, I started exploring the SDK to understand all the APIs I could use to fulfil my requirements.

Setup

First I've created a new application in Splitwise, this generates Consumer Key and Consumer Secret for us which we'll be using to authenticate our calls.

As I plan to use this for a personal use case I've generated an API key, instead of going through the hassle of setting up OAuth1/OAuth2.

Let's play with the API

Let's go through all the APIs that Splitwise supports and list down the methods in which we are interested.

Get all expenses of the current user

This API is so powerful that most of our logic will rely on this, the getExpenses method supports a wide variety of filters with which we can filter the data based on createdDate, updatedDate, groupIds, friendIds etc.., As I'll need the data for the past month, I'll just get all the expenses for the previous month date range. Unfortunately, this API doesn't give us all the details in one network call hence we need to get all the data by making multiple network calls.

s = Splitwise(CONSUMER_KEY, CONSUMER_SECRET, api_key=API_KEY)
expenses = []
current_page_num = 0
page_size = 100

while True:
    curr_page_expenses = s.getExpenses(
        dated_after=start_date,
        dated_before=end_date,
        limit=page_size,
        offset=current_page_num * page_size,
    )
    if len(curr_page_expenses) == 0:
        break
    current_page_num = current_page_num + 1
    expenses = expenses + curr_page_expenses

Get all Groups and Friends

The getGroups and getFriends method fetches us all the friends and group data to us which we can map with the expense data that we've fetched earlier.

groups = s.getGroups()
friends = s.getFriends()

We can traverse through friends' data and filter out friends where we owe them or where we are owed.

def get_friends_with_outstanding_table(friends):
    friends_with_outstanding_balances = []
    for f in friends:
        balance = sum(map(lambda b: float(b.amount), f.balances))
        if balance != 0:
            color = "red" if balance < 0 else "green"
            friends_with_outstanding_balances.append([f.first_name, balance, color])

    friends_with_outstanding_balances.sort(key=lambda f: f[1], reverse=True)
    return friends_with_outstanding_balances

Visualization

When it comes to visualization in python, most of us are familiar with matplotlib. The same is the case with me, so I used matplotlib to generate the charts.

Email notification

As I mentioned earlier, I wanted to use Gmail's SMTP server to send out emails. There are plenty of articles available on how to setup and send out emails using Gmail which I'll link at the end of the article.

Scheduling the script

Let's first create a Github action to run our script and send out the email. Let the name of the job be send-email

jobs:
  send-mail:
    runs-on: ubuntu-latest

    steps:

The first step of our job is to checkout the repository with the default branch. There is already an action available in GitHub Action Marketplace for this.

jobs:
  send-mail:
    runs-on: ubuntu-latest

    steps:    
      - name: Checkout Repos
        uses: actions/checkout@v2.4.2

Next, set up the python environment so that our script can run without any issues. I'll be using setup-python from marketplace to install python and pip.

jobs:
  send-mail:
    runs-on: ubuntu-latest

    steps:

      - name: Checkout Repos
        uses: actions/checkout@v2.4.2

      - name: Setup Python
        uses: actions/setup-python@v4.2.0
        with:
          python-version: 3.8.0
          cache: pip
          architecture: x64
          update-environment: true

Finally, let's install all the required libraries using pip and then run our script.

jobs:
  send-mail:
    runs-on: ubuntu-latest

    steps:

      - name: Checkout Repos
        uses: actions/checkout@v2.4.2

      - name: Setup Python
        uses: actions/setup-python@v4.2.0
        with:
          python-version: 3.8.0
          cache: pip
          architecture: x64
          update-environment: true

      - name: Install required dependencies
        run: python -m pip install -r requirements.txt

      - name: Run Automation
        run: python main.py

As our action is ready to go, let's now try to schedule the action at regular intervals. GitHub supports cron expressions as schedule which we'll be using to schedule our script to run on the 1st of every month.

on:
  push:
    branches: ["main"]
  schedule:
    - cron: 0 10 1 * *

The complete yml file can be found here.

References

Let's discuss commit messages

Manikanta Inugurthi — Sun, 10 Jul 2022 04:30:00 +0000

We all know what a commit message is, and probably write at least 1 commit message every day. But how confident we are that we'll understand the changes we made in a commit by reading only the message after some time?

There are multiple reasons why we don't write proper commit messages:

No proper structure is defined in the team/project
Laziness

There can be a wide variety of other reasons as well.

But wait!

Why should we write better commit messages?

If we browse through the git log of any of our repositories, there is a high chance that it looks something like this,

fixed NPE in test case
CSS fixes
added a button to send mail
updates versions in package.json

Hmm, so what is the fix in the latest commit? Yeah, even I can't tell until I look at the changes in the commit. Perhaps you have encountered code in a professional environment where you had no idea what it was doing or meant for.

Similar to having maintainable code and good documentation, having a good commit log is suggested to save yourself and/or co-workers hours of digging around in the repository.

Anatomy of a commit message

Most of us write only one line of commit message explaining what the changes are by running the following command.

git commit -m "<Message>"

But did you know that multi-line commits are possible?

git commit -m "<Message1>" -m "<Message2>"

Actually, for each -m param to the git commit command it adds a new line to the commit message.

This feature opens a whole new world for us to write commits, as we now know that we aren't blocked by writing about our changes in 1 line, we can have a summary in one line have a detailed explanation in the subsequent lines. The structure of the commitment will come as

git commit -m "<Title>" -m "<Message>" -m "<Footer>"

Pro Tip: Prefer using a text editor when writing commit messages instead of command-line :D

Let's talk about better commit messages

As we now have defined a basic structure of the commit message, let's add some rules along with the structure to have consistency across all of our commits.

These rules work for me, but might not work for you. My goal here is to give you some inspiration so that you can come up with your structure.

Type of Commit

Wait what!!! There are types of commits?

Technically there are no types, but logically we can have any number of types based on our needs. Following are a few of the widely accepted types of commits:

feat Added a new feature
fix Fixed a bug
chore Changes that do not relate to a fix or feature and don't modify src or test files (for example updating dependencies)
refactor refactored code that neither fixes a bug nor adds a feature
docs Updates to docs in the repository
style Format code
test Adding/fixing test cases
perf Performance improvements
ci Continuous Integration related
build Changes that affect the build system or external dependencies
revert Reverts a previous commit

To specify the type of the commit I use emojis like , and 🐛 for the feat, and bug respectively... more emojis for different types of commits can be found here. Specifying the type as text would also work, but I've transitioned to this way of writing commits for 2 reasons:

Git log looks cool 😎
I save some characters in the subject line

Scope of Commit

The scope is a noun that describes what section of the codebase is affected by this change/commit.

Example:

- feat(API) or (API)
- fix(orders) or 🐛(orders)
- docs(workflow) or 📝(workflow)

Subject

Summary of the changes in the commit.

Some key points to consider while writing the subject:

Have a hard limit of 50 characters
There is no need to add punctuation, be mindful of the hard limit we've set ;)
Use the imperative mood

Body

This is an optional section where we can provide a detailed explanation of what the change is, and why we've made the change.

Separate the subject from the body with a blank line
Try having a hard limit in each line of the body(I prefer to have around 70 to 75)

Footer

This section can be used as a meta section of the commit. Here we can specify things like the ticket or issue number that the change corresponds to. reviewer of the commit etc..,

Most of this information is inspired from Conventional Commits and they explain all the specs for writing a good commit in much detail. Do note that there is no one correct way of writing commits but having a team agreement can help maintain the repository easier.

Tools

If you've liked the concept of writing conventional commits and want to force them into your workflow, there are a lot of tools available to help you to do so.

If you prefer using the command line for writing commits, Commitizen would be a good choice for you, when you are about to commit your changes it asks a series of questions and forces you to write descriptive commits.
If you prefer using IDEs like IntelliJ or VS Code there are plugins available. I've used this plugin in IntelliJ and it worked for me, however, I haven't tried any in VSCode.

References

Conventional Commits
Gitmoji - An emoji guide for your commit messages
Commitizen - Command line tool for writing conventional commits
Conventional Commit Plugin - An IntelliJ plugin for writing conventional commits

Logging best practices that I follow

Manikanta Inugurthi — Sun, 10 Apr 2022 04:38:53 +0000

At the beginning of my coding career, I mostly wrote logs (print statements 😜) when debugging an issue and removed them (unsaid rule) after the bug was fixed. But when I became a backend developer, I realised the importance of logging. In this blog, I am listing down all the things that I've learned in the past few years and the rules I follow when writing log statements.

What exactly is logging?

In laymen's terms, we all know what logging is; it's printing messages, variables and exceptions to the console or a file. It is partially correct. However, what I feel is logging is a mechanism to record the state of the application and the actions it has taken at a given point in time.

If we think about it, both definitions are completely different. One suggests that we are indicating where we are in the codebase, and the second indicates the current state of request flow. This change in the thought process when developing an application helps us to write better logging statements and, in turn, helps us to improve the application.

Different levels of Logs

The level of logging may vary based on the team you're in, the application you're working on, and, most likely, the framework/language you're using. The most widely used logging levels are DEBUG, INFO, WARN and ERROR.

DEBUG 👨🏻‍💻

As the name suggests, these logs contain information that is most interesting for developers when trying to debug a problem. They are fine-grained statements concerning the state of the application.

In our production system, we have a huge number of DEBUG logs, and we tend to disable this level mostly. We enable this only when the need arises.

INFO ℹ️

Logs at this level are purely informational messages, as the name suggests and they should not be used to indicate a fault or error state in the application. To make the best use of this log level, consider what general information would be useful for diagnosing an application error when the primary interface is unavailable.

WARN ⚠️

Warning logs should be used when something happens that could potentially cause some oddities in the application. These logs give us some heads-up, indicating something that might need our attention.

ERROR 🚨

Logs at this level contain failures in the application. Use error logs when something that could've gone wrong has gone wrong and action needs to be taken by someone to resolve the issue. In the event there is no action needed, we can use the WARN level instead.

How much should we log?

Have fewer logs 🔇

There is no harm in actually having very few logs in our system, but this will make figuring out why something happened the way it happened a bit difficult.

Log everything 🔊

If logging is this good, why can't we write a log statement after every line of our code?

Is it too much?

How much logging is too much logging?

These types of questions were asked in a lot of forums, including StackOverflow, which led to this blog by Coding Horror

A few takeaways, according to me, are as follows:

The more logs you add, the larger the codebase grows.
There is a slight chance of your application becoming slower by having too many logs.

There is no one golden standard in the case of how much we need to log. It all depends on the application and who is writing the code. I suggest starting logging ERROR and WARN levels, and then INFO and DEBUG level logs as the need arises during development. Be mindful of what is being logged.

Other uses of logging

Having good logs helps us to debug and resolve the issue better and faster.
Structured logs help us visualise logs in viewers like Kibana, Datadog, etc. These visualisations help us derive a few patterns in our application, which in turn can be used to make the application better.

Rules I follow while writing logs 🪵

So we've read about what logging is, the different levels of logs, and the advantages of having good logs. The following are the rules and guidelines that I follow on a day-to-day basis when writing logs. It might not be exhaustive, but it should give a very good idea of the different things to consider while writing logs.

Not all exceptions are errors.

I used to have the habit of logging the exception messages and categorising them as errors. Later down the line, I realised that not all exceptions are errors, and some might be expected. These can be logged as either warnings or informational messages.

Log after an action is performed, not before.

Logging before an action is taken might give you some contextual information about what action would be taken, but we would never know whether the action was successful or not. Having a log message after the action is taken would inform us that the action was performed and the result of the action, if available. There are scenarios where we log before and after the action is taken. This is done to measure the amount of time it takes to complete the action.

Either log or throw exceptions, but not both.

As mentioned in RSPEC-2139, it is not a good idea to log and throw exceptions at the same time. This is done to avoid a large number of identical logs, which would lead to noise in the log files.

Define a logging format for your project.

Having a predefined structure for log messages would make reading the logs easier. Additionally, this structure would help to visualise the logs in log viewers like Kibana, Datadog, etc...

Try not to merge log messages and variables when writing the logs. A good log format that I follow is

LOG.level("{Message} {Variables}");

Some examples:

LOG.info("Fetched the employee data for a company. [companyId={}]", companyId);
LOG.error("Error occurred while fetching the employee data for the company. [companyId={}]", companyId, e);

References

Coding Horror blog on logging
Code project blog on The Art of Logging

How-to: Python Paramiko

Manikanta Inugurthi — Sun, 06 Mar 2022 18:00:19 +0000

Background

I recently had a task that required me to connect to an SFTP server, download particular files (if they existed), and then process the contents. We were given the option of using any language to fulfil the criteria, so I chose Python to tackle the problem. This blog outlines the problems I encountered when attempting to build the script, as well as a tutorial on how to use the library.

Why not an alternative lib something like `pysftp`

This StackOverflow answer lists the differences between the libraries.

Requirements

Verify Python

We need to have some version of python installed in our machine to use the library, in my case I am currently using python 3.8.

Install Paramiko

Run the following command in the terminal to install the paramiko library globally

pip install paramiko

SFTP Server

It is suggested that we have an SFTP server up and running in order to test the script. Our server can assist us in configuring the setup to our liking.

If you do not want to set up or have an SFTP server to test on, we can utilise several publicly available servers. A list of servers may be found here.

How to use the library

Now let's start discussing how to use the library. We'll be talking about the following in upcoming sections.

Connecting to the remote server
Executing commands on the server
Transferring files (Upload/Download)
The final script

Connecting to the remote server

First let's define the constants for connection properties like hostname, username, password, public key, private key etc.., In a real-world application, these properties should come from a configuration file/system.

# There might be others based on the SFTP configuration
HOST_NAME = "test.rebex.net"
USER_NAME = "demo"
PASSWORD = "password"
PRIVATE_KEY_FILE = '/some/file/path/private_key.txt'
PORT = 22

To connect to the server we create an instance client of paramiko.SSHClient(). Paramiko.SSHClient is the primary class used to make connections to the remote server and execute commands. This class wraps Transport, Channel, and SFTPClient to take care of most(not all as discussed below) aspects of authenticating and opening channels. The creation of an SSHClient object allows establishing server connections via the connect() method.

Password-based auth

client = paramiko.SSHClient()
client.connect(hostname=HOST_NAME, port=PORT, username=USER_NAME, password=PASSWORD)

connect() method has support for a wide variety of ways to authenticate before connecting to the server for a user. The above-mentioned method is for password-based authentication.

For private key-based auth, instead of passing the password parameter, we can instead pass pkey an object of type PKey or key_filename the list of private key file names.

Private key-based auth

# Passing the PKey object
client = paramiko.SSHClient()
pkey = paramiko.RSAKey.from_private_key_file(PRIVATE_KEY_FILE)
client.connect(hostname=HOST_NAME, port=PORT, username=USER_NAME, pkey=pkey)

# Passing private key file path
client = paramiko.SSHClient()
client.connect(hostname=HOST_NAME, port=PORT, username=USER_NAME, key_filename=PRIVATE_KEY_FILE)

However, connect() method doesn't support 2-Factor authentication. Following are a few examples where the above method might not work.

SFTP server is enabled with both password and private key-based auth.
SFTP server needs an OTP post entering the password to login into the server.

For use cases like these instead of creating an object of Paramiko.SSHClient we can create Paramiko.Transport instead. Transport method has a support to authenticate the connection after successful negotiation by the methods like auth_password, auth_publickey, auth_interactive etc..,

Password and Private key-based auth

Following are a few sample code snippets for the usage of above mentioned methods

transport = paramiko.Transport()
transport.start_client()
transport.auth_password(USER_NAME, PASSWORD)
pkey = paramiko.RSAKey.from_private_key_file(PRIVATE_KEY_FILE)
transport.auth_publickey(USER_NAME, pkey)

Password and OTP based auth

Here we use auth_interactive method which accepts a handler for passing the values to the prompt. The custom handler that we write has the logic to pass the OTP or Password etc.. based on the output in the prompt. If we want to get the input to the handler from stdin instead then we can use the auth_interactive_dumb method.

transport = paramiko.Transport()
transport.start_client()
transport.auth_interactive(username, handler)

def handler(self, title, instructions, prompt_list):
        answers = []
        for prompt_, _ in prompt_list:
            prompt = prompt_.strip().lower()
            if prompt.startswith('password'):
                answers.append(PASSWORD)
            elif prompt.startswith('verification'):
                answers.append(OTP)
            else:
                raise ValueError('Unknown prompt: {}'.format(prompt_))
        return answers

Executing commands on the server

After we have successfully logged into the server by any one of the above-mentioned methods executing the commands is as simple as passing your command to exec_command.

command = "ls -l"
stdin, stdout, stderr = client.exec_command(command)
print(stdout.read())

However paramiko has some very useful util methods to run SFTP commands on the server, following are a few of them

sftp_client = client.open_sftp()
sftp_client.chdir() # Changing the directory
sftp_client.cwd() # Get the current working directory
sftp_client.listdir() # List all the contents of the current working directory

For other supported commands, we can refer to this documentation.

File Transfers

Paramiko allows to programmatically send and receive files using the SFTP protocol. To upload/download file we need to create an SFTPClient session object by calling open_sftp(). This object allows to perform common SFTP operations like get(), put().

Upload File

sftp_client= client.open_sftp()
sftp_client.put(uploadlocalfilepath,uploadremotefilepath)
sftp_client.close()

Download File

sftp_client= client.open_sftp()
sftp_client.get(downloadremotefilepath,downloadlocalfilepath)
sftp_client.close()

The final script

Navigate to this link for the gist of the following script. Star ⭐️ the gist if it helped you.

References

Official Paramiko documentation can be a good start for exploring other features of the library
pysftp vs paramiko
SSH using Python Paramiko

Forem: Manikanta Inugurthi

Splitwise Automation to send expense updates

Introduction

Requirements

Research

Fetching data

Email notification

Scheduling

Implementation

Setup

Let's play with the API

Get all expenses of the current user

Get all Groups and Friends

Visualization

Email notification

Scheduling the script

References

Let's discuss commit messages

Why should we write better commit messages?

Anatomy of a commit message

Let's talk about better commit messages

Type of Commit

Scope of Commit

Subject

Body

Footer

Tools

References

Logging best practices that I follow

What exactly is logging?

Different levels of Logs

DEBUG 👨🏻‍💻

INFO ℹ️

WARN ⚠️

ERROR 🚨

How much should we log?

Have fewer logs 🔇

Log everything 🔊

Other uses of logging

Rules I follow while writing logs 🪵

Not all exceptions are errors.

Log after an action is performed, not before.

Either log or throw exceptions, but not both.

Define a logging format for your project.

References

How-to: Python Paramiko

Background

Why not an alternative lib something like pysftp

Requirements

Verify Python

Install Paramiko

SFTP Server

How to use the library

Connecting to the remote server

Password-based auth

Private key-based auth

Password and Private key-based auth

Password and OTP based auth

Executing commands on the server

File Transfers

Upload File

Download File

The final script

References

Why not an alternative lib something like `pysftp`