Forem: Aly Sivji

Testing 101: Introduction to Testing

Aly Sivji — Mon, 22 Oct 2018 18:43:24 +0000

This post was originally published on Siv Scripts

Programming is writing code to solve problems. Software Engineering is the practice of using a structured process to solve problems. As engineers, we want to have a codebase we can change, extend, and refactor as required. Tests ensure our program works as intended and that changes to the codebase do not break existing functionality.

At my last job, I worked with a Senior Engineer to build out a microservices-based backend to replace our existing Django monolith. It was a greenfield project and we were encouraged to try new things. I was reading Python Testing with pytest and convinced the Senior Engineer to let me bring pytest into our project. This was fortuitous as it forced me to take the lead in writing the initial set of tests we used as a template for all of our services.

This experience reinforced the principles highlighted in The Pragmatic Programmer. It's about being pragmatic in what we test, how we test, and when we test; we should leverage tools and techniques that allow us to test our code as efficiently as possible. Testing needs to be easy and free of barriers; once testing feels like a chore, programmers won't do it... and this is how software quality slips.

We dread going into the code because either there are no tests or the tests that exist are so brittle that we're forced to rewrite tests as we write code. This is not what Software Engineering is about. Test should enable refactoring, not hamper our ability to make changes to the codebase. We should spend our time writing business logic, not wrestling with tests.

Testing is folklore in the sense that best practices and techniques are passed down from programmer to programmer while working on projects as part of a team. If you are new to the industry and are trying to grok testing, it's hard to figure out how to get started. It feels like there is a lot of conflicting advice out there, and that's because there is. Testing is opinionated, more-so than any other software engineering discipline. Folks are always arguing about what to test, how to test, and especially when to test.

This is the first in a series of posts that details my thought process for how I go about adding tests to a codebase. In this post, I provide a broad introduction to the world of testing so we can have a common vocabulary for future posts.

What is Testing
Benefits of Testing
Black Box vs White Box
Test Pyramid
Structuring Tests
What to Test
When to Write Tests
Conclusion

What is Testing

When we write code, we need to run it to ensure that it is doing what we expect it to. Tests are a contract with our code: given a value, we expect a certain result to be returned.

Running tests can be thought of as a feedback mechanism that informs us if our program works as intended:

While passing tests cannot prove the absence bugs, they do inform us that our code is working in the manner defined by the test. In contrast, a failing test indicates that something is not right. We need to understand why our test failed so we can modify code and/or tests, as required.

Properties of Tests

1. Fast

Tests give us confidence that our code is working as intended. A slower feedback loop hampers development as it takes us longer to find out if our change was correct. If our workflow is plagued by slow tests, we won't be running them as often. This will lead to problems down the line.

2. Deterministic

Tests should be deterministic, i.e. the same input will always result in the same output. If tests are non-deterministic, we have to find a way to account for random behavior inside of our tests.

While there is definitely non-deterministic code in production (i.e. Machine Learning and AI), we should try to make all our non-probabilistic code as deterministic as possible. There is no point of doing additional work unless our program requires it.

3. Automated

We can confirm our program works by running it. This could be manually running a command in the REPL or refreshing a webpage; in both cases, we are looking to see if our program does what it is supposed to do. While manual testing is fine for small projects, it becomes unmanageable as our project grows in complexity.

By automating our test suite, we can quickly verify our program works on-demand. Some developers even have their tests triggered to run on file save.

Formal Definition

Let's go over some definitions so we have a common vocabulary going forward.

A System Under Test (SUT) is the entity that is currently being tested. This could be a line of code, a method, or an entire program.

Acceptance Criteria refers to the check we perform that allows us to accept output from the system. The specificity and range of acceptance criteria depends on what we are testing: medical device and aerospace require tests to be specific as there is a lot less room for error.

If Amazon makes a bad recommendation, it's not the end of the world. If IBM's Watson suggests the wrong surgery, it can be life-threatening.

Testing refers to the process of entering Inputs into our System Under Test and validating Outputs against our Acceptance Criteria:

If output is okay, our test passes.
If output is not okay, our test fails and we have to debug.

Hopefully the test failure provides enough contextual information for us to find out where to look.

Benefits of Testing

A well-thought-out testing strategy paired with thorough test cases provides the following benefits:

Modify Code with Confidence

If a program does anything of interest, it has interactions between functions, classes, and modules. This means a single line change can break our program in unexpected ways. Tests give us confidence in our code. By running our tests after we modify our code, we can confirm our changes did not break existing functionality as defined by our tests.

In contrast, modifying a code base without tests is a challenge. There is no way of knowing if things are working as intended. We are programming by the seat of our pants, which is quite a risky proposition.

Identify Bugs Early

Bugs cost money. How much depends on when you find them.

Fixing bugs gets more expensive the further you are in the Software Development Life Cycle (SDLC). True Cost of a Software Bug digs into this issue.

Improve System Design

This one is a bit controversial, but I think writing code with tests in mind improves system design. A thorough test suite shows that the developer has actually thought about the problem in some depth. Writing tests forces you to use your own API; this hopefully results in a better interface.

All projects have time constraints and it's quite easy to get into the habit of taking shortcuts that increase coupling between modules leading to complex interdependencies. We have to be cognizant of solving problems with spaghetti code.

Knowing we have to test our code forces us to write modular code. If something is clunky to test, there might be a better interface we can implement. Taking the time to write tests forces mindfulness upon us; we take a deep breath before looking at the problem from the perspective of a user.

Once you write testable code by using patterns like dependency injection, you'll see how adding structure makes it easier to verify our code is doing what we expect it to.

Black Box vs White Box

Tests can be broadly classified into two broad categories: black box testing and white box testing.

Black Box Testing refers to testing techniques in which the tester cannot see the inner workings of the item being tested.

White Box Testing is the technique in which the tester can see the inner workings of the item being tested.

As developers, we perform white box testing. We wrote the code inside of the box and know how to test it thoroughly. This is not to say that there is not a need for black box testing, we should still have somebody perform testing at a higher level; proximity to the code can lead to blind spots in our tests.

Test Pyramid

The Automated Test Pyramid provides guidance on how to structure our testing strategy. It says we should write lots of fast and cheap unit tests and a small number of slow and expensive end-to-end tests.

The Test Pyramid is not a hard and fast rule, but it provides a good place to start thinking about a testing strategy. A good rule of thumb is to write as many tests at each level as you need to have confidence in your system. We should be writing tests as we write code, iterating towards a testing strategy that works for the project we are working on.

Unit Tests

Unit tests are low-level tests that focus on testing a specific part of our system. They are cheap to write and fast to run. Test failures should provide enough contextual information to pinpoint the source of the error. These tests are typically written by developers during the Implementation phase of the Software Development Life Cycle (SDLC).

Unit tests should be independent and isolated; interacting with external components increases both the scope of our tests and the time it takes for tests to run. As we will see in a future post, replacing dependencies with test doubles results in deterministic tests that are quick to run.

How big should our unit test be? Like everything else in programming, it depends on what we are trying to do. Thinking in terms of a unit of behavior allows us to write tests around logical blocks of code.

The Test Pyramid recommends having a lot of unit tests in our test suite. These tests give us confidence that our program works as expected. Writing new code or modifying existing code might require us to rewrite some of our tests. This is standard practice, our test suite grows with our code base.

Try to be cognizant of our test suite growing in complexity. Remember, code that tests our production code is also production code. Take the time to refactor your tests to ensure they are efficient and effective.

Unit Test Example

Suppose we have the following function that takes a list of words and returns the most common word and the number of occurrences of that word:

def find_top_word(words)
    # Return most common word & occurrences
    word_counter = Counter(words)
    return word_counter.most_common(1)[0]

We can test this function by creating a list, running the find_top_word function over that list and comparing the results of the function to the value we expect:

def test_find_top_word():
    words = ["foo", "bar", "bat", "baz", "foo", "baz", "foo"]
    result = find_top_word(words)

    assert result[0] == "foo"
    assert result[1] == 3

If we ever wanted to change the implementation of find_top_words, we can do it without fear. Our test ensures that the functionality of find_top_word cannot change without causing a test failure.

Integration Tests

Every complex application has internal and external components working together to do something interesting. In contrast to units tests which focus on individual components, integration tests combine various parts of the system and test them together as a group. Integration testing can also refer to testing at service boundaries of our application, i.e. when it goes out to the database, file system, or external API.

These tests are typically written by developers, but they don't have to be. By definition, integration tests are larger in scope and take longer to run than unit tests. This means that test failures require some investigation: we know that one of the components in our test is not working, but the failure's exact location needs to be found. This is in contrast to unit tests which are smaller in scope and indicate exactly where things have failed.

We should try to run integration tests in a production-like environment; this minimizes the chance that tests fail due to differences in configuration.

Integration Test Example

Suppose we have the following function that takes in a URL and a tuple of (word, occurrences). Our function creates a records and saves it to the database:

def save_to_db(url, top_word):
    record = TopWord()
    record.url = url
    record.word = top_word[0]
    record.num_occurrences = top_word[1]

    db.session.add(record)
    db.session.commit()

    return record

We test this function by passing in known information; the function should save the information we entered into the database. Our test code pulls the newly saved record from the database and confirms its fields match the input we passed in.

def test_save_to_db():
    url = "http://test_url.com"
    most_common_word_details = ("Python", 42)

    word = save_to_db(url, most_common_word_details)

    inserted_record = TopWord.query.get(word.id)
    assert inserted_record.url == "http://test_url.com"
    assert inserted_record.word == "Python"
    assert inserted_record.num_occurrences == 42

Notice how this is the kind of testing we do manually to confirm things are working as expected. Automating this test saves us from having to repeatedly check this functionality each time we make a change to the code.

End-to-End

End-to-end tests check to see if the system meets our defined business requirements. A common test is to trace a path through the system in the same manner a user would experience. For example, we can test a new user workflow: simulate creating an account, "clicking" the link in the activate email, logging-in for the first time, and interacting with our web application's tutorial modal pop-up.

We can conduct end-to-end tests through our user interface (UI) by leveraging a browser automation tool like Selenium. This creates a dependency between our UI and our tests, which makes our tests brittle: a change to the front-end requires us to change tests. This is not sustainable as either our front-end will become static or our tests will not be run.

A better solution is to test the subcutaneous layer, i.e. the layer just below our user interface. For a web application, this would be testing the REST API, both sending in JSON and getting JSON out.

Our subcutaneous tests are our contracts with our front-end; they can be used by our front-end developers as a specification of the REST API. Tools, like swagger-meqa, that are built on top of the OpenAPI Specification can help us automate this process. We could also full-featured tools like Postman to test, debug, and validate our API.

End-to-end tests are considered black box as we do not need to know anything about the implementation in order to conduct testing. This also means that test failures provide no indication of what went wrong; we would need to use logs to help us trace the error and diagnose system failure.

End-to-End Test Example

Here we are using the Flask Test client to run subcutaneous testing on our REST API. There are a lot of things happening behind the scene and the result we get back (HTTP status code) lets us know that the test either passed or failed.

def test_end_to_end():
    client = app.test_client()

    body = {"url": "https://www.python.org"}
    response = client.post("/top-word", json=body)

    assert response.status_code == HTTPStatus.OK

Resources

Martin Fowler Wiki: TestPyramid | UnitTest | IntegrationTest | EndToEndTest
Google Testing Blog: Just Say No to More End-to-End Tests
Automation Panda: The Testing Pyramid

Structuring Tests

Each test case can be separated into the following phases:

setting up the system under test (SUT) to the environment required by the test case (pre-conditions)
performing the action we want to test on SUT
verifying if the expected outcome occurred (post-conditions)
tearing down SUT and putting the environment back to the state we found it in

There are two widely used frameworks for structuring tests: Arrange-Act-Assert and Given-When-Then.

Arrange-Act-Assert (AAA)

The AAA pattern is abstraction for separating the different part of our tests:

Arrange all necessary pre-conditions
Act on the SUT
Assert that our post-conditions are met

Arrange-Act-Assert Example

def test_find_top_word():
    # Arrange
    words = ["foo", "bar", "bat", "baz", "foo", "baz", "foo"]

    # Act
    result = find_top_word(words)

    # Assert
    assert result[0] == "foo"
    assert result[1] == 3

The clear separation between the phases allows us to see if our test method is trying to test too many different things at once. Arrange-Act-Assert is the pattern I use when writing tests.

Given-When-Then (GWT)

GWT provides a useful abstraction for separating the different phases of our test:

Given a set of pre-conditions
When we perform an action on the SUT
Then our post-conditions should be as follows

GWT is widely used in Behavior Driven Development (BDD).

Given-When-Then Example

def test_find_top_word():
    # Given a list of word
    words = ["foo", "bar", "bat", "baz", "foo", "baz", "foo"]

    # When we run the function over the list
    result = find_top_word(words)

    # Then we should see `foo` occurring 3 times
    assert result[0] == "foo"
    assert result[1] == 3

Resources

Gerard Meszaros: Four-Phase Test
Martin Fowler Wiki: GivenWhenThen
C2 Wiki: Arrange Act Assert
James Cooke: Arrange Act Assert Pattern for Python Developers
Automation Panda: Behavior-Driven Python

What to Test

In order to prove that our program is correct, we have to test it against every conceivable combination of input values. This type of exhaustive testing is not practical so we need to employ testing strategies that allow us to select test cases where errors are most likely to error.

Seasoned developers can balance writing code to solve business problems with writing tests to ensure correctness and prevent regression. Finding this balance and knowing what to test can feel more like an art than a science. Fortunately, there are a few rules of thumb we can follow to make sure our testing is thorough.

Functional Requirements

We want to make sure that all relevant requirements have been implemented. Our test cases should be detailed enough to check business requirements. There is no point building something if doesn't it meet the criteria you set forth.

Basis Path Testing

We have to test each statement at least once. If the statement has a conditional (if or while), we have to vary our testing to make sure we test all branches of the conditional. For example, if we have the following code:

if x > 18:
    # statement1
elif 18 >= x >= 35:
    # statement2
else:
    # statement3

To make sure we hit all branches of the above conditional, we need to write the following tests:

x < 18
18 <= x <= 35
x > 35

Equivalence Partitioning

Two test cases that result in the same output are said to be equivalent. We only require one of the test cases in order to cover that class of errors.

Boundary Analysis

"There are 2 hard problems in Computer Science: cache invalidation, naming things, and off-by-1 errors."

This is one of the oldest jokes in programming, but there is a lot of truth behind it, we often confuse if we need a < or a <=. This is why we should always test the boundary conditions. Given the following example:

if x > 18:
    # statement1
else:
    # statement2

To ensure we thoroughly test the boundary conditions of the code snippet above, we would to have test cases for x=17, x=18, and x=19. Be aware that writing test cases becomes more complicated if our boundary has compound conditionals.

This is a great guide on testing boundary conditions.

Classes of Bad Data

This refers to any of the the following cases:

Too little data (or no data)
Too much data
Invalid data
Wrong size of data
Uninitialized data

Data Flow Testing

Focuses on tracing the control flow of the program with a focus on exploring the sequence of events related to the status of data objects. For example, we get an error if we try to access a variable that has been deleted. We can use Data Flow testing to come up with additional test cases for variables that have not be tested by other tests.

Error Guessing

Past experience provides insights into parts of our code base that can lead to errors. Keeping a record of previous errors can improve the likelihood that you will not make that same mistake again in the future.

Recap

Figuring out what to test and doing it efficiently is what I mean when I say Art of Developer Testing. The only way to get better at testing is by writing tests, coming up come up better testing strategies, and learning about different testing techniques. Just like in software development, the more you know about something, the better you will become at it.

When to Write Tests

While there is a lot of interesting discussion about when to write tests, I feel it takes away from the point of testing. It doesn't matter when you write tests, it just matters that you write tests.

If you are interested in exploring this topic, I recommend the following links:

RubyOnRails creator David Heinemeier Hansson (DHH) criticizing TDD at RailsConf 2014
Is TDD Dead? discussion with DHH, Martin Fowler, and Kent Beck
TDD: A Academic Survey by Ted M. Young

Conclusion

In this post, we got a broad introduction to the world of testing. Now that we are all on the same page, we can explore testing in more depth in future posts.

Additional Resources

Code Complete Chapter 22: Developer Testing
Code Simplicity: The Philosophy of Testing
Katy Huff: Python Testing and Continuous Integration
Test and Code Podcast

Finding Common Security Issues in Python Code with Bandit

Aly Sivji — Thu, 20 Sep 2018 11:55:47 +0000

This post was originally published on Siv Scripts

Bandit is a static analysis tool that can find common security issues in Python code. The command line utility scans .py files and generates a report detailing issues by confidence and severity.

I ran Bandit on a few of my repositories and found I was using md5 to hash passwords in one of my side projects. It was a hack with a TODO fix comment, but I had forgotten about it. Thanks to Bandit, I have changed my password hashing algorithm to bcrypt2. Also learned about PassLib as I was researching how to salt and hash passwords.

The Rackspace blog has a post on Getting Started with Bandit that I recommend checking out.

Implementing a Plugin System in Python

Aly Sivji — Mon, 10 Sep 2018 18:12:26 +0000

This post was originally published on Siv Scripts

In July, I released my first open source project. It's an apispec plugin that generates OpenAPI Specification (aka Swagger docs) for Falcon web applications.

Apispec's design made it easy to extend core functionality for a specific use case. I extended the apispec.BasePlugin class, overrode a couple of methods, and I was done. Had to dig into apispec internals to figure out how things were wired together, but it was easy to build upon what was already there

This got me thinking about how to implement a plugin system in one of my own applications. Creating a plugin architecture? It sounded hard. An advanced computer science concept, if you will.

I mulled over a few different implementations based on software I had previously used. I realized this wasn't a difficult problem. Like everything else in programming, once we deconstruct the problem into smaller chunks, we can reason about implementation details clearly.

We assume things are more difficult than they appear. This is especially true for problems we have not seen before. Take a step back. Breathe. Break the problem down into smaller pieces. You got this.

In this Quick Hit, we will walk through the implementation of a simple plugin system.

Background

A plugin is a software component that adds a specific feature to an existing computer program. When a program supports plug-ins, it enables customization (Wikipedia)

There are many benefits to building apps with a plugin framework:

3rd party developers can create and extend upon your app
new features are easier to developer
your application becomes smaller and easier to understand

Sample Application Flow

We have a program that starts, does a few things, and exits.

Plugin Architecture Workflow

We refactored our Business Logic into a Plugin Framework that can run registered plugins. The plugins will need to meet the specifications defined by our framework in order to run.

Toy Example

Let's implement a toy example where we have a plugin framework to print to the console. Our project will have the following structure:

.
├── internal.py  # internal business logic
├── external.py  # contains user-created plugins
└── main.py      # initialize and run application

Internal

This module contains the application class and an internal plugin.

# internal.py

class InternalPrinter:
    """Internal business logic"""
    def process(self):
        print("Internal Hello")


class MyApplication:
    """First attempt at a plugin system"""
    def __init__(self, *, plugins: list=list()):
        self.internal_modules = [InternalPrinter()]
        self._plugins = plugins

    def run(self):
        print("Starting program")
        print("-" * 79)

        modules_to_execute = self.internal_modules + self._plugins
        for module in modules_to_execute:
            module.process()

        print("-" * 79)
        print("Program done")
        print()

External

# external.py

class HelloWorldPrinter:
    def process(self):
        print("Hello World")


class AlohaWorldPrinter:
    def process(self):
        print("Aloha World")

Main

In this module, we run instances of our application with the external plugins we want to enable.

# main.py

from internal import MyApplication
from external import HelloWorldPrinter, AlohaWorldPrinter


if __name__ == "__main__":
    # Run with one plugin
    app = MyApplication(plugins=[HelloWorldPrinter()])
    app.run()

    # Run with another plugin
    app = MyApplication(plugins=[AlohaWorldPrinter()])
    app.run()

    # Run with both plugins
    app = MyApplication(plugins=[HelloWorldPrinter(), AlohaWorldPrinter()])
    app.run()

Discussion

The Application's plugin framework works for both internal and external plugins. We define internal plugins in the application code. External plugins are initialized and passed into the application at runtime.

Each plugin inherits from the base Python object and has a process() method. Nothing complex, want to keep this example as simple as possible.

We can run our plugins by calling the application's run() method. This method loops over all the plugins and calls each instance's process() function. As we see from the output above, the plugins are executed in the same order as the list.

Running Toy Application

$ python main.py
Starting program
-------------------------------------------------------------------------------
Internal Hello
Hello World
-------------------------------------------------------------------------------
Program done

Starting program
-------------------------------------------------------------------------------
Internal Hello
Aloha World
-------------------------------------------------------------------------------
Program done

Starting program
-------------------------------------------------------------------------------
Internal Hello
Hello World
Aloha World
-------------------------------------------------------------------------------
Program done

Real World Example

This pattern can be used in conjunction with the Adapter pattern to simplify application development.

Let's say we have a large number of external clients we want to interface with. Each API is different, but the tasks we need to perform for each client are the same.

One possible implementation of this is to write an adapter around each of the client APIs, resulting in a common interface. Next, we can leverage the plugin framework to solve our business problem, and then we can use plugins to make it work for all of our clients.

This is a very high level description of the solution. I leave implementation as an exercise to the reader.

Additional Resources

A Simple Plugin Framework
Plugin libraries

PHP + MySQL using Docker Compose

Aly Sivji — Tue, 27 Feb 2018 02:04:20 +0000

This post was originally published on Siv Scripts

Being a Software Engineer isn't just about being effective in a specific programming language, it's about being able to solve any given problem using the tools at hand.

This week at work I have to extend the functionality of a WordPress plug-in so it can fit into our microservices-based backend architecture. This means learning enough PHP to write some production-grade code.

I don't want to install PHP on my local machine so this is the perfect use case for Docker! In this Quick Hit, I will describe how to create a containerized PHP + MySQL development environment using Docker Compose.

The LAMP Stack is back!

Instructions

We'll start by creating a folder for this project:

mkdir lamp-stack && cd lamp-stack
Create another subdirectory, /php which contains the following index.php file:

<!-- ./php/index.php -->

<html>
    <head>
        <title>Hello World</title>
    </head>

    <body>
        <?php
            echo "Hello, World!";
        ?>
    </body>
</html>

Populate docker-compose.yml with the following configuration:

# ./docker-compose.yml

version: '3'

services:
  db:
    image: mysql:5.7
    environment:
      MYSQL_ROOT_PASSWORD: my_secret_pw_shh
      MYSQL_DATABASE: test_db
      MYSQL_USER: devuser
      MYSQL_PASSWORD: devpass
    ports:
      - "9906:3306"
  web:
    image: php:7.2.2-apache
    container_name: php_web
    depends_on:
      - db
    volumes:
      - ./php/:/var/www/html/
    ports:
      - "8100:80"
    stdin_open: true
    tty: true

Our directory structure should look as follows:

$ tree
.
├── docker-compose.yml
└── php
    └── index.php

Execute docker-compose up -d in the terminal and load http://localhost:8100/ in your browser.

Time to learn some P H P!

Notes

We use port-forwarding to connect to the inside of containers from our local machine.
- webserver: http://localhost:8100
- db: mysql://devuser:devpasslocalhost:9906/test_db
Our local directory, ./php, is mounted inside of the webserver container as /var/www/html/
- The files within in our local folder will be served when we access the website inside of the container

Data Science Workflows Using Docker Containers

Aly Sivji — Mon, 27 Nov 2017 18:22:32 +0000

I gave a talk last month on how to incorporate Docker into Data Science projects.

Slides are available at http://bit.ly/docker-for-data-science
Link to my blog: https://alysivji.github.io

Interactive, Web-Based Dashboards in Python

Aly Sivji — Thu, 16 Nov 2017 00:18:11 +0000

This post was originally published on Siv Scripts

Summary

Explore Plotly's new Dash library

Discuss how to structure Dash apps using MVC

Build interactive dashboard to display historical soocer results

I spent a good portion of 2014-15 learning JavaScript to create interactive, web-based dashboards for a work project. I wrapped D3.js with Angular directives to create modular components that were used to visualize data.

Data Analysis is not one of JavaScript's strengths; most of my code was trying to cobble together DataFrame-esque operations with JSON data. I missed R. I missed Python. I even missed MATLAB.

When I found Dash a couple of months ago, I was blown away.

With Dash, we can create interactive, web-based dashboards with pure Python. All the front-end work, all that dreaded JavaScript, that's not our problem anymore.

How easy is Dash to use? In around an hour and with <100 lines of code, I created a dashboard to display live streaming data for my Data Science Workflows using Docker Containers talk.

Dash is a powerful library that simplifies the development of data-driven applications. Dash enables Data Scentists to become Full Stack Developers.

In this post we will explore Dash, discuss how the Model-View-Controller pattern can be used to structure Dash applications, and build a dashboard to display historical soccer results.

Dash Overview

Dash is a Open Source Python library for creating reactive, Web-based applications

Dash apps consist of a Flask server that communicates with front-end React components using JSON packets over HTTP requests.

What does this mean? We can run a Flask app to create a web page with a dashboard. Interaction in the browser can call code to re-render certain parts of our page.

We use the provided Python interface to design our application layout and to enable interaction between components. User interaction triggers Python functions; these functions can perform any action before returning a result back to the specified component.

Dash applications are written in Python. No HTML or JavaScript is necessary.

We are also able to plug into React's extensive ecosystem through an included toolset that packages React components into Dash-useable components.

Dash Application Design: MVC Pattern

As I worked my way through the documentation, I kept noticing that every Dash application could be divided into the following components:

Data Manipulation - Perform operations to read / transform data for display
Dashboard Layout - Visually render data into output representation
Interaction Between Components - Convert user input to commmands for data manipulation + render

That's the Model-View-Controller (MVC) Pattern's music! (Note: I covered MVC in a previous post)

When designing a Dash application, we should stucture our code into three sections:

Data Manipulation (Model)
Dashboard Layout (View)
Interaction Between Components (Controller)

I created the following template to help us get started:

# standard library
import os

# dash libs
import dash
from dash.dependencies import Input, Output
import dash_core_components as dcc
import dash_html_components as html
import plotly.figure_factory as ff
import plotly.graph_objs as go

# pydata stack
import pandas as pd
from sqlalchemy import create_engine

# set params
conn = create_engine(os.environ['DB_URI'])


###########################
# Data Manipulation / Model
###########################

def fetch_data(q):
    df = pd.read_sql(
        sql=q,
        con=conn
    )
    return df


#########################
# Dashboard Layout / View
#########################

# Set up Dashboard and create layout
app = dash.Dash()
app.css.append_css({
    "external_url": "https://codepen.io/chriddyp/pen/bWLwgP.css"
})

app.layout = html.Div([

    # Page Header
    html.Div([
        html.H1('Project Header')
    ]),

])


#############################################
# Interaction Between Components / Controller
#############################################

# Template
@app.callback(
    Output(component_id='selector-id', component_property='figure'),
    [
        Input(component_id='input-selector-id', component_property='value')
    ]
)
def ctrl_func(input_selection):
    return None


# start Flask server
if __name__ == '__main__':
    app.run_server(
        debug=True,
        host='0.0.0.0',
        port=8050
    )

Historical Matchup Dashboard

In this section, we will create a full-featured Dash application that can be used to view historicial soccer data.

We will use the following process to create / modify Dash applications:

Create/Update Layout - Figure out where components will be placed
Map Interactions with Other Components - Specify interaction in callback decorators
Wire in Data Model - Data manipulation to link interaction and render

Setting Up Environment and Installing Dependencies

There are installation instructions in the Dash Documentation. Alternatively, we can create a virtualenv and pip install the requirements file.

mkdir historical-results-dashboard && cd historical-results-dashboard
mkvirtualenv dash-app
wget https://raw.githubusercontent.com/alysivji/historical-results-dashboard/master/requirements.txt
pip install -r requirements.txt

Data is stored in an SQLite database:

wget https://github.com/alysivji/historical-results-dashboard/blob/master/soccer-stats.db?raw=true soccer-stats.db

Download the Dash application template file from above:

wget https://gist.githubusercontent.com/alysivji/e85a04f3a9d84f6ce98c56f05858ecfb/raw/d7bfeb84e2c825cfb5d4feee15982c763651e72e/dash_app_template.py app.py

Dashboard Layout (View)

Our app will look as follows:

Users will be able to select Division, Season, and Team via Dropdown components. Selection will trigger actions to update tables (Results + Win/Loss/Draw/Points Summary) and a graph (Points Accumulated vs Time)

We begin by translating the layout from above into Dash components (both core + HTML components will be required):

#########################
# Dashboard Layout / View
#########################

def generate_table(dataframe, max_rows=10):
    '''Given dataframe, return template generated using Dash components
    '''
    return html.Table(
        # Header
        [html.Tr([html.Th(col) for col in dataframe.columns])] +

        # Body
        [html.Tr([
            html.Td(dataframe.iloc[i][col]) for col in dataframe.columns
        ]) for i in range(min(len(dataframe), max_rows))]
    )


def onLoad_division_options():
    '''Actions to perform upon initial page load'''

    division_options = (
        [{'label': division, 'value': division}
         for division in get_divisions()]
    )
    return division_options


# Set up Dashboard and create layout
app = dash.Dash()
app.css.append_css({
    "external_url": "https://codepen.io/chriddyp/pen/bWLwgP.css"
})

app.layout = html.Div([

    # Page Header
    html.Div([
        html.H1('Soccer Results Viewer')
    ]),

    # Dropdown Grid
    html.Div([
        html.Div([
            # Select Division Dropdown
            html.Div([
                html.Div('Select Division', className='three columns'),
                html.Div(dcc.Dropdown(id='division-selector',
                                      options=onLoad_division_options()),
                         className='nine columns')
            ]),

            # Select Season Dropdown
            html.Div([
                html.Div('Select Season', className='three columns'),
                html.Div(dcc.Dropdown(id='season-selector'),
                         className='nine columns')
            ]),

            # Select Team Dropdown
            html.Div([
                html.Div('Select Team', className='three columns'),
                html.Div(dcc.Dropdown(id='team-selector'),
                         className='nine columns')
            ]),
        ], className='six columns'),

        # Empty
        html.Div(className='six columns'),
    ], className='twleve columns'),

    # Match Results Grid
    html.Div([

        # Match Results Table
        html.Div(
            html.Table(id='match-results'),
            className='six columns'
        ),

        # Season Summary Table and Graph
        html.Div([
            # summary table
            dcc.Graph(id='season-summary'),

            # graph
            dcc.Graph(id='season-graph')
            # style={},

        ], className='six columns')
    ]),
])

Notes:

We used HTML <DIV> elements and the Dash Style Guide to design the layout
Tables can be rendered two different ways: Native HTML or Plotly Table
We just wireframed components in this section, data will be populated via Model and Controller

Interaction Between Components (Controller)

Once we create our layout, we will need to map out the interaction between the various components. We do this using the provided app.callback() decorator.

The parameters we pass into the decorator are:

Output component + property we want to update
list of all the Input components + properties that can be used to trigger the function

Our code looks as follows:

#############################################
# Interaction Between Components / Controller
#############################################

# Load Seasons in Dropdown
@app.callback(
    Output(component_id='season-selector', component_property='options'),
    [
        Input(component_id='division-selector', component_property='value')
    ]
)
def populate_season_selector(division):
    seasons = get_seasons(division)
    return [
        {'label': season, 'value': season}
        for season in seasons
    ]


# Load Teams into dropdown
@app.callback(
    Output(component_id='team-selector', component_property='options'),
    [
        Input(component_id='division-selector', component_property='value'),
        Input(component_id='season-selector', component_property='value')
    ]
)
def populate_team_selector(division, season):
    teams = get_teams(division, season)
    return [
        {'label': team, 'value': team}
        for team in teams
    ]


# Load Match results
@app.callback(
    Output(component_id='match-results', component_property='children'),
    [
        Input(component_id='division-selector', component_property='value'),
        Input(component_id='season-selector', component_property='value'),
        Input(component_id='team-selector', component_property='value')
    ]
)
def load_match_results(division, season, team):
    results = get_match_results(division, season, team)
    return generate_table(results, max_rows=50)


# Update Season Summary Table
@app.callback(
    Output(component_id='season-summary', component_property='figure'),
    [
        Input(component_id='division-selector', component_property='value'),
        Input(component_id='season-selector', component_property='value'),
        Input(component_id='team-selector', component_property='value')
    ]
)
def load_season_summary(division, season, team):
    results = get_match_results(division, season, team)

    table = []
    if len(results) > 0:
        summary = calculate_season_summary(results)
        table = ff.create_table(summary)

    return table


# Update Season Point Graph
@app.callback(
    Output(component_id='season-graph', component_property='figure'),
    [
        Input(component_id='division-selector', component_property='value'),
        Input(component_id='season-selector', component_property='value'),
        Input(component_id='team-selector', component_property='value')
    ]
)
def load_season_points_graph(division, season, team):
    results = get_match_results(division, season, team)

    figure = []
    if len(results) > 0:
        figure = draw_season_points_graph(results)

    return figure

Notes:

Each app.callback() decorator can be bound to a single Output (component, property) pair
- We will need to create additional functions to change multiple Output components
We could add Data Manipulation code in this section, but separating the app into components makes it easier to work with

Data Manipulation (Model)

We finish the dashboard by wiring our Model into both the View and the Controller:

###########################
# Data Manipulation / Model
###########################

def fetch_data(q):
    result = pd.read_sql(
        sql=q,
        con=conn
    )
    return result


def get_divisions():
    '''Returns the list of divisions that are stored in the database'''

    division_query = (
        f'''
        SELECT DISTINCT division
        FROM results
        '''
    )
    divisions = fetch_data(division_query)
    divisions = list(divisions['division'].sort_values(ascending=True))
    return divisions


def get_seasons(division):
    '''Returns the seasons of the datbase store'''

    seasons_query = (
        f'''
        SELECT DISTINCT season
        FROM results
        WHERE division='{division}'
        '''
    )
    seasons = fetch_data(seasons_query)
    seasons = list(seasons['season'].sort_values(ascending=False))
    return seasons


def get_teams(division, season):
    '''Returns all teams playing in the division in the season'''

    teams_query = (
        f'''
        SELECT DISTINCT team
        FROM results
        WHERE division='{division}'
        AND season='{season}'
        '''
    )
    teams = fetch_data(teams_query)
    teams = list(teams['team'].sort_values(ascending=True))
    return teams


def get_match_results(division, season, team):
    '''Returns match results for the selected prompts'''

    results_query = (
        f'''
        SELECT date, team, opponent, goals, goals_opp, result, points
        FROM results
        WHERE division='{division}'
        AND season='{season}'
        AND team='{team}'
        ORDER BY date ASC
        '''
    )
    match_results = fetch_data(results_query)
    return match_results


def calculate_season_summary(results):
    record = results.groupby(by=['result'])['team'].count()
    summary = pd.DataFrame(
        data={
            'W': record['W'],
            'L': record['L'],
            'D': record['D'],
            'Points': results['points'].sum()
        },
        columns=['W', 'D', 'L', 'Points'],
        index=results['team'].unique(),
    )
    return summary


def draw_season_points_graph(results):
    dates = results['date']
    points = results['points'].cumsum()

    figure = go.Figure(
        data=[
            go.Scatter(x=dates, y=points, mode='lines+markers')
        ],
        layout=go.Layout(
            title='Points Accumulation',
            showlegend=False
        )
    )

    return figure

Notes:

Data is queried from the database each time there is an interaction
- We can improve performance through memoization
We used f-strings when creating SQL queries. Code is Python 3.6+

Run Application

Let's run app.py to make sure everything works.

$ export DB_URI=sqlite:///soccer-stats.db
$ python app.py
* Running on http://0.0.0.0:8050/ (Press CTRL+C to quit)
* Restarting with stat

Open a web browser...

And we're good to go!

Conclusion

Dash is a Python library that simplifies data-driven web app development. It combines Python's powerful data ecosystem with one of JavaScript's most popular front-end libraries (React).

In a future post, I will walk through the process of converting a React component from npm into a Dash-useable component. Stay tuned.

Additional Resources

First Month as a Professional Developer

Aly Sivji — Mon, 06 Nov 2017 06:04:55 +0000

This post was originally published on Siv Scripts

The best laid schemes o' mice an' men
Gang aft agley

I started a new job as a Mathematician / Software Engineer at a Chicago-based healthcare startup in October. It's my first proper development job and I could not be more excited.

I get paid to write code. It's pretty awesome.

Had always thought I would work in data, maybe transition into a data engineer role. But here we are. I'm working on a team of ~10 developers building a clinical decision support web application.

Guess this change isn't entirely out of left field; I had started to dig deeper into software engineering best practices over the past few months. Became fascinated by the different methodologies and frameworks that facilitate the production of good code. As a self-taught developer, I desired formality. There's only so much I could absorb by reading about other people's experiences and working on open source projects after work.

I needed to get my hands dirty and get some real world practice writing code as part of a team. Thought it would be good to experience the process-based workflow of a structured Agile/Scrum development environment; to understand the pain of working (effectively) with legacy code; and to have no choice but to write tests for the code I ship. And, preferrably, have those tests be written first.

It's only been a month, but I think I made the right decision. In one of my first commits, I made a change and took time to refactor code to increase readability ala Code Complete. Got a shoutout during code review. That felt awesome! Software Craftsmanship FTW!

I will say that I find myself constantly studying after work and on weekends so that I'm not completely useless, but I wouldn't want it any other way. Coding at the edge of my abilities will increase my skills exponentially.

While I eventually plan on going back to data, let's hold off thinking about when. Right now I'm enjoying life and want to stay on Mr. Bones' Wild Ride.

Mastering Development Tools

The main reason I took a backend developer role was to level up my dev chops. This means many different things such as writing clean code, being able to identify and correctly apply design patterns, recognizing anti-patterns (code smells), and, of course, learning the tools of the trade.

Over the past couple of weeks, I've started transitioning away from the friendly confines of VSCode and iTerm2 into the magical world of Vim and tmux. Even though I'm fairly proficient with a graphical IDE, I thought it was worth investing in tools that enable me to work as fast as I think.

By switching to a terminal-only development environment, I hope to reduce the context switching that occurs everytime I reach for my mouse or trackpad. As an added bonus, I can SSH into any machine and feel at home. No more having to set up rsync or use nano as a crutch.

How I'm Learning Vim

Worked my way through $ vimtutor. It's a rite of passage.
Enabled Vim keybindings in all my text editors (VSCode & Sublime) to get comfortable using the h j k l movement keys. Being in a familiar environment where I could get work done without having to Google every command did help flatten the learning curve.
Once I was able to make the cursor go where I wanted, I watched Learning Vim in a Week. Key takeaways:
- Set up a basic .vimrc file and throw it in Verison Control
  - My dotfile repo - Warning: Use at your own risk ðŸ˜œ
- Remap [CapsLock] to [Esc]
  - I went a step further
- Use plugins for development, with included recommendations
  - I recommend the vim-plug plugin manager

Last weekend, I installed tmux on my home machine so I could start getting familiar with how it works. My .tmux.conf is very basic, but all the pieces are in place to enable me to learn at my own pace. I just have to make an effort to learn a couple of new things every week. Like everything else in life, it's about putting in the reps.

You're only as good as your tools so it's worth making an investment into optimizing your workflow. We're developers and we should always be tinkering with the software we use on a daily basis. It doesn't have to be Vim and I'm not suggesting everybody go out and learn it. But do take some time to learn your tools and the plugin ecosystem of your IDE.

Shoutout to the folks who responded to my tweet with helpful suggestions. You Da Real MVPs!

Additional Resources

Terminal Tricks: Directory Bookmarks

Aly Sivji — Sat, 07 Oct 2017 17:05:48 +0000

This post was originally published on Siv Scripts

Even though my development folders are well organized, I still have to feel my way around the filesystem when I'm looking for a project directory. It's like trying to find a lightswitch in a dark room: a cd here, an ls there, maybe a find when I'm stuck.

I always get to where I want to go, but not without some frustration along the way. Plus, all the seconds spent navigating directories starts to add up.

Wait a minute. I use bookmarks in my browser, so why am I not using them in my shell?

A quick Google search led me to Bashmarks. Absolute game changer. I can move around the filesystem with ease.

Feel a bit like Dr. Who.

In this Quick Hit, we will explore Bashmarks, walk through the installation process, and get a feel of the most commonly used commands.

Bashmarks

Bashmarks is a [bash] shell script that allows you to save and jump to commonly used directories

It supports tab completion (!!!) and has a very simple interface with only 5 commands to memorize.

Use another shell? Not a problem. There are ports of bashmarks for fish and zsh.

Installation

Download the files into a temporary directory and install using GNU make:

mkdir temp && cd temp
git clone git://github.com/huyng/bashmarks.git
cd bashmarks
make install
echo "source ~/.local/bin/bashmarks.sh" >> ~/.bash_profile
source ~/.bash_profile

And we're good to go!

Commands

s <bookmark_name> - Saves the current directory as "bookmark_name"
g <bookmark_name> - Goes (cd) to the directory associated with "bookmark_name"
p <bookmark_name> - Prints the directory associated with "bookmark_name"
d <bookmark_name> - Deletes the bookmark
l                 - Lists all available bookmarks

We can save bookmarks using the s [bookmark_name] command:

List all bashmarks with l:

Navigate to project directory using g [bookmark_name]:

Conclusion

Bashmarks is a tool that increases developer productivity.

Do you have an interesting terminal workflow or trick to share? Please comment below!

Book Review: Code Complete

Aly Sivji — Fri, 22 Sep 2017 16:10:57 +0000

This post was originally published on Siv Scripts

2017 is the year I became deliberate about my approach to programming. Like most newbie developers, I hacked together spaghetti code around chunks of answers I found on StackOverflow (Praise Be).

I didn't care how I solved the problem, only that it was solved enough to cross off my todo list.

By ignoring recommended software development practices such as coding standards, version control, and testing, I ended up making things harder for myself in the long run.

I worried about introducing bugs everytime I touched the codebase. I couldn't reuse code because my functions and modules were tightly coupled. It was difficult to wrap my brain around the complexities of large projects with many moving parts.

Then I discovered Code Complete. Steve McConnell changed the way I thought about code. He made me realize that the process of doing things methodically is a lot more important than simply solving the problem at hand.

If we break down tasks into manageable components, we can build anything we can imagine.

Go read Code Complete. It will change your life the way it changed mine. Doesn't matter if you're a beginner or a seasoned professional, you'll become a better developer just by reading this book.

In this post, I will summarize the themes of Code Complete.

Themes

Reduce Complexity

A primary goal of software design and construction is conquering complexity. The motivation behind many programming practices is to reduce a program’s complexity, and reducing complexity is arguably the most important key to being an effective programmer.

As we increase the number of things we keep track of, we increase our chances of introducing a bug in our program. If our code is readable, we are more likely to find logic errors.

Two ways to reduce complexity:

Use helper functions
Refactor conditional tests into readable boolean expressions

What's easier to read:

# okay
for employee in all_employees:
    if employee.active == True:
        print(employee)

# better
for employee in all_employees:
    currently_employed = (employee.active == True)

    if currently_employed:
        print(employee)

Process Matters

Observing large processes and small processes means pausing to pay attention to how you create software. It’s time well spent. Saying that “code is what matters; you have to focus on how good the code is, not some abstract process is shortsighted and ignores mountains of experimental and practical evidence to the contrary. Software development is a creative exercise. If you don’t understand the creative process, you’re not getting the most out of the primary tool you use to create software–your brain. A bad process wastes your brain cycles. A good process leverages them to maximum advantage.

It's about doing things in a way we can best guarantee success. Sure we can create a product that works, but it only works now. What if requirements change? How are we going to handle those changes?

We are dealing with a living, breathing product. We need to leverage processes that help us handle changes as they arise.

One of the reasons I love Code Complete is that it introduced me to the concepts of TDD (Test Driven Development). After years of struggling with regression errors, I finally understood why testing is a large component of proper development methodologies. Tests make our lives easier in the long run.

Reability

You should go to the effort of writing good code, which you can do once, rather than the effort of reading bad code, which you’d have to do again and again.

Code is read by humans far more than it is written. Your interpreter or compiler doesn't care if it's clean or messy so write for your audience, i.e. you in six month's time.

Both of these blocks of code produce the same output, but which one makes more sense?

# bad
for x in my_list:
  print(x[0])

# good
for employee in company_roster:
  print(employee.name)

Program into Your Language, Not in It

Don’t limit your programming thinking only to the concepts that are supported automatically by your language. The best programmers think of what they want to do, and then they assess how to accomplish their objectives with the programming tools at their disposal.

This is a topic I explored when I dug into Context Managers: doing things a certain way in one language doesn't mean that we should follow the same pattern in another language. When I came into Python from C# and JavaScript, I brought all my habits with me. Instead of looking for a Pythonic solution, I looked for the Python syntax to do things the way I've always done them.

Always try to use your language of choice idiomatically.

Conventions for Clarity

The specific way in which ... a question is answered is less important than that it be answered consistently each time. Conventions save programmers the trouble of answering the same questions, making the same arbitrary decisions–again and again.

Programming is hard enough. Use style guides and logical variable names to make it easy for yourself and your team. Doing things in a standard way builds competency. Look for best practices and incorporate them into your process.

Program in Terms of the Problem Domain

Another specific method of dealing with complexity is to work at the highest possible level of abstraction. One way of working at a high level of abstraction is to work in terms of the programming problem rather than the computer science solution.

Keep business logic separate from the underlying implementation. Separate out parts of the project with helper functions, classes, or a REST API.

The This Developer's Life podcast featured an abstraction episode which discussed how we primarily concern ourselves with the layer we are currently working in. We might need to peel a layer or two from the onion when we encounter problems, but that's more on a need to know basis.

Watch for Warning Signs

Warning signs in programming alert you to the possibility of problems, but they’re usually not as blatant as a road sign that says “Watch for falling rocks.”

Code smells. Look out for patterns and symptoms that indicate a deeper problem. I'll dig into this more once I read Uncle Bob's Clean Code.

Iterate, Repeatedly, Again and Again

The more you iterate in each development activity, the better the product of that activity will be.

Iterating to a final product is a great way to build software since we can be sure that early versions of the feature are implemented in a manner that will be used.

This is also a good approach to life. Iterate your processes, optimize your workflow as you figure things out.

Don't be Religious about Software

Dogmatic methodologies and high-quality software development don’t mix. Fill your intellectual toolbox with programming alternatives, and improve your skill at choosing the right tool for the job.

Always use the best tool for the job at hand. This might require us to learn a new language, tool, technique, or framework. That's okay. As programmers, we are meant to solve problems. Sometimes this requires us to piece together a solution from many different components.

Conclusion

The first few chapters of Code Complete totally changed the way I look at software development. I've started to approach programming like a craftsman learning a trade. I am but a simple apprentice learning from the masters who have come before me.

I need to stay humble and realize that I don't know everything, but that I have the ability within me to figure out how to solve problems. Stand on the shoulders of giants. Use design patterns, algorithms, techniques, frameworks, or whatever else makes life easy. Just remember to approach development pragmatically.

References

McConnell, S. (2004) Code Complete: A Practical Handbook of Software Construction, Second Edition. Microsoft Press. [ISBN: 0790145196705]

Ultimate Solution to Python Virtual Environments: pyenv + virtualenvwrapper

Aly Sivji — Tue, 29 Aug 2017 18:18:41 +0000

(This post was originally published on Siv Scripts)

I love using the Command Line; it's my favourite part of programming. It wasn't always this way. I used to worry that the commands I entered would destory the delicate equilibrium where everything just worked. Would this be the pip install that ended it all?

Enter virtual environments.

A virtual environment (venv) is a tool that allows us to keep project dependencies isolated. If Project A requires pandas version 0.17 and Project B requires pandas version 0.21, we can create a venv for each project and keep workflows independent. More info in the Python documentation.

It's been around a year since I started using virtual environments in my projects, but I had yet to find the perfect dev workflow.

Tried Anaconda; it was too bloated for non-Data Science work.

Next came virtualenv + virtualevnwrapper. Did everything I wanted, but I had to manually manage Python installations when setting up environments for different versions.

Like everyone who learns Docker, I went thru a containerize all the things phase. The non-existent battery life on my Mac told me this wasn't the most feasible solution.

Then I found pyenv...

In this post, I will describe how to install pyenv and pyenv-virtualenvwrapper. I will also walk through the steps required to setup and use a new virtual environment.

pyenv

pyenv lets you easily switch between multiple versions of Python

Background

Anytime we run a command, the OS goes looking for the executible within the list of directories in the $PATH environment variable.

$ echo $PATH
/Users/alysivji/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/git/bin:/Users/alysivji/bin
$ which python
/usr/bin/python
$ which python3
/usr/local/bin/python3

We see both the default Python and Python3.6 installations are located in the $PATH var.

When we install pyenv, we add an additional directory of shims to the front of our $PATH. Each shim intercepts commands (python, pip, etc) and redirects them to the appropriate location.

Installation

Check the project's Github for detailed installation instructions. These instructions assume you have macOS + Homebrew installed.

brew update
brew install pyenv

Add the following line to ~/.bash_profile:

eval "$(pyenv init -)"

Let's make sure our installation worked:

$ source ~/.bash_profile
$ echo $PATH
/Users/alysivji/.pyenv/shims:/Users/alysivji/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/git/bin:/Users/alysivji/bin

Looks good! The shims at the front of our $PATH variable will intercept and redirect all python-related commands.

Commands

We use pyenv install --list to get a list of all Python versions available for installation. Let's install a few different versions of Python so we can see how pyenv works.

pyenv install 2.7.13
pyenv install pypy-5.7.1
pyenv install 3.6.2
pyenv install 3.7-dev

Set the Python we want as our default version:

$ pyenv global 3.6.2
$ pyenv versions
  system
  2.7.13
* 3.6.2 (set by /Users/alysivji/.pyenv/version)
  3.7-dev
  pypy-5.7.1

How does the shim know which version of Python to use? It goes down the hierarchy until it finds what it's looking for.

Activate different versions of Python using the pyenv shell [version] command.

$ which python
/Users/alysivji/.pyenv/shims/python
$ python
Python 3.6.2 (default, Aug 24 2017, 00:00:01)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
$ pyenv shell 2.7.13
$ which python
/Users/alysivji/.pyenv/shims/python
$ python
Python 2.7.13 (default, Aug 24 2017, 00:14:24)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

Notice how the pyenv shims take care of everything for us. Next, we'll learn how to create a venv.

pyenv-virtualenvwrapper

pyenv-virtualenvwrapper is a pyenv plugin ... to manage your virtualenvs with virtualenvwrapper

Installation

Detailed instructions are on Github. macOS + Homebrew users can follow along:

brew install pyenv-virtualenvwrapper

Append to ~/.bash_profile:

export PYENV_VIRTUALENVWRAPPER_PREFER_PYVENV="true"
export WORKON_HOME=$HOME/.virtualenvs
pyenv virtualenvwrapper_lazy

Reload the shell:

source ~/.bash_profile

Commands

Anytime we install a new version of Python, we will need to install virtualenvwrapper. This is done with either the pyenv virtualenvwrapper or pyenv virtualenvwrapper_lazy (preferred) command:

$ pyenv shell 3.6.2
$ pyenv virtualenvwrapper_lazy
Collecting virtualenvwrapper
Collecting stevedore (from virtualenvwrapper)
  Using cached stevedore-1.25.0-py2.py3-none-any.whl
Collecting virtualenv (from virtualenvwrapper)
  Using cached virtualenv-15.1.0-py2.py3-none-any.whl
Collecting virtualenv-clone (from virtualenvwrapper)
  Using cached virtualenv-clone-0.2.6.tar.gz
Collecting pbr!=2.1.0,>=2.0.0 (from stevedore->virtualenvwrapper)
  Using cached pbr-3.1.1-py2.py3-none-any.whl
Collecting six>=1.9.0 (from stevedore->virtualenvwrapper)
  Using cached six-1.10.0-py2.py3-none-any.whl
Installing collected packages: pbr, six, stevedore, virtualenv, virtualenv-clone, virtualenvwrapper
  Running setup.py install for virtualenv-clone ... done
Successfully installed pbr-3.1.1 six-1.10.0 stevedore-1.25.0 virtualenv-15.1.0 virtualenv-clone-0.2.6 virtualenvwrapper-4.7.2

Now we can use the virtualenvwrapper shell commands to manage our environment:

$ mkvirtualenv venv_test
Using base prefix '/Users/alysivji/.pyenv/versions/3.6.2'
New python executable in /Users/alysivji/.virtualenvs/venv_test/bin/python3.6
Also creating executable in /Users/alysivji/.virtualenvs/venv_test/bin/python
Installing setuptools, pip, wheel...done.
virtualenvwrapper.user_scripts creating /Users/alysivji/.virtualenvs/venv_test/bin/predeactivate
virtualenvwrapper.user_scripts creating /Users/alysivji/.virtualenvs/venv_test/bin/postdeactivate
virtualenvwrapper.user_scripts creating /Users/alysivji/.virtualenvs/venv_test/bin/preactivate
virtualenvwrapper.user_scripts creating /Users/alysivji/.virtualenvs/venv_test/bin/postactivate
virtualenvwrapper.user_scripts creating /Users/alysivji/.virtualenvs/venv_test/bin/get_env_details
(venv_test) $ which python
/Users/alysivji/.virtualenvs/venv_test/bin/python
(venv_test) $ python
Python 3.6.2 (default, Aug 24 2017, 00:00:01)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

We can also switch between virtual environments using the deactivate and workon commands:

(venv_test) $ deactivate
$ workon venv_test
(venv_test) $ python
Python 3.6.2 (default, Aug 24 2017, 00:00:01)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

That's pretty much it. We can create a virtual environment for any version of Python in two commands.

Notes

virtualenvwrapper_lazy decreases shell load time

Conclusion

We should always look for ways to optimize our development workflows. There are tools out there to help us manage complexity and make our life easy. Find these tools. Use them. It'll make you more productive in the long run.

Additional Resources

Real Python Primer on Virtual Environments
virtualenvwrapper Command List
pyenv Command Reference
pyenv-virtualenv - Alternative virtual environment manager for pyenv
direnv

Forem: Aly Sivji

Testing 101: Introduction to Testing

Table of Contents

What is Testing

Properties of Tests

1. Fast

2. Deterministic

3. Automated

Formal Definition

Benefits of Testing

Modify Code with Confidence

Identify Bugs Early

Improve System Design

Black Box vs White Box

Test Pyramid

Unit Tests

Unit Test Example

Integration Tests

Integration Test Example

End-to-End

End-to-End Test Example

Resources

Structuring Tests

Arrange-Act-Assert (AAA)

Arrange-Act-Assert Example

Given-When-Then (GWT)

Given-When-Then Example

Resources

What to Test

Functional Requirements

Basis Path Testing

Equivalence Partitioning

Boundary Analysis

Classes of Bad Data

Data Flow Testing

Error Guessing

Recap

When to Write Tests

Conclusion

Additional Resources

Finding Common Security Issues in Python Code with Bandit

Implementing a Plugin System in Python

Background

Sample Application Flow

Plugin Architecture Workflow

Toy Example

Internal

External

Main

Discussion

Running Toy Application

Real World Example

Additional Resources

PHP + MySQL using Docker Compose

Instructions

Notes

Data Science Workflows Using Docker Containers

Interactive, Web-Based Dashboards in Python

Summary

Dash Overview

Dash Application Design: MVC Pattern

Historical Matchup Dashboard

Setting Up Environment and Installing Dependencies

Dashboard Layout (View)

Notes:

Interaction Between Components (Controller)

Notes:

Data Manipulation (Model)

Notes:

Run Application

Conclusion

Additional Resources

First Month as a Professional Developer

Mastering Development Tools

How I'm Learning Vim

Additional Resources

Terminal Tricks: Directory Bookmarks

Bashmarks

Installation

Commands