Forem: Muhammad Osama Arshad

Data Driven Testing - How we went from 150+ test cases to 1

Muhammad Osama Arshad — Sat, 04 May 2019 22:10:34 +0000

This is one of my favorites.

The Problem

Finding an efficient way to test over 150+ functions.

This happened while I was working on an enhancement of the Open edX platform. Currently, Open edX uses a custom event tracking library for tracking user generated events. These events are simple JSON objects that are fed into the Open edX Insights app.

Each user generated event in Open edX is passed through a series of processors, which are basically Python classes or functions. The event passes through each of these processors (which add specific data to the event), before finally going through a router that logs the event into a log file. However, currently these is no standardized schema for the events. An event can contain any arbitrary data and the insights app can process it however it wants to. Not an ideal scenario for accurate and real-time analytics.

We were assigned to standardize these logs into a consistent format, namely the Caliper Analytics Specification by IMS Global. That way we would be enabling real-time analysis of the logs as well as consolidating a fixed structure for each event.

For example, here is an event that is generated by Open edX

{
    "accept_language": "en-US,en;q=0.9",
    "agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36",
    "context": {
        "course_id": "",
        "org_id": "",
        "path": "/api/bookmarks/v1/bookmarks/",
        "user_id": 17
    },
    "event": {
        "bookmark_id": "ali,block-v1:edX+DemoX+Demo_Course+type@vertical+block@vertical_0270f6de40fc",
        "component_type": "vertical",
        "component_usage_id": "block-v1:edX+DemoX+Demo_Course+type@vertical+block@vertical_0270f6de40fc",
        "course_id": "course-v1:edX+DemoX+Demo_Course"
    },
    "event_source": "server",
    "event_type": "edx.bookmark.added",
    "host": "9c680a5d44c8",
    "ip": "172.18.0.1",
    "name": "edx.bookmark.added",
    "page": null,
    "referer": "http://localhost:18000/courses/course-v1:edX+DemoX+Demo_Course/courseware/d8a6192ade314473a78242dfeedfbf5b/edx_introduction/1?activate_block_id=block-v1%3AedX%2BDemoX%2BDemo_Course%2Btype%40vertical%2Bblock%40vertical_0270f6de40fc",
    "session": "bf5ee12841d50f244b738e79ada5d1fc",
    "time": "2018-12-07T14:24:26.143437+00:00",
    "username": "ali"
}

And here is the same event in Caliper compliant format

{
  "@context": "http://purl.imsglobal.org/ctx/caliper/v1p1",
  "action": "Bookmarked",
  "actor": {
    "id": "http://localhost:18000/u/ali",
    "name": "ali",
    "type": "Person"
  },
  "eventTime": "2018-12-07T14:24:26.143Z",
  "extensions": {
    "extra_fields": {
      "accept_language": "en-US,en;q=0.9",
      "agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36",
      "event_source": "server",
      "event_type": "edx.bookmark.added",
      "host": "9c680a5d44c8",
      "ip": "172.18.0.1",
      "org_id": "",
      "page": null,
      "path": "/api/bookmarks/v1/bookmarks/",
      "session": "bf5ee12841d50f244b738e79ada5d1fc",
      "user_id": 17
    }
  },
  "id": "urn:uuid:d4618c23-d612-4709-8d9a-478d87808067",
  "object": {
    "id": "http://localhost:18000/courses/course-v1:edX+DemoX+Demo_Course/courseware/d8a6192ade314473a78242dfeedfbf5b/edx_introduction/1?activate_block_id=block-v1%3AedX%2BDemoX%2BDemo_Course%2Btype%40vertical%2Bblock%40vertical_0270f6de40fc",
    "type": "Page",
    "extensions": {
      "course_id": "course-v1:edX+DemoX+Demo_Course",
      "bookmark_id": "ali,block-v1:edX+DemoX+Demo_Course+type@vertical+block@vertical_0270f6de40fc",
      "component_type": "vertical",
      "component_usage_id": "block-v1:edX+DemoX+Demo_Course+type@vertical+block@vertical_0270f6de40fc"
    }
  },
  "referrer": {
    "id": "http://localhost:18000/courses/course-v1:edX+DemoX+Demo_Course/courseware/d8a6192ade314473a78242dfeedfbf5b/edx_introduction/1?activate_block_id=block-v1%3AedX%2BDemoX%2BDemo_Course%2Btype%40vertical%2Bblock%40vertical_0270f6de40fc",
    "type": "WebPage"
  },
  "type": "AnnotationEvent"
}

That is only 1 of the around 150 events that we had to transform and we had to write a separate function for each event. Okay, so writing a transformer function is not an issue. The real challenge was how to test all of the transformers in an efficient way.

The Solution

One way would be to add the testing responsibility to the developer, so who ever picks an event to transform would also be writing the test cases for it.

Since we were using Python, each developer would have to first construct a few input “dicts” and then run them through the relevant transforming function and then validate the output dict. If left to the developers, this would’ve added considerable monotonous effort on the part of each member of our team. So we thought that there has to be a better way to do this.

There was and it’s called Data Driven Testing (DDT).

Data driven testing is defined as a software testing methodology in which the inputs are not hard coded in the test cases and are provided through some external source. That source can be a database table, HTTP requests from another system, or (as in our case) simple JSON files.

Our solution was to simply create, 2 directories. current and expected. We wrote only one test case that performed the following operations.

Read a JSON file from the current directory
Parse the file and run it through the corresponding transformer
Read the corresponding JSON file from the expected directory
Assert that both dicts are equal
Repeat

The original PR for that test case can be found here, however here is the main test case function.

def test_caliper_transformers(self):
    test_files = [file for file in os.listdir(
        '{}current/'.format(TEST_DIR_PATH)) if file.endswith(".json")]

    for file in test_files:
        input_file = '{}current/{}'.format(
            TEST_DIR_PATH,
            file
        )
        output_file = '{}expected/{}'.format(
            TEST_DIR_PATH,
            file
        )

        with open(input_file) as current, open(output_file) as expected:
            event = json.loads(current.read())
            expected_event = json.loads(expected.read())

            expected_event.pop('id')

            caliper_event = CaliperBaseTransformer(event).transform_event()
            related_function = EVENT_MAPPING[event.get('event_type')]
            caliper_event = related_function(event, caliper_event)

            caliper_event.pop('id')

            self.assertDictEqual(caliper_event, expected_event)

After that test case was written, each developer had to only add the necessary JSON files in the necessary folders and they would automatically be tested while running our CI build. Leaving more time to write code and optimize for performance.

I’m hopeful that this code will be added to the main Open edX platform. and I would consider that a major contribution on our part towards Arbisoft, UC San Deigo and Open edX.

If you were added to a team with a horrible code base and an unhappy client, what steps would you take to fix everything?

Muhammad Osama Arshad — Wed, 12 Dec 2018 05:56:01 +0000

I've seen projects that have been successful because they had talented people and good processes from the start, however I was just curious on how one could recover from a situation in which people did not have the foresight to establish good practices early on.

Any suggestions?

How to install Xi Editor

Muhammad Osama Arshad — Mon, 24 Sep 2018 18:00:49 +0000

I recently went though this blog post by Nikita Prokopov. That post was both an eye-opener and a relief that someone has finally pointed out the extreme flaw in the design of most modern applications.

Nikita referenced a few projects in his post that are performance focused and are under active development. One of these caught my eye.

And of course, it was a text editor. Xi Editor has been conceived by Raph Levien and is focused on all the right stuff.

High Performance
Reliability
and Customizability

It's built using Rust and is still in early stages of development so, it's not really usable at this point. There is currently no installer or package available. The only way to install it is to build directly from the source.

It took me some time to figure out how to install and run Xi, so I decided to write a post to ease the process for others.

Xi is based on a client-server model. There is a backend process known as the xi-core which basically runs in the background and is responsible for all the editing of the files.

Installing the backend

The Xi backend is just a process that is running in the background. To install and compile it make sure you have the latest version of Rust installed on your system.
First step is to simply clone the Xi repo to a folder on your machine.

$ git clone https://github.com/xi-editor/xi-editor.git

Navigate to the rust folder inside the xi-editor repo,

$ cd xi-editor/rust/

And run the build command to compile Xi,

$ cargo build

Installing the frontend

The above commands will install the backend of Xi. To actually use the editor, you will need to install one of the multiple frontends given on the official Repo. For the purposes of this tutorial, I will install the gxi frontend. It does not have a lot of features but it does enable us to edit and save basic files.

To install the gxi frontend, clone its in a folder of your choice

$ git clone https://github.com/bvinc/gxi.git

Install the dependencies,

$ sudo apt-get install libgtk-3-dev

Now navigate to the gxi folder simply run the run command,

$ cd gxi
$ cargo run

This will open a basic text editor sort of like Notepad on Windows, only with less features. As I said before, Xi is not really usable at this point but I'm hopeful for the future. I hope I can finally be able to get an editor that is faster then Sublime Text and is open-source.