Forem: Brad Richardson

Like Miracle Grow For Your Garden

Brad Richardson — Fri, 07 Jun 2019 00:02:27 +0000

In the last post I did about the Vegetables testing framework I developed, I talked about how much of a success it had been. Now I’ve cranked it up to 11. I’ve now got what is effectively QuickCheck level functionality implemented in a Fortran testing framework.

For anyone not familiar with QuickCheck, it is effectively a library for generating random values to use as examples for your tests. But the really useful part is that if it finds an example that causes your test to fail, it then attempts to find the simplest example that causes your test to fail, and reports that to you.

The simplest example I have is just that an integer should equal itself.

function checkPassForSameInteger(input) result(result_)
    use Vegetables_m, only: Result_t, assertEquals, assertThat, fail

    class(*), intent(in) :: input
   type(Result_t) :: result_

    type(Result_t) :: example_result

    select type (input)
    type is (integer)
        example_result = assertEquals(input, input)
        result_ = assertThat( &
                example_result%passed(),
                example_result%verboseDescription(.false.))
    class default
        result_ = fail("Expected to get an integer")
    end select
end function checkPassForSameInteger

Running just this test now gives the following result.

$ test_build/vegetable_driver -q -v -f "passes with the same integer"
Running Tests

A total of 1 test cases

All Passed
Test that
    assertEquals with integers
        passes with the same integer
            Passed after 100 examples
A total of 1 test cases containg a total of 1 assertions

Then, if I change the assert to assertEquals(input, input+1), I get the following.

$ test_build/vegetable_driver -q -v -f "passes with the same integer"
Running Tests

A total of 1 test cases

Failed
Test that
    assertEquals with integers
        passes with the same integer
            Fails with the simplest possible example
            Expected to be true
                User Message:
                    [Expected
                             [0]
                     but got
                             [1]]
1 of 1 cases failed
2 of 2 assertions failed

Or, if I change the assert to assertEquals(input, input/2), I get the following.

$ test_build/vegetable_driver -q -v -f "passes with the same integer"
Running Tests

A total of 1 test cases

Failed
Test that
    assertEquals with integers
        passes with the same integer
            Found simplest example causing failure
            Expected to be true
                User Message:
                    [Expected
                             [1]
                     but got
                             [0]]
1 of 1 cases failed
2 of 2 assertions failed

In order to use the QuickCheck functionality you need to create a derived type extended from the Generator_t type. This requires you to implement two functions for it. You need generate, which takes your generator type as an argument, and produces a random value wrapped in a Generated_t type. Then you need shrink, which takes one of your generated values as a class(*), and returns a “simpler” value than the one it was given, wrapped in either a ShrunkValue_t or a SimplestValue_t to indicate the value can’t be made any simpler.

I’ve implemented a couple of basic Generators and some helper functions for generating some random values, but for most things, you’ll probably need to create your own. But if you can, and you can specify some properties of your code that should hold for any random value, you can really stress test your code.

I think at this point I have one of the most feature-full testing frameworks I’ve ever used. And I think in fewer lines of code than any other testing framework I’ve ever seen. Functional programming principles for the win!

The Power of a Testing Framework

Brad Richardson — Sun, 10 Feb 2019 23:11:36 +0000

I am now ready to call Vegetables a huge success. In my last post I discussed its design and construction. I am now using it in a project at work. The power and convenience it has brought us is huge.

The design of Vegetables is such that its use highly encourages writing your tests in a BDD, or at least a specification style. This means that the act of writing tests encourages you to think more explicitly about the requirements of the code you’re writing.

Now that your test descriptions are written as a requirements specification, the testing framework can report those requirements back to you. We’ve got our tests and our requirements developed as part of a single effort; two birds with one stone.

Next, with all the information collected by the testing framework as part of running the tests, and the ability to ask it be verbose when reporting the results, means our test suite can effectively generate our V&V report for us. That makes 3 things now that we have gotten out of the single effort of developing our test suite.

Finally, I added in the ability to ask the framework to run only a subset of the tests. Add this to the verbose outputs and I can get detailed feedback about the piece of code I’m currently working on, without having to wade through the outputs from the rest of the test suite. This shortens and simplifies the feedback cycle and makes TDD, or at least writing the tests very shortly after the code, much more useful. In fact, I put the following script into our repository for everybody to use while developing or debugging.

#!/bin/bash

DRIVER_PROGRAM="unit_test_build/vegetable_driver"

./shake.sh "${DRIVER_PROGRAM}" && $DRIVER_PROGRAM -q -v -f "${1}"

I think having this level of power and flexibility from my testing framework is going to be a must have for any project of significant size in the future. It has proved to be too useful to do without. And if one like this doesn’t exist in the language I want to use, I’ll now be confident about creating it, since I’ve now done it for Fortran.

Eat Your Vegetables

Brad Richardson — Mon, 19 Nov 2018 02:41:47 +0000

About a year ago, I was starting a new project at work using Fortran. For anybody who didn't know, the tools and libraries available to Fortran programmers are few and far between, and not particularly extensible or adaptable. But, being the testing and automation proponent I am, I went looking for a testing framework.

The only decent, open-source option I found was called fruit. It wasn't particularly well documented, but it purported to be able to find your tests suites at compile time, so you wouldn't have to manually maintain a driver program. It used Rake as a build system, and the Fortran part was contained in a single source file.

It was unmaintained, and only available from SourceForge, so I figured if I was going to use it, I better take over its maintenance. I grabbed a copy, put it in git, and got to work. I've since cleaned up all the build system components, and even fixed a few of the bugs. But I've made very little changes to the testing framework itself. The framework itself can be found here, and an example hello world type example of how to use it can be found here.

Having done all this, I was terribly unsatisfied with it. It's very imperative by nature, makes heavy use of internal state, and it's output leaves a lot to be desired. Having been maintainer of fruit for about a year now, having seen plenty of other testing frameworks in use, and having been bitten by the functional programming bug, I thought, surely I could write a better testing framework. How hard could it be.

And so, I announce to you now, a new Fortran testing framework: Vegetables. For a healthier code base, be sure to eat your vegetables. I've still got lots to do, but I believe the foundation is sound and proven at this point. I wanted to explain a bit about how I put it together.

My first couple attempts (which you can find in the git history if you are truly curious) just didn't work out. It turns out, Fortran just can NOT handle a recursive data structure properly. So I ended up having to link to C++. This isn't particularly ideal, but it was the only way I could get it to work.

Basically all of the heavy lifting of the framework happens on the C++ side. But all of the interactions happen on the Fortran side by exposing equivalent types and methods. The Fortran types just hold onto a C++ pointer, and pass it back to functions which call the appropriate methods in C++. This means, that unless you really start looking into how the framework is constructed, you won't really be able to tell. A user won't have to concern themselves with dealing with the C++ side of things.

There is downside to doing things this way. I've basically made a memory leak. The good news, it's memory that under normal circumstances wouldn't be freed until the end of execution anyway. Basically, the whole process of defining your tests builds up a single data structure containing all of them. Then, running the tests creates a second, identically shaped data structure containing all the results. But neither of these go out of scope until the end of the program, so their destructors wouldn't be called until then anyway. The only time you'd even be able to tell is if you're doing something really weird, or testing the testing framework.

Which brings me to my next point. The framework is fully tested, using itself as the testing framework.

Ok, maybe it's not that mind blowing. I'm sure some other testing frameworks have done this. But it was a little tricky trying to do TDD this way. Oh, yeah, I used TDD to build my testing framework. Yeah, that one does deserve a mind explosion.

At the early stages of the process the tests weren't really run, they only compiled. But they at least existed and trying to write them was the impetus to say which function or type needed to be written next. I had a pretty good idea for the overall design before I started, which really helped too.

The basic strategy/design is simply thus: a test is either a test case, or a collection of tests. Hence, a recursive data structure. Then running a test case produces a test result, and running a test collection produces a collection of results; an identically shaped recursive data structure.

A test collection has a description, and a collection of tests. A test case has a description and a function which returns an assertion result. Assertion results can be .and.ed together, so a result from a test case can even tell how many assertions were made as a part of it.

Enough of the abstract, how do you use this thing. Pretty simple, actually. You just define your tests like so.

function test_passing_case_behaviors() result(test)
    use Vegetables_m, only: TestCollection_t, given, then, when

    type(TestCollection_t) :: test

    test = given("a passing test case", &
            [when("it is run", &
                    [then("it knows it passed", checkCasePasses), &
                    then("it has 1 test case", checkNumCases), &
                    then("it has 1 passing case", checkNumPassingCases), &
                    then("it has no failing case", checkNumFailingCases), &
                    then("it's verbose description still includes the given description",  checkVerboseDescription), &
                    then("it's failure description is empty", checkFailureDescriptionEmpty), &
                    then("it knows how many asserts there were", checkNumAsserts), &
                    then("it has no failing asserts", checkNumFailingAsserts), &
                    then("it knows how many asserts passed", checkNumPassingAsserts)])])
end function test_passing_case_behaviors

given is a function which takes a description and an array of test collections and returns a test collection. when is a function which takes a description and an array of test cases and returns a test collection. then is a function which takes a description and a function, and returns a test case. The function must take no arguments, and return a result from one or more assert functions. Like so:

function checkCasePasses() result(result_)
    use example_cases_m, only: examplePassingTestCase
    use Vegetables_m, only: &
            Result_t, &
            TestCase_t, &
            TestCaseResult_t, &
            operator(.and.), &
            assertNot, &
            assertThat

    type(Result_t) :: result_

    type(TestCase_t) :: test_case
    type(TestCaseResult_t) :: test_result

    test_case = examplePassingTestCase()
    test_result = test_case%run()
    result_ = assertThat(test_result%passed()).and.assertNot(test_result%failed())
end function checkCasePasses

There are also describe and it functions which take a description and an array of test cases, and description and function, respectively if you don't need full BDD style given-when-then. At the top level, a function testThat accepts an array of test collections and returns a single test collection. A single subroutine runTests accepts a test collection as argument, runs it, and reports the results.

The test specification is reported to the user prior to running the tests, using indentation to signify the nesting of the test collections. Then, once the tests are run, any failures can be reported exactly the same way.

The result is a composable, pure-functional, testing framework, with a very nice output as shown below, somewhat abbreviated. See if you can figure out what's broken from the results.

Running Tests

Test that
    A test collection
        can tell how many tests it has
        includes the given description
        includes the individual test descriptions

...

A total of 100 test cases

Failed

2 of 50 cases failed
2 of 59 assertions failed

Test that
    A test collection
        can tell how many tests it has
            Expected '2' but got '4'
    A test case
        Only has 1 test case
            Expected '1' but got '2'

There is still a fair amount of work left to do, but I am now confident that it will work and is something worth using. Check it out here if you're curious, and feel free to ask any questions you might have or make any suggestions.

I also have a question for anybody who might consider using this. Would it be worth supporting a way to report all of the assertions that passed? Right now I've only got the mechanics to show all of the test cases, and the results of any failed assertions, but not the passing assertions. It wouldn't be hard to add, I just haven't decided if it would be worth it. What do you guys think?

EDIT:

I've actually managed to get the framework working using only Fortran. The data structures and functionality are the same as described above, but there's no more C++ that needs to be linked in, making this a single file drop in for anyone that doesn't want to use the same build system and automation as me.