Forem: Daniel Waller (he/him)

JetBrains Code With Me - Collaborative Coding in IntelliJ

Daniel Waller (he/him) — Thu, 01 Oct 2020 10:45:00 +0000

Has anyone else completely missed JetBrain's early access announcement for the Code With Me plugin that happened 3 days ago!?

It is finally happening, collaborative coding in IntelliJ! And it's not even restricted to IntelliJ Ultimate (for now).

All you need is an IntelliJ 2020.2.x build and an internet connection. Just install the plugin and share a collaboration link.
The other side then has to quickly install a dedicated IntelliJ Client and you're good to go.

From what I've seen during a quick test run it looks pretty stable and promising. And I'm not sure if I'm interpreting the connection UI correctly...but it seemed that the connection was peer-to-peer, with JetBrain's servers only used for lobby management. That would be a huge plus if true!

Show and tell: Show off your VS Code setup :)

Daniel Waller (he/him) — Thu, 17 Sep 2020 12:56:15 +0000

Hey DEV community 👋

I primarily use VS Code for working with React/TypeScript and Python.
I could use some help and inspiration for really getting the most out of my VS Code experience!

I'd love to see your setups :)
From pretty themes and icon sets to your favorite plugins for productivity and ease of use.

Cookie Expiry Pitfalls

Daniel Waller (he/him) — Wed, 24 Jul 2019 14:05:39 +0000

I just stumbled upon some browser behavior regarding cookies that seemed very strange to me but turned out to be actually pretty obvious and well documented.
I thought I'd share what I've learned just in case some of you might not have come across this either.

Session Cookie Expiry

This one should be pretty straight forward, right? Omitting the Expires property will result in a cookie that is deleted when a user ends the session, i.e. closes the browser.
Except, as MDN will inform you, if the user has set their browser to restore the previous session when closing.

If you are like me and carry around 100+ open tabs at all times, chances are you'll have this setting activated.

Chrome also has this feature of course.

While this is nothing new, it had never manifested itself as a problem for me until now...and it's easy to forget about when conceptualizing cookie use. So be warned.

👻 Timezones 👻

So you set an expiry date on your cookie, hm? Would be a shame if you didn't think about the possible discrepancies between your server's timezone and the browser using your site 😈
Again, as MDN explains:

When an expiry date is set, the time and date set is relative to the client the cookie is being set on, not the server.

This is fairly obvious when you think about it but something that is easily missed when thinking about setting cookies.

So there you have it. Two small details about setting cookies on server responses that could easily be missed and could lead to data being stored on a client for much longer than you intended.

Additionally this can become a fun source of obscure bugs that will manifest themselves with some classic "works on my machine 🤷" behavior in the future.

Happy coding! :)

EFAIL mitigations

Daniel Waller (he/him) — Tue, 15 May 2018 21:54:51 +0000

As you might have heard there is a flaw in many popular email clients that allows an attacker to decrypt PGP or S/MIME encrypted email content.

Here's the website with all of the info about the attacks: EFAIL

Let's collect mitigation techniques especially for the clients affected by the direct exfiltration attack (Apple Mail, iOS Mail, and Mozilla Thunderbird)

Some immediate mitigations I've read about so far:

Turn off OpenPGP and S/MIME encryption in your mail client and use an external tool (eg. Keybase) for encryption/decryption by copy-pasting content.
Disable HTML support. Only show plaintext
(For Thunderbird) Leave OpenPGP encryption enabled but turn off automatic decryption/verification of messages in Enigmail settings

(For Thunderbird) Disallow remote content in messages.

Note

The last 2 just protect you from stumbling into the attack. You can still be exploited if you accept remote content for that particular mail and/or if you manually decrypt the message.

Test organization in Python

Daniel Waller (he/him) — Sat, 05 May 2018 09:32:43 +0000

For quite some time now Python has been my hacking and proof-of-concept language of choice.

As such, testing in Python never really was an important issue for me because I wouldn't use the code in production or even revisit it most of the time.

Recently, I have been writing a small flask app in python to calibrate and continuously read from an HX711 load cell with a Raspberry Pi Zero W.

Since I'm using this code as part of my bachelor thesis I consider it to be 'production' code and most importantly I want it to be easy to refactor and extend.

So for the first time I had to think about testing in Python.

The setup

First things first, my project without tests looks like this:

As you can see, it is a very simple setup with 2 modules backend and service and an entry point server.py.

The Goal

For easy testing I want to be able to define tests for each module and be able to run them separately, as well as an easy way of running all tests of all modules.

The solution

For testing I will use pythons unittest module. The smallest unit for testing with unittest is the TestCase which can have 1 or more methods of the form def test_*():.

To begin, I tried to separate the different responsibilities of my modules into TestCase classes, eg.:

    import unittest

    class TestCalibrationChecks(unittest.TestCase):

        """
        methods that test whether all checks on the calibration state of the scale work
        """

    class TestCalibrationProcess(unittest.TestCase):

        """
        methods that test calibration of the scale
        """

After writing test cases for the modules my project looked like this:

Running a TestCase from the project root looks like this: python -m unittest backend.test.calibration_test.TestCalibrationChecks

Now I wanted to be able to run all tests in each module so I looked around the docs and came to unittest's TestSuite.

A TestSuite is a collection of TestCases and/or TestSuites and can be used to group tests together.

So in the root of my modules I wrote a file tests.py to combine all TestCases from that module's test folder into a single suite.
For the backend module it looked like this:

    import unittest

    from backend.test import TestCalibrationChecks
    from backend.test import TestCalibrationProcess
    from backend.test import TestSystemHealthChecks

    def test_scale_suite():
        scale_test_suite = unittest.TestSuite([
            unittest.TestLoader().loadTestsFromTestCase(TestCalibrationChecks),
            unittest.TestLoader().loadTestsFromTestCase(TestCalibrationProcess),
            unittest.TestLoader().loadTestsFromTestCase(TestSystemHealthChecks)
        ])
        result = unittest.TestResult()
        runner = unittest.TextTestRunner()
        print(runner.run(scale_test_suite))

    if __name__ == '__main__':
        test_scale_suite()

After writing a file like that for each module my project looked like this:

I was now able to run tests for each module from the project root with a command like this: python -m unittest backend.tests.

But what about running all tests from all modules with a single command?

With a tiny refactor of backend/tests.py and service/tests.py I was able to write a tests.py in the root directory and combine the backend and service test suites to a larger TestSuite

First I changed the tests.py in each module so that the TestSuite isn't created in the function but is an exported variable:

    import unittest

    from backend.test import TestCalibrationChecks
    from backend.test import TestCalibrationProcess
    from backend.test import TestSystemHealthChecks

    scale_test_suite = unittest.TestSuite([
        unittest.TestLoader().loadTestsFromTestCase(TestCalibrationChecks),
        unittest.TestLoader().loadTestsFromTestCase(TestCalibrationProcess),
        unittest.TestLoader().loadTestsFromTestCase(TestSystemHealthChecks)
    ])

    def test_scale_suite():
        result = unittest.TestResult()
        runner = unittest.TextTestRunner()
        print(runner.run(scale_test_suite))

    if __name__ == '__main__':
        test_scale_suite()

Now I was able to define tests.py in the project root like this:

    import unittest

    from backend import tests as backend_tests
    from service import tests as service_tests

    complete_test_suite = unittest.TestSuite([
        backend_tests.scale_test_suite,
        service_tests.service_test_suite
    ])


    def run_all_suites():
        result = unittest.TestResult()
        runner = unittest.TextTestRunner()
        print(runner.run(complete_test_suite))


    if __name__ == '__main__':
        run_all_suites()

The finished project now looks like this:

And from the project root I am able to run

all tests python -m tests
backend tests python -m unittest backend.tests
service tests python -m unittest service.tests

Mission Accomplished

Notes

I am aware there is a function unittest.TestLoader().discover() which will automatically discover all TestCases in a file tree. I chose not to use it because I like to explicitly declare what to run so I can see what to expect and avoid some tests not running because the discoverer missed them for some reason.
I am not saying this is the right way to organize your tests in Python, it just seems to work very well for me so far. Happy to hear your comments on this :)

From Network Response to Algebraic Data Type with Kotlin

Daniel Waller (he/him) — Mon, 15 Jan 2018 13:10:36 +0000

Or, what I learned from trying to stop handling HTTP error codes as exceptions.

The other day I finally understood something that had eluded me for months now: How can we avoid using exceptions for handling HTTP error codes? More specifically, how can we cast them to something different before they are being thrown as exceptions?
Here's what I learned.

If you're using Kotlin you may have already heard about the concept of Algebraic Data Types (ADTs).
If not, there is a cool post from Dmitry Zaytsev and another one by Mike Gehard.

In short, here's what we need to know about them for this post:
The idea is to wrap a bunch of objects and data classes inside a sealed class which they all inherit from.

typealias Second = Int

sealed class PowerUp {

    object FireFlower : PowerUp()

    object SuperMushroom : PowerUp()

    data class SuperStar(val duration: Second) : PowerUp()
}

You can then use them very effectively together with Kotlin's when statement:

fun Mario.collect(powerUp: PowerUp) { 
    // thanks to the 'return' a non-exhaustive 'when' will result in a compiler error
    return when(powerUp) {
        is FireFlower -> enableFlameThrowing()
        is SuperMushroom -> grow()
        // 'powerUp' is smart cast to 'SuperStar' so we can access duration on it
        is SuperStar -> startGodMode(powerUp.duration) 
    }
}

The cool thing here is that the compiler can infer whether a when expression is exhaustive or not because the type being evaluated is a sealed class.
Sealed classes can only be extended within their own .kt file so the compiler can guarantee that FireFlower, SuperMushroom, and SuperStar are the only possible values the powerUp variable can take on.
If we then use expression syntax rather than statement syntax for the when, ie. val result = when(..) {} or return when(..) {}, the compiler will treat this as an error and not build your program.

This can come in handy when you add more objects to your sealed class but forget to implement how to react to them in the when expression. Instead of your app crashing or entering an undefined state you'll be notified of the error at compile time.

Another powerful feature, smart casting, can be seen on the last branch. Here powerUp on the right-hand side of the branch is automatically cast to the type being checked for on the left-hand side of the branch. So we can use SuperStar's duration property without explicitly writing (powerUp as SuperStar).duration.

Now as Collinn Flynn explains, this Algebraic Data Structure can be really useful for error handling without resorting to exceptions.

Error handling for network requests

When we make network requests there are many things that can go wrong and many networking libraries react to any HTTP response code outside of the 200s with an exception. This is weird because in the context of most apps, codes such as 404 (Resource not found), 401 (Unauthorized) or 403 (Forbidden) are actually meaningful to our app's business logic and should be handled accordingly to enhance user experience, eg.:

redirect to login on 401
offer premium upgrade for some subscription service on 403
show cutesy 404 page

Traditionally we have to catch any exceptions resulting from our network calls and handle them in order to implement these cases. This can make our network calling code messy and tricky to adapt to changing requirements.

Imagine you have network calling code in many places in your program and have to add a new HTTP code to react to. For example your team now also wants to show a cutesy service unavailable page on 503.

This can become a fun source of errors especially when new developers, unfamiliar with the codebase make the change (or even just you 3 months later 😉).

Also there are some people with opinions about driving control flow through exceptions 👀.

Context shift. We're now switching from Mario to a simple HackerNews reader

Using Algebraic Data Types we can avoid this by defining any response that has meaning to our app inside a sealed response class:

sealed class NetworkResult

sealed class Payload : NetworkResult() {

    data class AggregateArticles(val articleIds: List<Long>) : Payload()

    data class SingleArticle(val article: HackerNewsItem): Payload()
}

sealed class HttpError : NetworkResult() {

    object ResourceNotFound : HttpError()

    object ServiceUnavailable : HttpError()

    data class UnknownError(val code: Int, val message: String) : HttpError()
}

Now we can use the when expression at the network call site like this:

    fun handleNetworkResult(result: NetworkResult) {
        return when(result) {
            is Payload -> handleResult(result)
            is HttpError -> handleError(result)
        }
    }

    fun handleResult(result: Payload) {
        return when(result) {
            is AggregateArticles -> loadArticleDetails(result.articleIds)
            is SingleArticle -> showArticle(result.article)
        }
    }

    // this would probably live in an injected global error handler
    fun handleError(result: HttpError) {
        return when(result) {
            is ResourceNotFound -> show404Screen()
            is ServiceUnavailable -> show503Screen()
            is UnknownError -> showError("Unknown error ${result.code}: ${result.message}")
        }
    }

This is cool because as we've shown above you can't forget to implement a case now cause you'll get a compiler error on the when expression where you're missing a branch.

Great! But there's already loads of articles out there that talk about all this.
How do we actually get from a network layer response object to our sealed class members!?

How to integrate with the network layer

The following code will be Android specific and assume a Dagger 2 setup

In Android development we often use OkHttp + Retrofit to define our network layer and RxJava to handle the asynchronous nature of network calls.
A typical Retrofit initialization would look like this


    @Provides
    @Singleton
    fun provideRetrofit(): Retrofit {
        val rxCallAdapter = RxJava2CallAdapterFactory.createWithScheduler(Schedulers.io())
        return Retrofit.Builder()
                .baseUrl("https://hacker-news.firebaseio.com/v0/")
                .addConverterFactory(GsonConverterFactory.create(gson))
                .addCallAdapterFactory(rxCallAdapter)
                .build()
    }

Here we add a call transformer that will enable us to have every network call wrapped in an RxJava Observable type and a converter to automatically parse the network response's JSON to a data type.
A typical API interface defined with Retrofit, and a Service using it could then look like this:

interface NewsAggregateApi {
    @GET("topstories.json")
    fun loadTopStories(): Single<List<Long>>

    @GET("newstories.json")
    fun loadNewStories(): Single<List<Long>>
}

class NewsAggregateService(private val aggregateApi: NewsAggregateApi) {

    fun loadTopStories(): Single<List<Long>> = aggregateApi.loadTopStories()

    fun loadNewStories(): Single<List<Long>> = aggregateApi.loadNewStories()
}

If everything goes smoothly while calling /topstories.json our Observable will get a List of articleId: Long but if we encounter any HTTP code not in the 200s the Observables onError method will be triggered. This is bad because it happens before we can wrap anything in our nice sealed class types.
In an old answer to Retrofit Issue #1218 Jake Wharton has a solution for our problem:
, Observable>, or Observable>. For the first version, there's nowhere to hang non-200 response information so it is included in the exception passed to onError. For the latter two, the data is encapsulated in the Response object and can be accessed by calling errorBody().""/>

So going by that, we can now change our API interface accordingly to look like this:

interface NewsAggregateApi {
    @GET("topstories.json")
    fun loadTopStories(): Single<Response<List<Long>>>

    @GET("newstories.json")
    fun loadNewStories(): Single<Response<List<Long>>>
}

And make some changes to our service and our sealed class so that the return type of the service is always a NetworkResult and we can map our Response to one of the NetworkResult subtypes.

class NewsAggregateService(private val aggregateApi: NewsAggregateApi) {

    fun loadTopStories(): Single<NetworkResult> {
        return aggregateApi.loadTopStories()
                .map { it.toAggregateResult() }
    }

    fun loadNewStories(): Single<NetworkResult> {
        return aggregateApi.loadNewStories()
                .map { it.toAggregateResult() }
    }

    private fun Response<List<Long>>.toAggregateResult() =
            if (this.isSuccessful) {
                NetworkResult.fromAggregateResponse(this.body() ?: emptyList())
            } else {
                NetworkResult.fromErrorResponse(this.code(), this.message())
            }
}

sealed class NetworkResult {

    // unmodified code omitted

    companion object {
        fun fromAggregateResponse(payload: List<Long>): NetworkResult {
            return Payload.AggregateArticles(payload)
        }

        fun fromErrorResponse(code: Int, message: String?): NetworkResult {
            return when (code) {
                404 -> HttpError.ResourceNotFound
                503 -> HttpError.ServiceUnavailable
                else -> HttpError.UnknownError(code, message ?: "")
            }
        }
    }

}

And that's all there is to it! Now every valid network response will be wrapped in your ADT class.
There are some caveats though. This will only take care of valid responses that are successfully parsed. Any errors during JSON deserialization or network stack exceptions such as timeouts and SSL protocol exceptions will still be thrown and have to be handled elsewhere.

But we now have a nice (and easily testable!) way to handle non-200 server responses that have meaning to our business logic.

Thanks for reading :)

P.S. If you would like to see the code samples with some more context check out this repo.

Happy Holidays! Have some graph theory...

Daniel Waller (he/him) — Tue, 12 Dec 2017 17:10:42 +0000

So its more or less Christmastime and I was reminded the other day of a popular childrens puzzle in German-speaking countries called Haus des Nikolaus which roughly translates to House of Santa Claus¹.
Being a child I always solved it with one solution I had memorized but only a few years ago I learned about the theoretical basis of it during a university math class by accident.

So having a little downtime and thinking it could be fun, welcome to my first ever post on the internet!

The puzzle is to draw a house like the one in the title image, beginning in any one of its corners, never retracing a line you've already drawn and never lifting your pen.
The end result should look like this:

Take out a piece of paper and try it! Can you solve it? Can you think of multiple solutions to the problem?

Did it? Couldn't do it? I came here to read not to draw!?
Let's see how we can formalize and solve the problem using graphs!

For starters let's have some definitions

A graph consists of a set of nodes (or vertices) and edges connecting them. In our special case we are talking about an undirected graph, ie. a graph in which you can traverse the edges in either direction.
Another definition we have to make is the degree of a node, ie. the number of edges connected to a node.

So with our new knowledge let's setup santa's house as a graph with each node's degree denoted next to it.

Brief course in history

The Haus des Nikolaus problem could be interpreted as a variation of an early 18th century problem, the Seven Bridges of Königsberg, addressed by mathematician Leonhard Euler (1707-1783).
The question was whether it was possible to devise a round-trip through the then capital of East Prussia, the city of Königsberg, crossing all of its seven bridges exactly once and end up where you started your trip.

Euler's solution to the problem is today regarded as the first ever published paper on graph theory.
He observed that when you enter a landmass coming across a bridge you must also exit that landmass using a bridge. Since you aren't allowed to use the same bridge twice for either entering or exiting, each landmass must have an even number of bridges connected to it. One for entering and one for exiting the landmass.

He concluded that the Seven Bridges of Königsberg problem has no solution because each of the landmasses had an odd number of bridges connected to it.
To state the problem and Eulers observation more generally:
Find a way through a graph that
1) visits all nodes,
2) traverses every edge exactly once,
3) and starts and ends in the same node.
This way is called a Eulerian cycle.
An undirected graph has a Eulerian cycle if and only if every node has an even degree.

For a more complete and coherent post about the Seven Bridges of Königsberg problem check out Vaidehi Joshi's awesome post on it!

Back to our house

The constraints for our house drawing stated earlier can be rephrased as a weaker variation of the Eulerian cycle:
Find a way through a graph that
1) visits all nodes,
2) traverses every edge exactly once,
3) BUT start and end node need not be the same.
This way is called a Eulerian trail.
From our observations above we can derive that a Eulerian trail is present in a graph if and only if exactly zero or two nodes have an odd degree.

As we can see from our Haus des Nikolaus graph representation luckily we have exactly 2 nodes with odd degree in our graph so we can conclude that a solution is in fact possible!
But there's a catch.
The only way you can make your drawing work is when you start and end your path in one of the odd degree nodes.

To make this fact more apparent, let's tag each edge with either 'in' or 'out' depending on which direction you want to use them in your path.

Now we can immediately see that one of the bottom nodes has 2 inputs but only 1 output, meaning you'll inevitably end up getting trapped in it. This naturally must be your end node.
The other of the two bottom nodes has 2 outputs but just 1 input, meaning that in order to traverse all the edges in the graph one of the 2 outputs needs to be the first edge you traverse. Making that node your start node.

So there we have it. A simple childrens puzzle solved with far too much effort!
I have the feeling that this is the kind of useless information that tends to stick with you forever. Also it's always interesting to encounter and identify variations of famous mathematical problems in the wild.

Hope you had fun :)

Extra Credit

How many possible solutions to this problem are there?

^{1 That's a lie Nikolaus does not translate to Santa Clause. He is Saint Nicholas. The rhyme just works better like that :P}

Hi, I'm Daniel Waller

Daniel Waller (he/him) — Wed, 24 May 2017 14:22:12 +0000