Forem: Dmitry Yakimenko

Telegram bot in Go: charts

Dmitry Yakimenko — Thu, 06 Jun 2019 21:58:39 +0000

It's been a long time since I wrote about the bot I'm building. Life happened. A lot of it. Now, when things are settling down, I might have again the time and the energy to program the bot some more and write about it in the process.

Last time the bot learned some text commands. Now it's time to draw some pretty pictures. Today I'd like to add a couple of chart commands.

Using Telegram API it possible to send an image (as well as a video or any other document), it's not limited to just text messages. All I need is a way to generate an image and push it down the wire. After a quick search, I stumbled upon go-chart. go-chart renders charts into in-memory PNG and SVG files and this is exactly what I need. Here's an example:

I'm not so interested in visualizing stock prices though. I have my own data to deal with. The first step would be to draw something simple. Last time I added the /top command that shows 10 most logged events. It would be good to see that as a bar chart as well. Please welcome the /topChart command.

The charts look a bit too small and ugly like this, but they blow up once you click on them. The code for this chart is pretty simple:

func (c context) topChart(args string) {
    num := parseTopArgs(args)

    // Get values from the DB and convert
    values := make([]chart.Value, 0, num)
    for _, e := range c.getTopEvents(num) {
        values = append(
            values,
            chart.Value{Label: e.name, Value: float64(e.count)},
        )
    }

    // Chart settings
    response := chart.BarChart{
        Title:      fmt.Sprintf("Top %d events", num),
        TitleStyle: chart.StyleShow(),
        Background: chart.Style{
            Padding: chart.Box{
                Top: 40,
            },
        },
        Width:    num * 100,
        Height:   512,
        BarWidth: 80,
        XAxis:    chart.StyleShow(),
        YAxis: chart.YAxis{
            Style:          chart.StyleShow(),
            ValueFormatter: chart.IntValueFormatter,
            Range: &chart.ContinuousRange{
                Min: 0,
                Max: values[0].Value,
            },
        },
        Bars: values,
    }

    // Render and send
    c.sendChart(response)
}

To get the values I use the following query:

SELECT name, COUNT(name) freq FROM events
    WHERE user = ?
    GROUP BY name
    ORDER BY freq DESC
    LIMIT 10

And rendering the cart is trivial with go-chart

buffer := &bytes.Buffer{}
err := chartSettings.Render(chart.PNG, buffer)

Now the buffer holds the bytes for a PNG file. It could be saved to disk or sent over the wire via Telegram API like this:

image := tgbotapi.FileBytes{Name: "chart.png", Bytes: buffer.Bytes()}
_, err := c.bot.Send(tgbotapi.NewPhotoUpload(c.message.Chat.ID, image))

Easy-peasy.

Another idea is to use a bar chart to show daily activity for a specific event like this:

Up close it looks like this:

What I thought would be very cool is to draw a diagram similar to GitHub activity for a selected event. On GitHub it looks like this:

Easy? Well, not really. I spent a bunch of time trying to replicate it. I tried to save myself some trouble at first and use the stacked bar chart, but it didn't work out. So I had to start from scratch, roll up the sleeves and do it myself. About five hundred lines later I have this:

Close enough for my needs. There's quite a bit of code to be covered in this post. Maybe I make a separate post some time where I describe what it takes to draw a chart using go-chart. It is doable. The problem is testing. Especially if you want to support all kinds of different options, like configurable axis labels, legend, title, font sizes and so on. It's quite tedious. I declare: works on my machine! Ship it!

What's next

The bot seems to have all the basic features. Sure, they are not very robust and not polished, but they kinda work. It's time to start using it. I need to figure out a way to deploy it somewhere and have it running. DevOps stuff. I wonder if I'm gonna need a Kubernetes cluster for that.

If you're curious, the code is available on GitHub. This version is tagged day-6.

Originally published on detunized.net. Follow me on Twitter or LinkedIn.

Don't make me hack your software

Dmitry Yakimenko — Wed, 17 Apr 2019 12:23:44 +0000

I got a new corporate VPN tool the other day. It's called Pulse Secure. It worked fine, thank you very much, no complains there. But then I tried to quit

and I got this

That pissed me off right then and there. Why would any remote admin tell me what to do on my local machine? Even if it's a company machine. When I do not use the VPN or maybe I'm not even connected to any WiFi, why should I have this running? It's just silly to try to prevent me from doing something on the machine where I have the root access. When this gets pushed onto a developer's laptop (as opposed to an accountant or an HR clerk) it simply becomes a challenge.

It has no impact on the security of the machine or the corporate network when it's simply running in the background. This is not anti-virus software. It has an impact on the battery life and other resources like RAM and CPU though. And I need those. It's exactly this type of programs that don't exit, hang around and prevent me from running yet another Electron app on my laptop. Sometimes one is not enough, you know.

I guess I'd have to get my hands dirty instead of doing something I was actually going to do. Simply killing it from the command line didn't work. It just starts over. Investigation it is, then.

First, I took a quick look at the list of the open files from the Activity Monitor. There I found a log file (bottom row):

The tool developers were very kind and dumped about half a meg of stuff on every restart cycle into the log. So it's wasting my disk space as well then. Amongst thousands of lines, I found a reference to /Library/Application Support/Pulse Secure/Pulse/connstore.dat. It sounded promising. After poking around in that file and trying this and that, I found a parameter that is responsible for that.

ive "921sn438-qoo8-4pp4-85p7-68q5s2592o32" {
    ...
    connection-policy-override: "false"
    ...
}

When this parameter is changed to true and the tool is restarted I was able to quit. Voilà!

Yes, I do.

And here the script that automates the whole thing:

cd '/Library/Application Support/Pulse Secure/Pulse'
sudo sed -i '' \
      s'/connection-policy-override: "false"/connection-policy-override: "true"/' \
      connstore.dat
sudo killall PulseTray 'Pulse Secure' dsAccessService

Dear developers and sysadmins, please don't do that again. You're just wasting everybody's time. Rather put your efforts into making the software more reliable, less resource hungry, more secure.

Originally published on detunized.net

Telegram bot in Go: speak robot

Dmitry Yakimenko — Tue, 09 Apr 2019 11:14:50 +0000

Last time I taught my bot to speak human. This time I'm gonna teach it to speak robot. I'm going to add a few bot commands. In Telegram the bots receive text exactly as you send it. By convention though, when the first word starts with a backslash (/) it's interpreted as a command. Commands are used to tell the bot what to do. It's a bit like shell, where you spend, I assume, most of your time.

The go-telegram-bot-api/telegram-bot-api package provides some functions to check whether there's a command in the received message and lets you extract it and its arguments.

for update := range updates {
    if update.Message.IsCommand() {
        command := update.Message.Command()
        arguments := update.Message.CommandArguments()
    }
}

After some light refactoring my bot is ready to accept commands as well as regular text. Actually, the plain text is simply mapped to the /add command.

if message.IsCommand() {
    switch command := message.Command(); command {
    case "a", "add":
        c.add(message.CommandArguments())
    case "e", "export":
        c.export()
    case "h", "help":
        c.help()
    case "s", "since":
        c.since(message.CommandArguments())
    case "t", "top":
        c.top()
    case "test":
        c.test()
    default:
        c.sendText(fmt.Sprintf("Eh? /%s?", command))
    }
} else {
    c.add(message.Text)
}

/test

The simplest command would be /test, it just sends some hardcoded text back. It's pretty trivial to implement:

func (c context) test() {
    c.bot.Send(tgbotapi.NewMessage(c.message.Chat.ID, "It works"))
}

/export

Export is a very important command. I consider it so important that I implemented it first. It allows the user to download all of its data stored in the bot's database. Say no to vendor lock-in.

This is really annoying these days. Take any app, like a fitness tracker app for example. It keeps track of your runs and stores your data somewhere on the phone or in the cloud. And one day it stops working or you decide to start using another app. And what do you do? How do you migrate your data? Usually, there's no easy way or no way at all. You'd have to abandon all of your history and your progress and start over in the new app.

I don't want that for my yet non-existent users. Let them take their data home whenever they want. That's why I have the /export command. Just say /export to the bot and it's happy to comply:

To generate the CSV file I use the built-in package encoding/csv. It's very easy to export all the rows into a CSV file :

buffer := &bytes.Buffer{}
csv := csv.NewWriter(buffer)

// For each row
for ... {
    csv.Write([]string{name, date})
}

// Flush when done (important!)
csv.Flush()

// Send
c.bot.Send(tgbotapi.NewDocumentUpload(
    c.message.Chat.ID,
    tgbotapi.FileBytes{
        Name:  "data.csv",
        Bytes: buffer.Bytes()}))

/add and /since

The /add and the /since commands are bread and butter of this bot. As the names imply, one is for adding the events and the other is for checking when the event was last added. /add is a 2-in-1 command, like shampoo & conditioner in one bottle, it displays the time before adding.

/top

To practice SQL, I added a relatively useless command to display 10 most logged events.

SELECT name, COUNT(name) freq FROM events
    WHERE user = <user-id>
    GROUP BY name
    ORDER BY freq DESC
    LIMIT 10

On my really dumb dataset it looks like this:

I have to say that I'm pretty impressed with SQL so far. I knew theoretically what it could do. But now I get to try it and it's really cool how I can just write a simple query instead of writing a bunch of code with loops and variables. Probably more efficient too.

What's next

The bot is getting smarter. Now it could respond in different ways to a few basic commands. More is coming. I'd like the bot to be able to draw charts and show some statistics, like how often something happened or distribution throughout the day/week/month. Do you have any ideas?

If you're curious, the code is available on GitHub. This version is tagged day-5.

Originally published on detunized.net

Telegram bot in Go: speak human

Dmitry Yakimenko — Thu, 04 Apr 2019 22:27:29 +0000

Last time I was bulletproofing my SQLite access foundation. Let's see if it holds in production. Too bad I'm never gonna get that high load that is supposed to break things. Oh well, let's wait and see. Maybe I get lucky and two sad concurrent requests would eventually make my Go binary panic.

Today, I'm going to focus on features. I'd like to make my bot intelligent. Not really AI like, rather just sounding a bit better than cmd.exe when you type in a wrong command. I'm not gonna make anything big, more like dipping the toe in the water to see how it is.

First, I'd like to find the last event with the same name, since I'm storing them in the database, and send back the date the event. Imagine the interaction like this:

me: eat
bot: Previous 'eat' happened 6 hours ago

To find the event with the same name I have to ask SQLite to give me the row where the user is the same, the name of the event it the same and I want the last row after it's sorted by date. Converting English to SQL I get:

SELECT date FROM events
    WHERE user = ? AND name = ?
    ORDER BY date
    DESC LIMIT 1

Pretty simple, eh? Now, converting this to Go and crawshaw.io/sqlite I get the following:

// Default response
response := fmt.Sprintf("Fist time for '%s'", name)

// Get the last event with the same name and format the response
err := sqlitex.Exec(connection,
    "SELECT date FROM events "+
        "WHERE user = ? AND name = ? "+
        "ORDER BY date "+
        "DESC LIMIT 1",
    func(s *sqlite.Stmt) error {
        response = formatResponse(s.GetInt64("date"), name)
        return nil
    },
    message.From.ID,
    name)

// Send the message back to the user
go func() {
    bot.Send(tgbotapi.NewMessage(message.Chat.ID, response))
}()

Where formatResponse is very rudimentary:

func formatResponse(date int64, name string) string {
    last := time.Unix(date, 0)
    return fmt.Sprintf("Previous '%s' happened on '%v'", name, last)
}

Now the interaction looks like this:

Not exactly the smartest bot on the planet, but we're going somewhere.

Now, I would like to make it sound a bit less dumb and bit more human, which are not always going together. In this case, they are. I don't want the bot to say things like happened on '2019-04-04 14:13:57 +0200 CEST', but rather something like 8 minutes since or 1 year since. Believe it or not, but there's a package for exactly that. Welcome hako/durafmt. With its help it's very easy to turn the conversation into something like this:

It is much more readable. To make it work I just had to modify the formatResponse function a bit (now it also has to take the current message date):

func formatResponse(name string, date int64, prevDate int64) string {
    prev := time.Unix(prevDate, 0)
    now := time.Unix(date, 0)
    duration := durafmt.ParseShort(now.Sub(prev))
    return fmt.Sprintf("%s since last '%s'", duration, name)
}

Easy peasy. With not so many lines and in under an hour of work I have a bot that speaks human and can tell me when something happened the last time. Ship it! 🚢

If you're curious, the code is available on GitHub. This version is tagged day-4.

Originally published on detunized.net

Telegram bot in Go: concurrent SQLite

Dmitry Yakimenko — Mon, 01 Apr 2019 14:03:45 +0000

Last time I added SQLite to my bot and at the same time I moved the request processing into a goroutine. Which means I introduced concurrent database access to my codebase.

Normally one should think, then do. Though not ideal, it's also possible to do it the other way around when you use git. But it's important not to forget to think at some point. Like I almost did and almost moved on to pile on more bugs on top of what I just introduced.

I remembered reading or watching something about SQLite the other day and they made it clear that I'd have to make sure not to access the database from multiple places at the same time. Go is made for concurrency, so I'm sure go-sqlite3 I was using has it under control. Not exactly, as it turned out.

From the FAQ:

Can I use this in multiple routines concurrently?

Yes for readonly. But, No for writable. See #50, #51, #209, #274.

And just like that I went down the major rabbit hole. I searched and researched, googled and binged (no, not really), read the source of go-sqlite3 and the docs for SQLite (which are pretty good by the way). All of that just to find myself in a situation that I don't know if I can trust the go-sqlite3 package to handle my database work. Even though the FAQ states that write concurrency is not supported, I found contradicting statements. People were saying it's fine if I use multiple connections. But I wasn't going to open a new connection every time.

Long story short, I found a blog post which addressed exactly the same problem and offered a solution for it (and a couple of others) in a form of a Go package.

go get -u crawshaw.io/sqlite

According to David, the author of the post and the package, I should be able to trust this package to handle concurrency reliably. Now, I don't simply establish a connection to the database, but rather create an explicit pool of those of the size I desire (16 in this case):

import "crawshaw.io/sqlite/sqlitex"

func openDB() *sqlitex.Pool {
    db, err := sqlitex.Open("./since.db", 0, 16)
    if err != nil {
        log.Panic(err)
    }

    return db
}

Now, every time I want to access the database I have to get a connection from the pool and not to forget to put it back when I'm done. Getting a connection from the pool might block until a connection becomes available.

func execSQL(db *sqlitex.Pool, sql string) {
    connection := db.Get(nil)
    defer db.Put(connection)

    err := sqlitex.Exec(connection, sql, nil)
    if err != nil {
        log.Panic(err)
    }
}

A side note on the error handling. I currently do not handle errors on purpose to not slow myself down. But I don't ignore them either. Like any self-respecting software engineer I panic when I receive an error. For some errors, like not being able to open the database on startup, it's totally fine to panic. But if there was an error with one of the messages, panic is not such a good choice. It's like shutting down the server when we should have just returned HTTP/404.

Now, storing the incoming message in the database becomes this:

func store(message *tgbotapi.Message, db *sqlitex.Pool) {
    connection := db.Get(nil)
    defer db.Put(connection)

    err := sqlitex.Exec(
        connection,
        "INSERT INTO events (user, name, date) VALUES (?, ?, ?);",
        nil,
        message.From.ID,
        message.Text,
        message.Date)

    if err != nil {
        log.Panic(err)
    }
}

One small thing I don't like about this is that I'm forced to use positional SQL arguments if I want to use sqlitex.Exec. If I wanted to use the column names like "... VALUES ($user, $name, $date)", I'd have to use a much wordier API. Prepare statement myself and then step through it. Like this:

func store(message *tgbotapi.Message, db *sqlitex.Pool) {
    connection := db.Get(nil)
    defer db.Put(connection)

    insert := connection.Prep("INSERT INTO events (user, name, date) VALUES ($user, $name, $date);")
    insert.SetInt64("$user", int64(message.From.ID))
    insert.SetText("$name", message.Text)
    insert.SetInt64("$date", int64(message.Date))

    _, err := insert.Step()
    if err != nil {
        log.Panic(err)
    }

    // Done with this query
    // TODO: Is it really needed? What happens when this isn't called?
    err = insert.Reset()
    if err != nil {
        log.Panic(err)
    }
}

If I see that positional arguments become a problem, I'll switch to this way of doing things.

As much as I wanted to make my bot less dumb this time, I only managed to switch libraries while trying to bullet-proof my SQLite access methods. Just refactoring, no features. Here's a typical day of a software engineer for you. Well, the next day then.

If you're curious, the code is available on GitHub. This version is tagged day-3.

Originally published on detunized.net

Telegram bot in Go: database

Dmitry Yakimenko — Fri, 29 Mar 2019 22:36:45 +0000

Today I'm gonna add a database to my bot. As I mentioned in the previous post, I'm going to use SQLite to keep it simple and because the management turned down my budget request for Oracle on this project.

I quickly shopped around for an SQLite package for Go and found go-sqlite3. This seems to be a very popular package. It allows me to use the standard database/sql package and it acts as a driver, which allows the Go database engine to talk to SQLite databases. So far so good. I'll use that then.

Since I'm writing everything in Go, I desperately need to use goroutines somewhere. So the first thing I did was to move the reply functionality into a goroutine.

for update := range updates {
    go reply(bot, update)
}

That should make things go FAST! Never mind the fact that Telegram will not allow frequent requests for any extended period of time. I've gotta try anyway.

To use the database I have to open it first. It's done like this:

func openDatabase() *sql.DB {
    db, err := sql.Open("sqlite3", "./since.db")
    if err != nil {
        log.Panic(err)
    }

     _, err = db.Exec("CREATE TABLE IF NOT EXISTS events (" +
        "id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, " +
        "user INTEGER, " +
        "name TEXT, " +
        "date INTEGER);")

     if err != nil {
        log.Panic(err)
    }

     return db
}

I also create a table if it doesn't exist. Since I barely use SQL, I prefer to SCREAM so people can hear me. And maybe the DB will get a sense of urgency and process my queries faster.

Once the database successfully opened and no one panicked, it's time to start saving incoming events to it. I do it like this:

func store(message *tgbotapi.Message, db *sql.DB) {
    _, err := db.Exec("INSERT INTO events (user, name, date) VALUES ($1, $2, $3);",
        message.From.ID,
        message.Text,
        message.Date)
    if err != nil {
        log.Panic(err)
    }
}

store(update.Message, db)

Now, every time I send a message to my bot it stores its content in the database. It's a start. The bot is still responding with the same reply as before. It's still really dumb. We'll get to make it smarter in the next day.

If you're curious, the code is available on GitHub. This version is tagged day-2.

Originally published on detunized.net

22-2222-22

Dmitry Yakimenko — Fri, 29 Mar 2019 14:43:01 +0000

Opened my dashboard just now and that what I saw there =)

Thanks for following, guys!

Telegram bot in Go

Dmitry Yakimenko — Thu, 28 Mar 2019 16:14:06 +0000

This is my fourth or fifth attempt to like Go. Kinda didn't work all the previous times. I blame myself, though. I believe I chose Go for the wrong types of projects. This time it's what's it was made to do. It's a backend application, it's deployed to a remote server, it's using concurrency and not too complex. It's a Telegram bot that helps me track random things.

Idea

I want to be able to send simple commands to the chat and let the bot remember what happened and when. It's a bit like a log file. I want to be able to ask the bot when or how often something happened. This is what it could look like:

me: eat
bot: Recorded 'eat'. Last 'eat' happened 8 hours ago.

me: sleep, 10 hours ago
bot: Recorded 'sleep' at 23:15, yesterday.
     Last 'sleep' happened 1 day 12 hours ago.

me: /since run
bot: Last 'run' happened 2 days 15 hours ago.

me: /stats blog
bot: So far 'blog' happened 20 times, average
     time between events is 4 days 5 hours.

The questions and answers are a bit clumsy, but I don't think I'm going to implement the whole natural language processing thing here. It's a simple bot, remember? I have ideas for many more commands that could be useful, but I'll start small with recording and basic querying.

Architecture

It's kind of a big word, eh? To talk to the Telegram servers I'm going to use Go bindings for the Telegram Bot API. I'm planning on using webhooks in the final version. I'll start with the poll loop though since it's possible to run the polling bot locally.

I'm planning to use SQLite to store the data. I like zero configuration and setup. I like that I could just copy the database anywhere I want. This way I could organize backups very easily. For example, I could just email it to myself every now and then. Unless I open the bot to the world, I don't expect the database to grow much.

Code

Let's jump straight to coding then. The Go Telegram Bot API provides a good starting point in the README. It's an echo bot. It just sends the received text back to the user. I'll start with that.

For the bot to connect to the Telegram network it has to have a username and an API token. Both thing you could get by talking to a special bot on Telegram. The bot is called The Botfather.

The interaction is quite trivial and pleasant. Just follow the dialog and he'll give you what you need. Just make sure not to ask for too much.

The API token I got from the Botfather should be kept private. Who owns the token owns the bot. To keep it out of the repo I added a bit of code to read it from a simple JSON config. Besides that, it's just the code from the README.

// Config represents the structure of the config.json file
type Config struct {
    Token string `json:"token"`
}

func readConfig() Config {
    file, err := os.Open("config.json")
    if err != nil {
        log.Panic(err)
    }

    defer file.Close()
    bytes, err := ioutil.ReadAll(file)
    if err != nil {
        log.Panic(err)
    }

    var config Config
    err = json.Unmarshal(bytes, &config)
    if err != nil {
        log.Panic(err)
    }

    return config
}

That should do it for day 1. If you're curious, the code is available on GitHub. This version is tagged day-1.

Originally published on detunized.net

NUnit to xUnit automatic test conversion: source code transformation

Dmitry Yakimenko — Tue, 26 Mar 2019 20:52:23 +0000

In the previous post I wrote about how I find the patterns in the code that I would like to refactor using simple C# syntax. Basically, I write the exact expression I would like to find with some wildcards that match the varying parts and the rest is matched as is. A bit like a regexp or a shell file glob. Like this:

Assert.That(_, Is.EqualTo(_))
Assert.That(_, Throws.TypeOf<_>())

_._(_, _)

if you'd like to go extreme and match every member function call with two parameters.

What I would like to be able to do, though, is to transform the code, not just match. I'd like to specify how to convert the patterns to the form I'm after. For example:

Assert.That(@actual, Is.EqualTo(@expected)) -> Assert.Equal(@expected, @actual)

Well, this is exactly the syntax I'm going to use. The placeholders @actual and @expected match any expression subtrees in the AST, anything that could be placed as arguments in those two positions. It's actually a valid C# code as it allows @ at the beginning of an identifier. In my matcher I treat any identifier that starts with an @ as a placeholder.

The code in the previous post only matched the patterns (_ matches anything, a bit like .* in a regex). Now we need to store the named matches, like the match groups in a regex. The code would have to be changed from

// A placeholder matches anything
if (IsPlaceholder(pattern))
    return true;

// A placeholder matches anything
if (IsPlaceholder(pattern))
{
    var name = pattern.ToFullString();
    if (name != "_")
        Matches[name] = code;

    return true;
}

The rest of the matching code stays practically untouched. We have a matcher, the next step would be to write a "substituter" or "replacer", the replace part of the find-and-replace tool.

Since I like to keep my code DRY, first I spent a bunch of time thinking about how to reuse the matching code for the substitution part. I failed there. So I decided to go with WET instead and rewrite that giant switch once again.

// _variables is populated in the constructor
public SyntaxNode Replace(SyntaxNode template)
{
    // A placeholder found. Substitute.
    if (IsPlaceholder(template))
    {
        var name = template.ToFullString();
        if (_variables.TryGetValue(name, out var v))
            return v;

        return template;
    }

    switch (template)
    {
    case ArgumentSyntax t:
        return t.Update(t.NameColon,
                        t.RefKindKeyword,
                        (ExpressionSyntax)Replace(t.Expression));
    case ArgumentListSyntax t:
        return t.Update(t.OpenParenToken,
                        Replace(t.Arguments),
                        t.CloseParenToken);
    case IdentifierNameSyntax t:
        return t.Update(Replace(t.Identifier));
    case InvocationExpressionSyntax t:
        return t.Update((ExpressionSyntax)Replace(t.Expression),
                        (ArgumentListSyntax)Replace(t.ArgumentList));
    case LiteralExpressionSyntax t:
        return t.Update(Replace(t.Token));
    case MemberAccessExpressionSyntax t:
        return t.Update((ExpressionSyntax)Replace(t.Expression),
                        t.OperatorToken,
                        (SimpleNameSyntax)Replace(t.Name));
    case GenericNameSyntax t:
        return t.Update(Replace(t.Identifier),
                        (TypeArgumentListSyntax)Replace(t.TypeArgumentList));
    case TypeArgumentListSyntax t:
        return t.Update(t.LessThanToken, Replace(t.Arguments), t.GreaterThanToken);
    default:
        return template;
    }
}

This code recursively walks the replace template AST and substitutes the placeholders (like @actual and @expected) with the matches found in the previous step. The end result of this substitution is then swapped with the matched expression node in the original source file AST. And that's it. Works like a charm.

A side note on the choice of keywords here. I'm not an OOP aficionado. I think it has its place and in C# it's quite natural to use classes and member functions to represent and do stuff. Though most of the time I try to stick to more of a functional style, where functions don't have any hidden state and take all their input as parameters and return the result. Most of my functions and classes are static, actually. But common sense wins most of the time for me (or so I hope). So, in this case, I went with a class and non-static functions to not to pass around the state into every function, as there are quite many recursive calls. I kept it DRY in a sense.

I wrote a small driver application for this algorithm. It takes care of the loading, parsing, writing out the result. All the patterns are hardcoded for now, but there's nothing but my laziness stopping me from putting that into a config file.

Here are the patterns I used to convert the bulk of my tests:

Assert.That(@actual, Is.EqualTo(true))      -> Assert.True(@actual)
Assert.That(@actual, Is.EqualTo(false))     -> Assert.False(@actual)
Assert.That(@actual, Is.EqualTo(@expected)) -> Assert.Equal(@expected, @actual)
Assert.That(@code, Throws.TypeOf<@type>())  -> Assert.Throws<@type>(@code)

Conclusion

The solution described here is not a working tool ready to be picked up and used by anyone. It's a proof of concept that happens to work and it has value. At least for me. I was able to convert hundreds of tests across dozens of files, saving myself a couple of hours of tedious manual labor. The time I put into this tool I didn't get back. Pure ROI is negative here. But it's not only about time. I got plenty of satisfaction from learning something new, from automating an annoying task, from blogging about it and sharing it with the world.

The code could be found here.

Future work

Not every AST node type is covered by the pattern matching code and not every possible Assert expression is covered by the patterns. That is something to expand on. Maybe this could also be turned into a Visual Studio [Code] extension. In general this doesn't have to be a unit test conversion tool. With this mini-DSL it's possible to refactor/convert any code really. It's also possible to search for the matches in the codebase. Lots of ideas and not so much time.

Originally published on detunized.net

NUnit to xUnit automatic test conversion: pattern match

Dmitry Yakimenko — Mon, 18 Mar 2019 07:57:54 +0000

In the previous post I described how to use the Roslyn API to find code patterns in the C# AST and how to change the AST to rewrite the original code to something else. The goal was to automate the conversion of NUnit tests to xUnit. The approach I used was quite tedious, as I had to write a very long chain or ifs and typecasts to get the job done. Let's try to do better this time. Let's start with just the search part in our search-and-replace tool.

What would be great is to be able to specify structural patterns like this:

Assert.That(_, Is.EqualTo(_))
Assert.That(_, Is.EqualTo(true))
Assert.That(_, Is.Throws.TypeOf<_>())

And they would match the actual code:

// Matched by 'Assert.That(_, Is.EqualTo(_))'
Assert.That(account.Id, Is.EqualTo(id))
Assert.That("".ToBytes(), Is.EqualTo(new byte[] {}))

// Matched by 'Assert.That(_, Is.EqualTo(true))'
Assert.That(info.IsMd5, Is.EqualTo(true));
Assert.That(token.BoolAt(path, true), Is.EqualTo(true));

// Matched by 'Assert.That(_, Is.Throws.TypeOf<_>())'
Assert.That(() => Quad[-1], Throws.TypeOf<ArgumentOutOfRangeException>())
Assert.That(() => access(token, path), Throws.TypeOf<JTokenAccessException>())

At first it looks like a quite difficult task. But as it turns out in its simple form is not even that hard. I got the idea first when I was generating code for AST replacement with Roslyn Quoter. Looking at its source code I discovered a bunch of Parse* methods of the SyntaxFactory class.

So basically one function call will parse the snippet and return an AST for the given pattern:

var patternAst = SyntaxFactory.ParseExpression("Assert.That(_, Is.EqualTo(_))");

The one line above is equivalent to a wall of code like this:

var patternAst =
    InvocationExpression(
        MemberAccessExpression(
            SyntaxKind.SimpleMemberAccessExpression,
            IdentifierName("Assert"),
            IdentifierName("That")))
    .WithArgumentList(
        ArgumentList(
            SeparatedList<ArgumentSyntax>(
                new SyntaxNodeOrToken[]{
                    Argument(
                        IdentifierName("_")),
                    Token(SyntaxKind.CommaToken),
                    Argument(
                        InvocationExpression(
                            MemberAccessExpression(
                                SyntaxKind.SimpleMemberAccessExpression,
                                IdentifierName("Is"),
                                IdentifierName("EqualTo")))
                        .WithArgumentList(
                            ArgumentList(
                                SingletonSeparatedList<ArgumentSyntax>(
                                    Argument(
                                        IdentifierName("_"))))))})));

It feels like a total win already and we have not even done anything useful yet. But let's find this pattern in a source AST. First, we need to parse the file we're searching in:

var sourceAst = CSharpSyntaxTree.ParseText(File.ReadAllText(filename));

This gives us the list of all expression nodes in the AST:

var nodes = sourceAst.GetRoot().DescendantNodes().OfType<ExpressionSyntax>();

And now we find the nodes that match:

foreach (var e in nodes)
{
    if (Ast.Match(e, patternAst))
    {
        var line = e.GetLocation().GetLineSpan().StartLinePosition.Line;
        var code = e.NormalizeWhitespace();
        Console.WriteLine($"  {line}: {code}");
    }
}

Obviously the Ast.Match function is the tricky one. But not as tricky, really. We recursively traverse both ASTs in parallel and see if they match:

public bool Match(SyntaxNode code, SyntaxNode pattern)
{
    // A placeholder matches anything
    if (IsPlaceholder(pattern))
        return true;

    // Node types don't match. Clearly not a match.
    if (code.GetType() != pattern.GetType())
        return false;

    switch (code)
    {
    case ArgumentSyntax c:
        {
            var p = (ArgumentSyntax)pattern;
            return Match(c.Expression, p.Expression);
        }
    case ArgumentListSyntax c:
        {
            var p = (ArgumentListSyntax)pattern;
            return Match(c.OpenParenToken, p.OpenParenToken)
                && Match(c.Arguments, p.Arguments)
                && Match(c.CloseParenToken, p.CloseParenToken);
        }
    case IdentifierNameSyntax c:
        {
            var p = (IdentifierNameSyntax)pattern;
            return Match(c.Identifier, p.Identifier);
        }
    case InvocationExpressionSyntax c:
        {
            var p = (InvocationExpressionSyntax)pattern;
            return Match(c.Expression, p.Expression)
                && Match(c.ArgumentList, p.ArgumentList);
        }
    case LiteralExpressionSyntax c:
        {
            var p = (LiteralExpressionSyntax)pattern;
            return Match(c.Token, p.Token);
        }
    case MemberAccessExpressionSyntax c:
        {
            var p = (MemberAccessExpressionSyntax)pattern;
            return Match(c.Expression, p.Expression)
                && Match(c.Name, p.Name);
        }
    case GenericNameSyntax c:
        {
            var p = (GenericNameSyntax)pattern;
            return Match(c.Identifier, p.Identifier)
                && Match(c.TypeArgumentList, p.TypeArgumentList);
        }
    case TypeArgumentListSyntax c:
        {
            var p = (TypeArgumentListSyntax)pattern;
            return Match(c.LessThanToken, p.LessThanToken)
                && Match(c.Arguments, p.Arguments)
                && Match(c.GreaterThanToken, p.GreaterThanToken);
        }
    default:
        return false;
    }
}

So it's basically a giant switch with every node type in it. By far not every type is covered here, just those that I needed to get my examples to work. I imagine to cover the most of C# syntax I'd have to tediously write a couple of thousand lines of repetitive code. I'm not going to do it all any time soon. Just the stuff I need to cover my use cases.

With a few more lines of code added this already becomes a useful tool for searching for code patterns in a codebase. Next time we see how we can implement the replace part. The goal was to refactor, not just to search, wasn't it? I have some ideas on how it could be done. See you next time.

Conclusion

Thanks to Roslyn awesome API with just 172 lines of code we have a pretty advanced code grep. Surely, it's just a toy and a proof of concept at the moment. It would take a serious effort to make it something more than that. But I'm happy with what is possible with so little effort. Amazing.

Originally published on detunized.net

NUnit to xUnit automatic test conversion

Dmitry Yakimenko — Tue, 12 Mar 2019 23:45:32 +0000

I'm currently working on a major refactoring of a C# library which has many NUnit tests. I decided, without having any good reason, it would be a good idea to migrate them to xUnit. I did a few by hand and it turns out to be tedious. Like really tedious. The most common pattern is the following:

The test in NUnit

Assert.That(actual, Is.EqualTo(expected));

becomes the test in xUnit:

Assert.Equal(expected, actual);

To convert each by hand requires a lot of patience and stamina. Since I don't have either, after doing a few dozens manually I decided to automate the whole thing. The first most obvious approach would be to use a regexp and convert one to the other like so:

/Assert\.That\((.*?), Is\.EqualTo\((.*)\)\)/ -> Assert.Equal($2, $1)

It does work for some simple cases, but throw in something a bit hairier and the whole thing goes sideways. A perfectly valid small test tears this regexp to shreds:

Assert.That("));", Is.EqualTo("));")); -> Assert.Equal(", "));");;"));

Not good.

The better way to do that would be to parse the source file into an AST (Abstract Syntax Tree) and perform source to source transformations on it. I've done a bit of this in the past with C++ using Clang/LLVM and with JavaScript using Acorn parser. For C# there's Roslyn.

How difficult could this be? Let's find out. There's some amount of documentation out there and some samples. Also, it's possible to generate a starter project with VS2017 that would do the file loading and minimal AST traversal. It's a good start, we can build on it. Here's a good starting point for source transformation, for example.

So here's a simplest NUnit module:

using NUnit.Framework;

namespace Test
{
    [TestFixture]
    class DumpTests
    {
        [Test]
        public void One_plus_one_should_be_two()
        {
            Assert.That(1 + 1, Is.EqualTo(2));
        }
    }
}

When converted to xUnit, it becomes this:

using Xunit;

namespace Test
{
    class DumpTests
    {
        [Fact]
        public void One_plus_one_should_be_two()
        {
            Assert.Equal(2, 1 + 1);
        }
    }
}

Precisely the following needs to be done:

change using directive
remove TextFixture class attribute
replace Test attribute with Fact
change Assert.That to Assert.Equal and swap arguments

The syntax rewriter does all the work, we just need to fill in some logic:

public class NunitToXunitRewriter: CSharpSyntaxRewriter
{
    ...
}

Let's start with the easiest, removing the TextFixture attribute:

public class NunitToXunitRewriter: CSharpSyntaxRewriter
{
    public override SyntaxNode VisitAttributeList(AttributeListSyntax node)
    {
        if (ShouldRemoveTestFixture(node))
            return null;

        return base.VisitAttributeList(node);
    }

    // Checks if the node is "[TestFixture]" and should be removed
    private bool ShouldRemoveTestFixture(AttributeListSyntax node)
    {
        return node.Attributes.Count == 1
            && node.Attributes[0].Name.ToString() == "TestFixture"
            && node.Parent is ClassDeclarationSyntax;
    }
}

The code, in this case, is quite simple. The VisitAttributeList function gets called for every attribute in the source file. We just check that the attribute list has only one attribute, that its name is TextFixture and the parent node is a class. If it's all true, then we just return null from the visitor method to indicate that the node should be deleted from the syntax tree. Easy.

Next up is the Test attribute:

public class NunitToXunitRewriter: CSharpSyntaxRewriter
{
    public override SyntaxNode VisitAttributeList(AttributeListSyntax node)
    {
        var newNode = TryConvertTestAttribute(node);
        if (newNode != null)
            return newNode;

        return base.VisitAttributeList(node);
    }

    // Converts "[Test]" to "[Fact]"
    private SyntaxNode TryConvertTestAttribute(AttributeListSyntax node)
    {
        if (node.Attributes.Count != 1)
            return null;

        if (node.Attributes[0].Name.ToString() != "Test")
            return null;

        if (!(node.Parent is MethodDeclarationSyntax))
            return null;

        return
            AttributeList(
                AttributeList<AttributeSyntax>(
                    Attribute(
                        IdentifierName("Fact"))))
            .NormalizeWhitespace()
            .WithTriviaFrom(node);
    }
}

What we do here is quite similar to the previous example, with one exception that we're not deleting the node, but replacing it with something else. First, we check that it's a single attribute named Test and it's attached to a function. To replace it, we need to construct a new syntax node. In this case, it's the same thing, just the name is different. To build the syntax node we use SyntaxFactory methods, like AttributeList, Attribute and so on. The small quirk is the NormalizeWhitespace and
WithTriviaFrom bits. Those make sure the resulting code is formatted and has the whitespace copied from the original node. Otherwise, the output code would look out of place and would require reformatting.

The using directive change is also trivial. It's very similar to the Fact attribute situation above:

public class NunitToXunitRewriter: CSharpSyntaxRewriter
{
    public override SyntaxNode VisitUsingDirective(UsingDirectiveSyntax node)
    {
        var newNode = TryConvertUsingNunit(node);
        if (newNode != null)
            return newNode;

        return base.VisitUsingDirective(node);
    }

    // Converts "using NUnit.Framework" to "using Xunit"
    private SyntaxNode TryConvertUsingNunit(UsingDirectiveSyntax node)
    {
        if (node.Name.ToString() != "NUnit.Framework")
            return null;

        return
            UsingDirective(IdentifierName("Xunit"))
            .NormalizeWhitespace()
            .WithTriviaFrom(node);
    }
}

The Assert conversion is a much more complicated case. The problem that the expression we want to match is quite complex, even though it doesn't look like that. There's a member function access Assert.That and a function call Assert.That(...) and the argument list made up of two arguments, where the second one is a member function call as well: Assert.That(actual, Is.EqualTo(expected)). Using Roslyn Quoter tool it's possible to generate the code that creates such an expression:

InvocationExpression(
    MemberAccessExpression(
        SyntaxKind.SimpleMemberAccessExpression,
        IdentifierName("Assert"),
        IdentifierName("That")))
.WithArgumentList(
    ArgumentList(
        SeparatedList<ArgumentSyntax>(
            new SyntaxNodeOrToken[]{
                Argument(
                    IdentifierName("actual")),
                Token(SyntaxKind.CommaToken),
                Argument(
                    InvocationExpression(
                        MemberAccessExpression(
                            SyntaxKind.SimpleMemberAccessExpression,
                            IdentifierName("Is"),
                            IdentifierName("EqualTo")))
                    .WithArgumentList(
                        ArgumentList(
                            SingletonSeparatedList<ArgumentSyntax>(
                                Argument(
                                    IdentifierName("expected"))))))})))

In the AST form this little snippet of code looks pretty huge. When we want to replace this pattern with a different piece of code, we need to find it first. And that means we need to check against the structure of every function call expression in the file and see if it's similar:

public class NunitToXunitRewriter: CSharpSyntaxRewriter
{
    public override SyntaxNode VisitInvocationExpression(InvocationExpressionSyntax node)
    {
        var newNode = TryConvertAssertThatIsEqualTo(node);
        if (newNode != null)
            return newNode;

        return base.VisitInvocationExpression(node);
    }

    // Converts Assert.That(actual, Is.EqualTo(expected)) to Assert.Equal(expected, actual)
    private SyntaxNode TryConvertAssertThatIsEqualTo(InvocationExpressionSyntax node)
    {
        // Check it's Assert.That member
        if (!IsMethodCall(node, "Assert", "That"))
            return null;

        // It must have exactly two arguments
        var assertThatArgs = GetCallArguments(node);
        if (assertThatArgs.Length != 2)
            return null;

        // The second argument must be a `Is.EqualTo`
        var isEqualTo = assertThatArgs[1].Expression;
        if (!IsMethodCall(isEqualTo, "Is", "EqualTo"))
            return null;

        // With exactly one argument
        var isEqualToArgs = GetCallArguments(isEqualTo);
        if (isEqualToArgs.Length != 1)
            return null;

        // Grab the arguments
        var expected = isEqualToArgs[0];
        var actual = assertThatArgs[0];

        // Build a new AST with the actual and expected nodes inserted into it
        return
            InvocationExpression(
                MemberAccessExpression(
                    SyntaxKind.SimpleMemberAccessExpression,
                    IdentifierName("Assert"),
                    IdentifierName("Equal")))
            .WithArgumentList(
                ArgumentList(
                    SeparatedList<ArgumentSyntax>(
                        new SyntaxNodeOrToken[] {expected, Token(SyntaxKind.CommaToken), actual})))
            .NormalizeWhitespace()
            .WithTriviaFrom(node);
    }
}

To match the expression we have to drill down into the AST and compare node by node. It's very tedious, but luckily after the code is written it will convert all the tests that have a similar structure. Write once, run many times. The two helper functions that are used in this matching code look like this:

private bool IsMethodCall(ExpressionSyntax node, string objekt, string method)
{
    var invocation = node as InvocationExpressionSyntax;
    if (invocation == null)
        return false;

    var memberAccess = invocation.Expression as MemberAccessExpressionSyntax;
    if (memberAccess == null)
        return false;

    if ((memberAccess.Expression as IdentifierNameSyntax)?.Identifier.ValueText != objekt)
        return false;

    if (memberAccess.Name.Identifier.ValueText != method)
        return false;

    return true;
}

private ArgumentSyntax[] GetCallArguments(ExpressionSyntax node)
{
    return ((InvocationExpressionSyntax)node).ArgumentList.Arguments.ToArray();
}

In case the expression is a match, we take the expected and actual arguments, or the AST nodes that represent them, to be exact and wrap them into a different AST that represents the xUnit equivalent: Assert.Equal(expected, actual).

Not that crazy difficult. But now we have a tool that can convert a majority of tests from NUnit to xUnit automagically. And it not only converts the Assert expressions but the whole file. Nice!

The sucky part is that the matching code is very specific to the expression we're trying to convert. So if we have a few variations of the Assert it would take writing so much code for every case. It's gonna very quickly get out of control. Imagine just a few very simple variations:

Assert.That(actual, Is.True());
Assert.That(actual, Is.EqualTo(true));
Assert.That(actual, Is.False());
Assert.That(actual, Is.EqualTo(false));

To cover most common NUnit cases we'd have to write hundreds of those matching functions with very repetitive code. That would be a LOT of work. Can we do better? Yes, we can! I have an idea and I'll describe in the next post.

Conclusion

In only 175 lines of code we have a fully functional converter that does in a second what takes a lot of time to do by hand. Even though it's just a proof of concept and doesn't cover any significant amount of NUnit assertions, I was able to convert a few files with tests with almost no additional fixing.

Originally published on detunized.net

Base64 decoding bug that is present in all version of .NET

Dmitry Yakimenko — Wed, 06 Mar 2019 22:29:13 +0000

One sunny morning I was sitting in front of my laptop refactoring some C# code. Everything was going very smooth and it was going to be a productive day. And then I added one too many equal signs to a constant string literal and things just blew up. Gone the productivity. Gone the peaceful refactoring Sunday. Even the sun decided to hide behind the cloud.

After spending 30-40 minutes trying to figure out what I did wrong, I realized it wasn't me. It was Microsoft. Apparently, I stumbled upon an ancient bug in Base64 decoding function. This bug must be present since the introduction of Convert.FromBase64String in .NET 1.1 in the year of 2003. Whoa! That's old. And it's not very difficult to reproduce. Here you go:

Convert.FromBase64String("abc==");

Technically this is an illegal Base64. The legal version is "abc=". Notice only one padding character =. The Base64 encoding represents every 6 bits of the binary input with one ASCII character. This means every 4 characters in the Base64 encoded string represent 3 bytes. When the encoded data is not a multiple of 3 bytes Base64 encoder adds padding characters to make the Base64 a multiple of 4 characters. This makes "abc=" a correctly padded Base64 string. Adding another = to it makes it invalid.

Base64 "abc=" decodes to two bytes [105, 183]. This is correct. Adding another padding character at the end shouldn't really change the encoded value. It's like adding a space at end of the sentence. Yes, it's there, but it doesn't change the meaning of the sentence. But .NET doesn't think so. "abc==" decodes to one byte of [109]. Not only it got shorter, which is weird since we made the input longer. It also got different. The first byte changed from 105 to 109. And an exception didn't get thrown either. Add another = and you'll get an exception. Amazing!

Code:

using System;

public class Program
{
    public static void Main()
    {
        DecodeAndPrint("abc=");
        DecodeAndPrint("abc==");
    }

    static void DecodeAndPrint(string base64)
    {
        Console.WriteLine(
            "'{0}' -> [{1}]",
            base64,
            string.Join(", ", Convert.FromBase64String(base64)));
    }
}

Output:

'abc=' -> [105, 183]
'abc==' -> [109]

And what is really amazing, is that no one discovered this for so many years. Or it got discovered, but it didn't get fixed. Base64 is quite fundamental in the information exchange over the wire. It is used all over the place. Yet, .NET got away with a buggy Base64 decoder for so many years.

At first I couldn't believe it and started to investigate it. I googled for a while and didn't really find much. Then I posted on StackOverflow, but didn't get much luck there either. I had to even answer my own question once I figured out what's going on. After searching on GitHub for a while I stumbled upon a fix in .NET Core made in July 2018. So the latest .NET Core version handles this correctly and throws an exception:

Unhandled Exception: System.FormatException: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.
   at System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength)
   at System.Convert.FromBase64String(String s)
   at Program.DecodeAndPrint(String base64) in ./base64/Program.cs:line 13
   at Program.Main() in ./base64/Program.cs:line 8

It took them about 15 years to find this and fix it. What's interesting is that no one really tried to fix it specifically. It happened while they rewrote the code to make it faster:

Convert.FromBase64() had a subtle bug where an illicit
second padding character at the end of the string caused
the decode to "succeed" by dropping the fifth to
last character.

We inadvertently fixed this bug while optimizing that
api in .NetCore 2.1. Adding test to document bug and
ensure we don't regress.

So this is fixed in .NET Core 2.2. But it's still broken in the current latest version of .NET Framework 4.7.2. And it looks like it's broken in Mono too.

A workaround for .NET 4.7.2 would be to repad incorrectly padded strings with something like this:

// This only works for base64 without spaces or linebreaks.
string Repad(string base64)
{
    var l = base64.Length;
    return l % 4 == 1 && base64[l - 1] == '='
        ? base64.Substring(0, l - 1)
        : base64;
}

Originally published on detunized.net