Forem: Bruno Ferreira

Boost your terraform automation

Bruno Ferreira — Mon, 21 Dec 2020 00:39:31 +0000

After some time working with CloudFormation to manage infrastructure on AWS, a few months ago I switched jobs and started to work with terraform. I feel that the learning curve was really smooth, despite the differences between these tools.
After using the two, I feel that HCL is great to avoid all the mess one can do with yaml and if you are using multiple cloud services, for example Fastly or Cloudflare for CDN instead of Cloudfront, or running a couple of workloads on different cloud providers such as GCP or Azure, terraform is a great tool to help you manage everything with a unified tool and language.
I was also surprised with the amount of useful tools and resources available to work with terraform (check the Awesome Terraform repository!).

Terraform workflow

The typical terraform workflow might look similar to this:

You start by opening a pull request to a repository that contains the infrastructure code and the first step is to check if the files are well formatted by running terraform fmt --check. After that, you run terraform init to initialize the working directory by downloading the different terraform modules you might be using and initialize the backend, where the status is stored.
The third step is to run terraform plan that outputs the changes that will be done on the infrastructure based on the modifications you did on the code. You can then analyze this output and see what will be modified, created and/or deleted. If you and your peers are ok with the output, you get your pull request approved, merge it to the main branch and go for the terraform apply that will apply the changes. Otherwise, if you are not ok with the output of the plan, you do another commit to fix what is wrong and the workflow starts from the beginning. Pretty straightforward, right?

Apply and... Fail!

However, even if the plan executes successfully and the output seems fine to you and your peers, it is possible that the terraform apply command fails and, since it is not transactional, your infrastructure will be in an inconsistent state until someone applies a patch to fix the problem. The apply command can fail for multiple reasons such as typo on the instance type or missing permissions on an iam role that makes it impossible to do a specific operation.

So, what can we do to reduce the likelihood of failing at the `apply` step?

In this blog post I will do a brief introduction to some of the tools you can add to your terraform workflow to help you to avoid this problem and, on top of that, add automated checks on security and infrastructure costs.

TFLint

terraform-linters / tflint

A Pluggable Terraform Linter

Let's start with TFLint, a static code analysis tool focused on catching possible errors and enforcing best practices. Let's check the following example:

In this pull request, just by simply adding the reviewdog github action for TFLint, I was able to get an automated review reporting that the instance type, the ebs volume type and also the AMI id were invalid. These kind of errors are easy to catch, but also easy to miss in the review process and would fail in the apply, as previously described.

TFLint has more than 700+ rules available and also supports custom rules, for example to enforce patterns in S3 bucket names. It supports AWS, Azure and GCP.

Terratest

gruntwork-io / terratest

Terratest is a Go library that makes it easier to write automated tests for your infrastructure code.

And what if we could actually test the apply step to check if it will run successfully? 🤔 By running the applystep on a different environment and making automated assertions on the side effects of it, we can also detect problems before running it in the production environment. For this we have Terratest, a Go library to write automated tests for infrastructure code.

localstack / localstack

💻 A fully functional local AWS cloud stack. Develop and test your cloud & Serverless apps offline

The first time I tried Terratest, I used it with Localstack that is an AWS "emulator" you can run locally on your machine or anywhere else. The main reasons for this experiment were to avoid having a different AWS account to run the tests and to be able to run it locally and discard everything easily just by destroying localstack's docker container.

In order to make terraform execute against your localstack's container, you have to add some settings on your provider block, as you can see in the following example. The most important part is the endpoints block that should contain the url for the different AWS services you are interacting with on your terraform code.

provider "aws" {
  region                      = "us-east-1"
  access_key                  = "mock_access_key"
  secret_key                  = "mock_secret_key"
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  s3_force_path_style         = true
  endpoints {
    ec2 = "http://localhost:4566"
    iam = "http://localhost:4566"
    s3  = "http://localhost:4566"
  }
}

And here's an example of a simple test that checks if the created bucket with the name "awesome-bucket" has versioning enabled:

terraformOptions := &terraform.Options{
    TerraformDir: "../../local",
    EnvVars: map[string]string{
        "AWS_REGION":         awsRegion,
    },    
}

defer terraform.Destroy(t, terraformOptions)

terraform.InitAndApply(t, terraformOptions)

actualStatus := aws.GetS3BucketVersioning(t, 
    awsRegion, "awesome-bucket")
expectedStatus := "Enabled"
assert.Equal(t, expectedStatus, actualStatus)

Unfortunately, this test does not run successfully with localstack (yet!). The assertion to check if the bucket has versioning enabled ignores the endpoint set on terraform and calls the real aws endpoint, therefore failing the assertion. This issue is actually open on the terratest repository as you can see below:

Assert failing if you are using local configuration for testing (i.e: localstack locally) #494

ffernandezcast posted on Apr 07, 2020

I'm using localstack locally to test my terraform code:

This is my provider configuration

provider "aws" {
  access_key                  = "mock_access_key"
  region                      = "eu-west-2"
  s3_force_path_style         = true
  secret_key                  = "mock_secret_key"
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true

  endpoints {
    apigateway     = "http://localhost:4567"
    cloudformation = "http://localhost:4581"
    cloudwatch     = "http://localhost:4582"
    dynamodb       = "http://localhost:4569"
    es             = "http://localhost:4578"
    firehose       = "http://localhost:4573"
    iam            = "http://localhost:4593"
    kinesis        = "http://localhost:4568"
    lambda         = "http://localhost:4574"
    route53        = "http://localhost:4580"
    redshift       = "http://localhost:4577"
    s3             = "http://localhost:4572"
    secretsmanager = "http://localhost:4584"
    ses            = "http://localhost:4579"
    sns            = "http://localhost:4575"
    sqs            = "http://localhost:4576"
    ssm            = "http://localhost:4583"
    stepfunctions  = "http://localhost:4585"
    sts            = "http://localhost:4592"
  }
}

I.E: I'm using aws.AssertS3BucketExists method

But I constantly receiving an error about the credentials (In my case I have fake one and receive 403). This mean the aws import dependency is not using the terraform provider configuration and continue using the default aws cli config credentials.

I have checked the code and I realized the module https://github.com/gruntwork-io/terratest/blob/master/modules/aws/auth.go doesn't have the option to use local custom services endpoint

View on GitHub

There's already an ongoing fix for this problem, which I tried following the instructions described in detail here, which will basically require you to set again the endepoints on the terratest code like this:

var LocalEndpoints = map[string]string{
    "iam":            "http://localhost:4566",
    "s3":             "http://localhost:4566",
    "ec2":             "http://localhost:4566",
}
aws.SetAwsEndpointsOverrides(LocalEndpoints)

On top of this "bump on the road", I also had other issues such as some flakiness on the tests because of the container getting stuck during resource creation and, in the end, I felt that I was spending more time dealing with localstack than writing new tests and taking value out of it. I hope that in the future it will be more reliable but for now, if you wish to use terratest to test your infrastructure, I recommend you to run it against a sandbox AWS account where you can run aws-nuke or cloud-nuke once in a while.

Checkov

bridgecrewio / checkov

Prevent cloud misconfigurations and find vulnerabilities during build-time in infrastructure as code, container images and open source packages with Checkov by Bridgecrew.

Just like TFLint, Checkov is also a static analysis tool that has more than 400 rules focused on security and compliance best practices for AWS, Azure and Google Cloud.

It can report situations such as the following, where I created an ebs volume without having encryption enabled:

Or this one, where I created an S3 bucket without access logging, that could be handy for further analysis of suspicious assesses to the bucket:

What I enjoy most in this tool is that every issue reported has a url with the rationale, so you can learn why it is important to fix and how to do it. Kudos to BridgeCrew for the great work!

Infracost

infracost / infracost

Cloud cost estimates for Terraform in pull requests💰📉 Shift FinOps Left!

This tool really got me "wow, this is cool" 😲. Basically, Infracost estimates the costs automatically in every pull request, based on the changes you did on your terraform code. How many times you didn't realize that the changes you were doing were going to cost you 2x more in the end of the month? Or you suspected that it would be expensive but you had to jump to AWS Pricing calculator and do the math?
In the following example, I just increased the instance type from t3.micro to t3.xlarge and automatically (powered by this infracost github action), I got a comment on my pull request telling me that the monthly bill would increase from ~114$ to ~239$:

That's it! 👋

If you're interested and want to quickly try these tools, I set up a Github Actions workflow with all of them in this repository, feel free to use it!

bmbferreira / awesome-terraform-pipeline

Sample terraform repo with multiple resources and a workflow that uses tools to automate cost management, security checks, best practices, tests, documentation and more!

I hope that you find at least some of these tools useful and please tell me on the comments what other tools are you using in your day-to-day to improve your work with terraform! 😄

From Postgres RDS to Aurora

Bruno Ferreira — Tue, 17 Dec 2019 23:27:26 +0000

It is no secret that Codacy's underlying infrastructure relies heavily on AWS and, one of the services that we use most, is AWS's Relational Database Service (RDS) to host our Postgres databases.
In this blog post we'll describe our experience migrating a critical database with 18Tb of data from RDS to Aurora, with minimal downtime. Hopefully, this can be useful to you if you're dealing with a similar task. 😄

Analysis workflow, behind the scenes

Let's start with a brief description of the analysis process on Codacy, from the git providers (GitHub, Gitlab or Bitbucket) to the storage of results on the databases.
Whenever a commit or Pull/Merge request is done over a repository, it is detected on our side through the use of hooks [1]. Our internal component named "Repository Listener" is then responsible to request an analysis [2] which will be queued and picked by our "Workers Manager" that will launch the analysis over the repository [3]. The analysis will be done by the short-living "Workers" that clone the repository [4], execute multiple linters (eslint, pylint, checkstyle, etc.) and the data of the final results are then saved mostly in two different databases [5]: the "Results DB", which stores data related to the issues detected on the repositories, and the "Analysis DB", which stores data related to commits, pull requests, etc.

Data overload - Results DB

Codacy was born in 2012 and, by January 2017, the "Results DB" had approximately 6Tb of data, being the limit on postgres RDS at the time of 8Tb. Since we were having more and more analysis being done on Codacy, we were running out of time.
To solve this issue, by July 2017, a new database named "Results DB 2017" was created and new results started to be saved on this new database.

And here's a "funny" story: a few months later, AWS announced support for up to 16Tb of data on RDS, but that was already too late. 😥

Data overload - Analysis DB

Time goes by and, by September 2018, we have the same problem on the other database, the "Analysis DB". It had almost 14Tb of data, growing approximately 500Gb per month and the limit was 16Tb so, the math here is simple, we had 4 months to do something. In this case, because of the schema on this database, a simple solution similar to the one taken previously was almost impossible. 😨

Aurora

While researching for solutions to this problem, Aurora started to show up as a good candidate, since it automatically grows storage as needed, up to 64Tb. Among other benefits announced on Aurora's product page, the ease of migration from RDS and the performance benefits, caught our attention.
Regarding performance benefits, Aurora's official docs claimed up to a 3x increase in throughput performance over stock PostgreSQL 9.6 on similar hardware, but there were also testimonials that the performance increased 12x, just by migrating to Aurora.

Migration process

After deciding to go for Aurora, we started to investigate on how to do the migration with minimal downtime. We had 3 different options for it:

Use a database snapshot from the original database and migrate it to Aurora. This option would require a huge downtime to complete (according to AWS support, 12 hours per Tb). 👎
Use AWS Database Migration Service (DMS). This option, on the positive side, would not require any downtime during the migration and it would automatically upgrade our database from Postgres 9.4 to 9.6, which was a requirment since 9.6 is the oldest version compatible with Aurora. However, on the negative side, despite having no downtime, the whole process would still be very slow, since it would take around 12 hours per Tb and indexes would need to be recreated manually because DMS doesn't propagate them. This would be problematic since it could take a large amount of time to recreate the indexes for our biggest tables. 👎
Use an Aurora Read Replica. This looked the best solution to do the migration because it would require minimal downtime and, during dry runs, we verified that it was taking between 2 and 3 hours per TB, which was acceptable since we could leave Codacy running during that period on the old database, without having any impact. We would also need to upgrade manually the database to 9.6, but the process seemed to be easy, with minimal downtime. 👍

1st round

As previously referred, we had to upgrade our database from Postgres 9.4 to 9.6. This basically consisted in running two commands using the AWS CLI, as described here, being the first the upgrade to 9.5 and then to 9.6. Since the whole process for the upgrade could take around 1 hour of downtime, we planned to do it on a Saturday morning when we had less people running analysis, warned our users some days before, deployed Codacy's maintenance page and started the upgrade.

Oops! 😱 After the upgrade, Codacy was looking really slow, taking several minutes to load data from the database. After some minutes, completely clueless, we decided to revert everything and investigate the problem later. 😞
After taking a better look to the aws official docs, we noticed a useful recommendation about running the ANALYZE operation to refresh the pg_statistic table. This internal table is important for query performance since it stores statistics that are used by the query planner (you can read more about it here).
Before trying this again, we tested the ANALYZE operation on a test database and noticed that it could take several hours to finish. Therefore, we started to research ways to run it faster and we ended up using the vacuumdb utility. Here's the command that we ran:

vacuumdb -Ze -h analysisdb --analyze-in-stages -j 20

These are the important parts:

analyze-in-stages: some statistics are created as fast as possible, to make the database usable and then, the full statistics are be produced in the subsequent stages. There are 3 stages in total and after the first one, we verified that the database had already an acceptable performance. And it took only some seconds to finish this first stage!
-j 20: it will start 20 jobs to run the analyze command in parallel, reducing the time of the processing.

2nd round

After successfully upgrading the database, the second, and final, step was to create an Amazon Aurora PostgreSQL Read Replica using the previous upgraded database as a source. It took around two entire days to the lag between the RDS instance and the Aurora Read Replica got close to 0.

When it happened, we promoted the read replica to master, breaking the link with the source database and therefore, stopping the replication. In this last step we had to stop every write operation on the source database before doing the promotion, to guarantee the full replication of the two instances. Again, we had to do this last step in another Saturday morning, since it required some minutes of downtime.

After removing the maintenance page, we started testing Codacy, now running on Aurora, and we faced performance issues, again. Our first thought was that we would have to run the ANALYZE operation again, and so we did. But it was still slow. 😨
We decided to execute an EXPLAIN to show the execution plan of a select statement over one of the most used tables and we noticed that the query planner was mistakenly preferring sequential scans over index scans. After some more investigation in place we found the root cause of this issue: with the migration we "lost" the previous parameter group that we were using for RDS and started using the default. In that parameter group, we had the random_page_cost set to 1, provided that the default PostgreSQL value of 4 is tuned for HDD, where random access to disk is more expensive than sequential access. Since Aurora is using solid state drives behind the scenes we had to reduce this parameter to 1 again to start using the indexes instead of doing sequential scans. 🎊

Here's another "funny" story: while we were struggling with the migration and almost running out of time, AWS announced another storage limit increase on RDS for up to 32Tb. However, this didn't came out of the box and the AWS support team had to upgrade the underlying filesystem of the database from 32 to 64 bits to be able to use more than 16Tb. We had to stop any transactions on the database for this upgrade to happen on their side and the whole operation took around 2 hours of downtime. Despite this limit increase, we were already commited with the migration to Aurora and decided to go for it. This also gave us a bit more time to prepare it and we ended up doing it only on May 2019, with a bit more than 18Tb on the database.

Wins

Besides having automatic storage provisioning as needed, up to 64Tb, what are the other positive outcomes of this migration?

Performance Insights

Previously, we were using pgbadger to generate HTML reports from the logs and get useful statistics about query performance on the database. With Aurora we got Performance Insights out of the box and we are now able to get useful, always updated, statistics straight from the AWS Management Console.

Write latency

We also had some performance improvements, especially regarding to the latency of the write operations.

As it is visible, the latency was slowly increasing on RDS, reaching an average of 23ms on November 2018.

On Aurora, the write latency is now in the order of microseconds. Not bad! 😄

Future

And that's it! What should we be doing next?

Aurora Serverless

Databases on AWS can get expensive, especially with the large amounts of data that we have on some databases. Using Aurora Serverless, since it is able to seamlessly scale compute and memory capacity as needed, we could reduce the resources for our database automatically when we have less load on it, for instance, during weekends.

Consider different types of databases, other than Postgres

Postgres is great, but we are using it for almost everything and, in some situations, maybe we shouldn't. To be able to properly scale horizontally, we should start considering other types of databases, like NoSQL databases such as MongoDB, DocumentDB or CouchDB.

DNS Hell in k8s

Bruno Ferreira — Sun, 03 Nov 2019 00:10:42 +0000

Here at Codacy, everyone's been working really hard in the last few months to move all of our services to kubernetes. And it has been a bumpy road... From having to run a nfs-server provisioner, to be able to share files between pods, to launching our own scheduler to avoid scaling issues, we can say that we're hitting some interesting problems along the way, giving us the opportunity to do and learn new things everyday - which is pretty cool.

In this blog post I will talk about one of the most common issues that almost everyone, including us, seems to struggle with when moving to k8s: DNS intermittent delays

One of our components, we simply call it worker, runs as a short-living pod and is responsible to spawn other pods that run the analysis tools (eslint, pylint, etc.), gather their data and save it.
Because of some pitfalls on our architecture, we agreed that this was one of the toughest parts of the system to put on k8s and therefore, we decided to tackle it first.
During one of the first attempts to put these workers running on k8s in production, after some minutes we noticed a problem that we didn't get during our tests on the development and staging environments.
Some workers were throwing UnknownHostExceptions while trying to access graylog and the databases, that were running outside the cluster. This seemed to increase when the number of running workers increased on the nodes. After some research, we found lots of users complaining about DNS intermittent delays of ~5s in this github issue. This was a problem, since the default timeout for the DNS is 5 seconds. We went through almost all of the solutions referred in the issue thread:

"arp table overflow on the nodes (arp -n showing more than 1000 entries). Increasing the limits solved the problem"

We checked this on our nodes and this was not a problem for us, since we had around 50 entries in all of the nodes.
"dnsPolicy: Default works without delays"

Well, actually, this was a bit confusing for us because despite this dnsPolicy being called Default, it is not the default policy in k8s. The default DNS policy is "ClusterFirst", i.e., "any DNS query that does match the configured cluster domain suffix, is forwarded to the upstream nameserver inherited from the node", while the "Default" DNS policy just "inherits the name resolution configuration from the node that the pods run on". We tested this, by running some test pods trying to resolve www.google.com and this configuration decreased from an average of ~5s to ~2s... It is still a lot of time to resolve the name but we then decided to try it on our workers. However, after some minutes we still got the UnknownHostExceptions while trying to access the external services on startup.
"Use fully-qualified names when possible"

As explained by Marco Pracucci in his blog, in k8s if you're trying to resolve a name of something outside of the cluster, with the default configuration on /etc/resolv.conf, any request for resolution that contains fewer than 5 dots will cycle through all of the search domains as well in an attempt to resolve. For example, to resolve codacy.com, codacy.com.kube-system.svc.cluster.local., codacy.com.svc.cluster.local., codacy.com.cluster.local., codacy.com.ec2.internal. and finally codacy.com. must be looked up, for both A and AAAA records. While this didn't solve our issue, it was something good to be aware to tweak our apps and get a better performance.
"using the option single-request-reopen on /etc/resolv.conf fixed the problem"

Fortunately, the guys from weaveworks and also from XING investigated this problem in depth and explain in detail why this can be a solution. Basically, the root cause of these delays is the Linux connection tracking mechanism, aka conntrack, which is implemented as a kernel module and is inherently racy. According to the man page for resolv.conf, the single-request-reopen option enables sequential lookups using different sockets for the A and AAAA requests, reducing the race conditions. We also tried this but, after some minutes, we continued to see workers failing and while running conntrack -S on the nodes, the insert_failed counter was still increasing.
"use tcp for DNS requests"

Since UDP is lossy, one of the UDP packets might get dropped by the kernel due to the races, so the client will try to re-send it after a timeout (5 seconds). Therefore, we decided to try using TCP for DNS as a workaround for this issue, as it was also one of the workarounds suggested by Pavithra Ramesh and Blake Barnett in their recent talk at Europe's Kubecon. Despite this being a bit slower because we are now using TCP, it actually solved the problem for us and the pods stopped failing.

Some good news is that these DNS problems caused by race conditions in the connection tracking mechanism (fun fact: this is briefly "documented" in Linux's source code) already have two patches to fix it (if you're brave enough, you can take a look at them here and here).
However, the most recent patch is only available since version 5 of the linux kernel and it's not always possible to control the kernel version of the nodes where your pods will run. In our case, since we are running on EKS, we are using Amazon Linux 2 and the most recent update (7/18/2019) only supports the 4.19 kernel, which only contains one of the patches.

Meanwhile, the most proper solution seems to be the usage of a NodeLocal DNScache (beta in k8s v1.15), already detailed in the k8s official documentation. This solution aims to improve the overall DNS performance on the cluster by running DNS caching agents on every node as a DaemonSet so pods can reach out to these agents running on the same node, thereby reducing the number of upstream trips which would still use conntrack and increase the probability of being impacted by the race conditions previously referred.

Other solutions, such as using a sidecar on every pod running tc to delay DNS packages, we actually didn't try since they seemed more complex and required more configuration. We also discarded every solution that required any modification on the nodes configuration since we can also deploy our application on-premises and, in this case, the cluster nodes are managed by our customers.

In the end, after some research and the typical "trial and error", we were able to find a workaround for this issue until we get to a proper solution in the future, enabling us to proceed and hit the next problem.
😄

See the original article here

Putting BDD in practice using Scala

Bruno Ferreira — Sat, 08 Sep 2018 19:22:51 +0000

This article aims to give a brief explanation about what Behaviour-Driven Development (BDD) is and how it can be used to fill the information gap between stakeholders and development teams, ensuring everyone (technical or not) is involved in the project’s progress.

The first time I had contact with this approach of software development was a few years ago during a Massive Open Online Course (MOOC) and, I must warn you, I've never applied it in production. Nevertheless, I'm fully aware this is not the holy grail and it might not work so well as advertised but, the idea of having everyone in the team (not only developers) collaborating on the development process, sounded very romantic to me.

In the past few months, I've been working with Scala and I've been thinking about what would be the best way to apply BDD using the most popular testing tools in the Scala environment (ScalaTest and Specs2) and the most popular BDD tool: Cucumber. In this article I will compare these three tools using a very simple example.

What's BDD?

BDD can be described, in very simplistic terms, as an extension for Test-Driven Development (TDD).
The typical TDD cycle, aka Red/Green/Refactor, starts with the development of a failing test. The next step is to write the code to make this test pass and finally, refactor the code in order to improve its readability and maintainability.

In comparison, the BDD cycle is almost identical to the latter except the first step is replaced by the writing of a specification, rather than the development of a test. Meaning this first step can be done by a non-technical person, for instance a member of the business team or a functional analyst with an understanding of how the system should behave. The best part is this specification can be used for tests automation in the development process.

BDD can also be described as the intersection between Domain-Driven Design (DDD) and TDD.
Domain-Driven design is an approach to software development aiming to deal with projects with complex business rules where collaboration between developers and domain experts (typically, this role is assumed by the client) is necessary to set a common language and a domain model that can be translated into very specific and detailed requirements. In BDD, this specific requirements are used to drive the development process in the first step of its cycle.

BDD in practice

Obviously, BDD can be put into practice without using a specific tool or framework, you just have to get people to collaborate, record that collaboration in some form of specification, and then automate that specification to drive out the implementation. However, there are tools that can help significantly in the automation process, so we don't have to find a new way to extract values from the requirements to automate tests.

Cucumber

Cucumber is probably the most popular tool when it comes to BDD and it was the first one I worked with. It is advertised as being capable to ease tests automation using executable specifications and therefore, generating living documentation that is easily understood by everyone.

It was originally developed in Ruby, but it is also officially available for Java and Javascript.

The specifications are written in Gherkin, a simple set of grammar rules generating plain text structured enough for Cucumber to understand. Here's a simple example of a Bank Account feature specification from where the holder can withdraw cash:

Feature: Account Holder withdraws cash

  Scenario: The account has sufficient funds
    Given an account with a balance of $100.5
    When the Account Holder requests $20.25
    Then the account balance should be $80.25

  Scenario: The account has insufficient funds
    Given an account with a balance of $80.25
    When the Account Holder requests $100.75
    Then the Account Holder should be notified that overdrafts are not permitted

Gherkin documents are stored in .feature text files and it supports different dialects, so you can even write your specifications in Esperanto!

The Scala implementation for the previous feature is something like this:

class BankAccount(val balance: Double) {

  def debit(amount: Double): Either[String, BankAccount] = {
    if (amount < 0.0) {
      Left(s"The debit amount must be > 0.0")
    } else if (balance - amount < 0.0) {
      Left("Overdrafts are not permitted")
    } else {
      Right(new BankAccount(balance - amount))
    }
  }

  def credit(amount: Double): Either[String, BankAccount] = {
    if (amount < 0.0) {
      Left("The credit amount must be > 0.0")
    } else {
      Right(new BankAccount(balance + amount))
    }
  }
}

I believe the code doesn't require any detailed explanation. It basically enables the creation of a BankAccount instance with a given amount of cash that can be debited and credited, but overdrafts are forbidden.

Using this library I managed to automate the execution of the previous feature in a Scala project. It looks like this:

class StepDefinitions extends ScalaDsl with EN {

  var bankAccount: BankAccount = _
  var result: Either[String, BankAccount] = _

  Given("""^an account with a balance of \$(.+)$""") { (balance: Double) =>
    bankAccount = new BankAccount(balance)
  }

  When("""^the Account Holder requests \$(.+)$""") { (amount: Double) =>
    result = bankAccount.debit(amount)
  }

  Then("""^the account balance should be \$(.+)$""") { (expectedResult: Double) =>
    assert(result.right.exists(_.balance == expectedResult))
  }

  Then("""^the Account Holder should be notified that overdrafts are not permitted$""") { () =>
    assert(result.left.exists(_.equalsIgnoreCase("overdrafts are not permitted")))
  }
}

Basically, we're just extracting the values from each one of the specification sentences and doing operations and assertions with those values.

As you might have noticed (codacy obviously noticed it), I had to sacrifice immutability, one of the building blocks of functional programming, by using two mutable variables to be able to do the assertions in the last steps. It would be cleaner if each step could return a value to be used in the next steps and do the assertions in the end.
I wasn't the only one that noticed this limitation and some issues were already reported to make this more functional. However, it seems these improvements won't happen in the near future as it was announced the support for JVM languages was dropped, and this obviously includes Scala.

ScalaTest

ScalaTest is probably the most popular testing tool in the Scala ecosystem. It provides some traits facilitating BDD style, enabling a more grammatical structure in order to write tests as specifications.

The tests for the BankAccount implementation previously shown looks like this:

class BankAccountSpecs extends FeatureSpec with GivenWhenThen {

  feature("Account Holder withdraws cash") {
    scenario("The account has sufficient funds") {
      val balance = 100.5
      Given(s"""an account with a balance of $$$balance""")
      val bankAccount = new BankAccount(balance)
      val withdrawAmount = 20.25
      When(s"the Account Holder requests $$$withdrawAmount")
      val resultBankAccount = bankAccount.debit(withdrawAmount)
      val expectedBalance = 80.25
      Then(s"the account balance should be $$$expectedBalance")
      assert(resultBankAccount.right.exists(_.balance == expectedBalance))
    }

    scenario("The account has insufficient funds") {
      val balance = 80.25
      Given(s"""an account with a balance of $$$balance""")
      val bankAccount = new BankAccount(balance)
      val withdrawAmount = 100.75
      When(s"the Account Holder requests $$$withdrawAmount")
      val resultBankAccount = bankAccount.debit(withdrawAmount)
      val expectedMessage = "overdrafts are not permitted"
      Then(s"the Account Holder should be notified that $expectedMessage")
      assert(resultBankAccount.left.exists(_.equalsIgnoreCase(expectedMessage)))
    }
  }
}

The main difference I noticed when using ScalaTest compared with Cucumber is the specification text is inlined with the code and the Given/When/Then statements are basically just printed logs to the console, giving a greater level of granularity than just the test names. Therefore, the advantage of having the specification completely separated from the code is lost.

Specs2

Specs2 is another popular testing tool in the Scala ecosystem and it's my favourite one. It also provides a trait enabling the BDD style of using specifications for tests and, in my opinion, this is the best option to practice BDD in Scala. Here's the code for the successfull withdraw scenario:

class BankAccountWithSufficientFunds extends Specification with GWT {

  def is =
    s2"""
     Given an account with a balance of $$100.5     ${bankAccount.start}
     When the Account Holder requests $$20.25
     Then the account balance should be $$80.25     ${bankAccount.end}
    """

  private val bankAccount =
    Scenario("bankAccount")
      .given(aBankAccount)
      .when(aDouble) { case debit :: bAccount :: _ => bAccount.debit(debit) }
      .andThen(aDouble) {
        case expectedBalance :: bAccount :: _ =>
          bAccount.right.exists(_.balance == expectedBalance)
      }
}

As it is visible, the specification text is also not completely separated from the code but it is not inlined. It is in a separated string that could be easily extracted from some other place. In this string, we have to specify where the scenario starts, where it ends, and every line between these two points must correspond to a given, when or andThen call on the scenario.

We also have to write some custom step parsers for the balance and withdraw values and to instantiate the BankAccount when the values are extracted from the specification string. Here's the code:

object CustomStepParsers extends StepParsers {

  private val regex = ".*\\$(.*)$"

  def aDouble: StepParser[Double] = readAs(regex).and((s: String) => s.toDouble)

  def aBankAccount: StepParser[BankAccount] =
    readAs(regex).and((s: String) => new BankAccount(s.toDouble))

}

And finally, the code for an unsuccessfull withdraw operation, where the account holder is notified overdrafts are not permitted:

class BankAccountWithInsufficientFunds extends Specification with GWT {

  def is =
    s2"""
     Given an account with a balance of $$80.25                                        ${bankAccount.start}
     When the Account Holder requests $$100.75
     Then the Account Holder should be notified that overdrafts are not permitted      ${bankAccount.end}
    """

  private val bankAccount =
    Scenario("A Bank Account with Insufficient Funds")
      .given(aBankAccount)
      .when(aDouble) { case debit :: bAccount :: _ => bAccount.debit(debit) }
      .andThen() {
        case insufficientFunds :: bAccount :: _ =>
          bAccount.left.exists(msg => insufficientFunds.toLowerCase.contains(msg.toLowerCase))
      }
}

Conclusion

I think BDD, if correctly applied and with everyone commited, can be a very efficient approach to software development. Especially for projects where a significant number of team members don't have a deep understanding of the business logic. Having a way to formally specify what the system should do and how it should behave and use this specification for testing and documentation, can avoid many issues during the development process.

About the tools comparison, despite Cucumber being the standard tool to practice BDD, I think Specs2 was definetely the best testing tool for this simple example in Scala.

The project with the code used for this blog post is available in GitHub, feel free to check it out.