Forem: Martijn Versluis

TIL about VCR

Martijn Versluis — Mon, 04 Nov 2019 00:00:00 +0000

A while ago I contributed a change to the well-known Ruby gem VCR. You can read all about it in this pull request, but basically it allows you to prevent interactions from being recorded until your test passes.

To be able to make this change, I obviously had to dive into (a portion of) the source code to understand what I had to change. By reading the code I discovered some VCR features I did not know, so they might be new to you too. Here are six things you might not know about VCR.

1. You can change the serialization

VCR is well known for the YAML files it uses to persist cassettes. By default that isn't a bad format: it is readable and Ruby developers are generally familiar with it. But if you would want to change the serialization, you can set it in the configuration. That could come in handy, for example if you're exporting the cassettes to a separate system that does not support YAML.

VCR.configure do |config|
  config.default_cassette_options = {
    serialize_with: :json
  }
end

The available serializers are: :yaml, :syck or :psych for YAML format, :json for JSON format (powered by MultiJson) or :compressed, which adds compression to the :yaml serializer. You can even create and use your own serializer. It should respond to file_extension, serialize and deserialize:

module VcrXmlSerializer
  def file_extension
    'xml'
  end

  def serialize(hash)
    Qyoku.xml(hash, key_converter: :none)
  end

  def serialize(xml)
    Nori.new(parser: :nokogiri).parse(xml)
  end
end

VCR.cassette_serializers[:xml] = VcrXmlSerializer

VCR.configure do |config|
  config.default_cassette_options = {
    serialize_with: :xml
  }
end

2. You can change the persistence

By default VCR generates files, which works fine for most use cases. You can implement your own storage though, for example to write all interactions to a database:

module VcrDatabasePersister
  def []=(file_name, content)
    db.execute("INSERT INTO http_interactions (name, content) VALUES (?, ?)", [name, content])
  end

  def [](file_name)
    db
      .get_first_row("SELECT content FROM http_interactions WHERE name = ?", [name])
      .fetch('content')
  end

  private

  def db
    SQLite3::Database.new("test.db")
  end
end

VCR.cassette_persisters[:database] = VcrDatabasePersister

VCR.configure do |config|
  config.default_cassette_options = {
    persist_with: :database
  }
end

3. You can hook into events

VCR supports a number of hooks that allows you to change data before it is used or persisted by VCR:

before_record: change the HTTP interaction before it is recorded
before_playback: change a cassette before it is registered for use
before_http_request, after_http_request and around_http_request to perform an action before and/or after a HTTP request is performed (either stubbed or real requests).

For example, you can wrap a timeout around each request:

VCR.configure do |config|
  config.around_http_request do |request|
    Timeout::timeout(500, &request)
  end
end

See: https://relishapp.com/vcr/vcr/v/5-0-0/docs/hooks

4. You can use placeholders

You can use the config option define_cassette_placeholder, aliased as filter_sensitive_data to replace data before it is written to a file. You can, for example, use it to filter passwords:

VCR.configure do |config|
  config.filter_sensitive_data('<MASKED>') { ENV['API_PASSWORD'] }
end

See: https://relishapp.com/vcr/vcr/v/5-0-0/docs/configuration/filter-sensitive-data

5. You can use cassette data inside your test

One use case is when a HTTP request is time sensitive (eg. when signing a request). When using use_cassette the cassette object is passed to the block, allowing to grab the recording time:

VCR.use_cassette('example') do |cassette|
  request = build_api_request
  timestamp = cassette.originally_recorded_at || Time.now
  request.sign_with('some secret key', timestamp)
end

See: https://relishapp.com/vcr/vcr/v/5-0-0/docs/cassettes/freezing-time

Besides the recording time, the cassette exposes more metadata like the name, configuration and the file path. For more on the public API of Cassette see: https://rubydoc.info/gems/vcr/VCR/Cassette.

6. You can use ERB in cassettes

Yes, you can use ERB in a cassette. There are probably not many cases where you should do it, but here is an example just to show what is possible. We use ERB to inject an API key in a cassette:

---
http_interactions:
- request:
    method: get
    uri: http://example.com/time?api_key=<%= api_key %>
    body:
      encoding: UTF-8
      string: ''
    headers: {}
  response:
    status:
      code: 200
      message: OK
    headers:
      Content-Type:
      - application/json;charset=UTF-8
      Content-Length:
      - '47'
    body:
      encoding: UTF-8
      string: "{\"time\":\"2019-08-08T20:08:25.604188+02:00\"}"
    http_version: '1.1'
  recorded_at: Tue, 01 Nov 2011 04:58:44 GMT
recorded_with: VCR 2.0.0

VCR.use_cassette('time', erb: { api_key: ENV['TIME_API_KEY'] }) do
  response = Net::HTTP.get_response('example.com', "/time?api_key=#{ENV['TIME_API_KEY']}")
  puts "Response: #{response.body}"
end

See: https://relishapp.com/vcr/vcr/v/5-0-0/docs/cassettes/dynamic-erb-cassettes

That's all, folks!

Hopefully these tips were useful to make testing your application easier. Do you need more help? At Kabisa we know a lot about writing good tests. Leave us a message if you would like to get in touch.

Matching multipart request bodies with VCR

Martijn Versluis — Tue, 20 Aug 2019 13:35:23 +0000

When using VCR to test a HTTP multipart request, you might experience mismatching cassettes because of multipart boundaries. Multipart boundaries are most often randomly generated by the HTTP library or browser, so when VCR tries to match a multipart request with a cassette, the boundaries will be different and the match will fail.

However, it is quite simple to replace the random boundaries by fixed ones before matching requests. A multipart request will have a Content-Type header, which not only makes the request recognizable as multipart request but also specifies the exact boundary that is used. In this case I took a cassette from a Ruby project, so the Content-Type looks like this:

multipart/form-data; boundary=----RubyFormBoundaryTsqIBL0iujC5POpr

Given this header we should be able to simply replace the boundaries with a fixed value. VCR has a configuration option called match_request_on, which is an array that accepts symbols for predefined request matchers (see https://relishapp.com/vcr/vcr/v/5-0-0/docs/request-matching) or a callable (Proc, Lambda). I'm not a real fan of using procs for this but as long as we create an object that responds to call we will be fine:

class VCRMultipartBodyMatcher
  MULTIPART_HEADER_MATCHER = %r{^multipart/form-data; boundary=(.+)$}

  def call(request_1, request_2)
    normalized_multipart_body(request_1) == normalized_multipart_body(request_2)
  end

  private

  def normalized_multipart_body(request)
    content_type = (request.headers['Content-Type'] || []).first.to_s

    return request.body unless multipart_request?(content_type)

    boundary = MULTIPART_HEADER_MATCHER.match(content_type)[1]
    request.body.gsub(boundary, '----RubyFormBoundaryTsqIBL0iujC5POpr')
  end

  def multipart_request?(content_type)
    return false if content_type.empty?

    MULTIPART_HEADER_MATCHER.match?(content_type)
  end
end

Now we can tell VCR to use our custom matcher:

VCR.configure do |config|
  config.default_cassette_options = {
    match_requests_on: [:method, :uri, VCRMultipartBodyMatcher.new]
  }
end

Now, when running the test the boundaries of the request and the cassette will be equal, so the cassette will only mismatch if the request parameters differ. 👍

Do you need help with testing your application? At Kabisa we know a lot about writing good tests. Leave us a message if you would like to get in touch.

Matching multipart request bodies with VCR

Martijn Versluis — Tue, 20 Aug 2019 00:00:00 +0000

multipart/form-data; boundary=----RubyFormBoundaryTsqIBL0iujC5POpr

class VCRMultipartBodyMatcher
  MULTIPART_HEADER_MATCHER = %r{^multipart/form-data; boundary=(.+)$}

  def call(request_1, request_2)
    normalized_multipart_body(request_1) == normalized_multipart_body(request_2)
  end

  private

  def normalized_multipart_body(request)
    content_type = (request.headers['Content-Type'] || []).first.to_s

    return request.body unless multipart_request?(content_type)

    boundary = MULTIPART_HEADER_MATCHER.match(content_type)[1]
    request.body.gsub(boundary, '----RubyFormBoundaryTsqIBL0iujC5POpr')
  end

  def multipart_request?(content_type)
    return false if content_type.empty?

    MULTIPART_HEADER_MATCHER.match?(content_type)
  end
end

Now we can tell VCR to use our custom matcher:

VCR.configure do |config|
  config.default_cassette_options = {
    match_requests_on: [:method, :uri, VCRMultipartBodyMatcher.new]
  }
end

Now, when running the test the boundaries of the request and the cassette will be equal, so the cassette will only mismatch if the request parameters differ. 👍

Do you need help with testing your application? At Kabisa we know a lot about writing good tests. Leave us a message if you would like to get in touch.

How to handle JavaScript/frontend errors with Selenium Webdriver

Martijn Versluis — Wed, 24 Apr 2019 00:00:00 +0000

Dotfiles are text files that live in your $HOME directory and help to configure your terminal and programs. When you are moving to a new computer, having dotfiles in a git repository makes copying these configurations a breeze.

There are quite a lot dotfile managers written in various languages. Basically these dotfile managers create symlinks to the dotfiles in the home directory.

But do we really need a dotfile "manager" to symlink some files?

Meet GNU Stow 🔨

GNU Stow is a symlink farm manager, that makes packages of software appear in different locations on the filesystem.

We can install the software with brew:

$ brew install stow

Making yourself at home 🏠

Let's create a repository dotfiles and create our first dotfile that will be linked to the home directory.

$ mkdir ~/dotfiles
$ cd ~/dotfiles
$ git init .
$ echo "A simple dotfile configuration." > .mydotfile
$ git add .mydotfile
$ git commit -m "Create dotfiles repository"

Now we have a repository, you can easily transfer. Next we need to symlink mydotfile into the home directory.

$ cd ~
$ ln -s ~/dotfiles/mydotfile .mydotfile
$ ls -la | grep mydotfile
lrwxr-xr-x    1 e  ff      20B 29 May 11:11 .mydotfile -> dotfiles/.mydotfile

Now we have symlinked the dotfile in the repository to our home directory. The symlink, which is a reference to the original file, can be edited and viewed. However when you delete the symlink, the original file remains as is.

We can automate the process of manually symlinking the files with Stow. As an example, let's create the following structure to setup neovim configuration files:

$ tree -a ~/dotfiles
.
└── neovim
        └── .config/nvim
        └── .init.vim
    └── .vimrc

With Stow you can link those files by simply running one command.

$ cd ~/dotfiles
$ stow neovim

When we look in our home directory we see the following.

$ ls -la ~ | grep vimrc

lrwxr-xr-x     1 e  ff    22B May 29 11:12 .vimrc ->  dotfiles/neovim/.vimrc

$ ls -la ~/.config/nvim/

lrwxr-xr-x   1 e  ff    43B May 29 11:36 init.vim -> ../../dotfiles/neovim/.config/nvim/init.vim

Let's reflect on what happened 📝

We ran stow with our neovim dotfiles directory.
Stow finds the .vimrc and the subdirectory .config/nvim/ with an .init.vim.
Stow creates a symbolic link one directory above the current location and also creates the structure in the home directory for ~/.config/nvim/ and symlinks .init.vim

Stow makes an assumption that you want to symlink the files one directory above the directory where you have executed the stow command. Having the git repository in ~/dotfiles makes stow create the correct symlinks for us, in the $HOME directory.

In case you want to remove the symlink, you can type stow -D neovim, which will unlink the files. Stow will warn you in case you are about to overwrite existing directories or files.

I am using the described technique to symlink my own dotfiles.