Forem: Jason Dinsmore

Active Record Encryption in Rails 7

Jason Dinsmore — Wed, 07 Jul 2021 04:47:04 +0000

Rails 7 will be introducing a very cool new feature for ActiveRecord - application-level encryption, summoned by the mighty encrypts declaration in a model. This new feature provides a layer of encryption that sits between our application code and the database. In essence, when our data using Active Record Encryption has been loaded into an AR object, it will be unencrypted, and when it is sitting in the database it will be encrypted.

In this post, we will take a high-level look at how to use this functionality, discuss some cool things it can do, and acknowledge some of its limitations.

Before we dive in, I would be remiss if I didn't point to the excellent documentation in the Rails guide.

For the duration of the post, I will refer to the feature as encrypts for brevity (and also because I am not sure what else to call it 😉).

Backstory

encrypts was merged into Rails in PR 41659 by @jorgemanrubia. In the PR description he states that the functionality is an extraction from HEY, which had the implementation reviewed by a security firm. If you are interested in hearing more of the story behind the feature, I'd recommend Jorge's blog post on the subject - it's an interesting read.

Deterministic Sidebar

Let's take a quick second to discuss the difference between deterministic and non-deterministic encryption. It's really pretty simple, but is a central tenet in understanding how to use encrypts.

Think of encryption as a function that is applied to some input (text), resulting in some output (encrypted text).

If the function is deterministic, then any time the function is applied to the same text, it will output the same result.

If the encryption function is non-deterministic, then we can't predict what the output will be the second time we encrypt a given value. In theory, it is possible we would get the same result we got the first time, but the odds of that happening are very, very low. In the context of encrypts with its default non-deterministic configurations, subsequent encryptions of the same plaintext are almost certain to produce different ciphertexts.

If we are using deterministic encryption with encrypts for a model attribute, then any two rows in the database that would have the same plaintext value will also have the same stored encrypted value. If we using non-deterministic encryption, then two rows that have the same plaintext value will generally have different encrypted values. As we'll see in a bit, this has implications as to whether we can query the encrypted data.

Setup

There's not a ton of configuration required to get going with encrypts, but there are some things to be aware of.

Keys

The main requisite is that we will need to generate a set of keys and add them to our credentials file(s). We can generate the keys to add by running: bin/rails db:encryption:init, which will output something like:

Add this entry to the credentials of the target environment:

active_record_encryption:
  primary_key: zxMXS0hBbpa5BzRKPv9HOSF9etBySiHQ
  deterministic_key: 0pM2UHHBQr1kf1irO6JgakcSOXu0r1Vn
  key_derivation_salt: I5AkViD0UJhSqK3NY49Zvsls3ZoifyXx

The primary key is used to derive the root encryption key for non-deterministic encryption. Note that the primary_key value in the credentials file can also be a list of keys.

The deterministic_key is used for deterministic encryption. If you recall from the section on determinism above, we'll get the same result if we encrypt the same data with this key multiple times. Currently, encrypts does not support using a list of keys for deterministic encryption. If we want to completely disable deterministic encryption, not providing the key is a sure-fire off switch.

The key_derivation_salt is used to derive encryption keys.

App Configuration

The encrypts API provides several configuration options. All of the options are defined in the config.active_record.encryption namespace and I'd encourage reading through them if you are going to use this feature. I believe you'll find that most of the options have pretty reasonable defaults.

I will mention the config.active_record.encryption.extend_queries option as it defaults to false, but enabling it has several implications:

enables querying unencrypted data in an encrypted column (also need to enable config.active_record.encryption.support_unencrypted_data for this)
allows supporting multiple encryption schemes
enables support for uniqueness validations

Database

When an encrypted string or text attribute is stored in the database, it isn't stored as an ordinary string or text - it is stored as a more complex data structure that is serialized when written and deserialized when read. This data structure allows some meta information to be stored along with the encrypted text, which gives the app some clues about how the text was encrypted.

This extra meta-info introduces some storage overhead - up to 250 bytes.

The guide recommends increasing the size of a 255 byte string field to 512 bytes if using encryption on a column. It also says that for a text field, the overhead is generally negligible.

Invocation

Finally, we can talk about actually using the thing!

In the most basic use case, to encrypt a single column, we simply add an encrypts declaration to our model for the attribute we want to encrypt. For example, if we had a Dog model with a toy_location field (dogs like to hide their toys, you know) that needs encryption, our model would look like:

  class Dog < ApplicationRecord
    encrypts :toy_location

Pretty simple, eh?

Writing

Writing an encrypted attribute is completely transparent. We just do what we would normally do in Rails:

> dog = Dog.create!(name: 'Bruno', toy_location: 'top secret')

If we were to look at the content sitting in the database directly, we would see something like:

> result = Dog.connection.execute('SELECT toy_location FROM dogs LIMIT 1').first
   (1.4ms)  SELECT toy_location FROM dogs LIMIT 1
=> {"toy_location"=>"{\"p\":\"oVgEJvRaX6DJvA==\",\"h\":{\"iv\":\"WYypcKysgBY05Tum\",\"at\":\"OaBswq+wyriuRQO8yCVD3w==\"}}"}

The value here is just serialized JSON, let's go ahead and parse it:

> JSON.parse(result['toy_location'])
=> {"p"=>"oVgEJvRaX6DJvA==", "h"=>{"iv"=>"WYypcKysgBY05Tum", "at"=>"OaBswq+wyriuRQO8yCVD3w=="}}

That gave us a Hash. Most of the keys in this Hash are defined in the ActiveRecord::Encryption::Properties::DEFAULT_PROPERTIES constant. p is the payload, aka the encrypted plaintext. h is a Hash of headers that contain information relating to the encryption operation. Here, iv is the initialization vector the plaintext was encrypted with - more about this in the next section on searching, and at is an auth_tag that will be used during the decryption process to verify that the encrypted text hasn't been altered. You may notice other headers from the DEFAULT_PROPERTIES Hash above depending on how your encryption is set up and being used.

Reading

When we load a model with an encrypted attribute, Rails will seamlessly decrypt the encrypted value. Let's find the Dog we created above by his name:

> Dog.find_by!(name: 'Bruno').toy_location
=> <Dog id: 1, name: "Bruno", toy_location: "top secret", created_at: "2021-05-28 22:41:23.142635000 +0000", updated_at: "2021-05-28 22:41:23.142635000 +0000">

As you can see, the encrypted value was automatically translated to a readable attribute on our model instance - pretty slick.

Searching

What if we wanted to find Bruno by his toy_location instead of his name? We can do that just like we would if the field were not encrypted:

> dog = Dog.find_by!(toy_location: 'top secret')
  Dog Load (2.1ms)  SELECT "dogs".* FROM "dogs" WHERE "dogs"."toy_location" = ? LIMIT ?  [["toy_location", "{\"p\":\"oVgEJvRaX6DJvA==\",\"h\":{\"iv\":\"WYypcKysgBY05Tum\",\"at\":\"OaBswq+wyriuRQO8yCVD3w==\"}}"], ["LIMIT", 1]]
=> #<Dog id: 1, name: "Bruno", toy_location: "top secret", created_at: "2021-05-28 22:41:23.142635000 +0000", updated_at: "2021-05-28 22:41:23.142635000 +0000">

Notice that our query string was automatically converted into the encrypted JSON string we saw when we looked in the database.

Initialization Vector/Determinism

When using deterministic encryption, all records with the same plaintext value will use the same initialization vector to encrypt. This is so ActiveRecord will generate the same ciphertext for the same input, which is a prerequisite for being able to search the encrypted data. Under the hood, Rails uses the plaintext to generate the initialization vector for deterministically encrypted data - otherwise the IV is randomly generated.

If two rows with the same plaintext were to use different initialization vectors to perform the encryption, the serialized JSON that ends up in the database would be completely different.

In order to be able to perform searches on the encrypted data, the stored values need to be exactly the same.

This means that all of the stored values in the serialized hash need to be identical for two rows that have the same text value, AND that Rails can re-compute the exact same hash on the fly to find rows that are matches for the search string.

Determinism at its finest.

Searching Plaintext

What if we did not have the luxury of starting with pristine data? For example, if our Dog table already existed and had a pre-existing toy_location column on it that was not encrypted?

Well, as we can see by the query generated above, if we had a Dog record with top secret (unencrypted) as its toy_location, that query aint gonna find it. Also, it seems pretty likely that if we try to load a Dog record with stored plaintext into memory, Rails is going to have problems when it attempts to decrypt the plaintext.

One option we have would be to convert our plaintext data to encrypted data, which seems ideal to me. We may have reasons to want to avoid doing a data migration like that, however.

In that case, encrypts will allow us to keep storing the plaintext values as plaintext, and will encrypt any new or updated data. To opt-in to supporting a mix of encrypted/unencrypted data, enable the config.active_record.encryption.support_unencrypted_data configuration option.

Enabling this behavior will prevent the errors we would get when it tries to decrypt the plaintext and will also allow us to perform searches when a column contains a mismatch of plaintext and encrypted data.

If we enable the setting and re-run our search query above, we'll see:

Dog Load (0.3ms)  SELECT "dogs".* FROM "dogs" WHERE "dogs"."toy_location" IN (?, ?) LIMIT ?  [["toy_location", "{\"p\":\"Bd+/TzEysF2CCQ==\",\"h\":{\"iv\":\"R2IUJJ+EmnDnZvQP\",\"at\":\"zqG5WAJql1zgctRCPpoBkQ==\"}}"], ["toy_location", "top secret"], ["LIMIT", 1]]

Now, it is looking for records that have the encrypted content or have the plaintext version of that content. Perfect!

Case Insensitive Searches

By default, searching is case sensitive. If we need to ignore case when searching for some reason, we have a few options:

Option 1:

We can query for all of the case variations we need to match - eg. Dog.where(toy_location: ['Top secret', 'top secret']).

Option 2:

We can specify downcase: true on our encrypts declaration. This will cause the text to be downcased before it is stored. ActiveRecord will automatically downcase our search text when performing queries. The downside here is that all case information is lost when it is downcased. Sorry to be a downer.

Option 3:

We can specify ignore_case: true on the encrypts declaration and add an original_column_name column to our database (eg. original_toy_location). With this in place, if we created a dog with an uppercase letter in the encrypted field:

Dog.create!(name: 'Max', toy_location: 'Top secret')

the toy_location column would be populated with the encrypted form of 'top secret' (the value downcased), and the original_toy_location column will have the encrypted form of 'Top secret' (the value we set).

Any searches would be done against the toy_location column, and the model's toy_location attribute would be populated from the original_toy_location column when it is loaded into memory.

One thing to note here - while the toy_location column is encrypted deterministically in this situation (so it can be searched), the original_toy_location column appears to be encrypted non-deterministically. This makes sense, since that column does not need to support searching. This can be confirmed by comparing the toy_location and original_toy_location values for two records with the same plaintext value. As you can see below, they have the same stored values (initialization vector, payload, etc) for the toy_location column (searchable and downcased), and different stored values for the original_toy_location column (not searchable, case preserved):

{ "toy_location"          => "{\"p\":\"Bd+/TzEysF2CCQ==\",\"h\":{\"iv\":\"R2IUJJ+EmnDnZvQP\",\"at\":\"zqG5WAJql1zgctRCPpoBkQ==\"}}",
  "original_toy_location" => "{\"p\":\"5syLqDK6GCbBDw==\",\"h\":{\"iv\":\"KBGp4FrI7oL4/a3p\",\"at\":\"JnH6hxLX35cAwroImk2XqQ==\"}}" },
  "toy_location"          => "{\"p\":\"Bd+/TzEysF2CCQ==\",\"h\":{\"iv\":\"R2IUJJ+EmnDnZvQP\",\"at\":\"zqG5WAJql1zgctRCPpoBkQ==\"}}",
  "original_toy_location" => "{\"p\":\"0246w4+SSqqlJw==\",\"h\":{\"iv\":\"1uEnjlCNot9sYNgR\",\"at\":\"UhkhK6YlOTxJg75juqIMGA==\"}}" }

Other Cool things

encrypts does even more than we have looked at so far. In the interest of not writing a book, I'm not going to go into much detail here, but would like to mention some of its other capabilities.

We have only looked at encrypting simple strings, but encrypts can encrypt rich text attributes too.

It also provides support for previous encryption schemes. This means that we can start out using non-deterministic encryption on a column and change to using deterministic encryption later on. I would definitly recommend reading the fine print before using this feature.

We can rotate our (non-deterministic) keys. This is pretty cool, just note that it is not currently supported for deterministic encryption.

Related to the rotating keys, we can configure encrypts to store a reference to the key used to encrypt in the encrypted data itself.

If using deterministic encryption, encrypts supports using unique constraints. If we need to ensure uniqueness in any of our encrypted columns, there are some things to be aware of. Be sure to read up on it first.

Encrypted columns are automatically filtered from Rails logs by default. encrypts provides a way to disable this functionality.

This might be a good place to mention that the implementation is modular, and allows quite a bit of customization. Many of encrypts options can be set on a per-attribute basis, or at more global levels.

Limitations

Even with all of its glory, encrypts does have limitations. We'll take a look at a few that stood out to me. Given the variety of applicable use cases and the breadth of the functionality, I am sure other folks will have their own list.

Fuzzy searching - The search functionality encrypts provides requires an exact match on search text. This means doing a LIKE query, for example, won't work. It also means that any queries done on encrypted columns will need to go through Rails and ActiveRecord vs being manually crafted in SQL.
Rich text search - While it is rad that encrypts can encrypt rich text, it can only do so non-deterministically right now. This means that we won't be able to search rich text.
Deterministic searching does not support multiple keys - Something good to be aware of going in - if using deterministic encryption/searching, we won't have the ability to use more than one key at a time. If we need to change keys, we'll likely need to do something fancy.
Rails console exposes data - This may seem obvious, but if a malicious person gets access to our Rails console, they can load encrypted data into objects and view the plaintext all day long. In Jorge's post, he mentioned that HEY is using a console extension that sits on top of the encryption feature that protects and audits console access. Unfortunately, that is a private gem (named console1984) and not available in Rails (at this time).
Deterministic encryption reduces security - I don't think this is a fault of the implementation per se, but if we use deterministic encryption, then any two rows that have the same value for an encrypted attribute will have the same value stored in the database. While we can't necessarily reverse engineer how it was encrypted, if we know what one of the row's plaintext value is, then we know what the other row's plaintext value is as well. Non-deterministic encryption doesn't have this same weakness.

Summary

TLDR: I am pretty excited and intrigued by this feature. I think it will be cool to see how people use it and how it evolves over time. My hunch is that some of the current limitations will go away (like not supporting multiple keys for deterministically encrypted attributes) as more people begin using it and digging through the code.

What's Cooking in Rails 7?

Jason Dinsmore — Wed, 21 Apr 2021 16:16:09 +0000

A new version of Rails always brings new and exciting features. This writeup takes a look at some of the things Rails 7 has in store for us.

To find these examples, I dug through the Changelogs and selected some of the features that have me most excited so far, primarily based on how I currently use Rails.

I encourage you to do the same - you may find something awesome that impacts a feature you use regulary or provides functionality you've been lying awake at night fantasizing about.

For each feature, I have included the PR that introduced it and the GitHub user who submitted the PR.

Note that several of these features have followup PRs from people making essential contributions by updating documentation, adding changelog entries, tweaking implementations, etc. If you are one of those folks, please know that while you're not given explicit credit here, we are very grateful for all you do! 🙏

Railties

The main Branch

PR 40254 by @prateekkish

When generating a new Rails project or plugin without the --skip-git flag, the default branch of the created repository will now be main instead of master. If you've got a different branch set in your git config's init.defaultBranch, that default will be used instead.

Rails began initializing the Git repository in 5.1 via PR 27632 by @dixpac.

benchmark Anywhere

PR 40734 by @semaperepelitsa

This PR adds the ActiveSupport::Benchmarkable.benchmark method to the Rails namespace, which allows you to easily benchmark a block of code anywhere without having to declare a logger or extend/include the ActiveSupport::Benchmarkable module.

To use the new functionality, you'd do something like this:

  Rails.benchmark('Print bar') do
    # code to benchmark
    100.times { puts 'bar' }
  end

After your code executes, the message you provided and the time it took the code to execute in milliseconds will be passed along to the Rails logger, ie: Print bar (0.6ms). The benchmark call will return the return value of the block, 100 in this case.

Support Stats For Stylesheets and ERB Views

PR 40597 by @joelhawksley

If you've run rails stats, you know that it provides various statistics about your app's codebase. This new functionality adds Views and Stylesheets to the output, providing stats for files in app/views and app/assets/stylesheets.

Note that stylesheets need to have a .css or .scss extension, and view files need to have the .erb extension in order to be processed.

ActionPack

Redirect Back or To

PR 40671 by @dhh

Adds redirect_back_or_to(fallback, **) as a nice shorthand for redirect_back(fallback_location:, **).

This avoids having to specify the fallback_location kwarg (keyword argument) to redirect_back. redirect_back_or_to will redirect the user back to whence they came if it can, and will redirect to the fallback location if it can't.

The old syntax still works and hasn't been explicitly deprecated, but now redirect_back calls redirect_back_or_to under the hood.

Unpermitted Parameter Context

PR 41809 by @bbuchalter

This PR provides controller and request context when the parameters object is initialized so that when unpermitted attributes are logged, the controller, action, request, and filtered parameters will be logged as well.

ActionView

Lazy Load Images

PR 38452 by @jonathanhefner

Currently, apps can take advantage of the HTML standard's lazy loading of images by manually specifying the loading attribute as lazy in calls to image_tag. This new configuration allows you to set the default to "lazy" app-wide, so that the attribute will only need to be specified for places you want an image to be eager loaded.

Tag Attributes From Hash

PR 40657 by @seanpdoyle

This cool addition provides a slick way to tranform a hash of attributes into attributes on an ERB tag.

For example:

<div <%= tag.attributes({ id: 'percent-loaded', role: 'progressbar', aria: { valuenow: '75' }}) %>>

Would render:

<div id="percent-loaded" role="progressbar" aria-valuenow="75">

Perhaps not the greatest example, but if the hash were generated programmatically and required logic, this could be pretty useful.

ActiveModel

Add Numericality In Range Validation

PR 41022 by @mpapis

When validating numericality of a model attribute, this new syntax lets you specify a range for the numericality validator versus having to sandwich between the greater_than_or_equal_to and less_than_or_equal_to options.

Example usage:

validates :latitude, numericality: { in: -90..90 }

ActiveRecord

Invert a where Clause

PR 40249 by @kddeisz

This addition provides a handy way to get at the inverse of a where clause by via invert_where.

For example:

good_students = Student.where(grade: 80..100)
# SELECT \"students\".* FROM \"students\" WHERE \"students\".\"grade\" BETWEEN 80 AND 100

bad_students = good_students.invert_where
# SELECT \"students\".* FROM \"students\" WHERE NOT (\"students\".\"grade\" BETWEEN 80 AND 100)

Exclude a Record From Results

PR 41439 by @GlenCrawford

Ever need a way to get all records matching some condition except a record you already have? excluding may be your jam.

Instead of:

other_users = User.where(rating: 80..).where.not(id: primary_user)

You can:

other_users = User.where(rating: 80..).excluding(primary_user)

Build or Create Association on Has One Through

PR 40007 by @perezperret

Enables the build_association and create_association functionality for has_one through: relations. Previously these were not available on through associations.

For example, if you had:

class Dog
  has_many :toys
  has_one :toy_box
end

class Toy
  belongs_to :dog
  has_one :toy_box, through: :dog
end

class ToyBox
  belongs_to :dog
end

You'll now be able to:

toy.build_toy_box
# <ToyBox:0x00007f572007e170 id: nil, dog_id: 3>

Or:

toy.create_toy_box
# <ToyBox:0x00005601f2ac09a0 id: 5, dog_id: 3>

Respect Column Type When Calculating Average

PR 40351 by @schmijos

In prior versions of Rails, calling ActiveRecord::Caculations#calculate with :average would result in a BigDecimal, even if a column was of a Float or Integer type.

For example:

Coordinate.last.longitude.class
# Float
Coordinate.calculate(:average, :longitude).class
# BigDecimal
Coordinate.calculate(:average, :longitude)
# 0.13002356e3

With the new behavior, it honors the type of the database column:

Coordinate.calculate(:average, :longitude).class
# Float
Coordinate.calculate(:average, :longitude)
# 130.02356

One of the motivators for this change was that JSON conversions of BigDecimal result in a string value, whereas Float results in a numeric value.

JSON.parse({ avg_longitude: 130.02356.to_d }.to_json)
# { "avg_longitude" => "130.02356" }
JSON.parse({ avg_longitude: 130.02356 }.to_json)
# { "avg_longitude" => 130.02356 }

Automatically Encrypt/Decrypt a Model Attribute

PR 41659 by @jorgemanrubia

Finally, we get to encrypts.

This is deserving of a post all its own. The TLDR; version is that this feature adds a mechanism to auto-encrypt/decrypt an ActiveRecord attribute by declaring encrypts attr_name in the model. When a record is written to the database, it will automatically be encrypted, and when it is loaded from the database, the attribute will automatically be decrypted.

encrypts also provides a workflow for changing the encryption scheme being used, without needing to re-encrypt all previously encrypted records under the new scheme.

Data can be encrypted in a non-deterministic manner (encrypting the same text twice will result in different ciphertexts), or can be encrypted deterministically - which allows querying the encrypted attribute.

Other niceties are the capability to encrypt Action Text and to filter encrypted attributes from the application log files.

It really does quite a lot and provides several ways to customize behavior, depending on your needs.

What Else?

I touched on the things I am most excited about so far, but there is other great work underway. ActionPack, ActionView, ActiveRecord, and ActiveSupport in particular, all have rapidly growing lists of changes.

Thanks again to all the contributors who are making this next release possible. It's shaping up nicely!

5 Ways to Splat in Ruby

Jason Dinsmore — Tue, 05 May 2020 15:14:28 +0000

Originally posted on Hint's blog.

In honor of it being 5/5 today (aka Cinco de Mayo), I thought we'd continue the pattern and take a quick look at 5 different ways to use Ruby's infamous splat (*) operator.

Let's hop right into it!

1. Interpolate Into an Array

Splat can be used to expand the contents of one array into another:

middle = %i(bar baz)
[:foo, *middle, :qux]

# [:foo, :bar, :baz, :qux]

2. Capture an Argument List

You can use splat to capture a list of arguments into an Array. Below, we're capturing the entire list of arguments being passed to foo into an Array named bar:

def foo(*bar)
  "#{bar.class.name}: #{bar}"
end

foo(:a, :b, :c)

# Array: [:a, :b, :c]

3. Capture First/Last or First/Rest Values

Splat can also be leveraged to split up an array into parts. Kinda like pattern matching:

arr = [1, 2, 3, 4, 5]
first, *, last = arr
puts first # 1
puts last # 5

first, *rest = arr
puts first # 1
puts rest # [2, 3, 4, 5]

4. Coerce a Value Into an Array

If you want to ensure the thing you are dealing with is an Array, you can splat it.

foo = 1
bar = *foo
puts bar # [1]

foo = [1, 2, 3]
bar = *foo
puts bar # [1, 2, 3]

5. Convert an Array Into a Hash

Lastly, splat can convert an Array into a Hash. Your Array needs to contain an even number of elements for this to work. If the Array were grouped into pairs, the first element in each pair will become a key in your Hash, and the second element of the pair will be the corresponding value.

arr = [:foo, 1, :bar, 2, :baz, 3]
Hash[*arr]
# { foo: 1, bar: 2, baz: 3 }

10 New Things in Active Record

Jason Dinsmore — Wed, 20 Nov 2019 20:14:05 +0000

Originally posted on Hint's blog.

In this post, we will take a look at 10 new additions to Active Record in Rails 6. For each addition, I'll include a link to the PR the feature was introduced in, a link to the author's GitHub profile, and a brief description of what the feature provides.

We've got a lot to cover, so let's get going!

1. `rails db:prepare`

PR 35678 by @robertomiranda

When this rake task is run, if the database exists, it runs any pending migrations. If a database does not exist, it runs the db:setup rake task.

This feature was designed to be idempotent, allowing it to be run over and over again until it completes successfully.

The rake db:prepare task functions like this:

If you're not familiar with it, the db:setup rake task:

Creates the database
Loads the schema.rb or structure.sql file (whichever your application is configured to use)
Runs the db:seed task, which executes the code in the db/seeds.rb file

2. `rails db:seed:replant`

PR 34779 by @bogdanvlviv

This new db:seed:replant rake task does two things:

Runs truncate on all tables managed by ActiveRecord in the Rails environment the rake task is being run under. (Note that truncate deletes all of the data in the table, but does not reset the table's auto-increment (ID) counter)
Runs the db:seed Rails rake task to populate seed data

3. Automatic Database Switching

PR 35073 by @eileencodes

Rails 6 provides a framework for auto-routing incoming requests to either the primary database connection, or a read replica.

By default, this new functionality allows your app to automatically route read requests (GET, HEAD) to a read-relica database if it has been at least 2 seconds since the last write request (any request that is not a GET or HEAD request) was made.

The logic that specifies when a read request should be routed to a replica is specified in a resolver class, ActiveRecord::Middleware::DatabaseSelector::Resolver by default, which you would override if you wanted custom behavior.

The middleware also provides a session class, ActiveRecord::Middleware::DatabaseSelector::Resolver::Session that is tasked with keeping track of when the last write request was made. Like the resolver, this class can also be overridden.

To enable the default behavior, you would add the following configuration options to one of your app's environment files - config/environments/production.rb for example:

config.active_record.database_selector = { delay: 2.seconds }
config.active_record.database_resolver = 
  ActiveRecord::Middleware::DatabaseSelector::Resolver
config.active_record.database_operations = 
  ActiveRecord::Middleware::DatabaseSelector::Resolver::Session

If you decide to override the default functionality, you can use these configuration options to specify the delay you'd like to use, the name of your custom resolver class, and the name of your custom session class, both of which should be descendants of the default classes.

4. Negative Scopes for Enums

PR 35381 by @dhh

While enum has traditionally provided scopes to find items by their enum value, it has not provided scopes to find items not matching a specific enum value.

For example, given a Post model for a blog, with an enum on the status field:

enum status %i(draft published archived)

The following scopes have been provided automatically for some time:

scope :draft, -> { where(status: 0) }
scope :published, -> { where(status: 1) }
scope :archived, -> { where(status: 2) }

Now, the following negative scopes are also available:

scope :not_draft, -> { where.not(status: 0) }
scope :not_published, -> { where.not(status: 1) }
scope :not_archived, -> { where.not(status: 2) }

This makes it easy, for example, to find unpublished posts:

Post.not_published

5. `#extract_associated`

PR 35784 by @dhh

The new #extract_associated method is literally just a shorthand for preload plus map/collect.

Here's the source code of the method:

def extract_associated(association)
  preload(association).collect(&association)
end

Be aware that preload does not allow you to specify conditions on the association being "preloaded". You'd want to use a different eager-loading mechanism for that, likely includes, eager_load or joins.

Usage of #extract_associated might look like this:

commented_posts = user.comments.extract_associated(:post)

6. `#annotate`

PR 35617 by @mattyoho

This is a nifty addition that could be used to add useful information to your application's log files. The #annotate method provides a mechanism to embed comments into the SQL generated by ActiveRecord queries. As an added benefit, the comments it generates could be completely dynamic.

Inserting annotate into your query chain like below:

User
  .annotate('there can be only one!')
  .find_by(highlander: true)

Would generate the following SQL:

SELECT "users".*
FROM "users"
WHERE "users"."highlander" = ? /* there can be only one! */
LIMIT ? [["highlander", 1], ["LIMIT", 1]]

7. `#touch_all`

PR 31513 by @fatkodima

Another ActiveRecord::Relation method, #touch_all touches all records in the current scope, updating their timestamps.

You can pass an array of columns to touch, and optionally provide a time value to use. touch_all defaults to the current time in whatever timezone the app's config has set for config.active_record.default_timezone (the setting defaults to UTC).

For example, to update the updated_at field of all comments associated with a given blog post, @post, you could:

@post.comments.touch_all

To update a given field on the comments, say :reviewed_at, you would provide the column name:

@post.comments.touch_all(:reviewed_at)

And, to specify a time value, you would:

@post.comments.touch_all(:reviewed_at, time: the_time)

8. `#destroy_by` and `#delete_by`

PR 35316 by @abhaynikam

The destroy_by and delete_by methods are intended to provide symmetry ("in spirit") with ActiveRecord's find_by and find_or_create_by methods.

I believe there is an important distinction you should be aware of. find_by returns one record, or nil, whereas destroy_by and delete_by will match on entire collections of records.

Using find_by like this:

User.find_by(admin: true)

Generates the following SQL:

SELECT  "users".*
FROM "users"
WHERE "users"."admin" = $1
LIMIT 1  [["admin", 1]]

Whereas, using delete_by with the same parameters:

User.delete_by(admin: true)

Results in the following SQL:

DELETE FROM "users"
WHERE "users"."admin" = ?  [["admin", 1]]

Definitely something to keep in mind when using these methods!

Also, it's worth noting that there are no bang ( !) versions of the delete_by/destroy_by methods.

9. Endless Ranges in `#where`

PR 34906 by @gregnavis

Ruby 2.6 introduced infinite ranges. This new feature lets you use them in Rails' #where clauses.

For example, when trying to find a Post with more than 10 comments (contrived example, I know).

Before you would have to use SQL like:

Post.where('num_comments > ?', 10)

Now you can use syntax that is more idiomatic Ruby:

User.where(num_comments: (10..))

10. Implicit Ordering

PR 34480 by @tekin

This feature makes implicit ordering configurable for a database table. Rails implicitly orders results by the table's primary key, which can give surprising results when the primary key is something that isn't an auto-incrementing integer (like a UUID).

Setting a table's implicit order column allows you to specify a default order, without using a default scope, meaning that you don't need to use reorder in your code to change the order downstream, you can use order (assuming you have not already specified explicit ordering earlier in the query chain).

For example, if you declare an implicit ordering on your Post table:

class Post < ActiveRecord::Base
  self.implicit_order_column = 'title'
end

Be aware, that if you declare implicit ordering on a column that does not ensure unique values, your results may not be what you expect.

Have you called your method object lately?

Jason Dinsmore — Mon, 10 Jun 2019 20:20:12 +0000

Originally posted on Hint's blog

Allow me to introduce you to another library that has proven itself useful when writing Ruby: the Procto gem.

Procto Overview

Procto is a gem that lets you turn a Ruby class into a method object. This helps clean up code that instantiates a class to perform a single task, calculation, etc.

To better understand what it's trying to facilitate, perhaps a review of method objects in Ruby would help.

Ruby Method Objects

A method object is essentially an object that has a bound context and provides a single method.

Consider this example:

  >> meth_obj = [:foo, :bar, :baz].method(:size)
  => Array#size()
  >> meth_obj.class
  => Method < Object
  >> meth_obj.call
  => 3

Above, meth_obj is a method object that is bound to the array containing [:foo, :bar, :baz]. In this case, when call is executed on the method object, the size message is passed to the context the method object is bound to ([:foo, :bar, :baz]) and the result is 3. Interestingly (though unrelated), Ruby allows you to unbind and rebind in cases like this.

For example:

  >> meth_obj = [:foo, :bar, :baz].method(:size)
  >> meth_obj.call
  => 3
  >> meth_obj = meth_obj.unbind.bind([:waldo, :fred])
  >> meth_obj.call
  => 2

Use

To use Procto, you'll need to ensure you've got the gem installed and that it has been loaded by your program or application. If the gem is available, but hasn't been loaded, you can just use require 'procto' at the top of your Ruby class.

Once the gem is loaded and available for you to use, you'll just need to: include Procto.call right after your class definition, ala:

require 'procto'

class Foo
  include Procto.call

  def initialize(bar)
    @bar = bar
  end

  def call
    # do stuff
  end
end

When invoking your class, you will supply call with the parameters you would normally pass to your class's initializer.

E.g.:

Instead of Foo.new(bar).call, you would do Foo.call(bar).

Blocks

Although Procto is very useful, it's not the answer for everything. One thing you might find yourself wanting to do is pass a block to your call method. Unfortunately, Procto does not support this.

Consider the following code:

class FooWithBlock
  include Procto.call

  def initialize(bar)
    @bar = bar
  end

  def call
    # do something

    yield if block_given?
  end

  protected

  attr_reader :bar
end

You might expect to be able to use the above code like this:

FooWithBlock.call('Hello') do
  puts "I'm in a block!"
end

Sadly, your block will never be executed.

If you need to do the above, you'll have to do so without using Procto, so you'd need to remove include Procto.call from your class, and invoke your class as follows:

FooWithBlock.new('Hello').call do
  puts "I'm in a block!"
end

Bonus

If you haven't used it before, I also highly recommend the Concord gem. When used in conjunction with Procto, it really creates a nice interface for invoking method objects.

Concord abstracts away having to define the initializer and attribute accessors for your class and its attributes. It also happens to play very nicely with Procto.

Using both, you can turn:

class Foo

  def initialize(bar, baz)
    @bar = bar
    @baz = baz
  end

  def call
    # do stuff
  end

  protected

  attr_reader :bar, :baz
end

For.new(bar, baz).call

Into:

class Foo
  include Concord.new(:bar, :baz)
  include Procto.call

  def call
    # do stuff
  end
end

Foo.call(bar, baz)

I should mention that Concord limits your initializer to three parameters. You can work around this by passing a more complex data structure as one (or more) of the parameters, and extracting the parameters from that object, or by using the [Anima gem (https://github.com/mbj/anima) instead, which does not cap the number of parameters you can pass, but takes an initialization hash (keyword arguments) instead of parameters.

Summary

Procto is a fantastic tool for cleaning up your class's interface. When used in conjunction with the Concord gem, you'll find yourself writing significantly less boilerplate code to get your classes up and running. Your code will also end up generating a smaller abstract syntax tree (AST), which is generally a very good thing.

Gems, Some of My Favorites - PaperTrail + PaperTrailScrapbook

Jason Dinsmore — Mon, 06 May 2019 22:16:12 +0000

Originally posted on Hint's blog.

This is the first in a series of blog posts taking an introductory look at various Ruby gems that we find useful and regularly use or have used in our projects over the years.

Today I am going to cover two related gems: PaperTrail, and PaperTrailScrapbook. These gems facilitate keeping an auditable trail of changes made to rows of a tracked table in your Rails application's database.

PaperTrail

Overview

The PaperTrail gem allows you to track changes made to your Rails models over time. Any time a tracked record in your database is changed, it creates a corresponding entry in a PaperTrail::Version table. It accomplishes this by tying into the ActiveRecord callback chain and storing a new version when the record is created, updated, or destroyed.

Setup

To start using PaperTrail, you'll need to follow the steps outlined in the README on the project's GitHub page, which essentially guide you through the process of adding the gem to your gemfile, bundling, and then generating and running a migration that creates the PaperTrail::Version table. This table will be leveraged to store an entry for each "version" of your model.

Use

Once you've got the gem set up, you can start using it. To start tracking versions for a model, you'll just need to add has_paper_trail to your model file (usually near the top, inside the class):

class MyModel < ApplicationRecord
  has_paper_trail
  ...
end

Now that your versions are being tracked, you will be able to go into your Rails console and view the various versions of your models using PaperTrail's API.

For a specific object, obj, you can access a collection of its versions using: obj.versions. This will show you a list of versions that each have this structure:

#<PaperTrail::Version:0x007fe111b03f70> {
                    :id => 314159,
             :item_type => "Invoice",
               :item_id => 555,
                 :event => "update",
             :whodunnit => "1234",
                :object => "---\nid: 5455\nactor_id: 2183\nsale_id: \namount: !ruby/object:BigDecimal 27:0.1276E3\nauthorization_id: \npaid_on: \ncreated_at: 2018-02-15 17:14:16.718224000 Z\nupdated_at: 2018-02-16 17:14:16.718224000 Z\nstatus: active\n",
            :created_at => Mon, 19 Feb 2019 11:59:49 PST -08:00,
        :object_changes => "---\nsale_id:\n- \n - 123\npaid_on:\n- \n- 2018-02-20 20:39:19.956256107 Z\nauthorization_id:\n- 412\nupdated_at:\n- 2018-02-19 20:39:19.956256107 Z\n- 2018-02-19 19:59:49.270910041 Z\n"
    }

The fields of a version break down like this:

Field	Description
`id`	The primary key for this `PaperTrail::Version`
`item_type`	Type of record being tracked
`item_id`	The primary key of the record being tracked
`event`	The type of change that occurred
`whodunnit`	The `id` of the person making the change (you can set the class of the `whodunnit` in your PaperTrail configuration, ie. `User`, `Actor`, etc.)
`object`	A serialized version of the object being updated in its pre-update state
`created_at`	When this version was created
`object_changes`	A serialized list of changes that were applied to the object as shown in the `object` field of the version

PaperTrail also provides the ability to revert an object to a previous state. To restore an object to a previous version, you can call reify (which means to make something abstract more concrete or real) on the version.

For example, using the version from our last example:

last_version = object.versions.last  # gets PaperTrail::Version shown above
puts last_version.reify

Invoice 314159 {
                  :id => 314159,
            :actor_id => 2183,
             :sale_id => nil,
              :amount => 27.0,
    :authorization_id => nil,
             :paid_on => nil,
          :created_at => Mon, 19 Feb 2018 11:59:49 PST -08:00,
          :updated_at => Mon, 19 Feb 2018 11:59:49 PST -08:00,
              :status => 'active'
}

reify will give you an instance of the object that corresponds to the version before the object_changes have been applied, which matches what is in the object field of the version.

Calling reify does not affect the object's current state in the database unless you call save on the reified object. Doing so will persist the reified version to the database, creating a new version for the object in the process.

Caveats

PaperTrail falters a bit when tracking changes to a model's associations. For example, consider an Invoice class that has_many :line_items where LineItem belongs_to :invoice. If you had an invoice that has several line_items associated with it, you could remove the line items by calling some code like:

invoice.line_items = []
invoice.save!

If you did so and took a look at the tracked versions of one of the line_items that had been associated with the invoice, you'd see that the removal of the invoice_id from the line_item was not captured in a version.

PaperTrail has some information in the project's README mentioning this and suggests some "experimental" workarounds, but your mileage may vary.

In my opinion, PaperTrail's API is a bit cumbersome and does not provide a concise summary of the changes that have been made to a model over time. The information is definitely there, but extracting it into a meaningful, complete history can be painful.

This is where PaperTrailScrapbook comes in.

PaperTrailScrapbook

Overview

PaperTrailScrapbook was conceptualized by Tim Chambers in 2017. Its intent is to provide a human-readable summary of changes made to a model tracked by PaperTrail. It accomplishes this by providing a simple interface to obtain either a complete history of an object, or a list of changes a specific person has made. Disclaimer: I am a contributor on this project :)

Setup

To set up PaperTrailScrapbook, you'll just need to add gem 'paper_trail_scrapbook' to your Gemfile and bundle your application. The other thing you'll need to do is specify the class of your app's whodunnit. Most often, this is going to be User. I'd recommend adding an initializer (e.g. config/initializers/paper_trail_scrapbook.rb) for PaperTrailScrapbook.

If your whodunnit class is User, your configuration would look similar to this:

PaperTrailScrapbook.configure do |config|
  config.whodunnit_class = User
end

Once you've got the configuration in place, restart your app server and you should be good to go.

Use

PaperTrailScrapbook currently has two modes of use, LifeHistory and UserJournal. Each mode provides a story for a given object or whodunnit.

LifeHistory

The LifeHistory module takes a model instance as its argument, and generates a list of changes made to that object over time.

For example, the following query for history:

widget = Widget.find(123)

puts PaperTrailScrapbook::LifeHistory.new(widget).story

Could output the following story:

On Friday, 08 Dec 2017 at 8:01 AM PST, Fred Flintstone <fred@flinstone.com> created the following Widget info:
 • name: Tricky Spinner
 • description:
 • notes: Recommended for ages 3 an up
 • created by: Fred Flintstone[123]
 • pattern: Widget Template[386]
 • price: 12.34
 • status: inactive
 •: 123

On Wednesday, 20 Dec 2017 at 12:53 PM PST, Barney Rubble <barney@rubble.com> updated the following Widget info:
 • name: Recommended for ages 3 an up -> Recommended for ages 3 and up
 • description: -> A fun spinning widget to keep your active fingers busy!
 • price: 12.34 -> 12.99

On Thursday, 21 Dec 2017 at 12:34 PM PST, Wilma Flintstone <wilma@flinstone.com> updated the following Widget info:
 • price: 12.99 -> 11.99

On Wednesday, 03 Jan 2018 at 10:24 AM PST, Betty Rubble <betty@rubble.com> updated the following Widget info:
 • name: Tricky Spinner -> Spinning Trick Widget
 • price: 11.99 -> 9.99
 • status: inactive -> active

On Wednesday, 05 Jan 2018 at 3:24 PM PST, Fred Flintstone <fred@flinstone.com> updated the following Widget info:
 • name: Spinning Trick Widget -> Spinning Brontosaurus Widget
 • price: 11.99 -> 9.99
 • status: inactive -> active

UserJournal

The UserJournal module takes a whodunnit instance as its primary argument and generates a summary of changes made by that person over time. It also provides options to scope the summary to a specific class and/or date range. This can be really useful as your app ages since the number of changes made by a person could grow quite large over time.

For example, searching by a specific whodunnit:

fred = Person.find(5)

puts PaperTrailScrapbook::UserJournal.new(fred).story

Could output the following history:

Between Friday, 08 Dec 2017 at 8:01 AM PST and Wednesday, 05 Jan 2018 at 3:24 PM PST, Fred Flintstone made the following changes:

On Friday, 08 Dec 2017 at 8:01 AM PST, created Widget[123]:
 • name: Tricky Spinner
 • description:
 • notes: Recommended for ages 3 an up
 • created by: Fred Flintstone[123]
 • pattern: Widget Template[386]
 • price: 12.34
 • status: inactive
 •: 123

On Wednesday, 12 Dec 2017 at 2:31 PM PST, updated Wobble[538]:
 • on_hand: 5 -> 12

On Thursday, 21 Dec 2017 at 12:34 PM PST, updated User[5]:
 • favorite saying: Yabba dabba -> Yabba dabba doo!

On Wednesday, 05 Jan 2018 at 3:24 PM PST, updated the following Widget[123]:
 • name: Spinning Trick Widget -> Spinning Brontosaurus Widget
 • price: 11.99 -> 9.99
 • status: inactive -> active

You can also scope changes to a specific class to hone in on the changes you're after. Consider the following where we look for changes Fred has made to instances of the Widget class:

fred = Person.find(5)

puts PaperTrailScrapbook::UserJournal.new(fred, klass: Widget).story

Which would output the following:

Between Friday, 08 Dec 2017 at 8:01 AM PST and Wednesday, 05 Jan 2018 at 3:24 PM PST, Fred Flintstone made the following changes:

On Friday, 08 Dec 2017 at 8:01 AM PST, created Widget[123]:
 • name: Tricky Spinner
 • description:
 • notes: Recommended for ages 3 an up
 • created by: Fred Flintstone[123]
 • pattern: Widget Template[386]
 • price: 12.34
 • status: inactive
 •: 123

On Wednesday, 05 Jan 2018 at 3:24 PM PST, updated the following Widget[123]:
 • name: Spinning Trick Widget -> Spinning Brontosaurus Widget
 • price: 11.99 -> 9.99
 • status: inactive -> active

History can also be constrained to a specific period of time, using the start and end parameters:

fred = Person.find(5)

puts PaperTrailScrapbook::UserJournal.new(fred, klass: Widget, 
                                                start: Date.parse('2018-01-01'), 
                                                end:   Time.zone.now).story

Which would ouput this:

Between Monday, 01 Jan 2018 at 12:00 AM +00:00 and Friday, 23 Feb 2018 at 11:05 AM PST, Fred Flintstone made the following changes:

On Wednesday, 05 Jan 2018 at 3:24 PM PST, updated the following Widget[123]:
 • name: Spinning Trick Widget -> Spinning Brontosaurus Widget
 • price: 11.99 -> 9.99
 • status: inactive -> active

Customization

If your system allows for instances of your whodunnit class to be deleted, you may run across a situation where the whodunnit for a version no longer exists in your database. You can customize the behavior of PaperTrailScrapbook for that scenario by providing a proc for invalid whodunnits in your configuration file, as follows:

 config.invalid_whodunnit = proc { |id| "** DELETED: #{id}**"}

Caveats

This project is still in its infancy, so you may hit occasional quirks or behavior that isn't ideal for your application. If you do, I'd consider submitting an issue or even a pull request. Any contributions are definitely welcomed and appreciated.

That said, there are a few known behaviors that I'd like to mention.

Querying for changes made by a person can take a very long time if your project has several large tables and the person has been active. If you can, scoping by class or time period will improve performance drastically.

Another area that could use improvement is in the presentation of serialized fields.

Imagine you have a Container class with a serialized Hash field, called capacity. If you were to populate the capacity field with the following hash:

{
  "weight" => {
    "min" => "25",
    "max" => "50"
  }
}

and ran history on your object instance, you would see something like this for the change on capacity:

• capacity: --- !ruby/hash:ActiveSupport::HashWithIndifferentAccess
weight: !ruby/hash:ActiveSupport::HashWithIndifferentAccess
min: '25'
max: '50'
added

and then if you updated the value to include height limits so the hash was:

{
  "weight" => {
    "min" => "20",
    "max" => "40"
  },
  "height" => {
    "min" => "55",
    "max" => "65"
  }
}

and re-ran history, you'd see this for the change:

• capacity: --- !ruby/hash-with-ivars:ActionController::Parameters
elements:
weight: !ruby/hash-with-ivars:ActionController::Parameters
elements:
min: '25'
max: '50'
ivars:
:@permitted: false
ivars:
:@permitted: false
-> --- !ruby/hash:ActiveSupport::HashWithIndifferentAccess
weight: !ruby/hash:ActiveSupport::HashWithIndifferentAccess
min: '20'
max: '40'
height: !ruby/hash:ActiveSupport::HashWithIndifferentAccess
min: '55'
max: '65'

All of the information is there, but as you can see, it can get a little unwieldy to wrap your head around as the hash grows in size. Serialized arrays face similar challenges, but the amount of data displayed for a change is nowhere near as verbose.

Summary

These two gems provide a great way to track changes made to your application's data.

PaperTrail is a well established gem and is used by many applications in their production environment.

PaperTrailScrapbook is hugely helpful for following changes made over time. It provides useful tools for observing how people are using your application and the sorts of changes they are making to data. I am excited to see how the project continues to evolve and grow as time goes on.

Next Time

In the next post in this series, we'll take a look at the handy-dandy Procto gem, which provides an excellent way to turn "your ruby object into a method object".

Forem: Jason Dinsmore

Active Record Encryption in Rails 7

Backstory

Deterministic Sidebar

Setup

Keys

App Configuration

Database

Invocation

Writing

Reading

Searching

Initialization Vector/Determinism

Searching Plaintext

Case Insensitive Searches

Other Cool things

Limitations

Summary

What's Cooking in Rails 7?

Railties

The main Branch

PR 40254 by @prateekkish

benchmark Anywhere

PR 40734 by @semaperepelitsa

Support Stats For Stylesheets and ERB Views

PR 40597 by @joelhawksley

ActionPack

Redirect Back or To

PR 40671 by @dhh

Unpermitted Parameter Context

PR 41809 by @bbuchalter

ActionView

Lazy Load Images

PR 38452 by @jonathanhefner

Tag Attributes From Hash

PR 40657 by @seanpdoyle

ActiveModel

Add Numericality In Range Validation

PR 41022 by @mpapis

ActiveRecord

Invert a where Clause

PR 40249 by @kddeisz

Exclude a Record From Results

PR 41439 by @GlenCrawford

Build or Create Association on Has One Through

PR 40007 by @perezperret

Respect Column Type When Calculating Average

PR 40351 by @schmijos

Automatically Encrypt/Decrypt a Model Attribute

PR 41659 by @jorgemanrubia

What Else?

5 Ways to Splat in Ruby

1. Interpolate Into an Array

2. Capture an Argument List

3. Capture First/Last or First/Rest Values

4. Coerce a Value Into an Array

5. Convert an Array Into a Hash

10 New Things in Active Record

1. rails db:prepare

PR 35678 by @robertomiranda

2. rails db:seed:replant

PR 34779 by @bogdanvlviv

3. Automatic Database Switching

PR 35073 by @eileencodes

4. Negative Scopes for Enums

PR 35381 by @dhh

5. #extract_associated

PR 35784 by @dhh

6. #annotate

PR 35617 by @mattyoho

7. #touch_all

PR 31513 by @fatkodima

8. #destroy_by and #delete_by

PR 35316 by @abhaynikam

9. Endless Ranges in #where

PR 34906 by @gregnavis

10. Implicit Ordering

PR 34480 by @tekin

Have you called your method object lately?

Procto Overview

1. `rails db:prepare`

2. `rails db:seed:replant`

5. `#extract_associated`

6. `#annotate`

7. `#touch_all`

8. `#destroy_by` and `#delete_by`

9. Endless Ranges in `#where`