Forem: Hernan Velasquez

Be careful when using assign_attributes with has_one relations in Rails 7

Hernan Velasquez — Mon, 30 Oct 2023 07:20:36 +0000

Recently I was tasked to solve a bug on a feature that allows a user to mass import the relationships of an existing record of a model via a yml file.

Lets say you have 2 models, course and professor as:

class Course < ApplicationRecord
  has_one :professor

  accepts_nested_attributes_for :professor
end

class Professor < ApplicationRecord
  belongs_to :course
end

And the user can update existing records with a yml like:

---
name: Math 101
:professor_attributes:
  name: John Doe

The importer just parse this yml into a object like:

params = {
  name: 'Math 101',
  professor_attributes: {
    name: 'John Doe'
    }
}

For the sake of the example, lets say we want to update course with id 1, so the operation the importer is doing is:

course = Course.find(1)
course.assign_attributes(params)
course.save

It seems pretty straightforward right? Well, unfortunately not in all cases, so let's break this out:

A professor's record for existing course doesn't exist:

This case works. If you run this on a rails console you'll get:

course.assign_attributes(params)
course.save

  TRANSACTION (0.2ms)  BEGIN
  Professor Create (3.0ms)  INSERT INTO "professors" ("name", "course_id", "created_at", "updated_at") VALUES ($1, $2, $3, $4) RETURNING "id"  [["name", "John Doe"], ["course_id", 1], ["created_at", "2023-10-30 06:55:41.912606"], ["updated_at", "2023-10-30 06:55:41.912606"]]
  TRANSACTION (2.3ms)  COMMIT

A new record is inserted for the relation and we are all happy.

A professor's record for existing course exists, but I don't know it's id:

Here's where things start to get tricky. As you can wonder, the system user doesn't have to know the internals of a database, therefore they don't know the professors primary key. That's why the yml file doesn't have it.

As we have here a has_one relation, you can think "well, why do I need to know the professors primary key? it's only one record anyway!".

Well, lets see what rails do in this case:

course.assign_attributes(params)

  Professor Load (0.4ms)  SELECT "professors".* FROM "professors" WHERE "professors"."course_id" = $1 LIMIT $2  [["course_id", 1], ["LIMIT", 1]]
  TRANSACTION (0.1ms)  BEGIN
  Professor Update (0.7ms)  UPDATE "professors" SET "course_id" = $1, "updated_at" = $2 WHERE "professors"."id" = $3  [["course_id", nil], ["updated_at", "2023-10-30 07:01:44.880614"], ["id", 5]]
  TRANSACTION (2.6ms)  COMMIT

Weird right? with only calling assign_attributes rails is going to the database to execute an update setting professor's foreign key to null.

If then, you run save:

  TRANSACTION (0.2ms)  BEGIN
  Professor Create (0.9ms)  INSERT INTO "professors" ("name", "course_id", "created_at", "updated_at") VALUES ($1, $2, $3, $4) RETURNING "id"  [["name", "John Doe"], ["course_id", 1], ["created_at", "2023-10-30 07:05:05.858188"], ["updated_at", "2023-10-30 07:05:05.858188"]]
  TRANSACTION (1.8ms)  COMMIT

Wow, rails is now inserting a new brand record with the "updated" fields, so at the end you will have a duplicated record in the database:

select id,name,course_id from professors;

 id |   name   | course_id 
----+----------+-----------
  5 | Jhon Doe |            <- null
  6 | Jhon Doe |         1

So you'll end up 2 records, one of them with its foreign key in null, possibly raising an active record exception if you have strict constraints on the foreign keys.

This of course will not happen if you include all proper ids in the params hash you are passing to assign_attributes as:

params = {
  name: 'Math 101',
  professor_attributes: {
    id: 5 
    name: 'John Doe'
  }
}

Conclusion

Be very careful when using assign_attributes with accepts_nested_attributes_for's models, and don't trust on the fact that has_one relations should be easy to locate. If you can't pass proper primary/foreign keys to the hash object you are passing to assign_attributes it is better to manually load the relation and update it separately like:

params = {
  name: 'Math 101',
  professor_attributes: {
    name: 'John Doe'
    }
}
course = Course.find(1)
if course.professor
  params[:professor_attributes][:id] = course.id
end
course.assign_attributes(params)
course.save

Hope you find this useful.

The optimal way to create a set of records with FactoryBot.create_list & FactoryBot.build_list

Hernan Velasquez — Sat, 19 Jun 2021 09:23:17 +0000

The purpose

Recently I advised a fellow Rubyist trying to create 20 records in rspec to change something like:

# arbitrarily, assign even numbers to employee_number in this example
let(:first) { create (:employee, employee_number: 2) }
let(:second) { create (:employee, employee_number: 4) }
let(:third) { create (:employee, employee_number: 6) }
let(:fourth) { create (:employee, employee_number: 8) }
let(:fifth) { create (:employee, employee_number: 10) }
...
let(:tenth) { create (:employee, employee_number: 20) }

To something like:

let(:employees) { create_list(:employee, 20) }

But unfortunately this wasn't that simple, so I want to share this quick write up to expose the problems we faced and the final solution we reached.

The uniqueness validation

For the sake of this example, let's build our employee model as:

class Employee
   validates_uniqueness_of :employee_number
end

And the migration:

   add_column :employees, :employee_number, :integer, default: 0

So any attempt to call create_list will result in an exception Validation failed: employee_number should be unique since FactoryBot will try to create the 20 records with this field in 0. Even trying:

let(:employees) { create_list(:employee, 20, employee_number: 20) }

Will also fail since FactoryBot will try to create the 20 records with this field in 20.

First approach, the block

Reading FactoryBot documentation, we found that you can pass a block to create_list to manipulate the record to be created, so our next approach was doing:

let(:employees) { 
  create_list(:employee, 20) do |record, i|
    # arbitrarily, assign even numbers to employee_number in this example
    record.employee_number = i * 2
  end
}

First approach failed, why?

Unfortunately FactoryBot folks haven't documented the use of the block properly. When we read the docs the first time, we believed that create_list was built as something like

def create_list
  object = build the object
  yield(object) if block_given?
  object.save!
end

But the way they built this was something like:

def create_list
  object = build the object
  object.save!
  yield(object) if block_given?
end

So what we see here is the same effect of:

let(:employees) { create_list(:employee, 20) }

Trying to create 20 employees with employee_number set to its default value, 0, basically because he is saving the object before giving us the opportunity to manipulate the employee_number as we see fit.

Second approach, saving in the block.. will it work?

So yeah... next think we thought of was doing:

let(:employees) { 
  create_list(:employee, 20) do |record, i|
    # arbitrarily, assign even numbers to employee_number in this example
    record.employee_number = i * 2
    record.save!
  end
}

But, this will work? The answer ..... NOPE... but, why?

On the first iteration, it will create an employee with employee_number 0 * 2 = 0

On the second iteration, expect FactoryBot to create an employee with employee_number 1 * 2 = 2, but as he creates the record before yielding, then it first creates the record with its default (guess what, 0), so yeap, we'll get Validation failed: employee_number should be unique again.

Even if we don't have the uniqueness validation, it seems weird to create a persisted object (insert) to immediately update it (update).

Third and final approach, using build_list instead.

So, how is the best way to create a set of records using FactoryBot when you have weird validations to take care of?

let(:employees) { 
  build_list(:employee, 20) do |record, i|
    # arbitrarily, assign even numbers to employee_number in this example
    record.employee_number = i * 2
    record.save!
  end
}

build_list was intended to just build the object in memory without persisting them, so we can use it in combination to save! within a block to create the list we wanted without triggering the validations before saving them.

Its better to use save! instead of save so you will see any other validation error in case it happens to you.

Happy testing!

Understanding and tuning fuzzy queries in ElasticSearch by example

Hernan Velasquez — Wed, 19 Feb 2020 21:48:08 +0000

What's a fuzzy query?

Fuzzy queries are one of the most powerful features available in ElasticSearch (ES). It allows to search for similar terms across thousands of documents on an existing index. For example, with fuzzy queries a search term like "colombia" is found on entries like "columbia", "colombie" or "locombia" depending on how tuned we configure the request query.

Here is assumed that the reader has some basic knowledge on Elasticsearch, in particular how to send query requests to the server. There are a ton of great resources to catch up before going forward with this post.

To execute fuzzy queries, it's important to understand that ElasticSearch measures similarity using the Levenshtein Edit Distance algorithm, defined as the number of individual changes required to make one term identical to the another one. A change can be an insert, edit or delete operation (ES can also handle transpositions but we'll leave it for another post). It's not that complicated. Here are two examples:

The Levenshtein distance between colombia and columbia is 1 because you only need to make one change: change the 'u' for a 'o' in in the term columbia.

The Levenshtein distance between colombia and columbi is 2 because you only need to make two changes: change the 'u' for a 'o' in in the term columbia and insert an 'a' at the end of the term.

Executing a fuzzy query in ElasticSearch

Executing a fuzzy query is not that far from executing a typical search query in ElasticSearch. It's important to add the fuzziness parameter (even the fact that its optional) in the json request with the maximum Levenshtein distance you want to allow. Let's assume we have and index with this documents:

doc1: "i will marry you because I love you"
doc2: "i will live with harry"
doc2: "i'm sorry for your loss"

The query

GET /_search
{
  "query": {
    "fuzzy": {
      "name": {
        "value": "harry"
        "fuziness": 1
      }
    }
  }
}

Will return documents doc1 and doc2 because

Levenshtein('harry', 'marry') = 1 in doc1
Levenshtein('harry', 'harry') = 0 in doc2
Levenshtein('harry', 'sorry') = 2 in doc2

Of course, if we change fuziness from 1 to 2, all doc1, doc2 and doc3 documents will be returned.

Undestanding "fuziness": "auto"

Sometimes we need more flexibility on how long we want the maximum Levenshtein Distance to be defined. For short terms it makes no sense to have a distance greater that 2 because it can generate a lot of false positives.

In this sense, ElasticSearch allows the value "auto" to the fuziness parameter that will work in this way:

If the length of the term is 0, 1 or 2 the maximum Levenshtein distance allowed will be 0. In other words, only terms with exact matches will be returned.

If the length of the term is 3, 4 or 5 the maximum Levenshtein distance allowed will be 1.

If the length of the term is greater than 5 the maximum Levenshtein distance allowed will be 2.

To illustrate this, lets consider this example:

doc1: "don't be so hard on me"
doc2: "i will live with my friends"
doc2: "love this twitter trends"

The query

GET /_search
{
  "query": {
    "fuzzy": {
      "name": {
        "value": "me"
        "fuziness": "auto"
      }
    }
  }
}

Will return doc1 since it has the exact term "me". On doc2 although Levenshtein('me', 'my') = 1, the length of the term is 2 characters so it will be discarded.

Lets consider this other query over the same documents:

GET /_search
{
  "query": {
    "fuzzy": {
      "name": {
        "value": "friends"
        "fuziness": "auto"
      }
    }
  }
}

This will return:

doc2 because Levenshtein('friends', 'friends') = 0 (exact match) and length of the term is 6

doc3 because Levenshtein('friends', 'trends') = 2 (change 'f' -> 'r' and insert 'i' in term 'trends' and length of the term is 6

Tuning up "fuziness": "auto"

Although the default definition for auto fuziness in ES is a widely tested and accepted standard, sometimes you'll want to tweak a little bit the relation between Levenshtein Distance and the length of the terms.

For instance, I find sometimes inconvenient to make fuzzy queries on terms with length 3, so I'm inclined to look for exact matches on these terms. To achieve that, we can tweak the query request passing length ranges on the fuzziness entry. For example:

"fuziness": "auto:4,6"

Will work in this way:

If the length of the term is 0, 1, 2 or 3 a exact match is expected.

If the length of the term is 4 or 5 the maximum Levenshtein distance allowed will be 1.

If the length of the term is greater than 5 the maximum Levenshtein distance allowed will be 2.

Conclusion

ElasticSearch is a great tool that may be sometime used under its real capabilities. This small post just wanted to illustrate how fuzzy queries work using simple examples. I hope the reader find this useful.