Forem: Jan Bajena

Manage PostgreSQL default column values with Rails without downtime

Jan Bajena — Thu, 15 May 2025 06:42:28 +0000

Recently, my team at Fresha needed to introduce a type column to an existing table to represent different payment method types using a single table.

The migration had to be executed carefully and with zero downtime due to the large volume of records (50 million+) and the high frequency of reads and writes on the table.

Of course, not everything went according to plan. In this post, I’ll share some lessons learned so that you don’t have to ask the same questions or make the same mistakes as I did. 😅

Migration Plan

The migration consisted of the following steps:

Add the type column to the table with a NOT NULL constraint and a default value of "card".
The default value acts as a kind of backfill here.
Ensure all places in the application that write to the table explicitly specify the type attribute.
Drop the default value from the column.
This step prevents a situation where a record is created with an incorrect type value if a developer forgets to explicitly specify it during an insert.

Takeaways

Adding a Static Default Is a Safe Operation in Postgres 11+

In older versions of Postgres (<11), adding a default to a column would rewrite all existing rows to physically store the default value. This would lock the table and place higher pressure on the database which is especially problematic with large tables.

In PostgreSQL 11+, adding a column with a default uses metadata-only changes—no table rewrite, no locks.

For existing rows, the default value appears "dynamically" via catalog-stored defaults, but it isn’t physically written until the rows are updated.

New inserts write the value explicitly, materializing it in the column.

Important: Keep in mind that this is only safe for static defaults. If you're adding a column with a dynamic default (e.g., DEFAULT public.gen_random_uuid() for a UUID column), a full table rewrite will still be triggered.

📘 Check the PostgreSQL docs for more details

Adding a Default with a `NOT NULL` Constraint is a Safe Operation

We know that adding a column with a default is safe—but what about adding a default value and a NOT NULL constraint? Will it work for existing rows where the value hasn’t been physically written?

The answer is yes, it will. PostgreSQL uses the default value when validating the NOT NULL constraint, even if it hasn’t been physically written to the row yet.

Dropping a Default Is a Safe Operation

Dropping a default is also a metadata-only operation—it doesn’t lock the table.

This removes the default for future inserts but doesn’t affect existing rows. They either already have the value materialized (from earlier inserts) or still reference the historical default (for unchanged rows).

PostgreSQL handles legacy defaults internally to satisfy NOT NULL, ensuring safety without downtime.

Rails’ `partial_inserts` Option Can Make Dropping a Default Unsafe

The partial_writes option (renamed to partial_inserts in newer Rails versions) was introduced as an optimization. It minimizes the size of insert queries by omitting column values that match the table’s default.

While the idea is sound, in our case, it led to unexpected downtime.

As you may know, Rails maintains a database schema cache. After we dropped the default value from the column, there was a short period during which the schema cache had not yet been refreshed in all pods. Rails still believed the default value existed.

Since we had partial_writes enabled, Rails omitted the type: "card" value from the query even though we explicitly set it in the code. This caused the NOT NULL constraint to reject the inserts.

The partial_writes option is not very well known, and its name isn’t particularly descriptive.

Older versions of Rails (<7) had partial_writes enabled by default.

Fortunately, Rails 7 disables this option by default.

So if you're on Rails 7 or newer, you're probably fine—but it’s still worth double-checking whether it’s enabled in your application before running a similar migration.

Summary

Managing a column with a default value in PostgreSQL 11+ is a safe, metadata-only operation, but it comes with caveats, especially when combined with NOT NULL constraints or dynamic defaults.

Rails' partial_inserts (formerly partial_writes) feature can silently break inserts after dropping a default.

Mastering Large Backfill Migrations in Rails and Sidekiq

Jan Bajena — Mon, 25 Mar 2024 07:57:49 +0000

Migrating large datasets within a Rails application can be a daunting task. We've recently learned that the hard way at Productboard when trying to backfill data from a 230GB PostgreSQL table into a new one. It took us a few weeks, 2 database incidents, and a lot of nerves to get the task done. Was it perfect? Definitely not, but the upside of this process is that we've learned a lot, and now we can share it with you, dear reader :)

In this post, I'll explore effective strategies and best safety practices for tackling big data migrations using Rails and Sidekiq. Hopefully, it'll help you avoid repeating the same mistakes we made during our adventure.

Why Use Rails at All?

Some could argue whether using Rails is necessary at all if the only thing we need to do is to extract the data from one PostgreSQL table into another. The thing is, this migration involved a lot of business logic, which is very well encapsulated in our application and would be very time-consuming and error-prone to duplicate into SQL, so we quickly abandoned this idea in favor of loading the records in the application, doing necessary transformations, and storing the data in the new format.

One additional argument for using Rails was the ability to use Sidekiq to parallelize the workload and process each of our DB tenants (called spaces in our case) in a separate job.

Rails Tips

ActiveRecord is a great pattern which abstracts a lot of DB complexity from the programmer. However, when manipulating large data sets without a deep understanding of what's hapenning under the hood, it's easy to shoot yourself in the foot.

Below, I present a few tips that can help you avoid some headscratches :)

Use DB Replica

In the first iteration we've run our migration on the primary database that's used for most important user flows - the ones involving writes. There's nothing worse than e.g. writing a long comment just to see that saving failed when pressing ENTER.

That's why in the next iteration instead of the primary database, we utilise a read replica for fetching the data.

ActiveRecord provides an easy way of querying data from read replicas. Follow the Rails guide to set up the connection to the secondary database and then just wrap your selects in connected_to method as in the example below:

scope = ActiveRecord::Base.connected_to(role:  :reading, prevent_writes:  true) { SourceModel.all }

# Process the record set as always...

Use `find_each` / `in_batches` for fetching the data

This tip is quite basic for seasoned Rails developers, but still worth keeping in mind: when processing large amounts of records always make sure to query your tables in batches. You really don't want to run a single DB query for each of your 10 million records in the table ;)

Rails provides two methods for this purpose:

in_batches - loads records from table in batches (by default 1000 records per batch) and yields each batch in a block
find_each - loads records from table in batches and yields each record separate in a block

SourceModel.all.in_batches do |records|
  # Process batch of records at once
end

SourceModel.all.find_each do |record|
  # Process a single record
end

Warning: One commonly overlooked thing about these batch methods is that YOU CAN'T SORT the result set. Rails will always use the primary key for iterating. I've been hit by this problem quite a few times already 🙈

Use Rails' `insert_all` methods to speed up writes

Similarly to the read methods listed above, Rails supports batch inserts and upserts (from version 6.0.0).

insert_all allows you to bulk insert records, significantly speeding up the process compared to individual INSERT statements. This method is ideal for quickly populating the new table with transformed data.

One great thing about this method is that in case if e.g. you're restarting the migration and you already have some records inserted in the table with unique index applied insert_all will automatically skip duplicate inserts. If you have multiple unique constraints on the same table, you can specify the desired one by providing unique_by option.

models_data = [
  { first_name: "Foo", last_name: "Bar" },
  { first_name: "Eloquent Ruby", last_name: "Russ" }
]

DestinationModel.insert_all(models_data)

Warning: There are two caveats to using this method:

Keep in mind that the array (models_data in the example above) you're using for insert_all needs to have at least 1 element. Otherwise the call will crash.
All the hashes in the array have to have the same set of attributes, so even if e.g. one of your columns has some default value defined, you'll still have to provide that attribute in the hash.

Prevent N+1 queries when loading associated records

Often it happens that in your migrations you need access to multiple related records. For example when migrating Comments you might also need their author information.

Naive implementation could look like this:

Comment.all.find_each do |comment|
  next if comment.author.inactive?

  # Other migration logic
end

Can you spot the problem already? Calling comment.author will fire a separate SQL query for each comment.

This can be solved by calling includes(:author) on your ActiveRecord query:

Comment.all.find_each do |comment|
  next if comment.author.inactive?

  # Other migration logic
end

Sometimes calling includes might not be possible, for example when comments and authors exist in different databases.

In such cases, you can simply fire a separate, explicit query to load authors and use index_by(&:id) to speed up accessing them by IDs in the memory:

Comment.all.in_batches do |comments_batch|
  authors = User.where(id: comments_batch.map(&:author_id).index_by(&:id)

  comments_batch.each do |comment|
    author = authors[comment.author_id]
    next if author.inactive?

    # Other migration logic
  end
end

Use `select` to Load Only Necessary Columns

Rails by default loads all the columns when querying the table. It can be unnecessarily heavy e.g. when the table has text fields with potentially long contents.

When you only need certain fields from the database, use select to load just those columns. This reduces memory usage and speeds up the data loading process.

SourceModel.all.select(:id, :name).find_each do |model|
  # migration logic
end

Sidekiq Tips

When processing large data sets in Ruby, Sidekiq can be extremely helpful for splitting the work into smaller, parallel chunks. Below you can find a few tips that can help you making your migrations bulletproof.

Use `push_bulk` to schedule effectively

Sometimes you'll need to push lots of jobs to Sidekiq in order to process your migration. Scheduling each job using perform_async method is ineffective as it causes a lot of unnecessary round-trips to Redis.

Fortunately Sidekiq has a way to push jobs in batches. You can do it by using push_bulk method as in the example below:

class MigrationJob
  include Sidekiq::Job

  def perform(record_id)
  end
end

class LoaderWorker
  include Sidekiq::Job

  SIZE = 1000

  def perform(idx)
    # assume we want to create a job for each of 200,000 database records
    # query for our set of 1000 records

    SourceModel.all.in_batches(of: 1000) do |batch|
      array_of_args = batch.map { |record| [record.id] }

      # push 1000 jobs in one network call to Redis, saves 999 round trips
      Sidekiq::Client.push_bulk('class' => MigrationJob, 'args' => array_of_args)
    end
  end
end

It's also worth mentioning that in version 6.3.0 Sidekiq introduced perform_bulk method.

The code below is equivalent to the one above. It looks much cleaner

class LoaderWorker
  include Sidekiq::Job

  def perform
    SourceModel.all.in_batches(of: 1000) do |batch|
      array_of_args = batch.map { |record| [record.id] }

      MigrationJob.perform_bulk(array_of_args)
    end
  end
end

Use Sidekiq Batches to monitor the progress

When processing a huge volume of migration Sidekiq jobs it might be difficult to track overall progress. This is where Sidekiq batches come in handy. By scheduling jobs within a batch you can later easily see the percentage of completion and view all the failed executions in Sidekiq dashboard.

You can also define a callback job to be performed once a batch is finished. Such callback job can for example send you a Slack notification with an information whether the batch succeeded or failed freeing you of constantly checking the Sidekiq

Sidekiq Batches allow you to group jobs and monitor their overall progress. This is especially useful for large migrations, as it provides visibility into the process and helps identify issues early.

In the snippet below you can find an example of a generic job that allows us to schedule a defined job class for each of the Productboard spaces:

module Sidekiq::Migrations
  class ScheduleForAllSpacesJob < ApplicationJob
    sidekiq_options(queue: :low)

    def perform(job_class, *arguments)
      batch = Sidekiq::Batch.new

      batch.description = "#{job_class}Batch"
      batch.on(:complete, Sidekiq::Migrations::ScheduleForAllSpacesJob::Callback, 'job_class' => job_class)
      batch.jobs do
        Space.all.select(:id).find_in_batches do |spaces|
          Sidekiq::Client.push_bulk('class' => job_class.constantize, 'args' => spaces.map { |space| [space.id] + arguments })
        end
      end
    end
  end
end

class Sidekiq::Migrations::ScheduleForAllSpacesJob::Callback
  def on_complete(status, options)
    Rails.logger.info "[#{self.class.name}] Batch completed: #{status.as_json}, #{options}"
  end
end

# And then use it like this:
Sidekiq::Migrations::ScheduleForAllSpacesJob.perform_async(
  History::Sidekiq::Migrations::BackfillHistoryEntriesForFeaturesJob
)

Leverage Sidekiq::Iteration Gem

For processing large datasets in batches, the sidekiq-iteration gem is invaluable. It allows you to iterate over a table and process the records in manageable chunks.

What's great is that the gem takes care of handling job interruptions caused e.g. by new version deploys or DB connectivity issues. When a situation like this takes place, sidekiq-iteration saves the iteration cursor position and reschedules the job.

What you need to do is to include SidekiqIteration::Iteration module, define the collection to iterate over (build_enumerator method) and define your logic for processing a batch of records (each_iteration method):

 class BackfillSourceRecordsJob
   include Sidekiq::Worker
   include SidekiqIteration::Iteration

   # After 5 minutes the job will be rescheduled with current cursor position
   self.max_job_runtime = 5.minutes

   def build_enumerator(space_id, cursor: nil)
     scope = SourceRecord.where(space_id:)

     active_record_batches_enumerator(scope, cursor:, batch_size: 1000)
   end

   def each_iteration(records, *)
     # Process a batch of records
   end
end

Then simply schedule the job as you'd schedule any sidekiq job:

BackfillSourceRecordsJob.perform_async(123)

Control the number of concurrently processed jobs

Splitting a big migration into smaller chunks is a great way to process the data in a reasonable time. Unfortunately, if not done carefully, it can slow your system down significantly.

When scheduling jobs using standard perform_async, Sidekiq will start picking up them up from the queue as fast as it can, filling all the worker threads possible.

In case your jobs execute heavy database queries or have long runtime it can slow the rest of your system down significantly.

That's why it's reasonable to limit the number of concurrent migration jobs. Below you can find a few strategies that can help you achieve that.

Option 1: Use `perform_in`

If you know that each of your jobs takes a similar amount of time, you can easily plan split the execution into time windows by scheduling jobs to be ran at a specific time, using perform_in.

PARALLEL_JOBS = 5
SPACE_IDS = [1, 2, 5245, ...]

perform_in = 0

SPACE_IDS.each_slice(PARALLEL_JOBS) do |space_ids|
  space_ids.each do |space_id| 
    History::Sidekiq::Migrations::BackfillHistoryEntriesForFeaturesJob.perform_in(
      perform_in.minutes, 
      space_id
    )
  end

  perform_in += 1
end

It's worth noting push_bulk also supports at option, so the snippet above could also look like this:

PARALLEL_JOBS = 5

perform_in = 0

SPACE_IDS.each_slice(PARALLEL_JOBS) do |space_ids|
   Sidekiq::Client.push_bulk(
     'class' => History::Sidekiq::Migrations::BackfillHistoryEntriesForFeaturesJob,
     'args' => space_ids.map { |space_id| [space_id] },
     'at' => [perform_in.minutes.from_now] * space_ids.length
   )
end

Option 2: Dedicated queue and instances

Another option to limit the concurrency could be to schedule all your migration jobs into a separate queue dedicated only to your task and spin up Sidekiq server instances (e.g. via a dedicated Kubernetes deployment) configured to process that queue only.

# config/sidekiq-migration.yaml file:
:queues:
  - "migration"

In initializers/sidekiq.rb:

Sidekiq.configure_server do |config|
  config.concurrency = ENV['SIDEKIQ_CONCURRENCY']

  # rest of your Sidekiq config
end

and then initialize your server providing the desire concurrency and queues config:

SIDEKIQ_CONCURRENCY=10 bundle exec sidekiq -v -C config/sidekiq-migration.yml

Option 3: Use `Sidekiq::Limiter`

In our case each of the migration jobs had a different execution time, so using perform_in wasn't an option. We also didn't want to deal with introducing new Kubernetes deployments, so in the end we've ended up with a solution that involved Sidekiq::Limiter.concurrent from sidekiq enterprise.

The idea is that each job's data processing section is wrapped in a Sidekiq::Limiter. At the same time the main, "scheduler" job is also guarded using the same limiter key, so it'd only schedule a new job if it sees that there's a free "spot" in the limiter.

We had to "hack" sidekiq-iteration gem slightly in order to make this idea work, but in the end it served its purpose really well.

Here's how the "scheduler" job that enqueues the jobs for each of the Productboard workspaces looks like:

 class ScheduleForAllSpacesThrottledJob < ApplicationJob
   include SidekiqIteration::Iteration

   # Sidekiq-iteration needs an object that responds to `backoff` when throttling
   ThrottleCondition = Struct.new(:backoff)

   MAX_WORKERS_COUNT = 5
   BACKOFF_SECONDS = 30

   self.max_job_runtime = 1.minute

   sidekiq_options(
     # These retries will be just caused by sidekiq iteration, so we can retry indefinitely
     max_retries: 9999999
   )

   attr_reader :job_class

   def build_enumerator(job_class, cursor: nil)
     @job_class = job_class.constantize

     active_record_records_enumerator(Space.all.select(:id), cursor:, batch_size: MAX_WORKERS_COUNT)
   end

   def each_iteration(space, *)
     # Schedule a new job only there's less than MAX_WORKERS_COUNT jobs being processed currently
     within_limit do
       job_class.perform_async(space_id)
     end
   rescue Sidekiq::Limiter::OverLimit
     @over_limit = true
   end

   # This overrides `sidekiq-iteration` method
   def find_throttle_condition
     return unless @over_limit

     ThrottleCondition.new(backoff_seconds.seconds)
   end

   def within_limit
     Sidekiq::Limiter.concurrent(
       job_class::CONCURRENCY_KEY, 
       job_class::MAX_WORKERS_COUNT, 
       wait_timeout: 0
     ).within_limit do
       yield
     end
   end
 end

And here's the definition of the job that processes data for a single space:

  class BackfillHistoryEntriesJob < ApplicationJob
    CONCURRENCY_KEY = 'backfill_history_entries'.freeze
    MAX_WORKERS_COUNT = 5

    include SidekiqIteration::Iteration

    self.max_job_runtime = 1.minute

    sidekiq_options(
      # These retries will be just caused by sidekiq iteration, so we can retry indefinitely
      max_retries: 999999
    )

    attr_reader :space_id

    # We've overridden this method from `sidekiq-iteration`:
    def iterate_with_enumerator(*args)
      within_limit do
        super(*args)
      end
    rescue Sidekiq::Limiter::OverLimit
      # Setting following two instance variables will tell sidekiq-iteration to reenqueue the job and try processing again in a few seconds.
      @job_iteration_retry_backoff = backoff_seconds.seconds
      @needs_reenqueue = true

      false
    end

    def build_enumerator(space_id, cursor: nil)
      @space_id = space_id

      active_record_batches_enumerator(
        Note.all.select(:id).where(space_id:), 
        cursor:
      )
    end

    def each_iteration(records, *)
      # Migration logic here
    end

    def within_limit
      Sidekiq::Limiter.concurrent(
        CONCURRENCY_KEY, 
        MAX_WORKERS_COUNT
      ).within_limit do
        yield
      end
    end
  end
end

Option 4: Use `Sidekiq::Limiter` and control it through feature flags

This solution from the previous section did the job and prevented us from overloading the database. However, while monitoring the DB CPU and IOPS we've noticed that the database is underutilised, so we made one tweak to that solution. Instead of storing MAX_WORKERS_COUNT and BACKOFF_SECONDS in constants, we stored those values in variations of a LaunchDarkly flag and read them in runtime. This allowed us to dynamically adjust the job parameters according to the current system load.

That's how the limiter definition looked like after the changes:

def within_limit
  Sidekiq::Limiter.concurrent(
    CONCURRENCY_KEY, 
    max_workers_count
  ).within_limit do
    yield
  end
end

def backoff_seconds
  job_config[:backoff_seconds] || 5
end

def max_workers_count
  job_config[:workers] || 5
end

def job_config
  # Our own implementation of LaunchDarkly client 
  PbLaunchdarklyClient.global.variation(
    'main.multiplayer.history.backfill-configuration', 
    default_value: {}
  ).with_indifferent_access
end

In the screenshot below you can see how happy I was when I increased the workers count from 5 to 10 and the migration started picking up the pace 🎉

Implement a Kill-Switch

It's a good practice to have a kill-switch mechanism in place to immediately halt the migration if something goes wrong.

We've implemented our kill-switch using a LaunchDarkly flag. This little condition saved us a few times already. Highly recommended!


def build_enumerator(job_class, cursor: nil)
  if PbLaunchdarklyClient::Global.flag?('main.multiplayer.history.backfill-kill-switch')
    Rails.logger.info("[#{self.class.name}][cursor:#{cursor}] Killswitch enabled. Skipping.")

    return [].to_enum
  end

  active_record_records_enumerator(Space.all.select(:id), cursor:)
end

In Conclusion

Successfully navigating large backfill migrations in Rails, as illustrated by our experience at Productboard, requires a blend of strategic planning, robust tool utilization, and a deep understanding of both Rails and Sidekiq. From grappling with a massive 230GB PostgreSQL table migration to overcoming database incidents and performance bottlenecks, our journey was full of challenges but ultimately led to valuable insights.

I hope this post was useful and will serve you as a pre-mortem to a never existing post mortem :)

Is programming a form of art? Learn what philosophy has to say about it 🤓

Jan Bajena — Sun, 24 Dec 2023 12:01:28 +0000

Have you ever wondered if programing can be considered a form of art? Do you feel as an artist sometimes?

I did wonder so I went deeply into the topic - asking my programmer friends about it and reading philosophical articles about what defines an art.

In the video below I gathered all my knowledge and tried answering the question as a true engineer - using boolean logic 😉

Do you agree with my point of view?

Employee stock options - trick or treasure?

Jan Bajena — Sun, 12 Nov 2023 11:32:25 +0000

Employee stock options can be a great addition to software developer's job offer, but before blindly accepting it, it's worth knowing what risks and problems they're associated with.

Here's a video I've recorded in which I'm explaining how the stock options work and what's the meaning of the wording associated with them.
Also, based on a specific example of a job offer, I'm divagating whether it's worth accepting it or not.

I know I just scratched the surface here. What's your opinion about the stock options? Would you accept a low-salary offer if the options package was high? Let's discuss!

Toxic Code Reviews

Jan Bajena — Fri, 27 Oct 2023 19:42:34 +0000

Here's a little fun video tutorial I made describing how (not) to become a toxic communicator in code reviews.

Have you experienced any of the situations from the video in your workplace? Or maybe you had a different situation that made you clinch your teeth? Let's discuss :)

P.S. This is a first video I've ever uploaded to YouTube. Tips and feedback welcome :)

Moving a Hanami application from Heroku to Railway

Jan Bajena — Thu, 08 Dec 2022 15:00:46 +0000

Recently Heroku dropped their free tier which was a go-to platform for many pet projects. My simple Hanami flashcard application (I wrote about it in my previous article) got deleted and I still wanted to continue learning Italian, so I started looking for an alternative to Heroku.

A colleague recommended Railway.app to me, so I decided to give it a shot.
Eventually I managed to migrate the application, but there were a few problems that I experienced, so I thought to share with you how I managed to solve them.

1. Creating and deploying the application

This part was super straightforward in my case. The only thing I needed to do is to pick "Deploy from GitHub repo" and that was that! Railway picked up the code and started a deploy automatically.

2. Adding a Postgres database to the project

The first deploy immediately failed. When I checked the deploy logs I've noticed that the application failed to connect to the Postgres database and well... that's understandable, because I didn't add the database to my project.

What I had to do was to click "+New" in my project and select a Postgres database.

Adding a Postgres to my project automatically injected necessary environment variable: DATABASE_URL.

3. Migrating environment variables

The next problem I faced was that the application wouldn't start due to missing environment variables.

So, what I did was that I went to my Heroku project settings and copy pasted the variables:

4. Solving assets precompilation

Unfortunately at this point my app still didn't want to start.
Once I checked my project's Sentry I saw Hanami::Assets::MissingManifestFileError:

It turned out that Railway's deployment would only spin up the server without precompiling the assets which was different from Heroku and gave me a lot of headache before I managed to fix that.

What worked for me in the end was to:

Set SERVE_STATIC_ASSETS enviromnent variable as true
Add a file called Procfile to the root directory of GitHub repository. The file has the following content web: bundle exec hanami assets precompile && bundle exec hanami server -p $PORT

Note the -p $PORT param. $PORT variable is provided by default by Railway and it's super important to set it when starting the server, because otherwise Railway won't know where your application lives.

5. Exporting and restoring PostgreSQL backup

My app's database contained over a thousand of language flashcards and I didn't want to loose them.
So, what I've done is I went to Heroku's PostgreSQL "Durability" tab and downloaded an export file of my database:

Next, I opened the Postgres addon in my Railway project and found the connection URL:

Having that URL, I launched pgAdmin on my computed, connected to the database, clicked "Restore" and picked the previously exported Heroku database file:

Summary

The migration wasn't as seamless as expected, but eventually everthing seems to be working fine. Note that my application is a Hanami 1.3. application, so things may have changed in recently released 2.0. version.

Happy hacking!

Eliminate hardcore nesting in your RSpec tests by using Ruby procs

Jan Bajena — Mon, 18 Jul 2022 10:14:27 +0000

Recently I was testing a service class in Ruby and I needed to make sure that specific combinations of conditions produce desired effects.

First, I started nesting the conditions in contexts like I'd do normally, so my tests would look like this:

context 'when A1' do
  let(:a1) {}

  it 'R1' do
    # ...
  end

  context 'and B1' do
    let(:b1) {}

    it 'R1' do
      # ...
    end
  end

  context 'and B2' do
    let(:b2) {}

    it 'R1' do
      # ...
    end
  end
end

context 'when A2' do
  let(:a2) {}

  it 'R2' do
    # ...
  end

  context 'and B1' do
    let(:b1) {}

    it 'R3' do
      # ...
    end
  end

  context 'and B2' do
    let(:b2) {}

    it 'R2' do
      # ...
    end
  end
end

In my case there was one more level of nesting, so the code was really unreadable and hard to understand. Here I removed one level and cleaned up the context contents to save your eyes dear reader :)

As you can see there was also a lot of repetition, so the first thing I did was extracting the repeating contexts into shared_context blocks and repeating test cases into shared_examples, so the code became similar to this:

shared_context 'A1' do
  let(:a1) {}
end

shared_context 'A2' do
  let(:a2) {}
end

shared_context 'B1' do
  let(:b1) {}
end

shared_context 'B2' do
  let(:b2) {}
end

shared_examples 'R1' do
  it 'R1' do
    # ...
  end
end

shared_examples 'R2' do
  it 'R2' do
    # ...
  end
end

shared_examples 'R3' do
  it 'R3' do
    # ...
  end
end

context 'A1' do
  include_context 'A1'

  include_examples 'R1'

  context 'B1' do
    include_context 'B1'

    include_examples 'R1'
  end

  context 'B2' do
    include_context 'B2'

    include_examples 'R1'
  end
end

context 'A2' do
  include_context 'A2'

  include_examples 'R2'

  context 'B1' do
    include_context 'B1'

    include_examples 'R3'
  end

  context 'B2' do
    include_context 'B2'

    include_examples 'R2'
  end
end

Ok, at least now, whenever the details of a test or context case change I wouldn't have to update all the places.

However, the tests would still be difficult to read due to nesting, so I thought that it might be cool to define each scenario in a single line instead of having to browse through endless nested blocks.

Imagine something like in the snippet below. I translated the nested contexts and examples from previous code block, so that in each line the last element of the array defines the expected shared_examples group and all the previous elements are names of shared_contexts that need to be deeply nested one after another:

[
  ['A1', 'R1'],
  ['A1', 'B1', 'R1'],
  ['A1', 'B2', 'R1'],
  ['A2', 'R2'],
  ['A2', 'B1', 'R3'],
  ['A2', 'B2', 'R2'],
].each do |scenario|
  # ... do some Ruby + rspec magic here
end

Reads much better, right? Let's see how the full solution looks like:


[
  ['A1', 'R1'],
  ['A1', 'B1', 'R1'],
  ['A1', 'B2', 'R1'],
  ['A2', 'R2'],
  ['A2', 'B1', 'R3'],
  ['A2', 'B2', 'R2'],
].each do |scenario|
  # All but last element of the array
  contexts = scenario[0..-2]
  test_example = scenario.last

  # Let's prepare a deeply nested set of Procs with all the required nesting to achieve the same result. 
  # A tricky thing to understand here is that we need to start from the most nested block and go up,
  # until the most outer context.
  contexts.reverse.inject(proc { include_examples(test_example) }) do |inner, ctx|
    proc do
      context ctx do
        include_context(ctx, &inner)
      end
    end
  end.call
end

I understand that this solution might be very specific to the problem I had, but perhaps it might inspire you to refactor your tests and improve their readability.

If you know a simpler or better method (a Ruby gem? Built-in RSpec functionality?) of dealing with such problems please let me know in a comment or private message.

Think twice before using AGE in PotgreSQL

Jan Bajena — Mon, 09 May 2022 19:03:12 +0000

Recently my friend at Productboard noticed an interesting bug in one of our services. For some reason our code responsible for calculating how many days our customers' features spend in certain states (Idea, Discovery, Delivery, etc) in some cases would give us wrong results.

For example: even though you can tell by looking at the calendar that the days difference between 2021-02-28 00:00:00 and 2022-05-03 00:00:00 is 429 days, our code would return 428.25 days for that timespan.

After some research it turned out that the problem was caused by the fact that we were using PostgreSQL's AGE function for calculating days difference.

According to PostgreSQL's docs AGE function calculates “symbolic” result that uses years and months, rather than just days.
It wasn't super clear to me what "symbolic" means, so I started digging a bit and I realised that the purpose of AGE is not to calculate a precise time difference, but rather calculate age the way we, humans do it. Instead of subtracting UNIX timestamps like computers do, we subtract each component of the date and then adjust the negative values.

So, in the previous example (2021-02-28 - 2022-05-03):

Years difference is 1
Months difference is 3
Days difference is -25, so we subtract 1 month, check how many days are left in February (0) and then add the days from May (3). Eventually we end up with 3 days.

We end up with 1 year, 2 months and 3 days. Now, why does Postgres return 428.25 here?

It's because:

The number of days returned by AGE is 365.25 - it's an average number of days in a year when we take leap years into consideration.
Postgres uses 30 as number of days in each month

So, now it all makes sense - 1 year, 2 months and 3 days leave us with 365.25 + 2 * 30 + 3 = 428.25 days 🤓.

Fortunately the solution to our problem was extremely simple - we just had to replace the AGE function with the subtraction operator.
In order to show you the difference, here's a query that I ran in an online postgres query tool:



SELECT 
    EXTRACT(epoch FROM ('2022-05-03 00:00:00'::timestamp - '2021-02-28 00:00:00'::timestamp)) / (3600 * 24) as subtraction_days, 
    EXTRACT(epoch FROM AGE('2022-05-03 00:00:00'::timestamp, '2021-02-28 00:00:00'::timestamp)) / (3600 * 24) as age_days 
FROM "current_schema"()

The moral of this story is that time calculations are an extremely sensitive matter that can be approached by computers and humans differently. Fortunately PostgreSQL has all of the possible approaches covered. We, as developers, just have to understand our use case, read the documentation and think twice in order to pick an appropriate one.

Rendering a Liquid template with highlight tags in Jekyll

Jan Bajena — Thu, 29 Jul 2021 07:12:35 +0000

While developing a blog in Jekyll it may happen that you'll have a need for rendering a custom template. I had this situation when trying to build a code snippet block with Contentful that'd eventually be rendered in a Jekyll blog.

Jekyll's docs suggest that you should parse the template like this:

content = "<pre>{{ contentToDisplay }}</pre>"
template = Liquid::Template.parse(content)

Then you can render the template (passing optionally some variables):

template.render('contentToDisplay' => "x = 1")
"<pre>x = 1</pre>"

It works nicely as long as long as you don't want to use a Jekyll Liquid tag, like the highlight tag that I needed.

When you change the content to:

{% highlight ruby %}
{{ contentToDisplay }}
{% endhighlight %}

then rendering such template will raise a following exception:

undefined method `safe' for nil:NilClass

It took me some time to figure out what the problem was, but eventually it turned out that the Jekyll::Tags::HighlightBlock class expects that there's site in template's context.

So, I ended up having three new problems:

What should be the type of the site variable?
Where do I get the variable from?
How do I pass it to the highlight tag?

After trials & errors I managed to answer all of these questions:

The tag expects site to be a Jekyll::Site
You can get the site object by calling Jekyll.sites.first (I suppose that in 99.9% of cases there's just one site so you shouldn't worry too much)
In order to pass the site context when rendering a template the previous call needs to be converted like this:

@template.render!({ 'contentToDisplay' => "x = 1" }, { registers: { site:  Jekyll.sites.first } })

That's it! Now my template got rendered correctly and the syntax is nicely highlighted:

<figure class="highlight">
  <pre>
    <code class="language-ruby" data-lang="ruby">
      <span class="n">x</span> <span class="o">=</span> <span class="mi">1</span>
    </code>
  </pre>
</figure>"

Hope you find this post useful and that it'll save you some time :)

Ruby TIL: Bubble babble encoding algorithm

Jan Bajena — Fri, 28 May 2021 18:12:10 +0000

Today I'd like to share an interesting finding from Ruby's standard library. The method is called bubblebabble and lives in the Digest module.

The main point of this method is to create digests that look and sound similarly to human words, so that they can be recognized and visually compared more easily than hexadecimal digests.

You can either use the method directly on Digest module (it'll run the encoding algorithm directly on the provided string) or on a specific digest class, like Digest::SHA1 (it'll then run the algorithm on a SHA1 digest of the input string):

irb(main):001:0> require 'digest/bubblebabble'
=> true
irb(main):002:0> Digest.bubblebabble('a')
=> "ximex"
irb(main):003:0> Digest::SHA1.bubblebabble('a')
=> "xociz-lynaf-livip-huniz-samah-tolat-sivov-pipiv-petel-kynyr-mexix"

Some history

The bubble babble encoding was invented in 2000 by Atti Huima. You can find the original document with algorithm's description at http://web.mit.edu/kenta/www/one/bubblebabble/spec/jrtrjwzi/draft-huima-01.txt.

According to the author, the name combines the name of a video game classic Bubble Bobble and the fact that the generated strings can be pronounced but sound like babbling.

Is this used anywhere for real?

Yes, this funny encoding method is used by the SSH2 suite to display easy-to-remember key fingerprints. The key is converted into a textual form, digested using SHA1, and run through bubble babble to create the key fingerprint.

You can test it out yourself by running ssh-keygen with -B option (http://man.openbsd.org/ssh-keygen.1#B).

Ruby Hash trick for creating an in-memory cache

Jan Bajena — Tue, 11 May 2021 07:40:23 +0000

In this short post I'd like to show you a smart trick for memoizing results of computations in memory. Sometimes you need to store results of multiple operations in memory for reuse and typically it's done using a Hash instance.

E.g. let's say that we have a method called for checking whether a Star Wars character is from the dark side:

def dark_side?(character_name)
  StarWars::Character.find_by(name: character_name).dark_side?      
end

The method is heavy, because it needs to run a DB query in order to get a result for a given input, so if we know that there might be a need for calling it many times with the same character_name it might make sense to store the result for future use.

Here's one possibility of how the results can be memoized:

def dark_side?(character_name)
  @dark_side_cache ||= {}

  @dark_side_cache[character_name] = StarWars::Character.find_by(name: character_name).dark_side? unless @dark_side_cache.key?(character_name)      

  @dark_side_cache[character_name]
end

However, Ruby's Hash has a nice constructor variant that allows passing a block.

Check this solution out, isn't it a pure Ruby beauty? 💅

def dark_side?(character_name)
  @dark_side_cache ||= Hash.new do |hash, char_name|
    hash[char_name] = StarWars::Character.find_by(name: char_name).dark_side?
  end
  @dark_side_cache[character_name]
end

How to setup Dynamoid in Ruby projects to simplify DynamoDB interactions

Jan Bajena — Fri, 07 May 2021 14:27:54 +0000

Working with DynamoDB in Ruby can be painful compared to how easy it is to configure and use SQL databases via ActiveRecord. Fortunately there's a way to speed things up thanks to Dynamoid gem which introduces ActiveRecord-like abstractions for interacting with DynamoDB.

In this article I'd like to show you how you can fully setup your project to work with Dynamoid.

Local DynamoDB setup

Before we start with Dynamoid itself I'd recommend you to setup DynamoDB locally.
First, start dynamodb-local Docker image.

Next you can play around with your local DDB instance by using the NoSQL Workbench from Amazon:

Dynamoid installation and configuration

After adding gem "dynamoid" to your Gemfile you'll need to configure the gem. If you're using Rails I suggest doing it in an initializer (e.g. in config/initializers/dynamoid.rb file).

You can find the full list of options in Dynamoid's README but here's the config I'm using

require "dynamoid"

Dynamoid.configure do |config|
  # Local DDB endpoint:
  config.endpoint = "http://localhost:8000"

  # Fake AWS credentials for local development purposes:
  config.access_key = "abc"
  config.secret_key = "xyz"
  config.region = "localhost"

  # Do not add prefixes to table names. By default dynamoid uses `dynamoid_#{application_name}_#{environment}` prefix:
  config.namespace = nil

  # Tells Dynamoid to use exponential backoff for batch operations (BatchGetItem, BatchPutItem)
  config.backoff = { exponential: { base_backoff: 0.2.seconds, ceiling: 10 } }

  # Do not add timestamps (created_at, updated_at) fields by default
  config.timestamps = false

  # Store datetimes as ISO-8601 strings by default. Otherwise UNIX timestamps will be used.
  config.store_datetime_as_string = true
end

Defining models

Let's say we're building an application for storing lists of flashcards, similar to Flashcard Genius.
In such app there could be a model called FlashcardList with:

user_id partition key
list_id sort key
name attribute
created_at attribute

in order to reflect that in Dynamoid model you can define a following class:

class FlashcardList
  include Dynamoid::Document

  table(
    name: "flashcard-lists",
    # Defines getters/setters for `user_id` field as well
    key: :user_id
  )

  # Defines getters/setters for `list_id` field as well
  range :list_id, :string

  field :name, :string
  field :created_at, :datetime
end

That's it! Now you can read/update your data in a manner similar to ActiveRecord:

[6] pry(main) FlashcardList.create_table(sync: true)
[7] pry(main)> FlashcardList.create(name: "My list", user_id: 1, list_id: "123", created_at: Time.zone.now)
=> #<FlashcardList:0x00007fb2c1a4ff08
 @_touch_record=nil,
 @associations={},
 @attributes={:user_id=>"1", :list_id=>"123", :name=>"My list", :created_at=>Thu, 29 Apr 2021 14:33:15 +0000},
 @attributes_before_type_cast={:user_id=>1, :list_id=>"123", :name=>"My list", :created_at=>Thu, 29 Apr 2021 14:33:15.977495000 UTC +00:00},
 @changed_attributes={},
 @errors=#<ActiveModel::Errors:0x00007fb2c1a4f210 @base=#<FlashcardList:0x00007fb2c1a4ff08 ...>, @errors=[]>,
 @new_record=false,
 @previously_changed={"user_id"=>[nil, "1"], "list_id"=>[nil, "123"], "name"=>[nil, "My list"], "created_at"=>[nil, Thu, 29 Apr 2021 14:33:15 +0000]},
 @validation_context=nil>
[8] pry(main)> FlashcardList.create(name: "My list 2", user_id: 2, list_id: "234", created_at: Time.zone.now)
=> #<FlashcardList:0x00007fb2c1bb5dc0
 @_touch_record=nil,
 @associations={},
 @attributes={:user_id=>"2", :list_id=>"234", :name=>"My list 2", :created_at=>Thu, 29 Apr 2021 14:33:25 +0000},
 @attributes_before_type_cast={:user_id=>2, :list_id=>"234", :name=>"My list 2", :created_at=>Thu, 29 Apr 2021 14:33:25.832570000 UTC +00:00},
 @changed_attributes={},
 @errors=#<ActiveModel::Errors:0x00007fb2c1bb50c8 @base=#<FlashcardList:0x00007fb2c1bb5dc0 ...>, @errors=[]>,
 @new_record=false,
 @previously_changed={"user_id"=>[nil, "2"], "list_id"=>[nil, "234"], "name"=>[nil, "My list 2"], "created_at"=>[nil, Thu, 29 Apr 2021 14:33:25 +0000]},
 @validation_context=nil>
[9] pry(main)> FlashcardList.import([{ name: "My list 3", user_id: 3, list_id: "456", created_at: Time.zone.now}, { name: "My list 4", user_id: 3, list_id: "567", created_at: Time.zone.now }])
=> [#<FlashcardList:0x00007fb2c134e6f0
  @associations={},
  @attributes={:user_id=>"3", :list_id=>"456", :name=>"My list 3", :created_at=>Thu, 29 Apr 2021 14:34:09 +0000},
  @attributes_before_type_cast={:user_id=>3, :list_id=>"456", :name=>"My list 3", :created_at=>Thu, 29 Apr 2021 14:34:09.230243000 UTC +00:00},
  @changed_attributes={"user_id"=>nil, "list_id"=>nil, "name"=>nil, "created_at"=>nil},
  @new_record=false>,
 #<FlashcardList:0x00007fb2c134da70
  @associations={},
  @attributes={:user_id=>"3", :list_id=>"567", :name=>"My list 4", :created_at=>Thu, 29 Apr 2021 14:34:09 +0000},
  @attributes_before_type_cast={:user_id=>3, :list_id=>"567", :name=>"My list 4", :created_at=>Thu, 29 Apr 2021 14:34:09.230263000 UTC +00:00},
  @changed_attributes={"user_id"=>nil, "list_id"=>nil, "name"=>nil, "created_at"=>nil},
  @new_record=false>]
[10] pry(main)> FlashcardList.all
=> #<Enumerator::Lazy: ...>
[11] pry(main)> FlashcardList.all.to_a
=> [#<FlashcardList:0x00007fb2a58ff458 @associations={}, @attributes={:user_id=>"1", :list_id=>"123", :name=>"My list", :created_at=>Thu, 29 Apr 2021 14:33:15 +0000}, @attributes_before_type_cast={:user_id=>"1", :list_id=>"123", :name=>"My list", :created_at=>Thu, 29 Apr 2021 14:33:15 +0000}, @changed_attributes={}, @new_record=false, @previously_changed={}>,
 #<FlashcardList:0x00007fb2a50c53c8
  @associations={},
  @attributes={:user_id=>"3", :list_id=>"456", :name=>"My list 3", :created_at=>Thu, 29 Apr 2021 14:34:09 +0000},
  @attributes_before_type_cast={:user_id=>"3", :list_id=>"456", :name=>"My list 3", :created_at=>Thu, 29 Apr 2021 14:34:09 +0000},
  @changed_attributes={},
  @new_record=false,
  @previously_changed={}>,
 #<FlashcardList:0x00007fb2a780f230
  @associations={},
  @attributes={:user_id=>"3", :list_id=>"567", :name=>"My list 4", :created_at=>Thu, 29 Apr 2021 14:34:09 +0000},
  @attributes_before_type_cast={:user_id=>"3", :list_id=>"567", :name=>"My list 4", :created_at=>Thu, 29 Apr 2021 14:34:09 +0000},
  @changed_attributes={},
  @new_record=false,
  @previously_changed={}>,
 #<FlashcardList:0x00007fb2a50bc8e0
  @associations={},
  @attributes={:user_id=>"2", :list_id=>"234", :name=>"My list 2", :created_at=>Thu, 29 Apr 2021 14:33:25 +0000},
  @attributes_before_type_cast={:user_id=>"2", :list_id=>"234", :name=>"My list 2", :created_at=>Thu, 29 Apr 2021 14:33:25 +0000},
  @changed_attributes={},
  @new_record=false,
  @previously_changed={}>]
  [12] pry(main)> FlashcardList.delete_table
=> "flashcard-lists"

Here's a corresponding log of DDB queries produced by the commands above. As you can see Dynamoid freed us from writing these huge ass queries which tidies up the code a lot.

D, [2021-04-29T16:32:23.509266 #43231] DEBUG: [Aws::DynamoDB::Client 200 0.065601 0 retries] create_table(table_name:"flashcard-lists",key_schema:[{attribute_name:"user_id",key_type:"HASH"},{attribute_name:"list_id",key_type:"RANGE"}],attribute_definitions:[{attribute_name:"user_id",attribute_type:"S"},{attribute_name:"list_id",attribute_type:"S"}],billing_mode:"PROVISIONED",provisioned_throughput:{read_capacity_units:100,write_capacity_units:20})

D, [2021-04-29T16:32:23.510232 #43231] DEBUG: (68.0 ms) CREATE TABLE
D, [2021-04-29T16:33:16.025411 #43231] DEBUG -- [Aws::DynamoDB::Client 200 0.046285 0 retries] put_item(table_name:"flashcard-lists",item:{"user_id"=>{s:"1"},"list_id"=>{s:"123"},"name"=>{s:"My list"},"created_at"=>{s:"2021-04-29T14:33:15+00:00"}},expected:{"user_id"=>{exists:false},"list_id"=>{exists:false}})

D, [2021-04-29T16:33:16.026200 #43231] DEBUG -- (47.79 ms) PUT ITEM - ["flashcard-lists", {:user_id=>"1", :list_id=>"123", :name=>"My list", :created_at=>"2021-04-29T14:33:15+00:00"}, {}]
D, [2021-04-29T16:33:25.881593 #43231] DEBUG -- [Aws::DynamoDB::Client 200 0.047884 0 retries] put_item(table_name:"flashcard-lists",item:{"user_id"=>{s:"2"},"list_id"=>{s:"234"},"name"=>{s:"My list 2"},"created_at"=>{s:"2021-04-29T14:33:25+00:00"}},expected:{"user_id"=>{exists:false},"list_id"=>{exists:false}})

D, [2021-04-29T16:33:25.882293 #43231] DEBUG -- (49.18 ms) PUT ITEM - ["flashcard-lists", {:user_id=>"2", :list_id=>"234", :name=>"My list 2", :created_at=>"2021-04-29T14:33:25+00:00"}, {}]
D, [2021-04-29T16:34:09.341679 #43231] DEBUG -- [Aws::DynamoDB::Client 200 0.109989 0 retries] batch_write_item(request_items:{"flashcard-lists"=>[{put_request:{item:{"user_id"=>{s:"3"},"list_id"=>{s:"456"},"name"=>{s:"My list 3"},"created_at"=>{s:"2021-04-29T14:34:09+00:00"}}}},{put_request:{item:{"user_id"=>{s:"3"},"list_id"=>{s:"567"},"name"=>{s:"My list 4"},"created_at"=>{s:"2021-04-29T14:34:09+00:00"}}}}]},return_consumed_capacity:"TOTAL",return_item_collection_metrics:"SIZE")

D, [2021-04-29T16:34:09.342284 #43231] DEBUG -- (111.47 ms) BATCH WRITE ITEM - ["flashcard-lists", [{:user_id=>"3", :list_id=>"456", :name=>"My list 3", :created_at=>"2021-04-29T14:34:09+00:00"}, {:user_id=>"3", :list_id=>"567", :name=>"My list 4", :created_at=>"2021-04-29T14:34:09+00:00"}]]
D, [2021-04-29T16:35:00.258852 #43231] DEBUG -- (0.04 ms) SCAN - ["flashcard-lists", {}]
D, [2021-04-29T16:35:00.283511 #43231] DEBUG -- [Aws::DynamoDB::Client 200 0.023691 0 retries] describe_table(table_name:"flashcard-lists")

D, [2021-04-29T16:35:00.305328 #43231] DEBUG -- [Aws::DynamoDB::Client 200 0.020683 0 retries] scan(table_name:"flashcard-lists",scan_filter:{},attributes_to_get:nil)

D, [2021-04-29T16:36:50.309800 #43231] DEBUG -- [Aws::DynamoDB::Client 200 0.057094 0 retries] delete_table(table_name:"flashcard-lists")

D, [2021-04-29T16:36:50.310466 #43231] DEBUG -- (58.15 ms) DELETE TABLE

Development environment setup

First create a following file:

class Dev::DynamoidSetup
  # List all your Dynamoid models here:
  DYNAMOID_MODELS = [FlashcardList].freeze

  class << self
    def create_tables
      check_env!

      DYNAMOID_MODELS.each do |m|
        puts "Creating table: #{m.table_name}..."
        m.create_table(sync: true)
      end
    end

    def delete_tables
      check_env!

      DYNAMOID_MODELS.each do |m|
        m.delete_table
      end
    end

    def check_env!
      return if Rails.env.development? || Rails.env.test?

      raise "Do not run on production envs!"
    end
  end
end

Now you can use Dev::DynamoidSetup.create_tables to automatically setup all required tables in dev environment.

You could optionally wrap it in a rake task and call it like bundle exec rake setup_dynamodb:

desc "Sets up DDB tables for local development"
task setup_dynamodb: :environment do
  Dev::DynamoidSetup.create_tables
end

Test environment (RSpec) setup

Once you have your development environment ready it's time to configure the tests.

First in spec_helper.rb add a following hook:

config.before(:suite) do
  Dev::DynamoidSetup.delete_tables
end

This'll make sure there are no leftover tables from previous suite runs. In CI you could optionally skip this line as DynamoDB container should always be fresh there.

Then in order to manipulate the data in tables during the specs - I suggest having shared contexts for each table and including them only in the specs that really need access to a given table.

Check out e.g. this one:

RSpec.shared_context "uses_flashcard_lists_table" do
    around do |example|
      FlashcardList.create_table(sync: true)
      example.run
      FlashcardList.delete_table
    end
end

RSpec.describe FlashcardList do
  include_context "uses_flashcard_lists_table"

  it "stores data in the DDB table" do
    expect { FlashcardList.create(name: "My list 3", user_id: 3, list_id: "456") }.to change(FlashcardList, :count).by(1)
  end
end

Summary

Hope you found this article useful. Also if you had already used Dynamoid in your projects before don't hesitate to comment about your approaches.

Forem: Jan Bajena

Manage PostgreSQL default column values with Rails without downtime

Migration Plan

Takeaways

Adding a Static Default Is a Safe Operation in Postgres 11+

Adding a Default with a NOT NULL Constraint is a Safe Operation

Dropping a Default Is a Safe Operation

Rails’ partial_inserts Option Can Make Dropping a Default Unsafe

Summary

Mastering Large Backfill Migrations in Rails and Sidekiq

Why Use Rails at All?

Rails Tips

Use DB Replica

Use find_each / in_batches for fetching the data

Use Rails' insert_all methods to speed up writes

Prevent N+1 queries when loading associated records

Use select to Load Only Necessary Columns

Sidekiq Tips

Use push_bulk to schedule effectively

Use Sidekiq Batches to monitor the progress

Leverage Sidekiq::Iteration Gem

Control the number of concurrently processed jobs

Option 1: Use perform_in

Option 2: Dedicated queue and instances

Option 3: Use Sidekiq::Limiter

Option 4: Use Sidekiq::Limiter and control it through feature flags

Implement a Kill-Switch

In Conclusion

Is programming a form of art? Learn what philosophy has to say about it 🤓

Employee stock options - trick or treasure?

Toxic Code Reviews

Moving a Hanami application from Heroku to Railway

1. Creating and deploying the application

2. Adding a Postgres database to the project

3. Migrating environment variables

4. Solving assets precompilation

5. Exporting and restoring PostgreSQL backup

Summary

Eliminate hardcore nesting in your RSpec tests by using Ruby procs

Think twice before using AGE in PotgreSQL

Rendering a Liquid template with highlight tags in Jekyll

Ruby TIL: Bubble babble encoding algorithm

Some history

Is this used anywhere for real?

Ruby Hash trick for creating an in-memory cache

How to setup Dynamoid in Ruby projects to simplify DynamoDB interactions

Local DynamoDB setup

Dynamoid installation and configuration

Defining models

Development environment setup

Test environment (RSpec) setup

Summary

Adding a Default with a `NOT NULL` Constraint is a Safe Operation

Rails’ `partial_inserts` Option Can Make Dropping a Default Unsafe

Use `find_each` / `in_batches` for fetching the data

Use Rails' `insert_all` methods to speed up writes

Use `select` to Load Only Necessary Columns

Use `push_bulk` to schedule effectively

Option 1: Use `perform_in`

Option 3: Use `Sidekiq::Limiter`

Option 4: Use `Sidekiq::Limiter` and control it through feature flags