Parallelism and custom variables in CircleCI

Mrinmoy Das — Wed, 02 Mar 2022 20:14:47 +0000

CircleCI allows us to specify a job’s parallelism level. In this post, we will try to further customize that using a custom CircleCI parameter.

Premise

We can use CircleCI’s parallelism feature to split a job in parallel by spreading them across multiple separate executors.

But sometimes we want to customize the parallelism count based on various other parameters.

One example of this would be, let’s say we have two kinds of workflows. A ci-workflow that runs every time. And a nightly workflow that runs at a certain time every night and builds the nightly release.

jobs:
  rspec:
    executor: rspec-executor
    parallelism: 30
    steps: ...

workflows:
  nightly:
    triggers:
      - schedule:
          cron: "00 10 * * *"
          filters:
            branches:
              only:
                - master

    jobs:
      - rspec:
          requires:
            - bundle-install

  ci-workflow:
    jobs:
      - rspec:
          requires:
            - bundle-install

We are using parallelism of 30 for our RSpec tests. For both the workflows. Now nightly runs happen at a low traffic time when most of the devs are asleep. We can reduce the number of parallelism for just the nightly workflow.

Solution

CircleCI does not provide an out-of-the-box solution for the above problem. We will need to rely on a custom parameter to solve this problem.

jobs:
  rspec:
    executor: rspec-executor
    parameters:
      machines:
        type: integer
        default: 30
    parallelism: << parameters.machines >>
    steps: ...

workflows:
  nightly:
    triggers:
      - schedule:
          cron: "00 10 * * *"
          filters:
            branches:
              only:
                - master

    jobs:
      - rspec:
          requires:
            - bundle-install

  ci-workflow:
    jobs:
      - rspec:
        machines: 10
          requires:
            - bundle-install

Using a pipeline parameter like above we can customize the parallelism value in CircleCI.

Until next time! ❤️

Sql injection and prepared statements

Mrinmoy Das — Sat, 06 Nov 2021 21:30:47 +0000

We will explore how to be safe against sql 💉 attacks. Using bound parameters and prepared statements.

Most of the time when interacting with the database from rails applications, we use ActiveRecord and most of the time we can just use the ORM to generate the sql query.

But sometimes we might need to venture out of the usual path and write our own queries. Sometimes we need to write complex reporting queries.

For example, let us say we need to find the distance between two coordinates or words. In these cases, there is no ready-made help available from active-record. We will need to write custom sql queries.

Let us move on to a concrete example.

Seed Database

To setup our test environment, we will insert an entire dictionary to the database as seed data.

We will start with setting up a Postgres instance and also check the IP address which will be helpful later.

Next, we can fire the psql shell and create words tables.

We will start a ruby container and install a dictionary.

Time to write some code to seed the database with the dictionary.

We have successfully seed the database with 102774 records.

SQL 💉

Now that we have the database setup, let us define the task at hand.

We would like to grab the 5 closest matches for an user input word.

We can use trigram search for this. And Postgres has pg_trgm module which provides us with the necessary tools. The <-> operator Returns the ‘distance’ between the arguments.

And to run the query we can use ActiveRecord::Base.connection.execute

But that is prone to sql injection.

There are roughly two well-known safe-guards present to solve this.

Quoting
Bind Parameters

Quoting

We can quote the column value to help prevent sql injection.

ActiveRecord provides us with ActiveRecord::Base.connection.quote.

Important to remember

Connection quoting in Rails uses a db connection from the existing pool, a repeated call and a heavy load might affect performance.

Bind Parameters And Prepared Statements

In most DBMS we can use a prepared statement, which pre-compiles the SQL query, separating it from data.

It is a server-side object that can be used to optimize performance, reducing/eliminating SQL injection attacks.

Postgresql support PREPARED statement, you can read more about it here.

Let us fire up the psql shell and use PREPARED statements.

We can check the pg log to see how it works.

Back to rails. How do we achieve this using ActiveRecord?

Well ActiveRecord has find_by_sql method, which helps us with this.

Let us check the pg log. To verify if it is working as intended.

Important to note, prepared statement’s lifecycle is per-connection basis.

Prepared statements only last for the duration of the current database session. When the session ends, the prepared statement is forgotten.

We can verify this by disconnecting and checking the log.

Alright, that is all I had to share on sql injections.

Until next week! ❤️

Forem: Mrinmoy Das