Forem: 2n IT

Weekly programmers report 30/10/2023

2nITArticles — Mon, 30 Oct 2023 13:25:34 +0000

Hi! Happy Monday!

We would love to invite you to our weekly report, where we unravel the solutions and technologies that have been instrumental in our continued growth.

The content for this report was extracted from our internal standup meetings, and ChatGPT has provided clear and concise report text.
The automation of data aggregation is made possible through a custom Slack plugin integrated with make.com, which communicates with OpenAI on our behalf.

Mariusz:

What functionalities have been delivered?

The programmer has made several corrections in the code responsible for drawing the construction after the architecture change. These include usage of the render? method in the viewComponent being rendered after the condition is met, resigning from guard clauses in favor of case statements, and placing all strings in constants defined in a class serving as a constants' storehouse. He has also added a mechanism that calculates the price based on chosen construction and door dimensions. Furthermore, he resolved issues with select dimensions blockage and input blockage in the admin wizard. Lastly, Mariusz managed to finalize all his features, updated the first step in the salesperson's order position configuration, and wrote two scripts exporting data of sub-aspects and aspects of each group of doors.

What problems have been solved?

Issues resolved include achieving a smoother functionality with viewComponent's render? method, enhancing code clarity with case statements approach, and eliminating hard-coded values in the JavaScript. Some problems concerning the blocking options of Selects were resolved by placing the blocking function in its own controller. A mistaken "bang" in checking the current price type was identified and corrected, which previously kept all inputs disabled.

What technologies, tools or solutions have been used?

rspec, stimulus, stimulusReflex, csv

Mateusz:

What functionalities have been delivered?

Improved user listing by adding pagination and search function, and allowing users to be removed or edited on the list.
Explained and addressed issues related to external_id to the client.
Added search by first_name and last_name on the user list in the admin side.
Resolved duplication issues in industries.
Undertook some refactor of tests in users_spec.rb.
Fixed a 500 error on password reset.
Updated the link used to redirect users who get an email about their account migration from another platform.
Made changes to user import based on client feedback.
Modified weekly report generation for 2n things, based on Karolina's request.
Began moving report generation to the company account.
Uploaded a copy of the staging database to demo for client's usage.
Modified user import by assigning industries and company to user profile.
Added three email notifications for users, subject to text templates from client.
Started refactoring tests in users_spec in response to PM's approval.

What problems have been solved?

Resolved server overload issue caused by the user list feature.
Addressed a 500 error in password reset.
Fixed issues found during user importing.
Transferred the staging database to demo for client's presentations, allowing staging to be modified.
Handling problems with random system shutdowns on manjaro.

What technologies, tools or solutions have been used?

active_admin, bullet, rspec, csv, make.com, AI, make

Maciej:

What functionalities have been delivered?

Maciej has delivered the functionality of adding styling to a div to make it match the style on another webpage. Utilizing the .from-wysiwyg style, the application can now ensure each specified tag nested in an element with the .from-wysiwyg style will have this style applied. He also made progress with Shopify: he wrote to subscriptions and is now waiting for the necessary collaboration request from support. Additionally, he added alignment to the Trix component in the admin panel, allowing for better text format. Maciej has also managed to align the divs in the document's title.

What problems have been solved?

Maciej faced and resolved the issue of waiting for support from Shopify when making more unconventional style changes to the store. Other more ready-to-use libraries were unresponsive to the changes he made. He was also able to solve the problem with justifying text, which was in bullet points, by creating a JavaScript in the stimulus controller that disabled sub-points with each click. Lastly, he managed to stop sending mails on staging in the shopify-api app.

What technologies, tools or solutions have been used?

Maciej utilized a variety of technologies including Spina, CSS, Tailwind, Shopify, Trix, JavaScript, and HTML tags.

Karolina:

What functionalities have been delivered?

The infographic has been finished, an announcement for the November BRUG has been shared globally, and sponsor texts have been prepared. Some tests were done in several projects. In one of them, most of the checked functionalities have been approved, however, some issues still need to be closely examined. Other tasks included work on the AI report, which is already available on the blog and on 2n social media, and creating tickets for mobile view in the project platform.

What problems have been solved?

Minor login issues on the project platform have been encountered and subsequently resolved.

What technologies, tools or solutions have been used?

Clickup, Slack, Google Doc, Markdown

Krzysiek:

What functionalities have been delivered?

Version 1.0.9 of the app has been released, with updates from both Krzysiek and Maciek, including the extension of Trix. Mainly graphic improvements or minor corrections have been implemented. The major issue was with the fonts and also with the Forticlient, which significantly delayed the release.

What problems have been solved?

The issue with fonts not being read on Windows has been resolved. The issue was present when two stylesheets were incorporated, one loading the other, normally compiled into one when the page is rendered. However, imports would remain as they were, not merging into an import section at the top of the final file. This usually isn't a problem unless you're on Windows. Therefore, the import statement was moved to the top of the file which is loaded first for compilation, and this solved the issue. The discovery of this solution demanded a significant amount of time.

What technologies, tools or solutions have been used?

dokku, css, js, spina, tailwind, rails

Radek:

What functionalities have been delivered?

The creation of the validator was partially finished but not completed. No new functionality was added. Work continued on writing the validator for walking spaces. The only function delivered was a button for the admin panel.\

What problems have been solved?

The day involved resolving a series of small issues. There were some challenges with deploying changes to staging and understanding the flow of it. Errors detected by colleague Karolina and a few self-found ones were fixed in collaboration with Kuba. It was discovered how to add custom Tailwind classes with lower priority that can be overwritten later. The Avo application was checked and confirmed it could be used in Polish. The absence of a visible avatar on staging was investigated and it turned out to be a problem with Dokku not having the option set to keep storage during deploys. There were problems with the application crashing on staging right after admin login, and a local Avo license warning. The issues were eventually resolved. Another problem solved was fixing a tab in the admin panel and modifying includes to stop Bullet from throwing warnings.

What technologies, tools or solutions have been used?

rspec, rails, validator, tailwind, active storage, dokku, avo

Jakub:

What functionalities have been delivered?

Improved minor elements in the login components and landing page, including introduction of classes to redundant items like buttons and inputs
Adjusted login components according to prototype designs
Tested changes in logging and start pages, removing discovered bugs
Completed initial work on a hamburger type menu, pending modal incorporation
Finished adjustments to the tailwindconfig branch and merged it.

What problems have been solved?

No specific problems indicated.

What technologies, tools or solutions have been used?

Tailwind
Figma
Git
Stimulus
Turbo

Arkadiusz:

What functionalities have been delivered?

New columns according to the ticket information have been added in the forms when generating reports, dynamic reports, and in pivot tables. Two tickets for production issues related to incorrect assignment of sales date were resolved. The status update for newly created apartments from the price list is now not affected when the 'do not update prices and statuses' checkbox is selected, which was not considered before in newly created apartments.

What problems have been solved?

An incorrect use of the design pattern (causing the n+1 problem in several places) was identified and one function was adjusted to eliminate and slightly speed up this issue. Uncertainties regarding labels and dot templates/map templates have been communicated in the ticket and feedback from the client is awaited. With pivot tables, a refresher on using 'scenic' was needed. It was discovered that missing indices may be causing data retrieval problems in map reports and dynamic reports, some indices were added on staging and their effect is being evaluated.

What technologies, tools or solutions have been used?

ActiveRecord, scenic, Benchmark, ActiveRecord::Fixtures, AjaxDatatable, pg_dump, EXPLAIN ANALYZE, pg_indexes

Szymon:

What functionalities have been delivered?

The creation of a service object was unsuccessful. After facing difficulty, the plan to create it was postponed for the next day. A folder for the services and the first file was created, but it’s unsure if it's functioning as expected. However, the entire planned functionalities were delivered subsequently. Point collection for article subscription was added including the ability to subscribe to articles. An email notification feature was implemented that informs whenever someone subscribes or unsubscribes. Lastly, corrections were made to a previous project, some changes were introduced and an error was fixed.

What problems have been solved?

Adding and subtracting points, subscribing and unsubscribing from articles, and sending emails to users were the issues addressed.

What technologies, tools or solutions have been used?

service object, ruby, js, tailwind, devise, mailer

Marcin:

What functionalities have been delivered?

Work has been done on a ticket concerning the history of orders for each item, which was previously displayed for a past three-day span, and was increased to a seven-day span recently. While there is a slight confusion about the particular order information that needs to be displayed, the functionality has been working for 4 years without complaints until a ticket reported an issue. Before committing any corrections to the code, clarifying questions have been asked under the ticket to understand what exactly needs to be fixed. In another task, updates were done to the content of an email. To do this, access to SendGrid was acquired and despite the presence of several email templates, the required one was missing. While awaiting access to MailChimp, more digging into the code discovered that EmailTemp records are used to generate email content templates with the Liquid gem.

What problems have been solved?

No specific problems have been listed as solved yet.

What technologies, tools or solutions have been used?

rails, docker-compose, ruby, pry, liquid, sendgrid.

Weekly programmers report 23/10/2023

2nITArticles — Mon, 23 Oct 2023 14:57:16 +0000

Hey everyone!

It's that time of the week when we share our insights into the solutions and technologies we've been diligently working on.

Krzysiek:

What functionalities have been delivered?

Assisted with tasks including organizing emails and discussing client briefings.
Worked on setting up access and had an overview of how a production infrastructure update would be established. However, this was more complex than initially thought and was delegated solely to Marek.
Reviewed code in the project and discussed features with Mariusz, including past failures and potential improvements.
A ticket was moved to staging for final amendments to authorisation and template editing.
Adjustments were made to the project server, where the port mapping from HTTP and HTTPS to 5000 was corrected as Puma was running at 3000.
Demonstrated deployment process to team members and granted access rights to various projects.
Began writing secondary documentation.
Worked on several tickets mainly regarding bug fixes and refining yesterday's feature. Discovered unutilised user privilege switches.

What problems have been solved?

Assisted Mariusz in understanding certain aspects of various programming languages such as constructors, setters, and getters.
Setup for a new laptop was completed alongside giving access to several projects for various team members.

What technologies, tools or solutions have been used?

javascript, kubernetes, docker, ruby, spinacms, linux, turbo, hotwire, actionmailer, rails, notion

Karolina:

What functionalities have been delivered?

Karolina has created documentation that provides step-by-step instructions on how to add a document in the admin panel and how to enable maintenance mode on the website. She has also worked on an AI report which is now available as a blog post on the 2n website. In another project, she has created three additional test cases for the project, including ones about adding centers, creating trainings, and setting celebratory themes. Additionally, she posted about the October BRUG on Facebook.

What problems have been solved?

Karolina has prepared documentation aimed at resolving concerns and doubts of a client regarding the project. This will guide the client on where to click and what to do so that everything works as it should.

What technologies, tools or solutions have been used?

Slack, GoogleDocs, Markdown, Clickup

Mateusz:

What functionalities have been delivered?

Mateusz fixed a bug preventing access to a non-existent user, merged a pull request that removed the username from the entire application and handled a code review involving regular expressions. He added jersey_number and position to user teams and ensured successful import and data storage. In discount coupons, he moved a field from subscription serializer to team_serializer and revised the tests. He also spent time working on teams/join endpoint, added new fields and wrote many tests. Mateusz also faced a challenge with make.com where he had been sending article data in JSON format but managed to fix the issue by sending the data without JSON. Later on, he found a workaround for why the finished blog post wouldn't update, completed the integration of make with dev.to, made some changes to the user import feature, started work on emails, and began developing a standard format for standups. Next, he improved the look of the top menu, tested the production environment, developed a regular expression to eliminate special characters in external_id and prepared the homework for his evaluation with Mehdi and finally worked on deploying and testing the demo environment.

What problems have been solved?

He was able to resolve the problem of the JSON parsing in make. The issue with the faulty deployment of the production environment via GitHub was also solved by resetting the git refs.

What technologies, tools or solutions have been used?

Make.com, rspec, docker. stripe. csv, mysql, git, dokku, vps, ubuntu, regex, grape

Szymon:

What functionalities have been delivered?

Szymon delivered functionalities such as creating his own application, creating new models, views, controllers, and working with gemfiles and databases. He learned how to work on a database, including adding items to tables and relating them to users. He started authentication with Devise, managed to log in, create an account, and is now working on improving Devise and adding things to the table from the browser level. With help, he managed to set up Tailwind, realized he was using an outdated form and exchanged it for the current standard. He also developed an article creator, edited basic Devise files, and implemented login, password changing, and account creation. He changed the database from SQLite3 to PostgreSQL and implemented corresponding functionalities. He added new articles and a page for creating articles.

What problems have been solved?

Szymon resolved problems like creating his own application including new models, views, controllers, working with gemfile, routes.rb and databases. He solved a problem with entering things into the database from the terminal level and authentication with Devise. He managed to configure Tailwind CSS and refine the visual design of the entire program. He switched the database from SQLite3 to PostgreSQL, enabled article addition, user creation, and logout feature. He also implemented Tailwind into API however new problems appeared.

What technologies, tools or solutions have been used?

Devise, Ruby, SQLite3, Tailwind, PostgreSQL, JS

Jakub:

What functionalities have been delivered?

Improvements in login view and landing page were made. Learned to use git stash. Deployment procedures were also mastered and continued work on views. Configuration files were successfully implemented and fonts for desktop were adjusted. Improvements on repeated code in case of buttons are left for further work along with continuing view improvements.

What problems have been solved?

Small error fixes, understanding git specifics, resolving issues with Tailwind config- required removal of tailwind-css-rails gem and references to it in procfile. Conflict resolution skills in Git were improved and code readability was increased by wrapping long lines of code and ensuring adherence to DRY principles.

What technologies, tools or solutions have been used?

Tailwind, git, ruby, docker, CSS

Radek:

What functionalities have been delivered?

Radek improved the appearance of the app and added a centering functionality for maps when the user is adding a marker. He also improved the format of displayed addresses and their responsiveness when searching via autocomplete. Changes were made to how the model for the walk was structured. Furthermore, Radek created a validator for creating a walk and adjusted the format of incoming coordinates from the front end.

What problems have been solved?

Radek found and fixed an error in the admin panel and confirmed it didn't cause further issues. Initially, he addressed a problem of double markers being added, later realizing some were tripled; however, this was resolved the next day. The changes also ensured markers no longer multiply. Furthermore, Radek refined how everything would be retained in the database.

What technologies, tools or solutions have been used?

avo, tailwind, stimulus, leaflet, openrouteservice, activerecord, rspec, rails

Mariusz:

What functionalities have been delivered?

Mariusz successfully implemented changes to the door structure drawing architecture and introduced the Overall and Frame categories of the drawing batch. He also made considerable progress with the feature for calculating the price per construction and frame surface area. In addition, he added an option in the selectors for the door group dimensions to restrict the possible measurements based on the total height of the construction. He improved the calculations for the price per frame surface area and construction. Fixing a mistake that was made when calculating the height of the construction in the canvas drawing, he sent his work to the staging area for testing. Furthermore, Mariusz fixed a bug related to displaying attachments available for cloned relational objects of a product when the attachment has been deleted from another product (which is a clone). He also introduced a block on price inputs and radio buttons for selecting their type when the current step of the admin wizard deals with an aspect defined in the constants as aspect_without_price.

What problems have been solved?

Problems with the implementation of getters and setters were solved, as the class variables built in the constructor should not have the same names as the getters and setters. A significant issue was found with the attachment display for cloned relational objects of a product, which was fixed by introducing an additional operation in the cloning service. Troubles associated with selecting the door group's dimensions which affects the total height of the construction were also addressed.

What technologies, tools or solutions have been used?

stimulus, rspec, StimulusReflex, Shrine, Tailwind

Arkadiusz:

What functionalities have been delivered?

The programmer has delivered several functionalities. Specifically, he has made it possible to input values into the table structure of sold items, including the minimum, maximum, and average price, as well as size. It is also possible to recalculate these values based on the price list. He added feature reset checkbox values in the sales section when creating a new historical version. A functionality has also been introduced to block the possibility of adding new phases in finished investments. While the application work on learning microservices progresses, initial pseudocode has been created to guide the development process.

What problems have been solved?

Several issues were addressed. Integrating the checkbox functionality into the existing mechanism posed a challenge due to the NOT NULL constraint, thus a separate array was used. In-depth discussions were held on applying the Boolean on the checkbox and the select in the logic with the possibilities of NULL values. An issue with the change of sale dates in the new historical version of a unit was addressed. The problem was likely linked to the price list addition. The solution for the issue of adding new phases in finished investments resulted in being able to visually conclude investments. Finally, the proper connection of Tailwind to the application was managed with significant help in understanding dependencies, utilization of button_to and link_to, and log digging.

What technologies, tools or solutions have been used?

ActiveRecord, ActiveRecord::Migration, CoffeeScript, JS, haml, ActiveAdmin, Pundit, ssh, scp, Chipmunk, pry, devise, tailwindcss

Marcin:

What functionalities have been delivered?

The date of the offer's end in the restaurant has been quickly changed, which was just a text change. Additionally, the content of the return policy on the website under the help section and in the email confirming the order has been edited.

What problems have been solved?

There was a problem with locating the content from the help section within the project. Once located, the old record has been hidden so it would not be deleted, with a new record added in the same place. An obstacle was also encountered when trying to alter the contents of the confirmation email - the text couldn't be found within the project to be updated. Access to SendGrid to modify the email's content was granted, but a two-step verification was required to log in. The code provided was no longer valid when it was attempted to be used for access, a further attempt to gain access will be made.

What technologies, tools or solutions have been used?

ruby, rails, docker-compose, sendgrid, mailchimp

Maciej:

What functionalities have been delivered?

Maciej successfully implemented functionalities related to Pundit (policy regarding the use of admin and user). The policy was carefully incorporated into the appropriate segments of the code. The issue concerning username change error was resolved. When a folder was deleted, it was impossible to rename it - however, this error was fixed as well. He also introduced an adjustment concerning layout changes and checkboxes. Lastly, he spent some time figuring out subscriptions, although more conceptual information is necessary on this aspect.

What problems have been solved?

Maciej successfully solved problems with name changing, implemented policy into numerous segments in the code, modified the checkbox layout, and added a fix to the folder renaming issue.

What technologies, tools or solutions have been used?

Shopify, Spina, Pundit, RSpec, Stub

Using Docker for Rails development

JaafarMehdi — Tue, 14 Mar 2023 09:32:27 +0000

Recently I got assigned to an old project and while luckily it had instructions on how to set it up locally in the Read.me the number of steps was damn too high. So instead of wasting half a day executing them, I wasted 2 days automating them (future devs will thank me… maybe).
I wanted to get as close as possible to a one command local env setup.

To simplify the process as you have guessed from the title of the article I decided to set up docker and docker-compose. I started with an example config from awesome-compose.

Level 0: just get the app to run on docker

FROM ruby:2.6.5


WORKDIR /myapp
COPY Gemfile /myapp/Gemfile
COPY Gemfile.lock /myapp/Gemfile.lock
RUN bundle install


COPY . /myapp
COPY config/database.yml.example /myapp/config/database.yml


CMD bundle exec rails s -p 8080 -b '0.0.0.0'

I won’t enter into details too much the main concept is:

first, copy just the gemfiles and install gems (this is done to take advantage of the caching mechanism in building images to speed up rebuilding)
copy the rest of the app code
create a config file for the database based on a template (the details will be passed through ENV variables)

Since this time the goal is local developments assets compilation is not an issue we want to optimize.

Next, the maestro orchestrating the local env: docker-compose

services:
  db:
    image: postgres
    volumes:
      - ./tmp/db:/var/lib/postgresql/data
    environment:
      POSTGRES_PASSWORD: password
  web:
    build: .
    command: bash -c "rm -f tmp/pids/server.pid && bundle exec rails s -p 3000 -b '0.0.0.0'"
    ports:
      - "3000:3000"
    environment:
      POSTGRES_HOST: db
      REDIS_URL: redis://redis
      REDIS_HOST: redis
    depends_on:
      - db
      - redis
  redis:
    image: 'redis:5-alpine'
    command: redis-server

Most of it is thanks to the help of awesome-compose
What we want to achieve here:

Run our app alongside a postgres database and a redis server
Have the postgres database data persist
Use env variables to pass the component names for routing

As a warning note: avoid mapping any port that’s not necessary. So no mapping of 3456:3456 for postgres db. While it’s not an issue for local dev if you use a similar config on a production server it could open you to some brute-force attacks.

The odd thing: you may have noticed that we have both a REDIS_URL and a REDIS_HOST env. It’s because the default redis cache use requires the url param to have the protocol redis:// at the beginning, but our app also uses redis_store for session_store which requires just the host without the protocol. Chances are you don’t use redis_store and can safely remove the unnecessary env.

Level 1: add some dev quality-of-life configs

So this config was enough for me to get rolling and work on the project and contribute but it’s not the most comfortable thing due to a couple of issues:

Any change to the code required turning off the docker-compose instances, rebuilding and turning the infrastructure on again and that adds up.
I can't use binding.pry do debug

So to be able to cut on the restarts necessary and auto update code we will use volumes:

  web:
    build: .
    volumes:
      - .:/myapp

Adding a volume will mirror the current dir to the app dir in the image will automatically mirror any change we make to the code. While this won't help with changes to config files it will handle most changes in real-time (Assuming we run on typical dev env configs)

To enable access to binding.pry you need to add the following to the web service

  web:
    build: .
    volumes:
      - .:/myapp
    tty: true # for binding.pry
    stdin_open: true # for binding.pry

But adding this won’t allow us to just access the debug console from the console used to run docker-compose we will need to find the id of the web process and connect to it:

docker ps
 docker attach 75cde1ab8133

Level 2: add some reusability and sidekiq

One last thing missing now in my case was the possibility to also run sidekiq. It was not necessary in the beginning but once the development went on we needed to work on some async tasks to handle long data import processes.

x-my-app: &my_app
 build: .
 volumes:
   - .:/myapp
 environment:
   POSTGRES_HOST: db
   REDIS_URL: redis://redis
   REDIS_HOST: redis
 depends_on:
   - db
   - redis
 tty: true # for binding.pry
 stdin_open: true # for binding.pry


services:
 db:
   image: postgres
   volumes:
     - ./tmp/db:/var/lib/postgresql/data
   environment:
     POSTGRES_PASSWORD: password
 web:
   <<: *my_app
   command: bash -c "rm -f tmp/pids/server.pid && bundle exec rails s -p 3000 -b '0.0.0.0'"
   ports:
     - "3000:3000"
 sidekiq:
   <<: *my_app
   command: 'sidekiq -L ./log/sidekiq.log -C ./config/sidekiq-dev.yml -P tmp/pids/sidekiq.pid'
 redis:
   image: 'redis:5-alpine'
   command: redis-server

So as you can see before starting to define services we created a block that will contain all the common configs between the rails app and the sidekiq process containers. The only differences are the command and the ports since sidekiq doesn't need any open and we can’t have 2 containers trying to occupy the same port anyway.

And with this addition, I’d consider the docker config done for this project. While it still requires more than one command to setup initially with the database creation/migration/seeding it’s a step requiring additional scripting on top of docker-compose. Maybe for another article.
The commands toolbox

As a parting note i’d like to list the docker command i found myself using most often:

docker-compose up --build // start the app
docker-compose down // stop it if running in the background
docker-compose ps // check the status of currently running images
docker-compose run web bash // run commands
docker images // show list of built images
docker image prune // delete unused images since they can take up a lot of hard drive
docker ps / docker kill // if docker-compose not doing the job

Tip: you may consider creating an alias for docker-compose to avoid typing it every time.

Refinements in ruby

Adam Piotrowski — Tue, 22 Mar 2022 09:07:39 +0000

Today I had the opportunity to use https://docs.ruby-lang.org/en/2.4.0/syntax/refinements_rdoc.html for the first time in my almost 8 years of Ruby programming. So in general it works in the way that you can override some classes in Ruby but the change is not globally. It creates the module, applies monkey patching, and using ModuleName. That makes things overridden only in the context of the runtime of that using.

Let's describe it with my use case example. In application, we had an evaluator for formulas. Implementation details are not important, what we should focus on is that after parsing data we had a simple eval like that:

max( nil * 10, 20 )

Normally any operation like * on a nil object raises an error and that was ok. But in this case, we have a proper value of 20 which should be the result of that code since it is the maximum value from passed values.

So my solution is to expand basic arithmetic methods in NilClass, and thanks to using refinement i did it only in the context of our evaluator without causing any changes in the rest of application.

  class Evaluator
    Refinement = Module.new do
      refine NilClass do
        %i[- + / *].each do |operator|
          define_method(operator) { |_| nil }
        end
      end
    end

    using Refinement

    def initialize
      (...)
     end
  end

Graphql-ruby and passing ISO8601 DateTimes without timezone

Adam Piotrowski — Fri, 25 Jun 2021 10:00:00 +0000

If you are writing web apps for some time, you had to encounter at least a few issues/situations when you had to deal with Timezones. If you are a Ruby developer, and you use Ruby on Rails then you are probably familiar with the difference of Time.now, Time.current and Time.zone.now. (if not, that is quick blogpost for one of those teams that are setting the standard of how to write Ruby code : https://thoughtbot.com/blog/its-about-time-zones)

Conclusions of that blogpost (and many others) are simple: use those methods that are referring to your app timezone and do not mess with that if you want to have smooth to maintain time-related code.

Now, let’s focus for a moment on a gem that helped a lot of us to have a nice time during working with our graphl API: https://github.com/rmosolgo/graphql-ruby .

What is easy to forget, that this gem is for all ruby applications. Not only for rails-based.

If you are thinking about that gem as an addition to your rails app that is following “standard” approaches: remember that this gem doesn’t have access to Rails DateAndTime::Zones class for example.

That is Time class ancestors from gem:

$: Time.ancestors
=> [Time, Comparable, Object, ErrorBubblingHelpers, Minitest::Expectations, PP::ObjectMixin,
 JSON::Ext::Generator::GeneratorMethods::Object, Kernel, BasicObject]

and that one is how Time ancestors looks like if we load default rails console:

$: Time.ancestors

=> [Time, DateAndTime::Compatibility, DateAndTime::Calculations, DateAndTime::Zones, Comparable, ActiveSupport::ToJsonWithActiveSupportEncoder, Object, ActiveSupport::Dependencies::Loadable, PP::ObjectMixin, JSON::Ext::Generator::GeneratorMethods::Object, ActiveSupport::Tryable, Kernel, BasicObject]

Why is it important? Because when you realize that, you can assume and check it in the gem codebase to be sure how exactly it works if it comes to ISO8601DateTime types when they don't have a TimeZone passed from client side.

As you probably already suspect - it parse it using server timezone, not application timezone (because it doesn’t have access to it by default).

Conslusion:

Remember that graphql-ruby is a gem prepared not only for rails environment, so do not expect it will be operating on your rails app timezone. Ensure that your system have the same timezone as your app, and enjoy graphl-ruby gem, as you were before!

P.S. (few tips for setup that we mostly work with)

App: To change timezone in app, go to config/environments/.rb and set:

config.time_zone = "UTC"

System To change linux based system timezone just run

 sudo ln -fs /usr/share/zoneinfo/UTC /etc/localtime

Postgres Database :
- Run psql
- show config_file;
- because SET TIME ZONE 'UTC; -from psql will work only until next restart of postgres to change timezone open config file and set

  timezone = 'UTC'

Rails CD with Docker, Github Actions and VPS

Marek Stocki — Wed, 09 Jun 2021 11:46:19 +0000

I want to show you how to deploy your app to production with minimal cost and make the deployment process fully automated. If you have never done it before, this post will show you how to achieve it step by step. Maybe you have already deployed some apps, then you know that there are always some problems, especially when the server is used by multiple applications. This approach isn’t something innovative, there are many blog posts where you can learn how to dockerize apps, how to use GitHub Actions, and how to deploy code to VPS, but this tutorial brings it all together.

Docker

The whole idea is based on Docker's image. So the first thing to do is Docker installation. You can skip that part if you have already installed it.

Install Docker

For more details check the official site.

# Update the apt package index and install packages to allow apt to use a repository over HTTPS:
$ sudo apt-get update && apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common

# Add Docker’s official GPG key
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

# Update the apt package index and install the latest version of Docker Engine and Containerd
$ sudo apt-get update && apt-get install docker-ce docker-ce-cli containerd.io

# Verify that Docker Engine is installed correctly by running the hello-world image
$ sudo docker run hello-world

Manage Docker as a non-root user

For more details check official site

# Create the docker group
$ sudo groupadd docker

# Add your user to the docker group
$ sudo usermod -aG docker $USER

# Activate the changes to groups (only Linux)
$ newgrp docker

# Verify that you can run docker commands without sudo
$ docker run hello-world

Dockerfile

Docker is creating images using Dockerfile - it's a file with all commands that are executed during the build. I will show you the simplest version that will work. Later I will improve it and shorten the build time. Create a file Dockerfile in the main app directory.

#1 This is the official Ruby image (https://hub.docker.com/_/ruby) - a complete Linux system with Ruby installed
FROM ruby:3.0.1

#2 Install applications needed for building Rails app
RUN apt-get update && apt-get install -y \
 build-essential libpq-dev nodejs zlib1g-dev liblzma-dev

#3 The WORKDIR instruction sets the working directory for any RUN, CMD, ENTRYPOINT, COPY and ADD
# If a directory doesn’t exist, it will be created
WORKDIR /app

#4 Copy files from current location to image WORKDIR
COPY . . 

#5 Install gems in the image
RUN bundle install

#6 Command that will be executed when you run the image
CMD bundle exec rails s -p 3000 -b '0.0.0.0'

Now let’s test it and create an image with the name rails_app.

$ docker build -t rails_app .
Sending build context to Docker daemon  86.65MB
Step 1/6 : FROM ruby:3.0.1
3.0.1: Pulling from library/ruby
d960726af2be: Pull complete 
## part ommited
Status: Downloaded newer image for ruby:3.0.1
 ---> 9cba361e78fe
Step 2/6 : RUN apt-get update && apt-get install -y  build-essential libpq-dev nodejs zlib1g-dev liblzma-dev
 ---> Running in fa0bce0b6b81
Get:1 http://deb.debian.org/debian buster InRelease [121 kB]
## part ommited
Removing intermediate container 40b752bd0ef3
 ---> 7d09aa5c9ced
Step 3/6 : WORKDIR /app
 ---> Running in 427dea58acb0
Removing intermediate container 427dea58acb0
 ---> 8ed87d4b0643
Step 4/6 : COPY . .
 ---> 0b3a695a0987
Step 5/6 : RUN bundle install
 ---> Running in 65a2592eca90
Fetching gem metadata from https://rubygems.org/............
Fetching rake 13.0.3
Installing rake 13.0.3
## part ommited
Removing intermediate container 65a2592eca90
 ---> 55d9368c4b98
Step 6/6 : CMD bundle exec rails s -p 3000 -b '0.0.0.0'
 ---> Running in 795356f8553e
Removing intermediate container 795356f8553e
 ---> 2466c41ac676
Successfully built 2466c41ac676
Successfully tagged rails_app:latest

The image is successfully built, to check available images you can use this command

$ docker images

Now it's time to run the container with the application and check if it works.

#-p parm allows to map ports with scheme EXPOSED_PORT:IMAGE_INTERNAL_PORT
$ docker run -p 3001:3000 rails_app

Open the browser and go to http://localhost:3001/ - there is a little success, Rails application is working partially:

This is an error from Rails, so Rails is working. Still, there is a problem with the database. There must be another container with the Postgres application and connection between these containers. To achieve it I will use Docker Compose.

Docker Compose

This is a tool that allows to run multiple containers and create a network between them. The configuration file is stored as YAML.

Install Docker Compose

For more details check official site

# Download the current stable release of Docker Compose
# To install a different version of Compose, substitute 1.29.2 with the version of Compose you want to use.
$ sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

# Apply executable permissions to the binary
$ sudo chmod +x /usr/local/bin/docker-compose

# Test the installation
$ docker-compose --version

Compose config file

Create a file docker-compose.yaml in the main app directory.

version: "3"
services:
  database:
    # Official postgres image available in https://hub.docker.com/
    image: postgres
    # There are many types of volumes, this is a named volume, which will store database in docker directory
    # Named volumes must be listed under the top-level volumes key, as shown at bottom of the file
    volumes:
      - db_data:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=password
  web:
    image: rails_app
    # Command will replace CMD from Dockerfile
    command: bash -c "rm -f tmp/pids/server.pid && bundle exec rails s -p 3000 -b '0.0.0.0'"
    # Path on the host, relative to the Compose file. 'app' is a WORKDIR name from Dockerfile
    # This volume will allow you to run the Rails app with Docker Compose
    # and made live changes without rebuilding the image
    volumes:
      - .:/app
    ports:
      - "3001:3000"
    # 'database' is Postgres service name from the top of the file - it will allow communication between containers
    depends_on:
      - database
    environment:
      - POSTGRES_PASSWORD=password
      - POSTGRES_USERNAME=postgres
      - POSTGRES_HOST=database  # it's Postgres service name from the top of the file
volumes:
  db_data:

Now it's time to run Rails application and Postgres database with Docker Compose, but before you must update Rails database config file, create a database, and run migrations.

#config/database.yaml

default: &default
  adapter: postgresql
  encoding: unicode
  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
  username: <%= ENV['POSTGRES_USERNAME'] %>
  password: <%= ENV['POSTGRES_PASSWORD'] %>
  host: <%= ENV['POSTGRES_HOST'] %>

development:
  <<: *default
  database: rails_app_development

test:
  <<: *default
  database: rails_app_test

production:
  <<: *default
  database: rails_app_production

After updating the database file Docker image needs to be rebuild.

$ docker build -t rails_app .

The first 3 steps are cached, but changes in the application directory cause gems installation. I will show you later how to avoid it and use cache.

Now start containers and in another terminal window run a command to create a database.

# Run command and leave it running
$ docker-compose up
# From another terminal window
$ docker-compose run web rake db:create db:migrate

Open the browser and go to http://localhost:3001/ and... You just run the Rails app with Docker.

VPS

The next piece of the puzzle is VPS - a place where you deploy application. You can find many companies that provide cloud services and it's your decision which one you choose. I wanna show you an example based on a server with Ubuntu. Like on your localhost, firstly you install Docker and Docker Compose on VPS. Use steps from the beginning of this post. You will need two additional non-root users: nginx_proxy and rails_app.

$ sudo adduser nginx_proxy
$ sudo adduser rails_app
# Add new users to the docker group
$ sudo usermod -aG docker nginx_proxy
$ sudo usermod -aG docker rails_app

HTTP server

For HTTP server I will use NGINX with this awesome application nginx-proxy and acme-companion for automatic SSL certificate generation. Connect to the server as nginx_proxy user and create two files docker-compose.yaml and nginx_custom.conf.

$ cd ~ && touch docker-compose.yaml nginx_custom.conf

I will show you the basic configuration of these two applications. For more details check the app's documentation from the links above.

# docker-compose.yaml
version: '3.9'

services:
  nginx-proxy:
    restart: always
    image: nginxproxy/nginx-proxy
    container_name: nginx-proxy
    ports:
      - 80:80
      - 443:443
    volumes:
      - conf:/etc/nginx/conf.d
      - vhost:/etc/nginx/vhost.d
      - html:/usr/share/nginx/html
      - dhparam:/etc/nginx/dhparam
      - certs:/etc/nginx/certs:ro
      - /var/run/docker.sock:/tmp/docker.sock:ro
      - ./nginx_custom.conf:/etc/nginx/conf.d/nginx_custom.conf
    networks:
      nginx-proxy-network:

  letsencrypt:
    restart: always
    image: nginxproxy/acme-companion
    container_name: nginx-proxy-acme
    volumes_from:
      - nginx-proxy
    volumes:
      - certs:/etc/nginx/certs:rw
      - acme:/etc/acme.sh
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      nginx-proxy-network:

volumes:
  conf:
  vhost:
  html:
  dhparam:
  certs:
  acme:

networks:
  nginx-proxy-network:
    name: "nginx-proxy-network"

# nginx_custom.conf
# here you can customize NGINX
server_tokens off;
client_max_body_size 100m;

When these two files are created and filled with content, let's run the NGINX server with deamon (-d param)

$ docker-compose up -d

And that's all you need to do with the HTTP server - this app will handle all new Rails applications on your server with few ENV variables that you will add to the Rails app docker-compose files.

Rails app - Production Docker Compose file

Let's connect to the server as rails_app user. You must create two files docker-compose.yaml and .env on the server and copy the below content to these files.

$ cd ~ && touch docker-compose.yaml .env

In the production version you must pass more ENV variables, so let's create a file to store these variables separately. Also, you must remember that every file created in the Docker image during the app life cycle will be deleted with the new app version release. So e.g. files from ActiveStorage or logs need to be stored outside of the image.

version: "3"
services:
  database:
    # restart docker container when there will be a crash
    restart: always
    image: postgres
    volumes:
      - db_data:/var/lib/postgresql/data
    # instead of environment let's use the env file
    env_file: .env
  web:
    restart: always
    image: rails_app
    command: bash -c "rm -f tmp/pids/server.pid && bundle exec rails s -p 3000 -b '0.0.0.0'"
    env_file: .env
    environment:
      - VIRTUAL_HOST=your_dns_for_rails_app.com # it will allow nginx-proxy to redirect HTTP request to your Rails app
      # LETSENCRYPT variables are used by acme-companion and it will create SSL certificate for those params
      - LETSENCRYPT_HOST=your_dns_for_rails_app.com
      - LETSENCRYPT_EMAIL=some_user@your_dns_for_rails_app.com
    volumes:
      - ./storage:/app/storage # store ActiveStorage files in `storage` directory
      - ./log:/app/log # store logs in `log` directory
    ports:
      - 3001:3000
    depends_on:
      - database
volumes:
  db_data:
networks:
  default:
    external:
      name: nginx-proxy-network

Example env file:

POSTGRES_PASSWORD=password
POSTGRES_USERNAME=postgres
POSTGRES_HOST=database
RAILS_ENV=production
SECRET_KEY_BASE=some_secret_key
RAILS_LOG_TO_STDOUT=true
RAILS_SERVE_STATIC_FILES=true

Rails app - Production Dockerfile

The main difference is a need to precompile assets to run the production environment. To do it with Rails and Webpacker, then also Yarn is needed. Let’s update Dockerfile to handle it and fix gems caching.

FROM ruby:3.0.1
# add yarn to apt list
RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - \
  && echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
# add yarn to installed apps
RUN apt-get update && apt-get install -y \
 build-essential libpq-dev nodejs zlib1g-dev liblzma-dev yarn
WORKDIR /app
# copy Gemfile and Gemfile.lock and install gems before copying rest of the application
# so the steps will be cached until there won't be any changes in Gemfile
COPY Gemfile* ./
RUN bundle install
COPY . .
# precompile assets with temporary secret key base
RUN SECRET_KEY_BASE="assets_compile" RAILS_ENV=production bundle exec rake assets:precompile
CMD bundle exec rails s -p 3000 -b '0.0.0.0'

GitHub Actions

When the production Rails app on Docker image is fully working and VPS is ready, it's time to create an image with GitHub Actions and store it in GitHub Container Registry. Before I show you the config file, there are few things to do in GitHub.

GitHub Container Registry (GHCR) is in an experimental state, so you must enable that feature with this tutorial.
Second thing needed is token, which allow to login to GHCR - tutorial (select two scopes: write:packages and delete:packages)
Create repository secrets. Go to your repository -> Settings -> Secrets and add New repository secret and create two secrets: CR_PAT with GHCR token and VPS_PASSWORD - its password for user rails_app.

Then log in to your server with rails_app user and edit bashrc file. Add a line at the end of the file:

export CR_PAT=<your GHCR token>`

In your project create a file in that path /.github/workflows/deploy.yml

name: Deploy
on:
  push:
    branches:
      # Run deploy job on every push to the master branch
      - master
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      -
        name: Checkout
        uses: actions/checkout@v2
      -
        name: Login to GitHub Container Registry
        run: echo ${{ secrets.CR_PAT }} | docker login ghcr.io -u <YOUR GITHUB LOGIN> --password-stdin
      -
        name: Pull image to use as a cache
        run: docker pull ghcr.io/<YOUR GITHUB LOGIN>/rails_app:latest || exit 0
      -
        name: Build Docker image
        run: docker build . --cache-from ghcr.io/<YOUR GITHUB LOGIN>/rails_app:latest --tag ghcr.io/<YOUR GITHUB LOGIN>/rails_app:latest
      -
        name: Push the image to GitHub Container Registry
        run: docker push ghcr.io/<YOUR GITHUB LOGIN>/rails_app:latest
      -
        name: VPS - pull image and run app containters
        uses: appleboy/ssh-action@master
        with:
          host: <your-server-ip>
          username: rails_app 
          password: ${{ secrets.VPS_PASSWORD }}
          script: |
            echo $CR_PAT | docker login ghcr.io -u <YOUR GITHUB LOGIN> --password-stdin
            docker-compose pull web
            docker-compose up -d --no-deps

After first successful deploy, login to your server as rails_app and create database with command:

$ docker-compose run web rake db:create db:migrate

The last improvement

The final touch to make deployment fully automated is the migration script. Create a file docker-entrypoint.sh in your project main directory and paste the below content.

#!/bin/sh
set -e

if [ -f tmp/pids/server.pid ]; then
  rm tmp/pids/server.pid
fi

bundle exec rails db:migrate 2>/dev/null
exec bundle exec "$@"

And then few changes are needed in Dockerfile:

FROM ruby:3.0.1
RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - \
  && echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
RUN apt-get update && apt-get install -y \
 build-essential libpq-dev nodejs zlib1g-dev liblzma-dev yarn
WORKDIR /app
COPY Gemfile* ./
RUN bundle install
COPY . .
RUN SECRET_KEY_BASE="assets_compile" RAILS_ENV=production bundle exec rake assets:precompile
# Add entrypoint script to handle migrations
ENTRYPOINT [ "./docker-entrypoint.sh" ]
EXPOSE 3000
CMD ["rails", "server", "-b", "0.0.0.0"]

You just created a fully working continuous deployment. You don't have to worry about errors on your local machine or some problems with the internet connection anymore. Just write your code and simply push commit and the rest is magic. Below some useful commands that may help you.

# view logs from Postgres
$ docker-compose logs database -f

# view logs from Rails
$ docker-compose logs web -f

# run Rails console inside Docker container
$ docker-compose run web rails c

# list available images
$ docker images

# list running containers
$ docker ps

# stop containers
$ docker-compose down

# remove old images/containers
$ docker system prune

Application testing with Apache JMeter

IamMatt — Wed, 07 Oct 2020 10:14:41 +0000

What is Apache JMeter?

It is a tool for carrying out performance tests of the server and database used by our application. Such tests allow us to estimate how many users of our application will be able to use it in comfortable conditions and will allow us to see when we should change to a more efficient server. Such tests also allow us to verify which parts of the code are inefficient and require rewriting.

When we run JMeter for the first time, we have the impression that it is quite a complicated tool, but there is nothing wrong with it. The GUI is very intuitive, and after a few minutes, we can learn the ropes of it, and start using it for its basic functions. We will use BlazeMeter to prepare the tests. It is an extension to the Chrome browser, which saves everything we do in the program and can play it back later. Based on these recordings, we can also export the file that we can add to the JMeter instead of adding everything manually.

Application testing

The development process was practically finished. The client decided that it was a good time to put the application into use for ordinary users, so we decided to check for the last time how the application will behave with really large numbers of users. We are talking about 20,000 people online. I recorded the registration process and job search using BlazeMeter and exported a JMX file in which all the activities that JMeter needs to run are saved.

The next thing you need to do is to determine how many users you want JMeter to fake. In my case, it was 20,000, so we also need to determine how many of them should perform their tasks per second. I think the 10-50 range will be appropriate in this situation. So we click on "Thread group" and set specify those options:

Number Of Threads - the number of threads used to send requests,
Ramp-Up Period - in how many seconds the requests are to be sent,
Loop Count - how many times a given test is to be repeated,
Delay Thread creation until needed - without this option checked, JMeter will allocate all memory for all threads immediately. This means that even if the user performs his sample after e.g. 30 minutes, the memory will be reserved for him immediately after running the script,
Scheduler - we can set what day and time we want to run the test.

Enabling tests is very easy. It only requires clicking on the "PLAY" icon. Now we have to wait. How much? It is hard to say because it depends on how many tests we do, how many users, and what computer we have.

Summary of tests

It turned out that the application is inefficient in several places, and it takes too long for the server to respond. We managed to identify places that should be changed or rewritten. Where the code was very slow, we decided to use SQL Views to get the best server response time.

Source: jmeter.apache.org

How to make UI for neovim plugins in Lua

Rafał Camlet — Mon, 20 Jul 2020 11:14:45 +0000

In the last article, we saw the basics of creating plugins in Lua using floating windows. Now it's time for a more traditional approach. Let's create a simple plugin that will show us last opened files in handy side navigation. As we focus on learning the interface, we will use vim native oldfiles list for this purpose. It will look something like this:

If you didn't read previous article, I highly recommend you to do so, because this article expands on the ideas from the last one and is full of new things in comparison.

Plugin window

Ok, so we should start by writing a function that will create our first window, where the oldfiles list will be displayed. But first, we will declare three variables in the main scope of our script: buf and win that will contain our navigation window and buffer references and start_win that will remember the position where we opened our navigation. We will be using these often across our plugin functions.

-- It's our main starting function. For now we will only creating navigation window here.
local function oldfiles()
  create_win()
end

local function create_win()
  -- We save handle to window from which we open the navigation
  start_win = vim.api.nvim_get_current_win()

  vim.api.nvim_command('botright vnew') -- We open a new vertical window at the far right
  win = vim.api.nvim_get_current_win() -- We save our navigation window handle...
  buf = vim.api.nvim_get_current_buf() -- ...and it's buffer handle.

  -- We should name our buffer. All buffers in vim must have unique names.
  -- The easiest solution will be adding buffer handle to it
  -- because it is already unique and it's just a number.
  vim.api.nvim_buf_set_name(buf, 'Oldfiles #' .. buf)

  -- Now we set some options for our buffer.
  -- nofile prevent mark buffer as modified so we never get warnings about not saved changes.
  -- Also some plugins treat nofile buffers different.
  -- For example coc.nvim don't triggers aoutcompletation for these.
  vim.api.nvim_buf_set_option(buf, 'buftype', 'nofile')
  -- We do not need swapfile for this buffer.
  vim.api.nvim_buf_set_option(buf, 'swapfile', false)
  -- And we would rather prefer that this buffer will be destroyed when hide.
  vim.api.nvim_buf_set_option(buf, 'bufhidden', 'wipe')
  -- It's not necessary but it is good practice to set custom filetype.
  -- This allows users to create their own autocommand or colorschemes on filetype.
  -- and prevent collisions with other plugins.
  vim.api.nvim_buf_set_option(buf, 'filetype', 'nvim-oldfile')

  -- For better UX we will turn off line wrap and turn on current line highlight.
  vim.api.nvim_win_set_option(win, 'wrap', false)
  vim.api.nvim_win_set_option(win, 'cursorline', true)

  set_mappings() -- At end we will set mappings for our navigation.
end

Drawing function

Okay, so we have a window, now we need something to display in it. We will use vim oldfiles special variable, which stores paths to previously opened files. We will take as many items from it, as we can display without scrolling, but of course, you can take as many as you want in your script. We will call this function redraw because it can be used to refresh navigation content. File paths might be long, so we will try to make them relative to the working directory.

local function redraw()
  -- First we allow introduce new changes to buffer. We will block that at end.
  vim.api.nvim_buf_set_option(buf, 'modifiable', true)

  local items_count =  vim.api.nvim_win_get_height(win) - 1 -- get the window height
  local list = {}

  -- If you using nightly build you can get oldfiles like this
  local oldfiles = vim.v.oldfiles
  -- In stable version works only that
  local oldfiles = vim.api.nvim_get_vvar('oldfiles')

  -- Now we populate our list with X last items form oldfiles
  for i = #oldfiles, #oldfiles - items_count, -1 do

    -- We use build-in vim function fnamemodify to make path relative
    -- In nightly we can call vim function like that
    local path = vim.fn.fnamemodify(oldfiles[i], ':.')
    -- and this is stable version:
    local path = vim.api.nvim_call_function('fnamemodify', {oldfiles[i], ':.'})

    -- We iterate form end to start, so we should insert items
    -- at the end of results list to preserve order
    table.insert(list, #list + 1, path)
  end

  -- We apply results to buffer
  vim.api.nvim_buf_set_lines(buf, 0, -1, false, list)
  -- And turn off editing
  vim.api.nvim_buf_set_option(buf, 'modifiable', false)
end

We can now update our main function. We will also add some code that prevents opening multiple navigation windows. For this purpose, we can use nvim_win_is_valid which checks if our plugin window already exists.

local function oldfiles()
  if win and vim.api.nvim_win_is_valid(win) then
    vim.api.nvim_set_current_win(win)
  else
    create_win()
  end

  redraw()
end

Openings files

We can now look at our oldfiles, but it would be much handier if we can also open them. We will allow users to open files in 5 different ways! In a new tab, in horizontal or vertical splits, in the current window and in preview mode, which will keep the focus on navigation.

Let's start by opening files in the current window. We should prepare for two scenarios:

Opening a file in the window from which the user opens navigation.
Closing the starting window, when we will create a new one for opening file.

local function open()
  -- We get path from line which user push enter on
  local path = vim.api.nvim_get_current_line()

  -- if the starting window exists
  if vim.api.nvim_win_is_valid(start_win) then
    -- we move to it
    vim.api.nvim_set_current_win(start_win)
    -- and edit chosen file
    vim.api.nvim_command('edit ' .. path)
  else
    -- if there is no starting window we create new from lest side
    vim.api.nvim_command('leftabove vsplit ' .. path)
    -- and set it as our new starting window
    start_win = vim.api.nvim_get_current_win()
  end
end

-- After opening desired file user no longer need our navigation
-- so we should create function to closing it.
local function close()
  if win and vim.api.nvim_win_is_valid(win) then
    vim.api.nvim_win_close(win, true)
  end
end

-- Ok. Now we are ready to making two first opening functions

local function open_and_close()
  open() -- We open new file
  close() -- and close navigation
end

local function preview()
  open() -- WE open new file
  -- but in preview instead of closing navigation
  -- we focus back to it
  vim.api.nvim_set_current_win(win)
end

-- To making splits we need only one function
local function split(axis)
  local path = vim.api.nvim_get_current_line()

  -- We still need to handle two scenarios
  if vim.api.nvim_win_is_valid(start_win) then
    vim.api.nvim_set_current_win(start_win)
    -- We pass v in axis argument if we want vertical split
    -- or nothing/empty string otherwise.
    vim.api.nvim_command(axis ..'split ' .. path)
  else
    -- if there is no starting window we make new on left
    vim.api.nvim_command('leftabove ' .. axis..'split ' .. path)
    -- but in this case we do not need to set new starting window
    -- because splits always close navigation 
  end

  close()
end

And in the end the simplest opening in new tab.

local function open_in_tab()
  local path = vim.api.nvim_get_current_line()

  vim.api.nvim_command('tabnew ' .. path)
  close()
end

For everything to work, we need to add the key mappings, export all public functions, and add a command to trigger our navigation.


local function set_mappings()
  local mappings = {
    q = 'close()',
    ['<cr>'] = 'open_and_close()',
    v = 'split("v")',
    s = 'split("")',
    p = 'preview()',
    t = 'open_in_tab()'
  }

  for k,v in pairs(mappings) do
    -- let's assume that our script is in lua/nvim-oldfile.lua file.
    vim.api.nvim_buf_set_keymap(buf, 'n', k, ':lua require"nvim-oldfile".'..v..'<cr>', {
        nowait = true, noremap = true, silent = true
      })
  end
end

-- at file end
return {
  oldfiles = oldfiles,
  close = close,
  open_and_close = open_and_close,
  preview = preview,
  open_in_tab = open_in_tab,
  split = split
}

command! Oldfiles lua require'nvim-oldfile'.oldfiles()

And that's it! Have fun and make grate things!

The whole plugin

local buf, win, start_win

local function open()
  local path = vim.api.nvim_get_current_line()

  if vim.api.nvim_win_is_valid(start_win) then
    vim.api.nvim_set_current_win(start_win)
    vim.api.nvim_command('edit ' .. path)
  else
    vim.api.nvim_command('leftabove vsplit ' .. path)
    start_win = vim.api.nvim_get_current_win()
  end
end

local function close()
  if win and vim.api.nvim_win_is_valid(win) then
    vim.api.nvim_win_close(win, true)
  end
end

local function open_and_close()
  open()
  close()
end

local function preview()
  open()
  vim.api.nvim_set_current_win(win)
end

local function split(axis)
  local path = vim.api.nvim_get_current_line()

  if vim.api.nvim_win_is_valid(start_win) then
    vim.api.nvim_set_current_win(start_win)
    vim.api.nvim_command(axis ..'split ' .. path)
  else
    vim.api.nvim_command('leftabove ' .. axis..'split ' .. path)
  end

  close()
end

local function open_in_tab()
  local path = vim.api.nvim_get_current_line()

  vim.api.nvim_command('tabnew ' .. path)
  close()
end


local function redraw()
  vim.api.nvim_buf_set_option(buf, 'modifiable', true)
  local items_count =  vim.api.nvim_win_get_height(win) - 1
  local list = {}
  local oldfiles = vim.api.nvim_get_vvar('oldfiles')

  for i = #oldfiles, #oldfiles - items_count, -1 do
    pcall(function()
      local path = vim.api.nvim_call_function('fnamemodify', {oldfiles[i], ':.'})
      table.insert(list, #list + 1, path)
    end)
  end

  vim.api.nvim_buf_set_lines(buf, 0, -1, false, list)
  vim.api.nvim_buf_set_option(buf, 'modifiable', false)
end

local function set_mappings()
  local mappings = {
    q = 'close()',
    ['<cr>'] = 'open_and_close()',
    v = 'split("v")',
    s = 'split("")',
    p = 'preview()',
    t = 'open_in_tab()'
  }

  for k,v in pairs(mappings) do
    vim.api.nvim_buf_set_keymap(buf, 'n', k, ':lua require"nvim-oldfile".'..v..'<cr>', {
        nowait = true, noremap = true, silent = true
      })
  end
end

local function create_win()
  start_win = vim.api.nvim_get_current_win()

  vim.api.nvim_command('botright vnew')
  win = vim.api.nvim_get_current_win()
  buf = vim.api.nvim_get_current_buf()

  vim.api.nvim_buf_set_name(0, 'Oldfiles #' .. buf)

  vim.api.nvim_buf_set_option(0, 'buftype', 'nofile')
  vim.api.nvim_buf_set_option(0, 'swapfile', false)
  vim.api.nvim_buf_set_option(0, 'filetype', 'nvim-oldfile')
  vim.api.nvim_buf_set_option(0, 'bufhidden', 'wipe')

  vim.api.nvim_command('setlocal nowrap')
  vim.api.nvim_command('setlocal cursorline')

  set_mappings()
end

local function oldfiles()
  if win and vim.api.nvim_win_is_valid(win) then
    vim.api.nvim_set_current_win(win)
  else
    create_win()
  end

  redraw()
end

return {
  oldfiles = oldfiles,
  close = close,
  open_and_close = open_and_close,
  preview = preview,
  open_in_tab = open_in_tab,
  split = split
}

Trailblazer is back, better then ever!

Łukasz Żarski — Fri, 03 Jul 2020 13:09:49 +0000

For those who know Trailblazer - you know that for the last 6 years TRB evolved a few times, introduced a lot of great ideas but also had it's problems: missing documentation, lack of communication with the community, and inconsistent approach to some problems. We strongly believe the current version, along with the support of the core team and the new landing page (with complete documentation) is worth working with.

For those who somehow haven't hear about Trailblazer - check it out. Probably for some complex projects where Rails aren't sufficient that tool could be really helpful and efficent.

The main changes coming with the new version of TRB relate to:

Major API change,
Autoloading,
New website with complete documentation,
Tracing.

API CHANGES

The updated version 2.1 contains some necessary and sound improvements in the method signatures. One of these changes is the replacement of the automatic merging of params part and additional options with the possibility to pass one hash to the call and do the merging. Second major change in API is that ctx elements are now accessible both as strings and symbols like in a hash. It’s worth noticing that ctx object is now documented in its own section.

# old style
result = Memo::Create.(params, "current_user" => current_user)

# new style
result = Memo::Operation::Create.(params: params, current_user: current_user)

AUTOLOADING

The significant change to autoloading is return to Rails file and class naming structure for operations. This change allowed removing the trailblazer-loader gem, which was responsible for causing some issues and inconsistencies.

NEW WEBSITE

The goal behind creating a new website was to make it as simple as possible. A lot of pages accessible through the top navigation, and then the documentation behind the DOCS link.

The documentation has been structured in such a way as to ensure the fastest possible access to consistent information. There's even added a new search function - something that TRB user base has repeatedly asked for. Just press the / (or Shift+7) button in the docs.

TRACING

The 2.1 version comes with the ability to track the operations - something that was missing in all the prior releases. This change makes debugging deeply nested operations way more pleasant.

Either you know TRB already or you never heard about it, check out the blog post about the history and changes of trailblazer 2.1 and give the core team some feedback!

Dead rows in a materialized view

blysband — Tue, 16 Jun 2020 11:23:00 +0000

In the application that I work on, we have a materialized view. It was created from several joined tables. It’s used to speed up searching for data without joining seven or eight tables in every query. At least it should make searching faster but in practice, it didn’t and I will describe to you why.

Queries to that view were really slow, but only on the production server, on staging everything was fine. That view was exactly the same as on staging server, but queries were ~50 times slower on production.

The first step was to check execution plans for any query on both servers.

EXPLAIN ANALYZE SELECT COUNT(*) FROM table_name;

On staging execution plan involved parallel scan, on production index scan.

When I discovered that, the first thing that came to my mind was the possibility of different configurations that disabled parallel queries on production. I checked that and it turned out to be false. My next attempts were to check various Postgres views that kept some data about tables. After looking through many numbers that didn’t help I finally discovered that our table (materialized view actually, but in this context, it makes no difference) on the production server it took 7GB of disc space and on staging only 300MB.

“SELECT pg_size_pretty( pg_total_relation_size('table_name') );”

I expected that row count had to be totally different but unexpectedly it was ~320k on production and ~310k on staging.

After a really long investigation, I found the reason why those numbers were so different, the table on production contained an enormous number of dead rows.

“SELECT n_dead_tup 
FROM pg_stat_all_tables
WHERE relname = ‘table_name’”

At this time I was almost sure that’s the reason for our whole problem (or at least I hoped so). I started researching those dead rows, what they are, and why they appear. The major defect of that approach is that after every single update on one of the records from joined tables, the whole materialized view is refreshed. Refreshed i.e. whole view is dropped and then it’s building query is run. We have two types of a refresh in PostgreSQL:

non concurrently
This refresh type does not produce any dead rows, but for the time of refreshing view is locked, and no data can be read from it. This is an unacceptable solution for my project.
concurrently
It allows reading data during the process. It’s possible thanks to duplicating all data before deleting it. During refresh, all SELECT queries see that duplicated data, and after the process, all queries have access to newly created view, and duplicates remain as dead rows. It’s the way how the view is bloated with tons of unnecessary data.

And here comes VACUUM mechanism that is used to remove all dead rows from the table or materialized view.

“VACUUM table_name”

This command removes all dead queries from the given table, but it has to be run ‘manually’, or by some application code. There is also a mechanism called autovacuum . PostgreSQL has some workers (quantity set in configuration) that all the time search our database and run VACUUM on tables that are in need. What do we know now? We have many dead rows in the materialized view, we also have a mechanism that should clean them. Why isn’t it? In order to check that I added query that logs last auto and vacuum time with dead rows count.

“SELECT n_dead_tup, last_autovacuum, last_vacuum 
FROM pg_stat_all_tables
WHERE relname = ‘table_name’”

After a few days, I checked that logs. The first thing to notice was that the Autovacuum process is working but it’s triggered only in evenings. All dead rows are cleaned and the next day from something like 7 a.m. they appear and it’s count is growing. I checked our search in the morning and indeed it was as fast as it should be, but within a day it was getting much slower. Following really long research why is that happening I found that Autovacuum cannot be run on the table when it’s locked by SHARE UPDATE EXCLUSIVE lock. What locks a table that way? Obviously it’s REFRESH MATERIALIZED VIEW CONCURRENTLY. When that view is refreshed in our application? Other logs added and the answer is: the view is refreshed almost whole time (during a workday from morning to evening). Refresh is as I mentioned triggered by every data update on each of the tables that problematic view is made of. And here comes our final answer. Dead rows aren’t cleaned because Autovacuum cannot be run during refresh and refresh is running continuously from morning to evening.

Possible solutions:

Architectural changes that would prevent an application from refreshing whole materialized view during every data update. This is definitely possible to just update those rows that really changed without dropping and building from the scratch whole table. This is definitely the best solution and I would choose that if I had much more time to spend on fixing that problem, but it would be too long to refactor every place in the application that can update one of the included tables.
The second possible fix is to append some breaks between refreshes that AUTOVACUUM process could be fired on our view. Theoretically, it is a fine idea, it would reduce database overload, it should be quite fast to implement, but I think there is too much room for unforeseen consequences and finally, I decided to give up that idea. Even though adding breaks would not take much time to implement, testing it carefully would and as I noticed above there were no more time after that really long investigation
The third and final solution I anticipated was the simplest one. In the job used to refresh the materialized view, I added code that ‘manually’ vacuums view from dead rows. During that whole research on I run vacuum manually many times and I knew it worked. Going this way assures us that there is no possibility that dead rows stay in view because they are vacuumed as soon as refresh (that creates them) ends.

Weighing up the pros and cons I finally chose the third solution. Time was a problem and it was the only one that could be tested immediately. It worked, but I think refactoring the application to not refresh the whole view would be the best option.

The Art of Project Management - part 2

Łukasz Żarski — Thu, 14 May 2020 11:34:20 +0000

INTRODUCTION

In the first part of the cycle, we talked about initial estimations (in our case project concept) and characteristics of a good leader based on the first chapter of Sun Tzu’s TAoW. Since we established that the preparation for the upcoming new challenges is crucial in terms of accomplishing success, we can now move to adopt a strategy and adjust tactics to our situation while still focusing on our ancient guidance. So let’s try to examine in this light the second and third TAoW chapters!

Waging War - Strategy

The second chapter of TAoW is focusing solely on a concept of waging war, estimating available resources, and how to avoid weakening your army by running out of them. Of course, we don’t have to gather a thousand swift chariots or a hundred thousand mail-clad soldiers, thankfully. Nevertheless, we have to be able to predict how many resources will be needed to complete our campaign - we need to create a project cost estimation.

"He who wishes to fight must first count the cost"

The key to estimating it properly is to prepare beforehand, just like we said in the previous part. The knowledge of what to expect is crucial. It allows us to evaluate how many troops we need to assign for the whole operation and also how to provide for their subsistence. In our case, the base of estimation should be an in-depth discussion with our client. It’s not uncommon that our customer can table indispensable cornerstones of the project that are reasonable, but it’s getting harder for him to describe the smaller elements that might not seem important at first sight (or even unnoticeable). It’s our job to extract as much information as possible, mainly by asking a lot of questions. Those questions are intended to enable us to better understand his expectations and the amount of work that will be required. It’s worth keeping in mind that this estimation won’t be certain. It’s more about predicting the cost than counting the actual cost.

After getting all the information that we could get from our client, we could now prepare some scenarios of how using the actual app would look like. I mean, of course, well-known user stories. Those are short descriptions of functionality told from the user's perspective - user cases. The idea behind them is to organize work - basing on them it is easier to section off smaller parts of tasks from the entirety of a project. “Ok, but why are user stories important while estimating resources?” - you might ask. Well, I believe that they allow specifying all of the moving parts within a project, their inner connections, and the order of completion. They help to ensure that none of the aspects of a project are overlooked and every one of them is properly tested. From the customer’s perspective, those initial user stories could be helpful after the project is given to him, simply because he can recreate any of them to find out if requested features are present and are working as they should be.

But user stories are not everything we need to prepare accurate estimations for - we need to go deeper. For example, one of our use cases could be the need to log in to the app by our user. It’s pretty simple, right? But what has to be done to make this possible while making it a pleasant experience? It’s required to design the UX interface, prepare the graphic design of the site, and implement code responsible for the validation of data put by the user in the given fields (at least). So basically we have to divide all of the use cases for smaller tasks and THEN we can think about estimating one case after another. This approach allows us to make predictions that are closest to reality. Without focusing on all the smallest factors it’s possible, that we could draw our estimations right out of the hat as well.

As I said in the previous part of the cycle, we can also align ourselves by using different methods like building BPMN diagrams or creating a schema of the project. Finding the right tools shouldn’t be a problem - there are already plenty of them all over the Internet, and many more still emerging (just like GraphQL Editor mentioned in a previous post for creating GraphQL schema, documenting it etc.).

But even if we manage to prepare a lot of information and ancillary materials there is always a risk of a huge under/overestimation. It’s a human factor. That’s why it is always safer to go through initial estimation with other members of your team. Sure, the estimation of every task won’t need to be double-checked but look from other perspectives will allow us to minimize the aforementioned risk.

All of the above sound nice and easy, right? Almost ideal. It is, but it requires almost ideal conditions as well - time and budget, that could be strictly limited. For example, our client knows what he wants from his potential app, but has little to none idea what should this app contain and how to execute this. On the other hand, he seeks to apply for donation and there are just a few days left to present the cost predictions to the granter.
So what when the customer that wants to engage in a project does not have enough time or other factors are in the way of preparing complex estimation? Well, assuming that our customer has limited resources, skipping estimation altogether would be a mistake because it could lead to stretching his budget a lot in the future and not meeting the set deadlines. Instead, we could make use of our experiences with other projects and try to guide our customers with propositions based on them. A whole lot of projects have some similarities.

Let’s take the previous example of a user's login into an app - that’s pretty common so most PMs and their team members know how to handle that. The same goes for things like online payments, placing an order, etc. Using those experiences might let us focus more on unique parts of the project while not letting those other parts slow our progress down.

But it is possible to make the process of estimating simpler and shorter. It won’t be as accurate of course, but it will give us the overall picture. In this scenario, it is very important to determine as many general aspects of the project as possible by communicating with the client. It’s even more important than in our close-to-ideal scenario before. Why? Simply because we now should be focusing on the bigger parts of the project and we can assume that we will have to take care of the details. At this stage of the project, it might not be important, for example, how something will look like or how, let’s say, restoring a forgotten password would work. Based on information acquired from the client we can now prepare, you guessed it, user stories. The difference here is that we won’t be splitting those user stories into smaller tasks to estimate them individually, saving us some time. And doing that shouldn’t be that hard - a whole lot of projects have some similarities between them, just like logging into an app in the example mentioned before.

I have stated before (twice already and I can’t promise that I won’t do that in the future) that using other tools, like:

creating a schema,
BPMN diagrams,
preparing mockups.

All of above could help us to guide our project from the beginning and estimate the costs. I stand by it, but it is worth saying that using those is not essential. When the circumstances aren’t as we hope for them to be, we can skip them and go back to them at later stages. It will allow us to create a faster, but slightly less precise estimation.

“Thus, though we have heard of stupid haste in war, cleverness has never been seen associated with long delays.”

According to TAoW a country never takes advantage of prolonged warfare. It can exhaust the country's resources to a minimum and bring many to a fringe of poverty. Maybe poverty itself is not a real issue in our case, but it definitely could influence the financial state of our company. That’s why we should prevent stagnation in our project - this includes preparing estimation. Sure, the first approach that I talked about in this article will prolong the process somehow, but the key is to try our hardest to make this swift. For example, an email exchange with our customer that was supposed to bring us more light on the project could take days, taking into account that it is not our and our customer's only seizure. I know from my experience that setting up a call or meeting with a customer in person might be way more efficient than sending a bunch of questions and waiting for a response from the other side. The key here is to come prepared, so before the said meeting ask yourself a bunch of questions about the specific parts of the project. For example, if the user of an app should be able to create an account in it, then those questions could look like: “What data user is supposed to fill-in? All of that data is supposed to be editable or just only some fields?”. This will help us to stay on topic instead of conducting a meeting about every small piece or the subjects that are not that relevant (at the moment at least). Also, a lot of the previously unspoken issues can come up during such a short meeting so we could say that profit coming with them is double. If that’s not an option, then try to make your emails to be as precise as they can be and point out those aspects of a new project that need to be discussed more further. While asynchronous work is the approach to go with by default, it is worth realizing that there are some scenarios when actual talking live could be more fruitful.

So, to conclude, how exactly should we choose “strategy to adapt and tactics to adjust to”? Well, it depends. Just like a commander of an army, in the face of a new challenge, we have set goals to accomplish while making sure that those goals are achievable. That’s why we need to be fully aware of the resources to our disposal and tailor our plans according to that knowledge. By following the same approach in every situation we expose ourselves to a risk of losing otherwise winnable battles and, in the long term, even a war.

Keep in mind that's my interpretation of a second and third chapters of TAoW. I'm aware that many of you will have different thoughts about the subject and I'm really looking forward to read about them in the comments section!

How to write neovim plugins in Lua

Rafał Camlet — Wed, 05 Feb 2020 10:08:02 +0000

One of goals which neovim devs set for themselves, was making lua the first-class scripting language alternative to viml. Since version 0.4 its' interpreter along with 'stdlib' have been already built into the editor. Lua is quite simple to learn, very fast and widely used in gamedev community. In my opinion it also has much lower learning curve than viml, which may encourage new people to begin their journey with extending neovim capabilities or just making simple scripts for one time purposes. So how about we try it out? Let's write simple plugin to display the files we have recently worked on. How should we name it? Maybe... "What have I done?!". It will look something like that:

Plugin directories structure

Our plugin should have at least two directories: plugin where we put its main file and lua with whole codebase. Of course if we really want it, we can put everything in one file, but please, let's not be this guy (or gal). So plugin/whid.vim and lua/whid.lua will be fine. We can start from:

" in plugin/whid.vim
if exists('g:loaded_whid') | finish | endif " prevent loading file twice

let s:save_cpo = &cpo " save user coptions
set cpo&vim " reset them to defaults

" command to run our plugin
command! Whid lua require'whid'.whid()

let &cpo = s:save_cpo " and restore after
unlet s:save_cpo

let g:loaded_whid = 1

let s:save_cpo = &cpo is a common practice preventing custom coptions (sequence of single character flags) to interfere with the plugin. For our own purposes, the lack of this line would probably not hurt, but it is considered as good practice (at least according to the vim help files). There is also command! Whid lua require'whid'.whid() which requires plugin's lua module and calls its main function.

Floating window

Okey let's start with something fun. We should create a place where we can display things. Thankfully neovim (now vim too) has neat feature called floating windows. It's a window that is displayed over top of other windows, like in OS.

-- in lua/whid.lua

local api = vim.api
local buf, win

local function open_window()
  buf = api.nvim_create_buf(false, true) -- create new emtpy buffer

  api.nvim_buf_set_option(buf, 'bufhidden', 'wipe')

  -- get dimensions
  local width = api.nvim_get_option("columns")
  local height = api.nvim_get_option("lines")

  -- calculate our floating window size
  local win_height = math.ceil(height * 0.8 - 4)
  local win_width = math.ceil(width * 0.8)

  -- and its starting position
  local row = math.ceil((height - win_height) / 2 - 1)
  local col = math.ceil((width - win_width) / 2)

  -- set some options
  local opts = {
    style = "minimal",
    relative = "editor",
    width = win_width,
    height = win_height,
    row = row,
    col = col
  }

  -- and finally create it with buffer attached
  win = api.nvim_open_win(buf, true, opts)
end

On top of the file we define win and buf variables in the highest scope, which will be often referenced by the other function. Empty, at this moment, the buffer will be the place where we put our results. It was created as not listed buffer (first argument) and "scratch-buffer" (second argument; see :h scratch-buffer). Also we set it to be deleted when hidden bufhidden = wipe.

With nvim_open_win(buf, true, opts) we create new window with previously created buffer attached to it. Second argument makes the new window focused. width and height are pretty self explanatory. row and col are starting position of our window calculated from the upper left corner of editor relative = "editor". style = "minimal" is handy option that configures appearance of window and here we disable many unwanted options, like line numbers or highlighting of spelling errors.

So now we have floating window, but we can make it look even better. Neovim currently doesn't support widgets like border, so we should create one by ourselves. It's quite simple. We need another floating window, slightly bigger than the first one and placed under it.

local border_opts = {
  style = "minimal",
  relative = "editor",
  width = win_width + 2,
  height = win_height + 2,
  row = row - 1,
  col = col - 1
}

We will fill it with "box-drawing" characters.

local border_buf = api.nvim_create_buf(false, true)

local border_lines = { '╔' .. string.rep('═', win_width) .. '╗' }
local middle_line = '║' .. string.rep(' ', win_width) .. '║'
for i=1, win_height do
  table.insert(border_lines, middle_line)
end
table.insert(border_lines, '╚' .. string.rep('═', win_width) .. '╝')

api.nvim_buf_set_lines(border_buf, 0, -1, false, border_lines)
-- set bufer's (border_buf) lines from first line (0) to last (-1)
-- ignoring out-of-bounds error (false) with lines (border_lines)

Of course we must open windows in proper order. And one more thing. Both windows should always close together. It will be quite odd if after closing first the border is still there. Currently viml autocomand is the best solution for this.

local border_win = api.nvim_open_win(border_buf, true, border_opts)
win = api.nvim_open_win(buf, true, opts)
api.nvim_command('au BufWipeout <buffer> exe "silent bwipeout! "'..border_buf)

Get some data

Our plugin is designed to show latest files that we have worked on. We will use simple git command to do that. Something like:

git diff-tree --no-commit-id --name-only -r HEAD

Let's create function, that will put some data in our pretty window. We will call it frequently, so let's name it update_view.

local function update_view()
  -- we will use vim systemlist function which run shell
  -- command and return result as list
  local result = vim.fn.systemlist('git diff-tree --no-commit-id --name-only -r HEAD')

  -- with small indentation results will look better
  for k,v in pairs(result) do
    result[k] = '  '..result[k]
  end

  api.nvim_buf_set_lines(buf, 0, -1, false, result)
end

Hmm... it isn't very handy if we can look only for current files. Because we will directly call this function to update our view, we should accept param with information on whether we want show older or newer state.

local position = 0

local function update_view(direction)
  position = position + direction
  if position < 0 then position = 0 end -- HEAD~0 is the newest state

  local result = vim.fn.systemlist('git diff-tree --no-commit-id --name-only -r  HEAD~'..position)

  -- ... rest of the code 
end

Do you know what is still missing here? Our plugin's title. Centered! This function will help us put some text in the middle.

local function center(str)
  local width = api.nvim_win_get_width(0)
  local shift = math.floor(width / 2) - math.floor(string.len(str) / 2)
  return string.rep(' ', shift) .. str
end

  api.nvim_buf_set_lines(buf, 0, -1, false, {
      center('What have i done?'),
      center('HEAD~'..position),
      ''
    })
end

Some highlights will be nice. There are a few options we can chose: defining custom syntax file (you can match pattern based on line number) or use virtual text annotation instead of normal text (it is possible, but in that case our code would be more complicated) or... we can use position based highlighting nvim_buf_add_highlight. But first we must declare our highlights. We will link to existing default highlights group instead of setting color by ourselves. This way it will match user colorsheme.

" in plugin/whid.vim after set cpo&vim

hi def link WhidHeader      Number
hi def link WhidSubHeader   Identifier

Now let's add highlighting

api.nvim_buf_add_highlight(buf, -1, 'WhidHeader', 0, 0, -1)
api.nvim_buf_add_highlight(buf, -1, 'WhidSubHeader', 1, 0, -1)

We add highlights to buffer buf as ungrouped highlight (second argument -1). We can pass the namespace id here, which give us possibility to clear all highlights in group at once, but in our case we don't need that. Next is line number and last two params are start and end (byte-indexed) column range.

Whole function will look like this:

local function update_view(direction)
  -- Is nice to prevent user from editing interface, so
  -- we should enabled it before updating view and disabled after it.
  api.nvim_buf_set_option(buf, 'modifiable', true)

  position = position + direction
  if position < 0 then position = 0 end

  local result = vim.fn.systemlist('git diff-tree --no-commit-id --name-only -r  HEAD~'..position)
  for k,v in pairs(result) do
    result[k] = '  '..result[k]
  end

  api.nvim_buf_set_lines(buf, 0, -1, false, {
      center('What have i done?'),
      center('HEAD~'..position),
      ''
  })
  api.nvim_buf_set_lines(buf, 3, -1, false, result)

  api.nvim_buf_add_highlight(buf, -1, 'WhidHeader', 0, 0, -1)
  api.nvim_buf_add_highlight(buf, -1, 'whidSubHeader', 1, 0, -1)

  api.nvim_buf_set_option(buf, 'modifiable', false)
end

User input

Now we should make our plugin interactive. Nothing too complicated, only simple features like changing currently previewed state or selecting and opening files. Our plugin will receive user input via mappings. Pressing a key will trigger certain action. Let's look how mappings are defined in lua api.

api.nvim_buf_set_keymap(buf, 'n', 'x', ':echo "wow!"<cr>', { nowait = true, noremap = true, silent = true })

First argument, as usual, is the buffer. These mappings will be scoped to it. Next is mode short-name. We define all ours mappings in normal mode n. Then is a "left" keys combination (I choose x as example) mapped to "right" keys combination (we tell neovim to enter command-line mode, typing some viml and push enter <cr>). At last there are some options. We want neovim to trigger mapping as soon as it matches a pattern, so we set nowait flag, we prevent it from triggering our mapping via others mappings noremap and don't show typing to user silent. It is quite a long line, so we use array to save some writing.

local function set_mappings()
  local mappings = {
    ['['] = 'update_view(-1)',
    [']'] = 'update_view(1)',
    ['<cr>'] = 'open_file()',
    h = 'update_view(-1)',
    l = 'update_view(1)',
    q = 'close_window()',
    k = 'move_cursor()'
  }

  for k,v in pairs(mappings) do
    api.nvim_buf_set_keymap(buf, 'n', k, ':lua require"whid".'..v..'<cr>', {
        nowait = true, noremap = true, silent = true
      })
  end
end

We can also disable not used keys (or not, whichever you like).

local other_chars = {
  'a', 'b', 'c', 'd', 'e', 'f', 'g', 'i', 'n', 'o', 'p', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'
}
for k,v in ipairs(other_chars) do
  api.nvim_buf_set_keymap(buf, 'n', v, '', { nowait = true, noremap = true, silent = true })
  api.nvim_buf_set_keymap(buf, 'n', v:upper(), '', { nowait = true, noremap = true, silent = true })
  api.nvim_buf_set_keymap(buf, 'n',  '<c-'..v..'>', '', { nowait = true, noremap = true, silent = true })
end

Public function

Okey, but there are some new functions mentioned in the mapping. Let's look at them.

local function close_window()
  api.nvim_win_close(win, true)
end

-- Our file list start at line 4, so we can prevent reaching above it
-- from bottm the end of the buffer will limit movment
local function move_cursor()
  local new_pos = math.max(4, api.nvim_win_get_cursor(win)[1] - 1)
  api.nvim_win_set_cursor(win, {new_pos, 0})
end

-- Open file under cursor
local function open_file()
  local str = api.nvim_get_current_line()
  close_window()
  api.nvim_command('edit '..str)
end

Our file list is quite simple. We can just get line under the cursor and tell neovim to edit it. Of course we can built more sophisticated mechanism. We can get line number (or even column) and then, based on it, trigger specific action. It will allow to separate view form logic. But for our purposes it is enough.

However non of these functions can be called if we don't export them first. At the bottom of the file we will return associative array with publicly available functions.

return {
  whid = whid,
  update_view = update_view,
  open_file = open_file,
  move_cursor = move_cursor,
  close_window = close_window
}

And of course the main function!

local function whid()
  position = 0 -- if you want to preserve last displayed state just omit this line
  open_window()
  set_mappings()
  update_view(0)
  api.nvim_win_set_cursor(win, {4, 0}) -- set cursor on first list entry
end

The whole plugin...

... with small improvments (you can find them by looking for the comments).

" plugin/whid.vim
if exists('g:loaded_whid') | finish | endif

let s:save_cpo = &cpo
set cpo&vim

hi def link WhidHeader      Number
hi def link WhidSubHeader   Identifier

command! Whid lua require'whid'.whid()

let &cpo = s:save_cpo
unlet s:save_cpo

let g:loaded_whid = 1

-- lua/whid.lua
local api = vim.api
local buf, win
local position = 0

local function center(str)
  local width = api.nvim_win_get_width(0)
  local shift = math.floor(width / 2) - math.floor(string.len(str) / 2)
  return string.rep(' ', shift) .. str
end

local function open_window()
  buf = api.nvim_create_buf(false, true)
  local border_buf = api.nvim_create_buf(false, true)

  api.nvim_buf_set_option(buf, 'bufhidden', 'wipe')
  api.nvim_buf_set_option(buf, 'filetype', 'whid')

  local width = api.nvim_get_option("columns")
  local height = api.nvim_get_option("lines")

  local win_height = math.ceil(height * 0.8 - 4)
  local win_width = math.ceil(width * 0.8)
  local row = math.ceil((height - win_height) / 2 - 1)
  local col = math.ceil((width - win_width) / 2)

  local border_opts = {
    style = "minimal",
    relative = "editor",
    width = win_width + 2,
    height = win_height + 2,
    row = row - 1,
    col = col - 1
  }

  local opts = {
    style = "minimal",
    relative = "editor",
    width = win_width,
    height = win_height,
    row = row,
    col = col
  }

  local border_lines = { '╔' .. string.rep('═', win_width) .. '╗' }
  local middle_line = '║' .. string.rep(' ', win_width) .. '║'
  for i=1, win_height do
    table.insert(border_lines, middle_line)
  end
  table.insert(border_lines, '╚' .. string.rep('═', win_width) .. '╝')
  api.nvim_buf_set_lines(border_buf, 0, -1, false, border_lines)

  local border_win = api.nvim_open_win(border_buf, true, border_opts)
  win = api.nvim_open_win(buf, true, opts)
  api.nvim_command('au BufWipeout <buffer> exe "silent bwipeout! "'..border_buf)

  api.nvim_win_set_option(win, 'cursorline', true) -- it highlight line with the cursor on it

  -- we can add title already here, because first line will never change
  api.nvim_buf_set_lines(buf, 0, -1, false, { center('What have i done?'), '', ''})
  api.nvim_buf_add_highlight(buf, -1, 'WhidHeader', 0, 0, -1)
end

local function update_view(direction)
  api.nvim_buf_set_option(buf, 'modifiable', true)
  position = position + direction
  if position < 0 then position = 0 end

  local result = vim.fn.systemlist('git diff-tree --no-commit-id --name-only -r  HEAD~'..position)
  if #result == 0 then table.insert(result, '') end -- add  an empty line to preserve layout if there is no results
  for k,v in pairs(result) do
    result[k] = '  '..result[k]
  end

  api.nvim_buf_set_lines(buf, 1, 2, false, {center('HEAD~'..position)})
  api.nvim_buf_set_lines(buf, 3, -1, false, result)

  api.nvim_buf_add_highlight(buf, -1, 'whidSubHeader', 1, 0, -1)
  api.nvim_buf_set_option(buf, 'modifiable', false)
end

local function close_window()
  api.nvim_win_close(win, true)
end

local function open_file()
  local str = api.nvim_get_current_line()
  close_window()
  api.nvim_command('edit '..str)
end

local function move_cursor()
  local new_pos = math.max(4, api.nvim_win_get_cursor(win)[1] - 1)
  api.nvim_win_set_cursor(win, {new_pos, 0})
end

local function set_mappings()
  local mappings = {
    ['['] = 'update_view(-1)',
    [']'] = 'update_view(1)',
    ['<cr>'] = 'open_file()',
    h = 'update_view(-1)',
    l = 'update_view(1)',
    q = 'close_window()',
    k = 'move_cursor()'
  }

  for k,v in pairs(mappings) do
    api.nvim_buf_set_keymap(buf, 'n', k, ':lua require"whid".'..v..'<cr>', {
        nowait = true, noremap = true, silent = true
      })
  end
  local other_chars = {
    'a', 'b', 'c', 'd', 'e', 'f', 'g', 'i', 'n', 'o', 'p', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'
  }
  for k,v in ipairs(other_chars) do
    api.nvim_buf_set_keymap(buf, 'n', v, '', { nowait = true, noremap = true, silent = true })
    api.nvim_buf_set_keymap(buf, 'n', v:upper(), '', { nowait = true, noremap = true, silent = true })
    api.nvim_buf_set_keymap(buf, 'n',  '<c-'..v..'>', '', { nowait = true, noremap = true, silent = true })
  end
end

local function whid()
  position = 0
  open_window()
  set_mappings()
  update_view(0)
  api.nvim_win_set_cursor(win, {4, 0})
end

return {
  whid = whid,
  update_view = update_view,
  open_file = open_file,
  move_cursor = move_cursor,
  close_window = close_window
}

Now you should have a basic knowledge, enough to write simple TUI for your lua neovim scripts. Have fun!
The code can be found also here https://github.com/rafcamlet/nvim-whid