Forem: Fernando Karchiloff

How to debug Django inside a Docker container with VSCode

Fernando Karchiloff — Fri, 26 Jan 2024 16:46:23 +0000

“If you want a thing done well, do it yourself.” ― Napoleon Bonaparte

This tutorial is valid for these platforms:

Linux ✅
Windows ❌
OSX ❌

Changes will be indicated along with a changelog.

Table Of Contents
1. Requirements
2. Motivation
3. How to set up for this example
- 3.1. Clone the project and have some requirements ready
- 3.2. Enable virtual environment
- 3.3. Add debugpy
- 3.4. Install the packages for definitions
- 3.5. Build the project for Docker
- 3.6. Create the debug file
- 3.7. Create the launch.json file to allow VSCode to attach to our application
- 3.8. Create a test view
4. Debugging your code
- 4.1. Execution of docker compose up with debug
- 4.2. Attach debugger to application
- 4.3. Request the URL and debug your code
5. Debugging library code
- 5.1. Changing launch.json to work with library code
- 5.2. Configuring variables for library code debug
- 5.3. Debugging the library code
6. Conclusion
7. References
- 7.1 Docker Engine and Docker Compose
- 7.2 Pyenv
- 7.3 Poetry
- 7.4 Git
- 7.5 Project
- 7.6 Debugpy
- 7.7 VSCode

1. Requirements

Docker Engine
Docker Compose v2.23.3 (v2.24.1 has a bug when using two docker-compose.yml files that override)
Python version manager pyenv
Python >= 3.8 (for this example, 3.11.x is used) through pyenv
Package manager Poetry v1.4.2
VSCode editor at least v1.85
Python package debugpy >= 1.8
For this demonstration, this project will be used: https://github.com/wiamsuri/django-gunicorn-nginx-docker

2. Motivation

Many tutorials available online don't acknowledge the problems of debbuging Django applicants with Docker, usually they explain basic functioning and sometimes are just as vague on how it works. My intention is explaining how to set up a debugger for Django applications that use Docker, Poetry and VSCode, correctly.

One main problem that I constantly see is the debugpy module always being installed when executing the container for debugging, which consumes bandwidth and time. It should be installed as a development dependency for the project, so you can always use it when needed.

3. How to set up for this example

If you already have an project configured with pyenv, Poetry and Docker. Skip steps 1, 2 and 8.

3.1. Clone the project and have some requirements ready

Install the pyenv Python version manager, there are various ways to do it here. My personal choice is the automatic installer. Also check the shell set up, it will enable your environment to always have pyenv easily from your terminal.

After that, install the latest Python version available of 3.11, do it using the following:



pyenv install 3.11

As soon as you installed the specific Python version, install the Poetry package manager. I recommend using the official installer, once done, proceed to clone the repository.

Clone the project using Git:



git clone https://github.com/wiamsuri/django-gunicorn-nginx-docker

You will need to specify the Python version to 3.11.x at the root of the project:



pyenv local 3.11.<hotfix_version_installed>

At this point, I'm assuming you also have the Docker Engine and Docker Compose V2 installed.

3.2. Enable virtual environment

Enable the virtual environment. If you don't have one, it will create for you with the command below:



poetry shell

3.3. Add debugpy

debugpy description from it's repository:

An implementation of the Debug Adapter Protocol for Python 3.

It is a module maintained by Microsoft in order to debug Python programs, allowing connection to a debugger.

If the project does not have the debugpy module, add it with the command below:



poetry add --group=dev debugpy

Done this way, debugpy will always be available at development and you won't consume bandwidth and lose your time waiting for the debugger module to install everytime.

3.4. Install the packages for definitions

The command below will install all the packages needed for the project so VSCode can identify the packages in the source code. This step is needed to allow definitions for library's to be available:



poetry install

3.5. Build the project for Docker

Our project example has a README.md file which explains how to build the project, each project has it's own configuration and explanations on how to build them. You can read it here or use the following commands sequentially:



cp django.env.example django.env



docker compose build

3.6. Create the debug file

This file is essential to allow debugging, as it will reuse all configurations defined in other files and only changes what is needed to accomplish our task.

At the same level of docker-compose.yml, create a new file called docker-compose.debug.yml.
It should have the service you need to debug, as also the ports and command.
It must have the same port used for the application and debug port defined.

File example:



version: "3.7"

services:
  app:
    ports:
      - "8000:8000"
      - "5678:5678"
    command: ["sh", "-c", "python3 -m debugpy --listen 0.0.0.0:5678 manage.py runserver 0.0.0.0:8000"]

As shown in the file, it's calling the debugpy module and listening for connections at the port 5678, but it isn't waiting the connection of the debugger, it will run the application just as normal. If you need to wait for it to connect, you might need the flag --wait-for-client, then the application will only start when the debugger is connected. Check more information in the debugpy module documentation.

3.7. Create the launch.json file to allow VSCode to attach to our application

In case you don't have a launch.json file, create one. VSCode offers a template as default.

Click on create a launch.json file.

You might need to select the debugger, choose Python.
Then choose the debug configuration, go for Remote Attach.
It will ask configurations for the remote debugging, stick with the default, or change to your needs.

The result will be something like this:



{
  // Use IntelliSense to learn about possible attributes.
  // Hover to view descriptions of existing attributes.
  // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Python: Remote Attach",
      "type": "python",
      "request": "attach",
      "connect": {
        "host": "localhost",
        "port": 5678
      },
      "pathMappings": [
        {
          "localRoot": "${workspaceFolder}",
          "remoteRoot": "."
        }
      ],
      "justMyCode": true
    }
  ]
}

Change the property justMyCode to false, so that the debugger can have access to code from the library that your code might call to see it's behavior.

3.8. Create a test view

As the project is just an example, it has a basic template for running Django with Docker. We have to create a view to test our debugger and check if it's running correctly.

At the urls.py file, in the config/ folder, create a hello_view function and add it to the urlpatterns list.



from django.contrib import admin
from django.http import HttpResponse
from django.urls import path

def hello_view(request):
    if request.method == "GET":
        return HttpResponse("Hello, world!")

urlpatterns = [
    path('admin/', admin.site.urls),
    path('hello/', hello_view)
]

With this function, the debugger will be able to stop the execution of our code and check what's happening inside the program.

4. Debugging your code

4.1. Execution of docker compose up with debug

If everything was done correctly, we can start our application with the debugging command. In order for this event to happen, specify the debug file in the command. Do it as follows:



docker compose -f docker-compose.yml -f docker-compose.debug.yml up

The Docker Compose will overlap the files and use the debug file specified. As it's using a debug file, a base file must be determined for the services. As you may have other docker-compose files, just ensure the docker-compose.debug.yml is the last one, so the program can overlap the files and get their configurations correctly.

4.2. Attach debugger to application

With the application running, select the "Run and Debug" tab at VSCode, it will look like this if the launch.json file is present:

Click the play button indicated as Python: Remote Attach. In normal scenarios, it will connect the debugger to your application and a toolbar will appear on screen.

This means the debugger is working now, let's test it.

4.3. Request the URL and debug your code

Open your browser of choice and type localhost:8000/hello/, a call to the test view will be made. The result should be a "Hello, world!":

Notice that the debugger hasn't triggered yet, that's because we didn't specified any breakpoints in the code.

Set up a breakpoint on line 22 (L22) by clicking on the left side of the line number:

Now we refresh the page and you will see the magic happening.

Our code execution stopped at the breakpoint we've just specified, you will see variables, call stack and even watch for specific values or expressions. You can learn more about the debugger screen in the VSCode documentation.

That's the basic to debug your code and see how it's behaving. However, this configuration does not allow to make breakpoints inside libraries that you might have in your project.

The next section will explain what's needed to do in order to debug libraries directly from inside your project.

5. Debugging library code

Sometimes you're not interested in debugging your code. Instead, you want to understand the library logic or behavior in order to fix your code, or the library itself.

In this scenario, we need breakpoints inside the library code. However, by default, the configuration created above (which solves most problems) isn't enough to allow debugging library code.

Under these circumstances, you will get this grey empty dot:

In order to solve this, changes are needed at the launch.json, environment variables in .bashrc and settings.json to make it work.

5.1. Changing `launch.json` to work with library code

To make the debugger aware that the library code is available, some modifications are needed.

To accomplish this, changes to launch.json file must be done, so that the debugger can seek for the path were the library code is. By adding another object at pathMappings, we make the code available to the debugger.



{
  // Use IntelliSense to learn about possible attributes.
  // Hover to view descriptions of existing attributes.
  // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Python: Remote Attach",
      "type": "python",
      "request": "attach",
      "connect": {
        "host": "localhost",
        "port": 5678
      },
      "pathMappings": [
        {
          "localRoot": "${workspaceFolder}",
          "remoteRoot": "."
        },
        {
          "localRoot": "${env:PACKAGES_ENV_PATH}/${config:python.poetry.project.venv}/lib/python3.11/site-packages/",
          "remoteRoot": "/usr/local/lib/python3.11/site-packages/"
        }
      ],
      "justMyCode": false
    }
  ]
}

With this configuration provided for the debugger, which contains the library code path, it will seek for the files and allow to set a breakpoint.

Although, I have to explain what is that object and how you can configure it.

5.2. Configuring variables for library code debug

${env:PACKAGES_ENV_PATH} is an environment variable defined in .bashrc (or similar) that tells us where to look for the libraries source code. This path can be obtained by using:



poetry env info --path

The result will be similiar to:



/home/youruser/.cache/pypoetry/virtualenvs/django-gunicorn-nginx-docker-ucLVPu3k-py3.11

Until virtualenvs/, the path leads to all virtual environments that your Poetry creates. After it, is the specific virtualenv for the project.

The environment variable VSCode needs to read, must be defined at .bashrc (or similars) as:



#VSCode debug
export PACKAGES_ENV_PATH="/home/youruser/.cache/pypoetry/virtualenvs"

After adding this piece of code to .bashrc, VSCode must be restarted to allow the process to get the new defined env.
This process has to be done due to how VSCode is opened and how processes work, for more info, see here.

${config:python.poetry.project.venv} is a user defined variable inside settings.json which tells us which virtualenv the debugger must look for.

The last part of the command mentioned above can be used to fulfill this part. If the project doesn't have a settings.json file, just create one under .vscode/ folder, in case you have it, define the new variable to use:



{
  "python.poetry.project.venv": "django-gunicorn-nginx-docker-ucLVPu3k-py3.11"
}

I tried to came up with a key that won't affect the VSCode ecosystem or any extensions (that I've heard of).

This was the best way I've found that allowed to easily define the path in order to share the launch.json file with other coworkers through source control software.

Observation: you can also hardcode the path to make your life easier, .

The Python version won't change frequently, so I fixed it in the configuration.

In this context, I'm assuming that settings.json will never be commited to the source control, and the user can change as he desires. If that's not the case, I would recommend to hardcode the path in the localRoot, but every developer path will be different, be aware. DO NOT FORGET to never include this change to your commits.

If you know about a better way to do this configuration for library source code debugging, let me know in the comments or reach out to me.

5.3. Debugging the library code

Commands are the same from the Debugging your code section, the only difference is where the breakpoint is indicated.

For the example, I'd choosen the WSGIHandler class as it's purpose is receiving the requests and serving them.

I set the breakpoint at line 123 (L123). If the dot is red, it means the debugger can use the breakpoint properly. Then I refreshed the page for our test view, and this is the result:

From there, you can happily (or not) debug the code.

6. Conclusion

Debugging is not a easy task, but with this tutorial I expect that somehow I made your life easier. So you can achieve your objective and/or catch that bug that is annoying you.

Feel free to comment and warn me about any mistakes that this article may have.

Thanks for reading until here, Ferka out ✌️.

7. References

7.1 Docker Engine and Docker Compose

https://docs.docker.com/engine/install/
https://docs.docker.com/compose/install/linux/#install-the-plugin-manually (be aware of the v2.24.1 bug with overlap compose files)

7.2 Pyenv

7.3 Poetry

7.4 Git

https://git-scm.com/

7.5 Project

https://github.com/wiamsuri/django-gunicorn-nginx-docker#for-development

7.6 Debugpy

7.7 VSCode

How to avoid messing up squash migration in Hasura

Fernando Karchiloff — Thu, 05 Jan 2023 20:54:27 +0000

Versão em Português Brasileiro

Summary

Technology versions
Background knowledge needed
Why I decided to write it
Too long; didn't read
Database example
Squash guide
Conclusion

Technology versions

Hasura v2.0.9
Hasura CLI v2.0.9
PostgreSQL 12

Background knowledge needed

Understand SQL and it's commands
Understand how migrations work
How Hasura GraphQL Engine works
How Git works

Why I decided to write it

During the development of one project at the e-commerce company I'd worked for, we had a task that would require to change the database structure, many tables and columns had to be changed to complete the task.

As Hasura GraphQL Engine offers a web page to make the structural changes at the database, I made the changes using this interface and executed the manual queries that were necessary. All of these modifications created migrations in the repository, as expected for a project with migrations enabled in Hasura.

After the modifications, it was noticeable the amount of folders with modifications that were being saved in the migration format. I decided to use the command to compress all of them into a single file, which would be more readable. That's when my mistake happened: without making basic validations beforehand, I executed the command and deleted the previous migrations. That's how I spent more than a day trying to fix a mistake that could have been avoided with a simple checklist before using the command. How can we avoid this? I will explain in the next topics.

Too long; didn't read

Make your changes using migrations.
Using Git or similar, make a commit to save your migrations.
Make sure that the up.sql and down.sql files execute as desired using hasura migrate apply and hasura migrate apply --down <N>. Do manual fixes (mistakes can happen) using the SQL tab without using migrations.
Commit your complete migrations to the version control system.
Use the command hasura migrate squash --from <num_migration> --name <name> to execute the squash, delete the old migrations (you already have them at the version control).
Mark the created migration as fulfilled at the database using hasura migrate apply --skip-execution (if the removed migrations weren't applied, don't use the flag).
Use again the command hasura migrate apply --down <N> to remove the migrations, and then use it without the flag to make sure that they're working.
Commit your changes to the version control system.
(Optional) Do optimizations if needed, for example: remove an ALTER TABLE if there's a DROP TABLE (without losing data of course).

Database example

To demonstrate how to do a squash correctly, I will use a simple database with a single table, but in a real case you can have various operations among N tables, which may lead to a larger and complex squash.

Our database contains the students table defined through this structure:

CREATE TABLE "public"."students" (
    "id" serial NOT NULL,
    "name" Text NOT NULL,
    "score" text NOT NULL,
    PRIMARY KEY ("id") 
);

It was created after we had already connected to our database created in PostgreSQL, disabled the console, and started migrations in the project. You can see more details on how to enable migrations in this section of Migrations & Metadata (CI/CD) in the documentation for the Hasura GraphQL Engine.

Let's say that this table was filled with some student data:

Somewhere along the way, we realize that the data in the score column, which cannot be text because we would like to do some arithmetic with them. However, we cannot change this data if it is being used in production! (Assume that you have thousands of important data and cannot lose them)

What to do then? In the development environment, we will create a new column to accommodate this data. We will name it score_decimal because it represents the score as a decimal number allowing arithmetic operations. Go to the students table's Modify tab and click Add new column. We will define it as a Numeric column.

Once the new column is created, we can migrate the data from the score column to score_decimal. To do this, we will use the SQL section located just below the tables in our database.

In this tab you can run SQL code directly against the selected database, marking these queries as migrations or not. Depending on the code, Hasura may be smart enough to identify if it is a structural or data change and mark it as a migration automatically, so always pay attention before running the query to see if the "This is a migration" box is enabled.

Putting the query that converts the scores in the text field, we can run it without adding it as a migration in order to verify that the query is being executed as described. After checking, we can enable the migration box and write a name for it, I used a descriptive name cast_students_score_to_decimal_migration. This is the query made to convert the data:

UPDATE students
SET score_decimal=CAST(score AS DECIMAL)
WHERE students.id = students.id;

NOTE: always be careful when making structural changes that change field categories, so do it in steps and update the view layer (front-end), if used, to change the new type without breaking the application for clients.

After running the update of the data, we can rename the columns and delete the old one. The name used for the migration was rename_columns_and_remove_old_migration.

ALTER TABLE students RENAME COLUMN "score" TO "score_text";
ALTER TABLE students RENAME COLUMN "score_decimal" TO "score";
ALTER TABLE students DROP COLUMN "score_text";

With that, we finished our modification to run our arithmetic operations, but we ended up with some separate migrations that can be joined through a squash operation, as they can be executed all together.

Squash guide

For the squash process of the migrations to occur "correctly" (there can always be an unexpected error in the middle of the process), we need to ensure that some steps are followed. I will list some of them to avoid errors in this process.

Save the made migrations in the repository
Ensure that the migrations can be made and undone without errors
Save the complete migrations
Squash the migrations
Mark the migration as executed
Save the changes
(Optional) Modify the compressed migration as desired (and save the changes)

1. Save the made migrations in the repository

To keep track of our migrations and not lose progress, we save the changes in a commit, we can then proceed to change them without the risk of losing what we have done, this was one of the mistakes I made earlier, not saving in the version control.

2. Ensure that the migrations can be made and undone without errors

With the migrations saved in the repository, we can start modifying those that have incomplete down.sql files. Many of the changes we make to the database do not always reflect in the files due to being complex changes, such as renaming columns and deleting a column in the same migration. For example, in the last migration, the down.sql file was filled out as follows:

-- Could not auto-generate a down migration.
-- Please write an appropriate down migration for the SQL below:
-- ALTER TABLE students RENAME COLUMN "score" TO "score_text";
-- ALTER TABLE students RENAME COLUMN "score_decimal" TO "score";
-- ALTER TABLE students DROP COLUMN "score_text";

For the migration to work back, that is, remove the changes, we will reverse the commands made in up.sql.

ALTER TABLE students ADD COLUMN "score_text" text;
ALTER TABLE students RENAME COLUMN "score" TO "score_decimal";
ALTER TABLE students RENAME COLUMN "score_text" TO "score";

We will do the same with the migration that creates the new column and reverse the command.

ALTER TABLE "public"."students" DROP COLUMN "score_decimal";

With that, we can use the --up and --down flags of hasura migrate apply. The command changes the database according to the .sql defined in the migrations. If you relate the files to their respective commands, it will make more sense, the command that uses --down will remove the applied migration changes, while --up will apply them (--up is applied by default if not defined). To remove only some, use:

hasura migrate apply --down <N>

In the command, N is the number of migrations that will be removed. Use hasura migrate apply --help to see more details.

Note: If you ran a migration and it failed, if you have more than one SQL command in the same file, try to fix it through the SQL tab (without creating a new one), correct it and try again.

3. Save the complete migrations

Since you have tested your migrations and they are working, save them again in a commit, now with the complete migrations.

4. Squash the migrations

With all migrations completed and working, we can squash them, that is, compress them into a single migration. To do this, we will run the command:

hasura migrate squash --from <num_migracao> --name <name>

It compress the migration files for us. In this example, I will do like this:

hasura migrate squash --from 1638852925395 --name "changing_column_type"

This command will merge all migrations starting from the prefix 1638852925395 into a single one, named "changing_column_type". I recommend you delete the previous migrations because they are already saved in the repository.

5. Mark the migration as executed

Once we have created our new compressed migration as a result of the operation, if the previous migrations were not applied to the database, we will run the command to mark it as if the migration we just generated had already been applied:

hasura migrate apply --skip-execution

Remember that this command should only be run if we actually applied the changes of the other migrations to the database, otherwise, run the command without the --skip-execution flag to apply it normally.

6. Save the changes

Once this is done, save the changes again in the repository's version control to avoid losing your changes.

7. (Optional) Modify the compressed migration as desired (and save the changes)

If your migration has many commands, evaluate the file and see if, depending on the changes made, you can simplify the executed SQL code. For example, you create a table, and then in another migration you add a column, if you have not added any data in the middle of the way, or it has not gone to production yet, directly add the column when creating the table, which will simplify the code.

Conclusion

I hope that these instructions serve as a guide so that you don't make the same mistakes I did during the project I was developing. I tried to put as many details as possible in the step-by-step instructions and the reasons behind them.

These instructions can be replicated in other projects, even if they don't follow the same technology, but if their concepts are similar, it's possible. If you don't apply them immediately, at least I hope you've learned something from what I demonstrated in this article.

Any positive criticism that can enhance this article is welcome! Share with other people who need this content!

Thank you for your attention.

Reduzindo as chances de errar um squash de migrações no Hasura

Fernando Karchiloff — Wed, 15 Dec 2021 15:55:59 +0000

Sumário

Versão das tecnologias
Requisitos mínimos de conhecimento para entender o artigo
Por quê deste artigo
Para os apressados
Exemplo de base de dados
Processo de fazer um squash
Conclusão

Versão das tecnologias

Hasura v2.0.9
Hasura CLI v2.0.9
PostgreSQL 12

Requisitos mínimos de conhecimento para entender o artigo

Entender de comandos SQL
Como migrações funcionam
Como Hasura GraphQL Engine funciona
Como Git funciona

Por quê deste artigo

Durante o desenvolvimento de um dos projetos de uma empresa de e-commerce, tivemos uma tarefa que seria necessário mudar a estrutura do banco de dados, esta alteração mexeria com diversas tabelas e colunas para que o objetivo da tarefa fosse concluído.

Como o Hasura GraphQL Engine oferece uma interface para fazer as mudanças na estrutura do banco de dados, modifiquei por meio desta e executei as queries manuais que se mostraram necessárias, todas essas modificações criavam migrações no repositório do projeto, assim como esperado para um projeto com migrações habilitadas no Hasura.

Após as modificações, era notável a quantidade de pastas com modificações que foram sendo gravadas no formato de migrações, decidi então usar o comando para comprimir todas elas em um único arquivo, o que seria mais legível, e minha falha se concretizou no momento que executei o comando.

Sem ter feito validações básicas antes, executei o comando e excluí as migrações anteriores. E assim passei mais de um dia tentando corrigir o erro que cometi por não fazer uma lista de verificação básica antes de usar o comando. Como evitar isso? Vou te explicar no decorrer do artigo.

Para os apressados

Faça as suas modificações usando migrações.
Usando Git ou outro versionador, faça um commit para salvar as migrações.
Garanta que os arquivos de up.sql e down.sql executam como deveriam usando hasura migrate apply e hasura migrate apply --down <N>. Correções manuais (caso erros aconteçam), faça pela aba SQL sem usar migrações.
Salve as migrações completas no versionador.
Use o comando hasura migrate squash --from <num_migracao> --name <name> para executar o squash, remova as migrações anteriores (pois você já salvou no versionador).
Marque a migração que foi criada como preenchida no banco, usando hasura migrate apply --skip-execution (se as que serão removidas não haviam sido aplicadas, não use a flag).
Use novamente o hasura migrate apply --down <N> para retirar, e sem a flag para colocar as migrações, garantindo que sua migração funciona.
Grave as alterações no versionador.
(Opcional) Faça otimizações de comandos SQL se achar necessário, exemplo: remover ALTER TABLE caso exista um DROP TABLE para uma tabela (sem perda de dados).

Exemplo de base de dados

Para demonstrar como fazer um squash corretamente, irei utilizar uma base de dados de exemplo simples com uma única tabela, mas no caso real você pode ter diversas operações entre N tabelas, o que pode tornar o squash maior e mais complexo.

A nossa base contém a tabela students definida por meio desta estrutura:

CREATE TABLE "public"."students" (
    "id" serial NOT NULL,
    "name" Text NOT NULL,
    "score" text NOT NULL,
    PRIMARY KEY ("id") 
);

Ela foi criada depois que já executamos a conexão com a nossa base de dados criada no PostgreSQL, desativamos o console e iniciamos as migrações no projeto. Você pode ver mais detalhes de como ativar migrações nesta seção de Migrations & Metadata (CI/CD) na documentação do Hasura GraphQL Engine.

Digamos então que esta tabela foi preenchida com alguns dados de estudantes:

No meio do caminho, percebemos que os dados das notas localizados na coluna score, não poderiam ser textos, pois gostaríamos de fazer algumas aritméticas com eles. Entretanto, não podemos mudar estes dados caso estejam em uso no ambiente de produção! (Suponha que você tenha milhares de dados importantes e não pode perdê-los)

O que fazer então? No ambiente de desenvolvimento, iremos criar uma coluna para acomodar esses dados, vamos nomeá-la score_decimal, pois representa a nota em número decimal permitindo operações aritméticas. Vá até à aba Modify da tabela students e clique em Add new column. Vamos definir como uma coluna Numeric.

Uma vez criada a nova coluna, podemos fazer a migração dos dados da coluna score para score_decimal. Para isso ser possível, vamos usar a seção de SQL que fica logo abaixo das tabelas da nossa base de dados.

Nessa aba você pode rodar código SQL diretamente contra o banco de dados selecionado, podendo marcar estas consultas como migrações ou não. Dependendo do código, o Hasura talvez seja inteligente para identificar ser uma alteração estrutural ou de dados e marque como migração automaticamente, então sempre preste atenção antes de rodar a consulta para ver se a caixa de "This is a migration" está habilitada.

Colocando a consulta que converte as notas no campo de texto, podemos rodar sem adicionar como migração com intenção de verificar se a consulta está sendo executada como descrita. Após a verificação, podemos ativar a caixa de migração e escrever um nome para ela, usei um nome descritivo cast_students_score_to_decimal_migration.
Esta é a consulta feita para converter os dados:

UPDATE students
SET score_decimal=CAST(score AS DECIMAL)
WHERE students.id = students.id;

OBSERVAÇÃO: sempre tome cuidado para fazer alterações estruturais que mudam categorias de campos, então faça por partes e vá atualizando a camada de visualização (front-end), caso sejam utilizadas, para mudar o novo tipo sem quebrar a aplicação para os clientes.

Após rodar a atualização dos dados, podemos renomear as colunas e deletar a antiga. O nome usado para migração foi rename_columns_and_remove_old_migration.

ALTER TABLE students RENAME COLUMN "score" TO "score_text";
ALTER TABLE students RENAME COLUMN "score_decimal" TO "score";
ALTER TABLE students DROP COLUMN "score_text";

Com isso, finalizamos nossa modificação para executarmos nossas operações aritméticas, mas acabamos com algumas migrações separadas que podem ser unidas através de uma operação de squash, pois podem ser executadas todas juntas.

Processo de fazer um squash

Para que o processo de squash das migrações ocorra "corretamente" (sempre pode ocorrer algum erro inesperado no meio do processo), precisamos garantir que algumas etapas sejam seguidas. Vou listar algumas delas para evitar erros desse processo.

Gravar as migrações feitas no repositório
Garantir que as migrações podem ser feitas e desfeitas sem erros
Gravar as migrações completas
Fazer o squash das migrações
Marcar a migração como executada
Grave as alterações
(Opcional) Modifique a migração comprimida como desejar (e grave as alterações)

1. Gravar as migrações feitas

Para mantermos controle das nossas migrações e não perdermos o progresso feito, gravamos as alterações em um commit do versionador, podemos então prosseguir em alterá-las sem risco de perder o que já fizemos, esse foi um dos erros que cometi anteriormente, não ter salvo no versionador.

2. Garanta que as migrações podem ser feitas e desfeitas sem erros

Com as migrações salvas no repositório, podemos começar a modificação das que possuem arquivos down.sql incompletos. Muitas das alterações que executamos no banco de dados, nem sempre refletem nos arquivos por serem modificações complexas, como a renomeação das colunas e deleção de coluna na mesma migração. Por exemplo, a última migração, o arquivo down.sql foi preenchido desta forma:

-- Could not auto-generate a down migration.
-- Please write an appropriate down migration for the SQL below:
-- ALTER TABLE students RENAME COLUMN "score" TO "score_text";
-- ALTER TABLE students RENAME COLUMN "score_decimal" TO "score";
-- ALTER TABLE students DROP COLUMN "score_text";

Para que a migração funcione na volta, ou seja, remover as mudanças, vamos inverter os comandos feitos no up.sql.

ALTER TABLE students ADD COLUMN "score_text" text;
ALTER TABLE students RENAME COLUMN "score" TO "score_decimal";
ALTER TABLE students RENAME COLUMN "score_text" TO "score";

Faremos o mesmo com a migração que cria a nova coluna e inverteremos o comando.

ALTER TABLE "public"."students" DROP COLUMN "score_decimal";

Com isso, podemos usar as flags de --up e --down do hasura migrate apply. O comando altera o banco de dados conforme os .sql definidos nas migrações. Se você relacionar os arquivos aos seus respectivos comandos, fará mais sentido, o comando que usa --down vai remover as alterações de migrações já aplicadas, enquanto o --up irá aplicá-las (o --up é aplicado por padrão se não for definido). Para remover somente algumas, use:

hasura migrate apply --down <N>

No comando, N é a quantidade de migrações que serão removidas. Use hasura migrate apply --help para mais detalhes.

Observação: se você executou alguma migração e ela falhou, caso tenha mais de um comando SQL no mesmo arquivo, tente corrigir pela aba SQL (sem fazer uma nova), corrigi-la e tentar novamente.

3. Grave as migrações completas

Dado que você testou suas migrações, e elas estão funcionando, grave novamente em um commit, agora com as migrações completas.

4. Fazer o squash das migrações

Com todas as migrações completas e funcionando ida e volta, podemos agora fazer o squash delas, ou seja, comprimi-las em uma única só migração, para isso, vamos rodar o comando:

hasura migrate squash --from <num_migracao> --name <name>

Ele executa a compressão dos arquivos das migrações para nós. No meu caso, farei dessa forma:

hasura migrate squash --from 1638852925395 --name "changing_column_type"

Este comando irá a partir da migração de prefixo 1638852925395 juntar todas em uma única, de nome "changing_column_type". Recomendo você deletar as migrações anteriores pois já estão salvas no repositório.

5. Marcar a migração como executada

Assim que criamos a nossa nova migração comprimida como resultado da operação, caso as migrações anteriores não estivessem aplicadas no banco de dados, vamos rodar o comando para marcar como se a migração que acabamos de gerar, já tivesse sido aplicada:

hasura migrate apply --skip-execution

Lembrando que este comando só deve ser executado se efetivamente aplicamos as alterações das outras migrações no banco de dados, caso contrário, rode o comando sem a flag --skip-execution para aplicar normalmente.

6. Grave as alterações

Feito isso, grave novamente as alterações na versão de controle do repositório para não perder suas alterações.

7. (Opcional) Modifique a migração comprimida como desejar (e grave as alterações)

Se a sua migração possui muitos comandos, avalie o arquivo e verifique se, dependendo das alterações feitas, você pode simplificar o código SQL executado. Por exemplo, você cria uma tabela, e depois em outra migração adiciona uma coluna, caso você não tenha adicionado nenhum dado no meio do caminho, ou não foi para produção ainda, coloque diretamente a coluna na criação da tabela, o que simplificará o código.

Conclusão

Espero que estas instruções sirvam de guia para que não cometa os mesmos erros que eu durante o projeto que estava desenvolvendo. Tentei colocar o máximo de detalhes no passo a passo e seus motivos.

Essas instruções podem ser replicadas em outros projetos, mesmo que não sigam a mesma tecnologia, mas se seus conceitos forem parecidos, é possível. Caso você não aplique imediatamente, ao menos espero que tenha aprendido algo com o que demonstrei neste artigo.

Qualquer crítica positiva que possa incrementar este artigo é bem-vinda! Compartilhe com outras pessoas que precisam deste conteúdo!

Obrigado por sua atenção.