Forem: Kaio Marx

Engenharia de Dados: uma introdução

Kaio Marx — Tue, 30 May 2023 18:24:41 +0000

O mundo da tecnologia sempre surpreende com alguma coisa nova que você particularmente não sabia que existia, mas hoje não sabe como sobreviveu tantos pull requests sem (obrigado CI/CD, obrigado!). Com Engenharia de Dados não é diferente, não é uma tecnologia em si, mas sim uma posição pouco conhecida a qual tem impacto gigante até o presente momento da indústria, com promessas de continuar sendo uma das principais carreiras para o futuro mundo repleto de IA que nos aguarda.

Primeiramente, vale sempre a definição, já que é uma posição pouco conhecida (duvido que você saiba as atribuições só pelo nome), eu particularmente entendo engenharia de dados como o DBA do novo século. Alguns autores do setor são mais precisos do que eu, segundo alguns deles o engenheiro de dados é aquele profissional encarregado de garantir o dado correto, no formato certo, para pessoa certa, da forma mais eficiente possível.

Dessa forma, ao pensar na estrutura das empresas atuais, podemos perceber que o data engineer é praticamente o precursor da cultura de dados da empresa INTEIRA, end-to-end. Isso porque tanto o time de devs, quanto de machine learning, quanto de BI (business intelligence), vai acabar recorrendo a este profissional quando precisar acessar ou apenas entender melhor a infraestrutura de dados da empresa.

O engenheiro de dados é especialmente importante para o time de machine learning, pelo fato da consciência difundida que esse pessoal tem sobre o conceito de "garbage in, garbage out”, eles praticamente já têm trabalho suficiente apenas se preocupando com o modelo de machine learning “o que”, “como”, mas “quais os dados” é uma resposta que só o engenheiro de dados pode responder com propriedade.

Mas você pode pensar “nossa criaram uma carreira para cuidar dos dados só para treinar inteligência artificial direito”, essa é uma visão reducionista sobre engenheira de dados, na verdade a posição existe desde o primeiro computador que conseguiu fazer armazenamento (provavelmente em torno de 1815) e veio evoluindo junto da indústria, hoje as principais atribuições de um data engineer serão a respeito do “pipeline” da empresa.

Nesse momento, chegamos em uma buzzword, “pipeline” na tradução direta significa cano, resumindo o conceito então vemos que sempre existiu o encanador da empresa, a pessoa que cuida do fluxo que os dados fazem dentro dos canos, mais do que nunca essa pessoa utiliza do advento da cloud, batch processing, stream processing, para entregar o melhor dado possível, na maior velocidade que faz sentido para empresa.

Portanto, as principais ferramentas que compõem o dia a dia da engenharia de dados são aquelas que auxiliam na construção e manutenção do fluxo de dados de toda a organização. Temos como exemplo para a construção dessa primeira infraestrutura o Apache Airflow. Para o processamento e tratamento, pode ser necessário o uso de processamento distribuído, entrando então em cena o Apache Spark. Como é uma posição multidisciplinar que envolve engenharia de software, temos o uso intensivo de SQL e Python, que são as principais linguagens de programação. Juntas, elas representam quase 50% de todas as ferramentas utilizadas.

Pattern for Git commits

Kaio Marx — Tue, 15 Mar 2022 12:36:11 +0000

It's boring to write commit, but even more boring is not knowing where that change happened.

Many of you may have already uploaded at least one commit to a github repository, but as not every junior is committed to open source to take inspiration as they do, maybe you've made commits without any pattern in the message. totally normal.

In my case I also do this from time to time, in the projects I want to commit quickly. Now in a project with several devs it's much easier by the title of your commit to know what it's about, standardizing this first interaction with the new code then becomes a must-have.

Is there an industry standard?
Look, each team chooses how they do it, there are people who use cute icons, there are people who go through the previous projects they worked on, but I use and recommend ConventionalCommits.org, based on that structure you're ready to send code to the whole world.

The structure is very simple, as follows:

[optional scope]: description

Just looking at this cake recipe may seem abstract to some, so let's put it in real examples:

Commit with new feature.

feat: "Add payment methods to xyzUseCase"

Commit with bug fix.
I've never seen a developer not have doubts about this one, because in our minds we thought it should be like this "fix: "fix the something..." But the proper way would be:

fix: "Correct the foo class method"

Documentation commit.

docs: (changes to the documentation)

Another interesting examples.

style: (formatting, missing semi colons, etc; no production code change)
refactor: (refactoring production code, eg. renaming a variable)
test: (adding missing tests, refactoring tests; no production code change)
chore: (updating grunt tasks etc; no production code change)

That's it for today, I hope you adopt these more organized commits, after automating this routine in your development process, it becomes easier to organize your day-to-day.

How to organize typescript projects with prisma.io

Kaio Marx — Mon, 28 Feb 2022 18:37:17 +0000

First things, first!

The idea here is to show a point of view on the code architecture and improve the format by your feedback in the comments, so feel free to demonstrate your way of orchestrating projects with Node/Typescript.

Let's Understand

Some tools used in typescript development can declare an opinion about your code, others like prism leave this responsibility of organization to you. I prefer the prism.

With the prism we have a configuration optimization because most of the prism does it for you, but this ORM doesn't organize its connections by "Entities". This implies that your connection client will bring all your tables in a single class, at this moment it is important to divide it with layers of responsibility, the famous repositories.

What are Repositories?

Repositories are classes or components that encapsulate the logic required to access data sources. They centralize common data access functionality, providing better maintainability and decoupling the infrastructure or technology used to access databases from the domain model layer

To demonstrate this abstraction of code construction, let's analyze this diagram that I use in the development routine:

.
├── src
│   ├── config
│   │   └── prisma.ts
│   ├── modules
│   │   └── domain_name
│   │       ├── dtos
│   │       ├── infra
│   │       │   ├── repository
│   │       │   └── IdomainRepository.ts
│   │       └── useCases
│   │           └── -subDomain
│   │               ├── -SubDomainController.ts
│   │               └── -SubDomainUseCase.ts

For all class files or library configuration we have the configs directories, in projects with prisma it is appropriate to create this file "dbClient.ts" to instantiate a single connection to the database through the client class (PrismaClient), as you can see in the following code:

import { PrismaClient } from '@prisma/client';

export const prisma = new PrismaClient();

Domains

"The domain is the world of the business you are working with and the problems they want to solve. This will typically involve rules, processes and existing systems that need to be integrated as part of your solution. The domain is the ideas, knowledge and data of the problem you are trying to solve."

So imagine an ecommerce where we have a database with tables named "Products", "Customers", "Users". our domains will be respectively these tables, everything that involves a "product" will be in the Products directory, being this product update, deletion, creation, queries.

Domains have a very simple organization when implemented with typescript.

Dtos: refers to the acronym Data Transport Object, it is the folder where interfaces will be created and maintained, they are generally used to define the types of parameters passed in the "useCases" functions.

export interface ICreateClientDTO {
  username: string;
  password: string;
}

The "I" starting the filename and the class is just a customization convention by eslint rules, to identify Interfaces.

infra/Repository: as explained earlier, here will be the classes that take care of the connection with the database, these classes must have connectivity directly related to the domain, to maintain a division of responsibilities implement contracts in your class, dictating what methods it must show for the useCases.

import { Clients, PrismaClient } from "@prisma/client"
import { prisma } from "../../../../config/prisma"
import { ICreateClientDTO } from "../../dtos/ICreateClientDTO"
import { IClientRepository } from "../IClientRepository"

class ClientRepository implements IClientRepository {
  private repository: PrismaClient

  constructor() {
    this.repository = prisma
  }

  async findOneByUsername(username: string): Promise<Clients> {}

  async create({ username, password }: ICreateClientDTO): Promise<void> {}

  async findAll(): Promise<Clients[]> {
    //Your code
  }
}

export { ClientRepository }

infra/IdomainRepository: These interfaces serve to define custom methods serving as contracts that our repository class must follow. This ensures that even if one day the connection implementation changes it will serve the application without releasing side effects in the code.

import { Clients } from "@prisma/client";
import { ICreateClientDTO } from "../dtos/ICreateClientDTO";

export interface IClientRepository {
  findAll(): Promise<Clients[]>
  findOneByUsername(username: string): Promise<Clients>
  create({ username, password }: ICreateClientDTO): Promise<void>
}

UseCases

the use cases will have the subdomains, subdomain can be classified as the action to be performed, for example in the Domain "Products" we will have a logical subdomain "createProducts". the structure of a subdomain does not change and is composed of Controller and useCase, this generates a request manager (Controller) and a manager of the business rule and validations (useCase).

Controller:
 import { Request, Response } from "express";
import { container } from "tsyringe";
import { CreateClientsUseCase } from "./CreateClientsUseCase";

class CreateClientsController {
  async handle(req: Request, res: Response): Promise<Response> {
    const { username, password } = req.body;

    const createClientsUseCase = container.resolve(CreateClientsUseCase)

    await createClientsUseCase.execute({
      username,
      password
    })

    return res.status(201).send()
  }
}

export { CreateClientsController }

import { PrismaClient } from "@prisma/client";
import { hash } from "bcrypt";
import { inject, injectable } from "tsyringe";
import { AppError } from "../../../../shared/errors/AppError";
import { ICreateClientDTO } from "../../dtos/ICreateClientDTO";
import { IClientRepository } from "../../infra/IClientRepository";

@injectable()
class CreateClientsUseCase {
  constructor(
    @inject('ClientsRepository') private clientRepository: IClientRepository
  ) {}

  async execute({ username, password }: ICreateClientDTO) {
    const userAlreadyExists = await this.clientRepository.findOneByUsername(
      username
    )

    if (userAlreadyExists) {
      throw new AppError("User Already Exists")
    }

    const encryptedPassword = await hash(password, 10)

    await this.clientRepository.create({
      username,
      password: encryptedPassword
    })
  }
}

export { CreateClientsUseCase }

Considering that architecture is much more theoretical and comprehensive than the simplification made here, the code and structure presented above serves as a guide to make it easier to organize. In future posts I will put these ideas into practical examples applied to real problems.

If you want to ask about any topic of the code, feel free to contact me through the information on my profile or on my twitter.