Forem: Aluisio Amorim C.

O que é IA Generativa? Um Glossário de Termos Essenciais

Aluisio Amorim C. — Tue, 19 Nov 2024 14:27:33 +0000

Nos últimos anos, a Inteligência Artificial Generativa tem se tornado um dos avanços tecnológicos mais comentados. Capaz de criar texto, imagens, áudio e até vídeos de forma autônoma, essa tecnologia impacta as mais diversas indústrias. Para compreender como a IA generativa funciona, é essencial conhecer os principais conceitos que a impulsionam.

LLM (Large Language Model)

LLM, ou Large Language Model, refere-se a um modelo de linguagem de grande escala, treinado em vastos conjuntos de dados textuais. Esses modelos, como GPT, Claude ou Llama, são compostos de bilhões ou até trilhões de parâmetros, o que lhes confere uma capacidade impressionante de gerar texto coerente, traduzir linguagens, responder a perguntas complexas e muito mais. O tamanho do modelo é um fator crucial para sua eficácia, pois modelos maiores tendem a capturar mais nuances da linguagem e gerar saídas mais sofisticadas.

Atualmente, quando falamos de IA generativa, usualmente estamos nos referindo ao poder de geração de texto desses LLMs, que são fundação para aplicações como o Chat GPT.

O LLM é o Transformer altamente treinado, que compreende e gera séries de tokens sofisticadas.

Transformer

O Transformer é a arquitetura de rede neural que revolucionou o campo do processamento de linguagem natural (NLP). Introduzido no artigo “Attention is All You Need” de 2017, ele é a base para os grandes modelos de linguagem, como GPT, BERT e outros. Ao contrário de arquiteturas anteriores, como redes recorrentes (RNNs), o Transformer utiliza o mecanismo de atenção, que o permite capturar dependências de longo prazo, ou seja, considerar o conteúdo relevante de qualquer parte do texto. Sobretudo, esta arquitetura permite processamento paralelo, o que torna possível treinar modelos tão grandes em um “curto” período.

Token

No contexto de LLMs, um token é uma unidade básica de texto. Pode ser uma palavra, parte de uma palavra ou até mesmo um símbolo, dependendo de como o modelo é treinado. A geração de texto pelo modelo ocorre token por token, com a previsão de qual será o próximo baseada nos anteriores.

Embedding

Um embedding, no contexto de processamento de linguagem natural, é a representação numérica do token em um espaço vetorial, em outras palavras, um embedding é um vetor. O que significa dizer que os embeddings são responsáveis por capturar relações sintáticas e semânticas entre os tokens. Por exemplo, os embeddings das palavras “rei” e “príncipe” estão mais próximos do que os embeddings de “rei” e “Python”, permitindo assim, a geração de texto contextualizado e coeso.

Inferência

Na IA generativa, a inferência refere-se ao processo de utilizar um modelo treinado para prever o próximo token em uma sequência de texto. Durante a inferência, o modelo analisa o contexto fornecido pelos tokens anteriores e gera previsões baseadas nos padrões aprendidos durante o treinamento. A qualidade dessa inferência determina a fluidez, coerência e relevância do texto gerado. Para tarefas como escrita automática ou diálogos, essa capacidade inferencial é o núcleo do funcionamento dos modelos de linguagem.

Janela de Contexto (Context Window)

A janela de contexto é a quantidade de texto ou tokens que um modelo de linguagem pode processar de uma vez. A janela de contexto define o número de tokens que podem ser considerados simultaneamente para gerar uma resposta. Se o contexto for muito longo e exceder o limite da janela, o modelo não poderá levar em conta a totalidade da informação passada. A escolha de uma janela de contexto adequada é essencial para garantir que o modelo tenha informações suficientes para tomar decisões precisas.

Habilidades Emergentes

As habilidades emergentes referem-se a comportamentos ou capacidades inesperadas que os modelos de IA exibem à medida que se tornam maiores e mais complexos. Essas habilidades não foram explicitamente programadas, mas emergem naturalmente durante o processo de treinamento em grandes volumes de dados. Um exemplo é a capacidade de resolver problemas matemáticos complexos ou gerar código em linguagens de programação específicas. Esses fenômenos demonstram o poder dos LLMs para aprender padrões além das intenções dos desenvolvedores.

Treinamento

O treinamento de um modelo envolve ajustar seus parâmetros por meio da exposição a grandes quantidades de dados. Para modelos de linguagem generativa, o treinamento consiste em alimentar o modelo com vastos conjuntos de dados de texto, permitindo que ele aprenda padrões e contextos. Durante o processo, o modelo passa por muitas iterações, ajustando seus pesos e viéses com base em erros anteriores, para melhorar sua capacidade de prever o próximo token em uma sequência de texto.

Fine-tuning

O fine-tuning é o processo de ajustar um modelo já treinado para tarefas específicas. Por exemplo, após o treinamento geral em um vasto conjunto de dados, um modelo pode ser refinado para tarefas como geração de e-mails ou atendimento ao cliente em um domínio específico. O fine-tuning permite que um modelo genérico seja especializado para contextos mais específicos, sem precisar começar o treinamento do zero.

Prompts

Um prompt é a entrada inicial fornecida a um modelo de linguagem generativa para gerar uma resposta. É a “semente” que desencadeia o processo de geração de texto. O design do prompt é crucial para obter resultados satisfatórios. Dependendo de como o prompt é estruturado, o modelo pode produzir respostas muito diferentes.

Viés (Bias)

No contexto dos modelos de IA, o termo bias refere-se a um parâmetro que influencia a forma como o modelo aprende e generaliza. Esse tipo de bias, na matemática das redes neurais, ajuda a ajustar o modelo para que ele possa capturar melhor os padrões nos dados de treinamento e alcançar maior precisão nas previsões. Essencialmente, ele atua como uma constante que permite ao modelo se adaptar mais facilmente a um conjunto de dados, tornando a aprendizagem mais eficiente e flexível. Esse tipo de parâmetro bias não está diretamente relacionado a influências culturais ou sociais, mas ao ajuste de pesos para otimizar a capacidade do modelo em realizar tarefas preditivas com menor erro.

No entanto, o termo bias também refere-se ao viés presente nos dados e, por consequência, nas respostas geradas pelo modelo. Esse viés surge dos padrões contidos nos dados de treinamento, que podem refletir preconceitos e estereótipos humanos. Quando exposto a dados enviesados, um modelo de IA pode gerar saídas que perpetuam ou até amplificam preconceitos, resultando em textos racistas, sexistas ou de outra forma prejudiciais. Isso ocorre porque o modelo aprende com grandes quantidades de dados reais, que podem carregar vieses culturais, históricos e sociais. Combater esses vieses exige um trabalho cuidadoso de seleção e tratamento dos dados, além de ajustes no próprio processo de treinamento para minimizar outputs indesejáveis.

Alucinação (Hallucination)

No contexto de modelos de IA generativa, alucinação refere-se ao fenômeno em que o modelo gera informações falsas, incoerentes ou não fundamentadas nos dados fornecidos. Isso ocorre porque os LLMs não têm acesso direto a fatos ou à realidade durante o processo de geração; eles simplesmente seguem padrões aprendidos no treinamento para prever o próximo token.

Por exemplo, um modelo pode inventar nomes de livros, criar citações inexistentes ou afirmar informações factualmente incorretas como verdadeiras. Alucinações podem ser problemáticas, especialmente em aplicações críticas como saúde, direito, educação, ou seguro onde a precisão é essencial.

Minimizar alucinações exige melhorias contínuas nos processos de treinamento, maior supervisão humana e, em muitos casos, a integração com sistemas que validem os outputs gerados pelo modelo. O design de prompts claros e específicos também ajuda a mitigar esse comportamento.

Guard Rails

Guard rails referem-se a mecanismos implementados para limitar o comportamento dos modelos de IA generativa, garantindo que suas respostas sejam seguras, éticas e alinhadas aos objetivos pretendidos. Esses mecanismos são projetados para prevenir o uso indevido, reduzir outputs prejudiciais, e proteger contra alucinações ou vieses problemáticos.

Os guard rails podem ser implementados em várias formas, incluindo:

Filtros de conteúdo: Bloqueiam respostas inadequadas, como discurso de ódio ou informações perigosas.
Validação de saídas: Integração com sistemas externos para verificar fatos ou assegurar a precisão de respostas.
Limitação de contexto: Restringe o escopo do modelo para tópicos relevantes à aplicação.
Instruções específicas no treinamento: Ajustam os dados de treinamento e o fine-tuning para moldar o comportamento do modelo.

Um exemplo comum de guard rail é a configuração de um modelo para evitar responder perguntas fora de um domínio específico, como o contexto de seguros que estamos inseridos.

"Os limites de minha linguagem são os limites de meu mundo."

— Ludwig Wittgenstein

Dominar o vocabulário técnico da área prepara o terreno para estudos mais aprofundados.

O intuito desse texto é se tornar um glossário do time da bem-te-vi, compartilhado com a comunidade. Sinta-se a vontade para sugerir adições e edições.

Referência

Bouchard, Louis-Francois, and Louie Peters. Building LLMs for Production. Towards AI, 2024.

Implementing end-to-end encryption (E2EE) to a Planning Poker game

Aluisio Amorim C. — Tue, 02 Apr 2024 14:12:27 +0000

Planning Poker® is a consensus-based technique for agile estimating. It is a fun and engaging way for teams to apply relative estimates to planned work. In this blog post, we're diving into how we can enhance the security of Planning Poker sessions using end-to-end encryption (E2EE).

The code of the Qwikens Free Planning Poker application is open-source. 🚀🥳

Take a look at the repository:

https://github.com/qwikens/planning-poker

What is End-to-End Encryption?

End-to-End Encryption (E2EE) is a method of secure communication that prevents third-parties from accessing data while it's transferred from one end system or device to another. In E2EE, the data is encrypted on the sender's system or device and only the recipient is able to decrypt it. Nobody in between, be it internet service providers, hackers, or even the platform provider itself, can read it or tamper with it.

The principle behind E2EE is that it uses cryptographic keys to encrypt and decrypt the data. Only the communicating users possess the keys to unlock and read the messages. This ensures that the data remains confidential and integral, providing a secure channel for communication over potentially insecure networks.

Examples of E2EE in Use

One of the most well-known examples of E2EE in use today is WhatsApp. WhatsApp encrypts messages in a way that even WhatsApp servers cannot decrypt them. This means that when you send a message, photo, video, or file, everything is encrypted automatically, and only the recipient can decrypt and view the message.

Other notable examples include:

Signal: A messaging app that is widely regarded for its state-of-the-art end-to-end encryption for voice calls, video calls, and instant messaging.
Telegram: Offers optional end-to-end encrypted secret chats that are not stored on their servers.
ProtonMail: An email service that secures emails with end-to-end encryption, meaning that even ProtonMail cannot access the content of your emails.

These examples highlight the growing importance and implementation of E2EE in various forms of digital communication, ensuring privacy and security for users worldwide.

Why E2EE?

Our infrastructure already provides a secure channel for communication (TLS/SSL), so why do we need E2EE?

While TLS/SSL secures the communication between the client and the server, it does not protect the data once it reaches the server. This means that the data is decrypted on the server and can potentially be accessed by the service provider or malicious actors who gain access to the server.

More information about TLS/SSL:
https://www.cloudflare.com/learning/ssl/transport-layer-security-tls/

We are worried about the Planning Poker issues being exposed to the Qwikens's infrastructure, we want to ensure that the issues remains confidential even when it's stored on the server. This is where E2EE comes into play. By encrypting the data on the client-side before it's sent to the server, we can ensure that only the intended recipient can decrypt and access the data.

E2EE in Planning Poker

This is not a trivial task, since we are working with a real-time collaborative tool, we need to ensure that the encryption and decryption process does not introduce significant latency or complexity to the user experience.

The first point to consider is that we are dealing with a group of participants. So, it is not enough to encrypt the data with the recipient's public key, as this would only allow the recipient to decrypt the data.

We need to encrypt the data with the public keys of all participants, so that all participants can decrypt the data.

The second point to consider is that we are dealing with a real-time application. This means that we need to ensure that the encryption and decryption process is fast enough to keep up with the pace of the planning poker game. We need to carefully consider the encryption algorithm and key size to ensure that the process is efficient and secure.

Based on these constraints, we had to analyze some approaches.

Peer-to-peer encryption

Each participant encrypts the data with the public keys of all participants and sends the encrypted data to all participants. This approach is not scalable, as the number of messages grows quadratically with the number of participants.

In the context of Planning Poker, where the number of participants is relatively small, this approach could be feasible. However, it introduces additional complexity and overhead, as each participant needs to encrypt the data multiple times and create multiple issues. This could lead to synchronization issues and potential performance problems.

Shared key

All participants share a symmetric key that is used to encrypt and decrypt the data. This approach is simpler and more efficient, as it avoids the need to encrypt the data multiple times and create multiple encrypted issues.

When creating a room, the client generates a key and shares it with all participants.

Although this approach is simpler and more efficient, since the data is encrypted with a single key, there are two main drawbacks:

Single point of failure: If the key is compromised, all data is exposed. This means that the security of the system relies on the secrecy of the key;
Key management: Since all participants share the same key, there is a risk that the key could be lost or compromised. This could lead to data loss or exposure. Also, the key is a huge string, so it is not easy to share.
- A possible solution is to share the key within the URL, as a query parameter, but this results in a very long URL, which is not ideal, since it could be truncated by some services (like Google Meet, for example).

Hybrid approach

What about a hybrid approach? 🤔

The idea is to combine the benefits of both approaches:

Do not need to share keys "manually";
Do not need to maintain multiple encrypted issues state.

The idea is to use a symmetric key to encrypt the data, but the key is encrypted with the public keys of all participants. This way, only the participants can decrypt the key and access the data.

When creating a room, the user generates a pair of keys (public and private) and shares the public key;

/**
 * Generates a RSA key pair using a specified bit length.
 * @returns An object containing the PEM encoded public and private keys.
 * @example
 * const keyPair = generateKeyPair();
 * console.log(keyPair.publicKey); // PEM encoded public key
 * console.log(keyPair.privateKey); // PEM encoded private key
 */
export const generateKeyPair = async (): Promise<{
  publicKey: string;
  privateKey: string;
}> =>
  new Promise((resolve, reject) => {
    forge.pki.rsa.generateKeyPair({ bits: 2048 }, (err, keys) => {
      if (err) {
        return reject(err);
      }

      return resolve({
        publicKey: forge.pki.publicKeyToPem(keys.publicKey),
        privateKey: forge.pki.privateKeyToPem(keys.privateKey),
      });
    });
  });

A key point here is to generate the key pair inside a Promise. Generating the key pair synchronously could block the JS event loop's main thread, causing the application to freeze. By using a Promise, we ensure that the key pair is generated asynchronously, outside the main thread, and the application remains responsive.

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Event_loop#never_blocking

For the sake of simplicity, we are using a 2048-bit RSA key pair. This is a common key size for RSA encryption and provides a good balance between security and performance. However, the key size can be adjusted based on the desired level of security and performance.

The user also generates a symmetric key and encrypts it with its own public key;

/**
 * Generates a symmetric key using a specified bit length.
 * @param {number} [bitLength=256] - The bit length for the symmetric key. Default is 256 bits.
 * @returns {string} The generated symmetric key, base64 encoded.
 * @example
 * const symmetricKey = generateSymmetricKey();
 * console.log(symmetricKey); // Base64 encoded symmetric key
 */
export const generateSymmetricKey = (bitLength = 256) => {
  const bytes = bitLength / 8;
  const key = forge.random.getBytesSync(bytes);
  return forge.util.encode64(key);
};

When any participant enters the room, the new participant generates a pair of keys and shares the public key;
Then, any of the current participants encrypts the symmetric key with the new participant's public key and sends it to the server;

/**
 * Encrypts a message using a given public key.
 * @param message - The plaintext message to be encrypted.
 * @param publicKey - The PEM encoded public key used for encryption.
 * @returns The encrypted message, base64 encoded.
 * @example
 * const encryptedMessage = asymmetricEncrypt("Hello, world!", publicKey);
 * console.log(encryptedMessage); // Encrypted and base64 encoded message
 */
export const asymmetricEncrypt = (message: string, publicKey: string) => {
  const publicKeyRsa = forge.pki.publicKeyFromPem(publicKey);
  const encrypted = publicKeyRsa.encrypt(forge.util.encodeUtf8(message));
  return forge.util.encode64(encrypted);
};

With the encrypted symmetric key in hands, the new participant can decrypt it with its private key and access the data;

/**
 * Decrypts an encrypted message using a given private key.
 * @param encryptedMessage - The encrypted message, base64 encoded.
 * @param privateKey - The PEM encoded private key used for decryption.
 * @returns The decrypted plaintext message.
 * @example
 * const decryptedMessage = asymmetricDecrypt(encryptedMessage, privateKey);
 * console.log(decryptedMessage); // Decrypted plaintext message
 */
export const asymmetricDecrypt = (
  encryptedMessage: string,
  privateKey: string
) => {
  const privateKeyObj = forge.pki.privateKeyFromPem(privateKey);
  const encrypted = forge.util.decode64(encryptedMessage);
  const decrypted = privateKeyObj.decrypt(encrypted);
  return forge.util.decodeUtf8(decrypted);
};

The application needs to handle two states: issues and decryptedIssues;

type Issue = {
  id: string;
  storyPoints?: number;
  createdAt: number;
  createdBy: string;
  title: string;
};

const issuesState = proxy<Issue[]>([]);
const decryptedIssuesState = proxy<Issue[]>([]);

Each issue is encrypted with the symmetric key before being sent to the server;

/**
 * Encrypts a message using a symmetric key.
 * @param message - The plaintext message to be encrypted.
 * @param symmetricKey - The base64 encoded symmetric key used for encryption.
 * @returns The encrypted message, base64 encoded.
 * @example
 * const encryptedMessage = symmetricEncrypt("Hello, world!", symmetricKey);
 * console.log(encryptedMessage); // Encrypted and base64 encoded message
 */
export const symmetricEncrypt = (message: string, symmetricKey: string) => {
  const key = forge.util.decode64(symmetricKey);
  const iv = forge.random.getBytesSync(16); // Initialization vector
  const cipher = forge.cipher.createCipher("AES-CBC", key);
  cipher.start({ iv: iv });
  cipher.update(forge.util.createBuffer(forge.util.encodeUtf8(message)));
  cipher.finish();
  const encrypted = cipher.output.getBytes();
  const encryptedMessageWithIv = `${iv}${encrypted}`;
  return forge.util.encode64(encryptedMessageWithIv);
};

When the issues state is synchronized, the client decrypts the issues with the symmetric key and updates the decryptedIssues state.

/**
 * Decrypts an encrypted message using a symmetric key.
 * @param encryptedMessage - The encrypted message, base64 encoded.
 * @param symmetricKey - The base64 encoded symmetric key used for decryption.
 * @returns The decrypted plaintext message.
 * @example
 * const decryptedMessage = symmetricDecrypt(encryptedMessage, symmetricKey);
 * console.log(decryptedMessage); // Decrypted plaintext message
 */
export const symmetricDecrypt = (
  encryptedMessage: string,
  symmetricKey: string
) => {
  const key = forge.util.decode64(symmetricKey);
  const encryptedBytesWithIv = forge.util.decode64(encryptedMessage);
  const iv = encryptedBytesWithIv.slice(0, 16);
  const encryptedBytes = encryptedBytesWithIv.slice(16);
  const decipher = forge.cipher.createDecipher("AES-CBC", key);
  decipher.start({ iv: iv });
  decipher.update(forge.util.createBuffer(encryptedBytes));
  decipher.finish();
  return forge.util.decodeUtf8(decipher.output.getBytes());
};

Diffie–Hellman key exchange

Diffie–Hellman key exchange is a mathematical method of securely exchanging cryptographic keys over a public channel and was one of the first public-key protocols.
https://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exchange

The Diffie–Hellman key exchange is a method for securely exchanging cryptographic keys over a public channel. It allows two parties to agree on a shared secret key without exchanging the key directly. This is achieved by each party generating a public-private key pair and exchanging public keys. The shared secret key is then derived from the combination of the private key and the other party's public key.

In the context of Planning Poker, the Diffie–Hellman key exchange could be used to establish a shared symmetric key between all participants. This would eliminate the need for a single point of failure.

The problem with this approach is that it does not solve the problem of N * M issues being created, where N is the number of players and M is the number of issues.
The secret key now is the combination of pairs of keys, so the application needs to handle a issuesStatefor each pair of keys.

Benefits of E2EE in Planning Poker

Privacy: Ensures that issues remain confidential among the participants.
Security: Protects against potential eavesdropping or data breaches.
Trust: Builds trust among team members, knowing that their issues are secure.

Next steps and Considerations

Room IDs: A UUID is secure enough to be the room identifier?
Key size: What is the ideal key size for the symmetric key and the RSA key pair?
Performance: How to ensure that the encryption and decryption process is fast enough to keep up with the pace of the Planning Poker game?
RSA vs. ECC: Should we consider using Elliptic Curve Cryptography (ECC) instead of RSA for key generation? ECC is known for its shorter key lengths and faster performance compared to RSA. RSA is more widely used and supported, but the key-pair generation and encryption/decryption process can be really slow.

Conclusion

In this article, we have explored the concept of end-to-end encryption (E2EE) and its potential application in Planning Poker. We have discussed the importance of E2EE in ensuring privacy, security, and trust among team members.

We have also examined the challenges of implementing E2EE in a real-time collaborative tool like Planning Poker, such as the need to encrypt data with the public keys of all participants and the need to ensure that the encryption and decryption process is fast enough to keep up with the pace of the game.

We have proposed a hybrid approach that combines the benefits of both symmetric and asymmetric encryption. This approach involves using a symmetric key to encrypt the data, but the key is encrypted with the public keys of all participants. This ensures that only the participants can decrypt the key and access the data, eliminating the need for a single point of failure and reducing the complexity of key management.

However, there are still many considerations and potential challenges to address, such as the choice of key size, the performance of the encryption and decryption process, and the choice between RSA and ECC for key generation.

In conclusion, while E2EE can significantly enhance the security of Planning Poker, its implementation requires careful consideration and planning. It is not a one-size-fits-all solution, but rather a tool that can be tailored to meet the specific needs and constraints of each application.

How to Successfully Integrate with Legacy APIs Using NodeJS

Aluisio Amorim C. — Mon, 11 Dec 2023 18:20:08 +0000

Hello, I'm Aluisio, a software engineer with a rich history of integrating with Brazilian banks, insurance companies, and ERPs.

In this article, I'll share my experience and guide you through the process of seamless integration. For instructional purposes, I'll exaggerate partner inefficiencies, but it's crucial to tailor these strategies to your unique context.

Document Everything

When embarking on the integration journey, the initial step is to request documentation. However, the reality is that not every company excels at documentation.

Uncertainty looms over the structure of responses.
The configuration of the POST body remains a mystery.
Required parameters are shrouded in ambiguity.
The meanings of various fields are unclear.

The solution? Write your documentation!

Begin by selecting your preferred platform for API development (e.g., Postman, Insomnia), then create a collection and commence drafting your requests.

If feasible, consider importing the requests provided by your partner.

Create a Dedicated Section for Your Documentation

Establish a dedicated section for your documentation, whether it's a Wiki within your repository, a 'docs' folder in your monorepo, a page on Notion, or any other suitable platform.

This documentation should encompass the following key components:

Docs: Include links to the partner's documentation. This serves as a quick reference for your team to access essential information.

Contacts: Documenting available contacts is crucial. Specify people's names and their respective positions. This not only provides clarity on the support network but also ensures that in the absence of the designated integration contact, another developer can seamlessly take over.

Working Requests: Compile a comprehensive list of all endpoints that are functional and yield valuable data for your business. Each entry in this list should include a detailed description (as interpreted by your team) of the endpoint, its required parameters, and a breakdown of each response field. For any unknown fields, mark them with a 'TODO,' and ensure that the relevant partner contacts are activated for swift resolution.

This structured documentation will significantly enhance your integration process, fostering clarity and efficiency within your team.

Write an RFC (Request for Comments)

Consider initiating the integration process by drafting a Request for Comments (RFC). Refer to the concept of RFC for guidance. Emphasize the importance of maintaining clean and well-documented code to facilitate comprehension and collaboration among developers working on the integration.

Conduct a thorough evaluation of existing integrations within your team's codebase. Identify commonalities, potential areas for abstraction, and opportunities for optimization such as concurrency and batch processing. Advocate for asynchronous and efficient communication, providing practical implementation suggestions along with code examples.

Once your team aligns on the integration strategy, proceed to the coding phase.

Mock the API

In my experience, achieving a quality delivery is inseparable from rigorous testing. Begin by creating test scenarios, especially when integrating with complex systems like ERPs.

Consider a hypothetical scenario where data from a list of companies within an ERP needs to be retrieved. As a personal recommendation, leverage tools like MSW for top-level mocks, which can significantly enhance the testing process.

Prioritizing testing not only ensures the reliability of your integration but also streamlines the development workflow, ultimately leading to a more robust and maintainable codebase.

So, firstly, setup your handlers:

const server = setupServer(
  // Describe network behavior with request handlers.
  // Tip: move the handlers into their own module and
  // import it across your browser and Node.js setups!
  http.get('/companies', ({ request, params, cookies }) => {
    return HttpResponse.json([
      {
        id: 'f8dd058f-9006-4174-8d49-e3086bc39c21',
        tradeName: `My Test Company`,
        employees: 10,
      },
      {
        id: '8ac96078-6434-4959-80ed-cc834e7fef61',
        tradeName: `My Test Company 2`,
        employees: 30,
      },
    ]);
  }),
);

// Enable request interception.
beforeAll(() => server.listen())

// Reset handlers so that each test could alter them
// without affecting other, unrelated tests.
afterEach(() => server.resetHandlers())

// Don't forget to clean up afterwards.
afterAll(() => server.close())

Then, create your test:

it('gets companies', async () => {
    // Your function to fetch the endpoint data (it can use fetch,
    // axios, etc).
    const companies = await getCompanies();

    expect(companies).toEqual([
      {
        id: 'f8dd058f-9006-4174-8d49-e3086bc39c21',
        tradeName: 'My Test Company',
        employees: 10,
      },
      {
        id: '8ac96078-6434-4959-80ed-cc834e7fef61',
        tradeName: 'My Test Company 2',
        employees: 30,
      },
    ]);
});

Type the API

After creating the test cases, the next crucial step is to define the types for your API endpoints. I highly recommend leveraging Zod💫 for its exceptional type safety and developer experience (DX).

An innovative approach is to integrate GPT into this process. Here's how:

Having integrated with endpoints featuring over 80 fields in the response, the time-saving potential of GPT becomes truly apparent.

Response:

Wonderful, GPT saved us a lot of time, right?

One remarkable advantage of using Zod schemas is that they are not strict. In cases where certain fields, like foo, are not pertinent, Zod allows for their effortless removal. This flexibility ensures that your schema definitions remain concise and directly aligned with your specific needs.

// Zod schema without `foo`
export const getCompaniesResponseSchema = z.array(
  z.object({
    id: z.string().uuid(), // Improved schema definition
    tradeName: z.string(),
    employees: z.number().min(0), // A company cannot have less than 0 employees
  })
);

// Response type
export type GetCompaniesResponse = z.infer<typeof getCompaniesResponseSchema>

Note that you can improve GPT responses with some prompt engineering to add restrictions, descriptions, and possible meanings for each field, in the schema definition.
Feel free to explore.

Finally, use the schema to parse the responses:

/**
 * Gets companies list
 *
 * @example
 *   const companies = await getCompanies();
 *
 * @throws {ZodError} If parsing fails (response is not valid)
 */
const getCompanies = async (): Promise<GetCompaniesResponse> => {
  const response = await fetch('https://my-api.com/companies');

  return getCompaniesResponseSchema.parse(await response.json());
};

Definitions of Done

As the integration process progresses, it's imperative to establish clear conditions for its conclusion. Ideally, these criteria should be aligned even before the initiation of RFCs, ensuring a cohesive and well-orchestrated development journey.

However, the final stage of delivery is the "Definitions of Done". It encapsulates all the necessary conditions for integration completion. These conditions encompass every integration requirement, essentially encapsulating all the data required for resolving the targeted problem.

Once these defined requirements are met, the integration achieves its state of readiness. These "Definitions of Done" serve as a comprehensive checklist, assuring that every aspect of the integration is not only functional but also aligned with the broader objectives it aims to address.

In essence, the clarity provided by these conclusive criteria ensures that the integration is not only technically sound but also effectively solves the targeted problem, contributing to a successful and impactful project delivery.