Forem: Daniel Reis

Data Engineering 101 - A real beginner's approach

Daniel Reis — Tue, 14 Oct 2025 00:30:14 +0000

This is the article about Data Engineering that you find if you search the subject on Google and get redirected after click Feeling Lucky.

Also, this article is a POV of an experienced webdev engineer that started a new topic in his career, that means: that's my study, my research, if you have anything to add, feel free to teach me in the comments below! I would love to learn more!

1. Prologue
2. First Impressions: Exploring Bruin’s Structure
- 2.1. Core File: .bruin.yml
- 2.2. Pipeline: A Way Easier Apache Airflow
- 2.3. Assets: The Building Blocks of Data Products
- 2.4. Policies: Enforcing Quality and Governance
- 2.5. Glossary: Speaking the Same Language
3. Building Our First Pipeline
- 3.1. Step 1: Ingest from Your Source to a Data Lake
- 3.2. Step 2: Formatting and Validating the Data (Staging Layer)
- 3.3. Step 3: Designing the Analytics (Mart) Layer
4. Full Preview Data Flow
5. Overpowered VS Code Extension
6. Conclusion

1. Prologue

I have to admit: whenever someone mentioned Data Engineering, I used to tune out. It always sounded like something impossibly complex — almost magical.

This week, I finally decided to dive in. I thought it would be fairly straightforward, but it didn’t take long to realize how deep the rabbit hole goes.

This field isn’t just a few scripts or SQL queries; it’s an entire ecosystem of interconnected tools, concepts, and responsibilities that form the backbone of modern data systems.

Concepts like:

Data Catalogs and Governance: understanding who owns the data, how to ensure quality, and how to track lineage.
Orchestration: coordinating dependencies and workflows with tools like Apache Airflow or Dagster.
Transformation (ETL/ELT): cleaning and standardizing data using tools such as dbt or Fivetran.
Ingestion and Streaming: connecting sources and moving data in real time with Kafka, Airbyte, or Confluent Cloud.
Observability and Quality: monitoring data health with solutions like Monte Carlo and Datafold.

For each article clicked, just opened up a new world of tools, words, frameworks, architectures, and best practices.
And somehow, all of it has to work together — governance, orchestration, transformation, ingestion, observability, infrastructure.

As a developer, I’m used to learning a language and a framework and then getting to work.
But in data engineering, it’s different.

It’s about understanding an entire ecosystem and how each piece connects to the next.

After hours of reading docs, chasing GitHub repos, and jumping between tools, articles, and endless definitions, I finally found the tool that made everything click — Bruin.

Imagine a single framework that:

Pipelines as Code — Everything lives in version-controlled text (YAML, SQL, Python). No hidden UIs or databases. Reproducible, reviewable, and automatable.
Multi-Language by Nature — Native support for SQL and Python, plus the ability to plug in binaries or containers for more complex use cases.
Composable Pipelines — Combine technologies, sources, and destinations in one seamless flow — no glue code, no hacks.
No Lock-In — 100% open-source (Apache-licensed) CLI that runs anywhere: locally, in CI, or in production. You keep full control of your pipelines and data.
Built for Developers and Data Quality — Fast local runs, integrated checks, and quick feedback loops. Data products that are tested, trusted, and easy to ship.

…and it fits all the core Data Engineering concepts I just mentioned earlier.

I’ll admit it — I’m the kind of person who embraces productive laziness. If there’s a way to do more with fewer tools and less friction, I’m in.

So before we get started, here’s the plan:

In most setups, data flows from OLTP databases → ingestion → data lake/warehouse → transformation → marts → analytics dashboards.

Tools like Airbyte handle ingestion, dbt handles transformation, Airflow orchestrates dependencies — and Bruin combines those layers
into one unified framework.

This article will walk through the fundamental principles of Data Engineering, while exploring how Bruin brings them all together through a simple, real-world pipeline.

2. First Impressions: Exploring Bruin’s Structure

I’ll be honest — I used to have a bit of a bias against Data Science/Engineering projects. Every time I looked at one, it felt messy and unstructured, with files and notebooks scattered everywhere. Coming from a software development background, that kind of chaos always bothered me.

But once I started looking at Bruin’s project structure, that perception completely changed. Everything suddenly felt organized and intentional.
The framework naturally enforces structure through its layers — and once you follow them, everything starts to make sense.

Example: Project Structure

├── duckdb.db  
├── ecommerce-mart  
│ ├── assets  
│ │ ├── ingestion  
│ │ │ ├── raw.customers.asset.yml  
│ │ │ ├── raw.order_items.asset.yml  
│ │ │ ├── raw.orders.asset.yml  
│ │ │ ├── raw.products.asset.yml  
│ │ │ └── raw.product_variants.asset.yml  
│ │ ├── mart  
│ │ │ ├── mart.customers-by-age.asset.py  
│ │ │ ├── mart.customers-by-country.asset.yml  
│ │ │ ├── mart.product_performance.sql  
│ │ │ ├── mart.sales_daily.sql  
│ │ │ └── mart.variant_profitability.sql  
│ │ └── staging  
│ │ ├── stg.customers.asset.yml  
│ │ ├── stg.order_items.sql  
│ │ ├── stg.orders.sql  
│ │ ├── stg.products.sql  
│ │ └── stg.product_variants.sql  
│ └── pipeline.yml  
├── glossary.yml  
├── policy.yml  
└── .bruin.yml

What Each Part Does

.bruin.yml
- The main configuration file for your Bruin environment.
- Defines global settings like default connections, variables, and behavior for all pipelines.
policy.yml
- Your data governance and validation policy file.
- Defines data quality rules, access controls, and compliance checks that Bruin can automatically enforce before shipping data products.
glossary.yml
- Works as a lightweight data catalog for your project.
- Documents terms, metrics, and datasets so everyone on the team speaks the same language.
- Also helps with lineage, documentation, and discoverability.
some-feature/pipeline.yml
- Defines a specific pipeline for a domain or project (in this example, ecommerce).
- Describes the end-to-end data flow — which assets to run, their dependencies, and schedules.
- Pipelines are modular, so you can maintain separate ones for different business domains.
some-feature/assets/*
- Contains all the assets — the building blocks of your data pipelines.
- Each asset handles a distinct task: ingesting raw data, transforming it, or generating analytical tables.
- Since every asset is a file, it’s version-controlled, testable, and reusable — just like code.

With just that, we're able to run a full pipeline. However, I still think we need to go through each step and file individually — I promise it’ll be quick!

2.1. Core File: `.bruin.yml`

Think of .bruin.yml as the root configuration of your project — the file that tells Bruin how and where to run everything.

Instead of scattering settings across scripts or environment variables, Bruin centralizes them here: connections, credentials, and environment-specific configurations all live in one place.
It also serves as Bruin’s default secrets backend, so your pipelines can access databases or warehouses securely and consistently.

bruin run ecommerce/pipeline.yml --config-file /path/to/.bruin.yml

A simple example:

default_environment: default

environments:
  default:
    connections:
      postgres:
        - name: pg-default
          username: postgres # (hardcoded as well)
          password: ${PG_PASSWORD}
          host: ${PG_HOST}
          port: ${PG_PORT}
          database: ${PG_DATABASE}

      duckdb:
        - name: duckdb-default
          path: duckdb.db

What’s Happening Here

default_environment — sets the environment Bruin will use unless specified otherwise.
environments — defines multiple setups (e.g., dev, staging, prod), each with its own configuration.
connections — lists every system Bruin can connect to, like Postgres or DuckDB. Each connection gets a name (e.g., pg-default) that you’ll reference across pipelines and assets.
Environment variable support — any value wrapped in ${...} will be automatically read from your system environment.

This means you can keep credentials out of source control while still running locally or in CI/CD environments.

This design keeps everything centralized, secure, and version-controlled, while giving you the flexibility to inject secrets dynamically through environment variables — perfect for switching between local, staging, and production without touching the code.

2.2. Pipeline: A WAY EASIER Apache Airflow

For each feature you have, it will have to come with a pipeline.yml file.
This is the file that will group all your assets and understand that it's not a single asset running, but a chained list of assets.

- ecommerce-mart/
  ├─ pipeline.yml -> you're here
  └─ assets/
    ├─ some-asset.sql
    ├─ definitely-an-asset.yml
    └─ another-asset.py

There you also configure each connection you want to use on the specific pipeline:

name: product_ecommerce_marts
schedule: daily # relevant for Bruin Cloud deployments

default_connections:
  duckdb: "duckdb-default"
  postgres: "pg-default"

2.3. Assets: The Building Blocks of Data Products

Every data pipeline in Bruin is composed of assets — modular, self-contained units that define a specific operation: ingesting, transforming, or producing a dataset.

Each asset exists as a file under the assets/ directory, and its filename doubles as its identity inside the pipeline graph.

If you remember the file structure shown in the beginning, you must remember that I have multiple types of assets in the pipeline. That's the most cool part, since you can write down in multiple languages and still being something simple. Here's some possibilities:

Type	Description	Filename (in the file tree)
YAML	Declarative configuration for ingestion or metadata-heavy assets	`raw.customers.asset.yml`
SQL	Pure transformation logic — think dbt-style models	`stg.orders.sql`
Python	Custom logic or integrations (e.g., APIs, validations, or machine learning steps)	`mart.sales_daily.asset.py`

You’re free to organize assets however you like — there’s no rigid hierarchy to follow.
The key insight is that the orchestration happens implicitly through dependencies, not through an external DAG engine like Airflow.

Each asset declares what it depends on, and Bruin automatically builds and executes the dependency graph for you.

Example:

raw.orders.asset.yml

# raw.orders.asset.yml
name: raw.orders
type: ingestr
description: Ingest OLTP orders from Postgres into the DuckDB raw layer.
parameters:
  source_connection: pg-default
  source_table: "public.orders"
  destination: duckdb

raw.order_items.asset.yml

# raw.order_items.asset.yml
name: raw.order_items
type: ingestr
description: Ingest OLTP order_items from Postgres into the DuckDB raw layer.
depends:
  - raw.orders  # declares a dependency on the 'raw.orders' asset
parameters:
  source_connection: pg-default
  source_table: "public.order_items"
  destination: duckdb

... turns into:

graph TD
    raw.orders --> raw.order_items;

By chaining assets like this, you describe logical relationships between data operations rather than manually orchestrating steps.
The result is a declarative, composable, and maintainable pipeline — easy to read, version, and extend just like application code.

One of the most powerful aspects of Bruin is how it connects data quality and governance directly into your assets.
By defining checks under each column, you’re not only validating your data but also documenting ownership, expectations, and constraints — all version-controlled and enforceable at runtime.

This means Bruin doesn’t just run pipelines — it audits, documents, and governs them as part of the same workflow.

2.4. Policies: Enforcing Quality and Governance

Policies in Bruin act as the rulebook that keeps your data pipelines consistent, compliant, and high quality.
They ensure every asset and pipeline follows best practices — from naming conventions and ownership to validation and metadata completeness.

At their core, policies are defined in a single policy.yml file located at the root of your project.
This file lets you lint, validate, and enforce standards automatically before a pipeline runs.

Quick Overview

rulesets:
  - name: standard
    selector:
      - path: .*/ecommerce/.*
    rules:
      - asset-has-owner
      - asset-name-is-lowercase
      - asset-has-description

Each ruleset defines:

where the rule applies (selector → match by path, tag, or name),
what to enforce (rules → built-in or custom validation rules).

Once defined, you can validate your entire project:

bruin validate ecommerce

# Validating pipelines in 'ecommerce' for 'default' environment...
# Pipeline: ecommerce_pg_to_duckdb (.)
#   raw.order_items (assets/ingestion/raw.order_items.asset.yml)
#     └── Asset must have an owner (policy:standard:asset-has-owner)

Bruin automatically lints assets before execution — ensuring that non-compliant pipelines never run.

Built-in and Custom Rules

Rule	Target	Description
`asset-has-owner`	asset	Each asset must define an owner.
`asset-has-description`	asset	Assets must include a description.
`asset-name-is-lowercase`	asset	Asset names must be lowercase.
`pipeline-has-retries`	pipeline	Pipelines must define retry settings.

You can also define your own rules:

custom_rules:
  - name: asset-has-owner
    description: every asset should have an owner
    criteria: asset.Owner != ""

Rules can target either assets or pipelines, and they use logical expressions to determine compliance.

Policies transform Bruin into a self-governing data platform — one where best practices aren’t optional, they’re enforced.
By committing your rules to version control, you make data governance part of the development workflow, not an afterthought.

2.5. Glossary: Speaking the Same Language

In data projects, one of the hardest problems isn’t technical — it’s communication.
Different teams often use the same word to mean different things.
That’s where Bruin’s Glossary comes in.

A glossary is defined in glossary.yml at the root of your project.
It acts as a shared dictionary of business concepts (like Customer or Order) and their attributes, keeping teams aligned across pipelines.

entities:
  Customer:
    description: A registered user or business in our platform.
    attributes:
      ID:
        type: integer
        description: Unique customer identifier.

You can reference these definitions inside assets using extends, avoiding duplication and ensuring consistency:

# raw.customers.asset.yml
name: raw.customers
type: ingestr

columns:
  - name: customer_id
    extends: Customer.ID

This automatically inherits the type and description from the glossary.
It’s a simple idea, but a powerful one — your data definitions become version-controlled and shared, just like code.

3. Building Our First Pipeline

Now that we’ve explored the structure and philosophy behind Bruin, it’s time to build an end-to-end pipeline.

We’ll go from raw ingestion to a clean staging layer, and finally, to analytics-ready marts — all defined as code.

We’ll assume you already have:

A Postgres database as your data source.
A DuckDB database as your analytical storage.
A working .bruin.yml file configured with both connections ### 3.1 Step 1 : Ingest from Your Source to a Data Lake

The first step is to move data from Postgres into DuckDB.

This creates your Raw Layer — data replicated from the source with minimal transformation.

Create an ingestion asset file:

touch assets/ingestion/raw.customers.asset.yml

Then define the asset:

# assets/ingestion/raw.customers.asset.yml
name: raw.customers
type: ingestr

description: Ingest OLTP customers from Postgres into the DuckDB raw layer.

parameters:
  source_connection: pg-default
  source_table: "public.customers"
  destination: duckdb

columns:
  - name: id
    type: integer
    primary_key: true
    checks:
      - name: not_null
      - name: unique

  - name: email
    type: string
    checks:
      - name: not_null
      - name: unique

  - name: country
    type: string
    checks:
      - name: not_null

This tells Bruin to extract data from your Postgres table public.customers, validate column quality, and store it in the DuckDB raw layer.

Running the Asset

bruin run ecommerce/assets/ingestion/raw.customers.asset.yml

Expected output:

Analyzed the pipeline 'ecommerce_pg_to_duckdb' with 13 assets.
Running only the asset 'raw.customers'

  Pipeline: ecommerce_pg_to_duckdb (../../..)
  No issues found                                                                                                                                                                                                                                  
✓ Successfully validated 13 assets across 1 pipeline, all good.                                                                                                 
  Interval: 2025-10-12T00:00:00Z - 2025-10-12T23:59:59Z

Starting the pipeline execution...
  PASS raw.customers ........

  bruin run completed successfully in 2.095s

✓ Assets executed      1 succeeded

You can now query the ingested data:

bruin query --connection duckdb-default --query "SELECT * FROM raw.customers LIMIT 5"

Result:

┌────┬───────────────────┬───────────────────────────┬───────────┬──────────────────┬──────────────────────────────────────┬──────────────────────────────────────┐
│ ID │ FULL_NAME         │ EMAIL                     │ COUNTRY   │ CITY             │ CREATED_AT                           │ UPDATED_AT                           │
├────┼───────────────────┼───────────────────────────┼───────────┼──────────────────┼──────────────────────────────────────┼──────────────────────────────────────┤
│ 1  │ Allison Hill      │ donaldgarcia@example.net  │ Uganda    │ New Roberttown   │ 2025-10-10 18:19:13.083281 +0000 UTC │ 2025-10-10 00:42:59.71112 +0000 UTC  │
│ 2  │ David Guzman      │ jennifermiles@example.com │ Cyprus    │ Lawrencetown     │ 2025-10-10 07:52:47.643619 +0000 UTC │ 2025-10-10 06:23:42.864287 +0000 UTC │
│ 3  │ Caitlin Henderson │ eric51@example.org        │ Hong Kong │ West Melanieview │ 2025-10-10 21:06:02.639412 +0000 UTC │ 2025-10-10 19:23:17.540169 +0000 UTC │
│ 4  │ Monica Herrera    │ smiller@example.net       │ Niger     │ Barbaraland      │ 2025-10-11 01:33:43.032929 +0000 UTC │ 2025-10-10 02:29:27.22515 +0000 UTC  │
│ 5  │ Darren Roberts    │ wyattmichelle@example.com │ Fiji      │ Reidstad         │ 2025-10-10 12:05:18.734246 +0000 UTC │ 2025-10-10 00:51:13.406526 +0000 UTC │
└────┴───────────────────┴───────────────────────────┴───────────┴──────────────────┴──────────────────────────────────────┴──────────────────────────────────────┘

Your raw layer is now established and validated.

3.2 Step 2: Formatting and Validating the Data (Staging Layer)

Next, we’ll clean and standardize the ingested data before using it in analytics.

This layer is called Staging (stg) — it’s where you enforce schema, column consistency, and apply business rules.

Create the file:

touch ecommerce/assets/staging/stg.customers.asset.sql

And define it as follows:

/* @bruin
name: stg.customers
type: duckdb.sql
materialization:
  type: table

depends:
  - raw.customers

checks:
  columns:
    id:
      - not_null
    email:
      - not_null
      - unique
    country:
      - not_null
@bruin */

SELECT id::INT AS customer_id,
       COALESCE(TRIM(email), '') AS email,
       COALESCE(TRIM(country), 'Unknown') AS country,
       created_at,
       updated_at
FROM raw.customers
WHERE email IS NOT NULL;

Here’s what’s happening:

The Bruin annotation block (@bruin) defines metadata for the asset.
The depends key ensures this staging step only runs after raw.customers completes — Bruin automatically manages the dependency chain.
The column checks ensure data quality before and after the transformation.
The SQL query itself performs light cleaning and enforces type consistency.

This design mimics orchestration tools like Airflow, but without external schedulers — dependencies and checks are declared right inside your code.

Validation and Execution

Before running, validate the asset:

bruin validate ecommerce/assets/staging/stg.customers.asset.sql

Expected output:

Pipeline: ecommerce_pg_to_duckdb (.)
  No issues found      

✓ Successfully validated 13 assets across 1 pipeline, all good.

Now execute it:

bruin run ecommerce/assets/staging/stg.customers.asset.sql

Result:

bruin run ecommerce/assets/staging/stg.customers.asset.sql
Analyzed the pipeline 'ecommerce_pg_to_duckdb' with 15 assets.
Running only the asset 'stg.customers'

Pipeline: ecommerce_pg_to_duckdb (../../..)
  No issues found                                                                                                                                                                                                                                   

✓ Successfully validated 15 assets across 1 pipeline, all good.                                                       

Interval: 2025-10-12T00:00:00Z - 2025-10-12T23:59:59Z

Starting the pipeline execution...

[21:28:16] Running:  stg.customers
[21:28:16] Finished: stg.customers (41ms)

==================================================

PASS stg.customers 


bruin run completed successfully in 41ms

 ✓ Assets executed      1 succeeded

Confirm that the transformation worked:

bruin query "SELECT country, COUNT(*) AS customers FROM stg.customers GROUP BY country ORDER BY customers DESC;"

Sample output:

country      | customers
--------------+-----------
BRAZIL        | 420
GERMANY       | 255
UNITED STATES | 198
ARGENTINA     | 190
SOUTH KOREA   | 182

At this point, you have a clean, validated dataset ready for analytics.

3.3 Step 3: Designing the Analytics (Mart) Layer

The final step is to build your Mart Layer, where business-ready data lives.

This is the layer analysts and dashboards query directly.

Each Mart asset aggregates or reshapes the staging data into meaningful datasets for reporting and analysis.

3.3.1 Asset: `mart.customers_by_country.asset.sql`

Create the following file:

touch ecommerce/assets/mart/mart.customers_by_country.asset.sql

Then define the asset:

/* @bruin
name: mart.customers_by_country
type: duckdb.sql

materialization:
  type: table

depends:
  - stg.customers
@bruin */

SELECT
    country,
    COUNT(*) AS total_customers
FROM stg.customers
GROUP BY country
ORDER BY total_customers DESC;

This SQL asset aggregates customers by country and depends on stg.customers, ensuring the staging layer runs first.

It materializes as a table inside DuckDB.

Run it:

bruin run ecommerce/assets/mart/mart.customers_by_country.asset.sql

Expected output:

Pipeline: ecommerce_pg_to_duckdb (.)
Running mart.customers_by_country
✓ Table materialized successfully in DuckDB

Verify the results:

bruin query --connection duckdb-default --query "SELECT * FROM mart.customers_by_country LIMIT 5;"

Result:

┌───────────────┬───────────────────┐
│ COUNTRY       │ TOTAL_CUSTOMERS   │
├───────────────┼───────────────────┤
│ BRAZIL        │ 420               │
│ GERMANY       │ 255               │
│ UNITED STATES │ 198               │
│ ARGENTINA     │ 190               │
│ SOUTH KOREA   │ 182               │
└───────────────┴───────────────────┘

3.3.2 Asset: `mart.customers_by_age.asset.sql`

Now, let’s create a second mart that segments customers into age groups.

Create the file:

touch ecommerce/assets/mart/mart.customers_by_age.asset.sql

Define it as follows:

/* @bruin
name: mart.customers_by_age  
type: duckdb.sql  

materialization:  
  type: table  
depends:  
  - stg.customers  

@bruin */  

WITH src AS (
  SELECT 
    CASE  
      WHEN age < 25 THEN '18-24'  
      WHEN age BETWEEN 25 AND 34 THEN '25-34'  
      WHEN age BETWEEN 35 AND 49 THEN '35-49'  
      ELSE '50+'  
    END AS age_group  
  FROM stg.customers
)
SELECT 
  age_group,  
  COUNT(*) AS total_customers  
FROM src  
GROUP BY age_group  
ORDER BY total_customers DESC;

This asset computes customer distribution by age brackets using a simple CASE expression and aggregates the results.

Run it:

bruin run ecommerce/assets/mart/mart.customers_by_age.asset.sql

Expected output:

bruin run ecommerce/assets/mart/mart.customers_by_age.asset.sql
Analyzed the pipeline 'ecommerce_pg_to_duckdb' with 15 assets.
Running only the asset 'mart.customers_by_age'

Pipeline: ecommerce_pg_to_duckdb (../../..)
  No issues found

✓ Successfully validated 15 assets across 1 pipeline, all good.

Interval: 2025-10-12T00:00:00Z - 2025-10-12T23:59:59Z

Starting the pipeline execution...

[21:10:24] Running:  mart.customers_by_age
[21:10:24] Finished: mart.customers_by_age (39ms)

==================================================

PASS mart.customers_by_age 


bruin run completed successfully in 39ms

 ✓ Assets executed      1 succeeded

Confirm that the mart is populated correctly:

bruin query --connection duckdb-default --query "SELECT * FROM mart.customers_by_age;"

Result:

┌─────────────┬───────────────────┐
│ AGE_GROUP   │ TOTAL_CUSTOMERS   │
├─────────────┼───────────────────┤
│ 25-34       │ 460               │
│ 35-49       │ 310               │
│ 18-24       │ 250               │
│ 50+         │ 225               │
└─────────────┴───────────────────┘

4. Full Preview Data Flow

Hopefully, if I got it right, the concepts explained in the beginning didn't got that messy and we actually made the pipeline become a real thing only using Bruin and a couple of queries!

Our example pipeline look like this:

graph TD
    raw.customers --> stg.customers
    stg.customers --> mart.customers_by_country
    stg.customers --> mart.customers_by_age

At this stage, you’ve built a fully declarative, end-to-end data pipeline with Bruin — ingestion, staging, and analytics — all governed, validated, and reproducible, without any external orchestration layer.

5. Overpowered VS Code Extension

When I first installed the Bruin VS Code Extension, I had no idea what I was doing.

At that time, I didn’t really understand how Bruin — or even Data Engineering — worked.

I clicked around, saw a bunch of YAMLs and metadata markers, and quickly gave up.

Week later, after finally understanding the ecosystem — ingestion, staging, marts, and governance — I decided to open the extension again, ad that’s when it all clicked.

It wasn’t just a helper. It was the missing piece.

The extension brings the same declarative power of Bruin’s CLI into a visual, developer-friendly environment.

It automatically scans your assets, validates configurations, runs queries directly against your databases, and even manages your YAMLs in real time.

Everything happens inside VS Code — validation, lineage exploration, metadata checks, and query previews.

What impressed me most was how fluid and open it is. There’s no vendor lock-in here — it’s completely open source.

You can fork it, extend it, or contribute back to the community just like the CLI itself.

In short, the Bruin VS Code Extension isn’t just a companion — it’s the natural evolution of the workflow. Once you understand Bruin, this tool feels like magic finally explained.

6. Conclusion

This study was my first real attempt to understand Bruin and what Data Engineering actually means in practice. What once felt abstract and complicated started to make a bit more sense after breaking things apart, experimenting, and connecting the dots one step at a time.

I wouldn’t say I fully understand everything yet—far from it—but I can finally see how the pieces fit together: ingestion, staging, marts, validation, and governance. Bruin helped me get hands-on with those concepts in a way that felt approachable instead of overwhelming.

The process of exploring, failing, reading, and rebuilding turned out to be the most valuable part. There’s still a lot to learn, but this was a solid first step toward really understanding how data moves, transforms, and tells a story (just like this article).

GitHub Demo: E-commerce Pipeline with Bruin
Bruin Website

Have you used Rust (or Go, or anything else) and just felt relieved by how they treat errors?

Daniel Reis — Mon, 31 Mar 2025 16:59:34 +0000

Alright, I need to vent. I've reflecting about engineering recently I swear, exception handling in PHP feels like duct taping over a leaking pipe.

Like, why is everything so optional? You can throw exceptions, sure, but half the time you're just praying that some function doesn't silently fail or spit out false without context.

Meanwhile, I’ve been playing with Rust and Go lately and bro... it’s like, it feels safer (?). Every error is explicit. Either it worked (Ok) or it didn’t (Err), and you have to handle it. You don’t get to ignore it and pretend everything’s fine. The compiler literally says: "Nope, go back and deal with it."

But I feel that do that in PHP would be an Anti-Pattern but at the same time would be safer to work with this type of "Result Objects".

With Exceptions

function getUser(int $id): array {
    // Simulate DB access or something that might throw
    if ($id <= 0) {
        throw new InvalidArgumentException("Invalid user ID");
    }

    return ['id' => $id, 'name' => 'Daniel'];
}

try {
    $user = getUser(0); // ⚠️ This will throw
    echo "User name: " . $user['name'];
} catch (InvalidArgumentException $e) {
    echo "Caught exception: " . $e->getMessage();
} catch (Exception $e) {
    echo "Something went wrong: " . $e->getMessage();
}

With a Result Class

class Result {
    private function __construct(
        public bool $isOk,
        public mixed $value = null,
        public mixed $error = null
    ) {}

    public static function ok(mixed $value): self {
        return new self(true, $value);
    }

    public static function err(mixed $error): self {
        return new self(false, null, $error);
    }

    public function isOk(): bool {
        return $this->isOk;
    }

    public function unwrap() {
        if (!$this->isOk) {
            throw new RuntimeException("Tried to unwrap an Err: {$this->error}");
        }
        return $this->value;
    }

    public function unwrapOr(mixed $fallback): mixed {
        return $this->isOk ? $this->value : $fallback;
    }

    public function getError(): mixed {
        return $this->error;
    }
}

// --------------

function getUser(int $id): Result {
    if ($id <= 0) {
        return Result::err("Invalid user ID");
    }

    return Result::ok(['id' => $id, 'name' => 'Daniel']);
}

$result = getUser(0);

if ($result->isOk()) {
    $user = $result->unwrap();
    echo "User name: " . $user['name'];
} else {
    echo "Error: " . $result->getError();
}

I know that the "lack" of generics (annotations solve this partially) makes it harder to accomplish, but I mean, this would improve the whole ecosystem IMHO.

Also I just started to think about that because recently Microsoft started the revamp/reload/renew of typescript in go and I was wondering: what would be if PHP got a new engine (since Zend Engine is a total mess) with a couple of new features? Would be amazing.

How I learn any type of new technology (As a Senior Developer)

Daniel Reis — Thu, 20 Mar 2025 19:40:16 +0000

Recently, I had a task of learning a new tool that I had never used or seen anything related to, and I thought: why not write about how I learned it?

This is the kind of article that teaches you to learn things from a different perspective. Not the best way, but you can reuse some of the concepts written here.

Table of Content

1. Prologue
2. The Basics
- 2.1 Break it until you make it
- 2.2. But... It's not that obvious
3. The Documentation
4. The Source Code
5. The Network Tab
6. The Implementation
7. Conclusion

1. Prologue

Recently I've started a new project (Yet Another Poor SaaS Application) with my friends using PHP and Laravel. The project doesn't really matter, but the stack does. Why is that? Because we're not rich enough to spend money on self-hosted infrastructure, nor on good paid tools... So it's important.

So, if we're going to build something that has any kind of integration, it needs to at least have a free tier to consume and not be a living hell to implement. And in thinking about that, we stumbled upon: how are we going to do the user tracking/metrics?

My goal is to show the analytics per page of the SaaS user The average developer might think: we can use Google Analytics and Google Tag Manager, right? Well... Let me tell you something that Chester said once:

I tried so hard and got so far. But in the end, it doesn't even matter - Chester Bennington

Dude, I've always been a Google UI hater. But this time it's not for my job but for me and I have no time to waste. So, I spent 4 hours trying to do... the basics of setting it up on my project and understanding how to create a simple tag? And at the end I wasted my time since nothing worked.

Like, why all Google products have an interface that you need a fucking PhD to use it? Too much information everywhere!

That's when my friend told me about PostHog, an open-source analytics platform that lets you track user behavior with self-hosted or cloud options (and beat GTM/GA4 in matter of implementation). But at the same time, I'd like to ask you to stay until the end of this article, because it took me some hours and lines of code to do this study.

2. The Basics

Ok, this is the most obvious thing, but it has to be said: when you try a new tool, you have to play with every aspect of it. I know we're developers, but the UI of a product is trying to tell you something.

2.1 Break it until you make it

This PostHog process took infinitely less time to set up and that was definitely a plus. However, the UI has too many things happening and my first idea was to try to click everything in and break the project I was on in record time.

Just think with me: if you're afraid to use a platform/product just because you're in a cloud or "production" environment, you're not going to learn anything in the end.

At the end of this step, I just deleted the messed up project and started a new one. But at this point, I know how the UI works.

DISCLAIMER: DON'T DO IT AT YOUR COMPANY'S CLOUD ACCOUNT, CREATE ONE AND EXPLODE YOUR OWN ENVIRONMENT!!

2.2 But... It's not that obvious

Ok... I have to be honest with you, sometimes breaking everything is not the answer because it's not even an question depending on the feature you're using. At PostHog they have things like "TrendsQuery", "RetentionQuery", "Web Vitals" and a really fancy UI query builder for each type of query. Really impressive! But... I couldn't use it right away.

Since I have this amazing condition called ADHD, when I see minimally complex UI's with conditional components appearing on the screen, I just freak out and start panicking. But that's no reason to give up, right?

This section is not a criticism of the product, but me getting it out of my chest. Data science is an area of infinite conditional rules, where the more information you have, the more accurate it will be, and I'm taking my first steps to learn it.

3. The Documentation

Sure, the documentation! Why haven't I played with this before? I mean, this is the place for developers, right? Right! But we're developers who are going to use the product, so the first step is a must.

I don't usually go through tutorials, even though I know everyone should. I just don't like it, to me good API references must tells you everything, but what would be everything in my opinion?

"Indexable" Title: under a table of contents, this helps a lot.
Refined Description: tells everything that you should know in a first moment to use.
Implementation Examples: using languages like PHP, JS, Go, Rust and mainly cURL
Expected Responses: which payload with a "real data" will be served based on the response status (200, 401, 422, 500).
Implementations Reference: Where you can find the actual code to read and understand how things works at the client/server side.
Required Scopes: Usually you should have scopes for the endpoint.

In most cases you'll have many elements like these which I listed above, but it depends on the API you're using. In my case with this PostHog specific implementation, the challenge is that the /query/create endpoint has a polymorphic request and response and so there's no response payload there.

So that's when we start digging into the real code because we need to understand how these endpoints works with the actual implementation.

To avoid I could just write down a HogQL Query, but learning it (for me) would take way more time.

4. The Source Code

We know what we want at this point, right? We want to understand what payload is being sent for each type of request. In my case, I want to understand how Retention and Trends Query works on PostHog. And since the product is open source, I can read it directly from GitHub!

The front-end is written in TypeScript and the back-end is written in Python. Since I'm a back-end developer, you'd think I'd pick the Python implementation to dig into, right? Well, actually, I'm a Python hater for many reasons. Also, TypeScript gives me a type safety to use as a reference for code search.

In the TypeScript source I found the implementation of everything I needed on the client side, but after seeing it I went straight down the rabbit hole and you will understand why.

Take a look at this snippet (reference):

export interface TrendsQuery extends InsightsQueryBase<TrendsQueryResponse> {
    kind: NodeKind.TrendsQuery
    /**
     * Granularity of the response. Can be one of `hour`, `day`, `week` or `month`
     *
     * @default day
     */
    interval?: IntervalType
    /** Events and actions to include */
    series: AnyEntityNode[]
    /** Properties specific to the trends insight */
    trendsFilter?: TrendsFilter
    /** Breakdown of the events and actions */
    breakdownFilter?: BreakdownFilter
}

Here we can assume a few things:

There's an inherited InsightsQueryBase, so other queries have the same base.
The TrendsQueryResponse is available and must follow a pattern, so we can check it later.
The query "concerns" (which you can think of as a filter) isn't as explicit and will require more research.

First, we need to know what is in the basis and look for the InsightsQuery Concerns:


/** Base class for insight query nodes. Should not be used directly. */
export interface InsightsQueryBase<R extends AnalyticsQueryResponseBase<any>> extends Node<R> {
    /** Date range for the query */
    dateRange?: DateRange
    /**
     * Exclude internal and test users by applying the respective filters
     *
     * @default false
     */
    filterTestAccounts?: boolean
    /**
     * Property filters for all series
     *
     * @default []
     */
    properties?: AnyPropertyFilter[] | PropertyGroupFilter
    /**
     * Groups aggregation
     */
    aggregation_group_type_index?: integer | null
    /** Sampling rate */
    samplingFactor?: number | null
    /** Colors used in the insight's visualization */
    dataColorTheme?: number | null
    /** Modifiers used when performing the query */
    modifiers?: HogQLQueryModifiers
}

And that's the moment you understand that you are digging in a rabbit hole because these concerns seem to be HUGE objects that grant different behaviors within your API.

After a few hours into it, I started to realize how these concerns affect the query, things like:

Breakdown: More like a a GroupBy clause
Properties: Polymorphic Filtering which totally depends on the query
Series: The type of the data being retrieved (e.g. "views count")
Interval: a from/to date filtering.

This is just the tip of the iceberg. There are plenty of other concerns surrounding the many requests on PostHog. But here are the questions:

If there is an InsightsQueryBase, wouldn't these concerns be shared with other types of queries of that category?
Is there a way for other types of base queries to have the same concerns?
Should I understand how each type of **concern affects the final query?

I mean, we have to understand. Maybe not that deep, but a general knowledge. But how do we get proof if the documentation doesn't give us examples?

Well, that's where I explain my favorite thing about a browser: the Network tab!

5. The Network Tab

I just learned all about how to "explore" web applications directly from the browser by doing A-B testing with endpoints. You might think:

"Depending on the service, you can't build a self-user bot because it may violate the terms of service" - Normal User

and I would answer that I couldn't care less because I'm doing it for educational purposes. If I'm doing it, it's to learn how things work, to build my own tools and use their product.

Just a random fact: I've been inspecting elements/requests since 2014. My first "real" GitHub project was basically inspecting all requests from a browser game called "TribalWars" and building a CLI app/bot to play the game through my terminal with the ultimate goal of not playing it (building a self-bot account).

Back to the PostHog implementation, when I started inspecting, I found different types of payloads for each type of query. With that, I could make more assumptions and use it for the next step.

{
    "client_query_id": "54f620e7-fdfe-4749-af41-caed0a3fe671",
    "query": {
        "breakdownFilter": {
            "breakdown": "$geoip_country_code",
            "breakdown_type": "event"
        },
        "conversionGoal": null,
        "dateRange": {
            "date_from": "-7d",
            "date_to": null
        },
        "filterTestAccounts": false,
        "kind": "TrendsQuery",
        "properties": [],
        "series": [
            {
                "event": "$pageview",
                "kind": "EventsNode",
                "math": "dau",
                "name": "Pageview"
            }
        ],
        "trendsFilter": {
            "display": "WorldMap"
        }
    },
    "refresh": "async"
}

and the second query just in case :p

{
    "client_query_id": "25d4b82f-5bbe-4665-89a7-7aedcc7b103e",
    "query": {
        "dateRange": {
            "date_from": "-7d",
            "date_to": null
        },
        "filterTestAccounts": false,
        "kind": "RetentionQuery",
        "properties": [],
        "retentionFilter": {
            "period": "Week",
            "retentionReference": "total",
            "retentionType": "retention_first_time",
            "totalIntervals": 8
        }
    },
    "refresh": "async"
}

I would include the results query, but that would be too much code and the point of this section is already made. Now it's time to finish this entire flow.

6. The Implementation

That's when you start thinking, "Does the tooling available meet my needs? If the SDK doesn't give me a fully typed API, I'd rather build my own stuff. I mean, I like to do things from TOTALLY SCRATCH! Yeah, the whole reinventing the wheel thing, you know?

The reason is simple: I'm a developer who learns how things work by rebuilding them and observing how they behave. And with that, I decided to build a PostHog Dedicated Query Builder with PHP to learn how things work on my end.

So far this is what I've got:

We have multiple query types;
These queries can inherit properties/concerns but not necessarily use all of them;
Queries share filters/concerns;

So how do we organize it? And after hours of digging in the Source and Network tabs, this was my answer:

------
- QueryBuilder Feature at my own PostHogSDK
------

Query
├── Builders
│   ├── AbstractQueryBuilder.php
│   └── Insights
│       ├── AbstractInsightsQueryBuilder.php
│       ├── RetentionQueryBuilder.php
│       └── TrendsQueryBuilder.php
├── Filters
│   ├── Breakdown
│   │   ├── BreakdownFilter.php
│   │   ├── BreakdownTypeEnum.php
│   │   └── Concerns
│   │       └── InteractsWithBreakdown.php
│   ├── Compare
│   │   ├── CompareFilter.php
│   │   └── Concerns
│   │       └── InteractsWithCompare.php
│   ├── ConversionGoal
│   │   ├── ActionConversionGoal.php
│   │   ├── Concern
│   │   │   └── InteractsWithConversionGoal.php
│   │   ├── Contracts
│   │   │   └── ConversionGoalContract.php
│   │   └── CustomEventConversionGoal.php
│   ├── DateRange
│   │   ├── Concerns
│   │   │   └── InteractsWithDateRange.php
│   │   └── DateRangeFilter.php
│   ├── Interval
│   │   ├── Concerns
│   │   │   └── InteractsWithInterval.php
│   │   └── QueryIntervalEnum.php
│   ├── Node
│   │   ├── ActionNode.php
│   │   ├── Concerns
│   │   │   ├── InteractsWithMath.php
│   │   │   └── InteractsWithSeries.php
│   │   ├── Contracts
│   │   │   └── EntityNodeContract.php
│   │   ├── EntityNodeKindEnum.php
│   │   ├── EntityNode.php
│   │   ├── EventsNode.php
│   │   └── MathEnum.php
│   ├── Properties
│   │   ├── BaseProperty.php
│   │   ├── Concerns
│   │   │   └── InteractsWithProperties.php
│   │   ├── Contracts
│   │   │   └── PropertyFilterContract.php
│   │   ├── Filters
│   │   │   ├── EventPropertyFilter.php
│   │   │   └── SessionPropertyFilter.php
│   │   ├── PropertyFilterKind.php
│   │   └── PropertyOperator.php
│   └── Retention
│       ├── Concerns
│       │   └── InteractsWithRetention.php
│       ├── Enums
│       │   ├── RetentionPeriodEnum.php
│       │   ├── RetentionReferenceEnum.php
│       │   └── RetentionTypeEnum.php
│       └── RetentionFilter.php
├── QueryBuilderInterface.php
├── QueryFactory.php
└── QueryKindEnum.php

Every filter has your own dedicated spot, where it can be inherited to any type of builder using PHP Traits:

trait InteractsWithDateRange  
{  
    protected ?DateRangeFilter $dateRange = null;  
    public function setDateRange(DateRangeFilter $dateRange): self  
    {  
        $this->dateRange = $dateRange;  

        return $this;  
    }    public function getDateRange(): ?DateRangeFilter  
    {  
        return $this->dateRange;  
    }  

    private function buildDateRange(array &$payload): void  
    {  
        if ($this->dateRange !== null) {  
            $payload['dateRange'] = [  
                'date_from' => $this->dateRange->from,  
                'date_to' => $this->dateRange->to,  
            ];        
        }    
    }
}

So the only requirement of a query builder is to have the "build()" method provided by my QueryBuilderContract, so I can ensure that at least the typing will be satisfied at the end. Each use in the class below is a different common QueryBuilder concern.

class TrendsQueryBuilder extends AbstractInsightsQueryBuilder  
{  
    use InteractsWithInterval,  
        InteractsWithDateRange,  
        InteractsWithBreakdown,  
        InteractsWithCompare,  
        InteractsWithConversionGoal,  
        InteractsWithProperties,  
        InteractsWithSeries;  

    protected QueryKindEnum $queryType = QueryKindEnum::TrendsQuery;  

    public static function make(): self  
    {  
        return new self();  
    }  


    public function build(): array  
    {  
        return $this->jsonSerialize();  
    }  

    public function jsonSerialize(): array  
    {  
        $payload = parent::jsonSerialize();  

        $payload['kind'] = $this->queryType->value;  

        $this->buildDateRange($payload);  
        $this->buildInterval($payload);  
        $this->buildSeries($payload);  
        $this->buildCompare($payload);  
        $this->buildProperties($payload);  
        $this->buildConversionGoal($payload);  
        $this->buildBreakdown($payload);  

        return $payload;  
    }
}

While we are coding, we have to pay double attention to the minimal details around the implementation that is being made. And frankly, when I build this kind of tool, I mostly think that someone will use it, which raises the attention bar even higher.

The end result of this study is a fully functional Trends and Retention QueryBuilder for the PostHog API's:


test('can build a retention query', function () {  
    $expected = [  
        "kind" => "RetentionQuery",  
        "dateRange" => [  
            "date_from" => "-7d",  
            "date_to" => null  
        ],  
        "filterTestAccounts" => false,  
        "retentionFilter" => [  
            "retentionType" => "retention_first_time",  
            "retentionReference" => "total",  
            "totalIntervals" => 8,  
            "period" => "Week"  
        ]  
    ];  
    $queryBuilder = RetentionQueryBuilder::make()  
        ->setDateRange(DateRangeFilter::from('-7d'))  
        ->setRetention(RetentionFilter::weekly()  
            ->setRetentionType(RetentionTypeEnum::FIRST_TIME)  
            ->setRetentionReference(RetentionReferenceEnum::Total)  
        );  
    expect($queryBuilder)->toBeInstanceOf(RetentionQueryBuilder::class)  
        ->and($queryBuilder->build())->toMatchArray($expected);  
});

test('can build a trends-query', function () {  
    $expected = [  
        'kind' => 'TrendsQuery',  
        'filterTestAccounts' => false,  
        'dateRange' => [  
            'date_from' => '-7d',  
            'date_to' => null,

        ],        
        'interval' => 'day',  
        'series' => [  
            [                
                'event' => '$pageview',  
                'kind' => 'EventsNode',  
                'math' => 'dau',  
                'name' => 'Pageview',  
            ],        
        ],        
            'compareFilter' => [  
            'compare' => true,  
        ],        
        'properties' => [  
            [                'type' => 'event',  
                'key' => '$host',  
                'operator' => 'exact',  
                'value' => 'api-main-ofjibb.laravel.cloud',  
            ],        
        ],    
    ];  
    $actual = TrendsQueryBuilder::make()  
        ->setDateRange(DateRangeFilter::from('-7d'))  
        ->setInterval(QueryIntervalEnum::Day)  
        ->addCompareFilter()  
        ->addSeries(EventsNode::make('$pageview', MathEnum::DAU, 'Pageview'))  
        ->addProperty(EventPropertyFilter::make('$host', PropertyOperator::Exact, 'api-main-ofjibb.laravel.cloud'));  


    expect($actual)->toBeInstanceOf(TrendsQueryBuilder::class)  
        ->and($actual->build())->toMatchArray($expected);  
});

You can check the Pull Request for this implementation clicking here. Feel free to contribute on this project too!

7. Conclusion

This is just a study in the PostHog API that I enjoyed a lot and decided to write about. Building things from scratch and doing proper research is something that can help you learn things on another level.

Now I can tell you that I can play with the PostHog platform in a really smooth way, without the worries mentioned in the first section of this article since I just breaked it once, divided concerns and studied each of them separately.

This approach—breaking things, diving into the docs, exploring the network tab, and rebuilding from scratch—can be applied to any technology. Whether it's PostHog, a new API, or a framework, understanding the internals makes you a better developer.

That's it! Hope you enjoyed and don't forget to drink water!

[Boost]

Daniel Reis — Fri, 21 Feb 2025 16:48:40 +0000

#DevOps para noobs - 3 Boas Práticas de Bancos de Dados para compartilhar com os Devs (AWS RDS)

Camila Figueira ・ Feb 20

This is all what I've learned about Go in TWO Weeks!

Daniel Reis — Fri, 25 Oct 2024 21:38:22 +0000

What is your approach when you need to learn something new? I have a very specific one and once again I tested it while learning Golang!

There's too much content to talk about, but my aim here is to list things that I found useful and that I specifically took the time to learn properly.

1. Prologue
2. Meet the CLI
- 2.1 CLI: go mod
- 2.2 CLI: go run
3. Comparing Different Syntax's
- 3.1 Syntax: Classes/Structs and the API Encapsulation
- 3.2 Syntax: Interface Implementation is WEIRD AS FUC*
4. Stdlib: definitely an AWESOME toolkit
- 4.1 Packages: Primitive Types
- 4.2 Packages: Useful Stuff
5. Tests in Go are THAT SIMPLE
- 5.1 Tests: Basic Testing
- 5.2 Tests: Jetbrains Boilerplate
- 5.3 Tests: Running Tests
6. Beware Cyclic Imports
7. Defer this, defer that... But, what is a Defer?
8. Error Management for Noobs
9. Conclusion feat: Low Latency Coding Challenge

1. Prologue

For the last 2 weeks I've been learning and building small applications with Golang. At the moment it's been almost 50h of code through many livestreams and it's been pretty awesome to learn something that I previously had a some small issues with the language.

In this two weeks journey I've crafted:

A small and REALLY simple shell
A Redis Basic implementation
HTTP 1.1 protocol implementation
A DNS server implementation
and a job test for a really cool company (which will be available at the end of this article).

And all this because my boss asked me, once again, to learn a new technology to work on some ScyllaDB PoC's and demos... I wasn't too happy with the decision, but well, it's my job.

During the last year I've been studying Rust, and it's probably still too complex for me, but I've learnt some really cool concepts that made my switch to Go work like a charm!

In this article I'll give you some tips and advice to speed up your learning flow.

2. Meet the CLI

I'm a PHP developer and I'm used to the BEST CLI ever made (yes, it's Artisan), however through my journey as a developer I've been through awesome projects many of which have been..:

Cargo (Rust)
NPM (JS)
Composer (PHP)
and so on...

When I got to the Go environment, it started as a real problem. At least for me, the developer experience of Golang in terms of tools and documentation could be much better. Thinking about this, I decided to go through 3 commands that you HAVE TO LEARN at the beginning.

Remember: this is just a walkthrough with my own explanation of things. If you want detailed information, open the docs :)
Also: go docs sucks please someone put a syntax highlighter there

2.1 CLI: go mod

Depending on whether you want to modularise your application or have an organised environment, this will be the most useful command at first.

The go mod command manages all the dependencies within your project and also takes care of autoremoving anything that's no longer used.

First, inside your new empty folder, let's init a new module inside the project with go mod init:

mkdir go-fodase
cd go-fodase

# go mod init <module-name>
go mod init github.com/danielhe4rt/go-fodase

This will create a new file in the project root called go.mod, which is basically the contents at the moment:

The module name
Your Go version

Here's the file, if you want to check it yourself:

# folder: ~/go-fodase
cat go.mod

# module github.com/danielhe4rt/gofodase
# 
# go 1.23.2

After that, the next thing I really liked was the go mod tidy, which basically adds any missing dependencies and removes unused ones.

go mod tidy

This second one is just to keep into your mind that this exists and it's really useful! Probably your environment will run it EVERY TIME and you will get used to see imports vanishing :p

2.2 CLI: go run

This is probably the most common command you'll use, since you HAVE to run your project, but here's how it works:

You should point to the file that contains the main() function.
This file doesn't have to be in the root of your folder.

The most important thing to remember about this command is that when you run the go run command, it will look for the go.mod file in your current directory and use it as the basis for mapping your whole project (imports, packages, etc). Here's some examples:

# go run <your-main-file>

# Environment 1
# .
# ├── go.mod
# └── main.go
go run app.go

# Environment 2
# .
# ├── go.mod
# └── src
#     └── main.go
go run src/app.go

Here's our app.go content:

package main

import(
    "fmt"
)

func main () {
    fmt.Println("whats up! don't forget to like this article <3")
}

Now you know the basics to execute an project! Literally, hello world!

3. Comparing Different Syntax's:

My problem with Go has always been the way it's written, but after hours of coding I realised that it's simpler than I thought. As you might have guessed, I have a strong background in PHP and some experience with Rust.

When I started to learn Rust in 2023, fortunately a guy I'm a big fan of, Nuno Maduro (Laravel), gave a talk called "PHP for Rust Developers", which gave me some basic introduction to the syntax and gave me some breathing space while I was completely STOMPED by Rust.

Anyway, it was useful to me at the time, so why not do some comparisons?

3.1 Syntax: Classes/Structs and the API Encapsulation

In OOP we have classes, which is a really cool way of abstracting your code into small pieces, and you have something "like that". Golang can be seen as an odyssey, because it can be an epic development to turn the environment into whatever you want it to be.

Remember, Golang is a "high level language" that provides a "system level" syntax that allows you to easily work with "low level" implementations.

Under the Go syntax, you can

[Struct] Define a struct by prefixing it with type, adding your "class/struct" name and then adding a suffix of struct.
[Encapsulation] Define the exposure of your class/structure related elements by starting them with UpperCase or LowerCase names.
- [Visibility: "public"]: Set the item name to uppercase.
- [Visibility: "private/protected"]: Set the item name in lower case.

And you can use it for: Structs, Struct Fields, Struct Methods. Take a closer look:

// -----------------------
// file: src/users/user.go
package users

type User struct {  
    name string  // starts with uppercase: public
    Age  uint8  // starts with uppercase: public
}  

// Starts with lowercase: private
func (u *User) getName() string {  
    return u.name  
}  

// Starts with uppercase: public
func (u *User) Greetings() string {  
    cheering := "Don't forget to follow me back!"

    // Can consume same struct functions.
    return fmt.Sprintf("Hey, %v (%v)! %v !", u.getName(), u.Age, cheering)
} 


// -----------------
// file: src/main.go
package main  

import "github.com/danielhe4rt/go-fodase/src/users"  

func main() {  
    // ❌ You CAN'T start 'name'. Use a Setter Function for it.    
    // user := users.User{    
    //     name: "danielhe4rt", // ❌
    //     Age:  25,            // ✅
    // }

    // ✅ Now you're only initializing what you need.    
    user := users.User{Age: 25}  

    // Methods Calls  
    user.SetName("danielhe4rt") // ✅
    user.getName()              // ❌
    user.Greetings()            // ✅

    currentName := user.name() // ❌ 
    currentAge := user.Age     // ✅ 
}

In Rust, you have an more explicit approach (more oop like languages) where:

[Struct] Define an struct using the prefix struct, adding your "Class/Struct" name and that's it.
[Encapsulation] If you want something to be public to other "crates", you should add the pub prefix in the part of the code you want to expose.

// file: src/users/mod.rs

// Make the struct `User` public
pub struct User {
    // Private field: accessible only within the `users` module
    name: String,
    // Public field: accessible from other modules
    pub age: u8,
}

impl User {
    // Public method to create a new User with a name and age
    pub fn new(age: u8) -> Self {
        User {
            name: String::new(), // Initialize with an empty name
            age,
        }
    }

    // Public setter method to set the private `name` field
    pub fn set_name(&mut self, name: String) {
        self.name = name;
    }

    // Private method to get the name
    fn get_name(&self) -> &str {
        &self.name
    }

    // Public method that uses both public and private fields/methods
    pub fn greetings(&self) -> String {
        let cheering = "Don't forget to follow me back!";
        format!(
            "Hey, {} ({} years old)! {}",
            self.get_name(),
            self.age,
            cheering
        )
    }
}

// file: src/main.rs
mod users;  

use crate::users::User;  

fn main() {
    // ❌ You CAN'T directly set the private `name` field
    // let user = User {
    //     name: String::from("danielhe4rt"), // ❌
    //     age: 25,                           // ✅
    // };

    // ✅ Initialize the User using the constructor
    let mut user = User::new(25);

    // ✅ Use the setter method to set the private `name`
    user.set_name(String::from("danielhe4rt"));

    let greeting = user.greetings();    // ✅ Call the public `greetings` method
    let current_age = user.age;         // ✅ Access the public `age` field directly
    let current_name = user.name;       // ❌ You CAN'T access the private `name` field directly
    let current_name = user.get_name(); // ❌ You CAN'T call the private `get_name` method directly
}

I'd like to make things explicit like PHP, Java and so on but if you stop to think is LESS CODE to write, but it also impacts the readability.

3.2 Syntax: Interface Implementation is WEIRD AS FUC*

To be really honest, I'm the kind of person who would try to put, I don't know... LARAVEL in Go Environment, but that was already done in Goravel. Anyway, I really like the idea of working with "Interface/Contract Driven Development", and for the first time I got stuck with it in a language.

In Go, interfaces aren't "implemented" in a structure/class, and for an OOP guy like me, it's just crazy to have such a design decision fit into my head. Have a look at what is expected:

interface OAuthContract {
    public function redirect(string $code); 
    public function authenticate(string $token);
}

class GithubOAuthProvider implements OAuthContract{
    public function redirect(string $code) {}
    public function authenticate(string $token) {}
}

class SpotifyAuthProvider implements OAuthContract{
    public function redirect(string $code) {}
    public function authenticate(string $token) {}
}

function authenticate(string $routeProvider): OAuthContract {
    return match($provider) {
        'github' => new GithubOAuthProvider(),
        'spotify' => new SpotifyAuthProvider(),
        default => throw OAuthProviderException::notFound(),
    };
}

authenticate('github');

Now, when it comes to go: you don't have this explicit implementation of an "interface" inside a structure, and that's, hmm... weird? Instead, you just implement the interface's required methods, which go will check for you at compile time. It's fair to know that this is a compiled language and it should never be a problem, but I'm talking about my perspective with Developer Experience!

import (
    "errors"
    "fmt"
)

type OAuthContract interface {
    Redirect(code string) error
    Authenticate(token string) error
}

// GithubOAuthProvider implements the OAuthContract interface for GitHub.
type GithubOAuthProvider struct{}

// Redirect handles the redirection logic for GitHub OAuth.
func (g *GithubOAuthProvider) Redirect(code string) error {
    // Implement GitHub-specific redirection logic here.
    return nil
}

// Authenticate handles the authentication logic for GitHub OAuth.
func (g *GithubOAuthProvider) Authenticate(token string) error {
    // Implement GitHub-specific authentication logic here.
    return nil
}

// SpotifyOAuthProvider implements the OAuthContract interface for Spotify.
type SpotifyOAuthProvider struct{}

// Redirect handles the redirection logic for Spotify OAuth.
func (s *SpotifyOAuthProvider) Redirect(code string) error {
    // Implement Spotify-specific redirection logic here.
    fmt.Printf("Redirecting to Spotify with code: %s\n", code)
    return nil
}

// Authenticate handles the authentication logic for Spotify OAuth.
func (s *SpotifyOAuthProvider) Authenticate(token string) error {
    // Implement Spotify-specific authentication logic here.
    fmt.Printf("Authenticating Spotify token: %s\n", token)
    return nil
}

// AuthenticateFactory is a factory function that returns an instance of OAuthContract
func AuthenticateFactory(provider string) (OAuthContract, error) {
    switch provider {
    case "github":
        return &GithubOAuthProvider{}, nil
    case "spotify":
        return &SpotifyOAuthProvider{}, nil
    default:
        return nil, &OAuthProviderError{Provider: provider}
    }
}

In any case, with some time coding in the language you will get used with it. Now, let's talk about what the base environment offers to you without download anything!

4. Stdlib: definitely an AWESOME toolkit

Now I'm talking about everything that Go serves you with the Standard Library, without download an third party package. Here's some chronological timeline for you:

1st day: WHAT? WHY NOT LIKE JS/Java IN THE MATTER THAT THE TYPE CARRIES ALL METHODS? (And I hate both of them)
1st week: Wait, maybe that's good shit (after understand the packages for primitive types)
2nd week: WHAT? WHY NOT OTHER LANGS HAVE SUCH A GOOD LIBRARIES BY DEFAULT?

I'm not joking about that, every day that I explore go I found some cool library under the standard ones. So, let's start talking about primitive types.

4.1 Packages: Primitive Types

Like PHP, and unlike many other languages (Rust, Java, JS, etc), Golang needs "helper" functions to perform most of the related type operations. We can think of them as "anemic" types, since they don't have "utility" attached to them.

// PHP Example (with a REALLY UGLY API)
// String manipulation using built-in PHP functions
$str = "Hello, World!";

// Check if the string contains a substring
$contains = strpos($str, "World") !== false;
echo "Contains 'World': " . ($contains ? "true" : "false") . "\n"; // Output: true

// Convert the string to uppercase
$upper = strtoupper($str);
echo "Uppercase: " . $upper . "\n"; // Output: HELLO, WORLD!

// Split the string by comma
$split = explode(",", $str);
echo "Split by comma: ";
print_r($split); // Output: Array ( [0] => Hello  [1] =>  World! )

So if you're working with a "String" type, you have other packages like strconv or strings to manipulate it! But here's a golden rule to never forget which package to look at: if your type is string, look for the same package with a pluralised name!

In a nutshell, this will give you functions related to []Type and Type:

String type -> import ("strings") for operations like: Contains(), Upper(), Split() ...
Bytes type -> import ("bytes") for operations like Include(), Compare(), Split() ...
and so on!

Take a look at the code, so you can validate by yourself:

package main

import (
    "bytes"
    "fmt"
    "strings"
)

func main() {
    // String manipulation using the "strings" package
    str := "Hello, World!"

    // Check if the string contains a substring
    contains := strings.Contains(str, "World")
    fmt.Printf("Contains 'World': %v\n", contains) // Output: true

    // Convert the string to uppercase
    upper := strings.ToUpper(str)
    fmt.Printf("Uppercase: %s\n", upper) // Output: HWith this intELLO, WORLD!

    // Split the string by comma
    split := strings.Split(str, ",")
    fmt.Printf("Split by comma: %v\n", split) // Output: [Hello  World!]
}

That's supposed to be simple, but I struggled with that for a while until gets into my head. Maybe using Laravel and their helper functions for too many years made me forget how tough is to code without a Framework :D

4.2 Packages: Useful Stuff

While I was exploring tools and projects, I got a really good introduction to many projects and I'd like to list each of them and the libs I've used:

Build your own Shell Challenge:
- packages:
  - fmt: I/O library (Scan/Read and Write stuff on your screen)
  - os: functions and helpers that talks directly with your Operational System.
  - strconv: cast specific data-types to string or cast string to any defined type.
Build your own (HTTP|DNS) Server Challenge:
- packages:
  - net: integration layer with network I/O protocols such as HTTP, TCP, UDP and Unix Domain Sockets
  - [previous packages...]
Mid Level Homework Task assignment?
- packages:
  - flag: Captures your CLI arguments into variables
  - log: Adds Log's channels to your application
  - crypto/rand: Secure Cryptographic Random Generator
  - math/rand: Math Numbers Random Generator
  - time: Duration/Time Lib

Here's a scrollable view of all the package implementations so you can check them out. There are PLENTY of cool std packages that can be cited here.

ATTENTION: that's a LOT OF CODE! :p
Don't forget to comment your favorite features (besides goroutines and channels) :p

package main

import (
    crypto "crypto/rand"
    "encoding/hex"
    "flag"
    "fmt"
    "log"
    "math/rand"
    "net/http"
    "os"
    "strconv"
    "time"
)

// =========================
// Example 1: fmt Package
// =========================

// fmtExample demonstrates basic input and output operations using the fmt package.
func fmtExample() {
    fmt.Println("=== fmt Package Example ===")
    var name string
    var age int

    // Prompt the user for their name
    fmt.Print("Enter your name: ")
    _, err := fmt.Scanln(&name)
    if err != nil {
        fmt.Println("Error reading name:", err)
        return
    }

    // Prompt the user for their age
    fmt.Print("Enter your age: ")
    _, err = fmt.Scanln(&age)
    if err != nil {
        fmt.Println("Error reading age:", err)
        return
    }

    // Display the collected information
    fmt.Printf("Hello, %s! You are %d years old.\n\n", name, age)besides
}

// =========================
// Example 2: os Package
// =========================

// osExample showcases how to interact with the operating system, such as reading environment variables and exiting the program.
func osExample() {
    fmt.Println("=== os Package Example ===")

    // Retrieve the HOME environment variable
    home := os.Getenv("HOME")
    if home == "" {
        fmt.Println("HOME environment variable not set.")
        // Exit the program with a non-zero status code to indicate an error
        // Uncomment the next line to enable exiting
        // os.Exit(1)
    } else {
        fmt.Printf("Your home directory is: %s\n", home)
    }

    // Demonstrate exiting the program successfully
    // Uncomment the next lines to enable exiting
    /*
        fmt.Println("Exiting program with status code 0.")
        os.Exit(0)
    */
    fmt.Println()
}

// =========================
// Example 3: strconv Package
// =========================

// strconvExample demonstrates converting between strings and other data types.
func strconvExample() {
    fmt.Println("=== strconv Package Example ===")

    // Convert string to integer
    numStr := "456"
    num, err := strconv.Atoi(numStr)
    if err != nil {
        fmt.Println("Error converting string to int:", err)
        return
    }
    fmt.Printf("Converted string '%s' to integer: %d\n", numStr, num)

    // Convert integer back to string
    newNumStr := strconv.Itoa(num)
    fmt.Printf("Converted integer %d back to string: '%s'\n\n", num, newNumStr)
}

// =========================
// Example 4: net Package
// =========================

// netExample demonstrates making a simple HTTP GET request using the net package.
func netExample() {
    fmt.Println("=== net Package Example ===")

    // Make an HTTP GET request to a public API
    resp, err := http.Get("https://api.github.com")
    if err != nil {
    let greeting = user.greetings();
        fmt.Println("Error making HTTP GET request:", err)
        return
    }
    defer resp.Body.Close()

    // Display the HTTP status code
    fmt.Printf("Response Status: %s\n\n", resp.Status)
}

// =========================
// Example 5: flag Package
// =========================

// flagExample demonstrates how to capture command-line arguments using the flag package.
func flagExample() {
    fmt.Println("=== flag Package Example ===")

    // Define command-line flags
    name := flag.String("name", "World", "a name to say hello to")
    age := flag.Int("age", 0, "your age")

    // Parse the flags
    flag.Parse()

    // Use the flag values
    fmt.Printf("Hello, %s!\n", *name)
    if *age > 0 {
        fmt.Printf("You are %d years old.\n\n", *age)
    } else {
        fmt.Println()
    }
}

// =========================
// Example 6: log Package
// =========================

// logExample demonstrates logging messages to a file using the log package.
func logExample() {
    fmt.Println("=== log Package Example ===")

    // Open or create a log file
    file, err := os.OpenFile("app.log", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
    if err != nil {
        log.Fatalf("Failed to open log file: %s", err)
    }
    defer file.Close()

    // Set the log output to the file
    log.SetOutput(file)

    // Log messages
    log.Println("Application started")
    log.Printf("Logging an event at %s", "some point")
    log.Println("Application finished")

    fmt.Println("Logs have been written to app.log\n")
}

// =========================
// Example 7: crypto/rand Package
// =========================

// cryptoRandExample demonstrates generating a secure random string using the crypto/rand package.
func cryptoRandExample() {
    fmt.Println("=== crypto/rand Package Example ===")

    // Generate a secure random string of 16 bytes (32 hex characters)
    randomStr, err := generateSecureRandomString(16)
    if err != nil {
        fmt.Println("Error generating secure random string:", err)
        return
    }
    fmt.Println("Secure Random String:", randomStr, "\n")
}

// generateSecureRandomString generates a hexadecimal string of the specified byte length using crypto/rand.
func generateSecureRandomString(length int) (string, error) {
    bytes := make([]byte, length)
    _, err := crypto.Read(bytes)
    if err != nil {
        return "", err
    }
    return hex.EncodeToString(bytes), nil
}

// =========================
// Example 8: math/rand Package
// =========================

// mathRandExample demonstrates generating pseudo-random numbers and shuffling a slice using the math/rand package.
func mathRandExample() {
    fmt.Println("=== math/rand Package Example ===")

    // Seed the random number generator to ensure different outputs each run
    rand.Seed(time.Now().UnixNano())

    // Generate a random integer between 0 and 99
    randomInt := rand.Intn(100)
    fmt.Printf("Random integer (0-99): %d\n", randomInt)

    // Generate a random float between 0.0 and 1.0
    randomFloat := rand.Float64()
    fmt.Printf("Random float (0.0-1.0): %f\n", randomFloat)

    // Shuffle a slice of integers
    nums := []int{1, 2, 3, 4, 5}
    rand.Shuffle(len(nums), func(i, j int) {
        nums[i], nums[j] = nums[j], nums[i]
    })
    fmt.Printf("Shuffled slice: %v\n\n", nums)
}

// =========================
// Example 9: time Package
// =========================

// timeExample demonstrates working with current time, formatting, parsing, sleeping, and measuring durations using the time package.
func timeExample() {
    fmt.Println("=== time Package Example ===")

    // Current time
    now := time.Now()
    fmt.Println("Current time:", now)

    // Formatting time
    formatted := now.Format("2006-01-02 15:04:05")
    fmt.Println("Formatted time:", formatted)

    // Parsing time from a string
    parsed, err := time.Parse("2006-01-02", "2024-10-25")
    if err != nil {
        fmt.Println("Error parsing time:", err)
        return
    }
    fmt.Println("Parsed time:", parsed)

    // Sleep for 2 seconds
    fmt.Println("Sleeping for 2 seconds...")
    time.Sleep(2 * time.Second)
    fmt.Println("Awake!")

    // Measuring duration
    start := time.Now()
    time.Sleep(1 * time.Second)
    elapsed := time.Since(start)
    fmt.Printf("Elapsed time: %s\n\n", elapsed)
}

// =========================
// Main Function
// =========================

func main() {
    // Execute each example function sequentially
    fmtExample()
    osExample()
    strconvExample()
    netExample()
    flagExample()
    logExample()
    cryptoRandExample()
    mathRandExample()
    timeExample()

    fmt.Println("All examples executed.")
}

Seriously, that's just amazing! So, lets keep going for tests now.

5. Tests in Go are THAT SIMPLE

In my second project using Go, I saw an opportunity to learn tests while creating Requests and Responses objects. Inside the PHP environment, you are probably using a 3rd party library like PHPUnit or Pest. Right? Inside the Go environment, this is EASY! All you need to do is:

Create a file inside a package: In person.go you will write the functions you want to test;
Create a test file for your package :** create a file called person_test.go and start writing your own tests!

Let's say we have requests.go and requests_test.go in our package folder, where requests.go is:

package request  

import (  
    "bytes"  
    "fmt")  

type VerbType string  

const (  
    VerbGet  VerbType = "GET"  
    VerbPost VerbType = "POST"  
)  

type Request struct {  
    Verb    VerbType  
    Version string  
    Path    string  
    Headers map[string]string  
    Params  map[string]string  
    Body    string  
}  

func NewRequest(payload []byte) Request {  

    payloadSlices := bytes.Split(payload, []byte("\r\n"))  

    verb, path, version := extractRequestLine(payloadSlices[0])  
    headers := extractHeaders(payloadSlices[1 : len(payloadSlices)-2])  
    body := payloadSlices[len(payloadSlices)-1]  
    req := Request{  
       Verb:    VerbType(verb),  
       Version: version,  
       Path:    path,  
       Headers: headers,  
       Params:  map[string]string{},  
       Body:    string(body),  
    }  
    return req  
}  

func extractHeaders(rawHeaders [][]byte) map[string]string {  
    headers := make(map[string]string)  

    for _, headerBytes := range rawHeaders {  
       fmt.Printf("%v\n", string(headerBytes))  
       data := bytes.SplitN(headerBytes, []byte(": "), 2)  

       key, value := string(data[0]), string(data[1]) // Accept (?)  
       headers[key] = value  
    }  

    return headers  
}  

func extractRequestLine(requestLine []byte) (string, string, string) {  

    splitRequestLine := bytes.Split(requestLine, []byte(" "))  
    verb := string(splitRequestLine[0])  
    path := string(splitRequestLine[1])  
    version := string(splitRequestLine[2])  
    return verb, path, version  
}

5.1 Tests: Basic Testing

A test in Go is considered PASSED (green) if (t *Testing.T).Errorf() is not called within your test function. It also follows the same concept of encapsulation introduced earlier:

Test Functions starting with uppercase are identified by the Test Runner
Test Functions starting with lowercase are ignored (usually helper functions)

package request

import (
    "reflect"
    "testing"
)

func TestNewRequest(t *testing.T) {
    // Prepare
    payload := []byte("GET /index.html HTTP/1.1\r\nHost: localhost:4221\r\nUser-Agent: curl/7.64.1\r\nAccept: */*\r\n\r\n")

    // Act
    got := NewRequest(payload);

    // Assert 
    want := Request{
        Verb:    VerbGet,
        Version: "HTTP/1.1",
        Path:    "/index.html",
        Headers: nil,
    }

    if !reflect.DeepEqual(got, want) {
        t.Errorf("FAILED! NewRequest() = %v, want %v", got, want)
    }
}

You can do your own helper functions to test. Just make sure to not trespass the module domain on these tests!

5.2 Tests: Jetbrains Boilerplate

I've been using Goland since day one, so most things have been easier for me. So every time I start a new test, I get this boilerplate with a Unity test structure that runs in parallel (goroutines) by default.

package request

import (
    "reflect"
    "testing"
)

func TestNewRequest(t *testing.T) {
    type args struct {
        payload []byte
    }
    tests := []struct {
        name string
        args args
        want Request
    }{
        {
            name: "Base Request Argless",
            args: args{
                payload: []byte("GET /index.html HTTP/1.1\r\nHost: localhost:4221\r\nUser-Agent: curl/7.64.1\r\nAccept: */*\r\n\r\n"),
            },
            want: Request{
                Verb:    VerbGet,
                Version: "HTTP/1.1",
                Path:    "/index.html",
                Headers: nil,
            },
        },
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            if got := NewRequest(tt.args.payload); !reflect.DeepEqual(got, tt.want) {
                t.Errorf("NewRequest() = %v, want %v", got, tt.want)
            }
        })
    }
}

5.3 Tests: Running Tests

Ok, now we know how easy it is to write tests in Go, but how about running them? Simple task! All you need to do is navigate to the package folder and run:

# ~/go_fodase
cd requests
ls
# requests.go request_test.go
go test # Run all tests inside the package in all files with suffix "_go"
# ...
# PASS
# ok      github.com/danielhe4rt/go_fodase/requests/request      0.001s

go test -run=TestNewRequest # Filter an specific test
# PASS
# ok      github.com/danielhe4rt/go_fodase/requests/request      0.001s

Please write down some tests for your stuff. It's not that hard if you decouple what's needed :p

6. Beware Cyclic Imports

During my last few years of development, I've always tried to modularise all my projects in a way that suits my needs, without getting stuck on "Clean Arch" or "Domain Driven Design" stuff. However, in my first attempts at splitting my packages, I got the "Cyclic Import" error and thought to myself: HOW LONG SINCE I'VE SEEN SOMETHING LIKE THAT?

During my 2 years in PHP, I had the same problem with import hell, where you couldn't not import the same file TWICE in a particular flow. This was before I met the PSR-4 (Autoloading) (which changed my PHP days forever!!) and now, years ago, I'm struggling with this in Go.

Let's consider a scenario of cyclic imports:

// ------------------
// A.php

require_once 'B.php';

class A {
    public function __construct() {
        $this->b = new B();
    }

    public function sayHello() {
        echo "Hello from A\n";
    }
}

// ------------------
// B.php

require_once 'A.php';

class B {
    public function __construct() {
        $this->a = new A();
    }

    public function sayHello() {
        echo "Hello from B\n";
    }
}
// ------------------
// index.php

require_once 'A.php';

$a = new A();
$a->sayHello();
// Fatal error: Uncaught Error: Maximum function nesting level of '256' reached...

When you try to compile something that flags Cyclic Imports in Go, you will receive an error like:

import cycle not allowed
package path/to/packageA
    imports path/to/packageB
    imports path/to/packageA

And in this moment, you have to start breaking down your dependencies/packages in order to avoid it.

TLDR: don't import the same package in a place that will be loaded many times.

7. Defer this, defer that... But, what is a Defer?

I didn't look it up, but it was the first time I'd seen the reserved word defer in a programming language. And since it was not part of the "generic reserved words", I ignored it for a whole week!

Then one of my work mates, Dusan, gave me a memory management lesson in Go after seeing me struggle with the language for a couple of hours. (Yes, this is a shout-out :p)

The thing is: whenever you open a buffer/connection to something, you SHOULD CLOSE IT! I remember when I was working with MapleStory servers (Java) in 2014, the most common problem was memory leaks, simply because the developers didn't close the connections to the DB.

This is OK to FORGET! But it's not OK to pass in the Code Review LOL

Here's an example in Java:

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.Statement;
import java.sql.SQLException;

public class UnclosedConnectionExample {
    public static void main(String[] args) {
        // Database credentials
        String url = "jdbc:mysql://localhost:3306/mydatabase";
        String user = "username";
        String password = "password";

        Connection conn = null;
        Statement stmt = null;
        ResultSet rs = null;

        while {
            try {
                // Establishing the connection
                conn = DriverManager.getConnection(url, user, password);
                System.out.println("Database connected successfully.");

                // Creating a statement
                stmt = conn.createStatement();

                // Executing a query
                String sql = "SELECT id, name, email FROM users";
                rs = stmt.executeQuery(sql);

                // Processing the result set
                while (rs.next()) {
                    int id = rs.getInt("id");
                    String name = rs.getString("name");
                    String email = rs.getString("email");

                    System.out.println("ID: " + id + ", Name: " + name + ", Email: " + email);
                }

                // Intentionally not closing the resources
                // rs.close() // <-------------------
                // This leads to resource leaks
            } catch (SQLException e) {
                // rs.close() // <-------------------
                e.printStackTrace();
            }
        }
    }
}

While coding Golang, they give this defer attribute for you to remember to close your stuff right after opening it.

Defer stands for "Deference" which is a way to "Clean" your resources after the execution of that specific portion of code ends.

import(
    "github.com/gocql/gocql"
)

func main() {
    cluster := gocql.NewCluster("localhost:9042")  
    session, err := cluster.CreateSession()  

    if err != nil {  
        log.Fatal("Connection Refused!")  
    }  
    defer session.Close() // After the function ends, the method will be called

    // Migrate stuff  
    fmt.Println("Migrating Tables...")
    database.Migrate(session)
    // ... rest of your code

    // After the last line of code, the function `session.Close()` will run without you trigger it. 
}

You can also have many defer's within a function and the DEFER ORDER matters! If you defer database2 and then defer database1, both processes will be cleaned in the same order.

import(
    "github.com/gocql/gocql"
)

func main() {
    cluster := gocql.NewCluster("localhost:9042")  

    loggingSession, _ := cluster.CreateSession()  
    appSession, _ := cluster.CreateSession()  

    defer appSession.Close()
    defer loggingSession.Close()

    // ... rest of your code
    // Defer Order
    // ...
    // appSession.close() 1st
    // loggingSession.close() 2nd
}

This is a really simple way to ~~not fuck up~~ prevent your project from having a memory leak. Please remember to use it whenever you stream anything.

8. Error Management for Noobs

Error handling at first will be something like: check if the function you're using returns an error type and validate it EVERY ~~FUCKING~~ TIME! Here's an example of what I'm talking about:

cluster := gocql.NewCluster("localhost:9042")  
appSession, err := cluster.CreateSession()  

// NIL STUFF
if err != nil {
    // do something
    log.Fatal("Well, something went wrong.")
}
defer appSession.close()
// Keep your flow. 
// ...

To be honest, I HATE this syntax. However, it's part of the language and will be something you'll come across during your days of coding.

Functions with errors can return error or (T, error), and fortunately Go will not let you forget that.

func SumNumbers(a, b int) (int, error) {
    result := a + b 
    if result < 0 {
        return nil, fmt.Errorf("The result is negative.")
    }

    return result, nil
}

// Using it
* always return when possible
    * spam the type error 
result, err := SumNumbers(-11, 5);

if err != nil {
    log.Fatal(err) // Ends the app or handle in the way you want.
}

Spam your code with err != nil and you will be fine, I promise! :D

9. Conclusion feat: Low Latency Coding Challenge

Aside from all the stress and hours spent trying to understand the environment, it was a cool challenge to learn a new language together with my Twitch viewers. Many of them have been asking me for a long time to check it out and here we are.

All of these points reflect my personal development experience with the language, and the goal was to share things I've gone through during these 2 weeks of studying it.

9.1 Bonus: Coding Challenge

Recently I was challenged by my teammate to complete a ScyllaDB challenge and it taught me a lot about parallelism, pools and rate limiting. This is the kind of challenge that many companies face to make their products perform better!

The goal of the challenge is to create a small Go command line application that inserts some random data into ScyllaDB while rate limiting the number of requests.

You can find the repository challenge here: github.com/basementdevs/throttling-requests-scylla-test. We're also hiring! You can find the open positions in our careers section!

Thank you for reading! I hope this article has provided valuable insights into learning Golang. Feel free to share your thoughts or experiences.

Database 101: SSL/TLS for beginners

Daniel Reis — Fri, 04 Oct 2024 15:33:01 +0000

Once again, I'm bringing some knowledge from my day-to-day work as a Developer Advocate at ScyllaDB, and this time I'm going to teach you about secure and encrypted connections ! It's a topic that can cause a lot of anxiety for newbies, but I'm going to make it easier for you.

If you're just getting started with databases in general or databases in particular, you might want to start by reading my first article, Database 101: Data Consistency for Beginners.

This article captures my own exploration of how many database paradigms exist as I look far beyond my previous experience with just SQL and MySQL. I'm keeping track of my studies in this Database 101 series.

1. Prologue
2. What is SSL and TLS?
3. OpenSSL for Noobs
4. Database Authentication
- 4.1 No Authentication
- 4.2 Username/Password Authentication
- 4.3 Creating Users and Roles
- 4.4 Certificated / Role Certificated Authentication
  - 4.4.1 Change the Authentication Type
  - 4.4.2 Enable the Encryption Port
  - 4.4.3 Enable the Client Encryption Port
5. Testing Encrypted Connections
6. Conclusion

1. Prologue

Working on a NoSQL database is challenging in the sense that I have to learn stuff that I'd NEVER touch or study as a regular web developer. I mean, mostly you build a CRUD application, try your best not to screw up the database indexes (right guys ??????) and paint a few buttons.

But if you've been following my Database 101 journey, you probably know that I came to Scylla knowing NOTHING about databases except what MySQL was and how to build things with Laravel, and now I'm into it:

Building Highly Scalable Applications with ScyllaDB
Still learning Rust - 1 year and 6 months (and I feel like I know shit about it).
Started learning ShellScript, Go, Python, JavaScript, and anything that has a ScyllaDB driver
Learned the basics of observability using Grafana/Prometheus
and my new friend Transport Layer Security (a.k.a. TLS)

If you're setting up your own servers, you've probably already used certbot to install certificates to run HTTPS smoothly, and that's what I've been doing for the last 6~7 years. To be really honest, knowing how to use a tool is what we developers do.

It even seems like an omen, but I'm working on a browser extension whose backend is written in Rust, and I didn't manage to use certbot there, so I had to create my certificate using weird commands and add a weird openssl crate to my project and trust that it would work:

let mut certs_file = BufReader::new(File::open(cert_path).unwrap());
let mut key_file = BufReader::new(File::open(key_path).unwrap());

// load TLS certs and key to create a self-signed temporary cert for testing:
// `openssl req -x509 -newkey rsa:4096 -nodes -keyout key.pem -out cert.pem -days 365 -subj '/CN=localhost'`
let tls_certs = rustls_pemfile::certs(&mut certs_file)
    .collect::<Result<Vec<_>, _>>()
    .unwrap();
let tls_key = rustls_pemfile::pkcs8_private_keys(&mut key_file)
    .next()
    .unwrap()
    .unwrap();

After a few hours of trying/catching, I finally managed to deploy the API with the certificate, but I didn't learn anything about it. THEN, a few days later, my boss asked me to extend the ScyllaDB Security Pages and learn how Certificated Based Connection works, since I'd never used it.

I spent 10 hours on it and managed to get it to work, but mostly it wasn't about the documentation, it was about my lack of knowledge in this area, which is no longer a problem and I will teach you everything I've learned so far!

2. What is SSL and TLS?

TLDR: this is that boring section which you can skip if you want.

SSL (Secure Sockets Layer) was introduced in 1995 by Netscape to provide privacy, authentication, and data integrity for internet communications, primarily in browsers. TLS (Transport Layer Security), which succeeded SSL, is now the standard security protocol used across the internet for applications like email, messaging, and VoIP.

Without SSL/TLS, data transmitted between a client and a server is unencrypted and vulnerable to interception and misuse, making it an easy target for phishing and other attacks. SSL/TLS ensures that a secure "handshake" occurs before data transmission, meaning that all data is encrypted for that specific session.

However, even though SSL/TLS provides encryption, it doesn’t guarantee that the server itself is fully trustworthy or secure. SSL, being deprecated since 1996 due to its vulnerabilities, is still often referenced alongside TLS by developers as "SSL/TLS" because of its historical significance.

With this understanding of SSL/TLS, let’s now dive into the tools that help manage secure connections.

3. OpenSSL for Noobs

There are many data encryption toolkits available today, but we're going to talk about a very specific one: the openssl-library.org.

This library, which had its first release in 1998, is attached to every single tutorial on data encryption, and is used by many other tools around the open/closed source environment - such as Let's Encrypt and Certbot - and the one we're going to use here.

The purpose of this tool is to create keychains and credentials to authorize someone to do whatever you want.

Imagine you work in a commercial building where you have to show your badge to security to get in:

Your company creates a "master key" and generates a certificate for it.
You also generate a key and certificate for the security guard and give them to him along with a list of all the employee's certificates.
The guard has a truststore keychain with all sorts of tags on it.
When you arrive at the entrance, he will check that your tag has the same root certificate as his to decide whether you can pass or not.

In matter of commands, we did something like this:

# Creating the root/master key and certificate
# -----
openssl genpkey -algorithm RSA -out root_key.pem -pkeyopt rsa_keygen_bits:2048 # Generate the root key
openssl req -x509 -new -key root_key.pem -days 3650 -out root_cert.pem -subj "/CN=administrator" # Generate the root certificate
# -----

# Creating the Employee Key and Certificate
# -----
openssl genpkey -algorithm RSA -out employee_key.pem -pkeyopt rsa_keygen_bits:2048 # Generate the employee base key
openssl req -new -key employee_key.pem -out employee.csr -subj "/CN=employee" # Generate the employee CSR
openssl x509 -req -in employee.csr -CA root_cert.pem -CAkey root_key.pem -CAcreateserial -out employee_cert.pem -days 365 # Sign and generate the employee certificate with the root/master key and certificate
cat root_cert.pem employee_cert.pem > employee_truststore.pem # Create a truststore (tag) to be used together with the key.
# -----

# Creating the Security (server) Key and Certificate
# -----
openssl genpkey -algorithm RSA -out security_key.pem -pkeyopt rsa_keygen_bits:2048 # Generate the security key
openssl req -new -key security_key.pem -out security.csr -subj "/CN=server" # Generate the security CSR
openssl x509 -req -in security.csr -CA root_cert.pem -CAkey root_key.pem -CAcreateserial -out security_cert.pem -days 365 # Sign and generate the security certificate with the root/master key and certificate
cat root_cert.pem security_cert.pem > security_truststore.pem # Create a truststore (tag) to be used together with the key.
# -----


# Creating the Malicious Root and Client Key and Certificate
# -----
openssl genpkey -algorithm RSA -out malicious_key.pem -pkeyopt rsa_keygen_bits:2048 # Generate the malicious root key
openssl req -x509 -new -key malicious_key.pem -days 3650 -out malicious_cert.pem -subj "/CN=fake-admin" # Generate the malicious root certificate
cat root_cert.pem malicious_cert.pem > malicious_truststore.pem # Combine root and malicious root certificates into a truststore
# -----

There are too many commands, but basically this is what happened:

A root_key.pem and root_cert.pem were generated.
These keys and certificates were used to sign other keys and certificates for employees:
- security_key.pem was signed by root_key.pem.
- employee_key.pem was signed by root_key.pem.
- server_key.pem has been signed by root_key.pem.
All these keys are in a file called security_truststore.pem.
Someone tried to create a fake key called malicious_key.pem using the root_cert.pem.

Now let's do some validation. Theoretically, only the keys in security_truststore.pem should be allowed to join:

# Validating the certificates with the truststores
# ----- 
openssl verify -CAfile security_truststore.pem employee_cert.pem # Check if the certificate is valid
# employee_cert.pem: OK

openssl verify -CAfile security_truststore.pem malicious_cert.pem # Check if the certificate is valid
# CN = fake-admin
#  error 18 at 0 depth lookup: self-signed certificate
#  error malicious_cert.pem: verification failed
# -----

And as expected, since the malicious key is not signed by the root user, authentication fails and that person is kicked out.

So now we know the basics of encryption. But where do you use it in the real world? Let me tell you more about my current task and then you will understand.

4. Database Authentication

Say you're running any database (e.g. MySQL, Postgres, ScyllaDB, Redis, etc.) in Docker. When you spin up these instances in a containerized environment, you usually don't need a password unless you set one. In any case, we have a number of options when we talk about authentication in any part of the modern web, and the same is true when we're talking about databases.

Were going to use ScyllaDB in this example however this flow is pretty common in many databases.

At this step, we're going to learn about:

No Authentication: Literally no credentials required, just spin your instance and be happy;
Username/Password Authentication: Some basic credentials are fine to keep people away from your data;
Certified / Role Certified Authentication: This is where the big companies like to play and you should learn that.

After this brief introduction of encryption 101, we can get started with the content itself.

4.1 No Authentication

In Scylla, if you're running a node or cluster in the development environment (locally or using Docker), there is no password by default. Also, ScyllaDB uses the CQL protocol on port 9042 by default, which can be easily changed in the config files.

Why am I telling you this port information? Because this information will be important later, trust me.

You can spin up a Docker instance by running it:

# Running a ScyllaDB Node with Docker
docker  run --name some_scylla -p 9042:9042 -p 9142:9142 -d scylladb/scylla:6.1.2 \
  --overprovisioned 1 --smp 1 

# Checking the node status -> Expect for UN (Up And Running)Cassandra
docker exec -it some_scylla nodetool status
# Datacenter: datacenter1
# =======================
# Status=Up/Down
# |/ State=Normal/Leaving/Joining/Moving
# -- Address   Load      Tokens Owns Host ID                              Rack 
# UN 10.10.5.2 509.46 KB 256    ?    9f597eb5-a77f-493f-9835-85dd1e571fcc rack1
# --

After that, you can login at cqlsh (CQL Shell) by running:

docker exec -it some_scylla cqlsh
# Connected to  at 10.10.5.5:9042
# [cqlsh 6.0.18 | Scylla 6.0.1-0.20240612.bc89aac9d017 | CQL spec 3.3.1 | Native protocol v4]
# Use HELP for help.
# cqlsh> select address, port, client_type, username from system.clients;
#
#  address   | port  | client_type | username
# -----------+-------+-------------+-----------
#  10.10.5.5 | 50854 |         cql | anonymous
#  10.10.5.5 | 50868 |         cql | anonymous
# -----------+-------+-------------+-----------

Yeah, we're connected as anonymous users! Perfect for developing new stuff and testing locally. But then you decide to deploy the same configuration into production...

And OF COURSE that, in less than an hour you'll be owned by some random database crawler LOL.

And you can read more about this ransomware if you want. Anyway, let's add some base credentials and configure it properly in the next step.

Seriously, don't open your database to the Internet with the default configurations or without binding a unique IP address to access it.

4.2 Username/Password Authentication

While running any of these other databases, you have the ability to switch authentication types by changing a bunch of configuration files. At ScyllaDB, we're really proud of how modular and easy the configuration is.

If you look at /etc/scylla/scylla.yaml in your docker (you can get a better look at the repository by clicking here (file: scylla.yaml#247), you will find these configuration flags:

# file: /etc/scylla/scylla.yaml
# ...
# ...
# Authentication backend, identifying users
# Out of the box, Scylla provides org.apache.cassandra.auth.{AllowAllAuthenticator,
# PasswordAuthenticator}.
#
# - AllowAllAuthenticator performs no checks - set it to disable authentication.
# - PasswordAuthenticator relies on username/password pairs to authenticate
#   users. It keeps usernames and hashed passwords in system_auth.credentials table.
#   Please increase system_auth keyspace replication factor if you use this authenticator.
# - com.scylladb.auth.TransitionalAuthenticator requires username/password pair
#   to authenticate in the same manner as PasswordAuthenticator, but improper credentials
#   result in being logged in as an anonymous user. Use for upgrading clusters' auth.
# authenticator: AllowAllAuthenticator
# ...

The default is the AllowAllAuthenticator which allows anonymous connection with superpowers. Now, we'll uncomment the # authenticator: line and replace with PasswordAuthenticator:

# file: /etc/scylla/scylla.yaml
# ...
-#authenticator: AllowAllAuthenticator  # line ~247
+authenticator: PasswordAuthenticator

Okay! Now we just need to tell our Scylla node that we have made this change. Since this is a "critical" change to our system environment, we'll need to drain and then restart the node. Here's a step-by-step guide:

# Before Editing
docker exec -it some_scylla nodetool drain # Stop gossiping and preparing to shut down
docker exec -it some_scylla supervisorctl stop scylla # Stop ScyllaDB
docker exec -it some_scylla cat /etc/scylla/scylla.yaml | grep authenticator: # Check the current flag

# Edit /etc/scylla/scylla.yaml ...
docker exec -it some_scylla sed -i 's/# authenticator:.*/authenticator: PasswordAuthenticator/' /etc/scylla/scylla.yaml

# After Edit
docker exec -it some_scylla cat /etc/scylla/scylla.yaml | grep authenticator: # Check if the update is there
docker exec -it some_scylla supervisorctl start scylla # Run ScyllaDB

With that we're finally able to run cqlsh and receives a huge "YOU NEED CREDENTIALS, BITCH":

docker exec -it some_scylla cqlsh
Connection error: ('Unable to connect to any servers', {'10.10.5.5:9042': AuthenticationFailed('Remote end requires authentication')})

and by default, at ScyllaDB, we can login using the username/password "cassandra" (superuser):

docker exec -it some_scylla cqlsh -u cassandra -p cassandra
# Connected to  at 10.10.5.5:9042
# [cqlsh 6.0.18 | Scylla 6.0.1-0.20240612.bc89aac9d017 | CQL spec 3.3.1 | Native protocol v4]
# Use HELP for help.
# cassandra@cqlsh> select address, port, client_type, username from system.clients; 

#  address   | port  | client_type | username
# -----------+-------+-------------+-----------
#  10.10.5.5 | 56998 |         cql | cassandra
#  10.10.5.5 | 57014 |         cql | cassandra
# -----------+-------+-------------+-----------

... with that, we know the basics of how to configure authentication in ScyllaDB, and if you want to know why the user/password is "cassandra", you can read my first article from this series. Also, be sure to change your default credentials before deploying it.

The CQL protocol runs on port 9042 by default and is well known. In the same way that you can get owned by just letting your database run without authentication, you can get owned by letting the default credentials there.

At the moment, all connections still unencrypted but at least we have a minimum security, but don't worry! We're about fix the encryption soon! Before that, let's create some users/roles inside our database.

4.3 Creating Users and Roles

Before creating new users for this database, we need to understand how it works. At Scylla we use Role Based Authentication every day. So every user is placed in system.roles as a role.

This is not a problem because you can set a PASSWORD to a role. It looks weird, but I'll show you how it works. Check it out:

scylladb@cqlsh> desc system.roles;

CREATE TABLE system.roles (
    role text,
    can_login boolean,
    is_superuser boolean,
    member_of set<text>,
    salted_hash text,
    PRIMARY KEY (role)
);

scylladb@cqlsh> select * from system.roles;

 role        | can_login | is_superuser |     member_of | salted_hash
-------------+-----------+--------------+---------------+--------------------
 scylladb    |      True |         True |          null | $6$rAJ6FflUo8Chf...
 employee    |     False |        False |          null | null
 danielhe4rt |      True |        False |  {'employee'} | $6$AF4F6CflAA8Cw...
-------------+-----------+--------------+---------------+--------------------

This output gives us some information about the authentication itself, such as

A role becomes an authenticatable when the can_login flag is set to true;
A role must/can have a password after it becomes an authenticatable;
A role can belong to another role, which will inherit all privileges from it.

This is just an environment created to give you an example of how the modeling and features work. Be sure to read the documentation.

In a matter of DCL (Data Control Language) and DML (Data Manipulation Language), we'll do basic commands like:

-- Create our user/roles
CREATE ROLE developer;
CREATE ROLE danielhe4rt WITH PASSWORD = 'some_cool_password' AND LOGIN = true;

-- Grant roles to our users
GRANT developer TO danielhe4rt;

... which we will be running at our ScyllaDB Docker Instance following the keys created in the beginning of the article:

docker exec -it some-scylla cqlsh -u cassandra -p cassandra -e "CREATE ROLE IF NOT EXISTS 'employee' WITH LOGIN = true;";
docker exec -it some-scylla cqlsh -u cassandra -p cassandra -e "CREATE ROLE IF NOT EXISTS 'server' WITH LOGIN = true;";

Note: You can only create new roles/users using an authenticated account

4.4 Certificated / Role Certificated Authentication

At the beginning of this tutorial we introduced the topic by talking about a story of someone with valid keys to enter a building and now the goal is to understand what is different from a regular authentication. As I said, without TLS/SSL, your data is in transit without any encryption, and that's BAD depending on the content you're sending/receiving.

Using a certificate means that before you send your data out, we have guarantees that if someone intercepts the data, it won't be a problem if they don't have the key to decrypt it.

By default, ScyllaDB listens to the CQL protocol on port 9042, which can be configured using the native_transport_port configuration option.
Scylla also supports the CQL protocol via TLS/SSL encryption, which is disabled by default and can be enabled using the native_transport_port_ssl configuration option.

The traditional choice of port for secure connections is 9142, but if client_encryption_options is specified and native_transport_port_ssl is not, then native_transport_port will only handle encrypted connections. The same thing happens if native_transport_port and native_transport_port_ssl are set to the same value.

I know, it seems a bit crazy with all these flags and options, but I'll try to make it simpler. Check out the rules that govern port assignment/encryption, which are summarized in the table below:

np  := native_transport_port is set
nps := native_transport_port_ssl is set
ceo := client_encryption_options are enabled
eq  := native_transport_port_ssl == native_transport_port

+-----+-----+-----+-----+
|  np | nps | ceo |  eq |
+-----+-----+-----+-----+
|  0  |  0  |  0  |  *  |   =>   listen on native_transport_port, unencrypted
|  0  |  0  |  1  |  *  |   =>   listen on native_transport_port, encrypted
|  0  |  1  |  0  |  *  |   =>   don't listen
|  0  |  1  |  1  |  *  |   =>   listen on native_transport_port_ssl, encrypted
|  1  |  0  |  0  |  *  |   =>   listen on native_transport_port, unencrypted
|  1  |  0  |  1  |  *  |   =>   listen on native_transport_port, encrypted
|  1  |  1  |  0  |  *  |   =>   listen on native_transport_port, unencrypted
|  1  |  1  |  1  |  0  |   =>   listen on native_transport_port, unencrypted + native_transport_port_ssl, encrypted
|  1  |  1  |  1  |  1  |   =>   listen on native_transport_port(_ssl - same thing), encrypted
+-----+-----+-----+-----+

// More at: https://github.com/scylladb/scylladb/blob/master/docs/dev/protocols.md#cql-client-protocol

This tells us that if we want to enable encryption, we need to update a few more things in scylla.yaml. Instead of using the PasswordAuthenticator, we'll switch to the com.scylladb.auth.CertificateAuthenticator feature.

Before we change anything, let's drain and stop our cluster:

docker exec -it some_scylla nodetool drain # Stop gossiping and preparing to shut down
docker exec -it some_scylla supervisorctl stop scylla # Stop ScyllaDB

In the previous step we created some roles with LOGIN enabled. And we added these "roles" to our OpenSSL commands at the top with a CN=server/blabla, right? Now it's time to make it useful and not use credentials anymore.

Private Keys + Certs can hold more information, like raw strings that can be used after the handshake is done. In this case, we have stored the user/role that will be used to log in inside our keys. With this, we can do a pattern match with the certificate content and check if there's any CN=something that matches our select * from system.roles where role = 'something'.

Let's start the modifications by entering our ScyllaDB instance:

# Enter the ScyllaDB Instance
docker exec -it some_scylla shell

# root@c531i213hu:/#
# Install any editor of your choice
apt install nano

# Enter the ScyllaDB Config File
nano -l /etc/scylla/scylla.yaml

With your config file opened, let's work on the modifications needed.

4.4.1 Change the Authentication Type

This is the last time we'll open this file, I promise! The idea here is to change the authentication type from PasswordAuthenticator to com.scylladb.auth.CertificateAuthenticator. We're also going to set a rule that will extract the CN= flag from each certificate and use the content for authentication purposes:

# file: /etc/scylla/scylla.yaml

-authenticator: PasswordAuthenticator # line ~247
+authenticator: com.scylladb.auth.CertificateAuthenticator
+auth_certificate_role_queries:
+  - source: SUBJECT
+    query: CN=([^,\s]+)

4.4.2 Enable the Encryption Port

We still haven't enabled TLS/SSL even after setting the authenticator. We still need to uncomment the native_transport_port_ssl to get port 9142 (which is used to transport our encrypted data) into the game. So, back to the scylla.yaml, lets change it:

# file: /etc/scylla/scylla.yaml
# ...
-# native_transport_port_ssl: 9142 # <- line ~131
+native_transport_port_ssl: 9142

4.4.3 Enable the Client Encryption Port

Step by step, we'll make it! Our final change is to the client_encryption_options as defined in the table above. We need to uncomment everything and make sure everything matches in these configurations. Here's a brief explanation of each configuration:

enabled: enables TLS/SSL encryption -> change to true (default: false)
certificate: absolute path to your server certificate (security_cert.pem / server_cert.pem)
keyfile: absolute path your server key signed by the root_key (security_key.pem / server_key.pem)
truststore: absolute path your server truststore containing server + root certificates (security_truststore.pem / server_truststore.pem)
require_client_auth: your server must receive a certificate to authenticate -> change to true

So, let's work on these changes:

# file: /etc/scylla/scylla.yaml
# ...
# enable or disable client/server encryption.
-# client_encryption_options:
-#   enabled: true
-#   certificate: /etc/scylla/certs/cert.pem
-#   keyfile: /etc/scylla/certs/key.pem
-#   truststore: /etc/scylla/certs/truststore.pem
-#   require_client_auth: true
+client_encryption_options:
+  enabled: true
+  certificate: /etc/scylla/certs/server_cert.pem
+  keyfile: /etc/scylla/certs/server_key.pem
+  truststore: /etc/scylla/certs/server_truststore.pem
+  require_client_auth: true

Ok! Now we're good to go. Let's turn on our ScyllaDB cluster by running:

docker exec -it some_scylla supervisorctl start scylla # Start ScyllaDB

Let's see if our port 9142 is running and listening for TLS/SSL connections:

openssl s_client -connect localhost:9042 
# CONNECTED(00000003)
# 40873B60D2750000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../ssl/record/ssl3_record.c:354:
# no peer certificate available
# No client certificate CA names sent

openssl s_client -connect localhost:9142
# CONNECTED(00000003)
# Can't use SSL_get_servername
# depth=0 CN = server
# verify error:num=20:unable to get local issuer certificate
# verify return:1
# depth=0 CN = server
# verify error:num=21:unable to verify the first certificate
# verify return:1
# depth=0 CN = server
# verify return:1
# ---
# Certificate chain
# 0 s:CN = server
#   i:CN = administrator
#   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
#   v:NotBefore: Aug  1 20:09:38 2024 GMT; NotAfter: Aug  1 20:09:38 2025 GMT
# ---
# YAY IT'S ASKING FOR CERTIFICATES!!!

At this point, we are 100% sure that TLS is enabled and requesting the certificates that match our previously generated keys.

5. Testing Encrypted Connections

Too much stuff to configure, but how do we connect to it? Instead of using CQLSH (because it's a pain in the ass to set it up and I LITERALLY PREFERED TO MAKE IT WORK on Node than explain how to use it), we're going to use NodeJS for the sake of simplicity.

First, let's quick setup our driver by running:

npm install @lambda-group/scylladb

After that, you can already create a demo script pointing your nodes and don't forget to switch the ports at the connection string:

import { Cluster } from "@lambda-group/scylladb";

const cluster = new Cluster({  
    nodes: ["127.0.0.1:9142"],
    ssl: {  
        enabled: true,  
        truststoreFilepath: "/your/path/to/certificates/developer_cert.pem",  
        privateKeyFilepath: "/your/path/to/certificates/developer_key.pem",  
        caFilepath: "/your/path/to/certificates/developer_truststore.pem",  
        verifyMode: VerifyMode.Peer,  
    }});
let result = await session.execute(
    "SELECT address, port, username, driver_name, driver_version FROM system.clients"
    );

console.log(result)
// [
//  {
//     address: '127.0.0.1',
//     driver_name: 'scylla-js-driver',
//     driver_version: '0.0.1',
//     port: 58846,
//     username: 'developer' // We're logged in as the role 'developer'
//  }
// ]

As the output tells us, we managed to connect by only sending our key and certificate to the server!

6. Conclusion

This "article/tutorial" took me a long time to write for several reasons, but one of them was to make sure of the content by going through it a few times.

I also made a demo (gh: danielhe4rt/scylladb-role-tls-auth) where you can run a makefile command and set up all these steps. So if you got here somehow, please don't forget to drop a star :D

Security is one of the topics that people are starting to pay more attention to nowadays and learning more about it was very exciting. Anyway, let me know what topics you would like to see in this series!

Stay safe and don't forget to drink water!

Laravel inside Rust? I have a reason for that.

Daniel Reis — Mon, 26 Feb 2024 14:22:06 +0000

What's up everyone!

I've been study Rust for the last couple of months and always trying to get closer of my actual stack which is around Laravel environment.

For me, as a PHP Developer, jump into Rust is a truly hard task since I never went to Functional Programming before, however I found a way to make this "easier".

Rust AFAIK

The whole ecosystem is made up of different packages and you should add it as you need it, unlike Laravel which has a really robust environment around the framework.

So my first impression was that I had to learn how to use the base packages:

Dotenvy
Chrono
Time
Uuid
and so on.

But when it comes to Web Development (API mostly) you have to choose an framework among many which is already there. Like:

Actix (currently using)
Axum (evaluating)
Rocket (people always tell me to not use it, still don't know why)

and none of them has the opinionated "structure" of Laravel, so I decided to create mine.

Laravel inside Rust

Why am I talking about Laravel? Because the "MVC" structure it gives us is elegant enough for small projects with Rust Web.

Things started to make sense when I coded Rust using the opinionated Laravel framework. Here's a structure of what I'm talking about:

./src
├── app.rs
├── config.rs
├── http
│   ├── controllers
│   │   ├── leaderboard_controller.rs
│   │   ├── mod.rs
│   │   └── submissions_controller.rs
│   ├── mod.rs
│   └── requests
│       ├── leaderboard_request.rs
│       ├── mod.rs
│       └── submission_request.rs
├── main.rs
├── models
│   ├── leaderboard.rs
│   ├── mod.rs
│   └── submission.rs
└── repositories
    ├── leaderboard_repository.rs
    ├── mod.rs
    └── submission_repository.rs

This project is currently using:

and you can check my mess in this Pull Request

The question is: would be a good thing to follow this thoughts while working with Rust? The simplicity and good structure for maintaining is what I'm always looking in a project.

Also I'm thinking to write some series on how is my process migrating from PHP to Rust, would interest you read something like that?

Hope to see your thoughts here and thanks!

What we've achieved with 1 Month of Basement Devs

Daniel Reis — Mon, 19 Feb 2024 15:29:22 +0000

Running a community is a tough job, but after a few years of doing it, it becomes an easy task. This is the first month of our community and I want to show you what we have accomplished so far.

If you're planning to run a community or even join one, this series will show you how to measure the health of the community using Basement Developers as an example.

1. General Achievements
2. Demographics
- 2.1 New Members
- 2.2 Members Activation
- 2.3 Member Visits & Comunicators
3. Events
4. Partnerships
5. Published Content
6. What now?

1. General Achievements

First of all, I'd like to thank a few readers who joined the community. Now we have a couple of Polish people who engage there on a daily basis.

FYI: the community is looking for members like you, so join us ASAP! :D

Peak of 71 members in our latest weekly meeting.

Since we started the community, our weekly meetings have been the place to discuss new opportunities, projects, and other opportunities down there, and to listen to what the members have to say.

Your browser does not support the video tag.

Speaking of community feedback, most of our members are Brazilians who want to get used to English, so the first project made in the community was an overlay to translate everything we say in our weekly meetings from English to Brazilian-Portuguese! Cheers to Pantotone, the creator of this amazing tool!

Also, since we're starting to create projects, a new GitHub organization has been created to host all projects related to our community. Don't forget to follow our community to get the latest project updates!

2. Demographics

When we talk about growth, we talk about numbers! Here's a recap from January 14 to February 12.

2.1 New Members

Since January 15th we have doubled the number of members! Currently we have 2197 active members on the server and many of them are practicing English every day in voice channels (in groups or one-on-one) and text channels of different styles.

2.2 Members Activation

One of the metrics we'll be working on this month is the "activation" of a new member, which is when they understand what the community is about and why it's a good fit for them.

We did great in the first month and you can see the graph going straight to the bottom, but that's just because this last week we had Carnaval here in Brazil, so it was expected and our Brazilian developers deserve this amazing event and a good vacation.

2.3 Member Visits & Comunicators

Our average number of communicators or engagement is 8%. Our goal is to get to 15% by the end of March. How are we going to do that? I don't know yet, but with this list of weekly events, it should be a good bet.

3. Events

Events are at the heart of our community, and last month, 16 pre-scheduled events were held, divided into 5 models:

Basement Meeting: Our weekly meeting held every Tuesday at 07:00 PM BRT. Its purpose is to share the plans of our community, announce new partnerships, update members on news, and of course, practice English.
Practicing with Renato: Every Wednesday, our community gathers to practice English with Renato. In this practice session, our members are given a topic, and each one engages in a conversation with Renato about it, always introducing new words to expand the community's vocabulary.
Traveling with JP: In this session, our members will plan their dream trip together with JP, discussing costs, places to visit, local food, and much more! Come and plan your trip with us.
Practice with Leonardo: Every Thursday morning, our community participates in an English practice session with Leonardo. Participants are assigned a topic and engage in conversation with Leonardo, incorporating new words to build vocabulary.
Coding Dojo: In early February, we launched the Basement Coding Dojo, an event where community members take on roles in a development team during a simulated sprint. Our first Dojo Dev Team is made up of Gabriel, Lorena, Lucas, Pedro, Renato and Yuri. Over the course of a month, the team aims to build a simple but complete application, working on design, backend and frontend, attending weekly meetings and interacting with an open-source project. You can find more information about this event in this repository.

4. Partnerships

Right now our biggest goal is to find HR companies or even companies that are actively recruiting developers.

For this month, our goal is to build a demographic with more information about things that will be interesting to bring these opportunities to our members, e.g: Seniority, Stack and others. So far we have some results, but nothing that's meaningful yet, since it's a new approach on our side.

If you have an open position in your company, please email me at danielhe4rt@gmail.com and I'll forward it to our best members to apply!

5. Published Content

During the last month, members of our community have received several recognitions for the articles produced on this platform. We would like to highlight the following articles, which were among the most relevant articles of their posting weeks (Top 7).

Database 101: How to Model Leaderboards for 1M Player's Game.

Daniel Reis for ScyllaDB ・ Jan 29 '24

#scylladb #database #yarg #tutorial

"In the latest installment of his "Database 101" series, @danielhe4rt explores the intricate art of modeling leaderboards for games with a million players. Drawing on personal experiences with the open-source rhythm game YARG and leveraging ScyllaDB's efficiency, the article delves into Query Driven Data Modeling and wide-column database concepts, providing a comprehensive guide for crafting performative leaderboards."

Everything you need to know about GIT

Renato Teixeira for Basement Developers ・ Feb 5 '24

#beginners #tutorial #learning #git

"Discover the world of code versioning with GIT through this comprehensive beginner's guide! Authored by first-time DEV contributor @reenatoteixeira, this breakout article meticulously unpacks all the essentials of GIT, ensuring a smooth learning journey. With crystal-clear explanations and detailed step-by-step instructions, you'll master the art of GIT, empowering you with seamless version control capabilities."

Staircase detail

Klecianny Melo ・ Feb 16 '24

#tutorial #programming #beginners #learning

Discover how to build staircases in programming for beginners with this fantastic step-by-step tutorial created by our contributor @kebcm! A staircase is a visual representation formed by a series of steps, each represented by the # character. Explore how to create a function that prints a custom-sized staircase.

Be a better developer with these Git good practices

Anthony Vinicius for Basement Developers ・ Feb 16 '24

#git #github #webdev #beginners

If you're a developer, you likely use Git regularly. It's vital for app development, whether solo or in a team. But many struggle with messy repositories and unclear commit messages / branch names. Learning Git well and sticking to best practices is one of the keys for career growth.

6. What now?

So far our community is growing in a way we didn't expect, but still in a healthy way. Many people who have never tried Technical English before have taken their first steps to practice and have begun to glimpse new possibilities, and that's the fuel we need.

If you want to be a part of this transformative journey, please join our Discord Server and don't forget to like and comment on this article!

Any tip to make our community better is welcome, ok? Stay hydrate and see you next month!

Btw, cheers to Renato and Anthony for helping me on this report :D

Database 101: How to Model Leaderboards for 1M Player's Game.

Daniel Reis — Mon, 29 Jan 2024 17:34:16 +0000

Ever wondered how a game like League of Legends, Fortnite, or even Rockband models its leaderboards? In this article, we'll find out how to properly model a schema to handle them in a monstrously performative way!

If you’re just getting started with databases in general or databases, you might want to start off by reading my initial post, Database 101: Data Consistency for Beginners for Beginners. That article captures my own exploration of how many database paradigms exist as I look far beyond my previous experience with just SQL and MySQL. I’m keeping track of my studies in this Database 101 series.

It has been almost a year since I published my first article in this series! Thank you for being here with me as I learn this subject. Your comments and thoughts are always very helpful!

1. Prologue

Ever since I was a kid, like most average developers, I've been fascinated by games and how they're made. Speaking of which, let me introduce you to my favorite childhood game: Guitar Hero 3: Legends of Rock.

Well, more than a decade later, I decided to try to contribute with some games in the open source environment, like rust-ro (Rust Ragnarok Emulator) and also the main character of this article: the YARG (Yet Another Rhythm Game).

YARG is literally another rhythm game, but the difference about this project is that it is completely open source and they united legendary contributors in game development and also design to make this project works.

Suddenly the game was being picked up and played mostly by Guitar Hero/Rockband streamers on Twitch, and I thought: well, it's an open source project, so maybe I can use my database skills for something like creating a monstrously fast leaderboard or store the past games.

It started as a simple chat on their Discord, which turned into a long discussion about how to make this project grow faster.

Then I decided to talk to my boss and ask him if I could work with the YARG guys and the condition was to create something cool enough to implement ScyllaDB (NoSQL Wide-column Database) since I'm working as a Developer Advocate there. You won't believe how the simplicity and scalability brought by ScyllaDB perfectly fit the needs of YARG.in!

Anyway, talk is cheap. Let me show you some code and concepts!

2. QDD - Query Driven Data Modeling

When we're talking about develop with NoSQL, mostly we should understand that depending on the paradigm (document, graph, wide-column etc) you should first understand which query you want to run.

While in MySQL the main goal is to understand the consistency, in Scylla you should focus on the query and create your schema based on that query.

At this project, we will handle two types of paradigm, which are:

Key-Value
Wide Column (Clusterization)

Now let's talk about the queries/features of our modeling.

2.1 Feature: Storing the matches

Every time that you finish a YARG gameplay, the most interest thing would be submitting your scores together with many other in-game metrics.

Basically it will be a single query based in a main index and that's all.

SELECT score, stars, missed_notes, instrument, ...  
FROM leaderboard.submisisons  
WHERE submission_id = 'some-uuid-here-omg'

2.2 Feature: Leaderboard

And now our main goal: a super cool leaderboard that you don't need to care about after a good data modeling. The leaderboard is per song, so every time you play a specific song, your best score will be saved and ranked.

However there's a big point on this interface, which is having filters to know exactly "which" leaderboard to bring:

song_id: required
instrument: required
modifiers: required
difficulty: required
player_id: optional
score: optional

So imagine our query looks like this, and it returns the results sorted by score in descending order:

SELECT 
    player_id, score, ...
FROM 
    leaderboard.song_leaderboard 
WHERE 
    instrument = 'guitar' AND
    difficulty = 'expert' AND
    modifiers = {'none'} AND
    track_id = 'dani-california' 
LIMIT 
    100;

--   player_id  | score
----------------+-------
--        tzach | 12000
--  danielhe4rt | 10000
--     kadoodle |  9999
----------------+-------

Now we know the features that will be worked here, but can you already imagine how will be the final schema?

No? Ok, lemme help you on that!

3. Data Modeling time!

Time to take a deep dive into data modeling with ScyllaDB and better understand how to scale it.

3.1 - Matches Modeling

First, let us understand a little more about the game itself:

It's a rhythm game;
You play a certain song at a time;
You can activate "modifiers" to make your life easier or harder before the game;
You have to choose an instrument (e.g. guitar, drums, bass and microphone).
Every aspect of the gameplay is tracked, such as:
- Score;
- Missed notes;
- Overdrive count;
- Play speed (1.5x ~ 1.0x);
- Date/time of gameplay;
- And other cool stuff.

Thinking on that, we can easily start our data modeling, which will turn in something like this:

CREATE TABLE IF NOT EXISTS leaderboard.submissions (
    submission_id uuid,
    track_id text,
    player_id text,
    modifiers frozen<set<text>>,
    score int,
    difficulty text,
    instrument text,
    stars int,
    accuracy_percentage float,
    missed_count int,
    ghost_notes_count int,
    max_combo_count int,
    overdrive_count int,
    speed int,
    played_at timestamp,
    PRIMARY KEY (submission_id, played_at)
);

Let's skip all the int/text values and jump to the set<text>.

The set type allows you to store a list of items of a particular type. I decided to use this list to store the modifiers because it's a perfect fit. Look at how the queries are executed:

INSERT INTO leaderboard.submissions (
    submission_id, 
    track_id,
    modifiers, 
    played_at
) VALUES (
    some-cool-uuid-here,
    'starlight-muse'
    {'all-taps', 'hell-mode', 'no-hopos'},
    '2024-01-01 00:00:00'
);

With this type, you can easily store a list of items to retrieve later.

Another cool piece of information is that this query is a key-value like! What does that mean?

Since you will always query it by the submission_id only, it can be categorized as a key-value.

3.2 Leaderboard Modeling

This is the part of the article where you will learn some cool wide-column database concepts.

On our leaderboard query, as mentioned earlier, we will always need some dynamic values in the WHERE clauses, which means that these values will belong to the Partition Key while the Clustering Keys will have value that can be "optional".

A partition key is a hash based on a combination of fields that you added to identify a value. Did you get it? No? Well, it took me a while to understand that too, but let me show you something:

Let's imagine that you played Starlight - Muse 100x times. If you would query this information, would be 100x different results differentiated by Clustering Keys like score or player_id.

SELECT 
    player_id, score ---
FROM 
    leaderboard.song_leaderboard 
WHERE 
    track_id = 'starlight-muse' 
LIMIT 
    100;

If 1.000.000 players play this song, your query will become slow and it will become a problem in the future, because your partition key consists of only one field, which is track_id.

However, if you add more fields to your Partition Key, like mandatory things before playing the game, maybe we can shrink these possibilities for a faster query. Now you see the big picture? Adding the fields like: Instrument, Difficulty and Modifiers will give you a way to split the information about that specific track evenly.

Let's imagine with some simple numbers:


-- Query Partition ID: '1'
SELECT 
    player_id, score, ...
FROM 
    leaderboard.song_leaderboard 
WHERE 
    instrument = 'guitar' AND
    difficulty = 'expert' AND 
    modifiers = {'none'} AND -- Modifiers Changed
    track_id = 'starlight-muse' 
LIMIT 
    100;

-- Query Partition ID: '2'
SELECT 
    player_id, score, ...
FROM 
    leaderboard.song_leaderboard 
WHERE 
    instrument = 'guitar' AND
    difficulty = 'expert' AND
    modifiers = {'all-hopos'} AND -- Modifiers Changed
    track_id = 'starlight-muse' 
LIMIT 
    100;

So, if you build the query in a specific shape it will always look for a specific token and retrieve the data based on these specifics Partition Keys.

Let's take a look at the final modeling and talk about the clustering keys and the application layer:

CREATE TABLE IF NOT EXISTS leaderboard.song_leaderboard (
    submission_id uuid,
    track_id text,
    player_id text,
    modifiers frozen<set<text>>,
    score int,
    difficulty text,
    instrument text,
    stars int,
    accuracy_percentage float,
    missed_count int,
    ghost_notes_count int,
    max_combo_count int,
    overdrive_count int,
    speed int,
    played_at timestamp,
    PRIMARY KEY ((track_id, modifiers, difficulty, instrument), score, player_id)
) WITH CLUSTERING ORDER BY (score DESC, player_id ASC);

The partition key was defined as mentioned above, consisting of our REQUIRED PARAMETERS such as: track_id, modifiers, difficulty and instrument. And on the Clustering Keys we added score and player_id.

Note that by default the clustering fields are ordered by score DESC and just in case a player has the same score, the criteria to choose the winner will be alphabetical ¯\(ツ)/¯.

First is good to understand that we will have only ONE SCORE PER PLAYER, but with this modeling like that if the player goes through the same track twice in different scores, it will generate two different entries.

INSERT INTO leaderboard.song_leaderboard  (
    track_id, 
    player_id,
    modifiers,
    score,
    difficulty,
    instrument,
    stars, 
    played_at
) VALUES (
    'starlight-muse',
    'daniel-reis', 
    {'none'}, 
    133700, 
    'expert', 
    'guitar', 
    '2023-11-23 00:00:00'
);

INSERT INTO leaderboard.song_leaderboard (
    track_id,
    player_id,
    modifiers,
    score,
    difficulty,
    instrument,
    stars, 
    played_at
) VALUES (
    'starlight-muse',
    'daniel-reis', 
    {'none'}, 
    123700, 
    'expert', 
    'guitar', 
    '2023-11-23 00:00:00'
);


SELECT 
    player_id, score
FROM 
    leaderboard.song_leaderboard 
WHERE 
    instrument = 'guitar' AND
    difficulty = 'expert' AND
    modifiers = {'none'} AND
    track_id = 'starlight-muse' 
LIMIT 
    2;

--   player_id  | score
----------------+-------
--  daniel-reis | 133700
--  daniel-reis | 123700
----------------+-------

So how do we fix this problem? Well, it's not a problem per se. It's a feature! LOL

As a developer, you have to create your own business rules based on the project needs, and this is no different. What do I mean by that?

You can run a simple DELETE query before insert the new entry and guarantee that you will not have a specific data from the player_id with less than the new score inside that specific group of partition keys.

-- Before Insert the new Gampleplay

DELETE FROM 
    leaderboard.song_leaderboard 
WHERE 
    instrument = 'guitar' AND
    difficulty = 'expert' AND
    modifiers = {'none'} AND
    track_id = 'starlight-muse' AND
    player_id = 'daniel-reis' AND
    score <= 'your-new-score-here';

-- Now you can insert the new payload...

And with that we finished our simple leaderboard system, the same one that runs in YARG and can also be used in games with MILLIONS of entries per second :D

4. How to Contribute to YARG

Here's the part of the text that I'll invite you to contribute on this wonderful open source project!

Today we're building a brand new platform for all the players using:

Game: Unity3d (Repository)
Front-end: NextJS (Repository)
Back-end: Laravel 10.x (Repository)

and we will need many developers and testers as possible to discuss future implementations of the game together with the main contributors!

First, make sure to join their Discord Community. There is the place where all the technical discussions happen with the back of the community before going to the development board.

Also, outside of Discord, the YARG community is mostly focused on the EliteAsian (core contributor and project owner) Twitter account for development showcases. Be sure to follow him there as well.

// Detect dark theme var iframe = document.getElementById('tweet-1736149319382671766-123'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1736149319382671766&theme=dark" }

And FYI, the Lead Artist of the game, aka Kadu is also a Broadcast Specialist and Product Innovation Developer at Elgato that worked with streamers like:

Ninja
Nadeshot
StoneMountain64
and the legendary DJ Marshmello.

Kadu also use his twitter to share some insights and early previews of new features and experimentations for YARG. So, don't forget to follow him on Twitter as well!

// Detect dark theme var iframe = document.getElementById('tweet-1689489132060397568-629'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1689489132060397568&theme=dark" }

Here is some useful links to know more about the project:

Fun fact: YARG got noticed by Brian Bright, project lead on Guitar Hero, who liked the fact that the project was open source. Awesome, right?

5. Conclusion

Data modeling is sometimes challenging, and this study took 3 months of many new ScyllaDB concepts and a lot of testing together with my community at Twitch.

I have also published a Gaming Leaderboard Demo, where you can get some insights on how to implement the same project using NextJS and ScyllaDB!

Also, if you liked ScyllaDB and want to learn more about, I strongly suggest you to watch our free Masterclass Courses or vising ScyllaDB University!

Don't forget to like this post, follow me on the socials and fill your water bottle xD

See you in the next article!

Follow me on Twitter
Follow me on Github
Follow me on Github
Follow and Subscribe at my Twitch Channel

Translating Database Queries

Daniel Reis — Sun, 28 Jan 2024 16:01:37 +0000

Did you ever wondered a tool that can be translating many database syntax's at the same time?

1. Prologue
2. Once upon a time
3. Translating SQL
4. Final Considerations

1. Prologue

I've been working professionally as a developer since 2018, and one of the hardest things to decide when I decided to start writing feature tests was: which database should I use as the base for our test cases?

Why do I say this? I mean, most of the tutorials on the internet at that time tell us to use SQLITE, which is strange since you probably don't run it in production.

As far as I can remember, videos and articles creating CRUD solutions using PHP, C#, Java and Python point directly to:

Oracle
MariaDB
MySQL
PostgreSQL

and of course that there's a lot of differentiation on these databases syntax.

2. Once upon a time

This is one of many scripts that I decided to convert between SQLITE and MySQL in a very specific project and I can tell you that is a PAIN IN THE ASS!

Being a junior/mid level developer getting weird tasks that might not be the best approach to solve a problem sucks totally, mostly because you have to stop and study the differences between databases and their syntax with a short deadline.

Since the ISO (International Organization for Standardization) and ANSI (American National Standards Institute) have decided that if you want to implement the SQL standard, you HAVE to implement the function, the name of the function itself doesn't really matter. Right? RIGHT??

Aggregators, functions and any other type of syntax related can be written in different styles and since it's implemented following the SQL-92 standards, will be recognized as an SQL database.

Take a look at the script below:

-- Enabling Foreign Keys in different databases
SET CONSTRAINTS ALL DEFERRED; -- PostgreSQL
SET FOREIGN_KEY_CHECKS = 1;   -- MySQL
PRAGMA foreign_keys = ON;     -- SQLite

-- Disabling Foreign Keys in different databases
PRAGMA foreign_keys = OFF;    -- SQLite
SET FOREIGN_KEY_CHECKS=0;     -- MySQL
COMMIT;                       -- PostgreSQL (after disabling it)

You can do exactly the same thing in different databases, but it's not that common to find a tool that will translate it from a single source.

3. Translating SQL

Imagine a tool that allows you to write a statement in SQL or even high order functions that are executed inside a database.

This tool can also be used to generate the same query that will be executed, but for different relational databases and even in other paradigms like Wide Column (ScyllaDB, CassandraDB).

Did you visualized it? No? So let me present you the esProc!

esProc is a powerful programming language that can work as a data warehouse, having many capabilities and one of them is to translate query statements to many different databases:

sql = select * from Orders where ADDDAYS(OrderDate,3) > ShipDate
sql.sqltranslate(MySQL) -> select * from Orders where (OrderDate + INTERVAL 3 DAY)>ShipDate
sql.sqltranslate(Oracle) -> select * from Orders where (OrderDate + NUMTODSINTERVAL(3, DAY)>ShipDate
sql.sqltranslate(DB2) -> select * from Orders where (OrderDate + 3 DAYS )>ShipDate

This is a powerful and useful way to consume many database sources based in one unique language.

4. Final Considerations

Having multiple databases inside a project would require tons of abstractions and implementations, however with a good tool.

EsProc can bring you many possibilities to explore and I'll be inviting your to know more about them and star the repository on Github!

EsProc Github Link

I'm creating a new tech community

Daniel Reis — Mon, 15 Jan 2024 15:01:52 +0000

What's up folks! I'm here to recruit the coolest people to join my new community.

Recently I decided to step down from my previous community, He4rt Developers (PT-BR) community and start a new thing focused in the international environment.

Yet Another Tech Community

First weekly community meeting with a peak of 54 members

The community is called Basement Devs, a community focused to help people to achieve better opportunities in Europe mostly (US can be a thing as well).

Why I decided to start that: I want to give more people a safe place to practice English and get better opportunities around the world through my network with the Microsoft MVP program, Developer Relations communities, and tech events I'm currently attending.

I've been working with communities for 5 years so far, but this is the first time I've trying to make this a global thing, so it would be cool to have more people join us!

So far the community has 1,160 members and 1 week of existence and most of the members are mid/senior level, Brazilians, from all possible backgrounds and stacks who want to mentor a non-Portuguese speaker (any level) to practice their English.

Also if you know anyone that wants to learn about programming, we'll be really glad to help them!

So what? Who can join us?

If you're starting at programming or is already at the market and looking for guidance, our community is what you're looking for!

We have members that works in huge companies like X-Team, Spotify and GitHub (among many others) that will gladly help you when possible!

And if you're in university, it will also be a cool thing for you to share some of your problems and solutions for specific classes.

The only thing that we ask you for is to respect the community rules and the Discord guidelines.

Next Steps: stick together ft. Rust!

Our focus is on the Discord community, and I also decided to use this fresh start to learn more about Rust. So all the tools/bots/whatever we're going to build there will use it.

Also I already scheduled a weekly growth meeting, a very special moment to understand from the community what will be our next steps and also bring some good news! This meeting happens every Tuesday at:

5PM (EST)
7PM (BRT)
11PM (CET)

... so feel free to join us if possible!

Usually I do 6h at least of LiveCoding at Twitch and I'm getting at least 1h to discuss about the community together with the members, so, feel free to join us there as well!

Anyway, thanks for reading this until now and welcome to our community!

Link to Discord Community
Link to Twitch Channel
Link to Twitter Profile

Criando Exceptions para impressionar no Teste Técnico

Daniel Reis — Tue, 10 Oct 2023 01:03:22 +0000

Exceptions sempre vai ser um assunto constante quando o tópico for Orientação à Objetos e hoje vamos descobrir como criá-las como um artesão de software!

Tabela de Conteúdo

1. Prólogo
2. O que gostariamos de EVITAR
3. Refatoração 1: Criando Exceptions
4. Design Patterns: Factory Pattern
5. Refatoração 2: Refinando as Exceptions
6. Conclusão

1. Prólogo

Quando eu comecei a estudar programação, um dos assuntos que sempre me assustava eram "erros" ou qualquer coisa que se relaciona à isso, porém após começar a estudar com mais frequencia, eu entendi que os erros e/ou Exceptions são muito mais amigas do que inimigas. Mas é claro que pra isso, você precisa entender como utilizar de um jeito interessante pro seu projeto.

No meu caso, acabava usando o trecho throw new Exception() pra literalmente qualquer coisa e me perdia facilmente na codebase, por conta de uma Exception genérica espalhada no meio de tantas outras. No início não tinha problema, eu ainda não trabalhava com time que tinha observabilidade, então tá tudo certo.

Passado o tempo, entrei em mais empresas FODAS e me deparei com excelentes implementações de Exceptions, principalmente o Factory Pattern. Esse método me deixou maravilhado em como as coisas podem ser simples e elegantes, mesmo quando se trata de erros.

E hoje vou mostrar pra vocês como criar um certo gosto em escrever exceções elegantes pra não encher seu código com 2km de mensagem de erro dentro da regra de negócio.

2. O que gostariamos de EVITAR

Vamos começar dando um pouco de contexto para esse tutorial: imagine que você tá desenvolvendo um sistema de RPG e nele você precisa criar um inventário simples pro seu personagem.

src
├── Item
│   └── Item.php
└── Player
    ├── Inventory.php
    └── Player.php

Dentro desse contexto, imagine que você está tentando equipar um item no seu personagem. Porém, é lógico que nós vamos colocar algumas regras de validação com suas devidas Exceptions.

namespace DanielHe4rt\Player;

use DanielHe4rt\Item\Item;

class Player {

    public function __construct(
        public string $username,
        public int $level,
        protected Inventory $inventory,
    ) {

    }

    public function equipItem(Item $item): void
    {

        if ($this->inventory->hasItem($item)) {
            throw new \Exception(
                'Você não possui o item "' . $item->name . '". '
            );
        }

        if ($item->minLevel > $this->level) {
            throw new \Exception(
                'Você não pode equipar o item ' . $item->name . 'pois o nível minimo é ' . $item->minLevel 
            );
        }

        $this->setupItem($item);
    }


    private function setupItem(Item $item): void 
    {
        // faça coisas iradas
    }
}

Vimos que existem duas regras de validação que jogam exceções diferentes pro nosso cliente. E pasmem: isso funciona (num cenário que o código tá completinho) e cumpre o papel de validação... MASSSSS, depois que eu aprendi que em testes de emprego o que é visto é a QUALIDADE DA ENTREGA e não a agilidade que foi criado, meu mundo deu uma leve mudada para entender como transformar coisas que parecem "estranhas e feias" em coisas "simples e elegantes".

Nesse caso, eu gostaria muito de evitar duas coisas:

Exceptions genéricas;
Exceptions que tomam conta de locais para regra de negócio.

Não entenda errado, as exceptions vão continuar onde elas estão, porém vamos melhorar a legibilidade do código.

3. Refatoração 1 : Criando Exceptions

src
├── Item
│   └── Item.php
└── Player
    ├── Exceptions
    │   ├── PlayerException.php
    │   └── PlayerInventoryException.php
    ├── Inventory.php
    └── Player.php

Beleza, agora passamos para parte que criamos nossa primeira Exception "customizada", onde só estendemos a Exception base para uma nova classe. Não é nada de outro mundo, mas já melhora nossa legibilidade e entendimento do código em alguns vários pontos.

namespace DanielHe4rt\Player; 

class PlayerException extends \Exception
{}

class PlayerInventoryException extends \Exception
{}

E faremos uma refatoração simples na nossa função equipItem(), reposicionando as exceptions padrões pela nova exception que criamos.


namespace DanielHe4rt\Player;

use DanielHe4rt\Item\Item;
use DanielHe4rt\Player\PlayerException;
use DanielHe4rt\Player\PlayerInventoryException;

class Player {

    public function __construct(
        public string $username,
        public int $level,
        protected Inventory $inventory,
    ) {

    }

    public function equipItem(Item $item): void
    {
        if (!$this->inventory->hasItem($item)) {
            throw new PlayerInventoryException(
                'Você não possui o item "' . $item->name . '". '
            );
        }

        if ($item->minLevel > $this->level) {
            throw new PlayerException(
                'Você não pode equipar o item ' . $item->name . 'pois o nível minimo é ' . $item->minLevel 
            );
        }

        $this->setupItem($item);
    }


    private function setupItem(Item $item): void 
    {
        // faça coisas iradas
    }
}

Com as novas Exceptions, agora sabemos exatamente do que se trata e principalmente onde buscar na nossa codebase quando essa Exception estourar. É literalmente um CTRL + ALT + F e pesquisar o nome "PlayerInventoryException". Facilita sua vida, a vida do DevOps que vai meter isso num NewRelic/DataDog da vida e assim seguimos.

Porém algo ainda me incomoda muito... Por quê essas mensagens gigantes estão no meio da regra de negócio? Misturar pt-br com en desse jeito é triste d+ pra mim desculpa amigos!! Vamos aprender um jeito de por isso debaixo dos panos, porém antes precisamos passar num tópico de Design Pattern chamado Factory!

4. Design Patterns: Factory Pattern

Se você já ouviu falar de Design Patterns, provavelmente já entende um pouco sobre o que isso resolve. Mas caso não, eu te explico!

"Design Patterns são soluções genéricas para problemas genéricos." - Alguém por ai

Nessa ideia de problemas genéricos, uma galera se reuniu e começou a criar alguns principios de Design de Software pra você resolver problemas do dia a dia com uma certa agilidade. Os Design Patterns são divididos em três tipos:

Padrões Comportamentais;
Padrões Criacionais;
Padrões Estruturais.

e você pode ler mais sobre eles no site https:/refactoring.guru e eu recomendo MUITO pra qualquer pessoa desenvolvedora explorar essa documentação e se auto desenvolver. Ok, mas vamos focar nele, o tal do Criacional de Fábrica (ou Factory Method).

A ideia desse padrão é você criar objetos sem ter que instaciar mil coisas em classes diferentes, você literalmente fabricar alguma instância de algo e só receber numa chamada simples de alguma função. No mundo da programação existem centenas de milhões de chamadas como Models::make(), Exception::create(), ApiQualquer::factory() pra você não ter que acessar o método construtor de uma respectiva classe.

Dando o exemplo de um Client de API, onde deixamos o construtor modular pra caso precisemos trocar a chave e segredo PORÉM ainda damos a possibilidade de uma chamada rápida fabricando o objeto final:


class GithubClient {

    public function __construct(string $clientId, string $clientSecret) 
    {
        $this->client = new Client([
            'clientId' => $clientId,
            'clientSecret' => $clientSecret,
        ]);
    }

    public static function make(): self
    {
        return new self(
            env('github.client_id'),
            env('github.client_secret'),
        );
    }

    public function getUserByUsername(string $username = 'danielhe4rt'): array 
    {
        // faça uma chamada pro github..
    }
}

// Chamando sem fabricar o objeto

$client = (new GithubClient('client-id-foda', 'client-secret-foda'))
    ->getUserByUsername('danielhe4rt');

// Chamando usando o Factory 

$client = GithubClient::make()
    ->getUserByUsername('danielhe4rt');

Nós fizemos uma chamada estática fabricando todos os parâmetros de um jeito sucinto. Esse "make/factory" ou o que você quiser chamar, pode ser um método bem extenso dependendo do que você for injetar mas isso não chega a ser problema.

Mas de qualquer forma, vimos que a legibilidade usando o Factory Pattern foi melhorada, claro que você pode colocar melhores nomes pras funções mas na base é isso. Agora voltemos para nossas exceptions!

5. Refatoração 2: Refinando as Exceptions

Show, aprendemos um pouquinho sobre o factory, agora vamos aplicar.

Criaremos um método de factory para nossa exception que faça sentido com o contexto do que tá acontecendo. Pois é, nada de usar "make" ou "create" nesses momentos. Exception precisam contar minimamente uma história pro usuário ou pro desenvolvedor do quê tá acontecendo e vamos focar nisso.

Depois de uma leve refatoração na nossa PlayerInventoryException, temos o resultado de:

class PlayerInventoryException extends \Exception 
{
    public static function itemNotFound(string $itemName): self
    {
        $message = sprintf('Você não possui o item "%s".', $itemName);
        return new self(
            message: $message,
            code: 403 // Forbidden
        );
    }
}

E após chamarmos essa factory em nosso código, podemos perceber uma melhora de leitura e mantenibilidade já que isolamos as informações da Exception dentro da mesma.

public function equipItem(Item $item): void
{

    if (!$this->inventory->hasItem($item)) {
        throw PlayerInventoryException::itemNotFound($item->name);
    }

    if ($item->minLevel > $this->level) {
        throw new PlayerException(
            'Você não pode equipar o item ' . $item->name . 'pois o nível minimo é ' . $item->minLevel 
        );
    }

    $this->setupItem($item);
}

Agora refatorando a próxima, temos a mesma ideia de trocar a PlayerException.

class PlayerException extends \Exception 
{
    public static function lowLevelForThisEquipment(string $itemName, int $itemLevel): self
    {

        $message = sprintf(
            'Você não pode equipar o item %s pois o nível minimo é %s.',
            $itemName,
            $itemLevel
        );

        return new self(
            message: $message,
            code: 403 // Forbidden
        );
    }
}

e agora nossa equipItem() tá ó, uma maravilha!

public function equipItem(Item $item): void
{
    if (!$this->inventory->hasItem($item)) {
        throw PlayerInventoryException::itemNotFound($item->name);
    }

    if ($item->minLevel > $this->level) {
        throw PlayerException::lowLevelForThisEquipment($item->name, $item->minLevel);
    }

    $this->setupItem($item);
}

Tá uma maravilha? Tá. Mas ainda tem algo me incomodando... Por quê passar os tipos primitivos sendo que essas exceptions estão se "comunicando" com classes?

Ficaria bem mais limpo se passarmos a referência do objeto inteiro pra Exception e lá dentro ela resolve o que precisa usar. Afinal, vai que precisamos de mais algo num futuro e não custa nada já deixar bonitinho, né?

namespace DanielHe4rt\Player;

use DanielHe4rt\Item\Item;

class PlayerException extends \Exception 
{
    public static function lowLevelForThisEquipment(Item $item): self
    {

        $message = sprintf(
            'Você não pode equipar o item %s pois o nível minimo é %s.',
            $item->name,
            $item->minLevel
        );

        return new self(
            message: $message,
            code: 403 // Forbidden
        );
    }
}

class PlayerInventoryException extends \Exception 
{
    public static function itemNotFound(Item $item): self
    {

        $message = sprintf('Você não possui o item "%s".', $item);

        return new self(
            message: $message,
            code: 403 // Forbidden
        );
    }
}

E o resultado final do nosso método fica só o charme, tendo exceptions encapsuladas e ainda vai te gerar ótimos feedbacks na sua entrevista de emprego.

namespace DanielHe4rt\Player;

use DanielHe4rt\Item\Item;
use DanielHe4rt\Player\PlayerException;
use DanielHe4rt\Player\PlayerInventoryException;

class Player {

    public function __construct(
        public string $username,
        public int $level,
        protected Inventory $inventory,
    ) {

    }

    public function equipItem(Item $item): void
    {
        if (!$this->inventory->hasItem($item)) {
            throw PlayerInventoryException::itemNotFound($item);
        }

        if ($item->minLevel > $this->level) {
            throw PlayerException::lowLevelForThisEquipment($item);
        }

        $this->setupItem($item);
    }


    private function setupItem(Item $item): void 
    {
        // faça coisas iradas
    }
}

6. Conclusão

Exceptions são de longe uma das coisas mais "chatas" de se lidar. Afinal ninguém quer erro estourando na tela do cliente, mas no geral elas só precisam ter uma boa escrita e adicionar um pouquinho de charme com chamadas estáticas e PLAU tu ganha um elogio e ponto positivo na entrevista de emprego.

Espero que vocês tenham curtido o conteúdo e não esqueça de me seguir nas redes sociais!

Referência: Formatting Exception Messages

Forem: Daniel Reis

Data Engineering 101 - A real beginner's approach

Table of Contents

1. Prologue

Imagine a single framework that:

2. First Impressions: Exploring Bruin’s Structure

Example: Project Structure

What Each Part Does

2.1. Core File: .bruin.yml

What’s Happening Here

2.2. Pipeline: A WAY EASIER Apache Airflow

2.3. Assets: The Building Blocks of Data Products

2.4. Policies: Enforcing Quality and Governance

Quick Overview

Built-in and Custom Rules

2.5. Glossary: Speaking the Same Language

3. Building Our First Pipeline

Running the Asset

3.2 Step 2: Formatting and Validating the Data (Staging Layer)

Validation and Execution

3.3 Step 3: Designing the Analytics (Mart) Layer

3.3.1 Asset: mart.customers_by_country.asset.sql

3.3.2 Asset: mart.customers_by_age.asset.sql

4. Full Preview Data Flow

5. Overpowered VS Code Extension

6. Conclusion

Have you used Rust (or Go, or anything else) and just felt relieved by how they treat errors?

With Exceptions

With a Result Class

How I learn any type of new technology (As a Senior Developer)

Table of Content

1. Prologue

2. The Basics

2.1 Break it until you make it

2.2 But... It's not that obvious

3. The Documentation

4. The Source Code

5. The Network Tab

6. The Implementation

7. Conclusion

[Boost]

#DevOps para noobs - 3 Boas Práticas de Bancos de Dados para compartilhar com os Devs (AWS RDS)

Camila Figueira ・ Feb 20

This is all what I've learned about Go in TWO Weeks!

Table of Contents

1. Prologue

2. Meet the CLI

2.1 CLI: go mod

2.2 CLI: go run

3. Comparing Different Syntax's:

3.1 Syntax: Classes/Structs and the API Encapsulation

3.2 Syntax: Interface Implementation is WEIRD AS FUC*

4. Stdlib: definitely an AWESOME toolkit

4.1 Packages: Primitive Types

4.2 Packages: Useful Stuff

5. Tests in Go are THAT SIMPLE

5.1 Tests: Basic Testing

5.2 Tests: Jetbrains Boilerplate

5.3 Tests: Running Tests

6. Beware Cyclic Imports

7. Defer this, defer that... But, what is a Defer?

8. Error Management for Noobs

9. Conclusion feat: Low Latency Coding Challenge

9.1 Bonus: Coding Challenge

Database 101: SSL/TLS for beginners

Table of Contents

1. Prologue

2. What is SSL and TLS?

3. OpenSSL for Noobs

4. Database Authentication

4.1 No Authentication

4.2 Username/Password Authentication

4.3 Creating Users and Roles

4.4 Certificated / Role Certificated Authentication

4.4.1 Change the Authentication Type

4.4.2 Enable the Encryption Port

4.4.3 Enable the Client Encryption Port

5. Testing Encrypted Connections

6. Conclusion

Laravel inside Rust? I have a reason for that.

2.1. Core File: `.bruin.yml`

3.3.1 Asset: `mart.customers_by_country.asset.sql`

3.3.2 Asset: `mart.customers_by_age.asset.sql`