Forem

# dataengineering

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
Building a Real-Time Crypto Pipeline with Binance APIs, PostgreSQL, Debezium, Kafka, Spark & Cassandra
Cover image for Building a Real-Time Crypto Pipeline with Binance APIs, PostgreSQL, Debezium, Kafka, Spark & Cassandra

Building a Real-Time Crypto Pipeline with Binance APIs, PostgreSQL, Debezium, Kafka, Spark & Cassandra

2
Comments
6 min read
BEGINNER'S GUIDE TO STREAM REAL-TIME DATA USING APACHE KAFKA
Cover image for BEGINNER'S GUIDE TO STREAM REAL-TIME DATA USING APACHE KAFKA

BEGINNER'S GUIDE TO STREAM REAL-TIME DATA USING APACHE KAFKA

Comments
4 min read
Stop Drawing ETL Diagrams — Your Python Code Visualizes Itself

Stop Drawing ETL Diagrams — Your Python Code Visualizes Itself

4
Comments
4 min read
⚡ Kafka ClickHouse: Real-Time Data Pipeline for Beginners
Cover image for ⚡ Kafka ClickHouse: Real-Time Data Pipeline for Beginners

⚡ Kafka ClickHouse: Real-Time Data Pipeline for Beginners

2
Comments
2 min read
Become the Serverless DJ. How to process audio using AWS?
Cover image for Become the Serverless DJ. How to process audio using AWS?

Become the Serverless DJ. How to process audio using AWS?

2
Comments
8 min read
Discussion about Data Science project idea

Discussion about Data Science project idea

Comments
1 min read
Why Data Formats Matter More Than You Think
Cover image for Why Data Formats Matter More Than You Think

Why Data Formats Matter More Than You Think

1
Comments
19 min read
Troubleshooting SeaTunnel Cluster Split-Brain: A Deep Dive into Hazelcast Configuration and GC-Induced Failures

Troubleshooting SeaTunnel Cluster Split-Brain: A Deep Dive into Hazelcast Configuration and GC-Induced Failures

Comments
9 min read
A Simple Fix for SeaTunnel Excel Failing to Convert Numeric Types to Strings | With Source Code Packaging

A Simple Fix for SeaTunnel Excel Failing to Convert Numeric Types to Strings | With Source Code Packaging

Comments
3 min read
Personal Picks: Data Product News (May 28, 2025)

Personal Picks: Data Product News (May 28, 2025)

Comments
7 min read
Big Data Fundamentals: spark example

Big Data Fundamentals: spark example

1
Comments
5 min read
Building Multi-Tenant Analytics with Snowflake RBAC and Sigma Computing: Part 3

Building Multi-Tenant Analytics with Snowflake RBAC and Sigma Computing: Part 3

Comments
3 min read
Big Data: Distributed Computing - Your Essential Resource Guide

Big Data: Distributed Computing - Your Essential Resource Guide

Comments
3 min read
🚀 Dagster 2025: Not Just ETL — A Data Asset Mindset

🚀 Dagster 2025: Not Just ETL — A Data Asset Mindset

Comments
1 min read
Big Data Fundamentals: hadoop tutorial

Big Data Fundamentals: hadoop tutorial

2
Comments
6 min read
Top 5 Open Source Tools Every Data Engineer Should Know About (2025 Edition)

Top 5 Open Source Tools Every Data Engineer Should Know About (2025 Edition)

Comments
3 min read
Data Ingestion using Logstash: PostgreSql to Elastic

Data Ingestion using Logstash: PostgreSql to Elastic

1
Comments 2
5 min read
How to Document SQL Server Schemas Visually in 2025
Cover image for How to Document SQL Server Schemas Visually in 2025

How to Document SQL Server Schemas Visually in 2025

12
Comments 1
4 min read
Top 5 Challenges in Migrating to Snowflake and How to Overcome Them
Cover image for Top 5 Challenges in Migrating to Snowflake and How to Overcome Them

Top 5 Challenges in Migrating to Snowflake and How to Overcome Them

1
Comments
9 min read
[Snowflake's New Feature]dbt Projects on Snowflake: Run Your Entire dbt Workflow Directly in Snowflake

[Snowflake's New Feature]dbt Projects on Snowflake: Run Your Entire dbt Workflow Directly in Snowflake

Comments
6 min read
Building Multi-Tenant Analytics with Snowflake RBAC and Sigma Computing: Part 2

Building Multi-Tenant Analytics with Snowflake RBAC and Sigma Computing: Part 2

Comments
3 min read
Como Processar 60+ milhões de CNPJs com Python: Arquitetura e Decisões Técnicas

Como Processar 60+ milhões de CNPJs com Python: Arquitetura e Decisões Técnicas

5
Comments 1
4 min read
Building Multi-Tenant Analytics with Snowflake RBAC and Sigma Computing: Part 1 - Foundation and Data Modeling

Building Multi-Tenant Analytics with Snowflake RBAC and Sigma Computing: Part 1 - Foundation and Data Modeling

Comments
3 min read
How to Design a Relational Database Schema in 2025
Cover image for How to Design a Relational Database Schema in 2025

How to Design a Relational Database Schema in 2025

17
Comments 1
8 min read
StrataScratch's Advanced 25 Hard SQL Questions

StrataScratch's Advanced 25 Hard SQL Questions

1
Comments
2 min read
loading...