Forem

# dataengineering

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
Apache Arrow dev list digest (Aug 25–29 2025)

Apache Arrow dev list digest (Aug 25–29 2025)

Comments
4 min read
Anomaly Detection in Financial Transactions: Algorithms and Applications
Cover image for Anomaly Detection in Financial Transactions: Algorithms and Applications

Anomaly Detection in Financial Transactions: Algorithms and Applications

2
Comments
10 min read
Apache Kafka Deep Dive: Core Concepts, Data Engineering Applications, and Real-World Production Practices
Cover image for Apache Kafka Deep Dive: Core Concepts, Data Engineering Applications, and Real-World Production Practices

Apache Kafka Deep Dive: Core Concepts, Data Engineering Applications, and Real-World Production Practices

Comments 1
6 min read
(I) Principles of Data Model Architecture: Four Layers and Seven Stages

(I) Principles of Data Model Architecture: Four Layers and Seven Stages

5
Comments
7 min read
Revamping Real-Time Data Ingestion for Scalable Media Intelligence
Cover image for Revamping Real-Time Data Ingestion for Scalable Media Intelligence

Revamping Real-Time Data Ingestion for Scalable Media Intelligence

Comments
4 min read
Two Years of Microsoft Fabric: Game Changer or Still Leveling Up? 🚀
Cover image for Two Years of Microsoft Fabric: Game Changer or Still Leveling Up? 🚀

Two Years of Microsoft Fabric: Game Changer or Still Leveling Up? 🚀

Comments
2 min read
Scraping the Schema of NetSuite

Scraping the Schema of NetSuite

Comments
2 min read
Docker for Data Engineers: The Complete Beginner’s Guide
Cover image for Docker for Data Engineers: The Complete Beginner’s Guide

Docker for Data Engineers: The Complete Beginner’s Guide

3
Comments
6 min read
Dynamic Routing Lightweight ETL with AWS Lambda, DuckDB, and PyIceberg

Dynamic Routing Lightweight ETL with AWS Lambda, DuckDB, and PyIceberg

6
Comments 4
4 min read
🚀 The Future of Data Engineering: How AI and Automation are Changing the Game
Cover image for 🚀 The Future of Data Engineering: How AI and Automation are Changing the Game

🚀 The Future of Data Engineering: How AI and Automation are Changing the Game

Comments
2 min read
Career Opportunities After Completing AI & Data Science Degree

Career Opportunities After Completing AI & Data Science Degree

Comments
3 min read
The Blueprint of a Data Team: Roles, Responsibilities, and Specializations
Cover image for The Blueprint of a Data Team: Roles, Responsibilities, and Specializations

The Blueprint of a Data Team: Roles, Responsibilities, and Specializations

2
Comments
10 min read
Building an End-to-End Data Engineering Pipeline with DuckDB and Python
Cover image for Building an End-to-End Data Engineering Pipeline with DuckDB and Python

Building an End-to-End Data Engineering Pipeline with DuckDB and Python

Comments
5 min read
Big Data Fundamentals: real-time analytics project

Big Data Fundamentals: real-time analytics project

Comments
6 min read
Data Mesh: The Decentralized Revolution That Will Transform Your Data Architecture
Cover image for Data Mesh: The Decentralized Revolution That Will Transform Your Data Architecture

Data Mesh: The Decentralized Revolution That Will Transform Your Data Architecture

1
Comments
4 min read
The Case for Apache Airflow and Kafka in Data Engineering
Cover image for The Case for Apache Airflow and Kafka in Data Engineering

The Case for Apache Airflow and Kafka in Data Engineering

1
Comments
2 min read
🛠️ SiliconPrimeX – Healing AWS Glue Jobs Autonomously with Gemini & Lambda 🚑✨

🛠️ SiliconPrimeX – Healing AWS Glue Jobs Autonomously with Gemini & Lambda 🚑✨

Comments
2 min read
What is the Modern Data Stack?

What is the Modern Data Stack?

Comments
3 min read
Real-Time Fraud Detection Using Kafka and Machine Learning
Cover image for Real-Time Fraud Detection Using Kafka and Machine Learning

Real-Time Fraud Detection Using Kafka and Machine Learning

Comments
5 min read
🔄 ETL vs ELT: The Backbone of Data Engineering

🔄 ETL vs ELT: The Backbone of Data Engineering

1
Comments
1 min read
🕒 Cumulative Data Without the Pain: PostgreSQL Rollups with Time Buckets
Cover image for 🕒 Cumulative Data Without the Pain: PostgreSQL Rollups with Time Buckets

🕒 Cumulative Data Without the Pain: PostgreSQL Rollups with Time Buckets

Comments
3 min read
Check Out 3 Awesome Open Source Tabular Data Wrangling Apps

Check Out 3 Awesome Open Source Tabular Data Wrangling Apps

1
Comments
3 min read
15 Core Concepts of Data Engineering

15 Core Concepts of Data Engineering

Comments
9 min read
Building a Food Price & Inflation Analysis Pipeline in Kenya (2006–2024)
Cover image for Building a Food Price & Inflation Analysis Pipeline in Kenya (2006–2024)

Building a Food Price & Inflation Analysis Pipeline in Kenya (2006–2024)

Comments
3 min read
Understanding Data Warehousing for Retail Analytics: A Comprehensive Guide
Cover image for Understanding Data Warehousing for Retail Analytics: A Comprehensive Guide

Understanding Data Warehousing for Retail Analytics: A Comprehensive Guide

1
Comments
3 min read
loading...