Forem

# dataengineering

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
🔥 Day 4: RDD Internals - Partitions, Shuffles & Repartitioning Demystified
Cover image for 🔥 Day 4: RDD Internals - Partitions, Shuffles & Repartitioning Demystified
sandeepk27 profile

🔥 Day 4: RDD Internals - Partitions, Shuffles & Repartitioning Demystified

#spark #dataengineering #bigdata #python
Comments
2 min read
The Developer's Guide to Normalizing Historical Airline Flight Data for Machine Learning
Cover image for The Developer's Guide to Normalizing Historical Airline Flight Data for Machine Learning
rameshchauhan profile

The Developer's Guide to Normalizing Historical Airline Flight Data for Machine Learning

#dataengineering #datascience #machinelearning
Comments
6 min read
Overview of Real-Time Data Synchronization from MySQL to VeloDB
apachedoris profile

Overview of Real-Time Data Synchronization from MySQL to VeloDB

#database #aws #dataengineering #mysql
5
Comments
5 min read
Stop Writing df.describe(): Automate EDA with D-Tale (The Lazy Engineer's Way)
theaidataengineer1010 profile

Stop Writing df.describe(): Automate EDA with D-Tale (The Lazy Engineer's Way)

#python #dataengineering #productivity #beginners
Comments
3 min read
CHW Monthly Activity Aggregation: Turning Visit Logs into Insight
oliver_samuel_028c6f65ad6 profile

CHW Monthly Activity Aggregation: Turning Visit Logs into Insight

#analytics #sql #dataengineering #tooling
Comments
5 min read
🔥 Day 2: Understanding Spark Architecture - How Spark Executes Your Code Internally
Cover image for 🔥 Day 2: Understanding Spark Architecture - How Spark Executes Your Code Internally
sandeepk27 profile

🔥 Day 2: Understanding Spark Architecture - How Spark Executes Your Code Internally

#spark #python #dataengineering #bigdata
Comments
2 min read
When models suggest deprecated Pandas APIs: a small mistake that cascades
olivia_perell_ profile

When models suggest deprecated Pandas APIs: a small mistake that cascades

#codequality #dataengineering #llm #python
Comments
3 min read
Marmot: Data catalog without the complex infrastructure
Cover image for Marmot: Data catalog without the complex infrastructure
charliehaley profile

Marmot: Data catalog without the complex infrastructure

#showdev #dataengineering #opensource #go
1
Comments
3 min read
TDD for dbt: unit testing the way it should be
niclasolofsson profile

TDD for dbt: unit testing the way it should be

#codequality #dataengineering #softwareengineering #testing
2
Comments
12 min read
When code-gen suggests deprecated Pandas APIs: a case study in subtle breakage
gabrieal845 profile

When code-gen suggests deprecated Pandas APIs: a case study in subtle breakage

#codequality #dataengineering #llm #python
Comments
3 min read
Schema, COPY, MERGE, and Immutability — A First-Principles Guide for Data Engineers
shrinivasv73 profile

Schema, COPY, MERGE, and Immutability — A First-Principles Guide for Data Engineers

#dataengineering
Comments
5 min read
HackerRank 'The Pads' MySQL
carol_caillaud profile

HackerRank 'The Pads' MySQL

#dataengineering #beginners #sql #mysql
Comments
3 min read
🔥 Day 5: Introduction to DataFrames - The Most Importantce of Spark API
Cover image for 🔥 Day 5: Introduction to DataFrames - The Most Importantce of Spark API
sandeepk27 profile

🔥 Day 5: Introduction to DataFrames - The Most Importantce of Spark API

#dataengineering #python #spark #bigdata
Comments
2 min read
Generating Table Schema from AWS Glue Table
dendihandian profile

Generating Table Schema from AWS Glue Table

#aws #athena #dataengineering #sql
2
Comments
1 min read
Comparing Great Expectations and CsvPath Framework
Cover image for Comparing Great Expectations and CsvPath Framework
david_kershaw_b6916404da6 profile

Comparing Great Expectations and CsvPath Framework

#data #dataengineering #csv #database
Comments
8 min read
👋 Sign in for the ability to sort posts by relevant, latest, or top.