Forem

# dataengineering

Posts

đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
Refactoring a Mature Airflow Project: A Practical Guide to Scaling from Solo Development to Team Collaboration
ajitkumar profile

Refactoring a Mature Airflow Project: A Practical Guide to Scaling from Solo Development to Team Collaboration

#airflow #python #datascience #dataengineering
Comments
4 min read
Apache Dev List Digest: Iceberg, Polaris, Arrow & Parquet (Nov 24-Dec 8, 2025)
Cover image for Apache Dev List Digest: Iceberg, Polaris, Arrow & Parquet (Nov 24-Dec 8, 2025)
alexmercedcoder profile

Apache Dev List Digest: Iceberg, Polaris, Arrow & Parquet (Nov 24-Dec 8, 2025)

#dataengineering #opensource #architecture #community
Comments
6 min read
2025 Year in Review: Apache Iceberg, Polaris, Parquet, and Arrow
Cover image for 2025 Year in Review: Apache Iceberg, Polaris, Parquet, and Arrow
alexmercedcoder profile

2025 Year in Review: Apache Iceberg, Polaris, Parquet, and Arrow

#architecture #bigdata #opensource #dataengineering
Comments
6 min read
Context Engineering (Part 1): The Architecture of Recall
mosiddi profile

Context Engineering (Part 1): The Architecture of Recall

#dataengineering #rags #informationretrieval #vectorsearch
Comments 1
3 min read
Day 9: Spark SQL Deep Dive - Temp Views, Query Execution & Optimization Tips for Data Engineers
Cover image for Day 9: Spark SQL Deep Dive - Temp Views, Query Execution & Optimization Tips for Data Engineers
sandeepk27 profile

Day 9: Spark SQL Deep Dive - Temp Views, Query Execution & Optimization Tips for Data Engineers

#python #dataengineering #spark #bigdata
Comments
2 min read
AWSChallenge - Week 2
Cover image for AWSChallenge - Week 2
andresguecha profile

AWSChallenge - Week 2

#ai #dataengineering #aws #programming
Comments
4 min read
Day 10: Partitioning vs Bucketing - The Spark Optimization Guide Every Data Engineer Needs
Cover image for Day 10: Partitioning vs Bucketing - The Spark Optimization Guide Every Data Engineer Needs
sandeepk27 profile

Day 10: Partitioning vs Bucketing - The Spark Optimization Guide Every Data Engineer Needs

#python #dataengineering #spark #bigdata
Comments
2 min read
Deepening My Roots in the Data Ecosystem - Choosing Depth Over Breadth
jongwan93 profile

Deepening My Roots in the Data Ecosystem - Choosing Depth Over Breadth

#career #dataengineering #devjournal
Comments
2 min read
Automate Python Manual Extraction: Build End-to-End PDF -> LLM -> SQL Flows with CocoIndex, Ollama, and Postgres
Cover image for Automate Python Manual Extraction: Build End-to-End PDF -> LLM -> SQL Flows with CocoIndex, Ollama, and Postgres
badmonster0 profile

Automate Python Manual Extraction: Build End-to-End PDF -> LLM -> SQL Flows with CocoIndex, Ollama, and Postgres

#python #llm #dataengineering #etl
Comments
3 min read
The Boring Debug Checklist That Fixes Most “RAG Failures”
Cover image for The Boring Debug Checklist That Fixes Most “RAG Failures”
dowhatmatters profile

The Boring Debug Checklist That Fixes Most “RAG Failures”

#rag #dataengineering #llm #architecture
Comments
2 min read
Function Calling and Tool Use: Turning LLMs into Action-Taking Agents
qvfagundes profile

Function Calling and Tool Use: Turning LLMs into Action-Taking Agents

#ai #llm #mcp #dataengineering
Comments
18 min read
dremioframe & iceberg: Pythonic interfaces for Dremio and Apache Iceberg
Cover image for dremioframe & iceberg: Pythonic interfaces for Dremio and Apache Iceberg
alexmercedcoder profile

dremioframe & iceberg: Pythonic interfaces for Dremio and Apache Iceberg

#python #tooling #database #dataengineering
Comments
8 min read
AWS Lambda and AWS Glue Python Shell in the Context of Lightweight ETL
datapenguin profile

AWS Lambda and AWS Glue Python Shell in the Context of Lightweight ETL

#aws #dataengineering #etl
3
Comments
7 min read
SQL: Doing GROUP BY in CsvPath
Cover image for SQL: Doing GROUP BY in CsvPath
david_kershaw_b6916404da6 profile

SQL: Doing GROUP BY in CsvPath

#sql #dataengineering #csv #csvpath
Comments
5 min read
🔥 Day 3: RDDs - The Foundation of Spark
Cover image for 🔥 Day 3: RDDs - The Foundation of Spark
sandeepk27 profile

🔥 Day 3: RDDs - The Foundation of Spark

#distributedsystems #interview #dataengineering #tutorial
Comments
2 min read
đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.