DEV Community

# bigdata

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
Inside Apache SeaTunnel CDC: How the System Really Works

Inside Apache SeaTunnel CDC: How the System Really Works

Comments
10 min read
Apache Doris IP change problem handling method

Apache Doris IP change problem handling method

Comments
4 min read
Overview of Real-Time Data Synchronization from PostgreSQL to VeloDB

Overview of Real-Time Data Synchronization from PostgreSQL to VeloDB

Comments
6 min read
Day 16: Delta Lake Explained - How Spark Finally Became Reliable for Production ETL

Day 16: Delta Lake Explained - How Spark Finally Became Reliable for Production ETL

Comments
2 min read
Day 15: Running Spark in the Cloud - Dataproc vs Databricks

Day 15: Running Spark in the Cloud - Dataproc vs Databricks

Comments
2 min read
Day 14: Building a Real Retail Analytics Pipeline Using Spark Window Functions

Day 14: Building a Real Retail Analytics Pipeline Using Spark Window Functions

Comments
1 min read
Day 13: Window Functions in PySpark

Day 13: Window Functions in PySpark

Comments
2 min read
Day 17: Building a Real ETL Pipeline in Spark Using Bronze-Silver-Gold Architecture

Day 17: Building a Real ETL Pipeline in Spark Using Bronze-Silver-Gold Architecture

Comments
1 min read
Day 12: UDF vs Pandas UDF

Day 12: UDF vs Pandas UDF

Comments
2 min read
Agent Facing Analytics with High Concurrency: Doris vs Clickhouse vs Snowflake

Agent Facing Analytics with High Concurrency: Doris vs Clickhouse vs Snowflake

Comments
5 min read
Connector Fixes, Core API Enhancements, and Ecosystem Updates: Apache SeaTunnel’s Progress in November

Connector Fixes, Core API Enhancements, and Ecosystem Updates: Apache SeaTunnel’s Progress in November

Comments
6 min read
Day 11: Choosing the Right File Format in Spark

Day 11: Choosing the Right File Format in Spark

Comments
2 min read
From Bug Fixes to Ecosystem Enhancements: Key Highlights from DolphinScheduler’s November Updates

From Bug Fixes to Ecosystem Enhancements: Key Highlights from DolphinScheduler’s November Updates

Comments
5 min read
Day 7: Mastering Joins, Unions, and GroupBy in PySpark - The Core ETL Operations

Day 7: Mastering Joins, Unions, and GroupBy in PySpark - The Core ETL Operations

Comments
2 min read
Day 9: Spark SQL Deep Dive - Temp Views, Query Execution & Optimization Tips for Data Engineers

Day 9: Spark SQL Deep Dive - Temp Views, Query Execution & Optimization Tips for Data Engineers

Comments
2 min read
Day 10: Partitioning vs Bucketing - The Spark Optimization Guide Every Data Engineer Needs

Day 10: Partitioning vs Bucketing - The Spark Optimization Guide Every Data Engineer Needs

Comments
2 min read
From Raw Claims and Clinical Data to PCORnet CDM: End-to-End ETL on Snowflake

From Raw Claims and Clinical Data to PCORnet CDM: End-to-End ETL on Snowflake

Comments
7 min read
GSoC Student Crushes It! The Inside Story Behind the OIDC Upgrade for Apache DolphinScheduler

GSoC Student Crushes It! The Inside Story Behind the OIDC Upgrade for Apache DolphinScheduler

Comments
10 min read
🔥 Day 4: RDD Internals - Partitions, Shuffles & Repartitioning Demystified

🔥 Day 4: RDD Internals - Partitions, Shuffles & Repartitioning Demystified

Comments
2 min read
🔥 Day 2: Understanding Spark Architecture - How Spark Executes Your Code Internally

🔥 Day 2: Understanding Spark Architecture - How Spark Executes Your Code Internally

Comments
2 min read
🔥 Day 5: Introduction to DataFrames - The Most Importantce of Spark API

🔥 Day 5: Introduction to DataFrames - The Most Importantce of Spark API

Comments
2 min read
Day 8: Accelerating Spark Joins - Broadcast, Shuffle Optimization & Skew Handling

Day 8: Accelerating Spark Joins - Broadcast, Shuffle Optimization & Skew Handling

Comments
2 min read
From Raw to Refined: Data Pipeline Architecture at Scale

From Raw to Refined: Data Pipeline Architecture at Scale

Comments
12 min read
Data Quality at Scale: Why Your Pipeline Needs More Than Green Checkmarks

Data Quality at Scale: Why Your Pipeline Needs More Than Green Checkmarks

Comments
8 min read
Building Real-Time Lakehouse with S3 Tables, AWS Glue, and Apache Doris

Building Real-Time Lakehouse with S3 Tables, AWS Glue, and Apache Doris

Comments
3 min read
loading...