Databricks Data Engineering Concepts

Procedural vs. Declarative Data Processing

Overview

• Procedural and declarative programming are two fundamental programming paradigms representing different approaches to structuring and executing instructions.
• Choosing between these models when designing data pipelines impacts workflow complexity, maintainability, and efficiency.

Procedural Data Processing

• Definition: A structured approach that defines explicit steps to manipulate data, emphasizing a command sequence that dictates how data should be processed.
• Characteristics:
  • Step-by-step execution: The developer explicitly defines the order of operations.
  • Control structures: Uses loops, conditionals, and functions to manage execution flow.
  • Detailed resource control: Enables fine-grained optimizations and manual performance tuning.
• Common Use Cases:
  • Custom ETL pipelines requiring procedural logic.
  • Low-level performance optimizations in batch and streaming workflows.
  • Legacy systems or existing imperative scripts.
• Databricks Context: Apache Spark primarily follows a procedural model. Lakeflow Jobs can be used to add explicit execution logic to define step-by-step transformations and actions on distributed data.

In a procedural processing model, you explicitly specify each step-by-step operation to transform and load the data. You control the exact operational control flow (read -> filter -> aggregate -> write)

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.appName("ProceduralPipeline").getOrCreate()

# Step 1: Explicitly load raw data
raw_df = spark.read.json("s3://my-bucket/data/orders.json")

# Step 2: Manually filter completed orders
filtered_df = raw_df.filter(col("status") == "COMPLETED")

# Step 3: Explicitly define aggregation logic
aggregated_df = filtered_df.groupBy("customer_id") \
                           .sum("amount") \
                           .withColumnRenamed("sum(amount)", "total_spent")

# Step 4: Manually execute write operation to destination
aggregated_df.write.format("delta").mode("overwrite").saveAsTable("customer_totals")

Declarative Data Processing

• Definition: Abstracting the how, this model focuses on defining what needs to be achieved (the desired result), leaving the underlying system to determine the most efficient execution plan.
• Characteristics:
  • Abstraction of execution details: Users describe the desired outcome rather than the execution steps.
  • Automatic optimization: The system applies query planning and execution tuning automatically.
  • Reduced complexity: Eliminates the need for explicit control structures, improving maintainability.
  • Paradigm relation: Includes domain-specific and functional programming paradigms.
• Common Use Cases:
  • SQL-based transformations in batch and streaming workflows.
  • High-level data processing frameworks such as pipelines.
  • Scalable, distributed data workloads requiring automated optimizations.
• Databricks Context: Lakeflow pipelines are built on Apache Spark Declarative Pipelines (SDP). By specifying what data to ingest and how to transform it, pipelines automate orchestration, compute management, monitoring, data quality enforcement, and error handling.

In a declarative processing model, you describe what target table or view should exist, and let the system (such as Lakeflow pipelines) automatically figure out optimization, execution order, and resource management.

-- Define the target view/table declaration
CREATE OR REFRESH MATERIALIZED VIEW customer_totals AS
SELECT 
  customer_id,
  SUM(amount) AS total_spent
FROM 
  STREAM(live.raw_orders)
WHERE 
  status = 'COMPLETED'
GROUP BY 
  customer_id;

Key Differences Comparison

When to Choose Which Approach

• Choose Procedural Processing when:
  • Fine-grained control over execution logic is required.
  • Transformations involve complex business rules that are difficult to express declaratively.
  • Performance optimizations necessitate manual tuning.
• Choose Declarative Processing when:
  • Simplified development and maintenance are priorities.
  • SQL-based transformations or managed workflows eliminate the need for procedural control.
  • Data processing frameworks (such as pipelines) provide built-in optimizations.

How to Think About It

• Procedural (Baking from Scratch): You write out every exact action step by step (crack eggs, mix, check smoothness, bake). You control every move, but you have to manage all the effort yourself.
• Declarative (Ordering at a Bakery): You simply say, “I want a chocolate cake.” You don’t care how the baker mixes the flour or sets the oven temperature—the system handles all the steps to give you the final result.