Apache Spark is an open-source distributed data processing engine designed for large-scale data analytics, ETL, machine learning, and real-time stream processing.
Instead of processing data on a single machine, Spark distributes work across multiple machines (nodes or workers) and processes data in parallel, making it significantly faster and more scalable than traditional approaches.
Key Benefits
Traditional Hadoop MapReduce processed data where each stage required reading and writing back to disk, causing significant overhead.
Read -> Process -> Write
Read -> Process -> Write
Read -> Process -> WriteWhereas Spark introduced in-memory processing that dramatically improved performance for many data processing workloads.
Read -> Process in Memory -> Process in Memory -> Write
The driver:
Executors:
Spark consists of several modules built on top of Spark Core:
The foundation of Spark
Example: everything in Spark ultimately executes through Spark Core.
rdd = spark.sparkContext.parallelize([1,2,3,4]Used for processing structured and semi-structured data. It supports,
spark.sql("""
SELECT * FROM customers
""")or
df = spark.read.table("customers")In Databricks and modern Spark development, Spark SQL is the most frequently used component.
Provided real-time stream processing. Example pipeline:
Kafka -> Spark Stream -> Delta Table -> DashboardCommon use cases:
MLlib is Spark’s distributed machine learning library. It provides scalable implementations of common ML algorithms, feature extraction tools, and pipeline management.
from pyspark.ml.classification import LogisticRegression
from pyspark.ml.feature import VectorAssembler
# Combine features into a single vector column
assembler = VectorAssembler(inputCols=["age", "income"], outputCol="features")
feature_df = assembler.transform(df)
# Train a distributed logistic regression model
lr = LogisticRegression(featuresCol="features", labelCol="label")
model = lr.fit(feature_df)GraphX is Spark’s API for graph-parallel computation. It extends RDDs to support directed multigraphs with attributes attached to vertices and edges.
(Customer)-[:OWNS]->(Account)Spark provides APIs for multiple programming languages.
Spark was originally developed in Scala.
val df = spark.read.table("customers")Advantages:
It is used in enterprise Java applications.
Dataset<Row> df = spark.read().json("data.json");It is most commonly used today. Popular in:
df = spark.read.table("customers")It is mainly used by statistical analytics.
df <- read.df("customers.csv")It is supported directly through Spark SQL.
SELECT * FROM customersPySpark is the Python API for Apache Spark. It is NOT a separate processing engine. PySpark simply allows Python applications to control Spark.
Yes, a standard Spark installation already contains PySpark.
spark/
├── bin/
├── jars/
└── python/
└── pyspark/Therefore, you can immediately use it without installing anything additional.
from pyspark.sql import SparkSessionThis provides:
pip install pysparkFor Python developers who want Spark without manual installation.
Spark itself runs on the JVM. PySpark communicates through a bridge.
PySpark provides the Python interface, while the actual Spark engine runs in the JVM. The Python process communicates with the JVM-based Spark application through a bridge such as Py4J.
The entry point to Spark.
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()Everything starts with spark object.
Distributed table-like structure.
df = spark.read.csv("customer.csv")
print(df)id name city
1 John Pune
2 Smith MumbaiTransformations are operations that construct a new DataFrame from an existing one without immediately evaluating the data (e.g., filter(), select(), groupBy(), join()).
Spark records these operations in a Lineage Graph (Directed Acyclic Graph or DAG). Execution is deferred until an action is called, allowing Spark’s Catalyst Optimizer to optimize the execution plan (e.g., predicate pushdown and column pruning).
df.filter(df.age > 18)
df.select("name")Actions trigger the physical computation of the execution plan built by transformations. They compile the DAG into physical stages and tasks distributed across worker nodes, returning results to the Driver process or saving data to external storage.
# Triggers execution and displays top rows
df.show(5)
# Triggers execution to aggregate matching rows
total_count = df.count()
# Triggers physical write to storage
df.write.mode("overwrite").parquet("/mnt/data/output")
# Triggers physical write to storage
df.write.mode("overwrite").parquet("/mnt/data/output")df = spark.read.table("customers")Many developers think this is a Spark Core operation, but actually the primary component involved is Spark SQL.
Databricks is a managed platform built around Spark. It provides: