Back to Articles/The Practical Guide to Apache Spark & PySpark

Sections

Introduction What is Spark?Spark Architecture Partitions & Parallelism RDDs, DataFrames, Datasets Transformations & Actions Lazy Evaluation & the DAG The Shuffle Catalyst & Tungsten SparkSession Getting Started DataFrame Operations Spark SQL UDFs & Pandas UDFs Structured Streaming MLlib Pipelines Caching & Persistence Joins & Broadcast Performance Tuning Glossary & Cheatsheet

Glossary & Cheatsheet

10 min read

Part V — Running It Well · Module 18 of 18 Prerequisites: none — this is a reference. Skim after your first read-through, then keep it open while you work. You will find: a plain-language glossary of every term the guide defines, followed by a one-page PySpark cheat sheet of the code and configs you'll reach for most.

This final module is designed for lookup, not linear reading. Each glossary entry links back to the module where the concept is explained in full.

Glossary

A

Action — An operation that triggers execution of the accumulated transformations and returns a value or writes output (e.g., show(), count(), collect(), write.save()). One action = one job. (Module 05)

Adaptive Query Execution (AQE) — A Spark 3.0 feature that re-optimizes the query plan at runtime using statistics from completed stages — coalescing small shuffle partitions, handling skew, and switching join strategies. (Module 17)

Analyzer — The Catalyst phase that resolves column/table names and types against the Catalog, turning an unresolved logical plan into a resolved one. (Module 08)

B

Broadcast hash join (BHJ) — A join strategy that sends a full copy of a small table to every executor so the large table is joined locally with no shuffle. Triggered by the broadcast() hint or automatically under spark.sql.autoBroadcastJoinThreshold. (Module 16)

Bucketing — Pre-partitioning and pre-sorting a table into a fixed number of buckets on a key at write time, so later joins on that key need no shuffle. (Modules 16, 17)

C

Cache / Persist — Storing a DataFrame's computed result (in memory and/or on disk) so repeated actions reuse it instead of recomputing. cache() uses the default level; persist(level) chooses the storage level. Lazy — must be materialized with an action. (Module 15)

Catalog — Spark's internal metadata registry of tables, views, columns, and types; the Analyzer consults it to resolve names. Browsable via spark.catalog. (Modules 08, 11)

Catalyst optimizer — Spark SQL's rule- and cost-based query optimizer; transforms a query through four phases (analysis, logical optimization, physical planning, code generation). (Module 08)

Cluster manager — The external master process (Standalone, YARN, Mesos, Kubernetes) that owns cluster machines and allocates resources to Spark applications. (Module 02)

Coalesce — A narrow transformation that reduces partition count by merging adjacent partitions on the same node — no shuffle. Use to shrink partitions cheaply. (Modules 03, 17)

Cost-Based Optimizer (CBO) — The part of Catalyst's physical planning that estimates the cost of candidate physical plans using data statistics and picks the cheapest. (Module 08)

D

DAG (Directed Acyclic Graph) — The one-way, no-loops graph of computation Spark builds from your transformations before execution. (Module 06)

DataFrame — A distributed, in-memory table with named columns and a schema; the recommended default API. Optimized by Catalyst; same performance across languages. (Modules 04, 10)

Data locality — Scheduling each task on the executor physically closest to its data to minimize network transfer (moving computation to data). (Module 03)

Dataset — A strongly-typed, type-safe structured API available only in Scala/Java; adds compile-time safety at the cost of serialization overhead. (Module 04)

Driver — The single process that runs your program, holds the SparkSession, builds the plan, and schedules tasks — the "brain" of the application. (Module 02)

Dynamic allocation — Letting Spark scale the number of executors up and down with the workload rather than using a fixed count. (Module 17)

E

Estimator — An MLlib component that learns from data via .fit(), producing a transformer (e.g., StringIndexer, a training algorithm). (Module 14)

Exchange — The word for a shuffle in a Spark query plan; each Exchange marks a wide transformation and a stage boundary. (Modules 06, 07)

Executor — A JVM worker process (one per worker node, typically) that runs tasks on data partitions and provides in-memory cache storage. (Module 02)

Execution modes — Cluster (driver + executors in the cluster; production), client (driver on the submitting machine; interactive), local (everything on one machine; dev/testing). (Module 02)

J

Job — All the work triggered by a single action; split into stages, which split into tasks. (Module 02)

L

Lazy evaluation — Spark records transformations without running them, deferring all computation until an action, so it can optimize the whole plan at once. (Module 06)

Lineage — The recorded chain of transformations from source to a given DataFrame; enables fault-tolerant recomputation of lost partitions. (Modules 05, 06)

M

MLlib — Spark's DataFrame-based machine learning library (transformers, estimators, pipelines). (Module 14)

N

Narrow transformation — A transformation where each input partition feeds at most one output partition (e.g., filter, select, withColumn); no shuffle, can be pipelined. (Module 05)

P

Pandas UDF (vectorized UDF) — A UDF that uses Apache Arrow to move data across the JVM↔Python boundary in columnar batches and process them with vectorized Pandas ops — far faster than a standard UDF. (Module 12)

Partition — A chunk of a DataFrame's rows living on one machine; the atomic unit of parallelism (one task per partition per core). (Module 03)

Pipeline — An MLlib estimator that chains ordered stages (transformers/estimators) into one repeatable workflow; .fit() yields a PipelineModel. (Module 14)

Predicate pushdown — An optimization that moves filters as close to the data source as possible so less data is read. (Modules 06, 08)

Project Tungsten — Spark's execution backend: off-heap binary storage (no per-object overhead, no GC pressure) and whole-stage code generation. (Module 08)

R

RDD (Resilient Distributed Dataset) — Spark's low-level, immutable, partitioned collection of raw objects; defined by dependencies, partitions, and a compute function. Opaque to Catalyst — no automatic optimization. (Module 04)

Repartition — A wide transformation that shuffles all data into n balanced partitions; the way to increase or rebalance partitions. (Modules 03, 17)

S

Schema — The column names and types of a DataFrame. Define it explicitly (DDL string or StructType) to skip costly inference. (Module 09)

Shuffle — Redistributing data across partitions/executors so records sharing a key meet; the most expensive operation (network + disk I/O). Caused by wide transformations. Phases: map → shuffle → sort → reduce. (Module 07)

Shuffle sort-merge join (SMJ) — The default join for two large tables: shuffle and sort both, then merge by key. (Module 16)

Skew (data skew) — When one partition/key holds disproportionately much data, making one task run far longer than its peers. AQE can mitigate it. (Module 17)

Small-file problem — Many tiny files/partitions bloating metadata and per-task overhead; fix with maxRecordsPerFile or coalesce. (Modules 03, 17)

SparkSession — The unified entry point to all Spark functionality; created with the builder pattern and getOrCreate(). (Modules 02, 09)

Stage — A group of tasks that run without moving data; stage boundaries fall at shuffles. (Modules 02, 07)

Storage level — How a cached DataFrame is stored: MEMORY_ONLY, MEMORY_ONLY_SER (compact), MEMORY_AND_DISK, DISK_ONLY, plus replicated variants. (Module 15)

Structured Streaming — Processing unbounded data as an ever-growing table using the same DataFrame API; supports event-time windows, output modes, and triggers. (Module 13)

T

Task — The smallest unit of execution: one computation on one partition on one core. (Module 02)

Transformation — A lazy operation that describes deriving a new DataFrame from an existing one (e.g., select, groupBy, join); returns a new DataFrame, runs nothing until an action. (Module 05)

Transformer — An MLlib component that applies a known rule via .transform() (e.g., OneHotEncoder, a trained model). (Module 14)

Temporary view — A session-scoped name registered for a DataFrame so it can be queried with SQL; stores no data. (Module 11)

U

UDF (User-Defined Function) — Custom Python logic run as a column function. Standard UDFs pay a per-row JVM↔Python serialization tax and are opaque to Catalyst — prefer built-ins, then Pandas UDFs. (Module 12)

Unified analytics engine — Spark's defining trait: one engine + consistent APIs for batch, SQL, streaming, ML, and graph. (Module 01)

W

Whole-stage code generation — Tungsten's technique of fusing a whole stage of operators into one compact JVM function, eliminating per-row virtual calls. (Module 08)

Wide transformation — A transformation where one output partition depends on many input partitions (e.g., groupBy, join, orderBy, distinct); forces a shuffle and a new stage. (Module 05)

PySpark Cheat Sheet

A condensed reference to the code you'll use most. Everything here is explained in the modules linked above.

Create a SparkSession

from pyspark.sql import SparkSession

spark = (SparkSession.builder
         .appName("MyApp")
         .config("spark.sql.shuffle.partitions", 8)
         .getOrCreate())

Read data (with an explicit schema)

schema = "date STRING, delay INT, distance INT, origin STRING, destination STRING"

df = (spark.read.format("csv")
      .option("header", "true")
      .schema(schema)                 # explicit — avoids inference pass
      .load("/path/data.csv"))

parquet_df = spark.read.parquet("/path/data.parquet")   # preferred format

Inspect

df.printSchema()      # columns + types (metadata only)
df.show(5)            # first rows (ACTION)
df.count()            # row count (ACTION)
df.columns            # list of column names
df.explain()          # physical plan (look for Exchange = shuffle, PushedFilters)

Core DataFrame operations

from pyspark.sql.functions import col, expr, count, avg, desc

df.select("origin", "delay")                                   # projection
df.filter((col("delay") > 60) & (col("origin") == "SFO"))      # selection (parenthesize!)
df.withColumn("delay_hrs", col("delay") / 60)                  # add/replace column
df.withColumnRenamed("delay", "delay_min")                     # rename
df.drop("distance")                                            # drop column(s)

(df.groupBy("origin")                                          # summarize (WIDE → shuffle)
   .agg(count("*").alias("n"), avg("delay").alias("avg_delay"))
   .orderBy(desc("n")))                                        # sort (WIDE → shuffle)

Spark SQL

df.createOrReplaceTempView("flights")
spark.sql("""SELECT origin, COUNT(*) AS n
             FROM flights GROUP BY origin ORDER BY n DESC LIMIT 5""").show()

Joins

from pyspark.sql.functions import broadcast

big.join(small, "id")                        # default: shuffle sort-merge join
big.join(broadcast(small), "id")             # broadcast small side (no big shuffle)
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", 50*1024*1024)   # tune (default 10MB)

Caching

df.cache()            # mark for caching (lazy)
df.count()            # materialize the cache NOW (action)
# ... reuse df across actions ...
df.unpersist()        # free memory when done

from pyspark import StorageLevel
df.persist(StorageLevel.MEMORY_ONLY_SER)     # compact form for large data

Partitioning

df.rdd.getNumPartitions()        # inspect count
df.coalesce(10)                  # shrink (narrow, no shuffle)
df.repartition(200)              # grow/rebalance (wide, full shuffle)

Pandas UDF (when built-ins won't do)

import pandas as pd
from pyspark.sql.functions import pandas_udf, col
from pyspark.sql.types import LongType

@pandas_udf(LongType())
def cubed(a: pd.Series) -> pd.Series:
    return a * a * a

df.select("id", cubed(col("id")))

High-value tuning configs

spark.conf.set("spark.sql.shuffle.partitions", 8)                  # match to workload (default 200)
spark.conf.set("spark.sql.adaptive.enabled", True)                # enable AQE
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", True)
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", True)
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", 10*1024*1024)

Safe vs. dangerous actions

df.show(20)              # safe: bounded
df.take(20)              # safe: bounded
df.write.parquet("/out") # safe: writes to sink
df.collect()             # DANGER on large data: pulls everything to the driver (OOM)
df.toLocalIterator()     # safer full-scan: partition-by-partition

Where to go next

You've reached the end of the guide. A natural progression from here:

Practice — run the companion notebook on the included sample data, then build a pipeline end to end: read with a schema, transform, join, aggregate, write out. Watch it in the Spark UI.
Read plans — run explain() on your queries and find the Exchange nodes; make a shuffle disappear with a broadcast join or an early filter.
Go deeper — the resources below expand every topic here with more examples and edge cases.

Thanks for reading — now go build something fast. 🚀

Sources & further reading

This guide synthesizes and re-explains material from these open resources — all freely available:

Official Apache Spark documentation

Books (free via Databricks)

Learning Spark, 2nd Edition — Jules S. Damji, Brooke Wenig, Tathagata Das & Denny Lee.
Spark: The Definitive Guide — Bill Chambers & Matei Zaharia.

Background reading

Databricks: What is Apache Spark?

Back to the guide

Return to the README for the full module index, or revisit any concept via the glossary links above.