Transformations & Actions

Part III — The Execution Engine · Module 5 of 18 Prerequisites: Spark Architecture, Partitions & Parallelism, RDDs, DataFrames, Datasets You will learn: the single most important distinction in day-to-day Spark — transformations (which describe work) vs. actions (which trigger it) — why Spark data is immutable, and the make-or-break difference between narrow and wide transformations that determines whether Spark must shuffle.

---

Every Spark operation is one of two things

You've now met the data structures. This module is about operating on them — and the beautiful thing is that in Spark, every operation you can perform falls into exactly one of two categories:

• Transformations describe how you want to change a DataFrame. They are instructions, recorded but not run.
• Actions trigger the actual computation and produce a result.

Internalize this split and Spark stops being mysterious. Nearly every "why didn't my code do anything?" or "why did everything suddenly run at once?" moment traces back to whether you called a transformation or an action.

---

Transformations: instructions, not execution

A transformation is a logical instruction that specifies how to derive a new DataFrame from an existing one. The critical, counterintuitive fact: transformations do not run when you write them. Spark simply records them.

Consider:

from pyspark.sql.functions import col

# None of these lines processes any data. Spark just records the recipe.
df2 = df.select("State", "Color", "Count")
df3 = df2.filter(col("Count") > 100)
df4 = df3.orderBy(col("Count").desc())
# ...still nothing has been computed. No rows have been read.

If you ran this in a notebook, it would return almost instantly — not because Spark is fast here, but because it hasn't done anything yet. It has only built up a plan: "when someone finally asks for a result, select these columns, then filter, then sort." This deferred behavior is lazy evaluation, and it's so central it gets its own module next (06). For now, just hold the mental image: transformations are you writing down a recipe, not cooking.

Immutability: transformations never change the original

Spark's data structures are immutable — they cannot be modified in place. A transformation never mutates the DataFrame you call it on; it returns a brand-new DataFrame representing the transformed state. In the snippet above, df is untouched after every line; df2, df3, df4 are each new logical DataFrames.

This isn't a quirk — it's what makes Spark's fault tolerance possible. Because each DataFrame is derived from its parents by a known transformation, Spark can always recompute any piece that gets lost (a machine dies mid-job) by replaying the recipe from the source. That recorded chain of derivations is called lineage, and immutability is what keeps the lineage trustworthy: the inputs a transformation depended on can never have changed underneath it.

Analogy — a recipe vs. the cooked dish. Writing transformations is like writing down a recipe: "chop onions, then sauté, then add stock." Reading the recipe changes nothing in your kitchen — no onion is harmed. Immutability means each step produces a new bowl rather than altering the previous one, so if you drop a bowl (lose a partition), you can always re-cook it from the written steps. The cooking itself only starts when someone asks to eat — that's an action.

---

Actions: the trigger

An action is an operation that triggers the physical execution of all the accumulated transformations. When you call an action, Spark finally looks at the whole recipe it has recorded, optimizes it, and runs it across the cluster.

Actions do one of three things:

1. View data in the console (e.g., show()).
2. Collect data to native objects in the driver (e.g., collect(), take(), count()).
3. Write data out to a storage sink (e.g., write.save()).

# Building the plan (transformations — instant, nothing runs):
result = (df
          .select("State", "Color", "Count")
          .filter(col("Count") > 100)
          .orderBy(col("Count").desc()))

# The moment of truth (an ACTION — NOW Spark reads data and computes):
result.show(10)        # triggers execution and prints rows
print(result.count())  # another action → another job

Recall the vocabulary from Module 02: one action triggers one job. Every time you call show(), count(), or save(), you kick off a fresh job that Spark plans and executes. (This is also why calling multiple actions on the same un-cached DataFrame recomputes it each time — a motivation for caching in Module 15.)

A quick reference

Common transformations (lazy)	Common actions (trigger execution)
select(), filter() / where()	show()
groupBy(), agg()	count()
join(), orderBy() / sort()	collect(), take(n), first()
withColumn(), withColumnRenamed(), drop()	write.save() / write.format(...).save()
repartition(), coalesce()	foreach()

The tell: if an operation returns a new DataFrame, it's a transformation (lazy). If it returns a value, a list, or writes to disk — something that isn't a DataFrame — it's an action (it runs).

---

Narrow vs. wide: the distinction that governs performance

Not all transformations are equal. They split into two kinds based on a single question: to compute one output partition, does Spark need data from just one input partition, or from many? This is the difference between narrow and wide transformations, and it's the biggest lever on Spark performance.

Narrow transformations: no data movement

In a narrow transformation, each input partition contributes to at most one output partition. The computation for a given output partition needs only the data already sitting in a single input partition — nothing has to travel across the network.

Examples: filter(), select(), withColumn(), contains().

Because the work stays local, Spark can do something clever called pipelining: it chains multiple narrow transformations together and runs them in a single pass, in memory, on each partition — without writing intermediate results to disk. select().filter().withColumn() on a partition all happen back-to-back on the same core, on the same data, with zero shuffling. Narrow transformations are cheap and embarrassingly parallel.

Analogy — each proofreader works alone. Back to our torn-up book from Module 03. A narrow transformation is like telling every proofreader, "highlight all the typos in your section." Each works entirely within their own stack of pages — no one needs to consult anyone else. Ten proofreaders finish in parallel with no coordination. Fast.

Wide transformations: the shuffle

In a wide transformation, a single output partition can depend on data from many input partitions spread across the cluster. To compute the result, Spark must physically move data across the network so that related records end up together. This movement is called a shuffle, and it's the most expensive thing Spark does.

Examples: groupBy(), join(), orderBy() / sort(), repartition(), distinct().

Why so expensive? Because a shuffle breaks the data locality we prized in Module 03. Records that were happily local now have to be sent over the network to new partitions based on their key. Worse, for fault tolerance Spark writes these intermediate shuffle files to local disk before the exchange — so a wide transformation involves network and disk I/O, not just computation. A shuffle is also where Spark splits your job into separate stages (Module 02): a stage boundary falls exactly at each shuffle.

Analogy — regrouping the whole class by birthday month. Now imagine you ask the ten proofreaders to physically regroup so that everyone born in the same month sits together. Suddenly there's chaos: people get up, cross the room, and reshuffle into twelve new groups. Nobody can do this alone — it requires everyone moving at once, coordinated across the whole room. That upheaval is a shuffle. groupBy("birth_month") is exactly this: records must move so that all rows sharing a key land in the same partition.

Why you should care

This distinction is the practical intuition behind most Spark tuning. When a job is slow, an experienced engineer's first instinct is: where are the shuffles? Every wide transformation is a candidate for optimization — can you filter before the shuffle to move less data? Can you replace a shuffle-heavy join with a broadcast join (Module 16)? Can you avoid an unnecessary distinct() or orderBy()? You can't eliminate shuffles entirely — some regrouping is genuinely required — but knowing which operations trigger them tells you exactly where the cost lives. We devote all of Module 07 to the shuffle for this reason.

	Narrow	Wide
Input→output partitions	One input → one output	Many inputs → one output
Data movement	None (stays local)	Shuffle across network
Disk I/O	No intermediate writes	Writes shuffle files to disk
Cost	Cheap, pipelined	Expensive
Stage boundary?	No	Yes — a new stage starts
Examples	filter, select, withColumn	groupBy, join, orderBy, distinct

---

Key takeaways

• Every Spark operation is either a transformation (describes work, lazy, returns a new DataFrame) or an action (triggers execution, returns a value or writes output). One action → one job.
• Spark data is immutable: transformations never mutate; they return new DataFrames. This enables lineage and fault tolerance — lost partitions are recomputed from the recorded recipe.
• Narrow transformations (filter, select, withColumn) need only one input partition per output partition, cause no data movement, and can be pipelined in one in-memory pass.
• Wide transformations (groupBy, join, orderBy, distinct) need data from many partitions, forcing a shuffle — network and disk I/O — and each shuffle starts a new stage. Shuffles are the most expensive operation in Spark.
• Diagnosing performance almost always starts with "where are the shuffles?"

---

Check your understanding

1. You write five chained transformations in a notebook cell and it returns in milliseconds. Did Spark process your data? Explain what actually happened.
2. Classify each as narrow or wide, and say whether it causes a shuffle: select(), groupBy(), filter(), orderBy(), withColumn().
3. Why does immutability make Spark's fault tolerance possible? Tie your answer to the concept of lineage.

---

Up next

Lazy Evaluation & the DAG — Lazy Evaluation & the DAG. We kept saying transformations are "recorded but not run." Next we look at what Spark records — the lineage and the DAG — and why waiting until the last moment lets Spark produce a dramatically more efficient plan than running each step eagerly.