Interview Prep Guide
Apache Spark Data Engineer Interview Questions and Production Scenarios
Apache Spark questions covering execution architecture, DAGs, Catalyst, memory, shuffles, Structured Streaming, watermarks, scaling, and incident diagnosis.
Spark Engine Fundamentals
What are the responsibilities of the Spark driver and executors?
The driver builds and coordinates the application plan, while executors run tasks, hold cached or shuffle data, and report results and metrics.
How do jobs, stages, tasks, and partitions relate in Spark?
An action creates a job; shuffle boundaries divide it into stages; each stage runs tasks over its partitions.
What do Catalyst, Tungsten, and Adaptive Query Execution do?
Catalyst analyzes and optimizes queries, Tungsten improves physical execution and memory efficiency, and AQE adjusts parts of a running plan using observed statistics.
What is the difference between narrow and wide transformations?
Narrow transformations consume local parent partitions, while wide transformations redistribute data across partitions through a shuffle.
Advanced Spark and Streaming Questions
How does Spark manage execution and storage memory?
Spark shares executor memory between execution work and cached storage, subject to configured limits, eviction, serialization, and off-heap choices.
How do checkpoints and sinks affect exactly-once behavior in Structured Streaming?
Checkpoints track query progress and state, but end-to-end exactly-once behavior also depends on replayable sources and an idempotent or transactional sink.
How do watermarks handle late data in stateful streaming?
A watermark estimates event-time progress and bounds how long state is retained; records later than the permitted threshold may be dropped from stateful results.
How do dynamic allocation and speculative execution solve different problems?
Dynamic allocation changes executor capacity with workload demand; speculation duplicates unusually slow tasks to reduce straggler impact.
Apache Spark Production Scenarios
One task runs for 40 minutes while thousands finish in seconds. What does that indicate?
It usually indicates skew, an unusually large partition, slow input locality, or a failing executor; compare task input, shuffle, spill, and host metrics.
A Structured Streaming query is falling behind its input rate. How do you respond?
Measure processing and input rates, batch duration, state size, source lag, shuffle, sink latency, and cluster utilization, then address the limiting stage.
A Spark upgrade changes a production query result. How do you investigate safely?
Compare logical and physical plans, SQL compatibility settings, null and timestamp behavior, UDFs, data-source readers, and representative outputs between versions.
Spark cluster cost increased without a matching data-volume increase. What would you inspect?
Break cost down by job, runtime, executor utilization, retries, idle time, shuffle, storage I/O, autoscaling, and recent code or configuration changes.
Spark Engine and Streaming Depth
How do jobs, stages, and tasks relate in Spark?
An action creates a job, shuffle boundaries divide the job into stages, and each stage runs tasks across its input partitions.
What do Catalyst and Tungsten do?
Catalyst analyzes and optimizes logical and physical query plans; Tungsten improves execution through memory management, binary processing, and code generation.
How does Adaptive Query Execution improve Spark workloads?
AQE uses runtime statistics to revise decisions such as shuffle partition counts, skew handling, and join strategies after execution has begun.
Why does serialization matter in Spark?
Spark repeatedly moves and stores objects, so inefficient serialization increases CPU, network traffic, memory use, and garbage collection pressure.
What is the difference between checkpointing and caching?
Caching accelerates recomputation but retains lineage; checkpointing writes a reliable materialized state and truncates lineage for recovery or plan control.
How do you control input rate in Structured Streaming?
Match trigger and source-rate settings to sustainable processing capacity, monitor backlog and batch duration, and scale or reduce work before state and latency grow without bound.
How do watermarks affect correctness and state in Spark streaming?
A watermark defines how late event-time data may arrive before Spark can finalize windows and evict old state, trading completeness for bounded resource use.
When is speculative execution useful in Spark?
It can launch duplicate copies of unusually slow tasks and keep the first result, helping with transient stragglers but not deterministic skew or consistently slow partitions.
Apache Spark Coding Round
Write a skew-resistant Spark join for one hot key
Salt only the known hot keys, replicate the matching small-side rows across the same salt range, join on key plus salt, and remove the helper column afterward. from pyspark.sql import functions as F salt_count = 8 hot_keys = ["UNKNOWN"] large_salted = large.withColumn( "salt", F.when(F.col("join_key").isin(hot_keys), (F.rand() * salt_count).cast("int")) .otherwise(F.lit(0)) ) small_salted = (small .withColumn( "salt_values", F.when(F.col("join_key").isin(hot_keys), F.sequence(F.lit(0), F.lit(salt_count - 1))) .otherwise(F.array(F.lit(0))) ) .withColumn("salt", F.explode("salt_values")) .drop("salt_values")) result = large_salted.join(small_salted, ["join_key", "salt"]).drop("salt") Confirm the hot-key distribution and task skew in the Spark UI before applying this pattern.
Build an idempotent incremental Delta Lake merge
Resolve multiple source events per key before merging, then advance the external watermark only after the transaction succeeds. from delta.tables import DeltaTable from pyspark.sql import functions as F from pyspark.sql.window import Window w = Window.partitionBy("customer_id").orderBy( F.col("event_time").desc(), F.col("event_id").desc() ) changes = (source_changes .withColumn("rn", F.row_number().over(w)) .filter("rn = 1") .drop("rn")) target = DeltaTable.forPath(spark, target_path) (target.alias("t") .merge(changes.alias("s"), "t.customer_id = s.customer_id") .whenMatchedDelete(condition="s.operation = 'D'") .whenMatchedUpdateAll(condition="s.operation <> 'D'") .whenNotMatchedInsertAll(condition="s.operation <> 'D'") .execute())
Write a stateful Structured Streaming aggregation with a watermark
Parse event time, bound state with a business-approved lateness threshold, and store checkpoints separately from output. from pyspark.sql import functions as F result = (events .withColumn("event_time", F.to_timestamp("event_time")) .withWatermark("event_time", "20 minutes") .groupBy(F.window("event_time", "5 minutes"), "product_id") .agg(F.sum("amount").alias("revenue"))) query = (result.writeStream .outputMode("append") .option("checkpointLocation", checkpoint_path) .format("delta") .start(output_path)) The watermark limits retained state; it does not guarantee that every arbitrarily late event is included.
Control Spark output file sizes without forcing one partition
Estimate a useful partition count from output size, repartition by meaningful low-cardinality columns when required, and cap records per file. target_partitions = 96 (result .repartition(target_partitions, "event_date") .write .mode("overwrite") .option("maxRecordsPerFile", 2_000_000) .partitionBy("event_date") .parquet(output_path)) Validate actual compressed file sizes and downstream read behavior; row count alone does not predict byte size.
Inspect and verify a Spark join strategy before tuning it
Explain the formatted plan, verify table statistics and size, then apply a broadcast hint only when the small side safely fits executor memory. from pyspark.sql import functions as F candidate = facts.join(dimensions, "dimension_id") candidate.explain(mode="formatted") optimized = facts.join( F.broadcast(dimensions.select("dimension_id", "category")), "dimension_id" ) optimized.explain(mode="formatted") The final answer should connect the plan to Spark UI evidence such as shuffle bytes, spill, task skew, and executor memory pressure.