Databricks Certified Associate Developer for Apache Spark 3.5 - Python Sample Questions:

1. 42 of 55.
A developer needs to write the output of a complex chain of Spark transformations to a Parquet table called events.liveLatest.
Consumers of this table query it frequently with filters on both year and month of the event_ts column (a timestamp).
The current code:
from pyspark.sql import functions as F
final = df.withColumn("event_year", F.year("event_ts")) \
.withColumn("event_month", F.month("event_ts")) \
.bucketBy(42, ["event_year", "event_month"]) \
.saveAsTable("events.liveLatest")
However, consumers report poor query performance.
Which change will enable efficient querying by year and month?

A) Add .sortBy() after .bucketBy()
B) Change the bucket count (42) to a lower number
C) Replace .bucketBy() with .partitionBy("event_year", "event_month")
D) Replace .bucketBy() with .partitionBy("event_year") only

2. A data engineer needs to write a Streaming DataFrame as Parquet files.
Given the code:

Which code fragment should be inserted to meet the requirement?
A)

B)

C)

D)

Which code fragment should be inserted to meet the requirement?

A) .format("parquet")
.option("path", "path/to/destination/dir")
B) .option("format", "parquet")
.option("location", "path/to/destination/dir")
C) CopyEdit
.option("format", "parquet")
.option("destination", "path/to/destination/dir")
D) .format("parquet")
.option("location", "path/to/destination/dir")

3. 17 of 55.
A data engineer has noticed that upgrading the Spark version in their applications from Spark 3.0 to Spark 3.5 has improved the runtime of some scheduled Spark applications.
Looking further, the data engineer realizes that Adaptive Query Execution (AQE) is now enabled.
Which operation should AQE be implementing to automatically improve the Spark application performance?

A) Dynamically switching join strategies
B) Optimizing the layout of Delta files on disk
C) Improving the performance of single-stage Spark jobs
D) Collecting persistent table statistics and storing them in the metastore for future use

4. 24 of 55.
Which code should be used to display the schema of the Parquet file stored in the location events.parquet?

A) spark.read.parquet("events.parquet").printSchema()
B) spark.sql("SELECT schema FROM events.parquet").show()
C) spark.sql("SELECT * FROM events.parquet").show()
D) spark.read.format("parquet").load("events.parquet").show()

5. 46 of 55.
A data engineer is implementing a streaming pipeline with watermarking to handle late-arriving records.
The engineer has written the following code:
inputStream \
.withWatermark("event_time", "10 minutes") \
.groupBy(window("event_time", "15 minutes"))
What happens to data that arrives after the watermark threshold?

A) Records that arrive later than the watermark threshold (10 minutes) will automatically be included in the aggregation if they fall within the 15-minute window.
B) Data arriving more than 10 minutes after the latest watermark will still be included in the aggregation but will be placed into the next window.
C) The watermark ensures that late data arriving within 10 minutes of the latest event time will be processed and included in the windowed aggregation.
D) Any data arriving more than 10 minutes after the watermark threshold will be ignored and not included in the aggregation.

Solutions: