0

The Question is:

Complete the writeToBronze function to perform the following tasks:

  1. Write the stream from gamingEventDF -- the stream defined above -- to a bronze Delta table in path defined by outputPathBronze.
  2. Convert the (nested) input column client_event_time to a date format and rename the column to eventDate
  3. Filter out records with a null value in the eventDate column
  4. Make sure you provide a checkpoint directory that is unique to this stream

Code :

def writeToBronze(sourceDataframe, bronzePath, streamName):
  (sourceDataframe    
     .withColumn("eventDate", 
        to_date(col("eventParams.client_event_time"), "yyyy-MM-dd"))
     .filter(col("eventDate").isNotNull())     
     .writeStream
     .format("delta")
     .option("checkpointLocation", f"{bronzePath}_checkpoint")
     .queryName(streamName)
     .outputMode("append") 
     .start(outputPathBronze)
  )
    
writeToBronze(gamingEventDF, outputPathBronze, "bronze_stream")

Below is the Error I get:

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
Sivaani N
  • 19
  • 1
  • 8

1 Answers1

1

I was stuck on this too but have just figured it out In the provided template for this capstone cell they have included the line: .start(outputPathBronze) This is referring to a parameter that is outside of the function, not the path passed in as an argument. The reality check passes a different path in that variable. If you change this line to use the variable bronzePath you should be able to get past that step (now I have to get my count to match their expected count...)

Rebekah Taylor
  • 11
  • 1