Scala Programming Projects

Aggregating streaming transactions

Add a new paragraph with the following code:

val aggDfStream = {   dfStream    .withWatermark("timestamp", "1 second")    .groupBy(window($"timestamp", "10 seconds").as("window"))    .agg(      count($"tid").as("count"),       avg("price").as("avgPrice"),      stddev("price").as("stddevPrice"),      last("price").as("lastPrice"),      sum("amount").as("sumAmount")    )    .select("window.start", "count", "avgPrice", "lastPrice",     "stddevPrice", "sumAmount")}

This code is very similar to the one we wrote in the section Drawing our first chart. The only difference is the call to withWatermark, but the rest of the code is the same. This is one of the main benefits of using Spark-structured streaming—we can reuse the same code for transforming batch ...