7

I have two spark streams, in the first comes data related to products: their price to the supplier, the currency, their description, the supplier id. These data are enriched by the category, guessed by the analysis of the description and the price in dollars. Then they are saved in a parquet dataset.

The second stream contains data on the auctioning of these products, then the cost at which they were sold and the date.

Given the fact that a product can arrive in the first stream today and be sold in a year, how can I join the second stream with all the history contained in the parquet dataset of the first stream?

The result to be clear should be the average daily earnings per price range ...

Claudio D'Alicandro
  • 73
  • 5

2 Answers2

1

I found a possible solution with snappydata, using its mutable DataFrame:

https://www.snappydata.io/blog/how-mutable-dataframes-improve-join-performance-spark-sql

The reported example is very similar to the one described by claudio-dalicandro

giorrrgio
  • 66
  • 5
0

If you are leveraging structured streaming in Spark then you can load the parquet files of first stream into a dataframe.

parquetFileDF = spark.read.parquet("products.parquet")

Then you can get your second stream and join with the parquet file.

streamingDF = spark.readStream. ...
streamingDF.join(parquetFileDF, "type", "right_join")

Even you can join with your first stream to second stream as well.

Hope, this helps.

Gourav Dutta
  • 533
  • 4
  • 10
  • Unfortunately I'm using the classic streams, but even if it were not so with your solution I would not consider the updates of the parquet, which is rewritten with a window of 30 minutes from the first stream... If there was a way to read the dataframe every 30 minutes would be great! Alternatively it would be enough for me to be able to join two structured streams... – Claudio D'Alicandro Jan 18 '18 at 09:52