0

Each record in Kafka is a list of dictionaries (serialized using lambda x: json.dumps(x).encode('utf-8') and I'm trying to use Faust. I understand that I can pass the schema by creating a class like so:

class row(faust.Record):
    ID: int 
    Severity: str 
    Start_Time: str 
    End_Time: str 
    Start_Lat: str 
    Start_Lng: str 
    End_Lat: str 
    End_Lng: str 
    Distance_mi: str 
    Description: str 
    Number: str 
    Street: str 
    Side: str  
    ...

As I mentioned I have a list of this class. For something like pydantic I could use:

from typing import List
class rows:
    __root__:List[row]

I tried passing value_type=list(row) and mimicking pydantic's way, but both failed.

How do I do that in Faust?

Note: Not passing the value_type parameter to app.topic works fine. I would like to add this extra layer of verification to the process.

Mohamed Yasser
  • 641
  • 7
  • 17

1 Answers1

0

I'd create a

class Rows(Record):
  entries: List[row]

and then value_type=Rows

Georgi Tenev
  • 338
  • 4
  • 18
  • I ditched Faust for that project due to its slow read and low throughput. I'll try to verify your answer as soon as possible. However, doesn't that require the list to be a JSON value with a key 'entries'? – Mohamed Yasser Aug 08 '22 at 10:53