how to create pipeline in databricks using delta live table to read data from kafka

Question

1. read from kafka this worked

raw_kafka_test = (spark.readStream
.etc
)
@dlt.table(
    table_properties={"pipelines.reset.allowed":"false"}
)
def raw_kafka():
    return raw_kafka_test

2. read from delta live table not worked

@dlt.table(
    comment="real schema for Kakfa payload",
    temporary=True
)
def data_kafka():
  return (
    dlt.read_stream("raw_kafka")

  )

what should i do with read from delta live table

Answer 1

You need to put spark.readStream inside the function, not outside of it.

@dlt.table(
    table_properties={"pipelines.reset.allowed":"false"}
)
def raw_kafka():
    raw_kafka_test = (spark.readStream
.etc
)
    return raw_kafka_test

@dlt.table(
    comment="real schema for Kakfa payload",
    temporary=True
)
def data_kafka():
  return dlt.read_stream("raw_kafka")

In general, all operations should be defined inside the functions - outside you need to have only things like constants definitions, etc. See documentation for more details.