2

One of the field names in my JSON is "User-Agent". KSQL doesn't like the dash when I try to create a stream with that field name. I've tried single, double quotes, escaping it various ways--no luck.

How do I get that field from json in a topic to be in a KSQL stream?

Here's a log of some of my attempts:

ksql> create stream tmpstream (startTimeEpoch bigint, type VARCHAR, src VARCHAR, 'user-agent' varchar) with (KAFKA_TOPIC='foo', VALUE_FORMAT='JSON');
line 1:76: extraneous input ''user-agent'' expecting {'ADD', ...}

ksql> create stream tmpstream (startTimeEpoch bigint, type VARCHAR, src VARCHAR, user-agent varchar) with (KAFKA_TOPIC='foo', VALUE_FORMAT='JSON');
line 1:80: extraneous input '-' expecting {'ADD', ...}

ksql> create stream tmpstream (startTimeEpoch bigint, type VARCHAR, src VARCHAR, "user-agent" varchar) with (KAFKA_TOPIC='foo', VALUE_FORMAT='JSON');

 Message
----------------
 Stream created
----------------
ksql> select * from tmpstream;
Code generation failed for SelectValueMapper
Caused by: Line 1, Column 15: Expression "TMPSTREAM_user" is not an rvalue

ksql> drop stream tmpstream;
 Message
--------------------------------
 Source TMPSTREAM was dropped.
--------------------------------
ksql> create stream tmpstream (startTimeEpoch bigint, type VARCHAR, src VARCHAR, "user\-agent" varchar) with (KAFKA_TOPIC='foo', VALUE_FORMAT='JSON');

 Message
----------------
 Stream created
----------------
ksql> select * from tmpstream;
Code generation failed for SelectValueMapper
Caused by: Line 1, Column 15: Expression "TMPSTREAM_user" is not an rvalue
medloh
  • 941
  • 12
  • 33

1 Answers1

4

Unfortunately, having a dash within a field name is not supported.

KSQL issues

Please vote on the tickets which are more crticial for your use case.


One reason would be that all fields should adhere to Java variable naming conventions (minus capitalization rules). Meaning, alphanumeric and underscore characters are allowed.


The workarounds would require you to edit the producer code, or write a Kafka Streams task that can do a map() renaming to a supported field name structure.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245