2

I'm trying to add a header with a key of "order_id" and a value based on a property in the payload to my messages. I then send the result to a log sink where I can inspect the headers after the header processor. Here's the stream:

stream create --name add-header-to-message-stream 
  --definition 
    ":aptly-named-destination 

    > add-order_id-header: header-enricher          
      --header.enricher.headers='order_id=payload.order.id \\n fizz=\"buzz\"' 

    | log 
      --log.expression=headers"

I do not see keys of "order_id" or "fizz" in the headers map when I tail the log sink. I'm able to deploy the stream and run data through the pipeline with no errors. How do I add headers to my messages?

James Hines
  • 341
  • 1
  • 4
  • 12
  • I can only get it to work with one header at a time; there doesn't seem to be a way to pass a newline from the shell within a property. – Gary Russell Jun 20 '17 at 22:09
  • How are you able to get it to work in the shell? Even If I only set one header using the shell with single quotes around the entire expression and escaped single or double quotes around the header value, it still doesn't work. Adding headers using the header.enricher.headers config property works fine when running integration tests. – James Hines Jun 21 '17 at 05:04
  • See my answer for what I found works, and what doesn't. – Gary Russell Jun 21 '17 at 12:42

1 Answers1

3

This works fine for me, but only with a single header...

dataflow:>stream create foo --definition "time --fixedDelay=5 | 
  header-enricher --headers='foo=payload.substring(0, 1)' | 
  log --expression=#root " --deploy

With result

2017-06-21 08:28:38.459 INFO 70268 --- [-enricher.foo-1] log-sink : GenericMessage [payload=06/21/17 08:28:38, headers={amqp_receivedDeliveryMode=PERSISTENT, amqp_receivedRoutingKey=foo.header-enricher, amqp_receivedExchange=foo.header-enricher, amqp_deliveryTag=1, foo=0, amqp_consumerQueue=foo.header-enricher.foo, amqp_redelivered=false, id=302f1d5b-ba90

I am told that this...

--headers='foo=payload.substring(0, 1) \n bar=payload.substring(1,2)'

...or this...

--headers='foo=payload.substring(0, 1) \u000a bar=payload.substring(1,2)'

should work, but I get a parse error...

Cannot find terminating ' for string time --fixedDelay=5 | header-enricher --headers='foo=payload.substring(0, 1) 
 bar=payload.substring(1,2)' | log --expression=#root

...I am reaching out to the shell/deployer devs and will provide an update if I have one.

I tested with a literal value (single header) too...

dataflow:>stream create foo --definition "time --fixedDelay=5 | 
header-enricher --headers='foo=\"bar\"' | 
log --expression=#root " --deploy

2017-06-21 08:38:17.684 INFO 70916 --- [-enricher.foo-1] log-sink : GenericMessage [payload=06/21/17 08:38:17, headers={amqp_receivedDeliveryMode=PERSISTENT, amqp_receivedRoutingKey=foo.header-enricher, amqp_receivedExchange=foo.header-enricher, amqp_deliveryTag=8, foo=bar, amqp_consumerQueue=foo.header-enricher.foo, amqp_redelivered=false, id=a92f4908-af13-53aa-205d-e25e204d04a3, amqp_consumerTag=amq.ctag-X51lhhRWBbEDVSyzp3rGmg, contentType=text/plain, timestamp=1498048697684}]

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • The related GH issue: https://github.com/spring-cloud-stream-app-starters/header-enricher/issues/6 – Artem Bilan Jun 21 '17 at 12:46
  • Thanks Gary, you've been very helpful. As a workaround of sorts, I decided to just create a custom processor for my use case. Here's a gist: https://gist.github.com/jhines2k7/8d10c59f6479536e1e5fa7ccb4c554b5 I'll give it a try using the RabbitMQ binder. Right now I'm using Kafka with my Spring Cloud Dataflow apps. I should probably mention the unit test will fail. I wanted a quick and dirty way to experiment with a solution I thought could work. – James Hines Jun 21 '17 at 18:13
  • We [fixed the data flow servers](https://github.com/spring-cloud/spring-cloud-dataflow/pull/1540). You could try a snapshot. It will be in the next release. – Gary Russell Jun 21 '17 at 20:04
  • 1
    Gary, your earlier comment gave me clue. There is a Kafka binder property called "spring.cloud.stream.kafka.binder.headers". You have to set the value of that config property to the key or keys of the header you want to be transported by the binder. In my case, I set a deployer property like so: "app.header-enricher.spring.cloud.stream.kafka.binder.headers=order_id". In the shell: stream create --name header-processor-stream --definition "time --trigger.time-unit=SECONDS --trigger.fixed-delay=5 | header-enricher --header.enricher.headers='order_id=\"foo\"' | log --log.expression=headers" – James Hines Jun 21 '17 at 21:08
  • I'll add a more properly formatted answer this evening. Love the tech! – James Hines Jun 21 '17 at 21:10