2

WSO2 SP 4.3.0 seems not to work with JSON events bigger than 1024 characters. Is there any limitation?

I'm using WSO2 Stream Processor version 4.3.0. I have a Shiddi application deployed using an HTTP source with JSON mapping. Everything works ok as long as the JSON event is less than 1024 characters. If the JSON's lenght is more than 1024 characters, then it looks like SP is splitting the message into several different events, and it fails when trying to parse the JSON content.

It is easy to reproduce using the included samples, "ReceiveHTTPInJsonFormatWithDefaultMapping".

@Source(type = 'http', receiver.url='http://localhost:8006/productionStream', basic.auth.enabled='false',
    @map(type='json'))
define stream SweetProductionStream (name string, amount double);

@sink(type='log')
define stream LowProductionAlertStream (name string, amount double);

-- passthrough data in the SweetProductionStream into LowProducitonAlertStream

@info(name='query1')
from SweetProductionStream
select *
insert into LowProductionAlertStream;

Then try to send a small event:

curl -X POST -d "{\"event\": {\"name\":\"sugar\",\"amount\": 20.5}}"  http://localhost
:8006/productionStream --header "Content-Type:application/json"

This will work. Then send a long event (more than 1024 characters)

curl -X POST -d "{\"event\": {\"name\":\"sugarlongggggggggggggggggg....gggggg\",\"amount\": 20.5}}"  http://localhost
:8006/productionStream --header "Content-Type:application/json"

In the logs you can see:

ERROR {org.wso2.extension.siddhi.map.json.sourcemapper.JsonSourceMapper} - Invalid Json String :{"event": {"name":"sugarlonggggggggggggggggggggggggggggggggggggggggggggggggggggggggg
26/3/2019 20:38:15[2019-03-26 19:38:15,789] ERROR {org.wso2.extension.siddhi.map.json.sourcemapper.JsonSourceMapper} - Invalid Json String :gggggggggggggggggggggggggggggggggggggggggggggggggggggggg","amount": 20.5}}
26/3/2019 20:38:15[2019-03-26 19:38:15,791] ERROR {org.wso2.transport.http.netty.common.Util} - Remote client closed the connection before completing outbound response io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: cannot send more responses than requests
26/3/2019 20:38:15  at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106)
26/3/2019 20:38:15  at io.netty.handler.codec.MessageToMessageCodec.write(MessageToMessageCodec.java:116)
26/3/2019 20:38:15  at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
26/3/2019 20:38:15  at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
26/3/2019 20:38:15  at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
26/3/2019 20:38:15  at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
26/3/2019 20:38:15  at io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:304)
26/3/2019 20:38:15  at io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:137)
26/3/2019 20:38:15  at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
26/3/2019 20:38:15  at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768)
26/3/2019 20:38:15  at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749)
26/3/2019 20:38:15  at io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117)
26/3/2019 20:38:15  at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
26/3/2019 20:38:15  at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802)
26/3/2019 20:38:15  at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814)
26/3/2019 20:38:15  at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794)
26/3/2019 20:38:15  at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831)
26/3/2019 20:38:15  at org.wso2.transport.http.netty.contractimpl.HttpOutboundRespListener.writeOutboundResponseHeaderAndBody(HttpOutboundRespListener.java:186)
26/3/2019 20:38:15  at org.wso2.transport.http.netty.contractimpl.HttpOutboundRespListener.writeOutboundResponse(HttpOutboundRespListener.java:138)
26/3/2019 20:38:15  at org.wso2.transport.http.netty.contractimpl.HttpOutboundRespListener.lambda$null$35(HttpOutboundRespListener.java:94)
26/3/2019 20:38:15  at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
26/3/2019 20:38:15  at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
26/3/2019 20:38:15  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
26/3/2019 20:38:15  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
26/3/2019 20:38:15  at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
26/3/2019 20:38:15  at java.lang.Thread.run(Thread.java:748)
26/3/2019 20:38:15Caused by: java.lang.IllegalStateException: cannot send more responses than requests
26/3/2019 20:38:15  at io.netty.handler.codec.http.HttpContentEncoder.encode(HttpContentEncoder.java:110)
26/3/2019 20:38:15  at io.netty.handler.codec.http.HttpContentEncoder.encode(HttpContentEncoder.java:51)
26/3/2019 20:38:15  at io.netty.handler.codec.MessageToMessageCodec$1.encode(MessageToMessageCodec.java:67)
26/3/2019 20:38:15  at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88)
26/3/2019 20:38:15  ... 25 more
antonyh
  • 2,131
  • 2
  • 21
  • 42
jlmate
  • 23
  • 2

2 Answers2

3

It is a known issue and is fixed from the siddhi-io-http-1.0.40.

In order to apply the latest version on your Stream Processor pack, please follow the below steps,

  1. Download siddhi-io-http-1.0.40.jar from the following link. https://store.wso2.com/store/assets/analyticsextension/details/63a5db75-586e-48d0-8244-4c16774e8846
  2. Replace the existing siddhi-io-http-1.0.39.jar which can be found in the /lib directory with the latest siddhi-io-http-1.0.40.jar.
Niveathika
  • 1,319
  • 2
  • 8
  • 17
  • For future reference - the current version is higher than 1.0.40 (at the time of writing this 1.2.1) but it's not compatible with the other libs matching the faulty 1.0.39... so if you do see this error and are running 1.0.39 migrate just a notch up to 1.0.40 and don't go the full monty... Direct link to 1.0.40: https://repo1.maven.org/maven2/org/wso2/extension/siddhi/io/http/siddhi-io-http/1.0.40/siddhi-io-http-1.0.40.jar – roy650 Jul 02 '19 at 13:34
2

This issue has been already identified[1] and fixed in the latest version. In order to apply the latest version to SP, please follow the steps given below.

  1. Download the new version of siddhi-io-http extension from WSO2 Extension Store[2].
  2. Replace the existing siddhi-io-http-x.x.x.jar which is in SP-HOME/lib directory with the above downloaded one.
  3. Restart the server.

[1] https://github.com/wso2-extensions/siddhi-io-http/issues/84

[2] https://store.wso2.com/store/assets/analyticsextension/details/63a5db75-586e-48d0-8244-4c16774e8846

Minudika
  • 851
  • 6
  • 23