0

Using Netty, I'm receiving multiple asynchronous messages from a framework on multiple threads. I need to send these messages to a network device (UDP) which uses a synchronous, stateful protocol. So, I need to use a state variable, and only allow one message to be sent at a time, which should only happen when the client is in the "idle" state.

In addition, the state-machine will need to send it's own internally generated messages - retrys - ahead of whatever is waiting in queue. For this use case I know how to inject messages into the pipeline, which would work as long as outbound messages can be held at the head of the pipeline.

Any idea how to control the output using a client state?

TIA

tlum
  • 913
  • 3
  • 13
  • 30

1 Answers1

0

I've come up with a proof-of-concept / proposed solution, although, hopefully someone knows a better way. While this works as intended it introduces a number of undesirable side effects which would need to be solved for.

  • Run a Blocking Write Handler on its own thread.
  • Put the thread to sleep if the state isn't idle, and wake it when it becomes Idle.
  • If the State was Idle, or becomes Idle, send the message on its way.

This is the bootstrap I used

public class UDPConnector {
    
    public Init() {

        this.workerGroup = EPOLL ? new EpollEventLoopGroup() : new NioEventLoopGroup();
        this.blockingExecutor = new DefaultEventExecutor();

        bootstrap = new Bootstrap()
                .channel(EPOLL ? EpollDatagramChannel.class : NioDatagramChannel.class)
                .group(workerGroup)
                .handler(new ChannelInitializer<DatagramChannel>() {
                    @Override
                    public void initChannel(DatagramChannel ch) throws Exception {
                        ch.pipeline().addLast("logging", new LoggingHandler());
                        ch.pipeline().addLast("encode", new RequestEncoderNetty());
                        ch.pipeline().addLast("decode", new ResponseDecoderNetty());
                        ch.pipeline().addLast("ack", new AckHandler());
                        ch.pipeline().addLast(blockingExecutor, "blocking", new BlockingOutboundHandler());
                    }
                });
    }
}

The Blocking Outbound Handler looks like this

public class BlockingOutboundHandler extends ChannelOutboundHandlerAdapter {

    private final Logger logger = LoggerFactory.getLogger(BlockingOutboundHandler.class);
    private final AtomicBoolean isIdle = new AtomicBoolean(true);

    public void setIdle() {
        synchronized (this.isIdle) {
            logger.debug("setIdle() called");
            this.isIdle.set(true);
            this.isIdle.notify();
        }
    }

    @Override
    public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise promise) throws Exception {
        synchronized (isIdle) {
            if (!isIdle.get()) {
                logger.debug("write(): I/O State not Idle, Waiting");
                isIdle.wait();
                logger.debug("write(): Finished waiting on I/O State");
            }
            isIdle.set(false);
        }
        logger.debug("write(): {}", msg.toString());
        ctx.write(msg, promise);
    }
}

Finally, when the StateMachine transitions to idle the the block is released

Optional.ofNullable((BlockingOutboundHandler) ctx.pipeline().get("blocking")).ifPresent(h -> h.setIdle());

All of this results in the Outbound messages being synchronized with the synchronous, statefull responses from the device.

Of course, I'd prefer not to have to deal with additional threads and the synchronization which come with them. I'm also not sure what kind of "yet to be discovered" issues I'm going to run into doing it this way. It does have the side effect of causing the main handler to be visited by multiple threads which just creates a new problem hich needs to be solved.

Also, next up, is implementation of a timeout and retry with back-off; this thread can't stay blocked indefinitely.

15:07:16.539 [DEBUG] [.netty.handler.logging.LoggingHandler] - [id: 0xb19e73e2] REGISTERED
15:07:16.540 [DEBUG] [.netty.handler.logging.LoggingHandler] - [id: 0xb19e73e2] CONNECT: portserver1.tedworld.net/192.168.2.173:2102
15:07:16.541 [DEBUG] [internal.connection.UDPConnectorNetty] - connect(): connect() complete
15:07:16.541 [DEBUG] [.netty.handler.logging.LoggingHandler] - [id: 0xb19e73e2, L:/192.168.2.186:47010 - R:portserver1.tedworld.net/192.168.2.173:2102] ACTIVE
15:07:16.542 [DEBUG] [c.projector.internal.ProjectorHandler] - scheduler.execute: creating test message
15:07:16.543 [DEBUG] [c.projector.internal.ProjectorHandler] - scheduler.execute: sending test message
15:07:16.544 [DEBUG] [internal.connection.UDPConnectorNetty] - sendRequest: Adding msg to queue { super={ messageType=21, channelId=test, data= } }
15:07:16.546 [DEBUG] [c.projector.internal.ProjectorHandler] - scheduler.execute: Finished
15:07:16.545 [DEBUG] [rnal.protocol.BlockingOutboundHandler] - write { super={ messageType=21, channelId=test, data= } }
15:07:16.547 [DEBUG] [internal.connection.UDPConnectorNetty] - sendRequest: Adding msg to queue { super={ messageType=3F, channelId=lamp, data= } }
15:07:16.548 [DEBUG] [rnal.protocol.BlockingOutboundHandler] - write(): I/O State not Idle, Waiting
15:07:16.548 [DEBUG] [.netty.handler.logging.LoggingHandler] - [id: 0xb19e73e2, L:/192.168.2.186:47010 - R:portserver1.tedworld.net/192.168.2.173:2102] WRITE: 6B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 21 89 01 00 00 0a                               |!.....          |
+--------+-------------------------------------------------+----------------+
15:07:16.550 [DEBUG] [.netty.handler.logging.LoggingHandler] - [id: 0xb19e73e2, L:/192.168.2.186:47010 - R:portserver1.tedworld.net/192.168.2.173:2102] FLUSH
15:07:16.567 [DEBUG] [.netty.handler.logging.LoggingHandler] - [id: 0xb19e73e2, L:/192.168.2.186:47010 - R:portserver1.tedworld.net/192.168.2.173:2102] READ: DatagramPacket(/192.168.2.173:2102 => /192.168.2.186:47010, PooledUnsafeDirectByteBuf(ridx: 0, widx: 6, cap: 2048)), 6B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 06 89 01 00 00 0a                               |......          |
+--------+-------------------------------------------------+----------------+
15:07:16.568 [DEBUG] [nternal.protocol.ResponseDecoderNetty] - decode DatagramPacket(/192.168.2.173:2102 => /192.168.2.186:47010, PooledUnsafeDirectByteBuf(ridx: 0, widx: 6, cap: 2048))
15:07:16.569 [DEBUG] [rojector.internal.protocol.AckHandler] - channelRead0 { super={ messageType=06, channelId=test, data= } }
15:07:16.570 [DEBUG] [rnal.protocol.BlockingOutboundHandler] - setIdle called
15:07:16.571 [DEBUG] [rnal.protocol.BlockingOutboundHandler] - write(): Finished waiting on I/O State
15:07:16.571 [DEBUG] [.netty.handler.logging.LoggingHandler] - [id: 0xb19e73e2, L:/192.168.2.186:47010 - R:portserver1.tedworld.net/192.168.2.173:2102] READ COMPLETE
15:07:16.571 [DEBUG] [rnal.protocol.BlockingOutboundHandler] - write { super={ messageType=3F, channelId=lamp, data= } }
15:07:16.573 [DEBUG] [.netty.handler.logging.LoggingHandler] - [id: 0xb19e73e2, L:/192.168.2.186:47010 - R:portserver1.tedworld.net/192.168.2.173:2102] WRITE: 6B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 3f 89 01 50 57 0a                               |?..PW.          |
+--------+-------------------------------------------------+----------------+
15:07:16.573 [DEBUG] [.netty.handler.logging.LoggingHandler] - [id: 0xb19e73e2, L:/192.168.2.186:47010 - R:portserver1.tedworld.net/192.168.2.173:2102] FLUSH
15:07:16.587 [DEBUG] [.netty.handler.logging.LoggingHandler] - [id: 0xb19e73e2, L:/192.168.2.186:47010 - R:portserver1.tedworld.net/192.168.2.173:2102] READ: DatagramPacket(/192.168.2.173:2102 => /192.168.2.186:47010, PooledUnsafeDirectByteBuf(ridx: 0, widx: 6, cap: 2048)), 6B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 06 89 01 50 57 0a                               |...PW.          |
+--------+-------------------------------------------------+----------------+
15:07:16.588 [DEBUG] [nternal.protocol.ResponseDecoderNetty] - decode DatagramPacket(/192.168.2.173:2102 => /192.168.2.186:47010, PooledUnsafeDirectByteBuf(ridx: 0, widx: 6, cap: 2048))
15:07:16.589 [DEBUG] [rojector.internal.protocol.AckHandler] - channelRead0 { super={ messageType=06, channelId=lamp, data= } }
15:07:16.590 [DEBUG] [rnal.protocol.BlockingOutboundHandler] - setIdle called
15:07:16.591 [DEBUG] [.netty.handler.logging.LoggingHandler] - [id: 0xb19e73e2, L:/192.168.2.186:47010 - R:portserver1.tedworld.net/192.168.2.173:2102] READ COMPLETE
15:07:16.592 [DEBUG] [.netty.handler.logging.LoggingHandler] - [id: 0xb19e73e2, L:/192.168.2.186:47010 - R:portserver1.tedworld.net/192.168.2.173:2102] READ: DatagramPacket(/192.168.2.173:2102 => /192.168.2.186:47010, PooledUnsafeDirectByteBuf(ridx: 0, widx: 7, cap: 2048)), 7B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 40 89 01 50 57 30 0a                            |@..PW0.         |
+--------+-------------------------------------------------+----------------+
15:07:16.593 [DEBUG] [nternal.protocol.ResponseDecoderNetty] - decode DatagramPacket(/192.168.2.173:2102 => /192.168.2.186:47010, PooledUnsafeDirectByteBuf(ridx: 0, widx: 7, cap: 2048))
15:07:16.594 [DEBUG] [.netty.channel.DefaultChannelPipeline] - Discarded inbound message { super={ messageType=40, channelId=lamp, data=30 } } that reached at the tail of the pipeline. Please check your pipeline configuration.
15:07:16.595 [DEBUG] [.netty.channel.DefaultChannelPipeline] - Discarded message pipeline : [logging, encode, decode, ack, blocking, DefaultChannelPipeline$TailContext#0]. Channel : [id: 0xb19e73e2, L:/192.168.2.186:47010 - R:portserver1.tedworld.net/192.168.2.173:2102].
15:07:16.596 [DEBUG] [.netty.handler.logging.LoggingHandler] - [id: 0xb19e73e2, L:/192.168.2.186:47010 - R:portserver1.tedworld.net/192.168.2.173:2102] READ COMPLETE
tlum
  • 913
  • 3
  • 13
  • 30