The question about integrating Netty 3 with JCA Resource adapter was already asked. The solution was quite straightforward: write a custom Executor that wraps JCA WorkManager and pass it to NioServerSocketChannelFactory constuctor.
However, threading seems to be heavily refactored in Netty 4 and this approach doesn't work (there's no NioServerSocketChannelFactory class, to start with). There is an option to supply your own ThreadFactory, but obviously, this is not good enough for JCA, since only WorkManager is exposed, not threads, so a simple facade is not possible anymore.
So I think I'm stuck. Is what I am trying to do even possible without lots of code being written?
EDIT: In the end I asked myself why making Resource adapter at all. Instead I just use JMS queues (inbound and outbound) as my integration points between our EE application and standalone server that uses netty and it works fine.