File handing in WebFlux (Reactor)

Question

I'm working on a greenfield reactive project where a lot of file handling IO is going on. Is it sufficient if I write the IO code in an imperative blocking manner and then wrap them in a Mono, publish them on boundedElastic scheduler? Will the boundedElastic pool size limit the number of concurrent operations?

If this is not the correct method, can you show an example how to write bytes to a file using Reactor?

Its hard to answer the questions without the context, if you could give an example of what does one of these transactions look like, that might be helpful. Saying, what do you do before the IO handling, whats after, who triggers this transaction, etc. — kevinjom, Jun 04 '20 at 09:19
Does this answer your question? [Why FileChannel in Java is not non-blocking?](https://stackoverflow.com/questions/3955250/why-filechannel-in-java-is-not-non-blocking) — pixel, Mar 30 '22 at 19:51

Michael Berry · Accepted Answer · 2020-06-04T12:26:48.310

Is it sufficient if I write the IO code in an imperative blocking manner and then wrap them in a Mono, publish them on boundedElastic scheduler?

This comes down to opinion on some level - but no, certainly not ideal not for a reactive greenfield project IMHO. boundedElastic() schedulers are great for interfacing with blocking IO when you must, but they're not a good replacement when a true non-blocking solution exists. (Sometimes this is a bit of a moot point with file handling, since it depends if it's possible for the underlying system to do it asynchronously - but usually that's possible these days.)

In your case, I'd look at wrapping AsynchronousFileChannel in a reactive publisher. You'll need to use create() or push() for this and then make explicit calls to the sink, but exactly how you do this depends on your use case. As a "simplest case" for file writing, you could feasibly do something like:

static Mono<Void> writeToFile(AsynchronousFileChannel channel, String content) {
    return Mono.create(sink -> {
        byte[] bytes = content.getBytes();
        ByteBuffer buffer = ByteBuffer.allocate(bytes.length);
        buffer.put(bytes);
        buffer.flip();

        channel.write(buffer, 0, null, new CompletionHandler<>() {
            @Override
            public void completed(Integer result, Object attachment) {
                sink.success();
            }

            @Override
            public void failed(Throwable exc, Object attachment) {
                sink.error(exc);
            }
        });
    });
}

A more thorough / comprehensive example of bridging the two APIs can be found here - there's almost certainly others around also.

Thank you, I'll dig into this! I still should publish these on boundedElastic right? — Peter, Jun 05 '20 at 10:14
@Peter No, you shouldn't need to publish these on any specific scheduler - reading and writing in this way is completely asynchronous, so you can use your event loop as normal. Note that this (unfortunately) isn't guaranteed for actually *opening* the `AsynchronousFileChannel` in the first place, so you (may, depending on context) wish to wrap the open call in a boundedElastic scheduler or similar. — Michael Berry, Jun 05 '20 at 10:17
After reading every documentation and resource I found about AsynchronousFileChannel, just some added thoughts: the asynchronous part comes from the fact that **your** thread is not blocked, instead the JVM provides a dedicated pool where the blocking occurs. https://docs.oracle.com/javase/7/docs/api/java/nio/channels/AsynchronousChannelGroup.html Also, Linux would not support truly non-blocking IO, only newer Windows systems do. — Peter, Jun 28 '20 at 05:11
@Peter It looks like Linux has io_uring which is asynchronous and it's going to be supported in Loom: https://twitter.com/ijuma/status/1261486739769573376 — pixel, Mar 30 '22 at 19:52

kerbermeister · Answer 2 · 2023-02-13T07:42:12.897

After some researching the java.nio and Spring library I have found the convenient approach to write strings to file as DataBuffers (which perfectly connect with WebFlux) into AsynchronousFileChannel using Spring classes.

It's not "truly" reactive way to write lines in file, but asyncronous and it is still better than using some standard blocking API.

public Mono<Void> writeRows(Flux<String> rowsFlux) {
    DefaultDataBufferFactory bufferFactory = new DefaultDataBufferFactory();
    CharSequenceEncoder encoder = CharSequenceEncoder.textPlainOnly();

    Flux<DataBuffer> dataBufferFlux = rowsFlux.map(line ->
            encoder.encodeValue(line, bufferFactory, ResolvableType.NONE, null, null)
    );
    return DataBufferUtils.write(
            dataBufferFlux,
            Path.of("/your_path_to_save_file/file.txt"),
            StandardOpenOption.CREATE_NEW
    );
}

Of course, for better performance in this case you can buffer your strings in flux and then append those strings to one string and create a data buffer from it.

Or if you already have Flux of data buffers you can write them to file using DataBufferUtils directly.

File handing in WebFlux (Reactor)

2 Answers2