Asynchronous write (append) to a file by multiple parallel threads NIO

Question

Can I leverage NIO Async APIs to write (append) in a single file by multiple parallel threads? What I am trying to do is break the queries into different categories and assign a thread to each category to get the records from DB and write to a file (say user-records.json).

Thread A Users of some category A
Query by user ids - write the resultset to user-records.json

Thread B Users of some category B
Query by user ids - append the resultset to user-records.json

Thread C Users of some category C
Query by user ids - append the resultset to user-records.json

and so on ....

Here are the classes which are not the actual implementation but just a sample to enable Async file write/append -

public class AsyncAppender {

    private final AsynchronousFileChannel channel;
    /** Where new append operations are told to start writing. */
    private final AtomicLong              projectedSize;

    AsyncAppender( AsynchronousFileChannel channel) throws IOException {
        this.channel = channel;
        this.projectedSize = new AtomicLong(channel.size());
    }

    public void append( ByteBuffer buf) throws IOException {
        final int buflen = buf.remaining();
        long size;
        do {
            size = projectedSize.get();
        } while (!projectedSize.compareAndSet(size, size + buflen));

        channel.write(buf, channel.size(), channel, new WriteOp(buf, size));
    }
}

public class WriteOp implements CompletionHandler<Integer, AsynchronousFileChannel> {

    private final ByteBuffer buf;
    private long             position;

    WriteOp( ByteBuffer buf, long position) {
        this.buf = buf;
        this.position = position;
    }

    @Override
    public void completed( Integer result, AsynchronousFileChannel channel) {
        if (buf.hasRemaining()) { // incomplete write
            position += result;
            channel.write(buf, position, channel, this);
        }
    }

    @Override
    public void failed( Throwable ex, AsynchronousFileChannel channel) {
        // ?
    }
}

The main class -

public class AsyncWriteMain {

    public static void main( String[] args) {
        AsyncWriteMain m = new AsyncWriteMain();
        m.asyncWrite();
    }

    public void asyncWrite() {
        try {
            String filePath = "D:\\temp\\user-records.txt";
            Path file = Paths.get(filePath);
            AsynchronousFileChannel asyncFile = AsynchronousFileChannel.open(file, StandardOpenOption.WRITE, StandardOpenOption.CREATE);

            AsyncAppender aa = new AsyncAppender(asyncFile);

            for ( int i = 0; i < 10; i++) {
                aa.append(ByteBuffer.wrap((i + " Some text to be written").getBytes()));
            }
        } catch (IOException e) {
            System.out.println(e.getMessage());
        }
    }
}

Qingfei Yuan · Answer 1 · 2019-05-28T19:01:17.797

0

I think you can. Bascically, you need file lock

Future<FileLock> featureLock = asynchFileChannel.lock();
        log.info("Waiting for the file to be locked ...");
        FileLock lock = featureLock.get();
        if (lock.isValid()) {...}

coming from here

edited May 28 '19 at 19:01

answered May 28 '19 at 18:47

Qingfei Yuan

1,196
1
8
12

1

Thanks. But how can I enable parallel threads to call asyncWrite() method. seems using CompletableFuture.runAsync(() -> asyncWrite()); is not working. It exiting from asyncWrite() method without writing anything. Basically, I want to exit from main thread after invoking asyncWrite(). – Saurabh May 29 '19 at 21:50

Asynchronous write (append) to a file by multiple parallel threads NIO

1 Answers1