1

I've got a problem with MappedByteBuffer specially how it works internally. The way I understand it the caching is done completely by the Operating System. So if I read from the file (using MappedByteBuffer) the OS will read whole pages from the hard drive and saves the page in RAM for faster access when needed again. This also allows to provide a shared cache for multiple applications/processes which access the same file. Is this correct?

If so, how is it possible to invalidate this cache? Just reinitializing the Mapped-Object shouldn't work. I have written an application which reads a lot from the hard drive. I need to do a few benchmarks, so I need to clear this cache when needed. I've tried to use "echo 3 > /proc/sys/vm/drop_caches" but this doesn't make a difference so I think it is not working.

Karamba
  • 138
  • 2
  • 15
  • This is OS dependent; it seems you are using Linux here, but what you should really do is `.force()` and unmap, before mapping again. – fge Apr 19 '14 at 11:09
  • http://docs.oracle.com/javase/6/docs/api/java/nio/channels/FileChannel.html says that "A mapping, once established, is not dependent upon the file channel that was used to create it. Closing the channel, in particular, has no effect upon the validity of the mapping." – Karamba Apr 19 '14 at 11:17
  • @Karamba Once you hold a mapping to a file, it remains open until you call close() and all the mappings have been cleaned up. – Peter Lawrey Apr 19 '14 at 11:30
  • Could you please tell where can I found this in the specification? I only see the following: http://docs.oracle.com/javase/7/docs/api/java/nio/channels/FileChannel.html#map%28java.nio.channels.FileChannel.MapMode,%20long,%20long%29 - "A mapping, once established, is not dependent upon the file channel that was used to create it. Closing the channel, in particular, has no effect upon the validity of the mapping." Thus I cannot see how closing File/FileChannel could help reclaiming RAM from MappedByteBuffer. – aljipa Apr 19 '14 at 11:34
  • As written, this appears to be an "X-Y" problem: you want to do X, and you think that Y is the best way to approach it, so you ask how to do Y. What is the real "X" that you're trying to accomplish? In other words, what do you plan to demonstrate/validate/test with your benchmarks? – kdgregory Apr 19 '14 at 12:47

2 Answers2

1

This also allows to provide a shared cache for multiple applications/processes which access the same file. Is this correct?

This is how it works on Linux, Windows and MacOS. On other OSes, it probably is the same.

If so, how is it possible to invalidate this cache?

delete the file and it will not longer be valid.

I need to do a few benchmarks, so I need to clear this cache when needed.

That is what the OS is for. If you need to force the cache to be invalid, this is tricky and entirely OS dependant.

I've tried to use "echo 3 > /proc/sys/vm/drop_caches" but this doesn't make a difference so I think it is not working.

It may have no impact on your benchmark. I suggest you look at /proc/meminfo for 

Cached:           588104 kB
SwapCached:          264 kB

BTW if you want to unmap a MappedByteBuffer I do the following

public static void clean(ByteBuffer bb) {
    if (bb instanceof DirectBuffer) {
        Cleaner cl = ((DirectBuffer) bb).cleaner();
        if (cl != null)
            cl.clean();
    }
}

This works for direct ByteBuffers as well, but probably won't work in Java 9 as this interface will be removed.

Peter Lawrey
  • 525,659
  • 79
  • 751
  • 1,130
  • deleting the file is no option. I will try looking at /proc/meminfo – Karamba Apr 19 '14 at 11:21
  • Let's assume I do one run in my application. After calling clean(), no cached page from the first run will be used in the next run? – Karamba Apr 19 '14 at 11:30
  • 1
    @Karamba If you use `drop_caches` nothing is kept at that point. In `top` you can see there is little cached data. It appears the kernel keeps some pages for it's own purposes. A good test is to perform an `vmstat -s` before and after the test and take the difference. This will tell you the total IOs performed since boot. If you are getting the same number of IOs you know the number of reads/writes is the same. – Peter Lawrey Apr 19 '14 at 11:37
  • Sorry, but I still don't get it. So drop_caches should work for just not using cached pages of a previous run? – Karamba Apr 19 '14 at 11:51
  • @Karamba It should clear all cached data. It is possible it doesn't clear mapped data, but I don't see any which says this is the case. – Peter Lawrey Apr 19 '14 at 12:34
  • @Karamba The most important tests is to see if you read/write the same amount in the test at the OS level. If this is the same, you have a reproducable test. – Peter Lawrey Apr 19 '14 at 12:35
0

It's known sad issue (which is still unresolved in JDK): http://bugs.java.com/view_bug.do?bug_id=4724038

But even though there's no public API for that there's a dangerous workaround (use at your own risk): https://community.oracle.com/message/9387222

An alternative is not to use huge MappedByteBuffers and hope that they eventually will be garbage-collected.

In case what is needed is for different programs that want to map this file use their very own MappedByteBuffer copy out of this File, MapMode.PRIVATE could help.

Hope that helps.

aljipa
  • 716
  • 4
  • 6
  • I don't think he is trying to unmap the mapping but clear the disk cache for reproducable benchmark results. – Peter Lawrey Apr 19 '14 at 11:20
  • Could you please explain, what do you mean by 'clear the disk cache', MappedByteBuffer maps directly into RAM as far as I remember, – aljipa Apr 19 '14 at 11:31
  • Yes, I just want to have reproduceable benchmarks. I don't really care about getting back used RAM instantly. I want that cached pages from a previous run are not used again. – Karamba Apr 19 '14 at 11:49
  • Is MapMode.PRIVATE what you need? Updated the answer. – aljipa Apr 19 '14 at 11:57