This isn't directly built into java. Your best bet is to jump into a native OS solution. This can be tedious if you want to support multiple systems though.
If you can get away with supporting windows take a look at THIS LINK . Scroll down to the bottom and look at similar apps. You would be interested in any app that contains a command line interface. What you will need to do is install one of the software and then kick off a process using Runtime.exec. You could potentially just use a direct dll, but I'm not qualified to tell you which dll will give you that information or if it even exists. It might be something you want to look into though if you do not want a 3rd party dependency.
You will read the results of the process that hooks into the windows dll's and will tell you if the file is currently open (See this link for more details). Your application will have to pull data (consistently asking the Application if the file is open). It is not ideal, but probably it is a potential solution.