You can lock/unlock files and folders in Java but only by application that locked them (you programmed). However there is no Java method/class which can unlock file used by other process.
You should bundle your application with another (native) software. For example you could create shell script for Linux systems and execute it. In Java application detect in which OS it is running so you can execute proper script/software.
When application requires RW lock, system must ensure that no one else have rights to modify it, thats why you need to kill process that is using it.
If you have access to source code of that 3rd party Java application (that is actually locking file you need), then you could implement server side which will listen requests for unlocking file and approval for locking it back again.
By my opinion better approach would be to transfer file by that application to yours, then do what you want and 3rd party app can run without interruption (shouldn't be noticeable). If you need to modify it, then 3rd should wait, your modifies and send back an updated version, 3rd continue to work.