You could I suppose, in a specific thread, regularly poll currently running processes to see if the file name is contained within a process title.
As for .docx files for example, WORD would have this as its process title:
Sample.docx - Microsoft Word
You would need to utilize a JNA method named getAllWindowNames() to acquire a list of Window Names. This method works quite well. When Sample.docx is detected within the acquired list then start whatever file or files you like.
Keep in mind however, your Java application would need to always be running in the background and because of file association (as mentioned to you in a previous post) this technique would run the files you have associated with Sample.docx regardless of how the file was run (from a double-click in Windows Explorer, a shortcut on Desktop, opened from MS WORD itself, etc).
I have actually created a small demo application that does exactly what you are trying to accomplish however it is too large to post here. There is no tutorial that I know of for this sort of thing, it's just a matter of doing it....that is if the concept appeals to you.
Yes!...most people would not want this sort of thing dancing around on their System(s).