4

I tought that it might work using the Vulkan's external memory extension by allocating the memory in process A, create a HANDLE to that memory and pass that HANDLE via shared memory to process B.

But this seems is not possible as HANDLE is actually just a void* and for different processes exist different memory mapping, so the pointer/HANDLE is actually not transferable between processes.

Is there some way how to get the handle somehow and recreate the GL texture/ VK image in different process without copying all images's pixel data (like when PBO is used)?

CJ_Notned
  • 248
  • 2
  • 11
  • 1
    I'm not sure if I can follow: I guess you reimport the HANDLE in the second process with `ImportMemoryWin32HandleEXT` ? Handle is a technically a void*, but why would that be a problem and what does it have to do with memory mapping? As far as I understand the external memory handling, HANDLE is a pointer to external memory, not to memory in your programs address space. And you don't access that pointer with anything else than `ImportMemoryWin32HandleEXT` anyway, so as long as that method handles it correctly, it should be fine. – BDL Jul 24 '23 at 12:38
  • The [extension spec](https://registry.khronos.org/OpenGL/extensions/EXT/EXT_external_objects_win32.txt) also states this: " is a windows HANDLE cast to the type "void*"". It's not a pointer, it's a handle cast into a pointer. – BDL Jul 24 '23 at 12:42
  • As `winnt.h` states `typedef void *HANDLE;`, so yes HANDLE is just typedef for void* I should tag the questuion C++/C. Although I obtain HANDLE in process A, via `vkGetMemoryWin32HandleKHR` and then I store that HANDLE in a shared memory and try to reuse it in process B, but it is not working. In process B it is invalid, as the pointer value (imported HANDLE) there points to different memory space, or at last it seems so. If you know about any example using Vulkan + memory HANDLES in IPC way, pls let me know, as I did not found anything. – CJ_Notned Jul 24 '23 at 12:54
  • To be clear I am inporting the HANDLE in process B using `glImportMemoryWin32HandleEXT`. Maybe I should avoid that GL call and use something for Vulkan instead (but for interop within one process export in VK and import in GL works OK)? – CJ_Notned Jul 24 '23 at 13:01
  • 2
    Using `glImportMemoryWin32HandleEXT` is exactly what you are supposed to call. I think there is still a conceptual misunderstanding: Even though HANDLE is a void* it is not a "normal" memory pointer. It's a pointer to external memory, not to your applications memory. You can't access it with a debugger; how do you currently identify that the pointer is invalid? Any exception? Error message? I used this extension ages ago and couldn't find the blog anymore, but this one (though not in C++) uses basically the same steps: https://bxt.rs/blog/fast-half-life-video-recording-with-vulkan/ – BDL Jul 24 '23 at 13:04
  • @BDL Then it would make sense (that the handle is actaully different memory not app's memory map based one), but that mean that I should be completely OK to obtain a handle via `vkGetMemoryWin32HandleKHR` and put the value to the shared memory, open that shared mem. in process B read the value and use it with `glImportMemoryWin32HandleEXT`, or not? – CJ_Notned Jul 24 '23 at 13:28
  • The way you describe it is exactly how it should work. If it doesn't work, then you should probably ask a new question and show your code – BDL Jul 24 '23 at 13:56
  • The hlaf-life link is just showing the multithread approach, not multiprocess, as what I am able to understand with mine limited Rust knowledge. This is not the case, as such approach I already have working. `DuplicateHandle` seems the way to go for me, though I am still getting some errors, but at last I got some progress. – CJ_Notned Jul 26 '23 at 12:26

1 Answers1

6

To transfer a HANDLE to a different process use the function DuplicateHandle, which will create a duplicate handle referencing the same object, but in the other process. Then you'll have to pass the value of that new HANDLE (which isn't usable in the original process) to the other process by some means of IPC. You could use shared memory, or send a message, or… well whatever that allows you to send the other process an integer of sizeof(void*), i.e. in terms of C a uintptr_t. You then just cast the HANDLE to a uintptr_t send that numeric value to the other process, where you cast it back to a HANDLE.

You have to do this manually, since there's no mechanism in place, that tells a process that suddenly and out of the blue an entirely new HANDLE was magicked into their task context.

datenwolf
  • 159,371
  • 13
  • 185
  • 298
  • That is exactly what I was doing, except calling `DuplicateHandle`, I hope this was the problem, thank you very much. I will try it out and accept the answer, if it solves the issue. – CJ_Notned Jul 25 '23 at 07:31
  • Yes, this worked for me in the end. Just note for anybody reading this, you can `DuplicateHandle` just once in the client (which is what you need to do, if you do not know the PID (or names) of potential clients within server process). – CJ_Notned Jul 27 '23 at 11:50