Is it possible to allocate host memory by overriding PFN_vkAllocationFunction and passing the callback to a Vulkan function

Question

I have implemented a custom allocator to track the memory that is allocated by vkAllocateMemory. I have also studied the official Khronos documentation regarding this topic but I could not found an answer to a following questions:

1. What is acutally the size parameter PFN_vkAllocationFunction? I have found out that apparently it is not the size of the actual buffer we want to allocate the memory for. I am suspecting that this is size of a Vulkan structures or any other Vulkan internal buffers. No matter how big memory chunk I want to allocate, it the size is always set to 200 (it is machine/GPU/driver dependent value but it is a constant value). For the testing purposes I used triangle from: https://github.com/SaschaWillems/Vulkan and an allocator from a similar question: Vulkan's VkAllocationCallbacks implemented with malloc/free()

triangle before vkAllocateMemory: memAlloc.allocationSize: 7864320 sizeof(memAlloc): 32
triangle after vkAllocateMemory: memAlloc.allocationSize: 7864320
pAllocator's allocationFunction: <Memory>, size: 200, alignment: 8, allocationScope: 1, return ptr* : 0x0x5564ac917b20

I have also found out that for other calls, this size is different for different vulkan calls, like vkCreateRenderPass or vkCreateImageView.

pAllocator's allocationFunction: <ImageView>, size: 160, alignment: 8, allocationScope: 1, return ptr* : 0x0x5564acdf8390 
pAllocator's allocationFunction: <ImageView>, size: 824, alignment: 8, allocationScope: 1, return ptr* : 0x0x5564acdf8440 
pAllocator's allocationFunction: <RenderPass>, size: 200, alignment: 8, allocationScope: 1, return ptr* : 0x0x5564ac950ee0 
pAllocator's allocationFunction: <RenderPass>, size: 88, alignment: 8, allocationScope: 1, return ptr* : 0x0x5564acdf8780 
pAllocator's allocationFunction: <RenderPass>, size: 56, alignment: 8, allocationScope: 1, return ptr* : 0x0x5564acdf7a00 
pAllocator's allocationFunction: <RenderPass>, size: 344, alignment: 8, allocationScope: 1, return ptr* : 0x0x5564ac6a07c0 
pAllocator's allocationFunction: <RenderPass>, size: 8, alignment: 8, allocationScope: 1, return ptr* : 0x0x5564acdf87e0 

2. Is it possible to allocate the host-visible memory using those callbacks? I would like to emulate the behaviour of VK_EXT_external_memory_host, VK_KHR_external_memory_fd or VK_EXT_external_memory_dma_buf but I don't know if this approach (that is implementing own allocator) can be useful for it.

Answer 1

To understand what's going, you need to understand what VkAllocationCallbacks are for. So let's dive in.

When you call vkCreateDevice for example, a Vulkan implementation needs to return a VkDevice handle. This is undoubtedly a pointer to an object. So... where did it's memory come from? There must be some internal, implementation-defined data structure in play. And that likely requires heap allocation.

However, for a low-level system, it is generally considered rude for it to arbitrarily allocate heap memory without the knowledge or consent of the application using that system. The larger application may want to pre-allocate a bunch of memory specifically for the Vulkan implementation's use, for any number of reasons.

Being able to do that requires being able to replace the Vulkan implementation's heap allocation functions with your own. That is what VkAllocationCallbacks are for: allowing your program to provide heap-allocation services to the Vulkan implementations, which will use them for its own internal data structures. And that last part is important: these allocations are for internal use.

vkAllocateMemory is a function for allocating device-accessible memory. Now yes, depending on the memory type used, it may allocate from the same pool of memory as CPU-accessible memory. But the memory vkAllocateMemory is allocating from is not for the Vulkan implementations usage; it's for your application's usage, mediated through the Vulkan API.

Now, above I said "in certain contexts". When you create a device using VkAllocationCallbacks, the callbacks will be used to allocate implementation memory for any Vulkan function using that device. That is the context of those callbacks. Other functions have their own context. For example, when you call vkCreateDescriptorPool, you also give it callbacks. These callbacks, if specified, will be used for any allocations used for descriptor pools.

Now, as stated above, vkAllocateMemory allocates device-accessible memory. However, it also creates a VkDeviceMemory object that represents this storage. This object lives in CPU space, and therefore it must have been allocated from CPU storage.

That's what the 200 bytes you saw represents: not the device memory allocation itself, but the allocation of CPU storage used to manage that device memory. You could say that the internal implementation object represented by VkDeviceMemory takes up 200 bytes.

Overall, you cannot track device-accessible allocations through a system intended to provide you with access to how much CPU-accessible memory is being allocated by Vulkan.