CUDA context destruction at host process termination

Question

If my host program [exit]/[segfault]/[is killed] what are the corresponding behaviors regarding the CUDA context destruction and corresponding allocated resources ?

By "behavior" I mean automatic GPU driver side mechanism if I never explicitly call cudaDeviceReset in my program.

I'm working under Linux and I've seen my GPU memory usage growing incrementally over several days (or even one single day) while I was launching CUDA code that was some times terminated through (CTRL-C) or simply segfaulting (on the host side) ...

score 3 · Answer 1 · answered Oct 04 '13 at 15:56

3

Perhaps, if you correctly manage exceptions, you can use atexit()? An exemple of using atexit() follows:

void ExitFunction() {
    cudaDeviceReset();
}

int main(...) {
    atexit(ExitFunction); 
    ...
    return 0;
}

answered Oct 04 '13 at 15:56

Vitality

20,705
4
108
146

score 3 · Answer 2 · answered Oct 04 '13 at 16:29

All allocations on the device are encapsulated in a CUDA context. When using the runtime API, CUDA contexts are created automatically "under the hood".

When an application exits, regardless of how, the context (and any associated allocations that are encapsulated within) should be destroyed.

If you are experiencing "small" changes in free memory after various events, this may be simply the fact that the driver is allocating additional space for it's own usage for internal housekeeping tasks. (The driver is akin to an operating system for the device.) If you are experiencing "large" changes in free memory after various application exit events, then I would say something is wrong, but since your question provides no details or indication of how to reproduce it, I'm not sure what else can be said.

If you want to force a driver "reset" externally, you could try unloading the driver via:

sudo rmmod nvidia
sudo nvidia-smi -a

And your free memory should be back to "normal".

CUDA context destruction at host process termination

2 Answers2