Omitting core dumps via C a #define (or other in code/compile time solutions)?

Question

I understand that the operating system generates a core dump sometimes when a signal is sent (usually upon a crash). Is there a way to tell the operating system from inside C/C++ via a #define or compiler flag that no, or a somehow specifically limited core dump can be generated via that executable? The only way I know of to control it is via ulimit -c. My test system is Linux. The no core dump does not need to be implemented system wide, only for a specific program.

For the interested, this has to do with CVE-2019-15947 in Bitcoin Core's bitcoin-qt which still has no solution.

Some of this discussion is at the Bitcoin GitHub bug tracking page.

The other option would be to obfunscate and or encrypt the wallet.dat in memory so it's not easily retrievable via core dumps. Please note the second option can already be accomplished, though is not enabled by default.

Answer 1

Depending on your definition of "in code/compile-time", you can install a signal handler and wipe memory upon receiving that signal.

However, crashes are handled by the kernel, not the compiler or the executable. You cannot stop memory from being dumped by the kernel into a core from inside the executable, no matter what you do.

Therefore, the other option sounds best.

Answer 2

The key primitive you will want to use is madvise(..., MADV_DONTDUMP), which notifies Linux (since 3.4) that you do not wish a series of pages to be included in the dump. The flag is also called VM_DONTDUMP in kernel space. (Note that some versions of gdb do not respect this flag, which could be relevant for cores generated by gcore or other helpers rather than by the kernel.)

You will also need to ensure that when processing keys and other sensitive data stored in these pages information is not disclosed via registers or spilled to the stack sufficient for a compromise should a core dump after that time.

Answer 3

It's theoretically possible to set the limit from within a program with setrlimit():

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include <sys/resource.h>
#include <errno.h>


int setNoCores( void )
{
    int result;
    struct rlimit core_limiter;

    core_limiter.rlim_cur = 0;    // *no* core
    core_limiter.rlim_max = 0;

    // Tell the OS no core-files
    result = setrlimit( RLIMIT_CORE, &core_limiter );

    // Was it OK for you?
    if ( result != 0 )
    {
        switch( errno )
        {
            case EFAULT:   fprintf( stderr, "setNoCores() - EFAULT\n" ); break;
            case EINVAL:   fprintf( stderr, "setNoCores() - EINVAL\n" ); break;
            case EPERM:    fprintf( stderr, "setNoCores() - EPERM - No Permissions!\n" ); break;
            case ESRCH:    fprintf( stderr, "setNoCores() - ESRCH\n" ); break;
        }
    }

    return result;
}


int main( int argc, char **argv )
{
    if ( setNoCores() == 0 )
    {
        printf( "Core Files off\n") ;
    }
    else
    {
        printf( "Failed to change limits, core-file generation at default\n" );
    }

    return 0;
}

Of course, the process needs permission to adjust the limits. Depending on the user-id, the request may be denied, and EPERM error set.

Core files are also controllable in your OS via the ulimit command. With the -c parameter, this sets the maximum size of the core file. The size-number is the number of 1024 or 512-byte blocks [ref: Solaris man page (512), Linux '--help' (1024)]. Setting this in the shell the program executes from should limit the result for only that shells child-processes.

For no core-files:

    ulimit -c 0 

To turn it back on again:

ulimit -c <some large number>

For more details, try ulimit --help in your shell.