I ran Valgrind but the output points to generic malloc calls as the source of the leak. Why is it hiding the details with "???"?

Question

I'm abbreviating my Valgrind output below because it repeats numerous times with the same unhelpful output.

"Unhelpful" because what am I supposed to do with these obfuscated call stacks :

by 0x115EA8: ??? (in /usr/bin/dash)

In place of the ??? and the 0x115EA8 I was expecting some actionable info such as a function name or the name of a code file. I really don't know what to do with this and as, indicated below, I'm using the standard command-line options to start Valgrind.

==23991== 160 bytes in 1 blocks are still reachable in loss record 18 of 25
==23991==    at 0x48406C4: malloc (vg_replace_malloc.c:380)
==23991==    by 0x115EA8: ??? (in /usr/bin/dash)
==23991==    by 0x11546B: ??? (in /usr/bin/dash)
==23991==    by 0x10F78C: ??? (in /usr/bin/dash)
==23991==    by 0x1121AE: ??? (in /usr/bin/dash)
==23991==    by 0x112E76: ??? (in /usr/bin/dash)
==23991==    by 0x10F27F: ??? (in /usr/bin/dash)
==23991==    by 0x10E20E: ??? (in /usr/bin/dash)
==23991==    by 0x10E1E1: ??? (in /usr/bin/dash)
==23991==    by 0x115BC4: ??? (in /usr/bin/dash)
==23991==    by 0x10C70E: ??? (in /usr/bin/dash)
==23991==    by 0x4887189: (below main) (libc_start_call_main.h:58)

Is there anything to do here to get more info?

I'm using these standard options for running Valgrind:

valgrind --leak-check=full \
         --show-leak-kinds=all \
         --track-origins=yes \
         --verbose \
         --log-file=valgrind-out.txt \
         ./executable exampleParam1

Does anyone see anything useful in this output that I'm missing?

Or should I use different command-line options?

Adding some additional context regarding OS:

==23985== Memcheck, a memory error detector
==23985== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==23985== Using Valgrind-3.19.0-8d3c8034b8-20220411 and LibVEX; rerun with -h for copyright info
==23985== Command: /root/.sdkman/candidates/gradle/current/bin/gradle apprun --console=plain
==23985== Parent PID: 8
==23985== 
--23985-- 
--23985-- Valgrind options:
--23985--    --leak-check=full
--23985--    -v
--23985--    --show-leak-kinds=all
--23985--    --track-origins=yes
--23985--    --log-file=vg_logfile.out
--23985-- Contents of /proc/version:
--23985--   Linux version 4.4.0-19041-Microsoft (Microsoft@Microsoft.com) (gcc version 5.4.0 (GCC) ) #2311-Microsoft Tue Nov 08 17:09:00 PST 2022
--23985-- 
--23985-- Arch and hwcaps: AMD64, LittleEndian, amd64-cx16-lzcnt-rdtscp-sse3-ssse3-avx-avx2-bmi-f16c-rdrand-rdseed
--23985-- Page sizes: currently 4096, max supported 4096
--23985-- Valgrind library directory: /usr/libexec/valgrind
--23985-- Reading syms from /usr/bin/dash
--23985--    object doesn't have a symbol table
--23985-- Reading syms from /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
--23985--   Considering /usr/lib/debug/.build-id/4f/536ac1cd2e8806aed8556ea7795c47404de8a9.debug ..
--23985--   .. build-id is valid
--23985-- Reading syms from /usr/libexec/valgrind/memcheck-amd64-linux
--23985--   Considering /usr/lib/debug/.build-id/82/26c2aa6b808ebd5a6fafb694a7fb3287f33590.debug ..
--23985--   .. build-id is valid
--23985--    object doesn't have a dynamic symbol table
--23985-- Scheduler: using generic scheduler lock implementation.
--23985-- Reading suppressions file: /usr/libexec/valgrind/default.supp
==23985== embedded gdbserver: reading from /tmp/vgdb-pipe-from-vgdb-to-23985-by-root-on-???
==23985== embedded gdbserver: writing to   /tmp/vgdb-pipe-to-vgdb-from-23985-by-root-on-???
==23985== embedded gdbserver: shared mem   /tmp/vgdb-pipe-shared-mem-vgdb-23985-by-root-on-???
==23985== 
==23985== TO CONTROL THIS PROCESS USING vgdb (which you probably
==23985== don't want to do, unless you know exactly what you're doing,
==23985== or are doing some strange experiment):
==23985==   /usr/bin/vgdb --pid=23985 ...command...
==23985== 
==23985== TO DEBUG THIS PROCESS USING GDB: start GDB like this
==23985==   /path/to/gdb /root/.sdkman/candidates/gradle/current/bin/gradle
==23985== and then give GDB the following command
==23985==   target remote | /usr/bin/vgdb --pid=23985
==23985== --pid is optional if only one valgrind process is running
==23985== 
==23985== error calling PR_SET_PTRACER, vgdb might block
--23985-- REDIR: 0x4023860 (ld-linux-x86-64.so.2:strlen) redirected to 0x580bb0e2 (vgPlain_amd64_linux_REDIR_FOR_strlen)
--23985-- REDIR: 0x4022040 (ld-linux-x86-64.so.2:index) redirected to 0x580bb0fc (vgPlain_amd64_linux_REDIR_FOR_index)
--23985-- Reading syms from /usr/libexec/valgrind/vgpreload_core-amd64-linux.so
--23985--   Considering /usr/lib/debug/.build-id/ad/f1388be4d8781737b0c83fe111a5a9c6e930aa.debug ..
--23985--   .. build-id is valid
--23985-- Reading syms from /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so
--23985--   Considering /usr/lib/debug/.build-id/d8/ec66cffcb23a75c3f15940674d6028709121f8.debug ..
--23985--   .. build-id is valid
==23985== WARNING: new redirection conflicts with existing -- ignoring it
--23985--     old: 0x04023860 (strlen              ) R-> (0000.0) 0x580bb0e2 vgPlain_amd64_linux_REDIR_FOR_strlen
--23985--     new: 0x04023860 (strlen              ) R-> (2007.0) 0x048468a0 strlen
--23985-- REDIR: 0x4022260 (ld-linux-x86-64.so.2:strcmp) redirected to 0x4847780 (strcmp)
--23985-- REDIR: 0x40214d0 (ld-linux-x86-64.so.2:mempcpy) redirected to 0x484b1a0 (mempcpy)
--23985-- Reading syms from /usr/lib/x86_64-linux-gnu/libc.so.6
--23985--   Considering /usr/lib/debug/.build-id/4a/ff0f9d796e67d413e44f332edace9ac0ca2401.debug ..
--23985--   .. build-id is valid
==23985== WARNING: new redirection conflicts with existing -- ignoring it
--23985--     old: 0x048f93e0 (memalign            ) R-> (1011.0) 0x04845bc0 memalign
--23985--     new: 0x048f93e0 (memalign            ) R-> (1017.0) 0x04845b90 aligned_alloc
==23985== WARNING: new redirection conflicts with existing -- ignoring it
--23985--     old: 0x048f93e0 (memalign            ) R-> (1011.0) 0x04845bc0 memalign
--23985--     new: 0x048f93e0 (memalign            ) R-> (1017.0) 0x04845b60 aligned_alloc
==23985== WARNING: new redirection conflicts with existing -- ignoring it
--23985--     old: 0x048f93e0 (memalign            ) R-> (1011.0) 0x04845bc0 memalign
--23985--     new: 0x048f93e0 (memalign            ) R-> (1017.0) 0x04845b90 aligned_alloc
==23985== WARNING: new redirection conflicts with existing -- ignoring it
--23985--     old: 0x048f93e0 (memalign            ) R-> (1011.0) 0x04845bc0 memalign
--23985--     new: 0x048f93e0 (memalign            ) R-> (1017.0) 0x04845b60 aligned_alloc
--23985-- REDIR: 0x48ff330 (libc.so.6:strnlen) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x48ff3c0 (libc.so.6:strpbrk) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x48fd4e0 (libc.so.6:strcmp) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x49122e0 (libc.so.6:wcsnlen) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x48fc620 (libc.so.6:memset) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x4910c20 (libc.so.6:wcslen) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x48fbc50 (libc.so.6:memcpy@@GLIBC_2.14) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x4910a50 (libc.so.6:wcschr) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x48fd3d0 (libc.so.6:index) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x48ff3f0 (libc.so.6:rindex) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x4910ae0 (libc.so.6:wcscmp) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x48fc860 (libc.so.6:stpncpy) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x4911030 (libc.so.6:wmemchr) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x48ff1e0 (libc.so.6:strncmp) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x48fc8d0 (libc.so.6:strcasecmp) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x48fe740 (libc.so.6:strcspn) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x4910b70 (libc.so.6:wcscpy) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x48fd350 (libc.so.6:strcat) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x48ff0d0 (libc.so.6:strncasecmp_l) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x48fbb60 (libc.so.6:bcmp) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x48fc590 (libc.so.6:memrchr) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x48fd450 (libc.so.6:strchrnul) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x48fe6c0 (libc.so.6:strcpy) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x48fc970 (libc.so.6:strcasecmp_l) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x48fefa0 (libc.so.6:strlen) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x48ff280 (libc.so.6:strncpy) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x48fc7e0 (libc.so.6:stpcpy) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x48fc370 (libc.so.6:memmove) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
==23985== Preferring higher priority redirection:
--23985--     old: 0x049b2640 (__memcpy_avx_unalign) R-> (2018.0) 0x04848a60 __memcpy_avx_unaligned_erms
--23985--     new: 0x049b2640 (__memcpy_avx_unalign) R-> (2018.1) 0x0484a2b0 memmove
--23985-- REDIR: 0x48fbae0 (libc.so.6:memchr) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x48ff5b0 (libc.so.6:strspn) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x48fc480 (libc.so.6:mempcpy) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x48ff030 (libc.so.6:strncasecmp) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x48fc720 (libc.so.6:rawmemchr) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x48ffd50 (libc.so.6:strstr) redirected to 0x48371b0 (_vgnU_ifunc_wrapper)
--23985-- REDIR: 0x49b8710 (libc.so.6:__strrchr_avx2) redirected to 0x48462e0 (rindex)
--23985-- REDIR: 0x48f8700 (libc.so.6:malloc) redirected to 0x4840740 (malloc)
--23985-- REDIR: 0x49b52e0 (libc.so.6:__strchrnul_avx2) redirected to 0x484aca0 (strchrnul)
--23985-- REDIR: 0x49b50a0 (libc.so.6:__strchr_avx2) redirected to 0x4846460 (index)
--23985-- REDIR: 0x49b5dc0 (libc.so.6:__strlen_avx2) redirected to 0x4846780 (strlen)
--23985-- REDIR: 0x49b2640 (libc.so.6:__memcpy_avx_unaligned_erms) redirected to 0x484a2b0 (memmove)
--23985-- REDIR: 0x49b25f0 (libc.so.6:__mempcpy_avx_unaligned_erms) redirected to 0x484adb0 (mempcpy)
--23985-- REDIR: 0x49b54d0 (libc.so.6:__strcmp_avx2) redirected to 0x4847680 (strcmp)
--23985-- REDIR: 0x49d2030 (libc.so.6:__strcspn_sse42) redirected to 0x484b570 (strcspn)
--23985-- REDIR: 0x48f8cc0 (libc.so.6:free) redirected to 0x4843110 (free)
--23985-- REDIR: 0x49d4bd0 (libc.so.6:__strpbrk_sse42) redirected to 0x484b520 (strpbrk)
--23985-- REDIR: 0x49b5a30 (libc.so.6:__strcpy_avx2) redirected to 0x48468d0 (strcpy)
--23985-- REDIR: 0x48f8f00 (libc.so.6:realloc) redirected to 0x48457b0 (realloc)
--23986-- REDIR: 0x49d4cc0 (libc.so.6:__strspn_sse42) redirected to 0x484b670 (strspn)
==23986== 
==23986== HEAP SUMMARY:
==23986==     in use at exit: 10,426 bytes in 50 blocks
==23986==   total heap usage: 64 allocs, 14 frees, 21,089 bytes allocated
==23986== 
==23986== Searching for pointers to 50 not-freed blocks
==23986== Checked 117,136 bytes
==23986== 
==23986== 25 bytes in 1 blocks are still reachable in loss record 1 of 16
==23986==    at 0x48407B4: malloc (vg_replace_malloc.c:381)
==23986==    by 0x115E88: ??? (in /usr/bin/dash)
==23986==    by 0x10FC68: ??? (in /usr/bin/dash)
==23986==    by 0x110323: ??? (in /usr/bin/dash)
==23986==    by 0x10F002: ??? (in /usr/bin/dash)
==23986==    by 0x10E20E: ??? (in /usr/bin/dash)
==23986==    by 0x10E20E: ??? (in /usr/bin/dash)
==23986==    by 0x10E8AD: ??? (in /usr/bin/dash)
==23986==    by 0x10E20E: ??? (in /usr/bin/dash)
==23986==    by 0x115BC4: ??? (in /usr/bin/dash)
==23986==    by 0x10C70E: ??? (in /usr/bin/dash)
==23986==    by 0x4887189: (below main) (libc_start_call_main.h:58)

How did you compile the program you’re running valgrind on? Did you include debug symbols? — templatetypedef, Feb 25 '23 at 07:18
I just built it normally with `gradle`. I didn't think any changes were needed to the program for Valgrind to work. — pio1dcaqr, Feb 25 '23 at 13:13

Paul Floyd · Accepted Answer · 2023-02-25T14:56:12.947

You don't say anything about the OS or platform that you are using.

I'm going to guess that it is Arch Linux on amd64.

There is no problem with Valgrind. Valgrind is a binary dynamic analysis tool. When an error occurs in the guest executable it will try to use any information it can get to provide the user with details. But if the guest executable has no extra information then Valgrind can't magic it out of thin air.

So the problem is with your /usr/bin/dash. Most likely this has been stripped.

You have two choices

If your distro has 'debuginfo' packages, install the debuginfo for dash
build your own copy of dash that isn't stripped.

EDIT:

Roughly speaking, when you build an executable on an ELF unix-like system there are 3 possibilities

A plain "release" build. This will be optimized and contain information about symbols but no debug info. You'll get function names but not variables or line numbers.
A debug build which contains debuginfo. You'll get function names, and info on variables and line numbers.
A "stripped release" build. This uses stripto remove any symbol or debug information. You won't get any function names in this case, just ???.

One of the features of the strip command is to save the stripped information to a separate file.

I mainly use FreeBSD and Fedora. On FreeBSD at install time you can choose to install debug packages for the system (and the i386 components on amd64). There isn't a mechanism at present for third party components in the ports system to install debuginfo.

On Fedora you can install -debuginfo packages, more details here.

Lastly Valgrind can download debuginfo over the internet (this works with Fedora and RHEL at least).

I don't know if Kali supports any of the above.

I appreciate your help (and your work!). I have added some additional output from the start of the report. I'm running Valgrind on Kali Linux running under MS-WSL. So, when you say the problem is with `dash` being "stripped", I'm not sure what that means. Can you clarify? Thanks. I'm also not sure how to install debuginfo for dash. — pio1dcaqr, Feb 25 '23 at 13:20
Great explanation. I wouldn't have suspected that the distro might be playing a role. I'm going to try running it on a different box. Much appreciated, man. — pio1dcaqr, Feb 25 '23 at 17:19

I ran Valgrind but the output points to generic malloc calls as the source of the leak. Why is it hiding the details with "???"?

1 Answers1