3

enter image description here I have a memory leak problem on a AWS dockerized microservice deployed with ECS. I wanted to analyze the dump with DotMemory so I used exec to connect to the container, gcore to save the dump and then transfer that dump to an S3 bucket so I could download it. Problem is, when I open the dump everything is "Unresolved" and I can't understand what's going on.

I've tried to run the same microservice locally and take a memory dump with Windows Task Manager and everything worked fine. Sadly, since is a complex system I can't replicate locally exactly what's happening when it's deployed so I need to create a readable dump from my deployed microservice. How do I fix the issue?

MrDemien
  • 74
  • 8
  • Hello @MrDemien ! I have a similar issue (except the dump is made with dotnet-monitor as a sidecar). I updated the coredump_filter flag with 0x3f and did a new dump but no luck, everything is still "Unresolved" in DotMemory. Did you find a solution? – Bluezen Jan 09 '23 at 21:00
  • Not yet unfortunately. Have you tried with "0x8f"? – MrDemien Jan 19 '23 at 12:00
  • 1
    "0x3f" already set the first 6 bit to 1. The coredump_filter takes into account 8 bits max so we could try "0xff" to set every bit to 1 but in my case it shouldn't change anything. You can see in the following doc what each bit means: https://man7.org/linux/man-pages/man5/core.5.html – Bluezen Jan 20 '23 at 13:23
  • 1
    I have create a way to [reproduce the issue](https://youtrack.jetbrains.com/issue/DMRY-9646/NullReferenceException-when-reading-a-linux-dump-taken-with-dotnet-monitor-dump#focus=Comments-27-6737589.0-0) and JetBrains has created a [ticket](https://youtrack.jetbrains.com/issue/DMRY-9990/All-types-are-unresolved-in-the-dump-file-taken-on-Linux-machine-even-if-coredumpfilter-is-set-to-0x3f) – Bluezen Jan 20 '23 at 13:27

1 Answers1

0

This issue might appear if the dump is missing the segments with metadata.

Since kernel 2.6.23, the Linux-specific /proc/[pid]/coredump_filter file can be used to control which memory segments are written to the core dump file in the event that a core dump is performed for the process with the corresponding process ID.

See core man section "Controlling which mappings are written to the core dump" for more information.

In order to get a proper dump of the dotNet process coredump_filter should be set to at least 0x3f.

You can check what the current filter set for your process by executing:

cat /proc/<pid>/coredump_filter

To set the proper coredump_filter type:

echo "0x3f" > /proc/<pid>/coredump_filter

<pid> should be replaced with your process ID, for example:

echo "0x3f" > /proc/144/coredump_filter

UPDATE: May 10, 2023

I've managed to collect a full dump with all the necessary information by using createdump utility:

createdump -u -f /tmp/coredump <pid>

The -u option tells createdump to generate a full memory dump, including all memory-mapped files. Note that this will result in a very large dump file for applications that use a lot of memory or have many memory-mapped files.

Example:

docker container exec -it \
  -u root --privileged \
  <container-id> \
  /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.16/createdump -u -f /tmp/coredump <pid>

docker cp <container-id>:/tmp/coredump .
Lenar
  • 298
  • 3
  • 8
  • I tried to set "0x3f" but I got the same result. Should I try with "0x8f"? It takes me quite sometime to try it because it's a 12Gb dump file. – MrDemien Dec 12 '22 at 19:16
  • There is a known issue with importing memory dumps taken in containers. Please, see the issue in JetBrains issue tracker: https://youtrack.jetbrains.com/issue/DMRY-9990 – Lenar Mar 10 '23 at 09:54