Symbolicate calls into macOS system libraries?

Question

I'm working on a macOS application that uses Google Crashpad to upload customer crash reports back to us. The crash dumps generated by that tool are fully symbolicated for our app, but NOT for the system libraries.

This is the opposite of the crash files located in /Library/Logs/DiagnosticReports which have symbols for system libraries but not typically for the application that crashed.

My question is, how do I symbolicate function calls within system libraries? I can already do this for an application I've built myself using atos and a dSYM bundle. However I'm having trouble finding documentation on how to do this for system libraries.

Crashpad's documentation is not very enlightening, as it is mainly focused on Linux and Windows: https://www.chromium.org/developers/decoding-crash-dumps

However, I should in theory be able to do this (after all, macOS's system crash service does do this when it generates its own reports - or, maybe I just need to pass some debug info from the Xcode SDK into atos?)

Here's an example dump, with references to my application removed.

Operating system: Mac OS X
                  10.9.5 13F1911
CPU: amd64
     family 6 model 42 stepping 7
     4 CPUs

GPU: UNKNOWN

Crash reason:  EXC_BREAKPOINT / EXC_I386_BPT
Crash address: 0x7fff669d609d
Process uptime: 3 seconds

Thread 8 (crashed)
 0  dyld + 0x109d
    rax = 0x00007fff66a0adb0   rdx = 0x0000000000000000
    rcx = 0x0000000000000000   rbx = 0x00007fd96a50de60
    rsi = 0x0000000000000000   rdi = 0x00007fff66a0af20
    rbp = 0x000000010c2ee700   rsp = 0x000000010c2ee6e8
     r8 = 0x00007fff669f5b8c    r9 = 0x0000000000000000
    r10 = 0x00007fff669f8d26   r11 = 0x00007fff66a0af20
    r12 = 0x005b327ab9000001   r13 = 0x0000000000000000
    r14 = 0x00007fff66a0af20   r15 = 0x00007fd96a508d10
    rip = 0x00007fff669d609d
    Found by: given as instruction pointer in context
 1  dyld + 0x3df6
    rbp = 0x000000010c2ee720   rsp = 0x000000010c2ee710
    rip = 0x00007fff669d8df6
    Found by: previous frame's frame pointer
 2  libdyld.dylib + 0x12dd
    rbp = 0x000000010c2ee870   rsp = 0x000000010c2ee730
    rip = 0x00007fff86d462dd
    Found by: previous frame's frame pointer
 3  0x109d90008
    rbp = 0x000000010c2ee8d0   rsp = 0x000000010c2ee880
    rip = 0x0000000109d90008
    Found by: previous frame's frame pointer
[...]
Loaded modules:
[...]
0x7fff669d5000 - 0x7fff66a08fff  dyld  0.0.0.0  (WARNING: No symbols, dyld, 1D3130FEFE7E3C4C8E74EB51895B6BA50)
0x7fff83520000 - 0x7fff83593fff  SecurityFoundation  55122.3.0.0
0x7fff8362d000 - 0x7fff837e5fff  libicucore.A.dylib  51.1.0.0
0x7fff837e6000 - 0x7fff8380dfff  libsystem_info.dylib  449.1.4.0
0x7fff83abf000 - 0x7fff83accfff  libxar.1.dylib  1.0.0.0
0x7fff83c04000 - 0x7fff83c0bfff  liblaunch.dylib  842.92.1.0
0x7fff83c0c000 - 0x7fff83cd6fff  LaunchServices  572.32.0.0
0x7fff83dd2000 - 0x7fff83dd6fff  libGIF.dylib  1.0.0.0
0x7fff83e12000 - 0x7fff83e42fff  libncurses.5.4.dylib  5.4.0.0
0x7fff841ae000 - 0x7fff8423efff  Metadata  800.30.0.0
0x7fff842d4000 - 0x7fff842d5fff  libremovefile.dylib  33.0.0.0
0x7fff84387000 - 0x7fff84475fff  libJP2.dylib  1.0.0.0
0x7fff84476000 - 0x7fff84481fff  libkxld.dylib  1.0.0.0
0x7fff844a0000 - 0x7fff844a8fff  libsystem_dnssd.dylib  522.92.3.0
0x7fff846b4000 - 0x7fff846cbfff  CFOpenDirectory  1.0.0.0
0x7fff84e1c000 - 0x7fff851fdfff  libLAPACK.dylib  1.0.0.0
0x7fff85bd0000 - 0x7fff85c92fff  CoreText  1.0.0.0
0x7fff85c93000 - 0x7fff85ce0fff  PrintCore  428.0.0.0
0x7fff85d6d000 - 0x7fff85de4fff  OSServices  600.4.0.0
0x7fff85de5000 - 0x7fff85deffff  ServiceManagement  1.0.0.0
0x7fff85e38000 - 0x7fff85e53fff  libCRFSuite.dylib  1.0.0.0
0x7fff85e84000 - 0x7fff85e84fff  libkeymgr.dylib  28.0.0.0
0x7fff85e85000 - 0x7fff85e89fff  libheimdal-asn1.dylib  1.0.0.0
0x7fff85ea1000 - 0x7fff85ea8fff  libcopyfile.dylib  103.92.1.0
0x7fff85ea9000 - 0x7fff85eb9fff  libbsm.0.dylib  1.0.0.0
0x7fff85eba000 - 0x7fff85ec1fff  libsystem_pthread.dylib  53.1.4.0  (WARNING: No symbols, libsystem_pthread.dylib, AB498556B555310E9041F67EC9E00E2C0)
0x7fff85ec2000 - 0x7fff85ec2fff  CoreServices  59.0.0.0
0x7fff85f20000 - 0x7fff85f27fff  NetFS  1.0.0.0
0x7fff864b2000 - 0x7fff864bdfff  NetAuth  1.0.0.0
0x7fff864e1000 - 0x7fff86751fff  Security  55471.14.40.0
0x7fff867a4000 - 0x7fff867adfff  libsystem_notify.dylib  121.20.1.0
0x7fff867d7000 - 0x7fff867dcfff  libunwind.dylib  35.3.0.0
0x7fff86813000 - 0x7fff86815fff  libquarantine.dylib  71.0.0.0
0x7fff86816000 - 0x7fff86827fff  libsystem_asl.dylib  217.1.4.0
0x7fff86887000 - 0x7fff868abfff  libxpc.dylib  300.90.2.0
0x7fff869ad000 - 0x7fff86aa7fff  libFontParser.dylib  1.0.0.0
0x7fff86ad3000 - 0x7fff86b2cfff  libTIFF.dylib  1.0.0.0
0x7fff86b69000 - 0x7fff86cbdfff  AudioToolbox  492.0.0.0
0x7fff86d45000 - 0x7fff86d48fff  libdyld.dylib  239.5.0.0  (WARNING: No symbols, libdyld.dylib, CAE7A50DF1563D4781C0DC41EB975D380)
0x7fff878c0000 - 0x7fff8792dfff  ATS  236.0.0.0
0x7fff8794d000 - 0x7fff8794dfff  vecLib  423.32.0.0
0x7fff87efb000 - 0x7fff87f40fff  HIServices  468.0.0.0
0x7fff87f4c000 - 0x7fff87f56fff  libcommonCrypto.dylib  60049.0.0.0
0x7fff88229000 - 0x7fff8823afff  libz.1.dylib  1.2.5.0
0x7fff8841f000 - 0x7fff88421fff  libsystem_configuration.dylib  596.15.0.0
0x7fff88422000 - 0x7fff88423fff  libsystem_sandbox.dylib  278.11.2.0
0x7fff8842b000 - 0x7fff8842bfff  ApplicationServices  48.0.0.0
0x7fff88441000 - 0x7fff88442fff  TrustEvaluationAgent  25.0.0.0
0x7fff8848a000 - 0x7fff884a5fff  libsystem_malloc.dylib  23.10.1.0  (WARNING: No symbols, libsystem_malloc.dylib, A695B4E438E9332EA77229D31E3F13850)
0x7fff884a9000 - 0x7fff8859afff  libiconv.2.dylib  7.0.0.0
0x7fff886d0000 - 0x7fff886dcfff  OpenDirectory  1.0.0.0
0x7fff88758000 - 0x7fff887a6fff  libcorecrypto.dylib  1.0.0.0
0x7fff887d2000 - 0x7fff8885bfff  ColorSync  4.7.0.0
0x7fff89c2a000 - 0x7fff89c43fff  Kerberos  6.0.0.0
0x7fff89c44000 - 0x7fff89c45fff  liblangid.dylib  1.0.0.0
0x7fff89ea8000 - 0x7fff89eadfff  libmacho.dylib  845.0.0.0
0x7fff8a24a000 - 0x7fff8a266fff  libsystem_kernel.dylib  2422.115.15.0  (WARNING: No symbols, libsystem_kernel.dylib, 34ABAF79E1473C34B05D46A566E689CC0)
0x7fff8a278000 - 0x7fff8a328fff  libvMisc.dylib  423.32.0.0
0x7fff8a342000 - 0x7fff8a3a5fff  SystemConfiguration  596.15.0.0
0x7fff8a425000 - 0x7fff8a42efff  CommonAuth  1.0.0.0
0x7fff8a5a7000 - 0x7fff8a5d0fff  libc++abi.dylib  49.1.0.0
0x7fff8a5d1000 - 0x7fff8a5edfff  libresolv.9.dylib  1.0.0.0
0x7fff8a6eb000 - 0x7fff8a71afff  libsystem_m.dylib  3047.16.0.0
0x7fff8a730000 - 0x7fff8a757fff  libsystem_network.dylib  241.4.0.0
0x7fff8abd1000 - 0x7fff8abf5fff  libJPEG.dylib  1.0.0.0
0x7fff8ad46000 - 0x7fff8b01afff  vImage  271.0.0.0
0x7fff8b20b000 - 0x7fff8b294fff  libsystem_c.dylib  997.90.5.0  (WARNING: No symbols, libsystem_c.dylib, 889AA3F9121C39858B1D5E908C5693BC0)
0x7fff8b4ac000 - 0x7fff8b4b0fff  libsystem_stats.dylib  93.90.3.0
0x7fff8b569000 - 0x7fff8b578fff  LangAnalysis  1.0.0.0
0x7fff8b579000 - 0x7fff8b579fff  Accelerate  4.0.0.0
0x7fff8bca4000 - 0x7fff8bcbffff  libPng.dylib  1.0.0.0
0x7fff8bccd000 - 0x7fff8bccdfff  libOpenScriptingUtil.dylib  1.0.0.0
0x7fff8be35000 - 0x7fff8c75efff  CoreGraphics  600.0.0.0
0x7fff8cac1000 - 0x7fff8cb08fff  libFontRegistry.dylib  1.0.0.0
0x7fff8cbaf000 - 0x7fff8cbb0fff  libsystem_blocks.dylib  63.0.0.0
0x7fff8d035000 - 0x7fff8d03bfff  libsystem_platform.dylib  24.90.1.0
0x7fff8d03c000 - 0x7fff8d03ffff  IOSurface  1.0.0.0
0x7fff8d071000 - 0x7fff8d371fff  Foundation  1056.17.0.0
0x7fff8d372000 - 0x7fff8d3d6fff  DataDetectorsCore  354.5.0.0
0x7fff8d4fd000 - 0x7fff8d4fffff  libRadiance.dylib  1.0.0.0
0x7fff8d500000 - 0x7fff8d501fff  libDiagnosticMessagesClient.dylib  1.0.0.0
0x7fff8df59000 - 0x7fff8df66fff  libbz2.1.0.dylib  1.0.5.0
0x7fff8e11e000 - 0x7fff8e127fff  SpeechSynthesis  1.0.0.0
0x7fff8e1df000 - 0x7fff8e2c9fff  libsqlite3.dylib  158.0.0.0
0x7fff8e2ca000 - 0x7fff8e2cdfff  TCC  1.0.0.0
0x7fff8e2ce000 - 0x7fff8e2cffff  libSystem.B.dylib  1197.1.1.0
0x7fff8e32c000 - 0x7fff8e387fff  AE  665.6.0.0
0x7fff8ea6a000 - 0x7fff8ea92fff  libxslt.1.dylib  3.26.0.0
0x7fff8ea93000 - 0x7fff8eb7bfff  libxml2.2.dylib  10.9.0.0
0x7fff8eb86000 - 0x7fff8ebebfff  Heimdal  1.0.0.0
0x7fff8ec48000 - 0x7fff8ef32fff  CarbonCore  1077.17.0.0
0x7fff8ef33000 - 0x7fff8f03afff  ImageIO  1.0.0.0
0x7fff8f0c1000 - 0x7fff8f12dfff  IOKit  275.0.0.0
0x7fff8f21e000 - 0x7fff8f3cbfff  libobjc.A.dylib  228.0.0.0
0x7fff8f58c000 - 0x7fff8f58dfff  libunc.dylib  28.0.0.0
0x7fff8f58e000 - 0x7fff8f659fff  libvDSP.dylib  423.32.0.0
0x7fff8f65a000 - 0x7fff8f674fff  libdispatch.dylib  339.92.1.0  (WARNING: No symbols, libdispatch.dylib, C4E4A18D3C3B3C9C8709A4270D998DE70)
0x7fff8f71f000 - 0x7fff8f771fff  libc++.1.dylib  120.0.0.0
0x7fff8fca0000 - 0x7fff8fcd9fff  QD  298.0.0.0
0x7fff8ff1e000 - 0x7fff8ff22fff  libcache.dylib  62.0.0.0
0x7fff8ff23000 - 0x7fff8ff65fff  libauto.dylib  1.0.0.0
0x7fff8ff66000 - 0x7fff8ff6dfff  libcompiler_rt.dylib  35.0.0.0
0x7fff8ff71000 - 0x7fff8ff75fff  libpam.2.dylib  3.0.0.0
0x7fff902c7000 - 0x7fff90318fff  CoreAudio  1.0.0.0
0x7fff90319000 - 0x7fff9031efff  DiskArbitration  1.0.0.0
0x7fff903aa000 - 0x7fff90545fff  CFNetwork  673.6.0.0
0x7fff90546000 - 0x7fff9056ffff  DictionaryServices  1.0.0.0
0x7fff905ee000 - 0x7fff907d3fff  CoreFoundation  855.17.0.0
0x7fff90833000 - 0x7fff909a1fff  libBLAS.dylib  1.0.0.0
0x7fff90c01000 - 0x7fff90c48fff  libcups.2.dylib  2.10.0.0
0x7fff90c49000 - 0x7fff90c78fff  GSS  1.0.0.0
0x7fff90c79000 - 0x7fff90ce6fff  SearchKit  200.1.0.0

Mattie · Answer 1 · 2018-07-11T18:35:05.617

I'm not terribly familiar with Crashpad, but from looking at the report, I think I have an idea of what's going on.

Generally speaking, your process for symbolication will work for any library - yours or Apple's. You just have to point atos at a symbol source, give it the load address + offset and you are good. dSYMs are great sources, but executables can be sources too.

However, the critical part is having access to a symbol source that matches the version of the library loaded into your process when the crash occurred. To help with this process, binaries on Apple's platforms are identified by a UUID. You can read out this uuid with dwarfdump -u /path/to/binary.

One way to do this is just to attempt the symbolication process on the device where the crash occurred. This is how ReportCrash works. And, why it can symbolicate Apple's binaries. The reason (I assume) it fails to work for your app is that you stripe symbols from your artifact(s).

There are some downsides to doing symbolication this way, however. One big one is you generally don't have access to dSYMs. dSYMs contain way more information that just the address->symbol info. So, you definitely want use them, if you can.

The alternative is to do symbolication after the fact, with only the data from the report. The huge problem with doing this is you still need access to symbol sources. Going by the report you posted again, it appears that Crashpad's symbolication system could find/load symbols for dyld. Helpfully, it includes the uuid for that particular binary. But, if you don't happen to have a copy of the dyld that shipped with 10.9.5 (13F1911), you're out of luck. Even access to the macOS SDK for 10.9 is insufficient, because you need that particular build.

Given that CrashPad has produced warnings for so few binaries, it seems like it should have at least tried to do some symbolication. Perhaps there's some extra configuration required? I know that Apple made some modifications to the symbol lookup APIs years ago (iOS 7 timeframe I believe), which made the on-device symbolication process way more complex. Could it be that CrashPad has not been updated since then?

This is one of the reasons why hosted crash reporting systems exist. Crashlytics (a service I used to work on) indexes every Apple OS release for all platforms, so that it can do high quality server-side symbolication to solve exactly this problem. There are a bunch of other services too. Have you considered a hosted solution?

Some good ideas here. We are considering changing things so we do symbolication in two passes - first on the device (to take advantage of the fact that the system binaries are there), and then finish in our environment where we have our own dSYMs. A hosted crash reporting service with a full collection of binaries is something I hadn't considered. — craig65535, Jul 11 '18 at 17:35
Two passes are pretty much required. Doing things on-device is convenient, but tricky. For example, there's no guarantee that the re-launch of your app, where you'll do your symbolication, is running on the same OS version that the crash occurred on. Stupid edge case, but definitely comes up. Good luck with your work, and if you have any other questions about this kind of stuff, let me know I'd be happy to try to help. — Mattie, Jul 11 '18 at 18:37

Symbolicate calls into macOS system libraries?

1 Answers1