Using otool (recursively) to find shared libraries needed by an app

Question

I have a Cocoa app that uses otool to find required shared libraries that an app needs to function properly. For example, say I run otool -L on an app that uses QTKit.framework. I get a list of the shared libraries used by the program (including the basic frameworks like Cocoa.framework and AppKit.framework):

/System/Library/Frameworks/QTKit.framework/Versions/A/QTKit (compatibility version 1.0.0, current version 1.0.0)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 476.0.0)
    /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 949.0.0)

..... and so on for a bunch of other frameworks

Which shows that the app uses QTKit.framework. However if I use "otool -L" again on the binary for QTKit.framework (/System/Library/Frameworks/QTKit.framework/Versions/A/QTKit) I get this:

/System/Library/Frameworks/QTKit.framework/Versions/A/QTKit (compatibility version 1.0.0, current version 1.0.0)
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox (compatibility version 1.0.0, current version 1.0.0)
/System/Library/PrivateFrameworks/CoreMedia.framework/Versions/A/CoreMedia (compatibility version 1.0.0, current version 1.0.0)
/System/Library/PrivateFrameworks/MediaToolbox.framework/Versions/A/MediaToolbox (compatibility version 1.0.0, current version 1.0.0)
/System/Library/PrivateFrameworks/VideoToolbox.framework/Versions/A/VideoToolbox (compatibility version 1.0.0, current version 1.0.0)
/System/Library/PrivateFrameworks/CoreMediaIOServices.framework/Versions/A/CoreMediaIOServices (compatibility version 1.0.0, current version 1.0.0)
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 751.0.0)
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 1038.0.0)
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
/System/Library/Frameworks/QuickTime.framework/Versions/A/QuickTime (compatibility version 1.0.0, current version 1584.0.0)
/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio (compatibility version 1.0.0, current version 1.0.0)
/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore (compatibility version 1.2.0, current version 1.6.0)
/System/Library/Frameworks/IOSurface.framework/Versions/A/IOSurface (compatibility version 1.0.0, current version 1.0.0)
/System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox (compatibility version 1.0.0, current version 435.0.0)
/usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.9.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 123.0.0)
/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 227.0.0)
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 44.0.0)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 550.0.0)
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 38.0.0)
/System/Library/Frameworks/CoreVideo.framework/Versions/A/CoreVideo (compatibility version 1.2.0, current version 1.6.0)

That shows a load more frameworks that the original otool output on the app binary showed. Is there a way to have otool run recursively, meaning it grabs the frameworks that the app needs, then goes in and searches each of those frameworks for dependencies?

Nicholas Riley · Accepted Answer · 2009-10-04T23:56:24.663

11

No, you'll have to run otool repeatedly, or incorporate its parsing code (here). Don't forget about handling @executable_path.

Here it is in Python (without @executable_path, canonicalization, or filenames-with-spaces supported), since this was easier than trying to debug pseudocode:

import subprocess

def otool(s):
    o = subprocess.Popen(['/usr/bin/otool', '-L', s], stdout=subprocess.PIPE)
    for l in o.stdout:
        if l[0] == '\t':
            yield l.split(' ', 1)[0][1:]

need = set(['/Applications/iTunes.app/Contents/MacOS/iTunes'])
done = set()

while need:
    needed = set(need)
    need = set()
    for f in needed:
        need.update(otool(f))
    done.update(needed)
    need.difference_update(done)

for f in sorted(done):
    print f

edited Oct 04 '09 at 23:56

answered Oct 04 '09 at 23:06

Nicholas Riley

43,532
6
101
124

I think running it repeatedly would probably work best. What would be a logical way of going about doing this? Having it just go through everything would make it go into an infinite loop... – indragie Oct 04 '09 at 23:29
Two sets (NSMutableSet) should do it: one set for paths you've already processed, and one set for paths you need to process. First populate the second set by running otool on the app. Then make a copy of the second set, empty it and run otool on each item in the copy. Don't forget to canonicalize paths after resolving @executable_path and before adding to a set (you can use realpath(3)). – Nicholas Riley Oct 04 '09 at 23:55
1

For me as well, with small adjustments. Thanks again! – Ákos Feb 21 '14 at 15:01
1

Many thanks for this, it works a treat almost a decade later! I've added a solution in the answers which builds on this which also corrects for `@executable_path` and others, and also applies `install_name_tool` to fix the references. – Josh Feb 19 '19 at 18:27
It had been long enough I had no memory of this answer whatsoever :-) Glad you found it useful as a starting point! – Nicholas Riley Feb 22 '19 at 20:51
Is there a way to adapt this code to only return libraries for which `otool -L` has current version < compatibility version? (i.e. problems) – stevec Apr 09 '20 at 16:29

score 3 · Answer 2 · answered Feb 19 '19 at 18:24

Here's my solution that I use to fix macdeployqt's output when using Homebrew-installed libraries. What I've found is that macdeployqt does a good job of putting the dylibs in the Framework folder, but it fails to fix the paths.

https://github.com/jveitchmichaelis/deeplabel/blob/master/fix_paths_mac.py

I've modified Nicholas' script to be a bit more usable - it corrects for @executable_path, @rpath and @loader_path. This isn't exactly production code, but it has let me run apps on other Macs without any dependencies already installed.

Run with: python fix_paths_mac.py ./path/to/your.app/Contents/MacOS/your_exe. i.e. point it to the binary inside an app package and it'll figure out the rest.

I've assumed that most of the problems come from stuff linked to /usr/local. So if the code detects that there's a dependency that points to a file in /usr/local, it'll fix the paths appropriately. You could change the pass statement to copy in a file if it's not in the Frameworks folder, but I've not encountered a situation where there's a missing dylib, it's just linked wrong.

import subprocess
import os
import sys
from shutil import copyfile

executable = sys.argv[1]
app_folder = os.path.join(*executable.split('/')[:-3])
content_folder = os.path.join(app_folder, "Contents")
framework_path = os.path.join(content_folder, "Frameworks")

print(executable)
print("Working in {} ".format(app_folder))

def file_in_folder(file, folder):
    return os.path.exists(os.path.join(folder, file))

def otool(s):
    o = subprocess.Popen(['/usr/bin/otool', '-L', s], stdout=subprocess.PIPE)

    for l in o.stdout:
        l = l.decode()

        if l[0] == '\t':
            path = l.split(' ', 1)[0][1:]

            if "@executable_path" in path:
                path = path.replace("@executable_path", "")
                # fudge here to strip /../ from the start of the path.
                path = os.path.join(content_folder, path[4:])

            if "@loader_path" in path:
                path = path.replace("@loader_path", framework_path)

            if "@rpath" in path:
                path = path.replace("@rpath", framework_path)

            dependency_dylib_name = os.path.split(path)[-1]

            if "usr/local" in path:
                if app_folder in s:

                    print("Warning: {} depends on {}".format(s, path))

                    if file_in_folder(dependency_dylib_name, framework_path):
                        print("Dependent library {} is already in framework folder".format(dependency_dylib_name))

                        print("Running install name tool to fix {}.".format(s))

                        if dependency_dylib_name == os.path.split(s)[-1]:
                            _ = subprocess.Popen(['install_name_tool', '-id', os.path.join("@loader_path", dependency_dylib_name), s], stdout=subprocess.PIPE)

                        _ = subprocess.Popen(['install_name_tool', '-change', path, os.path.join("@loader_path", dependency_dylib_name), s], stdout=subprocess.PIPE)
                else:
                    # Potentially you could copy in the offending dylib here.
                    pass

            yield path

need = set([executable])
done = set()

while need:
    needed = set(need)
    need = set()
    for f in needed:
        need.update(otool(f))
    done.update(needed)
    need.difference_update(done)

Kamil.S · Answer 3 · 2022-07-13T09:51:30.373

Here's my take on the topic. My script is intended to start with the app main executable and traverse recursively all the frameworks. My use is around verifying if the app referenced frameworks match the embedded ones by Xcode. The key assumptions I made to focus on non-system frameworks were:

import path must begin with @rpath
must be a framework of X.framework/X format
weak frameworks are ignored

If any of these is not needed the awk regex /weak\)$/ { next }; match($1, /^@rpath.*(.framework)/) { ... } may be modified.
First I wrote a shell script:

#/bin/sh
recursiveFrameworksParseStep() {
#fail on 1st otool error
set -e
set -o pipefail #not really POSIX compliant but good enough in MacOS where sh is emulated by bash
otool -L $1|awk -v pwd=${PWD} '/weak\)$/ { next }; match($1, /^@rpath.*(.framework)/) { gsub("@rpath",pwd"/MyApp.app/Frameworks",$1); print $1 }'| while read line; do
   if [ $1 != $line ]; then #safety check for otool -L output not to self reference resulting in infinite loop
      recursiveFrameworksParseStep $line
   fi
done
}
recursiveFrameworksParseStep MyApp.app/MyApp

It will fail on first referenced framework not found on the filesystem. That's all great, but the drawback is there is no track of visited frameworks, and there might be a lot of duplicate checks. Shell isn't particularly suited for global dictionary like structure to keep track of that.
That's why I rewrote this script using a python3 wrapper:

#!/usr/bin/python3
import subprocess
import os.path
from sys import exit

visitedFrameworks = set()

def fn(executableToProcess):
    try:
        otoolOut = subprocess.check_output(['otool','-L',executableToProcess])
    except subprocess.CalledProcessError: 
        exit(-1)

    pipeOutput = subprocess.Popen(['awk', '-v', os.path.expandvars('pwd=$PWD'),'/weak\)$/ { next };match($1, /@rpath.*(.framework)/) { gsub(\"@rpath\",pwd\"/MyApp.app/MyApp\",$1); print $1 }'], 
        stdin=subprocess.PIPE, stdout=subprocess.PIPE).communicate(otoolOut)

    lines = pipeOutput[0].decode('utf-8').split('\n')

    for outputLine in lines[1:-1]:
        if executableToProcess != outputLine:
            if outputLine not in visitedFrameworks:
                visitedFrameworks.add(outputLine)
                fn(outputLine)

fn("MyApp.app/MyApp")

Conceptually the only difference is keeping track of visited frameworks which leads to dramatic elapsed time reduction (in my case from 7-8s to less than a second).

Finally this can be made an Xcode shell script in the Target build process (shell interpreter likewise set to /usr/bin/python3).

import subprocess
import os.path
from sys import exit

visitedFrameworks = set()
missingFrameworksCandidates = set()
numberOfMissingFrameworks = 0

def fn(executableToProcess):
    global numberOfMissingFrameworks

    if os.path.exists(executableToProcess):
        otoolOut = subprocess.check_output(['otool','-L',executableToProcess])
    else:
        missingFrameworksCandidates.add(executableToProcess)
        return

    pipeOutput = subprocess.Popen(['awk', '-v', os.path.expandvars('frameworkPath=$TARGET_BUILD_DIR/$FRAMEWORKS_FOLDER_PATH'),'/weak\)$/ { next };match($1, /@rpath.*(.framework)/) { gsub(\"@rpath\",frameworkPath,$1); print $1 }'], 
        stdin=subprocess.PIPE, stdout=subprocess.PIPE).communicate(otoolOut)
    lines = pipeOutput[0].decode('utf-8').split('\n')

    linesWithoutSubFrameworks = [] 

    for outputLine in lines[1:-1]:
        frameworkPathToCheck = os.path.dirname(outputLine) + "/Frameworks/"
        #check for Frameworks within Frameworks case 
        if os.path.exists(frameworkPathToCheck) == False:
            linesWithoutSubFrameworks.append(outputLine)
            continue
        #else it's framework within framework which is processed first
        if executableToProcess != outputLine:
            if outputLine not in visitedFrameworks:
                visitedFrameworks.add(outputLine)
                fn(outputLine)

    for outputLine in linesWithoutSubFrameworks:
        if executableToProcess != outputLine:
            if outputLine not in visitedFrameworks:
                visitedFrameworks.add(outputLine)
                fn(outputLine)

# in SPM dependencies the frameworks may end up not in BrandName.app/Frameworks but in BrandName.app/Frameworks/SomeLib.framework/Frameworks
# this check accounts for that
for missingFrameworksCandidate in missingFrameworksCandidates:
    if os.path.basename(missingFrameworksCandidate) in visitedFrameworks == False:
        print(missingFrameworksCandidate," is missing")
        numberOfMissingFrameworks += 1

fn(os.path.expandvars('$TARGET_BUILD_DIR/$EXECUTABLE_PATH'))
exit(numberOfMissingFrameworks)

Validus Oculus · Answer 4 · 2020-08-09T05:50:33.473

I have following dummy script that works well. Did not overengineer as it is just a simple utility script that is rarely used for debugging purposes.

#!/usr/bin/env python

import subprocess
import re
import sys

discovered = []

def library_finder(lib):
    lib = lib.split(':')[0]
    lib = lib.split(' ')[0]
    lib = re.sub(r"[\n\t\s]*", "", lib)

    if lib in discovered:
        return

    discovered.append(lib)

    print(lib)

    if lib.startswith("@rpath"):
      return

    process = subprocess.Popen(['otool', '-L', lib],
                               stdout=subprocess.PIPE,
                               universal_newlines=True)

    deps = process.stdout.readlines()

    for dep in deps:
        library_finder(dep)

if len(sys.argv) < 2:
    print("usage: {} <binary path>".format(sys.argv[0]))
    sys.exit(1)

library_finder(sys.argv[1])

Using otool (recursively) to find shared libraries needed by an app

4 Answers4

Linked