Android trace file descriptors leak in production

Question

Sometimes our users having an app crash due to sudden inaccessibility of database:

1) android.database.sqlite.SQLiteCantOpenDatabaseException: unable to open database file (code 14)

or

2) java.lang.RuntimeException: Could not read input channel file descriptors from parcel.

or

3) android.database.CursorWindowAllocationException: Cursor window allocation of 2048 kb failed. # Open Cursors=1 (# cursors opened by this proc=1)

And I noticed that in that moment I can't even extract logcat:

07-17 16:57:20.722 E/APP ( 2301): 2019-07-17_16:57:20.598+0300 [FileUtils] logcat extraction error
07-17 16:57:20.722 E/APP ( 2301): java.io.IOException: Error running exec(). Command: [logcat, -d, -v, time] Working Directory: null Environment: null
07-17 16:57:20.722 E/APP ( 2301):   at java.lang.ProcessManager.exec(ProcessManager.java:211)
07-17 16:57:20.722 E/APP ( 2301):   at java.lang.Runtime.exec(Runtime.java:174)
07-17 16:57:20.722 E/APP ( 2301):   at java.lang.Runtime.exec(Runtime.java:247)
07-17 16:57:20.722 E/APP ( 2301):   at java.lang.Runtime.exec(Runtime.java:190)
....
07-17 16:57:20.722 E/APP ( 2301): Caused by: java.io.IOException: Too many open files
07-17 16:57:20.722 E/APP ( 2301):   at java.lang.ProcessManager.exec(Native Method)
07-17 16:57:20.722 E/APP ( 2301):   at java.lang.ProcessManager.exec(ProcessManager.java:209)
07-17 16:57:20.722 E/APP ( 2301):   ... 8 more

So I'm getting a question - is it possible to get in runtime (on user's device) a list of all opened file descriptors in my process (and pull it to logcat, for example)?

P.S. I failed to run a process to write logcat to file. But getting a dump app.hprof file working... Although I haven't found anything special here.

score 1 · Answer 1 · answered Oct 18 '19 at 19:49

1

Not an answer to the exact question asked, but one that might be able to help you solve the problem: ensure that all of your file descriptors are wrapped in RAII types so you can't forget to close them. In Android we use unique_fd for this: https://android.googlesource.com/platform/system/core/+/master/base/include/android-base/unique_fd.h.

This of course only helps with a subset of leaks. It won't help if the problem actually is that you're using too many file descriptors at a time, but it will help you with the more common case of just forgetting to close a file descriptor.

answered Oct 18 '19 at 19:49

Dan Albert

10,079
2
36
79

1

I think it's not real file objects leaking, it's may be broadcasts, intents or smth like this...as pointed in answer https://stackoverflow.com/a/36569771 – Mike Oct 18 '19 at 20:10
thank you for your reply. But we haven't native code in our project. Only one dependency has some native code - implementation 'pl.droidsonroids.gif:android-gif-drawable:1.1.+' – Mike Oct 19 '19 at 06:42
how do I use unique_fd in NDK project? I can't find this header filer under NDK's directory. – Ted Nov 20 '19 at 05:20
1

It's not a part of the NDK. Just copy the file I linked. – Dan Albert Nov 20 '19 at 21:50

score 0 · Answer 2 · answered Oct 19 '19 at 11:58

0

You can read and dump (I prefer Crashlytics for this kind of reporting) the list of open file descriptors from /proc/self/fd.

answered Oct 19 '19 at 11:58

Alex Cohn

56,089
9
113
307

score 0 · Answer 3 · edited Sep 06 '20 at 13:56

Following @Alex Cohn's advice I've written some code that stores

/proc/self/fd

information in cache

import android.support.v4.util.LruCache;
import android.text.format.DateUtils;
import android.util.Log;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.Date;
import java.util.Map;

public class CrashlyticsUtils {
    private static final String TAG = "Crashlytics";
    private static final LruCache<java.util.Date, String> crashlytics = new LruCache<>(20);
    private static final long INTERVAL = 10 * DateUtils.MINUTE_IN_MILLIS;
    private static long lastTime = 0;

    private static String extractShellCommand(String command) {
        Process process = null;
        BufferedReader bufferedReader = null;
        try {
            process = Runtime.getRuntime().exec(command);
            bufferedReader = new BufferedReader(
                    new InputStreamReader(process.getInputStream()));

            StringBuilder log=new StringBuilder();
            String line;
            log.append(String.format("Executing a command '%s'", command)).append("\n");
            while ((line = bufferedReader.readLine()) != null) {
                log.append(line).append("\n");
            }

            return log.toString();

        }

        catch (IOException e) {
            Log.e(TAG, "logcat extraction error", e);
            return "";
        }
        finally {
            if (bufferedReader != null) {
                try {
                    bufferedReader.close();
                } catch (IOException e) {
                    Log.e(TAG, "logcat reading error", e);
                }
            }
            if (process != null) {
                process.destroy();
            }
        }
    }

    private static void addLogNow(String log) {
        Date now = now();
        if (now.getTime() > lastTime + INTERVAL) {
            crashlytics.put(now, log);
        }
    }

    public static void collectSomeLogs() {
        addLogNow( extractShellCommand("ls -l /proc/self/fd") );
        addLogNow( extractShellCommand("lsof") );
        waitSomeTimeout();
    }

    public static void waitSomeTimeout() {
        lastTime = now().getTime();
    }

    public static void printCrashlyticsInfo() {
        Map<Date, String> snapshot = crashlytics.snapshot();
        for (Map.Entry<Date, String> entry : snapshot.entrySet()) {
            Log.d(TAG, String.format("record from %s", entry.getKey()));
            String[] strings = entry.getValue().split("\n");
            for (String line : strings) {
                Log.d(TAG, line);
            }
        }
    }

    private static Date now() {
        return new java.util.Date();
    }
}

So I'm written calls among my project

CrashlyticsUtils.collectSomeLogs()

and then (when uncaught exception happened) write logs to logcat

CrashlyticsUtils.printCrashlyticsInfo()

(you can write in print method anything else that desirable)

Android trace file descriptors leak in production

3 Answers3