0

We are using Dropbox API v2 in our Android app to upload database files (.db) to a user's Dropbox App folder. Below is the code for uploading file:

    InputStream inputStream = null;
    FileMetadata obj = null;
    try {
        inputStream = new FileInputStream(dbFile);
        obj = mDbxClient.files().uploadBuilder(remoteFilePath)
                .withMode(WriteMode.OVERWRITE)
                .uploadAndFinish(inputStream);
    } catch (DbxException | IOException e) {
        obj = null;
        mException = e;
    } finally {
        if (inputStream != null)
            try {
                inputStream.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
    }

The same file is uploaded in quick succession 3-4 times. Now, everything works fine when Internet speed is fast (4G, Wifi) but when a user switches to 2G/3G network, the file isn't uploaded fully. The callback method onUploadComplete() is called but the file size here has drastically reduced. I have attached the version history of a database file. https://www.dropbox.com/s/s7y43707ic1kqxg/dropbox.jpg?dl=0

You can see that the file size suddenly reduces to 104KB. This becomes a corrupt database file.

Is there any way to fix this? Should I change the upload method? (maybe use UploadSession() Or Uploads shouldn't be called in quick succession? Please guide

EDIT:

Below is the call to upload file:

            AsyncTaskCompat.executeParallel(new DropboxUploadFileTask(DropboxClientFactory.getClient(), new DropboxUploadFileTask.Callback() {
            @Override
            public void onUploadComplete(FileMetadata result) {
                String message = result.getName() + " size " + result.getSize() + " modified " +
                        java.text.DateFormat.getDateTimeInstance().format(result.getClientModified());
                Log.i("BKSync upload ends: ", message);
            }

            @Override
            public void onError(Exception e) {
                Log.e("BKSync", "Failed to upload file.", e);
            }
        }, context), glDbName, "");


        class DropboxUploadFileTask extends AsyncTask<String, Void, FileMetadata> {

private final DbxClientV2 mDbxClient;
private final Callback mCallback;
private Exception mException;
private Context mContext;

public interface Callback {
    void onUploadComplete(FileMetadata result);
    void onError(Exception e);
}

DropboxUploadFileTask(DbxClientV2 dbxClient, Callback callback, Context context) {
    mDbxClient = dbxClient;
    mCallback = callback;
    mContext = context;
}
@Override
protected void onPreExecute() {
    Log.e("BKSync", "upload starts");
}

@Override
protected void onPostExecute(FileMetadata result) {
    Log.e("BKSync", "upload ends");
    super.onPostExecute(result);
    if (mException != null) {
        mCallback.onError(mException);
    } else if (result == null) {
        mCallback.onError(null);
    } else {
        mCallback.onUploadComplete(result);
    }
}

@Override
protected FileMetadata doInBackground(String... params) {
    String dbFileName = params[0];
    File dbFile = mContext.getDatabasePath(dbFileName);
    String remoteFolderPath = params[1];
    String remoteFileName = dbFile.getName();

    InputStream inputStream = null;
    FileMetadata obj = null;
    try {
        inputStream = new FileInputStream(dbFile);
        obj = mDbxClient.files().uploadBuilder(remoteFolderPath + "/" + remoteFileName)
                .withMode(WriteMode.OVERWRITE)
                .uploadAndFinish(inputStream);
    } catch (DbxException | IOException e) {
        obj = null;
        mException = e;
        Log.e("BKSync", "exception1");
        mException.printStackTrace();
    } finally {
        if (inputStream != null)
            try {
                inputStream.close();
            } catch (IOException e) {
                Log.e("BKSync", "exception2");
                e.printStackTrace();
            }
    }

    return obj;
}

}

And below is the log:

            05-29 12:08:06.322 11232-11232/com.bk E/BKSync: upload starts
            05-29 12:08:13.129 11232-11232/com.bk E/BKSync: upload ends
            05-29 12:08:13.130 11232-11232/com.bk I/BKSync upload ends:: demo2_local.db size 692224 modified May 29, 2018 12:08:13 PM
            05-29 12:08:13.471 11232-11232/com.bk E/BKSync: upload starts
            05-29 12:08:22.992 11232-11232/com.bk E/BKSync: upload ends
            05-29 12:08:22.993 11232-11232/com.bk I/BKSync upload ends:: demo2_local.db size 692224 modified May 29, 2018 12:08:23 PM
            05-29 12:08:45.181 11232-11232/com.bk E/BKSync: upload starts
            05-29 12:08:48.692 11232-11232/com.bk E/BKSync: upload starts
            05-29 12:08:49.137 11232-11232/com.bk E/BKSync: upload starts
            05-29 12:08:51.962 11232-11232/com.bk E/BKSync: upload starts
            f05-29 12:09:24.183 11232-30020/com.bk E/BKSync: exception1
            05-29 12:09:24.184 11232-30020/com.bk W/System.err: com.dropbox.core.NetworkIOException: Unable to resolve host "content.dropboxapi.com": No address associated with hostname
                    at com.dropbox.core.DbxUploader.finish(DbxUploader.java:235)
                    at com.dropbox.core.DbxUploader.uploadAndFinish(DbxUploader.java:106)
                    at com.dropbox.core.v2.DbxUploadStyleBuilder.uploadAndFinish(DbxUploadStyleBuilder.java:92)
            05-29 12:09:24.185 11232-30020/com.bk W/System.err:     at com.bk.DropboxUploadFileTask.doInBackground(DropboxUploadFileTask.java:81)
                    at com.bkbk.DropboxUploadFileTask.doInBackground(DropboxUploadFileTask.java:20)
                    at android.os.AsyncTask$2.call(AsyncTask.java:304)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:237)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
                    at java.lang.Thread.run(Thread.java:762)
            05-29 12:09:24.189 11232-30020/com.bk W/System.err: Caused by: java.net.UnknownHostException: Unable to resolve host "content.dropboxapi.com": No address associated with hostname
                    at java.net.Inet6AddressImpl.lookupHostByName(Inet6AddressImpl.java:125)
                    at java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:74)
                    at java.net.InetAddress.getAllByName(InetAddress.java:752)
                    at okhttp3.Dns$1.lookup(Dns.java:39)
                    at okhttp3.internal.connection.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:185)
                    at okhttp3.internal.connection.RouteSelector.nextProxy(RouteSelector.java:149)
                    at okhttp3.internal.connection.RouteSelector.next(RouteSelector.java:84)
            05-29 12:09:24.190 11232-30020/com.bk W/System.err:     at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:213)
                    at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:134)
                    at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:113)
                    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
                    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
                    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
            05-29 12:09:24.193 11232-30020/com.bk W/System.err:     at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
                    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
            05-29 12:09:24.194 11232-30020/com.bk W/System.err:     at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
                    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
            05-29 12:09:24.195 11232-30020/com.bk W/System.err:     at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
                    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:125)
                    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
                    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
                    at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
                    at okhttp3.RealCall$AsyncCall.execute(RealCall.java:147)
                    at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
                    ... 3 more
            05-29 12:09:24.196 11232-30020/com.bk W/System.err: Caused by: android.system.GaiException: android_getaddrinfo failed: EAI_NODATA (No address associated with hostname)
                    at libcore.io.Posix.android_getaddrinfo(Native Method)
                    at libcore.io.ForwardingOs.android_getaddrinfo(ForwardingOs.java:55)
                    at java.net.Inet6AddressImpl.lookupHostByName(Inet6AddressImpl.java:106)
                    ... 26 more
            05-29 12:09:24.211 11232-11232/com.bk E/BKSync: upload ends
                Failed to upload file.

            05-29 12:09:24.300 11232-11232/com.bk E/BKSync: upload starts
            05-29 12:09:24.314 11232-30020/com.bk E/BKSync: exception1
            05-29 12:09:24.315 11232-30020/com.bk W/System.err: com.dropbox.core.NetworkIOException: Pipe closed
                    at com.dropbox.core.DbxUploader.uploadAndFinish(DbxUploader.java:103)
                    at com.dropbox.core.v2.DbxUploadStyleBuilder.uploadAndFinish(DbxUploadStyleBuilder.java:92)
                    at com.bk.DropboxUploadFileTask.doInBackground(DropboxUploadFileTask.java:81)
                    at com.bk.DropboxUploadFileTask.doInBackground(DropboxUploadFileTask.java:20)
                    at android.os.AsyncTask$2.call(AsyncTask.java:304)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:237)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
            05-29 12:09:24.316 11232-30020/com.bk W/System.err:     at java.lang.Thread.run(Thread.java:762)
                Caused by: com.dropbox.core.util.IOUtil$WriteException: Pipe closed
                    at com.dropbox.core.util.IOUtil.copyStreamToStream(IOUtil.java:61)
                    at com.dropbox.core.util.IOUtil.copyStreamToStream(IOUtil.java:68)
                    at com.dropbox.core.util.IOUtil.copyStreamToStream(IOUtil.java:43)
                    at com.dropbox.core.http.HttpRequestor$Uploader.upload(HttpRequestor.java:98)
                    at com.dropbox.core.DbxUploader.uploadAndFinish(DbxUploader.java:98)
                    ... 8 more
            05-29 12:09:24.318 11232-30020/com.bk W/System.err: Caused by: java.io.IOException: Pipe closed
                    at java.io.PipedInputStream.checkStateForReceive(PipedInputStream.java:267)
                    at java.io.PipedInputStream.receive(PipedInputStream.java:233)
                    at java.io.PipedOutputStream.write(PipedOutputStream.java:149)
                    at com.dropbox.core.util.IOUtil.copyStreamToStream(IOUtil.java:59)
                    ... 12 more
            05-29 12:09:24.441 11232-11232/com.bk E/BKSync: upload ends
            05-29 12:09:24.442 11232-11232/com.bk E/BKSync: Failed to upload file.
                com.dropbox.core.NetworkIOException: Pipe closed
                    at com.dropbox.core.DbxUploader.uploadAndFinish(DbxUploader.java:103)
                    at com.dropbox.core.v2.DbxUploadStyleBuilder.uploadAndFinish(DbxUploadStyleBuilder.java:92)
                    at com.bk.DropboxUploadFileTask.doInBackground(DropboxUploadFileTask.java:81)
                    at com.bk.DropboxUploadFileTask.doInBackground(DropboxUploadFileTask.java:20)
                    at android.os.AsyncTask$2.call(AsyncTask.java:304)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:237)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
                    at java.lang.Thread.run(Thread.java:762)
                 Caused by: com.dropbox.core.util.IOUtil$WriteException: Pipe closed
                    at com.dropbox.core.util.IOUtil.copyStreamToStream(IOUtil.java:61)
                    at com.dropbox.core.util.IOUtil.copyStreamToStream(IOUtil.java:68)
                    at com.dropbox.core.util.IOUtil.copyStreamToStream(IOUtil.java:43)
                    at com.dropbox.core.http.HttpRequestor$Uploader.upload(HttpRequestor.java:98)
                    at com.dropbox.core.DbxUploader.uploadAndFinish(DbxUploader.java:98)
                    at com.dropbox.core.v2.DbxUploadStyleBuilder.uploadAndFinish(DbxUploadStyleBuilder.java:92) 
                    at com.bk.DropboxUploadFileTask.doInBackground(DropboxUploadFileTask.java:81) 
                    at com.bk.DropboxUploadFileTask.doInBackground(DropboxUploadFileTask.java:20) 
                    at android.os.AsyncTask$2.call(AsyncTask.java:304) 
                    at java.util.concurrent.FutureTask.run(FutureTask.java:237) 
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133) 
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607) 
                    at java.lang.Thread.run(Thread.java:762) 
                 Caused by: java.io.IOException: Pipe closed
                    at java.io.PipedInputStream.checkStateForReceive(PipedInputStream.java:267)
                    at java.io.PipedInputStream.receive(PipedInputStream.java:233)
                    at java.io.PipedOutputStream.write(PipedOutputStream.java:149)
                    at com.dropbox.core.util.IOUtil.copyStreamToStream(IOUtil.java:59)
                    at com.dropbox.core.util.IOUtil.copyStreamToStream(IOUtil.java:68) 
                    at com.dropbox.core.util.IOUtil.copyStreamToStream(IOUtil.java:43) 
                    at com.dropbox.core.http.HttpRequestor$Uploader.upload(HttpRequestor.java:98) 
                    at com.dropbox.core.DbxUploader.uploadAndFinish(DbxUploader.java:98) 
                    at com.dropbox.core.v2.DbxUploadStyleBuilder.uploadAndFinish(DbxUploadStyleBuilder.java:92) 
                    at com.bk.DropboxUploadFileTask.doInBackground(DropboxUploadFileTask.java:81) 
                    at com.bk.DropboxUploadFileTask.doInBackground(DropboxUploadFileTask.java:20) 
                    at android.os.AsyncTask$2.call(AsyncTask.java:304) 
                    at java.util.concurrent.FutureTask.run(FutureTask.java:237) 
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133) 
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607) 
                    at java.lang.Thread.run(Thread.java:762) 

As you can see, exceptions raised by the app. Issue occurs when user starts upload on 4G and then due to incoming call, network switches to 3G. File size on Dropbox reduces drastically then.

Please help.

varun
  • 465
  • 6
  • 23
  • zip the file & upload – onkar May 23 '18 at 10:16
  • cant do that because the file is being used on other devices also e.g. windows – varun May 23 '18 at 11:08
  • Uploading multiple times in a row shouldn't matter. First, in order to track down where the loss in size is occurring, can you check the length of the local data before uploading? Second, does this _only_ happen when the user switches their network connection _during_ the upload, or does this happen for any upload on 2G/3G? – Greg May 23 '18 at 14:28
  • Hi, I just tested the entire scenario. It just happens when user switches network during an upload. From Wifi or 4G to 2G/3G during upload. Please help – varun May 24 '18 at 10:50
  • Thanks! It sounds like the file is getting truncated but somehow still getting committed when the network changes. I would expect an exception to be raised and the upload to fail in this case. It sounds like you're able to reproduce this. Can you share the rest of the code to reproduce the issue exactly? (E.g., the `onUploadComplete` you mentioned, etc.) By the way, I see you're just catching and storing exceptions (`mException = e;`). Can you double check if there are any exceptions raised there? – Greg May 24 '18 at 16:15
  • hi, let me log everything and rerun the case. will get back to you. thanks – varun May 26 '18 at 05:19
  • @Greg can you please check the question now? I have updated. – varun May 29 '18 at 07:42
  • @varun You mentioned earlier that when this issue occurs, `onUploadComplete` is called, but I don't see that in your output. I do see two successful uploads, and then a third failed one with a variety of exceptions. Can you clarify what you mean when you said `onUploadComplete` is called? Is this output not representative of the issue? The top exception seems to be a DNS failure, which would fail before any file data is uploaded. – Greg May 30 '18 at 19:40
  • @Greg I can see in the logs that onUploadComplete is not called for third upload. You are correct. After two successful uploads, third fails. But why does the file size reduce? Please note that we are calling upload() consecutively even if third upload fails. – varun May 31 '18 at 05:04

1 Answers1

0

I've created an issue on github: https://github.com/dropbox/dropbox-sdk-java/issues/191 for tracking. It looks like a bug in Uploader code. But we don't have plan to immediately fix it and release a new version.

So my suggestion of work around would be:

  1. If you get an error and meanwhile the partial file was uploaded to Dropbox, please make sure you retry after error.
  2. If you don't get error and the partial file was saved. (unlikely). You may want to compare the content hash of the file you just uploaded. The content hash is returned as the result of upload. And for the hash function you can refer https://www.dropbox.com/developers/reference/content-hash
k3dvJeGJLshdGLq8XtkB
  • 1
  • 1