14

First off, this is not a duplicate of any of these questions, which discuss using a single Dropbox account as the backend of a multi-user app

I have built a handful of apps, mostly on Windows Phone and Windows Store, and have lots more in the works.

I often find myself wanting to add data synching to these apps, between devices but for the same user. My apps are often free or very cheap, and I can't afford to fork out money for a database hosting server. I do have a paid hosting account with some limited DB space I could use, but in the (unlikely) event that one of my apps becomes a runaway success I'm suddenly stuck with a large hosting bill. In that case I'd have to suddenly start charging users a recurring amount for apps that were previously free, or worse, already paid for.

I keep coming back to this idea of using OneDrive, or Dropbox, or some other free cloud hosting as a database. Here's how I see it working.

Lets say we have a todo app called Jobbie

  • User installs Jobbie on both devices
  • User logs into OneDrive on both devices, creating a "jobbie" sync folder in their OneDrive folder
  • On Device A, User creates a new todo item "pick up dry cleaning"
  • Device A uploads a text file to OneDrive with the name "20153001-pick_up_dry_cleaning.item".
  • Device B scans the folder, finds a new file, and adds it to local database
  • On Device B, User marks "pick up dry cleaning" as "done", and the file is renamed to "x20153001-pick_up_dry_cleaning.item" (or deleted)
  • Device A scans the folder, sees the item has been renamed (or deleted) and removes it from local database

The obvious problems I see with this approach are

  1. items are limited to 255 - 9 characters (9 reserved for xYYYYMMDD), unless you want to download each file which would be quite slow
  2. there's no locking

Other than these two issues, are there any other problems I might face implementing such a system?

PS: I have also considered overwriting a stored SQLite file with a local copy but I figure this would be too much data overhead for a mobile device

UPDATE

I've accepted Peter Nied's answer below, which pointed out the issues I might encounter with such a system, which answered my question. However, Smarx pointed out in the comments that Dropbox has a free datastore API which I can use for my apps. This seems like a far better solution than trying to implement my own datastore on top of the file system, so I'm going with that.

UPDATE 2

The Datastore API was deprecated just 3 months after I updated the post, so its no longer available. Luckily I hadn't started developing against it at that point. You can use Dropbox as a standard flat-file storage very easily, but if you want to do any kind of sync you'll have to roll your own

UPDATE 3

June 2021

I did build something akin to "Jobbie" for personal use, but I ended up storing data in a text file using the Dropbox API, similar to how Todo.TXT works.

My Mobile app on Android (VueJs / Cordova) syncs my content changes to Dropbox, and then on any laptop I can open up the txt file in any text editor and make changes if I need to. It works very well for my use case.

Once or twice I've had the text file open in an editor with autosaving enabled (some markdown editor on Mac) and I have lost data when the latest version was overwritten with an older copy, but using Dropbox revision history it was easy enough to recover data.

If you're building a lightweight app and need to sync simple data between devices on the cheap, this works.

roryok
  • 9,325
  • 17
  • 71
  • 138
  • 1
    actually now that I think of it, you could get away with 7 characters if you use a 2 digit year format. this would work for the next 100 years at least. – roryok Jan 30 '15 at 10:33
  • 1
    I'm guessing you haven't heard of Dropbox's Datastore API. :-) https://www.dropbox.com/developers/datastore In particular, take a look at the Lists sample app: https://www.dropbox.com/developers/blog/108/new-lists-sample-app. – user94559 Jan 30 '15 at 16:07
  • If you do want to sync via files, I would suggest just a single file with all the tasks in it. (Personally, I'd use JSON.) The only tricky part with this sort of thing is dealing with conflicts created by offline devices. The simplest conflict resolution would be to just pick a winner (e.g. first writer wins). A better conflict resolution strategy would be to actually merge the lists. – user94559 Jan 30 '15 at 16:08
  • @smarx on mobile data plans that would be a lot of data overhead - sending what amounts to the entire local database every time a few tasks are added? Dropbox Datastore looks sweet, I think I'll use that! =) – roryok Jan 31 '15 at 11:00
  • "the entire local database" for a TODO list might be dwarfed by the HTTP headers themselves. It really depends on what you're actually doing. Also consider Firebase, Parse, and the other backend-as-a-service services. There are a lot of options. – user94559 Jan 31 '15 at 16:50
  • @smarx, I will be using the dropbox datastore API, thanks for the suggestion. I've accepted Peter's answer below because it answers the actual question I asked, but I've upvoted your comment because it was so helpful. – roryok Feb 02 '15 at 09:17
  • For anyone reading this - this was probably not the best choice? ;) https://blogs.dropbox.com/developers/2015/04/deprecating-the-sync-and-datastore-apis/ – RnR Aug 23 '16 at 13:39
  • You could use them as databases via some sqlite modification but you will have the same end result. If your app gets popular, you have a bill, not from the hosting provider but from dropbox. – Michael Chourdakis Dec 27 '16 at 12:08
  • 1
    @Michael the dropbox integration I was talking about would be on the user's own dropbox account, not a central one. – roryok Dec 27 '16 at 20:19
  • If the app becomes popular, everybody gets their own Dropbox and manages their own to-do list. Jobbie does not need to be designed to cross-connect every to-do list in the world, it can be a personal app with a personal Dropbox-as-a-database. I am not saying this is best in any particular application, but I found this whole idea (with modifications) inspiring, and something to consider in many situations. – DavidWalley Apr 19 '17 at 16:23
  • The dropbox datastore API has been deprecated. – Private Sep 18 '17 at 11:40

2 Answers2

2

While OneDrive and DropBox have ways to store file information and ensure some level of consistency with Etags/Ctags, building on top of of these file based systems would be hard to offer data-partitioning, transactions, and conflict management. It would be cool, but I doubt it would be cheap (time-wise) or bug free.

Using services that are already built to offer you the functional for data storage would be your best bet. There are multitudes upon multitudes of services that would give you database access to your information store. such as offerings from Microsoft's Azure product line, Amazon's AWS and many others that specialize in keeping data in sync across multiple platforms.

Peter Nied
  • 1,820
  • 1
  • 13
  • 22
  • 6
    I did mention I can't afford Azure or AWS. It's a case of scale. The apps are free, so the datastore must be free to provide. That said, your answer does cover what I asked in my question, so I'm accepting it. – roryok Feb 02 '15 at 09:15
0

I've been interested in implementing something similar for some time. I haven't yet implemented anything substantial, but I think the best solution would allow a DropBox/OneDrive/Box/SyncThing, etc. sync folder.

FYI, MoneyDance is a Quicken alternative which provides such syncing quite successfully.

I think using a consistent, Write Only, sync folder would provide not only a permanent backup of the data but also function as a "master" copy forever.

CodeMonkey
  • 101
  • 2
  • 2
  • I also think providing an SMB share as a sync option would be a great for business/enterprise users. NOTE: There are a few GitHub projects for consistent flat-file databases...sorry, I don't have the links handy at the moment. – CodeMonkey Apr 09 '23 at 14:06