Currently I am working on a data gathering project in which we are using deployed data gathering devices as well as a centralized server that will store all aggregated data from multiple deployed devices. Both the deployed data gathering devices and the server store data in the same general structure (mainly timestamp and associated values).
The deployed devices use a SQLite database for local storage while the central server will probably be using MySQL or PostgreSQL. I need to be able to have the deployed devices send data to the central server in as close to real time as possible while at the same time creating a local backup copy on their internal SQLite database. If the connection goes down the deployed devices need to still write the data locally and then update the server database with the unsent data as soon as a connection can be reestablished.
The deployed device sensor configuration will also be stored in the local SQLite database and should also be synced with the server database whenever changes are made and a connection is possible and any subsequent changes to the sensor config on either the local device or server should propagate both ways with the local configuration having priority if both are changed while there is no connection, that way after initial configuration is done on the deployed device any subsequent needed changes can be made either on the server or the device itself.
I found a number of options for syncing databases with Django but they did not appear to be in real time which is important for this. Both the deployed device and central server software will be very similar and both use a Django back-end with the server being able to view and store data from multiple deployed devices and able configure each one individually while each deployed device will only store its own captured data and its own configuration.
Are there any projects that may be of use for this type of data management or any suggestions of how to implement this?
