1

just started using Keen.IO…very nice product. I have a question please on data modeling:

I am tracking mobile app registration events that have the following attributes:

Device Id,
Platform, Manufacturer, Marketing Version,
Language and
Latitude and Longitude

The problem is that the mobile app sends this registration every time the device changes location. In a way, it is to maintain the device latest location and keep track of historic changes. I created a new collection for devices and it seems to be OK as I am able to select_unique the keen.location.coordinates to draw the devices on a map.

The question though is how would I know the latest location update? So now I have several events in the collection for the same device (uniquely identified by a device id) but no way of finding the latest!

Thank you for any pointer.

Regards, Khaled

Khaled Hikmat
  • 289
  • 2
  • 13

2 Answers2

1

It is technically possible to use existing Keen query types to retrieve the latest position of a device as you require, the problem is doing so is inefficient and doesn't scale well (mainly because Keen is designed to store/query event data, not entity data). You'd have to plug the Device Id you retrieve from the select_unique into individual queries for each device to find the latest location. Doing so would bump you up against concurrency and/or rate limiting fairly quickly.

That being said, that leaves you with a few options:

  1. Use Keen's S3 integration. This feature writes all of your raw events to S3 for consumption however you see fit. You could use this data to create a pipeline to a separate entity database which would allow you to retrieve the device's latest location. Note: there is an extra charge for this feature.

  2. Post device/location data to a separate entity database at the same time you're sending the event to Keen, and use that database as the source of the device's latest location.

Either of these options will work, it just depends on how you'd like to implement/manage data collection in your application.

terrhorn
  • 510
  • 4
  • 6
  • Terrhorn, thank you very much for your answer. Yes....of course I do have the latest update in my entity database. The problem is that I am building a dashboard that relies on Keen data and I really did not want to pull data from my entity database. – Khaled Hikmat Jun 23 '15 at 18:17
  • Given this is not going to work well the way I modeled it, what is a better approach to model my problem? Thanks. – Khaled Hikmat Jun 23 '15 at 18:29
0

I think you can achieve this by combining a select_unique query with a group_by clause and looking at the last item in the result.

Though, as terrhorn noted – it's not a scalable solution. I wouldn't use it for anything other than dashboarding.

Here's a quick example:

var query = new Keen.Query("select_unique", { eventCollection: "devices", targetProperty: "location", group_by: "device_id" });

The result will look something like this:

{ "result": [ { "device_id": "4252f729-7bdc-a487-be15-984999a96683", "result": [ "location_1", "location_2" ] } ] }

NOTE: This is theoretical – I'm not sure if Keen sorts the result of the group_by by the order in which the events came in or not – but it's a good assumption to test. I don't see why they wouldn't.

Nima Gardideh
  • 575
  • 1
  • 4
  • 10
  • Thank you sir! Yeah...that was one of the first things I thought of....but it was hard to make sure the location I got is actually the latest, especially that Keen does not guarantee any order and things might work differently when the data got bigger – Khaled Hikmat Jun 24 '15 at 05:28
  • As terrhorn advised, I want to try another approach that relies on my entity database. I will maintain in each device event latitude and longitude location adjustments! With help from my entity database, I know the first time the device registered.....so the first device event will contain the actual initial latitude and longitude. Then for each subsequent update event, I will store the adjustments made to the location. To know the actual/latest device locations, I will sum up the latitude and longitude grouped by the device id. Is this naive thinking? – Khaled Hikmat Jun 24 '15 at 05:36
  • One thing to keep in mind when summing* by lat/long grouped by the device id: the results you expect may not be what you receive depending on how frequently lat/long is updated. You may only ever receive a total of 1 for each lat/long. * - I believe you mean counting by lat/long. – terrhorn Jun 24 '15 at 13:06
  • Also, in regards to sorting of results, Keen does not sort the results of the `group_by` in any particular order. You're not guaranteed to receive the results in the order you expect each time you receive a result. – terrhorn Jun 24 '15 at 13:17
  • Ok...thank you sir. Let me run a few tests ...I will update this thread if I get something positive. Thanks. – Khaled Hikmat Jun 24 '15 at 18:55