Active Collab 5 Webhooks / Maintaining "metric" data

Question

I have an application I am working on that basically takes the data from Active Collab and creates reports / graphs out of the data. The API itself is insufficient to get the proper data on a per request basis so I resorted to pulling the data down into a separate data set that can be queried more efficiently.

So in order to avoid needing to query the entire API constantly I decided to make use of webhooks in order to make the transformations to the relevant data and lower the need to resync the data.

However I notice not all events are sent, notably the following.

TaskListUpdated
MemberUpdated
TimeRecordUpdated
ProjectUpdated

There is probably more but these are the main ones I noticed so far,

Time reports is probably the most important, in fact it missing from webhooks means that almost any application has a good chance of incorrect data if it needs time record data. Its fairly common to do a typo in a time record and then adjust it later.

So am I missing anything here? Is there some way to see these events reliably?

EDIT: In order to avoid a long comment to Ilija I am putting the bulk here.

Webhooks apart, what information do you need to pull? API that powers time tracking reports can do all sorts of cross project filtering, so your approach to keep a separate database may be an overkill.

Basically we are doing a multi-variable tiered time report. It can be sorted / grouped by any conceivable method you may want to look at.

http://www.appsmagnet.com/product/time-reports-plus/

This is the closest to what we are trying to do, back when we used Active Collab 4 this did the job, but even with it we had to consolidate it in our own spreadsheets.

So the idea of this is to better integrate our Active Collab data into our own workflow.

So the main data we are looking for in this case is

Job Types
Projects
Task Lists
Tasks
Time Records
Categories
Members / Clients
Companies

These items can feed not only our reports, but many other aspects of our company as well. For us Active Collab is the point of truth, so we want the data quickly accessible and fully query-able.

So I have set up a sync system that initially grabs all the data it can from Active Collab and then uses a mix of cron's and webhooks to keep it up to date.

Cron jobs work well for all aspects that do not have "sub items" (projects/tasks/task lists/time records). So those I need to rely on the webhook since syncing them takes to much time to be able to keep it up to date in real time.

For the webhook I noticed the above do not carry through. Time Records I figured out a way around it listed in my answer, and member can be done through the cron. However Task list and project updating are the only 2 of some concern. Project is fairly important as the budget can change and that would be used in reports, task lists has the start / end dates that could be used as well. Since going through every project / task list constantly to see if there is a change is really not a great idea I am looking for a way to reliably see updates for them.

I have based this system on https://developers.activecollab.com/api-documentation/ but I know there are at least a few end points that are not listed.

Cross-project time-record filtering using Active Collab 5 API

This question is actually from another developer on the same system (and also shows a TrackingFilter report not listed in the docs). Due to issues with maintaining an accurate set of data we had to adapt it. I actually notice that you (Ilija) are the person replying and did recommend we move over to this style of system.

Answer 1

This is not a total answer but a way to solve the issue with TimeRecordUpdated not going through the webhook.

There is another API endpoint for /whats-new This endpoint describes changes for the last day or so and it has a category called TrackingObjectUpdatedActivityLog this refers to an updated time record.

So I set up a cron job to check this fairly consistently and manually push the TimeRecordUpdated event through my system to keep it consistent.

For MemberUpdated since the data for a member being updated is unlikely to affect much, having a daily cron for checking the users seems good enough.

ProjectUpdated could technically be considered the same, but with the absence of TaskListUpdated that leads to far to many api calls to sync the data. I have not found a solution for this yet unfortunately.