Case:
I am aware this question is a bit long (sorry for that). I think a detailed description explains my case better. I receive device data from a streaming endpoint. Each payload in stream includes multiple data entity. I create following structure to be stored in dynamo
db_pk = str(msg.get('key'))
db_sk = str(msg.get('sortKey'))
area_id = str(msg.get('area'))
## other entities in following mapped_values...
mapped_values = {
startTime: {
'alert_msg': alert,
'area ': area_id,
'speed': speed,
'state': state,
'startTime': startTime,
'duration': duration,
'inspector': inspector_id
}
}
# Data item
lt_item = {
'id': db_pk,
'serialNo': db_sk,
'value_map': [mapped_values]
}
I use the following update statement to create a mapping of incoming data in each payload.
table = dynamo_db.Table('device')
table.update_item(
Key = {
'id': db_pk,
'serialNo': db_sk
},
UpdateExpression = 'SET #value_map = list_append(#value_map, :mapped_values)',
ConditionExpression = 'attribute_exists(#value_map)',
ExpressionAttributeValues = {
':mapped_values': [mapped_values]
},
ExpressionAttributeNames = {
'#value_map': 'value_map'
}
)
Current Result
The above update expression creates a map of messages under the value_map
key in dynamo, and whenever a new message received, it is appended into the map like the following:
{
"id": "KD_125",
"serialNo": "KDCRT-231"
"value_map": [
{
"2019-05-29 14:36:03": { #first msg in payload
"alert_msg": "0x12 LOGICAL ERROR AT DIRECTION SELECT SIGNAL",
"area": "TX-112",
"speed": "65",
"startTime": "2019-05-29 14:36:03",
"state": "ACTIVE",
"duration": "None"
"inspector": "None"
}
},
{
"2019-05-29 14:36:03": { # second msg, same timestamp with first one
"alert_msg": "0x12 LOGICAL ERROR AT DIRECTION SELECT SIGNAL",
"area": "None",
"speed": "None",
"startTime": "2019-05-29 14:36:03",
"state": "PASSIVE",
"duration": "1200"
"inspector" "422TX19"
}
}
#another message with different timestamp and data values
]
}
Problem and Requirement
Some of the messages are sent with same timestamp (startTime
). If state='ACTIVE'
in message, it includes all data except duration
and inspector
; and If state='PASSIVE'
, it includes duration, inspector
and others except area
and speed
(see above sample).
If the same timestamp
is sent in the payload, I need to replace the None
values (duration
and inspector
) and state
of 'ACTIVE'
state message with the 'PASSIVE'
state message instead of creating two objects with the same timestamp
. So the final shape should be like:
{
"id": "KD_125",
"serialNo": "KDCRT-231"
"value_map": [
{
"2019-05-29 14:36:03": {
"alert_msg": "0x12 LOGICAL ERROR AT DIRECTION SELECT SIGNAL",
"area": "TX-112",
"speed": "65",
"startTime": "2019-05-29 14:36:03",
"state": "PASSIVE", # updated from PASSIVE state msg
"duration": "1200" # updated values from PASSIVE state msg
"inspector": "422TX19" # updated values from PASSIVE state msg
}
}
# other messages with different timestamps
]
}
I tried different update expressions such as if_not_exists
. In addition, tested with
ExpressionAttributeNames = {
'#value_map': 'value_map.startTime'
}
option.. But no success so far. I hope someone can suggest me a way to make this update..