3

Case:

I am aware this question is a bit long (sorry for that). I think a detailed description explains my case better. I receive device data from a streaming endpoint. Each payload in stream includes multiple data entity. I create following structure to be stored in dynamo

db_pk = str(msg.get('key'))
db_sk = str(msg.get('sortKey'))
area_id = str(msg.get('area'))
## other entities in following mapped_values...

mapped_values = {
    startTime: {
        'alert_msg': alert,
        'area ': area_id,
        'speed': speed,
        'state': state,
        'startTime': startTime,
        'duration': duration,
        'inspector': inspector_id
    }
}
# Data item
lt_item = {
    'id': db_pk,
    'serialNo': db_sk,
    'value_map': [mapped_values]
}

I use the following update statement to create a mapping of incoming data in each payload.

table = dynamo_db.Table('device')
table.update_item(
        Key = {
            'id': db_pk,
            'serialNo': db_sk
        },
        UpdateExpression = 'SET #value_map = list_append(#value_map, :mapped_values)',
        ConditionExpression = 'attribute_exists(#value_map)',
        ExpressionAttributeValues = {
            ':mapped_values': [mapped_values]
        },
        ExpressionAttributeNames = {
            '#value_map': 'value_map'
        }
    )

Current Result

The above update expression creates a map of messages under the value_map key in dynamo, and whenever a new message received, it is appended into the map like the following:

{
   "id": "KD_125",
   "serialNo": "KDCRT-231"
   "value_map": [
      {
         "2019-05-29 14:36:03": { #first msg in payload
             "alert_msg": "0x12 LOGICAL ERROR AT DIRECTION SELECT SIGNAL",
             "area": "TX-112",
             "speed": "65",
             "startTime": "2019-05-29 14:36:03",
             "state": "ACTIVE",
             "duration": "None"
             "inspector": "None"
             }
      },
      {
        "2019-05-29 14:36:03": { # second msg, same timestamp with first one
             "alert_msg": "0x12 LOGICAL ERROR AT DIRECTION SELECT SIGNAL",
             "area": "None",
             "speed": "None",
             "startTime": "2019-05-29 14:36:03",
             "state": "PASSIVE",
             "duration": "1200"
             "inspector" "422TX19"
             }
        }
          #another message with different timestamp and data values
     ]
}

Problem and Requirement

Some of the messages are sent with same timestamp (startTime). If state='ACTIVE' in message, it includes all data except duration and inspector; and If state='PASSIVE', it includes duration, inspector and others except area and speed (see above sample).

If the same timestamp is sent in the payload, I need to replace the None values (duration and inspector) and state of 'ACTIVE'state message with the 'PASSIVE'state message instead of creating two objects with the same timestamp. So the final shape should be like:

{
   "id": "KD_125",
   "serialNo": "KDCRT-231"
   "value_map": [
      {
         "2019-05-29 14:36:03": { 
             "alert_msg": "0x12 LOGICAL ERROR AT DIRECTION SELECT SIGNAL",
             "area": "TX-112",
             "speed": "65",
             "startTime": "2019-05-29 14:36:03",
             "state": "PASSIVE", # updated from PASSIVE state msg
             "duration": "1200" # updated values from PASSIVE state msg
             "inspector": "422TX19" # updated values from PASSIVE state msg
             }
      }
      # other messages with different timestamps
   ]
}

I tried different update expressions such as if_not_exists. In addition, tested with

ExpressionAttributeNames = {
            '#value_map': 'value_map.startTime'
        }

option.. But no success so far. I hope someone can suggest me a way to make this update..

ylcnky
  • 775
  • 1
  • 10
  • 26
  • 1
    I think part of your problem is that you are building a list of maps - what you really want is a map of maps (because element names in a map are unique). Then you can build a condition expression around attribute_exists or attribute_not_exists (using the full path of the nested map). Does that make sense? – NoSQLKnowHow May 31 '19 at 21:09
  • I am testing the same direction as well. I guess with one update statement, the logic will be complicated. So perhaps, multiple if-else conditions and corresponding update_expressions might work. I will put my answer if I reach sth. – ylcnky Jun 01 '19 at 12:06

0 Answers0