I am writing a Java application where I am receiving related data from two sources, from first source I receive all data item one time and from second source I am receiving data as continues stream. There are some common fields in both data items, I need to perform search operation on data from first source while manipulating data from second source so I an add required fields in the data of second source. Not sure what is best data structure to store data from first source so I can search the related item while handling data from second source.
I am explaining the data here, both are in JSON format First source Data is from AWS s3 bucket and its actually a metadata item and I am reading entire data from bucket before manipulating data from second source
{
"id":"abc",
"entity":"some_entity",
"feature":"some_feature",
"originalFeature":"some_feature",
"dataType":"date",
"example":"01/10/2018T12:00:00.444",
"source":"some_source",
"priority":1,
"ts":"2018-10-30T10:19:57.556"
}
Data from second source with continuous stream
{
"id":"xyz",
"entity":"some_entity",
"feature":"some_feature",
"value":"some_value",
"source":"some_source",
"seq":1,
"ts":"2019-04-24T16:03:32.398"
}
The common data items are entity, feature and source and I need to get priority and dataType information from metadata (first source) while manipulating data from second source.
I hope it might give better picture of my problem so I need to first keep meta data in some data structure before manipulating data from second source so I can search required values efficiently.
