How to read a Volatile layer from a batch pipeline?

Question

How can I read from a Volatile Layer using a Batch pipeline ?

public IntermediateData compileInFn(Pair<Key, Meta> in, LogContext logContext) {
    String partitionID = in.getKey().partition().toString();        
    try {
        if (!partitionID.isEmpty()) {
            // Retrieve the partition.
            Payload payload = retriever.getPayload(in.getKey(), in.getValue(), logContext);          

            }
        }
    } catch (Exception e) {

    }        
    return new IntermediateData(in.getKey(), testResults);
}

score 3 · Answer 1 · answered Aug 29 '18 at 20:10

3

Well, it's actually no different than reading from a versioned catalog. However, this probably isn't what you actually want to do. A scheduled batch pipeline running a DPL compiler only triggers when a new version is published. In volatile layers, metadata isn't necessarily published (which triggers a version update) whenever new data is pushed to the volatile layer. So, unless the data provider is actually updating metadata every time they publish to a volatile layer, the version might not update and the batch pipeline may not get triggered. You could manually initiate a batch pipeline job and set the processing type to "reprocess" which will read the entire catalog but this will only run the compiler once.

answered Aug 29 '18 at 20:10

ichow

131
4

I am new to types of layers HERE offers, can you please provide the details on the differences between a versioned vs volatile layer. – Vishrant Jan 19 '19 at 19:22
2

@Vishrant Versioned layers contain versioned data so one can retrieve previous versions of a partition. Volatile data is essentially a key/value store where only the "current" version of a key can be retrieved. Please read [the documentation](https://developer.here.com/olp/documentation/get-started/dev_guide/shared_content/topics/olp/concepts/layers.html) for further details. – ichow Feb 04 '19 at 15:36

How to read a Volatile layer from a batch pipeline?

1 Answers1