How do I load data from a Hadoop Avro source into ActivePivot store?

Question

Please see [ask] and [The perfect question](http://codeblog.jonskeet.uk/2010/08/29/writing-the-perfect-question/). — Paul Roub, Jul 31 '17 at 21:34

score 0 · Answer 1 · answered Jul 31 '17 at 21:46

0

Look in the ActivePivot Sandbox project you should have access to and check out where channels are use to persist data to datastores.

answered Jul 31 '17 at 21:46

UserF40

3,533
2
23
34

1

I dont think the ActivePivot Sandbox project is available with me. But, I have verified the ActivePivot support URL (https://support.quartetfs.com/). I could not able to see any samples to load custom source. My requirement is to load the data into the datastore which is received as avro file (ex: Employee.avro). Only thing I can see to write the code for custom source is, it should Implement the ISource interface and override fetch & listen methods. – user3090026 Aug 01 '17 at 15:25

score 0 · Answer 2 · answered May 16 '18 at 10:13

Suppose you have class that should be posted to the cube.

public class PostVectorFact {
     private final LocalDate date;
     //other fields
     //getters, equals, hashCode
}

In your spring configuration you need to define message chanell

@Autowired
protected IDatastore datastore;

@Autowired
protected KeepArbitraryEpochPolicy epochPolicy;

@Autowired
protected LocationHelper locationHelper;

public IMessageChannel<String, Object> returnsRealTimeChannel() {
    return pojoMessageChannelFactory().createChannel("RealTimeReturns", DataStoreConstants.RETURNS_STORE, tuplePublisher());
}

@Bean
ITuplePublisher<String> tuplePublisher() {
    return new VersionCheckingTuplePublisher(datastore, DataStoreConstants.RETURNS_STORE, DataStoreConstants.LATEST_RETURNS_STORE, 2L, epochPolicy, locationHelper);
}

@Bean
public CubeReturnsFactPublisher cubeReturnsFactPublisher() {
    return new CubeReturnsFactPublisher(returnsRealTimeChannel());
}

And the publisher class itself:

public class CubeReturnsFactPublisher {

    IMessageChannel<String, Object> messageChannel;

    public CubeReturnsFactPublisher(IMessageChannel<String, Object> messageChannel) {
        super();
        this.messageChannel = messageChannel;
    }

    public void publish(List<ReturnVectorFact> facts) {
        messageChannel.sendMessage("", facts);
    }
}

The way how you publish your data:

cubeReturnsFactPublisher.publish(Collections.singletonList(new PostVectorFact(...)));

How do I load data from a Hadoop Avro source into ActivePivot store?

2 Answers2