Writing to memcache from within a streaming dataflow pipeline

Question

Is it possible to write to memcache from a streaming data flow pipeline? or do I need to write to a pubsub and create another compute engine or app engine?

score 1 · Answer 1 · answered Jun 20 '16 at 17:39

1

Yes, the Dataflow workers can communicate with any external services that you need; they are just VMs with no special restrictions or permissions.

If you are just writing out data to memcache, the Sink API will likely be useful

answered Jun 20 '16 at 17:39

danielm

3,000
10
15

Thanks but could you be more specific. Can you give me an example Sink call that also shows how I authenticate ? – Melissa Stockman Jun 20 '16 at 19:42
If you can go into your use case a bit more, I can give an example of how to use the Sink API to implement it. With regards to authentication, for communicating with any Google-provided service you can use the credentials in options.getGcpCredential() – danielm Jun 20 '16 at 19:47
From the dataflow custom sink page it says that you can only write bounded data but I need to do it from a streaming pipeline."You can use the extensible API to create custom sources that read either bounded (batch) or unbounded (streaming) data, and to create sinks that write bounded data only." – Melissa Stockman Jun 20 '16 at 19:52
I have a streaming dataflow pipeline that is reading a pubsub and writing the data to a BigQuery table. I want to also be able to rollup that data and write the rollups to memcache for quick current access from clients and then also write those rollups to datastore. I would like to do this all from the same pipeline. – Melissa Stockman Jun 20 '16 at 20:01
Ah, for now in streaming, you can just make direct calls to the memcache API from a DoFn – danielm Jun 20 '16 at 21:08
Yes but the memcache is not residing in the dataflow vm (or do you need to ssh in and install it?). I don't want to create an app engine on it's own just to hold the cache. – Melissa Stockman Jun 20 '16 at 22:10

score 0 · Accepted Answer · edited May 23 '17 at 12:33

0

For redis I created DoFn with redis client. It is possible to do some tricks if you need batch writing. For example: link

edited May 23 '17 at 12:33

Community

1
1

answered Jul 29 '16 at 14:10

al32

109
3

Thanks, I've switched to redis now also. I wasn't batch writing but this link is insightful. – Melissa Stockman Jul 31 '16 at 00:53

Writing to memcache from within a streaming dataflow pipeline

2 Answers2