Scio is a Scala API for Google Cloud Dataflow and Apache Beam inspired by Spark and Scalding.
Questions tagged [spotify-scio]
80 questions
0
votes
1 answer
Scala macros: string literal assigned to a variable does not match
I am using a macro annotation from Spotify's Scio library. I would like to define a variable of String type and annotate like this:
val schemaString = """schema here"""
@BigQueryType.fromSchema(outputString) class BigQuery
This does not compile,…
jamborta
- 5,130
- 6
- 35
- 55
0
votes
0 answers
Scio: How can I combine messages sent from cloud pub sub using Apache Beam?
I am currently using Apache Beam's scala wrapper library, scio.
The thing you want to do is combine different types of messages sent from CloudPubSub based on ID.
The message A is sent every second, and the message B is sent once every three…
SakaT
- 1
- 2
0
votes
2 answers
SBT Verify Error caused by multiple protobuf 2/3 dependencies in spite of shading
I am struggling with Verify Errors with this below sample project using Scio/Bigtable/HBase. The dependency tree requires protobuf version (2.5, 2.6.1, 3.0, 3.1) and seems to default to 3.2. I used the shading component of sbt-assembly, not sure I…
ogen
- 802
- 2
- 7
- 23
0
votes
1 answer
Transform HBase Scan to RowFilter
I'm using scio from spotify for my Dataflow jobs.
In last scio version, new bigtable java api is used (com.google.bigtable.v2)
Now scio bigtable entry point required "RowFilter" to filter instead of Hbase "Scan". Is there a simple way to transform…
harscoet
- 340
- 1
- 11
0
votes
2 answers
How to save SCollection elements to different BigQuery tables?
I need to save SCollection elements into different hourly BigQuery tables on the basis of their timestamps. I tried the following methods -
Group elements by (TableName, Iterable[TableRow]) and then save each Iterable[TableRow] to their respective…
Kakaji
- 1,421
- 2
- 15
- 23