Stack Overflow
Stack Overflow

Questions tagged [beam]

This tag should be used for questions about the BEAM, the Erlang virtual machine.

The BEAM (Bogdan/Björn's Erlang Abstract Machine) is the Erlang virtual machine. Besides , there are also other languages that can target the BEAM virtual machine, such as Joxa, , , and others.

Disambiguation

  • Use for questions related to Apache Beam, an SDK for batch and stream processing.
  • Use for questions related to Android Beam, the NFC peer-to-peer mode NDEF message exchange mechanism in Android.
  • Use for questions related to the heuristic search algorithm beam search.
106 questions
0
votes
0 answers

Scio (Apache Beam) BigQuery struct (Record) type issue

I`m facing an issue during ingestion process of a BigQuery table which contains RECORD (STRUCT) fields. The Apache Beam (I`m using Scio with Scala) load process extracts a JSON file with a flat values for the RECORD fields. As shown…
Bruno Faleiros
  • 1
0
votes
1 answer

Why are erlang lib changes not being incorporated into my project?

I'm using eclipse 3.6.2 with erlang 5.8.1.1 on a fairly large project that we're not ready to move to a more modern version of the language, so I'm stuck with a bug in eprof: string_bp_mfa([{Mfa, {Count, Time}}|Mfas], Tus, {MfaW, CountW, PercW,…
Sniggerfardimungus
  • 11,583
  • 10
  • 52
  • 97
0
votes
0 answers

Convert a Beam PCollection to HashMap after Combine.globally()

I want to convert a small PCollection into a List and construct a HashMap. I managed to Combine the elements of the PCollection globally into an Iterable, but the Iterable is inside a PCollection. How do I extract the Iterable (which now contains…
AlpsToronto
  • 25
  • 4
0
votes
1 answer

CassandraIO + Apache Beam

I have been attempting to find information on how to utilize Apache Beam with Cassandra in Python, but I have not been successful in my search. I am curious if the use of cassandraio library is supported for Python? If anyone could provide any…
Udemytur
  • 79
  • 1
  • 5
0
votes
3 answers

Curve fitting of the laser beam radius W(z) versus propagation distance z

I am taking 5 measurements of the laser beam radius at various distances between 1-200 cm. Without knowing where the beam waist Wo is, I am trying to use the curve_fit() function to find the opt value of Wo and Zo. The output I am getting seems…
Kelia
  • 1
0
votes
1 answer

Apache Beam for feature creation

I'm trying to use Apache Beam for feature creation. I've looked around SO and the Beam Dataframe API docs but I haven't seen it address the issue I have. From what I looked at from the docs, each row is a PCollection and processed using a Transform.…
Minh
  • 2,180
  • 5
  • 23
  • 50
0
votes
1 answer

Transform PCollection to custom class

My goal is to read a file from GCS and write it to Cassandra. New to Apache Beam/Dataflow, I could find most of the hand on build with Python. Unfortunately CassandraIO is only Java native with Beam. I used the word count example as a template and…
Ragnar
  • 2,550
  • 6
  • 36
  • 70
0
votes
0 answers

How to change window trigger after generating data in ParDo in streaming dataflow job?

I have a streaming dataflow job that reads new messages for PubSub and triggers after n messages. Windowing options used by me are presented in the code below. Next, I group items and generate some new items (in two branches) in ParDo based on…
Sienki
  • 81
  • 1
  • 5
0
votes
1 answer

merge collection with different window strategy

we have 3 different data sources which eventually we need to do some kind of inner join between them we create all pcollections with group by key pcollectionA - Implemented using state (the data is not changed) pcollectionB - windowed for 5h. if…
Violet
  • 1
  • 1
0
votes
1 answer

Reading Avro files in GCS as PCollection

Our Dataflow job, written in Python, listens from a Pubsub subscription. The messages are strings of GCS file path of Avro files (gs://bucket/file-timestamp.avro). The avro files are not of uniform schema, but Beam Python's avroio.ReadAllFromAvro is…
oikonomiyaki
  • 7,691
  • 15
  • 62
  • 101
0
votes
0 answers

Error reporting while running a java program, which is about a beam scenario

I am writing to enquire some problems about what I encountered while learning BEAM,which is about transportation simulation .First, the application of Java 1.8, Gradle 7.5.1, and GIT-IFS 2.3.4 had been installed and verified according to this…
qiqi
  • 1
0
votes
0 answers

'Cannot set destination table in jobs with DML statements'

This is my beam job I can run only select * statement but I cannot run delete statement so, let me know how to resolve this issue Thanks in advance pipeline_options = PipelineOptions(options) qry = ''' delete from project-id.dataset-id.table where…
Shankar Patil
  • 1
0
votes
1 answer

Apache Beam: Unable to serialize DoFnWithExecutionInformation because of PipelineOptions not serializable

Getting the following error when running my Beam pipeline: java.lang.IllegalArgumentException: unable to serialize DoFnWithExecutionInformation{doFn=WriteWithAppendToGoFile$CreateTrailerDoFn@57b711b6, mainOutputTag=Tag, sideInputMapping={},…
Priyab
  • 11
  • 3
0
votes
0 answers

Perform an UPSERT with apache beam MongoDB in Python

I'm trying to perform an "upsert" on a collection in MongoDB using python & apache beam. My collection (called people) contains this sample data : { "asset" : "41299e41-4c8f-47cb-83a7-4e718b437c16", "appId" : "my-app", "udid" :…
Amine Gharbi
  • 1
  • 2
0
votes
1 answer

elixir - error loading module ranch_server - recompile this module with an 20 compiler

I'm trying to run an existing elixir project that works on my other computer, on a new computer (both are mac). When I run iex -S mix I get: iex -S mix Erlang/OTP 20 [erts-9.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [hipe]…
BeniaminoBaggins
  • 11,202
  • 41
  • 152
  • 287
1 2 3 4 5
6
7 8