Configure Linux Distro on Google Dataflow Workers?

Question

My Beam/Dataflow pipeline involves some data processing which uses a Linux CLI tool (specifically, osmium-tool version >= 1.9.1) which is only available for some newer Ubuntu distros (cosmic or disco). I cannot find any documentation on the OS specs of Dataflow Workers, or if it is configurable.

Any help would be appreciated, thanks!

score 1 · Accepted Answer · answered Oct 10 '19 at 20:27

1

One of the goals of the Beam portability effort is to allow customizations such as this. Though it's still experimental at this point, you can follow the instructions at https://beam.apache.org/documentation/runtime/environments/ to completely configure the environment your user code runs in.

answered Oct 10 '19 at 20:27

robertwb

4,891
18
21

Thanks, that's exactly the documentation I was looking for! – manesioz Oct 10 '19 at 21:01
Do you have any example modified Dockerfiles that use a newer Ubuntu distro? – manesioz Oct 10 '19 at 21:08
No, I don't have an example modified Dockerfile. You might have to go up to the base images a bit to find where the OS chosen. – robertwb Oct 11 '19 at 22:25

Configure Linux Distro on Google Dataflow Workers?

1 Answers1