I know that presently order by is not there in BeamSQL is there any work around for it

Question

This is present

PCollection<BeamRecord> rec = rec_out.apply(BeamSql.query(
  "SELECT bnk_name,state_name,val from PCOLLECTION order by val desc limit 2"));

But I need

PCollection<BeamRecord> rec = rec_out.apply(BeamSql.query(
  "SELECT bnk_name,state_name,val from PCOLLECTION order by val desc "));

What will you do with the data after it is output? What is the purpose of the order? — Kenn Knowles, Nov 28 '18 at 15:30

score 0 · Answer 1 · answered Nov 28 '18 at 15:29

0

This actually wouldn't help. PCollections are unordered in Beam, so you cannot apply ordering to the whole output.

Beam does not have ordering because it is usually not necessary, often not possible. Most problems where ordering is used have another solution that does not require global sorting.

If you describe more about what you will do with the output of the SQL query, I will update my answer with advice.

If you cannot share more, then you have the option of using Beam's sorter extension outside of the SQL query, but again it is not really for global sorting, but per-key order after a shuffle.

answered Nov 28 '18 at 15:29

Kenn Knowles

5,838
18
22

I will store my output Pcollection to a csv file or any database . – TEJASWAKUMAR Dec 01 '18 at 13:19
That makes sense - if your data is small enough to store in a database then it is a great choice. – Kenn Knowles Dec 01 '18 at 17:22

I know that presently order by is not there in BeamSQL is there any work around for it

1 Answers1