How to do group by key on custom logic in cloud data flow

Question

I am trying to achieve the Groupby key based on custom object in cloud data flow pipe line.

 public static void main(String[] args) {
   Pipeline pipeline = Pipeline.create(PipelineOptionsFactory.create());
   List<KV<Student,StudentValues>> studentList = new ArrayList<>();
   studentList.add(KV.of(new Student("pawan", 10,"govt"),
                         new StudentValues("V1", 123,"govt")));
   studentList.add(KV.of(new Student("pawan", 13223,"word"),
                         new StudentValues("V2", 456,"govt")));

   PCollection<KV<Student,StudentValues>> pc = 
     pipeline.apply(Create.of(studentList));
   PCollection<KV<Student, Iterable<StudentValues>>> groupedWords =
     pc.apply(GroupByKey.<Student,StudentValues>create());
}

I just wanted to groupBy both the PCollection record based on the Student object.

@DefaultCoder(AvroCoder.class)
static class Student /*implements Serializable*/{
  public Student(){}
  public Student(String n, Integer i, String sc){
    name = n;
    id = i;
    school = sc;
  }
  public String name;
  public Integer id;
  public String school;

  @Override
  public boolean equals(Object obj) {
    System.out.println("obj = "+obj);
    System.out.println("this = "+this);

    Student stObj= (Student)obj;
    if (stObj.Name== this.Name){
      return true;
    } else{
      return false;
    }
  }
}

I have overridden the equals method of my custom class, but each time i am getting same instance of Student object to compare inside equals method. Ideally it sholud compare first student key with second one.

Whats wrong i am doing here.

score 0 · Answer 1 · answered Apr 05 '17 at 18:50

Why do you think you are doing anything wrong? The keys of each element are serialized (using the AvroCoder you specified) and the GroupByKey can group all of the elements with the same serialized representation together. After that it doesn't need to compare the students to make sure that the values with the same key have been grouped together.

How to do group by key on custom logic in cloud data flow

1 Answers1