Gemfire Composite key (pojo) as gemfire key

Question

I have following type of data into my database and want to store into Gemfire.

+------------------------------+--------------------------------+----------------------------------------+----------------------------------+----------------------------------+------------------------------------+-----------------------------------+------------------------------+------------------------+----------------------------+-------------------------------------+--+
| A                            | B                              | C                                      | D                                | E                                | F                                  | G                                 | H                            | I                      | J                          | date                                |
+------------------------------+--------------------------------+----------------------------------------+----------------------------------+----------------------------------+------------------------------------+-----------------------------------+------------------------------+------------------------+----------------------------+-------------------------------------+--+
| VK                           | XXXXXXXXXXXXXXXX               | YYY                                    | VBGFD                            | 9                                | 193.83                             | 9                                 | T                            |                        | False                      | 2017-08-25                          |
| VK                           | XXXXXXXXXXXXXXXX               | YYY                                    | VBGFD                            | 25.22                            | 193.83                             | 9                                 | T                            |                        | False                      | 2017-08-25                          |
| VK                           | XXXXXXXXXXXXXXXX               | YYY                                    | VBGFD                            | 9                                | 112.23                             | 9                                 | T                            |                        | False                      | 2017-08-25                          |
| VK                           | XXXXXXXXXXXXXXXX               | YYY                                    | VBGFD                            | NULL                             | 89.98                              | NULL                              | T                            |                        | False                      | 2017-08-25                          |
+------------------------------+--------------------------------+----------------------------------------+----------------------------------+----------------------------------+------------------------------------+-----------------------------------+------------------------------+------------------------+----------------------------+-------------------------------------+--+

4 rows selected (2.248 seconds)

1) You can see data is duplicate and any one column with different values would be consider as unique row.

2) I cannot use any column as Gemfire key as will override values (since data is duplicate )

I have following two possible solutions:

Approach 1

I can use composite key (pojo) as gemfire key but issue is that i need to use same pojo as key and value (as per data )

package  data;
Class KEY{
private String A;
private String B ;
private String C ;
private String D ;
private String E ;
private String F;
private String G ;
private String H ;
private String I ;
private String J ;
private String  date ;
}

package  data
Class Value{
private String A;
private String B ;
private String C ;
private String D ;
private String E ;
private String F;
private String G ;
private String H ;
private String I ;
private String J ;
private String  date ;
}


put --key=(KEY) --value=(Value) --region=/region1 --key-class=data.KEY --value-class=data.Value

Issue with this approach is that , it will double my data size in region and my loader start throwing out of memory error.

Approach 2

Create auto increment id into database and use this id as key but then i need to create index ,increment id would not be part of my sql.

My Query

query --query='select a,b,c,d,e,f,g,h,i,j,date from / myRegion where a=VK and b=XXXXXXXXXXXXXXXX and c=YYY and F=193.83'

I am wonder if anyone come across same scenario ?

Update : I have decided to use auto increment id (Long) as key and pojo as value. Planing to create Range index on column.

Swapnil · Answer 1 · 2017-10-04T22:27:39.750

1

What kind of access patterns you have for the data? If you are only going to use OQL or Functions to query the data, I would suggest to generate a random key and let the value be your data.

I don't see any value in keeping all data in your key. If you were to do a lookup (using region.get(key)) you will need all the data to create the key, at which point your get() will be of no value.

(update)

Your key should have the minimum number of fields that you always have available while doing a lookup. Since in your query you are using a, b, c and d, I would suggest that your key be composed of a, b, c, d and your value contain the remaining fields. At this point you can just do a region.get(key), no need to write a query.

edited Oct 04 '17 at 22:27

answered Oct 03 '17 at 19:09

Swapnil

1,191
7
8

I am exposing rest micro-service ,i was thinking to use auto increment key as gemfire key but issue is that i have billions of records into gemfire region and i need to create index on key . – vaquar khan Oct 03 '17 at 19:13
Hey Khan, as mentiond by @Swapnil, you should think about how the data is going to be accessed/read (you didn't answer this part). In case you didn't get it, he was asking will the data reading be like querying on one of the column, for example "select * from /region1 where A='VK' " and get a list and do something after. Or what will it be like? – Xiawei Zhang Oct 04 '17 at 07:38
Thanks Zhang , i have updated question with required details – vaquar khan Oct 04 '17 at 18:56
@Swapnil really appreciated your quick response and help, i have added one more row now you can see my query should return two row on given criteria however if key on this four column it will override values in genmfire and store only second record , so query will return only one result which is not correct . – vaquar khan Oct 05 '17 at 14:33

Gemfire Composite key (pojo) as gemfire key

1 Answers1