0

In Elassandra, the cassandra data size is 8GB but the elasticsearch.data size is 83GB. We have data-in flow of 5 msgs/sec and below is the query used to create tables and indexes:

Table creation:

CREATE TABLE IF NOT EXISTS x.abc (
        internal_tag text,
        generated_at timestamp,
        collected_at timestamp,
        data_type text,
        metadata text,
        recorded_at timestamp,
        value text,
        PRIMARY KEY(internal_tag, generated_at)
)
WITH CLUSTERING ORDER BY(generated_at ASC)
AND bloom_filter_fp_chance = 0.01
AND caching = { 'keys': 'ALL', 'rows_per_partition': 'NONE' }
AND comment = ''
AND compaction = { 'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4' }
AND compression = { 'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor' }
AND crc_check_chance = 1.0
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';

Index creation :

curl -XPUT -H 'Content-Type: application/json' 'http://10.0.0.01:9200/x_abc_index' -d '{
        "settings": {
        "keyspace": "x"
    },
        "mappings":{
                "abc" : {
                        "discover":".*"
                }
        }
}'

Please suggest any solution to overcome the data size issue. Thanks

Zeal Shah
  • 1
  • 2
    You have to create a mappings for what you need. Focus on your text field and ask yourself if you cant index them as keyword or if you dont have to just store them (without indexation). It s difficult to help you as we doesnot know witch data you will search. Also take a loot to your cluster, how many shards + replica are configured on elastic? – LeBigCat Feb 20 '20 at 09:17

1 Answers1

2

I suggested by LeBigCat, you can reduce the size of elasticsearch indices by reducing the number of indexed fields in your mapping or choose the right mapping.

vroyer
  • 31
  • 2