0

I am trying to setup a simple cluster for influxdb on AWS using EFS (elastic file storage which is similar to NFS) where 2 different influx processes will share database. I have 2 EC2 instances (named: EC2_1, EC2_2) running their own influxdb processes. Both EC2 instances are mounted with a common EFS directory called "efs" at /opt/efs. I modified influxdb.conf on both ec2 instances to have it point to common /opt/efs/influxdb/data directory for data, /opt/efs/influxdb/wal for wal,/opt/efs/influxdb/meta for meta directories.

What I am trying to achieve by this is when EC2_1 influx process writes something to database it can be immediately read by influx process on EC2_2.

To achieve this I am trying to find out a setting that can disable WAL or make writing to disk very frequent (once/sec) so that another process can read directly from disk. Read/Write performance or data loss is not a major concern for me right now.

I tried changing below setting hoping that it will make WAL flush frequent but it did not work the way I expected:

# CacheSnapshotMemorySize is the size at which the engine will
# snapshot the cache and write it to a TSM file, freeing up memory
# cache-snapshot-memory-size = 26214400
  cache-snapshot-memory-size = 2

# CacheSnapshotWriteColdDuration is the length of time at
# which the engine will snapshot the cache and write it to
# a new TSM file if the shard hasn't received writes or deletes
# cache-snapshot-write-cold-duration = "1h"
  cache-snapshot-write-cold-duration = "1s"

If EC2_1 writes something to influxdb, querying EC2_2 does not show that data unless influxdb process on EC2_2 is restarted and vice versa.

Ashish C
  • 77
  • 6
  • Can you add a bit more detail about your use case? – Michael Desa Aug 22 '16 at 16:44
  • Edited my original question @MichaelDesa – Ashish C Aug 23 '16 at 23:02
  • I see. I'd open a [feature request](https://github.com/influxdata/influxdb/issues/new) on the InfluxDB repo asking for this functionality. I feel like there used to be a way to control WAL flushes, but I'm not sure if thats still the case. Whats the reason you want to have two instances writing and reading from the same disk? – Michael Desa Aug 24 '16 at 15:13
  • Thanks, I will request it on github. I am trying to achieve a customized clustering where EFS snapshots are taken by lambda functions in AWS and I dont have to rely on influxdb replication. – Ashish C Aug 24 '16 at 19:30
  • I see. Running multiple InfluxDB instances on the same file system will lead to data corruption. – Michael Desa Aug 26 '16 at 16:38

1 Answers1

0

So there's not a way to disable the WAL entirely or to make it flush every second.

Michael Desa
  • 4,607
  • 24
  • 19