0

I have a requirement that since the Redis-Stream structure holds a specified length of data that I wish to transfer to influxDB in bulk, but during the transfer process the data inserted in bulk will not be in the same order as the original data

An example of the data generation code is shown below

try:
    rs = redis.Redis(host="localhost", port=6379, db=0)
except Exception as e:
    print(e)
count = 0
while(True):
    Ts = int(round(time.time() * 1000))
    count = count + 1
    Count = count

    data = {
        "Ts": Ts,
        "Count":Count
    }
    rs.xadd("simulation", data, maxlen=1000, approximate=False)
    time.sleep(0.1)

Examples of the data read and save code are shown below

def parse(datas):
    ts,data = datas
    w_json = {
    "measurement": 'test1',
    "tags": {
        'Ts': data[b'Ts'],
        },
    "fields": {
        "Count":data[b'Count'].decode('utf-8'),
        'Ts': data[b'Ts']
        }
    }
    return w_json


try:
    rs = redis.Redis(host="localhost", port=6379, db=0)
except Exception as e:
    print(e)
client = InfluxDBClient(host="localhost", port=8086,database='test')

results = rs.xrange("simulation",count=1000)

influxdb_data = list(map(parse,results))
client.write_points(influxdb_data,batch_size=500)
print('insert success')

When I look at the data, you see that the Count data order is not output in that order

> select Count,Ts from test1 limit 30;
name: sensor1
time                Count Ts
----                ----- --
1594003971238615143 1     1594003959534
1594003971238615143 10    1594003960446
1594003971238615143 100   1594003969537
1594003971238615143 101   1594003969638
1594003971238615143 102   1594003969739
1594003971238615143 103   1594003969840
1594003971238615143 104   1594003969941
1594003971238615143 105   1594003970042
1594003971238615143 106   1594003970143
1594003971238615143 107   1594003970244
1594003971238615143 108   1594003970345
1594003971238615143 109   1594003970446
1594003971238615143 11    1594003960547
1594003971238615143 110   1594003970547
1594003971238615143 111   1594003970648
1594003971238615143 112   1594003970749
1594003971238615143 12    1594003960648
1594003971238615143 13    1594003960749
1594003971238615143 14    1594003960850
1594003971238615143 15    1594003960951
1594003971238615143 16    1594003961053
1594003971238615143 17    1594003961154
1594003971238615143 18    1594003961255
1594003971238615143 19    1594003961356
1594003971238615143 2     1594003959638
1594003971238615143 20    1594003961457
1594003971238615143 21    1594003961558
1594003971238615143 22    1594003961659
1594003971238615143 23    1594003961760
1594003971238615143 24    1594003961861

Is there some way to implement sequential batch inserts?

I would appreciate it if you could tell me how to solve it?

moluzhui
  • 1,003
  • 14
  • 34
  • side question, why do you need to move the data to influx? – Guy Korland Jul 06 '20 at 07:56
  • Since the redis-Stream data set can only store a certain length of data, and the data is continuously generated, all of them need to be saved. Since the data is related to the time series, subsequent processing of the transferred data will be done for the influxDB. The fields contain a long array that may not be appropriate for influxDB, but I am not aware of anything else that might be more appropriate – moluzhui Jul 06 '20 at 11:33
  • Did you check RedisTimeSeries? – Guy Korland Jul 06 '20 at 12:02
  • I've tried to print the results of reading Redis-Stream, and it seems to work. It now appears that when inserted in the influxdb, it is sorted lexicographically by Count and not by the number size of Count itself – moluzhui Jul 06 '20 at 12:15

0 Answers0