4

I'm trying to benchmark our Aerospike cluster of 3 nodes with build-in Java benchmark tool.

When I check AMC (Aerospike Management Console) I see there are no successful reads. Benchmark is reporting that everything is ok and that's the problem.

2016-12-16 15:34:21.674 write(tps=14851 timeouts=0 errors=0) read(tps=15030 timeouts=0 errors=0) total(tps=29881 timeouts=0 errors=0)
2016-12-16 15:34:22.674 write(tps=21160 timeouts=0 errors=0) read(tps=21284 timeouts=0 errors=0) total(tps=42444 timeouts=0 errors=0)
2016-12-16 15:34:23.675 write(tps=22868 timeouts=0 errors=0) read(tps=22312 timeouts=0 errors=0) total(tps=45180 timeouts=0 errors=0)
2016-12-16 15:34:24.676 write(tps=22443 timeouts=0 errors=0) read(tps=22795 timeouts=0 errors=0) total(tps=45238 timeouts=0 errors=0)

Do you have any idea why AMC do not show those read requests as successful since benchmark reports reading with no error or timeout?

AMC view, showing 0 success reads

My benchmark configuration (modified example 3) is bellow.

./run_benchmarks -h aero1.db.test.env -p 3000 -n namespace -k 1000000000000000 -S 1 -o S:50 -w RU,50 -z 1 -async -asyncMaxCommands 300 -asyncSelectorThreads 8 -e 1 -T 500
Northys
  • 1,305
  • 3
  • 16
  • 32

1 Answers1

2

That's a lot of keys you have there (1 Million Billion keys, or 10^15)... and doing an RU workload...

Your read failures are in fact 'not found'. I am not a statistician, but I would expect the chances of randomly reading a key that has been already inserted from such a large sample to be very very low for a very very long time. You can check the stats and verify that all those reads are indeed 'not found'. Those are technically reported separately from the error in the benchmark tool, which is, I give you that, misleading at best since you wouldn't notice.

I am also not sure what type of RAM/Storage you have on a 3 node cluster to handle that many records, as small as they are.

For proper benchmark testing, you should first load all your keys with an insert only workload (-w I) and then trigger the read update workload.

Meher
  • 2,804
  • 9
  • 7
  • Cluster is memory only. – Northys Dec 16 '16 at 16:24
  • 1
    Memory only would make it even more expensive to store all of this, the index only requirements would be around 58 PetaBytes (64 bytes per record), so close to 20PetaByte of RAM per node for the index. By the way, Aerospike currently has a limitation (not for too long hopefully) where a maximum of 2^32 (~4Billion) keys per node per namespace can be stored. – Meher Dec 16 '16 at 16:41
  • So you are talking about the huge number I put inside the benchmark? It was only random (high) number number to make it run almost forever. I think it has nothing in common why benchmark has data from aerospike but aerospike reports 0 successful reads. – Northys Dec 16 '16 at 22:48
  • 2
    Ah, yes, sorry if that was not clear. The -k option is for the number of records to use for the benchmark. A workload of -w RU (read update) will always run until terminated, so you shouldn't have to worry about how long it would run. You probably typically want to first do an insert only workload -w I (uppercase i) with a good number of records for your use case (-k option) and then, after the load is complete, you can run the read update benchmark with the same -k option to make sure all keys will be found on read. – Meher Dec 17 '16 at 00:55