0

A multi-day test run with Spanner has shown an massively increasing CPU usage with growing table size. I'm trying to figure out what could be the reason. Summary of the usage:

  • Small size rows (400 byte per row)
  • Average around 500 reads (with 0 result) and 500 inserts per second
  • Running on 4 nodes
  • Growing data size (only inserts, no updates at the moment)
  • Composite primary key {a, b, c}
  • Reads by secondary index {a, b} (with 0 result)

How can I debug this?

Spanner nicely shows execution plans for manual SQL queries in the console.
The client is in Golang, is there a way to print the actual SQL on client side to execute it manually? Or can I somewhere see the latest executed queries in the spanner console? So far I couldn't find a way to find out what's going on.

What could be causing this CPU increase? It seems to me that there is some processing of the whole table, such as a table scan. I can hardly imagine that reading or updating the secondary index creates this increase.

I appreciate any ideas and hints.

Chrisport
  • 2,936
  • 3
  • 15
  • 19
  • Yes, I would recommend to start with verifying that queries behave as expected (e.g. not doing a full scan). You could print the actual sql string using Statement.SQL, see https://github.com/GoogleCloudPlatform/google-cloud-go/blob/master/spanner/statement.go#L42 We are working on better debugging tools. – Mairbek Khadikov Oct 11 '17 at 19:00
  • Thanks for your quick response. Since we are using transactions, there is no Statement involved; I will try to rewrite it. As improvement it would be really useful to see the latest executed SQL on spanner's server side in order to understand what is going on and what we are doing wrong. – Chrisport Oct 12 '17 at 10:29

0 Answers0