Handling High-Velocity Data Ingestion with TDengine

Question

I'm working on a project that involves a high influx of time-series data and considering using TDengine. Given its claim of supporting high-speed data ingestion, could anyone provide a more detailed explanation of how TDengine manages this? What is its actual write speed and how does it maintain performance when data is coming in at a rapid pace?

Answer 1

From the perspective of application program, you need to consider:

The data size of each single write, also known as batch size. Generally speaking, higher batch size generates better writing performance. However, once the batch size is over a specific value, you will not get any additional benefit anymore. When using SQL to write into TDengine, it's better to put as much as possible data in single SQL. The maximum SQL length supported by TDengine is 1,048,576 bytes, i.e. 1 MB.

The number of concurrent connections. Normally more connections can get better result. However, once the number of connections exceeds the processing ability of the server side, the performance may downgrade.

The distribution of data to be written across tables or sub-tables. Writing to single table in one batch is more efficient than writing to multiple tables in one batch.

Data Writing Protocol.
    Parameter binding mode is more efficient than SQL because it doesn't have the cost of parsing SQL.
    Writing to known existing tables is more efficient than writing to uncertain tables in automatic creating mode because the later needs to check whether the table exists or not before actually writing data into it.
    Writing in SQL is more efficient than writing in schemaless mode because schemaless writing creates table automatically and may alter table schema.

Application programs need to take care of the above factors and try to take advantage of them. The application program should write to single table in each write batch. The batch size needs to be tuned to a proper value on a specific system. The number of concurrent connections needs to be tuned to a proper value too to achieve the best writing throughput.