Azure MSSQL Server slow ingestion

Question

I am trying to ingest 9,000,000 rows in an elastic pool database with 6 Vcore. Data ingestion using python (pyodbc).

Since data is large, I am ingesting the data in chunks.

I am getting weird behaviour after the 9th chunk of the ingestion. Process disappear and randomly appears after an hour.

Is there any solution for this?

can you give a screenshot or any concrete example of this "weird behavior"? — physincubus, Aug 11 '22 at 15:18

score 0 · Answer 1 · edited Sep 19 '22 at 01:09

0

My suggestion use non-durable memory-optimized table to speed up data ingestion, while managing the In-Memory OLTP storage footprint by offloading historical data to a disk-based Columnstore table. Use a job to regularly batch-offload data to a disk-based Columnstore table.

With that adjustment you can obtain 1.4 million sustained rows per second during ingestion.

edited Sep 19 '22 at 01:09

ChrisGPT was on strike

127,765
105
273
257

answered Aug 09 '22 at 17:10

Alberto Morillo

13,893
2
24
30

Why do you **bold all your links?** Please don't. Generally speaking, linking is enough to draw users' attention to things. – ChrisGPT was on strike Sep 19 '22 at 01:09

Azure MSSQL Server slow ingestion

1 Answers1