I'm experiencing this kind of problem for the first time. I'm running an application in a container with some hosted services (AspNet, Core C# 8.0, Docker container Linux). The first service do a big data ingestion in my database. I will explain the service: it just start, download from the web and download a 2gb file with a lot of JSON inside. Then it add one by one every JSON file deserialized in my database. It starts pretty well, but after 500.000 records starting to slow up and does not end the data ingestion. If I run
docker logs -f <container>
I get this message:
With docker stats, I see my container using all the resource possible. Thanks a lot for any assistance!
This is the HS:
private void DoWork(object state)
{
using (IServiceScope scope = _services.CreateScope())
{
executionCount++;
IVUManager _vuManager = scope.ServiceProvider.GetRequiredService<IVUManager>();
_vuManager.BulkLoaderVuMassive();
}
}
This is the code part managing the JSON file:
foreach (string filename in filesnumber)
{
using (StreamReader r = new StreamReader(filename))
{
//Parsin JSON
aInt++;
string jsonFull = r.ReadToEnd();
JsonDocument jsonDoc = JsonDocument.Parse(jsonFull);
JsonElement jsonParsed = jsonDoc.RootElement;
ASoVu newVU = new ASoVu();
newVU = _importStrategy.VU();
_repository.MassiveImportNVD(_context, newVUL)
if (aInt == 9999)
{
_repository.Save(_context);
}
}
}
This is the code part that make the insert inside the database:
public class VURepository : IVURepository { private readonly ILogger _logger;
public VURepository(ILogger<VURepository> logger)
{
_logger = logger;
}
public void MassiveImport(VMContext _context, ASoVu newVU)
{
_context.Vu.Add(newVU);
}