How can I write multiple updates in one BulkAll method in ElasticSearch NEST 7.13.2

Question

Using ElasticSearch NEST .Net package 7.13.2 in Visual Studio 2019

For a list of products I am currently updating existing documents in my product index by using the following code:

var productIndex = "productindex";
foreach (var product in products)
{
  productClassIdScript = $"ctx._source.productClassId = \"{product.ProductClassId}\"; ";
  elasticClient.Update<productIndex, object>(product.Id,
               q => q.Script(s => s.Source(productClassIdScript).Lang("painless")));
}

I do this for more than 10000 products and it takes about 2 hours. I know I can insert new documents with the Bulk API. Can I do the updates with the BulkAll method ?

Something like this:

var bulkAllObservable = elasticClient.BulkAll<Product>(myBulkAllRequest)
                        .Wait(TimeSpan.FromMinutes(15), next =>
                        {
                            // do something e.g. write number of pages to console
                        });

How should I construct myBulkAllRequest ?

Any help is much appreciated.

score 0 · Accepted Answer · answered Jul 12 '21 at 11:31

Bulk index will drastically reduce your indexing / updating time, so this is a good way to go.

You can still use BulkAll for updates, in case elasticsearch already has document with provided id, the document will be updated.

var bulk = elasticClient.BulkAll<EsDocument>(new List<EsDocument> { new EsDocument { Id = "1", Name = "1" }}, d => d);
using var subscribe = bulk.Subscribe(new BulkAllObserver(onNext: response => Console.WriteLine("inserted")));
bulk.Wait(TimeSpan.FromMinutes(1), response => Console.WriteLine("Bulk insert done"));

var bulk2 = elasticClient.BulkAll<EsDocument>(new List<EsDocument> { new EsDocument { Id = "1", Name = "1_updated"  }}, d => d);
using var subscribe2 = bulk2.Subscribe(new BulkAllObserver(onNext: response => Console.WriteLine("inserted")));
bulk2.Wait(TimeSpan.FromMinutes(1), response => Console.WriteLine("Bulk insert done"));

First BulkAll will insert document with Id "1" second, will update document with Id "1".

Index state after the first bulkd

and after second one

How can I write multiple updates in one BulkAll method in ElasticSearch NEST 7.13.2

1 Answers1