2

I have been reading up on Point in time API and wanted to implement it using NEST in my .net application. However, when reading that article (.net application hyperlink), I saw the Fluent DSL example as shown below. Is there a way to find that ID without having to go to kibana on the console and doing an query search to get the id to then later place that id inside of "a-point-in-time-id"? or does "a-point-in-time-id" do that for you like is that mapped to the ID?

s => s
.PointInTime("a-point-in-time-id", p => p
.KeepAlive("1m"))

I know in the kibana cli console if you do:

POST /app-logs*/_pit?keep_alive=5m it will give you a PIT (point in time) id. How do you go about retrieving that in NEST?

and when reading up on search_after and attempting to implement it using the search after usage for the .net client using the Fluent DSL Example. I noticed that they had they word "Project" but it does not say what "Project is in the example. What would that be exactly?

s => s
.Sort(srt => srt
    .Descending(p => p.NumberOfCommits)
    .Descending(p => p.Name)
)
.SearchAfter(
    Project.First.NumberOfCommits,
    Project.First.Name
)

Here I attempted to implement .PointInTime() and .Sort() and .SearchAfter() but got stuck.

var response = await _elasticClient.SearchAsync<Source>(s => s
                  .Size(3000) // must see about this
                  .Source(src => src.Includes(i => i
                                    .Fields(f => f.timestamp,
                                            fields => fields.messageTemplate,
                                            fields => fields.message)))
                  .Index("app-logs*"
                  .Query(q => +q
                      .DateRange(dr => dr
                          .Field("@timestamp")
                              .GreaterThanOrEquals("2021-06-12T16:39:32.727-05:00")
                              .LessThanOrEquals(DateTime.Now))))
                  .PointInTime("", p => p
                                   .KeepAlive("5m"))
                  .Sort(srt => srt
                               .Ascending(p => p.timestamp))
                  .SearchAfter()

I know when you are using PIT ID you do not need the index in the search, but in the hyperlink example it does not show how you would go about implementing that. So just a bit lost on how to do so. Any pointers/guidance/tutorials would be great!

But just trying to see how I can do this in NEST but if you are saying its apart of the XPACK, I would understand.

MarkCo
  • 810
  • 2
  • 12
  • 29
  • I am not able to make this work via HTTP API with my ES 7.10 Although I think that the idea is to a) first do the PIT request, fetch the PIT ID from it (looks like [here's an example](https://www.elastic.co/guide/en/elasticsearch/client/net-api/master/point-in-time-usage.html)), and then b) use it in the subsequent queries. – Nikolay Vasiliev Jul 14 '21 at 10:02
  • How would you go about requesting to get the PIT ID in nest though? @NikolayVasiliev – MarkCo Jul 14 '21 at 15:15
  • As soon as you've got the ID I think `.PointInTime("",...)` is supposed to do it. Btw are you sure you're strictly required to use "point in time"? Maybe the data changing in the index is not a problem (maybe the data does not even change), then you don't need PIT. – Nikolay Vasiliev Jul 15 '21 at 07:41
  • Sorry I meant like how do I get the ID? I know you have to go to kibana and run the command `POST /customer-simulation-es-app-logs*/_pit?keep_alive=5m` in order to get an ID. How do you do that in NEST not having to go to Kibana? I mean is there another way to retrieve all the logs even if its more than 10,000 results, besides using PIT with search after? Because I know search api only can handle up to 10,000 results – MarkCo Jul 15 '21 at 14:51
  • 1
    You can use [Scroll API](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/paginate-search-results.html#scroll-search-results) to get all results. Moreover, I figured why I could not use `_pit` endpoint - because [it's part of XPack](https://stackoverflow.com/questions/65114162/how-to-create-elasticsearch-point-in-time-pit#comment118070392_65114818) which is not available in open source distros. – Nikolay Vasiliev Jul 15 '21 at 16:56
  • But the issue with the scroll api is the following: "We no longer recommend using the scroll API for deep pagination. If you need to preserve the index state while paging through more than 10,000 hits, use the search_after parameter with a point in time (PIT)." So if I need something to retrieve all the hits regardless if it is more than 10,000 hits I would have to use search after right which would require a PIT ID? So in NEST you cannot get the `_pit id` because its part of XPACK which you would have to pay for? Or is that for the CLI? – MarkCo Jul 15 '21 at 17:52
  • When I do the POST command I actually get a PIT ID back, please check the original post, I will show @NikolayVasiliev – MarkCo Jul 15 '21 at 17:54

1 Answers1

6

I was able to combine PIT and SearchAfter with the code sample below:

var pit = string.IsNullOrEmpty(pitId) ? client.OpenPointInTime(new OpenPointInTimeDescriptor().KeepAlive(ServerDetail.TTL).Index(Indices.Index(Name))) : null;

            var request = new SearchRequest(Name)
            {
                SearchAfter = string.IsNullOrEmpty(pitId) ? null : new string[] { last },
                Size = ServerDetail.PageSize,
                PointInTime = new PointInTime(pit == null ? pitId : pit.Id)
            };
 List<FieldSort> sorts = new List<FieldSort>();
            foreach (var item in sortBy)
            {
                sorts.Add(new FieldSort() { Field = item.Field, Order = item.Direction == SortDirection.Asc ? SortOrder.Ascending : SortOrder.Descending });
            }
            request.Sort = sorts.ToArray();

Your SearchAfter value should be values from your sort fields for the last object of your previous search result.

For the 1st search pitId is null so a new one is created, for subsequent requests, pitId is passed.

Hope that works for you?

Abimbola Omotoso
  • 76
  • 1
  • I follow what you did, this is great! Thank you the documentation was a bit confusing on how I could request the PIT id. Thanks again! – MarkCo Jul 19 '21 at 13:30