how do I scan and scroll with elasticsearch-js and elasticsearch 5.0

Question

I was using elasticsearch-scrolltoend as a plugin previously, after upgrading to 5.0 the plugin doesn't seem to work. How can I scan and scroll a large dataset with elasticsearch 5.0?

I also received an error when attempting to use the implementation in the elasticsearch-js docs:

json { "error": { "root_cause": [ { "type": "illegal_argument_exception", "reason": "Failed to parse request body" } ], "type": "illegal_argument_exception", "reason": "Failed to parse request body", "caused_by": { "type": "json_parse_exception", "reason": "Unrecognized token 'DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAAHKFnV6a2NabEh4VDZLQmdzUzY0Y2tpd0EAAAAAAAAByxZ1emtjWmxIeFQ2S0Jnc1M2NGNraXdBAAAAAAAAAcwWdXprY1psSHhUNktCZ3NTNjRja2l3QQAAAAAAAAHOFnV6a2NabEh4VDZLQmdzUzY0Y2tpd0EAAAAAAAABzRZ1emtjWmxIeFQ2S0Jnc1M2NGNraXdB': was expecting ('true', 'false' or 'null')\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@67ba4d99; line: 1, column: 457]" } }, "status": 400 }

score 0 · Answer 1 · answered Apr 18 '17 at 17:49

The unrecognized token response error is a bug that was fixed in 13.0.0-rc1 -- see this issue for reference

As for the implementation, heres how I did it without the elasticsearch-scrolltoend plugin

  // methods in some ES6 class
  ...

  fetchStuff(min, max = min) {

    const body = new Bodybuilder();

    body.size(3);
    body.filter('range', 'timestamp', {
      gte: min,
      lte: max
    });

    return this.elasticsearch.search({
      index: 'my-alias',
      type: 'my-doc',
      body: body.build('v2'), // have not upgraded to newer bodybuilder package yet
      scroll: '15s'
    })
    .then((res) => this._scrollToEnd(res, []))
    .then((events) => {

      // sort by timestamp then _id
      events = _.sortBy(events, ['_.source.timestamp', '_id']);

      return events;

    });

  }

  _scrollToEnd(res, events) {
    return Promise.resolve().then(() => {

      events = events.concat(_.get(res, 'hits.hits', []));

      if (res.hits.total > events.length) {
        return this.elasticsearch.scroll({
          scrollId: res._scroll_id,
          scroll: '15s'
        })
        .then((res) => this._scrollToEnd(res, events));
      }

      return events;

    });
  }

how do I scan and scroll with elasticsearch-js and elasticsearch 5.0

1 Answers1