What are Output Errors in Elastic APM?

Question

When updating parameters for the APM agent, we noticed that Output Errors would increase upon restarts, and would see an increase even after letting the servers stabilize. Syslog output did not show any errors. Where are these errors coming from?

Answer 1

Inside the code for Libbeat, which is utilized by the Beat family of tools:

    //
    // Output event stats
    //
    batches *monitoring.Uint // total number of batches processed by output
    events  *monitoring.Uint // total number of events processed by output

    acked      *monitoring.Uint // total number of events ACKed by output
    failed     *monitoring.Uint // total number of events failed in output
    active     *monitoring.Uint // events sent and waiting for ACK/fail from output
    duplicates *monitoring.Uint // events sent and waiting for ACK/fail from output
    dropped    *monitoring.Uint // total number of invalid events dropped by the output
    tooMany    *monitoring.Uint // total number of too many requests replies from output

    //
    // Output network connection stats
    //
    writeBytes  *monitoring.Uint // total amount of bytes written by output
    writeErrors *monitoring.Uint // total number of errors on write

    readBytes  *monitoring.Uint // total amount of bytes read
    readErrors *monitoring.Uint // total number of errors while waiting for response on output
}

When you're querying in Elastic for results of Libbeat (see below), the Output Errors is derived from the measured delta between the initial timestamp's readErrors + writeErrors and the latest timestamp's readErrors + writeErrors. According to the code commentary then, Output Errors is the number of network packets experiencing errors.

The below example is utilizing apm-server as the beat type, but you can replace it to suit your needs. It won't give you why you're having network errors, but it'll split up the data so you can identify if it's a Read Error or a Write Error.

GET _search

    "query": {
      "bool": {
        "filter": [
          {
            "bool": {
              "should": [
                {
                  "term": {
                    "data_stream.dataset": "beats.stats"
                  }
                },
                {
                  "term": {
                    "metricset.name": "stats"
                  }
                },
                {
                  "term": {
                    "type": "beats_stats"
                  }
                }
              ]
            }
          },
          {
            "term": {
              "cluster_uuid": "CLUSTER_UUID"
            }
          },
          {
            "range": {
              "beats_stats.timestamp": {
                "format": "epoch_millis",
                "gte": 1665053615330,
                "lte": 1665054515330
              }
            }
          },
          {
            "bool": {
              "must": {
                "term": {
                  "beats_stats.beat.type": "apm-server"
                }
              }
            }
          }
        ]
      }
    },
    "collapse": {
      "field": "beats_stats.metrics.beat.info.ephemeral_id",
      "inner_hits": {
        "name": "earliest",
        "size": 1,
        "sort": [
          {
            "beats_stats.timestamp": {
              "order": "asc",
              "unmapped_type": "long"
            }
          },
          {
            "@timestamp": {
              "order": "asc",
              "unmapped_type": "long"
            }
          }
        ]
      }
    },
    "sort": [
      {
        "beats_stats.beat.uuid": {
          "order": "asc",
          "unmapped_type": "long"
        }
      },
      {
        "timestamp": {
          "order": "desc",
          "unmapped_type": "long"
        }
      }
    ]
}