Indefinite retry and elastic search index pollution

[Ostico]

When a log event fails on elasticsearch ( even if it accept the event ) the message is re-sent multiple times ( a single event can be logged on ESearch 320 times ):

elasticsearch indexing error {
  type: 'mapper_parsing_exception',
  reason: "failed to parse field [subtitle.time_activity] of type [long] in document with id 'rbfgjoABJah9DI4rZNHN'. Preview of field's value: '9.223372036854776E31'",
  caused_by: {
    type: 'input_coercion_exception',
    reason: 'Numeric value (9.223372036854776e+31) out of range of long (-9223372036854775808 - 9223372036854775807)\n' +
      ' at [Source: (ByteArrayInputStream); line: 1, column: 752]'
  }
} {
  '@timestamp': '2022-05-04T11:40:20.874Z',
  severity: 'debug',
  mesage: 'Send aSync update',
  socketId: '56moxoTSOe4qLnFzAABT',
  clientAddress: 'xxxxxxxxxxxxx',
  messageType: 'update',
  requestId: '67116286-12c7-4e93-83e3-90e3aa843294',
  user_id: 'a0f4ae5f-0ec0-4b6d-80b9-47af97be41c5',
  subtitle: {
    text: [
      'text',
      'text'
    ],
    settings: {},
    project_id: '78c9882d-0ab3-407e-9510-608fd4e73769',
    target_id: '0a47eef2-2a02-412b-b5f4-dc643c55877b',
    status: 'draft',
    start: 2.211097204839383,
    end: 5.882352941176471,
    time_activity: 9.223372036854776e+31,
    hash_id: '3e934c7c-baf0-44bd-9bd3-56438871410e',
    qaErrors: { errors: [] }
  }
}

[vanthome]

Can you try to reduce the retryLimit option? Its default is 400 and after that it should five up.

[eye1]

Hello,

We encountered the same issue. I think this is even worse when you have many logs in the buffer. Then all of them got inserted 400? times and it went bananas real quick. If we turn off retries ultimately? What about buffering can it also be turned off?

[kuldarim]

@eye1

There are two things with buffering wich you can control.

// turn on or of the buffering at all
buffering: false,
// max number of messages in buffer
bufferLimit: 20,