Cortex will fail to run analyzers #182

obikao · 2019-04-12T03:08:18Z

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	Debian 9
OS version (client)	Win 10
Cortex version / git hash	3.0.0-RC1 - `62e07ec`
Package Type	From source
Browser type & version	Chrome 73.0.3683.86

Problem Description

Cortex fails to run analyzers, returning the error "Binary fields do not support searching". This does not happen all the time, but once it starts, much like popping a can of Pringles, it just doesn't stop. I attempted using a fresh index of the database and this did not fix the issue either. I had to roll back to version 2. Running the analyzers from Cortex or TheHive made no difference.

Steps to Reproduce

Run analyzer
No idea, it kept happening around the 5th or 6th analyzer run

Complementary information

log entry from the hive

Apr 11 20:29:01 ogmpl01 thehive[27617]: [info] o.e.ErrorHandler - POST /api/connector/cortex/job returned 500
Apr 11 20:29:01 ogmpl01 thehive[27617]: connectors.cortex.services.CortexError: Cortex error on http://10.96.0.200:9001/api/analyzer/792b69494f2d7729ecd08514d348d84e/run (400)
Apr 11 20:29:01 ogmpl01 thehive[27617]: {"type":"Invalid search query","message":"Binary fields do not support searching"}
Apr 11 20:29:01 ogmpl01 thehive[27617]:         at connectors.cortex.services.CortexClient.$anonfun$request$3(CortexClient.scala:95)
Apr 11 20:29:01 ogmpl01 thehive[27617]:         at scala.util.Success.$anonfun$map$1(Try.scala:251)
Apr 11 20:29:01 ogmpl01 thehive[27617]:         at scala.util.Success.map(Try.scala:209)
Apr 11 20:29:01 ogmpl01 thehive[27617]:         at scala.concurrent.Future.$anonfun$map$1(Future.scala:288)
Apr 11 20:29:01 ogmpl01 thehive[27617]:         at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
Apr 11 20:29:01 ogmpl01 thehive[27617]:         at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
Apr 11 20:29:01 ogmpl01 thehive[27617]:         at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
Apr 11 20:29:01 ogmpl01 thehive[27617]:         at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
Apr 11 20:29:01 ogmpl01 thehive[27617]:         at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
Apr 11 20:29:01 ogmpl01 thehive[27617]:         at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)

The text was updated successfully, but these errors were encountered:

obikao · 2019-04-13T00:19:10Z

I was able to get Cortex to fail in this way again. Here are the logs:

2019-04-12 19:14:23,365 [INFO] from org.thp.cortex.services.JobSrv in application-akka.actor.default-dispatcher-15 - Looking for similar job in the last 10 minutes (worker=c657cb8859d5d79a6aaec4dd90144c36, dataType=ip, data=Left(8.8.8.8), tlp=2, parameters={})
2019-04-12 19:14:23,370 [INFO] from org.thp.cortex.services.ErrorHandler in application-akka.actor.default-dispatcher-14 - POST /api/analyzer/c657cb8859d5d79a6aaec4dd90144c36/run returned 400
org.elasticsearch.transport.RemoteTransportException: [KDXyIqC][10.96.0.2:9300][indices:data/read/search]
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:272)
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:130)
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:241)
        at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:107)
        at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:49)
        at org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:217)
        at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:171)
        at org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:49)
        at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:217)
        at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51)
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1077)
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1181)
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1159)
        at org.elasticsearch.transport.TransportService$7.onFailure(TransportService.java:665)
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:659)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.elasticsearch.index.query.QueryShardException: Binary fields do not support searching
        at org.elasticsearch.index.mapper.BinaryFieldMapper$BinaryFieldType.termQuery(BinaryFieldMapper.java:134)
        at org.elasticsearch.index.query.TermQueryBuilder.doToQuery(TermQueryBuilder.java:143)
        at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:96)
        at org.elasticsearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:444)
        at org.elasticsearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:418)
        at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:96)
        at org.elasticsearch.index.query.QueryShardContext.lambda$toQuery$1(QueryShardContext.java:313)
        at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:325)
        at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:312)
        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:617)
        at org.elasticsearch.search.SearchService.createContext(SearchService.java:485)
        at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:461)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:343)
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:340)
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:654)
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        ... 3 common frames omitted

obikao · 2019-04-13T01:07:58Z

Lastly, here is the query from the elastic logs:

[2019-04-12T19:56:43,885][DEBUG][o.e.a.s.TransportSearchAction] [KDXyIqC] [cortex_dev_3][3], node[KDXyIqCwQr66kTJ6gFT71Q], [P], s[STARTED], a[id=jy_99Bm1QX-6tfLxzp2zDg]: Failed to execute [SearchRequest{search
Type=QUERY_THEN_FETCH, indices=[cortex_dev_3], indicesOptions=IndicesOptions[id=38, ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_alisases_to
_multiple_indices=true, forbid_closed_indices=true], types=[job], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=5, batchedReduceSize=512, preFilterShardSize=128,
 source={
  "from" : 0,
  "size" : 1,
  "query" : {
    "bool" : {
      "must" : [
        {
          "term" : {
            "workerId" : {
              "value" : "c657cb8859d5d79a6aaec4dd90144c36",
              "boost" : 1.0
            }
          }
        },
        {
          "bool" : {
            "must_not" : [
              {
                "term" : {
                  "status" : {
                    "value" : "Failure",
                    "boost" : 1.0
                  }
                }
              }
            ],
            "disable_coord" : false,
            "adjust_pure_negative" : true,
            "boost" : 1.0
          }
        },
        {
          "bool" : {
            "must_not" : [
              {
                "term" : {
                  "status" : {
                    "value" : "Deleted",
                    "boost" : 1.0
                  }
                }
              }
            ],
            "disable_coord" : false,
            "adjust_pure_negative" : true,
            "boost" : 1.0
          }
        },
        {
          "range" : {
            "startDate" : {
              "from" : "1555116403885",
              "to" : null,
              "include_lower" : true,
              "include_upper" : true,
              "boost" : 1.0
            }
          }
        },
        {
          "term" : {
            "dataType" : {
              "value" : "ip",
              "boost" : 1.0
            }
          }
        },
        {
          "term" : {
            "tlp" : {
              "value" : 2,
              "boost" : 1.0
            }
          }
        },
        {
          "term" : {
            "data" : {
              "value" : "8.8.8.8",
              "boost" : 1.0
            }
          }
        },
        {
          "term" : {
            "parameters" : {
              "value" : "{}",
              "boost" : 1.0
            }
          }
        }
      ],
      "disable_coord" : false,
      "adjust_pure_negative" : true,
      "boost" : 1.0
    }
  },
  "version" : true,
  "stored_fields" : [
    "_source",
    "_routing",
    "_parent"
  ],
  "sort" : [
    {
      "createdAt" : {
        "order" : "desc"
      }
    },
    {
      "_uid" : {
        "order" : "desc"
      }
    }
  ]
}}] lastShard [true]

obikao · 2019-04-13T01:57:24Z

I've confirmed that this portion of the query is breaking it:

{
          "term" : {
            "data" : {
              "value" : "8.8.8.8",
              "boost" : 1.0
            }
          }
        }

I can reproduce the error consistently by reposting this query. What is weird is that it only fails on 2 of the 5 shards

{u'hits': {u'hits': [], u'total': 0, u'max_score': None}, u'_shards': {u'successful': 3, u'failed': 2, u'skipped': 0, u'total': 5, u'failures': [{u'node': u'KDXyIqCwQr66kTJ6gFT71Q', u'index': u'cortex_dev_3', u'reason': {u'index_uuid': u'XCOqWBZXTgaJ9drDa5Mdqg', u'index': u'cortex_dev_3', u'reason': u'Binary fields do not support searching', u'type': u'query_shard_exception'}, u'shard': 2}]}, u'took': 3, u'timed_out': False}

obikao · 2019-04-13T02:08:14Z

Lastly, just to ensure that there wasn't anything wrong with my instance of elasticsearch, I installed a brand new instance and tested the same query. It reacted in the same manner.

obikao · 2019-04-13T16:44:43Z

So, not being really well versed in scala, I think I ended up finding the issue. In 8e570b1 the data field was changed from F.stringFmt to F.rawFmt. The e4p model has this defined as a binary field. The only thing I can't figure out is why this only happens some of the time.

To-om · 2019-05-02T09:26:18Z

Sorry for the late answer. This problem is related to #178. The format of the data to analyze has changed in order to remove its size limitation. The data is stores using binary mapping, and become unsearchable.

This breaks job creation if cache is enable because Cortex searches old similar jobs (with the same data).

To solve this issue, I've added a cacheTag attribute which contains a hash of job information (data, dataType, tlp, analyzer/responder ID and parameters).

joseluratm · 2019-05-06T13:14:48Z

Hi!
In case it happens to someone, if after updating you get the error: Worker cannot be run
The solution is to disable the analyzers and re-enable them.

I mention it here, because I think it's related.
Thanks!

BrijJhala · 2020-09-02T20:37:06Z

hi @joseluratm some reason parameters is not picking up even though i have specified same as api doc.
{
"data":"8.8.8.8",
"dataType":"ip",
"tlp":0,
"message": "A message that can be accessed from the analyzer",
"parameters": {
"key1": "value1",
"key2": "value2"
}
}

obikao mentioned this issue Apr 13, 2019

Resolve "Binary fields do not support searching" errors #183

Closed

ag-michael mentioned this issue Apr 23, 2019

Responder not working after a Cortex upgrade #186

Closed

To-om self-assigned this May 2, 2019

To-om added the bug label May 2, 2019

To-om added this to the 3.0.0-RC2 milestone May 2, 2019

To-om added a commit that referenced this issue May 2, 2019

#182 Add cacheTag attribute

0942846

To-om closed this as completed May 2, 2019

ptobis mentioned this issue Oct 10, 2019

[Bug] Custom responder not working after upgrade to cortex 3 TheHive-Project/Cortex-Analyzers#542

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cortex will fail to run analyzers #182

Cortex will fail to run analyzers #182

obikao commented Apr 12, 2019 •

edited

Loading

obikao commented Apr 13, 2019

obikao commented Apr 13, 2019

obikao commented Apr 13, 2019

obikao commented Apr 13, 2019

obikao commented Apr 13, 2019

To-om commented May 2, 2019

joseluratm commented May 6, 2019

BrijJhala commented Sep 2, 2020

Cortex will fail to run analyzers #182

Cortex will fail to run analyzers #182

Comments

obikao commented Apr 12, 2019 • edited Loading

Request Type

Work Environment

Problem Description

Steps to Reproduce

Complementary information

obikao commented Apr 13, 2019

obikao commented Apr 13, 2019

obikao commented Apr 13, 2019

obikao commented Apr 13, 2019

obikao commented Apr 13, 2019

To-om commented May 2, 2019

joseluratm commented May 6, 2019

BrijJhala commented Sep 2, 2020

obikao commented Apr 12, 2019 •

edited

Loading