Responder not working after a Cortex upgrade

[ag-michael]

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	Redhat
OS version (client)	7.6
Cortex version / git hash	3.0.0-RC1
Package Type	RPM

Problem Description

After upgrading cortex to 3.0.0-RC1 ,analyzers and responders stopped working.
#182 exists for the analyzer issue, however, I'm also running into a Responder issue.
Responders are failing with no error output at all.

Steps to Reproduce

1. Upgrade to Cortex
2. Attempt to run the FalconCustomIOC responder (or any other responder)

Complementary information

2019-04-23 19:40:26,284 [INFO] from org.thp.cortex.services.JobSrv in application-akka.actor.default-dispatcher-11 - Job cache is disabled
2019-04-23 19:40:27,505 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-5 - Register new listener for job AWpLt2afXww3rV7yqcne (Actor[akka://application/temp/$he])
2019-04-23 19:40:27,614 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in application-akka.actor.default-dispatcher-5 - Execute /opt/Cortex-Analyzers/responders/FalconCustomIOC/FalconCustomIOC.py in /opt/Cortex-Analyzers/responders, timeout is none
2019-04-23 19:40:27,615 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-9 - Job AWpLt2afXww3rV7yqcne has be updated (JsDefined("InProgress"))
2019-04-23 19:40:27,823 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-816 -   Job AWpLt2afXww3rV7yqcne: Traceback (most recent call last):
2019-04-23 19:40:27,823 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-816 -   Job AWpLt2afXww3rV7yqcne:   File "/opt/Cortex-Analyzers/responders/FalconCustomIOC/FalconCustomIOC.py", line 56, in <module>
2019-04-23 19:40:27,823 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-816 -   Job AWpLt2afXww3rV7yqcne:     FalconCustomIOC().run()
2019-04-23 19:40:27,823 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-816 -   Job AWpLt2afXww3rV7yqcne:   File "/opt/Cortex-Analyzers/responders/FalconCustomIOC/FalconCustomIOC.py", line 14, in __init__
2019-04-23 19:40:27,823 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-816 -   Job AWpLt2afXww3rV7yqcne:     Responder.__init__(self)
2019-04-23 19:40:27,823 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-816 -   Job AWpLt2afXww3rV7yqcne:   File "/usr/lib/python2.7/site-packages/cortexutils/responder.py", line 11, in __init__
2019-04-23 19:40:27,823 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-816 -   Job AWpLt2afXww3rV7yqcne:     Worker.__init__(self)
2019-04-23 19:40:27,823 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-816 -   Job AWpLt2afXww3rV7yqcne:   File "/usr/lib/python2.7/site-packages/cortexutils/worker.py", line 20, in __init__
2019-04-23 19:40:27,823 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-816 -   Job AWpLt2afXww3rV7yqcne:     self._input = json.load(self.fpinput)
2019-04-23 19:40:27,823 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-816 -   Job AWpLt2afXww3rV7yqcne:   File "/usr/lib64/python2.7/json/__init__.py", line 290, in load
2019-04-23 19:40:27,823 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-816 -   Job AWpLt2afXww3rV7yqcne:     **kw)
2019-04-23 19:40:27,823 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-816 -   Job AWpLt2afXww3rV7yqcne:   File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
2019-04-23 19:40:27,823 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-816 -   Job AWpLt2afXww3rV7yqcne:     return _default_decoder.decode(s)
2019-04-23 19:40:27,823 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-816 -   Job AWpLt2afXww3rV7yqcne:   File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
2019-04-23 19:40:27,823 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-816 -   Job AWpLt2afXww3rV7yqcne:     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2019-04-23 19:40:27,823 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-816 -   Job AWpLt2afXww3rV7yqcne:   File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
2019-04-23 19:40:27,824 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-816 -   Job AWpLt2afXww3rV7yqcne:     raise ValueError("No JSON object could be decoded")
2019-04-23 19:40:27,824 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-816 -   Job AWpLt2afXww3rV7yqcne: ValueError: No JSON object could be decoded
2019-04-23 19:40:28,624 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-12 - Job AWpLt2afXww3rV7yqcne has be updated (JsDefined("Failure"))
2019-04-23 19:40:28,624 [INFO] from org.thp.cortex.services.JobSrv in application-akka.actor.default-dispatcher-10 - Job AWpLt2afXww3rV7yqcne has finished with status Failure

[To-om]

Cortex 3.x requires cortexutils 2.0. Have you updated it ?

[ag-michael]

@To-om Yes sir, I have cortexutils 2.0 available under all python interpreters.

[To-om]

I don't know why but your responder uses an old version of cortexutils. According to your stacktrace, line 20 of worker.py is self._input = json.load(self.fpinput) which doesn't match the version 2.0 but a version 1.3.

[ag-michael]

@To-om I believe that was my issue. Sorry for the confusion, I checked 'python' and 'python3', apparently on my install they both mean the same now. my responder was using python2.7 which had the outdated cortexutils version.

Closing this out since analyzers and responders are working expected after my upgrade to the latest cortex.