Can not list all cortex jobs through API #133

s-shubh · 2018-09-24T19:32:06Z

Work Environment

Question	Answer
OS version (server)	CentOS 7
OS version (client)	CentOS
Cortex version / git hash	2.x, hash of the commit
Package Type	Binary,

Problem Description

I want to list all the jobs of an organization but could not do so with using API. It shows me below error logs in /var/log/messages :

_Sep 24 15:17:51 TImachine cortex: [#033[37minfo#033[0m] o.t.c.s.ErrorHandler - GET /api/job/_search?range=all returned 500
Sep 24 15:17:51 TImachine cortex: java.util.NoSuchElementException: head of empty stream
Sep 24 15:17:51 TImachine cortex: at akka.stream.scaladsl.Sink$.$anonfun$head$3(Sink.scala:162)
Sep 24 15:17:51 TImachine cortex: at scala.Option.getOrElse(Option.scala:121)
Sep 24 15:17:51 TImachine cortex: at akka.stream.scaladsl.Sink$.$anonfun$head$2(Sink.scala:162)
Sep 24 15:17:51 TImachine cortex: at scala.util.Success.$anonfun$map$1(Try.scala:251)
Sep 24 15:17:51 TImachine cortex: at scala.util.Success.map(Try.scala:209)
Sep 24 15:17:51 TImachine cortex: at scala.concurrent.Future.$anonfun$map$1(Future.scala:289)
Sep 24 15:17:51 TImachine cortex: at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
Sep 24 15:17:51 TImachine cortex: at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
Sep 24 15:17:51 TImachine cortex: at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
Sep 24 15:17:51 TImachine cortex: at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)

Find Cortex Application Logs:

2018-09-24 15:27:47,905 [INFO] from org.thp.cortex.services.ErrorHandler in application-akka.actor.default-dispatcher-15898 - GET /api/job/_search?range=all returned 500
java.util.NoSuchElementException: head of empty stream
        at akka.stream.scaladsl.Sink$.$anonfun$head$3(Sink.scala:162)
        at scala.Option.getOrElse(Option.scala:121)
        at akka.stream.scaladsl.Sink$.$anonfun$head$2(Sink.scala:162)
        at scala.util.Success.$anonfun$map$1(Try.scala:251)
        at scala.util.Success.map(Try.scala:209)
        at scala.concurrent.Future.$anonfun$map$1(Future.scala:289)
        at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
        at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
        at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:73)
        at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:78)
        at akka.dispatch.BatchingExecutor.execute(BatchingExecutor.scala:120)
        at akka.dispatch.BatchingExecutor.execute$(BatchingExecutor.scala:114)
        at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:77)
        at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
        at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
        at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
        at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
        at scala.concurrent.Promise.trySuccess(Promise.scala:90)
        at scala.concurrent.Promise.trySuccess$(Promise.scala:90)
        at scala.concurrent.impl.Promise$DefaultPromise.trySuccess(Promise.scala:183)
        at akka.stream.impl.HeadOptionStage$$anon$3.onUpstreamFinish(Sinks.scala:251)
        at akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:504)
        at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:378)
        at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:585)
        at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:469)
        at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:560)
        at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:742)
        at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$shortCircuitBatch(ActorGraphInterpreter.scala:732)
        at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:758)
        at akka.actor.Actor.aroundReceive(Actor.scala:517)
        at akka.actor.Actor.aroundReceive$(Actor.scala:515)
        at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:667)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:590)
        at akka.actor.ActorCell.invoke(ActorCell.scala:559)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)_

Steps to Reproduce

Fetch API key of any ORGadmin user of required organization from cortex.
Use the below Request to list all the jobs:
curl -H 'Authorization: Bearer API_KEY' 'https://CORTEX_APP_URL:9001/api/job/_search?range=all'
You will find the below error on console.
{"type":"java.util.NoSuchElementException","message":"head of empty stream"}

Kingly Help..!

The text was updated successfully, but these errors were encountered:

To-om · 2018-10-08T08:57:21Z

I can't reproduce your problem. Do you have any job in your Cortex instance ?

s-shubh · 2018-10-08T09:43:54Z

I have about 6k jobs jn cortex instance and I want to delete them at once but somehow I want to save some jobs. So whole going to list these jobs through API, I got an errror. SHUBHAM

…

On Mon 8 Oct, 2018, 2:27 PM To-om, ***@***.***> wrote: I can't reproduce your problem. Do you have any job in your Cortex instance ? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#133 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AYMDjpBQLkxdVWCdbtSVQgtFG5c4KkA9ks5uixN2gaJpZM4W3UEd> .

To-om · 2018-10-08T09:45:39Z

With another range (rang=0-20 for example), does it work ?

s-shubh · 2018-10-08T10:04:54Z

Nop, It is not working in range.

…

On Mon 8 Oct, 2018, 3:15 PM To-om, ***@***.***> wrote: With another range (rang=0-20 for example), does it work ? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#133 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AYMDjmzfmYglh0ON5JrNhxaSdcmrVLa6ks5uix7GgaJpZM4W3UEd> .

To-om · 2018-10-08T12:03:39Z

what is the role of the user associated with the API key ? Is it in the same organization of the jobs ?

s-shubh · 2018-10-08T12:10:31Z

The user belongs to same organization and it has all three admin rights. Shubham

…

On Mon 8 Oct, 2018, 5:33 PM To-om, ***@***.***> wrote: what is the role of the user associated with the API key ? Is it in the same organization of the jobs ? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#133 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AYMDjmfDNSfsjVOOhi7yV7a9NOF2tMKfks5uiz8dgaJpZM4W3UEd> .

To-om · 2018-10-08T13:11:57Z

You probably use Cortex 2.0.4. Please update your Cortex and check if the error still occurs.

s-shubh · 2018-10-08T13:18:38Z

I have already used cortex 2.0.4. please let me know whether any new version needs to be updated. Shubham

…

On Mon 8 Oct, 2018, 6:42 PM To-om, ***@***.***> wrote: You probably use Cortex 2.0.4. Please update your Cortex and check if the error still occurs. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#133 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AYMDjjixN2v5xsVW9PStntP1aVBE-Mxmks5ui08ggaJpZM4W3UEd> .

To-om · 2018-10-08T13:35:58Z

The latest version of Cortex is 2.1.0 (2.1.1 will be released soon).

s-shubh · 2018-10-08T13:44:23Z

I have just installed latest verison and let me check these. I will get bach to you once any problem appears. Shubh

…

On Mon 8 Oct, 2018, 7:06 PM To-om, ***@***.***> wrote: The latest version of Cortex is 2.1.0 (2.1.1 will be released soon). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#133 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AYMDjgIJS3kohXulXs4n1jj29YGbsUkpks5ui1TCgaJpZM4W3UEd> .

s-shubh · 2018-10-08T13:50:18Z

[root@ti~]# curl -H 'Authorization: Bearer s7FTpcrb6bdmhrUa5mNjT2NK+3bDoo4H' 'http://192.168.0.170:9001/api/job/_search?range=all'
{"type":"NotFoundError","message":"job _search not found"}

Again same error occured.

To-om · 2018-10-08T13:51:49Z

You have this error because you do a GET not a POST. Add -XPOST parameter to your command.

nadouani · 2018-10-08T13:52:25Z

This sounds like a doc error in fact

To-om · 2018-10-08T13:53:31Z

Right !

nadouani · 2018-10-08T13:54:00Z

Fixed https://github.com/TheHive-Project/CortexDocs/blob/master/api/api-guide.md#list-and-search

s-shubh · 2018-10-08T13:54:19Z

As I have soun d knowledge of API thing. Let me check this out.

s-shubh · 2018-10-08T13:56:57Z

Yea, It gets worked. A last single question, Can I delete all jobs at once ???

s-shubh · 2018-10-08T14:05:10Z

If I get all the jobs gets deleted at once then it would be best for me.

In addition to this, is there any settings or API to delete these job entries from database ?

To-om · 2018-10-08T14:05:41Z

No you can't delete all job with only one API call. You can list all jobs, get there ID and delete them one by one. The script should not be hard to write.
Note that jobs are not really deleted in the database. Deleting a job is updating its status.

s-shubh · 2018-10-08T14:07:36Z

Okay, script can be worked. But can't we have any way to delete them from database ?

To-om · 2018-10-08T14:15:21Z

Working directly on the database is possible but not recommended as consistency of the data can be broken.
If it is an important feature for you, you can open an issue or a PR for that.

s-shubh · 2018-10-08T14:19:27Z

Actually, I used it only for threat intelligence. I also subscribed many analyzers. But thing is that, If it gets deleted from database then we can also minimize system resource usage.

I will work on that and it it gets required then I will probably raise the request.

But thank you for your great support and suggestions.

I appreciate this. Have a great time ahead..!

To-om self-assigned this Oct 8, 2018

To-om closed this as completed Oct 8, 2018

To-om added the question label Oct 9, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can not list all cortex jobs through API #133

Can not list all cortex jobs through API #133

s-shubh commented Sep 24, 2018 •

edited by To-om

Loading

To-om commented Oct 8, 2018

s-shubh commented Oct 8, 2018 via email

To-om commented Oct 8, 2018

s-shubh commented Oct 8, 2018 via email

To-om commented Oct 8, 2018

s-shubh commented Oct 8, 2018 via email

To-om commented Oct 8, 2018

s-shubh commented Oct 8, 2018 via email

To-om commented Oct 8, 2018

s-shubh commented Oct 8, 2018 via email

s-shubh commented Oct 8, 2018

To-om commented Oct 8, 2018

nadouani commented Oct 8, 2018

To-om commented Oct 8, 2018

nadouani commented Oct 8, 2018

s-shubh commented Oct 8, 2018

s-shubh commented Oct 8, 2018

s-shubh commented Oct 8, 2018

To-om commented Oct 8, 2018

s-shubh commented Oct 8, 2018

To-om commented Oct 8, 2018

s-shubh commented Oct 8, 2018

Can not list all cortex jobs through API #133

Can not list all cortex jobs through API #133

Comments

s-shubh commented Sep 24, 2018 • edited by To-om Loading

Work Environment

Problem Description

I want to list all the jobs of an organization but could not do so with using API. It shows me below error logs in /var/log/messages :

Find Cortex Application Logs:

Steps to Reproduce

To-om commented Oct 8, 2018

s-shubh commented Oct 8, 2018 via email

To-om commented Oct 8, 2018

s-shubh commented Oct 8, 2018 via email

To-om commented Oct 8, 2018

s-shubh commented Oct 8, 2018 via email

To-om commented Oct 8, 2018

s-shubh commented Oct 8, 2018 via email

To-om commented Oct 8, 2018

s-shubh commented Oct 8, 2018 via email

s-shubh commented Oct 8, 2018

To-om commented Oct 8, 2018

nadouani commented Oct 8, 2018

To-om commented Oct 8, 2018

nadouani commented Oct 8, 2018

s-shubh commented Oct 8, 2018

s-shubh commented Oct 8, 2018

s-shubh commented Oct 8, 2018

To-om commented Oct 8, 2018

s-shubh commented Oct 8, 2018

To-om commented Oct 8, 2018

s-shubh commented Oct 8, 2018

s-shubh commented Sep 24, 2018 •

edited by To-om

Loading