Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature taxonomy #1668

Merged
merged 59 commits into from
Jan 21, 2021
Merged
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
d1935bc
Added taxonomy to database's schema
rriclet Nov 11, 2020
5f9d4b9
WIP adding taxonomy routes & mapping
rriclet Nov 12, 2020
95a425d
WIP Continued mapping for taxonomies
rriclet Nov 12, 2020
08c0a5c
WIP create taxonomiy / get works
rriclet Nov 16, 2020
a6cfdb3
WIP Custom taxonomy when new organisation is created
rriclet Nov 16, 2020
80ffdd2
WIP Added taxonomy activate / deactivate
rriclet Nov 16, 2020
279f8af
Fixed taxonomy values parsing
rriclet Nov 16, 2020
2e25d59
Idempotent TheHive schema
rriclet Nov 17, 2020
49e9e0f
Checked if taxonomy namespace is present before creating
rriclet Nov 17, 2020
cda163d
Correct output format for taxonomies
rriclet Nov 17, 2020
eb91a5b
Query for taxonomies
rriclet Nov 17, 2020
858ba48
Basic zip import
rriclet Nov 18, 2020
366ef7b
Handled zip import errors
rriclet Nov 18, 2020
2c7e887
Review changes
rriclet Nov 19, 2020
1146010
Added (de)activation & deletion
rriclet Nov 24, 2020
5e88dba
Used correct Scalligraph commit
rriclet Nov 24, 2020
bc1b507
Edit drone.yml
rriclet Nov 24, 2020
e4d43cb
Fixed schema erros & (de)activation
rriclet Nov 24, 2020
a6d7dfc
Fixed schema
rriclet Nov 24, 2020
09f0dc7
Fixed unit test for taxonomy
rriclet Nov 25, 2020
6f8c4fe
Fixed user permission test
rriclet Nov 25, 2020
471b894
Review changes
rriclet Dec 8, 2020
673ded1
Added taxonomy to database's schema
rriclet Nov 11, 2020
fb2c911
WIP adding taxonomy routes & mapping
rriclet Nov 12, 2020
2327ffa
WIP Continued mapping for taxonomies
rriclet Nov 12, 2020
9960e4d
WIP create taxonomiy / get works
rriclet Nov 16, 2020
f285f2b
WIP Custom taxonomy when new organisation is created
rriclet Nov 16, 2020
8a4d73e
WIP Added taxonomy activate / deactivate
rriclet Nov 16, 2020
a092f15
Fixed taxonomy values parsing
rriclet Nov 16, 2020
8863acb
Idempotent TheHive schema
rriclet Nov 17, 2020
2be85d8
Checked if taxonomy namespace is present before creating
rriclet Nov 17, 2020
54218aa
Correct output format for taxonomies
rriclet Nov 17, 2020
db3a944
Query for taxonomies
rriclet Nov 17, 2020
c5e1385
Basic zip import
rriclet Nov 18, 2020
1208235
Handled zip import errors
rriclet Nov 18, 2020
de6f2df
Review changes
rriclet Nov 19, 2020
a0a4ca4
Added (de)activation & deletion
rriclet Nov 24, 2020
c53c619
Used correct Scalligraph commit
rriclet Nov 24, 2020
c87f090
Edit drone.yml
rriclet Nov 24, 2020
a934a82
Fixed schema erros & (de)activation
rriclet Nov 24, 2020
f2248d3
Fixed schema
rriclet Nov 24, 2020
6234bc0
Fixed unit test for taxonomy
rriclet Nov 25, 2020
872a7d5
Fixed user permission test
rriclet Nov 25, 2020
91861c5
Review changes
rriclet Dec 8, 2020
4fe3111
Correct Scalligraph version
rriclet Dec 8, 2020
a4f0aaf
resolved conflicts
rriclet Dec 8, 2020
65189f5
Fixed build
rriclet Dec 8, 2020
0bfdf13
Zip import works with folders
rriclet Dec 8, 2020
0d1a8b7
#1668 WIP: Add taxonomies list admin page
nadouani Dec 9, 2020
eecd191
Changed freetags namespace & fixed visibily issue for listing taxonom…
rriclet Dec 10, 2020
aac858a
Removed freetags from admin list taxonomies
rriclet Dec 10, 2020
c1a6f7d
Zip import can fail and still continue importing
rriclet Dec 10, 2020
213d94c
Changed field names for zip import
rriclet Dec 10, 2020
4574830
Fixed broken tests
rriclet Dec 10, 2020
04afbe6
#1668 WIP: Update taxonomies list admin page: add filtering, paginati…
nadouani Dec 10, 2020
e9ba4aa
#1668 WIP: add taonomy view dialog to list tags
nadouani Dec 11, 2020
745fcea
#1668 WIP: allow filtering by taxonomy status
nadouani Dec 11, 2020
eae7bbc
WIP Changed tag colour property type - Cortex tests failing
rriclet Dec 11, 2020
5365d7e
Merge branch 'develop-th4' into feature-taxonomy
To-om Jan 5, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .drone.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ steps:
- name: submodules
image: alpine/git
commands:
- git submodule update --recursive --init --remote
- git submodule update --recursive --init

# Restore cache of downloaded dependencies
- name: restore-cache
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ object InputCustomFieldValue {
}
case _ => Good(Nil)
}

implicit val writes: Writes[Seq[InputCustomFieldValue]] = Writes[Seq[InputCustomFieldValue]] { icfv =>
val fields = icfv.map {
case InputCustomFieldValue(name, Some(s: String), _) => name -> JsString(s)
Expand Down
15 changes: 15 additions & 0 deletions dto/src/main/scala/org/thp/thehive/dto/v1/Tag.scala
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
package org.thp.thehive.dto.v1

import play.api.libs.json.{Json, OFormat}

case class OutputTag(
namespace: String,
predicate: String,
value: Option[String],
description: Option[String],
colour: Int
)

object OutputTag {
implicit val format: OFormat[OutputTag] = Json.format[OutputTag]
}
73 changes: 73 additions & 0 deletions dto/src/main/scala/org/thp/thehive/dto/v1/Taxonomy.scala
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
package org.thp.thehive.dto.v1

import java.util.Date

import play.api.libs.json.{Json, OFormat}

/*
Format based on :
https://tools.ietf.org/id/draft-dulaunoy-misp-taxonomy-format-04.html
*/

case class InputTaxonomy(
namespace: String,
description: String,
version: Int,
`type`: Option[Seq[String]],
exclusive: Option[Boolean],
predicates: Seq[InputPredicate],
values: Option[Seq[InputValue]]
)

case class InputPredicate(
value: String,
expanded: Option[String],
exclusive: Option[Boolean],
description: Option[String]
)

case class InputValue(
predicate: String,
entry: Seq[InputEntry]
)

case class InputEntry(
value: String,
expanded: Option[String],
colour: Option[String],
description: Option[String],
numerical_value: Option[Int]
)

object InputTaxonomy {
implicit val format: OFormat[InputTaxonomy] = Json.format[InputTaxonomy]
}

object InputPredicate {
implicit val format: OFormat[InputPredicate] = Json.format[InputPredicate]
}

object InputValue {
implicit val format: OFormat[InputValue] = Json.format[InputValue]
}

object InputEntry {
implicit val format: OFormat[InputEntry] = Json.format[InputEntry]
}

case class OutputTaxonomy(
_id: String,
_type: String,
_createdBy: String,
_updatedBy: Option[String] = None,
_createdAt: Date,
_updatedAt: Option[Date] = None,
namespace: String,
description: String,
version: Int,
tags: Seq[OutputTag]
)

object OutputTaxonomy {
implicit val format: OFormat[OutputTaxonomy] = Json.format[OutputTaxonomy]
}
23 changes: 15 additions & 8 deletions thehive/app/org/thp/thehive/controllers/v0/CaseCtrl.scala
Original file line number Diff line number Diff line change
Expand Up @@ -195,13 +195,17 @@ class PublicCase @Inject() (
with CaseRenderer {
override val entityName: String = "case"
override val initialQuery: Query =
Query.init[Traversal.V[Case]]("listCase", (graph, authContext) => organisationSrv.get(authContext.organisation)(graph).cases)
override val getQuery: ParamQuery[EntityIdOrName] = Query.initWithParam[EntityIdOrName, Traversal.V[Case]](
"getCase",
FieldsParser[EntityIdOrName],
(idOrName, graph, authContext) => caseSrv.get(idOrName)(graph).visible(authContext)
)
override val pageQuery: ParamQuery[OutputParam] = Query.withParam[OutputParam, Traversal.V[Case], IteratorOutput](
Query.init[Traversal.V[Case]]("listCase", (graph, authContext) =>
organisationSrv.get(authContext.organisation)(graph).cases
)
override val getQuery: ParamQuery[EntityIdOrName] =
Query.initWithParam[EntityIdOrName, Traversal.V[Case]](
"getCase",
FieldsParser[EntityIdOrName],
(idOrName, graph, authContext) => caseSrv.get(idOrName)(graph).visible(authContext)
)
override val pageQuery: ParamQuery[OutputParam] =
Query.withParam[OutputParam, Traversal.V[Case], IteratorOutput](
"page",
FieldsParser[OutputParam],
{
Expand All @@ -215,7 +219,10 @@ class PublicCase @Inject() (
}
}
)
override val outputQuery: Query = Query.outputWithContext[RichCase, Traversal.V[Case]]((caseSteps, authContext) => caseSteps.richCase(authContext))
override val outputQuery:
Query = Query.outputWithContext[RichCase, Traversal.V[Case]]((caseSteps, authContext) =>
caseSteps.richCase(authContext)
)
override val extraQueries: Seq[ParamQuery[_]] = Seq(
Query[Traversal.V[Case], Traversal.V[Observable]]("observables", (caseSteps, authContext) => caseSteps.observables(authContext)),
Query[Traversal.V[Case], Traversal.V[Task]]("tasks", (caseSteps, authContext) => caseSteps.tasks(authContext))
Expand Down
25 changes: 25 additions & 0 deletions thehive/app/org/thp/thehive/controllers/v1/Conversion.scala
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import java.util.Date
import io.scalaland.chimney.dsl._
import org.thp.scalligraph.controllers.Renderer
import org.thp.scalligraph.models.Entity
import org.thp.thehive.dto.v1.{InputTaxonomy, OutputTaxonomy}
import org.thp.thehive.dto.v1._
import org.thp.thehive.models._
import play.api.libs.json.{JsObject, JsValue, Json}
Expand Down Expand Up @@ -251,6 +252,29 @@ object Conversion {
.transform
}

implicit class InputTaxonomyOps(inputTaxonomy: InputTaxonomy) {

def toTaxonomy: Taxonomy =
inputTaxonomy
.into[Taxonomy]
.transform
}

implicit val taxonomyOutput: Renderer.Aux[RichTaxonomy, OutputTaxonomy] =
Renderer.toJson[RichTaxonomy, OutputTaxonomy](
_.into[OutputTaxonomy]
.withFieldComputed(_._id, _._id.toString)
.withFieldConst(_._type, "Taxonomy")
.withFieldComputed(_.tags, _.tags.map(_.toOutput))
.transform
)

implicit val tagOutput: Renderer.Aux[Tag, OutputTag] =
Renderer.toJson[Tag, OutputTag](
_.into[OutputTag]
.transform
)

implicit class InputUserOps(inputUser: InputUser) {

def toUser: User =
Expand Down Expand Up @@ -335,6 +359,7 @@ object Conversion {
.withFieldComputed(_.tlp, _.tlp.getOrElse(2))
.transform
}

implicit val observableOutput: Renderer.Aux[RichObservable, OutputObservable] = Renderer.toJson[RichObservable, OutputObservable](richObservable =>
richObservable
.into[OutputObservable]
Expand Down
8 changes: 8 additions & 0 deletions thehive/app/org/thp/thehive/controllers/v1/Properties.scala
Original file line number Diff line number Diff line change
Expand Up @@ -375,4 +375,12 @@ class Properties @Inject() (
.property("data", UMapping.string.optional)(_.select(_.data.value(_.data)).readonly)
// TODO add attachment ?
.build

lazy val taxonomy: PublicProperties =
PublicPropertyListBuilder[Taxonomy]
.property("namespace", UMapping.string)(_.field.readonly)
.property("description", UMapping.string)(_.field.readonly)
.property("version", UMapping.int)(_.field.readonly)
.build

}
8 changes: 8 additions & 0 deletions thehive/app/org/thp/thehive/controllers/v1/Router.scala
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ class Router @Inject() (
taskCtrl: TaskCtrl,
customFieldCtrl: CustomFieldCtrl,
alertCtrl: AlertCtrl,
taxonomyCtrl: TaxonomyCtrl,
auditCtrl: AuditCtrl,
statusCtrl: StatusCtrl,
authenticationCtrl: AuthenticationCtrl,
Expand Down Expand Up @@ -90,6 +91,13 @@ class Router @Inject() (
// DELETE /alert/:alertId controllers.AlertCtrl.delete(alertId)
// POST /alert/:alertId/merge/:caseId controllers.AlertCtrl.mergeWithCase(alertId, caseId)

case POST(p"/taxonomy") => taxonomyCtrl.create
case POST(p"/taxonomy/import-zip") => taxonomyCtrl.importZip
case GET(p"/taxonomy/$taxoId") => taxonomyCtrl.get(taxoId)
case PUT(p"/taxonomy/$taxoId/activate") => taxonomyCtrl.toggleActivation(taxoId, isActive = true)
case PUT(p"/taxonomy/$taxoId/deactivate") => taxonomyCtrl.toggleActivation(taxoId, isActive = false)
case DELETE(p"/taxonomy/$taxoId") => taxonomyCtrl.delete(taxoId)

case GET(p"/audit") => auditCtrl.flow
// GET /flow controllers.AuditCtrl.flow(rootId: Option[String], count: Option[Int])
// GET /audit controllers.AuditCtrl.find()
Expand Down
Loading