Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] Unable to Migrate from TheHive 3.5.1 --> 4.1.9 to a new system #2163

Open
Fedora7830 opened this issue Aug 10, 2021 · 1 comment
Open

[Bug] Unable to Migrate from TheHive 3.5.1 --> 4.1.9 to a new system #2163

Fedora7830 opened this issue Aug 10, 2021 · 1 comment
Labels
bug TheHive4 TheHive4 related issues

Comments

@Fedora7830
Copy link

Request Type

Bug

Work Environment

Question Answer
OS version (server) RedHat --> RockyLinux
OS version (client) RedHat7 --> RockyLinux8
Virtualized Env. True
Dedicated RAM 12 GB
SWAP 4 GB
vCPU 2
TheHive version / git hash 3.5.1-1 (RedHat) --> 4.1.9-1 (Rocky)
Package Type RPM
Database Cassandra
Index type Elasticsearch
Attachments storage Local

Problem Description

I am unable to Migrate thehive instance that is currently running on RedHat 7, running TheHive Version 3.5.1-1 installed from the yum repo. We are migrating and upgrading to a new system running RockLinux 8.4 where TheHive version 4.1.9 is installed.

Current Instance New Instance
cassandra.noarch (3.11.10-1)
elasticsearch.x86_64 (7.13.4-1) elasticsearch.x86_64 (7.13.4-1)
thehive.noarch (3.5.1-1) thehive4.noarch 4.1.9-1

On both servers, all of the applications are running on a single server. The Elasticsearch instance is running with a single node on each respective server, listening only to localhost.

Initially, the migration starts on the current RedHat (RHEL) server. On this server, I updated thehive to the 3.5.1 version. No issues were encountered during this update. Elastic's index was then migrated to the RockyLinux Server. On the RockyLinux server, the above packages were installed and the configurations set up to match the (as close as possible) the configurations on the RHEL server. By itself prior to the migration, TheHive (v4) runs as expected on the RockyLinux server.

Steps to Reproduce

  1. Stopped thehive service

    systemctl stop thehive

  2. Listed the existing indexes on the Elasticsearch instance on RockyLinux

    curl -XGET http://localhost:9200/_alias?pretty
{
"new_the_hive_15" : {
    "aliases" : {
    "the_hive" : { },
    "the_hive_15" : { },
    "the_hive_17" : { }
    }
},
"thehive_global" : {
    "aliases" : { }
}
}

    The aliases have been added through previous upgrades to our instance of thehive.

  3. thehive_global instance is removed

    curl -XDELETE 'http://localhost:9200/thehive_global'

    This process has been attempted with and without running the above delete command. The same errors will occur in both instances.

  4. The migration script is then run

    opt/thehive/bin/migrate -d --input /opt/thehive/bin/thehive3.conf --output /etc/thehive/application.conf --es-uri http://127.0.0.1:9200   --es-index the_hive -m mainOrg

    The above command has been run with and without the -m option. No changes in the errors from the migration are observed.

    The migration will run to completion, but will contain many errors. Several Examples are below:

    • Example 1

      [info] Initialising cluster
[info] Member is Up: akka://[email protected]:35105
[info] Full-text index is available (elasticsearch:[127.0.0.1]) single node
[warn] An error occurs (org.janusgraph.core.JanusGraphException: Could not call index), retrying (1)
[info] [Migrate users] Organisation:1/1(2119ms)
[warn] An error occurs (org.janusgraph.core.SchemaViolationException: Adding this property for key [_label] and value [User] violates a uniqueness constraint [UserLogin]), retrying (1)
[warn] An error occurs (org.janusgraph.core.SchemaViolationException: Adding this property for key [_label] and value [User] violates a uniqueness constraint [UserLogin]), retrying (2)
[warn] An error occurs (org.janusgraph.core.SchemaViolationException: Adding this property for key [_label] and value [User] violates a uniqueness constraint [UserLogin]), retrying (3)
[warn] An error occurs (org.janusgraph.core.SchemaViolationException: Adding this property for key [_label] and value [User] violates a uniqueness constraint [UserLogin]), retrying (4)
[warn] An error occurs (org.janusgraph.core.SchemaViolationException: Adding this property for key [_label] and value [User] violates a uniqueness constraint [UserLogin]), retrying (5)
[error] An error occurs
org.janusgraph.core.SchemaViolationException: Adding this property for key [_label] and value [User] violates a uniqueness constraint [UserLogin]
    at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.addProperty(StandardJanusGraphTx.java:832)
    at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.addProperty(StandardJanusGraphTx.java:754)
    at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.addProperty(StandardJanusGraphTx.java:750)
    at org.janusgraph.graphdb.vertices.AbstractVertex.property(AbstractVertex.java:152)
    at org.janusgraph.core.JanusGraphVertex.property(JanusGraphVertex.java:72)
    at org.janusgraph.core.JanusGraphVertex.property(JanusGraphVertex.java:33)
    at org.thp.scalligraph.models.SingleMapping.setProperty(Mapping.scala:189)
    at org.thp.scalligraph.janus.JanusDatabase.createVertex(JanusDatabase.scala:460)
    at org.thp.scalligraph.services.VertexSrv.createEntity(VertexSrv.scala:40)
    at org.thp.thehive.migration.th4.Output.$anonfun$createUser$4(Output.scala:330)
[error] User creation failure: org.thp.scalligraph.models.DatabaseException: Violation of database schema

    • Example 2

      [warn] An error occurs (org.janusgraph.core.SchemaViolationException: Adding this property for key [_label] and value [CustomField] violates a uniqueness constraint [CustomfieldName]), retrying (1)
[warn] An error occurs (org.janusgraph.core.SchemaViolationException: Adding this property for key [_label] and value [CustomField] violates a uniqueness constraint [CustomfieldName]), retrying (2)
[warn] An error occurs (org.janusgraph.core.SchemaViolationException: Adding this property for key [_label] and value [CustomField] violates a uniqueness constraint [CustomfieldName]), retrying (3)
[warn] An error occurs (org.janusgraph.core.SchemaViolationException: Adding this property for key [_label] and value [CustomField] violates a uniqueness constraint [CustomfieldName]), retrying (4)
[info] [Migrate custom fields] Organisation:1/1 User:15/15 CustomField:17/19(3538ms)
[warn] An error occurs (org.janusgraph.core.SchemaViolationException: Adding this property for key [_label] and value [CustomField] violates a uniqueness constraint [CustomfieldName]), retrying (5)
[error] An error occurs
org.janusgraph.core.SchemaViolationException: Adding this property for key [_label] and value [CustomField] violates a uniqueness constraint [CustomfieldName]
    at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.addProperty(StandardJanusGraphTx.java:832)
    at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.addProperty(StandardJanusGraphTx.java:754)
    at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.addProperty(StandardJanusGraphTx.java:750)
    at org.janusgraph.graphdb.vertices.AbstractVertex.property(AbstractVertex.java:152)
    at org.janusgraph.core.JanusGraphVertex.property(JanusGraphVertex.java:72)
    at org.janusgraph.core.JanusGraphVertex.property(JanusGraphVertex.java:33)
    at org.thp.scalligraph.models.SingleMapping.setProperty(Mapping.scala:189)
    at org.thp.scalligraph.janus.JanusDatabase.createVertex(JanusDatabase.scala:460)
    at org.thp.scalligraph.services.VertexSrv.createEntity(VertexSrv.scala:40)
    at org.thp.thehive.services.CustomFieldSrv.createEntity(CustomFieldSrv.scala:34)
[error] CustomField creation failure: org.thp.scalligraph.models.DatabaseException: Violation of database schema

    • Example 3

      error] ObservableType creation failure: org.thp.scalligraph.CreateError: Observable type ip already exists
[error] uncaught error, not retrying
org.thp.scalligraph.CreateError: Observable type uri_path already exists
    at org.thp.thehive.services.ObservableTypeSrv.create(ObservableTypeSrv.scala:33)
    at org.thp.thehive.migration.th4.Output.$anonfun$createObservableTypes$2(Output.scala:386)
    at org.thp.thehive.migration.th4.Output.$anonfun$authTransaction$1(Output.scala:280)
    at org.thp.scalligraph.janus.JanusDatabase.$anonfun$tryTransaction$7(JanusDatabase.scala:241)
    at scala.util.Try$.apply(Try.scala:213)
    at org.thp.scalligraph.janus.JanusDatabase.$anonfun$tryTransaction$6(JanusDatabase.scala:241)
    at scala.util.Try$.apply(Try.scala:213)
    at org.thp.scalligraph.utils.DelayRetry.withTry(Retry.scala:93)
    at org.thp.scalligraph.janus.JanusDatabase.tryTransaction(JanusDatabase.scala:238)
    at org.thp.thehive.migration.th4.Output.authTransaction(Output.scala:279)
[error] Exception raised, rollback (Observable type uri_path already exists)
[error] ObservableType creation failure: org.thp.scalligraph.CreateError: Observable type uri_path already exists
[error] uncaught error, not retrying
org.thp.scalligraph.CreateError: Observable type hash already exists
    at org.thp.thehive.services.ObservableTypeSrv.create(ObservableTypeSrv.scala:33)
    at org.thp.thehive.migration.th4.Output.$anonfun$createObservableTypes$2(Output.scala:386)
    at org.thp.thehive.migration.th4.Output.$anonfun$authTransaction$1(Output.scala:280)
    at org.thp.scalligraph.janus.JanusDatabase.$anonfun$tryTransaction$7(JanusDatabase.scala:241)
    at scala.util.Try$.apply(Try.scala:213)
    at org.thp.scalligraph.janus.JanusDatabase.$anonfun$tryTransaction$6(JanusDatabase.scala:241)
    at scala.util.Try$.apply(Try.scala:213)
    at org.thp.scalligraph.utils.DelayRetry.withTry(Retry.scala:93)
    at org.thp.scalligraph.janus.JanusDatabase.tryTransaction(JanusDatabase.scala:238)
    at org.thp.thehive.migration.th4.Output.authTransaction(Output.scala:279)
[error] Exception raised, rollback (Observable type hash already exists)
[error] ObservableType creation failure: org.thp.scalligraph.CreateError: Observable type hash already exists

    • Example 4

      [warn] Unable to set custom field encryptionStatus=null: org.thp.scalligraph.NotFoundError: Custom field encryptionStatus not found
[warn] Unable to set custom field remediation=null: org.thp.scalligraph.NotFoundError: Custom field remediation not found
[warn] Unable to set custom field hostName=null: org.thp.scalligraph.NotFoundError: Custom field hostName not found
[warn] Unable to set custom field ownedBy=null: org.thp.scalligraph.NotFoundError: Custom field ownedBy not found
[warn] Unable to set custom field clientPhone=null: org.thp.scalligraph.NotFoundError: Custom field clientPhone not found
[warn] Unable to set custom field natureOfIncident=null: org.thp.scalligraph.NotFoundError: Custom field natureOfIncident not found
[warn] Unable to set custom field typeOfDevice=null: org.thp.scalligraph.NotFoundError: Custom field typeOfDevice not found
[warn] Unable to set custom field clientEmailAddress=null: org.thp.scalligraph.NotFoundError: Custom field clientEmailAddress not found
[warn] Unable to set custom field incidentType="Security": org.thp.scalligraph.NotFoundError: Custom field incidentType not found
[warn] Unable to set custom field iPAddress=null: org.thp.scalligraph.NotFoundError: Custom field iPAddress not found
[warn] Unable to set custom field clientName=null: org.thp.scalligraph.NotFoundError: Custom field clientName not found
[warn] Unable to set custom field timeClose=null: org.thp.scalligraph.NotFoundError: Custom field timeClose not found
[warn] Unable to set custom field countTowardsIncident=true: org.thp.scalligraph.NotFoundError: Custom field countTowardsIncident not found
[warn] Unable to set custom field timeStart=null: org.thp.scalligraph.NotFoundError: Custom field timeStart not found
[warn] Unable to set custom field encryptionStatus=null: org.thp.scalligraph.NotFoundError: Custom field encryptionStatus not found
[warn] Unable to set custom field hostName=null: org.thp.scalligraph.NotFoundError: Custom field hostName not found
[warn] Unable to set custom field ownedBy=null: org.thp.scalligraph.NotFoundError: Custom field ownedBy not found
[warn] Unable to set custom field typeOfDevice=null: org.thp.scalligraph.NotFoundError: Custom field typeOfDevice not found
[warn] Unable to set custom field clientEmailAddress=null: org.thp.scalligraph.NotFoundError: Custom field clientEmailAddress not found
[warn] Unable to set custom field incidentType="Security": org.thp.scalligraph.NotFoundError: Custom field incidentType not found
[warn] Unable to set custom field clientName=null: org.thp.scalligraph.NotFoundError: Custom field clientName not found
[warn] Unable to set custom field countTowardsIncident=true: org.thp.scalligraph.NotFoundError: Custom field countTowardsIncident not found
[warn] Unable to set custom field encryptionStatus=null: org.thp.scalligraph.NotFoundError: Custom field encryptionStatus not found
[warn] Unable to set custom field remediation=null: org.thp.scalligraph.NotFoundError: Custom field remediation not found
[warn] Unable to set custom field hostName=null: org.thp.scalligraph.NotFoundError: Custom field hostName not found
[warn] Unable to set custom field ownedBy=null: org.thp.scalligraph.NotFoundError: Custom field ownedBy not found
[warn] Unable to set custom field clientPhone=null: org.thp.scalligraph.NotFoundError: Custom field clientPhone not found
[warn] Unable to set custom field natureOfIncident=null: org.thp.scalligraph.NotFoundError: Custom field natureOfIncident not found
[warn] Unable to set custom field typeOfDevice=null: org.thp.scalligraph.NotFoundError: Custom field typeOfDevice not found
[warn] Unable to set custom field clientEmailAddress=null: org.thp.scalligraph.NotFoundError: Custom field clientEmailAddress not found
[warn] Unable to set custom field incidentType="Security": org.thp.scalligraph.NotFoundError: Custom field incidentType not found
[warn] Unable to set custom field iPAddress=null: org.thp.scalligraph.NotFoundError: Custom field iPAddress not found
[warn] Unable to set custom field clientName=null: org.thp.scalligraph.NotFoundError: Custom field clientName not found
[warn] Unable to set custom field didTheAttackResultInALossCompromiseOfSensitivePersonalInformation=null: org.thp.scalligraph.NotFoundError: Custom field didTheAttackResultInALossCompromiseOfSensitivePersonalInformation not found
[warn] Unable to set custom field timeClose=null: org.thp.scalligraph.NotFoundError: Custom field timeClose not found
[warn] Unable to set custom field firstResponder=null: org.thp.scalligraph.NotFoundError: Custom field firstResponder not found
[warn] Unable to set custom field countTowardsIncident=true: org.thp.scalligraph.NotFoundError: Custom field countTowardsIncident not found
[warn] Unable to set custom field timeStart=null: org.thp.scalligraph.NotFoundError: Custom field timeStart not found
[warn] Unable to set custom field countTowardsIncident=null: org.thp.scalligraph.NotFoundError: Custom field countTowardsIncident not found
[warn] Unable to set custom field incidentType=null: org.thp.scalligraph.NotFoundError: Custom field incidentType not found
[warn] An error occurs (org.janusgraph.core.SchemaViolationException: Adding this property for key [_label] and value [ResolutionStatus] violates a uniqueness constraint [ResolutionstatusValue]), retrying (1)
[warn] An error occurs (org.janusgraph.core.SchemaViolationException: Adding this property for key [_label] and value [ResolutionStatus] violates a uniqueness constraint [ResolutionstatusValue]), retrying (2)
[warn] An error occurs (org.janusgraph.core.SchemaViolationException: Adding this property for key [_label] and value [ResolutionStatus] violates a uniqueness constraint [ResolutionstatusValue]), retrying (3)
[warn] An error occurs (org.janusgraph.core.SchemaViolationException: Adding this property for key [_label] and value [ResolutionStatus] violates a uniqueness constraint [ResolutionstatusValue]), retrying (4)
[warn] An error occurs (org.janusgraph.core.SchemaViolationException: Adding this property for key [_label] and value [ResolutionStatus] violates a uniqueness constraint [ResolutionstatusValue]), retrying (5)

  5. Upon completion the following is displayed by the migration script

    [info] Member is Removed: akka://[email protected]:42281 after Exiting
[info] Stage: Finalisation
Action: 0/0
Alert: 0/0
Audit: 0/1628
Case/Observable: 0/6
Case/Task/Log: 0/187
Case/Task: 0/335
Case: 0/46 (45 failures, 1 exists) avg:3643ms
CaseTemplate: 0/5 (5 exists)
CustomField: 0/19 (19 exists)
ImpactStatus: 0/0
Job: 0/0
ObservableType: 0/15 (14 failures, 1 exists) avg:7ms
Organisation: 1/1 avg:1644ms
Profile: 0/0
ResolutionStatus: 0/0
User: 0/15 (15 exists)

Possible Solutions

Prior to creating this issue, I found the below issues that seemed like they may be relevant.

Complementary information

Attached are the following files:
@Fedora7830 Fedora7830 added bug TheHive4 TheHive4 related issues labels Aug 10, 2021
@Fedora7830
Copy link
Author

I was able to perform the migration to 4.1.9 on a Red Hat 7 Clone. I mentioned this as well in Issue #2164. I believe there is a difference in how Red Hat Distributions (CentOS 8, RockyLinux 8) are handling the TheHive application. On the Red Hat 7 server, where I was able to perform a successful migration to 4.1.9, the same commands and configurations had been copied over from the RockyLinux 8 Host, where the migration was failing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug TheHive4 TheHive4 related issues
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Fedora7830