From 8eb1d3bd7485981a3270ad6613978221a8eb90eb Mon Sep 17 00:00:00 2001 From: To-om Date: Mon, 30 Jul 2018 14:02:30 +0200 Subject: [PATCH 1/3] Update repository --- README.md | 2 +- analyzer_requirements.md | 2 +- api/how-to-create-an-analyzer.md | 4 ++-- installation/install-guide.md | 29 +++++++++++++++++++---------- 4 files changed, 23 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 90413c6b0..f93248f2f 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ Starting from Cortex version 2, you can create and manage multiple organizations **Notes:** -- This is the Cortex documentation repository. If you are looking for its source code, please visit [https://github.com/CERT-BDF/Cortex/](https://github.com/TheHive-Project/Cortex/). +- This is the Cortex documentation repository. If you are looking for its source code, please visit [https://github.com/TheHive-Project/Cortex/](https://github.com/TheHive-Project/Cortex/). - [Cortex4py](https://github.com/TheHive-Project/Cortex4py), the FOSS Python library we provide to submit observables in bulk mode through the Cortex REST API from alternative SIRP platforms & custom scripts, is compatible with Cortex 2 starting from v 2.0.0. - If you are looking for the Cortex 1 documentation, please check the [cortex-1](https://github.com/TheHive-Project/CortexDocs/tree/cortex-1) branch. diff --git a/analyzer_requirements.md b/analyzer_requirements.md index c0baee96e..0342f9386 100644 --- a/analyzer_requirements.md +++ b/analyzer_requirements.md @@ -208,7 +208,7 @@ The analyzer comes in two flavors: As long as you are using the **Status** flavor you don't need API key. If you want more details using the **Detail** flavor, you need to [sign up for a hashdd.com account and obtain an API](https://www.hashdd.com/). ### Hippocampe -Query threat feeds through [Hippocampe](https://github.com/CERT-BDF/Hippocampe), +Query threat feeds through [Hippocampe](https://github.com/TheHive-Project/Hippocampe), a FOSS tool from TheHive Project that centralizes feeds and allows you to associate a confidence level to each one of them (that can be changed over time) and get a score indicating the data quality. diff --git a/api/how-to-create-an-analyzer.md b/api/how-to-create-an-analyzer.md index 87e875f7b..b85978c29 100644 --- a/api/how-to-create-an-analyzer.md +++ b/api/how-to-create-an-analyzer.md @@ -193,7 +193,7 @@ submitting your analyzer. #### url The URL where the analyzer is stored. This should ideally be -`https://github.com/CERT-BDF/Cortex-Analyzers` +`https://github.com/TheHive-Project/Cortex-Analyzers` #### license The license of the code. Ideally, we recommend using the AGPL-v3 @@ -549,7 +549,7 @@ VirusTotal JSON analyzer configuration file(s): "name": "VirusTotal_GetReport", "version": "3.0", "author": "CERT-BDF", - "url": "https://github.com/CERT-BDF/Cortex-Analyzers", + "url": "https://github.com/TheHive-Project/Cortex-Analyzers", "license": "AGPL-V3", "description": "Get the latest VirusTotal report for a file, hash, domain or an IP address", "dataTypeList": ["file", "hash", "domain", "ip"], diff --git a/installation/install-guide.md b/installation/install-guide.md index 046c69a54..f22a8bc78 100644 --- a/installation/install-guide.md +++ b/installation/install-guide.md @@ -38,7 +38,7 @@ RPM packages are published on a Bintray repository. All packages are signed usin First install the RPM release package: ``` -yum install https://dl.bintray.com/cert-bdf/rpm/thehive-project-release-1.0.0-3.noarch.rpm +yum install https://dl.bintray.com/thehive-project/rpm-stable/thehive-project-release-1.1.0-2.noarch.rpm ``` This will install TheHive Project's repository in `/etc/yum.repos.d/thehive-rpm.repo` and the corresponding GPG public key in `/etc/pki/rpm-gpg/GPG-TheHive-Project`. @@ -50,6 +50,9 @@ yum install cortex Once the package is installed, [install the analyzers](#analyzers-1) as outlined in the next section and proceed to the configuration using the [Quick Start Guide](../admin/quick-start.md). For more advanced configuration options, please refer to the [Administration Guide](../admin/admin-guide.md). +#### Pre-release versions +The RPM release package install two repositories `thehive-project-stable` and `thehive-project-beta`. The latter contains pre-release version and is disabled by default. If you want to install them and help us to find bugs, you can enable the repository by editing `/etc/yum.repos.d/thehive-rpm.repo` and set `enable` value to `1` for `thehive-project-beta` repository. + ### DEB Debian packages are published on a Bintray repository. All packages are signed using our GPG key [562CBC1C](https://raw.githubusercontent.com/TheHive-Project/Cortex/master/PGP-PUBLIC-KEY). Its fingerprint is: @@ -57,7 +60,7 @@ Debian packages are published on a Bintray repository. All packages are signed u To install the Cortex Debian package, use the following commands: ``` -echo 'deb https://dl.bintray.com/cert-bdf/debian any main' | sudo tee -a /etc/apt/sources.list.d/thehive-project.list +echo 'deb https://dl.bintray.com/thehive-project/debian-stable any main' | sudo tee -a /etc/apt/sources.list.d/thehive-project.list sudo apt-key adv --keyserver hkp://pgp.mit.edu --recv-key 562CBC1C sudo apt-get update sudo apt-get install cortex @@ -69,6 +72,12 @@ Some environments may block access to the `pgp.mit.edu` key server. As a result, Once the package is installed, [install the analyzers](#analyzers-1) as outlined in the next section and proceed to the configuration using the [Quick Start Guide](../admin/quick-start.md). For more advanced configuration options, please refer to the [Administration Guide](../admin/admin-guide.md). +#### Pre-release versions +If you want to install pre-release version of packages and help us to find bugs, you can add the pre-release repository with the command: +```bash +echo 'deb https://dl.bintray.com/thehive-project/debian-beta any main' | sudo tee -a /etc/apt/sources.list.d/thehive-project.list +``` + ### Docker To use the Docker image, you must use [Docker](https://www.docker.com/) (courtesy of Captain Obvious). @@ -94,7 +103,7 @@ services: - thread_pool.search.queue_size=100000 - thread_pool.bulk.queue_size=100000 cortex: - image: certbdf/cortex:latest + image: thehiveproject/cortex:latest depends_on: - elasticsearch ports: @@ -122,7 +131,7 @@ volumes: Elasticsearch can be installed on the same server as Cortex or on a different one. You can then configure Cortex according to the [documentation](../admin/admin-guide.md) and run Cortex docker as follow: ``` -docker run --volume /path/to/cortex/application.conf:/etc/cortex/application.conf certbdf/cortex:latest --no-config +docker run --volume /path/to/cortex/application.conf:/etc/cortex/application.conf thehiveproject/cortex:latest --no-config ``` You can add the `--publish` docker option to expose the Cortex HTTP service. @@ -134,7 +143,7 @@ By default, the Cortex Docker image has minimal configuration: This behavior can be disabled by adding `--no-config` to the Docker command line: -`docker run certbdf/cortex:latest --no-config` +`docker run thehiveproject/cortex:latest --no-config` Or by adding the line `command: --no-config` in the `cortex` section of docker-compose file. @@ -157,7 +166,7 @@ Analyzers are embedded in the docker image under `/opt/Cortex-Analyzers/analyzer [install them](#analyzers-1) outside of Docker and overwrite the existing ones by adding the following parameter: ``` ---volume /path/to/analyzers:/opt/Cortex-Analyzers/analyzers:ro certbdf/cortex:latest +--volume /path/to/analyzers:/opt/Cortex-Analyzers/analyzers:ro thehiveproject/cortex:latest ``` #### What to Do Next? @@ -202,13 +211,13 @@ sudo apt-get install openjdk-8-jre-headless To install Elasticsearch, please read the [Elasticsearch Installation](#elasticsearch-installation) section below. #### 4. Install Cortex -Binary packages can be downloaded from [Bintray](https://dl.bintray.com/cert-bdf/cortex/). The latest version is called [cortex-latest.zip](https://dl.bintray.com/cert-bdf/cortex/cortex-latest.zip). +Binary packages can be downloaded from [Bintray](https://dl.bintray.com/thehive-project/binary/). The latest version is called [cortex-latest.zip](https://dl.bintray.com/thehive-project/binary/cortex-latest.zip). Download and unzip the chosen binary package. Cortex files can be installed wherever you want on the filesystem. In this guide, we assume you have chosen to install them under `/opt`. ``` cd /opt -wget https://dl.bintray.com/cert-bdf/cortex/cortex-latest.zip +wget https://dl.bintray.com/thehive-project/binary/cortex-latest.zip unzip cortex-latest.zip ln -s cortex-x.x.x cortex ``` @@ -263,7 +272,7 @@ restart the service. ``` service cortex stop cd /opt -wget https://dl.bintray.com/cert-bdf/cortex/cortex-latest.zip +wget https://dl.bintray.com/thehive-project/binary/cortex-latest.zip unzip cortex-latest.zip rm /opt/cortex && ln -s cortex-x.x.x cortex chown -R cortex:cortex /opt/cortex /opt/cortex-x.x.x @@ -482,7 +491,7 @@ for I in /path/to/Cortex-Analyzers/analyzers/*/requirements.txt; do sudo -H pip3 ``` After running these commands, read the Analyzer Requirements Guide,  log into the Cortex 2 Web UI as an `orgAdmin`, click on the Refresh Analyzers button in the Cortex Web UI, configure the new analyzers and enjoy! -If you are using TheHive, get the [latest version of the report templates](https://dl.bintray.com/cert-bdf/thehive/report-templates.zip) and import them into TheHive. +If you are using TheHive, get the [latest version of the report templates](https://dl.bintray.com/thehive-project/binary/report-templates.zip) and import them into TheHive. ### Additional Analyzers The following analyzers are not supported by THeHive Project at this time: From 626984d60afb139f970acbc6aa99e89aedfe79a3 Mon Sep 17 00:00:00 2001 From: To-om Date: Mon, 30 Jul 2018 16:39:57 +0200 Subject: [PATCH 2/3] Explain how to create a responder --- api/how-to-create-a-responder.md | 409 +++++++++++++++++++++++++++++++ api/how-to-create-an-analyzer.md | 66 ++--- 2 files changed, 435 insertions(+), 40 deletions(-) create mode 100644 api/how-to-create-a-responder.md diff --git a/api/how-to-create-a-responder.md b/api/how-to-create-a-responder.md new file mode 100644 index 000000000..a4759c497 --- /dev/null +++ b/api/how-to-create-a-responder.md @@ -0,0 +1,409 @@ +# How to Write and Submit a Responder + +## Table of Contents + * [Writing a Responder](#writing-an-responder) + * [The Program](#the-program) + * [Service Interaction Files (Flavors)](#service-interaction-files-flavors) + * [Python Requirements](#python-requirements) + * [Example: Mailer Responder Files](#example-mailer-responder-files) + * [Input](#input) + * [Service Interaction Configuration Items](#service-interaction-configuration-items) + * [Responder Configuration in the Global Configuration File](#responder-configuration-in-the-global-configuration-file) + * [Output](#output) + * [The Cortexutils Python Library](#the-cortexutils-python-library) + * [Submitting a Responder](#submitting-an-responder) + * [Check Existing Issues](#check-existing-issues) + * [Open an Issue](#open-an-issue) + * [Review your Service Interaction File(s)](#review-your-service-interaction-files) + * [Provide the List of Requirements](#provide-the-list-of-requirements) + * [Verify Execution](#verify-execution) + * [Create a Pull Request](#create-a-pull-request) + * [Need Help?](#need-help) + +## Writing a Responder +A responder is a program that takes JSON input and do an action and produces a basic result of that action. +It is made of at least 2 types of files: + +- The program itself +- One or several service interaction files or flavors +- A Python requirements file, which is only necessary if the responder is +written in Python. + +### The Program +The first type of files a responder is made of is the core program that +performs actions. It can be written in any programming +language that is supported by Linux. + +While many responders are written in Python (`*.py` files), you can write yours +in Ruby, Perl or even Scala. However, the very handy `Cortexutils` library +[described below](#the-cortexutils-python-library) is in Python. It greatly facilitates responder development and +it also provides some methods to quickly format the output to make it compliant +with the JSON schema expected by [TheHive](https://github.com/TheHive-Project/TheHive/). + +### Service Interaction Files (Flavors) +A responder must have at least one service interaction file. Such files +contain key configuration information such as the responder's author +information, the datatypes (thehive:case, thehive:alert, ...) the responder accepts as +input, the TLP and PAP above which it will refuse to execute to protect against data +leakage and to enforce sane OPSEC practices and so on. + +A responder can have two or more service interaction files to allow it to +perform different actions. We speak then of flavors. For example, a mailer +responder can send message using several body template. + +### Python Requirements +If the responder is written in Python, a `requirements.txt` must be provided +with the list of all the dependencies. + +### Example: Mailer Responder Files +Below is a directory listing of the files corresponding to the Mailer +responder. + +```bash +responders/Mailer +|-- Mailer.json +|-- requirements.txt +`-- mailer.py +``` + +### Input +The input of a responder can be any JSON data, even a simple string. The submitter must send data with +the structure expected by the program. The acceptable datatypes described in the +Service Interaction files indicate what kind of data is expected. +For example, if the program requires a `thehive:case`, input must be comply with +TheHive case. Below an example of `thehive:case` input. + +```json + { + "caseId":1484, + "title":"Invoice mail", + "description":"...", + "startDate":1532586900000, + "customFields":{}, + "pap":2, + "flag":false, + "status":"Open", + "owner":"me", + "severity":2, + "tlp":2, + "tags":["phishing"], + "metrics":{}, + "createdBy":"me", + "createdAt":1532586924589, + "updatedBy":"me", + "updatedAt":1532697739274, + "_type":"case", + "_routing":"AWTVTJzWpVfF5eRPi-qd", + "_parent":null, + "_id":"AWTVTJzWpVfF5eRPi-qd", + "_version":3, + "id":"AWTVTJzWpVfF5eRPi-qd", + "config":{ + "key":"1234567890abcdef", + "max_tlp":1, + "check_tlp":true, + "max_pap":1 + "check_pap":true, + [..] + }, + "proxy":{ + "http":"http://myproxy:8080", + "https":"https://myproxy:8080" + } + } +``` + +From the input send by the submitter, Cortex add `config` which is the responder's specific +configuration provided by org admin when the responder has been enabled. + +#### Example: Service Interaction File for VirusTotal GetReport +The `<==` sign and anything after it are comments that do no appear in the +original file. +```json +{ + "name": "Mailer", + "version": "1.0", + "author": "CERT-BDF", + "url": "https://github.com/TheHive-Project/Cortex-Analyzers", + "license": "AGPL-V3", + "description": "Send an email with information from a TheHive case or alert", + "dataTypeList": ["thehive:case", "thehive:alert"], + "command": "Mailer/mailer.py", <== Program to run when invoking the responder + "baseConfig": "Mailer", <== name of base config in Cortex responder config page + "config": { + <== any configuration items + }, + "configurationItems": [ <== list of configuration items the responder needs to operate (api key etc.) + { + "name": "from", + "description": "email address from which the mail is send", + "type": "string", <== defines what kind of data type the configuration item is (string, number) + "multi": false, <== setting multi to true allows to pass a list of items + "required": true + }, + { + "name": "smtp.host", + "description": "SMTP server used to send mail", + "type": "string", + "multi": false, + "required": true, + "defaultValue": "localhost" + } + ] +} +``` + +### Service Interaction Configuration Items +#### name +Name of the specific service (or flavor) of the responder. + +If your responder has only one service interaction (i.e. performs only one +action), it is the name of the responder's directory. + +If your responder performs several actions (i.e. comes in several flavors), +you have to give a specific and meaningful name to each flavor. + +Each flavor's name appear in TheHive's responder list and in MISP when you +use Cortex for attribute enrichment. + +#### version +The version of the responder. + +You **must** increase major version numbers when new features are added, +modifications are made to take into account API changes, report output is +modified or when report templates (more on this later) are updated. + +You must increase minor version numbers when bugs are fixed. + +#### author +You must provide your full name and/or your organization/team name when +submitting a responder. Pseudos are not accepted. If you'd rather remain +anonymous, please contact us at support@thehive-project.org prior to +submitting your responder. + +#### url +The URL where the responder is stored. This should ideally be +`https://github.com/TheHive-Project/Cortex-Analyzers` + +#### license +The license of the code. Ideally, we recommend using the AGPL-v3 +license. + +Make sure your code's license is compatible with the license(s) of the +various components and libraries you use if applicable. + +#### description +Description of the responder. Please be concise and clear. The description is + shown in the Cortex UI, TheHive and MISP. + +#### dataTypeList +The list of TheHive datatypes supported by the responder. Currently TheHive +accepts the following datatypes: + +- thehive:case +- thehive:case_artifact +- thehive:alert +- thehive:case_task +- thehive:log + +#### baseConfig +Name used to group configuration items common to several responders. This +prevent the user to enter the same API key for all responder flavors. +The Cortex responder config page group configuration items by there baseConfig. + +#### config +Configuration dedicated to the responder's flavor. This is where we + typically specify the TLP level of observables allowed to be analyzed with the + `check_tlp` and `max_tlp` parameters. For example, if `max_tlp` is set to `2` (TLP:AMBER), + TLP:RED observables cannot be analyzed. + +##### max_tlp +The TLP level above which the responder must not be executed. + +| TLP | max_tlp value | +|:----------:|:-------------:| +| Unknown | -1 | +| WHITE | 0 | +| GREEN | 1 | +| AMBER | 2 | +| RED | 3 | + + +##### check_tlp +This is a boolean parameter. When `true`, `max_tlp` is checked. And if the +input's TLP is above `max_tlp`, the responder is not executed. + +For consistency reasons, we do recommend setting both `check_tlp` and +`max_tlp` even if `check_tlp` is set to `false`. + +##### max_pap +The PAP level above which the responder must not be executed. + +| TLP | max_tlp value | +|:----------:|:-------------:| +| Unknown | -1 | +| WHITE | 0 | +| GREEN | 1 | +| AMBER | 2 | +| RED | 3 | + + +##### check_pap +This is a boolean parameter. When `true`, `max_pap` is checked. And if the +input's PAP is above `max_pap`, the responder is not executed. + +For consistency reasons, we do recommend setting both `check_pap` and +`max_pap` even if `check_pap` is set to `false`. + +##### command +The command used to run the responder. That's typically the full, absolute +path to the main program file. + +#### configurationItems +The list of configurationItems is necessary in order to be able to set all configuration variables for responders directly in the Cortex 2 user interface. As in the VirusTotal example above can be seen, every item is a json object that defines: +- name (string) +- description (string) +- type (string) +- multi (boolean) +- required (boolean) +- defaultValue (according to type, optional) + +The `multi` parameter allows to pass a list as configuration variable instead of a single string or number. This is used e.g. in the MISP responder that queries multiple servers in one run and needs different parameters for that. + +### Output +The output of a responder depends on the success or failure of its execution. + +If the responder **fails** to execute: + +```json +{ + "success": false, + "errorMessage":".." +} +``` + +- When `success` is set to `false`, it indicates that something went wrong + during the execution. +- `errorMessage` is free text - typically the error output message. + +If the responder **succeeds** (i.e. it runs without any error): + +```json +{ + "success":true, + "full":{ "message": ".." }, + "operations":[] +} +``` + +- When `success` is set to `true`, it indicates that the responder ran + successfully. +- `full` is the full report of the responder. It must contain at least + a message. +- `operations` is a list what the submitter system should execute. + Currently, TheHive accepts the following operations: + - `AddTagToCase` (`{ "type": "AddTagToCase", "tag": "tag to add" }`): add + a tag to the case related to the object + The list of acceptable operation will increase in future release of TheHive. + +### The Cortexutils Python Library +So far, all the published responders have been written in Python. We +released a special Python library called `cortexutils` to help developers easily write their programs. Note though that Python is not mandatory for responder coding and any language that runs on Linux can be used, though you won't have the benefits of the CortexUtils library. + +Cortexutils can be used with Python 2 and 3. +To install it : + +```bash +pip install cortexutils +``` + +or + +```bash +pip3 install cortexutils +``` + +This library is already used by all the responders published in our [Github +repository](https://github.com/TheHive-Project/Cortex-Analyzers). Feel free to +start reading the code of some of them before writing your own. + +## Submitting a Responder +We **highly encourage you to share your responders** with the community through our Github repository. To do so, we invite you to follow a few steps before submitting a pull request. + +### Check Existing Issues +Start by checking [if an issue already exists](https://github.com/TheHive-Project/Cortex-Analyzers/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aopen%20label%3A%22feature%20request%22%20label%3Aresponder) for the responder you'd like to write and contribute. Verify that nobody is working on it. If an issue exists and has the **in progress**, **under review** or **pr-submitted** label, it means somebody is already working on the code or has finished it. + +If you are short on ideas, check issues with a [**help wanted** label](https://github.com/TheHive-Project/Cortex-Analyzers/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aopen%20label%3A%22help%20wanted%22). If one of those issues interest you, indicate that you are working on it. + +### Open an Issue +If there's no issue open for the responder you'd like to contribute, [open one](https://github.com/TheHive-Project/Cortex-Analyzers/issues/new). Indicate that you are working on it to avoid having someone start coding it. + +You have to create an issue for each responder you'd like to submit. + +### Review your Service Interaction File(s) +Review your service interaction files. For example, let's check the +Mailer JSON responder configuration file(s): + +```json +{ + "name": "Mailer", + "version": "1.0", + "author": "CERT-BDF", + "url": "https://github.com/TheHive-Project/Cortex-Analyzers", + "license": "AGPL-V3", + "description": "Send an email with information from a TheHive case or alert", + "dataTypeList": ["thehive:case", "thehive:alert"], + "command": "Mailer/mailer.py", <== Program to run when invoking the responder + "baseConfig": "Mailer", <== name of base config in Cortex responder config page + "config": { + <== any configuration items + }, + "configurationItems": [ <== list of configuration items the responder needs to operate (api key etc.) + { + "name": "from", + "description": "email address from which the mail is send", + "type": "string", <== defines what kind of data type the configuration item is (string, number) + "multi": false, <== setting multi to true allows to pass a list of items + "required": true + }, + { + "name": "smtp.host", + "description": "SMTP server used to send mail", + "type": "string", + "multi": false, + "required": true, + "defaultValue": "localhost" + } + ] +} + +``` +Ensure that all information is correct and particularly the `author` and +`license` parameters. + +### Provide the List of Requirements +If your responder is written in Python, make sure to complete the +`requirements.txt` file with the list of all the external libraries that are +needed to run the responder correctly. + +### Verify Execution +Use these three simple checks before submtting your responder: + +- Ensure it works with the expecyed configuration, TLP or dataType. +- Ensure it works with missing configuration, dataType or TLP: your +responder must generate an explicit error message. +- Ensure the long report template handles error messages correctly. + +### Create a Pull Request +Create one Pull Request per responder against the **develop** branch of the +[Cortex-Analyzers](https://github.com/TheHive-Project/Cortex-Analyzers/) repository. Reference the issue you've created in your PR. + +We have to review your responders. Distinct PRs will allow us to review them +more quickly and release them to the benefit of the whole community. + +## Need Help? +Something does not work as expected? No worries, we got you covered. Please +join our [user forum](https://groups.google.com/a/thehive-project.org/forum/#!forum/users), + contact us on [Gitter](https://gitter.im/TheHive-Project/TheHive), or send us + an email at [support@thehive-project.org](mailto:support@thehive-project + .org). We are here to help. diff --git a/api/how-to-create-an-analyzer.md b/api/how-to-create-an-analyzer.md index b85978c29..05b0ae955 100644 --- a/api/how-to-create-an-analyzer.md +++ b/api/how-to-create-an-analyzer.md @@ -48,7 +48,7 @@ with the JSON schema expected by [TheHive](https://github.com/TheHive-Project/Th An analyzer must have at least one service interaction file. Such files contain key configuration information such as the analyzer's author information, the datatypes (IP, URL, hash, domain...) the analyzer accepts as -input, the TLP above which it will refuse to execute to protect against data +input, the TLP and PAP above which it will refuse to execute to protect against data leakage and to enforce sane OPSEC practices and so on. An analyzer can have two or more service interaction files to allow it to @@ -102,9 +102,7 @@ pieces of information. For example, to use the VirusTotal analyzer's `data`, `dataType` and `tlp` are the observable-related information generated by TheHive or any other program that is calling Cortex. `config` is the -analyzer's specific configuration items, which are obtained from the -Cortex configuration file - `/etc/cortex/application.conf` - and the service - interaction file, `VirusTotal_GetReport.json`. +analyzer's specific configuration provided by org admin when the analyzer has been enabled. Let's take the **GetReport** flavor of the VirusTotal analyzer as an example again. @@ -133,7 +131,7 @@ original file. "description": "Get the latest VirusTotal report for a file, hash, domain or an IP address.", "dataTypeList": ["file", "hash", "domain", "ip"], "command": "VirusTotal/virustotal.py", <== Program to run when invoking the analyzer - "baseConfig": "VirusTotal", <== name of base config in Cortex application.conf (/etc/cortex/application.conf) + "baseConfig": "VirusTotal", <== name of base config in Cortex analyzer config page "config": { "service": "get" }, @@ -229,8 +227,9 @@ If you need additional datatypes for your analyzer, please let us know at [support@thehive-project.org](mailto:support@thehive-project.org). #### baseConfig -Name used for the analyzer in the global Cortex configuration file -(`/etc/cortex/application.conf`). +Name used to group configuration items common to several analyzer. This +prevent the user to enter the same API key for all analyzer flavors. +The Cortex analyzer config page group configuration items by there baseConfig. #### config Configuration dedicated to the analyzer's flavor. This is where we @@ -257,6 +256,25 @@ input's TLP is above `max_tlp`, the analyzer is not executed. For consistency reasons, we do recommend setting both `check_tlp` and `max_tlp` even if `check_tlp` is set to `false`. +##### max_pap +The PAP level above which the analyzer must not be executed. + +| TLP | max_tlp value | +|:----------:|:-------------:| +| Unknown | -1 | +| WHITE | 0 | +| GREEN | 1 | +| AMBER | 2 | +| RED | 3 | + + +##### check_pap +This is a boolean parameter. When `true`, `max_pap` is checked. And if the +input's PAP is above `max_pap`, the analyzer is not executed. + +For consistency reasons, we do recommend setting both `check_pap` and +`max_pap` even if `check_pap` is set to `false`. + ##### command The command used to run the analyzer. That's typically the full, absolute path to the main program file. @@ -272,38 +290,6 @@ The list of configurationItems is necessary in order to be able to set all confi The `multi` parameter allows to pass a list as configuration variable instead of a single string or number. This is used e.g. in the MISP analyzer that queries multiple servers in one run and needs different parameters for that. -### Analyzer Configuration in the Global Configuration File -An analyzer might need specific configuration parameters such as a username and -password, an API key, a URL and so on. These items must be added to the -relevant section in the Cortex global configuration file -(`/etc/cortex/application.conf`). - -Please respect the following variable calling convention: - -| Type of data | Variable name | -|:----------:|:-------------:| -| An API key | `key` (lowercase) | -| A username | `username` | -| A password | `password` | -| a URL | `url` | - -Here is how the configuration section of the VirusTotal analyzer looks in `/etc/cortex/application.conf`: - -``` -analyzer { - config { - global { - proxy { - http="http://myproxy:8080", - https="https://myproxy:8080" - } - [..] - VirusTotal { - key="1234567890abcdef" - } - [..] -``` - ### Output The output of an analyzer depends on the success or failure of its execution. @@ -324,7 +310,7 @@ If the analyzer **succeeds** (i.e. it runs without any error): ```json { - "success":true + "success":true, "artifacts":[..], "summary":{ "taxonomies":[..] From 4a08e1d9545e6eb229866b2a2a1f6282b0df9cc7 Mon Sep 17 00:00:00 2001 From: Saad Kadhi Date: Tue, 31 Jul 2018 10:54:24 +0200 Subject: [PATCH 3/3] Proof-reading --- api/how-to-create-a-responder.md | 63 ++++++++++++++------------------ api/how-to-create-an-analyzer.md | 7 ++-- installation/install-guide.md | 9 ++++- 3 files changed, 37 insertions(+), 42 deletions(-) diff --git a/api/how-to-create-a-responder.md b/api/how-to-create-a-responder.md index a4759c497..48055e2d0 100644 --- a/api/how-to-create-a-responder.md +++ b/api/how-to-create-a-responder.md @@ -21,8 +21,7 @@ * [Need Help?](#need-help) ## Writing a Responder -A responder is a program that takes JSON input and do an action and produces a basic result of that action. -It is made of at least 2 types of files: +A responder is a program that takes JSON input and do an action and produces a basic result of that action. Responders are very similar to analyzers though they have different purposes. Responders are made of at least 2 types of files: - The program itself - One or several service interaction files or flavors @@ -31,32 +30,29 @@ written in Python. ### The Program The first type of files a responder is made of is the core program that -performs actions. It can be written in any programming -language that is supported by Linux. +performs actions. It can be written in any programming language that is supported by Linux. -While many responders are written in Python (`*.py` files), you can write yours -in Ruby, Perl or even Scala. However, the very handy `Cortexutils` library -[described below](#the-cortexutils-python-library) is in Python. It greatly facilitates responder development and +you can write your responders in Python, Ruby, Perl or even Scala. However, the very handy `Cortexutils` library [described below](#the-cortexutils-python-library) is in Python. It greatly facilitates responder development and it also provides some methods to quickly format the output to make it compliant with the JSON schema expected by [TheHive](https://github.com/TheHive-Project/TheHive/). ### Service Interaction Files (Flavors) A responder must have at least one service interaction file. Such files contain key configuration information such as the responder's author -information, the datatypes (thehive:case, thehive:alert, ...) the responder accepts as -input, the TLP and PAP above which it will refuse to execute to protect against data +information, the datatypes (`thehive:case`, `thehive:alert`, ...) the responder accepts as +input and to which it applies to if used from TheHive, the TLP and PAP (or [Permissible Actions Protocol](https://www.misp-project.org/taxonomies.html#_pap)) above which it will refuse to execute to protect against data leakage and to enforce sane OPSEC practices and so on. A responder can have two or more service interaction files to allow it to -perform different actions. We speak then of flavors. For example, a mailer -responder can send message using several body template. +perform different actions. We speak then of flavors. For example, a Mailer +responder can send message using several body templates. ### Python Requirements If the responder is written in Python, a `requirements.txt` must be provided with the list of all the dependencies. ### Example: Mailer Responder Files -Below is a directory listing of the files corresponding to the Mailer +Below is a directory listing of the files corresponding to a Mailer responder. ```bash @@ -70,8 +66,7 @@ responders/Mailer The input of a responder can be any JSON data, even a simple string. The submitter must send data with the structure expected by the program. The acceptable datatypes described in the Service Interaction files indicate what kind of data is expected. -For example, if the program requires a `thehive:case`, input must be comply with -TheHive case. Below an example of `thehive:case` input. +For example, if the program requires a `thehive:case` (i.e. it applies at the case level in TheHive), input must comply with TheHive case. Below an example of `thehive:case` input. ```json { @@ -113,10 +108,9 @@ TheHive case. Below an example of `thehive:case` input. } ``` -From the input send by the submitter, Cortex add `config` which is the responder's specific -configuration provided by org admin when the responder has been enabled. +In the addition to the input sent by the submitter, Cortex adds the `config` section which is the responder's specific configuration provided by an `orgAdmin` user when the responder is enabled in the Cortex UI. -#### Example: Service Interaction File for VirusTotal GetReport +#### Example: Service Interaction File for the Mailer Responder The `<==` sign and anything after it are comments that do no appear in the original file. ```json @@ -194,28 +188,27 @@ various components and libraries you use if applicable. #### description Description of the responder. Please be concise and clear. The description is - shown in the Cortex UI, TheHive and MISP. + shown in the Cortex UI and TheHive. #### dataTypeList The list of TheHive datatypes supported by the responder. Currently TheHive accepts the following datatypes: -- thehive:case -- thehive:case_artifact -- thehive:alert -- thehive:case_task -- thehive:log +- `thehive:case` +- `thehive:case_artifact` (i.e. observable) +- `thehive:alert` +- `thehive:case_task` +- `thehive:log` (i.e. task log) #### baseConfig Name used to group configuration items common to several responders. This prevent the user to enter the same API key for all responder flavors. -The Cortex responder config page group configuration items by there baseConfig. +The Cortex responder config page group configuration items by their `baseConfig`. #### config Configuration dedicated to the responder's flavor. This is where we typically specify the TLP level of observables allowed to be analyzed with the - `check_tlp` and `max_tlp` parameters. For example, if `max_tlp` is set to `2` (TLP:AMBER), - TLP:RED observables cannot be analyzed. + `check_tlp` and `max_tlp` parameters. For example, if `max_tlp` is set to `2` (TLP:AMBER), TLP:RED observables cannot be analyzed. ##### max_tlp The TLP level above which the responder must not be executed. @@ -301,14 +294,14 @@ If the responder **succeeds** (i.e. it runs without any error): - `full` is the full report of the responder. It must contain at least a message. - `operations` is a list what the submitter system should execute. - Currently, TheHive accepts the following operations: + As of version 3.1.0, TheHive accepts the following operations: - `AddTagToCase` (`{ "type": "AddTagToCase", "tag": "tag to add" }`): add a tag to the case related to the object - The list of acceptable operation will increase in future release of TheHive. + + The list of acceptable operations will increase in future releases of TheHive. ### The Cortexutils Python Library -So far, all the published responders have been written in Python. We -released a special Python library called `cortexutils` to help developers easily write their programs. Note though that Python is not mandatory for responder coding and any language that runs on Linux can be used, though you won't have the benefits of the CortexUtils library. +So far, all the published responders have been written in Python. We provide a Python library called `cortexutils` to help developers easily write their programs. Note though that Python is not mandatory for responder coding and any language that runs on Linux can be used, though you won't have the benefits of the CortexUtils library. Cortexutils can be used with Python 2 and 3. To install it : @@ -324,8 +317,7 @@ pip3 install cortexutils ``` This library is already used by all the responders published in our [Github -repository](https://github.com/TheHive-Project/Cortex-Analyzers). Feel free to -start reading the code of some of them before writing your own. +repository](https://github.com/TheHive-Project/Cortex-Analyzers). Feel free to start reading the code of some of them before writing your own. ## Submitting a Responder We **highly encourage you to share your responders** with the community through our Github repository. To do so, we invite you to follow a few steps before submitting a pull request. @@ -387,12 +379,11 @@ If your responder is written in Python, make sure to complete the needed to run the responder correctly. ### Verify Execution -Use these three simple checks before submtting your responder: +Use these three simple checks before submitting your responder: -- Ensure it works with the expecyed configuration, TLP or dataType. -- Ensure it works with missing configuration, dataType or TLP: your +- Ensure it works with the expected configuration, TLP, PAP or datatype. +- Ensure it works with missing configuration, PAP, datatype or TLP: your responder must generate an explicit error message. -- Ensure the long report template handles error messages correctly. ### Create a Pull Request Create one Pull Request per responder against the **develop** branch of the diff --git a/api/how-to-create-an-analyzer.md b/api/how-to-create-an-analyzer.md index 05b0ae955..31d452c11 100644 --- a/api/how-to-create-an-analyzer.md +++ b/api/how-to-create-an-analyzer.md @@ -48,8 +48,7 @@ with the JSON schema expected by [TheHive](https://github.com/TheHive-Project/Th An analyzer must have at least one service interaction file. Such files contain key configuration information such as the analyzer's author information, the datatypes (IP, URL, hash, domain...) the analyzer accepts as -input, the TLP and PAP above which it will refuse to execute to protect against data -leakage and to enforce sane OPSEC practices and so on. +input, the TLP and PAP ([Permissible Actions Protocol](https://www.misp-project.org/taxonomies.html#_pap)) above which it will refuse to execute to protect against data leakage and to enforce sane OPSEC practices and so on. An analyzer can have two or more service interaction files to allow it to perform different actions. We speak then of flavors. For example, a sandbox analyzer can analyze a file with or without an @@ -102,7 +101,7 @@ pieces of information. For example, to use the VirusTotal analyzer's `data`, `dataType` and `tlp` are the observable-related information generated by TheHive or any other program that is calling Cortex. `config` is the -analyzer's specific configuration provided by org admin when the analyzer has been enabled. +analyzer's specific configuration provided by an `orgAdmin` users when the analyzer is enabled in the Cortex UI. Let's take the **GetReport** flavor of the VirusTotal analyzer as an example again. @@ -229,7 +228,7 @@ If you need additional datatypes for your analyzer, please let us know at #### baseConfig Name used to group configuration items common to several analyzer. This prevent the user to enter the same API key for all analyzer flavors. -The Cortex analyzer config page group configuration items by there baseConfig. +The Cortex analyzer config page group configuration items by their `baseConfig`. #### config Configuration dedicated to the analyzer's flavor. This is where we diff --git a/installation/install-guide.md b/installation/install-guide.md index f22a8bc78..590dd3000 100644 --- a/installation/install-guide.md +++ b/installation/install-guide.md @@ -51,7 +51,7 @@ yum install cortex Once the package is installed, [install the analyzers](#analyzers-1) as outlined in the next section and proceed to the configuration using the [Quick Start Guide](../admin/quick-start.md). For more advanced configuration options, please refer to the [Administration Guide](../admin/admin-guide.md). #### Pre-release versions -The RPM release package install two repositories `thehive-project-stable` and `thehive-project-beta`. The latter contains pre-release version and is disabled by default. If you want to install them and help us to find bugs, you can enable the repository by editing `/etc/yum.repos.d/thehive-rpm.repo` and set `enable` value to `1` for `thehive-project-beta` repository. +The RPM release package installs two repositories: `thehive-project-stable` and `thehive-project-beta`. The latter contains pre-release, beta versions and is disabled by default. If you want to install them and help us find bugs to the benefit of the whole community, you can enable it by editing `/etc/yum.repos.d/thehive-rpm.repo` and set `enable` value to `1` for `thehive-project-beta` repository. ### DEB Debian packages are published on a Bintray repository. All packages are signed using our GPG key [562CBC1C](https://raw.githubusercontent.com/TheHive-Project/Cortex/master/PGP-PUBLIC-KEY). Its fingerprint is: @@ -73,7 +73,7 @@ Some environments may block access to the `pgp.mit.edu` key server. As a result, Once the package is installed, [install the analyzers](#analyzers-1) as outlined in the next section and proceed to the configuration using the [Quick Start Guide](../admin/quick-start.md). For more advanced configuration options, please refer to the [Administration Guide](../admin/admin-guide.md). #### Pre-release versions -If you want to install pre-release version of packages and help us to find bugs, you can add the pre-release repository with the command: +If you want to install pre-release, beta versions of TheHive packages and help us find bugs to the benefit of the whole community, you can add the pre-release repository with the command: ```bash echo 'deb https://dl.bintray.com/thehive-project/debian-beta any main' | sudo tee -a /etc/apt/sources.list.d/thehive-project.list ``` @@ -172,6 +172,9 @@ Analyzers are embedded in the docker image under `/opt/Cortex-Analyzers/analyzer #### What to Do Next? Once the Docker image is up and running, proceed to the configuration using the [Quick Start Guide](../admin/quick-start.md). For more advanced configuration options, please refer to the [Administration Guide](../admin/admin-guide.md). +#### Pre-release Versions +If you would like to use pre-release, beta versions of our Docker images and help us find bugs to the benefit of the whole community, please use `thehiveproject/cortex:version-RCx`. For example `thehiveproject/cortex:2.1.0-RC1`. + ### Binary The following section contains the instructions to manually install Cortex using binaries on **Ubuntu 16.04 LTS**. @@ -222,6 +225,8 @@ unzip cortex-latest.zip ln -s cortex-x.x.x cortex ``` +**Note**: if you would like to use pre-release, beta versions of and help us find bugs to the benefit of the whole community, please download `https://dl.bintray.com/thehive-project/binary/cortex-version-RCx.zip`. For example `https://dl.bintray.com/thehive-project/binary/cortex-2.1.0-RC1.zip`. + #### 5. First start It is recommended to use a dedicated, non-privileged user account to start Cortex. If so, make sure that the chosen account can create log files in `/opt/cortex/logs`.