Feature/SubmissionProfiles (master)

README.md

@@ -1,27 +1,80 @@
-# Assemblyline 4 - Automated malware analysis framework
+[![Discord](https://img.shields.io/badge/chat-on%20discord-7289da.svg?sanitize=true)](https://discord.gg/GUAy9wErNu)
+[![](https://img.shields.io/discord/908084610158714900)](https://discord.gg/GUAy9wErNu)
+[![Static Badge](https://img.shields.io/badge/github-assemblyline-blue?logo=github)](https://github.com/CybercentreCanada/assemblyline)
+[![Static Badge](https://img.shields.io/badge/github-assemblyline--base-blue?logo=github)](https://github.com/CybercentreCanada/assemblyline-base)
+[![GitHub Issues or Pull Requests by label](https://img.shields.io/github/issues/CybercentreCanada/assemblyline/base)](https://github.com/CybercentreCanada/assemblyline/issues?q=is:issue+is:open+label:base)
+[![License](https://img.shields.io/github/license/CybercentreCanada/assemblyline-base)](./LICENCE.md)
 
-AssemblyLine 4 is an open source malware analysis framework. It leverages Kubernetes and Docker to adapt to many use cases; from a small appliance for supporting manual malware analysis and security teams to large-scale enterprise security operations scanning millions of files a day and providing triage capabilities.
+# Assemblyline 4 - Base Package
 
-AssemblyLine can be easily integrated in your environment using it’s powerful rest API and web interfaces. The platform comes with dozens of services to provide deep file analysis and enable integration with other security platforms such as anti-virus, malware-detonation sandboxes and threat knowledge bases. Best of all, with a little bit of Python code you can extend it yourself by creating new analysis and integration services.
+This repository provides Assemblyline with common libraries, cachestore, datastore, filestore, ODM and remote datatypes.
 
-### Repository information
+## Image variants and tags
 
-This is Assemblyline 4 base repository. It provides Assemblyline with common libraries, cachestore, datastore, filestore, ODM and remote datatypes.
+| **Tag Type** | **Description**                                                                                  |      **Example Tag**       |
+| :----------: | :----------------------------------------------------------------------------------------------- | :------------------------: |
+|    latest    | The most recent build (can be unstable).                                                         |          `latest`          |
+|  build_type  | The type of build used. `dev` is the latest unstable build. `stable` is the latest stable build. |     `stable` or `dev`      |
+|    series    | Complete build details, including version and build type: `version.buildType`.                   | `4.5.stable`, `4.5.1.dev3` |
 
-#### System requirements
+## System requirements
 
-Assemblyline 4 will only work on systems running python3.11 and was only tested on linux systems.
+Assemblyline 4 will only work on systems running Python 3.11 and was only officially tested on Linux systems by the Assemblyline team.
 
-#### Installation requirements
+## Installation requirements
 
-If used outside of our normal container this library requires outside linux libraries.
+The following Linux libraries are required for this library:
 
 - libffi8 (dev)
 - libfuxxy2 (dev)
 - libmagic1
 - python3.11 (dev)
 
 Here is an example on how you would get those libraries on a `Ubuntu 20.04+` system:
+```bash
+sudo add-apt-repository ppa:deadsnakes/ppa
+sudo apt install libffi8 libfuzzy2 libmagic1 build-essential libffi-dev python3.11 python3.11-dev python3-pip libfuzzy-dev
+```
 
-    sudo add-apt-repository ppa:deadsnakes/ppa
-    sudo apt install libffi8 libfuzzy2 libmagic1 build-essential libffi-dev python3.11 python3.11-dev python3-pip libfuzzy-dev
+**Note:** Installation of the libraries are not required if using the `cccs/assemblyline` container image
+
+## Documentation
+
+For more information about these Assemblyline components, follow this [overview](https://cybercentrecanada.github.io/assemblyline4_docs/overview/architecture/) of the system's architecture.
+
+# Assemblage 4 - Paquet de base
+
+Ce dépôt fournit à Assemblyline les bibliothèques communes, le cachestore, le datastore, le filestore, l'ODM et les types de données à distance.
+
+## Variantes et étiquettes d'image
+
+| **Type d'étiquette** | **Description**                                                                                                  |  **Exemple d'étiquette**   |
+| :------------------: | :--------------------------------------------------------------------------------------------------------------- | :------------------------: |
+|       dernière       | La version la plus récente (peut être instable).                                                                 |          `latest`          |
+|      build_type      | Le type de compilation utilisé. `dev` est la dernière version instable. `stable` est la dernière version stable. |     `stable` ou `dev`      |
+|        séries        | Le détail de compilation utilisé, incluant la version et le type de compilation : `version.buildType`.           | `4.5.stable`, `4.5.1.dev3` |
+
+## Système requis
+
+Assemblyline 4 ne fonctionnera que sur des systèmes utilisant Python 3.11 et n'a été officiellement testé que sur des systèmes Linux par l'équipe Assemblyline.
+
+## Configuration requise pour l'installation
+
+Les bibliothèques Linux suivantes sont requises pour cette bibliothèque :
+
+- libffi8 (dev)
+- libfuxxy2 (dev)
+- libmagic1
+- python3.11 (dev)
+
+Voici un exemple de la manière dont vous obtiendrez ces bibliothèques sur un système `Ubuntu 20.04+` :
+```bash
+sudo add-apt-repository ppa:deadsnakes/ppa
+sudo apt install libffi8 libfuzzy2 libmagic1 build-essential libffi-dev python3.11 python3.11-dev python3-pip libfuzzy-dev
+```
+
+**Note:** L'installation des bibliothèques n'est pas nécessaire si vous utilisez l'image conteneur `cccs/assemblyline`.
+
+## Documentation
+
+Pour plus d'informations sur ces composants Assemblyline, suivez ce [overview](https://cybercentrecanada.github.io/assemblyline4_docs/overview/architecture/) de l'architecture du système.

assemblyline/common/caching.py

@@ -149,7 +149,7 @@ def keys(self):
             return self.cache.keys()
 
 
-def generate_conf_key(service_tool_version: Optional[str] = None, task: Optional[Task] = None) -> str:
+def generate_conf_key(service_tool_version: Optional[str] = None, task: Optional[Task] = None, partial_result: bool = False) -> str:
     ignore_salt = None
     service_config = None
     submission_params_str = None
@@ -164,7 +164,7 @@ def generate_conf_key(service_tool_version: Optional[str] = None, task: Optional
         }
         submission_params_str = json.dumps(sorted(submission_params.items()))
 
-        if task.ignore_cache:
+        if task.ignore_cache or partial_result:
             ignore_salt = get_random_id()
 
     if service_tool_version is None and \

assemblyline/common/constants.py

@@ -46,21 +46,27 @@ class ServiceStatus(enum.IntEnum):
 DEFAULT_SERVICE_ACCEPTS = ".*"
 DEFAULT_SERVICE_REJECTS = "empty|metadata/.*"
 
+# Priority used to drop tasks in the ingester
+DROP_PRIORITY = 0
+
+# Maximum priority that can be assigned to a submission
+MAX_PRIORITY = 1500
+
 # Queue priority values for each bucket in the ingester
 PRIORITIES = {
-    'low': 100,  # 0 -> 100
-    'medium': 200,  # 101 -> 200
-    'high': 300,
-    'critical': 400,
-    'user-low': 500,
-    'user-medium': 1000,
-    'user-high': 1500
+    'low': 100,                 # 1 -> 100
+    'medium': 200,              # 101 -> 200
+    'high': 300,                # 201 -> 300
+    'critical': 400,            # 301 -> 400
+    'user-low': 500,            # 401 -> 500
+    'user-medium': 1000,        # 501 -> 1000
+    'user-high': MAX_PRIORITY   # 1001 -> 1500
 }
-MAX_PRIORITY = 2000
+
 
 # The above priority values presented as a range for consistency
 PRIORITY_RANGES = {}
-_start = -1
+_start = DROP_PRIORITY
 for _end, _level in sorted((val, key) for key, val in PRIORITIES.items()):
     PRIORITY_RANGES[_level] = (_start + 1, _end)
     _start = _end

assemblyline/common/custom.magic

@@ -156,3 +156,12 @@
 0 string \#\!\/usr\/bin\/python custom: code/python
 0 string \#\!\/usr\/local\/bin\/python custom: code/python
 0 string \#\!\/usr\/bin\/env\ python custom: code/python
+# Pickle files: Starts with specific bytes, and end with a STOP (period ".")
+0 beshort 0x8002
+>-1 byte 0x2e custom: resource/pickle/v2
+0 beshort 0x8003
+>-1 byte 0x2e custom: resource/pickle/v3
+0 string \x80\x04\x95
+>-1 byte 0x2e custom: resource/pickle/v4
+0 string \x80\x05\x95
+>-1 byte 0x2e custom: resource/pickle/v5