Mitre import slow

[b4dpxl]

Description

Fresh install from Docker, running on a VM with 6 cores and 16GB RAM. No changes to memory configuration. At first start, the Mitre connector loads in ~5,750 messages. In ~85 hours, less than 5000 messages have been processed. Every restart also loads an additional ~6k messages onto the queue.

Environment

1. OS (where OpenCTI server runs): Digital Ocean Ubuntu Docker Appliance
2. OpenCTI version: latest using docker-compose from GitHub
3. OpenCTI client: frontend
4. Other environment details: 16GB RAM, 6 cores

[b4dpxl]

Is it possible to run more than one worker-import in a docker-compose deployment?

[richard-julien]

Hi @b4dpxl , yes you can scale the number of workers to improve the speed.
For the redo everything each time, its a current limitation we need to improve. See OpenCTI-Platform/connectors#14

[b4dpxl]

Thanks @richard-julien. Is it just a case of duplicating the worker-import section in docker-compose.yml to add additional workers?

It also seemed to be taking ~30 seconds (if I'm reading the logs right) to process each mitre message. Is that normal? It took roughly 5 days to import the full mitre list with a single worker.

[richard-julien]

Hi @b4dpxl sorry for the delay. Yes you can duplicate the worker import or better use the compatibility mode. docker-compose --compatibility up. It will bootstrap 4 workers according to the latest version of https://github.com/OpenCTI-Platform/docker/blob/master/docker-compose.yml

For the speed it can take times yes because:

• One message have multiple objects to create
• We also need to check every object before the creation to know what to do

Maybe you could add more memory/run on a SSD your grakn database?

[b4dpxl]

Thanks @richard-julien, I'll try compatibility. The server has 16gb and is using SSD, so we could assign more memory to Grakn, but it didn't seem memory or I/O limited. The limit seemed to be the CPU usage on the Grakn processes.

[richard-julien]

Hum I also forget but in version 1.1.1 we introduce a performance regression. #185. It will be faster after the release of the 1.1.2

[b4dpxl]

cool, thanks again @richard-julien. I'm really liking what I've seen of this project so far :)