Make IntelMQ plugable

[dmth]

We've already discussed this some times in the dev - calls but I couldn't find an issue which states this demand.

From my point of view IntelMQ will get better when using a plug-able structure, allowing to add own plug-ins with more ease.

I try to start anew with a collection of already existing Issues and code which are / is covering this topic.

• Is Enable separate packaging of bots by allowing addition and removals to the config. #552
• Is Where bot configs should be stored in repo? #368
• Is Clearly define all parameters used in a bot #757
• Is Very long BOTS file #668
• Replace-Bots branch of @sykaeh https://github.com/sykaeh/intelmq/tree/replace-bots Clearly define all parameters used in a bot #757 (comment)
• ML http://lists.cert.at/pipermail/intelmq-dev/2016-April/000033.html

I understood that we've already reached some kind of consensus that this structure is something we want to pursue. So let's just start to discuss and collect what's required to achieve this goal.

Maybe we can start by collecting some, even fictional, requirements:

• As an operator of an IntelMQ instance, I'd like to add my own code for bots which processes internal information without the need of manually editing the bots file.
• As an admin I want to install IntelMQ without the need of installing libraries which are not required for my use-case. Code for bots that is not necessary should not exist on my systems.

please continue...

[gethvi]

As an admin I want to install IntelMQ without the need of installing libraries which are not required for my use-case. Code for bots that is not necessary should not exist on my systems.

I have a suggestion for this. We could use setuptools with extras for installing bots with specific dependencies like this for example:

pip3 install intelmq[maxmind_geoip]

More:
https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies

@wagner-certat Do you think this would be sensible approach?

[ghost]

I thought about this approach for the contrib-tools and for the optional dependencies, e.g.

• intelmq[contrib]: installs all contrib tools
• intelmq: like current
• intelmq[all]: Installs all dependencies of bots

If you also want to do that per bot, we can have a minimal core installation (no bots, just libs) + one package per bot. But keep in mind, that this has a few dependencies on other issues solved first or together and the work needed for adding new bots should - all in all - be about the same as currently.