Allow custom install locations (instead of /opt/intelmq)

[sykaeh]

We would like to install IntelMQ at a different location (not the default /opt/intelmq). Currently, there is no easy way to do this as the path is hard-coded in several places (setup.py and intelmq/__init__.py and in various documentations).

The reason for this is that we would like to allow several developers to have their own instance of IntelMQ on our shared development server.

Is there a reason why the path is hard-coded and it is not configurable? Is there any reason not to make it configurable (I would still keep at it as the default)?

[sebix]

The root path is only in __init__.py, and for installation of data files (configuration etc) in setup.py. In the latter we could import the first one, tough I don't see much advantage in reducing the number of places from two to one.

We could remove the the prefix alltogether and only use the data-path from setuptools: https://docs.python.org/3/distutils/setupscript.html#installing-additional-files

Also note that in packages we only use LSB.

[sebix]

Not to forget the setup.cfg, where the data path is actually given.

[sykaeh]

What does LSB stand for? I am not familiar with that abbreviation.

I think there are two main issues here:

1. Hard-coded names in the code

Remove any hard-coded paths inside the package itself (i.e. inside the intelmq directory) and make them configurable. IMHO, the code should not depend on certain files being in a specific location without being able to change it.

I have used environment variables in intelmq/__init__.py to be able to configure the locations dynamically and so far it seems to be working well:

ROOT_DIR = os.getenv("INTELMQ_ROOT_DIR", "/opt/intelmq/")
DEFAULT_LOGGING_PATH = "/var/log/"

VAR_RUN_PATH = os.getenv("INTELMQ_RUN_DIR", os.path.join(ROOT_DIR, "var/run/"))

CONFIG_DIR = os.getenv("INTELMQ_CONFIG_DIR", os.path.join(ROOT_DIR, "etc/"))
BOTS_FILE = os.getenv("INTELMQ_BOTS_FILE", os.path.join(CONFIG_DIR, "BOTS"))
DEFAULTS_CONF_FILE = os.path.join(CONFIG_DIR, "defaults.conf")
HARMONIZATION_CONF_FILE = os.path.join(CONFIG_DIR, "harmonization.conf")
PIPELINE_CONF_FILE = os.path.join(CONFIG_DIR, "pipeline.conf")
RUNTIME_CONF_FILE = os.path.join(CONFIG_DIR, "runtime.conf")
STARTUP_CONF_FILE = os.path.join(CONFIG_DIR, "startup.conf")
SYSTEM_CONF_FILE = os.path.join(CONFIG_DIR, "system.conf")

Similarly, I also replaced intelmq/bin/intelmqctl.py#L172 with __version__ imported from intelmq. This way it still retrieves the version number automatically, but does not assume that the package name is intelmq (we have some additions that we cannot make public and thus named the package differently so that it is not confused with the official version).

2. Packaging issues

Packaging the code so that it is easy to use, configurable and works on various systems. This concerns setup.py, setup.cfg and possible other files. To be honest, I do not quite understand how the following are related: package_data in setup(), data_files in setup(), MANIFEST.in and setup.cfg.

In our setup we install intelMQ in a virtual environment and use Ansible to install everything. So for our case, I removed all of DATA from setup.py since most of the files there are just examples anyways and we deploy our configuration files with Ansible.

I understand the desire to configure everything with pip (thus copying examples, creating directories), but I also think that things like that depend on the system and are thus should not necessarily part of the Python package. I currently do not know of a good solution and am happy to hear if anyone has any recommendations or best practices.

Other issues

1. (related to hard-coded names in the code) intelmqctl and intelmqdump do not use the logging configuration that is specified in defaults.conf. Is there a reason for this? Can we change it so that they do? Currently it just uses DEFAULT_LOGGING_PATH from intelmq/__init__.py. Otherwise there is no way to change it and you are very likely to run into permission problems if you try to install IntelMQ in a different place from /opt/intelmq.

2. (related to packaging) Why is there a REQUIREMENTS file and why is it not the same as the REQUIRES in setup.py (both the libraries and the versions of the libraries)? In the installation guide it says that the REQUIREMENTS files define a list python packages and versions, which are necessary to run all components of IntelMQ. However that is not true since there are other requirements that are missing (like rt, imbox, stomp).

[sebix]

Two quick answers:

LSB stands for Linux Standard Base, see also #470. What I mean here is that for installations with packages (the RPMs are quite mature, the debs are still unstable) we should not use /opt, but /var/lib/ /run, /etc and so on.

(related to packaging) Why is there a REQUIREMENTS file and why is it not the same as the REQUIRES in setup.py (both the libraries and the versions of the libraries)?

The requires are absolute requirements, you can't go lower than that. The requirements are recommended, higher versions (as they are better tested, more fixes, ...) and include more 3rd party packages needed by bots.

[sebix]

My proposal is to stick to /opt/intelmq in the repository for installations via pypi/pip so we have separated installation for developers environments. For proper installations we packages and use LSB.

1. Hard-coded names in the code

For packaging I can't use ROOT_DIR, I'd need something like prefix and a postfix.

pip installation:

ROOT_DIR_PRE = '/opt/intelmq'
ROOT_DIR_POST = ''
CONFIG_DIR = os.path.join(ROOT_DIR_PRE, "etc", ROOT_DIR_POST)

package installation:

ROOT_DIR_PRE = '/'
ROOT_DIR_POST = 'intelmq/'
CONFIG_DIR = os.path.join(ROOT_DIR_PRE, "etc", ROOT_DIR_POST)

can be combined with your environment variable proposal, but for sake of simplicity I did not use it here.

I also think the BOTS file should not be placed in a configuration directory, but in the library directory.

(related to hard-coded names in the code) intelmqctl and intelmqdump do not use the logging configuration that is specified in defaults.conf.

Which is a bug. However, they should not fail if it's not (properly) defined.

I think I now addressed everything you raised in your post.

[ghost]

It will be possible in 2.2.0 by setting the environment variable INTELMQ_ROOT_DIR