ENH: intelmqsetup: revise installation of manager

requires certtools/intelmq-manager#282
fixes #2064
updates, fixes and re-structures the documentation

debian/rules

@@ -52,7 +52,7 @@ override_dh_auto_install: $(BOTDOCS)
 	# remove program not needed for packages
 	rm debian/intelmq/usr/bin/intelmqsetup
 	# create directory layout and empty state file
-	ROOT_DIR=debian/intelmq/ PYTHONPATH=. python3 intelmq/bin/intelmqsetup.py --skip-ownership --state-file debian/intelmq/var/lib/intelmq/state.json --skip-api
+	ROOT_DIR=debian/intelmq/ PYTHONPATH=. python3 intelmq/bin/intelmqsetup.py --skip-ownership --state-file debian/intelmq/var/lib/intelmq/state.json --skip-api --skip-manager
 
 override_dh_install:
 	dh_install

docs/user/installation.rst

@@ -61,6 +61,7 @@ These are the operating systems which are currently supported by packages:
 Get the installation instructions for your operating system here: `Installation Native Packages <https://software.opensuse.org/download.html?project=home:sebix:intelmq&package=intelmq>`_.
 The instructions show how to add the repository and install the `intelmq` package. You can also install the `intelmq-manager` package to get the `Web-Frontend IntelMQ Manager <https://github.com/certtools/intelmq-manager/>`_.
 
+.. _installation_docker:
 
 Docker
 ------

docs/user/intelmq-manager.rst

@@ -15,28 +15,48 @@ Its goal is to provide an intuitive tool to allow non-programmers to specify the
 Installation
 ************
 
-For the `intelmq-manager` webinterface any operating system that can serve HTML pages is supported.
-`intelmq-manager` can be installed via Python pip or via the operating systems package manager.
-We provide packages for the `intelmq-manager` for the same operating systems as we do for the `intelmq` package itself.
-For the list of supported distributions, please see the IntelMQ :doc:`installation` page.
+To use the `intelmq-manager` webinterface, a working `intelmq` installation which provides access to the :doc:`intelmq-api` is required.
+Please refer to the IntelMQ :doc:`installation` page.
 
-Our repository page gives `installation instructions for various operating systems <https://software.opensuse.org/download.html?project=home:sebix:intelmq&package=intelmq-manager>`_.
-No additional set-up steps are needed if you use these packages.
+`intelmq-manager` can be installed with different methods. Use the same one as you did for IntelMQ itself and the IntelMQ API.
+
+Native Packages
+===============
 
-To use the `intelmq-manager` webinterface, you have to have a working `intelmq` installation which provides access to the :doc:`intelmq-api`.
+As the repositories are already set-up on your system, you can simply install the package ``intelmq-manager``.
+
+Our repository page gives `installation instructions for various operating systems <https://software.opensuse.org/download.html?project=home:sebix:intelmq&package=intelmq-manager>`_.
+No additional set-up steps are needed.
 
-When using distribution packages, the webserver configuration (which is also shown below) for Apache will be automatically installed and the HTML files are stored under ``/usr/share/intelmq-manager/html``.
+The webserver configuration (which is also shown below) for Apache will be automatically installed and the HTML files are stored under ``/usr/share/intelmq-manager/html``.
 The webinterface is then available at ``http://localhost/intelmq-manager``.
 
+Docker
+======
+
+The IntelMQ Manager is included in our Docker-images. See the section :ref:`installation_docker` in our installation guide.
+
+
 Installation using pip
 ======================
 
+
 For installation via pip, the situation is more complex.
-``pip3 install intelmq-manager`` installs the HTML files in ``${PREFIX}/usr/share/intelmq-manager/html``.
-The value of ``${PREFIX}`` depends on your environment and is something like ``/usr/local/lib/pythonX.Y/dist-packages/`` (where ``X.Y`` is your Python version).
-You can either move the files to ``/usr/share/intelmq-manager/html`` or adapt the path in the webserver configuration, see below.
+The intelmq-manager package does not contain ready-to-use files, they need to be built locally.
+First, lets install the Manager itself:
+
+.. code-block::
+   pip3 install intelmq-manager
 
-`intelmq-manager` ships with a default configuration for the Apache webserver (in ``${PREFIX}/etc/intelmq/manager-apache.conf``):
+If your system uses wheel-packages, not the source distribution, you can use the ``intelmqsetup`` tool.
+``intelmqsetup`` which performs these set-up steps automatically but it may not detect all situations correctly.
+If it finds `intelmq-manager` installed, calls its build routine is called.
+The files are placed in ``/usr/share/intelmq_manager/html``, where the default Apache configuration expect it.
+
+If your system used the dist-package or if you are using a local source, the tool may not do all required steps.
+To call the build routine manually, use ``intelmq-manager-build --output-dir your/preferred/output/directory/``.
+
+`intelmq-manager` ships with a default configuration for the Apache webserver (``manager-apache.conf``):
 
 .. code-block::
 
@@ -50,14 +70,10 @@ You can either move the files to ``/usr/share/intelmq-manager/html`` or adapt th
    </Directory>
 
 This file needs to be placed in the correct place for your Apache 2 installation.
- - On Debian and Ubuntu, move the file to ``/etc/apache2/conf-available.d/manager-apache.conf`` and then execute ``a2enconf manager-apache``.
- - On CentOS, RHEL and Fedora, move the file to ``/etc/httpd/conf.d/``.
- - On openSUSE, move the file to ``/etc/apache2/conf.d/``.
-Don't forget to reload your webserver afterwards.
-
-IntelMQ 2.3.1 comes with a tool ``intelmqsetup`` which performs these set-up steps automatically.
-Please note that the tool is very new and may not detect all situations correctly. Please report us any bugs you are observing.
-The tools is idempotent, you can execute it multiple times.
+
+- On Debian and Ubuntu, the file needs to be placed at ``/etc/apache2/conf-available.d/manager-apache.conf`` and then execute ``a2enconf manager-apache``.
+- On CentOS, RHEL and Fedora, the file needs to be placed at ``/etc/httpd/conf.d/`` and reload the webserver.
+- On openSUSE, the file needs to be placed at ``/etc/apache2/conf.d/`` and reload the webserver.
 
 ***********************
 Security considerations

intelmq/bin/intelmqsetup.py

@@ -41,6 +41,13 @@
     import intelmq_manager
 except ImportError:
     intelmq_manager = None
+else:
+    try:
+        import intelmq_manager.build
+    except ImportError:
+        intelmq_manager_has_build = False
+    else:
+        intelmq_manager_has_build = True
 
 from termstyle import red
 from intelmq import (CONFIG_DIR, DEFAULT_LOGGING_PATH, ROOT_DIR, VAR_RUN_PATH,
@@ -255,15 +262,11 @@ def intelmqsetup_api_webserver_configuration(webserver_configuration_directory:
 
 
 def intelmqsetup_manager_webserver_configuration(webserver_configuration_directory: Optional[str] = None):
-    html_dir = Path(pkg_resources.resource_filename('intelmq_manager', '')).parent / '/usr/share/intelmq-manager/html'
-    html_dir_destination = Path('/usr/share/intelmq-manager/html')
-
-    if not html_dir_destination.as_posix().startswith('/usr/'):
-        # Paths differ in editable installations
-        print(red("Detected an editable (egg-link) pip-installation of intelmq-manager. Some feature of this program may not work."))
-
     webserver_configuration_dir = webserver_configuration_directory or find_webserver_configuration_directory()
-    manager_config = Path(pkg_resources.resource_filename('intelmq_manager', '')).parent / 'etc/intelmq/manager-apache.conf'
+    manager_config_1 = Path(pkg_resources.resource_filename('intelmq_manager', '')).parent / 'etc/intelmq/manager-apache.conf'
+    # IntelMQ Manager >= 3.1.0
+    manager_config_2 = Path(pkg_resources.resource_filename('intelmq_manager', '')) / 'manager-apache.conf'
+    manager_config = manager_config_2 if manager_config_2.exists() else manager_config_1
     apache_manager_config = webserver_configuration_dir / 'manager-apache.conf'
     if manager_config.exists() and not apache_manager_config.exists():
         shutil.copy(manager_config, apache_manager_config)
@@ -273,15 +276,22 @@ def intelmqsetup_manager_webserver_configuration(webserver_configuration_directo
         global NOTE_WEBSERVER_RELOAD
         NOTE_WEBSERVER_RELOAD = True
     elif not manager_config.exists() and not apache_manager_config.exists():
-        print(red(f'Unable to install webserver configuration manager-config.conf: Neither {manager_config!s} nor {apache_manager_config!s} exists.'))
+        print(red(f'Unable to install webserver configuration manager-config.conf: Neither {manager_config_1!s} nor {manager_config_2!s} nor {apache_manager_config!s} exist.'))
 
-    if html_dir.exists():
-        try:
-            shutil.copy(html_dir, '/')
-        except Exception as exc:
-            print(red(f"Unable to copy {html_dir} to {html_dir_destination}: {exc!s}"))
-        else:
-            print(f'Copied {html_dir!s} to {html_dir_destination!s}.')
+
+def intelmqsetup_manager_generate():
+    if not intelmq_manager_has_build:
+        print('Unable to build intelmq-manager files. Installed version of intelmq-manager is too old, at least version 3.1.0 is required.',
+              file=sys.stderr)
+        return
+    src_dir = Path(pkg_resources.resource_filename('intelmq_manager', ''))
+    html_dir_destination = Path('/usr/share/intelmq_manager/html')
+
+    if not src_dir.as_posix().startswith('/usr/'):
+        # Paths differ in editable installations
+        print(red("Detected an editable (egg-link) pip-installation of intelmq-manager. Some feature of this program may not work."))
+
+    intelmq_manager.build.buildhtml(html_dir_destination)
 
 
 def main():
@@ -299,24 +309,35 @@ def main():
     parser.add_argument('--skip-api',
                         help='Skip set-up of intelmq-api.',
                         action='store_true')
+    parser.add_argument('--skip-webserver',
+                        help='Skip all operations on the webserver configuration, affects the API and Manager.',
+                        action='store_true')
+    parser.add_argument('--skip-manager',
+                        help='Skip set-up of intelmq-manager.',
+                        action='store_true')
     args = parser.parse_args()
 
     basic_checks(skip_ownership=args.skip_ownership)
     intelmqsetup_core(ownership=not args.skip_ownership,
                       state_file=args.state_file)
     if intelmq_api and not args.skip_api:
-        print('Running setup for intelmq-api.')
+        print(f'Running setup for intelmq-api (version {intelmq_api.version}).')
         intelmqsetup_api(ownership=not args.skip_ownership,
                          webserver_user=args.webserver_user)
-        print('Running webserver setup for intelmq-api.')
-        intelmqsetup_api_webserver_configuration(webserver_configuration_directory=args.webserver_configuration_directory)
+        if not args.skip_webserver:
+            print('Running webserver setup for intelmq-api.')
+            intelmqsetup_api_webserver_configuration(webserver_configuration_directory=args.webserver_configuration_directory)
     else:
         print('Skipping set-up of intelmq-api.')
-    if intelmq_manager and not args.skip_api:
+    if intelmq_manager and not args.skip_manager and not args.skip_webserver:
         print('Running webserver setup for intelmq-manager.')
         intelmqsetup_manager_webserver_configuration(webserver_configuration_directory=args.webserver_configuration_directory)
     else:
-        print('Skipping set-up of intelmq-manager.')
+        print('Skipping intelmq-manager configuration.')
+    if intelmq_manager and not args.skip_manager:
+        manager_version = pkg_resources.get_distribution('intelmq-manager').version
+        print(f'Generate and save intelmq-manager (version {manager_version}) static files.')
+        intelmqsetup_manager_generate()
 
     if NOTE_WEBSERVER_RELOAD:
         print('Reload the webserver to make the changes effective.')