Script which can convert existing bots in the group collector to Systemd services #953
Conversation
Codecov Report
@@ Coverage Diff @@
## master #953 +/- ##
=========================================
Coverage ? 77.98%
=========================================
Files ? 221
Lines ? 9047
Branches ? 0
=========================================
Hits ? 7055
Misses ? 1992
Partials ? 0Continue to review full report at Codecov.
|
contrib/systemd/conf.py
Outdated
| import shutil | ||
|
|
||
| INTELMQ_DIR = '/opt/intelmq' | ||
| RUNTIME_CONF = INTELMQ_DIR+'/etc/runtime.conf' |
contrib/systemd/conf.py
Outdated
|
|
||
| INTELMQ_DIR = '/opt/intelmq' | ||
| RUNTIME_CONF = INTELMQ_DIR+'/etc/runtime.conf' | ||
| PIPELINE_CONF = INTELMQ_DIR+'/etc/pipeline.conf' |
There was a problem hiding this comment.
I did not imported intelmq, perhaps will be a good idea to do that.
ghost
self-requested a review
|
This needs a lot more testing. I have tested it only on one OS. Will be great if more folks can test it out on other systems. |
contrib/systemd/systemd.py
Outdated
| # Return only the bots which are directly connected | ||
| # If a bot has more than one inputs or outputs stop processing | ||
| # | ||
| def connected_bots(bot,rc_data,pipe_data): |
There was a problem hiding this comment.
I know. Not as of now, I am still trying to make the directly connected units (as explained in mailing list) work as connected systemd scripts. So can we please keep these? Unfortunately i cannot commit to any timeline, atleast this week looks pretty busy.
contrib/systemd/conf.py
Outdated
| DISABLE_IN_CONF = True | ||
| SET_RUNMODE_IN_CONF = True | ||
| INTELMQCTL_BIN = shutil.which('intelmqctl') | ||
| SYSTEMCTL_BIN = shutil.which('systemctl') |
There was a problem hiding this comment.
SYSTEMCTL_BIN is the only one which is never used, however it was used for making other units trigger after the collector.
contrib/systemd/systemd.py
Outdated
| tmr_file.write(timer_data) | ||
|
|
||
| if DISABLE_IN_CONF or SET_RUNMODE_IN_CONF: | ||
| shutil.move(RUNTIME_CONF, RUNTIME_CONF+'.bak') |
There was a problem hiding this comment.
if you copy here, you do not need to fix the permissions afterwards
|
I pushed a commit with a bunch of style fixes |
| shutil.move(RUNTIME_CONF_FILE, RUNTIME_CONF_FILE + '.bak') | ||
| rc_data = collections.OrderedDict(sorted(rc_data.items())) | ||
| data = json.dumps(rc_data, indent=4) | ||
| with open(RUNTIME_CONF_FILE, "w", encoding='utf-8') as rc_file: |
There was a problem hiding this comment.
You can reuse intelmqctl's write_updated_runtime_config here.
|
You are assuming that all collectors should be scheduled, but stream collectors (http stream, n6, xmpp, alienvault otx) can't run in this mode properly. |
|
I understand, however , I am unable to find a good solution to it, couple of things I can think of
2 can then have web UI page to automate the work. |
|
partially addresses the problem of selectively scheduling the bots. Unless there is a way to figure out which bots can be scheduled, there doesn't seems to be any decent way out of it. |
|
Could you reduce the number of tiny commits here (these "wip" and similar) please? Then I'll merge this and afterwards open a PR adding support for continuous collectors and some other fixes. |
|
Thanks. Sorry for be being too late, I have been pulled into too many things. |
#metoo ;) Cleaning up now to get the next feature release ready :) |
ghost
mentioned this pull request
Title says it all. However this is a hackish script. For those who know what they are doing.