Assigning a dictionary to the "extra" field does not handle missing keys or empty values correctly

[bernhard-herzog]

IntelMQ 1.1 changed how the "extra" field works but tries to provide a
backwards compatible interface. In particular, it's still possible to
assign a new dictionary to "extra" with add method with
overwrite=True. There are at least two cases that are not handled
correctly when doing this:

• Keys not present in the new dictionary that were present before are
  not removed

• Keys with empty values, e.g. empty dictionaries, are ignored and
  therefore are not changed.

Example session:

>>> import intelmq.lib.message
>>> m = intelmq.lib.message.Event()
>>> m["extra"] = {"a": {"x": 1}, "b": "foo"}
>>> m["extra"]
'{"b": "foo", "a": {"x": 1}}'

so far, so good.

>>> m.add("extra", {"a": {}}, overwrite=True)
True
>>> m["extra"]
'{"b": "foo", "a": {"x": 1}}'

I would have expected this to return {"a": {}}

[ghost]

So you need to keep empty values (empty dictionaries)? The current behavior is to ignore them (same as for all non-extra fields).

[bernhard-herzog]

So you need to keep empty values (empty dictionaries)?

Not necessarily. Removing the item altogether would also be OK for our purposes, so that assigning {"a": {}}' would be equivalent to assigning {}`

The current behavior is to ignore them (same as for all non-extra fields).

This is inconsistent with direct assignments to "extra.foo":

>>> import intelmq.lib.message
>>> m = intelmq.lib.message.Event()
>>> m["extra"] = {"a": {}, "b": 123}
>>> m["extra"]
'{"b": 123}'
>>> m["extra.a"] = {}
>>> m["extra"]
'{"b": 123, "a": {}}'

If this inconsistency is due to backwards compatibility it's best to keep that behavior, of course.

[ghost]

Pre 1.1 there were no restriction on anything inside extra, so I agree that we shouldn't ignore empty dicts there. Will create a patch for it.