[Bug] MISP 2.0 analyzer search crashes the MISP instance

[Kamforka]

Describe the bug
The MISP analyzer is "crashing" the MISP server when pymisp version 2.4.120 is used.

To Reproduce
Steps to reproduce the behavior:

1. Install pymisp client version 2.4.120
2. Run the MISP analyzer with any observable
3. Memory consumption of the MISP server will increase up to a set resource limit if such exists, otherwise just conumes all available system memory (at least I tested it with a 16GB machine and it always ate up all the mem and crashed)

Expected behavior
To run as before.

Work environment

• Client OS: Ubuntu 18.04
• Server OS: Ubuntu 18.04
• Browse type and version:
• Cortex version: 3.0.0-1
• Cortex Analyzer/Responder name: MISP
• Cortex Analyzer/Responder version: 2.0

Possible solutions
As a short term solution it's enough to pin the pymisp requirements to version 2.4.119.
For a long term solution I'd revisit the search queries used by the analyzer and limit them in a sensible way not to make MISP load a lot of resources into memory, although it's a shame that you can crash a system via its ReST API.

Additional context
The analyzer is using the legacy/deprecated client of pymisp which is never a good thing.

[3c7]

Not able to reproduce, is your MISP commit newer than MISP/MISP@f88743c?

[3c7]

Oh. Forget that, somehow mistyped the pip command to update PyMISP on my test machine.

[Kamforka]

Not able to reproduce, is your MISP commit newer than MISP/MISP@f88743c?

I tried with both MISP server version 2.4.119 and 2.4.120.

To be honest this bug should be there for quite some time, but the analyzer is using the legacy interface of pymisp that's why it worked, because the ExpandedPyMISP class was killing MISP with earlier pymisp versions with the same search command.

[3c7]

There were two issues, first the reponse dictionary changed and the search request must use value instead of values as parameter. Fixed in 2.4.0. Can you give the release/2.4.0 branch a shot, if you have the time?

[Kamforka]

Yep I tried with value instead of values already, so I think it should be fine.

Other than that, I'd suggest you to use the new ExpandedPyMISP client instead of the deprecated PyMISP one.

[3c7]

ExpandedPyMISP == PyMISP since 2.4.120.

[3c7]

See https://github.com/MISP/PyMISP/blob/master/pymisp/__init__.py#L37

[Kamforka]

@3c7 I know that but in case someone has an earlier version of pymisp installed on the Cortex machine, then the MISP 2.1 analyzer will be using two different classes in the background, (the legacy) PyMISP for versions <2.4.120 and (the new) ExpandedPyMISP for versions >=2.4.120.

So this search:

pymisp.search(value="192.168.0.1", type_attribute=["ip-src", "ip-dst"])

Will send out the following bodies to MISP:

pymisp < 2.4.120:

{'returnFormat': 'json', 'type': ['ip-src', 'ip-dst'], 'value': '192.168.0.1'}

pymisp >= 2.4.120:

{'category': None,
 'deleted': None,
 'enforceWarninglist': 0,
 'eventid': None,
 'eventinfo': None,
 'from': None,
 'headerless': 0,
 'includeContext': 0,
 'includeCorrelations': 0,
 'includeEventTags': 0,
 'includeEventUuid': 0,
 'includeSightings': 0,
 'limit': None,
 'metadata': 0,
 'org': None,
 'page': None,
 'published': None,
 'quickFilter': None,
 'requested_attributes': None,
 'returnFormat': 'json',
 'searchall': None,
 'sgReferenceOnly': 0,
 'tags': None,
 'to': None,
 'type': ['ip-src', 'ip-dst'],
 'uuid': None,
 'value': '192.168.0.1',
 'withAttachments': 0}

The newer body is basically a superset of the older body, so normally it should cause no problems, but I don't know if all the other body values could cause any inconsistency between the two searches.

[3c7]

That makes sense, thank you.

Edit: Also, because of the response it would crash with an earlier version of PyMISP.