Encoding error in Shodan results

[amr-cossi]

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	Ubuntu
OS version (client)	16.04
Cortex Analyzer Name	Shodan_Host
Cortex Analyzer Version	1.0
Cortex Version	2.0.4-1
Browser type & version	N/A

Description

If a scan result contains foreign characters, the job fails with an Invalid output error.

Steps to Reproduce

1. Launch job on Analyzer Shodan_Host and IP 211.104.160.152

Possible Solutions

A dirty workaround is to re-encode the result before sending it to report() but a better solution would probably be to handle this case in Cortex back-end.

        self.report(self.encode(results))
[...]
    def encode(self, x):
        if isinstance(x, str):
            return x.encode('utf-8', 'ignore').decode('utf-8', 'ignore')
        elif isinstance(x, dict):
            return {k: self.encode(v) for k, v in x.items()}
        elif isinstance(x, list):
            return [self.encode(k) for k in x]
        else:
            return x

[3c7]

Hey @amr-cossi, is this still an issue? Cannot reproduce it with the IP and Shodan_Host Analyzer.

[nadouani]

Up

[amr-cossi]

Hey,
I can confirm the bug still exists, but as it depends on the output of the query, my example is outdated.
I can reproduce it right now on Cortex 2.1.3 with a call to Shodan_Host_History_1_0 on IP 162[.]220[.]15[.]90.
Invalid output error with the official analyzer, works fine with my encoding hack.

[jeromeleonard]

tried to debug it with @To-om and got this error message :

"errorMessage": "Unexpected Error: 'utf-8' codec can't encode character '\\ud8ec' in position 2952: surrogates not allowed"

Still looking a way to solve it.

[jeromeleonard]

using the workaround waiting for another solution