New analyzer to query PhishTank for a URL

[ecapuano]

Another simple one that queries URL item types to determine if they are in the PhishTank database yet.

If a URL is already in the PhishTank database, we add a short report to the artifact title.

If it is not, we do not add anything to the artifact title.

Example long report of URL found in PhishTank

---

Great example of a false negative where a true phishing site is not in the PhishTank database yet. Context clues come from the URLQuery category validating the site as `Phishing`.

( Notice that there is no PhishTank short report on the title because the site is not currently in the PhishTank database, see long report below )

Also notice that if a site is not already listed on PhishTank, an option appears to Click here to submit this site to PhishTank which generates a pre-written email to PhishTank to report the phishing site.

---

We're even trying to account for unknowns and error situations by passing through the API error text in the event of a bad API call... This great example came from an analyst accidentally inputting an already defanged URL into TheHive (which is obviously unnecessary.)

[ecapuano]

Ah, forgot to mention.... This depends on obtaining a free API key from PhishTank and adding it to /etc/thehive/application.conf

analyzer {
  # Directory that holds analyzers
  path = analyzers
  # Analyzer configuration
  config {
    PhishTank_CheckURL {
        key="`<api key>`"
    }
  }
}

Not sure the best way to officially "add" that to the project... Please let me know what method you guys prefer.

[jeromeleonard]

Hi,
Thank you for this PR.
I studied this analyzer. First of all, the python code is OK and I have nothing to tell about it, it is doing the job.

Speaking now about PhishTank reports. Tell me if i'm wrong but, looking at the website:

• a URL can be submitted,
• the submitted URL can be verified or validated by other users who confirm that it is a phishing site. Otherwise it may be considered as false positive,
• the reported site is online, offline or its status is unknown.

I made some tests and compared the results obtained from the website to those produced by the API. I found that the API returns in_database:false unless a site:

• is in the database
• and confirmed as a phishing site
• and online.

For example, an API request for a recently submitted site that has not been verified returns "in_database:false" whilst the website shows it as present in the DB.

I think all these information are interesting:

• a phishing site may be in the database and not validated, so you can confirm that through TheHive (with the submission link you have added for ex.),
• A phishing site can now be offline (taken down for ex.), so the incident response may not be the same.

However the API does not provide the answers to such questions as we cannot know if a site has already been submitted (and not verified). Can you confirm that behavior?

Thus, we suggest to change some wordings in the long report and temper results "in_database:false" by writing "absent or not verified".

[ecapuano]

Very good feedback! I will take a look and make some modifications. Let's sit on this one a bit because I also need to push a fix to the short report for this one as well.

This line permits an empty short title in a situation when the short title should not exist.
https://github.com/CERT-BDF/TheHive/pull/27/files#diff-3a96ed88f7c911918c0b8c4597b382d6R3

[mthlvt]

Looking forward to this merge, any news? Thanks

[nadouani]

Hello @mthlvt yes, this will be released within TheHive 2.10.0 in the next fews days

[nadouani]

This PR will be merged manually into another repository that holds the new service responsible of running analyzers. This will be available in TheHive 2.10.0