Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] mispwarninglist update errors when using database backend #890

Closed
hoboristi opened this issue Nov 4, 2020 · 1 comment
Closed
Assignees
Labels
category:bug Issue is related to a bug scope:analyzer Issue is analyzer related

Comments

@hoboristi
Copy link

Describe the bug
In case of updating lists, a lot of list reports errors

To Reproduce
Steps to reproduce the behavior:

  1. I've set cronjob according to update postgres database, which is mentioned on https://blog.thehive-project.org/ MISPWarningLists reloaded (need for speed aka DB support) section
  2. In case of updates - when executing /usr/bin/python3 /MISPWarningLists/warninglists_create_db.py, it throws a lot of errors, like that (full list has ~ 1600 lines):
  • ERROR:root:NOT VALID: google.za [google]
  • ERROR:root:NOT VALID: com.home [cisco_top5k]
  • ERROR:root:NOT VALID: google.internal [cisco_top5k]
  • ERROR:root:NOT VALID: home [cisco_top5k]
  • ERROR:root:NOT VALID: TJMAXX [tlds]
  • ERROR:root:NOT VALID: TJX [tlds]
  • ERROR:root:NOT VALID: TK [tlds]
  • ERROR:root:NOT VALID: XN--90AE [tlds]
  • ERROR:root:NOT VALID: XN--90AIS [tlds]
  • ERROR:root:NOT VALID: XN--9DBQ2A [tlds]
  • ERROR:root:NOT VALID: XN--9ET52U [tlds]
  • ERROR:root:DB already exists

Expected behavior
process new warninglists to database without errors
Complementary information

Work environment

  • Client OS: n/A
  • Server OS: Ubuntu 18.04
  • Browse type and version: n/A
  • Cortex version: n/A
  • Cortex Analyzer/Responder name: MISPWarningLists
  • Cortex Analyzer/Responder version: 2.0

Possible solutions

Additional context

@hoboristi hoboristi added the category:bug Issue is related to a bug label Nov 4, 2020
@dadokkio
Copy link
Contributor

dadokkio commented Nov 4, 2020

Hello @hoboristi , the script is working properly.. is just the log that is very verbose and creates confusion.
There are a lot of items that we don't manage (eg. tlds) and other urls that are not valid because their tlds are not really valid [.lan, .home, etc...]. Each urls is checked with tld library that for them returns error and I show them in the log.

The interesting part is at the end:
image
On the right the first run, on the left one update.

Probably I could have done a better logging. I'll try to fix it.

@dadokkio dadokkio self-assigned this Nov 4, 2020
@jeromeleonard jeromeleonard added the scope:analyzer Issue is analyzer related label Nov 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category:bug Issue is related to a bug scope:analyzer Issue is analyzer related
Projects
None yet
Development

No branches or pull requests

3 participants