[VulnCheck] Initial Implementation of VulnCheck Connector

[maddawik]

Proposed changes

This is the initial implementation of an external import connector for VulnCheck. It transforms several different data sources from our API, using our python-sdk, into STIX objects/relationships.

Some data sources ingest substantial volumes of data for parsing and generate a significant number of STIX objects and relationships as a result. These resource-intensive data sources have been highlighted in the documentation for user awareness.

Related issues

#3522

Checklist

• I consider the submitted work as finished
• I tested the code for its functionality using different use cases
• I added/update the relevant documentation (either on github or on notion)
• Where necessary I refactored code to improve the overall quality

Further comments

We've made our best effort to make design decisions that are in-line with OpenCTI's plugin architecture - that said, please let me know if there are improvements that can be made!

[maddawik]

I see that isort is failing in CI, however I'm unable to reproduce that locally with the same version of black and isort - please advise!

❯ poetry run black --version
black, 24.10.0 (compiled: yes)
Python (CPython) 3.12.8
❯ poetry run isort --profile black --check src/vclib/config_variables.py --verbose

                 _                 _
                (_) ___  ___  _ __| |_
                | |/ _/ / _ \/ '__  _/
                | |\__ \/\_\/| |  | |_
                |_|\___/\___/\_/   \_/

      isort your imports, so you don't have to.

                    VERSION 5.13.2

else-type place_module for os returned STDLIB
from-type place_module for pathlib returned STDLIB
else-type place_module for yaml returned THIRDPARTY
from-type place_module for pycti returned THIRDPARTY
from-type place_module for vclib.sources.data_source returned FIRSTPARTY
SUCCESS: /Users/user-name/vulncheck/opencti-connector/external-import/vulncheck/src/vclib/config_variables.py Everything Looks Good!
❯ poetry run isort --profile black --check src/vclib/connector_client.py --verbose

                 _                 _
                (_) ___  ___  _ __| |_
                | |/ _/ / _ \/ '__  _/
                | |\__ \/\_\/| |  | |_
                |_|\___/\___/\_/   \_/

      isort your imports, so you don't have to.

                    VERSION 5.13.2

else-type place_module for gzip returned STDLIB
else-type place_module for json returned STDLIB
else-type place_module for os returned STDLIB
else-type place_module for zipfile returned STDLIB
from-type place_module for datetime returned STDLIB
from-type place_module for typing returned STDLIB
else-type place_module for requests returned THIRDPARTY
else-type place_module for vulncheck_sdk returned THIRDPARTY
from-type place_module for pycti returned THIRDPARTY
from-type place_module for pydantic returned THIRDPARTY
from-type place_module for vulncheck_sdk.models.advisory_botnet returned THIRDPARTY
from-type place_module for vulncheck_sdk.models.advisory_ip_intel_record returned THIRDPARTY
from-type place_module for vulncheck_sdk.models.advisory_ransomware_exploit returned THIRDPARTY
from-type place_module for vulncheck_sdk.models.advisory_threat_actor_with_external_objects returned THIRDPARTY
from-type place_module for vulncheck_sdk.models.advisory_vuln_check_kev returned THIRDPARTY
from-type place_module for vulncheck_sdk.models.api_exploit_v3_result returned THIRDPARTY
from-type place_module for vulncheck_sdk.models.api_initial_access returned THIRDPARTY
from-type place_module for vulncheck_sdk.models.api_nvd20_cve returned THIRDPARTY
from-type place_module for vulncheck_sdk.models.api_nvd20_cve_extended returned THIRDPARTY
from-type place_module for vclib.config_variables returned FIRSTPARTY
from-type place_module for vclib.sources returned FIRSTPARTY
SUCCESS: /Users/user-name/vulncheck/opencti-connector/external-import/vulncheck/src/vclib/connector_client.py Everything Looks Good!

[Powlinett]

Hi @maddawik, thank you for your contribution!

I succeed to launch the connector both locally and as a docker image (after a small fix, see my comment) ✅

For the formatting issue, where did you run isort? at the root of the connectors repository? or at the root of vulncheck directory?
If so, please try to run it at the root of the repo and see if it helps 🤞Let us know 😇

Thanks!

[maddawik]

Hi @maddawik, thank you for your contribution!

I succeed to launch the connector both locally and as a docker image (after a small fix, see my comment) ✅

For the formatting issue, where did you run isort? at the root of the connectors repository? or at the root of vulncheck directory? If so, please try to run it at the root of the repo and see if it helps 🤞Let us know 😇

Thanks!

Hey @Powlinett , thank you for the feedback! 🎉 I have made updates per your suggestion and have also pushed some additional commits. I hope you'll forgive me for the additional changes, but I think they will be welcome ones! To make it a little easier to digest I've included a summary of the changes below:

Features

• Updated the CONNECTOR_SCOPE config to use STIX objects for further filtering! 🥳
• Split each data source into separate works for better clarity and transparency on the administrative UI

Fixes

• Reduced the amount of in-memory processing required for parsing large data sources
• Reduced overall cyclomatic complexity for all parsing and gathering functionality
• Applied dependency injection where possible to make testing easier
• Applied Docker best practices to the Dockerfile

I was also able to resolve the formatting issues using pipx to install isort and black versions globally, and then running it against the external-import/vulncheck/ dir.

Please let me know if you have any additional feedback! 😊

[Powlinett]

Looks good to me 👍

• connector works both manually and as a docker image
• didn't see any unexpected errors in logs
• didn't see any errors during ingestion so far

Many thanks 🙌