[Import Document AI] Implement Import Document AI connector #3466

Cnstant · 2025-02-21T16:08:27Z

Proposed changes

implement a connector that can query an internal service extractor with AI to extract new entities from text

Related issues

[import-document-ai] Create the connector #3457

Checklist

I consider the submitted work as finished
I tested the code for its functionality using different use cases (integration tests were performed manually)
I added/update the relevant documentation (in the README)
Where necessary I refactored code to improve the overall quality

Further comments

flavienSindou

@Cnstant Thanks for your work!

Your contribution follows the current connector template, and the implementation and documentation are clear.

Could you havea look at the comments on example files, error handling, and file storage, please?

I also left a few (optional) suggestions to make the logging tools and message formatting more consistent.

It'd be nice if you could provide a fake web service respons json example, I’d then be able to quickly test it by simulating an API server.

flavienSindou · 2025-03-04T10:46:42Z

internal-import-file/import-document-ai/src/requirements.txt

I suggest you add explicitly used package and do not only rely on pycti sub dependencies

requests
stix2
yaml

what about versions ? should I stick to the versions in the package, or no version at all to make it more flexible ?

Thank you for your response

As the end user of the library, I think you must specify the exact (compatible) versions you need/tested with your use case. But as you pointed out, pycti currently does not provide loose version requirements for its dependencies. This is something we are correcting step by step. (see OpenCTI-Platform/client-python#859)

I think you can fix your versions to :

requests==2.32.3
stix2==3.0.1
PyYAML==6.0.2

internal-import-file/import-document-ai/src/reportimporter/core.py

internal-import-file/import-document-ai/docker-compose.yml.sample

internal-import-file/import-document-ai/src/reportimporter/core.py

Cnstant · 2025-03-05T15:12:00Z

Here are all the possible json responses at the moment. You have the return status in the file name, and the content is the actual response.json(). Let me know if it's enough
response_403_trial.json
response_400_missing.json
response_403_unauthorized.json
response_400_invalid.json
response_202.json

flavienSindou

Please find my last remarks. Thank you for the response examples you provide, I get back to you as soon as I have tested them.

flavienSindou · 2025-03-05T17:53:45Z

internal-import-file/import-document-ai/docker-compose.yml.sample

+version: '3'
+services:
+  connector-import-document-ai:
+    image: connector-template:6.3.3


#3466 (comment)

Sorry i was not clear in my previous comment. I think you should replace connector-template with the future image name

Suggested change

image: connector-template:6.3.3

image: opencti/connector-import-document-ai:6.3.3

flavienSindou · 2025-03-05T17:59:51Z

internal-import-file/import-document-ai/src/reportimporter/core.py

+        # On Windows, the invalid characters are different, so the behavior is not the same as Linux
+        # It only happens with free text on local setup running on windows. Never on prod.
+
+        os_system = os.name
+
+        # If windows detection, replacement of invalid characters
+        if os_system == "nt":
+            file_name = re.sub(r'[\\/:*?"<>|]', "_", file_name)


I think this is not necessary and you could remove it as you are not saving the file in the filesystem anymore.

Suggested change

# On Windows, the invalid characters are different, so the behavior is not the same as Linux

# It only happens with free text on local setup running on windows. Never on prod.

os_system = os.name

# If windows detection, replacement of invalid characters

if os_system == "nt":

file_name = re.sub(r'[\\/:*?"<>|]', "_", file_name)

flavienSindou · 2025-03-05T18:10:40Z

internal-import-file/import-document-ai/src/requirements.txt

Thank you for your response

As the end user of the library, I think you must specify the exact (compatible) versions you need/tested with your use case. But as you pointed out, pycti currently does not provide loose version requirements for its dependencies. This is something we are correcting step by step. (see OpenCTI-Platform/client-python#859)

I think you can fix your versions to :

requests==2.32.3
stix2==3.0.1
PyYAML==6.0.2

flavienSindou · 2025-03-05T19:25:04Z

internal-import-file/import-document-ai/src/reportimporter/core.py

+                    for entity_id in entities_ids:
+                        # Incident attributed-to Threats
+                        if (
+                            entity_id.startswith("threat-actor")


Note: Seems unused for now as the AI WebService does not extract threat actors

flavienSindou · 2025-03-05T19:25:58Z

internal-import-file/import-document-ai/src/reportimporter/core.py

+                        if (
+                            entity_id.startswith("threat-actor")
+                            or entity_id.startswith("intrusion-set")
+                            or entity_id.startswith("campaign")


Note: Seems unused for now as the AI WebService does not extract campaign

flavienSindou · 2025-03-05T19:31:11Z

internal-import-file/import-document-ai/README.md

Would you say it is correct to represent connector behavior like this:
Case 1 : from Data/Import UI

graph LR subgraph FileStorage direction LR pdf["PDF file"] text["Text file"] html["HTML file"] md["MD file"] end subgraph AIWebService end subgraph OpenCTI direction LR subgraph CreatedReport subgraph AddedDomainObjects direction LR OpenCTICountry[Country] OpenCTIMalware[Malware] OpenCTIIntrusionSet[Intrusion Set] OpenCTIVulnerability[Vulnerability] OpenCTIAttackPattern[Attack Pattern] end subgraph AddedObservables direction LR OpenCTIAutonomousSystem[Autonomous System] OpenCTIDomainName[Domain Name] OpenCTIEmailAddress[Email Address] OpenCTIPv4Address[IP V4 Address] OpenCTIPv6Address[IP V6 Address] OpenCTIMACAddress[MAC Address] OpenCTIURL[URL] OpenCTIFile[File] OpenCTIWindowsRegistryKey[Windows Registry Key] end end end FileStorage ==> AIWebService AIWebService ==> AddedDomainObjects & AddedObservables

Loading

Case2 : From Container UI (Report/Task/Case-Incidents, etc)

graph LR subgraph FileStorage end subgraph AIWebService end subgraph OpenCTI direction LR subgraph TriggeringContainer[Triggering Container] subgraph AddedDomainObjects[Added Domain Objects] end subgraph AddedObservables[Added Observables] end end end TriggeringContainer -->|Triggers| FileStorage FileStorage ==> AIWebService AIWebService ==> AddedDomainObjects & AddedObservables

Loading

Case 3 : from an OpenCTI Entity UI with Incident or Threat Actor special sub cases

graph LR subgraph FileStorage end subgraph AIWebService end subgraph OpenCTI direction LR subgraph TriggeringEntity[Triggering Entity] TriggeringIncident[Incident] TriggeringThreatActor[Threat Actor] TriggeringOthers["..."] end subgraph AddedDomainObjects[Added Domain Objects] OpenCTIIntrusionSet[Intrusion Set] OpenCTIVulnerability[Vulnerability] OpenCTIAttackPattern[Attack Pattern] OpenCTIOthers["..."] end subgraph AddedObservables[Added Observables] end end TriggeringEntity -.->|Triggers| FileStorage FileStorage ==> AIWebService AIWebService ==> AddedDomainObjects & AddedObservables AddedObservables -->|Related To| TriggeringEntity TriggeringIncident-->|Attributed To| OpenCTIIntrusionSet TriggeringIncident & TriggeringThreatActor -->|Targets| OpenCTIVulnerability TriggeringIncident & TriggeringThreatActor -->|Uses| OpenCTIAttackPattern

Loading

flavienSindou

I tested the connector with this fake rest server and everything worked fine

dev.zip

I'll be happy to approve your PR as soon as you address the previous remarks.

flavienSindou · 2025-03-06T12:23:36Z

internal-import-file/import-document-ai/src/config.yml.sample

+  validate_before_import: true 
+  log_level: 'info'
+  web_service_url: 'http://localhost:PORT'
+  licence_key_pem:|


Suggested change

licence_key_pem:|

licence_key_pem: |

Cnstant linked an issue Feb 21, 2025 that may be closed by this pull request

[import-document-ai] Create the connector #3457

Open

Cnstant force-pushed the issue/3457 branch 2 times, most recently from 33b3b50 to 090abc4 Compare February 21, 2025 16:17

helene-nguyen changed the title ~~feat: Implement Import Document AI connector~~ [Import Document AI] Implement Import Document AI connector Feb 21, 2025

helene-nguyen assigned Cnstant Feb 21, 2025

helene-nguyen added the filigran team use to identify PR from the Filigran team label Feb 21, 2025

flavienSindou self-requested a review February 24, 2025 13:01

Cnstant force-pushed the issue/3457 branch 4 times, most recently from 83e3d62 to b6c2782 Compare February 24, 2025 21:00

feat: Implement Import Document AI connector

7f899f6

Cnstant force-pushed the issue/3457 branch from b6c2782 to 7f899f6 Compare February 24, 2025 21:18

[connector] Ensure formatting

e3e42b0

Cnstant force-pushed the issue/3457 branch 2 times, most recently from 410503d to de91597 Compare February 28, 2025 13:59

feat: Add instance id in header

da793af

Cnstant force-pushed the issue/3457 branch from de91597 to da793af Compare February 28, 2025 14:21

flavienSindou requested changes Mar 4, 2025

View reviewed changes

Cnstant force-pushed the issue/3457 branch from c54b50d to 356de34 Compare March 5, 2025 15:14

review: Implement review feedbacks from @flavienSindou

0ec9850

Cnstant force-pushed the issue/3457 branch from 356de34 to 0ec9850 Compare March 5, 2025 15:17

Cnstant requested a review from flavienSindou March 5, 2025 15:18

flavienSindou requested changes Mar 5, 2025

View reviewed changes

flavienSindou requested changes Mar 6, 2025

View reviewed changes

flavienSindou reviewed Mar 6, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Import Document AI] Implement Import Document AI connector #3466

[Import Document AI] Implement Import Document AI connector #3466

Cnstant commented Feb 21, 2025 •

edited

Loading

flavienSindou left a comment •

edited

Loading

flavienSindou Mar 4, 2025 •

edited

Loading

Cnstant Mar 5, 2025

flavienSindou Mar 5, 2025 •

edited

Loading

Cnstant commented Mar 5, 2025

flavienSindou left a comment

flavienSindou Mar 5, 2025

flavienSindou Mar 5, 2025 •

edited

Loading

flavienSindou Mar 5, 2025 •

edited

Loading

flavienSindou Mar 5, 2025

flavienSindou Mar 5, 2025

flavienSindou Mar 5, 2025

flavienSindou left a comment •

edited

Loading

flavienSindou Mar 6, 2025

	image: connector-template:6.3.3
	image: opencti/connector-import-document-ai:6.3.3

[Import Document AI] Implement Import Document AI connector #3466

Are you sure you want to change the base?

[Import Document AI] Implement Import Document AI connector #3466

Conversation

Cnstant commented Feb 21, 2025 • edited Loading

Proposed changes

Related issues

Checklist

Further comments

flavienSindou left a comment • edited Loading

Choose a reason for hiding this comment

flavienSindou Mar 4, 2025 • edited Loading

Choose a reason for hiding this comment

Cnstant Mar 5, 2025

Choose a reason for hiding this comment

flavienSindou Mar 5, 2025 • edited Loading

Choose a reason for hiding this comment

Cnstant commented Mar 5, 2025

flavienSindou left a comment

Choose a reason for hiding this comment

flavienSindou Mar 5, 2025

Choose a reason for hiding this comment

flavienSindou Mar 5, 2025 • edited Loading

Choose a reason for hiding this comment

flavienSindou Mar 5, 2025 • edited Loading

Choose a reason for hiding this comment

flavienSindou Mar 5, 2025

Choose a reason for hiding this comment

flavienSindou Mar 5, 2025

Choose a reason for hiding this comment

flavienSindou Mar 5, 2025

Choose a reason for hiding this comment

flavienSindou left a comment • edited Loading

Choose a reason for hiding this comment

flavienSindou Mar 6, 2025

Choose a reason for hiding this comment

Cnstant commented Feb 21, 2025 •

edited

Loading

flavienSindou left a comment •

edited

Loading

flavienSindou Mar 4, 2025 •

edited

Loading

flavienSindou Mar 5, 2025 •

edited

Loading

flavienSindou Mar 5, 2025 •

edited

Loading

flavienSindou Mar 5, 2025 •

edited

Loading

flavienSindou left a comment •

edited

Loading