RegexCapture Action No Match

[lmahoney1]

Hello,

I'm running into issues with the RegexCapture action when the pattern supplied does not find a match in the value supplied.

Here's an example I built out for a sanity check:

<?xml version="1.0" encoding="UTF-8" ?>
<Workflow name="GitHub Audit" version="1.0" xmlns="http://qradar.ibm.com/UniversalCloudRESTAPI/Workflow/V1">
    <Actions>
        <Log type="INFO" message="regex no match test" />
        <RegexCapture pattern="NO_MATCH" value="hello world" savePath="/regex_output" />
        <Log type="INFO" message="after regex" />
    </Actions>
</Workflow>

When I run that workflow in our QRadar instance I only see the following entry in the log /var/log/qradar.log:

Feb 22 14:09:47 ::ffff:127.0.0.1 [ecs-ec-ingress.ecs-ec-ingress] [Thread-2225625] com.q1labs.semsources.sources.universalcloudrestapi.v1.workflow.action.LogAction: [INFO] [NOT:0000006000][<IP_ADDR>/- -] [-/- -]regex no match test

So the log statement after the RegexCapture action never runs, which is making me think the RegexCapture action is erroring? Side note: is there a place to see errors that occur when errors in actions happen? Not seeing them in this log file. Had a heck of a time figuring out an issue with the Create JWTAccessToken action where I was experiencing similar.

My actual use case: I'm making requests for events from the GitHub audit log API. If there are multiple pages of events, the link for the next page of events is returned in a 'Link' response header. However, there is other data in this header as well, so I need to parse the URL out with regex to use it in my next request. When I get to the last page of events, the Link header will no longer have the URL for the next page. This is when the regex doesn't match and I run into the error.

Here's an example of the 'Link' header value for the first page of events:

<https://api.github.com/organizations/<ORG_ID>/audit-log?after=MS42NzY5OTI2NjI2MjZlKzEyfGg2WGpEMjNZUW4wRHpEbHZ2WTJ5S0E%3D&before=>; rel="next"

I need to parse https://api.github.com/organizations/<ORG_ID>/audit-log?after=MS42NzY5OTI2NjI2MjZlKzEyfGg2WGpEMjNZUW4wRHpEbHZ2WTJ5S0E%3D&before= out of that header value, which works fine as long as the pattern matches.

Here's an example of the 'Link' header value for the second page of events (more data than just the 'next' URL - 'next', 'first', and 'prev' links):

<https://api.github.com/organizations/<ORG_ID>/audit-log?after=MS42NzY2NjgyMjg1NTNlKzEyfGY1a3ZSYi1CdldlSkxJSDlXZjljYUE%3D&before=>; rel="next", <https://api.github.com/organizations/<ORG_ID>/audit-log?after=&before=>; rel="first", <https://api.github.com/organizations/<ORG_ID>/audit-log?after=&before=MS42NzY5OTAyNTI3NzJlKzEyfEdQNTQwMXc5UThfRkdLaWlTU0pmZ0E%3D>; rel="prev"

Finally, here's an example of the 'Link header for the last page of events (no 'next' URL so regex pattern doesn't match):

<https://api.github.com/organizations/<ORG_ID>/audit-log?after=&before=>; rel="first", <https://api.github.com/organizations/<ORG_ID>/audit-log?after=&before=MS42NzQ0OTM5ODczNzZlKzEyfHBYWGJhX25sOUx2SmRmY3M3QWVVelE%3D>; rel="prev"

I was thinking it might make sense to check if "next" is in the header value before trying to parse the URL with regex, but I'm also struggling to figure out how to do this with any of the JPath functions.

Is the RegexCapture action expected to fail out when the pattern doesn't match? Or am I doing something wrong? Does anyone have any better ideas about how to handle this situation?

Any help would be greatly appreciated!

I'm running this on an instance of QRadar that's on version

7.5.0 UpdatePackage 3 (Build 20220829221022)
with interim fix IF02 applied

I'm not a QRadar expert, so please hold my hand if you notice anything. I typically working within the IBM SOAR platform but I have been tasked with creating a Universal REST API log source.

[lmahoney1]

In case anyone stumbles on this, I ended up splitting the value by commas and then using a regex pattern that was guaranteed to match.

In the context of my issue it would split the 'Link' header so then I would have a list of Link URLs. The Regex pattern would then parse out the 'type' of the link (next, last, first) and iterate over them. if the 'next' link was found the URL is parsed out of it and used in the next request.

I think allowing RegexCapture actions to specify patterns that don't match would be a beneficial feature.

[ChrisCollinsIBM]

The RegexCapture action does require at least one capture group.

https://www.ibm.com/docs/en/dsm?topic=actions-regexcapture

Changing the regex to "(NO MATCH)" instead doesn't abort it with an Exception

com.q1labs.semsources.sources.universalcloudrestapi.v1.workflow.action.SequenceAction] :: SequenceAction.execute(): LogAction, RegexCaptureAction, LogAction
regex no match test
[com.q1labs.semsources.sources.universalcloudrestapi.v1.workflow.action.RegexCaptureAction] :: RegexCaptureAction.execute()
[com.q1labs.semsources.sources.universalcloudrestapi.v1.workflow.WorkflowState] :: set(/regex_output, "", false)
after regex

Vs the output with just "NO MATCH"

[com.q1labs.semsources.sources.universalcloudrestapi.v1.workflow.action.SequenceAction] :: SequenceAction.execute(): LogAction, RegexCaptureAction, LogAction
regex no match test
[com.q1labs.semsources.sources.universalcloudrestapi.v1.workflow.action.RegexCaptureAction] :: RegexCaptureAction.execute()
[com.q1labs.semsources.sources.universalcloudrestapi.v1.workflow.WorkflowRunner] :: java.lang.Exception: The pattern must specified exactly one capture.
[com.q1labs.semsources.sources.universalcloudrestapi.provider.UniversalCloudRESTAPIProvider] :: onWorkflowStatusSet(ERROR, The pattern must specified exactly one capture.) called.

So that might have been the cause of your issue!

So the regex can fail to match, and the value saved to the path will just be empty "" so you can test it with the empty JPAth function to see if it matched or not.

empty(path) | Returns true if the value at a specific path expression is empty.