[RSS Feed] Error 403 on accessible public feeds

[Lhorus6]

Description

Some RSS feeds return errors 403 even though the links are public and accessible.

Examples:

• https://dailydarkweb.net/feed/
• https://cybersecurity.att.com/site/blog-all-rss
• https://www.securityweek.com/feed/

Environment

OCTI 6.3.6

Reproducible Steps

Steps to create the smallest reproducible scenario:

1. Create an RSS Feed with one of the examples in the description
2. Wait a bit and look at the platform logs.

Expected Output

Ingestion of the RSS feed

Actual Output

Error 403

Screenshots

[khalidelborai]

Same issue here.

[SamuelHassine]

Maybe a problem of user agent? @romain-filigran @nino-filigran maybe critical?

[romain-filigran]

@SamuelHassine: Not only a user-agent issue from my investigation, more complicated depending on the RSS source. Need to test which one is working with an external RSS tool to identify the problem.

[biastogit]

it seems this issue appeared on my instance 3 days ago after adding the RSS darkreading feed.
I started disabling all the RSS feeds creating errors, without success.
Also tried to delete the darkreading reports and have multiple update indexing fail errors (UI and logs)

[SamuelHassine]

Any news @romain-filigran @nino-filigran ?

[nino-filigran]

None @SamuelHassine we've increased priority to ensure it's looked over by devs. So far it seems sepcific to some RSS feeds & therefore the solution not straightforward, but we need to investigate.

[aHenryJard]

Update: RSS feed works fine when removing the AxiosAgent from request. I think the best is to have an option to use agent or not, but we will have to rebuild the proxy option when AxiosAgent is not use, something like (from https://axios-http.com/docs/req_config ):

  proxy: {
    protocol: 'https',
    host: '127.0.0.1',
    port: 9000,
    auth: {
      username: 'mikeymike',
      password: 'rapunz3l'
    }
  },

As side note changes in this PR could help in the "no agent" use case #6451

[JeremyCloarec]

I found an interesting issue on an other project related to the same problem we have: FreshRSS/FreshRSS#6533. It looks like the 403 errors come from a Cloudflare misconfiguration on the RSS server side rather than it being a problem on our client side.
Unfortunately, it is not really possible to implement a consistent workaround on our side: a fix that would work for now could break in the future depending on how Cloudflare update their anti-bot techniques.
I think the best way to proceed would be to contact the company publishing the RSS feed and ask them to fix their server configuration, letting their RSS feed be accessible without the same Cloudflare protection as the rest of their website.

[Lhorus6]

Just for the record -> Another issue that could be related: #8968

[JeremyCloarec]

Just for the record -> Another issue that could be related: #8968

We will be closing both issues, as there is no proper way on our side to fix accessing a feed protected by Cloudflare.
When that happens, the best course of action is to contact the author of the feed to inform them that their Cloudflare configuration seems incorrect

[SamuelHassine]

Are we sure we cannot do anything on our side?

[aHenryJard]

We could try making the user agent configurable in UI as advance configuration or in JSON/env configuration, in order to have different user agent per instance if several opencti in the same network.

[aHenryJard]

I was wondering if RSS feeds are not called too frequently also, but this would require to rework ingestion manager because all feed have the same frequency of http request call. For example we could have a parameter min time between calls (like 10 min) and skip the feed until this min time is not reached.

[aHenryJard]

Reading cloudflare documentation it's possible to know that it's a cloudflare challenge, so we can also have a dedicated error checking presence of header cf-mitigated

https://developers.cloudflare.com/waf/reference/cloudflare-challenges/#detecting-a-challenge-page-response

[aHenryJard]

For the record, https://cybersecurity.att.com/site/blog-all-rss is now moved to https://levelblue.com/site/blog-all-rss