Handle second/minute-rates limits on Flavors and Analyzers

[amr-cossi]

Request Type

Feature Request

The goal of this feature would be to handle APIs limiting the number of requests per second or minute. Right now, if you launch a lot of queries simultaneously, CORTEX launches them all together (at the parallelization rate configured) but fails for all queries hitting the short rate limit.
It would be great to be able to configure a rate limit in # queries per # seconds for a Flavor (endpoint limit) and an Analyzer (service limit) and have CORTEX wait between queries to avoid the threshold.

[amr-cossi]

Does this implementation handle a wait time between queries managed by CORTEX ?
In my understanding, with this modification, queries will fail because of the CORTEX limit instead of the distant endpoint limit.
The real feature would be to avoid failed queries altogether.

[nadouani]

Your understanding is right: Cortex will not enqueue the jobs based on that rate limit.

We have probably misunderstood the feature request.

[vazromju]

hello
has somebody been able to get this analyzer working ?
in my case, with a right api-key it is in progress until timeout and it is dropped
I have tested with curl and it works from cortex box
I have been reading (without knowlegde of python) the paloalto script, /usr/local/lib/python3.6/site-packages/autofocus/autofocus.py
and I miss some parameters as scope that I cannot realize where this parameter is calculated.

so, has somebody been able to get it working?

Thank you

[BrijJhala]

@nadouani its great feature to have it. Can we reopen this ticket ? Thank you. who knows we will pick up to fix too :)