engine/schedulemanager: isolate processing of individual schedules

[mastercactapus]

What problem would you like to solve? Please describe:
All schedules are updated in a single transaction:

• A single bug/problem causes all schedules to "freeze"
• All schedules, regardless of number, need to finish within the time limit

Describe the solution you'd like:
Process one schedule at a time, up to some limit per cycle

Describe alternatives you've considered:
Continued risk.

Additional context:
Things to watch out for in implementation:

• Ensure all schedules eventually get processed when multiple cycles are required
• Performance impact & timing

[mastercactapus]

Possible approach:

• Each cycle, fetch the list of all schedule IDs
• Store last processed time per schedule in memory truncated to, say, a minute
• shuffle the list of IDs, sorting oldest first
• have update function fetch all data for a single schedule
• Since we're fetching a single schedule, some queries may be combined
• Potentially use batch queries from pgx
• Move a schedule to the back of the list, pass-or-fail
• Log errors but keep processing other schedules
• Maybe set a fixed time allotment (e.g., process as many as possible within 2s) rather than a fixed number

[pavan541cs]

@mastercactapus Can I pick up this issue and work on this

[mastercactapus]

I'd say this one is still in the planning phase. I did a bit of testing on this and a big issue with the approach I outlined is just to walk through the schedules by locking alone (no actually fetching/updating) results in a > 10x slowdown.

[stale]

This issue has been automatically marked as stale because it has not had recent activity.

[mastercactapus]

Let's revisit this after the engine state is implemented (going in with metrics code).

[stale]

This issue has been automatically marked as stale because it has not had recent activity.

[mastercactapus]

Obsoleted by #4245