core: fix unrecoverable freezes of rabbit's consumer

[bougue-pe]

Hard fix (kind of) to kill the process if a thread's exception goes up to main(), or if rabbit's cancel/shutdown notification is received (even if other threads may run) and let orchestrator restart it.

Bump amqp-client on the way as it doesn't hurt.

Fixes #8621
Also reproduced and fixed the case that leads to the following (different) core logs:

[11:16:04,880] [INFO]          [WorkerCommand] consume shutdown: amq.ctag-LsXBfmFdL6n758icthlCYA, com.rabbitmq.client.ShutdownSignalException: connection error; protocol method: #method<connection.close>(reply-code=320, reply-text=CONNECTION_FORCED - broker forced connection closure with reason 'shutdown', class-id=0, method-id=0)
[11:16:04,883] [WARN]  [ForgivingExceptionHandler] An unexpected connection driver error occurred (Exception message: Connection reset by peer)
Exception in thread "main" com.rabbitmq.client.AlreadyClosedException: connection is already closed due to connection error; protocol method: #method<connection.close>(reply-code=320, reply-text=CONNECTION_FORCED - broker forced connection closure with reason 'shutdown', class-id=0, method-id=0)
        at com.rabbitmq.client.impl.AMQConnection.startShutdown(AMQConnection.java:1012)
        at com.rabbitmq.client.impl.AMQConnection.close(AMQConnection.java:1127)
        at com.rabbitmq.client.impl.AMQConnection.close(AMQConnection.java:1056)
        at com.rabbitmq.client.impl.AMQConnection.close(AMQConnection.java:1040)
        at com.rabbitmq.client.impl.AMQConnection.close(AMQConnection.java:1032)
        at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.close(AutorecoveringConnection.java:289)
        at kotlin.io.CloseableKt.closeFinally(Closeable.kt:56)
        at fr.sncf.osrd.cli.WorkerCommand.run(WorkerCommand.kt:319)
        at fr.sncf.osrd.App.main(App.java:44)

Hand-tested

• Run classic (one per infra) core with rabbitmq up, without editoast to load infra from: ✅core crashes.
• Run single-worker core with the whole stack running: ✅core works and OSRD does its job.
• Run single-worker core with the whole stack, then stop rabbitmq: ✅core exits (on shutdown notification).
• Run full stack single-worker mode, then remove the core-req-all queue to initiate cancel notification: ✅core logs "consumer cancelled" then stops (should cover Core message consumer fails and blocks the scenario #8621).
• Run full stack + single-worker editoast (no core), initiate some core requests (using front).
  • Then stop editoast and start single-worker core (so impossible to load infra inside DeliverCallback): ✅core crashes and releases unacked messages.
  • Then start editoast, then start core: ✅core works and OSRD does its job.

• Then stop editoast and start single-worker core (so impossible to load infra inside DeliverCallback): ✔️❓Exceptions in threads when trying to load infra for pending requests, core stays alive (leaving pending requests unacked until core is stopped - after step below).
  ➡️We can try/catch in callback function and exitProcess to force shutdown. DONE in last commit.
• Then start editoast (keep core as-is) and initiate some new core request (using front): ✅core "correctly" processing only new requests.
• Stop core: ✅Unacked requests are back to ready.
• Start core: ✅Ready requests are processed.

Understanding of previous and current work

Previous:

• core: multithread core workers #9591 messed-up with core crashing as expected when exception is raised up to main (still not understanding exactly what prevents the app from stopping).
• Especially core: improve worker lifetime #9439 was worked on (and in my memory correctly hand-tested) at the same time, but when merged (after core: multithread core workers #9591) it "silently" didn't work 😞.

Current:

• ShutdownCallback looks like it's executed in the main thread (triggering the newly added catch in the main(), then the final return) and it may be an idea to use shutdown notification to exit cleanly
• The thread executors are messing with the handling of exceptions
• Looks like the CancelCallback doesn't trigger a "join" or a shutodwn process (because of threads? maybe more because it's on the implementation to decide what's graceful?)

Looks like some improvements may be done (to be explored later, sorted by ROI)

• Improve/explicit some work applied in core: improve worker lifetime #9439 (use isRecoverable, cleanup consumer logs, properly close channels and connections)
• Avoid sharing channels between threads as stated in dedicated documentation
• Handle more standardly shutdown (on exceptions or on notification) by issuing a shutdown notification, or sharing a signal (or common variable?)

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.93%. Comparing base (b3a6f01) to head (e6c5f8c).
Report is 93 commits behind head on dev.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##              dev   #10594      +/-   ##
==========================================
- Coverage   81.93%   81.93%   -0.01%     
==========================================
  Files        1079     1079              
  Lines      107380   107376       -4     
  Branches      737      737              
==========================================
- Hits        87984    87978       -6     
- Misses      19356    19358       +2     
  Partials       40       40              

Flag	Coverage Δ
editoast	74.28% <ø> (-0.01%)	⬇️
front	89.47% <ø> (-0.01%)	⬇️
gateway	2.18% <ø> (ø)
osrdyne	3.28% <ø> (ø)
railjson_generator	87.50% <ø> (ø)
tests	88.14% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[bougue-pe]

A third commit was pushed, and the main comment updated (and all rebased on dev).

No more work is planned on this, please read the main comment, any feedback is welcome, and we should be good to go 🙏

[bougue-pe]

There is a bit more work, actually (for me): test if it improves the case described in #10704
EDIT: It does not improve things, looks like a different issue, not investigating more on it.

[eckter]

Thanks for handling this!

[Khoyo]

Great work!