[io.monitor] Upgrade dependencies and enable monitoring of ThreadPools

[holgerfriedrich]

• metrics-core from 4.0.7 to 4.2.30

Release notes:
https://github.com/dropwizard/metrics/releases

• micrometer-core from 1.6.3 to 1.14.4

Release notes:
https://github.com/micrometer-metrics/micrometer/releases

• HdrHistogram from 2.1.12 to 2.2.2

Release notes:
https://github.com/HdrHistogram/HdrHistogram/releases/tag/HdrHistogram-2.2.1 https://github.com/HdrHistogram/HdrHistogram/releases/tag/HdrHistogram-2.2.2

[holgerfriedrich]

This is still work in progress. It basically seems to work, with the modified metric binding from openhab/openhab-addons#18377; data shows up at http://localhost:8080/rest/metrics/prometheus as expected.

Though, during startup I can see a warning showing up several times:

[WARN ] [nt.binder.jvm.ExecutorServiceMetrics] - Failed to bind as org.openhab.core.common.ThreadPoolManager$UnstoppableExecutorService is unsupported.

Looking at the implementation, this seems to be caused by registering our thread pools for monitoring, in org.openhab.core.io.monitor/src/main/java/org/openhab/core/io/monitor/internal/metrics/ThreadPoolMetric.java.
Looking at https://github.com/micrometer-metrics/micrometer/blob/508d4dfaf4ce3d25623fdb2fb04783019a444525/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/jvm/ExecutorServiceMetrics.java#L296, the ExecutorService we try to register is not supported.
So my first approach was to modify UnstoppableExecutorService to base on one of supported classes and override unused functions, but I did not get this working properly yet (the best was to see the metrics for all thread pools for a few seconds in the rest API, before it stopped updating).

I have checked my 4.3.x installation and do not find thread pool data as well, so I am wondering if this has stopped working at some time in the past.
Can somebody confirm this is working in 4.3.x?

[holgerfriedrich]

The effects described above seemed to be caused by the garbage collector cleaning up the instance of ExecutorServiceMetrics.

I have a working implementation, which actually bases UnstoppableExecutorService on ThreadPoolExecutor (and overriding nearly all methods). This allows ExecutorServiceMetrics from micrometer-core to handle it properly.

This a part of what I see on the demo app on http://localhost:8080/rest/metrics/prometheus (after installing the modified metric binding from openhab/openhab-addons#18377 via the Webcosole http://localhost:8080/system/console/bundles):

# HELP executor_active_threads The approximate number of threads that are actively executing tasks
# TYPE executor_active_threads gauge
executor_active_threads{metric="openhab.core.metric.threadpools",name="ConfigStatusService",openhab_core_metric="true",pool="ConfigStatusService"} 0.0
executor_active_threads{metric="openhab.core.metric.threadpools",name="JsonStorage",openhab_core_metric="true",pool="JsonStorage"} 0.0
executor_active_threads{metric="openhab.core.metric.threadpools",name="OH-core-javasound-source",openhab_core_metric="true",pool="OH-core-javasound-source"} 0.0
executor_active_threads{metric="openhab.core.metric.threadpools",name="common",openhab_core_metric="true",pool="common"} 0.0
executor_active_threads{metric="openhab.core.metric.threadpools",name="discovery",openhab_core_metric="true",pool="discovery"} 0.0
executor_active_threads{metric="openhab.core.metric.threadpools",name="file-processing",openhab_core_metric="true",pool="file-processing"} 0.0
executor_active_threads{metric="openhab.core.metric.threadpools",name="safeCall",openhab_core_metric="true",pool="safeCall"} 0.0
executor_active_threads{metric="openhab.core.metric.threadpools",name="scheduler",openhab_core_metric="true",pool="scheduler"} 0.0
executor_active_threads{metric="openhab.core.metric.threadpools",name="thingManager",openhab_core_metric="true",pool="thingManager"} 0.0
executor_active_threads{metric="openhab.core.metric.threadpools",name="watchservice",openhab_core_metric="true",pool="watchservice"} 0.0
# HELP executor_completed_tasks_total The approximate total number of tasks that have completed execution
# TYPE executor_completed_tasks_total counter
executor_completed_tasks_total{metric="openhab.core.metric.threadpools",name="ConfigStatusService",openhab_core_metric="true",pool="ConfigStatusService"} 0.0
executor_completed_tasks_total{metric="openhab.core.metric.threadpools",name="JsonStorage",openhab_core_metric="true",pool="JsonStorage"} 0.0
executor_completed_tasks_total{metric="openhab.core.metric.threadpools",name="OH-core-javasound-source",openhab_core_metric="true",pool="OH-core-javasound-source"} 0.0
executor_completed_tasks_total{metric="openhab.core.metric.threadpools",name="common",openhab_core_metric="true",pool="common"} 304.0
executor_completed_tasks_total{metric="openhab.core.metric.threadpools",name="discovery",openhab_core_metric="true",pool="discovery"} 12.0
executor_completed_tasks_total{metric="openhab.core.metric.threadpools",name="file-processing",openhab_core_metric="true",pool="file-processing"} 525.0
executor_completed_tasks_total{metric="openhab.core.metric.threadpools",name="safeCall",openhab_core_metric="true",pool="safeCall"} 0.0
executor_completed_tasks_total{metric="openhab.core.metric.threadpools",name="scheduler",openhab_core_metric="true",pool="scheduler"} 0.0
executor_completed_tasks_total{metric="openhab.core.metric.threadpools",name="thingManager",openhab_core_metric="true",pool="thingManager"} 52.0
executor_completed_tasks_total{metric="openhab.core.metric.threadpools",name="watchservice",openhab_core_metric="true",pool="watchservice"} 0.0
# HELP executor_pool_core_threads The core number of threads for the pool
# TYPE executor_pool_core_threads gauge
executor_pool_core_threads{metric="openhab.core.metric.threadpools",name="ConfigStatusService",openhab_core_metric="true",pool="ConfigStatusService"} 1.0
executor_pool_core_threads{metric="openhab.core.metric.threadpools",name="JsonStorage",openhab_core_metric="true",pool="JsonStorage"} 5.0
executor_pool_core_threads{metric="openhab.core.metric.threadpools",name="OH-core-javasound-source",openhab_core_metric="true",pool="OH-core-javasound-source"} 5.0
executor_pool_core_threads{metric="openhab.core.metric.threadpools",name="common",openhab_core_metric="true",pool="common"} 5.0
executor_pool_core_threads{metric="openhab.core.metric.threadpools",name="discovery",openhab_core_metric="true",pool="discovery"} 3.0
executor_pool_core_threads{metric="openhab.core.metric.threadpools",name="file-processing",openhab_core_metric="true",pool="file-processing"} 5.0
executor_pool_core_threads{metric="openhab.core.metric.threadpools",name="safeCall",openhab_core_metric="true",pool="safeCall"} 1.0
executor_pool_core_threads{metric="openhab.core.metric.threadpools",name="scheduler",openhab_core_metric="true",pool="scheduler"} 5.0
executor_pool_core_threads{metric="openhab.core.metric.threadpools",name="thingManager",openhab_core_metric="true",pool="thingManager"} 5.0
executor_pool_core_threads{metric="openhab.core.metric.threadpools",name="watchservice",openhab_core_metric="true",pool="watchservice"} 5.0
# HELP executor_pool_max_threads The maximum allowed number of threads in the pool
# TYPE executor_pool_max_threads gauge
executor_pool_max_threads{metric="openhab.core.metric.threadpools",name="ConfigStatusService",openhab_core_metric="true",pool="ConfigStatusService"} 5.0
executor_pool_max_threads{metric="openhab.core.metric.threadpools",name="JsonStorage",openhab_core_metric="true",pool="JsonStorage"} 2.147483647E9
executor_pool_max_threads{metric="openhab.core.metric.threadpools",name="OH-core-javasound-source",openhab_core_metric="true",pool="OH-core-javasound-source"} 2.147483647E9
executor_pool_max_threads{metric="openhab.core.metric.threadpools",name="common",openhab_core_metric="true",pool="common"} 2.147483647E9
executor_pool_max_threads{metric="openhab.core.metric.threadpools",name="discovery",openhab_core_metric="true",pool="discovery"} 2.147483647E9
executor_pool_max_threads{metric="openhab.core.metric.threadpools",name="file-processing",openhab_core_metric="true",pool="file-processing"} 2.147483647E9
executor_pool_max_threads{metric="openhab.core.metric.threadpools",name="safeCall",openhab_core_metric="true",pool="safeCall"} 3.0
executor_pool_max_threads{metric="openhab.core.metric.threadpools",name="scheduler",openhab_core_metric="true",pool="scheduler"} 2.147483647E9
executor_pool_max_threads{metric="openhab.core.metric.threadpools",name="thingManager",openhab_core_metric="true",pool="thingManager"} 2.147483647E9
executor_pool_max_threads{metric="openhab.core.metric.threadpools",name="watchservice",openhab_core_metric="true",pool="watchservice"} 2.147483647E9
# HELP executor_pool_size_threads The current number of threads in the pool
# TYPE executor_pool_size_threads gauge
executor_pool_size_threads{metric="openhab.core.metric.threadpools",name="ConfigStatusService",openhab_core_metric="true",pool="ConfigStatusService"} 0.0
executor_pool_size_threads{metric="openhab.core.metric.threadpools",name="JsonStorage",openhab_core_metric="true",pool="JsonStorage"} 0.0
executor_pool_size_threads{metric="openhab.core.metric.threadpools",name="OH-core-javasound-source",openhab_core_metric="true",pool="OH-core-javasound-source"} 0.0
executor_pool_size_threads{metric="openhab.core.metric.threadpools",name="common",openhab_core_metric="true",pool="common"} 5.0
executor_pool_size_threads{metric="openhab.core.metric.threadpools",name="discovery",openhab_core_metric="true",pool="discovery"} 3.0
executor_pool_size_threads{metric="openhab.core.metric.threadpools",name="file-processing",openhab_core_metric="true",pool="file-processing"} 3.0
executor_pool_size_threads{metric="openhab.core.metric.threadpools",name="safeCall",openhab_core_metric="true",pool="safeCall"} 0.0
executor_pool_size_threads{metric="openhab.core.metric.threadpools",name="scheduler",openhab_core_metric="true",pool="scheduler"} 0.0
executor_pool_size_threads{metric="openhab.core.metric.threadpools",name="thingManager",openhab_core_metric="true",pool="thingManager"} 5.0
executor_pool_size_threads{metric="openhab.core.metric.threadpools",name="watchservice",openhab_core_metric="true",pool="watchservice"} 0.0
# HELP executor_queue_remaining_tasks The number of additional elements that this queue can ideally accept without blocking
# TYPE executor_queue_remaining_tasks gauge
executor_queue_remaining_tasks{metric="openhab.core.metric.threadpools",name="ConfigStatusService",openhab_core_metric="true",pool="ConfigStatusService"} 2.147483647E9
executor_queue_remaining_tasks{metric="openhab.core.metric.threadpools",name="JsonStorage",openhab_core_metric="true",pool="JsonStorage"} 2.147483647E9
executor_queue_remaining_tasks{metric="openhab.core.metric.threadpools",name="OH-core-javasound-source",openhab_core_metric="true",pool="OH-core-javasound-source"} 2.147483647E9
executor_queue_remaining_tasks{metric="openhab.core.metric.threadpools",name="common",openhab_core_metric="true",pool="common"} 2.147483647E9
executor_queue_remaining_tasks{metric="openhab.core.metric.threadpools",name="discovery",openhab_core_metric="true",pool="discovery"} 2.147483647E9
executor_queue_remaining_tasks{metric="openhab.core.metric.threadpools",name="file-processing",openhab_core_metric="true",pool="file-processing"} 2.147483647E9
executor_queue_remaining_tasks{metric="openhab.core.metric.threadpools",name="safeCall",openhab_core_metric="true",pool="safeCall"} 2.147483647E9
executor_queue_remaining_tasks{metric="openhab.core.metric.threadpools",name="scheduler",openhab_core_metric="true",pool="scheduler"} 2.147483647E9
executor_queue_remaining_tasks{metric="openhab.core.metric.threadpools",name="thingManager",openhab_core_metric="true",pool="thingManager"} 2.147483647E9
executor_queue_remaining_tasks{metric="openhab.core.metric.threadpools",name="watchservice",openhab_core_metric="true",pool="watchservice"} 2.147483647E9
# HELP executor_queued_tasks The approximate number of tasks that are queued for execution
# TYPE executor_queued_tasks gauge
executor_queued_tasks{metric="openhab.core.metric.threadpools",name="ConfigStatusService",openhab_core_metric="true",pool="ConfigStatusService"} 0.0
executor_queued_tasks{metric="openhab.core.metric.threadpools",name="JsonStorage",openhab_core_metric="true",pool="JsonStorage"} 0.0
executor_queued_tasks{metric="openhab.core.metric.threadpools",name="OH-core-javasound-source",openhab_core_metric="true",pool="OH-core-javasound-source"} 0.0
executor_queued_tasks{metric="openhab.core.metric.threadpools",name="common",openhab_core_metric="true",pool="common"} 4.0
executor_queued_tasks{metric="openhab.core.metric.threadpools",name="discovery",openhab_core_metric="true",pool="discovery"} 2.0
executor_queued_tasks{metric="openhab.core.metric.threadpools",name="file-processing",openhab_core_metric="true",pool="file-processing"} 0.0
executor_queued_tasks{metric="openhab.core.metric.threadpools",name="safeCall",openhab_core_metric="true",pool="safeCall"} 0.0
executor_queued_tasks{metric="openhab.core.metric.threadpools",name="scheduler",openhab_core_metric="true",pool="scheduler"} 0.0
executor_queued_tasks{metric="openhab.core.metric.threadpools",name="thingManager",openhab_core_metric="true",pool="thingManager"} 1.0
executor_queued_tasks{metric="openhab.core.metric.threadpools",name="watchservice",openhab_core_metric="true",pool="watchservice"} 0.0

From my point of view, we are good to go for a review.