Add additional metrics for worker saturation analysis and scaling

[jeisen]

This adds three new metrics that can be used to implement scaling rules:

• sidekiq_concurrency -- Total concurrency reported by sidekiq processes (i.e. max available)
• sidekiq_available_workers -- The number of threads able to handle jobs, calculated from each sidekiq process by subtracting its busy workers from its concurrency. In the case of a "quiet" process, which is not accepting any jobs, this metric reports 0 available.
• sidekiq_saturation -- The percentage of workers currently in use, calculated using the available_workers formula above. This is so that we count quiet processes as fully saturated; they cannot accept any more jobs, so from a scaling perspective they should not be counted as available.

We then scale in Kubernetes using the following metric:

((max(sidekiq_jobs_waiting_count) / max(sidekiq_concurrency) + max(sidekiq_saturation)) * 100

This lets us scale pods based on the current saturation plus the additional saturation required for jobs waiting in the queue.

[Envek]

Hi! Thank you for contributing to Yabeda!

My main concern here is that every worker process reports aggregated data about all workers (and there may be hundreds of them).

Isn't it better to report data about only this worker and let monitoring system to aggregate it? (By the way, what are you using?)

[Envek]

From the metric name I can't understand whether it is about processes or threads. I need to read the collector's code below to get it. Let's clarify:

Suggested change

	gauge :available_workers, tags: [], comment: "The number of workers available for new jobs across all processes."
	gauge :available_threads, tags: [], comment: "The number of threads available for job processing across all processes."

[jeisen]

This name was chosen to be consistent with how it’s reporting active workers — it looks like Sidekiq uses the term “worker” to mean “thread”.

[jeisen]

I originally tried to do it per-process and let Prometheus aggregate it, but Sidekiq isn’t reporting it that way so I had to do it like this.

The scaling is done with the Prometheus Exporter process making these stats available as custom metrics for the Kubernetes Horizontal Pod Autoscaler. To avoid it overloading Prometheus with queries, it’s set up as a service-level metric instead of per-pod.

[hbontempo-br]

Hi!

I saw the feature and specially the total number of threads is interesting (we already have the busy sidekiq_active_workers_count, therefore the available and saturation metrics are not really required since we can derive then from other metrics).

My main concern here is that every worker process reports aggregated data about all workers (and there may be hundreds of them).

By what I saw when this PR was opened there was no collect_cluster_metrics configuration yet.
Since we have it now, I believe that this is not a big concern as it was before, right?

@Envek are you open to consider this feature again? Maybe a bit smaller (just a sidekiq_workers_count) and under the collect_cluster_metrics configuration?
I open to help with code if you think the addition is valid and want help