Adds metrics on how different queues have different thread amounts allocated to them #37
Conversation
There was a problem hiding this comment.
Thank you for your pull request!
It is a bit unexpected to see desire to monitor underutilization and try to predict full utilization as, I believe, in many apps variability in load may result in spikes of 100% to 0% load. Tell me a bit more about your load profile, how you plan to fight this noise?
But I suppose that if aggregate over some big enough time window, it can make sense and predict full utilization in advance.
Also, can you please measure how long this metric calculation takes on your production (with a lot of running worker processes). I suppose that it should be fast as it doesn't iterate over jobs in queues, so no need to disable it by default (it was my first thought when I saw .each inside .each), but want to be sure.
| queue_data[queue][:available] += process['concurrency'] - process['busy'] | ||
| end | ||
| end | ||
| sidekiq_active_workers_count.set({}, process_data[:active]) |
There was a problem hiding this comment.
Is process_data[:active] the same as stats.workers_size? (it is better to keep backward compatibility)
| ::Sidekiq::ProcessSet.new.each do |process| | ||
| process_data[:active] += process['concurrency'] | ||
| process_data[:busy] += process['busy'] | ||
| process_data[:available] += process['concurrency'] - process['busy'] |
There was a problem hiding this comment.
What is the point of calculating available threads number here? It can be calculated in the monitoring system as well
Problem
Latency is great to measure workloads that are starving for resources, but some workloads are critical and we want to be able to see that a problem is forming before it starts.
How can we measure how can we measure if a given queue is about to "starve" for resources?
Idea
Measure how busy are the threads able to process a given queue.
If many threads are available then everything is fine, but once more and more threads become busy less available capacity we have for the queue until we reach a point where all threads are busy and we start to form a backlog.
Proposed Solution
Extract metrics from
::Sidekiq::ProcessSet.::Sidekiq::ProcessSetgives a set of Process. Each process has the list of queues it's processing and how busy its threads are. From it we can just aggregate the information.