kube_cronjob_status_active doesn't tell if a cronjob is running

[koote]

What happened:
I am using kube_cronjob_status_active to monitor my cronjob, it runs every 5 minutes.
In the given time window, kube_cronjob_status_active{cronjob=my-job} shows that there is no active cronjob running:

However if I check kube_job_status_start_time{job_name=my-job*} (my-job is a cronjob, every job instance has name like my-job-12345678) within same time window, it shows that there are jobs scheduled to run approximately every 5 minutes:

What you expected to happen:
I would like to see that the kube_cronjob_status_active matches kube_job_status_start_time.

How to reproduce it (as minimally and precisely as possible):
It is running in our internal k8s cluster so I don't know how to let others repo it, but I am happy to provide as much information as needed.

# An example: https://github.com/kubernetes/kube-state-metrics/issues/2223#issuecomment-1792850276
minikube start
...
go run main.go --custom-resource-state-only --custom-resource-state-config-file ksm-2223/custom-resource-config-file.yaml --kubeconfig ~/.kube/config

Anything else we need to know?:

Environment: AWS EKS

• kube-state-metrics version:
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration: AWS EKS
• Other info:

[dgrisonnet]

/assign @rexagod
/triage accepted

[Haleygo]

Hi @koote ,
How long does your job usually take to complete? You can find out by querying kube_job_status_completion_time{job_name=my-job*}-kube_job_status_start_time{job_name=my-job*}.
If the duration is too short (e.g., several seconds) and your scrape interval is relatively high (30s or 1m), you may miss the kube_cronjob_status_active and get zero values in database, as kube_cronjob_status_active reflects the running job at that time.
To validate this, try extending your job's exection time with deliberate sleep cmd, or reduce the scrape interval for KSM job.

[koote]

Hi @koote , How long does your job usually take to complete? You can find out by querying kube_job_status_completion_time{job_name=my-job*}-kube_job_status_start_time{job_name=my-job*}. If the duration is too short (e.g., several seconds) and your scrape interval is relatively high (30s or 1m), you may miss the kube_cronjob_status_active and get zero values in database, as kube_cronjob_status_active reflects the running job at that time. To validate this, try extending your job's exection time with deliberate sleep cmd, or reduce the scrape interval for KSM job.

Thanks @Haleygo, I checked last 3 days, usually the job runs for 40-80 seconds, so that is not the issue.

[Haleygo]

@koote hmm, that's odd, it works in my test with a cronjob running every 5m and completing in 1m.

What's your scrape interval for KSM job then? Can you get non-zero values for kube_cronjob_status_active by calling {ksm-address}/metrics directly?