QoS data missing on 21.08

[JaskaJ78]

Hello,

I did a clean docker installation of v21.08 and everything works like a charm and I can see the data from our clusters.
Only thing I'm missing is the QoS data for LUNs & volumes. I can see this information under the LUN page: "You must enable the Workload/QOS counters in $HARVEST/conf/zapiperf/default.yaml."
I did uncomment those rows, but nothing changed.
zapiperf.txt

What am I still missing?

[cgrinds]

hi @JaskaJ78 do you see any errors related to qos in your poller logs?
What if you hit the Prometheus endpoint directly like this:

curl -s 'http://localhost:12990/metrics' | grep "^qos" | head
qos_volume_read_data{cluster="umeng-aff300-05-06",datacenter="nane",wid="28856",svm="astra_vs2",volume="trident_pvc_84115bd1_3428_47eb_8b80_e6b5c242ac35",qtree="",lun="",file="",policy_group="User-Best-Effort"} 0
qos_volume_write_ops{cluster="umeng-aff300-05-06",datacenter="nane",wid="28856",svm="astra_vs2",volume="trident_pvc_84115bd1_3428_47eb_8b80_e6b5c242ac35",qtree="",lun="",file="",policy_group="User-Best-Effort"} 0
qos_volume_read_io_type{cluster="umeng-aff300-05-06",datacenter="nane",wid="28856",svm="astra_vs2",volume="trident_pvc_84115bd1_3428_47eb_8b80_e6b5c242ac35",qtree="",lun="",file="",policy_group="User-Best-Effort",metric="fc_miss"} 0
qos_volume_write_data{cluster="umeng-aff300-05-06",datacenter="nane",wid="28856",svm="astra_vs2",volume="trident_pvc_84115bd1_3428_47eb_8b80_e6b5c242ac35",qtree="",lun="",file="",policy_group="User-Best-Effort"} 0
qos_volume_sequential_reads{cluster="umeng-aff300-05-06",datacenter="nane",wid="28856",svm="astra_vs2",volume="trident_pvc_84115bd1_3428_47eb_8b80_e6b5c242ac35",qtree="",lun="",file="",policy_group="User-Best-Effort"} 0
qos_volume_read_io_type{cluster="umeng-aff300-05-06",datacenter="nane",wid="28856",svm="astra_vs2",volume="trident_pvc_84115bd1_3428_47eb_8b80_e6b5c242ac35",qtree="",lun="",file="",policy_group="User-Best-Effort",metric="disk"} 0
qos_volume_read_io_type{cluster="umeng-aff300-05-06",datacenter="nane",wid="28856",svm="astra_vs2",volume="trident_pvc_84115bd1_3428_47eb_8b80_e6b5c242ac35",qtree="",lun="",file="",policy_group="User-Best-Effort",metric="cache"} 0
qos_volume_read_latency{cluster="umeng-aff300-05-06",datacenter="nane",wid="28856",svm="astra_vs2",volume="trident_pvc_84115bd1_3428_47eb_8b80_e6b5c242ac35",qtree="",lun="",file="",policy_group="User-Best-Effort"} 0
qos_volume_read_io_type{cluster="umeng-aff300-05-06",datacenter="nane",wid="28856",svm="astra_vs2",volume="trident_pvc_84115bd1_3428_47eb_8b80_e6b5c242ac35",qtree="",lun="",file="",policy_group="User-Best-Effort",metric="ext_cache"} 0
qos_volume_latency{cluster="umeng-aff300-05-06",datacenter="nane",wid="28856",svm="astra_vs2",volume="trident_pvc_84115bd1_3428_47eb_8b80_e6b5c242ac35",qtree="",lun="",file="",policy_group="User-Best-Effort"} 0

I verified on one of our local setups that qos data is collected once those four lines are uncommented.

[JaskaJ78]

Hi,

That command returns nothing. Also if I remove the 'grep' it shows nothing:
[]# curl -s 'http://localhost:12990/metrics' | grep "^qos" | head
[]# curl -s 'http://localhost:12990/metrics'
[]# curl -s 'http://localhost:12991/metrics'
[]# curl -s 'http://localhost:12992/metrics'
[]# curl -s 'http://localhost:12993/metrics'

Ports 12991-12993 I've defined for exporters:
Exporters:
  prometheus:
    exporter: Prometheus
    local_http_addr: 0.0.0.0
    port: 12990
  prometheus1:
    exporter: Prometheus
    addr: 0.0.0.0
    port_range: 12991-12993

[cgrinds]

You mentioned that most things are working and that you're running via Docker. Are you exposing the Docker port? I ask because if you are, the curl command should work. Can you double check the exposed ports by pasting the results of docker ps -a and then making sure you use that same port with your curl command?

Another thought - perhaps your edits to uncomment the four qos lines are not seen in your docker volume. Not sure how familiar you are with Docker, but if you shell into your Docker container and look at the conf/zapiperf/default.yaml file that you edited outside the container, I'm betting those lines are still commented out.

If you just started, it's probably better to upgrade to 21.11 since we improved the docker volume mounting workflow to make it easier to keep the local Harvest changes in sync with the container. #511 mentioned in the 21.11 release notes.

[JaskaJ78]

Hi,

Yes, I decided to use the docker for simplicity even though I'm not so familiar with it.
I tried to update to the latest version of Harvest as per instructions, but no luck:
[]# docker pull rahulguptajss/harvest
Using default tag: latest
latest: Pulling from rahulguptajss/harvest
Digest: sha256:e1115992431bf431651c122c2f1f552fcee9adad74e7ebe5a57f36c1c6f6560b
Status: Image is up to date for rahulguptajss/harvest:latest
docker.io/rahulguptajss/harvest:latest

This also didn't update it:
[]# docker-compose -f prom-stack.yml -f harvest-compose.yml restart

So since this is a new setup, I just downloaded the latest version and copied the config files over. Now, everything works as intended and I can see the QoS data also.

Thank you for your help.