Flexgroups support ?

[wally007]

Hello,

I'm testing new Harvest on RHEL and I've read most of the documentation and GitHub issues but i cant figure out how to best approach FlexGroups. I thought I'd ask before i start trial and error hunting for the right config.

1, We don't use FlexVols at all so we have only FlexGroups (we will never use FlexVols). I'd like to do FlexVols constituents "aggregation" of statistics already in Harvest (instead of doing it Grafana with custom merge transformation)

• Currently i see ton of vol_name___001, vol_name___002 etc. in Grafana which is not very helpful of gauge performance.

2, On the same note, what would be the best way to "aggregate" NFS stats - regardless of the version ?
Again I'd like to do this already in harvest if possible.

• Basically we don't really care what NFS version end client is using - we just want SMB/NFS protocol usage breakdown on node and SVM level.

Thank you very much.

[Hardikl]

Hi @wally007 ,

1. We do monitor flexgroup constituents in Harvest, but we would change the style from flexgroup_constituent to flexgroup. https://github.com/NetApp/harvest/blob/main/conf/zapi/cdot/9.8.0/volume.yaml#L68

curl -s 'http://localhost:12997/metrics' | rg "^volume_labels" | grep flexgroup
volume_labels{datacenter="DC-06",cluster="optimus",volume="certl_nas_prod__0001",node="optimus-01",svm="svm_certl_nas",aggr="aggr1_optimus_01",style="flexgroup",state="online",is_sis_volume="true"} 1.0
volume_labels{datacenter="DC-06",cluster="optimus",volume="certl_jamaica_cache__0002",node="optimus-01",svm="svm_certl_nas",aggr="aggr1_optimus_01",style="flexgroup",state="online",is_sis_volume="false"} 1.0
volume_labels{datacenter="DC-06",cluster="optimus",volume="certl_jamaica_cache__0003",node="optimus-01",svm="svm_certl_nas",aggr="aggr1_optimus_01",style="flexgroup",state="online",is_sis_volume="false"} 1.0
volume_labels{datacenter="DC-06",cluster="optimus",volume="certl_nas_prod",node="",svm="svm_certl_nas",aggr="",style="flexgroup",state="online",is_sis_volume="true"} 1.0
volume_labels{datacenter="DC-06",cluster="optimus",volume="certl_jamaica_cache",node="",svm="svm_certl_nas",aggr="",style="flexgroup",state="online",is_sis_volume="false"} 1.0

Also, we do support the aggregation here, https://github.com/NetApp/harvest/blob/main/conf/zapi/cdot/9.8.0/volume.yaml#L70, you can customize if needed.

2. We do support the nfs stats perf counter at node level and svm level: https://github.com/NetApp/harvest/blob/main/conf/zapiperf/default.yaml
   where nfs version would be label in the metrics and can be utilise as required in promQL.

hardikl@hardikl-mac-0 harvest % curl -s 'http://localhost:12997/metrics' | rg "nfs_write_avg_latency"
node_nfs_write_avg_latency{cluster="optimus",datacenter="DC-06",nfsv="v4.1",node="optimus-01"} 0
node_nfs_write_avg_latency{cluster="optimus",datacenter="DC-06",nfsv="v4.1",node="optimus-02"} 0
svm_nfs_write_avg_latency{datacenter="DC-06",nfsv="v4",cluster="optimus",svm="DmoESX_prime-mc"} 0
svm_nfs_write_avg_latency{datacenter="DC-06",nfsv="v4",cluster="optimus",svm="DmoNAS_optimus"} 0
svm_nfs_write_avg_latency{datacenter="DC-06",nfsv="v4",cluster="optimus",svm="DmoNAS_prime-mc"} 0
svm_nfs_write_avg_latency{datacenter="DC-06",nfsv="v4",cluster="optimus",svm="DmoESX_optimus"} 0
node_nfs_write_avg_latency{datacenter="DC-06",nfsv="v3",cluster="optimus",node="optimus-02"} 0
node_nfs_write_avg_latency{datacenter="DC-06",nfsv="v3",cluster="optimus",node="optimus-01"} 14082.493066666666
svm_nfs_write_avg_latency{nfsv="v3",cluster="optimus",datacenter="DC-06",svm="svm_certl_src"} 0
svm_nfs_write_avg_latency{nfsv="v3",cluster="optimus",datacenter="DC-06",svm="DmoESX_optimus"} 14035.704697986577
svm_nfs_write_avg_latency{nfsv="v3",cluster="optimus",datacenter="DC-06",svm="DmoESX_prime-mc"} 0
svm_nfs_write_avg_latency{nfsv="v3",cluster="optimus",datacenter="DC-06",svm="DmoNAS_optimus"} 0
svm_nfs_write_avg_latency{nfsv="v3",cluster="optimus",datacenter="DC-06",svm="DmoNAS_prime-mc"} 0
svm_nfs_write_avg_latency{nfsv="v3",cluster="optimus",datacenter="DC-06",svm="svm_certl_nas"} 0
node_nfs_write_avg_latency{datacenter="DC-06",nfsv="v4",cluster="optimus",node="optimus-02"} 0
node_nfs_write_avg_latency{datacenter="DC-06",nfsv="v4",cluster="optimus",node="optimus-01"} 0
svm_nfs_write_avg_latency{datacenter="DC-06",nfsv="v4.1",cluster="optimus",svm="DmoESX_optimus"} 0
svm_nfs_write_avg_latency{datacenter="DC-06",nfsv="v4.1",cluster="optimus",svm="DmoESX_prime-mc"} 0
svm_nfs_write_avg_latency{datacenter="DC-06",nfsv="v4.1",cluster="optimus",svm="DmoNAS_optimus"} 0
svm_nfs_write_avg_latency{datacenter="DC-06",nfsv="v4.1",cluster="optimus",svm="DmoNAS_prime-mc"} 0

[wally007]

no2, -> thank you. I will make use of that when I try to modify/create new Dashboard.

no1, -> I'm not sure I follow. In your text output you can see volume constituents. I just tried your suggestion and it is even more confusing to me.

I'm trying to avoid having constituents in the output completely and to have all performance and space usage statistics for FlexGroups only - but any combinations I try in volume.yaml it always takes constituents as well (i'm not sure why this is a default setting though , as i dont think anyone using FlexGroups monitors each sub-volume separately)

Thank you

[rahulguptajss]

@wally007 you can ignore all flexvol style by adding a rule - style `flexvol in exclude_equals
Docs @ https://github.com/NetApp/harvest/blob/18d4b2105b385f953adb18adcd98300454464282/cmd/poller/plugin/README.md#exclude_equals
Example below.

plugins:
  LabelAgent:
    # metric label zapi_value rest_value `default_value`
    value_to_num:
      - new_status state online online `0`
    exclude_equals:
      - style `flexgroup_constituent`
      - style `flexvol`
    replace:
      - style style `flexgroup_constituent` `flexgroup`
  Aggregator:
    volume<style=flexgroup>volume node,svm,aggr,style

[wally007]

I assume to make it really work I have to also remove "replace" , correct ?

With your config as-is i was getting the constituents with the label "flexgroup"

plugins:
  LabelAgent:
    # metric label zapi_value rest_value `default_value`
    value_to_num:
      - new_status state online online `0`
    exclude_equals:
      - style `flexgroup_constituent`
      - style `flexvol`
    replace:
      - style style `flexgroup_constituent` `flexgroup`
  Aggregator:
    volume<style=flexgroup>volume node,svm,aggr,style

volume_labels{datacenter="EQX",cluster="TROY",volume="pcap_1Gbit__0002",node="TROY-02",svm="troy_svm",aggr="TROY_02_NVME_SSD_1",style="flexgroup",state="online",is_sis_volume="true"} 1.0

after I removed "replace", I'm finally getting only flexgroups

plugins:
  LabelAgent:
    value_mapping:
      - status state online `1`
    # metric label zapi_value rest_value `default_value`
    value_to_num:
      - new_status state online online `0`
    exclude_equals:
      - style `flexgroup_constituent`
      - style `flexvol`
  Aggregator:
    volume<style=flexgroup>volume node,svm,aggr,style

volume_labels{datacenter="EQX",cluster="TROY",volume="pcap_1Gbit",node="",svm="troy_svm",aggr="",style="flexgroup",state="online",is_sis_volume="true"} 1.0

[rahulguptajss]

yes, you can remove flexgroup_constituent as well by removing replace.

[wally007]

Thank you.
I just noticed that if FlexGroup is hosted on both nodes and aggregates (in HA pair) the volume performance statistics are reported per node ( which leads to single volume showing up 2 times in read and write performance stats ) for each aggregate/node.

[rahulguptajss]

@wally007 Could you file an issue for that. We'll take a look. Thanks!

[wally007]

done.
#706
Thank you.