Need Information Regarding the scrape interval & collector schedule

[jmg011]

I need some information about the below use-case.

1. If Prometheus servers are scraping Harvest targets (end-points) every minute, and the default zapi collectors have below schedule, how can I ensure that every poll has the raw data collected without aggregation from Netapp for any metric? Do I need to set "data: 60s" ?

From: zapi default yaml file:
schedule:

• instance: 600s
• data: 180s

2. If Prometheus scrapping takes more than 1 minute for a target, how does the harvest cached data work with the scheduled collectors?

3. Can you point me to the zapi document for 7-mode? I could find cmode document but not 7-mode API document.

[cgrinds]

The Zapi collectors don't do aggregation and Harvest's Prometheus exporter will cache the data it was given by the collector for 3m. This value can be changed with the key-value cache_max_keep in the exporter section of your harvest.yml

In your scenario, if Prometheus polls every minute and the collector polls every 3m. You'll have two out of every three Prometheus polls that return cached data.

Zapi Documentation for 7-mode

Can you say more about what you're looking for 7-mode wise? The way Harvest constructs, sends, and parses ZAPIs is the same between cDOT and 7-mode, but I don't think that's what your asking.

• Zapi 7-mode templates
• Zapi cDOT templates

[jmg011]

@cgrinds Thank you for the information.

Please correct me if I did not understand it properly.

I have data collected every 180s (3m), so the values of the metrics (latency, ops, throughput etc.) from Netapp are point in time values (not an average of 3 minutes, but cached the point in time value for 3 minutes)?. That means in Prometheus, 3-consecutive samples will be the exact same value because with 1-minute of scrape cycle, it is scraping the same cached value from Harvest for 3 scrape cycles.

In order for me to have non-cached data every 1 minute in Prometheus, it is recommended to use 60s data poll duration?

Regarding 7-mode, I enabled many custom templates for c-mode for the metric collection but the same APIs are not available for 7-mode data collection, at least I do not see the same attributes and counters. I want to find those correct APIs so I can enable some custom templates for 7-mode data collection as well. And yes, the documentation provided in 523 is what I'm looking for.

[cgrinds]

you've got it!

You mention latency, ops, xput. Those come from the ZapiPerf collector, which does some aggregation (earlier we were talking about the Zapi collector, which doesn't). The ZapiPerf collector needs two consecutive polls for calculations for rate, average, etc. And of course, some perf counters don't need two polls.

By default ZapiPerf collects data every 1m, which is a better fit freshness-wise, with your per minute Prometheus poll.

7-mode

Yes, some of the APIs are missing in 7-mode since those APIs were added to cDOT many years after 7-mode stopped receiving updates. Hopefully the pdf @ruanruijuan pasted in #523 has the information you need.

[jmg011]

@cgrinds Thank you for explaining in details. The document says "The collector collects a dynamic set of metrics. The metric values are calculated from two consecutive polls (therefore, no metrics are emitted after the first poll)"

• Does it mean that if my poll cycle is 3m, the aggregation happens at 6th minute?
• How do I find that aggregation applies the delta, average or rate for IOPS, Latency and Throughput?
• Is there a way to change the aggregation to RAW for specific type of metric collection like Latency and Ops to get raw samples?
• If I set poll cycle to 1 minute, it will ensure RAW data, correct? and poll cycle above 1 minute will try to aggregate consecutive polled data?

Regarding 7-mode, I'm still looking. I will reach out if I need any help.

[cgrinds]

• If you change your ZapiPerf data schedule to every 3m. You'll get something like this

---X---3m---6m---9m---12m time
   P   A    B    C     D

P is the first poll after you restart the poller. Since it's the first poll, no aggregated metrics will be emitted until another 3m have elapsed and deltas are calculated at poll A. Then every subsequent poll (B through D) will emit metrics since the one previous poll is cached.

• Aggregation is mostly dictated by the metadata returned from the ONTAP perf-* ZAPIs. The ZapiPerf collector PollCounter method sends the perf-object-counter-list-info ZAPI with the value of the query: from each Harvest template. This is the same information Harvest returns when using bin/harvest zapi -p infinity show counters --object volume). One of the pieces of metadata returned from this ZAPI is properties and it has values like: average, delta, percent, rate, raw, string. The addCounter method parses this metadata and saves it for later when the data is polled.

• Line 513 in zapiperf.go PollData is where the aggregations begin.

• There is a fair bit of logging around getting/setting properties, aggregation, metrics exporting. Another option is to enable debug logging to get a better idea how this works.

Is there a way to change the aggregation to RAW for specific type of metric collection like Latency and Ops to get raw samples?

• You can override the counter property that ONTAP returns by specifying an override in your template. You can see several examples of where we already do this in templates to fix ONTAP bugs.

For example, here's an example with lun. ONTAP says xcopy_reqs is a delta, when really it's a rate.

bin/harvest zapi -p u2 show counters --object lun | rg "properties|name"

    [name]        -  xcopy_reqs
    [properties]  -       delta  <=== ONTAP says delta, Harvest says otherwise

The conf/zapiperf/cdot/9.8.0/lun.yaml file includes

override:
  - writesame_reqs: rate
  - writesame_unmap_reqs: rate
  - caw_reqs: rate
  - unmap_reqs: rate
  - xcopy_reqs: rate             <==== override ONTAP's delta property with rate instead

If I set poll cycle to 1 minute, it will ensure RAW data, correct? and poll cycle above 1 minute will try to aggregate consecutive polled data?

No, the polling schedule doesn't change whether the data is aggregated or not. Schedule only controls when the data is polled. Aggregation is controlled by the properties mentioned above and the logic in the PollData method.

[jmg011]

@cgrinds Thank you so much for answering all the questions in details. Super useful.

[cgrinds]

Ah, you're probably asking about 7-mode documentation for #523