downsampling

Downsampling Template

Provided by: Russ Savage

This InfluxDB Template provides a set of Tasks that allow you to downsample your data for common Telegraf Input Plugins. It also includes a dashboard and a custom monitoring task to ensure everything is running smoothly.

Structure

This isn't a Template that you install with a single command. Each subfolder contains a telegraf configuration as well as a set of tasks to perform downsampling on that telegraf configuration. The following assumptions have been made:

• The bucket name for all Telegraf metrics is simply telegraf. If you are storing your Telegraf metrics in a different bucket, you will need to update the fromBucket variable at the top of each Task.
• The naming convention for the downsampled buckets is telegraf_<interval>. For example, for 1h rollups, the data is stored in telegraf_1h.
• The metrics being calculated for each field are as follows:
  • String Fields: last, count
  • Numeric Fields: last, count, mean, min, max, sum, p99.9, p99, p95, p90
  • Bool Fields: last, count, ratio (count(true)/count())
• All data is downsampled from the raw data bucket (telegraf) which is kept forever but can (and should) be changed. The minimum required time to keep the bucket in order to get the rollups is 24h + 5m in order to get the downsampled values.
• The tasks are named <num>. downsample_<plugin>_<every>

Included Resources

- Downsampling Buckets: `telegraf`, `telegraf_1m`, `telegraf_10m`, `telegraf_1h`, `telegraf_1d`
- Telegraf Configurations for each input plugin
- 1 Dashboard: `Downsampling Status`
- 1 Task: `00. Downsample Monitoring Task`

There are two paths you can take with this template: If you have a specific Telegraf plugin you would like to downsample your data for, you can leverage the specific downsampling tasks for that plugin located in the folders (see the list below). You might also choose this path if the task for downsampling everything is timing out. If you don't really care and would like to downsample all data from all plugins, you can use the tasks in the all_inputs directory. It will grab all the measurements this template can downsample data for and process them.

Note: You should not install the all inputs tasks as well as the individual plugin downsampling tasks. Just choose one or the other.

Telegraf Plugin	Subdirectory
all	all_inputs
cpu	default_inputs
disk	default_inputs
diskio	default_inputs
internal	internal_input
kernel	default_inputs
mem	default_inputs
net	net_inputs
netstat	net_inputs
processes	default_inputs
procstat	procstat_input
smart	smart_input
swap	default_inputs
system	default_inputs

Setup Instructions

General instructions on using InfluxDB Templates can be found in the use a template document.

The first thing you need to do to use the templates that are included for each of the Telegraf plugins is to set up the buckets, dashboard, and monitoring tasks. You can do that with the following command:

./influx apply -f buckets.yml -f dashboard.yml -f monitoring_task.yml

This will lay the foundation for the other parts of this template. Next, choose either the all plugins approach or a directory for the specific Telegraf input plugin metrics you are interested in downsampling. If you don't see the plugin listed, open a PR and add it.

You can then add the downsampling tasks using the following command:

./influx apply -f default_inputs/downsampling_tasks.yml

You can also host the telegraf config in InfluxDB if you like but it will probably be more useful to compare that to your telegraf inputs already configured and make any adjustments. If you do want to use the Telegraf Configs provided, you will need to set $INFLUX_TOKEN and $INFLUX_ORG env variables.

• This can be found on the Load Data > Tokens page in your browser: export INFLUX_TOKEN=TOKEN
• Your Organization name can be found on the Settings page in your browser: export INFLUX_ORG=my_org

Contact

Author: Russ Savage

Email: [email protected]

Github: @russorat

Influx Slack: @russ