How to deal with Space issues

[darthVikes]

Is your feature request related to a problem? Please describe.

Yes, local disk is getting used up, of course I can add more space but would like additional information on how harvest deals with rotating data. aka is there a way for us to specify how much old data exists? 1m/3m/6m before it removes old data or doesn't it currently remove old data out there?

Describe the solution you'd like
Would like some documentation on how to handle freeing up disk apce with regards to old data in harvest, and/or scripts added to be able to have it automatially clean up older disk space for data older than X.

Describe alternatives you've considered
manual method to quickly free up disk space and automated solution based on a parameter to allow X amounts of storage / older performance data?

Additional context
Depending on this is currently a feature or not this may also require additional tickets to be opened for issues around data grouth and space cleanup for old data. And documentation around said features.

[cgrinds]

hi @darthVikes - not sure if you saw this link in the wiki that discuss how much disk space Prometheus uses.

The out-of-the-box default for Prometheus is to retain data for 15d. Some folks are also using VictoriaMetrics to store and/or downsample older data.

[luddite516]

I ran into this same problem early on, and opened a discussion on it. The metrics have to reside somewhere, so it is up to you where you want the metrics to live. Many of the resources I had looked into, recommended keeping only recent data on the Prometheus servers. One factor in this decision is how much data you have and how many prometheus servers there are.

In our case, we decided to leave the most recent 15 days of metrics on the Prometheus server, and then use Thanos to offload older metrics to object storage. Thanos also performs the downsampling and compaction that allows us to store long-term data efficiently.

It was a lot more work to set this all up, but now that it is running it is easy enough to maintain and it works acceptably well.

HTH!