How Much Disk Space Do Prometheus Blocks Use?

Memory for ingestion is just one part of the resources Prometheus uses, let’s look at disk blocks. Every 2 hours Prometheus compacts the data that has been buffered up in memory onto blocks on disk. This will include the chunks, indexes, tombstones, and various metadata.

The main part of this should usually be the chunks themselves, and you can see how much space each sample takes on average by graphing: Combined with rate(prometheus_tsdb_head_samples_appended_total[1h]) for the samples ingested per second, you should have a good idea of how much disk space you need given your retention window. It’s a bit more complicated though, as there’s also indexes to consider. If you have lots of churn in your metrics these can end up taking a non-trivial amount of space.

We can use the jq tool on the meta.json in each block in Prometheus’s data directory to get a better idea of the ratio between samples ingested and ultimate storage costs:

Source: robustperception.io