For a while I’ve been keeping an eye on Prometheus as I’ve been fascinated with its architecture and philosophy. The one small detail that has prevented me from pursuing it as a production tool (for network monitoring specifically) is the limitations around long-term storage of metrics. Prometheus is designed to keep a certain period of data and then expire it out from its local storage. I’ve been monitoring this GitHub issues for quite a while as it expresses the need to allow Prometheus to forward metrics to other time series databases for long-term storage.
Paul Dix presented a talk at PromCon 2017 (blog at InfluxData) announcing that InfluxData have improved on the experimental remote read/write example code and added full support for InfluxDB as a remote storage backend. Metrics from Prometheus can be forwarded to InfluxDB as well as being stored locally for fast analysis for a defined time (e.g. 90 days) and allow PromQL to query data from both sources at the Prometheus layer.
This is something I’d love to spend some time investigating further as I really love the community around Prometheus as well as their design philosophy for the project.