Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with...
Transcript of Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with...
![Page 1: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/1.jpg)
Prometheus
Integrating Long-Term Storagewith Prometheus
Julius Volz, March 30, 2017
![Page 2: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/2.jpg)
Prometheus
Local Storage
Prometheus
data
Good:● Simple.● Avoids clustering.● Reliable with HA setup.
But...● Not durable.● Not long-term.● Not scalable.● No downsampling etc.● Not flexible.
![Page 3: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/3.jpg)
Prometheus
Remote Storage
Issue #10https://github.com/prometheus/prometheus/issues/10
![Page 4: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/4.jpg)
Prometheus
Legacy Write Support
For OpenTSDB, InfluxDB, Graphite
➔ replace with generic interface!
![Page 5: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/5.jpg)
Prometheus
Design Questions
Write Path
● Chunks vs. samples?
● Buffering and retries?
Read Path
● Distributed vs. central evaluation?
● protocol and encoding?
![Page 6: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/6.jpg)
Prometheus
Decision
Start simple, go from there.
![Page 7: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/7.jpg)
Prometheus
Write Path
![Page 8: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/8.jpg)
Prometheus
Samples vs. Chunks
Send samples. Prometheus has internal
chunking, but...
● What chunk size?
● What about write delay?
● Remote end not always a storage.
● Simplicity of remote end.
![Page 9: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/9.jpg)
Prometheus
Buffering and retries
● Write shards with dynamic parallelism.
● Minimal queueing and retries (memory!).
![Page 10: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/10.jpg)
Prometheus
● HTTP + protobuf.
● Lack of HTTP/2 support “out there”: no gRPC for now.
Protocol and Encoding
![Page 11: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/11.jpg)
Prometheus
Write Path (Native)
Prometheus
data
HTTP + protobuf
sample stream
NativeRemoteStorage
![Page 12: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/12.jpg)
Prometheus
Write Path (Bridge)
Prometheus
data
HTTP + protobuf
sample streamBridge Remote
Storage
custom protocol
![Page 13: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/13.jpg)
Prometheus
Or more generally...
Prometheus
data
HTTP + protobuf
sample streamBridge Arbitrary
Processor
custom protocol
![Page 14: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/14.jpg)
Prometheus
Configuration
remote_write: - url: "https://my-storage/write" write_relabel_configs: <...> remote_timeout: <...> basic_auth: <...> bearer_token: <...> proxy_url: <...> tls_config: <...>
![Page 15: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/15.jpg)
Prometheus
Read Path
![Page 16: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/16.jpg)
Prometheus
Distributed or Centralized Eval?
Distributed evaluation scales better, but:
● Requires remote knowledge of PromQL.
● Local eval good enough for most queries.
● Remote downsampling helps.
➔ Centralized PromQL evaluation for now.
![Page 17: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/17.jpg)
Prometheus
Read Path
Prometheus
data
HTTP + protobuf
request:matchers, ranges
Bridge Remote Storage
custom protocol
response:raw samples
![Page 18: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/18.jpg)
Prometheus
Read Path
Prometheus
data
HTTP + protobuf
request:matchers, ranges
Bridge Remote Storage
custom protocol
response:raw samples
PromQL evaluation in Prometheus!
![Page 19: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/19.jpg)
Prometheus
Future: Query Federation
Prometheus
data
remote read Prometheusshard 0
Prometheusshard 1
Prometheusshard 2
![Page 20: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/20.jpg)
Prometheus
Configuration
remote_read: - url: "https://my-storage/read" remote_timeout: <...> basic_auth: <...> bearer_token: <...> proxy_url: <...> tls_config: <...>
![Page 21: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/21.jpg)
Prometheus
Note!
● Rules query only local data.
Reliability!
● Metadata API only local so far.
● External label magic is applied
(TODO).
![Page 22: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/22.jpg)
Prometheus
Real-World Examples
CHRONIXQAware
Write-only
http://www.chronix.io/https://github.com/ChronixDB/chronix.ingester
WeaveworksRead + write
https://github.com/weaveworks/cortex
![Page 23: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/23.jpg)
Prometheus
Real-World Examples
CHRONIXQAware
Write-only
http://www.chronix.io/https://github.com/ChronixDB/chronix.ingester
WeaveworksRead + write
https://github.com/weaveworks/cortex
![Page 24: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/24.jpg)
Prometheus
Weave Cortex
● Horizontally scalable Prometheus.
● As a service or self-hosted (open source).
● Stores data in AWS (GCP support in the works).
● Normal querying API (use with Grafana, etc.).
● Plus generic remote read + write.
● Use and contribute: https://github.com/weaveworks/cortex.
![Page 25: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/25.jpg)
Prometheus
Weave Cortex
DEMO
![Page 26: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/26.jpg)
Prometheus
Build Your Own!
See protobuf definition at:https://github.com/prometheus/prometheus/blob/master/storage/remote/remote.proto
syntax = "proto3";
package remote;
message Sample {
double value = 1;
int64 timestamp_ms = 2;
}
message LabelPair {
string name = 1;
string value = 2;
}
...
![Page 27: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/27.jpg)
Prometheus
Beware
This is all still experimental.
Watch for the next release.
![Page 28: Integrating Long-Term Storage with Prometheus - Sched Long... · Integrating Long-Term Storage with Prometheus Julius Volz, March 30, 2017. Prometheus ... For OpenTSDB, InfluxDB,](https://reader034.fdocuments.net/reader034/viewer/2022052608/5a9f97717f8b9a7f178d0352/html5/thumbnails/28.jpg)
Prometheus
Thanks!