Accessing Prometheus

Cloudify’s Status Reporter comes with Prometheus at its core. It might be sometimes desirable to access it directly. This is a description of a few ways that might be achieved.

Cloudify’s installation of Prometheus comes with additional monitoring/ prefix to all paths. For example a query endpoint is not located at /api/v1/query but /monitoring/api/v1/query etc.

On the localhost

Prometheus is configured to listen on lo (localhost, 127.0.0.1) interface, on port 9090. To access it, it is necessary to obtain a SSH access to the machine it is running on (look here for details).

Below is the example of using curl to query all up metrics:

curl "http://localhost:9090/monitoring/api/v1/query?query=up"

To see the output nicely formatted and paginated consider using similarly pipe-d commands:

curl "http://localhost:9090/monitoring/api/v1/query?query=up" | python -m json.tool | less

External access on port 8009

Nginx HTTP reverse proxy, bundled with Status Reporter, listens on the port 8009 for incoming HTTPS requests and forwards them directly to Prometheus (running on the same node). The endpoint requires authentication which is configured in Cloudify’s config.yaml in the prometheus.credentials path.

Below is the example of using curl to query all up metrics:

curl \
--cacert /etc/cloudify/ssl/monitoring_ca_cert.pem \
--user a_user:a_password \
"https://172.20.0.3:8009/monitoring/api/v1/query?query=up"

Since the port is opened to the external traffic, it is possible to access Prometheus’s UI from the web browser. Just point it to https://{PUBLIC_IP}:8009/monitoring/graph.

This is the access used to build-up Prometheus federation, i.e. any Prometheus will try talking to other instances on that exact port, using the same HTTPS protocol and the same credentials as mentioned above.

External access with Cloudify’s authentication

Since Cloudify version 7.0.0 it is possible to access Prometheus using native authentication mechanisms – tokens. Tokens are passed as HTTP cookies named XSRF-TOKEN.

Below is an example of setting up and using this method with curl, but the same mechanism could be used to set up data sources in Grafana dashboards.

First a token should be created, here with an expiry date:

cfy token create --expiry +1h

Then the token might be used like that:

curl \
--cacert /etc/cloudify/ssl/cloudify_internal_ca_cert.pem \
--cookie "XSRF-TOKEN=ctok-fyzpRL3LHg-3W6IncAVwzldIKAGll0ADNeo1Vpt72Yy5UKwTrwx" \
"https://172.20.0.3/monitoring/api/v1/query?query=up"