Prometheus

Note

This plugin is currently being groomed to be an integration. It works, but we haven’t spent a lot of time on it yet to optimize the user experience. Feel free to try it out, just keep in mind that we will be enhancing the experience over time.

Overview

This plugin collects metrics from from HTTP servers exposing metrics in Prometheus format.

Setup

The Prometheus plugin is included with the AppOptics host agent by default. Please follow the directions below to configure it.

Configuration

The host agent provides an example configuration file to help you get started quickly. It defines the plugin and task file to be loaded by the agent, but requires you to provide the correct settings for your HTTP server(s) exposing metrics in Prometheus format. To enable the plugin:

  1. Make a copy of the prometheus example configuration file /opt/appoptics/etc/plugins.d/prometheus.yaml.example, renaming it to /opt/appoptics/etc/plugins.d/prometheus.yaml:
$ sudo cp /opt/appoptics/etc/plugins.d/prometheus.yaml.example /opt/appoptics/etc/plugins.d/prometheus.yaml
  1. Update /opt/appoptics/etc/plugins.d/prometheus.yaml configuration file with settings specific to your HTTP server(s) exposing metrics in Prometheus format, for example:
collector:
  prometheus:
    all:
      ## A list of urls to scrape metrics from.
      urls: |-
        http://localhost:9100/metrics
        http://otherhost:9100/metrics

      ## A list of Kubernetes services to scrape metrics from.
      # kubernetes_services: |-
      #   http://my-service-dns.my-namespace:9100/metrics

      ## Use bearer token for authorization
      # bearer_token: /path/to/bearer/token

      ## Specify timeout duration for slower prometheus clients (default is 3s)
      # response_timeout: "3s"

      ## Optional SSL Config
      # ssl_ca: /path/to/cafile
      # ssl_cert: /path/to/certfile
      # ssl_key: /path/to/keyfile

      ## Use SSL but skip chain & host verification
      # insecure_skip_verify: false

      ## Metric name prefix
      # bridge_prefix: "prometheus"
      # bridge_prefix: ""
load:
  plugin: snap-plugin-collector-bridge-prometheus
  task: task-bridge-prometheus.yaml
  • urls - list of URLs to scrape Prometheus metrics from.
  • kubernetes_services - urls listed here will be expanded by looking up all A records assigned to teh hostname as described in Kubernetes DNS service discovery. This method can be use to locate all of the Kubernetes headless services.
  • bearer_token - the file set on this parameter will be read on each interval and its contents will be appened to the Bearer string in the Authorization header
  • bridge_prefix - sets the default prefix used for metric names ; By default, it is set to prometheus
  1. Restart the host agent:
$ sudo service appoptics-snapteld restart
  1. Enable the Prometheus plugin in the AppOptics UI

    On the Integrations Page you will see the Kubernetes available if the previous steps were successful. If you do not see the plugin, see Troubleshooting.

    Select the Prometheus plugin to open the configuration menu in the UI, and enable the plugin.

    You should be able to select a set of metrics from the metrics tab and create a custom dashboard.

Metrics

The collected metrics will be prefixed with prometheus.* in AppOptics (or the configured value for bridge_prefix on the plugin configuration).

All metrics receive the url tag referencing the source URL in the configuration. If using Kubernetes service discovery, the address tag is also added indicating the discovered ip address.