Kubernetes

Overview

This plugin collects runtime metrics from Kubernetes masters and nodes within its cluster. It gathers information about resource usage and performance characteristics.

Note

This plugin is currently only available for x86_64 Linux.

Setup

The kubernetes plugin is included with the AppOptics host agent by default, please follow the directions below to enable it on a given host (each Kubernetes Master).

Prerequisites

This plugin requires that the host agent user appoptics has access to a valid kubeconfig and kube-apiserver.

Note

A kubeconfig file is used to configure access to Kubernetes clusters and is commonly found in ~/.kube/config. kubeconfig is a generic way of referring to the Kubernetes configuration files.

There are different ways to give the host agent access to Kubernetes clusters (to the Kubernetes API). One method is to copy your ~/.kube folder to /opt/appoptics/etc:

$ cd /opt/appoptics/etc
$ sudo cp -r ~/.kube .
$ sudo chown -R appoptics:appoptics .kube

To check that the host agent has access to the Kubernetes API, you can run the following command as the appoptics user, which retrieves the list of all pods in the current namespace:

$ sudo -u appoptics kubectl get pods --kubeconfig /opt/appoptics/etc/.kube/config
If the command does not execute successfully, ensure:
  1. both the symlinks and source files are owned by the appoptics group and user.
  2. the configured kubeconfigpath is valid.

Configuration

The host agent provides an example configuration file to help you get started quickly. It defines the plugin and task file to be loaded by the agent, but requires you to provide the correct settings for your Kubernetes deployment. To enable the plugin:

  1. Make a copy of the kubernetes example configuration file /opt/appoptics/etc/plugins.d/kubernetes.yaml.example, renaming it to /opt/appoptics/etc/plugins.d/kubernetes.yaml:
$ sudo cp /opt/appoptics/etc/plugins.d/kubernetes.yaml.example /opt/appoptics/etc/plugins.d/kubernetes.yaml
  1. Update the /opt/appoptics/etc/plugins.d/kubernetes.yaml configuration file with settings specific to your Kubernetes deployment, for example:
collector:
  kubernetes:
    all:
      incluster: false
      kubeconfigpath: "/opt/appoptics/etc/.kube/config"
      interval: "60s"

Note

  • incluster must be set to false (set to true only when running as a pod or daemonset).
  • kubeconfigpath a valid path to Kubernetes config files (e.g. ~/.kube/config).
  • interval sets time interval between subsequent metrics submissions (optional, by default set to 60s)
  1. Restart the host agent:
$ sudo service appoptics-snapteld restart
  1. Enable the Kubernetes plugin in the AppOptics UI

    On the Integrations Page you will see the Kubernetes available if the previous steps were successful. If you do not see the plugin, see Troubleshooting.

    Select the Kubernetes plugin to open the configuration menu in the UI, and enable the plugin.

    You should soon see the kubernetes metrics reported to your dashboard.

Metrics and Tags

All the metrics are fetched via the Kubernetes API. The tables below outline the default set of metrics collected by the kubernetes plugin along with the optional metrics available.

Pod Metrics

Namespace Description
kubernetes.pod.status.condition.ready specifies if the pod is ready to serve requests
kubernetes.pod.status.condition.scheduled status of the scheduling process for the pod
kubernetes.pod.status.phase.Pending This includes time before being bound to a node, as well as time spent pulling images onto the host
kubernetes.pod.status.phase.Running The pod has been bound to a node and all of the containers have been started.
kubernetes.pod.status.phase.Succeeded All containers in the pod have voluntarily terminated with a container exit code of 0, and the system is not going to restart any of these containers
kubernetes.pod.status.phase.Failed All containers in the pod have terminated, and at least one container has terminated in a failure
kubernetes.pod.status.phase.Unknown For some reason the state of the pod could not be obtained, typically due to an error in communicating with the host of the pod

Pod Metrics Tags

Tag Name Description
hostname Name of the host. Instead of using this tag we recommend using the @host alias
Pod Name of container group (Pod)
Node Kubernetes node name
Namespace Kubernetes namespace

Container Metrics

Namespace Description
kubernetes.container.requested.cpu.cores The limit on cpu cores to be used by a container.
kubernetes.container.limits.memory.bytes The limit on memory to be used by a container in bytes.
kubernetes.container.requested.cpu.cores The number of requested cpu cores by a container.
kubernetes.container.requested.memory.bytes The number of requested memory bytes by a container.
kubernetes.container.status.ready specifies whether the container has passed its readiness probe
kubernetes.container.status.restarts number of times the container has been restarted
kubernetes.container.status.running value 1 if container is running else value 0
kubernetes.container.status.terminated value 1 if container is terminated else value 0
kubernetes.container.status.waiting value 1 if container is waiting else value 0

Container Metric Tags

Tag Name Description
hostname Name of the host. Instead of using this tag we recommend using the @host alias
pod Name of container group (Pod)
node Kubernetes node name
namespace Kubernetes namespace
container Kubernetes container name

Node Metrics

Namespace Description
kubernetes.node.spec.unschedulable Whether a node can schedule new pods.
kubernetes.node.status.allocatable.cpu.cores The CPU resources of a node that are available for scheduling.
kubernetes.node.status.allocatable.memory.bytes The memory resources of a node that are available for scheduling.
kubernetes.node.status.allocatable.pods The pod resources of a node that are available for scheduling.
kubernetes.node.status.capacity.cpu.cores The total CPU resources of the node.
kubernetes.node.status.capacity.memory.bytes The total memory resources of the node.
kubernetes.node.status.capacity.pods The total pod resources of the node.
kubernetes.node.status.outofdisk True if there is insufficient free space on the node for adding new pods, otherwise False

Node Metric Tags

Tag Name Description
hostname Name of the host. Instead of using this tag we recommend using the @host alias
Node Kubernetes node name

Deployment Metrics

Namespace Description
kubernetes.deployment.metadata.generation The desired generation sequence number for deployment. If a deployment succeeds should be the same as the observed generation.
kubernetes.deployment.status.observedgeneration The generation sequence number after deployment.
kubernetes.deployment.status.targetedreplicas Total number of non-terminated pods targeted by this deployment (their labels match the selector).
kubernetes.deployment.status.unavailablereplicas Total number of unavailable pods targeted by this deployment.
kubernetes.deployment.status.availablereplicas Total number of available pods (ready for at least minReadySeconds) targeted by this deployment.
kubernetes.deployment.spec.desiredreplicas Number of desired pods.
kubernetes.deployment.status.deploynotfinished If desired and observed generation are not the same, then either an ongoing deploy or a failed deploy
kubernetes.deployment.spec.paused An optional boolean field for pausing and resuming a Deployment.
kubernetes.deployment.status.updatedreplicas Total number of non-terminated pods targeted by this deployment that have the desired template spec

Deployment Metric Tags

Tag Name Description
hostname Name of the host. Instead of using this tag we recommend using the @host alias
namespace Kubernetes namespace
deployment Kubernetes deployment name