Fluentd

Note

This plugin is currently being groomed to be an integration. It works, but we haven’t spent a lot of time on it yet to optimize the user experience. Feel free to try it out, just keep in mind that we will be enhancing the experience over time.

Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data. This is a fork of the Telegraf Fluentd plugin.

Setup

The fluentd plugin is included with the AppOptics host agent by default. Please follow the directions below to enable it for a host agent instance.

Prerequisites

Install Fluentd and open the configuration file /etc/td-agent/td-agent.conf.

$ sudo nano /etc/td-agent/td-agent.conf

Add the following code block into the configuration:

<source>
  @type monitor_agent
  bind 0.0.0.0
  port 24220
</source>

Save your changes and restart the Fluentd agent:

$ sudo /etc/init.d/td-agent restart

Note

See the monitoring page for more information on configuring the Fluentd agent.

You can check to see if metrics are coming in by running the following command (update the URI as needed):

$ curl http://host:24220/api/plugins.json
{
  "plugins":[
    {
      "plugin_id":"object:3fec669d6ac4",
      "type":"forward",
      "output_plugin":false,
      "config":{
        "type":"forward"
      }
    },
    {
      "plugin_id":"object:3fec669dfa48",
      "type":"monitor_agent",
      "output_plugin":false,
      "config":{
        "type":"monitor_agent",
        "port":"24220"
      }
    },
    {
      "plugin_id":"object:3fec66aead48",
      "type":"forward",
      "output_plugin":true,
      "buffer_queue_length":0,
      "buffer_total_queued_size":0,
      "retry_count":0,
      "config":{
        "type":"forward",
        "host":"192.168.0.11"
      }
    }
  ]
}

Configuration

The host agent provides an example configuration file to help you get started quickly. It defines the plugin and task file to be loaded by the agent, but requires you to provide the correct settings for your Fluentd installation. To enable the plugin:

  1. Make a copy of the fluentd example configuration file /opt/appoptics/etc/plugins.d/fluentd.yaml.example, renaming it to /opt/appoptics/etc/plugins.d/fluentd.yaml:
$ sudo cp /opt/appoptics/etc/plugins.d/fluentd.yaml.example /opt/appoptics/etc/plugins.d/fluentd.yaml
  1. Edit the configuration file with settings specific to your fluentd install:
$ sudo nano /opt/appoptics/etc/plugins.d/fluentd.yaml

In this configuration you’ll notice that it is already configured for localhost. If fluentd is running on another server you will need to edit the configuration to reflect the correct endpoint:

collector:
  fluentd:
    all:
      ## This plugin reads information exposed by fluentd (using /api/plugins.json endpoint).
      ##
      ## Endpoint:
      ## - only one URI is allowed
      ## - https is not supported
      endpoint: "http://localhost:24220/api/plugins.json"
      ## Define which plugins have to be excluded (based on "type" field - e.g. monitor_agent)
      exclude: |-
        monitor_agent
        dummy

load:
  plugin: snap-plugin-collector-bridge-fluentd
  task: task-bridge-fluentd.yaml

Save your changes and return to the command prompt.

  1. Restart the agent after any configuration changes have been made:
$ sudo service appoptics-snapteld restart
  1. Enable the Fluentd plugin

On the Integrations Page you will see Fluentd available if the previous steps were successful. It may take a couple minutes before the Fluentd plugin is identified.

Select the Fluentd plugin to open the configuration menu in the UI, and enable the plugin. If you do not see the plugin, see Troubleshooting.

Metrics

The table below outlines the default set of metrics collected by the fluentd plugin.

Namespace Description
fluentd.buffer_queue_length The length of the buffer queue.
fluentd.buffer_total_queued_size How many bytes of data are buffered in Fluentd for a particular output.
fluentd.retry_count How many times Fluentd retried to flush the buffer for a particular output.

Tags

The table below outlines the default set of tags provided for each metric.

Tags Description
plugin_category Category of the plugin
plugin_id ID of the plugin
plugin_type Plugin Type
hostname Name of the host. Instead of using this tag we recommend using the @host alias.